Production-grade guide to cloud load balancing advanced covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.
Cloud load balancers sit between clients and your compute fleet, and their default configurations often hide the trade-offs that surface at scale. This page is for engineers who need to move beyond “create service → get DNS” and actually understand how traffic is distributed, how failures are detected, and how to tune the balance point without introducing outages. It matters when health checks report green but users see errors, when a single zone absorbs all traffic despite multiple endpoints, or when you need to route based on request headers without modifying every microservice. The following sections are organized by what you’re trying to accomplish: verify distribution, design resilient health checks, manage session affinity, tune timeouts, debug misrouting, and integrate with service meshes or API gateways. Every command shown is a real, shipping tool; every configuration pattern describes a mechanics‑level pattern you can map to your own platform.
Before tuning, confirm that the load balancer is actually cycling traffic across your backend instances. In Kubernetes, the service abstraction masks the underlying cloud LB, so you need to probe both layers.
Check registered endpoints
kubectl get endpoints my-svc -o yamlThis output lists the subset of pod IPs the LB has registered. If a pod is absent, verify that its readiness probe is succeeding and that the cloud LB’s target‑group health check (if using an external LB) is returning healthy.
Probe the live balance point
for i in $(seq 1 10); do curl -s http://my-lb.example.com/health; doneRun this from a client outside the cluster. Compare the source IP or a unique header (e.g., X-Request-Id) across the ten runs. If you consistently see the same backend IP, either session affinity is enabled or the LB scheduler isn’t round‑robin.
Cross‑zone observation In multi‑zone deployments, many L7 load balancers route based on the client’s subnet or DNS geolocation. If you’ve added a new availability zone and traffic isn’t reaching it, verify that the load balancer has listeners attached to subnets in that zone and that the security group permits egress to the pod CIDR in all attached zones. Some providers also have a “zone affinity” setting that, when misconfigured, sends all traffic to the first‑registered zone.
The most dangerous failure mode is a load balancer that considers an endpoint healthy because TCP connect succeeded, while the application layer is saturated or misrouting requests.
Layer 4 vs Layer 7 health check scope A TCP health check only confirms that the kernel’s network stack is responding. An HTTP health check can verify status codes, response‑body patterns, and timing. If your app returns 200 OK but with a degraded payload (e.g., {"error":"degraded"}), a pure 200 check will keep sending traffic to a failing instance.
Multi‑condition health check pattern Configure the load balancer to require: (a) TCP connect, (b) HTTP 2xx status, and (c) response‑body match (e.g., contains a specific success indicator). If your platform supports it, add a timeout and an interval that is slower than your app’s cold‑start duration, so a pod that’s still initializing isn’t marked unhealthy mid‑startup.
Silent failure: health‑check interval vs pod startup duration If the health‑check interval is 5 seconds and a pod takes 45 seconds to become ready after its container starts, the cloud LB may repeatedly mark it unhealthy before the pod finishes initializing. The sharp edge is that the LB’s unhealthiness doesn’t surface in the pod’s own readiness probe—it only appears as “no endpoints available” in the LB’s target list. The fix is to align the LB health‑check timeout with your pod’s startup lifecycle, or use a separate health‑check path that simply returns 200 without side effects.
Pattern, not product‑specific keys Every L7 load balancer lets you set a request path, expected status codes, a timeout window, and a failure threshold. The exact config keys differ by vendor, but the mechanic is identical: the LB issues a GET (or HEAD) to the path, checks the status, and after N consecutive failures marks the target unhealthy. Map this pattern to your platform’s annotation, API field, or UI tab without relying on a vendor’s default.
Session affinity (or “stickiness”) ensures that all requests from a single client are routed to the same backend. It’s a blunt instrument that trades simplicity for scalability.
Client‑IP stickiness Most load balancers implement this by hashing the source IP modulo the number of healthy backends. The sharp edge: the hash table has a finite size. When the backend set grows or shrinks (autoscaling event), the hash distribution shifts, and many clients are remapped to different pods—often breaking stateful sessions that aren’t backed by an external store.
Why token‑based affinity scales better Instead of IP hashing, pass a session identifier (e.g., a JWT session ID or an app‑level ticket) in a request header or cookie, and configure the LB to hash that value. The hash space is then tied to the identifier, not the client IP, so adding or removing pods only affects clients whose identifier hash falls in the rebalanced range—typically a small fraction.
Confused options Engineers often set sessionAffinity: ClientIP in a Kubernetes Service spec expecting it to work like cookie‑based stickiness. In Kubernetes, ClientIP affinity is implemented via iptables rules that have limited entropy; it does not insert a cookie, and it doesn’t survive an LB restart or a cloud‑provider‑managed LB reconfigure. If you need cookie‑based stickiness, you must configure the LB’s listener rule to insert-cookie or set a Set-Cookie header, and the client must support it.
Command‑level check
kubectl get svc my-svc -o jsonpath='{.spec.sessionAffinity}'This returns None, ClientIP, or None (if using an external LB). If you see ClientIP and you’re observing session breaks during scale events, switch to a token‑based approach or move session state to a distributed store (Redis, Memcached) and drop stickiness entirely.
Default timeout values are calibrated for web‑request latency, not for long‑lived transfers, streaming, or background job processing.
Connection timeout Many cloud LBs default to a 5‑second connection timeout. If your app accepts WebSocket upgrades, gRPC streams, or request bodies larger than a few kilobytes within 5 seconds, the LB will tear down the connection and return 502/504. Raise the connection timeout to at least 2–3× your longest expected handshake latency, or configure the LB to keep‑alive connections when the backend supports it.
Idle timeout Idle timeout controls how long the LB keeps a connection open without sending data. Defaults are often 300 seconds (5 minutes). For APIs with periodic keep‑alive packets, this is fine. For long‑running batch imports or SSE connections, you’ll need to raise this to 600–900 seconds, or explicitly configure the LB to not terminate idle connections.
Request timeout Separate from connection timeout, the request timeout applies per request. If your backend processes take 2 minutes, set the request timeout to at least 3 minutes. Some LBs also have a “request body timeout” that fires if the client takes too long to send the body—useful for slow‑upload scenarios, but easy to misfire on high‑latency mobile networks.
Connection‑rate limits Burst traffic can hit the LB’s connection‑rate limit, causing 503 responses even when backends have capacity. The sharp edge: the rate limit is often per‑IP or per‑availability‑zone, not global. If you’re autoscaling across zones, a sudden influx may saturate one zone’s LB rate limit while other zones are underutilized. The fix is to over‑provision zone count or configure a global rate‑limiter upstream (e.g., Envoy, Cloudflare) if your traffic pattern is bursty.
Pattern (generic) Most LBs expose three timers: connection‑timeout, idle‑timeout, request‑timeout. Adjust each to at least 2× your observed 99th‑percentile latency for the given connection type. If your platform uses a config file or API patch, the fields are typically named connection_drain_timeout, session_timeout, and request_timeout—but the exact key name varies. Consult your LB’s documentation for the mapping.
Traffic that appears to be going to the right service may actually be hitting the wrong endpoint due to DNS caching, LB misconfiguration, or network policy gaps.
DNS TTL and propagation Load balancer DNS names often have short TTLs (60–300 seconds). If you recently changed the LB configuration, a client with a cached A record may still hit the old endpoint for up to the TTL duration. The sharp edge: some providers don’t honor TTL reductions on existing records until the record expires, especially in private DNS integrations.
Check the LB’s target registration
kubectl describe service my-svc | grep -A 20 'Endpoints'Look for the Endpoints field. If it lists IPs from only one zone, the LB is likely routing only to that zone, or health checks in other zones are failing.
Security group and network policy A common silent failure: the LB’s security group allows inbound traffic on the listener port, but the egress rule to the pod CIDR is missing or too restrictive. The LB can accept the connection but never receives a response, resulting in a client‑visible timeout or 504. Verify that the security group attached to the LB permits traffic to the node/pod IP range in all zones the LB spans.
Request path mismatches In L7 load balancers, the listener rule may forward based on a path prefix (e.g., /api/* → backend A, /static/* → backend B). If the LB is matching the longest prefix incorrectly, requests for /api/users might go to the static backend. Debug by curling the LB directly with specific paths and observing the backend IP via response headers (e.g., X-Backend-Name if your app injects it, or via kubectl get pods -o wide after the request).
DNS and LB name mismatch Ensure the client is resolving the intended LB DNS name. Run dig my-lb.example.com and compare the returned IPs with the LB’s configured subnet IPs. If the DNS resolution returns an anycast IP or a stale endpoint, flush the local resolver cache or use dig +short from a fresh client.
Modern architectures often place a service mesh (e.g., Istio, Linkerd) or an API gateway (e.g., Kong, Envoy) in front of or alongside the cloud load balancer. Understanding the boundary between them prevents routing loops and misplaced retries.
Mesh‑to‑LB boundary A service mesh’s ingress gateway typically exposes a single external IP or DNS name that then distributes traffic to internal services. The cloud LB sits above or below the mesh depending on deployment style. If the cloud LB is above the mesh, health checks and session affinity settings on the cloud LB may conflict with the mesh’s own load‑balancing logic. If the cloud LB is below the mesh, the mesh sees a single upstream target and applies its own service‑level load balancing (usually round‑robin or least‑connections).
Avoiding double‑load‑balance The sharp edge: running both a cloud LB and a mesh ingress gateway introduces two layers of timeout, health‑check, and affinity configuration. A request that times out at the cloud LB never reaches the mesh; a request that is routed correctly but meets a mesh‑level retry may hit the cloud LB again as a 502. To debug, insert a unique request‑id header at the client, trace it through the LB’s access logs (if available), and the mesh’s telemetry. If the same request‑id appears at both layers without a corresponding mesh retry, the cloud LB is the termination point.
API gateway routing rules When an API gateway sits in front of the cloud LB, the gateway often performs its own request validation, rate limiting, and auth checks before forwarding to the LB. The LB then sees forwarded requests with modified headers (e.g., X-Forwarded-For, X-Forwarded-Proto). Ensure the LB’s health check and affinity settings account for these forwarded headers; some LBs hash on X-Forwarded-For by default, which, behind a gateway, always resolves to the gateway’s IP, collapsing affinity to a single backend.
Pattern‑level takeaway Treat the cloud LB as the edge‑of‑infrastructure distributor and the mesh/gateway as the intra‑service distributor. Keep health checks, timeouts, and affinity as coarse as possible at the edge, and let the mesh/gateway handle fine‑grained routing. If you need request‑level routing (canary, blue‑green), prefer the mesh’s traffic‑splitting API over adjusting the cloud LB’s listener rules, because mesh‑level splits are atomic and reversible without reconfiguring the external LB.
X-Forwarded-For collapses to the gateway’s IP when an API gateway front‑ends the LB, effectively routing all traffic to one backend. This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy