Load Balancing Strategies

Load balancing strategies determine how incoming network traffic is distributed across a set of backend resources. They come into focus whenever you run more than one instance of a service and need to prevent overload, maintain availability under failure, or optimize resource utilization without over-provisioning. This page organizes the problem space by what you’re trying to accomplish, offers concrete commands and configuration patterns from widely deployed tooling, and highlights the failure modes that only appear after a production incident.

Distributing traffic across instances

The most common requirement is simply to spread new connections across all available backends. Most layer-4 and layer-7 implementations use a round-robin algorithm by default: each new connection is assigned to the next endpoint in a cyclical list. In Kubernetes, a Service of type ClusterIP or LoadBalancer implements this automatically; the kube-proxy component or cloud controller translates the Service selector into endpoint groups and cycles through them.

You can verify the distribution pattern with a simple loop:

for i in {1..15}; do
  curl -s -o /dev/null -w "Code %{http_code} Time %{time_total}s\n" http://my-svc.my-namespace.svc.cluster.local
done

The -w flag formats curl’s output using libcurl’s standard specifiers; %{http_code} prints the response status and %{time_total} records the full request duration. No vendor‑specific syntax is required.

When weighted distribution is needed—some backends have more capacity than others—the policy is expressed in the service or ingress specification rather than in the balancer’s core loop. For example, an ingress controller may accept a weight annotation on backend pods, or a cloud load balancer may read service.beta.kubernetes.io/amazonaws.com/load-balancer-target-weight labels. The exact key depends on the data plane, but the pattern is always: attach a numeric weight to each endpoint, and the balancer normalizes those weights into its selection probability.

If you need to change the service’s session handling mid‑lifecycle, kubectl patch can update the spec without a full recreate:

kubectl patch svc my-svc -p '{"spec":{"sessionAffinity":"ClientIP"}}'

The sessionAffinity field toggles whether the data plane attempts to route subsequent requests from the same source IP to the same backend. It is a boolean flag; there is no per‑connection weight attached to it.

Terminating TLS at the load balancer

When traffic arrives over HTTPS, the load balancer can terminate TLS and forward decrypted HTTP to backends. This offloads encryption cost from application servers and simplifies certificate management, since only the edge needs a valid public‑key pair.

In Kubernetes, TLS termination is configured through an Ingress resource that references a secret holding the private key and certificate:

kubectl create secret tls tls-cert \
  --key /path/to/private.key \
  --cert /path/to/public.crt

Once the secret exists, the Ingress spec references it by name:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
spec:
  tls:
  - hosts:
    - app.example.com
    secretName: tls-cert
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-svc
            port:
              number: 80

To test that the handshake completes and the backends receive plain HTTP, use curl with verbose mode:

curl -kv https://app.example.com 2>&1 | head -30

The -k flag permits connections to servers with invalid certificates (useful for internal testing); in production you would omit it and ensure the presented certificate is trusted. The -v output shows the TLS version, cipher, and SNI sent by the client, which you can cross‑check against the backend’s expected configuration.

If your environment uses a cloud provider’s native load balancer, the TLS configuration may live in the cloud console or via a cloud‑specific CLI. The underlying pattern—presenting a certificate, configuring a listener on port 443, and forwarding port 80 or 8080 to backends—remains the same across platforms.

Pinning users to specific backends

Session affinity (sometimes called “sticky sessions”) ensures that all requests from a particular client are routed to the same backend for the duration of a session. This is essential when backends maintain local state that isn’t replicated, such as an in‑memory cache or a TLS session ticket.

The most common affinity key is the client’s source IP address. In Kubernetes, setting sessionAffinity: ClientIP on a Service causes the data plane to hash the source IP and map it to a specific endpoint. The affinity timeout defaults to 3600 seconds but can be adjusted:

kubectl patch svc my-svc -p '{"spec":{"sessionAffinity":"ClientIP","sessionAffinityConfig":{"clientIP":{"timeoutSeconds":7200}}}}'

The sessionAffinityConfig key is part of the Kubernetes API; its nested structure varies slightly between versions but the timeoutSeconds field is consistent in recent releases.

A frequent misconfiguration occurs when the frontend proxy (e.g., a cloud load balancer or edge NGINX) forwards the original client IP via X-Forwarded-For instead of using the direct source IP. The downstream load balancer then hashes a header value that rarely matches any backend’s mapping, resulting in every request being sent to a random pod. The sharp‑edge section below covers this in more detail.

When backends are replaced or scaled up/down, affinity breakage is common. A pod restart, node drain, or HPA scale‑out event changes the endpoint set, and clients whose IP‑based affinity has timed out will be redistributed. There is no magic “preserve sessions across scale events” flag; the application must either replicate state or accept that affinity is a best‑effort mechanism.

Detecting and reacting to unhealthy backends

Health checking is the mechanism by which a load balancer learns which endpoints are capable of responding to requests. Probes may be layer‑4 (TCP connect) or layer‑7 (HTTP GET/HEAD), and they are configured independently of the traffic‑distribution algorithm.

In Kubernetes, liveness and readiness probes are defined in the pod spec:

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 10

The initialDelaySeconds field tells the kubelet to wait before sending the first probe; periodSeconds controls the frequency. If the HTTP response code is outside the 2xx or 3xx range, the endpoint is marked NotReady and the load balancer stops sending new connections to it.

You can inspect the current state of endpoints from the command line:

kubectl get endpoints my-svc

The output lists each ready endpoint alongside its allocated addresses. If an endpoint is missing from the list, its probe is likely failing.

A silently failing health check is one of the most common production breakages. If the probe path returns a 200 OK but the application is actually unable to serve traffic (e.g., a database connection pool is exhausted), the load balancer continues routing traffic to that pod, causing cascading failures. Conversely, a probe that is too aggressive—periodSeconds: 1 with timeoutSeconds: 1—can mark healthy pods NotReady during brief GC pauses, causing unnecessary request rejection and client‑visible 502 errors. Tuning probe parameters is an iterative process that often requires observing request latency and error rates over several minutes.

Adding or removing backends without downtime

Dynamic scaling is the reason load balancers exist, but the transition period—when a new pod is registered or an old one is drained—can cause request loss if not handled carefully.

Kubernetes implements graceful deletion by default: when a pod is deleted, the kubelet removes it from the Service’s endpoint list only after all existing connections have terminated or after a grace period (default 30 seconds). During that window, the load balancer may still have active TCP sessions or HTTP requests in flight.

To observe the drain behavior:

kubectl drain node worker-1 --ignore-daemonsets --delete-emptydir-data

The node’s pods are rescheduled, and the Service’s endpoint list updates incrementally. If you are using a cloud provider’s external load balancer, the IP address may remain allocated while the backend set changes; some providers support connection draining flags that close idle sessions after a configured timeout.

When adding new instances, the load balancer begins routing new connections to them immediately, but existing connections continue on older backends. There is no automatic “drain all old connections before scaling up” step; the operator must either accept a brief period of dual‑serving or manually cordon nodes before scaling.

Sharp edges – what silently fails, confused options, production lessons

Health check mismatch with application state: A pod may return 200 OK on /healthz while its internal caches are full or its database pool is exhausted. The load balancer continues sending traffic, and users see errors. Remedy: augment liveness probes with application‑specific checks or rely on external monitoring to trigger pod restart.

Session affinity + forward‑ed headers: When a frontend proxy adds X-Forwarded-For, the downstream load balancer hashing on source IP no longer matches the client’s actual address. Result: every request goes to a random pod, and affinity is effectively disabled. Remedy: either disable X-Forwarded-For at the edge, or configure affinity on the header value (sessionAffinity: Header with clientIP replaced by a custom header key) if the data plane supports it.

Probe thrash and node restart loops: If periodSeconds and timeoutSeconds are both set to 1, a brief GC pause or network blip can mark a pod NotReady. The pod controller may then restart it, creating a loop. Remedy: use periodSeconds >= 5 and timeoutSeconds >= 2, and monitor readinessGates if using structured pod termination.

TLS certificate renewal gaps: When a certificate expires, the load balancer may start returning 502 or 526 errors. If the secret is updated but the Ingress controller does not reload, the old certificate remains in use until a restart occurs. Remedy: configure the Ingress controller to watch the secret for changes (most modern controllers do this automatically), or automate secret rotation via a controller that sends a reload signal.

Port protocol mismatch: A common error is configuring a Service with type: LoadBalancer and specifying protocol: TCP on the cloud load balancer while the backend containers listen on HTTP. The connection succeeds at the TCP level but the HTTP request is malformed, resulting in 400 responses. Remedy: ensure the Service’s sessionAffinity and health‑check probe protocols match the application’s exposed protocol. In Kubernetes, this is usually HTTPGet for HTTP backends and TCPSocket for raw TCP services.

DNS TTL and propagation delays: When a Service’s ClusterIP changes (e.g., due to a re‑creation), DNS resolvers may cache the old address for the duration of the TTL. Clients that have already resolved the name continue hitting the old endpoint. Remedy: use short TTLs (30–60 seconds) for Service addresses in dynamic environments, or rely on environment variables/service discovery rather than DNS round‑robin.

These edges are not exhaustive, but they represent the categories of failure that repeatedly surface in conversations with operators who have lived through them. The pattern is always the same: a configuration option that seems straightforward acquires subtle dependencies on the surrounding data‑plane, and the failure mode only becomes visible when traffic volume or failure probability crosses a threshold.

Quick reference checklist

Goal	Typical tool / command	Key flag / key field
Round‑robin distribution	`kubectl get svc` / curl loop	Service `clusterIP`; no extra flag needed
Weighted distribution	Service/Ingress annotations or cloud‑provider labels	`weight` key (vendor‑specific)
TLS termination	`kubectl create secret tls` + Ingress `tls` block	`secretName` in Ingress spec
Session affinity (IP)	`kubectl patch svc … sessionAffinity`	`sessionAffinity: ClientIP`
Session affinity (header)	`kubectl patch svc … sessionAffinityConfig`	`clientIP.timeoutSeconds` or custom header key
Readiness probe	Pod spec `readinessProbe.httpGet`	`path`, `port`, `initialDelaySeconds`, `periodSeconds`
Graceful pod drain	`kubectl drain <node>`	Default 30‑second grace period; no flag to force immediate drain
Cloud‑provider TLS config	Varies (e.g., `aws elbv2 modify-listener`)	Listener ARN, certificate ARN, forward‑action rules

Use this checklist as a starting point; the exact flags and keys will shift depending on the load balancer you operate, but the underlying mechanisms—distribution algorithm, TLS handshake, affinity mapping, health‑check logic, and drain semantics—are shared across virtually all production‑grade implementations.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.