Choose and configure load balancing algorithms that match your workload characteristics. Covers L4 vs L7 balancing, health checks, connection draining, session affinity, and the failure modes that take down your entire service.
Load balancing beyond round robin is essential when traffic patterns, latency, and system health demand more than simple distribution across servers. When a single round robin policy begins to underperform—due to uneven server load, variable request duration, or regional user distribution—engineers must adopt strategies that adapt to real-time conditions. These strategies are not optional add-ons but core infrastructure components that directly influence service availability, response time, and resource utilization.
When backend servers vary in processing power or current load, simple round robin fails to balance work effectively. A server handling 100 requests per second may be overwhelmed while another idles with 10% CPU. The solution is to incorporate health checks and dynamic weight assignment.
Health checks are periodic probes sent to each backend server. The load balancer evaluates the response time, status code, and payload. A server returning a 200 OK within 50ms is considered healthy; one returning 503 or taking over 300ms is degraded. Based on these evaluations, the load balancer adjusts the server’s weight in the load distribution algorithm.
In practice, this is implemented using HTTP-based health checks. The load balancer sends a GET request to a /health endpoint on each server every 10 seconds:
GET /health HTTP/1.1
Host: backend.example.com
Accept: application/jsonA successful response returns a 200 status with a JSON body:
{
"status": "healthy",
"latency_ms": 42,
"memory_usage_percent": 67,
"queue_length": 12,
"uptime_seconds": 3672
}If the server responds with 503 or a latency above 300ms, it is marked as unhealthy and receives a reduced weight. Unhealthy servers are gradually removed from the load pool.
Weight adjustments are not static. A server with high memory usage and long queue length is assigned a lower weight. A server with low latency and low queue length receives a higher weight. The load balancer uses these weights in its selection algorithm:
The selection algorithm can be configured as a policy in the load balancer’s configuration:
load_balancer:
strategy: "least_time"
health_check:
interval_seconds: 10
timeout_seconds: 5
path: "/health"
healthy_threshold: 3
unhealthy_threshold: 2
initial_delay_seconds: 5
server_weights:
backend-01: 3
backend-02: 1
backend-03: 2
backend-04: 4A critical failure mode: if a health check fails but the server is still alive, the load balancer may not detect a slow server until the threshold is exceeded. A server that consistently takes 350ms becomes unhealthy after three consecutive failures, but during the second failure, the server is already overwhelmed. The system silently degrades until the third failure triggers a weight reduction.
Another edge: the health check path must be idempotent and lightweight. If /health returns a large JSON object with database stats, the check itself becomes a performance bottleneck. The load balancer may report a healthy server while the server is actually overloaded from the health check.
When users are distributed across geographies, the cost of network latency becomes a dominant factor. A user in Tokyo accessing a server in Frankfurt incurs a 150ms round trip. The same user accessing a server in Tokyo incurs 20ms. Even if all servers are equally loaded, the user experience differs dramatically.
To minimize latency, the load balancer must route requests based on the client’s IP address or DNS query origin.
The strategy uses DNS-based routing. When a client queries for api.example.com, the DNS resolver returns an IP address chosen by the load balancer. The decision is based on the client’s IP, resolved via GeoIP databases.
The load balancer maintains a list of backend servers with their geographic location, current load, and latency to a set of known clients. It uses a weighted algorithm that combines latency and server load.
A configuration might look like:
dns_routing:
strategy: "latency_and_load"
geoip_database: "/etc/geoip.mmdb"
location_map:
"frankfurt": { lat: 50.1109, lon: 8.6821 }
"tokyo": { lat: 35.6762, lon: 139.6503 }
"sfo": { lat: 37.7749, lon: -122.4194 }
server_locations:
backend-01: "frankfurt"
backend-02: "frankfurt"
backend-03: "tokyo"
backend-04: "sfo"
latency_thresholds:
"frankfurt": 100
"tokyo": 15
"sfo": 80When a request arrives from a client with IP 133.123.45.123, the load balancer:
latency_cost + load_penalty.The DNS response includes a TTL and a list of candidate IPs, allowing resolvers to cache and fall back:
api.example.com. 300 IN A 192.0.2.10
api.example.com. 300 IN A 203.0.113.20
api.example.com. 300 IN A 10.0.1.5The order of IPs matters: the first IP is the preferred server.
A common failure mode: GeoIP databases are updated infrequently. A client in Seoul may be routed to Tokyo, but the GeoIP database assigns Seoul to Tokyo, even though the city is 300km away. If the Tokyo server is overloaded, the load balancer may fail to route clients from Seoul to a nearby server in Osaka, even though the data is available.
Another edge: the load balancer must update its latency measurements continuously. A server in Tokyo may be fast for clients in Tokyo but slow for clients in Osaka due to a new network segment. The load balancer must maintain latency metrics per region and update them via periodic pings.
When a user performs a sequence of operations—such as adding items to a cart and checking out—the state of their session must be maintained across requests. Without session persistence, a user’s session data could be lost if the load balancer routes the next request to a different server.
Session persistence is implemented via sticky sessions, where the load balancer assigns a session identifier (cookie) to each client and routes all requests from that client to the same backend server.
The load balancer sets a cookie on the first response:
HTTP/1.1 200 OK
Content-Type: application/json
Set-Cookie: JSESSIONID=abc123xyz; Path=/; HttpOnly; SecureOn subsequent requests, the client sends the cookie:
GET /checkout HTTP/1.1
Host: api.example.com
Cookie: JSESSIONID=abc123xyzThe load balancer uses the JSESSIONID to look up the server assigned to that session.
The session persistence strategy can be configured with a timeout and a fallback:
session_persistence:
cookie_name: "JSESSIONID"
timeout_seconds: 1800
fallback_strategy: "least_connections"
sticky_policy: "consistent_hashing"The consistent_hashing policy minimizes session migration during server changes. It maps client identifiers (IP address, session ID, or user ID) to a hash ring. Each server owns a range of hash values. When a server is added or removed, only a subset of sessions need to be moved.
A key failure mode: when a server fails, the load balancer must reassign sessions. With simple session persistence, the load balancer sends a 302 Found response to redirect the client to the new server:
HTTP/1.1 302 Found
Location: https://api.example.com/new-server
Set-Cookie: JSESSIONID=abc123xyz; Path=/; HttpOnly; SecureBut the client may not update the cookie immediately. If the client sends a request to the old server while the redirect is in flight, the session may be split across two servers.
Another edge: the load balancer must handle session expiration. A session with a 30-minute timeout may expire while the user is idle. The load balancer must detect the session cookie and verify the session data on the backend server.
The session data may be stored in a shared cache, such as Redis, or in a distributed database. The load balancer does not manage the session data but relies on it being available across all servers.
When traffic patterns change—such as a spike during a product launch or a daily peak—load balancing must adapt not only to server health and location but also to traffic volume.
The load balancer monitors incoming request rate and adjusts its strategy dynamically. It uses a combination of thresholds and feedback loops.
A configuration might define a traffic shaping policy:
traffic_shaping:
strategy: "adaptive"
threshold:
requests_per_second: 1000
latency_threshold_ms: 200
error_rate_threshold: 0.05
policies:
"low_traffic":
strategy: "round_robin"
server_weights:
backend-01: 2
backend-02: 2
backend-03: 1
"high_traffic":
strategy: "least_time"
server_weights:
backend-01: 4
backend-02: 3
backend-03: 2
backend-04: 3
health_check:
interval_seconds: 5
timeout_seconds: 3
"burst_traffic":
strategy: "weighted_least_connections"
server_weights:
backend-01: 5
backend-02: 4
backend-03: 3
backend-04: 4
session_persistence:
timeout_seconds: 900
cookie_name: "SESSIONID"The load balancer continuously monitors metrics. When the request rate exceeds 1000 requests per second, it switches to the high_traffic strategy. When the latency exceeds 200ms or the error rate (500/503/504 responses) exceeds 5%, it triggers a shift to burst_traffic.
The strategy switch is not immediate. The load balancer uses a state machine with a cooldown period to prevent oscillation between strategies.
A common failure mode: the load balancer switches strategies too frequently. A spike from 990 to 1010 requests per second triggers a switch, but the next second drops to 980, triggering a back-switch. The system becomes unstable, and the load balancer constantly reconfigures.
Another edge: the load balancer must handle strategy transitions gracefully. During a switch, the load balancer maintains a queue of incoming requests and applies the new strategy to new requests while continuing to serve existing ones with the old strategy.
The load balancer can also be configured to trigger actions during a strategy change:
strategy_change_actions:
"high_traffic":
- "enable_logging_level=debug"
- "send_alert_to_slack"
- "scale_up_backends"
"burst_traffic":
- "enable_caching_layer"
- "disable_gzip_compression"These actions are executed via a simple command-line interface or HTTP API.
When implementing load balancing beyond round robin, engineers must remember that the load balancer is not a black box. It is a distributed system that must be monitored, tuned, and tested.
The most common silent failure: the load balancer does not propagate configuration changes. A new server is added, but the load balancer does not update its list of servers until the next health check. During that time, clients may receive 503s.
Another silent failure: the load balancer does not handle stale sessions. A server is removed from the pool but continues to serve requests for several minutes. The load balancer may route new requests to other servers while the old server is still active, leading to inconsistent state.
Finally, the load balancer must be tested under failure conditions: network partitions, server crashes, and slow health checks. The system must handle these with graceful degradation, ensuring that service remains available even when the load balancer itself is under stress.
This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy