Cloud Load Balancing Strategies

This reference describes specific techniques for distributing traffic across multiple backend services in cloud environments. These patterns apply when designing resilient systems that must handle varying request volumes, maintain availability during failures, and scale efficiently.

Distribute requests based on service capacity

When deploying multiple instances of a service, configure the load balancer to route requests according to current capacity rather than simple round-robin distribution.

Use health checks to monitor instance status and adjust traffic distribution. For example, in AWS Application Load Balancer configuration:

{
  "HealthCheckPath": "/health",
  "HealthyThresholdCount": 2,
  "UnhealthyThresholdCount": 2,
  "IntervalSeconds": 30
}

The load balancer considers an instance healthy only when it returns HTTP 200 for the health check endpoint. When an instance becomes unhealthy, traffic stops routing to it immediately.

Implement weighted request distribution

Distribute traffic according to instance capacity or priority using weights in your load balancer configuration.

For AWS Application Load Balancer, set weights on target groups:

{
  "TargetGroups": [
    {
      "TargetGroupArn": "arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067",
      "Weight": 3
    },
    {
      "TargetGroupArn": "arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a068",
      "Weight": 1
    }
  ]
}

The load balancer routes approximately 75% of traffic to the first target group (weight 3) and 25% to the second (weight 1). This allows for different instance sizes or service priorities without redeploying.

Route based on request characteristics

Direct requests to specific backend instances based on path, header, or query parameter values.

In a typical HTTP load balancer configuration, use rules to match request properties:

Rules:
  - Priority: 100
    Conditions:
      - Field: path-pattern
        Values: ["/api/v1/*"]
    Actions:
      - Type: forward
        TargetGroupArn: arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/api-v1/abcd1234
  - Priority: 200
    Conditions:
      - Field: header
        HeaderName: X-Environment
        Values: [staging]
    Actions:
      - Type: forward
        TargetGroupArn: arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/staging/efgh5678

These rules ensure API v1 requests go to a dedicated target group and staging environment requests go to a different backend.

Handle session affinity with sticky sessions

Maintain client sessions across multiple requests by routing subsequent requests from the same client to the same backend instance.

Configure session stickiness using cookies:

SessionCookieName: "JSESSIONID"
SessionStickinessEnabled: true

When enabled, the load balancer adds a cookie to the first response. Subsequent requests with that cookie are routed to the same backend instance.

Implement circuit breaker patterns

Prevent cascading failures by detecting and temporarily halting traffic to failing services.

Set up circuit breaker behavior in your load balancer configuration:

CircuitBreaker:
  Enabled: true
  MaxConnections: 1000
  Threshold: 50
  TimeoutSeconds: 30

When more than 50% of requests fail within a 30-second window, the circuit breaker opens and stops routing traffic to that backend for 30 seconds.

Configure health check thresholds

Establish appropriate thresholds for determining instance health to prevent routing to failing instances.

Set health check parameters:

HealthCheck:
  Path: "/health"
  Port: 8080
  Protocol: HTTP
  IntervalSeconds: 30
  TimeoutSeconds: 5
  HealthyThresholdCount: 2
  UnhealthyThresholdCount: 3

The load balancer waits 30 seconds between checks, considers a check successful if it completes within 5 seconds, and requires two consecutive successes to mark an instance healthy.

Use connection pooling with timeouts

Configure connection timeouts to prevent long-running requests from blocking backend resources.

Set connection timeout parameters:

ConnectionSettings:
  IdleTimeoutSeconds: 60
  MaxConnectionsPerInstance: 100

These settings ensure connections don't remain open indefinitely and that each backend instance handles a reasonable number of concurrent connections.

Implement graceful shutdown handling

Ensure traffic stops flowing to instances before they shut down to prevent dropped requests.

Set up lifecycle hooks:

Lifecycle:
  GracefulShutdownTimeoutSeconds: 30
  DrainConnections: true

When an instance is removed from the load balancer, it continues processing active requests for 30 seconds before terminating.

Monitor and respond to traffic spikes

Configure automatic scaling based on load balancer metrics:

ScalingPolicy:
  TargetValue: 70
  MetricType: RequestCountPerTarget

This policy scales the target group when average requests per instance exceed 70, ensuring sufficient capacity during traffic bursts.

Handle cross-zone load balancing

Control how traffic is distributed across availability zones.

Configure cross-zone settings:

CrossZoneLoadBalancing:
  Enabled: true

When enabled, traffic is distributed evenly across all available instances in all zones, rather than only within the same zone as the client.

Use path-based routing for microservices

Route requests to specific services based on URL paths.

Set up routing rules:

PathRouting:
  - Path: "/users/*"
    Service: "user-service"
  - Path: "/orders/*"
    Service: "order-service"

This configuration ensures that all requests to /users/ are routed to the user service and all requests to /orders/ go to the order service.

Configure timeout behavior

Set appropriate timeouts to prevent long-running requests from consuming resources.

TimeoutSettings:
  IdleTimeoutSeconds: 60
  RequestTimeoutSeconds: 30

Requests that remain idle for more than 60 seconds are closed, and requests that take longer than 30 seconds to complete are terminated.

Set up failover routing

Define backup targets when primary instances are unavailable.

FailoverSettings:
  PrimaryTargetGroupArn: arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/primary/12345678
  BackupTargetGroupArn: arn:aws:elasticloadbalancing:us-west-2:123456789012:targetgroup/backup/87654321

When primary instances are unhealthy, traffic routes to backup targets automatically.

Monitor traffic distribution

Track how requests are distributed across backend instances:

Monitoring:
  MetricsEnabled: true
  CustomMetrics: ["RequestCount", "Latency"]

These metrics help identify imbalances in traffic distribution and performance issues.

Handle connection draining

Prevent new connections to instances being terminated during updates.

ConnectionDraining:
  Enabled: true
  TimeoutSeconds: 30

When enabled, the load balancer waits 30 seconds before removing an instance from service, allowing active connections to complete.

Configure rate limiting

Control traffic volume to prevent backend overload.

RateLimiting:
  MaxRequestsPerSecond: 1000
  BurstLimit: 2000

This configuration limits requests to 1000 per second with a burst capacity of 2000 requests.

Implement retry logic

Configure automatic retries for transient failures.

RetrySettings:
  MaxRetries: 3
  BackoffSeconds: 1
  RetryOnStatusCodes: [500, 503]

When the backend returns a 500 or 503 error, the load balancer retries the request up to three times with increasing delays.

Handle health check failures

Define behavior when health checks fail:

FailureBehavior:
  UnhealthyThresholdCount: 3
  HealthyThresholdCount: 2

An instance must fail three consecutive health checks before being marked unhealthy, preventing premature removal during transient issues.

Set up multiple target groups

Route traffic to different services based on different criteria.

TargetGroups:
  - Name: "api-v1"
    Protocol: HTTP
    Port: 8080
  - Name: "api-v2" 
    Protocol: HTTP
    Port: 8081

This setup allows routing to different service versions or implementations.

Configure SSL termination

Handle secure connections at the load balancer level.

SSLSettings:
  CertificateArn: arn:aws:acm:us-west-2:123456789012:certificate/12345678-1234-1234-1234-123456789012
  SSLPolicy: ELBSecurityPolicy-2016-08

The load balancer terminates SSL connections and forwards traffic to backend instances over HTTP.

Manage certificate rotation

Update certificates without disrupting service availability.

CertificateManagement:
  AutoRenew: true
  RenewalThresholdDays: 30

Certificates automatically renew 30 days before expiration, preventing service interruption.

Handle regional failover

Route traffic to alternate regions when primary region fails.

RegionalFailover:
  Enabled: true
  BackupRegion: "us-east-1"
  RoutingPolicy: "weighted"

When primary region experiences issues, traffic routes to backup region with weighted distribution.

Configure custom headers

Add information to requests forwarded to backend services.

CustomHeaders:
  - Name: "X-Forwarded-For"
    Value: "$proxy_add_x_forwarded_for"
  - Name: "X-Forwarded-Proto"
    Value: "$scheme"

These headers help backend services understand the original request context.

Set up custom error handling

Define how to respond to failed requests.

ErrorHandling:
  CustomErrorPages:
    503:
      StatusCode: 503
      Content: "Service Unavailable"
  RetryBehavior:
    MaxRetries: 2

When a service returns 503, the load balancer shows custom error content and retries requests twice.

Monitor backend response times

Track performance of backend services.

ResponseMonitoring:
  LatencyThresholdSeconds: 10
  AlertOnSlowResponse: true

If backend responses exceed 10 seconds, monitoring systems can trigger alerts.

Handle request body routing

Route based on request content or headers.

ContentRouting:
  - Condition: "Content-Type"
    Value: "application/json"
    TargetGroup: "json-service"
  - Condition: "Accept"
    Value: "text/xml"
    TargetGroup: "xml-service"

Requests with JSON content go to one service, XML requests to another.

Implement traffic mirroring

Send copies of traffic to additional services for analysis.

TrafficMirroring:
  Enabled: true
  MirrorTarget: "analysis-service"
  SampleRate: 0.1

Ten percent of all requests are sent to an analysis service without affecting the primary response.

Configure automatic instance registration

Add new instances to load balancing automatically.

AutoRegistration:
  Enabled: true
  InstanceHealthCheckPath: "/health"
  RegistrationTimeoutSeconds: 60

New instances automatically register with the load balancer after passing health checks.

Handle traffic shaping

Control how traffic is distributed over time.

TrafficShaping:
  GradualRollout: true
  RolloutPercentage: 10
  RolloutIntervalSeconds: 300

When deploying updates, only 10% of traffic routes to new instances every 5 minutes.

Set up multiple load balancer types

Use different load balancer types for different requirements.

LoadBalancerTypes:
  - Type: "Application"
    Protocol: "HTTP"
    Port: 80
  - Type: "Network"
    Protocol: "TCP"
    Port: 443

Application load balancer handles HTTP/HTTPS traffic, while network load balancer handles TCP traffic.

Configure backend connection pooling

Manage connections to backend services.

BackendConnectionPool:
  MaxConnectionsPerInstance: 100
  ConnectionTimeoutSeconds: 5
  IdleTimeoutSeconds: 30

Each backend instance maintains up to 100 concurrent connections with 5-second timeouts and 30-second idle timeouts.

Monitor capacity utilization

Track how much of your backend capacity is being used.

CapacityMonitoring:
  Enabled: true
  MetricsToTrack: ["CPUUtilization", "NetworkIn", "NetworkOut"]

These metrics help determine when to scale services or adjust load distribution.

Handle instance lifecycle events

Manage traffic during instance startup and shutdown.

LifecycleEvents:
  InstanceStartupDelaySeconds: 60
  InstanceShutdownGracePeriodSeconds: 30

New instances wait 60 seconds after startup before receiving traffic, and shutdowns include a 30-second grace period.

Configure request buffering

Control how requests are buffered before processing.

Buffering:
  RequestBufferSizeBytes: 1048576
  ResponseBufferSizeBytes: 1048576

Requests larger than 1MB are buffered to prevent memory issues during processing.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.