Network Resilience Patterns

Network resilience is the ability of a network to continue delivering services despite component failures, transient outages, and evolving load. It matters when downtime is costly—when users expect uninterrupted access, when services are distributed across regions, and when failure propagation can cascade across systems. This page presents patterns, practices, and concrete configurations for building networks that survive and recover from failures, from single link drops to complete site outages.

Core Failure Modes and Their Resilience Patterns

Single-Link Failure: Fast Failover with ECMP and BFD

When a single network link fails—say, a fiber cut between a data center and a cloud provider—resilience demands fast detection and convergence. The most effective pattern uses Equal-Cost Multi-Path (ECMP) routing with Bidirectional Forwarding Detection (BFD).

ECMP enables multiple paths to the same destination, distributing traffic across them. BFD provides sub-second detection of link failures. Together, they reduce convergence time from minutes to under 100 milliseconds.

On Linux, BFD is managed via ip commands. To configure BFD on a link:

ip link add name bond1 type bond mode 802.3ad
ip link set dev eth0 master bond1
ip link set dev eth1 master bond1
ip link set dev bond1 up

ip -6 address add 2001:db8:1::1/64 dev bond1
ip -6 address add 2001:db8:2::1/64 dev bond1

ip route add 2001:db8:3::0/64 via 2001:db8:1::2 dev bond1 metric 10
ip route add 2001:db8:3::0/64 via 2001:db8:2::2 dev bond1 metric 10


ip bfd add 2001:db8:1::2 interval 100 min_rx 100 multiplier 3 dev bond1
ip bfd add 2001:db8:2::2 interval 100 min_rx 100 multiplier 3 dev bond1

# Verify BFD status
ip bfd show

BFD parameters are critical: interval sets the frequency of BFD control packets; min_rx is the minimum receive interval; multiplier determines the time after which a link is declared down (e.g., 3 × 100ms = 300ms). A common pitfall is setting interval too high (e.g., 1000ms) while min_rx remains at 100ms, causing BFD to fail to detect a link drop during a 200ms outage.

BFD sessions are independent of routing protocols. When a BFD session fails, the router immediately removes the associated route, triggering ECMP to shift traffic to the backup path. This pattern ensures that a 10ms network spike becomes a 100ms outage, not a 10-second failure.

Multi-Region Outage: Active-Active with Health Checks and Load Balancer Drift

When a full region—such as an AWS us-east-1 or a Google Cloud europe-west1—fails, resilience relies on active-active service deployment across regions, with dynamic load balancing and failover.

The core pattern involves a global load balancer (e.g., AWS ALB, Cloudflare Load Balancer, or HAProxy in front of multiple regional services) that performs health checks and routes traffic based on region availability.

Health checks are HTTP probes that measure endpoint responsiveness. They are typically configured with:

• Interval: 10 seconds
• Timeout: 3 seconds
• Unhealthy threshold: 3 failures
• Healthy threshold: 2 successes

A service is deemed down after three consecutive failures. The load balancer then begins routing traffic away from the failed region.

In HAProxy, this is configured with:

backend regional_services
    balance leastconn
    option httpchk GET /health HTTP/1.1\r\nHost:\ www.example.com
    http-check expect status 200
    server us-east-1-1 10.0.1.10:8080 check fall 3 rise 2
    server us-east-1-2 10.0.1.11:8080 check fall 3 rise 2
    server europe-west1-1 10.0.2.10:8080 check fall 3 rise 2 backup
    server europe-west1-2 10.0.2.11:8080 check fall 3 rise 2 backup

The fall 3 directive means a server must fail three checks in a row before being marked down. rise 2 means a server must pass two checks after being down before being promoted.

When the us-east-1 region fails, the load balancer gradually shifts traffic to europe-west1. But the pattern fails silently when health checks are misconfigured: a timeout of 5 seconds with a 10-second interval leads to a 15-second delay before a service is marked down—long enough for a 10-second spike in latency to go unnoticed.

A subtle edge is drift: a region’s load balancer may be configured to prefer its local services, but as traffic shifts, the regional load balancer in europe-west1 begins to receive more requests than it can handle. This is solved by enabling dynamic weight adjustment, where each region’s load balancer reports its current load (e.g., via curl -X POST /metrics -d '{"load": 1.2}'), and the global load balancer adjusts weights accordingly.

Latency-Induced Failures: Circuit Breakers and Graceful Degradation

High latency or jitter can cause services to time out, leading to cascading failures. The pattern here is circuit breaking: a client or gateway opens a circuit when latency exceeds a threshold, and during the open state, it skips downstream calls and serves degraded responses.

The circuit breaker is implemented via a state machine: closed, open, half-open.

• Closed: requests pass normally; a failure counter tracks errors.
• Open: after a threshold of failures (e.g., 5 errors in 10 seconds), the circuit opens. Subsequent requests fail immediately.
• Half-open: after a timeout (e.g., 30 seconds), the circuit attempts to validate the backend. If a request succeeds, the circuit closes.

This is typically implemented in an API gateway or service mesh. In Envoy, circuit breakers are configured per cluster:

static_resources:
  clusters:
    - name: user-service
      connect_timeout: 1s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      http2_protocol_options: {}
      load_assignment:
        cluster_name: user-service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: users.example.com
                      port_value: 8080
      circuit_breakers:
        thresholds:
          - priority: HIGH
            max_connections: 1000
            max_pending_requests: 100
            max_requests: 10000
            max_retries: 3
            min_rest_time: 10s
            retry_budget:
              retry_ratio: 0.5
              max_retries_per_request: 2
          - priority: DEFAULT
            max_connections: 500
            max_pending_requests: 50
            max_requests: 5000
            max_retries: 2
            min_rest_time: 5s
            retry_budget:
              retry_ratio: 0.7
              max_retries_per_request: 3

The min_rest_time setting is critical: it prevents the circuit from reopening too quickly after a burst of traffic. A common mistake is setting min_rest_time to 1 second but expecting it to apply only after the first successful request, not after the entire half-open period.

Graceful degradation complements circuit breaking. When a circuit is open, the client returns cached data, fallback responses, or a simplified UI. For example, a frontend service may serve a static "Offline Mode" page when the user service is unreachable.

Transient Outage: Retry with Exponential Backoff

When an outage is short-lived—lasting seconds to minutes—the pattern of retry with exponential backoff prevents overwhelming the backend and improves user experience.

A client makes a request, and if it receives a timeout or a 5xx error, it retries the request after a delay, doubling the delay on each failure.

The protocol is typically HTTP with Retry-After header and 503 Service Unavailable status.

A client implementation in Go might look like:

func callWithRetry(ctx context.Context, client *http.Client, url string, maxRetries int) (*http.Response, error) {
    var resp *http.Response
    var err error
    var backoff time.Duration = 1 * time.Second

    for attempt := 0; attempt <= maxRetries; attempt++ {
        req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
        req.Header.Set("Accept", "application/json")

        resp, err = client.Do(req)
        if err == nil && resp.StatusCode < 500 {
            return resp, nil
        }

        if err != nil {
            log.Printf("Attempt %d failed: %v", attempt+1, err)
        } else {
            log.Printf("Attempt %d returned status %d", attempt+1, resp.StatusCode)
        }

        if attempt == maxRetries {
            break
        }

        // Wait before retry
        select {
        case <-time.After(backoff):
        case <-ctx.Done():
            return resp, ctx.Err()
        }

        backoff *= 2
    }

    return resp, err
}

The Retry-After header is key. When the server returns 503 Service Unavailable, it may include:

HTTP/1.1 503 Service Unavailable
Content-Type: application/json
Retry-After: 30

{
  "message": "Service temporarily unavailable",
  "details": "Backend database is recovering"
}

This tells the client to wait at least 30 seconds before retrying. The client should respect Retry-After as the base delay, not just use a fixed backoff.

A silent failure occurs when the client uses Retry-After for the first retry but ignores it for subsequent retries. The pattern should be:

• First retry: use Retry-After as initial delay.
• Subsequent retries: use exponential backoff from the first delay, but cap it at Retry-After.

This prevents the client from retrying too frequently during a long outage.

Failure Cascading: Chaining Resilience Patterns

When one component fails, its failure propagates through the system. A pattern to combat this is cascading resilience: combining multiple patterns in sequence.

Consider a user-facing API that depends on a user service, a payment service, and a cache. Each service is resilient on its own, but when the user service fails, the API must:

1. Use circuit breaking to prevent overloading the user service.
2. Retry with exponential backoff to the user service.
3. Serve a degraded response using cached user data.
4. Fail over to a backup user service in another region.
5. Perform health checks on the backup service and gradually shift traffic.

The full sequence is orchestrated by a service mesh or API gateway. The API gateway configures:

• Circuit breaker for user service with 3 failure threshold and 30s open state.
• Retry policy: 3 retries, exponential backoff from 1s to 8s.
• Fallback: serve data from Redis when user service is unavailable.
• Health check: every 10s, check /health on user service.

When the user service becomes unreachable, the API gateway:

1. Opens the circuit after three consecutive failures.
2. Begins retrying with increasing delays.
3. After the third failure, it serves a cached user profile.
4. After 30 seconds, it begins routing traffic to a backup user service in a secondary region.
5. The backup service is promoted to primary after a successful health check.

This pattern ensures that a 30-second outage in the primary user service causes only a 30-second delay in service availability, with minimal impact on overall user experience.

Common Pitfalls and Silent Failures

• Health check intervals mismatched with failure thresholds: A 30-second interval with a 3-failure threshold means a service must be down for 60 seconds before being marked down—causing a 1-minute delay in failover.
• BFD session not synchronized with routing: A BFD session fails, but the routing protocol (e.g., OSPF) does not recompute routes until the next hello interval, leading to a 1-second delay.
• Circuit breaker not being reset on success: After a circuit opens, a single successful request does not close it; the number of successful requests must meet a threshold.
• Retry-with-backoff not respecting Retry-After: The client waits only the first Retry-After, but ignores it for subsequent retries, leading to suboptimal behavior.
• Graceful degradation not being tested under real outages: A fallback page is served, but the client does not load JavaScript or CSS, resulting in a non-interactive experience.

These patterns are not applied in isolation. The true resilience of a network emerges from their interplay, where a single failure triggers multiple responses, each tuned to different failure modes and response times.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.