Circuit Breaker Advanced | The Garnet Wiki

A circuit breaker is a pattern for managing failure in distributed systems, but advanced circuit breaker usage is not about setting a threshold and waiting for a failure. It’s about anticipating failure, reacting to cascading outages, and synchronizing resilience across services with precision. This page details the mechanics, configurations, and failure modes that distinguish a robust circuit breaker from a basic one. It applies when latency spikes, transient faults become persistent, and service degradation must be detected and acted upon before users notice.

Adaptive Failure Detection

Circuit breakers detect failure by counting errors, but the advanced case demands dynamic, context-aware thresholds. The core idea is that a failure threshold must adapt to the service’s typical behavior—what is “normal” today is “failure” tomorrow.

The circuit breaker tracks two metrics: error rate and latency. A request is considered failed if it exceeds a latency threshold (e.g., 500ms) or returns a non-2xx status code. The breaker evaluates these over a sliding window, typically using a time-based or count-based window.

circuit-breaker:
  failure:
    error-rate-threshold: 0.5
    min-request-count: 10
    latency-threshold: 500
    time-window: 60s
    sliding-window-type: "time"

This configuration means: if at least 10 requests in the last 60 seconds have either failed or exceeded 500ms latency, and the error rate (failed requests / total requests) exceeds 50%, the circuit breaker trips.

The sliding window is not a simple counter. Each request contributes to the current window, and old requests expire as time passes. The window is implemented as a ring buffer, where each slot stores a request’s result (success/failure, latency). When a new request arrives, the oldest slot is replaced, maintaining a fixed number of recent requests.

A common failure mode is misinterpreting the window type. A sliding-window-type: "count" uses the last N requests, but the window is not aligned with time. If the load fluctuates, a burst of requests in 10 seconds may not be captured in the next 30-second window. This leads to false positives: a circuit breaker trips during a burst, but the burst is not sustained.

The time-window must be aligned with the service’s typical request rate. If the service usually gets 100 requests per minute, a 60-second window holds ~100 requests. If the load drops to 10 per minute, the window contains only 10–15 requests, making the error rate estimate noisy. To fix this, use a dynamic window size:

circuit-breaker:
  failure:
    error-rate-threshold: 0.5
    min-request-count: 5
    latency-threshold: 500
    time-window: 60s
    dynamic-window-size: true
    window-size-multiplier: 1.5

With dynamic-window-size: true, the window expands or contracts based on the average request rate. If the average rate is 5 requests per minute, the window size becomes 60s × 1.5 = 90s. This prevents the circuit from tripping due to transient noise in low-load periods.

State Transitions and Timed Recovery

A circuit breaker has three states: closed, open, and half-open. The transition between them is not just based on thresholds but on timing.

When the circuit is closed, it allows requests through. If the failure threshold is exceeded, it transitions to open, blocking all requests. After a configured wait-time (e.g., 30 seconds), it becomes half-open, allowing a limited number of requests to test the health of the downstream service.

The wait-time is not a static value. It can be derived from the duration of the outage:

circuit-breaker:
  recovery:
    wait-time: 30s
    base-wait-time: 10s
    exponential-backoff: true
    wait-time-multiplier: 1.5

With exponential-backoff: true, the wait time doubles for each successive failure. After the first tripped circuit, the breaker waits 10s, then 15s, then 22.5s, etc.

The half-open state is where subtle failures arise. The breaker allows a max-concurrent-requests (e.g., 10) to pass through, and the first failure among them closes the circuit again. But the success threshold is critical:

circuit-breaker:
  recovery:
    success-threshold: 0.7
    min-success-count: 3
    max-concurrent-requests: 10
    wait-time: 30s

Here, the circuit becomes closed if at least 70% of the requests in the half-open state succeed, or at least 3 successes (whichever comes first). If only 2 of 10 requests succeed, the circuit remains open.

A sharp edge: success threshold and min-success-count are not redundant. If min-success-count is 3, the circuit opens only after three successes, but if success-threshold is 0.7, the circuit opens after 7 of 10 successes. The two can conflict. For example, if a service returns 4 successes in 6 requests, the success rate is 66.7%, which is below 70%. So even though 4 > 3, the circuit does not open.

Another edge: concurrent requests in half-open state are not processed in isolation. A request arriving during the half-open state may be served by a service that is still recovering. If the service uses a thread pool, and the pool is exhausted, a request may take 300ms to respond, but the breaker treats it as a success only if it returns before the pool stabilizes.

Circuit Breaker Coordination Across Services

In a microservices architecture, a circuit breaker is not isolated. It must coordinate with other services, especially when cascading failures occur.

A gateway service (e.g., an API gateway) may maintain a circuit breaker for each backend service. When the circuit for user-service trips, the gateway should:

Return a 503 Service Unavailable with Retry-After: 30 header.
Propagate the circuit breaker state to downstream services via metadata in HTTP headers.

HTTP/1.1 503 Service Unavailable
Retry-After: 30
X-Circuit-Breaker: open
X-Circuit-Breaker-Reason: "Latency > 500ms for 60s"
X-Circuit-Breaker-Open-At: "2024-05-15T12:34:56Z"

The downstream service can use this information to adjust its behavior. For example, a client may cache the circuit breaker state for a minute, avoiding redundant calls.

When a service is open, it should return a circuit breaker response to every request, even if the request succeeds:

{
  "status": "success",
  "data": { "id": 123, "name": "Alice" },
  "circuitBreaker": {
    "state": "open",
    "reason": "High latency and error rate",
    "openAt": "2024-05-15T12:34:56Z",
    "expectedRecovery": "2024-05-15T12:35:26Z"
  }
}

This enables clients to detect failure patterns and react—e.g., by switching to a fallback data source, or by queuing requests.

The circuit breaker state can be shared via a distributed cache (e.g., Redis) or event-driven coordination.

A service may publish a CircuitBreakerStateUpdated event to a message queue (e.g., Kafka, RabbitMQ) whenever the state changes:

{
  "eventId": "cb-7a1b2c3d",
  "service": "user-service",
  "state": "open",
  "reason": "Error rate > 50% for 60s",
  "timestamp": "2024-05-15T12:34:56Z",
  "metadata": {
    "errorRate": 0.63,
    "latencyMean": 612,
    "requestCount": 42
  }
}

Other services can subscribe to these events and react: a dashboard may display the current state, a monitoring system may trigger alerts, and a load balancer may route traffic based on circuit breaker state.

Advanced Configuration: Error Thresholds and Backpressure

The error threshold is not a single number. It can be adaptive, multi-stage, or context-aware.

A common pattern is to define multiple error thresholds based on request load:

circuit-breaker:
  failure:
    error-rate-thresholds:
      - load: "low", threshold: 0.4
      - load: "medium", threshold: 0.5
      - load: "high", threshold: 0.6
    load-detection-method: "request-rate"
    request-rate-thresholds:
      - load: "low", min: 5, max: 20
      - load: "medium", min: 21, max: 100
      - load: "high", min: 101

Here, the error threshold adapts to the request rate. If the request rate is 15 per minute, the service is in the "medium" load zone, so the threshold is 0.5. If the rate drops to 3, it switches to "low" load and uses a threshold of 0.4.

This configuration is not just a static table. The load detection is based on rolling average request rate over a 5-minute window. The service computes the average per-minute request rate, then maps it to a load category.

Another advanced pattern: backpressure during open state.

When the circuit is open, the service can reject new requests with a 429 Too Many Requests response:

HTTP/1.1 429 Too Many Requests
Retry-After: 10
X-Circuit-Breaker-State: open
X-Circuit-Breaker-Reason: "Service not responding"
Content-Type: application/json

{
  "error": "Circuit breaker open",
  "expectedRecoveryTime": "2024-05-15T12:36:00Z",
  "retries": 3
}

The Retry-After header is critical. It tells clients when to retry the request. Clients can implement exponential backoff based on this header.

Additionally, the service can queue requests during open state. When a request arrives, it is enqueued. A background worker processes the queue when the circuit becomes half-open.

The queue can be implemented using a message broker:

A circuit-breaker-queue receives requests when the circuit is open.
When the circuit transitions to half-open, the worker consumes up to max-concurrent-requests from the queue.
The worker processes each request, records the result, and updates the circuit breaker state.

This pattern ensures that even during prolonged outages, requests are not lost and the service can recover smoothly.

Failure Modes and Observability

Circuit breakers fail silently. The most common silent failure is stale state. A service may update the circuit breaker state, but the state is not observed by the caller.

For example, a client may cache the circuit breaker state for 1 minute, but the service updates it every 30 seconds. A client may use a stale state for 30–60 seconds, during which the circuit is open but the client thinks it is closed.

To detect this, include health check endpoints:

GET /health/circuit-breaker

Response:

{
  "service": "user-service",
  "state": "open",
  "openAt": "2024-05-15T12:34:56Z",
  "lastUpdated": "2024-05-15T12:35:23Z",
  "errorRate": 0.63,
  "latencyMean": 612,
  "requestCount": 42,
  "recoveryPredicted": "2024-05-15T12:36:00Z"
}

Clients can query this endpoint periodically to validate the circuit state and detect drift.

A sharp edge: latency measurement. A request may be considered failed if it takes more than 500ms, but the measurement method matters.

If the latency is measured from the moment the request is received to the moment the response is sent, the measurement includes processing time, network time, and I/O time. But if the measurement includes only the time from request start to first byte, it may miss delays in the upstream service.

A circuit breaker should track end-to-end latency from the client, and service latency from the server.

Finally, the circuit breaker should log detailed state transitions:

2024-05-15T12:34:56Z [user-service] CircuitBreaker: Closed → Open
  Reason: Error rate 0.63 in 60s, 42 requests, 27 failures
  Duration: 12s
2024-05-15T12:35:26Z [user-service] CircuitBreaker: Open → Half-Open
  Wait time: 30s
2024-05-15T12:35:56Z [user-service] CircuitBreaker: Half-Open → Closed
  Success rate: 0.76, 12 of 16 requests succeeded

These logs enable debugging cascading failures and tuning thresholds.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.