Implement the fundamental reliability patterns that keep distributed systems running under failure. Covers circuit breakers, bulkheads, timeouts, retries with backoff, graceful degradation, and fallback strategies.
This page describes concrete, reusable approaches for building reliable systems from the ground up. These patterns apply when you're designing or operating services that must maintain availability, handle failure gracefully, and recover quickly from incidents. They are not about incident response procedures or capacity planning.
A health check endpoint should return 200 for healthy, 503 for unhealthy, and never 2xx for degraded but still serving traffic.
GET /health HTTP/1.1
Host: example.com
HTTP/1.1 200 OK
Content-Type: application/json
{
"status": "healthy",
"timestamp": "2023-06-15T14:30:00Z"
}When a service is unhealthy, it must return 503 with a body indicating the reason.
HTTP/1.1 503 Service Unavailable
Content-Type: application/json
{
"status": "unhealthy",
"reason": "database connection failed",
"timestamp": "2023-06-15T14:30:00Z"
}Health checks that return 2xx when the system is actually unhealthy can cause load balancers to send traffic to broken instances, leading to cascading failures. Many platforms treat any 2xx response as healthy, so a 200 with a body indicating failure can silently pass health checks.
Services must implement graceful shutdown logic that allows for clean connection draining before process termination.
# Example systemd service configuration
[Service]
Type=simple
ExecStart=/usr/local/bin/my-service
TimeoutStopSec=30The shutdown process should:
A common failure mode occurs when a service does not handle SIGTERM properly, causing it to terminate abruptly and drop active connections.
Circuit breakers prevent cascading failures by stopping requests to failing services temporarily.
# Example circuit breaker implementation
import time
from enum import Enum
class CircuitState(Enum):
CLOSED = 1
OPEN = 2
HALF_OPEN = 3
class CircuitBreaker:
def __init__(self, failure_threshold=5, timeout=60):
self.failure_threshold = failure_threshold
self.timeout = timeout
self.failure_count = 0
self.last_failure_time = None
self.state = CircuitState.CLOSED
def call(self, func):
if self.state == CircuitState.OPEN:
if time.time() - self.last_failure_time > self.timeout:
self.state = CircuitState.HALF_OPEN
else:
raise Exception("Circuit breaker is open")
try:
result = func()
self._success()
return result
except Exception as e:
self._failure()
raise
def _success(self):
self.failure_count = 0
self.state = CircuitState.CLOSED
def _failure(self):
self.failure_count += 1
self.last_failure_time = time.time()
if self.failure_count >= self.failure_threshold:
self.state = CircuitState.OPENThe circuit breaker pattern requires careful configuration of failure thresholds and timeout periods. Setting a timeout too short can cause false positives, while setting it too long can delay recovery.
Retry mechanisms must include jitter and exponential backoff to prevent thundering herd behavior.
import random
import time
def retry_with_backoff(func, max_retries=3, base_delay=1):
for attempt in range(max_retries):
try:
return func()
except Exception as e:
if attempt == max_retries - 1:
raise
delay = base_delay * (2 ** attempt) + random.uniform(0, 1)
time.sleep(delay)Without jitter, retries can cause synchronized failures across multiple clients. The default retry behavior in many HTTP libraries is to retry immediately, which creates a thundering herd effect.
All external calls must have explicit timeouts configured at the network level.
import requests
import time
def call_external_api(url):
try:
response = requests.get(
url,
timeout=(5, 10) # (connect_timeout, read_timeout)
)
return response.json()
except requests.Timeout:
raise Exception("External API timed out")The connect and read timeouts should be different. A 5-second connect timeout with a 10-second read timeout prevents hanging connections while allowing longer-running requests.
A common mistake is using only a total timeout, which can cause connection hangs even when the server is responsive.
Services must provide fallback behavior when dependencies fail.
def get_user_data(user_id):
try:
return fetch_from_primary(user_id)
except DatabaseError:
# Fallback to cache
try:
return fetch_from_cache(user_id)
except CacheError:
# Fallback to default values
return {"id": user_id, "name": "Unknown User"}Fallbacks must be carefully designed to avoid cascading failures. A fallback that itself fails can cause a complete service outage.
Alerts should be based on measured SLIs, not system metrics alone.
{
"service": "user-api",
"sli": "latency_p95",
"threshold": 500,
"unit": "ms",
"window": "5m"
}Alerting rules must distinguish between:
A common mistake is alerting on raw error counts instead of error rates, which can cause false positives during low-volume periods.
Alerts must include sufficient context to make them actionable without requiring additional investigation.
{
"alert": "high_latency",
"severity": "warning",
"service": "payment-service",
"instance": "payment-worker-01",
"timestamp": "2023-06-15T14:30:00Z",
"metrics": {
"latency_p95": 1200,
"error_rate": 0.01
}
}Duplicate alerts should be suppressed for a reasonable time window to prevent alert fatigue.
Deployments must include rollback mechanisms and gradual rollout capabilities.
# Example deployment configuration
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend-app
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0The rolling update strategy should allow for zero downtime and prevent partial rollouts from causing issues.
Configuration changes must include validation before deployment.
# Validate configuration before applying
if ! validate_config /etc/my-service/config.yaml; then
echo "Configuration validation failed"
exit 1
fi
# Apply configuration
systemctl reload my-serviceConfiguration validation should check for:
A common failure occurs when services accept invalid configurations and fail silently, leading to unexpected behavior.
Operations that may be retried must be idempotent.
def process_payment(payment_id, amount):
# Check if already processed
if payment_already_processed(payment_id):
return {"status": "already_processed"}
# Process payment
result = charge_card(amount)
# Mark as processed
mark_as_processed(payment_id)
return resultIdempotent operations prevent duplicate processing, which can cause double charges or incorrect state transitions.
When multiple systems must coordinate, use transactional patterns or eventual consistency with compensating actions.
def transfer_money(from_account, to_account, amount):
# Step 1: Reserve funds
reserve_result = reserve_funds(from_account, amount)
if not reserve_result.success:
raise Exception("Could not reserve funds")
try:
# Step 2: Process transfer
transfer_result = process_transfer(from_account, to_account, amount)
if not transfer_result.success:
raise Exception("Transfer failed")
# Step 3: Confirm transaction
confirm_transaction(from_account, to_account, amount)
except Exception as e:
# Compensation: release reserved funds
release_reservation(from_account, amount)
raiseTransaction rollback logic must be implemented carefully to prevent partial state changes.
Logs must include correlation IDs to trace requests across services.
{
"timestamp": "2023-06-15T14:30:00Z",
"request_id": "req-12345",
"service": "user-service",
"method": "GET",
"path": "/users/123",
"level": "info",
"message": "Processing user request"
}Each log entry should include a unique request identifier to enable end-to-end tracing.
Services must clean up resources properly, especially when they fail.
import atexit
class DatabaseConnection:
def __init__(self):
self.connection = create_connection()
atexit.register(self.cleanup)
def cleanup(self):
if self.connection:
self.connection.close()Resource leaks can cause service degradation over time and eventually lead to complete outages.
All external calls must be wrapped in proper error handling with appropriate retry logic and fallback behavior.
def safe_external_call(url):
max_retries = 3
for attempt in range(max_retries):
try:
return requests.get(url, timeout=10)
except requests.RequestException as e:
if attempt == max_retries - 1:
raise
time.sleep(2 ** attempt) # Exponential backoffError handling patterns should distinguish between recoverable and unrecoverable failures to avoid unnecessary retries.
This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy