Build infrastructure that detects failures and recovers automatically. Covers health check design, auto-restart policies, auto-scaling, circuit breakers, automated runbooks, and the observability foundation required to trust automated remediation.
Self-healing infrastructure automates recovery from failures without human intervention. It detects system degradation or outages, diagnoses root causes, and applies corrective actions—ranging from restarting services to rolling back deployments—before users notice. This is essential in environments with high uptime requirements, rapidly changing systems, and distributed components where manual diagnosis and response are too slow.
Failures begin when a component deviates from expected behavior. Detection relies on monitoring signals: metrics, logs, traces, and health checks. A service is considered unhealthy when a metric crosses a threshold, a log contains an error, or a health check endpoint returns failure.
A typical health check endpoint returns 200 OK with a JSON body:
{
"status": "healthy",
"timestamp": "2024-05-17T10:30:00Z",
"services": {
"database": { "status": "up", "latency_ms": 12 },
"cache": { "status": "up", "hits": 94.3 }
}
}When the endpoint returns 503 Service Unavailable or a status of unhealthy, a monitoring system triggers an alert.
Failures are detected via periodic checks:
while true; do
curl -f http://localhost:8080/health
if [ $? -ne 0 ]; then
echo "Health check failed at $(date)" >> /var/log/health-check.log
# Send alert via HTTP
curl -X POST -d '{
"title": "Service Unavailable",
"message": "Health check failed at 2024-05-17T10:30:00Z",
"severity": "critical"
}' http://alerting-system:8080/alert
fi
sleep 10
doneThe key insight: a failure is not a single event—it is a sustained state. A transient network glitch may cause a single 503 response, but only if the same condition persists across multiple intervals is it considered a failure.
Thresholds define when a metric is considered abnormal. For example:
# Configuration for monitoring a service's response time
metrics:
- name: http_response_time_ms
source: prometheus
query: 'avg(rate(http_response_time_ms[5m]))'
threshold:
critical: 200
warning: 100
duration: 5m
evaluation:
- "all"
- "last"A metric is critical if it exceeds 200ms for five consecutive minutes. The evaluation strategy determines how values are combined: all requires every sample in the window to exceed the threshold; last only requires the most recent sample.
The duration field is often misunderstood: it governs how long the condition must hold before triggering. If duration is 5m, but checks occur every 30s, then five consecutive checks must pass before the failure is confirmed.
When a failure is detected, it is stored in a failure state database:
CREATE TABLE failures (
id UUID PRIMARY KEY,
service_name TEXT NOT NULL,
metric_name TEXT,
value FLOAT,
threshold FLOAT,
start_time TIMESTAMP,
end_time TIMESTAMP,
status TEXT CHECK (status IN ('detected', 'in_progress', 'resolved')),
metadata JSONB
);A failure propagates to downstream services. A database outage triggers alerts for all services that depend on it. The propagation is not just a notification—it is a failure graph:
{
"id": "f3a1b4c5-2d6e-7f8g-9h0i",
"service": "payment-service",
"cause": "database-unavailable",
"timestamp": "2024-05-17T10:31:22Z",
"affected": [
"checkout-service",
"user-service",
"notification-worker"
],
"root_cause": {
"service": "postgres",
"metric": "connection_pool_usage",
"value": 0.98,
"threshold": 0.95
}
}Once a failure is detected, the system diagnoses its root cause. Diagnosis involves correlating data from multiple sources: logs, metrics, traces, and configuration.
Logs are the primary source of context. A failure in a service’s error.log file:
2024-05-17T10:30:15Z [ERROR] [db] Failed to connect to PostgreSQL: timeout (5000ms)
2024-05-17T10:30:16Z [ERROR] [api] Database connection pool exhaustedThe diagnosis engine queries logs for all entries in the last 10 minutes, grouped by service and level:
SELECT
service,
level,
message,
count(*) as occurrences,
min(timestamp) as first_occurrence,
max(timestamp) as last_occurrence
FROM logs
WHERE timestamp BETWEEN '2024-05-17T10:20:00Z' AND '2024-05-17T10:30:00Z'
AND level = 'ERROR'
AND service = 'payment-service'
GROUP BY service, level, message
HAVING count(*) > 5
ORDER BY occurrences DESC;The most frequent error is likely the root cause. A message like “Database connection pool exhausted” with 12 occurrences is a strong candidate.
Metrics from multiple sources are combined. For example:
database.cpu_utilization > 90% for 10 minutesapplication.memory_usage > 95% for 15 minutesnetwork.latency > 200ms between application and databaseA diagnosis rule matches a combination:
diagnosis_rules:
- name: "Database_connection_pool_exhausted"
conditions:
- metric: "database.cpu_utilization"
threshold: 90
duration: 10m
- metric: "application.memory_usage"
threshold: 95
duration: 15m
- metric: "network.latency"
threshold: 200
duration: 5m
action: "restart-database"
priority: 1The system evaluates all rules against the current state. A rule fires when all conditions are met simultaneously.
Correlation is not just timing—it is causal inference. If application.memory_usage spikes 20 seconds after database.cpu_utilization exceeds 90%, the system infers that database load causes application memory pressure.
The inference engine uses time-lagged cross-correlation:
def find_root_cause(time_series, target_series, lag_range=(-30, 30)):
results = {}
for lag in lag_range:
shifted = target_series.shift(lag)
correlation = np.corrcoef(time_series, shifted)[0, 1]
results[lag] = correlation
best_lag = max(results, key=results.get)
return {
"cause": "target_series",
"effect": "time_series",
"lag_seconds": best_lag,
"correlation": results[best_lag]
}If the best lag is +15 seconds, the system concludes that database CPU load causes application memory usage with a 15-second delay.
Once root cause is identified, the system applies a remediation action. Actions are defined as idempotent operations that can be executed repeatedly without side effects.
Actions are defined in a standard format:
remediation_actions:
- name: "restart-database"
description: "Restart PostgreSQL service"
type: "systemd"
command: "systemctl restart postgresql"
timeout: 30s
conditions:
- metric: "database.cpu_utilization"
threshold: 90
duration: 5m
retry:
max_attempts: 3
backoff: "exponential"
delay: 5s
post_action:
- "send_alert"
- "update_dashboard"The type field determines how the action is executed. Common types:
systemd: runs systemctl commandskubernetes: applies a YAML manifest via kubectlhttp: sends a POST request to an endpointscript: executes a shell scriptThe execution engine runs actions in a queue, ensuring idempotency and reliability.
A systemd action runs:
# Executing a remediation action
systemctl restart postgresql
if [ $? -ne 0 ]; then
echo "Failed to restart postgresql" >> /var/log/remediation.log
exit 1
fi
# Verify service is up
until systemctl is-active --quiet postgresql; do
sleep 1
echo "Waiting for postgresql to become active..."
done
echo "PostgreSQL restarted successfully"The action is idempotent: running it twice produces the same result.
Actions can be chained. One action may trigger another:
remediation_actions:
- name: "restart-database"
type: "systemd"
command: "systemctl restart postgresql"
triggers:
- "restart-application"
- "send_alert"When restart-database completes, it triggers restart-application and send_alert. The triggering is not just a POST but a dependency graph:
{
"action": "restart-database",
"started": "2024-05-17T10:35:00Z",
"completed": "2024-05-17T10:36:45Z",
"dependencies": [
{
"action": "restart-application",
"status": "running",
"started": "2024-05-17T10:35:30Z"
},
{
"action": "send_alert",
"status": "completed",
"started": "2024-05-17T10:36:45Z"
}
]
}If a dependency fails, the parent action is marked as failed and may retry.
Retry logic is not just retry: 3—it must specify backoff strategy:
retry:
max_attempts: 3
backoff: "exponential"
base_delay: 2s
multiplier: 2
jitter: trueWith exponential backoff:
Jitter adds random variation:
def next_delay(base, multiplier, jitter):
delay = base * (multiplier ** attempts)
return delay * (1 + random.uniform(0, 0.5))A failure that occurs during a long-running action may be retried with increasing delays, avoiding cascading failures.
After applying actions, the system verifies that the failure has been resolved.
Verification checks whether the system has returned to expected state:
verification:
- metric: "database.cpu_utilization"
threshold: 70
duration: 5m
target: "steady"
- log: "PostgreSQL restarted successfully"
source: "postgresql"
level: "INFO"
min_occurrences: 1
- http:
url: "http://localhost:8080/health"
expected_status: 200
expected_body: {
"status": "healthy"
}
timeout: 60sThe system waits until all conditions are met before marking the failure as resolved.
If verification fails, the system can roll back to a previous state.
A rollback action:
remediation_actions:
- name: "rollback-deployment"
type: "kubernetes"
command: "kubectl rollout undo deployment/payment-service"
condition: "verification-failed"
fall_back:
- "restart-database"
- "scale-down-application"If the new deployment fails verification, the system rolls back and also takes additional actions to stabilize the system.
The system maintains a resolution state:
{
"failure_id": "f3a1b4c5-2d6e-7f8g-9h0i",
"resolved_at": "2024-05-17T10:40:12Z",
"diagnosis": {
"root_cause": "database.cpu_utilization > 90 for 10m",
"correlation": {
"lag": 15,
"confidence": 0.87
}
},
"actions_applied": [
"restart-database",
"send_alert",
"update_dashboard"
],
"verification": {
"status": "success",
"details": [
"cpu_utilization < 70 for 5m",
"health check returned 200",
"log message found"
]
}
}This state is used for auditing and future diagnosis.
A common failure: a metric is above threshold for 10 minutes, but the evaluation window is 5 minutes. The system expects the metric to be above threshold for 5 minutes, but the check runs every 10 seconds. If only three consecutive checks pass, the system may miss a transient spike.
The solution: ensure the evaluation window matches the check interval. Use duration: 5m and interval: 10s, but also add buffer: 2m so that a failure detected at t=10m is not confirmed until t=12m.
all and any EvaluationThe evaluation strategy is often misunderstood. all requires every sample in the window to meet the threshold. any requires only one. But all is not the same as all_samples.
In all, if the window is 10 minutes and checks occur every 30s, then 20 samples are collected. For the metric to be critical, all 20 must be above threshold.
In any, only one sample needs to be above threshold.
When two failures occur close in time, the system may merge them. A database outage starts at 10:30, a service outage starts at 10:33. If the second failure is detected within 1m of the first, the system merges them into a single failure event.
The merge is not just a union—it includes merge logic:
merge:
threshold: 60s
strategy: "overlap"
fields: ["service", "metric", "source"]
deduplication: "by_id"If two failures have the same service and metric, and their start_time is within 60 seconds, they are merged into one.
A frequent mistake: actions are applied, but verification is skipped.
For example:
remediation_actions:
- name: "restart-database"
command: "systemctl restart postgresql"
verification: falseThe system runs the command but does not check whether the database is actually healthy. The action is applied, but the failure is not resolved.
The fix: always require verification, and make it the default behavior.
An action is idempotent only if it returns the same result when run multiple times. A common failure: a script that starts a service but does not check whether it is already running.
# Non-idempotent script
systemctl start postgresql
echo "Started PostgreSQL"If the service is already running, this script does nothing—but it should not fail.
The correct idempotent version:
# Idempotent script
if ! systemctl is-active --quiet postgresql; then
systemctl start postgresql
echo "Started PostgreSQL"
else
echo "PostgreSQL already running"
fiThis pattern—checking before acting—is critical.
Self-healing infrastructure automates recovery from failures by detecting sustained deviations, diagnosing root causes through correlation and inference, and applying verified remediation actions. It relies on precise thresholds, duration evaluation, and idempotent actions. The system must be resilient to silent failures, evaluation confusion, and incomplete verification. The result is a system that heals itself, reducing downtime and operational overhead.
This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy