How to Build an Effective Incident Response Playbook

Incident response playbooks are structured, step-by-step guides that define how teams respond to specific types of incidents. They are critical during high-pressure events—when engineers, security analysts, and operations staff must act quickly, consistently, and correctly under stress. A playbook ensures that no critical step is missed, from initial detection to post-incident review, and it becomes the shared language across teams when the lights go out.

Define Incident Types and Categories

An effective playbook begins by clearly categorizing incidents. Each category should map to a distinct response workflow. Use a simple, consistent taxonomy: Severity, Impact, and Triggering Event. Define severity levels as *Minor*, *Major*, *Critical*, and *Catastrophic*, with clear criteria.

For example:

• Minor: Service degradation with no user impact, logs show error rate > 5% over 15 minutes.
• Major: Users report issues; SLA breached; error rate > 10% over 30 minutes.
• Critical: Outage affecting core functionality; users unable to complete key workflows; 90% error rate.
• Catastrophic: Complete service downtime; data loss or corruption; impact to multiple services.

Each incident type should be associated with a unique identifier: INC-2023-001, INC-2023-002, etc. Use this ID in all communications, alerts, and documentation.

Define the *triggering event* as the signal that activates the playbook. This is typically a metric threshold, a log pattern, or a specific alert condition. For example:

{
  "alert_name": "High Error Rate in Checkout Service",
  "trigger": "rate(errors, 10m) > 0.1",
  "conditions": [
    {
      "metric": "http.errors",
      "window": "10m",
      "threshold": 0.1,
      "aggregation": "rate",
      "comparison": "above"
    },
    {
      "log_query": "service:checkout AND level:error",
      "min_events": 50,
      "time_window": "5m"
    }
  ]
}

Use a standard alerting format across systems. Alert definitions should be stored in version-controlled files, such as alerts/checkout-service.yaml:

name: checkout-service-high-error-rate
description: >-
  Detects increased error rate in the checkout service.
  Triggers when error rate exceeds 10% over 10 minutes.
severity: Critical
labels:
  team: payments
  service: checkout
  incident_type: service-outage
for: 10m
query: |
  rate(
    http_errors{service="checkout"}[10m]
  ) > 0.1
notifications:
  - webhook: https://webhook.example.com/incident
    template: |
      Incident: {{.Incident.ID}}
      Service: {{.Service}}
      Error Rate: {{.Value}}%
      Duration: {{.Duration}}

Structure the Playbook Workflow

Each playbook should follow a defined lifecycle: *Detection*, *Triage*, *Containment*, *Eradication*, *Recovery*, *Post-Mortem*. Each phase contains a checklist of tasks, responsibilities, and expected outcomes.

Detection Phase

The first action after an incident is detected is to confirm the event using a canonical script. Use a check.incident.sh script that runs on all relevant systems:

#!/bin/bash

set -euo pipefail

readonly INCIDENT_ID="$1"
readonly SERVICE_NAME="$2"
readonly ALERT_SOURCE="$3"

echo "🔍 Validating incident: $INCIDENT_ID"
echo "  Service: $SERVICE_NAME"
echo "  Source: $ALERT_SOURCE"

# Check if service is up
if ! curl -sf http://$SERVICE_NAME:8080/health; then
  echo "❌ Service is down"
  exit 1
fi

# Check logs for recent errors
if ! psql -c "
  SELECT count(*) FROM logs
  WHERE service = '$SERVICE_NAME'
    AND level = 'error'
    AND created_at > now() - interval '10 minutes'
  " -t -A -F -q; then
  echo "❌ High error rate in logs"
fi

# Check queue backlog
if ! docker exec -i kafka-broker bin/kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list localhost:9092 \
  --topic checkout-events \
  --time -1; then
  echo "❌ Queue backlog is growing"
fi

echo "✅ Incident validated"

This script runs on every incident detection. It is executed via a CI/CD job or a runner service, and its output is logged and stored in a shared directory.

Triage Phase

Once confirmed, the incident enters triage. Assign a primary responder and a secondary. Use a standard triage template:

# Incident: INC-2023-001
## Triage Summary
- **Service**: Checkout
- **Severity**: Critical
- **Detected**: 2023-04-15T08:32:14Z
- **Primary Responder**: [email protected]
- **Secondary Responder**: [email protected]
- **Initial Observations**:
  - HTTP 5xx rate: 32%
  - 180 errors in last 10 minutes
  - Kafka queue backlog: 12,000 messages
- **Next Steps**:
  - [ ] Verify database connection pool exhaustion
  - [ ] Check recent deployments
  - [ ] Run `check.incident.sh` on all instances

The triage phase is documented in a shared document, updated in real time. Use a markdown file at playbooks/incidents/INC-2023-001.md.

Containment Phase

Containment aims to stabilize the incident and prevent further degradation. Use a contain.incident.sh script:

#!/bin/bash
# contain.incident.sh
set -euo pipefail

readonly INCIDENT_ID="$1"
readonly THRESHOLD="${2:-0.15}"

echo "🛡️ Containing incident: $INCIDENT_ID"

# Scale up service
kubectl scale deployment checkout --replicas=12 -n checkout

# Enable circuit breaker in API gateway
curl -X PUT http://api-gateway:8080/circuit-breakers/checkout \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "threshold": 0.15,
    "timeout": 10000,
    "failureThreshold": 5
  }'

# Update alert to reduce noise
curl -X PATCH https://alertmanager.example.com/api/v1/alerts \
  -H "Content-Type: application/json" \
  -d '{
    "matchers": [
      { "name": "incident_id", "value": "'$INCIDENT_ID'" }
    ],
    "labels": {
      "state": "containing",
      "severity": "warning"
    },
    "annotations": {
      "description": "Incident in containment phase"
    }
  }'

# Log containment action
echo "Containment applied at $(date)" >> /var/log/incidents/$INCIDENT_ID.log

This script is idempotent and runs on each incident. It can be triggered manually or automated via a webhook.

Eradication Phase

Eradication focuses on root cause analysis. Use a standard query to gather logs, metrics, and traces.

# Gather all data for incident
export INCIDENT_ID="INC-2023-001"
export START_TIME=$(date -Iseconds -d "10 minutes ago")

# Fetch logs from all services
for service in checkout payments inventory; do
  psql -c "
    COPY (
      SELECT * FROM logs
      WHERE service = '$service'
        AND created_at BETWEEN '$START_TIME' AND now()
      ORDER BY created_at
    ) TO '/tmp/incidents/$INCIDENT_ID/$service-logs.csv'
    WITH CSV HEADER;
  " -q
done

# Export metrics from Prometheus
curl -s "http://prometheus:9090/api/v1/query?query=rate(http_errors{service=\"checkout\"}[10m])&time=$(date -Iseconds)" \
  -o "/tmp/incidents/$INCIDENT_ID/metrics.json"

# Create a trace bundle
docker exec -i jaeger-query bin/jaeger-cli trace \
  --service=checkout \
  --start-time="$START_TIME" \
  --end-time=$(date -Iseconds) \
  --format=zip \
  > "/tmp/incidents/$INCIDENT_ID/trace.zip"

This bundle is uploaded to a shared storage (e.g., S3, NFS) and linked in the incident document.

Recovery Phase

Recovery restores full functionality. Use a recover.incident.sh script:

#!/bin/bash
# recover.incident.sh
set -euo pipefail

readonly INCIDENT_ID="$1"

echo "🔄 Recovering incident: $INCIDENT_ID"

# Roll back to stable version
kubectl set image deployment checkout \
  checkout=checkout:1.2.3 \
  -n checkout

# Disable circuit breaker
curl -X PUT http://api-gateway:8080/circuit-breakers/checkout \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": false
  }'

# Verify recovery
if ! curl -sf http://checkout:8080/health; then
  echo "❌ Health check failed after recovery"
  exit 1
fi

# Send recovery confirmation
curl -X POST https://webhook.example.com/incident \
  -H "Content-Type: application/json" \
  -d '{
    "incident_id": "'$INCIDENT_ID'",
    "status": "recovered",
    "message": "Service recovered successfully"
  }'

echo "✅ Incident $INCIDENT_ID recovered"

Post-Mortem Phase

After recovery, conduct a post-mortem. Use a standard template with the following sections:

• Incident Summary
• Timeline of Events
• Root Cause Analysis
• Actions Taken
• Lessons Learned
• Follow-up Tasks

Store this in postmortems/INC-2023-001.md. Use a checklist to ensure consistency:

- [ ] Incident title and ID included
- [ ] Timeline spans from detection to recovery
- [ ] Root cause identified and documented
- [ ] Actions taken listed with owners
- [ ] Lessons learned in bullet form
- [ ] Follow-up tasks with due dates
- [ ] Link to trace bundle, logs, and metrics

Maintain and Evolve the Playbook

A playbook is not static. It must be reviewed and updated after every incident. Create a playbook-review.sh script that runs after each incident:

#!/bin/bash
# playbook-review.sh
set -euo pipefail

readonly INCIDENT_ID="$1"

echo "📝 Reviewing playbook for $INCIDENT_ID"

# Identify missing steps
if ! grep -q "Check database connection pool" \
  playbooks/incidents/$INCIDENT_ID.md; then
  echo "⚠️ Missing step: Check database connection pool"
  echo "  Add to playbook: 'check.db.pool.sh'"
fi

# Check for outdated scripts
for script in check.incident.sh contain.incident.sh; do
  if ! git diff --exit-code origin/main -- "scripts/$script"; then
    echo "🔄 Script $script has changes not in main"
  fi
done

# Update playbook version
git add .
git commit -m "Update playbook for $INCIDENT_ID"
git push origin main

This script runs after every incident. It ensures the playbook evolves with experience.

Failure Modes and Silent Pitfalls

• Alerts fire but playbook is not triggered: Ensure alerting systems push to a central incidents queue (e.g., Kafka or RabbitMQ), and a worker consumes this queue to create a new playbook entry.
• Scripts fail silently: Add set -euo pipefail to all scripts. Use echo and date in logs for traceability.
• Incorrect severity assignment: Define severity thresholds in a config/severities.yaml file and validate them automatically:

  critical:
    error_rate: 0.1
    duration: 15m
    impact: "core"
  major:
    error_rate: 0.05
    duration: 30m
    impact: "significant"

• Missing post-mortem: Use a cron job that checks for incidents without post-mortems and sends reminders.
• Inconsistent incident IDs: Enforce a naming convention via a pre-commit hook in Git:

  # .git/hooks/pre-commit
  #!/bin/bash
  if ! git log -1 --pretty=format:%s | grep -q "INC-"; then
    echo "⚠️ Commit message must include incident ID"
    exit 1
  fi

Final Touches

• Standardize incident status labels: detected, triaged, containing, eradicating, recovered, post-mortem.
• Use a shared incident dashboard: A simple HTML page updated via CI that shows all active incidents with their status, severity, and primary responder.
• Document how to join the playbook: New engineers should be able to run playbook.join.sh to get access to all incident tools, alerting systems, and documentation.

A playbook is only as good as its execution. The most effective playbooks are not just documents—they are executable systems, continuously validated and refined by the very incidents they were built to answer.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.