DevOps Incident Management & On-Call

Build sustainable on-call practices. Covers incident response frameworks, escalation policies, on-call rotations, runbooks, communication protocols, and reducing alert fatigue.

This document describes operational practices for handling production incidents and managing on-call responsibilities in engineering teams. It covers how to respond to incidents, communicate during outages, and maintain alerting systems that distinguish between true problems and noise.

Incident Response Process

Initial Triage and Escalation

When an alert fires, the on-call engineer should immediately verify the alert's validity using the monitoring system's interface or API. For example, with a Prometheus-based system:

curl -H "Authorization: Bearer $TOKEN" \
  "http://prometheus.example.com/api/v1/query?query=up"

If the query returns result: [], then the alert is likely a false positive from a misconfigured rule or a temporary monitoring gap. The on-call engineer should acknowledge the alert and add a comment noting this verification.

The incident response process starts with a status page update. If using a tool like Statuspage, the following API call creates an incident:

curl -X POST \
  -H "Authorization: Bearer $STATUSPAGE_API_KEY" \
  -H "Content-Type: application/json" \
  "https://api.statuspage.io/v1/pages/$PAGE_ID/incidents" \
  -d '{"incident":{"name":"Database connection pool exhaustion","status":"identified","body":"Primary database experiencing connection pool exhaustion leading to application timeouts","impact":"major"}}'

Communication During Incident

All communication should happen through the established incident channel, typically a Slack or Microsoft Teams channel. The channel should be named #incidents-incident-name where incident-name is a descriptive term like db-connection-pool.

When adding team members to an incident, use the /invite command in Slack:

/invite @alice @bob @charlie

If an engineer is not available, they should update their status in the channel:

I'm on a call with the platform team and will return to this incident in 30 minutes.

Incident Documentation

Each incident requires a postmortem entry. The format uses a simple markdown structure that all engineers follow:

# Incident: Database Connection Pool Exhaustion

## Timeline

- 14:23: Alert fired from Prometheus
- 14:25: Initial investigation confirms connection pool exhaustion
- 14:30: Deployment of fix applied to staging
- 14:45: Fix validated in staging, deployed to production
- 15:00: Incident resolved

## Root Cause

The application was opening new database connections without closing them properly. This caused a gradual exhaustion of the connection pool.

## Resolution Steps

1. Restarted affected application pods
2. Applied fix to connection handling logic
3. Verified fix in staging environment
4. Deployed to production
5. Monitored for 30 minutes after deployment

## Postmortem

- [ ] Add alerting rule to detect connection leaks
- [ ] Review all database connection usage in the codebase

Alert Management

Alert Filtering and Suppression

Alerts should be filtered at source using configuration. A typical alert rule in Prometheus might look like:

groups:
- name: application-alerts
  rules:
  - alert: DatabaseConnectionTimeout
    expr: rate(db_connection_timeouts[5m]) > 1
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Database connection timeout detected"
      description: "The database is experiencing timeouts on {{ $labels.instance }}. Rate is {{ $value }} per second."

When an alert fires, the system should include a label with severity=warning for non-critical alerts. The configuration file for this rule must be version-controlled and reviewed during pull requests.

Alert Deduplication

Alert deduplication happens at the alertmanager level. A basic configuration might look like:

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: 'team-email'

receivers:
- name: 'team-email'
  email_configs:
  - to: '[email protected]'

If an alert manager is configured with group_wait: 10s, then all alerts that match the group criteria will be sent together after 10 seconds, rather than immediately.

Alert Suppression During Maintenance Windows

During scheduled maintenance windows, alerts should be suppressed. This can be done with a simple script:

#!/bin/bash
# Suppress alerts for the next 2 hours
curl -X POST \
  -H "Authorization: Bearer $ALERTMANAGER_TOKEN" \
  -H "Content-Type: application/json" \
  "http://alertmanager.example.com/api/v1/silences" \
  -d '{
    "matchers": [
      {"name": "alertname", "value": "DatabaseConnectionTimeout", "isRegex": false}
    ],
    "startsAt": "'$(date -u -Iseconds)'",
    "endsAt": "'$(date -u -Iseconds -d "+2 hours")'",
    "createdBy": "maintenance-script",
    "comment": "Suppressing database connection timeout alerts during maintenance window"
  }'

On-Call Scheduling and Rotation

Rotating Responsibilities

On-call rotation should be managed by a central system. The simplest approach is to use a shared calendar with recurring entries. Each engineer's name appears on the calendar for their assigned period.

The rotation must include a handoff process. When an engineer is handing off, they should:

  1. Document any ongoing issues
  2. Update the status page with current incident status
  3. Add a comment in the incident channel
  4. Send a message to the team's on-call Slack channel

Escalation Procedures

Escalation rules are defined in configuration files. A basic escalation policy might be:

escalation_policies:
  - name: "critical-alerts"
    steps:
      - level: 1
        target: "on-call-engineer"
        timeout_minutes: 10
      - level: 2
        target: "team-lead"
        timeout_minutes: 30
      - level: 3
        target: "engineering-manager"
        timeout_minutes: 60

When an alert reaches a step, the system should send a notification via email or Slack. The notification must include:

Tools and Automation

Status Page Integration

A basic status page update can be done through HTTP API calls:

curl -X PUT \
  -H "Authorization: Bearer $STATUSPAGE_API_KEY" \
  -H "Content-Type: application/json" \
  "https://api.statuspage.io/v1/pages/$PAGE_ID/incidents/$INCIDENT_ID" \
  -d '{"incident":{"status":"resolved","body":"Issue resolved after applying fix to connection handling"}}'

Health Checks and Probes

The health endpoint should return a specific HTTP status code and body structure:

curl -H "Accept: application/json" http://service.example.com/health

A healthy service returns 200 OK with a body like:

{
  "status": "healthy",
  "timestamp": "2023-10-15T14:30:00Z",
  "version": "1.2.3"
}

An unhealthy service returns 503 Service Unavailable with a body:

{
  "status": "unhealthy",
  "timestamp": "2023-10-15T14:30:00Z",
  "details": [
    {
      "component": "database",
      "status": "down",
      "message": "Connection timeout after 30 seconds"
    }
  ]
}

Common Failure Modes

Silent Alert Failures

Alerts may appear in the monitoring system but never reach engineers. This can happen when:

  1. The alert manager configuration is misconfigured
  2. The email or Slack integration fails
  3. The alert rule has incorrect label matching

The symptom is that alerts appear in the UI but no notifications are sent.

False Positives

False positive alerts occur when an alert fires for a problem that isn't actually affecting users. This often happens when:

  1. Alerts are too sensitive (too low thresholds)
  2. Alert rules don't consider system behavior
  3. Alerting is based on metrics rather than user experience

A common pattern is to have an alert fire when CPU usage exceeds 90%, but the service can handle 95% CPU without any user impact.

Escalation Delays

Escalation delays occur when:

  1. The first responder doesn't acknowledge an alert within the timeout period
  2. The escalation system isn't configured to automatically advance
  3. Communication channels are not properly set up

The symptom is that an engineer receives a notification but doesn't respond, and the alert never escalates.

Alert Suppression Issues

Alert suppression can silently fail when:

  1. A silence timer expires incorrectly
  2. The matching criteria for silences are too broad
  3. Silences are applied but not properly documented

The symptom is that alerts that should be suppressed start firing again without explanation.

Best Practices and Patterns

Standardized Response Templates

Use templates for common incident responses:

## Incident Response Template

### Initial Assessment
- [ ] Verify alert validity
- [ ] Check related systems
- [ ] Document findings in channel

### Investigation Steps
1. Review recent changes
2. Check system logs
3. Analyze metrics
4. Test fixes in staging

### Resolution Checklist
- [ ] Apply fix
- [ ] Deploy to production
- [ ] Monitor for 30 minutes
- [ ] Close incident

Communication Standards

All communication should follow these rules:

  1. All status updates go into the incident channel
  2. Responses to alerts must include a timestamp
  3. Escalations must include a reason for escalation
  4. Handoffs must include a summary of current state

Postmortem Process

Postmortems must include:

  1. Timeline of events
  2. Root cause analysis
  3. Resolution steps
  4. Action items for prevention
  5. Assignments for follow-up tasks

The postmortem should be shared with the team within 24 hours of incident resolution, and all action items must have owners assigned.

Configuration Examples

Alertmanager Configuration

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: 'team-email'

receivers:
- name: 'team-email'
  email_configs:
  - to: '[email protected]'
    from: '[email protected]'
    smarthost: 'smtp.example.com:587'
    auth_username: '[email protected]'
    auth_password: '$ALERTS_PASSWORD'

Prometheus Rule Configuration

groups:
- name: application-alerts
  rules:
  - alert: HighErrorRate
    expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "High error rate detected"
      description: "Error rate is {{ $value }} on {{ $labels.instance }}"

The configuration files should be version-controlled and reviewed in pull requests before deployment. Any changes to alerting rules or escalation policies require a code review process.

Troubleshooting

Alert Not Firing

When an alert doesn't fire, check:

  1. The rule expression in Prometheus
  2. Whether the alert manager has received the alert
  3. Whether the receiver configuration is correct
  4. Whether there are active silences that suppress the alert

Notification Delivery Failure

When notifications fail:

  1. Verify email or Slack integration credentials
  2. Check if the notification service is running
  3. Confirm the target email or channel exists
  4. Review logs for specific error messages

Escalation Not Working

When escalation fails:

  1. Check that the escalation policy matches alert labels
  2. Verify that all steps in the policy have correct timeouts
  3. Confirm that the escalation system has proper permissions
  4. Review whether the first responder acknowledged properly

This approach ensures consistent incident handling across teams and systems, with clear boundaries between alerting, communication, and resolution processes.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.