Incident Management

A durable model for running production incidents: roles and span of control, severity that binds a commitment, mitigate-before-diagnose, communication cadence, and blameless review that actually changes the system rather than the person.

When an alert fires and engineers must respond, incident management governs how teams organize, communicate, and resolve issues. This process begins when an alert transitions from passive monitoring to active human intervention, requiring structured coordination across roles, tools, and timelines.

Create and Assign Incidents

Create a new incident using the standard HTTP POST endpoint /incidents with JSON payload:

{
  "title": "Database connection pool exhausted",
  "description": "Primary database reports 500 errors from connection pool exhaustion",
  "severity": "P2",
  "owner": "database-team",
  "impact": "All user-facing API endpoints failing"
}

The system assigns a unique incident ID (UUID format) and creates an initial incident log entry. The response includes:

{
  "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "open",
  "created_at": "2023-10-15T14:30:00Z",
  "escalation_policy": "database-team-oncall"
}

Assign ownership using the PATCH endpoint /incidents/{id}:

{
  "owner": "db-sre-team",
  "status": "in_progress"
}

Update and Communicate

Update incident status with HTTP PUT to /incidents/{id}/status:

{
  "status": "resolved",
  "resolution": "Increased pool size from 20 to 100"
}

Add a log entry using POST to /incidents/{id}/log:

{
  "timestamp": "2023-10-15T14:45:00Z",
  "author": "[email protected]",
  "message": "Executed database restart script"
}

Escalate and Route

Escalate to another team using the HTTP POST to /incidents/{id}/escalate:

{
  "target_team": "frontend-team",
  "reason": "Database fix required frontend code changes"
}

Route incident to an external system using a webhook configuration. The platform sends JSON payloads to configured endpoints:

{
  "incident_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "in_progress",
  "title": "Database connection pool exhausted",
  "timestamp": "2023-10-15T14:30:00Z"
}

Track and Close

Close an incident by updating its status:

{
  "status": "closed",
  "resolution": "Database connection pool increased from 20 to 100. All endpoints restored.",
  "closed_at": "2023-10-15T15:30:00Z"
}

Configuration and Integration

Configure incident management through HTTP API endpoint /config/incident. The configuration includes:

{
  "default_severity": "P3",
  "auto_assign_to_owner": true,
  "escalation_rules": [
    {
      "condition": "incident_duration > 30m",
      "target_team": "database-team",
      "delay_minutes": 15
    }
  ],
  "webhook_endpoints": [
    {
      "url": "https://internal.slack.com/services/hooks/incident",
      "format": "json"
    }
  ]
}

Command Line Interface

Use the incidentctl command for local operations:

incidentctl create \
  --title "Database connection pool exhausted" \
  --description "Primary database reports 500 errors from connection pool exhaustion" \
  --severity P2 \
  --owner database-team

incidentctl update \
  --id a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
  --status resolved \
  --resolution "Increased pool size from 20 to 100"

incidentctl log \
  --id a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
  --message "Executed database restart script"

Failure Modes and Sharp Edges

Silent Failures: When the incident management system fails to send a webhook, there is no error returned from the POST request. The payload appears to succeed but silently drops messages.

Incorrect Severity Assignment: Using P1 for a minor configuration issue can result in 24/7 on-call rotation, while P4 issues may be ignored entirely. No validation prevents assignment of severity levels not defined in the system.

Escalation Delays: Escalation rules may trigger after a delay, but the system does not provide visibility into when escalation will occur or what happens if it fails to execute.

Duplicate Incident Creation: When multiple alerts fire simultaneously, the system may create multiple incidents instead of aggregating them. This occurs when the deduplication logic fails on non-unique fields.

Missing Ownership: If an incident has no owner assigned, the system does not automatically assign it to a default team. The incident remains unassigned until manually updated.

Timezone Handling: All timestamps are stored in UTC, but user interfaces may display times in local time zones without clear indication of which timezone is being used.

Common Error Responses

When creating an incident with invalid data:

{
  "error": "validation_failed",
  "details": [
    {
      "field": "severity",
      "message": "Severity must be one of P1, P2, P3, P4"
    }
  ]
}

When updating a non-existent incident:

{
  "error": "incident_not_found",
  "message": "Incident with ID a1b2c3d4-e5f6-7890-abcd-ef1234567890 does not exist"
}

Integration Points

The incident management system integrates with monitoring through webhook endpoints. When an alert fires, the monitoring system POSTs to /incidents/webhook with:

{
  "alert_id": "alert-123",
  "service": "database-service",
  "timestamp": "2023-10-15T14:30:00Z",
  "severity": "P2",
  "details": {
    "metric": "connection_pool_usage",
    "value": "98%",
    "threshold": "95%"
  }
}

The system returns:

{
  "incident_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "created"
}

Logging and Auditing

All incident updates are logged with full audit trail. The system stores:

The audit log can be retrieved via HTTP GET to /incidents/{id}/audit:

[
  {
    "timestamp": "2023-10-15T14:30:00Z",
    "user": "[email protected]",
    "action": "created",
    "details": "Incident created with severity P2"
  },
  {
    "timestamp": "2023-10-15T14:35:00Z",
    "user": "[email protected]",
    "action": "updated_status",
    "details": "Status changed from open to in_progress"
  }
]

Incident Templates

Predefined templates allow consistent incident creation. The system supports:

{
  "template_name": "database_outage",
  "title": "Database Service Outage",
  "description": "Database service is unavailable or responding slowly",
  "severity": "P1",
  "default_owner": "database-team"
}

Multi-Team Coordination

When incidents span multiple teams, the system supports:

The system allows for manual assignment of secondary teams via:

{
  "secondary_owners": ["frontend-team", "devops-team"]
}

Data Retention and Archiving

Closed incidents are retained for 90 days. After this period, the system archives them with reduced metadata access. The archiving process occurs automatically without user intervention.

The system does not provide export capabilities for incident data in bulk formats beyond individual JSON exports per incident.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.