Incident Management

Handle production incidents effectively from detection through resolution to postmortem. Covers incident severity classification, commander role, communication templates, timeline documentation, and building an incident management program that improves with every incident.

When an alert triggers or a service degrades, incident management is the structured process of identifying, analyzing, and resolving the problem while maintaining communication with stakeholders. This process includes defining the scope of impact, assigning ownership, tracking resolution steps, and documenting outcomes.

Defining an Incident

An incident begins when a system deviates from its expected behavior. The initial signal can come from monitoring tools, user reports, or automated alerts. To define an incident, you must establish a clear scope of what is broken and how it affects users.

Creating the Incident Record

Create a new incident by generating a unique identifier that will track all related activities. This identifier should be consistent across communication channels and systems. For example:

curl -X POST https://internal.ops/api/incidents \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"title": "Database connection pool exhausted", "priority": "P1"}'

The response includes an incident_id field that must be used in all subsequent API calls for this incident.

Assigning Incident Priority

Priority levels are defined as P0 (critical), P1 (high), P2 (medium), and P3 (low). The priority is set during creation and can be updated later.

curl -X PATCH https://internal.ops/api/incidents/$INCIDENT_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"priority": "P1"}'

The priority affects escalation rules and communication cadence.

Incident Response Process

Once an incident is defined, the response process begins with assigning responsibility and gathering information.

Assigning Ownership

Assign an incident commander who leads the response. This person coordinates all activities and ensures decisions are made quickly.

curl -X POST https://internal.ops/api/incidents/$INCIDENT_ID/assign \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"user_id": "[email protected]"}'

The user must be a member of the relevant team or have access to the system in question.

Updating Status and Communication

Keep all stakeholders informed by updating the incident status regularly. This can include technical details, progress updates, and planned fixes.

curl -X POST https://internal.ops/api/incidents/$INCIDENT_ID/status \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"status": "investigating", "message": "Root cause identified, working on fix"}'

Each status update should include a short message that explains the current state.

Root Cause Analysis

After resolving an incident, perform a root cause analysis to prevent future occurrences.

Documenting Findings

Record the sequence of events leading to the failure. Include logs, configuration changes, and any external factors.

curl -X POST https://internal.ops/api/incidents/$INCIDENT_ID/analysis \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "summary": "Database pool exhausted due to unbounded connection requests",
    "timeline": [
      {"time": "2024-01-01T10:00:00Z", "event": "Alert triggered"},
      {"time": "2024-01-01T10:05:00Z", "event": "Incident created"}
    ],
    "root_cause": "Application opened connections without proper closing"
  }'

The timeline must be in chronological order and include timestamps.

Identifying Corrective Actions

Define specific actions to reduce likelihood of recurrence. These should be measurable and owned by a team or individual.

curl -X POST https://internal.ops/api/incidents/$INCIDENT_ID/actions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "Implement connection pooling limits",
    "owner": "backend-team",
    "due_date": "2024-01-15T00:00:00Z"
  }'

Each action must have an owner and a due date.

Escalation Policies

Incidents may require escalation when they do not progress as expected or when the assigned team lacks resources.

Triggering Escalation

If no updates occur within a defined time window, escalate to the next tier of support.

curl -X POST https://internal.ops/api/incidents/$INCIDENT_ID/escalate \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"level": "P0", "message": "No progress in last 30 minutes"}'

Escalation messages must be informative and actionable.

Managing Escalated Incidents

Once escalated, the new team or individual should take over ownership and continue tracking the incident.

curl -X POST https://internal.ops/api/incidents/$INCIDENT_ID/assign \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"user_id": "[email protected]"}'

The escalation process must preserve all prior context.

Communication and Stakeholder Management

Effective communication during an incident is critical for maintaining trust and transparency.

Updating External Communication Channels

Notify stakeholders via email, Slack, or other systems using a standardized message format.

curl -X POST https://internal.ops/api/incidents/$INCIDENT_ID/notify \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "channel": "slack",
    "message": "Database incident ongoing. ETA for resolution: 30 minutes.",
    "recipients": ["team-leads", "executives"]
  }'

Notifications must specify the communication channel and intended audience.

Managing Internal Updates

Keep internal teams informed using structured updates that include status, impact, and next steps.

curl -X POST https://internal.ops/api/incidents/$INCIDENT_ID/internal-update \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Investigation shows memory leak in API gateway",
    "impact": "Users may experience timeouts",
    "next_steps": ["Rollback to previous version", "Deploy monitoring patch"]
  }'

Internal updates must be clear and actionable.

Post-Incident Review

After resolving an incident, conduct a review to identify improvements in process or system design.

Closing the Incident

Mark the incident as resolved once all corrective actions are complete.

curl -X PATCH https://internal.ops/api/incidents/$INCIDENT_ID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"status": "resolved", "resolution": "Deployed fix to connection pool handling"}'

The resolution message should summarize the actions taken.

Evaluating Performance

Review how long the incident took to resolve and what could have been done differently.

curl -X POST https://internal.ops/api/incidents/$INCIDENT_ID/performance \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "duration_minutes": 45,
    "impact_score": 3,
    "lessons_learned": ["Improve monitoring for connection pool usage"]
  }'

Performance metrics must be quantifiable and related to the incident's impact.

Common Failure Modes

Several issues can silently break an incident management process.

Silent Failures in API Calls

API calls may succeed without actually creating or updating the incident. For example, a malformed payload might return HTTP 200 but not create any record.

# This call returns success but does not create an incident
curl -X POST https://internal.ops/api/incidents \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"invalid_field": "value"}'

Always verify the response body for actual data creation.

Incorrect Priority Assignment

Assigning a priority level that doesn't match the severity can delay responses or lead to miscommunication.

# P0 should be reserved for outages affecting all users
curl -X POST https://internal.ops/api/incidents \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"title": "Minor UI bug", "priority": "P0"}'

Unassigned Incidents

An incident without an owner can stall progress indefinitely.

# Incident created but not assigned to anyone
curl -X POST https://internal.ops/api/incidents \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"title": "Service degradation", "priority": "P1"}'

Always assign an incident commander during creation.

Incomplete Root Cause Analysis

Incomplete analysis can lead to recurring issues without proper fixes.

# Missing timeline in analysis
curl -X POST https://internal.ops/api/incidents/$INCIDENT_ID/analysis \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "summary": "Service failed",
    "root_cause": "Wrong configuration"
  }'

The timeline must include all major events.

Configuration and Integration Points

Incident management integrates with several internal systems.

Monitoring Tool Integration

Ensure alerts from monitoring tools are correctly formatted for ingestion.

# Example alert payload
{
  "alert_name": "High CPU Usage",
  "service": "api-gateway",
  "timestamp": "2024-01-01T10:00:00Z",
  "severity": "warning"
}

Alerts must include a timestamp and service name.

Escalation Configuration

Define how incidents are escalated based on time or priority.

# Example escalation policy
escalation_rules:
  - priority: P0
    timeout_minutes: 30
    next_level: "on-call-engineers"
  - priority: P1
    timeout_minutes: 60
    next_level: "team-leads"

Escalation rules must be applied consistently.

Communication Templates

Use templates to standardize how incident updates are communicated.

# Example communication template
templates:
  status_update:
    subject: "[INCIDENT] {{title}} - Update"
    body: |
      Status: {{status}}
      Time: {{timestamp}}
      Details: {{message}}

Templates must include variables for dynamic content.

Sharp Edges and Gotchas

Several subtle issues can cause failures in incident management workflows.

Authentication Errors

The system may return a 200 status code even when authentication fails.

# This request might succeed despite invalid token
curl -X POST https://internal.ops/api/incidents \
  -H "Authorization: Bearer invalid-token" \
  -H "Content-Type: application/json" \
  -d '{"title": "Test incident"}'

Always validate the response body for successful creation.

Field Confusion

Fields like status and priority are often confused in API calls.

# Confusing priority with status
curl -X POST https://internal.ops/api/incidents \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"title": "Test", "priority": "investigating"}'

Priority should be a number (P0-P3), not a string like "investigating".

Missing Required Fields

API endpoints may silently ignore missing fields instead of returning errors.

# Title is required but ignored in some cases
curl -X POST https://internal.ops/api/incidents \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"priority": "P1"}'

Always check that required fields are included in requests.

Response Parsing Errors

Parsing responses incorrectly can lead to missed updates or duplicate actions.

# Incorrectly parsing incident ID from response
response=$(curl -s https://internal.ops/api/incidents)
incident_id=$(echo "$response" | jq -r ".id")

Always validate that the parsed value is present and correct.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.