Communicate effectively during production incidents. Covers status page management, stakeholder updates, customer communication templates, internal escalation, timeline documentation, and the patterns that maintain trust when things go wrong.
This playbook defines the specific actions, messages, and communication patterns used during incident response to ensure clarity, consistency, and operational transparency. It applies when incidents are declared, when status updates must be issued, or when coordination across teams requires structured messaging.
When an incident is declared, the first communication must include:
Example message format:
[INCIDENT-001] [P1] Database cluster unavailable
Started: 2023-10-27T14:30:00Z
Commander: [email protected]
Details: All database connections failing with "connection refused"The incident identifier must be generated using a stable, predictable format. A typical UUIDv4 string suffices: f47ac10b-58cc-4372-a567-0e02b2c3d479
The incident commander must maintain an active communication channel and update the incident log with:
Example log entry:
14:35:00Z - [john.doe] Promoted to incident commander
14:36:00Z - [ops-team] Created Slack channel #incident-001
14:37:00Z - [database-team] Verified cluster status via kubectl get pods -n databaseStatus updates must follow a consistent format. Each update includes:
Example:
[INCIDENT-001] [INVESTIGATING] 2023-10-27T14:45:00Z
Summary: Database cluster pods are in CrashLoopBackOff state. Initial investigation points to disk I/O errors.
Next steps: Investigating storage configuration and resource limits.Slack channels must be created for each incident using the naming convention #incident-<UUID>.
The channel should include:
Example channel configuration:
# Create Slack channel
curl -X POST -H 'Authorization: Bearer xoxb-...' \
-d 'name=incident-f47ac10b-58cc-4372-a567-0e02b2c3d479' \
https://slack.com/api/conversations.createEmail updates must be sent to:
Each email must include:
Example email body:
Subject: [INCIDENT-001] Status Update - Database Cluster Unavailable
Status: Investigating
Timestamp: 2023-10-27T14:45:00Z
The database cluster remains unavailable. We have identified a potential disk I/O issue that requires further investigation.
Next steps:
- Reviewing storage configuration
- Checking node resource constraints
For questions, contact [email protected].When multiple teams are involved, communication must be coordinated through the incident channel. Each team's lead should:
Example coordination message:
[database-team] Need help from network team to check firewall rules for database port 5432Escalations must follow a defined hierarchy. The primary escalation path is:
Each escalation must include:
Example escalation message:
[INCIDENT-001] [ESCALATION] P1 Database cluster failure
Escalation to: [email protected]
Reason: No progress in last 2 hours despite multiple teams engaged
Action requested: Review current incident plan and provide additional resourcesWhen an incident is resolved, the final update must include:
Example final update:
[INCIDENT-001] [RESOLVED] 2023-10-27T16:30:00Z
Root Cause: Storage I/O errors due to incorrect disk provisioning
Resolution: Added additional storage volume and updated configuration
Lessons Learned: Need to validate storage configuration before deploymentPost-incident documentation must include:
The incident command tool must support:
Example interface requirements:
POST /incidents/{uuid}/status
{
"status": "investigating",
"summary": "Database cluster pods failing",
"timestamp": "2023-10-27T14:45:00Z"
}Monitoring systems must provide:
Example monitoring update:
{
"incident_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
"status": "investigating",
"message": "Database cluster down - investigating disk I/O errors"
}Communication can silently fail when:
Example of silent failure:
# Channel created but no message sent to participants
curl -X POST -H 'Authorization: Bearer xoxb-...' \
-d 'channel=incident-f47ac10b-58cc-4372-a567-0e02b2c3d479' \
-d 'text=Incident declared' \
https://slack.com/api/chat.postMessageCommon configuration errors include:
Example incorrect status update:
# Should be "investigating", not "investigation"
{
"status": "investigation",
"summary": "Database cluster pods failing",
"timestamp": "2023-10-27T14:45:00Z"
}The incident channel can fail when:
Example verification command:
# Verify channel exists and has correct configuration
curl -X GET -H 'Authorization: Bearer xoxb-...' \
https://slack.com/api/conversations.info?channel=incident-f47ac10b-58cc-4372-a567-0e02b2c3d479[INCIDENT-{UUID}] [{SEVERITY}] {BRIEF_DESCRIPTION}
Started: {TIMESTAMP}
Commander: {CONTACT_EMAIL}
Details: {DETAILED_DESCRIPTION}[INCIDENT-{UUID}] [{STATUS}] {TIMESTAMP}
Summary: {SUMMARY}
Next steps: {NEXT_STEPS}[INCIDENT-{UUID}] [RESOLVED] {TIMESTAMP}
Root Cause: {ROOT_CAUSE}
Resolution: {RESOLUTION}
Lessons Learned: {LESSONS_LEARNED}This playbook ensures consistent communication during incidents. The specific formats, tools, and processes described here are designed to be immediately actionable and should be followed without modification unless explicitly authorized by the incident commander or escalation hierarchy.
This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy