Incident Postmortem Templates

Production engineering guide for incident postmortem templates covering patterns, implementation strategies, and operational best practices.

This page describes reusable templates for documenting incident postmortems. These templates are designed to guide structured analysis of incidents, from initial detection through resolution and follow-up actions. They apply when teams need to capture the sequence of events, root causes, and remediation steps in a consistent format that supports learning and process improvement.

Structure and Content Organization

Core Template Sections

The following sections form the foundation of an incident postmortem:

  1. Executive Summary
  2. Timeline
  3. Impact
  4. Root Cause Analysis
  5. Resolution Steps
  6. Follow-up Actions

Each section has a defined purpose and expected content. The format ensures that critical details are not omitted during the postmortem process.

Executive Summary

The executive summary should be a concise one-paragraph overview of the incident. It must include:

Example structure:

On 2024-03-15 at 14:30 UTC, a service degradation occurred in the payment processing pipeline due to a misconfigured database connection pool. The incident lasted approximately 45 minutes and affected 87% of transaction requests. The root cause was identified as an incorrect value in a configuration parameter that reduced the maximum number of concurrent connections from 100 to 10. Key actions included restoring the correct configuration and implementing automated validation checks.

Timeline

The timeline section records all significant events during the incident, including:

Each entry should include:

Example format:

14:30:12 UTC | Alerting system | Database connection pool exceeded threshold
14:31:05 UTC | SRE team | Initial investigation started
14:35:22 UTC | DevOps engineer | Checked configuration files
14:40:11 UTC | SRE team | Configuration restored
14:45:00 UTC | Monitoring system | Alert cleared

Impact

The impact section documents what was affected and how severely. Include:

Example:

Affected components:
- Payment gateway service
- User authentication service
- Database cluster

Metrics showing degradation:
- Response time increased from 100ms to 3500ms
- Error rate rose to 87%
- Database connection pool utilization peaked at 95%

User impact:
- 87% of transaction requests failed
- Users unable to log in or complete purchases

Root Cause Analysis

The root cause analysis section identifies the underlying issue. It must include:

Example:

Direct cause:
Configuration parameter `max_connections` was changed from 100 to 10 in the application settings.

Contributing factors:
- No automated validation for critical configuration parameters
- Change was deployed without review by senior SREs
- Monitoring did not alert on connection pool exhaustion

Evidence:
- Configuration file changes log showed modification at 14:15 UTC
- Database logs indicated connection timeouts starting at 14:28 UTC
- Application logs reported "connection pool exhausted" errors

Resolution Steps

This section documents the actions taken to resolve the incident. List each step in chronological order:

Example:

1. Identified configuration parameter `max_connections` as incorrect
2. Restored original value of 100 in configuration management system
3. Restarted affected services to pick up new configuration
4. Verified database connection pool behavior
5. Confirmed service stability

Follow-up Actions

The follow-up actions section lists items that must be completed after the incident. These are typically:

Example:

1. Implement automated validation for all critical configuration parameters
2. Add check for `max_connections` in deployment pipeline
3. Update SRE runbook to include configuration review step
4. Create new alert for database connection pool utilization above 80%

Specialized Templates for Common Incident Types

Configuration Error Template

For incidents caused by incorrect settings or configurations:

  1. Configuration Details
  1. Impact Analysis
  1. Recovery Steps

Infrastructure Failure Template

For incidents involving hardware, network, or platform outages:

  1. System State
  1. Failure Details
  1. Recovery Process

Performance Degradation Template

For incidents where performance dropped without full failure:

  1. Performance Metrics
  1. Analysis Steps
  1. Optimization Actions

Template Usage and Validation

Review Process

Postmortems should be reviewed by:

Review checklist:

Version Control

All postmortems should be stored in version control with:

Example git workflow:

git checkout -b postmortem/incident-2024-001
# Make changes to markdown file
git add incident-2024-001.md
git commit -m "Add postmortem for INC-2024-001"
git push origin postmortem/incident-2024-001

Common Pitfalls and Sharp Edges

Silent Failures

The most common silent failure in incident reporting is incomplete timeline data. The system may record alerts but not the human actions that followed. This leads to incorrect root cause assumptions.

Example of what fails:

Confusion Between Sections

Teams often conflate "impact" and "resolution steps." Impact should describe what was broken, while resolution steps describe how it was fixed.

Validation Issues

Postmortems are often reviewed without checking:

Template Customization Guidelines

Templates can be customized but must maintain core structure. Customization options include:

  1. Adding sections (e.g., "Lessons Learned")
  2. Modifying field names (e.g., "Affected Services" instead of "Impact")
  3. Adjusting required fields (e.g., adding "Security Impact" for security incidents)

Customizations must not:

Template Repository Structure

Postmortems should be organized in a consistent directory structure:

/incidents/
  /2024/
    /03/
      incident-2024-001.md
      incident-2024-002.md
    /04/
      incident-2024-003.md

Automation Considerations

Templates support automation through:

Example markdown linting command:

markdownlint -c .markdownlint.yaml incident-2024-001.md

Maintenance and Updates

Templates should be updated based on:

Update frequency should be quarterly or as needed based on incident trends.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.