Design on-call rotations that protect engineer health while maintaining service reliability. Covers rotation design, escalation policies, compensation models, burnout prevention, alert quality, and the organizational practices that make on-call sustainable.
This page describes practices for maintaining reliable on-call coverage without burnout or degradation in response quality. It applies when engineering teams must sustain incident response over extended periods, balancing urgency with long-term team health.
On-call rotation schedules should explicitly define who is responsible for what during each period. A typical rotation cycle spans 7–14 days, with clear handoffs between team members. Rotations are usually managed through shared calendar entries and ticketing systems. The schedule must be visible to all relevant stakeholders and updated regularly.
Example of a simple rotation configuration in a system that uses JSON for scheduling:
{
"rotation": {
"name": "backend-alerts",
"members": ["alice", "bob", "charlie"],
"duration": "7d",
"shift_start": "09:00",
"shift_end": "18:00"
}
}The system must track which member is on-call at any given time, typically using a timestamp-based lookup. When the rotation changes, the system should automatically update all alerting rules and escalation paths.
Each team must define acceptable alert thresholds and ensure alerts are actionable. Too many alerts lead to alert fatigue, reducing response quality. Teams should implement alert filtering and silencing mechanisms that prevent noise from overwhelming critical signals.
Alerts must include sufficient context to understand urgency. For example, an alert like Database connection pool exhausted is more useful than Service down. The alert should contain:
Use a consistent format for alert descriptions across tools. A minimal alert definition might look like:
alert:
name: "High CPU Usage"
severity: "warning"
description: "Instance i-1234567890abcdef0 has exceeded 80% CPU utilization for 5 minutes."
service: "webapp"
resource: "i-1234567890abcdef0"Some systems allow alert grouping or aggregation. Misconfigured grouping can mask issues or create false confidence.
Escalation rules define how alerts are handled if no action occurs within a defined window. Each rule must specify:
A sample rule might be:
escalation:
- condition: "alert.age > 10m"
action: "notify-slack"
recipients: ["#backend-team"]
- condition: "alert.age > 30m"
action: "call-oncall"
recipients: ["oncall-backend"]Escalations must be idempotent. Repeated escalation triggers should not create duplicate work or notifications.
When a team member leaves their on-call shift, they must complete an explicit handoff process. This includes:
Handoffs are often documented in a shared log file or ticket system. The process should not rely on informal communication like Slack messages.
Example handoff entry:
On-call handoff: alice (2024-05-15 09:00 - 18:00)
Active incidents:
- #12345: Database replication lag > 5m
Status: Investigating, waiting for logs from replica
- #12346: API latency spike
Status: Resolved (restarted service)During an incident, responders should document their actions in a shared workspace or ticketing system. This ensures knowledge sharing and enables future troubleshooting.
Responses should be categorized by type:
Each step must be timestamped and attributed to the responder.
Example incident log structure:
incident:
id: "INC-12345"
title: "Database replication lag"
status: "investigating"
started_at: "2024-05-15T10:15:00Z"
owner: "alice"
steps:
- timestamp: "2024-05-15T10:15:00Z"
action: "Check replication status"
result: "Replica lag is 8 minutes"
- timestamp: "2024-05-15T10:30:00Z"
action: "Restart replica service"
result: "Lag reduced to 2 minutes"On-call systems should minimize false positives that waste time. Alerts should be based on stable metrics rather than transient anomalies. For example, a metric like 5xx errors per minute is better than 5xx errors in last 30 seconds.
Tools that generate alerts from logs or traces must include filtering logic to avoid spurious notifications. A common mistake is alerting on any error log entry without considering whether the error is recoverable.
When someone cannot fulfill their on-call duty, coverage must be explicitly reassigned. This process includes:
Coverage gaps are often silent failures. The system must detect when a scheduled person is not available and trigger an alert or escalation.
On-call schedules must include adequate time off between shifts. A standard recommendation is 2–3 days off for every week of on-call duty. Teams should enforce this through calendar blocking and approval workflows.
Some systems track how much time each person spends in on-call mode. This data can help identify overworked individuals or inefficient scheduling patterns.
Sustainable on-call requires avoiding situations where teams are always "on." This means:
Teams should regularly review their on-call practices and make adjustments based on feedback.
Tracking how well the system supports on-call work is essential. Key metrics include:
These are usually collected through integration with ticketing or monitoring systems.
Example query to extract average time to resolution for recent incidents:
SELECT AVG(resolved_at - created_at) as avg_resolution_time
FROM incidents
WHERE created_at > '2024-05-01'
AND status = 'resolved';When escalation rules fail, they often do so silently. Common failure modes include:
Teams must have a process for detecting and correcting these failures.
To prevent alert fatigue, teams should regularly review their alerting rules and remove or modify ineffective ones. This includes:
Some systems support alert suppression for known issues, which can reduce noise during planned maintenance.
Teams should use consistent tools for managing on-call work. These include:
All tools must integrate properly to avoid data silos or missed information.
On-call readiness involves ensuring that all necessary resources are available. This includes:
Regular readiness checks should verify that these items are current and accessible.
When off-call, team members should not receive urgent notifications unless absolutely necessary. Systems should distinguish between different types of alerts:
Critical alerts should be routed to the on-call person via multiple channels (email, SMS, phone call). Warning alerts can be batched or sent during business hours.
Schedules must account for:
Systems should support flexible scheduling and provide visibility into future assignments.
Sustainable incident on-call requires intentional design of the entire process, from scheduling to response. Teams must ensure that alerts are meaningful, escalations work correctly, and handoffs are clear. Regular review and adjustment of these practices ensures that teams can respond effectively without burning out.
This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy