On-Call Engineering | The Garnet Wiki

On-call engineering is the discipline of ensuring systems remain stable, observable, and recoverable when engineers are not at their desks. It matters when a service fails at 2 a.m., when a database replica falls out of sync during a deployment, or when a third-party API begins returning 500s across a region. It is the practice of structuring responsibilities, tooling, and workflows so that anyone on-call—regardless of depth of domain knowledge—can triage, diagnose, and resolve incidents efficiently.

Incident Response Workflow

When an alert fires, the on-call engineer must act quickly. The goal is to reduce mean time to acknowledge (MTTA) and mean time to resolve (MTTR) through a well-structured, repeatable process.

Acknowledge the alert within 15 minutes of its first occurrence.

Use a command-line interface or dashboard to mark an alert as "acknowledged" with a timestamp and initial notes.
Example:

     curl -X POST \
       -H "Authorization: Bearer $ONCALL_TOKEN" \
       -H "Content-Type: application/json" \
       -d '{"incident_id": "INC-2023-087", "acknowledged_by": "sarah", "notes": "Initial triage: system is unresponsive."}' \
       https://alerting.internal/api/v1/incidents/acknowledge

The alert should be marked as “in progress” and assigned to the on-call engineer.

Diagnose the root cause using layered visibility:

Start with high-level metrics: latency, error rate, throughput.
Drill down into logs and traces:

     # Retrieve logs from the past 10 minutes using structured JSON logs
     kubectl logs deployment/api-gateway -n prod --since=10m --tail=100 | jq '.timestamp, .level, .message, .error, .trace_id'

Use distributed tracing to correlate requests across services:
Extract trace_id from a failed request in the logs.
Query a central tracing service (e.g., Jaeger, OpenTelemetry) using the trace_id to visualize the request flow.
Look for spans with high duration, errors, or missing dependencies.

Apply a fix based on the diagnosis:

Common actions:
Restart a service:

       systemctl restart api-gateway

Roll back a deployment:

       kubectl rollout undo deployment/api-gateway -n prod --to-revision=12

Scale a service:

       kubectl scale deployment/api-gateway -n prod --replicas=8

Update a configuration value in a live system:

       # /etc/config/api-gateway.yaml
       connection_pool:
         max_connections: 150
         idle_timeout: 300s
         max_lifetime: 1h

Apply with systemctl reload api-gateway.

Verify the fix and update the incident:

Confirm metrics have stabilized.
Validate with a manual request:

     curl -v https://api.example.com/v1/users/123

Update the incident with:
Summary of actions taken
Evidence (e.g., log snippets, trace IDs)
Next steps (e.g., “monitor for 1 hour”, “investigate slow DB queries”)
Close the incident when the system is stable and the fix is confirmed.

On-Call Rotation and Scheduling

An effective on-call rotation ensures consistent coverage and reduces burnout. The system must define who is on-call, when, and how they are notified.

Define a rotation using a shared calendar or a simple JSON file stored in version control:

  {
    "name": "API Team",
    "frequency": "weekly",
    "start_time": "2023-10-01T00:00:00Z",
    "rotations": [
      {
        "start": "2023-10-01T00:00:00Z",
        "end": "2023-10-08T00:00:00Z",
        "oncall": "alice",
        "backup": "bob"
      },
      {
        "start": "2023-10-08T00:00:00Z",
        "end": "2023-10-15T00:00:00Z",
        "oncall": "bob",
        "backup": "charlie"
      }
    ]
  }

Rotate on a fixed schedule: weekly, biweekly, or monthly.
Notify on-call engineers at the start of their shift via email, Slack, or a push notification.
Automatically update the on-call roster in a service:

  # Update on-call roster using a script triggered by CI/CD
  curl -X POST \
    -H "Content-Type: application/json" \
    -d '{"team": "api", "current_oncall": "alice", "start": "2023-10-01T00:00:00Z"}' \
    https://oncall.internal/api/v1/rotation

Alerting Configuration

Alerts must be actionable, timely, and avoid noise.

Define thresholds using a standard set of metrics:
error_rate > 0.05 over 5 minutes
latency_p95 > 1s over 10 minutes
service_down (e.g., HTTP 5xx for 2 consecutive minutes)
Alerting rules should include:
Alert name, description, and severity (critical, warning, info)
Evaluation window and aggregation
Silence and deduplication rules
Example alert configuration in YAML:

  alert: "High Error Rate in API Gateway"
  description: "API Gateway returns >5% errors over 5 minutes."
  severity: "critical"
  query: |
    rate(errors{service="api-gateway"}[5m]) / rate(requests{service="api-gateway"}[5m]) > 0.05
  for: "5m"
  labels:
    team: api
    service: api-gateway
  annotations:
    summary: "High error rate on {{ $labels.service }}"
    description: |
      API Gateway error rate has exceeded 5% over the last 5 minutes.
      Current value: {{ $value }}
      More details: https://monitoring.internal/incidents/{{ $labels.incident_id }}

Alerting is not just a configuration; it is a process:
Alerts must be tested before deployment (e.g., simulate 500s for 10 minutes).
Alerting tools must support:
Muting (silence for known issues)
Acknowledging (mark as in progress)
Escalating (notify backup if unacknowledged after 30 minutes)

Incident Documentation and Postmortems

Every incident is a learning opportunity. The documentation must capture not only what happened but why.

After resolving an incident, create a postmortem document using a standard template.
Include:
Incident timeline (with timestamps)
Summary of root cause
Actions taken
Lessons learned
Follow-up items
Store in a shared repository:

    # Postmortem: High Latency on Checkout Service (2023-10-04)

    ## Timeline
    - 02:15:42: Error rate spikes to 12%
    - 02:18:10: First alert fired
    - 02:20:03: On-call engineer acknowledges incident
    - 02:23:45: Logs show slow DB queries in `checkout` service
    - 02:31:00: DB index created; latency drops
    - 02:45:00: Incident resolved

    ## Root Cause
    - Missing index on `orders` table for `user_id` and `status`
    - Query: `SELECT * FROM orders WHERE user_id = ? AND status = 'pending'` took 800ms
    - Index created during previous deployment but not applied to production

    ## Actions Taken
    - Added index on `orders(user_id, status)`
    - Verified query performance with `EXPLAIN ANALYZE`
    - Updated deployment script to include index creation

    ## Lessons Learned
    - All database migrations should include a migration status check.
    - Add alert for "missing index" on high-load tables.
    - Include DB schema changes in deployment checklist.

Postmortems should be reviewed during team retrospectives.

On-Call Engineer Responsibilities

The on-call engineer is responsible for more than just responding to alerts.

Preparation:
Review incident history before shift.
Check system health dashboard (latency, error rate, queue depth, resource utilization).
Know the deployment process:
How to deploy a new version
How to roll back
How to promote from staging to production
Know the incident response playbook:
Common issues and their known solutions
Contact list: who to page for database, infrastructure, frontend issues

During Shift:
Monitor systems every 30 minutes:
Check alerting dashboard
Review logs from the past hour
Validate critical services
Respond to incidents with clarity and consistency:
Use a common incident template
Communicate via Slack or email:

      [INC-2023-087] High Error Rate in API Gateway

      Acknowledged by sarah (on-call)
      Status: In progress
      Summary: Error rate > 5% over 5 minutes.

      Next steps:
      - Investigate database queries
      - Check load balancer health
      - Review recent deployments

Escalate when needed:
Notify backup engineer via Slack or PagerDuty-style alert.
Page a team lead or domain expert if issue persists beyond 30 minutes.

After Shift:
Complete a shift report:
List of incidents handled
Observations (e.g., “DB CPU spiked at 3 a.m.”)
Suggestions (e.g., “add alert for slow cache hits”)
Update documentation:
Add new incident to the knowledge base
Update runbooks with new patterns or common fixes
Hand over to the next on-call engineer:
Share a concise handover document in the team’s shared drive
Highlight ongoing issues, known problems, and upcoming changes

Common Pitfalls and Silent Failures

Alerts that fire but are never acknowledged:
Engineers receive alert notifications but never check the dashboard.
Solution: set up automated reminders (e.g., daily email digest of unacknowledged alerts).

Confusing “critical” and “warning” alerts:
A warning alert is treated as critical, and a critical alert is ignored.
Solution: define clear severity criteria:
Warning: 3–5% error rate
Critical: >5% error rate or 10% latency increase

Incorrect or missing configuration:
A service starts but fails to read config.yaml because it’s in the wrong directory.
Use absolute paths in systemd units:

    [Unit]
    Description=API Gateway

    [Service]
    ExecStart=/usr/bin/api-gateway --config /etc/config/api-gateway.yaml
    WorkingDirectory=/opt/api-gateway
    Restart=always

Failure to verify fixes:
Engineers apply a fix but do not validate it.
Solution: require a “verify” step in the incident workflow:
Run a test query
Check metrics for 10 minutes
Confirm logs show success

Poor handover between on-call engineers:
The outgoing engineer forgets to mention a long-running issue.
Solution: use a handover checklist:
[ ] List of active incidents
[ ] Known issues
[ ] Recent deployments
[ ] Upcoming maintenance
[ ] Contacts for key teams

Tooling and Data Flow

On-call engineering relies on a few core tools and data flows.

Alerting:
Metrics collected via Prometheus (or similar)
Alerting rules evaluated by Alertmanager
Notifications sent via HTTP, SMTP, or WebSocket
Alert data stored in a database:

    CREATE TABLE alerts (
      id SERIAL PRIMARY KEY,
      incident_id VARCHAR(64) UNIQUE,
      title TEXT,
      description TEXT,
      severity TEXT,
      status TEXT, -- pending, acknowledged, in_progress, resolved
      created_at TIMESTAMPTZ,
      updated_at TIMESTAMPTZ,
      acknowledged_by TEXT,
      resolved_by TEXT
    );

Incident tracking:
Incidents stored in a shared database with a standard schema:

    CREATE TABLE incidents (
      id SERIAL PRIMARY KEY,
      incident_id VARCHAR(64) UNIQUE,
      title TEXT,
      description TEXT,
      status TEXT,
      created_at TIMESTAMPTZ,
      acknowledged_at TIMESTAMPTZ,
      resolved_at TIMESTAMPTZ,
      root_cause TEXT,
      solution TEXT,
      follow_ups JSONB,
      logs JSONB,
      traces JSONB
    );

On-call rotation:
Rotations stored as JSON in a version-controlled repository
Rotations updated by a CI/CD pipeline:
On merge to main, generate next rotation and deploy to on-call service
Rotate on a fixed interval (e.g., every Monday at midnight)

On-call engineering is not a role—it is a process, a culture, and a system. It thrives when every engineer knows how to respond to an alert, every incident is documented, and the handover between shifts is seamless. It is not enough to have alerts; the system must be designed so that anyone can become an on-call engineer after a single shift.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.