Runbook Engineering

Runbooks are the operational backbone of production systems. They encode the collective knowledge of teams into executable, repeatable sequences of actions that respond to known failure modes. Runbook Engineering is the practice of designing, building, and maintaining these runbooks—ensuring they are reliable, discoverable, and actionable across teams and environments. This matters most during incident response, onboarding, and regular maintenance, when consistency and speed are critical, and when the difference between a minor outage and a cascading disaster lies in how well the team follows the runbook.

Designing a Runbook

A runbook begins with a clear problem statement. It answers: *What is this runbook for? What does it do? When should it be run?* The best runbooks start with a single sentence: "This runbook restores service after a database replication lag exceeds 10 minutes."

Each runbook is a sequence of steps, each with a title, a description, and executable commands. Steps may be manual, automated, or hybrid. The structure of a runbook is typically a list of numbered steps, each containing:

• A concise title
• A description of purpose and expected outcome
• A list of commands, scripts, or UI actions
• Optional preconditions and postconditions
• Failure handling and retry logic

Example:

1. Confirm database replication lag
   - Description: Verify that the primary database is lagging behind the replica.
   - Precondition: Database connection is available.
   - Commands:
     psql -c "SELECT now(), pg_current_wal_lsn(), pg_last_wal_receive_lsn(), 
             (pg_current_wal_lsn() - pg_last_wal_receive_lsn()) / 1024 AS lag_mb 
             FROM pg_stat_replication;"
   - Expected output: lag_mb > 10240
   - If lag_mb ≤ 10240, go to step 3.
   - If lag_mb > 10240, proceed to step 2.

Executing a Runbook

Execution begins with a trigger: a monitoring alert, a scheduled job, or a manual invocation. The runbook is executed either by a person (manual runbook) or by a system (automated runbook), with the same interface in both cases.

Manual Execution

In manual execution, a human follows the runbook step-by-step. The runbook is stored in a shared location—e.g., a Markdown file in a Git repository under runbooks/.

Each step is marked as complete upon verification. A common pattern is to use a checklist with checkboxes:

- [x] Step 1: Confirmed replication lag
- [ ] Step 2: Promote replica to primary
- [ ] Step 3: Update application configuration
- [ ] Step 4: Verify service health

The executor logs actions as they go:

2025-04-05T12:30:00Z: Promoting replica to primary...
2025-04-05T12:32:15Z: pg_ctl promote -D /var/lib/postgresql/data succeeded.
2025-04-05T12:33:00Z: Running health checks...
2025-04-05T12:34:20Z: All services healthy. Proceeding to next step.

Automated Execution

Automated runbooks are triggered by events and executed by a workflow engine. Common triggers:

• A Prometheus alert firing
• A CloudWatch alarm
• A cron job
• A webhook from a monitoring service

The execution engine runs the runbook as a sequence of tasks. Each task is a step from the runbook, executed in order, with error handling.

Example of a runbook executed via a YAML-based workflow engine (like Argo Workflows or Tekton):

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: db-replication-lag-restore
spec:
  entrypoint: db-replication-lag-restore
  templates:
    - name: db-replication-lag-restore
      steps:
        - - name: confirm-replication-lag
            template: check-lag
        - - name: promote-replica
            template: promote-replica
            when: "{{steps.confirm-replication-lag.outputs.result}} > 10240"
        - - name: update-config
            template: update-config
            when: "{{steps.promote-replica.succeeded}}"
        - - name: verify-health
            template: verify-health
  templates:
    - name: check-lag
      container:
        image: postgres:15
        command:
          - sh
          - -c
          - |
            psql -h db-primary -U app_user -d app_db -c "
              SELECT now(), pg_current_wal_lsn(), pg_last_wal_receive_lsn(),
                     (pg_current_wal_lsn() - pg_last_wal_receive_lsn()) / 1024 AS lag_mb
              FROM pg_stat_replication;"
        outputs:
          result: "{{steps.check-lag.outputs.result}}"
    - name: promote-replica
      container:
        image: postgres:15
        command:
          - sh
          - -c
          - |
            pg_ctl promote -D /var/lib/postgresql/data &&
            echo "Promotion successful" > /tmp/promotion-success
    - name: update-config
      container:
        image: alpine:latest
        command:
          - sh
          - -c
          - |
            sed -i 's/primary: .*/primary: replica-01/g' /etc/app/config.yaml &&
            systemctl reload app-service
    - name: verify-health
      container:
        image: curlimages/curl:latest
        command:
          - sh
          - -c
          - |
            until curl -f http://app-service:8080/health; do
              echo "Waiting for service..." && sleep 5
            done
            echo "Service is healthy."

Structuring Runbooks for Discoverability

Runbooks must be easy to find. They live in a shared directory, version-controlled in Git. The root directory is runbooks/, and each runbook is a Markdown file with a .md extension.

Each runbook has a standard front matter:

---
title: "Restore Database Replication Lag"
description: |
  When the primary database falls behind its replica by more than 10MB, promote the replica to primary.
  This runbook handles the entire process from detection to validation.
tags:
  - database
  - high-availability
  - incident-response
  - postgres
trigger:
  - prometheus: "db_replication_lag_mb > 10240"
  - cloudwatch: "RDS/ReplicationLag > 10240"
  - manual: true
  - scheduled: "daily"
author: "sre-team"
created: "2025-01-10"
updated: "2025-04-05"
---

Runbooks are discoverable via:

• A searchable index built from tags and description
• A runbooks/ landing page listing all runbooks with their tags and trigger sources
• A runbooks/README.md that explains the structure and usage

Maintaining Runbooks

Runbooks evolve. A runbook that was correct when first written often fails when applied in a new environment or under different load.

Versioning and Change Tracking

Each runbook is versioned in Git. Changes are tracked via pull requests. A change to a runbook includes:

• A description of the change
• A reference to the incident or issue that prompted the change
• A checklist of steps to test the updated runbook

Example PR description: > Fix replication lag restore: add pg_wal_lsn check before promotion > > - Before: only pg_last_wal_receive_lsn() was used. > - After: now also checks pg_current_wal_lsn() to detect if the lag is truly increasing. > - Fixes: https://github.com/org/app/issues/1234 > - Test: ran against staging with 15-minute lag simulation.

Reviewing Runbooks

Runbooks are reviewed during post-mortems. When an incident occurs, the team evaluates whether the runbook was followed correctly and whether it was sufficient.

Key questions during review:

• Did the runbook cover all necessary steps?
• Were the commands correct and executable in the environment?
• Was the expected output clearly defined?
• Did the runbook account for common failure modes?

A common failure mode: the runbook assumes psql is available but does not specify the connection string. The runbook fails when the database server is behind a firewall or when credentials are missing.

Standardizing Runbook Structure

To avoid drift, teams adopt a standard runbook template. The template includes:

• A title and description
• tags for categorization
• trigger sources
• preconditions and postconditions
• A steps section with numbered steps
• An appendix section for related information (e.g., common errors, known issues)

Example appendix:

### Known Issues

- **Error: `psql: could not connect to server`**
  - Cause: `PGHOST` environment variable not set.
  - Fix: Set `export PGHOST=db-primary` before running the command.

- **Error: `pg_ctl promote: command not found`**
  - Cause: `pg_ctl` not in `PATH`.
  - Fix: Add `/usr/pgsql-15/bin` to `PATH`.

- **Error: `curl: (7) Failed to connect to app-service port 8080: Connection refused`**
  - Cause: Application service not running.
  - Fix: Run `systemctl start app-service` before health check.

Handling Runbook Failures

Runbooks fail silently. A command succeeds but produces incorrect output. A step is marked complete even when the action did not take effect.

Common Failure Modes

• Step completion without effect: A step says "Promote replica to primary", but the promotion does not update the application configuration.
• Fix: Add a verification step after promotion: "Check that primary field in config matches new primary."

• Incorrect command syntax: A command uses psql -h db-primary -d app_db but the database server is named db-primary.internal, and db-primary is not resolved.
• Fix: Use psql -h db-primary.internal -d app_db or add host: db-primary.internal to the config.

• Missing environment variables: A script assumes DB_HOST, DB_USER, DB_PASS are set, but they are not.
• Fix: Add a set -a at the start of the script, or use export DB_HOST=db-primary before running.

• Incorrect timing: A runbook assumes a service restart takes 10 seconds, but it takes 30 seconds in production.
• Fix: Add a sleep 30 after systemctl restart app-service.

Idiomatic Runbook Patterns

• Idempotent steps: A step should be safe to run multiple times. For example, systemctl restart app-service is idempotent; it works whether the service is running or stopped.

• Retryable steps: Steps that may fail transiently (e.g., network, database) should include retry logic:

  for i in {1..5}; do
    if curl -f http://app-service:8080/health; then
      echo "Success"
      break
    fi
    echo "Attempt $i failed. Waiting 10 seconds..."
    sleep 10
  done

• Conditional branching: Use when or if logic to guide the flow:

  2. Check service health
     - If status code is 200, go to step 4.
     - If status code is 503, go to step 3.

• Parallel steps: When steps can run independently, execute them in parallel:

  - [ ] Step 1: Update config on all servers (parallel)
  - [ ] Step 2: Restart app services (parallel)
  - [ ] Step 3: Run database migration (parallel)

Integrating Runbooks into Operations

Runbooks are not isolated documents. They are integrated into the operational workflow.

• Incident response: Runbooks are the first resource in a runbook-driven incident. The on-call engineer opens the relevant runbook immediately after alerting.
• Onboarding: New engineers follow runbooks to set up their local environment, deploy a service, or handle a common issue.
• Changelog: A runbooks/changelog.md tracks changes across runbooks, with dates, authors, and summaries.

Runbooks are also used in change management: before a release, the team runs a "release checklist" runbook that verifies all prerequisites are met.

Conclusion

Runbook Engineering is the craft of turning operational knowledge into executable, maintainable, and discoverable procedures. It is not about writing documentation—it is about building systems that ensure consistency, reduce cognitive load, and make teams resilient. The runbook is the place where theory meets practice, and where the difference between a good and a great operations team is written in plain text and executed in real time.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.