How to Build a Disaster Recovery Plan

Disaster recovery planning ensures systems survive catastrophic failures—data center outages, cloud region failures, network partitions, or complete platform crashes—by defining how to restore services within predictable time and data loss windows. This guide details the concrete steps to build a DR plan, from initial assessment to execution and validation, using real tools and workflows that span multiple cloud providers and on-premises infrastructure.

Assess Recovery Objectives

Define the recovery time objective (RTO) and recovery point objective (RPO) for each service. RTO is the maximum acceptable downtime; RPO is the maximum acceptable data loss. These drive all decisions: backup frequency, replication strategy, and failover automation.

For a database service using PostgreSQL, RPO is determined by the frequency of WAL (Write-Ahead Log) archiving. To achieve an RPO of 5 minutes, configure:


wal_level = replica
archive_mode = on
archive_command = 'cp %p /mnt/archives/%f'
max_wal_senders = 8
wal_keep_size = 2GB

When archive_command fails, WAL files are lost; the system must be manually restored to the last available archive. This failure is silent: pg_wal is filled, but no new archives appear. Monitoring must check pg_archivecleanup output and archive_status in pg_wal.

RTO is achieved via replication and failover. For a two-node cluster, use pg_ctl promote to elevate a standby. The command returns immediately but does not verify the new master is accepting connections:

pg_ctl promote -D /var/lib/postgresql/data
# → No confirmation that connections are active

A missing step: verify the new master is ready using pg_isready:

pg_isready -h localhost -p 5432 -t 30 -c 5

If pg_isready returns non-zero, the failover is incomplete. This error is common in production: the cluster appears up, but clients time out due to unready nodes.

Map Critical Systems and Dependencies

Document the end-to-end flow of each critical service. For a payment processing system, a transaction involves:

API gateway receives request
Order service writes to PostgreSQL
Payment service calls Stripe API
Inventory service updates stock
Message queue (Kafka) logs event

Use a standard tooling stack: kubectl for Kubernetes, docker-compose for local deployments, curl for API testing, and psql for database queries.

To validate end-to-end flow, write a test script using curl and jq:

#!/bin/bash
# test-payment.sh

PAYMENT_ID=$(curl -s -X POST http://api.example.com/payments \
  -H 'Content-Type: application/json' \
  -d '{"amount": 100, "currency": "USD"}' \
  | jq -r '.id')

# Wait for Kafka topic to receive event
kafka-console-consumer.sh --bootstrap-server kafka:9092 \
  --topic payments \
  --partition 0 \
  --offset latest \
  --max-messages 1 \
  --timeout-ms 10000 \
  | grep -q "$PAYMENT_ID"

if [ $? -ne 0 ]; then
  echo "Payment event not in Kafka after 10s"
  exit 1
fi

# Verify database state
psql -c "SELECT status FROM payments WHERE id = '$PAYMENT_ID'" \
  -t -A -q

This script captures a common failure: a missing --timeout-ms in Kafka consumer. Without it, the consumer exits immediately, missing events. The script also shows that psql -t -A -q is essential for consistent output: -t removes headers, -A uses aligned output, -q suppresses prompts.

Design Data Replication and Backup Strategy

For persistent data, implement asynchronous replication with periodic backups. Use rsync for file-level backup and pg_dump for database dumps.

To back up a PostgreSQL database every 15 minutes:

#!/bin/bash
# backup-postgres.sh

DATE=$(date +'%Y-%m-%d-%H-%M')
BACKUP_DIR="/backups/postgres/$DATE"
mkdir -p "$BACKUP_DIR"

pg_dump -U postgres -F c -v -Z 9 -f "$BACKUP_DIR/data.dump" payments_db
rsync -avz --delete "$BACKUP_DIR/" s3://backups/postgres/

The pg_dump -F c creates a custom-format dump, which is faster to restore than plain SQL. The -Z 9 flag enables maximum compression. However, rsync does not ensure atomicity: if the backup is interrupted, the data.dump file is incomplete.

To fix this, use tar with --checkpoint and --checkpoint-action:

tar -czf - -C /backups/postgres \
  --checkpoint=100 \
  --checkpoint-action='exec echo "Checkpoint: $TAR_CHECKPOINT"'

This provides real-time progress feedback and ensures the backup is consistent. The --checkpoint-action is often missed: without it, administrators assume a backup is complete when the tar process ends, but the file is still being flushed.

Use borg or restic for incremental, deduplicated backups. Both tools use a repository structure and support backup and prune commands.

restic backup /data --include=/data/payments \
  --exclude=/data/logs --exclude=/data/tmp \
  --cache-dir ~/.restic-cache

A common pitfall: restic does not automatically prune old snapshots. The prune command must be scheduled:

restic prune --keep-daily 7 --keep-weekly 4 --keep-monthly 12

If prune is not run, backup storage grows indefinitely. Also, prune must be run after backup to avoid full retention cycles.

Define Failover and Recovery Procedures

Failover moves services from primary to secondary systems. The process must be repeatable and documented.

Use a script to promote a PostgreSQL standby to master:

#!/bin/bash
# promote-postgres.sh

set -euo pipefail

export PGHOST=localhost PGUSER=postgres

# Step 1: Stop streaming replication
psql -c "SELECT pg_wake_replication();"

# Step 2: Promote standby
pg_ctl promote -D /var/lib/postgresql/data

# Step 3: Wait for readiness
until pg_isready -t 30; do
  echo "Waiting for PostgreSQL to become ready..."
  sleep 5
done

# Step 4: Update DNS and configuration
echo "192.168.1.100 payments.example.com" | tee -a /etc/hosts
echo "primary-db.example.com" > /etc/postgres-hosts

This script uses set -euo pipefail to catch errors: -e exits on any error, -u enforces variable existence, and -o pipefail ensures pipeline failures are propagated.

A subtle failure: pg_wake_replication() does not immediately stop WAL streaming. The standby continues sending WALs even after promotion. To fix this, use pg_wal_lsn_diff() to verify the WAL position before and after promotion:

SELECT pg_wal_lsn_diff(pg_current_wal_lsn(), pg_last_wal_receive_lsn()) AS lag;

If lag exceeds 100 MB, the standby is behind, and data loss may occur.

For cloud-based failover, use aws ec2 create-tags to mark instances as active:

aws ec2 create-tags \
  --resources i-12345678 i-87654321 \
  --tags Key=Role,Value=Primary-DB Key=Status,Value=Active

The --resources flag accepts a space-separated list. A common mistake: passing a comma-separated list without --resources, causing only the first instance to be tagged.

Automate Recovery with Runbooks

Create runbooks using standard tools: bash, yaml, json, curl, and kubectl.

A runbook for restoring a failed region might include:

# runbooks/restore-region.yml
---
name: Restore Region
description: "Bring up services in secondary region after primary failure"
steps:
  - name: Start database cluster
    command: |
      kubectl apply -f manifests/db.yaml
      pg_ctl start -D /var/lib/postgresql/data
      pg_isready -t 60
    expected:
      - "Database is ready and accepting connections"
      - "pg_isready returns 0"

  - name: Promote standby to primary
    command: |
      pg_ctl promote -D /var/lib/postgresql/data
      pg_isready -t 30
    expected:
      - "Promotion successful"
      - "Primary accepts connections"

  - name: Update load balancer
    command: |
      curl -X PUT http://lb.example.com/api/region \
        -H 'Content-Type: application/json' \
        -d '{"primary": "us-east-1", "secondary": "us-west-2"}'
    expected:
      - "Load balancer returns 200"
      - "Traffic is routed to us-west-2"

  - name: Verify end-to-end flow
    command: |
      ./test-payment.sh
    expected:
      - "Payment is created"
      - "Event appears in Kafka"
      - "Database state reflects payment"

This runbook uses kubectl apply for declarative deployment. The manifests/db.yaml file must exist and define a StatefulSet or Deployment for PostgreSQL.

A failure mode: kubectl apply does not report which resources were created, updated, or unchanged. To capture this, use --output yaml and --record:

kubectl apply -f manifests/db.yaml --record --output yaml

The --record flag updates the kubernetes.io/change-cause annotation, which is crucial for audit and troubleshooting.

Validate and Test the Plan

Test the DR plan quarterly using a simulation: simulate a region failure and measure RTO and RPO.

Use kubeadm to create a test cluster:

kubeadm init --pod-network-cidr=10.244.0.0/16 --kubernetes-version=v1.25.0

Deploy a test workload:

kubectl apply -f test-app.yaml

Then, simulate a failure:

# Stop all nodes in us-east-1
kubectl drain node us-east-1-worker-1 --force --delete-local-data
kubectl drain node us-east-1-worker-2 --force --delete-local-data

# Wait for recovery
sleep 120

Use kubectl get pods -w to watch for pod restarts. A common error: kubectl get pods does not show the full reason for pod failure. Use kubectl describe pod <pod-name> to uncover events like ImagePullBackOff, CrashLoopBackOff, or PodScheduled.

Validate the plan with a checklist:

[ ] All services are running
[ ] Database is promoting correctly
[ ] Load balancer routes traffic
[ ] Test script passes
[ ] Logs show no errors in journalctl or kubernetes events

Use journalctl to verify service startup:

journalctl -u postgresql.service --since "1 hour ago" --unit postgresql

The --since flag accepts relative time. A common mistake: using --since "1h" instead of --since "1 hour ago", which results in a different time window.

Maintain and Evolve the Plan

A DR plan is not static. Update it with each release and after every failure.

Use git to track changes:

git add .
git commit -m "Update DR plan for v2.3.0 release"
git push origin main

Create a docs/dr-plan.md file that includes:

Current RTO and RPO values
Runbooks
Diagrams (exported from draw.io or mermaid)
Checklists for testing

Use mermaid for diagrams:

graph TD
  A[Primary Region] -->|Replicates| B[Secondary Region]
  B -->|Promotes| C[New Primary]
  C -->|Serves Traffic| D[Users]
  D -->|Writes| A

This diagram is rendered with mermaid-cli or included in README.md via markdown syntax.

The final step: integrate DR plan into onboarding. New engineers run the test-payment.sh script, verify pg_isready, and observe journalctl output. They become familiar with the system before a failure occurs.

When a disaster hits, the team follows the runbook, validates each step, and updates the plan. The process becomes self-improving.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.