Design disaster recovery strategies that balance cost and recovery speed. Covers RPO and RTO definitions, DR tiers, pilot light and warm standby architectures, failover automation, and the patterns that keep businesses running when entire regions go down.
Cloud disaster recovery architecture ensures that systems remain available and data intact after a catastrophic failure—such as a regional outage, data center burnout, or complete cloud platform failure. It is essential when uptime is non-negotiable and data loss cannot be tolerated. This architecture is not a one-time setup but a continuous, monitored, and tested process that brings systems online in a secondary environment, typically across geographic zones or cloud providers.
Recovery across geographies requires a consistent strategy for data replication, network configuration, and service orchestration. The core principle is that systems must be able to fail over from a primary region to a secondary region with minimal data loss and service interruption.
Data is the foundation of recovery. To minimize data loss, systems use a combination of synchronous and asynchronous replication.
To configure this, systems use:
database:
replication:
mode: "synchronous"
standby:
- name: "primary-us-east-1"
host: "db-primary-us-east-1.example.com"
port: 5432
wal_level: "replica"
synchronous_standby_names: "standby-us-west-2"
wal_keep_segments: 128The synchronous_standby_names parameter defines which standby nodes must acknowledge writes before the primary commits. If a standby node fails to respond within the synchronous_commit_timeout, the primary falls back to asynchronous mode.
In the event of a failure, the secondary region can promote a standby database to primary with:
-- Promote a standby node to primary
SELECT pg_promote();This command returns immediately once the node becomes the primary, but clients must reconfigure their connection strings to point to the new primary.
For stateful services like application servers and message queues, recovery depends on consistent state across regions. This is achieved through incremental backups and log shipping.
Every 15 minutes, a backup job runs:
# Run incremental backup using rsync with hard links
rsync -a --link-dest=/backups/last \
/data/app-state/ \
/backups/$(date +%Y-%m-%d-%H:%M)This command creates a new backup directory with hard links to the previous full backup, reducing storage overhead. The --link-dest flag ensures that unchanged files are shared between backups.
The backup is then uploaded to a durable storage tier (S3, Azure Blob, GCP Cloud Storage):
aws s3 sync /backups/ s3://backup-registry-geo/ \
--exclude "*.tmp" \
--include "*.tar.gz" \
--deleteAfter the backup completes, the system logs the backup ID and timestamp in a metadata table:
CREATE TABLE backup_metadata (
id UUID PRIMARY KEY,
region TEXT NOT NULL,
start_time TIMESTAMP WITH TIME ZONE,
end_time TIMESTAMP WITH TIME ZONE,
size_bytes BIGINT,
status TEXT CHECK (status IN ('started', 'completed', 'failed'))
);When a failure occurs, the recovery process queries this table to identify the most recent successful backup and applies it to the secondary region.
Failover is not a manual process. It is driven by health checks and orchestrated by runbooks that define recovery steps.
Systems monitor the health of services and infrastructure using probes that check:
A health check is implemented as a simple HTTP endpoint:
{
"name": "api-health",
"path": "/health",
"method": "GET",
"timeout_ms": 5000,
"expected_status": 200,
"expected_body": {
"status": "healthy"
},
"interval_seconds": 30,
"failure_threshold": 3
}Each health check is polled by a monitoring system. If a check fails for three consecutive intervals, the system triggers a failure event.
A runbook is a series of executable steps that guide the failover process. It is defined as a sequence of commands and checks, stored in a version-controlled repository.
# failover-runbook.yaml
version: "1.0"
steps:
- name: "verify-primary-region-down"
description: "Check if primary region is completely unreachable"
command: |
curl -s http://api-primary.example.com/health | jq '.status'
expected: "unhealthy"
timeout_seconds: 60
retries: 3
- name: "promote-database"
description: "Promote standby database to primary"
command: |
psql -h db-standby-us-west-2.example.com -c "SELECT pg_promote();"
expected: "true"
- name: "update-dns"
description: "Update DNS to point to secondary region"
command: |
aws route53 change-resource-record-sets \
--hosted-zone-id Z1234567890 \
--change-batch file://dns-changes.json
- name: "restart-services"
description: "Start services on secondary region"
command: |
kubectl rollout restart deployment/api-server --namespace=prod
systemctl restart nginx
timeout_seconds: 120The dns-changes.json file defines the DNS changes:
{
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "api.example.com.",
"Type": "A",
"TTL": 300,
"ResourceRecords": [
{
"Value": "192.0.2.1"
}
]
}
}
]
}When a failure is detected, the runbook is executed in sequence. If any step fails, the system logs the error and halts execution.
After execution, the system validates that the failover succeeded:
# Validate failover by running end-to-end tests
curl -s http://api.example.com/v1/users | jq '.count' | grep -q '^[1-9][0-9]*$'
if [ $? -ne 0 ]; then
echo "Failover validation failed: user count not returned"
exit 1
fiValidation is performed by a script that calls key APIs and verifies responses. If validation fails, the system sends alerts via email and Slack.
Data consistency across regions is critical. Inconsistencies lead to corrupted state, lost transactions, and user confusion.
Point-in-time recovery allows systems to restore data to a specific moment, minimizing data loss after a failure.
To support PITR, systems maintain a continuous log of database changes (WAL logs) and regularly archive them.
The recovery process involves:
# Restore database from base backup and apply WAL logs
pg_basebackup -D /var/lib/postgresql/data \
-h db-primary.example.com \
-U backup_user \
-P \
-X stream
# Apply WAL logs to reach target time
pg_rewind --source-data=/path/to/base-backup \
--target-data=/var/lib/postgresql/data \
--timeline 10 \
--start-timeline 5
# Apply archived WAL logs
pg_waldump -p /var/lib/postgresql/wal/ -s 2024-01-01T00:00:00Z -e 2024-01-01T08:30:00ZThe pg_waldump command outputs the sequence of WAL records, which can be applied manually or by a recovery script.
Recovery points are identified by a recovery point identifier (RPI), a unique timestamp or sequence number that marks the state of the system.
When data is replicated across regions, conflicts arise when updates occur simultaneously in both regions.
A conflict resolution strategy must be defined. One common approach is last-write-wins:
CREATE TABLE user_profile (
id UUID PRIMARY KEY,
name TEXT,
updated_at TIMESTAMPTZ,
version BIGINT,
region TEXT
);
-- Update a user profile in primary region
UPDATE user_profile
SET name = 'Alice', updated_at = NOW(), version = version + 1
WHERE id = 'a1b2c3d4';
-- Update same user in secondary region
UPDATE user_profile
SET name = 'Alicia', updated_at = NOW(), version = version + 1
WHERE id = 'a1b2c3d4';If both updates are applied, the one with the later updated_at timestamp wins. If timestamps are equal, the region with the higher priority (e.g., us-east-1 > us-west-2) resolves the tie.
A more robust strategy uses version vectors or vector clocks:
{
"user_id": "a1b2c3d4",
"name": "Alicia",
"version": {
"us-east-1": 12,
"us-west-2": 8
},
"updated_at": "2024-01-01T08:30:00Z"
}The recovery system merges updates by combining the version vectors and resolving conflicts based on the highest version numbers.
A disaster recovery architecture is only effective if it is tested and monitored.
Key metrics include:
These metrics are collected via a monitoring system that logs events:
{
"event": "failover_started",
"timestamp": "2024-01-01T08:00:00Z",
"region": "us-east-1",
"backup_id": "2024-01-01-07:45",
"rpo_seconds": 60,
"rto_seconds": 120
}The system calculates RPO and RTO from the event logs and alerts when values exceed thresholds.
Disaster recovery drills are scheduled tests that simulate a failure and validate the entire recovery process.
A drill runs as a sequence of events:
The drill is scripted and automated:
#!/bin/bash
# disaster-drill.sh
set -euo pipefail
echo "Starting disaster recovery drill..."
echo "Step 1: Stop primary database"
kubectl scale deployment/db-primary --replicas=0
echo "Step 2: Simulate network partition"
# Using tc (traffic control) to delay and drop packets
sudo tc qdisc add dev eth0 root handle 1:0 netem delay 200ms 50ms loss 10% rate 1mbit
echo "Step 3: Run backup job"
./run-backup.sh
echo "Step 4: Execute failover runbook"
./runbook-runner.py failover-runbook.yaml
echo "Step 5: Validate service"
./validate-services.sh
echo "Drill completed successfully."After the drill, the system records the results and updates a runbook status tracker.
Several silent failures commonly occur in cloud disaster recovery:
--target flag in pg_basebackup.pg_promote() is called but the standby does not become primary due to missing recovery.conf file or incorrect standby.signal.The most common mistake: failing to test the recovery process in a real failure scenario. Teams assume that everything works because the runbook runs, but they forget to verify that the actual system is serving requests after the failover.
The architecture must include post-failover validation scripts that execute real user workflows, not just health checks.
Cloud disaster recovery architecture is not a collection of tools—it is a discipline of consistency, monitoring, and rigor. Its value emerges not in the quiet moments, but in the chaos of a real disaster.
This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy