Cloud Disaster Recovery Design

Production-grade guide to cloud disaster recovery design covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.

Cloud disaster recovery design ensures that systems remain available and data intact when a cloud region, availability zone, or entire data center fails. It is critical when downtime directly impacts revenue, user trust, or regulatory compliance—such as during a network partition, storage corruption, or a complete regional outage. The goal is not just to restore systems, but to minimize data loss and time-to-recovery, with recovery time objectives (RTO) and recovery point objectives (RPO) as measurable anchors.

Planning for Failure

Define RTO and RPO

RTO is the maximum acceptable time between failure and system restoration. RPO is the maximum acceptable amount of data loss measured in time. For example, an RTO of 15 minutes means the system must be usable within 15 minutes of failure; an RPO of 5 minutes means no more than 5 minutes of data can be lost.

A system with RTO = 10 minutes and RPO = 1 minute must be able to:

Failure detection is not automatic. It requires monitoring that continuously checks endpoints, health checks, and metrics. A common pattern uses a heartbeat mechanism:


- service: api-gateway
  endpoint: https://api.example.com/health
  interval: 10s
  timeout: 3s
  expected: { status: 200, body: "OK" }
  failure_threshold: 3
  recovery_threshold: 2

If three consecutive checks fail, the system marks the service as down. After two consecutive successful checks, it recovers. This simple pattern catches network partitions, instance crashes, and slow responses.

Choose Recovery Strategy

Recovery strategies fall into three tiers: cold, warm, and hot.

For a warm strategy, use a standby environment in a secondary region. Data is replicated via log streaming or database replication. For example, PostgreSQL uses WAL (Write-Ahead Logging) for continuous replication:

-- On primary database
ALTER SYSTEM SET wal_level = 'replica';
ALTER SYSTEM SET max_wal_senders = 10;
ALTER SYSTEM SET wal_keep_size = '1GB';

-- On replica
CREATE REPLICATION SLOT my_slot FROM pg_create_physical_replication_slot('my_slot');

The replica applies WAL records continuously. To trigger failover, promote the replica:

-- On replica node
SELECT pg_promote();

This promotes the standby to primary. Clients update their connection strings, and traffic shifts. The failover is complete when the new primary reports pg_is_in_recovery() as false.

Automate Failover

Failover must be automated. Manual steps introduce delays and human error. Use a script to orchestrate the process. For example, a disaster recovery script for a web application:

#!/bin/bash
# /opt/scripts/failover.sh

set -euo pipefail

REGION=${REGION:-"us-west-1"}
PRIMARY_DB=${PRIMARY_DB:-"primary-postgres"}
STANDBY_DB=${STANDBY_DB:-"standby-postgres"}
APPLICATION_NAME=${APPLICATION_NAME:-"web-app"}

# 1. Check if primary is down
if ! curl -sf http://$PRIMARY_DB/health; then
  echo "Primary DB is down. Initiating failover..."

  # 2. Promote standby DB
  psql -h $STANDBY_DB -c "SELECT pg_promote();"

  # 3. Update DNS to point to standby
  aws route53 change-resource-record-sets \
    --hosted-zone-id Z1234567890 \
    --change-batch '{
      "Changes": [{
        "Action": "UPSERT",
        "ResourceRecordSet": {
          "Name": "api.example.com.",
          "Type": "A",
          "TTL": 60,
          "ResourceRecords": [
            { "Value": "192.0.2.100" }
          ]
        }
      }]
    }'

  # 4. Start application in new region
  aws ec2 run-instances \
    --image-id ami-abc123 \
    --instance-type t3.large \
    --count 4 \
    --security-groups sg-123456 \
    --subnet-id subnet-789012 \
    --tag-specifications '[
      {
        "ResourceType": "instance",
        "Tags": [
          { "Key": "Name", "Value": "web-app-failover" },
          { "Key": "Environment", "Value": "dr" }
        ]
      }
    ]'

  # 5. Notify team
  curl -X POST -H "Content-Type: application/json" \
    -d '{"message": "Failover to $REGION completed", "level": "info"}' \
    https://hooks.slack.com/services/ABC123/XYZ456

  echo "Failover complete."
else
  echo "Primary DB is healthy. No failover needed."
fi

This script checks the primary, promotes the replica, updates DNS via AWS Route 53, launches EC2 instances in the secondary region, and sends a Slack alert. Each step is idempotent and can be run repeatedly.

Maintain Data Consistency

Data consistency is the most common failure point. Two systems, both updated during a failure, can diverge. To prevent this, use a dual-write pattern during the failover process.

When writing to the primary database, also write to a secondary database in the standby region. This ensures that no data is lost during the time the primary is down.

// In application code
func WriteUser(user User) error {
  // Write to primary
  if err := db.Write(user); err != nil {
    return err
  }

  // Write to secondary (asynchronously)
  go func() {
    if err := secondaryDB.Write(user); err != nil {
      log.Printf("Failed to write to secondary: %v", err)
    }
  }()

  return nil
}

During failover, the application continues writing to the secondary. The standby system applies these writes in real time, ensuring that no data is lost beyond the RPO.

When promoting the standby to primary, the new primary must process all pending writes from the secondary. This is done via a replay queue:

{
  "topic": "user-updates",
  "messages": [
    {
      "id": "12345",
      "timestamp": "2024-07-01T12:00:00Z",
      "data": {
        "id": 456,
        "name": "Alice",
        "email": "[email protected]"
      }
    }
  ]
}

The standby database reads from the queue and applies changes. This allows the system to catch up and become consistent.

Monitor and Validate

Monitoring must span both normal and disaster states. Use synthetic checks to validate recovery.

# /etc/monitoring/synthetic-checks.yaml
- name: "API Latency Post-Failover"
  type: "http"
  url: "https://api.example.com/users"
  interval: 1m
  threshold: { p95: 500ms }
  alert: {
    condition: "p95 > 500ms",
    message: "Post-failover API latency degraded"
  }

- name: "Database Replication Lag"
  type: "sql"
  query: "SELECT EXTRACT(EPOCH FROM now() - pg_last_wal_receive_lsn()::timestamp) as lag;"
  interval: 30s
  threshold: { max: 10 }
  alert: {
    condition: "lag > 10",
    message: "Replication lag exceeds 10 seconds"
  }

These checks run continuously and alert on deviations. When a failure occurs, validation includes:

A common mistake is assuming the database is up when it is not. Use a health endpoint that checks:

// GET /health
{
  "status": "healthy",
  "checks": {
    "database": {
      "status": "up",
      "latency": 12,
      "error": null
    },
    "cache": {
      "status": "up",
      "latency": 3,
      "error": null
    },
    "workers": {
      "status": "running",
      "count": 8,
      "error": null
    }
  },
  "timestamp": "2024-07-01T12:05:00Z"
}

Handle Partial Failures

Disaster recovery is not just for complete outages. It must handle partial failures: network partitions, disk failures, and memory exhaustion.

When a region fails partially, not all services are down. Use graceful degradation:

For example, during a network partition:

The system must handle split brain: two active databases, both accepting writes, but not synchronized.

To detect split brain:

{
  "primary": {
    "last_heartbeat": "2024-07-01T12:00:00Z",
    "status": "active"
  },
  "standby": {
    "last_heartbeat": "2024-07-01T11:59:30Z",
    "status": "active"
  }
}

If both are active and their heartbeats are far apart, the system detects split brain. Then, it can:

Prepare for Drills and Validation

Disaster recovery designs fail during actual disasters because the team has never practiced. Conduct regular disaster recovery drills.

A drill involves:

Use a drill playbook:

# Disaster Recovery Drill: Regional Outage

## Steps

1. Simulate outage:
   - Stop all instances in primary region.
   - Disable network connectivity.
   - Mark database as read-only.

2. Run failover script:
   - Execute `/opt/scripts/failover.sh`.

3. Validate:
   - Check `/health` endpoints.
   - Run synthetic tests.
   - Verify data consistency.

4. Document:
   - Time to failover.
   - Observed errors.
   - Duration of RTO and RPO.

## Success Criteria

- All services respond within 15 minutes.
- API latency < 500ms.
- Data consistency across databases.
- No user data loss.

Drills uncover silent failures:

Common Pitfalls

To address these, add a rollback script:

#!/bin/bash
# /opt/scripts/rollback.sh

set -euo pipefail

# Promote original primary back
aws ec2 start-instances --instance-ids i-1234567890

# Wait for DB to be ready
until pg_isready -h primary-db.example.com -d app; do
  sleep 5
done

# Update DNS to point to primary
aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "api.example.com.",
        "Type": "A",
        "TTL": 60,
        "ResourceRecords": [
          { "Value": "192.0.2.1" }
        ]
      }
    }]
  }'

# Wait for clients to switch
sleep 60

# Notify team
curl -X POST -H "Content-Type: application/json" \
  -d '{"message": "Rollback to primary region complete"}' \
  https://hooks.slack.com/services/ABC123/XYZ456

Rollback enables testing and recovery of the original system, ensuring that the DR strategy is not just a one-way process.

Final Considerations

Disaster recovery is not a one-time setup. It is a continuous process:

The design must be documented, versioned, and accessible. A single, well-maintained page with concrete commands, configurations, and failure scenarios becomes the anchor for the team’s resilience.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.