Disaster Recovery & Business Continuity

Disaster recovery (DR) and business continuity (BC) describe the policies, procedures, and technical mechanisms an organization uses to restore critical services and data after an interruption. In cloud environments, interruptions can range from a single-node process crash to an entire region outage. This reference page matters when you are designing or auditing a system that must meet stated availability guarantees, comply with regulatory retention requirements, or simply avoid prolonged downtime during incidents. The following sections are organized around the actions you need to take: defining what to protect, creating recoverable copies, designing continuity architectures, exercising restore procedures, and recognizing the common pitfalls that catch teams off guard.

Define Critical Scope & Inventory

Before you can protect anything, you must know what constitutes the “critical path.” Start by inventorying the components that, if unavailable, would violate an SLA or halt business operations. This typically includes primary database instances, authentication directories, object storage buckets containing active user content, API gateway routes, and any downstream services your platform calls synchronously. A practical first pass can be achieved by querying your orchestration layer.

For a Kubernetes-based workload, the following command lists every resource across all namespaces, which can be redirected to a file for auditing:

kubectl get all,configmaps,secrets,persistentvolumeclaims -A --no-headers > /tmp/inventory.txt

If your system runs outside Kubernetes, equivalent inventory steps might involve systemctl list-units --type=service --state=running, psql \l to list databases, or aws ec2 describe-instances (referenced here only as a pattern example; the mechanics of enumeration are generic). The output should be sorted by criticality, tagged with owner, and stored in a version‑controlled location so that a DR runbook can reference it without guessing.

Create Recoverable Backups & Snapshots

Once the scope is known, the next action is to produce copies that can be restored independently of the live environment. The pattern is consistent across services: export or dump state to a portable format, compress if beneficial, generate an integrity checksum, and write the artifact to a separate location—ideally a different failure domain (another region, another account, or an on‑premises site).

Database dump (PostgreSQL example)

pg_dump -Fc -f /var/backups/db_$(date +%Y%m%d).dump
gzip -c /var/backups/db_$(date +%Y%m%d).dump > /var/backups/db_$(date +%Y%m%d).dump.gz
sha256sum /var/backups/db_$(date +%Y%m%d).dump.gz > /var/backups/SHA256SUMS

Configuration directory archive

tar -czf /var/backups/config_$(date +%Y%m%d).tgz /etc/app/
sha256sum /var/backups/config_$(date +%Y%m%d).tgz >> /var/backups/SHA256SUMS

Push to an off‑site location via SSH

scp -r /var/backups user@offsite-host:/var/backups

Verify integrity after transfer

ssh user@offsite-host "sha256sum -c /var/backups/SHA256SUMS"

The same pattern applies to NoSQL stores, file systems, or container images: dump/export → compress → checksum → transfer. The critical detail often omitted is that checksums must be generated on the source, written to the artifact, and re‑verified on the destination. A common silent failure is a checksum written to a file that is later overwritten during the same run, rendering the verification step a no‑op.

Design Continuity Architecture

With backups in place, the architecture must define how traffic is redirected when the primary path becomes unhealthy. The pattern is independent of any single cloud vendor and instead relies on three generic mechanisms: health‑check‑driven DNS or load‑balancer routing, active‑active deployment across distinct failure domains, and explicit session or state management.

Health‑check endpoint

curl -sSf http://internal-service/healthz || { echo "upstream unhealthy"; exit 1; }

A non‑zero exit code from curl can trigger a script that updates a DNS A record via any provider’s API, or that shifts a cloud load balancer’s backend weight from one region to another. The key is that the health check must be idempotent, fast (sub‑second), and placed behind the same network path that clients use.

DNS TTL considerations Lower the TTL on critical records (e.g., 300 seconds) before planned maintenance so that cached resolutions expire quickly. After an unplanned outage, the cached TTL from the previous value can delay failover by minutes to hours depending on recursive resolver behavior. A practical habit is to run dig +short api.example.com and compare the observed TTL against the configured value after any change.

Active‑active deployment Deploy the same container image or binary to at least two distinct failure domains—typically separate cloud regions or a region paired with an on‑premises site. Use your orchestrator’s topology spread constraints or node affinity to ensure no single failure domain holds all replicas. Synchronize databases through change data capture or periodic logical replication; avoid relying on a single write endpoint unless your RTO permits the lag.

Session & state handling If your application maintains user session state (HTTP cookies, JWT tokens, in‑memory caches), design for statelessness or externalize state to a distributed store (Redis with replication, DynamoDB with cross‑region replication, PostgreSQL with streaming replication). Local ephemeral caches will be lost during any failover, causing authentication or lookup failures unless a warm‑up procedure is scripted.

Execute Failover & Restore

When an outage is confirmed, the restore sequence should be rehearsed and deterministic. The following steps illustrate a typical database‑first, then service, restore flow. Commands are shown for PostgreSQL and a generic Kubernetes deployment, but the order of operations is the pattern.

1. Detect and isolate

systemctl status app-service || true
kubectl get pods -A | grep -E 'Error|CrashLoop'

If the failure is node‑ or region‑scoped, confirm that no further traffic is being sent to the affected endpoint.

2. Restore the database

pg_restore -d production -j 4 /backups/db_20240115.dump

After restore, run psql -d production -c "SELECT count(*) FROM critical_table;" to confirm row counts match expectations. If you have checksums from the backup step, verify them now: sha256sum -c /backups/SHA256SUMS.

3. Restore configuration files

tar -xzf /backups/config_20240115.tgz -C /etc/app/

Reload or restart the service to pick up the new configuration. For systemd services: systemctl restart app-service. For Kubernetes: kubectl rollout restart deployment app.

4. Verify service availability

curl -sSf http://app.example.com/healthz

A 200 response with the expected body indicates the stack is up. If the health check returns an error, inspect recent pod logs (kubectl logs deployment/app) and confirm that environment variables, secrets, and network policies are correctly applied.

5. Cut over DNS or routing If using DNS‑based failover, update the A record to point to the healthy region’s endpoint. If using a global load balancer, shift traffic weights via its console or API. Document the exact command or API call used, because manual steps are the most common source of inconsistency in post‑mortems.

Run Drills & Validate RPO/RTO

A DR plan that is never exercised is a plan that will fail when it matters. Schedule full‑cycle drills at least quarterly. The drill should mimic a real outage: halt the primary region, restore from the most recent backup, bring services up in the secondary region, and measure the elapsed time.

Drill skeleton (Kubernetes example)

kubectl create namespace dr-test

# 2. Restore database snapshot into a test cluster
pg_restore -d drtest -j 2 /backups/db_$(date +%Y%m%d).dump

# 3. Deploy application manifests into dr-test
kubectl apply -f manifests/ -n dr-test

# 4. Run health checks against the test endpoint
while ! curl -sSf http://app.dr-test.example.com/healthz; do sleep 5; done
echo "Drill service up"

# 5. Record elapsed time from outage simulation to health check success
# 6. Tear down
kubectl delete namespace dr-test

If the measured time exceeds your declared RTO, the bottleneck is usually either transfer size (large dumps over public internet) or the number of restart steps required. Adjustments may include incremental backups, snapshot compression, or reducing the number of dependent services that must come online before the application is considered “up.”

Sharp Edges – What Silently Fails & Common Gotchas

• Incomplete snapshots during active writes: Volume snapshots capture the state at a single moment, but if a database is receiving writes at that exact moment, the dump may contain partial transactions. Use database‑level dump tools (e.g., pg_dump) rather than relying solely on filesystem snapshots for data stores.
• Checksum staleness: Generating a SHA‑256 hash and writing it to a file in the same step can mask errors if the file system caches the write. Always generate the checksum on the source file, store it separately, and verify on the destination after transfer.
• Restored credentials no longer valid: Secrets rotate automatically in many pipelines. A backup taken before a rotation will contain outdated keys. When restoring, have a documented key‑re‑wrap or migration step ready, and test it in a drill.
• DNS TTL persistence: Recursive resolvers may cache A records for the full TTL duration even after you lower it and update the authoritative record. Use dig +short from multiple networks and dig +trace to observe actual resolution behavior after a change.
• Cross‑region network ACLs blocking restore traffic: When pulling a backup from another cloud region or an on‑premises site, security groups or firewall rules may silently drop inbound SSH or rsync traffic. Verify connectivity with nc -zv <host> <port> before relying on automated restore scripts.
• Snapshot ≠ backup: A cloud volume snapshot is tied to the host’s lifecycle. If the entire availability zone is destroyed, the snapshot may be lost. True backups are copies retained independently of the source infrastructure.
• RPO vs. RTO confusion: RPO defines the maximum acceptable data loss (e.g., “no more than 15 minutes of writes”), while RTO defines the maximum acceptable downtime (e.g., “services back online within 30 minutes”). Teams often design backup frequency to meet RPO but then find the restore process takes longer than RTO, or vice versa. Explicitly measure both numbers in drills and adjust the architecture accordingly.
• Elasticsearch / OpenSearch index gaps: Snapshotting these engines requires a brief cluster freeze or a quiesce API call. Omitting this step produces indexes that are inconsistent after restore, leading to missing search results or mapping errors.
• Load balancer health‑check misconfiguration: A health check that only tests TCP connectivity (e.g., curl to port 80 without a path) may return success even when the application is stuck in an error loop. Define checks that hit a meaningful endpoint and validate expected response codes or body content.

These edges are learned in production far too often. Treat each as a checklist item in your runbook, and rotate them into every drill until they become second nature.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.