Disaster Recovery Testing | The Garnet Wiki

This page describes how to validate disaster recovery procedures through testing that exercises recovery workflows and confirms systems can restore from failure states. Testing occurs when a system's recovery mechanisms must be validated against real or simulated failures, including complete datacenter outages, service cascades, or component unavailability.

Test Planning and Setup

Define Recovery Scenarios

Disaster recovery tests should validate specific scenarios that match documented failure modes. The test scope includes:

Complete service interruption with no active components
Data loss in primary storage
Network partitioning between services
Failures in critical dependencies (databases, message queues, caches)

Use a test matrix to document which systems and data flows are covered by each test run.

Prepare Test Environment

Test environments must be isolated from production and include:

Standalone copies of all relevant configuration files
Replica sets or volumes matching production capacity
Network isolation rules for failure simulation
Credentials with limited permissions for test execution

Example test environment setup:

# Create test namespace
kubectl create namespace dr-test-$(date +%Y%m%d-%H%M%S)

# Apply test configuration
kubectl apply -n dr-test-$(date +%Y%m%d-%H%M%S) -f test-deployment.yaml

Configure Test Data

Test data must be representative but not sensitive. Create a known dataset with:

Predictable identifiers that can be verified post-recovery
Expected state values for validation
Sample transaction logs or audit trails

Execution and Validation

Run Recovery Workflows

Disaster recovery testing requires executing the documented recovery procedures in a controlled manner. Procedures include:

Isolate the failure zone
Trigger failover to backup systems
Validate data consistency
Confirm service availability

Example procedure:

# Simulate database outage
kubectl scale deployment database --replicas=0 -n production

# Wait for service to detect failure
kubectl get pods -n production -l app=database

# Verify failover process
kubectl get pods -n backup -l app=database

Validate Recovery State

Validation involves checking that systems have recovered correctly and are in a usable state. This includes:

Confirming all services are responding
Verifying data integrity through checksums
Testing end-to-end workflows
Checking for any error logs or warnings

Example validation command:

# Check service availability
curl -f http://service.example.com/healthz

# Verify data consistency
psql -d appdb -c "SELECT COUNT(*) FROM user_sessions;"

Monitor During Recovery

Monitoring during testing must capture:

System response times
Error rates
Resource utilization
Log entries from critical components

Use tools like Prometheus and Grafana for real-time monitoring:

# Query system metrics
curl http://prometheus.example.com/api/v1/query?query=rate(http_requests_total[5m])

Testing Failure Scenarios

Complete System Outage

Test complete system failure by:

Stopping all primary services
Verifying backup systems activate
Confirming data consistency between systems

Example command sequence:

# Stop primary database
kubectl scale deployment database-primary --replicas=0 -n production

# Verify failover to secondary
kubectl get pods -n production -l app=database-secondary

# Validate data integrity
psql -d appdb -c "SELECT COUNT(*) FROM transactions;"

Data Loss Simulation

Simulate data loss by:

Removing or corrupting critical data files
Verifying recovery from backups
Confirming data restoration completeness

Example:

# Remove backup file
rm /var/lib/database/backup_$(date +%Y%m%d).sql

# Trigger manual restore
kubectl exec -it deployment/database -- pg_restore -d appdb /backup/latest.sql

Network Partition Testing

Test network partition by:

Isolating service communication
Verifying internal service recovery
Confirming data consistency across partitions

Example:

# Apply network policy to isolate
kubectl apply -f isolation-policy.yaml

# Verify service responses
kubectl get pods -n production -l app=api-server

Validation and Verification

Data Consistency Checks

Verify that all data is consistent after recovery:

Compare checksums of critical datasets
Validate transaction logs
Check for missing or duplicate records

Example validation:

# Calculate data checksum
md5sum /var/lib/database/data.sql

# Compare with backup
diff <(md5sum /var/lib/database/data.sql) /backup/checksum.txt

Service Availability Verification

Confirm all services are functional:

Test API endpoints for expected responses
Verify authentication systems work
Check that internal service communication is restored

Example:

# Test service endpoint
curl -s http://api.example.com/status | jq '.status'

# Verify authentication
curl -H "Authorization: Bearer $TOKEN" http://api.example.com/secure

Performance Validation

Ensure recovery does not introduce performance degradation:

Compare response times with pre-failure measurements
Validate resource usage is within expected bounds
Check for any unexpected latency increases

Example:

# Query performance metrics
curl http://prometheus.example.com/api/v1/query?query=rate(http_request_duration_seconds_sum[5m])

Common Failure Modes and Gotchas

Silent Data Corruption

Data corruption can occur without visible errors. Watch for:

Inconsistent checksums after restore
Missing data in backup files
Partial transaction recovery

Example:

# Validate data integrity
psql -d appdb -c "SELECT COUNT(*) FROM user_sessions;" > /tmp/session_count.txt
psql -d appdb -c "SELECT COUNT(*) FROM transactions;" > /tmp/transaction_count.txt

Configuration Drift Issues

Configuration differences between test and production environments can cause failures:

Validate that all configuration files match
Check for environment-specific variables
Confirm service dependencies are correctly defined

Example:

# Compare configuration files
diff <(kubectl get configmap app-config -n production -o yaml) /tmp/test-config.yaml

Resource Limitation Failures

Insufficient resources in test environments can mask real-world issues:

Ensure test environments have adequate CPU and memory
Verify storage capacity matches production
Check network bandwidth limits

Example:

# Check resource allocation
kubectl top pods -n dr-test-$(date +%Y%m%d-%H%M%S)

Log and Monitoring Gaps

Missing logs can hide failures during testing:

Verify logging configuration in test environment
Confirm monitoring tools are capturing all events
Check for log rotation or retention issues

Example:

# Check for missing logs
kubectl logs deployment/api-server -n dr-test-$(date +%Y%m%d-%H%M%S) --since=1h

Test Documentation and Reporting

Capture Test Results

Document each test with:

Timestamps of all actions
Expected vs actual results
Any deviations from procedure
Error messages or warnings encountered

Example:

# Log test execution
echo "$(date): Started DR test for database outage" >> /var/log/dr-test.log

Report Failures and Issues

Record all problems found during testing:

Include error logs and stack traces
Document environmental factors
Note any configuration changes needed

Example:

# Log failure details
echo "Test failed at $(date): Database restore incomplete" >> /var/log/dr-test-failures.log

Update Recovery Procedures

After each test, update procedures based on findings:

Add missing steps from failures
Adjust timing or resource requirements
Document new troubleshooting approaches

Example:

# Update recovery documentation
git add recovery-procedures.md
git commit -m "Update procedure for database restore after outage"

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.