Reliability Testing

Production engineering guide for reliability testing covering patterns, implementation strategies, and operational best practices.

Reliability testing verifies that systems maintain specified performance and availability levels under defined workloads and failure conditions. It matters when you need to validate system behavior before production deployment, during capacity planning, or when evaluating changes to infrastructure or application code.

Test Design and Execution

Create Load Patterns for System Stress

Use ab (Apache Bench) to generate HTTP load patterns:

ab -n 1000 -c 10 -H "Authorization: Bearer token" http://service.example.com/api/endpoint

The -n flag sets total requests, -c sets concurrent connections. ab does not support custom headers in the same way as modern tools like wrk, but it's widely available and sufficient for basic load testing.

Simulate Network Conditions

Use tc (traffic control) to simulate network latency and packet loss:

tc qdisc add dev eth0 root netem delay 100ms loss 1%
tc qdisc del dev eth0 root

This affects all traffic on eth0. The netem module can also introduce correlation between packet loss and delay, which is critical for realistic failure simulation.

Configure Test Duration and Metrics

Test duration should match expected production workload patterns. For example:

timeout 3600 wrk -t12 -c400 -d30s http://service.example.com/api/endpoint

The wrk tool uses -t for threads, -c for connections, and -d for duration. The test should run long enough to observe steady-state behavior.

Test Validation

Validate Response Times and Error Rates

Use curl with timing options:

curl -w "@curl-format.txt" -o /dev/null -s http://service.example.com/api/endpoint

Where curl-format.txt contains:

time_namelookup:  %{time_namelookup}\n
time_connect:     %{time_connect}\n
time_appconnect:  %{time_appconnect}\n
time_pretransfer: %{time_pretransfer}\n
time_redirect:    %{time_redirect}\n
time_starttransfer: %{time_starttransfer}\n
speed_download:   %{speed_download}\n

This provides granular timing data that helps identify where delays occur.

Monitor System Resources During Tests

Use top or htop to observe CPU and memory usage:

top -p $(pgrep -f "service-name")

Or use vmstat for more detailed resource monitoring:

vmstat 1 60

These commands show real-time system load but don't provide structured metrics. For production testing, consider adding Prometheus metrics to your services and querying them with curl against the /metrics endpoint.

Test Configuration

Define Failure Scenarios

Use iptables rules to simulate service unavailability:

iptables -A INPUT -p tcp --dport 8080 -j DROP
iptables -D INPUT -p tcp --dport 8080 -j DROP

This drops all incoming traffic to port 8080. The -A flag appends a rule, -D deletes it.

Configure Test Data Sets

Create test data with psql:

INSERT INTO users (id, name) VALUES (generate_series(1,10000), 'Test User ' || generate_series(1,10000));

This inserts 10,000 rows into a users table. The generate_series() function is PostgreSQL-specific but widely available.

Set Up Test Environments

Use docker-compose to replicate production environments:

version: '3'
services:
  app:
    image: myapp:latest
    ports:
      - "8080:8080"
    environment:
      - DATABASE_URL=postgresql://user:pass@db:5432/test
  db:
    image: postgres:13
    environment:
      - POSTGRES_PASSWORD=pass

This creates a simple test environment with app and database containers. The docker-compose tool is available on most Linux systems.

Test Results Analysis

Parse and Store Metrics

Use jq to extract structured data from API responses:

curl http://service.example.com/metrics | jq '.[] | select(.name == "request_duration")'

This filters metrics for a specific metric name. The jq tool is essential for processing JSON output from monitoring systems.

Validate Service Health

Use kubectl to check pod status:

kubectl get pods -l app=service-name

This retrieves pods matching a label selector. The command fails with exit code 1 if no matching pods are found, which is important for automated testing scripts.

Identify Performance Degradation

Compare test results using diff:

diff baseline.txt test-run.txt

This compares two files containing performance data. For more complex comparisons, use awk to extract specific metrics.

Common Failure Modes

Silent Test Failures

The most common silent failure occurs when HTTP status codes are ignored:

curl -s http://service.example.com/api/endpoint

This command succeeds even if the server returns 500 errors. Always check exit codes:

curl -f -s http://service.example.com/api/endpoint

The -f flag makes curl fail on HTTP error codes.

Resource Exhaustion During Tests

When running tests with high concurrency, systems may exhaust memory or file descriptors. Monitor system logs for OOM killer activity:

dmesg | grep -i "killed"

This shows when processes were terminated by the kernel due to memory pressure.

Configuration Mismatches

Test configurations often differ from production setups. Use git diff to compare test and production configuration files:

git diff production.yml test.yml

This reveals differences in environment variables, timeouts, or connection parameters that may affect test validity.

Test Automation

Scripted Test Execution

Create a test script with proper error handling:

#!/bin/bash
set -e

echo "Starting reliability test..."
timeout 300 wrk -t12 -c400 -d30s http://service.example.com/api/endpoint
if [ $? -ne 0 ]; then
  echo "Test failed"
  exit 1
fi
echo "Test completed successfully"

The set -e option ensures the script exits on any command failure.

Continuous Test Integration

Use cron jobs to run periodic reliability tests:

0 2 * * * /usr/local/bin/reliability-test.sh

This runs a test script every day at 2 AM. The script should be idempotent and not interfere with production traffic.

Test Environment Isolation

Use Dedicated Test Resources

Ensure test environments don't interfere with production:

kubectl create namespace test-env
kubectl config set-context --current --namespace=test-env

This creates a dedicated namespace for tests and switches the current context to it. The kubectl command is available in Kubernetes environments.

Clean Up Test Artifacts

Always clean up after tests:

kubectl delete namespace test-env --wait=false

The --wait=false flag prevents the command from waiting for resources to be fully deleted, which speeds up cleanup.

Advanced Test Patterns

Multi-Service Dependency Testing

Test interactions between services:

curl -s http://service-a.example.com/dependent-call | jq '.service_b_status'

This validates that service A properly handles responses from service B. The test should fail if service B is unavailable.

Database Transaction Testing

Test database operations with psql:

psql -c "BEGIN; INSERT INTO test_table VALUES (1, 'test'); SELECT * FROM test_table WHERE id = 1; ROLLBACK;"

This tests transaction handling and ensures rollback behavior works correctly. The ROLLBACK statement is crucial for testing transactional integrity.

Service Degradation Simulation

Use iptables to simulate partial service failure:

iptables -A INPUT -p tcp --dport 8080 -m statistic --mode random --probability 0.1 -j DROP

This drops 10% of incoming requests to port 8080, simulating intermittent service unavailability.

Test Reporting

Generate Test Reports

Create structured test reports:

echo "Test: $(date)" > report.txt
curl -s http://service.example.com/metrics | jq '.[] | select(.name == "request_count")' >> report.txt

This appends timestamped metrics to a report file. The report should include success/failure status, duration, and key performance indicators.

Validate Test Results

Use grep to validate test outcomes:

curl -s http://service.example.com/metrics | grep "success_rate" | grep -q "95"

This checks if the service maintains a 95% success rate. The grep -q flag suppresses output and uses exit code for validation.

The key to reliable testing is careful attention to how failures manifest, consistent test environment setup, and thorough validation of results. Tests that appear to pass in isolation often fail when run in production conditions or when compared against expected behavior.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.