Chaos Engineering

Build confidence in system resilience by intentionally injecting failures. Covers chaos experiment design, blast radius control, game days, chaos in CI/CD, and building an organizational culture that embraces controlled failure.

Chaos Engineering is the practice of intentionally introducing failures into a system to validate its resilience and identify weaknesses before they manifest in production. It matters when you need to understand how your system behaves under stress, when you want to verify that your monitoring and alerting correctly detect problems, or when you're building confidence in an infrastructure that's expected to survive real-world failure conditions.

## Setting Up a Chaos Experiment

### Create the experiment definition

A chaos experiment begins with a clear hypothesis. You define what you expect to happen and what constitutes a successful outcome.

# Example experiment definition (not a specific product format)
---
experiment:
  title: "Database connection timeout under load"
  description: "Simulate database connection timeouts while serving requests"
  phases:
    - name: "baseline"
      duration: 300
      actions:
        - type: "latency"
          target: "database"
          duration: 10
          latency: 5000
    - name: "failure"
      duration: 60
      actions:
        - type: "failure"
          target: "database"
          duration: 30
          failure_rate: 0.1

This format is generic. You should implement it in your own system, not assume a specific tool has this exact YAML schema.

### Define the chaos infrastructure

You must have access to a control plane that can inject faults into your services. This can be a simple script or an orchestration layer:

# Example chaos control script
#!/bin/bash

# Start injecting network latency
tc qdisc add dev eth0 root netem delay 500ms

# Wait for duration
sleep 30

# Remove the delay
tc qdisc del dev eth0 root

### Configure chaos targets

You need to identify which services or components will be affected. This requires listing your service endpoints, container IDs, or process PIDs:

# List all running containers
docker ps

# Get process IDs for a service
pgrep -f myservice

# Get pod names from Kubernetes
kubectl get pods -n production

## Running Chaos Experiments

### Execute an experiment step-by-step

Run your chaos experiment by executing the defined phases in sequence:

# Run baseline phase (300 seconds)
./chaos-runner.sh --phase baseline --duration 300

# Run failure phase (60 seconds)
./chaos-runner.sh --phase failure --duration 60

You must ensure that each phase starts at the expected time and runs for the specified duration. If your chaos tool does not provide this functionality, you must implement it yourself.

### Monitor during execution

During a chaos experiment, you should monitor system behavior using standard tools:

# Watch logs in real time
kubectl logs -n production -l app=myapp --follow

# Monitor CPU and memory usage
top -p $(pgrep -f myservice)

# Check service availability
curl -v http://myapp.example.com/health

### Validate results

After running a chaos experiment, validate whether your system behaved as expected:

# Query metrics database for latency spikes
psql -c "SELECT avg(latency) FROM metrics WHERE timestamp > '2024-01-01' AND service = 'myapp'"

If your monitoring system is not capturing the chaos events, you will not be able to validate the experiment.

## Types of Chaos Failures

### Network failure injection

Network failures are among the most common and impactful. You can simulate them using tc (traffic control):

# Introduce packet loss
tc qdisc add dev eth0 root netem loss 10%

# Add latency and jitter
tc qdisc add dev eth0 root netem delay 100ms 20ms

# Simulate network partition
iptables -A OUTPUT -d 10.0.0.0/8 -j DROP

When you run these commands, ensure that they apply only to the target system and not globally. Incorrectly applied rules can break your control plane or make the system unreachable.

### Process failure simulation

You can simulate process crashes by sending signals to running processes:

# Kill a service process
kill -TERM $(pgrep -f myservice)

# Or send SIGKILL if graceful shutdown fails
kill -KILL $(pgrep -f myservice)

If your service does not handle SIGTERM gracefully, you may see unexpected restart patterns that do not reflect real-world failures.

### Resource exhaustion

Simulate resource constraints by consuming memory or CPU:

# Consume memory using a loop
while true; do
  dd if=/dev/zero of=/tmp/memfill bs=1M count=100
done

# Or use stress-ng for CPU usage
stress-ng --cpu 4 --timeout 60s

If you run out of disk space during the experiment, it can mask actual failures and make results unreliable.

## Managing Chaos Experiment Results

### Collecting metrics

During a chaos experiment, capture system metrics in a structured way:

# Sample CPU usage every second
sar -u 1 60 > cpu_metrics.log

# Capture memory usage
free -m > memory_metrics.log

# Monitor network traffic
iftop -t -s 60 > network_metrics.log

These metrics must be stored with timestamps that align with your chaos experiment phases.

### Logging failure events

Log all relevant system events during the experiment:

# Capture systemd logs
journalctl --since "2024-01-01 00:00:00" --until "2024-01-01 00:05:00" > chaos_logs.log

# Or collect from container logs
docker logs mycontainer > container_logs.log

If your logging system cannot capture logs at the right time, you will not have evidence to support or refute your experiment's results.

### Evaluating success criteria

Define what constitutes a successful experiment before starting:

# Check if service uptime meets expectations
curl -f http://myapp.example.com/health || echo "Service is down"

# Verify error rates remain within acceptable limits
psql -c "SELECT count(*) FROM errors WHERE timestamp > '2024-01-01' AND service = 'myapp'" > error_count.log

If your success criteria are not measurable, you cannot validate whether the chaos experiment succeeded or failed.

## Common Pitfalls and Failure Modes

### Incorrect failure injection

You may inject a failure that does not represent a real-world scenario. For example, introducing a 50ms delay when a real network issue causes a 500ms delay:

# This might be too subtle
tc qdisc add dev eth0 root netem delay 50ms

# While real failures are usually much longer
tc qdisc add dev eth0 root netem delay 500ms

If you use too small a delay, your system may never fail in production because it's resilient to that particular failure.

### Overlapping chaos phases

Running multiple chaos phases simultaneously can cause conflicts:

# Phase 1: Simulate disk I/O saturation
stress-ng --io 4 --timeout 60s &

# Phase 2: Introduce network latency
tc qdisc add dev eth0 root netem delay 100ms

# If the system is already under stress, the second failure may not manifest clearly

You must ensure that each chaos phase introduces a single, well-defined failure.

### Incomplete experiment capture

If you do not properly record what happened during an experiment, you cannot reproduce results or learn from them:

# Example of incomplete logging
# This only captures metrics but not the actual chaos actions taken
./chaos-runner.sh --phase failure --duration 60 > results.log

# Better approach: log all actions
echo "Starting chaos phase: failure" >> chaos.log
tc qdisc add dev eth0 root netem delay 100ms >> chaos.log
sleep 60
tc qdisc del dev eth0 root >> chaos.log

If your experiment logs only the final results and not the process, you cannot understand what caused which outcome.

### Failure in chaos tooling itself

Your chaos tools may themselves fail:

# If chaos runner crashes or exits early
./chaos-runner.sh --phase baseline --duration 300
# Exit code is 1, but no error message in logs

You must verify that your chaos tool runs to completion and handles errors gracefully.

### Misinterpretation of results

Chaos experiments can fail to show the real problem if you misinterpret the outcome:

# Example: system recovery looks good but was due to auto-restart, not resilience
# You see logs saying "service restarted", but it's just a default restart policy
kubectl get pods -n production | grep myapp

You must distinguish between automatic recovery and actual system resilience.

## Documenting Chaos Experiments

### Record experiment metadata

Always document what was tested, when, and why:

# Example metadata file
experiment_id: "chaos-001"
title: "Database connection timeout under load"
date: "2024-01-01"
author: "engineering-team"
target_system: "myapp"
expected_outcome: "system should continue serving requests with 5% error rate"
actual_outcome: "system served 95% of requests correctly"

### Capture all relevant system states

Store logs, metrics, and configuration snapshots at key points:

# Before experiment
kubectl get pods -n production > before.log
kubectl get nodes > before_nodes.log

# After experiment
kubectl get pods -n production > after.log
kubectl get nodes > after_nodes.log

## Scaling Chaos Experiments

### Run experiments in production

To get meaningful results, chaos experiments must run in production environments:

# Run on a subset of production traffic
kubectl patch deployment myapp -p '{"spec":{"replicas":10}}'
# Run chaos for 30 minutes
./chaos-runner.sh --phase failure --duration 1800

Running experiments only in staging can miss real-world edge cases.

### Automate experiment execution

Set up a schedule for regular chaos experiments:

# Example cron job to run weekly
0 0 * * 0 /opt/chaos/bin/run-weekly.sh

You must verify that automated runs complete successfully and produce reliable results.

Chaos Engineering is not just about breaking things — it's about understanding how your system behaves when it breaks. The key is to be precise in your failures, careful in your measurements, and honest in your interpretations.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.