Chaos Engineering

Injecting realistic faults as controlled experiments: defining steady state, forming a hypothesis, bounding blast radius, the fault classes that actually teach you something, game days, and the prerequisites that separate an experiment from a self-inflicted outage.

Chaos Engineering is the practice of proactively introducing controlled failures into a system to validate its resilience and identify weaknesses in production. It matters when systems are complex enough that failure modes cannot be fully anticipated or tested in staging environments, especially in distributed systems where interactions between components can produce unexpected outcomes.

How to Design and Run Chaos Experiments

Start with a hypothesis about system behavior under stress

A chaos experiment begins with a testable statement about how your system should behave when a specific component fails. The hypothesis must be precise enough that you can observe whether it holds true or not. For example, "If we simulate a 50% network latency between microservices, the API response time will increase by no more than 30%."

This approach requires identifying failure points that are likely to occur in production but aren't covered by standard testing. You should focus on areas where you have data about recent failures or where system behavior under stress isn't well understood.

Define your experiment scope and metrics

Before running any chaos experiment, determine which parts of your system will be affected and what metrics you'll monitor. For instance, if you're targeting a web application, you might choose to inject latency into the database layer. You should identify:

Use monitoring tools that provide metrics with sufficient granularity for analysis. If your system uses Prometheus, ensure you're collecting data from all relevant components including service mesh proxies, load balancers, and application instances.

Execute a controlled failure injection

The most common form of chaos experiment is injecting faults into your system using tools like iptables or container runtime commands. For example:

# Simulate network latency in Docker containers
docker exec <container_id> tc qdisc add dev eth0 root netem delay 100ms

Or with iptables for more granular control over network behavior:

# Introduce packet loss on a specific port
iptables -A OUTPUT -p tcp --sport 8080 -m random --random-probability 0.2 -j DROP

These commands must be applied carefully, with consideration for how they interact with existing network rules and whether they are reversible.

Monitor and measure outcomes

During the experiment, continuously monitor system metrics using tools like Prometheus or Grafana dashboards. Key indicators to track include:

If you observe that response times increase beyond expected thresholds, this indicates a weakness in your system's handling of the injected fault.

Common Failure Injection Techniques

Network faults

Network failures are among the most common types of chaos experiments. You can simulate various network conditions using standard Linux utilities:

# Add latency to outbound traffic
tc qdisc add dev eth0 root netem delay 50ms

# Introduce packet loss
tc qdisc add dev eth0 root netem loss 10%

# Add packet corruption
tc qdisc add dev eth0 root netem corrupt 5%

These commands work on Linux-based containers and hosts. They require root privileges and must be applied to the correct network interface (eth0 in the examples above). If you are running inside a container without proper network permissions, these commands will silently fail.

CPU or memory stress

Resource exhaustion can cause cascading failures. Stress tools like stress-ng or custom scripts can simulate resource constraints:

# Simulate high CPU load
stress-ng --cpu 4 --timeout 60s

# Allocate memory to force garbage collection
stress-ng --vm 2 --vm-bytes 512M --timeout 30s

These commands are not available in all environments, so ensure they are installed before beginning experiments. A common failure mode is that these tools do not accurately reflect real-world resource contention patterns.

Service disruptions

Simulate service unavailability by stopping or restarting processes:

# Stop a process and restart it after delay
systemctl stop myapp.service
sleep 30
systemctl start myapp.service

Use systemctl to manage services, but be aware that not all systems use systemd. On some platforms, you may need to use kill, pkill, or direct process management through a container orchestrator like Kubernetes.

Database or storage failures

For database systems, simulate connection timeouts or slow queries:

# Simulate slow queries using PostgreSQL's pg_sleep function
psql -c "SELECT pg_sleep(5);"

Or introduce artificial delays in database connections:

# Set a timeout on a database connection
PGCONNECT_TIMEOUT=30 psql -d mydb -c "SELECT * FROM users;"

These commands assume PostgreSQL, but similar concepts apply to other databases. The specific failure behavior depends on how each database implements timeouts and connection handling.

Managing Experiment Lifecycle

Prepare for rollback

Every chaos experiment should have a clear rollback plan. When applying changes, always record what was modified:

# Save current network configuration before injection
tc qdisc show dev eth0 > /tmp/network_config_before.txt

# Restore after experiment
tc qdisc del dev eth0 root

Failure to clean up after experiments can lead to persistent performance degradation or unexpected behavior in production.

Use idempotent operations

Ensure your chaos commands can be safely repeated. If you run the same command twice, it should have the same effect as running once:

# This may cause duplicate rules if run multiple times
tc qdisc add dev eth0 root netem delay 50ms

# Better: check for existing rule before applying
if ! tc qdisc show dev eth0 | grep -q "netem"; then
    tc qdisc add dev eth0 root netem delay 50ms
fi

Document experiment results

Capture logs and metrics from each run to build a history of system behavior under various failure conditions. This documentation helps inform future experiments and provides baseline data for comparisons.

Common Pitfalls and Silent Failures

Silent failures in network rules

Applying tc commands silently fails when the network interface doesn't exist or when you lack required permissions. For example:

# This command might succeed even though no rule was applied
tc qdisc add dev nonexistent_interface root netem delay 50ms

To detect such failures, check for existing rules before and after application:

tc qdisc show dev eth0
# If no output, the interface may not exist

Overlapping chaos injection tools

Multiple tools can interfere with each other when injecting similar faults. For example, using both iptables and tc simultaneously on the same network path can create conflicting rules:

# This could conflict with existing traffic control rules
iptables -A OUTPUT -p tcp --sport 8080 -j DROP
tc qdisc add dev eth0 root netem delay 100ms

Misunderstanding system behavior under load

Some experiments fail because the system under test behaves differently than expected. For example, a service might be resilient to CPU stress but fail when combined with network latency:

# This might appear successful
stress-ng --cpu 4 --timeout 60s

# But doesn't account for how it interacts with other components

Always validate that your experiment reproduces real-world conditions and doesn't oversimplify system interactions.

Resource exhaustion during experiments

Running multiple chaos experiments in parallel can overwhelm system resources. For example:

# Simultaneous stress tests can cause cascading failures
stress-ng --cpu 8 --timeout 120s &
stress-ng --vm 4 --vm-bytes 1G --timeout 120s &

Monitor resource usage during concurrent experiments to avoid overwhelming the system.

Inconsistent metrics collection

If your monitoring tools aren't configured to capture all relevant metrics, you might miss critical failures:

# Prometheus scrape configuration should include all relevant targets
scrape_configs:
  - job_name: 'myapp'
    static_configs:
      - targets: ['localhost:8080', 'localhost:8081']

Ensure that your monitoring setup captures metrics from all components involved in the chaos experiment.

Tools and Utilities

Linux utilities for network chaos

Use tc (traffic control) for network-related faults:

# Add delay to all traffic
tc qdisc add dev eth0 root netem delay 100ms

# Remove delay
tc qdisc del dev eth0 root

Process management tools

Use stress-ng, stress, or custom scripts for resource stress testing:

# Example of a simple memory stress script
#!/bin/bash
for i in {1..5}; do
    dd if=/dev/zero of=/tmp/stress_$i bs=1M count=100 &
done
wait
rm /tmp/stress_*

Container orchestration integration

When using Kubernetes, chaos experiments often involve pod disruptions:

# Delete a pod to simulate failure
kubectl delete pod <pod_name> --grace-period=0

# Or use a tool like Chaos Mesh for more advanced injection
kubectl apply -f chaos-mesh-pod-kill.yaml

These commands must be run with appropriate permissions and should be part of a larger orchestration strategy.

Best Practices for Reliable Experiments

Start small and scale gradually

Begin with simple experiments targeting single components before combining multiple failures:

# First, test network delay alone
tc qdisc add dev eth0 root netem delay 100ms

# Then combine with CPU stress
stress-ng --cpu 2 --timeout 60s

Test with realistic failure scenarios

Ensure your experiments reflect actual production failures. For example, instead of just dropping connections, simulate connection timeouts or partial data loss.

Validate experiment outcomes

Always confirm that the expected failure was injected and that system behavior matches expectations:

# Verify network delay was applied
tc qdisc show dev eth0

# Check for specific error logs
journalctl -u myapp.service --since "1 minute ago"

Document everything

Keep detailed records of each experiment including:

This documentation becomes invaluable for future experiments and incident analysis.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.