Chaos Engineering: Breaking Things on Purpose to Build Confidence

Chaos engineering is the practice of intentionally introducing failures into systems to validate their resilience and uncover hidden weaknesses. It matters when you need to ensure that your infrastructure, applications, or services can withstand real-world disruptions without degradation in performance or availability.

Creating a Chaos Experiment

To begin a chaos experiment, define a hypothesis about system behavior under specific failure conditions. For example: "If we simulate network latency between microservices, the application will continue serving requests with acceptable response times."

Use kubectl to create a pod that injects network delays into a service:

apiVersion: v1
kind: Pod
metadata:
  name: chaos-network-delay
spec:
  containers:
  - name: netem
    image: busybox
    command: ["sh", "-c", "tc qdisc add dev eth0 root netem delay 100ms"]

Apply this with:

kubectl apply -f chaos-network-delay.yaml

This uses Linux's traffic control (tc) to introduce a 100ms delay on all network packets through the eth0 interface.

Injecting Failures

You can simulate various failure types using standard tools and system-level commands.

Network Partition

To simulate a network partition between services, use iptables:

iptables -A INPUT -s <target-ip> -j DROP
iptables -A OUTPUT -d <target-ip> -j DROP

These rules block incoming and outgoing traffic from a specific IP address.

Process Failure

Simulate process death by killing a running application:

kill -9 $(pgrep myapp)

The kill command sends SIGKILL to the process, which terminates it immediately without cleanup.

Resource Exhaustion

Fill up memory or disk space to test how your system handles resource constraints.

To exhaust memory:

dd if=/dev/zero of=/tmp/fill bs=1M

This creates a large file in /tmp, consuming available disk space.

Running Experiments with Controlled Rollouts

Use Kubernetes jobs to run chaos experiments in a controlled environment. Define a job that runs a test script for a specific duration:

apiVersion: batch/v1
kind: Job
metadata:
  name: network-latency-job
spec:
  template:
    spec:
      containers:
      - name: chaos-runner
        image: busybox
        command: ["sh", "-c", "sleep 300"]
      restartPolicy: Never

This job runs for five minutes and allows time for monitoring to capture any impacts.

Monitoring During Chaos

Set up metrics collection during experiments using Prometheus or other monitoring tools. The following query targets latency data for a service:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="my-service"}[5m])) by (le))

This calculates the 95th percentile of HTTP request durations over a five-minute window.

Measuring Impact

Define acceptable thresholds before beginning an experiment. If requests exceed 1000ms response time more than 5% of the time, consider the experiment a failure.

Use tools like curl to measure latency:

curl -w "@curl-format.txt" -o /dev/null -s http://my-service/health

Where curl-format.txt contains:

     time_namelookup:  %{time_namelookup}\n
        time_connect:  %{time_connect}\n
     time_appconnect:  %{time_appconnect}\n
    time_pretransfer:  %{time_pretransfer}\n
       time_redirect:  %{time_redirect}\n
  time_starttransfer:  %{time_starttransfer}\n
                     time_total:  %{time_total}\n

This outputs timing details for each request.

Recovering from Failures

After injecting failures, ensure that systems recover automatically or within expected timeframes. For example, after dropping network connectivity, verify that:

kubectl get pods -o wide

Shows all pods in a healthy state.

Also check logs for error messages:

kubectl logs <pod-name> --previous

This retrieves logs from the previous instance of a pod, useful for diagnosing transient issues.

Handling Silent Failures

Silent failures are particularly dangerous. For example, when using iptables, the command may succeed even if it fails to apply rules correctly:

iptables -A INPUT -p tcp --dport 80 -j DROP

This adds a rule but does not verify that it was applied properly. To check:

iptables -L -n -v

This lists all active rules with packet and byte counters, allowing you to confirm that your rule is present and active.

Common Pitfalls

One common pitfall is confusing kill with kill -TERM. While kill sends SIGTERM (graceful shutdown), kill -9 sends SIGKILL (immediate termination). The difference matters for applications that handle shutdown signals:

# Graceful shutdown
kill $(pgrep myapp)

# Immediate shutdown
kill -9 $(pgrep myapp)

Another pitfall is applying network rules too broadly. For instance, blocking all traffic to an IP may disrupt unrelated services:

# Bad: blocks all outbound traffic
iptables -A OUTPUT -d 10.0.0.1 -j DROP

# Better: block only specific service traffic
iptables -A OUTPUT -p tcp -d 10.0.0.1 --dport 80 -j DROP

Logging and Reporting

Ensure that all chaos experiments log failures to a central system. A minimal logging script might look like this:

#!/bin/bash
echo "$(date): Chaos experiment started" >> /var/log/chaos.log
# ... experiment code ...
if [ $? -ne 0 ]; then
  echo "$(date): Chaos experiment failed" >> /var/log/chaos.log
fi

This writes timestamps and success/failure status to a log file.

Automating Experimentation

Use cron or orchestration tools to schedule chaos experiments:

# Run every weekday at 9 AM
0 9 * * 1-5 /usr/local/bin/run-chaos.sh

This executes a script daily during weekdays, ensuring consistent testing.

Conclusion

Chaos engineering helps validate system resilience through controlled failure injection. By using standard tools and clear metrics, you can build confidence in your systems' ability to handle real-world disruptions. The key is to define hypotheses, measure impact, and learn from both expected and unexpected outcomes.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.