Chaos Testing Playbook

A practical guide to running chaos experiments in production. Covers experiment design, blast radius control, steady-state hypothesis, automation with tools like LitmusChaos, and the patterns that build confidence in system resilience.

Chaos Testing Playbook is a living guide for engineers who need to validate system resilience under real-world failure conditions. It is used when a service must survive network partitions, hardware crashes, slow databases, and cascading outages—especially in distributed systems where failure is not exceptional but inevitable. Use this playbook when onboarding new teams, after a major release, or when a system has grown complex enough that monitoring alone no longer reveals failure modes.

Planning Chaos Experiments

Define the scope and objectives of a chaos test before launching it. Start by identifying the critical user journey—a sequence of actions that must remain functional under stress. For example: “User logs in, selects a product, adds to cart, and checks out.” Then, determine the failure mode to simulate: network latency, CPU saturation, database downtime, or a combination. The experiment must be measurable: define success criteria such as “95% of checkout requests complete within 2 seconds” or “no data loss in the order database.”

Use a standard format for experiment documentation:

experiment:
  name: "Checkout Flow Under Network Latency"
  description: "Simulate 500ms latency on the product service during checkout."
  user_journey:
    - "User logs in"
    - "Navigates to product page"
    - "Adds item to cart"
    - "Proceeds to checkout"
    - "Completes payment"
  failure_mode:
    - "Network latency: 500ms on product service"
    - "Database: 10% of queries timeout after 3s"
  success_criteria:
    - "Average checkout duration ≤ 2s"
    - "Error rate < 5%"
    - "All orders persisted to database"
  duration: "30 minutes"
  environment: "production"
  tags: ["checkout", "network", "latency"]

This file is stored in experiments/checkout-latency.yaml and is versioned with Git. Each experiment must have a failure seed—a set of initial conditions that can be reproduced exactly. For instance, a database state snapshot or a set of pre-populated queues.

Executing Chaos Experiments

Run experiments using a chaos orchestrator that applies failures across services, nodes, and network layers. The orchestrator must be able to:

A minimal orchestrator uses kubectl and curl with a simple workflow:


kubectl apply -f experiments/checkout-latency.yaml

# Inject network latency using `tc` (traffic control) on a pod
kubectl exec -it product-service-7c9b5d4f8c -- \
  sh -c 'tc qdisc add dev eth0 root handle 1:0 htb default 10'
kubectl exec -it product-service-7c9b5d4f8c -- \
  sh -c 'tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100mbit'
kubectl exec -it product-service-7c9b5d4f8c -- \
  sh -c 'tc qdisc add dev eth0 parent 1:1 handle 10: netem delay 500ms'

The tc command uses Linux’s netem to add latency. The key is to set parent 1:1 handle 10: netem delay 500ms—not parent 1:1 handle 10: netem delay 500ms and parent 1:1 handle 10: netem delay 500ms. The handle must match the classid, and the dev eth0 must be correct for the pod’s network interface. Misconfiguring the dev or using eth1 instead of eth0 introduces silent failure: latency applies only to one interface, but the service is bound to 0.0.0.0:8080 on eth0.

To verify the experiment is active, check the tc configuration:

kubectl exec -it product-service-7c9b5d4f8c -- \
  sh -c 'tc qdisc show dev eth0'

This outputs:

qdisc htb 1: root refcnt 2 r2q 1000
 Sent 123456 packets 12345678 bytes 12345678 packets in 10 seconds
 qdisc netem at 1:1 parent 1:1
    delay 500.0ms

If delay is missing or 500ms is not listed, the experiment failed to apply.

Monitoring and Observing During Chaos

During an experiment, collect and correlate metrics, logs, and traces. Use a unified observation stack consisting of:

A well-structured log entry:

{
  "timestamp": "2024-03-15T12:34:56.789Z",
  "level": "error",
  "service": "checkout",
  "trace_id": "a1b2c3d4e5f6",
  "span_id": "f6e5d4c3b2a1",
  "message": "Payment service timeout",
  "duration_ms": 3124,
  "error": {
    "code": "TIMEOUT",
    "details": "Request to payment-service:8080/api/v1/payments timed out after 3s"
  }
}

The traceparent header must be propagated across services. It is a single string in the format:

traceparent: 00-0a0b0c0d0e0f0a0b0c0d0e0f0a0b0c0d-0a0b0c0d0e0f0a0b-01

The trace_id, span_id, and trace_flags are encoded in 32 hex characters, and the trace_flags byte (last two digits) must be 01 for sampled traces. A common failure: trace spans are created but not linked, because the traceparent header is not set on HTTP requests.

Use curl to verify trace propagation:

curl -H "traceparent: 00-0a0b0c0d0e0f0a0b0c0d0e0f0a0b0c0d-0a0b0c0d0e0f0a0b-01" \
     http://checkout-service:8080/api/v1/checkout

If the traceparent is missing or malformed, traces are fragmented, and the root cause of a failure is difficult to locate.

Responding to Failure Events

When a chaos experiment runs, define failure detection rules that trigger alerts. Use a standard alerting.yaml file:

alerts:
  - name: "Checkout Failed"
    query: |
      rate(http_request_duration_seconds{job="checkout-service"}[5m]) > 0.5
    condition: "above"
    threshold: 0.5
    severity: "critical"
    actions:
      - "send_slack_notification"
      - "capture_screenshot"
      - "record_video"
    labels:
      "experiment": "checkout-latency"
      "environment": "production"

This rule triggers when the checkout service’s average request duration exceeds 500ms over 5 minutes. The rate function is crucial: it computes the rate of requests per second, but it must be applied to a histogram metric. Using avg instead of rate leads to misaligned thresholds: a slow service might have high average duration but low request rate, missing the alert.

When an alert fires, the system must capture a full diagnostic bundle:

The bundle is stored as a tarball:

tar -czf diagnostic-bundle-$(date +%Y%m%d-%H%M%S).tar.gz \
  pods.yaml \
  logs/ \
  traces/ \
  screenshots/ \
  video.mp4

This bundle is the single source of truth for post-mortems.

Automating Chaos Experiments

Automate experiments using CI/CD pipelines. A typical pipeline.yml runs a chaos test on every merge to main:

name: "Chaos Test on Main"
on:
  push:
    branches: [ main ]

jobs:
  chaos-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to staging
        run: |
          kubectl apply -f k8s/
          sleep 60
      - name: Run chaos experiment
        run: |
          kubectl apply -f experiments/checkout-latency.yaml
          sleep 1800
          kubectl delete -f experiments/checkout-latency.yaml
      - name: Collect diagnostic bundle
        run: |
          kubectl exec -it product-service-7c9b5d4f8c -- \
            sh -c 'tc qdisc show dev eth0 > tc-config.txt'
          kubectl logs -l app=checkout-service --tail=100 > logs/checkout-service.log
          curl -s http://prometheus:9090/api/v1/query?query=rate(http_request_duration_seconds{job="checkout-service"}[5m]) > metrics/latency.json
          tar -czf diagnostic-bundle.tar.gz \
            tc-config.txt \
            logs/ \
            metrics/ \
            screenshots/
      - name: Upload bundle to S3
        run: |
          aws s3 cp diagnostic-bundle.tar.gz s3://chaos-experiments/main/

The sleep 1800 is critical: it ensures the experiment runs for 30 minutes. A common mistake is using sleep 60—the experiment runs only one minute, missing the full duration.

Managing Experiment Lifecycle

Each experiment has a lifecycle that includes:

  1. Design: write experiment spec in experiments/
  2. Preparation: deploy services, seed data, warm caches
  3. Execution: apply failures, monitor, record
  4. Observation: collect metrics, logs, traces
  5. Analysis: review results, compare to success criteria
  6. Documentation: write post-mortem, update playbook
  7. Archival: save diagnostics, metadata, and results

Use a standard post-mortem template:

# Post-Mortem: Checkout Flow Under Network Latency

## Summary
- Experiment ran from 2024-03-15T12:34:56Z to 2024-03-15T13:04:56Z
- Duration: 30 minutes
- Failure mode: 500ms latency on product service, 10% timeout on database
- Success: 92% of checkouts completed within 2s, error rate 6.3%

## Root Cause
- Payment service timeout due to slow database query
- Database query took 3.2s, but service timeout was 3s
- Tracing showed that `SELECT * FROM products WHERE id = ?` was missing an index

## Actions Taken
- Added index on `products(id)`
- Increased payment service timeout to 4s
- Added circuit breaker to product service

## Lessons Learned
- Network latency alone is insufficient; database performance must be monitored concurrently.
- Tracing is essential for identifying bottlenecks but is often underused.
- Diagnostic bundles are critical for onboarding and review.

This template is stored in post-mortems/, and each post-mortem is linked to the original experiment spec and diagnostic bundle.

Common Failure Modes and Pitfalls

These pitfalls are not discovered in design but emerge during execution. The playbook must document them explicitly so that engineers can avoid them in future experiments.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.