Chaos Engineering Framework

This framework defines how to systematically introduce controlled failure into systems to discover weaknesses and build resilience. It matters when you want to validate system behavior under stress, test incident response procedures, or confirm that your monitoring and alerting work correctly in realistic failure scenarios.

Define Your Experiment Scope

Before running any chaos experiment, establish what constitutes a valid failure mode for your system.

Identify Critical System Components

Start by documenting which services, databases, or network paths are essential to your application's core functionality. For each component, define:

• The expected behavior under normal conditions
• Key metrics that indicate healthy operation (latency, error rates, throughput)
• Dependencies that must remain stable for the component to function correctly

Use kubectl get pods -o json to capture current pod states and their resource usage when defining baseline expectations. You can also run kubectl top pods to check CPU and memory consumption across all running workloads.

Select Failure Types

Choose from these concrete failure modes:

• Network partition (e.g., using iptables rules)
• Process termination
• Resource exhaustion (CPU, memory, disk)
• Latency injection via network delays or application-level timeouts

Each failure type should have a clear definition of how it manifests and what constitutes success or failure for the experiment.

Configure Experiment Execution

Set Up Control Plane

Create a configuration file (chaos-config.yaml) that defines how experiments are executed:

experiment:
  name: "network-latency-injection"
  duration: 300s
  targets:
    - service: "user-api"
      namespace: "production"
  failure-mode:
    type: "latency"
    duration: 60s
    delay: 100ms

The configuration must specify how long each experiment runs and which system components to target.

Implement Experiment Orchestration

Use a standard container runtime like Docker to run chaos experiments. For example:

docker run --rm \
  --network=host \
  --name=chaos-experiment \
  my-chaos-image \
  --target-service=user-api \
  --latency=100ms \
  --duration=60s

This command runs a containerized chaos experiment with specific parameters. The container must expose the necessary CLI flags for configuration.

Monitor During Execution

During experiment execution, collect metrics from Prometheus endpoints or logs to ensure the failure is propagating correctly. You can use curl to query health checks:

curl -s http://user-api:8080/health

If the service returns a 503 status, it indicates that the experiment is affecting availability as intended.

Execute and Validate Experiments

Run Individual Chaos Events

Use your containerized chaos tool to inject failures into selected services. For instance, to simulate a network partition:

docker run --rm \
  --network=host \
  --name=network-partition \
  chaos-toolbox \
  --partition-service=user-api \
  --duration=120s

The tool must handle the actual injection of failures and clean up afterward.

Validate Failure Impact

After running an experiment, check that the failure affected the expected components. Use kubectl describe pod to verify that pods are handling disruptions:

kubectl describe pod user-api-7d5b8c9f4-xyz12

Look for events indicating restarts, node issues, or resource constraints.

Confirm Recovery Behavior

Verify that systems recover automatically without manual intervention. Check if:

• Load balancers route traffic away from failed services
• Auto-scaling groups add new instances when resources are exhausted
• Monitoring systems detect and alert on the failure conditions

Analyze Results and Iterate

Capture Metrics and Logs

After each experiment, collect logs from all affected components and store them with timestamps. Use tools like kubectl logs to extract pod information:

kubectl logs user-api-7d5b8c9f4-xyz12 --since=10m

Also save metrics from Prometheus endpoints to analyze performance degradation.

Document Outcomes

Record whether the system behaved as expected under failure conditions. If a component failed to handle a specific failure type, note:

• The time of failure
• The observed behavior
• Any logs or metrics that show how the system responded
• Whether recovery was automatic or required manual intervention

Refine Experiment Parameters

Based on results, adjust your experiment configuration. For example, if latency injection didn't cause sufficient degradation:

experiment:
  name: "network-latency-injection"
  duration: 300s
  targets:
    - service: "user-api"
      namespace: "production"
  failure-mode:
    type: "latency"
    duration: 60s
    delay: 250ms

Common Failure Modes and Their Detection

Network Latency Injection

Use standard Linux tools like tc (traffic control) to introduce delays:

tc qdisc add dev eth0 root netem delay 100ms

To remove the delay after an experiment:

tc qdisc del dev eth0 root

This approach works across most containerized environments and doesn't require special software.

Process Termination

Terminate processes by sending signals to running containers:

docker kill --signal=SIGTERM user-api-7d5b8c9f4-xyz12

Or use kubectl delete pod to trigger graceful shutdowns:

kubectl delete pod user-api-7d5b8c9f4-xyz12

Resource Exhaustion

Force memory or CPU usage by running resource-intensive processes inside containers. For example, to consume 500MB of memory:

docker run --rm \
  --memory=512m \
  busybox \
  sh -c "dd if=/dev/zero of=/tmp/data bs=1M count=512"

Sharp Edges and Gotchas

Silent Failures in Experiment Cleanup

Failing to remove network rules or clean up containers can cause subsequent experiments to behave incorrectly. Always verify that cleanup commands execute successfully:

tc qdisc del dev eth0 root 2>&1 | grep -q "no queueing discipline"

If this command fails, the next experiment may see unexpected network behavior.

Confusing CLI Flag Names

Some chaos tools use flags like --duration and --time-limit. The difference matters:

• --duration: How long to run the experiment
• --time-limit: Maximum time allowed for cleanup or recovery

Using the wrong flag can cause experiments to stop prematurely or not clean up properly.

Misunderstanding Failure Scope

Including too many services in a single experiment makes it hard to isolate root causes. Start with minimal impact experiments and gradually expand scope. For example, testing network latency on a single pod instead of an entire service reduces noise in results.

Monitoring Gaps

Some experiments may fail silently if monitoring is insufficient. For instance, if you inject latency but don't check response times, the experiment might appear successful even though services aren't handling delays properly. Always validate that failure conditions are detectable through your monitoring systems.

Automation Considerations

Automate chaos experiments using standard orchestration tools like cron or Kubernetes CronJobs. Example CronJob definition:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: periodic-chaos-experiment
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: chaos-runner
            image: my-chaos-image
            args:
            - "--target-service=user-api"
            - "--latency=100ms"
            - "--duration=60s"
          restartPolicy: OnFailure

This approach ensures consistent, repeatable experiments without manual intervention.

Integration with Incident Response

Design chaos experiments to align with your existing incident response workflows. When an experiment introduces a failure, make sure:

• Alerting systems trigger appropriately
• Teams receive notifications via the same channels used for production incidents
• Recovery procedures can be tested alongside failure injection

This integration ensures that chaos testing builds real operational readiness rather than just technical capability.

The framework supports both manual and automated execution. Manual runs are useful for ad-hoc testing, while automated schedules provide regular stress testing of system resilience.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.