Chaos Engineering in Practice

Orientation

Chaos engineering is the disciplined practice of introducing controlled failures into a running system to validate resilience, uncover hidden dependencies, and prevent outages before they reach users. It matters most in distributed, cloud‑native, or microservice environments where failure modes are complex, state is asynchronous, and a single node loss should not cascade. This page walks through what you actually do: how to define a hypothesis, design a safe experiment, inject failure with existing tooling, observe the response, and close the loop with remediation. Every example uses real commands and standard interfaces you can run today.

1. Articulate a Resilience Hypothesis

Begin each experiment with a single, falsifiable statement. Write it in the “if‑then‑therefore” form so that the outcome is unambiguous.

> Example – “If I kill the primary database pod, the application will route reads to a replica within 30 seconds without an HTTP 5xx spike for any active client request.”

> Example – “If I drain node worker‑3, all pods rescheduled on worker‑1 will remain healthy and serve traffic within the SLA.”

A good hypothesis names the failure mode, the component affected, the observable metric (HTTP status, pod ready‑state, response latency), and the acceptable behavior if the system holds. If the experiment disproves the hypothesis, you have a concrete failure to investigate; if it passes, you have evidence to document and potentially extend.

2. Build a Safe Experiment Scaffold

Before any live disruption, establish boundaries that prevent the experiment from becoming an outage.

Boundary	Why it matters	Real‑world guardrail
Namespace or label scope	Limits the failure to a known subset of workloads	`kubectl label pod app=checkout chaos‑scope=true` then target only `chaos‑scope=true`
Time window	Prevents experiments from running past their intended duration	Use a watch or `timeout` wrapper; e.g., `timeout 120s kubectl …`
Rollback / revert plan	Guarantees you can restore the pre‑state instantly	Document the exact `kubectl apply -f <manifest>.yaml` or `helm rollback release v1` command before starting
Dry‑run flag	Lets you verify the action without effect	Many tools offer a `--dry-run` or `--confirm` switch; if absent, script the action with `echo` first

Practical scaffold – Write a short bash script that (1) records the current state, (2) performs the failure, (3) waits for a defined observation window, (4) records the post‑state, and (5) exits with a clear status code. Keep the script under version control so the experiment is reproducible and auditable.

3. Inject Failure with Standard Tooling

Use the same CLI tools you already operate daily. The following commands are all from shipped, maintained products; no custom SDK or invented flag is required.

3.1 Pod deletion (Kubernetes)


# The --force flag removes the pod from the API server immediately;
# --grace-period=0 ensures no shutdown hook runs.
kubectl delete pod primary-db-0 \
  --grace-period=0 \
  --force \
  --wait=false

If the pod is managed by a ReplicaSet or Deployment, Kubernetes will immediately create a replacement. To test resilience *without* automatic replacement, target a pod that has replicas: 1 in its Deployment, or temporarily scale the Deployment down first:

# Scale the Deployment to 0 – no pods remain running.
kubectl scale deployment primary-db --replicas=0

# After the experiment, restore:
kubectl scale deployment primary-db --replicas=1

3.2 Node drain (Kubernetes)

Draining a node schedules all its pods to other nodes, then marks the node unschedulable. This is the standard path for hardware maintenance or node‑level failure simulation.

# Drain node worker-3, evicting pods with a 30‑second pod-termination grace period.
# --ignore-daemonsets skips DaemonSet-managed pods (they’ll be rescheduled automatically).
kubectl drain worker-3 \
  --ignore-daemonsets \
  --grace-period=30 \
  --delete-emptydir-data \
  --timeout=5m

To bring the node back:

kubectl uncordon worker-3

3.3 Container kill (Docker)

# Find the container ID first:
docker ps --filter "name=app-api" --format "{{.ID}}"

# Send SIGTERM; the container has its default STOPDIME to shut down gracefully.
docker container kill --signal=SIGTERM <container-id>

To force‑stop without graceful shutdown:

docker container kill --signal=SIGKILL <container-id>

3.4 HTTP health‑check (curl)

Use curl to verify that the application surface remains responsive or that the expected error surface appears.

# Record the HTTP status code into a variable for scripting.
status=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health)

# If the experiment expects a brief outage, you might see 503 or 504 during the window.
# A successful resilience test should return 200 once the failover completes.
echo "Health check returned: $status"

3.5 Combined script skeleton

#!/usr/bin/env bash
set -euo pipefail

APP_URL="http://localhost:8080/health"
EXPERIMENT="kill-primary-db"

# 1. Record pre‑experiment state
echo "=== Pre‑experiment ==="
curl -s -o /dev/null -w "%{http_code} $APP_URL" > /tmp/pre_status.txt

# 2. Inject failure – example: delete the primary DB pod
kubectl delete pod primary-db-0 --grace-period=0 --force --wait=false

# 3. Observe – wait up to 45 seconds for failover
timeout 45 bash -c '
  while ! curl -s -o /dev/null -w "%{http_code}" "$APP_URL" | grep -q "200"; do
    sleep 2
  done
'

# 4. Record post‑experiment state
echo "=== Post‑experiment ==="
curl -s -o /dev/null -w "%{http_code} $APP_URL" > /tmp/post_status.txt

# 5. Compare and exit with meaning
pre=$(cat /tmp/pre_status.txt)
post=$(cat /tmp/post_status.txt)
if [ "$pre" = "$post" ] && [ "$post" = "200" ]; then
  echo "Hypothesis supported: system recovered within the window."
  exit 0
else
  echo "Hypothesis challenged: pre=$pre post=$post"
  exit 1
fi

Make the script executable (chmod +x run_chaos.sh) and keep it in a dedicated chaos/ directory of your repo. Run it from a CI job, a temporary dev cluster, or a local Kind/K3s setup.

4. Observe, Record, and Validate

Capture enough data to answer the hypothesis without guesswork. The minimum record set includes:

Timestamp – date --iso-8601=seconds at start and end.
Cluster state – kubectl get pods -o wide before and after.
Application response – curl‑captured HTTP status code, and optionally latency via curl -w "%{time_total}".
Pod lifecycle events – kubectl get events --field-selector involvedObject.name=<pod-name> for the 2‑minute window around the failure.
Application logs – kubectl logs <pod> --since=2m to see if the app logged retries, circuit‑breaker trips, or error cascades.

Store these artifacts in a dated directory (logs/2025-08-20/) so that repeated experiments are comparable over time.

5. Iterate and Harden

Outcome	Next step
Hypothesis supported (system behaved as expected)	Document the experiment in a “resilience registry”. Add the failure mode to a runbook or automated test suite so it can be re‑run on future releases.
Hypothesis challenged (unexpected failure)	Investigate the root cause: was the failure mode broader than anticipated? Did a circuit breaker not trigger? Was state replication lagging? Update the hypothesis to a more specific version, adjust infrastructure (e.g., add a replica, increase timeout), and rerun.
Hypothesis disproved by experiment design (the tool didn’t actually disrupt the intended component)	Verify the scope label, check that the pod/node was actually alive when the command ran, and re‑run with tighter scoping.

Close the loop by merging the updated hypothesis and any infrastructure changes into your configuration repository. Treat the experiment record as a first‑class artifact—versioned, reviewed, and linked to the associated CI/CD pipeline or feature flag.

Sharp Edges – What Catches People Out

Edge	Why it silently fails	What you learn the hard way
Pod already gone	`kubectl delete pod` returns “not found” with exit code 0, but no pod was ever running.	Always `kubectl get pod <name>` first, or use `--force` only after confirming existence.
`kubectl drain` vs `kubectl evict`	`drain` marks the node unschedulable and evicts pods; `evict` (alpha/beta depending on version) only evicts without marking unschedulable. Confusing the two can leave a node thought to be draining while pods still run on it.	Use `drain` for node‑level experiments; reserve `evict` only when you specifically need to force‑remove a single pod without affecting the node’s schedulability.
DaemonSet immunity	`kubectl drain --ignore-daemonsets` suppresses DaemonSet pod eviction, but DaemonSets will immediately reschedule on another node, masking the disruption.	If you want to take a DaemonSet-managed daemon offline, scale its Deployment to 0 or use `kubectl patch daemonset <name> -p '{"spec":{"replicas":0}}'` temporarily.
Grace‑period mis‑match	Setting `--grace-period=0` on a pod that has a pre‑stop hook can cause the hook to never run, leaving the application in a half‑shut state and producing misleading “still healthy” pings.	Test the pod’s shutdown behavior locally (`docker run --rm … /entrypoint`) before relying on the grace period in a chaos experiment.
Chaos injector as single point of failure	If the script or CLI used to inject failure runs on the same host as the component being disrupted, a host‑level outage can mask the experiment’s intent.	Run the experiment from a separate host, a bastion, or a CI runner that communicates over the network exactly like a user would.
Observation window too short	Some failover mechanisms (DNS TTL expiry, BGP convergence, state‑sync replication) take minutes, not seconds. A 30‑second `timeout` will record a failure that would have self‑corrected.	Align the observation window with the slowest documented recovery time in your SLA, or instrument a “wait‑for‑condition” loop that polls until a desired state is reached or a hard deadline is hit.

Production‑time checklist

Run the experiment in a staging or dev cluster that mirrors production topology (same pod shapes, same service mesh, same DNS).
Have a human‑verified runbook open before issuing any kubectl delete or kubectl drain.
Monitor the control plane (kubectl get api‑services, kubectl get nodes) during the experiment; unexpected API throttling can masquerade as application failure.
Capture the full kubectl explain output of the resources you’re manipulating once, so you’re never guessing a flag’s semantics mid‑experiment.
After the experiment, run a “post‑mortem‑light”: diff the pre‑ and post‑cluster states, note any manual interventions you performed, and record the decision.

Chaos engineering, at its core, is a feedback loop: you propose a resilience claim, you test it, you observe the actual system response, and you close the loop with a concrete improvement. The commands above are the mechanical building blocks; the discipline—hypothesis, safety, observation, iteration—is what turns a one‑off disruption into a lasting increase in system confidence. Use them, record the results, and let the evidence speak louder than assumptions.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.