Chaos Engineering Advanced

This page covers advanced techniques for designing and executing chaos experiments beyond basic failure injection. It addresses the specific needs of teams that have moved past simple outage simulations to intentional system stress testing, service degradation workflows, and complex multi-stage experiment orchestration.

Configure Complex Experiment Workflows

Define Multi-Stage Experiment Sequences

Chaos experiments can be structured as sequences of orchestrated stages that progress through different failure modes. For example:

experiment:
  name: "multi-stage-service-degradation"
  stages:
    - name: "baseline"
      duration: "30s"
      actions:
        - type: "http_request"
          target: "http://service-a:8080/health"
    - name: "latency-injection"
      duration: "2m"
      actions:
        - type: "network_delay"
          target: "service-b"
          delay: "100ms"
    - name: "resource-consumption"
      duration: "5m"
      actions:
        - type: "cpu_stress"
          target: "service-c"
          percentage: 80

The duration field controls how long each stage runs, while actions define the specific failure mechanisms applied. The sequence executes stages in order, with transitions between stages handled by a control loop.

Combine Failure Modes Across Services

A single experiment can introduce multiple failure types simultaneously across different services:

experiment:
  name: "distributed-system-stress"
  actions:
    - type: "network_partition"
      target: "service-a"
      partitions:
        - "frontend"
        - "backend"
    - type: "memory_leak"
      target: "service-b"
      rate: "1kb/sec"
    - type: "database_connection_pool_starvation"
      target: "service-c"
      connections: 200

Each action operates independently, allowing for realistic multi-vector failure scenarios. When actions overlap in scope, they must be coordinated to prevent conflicting behaviors.

Execute Controlled Chaos

Implement Experiment Rollback Strategies

Experiments should include explicit rollback steps that execute when a stage fails or exceeds thresholds:

experiment:
  name: "high-traffic-degradation"
  stages:
    - name: "stress-test"
      duration: "10m"
      actions:
        - type: "load_generator"
          target: "service-a"
          rps: 1000
      rollback:
        - type: "scale_down"
          target: "service-a"
          replicas: 1

Rollback steps must be idempotent and fast. If rollback fails, the system remains in a degraded state without clear recovery path.

Use External Triggers for Experiment Control

Experiments may require external signals to begin or end:

# Start experiment using HTTP API
curl -X POST http://chaos-engineering-api/v1/experiments/trigger \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"experiment": "high-traffic-degradation"}'

# Cancel running experiment
curl -X DELETE http://chaos-engineering-api/v1/experiments/active \
  -H "Authorization: Bearer $TOKEN"

The API endpoint returns a unique experiment ID. The system maintains an active experiment state until explicitly stopped or reaches timeout.

Validate Experiment Results Against Baselines

After each stage, validate whether observed behavior matches expected patterns:

# Query for metrics during experiment
curl http://monitoring-api/v1/metrics?query=rate(http_requests_total[5m]) \
  -H "Authorization: Bearer $TOKEN" | grep service-a

# Compare to baseline values
curl http://monitoring-api/v1/metrics?query=http_request_duration_seconds \
  -H "Authorization: Bearer $TOKEN" \
  -d 'labels={"service":"service-a", "experiment": "baseline"}'

If metrics deviate from expected ranges, the experiment should fail with a clear error message indicating which metric exceeded threshold.

Manage Experiment Dependencies

Coordinate Service State Changes

Experiments that affect multiple services must account for interdependencies:

experiment:
  name: "database-sharding-failure"
  dependencies:
    - service: "database-primary"
      state: "healthy"
    - service: "cache-layer"
      state: "ready"
  actions:
    - type: "database_write_failure"
      target: "database-primary"
      failure_rate: 0.1

The system validates that dependent services are in required states before starting the experiment. If a dependency fails, the experiment halts with error code 409.

Handle Service Restart Side Effects

Some experiments involve restarting services to test recovery behavior:

# Restart service and wait for readiness
kubectl rollout restart deployment/service-b
kubectl rollout status deployment/service-b

# Monitor until ready
kubectl get pods -l app=service-b -o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}'

If a service fails to restart within 30 seconds, the experiment fails with ServiceRestartTimeout error.

Control Experiment Scope and Duration

Limit Impact to Specific Environments

Control which environments can run experiments:

experiment:
  name: "staging-resource-stress"
  environment: "staging"
  targets:
    - service: "service-a"
      namespace: "frontend"
    - service: "service-b"
      namespace: "backend"

The system validates that the current environment matches the experiment's target. If not, it returns error code 403 with message "Environment mismatch".

Set Dynamic Timeouts

Configure timeout behavior based on workload characteristics:

experiment:
  name: "long-running-async-workload"
  timeout: "1h"
  grace_period: "5m"
  actions:
    - type: "job_queue_delay"
      target: "worker-pool"
      delay: "30s"

The grace_period defines how long to wait before declaring failure after timeout. If the experiment times out and grace period expires, it triggers cleanup procedures.

Adjust Failure Rate Based on Traffic Patterns

Modify failure injection rate dynamically:

# Query current traffic volume
curl http://monitoring-api/v1/metrics?query=rate(http_requests_total[1m]) \
  -H "Authorization: Bearer $TOKEN" | jq '.[] | select(.service == "service-a")'

# Adjust failure rate based on current load
if [ "$current_rps" -gt 100 ]; then
  echo "Setting failure rate to 5%"
else
  echo "Setting failure rate to 20%"
fi

The experiment must be responsive to real-time metrics. If the system is already under high load, it should reduce failure injection intensity.

Monitor and Log Experiment Events

Capture Detailed Event Logs

Each action in an experiment generates structured logs:

{
  "timestamp": "2023-10-01T10:00:00Z",
  "experiment_id": "abc123",
  "stage": "latency-injection",
  "action": {
    "type": "network_delay",
    "target": "service-b",
    "delay": "100ms"
  },
  "status": "success"
}

Logs are stored in a central logging system with fields for experiment ID, stage name, and action details.

Report Failures with Context

When actions fail, the system provides actionable error information:

# Example failed action response
{
  "error": "NetworkDelayInjectionFailed",
  "message": "Failed to inject delay on service-b: connection timeout",
  "experiment_id": "abc123",
  "stage": "latency-injection",
  "target": "service-b"
}

The error message includes both the immediate failure and context about the experiment state. If a critical action fails, it stops the entire experiment.

Handle Experiment Failures

Define Critical vs Non-Critical Failures

Some failures are acceptable during experimentation:

experiment:
  name: "non-critical-service-outage"
  actions:
    - type: "service_down"
      target: "service-a"
      critical: false
    - type: "database_connection_failure"
      target: "service-b"
      critical: true

If a non-critical action fails, the experiment continues. If a critical action fails, it immediately halts execution.

Implement Retry Logic for Transient Failures

Transient failures during experiments should retry:

# Retry logic for network failure injection
max_retries=3
retry_count=0
while [ $retry_count -lt $max_retries ]; do
  if curl -X POST http://chaos-engineering-api/v1/actions/inject \
    -d '{"type": "network_delay", "target": "service-b", "delay": "50ms"}'; then
    break
  else
    retry_count=$((retry_count + 1))
    sleep 1
  fi
done

If retries exceed maximum attempts, the experiment logs failure and proceeds.

Ensure Cleanup After Experiment Termination

Upon experiment completion, ensure proper cleanup:

# Example cleanup procedure
kubectl delete job --selector experiment=abc123
kubectl rollout status deployment/service-a
kubectl get pods -l app=service-a -o jsonpath='{.items[*].status.phase}'

Cleanup must be idempotent. If cleanup runs twice, it should not produce errors or duplicate actions.

Advanced Experiment Orchestration

Run Experiments in Parallel Across Clusters

Multiple experiments can run simultaneously:

experiment:
  name: "multi-cluster-stress"
  clusters:
    - name: "us-west-1"
      experiment: "high-traffic-degradation"
    - name: "eu-central-1"
      experiment: "high-traffic-degradation"

Each cluster runs its own instance of the experiment. Coordinators must ensure parallelism doesn't overload individual clusters.

Schedule Repeated Experiment Runs

Automate experiment execution on a schedule:

# Cron job to run weekly
0 0 * * 1 /usr/local/bin/chaos-engineer run --experiment "weekly-stress-test"

Scheduled experiments should include timestamp in their logs and metrics for tracking over time.

Integrate With Alerting Systems

Experiments can trigger alerts when failures exceed thresholds:

experiment:
  name: "critical-path-stress"
  alerting:
    - threshold: 0.1
      metric: "http_request_error_rate"
      action: "send_slack_notification"

If error rate exceeds 10%, a Slack notification is sent to the on-call team. Alerting must be configurable per experiment.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.