How to Automate Infrastructure Testing

Infrastructure testing ensures that cloud environments are not only deployed but also correct, consistent, and resilient. It matters when infrastructure is code—when a change to a cloud configuration must be validated before promotion to production, and when multiple teams maintain overlapping services across regions. This page details how to build and maintain automated infrastructure tests that verify infrastructure state, behavior, and performance, using real tools and practices that span most cloud environments.

Define Test Scope and Coverage

Infrastructure testing begins by answering: *What does “correct” mean for this infrastructure?* A test suite must cover three dimensions: state, behavior, and performance. State tests validate that infrastructure resources exist and match expected configuration. Behavior tests verify that systems interact correctly—networks route traffic, services respond, and data flows. Performance tests ensure that infrastructure meets latency, throughput, and capacity targets.

A typical test suite includes:

• Validation of resource creation (e.g., aws ec2 describe-instances output)
• End-to-end service validation via HTTP or gRPC
• Network connectivity checks (e.g., ping, traceroute)
• Health checks via health endpoints or logs
• Load and latency testing with synthetic traffic

Test scope should be defined in a single, versioned file. The most common format is a YAML file placed in the root of the infrastructure repository:

version: 1
targets:
  - name: staging
    environment: staging
    provider: aws
    region: us-east-1
    resources:
      - type: ec2
        name: web-server
        expected:
          ImageId: ami-12345678
          InstanceType: t3.large
          Tags:
            Environment: staging
            Owner: engineering
          SecurityGroups:
            - sg-abcdef12
      - type: rds
        name: primary-database
        expected:
          Engine: postgres
          DBInstanceClass: db.t3.medium
          AllocatedStorage: 20
          PubliclyAccessible: true
  - name: production
    environment: production
    provider: gcp
    region: europe-west1
    resources:
      - type: compute
        name: frontend
        expected:
          machineType: e2-medium
          tags: [web, production]
          metadata:
            startup-script: |
              #!/bin/bash
              apt update && apt install -y nginx
              systemctl enable nginx

This file drives a test runner that evaluates each resource against its expected configuration. When a resource fails, the runner produces a structured error:

{
  "test": "staging.web-server",
  "result": "failed",
  "details": {
    "actual": {
      "ImageId": "ami-12345678",
      "InstanceType": "t3.small",
      "Tags": { "Environment": "staging", "Owner": "engineering" },
      "SecurityGroups": [ "sg-abcdef12" ]
    },
    "expected": {
      "ImageId": "ami-12345678",
      "InstanceType": "t3.large",
      "Tags": { "Environment": "staging", "Owner": "engineering" },
      "SecurityGroups": [ "sg-abcdef12" ]
    },
    "diffs": [
      {
        "path": "InstanceType",
        "actual": "t3.small",
        "expected": "t3.large",
        "reason": "Instance type mismatch"
      }
    ]
  }
}

Implement State Verification with Infrastructure-as-Code Tools

State tests validate that deployed infrastructure matches a desired configuration. The most common approach uses the native query interface of infrastructure-as-code (IaC) tools: terraform, pulumi, cloudformation, bicep, or kubernetes manifests.

For a Terraform-based infrastructure, a test verifies that the state file matches expectations:

terraform plan -out=tfplan
terraform show -json tfplan > tfplan.json

The terraform show -json output is a structured representation of the planned infrastructure. A test script extracts values using jq:

#!/bin/bash
# test/state/terraform.sh

set -e

# Extract instance type from plan
INSTANCE_TYPE=$(jq -r '.planned_values.root_module.children[].values.instance_type' tfplan.json)

if [[ "$INSTANCE_TYPE" != "t3.large" ]]; then
  echo "Expected instance type: t3.large, got: $INSTANCE_TYPE"
  exit 1
fi

# Validate that a specific module exists
if ! jq -e '.planned_values.root_module.children[] | select(.address == "module.web")' tfplan.json > /dev/null; then
  echo "Module 'web' not found in plan"
  exit 1
fi

The terraform show -json output is deterministic and contains all values, including computed and interpolated ones. This enables fine-grained validation across IaC layers. A failure here often indicates a mismatch between code and deployment.

For Kubernetes, kubectl is the primary tool. A test validates that a deployment is rolled out correctly:

kubectl get deployment web-app -o json > deployment.json
kubectl get pods -l app=web-app -o wide > pods.txt

# Check that all pods are ready
if ! kubectl get pods -l app=web-app -o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}' | grep -v True; then
  echo "Not all pods are ready"
  exit 1
fi

# Check that replica count is correct
EXPECTED_REPLICAS=4
ACTUAL_REPLICAS=$(kubectl get deployment web-app -o jsonpath='{.spec.replicas}')
if [[ "$ACTUAL_REPLICAS" -ne "$EXPECTED_REPLICAS" ]]; then
  echo "Expected $EXPECTED_REPLICAS replicas, got $ACTUAL_REPLICAS"
  exit 1
fi

A common failure mode: the kubectl get pods command returns the *current* state, but the test expects *desired* state. Misunderstanding kubectl get output leads to flaky tests. The jsonpath syntax is easy to write but error-prone: kubectl uses @ for current object, @.path for property, and @.path[] for array iteration.

Validate Behavior with Integration and End-to-End Tests

Behavior tests verify that infrastructure works together. They use real network traffic, service calls, and data flow.

A typical setup uses curl, httpie, docker, and systemctl to simulate client behavior.

To validate that a web service is reachable and returns correct content:

#!/bin/bash
# test/behavior/web-service.sh

set -e

# Start a temporary HTTP client
docker run --rm --net=host alpine:latest \
  sh -c 'apk add --no-cache curl; \
         curl -f -s -H "Accept: application/json" http://web-app.staging.example.com/health'

# Check that the service returns a specific response
curl -s http://web-app.staging.example.com/api/v1/users > users.json

# Validate JSON structure
if ! jq -e '.users | length > 0' users.json; then
  echo "Expected at least one user, got empty response"
  exit 1
fi

if ! jq -e '.users[0].name | test("John")' users.json; then
  echo "Expected John in first user, got: $(jq -r '.users[0].name' users.json)"
  exit 1
fi

The curl -f flag is essential: it fails the test when the HTTP status code is 4xx or 5xx. The httpie tool offers a more expressive syntax, but curl is sufficient and universal.

For more complex behaviors, test scripts use systemctl to verify service state and logs:

# Check that the nginx service is active and running
if ! systemctl is-active --quiet nginx; then
  echo "nginx service is not active"
  exit 1
fi

# Check that a log file contains a specific message
if ! journalctl -u nginx --since "1 hour ago" | grep -q "Started nginx"; then
  echo "nginx did not start within the last hour"
  exit 1
fi

# Validate that a systemd service is configured correctly
if ! systemctl show nginx | grep -q "ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'" ; then
  echo "nginx ExecStart mismatch"
  exit 1
fi

A sharp edge: journalctl reads from the systemd journal, but logs written by nginx may be in /var/log/nginx/access.log. A test that checks only journalctl output will miss early startup logs. The solution: use journalctl for systemd logs and tail -f or grep for application logs.

Perform Performance and Load Testing

Performance tests measure infrastructure under load. They verify that services respond within latency targets, scale under traffic, and maintain throughput.

The most common approach uses ab (Apache Bench) or wrk to simulate HTTP traffic.

#!/bin/bash
# test/performance/load.sh

set -e

# Run 1000 requests over 30 seconds
ab -n 1000 -c 10 -t 30 http://web-app.staging.example.com/api/v1/users > ab-results.txt

# Parse results
LATENCY=$(awk '/Average/ {print $3}' ab-results.txt)
THROUGHPUT=$(awk '/Requests per second/ {print $4}' ab-results.txt)

if (( $(echo "$LATENCY > 200" | bc -l) )); then
  echo "Average latency ($LATENCY ms) exceeds 200ms"
  exit 1
fi

if (( $(echo "$THROUGHPUT < 15" | bc -l) )); then
  echo "Throughput ($THROUGHPUT req/sec) below target of 15"
  exit 1
fi

The ab tool produces a detailed report, including latency distribution, connection times, and error rates. A test that only checks average latency will miss tail latencies and outliers.

For more advanced performance testing, wrk is preferred for its high throughput and scripting support.

# Run wrk with a Lua script for dynamic requests
wrk -t4 -c100 -d30s -s test/performance/dynamic.lua http://web-app.staging.example.com

The Lua script can generate varied payloads, simulate user sessions, and validate response content:

-- test/performance/dynamic.lua
local http = require("http")
local json = require("json")

-- Simulate 50% of requests to /api/v1/users, 50% to /api/v1/posts
local path = math.random() < 0.5 and "/api/v1/users" or "/api/v1/posts"

-- Send request and validate response
local req = http.request("GET", path)
local res = http.request(req)

if res.status ~= 200 then
  print("Expected 200, got " .. res.status)
end

local data = json.decode(res.body)
if #data < 1 then
  print("Expected at least one item, got " .. #data)
end

A critical failure mode: wrk runs for a fixed duration, but the test expects steady-state metrics. The first 10 seconds of a 30-second run may be warm-up. The test must account for this by discarding early results or using wrk’s --duration and --warmup flags.

Integrate Tests into CI/CD Pipelines

Infrastructure tests are most valuable when executed automatically on every code change.

A typical CI pipeline uses git, bash, yaml, and docker to orchestrate the test suite.

# .github/workflows/test-infra.yml
name: Infrastructure Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: 1.5.0
      - name: Validate Terraform
        run: |
          terraform init
          terraform validate
      - name: Run State Tests
        run: |
          ./test/state/terraform.sh
      - name: Run Behavior Tests
        run: |
          ./test/behavior/web-service.sh
          ./test/behavior/database-connectivity.sh
      - name: Run Performance Tests
        run: |
          ./test/performance/load.sh
      - name: Upload Test Results
        uses: actions/upload-artifact@v3
        with:
          name: test-results
          path: |
            test/results/*.json
            test/results/*.txt

The actions/upload-artifact@v3 step collects test output and makes it available for review. A failed test appears immediately in the pull request, with logs and error details.

A key failure mode: the pipeline runs tests but does not fail the build on any test failure. The solution: use set -e in every script and exit 1 on any check. A script that returns 0 but logs an error is not sufficient.

Another edge: terraform apply generates a plan, but terraform show -json reads from terraform.tfstate. A test that runs terraform plan but does not terraform apply will validate the *planned* state, not the *applied* state. To validate applied state, the test must terraform apply first.

Maintain and Evolve the Test Suite

Infrastructure testing is not a one-time setup. Over time, tests grow, become brittle, and require refactoring.

A common practice is to run a test drift report: compare the current test configuration with the actual infrastructure state.

#!/bin/bash
# test/report/drift.sh

set -e

# Get current state from cloud provider
aws ec2 describe-instances --query 'Reservations[].Instances[].{InstanceId:InstanceId, ImageId:ImageId, InstanceType:InstanceType}' --output json > current.json

# Compare with expected configuration
jq -n --argjson expected "$(cat test/infrastructure.yaml)" '
  {
    drifts: (
      expected.targets[].resources
      | .[] as $r
      | $r.expected
      | . as $e
      | . as $i
      | [ $e | keys_unsorted ] as $keys
      | $keys
      | map({
          key: .,
          expected: $e[.],
          actual: $i[.]
        })
      | map(select(.expected != .actual))
    )
  }
' > drift.json

The report highlights differences between expected and actual configurations. When a test fails, the drift report helps determine whether the failure is due to a test configuration change or a real infrastructure change.

A mature test suite includes test metadata in a test/manifest.json file:

{
  "version": 1,
  "last-run": "2024-05-20T10:30:00Z",
  "total-tests": 24,
  "failed-tests": 3,
  "flaky-tests": [
    "test/behavior/web-service.sh",
    "test/performance/load.sh"
  ],
  "test-coverage": {
    "state": 0.95,
    "behavior": 0.82,
    "performance": 0.67
  }
}

This metadata enables continuous improvement: teams can track coverage, identify flaky tests, and prioritize test maintenance.

With these practices, infrastructure testing becomes not just a validation step, but a living, evolving part of the cloud architecture.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.