Infrastructure Testing: Validating Before You Deploy

Infrastructure testing is the practice of validating infrastructure configurations, deployment pipelines, and runtime environments before promoting changes to production. It matters when infrastructure is code—when servers, networks, databases, and services are defined in version-controlled files and managed through automated processes. This validation ensures that changes to infrastructure, such as new EC2 instances, updated Kubernetes manifests, or modified IAM policies, behave as expected across staging, pre-production, and production environments.

Validating Configuration Changes

When a developer updates an nginx.conf file or modifies a Terraform module, infrastructure testing ensures that the new configuration applies correctly and integrates with existing systems. This starts with static analysis and progresses to runtime validation.

Syntax and Linting

A configuration file must be syntactically correct and follow established conventions. Use terraform fmt to enforce consistent formatting across teams.

terraform fmt -recursive

A terraform validate command checks for structural issues: required variables, missing modules, and correct schema types.

terraform validate -check-variables

For YAML-based configurations like Kubernetes manifests, use yq to validate schema compliance.

yq eval '.metadata.name' -r -I 1 -o - config/deployment.yaml

The kustomize build command combines overlays and base configurations, and can fail on missing fields or incorrect paths.

kustomize build --enable-helm .

Common failure modes include missing apiVersion, incorrect kind, or misaligned spec fields. A kubectl apply without a --dry-run flag often reveals schema errors that only surface at runtime.

State and Drift Detection

After applying configuration, compare the actual state of infrastructure with its desired state. Use terraform plan to generate a diff between current and target states.

terraform plan -out=tfplan -var-file=staging.tfvars

The output includes +, -, and ~ markers for additions, deletions, and modifications. A plan command with --refresh ensures the current state is up to date.

terraform plan -refresh -out=tfplan -var-file=prod.tfvars

For Kubernetes, kubectl diff compares a live cluster with a local manifest.

kubectl diff -f manifests/

Drift occurs when live infrastructure diverges from configuration. Detect drift using terraform state show to inspect a resource and terraform show to compare two state files.

terraform state show aws_instance.app_server

A common silent failure: terraform apply runs successfully but does not update all resources because of a missing depends_on in the configuration. A terraform plan with --detailed-exitcode reveals this.

terraform plan -detailed-exitcode -out=tfplan

Exit code 1 means changes were detected, exit code 2 means plan failed, and exit code 3 means both.

Validating Deployment Pipelines

A deployment pipeline applies infrastructure changes in stages. Testing ensures each stage executes correctly and transitions data and services as expected.

Pre-Deployment Validation

Before deploying, verify that all required dependencies are available and that the environment is ready.

curl -s -f http://healthcheck-api.example.com/health

Use pg_isready to confirm PostgreSQL is available.

pg_isready -h db.example.com -p 5432 -U app_user -d app_db

For services behind a load balancer, verify that a target is registered.

aws elbv2 describe-target-health --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/app-tg/abcd1234

A wait command ensures that a resource reaches a desired state.

aws ec2 wait instance-status-ok --instance-ids i-1234567890abcdef0

Failures in pre-deployment steps often stem from mismatched environment variables. Use env to verify the current environment.

env | grep -E '(AWS|DB|APP)'

A common mistake: kubectl apply runs on a cluster, but the kubeconfig file is incorrect or the current context is wrong.

kubectl config current-context
kubectl config use-context staging

During Deployment

During deployment, validate that changes are applied incrementally and that services remain available.

Use kubectl rollout status to wait for a deployment to stabilize.

kubectl rollout status deployment/app-web -w

The -w flag waits for the rollout to complete, and returns exit code 0 on success, 1 on timeout, and 2 on error.

For rolling updates, verify that a new pod is running and that the old one is still serving traffic.

kubectl get pods -l app=web --watch

A silent failure: terraform apply succeeds, but the terraform state file is not committed and pushed to version control. Use terraform state push to upload the updated state.

terraform state push s3://my-tf-state/terraform.tfstate

A terraform apply without -input=false can fail in CI if the pipeline expects a non-interactive session.

terraform apply -input=false -auto-approve -state-out=terraform.tfstate

Post-Deployment Validation

After deployment, test that the infrastructure is functional and integrated.

Run a curl command against a newly deployed service.

curl -s http://app.example.com/api/health | jq .status

A 200 response with {"status": "OK"} is expected. A 404 or 500 indicates a problem.

Use kubectl logs with --tail to inspect logs from a specific pod.

kubectl logs deployment/app-web --tail=100

To validate a database schema change, run a SQL query.

psql -h db.example.com -U app_user -d app_db -c "SELECT COUNT(*) FROM users;"

A common failure: a new database migration runs successfully, but the migration script is not idempotent. Use psql with --echo-queries to log every executed statement.

psql -h db.example.com -U app_user -d app_db -c "SELECT 1;" --echo-queries

Validating Runtime Behavior

Infrastructure is not just configuration—it must behave correctly under load. Runtime validation ensures that services are responsive, scalable, and resilient.

Health and Readiness Checks

Validate that services are alive and ready to serve traffic.

curl -f http://app.example.com/health

A 200 response indicates health. A 503 indicates unavailability.

Use systemctl is-active to check if a service is running on a Linux host.

systemctl is-active nginx

A service may be active but not running.

systemctl is-active --quiet nginx && echo "Running" || echo "Not running"

For Kubernetes, use kubectl wait to wait for a condition.

kubectl wait --for=condition=Ready pod/app-web-7b8d9c4f56 --timeout=30s

A common edge case: kubectl wait times out, but the pod is ready. The cause is often a mismatch between readinessProbe and livenessProbe durations.

Performance and Load Testing

After deployment, simulate real-world usage to validate performance.

Use ab (Apache Bench) to test a web endpoint.

ab -n 1000 -c 10 http://app.example.com/api/users

The output includes Requests per second, Time per request, and Latency. A 1000 requests with 10 concurrent connections is a standard load test.

Use wrk for higher-performance testing.

wrk -c 100 -t 4 -d 30s http://app.example.com/api/users

A failure mode: wrk reports high latency, but nginx logs show no errors. The cause is often an insufficient worker_connections in nginx.conf.

worker_connections 1024;

Use top and htop to monitor CPU and memory during a load test.

htop -d 1

A silent failure: ab runs, but the output is not parsed. Use awk or jq to extract metrics.

ab -n 1000 -c 10 -g results.csv http://app.example.com/api/users

The -g flag generates a CSV file with timing data. Parse it with awk.

awk -F',' 'NR>1 {sum+=$3} END {print "Avg: " sum/NR}' results.csv

Integration and End-to-End Testing

Validate that multiple services work together as expected.

Use curl to simulate a user journey.

curl -X POST -H "Content-Type: application/json" \
  -d '{"name": "Alice", "email": "[email protected]"}' \
  http://app.example.com/api/users

# Get the user
curl http://app.example.com/api/users/1

Use jq to extract and validate JSON responses.

curl http://app.example.com/api/users/1 | jq '.name == "Alice" and .email == "[email protected]"'

A common failure: a service returns a 201 Created, but the response body is missing a required field.

{
  "id": 1,
  "name": "Alice"
}

The email field is missing. Use jq to validate the entire shape.

curl http://app.example.com/api/users/1 | jq 'has("email") and .email | test("^.+@.+\..+$")'

For database-backed services, use psql to verify that data is correctly persisted.

psql -h db.example.com -U app_user -d app_db -c "SELECT id, name, email FROM users WHERE name = 'Alice';"

A silent failure: a service depends on a message queue, but no messages are processed. Use rabbitmqctl to inspect queues.

rabbitmqctl list_queues -p /app

Use kafka-topics.sh to list topics and check their status.

kafka-topics.sh --bootstrap-server kafka.example.com:9092 --list

A common edge case: kafka-topics.sh runs successfully, but the KAFKA_ADVERTISED_LISTENERS are not configured correctly. The result is consumers connecting to the wrong port.

Handling Failures and Feedback

Infrastructure testing must not only validate but also communicate results clearly.

Use exit codes to signal success or failure.

if curl -f http://healthcheck-api.example.com/health; then
  exit 0
else
  exit 1
fi

For complex pipelines, use set -euo pipefail to catch errors early.

set -euo pipefail

• e: exit on error
• u: undefined variables cause failure
• o pipefail: pipeline returns exit code of first failing command

A silent failure: pipefail is missing, and a grep command fails silently in a pipeline.

curl http://app.example.com/api/users | grep "Alice" | sort | uniq

With pipefail, if grep returns no results, the pipeline fails.

Use stderr to provide detailed error messages.

echo "Failed to apply Terraform changes" >&2
echo "Error: $(terraform apply -input=false -auto-approve -state-out=terraform.tfstate)" >&2

A common edge case: terraform apply runs, but stderr is not captured. Use 2>&1 to merge stdout and stderr.

terraform apply -input=false -auto-approve -state-out=terraform.tfstate 2>&1 | tee tf-apply.log

Finally, use notify commands to alert teams when tests fail.

if ! curl -f http://app.example.com/health; then
  echo "App health check failed" | mail -s "Infrastructure Test Failure" [email protected]
fi

Use slack or curl to post to a Slack channel.

curl -X POST -H "Content-Type: application/json" \
  -d '{"text": "Infrastructure test failed on staging"}' \
  https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

Failures in infrastructure testing often stem from missing or incorrect assumptions. The most common are:

• The configuration file is not in the expected location
• The environment variable is misspelled
• The database connection string is incorrect
• The current-context in kubectl is wrong
• The state file is not updated after a terraform apply
• The wait command times out but the resource is ready

Each of these is a silent failure that only becomes apparent in production.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.