Infrastructure Testing

Infrastructure testing verifies that systems operate correctly under expected conditions and fail gracefully when they don't. This applies to network connectivity, service availability, resource constraints, and state consistency across components. Testing infrastructure requires validation of both static configurations and dynamic behaviors in test environments that mirror production.

Validate Network Connectivity and Services

Use ping with specific flags to test basic reachability:

ping -c 3 -W 5 example.com

The -c flag controls packet count, while -W sets timeout in seconds. A timeout of 5 seconds is appropriate for most internal services.

For TCP port testing, use nc (netcat) with explicit protocol flags:

nc -zv -w 5 database.example.com 5432

The -z flag enables scan mode without data transfer, -v shows verbose output, and -w sets connection timeout. This command returns exit code 0 for success, 1 for connection timeout, or 2 for port unreachable.

Test HTTP endpoints with curl using specific headers:

curl -f -s -o /dev/null -w "%{http_code}" --connect-timeout 5 https://api.example.com/health

The -f flag makes curl fail silently on HTTP errors, -s suppresses progress meter, -o /dev/null discards response body, and -w "%{http_code}" outputs only the HTTP status code. The --connect-timeout flag sets connection timeout.

Test Resource Constraints and Limits

Monitor CPU usage with top or htop using specific flags:

top -p 1234 -n 5 -d 2

The -p flag specifies process ID, -n sets update count, and -d sets delay between updates in seconds. For automated testing, combine with grep to extract values:

top -b -n 1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1

The -b flag runs top in batch mode for scripting.

Test memory consumption using free with specific output format:

free -m | grep Mem | awk '{print $3}'

This command shows used memory in megabytes. For consistent testing, use vmstat to monitor system load:

vmstat 1 5 | tail -n +3 | head -n 1

The first 1 sets interval seconds, 5 sets sample count, and the pipeline filters out headers.

Validate Configuration Consistency

Compare running configuration with stored baseline using diff:

diff /etc/nginx/nginx.conf /tmp/baseline-nginx.conf

If files differ, the command outputs lines that differ. For service-specific validation, test reload without restart:

systemctl reload nginx && systemctl is-active nginx

This command reloads configuration and checks if service remains active. If reload fails, the service continues running with old config.

Validate database connection strings with psql:

psql -h db.example.com -p 5432 -U appuser -d appdb -c "SELECT version();" --set=ON_ERROR_STOP=on

The --set=ON_ERROR_STOP=on flag ensures command stops on any error. The -c flag executes a single command.

Test Service Availability and Dependencies

Check service health with custom endpoints:

curl -f -s -o /dev/null https://service.example.com/healthz

This endpoint must return HTTP 200 for healthy status. For more detailed checks, use:

curl -f -s -H "Accept: application/json" https://service.example.com/healthz | jq '.status'

The jq command parses JSON output to extract health status.

Test inter-service communication with a simple loopback test:

for i in {1..5}; do
  curl -f -s -o /dev/null --connect-timeout 3 http://service.example.com/api/echo
  if [ $? -eq 0 ]; then echo "Success on attempt $i"; break; fi
  sleep 1
done

This loop retries up to five times with one-second delays. If all attempts fail, the script exits with non-zero status.

Validate Storage and Filesystem Behavior

Check disk space usage:

df -h | grep '/$' | awk '{print $5}' | cut -d'%' -f1

This command shows percentage of root filesystem usage. For specific mount points:

df -h /var/lib/docker | awk 'NR==2 {print $5}'

Test write performance to storage:

dd if=/dev/zero of=testfile bs=1M count=100 oflag=direct

The oflag=direct flag bypasses OS cache, testing actual storage performance.

Test Containerized Environments

Validate container startup with docker inspect:

docker inspect --format='{{.State.Running}}' mycontainer

This command returns true when container is running. For health check validation:

docker inspect --format='{{.State.Health.Status}}' mycontainer

The health status can be healthy, unhealthy, or starting.

Check container logs for errors:

docker logs --tail=100 mycontainer | grep -i error

This command shows last 100 lines of logs and filters for error messages.

Validate Orchestration State

Test Kubernetes deployment health:

kubectl get pods -l app=myapp -o jsonpath='{.items[*].status.phase}'

This command returns pod phases like Running, Pending, or Failed. For readiness checks:

kubectl get pods -l app=myapp -o jsonpath='{.items[*].status.containerStatuses[*].ready}'

This shows readiness status of containers within pods.

Monitor service endpoints:

kubectl get svc myservice -o jsonpath='{.spec.ports[*].port}'

This returns port numbers exposed by the service.

Handle Test Failures and Silent Errors

Many tests silently fail when they should report failure. For example, ping without -W may hang indefinitely:

# BAD: Silent timeout
ping -c 3 example.com

# GOOD: Explicit timeout
ping -c 3 -W 5 example.com

The -W flag is essential for test reliability.

Network tools like nc do not fail on timeout unless explicitly configured:

# BAD: Silent success
nc -zv database.example.com 5432

# GOOD: Explicit error handling
if ! nc -zv -w 5 database.example.com 5432; then
  echo "Connection failed"
  exit 1
fi

Storage tests may pass even when they should fail:

# BAD: File size check only
df -h /var/lib/docker

# GOOD: Size and usage validation
if [ $(df -m /var/lib/docker | awk 'NR==2 {print $5}' | cut -d'%' -f1) -gt 90 ]; then
  echo "Disk usage over 90%"
  exit 1
fi

Monitor Test Output and Metrics

Use structured output for automated testing:

curl -f -s -o /dev/null -w "%{http_code} %{time_total}s" https://api.example.com/health

This command outputs both HTTP status code and total request time. For more detailed metrics:

curl -f -s -w "@curl-format.txt" https://api.example.com/health

Where curl-format.txt contains:

HTTP: %{http_code}
Time: %{time_total}s
Size: %{size_download} bytes

Test Configuration Reloads

Validate configuration changes without service interruption:

# Test nginx config syntax
nginx -t

# If syntax is good, reload
if [ $? -eq 0 ]; then
  systemctl reload nginx
fi

The nginx -t command validates configuration and returns exit code 0 for success. For services that do not support graceful reloads:

systemctl restart service-name

This command stops and starts the service, ensuring new configuration takes effect.

Validate Rollback Scenarios

Test rollback procedures before deployment:

# Save current config
cp /etc/service.conf /etc/service.conf.backup

# Apply test config
cp test-config.conf /etc/service.conf

# Test functionality
curl -f -s -o /dev/null https://service.example.com/healthz

# If test fails, restore backup
if [ $? -ne 0 ]; then
  cp /etc/service.conf.backup /etc/service.conf
  systemctl restart service-name
fi

This pattern ensures configuration changes can be reverted without downtime.

Infrastructure testing requires careful attention to timeouts, exit codes, and structured output. Tests must fail fast when they detect issues rather than silently continuing with incorrect assumptions. The key is to validate both static configurations and dynamic behaviors in ways that closely mirror production conditions.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.