Infrastructure Testing Patterns

Test infrastructure code with the same rigor as application code. Covers Terraform plan testing, Pulumi unit tests, integration testing with real cloud resources, compliance testing, and the patterns that prevent infrastructure failures in production.

Orientation

When infrastructure changes touch networking, compute, or storage, a silent misconfiguration can propagate without immediate feedback. Infrastructure testing patterns give you automated guardrails that run every time code moves from repo to runtime. This page maps the patterns you can actually execute—no invented CLIs, no vendor-specific hooks, just the commands and config checks that real teams use to catch broken intent before it reaches users.

Validate syntax and intent before applying

The most immediate failure mode is pushing manifest that parses but doesn’t represent the desired state. A dry‑run against both the local filesystem and the API server catches this before any resources are touched.

kubectl apply --dry-run=client --dry-run=server -o yaml

In CI, a typical step looks like:

- name: Validate infrastructure manifest
  run: |
    set -euo pipefail
    kubectl apply --dry-run=client --dry-run=server -o yaml -f infra/manifests/ > /dev/null

If either dry‑run fails, the step exits non‑zero and the pipeline aborts, preventing a broken manifest from ever reaching apply.

Test deployed resources match intent

After apply, you need to confirm the cluster actually converged to the declared shape. A common pattern is to render the live state and diff it against the source-of-truth file.

Render live state

kubectl get all -n production -o yaml > live.yaml

Diff against source

diff -u infra/manifests/deployment.yaml live.yaml

If the manifests are identical, diff returns zero and exits silently. Any drift—extra labels, mismatched replica counts, changed image tags—appears as context lines prefixed with + or -, making the exact deviation visible without opening the YAML editor.

For schema‑level safety before the diff, kubectl explain can verify individual fields:

kubectl explain deployments.spec.replicas

This prints the field’s type, default, and validation rules. If the field has been removed or renamed in the installed Kubernetes version, explain returns Error: unable to introspect, flagging a version‑mismatch before you waste cycles on a full diff.

Detect drift and compliance after deploy

Drift isn’t always a YAML diff; sometimes a human operator changed a running service, or a pod restarted with a different environment. Standard tooling across OS and container runtimes gives you a repeatable check.

Service state

systemctl is-active myservice

Returns active, inactive, failed, or unknown. Paired with systemctl is-enabled myservice, you can verify both runtime state and persistence across reboots. A common pitfall: is-enabled checks the symlink in /etc/systemd/system, not whether the process is currently running. Use both flags in tandem.

Container runtime state

docker ps --filter "name=mycontainer"

Lists running containers matching the name. Add --filter "status=running" to exclude stopped containers. If the container was killed by the OOM killer, docker ps returns nothing, but docker logs mycontainer may show OOMKilled—a silent failure that is-active alone wouldn’t reveal.

Database connectivity

psql -U app -d prod -c "SELECT 1"

Exits 0 and prints SELECT 1 if the PostgreSQL server accepts the connection. If the server is down, the client returns psql: error: connection refused (or could not connect to server: Connection timed out). Exit code 2 indicates a connection failure, which you can capture in a shell script:

if ! psql -U app -d prod -c "SELECT 1" > /dev/null 2>&1; then
  echo "Database unreachable—blocking deployment"
  exit 1
fi

Gate changes in CI with real commands

Pipelines are the natural place to embed these checks. The goal is idempotent, fast feedback that blocks merge if the infrastructure state is unsatisfied.

Git pre‑commit hook

#!/usr/bin/env bash

set -euo pipefail

# Dry‑run the manifest that the commit touches
if ! kubectl apply --dry-run=client --dry-run=server -o yaml "$(git diff --cached --name-only | grep '\.yaml$' | head -1)" > /dev/null; then
  echo "⚠️ Manifest validation failed—commit blocked"
  exit 1
fi

# Verify a downstream service is reachable
if ! curl -sf http://localhost:8080/health > /dev/null 2>&1; then
  echo "⚠️ Health endpoint unreachable—commit blocked"
  exit 1
fi

Make it executable (chmod +x .git/hooks/pre-commit) and every local commit runs the guard before the object hits the staging area.

GitHub Actions step

- name: Validate K8s manifests
  run: |
    set -euo pipefail
    kubectl apply --dry-run=client --dry-run=server -o yaml -f ./k8s/ || exit 1

Add needs: deploy or if: github.ref == 'refs/heads/main' as needed. The step is pure shell—no custom SDK, no vendor‑locked syntax.

Validate running services with curl

Runtime health checks are the final boundary between “code deployed” and “service functional.” A one‑liner with explicit failure semantics is enough for most gate logic.

curl -sf --max-time 5 https://api.example.com/ready; echo "exit: $?"

A subtler edge: -s suppresses curl: (7) Failed to connect on stderr, so routing that output to a log requires 2>&1 or a dedicated stderr capture. In CI, set -e combined with curl -sf will abort the job on any non‑zero exit, but if you’re checking the exit code manually, remember that DNS failures also yield exit 22, indistinguishable from HTTP 5xx without extra parsing.

Sharp edges — what silently fails, what you only learn after it breaks

Dry‑run client vs server divergence --dry-run=client only validates local YAML syntax and known Kubernetes schema. It does not execute admission webhooks, validate custom resource constraints, or enforce namespace quotas. --dry-run=server hits the API server but may skip validation if the server is running a older kube-apiserver version or if the request hits a front‑proxy that strips custom headers. The two can disagree: a manifest may pass client but fail server because a required label was added by a webhook after the client check. Always run both, or at least one dry‑run followed by an actual apply --dry-run=server and inspect the returned object.

kubectl get vs kubectl apply --dry-run=server mismatch kubectl get deployment app -o yaml always returns the live desired state, even if the pod is stuck in Pending due to insufficient resources. apply --dry-run=server would reject the manifest if it specified replicas: 5 and the cluster can only schedule 3. Relying solely on get gives a false sense of convergence. Always pair live inspection with a dry‑run that enforces the intent.

curl -sf and silent DNS failures When the DNS name cannot be resolved, curl -sf exits 22 just like an HTTP 5xx. If your pipeline has both a health‑check step and a subsequent step that assumes the service is reachable, a DNS outage will block the pipeline at the health check, but downstream steps may also fail in ways that are harder to attribute. Explicitly log the exit code and, if possible, resolve the address beforehand with getent ahostsv4 <hostname> to distinguish DNS from application‑layer failures.

systemctl is-enabled and real‑time state is-enabled only checks the symlink state; it does not query systemd for the process status. A service can be enabled (symlink present) yet inactive (dead) or failed. Conversely, a service can be enabled and active after a manual systemctl start, but the symlink may have been removed by a configuration management run, causing the service to disappear on reboot. Pair is-enabled with is-active in the same command sequence, or use systemctl status and parse the Active: line.

Git pre‑commit hooks that assume a running environment A pre‑commit hook that runs kubectl apply --dry-run=server will succeed on a developer laptop with a local Kind cluster but may fail in CI where the API server is remote and requires authentication. Conversely, a hook that checks curl -sf http://localhost:8080/health will pass locally but break in CI where the service isn’t bound to localhost. Pin your hooks to the exact environment they’ll run in: use kind or a Docker‑in‑Docker runner for local validation, and inject CI‑scoped credentials (service accounts, token mounts) for remote checks.

Policy‑as‑code false positives on dynamic values Tools that evaluate YAML against OPA rules, Conftest policies, or tfsec checks often have rules that flag image: latest or hostPort: * as violations. In infrastructure where version pins are managed at a higher level, these flags are legitimate design choices, not misconfigurations. The sharp edge is that a policy that’s too strict will cause your CI to reject valid changes, and the team will either disable the policy or add suppressions—both of which erode the guardrail’s credibility. Write policies against anti‑patterns, not against your current implementation details, and version‑pin your policy bundles alongside your infrastructure code.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.