Infrastructure Drift Remediation

Infrastructure drift occurs when the actual state of a system diverges from its intended, declared configuration. This mismatch—between what is defined in code and what runs in production—accumulates over time due to manual changes, patching, scaling, and ad hoc operations. Infrastructure drift remediation is the automated process of detecting and correcting such deviations, ensuring systems remain aligned with their desired state. It matters most when systems grow complex and teams scale, especially in environments where infrastructure is managed via declarative tools like Terraform, Ansible, or Kubernetes.

Detecting Drift

Drift detection begins with comparing a system’s current state against its intended configuration. The most common approach is to compute a *delta* between a known desired state and the observed real state.

A typical drift detection workflow uses a snapshot of the target system, captured via a standard tool. For example, kubectl get all -o yaml exports the current state of a Kubernetes cluster. This output is compared against a reference configuration, such as a terraform plan -out=plan.tfplan file.

kubectl get all -A -o yaml > current-state.yaml

# Generate desired state from Terraform
terraform plan -out=plan.tfplan -refresh=true

# Compare current and desired states
diff -u <(yq '.resources[] | select(.type == "kubernetes_deployment") | .name' current-state.yaml) \
     <(yq '.resources[] | select(.type == "kubernetes_deployment") | .name' plan.tfplan)

The yq tool (from https://github.com/mikefarah/yq) parses JSON/YAML, and the diff command identifies discrepancies. A drift detection job logs changes in a structured format, such as a JSON file:

{
  "drift_detected": true,
  "timestamp": "2023-10-05T14:32:01Z",
  "changes": [
    {
      "resource": "Deployment/my-app",
      "type": "kubernetes_deployment",
      "expected": {
        "replicas": 3,
        "image": "myorg/app:v2.1",
        "ports": [8080]
      },
      "actual": {
        "replicas": 5,
        "image": "myorg/app:v2.0",
        "ports": [8080, 8081]
      },
      "details": [
        "Replicas mismatch: expected 3, got 5",
        "Image mismatch: expected myorg/app:v2.1, got myorg/app:v2.0",
        "Additional port 8081 not in desired state"
      ]
    }
  ]
}

Drift detection is not limited to configuration files. For instance, systemctl list-units --type=service --all --output=json captures the state of systemd services. A drift job may compare this output against a systemd unit file template stored in version control.

Remediation Execution

Once drift is detected, the system applies corrective actions. Remediation is typically driven by a script or job that executes a sequence of commands, often orchestrated via a CI/CD pipeline.

The most common remediation pattern is to apply a *patch* or *update* based on the drift report. This is done using a declarative tool that supports plan-and-apply workflows.

For example, when drift is detected in a Kubernetes cluster, a remediation job may use kubectl apply -f - to update resources:

#!/bin/bash
# remediate-drift.sh

set -euo pipefail

DRIFT_FILE="/tmp/drift-report.json"
OUTPUT_DIR="/tmp/remediation"

# Parse drift changes
jq -r '
  .changes[] |
  {
    name: .resource,
    type: .type,
    expected: .expected,
    actual: .actual
  } |
  .expected |
  .replicas as $replicas |
  .image as $image |
  .ports as $ports |
  [
    "kubectl scale deployment \(.name) --replicas=\($replicas)",
    "kubectl set image deployment \(.name) app=\($image)",
    "kubectl patch deployment \(.name) -p \(.ports | @json)"
  ] |
  .[]
' "$DRIFT_FILE" > "$OUTPUT_DIR/commands.sh"

# Execute all commands
bash "$OUTPUT_DIR/commands.sh"

# Record result
echo "Remediation completed at $(date)" >> "$OUTPUT_DIR/summary.log"

This script generates shell commands from the drift report and executes them. The commands are idempotent and safe to run multiple times.

Alternatively, a full remediation may use a declarative configuration tool. For instance, when using Terraform, a remediation job applies a terraform apply -auto-approve command using a plan.tfplan file generated by drift detection:

# After drift detection
terraform apply -auto-approve -input=false -state=prod.tfstate plan.tfplan

# Output includes:
# Plan: 12 to add, 3 to change, 4 to destroy
# Apply complete! Resources: 12 added, 3 changed, 4 destroyed
# Total time: 4m12s

Terraform’s plan.tfplan file is a binary-encoded state that includes full metadata about the desired state. The apply command reads this plan and applies it incrementally, ensuring atomic updates.

Drift Remediation Triggers

Remediation is not a one-off task. It must be triggered reliably by changes in the system. Common trigger mechanisms include:

• Scheduled jobs: Run drift detection every 15 minutes via cron.
• Event-driven: Triggered by an external event such as a new deployment or a service restart.
• On-demand: Run manually by an operator via a CLI or web interface.

A typical cron job runs drift detection and remediation daily:

# /etc/crontab
# Run drift detection and remediation every 15 minutes
*/15 * * * * root /opt/infra-drift/detect-and-remediate.sh

The script detect-and-remediate.sh performs the following:

#!/bin/bash
# detect-and-remediate.sh

set -euo pipefail

LOG_DIR="/var/log/infra-drift"
DATE=$(date +%Y-%m-%d_%H-%M-%S)

# Detect drift
detect-drift.sh > "$LOG_DIR/drift-$DATE.json"

# Check if drift is detected
if jq -e '.drift_detected' "$LOG_DIR/drift-$DATE.json"; then
  # Apply remediation
  remediate-drift.sh "$LOG_DIR/drift-$DATE.json"
  echo "Drift remediation applied at $DATE" >> "$LOG_DIR/summary.log"
fi

The detect-drift.sh script may use git diff to detect configuration changes, ps aux to capture process state, and docker inspect to gather container details.

For event-driven triggers, a service may listen to a message queue. For example, a drift-detector service listens to a Kafka topic named infra-events:

{
  "event": "service_started",
  "service": "app-server",
  "host": "prod-01",
  "timestamp": "2023-10-05T14:30:00Z"
}

Upon receiving such an event, the service runs systemctl is-active --wait app-server.service and compares the result against a known configuration. If the service is not running, it logs a drift event.

Failure Modes and Edge Cases

Several patterns silently fail in drift remediation workflows.

1. Drift Detection Fails to Capture All State

A common error is missing components in the state capture. For example, kubectl get all -o yaml includes only top-level resources but misses annotations and labels. A deployment may have a nodeSelector defined in the manifest, but kubectl get deployment my-app -o yaml does not include the nodeSelector field unless explicitly specified:

kubectl get deployment my-app -o yaml --show-labels --show-annotations

Failing to include these fields leads to false negatives: drift is missed because the nodeSelector is not in the current state.

2. Drift Remediation Overwrites Manual Changes

When a remediation job runs kubectl apply -f -, it overwrites the entire resource, including fields not in the desired configuration. For instance, a manual kubectl patch to add a tolerations field in a deployment is lost during a subsequent remediation.

To prevent this, remediation jobs should use kubectl apply with --prune and --force:

kubectl apply -f - --prune --prune-allow-missing-template-keys --force

The --prune flag ensures that resources not in the input file are deleted, while --force overwrites existing resources even if they are unchanged.

3. Drift Detection Runs Too Frequently

Running drift detection every 5 minutes may result in overlapping jobs. If a remediation job takes 3 minutes, a new detection job may start before the previous one finishes, leading to race conditions.

A solution is to use a lock file:

# Detect drift with file-based locking
LOCK_FILE="/tmp/dift-detection.lock"

if ! flock -x -w 300 "$LOCK_FILE"; then
  echo "Drift detection already running, skipping" >&2
  exit 1
fi

detect-drift.sh > "$LOG_DIR/drift-$(date +%s).json"

This prevents concurrent drift detection jobs from interfering.

4. Remediation Fails but Is Not Reported

A remediation job may succeed at the command level but fail silently at the infrastructure level. For example, terraform apply returns success, but a kubectl rollout status deployment/my-app fails due to a pod crash.

To catch such failures, remediation jobs should include post-apply validation:

#!/bin/bash
# remediate-drift.sh

# ... apply changes ...

# Validate deployment rollout
if ! kubectl rollout status deployment/my-app --timeout=300s; then
  echo "Deployment rollout failed" >&2
  exit 1
fi

# Validate service connectivity
if ! curl -f http://my-app-service:8080/health; then
  echo "Service health check failed" >&2
  exit 1
fi

echo "Remediation completed successfully"

Failure at any step halts the job and logs the error.

Drift Remediation in Practice

A typical drift remediation pipeline integrates detection, remediation, and validation into a single workflow. The following is a complete example using standard Unix tools:

#!/bin/bash
# /opt/infra-drift/pipeline.sh

set -euo pipefail

export PATH="/usr/local/bin:$PATH"

LOG_DIR="/var/log/infra-drift"
DATE=$(date +%Y-%m-%d_%H-%M-%S)
OUTPUT_FILE="$LOG_DIR/output-$DATE.json"

# 1. Detect drift
detect-drift.sh > "$OUTPUT_FILE"

# 2. Apply remediation if drift detected
if jq -e '.drift_detected' "$OUTPUT_FILE"; then
  remediate-drift.sh "$OUTPUT_FILE"
fi

# 3. Validate final state
validate-state.sh "$OUTPUT_FILE"

# 4. Record summary
echo "Pipeline completed at $(date)" >> "$LOG_DIR/summary.log"
echo "Output: $OUTPUT_FILE" >> "$LOG_DIR/summary.log"

The validate-state.sh script runs a series of health checks:

#!/bin/bash
# validate-state.sh

set -euo pipefail

DRIFT_FILE="$1"

# Check Kubernetes deployments
for DEPLOYMENT in $(jq -r '.changes[].resource | split("/") | .[1]' "$DRIFT_FILE"); do
  if ! kubectl rollout status deployment/$DEPLOYMENT --timeout=60s; then
    echo "Rollout failed for $DEPLOYMENT" >&2
    exit 1
  fi
done

# Check database connectivity
psql -h db-prod -U app_user -d app_db -c "SELECT 1;" > /dev/null || {
  echo "Database connection failed" >&2
  exit 1
}

# Check service endpoints
for SERVICE in $(jq -r '.changes[].resource | split("/") | .[1]' "$DRIFT_FILE"); do
  if ! curl -f "http://$SERVICE:8080/health"; then
    echo "Health check failed for $SERVICE" >&2
    exit 1
  fi
done

echo "All validations passed"

This pipeline ensures that drift is not only detected and corrected, but also verified to be effective.

Summary

Infrastructure drift remediation is the systematic process of identifying and correcting deviations between intended and actual system states. It relies on precise state capture, accurate delta computation, and idempotent correction actions. The most effective systems use a combination of declarative configuration tools, shell scripts, and validation checks, all orchestrated by a pipeline that runs continuously or on demand. The key to success lies in handling subtle failure modes: missing state fields, race conditions, silent failures, and incomplete validation. By combining detection, execution, and validation into a single, reliable workflow, teams can maintain infrastructure integrity across complex, evolving systems.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.