Configuration Management Automation

Configuration management automation is the practice of ensuring that a fleet of systems, containers, or virtual machines conform to a declared, version‑controlled desired state. Rather than manually logging into each host and running ad‑hoc commands, the operator maintains a source of truth—typically YAML, JSON, or a domain‑specific language—and applies it repeatedly and at scale. This matters when the number of hosts grows beyond a handful, when environments are ephemeral (e.g., CI/CD runners, Kubernetes namespaces), when compliance or audit requirements demand traceable change trails, or when disaster‑recovery procedures require reproducible stand‑up of infrastructure. The core promise is idempotence: applying the same definition multiple times produces the same result, and the system self‑corrects if drift occurs.

The following sections are organized around common reader tasks, each paired with concrete, real‑world commands and configuration patterns that exist in widely shipped tools. No vendor‑specific internals are exposed; everything maps to standard mechanisms you can translate to your own platform.

Version‑Controlling the Desired State

Every automation workflow begins with a declarative definition of what “configured” looks like. In practice this is a file checked into version control alongside application code.

Example: Git‑based YAML manifest for a web server

# files/nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.25-alpine
          ports:
            - containerPort: 80

To add this to a repository and push it:

git init
git add files/nginx-deployment.yaml
git commit -m "add nginx deployment manifest"
git remote add origin [email protected]:team/infra-config.git
git push -u origin main

Sharp edges: YAML anchors and aliases can silently merge or overwrite values when files are concatenated by a CI step. Always validate the merged output before applying. A common failure mode is a missing metadata.name field, which produces the exact error:

error: missing required field "metadata.name" in...

Applying Configuration to Targets

Once the desired state is in version control, the next task is to get it onto the targets. The pattern is typically: a controller reads the definition, compares it to the current state, and issues commands to close any gap.

Concrete: Applying the manifest with kubectl

# Dry‑run to see what would change, without actually mutating
kubectl apply -f files/nginx-deployment.yaml --dry-run=client --validate=strict

# Actual apply
kubectl apply -f files/nginx-deployment.yaml

If the manifest contains an unrecognized field, kubectl returns:

Error: failed to parse YAML: ...
error: unknown field "replicas" in io.k8s.api.apps.v1.DeploymentSpec

Sharp edges: --validate=strict rejects fields the API server doesn’t recognize, which is safe but can break pipelines that rely on forward‑compatible manifests. Omitting the flag lets the apply succeed but may result in silently ignored fields. The inverse—applying without validation—can leave the cluster in a partially configured state that only surfaces later through health checks.

Detecting Drift

Drift is the discrepancy between the declared state and the actual runtime state. Detecting it is essential before any remediation.

Concrete: Drift detection with kubectl diff

kubectl diff -f files/nginx-deployment.yaml

Output resembling:

--- 
+++ 
@@ -1,6 +1,7 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: nginx
-  labels:
-    app: nginx
+  labels:
+    app: nginx
+    version: v1
@@ -11,6 +12,7 @@
 spec:
   replicas: 3
   replicas: 3
   strategy:
     type: RollingUpdate
-  strategy:
-    type: RollingUpdate
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxSurge: 1

If drift has already occurred (e.g., someone manually scaled the deployment), the diff will show the actual current values versus the manifest.

Sharp edges: Not all platforms expose a diff command. On systems that only offer “apply”, you must first describe the current resource and compare outputs in a script. A subtle failure mode is timestamp‑based drift: a field that changes on every apply (such as metadata.generation) will always appear different, even though the meaningful configuration is identical. Filtering out meta‑fields is a common script step.

Automatic Remediation

When drift is detected, the next task is to reconcile the system back to the declared state. In a fully automated pipeline this happens without human intervention.

Concrete: Automatic remediation via ansible-playbook

ansible-playbook -i inventory.yml site.yml --check --diff

The --check flag (also called --no-op) simulates the run and reports changes. When piped into a CI job that auto‑approves, the same command without --check actually applies the changes. A typical error when a host is unreachable:

fatal: [web-01]: UNREACHABLE! => {
    "msg": "Failed to connect to the host via ssh.",
    "unreachable": true
}

Sharp edges: Running remediation without a dry‑run first is how production outages happen. A configuration key that means “apply only if current state differs” in one tool may mean “force replace” in another. For example, Ansible’s state: present versus state: latest have different idempotence semantics. Always inspect the --diff output before committing the run to production. Another silent failure: hosts: all combined with serial: 1 can cause the job to appear stuck if the inventory contains offline hosts; the play will retry those hosts indefinitely unless max_retries is set.

Promoting Changes Across Environments

Configuration often needs to move from development → staging → production. The mechanism is usually the same definition file with environment‑specific overrides.

Concrete: Using kustomize for layered overlays

# base deployment (checked into git)
kustomize edit set image nginx=nginx:1.25-alpine

# overlay for staging
kustomize create by default --resources ./base --namespace staging

# overlay for production
kustomize create by default --resources ./base --namespace prod

To render and apply the staging manifest:

kustomize build ./staging | kubectl apply -f -

To render and apply production:

kustomize build ./prod | kubectl apply -f -

Sharp edges: Kustomize overlays inherit all fields from the base, including metadata.uid and metadata.resourceVersion, which can cause AlreadyExists errors if the resource was already created with a different uid. The exact error:

Error: failed to create resource: ... already exists

To resolve, either force‑replace with kubectl replace --force or delete the existing resource first. Another common pitfall: secret keys referenced in the base but defined only in an overlay can cause key not found runtime errors if the overlay’s secretGenerator section is mis‑ordered.

Handling Secrets and Sensitive Data

Sensitive values (passwords, TLS keys, API tokens) must never appear in plain‑text manifests. The pattern is to reference external secret stores at apply time.

Concrete: Referencing a secret via kubectl and sealed-secrets

# Assume the secret already exists in the cluster
kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password=$(pass show db/password)

# Later, the manifest references it
apiVersion: v1
kind: Pod
spec:
  containers:
    - name: app
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password

To encrypt the secret for git storage:

sealed-secrets create-secret --name db-credentials \
  --from-literal=password=supersecret \
  --output yaml > secrets.yaml
git add secrets.yaml

Sharp edges: Sealed‑secrets uses an asymmetric key pair; the public key lives in the cluster, the private key is held off‑cluster. If the private key is lost, all sealed secrets become undecryptable, and the only recovery is re‑generation from the original plaintext (which must be retained securely). Another silent failure: kubectl apply on a sealed secret that has been modified offline but not re‑sealed results in a no‑op with no error, leaving the cluster running an outdated value.

Testing Before Application

Changes should be validated in a non‑production target before they touch production workloads.

Concrete: Linting and dry‑run with kubeval

kubeval files/nginx-deployment.yaml

If the manifest has a schema violation, kubeval exits with code 1 and prints:

Error: field "spec.replicas" must be an integer, string found

Concrete: ansible-playbook --check as a test gate

ansible-playbook -i inventory.yml site.yml --check > /tmp/playbook-check.log 2>&1
if grep -q "changed" /tmp/playbook-check.log; then
  echo "Changes detected; review before apply"
  cat /tmp/playbook-check.log
else
  echo "No changes; safe to proceed"
fi

Sharp edges: --check does not execute any state‑changing modules (e.g., file, service), so it will not detect drift in running services. It also cannot test connectivity or integration points. A common pattern is to run --check first, then apply to a staging cluster, run integration tests, and only then promote to production.

Recovery From Failed Apply

When an apply command fails part‑way through, the system may be left in a partially configured state. Knowing how to roll back or clean up is critical.

Concrete: Rolling back a Kubernetes deployment

kubectl rollout undo deployment/nginx

Output:

deployment "nginx" rolled back

To verify the rollback:

kubectl rollout status deployment/nginx

If the rollout is stuck, the exact error may be:

error: deployment "nginx" has timed out waiting for rollout to finish: "2 out of 3 old replicas are still running"

Sharp edges: kubectl rollout undo only goes back one revision. Multiple failures may require manual intervention, such as scaling the replica set down, deleting the faulty pod, and then scaling back up. Another silent failure mode: if the manifest contains a preStart hook that errors, the container enters CrashLoopBackOff, and kubectl rollout undo will not fix the underlying hook script. The operator must inspect kubectl describe pod <pod-name> to see the actual container exit status and error message.

Key Patterns Across the Workflow

Task	Common Command	Typical Failure Mode
Define desired state	git add … && git commit -m "…" && git push	Merge conflicts in YAML; missing required fields
Apply to target	kubectl apply -f manifest.yaml or ansible-playbook site.yml	Unknown API field; SSH unreachable on Ansible hosts
Detect drift	kubectl diff -f manifest.yaml or manual describe comparison	Timestamp‑based fields always differ; missing diff CLI
Remediate automatically	ansible-playbook site.yml (no --check)	Force‑replace vs. idempotent apply semantics
Promote across envs	`kustomize build …	kubectl apply -f -`	Inherited metadata.uid causing AlreadyExists
Handle secrets	kubectl create secret … or sealed-secrets	Lost private key → undecryptable secrets; plaintext leaked in git
Test before apply	kubeval manifest.yaml or ansible-playbook --check	--check skips service‑state modules
Recover from failure	kubectl rollout undo deployment/<name>	Only one revision back; hook‑script failures persist

Understanding these patterns—and the exact error texts and command flags that accompany them—lets an operator move from reactive firefighting to a predictable, repeatable configuration management lifecycle. The discipline of version‑controlling declarations, testing in isolation, and automated reconciliation pays off most sharply at scale, where manual steps become the bottleneck and the risk of configuration drift grows combinatorially.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.