Production-grade guide to infrastructure drift detection covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.
Infrastructure drift occurs when actual deployed resources differ from their declared configuration. This happens when changes are applied directly to infrastructure without updating source-of-truth files, or when automation fails to reconcile differences between state and reality. Drift detection identifies these inconsistencies before they cause outages.
Use kubectl diff to compare live resources with manifests stored in version control.
kubectl diff -f deployment.yamlThis command returns exit code 1 when drift is detected, with a diff output showing changes. The comparison uses the resource's live state versus what would be applied from the manifest.
When drift exists, kubectl diff shows differences like:
diff -u deployments/myapp-deployment.yaml deployments/myapp-deployment.yaml
--- deployments/myapp-deployment.yaml
+++ deployments/myapp-deployment.yaml
@@ -15,7 +15,7 @@
spec:
containers:
- image: myapp:v1.2.3
- name: myapp
+ name: myapp-container
ports:
- containerPort: 8080The command does not require cluster authentication if run from a context with kubectl config current-context set to a valid kubeconfig.
Helm charts can drift when values are changed directly in the cluster using helm upgrade --set without updating the values file.
To detect drift, compare the actual release values with the source:
helm get values myapp-releaseThis command outputs a YAML block containing the last applied values. If drift exists, it may show:
config:
database:
host: production-db.example.com
port: 5432
featureFlags:
enableLogging: trueIf helm upgrade is run without providing updated values, and changes were made manually in the cluster, this command shows what was last applied. A mismatch between this output and the repository's values.yaml indicates drift.
Use terraform plan to detect drift in Terraform-managed resources:
terraform planThis command will show changes that would be applied, including those due to manual modifications. A plan showing Plan: 0 to add, 0 to change, 1 to destroy indicates that a resource was deleted from the configuration but still exists in state.
When Terraform detects drift, it typically shows a message like:
Terraform will perform the following actions:
# module.db.aws_db_instance.main will be updated in-place
~ resource "aws_db_instance" "main" {
# ... (attributes unchanged)
~ allocated_storage = 20 -> 30
~ storage_type = "gp2" -> "io1"
}Terraform does not detect drift in resources that are not managed by the current configuration. The terraform plan command must be run from a directory containing the correct Terraform files.
Container image drift occurs when a deployment uses an image tag that differs from what is declared in source code or configuration.
Check deployed image tags using:
kubectl get pods -o jsonpath='{.items[*].spec.containers[*].image}'If this returns different values than expected, it indicates drift.
For example, if a deployment specifies nginx:1.21, but the running pods show nginx:latest, then drift exists.
Configuration files stored in Git repositories can drift when applied to infrastructure without version control tracking.
Compare local configuration with what is running:
kubectl get configmap myconfig -o yamlThis command outputs the current configmap. If this differs from what is in Git, drift exists.
Example output:
apiVersion: v1
kind: ConfigMap
metadata:
name: myconfig
data:
app.properties: |
server.port=8080
logging.level=INFOIf the app.properties content in this output does not match the file in Git, drift has occurred.
Network policy drift occurs when Kubernetes network rules are modified outside of declared policies.
Check current network policies:
kubectl get networkpolicies -ACompare with expected policy definitions. A missing policy or extra rule indicates drift.
Example output:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-internal
spec:
podSelector: {}
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: frontendIf a network policy exists in the cluster that is not defined in source code, drift has occurred.
Service account drift happens when permissions or annotations are modified directly on cluster objects.
Check service accounts:
kubectl get sa mysa -o yamlCompare with expected configuration. If additional annotations exist, drift is present.
Example output:
apiVersion: v1
kind: ServiceAccount
metadata:
name: mysa
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-roleIf this service account has extra annotations beyond what is expected, drift exists.
Use cloud provider APIs to compare actual resource configurations with expected definitions:
aws ec2 describe-instances --instance-ids i-1234567890abcdef0Compare the returned JSON output with expected configuration values. If an instance's tags differ from expected, drift exists.
Example response:
{
"Reservations": [
{
"Instances": [
{
"InstanceId": "i-1234567890abcdef0",
"Tags": [
{
"Key": "Name",
"Value": "web-server"
}
]
}
]
}
]
}If the Name tag is missing or incorrect, drift exists.
Use database client tools to check schema state against expected definitions:
psql -d mydb -c '\dt'This command lists all tables in the database. Compare this list with what is defined in source code or migration scripts.
If a table exists that is not declared in version-controlled migration files, drift exists.
Check load balancer attributes:
aws elbv2 describe-load-balancers --load-balancer-arns arn:aws:elb:us-west-2:123456789012:loadbalancer/app/mylb/1234567890123456Compare returned configuration with expected values. If a listener is added manually without updating infrastructure-as-code, drift exists.
Detect drift by comparing current state with source of truth:
git diff HEAD~1This command shows changes since the previous commit. If there are changes in infrastructure files but not in corresponding configuration files, drift may exist.
Include drift detection steps in pipeline definitions. Example using a bash script:
- name: Detect Drift
run: |
kubectl diff -f deployment.yaml
if [ $? -eq 1 ]; then
echo "Drift detected"
exit 1
fiThis step fails the pipeline if drift exists.
Configuration management tools like Puppet or Chef can detect drift by comparing current system state with desired state:
puppet resource package nginxThis command shows all installed packages and their versions. Compare this output with expected configuration to detect drift.
Check container image tags in registries:
docker inspect myapp:latestCompare the returned manifest with expected image metadata. If the digest differs from what is declared, drift exists.
Resources that are updated in-place without creating new objects can silently drift. For example, updating a ConfigMap's data field directly:
kubectl patch configmap myconfig --patch '{"data":{"key":"value"}}'This command modifies the existing resource without changing the source of truth, causing drift.
When environments are managed with different configurations, drift can occur if a change is applied to one environment but not others. For example:
kubectl apply -f config.yaml -n production
kubectl apply -f config.yaml -n stagingIf the config.yaml file has different content in each environment, drift occurs between environments.
kubectl diff only compares manifests against live resources. It does not detect drift in:
When multiple deployments modify the same namespace, drift can occur if one deployment's changes are overwritten by another:
kubectl apply -f deployment.yaml
kubectl apply -f service.yamlIf both files target the same namespace but were created independently, drift may exist.
When infrastructure is managed with immutable deployment strategies, drift detection becomes more critical. If a rolling update modifies resources that should remain static, drift occurs.
Example: A deployment's labels are modified directly:
kubectl label deployment myapp environment=productionThis change modifies the running deployment but does not update the source configuration file.
Secrets can drift when they are updated directly in the cluster:
kubectl get secret mysecret -o yamlIf this output differs from the version-controlled secret definition, drift exists. Secrets are often stored in base64 format, so compare encoded values carefully.
In systems where multiple pipelines can modify infrastructure, drift can occur when different processes update resources simultaneously:
helm upgrade --set image.tag=1.2.3 myapp ./chartThis command may introduce drift if the chart's default values differ from what is expected.
Schedule regular checks using cron jobs or CI/CD pipelines:
#!/bin/bash
kubectl diff -f deployment.yaml || exit 1Configure automation to detect and fix drift automatically:
- name: Auto-Detect Drift
run: |
kubectl diff -f . | grep -q "diff" && echo "Drift detected" && exit 1Ensure consistency across environments by running checks in each:
for env in staging production; do
kubectl diff -f deployment.yaml -n $env
doneThis approach ensures that all environments match the expected configuration.
Drift detection can produce false positives when:
Use kubectl diff with --output=json to get structured output for more precise comparisons.
This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy