Rollback Strategies | The Garnet Wiki

When a deployment introduces a defect that impacts service availability, user experience, or data integrity, rollback strategies provide the mechanism to restore a previous working state. These strategies are essential when a change cannot be fixed in-place or when immediate mitigation is required.

Restore from Known Good State

Git-based Rollback

Use git revert to create a new commit that undoes changes introduced by a previous commit:

git revert <commit-hash>

When reverting a merge commit, specify the parent:

git revert -m 1 <merge-commit-hash>

The command fails if conflicts exist. Use --no-commit to make changes without automatically committing.

Container Image Rollback

To roll back a deployment to a previous container image version:

kubectl set image deployment/<deployment-name> <container-name>=<old-image-tag>

This updates the deployment's spec. For immediate effect, use kubectl rollout restart:

kubectl rollout restart deployment/<deployment-name>

If rolling back to an image that is not present locally, ensure the registry still hosts the tag.

Database Schema and Data Changes

Rollback SQL Scripts

For database changes that require a rollback, maintain a separate script with inverse operations:

-- Example rollback script
BEGIN;
ALTER TABLE users DROP COLUMN last_login;
DROP INDEX idx_users_email;
COMMIT;

Use pg_dump to create a backup before major schema changes:

pg_dump -h <host> -U <user> <database> > backup_$(date +%Y%m%d_%H%M%S).sql

If a migration fails, the transaction must be rolled back manually. PostgreSQL will not automatically roll back failed migrations.

Versioned Data Migration

When data changes are part of a release, store the current schema version in a table:

CREATE TABLE schema_version (
  version TEXT PRIMARY KEY,
  applied_at TIMESTAMP DEFAULT NOW()
);

To rollback, apply the previous version's migration script and update the version table:

UPDATE schema_version SET version = '1.2.0' WHERE version = '1.3.0';

Configuration and Environment Changes

Rollback Configuration Updates

Use kubectl patch to modify a ConfigMap or Secret in place:

kubectl patch configmap/<config-name> -p '{"data":{"key":"old-value"}}'

To rollback an environment variable change, use kubectl set env with the old value:

kubectl set env deployment/<deployment-name> ENV_VAR=old_value

Configuration Management Rollback

For configuration stored in a repository, check out a previous commit containing known good settings:

git checkout <commit-hash> -- config/

This preserves the configuration files and allows for re-deployment.

Service Mesh and Network Changes

Rollback Traffic Routing

In Kubernetes, use kubectl patch to modify traffic rules in a service mesh:

kubectl patch virtualservice/<vs-name> -p '{"spec":{"http":[{"route":[{"destination":{"host":"<service>","subset":"<subset>"}}]}]}}'

If the service is already running with an incorrect configuration, update the configuration and restart the relevant pods:

kubectl rollout restart deployment/<deployment-name>

Infrastructure as Code Rollback

Terraform State Rollback

When infrastructure changes fail, use terraform state to revert the state file:

terraform state list
terraform state show <resource-name>
terraform state rm <resource-name>

To roll back a complete change, restore the previous state file from version control:

git checkout <commit-hash> -- terraform.tfstate
terraform apply

If the state file is corrupted, terraform state pull can help recover information.

Cloud Resource Rollback

For cloud resources that do not support atomic rollbacks, recreate them with known good parameters:

aws ec2 run-instances --image-id <ami-id> --count 1 --instance-type t3.micro

The rollback is only complete when the new resource is fully functional and tested.

Application Logic Rollback

Feature Flag Rollback

When a feature is controlled by a flag, disable it:

kubectl patch configmap/<config-name> -p '{"data":{"feature-flag":"false"}}'

For applications that do not reload configuration, restart the pods to pick up changes:

kubectl rollout restart deployment/<deployment-name>

Code Deployment Rollback

Use git checkout to revert to a previous commit and redeploy:

git checkout <commit-hash>
git push origin main

Ensure that the build pipeline re-deploys from this commit, not from a cached version.

Multi-Environment Rollback

Staging to Production Rollback

If production is in an inconsistent state after deployment, roll back from staging:

kubectl set image deployment/<staging-deployment> <container-name>=<staging-image>
kubectl rollout status deployment/<staging-deployment>

After confirming stability in staging, promote the same image to production:

kubectl set image deployment/<production-deployment> <container-name>=<staging-image>

Blue-Green Deployment Rollback

In a blue-green deployment, switch traffic back to the previous environment:

kubectl patch service/<service-name> -p '{"spec":{"selector":{"version":"blue"}}}'

If the current version is unstable, ensure all traffic routes are updated before proceeding.

Handling Rollback Failures

Partial Rollback Recovery

When a rollback partially succeeds, check for inconsistent states:

kubectl get pods
kubectl describe pod/<pod-name>

If some pods are in an error state, restart them manually:

kubectl delete pod/<pod-name>

Rollback Verification

After applying a rollback, run verification checks:

curl -f http://<service>/health
kubectl get deployment/<deployment-name> -o jsonpath='{.status.availableReplicas}'

If the service fails to respond, verify that all necessary dependencies are also restored.

Common Pitfalls and Silent Failures

Configuration Cache Invalidation

Configuration changes may not take effect if the application does not reload settings. For example, kubectl patch updates a ConfigMap, but if an application uses a cached copy, it will continue to use old values until restarted.

Rollback Order Dependencies

Rolling back services in the wrong order can cause cascading failures. Always rollback dependent services before their dependents:

kubectl rollout status deployment/<dependent-deployment>
kubectl rollout restart deployment/<dependent-deployment>

Image Tagging Errors

Using an incorrect image tag during rollback can lead to deploying an unintended version:

kubectl set image deployment/<deployment-name> <container-name>=<wrong-image-tag>

Ensure that the image tag points to a known good version before applying the change.

Rollback Script Failures

If a rollback script fails, it may leave the system in a partially consistent state. Always validate that all changes are applied correctly:

kubectl get configmap/<config-name> -o yaml | grep key

Emergency Rollback Procedures

Manual Override

In case of automated rollback failure, manually override configuration:

kubectl set env deployment/<deployment-name> ENV_VAR=old_value
kubectl set image deployment/<deployment-name> <container-name>=<image-tag>

Immediate Service Restoration

When service is down and rollback is not feasible, restore from backup:

pg_restore -h <host> -U <user> -d <database> backup_*.sql

This requires that backups are stored in a location accessible to the restoration process.

Rollback Testing

Automated Rollback Verification

Implement tests that verify rollback success:

kubectl rollout status deployment/<deployment-name>
kubectl get pods -l app=<app-name> | grep Running

Ensure that all pods are running with expected images and configurations.

Integration Test Rollback Validation

Run integration tests after a rollback to confirm functionality:

curl -X POST http://<service>/api/test-endpoint
kubectl logs deployment/<deployment-name>

The test should return success, and logs should not contain error messages related to the failed deployment.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.