Change Management Sre Practices

This page documents specific operational practices for managing changes in production systems, focusing on how to safely introduce modifications while minimizing risk and maintaining system stability. These are the techniques used when deploying code, configuration updates, infrastructure changes, or operational procedures that affect live services.

When Changes Matter

Changes in production environments require careful coordination to avoid unintended consequences. This includes any modification that can impact service availability, performance, data integrity, or user experience. The goal is to make these modifications predictable and reversible while maintaining confidence in system behavior.

How to Prepare for a Change

1. Create a Deployment Plan

Before executing a change, document the scope and rollback strategy. Include:

• What exactly will be modified
• Expected duration of the change
• Rollback procedure
• Test cases that verify correctness
• Communication plan for stakeholders

Use standard version control systems like Git to track changes. For example, create a branch with a descriptive name:

git checkout -b feature/upgrade-kubernetes-version

2. Validate Change Through Testing

Run tests in an environment that closely mirrors production. Use the same configuration files and service definitions. If using Kubernetes, validate manifests with:

kubectl apply --dry-run=client -f deployment.yaml

This command checks whether the YAML would apply correctly without actually making changes.

3. Set Up Monitoring Before Deployment

Configure alerts for critical metrics before applying any change. For instance, if you're updating a database schema, set up monitoring for:

• Query latency
• Connection pool usage
• Disk space
• Error rates

Use Prometheus or similar systems to define alert rules that trigger during deployment windows.

How to Execute a Change

1. Deploy Using Blue-Green or Rolling Updates

When deploying applications, prefer methods that allow immediate rollback:

Blue-Green Deployment Example

In a blue-green setup, update the inactive environment first:

kubectl set image deployment/myapp blue=mycompany/myapp:v2
kubectl rollout status deployment/myapp

Then switch traffic to the updated version:

kubectl patch service/myapp -p '{"spec":{"selector":{"version":"blue"}}}'

Rolling Update Example

For rolling updates, ensure that you do not exceed maximum unavailable pods:

kubectl set image deployment/myapp myapp=mycompany/myapp:v2
kubectl rollout status deployment/myapp --timeout=10m

If a deployment fails, roll back using:

kubectl rollout undo deployment/myapp

2. Perform Health Checks

After deploying, verify that services are responding correctly. Run health checks manually or programmatically. Example:

curl -f http://myapp.example.com/healthz

The -f flag makes curl exit with error code 22 if the server returns a 4xx or 5xx status.

3. Monitor for Anomalies

Check system metrics for unusual behavior post-deployment:

• CPU and memory usage
• Request latency
• Error counts
• Database connection states

How to Roll Back When Needed

1. Identify the Problem Quickly

When issues appear after deployment, check logs and error messages. Use standard tools like kubectl logs or system journal:

kubectl logs -l app=myapp --since=10m

This retrieves logs from the last ten minutes for pods labeled with app=myapp.

2. Revert to Known Good State

Revert using either versioned deployment or rollback commands:

kubectl rollout undo deployment/myapp --to-revision=1

Or reapply a previous configuration file:

kubectl apply -f deployment-v1.yaml

How to Handle Configuration Changes

1. Version Control Configuration Files

Treat all configuration as code. Store in version control systems like Git. Use consistent naming and directory structures.

Example of applying a new config map:

kubectl apply -f configmap.yaml
kubectl rollout status deployment/myapp

If the change fails, revert with:

kubectl apply -f configmap-old.yaml
kubectl rollout status deployment/myapp

2. Validate Configuration Before Applying

Use kubectl diff to compare desired state with actual state:

kubectl diff -f configmap.yaml

This command shows exactly what changes will be made without applying them.

How to Handle Infrastructure Changes

1. Use Infrastructure-as-Code Tools

Apply infrastructure changes through declarative tools like Terraform or Kubernetes manifests. Never manually modify infrastructure in production.

Example Terraform apply:

terraform apply -auto-approve

This command applies all changes from the current configuration without interactive confirmation.

2. Validate Before Applying

Before applying, preview what will change:

terraform plan

Check for any unexpected resource replacements or deletions.

How to Handle Multi-Step Changes

1. Break Large Changes into Smaller Units

Divide complex changes into incremental steps with clear boundaries. Each step should be independently verifiable.

Example: Instead of changing both database schema and application logic in one change, do:

1. Update schema
2. Test schema access
3. Deploy application with new schema usage

2. Use Feature Flags for Gradual Rollouts

Implement feature flags to enable changes gradually. This allows you to disable features if issues arise.

Example flag definition in configuration:

feature_flags:
  new_ui: true
  experimental_api: false

Common Failure Modes and Sharp Edges

1. Silent Failures in Rollbacks

Rollbacks may appear successful but leave behind partial state. For example, a Kubernetes deployment may complete, but some pods remain in a broken state due to failed readiness checks.

Use kubectl describe to inspect the deployment:

kubectl describe deployment/myapp

Look for events related to pod creation or health check failures.

2. Confusion Between Rollout and Rollback Commands

The commands kubectl rollout undo and kubectl rollout status are often confused. The undo command rolls back a deployment, while status shows the current state of the rollout process.

3. Incomplete Configuration Updates

Applying configuration changes without verifying that they were applied correctly can leave systems in an inconsistent state. Always validate with:

kubectl get configmap myconfig -o yaml

Compare this output to the expected configuration.

4. Overlooking Dependencies

Changes might not account for dependencies on other services or resources. Check for dependency issues before deployment by running:

kubectl get pods -l app=myapp --show-labels

Look for pods in CrashLoopBackOff or Pending states, which indicate dependency problems.

5. Race Conditions During Deployments

Concurrent deployments can cause conflicts. Ensure that only one change is active at a time by using deployment locks or coordinating with other teams.

6. Misunderstanding Dry-Run Behavior

The --dry-run=client flag does not check the server-side behavior. It only validates syntax and structure. Always follow dry runs with actual deployments.

Communication During Changes

1. Announce Changes in Advance

Send notifications to relevant teams before starting changes. Include:

• Expected start time
• Duration of change
• Affected services
• Rollback plan

2. Provide Real-Time Updates

During deployment, update stakeholders on progress and any unexpected issues.

Tools and Patterns

1. Use Standard CLI Commands for Consistency

Stick to commands that are available across different environments:

• kubectl apply for applying manifests
• kubectl rollout status for checking progress
• kubectl logs for retrieving logs
• kubectl diff for previewing changes

2. Automate Where Possible

Automate repetitive steps like rolling updates, health checks, and rollback procedures to reduce human error.

Summary of Key Practices

Changes in production should follow these principles:

1. Document everything
2. Test thoroughly before applying
3. Validate after applying
4. Monitor closely during and after deployment
5. Have clear rollback paths
6. Communicate clearly with all stakeholders

These practices ensure that changes are predictable, safe, and recoverable. They minimize the chance of unexpected failures while maintaining system stability and performance.

The key is to treat every change as a potential source of instability and prepare accordingly. The patterns described here work across platforms and tools because they rely on standard interfaces and well-understood mechanisms rather than vendor-specific features or configuration schemas.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.