Change Management Sre | The Garnet Wiki

Change management in SRE is the disciplined practice of planning, executing, and evaluating infrastructure and system changes to minimize risk, reduce downtime, and ensure reliability. It matters most when a team deploys a new service, upgrades a database cluster, migrates data centers, or rolls out a new monitoring stack—changes that are complex, time-consuming, and high-impact. This is not change tracking; it is change *engineering*: a repeatable, measurable, and observability-driven process where every change is a first-class citizen in the system’s reliability lifecycle.

Planning a Change

A change begins with a clear intent: what problem it solves, what success looks like, and how it will be measured. The change proposal is written in a standard format—typically a markdown file in changes/ in the team’s git repository. Each change includes a change.yaml file at the root, with required keys:

title: "Migrate user service to Kubernetes with Istio"
description: |
  Replace the legacy user service running on EC2 instances with a Kubernetes-based deployment using Istio for service mesh.
  This migration will reduce latency, enable canary deployments, and improve observability.
  The new service will support 10K RPS with 99.9% availability.
  The change is expected to reduce user service P99 latency from 450ms to 250ms.

planned_start: "2025-04-15T08:00:00Z"
planned_end: "2025-04-15T10:00:00Z"
duration: "2 hours"
impact: "high"
risk: "medium"
owners:
  - "[email protected]"
  - "[email protected]"
dependencies:
  - "database-migration-2025"
  - "monitoring-stack-upgrade"

The impact and risk fields are drawn from a fixed list: none, low, medium, high, critical. The planned_start and planned_end use ISO 8601 with UTC timezone. A change is considered *in progress* when a git branch is created with the naming convention change/<id>/<title>, where <id> is a unique identifier (e.g., change/0047/migrate-user-service-to-k8s).

Before a change is approved, the team runs a pre-flight checklist. This is a script executed via kubectl and curl:


kubectl get pods -n user-service
curl -s http://monitoring.company.com/api/v1/health?service=user-service | jq '.status == "healthy"'

# Validate database readiness
psql -h db-prod -U sre -d users -c "SELECT COUNT(*) FROM users WHERE created_at > '2025-04-01';"

If any check fails, the change is blocked. The failure is logged in the changes/0047/001-pre-flight-checks.log file, with timestamps and exit codes.

Executing a Change

Execution begins when the change is marked as *ready to execute* in the change tracking system. This involves creating a change-run.yaml file in the change’s directory:

run_id: "2025-04-15-0800"
started_by: "[email protected]"
status: "running"
steps:
  - id: "01"
    name: "Deploy new user service to staging"
    command: |
      kubectl apply -f manifests/staging/
      kubectl rollout status deployment/user-service -n staging
    expected_output: "Deployment user-service successfully rolled out"
    timeout: 300
  - id: "02"
    name: "Run smoke tests"
    command: |
      curl -s http://staging.user-service.com/health | jq '.status == "UP"'
      kubectl logs deployment/user-service -n staging --since=1m
    expected_output: "All tests pass"
    timeout: 60
  - id: "03"
    name: "Promote to production"
    command: |
      kubectl apply -f manifests/production/
      kubectl rollout status deployment/user-service -n production --timeout=5m
    expected_output: "Deployment is ready"
    timeout: 300

The change execution is orchestrated by a change-runner service, a simple HTTP API built with Go and net/http. It polls a changes/ directory for new change-run.yaml files and executes steps sequentially. Each step is recorded in changes/0047/002-execution.log with:

Start and end timestamps
Exit code
Standard output and error
Duration
Whether it succeeded or failed

A step fails if:

The command exits with non-zero status
The output does not contain expected_output (exact match, case-sensitive)
The step exceeds timeout seconds

Failure causes the runner to halt and send a notification via curl to an alerting service:

curl -X POST http://alerting.company.com/api/v1/alert \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Change execution failed",
    "message": "Step 02: Run smoke tests failed on change 0047",
    "severity": "critical",
    "details": {
      "change_id": "0047",
      "step_id": "02",
      "failed_command": "curl -s http://staging.user-service.com/health | jq '.status == \"UP\"'",
      "output": "Expected \"UP\", got \"DOWN\"",
      "run_id": "2025-04-15-0800"
    }
  }'

Monitoring During Change

During change execution, the system must be monitored in real time. SREs use a standardized monitoring layout: a dashboard with three key panels:

Change Status: a live view of the current change’s progress, showing which steps are running, which have succeeded, and which have failed.

Service Metrics: a time-series graph of key SLOs, including:

Request latency (P90, P99)
Error rate (requests with status 4xx or 5xx)
Throughput (RPS)

Change-Specific Signals: custom metrics defined in the change.yaml:

slos:
  - name: "User service P99 latency"
    metric: "http.server.request.duration.p99"
    target: "250ms"
    window: "10m"
  - name: "Error rate"
    metric: "http.server.errors.total"
    target: "0.01"
    window: "5m"

The monitoring system (built with Prometheus and Grafana) polls these metrics every 30 seconds. If a metric deviates by more than 15% from the baseline, it triggers a *change event*:

{
  "change_id": "0047",
  "event_type": "metric_threshold",
  "metric": "http.server.request.duration.p99",
  "value": "310ms",
  "baseline": "250ms",
  "timestamp": "2025-04-15T08:18:45Z",
  "details": {
    "window": "10m",
    "threshold": "250ms",
    "alert_id": "alert-123"
  }
}

These events are logged in changes/0047/events.json, a growing list of JSON objects. The change runner monitors this file and pauses execution if a critical event occurs.

Post-Change Validation

After the change completes, a post-change validation phase begins. This phase runs a set of validation checks, defined in validation.yaml:

checks:
  - name: "Verify user creation endpoint"
    endpoint: "http://user-service.company.com/v1/users"
    method: "POST"
    body: {
      "name": "Test User",
      "email": "[email protected]"
    }
    expected_status: 201
    expected_body: {
      "id": 123,
      "name": "Test User"
    }
    timeout: 10
  - name: "Validate data consistency"
    query: "SELECT COUNT(*) FROM users WHERE created_at > '2025-04-15'"
    expected_result: 1500
    timeout: 30

Each check is executed by a validator service that makes HTTP requests or runs SQL queries. The results are written to changes/0047/003-validation-results.json:

[
  {
    "check_name": "Verify user creation endpoint",
    "status": "success",
    "latency": 124,
    "response": {
      "id": 123,
      "name": "Test User",
      "email": "[email protected]"
    }
  },
  {
    "check_name": "Validate data consistency",
    "status": "failed",
    "latency": 28,
    "error": "Expected 1500, got 1432"
  }
]

A change is marked as *successful* if all checks pass. If any check fails, the change is marked *partially successful*, and an alert is sent to the change owners.

Handling Change Failures

When a change fails, the team follows a standard failure protocol. The failure is recorded in changes/0047/failure.yaml:

failure_id: "fail-001"
timestamp: "2025-04-15T08:34:22Z"
reason: "P99 latency exceeded 350ms for 10 minutes"
step_id: "03"
details: |
  The rollout to production took 12 minutes, but the P99 latency spiked from 250ms to 380ms
  after the first 5 minutes. The logs show increased CPU usage on the Istio ingress gateway.
  A recent change to the cache warmup job may have caused the issue.

The change runner then runs a rollback procedure, defined in rollback.yaml:

steps:
  - id: "01"
    name: "Revert to previous version"
    command: |
      kubectl rollout undo deployment/user-service -n production
      kubectl rollout status deployment/user-service -n production
    timeout: 300
  - id: "02"
    name: "Validate rollback"
    command: |
      curl -s http://user-service.company.com/health | jq '.status == "UP"'
    timeout: 60

Rollback steps are executed immediately and logged. If rollback succeeds, the change is marked *rollback complete*. If it fails, a new failure event is generated, and the change is flagged for a post-mortem.

Change Tracking and Review

Every change is tracked in a central database table changes with the following schema:

id	title	status	started_at	ended_at	duration	owner_emails	metadata
0047	Migrate user service to Kubernetes with Istio	completed	2025-04-15T08:00:00Z	2025-04-15T10:15:30Z	135.5	[email protected],[email protected]	{"risk": "medium", "impact": "high"}

The metadata field is a JSON object, allowing arbitrary key-value pairs. The status field is one of: planned, ready, running, completed, failed, rollback, reviewing.

A change is reviewed when it is closed. The review is a review.yaml file in the change’s directory:

review_id: "rev-001"
reviewer: "[email protected]"
timestamp: "2025-04-15T11:00:00Z"
status: "approved"
comments: |
  - The change was well-documented and executed with precision.
  - Consider adding a canary phase before full rollout.
  - The validation checks were thorough.
  - Rollback procedure could be automated via a CI job.

Changes are archived after review, with all files and logs moved to archive/changes/0047/. The change is now a reference for future work.

Common Failure Modes

Step timeout without output: a step runs for 300 seconds but produces no output. The runner assumes success, but the step actually failed silently.
Missing expected_output in step result: the command outputs Success, but the expected_output is Success!. The check passes despite mismatch.
Misaligned SLO windows: a 10-minute window starts at 08:15 but the change began at 08:00. The first 15 minutes of data are not included in the SLO calculation.
Rollback not triggered: the rollback.yaml file is present, but the change runner does not detect it. The team must manually trigger rollback.
Change status not updated: after a step completes, the change-run.yaml file is updated, but the status field is not persisted to the database. The change appears as running even after completion.

Change management SRE is not a checklist. It is a continuous loop of planning, execution, monitoring, validation, and learning—where every change is an artifact, every failure a signal, and every success a foundation.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.