Automated Change Management Workflow

Change management in engineering is the process of planning, testing, deploying, and validating code and infrastructure changes across environments. An automated change management workflow orchestrates this process end-to-end, ensuring consistency, traceability, and rollback capability with minimal human intervention. It is essential when teams deploy multiple services, maintain complex infrastructure, or operate under strict compliance requirements—especially when changes must be deployed across staging, production, and canary environments with zero downtime and full audit trails.

Triggering a Change

A change begins when a developer pushes code, updates configuration, or merges a pull request. The workflow is triggered by a git event: a push to main, a merge to develop, or a tag creation. The system watches for these events via a webhook, polling, or a message queue.

The initial trigger is a git push to origin main with a commit message containing [deploy]:

git commit -m "feat: add user profile endpoint [deploy]"
git push origin main

This generates a POST /webhook/git request to the change management service with the following JSON body:

{
  "ref": "refs/heads/main",
  "before": "a1b2c3d4e5f6",
  "after": "f6e5d4c3b2a1",
  "commits": [
    {
      "id": "f6e5d4c3b2a1",
      "message": "feat: add user profile endpoint [deploy]",
      "author": {
        "name": "Alice Chen",
        "email": "[email protected]"
      },
      "timestamp": "2023-10-05T14:22:01Z"
    }
  ],
  "repository": {
    "name": "user-service",
    "url": "https://git.company.com/user-service.git"
  }
}

The system inspects the message field for a [deploy] flag, and if present, creates a new change record in the workflow database.

Creating and Validating the Change

A change record is a structured entry in a database table named changes with the following schema:

column	type	description
id	uuid	unique change identifier
commit_hash	text	SHA-1 of the commit
status	text	`pending`, `validated`, `deployed`, `failed`
environment	text	`staging`, `production`, `canary`
triggered_by	text	`git`, `manual`, `cron`
created_at	timestamptz	when the change was created
started_at	timestamptz	when validation began
completed_at	timestamptz	when validation ended

When a change is created, the system runs a validation job using a validate command. This job runs a test script defined in the project’s test.sh file, which includes linting, unit testing, and integration testing.


#!/bin/bash
set -euo pipefail

echo "Running linting..."
npm run lint

echo "Running unit tests..."
npm run test:unit

echo "Running integration tests..."
docker-compose up -d db api
sleep 10
curl -f http://localhost:8080/health
npm run test:integration

echo "Running contract tests..."
npx pactum run tests/contract.spec.js

echo "All tests passed"

The validation job executes test.sh in a containerized environment using docker run with a Dockerfile that builds the application from source.

docker build -t user-service:latest .
docker run --rm \
  -v $(pwd):/app \
  -v /tmp/artifacts:/artifacts \
  -e TEST_ENV=staging \
  user-service:latest \
  /app/test.sh

If any test step fails, the job returns a non-zero exit code. The workflow captures the test output in a file at /artifacts/test.log. The validate command logs the result:

2023-10-05T14:23:15Z | INFO | Running linting
2023-10-05T14:23:16Z | INFO | Running unit tests
2023-10-05T14:23:18Z | ERROR | Integration test failed: curl: (22) The requested URL returned error: 503 Service Unavailable
2023-10-05T14:23:18Z | INFO | Running contract tests
2023-10-05T14:23:20Z | ERROR | Test failed: 3 of 10 contract assertions failed
2023-10-05T14:23:20Z | ERROR | Validation failed

The workflow updates the changes table with status = 'failed' and attaches the test log as a file resource.

Deploying to Staging

When validation passes, the workflow transitions the change to deploying and initiates a deployment to staging. The deployment is defined in a deploy.yaml file at the root of the project.

# deploy.yaml
version: 1
stages:
  - name: build
    image: node:18-alpine
    commands:
      - npm install
      - npm run build
    outputs:
      - dist/
      - build/
  - name: deploy
    image: alpine:latest
    commands:
      - apk add --no-cache curl jq
      - curl -X POST \
            -H "Content-Type: application/json" \
            -d '{"revision": "{{ commit_hash }}", "environment": "staging"}' \
            http://deployer.company.com/api/v1/deploy
    depends_on: [build]

The workflow engine reads deploy.yaml and runs each stage in sequence. The build stage runs inside a node:18-alpine container, installing dependencies and building the app. The deploy stage runs in alpine:latest, using curl and jq to call a deployment API.

The deploy API endpoint expects a JSON body with revision and environment. On success, it returns:

{
  "id": "d1234567-89ab-cdef-1234-56789abcdef0",
  "status": "accepted",
  "message": "Deployment queued for staging",
  "deployment_url": "https://deployer.company.com/deployments/d1234567-89ab-cdef-1234-56789abcdef0"
}

If the deployment API returns a 400 Bad Request, the workflow logs the error:

2023-10-05T14:24:30Z | ERROR | Deployment failed: HTTP 400
2023-10-05T14:24:30Z | ERROR | Response: {"error": "Invalid revision format", "details": ["revision must be a valid SHA-1"]}

The workflow reverts the change to failed status and sends a notification via POST /notifications/email with a template.

Promoting to Production

Once the staging deployment succeeds, the workflow waits for a manual approval or automatically promotes the change to production based on a promotion.yaml configuration.

# promotion.yaml
staging:
  auto_promote: true
  threshold:
    success_rate: 95
    duration_minutes: 30
  rollout:
    strategy: "blue-green"
    canary:
      - percent: 10
      - percent: 25
      - percent: 50
      - percent: 100

The workflow monitors the staging deployment using a GET /health?deployment_id=d1234567-89ab-cdef-1234-56789abcdef0 endpoint. It polls every 30 seconds and checks for:

HTTP 200 OK status
{"status": "healthy"} in the response body
95% of requests returning 2xx over a 30-minute window

When these conditions are met, the workflow triggers a blue-green deployment to production. It calls the deploy API with environment: production and strategy: blue-green.

The blue-green deployment works as follows:

The system identifies the current "blue" service (active) and prepares a "green" service with the new code.
It updates a load_balancer table with a target column, setting the green service as the new target.
After the switch, it performs a health check:

   curl -f http://lb.company.com/health

If the health check fails, the workflow rolls back by switching the load balancer back to the blue service.

Rollback is triggered by a POST /rollback request:

{
  "deployment_id": "d1234567-89ab-cdef-1234-56789abcdef0",
  "reason": "health check failed after 15 minutes",
  "from": "green",
  "to": "blue"
}

The system logs:

2023-10-05T14:30:00Z | INFO | Switching to green deployment
2023-10-05T14:30:05Z | ERROR | Health check failed: 503 Service Unavailable
2023-10-05T14:30:05Z | INFO | Rolling back to blue
2023-10-05T14:30:10Z | INFO | Rollback completed

Validating Production Deployment

After production deployment, the workflow runs a post-deploy validation job. This job checks:

The service is reachable at https://api.company.com/health
Database migrations have been applied
A sample request returns expected data

The validation script runs curl, psql, and jq:

#!/bin/bash
set -euo pipefail

echo "Checking health..."
curl -f http://api.company.com/health

echo "Verifying database migration..."
psql -U app -d user_db -c "SELECT version FROM schema_migrations ORDER BY version DESC LIMIT 1" | grep "20231005142200"

echo "Testing sample user retrieval..."
response=$(curl -s http://api.company.com/users/1)
echo "$response" | jq '.name == "Alice Chen" and .email == "[email protected]"'

echo "Post-deploy validation passed"

If the script exits with a non-zero code, the workflow updates the changes table with status = 'failed' and notifies the team via POST /notifications/slack.

Handling Failures and Retries

The workflow handles failures at every stage through a retry mechanism. Each job has a max_attempts and backoff configuration:

max_attempts: 3
backoff: exponential
initial_delay: 30s

When a job fails, the system enqueues a retry task. The retry task includes the job’s original arguments and metadata, such as the change_id and stage_name.

A job fails silently when:

The test.sh script exits with 1 but produces no output
The deploy API returns 201 Created but the deployment_url is not reachable
The psql command times out after 10 seconds without any output

The workflow logs these silent failures with a level: warning and a context key:

{
  "event": "job_failed",
  "job_id": "j1234567",
  "change_id": "c1234567",
  "stage": "post-deploy",
  "level": "warning",
  "context": {
    "timeout_seconds": 10,
    "command": "psql -U app -d user_db -c \"SELECT version FROM schema_migrations\"",
    "output": "",
    "error": "timeout"
  }
}

The workflow also supports manual retries. A user can trigger a retry via POST /change/{id}/retry with a reason and override_config.

{
  "reason": "manual retry after investigation",
  "override_config": {
    "test": {
      "timeout": "120s",
      "env": {
        "DATABASE_URL": "postgresql://prod-db:5432/user_db"
      }
    },
    "deploy": {
      "strategy": "rolling"
    }
  }
}

When a change is retried, the workflow updates the changes table with retried_at and retry_reason, and starts a new job with the overridden configuration.

Auditing and Reporting

Every change is audited in a change_events table, which logs every significant action:

event_type	change_id	timestamp	metadata
change_created	c1234567	2023-10-05T14:22:01Z	{"source": "git", "commit": "f6e5d4c3b2a1"}
validation_started	c1234567	2023-10-05T14:23:15Z	{"job_id": "j1234567"}
deployment_started	c1234567	2023-10-05T14:24:30Z	{"target": "staging", "deployment_id": "d1234567"}
promotion_requested	c1234567	2023-10-05T14:30:00Z	{"strategy": "blue-green"}

The workflow exposes a GET /changes/{id}/events endpoint that returns the audit trail:

[
  {
    "event_type": "change_created",
    "timestamp": "2023-10-05T14:22:01Z",
    "metadata": {
      "source": "git",
      "commit": "f6e5d4c3b2a1"
    }
  },
  {
    "event_type": "validation_started",
    "timestamp": "2023-10-05T14:23:15Z",
    "metadata": {
      "job_id": "j1234567"
    }
  },
  {
    "event_type": "validation_failed",
    "timestamp": "2023-10-05T14:23:20Z",
    "metadata": {
      "reason": "integration test failure",
      "log_url": "https://artifacts.company.com/test.log"
    }
  }
]

This audit trail enables traceability from code commit to production deployment, and serves as a foundation for incident response, compliance reporting, and retrospectives.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.