Design automated CI/CD pipelines that build, test, and deploy reliably. Covers pipeline architecture, stage design, parallel execution, artifact management, and rollback automation.
CI/CD pipeline automation patterns are the recurring, repeatable strategies for orchestrating the build, test, and deployment of code across environments with minimal human intervention. They matter when a team ships multiple times per day, needs consistent release processes, and must catch regressions early. These patterns solve the problem of how to reliably and efficiently move code from commit to production, regardless of the underlying tooling.
When a pipeline runs frequently—multiple times per hour—reducing build time is critical. Incremental builds compute only the changes since the last build, avoiding full rebuilds. The key is tracking file fingerprints and dependencies. A pipeline starts by computing a checksum over the source tree and a list of all dependency files (e.g., package-lock.json, pom.xml, go.sum). This checksum is used as a cache key.
The pipeline first attempts to restore the build cache from a remote storage (S3, GCS, or a shared filesystem) using the cache key. If a cache exists, it is restored to the build directory. Then, the build step runs with the restored dependencies and compiled artifacts. After the build, the pipeline computes a new cache key from the updated dependency files and uploads the new build artifacts and cache to storage.
CACHE_KEY=$( \
echo -n "$(git rev-parse HEAD)" \
&& echo -n "$(sha256sum package-lock.json pom.xml go.sum)" \
| sha256sum \
| cut -d' ' -f1 \
)
# Restore cache
curl -H "Authorization: Bearer $CACHE_TOKEN" \
-o /tmp/build-cache.tar.gz \
"https://cache.example.com/v1/artifacts/$CACHE_KEY"
tar -xzf /tmp/build-cache.tar.gz -C /build
# Run build
npm run build
make build
# Upload new cache
tar -czf /tmp/new-cache.tar.gz -C /build .
curl -H "Authorization: Bearer $CACHE_TOKEN" \
-X PUT \
--data-binary @/tmp/new-cache.tar.gz \
"https://cache.example.com/v1/artifacts/$CACHE_KEY"The failure mode: a dependency file changes but the checksum computation skips a file. The build uses outdated dependencies, and a regression goes unnoticed. This happens when a build script forgets to include package-lock.json in the checksum, even though it is updated by npm install.
Running all tests in a monorepo or large codebase can take 30 minutes. To reduce this, tests are split across multiple runners—test sharding. Each shard runs a subset of the test suite, and results are aggregated.
A test shard is determined by the total number of shards and the current shard index. The shard index is passed via environment variable, and the test runner selects a subset of test files based on this index.
# In a pipeline step
export SHARD_INDEX=1
export TOTAL_SHARDS=4
# Select test files for this shard
find tests/ -name "test_*.js" | \
sort -V | \
awk -v index="$SHARD_INDEX" -v total="$TOTAL_SHARDS" '
{
n = NR % total
if (n == index) print $0
}' > test-list.txt
# Run only selected tests
jest --config jest.config.js --testFilePattern test-list.txtThe pipeline must ensure all test shards report results to a common location. A JSON file per shard is uploaded to a shared storage, and a final aggregation step combines them.
{
"shard": 1,
"duration": 124.3,
"passed": 48,
"failed": 2,
"errors": [
{
"test": "user-login.spec.js",
"error": "TimeoutError: 5000ms exceeded",
"stack": "at userLogin.test.js:123"
}
]
}A silent failure: if a test fails on one shard but the overall pipeline passes, the failure is missed unless the aggregation step explicitly checks for any failure. Another edge: the test list is built from find, but find returns relative paths, and the test runner expects absolute paths. The test runner fails to locate test files when test-list.txt is not properly formatted.
A deployment must be available during updates. Rolling updates progressively replace instances of a service with new versions, maintaining availability.
The deployment pipeline first builds a new container image and pushes it to a registry.
docker build -t myapp:v1.2.3 .
docker push myapp:v1.2.3Then, it updates the Kubernetes deployment manifest with the new image and applies it.
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
replicas: 4
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: app
image: myapp:v1.2.3
ports:
- containerPort: 8080The rollout is configured with a rollingUpdate strategy. The cluster gradually replaces old pods with new ones, one by one or in batches.
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 1The pipeline monitors the rollout. After applying the manifest, it waits for the new replicas to become ready.
kubectl rollout status deployment/myapp --timeout=300sIf any pod is not ready after 300 seconds, the pipeline marks the deployment as failed. The pipeline then performs a health check: it sends a GET /health request to all instances and waits for a 200 response.
curl -f http://myapp.example.com/healthA failure occurs when the new image is built but the myapp:v1.2.3 image is not available in the registry. The deployment applies the manifest, but the pods fail to pull the image because the imagePullPolicy is IfNotPresent, and the image was never pulled before. The pods start but fail to start the application. The health check passes, but the application is unavailable because the health endpoint returns 500.
For high-traffic services, canary deployments reduce risk by gradually routing real traffic to a new version.
The pipeline starts by deploying the new version to a subset of nodes (e.g., 10% of the total). It uses a service mesh or ingress controller to split traffic.
The ingress controller uses a WeightedService or VirtualService configuration to route traffic based on weights.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: myapp
spec:
hosts:
- myapp.example.com
http:
- route:
- destination:
host: myapp
subset: v1
weight: 90
- destination:
host: myapp
subset: v2
weight: 10The pipeline configures the subset for each version in a DestinationRule.
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: myapp
spec:
host: myapp
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2The pipeline then runs a health check on the canary version. If all checks pass after 10 minutes, it increases the traffic to 30%. After 30 minutes, it increases to 60%, and after another hour, it promotes the new version to 100%.
A failure mode: the canary version is deployed but the version label is misspelled in the DestinationRule. The subset v2 is not matched, and traffic continues to the old version. The health check passes, but the canary is not active. Another edge: the pipeline assumes v2 is the canary version, but the v1 version is still being used in production. The deployment fails because the canary version is not the latest, and the health check uses the wrong version.
Feature flags allow deployment of code to production without activating it. The pipeline enables a feature flag at deployment time.
The pipeline exposes a flag management API. A deployment step calls an endpoint to create or update a flag.
curl -X POST \
-H "Authorization: Bearer $FLAG_TOKEN" \
-d '{
"name": "new-login-flow",
"description": "Enable new login flow with OAuth2",
"enabled": true,
"environments": ["staging", "production"],
"variants": {
"default": { "value": true },
"mobile": { "value": true, "weight": 0.3 }
}
}' \
https://flags.example.com/api/flagsThe application reads the flag at runtime. A feature flag is defined in a configuration file or database.
{
"flags": {
"new-login-flow": {
"enabled": true,
"variants": {
"default": true,
"mobile": true
}
}
}
}The pipeline triggers a release when a feature flag is enabled in a staging environment. It monitors the flag state and automatically promotes the feature to production after a specified duration or when a health check passes.
A silent failure: the flag name is case-sensitive, but the API expects new-login-flow, while the application reads NewLoginFlow. The feature is not activated. Another edge: the pipeline uses curl to call the flag API, but the server returns a 201 Created with a Location header. The pipeline expects the body to contain the flag ID, but it reads the Location header instead. The deployment continues, but the next step expects a flagId that does not exist.
A pipeline is defined in a declarative configuration file, versioned in Git. The pipeline is not a static series of steps but a living artifact.
The pipeline configuration file uses a standard format: YAML.
name: Build and Deploy
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- run: npm install
- run: npm run build
- name: Upload artifacts
uses: actions/upload-artifact@v3
with:
name: build
path: dist/
test:
runs-on: ubuntu-latest
needs: build
steps:
- uses: actions/checkout@v3
- uses: actions/download-artifact@v3
with:
name: build
path: dist/
- run: npm test
- name: Upload test results
uses: actions/upload-artifact@v3
with:
name: test-results
path: reports/
deploy:
runs-on: ubuntu-latest
needs: test
if: github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v3
- uses: actions/download-artifact@v3
with:
name: build
path: dist/
- run: |
kubectl apply -f k8s/
kubectl rollout status deployment/myapp --timeout=300sThe pipeline runs every time the configuration file changes. When pipeline.yml is committed, the pipeline rebuilds itself and applies the new configuration to the deployment environment.
A failure: the pipeline assumes pipeline.yml is the only configuration file, but another pipeline file ci.yml exists. The pipeline runs ci.yml but ignores pipeline.yml. The on section is missing, and the deploy job runs on every push, not just main. Another edge: the if condition uses github.ref, but github.ref is refs/heads/main, while github.ref is main. The condition github.ref == 'refs/heads/main' fails, and the deployment is skipped.
When a deployment fails, the pipeline rolls back to the previous version.
The pipeline records the current version in a registry or database. After a successful deployment, it updates a current-version record.
{
"service": "myapp",
"version": "v1.2.3",
"deployed_at": "2024-07-12T10:30:00Z",
"status": "success"
}On failure, the pipeline retrieves the previous version from the registry and deploys it.
# Get previous version
VERSION=$(curl -s https://registry.example.com/services/myapp/current \
| jq -r '.previous_version')
# Deploy previous version
kubectl set image deployment/myapp app=myapp:$VERSION
kubectl rollout status deployment/myapp --timeout=300sA failure occurs when the current-version endpoint returns a 404. The pipeline assumes the current-version endpoint is always available, but the registry is down. The rollback fails, and the pipeline logs the error but does not mark the deployment as failed.
Another edge: the pipeline uses kubectl rollout status, but the rollout takes longer than 300 seconds. The kubectl command exits with success, but the rollout is still in progress. The pipeline considers the deployment successful, but the service is not fully ready. The --timeout=300s is not enough, and the pipeline needs --wait or a polling loop.
This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy