Progressive Delivery

Ship features safely with progressive delivery techniques. Covers canary analysis, feature flags, dark launches, percentage rollouts, automated rollback, and the patterns that let you deploy with confidence and recover from failures fast.

Progressive delivery is a strategy for releasing software changes to users in stages, using real user traffic and observable metrics to guide decisions. It matters when a team needs to reduce risk, increase confidence, and improve velocity across multiple environments—especially when rolling out new features, performing infrastructure upgrades, or adopting new platforms. The core idea is to move beyond simple blue-green or rolling deployments by introducing a continuous feedback loop that adapts release decisions to actual system behavior.

Planning a Progressive Delivery Pipeline

A progressive delivery pipeline starts with a clear definition of release stages and success criteria. Each stage represents a segment of the user base or a set of infrastructure targets that receive a version of the application. The pipeline orchestrates the progression from development to production, using automated evaluation of metrics to decide whether to promote a version to the next stage.

The first step is defining the stages. A common sequence is:

  1. Staging: Automated tests run on a pre-production environment.
  2. Canary: A small percentage of real users receive the new version.
  3. Gradual rollout: Traffic increases incrementally over time.
  4. Production: Full deployment with automated promotion and monitoring.

Each stage must have one or more criteria for promotion. These criteria are evaluated after each deployment, and the pipeline advances only when all conditions are met.

Defining Promotion Criteria

Promotion criteria are derived from metrics collected during a release. The most common metrics are:

To define criteria, a configuration file is used. This file is checked into version control and applied as part of the release process.


stages:
  - name: staging
    criteria:
      - metric: http_success_rate
        threshold: 99.0
        window: 5m
        tolerance: 0.5
      - metric: error_rate
        threshold: 1.0
        window: 10m
        tolerance: 0.3
      - metric: p95_latency
        threshold: 300
        window: 1m
        tolerance: 25
    promotion:
      after: 15m
      when: all
      delay: 10m

  - name: canary
    criteria:
      - metric: http_success_rate
        threshold: 98.5
        window: 10m
        tolerance: 0.7
      - metric: checkout_events
        threshold: 1000
        window: 15m
        tolerance: 50
    promotion:
      after: 30m
      when: any
      delay: 15m

The window defines the time range over which metric data is aggregated. threshold is the required value, and tolerance is the acceptable deviation. The promotion block governs when to advance to the next stage: after defines the minimum time to wait before evaluating, when specifies whether all criteria must pass (all) or any one (any), and delay is an optional cooldown before the next stage begins.

Executing a Release

A release is initiated by pushing a new version of the application to a container registry and updating a manifest in Git.

# Build and push image
docker build -t registry.example.com/myapp:v1.2.3 .
docker push registry.example.com/myapp:v1.2.3

# Update image tag in k8s manifest
sed -i 's|image:.*|image: registry.example.com/myapp:v1.2.3|' deploy.yaml
git add deploy.yaml
git commit -m "Deploy v1.2.3"
git push origin main

The pipeline listens for changes to deploy.yaml, then triggers a new release. It fetches the manifest, deploys the new image to the staging environment, and begins collecting metrics.

Deploying to Staging

The pipeline deploys the new version to staging using kubectl.

kubectl apply -f deploy.yaml --context staging

It then starts monitoring using a metric collector service. This service exposes a /metrics endpoint that returns current values in JSON:

{
  "http_success_rate": 98.7,
  "error_rate": 1.3,
  "p95_latency": 294,
  "checkout_events": 872
}

The pipeline polls this endpoint every 30 seconds, recording values over time. After 15 minutes, it evaluates the criteria defined in pipeline.yaml.

If any criterion fails—say, error_rate exceeds 1.0—the pipeline pauses and sends an alert:

{
  "event": "promotion_failed",
  "stage": "staging",
  "timestamp": "2024-01-15T08:45:23Z",
  "details": [
    {
      "metric": "error_rate",
      "current": 1.42,
      "threshold": 1.0,
      "window": "10m",
      "sample": [
        { "timestamp": "2024-01-15T08:30:00Z", "value": 1.1 },
        { "timestamp": "2024-01-15T08:40:00Z", "value": 1.42 }
      ]
    }
  ]
}

The pipeline waits for 10 minutes before retrying. If the metric stabilizes, it proceeds to the next stage.

Rolling Out to Canary

When staging promotion succeeds, the pipeline deploys the version to the canary environment. This is typically a subset of users, defined by a percentage, a user ID range, or an HTTP header.

The deployment uses a feature flag or a traffic-splitting mechanism in Kubernetes. A TrafficSplit resource is applied:

apiVersion: split.example.com/v1
kind: TrafficSplit
metadata:
  name: myapp-canary
spec:
  service: myapp
  splits:
    - weight: 10
      version: v1.2.3
    - weight: 90
      version: v1.2.2

This routes 10% of incoming requests to the new version (v1.2.3), with the rest on the stable version (v1.2.2). The pipeline starts evaluating canary criteria, polling the /metrics endpoint every 30 seconds.

Evaluating Canaries

The canary evaluation is sensitive to both stability and business impact. The pipeline tracks two key indicators:

Each metric is evaluated independently. The pipeline waits 30 minutes after deployment to gather sufficient data, then checks whether any of the defined criteria pass.

If checkout_events reaches 1,000 within 15 minutes, the canary is considered successful, and the pipeline begins the next phase.

Handling Failure in Canary

A canary failure is not just a metric threshold. It includes silent failures—where the system behaves correctly but the business outcome is poor.

For example, a canary deployment shows:

The pipeline evaluates when: any, so even though only one metric fails, it advances to the next stage.

But a deeper failure occurs when the system reports metrics but the data is inconsistent. The /metrics endpoint returns:

{
  "http_success_rate": 98.5,
  "error_rate": 1.2,
  "p95_latency": 310,
  "checkout_events": 920,
  "last_updated": "2024-01-15T09:15:00Z"
}

However, the checkout_events value is outdated: the last_updated timestamp is 5 minutes behind the current time. The pipeline detects this mismatch and logs:

[canary-evaluation] WARNING: metrics are stale
  - checkout_events updated at 2024-01-15T09:10:00Z
  - current evaluation time: 2024-01-15T09:15:00Z
  - data is 5m behind

The pipeline pauses and sends a signal to the observability system: metric_staleness is true for checkout_events.

In production, this leads to a common mistake: teams assume metrics are up-to-date but neglect to validate the freshness of data. A metric that passes its threshold may be reporting on data from a past state.

Gradual Rollout and Production Promotion

After the canary succeeds, the pipeline begins a gradual rollout. This is a series of incremental traffic shifts, each triggered by a metric.

The rollout configuration defines the progression:

rollout:
  strategy: gradual
  steps:
    - weight: 25
      condition: p95_latency < 300
      duration: 15m
    - weight: 50
      condition: error_rate < 1.0
      duration: 30m
    - weight: 75
      condition: checkout_events > 1500
      duration: 45m
    - weight: 100
      condition: all
      duration: 60m

At each step, the pipeline updates the TrafficSplit resource:

# After first step
spec:
  service: myapp
  splits:
    - weight: 25
      version: v1.2.3
    - weight: 75
      version: v1.2.2

It then waits the specified duration, evaluating conditions continuously. When all conditions are met, it advances to the next step.

Failure Modes in Gradual Rollout

The gradual rollout phase has several subtle failure modes.

To handle these, the pipeline tracks metric stability using a sliding window. It calculates the standard deviation of the metric over the evaluation window. If the standard deviation is high, it marks the metric as unstable, and the promotion is delayed.

Final Promotion to Production

Once the rollout reaches 100% traffic, the pipeline evaluates final criteria:

If all conditions pass, the pipeline promotes the version to production. It updates the TrafficSplit to 100% new version and marks the release as complete.

The release is now considered stable. The pipeline records the release in a release database:

INSERT INTO releases (
  version,
  stage,
  started_at,
  completed_at,
  status,
  promotion_criteria,
  evaluation_metadata
) VALUES (
  'v1.2.3',
  'production',
  '2024-01-15T08:30:00Z',
  '2024-01-15T14:30:00Z',
  'successful',
  '{
    "http_success_rate": 99.1,
    "error_rate": 0.8,
    "p95_latency": 290,
    "checkout_events": 5200
  }',
  '{
    "evaluation_window": "30m",
    "stability": true,
    "metric_freshness": true
  }'
);

Common Pitfalls and Sharp Edges

Summary

Progressive delivery is not a sequence of steps. It is a feedback-driven process where decisions to promote a release are based on observable data. The mechanism is simple: a configuration file defines stages and criteria, a pipeline deploys, collects metrics, evaluates conditions, and advances releases. The sharp edges—stale data, overlapping conditions, atomicity, idempotency—reveal themselves only under load. The pattern is transferable: a team can implement it with standard tools—kubectl, curl, psql, git, and a REST API—without needing a specific platform. What matters is not the tool, but the flow: the data, the timing, the evaluation, the decision.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.