Progressive Delivery

Production engineering guide for progressive delivery covering patterns, implementation strategies, and operational best practices.

Progressive delivery is a deployment strategy that gradually introduces changes to users or systems, rather than releasing all at once. It reduces risk by allowing incremental rollout with monitoring and rollback capabilities. This approach matters when deploying code that may affect user experience, system stability, or business operations.

Understanding Rollout Strategies

Progressive delivery typically involves phased releases where a new version is gradually exposed to increasing portions of users or traffic. The key is managing the pace of change and having mechanisms for immediate response to issues.

Implementing Canary Deployments

Canary deployments involve routing a small percentage of traffic to new code versions. This allows testing in production with real user load before full rollout.

To configure a canary deployment, define traffic routing rules that direct a specific percentage of requests to the new version:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 80

The configuration above routes all traffic to a single service. To enable canary deployment, you would typically use an ingress controller that supports weighted routing or implement custom logic in your service mesh.

Using Feature Flags for Progressive Rollouts

Feature flags allow enabling or disabling functionality without redeploying code. This approach is especially useful when you want to control access to new features based on user segments or other criteria.

A basic feature flag implementation uses environment variables or a configuration store:

# Set feature flag via environment variable
export FEATURE_NEW_UI=true

In code, you might check for the flag like this:

import os

def render_new_ui():
    if os.environ.get("FEATURE_NEW_UI") == "true":
        return render_new_ui_template()
    else:
        return render_old_ui_template()

When rolling out features progressively, you can change the flag value without redeploying code. This allows for rapid toggling and experimentation.

Managing Rollout Progression

A rollout progression defines how traffic moves from one version to another. It typically includes stages like initial rollout, gradual increase, and final promotion.

In Kubernetes, this might be implemented using a deployment strategy:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

The maxSurge and maxUnavailable settings control how many pods are created or unavailable during updates. Setting maxUnavailable to zero ensures no downtime during rollouts.

Controlling Traffic with Service Mesh

Service mesh implementations like Istio enable sophisticated traffic management that supports progressive delivery patterns.

Example Istio VirtualService configuration:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example.com
  http:
  - route:
    - destination:
        host: app-v1
        port:
          number: 80
      weight: 90
    - destination:
        host: app-v2
        port:
          number: 80
      weight: 10

This configuration routes 90% of traffic to the stable version and 10% to the new version. Adjusting weights enables progressive rollout without requiring changes to infrastructure.

Implementing Health Checks for Rollout Decisions

Health checks determine whether a new version is ready to receive full traffic. These checks typically validate that services are responsive and returning expected responses.

A simple readiness probe configuration:

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
  - name: example-container
    image: example-image
    readinessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10

If a pod fails its readiness probe, it's removed from service until it passes validation. This prevents unhealthy versions from receiving traffic.

Rolling Back Changes

Rollbacks are essential for progressive delivery. They allow reverting to a previous stable version when issues arise.

In Kubernetes, rolling back a deployment:

kubectl rollout undo deployment/example-app --to-revision=1

This command reverts the deployment to a specific revision. The --to-revision flag accepts numeric values that correspond to deployment history.

Monitoring Rollout Progress

Monitoring helps track how a progressive rollout is performing. Metrics include request success rates, error rates, and latency.

Example Prometheus query for tracking success rate:

sum(rate(http_requests_total{status="200"}[5m])) / sum(rate(http_requests_total[5m]))

This query calculates the percentage of successful requests over a five-minute window. Monitoring these metrics helps identify when to pause or stop a rollout.

Handling Rollout Failures

When rollouts fail, it's important to have mechanisms for immediate action. This includes automated rollback triggers and manual intervention options.

Example failure detection in a script:

#!/bin/bash
if ! curl -f http://example.com/health; then
  echo "Health check failed, triggering rollback"
  kubectl rollout undo deployment/example-app
fi

This script checks service health and rolls back if it fails. The -f flag makes curl fail on HTTP error codes.

Configuring Gradual Traffic Increase

Gradual traffic increases are a core part of progressive delivery. They allow small increments in exposure to new versions.

In a service mesh, this might involve changing weights over time:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-service
spec:
  hosts:
  - example.com
  http:
  - route:
    - destination:
        host: app-v1
        port:
          number: 80
      weight: 80
    - destination:
        host: app-v2
        port:
          number: 80
      weight: 20

This configuration increases new version traffic from 10% to 20% over time.

Managing Multiple Rollouts

When multiple changes are being rolled out simultaneously, coordination becomes important. Each rollout should have clear boundaries and monitoring.

Example approach using labels:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app-v1
  labels:
    version: v1
spec:
  replicas: 3
  selector:
    matchLabels:
      version: v1

Using labels helps distinguish between different versions during rollout.

Using Manual Intervention Points

Manual intervention points allow human review before proceeding with a rollout. These are particularly useful for high-risk changes.

A simple manual approval system using environment variables:

# Before starting rollout, set approval flag
export ROLLOUT_APPROVED=true

Code can then check this flag before proceeding:

import os

def start_rollout():
    if os.environ.get("ROLLOUT_APPROVED") == "true":
        # Proceed with rollout
        pass
    else:
        # Wait for manual approval
        raise Exception("Rollout requires manual approval")

Implementing Time-Based Rollouts

Time-based rollouts gradually increase traffic over a specified period. This approach helps smooth out potential issues.

Example configuration for gradual rollout over time:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: example-destination
spec:
  host: example-service
  trafficPolicy:
    loadBalancer:
      simple: LEAST_CONN

This configuration ensures even distribution of traffic across available instances.

Handling Configuration Changes

Configuration changes during rollouts can cause issues. It's important to manage these carefully to prevent inconsistencies.

Example approach using config maps:

apiVersion: v1
kind: ConfigMap
metadata:
  name: example-config
data:
  feature_flag: "true"

Updating configuration in a rollout requires careful coordination to ensure all components see consistent values.

Dealing with Rollout Conflicts

Conflicting changes can occur when multiple teams work on the same system. Clear communication and coordination are essential.

Example conflict resolution approach:

# Check for conflicts before starting rollout
kubectl get deployments
kubectl describe deployment example-app

These commands help identify potential issues before beginning a rollout.

Monitoring Rollout Impact

Rollout impact monitoring helps understand how changes affect performance and user experience.

Example metrics to track:

These metrics can be collected using standard observability tools and alerting systems.

Managing Rollout Duration

Rollouts should have defined durations. Too short and issues aren't fully tested; too long and user experience suffers.

Example rollout timing configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

The maxSurge and maxUnavailable settings control how quickly changes are applied.

Testing Rollout Scenarios

Testing rollout scenarios in staging environments helps identify potential issues before production deployment.

Example testing approach:

# Test canary deployment
kubectl set image deployment/example-app app-container=example-image:v2
# Monitor for issues
kubectl get pods -w

This process helps validate that rollouts work as expected.

Handling Rollout Failures in Production

Production rollouts require robust failure handling mechanisms.

Example failure handling script:

#!/bin/bash
if ! kubectl rollout status deployment/example-app; then
  echo "Rollout failed, initiating rollback"
  kubectl rollout undo deployment/example-app
fi

This script monitors rollout status and rolls back automatically if it fails.

Managing Rollout Dependencies

Dependencies between services must be considered in progressive delivery. Changes to dependent systems may affect rollout success.

Example dependency management:

# Check dependencies before rollout
kubectl get pods -l app=dependent-service
kubectl describe service dependent-service

These checks ensure that dependencies are healthy before starting a rollout.

Documenting Rollout Procedures

Clear documentation of rollout procedures helps ensure consistency across teams and reduces errors.

Example documentation approach:

## Rollout Procedure

1. Prepare environment
2. Deploy new version
3. Monitor health checks
4. Gradually increase traffic
5. Validate results
6. Complete rollout or rollback

This structured approach ensures that rollouts are handled consistently.

Handling Rollout Rollbacks

Rollbacks must be tested and documented to ensure they work correctly in production.

Example rollback procedure:

# Check current deployment
kubectl get deployments
# Perform rollback
kubectl rollout undo deployment/example-app --to-revision=1
# Verify rollback
kubectl rollout status deployment/example-app

This sequence ensures that rollbacks are performed correctly and verified.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.