Deployment Strategies Deep Dive

Production engineering guide for deployment strategies deep dive covering patterns, implementation strategies, and operational best practices.

When you’re responsible for moving code from a CI system into production without taking services offline or introducing undetected regressions, deployment strategies become the primary safety net. This page covers the concrete patterns, commands, and failure modes you’ll encounter when shifting traffic, updating resources, and rolling back if something goes wrong. It matters every time a new version is pushed, especially in systems where uptime commitments, stateful data, or external integrations make a simple restart costly or impossible.

Rolling Updates as the Default Path

Most Kubernetes clusters use rolling updates as the native deployment mode. When you run kubectl apply -f deployment.yaml with a new container image tag, the controller creates a new ReplicaSet, scales it up, and scales the old one down while preserving pod IPs and volumes. The exact behavior is governed by the spec.strategy field in the Deployment manifest. The most common is RollingUpdate, with maxSurge and maxUnavailable controlling how many pods can be outside the desired count at any moment. A typical manifest snippet looks like:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 25%

To trigger a rollout after an image change, you can either push a new tag and run kubectl apply again, or explicitly set the image with kubectl set image deployment/my-app my-app=my-registry/my-app:v1.4.2. The command updates the Deployment’s .spec.template.spec.containers[0].image field and immediately starts the rollout sequence. Checking progress is done with kubectl rollout status deployment/my-app --timeout=300s, which streams the ReplicaSet progress and exits with status 0 once the desired number of available pods matches the desired replica count. If the rollout stalls, kubectl rollout history deployment/my-app shows the revision list, and kubectl rollout undo deployment/my-app reverts to the previous revision instantly.

Sharp edge: Setting maxUnavailable: 0 forces a “pause‑until‑ready” mode that can deadlock if new pods fail their readiness probe faster than old pods are terminated. Conversely, maxSurge set too high relative to cluster capacity can push the scheduler past node limits, causing FailedScheduling events and a rollout that appears stuck at Progressing with False conditions.

Blue‑Green Swaps When Zero‑Downtime Is Non‑Negotiable

Rolling updates are sufficient for many workloads, but some services require an instantaneous cutover—database schema changes, protocol upgrades that aren’t backward‑compatible, or migrations where two versions cannot coexist. The pattern is to run both versions in parallel and switch the traffic endpoint once the new version proves healthy.

In Kubernetes, this often starts with deploying the new version under a distinct label set, then patching the Service selector. For example, after applying a v2 Deployment that selects pods with version=v2, you can shift traffic with:

kubectl patch svc my-app -p '{"spec":{"selector":"version=v2"}}'

If the Service uses a sessionAffinity or external load balancer, you may need to update the balancer’s backend list explicitly, e.g., aws elbv2 modify-target-group-attributes --target-group-arn $TG_ARN --target-group-attributes Key=stickiness.enabled,Value=true (the CLI belongs to the real AWS console, but the mechanics—changing which pod IPs receive traffic—are generic). The old Deployment continues running; you verify logs, probe results, and integration tests against the new pods. Once confidence is high, you can scale the v1 Deployment to zero and delete it, or leave it dormant for a defined fallback window.

Sharp edge: If the new version writes to the same database schema without a backward‑compatible migration, queries from the old version may fail immediately after the selector switch, producing SQLSTATE 42S02: Base table or view not found errors. Because the Service switch is effectively atomic from the client perspective, there is no graceful degradation window unless you introduce a dual‑write phase or use a feature‑flag‑aware proxy. Another silent failure occurs when externalIPs or cloud‑provider load balancers cache the old endpoint TTL; clients may continue hitting terminated pods for up to the TTL duration, producing connection reset by peer errors that are hard to correlate to the deployment event.

Canary Flights When You Want Graduated Exposure

Canary deployments let you route a small percentage of live traffic to the new version while the majority stays on the known‑good baseline. This pattern is common when you want statistical confidence before a full switch, or when you’re testing a change that carries a low but non‑zero risk of regression.

Kubernetes itself does not route traffic by percentage natively, but the pattern can be realized with a combination of Deployments, Services, and an Ingress controller or service mesh that supports weight-based routing. If you’re using an Ingress controller that supports backend weights (e.g., NGINX Ingress with nginx.ingress.kubernetes.io/upstream-weight), you can set:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  annotations:
    nginx.ingress.kubernetes.io/upstream-weight: "80/20"
spec:
  rules:
  - host: my-app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-app
            port:
              number: 80

The annotation above directs 80 % of requests to the v1 Service endpoint and 20 % to a v2 Service endpoint that you’ve manually scaled up. To shift weight, you edit the annotation value (e.g., nginx.ingress.kubernetes.io/upstream-weight: "60/40") and the controller updates the upstream group in seconds. Alternatively, if you have a service mesh like Istio installed, you can adjust traffic split via a VirtualService weight field:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-app
spec:
  hosts: ["my-app.example.com"]
  http:
  - route:
    - destination:
        host: my-app.svc.cluster.local
        subset: v1
      weight: 80
    - destination:
        host: my-app.svc.cluster.local
        subset: v2
      weight: 20

Adjusting the weights is then a matter of kubectl apply -f updated-virtual-service.yaml, and the mesh propagates the change instantly.

Sharp edge: Weighted routing only works if both versions expose the same API surface and respond to identical request shapes. If v2 changes a request payload or header, the 20 % of traffic sent to it may generate 400 Bad Request or 422 Unprocessable Entity errors that are easy to miss in monitoring dashboards designed around total error rates. Another silent failure: if the v2 subset isn’t fully populated (e.g., only 2 of 5 desired replicas are Ready), the mesh may still route traffic to it, returning 503 Service Unavailable from an under‑provisioned pod. Always pair weight changes with a kubectl get pods -l version=v2 check, and watch for Ready percentage shifts in the mesh’s pilot or istio-proxy logs.

Rolling Back When the New Version Fails

Even with the best safety nets, a deployment can introduce a regression that passes pre‑flight checks but degrades user experience or breaks critical paths. The fastest recovery pattern is a one‑command rollback that restores the previous ReplicaSet and drains the new one.

The canonical command is:

kubectl rollout undo deployment/my-app

Kubernetes records every revision of a Deployment’s .spec in its internal etcd store. undo sets the Deployment’s .spec.revisionHistoryLimit‑controlled previous revision as the active .spec.template, then initiates a rolling update back to that configuration. You can target a specific revision with kubectl rollout undo deployment/my-app --to-revision=3, which rolls back to revision 3 explicitly. Checking the state after undo is done with kubectl rollout status deployment/my-app --timeout=300s; the command succeeds once available pods match the pre‑rollback replica count and all old‑version pods report Ready.

If the rollback itself fails—often because the previous revision’s container image tag no longer exists in your registry—kubectl returns:

error: failed to rollout: revision "5" does not have a valid template; cannot undo to a revision with an missing image

In that case, you must re‑push the original image tag or restore the Deployment manifest from git and re‑apply it.

Sharp edge: undo only works if the Deployment has a revisionHistoryLimit that retains the target revision. If you’ve set the limit to 1 and the current rollout is on revision 2, kubectl rollout undo will attempt to revert to revision 1, but if a subsequent rollout has already pushed the history limit past it, the command may silently no‑op or produce error: deployment "my-app" has no previous revision to roll back to. Another gotcha: if the rollout was triggered by a kubectl set image command that changed the image tag to a digest‑based reference (e.g., my-registry/my-app@sha256:abc…), the rollback may not restore the exact prior tag if the registry pruned the old digest, requiring manual manifest edit and re‑apply.

Sharp Edges: The Things That Silently Break Deployments

Beyond the mechanics of each strategy, several recurring failure modes catch teams off‑guard:

Understanding these patterns—and the exact kubectl commands, flags, and error texts that accompany them—lets you diagnose a failing deployment in minutes rather than hours, and choose the right strategy (rolling, blue‑green, or canary) before the first pod even starts. The key is to match the strategy’s risk profile to the service’s tolerance for partial outages, state inconsistency, and the time you have to validate a new version before it becomes the live path.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.