Production engineering guide for zero downtime deployment arch covering patterns, implementation strategies, and operational best practices.
Zero downtime deployment architecture ensures that services remain available and responsive during updates, even as new versions of code, configurations, or dependencies are rolled out. It matters most in systems with high availability requirements—e.g., financial transaction processing, real-time collaboration, or e-commerce platforms—where even a few seconds of downtime can cost thousands in revenue or user trust. This architecture enables continuous delivery with confidence: teams can push changes at any time, without coordination, without scheduling, and without risk of service interruption.
The foundation of zero downtime deployment is a rolling update strategy where new instances of a service are gradually introduced while old ones continue serving traffic. The key is to manage instance lifecycle through a load balancer, with health checks and readiness probes.
Before the rollout, a new version of the application is built, tested, and packaged into a container image. The deployment process starts by launching new instances (containers or VMs) with the updated version.
docker build -t registry.example.com/app:v2.1.3 .
docker push registry.example.com/app:v2.1.3
# Deploy new version to Kubernetes cluster
kubectl apply -f deployment.yamlThe deployment.yaml file defines a Deployment object with replicas: 4, strategy: RollingUpdate, and a readiness probe:
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
spec:
replicas: 4
selector:
matchLabels:
app: backend
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 2
maxUnavailable: 1
template:
metadata:
labels:
app: backend
spec:
containers:
- name: app
image: registry.example.com/app:v2.1.3
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 3The readiness probe is critical: a new pod is not added to the load balancer until it passes the GET /health check. The initialDelaySeconds: 10 ensures the app has time to start up, initialize connections, and warm caches.
As new pods become ready, the load balancer begins routing traffic to them. The roll-out proceeds in small batches, controlled by the maxSurge and maxUnavailable values.
maxSurge: 2 means up to two additional pods can be active during the update, allowing for smooth traffic ramp-up.maxUnavailable: 1 ensures that at least one pod remains available at all times.The load balancer (e.g., AWS ALB, NGINX, HAProxy, or Kubernetes Service) maintains a list of active endpoints. When a new pod becomes ready, it is added to the backend pool. As the rollout progresses, the old pods are gradually drained.
The deployment pipeline must orchestrate a graceful shutdown of old instances. This is not automatic: a pod may be marked ready but still serve requests while the new version is stabilizing.
# In the application code (Go example), handle SIGTERM for graceful shutdown
func main() {
server := &http.Server{Addr: ":8080"}
go func() {
if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Fatalf("Server failed: %v", err)
}
}()
// Graceful shutdown on SIGTERM
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGTERM)
<-sigChan
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := server.Shutdown(ctx); err != nil {
log.Printf("Server shutdown failed: %v", err)
}
log.Println("Server stopped gracefully")
}The maxUnavailable: 1 setting ensures that during the update, the load balancer can stop one old pod while the new ones take over. The old pod is removed from the backend list, and after a brief delay (e.g., 10 seconds), it begins draining incoming requests. Once it has processed all in-flight requests, it sends a SIGTERM, allowing the app to shut down cleanly.
While rolling updates work well for most services, zero downtime becomes more predictable and risk-aware when using blue-green or canary strategies.
In blue-green, two identical environments run in parallel: one active (blue), one inactive (green). When a new version is ready, it is deployed to the green environment. After validation, traffic is switched from blue to green.
# Switch traffic from blue to green using a load balancer API
curl -X PUT https://api.lb.example.com/v1/traffic \
-H "Content-Type: application/json" \
-d '{
"target": "green",
"strategy": "switch",
"delay": 60,
"timeout": 120
}'The switch strategy immediately redirects all traffic to green. The delay field allows for a warm-up period before the switch. The timeout defines how long the load balancer waits for the green environment to become healthy before finalizing the switch.
During the switch, both blue and green environments are running. If the new version fails, the load balancer can revert to blue immediately:
curl -X PUT https://api.lb.example.com/v1/traffic \
-d '{"target": "blue", "strategy": "rollback"}'This approach is ideal for large, complex releases where validation time is significant.
Canary deployment gradually introduces the new version to a subset of users, with traffic split between old and new versions.
# Example: traffic split in a Kubernetes Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
spec:
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: app-blue
port:
number: 8080
weight: 90
- path: /
pathType: Prefix
backend:
service:
name: app-green
port:
number: 8080
weight: 10Here, 90% of traffic goes to the current version (app-blue), and 10% to the new version (app-green). The weight field controls the split. This can be adjusted in real time:
# Update canary weights
kubectl patch ingress app-ingress \
-p '{
"spec": {
"rules": [
{
"host": "app.example.com",
"http": {
"paths": [
{
"path": "/",
"pathType": "Prefix",
"backend": {
"service": { "name": "app-blue", "port": { "number": 8080 } },
"weight": 70
}
},
{
"path": "/",
"pathType": "Prefix",
"backend": {
"service": { "name": "app-green", "port": { "number": 8080 } },
"weight": 30
}
}
]
}
}
]
}
}'Canaries are excellent for measuring performance, error rates, and user behavior before a full rollout.
A common failure is misconfiguring the readiness probe. If initialDelaySeconds is too short, the pod is marked ready before the database connection is established, leading to 503 Service Unavailable errors during the rollout.
A typical error seen in logs:
GET /health HTTP/1.1
Host: app.example.com
User-Agent: curl/7.68.0
Accept: */*
HTTP/1.1 503 Service Unavailable
Content-Type: application/json
{"status": "unhealthy", "message": "database connection failed"}The load balancer adds the pod to the backend but receives 503 on the first health check, causing it to fail the readiness probe.
If the health check endpoint takes longer than the periodSeconds interval, the load balancer may mark the instance as unhealthy during the rollout.
For example, if periodSeconds: 5 but the /health endpoint takes 7 seconds to complete, the load balancer will send a new request before the previous one finishes, resulting in overlapping checks. This can lead to flapping health status and unnecessary pod replacements.
If the SIGTERM is sent too early—before the load balancer has drained all in-flight requests—the old instance may terminate while still processing a request.
# Example: incomplete request during shutdown
2023-10-05T12:00:00Z [INFO] Handling request: GET /api/orders/123
2023-10-05T12:00:05Z [INFO] Request started
2023-10-05T12:00:10Z [INFO] Request in progress
2023-10-05T12:00:15Z [INFO] SIGTERM received
2023-10-05T12:00:16Z [INFO] Server shutting down
2023-10-05T12:00:17Z [INFO] Request finished, but server already shut downThe request completes, but the response is lost, leading to client timeouts or incomplete data.
A subtle but frequent failure occurs when the new version uses a different configuration than the old one. For example, the new version expects a DATABASE_URL environment variable, but the old version relies on DB_HOST and DB_PORT.
This leads to connection errors, especially when the load balancer serves traffic to both old and new versions simultaneously. The error appears as:
pq: could not connect to server: Connection refused
Is the server running on host "db.example.com" and accepting
TCP/IP connections on port 5432?The new version connects to db.example.com:5432, but the old version uses db.example.com:3306, and the database is not listening on port 5432.
When services depend on shared state (e.g., a message queue, a shared cache, or a database), a version mismatch can cause silent failures.
Consider a service that writes to a Kafka topic with a new schema. The old version reads from the topic using the old schema, but the new version writes using the new schema. If the schema is not backward compatible, the old version fails to parse messages.
// Old schema: {"id": 1, "name": "Alice"}
// New schema: {"id": 1, "name": "Alice", "email": "[email protected]"}The old version reads {"id": 1, "name": "Alice"} but fails to parse {"id": 1, "name": "Alice", "email": "[email protected]"} because it does not expect the email field.
Before declaring a zero downtime deployment successful, the system must validate that both versions are running and that traffic is balanced correctly.
# Check deployment status
kubectl get deployments app -o wide
kubectl get pods -l app=backend
# Verify rollout status
kubectl rollout status deployment/app
kubectl rollout history deployment/app
# Check health of both versions
curl -s http://app.example.com/health | jq .
curl -s http://app.example.com/health?version=blue | jq .
curl -s http://app.example.com/health?version=green | jq .The rollout status command waits until all new pods are ready and the old ones are successfully replaced. The rollout history shows the progression of revisions, enabling quick rollbacks.
When a deployment fails, the team can roll back to the previous version with a single command:
kubectl rollout undo deployment/appThis command reverts the Deployment to the previous revision, bringing back the old version with all its configuration and image.
This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy