Immutable Infrastructure | The Garnet Wiki

Immutable Infrastructure is a cloud architecture practice where system components—servers, containers, or virtual machines—are never modified after deployment. Instead, every change triggers a new, complete, versioned artifact that replaces the old one. It matters when consistency, reproducibility, and reliability across environments are critical, especially in distributed systems with frequent deployments, rolling upgrades, and disaster recovery needs.

Deploying a New Version

To deploy a new version of a service, you build a new image from source, then push it to a registry. This image becomes a self-contained unit that can be instantiated anywhere. The deployment process is deterministic: the same build command and configuration always produce the same artifact.

docker build -t myapp:v2.1.0 .
docker push myapp:v2.1.0

The image includes the application code, dependencies, runtime, and configuration. No post-deployment configuration or patching is allowed. Once the image is pushed, it is immutable: its contents cannot be changed.

The deployment itself is orchestrated by a scheduler or orchestrator. A typical workflow uses a declarative configuration file, such as a Kubernetes manifest, to describe how the new version should be rolled out.


apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 4
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
        version: v2.1.0
    spec:
      containers:
        - name: app
          image: myapp:v2.1.0
          ports:
            - containerPort: 8080
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: app-secrets
                  key: db_url

Apply the new version with kubectl apply -f deployment.yaml. The orchestrator creates new pods from the image, and during the rollout, old pods continue serving requests while new ones start. Once all new pods are ready, traffic is switched to them, and the old version is retired.

Rolling Out with Zero Downtime

A rolling update strategy ensures that the service remains available during deployment. The orchestrator gradually replaces old pods with new ones, maintaining a steady load.

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 1

This configuration allows up to two more pods than the desired number (maxSurge) and permits up to one pod to be unavailable during the update (maxUnavailable). During the rollout, the cluster maintains at least 4 running pods (the desired replica count), and the number of pods increases to 6 during peak update.

The health of a pod is determined by readiness and liveness probes. The pod is considered ready only when the application responds to a health check. The orchestrator uses readiness probes to delay the start of traffic to a pod until it is fully initialized.

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
  timeoutSeconds: 3
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 3
  timeoutSeconds: 2

If a pod fails its liveness probe, it is restarted. If it fails its readiness probe, it is removed from the load balancer until it becomes ready. A common failure mode is the “stuck readiness probe”: a pod reports ready but does not respond to requests, leading to partial outages.

Versioning and Release Management

Each deployment is versioned. The version is derived from a build artifact and recorded in the image and manifest. The orchestrator tracks which version is active, and every pod carries its version in metadata.

spec:
  template:
    metadata:
      labels:
        version: v2.1.0
        env: production

The version is used to identify the state of the system at any moment. You can inspect which version is running on which node, and you can roll back to a previous version by applying a manifest with an older image.

Rollback is simple: update the deployment’s image field to an earlier version and apply.

# deployment.yaml (rollback to v2.0.0)
spec:
  template:
    spec:
      containers:
        - name: app
          image: myapp:v2.0.0

Apply with kubectl apply -f deployment.yaml. The orchestrator creates new pods from the v2.0.0 image, and when the new pods are ready, it gradually switches traffic. Old pods are terminated after a grace period.

Rollbacks are instantaneous and reliable. If a new version causes a critical bug, the team can roll back within minutes, with no downtime and no manual intervention.

Configuration and Secrets Management

Configuration is managed separately from the application image. The image contains only the code and runtime. Configuration is injected at runtime via environment variables, configuration files, or mounted volumes.

The most common approach is to use Kubernetes ConfigMaps and Secrets.

# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-config
data:
  LOG_LEVEL: info
  CACHE_TTL: "300"
  FEATURE_FLAGS: |
    analytics: true
    search: false
    notifications: true

# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  db_url: bXlhcHA6Ly9kYi5leGFtcGxlLmNvbS9tYW5hZ2U=
  jwt_key: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

These are mounted into the pod:

spec:
  containers:
    - name: app
      image: myapp:v2.1.0
      envFrom:
        - configMapRef:
            name: myapp-config
        - secretRef:
            name: app-secrets
      volumeMounts:
        - name: config-volume
          mountPath: /etc/app-config
          readOnly: true
  volumes:
    - name: config-volume
      configMap:
        name: myapp-config

Configuration is immutable per deployment. Each version of the service uses a fixed set of config and secrets, which can be versioned and tested independently.

When a configuration change is needed, it is applied as a separate update. The orchestrator applies the new ConfigMap or Secret, and the next deployment rollout picks up the new configuration.

If a configuration change is rolled back, the version of the service and its configuration are decoupled. A v2.0.0 service can run with a v2.1.0 configuration, leading to subtle inconsistencies.

Infrastructure as Code and Build Pipelines

The entire infrastructure—networks, load balancers, storage—is defined in code. This code is versioned, tested, and deployed alongside the application.

The build pipeline is responsible for transforming source code into a deployable image.

# build.sh
#!/bin/bash
set -euo pipefail

# Build image
docker build -t myapp:$GIT_COMMIT .
docker tag myapp:$GIT_COMMIT myapp:latest

# Push to registry
docker push myapp:$GIT_COMMIT
docker push myapp:latest

# Deploy to Kubernetes
kubectl apply -f k8s/
kubectl set image deployment/myapp app=myapp:$GIT_COMMIT

# Wait for rollout
kubectl rollout status deployment/myapp

The pipeline is triggered by a git push. The GIT_COMMIT is derived from the commit SHA and used to tag the image. The pipeline runs in a CI system like GitHub Actions, GitLab CI, or CircleCI.

Each build produces a unique, immutable image. The image is tested in a staging environment before being promoted to production.

Failure Modes and Common Pitfalls

One silent failure is image drift. The build pipeline produces an image, but during the deployment, the orchestrator pulls the image from a registry. If the image is updated between the time of build and deployment, the running service may use a different version than expected.

This happens when the image tag is not fully qualified.

# ❌ Bad: uses latest tag
docker push myapp:latest

# ✅ Good: uses exact digest
docker push myapp@sha256:abc123...

The image digest ensures that the exact image is pulled, even if the latest tag changes.

Another common issue is missing configuration. A pod starts but fails to connect to the database because a required environment variable is missing.

# Error from pod
Error: Failed to connect to database: no such file or directory
Caused by: environment variable DATABASE_URL not set

This is often due to a typo in the environment variable name or a missing key in the ConfigMap.

A subtle failure is configuration inconsistency across environments. The production deployment uses a ConfigMap named myapp-config, but the staging deployment uses myapp-staging-config. This leads to differences in behavior that are hard to diagnose.

Another failure is version skew. Multiple versions of the service are running simultaneously, but they use different configurations and secrets. A new version is deployed with LOG_LEVEL: debug, but the old version still uses LOG_LEVEL: info. Over time, logs become inconsistent, and debugging becomes difficult.

Monitoring and Observability

The orchestrator exposes metrics about the deployment. The kubectl get deployments command shows the current status.

$ kubectl get deployments
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
myapp     4/4     4            4           2d

The kubectl rollout status command waits for a deployment to complete.

$ kubectl rollout status deployment/myapp
Waiting for deployment "myapp" rollout to finish: 2 of 4 updated replicas are available...
deployment "myapp" successfully rolled out

The kubectl describe deployment myapp command reveals detailed information about the rollout, including events, pod templates, and rollout history.

$ kubectl describe deployment myapp
Name:                   myapp
Namespace:              default
CreationTimestamp:      Mon, 15 Apr 2024 10:00:00 +0000
Labels:                 app=myapp
Annotations:            deployment.kubernetes.io/revision: 3
Selector:               app=myapp
Replicas:               4 desired | 4 current | 4 updated | 4 total
StrategyType:           RollingUpdate
MinReadySeconds:        30
RollingUpdateStrategy:  2 maxSurge, 1 maxUnavailable
Pod Template:
  Labels:  app=myapp
  Containers:
   app:
    Image:      myapp:v2.1.0
    Port:       8080/TCP
    Host Port:  0/TCP
    Environment:
      DATABASE_URL:  myapp://db.example.com/manage
    Mounts:
      /etc/app-config from config-volume (ro)
  Volumes:
   config-volume:
    ConfigMap:  myapp-config (existing)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  2m    deployment-controller  Scaled up replica set myapp-7b5c8f9d4d to 4
  Normal  ScalingReplicaSet  1m    deployment-controller  Scaled down replica set myapp-6c4f9d8b5c to 2
  Normal  ReplicaSetUpdated  1m    deployment-controller  Updated ReplicaSet myapp-7b5c8f9d4d

The kubectl rollout history deployment/myapp command shows the history of deployments.

$ kubectl rollout history deployment/myapp
REVISION  CHANGE-CAUSE
1         <none>
2         image updated to myapp:v2.0.0
3         image updated to myapp:v2.1.0

The kubectl rollout undo deployment/myapp command rolls back to the previous version.

$ kubectl rollout undo deployment/myapp
deployment.apps/myapp rolled back to revision 2

Summary

Immutable Infrastructure is not just about using containers. It is a discipline of treating every component as a fixed, versioned unit. Changes are made by building a new version, and deployments are atomic, predictable, and repeatable. The system is resilient to failure, and rollbacks are fast and reliable. The key is consistency: the same image, the same configuration, the same environment, applied across all stages of the software lifecycle.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.