ESC
Type to search guides, tutorials, and reference documentation.

Kubernetes: Control Loops, Workloads, and Failure Modes

The reconciliation model behind the Kubernetes API, the workload objects you actually use, how scheduling and probes really behave, and the failure modes that dominate real clusters.

This reference explains how Kubernetes control loops manage workloads and what can go wrong when those loops fail. It covers specific commands, configuration patterns, and failure modes you'll encounter in real deployments. Understanding these helps when debugging deployment issues, handling cluster failures, or designing resilient systems.

How Control Loops Work in Kubernetes

Kubernetes implements a control loop pattern where controllers continuously observe the desired state of resources and take actions to match that state. The main controller loops are:

  • Deployment Controller: Manages ReplicaSets and ensures the correct number of pods run.
  • ReplicaSet Controller: Ensures the right number of pods exist for a ReplicaSet.
  • StatefulSet Controller: Manages pods with stable network identities and persistent storage.
  • DaemonSet Controller: Ensures one pod runs on each node matching a selector.
  • Job Controller: Runs pods to completion and ensures they finish successfully.

Each controller polls the Kubernetes API server for changes to its managed resources. For example, when you create a Deployment with replicas: 3, the Deployment controller watches for:

kubectl get deployment myapp -o jsonpath='{.status.replicas}'

The controller compares .spec.replicas with .status.replicas and creates or deletes pods as needed.

Understanding Workload Types

Deployments

Deployments create ReplicaSets to manage pods. To scale a Deployment:

kubectl scale deployment myapp --replicas=5

Deployment status can be checked using:

kubectl get deployment myapp -o jsonpath='{.status}'

This returns fields like .status.replicas, .status.readyReplicas, and .status.updatedReplicas.

StatefulSets

StatefulSets maintain stable network identities for pods. They require a headless service:

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  clusterIP: None
  selector:
    app: nginx

To update a StatefulSet, use:

kubectl patch statefulset myapp -p '{"spec":{"replicas":3}}'

DaemonSets

DaemonSets ensure one pod per node. They can be updated with:

kubectl set image daemonset myapp nginx=nginx:1.21

Common Failure Modes

Control Loop Starvation

If a controller's API server watch fails, it may miss updates. This can happen when the API server returns 503 errors for watches. The controller will retry and potentially fall back to full resyncs.

To diagnose watch failures:

kubectl get pods -l app=myapp --watch

Watch failures show as Error events in pod status or FailedScheduling events in the node.

Deployment Replicas Not Matching Desired State

When .status.replicas does not equal .spec.replicas, check for:

  • Pod scheduling failures:
  kubectl describe deployment myapp

Look for FailedScheduling or Evicted events.

  • Image pull errors:
  kubectl get pods -l app=myapp -o jsonpath='{.items[*].status.containerStatuses[*].imageID}'
  • Resource constraints:
  kubectl describe pod myapp-7b5b8c9f4d-xyz12

Look for Evicted or OOMKilled events.

StatefulSet Pod Failures

StatefulSets can fail to update if pods are stuck in Pending state. Common causes include:

  • Persistent volume claims not binding:
  kubectl get pvc

Look for Pending PVCs with no matching PersistentVolume.

  • Storage class issues:
  kubectl get storageclass

Check if the provisioner is working or returning errors.

DaemonSet Rollout Issues

DaemonSets may fail to update when a node becomes unreachable. The controller will retry indefinitely on failed nodes:

kubectl get daemonset myapp -o jsonpath='{.status.numberUnavailable}'

A non-zero numberUnavailable indicates nodes where pods couldn't be scheduled.

Debugging Control Loop Issues

Check Controller Status

Use the Kubernetes API directly to inspect controllers:

kubectl get deployment myapp -o json | jq '.status'

Look for these fields:

  • .status.replicas: Total number of pods
  • .status.readyReplicas: Pods that are ready
  • .status.updatedReplicas: Pods with latest version
  • .status.availableReplicas: Ready pods that are available

Monitor Event Logs

Watch events related to your workload:

kubectl get events --sort-by=.metadata.creationTimestamp | grep -E "(myapp|Deployment|ReplicaSet)"

Look for these event types:

  • FailedScheduling: Pod couldn't be scheduled
  • FailedCreate: Controller failed to create pod
  • SuccessfulCreate: Pod successfully created

Inspect Pod Conditions

Pod conditions show why a pod is in a particular state:

kubectl get pods -l app=myapp -o jsonpath='{.items[*].status.conditions}'

Check for:

  • Ready: False with reason ContainerCreating
  • Ready: False with reason CrashLoopBackOff
  • Ready: False with reason ImagePullBackOff

Configuration Gotchas

Replica Count Mismatches

When manually scaling a Deployment, the controller may not respond as expected:

kubectl patch deployment myapp -p '{"spec":{"replicas":0}}'
kubectl get deployment myapp -o jsonpath='{.status.replicas}'

The status field may lag behind the spec until the controller reconciles.

Resource Limits and Requests

Misconfigured resource requests can cause pods to stay in Pending state:

resources:
  requests:
    memory: "64Mi"
    cpu: "250m"
  limits:
    memory: "128Mi"
    cpu: "500m"

If the cluster doesn't have enough resources, pods will remain Pending.

Image Pull Policies

The default image pull policy is IfNotPresent. For development:

imagePullPolicy: Always

This ensures the latest image is pulled each time. Without this, stale images can persist.

Silent Failure Patterns

Controller Loop Delays

Controllers may not react immediately to changes. The reconciliation loop runs every 30 seconds by default. This means:

kubectl scale deployment myapp --replicas=10
kubectl get deployment myapp -o jsonpath='{.status.replicas}'

The status might still show replicas: 3 for up to 30 seconds.

Rollout Race Conditions

When updating multiple resources in sequence:

kubectl set image deployment/myapp nginx=nginx:1.21
kubectl scale deployment/myapp --replicas=5

The controller may race between these operations, potentially leaving intermediate states.

Node Taints and Tolerations

Pods can be stuck in Pending due to taints without matching tolerations:

tolerations:
- key: "node-role.kubernetes.io/master"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"

Check node taints with:

kubectl get nodes -o jsonpath='{.items[*].spec.taints}'

Handling Cluster Failures

API Server Outages

If the API server is unreachable, controllers cannot reconcile:

kubectl get pods --all-namespaces | head -5

This command will hang or return connection errors. The controller logs may show:

error syncing pod: failed to create pod: Internal error occurred: failed calling webhook

Node Failures

When nodes go down, pods are rescheduled:

kubectl get nodes -o jsonpath='{.items[*].status.conditions[*].type}'

Look for Ready: False or DiskPressure: True.

Controller Pod Crashes

If a controller pod crashes, it will restart automatically. Check the controller manager logs:

kubectl logs -n kube-system deployment/kube-controller-manager

Errors like:

E1025 14:30:01.123456 12345 controller.go:123] error syncing deployment: failed to get pod: pods "myapp-7b5b8c9f4d-xyz12" not found

Recovery Patterns

Manual Controller Reconcile

Force a controller to reconcile by deleting its pod:

kubectl delete pod -n kube-system -l component=kube-controller-manager

This triggers a new controller manager that will begin reconciling all resources.

Deployment Rollback

If a deployment is broken, rollback to a previous version:

kubectl rollout undo deployment/myapp --to-revision=1

Check the rollout status:

kubectl rollout status deployment/myapp

StatefulSet Recovery

If a StatefulSet pod fails to come up:

kubectl delete pod myapp-0

The controller will recreate it. If this fails repeatedly, check for persistent volume issues.

Configuration Best Practices

Always set explicit resource requests and limits:

resources:
  requests:
    memory: "128Mi"
    cpu: "250m"
  limits:
    memory: "256Mi"
    cpu: "500m"

Use imagePullPolicy: Always for development deployments to avoid stale images.

Set appropriate replica counts based on workload requirements. For critical services:

replicas: 3
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 1
    maxSurge: 1

This ensures availability during updates.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.