Capacity Planning: Scaling Infrastructure Before You Need To

Capacity planning ensures systems can meet expected demand without over-provisioning or under-provisioning resources. This process focuses on identifying bottlenecks early, before performance degrades or outages occur. It is critical during growth phases, before major events, and when system behavior changes.

Assess Current Resource Utilization

Use kubectl top to view real-time CPU and memory usage across nodes and pods.

kubectl top node
kubectl top pod -n namespace-name

Monitor these metrics over time with a tool like Prometheus or InfluxDB. The key is tracking sustained load, not momentary spikes. For example, a pod may show 50% CPU usage during a 10-minute window but consistently exceed 80% during business hours.

Check system logs for resource contention errors:

OOMKilled
Error: failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused "process_linux.go:545: container init caused: write /sys/fs/cgroup/cpu,cpuacct/kubelet.slice/.../cpu.cfs_quota_us: invalid argument: unknown

These errors indicate a lack of CPU or memory allocation that must be addressed before scaling.

Identify Scaling Boundaries

Use kubectl describe to inspect pod and node resource limits:

kubectl describe pod pod-name -n namespace-name
kubectl describe node node-name

Look for events indicating resource exhaustion or scheduling failures. Common messages include:

Evicted due to NodeOutOfCPU
Evicted due to NodeOutOfMemory
FailedScheduling: 0/3 nodes are available: 3 Insufficient memory.

If a pod fails to start because of insufficient resources, check its requests and limits in the deployment manifest:

resources:
  requests:
    memory: "64Mi"
    cpu: "250m"
  limits:
    memory: "128Mi"
    cpu: "500m"

The system may schedule pods based on requests, but enforce limits at runtime. A pod that exceeds its memory limit will be terminated with an OOMKilled event.

Predict Future Load Requirements

Calculate expected load using historical metrics and projected growth. Use tools like kubectl get pods --sort-by=.status.startTime to identify how many pods are running in parallel during peak hours.

For CPU, compare actual usage against requested capacity:

kubectl top pod -l app=frontend --sort-by=cpu

If a deployment's CPU usage consistently reaches 80% of its limit, plan to increase the limit or add more replicas. The rule of thumb is to maintain headroom: avoid running at 95% utilization unless you have specific monitoring and alerting in place.

Plan Replicas and Horizontal Scaling

When scaling horizontally, consider both pod count and resource distribution. Use kubectl scale to adjust replica counts:

kubectl scale deployment frontend --replicas=10

Monitor the impact of scaling on service latency. If increasing replicas causes performance degradation, it might indicate a bottleneck in another part of the system — for example, database connection limits or shared cache contention.

Check if your system handles graceful shutdowns properly. When pods terminate, they should finish processing requests before exiting. Kubernetes sends SIGTERM to pods when terminating them, so applications must handle this signal appropriately.

Set Resource Limits and Requests Correctly

Misconfigured resource requests and limits cause unnecessary scheduling failures or overcommitment. If a pod's request is too low, it may be scheduled onto nodes with insufficient capacity. If limits are too high, the pod can monopolize resources, starving other workloads.

Use kubectl apply to update deployments with precise values:

resources:
  requests:
    memory: "128Mi"
    cpu: "500m"
  limits:
    memory: "256Mi"
    cpu: "1000m"

Avoid setting limits without matching requests. A pod with no request but a limit of 2GB may be scheduled on a node with only 1GB of free memory, leading to frequent evictions.

Monitor and Adjust Automatically

Set up alerts in your monitoring system to trigger when resource usage approaches thresholds. For example, alert when CPU utilization hits 80% for more than 15 minutes.

In Kubernetes, use Horizontal Pod Autoscaler (HPA) to automatically scale based on metrics:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: frontend-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: frontend
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

HPA adjusts replicas based on average CPU usage. If your workload is memory-bound, you can define custom metrics instead:

- type: Resource
  resource:
    name: memory
    target:
      type: AverageValue
      averageValue: 200Mi

If HPA doesn't respond as expected, check for missing metrics or incorrect metric types. A common mistake is using averageUtilization when the workload is memory-bound.

Handle Resource Contention

When multiple workloads share nodes, resource contention can occur even if individual pods have adequate limits. Use node labels and taints to isolate critical applications:

kubectl taint nodes node-name dedicated=production:NoSchedule

Then schedule pods with tolerations:

tolerations:
- key: "dedicated"
  operator: "Equal"
  value: "production"
  effect: "NoSchedule"

This ensures critical workloads don't get evicted due to resource pressure from less important services.

Prepare for Unexpected Events

Design systems to handle bursts in traffic or temporary failures gracefully. For instance, a service that uses Redis may experience slowdowns if the Redis instance is overloaded. Ensure your application has timeouts and retry logic to prevent cascading failures.

Monitor system logs during high-load periods to catch silent failures. A component may continue working but respond slowly. If response times increase beyond acceptable thresholds, investigate whether it's due to resource constraints or logic issues.

Evaluate Infrastructure Costs

Balance performance with cost. Over-provisioning increases expenses, while under-provisioning risks outages. Use kubectl top and your monitoring system to track actual usage against allocated resources.

If a deployment consistently runs below 20% CPU usage, consider reducing its resource allocation. If it exceeds 90% consistently, plan for more resources or better load distribution.

Use cost analysis tools like kube-cost or Prometheus with Grafana to correlate resource usage with billing data. This helps identify which workloads consume the most resources and whether scaling decisions are economically justified.

Plan for Cluster Growth

When adding new nodes, ensure they have compatible resource profiles. Check that new nodes have sufficient memory and CPU capacity to support existing workloads. Use kubectl drain before removing nodes:

kubectl drain node-name --ignore-daemonsets

This safely evicts pods from a node and redistributes them across other nodes. Verify that all pods can reschedule without issues, especially those with local storage or specific node affinity rules.

Handle Scaling Failures

If scaling fails due to resource constraints, check the cluster state:

kubectl get nodes
kubectl describe node node-name

Look for taints, conditions, or events indicating why nodes became unschedulable. If a node is marked as NotReady, investigate whether it's due to hardware failure, network issues, or resource exhaustion.

If pods are stuck in Pending state, review their scheduling constraints:

kubectl describe pod pod-name -n namespace-name

Look for events like 0/3 nodes are available: 1 node(s) didn't match node selector, which indicates a misconfigured node selector or label.

Validate Scaling Changes

After implementing capacity changes, validate that the system behaves as expected. Use load testing tools to simulate real-world traffic patterns and confirm that new limits or replica counts meet performance targets.

Monitor for regressions in application behavior. If a scaling change introduces latency or increases error rates, investigate whether it's due to resource contention or configuration errors.

Keep logs of all changes made during capacity planning sessions. This helps identify which modifications caused specific issues and allows for quick rollbacks if needed.

Use kubectl get events to check recent system events:

kubectl get events --sort-by=.metadata.creationTimestamp

This command shows the most recent events in chronological order, helping to correlate scaling actions with system behavior.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.