Kubernetes Resource Management: Stop Wasting Money on Over-Provisioned Pods

Master Kubernetes resource requests, limits, QoS classes, and horizontal/vertical autoscaling. Covers the real cost of over-provisioning, how to rightsize workloads, and the resource management patterns that save money without risking stability.

Kubernetes resource management is the art of matching CPU, memory, and storage to the real workloads of your applications. It matters when your clusters run for months with consistent pod counts, but still incur high costs from underused or over-provisioned pods. This page details the precise, repeatable actions to identify, diagnose, and fix over-provisioned pods—specifically, pods that consume more resources than their workloads justify.

Diagnose Over-Provisioned Pods

Start with a query that reveals resource utilization across the cluster. Use kubectl to gather metrics from the metrics-server and kubectl top for a snapshot of CPU and memory usage.

kubectl top pods --all-namespaces

This command outputs a table with the actual CPU and memory usage per pod. A pod consuming 1.5 GiB of memory but only using 100 MiB of resident set size (RSS) is a candidate for over-provisioning. To capture this over time, use kubectl top pods in a loop:

kubectl top pods --all-namespaces --interval=30s --watch

For deeper insight, use kubectl get pods -o json on a suspicious pod and examine the resources section in its spec:

resources:
  requests:
    cpu: "1"
    memory: "2Gi"
  limits:
    cpu: "2"
    memory: "4Gi"

A pod with a memory limit of 4 GiB but consistently using only 1.2 GiB is over-provisioned. The key signal is a gap between requests and limits, especially when limits are larger than what the workload requires.

To correlate this with real memory pressure, inspect the containerStatuses in the pod’s status:

kubectl get pod my-app-789f4d2c -o jsonpath='{.status.containerStatuses[0].memoryUsage}'

If memoryUsage is consistently below 2 GiB while the limits.memory is 4 GiB, the pod is over-allocated.

Identify the Root Cause: Requests vs. Limits vs. Actual Usage

The mismatch between requested, limited, and actual resource consumption is the core of over-provisioning. Use kubectl describe pod to see how the scheduler placed the pod and how resources were allocated:

kubectl describe pod my-app-789f4d2c

Look for:

A node with 16 GiB of memory, 8 CPUs, and 20 pods, each with 2 GiB memory requests, shows 40 GiB of requested memory—more than double the node’s capacity. This indicates that many pods are over-provisioned, and the cluster is underutilizing its memory.

Use kubectl top nodes to see node-level memory pressure:

kubectl top nodes --all-namespaces

If the Memory column shows 80% utilization on most nodes but kubectl describe node reveals Allocated resources at 100% for memory, then memory is the bottleneck and the cluster is likely oversubscribed.

Diagnose Memory Pressure and Garbage Collection

When memory usage is high but not fully utilized, memory pressure is the culprit. Check the pod’s log for GC events. In a Java-based pod, look for GC logs in /logs/gc.log or via journalctl:

journalctl -u my-app --since="1 hour ago" | grep "GC"

Look for:

In a Go application, use pprof to analyze heap profiles:

curl -s http://my-app:8080/debug/pprof/heap?seconds=30 > heap.pprof

Then use pprof to visualize the heap:

pprof -svg heap.pprof > heap.svg

A heap profile with many small, uncollected objects and a high live heap size indicates that the application is allocating memory faster than it can reclaim it.

Correlate with Application Metrics

For a complete view, pull application metrics from Prometheus. Query for container_memory_usage_bytes and container_cpu_usage_seconds_total:

rate(container_cpu_usage_seconds_total{container="app"}[5m])

Then overlay this with container_memory_usage_bytes and container_memory_working_set_bytes. If working_set is consistently below usage, then the application is using more memory than it needs, and the OS is caching data in memory that the app does not touch.

Also query for container_memory_rss and container_memory_cache. If rss is much smaller than usage, then the process is not using its full memory footprint, and the difference is mostly cache—suggesting that the memory limit is too large.

Fix Over-Provisioned Pods

Adjust Resource Requests and Limits

Start by adjusting resources.requests and resources.limits in the pod spec. For a pod with 2 GiB memory requests and 4 GiB limits, but using only 1.2 GiB, reduce the request to 1 GiB and the limit to 2 GiB.

Update the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 5
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: app
          image: my-app:v1.2
          resources:
            requests:
              memory: "1Gi"
              cpu: "500m"
            limits:
              memory: "2Gi"
              cpu: "1"

Apply the change:

kubectl apply -f deployment.yaml

After rolling update, verify the new resource usage:

kubectl top pods --all-namespaces

If the memory usage now stays around 1.8 GiB, and the pod’s memoryUsage in kubectl describe pod is close to 2Gi, the new limits are well-tuned.

Use Horizontal Pod Autoscaler (HPA) to Adapt to Load

For dynamic workloads, use HorizontalPodAutoscaler to scale pods based on CPU or memory.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 60

Apply the HPA:

kubectl apply -f hpa.yaml

The HPA will now scale the deployment up when CPU or memory utilization exceeds 70% or 60%, respectively. Monitor its behavior:

kubectl get hpa my-app-hpa -w

When the CurrentReplicas increases, check kubectl describe hpa my-app-hpa for the reason behind scaling decisions.

Tune the Scheduler for Memory Pressure

When pods are over-provisioned, the scheduler may not place pods efficiently. Use kubectl get pods -o wide to see which node each pod is on, and inspect the nodeSelector and affinity fields.

If many pods are on the same node, but the node is memory-constrained, add nodeAffinity to prefer nodes with more memory:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: "memory"
              operator: In
              values: ["high", "extra"]

Also, set resources.limits to ensure memory is respected during scheduling:

resources:
  limits:
    memory: "2Gi"
    cpu: "1"

Use kubectl describe node to see how the scheduler allocates resources and where pods are placed.

Reduce Memory Overhead with Pod-Level Configurations

For pods with low memory usage, reduce overhead by tuning resources.requests and resources.limits per container, and by optimizing the container image.

Use docker inspect to examine the image size and layer structure:

docker inspect my-app:v1.2

Look for:

If the image is 500 MiB but the running container uses only 150 MiB, then the image is bloated.

Optimize the image with docker build and --squash:

docker build --squash -t my-app:v1.3 .

Then reduce the base image to use alpine or distroless:

FROM golang:1.20-alpine AS builder


FROM golang:1.20-alpine AS runtime
COPY --from=builder /app/my-app /app/my-app
ENTRYPOINT ["/app/my-app"]

This reduces the image size from 500 MiB to 100 MiB, and the running container uses only 120 MiB, making it easier to tune memory limits.

Automate and Monitor

Set up a monitoring pipeline to detect over-provisioned pods daily.

Use a script to gather metrics and compare them:

#!/bin/bash
# detect-overprovisioned-pods.sh

for pod in $(kubectl get pods --all-namespaces -o jsonpath='{.items[*].metadata.name}'); do
  namespace=$(kubectl get pod "$pod" -o jsonpath='{.metadata.namespace}')
  memory_limit=$(kubectl get pod "$pod" -n "$namespace" -o jsonpath='{.spec.containers[0].resources.limits.memory}')
  memory_usage=$(kubectl get pod "$pod" -n "$namespace" -o jsonpath='{.status.containerStatuses[0].memoryUsage}')

  # Parse memory values (e.g., "2Gi", "100Mi")
  limit_bytes=$(echo "$memory_limit" | sed 's/Gi/000000000/g; s/Mi/000000/g; s/Ki/000/g; s/[^0-9]//g')
  usage_bytes=$(echo "$memory_usage" | sed 's/Gi/000000000/g; s/Mi/000000/g; s/Ki/000/g; s/[^0-9]//g')

  if [ "$usage_bytes" -lt "$((limit_bytes * 2 / 3))" ]; then
    echo "Over-provisioned: $namespace/$pod - Usage: $memory_usage, Limit: $memory_limit"
  fi
done

Schedule this script with cron to run daily and send alerts when over-provisioned pods are found.

Handle Silent Failures

Be aware of silent failures in resource management:

Common Pitfalls

Summary

Over-provisioned pods cost money without delivering performance. To fix them:

The key insight: a pod is over-provisioned when its actual memory usage is significantly below its memory limit, and the scheduler is allocating more memory than the workload needs.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.