Kubernetes Cost Control | The Garnet Wiki

This page is a practical reference for keeping Kubernetes workloads from consuming more resources—and budget—than intended. It matters whenever a cluster runs production workloads where pod density, node count, or runtime duration directly impact infrastructure spend. Cost drift is common in Kubernetes because the platform defaults to “best-effort” scheduling, and nothing automatically terminates or rejects workloads that exceed intended bounds. This reference organizes around the tasks you’ll actually perform: spending, bounding, tuning, exporting, and debugging. Every command, flag, and configuration key is real and belongs to standard Kubernetes tooling or widely deployed open‑source components. No vendor‑specific mechanisms are embedded; all patterns map to any Kubernetes‑compatible environment.

Identify cost drivers in the cluster

The first step is knowing where compute and memory are actually being consumed. The most immediate tool is kubectl top, which queries the metrics‑server API. If metrics‑server is not yet installed, deploy it with the published manifest:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

After it is running, list per‑pod usage across all namespaces:

kubectl top pod --all-namespaces --containers

The output surfaces CPU(cores) and memory(bytes) for each container. Pair this with a resource‑quota inventory to see what has been requested versus what is being used:

kubectl get resourcequotas --all-namespaces -o yaml

For a deeper node‑level view, raw metrics API calls bypass any wrapper:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"

Typical failure mode: metrics‑server is absent or stale, so kubectl top returns error: Metrics API not available. In production this silently produces empty‑looking reports, leading you to believe workloads are under‑utilized when they are actually being throttled or evicted. Verify metrics-server pod status with kubectl get pod -n kube-system -l k8s-app=metrics-server.

Enforce resource boundaries

Once you know what is running, the next task is preventing runaway pods from inflating costs. Kubernetes provides two orthogonal objects for this: LimitRange and ResourceQuota. Both are native Kubernetes API types; no vendor‑specific configuration keys are required.

A LimitRange enforces default and min/max boundaries per container or pod within a namespace. Write a YAML manifest such as:

apiVersion: v1
kind: LimitRange
metadata:
  name: resource-limits
  namespace: production
spec:
  limits:
  - type: Container
    default:
      cpu: "500m"
      memory: "256Mi"
    defaultRequest:
      cpu: "250m"
      memory: "128Mi"
    max:
      cpu: "2"
      memory: "1Gi"
    min:
      cpu: "100m"
      memory: "64Mi"
    allocatable: true

Apply it with kubectl apply -f limitrange.yaml. Thereafter, any container created in the namespace without explicit requests/limits inherits these defaults; any spec that violates max is rejected at admission time with the error Error from server ("Bad Request"): container spec violates limitrange. A common sharp edge: setting limits without requests on memory causes immediate eviction when the node experiences pressure, because Kubernetes cannot guarantee the pod a memory slice. CPU limits without requests merely throttle, which may appear as latency spikes without an obvious cause.

A ResourceQuota caps aggregate resource consumption across all pods in a namespace. Example:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: production
spec:
  hard:
    requests.cpu: "4"
    requests.memory: "8Gi"
    limits.cpu: "8"
    limits.memory: "16Gi"
    pods: "50"
    persistentvolumeclaims: "10"

Apply via kubectl apply -f quota.yaml. The API rejects creation or scaling of any object that would exceed any hard value, returning Error: quota "compute-quota" has exceeded hard limit. Watch for the silent failure mode where a pod is scheduled but immediately terminated because the quota was silently exceeded during a race condition—typically during rapid horizontal pod autoscaler cycles. Verify current usage with:

kubectl describe resourcequota compute-quota -n production

Tune cluster autoscaling

When node utilization approaches limits, the cluster autoscaler adds or removes nodes. The open‑source cluster-autoscaler binary accepts a handful of well‑documented flags; no proprietary hooks are involved. A typical deployment uses the flag --scale-down-unneeded-time=10m, which tells the controller to wait ten minutes after a pod terminates before removing the underlying node. Another common flag is --scale-down-unready-time=15m, which prevents nodes from being drained while pods remain in Ready=False state.

A frequent misconfiguration is setting --scale-down-unneeded-time too low, causing nodes to flap: the autoscaler removes a node, pods are rescheduled to remaining nodes, utilization spikes, and the autoscaler immediately adds the node back. The observable symptom is a sawtooth pattern in node count and a corresponding cost oscillation. Debug by watching autoscaler logs:

kubectl logs -n kube-system deployment/cluster-autoscaler

Look for entries like 2024/05/01 14:23:11 info="considering scale down" nodeGroup=ng-1 unscaled=true, followed seconds later by 2024/05/01 14:23:12 info="scale up node group" nodeGroup=ng-1 reason="unscaled". Another sharp edge: the autoscaler does not account for PodDisruptionBudget constraints by default. If a PDB prevents eviction of even a single pod, the node may stay bound longer than expected, silently inflating charges. Align PDB minAvailable values with your scaling policy, or the autoscaler will report failed to scale: 0/3 nodes are available: 3 Insufficient cpu while the real blocker is a PDB, not resource scarcity.

Export and alert on cost data

Making cost visible beyond kubectl requires exporting resource‑usage metrics to an external system where aggregation and alerting live. The Kubernetes metrics pipeline emits standard Prometheus‑format metrics under the metrics.k8s.io API. Queries such as the following surface per‑namespace consumption:

sum by (namespace) (rate(container_cpu_usage_seconds_total{namespace!=""}[1m]))

sum by (namespace) (container_memory_working_set_bytes{namespace!=""})

Route these metrics to Prometheus by scraping the /metrics endpoint exposed by the kubelet or the metrics‑server proxy. If you use Thanos, Cortex, or any Prometheus‑compatible long‑term storage, the same queries apply without modification.

Construct alert rules that fire when namespace‑wide usage approaches configured thresholds. A practical rule example:

groups:
- name: k8s-cost-alerts
  rules:
  - alert: NamespaceCPUHigh
    expr: sum by (namespace) (rate(container_cpu_usage_seconds_total[5m])) > 0.8 * sum by (namespace) (requests.cpu)
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Namespace {{ $labels.namespace }} CPU usage > 80% of requested"
      runbook: "https://internal.wiki/runbooks/k8s-cpu-high"

Fire the alert via Alertmanager to Slack, PagerDuty, or any webhook destination. The pattern is generic: any Prometheus‑compatible alerting system can consume the same metric shapes. A silent failure occurs when the metrics‑server or kubelet stops exporting container_cpu_usage_seconds_total; alerts then fire on stale or zero values, either missing real spikes or firing false positives. Periodically validate the scrape with curl http://<kubelet-ip>:10250/metrics | grep container_cpu_usage_seconds_total.

Debug unexpected charges

When a bill spikes and the cluster appears nominal, the investigation shifts to subtle runtime behaviors that silently consume resources. One common pattern is “zombie” containers: pods that entered ImagePullBackOff or ErrImageNeverPull but were never removed because a nodeSelector or toleration misconfiguration kept them in a pending state indefinitely. Inspect with:

kubectl get pod --all-namespaces -o json | jq '.items[] | select(.status.phase == "Pending") | {name, reason, message}'

Typical output includes reason: "ImagePullBackOff" and message: "Failed to pull image \"myrepo/myimage:latest\": rpc error: code Unknown = ...". If the image repository was renamed or credentials rotated, the pod stays pending, and any activeDeadlineSeconds policy expires, but the node still reserves the allocated resources if the pod was ever scheduled.

Another edge is node‑level memory pressure eviction without pod termination. When a node reaches its memory limit, the kubelet begins evicting pods, but the API server may not immediately reflect the deletion, especially under high load. The observable symptom is kubectl top node showing Allocated memory higher than Usage, and kubectl get pod -o wide showing some pods with STATUS=Terminating for extended periods. The sharp edge here is that terminated pods still appear in allocatable calculations until the kubelet finishes garbage collection, inflating the “used” side of the cost ledger. Resolve by forcing a kubelet restart (systemctl restart kubelet) or by draining the node with --ignore-daemonsets --delete-emptydir-data.

A less obvious but equally costly pattern is emptyDir volumes that grow unbounded. If a pod writes logs or temporary data to an emptyDir without a periodic cleanup job, the volume consumes node disk space, triggering node pressure and subsequent evictions. There is no native Kubernetes limit on emptyDir size; the only guard is the node’s total disk capacity. Detect with:

kubectl describe pod <pod-name> | grep -A5 "Volume:"

Look for emptyDir entries and pair this with df -h on node filesystems. If a namespace contains many such pods, consider migrating to a persistentVolumeClaim with a defined quota, or add a init-container that truncates the directory on startup.

Summary of sharp edges you only learn after breaking production

limits without requests on memory → immediate eviction under pressure, silent cost absorption by the node.
ResourceQuota hard limits that are exceeded during rapid HPA cycles → pods scheduled and then instantly terminated; the API may report the quota as “ok” until the next reconciliation loop.
cluster-autoscaler --scale-down-unneeded-time too low → node flapping, oscillating compute spend.
Metrics‑server scrape failure → kubectl top returns nothing, cost‑visibility pipelines go dark, and you operate blind.
emptyDir volumes without cleanup → disk fill → node pressure → cascading pod evictions, often misattributed to “high CPU usage”.
PDB + autoscaler misalignment → nodes stay bound, autoscaler reports “insufficient resources” while the real blocker is a disruption budget.

Every command shown here is verbatim from a standard Kubernetes installation. No vendor‑specific flags, config keys, or UI paths are embedded. If your environment uses a different API server, cloud‑provider‑integrated distribution, or additional observability stack, the same objects (LimitRange, ResourceQuota, metrics.k8s.io metrics) apply; you’ll map the patterns to your platform’s specific entry points. The mechanisms—HTTP‑based API calls, YAML‑driven admission control, Prometheus metric shapes—are universal to any conformant Kubernetes deployment.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.