Capacity Planning

Capacity planning is the process of determining the computing resources required to meet current and future demand for services. It matters when systems must scale predictably without unexpected outages or performance degradation.

Understanding Resource Consumption Patterns

Capacity planning requires measuring actual usage patterns across CPU, memory, disk I/O, and network bandwidth. Tools like top, htop, or vmstat provide real-time system metrics. On Linux, the command cat /proc/stat shows per-CPU statistics including idle time, which helps compute utilization percentage.

For containerized environments, docker stats displays live resource consumption for running containers. The output includes CPU %, memory usage, and network I/O. A common failure mode occurs when containers are not limited by resource constraints; this leads to resource starvation on shared hosts.

In Kubernetes clusters, kubectl top nodes and kubectl top pods show current resource consumption. These commands query metrics-server endpoints at /apis/metrics.k8s.io/v1beta1/. Misconfiguring resource requests or limits in pod specs causes scheduling issues or unexpected throttling.

Establishing Baseline Metrics

Begin capacity planning by collecting baseline metrics over a period of at least one week, ideally during peak load times. Use sar (System Activity Reporter) to capture system performance data at regular intervals:

sar -u 1 60

This command collects CPU utilization every second for 60 iterations. Similarly, use iostat to monitor disk I/O:

iostat -x 5 10

This outputs extended statistics every five seconds for ten iterations.

For network metrics, iftop provides real-time bandwidth usage per connection:

iftop -i eth0 -t

The -t flag enables text mode output suitable for logging. A silent failure occurs when monitoring tools do not account for time zones or clock drift, leading to misaligned metric aggregation.

Identifying Resource Saturation Points

To determine when a system becomes saturated, monitor resource utilization during high-load scenarios. For CPU-bound workloads, observe if idle time drops below 10% consistently. Memory saturation shows up as increased swap usage (swappiness) or frequent garbage collection cycles.

Disk I/O saturation manifests in high await times from iostat. The key metric is r_await and w_await, representing average read and write latency. If these exceed 10ms, the system may be experiencing bottlenecks.

Network saturation occurs when packet loss increases or latency grows beyond acceptable thresholds. Tools like ping with a high packet rate can help identify network limits:

ping -i 0.1 -c 1000 example.com

This sends 1000 packets at 10Hz, which is more aggressive than typical monitoring. A common mistake is assuming that increased throughput always correlates with better performance; some systems show degraded response times under high load due to resource contention.

Setting Up Resource Limits

Kubernetes pod specifications include resources.requests and resources.limits. These define minimum guaranteed resources (requests) and maximum allowed usage (limits). Misconfiguring these values can lead to either overcommitment or underutilization.

Example pod spec snippet:

resources:
  requests:
    memory: "64Mi"
    cpu: "250m"
  limits:
    memory: "128Mi"
    cpu: "500m"

The cpu value is expressed in millicores, where 1000m equals one full CPU core. Setting limits too low causes throttling, while setting them too high risks overcommitment.

In Docker containers, specify resource constraints using flags:

docker run --memory="128m" --cpus="0.5" myapp

The --memory flag sets maximum memory allocation in bytes or with suffixes like m for megabytes. The --cpus flag accepts floating-point values, where 0.5 equals half a CPU core.

Monitoring Resource Usage

Set up continuous monitoring of resource consumption using Prometheus. Define a scrape configuration that targets the /metrics endpoint exposed by applications:

scrape_configs:
  - job_name: "application"
    static_configs:
      - targets: ["localhost:8080"]

Prometheus stores metrics in an internal time-series database and allows querying with PromQL. A typical query to identify resource usage over time:

rate(container_cpu_usage_seconds_total[5m])

This calculates the per-second rate of CPU usage over the last five minutes. If a system is under-provisioned, this will show consistent high utilization.

Detecting Resource Drift

Resource requirements change over time due to application evolution or increased traffic. Regularly review resource consumption trends and adjust limits accordingly. Use SQL queries against monitoring databases to identify usage patterns:

SELECT 
  date_trunc('hour', timestamp) AS hour,
  avg(cpu_usage_percent) AS avg_cpu,
  max(memory_usage_mb) AS max_memory
FROM metrics
WHERE service_name = 'webapp'
GROUP BY hour
ORDER BY hour;

This query aggregates hourly averages of CPU and memory usage for a specific service. Without periodic review, resource constraints become outdated and cause unexpected failures.

Handling Resource Contention

In multi-tenant environments, resource contention can occur when multiple applications compete for shared resources. The Linux kernel's Completely Fair Scheduler (CFS) manages CPU allocation through cgroups. Ensure proper cgroup configuration to prevent one process from monopolizing CPU time.

Use cgroups tools to inspect resource allocation:

cat /sys/fs/cgroup/cpu/cpu.shares

This shows the relative weight of CPU allocation for a group. Values like 1024 represent default shares, while lower values indicate reduced priority.

Planning for Growth

Capacity planning must consider growth patterns, both linear and exponential. For applications with predictable growth, use historical data to project future needs. For unpredictable growth, employ load testing with increasing concurrency levels.

Run load tests using wrk:

wrk -t12 -c400 -d30s http://localhost:8080/api/users

This sends 400 concurrent connections for 30 seconds. The output includes requests per second (RPS) and latency metrics. If the system cannot maintain consistent response times under load, it indicates insufficient capacity.

Identifying Bottlenecks

When performance degrades, identify bottlenecks using profiling tools. For CPU-bound applications, use perf:

perf top -p <pid>

This shows real-time CPU usage by function. Memory leaks appear as increasing RSS (Resident Set Size) values. Use valgrind or similar tools to detect memory issues:

valgrind --tool=memcheck --leak-check=full ./myapp

A common mistake is treating symptoms as root causes. For example, high CPU usage might appear due to inefficient algorithms, but could also be caused by resource contention or disk I/O saturation.

Scheduling Capacity Updates

Capacity planning is not a one-time activity. Schedule regular reviews of resource consumption and system performance. Set up automated alerts when utilization exceeds certain thresholds:

# Example alert script
if [ $(df / | awk 'NR==2 {print $5}' | sed 's/%//') -gt 80 ]; then
  echo "Disk usage over 80%"
fi

This shell script checks disk usage and triggers an alert when it exceeds 80%. Automated capacity adjustment requires careful consideration of rolling updates, blue-green deployments, or canary releases.

Handling Over-Provisioning

Over-provisioning resources incurs unnecessary costs. Monitor resource utilization to identify when systems are underutilized. Tools like free show available memory:

free -m

This displays memory usage in megabytes. If available memory consistently exceeds 50% of total, consider reducing resource limits.

Under-provisioning causes more immediate issues, including timeouts, dropped requests, and application failures. A key indicator is an increasing number of requests failing with timeout errors or connection resets.

Managing Resource Constraints

When capacity is constrained, prioritize critical services using Quality of Service (QoS) classes. Kubernetes supports three QoS classes: Guaranteed, Burstable, and BestEffort.

In Guaranteed QoS:

resources:
  requests:
    memory: "64Mi"
    cpu: "250m"
  limits:
    memory: "64Mi"
    cpu: "250m"

Both requests and limits are equal, ensuring guaranteed resource allocation. In contrast, Burstable QoS allows containers to use more resources when available:

resources:
  requests:
    memory: "32Mi"
    cpu: "100m"
  limits:
    memory: "128Mi"
    cpu: "500m"

This configuration allows the container to consume up to 128MB of memory, but guarantees only 32MB. BestEffort QoS provides no resource guarantees, which can lead to unpredictable behavior.

Testing Capacity Limits

Before deploying new capacity plans, test them under realistic conditions. Use stress-ng to simulate various resource constraints:

stress-ng --vm-bytes 128M --vm 4 --timeout 60s

This command creates four virtual memory workers each using 128MB of memory for 60 seconds. Monitor system response during stress tests.

Addressing Unexpected Events

Capacity planning must account for unexpected events like sudden traffic spikes or hardware failures. Implement circuit breakers to prevent cascading failures:

# Example configuration for a circuit breaker in an application
circuit_breaker:
  timeout: 1000
  max_failures: 5
  reset_timeout: 60000

These settings define how long to wait before retrying failed requests, how many failures are acceptable before tripping the breaker, and how long to wait before resetting the breaker after success.

Tracking Capacity Changes

Document capacity changes in version control alongside application code. Use Git hooks or CI/CD pipelines to ensure that resource configuration changes are reviewed and tested before deployment.

Example commit message:

feat: increase memory limit for webapp from 128Mi to 256Mi

This ensures that capacity decisions are traceable and can be reverted if necessary.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.