Capacity Planning Advanced

Production-grade guide to capacity planning advanced covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.

This page describes advanced techniques for capacity planning in systems where resource constraints are not only about compute or memory, but also involve complex interdependencies, temporal patterns, and operational subtleties. It applies when standard forecasting tools fall short of capturing system behavior under stress, burst load conditions, or emergent performance characteristics.

Planning for Non-Linear Resource Consumption

Understanding Memory Leaks in Long-Running Processes

Memory usage in long-running services can grow unpredictably due to resource leaks. To detect such issues, inspect process memory maps using pmap:

pmap -x <pid>

This shows virtual memory layout and actual RSS (Resident Set Size). A growing RSS over time without corresponding heap growth may indicate a leak in off-heap allocations like file descriptors, network connections, or thread-local storage.

In systems that manage connection pools, monitor for uncleaned connections:

ss -tuln | grep :<port>

If connections remain in TIME_WAIT state beyond expected timeouts, they may not be properly closed. A common failure mode occurs when a service opens many connections but does not implement proper keepalive or connection timeout logic.

CPU Saturation and Contention

CPU usage is not always linear. When CPU-bound tasks are scheduled concurrently, the kernel can experience scheduling contention. Use perf to observe this:

perf stat -p <pid> -e cpu-cycles,instructions,cache-misses

A high ratio of cache misses compared to instructions suggests memory access patterns that may not scale well under load. If a process shows sustained high CPU usage with low I/O wait time, consider if it's spending cycles in user-space loops rather than waiting for resources.

Managing Storage Throughput and Latency

Predicting Disk I/O Patterns

Disk performance varies significantly based on the type of workload. For sequential workloads, throughput is more important than latency. For random access, latency becomes critical. Use iostat to monitor:

iostat -x 1 5

Watch for %util > 80%, which indicates disk saturation. If await values are high despite low %util, it might be due to queue depth limits or filesystem metadata contention.

In some cases, increasing the number of concurrent read/write operations can improve throughput, but this is limited by the underlying storage architecture. For example, if a database is configured with a single write-ahead log file, increasing concurrency beyond one writer will not help until that file is split.

Database Query Performance and Connection Limits

Database capacity planning requires understanding query execution patterns. Use SQL profiling tools to detect slow queries:

EXPLAIN ANALYZE <query>;

If a query shows high cost but low cardinality, it might be missing an index or using a suboptimal join strategy.

Some platforms limit the number of concurrent database connections. If a service creates new connections for each request and doesn't reuse them, you may hit connection limits before CPU or memory constraints. Use:

psql -c "SELECT count(*) FROM pg_stat_activity;"

To monitor active connections. A high number of idle connections can indicate inefficient connection pooling.

Handling Asynchronous Workloads

Queue Depth and Backpressure

Asynchronous systems often rely on message queues. If a consumer processes messages slower than they arrive, queue depth grows. Monitor this using:

kubectl get pods -o jsonpath='{.items[*].status.containerStatuses[*].ready}'

If containers are reporting readiness but not processing work, check for internal backpressure or deadlock conditions.

In some platforms, consumers can be configured to pause message delivery when their processing queue reaches a threshold. This is typically done through a configuration parameter:

backpressure_threshold: 1000

If this value is too low, it can cause unnecessary pauses; if too high, it may lead to memory exhaustion on the consumer side.

Scaling Workers Based on Workload

Some platforms allow scaling workers based on queue depth or task duration. For example, a worker pool might be configured with:

min_workers: 2
max_workers: 10
target_queue_length: 50

If target_queue_length is set too high, it can cause delayed processing. If it's too low, the system may overscale.

Advanced Monitoring and Alerting

Thresholds for Non-Standard Metrics

When planning capacity, consider metrics that are not directly related to CPU or memory but still affect performance. For instance, if a service uses a shared cache, monitor cache hit ratios:

curl http://localhost:8080/metrics | grep cache_hit_ratio

If this ratio drops below 0.8 and concurrent requests increase, it may be time to scale the cache layer or adjust caching strategy.

Latency SLOs and Tail Behavior

High-percentile latencies often dominate user experience. Monitor P95 and P99 latencies using:

kubectl top pods

If latency increases while resource usage remains stable, consider if there are bottlenecks in shared components like DNS, load balancers, or authentication services.

Detecting Resource Starvation

Some systems can silently fail when resources are constrained. For example, a service might return 200 OK responses even when processing is delayed by several seconds due to memory pressure or GC pauses.

Monitor for these signs using:

kubectl logs <pod-name> --previous

If the previous container instance shows out-of-memory (OOM) events, it indicates that the current instance may also be at risk.

Operational Edge Cases

Cold Starts and Resource Allocation

In containerized environments, cold starts can significantly impact performance. If a service is scaled down to zero and then needs to restart quickly, consider:

kubectl get pods -o jsonpath='{.items[*].status.containerStatuses[*].restartCount}'

High restart counts may indicate that the container is not ready before it receives traffic. Use readiness probes to prevent this:

readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

Handling Burst Traffic Without Overspending

Burst traffic can cause systems to allocate more resources than necessary during peak times. Monitor for this with:

kubectl top nodes

If nodes show sustained high CPU usage but low average utilization, it may indicate inefficient resource allocation.

Use horizontal pod autoscaling with custom metrics:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Pods
    pods:
      metric:
        name: requests_per_second
      target:
        type: AverageValue
        averageValue: 100

Dependency Chain Impact on Capacity

Capacity planning must account for cascading failures. If a service depends on an external API, monitor:

curl -w "@-"

With a custom format that includes response time and status code:

%{time_total} %{http_code}

If response times increase significantly or error rates rise, it may affect the capacity of dependent services.

Handling Data Growth and Retention

Predicting Storage Needs

Storage consumption can grow faster than expected. Use:

du -sh /var/lib/<service>

To monitor disk usage. If storage is not managed automatically, consider implementing retention policies or archival strategies.

In some systems, logs or temporary files can accumulate and fill up disk space. Monitor with:

df -h

If available space drops below 10%, it's time to investigate log rotation or cleanup processes.

Memory Usage in Distributed Systems

In distributed systems, memory usage across nodes may not be evenly distributed. Use:

kubectl get nodes -o jsonpath='{.items[*].status.allocatable.memory}'

To check available memory per node. If a few nodes are consistently using more memory than others, it might indicate unbalanced load distribution or uneven resource allocation.

Failures to Watch For

Silent Resource Exhaustion

Some systems silently degrade rather than failing outright. Monitor for:

kubectl describe pod <pod-name>

Look for events like OOMKilled or Evicted, which indicate that resources were exceeded but the system did not crash.

If a service runs with limited memory and is configured to use swap, it may appear functional while experiencing significant performance degradation due to page faults.

Incorrect Scaling Logic

Scaling decisions based on incorrect metrics can lead to over-provisioning or under-provisioning. If a system scales based on CPU but the workload is I/O-bound, it may not scale appropriately.

For example, a service with:

autoscaling:
  metric: cpu
  target: 70%

May not respond correctly if its actual bottleneck is disk I/O or network bandwidth.

Misconfigured Resource Limits

Containers can be configured with resource limits that are too restrictive. Use:

kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].resources}'

If memory requests and limits are mismatched, it can lead to unexpected eviction behavior or performance degradation.

Ignoring Time-Based Resource Patterns

Capacity planning should account for time-based usage patterns. If a service is configured with fixed resources without considering daily or weekly peaks, it may fail during high-demand periods.

Use:

kubectl get pods --sort-by=.status.startTime

To understand startup patterns and how they affect resource availability.

Misunderstanding Inter-Service Dependencies

When one service consumes another's resources, capacity planning must consider both. If a service uses 100MB of memory per request but depends on a backend that can only serve 50 concurrent requests, the total capacity is limited by the backend, not the frontend.

Monitor this with:

kubectl get pods -o jsonpath='{.items[*].status.containerStatuses[*].lastState}'

To detect if containers are restarting due to dependency failures.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.