Predict infrastructure capacity needs before they become outages. Covers demand forecasting models, resource utilization projections, capacity planning automation, and the patterns that ensure infrastructure scales ahead of growth instead of behind it.
Capacity forecasting in SRE is the process of predicting future resource needs based on historical usage patterns, workload characteristics, and growth trends to prevent performance degradation or outages before they occur. This technique becomes essential when systems must scale predictably, maintain SLA compliance, or operate under constrained infrastructure.
Capacity forecasting requires identifying consistent patterns in how resources are consumed over time. This involves collecting metrics such as CPU utilization, memory usage, disk I/O, network bandwidth, and request volume. The goal is to model these patterns so that future demand can be estimated with sufficient confidence.
For example, a web service might exhibit predictable daily cycles where traffic peaks during business hours and drops overnight. A forecasting model must capture this pattern to avoid under-provisioning during peak times or over-provisioning during low-traffic periods.
Historical data collection should focus on granular measurements that reflect actual system behavior rather than aggregate summaries. Tools like Prometheus, Grafana, or built-in OS monitoring utilities can provide the necessary telemetry.
The key is to gather data points at regular intervals (e.g., every 5 minutes) over a period sufficient to understand seasonal and cyclical trends. For instance, if you're forecasting for a service that runs daily, collecting data for at least 30 days allows you to detect weekly patterns.
Forecasting models typically use time-series analysis techniques. Simple approaches include moving averages or linear regression applied to historical data. More complex models may incorporate machine learning algorithms trained on past performance.
A basic forecasting function might look like:
def forecast_demand(history, window_size=7):
# history is list of (timestamp, value) tuples
recent_data = history[-window_size:]
total = sum(value for _, value in recent_data)
return total / len(recent_data)This approach works well for systems with stable or slowly changing demand patterns. It does not account for sudden changes or external events like marketing campaigns or seasonal shifts.
Statistical methods such as exponential smoothing or ARIMA (Autoregressive Integrated Moving Average) can improve forecast accuracy by accounting for trends and seasonality in the data.
Exponential smoothing applies a weighted average where recent observations are given more importance. The smoothing factor α controls how much weight is placed on recent data:
forecast[t+1] = α * actual[t] + (1 - α) * forecast[t]ARIMA models require more parameters and can capture complex temporal dependencies but also introduce greater complexity in implementation and tuning.
Once forecasts are generated, they must be used to trigger scaling actions. Kubernetes horizontal pod autoscalers (HPA) can respond to custom metrics provided by forecasting systems.
Example HPA configuration using a custom metric:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: example-app
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: example-app
minReplicas: 2
maxReplicas: 20
metrics:
- type: External
external:
metric:
name: forecasted_requests_per_second
target:
type: Value
value: "1000"The HPA will adjust replica count based on whether current requests exceed or fall below the forecasted threshold.
Capacity forecasting enables proactive alerting instead of reactive response. Alerts should be configured to trigger when projected resource usage approaches known limits.
For example, if a database has 8GB of RAM and historical data shows it typically uses 7GB, an alert might fire when projected usage exceeds 7.5GB within the next hour.
The alerting system must distinguish between temporary spikes and sustained growth. A short-term spike in CPU usage may not warrant scaling, while consistent growth over several hours indicates a need for capacity planning.
Forecasting models rarely predict perfectly. The confidence interval around each prediction should be tracked and used to set conservative thresholds for scaling or alerting.
A model might predict that demand will be 1000 requests per second with a 95% confidence interval of ±100. When setting an alert threshold, you might use 900 rather than 1000 to account for uncertainty.
Over time, systems evolve in ways that can make historical data less predictive. This drift may result from changes in user behavior, application performance characteristics, or infrastructure modifications.
Monitoring the accuracy of forecasts over time helps detect when models become obsolete. If forecast error consistently exceeds acceptable thresholds, it's time to re-evaluate the underlying assumptions or retrain the model.
Forecasting must account for physical constraints such as memory limits, disk space, network bandwidth, or license limitations. These constraints often have discrete thresholds rather than continuous capacity curves.
For example, a service might be limited by the number of concurrent database connections, which typically has a hard upper bound determined by licensing or configuration settings. Forecasting should include these hard limits to avoid planning for more capacity than is available.
Capacity forecasting should integrate into deployment workflows to ensure new services are provisioned appropriately from day one. This integration involves setting up initial capacity estimates based on similar systems or conservative assumptions.
A CI/CD pipeline might include a step that generates a forecast based on the expected workload profile and validates it against resource constraints before deploying.
While forecasting aims to prevent under-provisioning, it's equally important not to over-provision. Excessive resources increase costs and can mask performance issues.
The balance involves setting appropriate confidence levels in forecasts and choosing scaling policies that respond to actual demand rather than predicted peaks.
Accuracy tracking should include comparing forecasted values against actual measurements. If the system consistently overestimates or underestimates capacity, the model parameters may need adjustment.
Example metrics to track:
Many systems depend on external services or APIs that also require capacity forecasting. The capacity of dependent systems must be considered when planning resource allocation.
For example, if a web application depends on a third-party payment gateway, the forecast for that gateway's capacity should influence how many concurrent users the application can support.
Forecasting models must handle anomalies in data such as sudden spikes or drops in usage. These anomalies may be legitimate but could also indicate system issues or data quality problems.
Anomaly detection can be implemented using statistical methods like Z-scores or by establishing baseline behavior patterns that detect deviations from expected trends.
When forecasting capacity, consider both vertical and horizontal scaling options. Some systems scale better by adding more resources to existing instances (vertical), while others benefit from distributing load across multiple instances (horizontal).
The choice affects how forecasts are interpreted and applied. Horizontal scaling requires consideration of service discovery, load balancing, and data consistency concerns.
Forecasting impacts operations through alerting, automation decisions, and resource allocation. Operators must understand when and how forecasts influence system behavior.
For instance, if a forecast indicates that a service will reach capacity in 24 hours, operational procedures should include steps to either scale resources or implement rate limiting before the limit is reached.
Forecast models should be tested against historical data to validate their accuracy. This testing includes stress testing with various scenarios and edge cases.
A simple validation approach involves running the forecasting model on a subset of historical data and comparing results to known outcomes from that period.
All forecasting processes and assumptions must be documented for future reference. This documentation helps ensure consistency in how capacity is planned and reduces risk when team members change.
Documentation should include:
Several issues commonly arise during capacity forecasting. One frequent problem is failing to account for system load characteristics, such as burstiness or latency requirements.
Another common mistake is assuming that historical patterns will continue unchanged into the future. Changes in user behavior, technology stack, or business conditions can invalidate previous assumptions.
Failure to monitor forecast accuracy over time leads to outdated models that no longer reflect current system behavior. Regular review and updating of forecasting approaches is essential for maintaining their effectiveness.
Capacity forecasting is a critical tool for SRE teams seeking to maintain system reliability while optimizing resource usage. Effective forecasting requires careful data collection, appropriate modeling techniques, and continuous monitoring of model performance. The key is to balance precision with practicality, ensuring that forecasts inform decisions without becoming overly complex or brittle.
This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy