Production-grade guide to ai capacity prediction covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.
Ai Capacity Prediction is the practice of estimating the future resource demands—compute, memory, storage, and network—of AI workloads to ensure systems remain performant, cost-efficient, and resilient under load. It matters when deploying AI models at scale: during model serving, batch inference, training pipelines, or continuous learning systems. It is essential when a model's request rate, input size, or complexity grows unpredictably, and when resource provisioning is not purely manual or reactive.
Predicting capacity for model serving requires forecasting the number of inference requests per second, average latency, and memory footprint per request. The core input is a time series of request arrivals, typically collected via observability tools like Prometheus, OpenTelemetry, or logging pipelines.
To estimate required compute, define a request profile using the model’s input size and processing complexity. For instance, a vision model processing 224×224 RGB images requires 150 MB of memory per request. If the model’s average inference time is 80 ms and the system serves 1,200 requests per second (RPS), the total CPU demand is:
1,200 RPS × 80 ms = 96,000 ms/s = 96 CPU-seconds per secondThis implies a sustained CPU load of 96, or roughly 100% utilization on a 100-core cluster.
Use the following configuration in prometheus.yml to collect request metrics and generate capacity forecasts:
scrape_configs:
- job_name: 'ai-model-serving'
metrics_path: '/metrics'
static_configs:
- targets: ['model-svc-0:8080', 'model-svc-1:8080']
params:
'query': ['rate(model_request_count[5m])', 'avg(model_inference_duration_seconds)']The model_request_count metric is a counter, and the model_inference_duration_seconds is a histogram. Use promql to compute future load:
# Forecast next 24h request volume
rate(model_request_count[1h]) * 24 * 60 * 60
# Estimate memory usage over time
sum by (instance) (
rate(model_request_count[1h]) * 150 * 1024 * 1024 # 150MB per request
)A common failure: underestimating memory due to the model’s resident cache. The model weights are loaded into memory once, but each request adds a new activation tensor. The total memory per instance is:
model_weights + (requests_in_flight × activation_memory_per_request)If the model weighs 4 GB and each request adds 100 MB of activation memory, and the system supports 50 concurrent requests, then each serving instance needs:
4 GB + (50 × 100 MB) = 9 GBTo avoid OOM crashes, monitor container_memory_usage_bytes and process_resident_memory_bytes via cAdvisor or the containerd metrics.
Training pipelines often consist of multiple stages: data preprocessing, feature extraction, model training, and model validation. Each stage has different resource profiles, and the duration of one stage can affect the next.
Predict capacity by analyzing the throughput and volume of data processed per stage. The key is to model the pipeline as a set of queues, each with a rate of input data, processing time, and batch size.
Use dagster or prefect to define a training pipeline. Each job is a node with configurable batch_size, num_workers, and checkpoint_interval. The dagster config file defines:
solids:
data_preprocessing:
config:
input_path: '/data/raw/'
output_path: '/data/processed/'
batch_size: 128
num_workers: 8
shuffle: true
feature_extraction:
config:
model_name: 'bert-base'
input_dataset: 'processed_dataset'
output_path: '/features/'
batch_size: 64
num_workers: 4
model_training:
config:
model_type: 'transformer'
learning_rate: 0.001
epochs: 50
batch_size: 32
gradient_accumulation_steps: 4
checkpoint_interval: 1000To predict capacity, compute the time required per batch and the number of batches per day.
For example, if data_preprocessing takes 40 ms per batch and processes 1 million records per batch, then:
1,000,000 records / 40 ms = 25,000 batches per second
→ 25,000 × 8 workers = 200,000 records per secondNow, if new data arrives at 500,000 records per second, the preprocessing stage can keep up, but only if the batch size is aligned. If the batch size is too small, the pipeline stalls due to frequent I/O and metadata overhead.
The failure mode: batch size mismatch. A batch size of 32 in training, but 128 in preprocessing, causes the training stage to wait for data. The model_training stage starts processing only when a full batch is ready, leading to underutilized GPUs.
To detect this, monitor the pipeline_run_duration and input_queue_size in the orchestration system. Use curl to query the pipeline status:
curl -X GET http://pipeline-api:8000/api/v1/runs?status=running \
-H 'Authorization: Bearer $API_TOKEN' \
-H 'Accept: application/json'Response:
{
"runs": [
{
"id": "run-123",
"pipeline": "training_pipeline",
"status": "running",
"input_queue_size": 127,
"start_time": "2024-04-05T10:00:00Z",
"current_step": "model_training",
"progress": 0.63
}
]
}A high input_queue_size with low progress in model_training indicates that preprocessing is the bottleneck.
To predict capacity, calculate the total GPU hours per training run:
epochs × (dataset_size / batch_size) × (training_time_per_batch)If training takes 200 ms per batch on 8 GPUs, and there are 100,000 batches, then:
50 epochs × 100,000 batches × 200 ms = 1,000,000,000 ms = 1,000,000 seconds ≈ 278 GPU-hoursThis determines the required GPU cluster size. For example, 278 GPU-hours over 48 hours means:
278 / 48 ≈ 5.8 GPUs → 6 GPUs needed continuouslyIn continuous learning, models are retrained daily or hourly based on new data. The capacity prediction task is to estimate the required compute for training, the storage for model artifacts, and the bandwidth for data ingestion.
The key challenge is drift in data volume and feature dimensionality. A model trained on 100,000 features may grow to 150,000 features over time. The training time grows not only due to more data but also due to higher feature dimensionality.
Use kubernetes to manage the retraining pipeline. Define a CronJob that triggers a Job daily:
apiVersion: batch/v1
kind: CronJob
metadata:
name: model-retraining
spec:
schedule: "0 2 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: retrain
image: ai-retrain:v1.2
env:
- name: INPUT_DATA_PATH
value: "/data/ingested/"
- name: OUTPUT_MODEL_PATH
value: "/models/"
- name: TRAINING_DURATION_HOURS
value: "6"
resources:
requests:
cpu: "4"
memory: "16Gi"
limits:
cpu: "8"
memory: "32Gi"
restartPolicy: OnFailureThe ai-retrain:v1.2 image runs a script that:
pytorch with accelerate.To predict capacity, monitor the data_ingestion_rate and feature_dimension over time. Use psql to query a metadata table:
SELECT
date_trunc('day', timestamp) as day,
COUNT(*) as records,
AVG(feature_count) as avg_features,
SUM(data_size_bytes) as total_bytes
FROM ingestion_events
GROUP BY day
ORDER BY day DESC
LIMIT 30;If the average feature count increases by 500 features per month, and each feature adds 120 MB of memory during training, then:
120 MB × 500 = 60,000 MB/month ≈ 2 GB/monthAfter 6 months, the memory footprint increases by 12 GB, requiring more memory or more frequent checkpointing.
A silent failure: feature drift without retraining. The model is trained on 100,000 features, but new data arrives with 120,000 features. The model fails to process the new data because of shape mismatches, but the error is not logged. Use kubectl logs to inspect the retraining pod:
kubectl logs model-retraining-1712345678-abcdThe log shows:
WARNING: Feature dimension mismatch: expected 100000, got 120000
INFO: Rebalancing feature indices...
INFO: Training on 120000 features with 15% of data having missing featuresThis indicates that the capacity prediction must include feature dimension change as a critical variable.
To detect this, add a feature_schema table to the database:
CREATE TABLE feature_schema (
model_id UUID,
version VARCHAR(32),
feature_count INT,
last_updated TIMESTAMPTZ,
schema_hash BYTEA
);Each retraining job updates the schema_hash and compares it to the previous version. If the hash changes, the system schedules a data validation job.
Capacity prediction must account for throughput-latency trade-offs, especially in real-time systems. A model may support 1,000 RPS with 100 ms latency, but under load, latency degrades.
Use envoy as a load balancer and model gateway. Define a cluster configuration:
clusters:
- name: model-serving
type: STRICT_DNS
lb_policy: ROUND_ROBIN
http2_protocol_options: {}
connect_timeout: 10s
load_assignment:
cluster_name: model-serving
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: model-svc
port_value: 8080
circuit_breakers:
thresholds:
- priority: HIGH
max_requests: 1000
max_pending_requests: 500
max_connections: 200
- priority: DEFAULT
max_requests: 500
max_pending_requests: 250
max_connections: 100The circuit_breakers configuration prevents overload by limiting the number of concurrent requests. If the number of active requests exceeds 500, new requests are queued or rejected.
To predict when the system will saturate, use the queue_time metric from envoy:
avg(rate(envoy_cluster_upstream_rq_time_seconds_count[5m]))If the average queue time exceeds 50 ms, the system is under stress. The max_pending_requests threshold triggers a circuit breaker open state, which causes 503 Service Unavailable responses.
The failure mode: queue overflow without scaling. When requests arrive faster than the model can process, the queue grows, but the number of model instances does not increase. Use kubectl to scale the deployment:
kubectl scale deployment model-svc --replicas=12But the scaling is manual. To automate, use horizontalpodautoscaler with a custom metric:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: model-svc-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: model-svc
minReplicas: 4
maxReplicas: 50
metrics:
- type: External
external:
metric:
name: queue_length
selector:
matchLabels:
service: model-serving
target:
type: AverageValue
averageValue: "100"
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70The queue_length metric is collected by a prometheus job that scrapes envoy's envoy_cluster_upstream_rq_pending_count. When the average queue length exceeds 100, the system scales up.
Peak load predictions require analyzing historical request patterns, especially seasonal or daily cycles. Use timescaledb to store and analyze time-series data.
Create a hypertable:
CREATE TABLE request_logs (
time TIMESTAMP WITH TIME ZONE NOT NULL,
rps INT,
latency_ms DOUBLE PRECISION,
model_version VARCHAR(32)
) USING timescaledb;
SELECT create_hypertable('request_logs', 'time');Run a query to forecast next week’s peak load:
SELECT
time,
rps,
avg(latency_ms) as avg_latency,
lag(rps) OVER (ORDER BY time) as prev_rps,
(rps - lag(rps) OVER (ORDER BY time)) / lag(rps) OVER (ORDER BY time) as growth_rate
FROM request_logs
WHERE time >= now() - '7 days'::interval
ORDER BY time DESC
LIMIT 24;The growth rate helps predict when the system will exceed capacity. If the growth rate is 15% per day, and the current peak is 1,500 RPS, then:
Day 0: 1,500 RPS
Day 1: 1,725 RPS
Day 2: 1,983 RPS
Day 3: 2,281 RPSBy day 5, the system will be 50% over current peak capacity.
A sharp edge: overprovisioning without cost justification. The system is scaled to handle 2,500 RPS, but average load is only 1,200 RPS. The cost of idle resources is high. To fix this, use adaptive scaling based on both RPS and latency.
Define a capacity_plan table:
CREATE TABLE capacity_plan (
plan_id UUID PRIMARY KEY,
start_time TIMESTAMPTZ,
end_time TIMESTAMPTZ,
replicas INT,
cpu_cores INT,
memory_gb INT,
expected_rps INT,
expected_latency_ms DOUBLE PRECISION,
notes TEXT
);Each month, generate a plan using the forecast:
INSERT INTO capacity_plan (plan_id, start_time, end_time, replicas, expected_rps)
SELECT
gen_random_uuid(),
now(),
now() + '30 days'::interval,
CEIL(1.3 * avg_rps),
CEIL(1.3 * avg_rps)
FROM request_logs
WHERE time >= now() - '1 month'::interval
GROUP BY time_bucket('1 day', time)
ORDER BY avg_rps DESC
LIMIT 1;This ensures that capacity planning is not just reactive, but predictive and integrated into the operational workflow.
This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy