Time Series Forecasting in Production

End-to-end time series forecasting for production systems. Covers classical methods, modern ML approaches, forecast evaluation, and building automated forecasting pipelines.

Time series forecasting in production is the practice of generating reliable, automated, and monitored predictions of future values based on historical data, where forecasts are consumed by downstream systems and decisions are made based on them. It matters when systems must anticipate demand, detect anomalies ahead of time, schedule resources, or trigger actions based on predicted trends—such as inventory replenishment, server scaling, ad spend optimization, or electricity load forecasting.

Defining the Forecasting Problem

A forecasting task begins with a dataset of observations recorded at regular time intervals. Each observation is a point in a sequence, with a timestamp and one or more values. The goal is to predict future values of one or more target variables.

In practice, this means defining a forecasting model that takes past data and produces a forecast for a future horizon. The model must be trained, validated, and deployed so that it can generate predictions on a regular schedule or in response to new data.

The core components of a forecasting pipeline are:

Data Ingestion and Preparation

Data must be ingested into the forecasting system with consistent timing and reliability. The ingestion process should ensure that all required features are available at prediction time.

Data is typically stored as a table with a timestamp column and one or more value columns. For example:

CREATE TABLE sales (
    timestamp TIMESTAMPTZ NOT NULL,
    product_id INT NOT NULL,
    units_sold INT,
    revenue NUMERIC
);

A daily ingestion job runs at 02:00 UTC, consuming data from the past 24 hours.


docker run \
  --env DATABASE_URL=postgresql://prod/db \
  --env START_TIME='2024-04-15T00:00:00Z' \
  --env END_TIME='2024-04-16T00:00:00Z' \
  forecast-ingest:latest

The job reads from a source system (e.g., a Kafka topic or a data lake), transforms the raw events into a structured time series, and writes them to the sales table. A failure in ingestion causes a delay in forecasts: if the job fails, the forecast for the next day will miss data from the previous day.

Common failure modes:

Model Training and Versioning

Models are trained periodically—daily, weekly, or on-demand. Training is typically batched and runs in a scheduled job.

The model is defined by a set of hyperparameters, features, and a training algorithm. The training process outputs a model artifact (e.g., a serialized file) and metadata about the training run.

# train_forecast_model.py
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.preprocessing import StandardScaler
from joblib import dump

def train_model(data_path: str, output_path: str, horizon: int = 7):
    df = pd.read_parquet(data_path)
    df = df.sort_values("timestamp")
    
    # Feature engineering
    df["day_of_week"] = df["timestamp"].dt.dayofweek
    df["is_weekend"] = df["day_of_week"].isin([5, 6]).astype(int)
    df["rolling_mean_7"] = df["units_sold"].rolling(7).mean()
    
    # Prepare features and target
    X = df[["day_of_week", "is_weekend", "rolling_mean_7"]]
    y = df["units_sold"].shift(-horizon)  # forecast 7 days ahead
    
    # Train model
    model = RandomForestRegressor(n_estimators=100, random_state=42)
    model.fit(X, y)
    
    # Save model
    dump(model, output_path)
    return {"horizon": horizon, "last_timestamp": df["timestamp"].max()}

The model is versioned using a simple file-based system:

models/
  sales-forecast/
    v1/
      model.joblib
      metadata.json
    v2/
      model.joblib
      metadata.json

Each metadata.json file contains:

{
  "model_version": "v2",
  "training_start": "2024-04-15T00:00:00Z",
  "training_end": "2024-04-15T02:00:00Z",
  "horizon_days": 7,
  "feature_columns": ["day_of_week", "is_weekend", "rolling_mean_7"],
  "training_data_range": {
    "start": "2023-01-01T00:00:00Z",
    "end": "2024-04-14T23:59:59Z"
  }
}

Forecast Execution and Scheduling

Forecasts are generated by a scheduler (e.g., cron, Airflow, or a dedicated service). A typical forecast job runs daily at 04:00 UTC and uses the latest model.

# Run forecast job
docker run \
  --env MODEL_PATH="models/sales-forecast/v2" \
  --env FORECAST_START="2024-04-16T00:00:00Z" \
  --env FORECAST_HORIZON=7 \
  forecast-executor:latest

The executor:

CREATE TABLE forecasts (
    forecast_id UUID PRIMARY KEY,
    model_version TEXT,
    target_column TEXT,
    forecast_start TIMESTAMPTZ,
    forecast_end TIMESTAMPTZ,
    values JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

A forecast entry looks like:

{
  "forecast_id": "a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8",
  "model_version": "v2",
  "target_column": "units_sold",
  "forecast_start": "2024-04-16T00:00:00Z",
  "forecast_end": "2024-04-23T00:00:00Z",
  "values": [
    {"timestamp": "2024-04-16T00:00:00Z", "predicted": 1234.5},
    {"timestamp": "2024-04-17T00:00:00Z", "predicted": 1301.2},
    ...
  ]
}

A silent failure: if the forecast_start is not aligned with the model’s expected training data, the forecast may be incorrect. For example, a model trained on data up to 2024-04-14 but forecasted from 2024-04-16 will have a lag of two days in the first predictions.

Model Validation and Monitoring

After execution, forecasts must be validated and monitored for performance and correctness.

Validation is done using a test set of known future values. The model is evaluated using a metric such as Mean Absolute Error (MAE), Mean Squared Error (MSE), or Mean Absolute Percentage Error (MAPE).

# validate_forecast.py
def evaluate_forecast(actual_path: str, predicted_path: str, metric: str = "mae"):
    actual = pd.read_parquet(actual_path)
    predicted = pd.read_parquet(predicted_path)
    
    # Align predictions with actuals
    actual = actual.set_index("timestamp")
    predicted = predicted.set_index("timestamp")
    
    # Compute error
    error = actual["units_sold"] - predicted["predicted"]
    if metric == "mae":
        return error.abs().mean()
    elif metric == "mse":
        return (error ** 2).mean()
    elif metric == "mape":
        return (error / actual["units_sold"]).abs().mean() * 100

The evaluation is run after each forecast job, and results are stored in a model_evaluation table:

CREATE TABLE model_evaluation (
    evaluation_id UUID PRIMARY KEY,
    model_version TEXT,
    forecast_id UUID,
    metric TEXT,
    value FLOAT,
    evaluated_at TIMESTAMPTZ DEFAULT NOW()
);

Monitoring is done via a dashboard that tracks:

A common failure: the forecast model is trained on data up to 2024-04-14, but the actual data for 2024-04-15 arrives only on 2024-04-17. The model is evaluated on the 2024-04-15 data, but the forecast was already made on 2024-04-16—so the evaluation is delayed by two days, and the forecast is validated on data that was not available at prediction time.

Forecast Consumption and Integration

Forecasts are consumed by downstream systems via APIs or event streams.

A typical forecast API endpoint:

GET /api/v1/forecasts/sales?product_id=123&start=2024-04-16T00:00:00Z&end=2024-04-23T00:00:00Z

Response:

{
  "product_id": 123,
  "forecast_start": "2024-04-16T00:00:00Z",
  "forecast_end": "2024-04-23T00:00:00Z",
  "model_version": "v2",
  "values": [
    {
      "timestamp": "2024-04-16T00:00:00Z",
      "predicted": 1234.5,
      "lower_bound": 1150.0,
      "upper_bound": 1320.0
    },
    ...
  ],
  "metadata": {
    "horizon_days": 7,
    "training_data_range": {
      "start": "2023-01-01T00:00:00Z",
      "end": "2024-04-14T23:59:59Z"
    }
  }
}

Consumers include:

A critical issue: forecast staleness. The forecast is generated daily, but a downstream system assumes it is fresh every hour. If the forecast is updated only once a day, the system may use an outdated forecast for 23 hours. This leads to incorrect decisions: a warehouse may order inventory based on a forecast that is 18 hours old.

Handling Forecast Failures

Forecasting systems fail silently. The most common failures include:

A key diagnostic is forecast accuracy over time. A sharp increase in MAE indicates a problem—perhaps the model is not capturing a new seasonality, or the data pipeline is introducing delays.

To detect failures, systems use alerting based on:

A failure mode: forecast versioning confusion. A new model is trained and deployed, but the model_version field in the forecast is not updated correctly. The system uses v1 for a model that is actually v2, leading to inconsistent predictions.

Model Retraining and Rollout

Retraining is triggered either on a schedule or on demand. A new model is trained, validated, and deployed in a canary rollout:

  1. Train a new model on the latest data.
  2. Validate it against a test set.
  3. Deploy it to a staging environment.
  4. Run it in parallel with the current model for a week.
  5. Compare forecasts from both models.
  6. Promote the new model to production if accuracy improves by at least 5%.

The rollout is managed by a model registry that tracks:

A critical detail: model artifact consistency. The model file must include the full set of preprocessing steps, including feature engineering and scaling. If the model is trained with StandardScaler but inference does not apply the same scaler, the forecast will be biased.

Summary of Key Patterns

Time series forecasting in production is not just about making predictions. It is about ensuring that predictions are correct, timely, and trusted—so that decisions based on them are reliable, even when the system is running on autopilot.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.