Time Series Forecasting

Build reliable time series forecasting models for business metrics. Covers statistical methods (ARIMA, ETS), machine learning approaches, Prophet, neural forecasting, feature engineering for temporal data, and the evaluation frameworks that separate useful forecasts from noise.

Time series forecasting predicts future values based on historical sequences of observations recorded at regular intervals. It matters when decisions depend on knowing what will happen next—inventory replenishment, server capacity planning, sales projections, or demand for cloud compute. The core challenge is distinguishing signal from noise, seasonality from trend, and modeling both short-term dynamics and long-term patterns across time.

Forecasting with Standard Tools

Forecasting begins with data in a tabular format, typically stored in a database or data warehouse. The input is a table with a timestamp column and one or more numeric measurement columns. Each row represents a time bucket: minute, hour, day, or month. The output is a forecast table with the same schema, extended into the future.

Preparing and Validating Time Series Data

Data must be structured, clean, and aligned. A common failure is missing time buckets or irregular timestamps. Use SQL to validate and prepare data:

-- Ensure every day has a row in the sales table
SELECT
  DATE_TRUNC('day', ts) AS day,
  COUNT(*) AS observations,
  MIN(ts) AS first_observation,
  MAX(ts) AS last_observation
FROM sales
GROUP BY DATE_TRUNC('day', ts)
HAVING COUNT(*) < 1
ORDER BY day;

If the query returns rows, the data is incomplete. A missing day is a silent failure: the model learns from uneven gaps but assumes regular sampling. To fill gaps:

-- Create a complete calendar of days and left-join to sales
WITH calendar AS (
  SELECT generate_series(
    '2023-01-01'::date,
    '2024-12-31'::date,
    '1 day'::interval
  ) AS day
)
SELECT
  c.day,
  COALESCE(s.revenue, 0) AS revenue
FROM calendar c
LEFT JOIN sales s ON c.day = s.ts::date
ORDER BY c.day;

This pattern ensures the input to forecasting pipelines is uniformly sampled. A forecast trained on irregular data will mispredict during holidays or weekends when data arrives late.

Training Models with Python

Use scikit-learn and statsmodels for standard forecasting. The pipeline is:

  1. Load data
  2. Extract features (lag, rolling mean, seasonality)
  3. Fit a model
  4. Validate on held-out data
import pandas as pd
from statsmodels.tsa.seasonal import seasonal_decompose
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures


df = pd.read_csv("sales.csv", parse_dates=["ts"])
df.set_index("ts", inplace=True)

# Decompose time series
decomp = seasonal_decompose(df["revenue"], model="additive", period=7)

# Add lag features
df["revenue_lag_1"] = df["revenue"].shift(1)
df["revenue_lag_7"] = df["revenue"].shift(7)
df["revenue_rolling_mean_7"] = df["revenue"].rolling(7).mean()

# Prepare features and target
X = df[["revenue_lag_1", "revenue_lag_7", "revenue_rolling_mean_7"]]
y = df["revenue"].dropna()
X = X.loc[y.index]

# Fit model
model = LinearRegression()
model.fit(X, y)

# Predict future
future_lags = pd.DataFrame({
    "revenue_lag_1": [df["revenue"].iloc[-1]],
    "revenue_lag_7": [df["revenue"].iloc[-7]],
    "revenue_rolling_mean_7": [df["revenue"].iloc[-7:].mean()]
})
forecast = model.predict(future_lags)

Common failure: forgetting to align the target and features. If y is shifted but X is not, the model learns to predict one lag behind. Another subtle issue: shift() in pandas creates NaN values, and training data must be trimmed consistently. A model trained on data with NaN targets but not NaN features will underperform.

Evaluating Forecasts

Use rolling window validation. Split the data into training and test windows that slide forward in time.

def evaluate_forecast(df, window_size=365, horizon=30):
    results = []
    for start in range(0, len(df) - window_size - horizon, window_size):
        train = df.iloc[start:start + window_size]
        test = df.iloc[start + window_size:start + window_size + horizon]
        model = LinearRegression()
        X_train = train[["revenue_lag_1", "revenue_lag_7"]]
        y_train = train["revenue"].iloc[1:]
        X_train = X_train.iloc[1:]  # align with y_train
        model.fit(X_train, y_train)

        X_test = test[["revenue_lag_1", "revenue_lag_7"]]
        X_test = X_test.iloc[1:]  # use lag features from test
        y_test = test["revenue"].iloc[1:]

        y_pred = model.predict(X_test)
        mae = (y_test - y_pred).abs().mean()
        results.append(mae)
    return pd.Series(results).mean()

The key insight: the evaluation metric is not just accuracy, but robustness to changes in seasonality and trend. A model that performs well on the first half of data but poorly on the last half fails in production.

Deploying Forecasts with Batch and Real-Time Pipelines

Schedule forecasts using cron or systemctl. A typical batch job runs daily:

# /etc/systemd/system/forecast.service
[Unit]
Description=Daily Time Series Forecast
After=network.target

[Service]
User=forecast
WorkingDirectory=/opt/forecast
ExecStart=/usr/bin/python3 /opt/forecast/forecast.py
Restart=always

[Install]
WantedBy=multi-user.target

The forecast.py script reads from a database, fits a model, and writes predictions back:

# forecast.py
import sqlite3
import pandas as pd
from datetime import datetime, timedelta

conn = sqlite3.connect("/data/forecast.db")

# Read latest data
df = pd.read_sql("SELECT * FROM sales ORDER BY ts", conn, parse_dates=["ts"])

# Train and predict
# ... (same model as above)

# Write forecast to table
forecast_df = pd.DataFrame({
    "ts": pd.date_range(
        start=df["ts"].max() + timedelta(days=1),
        periods=30,
        freq="D"
    ),
    "predicted_revenue": forecast
})

forecast_df.to_sql("forecast", conn, if_exists="replace", index=False)
conn.commit()
conn.close()

A common failure: the forecast table has a ts column that is a datetime but the model expects a date. The to_sql call creates a column with datetime values, but the ts column in the database is date. This mismatch causes the next day’s forecast to be off by a day.

For real-time forecasting, use kafka to stream new observations into a forecasting service.

# Kafka topic: sales-events
kafka-console-consumer.sh \
  --bootstrap-server kafka:9092 \
  --topic sales-events \
  --partition 0 \
  --offset latest

A consumer reads new rows and triggers a forecast update:

# real_time_forecast.py
import json
import requests
from kafka import KafkaConsumer

consumer = KafkaConsumer(
    "sales-events",
    bootstrap_servers=["kafka:9092"],
    value_deserializer=lambda m: json.loads(m.decode("utf-8"))
)

for message in consumer:
    data = message.value
    # Update model incrementally
    model.partial_fit(
        [[data["revenue_lag_1"], data["revenue_lag_7"], data["revenue_rolling_mean_7"]]],
        [data["revenue"]]
    )
    # Send prediction to API
    response = requests.post(
        "http://forecast-api:8000/predict",
        json={
            "timestamp": data["ts"],
            "revenue": data["revenue"],
            "forecast": model.predict([[data["revenue_lag_1"], data["revenue_lag_7"], data["revenue_rolling_mean_7"]]])
        }
    )
    if response.status_code != 200:
        print(f"Failed to update forecast: {response.text}")

The silent failure: the partial_fit call updates the model but does not recompute the rolling mean. The lag features are static, but the rolling mean depends on the entire history. The model becomes outdated unless the rolling mean is recomputed on every new observation.

Monitoring Forecast Health

A forecast is only as good as its monitoring. A dashboard must show:

Use psql to extract metrics:

-- Weekly forecast error
SELECT
  DATE_TRUNC('week', ts) AS week,
  AVG(ABS(actual - forecast)) AS mae,
  COUNT(*) AS observations
FROM forecast
WHERE ts >= NOW() - INTERVAL '6 months'
GROUP BY DATE_TRUNC('week', ts)
ORDER BY week;

A common pattern: a forecast looks good, but the residual plot shows increasing variance over time. This indicates heteroscedasticity, which means the model needs to predict variance as well as mean.

Seasonality is often modeled as a feature, but the model must account for multiple seasonalities (e.g., daily and weekly). Use statsmodels’ STL decomposition:

from statsmodels.tsa.seasonal import STL

stl = STL(df["revenue"], period=7, robust=True)
result = stl.fit()

# Extract trend, seasonal, and residual components
trend = result.trend
seasonal = result.seasonal
residual = result.resid

# Forecast trend and seasonal separately
trend_forecast = trend.iloc[-1:].reindex(
    pd.date_range(
        start=df.index[-1] + pd.Timedelta(days=1),
        periods=30,
        freq="D"
    )
).interpolate(method="linear")
seasonal_forecast = seasonal.iloc[-7:].reindex(
    pd.date_range(
        start=df.index[-1] + pd.Timedelta(days=1),
        periods=30,
        freq="D"
    )
).reindex(
    pd.date_range(
        start=df.index[-1] + pd.Timedelta(days=1),
        periods=30,
        freq="D"
    )
).interpolate(method="linear")

The error: the seasonal component is indexed by day_of_week, but the forecast uses day_of_year. A mismatch of 7 days shifts the seasonal pattern by one week. The solution: align the seasonal component to the full time index.

Managing Model Versioning and Rollback

Each forecast model must be versioned. The model file and its configuration must be stored with metadata.

{
  "model_version": "2024.04.15",
  "training_data_start": "2022-01-01",
  "training_data_end": "2024-04-14",
  "features": [
    "revenue_lag_1",
    "revenue_lag_7",
    "revenue_rolling_mean_7"
  ],
  "model_type": "LinearRegression",
  "hyperparameters": {
    "fit_intercept": true,
    "normalize": false
  },
  "metrics": {
    "mae": 1247.3,
    "rmse": 1632.1,
    "mape": 0.078
  },
  "last_updated": "2024-04-15T08:32:11Z"
}

Store the model as a .pkl file and the metadata in model.json. When a new version is deployed, the system compares the new model’s predictions with the old one. If the new model has higher error, it triggers an alert.

Common Failure Modes

The model assumes NaN values are handled during training but fails during inference when NaN features are not filled.

The rolling_mean is computed on training data but not on new data. The model sees a different distribution of features at inference time.

Data from different regions arrives with different timezones. The model assumes UTC but receives data in America/New_York. The forecast drifts by hours.

A forecast table has rows but no ts column. The consumer assumes the first row is the first day, but the actual timestamps are off.

A model predicts 30 days ahead, but the output table has only 29 rows. The first day is missing.

The model uses a fixed window but the data grows. The model does not incorporate new observations into the rolling mean.

Each of these is a silent failure that can go unnoticed for weeks. The key is to build a monitoring system that validates the forecast pipeline end-to-end: from data ingestion to model inference to output validation.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.