Data Drift Monitoring

Data drift monitoring detects changes in the statistical properties of input data between training and inference, ensuring model performance remains stable over time. It is essential when data arrives continuously—batch or streaming—and when model predictions degrade silently due to shifts in feature distributions, label distributions, or feature-label relationships. It matters most in production systems where models serve real-time predictions, and where the cost of a failure is high: a recommendation engine serving ads, a fraud detection pipeline, or a clinical risk score.

Detecting Drift in Feature Distributions

To detect drift in feature distributions, compute statistical summaries of a reference dataset (typically training data) and compare them against a current dataset (inference or batch input). The reference is defined as a snapshot of data collected during model training, stored as a dataset in a data lake or warehouse.

A typical monitoring job computes the Kolmogorov-Smirnov (KS) statistic and Jensen-Shannon (JS) divergence between the reference and current feature distributions. The job runs daily or hourly, depending on data ingestion frequency.

python -m drift.monitor \
  --reference-data /data/reference.parquet \
  --current-data /data/ingested/2024-04-05.parquet \
  --output /results/drift-report-2024-04-05.json \
  --features "age", "income", "city_code", "user_type" \
  --thresholds "ks=0.1", "js=0.05"

The output is a JSON file with per-feature drift scores and global metrics:

{
  "global": {
    "ks": 0.12,
    "js": 0.08,
    "drift_detected": true,
    "timestamp": "2024-04-05T12:00:00Z"
  },
  "features": {
    "age": {
      "ks": 0.18,
      "js": 0.11,
      "drift_detected": true,
      "p_value": 0.003,
      "reference_mean": 34.2,
      "current_mean": 36.1,
      "reference_std": 12.3,
      "current_std": 14.7
    },
    "income": {
      "ks": 0.07,
      "js": 0.04,
      "drift_detected": false,
      "p_value": 0.15
    }
  }
}

A drift is flagged when either the global KS or JS exceeds a threshold, and each feature can be evaluated independently. The ks and js fields are computed using scipy.stats.ks_2samp and scipy.spatial.distance.jensenshannon, respectively.

Tracking Drift in Label Distributions

When labels are available at inference time—e.g., in supervised learning with feedback loops—monitoring label distribution drift is critical. A sudden shift in the label space can signal a change in user behavior, policy, or data collection.

The label distribution is computed from the current dataset and compared to the reference using the same statistical tests as for features.

# Extract labels and compute drift
python -m drift.label_drift \
  --reference-labels /data/labels-train.parquet \
  --current-labels /data/labels-inference-2024-04-05.parquet \
  --output /results/label-drift-2024-04-05.json \
  --method "chi2" \
  --threshold 0.05

The chi2 method computes the chi-squared test for categorical labels, while ks can be used for continuous labels. The output includes:

• chi2_statistic: observed chi-squared value
• p_value: significance of the test
• expected_frequencies: reference distribution
• observed_frequencies: current distribution
• drift_detected: boolean flag

A silent failure occurs when the label encoding differs between reference and current data. If the reference uses ["low", "medium", "high"] but the current data uses ["low", "med", "high"], the chi-squared test fails to detect drift even when the label distribution is significantly different. The solution is to normalize labels before computing drift.

Monitoring Feature-Label Dependencies

Drift in features or labels alone is insufficient when model performance degrades due to a shift in the relationship between input features and labels. This is known as *concept drift*, and it can be detected by computing the model’s prediction accuracy on the current data and comparing it to the expected accuracy on the reference.

The process involves:

1. Loading the trained model (e.g., as a .pkl or ONNX file).
2. Applying the model to the current dataset.
3. Computing accuracy (or AUC, F1, etc.) on the current labels.
4. Comparing it to the accuracy computed on the reference data.

# Compute model performance drift
python -m drift.concept_drift \
  --model /models/risk_model_v3.onnx \
  --reference-data /data/reference.parquet \
  --current-data /data/ingested/2024-04-05.parquet \
  --labels "risk_score" \
  --metrics "accuracy", "auc" \
  --output /results/concept-drift-2024-04-05.json

Output includes:

{
  "metrics": {
    "accuracy": {
      "reference": 0.83,
      "current": 0.78,
      "difference": -0.05,
      "significance": true,
      "p_value": 0.012
    },
    "auc": {
      "reference": 0.89,
      "current": 0.85,
      "difference": -0.04,
      "significance": true
    }
  },
  "drift_detected": true,
  "timestamp": "2024-04-05T12:00:00Z"
}

A key edge case: when the model is retrained on the current data, the reference accuracy is computed on the current data using the *old* model. This captures the concept drift that the old model cannot handle. If the model is retrained and the new model is evaluated on the same data, the accuracy will be higher, but it will not reflect the drift in the underlying concept.

Scheduling and Observing Drift Reports

Drift monitoring is not a one-off task. It must be scheduled and observed at regular intervals. A typical setup uses a cron job or a workflow orchestrator like Airflow.

# crontab entry to run drift monitoring
0 0 * * * /usr/local/bin/drift-monitor.sh --mode daily

The script orchestrates the full pipeline:

#!/bin/bash
# /usr/local/bin/drift-monitor.sh

set -e

export DATA_ROOT="/data"
export RESULTS_ROOT="/results"
export MODEL_ROOT="/models"

DATE=$(date +%Y-%m-%d)

# 1. Compute feature drift
python -m drift.monitor \
  --reference-data "${DATA_ROOT}/reference.parquet" \
  --current-data "${DATA_ROOT}/ingested/${DATE}.parquet" \
  --output "${RESULTS_ROOT}/drift-${DATE}.json" \
  --features "age", "income", "city_code", "user_type" \
  --thresholds "ks=0.1", "js=0.05"

# 2. Compute label drift
python -m drift.label_drift \
  --reference-labels "${DATA_ROOT}/labels-train.parquet" \
  --current-labels "${DATA_ROOT}/labels-${DATE}.parquet" \
  --output "${RESULTS_ROOT}/label-drift-${DATE}.json" \
  --method "chi2" \
  --threshold 0.05

# 3. Compute concept drift
python -m drift.concept_drift \
  --model "${MODEL_ROOT}/risk_model_v3.onnx" \
  --reference-data "${DATA_ROOT}/reference.parquet" \
  --current-data "${DATA_ROOT}/ingested/${DATE}.parquet" \
  --labels "risk_score" \
  --metrics "accuracy", "auc" \
  --output "${RESULTS_ROOT}/concept-drift-${DATE}.json"

# 4. Upload results to a central storage (e.g., S3)
aws s3 cp "${RESULTS_ROOT}/" s3://drift-reports/${DATE}/ --recursive

The orchestrator can trigger alerts when drift is detected. A simple alerting rule checks the drift_detected field in the output JSON.

def alert_on_drift(report_path: str) -> bool:
    with open(report_path) as f:
        report = json.load(f)
    return report["global"]["drift_detected"] or any(
        feat["drift_detected"] for feat in report["features"].values()
    )

The alerting system can send notifications via email, Slack, or push to a monitoring dashboard.

Failure Modes and Silent Pitfalls

Several common issues silently degrade drift monitoring:

• Mismatched time windows: The reference dataset is from a fixed training period, but the current dataset spans a different time window. If the current data is collected over a week but the reference is from a month, the comparison is biased. The solution is to align time windows by filtering or binning.

• Incomplete feature sets: The current dataset may lack some features present in the reference. The monitoring job fails silently if it assumes all features are present. The code must validate that all required features are present and handle missing ones by either imputing or flagging.

• Missing or misaligned labels: Labels may be missing for some rows, or the label column may have different names across datasets. The drift script must check for the presence of required columns and log warnings when they are missing.

• Drift threshold confusion: The ks=0.1 threshold is often misinterpreted. A KS statistic of 0.1 means the empirical CDFs differ by 0.1 on average, but it does not indicate the probability of the distributions being different. A common mistake is setting ks=0.1 without checking the p-value. The p-value from ks_2samp is critical: a KS of 0.1 with p-value 0.15 is not significantly different. The threshold should be combined with a p-value check.

• Thresholds not applied consistently: Some jobs compute drift per feature, but the global drift decision is based only on the average of feature scores, not the maximum or the number of drifted features. A system that uses the maximum feature KS may detect drift even if only one feature is drifting, while a system using the average may miss a critical shift.

• Configuration drift: The monitoring job may use different parameters across runs. The --features list, --thresholds, and --method flags are often hard-coded but not versioned. A change in the features list may go unnoticed. The solution is to store the configuration in a JSON file and version it with Git.

{
  "monitoring_config": {
    "reference_dataset": "/data/reference.parquet",
    "current_dataset": "/data/ingested/{date}.parquet",
    "features": [
      "age",
      "income",
      "city_code",
      "user_type"
    ],
    "thresholds": {
      "ks": 0.1,
      "js": 0.05,
      "p_value": 0.05
    },
    "method": "chi2",
    "output_format": "json"
  }
}

This configuration is loaded at runtime, and any mismatch between expected and actual features triggers a warning.

Visualizing and Debugging Drift

Visualizations help diagnose the cause of drift. A typical drift report includes:

• CDF plots of key features: overlaid reference and current empirical CDFs.
• Histograms with normalized binning for feature and label distributions.
• Heatmaps of feature-label joint distributions.

These visualizations are generated using matplotlib or plotly.

import matplotlib.pyplot as plt

def plot_cdf(feature: str, ref_data, curr_data, output_path: str):
    fig, ax = plt.subplots()
    ax.plot(ref_data[feature], label="Reference")
    ax.plot(curr_data[feature], label="Current")
    ax.set_title(f"CDF: {feature}")
    ax.legend()
    fig.savefig(output_path)

The output is stored as PNG or HTML and linked in the drift report.

A common mistake is to use the same binning for histograms across datasets. When the current data has a wider range, the histogram bins are misaligned. The solution is to compute a global binning strategy—e.g., using numpy.histogram_bin_edges on the reference data—and apply the same bins to the current data.

Summary of Key Actions

• Use scipy.stats.ks_2samp for feature and label drift.
• Compute drift per feature, then aggregate to global drift.
• Validate feature and label presence before computing drift.
• Use p_value alongside threshold values to decide drift significance.
• Align time windows when comparing reference and current data.
• Version monitoring configurations and store them in Git.
• Generate visualizations and store them with the report.
• Schedule daily or hourly monitoring with cron or Airflow.
• Trigger alerts based on drift_detected and p_value thresholds.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.