Production-grade guide to mlops production patterns covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.
MLOps Production Patterns is the collection of repeatable, reliable workflows and configurations that enable data science teams to deliver, monitor, and evolve machine learning models in production environments. It applies when models are expected to serve predictions continuously, update autonomously, and respond to data and system changes with minimal human intervention. This page details the core patterns that define successful MLOps, with emphasis on what must be done, how it must be done, and where it commonly fails.
Deploying a model is not just uploading a file. A model in production must be versioned, tested, and served with consistent inference behavior across environments.
Each model artifact must be built as a container image with a standard interface. A model is deployed using docker build with a Dockerfile that specifies a CMD entrypoint:
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY model.pkl .
COPY inference.py .
CMD ["gunicorn", "-w", "4", "-b", "0.0.0.0:8000", "inference:app"]The inference.py file defines a WSGI application serving a single endpoint at /predict:
import pickle
import numpy as np
from flask import Flask, request, jsonify
app = Flask(__name__)
with open("model.pkl", "rb") as f:
model = pickle.load(f)
@app.route("/predict", methods=["POST"])
def predict():
data = request.get_json()
X = np.array(data["features"])
prediction = model.predict(X).tolist()
return jsonify({"prediction": prediction})Model versions are tagged using semantic versioning in the container image: model-service:v1.2.3. Versioning is tracked in a model registry that stores metadata such as training dataset ID, hyperparameters, and evaluation metrics. The registry is updated with a curl command after training:
curl -X POST https://registry.example.com/models \
-H "Content-Type: application/json" \
-d '{
"name": "churn-predictor",
"version": "1.2.3",
"artifact_uri": "gcr.io/project/model-service:v1.2.3",
"training_dataset": "churn-v2023-12-01",
"metrics": {
"auc": 0.872,
"precision": 0.78
},
"metadata": {
"hyperparameters": {
"n_estimators": 100,
"max_depth": 10
}
}
}'A production deployment uses a Deployment object in Kubernetes:
apiVersion: apps/v1
kind: Deployment
metadata:
name: model-service
labels:
app: model-service
spec:
replicas: 4
selector:
matchLabels:
app: model-service
template:
metadata:
labels:
app: model-service
spec:
containers:
- name: model
image: gcr.io/project/model-service:v1.2.3
ports:
- containerPort: 8000
env:
- name: MODEL_NAME
value: "churn-predictor"
- name: MODEL_VERSION
value: "1.2.3"
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"Failure mode: the model container starts but returns 500 Internal Server Error on /predict. The logs show:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 589, in spawn_workers
worker.init_process()
File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/ggevent.py", line 208, in init_process
self.run()
File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/ggevent.py", line 126, in run
self.wsgi()
File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/base.py", line 123, in wsgi
self.respond()
File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/ggevent.py", line 188, in respond
self.handle_request()
File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/ggevent.py", line 227, in handle_request
self.application(self.environ, self.start_response)
File "/app/inference.py", line 25, in <lambda>
app = Flask(__name__)
File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1412, in __init__
self.config = self.make_config()
File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1462, in make_config
self.config.from_object(config)
File "/usr/local/lib/python3.11/site-packages/flask/config.py", line 252, in from_object
obj = import_string(obj)
File "/usr/local/lib/python3.11/site-packages/flask/utils.py", line 270, in import_string
raise ImportError(f"Could not import {import_name}")
ImportError: Could not import modelThis occurs when model.pkl is missing in the container, or when the model module is not in the Python path. The root cause is a mismatch between the build and runtime environment.
Models must adapt to new data. A continuous training pipeline runs daily, triggered by a new batch of data in a data lake.
The pipeline is defined as a Makefile with a train target:
train: model.pkl evaluate
model.pkl: data/processed/*.parquet feature_engineering.py
python feature_engineering.py
python train.py --input data/processed --output model.pkl
evaluate: model.pkl
python evaluate.py --model model.pkl --test data/test.parquet --output metrics.json
deploy: model.pkl
curl -X POST https://registry.example.com/models \
-H "Content-Type: application/json" \
-d '{
"name": "churn-predictor",
"version": "1.2.4",
"artifact_uri": "gcr.io/project/model-service:v1.2.4",
"training_dataset": "churn-v2024-01-15",
"metrics": { "auc": 0.876 }
}'
kubectl apply -f k8s/deployment.yamlThis Makefile is executed by a cron job on a CI system:
0 2 * * * cd /opt/pipelines && make trainEach run produces a new model.pkl and updates the model registry. The pipeline fails silently when train.py writes to model.pkl but feature_engineering.py does not produce data/processed/*.parquet. The Makefile depends on data/processed/*.parquet, but the data directory is not mounted correctly in the CI runner.
To detect such issues, a validate step is added:
validate:
ls -la data/processed/
[ -f data/processed/train.parquet ] || exit 1
[ -f data/processed/test.parquet ] || exit 1This step ensures that the preprocessing step completed successfully. Without it, a model trained on incomplete data is deployed with no indication of the problem.
A deployed model must be monitored for data and prediction drift. Monitoring is not just logging; it is continuous validation of model behavior.
Each model service logs predictions to a structured format:
{
"timestamp": "2024-01-15T08:30:00Z",
"model_version": "1.2.3",
"input": {
"features": [0.8, 0.2, 1, 0, 1]
},
"output": {
"prediction": 0.72,
"probability": [0.28, 0.72]
},
"request_id": "abc123"
}These logs are ingested into a time-series database such as InfluxDB or a columnar store like Apache Parquet in a data lake.
A drift detection job runs daily, comparing the current data distribution to the training data distribution:
import pandas as pd
from scipy.stats import wasserstein_distance
import json
def detect_drift(current_data, training_data, threshold=0.1):
drifts = {}
for col in current_data.columns:
dist = wasserstein_distance(current_data[col], training_data[col])
drifts[col] = dist
if dist > threshold:
print(f"Drift detected in {col}: {dist:.3f} (threshold: {threshold})")
return drifts
current = pd.read_parquet("data/processed/current.parquet")
training = pd.read_parquet("data/processed/training.parquet")
# Detect drift
drifts = detect_drift(current, training)
# Write results
with open("drift_report.json", "w") as f:
json.dump(drifts, f, indent=2)The report is stored and used to trigger alerts or retraining. A drift_report.json with a key monthly_churn having wasserstein_distance=0.23 triggers an alert because the threshold is 0.1.
A common failure: the model uses a feature total_minutes that is computed in a preprocessing step, but the current.parquet file includes only call_duration and streaming_duration. The drift detection compares call_duration to training.call_duration, but total_minutes is missing. The model has been trained on a derived feature that is not present in the current data.
To prevent this, a schema validation step is added before drift detection:
def validate_schema(data, expected_columns):
missing = set(expected_columns) - set(data.columns)
if missing:
raise ValueError(f"Missing columns: {sorted(missing)}")
return TrueThis step fails when total_minutes is not in current.parquet, and the pipeline stops before drift detection. The error is logged and surfaced in a monitoring dashboard.
Models do not stay in production forever. They are promoted, retired, and replaced. The lifecycle is managed through a state machine.
Each model has a state field in the registry:
{
"name": "churn-predictor",
"version": "1.2.3",
"state": "production",
"last_deployed": "2024-01-10T00:00:00Z",
"active": true
}The state transitions are managed by a promote command:
curl -X PUT https://registry.example.com/models/churn-predictor/versions/1.2.3 \
-H "Content-Type: application/json" \
-d '{
"state": "staging",
"note": "Deployed for A/B test with 10% traffic"
}'After validation, the model is promoted to production:
curl -X PUT https://registry.example.com/models/churn-predictor/versions/1.2.3 \
-H "Content-Type: application/json" \
-d '{
"state": "production",
"active": true,
"note": "Promoted after 48h monitoring"
}'A rollback command reverts to a previous version:
curl -X POST https://registry.example.com/models/churn-predictor/rollback \
-H "Content-Type: application/json" \
-d '{
"from_version": "1.2.3",
"to_version": "1.2.2",
"reason": "High error rate on new feature"
}'The service is updated to serve 1.2.2 while 1.2.3 is kept as a backup. A failure mode: a model is promoted to production but the active field is not set. The service continues serving the old version. The fix is to add a wait_for_active step in the deployment pipeline:
- name: wait-for-active
image: curlimages/curl
command: ["sh", "-c"]
args:
- |
until curl -f http://model-service:8000/health; do
echo "Waiting for model service to become active..."
sleep 5
doneThis step ensures that the new model is live and healthy before marking it as active.
Model behavior depends on configuration. Configuration must be versioned, applied consistently, and validated.
Configuration is stored in config.yml:
model:
name: churn-predictor
version: 1.2.3
threshold: 0.5
features:
- call_duration
- streaming_duration
- total_minutes
- avg_monthly_usage
hyperparameters:
n_estimators: 100
max_depth: 10
learning_rate: 0.1
monitoring:
drift_threshold: 0.1
log_predictions: trueThe model service reads this configuration at startup:
import yaml
import os
def load_config():
with open("config.yml", "r") as f:
return yaml.safe_load(f)
config = load_config()
threshold = config["model"]["threshold"]
features = config["model"]["features"]The configuration is stored in the container image and can be overridden at runtime using environment variables:
env:
- name: MODEL_THRESHOLD
valueFrom:
configMapKeyRef:
name: model-config
key: thresholdA common mistake: the config.yml file is updated but not rebuilt into the container image. The service runs with old configuration. To catch this, a validate-config step is added:
def validate_config(config_path, expected_keys):
with open(config_path, "r") as f:
config = yaml.safe_load(f)
for key in expected_keys:
if key not in config:
raise KeyError(f"Missing key: {key}")
return TrueThis step runs during container build and deployment, ensuring that the configuration is correct and up to date.
MLOps Production Patterns are the foundational workflows that ensure models are not just trained but deployed, monitored, and maintained in production. They depend on clear versioning, consistent deployment, continuous training, effective monitoring, and robust configuration. The patterns are not tied to any specific platform but are universal across systems that must deliver machine learning at scale.
This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy