Deploy ML models to production. Covers serving architectures, model versioning, A/B testing models, canary deployments, batch vs real-time inference, and model rollback strategies.
This page describes practical patterns for deploying machine learning models in production systems. These patterns address how to manage model versions, handle updates, ensure consistency between training and inference environments, and deal with failure scenarios during deployment. They apply when moving trained models from development to operational use, particularly in systems where model correctness and availability are critical.
Deployed models must be versioned to support rollback when new versions introduce errors or regressions. The simplest approach uses a model registry that stores each version with metadata including commit hash, training timestamp, and performance metrics.
A common pattern is to tag deployments with semantic version numbers (v1.0.0, v1.1.2) in the model metadata stored in a database or configuration store. This allows operators to select which version to serve traffic to.
When rolling back, the deployment system switches from serving one model version to another. In Kubernetes, this typically involves updating a Deployment's image reference and rolling out the change:
apiVersion: apps/v1
kind: Deployment
metadata:
name: model-deployment
spec:
replicas: 3
selector:
matchLabels:
app: model-server
template:
metadata:
labels:
app: model-server
spec:
containers:
- name: model-server
image: my-model-server:v1.1.2If an update fails, the system can roll back to the previous version by changing the image field to the known good version. The deployment controller ensures that new pods use the correct image while old ones are terminated.
The key failure mode here is updating to a version that silently introduces bugs not caught in testing. If no monitoring or rollback mechanism exists, a bad model version can serve traffic for hours or days before being detected.
Blue-green deployment involves running two identical production environments: one active (blue) and one inactive (green). When deploying a new model version, the system first deploys the new version to the green environment, tests it, then switches traffic to it.
In practice, this requires managing two separate deployments or services with different names. Traffic routing is handled by an external load balancer or service mesh that can switch between the two versions.
Example configuration for a service using blue-green strategy:
# Service definition for blue version
apiVersion: v1
kind: Service
metadata:
name: model-service-blue
spec:
selector:
app: model-server
version: blue
---
# Service definition for green version
apiVersion: v1
kind: Service
metadata:
name: model-service-green
spec:
selector:
app: model-server
version: greenThe routing logic updates the service's selector or uses an ingress controller to direct traffic to either blue or green.
A sharp edge in this pattern is that both versions must be able to handle the same input schema. If a new model expects different features, the system might silently fail requests that reach the wrong version.
Canary deployments gradually shift traffic to a new model version by routing a small percentage of requests to it. This allows observing performance and behavior in production before fully switching.
The deployment process starts with deploying the new model alongside the existing one, then incrementally increasing the traffic percentage using service mesh or load balancer rules.
A basic implementation uses a Kubernetes Service with multiple pods, each running a different version:
# Deployments for different versions
apiVersion: apps/v1
kind: Deployment
metadata:
name: model-deployment-v1
spec:
replicas: 90
selector:
matchLabels:
app: model-server
version: v1
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: model-deployment-v2
spec:
replicas: 10
selector:
matchLabels:
app: model-server
version: v2Traffic is routed based on labels, so a small portion of requests go to v2 while the rest go to v1. This requires updating routing rules in the load balancer or service mesh.
Failure mode: If the new version starts producing errors, the system must be able to quickly reduce or stop traffic to it. If the rollout process doesn't support gradual rollbacks, a broken model version can affect all traffic until manually corrected.
Model inference performance is critical in production systems. Caching strategies help avoid repeated computation of identical inputs.
A common pattern is caching predictions for frequently requested inputs using a key-value store like Redis or Memcached:
import redis
import pickle
r = redis.Redis(host='localhost', port=6379, db=0)
def predict_cached(model, input_data):
cache_key = f"model:{model.id}:input:{hash(str(input_data))}"
cached_result = r.get(cache_key)
if cached_result:
return pickle.loads(cached_result)
result = model.predict(input_data)
r.setex(cache_key, 3600, pickle.dumps(result))
return resultThis approach reduces the number of calls to the actual model, but introduces a trade-off between latency and consistency. Cached results may become stale if the underlying model is updated.
A sharp edge here is that cache invalidation is tricky. If a model updates without clearing the cache, old predictions continue to be served even though the model has changed. This can lead to silent degradation where models are updated but users see old behavior.
Models are typically served through HTTP endpoints that accept requests and return predictions. These endpoints must handle multiple concurrent requests efficiently.
A minimal serving implementation using Python Flask:
from flask import Flask, request, jsonify
import joblib
app = Flask(__name__)
model = joblib.load('model.pkl')
@app.route('/predict', methods=['POST'])
def predict():
data = request.get_json()
prediction = model.predict([data['features']])
return jsonify({'prediction': prediction[0]})
if __name__ == '__main__':
app.run(host='0.0.0.0', port=8080)This endpoint must be resilient to high load and handle various input types. It should also include error handling for malformed requests or model failures.
Failure mode: If the model fails during inference, the HTTP endpoint must return a meaningful error response. A common mistake is returning generic 500 errors without context about what went wrong in the model itself.
Models can degrade over time due to data drift or concept drift. Production systems must detect this degradation and trigger retraining or model updates.
Monitoring involves tracking prediction quality metrics such as accuracy, confidence scores, or specific performance indicators. This requires setting up logging systems that capture both inputs and outputs:
def log_prediction(input_data, prediction, model_version):
# Log to a monitoring system
log_entry = {
'input': input_data,
'prediction': prediction,
'model_version': model_version,
'timestamp': time.time()
}
# Send to logging backend
send_to_monitoring_system(log_entry)If metrics fall below thresholds, the system should trigger alerts or initiate a retraining pipeline. This requires establishing feedback loops between production systems and training environments.
The sharp edge here is that model degradation might be subtle. A model may still produce acceptable results but with reduced confidence or shifted decision boundaries. These changes can go unnoticed until they cause downstream failures.
Model deployments should integrate with standard CI/CD pipelines to ensure consistent and repeatable deployment processes. This involves automating the steps from model training to serving.
A typical pipeline includes:
Each step should have clear success criteria and failure handling. For example, a model must pass performance tests before being deployed.
Example workflow using GitOps principles:
# Commit model files to git
git add model.pkl
git commit -m "Update model to v2.1.0"
git push origin main
# Pipeline automatically builds container image
docker build -t my-model-server:v2.1.0 .
docker push my-model-server:v2.1.0
# Deploy to Kubernetes cluster
kubectl set image deployment/model-deployment model-server=my-model-server:v2.1.0Failure mode: If pipeline steps are not properly validated, a broken model can be deployed without detection. This requires ensuring that validation tests run before any production deployment.
Every deployment should include rollback mechanisms to quickly revert changes when issues are detected. This includes tracking which version is currently running and maintaining access to previous versions.
A simple approach uses version tags in the container image or configuration:
# Deployment with version tag
apiVersion: apps/v1
kind: Deployment
metadata:
name: model-deployment
spec:
replicas: 3
template:
spec:
containers:
- name: model-server
image: my-model-server:v2.1.0When rolling back, the system changes the version tag to a known working version.
The sharp edge is that rollbacks must be fast and reliable. If rollback mechanisms are complex or fail, the system may remain in an inconsistent state for extended periods.
Model serving systems must scale appropriately to handle varying loads. This includes setting resource limits for containers and configuring auto-scaling rules.
Example Kubernetes resource configuration:
apiVersion: apps/v1
kind: Deployment
metadata:
name: model-deployment
spec:
replicas: 3
template:
spec:
containers:
- name: model-server
image: my-model-server:v2.1.0
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"The system should also include horizontal scaling rules based on metrics like CPU usage or request queue length.
Failure mode: If resources are not properly configured, the system may either under-provision and fail during load spikes, or over-provision and waste resources. The key is finding the right balance between cost and performance.
Production systems must include monitoring of model behavior to detect anomalies or performance issues early. This includes tracking metrics like prediction latency, error rates, and throughput.
Alerting rules should trigger when metrics exceed thresholds or show unexpected patterns:
# Prometheus alert rule example
groups:
- name: model-alerts
rules:
- alert: ModelPredictionLatencyHigh
expr: histogram_quantile(0.95, sum(rate(model_prediction_duration_seconds_bucket[5m])) by (le))
for: 10m
labels:
severity: warning
annotations:
summary: "Model prediction latency is high"The monitoring system should also capture model-specific metrics like confidence scores or feature usage patterns.
A sharp edge is that alerting rules must be tuned carefully to avoid noise from normal variations. Too many alerts can lead to alert fatigue, while too few can miss real issues.
Updating models in production requires careful coordination between the training pipeline and deployment system. This includes version control, testing, and gradual rollout strategies.
The most common approach is to deploy new versions without immediate traffic shift, then validate performance before full rollout:
# Deploy new model version
kubectl set image deployment/model-deployment model-server=my-model-server:v2.1.0
# Wait for pods to become ready
kubectl rollout status deployment/model-deployment
# Validate with canary requests
curl -H "X-Model-Version: v2.1.0" http://model-service/predictFailure mode: If updates are not tested thoroughly or if rollback mechanisms are not in place, a model update can cause widespread service degradation.
Some systems serve multiple models simultaneously, either for A/B testing or handling different types of inputs. This requires managing routing between models based on input characteristics.
Feature flags can control which models are active or how traffic is distributed:
def route_model(input_data):
if input_data['user_type'] == 'premium':
return 'model-premium'
else:
return 'model-standard'This pattern allows for gradual rollout of new models or experimentation with different approaches.
The sharp edge is ensuring that all models in a multi-model setup are consistently updated and monitored. A mismatch between model versions can lead to inconsistent predictions or silent failures.
This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy