Model Serving Patterns | The Garnet Wiki

Model serving is the layer that translates a trained model artifact into a consumable prediction endpoint. It matters whenever you need to expose predictions over HTTP or gRPC, manage model versions without retraining, enforce access controls, or scale compute independently of training pipelines. This page maps the patterns you’ll actually use to get a model from artifact to endpoint, and the failure modes that catch teams off guard.

Deploying a New Model Version

The most common entry point is pushing a model directory or artifact to a serving endpoint. In a Kubernetes environment, this typically begins with a YAML descriptor and kubectl apply. A minimal deployment manifest might look like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: model-server
  template:
    metadata:
      labels:
        app: model-server
    spec:
      containers:
      - name: server
        image: placeholder-image
        ports:
        - containerPort: 8080
        env:
        - name: MODEL_NAME
          value: "my-model"
        - name: MODEL_PATH
          value: "/models/my-model"
        resources:
          limits:
            memory: "512Mi"
            cpu: "500m"

Applying it is an exact, real command:

kubectl apply -f model-deployment.yaml

If the image, path, or env keys are misaligned, the container starts but the model fails to load. A typical runtime error, expressed in the server’s log and surface‑level HTTP response, reads:

ERROR: Model directory /models/my-model not found or not readable
HTTP 503: Service Unavailable - model not loaded

Sharp edges. The most frequent surprise is a path mismatch between the host‑mounted volume and the container’s working directory. If the volumeMount references /data/models but the container expects /models, the server logs the “not found” error above without crashing, leaving the endpoint perpetually unavailable. Another common pitfall: signature version drift. If the model was trained with a TensorFlow/Keras signature serving_default but the server is configured to look for predict, every request returns {"error": "signature not found"} until the serving tag is aligned. Environment variables that control thread counts or GPU memory fractions are often ignored if the serving binary was compiled without those flags, resulting in OOM kills that appear as Exit code 137 with no explicit model‑loading error.

Rolling Out Incrementally

When you need to replace a model without taking traffic down, incremental rollouts—canary or blue‑green—are the pattern of choice. Kubernetes provides native tooling that avoids custom scripts.

To pause a rollout while you verify a new image or config:

kubectl rollout pause deployment/model-server

After making changes (updating the image tag, adjusting env vars, or patching resources), resume and monitor:

kubectl rollout resume deployment/model-server
kubectl rollout status deployment/model-server --timeout=300s

If the new version passes health checks, you can promote it fully. To undo a bad release instantly:

kubectl rollout undo deployment/model-server

Traffic shifting between versions is typically done via service selectors or ingress rules. A real kubectl patch example that switches the active model version label:

kubectl patch deployment model-server \
  -p '{"spec":{"template":{"spec":{"selector":{"matchLabels":{"version":"v2"}}}}}'

Sharp edges. Request routing can race with model‑state initialization. If a client sends a prediction request to a pod that has just received a new image but whose container is still loading the model, the request returns 503 Service Unavailable or, worse, a partially populated prediction that the client may cache as valid. Session affinity (sticky sessions) can mask this by always hitting the same pod, but it defeats the purpose of a canary if the “v1” pod still holds stale model weights. Another gotcha: database or cache migrations. If the new model version expects a different schema or key structure, and the migration runs only on the “v2” pod, requests routed to “v1” may fail with 400 Bad Request: invalid field while “v2” requests succeed, creating an inconsistent user experience that’s hard to debug in distributed logs.

Batch versus Real‑Time Inference

The choice between serving a model synchronously over HTTP/gRPC or as a batch job depends on latency requirements, data volume, and downstream consumption patterns.

Real‑time serving is the default for interactive applications. A client issues a POST and waits for a response. Using curl, a typical request looks like:

curl -s -X POST http://model-server.example.com/v1/models/my-model:predict \
  -H "Content-Type: application/json" \
  -d '{"instances": [[1.0, 2.3, 4.5]]}'

A successful response returns JSON (or protobuf) with predictions; a timeout or processing error returns HTTP 504 Gateway Timeout or {"code": "deadline_exceeded", "message": "prediction took too long"}.

Batch inference decouples request intake from prediction execution. A common pattern involves writing input records to a durable queue (e.g., a file per partition, Kafka topic, or cloud storage bucket), then running a worker process that reads, runs the model, and writes results back. A Docker‑based worker command that reads from a mounted input directory and writes to an output directory might be expressed in prose as:

docker run -v /input:/input -v /output:/output \
  -e MODEL_PATH=/models/my-model \
  my-worker-image --input /input --output /output --batch-size 100

The worker exits with 0 on success, non‑zero on failure, and its stdout/stderr flow into whatever log aggregation system the cluster provides.

Sharp edges. In batch pipelines, backpressure is the silent killer. If the queue accumulates faster than workers can process, memory on the worker node grows until the OOM killer intervenes, often without a clean shutdown signal. Timeout cascades occur when a batch job’s internal model‑warmup phase (loading the model into RAM or GPU) is not accounted for in the job’s wall‑time estimate, causing the job to be killed just as it begins producing predictions. Stale data is another edge case: if the input queue contains records generated before a model version swap, those records may be processed by the new model without validation, producing predictions based on an incompatible feature schema.

Auth, Validation, and Request Shaping

Every serving endpoint needs some form of request admission control. The pattern is typically a front‑proxy or API‑gateway layer that validates tokens, enforces rate limits, and shapes the inbound payload before it reaches the model container.

A client calling a protected endpoint with a bearer token uses a command like:

curl -s -X POST http://model-server.example.com/v1/predict \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"instances": [[1.0,2.3]]}'

If the token is missing or invalid, the gateway returns HTTP 401 Unauthorized or HTTP 403 Forbidden before the request touches the model process. Request bodies that exceed a configured size limit trigger HTTP 413 Payload Too Large; most serving setups cap payloads at a few megabytes to prevent denial‑of‑service amplification.

Configuration keys that control these behaviours are rarely product‑specific in their semantics. A typical gateway config might include:

auth.enabled: true – toggles token validation.
rate_limit.requests_per_minute: 100 – caps inbound requests per client.
max_request_bytes: 1048576 – hard limit on body size.

When these keys are mis‑set, the gateway may either pass unauthenticated traffic (security gap) or reject legitimate payloads (operational friction). A subtle edge is token propagation: some deployments forward the original client token to the model container via an environment variable or header, while others strip it and rely on internal service identities. If the model container expects an Authorization header and receives none, it may log authentication failed and refuse to run inference, returning 500 Internal Server Error to the caller.

Observability: Logs, Metrics, and Tracing

Model serving generates three categories of observability data: structured logs, time‑series metrics, and distributed traces. Collecting them without adding latency is a persistent concern.

Logs are the default output. Most serving containers write request‑level entries to stdout, structured as JSON lines:

{"time":"2024-01-15T12:34:56Z","level":"INFO","msg":"request received","model":"my-model","version":"v2","latency_ms":42}

Aggregating these into Elasticsearch, Loki, or a Splunk index is a deployment‑level concern, not a model‑level one.

Metrics are usually exposed via a /metrics endpoint in Prometheus format. Common labels include model_name, model_version, request_status (success, error, timeout), and latency_seconds. A real Prometheus query fragment that counts errors by model version might read:

sum by (model_version) (rate(model_requests_total{status="error"}))

When metrics are mis‑labeled—forgetting to add model_version or using inconsistent casing—the resulting dashboards show flat lines or aggregate across all versions, masking a degrading canary or a rogue rollout. Tracing (OpenTelemetry, Jaeger) adds request IDs that flow from the gateway through the model container to downstream databases. If the model container does not inject or propagate the trace context, downstream services cannot correlate latency, and “cold start” periods after a rollout appear as random spikes rather than predictable initialization phases.

Sharp edges. Log flooding is the most common production surprise. During a sudden traffic surge, stdout from 20 replica pods can exceed several gigabytes per minute, overwhelming log collectors and causing log loss. Metric aggregation delays of 1–2 minutes mean that a sudden spike in model_latency_seconds p99 may not appear on a Grafana dashboard until after the incident has resolved. Warmup latency is another hidden factor: after a pod restarts or a new image is pulled, the first ~10–50 requests often take 2–5× longer as the model loads into memory or initializes GPU contexts. If alerting thresholds are set on overall p99 latency, these warmup requests can trigger false‑positive alerts that trigger another rollout, compounding the problem.

Rollback, Version Pinning, and State Management

When a serving change introduces a regression, the fastest corrective path is a rollback. Kubernetes makes this a single command:

kubectl rollout undo deployment/model-server

The controller reverts the deployment to the previous .spec.revision, new pods start with the prior image/tag, and traffic gradually shifts back. If the rollout is stuck because of a Progressing condition (e.g., replicas unavailable: pod has unbound immediate PersistentVolumeClaim), the undo may report error: deployment "model-server" has progressing replicas, and manual intervention is required to resolve the volume binding before the rollback can advance.

Version pinning is the practice of locking a serving deployment to a specific model artifact, rather than relying on “latest” or ambiguous tags. In a Helm‑based or Kustomize‑driven workflow, this looks like:

args:
  - --model_name=my-model
  - --model_base_path=/models/[email protected]

The @v1.3.0 suffix is a convention, not a universal flag; the exact syntax depends on the serving binary. Pinning prevents accidental upgrades when a new model version is pushed to the same artifact repository, and it makes kubectl rollout history and kubectl rollout undo deterministic.

Sharp edges. Client‑side caches are a frequent blind spot. If front‑end applications cache model metadata (e.g., expected input schema, output format) based on the previous version’s response shape, a rollback to an older version may produce payloads that the client parses incorrectly, leading to TypeError: cannot read property X of undefined in the browser or mobile app. Database schema drift is another silent failure mode: if the new model version writes prediction results to a table with a added column, and the rollback reverts the model but not the table, subsequent runs by the old model may encounter column not found errors when trying to read columns that only the new model wrote. Lastly, partial rollouts—where only a subset of replicas are updated before a rollback is triggered—can leave a mix of old and new pods serving traffic, causing intermittent 502 Bad Gateway errors if the ingress layer’s health checks are mis‑configured to only route to the newest replica set.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.