AI Observability

Monitor LLM applications in production with trace logging, token usage dashboards, latency percentiles, and quality score tracking.

AI Observability is the practice of instrumenting, monitoring, and diagnosing the behavior, performance, and health of AI systems throughout their lifecycle — from training to inference, from deployment to feedback. It matters when a model fails silently in production, when latency spikes without warning, when predictions degrade over time, or when a new feature rollout introduces subtle but costly errors. Observability is not just logging; it is the ability to answer: *Why did this model make this prediction? How did it evolve? What inputs caused this drift? Where did the pipeline break?*

Instrumenting AI Systems

To observe AI systems effectively, every component must emit structured, traceable data. The foundation is a consistent telemetry format: structured logs, metrics, and traces, all linked by a shared context.

{
  "trace_id": "7a3c2b1d-4e5f-6a7b-8c9d-0e1f2a3b4c5d",
  "span_id": "1b2c3d4e-5f6a-7b8c-9d0e-1f2a3b4c5d6a",
  "timestamp": "2024-05-14T10:30:00.123Z",
  "service": "recommendation-engine",
  "operation": "predict",
  "inputs": {
    "user_id": 42381,
    "session_duration": 142,
    "items_viewed": [101, 102, 103],
    "context": "mobile"
  },
  "outputs": {
    "predicted_items": [204, 205, 206],
    "confidence_scores": [0.87, 0.74, 0.61],
    "model_version": "v3.2.1",
    "latency_ms": 84
  },
  "metadata": {
    "request_id": "req-7a3c2b1d",
    "batch_id": "batch-12345",
    "feature_vector_size": 128
  }
}

Each log entry must carry a trace_id and span_id to enable distributed tracing across microservices. Logs are emitted at multiple levels: model inference, preprocessing, feature extraction, and postprocessing. Each service logs to a standard schema, and a central logging system (e.g., Fluentd, Logstash, or Loki) aggregates them.

Metrics are collected via Prometheus-style exporters. A model server exposes a /metrics endpoint that returns:


# TYPE model_inference_latency_ms histogram
model_inference_latency_ms_bucket{le="50"} 12345
model_inference_latency_ms_bucket{le="100"} 18721
model_inference_latency_ms_bucket{le="200"} 19803
model_inference_latency_ms_bucket{le="+Inf"} 20000
model_inference_latency_ms_sum 1743210
model_inference_latency_ms_count 20000
model_inference_latency_ms{model="recsys-v3.2.1"} 84.2

A single metric model_inference_latency_ms captures distribution across all requests, and labels (model, environment, instance) allow filtering and alerting.

Tracing Model Execution

Tracing reveals how a prediction flows through the system. A request to /predict may span multiple services: feature store, preprocessing, model inference, and postprocessing.

A trace starts with an HTTP request to the API gateway, which propagates trace_id and span_id via HTTP headers:

GET /predict HTTP/1.1
Host: ai-api.example.com
X-Trace-ID: 7a3c2b1d-4e5f-6a7b-8c9d-0e1f2a3b4c5d
X-Span-ID: 1b2c3d4e-5f6a-7b8c-9d0e-1f2a3b4c5d6a
Content-Type: application/json

{
  "user_id": 42381,
  "session_duration": 142,
  "items_viewed": [101, 102, 103]
}

Each service extracts these headers, creates a new span, and logs to the central trace system. The trace includes:

The trace becomes a diagnostic map: if the total latency is 130ms but inference is only 84ms, the bottleneck is elsewhere. If feature_store_lookup shows a spike in p99 latency, the issue lies in the feature store — perhaps a missing index or a slow database query.

Detecting Model Drift and Data Skew

Drift detection is not a single metric but a pipeline of checks. A daily job runs across the latest batch of data to compute drift between training and serving data.

The drift pipeline uses pandas and scikit-learn to compare feature distributions:

import pandas as pd
from sklearn.metrics import mutual_info_score
from scipy.spatial.distance import jensenshannon

def compute_drift(reference_data: pd.DataFrame, current_data: pd.DataFrame, threshold: float = 0.1):
    drift_results = {}
    for col in reference_data.columns:
        ref_dist = reference_data[col].value_counts(normalize=True)
        curr_dist = current_data[col].value_counts(normalize=True)
        # Merge both distributions
        all_values = set(ref_dist.index) | set(curr_dist.index)
        ref_vec = [ref_dist.get(v, 0) for v in all_values]
        curr_vec = [curr_dist.get(v, 0) for v in all_values]
        js_div = jensenshannon(ref_vec, curr_vec)
        drift_results[col] = {
            "js_divergence": js_div,
            "is_drift": js_div > threshold,
            "reference_count": len(reference_data),
            "current_count": len(current_data)
        }
    return drift_results

The function runs nightly, outputting a drift_report.json:

{
  "timestamp": "2024-05-14T00:00:00Z",
  "drifts": {
    "user_age": {
      "js_divergence": 0.21,
      "is_drift": true,
      "reference_count": 120000,
      "current_count": 118500
    },
    "device_type": {
      "js_divergence": 0.08,
      "is_drift": false,
      "reference_count": 120000,
      "current_count": 118500
    }
  },
  "summary": {
    "total_features": 24,
    "drifted_features": 9,
    "drift_threshold": 0.1
  }
}

This report is ingested into a monitoring system. When a feature’s JS divergence exceeds 0.1, an alert triggers: Feature drift detected: user_age.

Debugging Model Predictions

When a model makes a wrong prediction, the question is: *What went wrong?* The answer lies in visualizing inputs, attention weights, and prediction confidence.

A common pattern is to store model predictions in a database with associated metadata:

CREATE TABLE model_predictions (
    id BIGSERIAL PRIMARY KEY,
    trace_id UUID NOT NULL,
    model_version TEXT NOT NULL,
    input_json JSONB NOT NULL,
    output_json JSONB NOT NULL,
    confidence_scores DOUBLE PRECISION[],
    explanation JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    FOREIGN KEY (trace_id) REFERENCES traces(id)
);

To debug a specific prediction, engineers query:

SELECT id, input_json, output_json, explanation
FROM model_predictions
WHERE trace_id = '7a3c2b1d-4e5f-6a7b-8c9d-0e1f2a3b4c5d'
ORDER BY created_at DESC
LIMIT 1;

The explanation field stores attention weights or SHAP values:

{
  "attention_weights": [
    [0.3, 0.1, 0.6],
    [0.2, 0.4, 0.4],
    [0.8, 0.1, 0.1]
  ],
  "shap_values": {
    "user_age": 0.42,
    "session_duration": 0.67,
    "items_viewed": [0.21, 0.34, 0.18]
  },
  "feature_importance": [
    {"feature": "session_duration", "importance": 0.67},
    {"feature": "user_age", "importance": 0.42},
    {"feature": "items_viewed", "importance": 0.34}
  ]
}

This allows interactive debugging: a user can click a prediction and see which features contributed most, how attention focused on certain input tokens, and why the model chose a specific output.

Monitoring Model Health and Performance

Model health is monitored via a suite of metrics collected at regular intervals:

A health check endpoint /health returns:

{
  "status": "ok",
  "timestamp": "2024-05-14T10:30:00.123Z",
  "services": {
    "feature_store": {
      "status": "up",
      "latency_ms": 18.4,
      "error_rate": 0.002
    },
    "model_inference": {
      "status": "degraded",
      "latency_ms": 95.2,
      "error_rate": 0.03,
      "gpu_utilization": 0.87,
      "memory_usage_mb": 4100
    },
    "postprocessor": {
      "status": "up",
      "latency_ms": 12.1
    }
  },
  "model_versions": {
    "recsys-v3.2.1": {
      "active": true,
      "last_updated": "2024-05-13T14:00:00Z",
      "uptime": "234h",
      "requests": 123456,
      "error_rate": 0.012
    },
    "recsys-v3.1.0": {
      "active": false,
      "last_updated": "2024-04-28T10:15:00Z",
      "uptime": "120h",
      "requests": 45231,
      "error_rate": 0.018
    }
  }
}

This endpoint is polled every 10 seconds by a health monitoring system. When model_inference has latency_ms > 80 and error_rate > 0.02, a new alert is created: Model inference degraded.

Alerting on Anomalies

Anomalies are detected by thresholding and statistical rules. A simple but effective system uses Prometheus alerts:

groups:
  - name: ai-model-alerts
    rules:
      - alert: ModelLatencyP99High
        expr: |
          rate(model_inference_latency_ms{operation="predict"}[5m]) > 100
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "P99 inference latency above 100ms"
          description: |
            The P99 latency for the model inference service has been above 100ms for 10 minutes.
            Current value: {{ $value }}ms
            Model version: {{ $labels.model_version }}

      - alert: ModelDriftDetected
        expr: |
          increase(drift_detected{feature="user_age"}[1d]) > 0
        for: 2h
        labels:
          severity: critical
        annotations:
          summary: "User age feature drifted"
          description: |
            The user_age feature has shown significant drift over the past 2 hours.
            Drift detected in the last 24 hours.

When a rule fires, the alert system triggers a notification (e.g., via Slack, PagerDuty, or email). The alert includes links to dashboards, traces, and raw logs.

Root-Cause Analysis and Failure Modes

The most common silent failure is model version mismatch: a request is sent to a service using model v3.1.0, but the feature store expects v3.2.1. The request fails silently, returning null predictions, with no error log.

The root cause lies in model_version mismatch between:

A common fix: add a validate_model_version middleware that checks the expected model version against the actual version used.

Another failure: data drift undetected due to infrequent evaluation. A model is trained monthly, but the data distribution shifts weekly. The training data becomes stale, but the model continues to serve. The solution is to run drift detection weekly, not just monthly.

A subtle but critical issue: feature store lag. The feature store updates every 15 minutes, but the model runs every 10 seconds. When a new user arrives, the feature vector is not available. The solution: introduce a feature_cache layer with TTL, and log a feature_missing metric.

Summary

AI Observability is not a set of tools but a mindset: every component must emit structured telemetry, linked by a shared context. It answers: *What did the model see? Why did it decide that? How long did it take? What went wrong?* The tools are not new — they are the standard stack of distributed systems: structured logs, metrics, traces, and databases. But their application to AI systems requires discipline: consistent schema, regular evaluation, and proactive alerting. The sharp edges — version mismatches, data lag, and silent failures — define the difference between a functioning AI system and a reliable one.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.