Build a centralized AI gateway for routing, rate limiting, cost tracking, and model fallback across multiple LLM providers.
An AI gateway sits at the heart of a production AI system, routing inference requests, managing model lifecycle, and coordinating between services. It matters when multiple models serve different parts of an application, when request patterns vary widely (batch vs. real-time), and when models must be versioned, monitored, and scaled independently. It is the single point of truth for AI services, transforming raw inputs into AI-powered outputs with consistency, observability, and resilience.
The gateway receives incoming requests and dispatches them to appropriate models. Each request carries metadata—model_version, priority, timeout, features—that guide routing decisions.
For a real-time inference request, a client sends a JSON payload over HTTP:
POST /predict HTTP/1.1
Host: ai-gateway.example.com
Content-Type: application/json
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yVqojrYHDijq
{
"input": "What is the capital of France?",
"model_version": "v2.1",
"features": {
"context_length": 128,
"embedding_size": 768
}
}The gateway matches the model_version and features against a routing table stored in a database. The routing table is a PostgreSQL table with columns: model_name, version, deployment_id, load_balancer_strategy, thresholds.
SELECT deployment_id, load_balancer_strategy
FROM model_routing
WHERE model_name = 'text-classifier'
AND version = 'v2.1'
AND (context_length >= 128 AND embedding_size >= 768);The gateway uses the load_balancer_strategy (round_robin, least_connections, consistent_hashing) to pick a deployment. If no match is found, it defaults to the latest stable version.
A common failure mode: requests fail silently when model_version is misspelled. The gateway logs model_version not found: v2.1 but returns HTTP 500 with no body. The root cause is a mismatch between the version string in the request and the database, where v2.1 is stored as v2.1.0. The solution is to normalize all version strings during registration.
Models are registered with the gateway before they can be used. Registration is a two-step process: first, the model is uploaded; second, it is deployed.
Model upload uses a standard multipart/form-data request:
POST /models/register HTTP/1.1
Host: ai-gateway.example.com
Authorization: Bearer <admin-token>
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="model_file"; filename="model.onnx"
Content-Type: application/octet-stream
<binary data of model.onnx>
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="metadata"
Content-Type: application/json
{
"name": "text-classifier",
"version": "v2.1",
"framework": "onnx",
"input_schema": {
"type": "string",
"max_length": 1024
},
"output_schema": {
"type": "array",
"element_type": "string",
"max_elements": 10
},
"dependencies": [
"numpy==1.24.0",
"onnxruntime==1.14.0"
],
"min_resources": {
"cpu": 2,
"memory": "4Gi",
"gpu": 1
},
"default_timeout_ms": 500,
"metrics": [
"latency_p95",
"throughput_qps",
"error_rate"
]
}
------WebKitFormBoundary7MA4YWxkTrZu0gW--Upon successful upload, the gateway stores the model in a shared object store (e.g., S3) and updates the models table:
INSERT INTO models (
name, version, file_uri, metadata, created_at, status
) VALUES (
'text-classifier', 'v2.1', 's3://models/text-classifier/v2.1.onnx',
'{"framework": "onnx", "input_schema": {"type": "string", "max_length": 1024}, ...}',
'2024-05-12T08:30:00Z',
'registered'
);The gateway then triggers a deployment job via a message queue (e.g., RabbitMQ). The job consumes the model, builds a Docker image, and registers it with a container orchestration system.
Deployment is asynchronous. A client can poll /models/{name}/{version}/status:
GET /models/text-classifier/v2.1/status HTTP/1.1
Host: ai-gateway.example.com
Authorization: Bearer <client-token>Response:
{
"model_name": "text-classifier",
"version": "v2.1",
"status": "deploying",
"progress": 0.67,
"logs": [
"Building image: python:3.10-slim",
"Copying model file to container",
"Running onnxruntime test suite",
"Starting service on port 8000"
],
"last_updated": "2024-05-12T08:35:12Z"
}A sharp edge: if the gateway receives a request before deployment completes, it returns HTTP 503 Service Unavailable with a Retry-After header. The client must handle Retry-After: 15 and backoff. Misconfiguring Retry-After leads to repeated client timeouts, especially when deployment takes 20 seconds but the gateway sets Retry-After: 5.
The gateway applies preprocessing pipelines to inputs before sending them to the model. This includes input validation, feature extraction, and format conversion.
A request to /predict may include a preprocessing_pipeline field:
{
"input": "What is the capital of France?",
"model_version": "v2.1",
"preprocessing_pipeline": [
"normalize_text",
"tokenize",
"embed_with_bert"
]
}The gateway checks the preprocessing_pipeline against a registry of available pipelines stored in a JSON table:
SELECT pipeline_id, steps, dependencies, description
FROM preprocessing_pipelines
WHERE name = 'text-classifier-preprocess'
AND version = 'v1.0';Each step is a function that transforms the input. The gateway applies them sequentially:
normalize_text: converts input to lowercase, removes extra whitespace, corrects punctuation.tokenize: splits text into tokens using a tokenizer (e.g., bert-base-uncased).embed_with_bert: runs a BERT model to produce embeddings.The transformed input is passed to the model as a dictionary:
{
"input_ids": [101, 2026, 2044, 102, 2124, 102],
"attention_mask": [1, 1, 1, 1, 1, 1],
"sequence_length": 6
}If a step fails, the gateway logs the error and returns HTTP 502 Bad Gateway with a detailed error message:
{
"error": "Preprocessing pipeline failed",
"details": {
"step": "embed_with_bert",
"error_code": "model_load_failed",
"message": "Failed to load BERT model from /models/bert-large-uncased/1.0.onnx",
"timestamp": "2024-05-12T08:40:03Z",
"retryable": true
},
"pipeline_steps": [
"normalize_text",
"tokenize",
"embed_with_bert"
]
}A silent failure: tokenize step uses a tokenizer trained on a different vocabulary than expected. The gateway logs Tokenizer mismatch: expected vocab size 30522, got 30512. The client sees correct output but with suboptimal results—no one notices until a model drift alert fires.
The gateway supports multiple model versions in parallel. It routes requests based on version and traffic weights.
To enable A/B testing, the gateway supports a traffic_split configuration:
{
"experiment_id": "ab-test-2024-05",
"model_versions": {
"v2.1": 0.7,
"v2.2": 0.3
},
"routing_strategy": "weighted_random",
"metadata": {
"variant_a": "v2.1",
"variant_b": "v2.2",
"target_accuracy": 0.91
}
}The gateway assigns each incoming request to a version with the specified probability. The routing_strategy determines how the choice is made:
weighted_random: each request is independently assigned.consistent_hashing: same input always goes to the same version, enabling session-level consistency.The gateway tracks A/B test metrics via a experiment_metrics table:
INSERT INTO experiment_metrics (
experiment_id, version, request_id, latency_ms, success, model_output, timestamp
) VALUES (
'ab-test-2024-05',
'v2.1',
'req-12345',
124,
true,
'{"label": "Paris", "confidence": 0.94}',
'2024-05-12T08:45:01Z'
);When a new version is promoted, the gateway updates the traffic_split and notifies downstream services via a POST /experiments/{id}/promote call. The promotion is idempotent: calling it multiple times with the same experiment_id and version updates only the active version.
A failure mode: traffic_split is updated but the gateway does not propagate the change immediately. Clients see version v2.1 for 10 minutes after v2.2 is promoted. The fix is to use a cache layer (e.g., Redis) with a TTL of 30 seconds, and to invalidate the cache on every traffic_split update.
The gateway exposes a health endpoint at /health:
GET /health HTTP/1.1
Host: ai-gateway.example.comResponse:
{
"status": "healthy",
"timestamp": "2024-05-12T08:50:00Z",
"uptime_seconds": 123456,
"services": {
"model_registry": {
"status": "healthy",
"latency_ms": 23,
"details": {
"database_connection": "up",
"cache_hit_rate": 0.87
}
},
"deployment_scheduler": {
"status": "healthy",
"pending_jobs": 12,
"last_job_completed": "2024-05-12T08:49:58Z"
},
"preprocessing_engine": {
"status": "warning",
"latency_ms": 112,
"details": {
"last_error": "Failed to load tokenizer from /models/tokenizers/2024-04-15.json",
"error_count": 4
}
}
},
"active_models": [
"text-classifier:v2.1",
"image-classifier:v1.3"
],
"error_logs": [
{
"timestamp": "2024-05-12T08:48:10Z",
"level": "error",
"message": "Failed to load model file from S3: s3://models/text-classifier/v2.1.onnx",
"correlation_id": "c-123456"
}
]
}Clients use this endpoint for liveness and readiness probes. The gateway also exposes metrics via Prometheus:
ai_gateway_request_duration_seconds: histogram of request latency by model and version.ai_gateway_model_deployment_status: gauge of deployment status (0 = failed, 1 = success).ai_gateway_experiment_traffic_split: vector of traffic weights per experiment.A common failure: a model version is registered but not yet deployed. The health check returns status: healthy, but active_models lists only deployed models. The gateway must poll the deployment system every 30 seconds to update active_models.
The gateway implements retry policies per model and request type. A request may fail due to model unavailability, timeout, or input schema mismatch.
On failure, the gateway returns a standard error structure:
{
"error": "ModelInferenceFailed",
"code": "model_inference_failed",
"message": "Inference request failed for text-classifier:v2.1",
"details": {
"model_version": "v2.1",
"input_schema": {
"expected": {
"type": "string",
"max_length": 1024
},
"actual": {
"type": "array",
"length": 3
}
},
"error_type": "schema_mismatch",
"retriable": true,
"retry_after": 5000,
"correlation_id": "c-789012"
},
"timestamp": "2024-05-12T08:55:30Z"
}The gateway retries failed requests using exponential backoff. The retry policy is configured per model:
{
"model_name": "text-classifier",
"version": "v2.1",
"retry_policy": {
"max_retries": 3,
"base_delay_ms": 100,
"max_delay_ms": 2000,
"backoff_factor": 1.5,
"retry_on": [
"timeout",
"network_error",
"schema_mismatch"
],
"ignore_errors": [
"model_not_found",
"version_not_available"
]
}
}If a request fails after 3 retries, the gateway sends it to a dead-letter queue (e.g., Kafka) for manual inspection. A consumer service processes the dead-letter queue, logs errors, and sends alerts via email or Slack.
A sharp edge: the gateway does not handle timeout correctly when the model service returns HTTP 200 but with a large Content-Length and slow streaming. The gateway considers the request successful even if the model takes 8 seconds and the gateway timeout is 5 seconds. The fix is to use read_timeout and body_timeout separately, and to validate that the entire response body is received before marking the request as complete.
This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy