Design production LLM applications that are reliable, cost-efficient, and maintainable. Covers prompt engineering patterns, model routing, caching strategies, evaluation frameworks, and the operational patterns for running LLMs at scale.
LLM application architecture extends beyond the basic POST /v1/completions call by orchestrating multiple LLM components—prompt engineering, caching, batching, state management, error recovery, and asynchronous execution—into a cohesive system. This architecture matters when a single API call becomes a bottleneck, when latency exceeds user tolerance, when prompt quality drifts over time, or when a single model cannot handle the full scope of a user’s intent. It is essential for production-grade applications that serve thousands of concurrent users, process complex workflows, or maintain consistent, high-fidelity interactions across sessions.
Prompt templates define the interface between application logic and LLM behavior. Templates must be versioned, tested, and parameterized. A typical prompt is stored as a JSON file, with embedded Jinja2-like syntax for dynamic content.
{
"name": "summarize-document",
"version": "v2.1",
"prompt": "You are an expert editor. Summarize the following document in {{ max_sentences }} sentences.\n\n---\n\n{{ content }}\n\n---\n\nOutput only the summary, no markdown, no preamble.",
"parameters": {
"max_sentences": {
"type": "integer",
"default": 3,
"description": "Number of sentences in the final summary"
},
"tone": {
"type": "string",
"enum": ["concise", "detailed", "academic", "journalistic"],
"default": "concise"
}
}
}Templates are loaded at runtime using a PromptLoader service that resolves a template by name and version. The loader accepts a template_id and parameters, and returns a rendered prompt string.
prompt = prompt_loader.load(
template_id="summarize-document",
version="v2.1",
parameters={"max_sentences": 5, "tone": "academic"}
)The rendering process fails silently if a required parameter is missing, but logs a structured error:
{
"error": "MissingRequiredParameter",
"template_id": "summarize-document",
"version": "v2.1",
"missing": ["max_sentences"],
"context": {
"request_id": "req-7a2b8c",
"timestamp": "2024-03-15T12:34:56Z"
}
}The system also supports fallbacks: if a template version is missing, it uses the latest stable version. If no template exists, it falls back to a default template defined in code.
Caching is essential for reducing LLM call frequency and latency. The system uses a two-tier cache: in-memory (Redis) and on-disk (local filesystem or S3).
Each cache entry is identified by a cache_key, derived from the input parameters and the template signature:
cache_key = f"llm:{template_id}:{version}:{hash(parameters)}"The hash function computes a SHA-256 digest of the parameters, ensuring that small changes in input yield distinct cache keys.
Cache operations are performed via a CacheManager service with the following interface:
result = cache_manager.get(cache_key)
if not result:
result = llm_client.generate(prompt)
cache_manager.set(cache_key, result, ttl=3600) # 1 hourThe cache layer handles cache misses, cache hits, and TTL expiration. On cache miss, the system generates the LLM output and stores it with a configurable TTL.
A common failure mode is cache stampede: multiple concurrent requests for the same cache_key when the cache is cold. The system resolves this using a distributed cache lock mechanism with Redis:
lock_key = f"cache:lock:{cache_key}"
acquired = redis.set(lock_key, "1", nx=True, ex=30) # EX: expire in 30s
if acquired:
# This thread will generate the result
result = llm_client.generate(prompt)
cache_manager.set(cache_key, result, ttl=3600)
redis.delete(lock_key) # Release lock
else:
# Wait for the result to be generated by another thread
while not redis.exists(cache_key):
time.sleep(0.1)
result = cache_manager.get(cache_key)Cache invalidation must be explicit. A new version of a template triggers a cache purge:
curl -X POST http://cache-service:8080/invalidate \
-H "Content-Type: application/json" \
-d '{"template_id": "summarize-document", "version": "v2.1"}'The system also supports warm-up jobs that pre-populate the cache with common inputs, reducing cold-start latency.
For high-throughput applications, LLM calls are batched to reduce per-call overhead. The system uses a message queue (e.g., RabbitMQ or Kafka) to collect LLM requests, then processes them in batches.
Each batch is a JSON array of requests:
[
{
"request_id": "req-1",
"template_id": "summarize-document",
"version": "v2.1",
"parameters": {
"max_sentences": 4,
"tone": "journalistic"
},
"input": "The company reported record revenue..."
},
{
"request_id": "req-2",
"template_id": "classify-text",
"version": "v1.0",
"parameters": {
"label_set": ["news", "marketing", "technical"]
},
"input": "The new AI model reduces inference latency by 40%."
}
]A worker process consumes from the queue, groups requests into batches of size N, and sends them to the LLM API using a batch_generate endpoint:
POST /v1/completions/batch
Content-Type: application/json
[
{
"prompt": "You are an expert editor. Summarize the following document in 4 sentences.\n\n---\n\nThe company reported record revenue...",
"max_tokens": 256
},
{
"prompt": "Classify the following text into one of: news, marketing, technical.\n\n---\n\nThe new AI model reduces inference latency by 40%.",
"max_tokens": 128
}
]The response is a list of results, preserving the order of requests:
[
{
"request_id": "req-1",
"result": "The company achieved record revenue in Q4, driven by strong cloud product adoption and expansion into Europe.",
"metadata": {
"model": "gpt-4-turbo",
"input_tokens": 128,
"output_tokens": 32,
"latency_ms": 240
}
},
{
"request_id": "req-2",
"result": "technical",
"metadata": {
"model": "gpt-4-turbo",
"input_tokens": 64,
"output_tokens": 12,
"latency_ms": 190
}
}
]Batching introduces new failure modes. If a request fails during batch processing, the system logs the error and retries the entire batch with exponential backoff.
When LLM processing takes longer than a user expects, the system returns a task_id immediately and performs the work asynchronously.
POST /api/v1/tasks
Content-Type: application/json
{
"template_id": "generate-report",
"parameters": {
"sections": ["executive-summary", "market-analysis", "financials"],
"tone": "academic"
},
"input": "Data from Q1 2024..."
}The response includes:
{
"task_id": "task-9f8e7d",
"status": "queued",
"created_at": "2024-03-15T12:34:56Z",
"result_url": "/api/v1/tasks/task-9f8e7d/result"
}A background worker polls a task queue (e.g., via SELECT * FROM tasks WHERE status = 'queued' FOR UPDATE SKIP LOCKED) and processes each task:
request_id and input in a Task table.status: "completed" and result.The task system supports polling, webhook, and long-polling for result retrieval:
GET /api/v1/tasks/task-9f8e7d/result?wait=30If wait=30, the server holds the connection open for up to 30 seconds, returning the result as soon as it’s ready.
Tasks are persisted in a PostgreSQL database with the following schema:
CREATE TABLE tasks (
task_id TEXT PRIMARY KEY,
status TEXT NOT NULL CHECK (status IN ('queued', 'processing', 'completed', 'failed')),
input JSONB,
parameters JSONB,
result JSONB,
error TEXT,
created_at TIMESTAMPTZ NOT NULL,
updated_at TIMESTAMPTZ NOT NULL,
model TEXT,
input_tokens INT,
output_tokens INT,
latency_ms INT
);A common failure is task loss: a worker crashes before updating a task. The system handles this via idempotent task processing: each task can be processed multiple times without side effects. The task_id is used as a unique identifier, and updates are idempotent through UPDATE ... WHERE task_id = ?.
LLM calls fail for many reasons: network timeouts, model unavailability, input validation errors, and model-specific issues. The system must handle these failures with a layered strategy.
Each LLM call includes a retry configuration:
{
"max_retries": 3,
"backoff": {
"type": "exponential",
"initial_delay_ms": 100,
"multiplier": 2,
"max_delay_ms": 3000
},
"retry_on": [
"timeout",
"5xx",
"408",
"429",
"model_not_available"
],
"timeout_ms": 5000
}The system retries on HTTP 408 (Request Timeout), 429 (Too Many Requests), and any 5xx status code. It also recognizes a custom error code model_not_available, returned when the LLM model is not loaded.
On retry, the system logs the attempt number, the error, and the current state:
{
"attempt": 2,
"error": "TimeoutError",
"message": "Request timed out after 5000ms",
"details": {
"request_id": "req-1234",
"model": "gpt-4-turbo",
"input_tokens": 1024,
"request_body": "..."
}
}If all retries fail, the task is marked failed, and the error is stored in the error column of the tasks table.
The system also supports circuit breaking: if a model fails consecutively for 10 minutes, the system marks it as "unavailable" and redirects all future calls to a fallback model.
The system is configured using a single app.yaml file, with no vendor-specific keys:
llm:
client:
endpoint: "https://api.openai.com/v1/completions"
api_key: "sk-abc123xyz"
timeout_ms: 5000
max_retries: 3
backoff:
type: "exponential"
initial_delay_ms: 100
multiplier: 2
max_delay_ms: 3000
cache:
type: "redis"
host: "redis:6379"
db: 0
ttl_seconds: 3600
warmup:
- template_id: "summarize-document"
version: "v2.1"
parameters: { max_sentences: 3 }
- template_id: "classify-text"
version: "v1.0"
parameters: { label_set: ["news", "marketing"] }
batching:
enabled: true
batch_size: 16
max_wait_ms: 200
queue: "llm-requests"
asynchronous:
task_timeout_ms: 60000
result_ttl_hours: 72
webhook_url: "https://webhook.example.com/llm-result"Deployment uses standard tooling: docker-compose.yml, systemd services, and kubectl for Kubernetes.
services:
llm-app:
image: ai-llm-app:latest
ports:
- "8080:8080"
volumes:
- ./config/app.yaml:/app/config/app.yaml
depends_on:
- redis
- queue
environment:
- LOG_LEVEL=info
- TASK_QUEUE=llm-requestsThe application starts with a health check:
GET /healthResponse:
{
"status": "healthy",
"details": {
"llm_client": "connected",
"cache": "redis://redis:6379",
"queue": "rabbitmq://queue:5672/llm-requests",
"tasks": {
"total": 127,
"in_progress": 4,
"failed": 2
}
}
}The system logs detailed traces for each LLM call using request_id as a correlation ID. Logs include:
template_id, version, parametersinput, prompt, model, max_tokensinput_tokens, output_tokens, latency_msretry_attempts, error, fallback_modelA typical error is prompt template mismatch: the parameters field is misaligned with the expected keys. The system logs:
{
"error": "TemplateParameterMismatch",
"template_id": "summarize-document",
"expected": ["max_sentences", "tone"],
"received": ["max_sentences", "tone", "word_count"],
"unknown": ["word_count"],
"context": {
"request_id": "req-abc123",
"timestamp": "2024-03-15T12:34:56Z"
}
}Another edge case: model drift. The LLM output changes over time due to updates in the model or prompt. The system captures a baseline and compares new outputs using a simple cosine similarity between embeddings of the first 100 tokens.
Finally, cache warming is not automatic. The system requires a scheduled job (cron or systemd timer) to load common templates into the cache:
0 8 * * * /app/scripts/cache-warmup.shThe script reads cache-warmup.json and populates the cache with pre-rendered prompts.
This architecture transforms the LLM from a black-box API call into a first-class application component—predictable, observable, and resilient.
This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy