RAG Architecture Patterns

RAG (Retrieval-Augmented Generation) architecture patterns solve the problem of enabling large language models to answer questions or generate content using external knowledge beyond their training data. This is essential when the model must synthesize answers from structured databases, unstructured documents, or real-time data sources. RAG patterns matter when the knowledge base is large, dynamic, or frequently updated, and when answers must be grounded in specific evidence from the source material.

Query-Time Retrieval Strategies

At query time, the model must retrieve relevant context before generating a response. The core decision is *how* and *when* to retrieve documents.

Vector Search with Approximate Nearest Neighbors

Use vector embeddings to retrieve the most semantically similar documents to a query. Embeddings are typically generated using a model like sentence-transformers/all-MiniLM-L6-v2 or cohere-embed-v3.

To retrieve documents, send a query vector to a vector database via HTTP. The database must support approximate nearest neighbor (ANN) search.

POST /v1/search
Content-Type: application/json

{
  "query_vector": [0.12, -0.34, 0.56, ...],
  "k": 5,
  "filter": {
    "source": "user_manuals",
    "updated_after": "2024-01-01T00:00:00Z"
  }
}

The vector database returns a list of document IDs and scores. Each document has a content field and metadata like title, source, url, and version.

A common failure mode is dimension mismatch: the query embedding is in a 384-dimensional space, but the stored vectors are 768-dimensional. The system silently returns low-scoring results with no error. The fix is to ensure both the embedding model and the vector index use the same dimensionality and normalization.

Hybrid Retrieval: Combining Vector and Keyword Search

For better recall, combine vector similarity with keyword matching. Use a two-stage retrieval:

1. Run a keyword search (e.g., BM25) over the full text.
2. Score each result using vector similarity.
3. Combine scores using a weighted sum: score = α × vector_score + (1−α) × keyword_score.

The optimal α is not always 0.5. In production, α = 0.7 often performs better due to long-tail queries with complex semantics.

{
  "query": "How do I configure the load balancer for high availability?",
  "hybrid": {
    "vector": {
      "model": "all-MiniLM-L6-v2",
      "k": 10
    },
    "keyword": {
      "field": "content",
      "boost": 1.5,
      "max_expansions": 3
    },
    "combine": {
      "method": "weighted_sum",
      "weights": {
        "vector": 0.7,
        "keyword": 0.3
      }
    }
  }
}

A subtle failure occurs when the keyword results are filtered *after* the vector search: the top-10 vector matches are scored, but then keyword search is applied only to those 10 documents. This loses precision when the top-10 vectors are from different domains.

Pre-Indexing and Document Processing

Before query time, documents are processed and indexed. The pipeline determines how well the retrieval system performs.

Chunking Strategies

Documents are split into chunks to balance retrieval precision and memory usage. The most common approach is sliding window with overlap.

For a document of 1000 tokens, use chunks of 512 tokens with 128-token overlap:

def chunk_document(text: str, chunk_size: int = 512, overlap: int = 128) -> list[dict]:
    chunks = []
    start = 0
    while start < len(text):
        end = min(start + chunk_size, len(text))
        chunk = {
            "id": f"{document_id}:{start}:{end}",
            "content": text[start:end],
            "metadata": {
                "source": document_id,
                "start_token": start,
                "end_token": end,
                "chunk_index": len(chunks)
            }
        }
        chunks.append(chunk)
        start += chunk_size - overlap
    return chunks

The overlap parameter is frequently misunderstood: it is not just "overlap between chunks" but the number of tokens carried from one chunk to the next. A common mistake is setting overlap to 256 but only copying 128 tokens, leading to gaps.

Metadata Enrichment and Filtering

Each chunk should carry metadata to enable filtering and ranking. Metadata fields include:

• source: the original document or source system (e.g., wiki, pdf, db:users)
• created_at, updated_at: timestamps
• version: version number or revision ID
• tags: list of keywords like api, tutorial, faq

This metadata enables filtering during retrieval, such as retrieving only "recent" or "high-priority" documents.

GET /v1/search?source=wiki&created_after=2024-01-01

A silent failure occurs when metadata is not indexed in the vector database. The system can filter on source but not on version or tags, even though the data is present. The solution is to index metadata as part of the vector record.

Generation with Retrieved Context

Once documents are retrieved, the LLM generates a response using the context.

Prompt Engineering with Context Formatting

The prompt must clearly separate the query from the retrieved documents. Use a standard format:

You are a helpful assistant. Answer the following question using the provided context.

Context:
1. [Title] - Source: [source], Updated: [updated_at]
   [content]

2. [Title] - Source: [source], Updated: [updated_at]
   [content]

Question: [query]

Answer:

The model must be trained to attend to the context. A common mistake is including the context in the prompt but not prepending a "Given the following information" instruction.

Context Re-Ranking and Filtering

After initial retrieval, filter and re-rank the results to improve quality.

Use a re-ranker model such as cross-encoder/ms-marco-MiniLM-L-6-v2 to score each retrieved document pair (query, document).

POST /v1/rerank
Content-Type: application/json

{
  "query": "How to set up CI/CD for a microservice?",
  "documents": [
    {
      "id": "doc-123",
      "title": "CI/CD Best Practices",
      "content": "Use GitHub Actions for simple repos..."
    },
    {
      "id": "doc-456",
      "title": "Advanced Jenkins Configurations",
      "content": "Set up multibranch pipelines..."
    }
  ]
}

The re-ranker returns a list of document_ids in descending order of relevance. This list becomes the final context.

A critical failure mode: the re-ranker is applied to the top-10 results, but the model is only fed the top-5. The top-6 to 10 documents are ignored, even though they contain valuable information.

Dynamic Context Selection

Instead of using all retrieved documents, select only the most relevant ones based on a threshold or a fixed number.

{
  "query": "What is the difference between a pod and a service in Kubernetes?",
  "retrieval": {
    "k": 10,
    "min_score": 0.4,
    "max_context_tokens": 2048
  }
}

The system retrieves 10 documents, then filters by min_score and truncates the total context to max_context_tokens. A common error is setting max_context_tokens to 2048 but not accounting for the prompt overhead (e.g., 300 tokens for the prompt and system message).

Caching and Performance Optimization

RAG systems often serve thousands of queries per second. Caching is essential.

Query Caching with TTL

Cache the results of a query using a key derived from the query text and metadata filters.

Use Redis or a key-value store:

SETEX "rag:query:How to deploy a service in Kubernetes?source=wiki" 300 \
  '{"results": [...], "retrieved_at": "2024-04-15T10:00:00Z"}'

The key format is critical: rag:query:{query_string}:{filter_key}:{filter_value}. A common error is using only query_string as the key, causing cache misses for queries with the same text but different filters.

Document Cache with Versioning

Cache document chunks to avoid re-embedding and re-indexing.

Use a document versioning system where each document has a version field. When a document is updated, its version increases.

When retrieving, check the version of each chunk:

GET /v1/chunk?doc_id=api-guide-v3&version=2

Return the chunk if the version matches; otherwise, return a 304 Not Modified response. This enables efficient delta updates.

A failure mode: the version number is stored as a string, but the system compares v2 and 2 as different values. Use consistent versioning: 1.2, v3, 2024-01-01, etc., and normalize them on ingestion.

Failure Modes and Debugging

RAG systems silently fail in production. Recognizing the symptoms helps diagnose issues.

Low Recall: Missing Relevant Documents

Symptoms:

• Answers are correct but lack detail.
• Users report "I saw that in the docs, but the model didn’t mention it."

Root causes:

• Vector dimension mismatch (e.g., 384 vs 768).
• Low k in retrieval (e.g., k=3 when k=10 is needed).
• Poor chunking: chunks are too small or overlap is too low.

Debugging:

• Log the top-5 retrieved documents and their scores.
• Compare the query embedding with the stored embeddings using cosine similarity.
• Use a test query with known relevant documents and measure precision at k.

High Latency: Slow Response Times

Symptoms:

• Average response time > 500ms, with spikes beyond 2s.

Root causes:

• Large context size: too many tokens in the prompt.
• Re-ranker not cached.
• Vector search not optimized (e.g., no indexing for the ANN algorithm).

Debugging:

• Profile the full pipeline: time spent in retrieval, re-ranking, embedding, and generation.
• Measure the time per chunk during chunking.
• Add a X-Trace-ID header to correlate logs across services.

Inconsistent Answers: Model Chooses Different Contexts

Symptoms:

• Same query returns different answers on repeated calls.
• The model references different documents in different runs.

Root causes:

• Unstable retrieval: no consistent query cache.
• Re-ranker model not synchronized across deployments.
• Metadata not indexed: filters are ignored.

Debugging:

• Log the full retrieval pipeline: query vector, filter parameters, retrieved document IDs, and final context.
• Use query_id to track a single request across services.

Suggested Configuration

A minimal, production-ready RAG configuration:

{
  "retrieval": {
    "strategy": "hybrid",
    "vector": {
      "model": "all-MiniLM-L6-v2",
      "k": 10,
      "dimension": 384,
      "normalize": true
    },
    "keyword": {
      "field": "content",
      "boost": 1.3
    },
    "combine": {
      "method": "weighted_sum",
      "weights": {
        "vector": 0.6,
        "keyword": 0.4
      }
    },
    "filter": {
      "max_age_days": 90,
      "sources": ["wiki", "docs", "faq"]
    },
    "reranker": {
      "model": "cross-encoder/ms-marco-MiniLM-L-6-v2",
      "k": 5,
      "cache": true
    },
    "context": {
      "max_tokens": 2048,
      "format": "list_with_metadata",
      "include_snippets": true
    },
    "caching": {
      "query": {
        "ttl_seconds": 300,
        "keys": ["query", "source", "created_after"]
      },
      "document": {
        "ttl_seconds": 86400,
        "version_based": true
      }
    }
  }
}

This configuration balances accuracy, latency, and maintainability. It handles dynamic knowledge and provides consistent, high-quality answers.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.