Design Retrieval-Augmented Generation systems that actually work in production. Covers chunking strategies, embedding models, hybrid search, reranking, evaluation metrics, and the failure modes that textbook RAG implementations ignore.
RAG architecture patterns solve the problem of retrieving relevant context from large knowledge bases to improve AI-generated responses. When vector search alone fails—because of noisy embeddings, ambiguous queries, or inconsistent document chunking—engineers must layer additional logic and structure. This page details the patterns that elevate RAG beyond basic retrieval, focusing on the actions engineers take to make retrieval robust, adaptive, and predictable.
When a user asks, *“What are the key changes in the Kubernetes 1.30 release?”*, a naive RAG system may query a vector index built from Kubernetes release notes, but the results vary wildly depending on how well the query matches the embedding space. The first failure mode is mismatched semantic scope: a query like *“Kubernetes 1.30 changes”* does not align with embeddings trained on full release notes, where the term *“changes”* appears in sub-sections titled *“Enhancements,”* *“Bug Fixes,”* and *“Deprecations.”*
To address this, engineers apply query refinement. The process begins with a preprocessor that normalizes and enriches the input query.
def refine_query(raw_query: str) -> str:
# Lowercase, remove extra whitespace, expand abbreviations
query = raw_query.strip().lower()
query = re.sub(r'\s+', ' ', query)
# Expand common abbreviations
abbreviations = {
'k8s': 'kubernetes',
'ingress': 'ingress controller',
'etcd': 'etcd database',
'cni': 'container network interface'
}
for abbr, full in abbreviations.items():
query = re.sub(rf'\b{abbr}\b', full, query)
# Add domain-specific terms to improve embedding alignment
domain_terms = ['release notes', 'feature highlights', 'deprecations', 'performance improvements']
for term in domain_terms:
if term not in query:
query += f' {term}'
return queryThis preprocessing reduces query drift: a query vector in the embedding space is more likely to match documents that were written with the same terminology. A query *“k8s 1.30 changes”* becomes *“kubernetes 1.30 release notes feature highlights deprecations performance improvements”*—a much better match.
Failure mode: when the preprocessor is applied but the vector index was trained on raw text without normalization. The embedding model sees *“k8s”* and *“Kubernetes”* as different terms, so the refined query fails to align with the original documents.
Vector search alone returns top-k nearest neighbors, but it often includes relevant but off-topic results. A query for *“how to scale a Kubernetes deployment”* might retrieve a document about *“pod lifecycle”* because both are related to Kubernetes, but the scaling guide is buried in the top-10 results.
To correct this, engineers implement multi-stage retrieval: first, a fast vector search over a large corpus, then a re-ranking phase using a more accurate but slower model.
curl -X POST http://vector-db:8080/search \
-H "Content-Type: application/json" \
-d '{
"query": "how to scale a kubernetes deployment",
"k": 50,
"filter": { "source": "kubernetes-docs" }
}' \
-o candidates.jsonThe output is a list of document IDs and scores from the vector model. These are then passed to a re-ranker:
def re_rank(candidates: list[dict], query: str) -> list[dict]:
# Load a cross-encoder model
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
# Prepare pairs: (query, document_text)
pairs = [
(query, doc["text"])
for doc in candidates
]
# Compute scores
scores = model.predict(pairs)
# Sort by re-ranker score
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
# Return top 10, enriched with re-ranker score
return [
{
"id": doc["id"],
"text": doc["text"],
"vector_score": doc["score"],
"re_rank_score": score,
"metadata": doc["metadata"]
}
for doc, score in ranked[:10]
]The final result includes a new re_rank_score field, which is used to decide which documents to pass to the LLM.
Critical failure mode: using a re-ranker model that was trained on a different domain than the target corpus. A model trained on MS MARCO fails to re-rank Kubernetes documentation because it expects general web queries, not technical documentation with fixed terminology.
A vector index is only as good as its document chunks. Too small, and context is lost; too large, and embeddings are noisy. A common failure is static chunking—splitting documents into 512-token chunks regardless of content.
Engineers improve this by applying dynamic chunking based on document structure.
def chunk_document(doc: dict) -> list[dict]:
# Extract sections from markdown or HTML
sections = parse_sections(doc["content"])
# Group by section type
chunks = []
current_chunk = {"text": "", "metadata": doc["metadata"].copy()}
for section in sections:
# Start new chunk if section is a heading of level 1 or 2
if section["level"] in [1, 2]:
if current_chunk["text"]:
chunks.append(current_chunk)
current_chunk = {
"text": section["text"],
"metadata": {**doc["metadata"], "section": section["title"]}
}
else:
current_chunk["text"] += "\n" + section["text"]
# Flush chunk if it exceeds 512 tokens
if len(current_chunk["text"].split()) > 512:
chunks.append(current_chunk)
current_chunk = {"text": section["text"], "metadata": doc["metadata"].copy()}
# Add final chunk
if current_chunk["text"]:
chunks.append(current_chunk)
return chunksThis produces chunks that align with logical units: each major section becomes its own chunk. A query about *“horizontal pod autoscaling”* now matches a chunk that contains only the relevant section, rather than a long document where the concept appears in the middle.
Failure mode: when the chunking logic is applied at indexing time, but the query pipeline assumes fixed-size chunks. The re-ranker or LLM receives inconsistent metadata: some chunks have section keys, others do not. The metadata is not normalized, leading to mismatched joins.
In systems with limited vocabulary, a query like *“how to auto-scale pods”* may miss documents that use *“autoscaling”*, *“scale pods automatically”*, or *“HPA configuration”*. To bridge this gap, engineers use query expansion.
The process starts with a static synonym map and expands the query using a lightweight model.
{
"synonyms": {
"scale": ["scale", "scale up", "scale out", "auto-scale"],
"pod": ["pod", "container", "k8s pod"],
"autoscaling": ["autoscaling", "auto-scale", "auto-scaling"],
"hpa": ["hpa", "horizontal pod autoscaler", "horizontal pod autoscaler (hpa)"]
}
}The query expansion function combines the original query with all synonyms and applies a weighted fusion:
def expand_query(query: str, synonyms: dict) -> str:
terms = query.split()
expanded = set(terms)
for term in terms:
if term in synonyms:
expanded.update(synonyms[term])
# Reconstruct query with higher weight on original terms
return " ".join([
f"{term}^1.5" if term in terms else term
for term in expanded
])The result is a query like *“scale^1.5 pod^1.5 autoscaling^1.5 auto-scale^1.5 hpa^1.5 horizontal pod autoscaler^1.5”*.
This query is then used in the vector search. The expansion step is often applied before the vector search, but can also be interleaved with re-ranking.
Failure mode: when the synonym map is not updated in sync with the index. A document about *“HPA”* is indexed with *“HPA”*, but the query expansion uses *“horizontal pod autoscaler”* only. The query *“hpa”* is expanded to *“horizontal pod autoscaler”*, but no document contains *“hpa”*—so the vector search misses the match entirely.
Vector search alone struggles with queries that mix semantic and keyword-based information. A query like *“how to configure HPA for stateful workloads”* requires both semantic similarity (e.g., *“pod autoscaling”*) and exact keyword matching (e.g., *“stateful”*, *“workload”*).
Hybrid search combines vector search with keyword search using a weighted fusion.
# Perform keyword search using Elasticsearch
curl -X POST http://es:9200/kubernetes-docs/_search \
-H "Content-Type: application/json" \
-d '{
"query": {
"multi_match": {
"query": "how to configure hpa for stateful workloads",
"fields": ["title^2", "content^1"],
"type": "best_fields"
}
},
"size": 50
}' \
-o keyword_results.jsonThe keyword results are scored using BM25. The vector results from the ANN search are scored using cosine similarity. These two sets of scores are combined:
def hybrid_search(vector_results, keyword_results, alpha=0.7):
# Normalize scores to [0,1]
vector_scores = normalize_scores([r["score"] for r in vector_results])
keyword_scores = normalize_scores([r["score"] for r in keyword_results])
# Combine using weighted fusion
combined = []
for v, k in zip(vector_results, keyword_results):
combined.append({
"id": v["id"],
"text": v["text"],
"vector_score": v["score"],
"keyword_score": k["score"],
"final_score": alpha * v["score"] + (1 - alpha) * k["score"]
})
# Sort by final score
return sorted(combined, key=lambda x: x["final_score"], reverse=True)The fusion parameter alpha controls the balance between semantic and keyword relevance. A high alpha (e.g., 0.8) emphasizes vector search, while a low alpha (e.g., 0.4) prioritizes keyword matching.
Failure mode: when the normalization is applied independently to each score, but the final score is not scaled properly. A vector score of 0.9 and a keyword score of 1.5 are added directly, but the keyword scores are in a different range. The result is a hybrid score where the keyword component dominates even when it should be secondary.
RAG systems often retrieve documents from multiple sources, versions, or domains. A query about *“Kubernetes service load balancing”* may pull results from *“Kubernetes User Guide,”* *“Cilium Documentation,”* and *“NGINX Ingress Controller Docs.”* The LLM needs to know which document to trust more.
To enable this, engineers enrich the retrieval pipeline with metadata filtering and context-aware selection.
{
"document_metadata": {
"source": "kubernetes-user-guide",
"version": "1.30",
"category": "configuration",
"author": "kubernetes team",
"last_updated": "2024-04-15",
"tags": ["networking", "load balancing", "service"]
}
}The retrieval pipeline applies filters based on the query and query context:
def filter_and_rank(candidates: list, query: str, context: dict) -> list:
# Filter by source and version
filtered = [
c for c in candidates
if c["metadata"]["source"] in context.get("preferred_sources", ["kubernetes-docs"])
and c["metadata"]["version"] >= context.get("min_version", "1.25")
]
# Weight results based on metadata match
weighted = []
for candidate in filtered:
score = candidate["final_score"]
# Boost if source matches context
if candidate["metadata"]["source"] == context.get("target_source"):
score *= 1.5
# Boost if tags match query
query_words = set(query.lower().split())
candidate_tags = set(candidate["metadata"]["tags"])
common_tags = query_words & candidate_tags
if common_tags:
score *= (1 + len(common_tags) * 0.2)
weighted.append({
"candidate": candidate,
"score": score
})
return sorted(weighted, key=lambda x: x["score"], reverse=True)This allows the system to prefer documents that are not only semantically relevant but also contextually aligned: a query about *“Kubernetes 1.30 load balancing”* favors documents from the 1.30 release and those tagged with *“load balancing.”*
Failure mode: when metadata is not indexed for filtering. A query filter on version fails because the version field is stored but not indexed in the vector database. The system must scan all documents, defeating the purpose of fast retrieval.
Even with all these patterns, retrieval quality degrades over time. Queries change, documents grow, and user expectations shift. Engineers implement feedback loops to adapt the retrieval pipeline.
Each query and its retrieved results are logged with user feedback:
{
"query": "how to configure hpa for stateful workloads",
"retrieved": [
{
"id": "k8s-1234",
"title": "HPA Configuration Guide",
"score": 0.87,
"selected": true,
"feedback": "user liked this result"
},
{
"id": "k8s-5678",
"title": "StatefulSets and HPA",
"score": 0.62,
"selected": false,
"feedback": "too general, missing HPA details"
}
],
"timestamp": "2024-04-15T12:34:56Z"
}This feedback is used to retrain the re-ranker model, adjust chunking logic, and update the synonym map. The system learns that *“stateful workloads”* is often paired with *“StatefulSets”*, and that users expect HPA configuration details when asking about HPA.
The pipeline is monitored with a set of health checks:
# Check if retrieval is within expected performance
curl -f http://metrics:8080/api/v1/retrieval/health \
-H "Accept: application/json" \
-d '{
"expected_latency": 200,
"min_recall_at_10": 0.8,
"max_mrr_at_5": 0.75
}'If any metric fails, the system triggers alerts and can roll back to a previous version of the retrieval pipeline.
Failure mode: when feedback is collected but not used. The system logs queries and selections, but the model is trained once a month, and changes are not reflected in the retrieval pipeline for weeks. The feedback loop is effectively a black box.
These patterns—query refinement, multi-stage retrieval, dynamic chunking, query expansion, hybrid search, metadata filtering, and adaptive feedback—form the foundation of a robust RAG architecture. Together, they transform vector search from a simple retrieval method into a dynamic, context-aware, and self-improving system.
This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy