Build production-grade RAG systems. Covers chunking strategies, embedding models, hybrid search, reranking, query transformation, evaluation, and advanced patterns for enterprise retrieval-augmented generation.
This reference describes how to design and operate retrieval-augmented generation systems beyond the simple document-store query patterns. It matters when you need to maintain accurate, up-to-date information in generative workflows—especially when documents change frequently or when the context window is limited.
The core of RAG systems is how they fetch relevant information for a query. A basic system retrieves top-k documents from a vector store, but advanced designs must handle multiple dimensions of relevance.
Vector search requires careful parameter tuning. For example, with a standard vector database, you might configure:
CREATE INDEX idx_vector_search ON documents USING ivfflat (embedding vector_l2_ops)
WITH (lists = 100, probes = 10);The lists parameter controls how many clusters to use for search; too few leads to poor recall, too many increases latency. The probes setting defines how many of these clusters to inspect during a search. A probe value of 1 gives fast but inaccurate results; higher values improve accuracy at cost of speed.
Embedding models must be tuned for retrieval quality. If using a model that outputs fixed-size vectors, you must ensure consistent input preprocessing:
# Example embedding pipeline
def preprocess_query(query):
return query.strip().lower().replace('\n', ' ')If the model expects sentences, but queries contain multiple paragraphs, results degrade significantly. Always validate that embeddings are generated with the same pipeline used for document embeddings.
Hybrid retrieval combines dense vector search with sparse keyword matching:
SELECT *,
(1.0 * bm25_score + 0.5 * vector_score) AS hybrid_score
FROM search_results
ORDER BY hybrid_score DESC
LIMIT 10;This requires a system that supports both retrieval types simultaneously. If your vector database does not support BM25 scoring, you must implement this logic in application code.
Document ingestion is where most RAG systems fail. The key issue is maintaining consistency between stored content and the information used for retrieval.
Chunking must balance semantic coherence with search efficiency:
def chunk_document(text, max_tokens=512):
sentences = text.split('. ')
chunks = []
current_chunk = []
token_count = 0
for sentence in sentences:
sentence_tokens = len(sentence.split())
if token_count + sentence_tokens > max_tokens and current_chunk:
chunks.append('. '.join(current_chunk))
current_chunk = [sentence]
token_count = sentence_tokens
else:
current_chunk.append(sentence)
token_count += sentence_tokens
if current_chunk:
chunks.append('. '.join(current_chunk))
return chunksThis approach ensures that each chunk fits within a token limit while preserving sentence boundaries. If chunks are too long, retrieval becomes imprecise. If too short, semantic context is lost.
Metadata fields must be indexed correctly for filtering:
CREATE TABLE documents (
id UUID PRIMARY KEY,
content TEXT,
title VARCHAR(255),
source_url TEXT,
created_at TIMESTAMP,
updated_at TIMESTAMP,
tags TEXT[]
);If a field is not indexed, filtering by it causes full table scans. For example, filtering by tags without an index leads to slow query performance:
SELECT * FROM documents WHERE 'important' = ANY(tags);Updates must maintain both content and embedding consistency:
UPDATE documents SET
content = 'Updated text',
updated_at = NOW(),
embedding = generate_embedding('Updated text')
WHERE id = 'some-id';If updates do not regenerate embeddings, the vector store contains outdated information. This mismatch is silent unless you validate it.
The prompt must control how retrieved information is used. Poor design leads to hallucination or irrelevant context.
Most models have hard limits on input length. You must truncate or summarize context accordingly:
def build_prompt(query, retrieved_docs, max_tokens=2048):
prompt = f"Question: {query}\n\n"
for doc in retrieved_docs:
prompt += f"Context: {doc['content']}\n\n"
# Truncate if needed
token_count = len(prompt.split())
if token_count > max_tokens:
# Remove oldest context items
while token_count > max_tokens and len(retrieved_docs) > 0:
retrieved_docs.pop(0)
prompt = f"Question: {query}\n\n"
for doc in retrieved_docs:
prompt += f"Context: {doc['content']}\n\n"
token_count = len(prompt.split())
return promptThis truncation must be consistent with the model's expected format. If you assume a different separator than the model expects, context becomes malformed.
Prompt templates must validate that all required fields are present:
def validate_prompt_template(template, variables):
required = set(re.findall(r'\{(\w+)\}', template))
provided = set(variables.keys())
missing = required - provided
if missing:
raise ValueError(f"Missing variables in prompt: {missing}")If variables are not provided, the model receives incomplete prompts. This often manifests as empty or meaningless responses.
The retrieval system must integrate with downstream components and provide observability.
You must track query latency and recall:
CREATE TABLE retrieval_metrics (
query_id UUID PRIMARY KEY,
query_text TEXT,
retrieved_count INTEGER,
latency_ms INTEGER,
recall_at_k REAL,
timestamp TIMESTAMP DEFAULT NOW()
);If you do not log these metrics, you cannot detect when retrieval quality degrades.
Retrieval failures must be handled gracefully:
try:
results = vector_search(query_embedding, top_k=10)
except VectorDatabaseError as e:
logger.error(f"Vector search failed: {e}")
# Fall back to keyword search or return empty results
results = keyword_search(query_text, top_k=10)If the system does not handle query failures, downstream models may generate responses based on stale or no context.
Caching improves performance but must avoid serving stale results:
def get_cached_results(query, cache_ttl=3600):
cache_key = f"rag:{hash(query)}"
cached = redis.get(cache_key)
if cached:
return json.loads(cached)
# Perform actual search
results = perform_search(query)
# Cache results
redis.setex(cache_key, cache_ttl, json.dumps(results))
return resultsIf you do not invalidate caches on document updates, users get outdated information.
Several problems silently affect RAG systems without clear symptoms.
If embedding models differ between indexing and querying, results are inconsistent. This manifests as low recall or irrelevant matches:
# Example mismatch
# Indexing uses model A, querying uses model B
# Results appear correct but have poor semantic accuracyThere is no error message; you simply get poor retrieval quality.
Context drift happens when retrieved documents are stale relative to the query:
-- Query for "latest quarterly report"
-- But documents are from last year's reports
SELECT * FROM documents
WHERE title LIKE '%quarterly%'
ORDER BY created_at DESC
LIMIT 10;If your search does not consider document freshness, you get outdated information.
Relevance scores may not align with actual usefulness:
-- If BM25 score is higher than vector score,
-- but the vector-retrieved documents are more relevant
SELECT *, (bm25_score + vector_score) / 2 AS combined_score
FROM results
ORDER BY combined_score DESC;You must validate that your scoring logic matches real-world relevance.
Deployment and maintenance require specific attention to avoid operational pitfalls.
Indices must be refreshed regularly:
# Example cron job for index updates
0 2 * * * /usr/local/bin/update_rag_index --force-refreshIf refreshes are skipped, the system becomes stale. If they fail silently, you have no indication of data inconsistency.
Memory and CPU usage must be monitored:
# Monitor memory usage
watch -n 1 'ps aux | grep rag_service'
# Monitor vector database load
curl -X GET http://vector-db:8080/statsOverloaded systems return incomplete or incorrect results.
If a change breaks retrieval, you must be able to revert:
git log --oneline
# Revert to previous working configuration
git checkout HEAD~1Without a rollback plan, a bad update can halt the entire system.
Debugging RAG systems requires attention to both input and output behavior.
Every query should log its components:
def debug_query(query, results):
logger.info(f"Query: {query}")
logger.info(f"Retrieved {len(results)} documents")
for i, doc in enumerate(results[:3]):
logger.info(f"Doc {i}: {doc['title'][:50]}...")This helps identify when retrieval fails or returns irrelevant content.
Embeddings should be validated:
def validate_embedding(embedding, expected_dim=768):
if len(embedding) != expected_dim:
raise ValueError(f"Invalid embedding dimension: {len(embedding)}")If you do not validate embeddings, the system runs with corrupted data silently.
You must capture user feedback on results:
CREATE TABLE user_feedback (
id UUID PRIMARY KEY,
query_text TEXT,
response_text TEXT,
relevance_score INTEGER,
timestamp TIMESTAMP DEFAULT NOW()
);Feedback helps tune retrieval parameters over time. Without it, you cannot improve performance.
This architecture is about handling complexity in information retrieval and integration, not about model selection or business alignment. The systems described here are concrete, actionable patterns that apply regardless of the specific tools used.
This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy