Build production RAG pipelines. Covers chunking strategies, embedding models, vector stores, retrieval techniques, evaluation, and common failure modes.
RAG combines language models with information retrieval to produce more accurate and contextually relevant responses by fetching relevant documents before generating text. This approach matters when you need factual accuracy, up-to-date information, or domain-specific knowledge without retraining the model.
The retrieval system requires a vector database capable of nearest-neighbor search over embeddings. Use pgvector extension for PostgreSQL or standalone vector stores like Weaviate or Qdrant.
To initialize a vector store with pgvector, run:
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE documents (
id UUID PRIMARY KEY,
content TEXT,
embedding VECTOR(1536)
);For document ingestion, generate embeddings using a pre-trained model such as sentence-transformers/all-MiniLM-L6-v2:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
embedding = model.encode("Your document text here")When indexing, ensure the embedding dimension matches your model output. Mismatched dimensions cause silent failures during similarity search.
Choose an embedding model based on performance vs. cost tradeoffs. For example, all-MiniLM-L6-v2 offers good balance for most use cases. Avoid models that do not support batch processing or lack public APIs for consistent inference.
If your vector store requires fixed-size embeddings, validate that the chosen model outputs vectors of correct dimensionality before deployment. Failure to do so results in failed queries or incorrect similarity scores.
Transform user input into embeddings using the same model used during indexing. The query embedding must match the stored embedding dimensions exactly.
Example processing flow:
Use LIMIT clauses in SQL to constrain search results:
SELECT content FROM documents
ORDER BY embedding <-> $1
LIMIT 5;If you do not limit results, the system may return too many irrelevant documents or timeout on large result sets.
Rank retrieved documents by similarity score. Some systems apply additional filters to remove low-confidence matches.
In PostgreSQL with pgvector, scores are automatically returned:
SELECT content, embedding <-> $1 AS distance
FROM documents
ORDER BY distance ASC
LIMIT 5;Filtering should exclude documents below a threshold score. If you do not filter out poor matches, generated answers will include outdated or incorrect information.
Construct prompts that include retrieved context alongside the original query. The prompt must clearly separate user input from retrieved content.
Example structure:
Context: [Retrieved document content here]
Question: What is the capital of France?
Answer:Use explicit delimiters like --- or === to separate different parts of the prompt. Without these, LLMs may misinterpret which information comes from context versus user query.
Set generation parameters carefully to prevent hallucinations and ensure factual responses. Parameters like temperature, max_tokens, and top_p affect output quality.
Example configuration for a typical LLM API call:
{
"prompt": "Context: ... Question: ... Answer:",
"temperature": 0.2,
"max_tokens": 200,
"top_p": 0.9
}High temperature values increase creativity but can introduce factual errors. Too many tokens may cause truncation or timeout issues.
The most common silent failure occurs when embedding dimensions do not match between indexing and querying phases. This leads to malformed similarity calculations without explicit error messages.
Another frequent issue is missing retrieval results. If no documents are returned for a query, the LLM generates based on general knowledge rather than retrieved context, leading to incorrect answers.
Ensure your system logs retrieval scores and handles empty result sets gracefully. For instance, if fewer than three documents are retrieved, consider returning a default response or flagging for manual review.
Integrate RAG components into existing application workflows. The retrieval step should be synchronous with generation, typically within 100ms to 500ms depending on vector database performance.
Use environment variables to configure embedding model paths and vector store connection strings:
export EMBEDDING_MODEL_PATH="sentence-transformers/all-MiniLM-L6-v2"
export VECTOR_DB_URL="postgresql://user:pass@localhost:5432/rag_db"Track retrieval accuracy using metrics like recall@k or mean reciprocal rank (MRR). Monitor generation latency and identify when slow queries impact user experience.
Log failed embeddings, query failures, and low-score retrievals. These help detect data quality issues in the document corpus.
Deploy both embedding and generation components with sufficient compute resources. Vector databases require memory proportional to stored embeddings. Generation models need GPU or TPU acceleration depending on model size.
Ensure all services can scale independently. For example, if retrieval becomes a bottleneck, add more database replicas or optimize indexing strategies.
Use containerization for consistent deployments:
FROM python:3.10-slim
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "app.py"]Test end-to-end workflows with sample queries and known document sets. Validate that retrieved documents are semantically relevant to queries.
Create test cases for edge cases such as:
Use unit tests to verify embedding generation and similarity search logic independently. Integration tests should validate full RAG pipeline behavior.
Store configuration in version-controlled files rather than hardcoded values. For example, use YAML for embedding model selection:
embedding_model:
name: "sentence-transformers/all-MiniLM-L6-v2"
dimension: 384
vector_store:
type: "pgvector"
connection_string: "postgresql://user:pass@localhost:5432/rag_db"Avoid hardcoding paths or model names in source code. Changes to these values should be managed through configuration updates, not code changes.
Before production deployment, validate:
Test with realistic document sizes and query patterns. Ensure that the system handles large-scale retrieval without timeouts or memory issues.
Maintain clean, well-formatted documents in your vector store. Poorly formatted content leads to suboptimal embeddings and poor retrieval quality.
Regularly audit stored embeddings for consistency. If embedding generation changes over time, reprocess existing documents to maintain alignment.
Avoid storing duplicate or near-duplicate documents in the same index, as they can skew similarity scores and reduce retrieval effectiveness.
Implement caching at multiple levels: document retrieval results and generated responses. Use short TTLs (minutes) for frequently changing content, longer TTLs (hours/days) for stable information.
Example caching approach:
import redis
cache = redis.Redis(host='localhost', port=6379, db=0)
def get_retrieved_documents(query):
cache_key = f"rag:{query}"
cached = cache.get(cache_key)
if cached:
return json.loads(cached)
# Perform actual retrieval
result = perform_vector_search(query)
cache.setex(cache_key, 3600, json.dumps(result)) # Cache for 1 hour
return resultThis approach prevents redundant computations and improves response times under high load.
This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy