Design hybrid search systems combining vector similarity, BM25 keyword matching, and cross-encoder reranking for production accuracy.
Semantic search architecture enables systems to retrieve information based on meaning, not just keywords. It matters when users expect natural, context-aware retrieval—when a query like *“What’s the best way to reduce latency in microservices?”* should surface documents discussing distributed tracing, circuit breakers, and service mesh tuning, even if those exact terms don’t appear in the original documents. This architecture is essential for documentation platforms, knowledge bases, AI agent backends, and search-driven user interfaces.
Documents arrive in a variety of formats—PDFs, Markdown, HTML, JSON, plain text—and must be transformed into dense, meaning-rich representations. The ingestion process begins with a preprocessing stage that normalizes input:
pdfminer.six -o /tmp/docs/12345.pdf -t -p 1000 -o /tmp/text/12345.txt
# Clean and split text into chunks
python -c '
import re
from pathlib import Path
def chunk_text(text, chunk_size=512):
words = text.split()
chunks = []
current = []
for word in words:
if len(" ".join(current + [word])) > chunk_size:
chunks.append(" ".join(current))
current = [word]
else:
current.append(word)
if current:
chunks.append(" ".join(current))
return chunks
text = (Path("/tmp/text/12345.txt").read_text(encoding="utf-8"))
chunks = chunk_text(text, chunk_size=512)
for i, chunk in enumerate(chunks):
(Path("/tmp/chunks/12345").mkdir(exist_ok=True) / f"{i:04d}.txt").write_text(chunk)
'Each chunk is then passed through a text embedding model to generate vector representations. The model is typically a fine-tuned Transformer such as all-MiniLM-L6-v2 or BAAI/bge-small-en.
# Generate embeddings using Hugging Face's `sentence-transformers`
curl -X POST http://embedder:8080/infer \
-H "Content-Type: application/json" \
-d '{
"texts": [
"Microservices should be independently deployable and loosely coupled.",
"Circuit breakers prevent cascading failures in distributed systems."
],
"model": "all-MiniLM-L6-v2"
}' \
-o /tmp/embeddings.jsonThe response includes a list of vectors, each of length 384 for all-MiniLM-L6-v2, with each vector stored as a floating-point array.
The vector embeddings are stored in a vector database with efficient similarity search. The database accepts a INSERT operation with a vector, metadata, and an ID.
-- Insert a chunk into a PostgreSQL table with pgvector extension
INSERT INTO documents (id, text, embedding, metadata, source)
VALUES (
'doc-12345-chunk-0012',
'Service mesh enables observability and traffic management across microservices.',
'{0.12, -0.34, 0.55, ...}', -- 384-element vector
'{"source": "docs/microservices.md", "page": 12, "section": "observability"}',
'docs/microservices.md'
);To perform a query, the system computes a query vector and retrieves the nearest neighbors:
-- Find the top 10 most similar documents to a query
SELECT id, text, metadata, 1 - (embedding <=> '{0.45, -0.11, 0.82, ...}') AS score
FROM documents
ORDER BY embedding <=> '{0.45, -0.11, 0.82, ...}'
LIMIT 10;The <= operator performs cosine similarity, and the => operator computes Euclidean distance. The query response includes id, text, metadata, and score fields.
When a user submits a query, the system applies query preprocessing:
def preprocess_query(query: str) -> str:
# Remove extra whitespace and normalize
query = re.sub(r'\s+', ' ', query.strip())
# Apply query expansion using synonyms
synonyms = {
"latency": ["delay", "response time", "latency"],
"microservices": ["services", "distributed services", "service-oriented architecture"],
"reduce": ["minimize", "decrease", "lower"]
}
expanded = []
for word in query.split():
expanded.append(word.lower())
if word.lower() in synonyms:
expanded.extend(synonyms[word.lower()])
return " ".join(set(expanded))The preprocessed query is then embedded:
curl -X POST http://embedder:8080/infer \
-H "Content-Type: application/json" \
-d '{
"texts": ["reduce latency in microservices"],
"model": "all-MiniLM-L6-v2"
}' \
-o /tmp/query_vector.jsonThe resulting vector is used in a similarity search. The system may apply query expansion by re-ranking results with a retrieval-augmented generation (RAG) step:
{
"query": "How to reduce latency in microservices?",
"top_k": 5,
"context": [
"Microservices should be independently deployable and loosely coupled.",
"Circuit breakers prevent cascading failures in distributed systems.",
"Service mesh enables observability and traffic management across microservices."
],
"retrieval_score": 0.87,
"rank": 1
}For better accuracy, systems combine vector search with keyword search. The two are fused using a weighted combination of scores:
def hybrid_search(query_vector: list, keyword_query: str, k: int = 10) -> list[dict]:
# Get vector results
vector_results = vector_db.search(query_vector, k=k)
# Get keyword results using full-text search
keyword_results = postgresql.query(f"""
SELECT id, text, metadata,
ts_rank_cd(to_tsvector('english', text), to_tsquery('english', ?)) AS score
FROM documents
WHERE to_tsvector('english', text) @@ to_tsquery('english', ?)
ORDER BY score DESC
LIMIT {k}
""", (keyword_query, keyword_query))
# Combine results with weighted scores
merged = {}
for result in vector_results:
merged[result["id"]] = {
"id": result["id"],
"text": result["text"],
"metadata": result["metadata"],
"score": result["score"] * 0.7, # vector weight
"type": "vector"
}
for result in keyword_results:
id = result["id"]
if id in merged:
merged[id]["score"] += result["score"] * 0.3
else:
merged[id] = {
"id": id,
"text": result["text"],
"metadata": result["metadata"],
"score": result["score"] * 0.3,
"type": "keyword"
}
return sorted(merged.values(), key=lambda x: x["score"], reverse=True)[:k]To reduce latency, systems use caching at multiple levels. A Redis instance stores recent query embeddings and retrieval results:
# Cache query embedding
redis-cli SETEX query-embedding-latency-in-microservices 300 \
'[0.45, -0.11, 0.82, ...]'
# Retrieve cached results
redis-cli HGETALL cached-retrieval-latency-in-microservicesThe cache key follows the pattern cached-retrieval-{query_hash} where the query is hashed using SHA-256. If a query is not in cache, the system computes it and stores it for future use.
New documents are ingested asynchronously. The system maintains an indexing queue using RabbitMQ or Kafka:
{
"event": "document_added",
"document_id": "doc-12345",
"source_path": "/data/docs/microservices.md",
"timestamp": "2024-04-15T10:30:00Z"
}Workers consume the queue and perform the following steps:
The system handles incremental indexing—it tracks the last processed document and resumes from the last offset, even across restarts. When an error occurs during embedding, the system logs the failure in a error log table:
CREATE TABLE ingestion_errors (
id SERIAL PRIMARY KEY,
document_id VARCHAR(255),
error_type VARCHAR(100),
error_message TEXT,
retry_count INTEGER DEFAULT 0,
created_at TIMESTAMPTZ DEFAULT NOW(),
processed_at TIMESTAMPTZ
);A background job scans this table and retries failed embeddings every 5 minutes.
The most common silent failure occurs when the embedding model returns a vector of incorrect dimensionality. A all-MiniLM-L6-v2 model expects 384-dimensional vectors, but the inference service returns 384.1 or 383. The vector database assumes the correct size, but the query returns incorrect results:
{
"error": "Vector dimension mismatch",
"expected": 384,
"actual": 383,
"query_vector": [0.1, -0.3, 0.5, ...],
"document_id": "doc-12345",
"timestamp": "2024-04-15T11:00:00Z"
}This issue is subtle: results look plausible but are consistently inaccurate. The system must validate vector dimensions on insert and query.
Query expansion using synonyms can introduce noise. A query for *“optimize microservices”* expands to *“optimize microservices, services, distributed services”*, but the system fails to normalize case and stemming. The term *“services”* is not matched to *“service”* in the vector database. The system must apply standardization:
def normalize_term(term: str) -> str:
return stemmer.stem(lemmatizer.lemmatize(term.lower()))Without this, a query for *“service”* fails to retrieve documents with *“services”*.
When a new document is added, the cache for related queries becomes stale. A user searches for *“microservices latency”*, and the cache returns results based on old data. The system must track cache dependencies:
{
"cache_key": "cached-retrieval-microservices-latency",
"dependencies": [
"doc-12345",
"doc-12346",
"doc-12347"
],
"last_updated": "2024-04-15T11:00:00Z"
}When any of the dependent documents are updated, the cache is marked as invalid and refreshed asynchronously.
As the dataset grows, the choice of indexing algorithm becomes critical. The system may use IVF-PQ (Inverted File with Product Quantization) for scalability, but the parameters are sensitive:
nlist: number of clusters (default 100)m: number of subquantizers (default 8)nbits: bits per subquantizer (default 8)A poorly tuned index leads to slow queries and high memory usage. The system logs the query latency and memory usage per index type and adjusts parameters dynamically.
Metadata fields like source, section, and page enable filtering. A query for *“tracing in microservices”* can filter to documents in the observability section:
SELECT id, text, metadata, 1 - (embedding <=> ?) AS score
FROM documents
WHERE metadata->>'section' = 'observability'
ORDER BY embedding <=> ?
LIMIT 10;But the system often applies filtering too late—after retrieving all vectors—leading to unnecessary computation. The optimal approach is to filter early in the retrieval pipeline.
Semantic search architecture is not a single service but a coordinated pipeline across preprocessing, embedding, storage, querying, and optimization. It succeeds when queries return relevant results even when key terms are absent. It fails silently when vector dimensions mismatch, caches go stale, or metadata filtering is applied too late. The system must be vigilant in its validation, error reporting, and configuration of the vector database, embedding models, and query pipelines.
This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy