Embedding Model Selection

Embedding models transform unstructured data—text, images, audio—into dense numerical vectors that capture semantic meaning. Choosing the right model for your use case is critical: a model mismatch can degrade retrieval accuracy, increase latency, and inflate inference costs. This page guides engineers through the practical decisions involved in selecting, evaluating, and deploying embedding models across diverse systems.

When to Select an Embedding Model

Embedding model selection matters when you are building a system that relies on semantic similarity: search, recommendation, clustering, classification, or retrieval-augmented generation (RAG). It is essential during the initial design phase and whenever you observe degradation in downstream accuracy, latency, or cost.

You should perform embedding model selection when:

• You are building a retrieval system (e.g., a question-answering engine) and need to rank documents by relevance.
• You are designing a recommendation pipeline that must understand user intent from free-form queries.
• You are integrating multiple modalities (text, image, video) into a single vector space.
• You are evaluating a new feature that depends on vector similarity (e.g., semantic deduplication, cross-encoder ranking).
• You are migrating from a static embedding pipeline to a dynamic, model-updatable system.

Evaluate Model Performance with a Standard Test Suite

To compare models, use a standard benchmark suite of tasks. The evaluation should be repeatable, automated, and grounded in real data. Use the following workflow:

1. Gather a representative test set: collect 1,000–5,000 query-document pairs from production data, or use a curated dataset like TREC, MS MARCO, or Natural Questions.
2. Generate embeddings for all queries and documents using the candidate models.
3. Compute similarity scores using cosine similarity.
4. Rank documents for each query.
5. Compute metrics such as recall@k, mean reciprocal rank (MRR), and normalized discounted cumulative gain (nDCG).

curl -X POST http://localhost:8080/embed \
  -H "Content-Type: application/json" \
  -d '{
    "text": "What is the capital of France?",
    "model": "all-mpnet-base-v2"
  }' \
  | jq -r '.embedding' > query_embedding.json

# Generate embeddings for all documents in parallel
find documents/ -name "*.txt" | xargs -P 8 -I {} \
  curl -X POST http://localhost:8080/embed \
    -H "Content-Type: application/json" \
    -d '{"text": "$(cat {})", "model": "all-mpnet-base-v2"}' \
    -o "embeddings/$(basename {}).json"

The key failure mode: inconsistent preprocessing. A model trained on tokenized text with SentencePiece may fail on input text that lacks sentence boundaries or includes Unicode normalization. Always verify that input text is preprocessed identically during training and inference.

Choose the Model Architecture Based on Use Case

The architecture of the embedding model determines its trade-off between accuracy, size, and speed.

Use all-mpnet-base-v2 for general-purpose tasks

This model, trained on MS MARCO and natural language inference datasets, excels in retrieval and classification. It is suitable when:

• You have a wide variety of query types (questions, descriptions, summaries).
• You need high accuracy with moderate inference latency.
• You are integrating into a RAG pipeline with a dense retriever.

{
  "model": "all-mpnet-base-v2",
  "dimensions": 768,
  "max_sequence_length": 512,
  "batch_size": 32,
  "temperature": 0.05
}

The model outputs a 768-dimensional vector. Use temperature to adjust the sharpness of similarity scores: lower values make top results more distinct.

Use bge-small for low-latency, edge deployments

When deploying to mobile, embedded, or serverless environments, use bge-small, a lightweight, 128-dimension model trained on a mix of web, academic, and query-document pairs.

# Deploy BGE-small via ONNX runtime
docker run -p 8080:8080 \
  -v ./models/bge-small.onnx:/model.onnx \
  -v ./data:/data \
  ghcr.io/onnxruntime/onnxruntime:latest \
  --model /model.onnx --port 8080

The model is typically quantized to INT8. Verify the quantization via onnxruntime’s GetInputMetadata and GetOutputMetadata calls. A silent failure: model input shape mismatch. The model expects [1, 512], but input is [1, 384] due to a misaligned tokenization step.

Use e5-small-v2 for query-focused retrieval

For systems where queries are short and critical (e.g., search engines), e5-small-v2 is optimized for query encoding. It applies query-specific fine-tuning and uses a dual-encoder architecture.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/e5-small-v2")
query = "How do I deploy a model on Kubernetes?"
query_embedding = model.encode(query)
document = "Kubernetes is an open-source container orchestration platform."
document_embedding = model.encode(document)

# Compute cosine similarity
similarity = (query_embedding @ document_embedding.T).item()

The model expects query and document inputs to be separately encoded, and the output vectors are not normalized by default. For optimal performance, normalize embeddings before querying:

import numpy as np

def normalize(vectors):
    return vectors / np.linalg.norm(vectors, axis=1, keepdims=True)

query_embeds = normalize(model.encode(queries))
doc_embeds = normalize(model.encode(documents))

Failure mode: failing to normalize leads to suboptimal ranking, especially when query and document embeddings are generated by different models.

Optimize for Deployment and Inference

Once you’ve selected a model, optimize for deployment across environments.

Use model versioning and manifest files

Maintain a model.yaml file in the model directory to describe its configuration and metadata.

# models/all-mpnet-base-v2/model.yaml
name: all-mpnet-base-v2
version: 2.0
description: "General-purpose sentence transformer for retrieval and classification"
dimensions: 768
max_sequence_length: 512
training_datasets:
  - ms_marco
  - natural_questions
  - snli
  - multi_nli
  - trec
inference:
  batch_size: 32
  default_temperature: 0.05
  output_format: "json"
  input_schema:
    text: string
    model: string
    options:
      tokenize: true
      add_special_tokens: true
      truncation: true
      padding: "max_length"

Use this file to validate model inputs and outputs during CI/CD.

Serve models with a standardized REST API

Define a consistent endpoint for embedding inference:

POST /embed
Content-Type: application/json

{
  "text": "What is the capital of Germany?",
  "model": "all-mpnet-base-v2",
  "options": {
    "normalize": true,
    "batch": true,
    "temperature": 0.1
  }
}

Return response:

{
  "embedding": [0.12, -0.45, 0.89, ...],
  "model": "all-mpnet-base-v2",
  "input_length": 12,
  "timestamp": "2024-04-15T12:34:56Z"
}

Failures to watch for:

• Missing text field: the server returns 400 Bad Request with {"error": "missing text field"}.
• Invalid model name: returns 404 Not Found with {"error": "model not found", "available": ["all-mpnet-base-v2", "bge-small", "e5-small-v2"]}.
• Large input size: a 10,000-token text fails with 413 Payload Too Large unless the server is configured to accept bodies up to 10 MB.

Use batching and caching to reduce latency and cost

Batch multiple embeddings into a single request:

POST /embed/batch
Content-Type: application/json

[
  {
    "text": "How does RAG work?",
    "model": "bge-small"
  },
  {
    "text": "What is ONNX?",
    "model": "bge-small"
  }
]

Return a list of embeddings, each with metadata.

Cache embeddings at the edge using Redis:

# Cache key: embedding:<model>:<text_hash>
SET embedding:all-mpnet-base-v2:abc123 "0.12,-0.45,0.89,..."
EXPIRE embedding:all-mpnet-base-v2:abc123 86400  # 1 day

Cache misses are common when queries are slightly paraphrased. To improve hit rate, compute text fingerprints using SHA-256 of the input text.

Handle Model Updates and A/B Testing

When rolling out a new embedding model, use a phased rollout with A/B testing.

Deploy using model aliases

Maintain aliases for models in a registry:

{
  "models": {
    "all-mpnet-base-v2": {
      "version": "2.1",
      "active": true,
      "tags": ["prod", "default", "baseline"],
      "created_at": "2024-04-10T08:00:00Z"
    },
    "bge-small-v2": {
      "version": "1.0",
      "active": false,
      "tags": ["testing", "ab-test"],
      "created_at": "2024-04-12T10:00:00Z"
    }
  }
}

Use the alias default to route traffic to the current production model. During A/B testing, route 50% of queries to bge-small-v2 and 50% to all-mpnet-base-v2.

Monitor model performance with logs and metrics

Log every embedding request with structured data:

{
  "timestamp": "2024-04-15T12:34:56Z",
  "request_id": "req-abc123",
  "model": "bge-small-v2",
  "input_length": 14,
  "embedding_duration_ms": 42,
  "cache_hit": true,
  "query": "How do I deploy a model?",
  "embedding": [0.21, -0.33, 0.78, ...]
}

Use these logs to detect:

• High embedding duration: indicates model under-provisioning or slow I/O.
• Low cache hit rate: suggests poor query similarity or short cache TTL.
• Mismatched model versions: a query returns embeddings from bge-small-v2 but the model manifest indicates bge-small-v1.

Common Pitfalls and Sharp Edges

• Model input preprocessing mismatch: the model expects text with sentence-transformers preprocessing (lowercase, strip, tokenize with | as sentence separator), but the input is raw HTML without normalization.
• Missing normalization: embedding vectors are not normalized, leading to inconsistent similarity scores across models.
• Incorrect batch size: batch size is set to 16, but the model was trained with batch size 32, causing suboptimal performance.
• Overfitting to training data: a model trained on academic papers fails on user-generated product reviews due to vocabulary mismatch.
• Inconsistent tokenization: the model uses BPE but the pipeline uses WordPiece, leading to mismatched embeddings.
• Version drift: the model version in model.yaml does not match the actual model file in model.onnx.

Summary

Selecting an embedding model is not a one-time decision. It is a continuous process involving evaluation, deployment, monitoring, and iteration. Use a standard test suite, validate preprocessing, standardize serving APIs, and instrument logging. Choose models based on task, deployment, and performance needs. Prioritize consistency across training and inference, and guard against subtle but costly failures.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.