Measure retrieval-augmented generation quality with faithfulness, relevance, and answer correctness scoring frameworks.
RAG evaluation metrics are the quantitative and qualitative signals that determine how well a Retrieval-Augmented Generation (RAG) system answers user queries by combining retrieved documents with a language model. They matter when a RAG pipeline is deployed to production, when new retrieval or generation components are added, and when comparing different retrieval strategies, chunking schemes, or embedding models.
To evaluate how effectively a RAG system retrieves relevant documents, compute precision and recall at various k values. For a query, the top-k retrieved documents are compared against a gold standard set of relevant documents.
def compute_precision_recall_at_k(retrieved_docs, relevant_docs, k=5):
retrieved = retrieved_docs[:k]
relevant = set(relevant_docs)
retrieved_set = set(retrieved)
tp = len(retrieved_set & relevant)
precision = tp / k
recall = tp / len(relevant)
return {"precision_at_k": precision, "recall_at_k": recall}Use k = 1, 3, 5, 10 as standard benchmarks. Precision-at-1 is critical: if the top document is not relevant, the model rarely needs to look further. Recall-at-10 captures whether the system finds most relevant documents even when the top-5 are insufficient.
A silent failure mode: ranked retrieval is evaluated as if top-k were a set, but the ordering matters. A query with top-3 documents in the correct order (e.g., most relevant first) receives a higher score than a shuffled list with the same documents. Use Mean Reciprocal Rank (MRR) to account for ordering.
def mrr(retrieved_docs, relevant_docs):
relevant = set(relevant_docs)
for i, doc in enumerate(retrieved_docs, start=1):
if doc in relevant:
return 1 / i
return 0MRR is sensitive to early hits: a relevant document in position 1 contributes 1.0, position 2 contributes 0.5, and so on. MRR is commonly used in information retrieval benchmarks and is ideal when the top result is most important.
Each retrieved document is assigned a relevance score (e.g., cosine similarity to the query). The evaluation process must validate that the scores are well-calibrated. A standard approach is to compute Average Precision (AP) and Mean Average Precision (MAP).
def average_precision(retrieved_docs, relevant_docs):
relevant = set(relevant_docs)
ap = 0
hits = 0
for i, doc in enumerate(retrieved_docs, start=1):
if doc in relevant:
hits += 1
precision_at_i = hits / i
ap += precision_at_i
return ap / len(relevant_docs)
def map(retrieval_results):
return sum(average_precision(results["retrieved"], results["relevant"]) for results in retrieval_results) / len(retrieval_results)AP rewards systems that retrieve relevant documents early and in order. MAP aggregates AP across multiple queries. Use AP and MAP when comparing different retrieval models (e.g., BM25 vs. dense retrieval) or different embedding dimensions.
Common mistake: assuming that a higher cosine similarity score implies better ranking, but failing to validate that scores are calibrated across queries. A model may rank well within a query but have poor cross-query calibration. Use Normalized Discounted Cumulative Gain (nDCG) to assess ranking quality across multiple relevance levels.
def ndcg(retrieved_docs, relevant_docs, relevance_scores):
# relevance_scores maps doc_id -> relevance level (e.g., 0, 1, 2, 3)
def dcg_at_k(retrieved_docs, k, relevance_scores):
dcg = 0
for i, doc in enumerate(retrieved_docs[:k], start=1):
rel = relevance_scores.get(doc, 0)
dcg += rel / (1 + math.log2(i))
return dcg
def idcg_at_k(retrieved_docs, k, relevance_scores):
# ideal ranking: sort by relevance descending
ideal_order = sorted(retrieved_docs, key=lambda d: relevance_scores.get(d, 0), reverse=True)
return dcg_at_k(ideal_order, k, relevance_scores)
k = min(10, len(retrieved_docs))
return dcg_at_k(retrieved_docs, k, relevance_scores) / idcg_at_k(retrieved_docs, k, relevance_scores)nDCG is especially useful when documents have graded relevance (e.g., 0 to 3 stars), and when the system is expected to return a ranked list where top results matter more.
After retrieval, the generator produces a natural language answer. Evaluate answer quality using Exact Match (EM) and F1 score.
EM checks whether the predicted answer matches the ground-truth answer exactly (token-by-token). F1 computes the harmonic mean of precision and recall between the predicted and expected answers.
def exact_match(prediction, truth):
return prediction.strip().lower() == truth.strip().lower()
def f1_score(prediction, truth):
pred_tokens = prediction.strip().split()
truth_tokens = truth.strip().split()
common = set(pred_tokens) & set(truth_tokens)
if len(common) == 0:
return 0
precision = len(common) / len(pred_tokens)
recall = len(common) / len(truth_tokens)
return 2 * (precision * recall) / (precision + recall)EM is strict: "The capital of France is Paris" fails if the model says "Paris is the capital of France". F1 is more forgiving and captures partial matches. Use both: EM for strict QA and F1 for robust assessment.
Sharp edge: F1 depends heavily on tokenization. A model using BPE may split "machine learning" differently than the gold standard. Apply tokenization-aware F1 by aligning tokens using the same tokenizer used during training.
A RAG system should not only generate correct answers but also ground those answers in retrieved context. Use Answer Relevance and Answer Consistency metrics.
Answer Relevance evaluates whether the generated answer is semantically aligned with the retrieved documents. Compute BERTScore using a pre-trained BERT model (e.g., bert-base-uncased) to measure similarity between the generated answer and each retrieved document.
python -c "
from transformers import AutoTokenizer, AutoModel
import torch
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModel.from_pretrained('bert-base-uncased')
def bert_score(answer, documents):
inputs = tokenizer(answer, documents, return_tensors='pt', padding=True, truncation=True)
with torch.no_grad():
outputs = model(**inputs)
# Use [CLS] token output for sentence pairs
cls_embeddings = outputs.last_hidden_state[:, 0, :]
# Compute cosine similarity between answer and each document
similarity = torch.cosine_similarity(cls_embeddings[0:1], cls_embeddings[1:], dim=1)
return similarity.mean().item()
"A high BERTScore indicates that the answer is semantically supported by the retrieved documents. Use this to detect hallucination: an answer that is fluent but not grounded in the retrieved context.
To measure whether the model used the correct documents, compute Contextual Accuracy (CA).
For each generated answer, extract answer spans (e.g., using a span-based NER or a simple regex). Then, for each span, determine which retrieved documents support it.
def contextual_accuracy(answer, retrieved_docs, spans):
# spans: list of (start, end, text) tuples
# For each span, find the best-matching retrieved document
# using overlap or embedding similarity
total = 0
correct = 0
for start, end, text in spans:
# Find document that covers this span best
best_doc = None
best_score = 0
for doc in retrieved_docs:
overlap = len(set(text.lower().split()) & set(doc.lower().split()))
score = overlap / len(text.split())
if score > best_score:
best_score = score
best_doc = doc
# If the best document contains the span (or is relevant), count as correct
if best_score >= 0.5:
correct += 1
total += 1
return correct / totalCA is sensitive to over-retrieval: a model may use a retrieved document that is only tangentially related. A CA score below 0.7 suggests that the model is not effectively grounding its answers.
Silent failure: the retrieval step returns 10 documents, but the generator only uses the top 3. Use Retrieved Document Usage (RDU) to track which documents are actually used.
def rdu(retrieved_docs, answer, spans):
# Map each span to the document that supports it
# Compute percentage of retrieved documents used
used_docs = set()
for start, end, text in spans:
best_doc = None
best_score = 0
for doc in retrieved_docs:
overlap = len(set(text.lower().split()) & set(doc.lower().split()))
score = overlap / len(text.split())
if score > best_score:
best_score = score
best_doc = doc
used_docs.add(best_doc)
return len(used_docs) / len(retrieved_docs)An RDU of 0.6 means that only 60% of retrieved documents are effectively used. Low RDU signals that retrieval and generation are decoupled.
Combine retrieval and generation into a single evaluation framework.
RAG Accuracy computes the fraction of queries for which the generated answer is correct (EM or F1) and the top-k retrieved documents contain at least one relevant document.
def rag_accuracy(retrieval_results, generation_results, threshold=0.5):
# retrieval_results: list of (retrieved_docs, relevant_docs)
# generation_results: list of (prediction, truth)
total = 0
correct = 0
for (retrieved_docs, relevant_docs), (prediction, truth) in zip(retrieval_results, generation_results):
# Check answer correctness
em = exact_match(prediction, truth)
# Check retrieval coverage
retrieved_set = set(retrieved_docs)
relevant_set = set(relevant_docs)
coverage = len(retrieved_set & relevant_set) / len(relevant_set)
if em and coverage >= threshold:
correct += 1
total += 1
return correct / totalRAG Accuracy captures the holistic performance: both generation and retrieval must succeed.
RAG Recall at k extends retrieval recall to the end-to-end pipeline: the model must generate an answer that includes a relevant document. This metric evaluates whether the retrieval step is sufficient for the generator.
When a RAG system fails, diagnose using these signals:
Use evaluation logs that record:
These logs enable post-hoc analysis and help identify patterns in failure across queries.
Evaluate RAG systems using a standard test set of queries and ground-truth answers. Store the evaluation results in a structured format:
[
{
"query": "What is the capital of France?",
"retrieved_docs": [
"Paris is the capital of France.",
"France is located in Western Europe.",
"The Eiffel Tower is in Paris."
],
"relevant_docs": [0, 2],
"generated_answer": "Paris is the capital of France.",
"truth_answer": "Paris",
"em": true,
"f1": 0.75,
"bert_score": 0.82,
"contextual_accuracy": 0.67,
"retrieved_document_usage": 0.75,
"mrr": 0.8,
"ndcg": 0.78
}
]Use jq, pandas, and json to process and analyze results. Run evaluations nightly using a CI pipeline:
# .github/workflows/rag-evaluation.yml
name: RAG Evaluation
on: schedule:
- cron: '0 3 * * *' # Every day at 3 AM UTC
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- run: pip install -r requirements.txt
- run: python evaluate_rag.py --test-set data/test.json --output results.json
- run: |
echo "Evaluating RAG metrics..."
python -c "
import json
import pandas as pd
with open('results.json') as f:
results = json.load(f)
df = pd.DataFrame(results)
print(df[['em', 'f1', 'mrr', 'ndcg', 'contextual_accuracy']].mean())
"These metrics form a foundation for tuning, monitoring, and maintaining RAG systems in production.
This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy