LLM Evaluation and Benchmarking

Systematically evaluate large language model performance for your use case. Covers evaluation frameworks, hallucination detection, human evaluation, automated metrics, A/B testing LLMs, and the patterns that prevent shipping LLM features that look impressive in demos but fail in production.

LLM Evaluation and Benchmarking is the systematic process of measuring a language model’s performance across a range of tasks, using standardized datasets and metrics, to guide model selection, hyperparameter tuning, and deployment decisions. It matters when you need to compare models, validate training pipelines, or justify changes to a production system—especially when model behavior is not obvious from inspection alone.

Defining the Evaluation Scope

Before measuring, you must define *what* to evaluate. A benchmark is not just a dataset; it is a defined set of tasks, input formats, expected outputs, and scoring rules. The most common structure uses a JSONL file where each line is a task instance. For example, a question-answering benchmark might have:

{"id": "q1", "context": "Paris is the capital of France...", "question": "What is the capital of France?", "answer": "Paris"}
{"id": "q2", "context": "The Eiffel Tower was built in 1889...", "question": "When was the Eiffel Tower built?", "answer": "1889"}

Each evaluation task must include a prompt field (string or list of strings) and a expected output (string or list of strings). The evaluation pipeline reads this and compares generated outputs against expected ones.

The evaluation scope is defined in a YAML configuration file, typically eval-config.yaml:

dataset:
  path: "data/squad-v2.jsonl"
  format: "jsonl"
  task: "question-answering"
  input_fields:
    - "question"
    - "context"
  output_field: "answer"
  id_field: "id"

metrics:
  - name: "exact-match"
    type: "exact_match"
    threshold: 0.5
  - name: "f1-score"
    type: "f1"
    normalize: true
  - name: "perplexity"
    type: "perplexity"
    context_field: "context"
    target_field: "answer"

model:
  name: "llama-3-8b"
  prompt_template: "question-context: {{question}} {{context}}"
  max_tokens: 256
  temperature: 0.7

This configuration is used across all evaluation runs. The prompt_template uses Jinja2-style syntax, but the evaluation engine does not require Jinja—only a string with placeholders in double curly braces.

Running Evaluations

The evaluation engine executes a model against a dataset and computes metrics. It requires a model endpoint, either local or remote.

For a local model served via text-generation-inference (TGI), the evaluation pipeline uses curl to send requests:

curl -X POST http://localhost:8080/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Question: What is the capital of France? Context: Paris is the capital of France.",
    "max_tokens": 256,
    "temperature": 0.7
  }' \
  -o output.json

The response is a JSON object:

{
  "generated_text": "Paris",
  "prompt_tokens": 12,
  "generated_tokens": 1,
  "latency_ms": 432
}

The evaluation script iterates over each dataset entry, constructs the prompt from input fields, sends the request, parses the response, and compares against expected.

For batch evaluation, the script writes a results.jsonl file:

{"id": "q1", "prediction": "Paris", "latency_ms": 432, "tokens_generated": 1}
{"id": "q2", "prediction": "1889", "latency_ms": 510, "tokens_generated": 1}

This file is then passed to a metric computation module.

Computing Metrics

Metrics are computed independently of the model inference. The evaluation pipeline uses a standard metric library, typically evaluate (Hugging Face’s evaluate library), which supports exact match, F1, BLEU, ROUGE, and custom metrics.

To compute exact match and F1 scores:

from evaluate import load

em_metric = load("exact_match")
f1_metric = load("f1")

predictions = ["Paris", "1889"]
references = ["Paris", "1889"]

results = {
    "exact_match": em_metric.compute(predictions=predictions, references=references),
    "f1": f1_metric.compute(predictions=predictions, references=references),
}

The result is a dictionary with exact_match and f1 as floats.

For metrics that require a per-sample prediction and reference, the evaluation engine expects predictions and references as lists of strings, or as a list of dictionaries with prediction and reference keys.

A common failure mode: the expected field in the dataset is an array, but the model outputs a string. The evaluation pipeline silently treats the string as a list with one element, leading to mismatched metric values. To fix this, the configuration includes a post_process step:

post_process:
  - name: "split-answers"
    type: "split"
    field: "answer"
    separator: ";"
    expected: true

This splits the answer field on semicolons before comparison. Without it, a dataset with answer: "New York; Boston" and model output New York yields exact_match: 0.5 instead of 1.0.

Benchmarking Across Models and Configurations

To compare models, you run the same evaluation pipeline with different model configurations. Each run is a “benchmark” and produces a report.

A benchmark is defined by a directory structure:

benchmarks/
├── llama-3-8b/
│   ├── config.yaml
│   ├── results.jsonl
│   └── report.json
├── phi-3-mini/
│   ├── config.yaml
│   ├── results.jsonl
│   └── report.json
└── mixtral-8x7b/
    ├── config.yaml
    ├── results.jsonl
    └── report.json

Each model’s configuration is stored in config.yaml, and the results are in results.jsonl. The benchmark report is generated by a script that aggregates results across models.

The report includes:

The report is stored as report.json:

{
  "benchmark": "squad-v2",
  "date": "2025-04-05",
  "models": {
    "llama-3-8b": {
      "exact_match": 0.82,
      "f1": 0.86,
      "avg_latency_ms": 450,
      "tokens_per_second": 278,
      "task_breakdown": {
        "short-context": {"em": 0.80, "f1": 0.84},
        "long-context": {"em": 0.85, "f1": 0.88}
      }
    },
    "phi-3-mini": {
      "exact_match": 0.78,
      "f1": 0.82,
      "avg_latency_ms": 310,
      "tokens_per_second": 350,
      "task_breakdown": {
        "short-context": {"em": 0.76, "f1": 0.80},
        "long-context": {"em": 0.80, "f1": 0.84}
      }
    }
  }
}

This report is used to generate a comparison table, often in benchmark-summary.md:

| Model            | EM   | F1   | Latency (ms) | TPS   |
|------------------|------|------|--------------|-------|
| llama-3-8b       | 0.82 | 0.86 | 450          | 278   |
| phi-3-mini       | 0.78 | 0.82 | 310          | 350   |
| mixtral-8x7b     | 0.84 | 0.88 | 610          | 240   |

Tracking Evaluation Artifacts

Every evaluation run must be tracked. The evaluation pipeline generates a unique run_id from a timestamp and a short hash of the configuration.

The run_id is used to tag and version artifacts:

{"run_id": "20250405T123456-abc123", "timestamp": "2025-04-05T12:34:56Z", "event": "start", "dataset": "squad-v2", "model": "llama-3-8b"}
{"run_id": "20250405T123456-abc123", "timestamp": "2025-04-05T12:35:10Z", "event": "task_completed", "task_id": "q1", "latency_ms": 432, "prediction": "Paris"}
{"run_id": "20250405T123456-abc123", "timestamp": "2025-04-05T12:36:45Z", "event": "report_generated", "metrics": {"em": 0.82, "f1": 0.86}}

The eval.log file is parsed by a monitoring system to detect regressions, latency spikes, or failed tasks.

Failure Modes and Common Pitfalls

Standardizing Evaluation Across Projects

To make evaluation consistent across teams, a project defines a standard-benchmarks/ directory with pre-configured datasets and evaluation workflows.

Each benchmark is a subdirectory with:

The run-eval.sh script uses standard tools:

#!/bin/bash
set -euo pipefail

DATASET="$1"
MODEL="$2"
OUTPUT_DIR="results/$(date +%Y%m%d-%H%M%S)-${MODEL}"

mkdir -p "$OUTPUT_DIR"


python infer.py \
  --model "$MODEL" \
  --dataset "$DATASET" \
  --output "$OUTPUT_DIR/results.jsonl"

# Compute metrics
python compute-report.py \
  --results "$OUTPUT_DIR/results.jsonl" \
  --config "eval-config.yaml" \
  --output "$OUTPUT_DIR/report.json"

# Save run metadata
cat <<EOF > "$OUTPUT_DIR/run_metadata.json"
{
  "model": "$MODEL",
  "dataset": "$DATASET",
  "start_time": "$(date -u +"%Y-%m-%dT%H:%M:%SZ")",
  "run_id": "$(uuidgen)"
}
EOF

echo "Evaluation complete: $OUTPUT_DIR"

The compute-report.py script uses the evaluate library and json standard library to generate the report.

This workflow ensures that any engineer can run an evaluation with just:

./run-eval.sh data/squad-v2.jsonl llama-3-8b

And get a complete, versioned evaluation with logs, results, and metrics.

Summary

LLM Evaluation and Benchmarking is not a one-off task. It is a repeatable process of defining tasks, running models, computing metrics, and tracking results. The pipeline relies on standard formats: JSONL for datasets, YAML for configuration, JSON for results and reports. The process exposes subtle failure modes—missing IDs, mismatched output types, unnormalized prompts—that silently degrade evaluation quality. By standardizing across projects, teams achieve consistency, enable comparison, and build trust in model performance. The core of the process is the evaluation run: a configuration, a dataset, a model, and a report. Every run is a unit of evidence.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.