LLM Application Testing

LLM applications introduce new failure modes beyond traditional software: hallucinated outputs, inconsistent reasoning, state drift, and prompt fragility. Testing them requires more than unit and integration tests—it demands validation of semantic fidelity, prompt robustness, and model behavior across diverse inputs. This page covers the essential practices, tools, and workflows engineers use to rigorously test LLM-driven applications, from the first pipeline to production deployment.

Testing Prompt Design and Output Quality

Validating a prompt begins with defining input-output pairs and evaluating whether the model produces expected responses across a representative sample. A prompt is not a string; it is a contract between the application and the model. The key test is whether a given prompt consistently produces correct, coherent, and contextually appropriate outputs.

For a prompt defined in a JSON file, testing involves:

{
  "prompt": "Summarize the following article in three bullet points:\n\n{article}",
  "inputs": {
    "article": "The rise of AI in healthcare has transformed patient diagnostics. Machine learning models now analyze medical images with accuracy surpassing human radiologists. In particular, deep learning models trained on thousands of MRI scans detect early-stage tumors with high precision. These systems reduce false positives and enable earlier interventions, improving survival rates."
  },
  "expected": [
    "Machine learning models analyze medical images with accuracy exceeding human radiologists.",
    "Deep learning models trained on MRI scans detect early-stage tumors with high precision.",
    "Early detection reduces false positives and improves patient survival rates."
  ]
}

A test suite runs this prompt against a collection of inputs using curl to invoke a local LLM server:

curl -X POST http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Summarize the following article in three bullet points:\n\nThe rise of AI in healthcare has transformed patient diagnostics. Machine learning models now analyze medical images with accuracy surpassing human radiologists. In particular, deep learning models trained on thousands of MRI scans detect early-stage tumors with high precision. These systems reduce false positives and enable earlier interventions, improving survival rates.",
    "max_tokens": 200,
    "temperature": 0.5
  }' | jq '.choices[0].text'

The output must be compared against expected values using a semantic similarity metric (e.g., cosine similarity between sentence embeddings). A failing test shows:

{
  "input": "The rise of AI in healthcare has transformed patient diagnostics...",
  "actual": [
    "AI in healthcare has improved diagnostics, with models analyzing medical images more accurately than radiologists.",
    "Deep learning models trained on MRI scans detect early-stage tumors with high precision.",
    "Early detection leads to better outcomes and reduced false positives."
  ],
  "expected": [
    "Machine learning models analyze medical images with accuracy exceeding human radiologists.",
    "Deep learning models trained on MRI scans detect early-stage tumors with high precision.",
    "Early detection reduces false positives and improves patient survival rates."
  ],
  "similarity_scores": [0.87, 0.92, 0.79],
  "threshold": 0.85,
  "pass": false,
  "failures": [
    {
      "index": 0,
      "actual": "AI in healthcare has improved diagnostics, with models analyzing medical images more accurately than radiologists.",
      "expected": "Machine learning models analyze medical images with accuracy exceeding human radiologists.",
      "reason": "Terminology mismatch: 'AI' vs. 'machine learning'; 'improved diagnostics' not in expected."
    }
  ]
}

A common failure mode: the temperature setting in the LLM call is misapplied. A temperature=0.0 should produce deterministic outputs, but if the model’s internal cache is not cleared between runs, the same prompt yields different outputs across test executions. This is silent until a test suite runs over multiple days and the output varies.

Evaluating Model Behavior Across Input Distributions

An LLM application must perform reliably across diverse and real-world input distributions. Testing must include synthetic and real data, with emphasis on edge cases and stress conditions.

To evaluate model behavior under distribution shift, use a test suite that runs the model on a dataset of inputs, each with a known expected output. The dataset is stored as a CSV with columns input, expected, category, and difficulty.

input,expected,category,difficulty
"Explain quantum entanglement in simple terms.", "Two particles become linked so that the state of one instantly affects the other, no matter the distance.", "explanation", "medium"
"Summarize the plot of 'The Great Gatsby'.", "Gatsby throws lavish parties to win back Daisy, a woman he loves. Despite wealth and fame, he fails to rekindle their romance. The story ends with Gatsby's death and Nick reflecting on the American Dream.", "summary", "hard"

A test script processes this dataset:

#!/bin/bash


INPUTS_CSV="test_inputs.csv"
RESULTS_JSON="test_results.json"

while IFS=, read -r input expected category difficulty; do
  response=$(curl -sX POST http://localhost:8080/v1/completions \
    -H "Content-Type: application/json" \
    -d "{
      \"prompt\": \"$input\",
      \"max_tokens\": 300,
      \"temperature\": 0.3
    }" | jq -c '.choices[0].text')

  echo "{
    \"input\": \"$input\",
    \"expected\": \"$expected\",
    \"actual\": $response,
    \"category\": \"$category\",
    \"difficulty\": \"$difficulty\",
    \"similarity\": $(python3 -c '
      import sys
      from sentence_transformers import SentenceTransformer
      model = SentenceTransformer("all-MiniLM-L6-v2")
      inputs = [sys.argv[1], sys.argv[2]]
      embeddings = model.encode(inputs)
      from sklearn.metrics.pairwise import cosine_similarity
      sim = cosine_similarity([embeddings[0]], [embeddings[1]])[0][0]
      print(f"{sim:.3f}")
    ' "$expected" "$response")
  }" >> $RESULTS_JSON
done < <(tail -n +2 $INPUTS_CSV)

The script outputs a JSON array of test results, which can be analyzed for accuracy per category and difficulty. A failure occurs when a high-difficulty input produces a response that is semantically similar but misses a key concept—e.g., for "Explain quantum entanglement," the model mentions "linked particles" but fails to explain the "instantaneous" nature of state correlation.

Testing Prompt Variants and A/B Experiments

When multiple prompt versions exist, testing must compare their outputs across the same input set. A/B testing in LLMs is not about UI changes but about prompt engineering and model configuration.

A directory structure defines different prompt variants:

prompts/
├── v1/
│   └── summary_prompt.txt
├── v2/
│   └── summary_prompt.txt
├── v3/
│   └── summary_prompt.txt

Each variant is evaluated on the same test set using a script that runs each prompt and collects metrics:

for PROMPT_DIR in prompts/*; do
  PROMPT_NAME=$(basename $PROMPT_DIR)
  OUTPUT_FILE="results_${PROMPT_NAME}.json"
  
  while IFS=, read -r input expected; do
    PROMPT=$(cat $PROMPT_DIR/summary_prompt.txt | sed "s/{article}/$input/g")
    
    response=$(curl -sX POST http://localhost:8080/v1/completions \
      -H "Content-Type: application/json" \
      -d "{
        \"prompt\": \"$PROMPT\",
        \"max_tokens\": 250,
        \"temperature\": 0.4
      }" | jq -c '.choices[0].text')

    echo "{
      \"input\": \"$input\",
      \"expected\": \"$expected\",
      \"actual\": $response,
      \"similarity\": $(python3 -c '
        import sys
        from sentence_transformers import SentenceTransformer
        model = SentenceTransformer("all-MiniLM-L6-v2")
        inputs = [sys.argv[1], sys.argv[2]]
        embeddings = model.encode(inputs)
        from sklearn.metrics.pairwise import cosine_similarity
        sim = cosine_similarity([embeddings[0]], [embeddings[1]])[0][0]
        print(f"{sim:.3f}")
      ' "$expected" "$response")
    }" >> $OUTPUT_FILE
  done < <(tail -n +2 test_inputs.csv)
done

The results are compared using jq and awk to produce a summary:

jq -r '[
  .[] | 
  { prompt: .prompt_name, avg_similarity: .similarity | add / length },
  { prompt: .prompt_name, total: length, failed: [.[] | select(.similarity < 0.8)] | length }
]' results_*.json

A common edge case: a prompt variant uses temperature=1.0 but the model server defaults to temperature=0.7. The test suite passes, but in production, the model runs at temperature=0.7, so outputs are more deterministic than expected. This discrepancy is silent until the evaluation is repeated with a test that includes a temperature field in the request and verifies that the model actually received the intended value.

Testing Application-Level Behavior and End-to-End Flows

An LLM application is not just a model inference service; it includes preprocessing, postprocessing, state management, and orchestration. Testing must validate the full pipeline.

A typical end-to-end test involves a user-facing API that receives a document, processes it through a series of LLM steps, and returns a structured result.

POST /api/v1/analyze-document
Content-Type: application/json

{
  "document": "Patient report: High blood pressure, fatigue, and shortness of breath. MRI shows left ventricular hypertrophy. Echocardiogram confirms reduced ejection fraction.",
  "steps": [
    {
      "name": "extract_symptoms",
      "prompt": "Extract all symptoms mentioned in the following clinical note.\n\n{note}",
      "output_format": "json"
    },
    {
      "name": "diagnosis_suggestion",
      "prompt": "Based on the symptoms and imaging results, suggest possible diagnoses.\n\nSymptoms: {symptoms}\nImaging: {imaging}",
      "output_format": "list"
    }
  ]
}

The test sends this request and validates the response:

{
  "steps": [
    {
      "name": "extract_symptoms",
      "output": [
        "High blood pressure",
        "Fatigue",
        "Shortness of breath",
        "Left ventricular hypertrophy",
        "Reduced ejection fraction"
      ]
    },
    {
      "name": "diagnosis_suggestion",
      "output": [
        "Heart failure",
        "Hypertensive heart disease",
        "Ischemic cardiomyopathy",
        "Valvular heart disease"
      ]
    }
  ]
}

The test ensures that:

• Each step is executed in sequence.
• The output of one step is correctly passed as input to the next.
• The model server correctly processes the prompt template and renders {note}, {symptoms}, etc.
• The output_format directive is respected: json results are parsed, list outputs are split by newline or bullet points.

Failure modes:

• A step expects a json output but receives list—the diagnosis_suggestion step fails to parse the output because it assumes a JSON array but gets a string with newline-separated items.
• The prompt template is missing a key, such as imaging being rendered as imaging: MRI shows left ventricular hypertrophy. Echocardiogram confirms reduced ejection fraction. instead of imaging: {imaging}—the model receives a single string but expects a structured object.

Monitoring and Regressions in Production

Once deployed, LLM applications must be tested continuously. A regression occurs when a new model version, updated prompt, or changed preprocessing pipeline causes output quality to degrade.

A monitoring pipeline uses systemctl to run a daily test job:

# /etc/systemd/system/llm-testing-daily.service
[Unit]
Description=Daily LLM Application Test Suite
After=network.target

[Service]
Type=oneshot
ExecStart=/opt/llm-testing/test.sh
User=llm
Environment=PATH=/usr/local/bin:/usr/bin:/bin
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

The test script runs the full suite and logs results:

#!/bin/bash
# test.sh

LOG_DIR="/var/log/llm-testing"
DATE=$(date +%Y-%m-%d)
LOG_FILE="$LOG_DIR/results-${DATE}.json"

# Run prompt evaluation
python3 -m prompt_evaluator --dataset test_inputs.csv --output $LOG_FILE

# Run A/B comparison
python3 -m ab_tester --prompt-v1 prompts/v1 --prompt-v2 prompts/v2

# Compare against baseline
python3 -m regression_checker --baseline $LOG_DIR/baseline.json --current $LOG_FILE

# Send alert if regression detected
if grep -q "regression" $LOG_DIR/latest.log; then
  echo "Regression detected: see $LOG_FILE" | mail -s "LLM Application Regression" [email protected]
fi

A regression is detected when the average similarity score drops below a threshold, or when a new prompt variant fails to meet a minimum threshold on 3 out of 5 test cases.

Failure mode: the regression_checker script uses jq to extract similarity values but assumes the output is a JSON array. However, the prompt_evaluator writes one result per line as JSON objects (JSONL), so the script reads the first object but ignores the rest. This leads to a false positive: the average similarity is computed over only the first test case.

Testing Error Handling and Failures

LLM applications fail in subtle ways: prompt templates miss keys, model timeouts, output parsing errors, and out-of-distribution inputs.

A test suite includes error cases:

{
  "input": "Patient report: Confusion, fever, and headache. Imaging: CT scan shows enlarged ventricles.",
  "expected": [
    "Meningitis",
    "Hydrocephalus",
    "Encephalitis"
  ],
  "error_case": true,
  "expected_error": "KeyError: 'imaging' not found in prompt template"
}

The test runs the same request but checks for an HTTP 500 error and logs the model’s internal error message:

curl -X POST http://localhost:8080/api/v1/analyze-document \
  -H "Content-Type: application/json" \
  -d '{"document": "Patient report: Confusion, fever, and headache. Imaging: CT scan shows enlarged ventricles.", "steps": [...]}' \
  -w "\nHTTP status: %{http_code}\n" \
  -o - \
  -s \
  | jq '.error | select(.code == 500) | .message'

The test expects the output to include the error_case and expected_error fields, and verifies that the model server logs an ERROR entry with the message.

A common silent failure: the model server assumes imaging is always a dictionary but receives a string. The prompt template renders {imaging} as a string but the model expects a list of findings. The test catches this, but the error is logged only in the model server logs, not in the application logs.

Summary of Key Practices

• Use curl and jq for end-to-end testing with real HTTP APIs.
• Store test inputs as CSV or JSON, with metadata for categorization and difficulty.
• Validate semantic similarity using sentence embeddings from standard models like all-MiniLM-L6-v2.
• Compare prompt outputs against expected outputs using exact and fuzzy matching.
• Run A/B tests across prompt variants with consistent evaluation metrics.
• Monitor production with daily test jobs using systemctl.
• Test error handling by including test cases with missing keys, malformed inputs, and expected errors.
• Use JSONL for test output to support scalable, streaming evaluation.

LLM application testing is not a one-time activity. It is a continuous process of validating prompt fidelity, model behavior, and end-to-end correctness across real-world data and evolving system states.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.