Testing In Production

Production engineering guide for testing in production covering patterns, implementation strategies, and operational best practices.

Testing in production is the practice of running tests on live systems under real user load, where the test data, infrastructure, and network conditions reflect actual usage. It matters when a feature release must be validated not just in staging, but in the environment where users interact with the system daily—when failures in production are costly, and when the cost of rolling back a release is nontrivial.

Running Real User Scenarios in Production

To validate that a new feature works end-to-end in production, engineers execute test scenarios that mimic actual user behavior. These scenarios are not isolated test cases but workflows that span services, databases, and external APIs. The key is to inject realistic data and simulate user actions while the system is live.

A common pattern is to use a test runner that executes a sequence of HTTP requests, recording and validating responses. The runner is deployed as a service alongside the application, either as a standalone process or as a Kubernetes pod.


testSuite:
  - name: "User checkout flow"
    steps:
      - method: POST
        url: "/api/v1/users"
        body:
          name: "Alice"
          email: "[email protected]"
        expectedStatus: 201
        assertions:
          - path: "$.id"
            type: "integer"
          - path: "$.email"
            value: "[email protected]"
      - method: POST
        url: "/api/v1/cart"
        body:
          userId: 123
          items:
            - productId: 456
              quantity: 2
        expectedStatus: 200
      - method: POST
        url: "/api/v1/orders"
        body:
          cartId: 789
          paymentMethod: "credit_card"
        expectedStatus: 201
        assertions:
          - path: "$.status"
            value: "confirmed"
          - path: "$.total"
            type: "number"
            greaterThan: 100.0

The test runner consumes this configuration and runs the suite at scheduled intervals. It logs each step with timestamps, request and response bodies, and assertion outcomes. Errors are written to a structured log stream, typically in JSON, and indexed in a monitoring system.

Failure is not just a test result but a full diagnostic context. When an assertion fails, the runner captures:

A test failure might appear as:

{
  "testId": "checkout-flow-2025-04-05",
  "timestamp": "2025-04-05T12:34:56Z",
  "status": "failed",
  "steps": [
    {
      "name": "Create user",
      "method": "POST",
      "url": "/api/v1/users",
      "status": 201,
      "response": {
        "id": 123,
        "email": "[email protected]"
      },
      "durationMs": 42,
      "assertions": [
        { "path": "$.id", "passed": true },
        { "path": "$.email", "expected": "[email protected]", "actual": "[email protected]", "passed": true }
      ]
    },
    {
      "name": "Add to cart",
      "method": "POST",
      "url": "/api/v1/cart",
      "status": 200,
      "response": { "cartId": 789 },
      "durationMs": 110,
      "assertions": [
        { "path": "$.cartId", "passed": true }
      ]
    },
    {
      "name": "Place order",
      "method": "POST",
      "url": "/api/v1/orders",
      "status": 201,
      "response": {
        "orderId": 301,
        "status": "confirmed",
        "total": 98.75
      },
      "durationMs": 612,
      "assertions": [
        { "path": "$.status", "expected": "confirmed", "actual": "confirmed", "passed": true },
        { "path": "$.total", "expected": "number", "actual": 98.75, "passed": false }
      ],
      "error": "Total expected to be greater than 100.0, but was 98.75"
    }
  ]
}

This level of detail allows engineers to diagnose the issue without accessing the application instance. The test runner can be triggered manually via a simple API call.

curl -X POST https://test-runner.example.com/v1/run \
  -H "Authorization: Bearer $TEST_RUNNER_TOKEN" \
  -d '{"suite": "checkout-flow", "config": {"userId": 123, "items": [{"productId": 456, "quantity": 2}]}' \
  -H "Content-Type: application/json"

Observing System Behavior Under Load

To assess how the system behaves under sustained use, engineers deploy load tests that run continuously in production. These tests simulate thousands of concurrent users and monitor system metrics in real time.

A typical load test is configured as a daemon that spawns worker processes, each executing a sequence of requests. The workers are controlled via a central configuration service.

# load-test-config.yaml
loadTest:
  name: "Peak-hour checkout stress"
  duration: "1h"
  concurrency: 500
  rampUp: "15m"
  target:
    - service: "order-service"
      endpoint: "/api/v1/orders"
      method: "POST"
      payloadTemplate: "order.json.j2"
      rate: 100
  metrics:
    - name: "latency"
      path: "$.durationMs"
      window: "5m"
      thresholds:
        - metric: "p95"
          expected: 500
          actual: 712
          status: "warning"
    - name: "errorRate"
      path: "$.errorCount"
      window: "10m"
      thresholds:
        - metric: "rate"
          expected: 0.01
          actual: 0.032
          status: "critical"

Each worker pulls the configuration from a shared storage, such as a database or a config service. The workers execute their test scenarios while streaming metrics to a time-series database.

# Worker script: run-load-test.sh
#!/bin/bash
set -euo pipefail

export TEST_CONFIG_URL="https://config.example.com/load-tests/peak-hour-checkout"
export METRICS_ENDPOINT="https://metrics.example.com/v1/ingest"

# Fetch current config
CONFIG=$(curl -s "$TEST_CONFIG_URL" | jq -c .)

# Parse configuration
CONCURRENCY=$(echo "$CONFIG" | jq -r '.concurrency')
DURATION=$(echo "$CONFIG" | jq -r '.duration')
TARGET_ENDPOINT=$(echo "$CONFIG" | jq -r '.target[0].endpoint')

# Run for duration
start_time=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
while true; do
  # Send test request
  RESPONSE=$(curl -s -w "%{time_total}" -o /dev/stdout \
    -X POST "$TARGET_ENDPOINT" \
    -H "Content-Type: application/json" \
    -d '{"userId": 123, "items": [{"productId": 456, "quantity": 1}]}' \
    -H "X-Trace-Id: $(uuidgen)")

  LATENCY=$(echo "$RESPONSE" | tail -n1 | tr -d '\n')
  RESPONSE_BODY=$(echo "$RESPONSE" | head -n -1)

  # Publish metrics
  curl -X POST "$METRICS_ENDPOINT" \
    -H "Content-Type: application/json" \
    -d '{
      "timestamp": "'"$(date -u +"%Y-%m-%dT%H:%M:%SZ")"'",
      "test": "peak-hour-checkout",
      "latencyMs": '"$LATENCY"',
      "success": true,
      "traceId": "'"$(echo "$RESPONSE_BODY" | jq -r '.traceId')"'"
    }'

  # Check if duration elapsed
  elapsed=$(($(date -u +"%s") - $(date -u -d "$start_time" +"%s")))
  if [ $elapsed -ge $((DURATION * 60)) ]; then
    break
  fi

  # Wait between requests
  sleep 0.5
done

The load test is considered to have passed if all metrics stay within their defined thresholds for a sustained period. A failure triggers alerts and generates a report that includes request traces, database query logs, and service-level monitoring data.

Detecting Regressions with Feature Flags

Feature flags are the backbone of testing in production, enabling gradual rollout and validation of features. A new feature is enabled via a flag, and tests are run to verify the behavior under real traffic.

Each feature flag is managed via a key-value store, where the value is a JSON object containing configuration and metadata.

{
  "feature": "checkout-v2",
  "enabled": true,
  "variants": {
    "default": {
      "description": "Baseline checkout flow",
      "weight": 0.7
    },
    "new-payment": {
      "description": "With 3D secure payment",
      "weight": 0.3
    }
  },
  "testing": {
    "testSuite": "checkout-flow",
    "runInterval": "1h",
    "baseline": {
      "metric": "orderSuccessRate",
      "expected": 0.98,
      "window": "1d"
    },
    "target": {
      "metric": "averageOrderTime",
      "expected": 4.2,
      "window": "1h"
    }
  }
}

A test runner continuously polls the feature flag store and triggers test suites whenever a flag is updated. The runner collects metrics from the production system and compares them against the baseline.

If the average order time increases from 3.8 to 4.5 minutes, the test runner logs the deviation and checks if it correlates with the rollout of the new variant.

The key insight is that the test is not just pass/fail but a continuous evaluation. A regression is detected not when a test fails, but when a metric drifts beyond its expected range over time.

Validating Data and Schema Changes

When a database schema evolves, testing in production ensures that the change works across all services and data pipelines. The validation process begins with a schema migration script that alters the database, followed by a data validation test.

The migration is deployed via a CI/CD pipeline, and a test checks that all required columns, constraints, and indexes exist.

# validate-schema.sh
#!/bin/bash
set -euo pipefail

DATABASE_URL="postgresql://user:[email protected]/app"

# Check if table exists
if ! psql -d "$DATABASE_URL" -c "\dt public.orders" | grep -q "orders"; then
  echo "Error: table public.orders does not exist"
  exit 1
fi

# Check if column exists
if ! psql -d "$DATABASE_URL" -c "SELECT column_name FROM information_schema.columns WHERE table_name = 'orders' AND column_name = 'total_amount'" | grep -q "total_amount"; then
  echo "Error: column total_amount missing in orders table"
  exit 1
fi

# Check if index exists
if ! psql -d "$DATABASE_URL" -c "\di" | grep -q "orders_total_amount_idx"; then
  echo "Error: index orders_total_amount_idx not created"
  exit 1
fi

# Validate data integrity
psql -d "$DATABASE_URL" -c "
  SELECT
    COUNT(*) AS total_orders,
    AVG(total_amount) AS avg_amount,
    MIN(total_amount) AS min_amount,
    MAX(total_amount) AS max_amount
  FROM public.orders
  WHERE created_at > '2025-04-01'
" | jq -r '
  .[] | 
  select(.total_orders > 1000) |
  select(.avg_amount > 100.0) |
  select(.min_amount >= 1.0) |
  select(.max_amount <= 10000.0) |
  {status: "passed", message: "Schema validation succeeded"}
'

The script is run as a job in the CI pipeline and triggered manually or automatically after a migration.

Handling Test Failures and Rollback

When a test fails, the system must respond quickly. A failure is not just a log entry but a trigger for actions.

The most common action is to roll back the change that introduced the regression. The rollback is initiated by a script that:

  1. Identifies the latest successful deployment
  2. Updates the deployment manifest to switch to the previous version
  3. Triggers a test suite to validate the rollback
# rollback-deployment.sh
#!/bin/bash
set -euo pipefail

DEPLOYMENT_SERVICE="https://deployments.example.com/v1"
ROLLBACK_TARGET=$(curl -s "$DEPLOYMENT_SERVICE/latest-successful" | jq -r '.version')

echo "Rolling back to version $ROLLBACK_TARGET"

curl -X PATCH "$DEPLOYMENT_SERVICE/current" \
  -H "Content-Type: application/json" \
  -d '{"targetVersion": "'"$ROLLBACK_TARGET"'"}'

# Wait for deployment to stabilize
sleep 60

# Run validation test
curl -X POST "$DEPLOYMENT_SERVICE/test" \
  -H "Authorization: Bearer $TEST_TOKEN" \
  -d '{"suite": "data-integrity-check", "target": "current"}'

The test runner monitors the rollout and marks the deployment as successful only if all tests pass. If any test fails during the rollback, the system sends an alert and halts the process.

Common Pitfalls and Silent Failures

The most subtle failure is the test not running at all: the test runner is deployed, but the scheduler is misconfigured, and the test suite runs only once a day instead of every hour. The system appears stable, but a regression goes undetected for days.

Testing in production is not just a set of tests—it is a continuous feedback loop where each test informs the next, and each failure becomes a trigger for action. It is the practice of validating software not in isolation, but in the context of its users, its data, and its environment.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.