Test Observability and Analytics

Turn test results into actionable intelligence. Covers test failure analysis, flaky test detection, test execution trends, coverage gap identification, and the patterns that transform testing from a pass/fail gate into a continuous feedback system.

Test Observability and Analytics is the practice of instrumenting test execution to capture, correlate, and analyze test behavior across environments, pipelines, and test types. It matters when tests fail without clear context—when a flaky E2E test fails in CI but the logs, traces, and metadata are scattered across systems, and when a regression in a contract test goes unnoticed until the next release. This is not just about collecting logs; it is about making test outcomes actionable, diagnosable, and predictive.

Instrumenting Tests for Observability

To turn tests into observability sources, every test must emit structured metadata about its execution. This includes timestamps, environment, test input, duration, and outcome. The test runner must record these details in a common schema.

A test suite written in Python using pytest emits structured output via the --json-report flag:

pytest --json-report --json-report-file=reports/test-results.json

The resulting test-results.json contains a list of test cases with keys such as name, duration, outcome, environment, and metadata. The metadata field is a dictionary that can include branch, commit, build_id, test_type, and test_config.

For a test run in a CI environment, the metadata might look like:

{
  "branch": "feature/user-profile",
  "commit": "a1b2c3d4e5f6",
  "build_id": "2023-10-05-1234",
  "test_type": "e2e",
  "test_config": {
    "browser": "chrome:115",
    "headless": true,
    "viewport": "1920x1080"
  }
}

This structured output is the foundation. Without it, no correlation across systems is possible.

Centralizing Test Data

Once tests emit structured output, they must be ingested into a central system. The ingestion pipeline uses a simple HTTP API to push test results. The standard format is JSON over HTTP POST to a /api/v1/test-results endpoint.

Example request body:

{
  "test_id": "e2e-user-login-123",
  "suite": "user-authentication",
  "start_time": "2023-10-05T12:34:56Z",
  "end_time": "2023-10-05T12:35:21Z",
  "duration": 25,
  "status": "passed",
  "environment": "staging",
  "artifacts": [
    {
      "type": "screenshot",
      "url": "https://artifacts.example.com/screenshots/e2e-user-login-123.png"
    },
    {
      "type": "trace",
      "url": "https://artifacts.example.com/traces/e2e-user-login-123.json"
    }
  ],
  "metadata": {
    "branch": "feature/user-profile",
    "commit": "a1b2c3d4e5f6",
    "build_id": "2023-10-05-1234"
  },
  "steps": [
    {
      "name": "Navigate to login page",
      "duration": 3,
      "status": "passed",
      "error": null
    },
    {
      "name": "Fill in credentials",
      "duration": 2,
      "status": "passed",
      "error": null
    },
    {
      "name": "Submit form",
      "duration": 10,
      "status": "failed",
      "error": {
        "message": "Network timeout after 8s",
        "type": "timeout",
        "stack": "at LoginPage.submit() at login-page.js:45\nat test-e2e.js:12"
      }
    }
  ]
}

The receiving service stores this data in a relational database using a table schema:

CREATE TABLE test_results (
  test_id VARCHAR(64) PRIMARY KEY,
  suite VARCHAR(64) NOT NULL,
  start_time TIMESTAMPTZ NOT NULL,
  end_time TIMESTAMPTZ,
  duration INTEGER,
  status VARCHAR(16) NOT NULL,
  environment VARCHAR(32),
  metadata JSONB,
  artifacts JSONB,
  steps JSONB
);

This enables fast querying and analysis across test runs.

Querying Test Outcomes

With data centralized, engineers can answer questions like:

To find flaky tests, query for tests that have failed in at least two of the last ten runs:

SELECT
  t.suite,
  t.test_id,
  COUNT(*) AS failure_count,
  AVG(t.duration) AS avg_duration,
  PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY t.duration) AS median_duration
FROM test_results t
WHERE t.status = 'failed'
  AND t.start_time >= NOW() - INTERVAL '10 days'
  AND t.suite IN (
    SELECT DISTINCT suite FROM test_results
    WHERE start_time >= NOW() - INTERVAL '10 days'
  )
GROUP BY t.suite, t.test_id
HAVING COUNT(*) >= 2
ORDER BY failure_count DESC;

This query returns a list of flaky tests, ranked by failure frequency. The results can feed into dashboards or alerting systems.

For time-series analysis, extract test durations over time:

SELECT
  DATE_TRUNC('hour', start_time) AS hour,
  suite,
  AVG(duration) AS avg_duration,
  STDDEV(duration) AS stddev_duration,
  COUNT(*) AS run_count
FROM test_results
WHERE start_time >= NOW() - INTERVAL '7 days'
GROUP BY hour, suite
ORDER BY hour, suite;

This output can be visualized as a line chart showing the trend of test execution times, revealing regressions or performance improvements.

Correlating Test Execution with System Events

A test failure is more valuable when it is linked to system state. The test runner must attach context from the environment: logs from services, database snapshots, and metrics from monitoring.

The test runner captures logs from dependent services using docker-compose logs and uploads them as artifacts:

docker-compose logs --no-color --timestamps --tail=100 api > logs/api.log
docker-compose logs --no-color --timestamps --tail=100 auth > logs/auth.log

These logs are uploaded alongside the test result with metadata:

"artifacts": [
  {
    "type": "log",
    "name": "api",
    "content": "docker-compose logs --no-color --timestamps --tail=100 api",
    "url": "https://artifacts.example.com/logs/api.log"
  },
  {
    "type": "log",
    "name": "auth",
    "content": "docker-compose logs --no-color --timestamps --tail=100 auth",
    "url": "https://artifacts.example.com/logs/auth.log"
  }
]

The test runner can also query the service’s health endpoint during the test:

curl -s http://api.example.com/health

If the response is {"status": "ok", "db": "healthy", "cache": "healthy"} and the test fails, the system assumes the API was healthy at the time of failure. If the response shows {"status": "unhealthy", "db": "down"}, the test failure is likely due to database issues.

The runner can also capture a snapshot of the database at the time of test execution:

pg_dump -U testuser -h db.example.com -d app_db > dumps/app_db_snapshot.sql

This dump is stored as an artifact and linked in the test result.

Visualizing Test Observability

Dashboards show the health of test suites at a glance. The core components are:

A test dashboard is built using a query that combines test results with metadata and artifacts.

Example query for a test status dashboard:

SELECT
  t.suite,
  t.status,
  COUNT(*) AS count,
  AVG(t.duration) AS avg_duration,
  ARRAY_AGG(t.test_id) AS test_ids,
  STRING_AGG(DISTINCT m.branch, ',') AS branches,
  STRING_AGG(DISTINCT m.environment, ',') AS environments
FROM test_results t
JOIN test_results_metadata m ON t.test_id = m.test_id
WHERE t.start_time >= NOW() - INTERVAL '1 day'
GROUP BY t.suite, t.status
ORDER BY t.suite, t.status;

This query feeds a dashboard that displays the percentage of passed, failed, and skipped tests per suite, with drill-downs to individual test runs.

Detecting and Alerting on Test Regressions

A test regression is a significant increase in test duration or failure rate. The detection mechanism uses a sliding window of runs and thresholds.

To detect a regression, compute the moving average of test durations over the last 7 days and compare it to the current run:

WITH recent AS (
  SELECT
    test_id,
    duration,
    start_time,
    AVG(duration) OVER (PARTITION BY test_id ORDER BY start_time ROWS BETWEEN 7 PRECEDING AND CURRENT ROW) AS rolling_avg
  FROM test_results
  WHERE start_time >= NOW() - INTERVAL '14 days'
)
SELECT
  test_id,
  duration,
  rolling_avg,
  (duration - rolling_avg) / rolling_avg AS z_score
FROM recent
WHERE start_time >= NOW() - INTERVAL '1 day'
  AND (duration - rolling_avg) > 0.3 * rolling_avg
ORDER BY z_score DESC;

This query identifies tests that are significantly slower than their recent average. A z-score > 0.3 indicates a regression.

Alerts are triggered when a test fails more than 30% of the time in the last 24 hours:

SELECT
  suite,
  COUNT(*) AS total_runs,
  SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) AS failed_runs,
  (SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) * 100.0 / COUNT(*)) AS failure_rate
FROM test_results
WHERE start_time >= NOW() - INTERVAL '24 hours'
GROUP BY suite
HAVING (SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) * 100.0 / COUNT(*)) > 30;

These alerts can be sent via email, Slack, or webhook to a notification system.

Common Pitfalls and Silent Failures

Integrating Test Analytics into Development Workflows

Test analytics are most valuable when they are part of the development cycle. A pull request is enriched with a summary of test results:

This summary is generated by a CI job that queries the test database and emits a Markdown report:



| Suite | Pass Rate | Duration (avg) | Flaky Tests |
|-------|-----------|----------------|-------------|
| e2e | 87% | 24s | login, signup |
| contract | 100% | 12s | - |

### Top 3 Failing Tests

1. `e2e-user-login-123` — Failed at `2023-10-05T12:35:21Z`
   - Error: `Network timeout after 8s`
   - [Screenshot](https://artifacts.example.com/screenshots/e2e-user-login-123.png)
   - [Trace](https://artifacts.example.com/traces/e2e-user-login-123.json)

2. `e2e-user-signup-456` — Failed at `2023-10-05T12:34:56Z`
   - Error: `Validation failed: email must be unique`
   - [Screenshot](https://artifacts.example.com/screenshots/e2e-user-signup-456.png)

### Duration Trend

- Current branch: avg 24s
- Base branch: avg 21s
- Regression: +14%

> 📌 **Recommendation**: Investigate `e2e-user-login-123` and `e2e-user-signup-456` before merging.

This report is attached to the pull request and becomes a shared reference for the team.

Summary

Test Observability and Analytics transforms tests from passive validation tools into active diagnostic systems. By instrumenting tests with structured metadata, centralizing execution data, and correlating test outcomes with system state, teams can identify flaky tests, detect regressions early, and diagnose failures with confidence. The practice relies on standard tools—HTTP, SQL, JSON, shell scripts—and avoids vendor-specific patterns. The result is a shared understanding of test health across the engineering organization, where a test failure is not just a red dot, but a story.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.