Test Reporting Dashboards | The Garnet Wiki

Test Reporting Dashboards surface test execution results in real time, enabling teams to monitor test health across environments, releases, and test suites. They are essential during continuous integration, release cycles, and post-deployment verification, where timely visibility into test outcomes determines whether a build qualifies for release, and whether a deployment succeeded.

Monitoring Test Health Across Environments

A test reporting dashboard provides a real-time view of test execution across multiple environments: development, staging, production, and canary. The dashboard displays pass/fail rates, execution duration, flaky test counts, and regression trends over time.

Each test run is reported as a test_run record with the following structure:

{
  "id": "tr-7a2f1b",
  "suite": "e2e",
  "environment": "staging",
  "commit_sha": "a3b4c5d6",
  "started_at": "2025-04-05T10:23:15Z",
  "finished_at": "2025-04-05T10:42:31Z",
  "duration_ms": 115600,
  "status": "failed",
  "artifacts": [
    { "name": "test-results.xml", "url": "/artifacts/tr-7a2f1b/test-results.xml" },
    { "name": "screenshots", "url": "/artifacts/tr-7a2f1b/screenshots.tar.gz" },
    { "name": "logs", "url": "/artifacts/tr-7a2f1b/app.log" }
  ],
  "test_cases": [
    {
      "id": "tc-101",
      "name": "User login with valid credentials",
      "status": "passed",
      "duration_ms": 2340,
      "error": null,
      "attachments": [
        { "name": "screenshot.png", "url": "/artifacts/tr-7a2f1b/screenshots/login.png" }
      ]
    },
    {
      "id": "tc-102",
      "name": "User cannot log in with invalid password",
      "status": "failed",
      "duration_ms": 1870,
      "error": {
        "message": "Expected element 'Error message' to be visible",
        "stack_trace": "at LoginPage.login()\n  at TestSuite.loginTest()\n  at TestRunner.run()"
      },
      "attachments": [
        { "name": "screenshot.png", "url": "/artifacts/tr-7a2f1b/screenshots/login_error.png" },
        { "name": "network.log", "url": "/artifacts/tr-7a2f1b/network.log" }
      ]
    }
  ]
}

When a CI job completes, it sends this report via HTTP POST to the reporting API endpoint:

curl -X POST https://reporting.example.com/api/v1/test-runs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $REPORTING_TOKEN" \
  -d @test-run.json

The dashboard subscribes to this endpoint via polling or WebSocket, refreshing every 30 seconds to display the latest results.

Silent Failures in Test Reporting

A common failure mode is the missing or malformed test case entry. If a test runner emits test_cases as a list but the dashboard expects testCases, the entire test suite appears as a single "flaky" case, even when individual test cases are detailed.

Another silent failure: test results missing duration_ms, causing the average test execution time to be skewed or unavailable. When duration_ms is absent, the dashboard defaults to 1000 milliseconds for each test, but this leads to overestimated overall run times.

When a test run is reported, the following checks are performed:

status must be one of: passed, failed, skipped, in_progress, timed_out
started_at and finished_at must be in ISO 8601 format
test_cases must be a non-empty array
artifacts URLs must be accessible

The dashboard logs a test_run_invalid event when any of these validations fail:

{
  "event": "test_run_invalid",
  "run_id": "tr-7a2f1b",
  "timestamp": "2025-04-05T10:43:00Z",
  "errors": [
    "Missing required field 'test_cases'",
    "Invalid status value: 'pass' (expected 'passed')",
    "Test case 'tc-102' has missing 'error.stack_trace'"
  ],
  "payload": { ... }
}

Drilling into Test Failures

When a test run fails, the dashboard allows teams to drill down into individual test cases. Clicking on a failed test case reveals the full stack trace, attached screenshots, and network logs.

The test case detail view pulls in associated artifacts via HTTP GET:

GET /artifacts/tr-7a2f1b/screenshots/login_error.png

The dashboard caches these assets locally and serves them with ETag and Last-Modified headers to avoid redundant downloads.

A common mistake is uploading artifacts to a non-standard path. The test runner may store screenshots in screenshots/ but the dashboard expects assets/screenshots/. This results in missing images and empty attachments. The dashboard logs a missing_attachment event when a URL returns 404 or 410.

To avoid this, the test runner should use a consistent artifact directory structure:

artifacts/
├── test-results.xml
├── screenshots/
│   ├── login.png
│   └── login_error.png
├── logs/
│   ├── app.log
│   └── network.log
└── videos/
    └── login.mp4

The reporting API expects this layout and automatically discovers all artifacts.

Flaky Test Detection

A flaky test is one that passes and fails unpredictably across runs. The dashboard detects flakiness by tracking test case status across multiple runs.

Each test case has a flaky_history array that records its status over time:

{
  "test_case_id": "tc-102",
  "name": "User cannot log in with invalid password",
  "flaky_history": [
    { "run_id": "tr-7a2f1b", "status": "failed", "timestamp": "2025-04-05T10:42:31Z" },
    { "run_id": "tr-7a2f1b", "status": "passed", "timestamp": "2025-04-04T14:10:22Z" },
    { "run_id": "tr-7a2f1b", "status": "failed", "timestamp": "2025-04-03T09:05:11Z" }
  ]
}

The dashboard computes flakiness score as:

flakiness_score = (number of failures) / (total number of runs)

A test case with a score > 0.3 is flagged as flaky in the dashboard.

The dashboard also shows test case trend lines, plotting pass/fail status over time. When a test case fails in three consecutive runs, it is highlighted with a red border.

Integrating with CI/CD Workflows

The test reporting dashboard is integrated into the CI/CD pipeline via a test-reporting job that runs after test execution.

The job is defined in .gitlab-ci.yml as:

test-reporting:
  image: node:18
  script:
    - npm install
    - npm run test:report -- --format=json --output=report.json
    - curl -X POST https://reporting.example.com/api/v1/test-runs \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $REPORTING_TOKEN" \
        -d @report.json
  artifacts:
    paths:
      - report.json
    expire_in: 1 week
  only:
    - main
    - release/*

The test:report script uses jest with a custom reporter that outputs the JSON structure above. The --format=json flag ensures consistent output.

A subtle failure occurs when the test-run.json file is not properly serialized. If the JSON is missing test_cases or uses testcase instead of test_cases, the dashboard fails to render the test list. The most common cause is a missing JSON.stringify call with null values.

To catch this, the dashboard validates every incoming test run against a JSON Schema:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["id", "suite", "environment", "commit_sha", "started_at", "finished_at", "test_cases"],
  "properties": {
    "test_cases": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["id", "name", "status"],
        "properties": {
          "status": { "enum": ["passed", "failed", "skipped", "in_progress", "timed_out"] },
          "duration_ms": { "type": "integer", "minimum": 0 },
          "error": {
            "type": "object",
            "properties": {
              "message": { "type": "string" },
              "stack_trace": { "type": "string" }
            }
          }
        }
      }
    }
  }
}

The dashboard rejects any report that fails this schema, logging a test_run_schema_validation_failed event.

Customizing the Dashboard View

The dashboard supports custom views via a dashboard_config file stored in the project root:

{
  "title": "E2E Test Health",
  "refresh_interval_seconds": 60,
  "default_filters": {
    "environments": ["staging", "production"],
    "test_suites": ["e2e", "api"],
    "status": ["failed", "in_progress"]
  },
  "widgets": [
    {
      "type": "test-suite-summary",
      "title": "Test Suite Summary",
      "query": "SELECT suite, COUNT(*), AVG(duration_ms) FROM test_runs GROUP BY suite"
    },
    {
      "type": "flaky-test-list",
      "title": "Flaky Tests (Last 7 Days)",
      "threshold": 0.3,
      "limit": 10
    },
    {
      "type": "test-run-timeline",
      "title": "Recent Test Runs",
      "time_window_hours": 24
    }
  ]
}

The dashboard applies the dashboard_config during startup and allows users to override it via query parameters:

https://reporting.example.com/dashboards/e2e?env=staging&suite=e2e&since=2025-04-04

The dashboard also supports test run tagging. When a test run is reported, it can include an optional tags array:

{
  "id": "tr-7a2f1b",
  "tags": ["regression", "smoke", "critical"],
  "test_cases": [ ... ]
}

These tags enable filtering, labeling, and grouping. The dashboard indexes tags and provides a tag cloud for quick navigation.

Handling Test Run Artifacts

Artifacts are stored in a durable object store (e.g., AWS S3, GCS). The reporting API generates public URLs for each artifact, with a consistent naming pattern:

https://artifacts.example.com/{run_id}/{path}

For example:

https://artifacts.example.com/tr-7a2f1b/screenshots/login_error.png
https://artifacts.example.com/tr-7a2f1b/logs/app.log

The dashboard preloads artifacts when a test run is selected. It also supports lazy loading: images are loaded only when the user scrolls to them.

A common issue is missing artifact compression. When screenshots are large, the dashboard takes too long to load. The solution is to compress images using pngquant or jpegoptim before uploading.

The test runner should include a step to optimize artifacts:


find screenshots/ -name "*.png" -exec pngquant --quality=65-90 {} \;

# Compress logs
gzip -9 logs/app.log

# Tar and compress all artifacts
tar -czf artifacts.tar.gz screenshots/ logs/ videos/

The reporting API expects compressed artifacts and applies Content-Encoding: gzip to artifacts.tar.gz.

Alerting on Test Run Status

The dashboard triggers alerts when a test run fails or when a test suite has a flakiness score above threshold.

Alerts are sent via HTTP POST to a webhook endpoint:

{
  "alert": "test_run_failed",
  "run_id": "tr-7a2f1b",
  "suite": "e2e",
  "environment": "staging",
  "message": "E2E test suite failed on staging after 115s",
  "details": {
    "failures": 3,
    "total": 15,
    "duration_ms": 115600,
    "failed_cases": [
      "User login with valid credentials",
      "User cannot log in with invalid password"
    ],
    "run_url": "https://reporting.example.com/test-runs/tr-7a2f1b"
  }
}

The dashboard sends alerts to Slack, email, or a monitoring system like Prometheus via Alertmanager.

A subtle point: alert deduplication. The dashboard deduplicates alerts based on run_id and suite. If the same test run fails twice, only one alert is sent. This prevents alert storms during CI bursts.

The dashboard also supports test run summaries. After a test run completes, it generates a summary email or Slack message with a table of pass/fail rates, duration, and links to the full report.

Summary

Test reporting dashboards are the central nervous system of test execution in engineering teams. They transform raw test results into actionable insights. Their success hinges on consistent data formats, robust artifact handling, and intelligent failure detection. When a test run fails silently, the dashboard becomes the first place engineers look. Its ability to surface flaky tests, drill into failures, and integrate with CI/CD workflows makes it indispensable.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.