Test Flakiness Reduction

Production-grade guide to test flakiness reduction covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.

Test flakiness reduction is the disciplined effort to eliminate intermittent test failures—tests that pass on one run but fail on the next without any code change. It matters when test suites are large, automated, and relied upon for release gates, CI/CD pipelines, and developer confidence. Flaky tests introduce noise, delay feedback, erode trust, and lead to overlooked bugs. This page details how to identify, diagnose, and eliminate flakiness in automated test suites, focusing on actionable practices and real-world configurations.

Identifying Flaky Tests

A flaky test is one that has inconsistent results across runs, typically with a pass/fail pattern over multiple executions. The key diagnostic step is to run the test suite multiple times and collect pass/fail data.

The most common method uses a test runner that outputs a JSON report per run. The runner must capture:

Example command using jest with --json and --outputFile:

jest --json --outputFile=test-results.json --maxWorkers=4

This generates a test-results.json file with an array of test objects, each containing name, status, duration, and error.

To correlate runs and detect flakiness, a script compares test results across multiple runs. A flaky test is one that has both passes and failures across runs. The simplest detection logic:

import json
from collections import defaultdict

def detect_flakiness(results_files):
    flaky_tests = defaultdict(list)
    for file_path in results_files:
        with open(file_path) as f:
            results = json.load(f)
        for test in results['testResults']:
            test_name = test['name']
            status = test['status']
            flaky_tests[test_name].append(status)
    
    return {
        test_name: test_runs
        for test_name, test_runs in flaky_tests.items()
        if len(set(test_runs)) > 1
    }

A test is flaky if it has at least one pass and one fail across runs. The script can output a list of flaky tests with their pass/fail history.

Reducing Flakiness: Diagnosis and Mitigation

Isolate Flaky Test Roots

A flaky test can fail due to timing, external dependencies, state pollution, or race conditions. The first step is to isolate the root cause.

Use --detectOpenHandles in Jest to catch unhandled promises, timers, or file descriptors:

jest --detectOpenHandles --runInBand --maxWorkers=1

This flag ensures that a test suite does not finish early due to async operations left uncompleted.

For E2E tests, capture screenshots and logs on failure. Use playwright with trace and screenshot options:

npx playwright test --trace on --screenshot on --video on

This generates a trace file and a video of the test run, enabling visual diagnosis of timing issues.

The trace file can be opened with playwright show-trace:

npx playwright show-trace trace.zip

This opens the trace in the Playwright Inspector, showing network requests, DOM changes, and user interactions.

Eliminate Timing Dependencies

Flakiness often stems from tests assuming fixed timing. A common failure is a test expecting a response within 100ms, but the server takes 150ms on a slow CI agent.

Use waitFor with dynamic conditions instead of fixed setTimeout.

Example in a test using puppeteer:

await page.waitForFunction(
  (timeout) => {
    const el = document.querySelector('#status');
    return el && el.textContent.includes('Ready') && el.offsetWidth > 100;
  },
  { timeout: 5000 }
);

This waits until the element is present, has correct text, and is rendered (not just laid out). It avoids relying on a fixed time.

For API tests, use curl with retries and backoff:

curl --retry 5 --retry-delay 2 --max-time 30 \
  --fail \
  --silent \
  --output /dev/null \
  http://api.example.com/v1/users/1

This command retries up to 5 times with 2-second delays between attempts, and fails on any HTTP error or timeout. It is resilient to transient network issues.

Manage Test State and Isolation

Flaky tests often share state, leading to interference between test runs. Use a test database per test or a test database with transaction rollbacks.

With psql, create a fresh database for each test run:

createdb --owner=test_user test_db_${BUILD_NUMBER}

Use docker-compose to spin up a test environment with isolated services:

version: '3.8'
services:
  db:
    image: postgres:15
    environment:
      POSTGRES_DB: test_db
      POSTGRES_USER: test_user
      POSTGRES_PASSWORD: test_pass
    ports:
      - "5432:5432"
    volumes:
      - db-data:/var/lib/postgresql/data
  api:
    build: .
    depends_on:
      - db
    environment:
      DATABASE_URL: postgres://test_user:test_pass@db:5432/test_db
    ports:
      - "8080:8080"
volumes:
  db-data:

Run tests with:

docker-compose up -d db api
docker-compose exec api npm test
docker-compose down

This ensures a clean, consistent environment for each test run.

Reduce External Dependencies

Tests relying on external services (APIs, message queues, caches) are prone to flakiness due to network delays, latency, or service outages.

Use testcontainers to spin up real dependencies in the test environment.

Example with testcontainers in a Node.js test:

const { Container } = require('testcontainers');

const container = new Container()
  .withImage('redis:7')
  .withExposedPort(6379)
  .withCommand(['redis-server', '--appendonly', 'yes']);

const redis = await container.start();
const redisHost = redis.getHostname();
const redisPort = redis.getMappedPort(6379);

// Use redisHost and redisPort in test

This ensures Redis is running in the same test environment as the application, eliminating flakiness due to external Redis instances.

For API tests, mock external APIs with wiremock or mockserver.

Example mockserver configuration via curl:

curl -X PUT \
  -H 'Content-Type: application/json' \
  -d '{
    "httpRequest": {
      "method": "GET",
      "path": "/api/users/1"
    },
    "httpResponse": {
      "statusCode": 200,
      "body": "{\"id\": 1, \"name\": \"Alice\"}"
    }
  }' \
  http://mockserver:1080/mockserver/mapping

This sets up a mock endpoint that returns a consistent response, reducing flakiness from external service variability.

Improve Retry and Recovery Strategies

Even after reducing flakiness, some tests remain slightly unreliable. Implement retry strategies at different levels.

In Jest, configure jest.config.js to retry failed tests:

module.exports = {
  retries: 2,
  testTimeout: 10000,
  setupFilesAfterEnv: ['<rootDir>/test/setup.js'],
};

This retries each test up to two times if it fails.

For CI pipelines, use GitHub Actions with retry logic:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm ci
      - run: npm test
        env:
          NODE_ENV: test
        continue-on-error: true
        retry:
          times: 3
          delay: 30

The retry: times: 3 block retries the entire test job up to three times, with 30 seconds between attempts.

Additionally, use try-catch blocks in test code to recover from transient failures:

async function withRetry(fn, maxRetries = 3, delay = 1000) {
  let lastError;
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (error) {
      lastError = error;
      await new Promise((resolve) => setTimeout(resolve, delay));
    }
  }
  throw lastError;
}

This function wraps any async test function with retry logic, backing off exponentially.

Record and Monitor Flakiness

Flakiness reduction is not a one-time task. Establish a process to monitor flakiness over time.

Store test results in a database. Use psql to create a table:

CREATE TABLE test_runs (
  id SERIAL PRIMARY KEY,
  run_id TEXT NOT NULL,
  test_name TEXT NOT NULL,
  status TEXT NOT NULL,
  duration_ms INTEGER,
  start_time TIMESTAMPTZ,
  end_time TIMESTAMPTZ,
  metadata JSONB,
  UNIQUE(run_id, test_name)
);

Insert test results with:

psql -c "
  INSERT INTO test_runs (run_id, test_name, status, duration_ms, start_time, end_time, metadata)
  VALUES ($1, $2, $3, $4, $5, $6, $7)
" --set=run_id=abc123 --set=test_name=TestUserCreate --set=status=pass --set=duration_ms=1245 --set=start_time='2024-05-15T10:00:00Z' --set=end_time='2024-05-15T10:00:01Z' --set=metadata='{"commit": "a1b2c3d"}'

A nightly job can then analyze flakiness and send alerts:

#!/bin/bash


# Run tests and collect results
jest --json --outputFile=test-results.json

# Insert into database
psql -c "COPY test_runs FROM STDIN WITH (FORMAT csv)" < test-results.csv

# Detect flaky tests
python detect_flakiness.py test-results.json

# Send alert if new flaky tests
if [ -s flaky-tests.json ]; then
  echo "Flaky tests detected:" >> alert.txt
  jq -r '.[].test_name' flaky-tests.json >> alert.txt
  mail -s "Flaky Tests Detected" [email protected] < alert.txt
fi

This script runs nightly, detects flakiness, and sends alerts when new flaky tests emerge.

Common Pitfalls and Sharp Edges

Flakiness reduction is not a one-off process. It is an ongoing practice of identifying, diagnosing, and eliminating intermittent failures through precise configuration, state isolation, and monitoring. The goal is not just to reduce flakiness but to build confidence that every test run is meaningful.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.