Visual Regression Testing | The Garnet Wiki

Visual regression testing is the process of automatically comparing rendered web or mobile interfaces across builds, ensuring that UI changes are intentional and consistent. It matters most when a team ships UI updates frequently and must catch subtle visual drift—such as font rendering differences, spacing shifts, or component layout bugs—before they reach users.

Defining the Visual Baseline

Visual regression testing begins with establishing a baseline: a set of reference images captured from a fully rendered UI. These images serve as the "truth" against which all future builds are compared.

The baseline is typically generated by running a test suite that renders each page or component in a controlled environment. This environment must include:

A fixed viewport size (e.g., 1920x1080)
A consistent set of fonts and assets
A fully hydrated DOM with all JavaScript executed
A stable network state (e.g., all assets loaded, no pending requests)

To generate the baseline, a test runner launches a headless browser (like Puppeteer or Playwright) and captures screenshots of key UI states. The most common workflow uses a script that:

Launches the application in a Docker container
Runs the test suite via a command-line interface
Takes screenshots and saves them in a directory structure that mirrors the test suite’s test cases


npx playwright test --config=playwright.config.ts --reporter=html

The playwright.config.ts file configures the test environment:

// playwright.config.ts
export default {
  use: {
    viewport: { width: 1920, height: 1080 },
    screenshot: { fullPage: true, timeout: 30000 },
    trace: 'on-first-retry',
  },
  projects: [
    {
      name: 'visual-baseline',
      testDir: 'tests/visual',
      outputDir: 'test-results/visual-baseline',
      expect: {
        timeout: 30000,
      },
    },
  ],
};

The resulting screenshots are stored in test-results/visual-baseline/, with a file naming convention like:

test-results/visual-baseline/checkout-page-1920x1080.png
test-results/visual-baseline/product-card-1920x1080.png

Running Visual Regression Tests

After the baseline is established, visual regression tests run on every commit or deployment. The test runner fetches the latest UI build, renders it, and captures a new screenshot.

The test compares the new screenshot to the corresponding baseline image using pixel-by-pixel comparison algorithms. The most common method is difference-based comparison, where a delta image is computed by subtracting the baseline from the current image.

The comparison can be configured with two primary parameters:

Threshold: the maximum allowed pixel difference (e.g., 5% of pixels differ by more than 10 units of color)
Tolerance: a threshold applied per pixel (e.g., a pixel is considered different if its RGB values differ by more than 10 in any channel)

// Example: configuration for visual comparison
{
  "visual": {
    "threshold": 0.05,
    "tolerance": 10,
    "include": [
      "checkout-page",
      "product-card"
    ],
    "exclude": [
      "admin-dashboard",
      "mobile-menu"
    ]
  }
}

The test runner evaluates each test case:

If the difference is below the threshold, the test passes.
If the difference exceeds the threshold, the test fails and generates a diff image.

A diff image is a side-by-side view of the baseline and current screenshots, with red pixels marking differences. It also overlays a heatmap showing the severity of deviation.

Handling Test Failures

When a visual regression test fails, the primary challenge is diagnosing *what* changed and *why*. The failure is not just a pass/fail result—it is a visual artifact.

The most common failure mode is unexpected visual drift. This occurs when a test fails due to a change that was not intentional, such as:

A font loading delay causing a layout shift
A CSS class being applied inconsistently due to a missing media query
A component rendering differently due to a data structure change

The test runner reports the failure as:

{
  "test": "checkout-page-1920x1080",
  "status": "failed",
  "actual": "test-results/visual-baseline/checkout-page-1920x1080.png",
  "expected": "baseline/checkout-page-1920x1080.png",
  "diff": "test-results/visual-baseline/checkout-page-1920x1080-diff.png",
  "details": {
    "diffThreshold": 0.05,
    "diffTolerance": 10,
    "pixelDifference": 12876,
    "totalPixels": 2073600,
    "failureRate": 0.0621,
    "reason": "Unexpected layout shift in the shipping address section"
  }
}

The diff image is essential. It reveals not just *where* the change occurred, but also *how* significant the change is. A red halo around a button indicates a subtle border color change. A filled rectangle in the header suggests a new background image was not loaded.

To aid debugging, the test runner often saves the full browser trace (recorded via Playwright or Puppeteer) and the DOM snapshot at the time of the screenshot.

Integration with CI/CD

Visual regression testing is most effective when integrated into a continuous integration pipeline. The typical setup runs on every pull request, with a dedicated CI job.

# .github/workflows/visual-regression.yml
name: Visual Regression
on: [pull_request]

jobs:
  visual:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm install
      - run: npm run build
      - run: npx playwright test --config=playwright.config.ts --reporter=html
      - uses: actions/upload-artifact@v4
        with:
          name: visual-regression-results
          path: test-results/visual-baseline/
          if-no-files-found: ignore
      - run: |
          if [ -f "test-results/visual-baseline/failure-summary.json" ]; then
            echo "Visual regression test(s) failed."
            exit 1
          fi

The CI job captures all test results and uploads the screenshots, diff images, and HTML report as artifacts. The PR is automatically marked with a status check that includes a link to the report.

When a test fails, the PR description is updated with the diff image and a summary of changes. The reviewer can click through to see the exact difference.

Common Pitfalls and Sharp Edges

Several subtle issues often go undetected until they cause production problems.

1. Time-Dependent Rendering

The most common silent failure is time-dependent rendering. A test may pass on a fast machine but fail on CI, where network latency or CPU load causes timing differences.

A setTimeout in a component may cause a popup to appear after 500ms, but the test captures the screenshot immediately after page.goto(), missing the animation.

The solution is to wait for a specific element or condition before capturing the screenshot:

await page.waitForSelector('#checkout-complete', { timeout: 10000 });
await page.screenshot({ path: 'screenshots/checkout-complete.png', fullPage: true });

2. Font and Resource Loading

Fonts and assets may not be ready at the time of screenshot capture. The test must wait for all resources:

await page.waitForLoadState('networkidle');
await page.screenshot({ path: 'screenshots/product-page.png', fullPage: true });

Without networkidle, the layout may shift as fonts load, introducing visual drift that is not part of the design.

3. Overlapping and Misconfigured Thresholds

A frequent mistake is confusing threshold and tolerance. A threshold of 0.05 means that 5% of the total pixels must differ by more than the tolerance value.

But if the tolerance is set too high (e.g., 50), small changes like a 1px border or a 5% color shift may go unnoticed.

Conversely, a low threshold with high tolerance can reject minor changes that are acceptable.

The correct approach is to tune the thresholds per component: a high-precision component like a chart may use a threshold of 0.01 and tolerance of 5, while a simple form may use 0.05 and 15.

4. Baseline Drift and Regression Bloat

Over time, the baseline image becomes outdated. A component that was once static now includes a dynamic loading state. The test passes, but the baseline no longer represents the intended state.

To mitigate this, teams periodically rebaseline all test cases:

npx playwright test --rebaseline --config=playwright.config.ts

This updates the baseline images with the current test output. The CI job can then compare the new baseline against the old one and report changes.

5. Diff Image Interpretation

The diff image is not just a report—it is a diagnostic tool. However, reviewers often miss critical details:

A small red dot in a corner may indicate a missing icon.
A blue tint across the entire page may suggest a background color change.
A jagged edge on a button may point to a font rendering difference.

The most effective practice is to review diffs in a dedicated tool like ImageMagick or a simple web-based viewer that allows zooming and pixel inspection.

Maintaining the Test Suite

As the test suite grows, it becomes unwieldy. The solution is to categorize tests and organize them by component, feature, or user journey.

Tests are grouped into directories:

tests/visual/
├── checkout/
│   ├── checkout-page.spec.ts
│   ├── payment-methods.spec.ts
│   └── order-summary.spec.ts
├── product/
│   ├── product-page.spec.ts
│   └── product-list.spec.ts
└── user/
    ├── login.spec.ts
    └── profile.spec.ts

Each test file includes a describe block that documents the test’s purpose:

describe('Checkout Page', () => {
  beforeEach(async () => {
    await page.goto('/checkout');
    await page.waitForLoadState('networkidle');
  });

  it('should render the checkout form with all required fields', async () => {
    await expect(page).toHaveScreenshot('checkout-page.png');
  });

  it('should display the shipping cost dynamically based on the selected method', async () => {
    await page.selectOption('#shipping-method', 'express');
    await expect(page).toHaveScreenshot('checkout-page-express-shipping.png');
  });
});

This structure makes it easy to add, modify, and maintain tests. It also allows for incremental test runs: playwright test tests/visual/checkout/.

Summary

Visual regression testing is not just capturing screenshots and comparing pixels. It is a disciplined process of defining, running, and maintaining a visual quality gate. It requires attention to rendering timing, resource loading, threshold configuration, and diff interpretation. The most valuable outcome is a test suite that not only catches bugs but also documents the UI’s intended behavior—so that every change is not only functional but also visually correct.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.