Mutation Testing Guide

Mutation Testing Guide

Mutation testing is a technique for evaluating the quality of a test suite by introducing small, deliberate changes—called *mutations*—into the source code. A good test suite should detect these mutations, meaning the tests fail when a change is introduced that is expected to break the system. This guide explains how to implement and maintain mutation testing in any project, from small libraries to large monorepos, using standard tools and established patterns.

Setting Up Mutation Testing

To begin mutation testing, you need a test runner, a mutation engine, and a way to report and compare results. Use jest as the test runner, stryker as the mutation engine, and json as the output format. The process starts with a single command:

npx stryker --configFile=stryker.conf.js

The configuration file stryker.conf.js must define the following keys:

module.exports = {
  mutator: 'javascript', // uses default JavaScript mutators
  testRunner: 'jest',
  reporters: ['console', 'html'],
  coverageAnalysis: 'perTest',
  timeoutMS: 10000,
  maxConcurrentTestRunners: 4,
  mutate: ['src/**/*.js'],
  tsConfigFile: 'tsconfig.json',
  projectType: 'node'
};

The mutate key specifies which files to apply mutations to. The tsConfigFile ensures that TypeScript compilation is used to resolve types during mutation. coverageAnalysis: 'perTest' enables per-test coverage, which allows the tool to track which tests fail due to which mutations.

When stryker runs, it creates a new project directory under stryker-results/ containing a mutations.json file and a report directory with HTML reports. The report directory includes a index.html file for interactive exploration of all mutations.

Running Mutations and Interpreting Results

After the mutation run, examine the mutations.json file:

{
  "id": "12345",
  "originalFile": "src/calculator.js",
  "mutation": {
    "type": "Addition",
    "line": 12,
    "column": 5,
    "original": "return a + b;",
    "mutated": "return a - b;"
  },
  "status": "Killed",
  "testRun": {
    "total": 12,
    "passed": 11,
    "failed": 1,
    "testNames": ["addition should return sum"],
    "durationMs": 342
  }
}

Each mutation is described by its type, location, original code, and mutated code. The status field indicates whether the mutation was *killed* (tests failed) or *lived* (tests passed). The testRun object shows which tests failed and the total time to execute the test suite against the mutated code.

A mutation is *killed* when at least one test fails. A mutation is *lived* when all tests pass. A mutation is *equivalent* if it produces the same result as the original code (e.g., a + b becomes b + a), and the test suite does not detect the difference.

The stryker CLI exposes a --reporter flag to choose between console, html, and json. Use console for CI, html for interactive review, and json for integration with other tools.

Refining the Mutation Test Suite

To improve test quality, focus on mutations that are *lived*, especially those that are *equivalent* or *uncovered*. Identify these by analyzing mutations.json or by using the HTML report.

To filter for lived mutations, run:

npx stryker --reporter=json --filter=lived

This outputs only mutations where all tests pass. The --filter option supports killed, lived, equivalent, and uncovered.

To reduce noise in the test suite, add a mutationThreshold to the configuration:

module.exports = {
  // ... other keys
  mutationThreshold: {
    "high": 85,
    "low": 70
  }
};

This sets thresholds for *high* and *low* mutation scores. The mutation score is the percentage of mutations that are killed. If the score is below low, the build fails. If it falls between low and high, it triggers a warning.

Use --concurrent to run multiple mutation test runs in parallel:

npx stryker --concurrent --maxConcurrentTestRunners=8

This uses up to 8 test runners simultaneously, each running on a separate mutation. The maxConcurrentTestRunners key controls the number of parallel test runners, and the tool automatically balances the load.

Integrating Mutation Testing into CI

Integrate mutation testing into CI using a .github/workflows/mutation.yml file:

name: Mutation Testing
on: [push, pull_request]

jobs:
  mutation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 18
      - name: Install dependencies
        run: npm ci
      - name: Run mutation testing
        run: npx stryker --configFile=stryker.conf.js --reporter=console
      - name: Upload results
        uses: actions/upload-artifact@v3
        with:
          name: stryker-results
          path: stryker-results/

The stryker-results/ directory is uploaded as an artifact, allowing developers to download and inspect the full report.

To enable mutation testing on pull requests, use the --reporter=html flag and publish the report to a static site:

npx stryker --reporter=html --outputDir=docs/stryker

Then, in CI, deploy the docs/stryker directory to a web server using gh-pages or s3.

Handling Mutation Failure Modes

Mutation testing fails silently when the test suite does not detect a mutation. The most common failure mode is *false positive equivalence*: the mutation changes code, but all tests pass, even though the behavior has changed.

For example, consider:

function add(a, b) {
  return a + b;
}

A mutation replaces a + b with a - b. If the test suite only includes add(1, 2) with expected 3, it fails to detect the change, because 1 - 2 also produces 3. This mutation is equivalent but not detected.

To catch these, use coverageAnalysis: 'perTest' and examine the testRun.testNames array. If a test fails only for a specific mutation, but not others, it suggests that the test is sensitive to a particular change.

Another failure mode is *test flakiness*. A mutation may be killed in one run but not in another. Use --timeoutMS to set a generous timeout. If a test takes longer than timeoutMS, it is considered to have timed out, and the mutation is marked as *failed*.

The --verbose flag provides detailed output, showing which test runner executed which mutation, and how long each test took:

npx stryker --verbose --reporter=console

This reveals performance bottlenecks: a single test taking 30 seconds may indicate that the test suite is not optimized for mutation testing.

Advanced Mutation Configurations

For complex projects, use stryker.conf.js to define advanced settings.

To run mutation testing only on changed files in a pull request, use a premutation script:

module.exports = {
  mutate: (context) => {
    const changedFiles = context.changedFiles;
    return changedFiles.filter(file => file.endsWith('.js'));
  },
  testRunner: 'jest',
  reporters: ['console'],
  coverageAnalysis: 'perTest',
  timeoutMS: 15000
};

This allows you to run mutation testing only on files modified in the PR, reducing CI time.

Use mutator to customize mutation types. For example:

mutator: {
  name: 'javascript',
  additionalMutators: ['StringLiteral', 'VariableDeclaration'],
  options: {
    removeUnusedImports: true,
    inlineVariable: true
  }
}

This enables the StringLiteral and VariableDeclaration mutators, and configures the base JavaScript mutator to remove unused imports and inline variables.

To handle large test suites, use --batch to group mutations into batches:

npx stryker --batch --batchSize=10

Each batch is a separate test run, and the tool reports results per batch. This reduces the number of test runner processes and improves memory usage.

Diagnosing and Fixing Mutation Test Issues

When mutation testing reports a low score, start by reviewing the mutations.json file. Focus on *lived* and *equivalent* mutations.

Use grep to find all equivalent mutations:

grep '"status": "Equivalent"' stryker-results/mutations.json

Then, examine the original and mutated fields to identify patterns. For example, if many equivalent mutations involve a + b → b + a, it suggests that the test suite lacks assertions on order-dependent operations.

To fix a mutation that is not detected, write a new test that specifically targets the change. For instance, to detect a + b → b + a, add:

test('addition order matters', () => {
  expect(add(1, 2)).toBe(3);
  expect(add(2, 1)).toBe(3);
  expect(add(1, 2)).toBe(3); // order preserved
});

This test will fail when a + b becomes b + a, because add(1, 2) returns 3, but add(2, 1) also returns 3, so the order is not preserved.

Use stryker's --include and --exclude flags to narrow down the scope:

npx stryker --include=src/math.test.js --exclude=src/utils.js

This runs mutation testing only on the math.test.js file, excluding utils.js, which is useful when debugging a specific test suite.

Final Check: Mutation Test Health

Before merging a PR, run mutation testing and verify that:

• All mutations are either killed or equivalent.
• The mutation score is above the threshold.
• The test suite runs within the timeout.
• The HTML report is generated and uploaded.
• The mutations.json file is committed to the repository.

Use a check-mutation-score script in package.json:

{
  "scripts": {
    "mutation": "npx stryker --reporter=console --configFile=stryker.conf.js",
    "check-mutation-score": "npx stryker --reporter=json --filter=lived --outputDir=results && cat results/mutations.json | jq -r '.[] | select(.status == \"Lived\") | .mutation.original, .mutation.mutated' | tee lived-mutations.txt"
  }
}

This script outputs a list of all lived mutations to lived-mutations.txt, which can be reviewed manually or used as a checklist for code reviews.

Mutation testing is not a one-time setup. It evolves with the codebase. By following this guide, teams ensure that their test suites are not just comprehensive, but *critical*—they detect subtle, meaningful changes that go unnoticed by traditional testing.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.