Property-Based Testing

Find edge cases your unit tests miss with property-based testing. Covers generators, shrinking, stateful testing, and integration with conventional test suites.

Property-based testing is a method for validating software behavior by generating a large number of test cases from formal properties—statements about correct behavior that must hold for all inputs. It matters when you need to test complex, stateful, or data-driven logic where example-driven testing (like unit tests with fixed cases) is insufficient or error-prone.

Defining Properties

A property is a universal statement: *“For all inputs of type X, the output satisfies condition Y.”* The test runner automatically generates a suite of inputs and verifies the condition for each. Unlike traditional unit tests that check specific examples, property-based tests discover edge cases and corner cases that manual examples often miss.

A property is written as a function that takes input and returns a boolean. The test runner applies this function to many randomly generated inputs and reports any counterexample—input that makes the property fail.

-- Example: property that reversing a list twice yields the original list
prop_reverse_twice :: [Int] -> Bool
prop_reverse_twice xs = reverse (reverse xs) == xs

In practice, this definition is used by a testing framework that generates test data and runs the property repeatedly. The framework reports the first failing input, often shrinking it to a minimal counterexample.

Generating Inputs

Input generation is the engine of property-based testing. The system must generate valid values of a given type—integers, lists, trees, custom records, and more—based on type signatures and domain constraints.

For a list of integers, the generator produces lists of varying lengths, including empty lists, single elements, and long sequences with duplicates, negative numbers, and zeros. The generator can be customized to produce more domain-specific data: for instance, generating only valid dates, or non-empty lists with increasing values.

The most common way to define a generator is by specifying a type and a set of rules. In Haskell’s QuickCheck, generators are defined using combinators:

-- Generate a list of integers between -100 and 100
genIntList :: Gen [Int]
genIntList = list (choose (-100, 100))

-- Generate a non-empty list of unique integers
genUniqueList :: Gen [Int]
genUniqueList = nonEmpty (list (distinct (choose (-50, 50))))

When a test fails, the framework shrinks the counterexample to the smallest possible input that still fails. For a failing reverse_twice test with [1,2,3,2,1], the system may reduce it to [1,2,1], then [1,1], then [1], revealing the minimal failing case.

Running Tests

A property-based test suite is run via a command-line interface. The test runner executes all defined properties, generates inputs, and reports results.

$ quickcheck --verbose --num-tests 1000 --shrink-depth 5

The output includes:

The failure message includes the failing input, the expected output, and the actual output. The runner may also show a summary of property statistics: average test duration, number of shrinks, and distribution of input sizes.

Common Patterns in Property Writing

Idempotence

A function f is idempotent if f (f x) == f x. This pattern checks that repeated application of a function stabilizes.

prop_idempotent :: (Int -> Int) -> Int -> Bool
prop_idempotent f x = f (f x) == f x

Useful for normalization, caching, or state updates.

Invariants

An invariant is a condition that must remain true across operations. For example, a stack must always have the same number of elements after push and pop.

prop_stack_invariant :: [Int] -> [Int] -> Bool
prop_stack_invariant initial ops = length (execStack initial ops) == length initial + length ops

Where execStack applies a sequence of push and pop operations.

Round-trip Properties

A pair of functions encode and decode form a round-trip if decode . encode == id.

prop_roundtrip :: String -> Bool
prop_roundtrip s = decode (encode s) == s

Common in serialization (JSON, binary, protocol buffers), configuration, and data transformation.

Equivalence

Two implementations of a function must produce the same result for all inputs.

prop_equivalence :: [Int] -> Bool
prop_equivalence xs = sort1 xs == sort2 xs

Useful when refactoring, validating algorithmic correctness, or comparing a new implementation with a known reference.

Shrinker Behavior and Failure Modes

The shrinker is critical but often misunderstood. It takes a failing test case and reduces it to a minimal counterexample. The shrinker works by applying a series of reduction rules: split lists, remove elements, decrease numbers, simplify nested structures.

A common failure is a *non-minimal* counterexample. The test runner reports a long list, but a smaller one would fail too. This indicates that the shrinker is not fully applied.

Another silent failure: shrinker over-shrinks. For instance, a test fails for [[1,2],[3,4]] but after shrinking, it passes. The shrinker reduces it to [[1],[3]], but [[1,2],[3]] also fails. This suggests a flaw in the shrinker logic, or the property is sensitive to order or nesting.

A subtle edge: shrinkers do not commute. Shrink by removing elements first, then decreasing values, versus decreasing first, then removing. The resulting minimal input depends on the order of shrink steps.

The default shrinker is often insufficient. For complex types—trees, graphs, maps—the default shrinker may miss key reductions. Manual shrinkers must be defined for better results.

instance Shrink Tree where
  shrink (Leaf x) = map Leaf (shrink x)
  shrink (Node l r) = [Node l' r | l' <- shrink l]
                    ++ [Node l r' | r' <- shrink r]
                    ++ [Node l r]  -- retain original

Configuring Test Runs

Property-based test suites are configured via a file or command-line flags. Common configuration keys:


test:
  num-tests: 2000
  shrink-depth: 10
  timeout: 10000
  seed: 42
  verbose: true
  parallel: true
  coverage: true

A test run with seed: 42 allows reproducing a failure exactly:

$ quickcheck --seed 42 --num-tests 5000

This produces the same sequence of inputs every time, enabling reproducible CI runs.

Integration with Continuous Integration

Property-based tests integrate into CI via standard test runners. A typical ci.yml workflow:

name: Property Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: stack build --test --only-dependencies
      - run: stack test --test-arguments "--num-tests 5000 --shrink-depth 10"
      - run: |
          # Extract and analyze failing test cases
          grep -A 10 "Falsifiable" test-results.txt
          # Report failure details
          cat test-results.txt | grep "Falsifiable"

The --test-arguments flag passes options to the test runner. The test output is parsed by CI tools to mark failures, show stack traces, and attach minimal counterexamples.

Common Mistakes and Silent Failures

Over-reliance on default generators

The default generator for Map String Int may produce maps with keys in random order, but the property assumes sorted keys. When the test fails, the input map is [("b", 2), ("a", 1)], but the expected result assumes ["a", "b"] order. The failure is not due to logic but to generator assumptions.

Missing properties for edge cases

A property prop_insert_sorted checks that inserting into a sorted list maintains sortedness. But it fails for [1,3,2]—a list that is not sorted. The test runner shows the failing input, but the reader assumes the list was sorted. The property must be augmented with a pre-condition:

prop_insert_sorted :: [Int] -> Int -> Bool
prop_insert_sorted xs x = sorted (insert xs x)
  where
    sorted [] = True
    sorted [x] = True
    sorted (x:y:xs) = x <= y && sorted (y:xs)

Poorly defined shrinkers

A shrinker for Tree Int may reduce Node (Leaf 10) (Node (Leaf 5) (Leaf 1)) to Node (Leaf 10) (Leaf 5), but the test fails at Node (Leaf 10) (Leaf 5) and passes at Node (Leaf 10) (Leaf 1). The shrinker should also shrink values in leaves.

Misunderstanding test count vs. test case count

A test suite runs num-tests: 1000, but the property prop_reverse_twice takes 10 seconds to complete. The test runner reports 1000 tests, but it actually runs 1000 * 10 = 10,000 test cases. The reader expects 1000 test cases, but the actual workload is ten times higher.

Tools and Ecosystem

The primary tool for property-based testing is QuickCheck, a Haskell library. It provides Gen, Property, Arbitrary, and Shrink instances for all common types.

Other tools include:

All tools share a core pattern: define properties as functions, generate inputs via combinators, run tests with configurable options, and report counterexamples with shrinkers.

No tool is unique to a platform. The patterns apply across languages and systems, from embedded devices to web services.

Summary of Best Practices

Property-based testing is not a replacement for traditional testing—it is its complement. Use it to validate correctness at scale, uncover subtle bugs, and provide confidence in complex systems.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.