Build scalable test data strategies for complex systems. Covers data factories, synthetic generation, database seeding, GDPR-compliant test data, and production data masking.
Test data management is the practice of creating, maintaining, and distributing consistent, realistic, and reusable data to support testing across all levels of the software lifecycle. It matters when tests fail due to missing, stale, or inconsistent data—when a test passes on a developer’s machine but fails in CI, or when integration tests reveal issues that only surface with data from a live production environment.
When writing tests, the data used to exercise the system is as important as the test logic itself. Poorly designed test data leads to flaky tests, long execution times, and difficulty reproducing bugs. The key is to define test data not as static input, but as a structured, versioned, and composable asset.
Test data should be designed around the test scope and test intent. For example, a test that validates user onboarding across multiple devices needs data that includes users with varying profiles, device types, and activity logs. The data must reflect real-world conditions, not just synthetic examples.
A test suite should define data in test data manifests—structured files that describe the expected state of the system before and after test execution. These manifests are typically in JSON or YAML and contain:
users: array of user objects with id, email, created_at, preferences, and statusorders: array of order objects with id, user_id, items, total, status, shipped_atproducts: array of product objects with id, name, price, category, in_stockA sample manifest for an e-commerce suite might look like this:
users:
- id: 1001
email: "[email protected]"
created_at: "2023-04-15T10:00:00Z"
preferences:
theme: "dark"
notifications: true
status: "active"
- id: 1002
email: "[email protected]"
created_at: "2023-04-16T14:30:00Z"
preferences: {}
status: "inactive"
orders:
- id: 2001
user_id: 1001
items:
- product_id: 3001
quantity: 2
price: 19.99
- product_id: 3002
quantity: 1
price: 45.00
total: 84.98
status: "pending"
shipped_at: null
products:
- id: 3001
name: "Wireless Headphones"
price: 19.99
category: "audio"
in_stock: true
- id: 3002
name: "Smart Watch"
price: 45.00
category: "wearables"
in_stock: falseThe manifest is consumed by a test runner that applies it to the system under test. This runner may use curl to upload data via an API, psql to seed a database, or a custom script that orchestrates multiple data sources.
Test data must be versioned alongside code. A test suite should be able to load a specific version of test data using a --data-version flag.
./run-tests.sh --data-version v1.2.3 --suite checkout-flowThe version is resolved by a data catalog, typically a directory or a Git repository. The catalog is organized by version, with each version containing a set of manifests and associated assets (e.g., seed files, fixtures, snapshot images).
data/
v1.0.0/
users.json
orders.yaml
products.sql
v1.1.0/
users.json
orders.yaml
products.sql
attachments/
order-12345.pdf
v1.2.3/
manifests/
checkout-flow.yaml
payment-verification.yaml
fixtures/
users/
1001.json
1002.jsonThe test runner reads data/version.txt to determine the current version. This file contains a single line with the version identifier.
v1.2.3When a new test data version is released, it is tagged in Git:
git tag -a v1.2.3 -m "Add test data for international shipping"
git push origin v1.2.3The version is then used to pin data in CI and deployment pipelines, ensuring that a test run on main uses the same data as the previous release.
Loading test data into the system under test requires a consistent interface. The most common pattern is to use data load scripts that execute in a predefined sequence.
A typical load script uses psql to seed a PostgreSQL database and curl to populate a REST API.
#!/bin/bash
# scripts/load-test-data.sh
set -euo pipefail
DATA_DIR="$1"
DB_NAME="${DB_NAME:-test}"
# Load users and orders from YAML into PostgreSQL
psql -d "$DB_NAME" -c "
CREATE TABLE IF NOT EXISTS users (
id BIGSERIAL PRIMARY KEY,
email TEXT NOT NULL UNIQUE,
created_at TIMESTAMPTZ NOT NULL,
preferences JSONB,
status TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS orders (
id BIGSERIAL PRIMARY KEY,
user_id BIGINT NOT NULL REFERENCES users(id),
total NUMERIC(10,2) NOT NULL,
status TEXT NOT NULL,
shipped_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE TABLE IF NOT EXISTS products (
id BIGSERIAL PRIMARY KEY,
name TEXT NOT NULL,
price NUMERIC(10,2) NOT NULL,
category TEXT NOT NULL,
in_stock BOOLEAN NOT NULL
);
"
# Convert YAML to JSON, then load into PostgreSQL
for file in "$DATA_DIR"/users.yaml "$DATA_DIR"/orders.yaml "$DATA_DIR"/products.yaml; do
table=$(basename "$file" .yaml | sed 's/-/_/g')
jq -c -r '
[
.[] | {
id: .id,
email: .email,
created_at: .created_at,
preferences: .preferences,
status: .status
}
]
' "$file" | \
psql -d "$DB_NAME" -c "
COPY $table (id, email, created_at, preferences, status)
FROM STDIN WITH (FORMAT CSV, HEADER FALSE, DELIMITER E'\t');
"
done
# Load data into REST API via curl
for file in "$DATA_DIR"/users.yaml "$DATA_DIR"/orders.yaml; do
table=$(basename "$file" .yaml | sed 's/-/_/g')
jq -c '
[
.[] | {
id: .id,
email: .email,
created_at: .created_at,
preferences: .preferences,
status: .status
}
]
' "$file" | \
curl -X POST http://api.test:3000/v1/$table \
-H "Content-Type: application/json" \
-d @-
doneThe script is idempotent: running it multiple times does not introduce duplicates. It assumes that the target system is ready and that the schema is already in place.
Test data often depends on other data. A test that verifies order fulfillment requires that the product data exists before creating orders.
The solution is data dependency resolution. A test suite maintains a dependency graph where each data set depends on one or more others.
For example, the checkout-flow test suite depends on users, products, and shipping-zones. The shipping-zones data set is used by multiple test suites and is updated independently.
To manage this, the test runner reads a dependencies.yaml file in the test data manifest:
# test/data/manifests/checkout-flow.yaml
dependencies:
users: v1.2.3
products: v1.2.3
shipping-zones:
version: v1.1.0
source: https://data.example.com/shipping-zones/v1.1.0The runner resolves dependencies by:
The dependency resolver uses curl to fetch remote data:
curl -s https://data.example.com/shipping-zones/v1.1.0 -o data/shipping-zones.yamlIt also validates that all dependencies are satisfied before executing tests.
If a required dependency is missing, the runner fails with:
ERROR: Missing test data dependency 'shipping-zones'
Expected: v1.1.0
Found: v1.0.0
Source: https://data.example.com/shipping-zones/v1.1.0This error is critical: it reveals that a test suite was run with outdated or incomplete data, and it points directly to the missing version and its source.
Test data is most valuable when shared across multiple test suites, projects, and teams.
Shared data is stored in a test data repository, a Git repository or a dedicated data service. The repository is versioned and accessible via HTTP.
Each test suite imports data using a data-import directive in its configuration.
# .test-config.yaml
data:
import:
- name: users
version: v1.2.3
from: https://data.example.com/users
- name: products
version: v1.2.3
from: https://data.example.com/products
- name: order-statuses
version: v1.0.0
from: https://data.example.com/order-statusesThe test runner fetches each named data set from its source, applying it to the system.
Shared data can be curated and validated. A data set may include a schema.json that defines its structure:
{
"type": "object",
"properties": {
"users": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": { "type": "integer" },
"email": { "type": "string", "format": "email" },
"created_at": { "type": "string", "format": "date-time" },
"preferences": { "type": "object" },
"status": { "type": "string" }
},
"required": ["id", "email", "created_at"]
}
}
},
"required": ["users"]
}The runner validates the data against the schema before loading it. If validation fails, it reports:
ERROR: Data validation failed for 'users'
- Missing required field 'status'
- Invalid email: '[email protected]' (expected: '[email protected]')
- Invalid date-time format: '2023-04-15T10:00:00' (expected: '2023-04-15T10:00:00Z')This ensures that shared data remains consistent and interoperable.
When a test fails, the data state is often the root cause. The test runner must allow inspecting the test data at runtime.
The most effective tool is data snapshots. After applying test data, the runner creates a snapshot of the system state.
# After loading test data, generate a snapshot
./run-tests.sh --data-version v1.2.3 --snapshot-dir ./snapshotsThe snapshot is a directory with:
schema.sql: current database schemadata/: all data sets in JSON and YAMLlogs/: test execution logsstate/: current state of external services (e.g., api-state.json, cache-state.json)The snapshot is stored with a timestamp:
snapshots/
2023-04-17T08:32:15Z/
schema.sql
data/
users.yaml
orders.yaml
logs/
test-run.log
state/
api-state.json
cache-state.jsonWhen a test fails, the CI system uploads the snapshot to a storage service (e.g., S3, NFS, or a shared file system). A test failure report includes a link to the snapshot, allowing engineers to:
The snapshot is also used for data drift detection. A nightly job compares the current test data version against the previous one, reporting differences:
DATA DRIFT DETECTED: v1.2.3 vs v1.2.2
- Added users:
- id: 1003
email: "[email protected]"
created_at: "2023-04-17T08:30:00Z"
- Removed products:
- id: 3003
name: "Bluetooth Speaker"
price: 29.99
- Changed order status:
- id: 2001
status: "shipped" → "delivered"This enables teams to track how test data evolves and to proactively detect issues before they break tests.
In CI, test data is managed as a build artifact. The test runner produces a test data archive that is cached and reused.
# .github/workflows/test.yml
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up test data
run: |
./scripts/build-test-data.sh --version v1.2.3
tar -czf test-data-v1.2.3.tar.gz data/
- name: Upload test data
uses: actions/upload-artifact@v3
with:
name: test-data-v1.2.3
path: test-data-v1.2.3.tar.gz
- name: Run tests
run: |
tar -xzf test-data-v1.2.3.tar.gz
./run-tests.sh --data-version v1.2.3 --suite checkout-flowThe test data archive is cached between runs. When a test job runs, it first checks if the required data version is cached. If not, it downloads the archive.
In deployment, test data is used to seed staging and production environments. A deployment script applies the current test data version to the target environment.
# deploy-staging.sh
DATA_VERSION=$(cat data/version.txt)
echo "Deploying test data version $DATA_VERSION"
curl -X POST http://staging.api.com/v1/data/seed \
-H "Content-Type: application/json" \
-d "{ \"version\": \"$DATA_VERSION\" }"The system responds with a confirmation and a status URL:
{
"status": "seeding",
"version": "v1.2.3",
"started_at": "2023-04-17T08:32:15Z",
"progress": 0.65,
"task_id": "task-7a3b2c1d"
}The deployment waits for the seeding to complete, checking the status every 10 seconds until the progress reaches 1.0.
This ensures that the deployed system starts with known, consistent data—reducing the risk of environment-specific test failures.
This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy