Data Contract Testing

Production engineering guide for data contract testing covering patterns, implementation strategies, and operational best practices.

Data contract testing verifies that data producers and consumers agree on the structure, format, and semantics of exchanged data. This process matters when systems must interoperate reliably, especially in distributed pipelines where a single field rename or type change can cascade into broken integrations.

What to Test

Schema Validation

Test that incoming data matches expected schema definitions. Tools like avro-tools validate Avro schemas against JSON data:

avro-tools validate schema.avsc data.json

When validation fails, the tool returns an error message indicating which field failed and why.

Use jsonschema to validate JSON documents against a schema:

jsonschema --schemafile schema.json --instance data.json

The tool outputs specific path and type mismatches.

Field Presence and Types

Ensure required fields exist and have correct types. For example, in a pipeline that consumes user events, test that user_id is present and numeric:

SELECT COUNT(*) FROM events WHERE user_id IS NULL OR user_id NOT REGEXP '^[0-9]+$';

If data lacks required fields, queries like this may silently return zero rows or crash downstream consumers.

Data Range and Format Constraints

Validate that values fall within acceptable ranges or formats. For date fields:

SELECT COUNT(*) FROM events WHERE event_timestamp < '2020-01-01' OR event_timestamp > NOW();

For numeric limits:

SELECT COUNT(*) FROM transactions WHERE amount < 0 OR amount > 999999.99;

Nullability and Default Values

Check that optional fields behave correctly when absent. A query testing for default values might look like:

SELECT COUNT(*) FROM users WHERE created_at IS NULL;

If a system expects a default timestamp but gets NULL, downstream logic may fail silently.

How to Implement

Using Command-Line Tools

For Avro data validation, use avro-tools with schema files and JSON samples:

avro-tools validate user-schema.avsc user-data.json

If the schema is invalid, the tool returns an error like:

Error: Schema validation failed: Missing required field 'email'

For JSON schemas, use jsonschema:

jsonschema --schemafile event-schema.json --instance event-data.json

Errors are verbose and include the specific path to the failing value.

Integration with CI/CD

Integrate contract tests into your build pipeline. Example using a shell script:

#!/bin/bash
set -e

for file in data/*.json; do
  jsonschema --schemafile schema.json --instance "$file"
done

This ensures that all new data files conform to the contract before deployment.

Testing with SQL Databases

Use SQL queries to validate schema compliance in database tables. For example, checking that a column is not null:

SELECT COUNT(*) FROM orders WHERE customer_id IS NULL;

If no rows are returned, the test passes; otherwise, it fails.

Testing Strategies

Snapshot Testing

Take snapshots of known good data and compare against new inputs. Use tools like jq to extract specific fields for comparison:

jq '.[] | {id, name}' data.json > snapshot.json

Then validate that future data matches this structure.

Negative Testing

Test edge cases such as missing fields or malformed values. For example, test what happens when a required field is omitted:

jq 'del(.email)' user.json > invalid-user.json
jsonschema --schemafile user-schema.json --instance invalid-user.json

If the schema doesn't require email, this may pass silently, but it's still important to validate expectations.

Incremental Contract Updates

When updating a contract, test both backward compatibility and new behavior. For example, if a new field is added:

# Test old data with new schema
jsonschema --schemafile updated-schema.json --instance old-data.json

If the schema allows optional fields, this may pass, but it's worth verifying that the new field behaves as expected.

Common Failure Modes

Silent Data Loss

A missing email field in a user record may not cause an immediate error, but downstream systems may silently ignore the user. This can be caught by:

SELECT COUNT(*) FROM users WHERE email IS NULL;

Schema Mismatch Errors

When data doesn't match schema expectations, validation tools return specific error messages. For example:

Error: Validation failed at path $.user.profile.name: Expected string, got number

These errors must be surfaced in logs and alerts.

Type Coercion Issues

A string field like age may be expected to contain numeric values. If a consumer tries to parse it as a number and fails:

jq '.age | tonumber' data.json

This can produce an error like:

jq: error (at <stdin>:1): number() requires numeric input, got string

Missing Required Fields

If a required field like timestamp is missing, a validation tool might return:

Error: Missing required field 'timestamp'

This is critical to catch during testing, as missing timestamps can break downstream logic.

Configuration and Setup

Schema Repository

Keep schemas in version-controlled files. For example, store Avro schema definitions in schemas/user.avsc. This ensures consistency across environments.

Validation Rules

Define validation rules in configuration or code. Example in Python:

import jsonschema
from jsonschema import validate

def validate_user(data):
    schema = {
        "type": "object",
        "properties": {
            "name": {"type": "string"},
            "email": {"type": "string", "format": "email"}
        },
        "required": ["name", "email"]
    }
    
    try:
        validate(instance=data, schema=schema)
        return True
    except jsonschema.exceptions.ValidationError as e:
        print(f"Validation error: {e.message}")
        return False

Test Coverage

Ensure that all data sources are tested against their contracts. This includes:

Monitoring and Alerts

Continuous Validation

Set up monitoring to validate data streams continuously. For example, a periodic check for schema compliance:

while true; do
  jsonschema --schemafile schema.json --instance latest-data.json
  sleep 300
done

Alerting on Failures

Configure alerts when validation fails. For instance, if the tool returns a non-zero exit code, trigger a notification.

Production Considerations

Graceful Degradation

In production, some data may not strictly match contracts but should still be processed. Implement fallback logic:

SELECT * FROM events WHERE event_timestamp IS NOT NULL;

This allows processing even if some fields are missing or malformed.

Logging Validation Errors

Log validation failures to help debug issues. For example:

jsonschema --schemafile schema.json --instance data.json 2>&1 | tee validation.log

This ensures that errors are captured and searchable.

Rollback Strategy

If a new contract breaks existing data, have a rollback plan. This might include:

Testing Best Practices

Early Detection

Run contract tests early in the pipeline to catch issues before they propagate. This includes:

Comprehensive Coverage

Ensure all data paths are tested, including:

Clear Error Reporting

Validation tools should provide clear error messages that indicate what went wrong and where. For example:

Error: Expected field 'user_id' to be numeric, got string "abc123"

This helps developers quickly identify and fix issues.

Versioned Contracts

Use versioned schemas to track changes over time. This allows you to validate data against the correct schema for its creation date.

Automated Testing

Integrate contract validation into automated testing workflows. This ensures that contracts are enforced consistently across environments and pipelines.

Contract testing is not about perfect data—it's about ensuring predictable, agreed-upon interfaces between systems. When a system fails to meet its contract, it must fail fast and clearly, not silently corrupt data or produce incorrect results.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.