Data Quality Frameworks

Build systematic data quality management into your data pipelines. Covers data quality dimensions, Great Expectations framework, data contracts, schema validation, data profiling, quality metrics, and the patterns that catch data problems before they reach consumers.

A data quality framework defines how to assess, monitor, and enforce consistency, accuracy, and reliability of data across systems. This applies when building pipelines that feed into analytics, operational dashboards, or machine learning models where poor input leads to broken outputs.

Define Metrics for Validation

Data quality frameworks begin with specifying what constitutes valid data for a given schema. You must define metrics such as nullability rules, field type constraints, value ranges, and cardinality requirements. These are expressed in configuration files or inline code that the validation engine evaluates.

Example SQL definition of a quality rule:

CREATE TABLE data_quality_rules (
    id SERIAL PRIMARY KEY,
    table_name VARCHAR(255),
    column_name VARCHAR(255),
    rule_type VARCHAR(50),
    rule_value TEXT,
    enabled BOOLEAN DEFAULT TRUE
);

The framework evaluates these rules against actual data during processing. A missing NOT NULL constraint can silently pass through rows with null values, leading to downstream failures in aggregation or join operations.

Implement Row-Level Checks

Row-level checks validate individual records based on business logic. For each data source, define checks that test whether a row meets expected criteria before it enters the system.

An example check using Python:

def validate_row(row):
    if not row['email'] or '@' not in row['email']:
        raise ValueError("Invalid email format")
    if row['age'] < 0 or row['age'] > 150:
        raise ValueError("Age must be between 0 and 150")

These checks often use SQL expressions or JSON configurations. The failure mode occurs when a check is too permissive or too strict, causing data to either pass incorrectly or fail unnecessarily.

Enforce Schema Compliance

Schema compliance ensures that all data conforms to predefined structural expectations. This includes checking column presence, data types, and constraints like primary keys or foreign key references.

Use a schema registry with a configuration file:

schema_version: 1
columns:
  - name: id
    type: integer
    required: true
  - name: name
    type: string
    required: true

The system compares incoming data against this schema. If a new column is added without updating the schema, it may pass validation but introduce unexpected behavior in downstream consumers.

Establish Data Freshness Requirements

Data freshness defines how recent data must be to remain valid for processing. Set time-based thresholds that determine when data becomes stale and triggers alerts or halts processing.

Example configuration:

freshness_threshold_minutes: 60

If a dataset hasn't updated in 60 minutes, the framework raises an alert. The silent failure here is when systems ignore the freshness check because it's not enforced at runtime.

Set Up Automated Monitoring

Automated monitoring runs checks periodically and reports results to operators or automated systems. This includes running validation scripts, comparing data against historical baselines, and generating alerts.

Example monitoring script:

#!/bin/bash
psql -c "SELECT COUNT(*) FROM users WHERE created_at > NOW() - INTERVAL '1 hour';"

Alerting can be configured via webhook endpoints or database triggers. The sharp edge here is that alerting systems often misfire due to temporary network issues or transient failures, leading to noise in the monitoring system.

Handle Data Profiling

Data profiling involves analyzing datasets to understand their characteristics and identify anomalies. This includes calculating statistics like mean, median, distinct counts, and detecting outliers.

Use a profiling query:

SELECT 
    COUNT(*) as total_rows,
    COUNT(email) as non_null_emails,
    COUNT(DISTINCT department) as unique_departments
FROM employees;

Profiling helps detect data drift over time. Without periodic profiling, you might not notice that a column's distribution changes significantly, leading to incorrect assumptions in analytics.

Create Validation Workflows

Validation workflows combine multiple checks into a sequence of operations that run on batches or streams of data. These are often implemented using SQL queries or custom scripts.

Example workflow structure:

steps:
  - name: null_check
    sql: "SELECT COUNT(*) FROM table WHERE column IS NULL"
  - name: range_check
    sql: "SELECT COUNT(*) FROM table WHERE value < 0 OR value > 100"

Each step must return a pass/fail status. If any step fails, the entire workflow stops or logs an error. The failure mode is when workflows don't stop on first failure but continue, masking issues in early stages.

Manage Data Quality Reports

Reports summarize validation results and help teams understand data health. These are typically generated from logs or database tables that store check outcomes.

Example report generation:

SELECT 
    rule_name,
    failed_count,
    timestamp,
    status
FROM data_quality_logs
WHERE timestamp > NOW() - INTERVAL '1 day'
ORDER BY timestamp DESC;

Reports should include failure details, affected rows, and timestamps. If reports are generated asynchronously, they may lag behind real-time changes, causing delays in issue detection.

Implement Data Quality Scores

Data quality scores aggregate multiple metrics into a single number representing overall dataset health. These scores help prioritize issues and track improvements over time.

Example scoring logic:

def calculate_quality_score(failures):
    total_checks = len(failures)
    failed_checks = sum(1 for f in failures if not f.passed)
    return (total_checks - failed_checks) / total_checks * 100

Scores are usually expressed as percentages. A score below 90% may trigger automatic remediation or escalation. The sharp edge is that scores can be misleading if some checks carry more weight than others without clear documentation.

Configure Alerting Rules

Alerting rules define when and how to notify stakeholders about data quality issues. These rules are typically configured in a configuration file or database table.

Example alert rule:

alert_rules:
  - name: critical_nulls
    condition: "failed_rows > 100"
    severity: "critical"
    recipients: ["[email protected]"]

Alerts can be sent via email, Slack, or HTTP hooks. The failure mode is when alerts are too noisy (firing on transient issues) or too quiet (missing real problems).

Handle Data Quality Failures

When data fails quality checks, the system must decide how to proceed. Options include halting processing, logging failures, or allowing the data through with warnings.

Example handling logic:

if not validation_passed:
    if rule.severity == "critical":
        raise Exception("Data failed critical validation")
    else:
        log.warning("Data failed non-critical validation")

The decision affects downstream consumers. If you allow bad data through, it can corrupt analytics or cause model drift. If you halt processing, it can create backlogs or require manual intervention.

Version Control Quality Definitions

Quality definitions should be version-controlled alongside code and configuration files. Changes to rules must be reviewed and tested before deployment.

Example Git workflow:

git add data_quality_rules.yaml
git commit -m "Update email validation rule"
git push origin main

This ensures that changes to quality requirements are tracked, reversible, and auditable. The failure mode is when definitions change without proper review or rollback procedures.

Define Data Quality Policies

Policies govern how quality checks are applied across different data sources and teams. These policies specify who owns what rules, how often checks run, and what actions follow failures.

Example policy configuration:

policy:
  owner: "data-engineering-team"
  frequency: "daily"
  retention_days: 30

Policies ensure consistency across projects. Without them, teams may implement inconsistent validation approaches, making data integration harder.

Integrate with Data Catalogs

Data quality frameworks integrate with catalogs to provide visibility into which datasets have which rules applied. This allows users to understand the quality characteristics of any table or view.

Example catalog entry:

dataset: "sales_transactions"
quality_rules: ["not_null", "date_range"]
last_validated: "2024-05-15T10:00:00Z"

Catalog integrations help teams make informed decisions about data usage. The failure mode is when catalogs are out of sync with actual validation rules, leading to confusion.

Track Quality Improvements

Tracking improvements over time shows whether quality efforts are successful. This includes measuring how often checks pass, how many issues are fixed, and how quickly problems are detected.

Example tracking query:

SELECT 
    date_trunc('week', timestamp) as week,
    COUNT(*) as total_checks,
    SUM(CASE WHEN passed = true THEN 1 ELSE 0 END) as passed_checks
FROM data_quality_logs
GROUP BY week
ORDER BY week;

Tracking helps justify investments in quality infrastructure. The sharp edge is that tracking systems may not capture all relevant information, leading to incomplete views of progress.

Implement Rollback Procedures

When quality issues are found, rollback procedures should exist to revert changes or reprocess data. These are typically implemented as scripts or database transactions.

Example rollback:

BEGIN;
DELETE FROM validated_data WHERE run_id = '2024-05-15';
INSERT INTO validation_log VALUES ('rollback', '2024-05-15', 'complete');
COMMIT;

Rollbacks must be tested regularly to ensure they work correctly. The failure mode is when rollbacks are not properly documented or tested, leaving teams unable to recover from data quality issues.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.