Production-grade guide to data classification engineering covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.
Data classification engineering is the practice of building and maintaining systems that automatically assign, manage, and enforce data sensitivity labels across an organisation’s datasets. It matters when compliance policies—like GDPR, HIPAA, or CCPA—require consistent handling of data based on its sensitivity, and when manual classification fails at scale. This page details how to implement and operate such systems, focusing on the tasks engineers perform to keep classification accurate, discoverable, and actionable.
Classification begins with defining rules that determine how data is labeled. Rules are evaluated in a priority order, matching metadata, content, or schema patterns. A rule applies when its conditions are satisfied, and it assigns one or more classification labels.
Each rule is a configuration object with id, name, priority, and conditions. Conditions are evaluated against a record’s metadata, such as file_path, table_name, column_name, and schema. They may use match, regex, in, or contains operators.
{
"id": "rule-001",
"name": "PII in Customer Table",
"priority": 10,
"conditions": [
{
"field": "table_name",
"operator": "equals",
"value": "customer"
},
{
"field": "column_name",
"operator": "contains",
"value": "email"
},
{
"field": "column_name",
"operator": "regex",
"value": "^(first|last|full)_name$"
}
],
"labels": [
"PII",
"HighlySensitive"
]
}Rules are stored in a central registry, often a database or a configuration service. They are versioned and deployed through a CI/CD pipeline.
When a new table is added to the data warehouse, a change event triggers a rule evaluation service. The service queries the registry, applies rules in priority order, and outputs a classification map.
A common failure is misinterpreting contains vs. regex. A column named customer_first_name matches contains("name") but not regex("^name$"). Engineers often assume in is case-sensitive; it is not by default, but many systems use lower() internally.
Classification must be enforced across data ingestion, transformation, and delivery. The most effective approach integrates classification into ETL pipelines and data catalog systems.
When a new CSV file is ingested into a cloud storage bucket, a data pipeline reads the file, infers schema, and applies classification rules. The pipeline uses file_path, file_size, and file_extension as metadata, and scans the first 100 rows to detect PII patterns.
steps:
- name: "ingest-csv"
image: "dataflow/ingest:v1.2"
inputs:
- bucket: "raw-data"
prefix: "customer/"
outputs:
- dataset: "customer_staging"
config:
schema_inference: true
classification:
rules: "s3://config/classification-rules.json"
metadata:
- "file_path"
- "file_size"
- "file_extension"
content_scanning:
- "regex"
- "fuzzy"
- "ngram"The classification engine scans the data using a set of pattern rules defined in a JSON file:
{
"patterns": [
{
"name": "Email",
"regex": "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$",
"min_confidence": 0.75
},
{
"name": "SSN",
"regex": "^\\d{3}-\\d{2}-\\d{4}$",
"min_confidence": 0.9
},
{
"name": "Phone",
"regex": "^\\+?\\d{10,15}$",
"min_confidence": 0.8
}
]
}During ingestion, the engine applies the patterns to each column. A column with 92% of values matching the email regex receives a confidence: 0.92, triggering a classification: PII label.
A silent failure occurs when min_confidence is misapplied. A column with 70% matches is labeled PII only if the rule explicitly sets min_confidence: 0.7. If the field is missing, the default of 0.5 is assumed, but many engineers expect 0.8. This mismatch causes false positives in production.
The pipeline outputs a classification report as a JSON file in the dataset’s metadata:
{
"dataset": "customer_staging",
"classification": [
{
"column": "email",
"labels": ["PII", "HighlySensitive"],
"confidence": 0.96,
"sources": ["regex", "fuzzy"]
},
{
"column": "address",
"labels": ["PII"],
"confidence": 0.67,
"sources": ["regex"]
}
]
}This report is used by downstream systems to generate audit trails and enforce access policies.
A data catalog surfaces classification to users and systems. It records metadata about datasets, including their labels, provenance, and classification history.
Each dataset has a classification object in its metadata. The catalog exposes a REST API endpoint:
GET /datasets/{id}/classificationResponse:
{
"dataset_id": "customer_staging",
"version": "2023-09-14T10:30:00Z",
"labels": [
"PII",
"HighlySensitive"
],
"sources": [
{
"type": "rule",
"rule_id": "rule-001",
"applied_at": "2023-09-14T10:30:00Z"
},
{
"type": "content_scanning",
"pattern": "Email",
"confidence": 0.96,
"applied_at": "2023-09-14T10:30:00Z"
}
],
"last_updated": "2023-09-14T10:30:00Z"
}A dashboard service calls this endpoint to display a dataset’s sensitivity level, color-coded by label: PII in blue, HighlySensitive in red.
When a user queries the catalog, the service filters datasets by labels:
GET /datasets?labels=PII&labels=HighlySensitiveThis query returns only datasets classified as both PII and HighlySensitive.
A common edge case is the label drift problem. A dataset labeled PII in January may have customer_phone added in March, but the classification engine only runs monthly. The catalog shows the dataset as PII but misses the new column. To fix this, a nightly job re-evaluates all datasets against the latest rule set, and updates the last_updated timestamp.
Another failure mode: the sources array in the classification response is often misused. Engineers assume that type: "rule" means the rule was applied, but they forget to include rule_id. Without rule_id, the catalog cannot link back to the rule definition or update the rule’s version history.
Classification systems maintain metadata that tracks how, when, and why a label was applied. This metadata enables traceability and supports compliance audits.
Each classification label has a definition object that includes name, description, standard, and retention_period. The definition is versioned and associated with a label ID.
{
"label_id": "PII",
"name": "Personally Identifiable Information",
"description": "Data that can identify an individual, including name, email, address, and phone.",
"standard": "GDPR",
"retention_period": "7 years",
"version": "v2.1",
"created_at": "2023-08-01T08:00:00Z"
}The system maintains a label_registry table in a relational database:
CREATE TABLE label_registry (
label_id VARCHAR(50) PRIMARY KEY,
name TEXT NOT NULL,
description TEXT,
standard VARCHAR(50),
retention_period INTERVAL,
version VARCHAR(20),
created_at TIMESTAMPTZ NOT NULL
);
CREATE TABLE rule_evaluation (
id SERIAL PRIMARY KEY,
dataset_id VARCHAR(100) NOT NULL,
rule_id VARCHAR(100) NOT NULL,
evaluated_at TIMESTAMPTZ NOT NULL,
applied_labels JSONB NOT NULL,
confidence_score DOUBLE PRECISION,
source TEXT
);When a new rule is applied, the system inserts into rule_evaluation. A dashboard uses this table to track classification accuracy over time.
A sharp edge: the confidence_score field is often ignored. Engineers store it but do not use it to trigger alerts. A rule with confidence_score < 0.8 is marked as "low confidence" and triggers an alert to the data steward.
The system also supports label inheritance. A dataset can inherit labels from its parent, such as a project or a data domain. A rule can define inherit: true, meaning the dataset takes on the labels of its parent if not explicitly labeled.
{
"rule_id": "rule-003",
"name": "Project-level PII",
"priority": 5,
"conditions": [
{
"field": "project_id",
"operator": "equals",
"value": "marketing"
}
],
"labels": ["PII"],
"inherit": true
}When a dataset belongs to the marketing project, it automatically gets the PII label, even if no rule matches it directly. This reduces configuration overhead but introduces a hidden dependency: if the marketing project is updated, all its datasets inherit the new labels.
A failure mode: label inheritance is enabled, but the project_id field is project_id in one system and project_id_str in another. The system expects project_id but finds project_id_str. The rule evaluates correctly, but the inheritance step fails silently.
Validation ensures that classification is correct and consistent. It is performed through automated checks and manual reviews.
A validation job runs daily and checks:
PII datasets have at least one column with confidence > 0.7.version field and are applied to at least one dataset.$ ./validate-classification --since "2023-09-13T00:00:00Z"The command outputs a report:
{
"validation_run_id": "2023-09-14-01",
"started_at": "2023-09-14T01:00:00Z",
"checks": [
{
"name": "At least one label per dataset",
"status": "failed",
"details": [
{
"dataset": "user_profiles",
"expected": true,
"actual": false,
"message": "No classification labels found"
}
]
},
{
"name": "PII confidence threshold",
"status": "passed",
"details": [
{
"dataset": "customer_staging",
"min_confidence": 0.7,
"actual": 0.82
}
]
}
]
}A failed check triggers a notification via email or Slack.
Auditing classification is done by querying the rule_evaluation table and joining it with label_registry.
SELECT
d.dataset_name,
r.name AS rule_name,
r.priority,
re.confidence_score,
l.name AS label_name,
l.standard
FROM rule_evaluation re
JOIN datasets d ON d.id = re.dataset_id
JOIN rules r ON r.id = re.rule_id
JOIN label_registry l ON l.label_id = ANY(re.applied_labels)
WHERE re.evaluated_at >= '2023-09-01'
ORDER BY d.dataset_name, r.priority;This query produces a compliance audit trail showing how each dataset was classified and by which rules, and which standards each label aligns with.
A common mistake is assuming that the applied_labels array is a list of strings, but it is actually a JSON array of strings. When a rule assigns multiple labels, the system stores them as ["PII", "HighlySensitive"], but a query using IN on applied_labels fails if the column is not JSONB or if the array is stored as TEXT.
Another failure: the standard field is often used to define the source of a label, but it is not indexed. Queries to filter by standard take minutes on large catalogs. Engineers should always create a GIN index on standard and label_id.
Classification systems generate events for changes. These events are consumed by downstream systems to trigger actions.
When a dataset’s classification is updated, a message is published to a streaming platform:
{
"event_type": "classification_updated",
"dataset_id": "customer_staging",
"version": "2023-09-14T10:30:00Z",
"labels": [
"PII",
"HighlySensitive"
],
"changed_at": "2023-09-14T10:30:00Z",
"source": "rule_evaluation"
}A consumer service listens to this event and updates access policies in a data lake:
def handle_classification_updated(event):
dataset_id = event["dataset_id"]
labels = event["labels"]
updated_at = event["changed_at"]
# Update access policies based on labels
for label in labels:
policy = get_policy_for_label(label)
apply_policy_to_dataset(dataset_id, policy, updated_at)A failure mode: the event is published but not acknowledged. If the consumer crashes before acknowledging the event, the same event is processed twice. To avoid this, the system uses idempotent event processing with a event_id field and a processed_events table.
CREATE TABLE processed_events (
event_id VARCHAR(100) PRIMARY KEY,
processed_at TIMESTAMPTZ NOT NULL
);Before processing, the service checks if event_id exists. If it does, it skips the event. This prevents duplicate policy updates and ensures consistency.
Classification engineering is complete when these events are not only consumed but also validated. A health check endpoint returns the number of unprocessed events, the last processed event time, and the average processing latency.
GET /health?component=classificationResponse:
{
"component": "classification",
"status": "healthy",
"details": {
"unprocessed_events": 127,
"last_event_processed": "2023-09-14T10:30:00Z",
"average_processing_latency": "1.2s"
}
}This endpoint is used in monitoring dashboards and alerting systems to detect classification system degradation.
This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy