Data Lake Governance

Production engineering guide for data lake governance covering patterns, implementation strategies, and operational best practices.

Data lake governance is the process of managing data quality, access control, metadata consistency, and lineage for data stored in a lake architecture. It matters when data teams need to ensure that multiple teams can safely share and consume datasets without breaking downstream systems or violating compliance requirements.

Establishing Access Control

Access control for data lakes requires both storage-level permissions and metadata-based policies. The most common approach uses POSIX file permissions combined with object storage ACLs.

For S3-compatible systems, use aws s3api put-bucket-acl to configure bucket-level access:

aws s3api put-bucket-acl \
  --bucket my-data-lake \
  --grant-read-acp uri="http://acs.amazonaws.com/groups/global/AllUsers" \
  --grant-write-acp uri="http://acs.amazonaws.com/groups/global/AllUsers"

Storage systems often require explicit permissions on individual objects. The command aws s3api put-object-acl sets access control for specific files:

aws s3api put-object-acl \
  --bucket my-data-lake \
  --key data/2024/01/sales.parquet \
  --grant-read uri="http://acs.amazonaws.com/groups/global/AllUsers"

A common failure occurs when permissions are applied at the bucket level but not propagated to existing objects. The system silently accepts the configuration but fails to enforce access control on pre-existing files.

Managing Data Quality Rules

Data quality rules can be enforced through metadata annotations and schema validation. Define schema requirements using JSON Schema files stored alongside data.

Create a schema file sales_record.json:

{
  "type": "object",
  "properties": {
    "customer_id": {"type": "string"},
    "amount": {"type": "number", "minimum": 0},
    "timestamp": {"type": "string", "format": "date-time"}
  },
  "required": ["customer_id", "amount", "timestamp"]
}

Validate data against this schema using jq:

jq -f sales_record.json data/2024/01/sales.parquet

The tool returns exit code 1 if validation fails, and error messages indicating which fields violated the schema.

Implementing Metadata Standards

Metadata management requires consistent naming conventions and metadata tags. The most reliable approach uses standard object storage tagging.

Add metadata to files using aws s3api put-object-tagging:

aws s3api put-object-tagging \
  --bucket my-data-lake \
  --key data/2024/01/sales.parquet \
  --tagging 'TagSet=[{Key=team,Value=finance},{Key=data-classification,Value=confidential}]'

Query tagged objects using aws s3api list-objects-v2 with prefix filtering:

aws s3api list-objects-v2 \
  --bucket my-data-lake \
  --prefix data/2024/01/ \
  --query 'Contents[?contains(Key, `.parquet`)]'

The most frequent failure mode occurs when metadata tags are inconsistently applied. Some teams use different tag keys or values than others, making automated queries unreliable.

Enforcing Data Lineage

Data lineage requires tracking source systems and transformations. The simplest approach uses file system metadata to record creation and modification information.

Record lineage in a JSON file lineage.json:

{
  "source": "sales-system",
  "target": "data/2024/01/sales.parquet",
  "transformations": [
    {
      "type": "ETL",
      "tool": "Apache Spark",
      "timestamp": "2024-01-15T10:30:00Z"
    }
  ]
}

Store this file alongside the data:

aws s3 cp lineage.json s3://my-data-lake/data/2024/01/sales.parquet.lineage

Use aws s3api head-object to retrieve lineage information:

aws s3api head-object \
  --bucket my-data-lake \
  --key data/2024/01/sales.parquet.lineage

A critical failure occurs when lineage records are not updated during transformation. The system continues to reference stale source files, leading to incorrect data consumption.

Creating Data Retention Policies

Data retention policies must specify how long data stays in the lake and when it should be archived or deleted. Most storage systems support lifecycle rules.

Configure retention using aws s3api put-bucket-lifecycle-configuration:

aws s3api put-bucket-lifecycle-configuration \
  --bucket my-data-lake \
  --lifecycle-configuration 'Rules=[{ID=archive-old-data,Status=Enabled,Filter={Prefix=data/2023/},Transitions=[{Days=30,StorageClass=STANDARD_IA},{Days=90,StorageClass=GLACIER}]}]'

The system automatically transitions objects based on age. A failure occurs when rules are not properly validated before deployment, causing data to remain in expensive storage classes longer than expected.

Monitoring Compliance

Compliance monitoring requires automated checks against access policies and data sensitivity. The approach uses a combination of access logs and metadata verification.

Query access logs using aws s3api list-objects-v2 with a custom prefix:

aws s3api list-objects-v2 \
  --bucket my-data-lake \
  --prefix data/2024/01/sales.parquet \
  --query 'Contents[?contains(Key, `.parquet`)]'

Verify data classification using jq against stored metadata:

aws s3api get-object-tagging \
  --bucket my-data-lake \
  --key data/2024/01/sales.parquet \
  | jq '.TagSet[] | select(.Key=="data-classification")'

The most common failure happens when compliance checks are not run regularly. Access permissions may change without corresponding metadata updates, causing systems to process unauthorized data.

Handling Data Ownership

Data ownership requires explicit assignment of responsibility for datasets. The system uses a combination of file permissions and metadata annotations.

Assign ownership using custom tags:

aws s3api put-object-tagging \
  --bucket my-data-lake \
  --key data/2024/01/sales.parquet \
  --tagging 'TagSet=[{Key=owner,[email protected]},{Key=team,Value=finance}]'

Query ownership information using aws s3api list-objects-v2:

aws s3api list-objects-v2 \
  --bucket my-data-lake \
  --prefix data/2024/01/sales.parquet \
  --query 'Contents[?contains(Key, `.parquet`)]'

A failure mode occurs when ownership tags are not consistently applied. Teams may assume responsibility for datasets without proper metadata annotations, leading to unclear accountability.

Managing Schema Evolution

Schema evolution requires versioned schema definitions and backward compatibility checks. Store multiple versions of schemas in a dedicated directory structure.

Create versioned schemas:

mkdir -p schemas/sales/v1
mkdir -p schemas/sales/v2

Store the v1 schema as schemas/sales/v1/schema.json:

{
  "type": "object",
  "properties": {
    "customer_id": {"type": "string"},
    "amount": {"type": "number"}
  }
}

Store the v2 schema as schemas/sales/v2/schema.json:

{
  "type": "object",
  "properties": {
    "customer_id": {"type": "string"},
    "amount": {"type": "number"},
    "currency": {"type": "string"}
  }
}

Validate against specific versions using jq:

jq -f schemas/sales/v1/schema.json data/2024/01/sales.parquet

A critical failure occurs when schema evolution is not tracked. New datasets may use newer schema versions while older systems expect the old format, causing processing failures.

Implementing Data Discovery

Data discovery requires metadata indexing and search capabilities. The approach uses a combination of file system tags and external search indexes.

Index data using custom metadata:

aws s3api put-object-tagging \
  --bucket my-data-lake \
  --key data/2024/01/sales.parquet \
  --tagging 'TagSet=[{Key=description,Value="Monthly sales data for Q1"},{Key=domain,Value=sales}]'

Query indexed data using aws s3api list-objects-v2:

aws s3api list-objects-v2 \
  --bucket my-data-lake \
  --query 'Contents[?contains(Key, `.parquet`)]'

A common failure occurs when search indexes are not updated in real-time. Data becomes discoverable only after scheduled updates, creating a gap between data availability and discovery.

Setting Up Audit Trails

Audit trails require logging all access and modification operations. The system uses standard object storage access logs and custom audit records.

Enable access logging:

aws s3api put-bucket-logging \
  --bucket my-data-lake \
  --bucket-logging-status 'LoggingEnabled={TargetBucket=my-data-lake-logs,TargetPrefix=access/}'

Create audit log entries in JSON format:

{
  "timestamp": "2024-01-15T10:30:00Z",
  "user": "[email protected]",
  "action": "read",
  "resource": "data/2024/01/sales.parquet",
  "ip": "192.168.1.1"
}

The system silently fails when audit log rotation is not configured, causing disk space exhaustion and loss of historical data.

Configuring Data Validation

Data validation requires automated checks that run during ingestion or on a schedule. The approach uses simple scripts that check file integrity and metadata consistency.

Validate file integrity using md5sum:

md5sum data/2024/01/sales.parquet | cut -d' ' -f1

Compare with stored checksums in metadata files:

cat data/2024/01/sales.parquet.md5

The most frequent failure occurs when validation scripts are not properly tested. A script may accept invalid data due to incorrect error handling or missing edge case checks.

Handling Data Partitioning

Data partitioning requires consistent directory structures and naming conventions. The system uses date-based partitioning with standard directory hierarchies.

Partition data by year/month:

mkdir -p data/2024/01/

Store files in the appropriate partition:

aws s3 cp sales.parquet s3://my-data-lake/data/2024/01/sales.parquet

Query partitions using standard file system operations:

ls data/2024/01/

A critical failure occurs when partitioning conventions are not enforced. Teams may store data in different formats or locations, making automated queries and processing impossible.

Managing Data Access Requests

Access request management requires a process for tracking who needs what data and when. The system uses simple metadata-based workflows.

Request access by adding a tag:

aws s3api put-object-tagging \
  --bucket my-data-lake \
  --key data/2024/01/sales.parquet \
  --tagging 'TagSet=[{Key=access-requested,Value=true},{Key=requestor,[email protected]}]'

Track requests using aws s3api list-objects-v2:

aws s3api list-objects-v2 \
  --bucket my-data-lake \
  --query 'Contents[?contains(Key, `.parquet`)]'

The system silently accepts requests without validation. Requests may not be properly reviewed or approved, leading to unauthorized data access.

Creating Data Provenance Records

Data provenance records require detailed tracking of where data came from and how it was processed. The approach uses metadata annotations that capture transformation details.

Record provenance in a JSON file:

{
  "dataset": "sales.parquet",
  "source": "sales-system",
  "transformations": [
    {
      "type": "ETL",
      "tool": "Apache Spark",
      "timestamp": "2024-01-15T10:30:00Z",
      "version": "3.4.1"
    }
  ]
}

Store this file with the dataset:

aws s3 cp provenance.json s3://my-data-lake/data/2024/01/sales.parquet.provenance

The most common failure occurs when provenance records are not updated. Systems continue to reference old transformation details, making data analysis and debugging difficult.

Enforcing Data Classification

Data classification requires consistent application of sensitivity labels. The system uses standard metadata tagging for classification.

Apply classification tags:

aws s3api put-object-tagging \
  --bucket my-data-lake \
  --key data/2024/01/sales.parquet \
  --tagging 'TagSet=[{Key=data-classification,Value=confidential},{Key=data-sensitivity,Value=high}]'

Query classified data using aws s3api list-objects-v2:

aws s3api list-objects-v2 \
  --bucket my-data-lake \
  --query 'Contents[?contains(Key, `.parquet`)]'

A critical failure occurs when classification is not enforced. Teams may store sensitive data without proper tags, making compliance audits difficult and potentially exposing confidential information.

Implementing Data Lineage Tracking

Lineage tracking requires capturing data flow from source to destination. The approach uses metadata annotations that track transformations.

Track lineage in a JSON file:

{
  "source": "sales-system",
  "target": "data/2024/01/sales.parquet",
  "transformations": [
    {
      "type": "ETL",
      "tool": "Apache Spark",
      "timestamp": "2024-01-15T10:30:00Z"
    }
  ]
}

Store this file with the dataset:

aws s3 cp lineage.json s3://my-data-lake/data/2024/01/sales.parquet.lineage

The system silently fails when lineage records are not maintained. The absence of proper tracking makes it difficult to understand data origins and impacts of changes.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.