Data Lakehouse Architecture

Production-grade guide to data lakehouse architecture covering architecture patterns, implementation strategies, testing approaches, and operational best practices for enterprise engineering teams.

A data lakehouse is a storage architecture that combines the scalability of object storage with the query performance and transactional capabilities of a data warehouse. It enables analytics workloads on raw data while maintaining ACID transactions for operational data processing.

Creating a Lakehouse Structure

To set up a lakehouse, you must define the physical storage layout and metadata schema. The root directory structure typically contains:

/lakehouse/
  /raw/              # Raw data files
  /bronze/           # Cleaned but unprocessed data
  /silver/           # Processed and validated data
  /gold/             # Business-ready aggregates
  /metadata/         # Schema and lineage information
  /system/           # System-managed files

The system requires a catalog to track schema evolution. The simplest implementation uses a relational database with tables for:

CREATE TABLE lakehouse_objects (
    name VARCHAR(255) PRIMARY KEY,
    path TEXT NOT NULL,
    schema_json TEXT,
    created_at TIMESTAMP,
    updated_at TIMESTAMP
);

CREATE TABLE lakehouse_partitions (
    object_name VARCHAR(255),
    partition_key VARCHAR(255),
    partition_value VARCHAR(255),
    file_path TEXT,
    created_at TIMESTAMP
);

Managing Data Ingestion

Data ingestion into a lakehouse requires careful handling of file formats and schema evolution. The recommended approach is to use Parquet files with explicit schema definitions.

When writing data, always specify the schema explicitly:

# Example using spark-submit
spark-submit \
  --conf spark.sql.adaptive.enabled=true \
  --conf spark.sql.adaptive.coalescePartitions.enabled=true \
  --conf spark.sql.parquet.writeLegacyFormat=false \
  --class com.example.DataWriter \
  data-lakehouse-app.jar \
  --output-path s3a://my-bucket/lakehouse/raw/events/ \
  --schema-file schema.json

Schema evolution is handled through versioned schema files stored alongside the data. When processing new data, always validate against the current schema:

# Example ingestion logic in Python
import json
from pyspark.sql import SparkSession

def write_with_schema(df, path, schema_path):
    with open(schema_path) as f:
        schema = json.load(f)
    
    # Validate schema matches
    df.write \
      .mode("append") \
      .format("parquet") \
      .option("path", path) \
      .save()

Querying Lakehouse Data

Queries against lakehouse data should use a SQL engine that supports ACID transactions and schema enforcement. The most common approach uses Spark SQL with explicit partition pruning.

To query data efficiently, always specify partition filters:

SELECT COUNT(*) FROM lakehouse_table
WHERE date_partition >= '2023-01-01'
  AND date_partition < '2023-02-01';

When running queries, ensure proper resource allocation:

# Example spark-submit for querying
spark-submit \
  --conf spark.sql.adaptive.enabled=true \
  --conf spark.sql.adaptive.coalescePartitions.enabled=true \
  --conf spark.executor.memory=4g \
  --conf spark.driver.memory=2g \
  --conf spark.sql.execution.arrow.pyspark.enabled=true \
  --class com.example.QueryRunner \
  query-app.jar \
  --query "SELECT * FROM lakehouse_table WHERE date_partition = '2023-01-01'"

Schema Evolution and Compatibility

Schema evolution is managed through explicit schema versioning. When adding fields, always create a new schema file:

{
  "version": "2.0",
  "fields": [
    {"name": "id", "type": "integer"},
    {"name": "name", "type": "string"},
    {"name": "email", "type": "string"},
    {"name": "created_at", "type": "timestamp"}
  ]
}

When reading data, always validate against the expected schema:

# Example schema validation
from pyspark.sql.types import StructType, StructField, StringType, IntegerType

expected_schema = StructType([
    StructField("id", IntegerType(), True),
    StructField("name", StringType(), True),
    StructField("email", StringType(), True),
    StructField("created_at", StringType(), True)
])

df = spark.read \
    .schema(expected_schema) \
    .parquet("s3a://bucket/lakehouse/raw/events/")

Data Quality and Validation

Data quality is enforced through explicit validation steps. Each data layer should have defined quality rules:

-- Example quality checks in SQL
CREATE TABLE data_quality_checks (
    check_name VARCHAR(255),
    table_name VARCHAR(255),
    column_name VARCHAR(255),
    rule TEXT,
    last_run TIMESTAMP,
    result BOOLEAN
);

Partition Management

Partitioning is critical for query performance. Partitions must be managed consistently across all layers:

# Create partitioned directory structure
mkdir -p s3a://my-bucket/lakehouse/raw/events/date_partition=2023-01-01/
mkdir -p s3a://my-bucket/lakehouse/raw/events/date_partition=2023-01-02/

When writing partitioned data, always ensure consistent naming:

# Example partitioning in Python
df.write \
  .mode("overwrite") \
  .partitionBy("date_partition", "hour_partition") \
  .format("parquet") \
  .save("s3a://bucket/lakehouse/raw/events/")

Metadata Management

Metadata is stored separately from data to enable fast schema lookups. The metadata system must support:

-- Metadata table structure
CREATE TABLE lakehouse_metadata (
    object_name VARCHAR(255) PRIMARY KEY,
    current_schema TEXT,
    partition_columns TEXT[],
    last_modified TIMESTAMP,
    created_at TIMESTAMP
);

Transaction Handling

ACID transactions are essential for maintaining data consistency. When writing to the lakehouse, always use transactional writes:

# Example transaction management with spark
spark-submit \
  --conf spark.sql.adaptive.enabled=true \
  --conf spark.sql.adaptive.coalescePartitions.enabled=true \
  --conf spark.sql.sources.partitionOverwriteMode=dynamic \
  --class com.example.TransactionalWriter \
  transaction-app.jar \
  --output-path s3a://bucket/lakehouse/silver/processed_data/

Performance Optimization

Optimize performance through proper data layout and query planning:

# Optimize query performance with spark configuration
spark-submit \
  --conf spark.sql.adaptive.enabled=true \
  --conf spark.sql.adaptive.coalescePartitions.enabled=true \
  --conf spark.sql.execution.arrow.pyspark.enabled=true \
  --conf spark.sql.files.maxPartitionBytes=134217728 \
  --conf spark.sql.parquet.blockSize=134217728 \
  --class com.example.OptimizedQuery \
  query-app.jar

Common Failure Patterns

Silent failures occur when schema mismatches are not detected:

# Common error when schema is not validated
Caused by: org.apache.spark.sql.AnalysisException: 
Cannot resolve column name "email" among [id, name, created_at]

Partition pruning fails silently when partitions are not properly maintained. Always validate that new data is written to the correct partition directories.

Configuration Best Practices

Always use explicit configuration for consistency:

# Recommended spark configuration for lakehouse operations
spark-submit \
  --conf spark.sql.adaptive.enabled=true \
  --conf spark.sql.adaptive.coalescePartitions.enabled=true \
  --conf spark.sql.parquet.writeLegacyFormat=false \
  --conf spark.sql.files.maxPartitionBytes=134217728 \
  --conf spark.sql.parquet.blockSize=134217728 \
  --conf spark.sql.execution.arrow.pyspark.enabled=true

Data Governance Considerations

Data governance in lakehouses requires explicit access controls on storage buckets and metadata databases. The system must track who can modify which layers and when changes occur.

Monitoring and Alerting

Monitor data freshness through timestamp checks:

-- Query for data freshness monitoring
SELECT 
    object_name,
    MAX(last_modified) as last_update,
    CURRENT_TIMESTAMP - MAX(last_modified) as age_hours
FROM lakehouse_metadata 
GROUP BY object_name;

Data Retention and Cleanup

Implement automatic cleanup of stale partitions:

# Example cleanup script for old partitions
find /lakehouse/raw/events/ -name "date_partition=*" -type d \
  -mtime +30 -exec rm -rf {} \;

The lakehouse architecture requires careful attention to data layout, schema management, and transaction handling to maintain consistency across the system. Each component must be explicitly configured and validated to prevent silent failures in production.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.