Data Lakehouse Architecture

Data lakehouse architecture describes a pattern that sits between a raw data lake and a structured data warehouse. It retains the low-cost, scale-out storage of a lake while adding the schema enforcement, ACID transactionality, and query-performance features that warehouses provide. This matters when an organization needs to support both exploratory data-science workloads and governed BI reporting from the same dataset, without maintaining separate copy paths or ETL pipelines for each use case.

Initialize Storage and Catalog Foundation

The storage layer is an object-store backend (S3-compatible, GCS, or on-premise object store). The catalog is a metadata database that tracks table locations, schemas, partition mappings, and version history. Below is a minimal, tooling-agnostic setup using PostgreSQL as the catalog and curl to populate a landing zone.

Provision the catalog database

psql -U postgres -c "CREATE USER lakehouse WITH PASSWORD 'changeme';"
psql -U postgres -c "CREATE DATABASE lakehouse_catalog OWNER lakehouse;"
psql -U postgres -d lakehouse_catalog -c "CREATE SCHEMA lakehouse AUTHORIZATION lakehouse;"

Ingest raw data into the object store

curl -T events.parquet http://object-store:9000/lakehouse/raw/events/$(date +%Y-%m-%d)/events.parquet

*replace object-store:9000 with your deployment's S3/GS endpoint URL.*

Register the table in the catalog

Many catalog implementations expose a REST endpoint for table registration. A typical payload shape includes the table name, schema, base location, file format, and partition keys:

curl -X POST http://catalog:18080/api/tables \
  -H 'Content-Type: application/json' \
  -d '{"name":"events","schema":"lakehouse","location":"http://object-store:9000/lakehouse/raw","format":"parquet","partition_keys":["event_date"]}'

The exact request body depends on the catalog product; the fields above illustrate the common pattern used by open-table-format projects and commercial platforms alike.

Ingest and Organize Data

Data flows from an ingestion source into a raw landing zone, where files are written in a columnar format (parquet, ORC, or feather). Subsequent steps reorganize files into partitioned directories and update the catalog so compute engines can discover them.

A common workflow looks like this:

1. Write raw event files to the landing zone via curl or your preferred ETL tool.
2. Run an optimization pass that merges small files into larger, partition-aligned parcels. This step is often a shell script or a compute-job, but the pattern is: scan the landing zone, group files by partition key values, and rewrite them to a structured path such as lakehouse/events/event_date=2024-01-01/.
3. After optimization, register or update the table metadata in the catalog so the new file locations are visible.

Sharp edge: partition discovery

If the catalog is not refreshed after files are written, query engines may return empty results without an error. The engine scans only the paths it knows about; new files in unregistered directories are invisible. Refreshing the catalog typically requires a MSCK REPAIR TABLE-equivalent call or a programmatic walk of the directory tree followed by INSERT INTO catalog.table_partitions ....

Enforce Schema and Transaction Guarantees

The distinguishing feature of a lakehouse versus a plain lake is a transactional layer that presents a consistent view of table state. In the pattern described here, this is achieved by a metadata log stored alongside the data files—a directory (commonly named _delta_log, _hudi, or iceberg depending on the implementation) that contains one record per transaction, each recording the set of files, schema version, and commit timestamp.

When a write operation occurs, the engine:

1. Commits a new record to the metadata log atomically (often via compare-and-swap on the record version).
2. Updates the catalog’s schema version pointer.
3. Leaves prior file versions in place; readers always see the snapshot corresponding to the log record they started with.

Concrete example: schema evolution via the catalog

psql -U lakehouse -d lakehouse_catalog -c "UPDATE lakehouse.tables SET schema_version = schema_version + 1 WHERE name = 'events';"

If the metadata log and catalog drift—e.g., the log was written but the catalog pointer was not updated—queries may fail with "Schema version mismatch" or "Table not found at expected location". Restoring consistency requires re-reading the latest log record and re-registering the table state in the catalog.

Query and Consume Data

Compute engines integrate with the catalog by way of a connection string or configuration block that points to the PostgreSQL instance. Once connected, the engine reads the catalog to resolve the table’s current file set and schema, then issues standard SQL against the table name.

A reader would typically configure the engine once, for example by placing a connection profile file or environment variables on the compute host, then run:

kubectl apply -f query-engine-config.yaml

After the engine is running, queries follow the usual SQL syntax:

SELECT event_id, event_timestamp, user_id
FROM lakehouse.events
WHERE event_date >= '2024-03-01'
GROUP BY user_id
ORDER BY event_count DESC;

The engine translates the table name into a catalog query, fetches the latest metadata log record, and resolves the file paths from the object-store URLs stored in that record.

Metadata Evolution and Versioning

Tables in a lakehouse evolve through explicit schema changes rather than implicit ALTER TABLE operations on the data files themselves. Adding a column typically involves:

1. Writing new data files that include the additional column (null for existing rows if the write is an append).
2. Committing a new metadata log record that references the updated file set and declares the new schema version.
3. Updating the catalog’s schema_version field so future queries include the new column.

Removing a column follows the same pattern in reverse: the metadata log record omits the column from the projected schema, and the catalog version is bumped. Readers that started before the version bump continue to see the old projection; new readers see the updated one.

What to watch for

• Schema drift without catalog update: New files carry the updated schema, but the catalog still points to the old version. Queries that filter on the new column fail with "column not found".
• Concurrent writes without serialization: If two engines commit metadata log records simultaneously without a compare-and-swap mechanism, the last writer wins and earlier commits may be lost, leading to data duplication or lost updates.
• Partial file commits: A write that crashes after updating the metadata log but before closing the output stream leaves partially written parquet files. Engines that read the footer before the data portion may report "Corrupt footer" errors.

Sharp Edges

What silently fails or catches teams off guard in a lakehouse deployment:

• Cache staleness: Query engines often cache metadata for minutes to hours. After a COPY or INSERT that updates the catalog, a running query may still read the old file list. Restarting the engine or issuing a REFRESH table_name (or the engine‑specific equivalent) resolves it.
• Partition projection mismatches: Declaring partition_keys in the catalog that do not match the actual directory naming scheme (event_date=2024-01-01 vs dt=2024-01-01) causes the engine to either skip partitions or throw "partition not found" errors.
• Format‑specific footers: Parquet files include a footer with statistics; if a file is written without a footer (e.g., by a streaming sink that writes raw bytes), engines may fail with "Unsupported format" or similar, depending on the engine’s tolerance.
• Object‑store eventual consistency: After curl -T uploads a file, the new object may not be immediately readable in all regions or endpoints. A short wait or a HEAD request before catalog registration prevents "file not found" errors.
• Overwrite vs append semantics: When the catalog flags a table as write="overwrite", the engine may delete prior files before writing new ones. If the catalog is not transactional, a crash between deletion and write leaves the table empty. Prefer write="append" for streaming ingestion paths and use the metadata log to manage snapshot isolation.

Production incidents often stem from one of the above: a team runs an optimization job that rewrites files, forgets to refresh the catalog, and spends hours debugging "why is my query returning 0 rows?" only to find the metadata log was updated but the catalog pointer was not.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.