Combine data lake flexibility with data warehouse performance. Covers lakehouse design principles, Delta Lake, Apache Iceberg, table formats, schema evolution, time travel, and the patterns that eliminate the data lake vs. warehouse tradeoff.
A data lakehouse architecture unifies the scalability of a data lake with the reliability and performance of a data warehouse. It matters when teams need to run analytical workloads on raw, evolving data—such as streaming logs, sensor feeds, or user event sequences—while simultaneously supporting transactional operations like real-time updates and incremental processing. This architecture enables data scientists to query and train models directly on raw data, while engineers maintain consistent, versioned datasets with ACID guarantees. It is the foundation for hybrid workflows where batch, streaming, and interactive analytics coexist.
To query raw, unstructured data stored in a data lake, use a SQL engine capable of reading from object storage. The query engine must resolve schema-on-read and support partitioned, columnar formats. For example, a query over a dataset of user clickstream logs stored in Parquet files in Amazon S3:
SELECT
user_id,
COUNT(*) AS session_count,
AVG(duration_seconds) AS avg_session_duration
FROM
clickstream_logs
WHERE
date BETWEEN '2024-01-01' AND '2024-01-31'
AND event_type = 'page_view'
GROUP BY
user_id
ORDER BY
avg_session_duration DESC
LIMIT 100;The engine resolves the clickstream_logs table by scanning a manifest.json file in the table’s root directory. This manifest contains a list of Parquet files, each with metadata like file size, column statistics, and partitioning keys. The query engine reads only the necessary columns and partitions, skipping irrelevant data.
If the Parquet files are not co-located with their corresponding metadata, the engine fails silently: queries return incomplete results but do not error. The most common cause is a missing or stale manifest. A manifest file may be generated by a batch job but not updated after new data is appended. When a new partition is added—say, date=2024-02-01—the engine must be explicitly told to refresh the manifest. Without this, queries over 2024-02-01 data return no rows, even though the files exist.
To update the manifest, run a command:
aws s3 cp manifest.json s3://analytics-data/clickstream_logs/manifest.jsonThe manifest must be in a standard format:
{
"version": 1,
"files": [
{
"path": "data/2024-01-01/001.parquet",
"size": 124387,
"partitionValues": { "date": "2024-01-01" },
"columnStats": {
"user_id": { "min": 1000, "max": 5000 },
"event_type": { "distinctCount": 3 }
}
}
]
}Failure to include columnStats or partitionValues in the manifest leads to suboptimal query planning. The engine may read full columns even when filtering on a single value.
To ingest new data, write new Parquet files into a partitioned directory structure. For example, every hour, a stream processor appends new clickstream data to s3://analytics-data/clickstream_logs/date=2024-02-01/hour=14/ with a filename like data-000123.parquet.
Each write must be atomic. If a new file is written directly to the directory without atomicity, readers may see partial data. A common failure is a file being written in place—data-000123.parquet is first written to a temporary location, then moved via mv or s3 mv—but the move is not atomic. During the move, some readers see the old file, others the new one.
To ensure atomicity, use a two-phase commit:
On AWS S3, use s3 cp --only-show-affected with --expected-etag to verify the copy. On POSIX, use rename(2) with renameat2(2).
To register new data in the table’s metadata, update the manifest. The update must be atomic. A race condition occurs if two writers update the manifest simultaneously: the final manifest may contain entries from both updates but omit some. The solution is to write to a temporary manifest file, then atomically move it into place.
aws s3 cp manifest.json.tmp s3://analytics-data/clickstream_logs/manifest.json.tmp
# Update manifest atomically
aws s3 mv s3://analytics-data/clickstream_logs/manifest.json.tmp \
s3://analytics-data/clickstream_logs/manifest.json \
--expected-etag "sha256:abc123..."The --expected-etag ensures that the move only succeeds if the current manifest has the expected ETag. This prevents concurrent updates from overwriting each other.
To support concurrent updates and reads with consistency, the lakehouse must provide ACID transactions. The most common pattern is to use a transaction log stored alongside the data.
Each write operation creates a new transaction log entry in a dedicated directory, such as s3://analytics-data/clickstream_logs/_transactions/. Each entry is a JSON file with metadata about the transaction:
{
"id": "txn-12345",
"timestamp": "2024-02-01T14:30:00Z",
"operations": [
{
"type": "add_file",
"path": "data/2024-02-01/hour=14/data-000123.parquet",
"size": 124387,
"partitionValues": { "date": "2024-02-01", "hour": 14 }
},
{
"type": "remove_file",
"path": "data/2024-01-31/hour=13/data-000122.parquet"
}
],
"commitMetadata": {
"author": "stream-processor",
"description": "Ingested 2024-02-01 14:00–15:00 click events"
}
}A transaction is committed when the transaction log entry is written and the manifest is updated atomically. A reader can now see all data written in a transaction by reading the log.
To read a consistent snapshot of the data, a query engine must use the transaction log to determine the set of files visible at a point in time. The engine reads the latest transaction log entry and applies all operations from the beginning up to that transaction ID.
If the transaction log is not updated atomically—e.g., the log entry is written, then the manifest is updated, but the manifest update fails—the engine may read a partially committed state. The most common failure is a missing transaction log entry: readers see the data from a previous transaction but miss recent updates.
To detect and recover from such failures, the engine maintains a last_committed_txn file in the table root. This file contains the transaction ID of the last successfully committed transaction. When a new transaction is started, the engine checks this file and reads the full transaction log from that point.
To support real-time analytics, data must be available in the lakehouse within seconds of ingestion. This requires a streaming ingestion pipeline that writes to a staging layer—usually a log file format like Apache Iceberg or Delta Lake.
The pipeline writes data in microbatches. Each microbatch is a sequence of records serialized as a Parquet file. The pipeline emits a new file every 30 seconds, and the file is written to a staging directory:
s3://analytics-data/clickstream_logs/staging/
├── 2024-02-01/
│ └── 14/
│ └── batch-000123.parquet
└── 2024-02-01/
└── 14/
└── batch-000124.parquetA separate process, the merge engine, periodically picks up these staged files and applies them to the main table.
The merge engine performs a merge operation, combining staged files with existing data. It uses a merge command that takes a list of files to merge and a target table:
merge \
--source s3://analytics-data/clickstream_logs/staging/ \
--target s3://analytics-data/clickstream_logs/ \
--partition-by date,hour \
--on "target.user_id = source.user_id AND target.date = source.date" \
--when-match-update "SET duration_seconds = source.duration_seconds" \
--when-not-match-insert "INSERT (user_id, date, duration_seconds)"The merge operation reads all staged files, applies the merge logic, and produces new Parquet files in the main data directory. The merge is idempotent: running it multiple times on the same set of staged files produces the same result.
If the merge engine fails partway through, it must be resilient to restart. It keeps track of processed files in a processed directory, and each processed file is marked with a checkpoint file:
{
"batch_id": "batch-000123.parquet",
"status": "completed",
"start_time": "2024-02-01T14:30:00Z",
"end_time": "2024-02-01T14:32:15Z",
"num_records": 12345
}On restart, the merge engine skips already processed files and resumes from the last checkpoint. A failure mode occurs when a file is marked as processed but the merge operation fails before writing the result. The result is duplicate data in the table.
To avoid this, the merge engine uses a two-phase commit:
_temp/).processed directory with the checkpoint.The move is atomic: if the merge fails during the move, the engine detects the partial state and can retry.
The lakehouse supports schema evolution. A table may change its schema over time. The engine must resolve the schema of each file and merge them into a unified view.
Schema evolution is managed via a schema.json file in the table root. It contains the current schema and a history of previous schemas:
{
"current_version": 3,
"schema": {
"type": "struct",
"fields": [
{ "name": "user_id", "type": "int64", "nullable": false },
{ "name": "event_type", "type": "string", "nullable": false },
{ "name": "duration_seconds", "type": "double", "nullable": true }
]
},
"versions": [
{
"version": 1,
"schema": {
"type": "struct",
"fields": [
{ "name": "user_id", "type": "int64", "nullable": false },
{ "name": "event_type", "type": "string", "nullable": false }
]
}
},
{
"version": 2,
"schema": {
"type": "struct",
"fields": [
{ "name": "user_id", "type": "int64", "nullable": false },
{ "name": "event_type", "type": "string", "nullable": false },
{ "name": "duration", "type": "int32", "nullable": true }
]
}
}
]
}When a query is executed, the engine resolves the schema from the schema.json and applies it to each file. If a file has a schema version 1 but the current table schema is version 3, the engine maps the fields using the schema history.
A common failure is a mismatch between the schema in the file and the schema in the schema.json. If the file schema does not match the current schema, the engine may drop columns or pad with nulls. The most silent failure is when a new column is added to the schema but not written to the file: the column appears as null for all rows.
To ensure consistency, the engine validates each file against the expected schema. A validation error is logged but not fatal. The query proceeds, but the reader may notice missing or incorrect data.
To improve performance, the engine caches schema mappings in a schema_cache.json file. The cache is updated during table scans and is used to resolve schemas without reading the full schema.json.
The most common failure is data drift: new data arrives with a different schema, partitioning, or file format than expected. The system does not fail, but queries return inconsistent results.
To detect failures, monitor the transaction log. A gap in transaction IDs indicates a missed batch. A missing transaction log entry indicates a failed write.
Another failure is manifest staleness. The manifest file is not updated after a batch of data is written. The engine reads data from the manifest but misses the latest files.
To detect this, use a manifest_age metric: the difference between the latest file modification time and the manifest’s last_updated timestamp. A high age indicates that the manifest is outdated.
For observability, the lakehouse exposes a status endpoint via HTTP. The endpoint returns the state of the table:
GET /api/table/clickstream_logs/status HTTP/1.1
Host: lakehouse.example.com
Authorization: Bearer abc123{
"table": "clickstream_logs",
"version": 123,
"latest_transaction": "txn-12345",
"last_updated": "2024-02-01T14:32:15Z",
"files": 456,
"total_size": "2.1 TB",
"manifest_age": "1m23s",
"schema_version": 3,
"partition_keys": [ "date", "hour" ],
"status": "healthy",
"warnings": [
"manifest_age > 1m",
"missing_transaction_log_entry: txn-12344"
]
}This endpoint is used by monitoring tools to alert on data freshness, schema drift, and transaction gaps.
A data lakehouse architecture integrates query, write, transaction, and streaming patterns into a single, consistent data layer. It relies on standardized formats—Parquet, JSON, manifest files—and atomic operations across object storage. The core is the transaction log and manifest, which together enable ACID transactions on data stored in a data lake. The architecture supports schema evolution, incremental updates, and real-time analytics. Its success depends on careful attention to atomicity, consistency, and observability. When it fails, the failures are subtle, silent, and pervasive—making observability essential.
This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy