Data Lake Architecture

Design and implement data lakes that scale from gigabytes to petabytes. Covers lakehouse architecture, storage formats (Parquet, Delta, Iceberg), partitioning strategies, data lifecycle management, query engines, and the patterns that prevent data lakes from becoming data swamps.

A data lake architecture is a storage system designed to hold vast amounts of raw, unprocessed data in its native format. This approach becomes essential when you need to store diverse data types without upfront schema definition and want to defer processing decisions until analysis time.

Storage Structure and Organization

Data lakes typically use object storage as their primary backing store. The most common implementation uses a hierarchical directory structure with timestamp-based partitioning:

/data-lake/
├── raw/
│   ├── events/
│   │   ├── 2023/
│   │   │   ├── 12/
│   │   │   │   ├── 01/
│   │   │   │   │   ├── event_20231201_000000.json
│   │   │   │   │   └── event_20231201_000001.json
│   │   │   │   └── 02/
│   │   │   │       ├── event_20231202_000000.json
│   │   │   └── 13/
│   │   └── user_sessions/
│   │       ├── 2023/
│   │       │   ├── 12/
│   │       │   │   ├── session_20231201_000000.json
│   │       │   │   └── session_20231201_000001.json
│   └── logs/
│       ├── application/
│       │   ├── 2023/
│       │   │   ├── 12/
│       │   │   │   ├── applog_20231201_000000.log
│       │   │   │   └── applog_20231201_000001.log

When using AWS S3, the most common implementation pattern involves setting bucket lifecycle policies to transition data between storage classes:

aws s3api put-bucket-lifecycle-configuration \
  --bucket my-data-lake \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "TransitionToIA",
        "Status": "Enabled",
        "Filter": {"Prefix": "raw/events/"},
        "Transitions": [
          {
            "Days": 30,
            "StorageClass": "STANDARD_IA"
          }
        ]
      }
    ]
  }'

Data Ingestion Patterns

Ingestion into a data lake works best with batch-oriented pipelines that write data in discrete chunks. The recommended approach uses a staging area with atomic writes:

# Example ingestion script
#!/bin/bash
set -e

SOURCE_FILE="/tmp/current_batch.json"
STAGING_PATH="s3://my-data-lake/staging/events/$(date +%Y/%m/%d/%H)/batch_$(date +%s).json"
FINAL_PATH="s3://my-data-lake/raw/events/$(date +%Y/%m/%d/%H)/batch_$(date +%s).json"

# Stage the file first
aws s3 cp "$SOURCE_FILE" "$STAGING_PATH"

# Validate file integrity
if ! aws s3api head-object --bucket my-data-lake --key "$(basename "$STAGING_PATH")" >/dev/null 2>&1; then
    echo "ERROR: File validation failed"
    exit 1
fi

# Atomically move to final location
aws s3 mv "$STAGING_PATH" "$FINAL_PATH"

Partitioning Strategy

Partitioning should align with query patterns. Common approaches include:

The key constraint is that partitions must be immutable once written. A typical implementation uses a directory structure like:

/data-lake/raw/events/2023/12/01/08/

Each partition should contain files of similar size (typically 100MB to 1GB). Smaller files cause performance issues due to metadata overhead.

Schema Evolution

Data lakes support schema evolution through format selection. Parquet is the most common choice for analytical workloads:

-- Example: reading parquet data with schema inference
CREATE EXTERNAL TABLE events_parquet (
    event_id STRING,
    user_id BIGINT,
    timestamp TIMESTAMP,
    payload STRUCT<type:STRING, data:STRING>
)
STORED AS PARQUET
LOCATION '/data-lake/raw/events/';

When writing to parquet, the schema is defined by the writer. The reader must be prepared to handle missing fields or field type changes:

# Example: reading with schema evolution
import pandas as pd

df = pd.read_parquet('s3://my-data-lake/raw/events/2023/12/01/08/batch_12345.parquet')
# Missing columns will appear as NaN in the resulting DataFrame

Query Interface Patterns

Query interfaces typically use SQL engines connected to object storage. Common implementations include:

The most common pattern involves creating external tables pointing to the lake's data:

CREATE EXTERNAL TABLE user_events (
    user_id BIGINT,
    event_type STRING,
    timestamp TIMESTAMP,
    metadata MAP<STRING, STRING>
)
STORED AS PARQUET
LOCATION 's3://my-data-lake/raw/events/';

Data Quality Considerations

Data lakes require careful attention to data quality. The most common failure mode occurs when ingestion pipelines write corrupt or incomplete files:

# Example: validation script for ingested data
#!/bin/bash
BUCKET="my-data-lake"
PREFIX="raw/events/2023/12/01/08"

# Check file count and size
FILES=$(aws s3 ls "$BUCKET/$PREFIX" | wc -l)
if [ "$FILES" -lt 1 ]; then
    echo "ERROR: No files found in partition"
    exit 1
fi

# Validate each file's integrity
for file in $(aws s3 ls "$BUCKET/$PREFIX"); do
    FILENAME=$(basename "$file")
    aws s3api head-object --bucket "$BUCKET" --key "$PREFIX/$FILENAME" >/dev/null 2>&1 || {
        echo "ERROR: Corrupt file detected: $FILENAME"
        exit 1
    }
done

Performance Optimization

Performance optimization requires understanding that data lakes work best with columnar formats like Parquet or ORC. Avoiding row-based formats is critical:

# Example: converting CSV to Parquet with spark
spark-submit \
  --class com.example.ConvertCsvToParquet \
  --master yarn \
  --deploy-mode cluster \
  my-conversion-job.jar \
  s3://my-data-lake/raw/events/2023/12/01/08/ \
  s3://my-data-lake/processed/events/2023/12/01/08/

The conversion process must be aware of data types and handle schema changes properly. Columnar formats should be used for analytical queries.

Data Governance

Data governance in a lake requires explicit metadata management. The system works best when:

# Example: setting access control on S3 objects
aws s3api put-object-acl \
  --bucket my-data-lake \
  --key raw/events/2023/12/01/08/batch_12345.json \
  --acl bucket-owner-full-control

Common Failure Modes

The most common failure mode is writing incomplete data to partitions. This occurs when:

  1. A pipeline fails mid-batch and leaves files in staging areas
  2. File uploads timeout or fail with partial writes
  3. Partition directories are not properly managed
# Example: cleanup script for failed batches
#!/bin/bash
STAGING_BUCKET="my-data-lake-staging"
FINAL_BUCKET="my-data-lake"

# Clean up stale staging files
aws s3 rm s3://$STAGING_BUCKET --recursive --exclude "*" --include "*/batch_*.json" --dryrun

Data Retention and Lifecycle Management

Lifecycle policies are essential for cost management. The typical implementation uses:

# Example: lifecycle policy for data retention
aws s3api put-bucket-lifecycle-configuration \
  --bucket my-data-lake \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "DeleteOldFiles",
        "Status": "Enabled",
        "Filter": {"Prefix": "raw/events/"},
        "ExpirationInDays": 365,
        "NoncurrentVersionExpirationInDays": 90
      }
    ]
  }'

Cost Optimization

Cost optimization requires careful attention to data access patterns. Hot data should remain in standard storage, while cold data can be moved to cheaper classes:

# Example: transitioning data between storage classes
aws s3api put-object-lifecycle-configuration \
  --bucket my-data-lake \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "TransitionToGlacier",
        "Status": "Enabled",
        "Filter": {"Prefix": "raw/logs/"},
        "Transitions": [
          {
            "Days": 90,
            "StorageClass": "GLACIER"
          }
        ]
      }
    ]
  }'

Monitoring and Alerting

Monitoring focuses on data freshness, ingestion success rates, and storage usage. The most effective approach uses:

# Example: monitoring script
#!/bin/bash
BUCKET="my-data-lake"
PREFIX="raw/events/$(date +%Y/%m/%d)"

# Check if partition exists
if ! aws s3 ls "$BUCKET/$PREFIX" >/dev/null 2>&1; then
    echo "ALERT: Missing partition for today"
    exit 1
fi

# Check file count in last hour
FILES=$(aws s3 ls "$BUCKET/$PREFIX" | wc -l)
if [ "$FILES" -lt 1 ]; then
    echo "ALERT: No files in partition"
    exit 1
fi

The architecture works best when pipelines are designed with atomic writes, clear partition boundaries, and explicit error handling. The system's strength lies in its ability to handle diverse data types without upfront schema definition while maintaining performance through proper format selection and partitioning strategies.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.