Big Data Architecture

Big Data Architecture orchestrates the end-to-end flow of large, complex, and rapidly evolving datasets across storage, processing, and consumption layers. It matters when data ingestion rates exceed gigabytes per minute, when processing workloads require parallel computation across hundreds of nodes, and when downstream systems depend on near-real-time availability of data with consistent quality and lineage. This architecture is not just about storing more data—it is about making data *usable*, *reliable*, and *timely* across diverse, often distributed, systems.

Ingesting Raw Data at Scale

Data arrives from sensors, logs, user actions, and external APIs in continuous streams or batch dumps. The architecture must handle high-throughput ingestion with low latency and fault tolerance.

The ingestion layer typically uses a message broker to decouple producers from consumers. A common pattern is to write data to a topic in Apache Kafka, where each record is a key-value pair with a timestamp and optional metadata. Producers publish to topics using a standard protocol such as the Kafka wire format over TCP. A typical producer configuration in a Python application using confluent-kafka includes:

from confluent_kafka import Producer

producer = Producer({
    'bootstrap.servers': 'kafka1:9092,kafka2:9092',
    'acks': 'all',
    'retries': 5,
    'batch.size': 16384,
    'linger.ms': 10,
    'compression.type': 'snappy',
    'value.serializer': lambda v: json.dumps(v).encode('utf-8'),
    'key.serializer': lambda k: k.encode('utf-8')
})

def send_event(topic, key, value):
    producer.produce(
        topic=topic,
        key=key,
        value=value,
        on_delivery=lambda err, msg: print(f"Delivered {msg.offset()}: {msg.value().decode('utf-8')}")
    )
    producer.flush()

The acks=all setting ensures that the producer waits for all in-sync replicas to acknowledge the write before considering the record committed. The linger.ms setting batches messages for up to 10 milliseconds, improving throughput at the cost of latency. When a broker fails, the leader for a partition is automatically elected, and consumers resume from the last committed offset.

In cases where ingestion must be resilient to network partitioning or producer crashes, producers use idempotent transactions. Each producer instance is assigned a unique client.id, and transactional producers use transactional.id to ensure exactly-once semantics across multiple topics. A producer can begin a transaction, write several messages, and commit the transaction atomically:

producer.init_transactions()
producer.begin_transaction()

try:
    for record in data_batch:
        producer.produce(topic='events', key=record['id'], value=record)
    producer.commit_transaction()
except Exception as e:
    producer.abort_transaction()

When a consumer group reads from Kafka, it maintains a consumer.group.id and stores its current offset in the __consumer_offsets topic. If a consumer crashes, another instance can resume from the last committed offset. The enable.auto.commit flag controls whether offsets are committed automatically; setting auto.commit.interval.ms=5000 ensures a commit every 5 seconds.

Failure modes include lag accumulation when consumers fall behind, caused by slow processing or high input volume. A consumer group’s lag is tracked in the kafka-consumer-groups.sh CLI tool:

./kafka-consumer-groups.sh --bootstrap-server kafka1:9092 \
  --describe --group my-group \
  --topic events

Output shows CURRENT-OFFSET, LOG-END-OFFSET, and LAG. A lag of 100,000 means the consumer is 100k messages behind. High lag can be diagnosed by checking kafka-server-start.sh logs for org.apache.kafka.server.metrics.KafkaMetrics reports showing MessagesPerSecond and NetworkProcessor load.

Processing Data in Parallel

Once ingested, data must be transformed, enriched, and aggregated. This is where batch and streaming processing engines come into play.

Apache Spark is a common choice for both batch and streaming workloads. A Spark job is submitted via spark-submit, with exact configuration keys in a spark-defaults.conf file:

spark.app.name=DataPipeline
spark.master=yarn
spark.executor.memory=4g
spark.executor.cores=4
spark.driver.memory=8g
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.sql.adaptive.enabled=true
spark.sql.adaptive.coalescePartitions.enabled=true

A Spark application reads data from Kafka using the Structured Streaming API. The following Scala code defines a streaming query that reads from a Kafka topic, applies a transformation, and writes output to a Delta Lake table:

val streamingQuery = spark
  .readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "kafka1:9092,kafka2:9092")
  .option("subscribe", "events")
  .option("startingOffsets", "latest")
  .load()
  .selectExpr("CAST(value AS STRING)")
  .as[RawEvent]
  .writeStream
  .format("delta")
  .option("checkpointLocation", "/checkpoints/events")
  .option("path", "/data/events")
  .outputMode("append")
  .start()

The checkpointLocation is critical: it stores the state of the streaming job—current offsets, aggregations, and metadata—on a durable file system like HDFS or S3. If the job fails, it resumes from the last checkpoint. The outputMode controls how data is written: append mode writes only new records, update mode updates existing aggregates, and complete mode writes the entire result set every time.

Spark’s adaptive query execution dynamically optimizes the physical plan based on runtime statistics. When a shuffle stage shows a skewed distribution of data across partitions, Spark automatically coalesces smaller partitions and repartitions data.

A common failure is data skew: one partition receives many more records than others. This leads to bottlenecks where a single executor becomes the bottleneck. Skew is diagnosed by examining Spark UI logs showing Stage 2: Shuffle Read and Shuffle Write times. The spark.sql.adaptive.skewJoin.enabled flag enables automatic skew handling, where Spark identifies skewed keys and repartitions those keys into separate partitions.

For long-running jobs, Spark jobs can be scheduled using cron or systemd. A service file spark-job.service ensures the job restarts on failure:

[Unit]
Description=Spark Batch Job
After=network.target

[Service]
Type=simple
User=spark
WorkingDirectory=/opt/spark-jobs
ExecStart=/opt/spark/bin/spark-submit --class MainApp --deploy-mode cluster job.jar
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

When a job fails, the exit code and logs are inspected using systemctl status spark-job.service and journalctl -u spark-job.service -f.

Storing Data for Querying and Analysis

After processing, data is stored in a central repository accessible to downstream systems. The storage layer must support both analytical queries and high-volume reads.

The choice of storage is often governed by query patterns: columnar formats for analytical workloads, and key-value stores for transactional access.

Apache Parquet is the de facto columnar format. Data is stored in .parquet files, with metadata describing the schema, encoding, and statistics per column. A query engine like Apache Hive or Presto reads Parquet files using a standard columnar scan:

SELECT user_id, AVG(duration), COUNT(*) 
FROM events 
WHERE event_type = 'click' 
  AND date >= '2024-01-01'
GROUP BY user_id
ORDER BY duration DESC;

The query engine prunes partitions based on date, reads only the user_id and duration columns, and applies filters on event_type using page-level statistics stored in the Parquet file.

Delta Lake is a storage layer that extends Parquet with transactional guarantees. It maintains a METADATA file and a VERSIONS table in the file system. Each write operation is a transaction, and the system supports ACID semantics. A MERGE operation updates existing records and inserts new ones:

MERGE INTO users AS target
USING new_users AS source
ON target.user_id = source.user_id
WHEN MATCHED THEN
  UPDATE SET name = source.name, email = source.email
WHEN NOT MATCHED THEN
  INSERT VALUES (source.user_id, source.name, source.email)

Delta Lake supports time travel, allowing queries to read data as it existed at a specific version:

SELECT * FROM users VERSION AS OF 123;

The version is a sequence number assigned with each commit. A VACUUM command cleans up old versions and deleted files:

-- In Delta Lake CLI
$ delta table users
> VACUUM 100

This command deletes all versions older than 100, reducing storage and improving performance.

When multiple systems query the same data, a data catalog becomes essential. The catalog is often a simple database that maps table names to locations and metadata. A typical schema uses a tables table with columns: name, location, format, partition_columns, created_at, and last_updated. Queries are stored in the catalog with a queries table containing name, sql, author, and created_at.

Serving Data to Applications

Final data products—aggregates, models, and derived datasets—are served to applications via APIs or direct access.

A common approach is to expose data through a REST API using a standard framework like Spring Boot or FastAPI. A typical endpoint returns JSON, with a well-defined schema:

{
  "user_id": 12345,
  "total_sessions": 7,
  "avg_duration_ms": 14200,
  "first_session": "2023-11-01T10:00:00Z",
  "last_session": "2024-01-15T14:30:00Z"
}

The API is deployed behind a reverse proxy like Nginx, configured to serve static assets and route requests:

server {
    listen 80;
    server_name analytics-api.example.com;

    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering on;
        proxy_buffer_size 16k;
        proxy_buffers 4 16k;
    }

    location /static {
        alias /app/static;
        expires 1y;
        add_header Cache-Control "public, immutable";
    }
}

When a client makes a request, the server logs the request_id, timestamp, status, and response_time to a structured log file in JSON format:

{
  "request_id": "a1b2c3d4",
  "timestamp": "2024-01-15T14:30:00Z",
  "method": "GET",
  "path": "/users/12345",
  "status": 200,
  "response_time_ms": 42,
  "user_id": 12345,
  "query": "SELECT * FROM users WHERE id = 12345"
}

For complex, real-time analytics, a materialized view is maintained in a columnar store. A CREATE MATERIALIZED VIEW statement defines a persistent, precomputed result set:

CREATE MATERIALIZED VIEW daily_user_summary AS
SELECT 
    DATE(event_timestamp) AS day,
    user_id,
    COUNT(*) AS events,
    SUM(duration_ms) AS total_duration
FROM events
GROUP BY day, user_id
WITH (
    refresh_interval = '1h',
    storage_format = 'parquet'
);

The view is refreshed periodically—either on a schedule or triggered by new data. When a query hits the view, it returns results immediately, without scanning the base table.

Monitoring and Failure Recovery

The architecture relies on monitoring to detect issues before users are affected.

A central monitoring system collects metrics from all components: Kafka, Spark, Delta Lake, and the API server. Metrics are exposed via HTTP endpoints, typically at /metrics, returning a text-based format:

# TYPE kafka_consumer_lag gauge
kafka_consumer_lag{group="user-events",topic="events"} 12450
kafka_consumer_lag{group="user-events",topic="logs"} 3210

# HELP spark_job_duration_seconds Duration of a Spark job
# TYPE spark_job_duration_seconds histogram
spark_job_duration_seconds_bucket{job="user_summary",le="10"} 15
spark_job_duration_seconds_bucket{job="user_summary",le="30"} 28
spark_job_duration_seconds_bucket{job="user_summary",le="60"} 35
spark_job_duration_seconds_sum{job="user_summary"} 1245.3
spark_job_duration_seconds_count{job="user_summary"} 35

A dashboard tool like Grafana ingests these metrics and displays alerts. An alert triggers when the Kafka consumer lag exceeds 50,000 for more than 10 minutes.

When a system fails, recovery is automated. A failed Spark job is restarted by a systemd service, which checks the job’s exit code and logs. If the job fails with exit code 137, it indicates an OOM (out-of-memory) error, often caused by insufficient spark.executor.memory.

A failed Delta Lake write may result in an inconsistent state if the MERGE transaction is interrupted. The delta-merge command logs a transaction_id and status to a transactions table:

SELECT transaction_id, status, start_time, end_time, error_message
FROM transactions
WHERE status = 'FAILED';

The error_message contains the full stack trace from the failure, often including the spark.sql.adaptive.advisory messages that suggest which optimization would help.

In the end, the architecture is not defined by its components, but by how they interact: how data flows, how failures propagate, and how recovery happens without manual intervention.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.