ESC
Type to search guides, tutorials, and reference documentation.

Real-Time Analytics and Stream Processing

Partitioned logs, event time versus processing time, watermarks, windows, state, delivery semantics, consumer lag and backpressure — and when streaming is not worth its cost.

Real-time analytics and stream processing enable systems to derive insights and trigger actions as data arrives, rather than waiting for batch windows. This matters when decisions must be made within seconds of event occurrence—such as fraud detection, live dashboards, real-time monitoring, and adaptive recommendation engines. The core trade-off is between latency, throughput, and correctness, and the challenge lies in ensuring that every event is processed exactly once, even under network partitions, node failures, and irregular arrival rates.

Ingesting Data Streams

Data ingestion is the first step in real-time analytics. Events arrive continuously from sources such as IoT devices, web applications, and microservices. The goal is to capture these events with minimal delay and preserve their order across producers.

To ingest data, a system must define a stream or topic, which is a durable, append-only sequence of events. Each event is a message with a key, value, and metadata. The ingestion process begins by configuring a producer to write to a specific stream.

For example, using Apache Kafka as the underlying stream engine, a producer sends events via the kafka-producer command-line tool:

kafka-producer \
  --bootstrap-server kafka.example.com:9092 \
  --topic user-events \
  --key-format json \
  --value-format json \
  --batch-size 1000 \
  --linger-ms 50 \
  --acks all \
  --max-retries 3

Key options:

  • --batch-size controls when messages are sent in bulk; default is 16,384 bytes.
  • --linger-ms delays sending until the batch is full or this time elapses, reducing network overhead.
  • --acks all ensures that the leader waits for all in-sync replicas before acknowledging, guaranteeing durability.
  • --max-retries enables idempotent producer behavior, reducing duplicate events.

A common failure mode is producer lag: the producer cannot keep up with the event rate. This is signaled by increasing pending-acks and buffer-full events in logs. When the producer buffer fills, new events are dropped or written to disk but not sent. The system must monitor producer-latency and producer-records-sent metrics and adjust linger-ms and batch-size accordingly.

Events are typically serialized as JSON or Avro. In Avro, schemas are registered in a schema registry. A producer must specify the schema ID or register it dynamically:

{
  "type": "record",
  "name": "UserEvent",
  "fields": [
    { "name": "userId", "type": "string" },
    { "name": "timestamp", "type": "long" },
    { "name": "action", "type": "string" }
  ]
}

If the schema evolves and producers write outdated data, consumers may fail to deserialize. A typical error is:

AvroSerdeError: Missing field 'action' in record UserEvent

To handle schema evolution, producers must publish new versions and consumers must be configured to accept backward- and forward-compatible schemas.

Processing Streams with Exactly-Once Semantics

Once data is ingested, processing begins. Stream processors apply transformations, aggregations, and joins to incoming events. The goal is to compute results that are both timely and correct.

A stream processor reads from one or more input streams, applies a set of operations, and writes output to one or more result streams. The processing model is stateful—the processor maintains state between events, such as windowed aggregates or lookup tables.

A typical configuration uses a stream processing engine with a declarative DSL. For example, a processor written in SQL-like syntax might define a windowed count of user actions per minute:

CREATE STREAM user_action_counts AS
SELECT
  window_start,
  window_end,
  user_id,
  COUNT(*) AS action_count
FROM user_events
GROUP BY user_id, TUMBLE(event_time, INTERVAL '1' MINUTE)
PARTITION BY user_id
EMIT CHANGES

This query:

  • Defines a tumbling window of one minute.
  • Groups events by user_id and window boundaries.
  • Emits a new row whenever a window completes.
  • Partitions the output stream by user_id, enabling scalable, parallel processing.

The processor maintains state in a state store, which is typically a key-value database. The state store must be durable and accessible to all processing instances. The processor can fail and restart, but must restore state from the store.

A common failure is state loss due to a node crash. When a processor instance fails, it must recover its state from the state store. The recovery process involves:

  1. Reading the latest checkpoint (a position in the input stream).
  2. Loading state from the store.
  3. Replaying events from the checkpoint onward.

If the state store is not replicated, a failure during processing may cause inconsistency—results may be duplicated or missed.

To achieve exactly-once processing, the system must coordinate three phases:

  1. Input processing: the processor reads and processes each event.
  2. Output writing: the processor writes results to output streams.
  3. Checkpointing: the system records progress and state.

A failure during output writing may result in duplicates. For example, if a processor writes to a Kafka topic and the broker fails after writing the message but before acknowledging, the message may be written twice.

To eliminate duplicates, the system uses idempotent sinks and transactional producers. The processor sends events in transactions, and the sink (e.g., a database) applies the transaction atomically.

A typical configuration for a transactional stream processor includes:

processor:
  id: user-action-processor
  input: user_events
  output: user_action_counts
  mode: exactly-once
  checkpoint-interval: 30s
  state-store:
    type: rocksdb
    path: /var/lib/processor/state
    replication: 3
  transaction:
    producer:
      bootstrap-server: kafka.example.com:9092
      transaction-timeout-ms: 60000
      acks: all

The processor writes state and output in a single transaction. If the transaction commits, the entire batch of events is applied exactly once. If it fails, the processor rolls back and retries.

A subtle failure mode is checkpointing delay: the processor may checkpoint after processing an event but before writing the result. If the processor crashes at that moment, the result may be lost. To mitigate, the system uses dual-phase checkpointing:

  • First, the processor writes state to the store.
  • Then, it writes a checkpoint record to a dedicated checkpoint stream.
  • The processor reads the checkpoint stream to recover its position.

Serving Real-Time Analytics

After processing, results must be available to consumers: dashboards, APIs, or downstream systems. The primary challenge is latency: how long after an event is received until it appears in a query result.

A real-time analytics system serves data through materialized views or streamed projections. These are precomputed results that are updated incrementally.

For example, a dashboard showing real-time user activity might query the user_action_counts stream. The query engine supports continuous queries that execute and update results as new data arrives.

A typical query in a real-time SQL engine:

SELECT
  user_id,
  SUM(action_count) AS total_actions,
  AVG(action_count) AS avg_actions_per_minute
FROM user_action_counts
WHERE window_end > NOW() - INTERVAL '5' MINUTE
GROUP BY user_id
ORDER BY total_actions DESC
LIMIT 10

This query:

  • Computes aggregates over the last five minutes.
  • Updates results continuously as new events arrive.
  • Returns a stream of result rows, each representing a new snapshot.

The query engine must maintain query state, including window bounds, aggregation buffers, and result caches.

A common issue is query staleness. If the query engine is not updated in real time, the dashboard may show outdated data. This happens when:

  • The query engine is not configured to refresh continuously.
  • The underlying data stream has high latency.
  • The query engine uses a batch-based execution model.

To fix, the query engine must be configured to:

  • Poll the input stream at regular intervals.
  • Push updates when new data arrives.
  • Use delta propagation to send only changes, not full results.

A typical configuration for a real-time query engine includes:

query-engine:
  name: real-time-analytics
  input: user_action_counts
  refresh-interval: 1s
  mode: continuous
  cache: true
  cache-size: 10000
  streaming: true
  delta-push: true
  buffer-size: 1024

If the system does not support delta push, the query engine may send full results after each refresh, causing high bandwidth usage and latency.

Another issue is query skew: results from different queries may be out of sync. For example, a query for total actions and another for average actions may be computed at slightly different times, leading to inconsistent totals.

To synchronize, the system uses global timestamps. Each event carries a timestamp, and the query engine aligns all computations to a common timeline. The timestamp source may be:

  • The event’s timestamp field.
  • The ingestion time (wall-clock time at producer).
  • A logical clock such as a sequence number.

A failure mode is timestamp drift: events arrive with incorrect or inconsistent timestamps. A typical error:

QueryResultError: Window end time missing in user_action_counts

The query engine assumes that window_end is available, but the event stream may omit it or write it with a delay.

Monitoring and Troubleshooting

A real-time analytics system must be monitored for latency, throughput, and correctness.

Key metrics include:

  • Event processing latency: time from event ingestion to final output.
  • Stream lag: how far behind the input stream the processor is.
  • Checkpoint interval deviation: how often checkpoints are written.
  • Query update frequency: how often results are refreshed.

Monitoring tools use standard formats: Prometheus metrics, OpenTelemetry traces, and structured logs.

Example logs from a stream processor:

[2023-09-15T12:34:56.789Z] INFO  processor:123 - Processed 12345 events in 1.2s, avg latency: 42ms
[2023-09-15T12:34:57.001Z] WARN  processor:123 - Checkpoint failed: state store unreachable
[2023-09-15T12:34:57.100Z] ERROR processor:123 - Duplicate event: user_id=123, action=click, timestamp=1694821234

A common failure is backpressure: the processor cannot keep up with incoming events. The system shows increasing pending-records, buffer-usage, and processing-latency.

The solution is to scale the processor:

  • Add more instances.
  • Increase parallelism per instance.
  • Tune batch-size, linger-ms, and checkpoint-interval.

A sharp edge is checkpointing overhead: if the checkpoint interval is too short, the system spends too much time writing checkpoints and less time processing. If it’s too long, results are delayed.

Another edge case is event reordering: events may arrive out of order due to network delays or multiple producers. The system must handle event time processing.

A typical reordering failure:

EventTimeError: Event with timestamp 1694821234 arrived after event with timestamp 1694821233

The processor must use watermarking to track progress and emit results at the correct time.

Watermarks are timestamps that indicate when all events up to that time have been processed. The system emits a watermark every few seconds, and the processor uses it to:

  • Trigger window computations.
  • Detect late events.
  • Adjust output timing.

A common misconfiguration is watermark interval mismatch: the watermark is emitted every 10 seconds, but the processor expects updates every 5 seconds. This leads to infrequent results.

To resolve, the system must align watermark emission with query refresh intervals and window sizes.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.