Stream Processing Patterns

Production engineering guide for stream processing patterns covering patterns, implementation strategies, and operational best practices.

Stream processing handles data flows that arrive continuously and must be processed in real time or near real time. This applies when systems need to react to events as they happen, maintain state across streams, or process high-volume, low-latency data without batching. The patterns described here are about how to structure processing logic, not about the tools themselves.

Processing Events as They Arrive

The fundamental pattern for handling streaming data is to consume messages from a topic or queue and process them immediately. An event-driven system reads from a message broker like Apache Kafka, Amazon Kinesis, or Redis Streams.

For example, with Kafka, you might use kafka-console-consumer.sh to test a stream:

kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic user-events \
  --from-beginning \
  --formatter kafka.tools.DefaultMessageFormatter

Processing logic should be stateless when possible. Each message is handled independently, with no assumptions about previous messages. The consumer must acknowledge successful processing to avoid duplicate handling.

A common failure mode occurs when consumers fail to commit offsets properly. If a consumer crashes before committing, it may reprocess the same messages on restart. You can see this in Kafka logs:

[Consumer clientId=consumer-1, groupId=stream-group] Offset commit failed for group stream-group

Handling Late Arriving Events

In streaming systems, events may arrive out of order or late due to network delays, processing delays, or clock drift. The pattern is to define a maximum allowed delay and apply windowing functions.

With Kafka Streams, you can specify event time windows:

StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> source = builder.stream("user-events");

source
  .groupByKey()
  .windowedBy(TimeWindows.of(Duration.ofMinutes(5)))
  .reduce((value1, value2) -> value1 + "," + value2)
  .toStream()
  .mapValues(value -> "Aggregated: " + value);

You must set a watermark delay to control how long to wait for late events:

processing.guarantee = exactly_once
commit.interval.ms = 5000
max.poll.interval.ms = 300000

If you do not configure sufficient time for late events, the system may drop them silently. The pattern requires explicit handling of event-time semantics and watermark calculation.

Aggregating Streams

When processing streams, you often need to aggregate data over time windows or sliding intervals. This pattern involves maintaining state and updating it incrementally.

For a count per minute aggregation, you might define:

KGroupedStream<String, String> grouped = source.groupByKey();
KTable<Windowed<String>, Long> counts = grouped
  .windowedBy(TimeWindows.of(Duration.ofMinutes(1)))
  .count();

This requires state stores to hold intermediate results. Kafka Streams uses RocksDB by default, and you must configure memory limits:

rocksdb.config.setter = org.apache.kafka.streams.state.RocksDBConfigSetter
state.dir = /var/lib/kafka-streams

If the state store fills up or becomes inconsistent, the aggregation may start dropping or duplicating results. You can detect this in logs:

[StateStore] State store is full or has corrupted data

Filtering and Transforming Streams

Filtering and transformation are core operations in streaming pipelines. A filter removes messages that do not match criteria, while a transform modifies the content.

With Kafka Streams, filtering is done with filter:

KStream<String, String> filtered = source.filter((key, value) -> 
  value.contains("important"));

Transformation uses mapValues or map for key-value changes:

KStream<String, String> transformed = source.mapValues(value -> 
  value.toUpperCase());

You must be careful with null values. A null return from a map function will cause the record to be skipped, which can silently affect downstream processing.

Handling Backpressure and Retry Logic

Streaming systems must handle situations where consumers cannot keep up with producers. The pattern is to implement backpressure through buffer limits or consumer lag monitoring.

You can configure Kafka consumers to limit how many messages are fetched at once:

max.poll.records = 100
fetch.max.bytes = 52428800

If the system becomes overloaded, you might see this in logs:

[Consumer] Consumer is falling behind on topic user-events

Retry logic for failed processing should be idempotent. A consumer that reprocesses a failed message must ensure it does not double-count or create duplicates.

Managing State Across Failures

Stateful stream processing requires durability and consistency. You must configure how state is checkpointed and recovered.

Kafka Streams supports state checkpointing:

processing.guarantee = exactly_once
commit.interval.ms = 1000

The system creates snapshots of the state store, which are stored in a dedicated topic. If a consumer fails, it can recover from the last checkpoint.

If you do not configure proper checkpointing, the system will lose state during restarts. You may see errors like:

[StateRestoreListener] Restore failed for store user-state

Processing Multiple Streams Together

When multiple data streams must be joined or coordinated, you use join patterns. This is common when combining events from different sources.

In Kafka Streams, you can join two streams:

KStream<String, String> stream1 = builder.stream("source-1");
KStream<String, String> stream2 = builder.stream("source-2");

KStream<String, String> joined = stream1.join(stream2,
  (value1, value2) -> value1 + " | " + value2,
  JoinWindows.of(Duration.ofMinutes(5))
);

You must define a window for joining, otherwise the join may be incomplete or miss events. If you do not specify a window, it defaults to no-time-bound joins, which can cause inconsistent results.

Detecting and Handling Duplicate Events

Duplicate detection is necessary when your processing system may reprocess messages due to failures or retries.

A common pattern is to track message IDs in a deduplication store:

KTable<String, String> seen = builder.table("seen-messages");
KStream<String, String> unique = source
  .filter((key, value) -> {
    // Check if message ID already processed
    return !seen.get(key).equals(value);
  });

You must maintain the deduplication store in a durable way. If it is not persisted, you may lose track of which messages were already handled.

Streaming Data to Multiple Destinations

A stream may need to be sent to more than one downstream system. This pattern uses fan-out logic.

In Kafka Streams, you can split a stream:

KStream<String, String> source = builder.stream("user-events");
source.to("events-for-analytics");
source.to("events-for-alerting");

Each output must be configured with appropriate replication and retention settings. If one destination fails, the system should not block the others.

Handling Failures in Stream Processing

Failures in streaming systems often manifest as lagging consumers or dropped messages. The pattern is to monitor consumer lag and implement circuit breaker logic.

You can monitor consumer lag via Kafka's admin API:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group stream-group

If a consumer group has high lag, it may be due to processing time or resource constraints. You should configure timeouts and backpressure to avoid indefinite blocking.

Emitting Events at Regular Intervals

Some patterns require emitting events periodically even when no new data arrives. This is useful for maintaining heartbeat or periodic aggregation.

You can use a scheduled task to emit periodic events:

ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
scheduler.scheduleAtFixedRate(() -> {
  // Emit heartbeat or periodic aggregation
}, 0, 60, TimeUnit.SECONDS);

This requires careful handling of timing and ensures that downstream consumers do not wait indefinitely for new data.

Streaming with Windowed Aggregations

Windowed aggregations process data in fixed or sliding time windows. This pattern is critical for metrics and reporting systems.

In Kafka Streams:

KGroupedStream<String, String> grouped = source.groupByKey();
KTable<Windowed<String>, Long> windowedCount = grouped
  .windowedBy(SlidingWindows.ofTimeDifferenceAndGrace(
    Duration.ofMinutes(10), 
    Duration.ofSeconds(30)
  ))
  .count();

The grace period allows late events to be included in the window. If you do not configure a grace period, late events may be dropped entirely.

Conclusion

Stream processing patterns are about managing continuous data flows with clear boundaries and well-defined behaviors. These patterns focus on how to structure logic around message consumption, aggregation, transformation, and failure handling without relying on specific vendor implementations or tools. The key is to understand the flow of events and ensure that processing remains consistent and reliable even under pressure or failure conditions.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.