Streaming Pipeline Patterns

This reference describes specific architectural approaches for building data pipelines that process continuous streams of events rather than discrete batches. These patterns apply when you need to handle real-time data flows where latency requirements exceed batch processing intervals, or when systems must react to events as they occur.

Processing Events as They Arrive

Fan-out Pattern

When a single event source must feed multiple downstream consumers, use a fan-out approach with message queuing. The event producer sends each message to a topic or queue that multiple consumers can subscribe to independently.

# Example using Kafka command line tools
kafka-topics.sh --create --topic user-events \
  --bootstrap-server kafka-broker:9092 \
  --partitions 6 --replication-factor 3

# Consumer groups process the same topic
kafka-console-consumer.sh --topic user-events \
  --bootstrap-server kafka-broker:9092 \
  --group analytics-group

kafka-console-consumer.sh --topic user-events \
  --bootstrap-server kafka-broker:9092 \
  --group reporting-group

The key configuration is the consumer group ID. Messages are distributed among consumers within the same group, but each consumer receives a copy of every message when different groups are used.

Processing Order and State

When event order matters for processing, use partitioning to ensure related events go to the same processing unit. For example, if processing user actions by user ID, partition by user_id to maintain ordering within partitions.

# Kafka producer with key-based partitioning
kafka-console-producer.sh --topic user-events \
  --bootstrap-server kafka-broker:9092 \
  --property parse.key=true \
  --property key.separator=:

Messages must have keys that are consistent across the stream to preserve order within partitions. If you do not specify a key, messages may be distributed randomly across partitions, breaking order guarantees.

Managing Data Flow Control

Backpressure Handling

When downstream systems cannot keep up with upstream event rates, implement backpressure by pausing or throttling producers. This can be achieved through consumer group coordination or explicit rate limiting.

# Example using Kafka consumer configuration
kafka-console-consumer.sh --topic user-events \
  --bootstrap-server kafka-broker:9092 \
  --max-poll-records 100 \
  --session-timeout-ms 10000 \
  --heartbeat-interval-ms 3000

The max-poll-records parameter limits how many records are fetched per poll cycle. If processing takes longer than the heartbeat interval, the consumer group will rebalance and potentially reprocess messages.

Dead Letter Queue (DLQ) Strategy

When events fail processing, route them to a separate queue for later inspection or retry. This prevents failed messages from blocking subsequent messages in the same stream.

# Configure Kafka consumer to send failures to DLQ topic
kafka-console-consumer.sh --topic user-events \
  --bootstrap-server kafka-broker:9092 \
  --property value.deserializer=org.apache.kafka.common.serialization.StringDeserializer \
  --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer \
  --offset.reset=earliest \
  --max.poll.records=100

Handling Failures and Retries

Idempotent Processing

Ensure that processing the same event multiple times produces the same result. This is critical for handling network failures or consumer restarts where events may be reprocessed.

# Example idempotent processing function
def process_event(event):
    # Use unique event ID to prevent duplicate processing
    event_id = event['id']
    
    # Check if already processed
    if check_processed(event_id):
        return
    
    try:
        # Process the event
        result = transform_event(event)
        save_result(result)
        
        # Mark as processed
        mark_processed(event_id)
    except Exception as e:
        # Log failure but do not mark as processed
        log_failure(event_id, str(e))
        raise  # Re-raise to trigger retry mechanism

Retry with Exponential Backoff

When processing fails, implement retries with increasing delays between attempts.

# Consumer configuration for retry behavior
kafka-console-consumer.sh --topic user-events \
  --bootstrap-server kafka-broker:9092 \
  --retries=3 \
  --retry.backoff.ms=1000 \
  --max.poll.interval.ms=300000

The max.poll.interval.ms controls how long a consumer can take to process messages before being considered dead and triggering a rebalance. If processing takes longer than this interval, the consumer group will rebalance, potentially causing duplicate processing.

Managing Data Consistency

Exactly-Once Processing Semantics

Use transactional producers and consumer groups to ensure each message is processed exactly once, even if failures occur during processing.

# Kafka transactional producer example
kafka-console-producer.sh --topic user-events \
  --bootstrap-server kafka-broker:9092 \
  --transactional.id=processing-1 \
  --enable.idempotence=true \
  --max.in.flight.requests.per.connection=5

The enable.idempotence=true flag ensures that producers can handle duplicate messages without creating duplicates in the destination. The transactional.id enables transactional semantics.

Windowed Processing

Aggregate events over time windows to build summaries or perform calculations that span multiple events.

-- Example SQL for windowed aggregation
SELECT 
    window_start,
    window_end,
    user_id,
    count(*) as event_count,
    avg(event_value) as average_value
FROM (
    SELECT 
        user_id,
        event_value,
        tumble(event_time, INTERVAL '10' MINUTE) as window
    FROM events_stream
) 
GROUP BY window_start, window_end, user_id

The tumble function creates fixed-time windows. If a window spans across midnight, you must handle the boundary correctly to avoid splitting events.

Controlling Data Flow with State

Rate Limiting by Source

When processing multiple sources, implement rate limiting to prevent overwhelming downstream systems.

# Kafka consumer with throttling
kafka-console-consumer.sh --topic user-events \
  --bootstrap-server kafka-broker:9092 \
  --max.poll.records=10 \
  --poll.interval.ms=1000

The poll.interval.ms parameter controls how frequently the consumer polls for new messages. Setting this too low can cause excessive polling overhead, while setting it too high can delay processing.

Conditional Processing

Only process events that meet specific criteria to reduce unnecessary work.

# Example conditional processing logic
def should_process_event(event):
    # Filter based on event type or content
    if event['type'] not in ['click', 'purchase']:
        return False
    
    # Filter based on business rules
    if event['user_id'] in ['internal_user_1', 'internal_user_2']:
        return False
    
    return True

def process_event(event):
    if not should_process_event(event):
        return
    
    # Process only qualifying events
    ...

Monitoring and Observability

Processing Lag Detection

Monitor how far behind the current time events are to detect processing delays.

# Kafka consumer lag monitoring
kafka-consumer-groups.sh --bootstrap-server kafka-broker:9092 \
  --describe --group analytics-group

The output includes Lag column showing how many messages behind the consumer is. If this value grows over time, it indicates processing is not keeping up.

Event Age Monitoring

Track how old events are when they reach processing to detect data freshness issues.

# Example age monitoring in processing function
def process_event(event):
    event_age = datetime.now() - event['timestamp']
    
    # Log if event is older than 5 minutes
    if event_age > timedelta(minutes=5):
        log_warning(f"Event is {event_age} old")
    
    # Process normally
    ...

Failure Modes and Sharp Edges

Consumer Group Rebalancing

When consumers join or leave a group, rebalancing can cause processing gaps or duplicates.

The session.timeout.ms controls how long the consumer can go without heartbeats before being considered dead. If this timeout is too short relative to processing time, consumers will be removed from the group unnecessarily.

Partition Assignment Issues

When partitions are reassigned, consumers may lose track of their current position.

Always use consistent partitioning keys across all consumers in a group to avoid message duplication or loss during rebalancing.

Serialization Errors

Misconfigured serialization can cause entire streams to stop processing.

If a consumer cannot deserialize messages from a topic, it will stop consuming and potentially fail to advance offsets. Always configure appropriate deserializers for each topic.

Memory Leaks in Long-Running Consumers

Consumers that run for extended periods may accumulate memory if not properly managing state or resources.

Monitor consumer memory usage and ensure proper cleanup of temporary objects between processing cycles.

Offset Management Failures

If consumers fail to commit offsets correctly, they may reprocess the same messages repeatedly.

The enable.auto.commit setting controls whether offsets are automatically committed. If set to false, you must manually commit offsets or risk message loss during failures.

Batch Size and Processing Time Mismatch

Setting max.poll.records too high relative to processing time can cause consumers to timeout and trigger rebalances.

Always measure actual processing time for your events and adjust batch sizes accordingly.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.