Build production streaming systems. Covers Kafka, Flink, Kinesis, event schema design, exactly-once processing, stream-table duality, windowing, and backpressure management.
This is an architecture pattern for systems that process data as it arrives, rather than in batches. It applies when data must be consumed within milliseconds or seconds of being generated, typically for event-driven applications, real-time dashboards, or immediate alerting.
Use a standard consumer group with a message broker like Apache Kafka to consume from topics:
kafka-console-consumer.sh \
--bootstrap-server kafka-broker:9092 \
--topic user-events \
--group event-processor \
--from-beginningThe key flag is --group, which defines the consumer group membership. Each consumer in the same group receives a partition assignment, and consumers in different groups process the same messages independently.
Set offset management strategy via consumer configuration:
# consumer.properties
group.id=event-processor
enable.auto.commit=true
auto.commit.interval.ms=5000
session.timeout.ms=30000
heartbeat.interval.ms=3000The enable.auto.commit flag controls whether the system automatically records offsets after each poll. When set to false, you must manually commit offsets using commitSync() or commitAsync().
A consumer group can trigger a rebalance when new consumers join or leave. During this process, partitions are reassigned, and processing may pause until the new assignment is complete.
Use a single consumer instance to subscribe to several topics:
kafka-console-consumer.sh \
--bootstrap-server kafka-broker:9092 \
--topic user-events,user-actions \
--group unified-processorThe --topic flag accepts comma-separated topic names. A consumer can only belong to one group, so it processes messages from all subscribed topics.
Use --value-deserializer and --key-deserializer flags for structured data:
kafka-console-consumer.sh \
--bootstrap-server kafka-broker:9092 \
--topic user-events \
--group event-processor \
--value-deserializer=org.apache.kafka.common.serialization.StringDeserializer \
--key-deserializer=org.apache.kafka.common.serialization.IntegerDeserializerDeserialization errors silently fail unless you configure error handling. When deserialization fails, messages are skipped and logged.
Use kafka-console-producer.sh to forward messages:
kafka-console-producer.sh \
--broker-list kafka-broker:9092 \
--topic enriched-events \
--property parse.key=true \
--property key.separator=:The --property parse.key=true flag indicates that input data contains keys separated by the key.separator.
Set producer acknowledgment level for reliability:
# producer.properties
acks=1
retries=3
batch.size=16384
linger.ms=5With acks=1, the producer waits for the leader to write messages before considering them acknowledged. With acks=0, it sends messages without waiting, which can cause data loss.
Use Kafka's built-in metrics:
kafka-consumer-groups.sh \
--bootstrap-server kafka-broker:9092 \
--describe \
--group event-processorThis command shows Lag for each partition, indicating how many messages behind the consumer is. A growing lag signals potential performance issues or system overload.
Use a standard HTTP client with timeout configuration:
curl -X POST http://sink-service:8080/events \
--connect-timeout 5 \
--max-time 10 \
--data-binary @- \
--header "Content-Type: application/json"If the sink service does not respond within the timeout, the stream processor fails the message and retries with exponential backoff.
When a consumer dies unexpectedly, Kafka triggers a rebalance. If this happens too frequently, you'll see repeated Rebalance started messages in logs.
Use the following to check group health:
kafka-consumer-groups.sh \
--bootstrap-server kafka-broker:9092 \
--describe \
--group event-processor \
--verboseThis includes Consumer ID and Host fields that help identify problematic consumers.
Use a simple Kafka Streams application:
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> source = builder.stream("user-events");
KStream<String, String> processed = source.mapValues(value -> value.toUpperCase());
processed.to("processed-events");The mapValues() operation applies a transformation to each message. If a transformation throws an exception, the stream stops processing until it is restarted.
Define a RocksDB-based state store:
KTable<String, Long> counts = builder.table(
"user-events",
Materialized.<String, Long>as("event-counts")
.withKeySerde(Serdes.String())
.withValueSerde(Serdes.Long())
);State stores are local to the application and persist data in memory or on disk. When a state store becomes corrupted, it may cause a full restart of the processing pipeline.
Adjust poll interval and batch size:
# consumer.properties
fetch.min.bytes=1024
fetch.max.wait.ms=500
max.poll.records=1000The fetch.min.bytes parameter ensures that a consumer waits for at least 1024 bytes of data before returning from a poll. This reduces the number of small fetches and improves throughput.
Use HTTP headers to control message rate:
curl -H "X-RateLimit-Limit: 100" \
-H "X-RateLimit-Remaining: 99" \
http://sink-service:8080/eventsA sink that does not implement rate limiting can become overwhelmed and drop messages. If a sink returns a 429 status code, the stream processor should retry with backoff.
Monitor Kafka metrics using JMX:
jconsole kafka-broker:9999Check Consumer and Producer MBeans for lag and throughput. Look for records-lag-max in consumer metrics to detect slow consumers.
Query metrics from a monitoring system like Prometheus:
max(kafka_consumer_fetch_manager_records_lag_max{group="event-processor"})If this value exceeds 1000 messages, trigger an alert. Lag can accumulate over time and indicate processing bottlenecks.
A growing consumer lag indicates that the processing rate is slower than the ingestion rate. This is not always visible in logs but becomes apparent through metrics.
When a consumer fails during processing, it may reprocess messages already handled. The enable.auto.commit=false setting requires manual offset management to prevent duplicates.
If a message cannot be deserialized, it silently skips and logs the error. This is not an exception but a silent data loss event. Always validate serialization before deploying to production.
A producer may timeout while waiting for acknowledgment from brokers. If retries are exhausted, messages are discarded.
State stores may become corrupted due to disk errors or memory issues. A corrupted store requires a full reset and replay of the stream. This can take hours depending on data volume.
If partitions are unevenly distributed across consumers, some consumers may be overloaded while others idle. Use kafka-reassign-partitions.sh to rebalance partitions manually.
To increase throughput, add more consumers to the same group:
kafka-console-consumer.sh \
--bootstrap-server kafka-broker:9092 \
--topic user-events \
--group event-processor \
--consumer-property num.stream.threads=4Each consumer in a group processes one or more partitions. The number of consumers should not exceed the number of partitions for optimal performance.
Use HTTP endpoints to expose readiness and liveness:
curl -f http://localhost:8080/healthA service that does not respond to /health is considered unhealthy and may be removed from a load balancer.
When updating consumer or producer settings, restart the application. Running with incompatible settings causes data loss or processing errors. Always validate configurations before deployment.
If a new version of the stream processor fails, rollback to the previous version using:
kubectl rollout undo deployment/stream-processorThis preserves the last known good configuration and avoids partial failures.
This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy