Real-Time Streaming with Kafka: Architecture Guide

This guide describes how to build and operate real-time streaming systems using Apache Kafka as the core messaging layer. It applies when data must flow continuously from producers to consumers with low latency, typically under 100 milliseconds end-to-end. This architecture is essential for live dashboards, real-time analytics, event-driven microservices, and streaming ETL pipelines.

Creating Topics and Configuring Partitioning

Use the kafka-topics.sh script to create topics with appropriate partition counts:

bin/kafka-topics.sh --create \
  --topic user_events \
  --bootstrap-server localhost:9092 \
  --partitions 12 \
  --replication-factor 3 \
  --config retention.ms=604800000 \
  --config segment.bytes=1073741824

Each partition is a separate queue that can be consumed in parallel. Use the --config flag to set topic-level retention and segment size. The default replication factor of 3 ensures availability even when one broker fails.

Avoid creating topics with too few partitions. Kafka's partitioning model means that each consumer in a group can only consume from one partition at a time, so 100 partitions allow for 100 concurrent consumers. Too few partitions limit throughput and create bottlenecks.

Producing Messages

Produce messages using kafka-console-producer.sh:

bin/kafka-console-producer.sh \
  --topic user_events \
  --bootstrap-server localhost:9092 \
  --property parse.key=true \
  --property key.separator=:

Input lines must follow the format key:value. For example:

user_123:{"event_type":"login","timestamp":1678886400}
user_456:{"event_type":"purchase","amount":99.99}

Kafka does not guarantee delivery order across partitions, but it preserves order within each partition. Messages without a key are assigned to partitions round-robin.

Consuming Messages

Consume messages using kafka-console-consumer.sh:

bin/kafka-console-consumer.sh \
  --topic user_events \
  --bootstrap-server localhost:9092 \
  --from-beginning \
  --formatter kafka.tools.DefaultMessageFormatter

Use the --from-beginning flag to read all messages, including those that were already consumed. The default consumer group name is console-consumer-<random_id>.

Consumer Groups and Offset Management

Consumer groups allow multiple consumers to work together on a topic. Each partition is assigned to exactly one consumer in the group:

bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group user_events_consumer_group

The output shows each partition's current owner, lag, and committed offset. If a consumer fails, its partitions are reassigned to other consumers in the same group.

Use --reset-offsets to manually manage offsets:

bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --group user_events_consumer_group \
  --reset-offsets \
  --to-earliest \
  --execute

This command resets the offset of all partitions to the earliest available message. If you run without --execute, it shows what would happen.

Handling Consumer Failures

Kafka uses a group coordinator to manage consumer lifecycle. When a consumer stops, Kafka waits for the configured session.timeout.ms (default 10 seconds) before considering the consumer dead and rebalancing partitions.

If consumers are too slow, they may hit the max.poll.interval.ms limit (default 5 minutes). If a consumer takes longer than this to poll messages, Kafka considers it dead and triggers a rebalance. This can cause duplicate processing or message loss if not handled carefully.

Configuring Retention and Cleanup

Kafka retains messages for a configurable time period. Set retention at topic level:

bin/kafka-topics.sh --alter \
  --topic user_events \
  --bootstrap-server localhost:9092 \
  --config retention.ms=2592000000

This sets retention to 30 days (2592000000 milliseconds). Use --config for all retention-related settings. Kafka cleans up old segments based on retention.bytes or retention.ms.

If a consumer group is inactive for longer than the retention period, messages may be deleted before they are consumed, leading to data loss.

Monitoring Partition Lag

Use kafka-consumer-groups.sh to check consumer lag:

bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group user_events_consumer_group \
  --verbose

The output includes the Lag column, which shows how many messages are behind. A growing lag indicates a consumer is too slow or has failed.

Rebalancing and Consumer Group Management

Consumer groups rebalance when new consumers join or leave. During rebalancing, all consumers pause processing while Kafka reassigns partitions.

Use --max.poll.records to control how many records a consumer fetches per poll cycle:

bin/kafka-console-consumer.sh \
  --topic user_events \
  --bootstrap-server localhost:9092 \
  --property print.key=true \
  --property key.separator=:

The default max.poll.records is 500. If a consumer is slow, it may not process messages quickly enough to avoid timeouts.

Dead Letter Queues and Error Handling

Kafka does not provide built-in dead letter queue handling. Implement error handling by having consumers send failed messages to a separate topic:

bin/kafka-topics.sh --create \
  --topic user_events_dlq \
  --bootstrap-server localhost:9092 \
  --partitions 6 \
  --replication-factor 3

In your consumer code, catch exceptions and write failed messages to the DLQ topic. This avoids blocking the entire pipeline when a single message fails.

Partition Key Strategy

Choose partition keys that distribute load evenly. Keys should be evenly distributed across the number of partitions. For example, if you have 12 partitions, a key that's always the same value will send all messages to one partition.

Avoid using timestamps or IDs that are monotonically increasing for partition keys. Instead, use a hash of user ID or event type to distribute messages evenly.

Message Serialization and Schema Evolution

Kafka stores messages as bytes. Use a serialization format like JSON, Avro, or Protobuf. For Avro, store schemas in a schema registry. When consuming, ensure your consumer can deserialize messages correctly.

If schemas evolve over time, use schema versioning. Kafka's --property key.separator and --property parse.key=true are important for correct message parsing during production.

Configuration for High Throughput

For high-throughput topics, tune these parameters:

bin/kafka-topics.sh --create \
  --topic user_events \
  --bootstrap-server localhost:9092 \
  --partitions 12 \
  --replication-factor 3 \
  --config retention.ms=604800000 \
  --config segment.bytes=1073741824 \
  --config max.message.bytes=1048588

The max.message.bytes setting controls the maximum message size. Messages larger than this may be rejected.

Handling Message Loss

Kafka's default settings are designed for durability, but they can still lose messages in failure scenarios. Use acks=all when producing to ensure messages are replicated to all in-sync replicas before acknowledging:

bin/kafka-console-producer.sh \
  --topic user_events \
  --bootstrap-server localhost:9092 \
  --property acks=all

If a producer sends a message and the broker fails before replication completes, that message is lost.

Consumer Offset Committing

Kafka supports both automatic and manual offset committing. Automatic commits happen every auto.commit.interval.ms (default 5 seconds). For high accuracy, use manual commits:

bin/kafka-console-consumer.sh \
  --topic user_events \
  --bootstrap-server localhost:9092 \
  --enable-auto-commit=false

Manual committing ensures that a consumer only acknowledges messages after they have been successfully processed. If you don't commit offsets, the same messages will be reprocessed on restart.

Scaling Consumers

Add consumers to a group to scale processing. Kafka automatically redistributes partitions among available consumers. The number of consumers should not exceed the number of partitions in a topic. Adding more consumers than partitions is wasteful.

Use consistent consumer group names to ensure consumers can resume from where they left off. If a consumer joins with a different group name, it starts from the beginning of the topic.

Partition Rebalancing Considerations

When partition rebalancing occurs, there is a brief pause while Kafka redistributes partitions. This can cause temporary drops in throughput or increase latency for some consumers.

If you have a large number of consumers, consider using --consumer-property to configure advanced settings:

bin/kafka-console-consumer.sh \
  --topic user_events \
  --bootstrap-server localhost:9092 \
  --consumer-property session.timeout.ms=30000

The default session.timeout.ms is 10000 milliseconds. Increase this value for consumers that may take longer to process messages.

Producer Acknowledgment Settings

Use acks configuration when producing:

• acks=0: Producer does not wait for acknowledgment from broker.
• acks=1: Producer waits for leader acknowledgment only.
• acks=all: Producer waits for all in-sync replicas.

Use acks=all for critical data where no message loss is acceptable. Use acks=1 for higher throughput with lower durability guarantees.

Handling Consumer Timeout

If consumers take longer than max.poll.interval.ms, Kafka considers them dead and triggers a rebalance. This can cause duplicate processing. Increase this value if your consumers are slow:

bin/kafka-console-consumer.sh \
  --topic user_events \
  --bootstrap-server localhost:9092 \
  --consumer-property max.poll.interval.ms=300000

This sets the maximum time between polls to 5 minutes. Use this setting when processing messages takes longer than the default 5-minute window.

Topic Cleanup and Log Retention

Kafka's log cleaner handles compaction of topics with keys. Enable compaction on topics that need to store only the latest value for each key:

bin/kafka-topics.sh --create \
  --topic user_profiles \
  --bootstrap-server localhost:9092 \
  --partitions 6 \
  --replication-factor 3 \
  --config cleanup.policy=compact \
  --config min.compaction.lag.ms=300000

The min.compaction.lag.ms setting controls how long a message must be in the log before compaction can occur.

Error Conditions and Failure Modes

Kafka does not validate message keys or values during production. If a message is malformed, consumers may fail silently. Always validate message format in consumer code.

If a consumer group has no active consumers for longer than retention.ms, messages may be deleted before being consumed. This is especially important for low-volume topics.

Kafka's default max.poll.records of 500 can cause issues if consumers take longer than expected to process each record. If processing takes more than max.poll.interval.ms, the consumer will be considered dead and trigger a rebalance.

If you use --from-beginning in a consumer, it reads all messages from the beginning of the topic. This is useful for testing but can cause performance issues on large topics with long retention periods.

Conclusion

Kafka's real-time streaming architecture requires careful attention to partitioning strategy, consumer group management, and offset handling. The system works best when consumers are designed to be stateless and can handle rebalancing gracefully. Message ordering is preserved within partitions but not across partitions, so design your keys accordingly.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.