Production engineering guide for event driven design patterns covering patterns, implementation strategies, and operational best practices.
Event-driven design patterns enable systems to react to changes in state through explicit events, decoupling producers and consumers. Use this approach when components must scale independently, respond to real-time changes, or coordinate across services without tight coupling. It matters most when data flows through multiple services, and the sequence of actions depends on state changes—such as order processing, user onboarding, or real-time analytics.
Publishing events begins with a well-defined schema and a reliable delivery mechanism. Events are structured as JSON payloads with a consistent top-level shape:
{
"id": "7e6f8b2c-1a9d-4c3e-9f0a-2b1c4d5e6f7a",
"type": "user.created",
"source": "auth-service",
"timestamp": "2025-04-05T12:34:56.789Z",
"data": {
"userId": "u12345",
"email": "[email protected]",
"name": "Alice Smith",
"metadata": {
"origin": "web",
"ipAddress": "192.168.1.10"
}
}
}Events are published via a message broker such as Apache Kafka, RabbitMQ, or Amazon SQS. Each event is written to a topic (Kafka), exchange (RabbitMQ), or queue (SQS) with a consistent naming convention: domain.event-type, such as users.created, orders.updated, payments.failed.
Consumers read events from a topic or queue using a consumer group or subscription. A typical consumer loop using kafka-console-consumer.sh reads from a topic with offset tracking:
kafka-console-consumer.sh \
--bootstrap-server kafka:9092 \
--topic users.created \
--group-id user-service-consumer \
--property print.key=true \
--property print.value=true \
--property key.separator=" | "Consumers must process events idempotently: duplicate or out-of-order events must not cause side effects. The id field in the event is used to deduplicate using a deduplication store (e.g., a PostgreSQL table):
CREATE TABLE event_dedup (
event_id TEXT PRIMARY KEY,
processed_at TIMESTAMPTZ NOT NULL,
processed_by TEXT,
metadata JSONB
);Failures manifest as unprocessed events, lagging consumer groups, or duplicate processing. The most common silent failure is missing event_id in the payload—when consumers assume it’s present, but some producers forget to set it. A missing id field causes repeated processing of the same event, especially when the message broker retries delivery.
Event sourcing captures state changes as a sequence of events rather than storing only the current state. The domain state is reconstructed by replaying all events since the beginning.
Each aggregate (e.g., Order, User) is identified by an aggregateId. Events are stored in a events table with schema:
CREATE TABLE events (
id UUID PRIMARY KEY,
aggregate_id TEXT NOT NULL,
version INTEGER NOT NULL,
type TEXT NOT NULL,
data JSONB NOT NULL,
metadata JSONB,
created_at TIMESTAMPTZ NOT NULL,
INDEX idx_aggregate_id (aggregate_id),
INDEX idx_version (version)
);To retrieve the current state of an order, a service queries all events for that aggregateId and applies them in order:
def load_order(order_id: str) -> Order:
events = db.query("""
SELECT id, type, data, metadata
FROM events
WHERE aggregate_id = %s
ORDER BY version
""", [order_id])
order = Order(id=order_id)
for event in events:
apply_event(order, event)
return orderThe version field ensures consistency: each event increments the version, and a write operation must include the expected version. A ConcurrentUpdateException occurs when a command tries to apply an event to a version that no longer matches:
{
"error": "concurrent_update",
"expectedVersion": 5,
"actualVersion": 4,
"eventId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
"aggregateId": "order-123"
}The most subtle failure is missing versioning during event application. If a consumer applies an event without checking the current version, two concurrent updates can overwrite each other without conflict. The pattern requires both version checking and event replaying for consistency.
CQRS separates reading and writing operations into distinct models. The write model is event-driven, and the read model is updated asynchronously via event handlers.
A command (e.g., CreateUserCommand) is dispatched to a command handler, which:
The read model is built from these events. For example, when UserCreated is published, a handler updates a user_profiles table:
CREATE TABLE user_profiles (
user_id TEXT PRIMARY KEY,
email TEXT NOT NULL,
name TEXT,
created_at TIMESTAMPTZ,
updated_at TIMESTAMPTZ
);Each event handler runs as a separate service, consuming events from a topic and updating the read model:
def handle_user_created(event: dict):
user_id = event["data"]["userId"]
email = event["data"]["email"]
name = event["data"]["name"]
db.execute("""
INSERT INTO user_profiles (user_id, email, name, created_at)
VALUES (%s, %s, %s, %s)
ON CONFLICT (user_id) DO UPDATE SET
email = EXCLUDED.email,
name = EXCLUDED.name,
updated_at = NOW()
""", [user_id, email, name, event["timestamp"]])The read model may be denormalized and optimized for queries. A user_search index is maintained with a user_search table that includes derived fields like full_name, active_status, and last_login.
A common failure occurs when an event handler processes an event but fails before committing the update. The event is consumed, but the read model is inconsistent. To fix this, handlers use idempotent updates: they track which events have been processed and retry on failure. The events_processed table records each event ID and the timestamp of processing.
Orchestration coordinates multiple services through a sequence of steps, where each step is triggered by an event.
A typical workflow for processing an order:
OrderCreated → start order processing.OrderShipped → notify customer.OrderPaid → update inventory.OrderDelivered → close order and send feedback request.Each step is a service that listens to events and emits the next. A workflow service maintains the state of each order using a order_states table:
CREATE TABLE order_states (
order_id TEXT PRIMARY KEY,
status TEXT NOT NULL,
current_step TEXT,
metadata JSONB,
updated_at TIMESTAMPTZ
);The workflow engine listens to OrderCreated and starts the process:
def start_order_workflow(event: dict):
order_id = event["data"]["orderId"]
db.execute("""
INSERT INTO order_states (order_id, status, current_step)
VALUES (%s, 'processing', 'order_created')
""", [order_id])
# Publish next event
publish_event({
"id": generate_uuid(),
"type": "inventory.reserve",
"data": { "orderId": order_id },
"source": "workflow-engine"
})When the inventory service reserves stock, it emits InventoryReserved. The workflow engine listens for this and transitions the order:
def handle_inventory_reserved(event: dict):
order_id = event["data"]["orderId"]
db.execute("""
UPDATE order_states
SET status = 'reserved',
current_step = 'inventory_reserved',
updated_at = %s
WHERE order_id = %s
""", [event["timestamp"], order_id])
publish_event({
"id": generate_uuid(),
"type": "payment.process",
"data": { "orderId": order_id },
"source": "workflow-engine"
})A failure mode is event loss: if InventoryReserved is published but the workflow engine crashes before processing it, the order remains in reserved state indefinitely. To recover, the workflow engine scans all events and reprocesses missing steps. A common mistake is not setting id on events in the workflow—without it, replaying the same event multiple times causes duplicate transitions.
Data pipelines process large volumes of events to generate analytics, summaries, or materialized views.
A pipeline reads from a raw_events topic and produces aggregated_events for daily reports:
kafka-topics.sh --bootstrap-server kafka:9092 \
--create --topic raw_events \
--partitions 10 --replication-factor 3
kafka-topics.sh --bootstrap-server kafka:9092 \
--create --topic daily_summary \
--partitions 4 --replication-factor 3A stream processor written in Python consumes raw_events, groups events by day, and emits summary events:
from datetime import date
from collections import defaultdict
def process_events():
consumer = KafkaConsumer(
'raw_events',
bootstrap_servers=['kafka:9092'],
value_deserializer=lambda m: json.loads(m.decode('utf-8')),
group_id='daily-summary-processor'
)
for message in consumer:
event = message.value
event_date = date.fromisoformat(event['timestamp'][:10])
summary_key = f"{event_date}-{event['type']}"
# Accumulate counts per type per day
daily_stats[summary_key] += 1
daily_total += 1
# Emit summary every 5 minutes or every 100 events
if len(daily_stats) % 100 == 0 or time() - last_emit > 300:
emit_summary(daily_stats, total=daily_total)
last_emit = time()
daily_stats.clear()The output is emitted as daily_summary events:
{
"id": "d12345",
"type": "daily.summary",
"data": {
"date": "2025-04-05",
"eventTypes": {
"user.created": 142,
"order.created": 89,
"payment.completed": 201
},
"totalEvents": 432
}
}A silent failure is late data: events arrive hours after their timestamp. Without watermarking, summary events are inaccurate. A second failure mode is event deduplication across parallel processors. If two instances of the processor run concurrently, they may emit overlapping summaries. To fix this, processors use a checkpoint table to track which events have been processed and use id to ensure no duplicates.
All event-driven systems must handle three guarantees: at-least-once, at-most-once, and exactly-once.
At-least-once is common: message brokers replay messages on failure. To achieve this, consumers use a processed_events table:
CREATE TABLE processed_events (
event_id TEXT PRIMARY KEY,
consumer_group TEXT,
processed_at TIMESTAMPTZ,
metadata JSONB
);Consumers check this table before processing each event:
def process_event(event: dict):
event_id = event["id"]
consumer_group = "user-service"
if db.query("""
SELECT 1 FROM processed_events
WHERE event_id = %s AND consumer_group = %s
""", [event_id, consumer_group]):
return # Already processed
# Apply the event logic
apply_event_logic(event)
# Mark as processed
db.execute("""
INSERT INTO processed_events (event_id, consumer_group, processed_at)
VALUES (%s, %s, %s)
""", [event_id, consumer_group, now()])At-most-once is achieved by setting acknowledge: true in the message broker. The broker removes the message after delivery, but if the consumer crashes before acknowledging, the message is lost.
Exactly-once requires both idempotent processing and a reliable transactional delivery mechanism. The most common approach is to use a two-phase commit: the consumer writes to the database and then sends an acknowledgment. If the consumer fails after writing but before acknowledging, the broker retries delivery and the idempotent handler processes it once.
The most subtle failure is mismatched version in event processing. If an event schema evolves, older consumers may not understand new fields. A user.created event from 2024 may lack metadata.origin, but a new consumer expects it. The system silently fails when the origin field is used but is null or missing.
Observability in event-driven systems relies on tracing, logging, and monitoring of event flows.
Each event carries a traceId and spanId:
{
"id": "e12345",
"type": "order.created",
"traceId": "t789abc",
"spanId": "s12345",
"data": { ... }
}Services log each event processing step with structured logs:
{
"timestamp": "2025-04-05T12:34:56.123Z",
"level": "info",
"service": "order-service",
"event": "order.created",
"traceId": "t789abc",
"spanId": "s12345",
"processed": true,
"durationMs": 142,
"event": {
"id": "e12345",
"data": { "orderId": "o123" }
}
}A failure in the pipeline is diagnosed by tracing a traceId across services. If an order.created event takes 3 seconds but a user.created event takes 200ms, the trace reveals the bottleneck.
The most common silent issue is missing traceId in event payloads. Without it, logs from different services cannot be correlated. Another failure is delayed processing: an event is published but not consumed for hours. Monitoring must track lag across consumer groups and alert when lag exceeds a threshold.
Event-driven systems are powerful but require disciplined handling of idempotency, versioning, deduplication, and tracing. Their success lies not in the tools, but in the rigor of event shape, processing logic, and observability.
This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.
We use cookies for analytics (Google Analytics) and advertising (Google AdSense) to improve your experience and support free content. Privacy Policy