Cloud Event Driven Architecture

Cloud Event Driven Architecture (EDA) is a design pattern where components communicate through events—discrete, meaningful facts about state changes in a system. It matters when systems must react to real-time data, scale independently, and tolerate failures gracefully: in microservices, real-time analytics, order processing, or IoT pipelines. Events decouple producers and consumers, enabling systems to evolve without coordination.

Core Components and Their Roles

An event is a self-contained message containing metadata and a payload. It is emitted when a significant change occurs—such as OrderCreated, PaymentProcessed, or UserSignedUp. The architecture relies on a message broker to store and deliver events reliably. The broker must persist events across restarts, support multiple consumers, and handle backpressure.

The primary operations are:

• Publishing an event: a producer sends a new event to the broker.
• Subscribing to events: a consumer registers interest in specific event types.
• Processing events: consumers read events from the broker and act on them.

The message broker typically supports:

• At-least-once delivery via persistent queues.
• Topic-based routing: events are published to topics (e.g., orders, users), and consumers subscribe to one or more topics.
• Consumer groups: multiple instances of a service can share the load of processing events from a single topic.

Publishing Events

A producer emits an event by sending a structured message to a broker endpoint. The message must include:

• event_id (UUID)
• event_type (e.g., OrderCreated)
• timestamp (ISO 8601)
• data (JSON object)
• schema_id (optional, for schema evolution)

Example of a published event:

{
  "event_id": "a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8",
  "event_type": "OrderCreated",
  "timestamp": "2024-09-18T14:32:15.123Z",
  "data": {
    "order_id": "ORD-2024-09-18-001",
    "customer_id": "CUST-1001",
    "items": [
      { "product_id": "P-101", "quantity": 2 },
      { "product_id": "P-102", "quantity": 1 }
    ],
    "total_amount": 159.99
  },
  "schema_id": "order.v1"
}

Publishing is done over HTTP POST to a topic endpoint:

curl -X POST https://events.example.com/topics/orders \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..." \
  -d '{
    "event_id": "a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8",
    "event_type": "OrderCreated",
    "timestamp": "2024-09-18T14:32:15.123Z",
    "data": {
      "order_id": "ORD-2024-09-18-001",
      "customer_id": "CUST-1001",
      "items": [
        { "product_id": "P-101", "quantity": 2 },
        { "product_id": "P-102", "quantity": 1 }
      ],
      "total_amount": 159.99
    },
    "schema_id": "order.v1"
  }'

The broker returns a 202 Accepted with the assigned event_id and offset in the response:

{
  "event_id": "a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8",
  "offset": 12345,
  "status": "accepted",
  "message": "Event successfully queued for delivery"
}

Consuming Events

Consumers poll or stream events from a topic. The most common approach is pull-based consumption using a start_offset and batch_size parameter.

A consumer reads events from a topic with:

• topic: the name of the topic (e.g., orders)
• group_id: the consumer group (e.g., order-processing)
• start_offset: the position in the log to begin reading
• batch_size: number of events to fetch per request

curl -X GET "https://events.example.com/topics/orders?group_id=order-processing&start_offset=12345&batch_size=10" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."

The response is a list of events:

{
  "events": [
    {
      "event_id": "a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8",
      "event_type": "OrderCreated",
      "timestamp": "2024-09-18T14:32:15.123Z",
      "data": {
        "order_id": "ORD-2024-09-18-001",
        "customer_id": "CUST-1001",
        "items": [
          { "product_id": "P-101", "quantity": 2 },
          { "product_id": "P-102", "quantity": 1 }
        ],
        "total_amount": 159.99
      },
      "schema_id": "order.v1"
    },
    {
      "event_id": "b2c3d4e5-f6g7-8901-h2i3-j4k5l6m7n8o9",
      "event_type": "OrderUpdated",
      "timestamp": "2024-09-18T14:32:16.456Z",
      "data": {
        "order_id": "ORD-2024-09-18-001",
        "status": "shipped",
        "tracking_number": "TRK-2024-09-18-001"
      },
      "schema_id": "order.v1"
    }
  ],
  "next_offset": 12357,
  "last_offset": 12357
}

The consumer updates its offset to next_offset and uses it in the next request.

Event Processing with Backpressure and Failure Handling

When a consumer processes events, it must acknowledge them upon successful handling. Acknowledgment prevents reprocessing and enables exactly-once semantics.

A consumer sends an acknowledgment:

curl -X PUT "https://events.example.com/consumers/order-processing/offsets" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..." \
  -d '{
    "group_id": "order-processing",
    "topic": "orders",
    "offset": 12357,
    "processed_at": "2024-09-18T14:33:00.000Z"
  }'

The broker records the offset per group and topic. If the consumer crashes after processing 10 events but before sending the ACK, the broker retries delivery from the last saved offset.

Failure modes:

• Event loss: if the broker crashes after accepting an event but before writing it to disk, the event may be lost.
• Duplicate processing: if the consumer acknowledges an event but the broker fails before persisting the offset, the same event may be processed twice.
• Stale offsets: if a consumer is slow and events arrive faster than it can process, the offset lags behind, leading to delayed processing.

To mitigate this, use idempotent event handlers:

• A handler checks whether an event has already been processed using event_id.
• A database table processed_events stores event_id and processed_at.

CREATE TABLE processed_events (
  event_id UUID PRIMARY KEY,
  processed_at TIMESTAMP WITH TIME ZONE,
  processed_by TEXT,
  metadata JSONB
);

When processing an event, the handler first checks:

SELECT event_id FROM processed_events WHERE event_id = 'a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8';

If present, it skips processing. If not, it processes the event and inserts the record.

Schema Evolution and Event Versioning

Events may evolve over time. A consumer must handle multiple versions of the same event type.

The schema_id field in the event payload allows versioning. The broker can store the schema for each schema_id and validate incoming events.

A schema might be:

{
  "title": "OrderCreated",
  "type": "object",
  "properties": {
    "order_id": { "type": "string" },
    "customer_id": { "type": "string" },
    "items": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "product_id": { "type": "string" },
          "quantity": { "type": "integer" }
        }
      }
    },
    "total_amount": { "type": "number" }
  },
  "required": ["order_id", "customer_id", "items", "total_amount"]
}

The broker validates incoming events against the schema. If validation fails, it returns:

{
  "error": "schema_validation_failed",
  "message": "Invalid schema for OrderCreated",
  "details": [
    "Missing required property: total_amount",
    "Invalid type for items: expected array, got object"
  ],
  "event_id": "a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8",
  "schema_id": "order.v1"
}

Consumers can handle schema evolution by:

• Reading schema_id and adapting their logic.
• Using schema registry services (e.g., Confluent Schema Registry, Avro schema repository).
• Supporting fallback logic for older schema versions.

Event-Driven Workflows and Chaining

Events can trigger cascading actions. A OrderCreated event may lead to:

• InvoiceGenerated (from an invoicing service)
• InventoryUpdated (from a stock service)
• EmailSent (from a notifications service)

Each service emits its own event, which may be consumed by other services.

A workflow might be:

OrderCreated → InvoiceGenerated → PaymentScheduled → ShipmentScheduled

This is achieved by:

• Each service publishing new events after processing.
• Using event types to route and trigger logic.

A service consuming OrderCreated may emit OrderProcessed:

{
  "event_id": "c3d4e5f6-g7h8-9012-i3j4-k5l6m7n8o9p1",
  "event_type": "OrderProcessed",
  "timestamp": "2024-09-18T14:33:15.000Z",
  "data": {
    "order_id": "ORD-2024-09-18-001",
    "invoice_id": "INV-2024-09-18-001",
    "payment_status": "scheduled",
    "shipment_status": "pending"
  },
  "schema_id": "order.processed.v1"
}

This event is published to the orders topic. Another service, shipment-service, subscribes to OrderProcessed and triggers a shipment.

Common Pitfalls and Sharp Edges

• Event drift: when consumers fall behind the event stream, leading to delayed processing. Monitoring lag (difference between latest and current offset) is critical.
• Schema mismatch: a consumer expects order.v1 but receives order.v2 with new fields. The consumer may fail silently if it does not handle unknown fields.
• Event ordering: events are not guaranteed to be delivered in order across topics. Use event_id and timestamp to reconstruct order.
• Event deduplication: a producer may emit the same event twice. Use event_id as a primary key to deduplicate.
• Idempotent processing: not all services are idempotent. A PaymentProcessed event may be applied twice, leading to duplicate payments.
• Failure to acknowledge: a consumer processes an event and sends ACK, but the broker fails before writing the offset. The event is processed again.
• Event format inconsistency: some producers use snake_case, others camelCase. The schema should enforce a canonical format.
• Event size limits: the broker may limit message size (e.g., 1MB). Large events must be split or compressed.
• Schema registry latency: if the schema registry is slow, event validation can delay processing.

Best Practices

• Use UUIDs for event_id to ensure global uniqueness.
• Include timestamp in ISO 8601 format for consistent time semantics.
• Use schema_id to enable versioned event contracts.
• Design event handlers to be idempotent and stateless.
• Monitor offset lag and event processing latency.
• Use batch_size and start_offset to tune consumer performance.
• Validate incoming events against their schema before processing.
• Log every event processed, including event_id, event_type, timestamp, and processing_duration.

Cloud Event Driven Architecture is not a technology stack—it is a discipline of designing systems around events. When done well, it delivers resilience, scalability, and responsiveness.

This page was rewritten on 16 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.