Saga Orchestration Patterns

Sagas orchestrate distributed transactions across microservices, enabling consistency in systems where individual services manage their own data. They are essential when a business operation spans multiple services that cannot coordinate via a single database transaction. Use sagas when you need eventual consistency, especially in systems with high latency, asynchronous workflows, or services that commit data independently. This pattern becomes critical when a single business action—like placing an order, processing a payment, or onboarding a user—must succeed across services, even if some steps fail and must be retried or compensated.

Event-Driven Saga Orchestration

A saga is a sequence of local transactions, each managed by a service, coordinated by a central orchestrator or via message events. The orchestrator publishes events, listens for state changes, and triggers the next step. This approach decouples services and allows for independent scaling.

The orchestrator typically maintains a saga state, recorded in a database or event store. Each step in the saga corresponds to a state transition. When an event is received, the orchestrator checks the current state, decides the next action, and emits new events.

State Management and Event Handling

The orchestrator maintains a saga table with columns: id, version, state, data, created_at, updated_at.

CREATE TABLE saga (
    id UUID PRIMARY KEY,
    version INT NOT NULL DEFAULT 1,
    state VARCHAR(50) NOT NULL,
    data JSONB NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

When a new saga begins—such as OrderPlaced—the orchestrator inserts a new saga record.

{
  "id": "a3d1e9b2-4c3f-1a2b-8d7c-6e5f4a3b2c1d",
  "state": "OrderReceived",
  "data": {
    "orderId": "ORD-1001",
    "customerId": "CUST-2003",
    "items": [
      { "productId": "PROD-101", "quantity": 2 }
    ]
  }
}

The orchestrator listens for events via a message broker like Apache Kafka, RabbitMQ, or AWS SQS. It uses a consumer group to ensure each event is processed exactly once.

A typical event payload follows a canonical format:

{
  "eventId": "e8b3f4a5-6c7d-8e9f-a1b2-c3d4e5f6a7b8",
  "eventType": "OrderPlaced",
  "aggregateId": "ORD-1001",
  "timestamp": "2024-04-05T12:34:56.789Z",
  "payload": {
    "orderId": "ORD-1001",
    "customerId": "CUST-2003",
    "items": [
      { "productId": "PROD-101", "quantity": 2 }
    ],
    "totalAmount": 199.98
  }
}

The orchestrator subscribes to OrderPlaced events and triggers the next step: CreateInventoryReservation.

Handling Step Failures and Retries

When a step in the saga fails—say, CreateInventoryReservation fails due to a network timeout—the orchestrator must handle the failure and decide on the next action.

The orchestrator logs the error and updates the saga state:

{
  "id": "a3d1e9b2-4c3f-1a2b-8d7c-6e5f4a3b2c1d",
  "state": "InventoryReservationFailed",
  "data": {
    "orderId": "ORD-1001",
    "inventoryReservationId": null,
    "error": "Failed to reserve inventory for PROD-101: Connection timed out",
    "retryCount": 3,
    "lastAttempt": "2024-04-05T12:35:10.123Z"
  }
}

The orchestrator then emits a InventoryReservationFailed event:

{
  "eventId": "f5a4b3c2-d1e2-f3a4-b5c6-d7e8f9a1b2c3",
  "eventType": "InventoryReservationFailed",
  "aggregateId": "ORD-1001",
  "timestamp": "2024-04-05T12:35:10.123Z",
  "payload": {
    "orderId": "ORD-1001",
    "error": "Failed to reserve inventory for PROD-101: Connection timed out",
    "retryCount": 3
  }
}

The orchestrator configures retry policies using exponential backoff and circuit breakers. Retries are tracked in the data field, and each attempt is recorded as a state change.

The orchestrator may also trigger a compensation action: if inventory reservation fails, it cancels the order.

{
  "id": "a3d1e9b2-4c3f-1a2b-8d7c-6e5f4a3b2c1d",
  "state": "Compensating",
  "data": {
    "compensationSteps": [
      {
        "step": "CancelOrder",
        "status": "Pending",
        "attempts": 1,
        "error": null
      }
    ],
    "originalState": "InventoryReservationFailed"
  }
}

Compensation and Undoing Steps

Compensation is critical in sagas. When a step fails and cannot be retried indefinitely, the orchestrator initiates compensation actions to undo earlier steps.

For example, if ProcessPayment fails after CreateInventoryReservation, the orchestrator sends a CompensatePaymentFailed event and starts the compensation process.

The compensation step logs the outcome:

{
  "id": "a3d1e9b2-4c3f-1a2b-8d7c-6e5f4a3b2c1d",
  "state": "Compensating",
  "data": {
    "compensationSteps": [
      {
        "step": "CancelOrder",
        "status": "Completed",
        "attempts": 1,
        "error": null
      },
      {
        "step": "ReleaseInventoryReservation",
        "status": "Pending",
        "attempts": 2,
        "error": "Failed to release reservation: Timeout during HTTP call to inventory service"
      }
    ]
  }
}

Compensation actions are idempotent and can be retried. The orchestrator ensures that each compensation step is executed exactly once, using message deduplication and idempotency keys.

State-Driven Orchestration with Decision Logic

The orchestrator evaluates the current state and makes decisions based on conditions. The decision logic is embedded in the orchestrator’s code or defined via configuration.

A simple decision function in a language like Go or Python:

func handleEvent(event Event) error {
    saga, err := getSaga(event.AggregateId)
    if err != nil {
        return err
    }

    switch saga.State {
    case "OrderReceived":
        if event.EventType == "OrderPlaced" {
            err = publishEvent("CreateInventoryReservation", event.Payload)
            if err != nil {
                return err
            }
            saga.State = "InventoryReservationRequested"
            saga.Data["reservationRequestedAt"] = time.Now().Format(time.RFC3339)
        }
    case "InventoryReservationRequested":
        if event.EventType == "InventoryReservationCreated" {
            saga.State = "InventoryReservationSucceeded"
            saga.Data["reservationId"] = event.Payload["reservationId"]
            err = publishEvent("ProcessPayment", event.Payload)
            if err != nil {
                return err
            }
        }
    case "InventoryReservationSucceeded":
        if event.EventType == "PaymentProcessed" {
            saga.State = "OrderConfirmed"
            saga.Data["paymentId"] = event.Payload["paymentId"]
            err = publishEvent("SendOrderConfirmationEmail", event.Payload)
            if err != nil {
                return err
            }
        }
    default:
        log.Printf("Unhandled state: %s, event: %s", saga.State, event.EventType)
        return errors.New("unhandled state transition")
    }

    return updateSaga(saga)
}

The orchestrator may also handle multiple concurrent sagas. Each saga runs independently, but the orchestrator shares a common event bus and database.

Eventual Consistency and Idempotency

Sagas rely on eventual consistency. The orchestrator assumes that events may arrive out of order, be duplicated, or be processed multiple times.

Idempotency is enforced through event deduplication. The orchestrator maintains a processed_events table:

CREATE TABLE processed_events (
    eventId UUID PRIMARY KEY,
    sagaId UUID,
    processedAt TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Before processing an event, the orchestrator checks if the event ID has already been processed:

SELECT 1 FROM processed_events WHERE eventId = 'e8b3f4a5-6c7d-8e9f-a1b2-c3d4e5f6a7b8';

If the event is already processed, the orchestrator skips the processing, avoiding side effects.

Common Failures and Their Symptoms

Event Lost or Duplicated: The orchestrator processes an event twice, leading to duplicate steps. The processed_events table helps detect and prevent this.
Saga State Divergence: A step updates the saga state, but the next step reads an outdated state due to delay in event propagation. This causes inconsistent state transitions.
Compensation Not Triggered: The orchestrator fails to emit a compensation event after a step failure, leading to unhandled errors.
Deadlock in Saga Execution: Two sagas depend on each other, and both wait for an event from the other, but neither triggers the next step.
Version Mismatch in Event Schema: An older service emits an event with a new field, but the orchestrator expects the old schema. The orchestrator fails to parse the event or misinterprets the state.

Configuration and Monitoring

The orchestrator is configured via environment variables and a single configuration file:


orchestrator:
  eventBus:
    type: "kafka"
    bootstrapServers: ["kafka1:9092", "kafka2:9092"]
    topic: "saga-events"
    groupId: "saga-orchestrator"
  database:
    url: "postgresql://db:5432/saga"
    maxConnections: 100
    idleTimeout: "10s"
  retry:
    maxAttempts: 5
    backoff:
      base: 1000
      factor: 2
      max: 30000
    circuitBreaker:
      threshold: 3
      timeout: "30s"
      reset: "1m"
  compensation:
    maxRetries: 3
    timeout: "5m"
    enableAutoCompensation: true

The orchestrator exposes a health endpoint:

GET /health
HTTP/1.1
Host: orchestrator.example.com

Response:
{
  "status": "healthy",
  "uptime": "123456s",
  "activeSagas": 247,
  "failedSagas": 8,
  "lastEventProcessed": "2024-04-05T12:35:10.123Z",
  "eventBus": {
    "connected": true,
    "lag": 234,
    "partition": 0
  }
}

Monitoring is done via structured logs. Each event processing step logs:

{
  "timestamp": "2024-04-05T12:34:56.789Z",
  "level": "info",
  "component": "saga-orchestrator",
  "action": "handle_event",
  "sagaId": "a3d1e9b2-4c3f-1a2b-8d7c-6e5f4a3b2c1d",
  "state": "InventoryReservationRequested",
  "event": {
    "eventId": "e8b3f4a5-6c7d-8e9f-a1b2-c3d4e5f6a7b8",
    "eventType": "InventoryReservationCreated",
    "aggregateId": "ORD-1001"
  },
  "durationMs": 142,
  "message": "Successfully processed inventory reservation created event"
}

Key Design Principles

Saga state must be immutable after creation: updates are versioned, and the orchestrator uses optimistic concurrency control.
Event payloads must be self-contained: no reliance on external schemas or shared metadata.
All actions must be idempotent: services can reprocess the same event without side effects.
Compensation must be reversible: each step must define how to undo it.
Orchestrator must handle duplicate events, out-of-order events, and partial failures.

This pattern is not tied to any specific platform. It works with any message broker, database, and programming language. It is the default approach for managing distributed transactions in microservices.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.