Outbox Pattern | The Garnet Wiki

The outbox pattern ensures that domain events are reliably published to an external system—such as a message broker, another service, or a data warehouse—across service boundaries, even when the primary transaction fails. It is essential when a service must both update its own data and emit a side-effect event, and where losing that event would break downstream integrations.

Core Mechanics

The outbox pattern decouples the *transaction* of updating a service’s data from the *publication* of its domain events. Instead of writing to the database and immediately sending a message, the service writes the event to a dedicated outbox table in the same transaction. This table acts as a durable, local queue that is committed atomically with the service’s business logic.

The pattern relies on a background process—often a worker service or a scheduled job—that periodically checks the outbox table for new entries and publishes them to a message broker like Kafka, RabbitMQ, or AWS SQS.

Event Persistence

Every domain event is stored in the outbox table as a row, with at least the following fields:

CREATE TABLE outbox (
    id UUID PRIMARY KEY,
    aggregate_id UUID NOT NULL,
    event_type VARCHAR(255) NOT NULL,
    payload JSON NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL,
    processed_at TIMESTAMP WITH TIME ZONE,
    version INTEGER NOT NULL DEFAULT 1
);

When a service handles a command, it performs a single database transaction that includes:

Updating the domain entity (e.g., UPDATE orders SET status = 'shipped' WHERE id = $1;)
Inserting a new outbox entry for the resulting domain event:

INSERT INTO outbox (id, aggregate_id, event_type, payload, created_at, version)
VALUES (
    'a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8',
    '12345678-90ab-cdef-1234-567890abcdef',
    'OrderShipped',
    '{"orderId": "12345678-90ab-cdef-1234-567890abcdef", "trackingNumber": "TRK123456789"}',
    '2025-04-05T12:00:00Z',
    1
);

This ensures that event persistence and domain state updates are atomic. If the database crashes after the UPDATE but before the INSERT, the event is still written and will be processed later.

Event Publication Loop

A worker process polls the outbox table at regular intervals. It selects all unprocessed events and publishes them to a message broker.

The worker uses a simple, idempotent loop:

while true; do
    psql -c "
        SELECT id, event_type, payload, created_at
        FROM outbox
        WHERE processed_at IS NULL
        ORDER BY created_at
        LIMIT 100
        FOR UPDATE SKIP LOCKED
    " \
    | while read id event_type payload created_at; do
        # Publish to message broker
        curl -X POST \
             -H "Content-Type: application/json" \
             -d "$payload" \
             http://message-broker:8080/events/$event_type

        # Mark as processed in the outbox
        psql -c "
            UPDATE outbox
            SET processed_at = NOW()
            WHERE id = '$id'
        "
    done

    sleep 1
done

The FOR UPDATE SKIP LOCKED clause prevents multiple workers from processing the same event. It also allows exactly-once delivery by ensuring that only one worker picks up a given event.

Processing Idempotency

Events are published to the message broker in a format that includes the outbox row id as the message’s messageId field. Consumers of the events use this messageId to deduplicate incoming messages.

The message payload includes:

{
  "messageId": "a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8",
  "eventType": "OrderShipped",
  "payload": {
    "orderId": "12345678-90ab-cdef-1234-567890abcdef",
    "trackingNumber": "TRK123456789"
  },
  "timestamp": "2025-04-05T12:00:00Z"
}

Consumers can use the messageId to implement idempotent processing: if they have already processed a message with the same messageId, they skip the work.

Common Failure Modes

Event Lost During Transaction

If the database crashes after writing the domain state but before committing the outbox insert, the event is lost. The transaction is rolled back, and the event is not in the outbox. This is silent: the service reports success, but the event never appears.

The fix is to ensure the outbox table is in the same database as the domain data, and to use a transaction that spans both the domain and outbox updates. The worker must run in a separate process, so it can recover from such a failure.

Duplicate Event Publication

When the worker processes an event and publishes it, but the service fails before marking the event as processed, the same event may be published multiple times.

This happens if the worker fails after the curl call but before the UPDATE outbox SET processed_at = NOW(). The messageId in the message broker is unique, but the worker process may restart and pick up the same event again.

To prevent this, use a two-phase commit or deduplication logic in the message broker. Alternatively, ensure that the UPDATE statement is atomic and that the worker process is idempotent.

Delayed Processing Due to Polling Interval

Polling every 1 second is common, but if the worker starts and the outbox table has 1000 unprocessed events, the first batch may take up to 1 second to process, during which new events arrive. This leads to event staleness and increased latency.

The solution is to increase the polling batch size (e.g., LIMIT 1000) and to process events in parallel using multiple worker threads.

Event Not Published Despite Being Processed

The processed_at field is set, but the message is never published due to a network error, broker outage, or a misconfigured endpoint.

The worker should log the messageId and event_type for every event it attempts to publish. When a message fails to send, it should log the HTTP response body and status code:

curl -X POST \
     -H "Content-Type: application/json" \
     -d "$payload" \
     http://message-broker:8080/events/$event_type \
     -w "\nStatus: %{http_code}\nBody: %{body}\n"

On error, the worker can requeue the event by setting a retries column in the outbox table.

Reentrant and Retried Processing

When an event fails during processing, the consumer can update the outbox table to record the failure:

ALTER TABLE outbox ADD COLUMN last_error TEXT;
ALTER TABLE outbox ADD COLUMN retries INTEGER DEFAULT 0;

The consumer updates the event after a failure:

UPDATE outbox
SET last_error = $1,
    retries = retries + 1
WHERE id = $2;

The worker can then reprocess events that have retries left, or move them to a dead-letter queue.

Outbox Table Growth

Over time, the outbox table grows large. If processed_at is never cleaned up, it can slow down queries, increase backup sizes, and consume disk space.

The worker should periodically archive or delete old processed events:

DELETE FROM outbox
WHERE processed_at < NOW() - INTERVAL '30 days'
  AND retries = 0;

This keeps the table lean and ensures that the SELECT ... FOR UPDATE query performs efficiently.

Advanced Usage

Multiple Event Types, Single Outbox

The outbox table supports multiple event types across services. A single worker can publish events from all services, or each service can have its own worker with filtering logic.

The worker can filter events by event_type:

SELECT id, payload, created_at
FROM outbox
WHERE processed_at IS NULL
  AND event_type = 'OrderShipped'
ORDER BY created_at
LIMIT 100
FOR UPDATE SKIP LOCKED;

This allows event specialization: some services publish events to a specific topic in the message broker, while others use a generic path.

Event Versioning and Schema Evolution

The version column allows for schema evolution. When an event is published, the worker can include the version in the message header:

{
  "messageId": "a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8",
  "eventType": "OrderShipped",
  "version": 2,
  "payload": {
    "orderId": "12345678-90ab-cdef-1234-567890abcdef",
    "trackingNumber": "TRK123456789",
    "shippedAt": "2025-04-05T12:00:00Z"
  }
}

Consumers can use version to adapt their logic, and the outbox table becomes a source of truth for event schema history.

Synchronous Event Publication

For services that require immediate delivery, a synchronous publish step can be added. After writing to the outbox, the service calls a dedicated endpoint that:

Inserts the event.
Immediately publishes it.
Returns the messageId.

This reduces latency and ensures that the event is available in the message broker within milliseconds.

POST /outbox/events
Content-Type: application/json

{
  "aggregateId": "12345678-90ab-cdef-1234-567890abcdef",
  "eventType": "OrderShipped",
  "payload": {
    "orderId": "12345678-90ab-cdef-1234-567890abcdef",
    "trackingNumber": "TRK123456789"
  }
}

The server returns:

{
  "messageId": "a1b2c3d4-e5f6-7890-g1h2-i3j4k5l6m7n8",
  "publishedAt": "2025-04-05T12:00:00Z"
}

This is useful for real-time dashboards, synchronous APIs, and event-driven workflows.

Configuration and Operational Concerns

Worker Configuration

The worker process is typically configured with:

POLL_INTERVAL: how often to poll the outbox (e.g., 1s, 5s)
BATCH_SIZE: number of events to fetch per poll (e.g., 100, 1000)
MAX_RETRIES: number of times to retry a failed event publish
ARCHIVE_AFTER_DAYS: days after which processed events are archived
MESSAGE_BROKER_URL: base URL of the message broker

These are set via environment variables or a simple configuration file:

outbox:
  poll_interval: 2s
  batch_size: 500
  max_retries: 3
  archive_after_days: 30
  message_broker:
    url: "http://message-broker:8080"
    timeout: 10s

Monitoring and Alerting

Key metrics to monitor:

outbox_unprocessed_count: number of events with processed_at IS NULL
outbox_processed_count: number of events processed in the last hour
outbox_failure_rate: percentage of publish attempts that failed
outbox_latency: average time from created_at to processed_at

Alerts should trigger when:

outbox_unprocessed_count > 1000 for 5 minutes
outbox_failure_rate > 5% in the last 15 minutes
outbox_latency > 10s for 10 consecutive polls

Schema Migrations

When the outbox table changes, migrations must be idempotent. For example:

-- Add retry column
ALTER TABLE outbox ADD COLUMN IF NOT EXISTS retries INTEGER DEFAULT 0;
ALTER TABLE outbox ADD COLUMN IF NOT EXISTS last_error TEXT;

-- Add version column if missing
ALTER TABLE outbox ADD COLUMN IF NOT EXISTS version INTEGER DEFAULT 1;

These can be run during deployments, and the worker continues to process events without interruption.

The outbox pattern is not a single tool, but a robust, general-purpose mechanism for reliable event publication. It is not tied to any specific platform, but can be implemented with standard SQL, HTTP, and process management. It handles transactional consistency, idempotency, failure recovery, and schema evolution, all with minimal infrastructure.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.