Webhook Delivery Patterns | The Garnet Wiki

Webhooks enable event-driven systems where one service notifies another of changes, typically via HTTP POST. They matter when components must react to real-time state changes—such as a payment processor informing an order service after a transaction completes, or a CI pipeline triggering a deployment after a code push. Delivery patterns define how these notifications are sent, retried, and acknowledged, ensuring reliability across network failures, service downtime, and high load.

Ensuring Reliable Delivery with Retry Strategies

When a webhook receiver is unreachable, the sender must reattempt delivery. The most common pattern is exponential backoff with jitter. The sender records a delivery attempt with a timestamp and status. If the response code is not 2xx, the delivery is considered failed.

A typical delivery attempt includes:

X-Webhook-ID: 12345
X-Webhook-Timestamp: 2024-03-15T12:34:56.789Z
X-Webhook-Attempt: 1
X-Webhook-Event-Type: payment.completed

On failure, the sender schedules a retry with delay calculated as base_delay × 2^attempt_number + jitter, where jitter is a random value between 0 and base_delay. For example:

Attempt 1: delay = 10s × 2^0 + random(0–10s) → 10–20s
Attempt 2: delay = 10s × 2^1 + random(0–10s) → 20–30s
Attempt 3: delay = 10s × 2^2 + random(0–10s) → 40–50s

The sender must persist retry state. A common schema in PostgreSQL:

CREATE TABLE webhook_deliveries (
    id UUID PRIMARY KEY,
    event_id UUID NOT NULL,
    target_url TEXT NOT NULL,
    payload JSONB NOT NULL,
    status TEXT NOT NULL CHECK (status IN ('pending', 'in-progress', 'failed', 'succeeded')),
    attempt_count INT NOT NULL DEFAULT 1,
    last_attempt_at TIMESTAMPTZ,
    next_attempt_at TIMESTAMPTZ,
    error TEXT,
    metadata JSONB
);

Delivery fails if the HTTP request times out (e.g., 10s), or the server returns a 5xx status code. A 4xx response is typically treated as transient failure—the receiver is available but cannot process the payload, so the sender should retry. A 400 or 404 response may indicate a malformed payload or invalid endpoint, which may be handled differently.

Acknowledging Receipt: The Role of HTTP Status and Headers

The receiver of a webhook must explicitly acknowledge receipt. A 200 OK status is sufficient, but for idempotency and clarity, the sender expects a structured acknowledgment.

The receiver should respond with:

Content-Type: application/json
X-Webhook-Receipt-ID: 67890
X-Webhook-Processing-Time: 125ms

A minimal successful response:

{
  "status": "received",
  "message": "Payload processed successfully",
  "receipt_id": "67890",
  "processed_at": "2024-03-15T12:34:57.123Z"
}

The sender uses X-Webhook-Receipt-ID to track delivery progress. If the receiver returns a 202 Accepted, the sender treats it as asynchronous acknowledgment: the payload is queued but not yet processed. The sender may consider the delivery complete after a fixed timeout or upon receiving a second webhook from the receiver.

A critical edge case: the receiver returns 200 but the connection drops before sending the response body. The sender assumes success, but the receiver never received the data. To prevent this, the sender should use idempotent delivery—each delivery includes a unique X-Webhook-ID and the receiver checks for duplicate deliveries using this ID.

Handling Idempotency with Deduplication

Idempotency ensures that multiple deliveries of the same event result in the same state. The sender generates a unique X-Webhook-ID per event. The receiver stores this ID and checks for duplicates before processing.

The receiver must:

Look up webhook_deliveries table by X-Webhook-ID in the request header.
If a delivery with that ID exists, return HTTP 200 with X-Webhook-Status: duplicate and the existing receipt_id.
Otherwise, process the event and insert a new record.

Example:

POST /webhooks/payment-completed HTTP/1.1
Host: api.example.com
Content-Type: application/json
X-Webhook-ID: 12345
X-Webhook-Timestamp: 2024-03-15T12:34:56.789Z
X-Webhook-Event-Type: payment.completed

{
  "payment_id": "pay_abc123",
  "status": "completed",
  "amount": 99.99
}

Receiver logic:

SELECT id, status, processed_at FROM webhook_deliveries
WHERE "webhook_id" = '12345';

If found, return:

HTTP/1.1 200 OK
Content-Type: application/json
X-Webhook-Status: duplicate
X-Webhook-Receipt-ID: 67890

{
  "status": "duplicate",
  "message": "Event already processed",
  "receipt_id": "67890",
  "processed_at": "2024-03-15T12:34:57.123Z"
}

If not found, process and insert:

INSERT INTO webhook_deliveries (id, event_id, target_url, payload, status, attempt_count, last_attempt_at, next_attempt_at, error, metadata)
VALUES (
  '12345',
  'event_abc123',
  'https://api.example.com/webhooks/payment-completed',
  '{"payment_id":"pay_abc123", "status":"completed", "amount":99.99}',
  'succeeded',
  1,
  '2024-03-15T12:34:56.789Z',
  '2024-03-15T12:35:06.789Z',
  NULL,
  '{"source": "stripe", "version": "v1"}'
);

A subtle failure: the sender assumes X-Webhook-ID is globally unique, but the receiver’s webhook_id column is TEXT and not indexed. As load grows, duplicate checks become slow and the system degrades. The solution is to create a unique index:

CREATE UNIQUE INDEX idx_webhook_deliveries_webhook_id ON webhook_deliveries (webhook_id);

Managing Delivery Order and Event Sequencing

Webhooks may arrive out of order, especially when multiple senders or delayed retries. The sender includes X-Webhook-Timestamp to help the receiver order events.

The receiver can:

Use X-Webhook-Timestamp to sort incoming events.
Maintain a last_processed_timestamp per event type in a webhook_sequence table:

CREATE TABLE webhook_sequence (
    event_type TEXT PRIMARY KEY,
    last_received_at TIMESTAMPTZ NOT NULL,
    last_processed_at TIMESTAMPTZ,
    current_offset BIGINT
);

On each delivery, the receiver:

Updates last_received_at to the X-Webhook-Timestamp.
Processes the event.
Updates last_processed_at and current_offset.

A failure occurs when the receiver processes an event before an earlier one arrives. To handle this, the receiver can track event offsets and apply events in order.

Example:

-- Receive event with timestamp 2024-03-15T12:34:55.000Z
INSERT INTO webhook_events (id, event_type, payload, received_at, processed_at)
VALUES ('event_abc123', 'payment.completed', '{}', '2024-03-15T12:34:55.000Z', NULL);

-- Update sequence
UPDATE webhook_sequence
SET last_received_at = '2024-03-15T12:34:55.000Z',
    last_processed_at = '2024-03-15T12:34:55.000Z',
    current_offset = 100
WHERE event_type = 'payment.completed';

If a new event arrives with timestamp 2024-03-15T12:34:53.000Z (earlier than the current one), the receiver must:

Enqueue it in a queue (e.g., RabbitMQ, Kafka).
Reorder events by timestamp before processing.

A common mistake: the receiver uses X-Webhook-Timestamp only for logging, but does not use it to reorder or deduplicate. The sender may assume the timestamp is authoritative, but the receiver’s received_at column is set at HTTP arrival time, which can be later than the event’s actual creation time.

Handling Concurrent and Overlapping Deliveries

When multiple events are delivered simultaneously, the sender may batch multiple events into a single HTTP request.

The receiver must handle batch delivery. The request body is a JSON array:

[
  {
    "id": "event_abc123",
    "type": "payment.completed",
    "timestamp": "2024-03-15T12:34:56.789Z",
    "payload": {
      "payment_id": "pay_abc123",
      "status": "completed",
      "amount": 99.99
    }
  },
  {
    "id": "event_def456",
    "type": "order.created",
    "timestamp": "2024-03-15T12:34:57.123Z",
    "payload": {
      "order_id": "ord_xyz789",
      "items": 3
    }
  }
]

The sender includes X-Webhook-Batch-ID and X-Webhook-Event-Count:

POST /webhooks/events HTTP/1.1
Host: api.example.com
Content-Type: application/json
X-Webhook-Batch-ID: batch_78901
X-Webhook-Event-Count: 2
X-Webhook-Source: stripe

[ ... ]

The receiver processes each event in the batch, and each event may be acknowledged individually.

A failure pattern: the receiver processes the first event successfully but crashes before processing the second. The sender sees the batch as partially delivered. To fix this, the receiver should:

Mark the entire batch as processed only after all events succeed.
Return a response that reflects batch-level status:

{
  "batch_id": "batch_78901",
  "total_events": 2,
  "successful_events": 1,
  "failed_events": 1,
  "errors": [
    {
      "event_id": "event_def456",
      "error": "Database connection lost",
      "timestamp": "2024-03-15T12:34:57.123Z"
    }
  ]
}

The sender can now retry the entire batch or individual failed events.

Detecting and Handling Delivery Timeouts

The sender must detect when a delivery has not been acknowledged within a defined window. A common pattern is timeout after delivery.

The sender sets X-Webhook-Timeout: 30s in the initial request. The receiver must respond within 30 seconds. If not, the sender marks the delivery as timed out and schedules a retry.

The sender tracks timeouts via a background job (e.g., using systemctl or cron):


0,5,10,15,20,25,30,35,40,45,50,55 * * * * /opt/webhook/bin/check-timeouts.sh

check-timeouts.sh:

#!/bin/bash
set -euo pipefail

DB_HOST=postgres.example.com
DB_NAME=webhooks
TIMEOUT_THRESHOLD=30  # seconds

# Find deliveries that have not been acknowledged within timeout window
psql -h "$DB_HOST" -d "$DB_NAME" -c "
SELECT id, target_url, payload, last_attempt_at
FROM webhook_deliveries
WHERE status = 'in-progress'
  AND next_attempt_at < NOW() + INTERVAL '$TIMEOUT_THRESHOLD seconds'
  AND last_attempt_at > NOW() - INTERVAL '1 hour'
" | while IFS= read -r line; do
  # Parse output: id, target_url, payload, last_attempt_at
  id=$(echo "$line" | cut -d'|' -f1 | xargs)
  url=$(echo "$line" | cut -d'|' -f2 | xargs)
  payload=$(echo "$line" | cut -d'|' -f3)
  last_attempt=$(echo "$line" | cut -d'|' -f4 | xargs)

  # Mark as timed out
  psql -h "$DB_HOST" -d "$DB_NAME" -c "
    UPDATE webhook_deliveries
    SET status = 'timed-out',
        error = 'Delivery did not receive acknowledgment within $TIMEOUT_THRESHOLD seconds'
    WHERE id = '$id'
  "

  # Trigger retry
  curl -X POST "$url" \
    -H "Content-Type: application/json" \
    -H "X-Webhook-ID: $id" \
    -H "X-Webhook-Attempt: $(psql -h "$DB_HOST" -d "$DB_NAME" -c \"SELECT attempt_count FROM webhook_deliveries WHERE id = '$id'\" -t -A -F '|' -c \"SELECT attempt_count FROM webhook_deliveries WHERE id = '$id'\" | head -1 | xargs)" \
    -H "X-Webhook-Timestamp: $(date -Iseconds)" \
    -d "$payload"
done

A subtle failure: the sender sets X-Webhook-Timeout but the receiver does not honor it. The receiver processes the event, updates the database, but does not send the response in time. The sender treats it as a timeout, but the event was actually processed. The solution is to use X-Webhook-Timeout as a deadline for response, and the receiver must send the response within that window, possibly using async processing.

Summary of Failure Modes and Common Pitfalls

Undelivered acknowledgments: sender assumes success, but receiver’s response is lost. Use idempotency with X-Webhook-ID.
Out-of-order events: sender assumes event order, but receiver processes events in different threads. Use X-Webhook-Timestamp and webhook_sequence.
Batch delivery without batch acknowledgment: sender receives a batch but does not know which events succeeded. Return structured batch responses.
Missing or incorrect retry logic: retries are too frequent or too infrequent. Use exponential backoff with jitter and persist retry state.
Idempotency not enforced: same event processed multiple times. Use unique webhook_id index and deduplication logic.
Timeouts not detected: deliveries are slow but not retried. Use a periodic timeout checker job.

The pattern is complete when each delivery is idempotent, ordered, acknowledged, and retried with exponential backoff—ensuring that events are delivered exactly once, even across failures.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.