Backend Idempotency Patterns

Design APIs that can be safely retried without side effects. Covers idempotency keys, at-most-once delivery, deduplication strategies, and the patterns that prevent duplicate charges, duplicate emails, and double-counted metrics in distributed systems.

Backend idempotency ensures that repeated application of the same operation produces the same result, regardless of how many times it is executed. This is essential in distributed systems where calls may be retried due to network issues, timeouts, or service failures—common in microservices, message queues, and API clients. Idempotency is not optional in systems where side effects must be predictable and consistent across retries, particularly in payment processing, order creation, user onboarding, and data synchronization.

Designing Idempotent Operations

An idempotent operation produces the same outcome when applied multiple times with identical inputs. The key insight is that the client or system must provide a unique identifier for each request, allowing the server to detect and avoid duplicate work. The most common approach is to use a client-provided idempotency_key in the request.

Using a Client-Supplied Idempotency Key

The client includes a unique identifier in each request, typically in the request headers or body, and the server uses this key to deduplicate incoming requests.

POST /api/v1/orders HTTP/1.1
Host: api.example.com
Content-Type: application/json
Idempotency-Key: 7a2e9c1d-8f5b-4a3c-9d1e-2b4c6d8e0f1a

{
  "customer_id": 123,
  "items": [
    { "product_id": 45, "quantity": 2 }
  ],
  "total": 99.99
}

The server must store the idempotency_key and its associated result (e.g., order ID, status, created_at) in a durable store. On receipt of a request with a key, the server checks if it has already processed a request with that key. If so, it returns the previously computed result without re-executing the operation.

Idempotency Key Storage and Lookup

The storage must support atomic reads and writes. A typical implementation uses a database table:

CREATE TABLE idempotency_keys (
  idempotency_key VARCHAR(255) PRIMARY KEY,
  request_payload JSONB NOT NULL,
  response_status INT NOT NULL,
  response_body JSONB,
  created_at TIMESTAMPTZ NOT NULL,
  updated_at TIMESTAMPTZ NOT NULL
);

When processing a request, the server performs a two-step operation:

  1. Check if key exists: 
   SELECT idempotency_key, response_body, response_status
   FROM idempotency_keys
   WHERE idempotency_key = $1;
  1. Insert new entry if missing (atomic): 
   INSERT INTO idempotency_keys (idempotency_key, request_payload, response_status, response_body, created_at, updated_at)
   VALUES ($1, $2, $3, $4, NOW(), NOW())
   ON CONFLICT (idempotency_key)
   DO NOTHING;

The ON CONFLICT (idempotency_key) DO NOTHING ensures that only the first request with a given key is processed, while subsequent ones are served from the cache.

Handling Partial Success and Retry Logic

When processing a request, the server may fail after updating some data but before saving the idempotency record. To avoid lost updates, the idempotency key must be recorded before the operation begins.

A robust pattern is to use a two-phase write:

  1. Prepare phase:
  1. Execute phase:

This allows clients to poll for progress and ensures that a request is idempotent even if the server crashes mid-process.

Idempotency with Message Queues

In event-driven systems, message queues like Apache Kafka, RabbitMQ, or AWS SQS are used to decouple producers and consumers. Idempotency is critical when messages are retried or delivered multiple times.

Each message contains a idempotency_key in its headers or body. The consumer checks if it has already processed the key before applying the message’s effect.

For example, in a Kafka consumer using kafka-python:

from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    'orders',
    bootstrap_servers=['kafka1.example.com', 'kafka2.example.com'],
    value_deserializer=lambda m: json.loads(m.decode('utf-8')),
    group_id='order-processor'
)


def process_message(message):
    key = message.headers.get('Idempotency-Key')
    if not key:
        raise ValueError("Missing Idempotency-Key header")

    # Check if already processed
    result = db.execute("""
        SELECT response_body, response_status
        FROM idempotency_keys
        WHERE idempotency_key = %s
    """, [key])

    if result:
        # Return cached response
        body, status = result[0]
        return {
            "statusCode": status,
            "body": body
        }

    # Process the message
    order_data = message.value
    order_id = create_order(order_data)
    db.execute("""
        INSERT INTO idempotency_keys (idempotency_key, request_payload, response_status, response_body, created_at, updated_at)
        VALUES (%s, %s, 201, %s, NOW(), NOW())
        ON CONFLICT (idempotency_key) DO NOTHING
    """, [key, json.dumps(order_data), json.dumps({"order_id": order_id, "status": "created"})])

    return {
        "statusCode": 201,
        "body": json.dumps({"order_id": order_id, "status": "created"})
    }

When a message is retried (e.g., due to a consumer crash), the consumer checks the idempotency_keys table and skips processing if the key is already present.

Idempotency in HTTP APIs

REST APIs commonly use idempotency keys in headers. Clients must generate consistent keys—typically using a UUID, a hash of the request body, or a combination of client ID and timestamp.

A standard practice is to return Idempotency-Key in the response header, allowing clients to re-try requests with confidence.

HTTP/1.1 201 Created
Location: /api/v1/orders/9876
Idempotency-Key: 7a2e9c1d-8f5b-4a3c-9d1e-2b4c6d8e0f1a
Content-Type: application/json

{
  "order_id": 9876,
  "status": "created",
  "total": 99.99
}

Clients can use this Idempotency-Key to re-send the same request on failure, knowing that the server will return the same result.

Idempotency and Transactional Consistency

In systems with multiple services, idempotency ensures that distributed transactions are resilient to partial failures. For example, a payment service may need to:

Each step must be idempotent, and the entire workflow should be orchestrated using a idempotency_key.

A key failure mode is duplicate processing due to delayed idempotency key persistence. If the idempotency record is written after the operation, a crash before the write may cause the same request to be processed twice, even though the key was generated.

To prevent this, the idempotency key should be written before the main operation begins:

def create_order_with_idempotency(request):
    # Step 1: Write idempotency key
    db.execute("""
        INSERT INTO idempotency_keys (idempotency_key, request_payload, status)
        VALUES (%s, %s, 'processing')
        ON CONFLICT (idempotency_key) DO NOTHING
    """, [request['idempotency_key'], json.dumps(request)])

    # Step 2: Process order
    order_id = db.execute("""
        INSERT INTO orders (customer_id, total, status)
        VALUES (%s, %s, 'pending')
        RETURNING id
    """, [request['customer_id'], request['total']])

    # Step 3: Update idempotency key with success
    db.execute("""
        UPDATE idempotency_keys
        SET response_status = 201,
            response_body = %s,
            updated_at = NOW()
        WHERE idempotency_key = %s
    """, [json.dumps({"order_id": order_id}), request['idempotency_key']])

    return {
        "order_id": order_id,
        "status": "created"
    }

This ensures that even if the server crashes after step 1 but before step 3, a retry will not re-execute the order.

Common Pitfalls and Silent Failures

Clients may send idempotency_key in the body but forget to set it in the header. Servers that only read the header miss keys from clients that use only the body.

Long-lived idempotency records consume storage. A common failure is stale entries from failed or long-running operations. Servers must periodically clean up idempotency_keys with status = 'pending' or updated_at < NOW() - INTERVAL '7 days'.

Clients often retry on 5xx or 408 (Request Timeout) but miss 429 (Too Many Requests) or 409 (Conflict) as retry triggers. A client that retries on 409 may reprocess a request even when the server already accepted it.

Using a simple UUID is reliable, but clients often use a combination of client_id + timestamp + payload_hash. If the hash algorithm differs between client and server (e.g., MD5 vs. SHA-256), the same request may be treated as different.

In a microservices architecture, idempotency keys must be propagated across services. A service may generate a key, but a downstream service may use a different key format or fail to persist the key. The system becomes idempotent only when all services are aligned.

Idempotency with API Versioning

Idempotency keys can be versioned to support backward compatibility. When a service evolves, clients may send requests with different payloads. The server can use the idempotency_key to store per-version responses.

For example, in a service that supports two versions of the create_order endpoint:

The idempotency_keys table can store the version information:

ALTER TABLE idempotency_keys ADD COLUMN api_version VARCHAR(10) DEFAULT 'v1';

When a request arrives, the server uses the api_version to determine how to process the request and which response format to return.

Summary: Idempotency as a Design Pattern

Idempotency is not a feature—it is a design principle for reliability. The core of the pattern lies in client-provided keys that the server uses to deduplicate requests. The pattern works best when:

When implemented correctly, idempotency eliminates the cost of retries, simplifies error recovery, and makes distributed systems predictable and resilient.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.