Idempotency in Distributed Systems

Design idempotent APIs and workflows that can be safely retried without side effects. Covers idempotency keys, exactly-once semantics, deduplication patterns, and the database techniques that make retries safe in distributed architectures.

Idempotency is the property of an operation that produces the same result when applied multiple times with the same input. In distributed systems, where requests can be lost, delayed, or retried, idempotency ensures that repeated invocations of the same operation do not cause unintended side effects—like duplicate orders, double payments, or redundant message deliveries. It matters most when network reliability is uncertain, components are stateless, and clients must handle retries autonomously, such as in microservices, event-driven architectures, and cloud-native applications.

When to Apply Idempotency

Designing Reusable APIs for Resilient Clients

In a distributed system, clients often send requests to services that may be unreachable, slow, or fail during the request lifecycle. A client may time out after 10 seconds and retry the request, only to find that the service already processed it. Without idempotency, a user placing an order might end up with two identical orders.

To make an HTTP API idempotent, the client must include an idempotency key in the request. The server uses this key to deduplicate incoming requests. The key is typically included in the request header:

POST /orders HTTP/1.1
Host: api.example.com
Content-Type: application/json
Idempotency-Key: 8d3f1a4e-2c9b-4a1a-8b5c-1d0e2f3a4b5c

{
  "customer_id": 123,
  "items": [
    { "product_id": 45, "quantity": 2 }
  ],
  "total": 99.99
}

The server checks if a request with this Idempotency-Key has been processed before. If yes, it returns the previously computed result instead of processing the request again. If no, it processes the request, stores the result, and records the key.

The response includes a Idempotency-Result header to indicate whether the result is from a new or cached operation:

HTTP/1.1 201 Created
Idempotency-Key: 8d3f1a4e-2c9b-4a1a-8b5c-1d0e2f3a4b5c
Idempotency-Result: new
Content-Type: application/json

{
  "order_id": 987,
  "status": "created",
  "total": 99.99
}

If the same request is retried with the same key, the server returns:

HTTP/1.1 200 OK
Idempotency-Key: 8d3f1a4e-2c9b-4a1a-8b5c-1d0e2f3a4b5c
Idempotency-Result: cached
Content-Type: application/json

{
  "order_id": 987,
  "status": "created",
  "total": 99.99
}

The client can then safely retry requests without fear of side effects.

Implementing Idempotent Message Processing

In message queues like Kafka, RabbitMQ, or Amazon SQS, producers send events that consumers process asynchronously. Messages may be redelivered when a consumer fails or times out. Without idempotency, a message processing step can be executed multiple times, leading to duplicate processing.

To ensure idempotency, consumers use message IDs as deduplication keys. The consumer maintains a set of processed message IDs. On receiving a message, it checks whether the message ID has already been seen. If so, it skips processing and acknowledges the message.

A typical message format includes an id field:

{
  "id": "c7a3b8d2-9f1e-4c5d-8b9a-1c2d3e4f5a6b",
  "event_type": "order.created",
  "payload": {
    "order_id": 987,
    "customer_id": 123,
    "items": [
      { "product_id": 45, "quantity": 2 }
    ]
  },
  "timestamp": "2025-04-05T12:00:00Z"
}

The consumer checks the message ID against a store (e.g., a database table, Redis set, or in-memory map). If the ID exists, it returns ACK without processing. If not, it processes the event and records the ID.

The key failure mode is the race between message receipt and ID insertion. If the consumer reads the message, checks the ID, and then inserts it, but the insertion fails before the ACK is sent, a subsequent retry may reprocess the same message.

To mitigate this, consumers use a two-phase idempotency strategy:

  1. Write-ahead logging: The consumer logs the message ID to a durable log (e.g., a processed_messages table) before processing.
  2. Idempotency key in the event payload: The id field is used as the primary key in the log table.
CREATE TABLE processed_messages (
  message_id UUID PRIMARY KEY,
  processed_at TIMESTAMP WITH TIME ZONE,
  result JSONB,
  processed_by TEXT
);

A consumer loop might look like:

for message in consumer.consume():
    message_id = message["id"]
    result = cache.get(f"processed:{message_id}")

    if result:
        # Already processed
        consumer.ack(message)
        continue

    # Process the message
    try:
        result = handle_event(message)
        # Log the processing
        db.execute(
            "INSERT INTO processed_messages (message_id, processed_at, result) VALUES (%s, %s, %s)",
            (message_id, now(), result)
        )
        # Confirm success
        consumer.ack(message)
        cache.set(f"processed:{message_id}", result, ttl=60*60*24)  # 24h
    except Exception as e:
        # Log failure but don’t ack
        log.error(f"Failed to process {message_id}: {e}")
        # Retry later
        consumer.nack(message)

Building Idempotent Batch Jobs

Batch jobs process large volumes of data in multiple steps. They are prone to partial failures and must be restartable. Idempotency ensures that restarting a job from any point yields the same final state.

Consider a job that imports customer data from a CSV file. The job is divided into stages: parse, validate, enrich, and load. Each stage can be idempotent.

The job maintains a job_run_id and stores the state of each stage in a database.

CREATE TABLE job_runs (
  job_run_id UUID PRIMARY KEY,
  status TEXT,
  started_at TIMESTAMP,
  completed_at TIMESTAMP,
  stage TEXT,
  last_processed_offset INT
);

Each stage updates the job run record and logs its progress. For example, the validation stage:

def validate_customers(job_run_id, data_source):
    # Get current state
    job = db.get("SELECT * FROM job_runs WHERE job_run_id = %s", job_run_id)

    # Read from offset
    records = read_csv(data_source, from_offset=job.last_processed_offset)

    for record in records:
        # Idempotent validation
        try:
            validated = validate(record)
            db.execute(
                "INSERT INTO validated_customers (customer_id, status, errors) VALUES (%s, %s, %s)",
                (validated.id, "valid", validated.errors)
            )
            # Update job state
            db.execute(
                "UPDATE job_runs SET last_processed_offset = %s, stage = 'validate' WHERE job_run_id = %s",
                (record.offset, job_run_id)
            )
        except Exception as e:
            db.execute(
                "INSERT INTO job_errors (job_run_id, stage, error, occurred_at) VALUES (%s, %s, %s, %s)",
                (job_run_id, "validate", str(e), now())
            )
            # Log and continue
            continue

    # Finalize
    db.execute(
        "UPDATE job_runs SET status = 'completed', stage = 'enrich', completed_at = %s WHERE job_run_id = %s",
        (now(), job_run_id)
    )

If the job crashes after validating 10,000 records, restarting it picks up from offset 10,000. The enrich stage uses the same job_run_id to deduplicate enrichments.

A common mistake is assuming that a job_run_id is sufficient for idempotency. But if the job writes to a table that doesn’t enforce uniqueness on job_run_id, duplicate enrichments can occur. The solution is to use the job run ID as a deduplication key in every stage.

Handling Idempotent State Transitions

Many systems model state machines: a user account can be pending, verified, suspended, deleted. Transitions between states must be idempotent.

For example, a verify_user endpoint that marks a user as verified. Without idempotency, calling it twice results in two verification events, even if the user was already verified.

To make the transition idempotent, the client includes a transition_id in the request:

POST /users/123/actions/verify HTTP/1.1
Host: api.example.com
Content-Type: application/json
Transition-Id: a1b2c3d4-e5f6-7890-a1b2-c3d4e5f6a7b8

{}

The server checks if a transition with transition_id has already been applied. If yes, it returns the existing state. If not, it updates the user and records the transition.

The database table for transitions:

CREATE TABLE user_transitions (
  transition_id UUID PRIMARY KEY,
  user_id INTEGER,
  from_state TEXT,
  to_state TEXT,
  applied_at TIMESTAMP WITH TIME ZONE,
  metadata JSONB
);

A query to apply a transition:

WITH transition AS (
  INSERT INTO user_transitions (transition_id, user_id, from_state, to_state, metadata)
  SELECT
    $1, $2, u.state, 'verified', $3
  FROM users u
  WHERE u.id = $2
  RETURNING transition_id, user_id, from_state, to_state
)
UPDATE users
SET state = 'verified', updated_at = NOW()
WHERE id IN (SELECT user_id FROM transition)
RETURNING users.id, users.state;

If the transition is applied twice, the second call returns 200 OK with the updated user, but does not change the state again.

Common Pitfalls and Silent Failures

Best Practices

Idempotency is not a feature—it is a discipline. It requires consistency across services, clear contracts, and careful handling of failure modes. In distributed systems, where failure is the norm and retries are inevitable, idempotency is not optional. It is the foundation of reliable, resilient, and predictable systems.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.