Distributed Transactions | The Garnet Wiki

Distributed transactions coordinate work across multiple database systems or services, ensuring that a set of operations—each potentially on a different node, storage engine, or network partition—either all succeed or all fail as a unit. They matter when a business operation spans more than one database, such as booking a flight and reserving a hotel room, where failure in either step must roll back the other. They are essential in microservices architectures, event-driven systems, and any application requiring strong consistency across boundaries.

Starting a Distributed Transaction

Begin a distributed transaction by initiating a global transaction context that spans multiple participants. The initiating service starts a transaction ID (XID) and broadcasts it to all involved databases and services.

POST /transactions
Content-Type: application/json

{
  "xid": "20240515T123456.789012",
  "isolation": "SERIALIZABLE",
  "timeout": 300
}

The transaction manager assigns a unique XID, typically a string with a timestamp and a sequence number, which becomes the transaction’s identity. This XID is used in every subsequent interaction.

Participants register with the transaction manager, often via a two-phase commit (2PC) protocol. Each participant logs the transaction in a local transaction log, including the XID, and sends a confirmation.

-- On a PostgreSQL participant
INSERT INTO transaction_log (xid, participant, state, registered_at)
VALUES ('20240515T123456.789012', 'flight-db', 'REGISTERED', NOW());

The XID must be passed across service boundaries via HTTP headers, message headers (e.g., in Kafka or RabbitMQ), or RPC call metadata.

Coordinating Multiple Participants

A distributed transaction involves multiple services, each managing its own data. To ensure consistency, all participants must agree on a common decision: commit or rollback.

Phase 1: Prepare

Each participant prepares its local transaction, locks required data, and writes a prepare record. The transaction manager queries each participant to prepare.

GET /transactions/20240515T123456.789012/prepare

Each participant responds with a 200 OK and a Prepared status, including a unique prepare token and a list of changes.

{
  "xid": "20240515T123456.789012",
  "participant": "hotel-db",
  "prepared_at": "2024-05-15T12:35:01.234Z",
  "prepare_token": "PT-8321",
  "changes": [
    { "table": "reservations", "rows": 1, "operation": "INSERT" },
    { "table": "inventory", "rows": 1, "operation": "UPDATE" }
  ],
  "status": "PREPARED"
}

If any participant fails to prepare, the transaction manager marks the global transaction as “in-doubt” and may retry or escalate.

Phase 2: Commit or Rollback

Once all participants have prepared, the transaction manager sends a commit or rollback command to all.

POST /transactions/20240515T123456.789012/commit
Content-Type: application/json

{
  "xid": "20240515T123456.789012",
  "prepared_tokens": [
    "PT-8321",
    "PT-4567",
    "PT-1234"
  ],
  "global_timestamp": "2024-05-15T12:36:00.000Z"
}

Each participant applies the commit, updates its log, and responds with success or failure.

200 OK
{
  "xid": "20240515T123456.789012",
  "participant": "flight-db",
  "committed_at": "2024-05-15T12:36:02.100Z",
  "status": "COMMITTED"
}

If a participant fails during commit, the transaction manager can retry the commit using the prepare token. If the coordinator fails after sending commit but before receiving all acknowledgments, the transaction is in a durable but uncertain state—a common production failure.

Handling Failures

Participant Failure During Prepare

If a participant crashes after writing to its transaction log but before sending a 200 OK to the prepare request, the coordinator must detect this and retry the prepare.

The coordinator logs the prepare request in a durable queue. On startup, it polls participants and re-sends prepare requests for any unacknowledged XIDs.

Coordinator Failure After Phase 1

After all participants have prepared, the coordinator crashes before sending commit. On restart, it checks the transaction log and finds that the transaction is in the “preparing” state. It sends a commit request to all participants, using the prepare tokens to ensure idempotency.

Participants use the prepare token to deduplicate commit requests. If a participant receives a commit request with a token it has already processed, it responds with 200 OK but does nothing.

-- On a MySQL participant
INSERT INTO transaction_log (xid, participant, state, prepare_token, committed_at)
VALUES ('20240515T123456.789012', 'payment-db', 'COMMITTED', 'PT-4567', NOW())
ON DUPLICATE KEY UPDATE
  state = VALUES(state),
  committed_at = VALUES(committed_at);

Participant Failure During Commit

If a participant fails after writing to its transaction log during commit, but before sending the response, the coordinator may assume the commit failed. The participant may later recover and replay the commit. To handle this, the commit operation must be idempotent.

The coordinator records each commit request in a durable log. On recovery, it replays all pending commits and ensures every participant has processed each request at least once.

Network Partition and Timeout

When the network splits, a participant may not receive the commit command. The transaction manager sets a global timeout—typically 300 seconds—after which it assumes a participant has failed.

Participants can register their heartbeat with the coordinator. If a participant fails to respond to a heartbeat within the timeout interval, the coordinator marks it as unreachable and can retry the prepare or commit.

POST /transactions/20240515T123456.789012/heartbeat
Content-Type: application/json

{
  "participant": "flight-db",
  "last_seen": "2024-05-15T12:38:00.000Z",
  "status": "ALIVE"
}

If the coordinator receives a heartbeat with a last_seen time earlier than the current time minus the timeout, it triggers a recovery process: sending missing prepare or commit requests.

Configuring and Monitoring

Configuration Keys

Transaction managers are configured via standard configuration files, with the following common keys:

transaction:
  coordinator:
    timeout: 300
    max_retries: 5
    heartbeat_interval: 30
    recovery_poll_interval: 10
    log_storage:
      type: "file"
      path: "/var/log/transactions"
      retention_days: 7
    protocol: "2pc"
    recovery_strategy: "retry"
  participants:
    - name: "flight-db"
      endpoint: "http://flight-db:8080/transactions"
      max_concurrent: 100
      isolation: "SERIALIZABLE"
      consistency: "strong"
    - name: "hotel-db"
      endpoint: "http://hotel-db:8080/transactions"
      max_concurrent: 100
      isolation: "REPEATABLE_READ"
      consistency: "eventual"

The coordinator uses the timeout to determine when a participant is unresponsive. The recovery_poll_interval governs how often the coordinator checks for unacknowledged transactions.

Monitoring and Diagnostics

A distributed transaction can be monitored via its XID. The coordinator exposes a status endpoint:

GET /transactions/20240515T123456.789012/status

Response:

{
  "xid": "20240515T123456.789012",
  "status": "COMMITTED",
  "started_at": "2024-05-15T12:34:56.789Z",
  "finished_at": "2024-05-15T12:36:02.100Z",
  "participants": [
    {
      "name": "flight-db",
      "status": "COMMITTED",
      "prepared_at": "2024-05-15T12:35:00.123Z",
      "committed_at": "2024-05-15T12:36:01.456Z"
    },
    {
      "name": "hotel-db",
      "status": "COMMITTED",
      "prepared_at": "2024-05-15T12:35:01.234Z",
      "committed_at": "2024-05-15T12:36:02.100Z"
    }
  ],
  "errors": [
    {
      "participant": "payment-db",
      "timestamp": "2024-05-15T12:36:01.000Z",
      "code": "504",
      "message": "Gateway Timeout during commit"
    }
  ],
  "duration_ms": 12600
}

Common Failure Modes

XID mismatch: A participant receives a transaction request with an XID it does not recognize. This often occurs due to stale or misconfigured transaction managers.
Idempotency failures: A participant processes a commit request twice, but the second execution introduces side effects (e.g., double-charging). The coordinator must ensure every request is idempotent via primary keys or unique tokens.
Prepare timeout: A participant takes longer than expected to prepare, causing the coordinator to timeout and start a rollback. The participant may finish its prepare after the coordinator has already rolled back, leading to phantom commits.
Two-phase commit stalls: The coordinator sends a commit request, but a participant fails before writing its log. On recovery, the participant reads the commit request and applies it, but the coordinator assumes the commit failed and retries. This causes duplicate commits.
Transaction log corruption: If a participant’s transaction log is on a network filesystem and the filesystem becomes unavailable, the participant may lose its log. The coordinator can detect this by checking the last_seen timestamp and triggering a recovery.

Advanced Patterns

Compensating Transactions

When a distributed transaction cannot be rolled back atomically (e.g., due to long-running processes), a compensating transaction is used. After a partial commit, the system runs a set of operations to reverse the changes.

POST /transactions/20240515T123456.789012/compensate
Content-Type: application/json

{
  "reason": "participant_failure",
  "failed_participant": "payment-db",
  "compensation_steps": [
    {
      "operation": "refund",
      "target": "customer-credit",
      "amount": 249.99,
      "currency": "USD"
    },
    {
      "operation": "release",
      "target": "flight-seats",
      "seat_ids": [ "A1", "A2", "B1" ]
    }
  ]
}

The compensation is logged and can be replayed in case of further failures.

Nested Transactions

A distributed transaction may contain nested sub-transactions. Each sub-transaction has its own XID and coordinates with the parent.

{
  "xid": "20240515T123456.789012",
  "parent_xid": "20240515T123456.789012",
  "sub_transaction": {
    "xid": "20240515T123456.789012-1",
    "participants": [ "email-service", "notification-db" ]
  }
}

The parent transaction waits for all sub-transactions to prepare before proceeding to phase 2.

Summary

Distributed transactions coordinate work across multiple databases and services using a global transaction ID. The two-phase commit protocol ensures atomicity: all participants either commit or roll back together. The coordinator manages the state, handles failures, and provides visibility into the transaction lifecycle.

Key challenges include handling network partitions, ensuring idempotency, recovering from coordinator crashes, and diagnosing partial failures. Configuration is standardized, and monitoring is built around the XID.

The success of a distributed transaction hinges on consistent use of the XID across services, idempotent operations, and robust recovery mechanisms. When these are missing, the transaction appears to succeed, but data is inconsistent.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.