Distributed Locking

Distributed locking enables multiple processes across a network to coordinate access to shared resources, ensuring that only one process can modify a resource at a time—even when those processes run on different machines, in different services, or across cloud regions. It is essential when a system must avoid race conditions in scenarios like leader election, file processing pipelines, database migration coordination, and idempotent job execution. Distributed locking becomes critical when a single-node locking mechanism—like a file lock or a database row lock—cannot scale, or when failures in one node must not block others from acquiring the lock.

Acquiring a Lock

To acquire a lock, a process sends a request to a shared lock service or to a shared data store, specifying the lock name and a duration for which it intends to hold the lock. The lock service returns a success or failure status, and the caller holds the lock until it explicitly releases it or until the lock expires.

The most common protocol is lease-based locking, where a process acquires a lock by writing a record to a shared store (e.g., a database table) with an identifier (e.g., a process ID or UUID), a lock name, and a timestamp for when the lock was acquired. The lock is considered held as long as the process keeps the lease alive—either by renewing it before expiration or by relying on a time-to-live (TTL).

-- Example schema for a distributed lock table
CREATE TABLE distributed_locks (
    lock_name VARCHAR(255) NOT NULL,
    owner_id VARCHAR(255) NOT NULL,
    acquired_at TIMESTAMP WITH TIME ZONE NOT NULL,
    expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
    PRIMARY KEY (lock_name)
);

A process acquires a lock by attempting to insert or update a record in this table. The operation must be atomic to avoid race conditions.

-- Attempt to acquire a lock named 'backup_jobs'
INSERT INTO distributed_locks (lock_name, owner_id, acquired_at, expires_at)
VALUES ('backup_jobs', 'worker-123', NOW(), NOW() + INTERVAL '30 seconds')
ON CONFLICT (lock_name)
DO UPDATE SET
    owner_id = EXCLUDED.owner_id,
    acquired_at = EXCLUDED.acquired_at,
    expires_at = EXCLUDED.expires_at
WHERE distributed_locks.expires_at < NOW()
RETURNING lock_name, owner_id, acquired_at, expires_at;

If the ON CONFLICT clause returns a row, the process successfully acquired the lock. If it returns no rows, the lock was already held by another process.

A common error occurs when a process acquires a lock but the database connection drops before the INSERT completes. The process may assume it holds the lock, but the insert was not committed, leading to a phantom lock. To prevent this, use SERIALIZABLE transaction isolation or ensure the insert is durable via COMMIT with WAL_SYNC enabled.

Renewing a Lock Lease

A lock acquired with a TTL is vulnerable to expiration if the process fails or becomes unresponsive. To prevent this, processes must renew the lock lease before it expires.

Renewal is typically done via a periodic background task or heartbeat. The process sends an update to the lock table, extending the expires_at timestamp.

-- Renew the 'backup_jobs' lock for another 30 seconds
UPDATE distributed_locks
SET expires_at = NOW() + INTERVAL '30 seconds'
WHERE lock_name = 'backup_jobs'
  AND owner_id = 'worker-123'
  AND expires_at > NOW();

The WHERE clause ensures that the process only updates the lock if it still owns it (i.e., the lock has not been taken by another process). If the UPDATE affects zero rows, the process has lost the lock—possibly due to a network failure or a concurrent acquisition.

This pattern is known as lease renewal with validation. It prevents lock stealing—where a new process takes over a lock while the original owner is busy—and ensures that only the current owner can renew the lock.

Releasing a Lock

A process releases a lock by updating the expires_at field to a time in the past, effectively invalidating the lease.

-- Release the 'backup_jobs' lock
UPDATE distributed_locks
SET expires_at = NOW() - INTERVAL '1 hour'
WHERE lock_name = 'backup_jobs'
  AND owner_id = 'worker-123';

This is idempotent: multiple calls to release the same lock by the same process are safe. If the process dies after writing the update but before committing, the lock is released, and a new process can take it.

A subtle but critical failure mode arises when two processes attempt to release the same lock concurrently. Without a WHERE clause, both processes might update the same row, overwriting each other’s work. The owner_id check ensures that only the correct process can release the lock, and it prevents double-release.

Handling Lock Contention and Failover

When multiple processes compete for the same lock, they must resolve contention. A process that fails to acquire a lock can either wait (spin or block) or retry.

A common pattern is exponential backoff with jitter. After failing to acquire a lock, a process waits a random time before retrying.

// Pseudocode for acquiring a lock with exponential backoff
func AcquireLock(lockName string, ownerID string, timeout time.Duration) (bool, error) {
    var attempts int
    for {
        query := `
            INSERT INTO distributed_locks (lock_name, owner_id, acquired_at, expires_at)
            VALUES ($1, $2, NOW(), NOW() + INTERVAL '30 seconds')
            ON CONFLICT (lock_name)
            DO UPDATE SET
                owner_id = EXCLUDED.owner_id,
                acquired_at = EXCLUDED.acquired_at,
                expires_at = EXCLUDED.expires_at
            WHERE distributed_locks.expires_at < NOW()
            RETURNING lock_name
        `
        _, err := db.Exec(query, lockName, ownerID)
        if err == nil {
            return true, nil
        }

        attempts++
        wait := time.Duration(2^attempts) * time.Second
        jitter := time.Duration(rand.Intn(1000)) * time.Millisecond
        time.Sleep(wait + jitter)

        if time.Since(startTime) > timeout {
            break
        }
    }
    return false, errors.New("failed to acquire lock")
}

This pattern avoids thundering herds—where many processes wake up simultaneously and attempt to acquire the same lock—by spreading the load over time.

In a failover scenario, if a process holding a lock dies, the lock expires and becomes available to other processes. However, the process may have been in the middle of work when it failed. The system must tolerate stale locks.

A process can detect that it has lost a lock by comparing the acquired_at timestamp of the lock with its last renewal. If the acquired_at has changed since the last renewal, the process has been preempted.

Detecting Lock Loss

A process can proactively detect lock loss by running a heartbeat task that periodically checks if the lock is still valid.

-- Check if the lock has been renewed recently
SELECT
    lock_name,
    owner_id,
    acquired_at,
    expires_at,
    EXTRACT(EPOCH FROM (NOW() - acquired_at)) AS age_seconds
FROM distributed_locks
WHERE lock_name = 'backup_jobs'
  AND owner_id = 'worker-123'
  AND expires_at > NOW();

If this query returns no rows, the process has lost the lock. It can then either reacquire it or log the loss and fail over.

To make this detection robust, processes should also record the last renewal time and compare it with the acquired_at. If the acquired_at has been updated by another process, the current owner has been preempted.

Coordinating Multiple Locks

In complex workflows, a process may need to hold multiple locks simultaneously. For example, a job scheduler might hold a lock on a job queue, a lock on a configuration file, and a lock on a database schema.

The order in which locks are acquired matters. A process should acquire locks in a canonical order, such as by lock name alphabetically or by priority.

-- Acquire multiple locks in order
BEGIN;
SELECT pg_advisory_lock(12345); -- global lock
SELECT pg_advisory_lock(67890); -- job queue lock
SELECT pg_advisory_lock(11223); -- config lock
-- Perform work
COMMIT;

Using PostgreSQL’s pg_advisory_lock, a process can acquire multiple locks in a transaction, ensuring that either all locks are acquired or none.

If a process fails between acquiring the first and second lock, the system can detect this and retry the entire sequence. The use of advisory locks ensures that locks are not tied to specific database rows or tables, making them lightweight and flexible.

Failure Modes and Pitfalls

• Clock drift: If the clocks on different nodes are not synchronized, a process may renew a lock too early or too late. A 1-second drift across nodes can cause lock loss when a process fails just after renewing the lock.

• Network partition: A process may renew a lock successfully, but the network partitions before the response is acknowledged. The process assumes it holds the lock, but the update is lost. The system must handle unacknowledged leases.

• TTL misalignment: A process sets a lock TTL to 30 seconds, but the database server applies a 25-second TTL due to pg_hba.conf settings. The lock expires 5 seconds earlier than expected, leading to premature lock loss.

• Multiple renewals from different processes: If two processes renew the same lock simultaneously, one may overwrite the other’s expires_at. Without a WHERE clause checking owner_id, this leads to stale updates.

• Stale lock detection without heartbeat: A process may believe it holds a lock, but the lock has been renewed by another process. Without periodic checks, the process may continue work using stale data.

• Lock leakage: A process acquires a lock but fails to release it. The lock remains in the system indefinitely, even after the process dies. This can be mitigated by using automatic cleanup—a background job that removes locks where expires_at < NOW().

Best Practices

• Use lease-based locking with TTLs and renewal.
• Always validate ownership when renewing or releasing a lock.
• Use exponential backoff with jitter when acquiring locks.
• Check lock status periodically, especially in long-running processes.
• Acquire multiple locks in a consistent order to avoid deadlocks.
• Use advisory locks for lightweight, non-transactional coordination.
• Monitor lock acquisition and renewal latency to detect performance degradation.
• Log lock acquisition and release with metadata: process ID, timestamp, and duration.

Distributed locking is not a single operation but a pattern of coordination across processes. It succeeds when the system behaves predictably under failure, and fails when locks are acquired, lost, or renewed without coordination. The pattern is general, portable, and robust—built on the foundation of a shared data store, a consistent clock, and a reliable network.

This page was rewritten on 18 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.