Distributed Systems Consensus

Consensus algorithms solve the problem of achieving agreement among multiple nodes in a distributed system when individual nodes may fail, messages may be lost, or network partitions may occur. This matters when building systems that require reliable state coordination across multiple machines.

When to Use Consensus

Use consensus when you need to maintain consistent state across multiple nodes and cannot tolerate divergent views of data. This includes:

• Implementing distributed databases
• Managing leader election in clusters
• Coordinating distributed transactions
• Maintaining replicated state machines
• Building fault-tolerant systems with no single point of failure

Consensus Algorithms and Their Properties

Raft Consensus Algorithm

Raft is a consensus algorithm designed for ease of understanding and implementation. It divides the consensus problem into three subproblems:

1. Leader election
2. Log replication
3. Safety

In Raft, each node can be in one of three states: follower, candidate, or leader.

To configure a Raft cluster, you typically define a set of peer addresses in your configuration:

raft.peers = ["node1:8080", "node2:8080", "node3:8080"]
raft.id = "node1"
raft.storage.path = "/var/lib/raft"

When a node starts, it begins as a follower. If no heartbeat is received from the leader within the election timeout, the node transitions to candidate and initiates an election.

If a majority of nodes vote for a candidate, that node becomes the leader. The leader then replicates log entries to followers.

Multi-Node Cluster Setup

For a Raft-based system, configure each node with:

# Example configuration file
[raft]
id = "node1"
peers = ["node1:8080", "node2:8080", "node3:8080"]
storage.path = "/var/lib/raft"
log.level = "info"

The node will attempt to join the existing cluster. If it's the first node, it creates a new cluster.

Leader Election

When leader election occurs:

1. A follower receives no heartbeat from the current leader
2. The follower increments its term and transitions to candidate
3. It sends RequestVote RPCs to all other nodes
4. If it receives votes from a majority of nodes, it becomes leader
5. If no leader emerges within the election timeout, another election begins

If a node fails during election, it will miss heartbeat messages and trigger a new election cycle.

Log Replication

Once a leader is established:

1. The leader receives client requests
2. It appends entries to its local log
3. It replicates log entries to followers via AppendEntries RPCs
4. When entries are replicated to a majority of nodes, they are considered committed
5. The leader applies committed entries to its state machine

Safety Properties

Raft ensures several safety properties:

• Leader election safety: Only one leader can be elected per term
• Log matching property: If two logs contain the same entry at the same index, then they have identical entries up to that index
• State machine safety: If a leader has applied an entry to its state machine, no follower can apply a different entry for that index

Consensus Failure Modes

Network Partition

When network partitions occur, Raft handles them by:

1. Nodes in the minority partition cannot elect a leader
2. Nodes in the majority partition continue operating normally
3. When partition heals, the system must handle log divergence

Example error from a node in minority partition:

raft: failed to replicate entry to follower: rpc error: connection refused

Node Failure During Election

If a node fails during election:

1. The election timeout expires for that node
2. It transitions to candidate and starts new election
3. If it receives a heartbeat from a newly elected leader, it becomes follower again

Log Divergence

When nodes have divergent logs after a partition heals:

1. The leader must replicate entries to catch up followers
2. Followers may need to truncate their logs
3. This can cause data loss if not handled carefully

Consensus Configuration Options

Election Timeout

The election timeout is typically configured in milliseconds:

raft.election.timeout = 5000

If a node doesn't receive a heartbeat within this time, it starts an election.

Heartbeat Interval

Heartbeat interval is usually shorter than election timeout:

raft.heartbeat.interval = 1000

This controls how often leaders send heartbeats to followers.

Log Persistence

Log entries are typically persisted to disk:

raft.log.persist = true
raft.log.storage.path = "/var/lib/raft/log"

Practical Implementation Considerations

Node Configuration

Each node must be configured with a unique identifier and list of peers:

# Config file example
[raft]
id = "node2"
peers = ["node1:8080", "node2:8080", "node3:8080"]
storage.path = "/var/lib/raft"

The system will attempt to connect to all peers listed in the configuration.

Starting a Cluster

To start a new cluster:

1. Configure each node with the same set of peer addresses
2. Start nodes one at a time
3. The first node to start becomes the initial leader
4. Subsequent nodes join the cluster

If multiple nodes are started simultaneously, the election process begins automatically.

Cluster Recovery

When restarting a cluster:

1. Nodes must be able to recover their state from persisted logs
2. If logs are inconsistent, recovery may require manual intervention
3. The system should attempt to maintain quorum when possible

Common Issues and Gotchas

Election Timeout Configuration

If election timeouts are too short, the system will experience frequent elections:

raft.election.timeout = 1000

This can lead to instability if network conditions cause temporary delays.

Log Persistence Failure

If log persistence fails, nodes may lose committed entries:

raft.log.persist = false

This should be avoided in production systems.

Node Joining

When a new node joins the cluster, it must catch up on all committed entries. If this fails, it can cause consensus failures.

Quorum Requirements

Raft requires a majority of nodes to form a quorum:

• 1 node: 1 quorum
• 2 nodes: 2 quorum (both required)
• 3 nodes: 2 quorum
• 4 nodes: 3 quorum
• 5 nodes: 3 quorum

If fewer than the required number of nodes are available, the system cannot make progress.

Consensus in Practice

Client Interaction

Clients interact with the consensus system through a leader:

1. Clients send requests to any node in the cluster
2. If the receiving node is not the leader, it forwards the request to the leader
3. The leader processes the request and replicates it to followers
4. Once committed, the result is returned to the client

Replication Lag

There may be a delay between when a request is made and when it's applied:

# Example response from cluster
{
  "status": "success",
  "committed_index": 100,
  "applied_index": 95,
  "timestamp": "2023-01-01T12:00:00Z"
}

The applied_index shows how far behind the node is in processing committed entries.

Monitoring Consensus Health

Key metrics to monitor include:

• Election count
• Log entry replication delay
• Heartbeat intervals
• Node status (follower, candidate, leader)

Handling Failures

When a node fails:

1. It stops sending heartbeats
2. Other nodes detect the failure after timeout
3. If it's the leader, a new election begins
4. If it's a follower, the cluster continues with remaining nodes

Recovery Process

When restarting a failed node:

1. The node reads its persisted state
2. It attempts to join the existing cluster
3. If it's behind, it replicates missing logs
4. It becomes a follower and starts participating in consensus

Configuration Examples

Minimal Configuration

[raft]
id = "node1"
peers = ["node1:8080"]
storage.path = "/var/lib/raft"

This creates a single-node cluster.

Multi-Node Configuration

[raft]
id = "node2"
peers = ["node1:8080", "node2:8080", "node3:8080"]
storage.path = "/var/lib/raft"
election.timeout = 5000
heartbeat.interval = 1000

This configures a three-node cluster with standard timeouts.

Runtime Behavior

Leader Responsibilities

The leader is responsible for:

• Replicating log entries to followers
• Processing client requests
• Sending periodic heartbeats
• Handling failures in the cluster

Follower Responsibilities

Followers are responsible for:

• Replying to heartbeat messages
• Replying to vote requests
• Applying committed log entries to their state machine
• Logging incoming entries

Candidate Behavior

When a node becomes a candidate:

1. It increments its term
2. It votes for itself
3. It sends RequestVote RPCs to all peers
4. It waits for a majority of votes or new leader information

Implementation Notes

Consensus implementations typically include:

• Log compaction to prevent logs from growing indefinitely
• Snapshotting to reduce log size
• Membership changes to add or remove nodes
• Client request handling with proper error reporting

The implementation should handle node failures gracefully and maintain consistency across all nodes in the cluster.

This page was rewritten on 15 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.