This README is the one indexed from the repository at its latest ref, not from the tag for this version.

raft.cr

A Crystal implementation of the Raft consensus algorithm for building distributed systems with strong consistency guarantees.

Features

• Leader election with pre-vote protocol (prevents term inflation from partitioned nodes)
• Log replication with conflict detection and per-follower replicator fibers
• Pluggable transport -- TCP with TLV framing and HMAC-SHA256 authentication, or in-memory for testing
• Pluggable storage -- in-memory or file-backed with crash recovery and log compaction
• Pluggable state machine -- implement three methods to replicate any application state
• Snapshots -- multi-chunk transfer for lagging followers
• Dynamic membership -- add/remove voters, learner (non-voting) members with safe promotion
• RTT auto-tuning -- leader measures round-trip time, distributes optimal heartbeat/election timeouts cluster-wide
• Active-passive mode -- optional quorum=1 for 2-node clusters (availability over consistency)
• Metrics -- elections, proposals, commits, term, and role observable via node.metrics
• Deterministic testing -- all non-determinism is injected; full cluster tests run in-process

Installation

Add the dependency to your shard.yml:

dependencies:
  raft:
    github: threez/raft.cr

Then run shards install.

Quick Start

require "raft"

# 1. Implement your state machine
class MyApp < Raft::StateMachine
  def apply(command : Bytes) : Bytes
    # execute command, return result
  end

  # Optional: override for bulk optimizations (e.g. a single DB transaction per batch)
  def apply(commands : Array(Bytes)) : Array(Bytes)
    commands.map { |cmd| apply(cmd) }
  end

  def snapshot : Bytes
    # serialize state
  end

  def restore(io : IO) : Nil
    # restore from snapshot
  end
end

# 2. Create and start a node
node = Raft::Node.new(
  id: "node-1",
  peers: ["node-2", "node-3"],
  state_machine: MyApp.new,
  transport: Raft::Transport::TCP.new("0.0.0.0", 7001, {
    "node-2" => {"192.168.1.11", 7002},
    "node-3" => {"192.168.1.12", 7003},
  }, cookie: "shared-secret"),
  log: Raft::Log::File.new("/var/lib/raft/node-1"),
)
node.start

# 3. Propose commands (leader only, blocks until committed)
result = node.propose("SET key value".to_slice)

# 4. Linearizable read (confirms leadership, then reads locally)
value = node.read("GET key".to_slice)

# 5. Membership changes
node.add_peer("node-4")           # add a voting member
node.add_learner("node-5")        # add a non-voting learner
node.promote_learner("node-5")    # promote to voter once caught up
node.remove_peer("node-4")        # remove a voter or learner

# 6. Shut down
node.stop

Learner Nodes

A learner is a non-voting member that receives log replication but does not participate in elections or quorum. This allows a new node to catch up on the log safely before becoming a voter -- avoiding the risk of reducing cluster availability during catch-up.

# On the leader: add a learner
node.add_learner("new-node")

# The learner receives all log entries and snapshots via replication.
# Once it has caught up, promote it to a full voting member:
node.promote_learner("new-node")

# Optionally remove an old member to maintain cluster size:
node.remove_peer("old-node")

RTT Auto-Tuning

When enabled, the leader periodically measures round-trip time to all peers via Ping/Pong probes and automatically adjusts heartbeat and election timeouts following etcd best practices. The tuned values are distributed to all followers via ConfigUpdate RPCs, ensuring cluster-wide consistency.

config = Raft::Config.new(
  rtt_tuning: true,          # enable auto-tuning
  rtt_probe_interval: 60,    # probe every 60 seconds (default)
)

Tuning formula:

• Heartbeat interval = 1.5x median RTT (clamped to 10ms–5000ms)
• Election timeout min = 10x median RTT
• Election timeout max = 20x median RTT

Active-Passive Mode

For 2-node clusters, standard Raft requires both nodes for quorum — if one fails, the cluster is unavailable. Active-passive mode lowers quorum to 1, allowing either node to operate independently.

config = Raft::Config.new(active_passive: true)

Trade-off: During a network partition, both nodes may accept writes. When the partition heals, the node with the lower term loses its divergent entries. Use only when availability is more important than strict consistency.

Example: Distributed KV Store

A complete 3-node KV store with an HTTP API lives in examples/kv_store/.

# Build and run the cluster
./examples/kv_store/run.sh

# In another terminal
bin/kv_client put hello world    # store a value
bin/kv_client get hello          # retrieve it
bin/kv_client delete hello       # remove it
bin/kv_client status             # view node metrics
bin/kv_client leader             # find the leader

# Or use curl directly
curl -X PUT localhost:8001/hello -d 'world'
curl localhost:8001/hello
curl localhost:8001/_status

The example demonstrates TCP transport, file-backed persistence, HMAC authentication, HTTP API with automatic leader redirection (307), and a CLI client.

To run a load test with k6:

./examples/kv_store/bench.sh

Architecture

Client API (propose / read / add_peer / add_learner / promote_learner / stop)
        |
   Raft::Node  [Follower <-> Candidate <-> Leader]
        |
   Log / RPC / Config
        |
  +-----------+--------------+
  |                          |
StateMachine (yours)    Transport (TCP / InMemory)

Each node runs a single event-loop fiber -- all state mutations happen on one fiber, eliminating locks. Leader spawns per-follower replicator fibers that send AppendEntries and heartbeats. Communication uses TLV binary framing with HMAC-SHA256 cookie authentication on TCP connections.

See the design documentation for details:

• Architecture -- design principles, component diagram, module structure
• Protocol -- TLV wire format, handshake protocol, on-disk log format
• Concurrency -- fiber model, epoch-gated timers, replicator design

API Overview

Class	Purpose
Raft::Node	Consensus node -- start, stop, propose, read, add_peer, add_learner, promote_learner, remove_peer
Raft::StateMachine	Abstract -- implement apply, snapshot, restore
Raft::Log::InMemory	In-memory log for testing
Raft::Log::File	File-backed log with crash recovery
Raft::Transport::TCP	TCP transport with TLV + HMAC-SHA256
Raft::Transport::InMemory	In-process transport for testing (with partition simulation)
Raft::RTTMonitor	Leader-driven RTT measurement and timeout auto-tuning
Raft::Config	Election timeouts, heartbeat interval, cookie, RTT tuning, active-passive
Raft::Metrics	Observable counters -- access via node.metrics

Full API documentation: crystal docs

Development

make           # clean, format, lint, docs, spec
make spec          # run tests (~107 specs)
make lint          # ameba linter
make fmt           # crystal format
make bench         # all benchmarks (codec, log, propose, cluster)
make bench-codec   # RPC encode/decode/roundtrip for all message types
make bench-log     # log append (InMemory + File)
make bench-propose # propose path micro-benchmarks
make bench-cluster # election latency + propose throughput
make bench-mt      # all benchmarks with preview_mt (multi-threaded)
make example       # build KV store server + client
make docs          # generate API docs

Requires Crystal >= 1.19.1.

Contributing

1. Fork it (https://github.com/threez/raft.cr/fork)
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

License

MIT -- Vincent Landgraf