Building a Distributed Counter with Raft and Gossip: My Experiment With Consensus and Cluster Coordination

2025-04-21

TLDR: Over the last few weekends, I’ve been experimenting with a distributed systems project that tackles a deceptively simple problem: generating atomic sequence numbers (like a distributed auto-increment ID generator) across multiple nodes — with fault tolerance, consistency, and scalability baked in.

Yes — there are existing solutions like Redis, Etcd, Consul, or even managed services that offer counters, coordination, and consistency out-of-the-box. But the goal of this project wasn’t just to build another distributed counter. The real purpose was to learn.

I set out to build the smallest possible service that still forces you to confront the big questions of distributed systems. The resulting project is a Go application that exposes a single HTTP interface, yet behind that simplicity hides a cluster of cooperating nodes. Each node participates in the Raft consensus protocol to replicate state and uses a Gossip layer (HashiCorp memberlist) to discover its peers. Snapshots allow nodes to restart without losing progress, while a recovery routine lets stale nodes pull the latest state from the leader before re-joining.

Why a distributed counter is a deceptively rich learning exercise

On paper a counter is a single uint64 that increments. In practice, keeping that value monotonic, unique and durable across machines is anything but trivial.

The exercise forces you to wrestle with:

I found that a counter is “just complex enough”: the business logic fits in a few lines, so every other line of code is about the distributed plumbing — perfect for honing intuition.

A real-world need: Global API rate-limiting

Imagine a SaaS platform with edge gateways in North America, Europe, and Asia. Each gateway must enforce a customer quota — say 10,000 requests per minute — without calling back to a single database on every request.

A Raft-backed, monotonic counter solves the problem by handing every request a globally ordered ticket, yet it still flows through one leader. To tighten accuracy while keeping latency low, we take advantage of Raft’s two roles:

Unified timeline, loose coupling. At the start of each minute a gateway reads the counter from its nearest non-voter — say the value is 42,710,000. Later it processes request 42,720,042 and instantly knows the tenant has used ~10,042 calls. Any in-flight replication lag will be a handful of IDs, not hundreds, because the three voters already agreed on order before the non-voters relayed the data.

Statistical fairness over mathematical perfection. Because every region — voter or non-voter — shares the same sequence, “bursting” traffic across continents no longer doubles the quota. The worst a client can do is sneak in a few calls already sitting in the replication pipeline; the error is bounded by the time it takes voters to ship a log entry to observers.

Operational safety valves. You can set the throttle to 10,000 + ε, where ε is your calculated propagation backlog (often ≤ 100). Voters ensure no ID is lost; non-voters ensure local reads are fast. Together they keep enforcement strict enough for billing while avoiding false positives during transient hiccups.

Automatic recovery. If a region partitions, its edge non-voters pause increments. When connectivity returns, they catch up from snapshots pushed by the voters and resume rate checks — no duplicate IDs, no quota drift.

By mixing voter nodes for strong consensus and non-voter observers for fast local reads, the distributed counter delivers near-exact, globally enforced rate limits — tight enough for real-world SLAs while still meeting edge-latency demands.

How Raft keeps the counter consistent — and where it bites back

Raft revolves around a single leader that ships an append-only log of commands to its followers. A command (e.g. {"op":"increment"}) is committed when the leader and a majority of followers have written it. Only then does the finite-state machine (FSM) apply the change, bumping the counter.

Strengths

Limitations

In the simple counter system, I mitigated these limits by keeping the write rate modest (a counter increment is cheap) and by allowing nodes to bootstrap from snapshots when logs grow long.

Working with Raft and Finite-State Machine

Raft governs how every change is agreed on, but it delegates what a change actually does to a user-supplied Finite-State Machine (FSM). Think of the FSM as “the business logic that runs once consensus is reached.”

Leader, log, commit — then FSM

  1. Client request: A gateway calls POST /next.
  2. Leader append: The current Raft leader appends a log entry that encodes the command {"op":"increment"}.
  3. Replication: The leader ships that entry to followers; when a majority acknowledges it, the entry is committed.
  4. FSM.Apply: Each node now invokes Apply() on its local FSM with that log entry; only at this point does the in-memory counter change.
// fsm.go (simplified)
type FSM struct {
    mu      sync.Mutex
    Counter uint64
}

func (f *FSM) Apply(logEntry *raft.Log) interface{} {
    f.mu.Lock()
    defer f.mu.Unlock()

    var cmd types.Command
    _ = json.Unmarshal(logEntry.Data, &cmd)

    if cmd.Op == "increment" {
        f.Counter++
        return f.Counter
    }
    return nil
}

func (f *FSM) Snapshot() (raft.FSMSnapshot, error) {
    f.mu.Lock()
    defer f.mu.Unlock()
    return &snapshot{Counter: f.Counter}, nil
}

func (f *FSM) Restore(rc io.ReadCloser) error {
    return json.NewDecoder(rc).Decode(&f.Counter)
}

Why this matters to learning

By writing the FSM yourself you feel the contract Raft enforces:

In my counter system, the FSM is tiny — just a locked uint64 — so you can focus on the protocol’s guarantees rather than domain complexity. Yet the exact same hooks scale to sophisticated state machines in real production clusters.

What Gossip adds — and why the blend works

Raft excels at agreement once you know who the servers are. It says nothing about discovering them. That’s where Gossip comes in.

Each node runs memberlist, broadcasting lightweight heartbeats over UDP. When a node sees a new peer, it merges that peer into its membership list and shares it onward. Failures are detected probabilistically: if several consecutive heartbeats are missed, the node is marked suspect and eventually dead.

Why pair the two protocols?

In practice the combination felt natural: memberlist gave me a near-real-time directory of HTTP addresses, and Raft gave me confidence that exactly one of those addresses would authoritatively hand out the next number.

This allowed me to simulate a real-world environment where nodes could appear or disappear without human intervention. That’s where HashiCorp’s memberlist came in. I used Gossip to detect and connect to other nodes automatically:

mlist, _ := memberlist.Create(memberlist.DefaultLANConfig())

go func() {
    for {
        time.Sleep(30 * time.Second)
        mlist.Join([]string{"127.0.0.1:7945", ...}) // try all known ports
    }
}()

Because I was running all of this on my local machine, my port assignment strategy is not suitable for a production environment. Ideally, the Gossip port should be the same on all machines to make the discovery simpler.

I also made a design decision that involves starting the Raft system and the HTTP API server after Gossip completes a first scan of all available nodes.

Testing

To validate the design, I spun up five local nodes, then:

Throughout these experiments, the counter remained monotonic and the service never returned an error to the client — exactly the resilience I set out to explore.

One of the interesting situations was the removal of dead nodes. Raft treats the cluster membership list as just another piece of replicated state, so adding or removing a server is itself a log entry that must be committed by a majority. When you remove a node through the leader, every follower sees exactly the same configuration change and agrees on the new quorum size. The situation can get more complicated when you end up with a cluster without a majority and no leaders, which can be a topic for another article.

What I Took Away from This

Join the conversation on LinkedIn