Coordination without a center

A gossip protocol is how a large group of machines agrees on facts about itself when no machine is in charge. The way to understand it is to look first at what this application does instead.

What this application does instead

There are three coordination mechanisms here, and all three are the same mechanism.

# config/queue.yml (abridged)
dispatchers:
  - polling_interval: 1
    batch_size: 500
workers:
  - queues: "default"
    threads: 3
    polling_interval: 0.1
  # two more worker pools, each with its own queue and interval

# config/cable.yml (production)
polling_interval: 0.1.seconds
message_retention: 1.day

# config/deploy.yml (proxy section)
healthcheck:
  path: /up
  interval: 5

A job dispatcher asks the database every second. A worker pool asks about every tenth of a second. Each server process's cable listener asks every tenth of a second, however many WebSocket subscribers it has. And during a deploy, the proxy asks the new container every five seconds until it answers.

Nobody tells anybody anything. Every participant keeps asking a single shared authority, either the database or the application itself, and that authority is the whole coordination mechanism.

Why that works here

It works because the authority exists and is reachable: one PostgreSQL node, two Puma processes, one proxy.

At this size, polling a shared store is the right design, and it's worth saying so plainly instead of treating it as a limitation. It's simple, it has no failure modes of its own beyond the store's, and it costs one query per interval per participant.

That last point is where it stops working. Cost per participant times the number of participants is the scaling property, and it's fine at three and impossible at three thousand.

flowchart TB
  subgraph "this application"
    W1["worker"] --> DB[("shared database")]
    W2["dispatcher"] --> DB
    C1["cable listener"] --> DB
    P["proxy, during a deploy"] --> APP["/up"]
  end
  DB --> N["one authority. everyone asks it. nobody tells anyone"]

When there is no authority

Gossip starts from this premise: there's no shared store every node can reach, or reaching it costs more than the coordination is worth.

That happens for two reasons. One is scale: a thousand nodes polling one coordinator make that coordinator the bottleneck and the single point of failure. The other is geography: nodes on three continents can't all reach one store cheaply, as the senior course "Architecture: multi-region database" measured at about 200ms per round trip between São Paulo and Frankfurt.

So the nodes talk to each other instead. Each node regularly picks a few peers at random and exchanges what it knows: who is alive, who joined, who left, and which version of the membership list it holds.

Nobody has the complete picture at any given moment, and everybody converges on it quickly.

The trade being made

Read the last row. Gossip isn't better. It's what you use when the row above has become unacceptable.

That row only becomes unacceptable at a scale this application is nowhere near. That's the fair framing for this course: you're learning the shape of a solution to a problem you should be able to recognize, and should not invent.

The one thing to take even at this size

Polling intervals are a coordination budget, and people usually choose them without one.

polling_interval: 0.1 on Solid Cable means each server process's listener issues ten queries per second, forever, whether anything is happening or not. With WEB_CONCURRENCY: 2 that's twenty queries per second: a known, bounded load. The number that matters is queries per second per participant times the number of participants. It's worth computing once, because the scale at which it stops being fine arrives without any code changing.

Key takeaways

• This application coordinates by polling a shared authority: the dispatcher every 1s, workers and the cable listeners every 0.1s, and the deploy health check every 5s.
• At this size, polling a shared store is the right design, and it costs one query per interval per participant.
• Gossip's premise is that no reachable shared authority exists, because of scale or geography.
• In gossip, nodes exchange what they know with a few random peers. Nobody has the whole picture, and everybody converges.
• Polling intervals are a coordination budget. Compute queries per second times participants once, because scale arrives without code changing.