When one server stops being enough

Load balancing is the answer to a question you should be able to state as a number, and stating it is more than half the work.

The two reasons to add a server

Capacity: requests arrive faster than one machine can serve them.

Availability: one machine's failure takes the site down, and that's no longer acceptable.

These are different problems with the same solution, and confusing them leads to adding a second server that helps with neither.

What capacity looks like as a number

Take a worked example: Puma with two workers and three threads each has six request slots.

A warm signed-in roadmap response in this application takes about 72 milliseconds. Six slots at 72 milliseconds each is roughly 83 requests per second, if every request were that page and nothing else competed.

The anonymous roadmap takes about 20 milliseconds, so the same six slots serve about 300 per second of that.

These are ceilings from arithmetic, not measurements, and the distinction matters. A real machine has garbage collection pauses, database contention and jobs running in the same process.

flowchart TB
  S["6 request slots, as a worked example"] --> A["at 72ms each: about 83 per second"]
  S --> B["at 20ms each: about 300 per second"]
  A --> C["arithmetic ceiling, not a measurement"]
  B --> C
  C --> D["the real number is lower and unmeasured here"]

The order of cheaper answers

Before a second server, make each request cheaper. The roadmap page in this project went from about 310 milliseconds and 311 queries to about 72 milliseconds and 46 queries. That change roughly quadrupled the same machine's capacity for that page, and it cost no hardware.

Then comes a bigger machine. More cores mean more workers, and it's still one machine to operate.

Then comes caching. A cached response that never reaches Ruby doesn't consume a slot at all.

Only then comes a second server, which is the first option that adds an operational commitment instead of a configuration change.

Availability is a different threshold

No amount of optimization makes one machine survive its own failure.

If the requirement is "the site stays up when a machine dies", the second server is required regardless of load, and it has to be a second server, not a bigger one.

Being clear about which of the two problems you have determines the whole design, because a capacity answer and an availability answer look identical in a diagram and behave completely differently.

Key takeaways

• Capacity and availability are different problems that both point at a second server.
• In the worked example, six slots at 72 milliseconds are roughly 83 requests per second, by arithmetic.
• Cheaper requests, a bigger machine and caching all come before a second server.
• The roadmap optimization roughly quadrupled that page's capacity with no hardware.
• No optimization makes one machine survive its own failure.