A plateau is not a slow query

There are two very different performance problems. They look the same from outside, and most wasted optimization comes from treating one as the other.

The two shapes

A slow endpoint is slow at one request per second and just as slow at fifty. The work itself takes too long: a missing index, a query in a loop, an image resized on the request path. Latency stays flat as load rises, and the fix is local: make that work smaller.

A plateau is fast until it isn't. Response times hold steady, then at some level of concurrency they climb and keep climbing, and nothing in a profile of a single request looks any different from before.

That's the signature: the individual request didn't get slower. It started waiting.

flowchart TB
  A["response time is bad"] --> Q{"is it bad at low load too?"}
  Q -->|yes| S["a slow path. profile one request"]
  Q -->|no| P["a plateau. count the servers, not the milliseconds"]
  S --> S1["fix: make the work smaller"]
  P --> P1["fix: find what is finite and being exhausted"]
  P1 --> P2["profiling one request will not show it"]

Why profiling fails on a plateau

A profiler measures where a request spends its time. It can't show you the request that wasn't allowed to start.

When a system plateaus, the request is waiting for a resource another request holds: a thread, a connection, a lock. The wait usually shows up as time inside something innocent, or in the gap between the request reaching the server and any application code running. Most tools don't instrument that gap by default.

So the tool depends on the shape of the problem. For a slow path, profile. For a plateau, count things: how many can run at once, and how many are trying.

What is finite here

A deployment states its limits in its configuration, and they're small, specific numbers. Take a deployment that sets:

WEB_CONCURRENCY: 2
RAILS_MAX_THREADS: 3

Puma reads WEB_CONCURRENCY as its number of worker processes, and this application's puma.rb sets the threads per process from the other value:

threads_count = ENV.fetch("RAILS_MAX_THREADS", 3)
threads threads_count, threads_count

Two processes with three threads each means six requests can be in flight at once. The seventh request doesn't run slowly. It doesn't run at all until one of the six finishes.

That number is the first thing to find out about any system you're asked to make faster. It's usually written down somewhere obvious, in a file nobody thinks of as a performance document.

The arithmetic that follows

Six concurrent slots with an average request of 100 milliseconds gives sixty requests per second. That isn't an estimate. It's what the configuration permits, before a single line of application code comes into it.

If requests average 500 milliseconds, the same six slots give twelve requests per second. The ceiling moved by a factor of five and nothing about the hardware changed.

The relationship worth learning is that throughput is concurrency divided by latency. It means a slow path and a plateau aren't independent after all: making the average request faster raises the ceiling, which is why the two problems are so easy to confuse.

They're still fixed differently, though. Halving the duration of one rare endpoint does nothing for the ceiling. Halving the average does.

Key takeaways

• A slow path is slow at any load. A plateau is fast until concurrency reaches a limit, then degrades.
• On a plateau the request didn't get slower. It started waiting, and profilers are poor at showing that.
• For a slow path, profile. For a plateau, count what can run at once and what's trying to.
• WEB_CONCURRENCY: 2 times RAILS_MAX_THREADS: 3 permits six concurrent requests.
• Throughput is concurrency divided by latency, so lowering the average latency raises the ceiling.