Why one number is not a measurement

Here's a benchmark. It upcases twenty thousand strings and times them with the clock everyone reaches for first.

require "benchmark"
WORDS = (1..20_000).map { |i| "row-#{i}" }
def build(words) = words.map(&:upcase)

5.times do |i|
  t = Time.now
  build(WORDS)
  puts "run #{i + 1}: #{((Time.now - t) * 1000).round(2)} ms"
end

Run it three times in a row, on the same machine in the same minute, with no other work started or stopped. Here's one machine on Ruby 4.0.1:

invocation 1:  4.24  3.93  3.35  3.48  2.91 ms
invocation 2:  2.19  2.12  1.87  2.03  1.87 ms
invocation 3:  1.93  1.93  1.71  1.86  1.80 ms

The slowest number is nearly 2.5 times the fastest, and the code never changed. Say you ran this once and reported 4.24 ms, and a colleague ran it once and reported 1.71 ms. You'd both be telling the truth, and you'd disagree by a factor of two.

What the clock measured

A single Time.now delta measures your machine's mood, and the mood has named parts.

Warm-up comes first. The first pass through build pays for method cache misses and inline cache setup, plus compilation if YJIT is on. Notice the shape inside every invocation above: run 1 is the slowest, and the numbers settle downward. That's a trend, not noise, and it repeats.

Lazy loading is next. The first call into a code path pulls in autoloaded constants, opens connections and allocates the buffers the path will reuse from then on. You paid a one-time cost and charged it to the algorithm.

Then there's the garbage collector. Ruby runs GC when allocation pressure says so, not when your measurement is over. If a collection fires inside your window, you timed a collection. If it fires just outside, you timed a lie in the other direction.

Last is the rest of the machine: CPU frequency scaling, thermal limits, another process waking up and the OS moving your threads to a different core. None of these ask your permission.

That's the spread across the three invocations above. It's why the difference between invocations is bigger than the difference between the first and last run inside one of them.

The claim you're making

When you say "this version is faster", you're making a claim somebody will test in review. There are three ways to lose that argument, and all three are avoidable.

You measured a warm-up cost and called it an algorithm. You measured once and got a number from the tail of a distribution. You compared two implementations whose real difference is smaller than the noise on your machine.

Each of those has a fix, and none of the fixes is complicated. They're steps the one-line Time.now version skips.

The previous course, and this one

The senior course "Performance: O-notation" closed by naming what it couldn't tell you. It told you which curve you're on: the shape your cost follows as the input grows. It never told you where on that curve you're sitting right now, because a curve is a prediction and a prediction isn't a measurement.

This is the measurement half. Complexity told you what to suspect. Benchmarking tells you whether you were right, and more often than a junior expects, the answer is that you weren't.

What this course covers

Lesson 2 is warm-up and steady state. It explains why Benchmark.bmbm runs your code twice and throws the first result away, and why that rehearsal fixes a specific problem rather than being a ritual.

Lesson 3 is variance. One number isn't a result. You report a spread, and two implementations whose spreads overlap aren't distinguishable. Saying one is faster is a claim you'll be held to.

Lesson 4 is iterations per second against elapsed time, and why the first is the better default for comparing two candidates. It also settles what's installed here, because that answer changed on Ruby 4.0.1.

Lesson 5 is Rails, where most benchmarks go wrong. The usual culprits are a cold database, the query cache, code reloading, logging and an object that was already memoized on the line above.

Lesson 6 is the question to ask before measuring anything: how much of the total can this candidate possibly touch? A 40% win on 0.3% of the request isn't a win.

Lesson 7 turns a measurement into a team standard. It covers a performance budget, where the threshold lives and the honest failure mode of performance tests on a noisy CI runner.

Start with the shape you already saw: run 1 is always the slowest. Lesson 2 is about paying that cost on purpose, before the clock starts.