Coverage says the line ran

This repository's test suite reports 82.99% line coverage over 1,206 examples. That is a respectable number, and this course is about what even a perfect one would not tell you.

$ bundle exec rspec
1206 examples, 1 failure
Line Coverage: 82.99% (4190 / 5049)

Here is what that number does and does not mean.

What line coverage measures

SimpleCov instruments the interpreter and records which lines executed during the run. 4190 lines out of 5049 ran at least once. That is the whole claim. (The one failure is a time-zone spec that depends on the local machine's tzdata.)

It says a line ran. It says nothing about whether anything checked the result. A spec that calls a method and asserts nothing produces the same coverage as a spec that asserts everything about it. And coverage cannot tell them apart because the distinction is invisible to the instrument.

flowchart TB
  A["a spec runs a line"] --> B["line coverage: covered"]
  A --> C{"did anything assert on the result?"}
  C -->|"yes"| D["the line is tested"]
  C -->|"no"| E["the line is executed"]
  B --> F["coverage cannot distinguish D from E"]

The experiment

Mutation testing answers the question coverage cannot. And it answers it the only way it can be answered: break the code on purpose, and see whether the suite notices.

Change one thing in the application. Run the suite. If a test fails, the mutation was killed: something was checking that behavior. If the suite still passes, the mutation survived, and you have found a line that runs without being tested.

Here is one, run against this repository. app/services/auth/password_policy.rb declares:

MIN_LENGTH = 8

Change it to 7 and run all 1,206 examples:

SURVIVED | MIN_LENGTH = 8 -> MIN_LENGTH = 7 | 1206 examples, 1 failure (the tzdata spec)

The suite stays green with the minimum password length weakened.

That line is covered. It is referenced by Identity::User's length validation, by three views, a Stimulus controller, and by a spec file named spec/services/auth/password_policy_spec.rb. Every one of those executes it. None of them checks it.

Why this is the honest measurement

A test suite exists to fail when the code is wrong. That is its only job. So the direct way to evaluate a suite is to make the code wrong and see whether it fails, which is what mutation testing does. And why its result is called mutation coverage rather than an estimate.

Line coverage is not worthless: a line at 0% is definitely untested, and finding those is cheap. It is a lower bound on ignorance, not a measure of quality. Mutation coverage is the measure of quality, and it costs far more to compute, which is the trade the rest of this course is about.

What to expect

The next lesson is the catalogue of mutations. Lesson 3 is the full experiment on this repository: eleven mutations, six killed, five survived, with the results. Lesson 4 is the part every honest account includes: mutations that survive and should, and why triage is the real cost.

Key takeaways

• This suite has 82.99% line coverage over 1,206 examples.
• Line coverage records that a line executed; it cannot see whether anything asserted on the result.
• Mutation testing breaks the code deliberately and asks whether the suite fails: killed, or survived.
• Measured here: MIN_LENGTH = 8 changed to 7 leaves all 1,206 examples green.
• Line coverage is a lower bound on ignorance; mutation coverage is the measure of whether behavior is verified.