The number and the lines behind it

This application reports 82.99% line coverage. That number is almost useless and the thing behind it is not, and the gap between the two is what this course is about.

The number

$ bundle exec rspec
1206 examples, 1 failure
Line Coverage: 82.99% (4190 / 5049)

(The one failure is a time-zone spec that depends on the local machine's tzdata.) What can you do with 82.99%? You can put it in a slide. You can watch it go up or down. You cannot act on it, because it does not name anything, and it does not even say what it measured.

The lines behind it

SimpleCov writes coverage/.resultset.json, which contains the per-line data the percentage was computed from. Reading it out gives something entirely different:

uncovered lines: 859, in 40 files
  727 in lib/tasks/ and lib/course_revision/ (this repository's course-authoring tooling)
    6 in lib/execution/ (the grading sandbox)
  126 in app/, across 23 files, for example:
    app/services/partners/webhook_transport.rb: 10, 12, 13, 14, 15, 17, 18, 20, 21, 23
    app/services/partners/fetch_site_icon_service.rb: 37, 59
    app/jobs/partners/fetch_site_icon_job.rb: 6, 7
    app/jobs/partners/deliver_webhook_job.rb: 19, 23
    app/controllers/admin/partners/partners_controller.rb: 29, 38, 39, 40

Two findings before reading a single line of code. Most of the gap is not the application at all: it is rake tasks that the suite loads and never runs. And the application's own gap is 126 lines in 23 files, each with a number. This is the same measurement, and it is actionable in a way the percentage is not, because every line can be looked at and given a decision.

(The rest of this course works from the 126 application lines and leaves the tooling out of scope.)

What the lines turn out to be

Three of them are worth looking at directly.

# app/services/partners/fetch_site_icon_service.rb, line 37
rescue URI::InvalidURIError, SocketError, Timeout::Error,
       OpenSSL::SSL::SSLError, Errno::ECONNREFUSED => e
  Result.new(success: false, url: nil, errors: [ e.message ])   # ← line 37

# app/jobs/partners/fetch_site_icon_job.rb
def perform(partner_id)
  partner = ::Partners::Partner.find_by(id: partner_id)                          # ← line 6
  ::Partners::FetchSiteIconService.call(partner: partner) if partner && ...      # ← line 7
end

# app/services/partners/webhook_transport.rb, lines 10 to 23
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
# ...
response = http.request(request)
return response if response.code.to_i.between?(200, 299)
raise DeliveryError, "#{url} responded #{response.code}"

The rescue clause for five exception classes, the entire body of the icon job, and the entire body of the adapter that delivers partner webhooks.

The first two are the code that runs when a partner's website is down: the exact situation the service was written for. The third is every line that talks to the network when the application notifies a partner, including the branch that raises DeliveryError so the job retries; the job's spec replaces it with a stub. So none of it has ever executed in a test.

flowchart TB
  A["82.99% line coverage"] --> B["a number you can put in a slide"]
  A --> C[".resultset.json: 126 application lines in 23 files"]
  C --> D["the rescue for 5 exception classes"]
  C --> E["the whole body of the icon job"]
  C --> H["the whole webhook delivery adapter"]
  D --> F["the code that runs when a partner's site is down"]
  E --> F
  H --> I["the code that runs every time a partner is notified"]

The point

Nobody needed 82.99% to find this. A grep for rescue and Net::HTTP and a question about which of them have specs would have found the same thing faster. And reading the application lines takes a few minutes.

The percentage did not surface it, hid the shape of it, and mixed it with seven hundred lines of tooling that nobody would ever test.

That is the pattern this course is about: a metric is a pointer, and the value is always in what it points at. A number reported without the thing behind it is a number that ended a conversation rather than starting one.

The rule for every metric in this course

If it cannot be turned back into a list of specific things, it is not worth tracking.

82.99% fails that test. "126 uncovered application lines, and three groups of them are the failure path of one outbound HTTP call and the whole of the other" passes it. And it is the same measurement: read differently.

Key takeaways

• This application reports 82.99% line coverage over 1,206 examples.
• coverage/.resultset.json turns that into 859 uncovered lines, 727 of them in course-authoring tooling and 126 in 23 application files.
• Three groups are worth a direct look: the five-class rescue clause, the whole body of the icon job, and the whole webhook delivery adapter.
• Those are the lines that run when a partner's website is down, or every time a partner is notified.
• A metric that cannot be turned back into a list of specific things is not worth tracking.