Repository navigation
Coverage is measured differently locally and on CI, hiding 68 untested app files #911
Description
The backend coverage figure means different things locally and on CI, and the local one is the misleading one. Reproduced exactly, not inferred.
$ bundle exec rspec 3772 / 3948 LOC (95.54%)
$ CI=true bundle exec rspec 4193 / 4856 LOC (86.35%) # identical to CI's report
Cause
Two settings that are each fine alone:
config/environments/test.rb—config.eager_load = ENV["CI"].present?. This is the stock Rails default, verbatim from therailtiesapp template; nobody misconfigured it.spec/spec_helper.rb— a bareSimpleCov.start, with no profile and therefore notrack_files.
Without track_files, SimpleCov measures only files that were actually loaded. GitHub Actions always sets CI=true, so Rails eager-loads all of app/; locally it does not, so only the files the specs happen to touch are loaded, and only those are measured.
What that hides
The delta accounts for itself exactly — 4856 − 3948 and 4193 − 3772:
70 files, 908 relevant lines, 421 covered. 68 of the 70 are app/ files, so 38% of the application is never loaded by the suite at all.
Those files are not reported as 0% locally. They are absent from the report. The local denominator is defined by what the tests already reach, which makes 95.54% close to circular.
The CI number is not trustworthy either, for a different reason: spec/ files are 1952 of its 4856 lines — 40% — at 98.51% covered, because the test_frameworks filter only ships with SimpleCov's rails profile, which is not loaded. The tests are grading themselves.
Honest figure for application code: 77.09%.
The 46% those 70 files appear to have on CI is pure load-time execution. Line by line on app/jobs/same_trackables_job.rb (18.2% on CI, invisible locally), the only covered lines are class, include Sidekiq::Worker, and two defs. Every method body is 0. The file is completely untested.
The obvious fix does not work
Adding track_files "{app,lib}/**/*.rb" makes it worse — the two environments then disagree on the denominator as well, because SimpleCov's static LinesClassifier counts relevant lines differently than runtime Coverage:
track_files + CI=true -> 4193 / 4856 (86.35%) unchanged
track_files + local -> 3772 / 5367 (70.28%)
I measured the filter options too. Every one leaves local ≠ CI, and filtering spec/ widens the gap (9.2pp → 15.2pp), since spec/ is the one bucket both environments agree on:
| Config | eager (CI) | lazy (local) |
|---|---|---|
| current | 86.35% | 95.54% |
filter spec/ |
78.17% | 92.64% |
rails profile equivalent |
77.09% | 92.31% |
eager_load is the whole story. No SimpleCov setting reconciles it.
The part that actually bites
Eager loading is in CI specifically to catch autoload and NameError breakage. Gating it on CI means that entire class of bug cannot fail locally — only on CI, after a push.
Proposed fix
config.eager_load = true in test, unconditionally. Measured cost: ~0.5s on boot, 2.7s → 3.2s.
Then backfill the coverage it exposes, which is the larger half of the work.
Separately worth deciding, but deliberately not bundled in: SimpleCov.start "rails" would stop the tests grading themselves and report ~77%. That is a policy change that visibly drops the headline number and should be a conscious call, not smuggled in behind a defect fix.
🤖 Generated with Claude Code
Activity
suttondemlong commentedon Sep 9, 2026
Update after simplecov 0.22.0 → 1.2.0 landed on
master(via #898 and #910). Re-measured; the defect is unchanged but two things in the issue above are now out of date.Still true, and now worse
Same cause, same reproduction. The gap has widened from 9.2pp to 14.5pp, which is exactly what the options table in the issue predicted would happen once
spec/was filtered —spec/was the one bucket both environments agreed on, so removing it leaves the eager-load divergence to dominate.No longer true: "the tests are grading themselves"
simplecov 1.2.0 loads the
test_frameworksprofile by default; 0.22.0 did not.spec/is now filtered automatically, so the criticism that spec files were 40% of the measured lines at 98.51% covered has been fixed by the dependency upgrade rather than by anyone deciding anything.That also settles the open question at the bottom of this issue. Adopting
SimpleCov.start "rails"was proposed to stop the tests grading themselves; simplecov now does that on its own, so there is nothing left to decide there. The remaining difference between the default profile andrailsis only thatrailsalso filtersconfig/anddb/and addstrack_files— andtrack_filesis still the wrong move here, for the reason already in the issue.Also stale: the headline figures
The "honest figure for application code: 77.09%" line was measured under 0.22.0. Under 1.2.0 it is 78.17%. The difference is not a real improvement — 0.22.0's
rails-profile-equivalent also filteredconfig/, which 1.2.0's default does not.Effect on #912
None on the fix; it is the same one line. The numbers in that PR are updated. For the record, on the branch local and CI now produce identical output —
— which is the actual point of the change, and is easier to see now that
spec/is out of the way.🤖 Generated with Claude Code