Skip to content

Collect per-test coverage in the Linux test cells for the CI dashboard - #7694

Draft
denik wants to merge 5 commits into
mainfrom
denik/ci-coverage
Draft

denik wants to merge 5 commits into
mainfrom
denik/ci-coverage

Conversation

@denik

@denik denik commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Collect coverage in the existing Linux cells of task test and upload it as a small artifact, coverage-linux-<cell>. It is for the CI dashboard, which will show coverage by package and file (unit, acceptance and combined) and which acceptance tests cover each line.

  • When: on every push to main, and on a PR whose head commit message contains with-coverage (for testing changes to this). Windows, macOS and the merge queue do not collect it.
  • Unit tests: when CLI_GOCOVERDIR_UNIT is set, test-unit-root runs with -cover -coverpkg=./... into that coverage directory. It is a directory rather than -coverprofile because --rerun-fails runs go test again with the same flags, which would overwrite the profile with the coverage of the rerun alone.
  • Acceptance tests: each per-test coverage directory gets a name file with t.Name(). The directory name ends in a variant index that does not say which variant ran.
  • tools/coverpack (new, Go) packs both into one zstd file. It reads Go's binary covmeta/covcounters files directly instead of running go tool covdata per test: each distinct meta-data file is parsed once, and a test costs one pass over its counter files. The format is documented at the top of format.go. In short: the block layout of each file once, then the distinct per-file bitmaps (one bit per Go block) once, then per test, which bitmap it has for each file it covered. go run ./coverpack -dump FILE prints it back as Go cover profile text.
  • The metadata records the commit that was checked out. For a PR that is GitHub's merge commit, which is what the block positions belong to.

Why

Each per-test coverage directory holds its own ~900 KB copy of the CLI's coverage meta-data, so the raw directories come to several GB per cell. Most tests cover a file the same way as many other tests (mostly the CLI's startup): 2,259 tests have about 3M (test, file) pairs but only 12k distinct bitmaps.

direct cell without coverage (mean of 5 main runs) this PR
Run tests 6m19s 8m56s
Pack coverage – 1s (0.7s for 10,846 counter files)
Whole job 7m20s 10m24s
Artifact – 339 KB (2.9 MB uncompressed)

The directdms cell gives 307 KB. The overhead is running the tests with -cover; packing is negligible.

Tests

  • tools/coverpack/main_test.go builds a small program with -cover, runs it as two "tests" and one unit run, and checks that the packed and dumped profiles equal go tool covdata textfmt, including a line with two blocks and a function that never runs. A second test checks that every truncation of a file is rejected.
  • On 358 local acceptance tests, the dump matches the per-test covdata textfmt output in all 360 sections.
  • This PR's own runs (its commit messages contain with-coverage) exercise the CI path end to end; see the coverage-linux-* artifacts of its latest run.

This pull request and its description were written by Isaac.

denik and others added 2 commits November 16, 2026 16:26
On push to main, and on a PR whose head commit message has "with-coverage",
the Linux cells of `task test` record unit coverage (-coverpkg=./...) and
per-test acceptance coverage, and upload it as coverage-linux-<cell>: one
zstd --long file with a section per acceptance test (tools/coverpack.py).

with-coverage

Co-authored-by: Isaac <no-reply@databricks.com>
with-coverage

Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 11081c1

Run: 44123123247

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 15 10:50
💚​ aws-windows-latest-4core-16gb 1 272 13 8:29
💚​ azure linux-2core-8gb 1 269 15 9:51
💚​ azure-windows-latest-4core-16gb 1 271 13 8:11
💚​ gcp linux-2core-8gb 1 270 15 12:32
💚​ gcp-windows-latest-4core-16gb 1 272 13 6:31
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 13 slowest tests (at least 2 minutes):
duration env testname
3:26 azure-windows-latest-4core-16gb TestAccept
3:24 gcp-windows-latest-4core-16gb TestAccept
3:09 aws-windows-latest-4core-16gb TestAccept
3:01 aws-windows-latest-4core-16gb TestExportDir
2:53 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
2:42 azure linux-2core-8gb TestFilerWorkspaceNotebook/pyNb.py
2:34 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete
2:28 aws-windows-latest-4core-16gb TestImportDirDoesNotOverwrite
2:15 aws linux-2core-8gb TestFilerReadWrite/workspace_files_extensions
2:14 azure-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions
2:06 azure linux-2core-8gb TestImportDirWithOverwriteFlag
2:03 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
2:00 azure linux-2core-8gb TestAccept

denik and others added 2 commits November 17, 2026 12:24
Binary format instead of text: 3.9k tests have 3M (test, file) pairs but
only 13k distinct bitmaps. Same content, checked with --dump.

with-coverage

Co-authored-by: Isaac <no-reply@databricks.com>
tools/coverpack (Go) replaces tools/coverpack.py: it decodes covmeta and
covcounters files itself instead of running `go tool covdata textfmt` per
test, so each distinct meta-data file is parsed once and a test costs one
pass over its counter files. Same output format; 0.27s instead of 10s on
358 local tests. A test compares it with covdata on a small program.

with-coverage

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the Dependencies Pull requests that update a dependency file label Nov 17, 2026
For a PR, CI tests GitHub's merge of it into main, not its head, so the
block positions belong to that commit.

with-coverage

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik changed the title Collect coverage in the Linux test cells for the CI dashboard Collect per-test coverage in the Linux test cells for the CI dashboard Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Dependencies Pull requests that update a dependency file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants