Repository navigation
Conversation
On push to main, and on a PR whose head commit message has "with-coverage", the Linux cells of `task test` record unit coverage (-coverpkg=./...) and per-test acceptance coverage, and upload it as coverage-linux-<cell>: one zstd --long file with a section per acceptance test (tools/coverpack.py). with-coverage Co-authored-by: Isaac <no-reply@databricks.com>
with-coverage Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: 11081c1
Top 13 slowest tests (at least 2 minutes):
|
Binary format instead of text: 3.9k tests have 3M (test, file) pairs but only 13k distinct bitmaps. Same content, checked with --dump. with-coverage Co-authored-by: Isaac <no-reply@databricks.com>
tools/coverpack (Go) replaces tools/coverpack.py: it decodes covmeta and covcounters files itself instead of running `go tool covdata textfmt` per test, so each distinct meta-data file is parsed once and a test costs one pass over its counter files. Same output format; 0.27s instead of 10s on 358 local tests. A test compares it with covdata on a small program. with-coverage Co-authored-by: Isaac <no-reply@databricks.com>
For a PR, CI tests GitHub's merge of it into main, not its head, so the block positions belong to that commit. with-coverage Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Collect coverage in the existing Linux cells of
task testand upload it as a small artifact,coverage-linux-<cell>. It is for the CI dashboard, which will show coverage by package and file (unit, acceptance and combined) and which acceptance tests cover each line.with-coverage(for testing changes to this). Windows, macOS and the merge queue do not collect it.CLI_GOCOVERDIR_UNITis set,test-unit-rootruns with-cover -coverpkg=./...into that coverage directory. It is a directory rather than-coverprofilebecause--rerun-failsrunsgo testagain with the same flags, which would overwrite the profile with the coverage of the rerun alone.namefile witht.Name(). The directory name ends in a variant index that does not say which variant ran.tools/coverpack(new, Go) packs both into one zstd file. It reads Go's binarycovmeta/covcountersfiles directly instead of runninggo tool covdataper test: each distinct meta-data file is parsed once, and a test costs one pass over its counter files. The format is documented at the top offormat.go. In short: the block layout of each file once, then the distinct per-file bitmaps (one bit per Go block) once, then per test, which bitmap it has for each file it covered.go run ./coverpack -dump FILEprints it back as Go cover profile text.Why
Each per-test coverage directory holds its own ~900 KB copy of the CLI's coverage meta-data, so the raw directories come to several GB per cell. Most tests cover a file the same way as many other tests (mostly the CLI's startup): 2,259 tests have about 3M (test, file) pairs but only 12k distinct bitmaps.
directcellThe
directdmscell gives 307 KB. The overhead is running the tests with-cover; packing is negligible.Tests
tools/coverpack/main_test.gobuilds a small program with-cover, runs it as two "tests" and one unit run, and checks that the packed and dumped profiles equalgo tool covdata textfmt, including a line with two blocks and a function that never runs. A second test checks that every truncation of a file is rejected.covdata textfmtoutput in all 360 sections.with-coverage) exercise the CI path end to end; see thecoverage-linux-*artifacts of its latest run.This pull request and its description were written by Isaac.