[MaxFreedomPollard]

Description

The nightly selective baseline merge never copies anything, so the benchmark baseline only survives every other nightly.

check_benchmark_regression.py lists benchmarks in its report by their benchmark_file field, which run_all_json.py sets to the script path relative to tests/benchmark (bench_softmax.py, experimental/bench_mhc.py). merge_baseline_selective.py uses those entries as file names inside the baseline and results directories, but run_all_json.py saves the results as bench_softmax_results.json and experimental_bench_mhc_results.json. No source file is ever found, so merged-baseline/ stays empty while the script exits 0 with "Updated baselines: 0 files", and "Update baseline (nightly only - selective or full)" has nothing to upload. The next nightly then finds no benchmark-baseline artifact on the latest successful run, skips the regression check, and uploads its own results as a first baseline. The names have never matched: when the merge was added in #15, benchmark_file was the bare script name and the results file was <stem>_results.json.

The Actions history shows the pattern. Of the last 60 tilegym-ci runs on main (Aug 25 to Oct 22), none of the 31 runs where "Merge baseline selectively (nightly only)" ran has a benchmark-baseline artifact, and all 29 runs where it was skipped have one. For example, run 39420331706 (Oct 20) skipped the merge and uploaded a baseline, then runs 39597578834 and 39776384161 (Oct 21 and 22) ran the merge and uploaded none. As a result, main is only checked for regressions every other night, PR builds have no baseline about half the time, and a regressed benchmark never keeps its old baseline.

The fix maps each report entry to its results file name the way run_all_json.py derives it (/ becomes _, .py becomes _results.json). The rule is repeated rather than imported because the aggregate job checks out only .github/scripts. The report format and the checker are unchanged.

Testing (CPU only):

• Added test_merge_keeps_old_baseline_only_for_regressed_files to .github/infra_tests/test_selective_baseline_merge.py. It lays out results the way run_all_json.py does (including a benchmark under experimental/), runs the checker and the merge, and checks that the regressed benchmark keeps the old file, the improved one takes the new file, and all_benchmarks.json is rebuilt from them. It fails on main with FileNotFoundError because the merged directory is empty, and passes with this change. pytest .github/infra_tests/ passes (63 tests).
• End to end: I ran run_all_json.py on two dummy benchmark scripts (bench_alpha.py, experimental/bench_beta.py) for a baseline and a second run with alpha 20% slower and beta 20% faster, then ran the two scripts with the same arguments as the workflow. Before, the merge wrote nothing. After, bench_alpha_results.json came from the old baseline, experimental_bench_beta_results.json from the new run, and a check against the merged directory still flags alpha.
• ruff check --select I ., ruff format --check ., the SPDX header check and the whitespace and end-of-file hooks pass on the changed files.

The merge step runs only on nightly builds, so PR CI can't exercise it end to end.

CI Configuration

config:
  build: true
  # valid options are "ops", "benchmark", and "sanity"
  test: []

Checklist

• Code formatted and imports sorted via repo specifications (./format.sh)
• Documentation updated (if needed)
• CI configuration reviewed

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.