Repository navigation
Conversation
Benchmarks passed: 1 new, 0 unchanged
Add the |
| vendor/bin/tally run benchmarks --output="$RUNNER_TEMP/tally-results" || [ $? -eq 1 ] | ||
| vendor/bin/tally accept "$RUNNER_TEMP/tally-results" benchmarks | ||
| echo "patch=$(git diff -- benchmarks | base64 --wrap=0)" >> "$GITHUB_OUTPUT" |
There was a problem hiding this comment.
Accept patch omits new expectation files because git diff skips untracked paths
git diff -- benchmarks only includes tracked files, so new expectation files written by tally accept never enter the patch and the commit job no-ops while still removing the label. Stage benchmarks first and export git diff --cached.
Evidence
- Measure runs
tally accept … benchmarksthen setspatchfromgit diff -- benchmarkswith nogit add. git diffdoes not include untracked files; onlygit add+git diff --cached(or equivalent) would.- This PR’s
benchmarks/tree has bench sources only—no committed expected-number fixtures—and the bench method has no expected-count attributes, so the first accept must create new files beside the benchmarks. - Commit runs only when
needs.measure.outputs.patch != '', so an empty patch skips the commit after the label is already removed.
Identified by Warden · code-review · TP4-QH7
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit be62a30. Configure here.
| PATCH: ${{ needs.measure.outputs.patch }} | ||
| HEAD_REF: ${{ github.event.pull_request.head.ref }} | ||
| run: | | ||
| echo "$PATCH" | base64 --decode | git apply --include='benchmarks/*' |
There was a problem hiding this comment.
Accept patch targets the wrong commit
Medium Severity
The measure job checks out the pull request merge commit and builds a git diff against that tree, but the commit job applies the patch onto github.event.pull_request.head.sha. When files under benchmarks/ differ between those commits, git apply fails after the accept-benchmarks label is already removed.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit be62a30. Configure here.
There was a problem hiding this comment.
Nice experiment. The two-job split in benchmarks-accept.yml keeps the write token away from PR code. Two things matter for whether the numbers are useful and whether the token isolation holds, plus some context on the bot findings:
1. The benchmark install has no lock, so expected numbers can drift without any SDK change. The repo has no composer.lock, so composer require phpstan/tally:1.0.1 --update-no-dev resolves the newest guzzlehttp/psr7, symfony/options-resolver, psr/log, jean85/pretty-package-versions, and Tally's own transitive deps on every run. Those packages run inside init() and the transport, and as the workflow comment says, every installed package ends up in the event's modules list. So any upstream patch release can move instructions and memory and flag the benchmark as slower or faster. Tally's README calls the environment part of the expectation ("Expected numbers measured in one environment only fit that environment"). If the numbers should be comparable over time, a committed lock for the benchmark install would remove most of this noise, for example a small Composer project under benchmarks/ with its own composer.lock that path-requires the SDK. Disabling ModulesIntegration in the bench setup would also take the package count out of the payload. php-version: '8.5' also follows PHP patch releases, but that one is harder to pin and the README already expects it.
2. In benchmarks.yml, the pull-requests: write token is not isolated from the steps that run PR and dependency code. The comment step is the only one with GH_TOKEN, but all steps share one job. An earlier step that runs untrusted code (tally run, pr-comment.php, any dependency) can append to $GITHUB_PATH or $GITHUB_ENV, for example to put its own gh first on PATH, and the comment step then runs it with the token (workflow commands docs). Fork PRs are excluded, so this mostly matters as supply-chain exposure through the floating deps from point 1. Using the same pattern as benchmarks-accept.yml would close it: give the measuring job contents: read only, and post the table from a second job that has pull-requests: write and only reads it (as a job output or artifact).
Context on the existing bot comments:
- The concern that
git diff -- benchmarksmisses untracked files does not apply here.tally acceptrewrites or adds the#[Expected]attribute and its import inside the tracked*Bench.phpfile (README,tally accept), so the diff is not empty. - The merge-commit vs. head-SHA point is real but narrow. The patch only touches
benchmarks/*Bench.php, sogit applyonly fails when the base branch changed the same bench file. In that case the label is removed and the job fails visibly. Measuring the merge commit also matches whatbenchmarks.ymlreports.


Adds Tally as a CI job and runs a small scenario where
captureExceptionis called.This is mostly an experiment to see how the results look like in CI and see if this is useful