Skip to content

feat(ci): add tally benchmarks - #2391

Open
Litarnus wants to merge 5 commits into
masterfrom
ci/tally-benchmarks
Open

Litarnus wants to merge 5 commits into
masterfrom
ci/tally-benchmarks

Conversation

@Litarnus

Copy link
Copy Markdown
Collaborator

Adds Tally as a CI job and runs a small scenario where captureException is called.

This is mostly an experiment to see how the results look like in CI and see if this is useful

@Litarnus
Litarnus requested review from a team and stayallive as code owners November 17, 2026 23:31
@getsentry getsentry deleted a comment from github-actions Bot Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Benchmarks passed: 1 new, 0 unchanged

Benchmark Instructions Instructions expected Cycles Cycles expected Memory Memory expected
captureException with 20 recursive frames 2.052M - 2.467M - 103.1KiB -

Add the accept-benchmarks label to commit the measured numbers as the expected ones.

Comment on lines +44 to +46
vendor/bin/tally run benchmarks --output="$RUNNER_TEMP/tally-results" || [ $? -eq 1 ]
vendor/bin/tally accept "$RUNNER_TEMP/tally-results" benchmarks
echo "patch=$(git diff -- benchmarks | base64 --wrap=0)" >> "$GITHUB_OUTPUT"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accept patch omits new expectation files because git diff skips untracked paths

git diff -- benchmarks only includes tracked files, so new expectation files written by tally accept never enter the patch and the commit job no-ops while still removing the label. Stage benchmarks first and export git diff --cached.

Evidence
  • Measure runs tally accept … benchmarks then sets patch from git diff -- benchmarks with no git add.
  • git diff does not include untracked files; only git add + git diff --cached (or equivalent) would.
  • This PR’s benchmarks/ tree has bench sources only—no committed expected-number fixtures—and the bench method has no expected-count attributes, so the first accept must create new files beside the benchmarks.
  • Commit runs only when needs.measure.outputs.patch != '', so an empty patch skips the commit after the label is already removed.

Identified by Warden · code-review · TP4-QH7

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit be62a30. Configure here.

PATCH: ${{ needs.measure.outputs.patch }}
HEAD_REF: ${{ github.event.pull_request.head.ref }}
run: |
echo "$PATCH" | base64 --decode | git apply --include='benchmarks/*'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accept patch targets the wrong commit

Medium Severity

The measure job checks out the pull request merge commit and builds a git diff against that tree, but the commit job applies the patch onto github.event.pull_request.head.sha. When files under benchmarks/ differ between those commits, git apply fails after the accept-benchmarks label is already removed.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit be62a30. Configure here.

@sentry-junior sentry-junior Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice experiment. The two-job split in benchmarks-accept.yml keeps the write token away from PR code. Two things matter for whether the numbers are useful and whether the token isolation holds, plus some context on the bot findings:

1. The benchmark install has no lock, so expected numbers can drift without any SDK change. The repo has no composer.lock, so composer require phpstan/tally:1.0.1 --update-no-dev resolves the newest guzzlehttp/psr7, symfony/options-resolver, psr/log, jean85/pretty-package-versions, and Tally's own transitive deps on every run. Those packages run inside init() and the transport, and as the workflow comment says, every installed package ends up in the event's modules list. So any upstream patch release can move instructions and memory and flag the benchmark as slower or faster. Tally's README calls the environment part of the expectation ("Expected numbers measured in one environment only fit that environment"). If the numbers should be comparable over time, a committed lock for the benchmark install would remove most of this noise, for example a small Composer project under benchmarks/ with its own composer.lock that path-requires the SDK. Disabling ModulesIntegration in the bench setup would also take the package count out of the payload. php-version: '8.5' also follows PHP patch releases, but that one is harder to pin and the README already expects it.

2. In benchmarks.yml, the pull-requests: write token is not isolated from the steps that run PR and dependency code. The comment step is the only one with GH_TOKEN, but all steps share one job. An earlier step that runs untrusted code (tally run, pr-comment.php, any dependency) can append to $GITHUB_PATH or $GITHUB_ENV, for example to put its own gh first on PATH, and the comment step then runs it with the token (workflow commands docs). Fork PRs are excluded, so this mostly matters as supply-chain exposure through the floating deps from point 1. Using the same pattern as benchmarks-accept.yml would close it: give the measuring job contents: read only, and post the table from a second job that has pull-requests: write and only reads it (as a job output or artifact).

Context on the existing bot comments:

  • The concern that git diff -- benchmarks misses untracked files does not apply here. tally accept rewrites or adds the #[Expected] attribute and its import inside the tracked *Bench.php file (README, tally accept), so the diff is not empty.
  • The merge-commit vs. head-SHA point is real but narrow. The patch only touches benchmarks/*Bench.php, so git apply only fails when the base branch changed the same bench file. In that case the label is removed and the job fails visibly. Measuring the merge commit also matches what benchmarks.yml reports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant