Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,4 @@
/mago.toml export-ignore
/decision-model-linter export-ignore
/warden.toml export-ignore
/benchmarks export-ignore
83 changes: 83 additions & 0 deletions .github/workflows/benchmarks-accept.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
name: Accept benchmarks

# Adding the `accept-benchmarks` label to a pull request measures its benchmarks again and
# commits the measured numbers to the pull request's branch as the expected ones.
# Only people with triage access or more can add a label.
on:
pull_request:
types:
- labeled

permissions:
contents: read

jobs:
# Runs the code of the pull request and its dependencies, so it only gets a read-only token
# and hands the changed expectations to the next job as a patch.
measure:
name: Measure
# The token of a pull request from a fork cannot push, its numbers are accepted locally.
if: github.event.label.name == 'accept-benchmarks' && github.event.pull_request.head.repo.full_name == github.repository
# Same runner image, PHP, and dependencies as benchmarks.yml, because the counts depend on them.
runs-on: ubuntu-24.04
outputs:
patch: ${{ steps.accept.outputs.patch }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Setup PHP
uses: shivammathur/setup-php@eb7c497e18156a6bbabfef1d3a82760b9eda3962 # v2
with:
php-version: '8.5'
extensions: posix
coverage: none

- name: Install dependencies and Tally
run: composer require phpstan/tally:1.0.1 --update-no-dev --no-progress --no-interaction --prefer-dist

- name: Measure and accept
id: accept
run: |
vendor/bin/tally run benchmarks --output="$RUNNER_TEMP/tally-results" || [ $? -eq 1 ]
vendor/bin/tally accept "$RUNNER_TEMP/tally-results" benchmarks
echo "patch=$(git diff -- benchmarks | base64 --wrap=0)" >> "$GITHUB_OUTPUT"

Check warning on line 46 in .github/workflows/benchmarks-accept.yml

View check run for this annotation

@sentry/warden / warden: code-review

Accept patch omits new expectation files because git diff skips untracked paths

`git diff -- benchmarks` only includes tracked files, so new expectation files written by `tally accept` never enter the patch and the commit job no-ops while still removing the label. Stage `benchmarks` first and export `git diff --cached`.
Comment on lines +44 to +46

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accept patch omits new expectation files because git diff skips untracked paths

git diff -- benchmarks only includes tracked files, so new expectation files written by tally accept never enter the patch and the commit job no-ops while still removing the label. Stage benchmarks first and export git diff --cached.

Evidence
  • Measure runs tally accept … benchmarks then sets patch from git diff -- benchmarks with no git add.
  • git diff does not include untracked files; only git add + git diff --cached (or equivalent) would.
  • This PR’s benchmarks/ tree has bench sources only—no committed expected-number fixtures—and the bench method has no expected-count attributes, so the first accept must create new files beside the benchmarks.
  • Commit runs only when needs.measure.outputs.patch != '', so an empty patch skips the commit after the label is already removed.

Identified by Warden · code-review · TP4-QH7


# Has the token that can push, and runs nothing from the pull request.
commit:
name: Commit
needs: measure
if: always() && needs.measure.result != 'skipped'
runs-on: ubuntu-24.04
permissions:
contents: write
pull-requests: write
steps:
# Also when measuring failed, so that adding the label again starts another attempt.
- name: Remove the label
env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: gh api --method DELETE "repos/$GITHUB_REPOSITORY/issues/$PR_NUMBER/labels/accept-benchmarks" --silent

- name: Checkout
if: needs.measure.outputs.patch != ''
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.event.pull_request.head.sha }}

# The push is rejected when the branch got new commits since the label was added.
# A push with this token does not start workflows, so CI only runs again with the next push.
- name: Commit the expected numbers
if: needs.measure.outputs.patch != ''
env:
PATCH: ${{ needs.measure.outputs.patch }}
HEAD_REF: ${{ github.event.pull_request.head.ref }}
run: |
echo "$PATCH" | base64 --decode | git apply --include='benchmarks/*'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accept patch targets the wrong commit

Medium Severity

The measure job checks out the pull request merge commit and builds a git diff against that tree, but the commit job applies the patch onto github.event.pull_request.head.sha. When files under benchmarks/ differ between those commits, git apply fails after the accept-benchmarks label is already removed.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit be62a30. Configure here.

git add benchmarks
git -c user.name='github-actions[bot]' -c user.email='41898282+github-actions[bot]@users.noreply.github.com' \
commit --message='chore(benchmarks): accept new expected numbers'
git push origin "HEAD:refs/heads/$HEAD_REF"
89 changes: 89 additions & 0 deletions .github/workflows/benchmarks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
name: Benchmarks

on:
pull_request:
push:
branches:
- master
- release/**

permissions:
contents: read

# Cancel in progress workflows on pull_requests.
# https://docs.github.com/en/actions/using-jobs/using-concurrency#example-using-a-fallback-value
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true

jobs:
tally:
name: Tally
# Tally compares instruction counts with the numbers committed next to each benchmark.
# The counts depend on the runner image and the PHP build, so neither floats with `latest`.
runs-on: ubuntu-24.04
permissions:
contents: read
pull-requests: write
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
# The token can comment on pull requests, so the code under test must not find it in .git/config.
persist-credentials: false

- name: Setup PHP
uses: shivammathur/setup-php@eb7c497e18156a6bbabfef1d3a82760b9eda3962 # v2
with:
php-version: '8.5'
extensions: posix
coverage: none

# Tally needs PHP 8.2+, so it cannot be in composer.json, which has to install on PHP 7.2.
# It is required without the dev dependencies, because every installed package ends up in
# the modules list of an event and changes the counts.
- name: Install dependencies and Tally
run: composer require phpstan/tally:1.0.1 --update-no-dev --no-progress --no-interaction --prefer-dist

# Exit code 1 means a benchmark is slower or could not be measured. That is reported in the
# job summary and the pull request comment, and does not fail the job. Exit code 2 means
# Tally itself could not run, which still fails, otherwise a broken setup would go unnoticed.
- name: Run benchmarks
run: |
vendor/bin/tally run benchmarks || [ $? -eq 1 ]
cp "$GITHUB_STEP_SUMMARY" "$RUNNER_TEMP/tally-report.md"
# Built here, because the next step has a token that code of the pull request must not run with.
php benchmarks/pr-comment.php "$RUNNER_TEMP/tally-report.md" > "$RUNNER_TEMP/tally-table.md"

# Pull requests from forks only get a read-only token, they have the report in the job summary.
# Dependabot's token is read-only too, so a failed comment does not fail the job either.
- name: Comment on the pull request
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
continue-on-error: true
env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: |
marker='<!-- tally-report -->'
report="$RUNNER_TEMP/tally-report.md"
comment="$RUNNER_TEMP/tally-comment.md"
# The comment is the headline and the table of all benchmarks. The full report with
# the patch of new expected numbers is in the job summary.
{
echo "$marker"
cat "$RUNNER_TEMP/tally-table.md"
echo
echo 'Add the `accept-benchmarks` label to commit the measured numbers as the expected ones.'
} > "$comment"

comment_id=$(gh api "repos/$GITHUB_REPOSITORY/issues/$PR_NUMBER/comments" --paginate \
--jq ".[] | select(.body | startswith(\"$marker\")) | .id" | head -n 1)

if [ -n "$comment_id" ]; then
# Keep the existing comment up to date, also once the benchmarks are unchanged again.
gh api --method PATCH "repos/$GITHUB_REPOSITORY/issues/comments/$comment_id" -F body=@"$comment" --silent
elif grep -q '^> \[!' "$report"; then
# Tally starts a section with an alert for slower, faster, new, and not measured
# benchmarks. Without one, every benchmark is unchanged and there is nothing to say.
gh api --method POST "repos/$GITHUB_REPOSITORY/issues/$PR_NUMBER/comments" -F body=@"$comment" --silent
fi
86 changes: 86 additions & 0 deletions benchmarks/CaptureExceptionBench.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
<?php

declare(strict_types=1);

namespace Sentry\Benchmarks;

use Sentry\HttpClient\HttpClientInterface;
use Sentry\HttpClient\Request;
use Sentry\HttpClient\Response;
use Sentry\Options;
use Tally\Attribute\After;
use Tally\Attribute\Before;
use Tally\Attribute\Warmup;

use function Sentry\captureException;
use function Sentry\init;

/**
* Measures captureException() from the global helper down to the serialized
* envelope. The real transport runs, only the HTTP request itself is replaced.
*/
final class CaptureExceptionBench
{
/**
* Frames added on top of Tally's own, so the stack trace has the depth of
* an exception thrown inside an application.
*/
private const STACK_DEPTH = 20;

private CountingHttpClient $httpClient;

private \Throwable $exception;

#[Before]
public function setUp(): void
{
$this->httpClient = new CountingHttpClient();

init([
'dsn' => 'https://public@example.com/1',
'http_client' => $this->httpClient,
]);

$this->exception = $this->createException(self::STACK_DEPTH);
}

#[Name('captureException with 20 recursive frames')]
#[Warmup(1)]
public function benchCaptureException(): void

Check warning on line 49 in benchmarks/CaptureExceptionBench.php

View workflow job for this annotation

GitHub Actions / Tally

Sentry\Benchmarks\CaptureExceptionBench::benchCaptureException has no expected numbers

instructions 2.052M cycles 2.467M memory 103.1KiB
{
captureException($this->exception);
}

/**
* Without this, an event dropped before the transport would be measured
* as a very fast capture.
*/
#[After]
public function assertEnvelopeWasSent(): void
{
if ($this->httpClient->requests === 0) {
throw new \LogicException('captureException() did not reach the HTTP client.');
}
}

private function createException(int $depth): \Throwable
{
if ($depth > 0) {
return $this->createException($depth - 1);
}

return new \RuntimeException('Benchmark exception');
}
}

final class CountingHttpClient implements HttpClientInterface
{
public int $requests = 0;

public function sendRequest(Request $request, Options $options): Response
{
++$this->requests;

return new Response(200, [], '');
}
}
17 changes: 17 additions & 0 deletions benchmarks/Name.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
<?php

declare(strict_types=1);

namespace Sentry\Benchmarks;

/**
* The name of a benchmark in the pull request comment, instead of its class and method.
* Tally itself does not know it and keeps reporting the case ID.
*/
#[\Attribute(\Attribute::TARGET_METHOD)]
final class Name
{
public function __construct(public readonly string $name)
{
}
}
87 changes: 87 additions & 0 deletions benchmarks/pr-comment.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
<?php

declare(strict_types=1);

/*
* Turns Tally's Markdown report into the body of the pull request comment: the headline and
* a table with one row per benchmark, with the names from #[Name] instead of the case IDs.
*
* Usage: php benchmarks/pr-comment.php <report.md>
*/

namespace Sentry\Benchmarks;

require_once __DIR__ . '/../vendor/autoload.php';
require_once __DIR__ . '/Name.php';

foreach (glob(__DIR__ . '/*Bench.php') ?: [] as $file) {
require_once $file;
}

$lines = file($argv[1], \FILE_IGNORE_NEW_LINES);
if ($lines === false || $lines === []) {
fwrite(\STDERR, \sprintf("Cannot read the report from %s.\n", $argv[1]));

exit(1);
}

/**
* The name from #[Name] of a case like `Class::method` or `Class::method@data set`.
*/
function displayName(string $cell): string
{
if (preg_match('/^`([^`@]+)::(\w+)(@.*)?`$/', $cell, $match) !== 1 || !method_exists($match[1], $match[2])) {
return $cell;
}

$attributes = (new \ReflectionMethod($match[1], $match[2]))->getAttributes(Name::class);
if ($attributes === []) {
return $cell;
}

return $attributes[0]->newInstance()->name . ($match[3] ?? '');
}

// The table of all benchmarks is the one inside <details>, the last of the report. It has a
// row per metric, with the case only in the first one:
// | Benchmark | Metric | Expected | Measured | Change | Allowed | Status |
$benchmarks = [];
$inTable = false;
foreach ($lines as $line) {
if (str_starts_with($line, '<details>')) {
$inTable = true;
}

if (!$inTable || !str_starts_with($line, '|')) {
continue;
}

[$benchmark, $metric, $expected, $measured, $change, $allowed] = array_map('trim', explode('|', trim($line, '|')));
if ($benchmark === 'Benchmark' || str_starts_with($benchmark, '-')) {
// the header and the separator below it
continue;
}

if ($benchmark !== '') {
$benchmarks[] = ['name' => displayName($benchmark), 'metrics' => []];
}

$benchmarks[array_key_last($benchmarks)]['metrics'][$metric] = [
$change === '-' ? $measured : \sprintf('%s (%s)', $measured, $change),
$allowed === '-' ? $expected : \sprintf('%s %s', $expected, $allowed),
];
}

echo $lines[0], "\n\n";
echo "| Benchmark | Instructions | Instructions expected | Cycles | Cycles expected | Memory | Memory expected |\n";
echo "|---|--:|--:|--:|--:|--:|--:|\n";

foreach ($benchmarks as $benchmark) {
$cells = [$benchmark['name']];
foreach (['instructions', 'cycles', 'memory'] as $metric) {
// a benchmark that could not be measured has no metrics
array_push($cells, ...($benchmark['metrics'][$metric] ?? ['-', '-']));
}

echo '| ', implode(' | ', $cells), " |\n";
}
Loading