[spawnia]

The benchmarks check fails on PRs that do not change performance, because wall times on hosted runners are noisy.
This PR counts instructions with Valgrind on base and head instead, and fails when head needs more than 1.1x.

The red check came from a failed PR comment, the alert from runner variance

On fork PRs, GITHUB_TOKEN is read-only, so comment-on-alert failed with "Resource not accessible by integration", for example in https://github.com/webonyx/graphql-php/actions/runs/38578052847.
The alert compared one run against a baseline from another runner at another time.
In that run, 30 of 31 benchmarks came out 1.75x to 2.5x slower by the same factor.

Instruction counts catch a regression and stay within 0.1% on identical code

In https://github.com/webonyx/graphql-php/actions/runs/40303300237, this PR changes no src/ code, and the largest ratio was 1.0007.
A temporary array_sum(range(1, 100)) in Lexer::advance() failed the check in https://github.com/webonyx/graphql-php/actions/runs/40303973119, with ratios of 1.35 and 1.31 for LexerBench and 1.11 for BuildSchemaBench.
An earlier version of this PR compared wall times of base and head on the same runner, and single passes of identical code drifted by up to 1.43x in https://github.com/webonyx/graphql-php/actions/runs/40290028062.

Each subject runs until it counts about 20 million extra instructions, so setup cost cancels out

benchmarks/run-subject.php runs a PHPBench subject with its @BeforeMethods, outside of PHPBench.
benchmarks/count-instructions.php runs it under valgrind --tool=cachegrind with 1 rev, with 2 revs to estimate the cost per rev, and with enough revs to count about 20 million more instructions than the 1-rev run.
The difference, divided by the extra revs, gives the instructions per rev.
A fixed share of @Revs was not enough: the counts of whole processes vary by about 1,000 instructions, which made ScalarOverrideBench::benchGetTypeWithTypesOverride (about 2,300 per rev) fail with 1.79 on identical code in https://github.com/webonyx/graphql-php/actions/runs/40298556123.
mt_srand(1) before setup makes the random schema of HugeSchemaBench the same in every run.
Opcache is off, so Valgrind sees all executed code.
Base is the first parent of the merge commit that GitHub checks out as head.
Base and head run in parallel, and the step takes about 6 to 8 minutes.
benchmarks/compare-results.php writes a table to the step summary, which needs no write token.
Instruction counts miss cache misses and branch mispredictions, which matter little for this CPU-bound library.
Hardware counters through perf are not available on Azure VMs: actions/runner-images#4974.

PHPBench no longer retries iterations, because unlimited retries hung CI

runner.retry_threshold: 2 reran iterations until their deviation was below 2%, and PHPBench has no retry limit by default.
On noisy runners this hung DeferredBench::benchManyNestedDeferreds for 18 minutes in https://github.com/webonyx/graphql-php/actions/runs/40285990012, and probably killed the runner in https://github.com/webonyx/graphql-php/actions/runs/40214562986.
doctrine/orm and phpactor set no retry threshold either.
Local composer bench and the benchmark-history job run without retries too, so their wall times may vary more.
"Count benchmark instructions" and the "Benchmarks" job in test.yml now have a timeout.

The job skips PRs that change no benchmarked code

A paths filter limits it to src/, benchmarks/, composer.json and the workflow.
"Count benchmark instructions" is not a required check, so a skipped run blocks no merge.
CONTRIBUTING.md explains how to reproduce the check locally.

Master history tracking still uses PHPBench wall times

The benchmark-history job still pushes PHPBench results to benchmark-results.

Changelog

None

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

The summary pipeline masks comparator failures, allowing detected performance regressions to pass the check.

1 open issue

• Enable pipefail so comparator failures fail the check · New

What changed in this PR

This PR replaces cross-runner PR benchmark comparisons with paired base/head runs on the same runner, reducing noise while preserving master benchmark history.

Changes:

• Runs three alternating base/head passes and compares median timing ratios against a 1.25× threshold.
• Reports results in the step summary and limits PR runs to performance-related paths.
• Shares XML parsing between comparison and history conversion scripts.

File	Description
benchmarks/​Utils/​BenchmarkResults.php	Extracts shared PHPBench XML parsing.
benchmarks/​convert-results.php	Uses the shared parser for history output.
benchmarks/​compare-results.php	Calculates median ratios and reports regressions.
.github/​workflows/​benchmarks.yml	Adds paired benchmark runs, path filtering, and summary output.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No unresolved issues remain, and the reported CI results demonstrate stable unchanged-code measurements and detection of an injected regression.

No open issues 🎉

1 resolved since last review

• Enable pipefail so comparator failures fail the check

_{Review effort: Balanced}