Skip to content

Pull requests now benchmark their changes against the base branch - #134

Merged
thunderer merged 1 commit into
masterfrom
tk-benchmarks
Nov 8, 2026
Merged

thunderer merged 1 commit into
masterfrom
tk-benchmarks

Conversation

@thunderer

Copy link
Copy Markdown
Owner

Pull requests now run PHPBench on the base branch and on their own code, and post the difference as one comment that each new run updates. PHPBench stays out of composer.json; the job installs it with composer require --dev on PHP 8.5, like the Infection job.

  • a benchmark job runs only on pull requests: it benchmarks src from the base branch, tags the run base, restores the pull request's src and prints the aggregate report against base,
  • the comment carries a hidden <!-- benchmark --> marker; the job edits the newest github-actions[bot] comment with that marker or posts a new one, with plain gh and no third-party action, and skips the comment for forks, whose token is read-only,
  • ParserBench runs RegularParser, RegexParser, HybridParser and WordpressParser on 5 inputs of about 10 KB: a page in English and Polish with a few shortcodes, 240 shortcodes with every parameter form, a page-builder layout nested 4 levels, 100 nested levels, and 28 KB of prose with citations and Markdown links whose unclosed tags are quadratic for 3 parsers,
  • ProcessorBench runs Processor with RegularParser and HybridParser on the page, 1000 shortcodes in 39 KB (the quadratic replacement step), the layout and the 100 levels,
  • each case repeats its call to take 60-150 ms per iteration, and the parser cases stay under 8 MB per call; larger working sets and shorter cases were 10-30% noisy on a loaded machine,
  • verified with PHPBench 1.7.0 on PHP 8.5: 4 runs of the same code differ by 2.8% on average and 8.9% at most per case, Iterations(5) and Warmup(1) beat 3 and 10 iterations and 2 warmups, one run takes 31 s; PHPUnit, Mago and Psalm do not load the benchmark files.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

Benchmark ⚡ improved ❌ regressed ✅ untouched 🆕 new
ProcessorBench 0 0 8 0
ParserBench 0 0 20 0

A benchmark is changed when its efficiency (BASE / HEAD - 1) is at least 10% in either direction.

All benchmarks
+----------------+--------------+--------------------+------+-----+----------------+------------------+------------------+
| benchmark      | subject      | set                | revs | its | mem_peak       | mode             | rstdev           |
+----------------+--------------+--------------------+------+-----+----------------+------------------+------------------+
| ProcessorBench | benchProcess | regular article    | 1    | 5   | 3.884mb 0.00%  | 78.640ms -2.70%  | ±2.03% +65.24%   |
| ProcessorBench | benchProcess | regular dense      | 1    | 5   | 16.017mb 0.00% | 114.017ms +3.96% | ±2.06% +15.35%   |
| ProcessorBench | benchProcess | regular nested     | 1    | 5   | 5.821mb 0.00%  | 91.204ms -2.33%  | ±3.13% -11.35%   |
| ProcessorBench | benchProcess | regular deep       | 1    | 5   | 2.516mb 0.00%  | 131.633ms -1.63% | ±1.41% +42.64%   |
| ProcessorBench | benchProcess | hybrid article     | 1    | 5   | 2.125mb 0.00%  | 100.231ms -2.61% | ±11.45% +988.61% |
| ProcessorBench | benchProcess | hybrid dense       | 1    | 5   | 4.206mb 0.00%  | 98.899ms +0.71%  | ±1.30% -33.18%   |
| ProcessorBench | benchProcess | hybrid nested      | 1    | 5   | 2.542mb 0.00%  | 125.333ms -1.37% | ±1.80% -2.64%    |
| ProcessorBench | benchProcess | hybrid deep        | 1    | 5   | 2.315mb 0.00%  | 64.062ms -3.94%  | ±1.29% +78.55%   |
| ParserBench    | benchParse   | regular article    | 1    | 5   | 4.540mb 0.00%  | 85.328ms -4.00%  | ±1.49% +8.25%    |
| ParserBench    | benchParse   | regular dense      | 1    | 5   | 6.204mb 0.00%  | 81.016ms -3.63%  | ±0.69% +3.91%    |
| ParserBench    | benchParse   | regular nested     | 1    | 5   | 7.011mb 0.00%  | 131.701ms -3.42% | ±2.36% +634.95%  |
| ParserBench    | benchParse   | regular deep       | 1    | 5   | 9.338mb 0.00%  | 86.463ms +1.54%  | ±2.98% +443.21%  |
| ParserBench    | benchParse   | regular brackets   | 1    | 5   | 12.082mb 0.00% | 72.943ms -0.71%  | ±1.92% +73.74%   |
| ParserBench    | benchParse   | regex article      | 1    | 5   | 2.125mb 0.00%  | 126.222ms +1.83% | ±1.78% +168.98%  |
| ParserBench    | benchParse   | regex dense        | 1    | 5   | 2.294mb 0.00%  | 88.442ms -0.68%  | ±0.58% -73.12%   |
| ParserBench    | benchParse   | regex nested       | 1    | 5   | 2.125mb 0.00%  | 85.628ms +4.69%  | ±2.15% +95.30%   |
| ParserBench    | benchParse   | regex deep         | 1    | 5   | 2.125mb 0.00%  | 103.030ms +0.72% | ±0.95% +49.27%   |
| ParserBench    | benchParse   | regex brackets     | 1    | 5   | 3.005mb 0.00%  | 121.036ms +2.07% | ±2.51% -18.98%   |
| ParserBench    | benchParse   | hybrid article     | 1    | 5   | 2.125mb 0.00%  | 102.529ms +3.05% | ±0.38% -85.22%   |
| ParserBench    | benchParse   | hybrid dense       | 1    | 5   | 2.182mb 0.00%  | 68.826ms +3.83%  | ±0.66% -58.71%   |
| ParserBench    | benchParse   | hybrid nested      | 1    | 5   | 2.395mb 0.00%  | 143.296ms +3.77% | ±0.81% -55.38%   |
| ParserBench    | benchParse   | hybrid deep        | 1    | 5   | 3.474mb 0.00%  | 141.184ms +2.56% | ±1.17% +12.78%   |
| ParserBench    | benchParse   | hybrid brackets    | 1    | 5   | 3.010mb 0.00%  | 137.129ms +1.57% | ±0.96% -45.49%   |
| ParserBench    | benchParse   | wordpress article  | 1    | 5   | 2.125mb 0.00%  | 70.747ms -1.32%  | ±0.89% -34.43%   |
| ParserBench    | benchParse   | wordpress dense    | 1    | 5   | 2.125mb 0.00%  | 132.554ms +1.79% | ±1.67% -25.66%   |
| ParserBench    | benchParse   | wordpress nested   | 1    | 5   | 2.125mb 0.00%  | 73.594ms +0.88%  | ±2.71% -4.57%    |
| ParserBench    | benchParse   | wordpress deep     | 1    | 5   | 2.125mb 0.00%  | 77.782ms -1.09%  | ±3.64% -48.49%   |
| ParserBench    | benchParse   | wordpress brackets | 1    | 5   | 2.966mb 0.00%  | 72.513ms +0.26%  | ±1.91% +13.09%   |
+----------------+--------------+--------------------+------+-----+----------------+------------------+------------------+

Comparing tk-benchmarks (04deee6) with master (8c3855b)

@thunderer
thunderer merged commit 7a5a0c8 into master Nov 8, 2026
18 checks passed
@thunderer
thunderer deleted the tk-benchmarks branch November 8, 2026 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant