Skip to content

Benchmark generated projects and time a chain of modules in CI #9282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
JonoPrest wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Benchmark generated projects and time a chain of modules in CI #9282

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ea9680b
Count the build's allocations on generated projects in the benchmarks
JonoPrest Nov 9, 2026
66b8aa3
Time a chain of modules with the base's and the head's build systems
JonoPrest Nov 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -455,7 +455,8 @@ jobs:
# to 0.2%. The runtime and Belt are built there too, since the
# replayed builds would otherwise write interfaces that don't
# match those of the files they don't replay yet. Both build the
# base's tests/tests, which the base's compiler builds already.
# base's tests/tests, which the base's compiler builds already,
# and the generated projects of this checkout's script.
measure() {
sudo mkdir -p /bench
sudo mount --bind "$1" /bench
Expand All @@ -466,10 +467,17 @@ jobs:
}
measure "$RUNNER_TEMP/base" "$RUNNER_TEMP/base.json"
measure "$GITHUB_WORKSPACE" "$RUNNER_TEMP/head.json"
# Wall time, with the two build systems taking turns on this
# runner. It only indicates a change that counting can't show, so
# a failure, or a run longer than 10 minutes, leaves it out of the
# report and fails nothing.
timeout --kill-after=30s 10m \
node scripts/compilerBenchmarks.js time "$RUNNER_TEMP/base" "$GITHUB_WORKSPACE" \
"$RUNNER_TEMP/timings.json" || echo "::warning::The wall-time benchmark failed."
status=0
report="$RUNNER_TEMP/benchmarks-report.md"
node scripts/compilerBenchmarks.js compare "$RUNNER_TEMP/base.json" "$RUNNER_TEMP/head.json" \
--threshold 1 --summary "$report" || status=$?
--threshold 1 --summary "$report" --timings "$RUNNER_TEMP/timings.json" || status=$?
cat "$report" >> "$GITHUB_STEP_SUMMARY"
printf '\nPull request %s compared with base %s in [this run](%s).\n' \
"$(git rev-parse --short=9 HEAD^2)" "$(git rev-parse --short=9 HEAD^1)" "$RUN_URL" \
Expand Down
2 changes: 2 additions & 0 deletions changes/generated-project-benchmarks.internal.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
- Count the words the build allocates on generated projects in the CI benchmarks: many modules using little of a few large dependencies, and a namespaced package of 1,000 modules.
- Time clean builds of a generated chain of modules with the base's and the head's build systems in turn in the CI benchmarks, and report the median ratio with a 95% confidence interval. The times are only indicative and never fail the job.
Loading
Loading