Repository navigation
ci(release): build the sixteen wheels in parallel jobs (platform x Python) - #94
Merged
Merged
Conversation
…orm x Python), not four jobs of four DECISION: the wheels matrix is platform x Python version. Each job builds one wheel with cibuildwheel (CIBW_BUILD=cpXYZ-*). Not a separate OpenCASCADE job, not fewer Pythons. PROBLEM A release took 1 h 48 min (2.2.9 dry run) to 1 h 40+ min (v2.3.0) and the wall-clock time was one job: the Windows one, which compiles the C++ module and links the static OpenCASCADE for cp310, cp311, cp312 and cp313 one after the other (1 h 32 min for the 2.2.9 dry run); macOS and Linux needed 48-69 min the same way. Nothing in the four builds of a platform depends on the others. CHANGE - plan: the matrix is the product of the chosen platforms and the chosen Pythons (new dry-run input `pythons`, default all, like `platforms`), 16 jobs by default. Checked by running the plan script for the default and for a narrowed dry run. - wheels: job name and artifact name carry the Python (`wheels-<platform>-cpXYZ`), because sixteen jobs uploading `wheels-<platform>` would overwrite each other. Every build step sets CIBW_BUILD. - test: its matrix carries the cp tag of each Python and downloads `wheels-linux-x86_64-<tag>`. - release: unchanged. It collects `artifacts/wheels-*/*.whl` and still requires exactly 16 wheels, four per platform, each with one compiled module for its own Python. WHAT IT COSTS, AND WHY IT IS ACCEPTED Every job restores the OpenCASCADE prefix from the cache (the key is the hash of build_occt.py, unchanged, so a hit). When build_occt.py does change, the four jobs of a platform each build OCCT at the same time: four times the runner minutes of that platform, the same wall-clock time as now, and the first to finish saves the cache (the others' save is continue-on-error). That happens rarely and the wall-clock is what hurts. A separate OCCT job that the wheel jobs wait for would avoid the duplicate work but adds a job and an artifact hand-off of a multi-gigabyte prefix; not worth it for a rare case. NOT KNOWN YET Restoring the cache sixteen times is sixteen downloads; the dry run of this branch measures what that costs. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The release workflow builds its sixteen wheels in sixteen parallel jobs (platform x Python) instead of four jobs that build four Pythons one after the other. The decision and its cost are in the commit message.
planexpands platforms x Pythons (new dry-run inputpythons, default all);wheelsjobs and their artifacts carry the Python (wheels-<platform>-cpXYZ);testdownloads the wheel of its own Python;releaseis unchanged and still demands exactly 16 wheels.Verified by a dry run of this branch (nothing published): all 16 wheel jobs, the four test jobs, the verification of the distributions and the skill build passed (run 43183454947).
Timing, honestly: that run had a cold OpenCASCADE cache (so every job built OCCT itself, the worst case this change costs): 1 h 13 min in total against 1 h 58 min for the real v2.3.0 release, which was cold as well. The caches are warm now; a second dry run measures the usual case and its result will be added here.
🤖 Generated with Claude Code