Repository navigation
Conversation
PyTorch publishes CUDA 13.2 and 13.4 wheels, and ExecuTorch supports both, but every job in cuda-windows.yml ran 13.2 only, because the cross-compile image carries one train's toolkits. The 13.4 Windows build was only exercised by the nightly wheel pipeline, so a pull request that broke it would land green. The model jobs now take the train from their matrix. A train other than the image's fetches its own Linux nvcc and Windows cudart with install_cuda_redist.py, the way the wheel pipeline does, so the image is unchanged. 13.4 runs on dinov2, the smallest model, since the toolkit is what differs between trains; the six models keep running on 13.2. The CUDA runtime unit tests run on both trains. The Windows GPU runners already carry the v13.4 toolkit. test_cuda_workflow.py now checks that these jobs cover every supported train, that export and e2e list the same rows, and that the export installs the toolkits for a train the image does not carry.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25307
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New Failure, 1 Unrelated Failure, 4 Unclassified FailuresAs of commit a874e5e with merge base 5e70618 ( NEW FAILURE - The following job has failed:
UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
FLAKY - The following job failed but was likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This PR needs a
|
|
@claude review effort=max |
|
Claude finished @Gasoonjia's task in 28m 19s —— View job Request Changes: the new CUDA 13.4 export row shares the e2e job's matrix-wide Detailed reviewPR Review: #25307SummaryThe PR adds a CUDA 13.4 dinov2 row to the Windows export and e2e matrices, runs
Infrastructure1. The new 13.4 export row can skip all seven e2e rows (
2. The export job now runs this script for the 13.4 row, but the script is missing from
3. (Low confidence) The 13.4 e2e row may load the 13.2 CUDA runtime Line 269 prepends only Every 13.x train names its runtime If the 13.2 copy wins, the row still passes, because Testing4. The 13.4 row covers less than its comment says (lines 109-110 and 244-245) The comment and the PR description say only the toolkit differs between trains. The Linux export also installs a different torch per train (
The model also decides which backend kernels run. So on 13.4, neither the int4 export path under that torch nor Either add one quantized 13.4 row, for example 5. The new tests check which trains the matrices list, not that the jobs use them
6. It used to assert
Assert that the non-dinov2 rows use Specific Comments
RecommendationRequest Changes. Finding 1 is a small Review steps
|
- test-model-cuda-windows-e2e no longer gates on the export matrix's overall result, so one train failing its export (say an expired torch pin) does not skip every row. Each row fails on its own missing artifact; a skipped export (fork pull request) still skips this. - The e2e job puts CUDA_HOME\bin\x64 ahead of bin, as the runtime and wheel e2e scripts do, so the 13.4 row loads the 13.4 runtime. - Trigger on install_cuda_redist.py, which the export now runs. - Add parakeet-tdt int4 on 13.4: each train also installs its own torch, and the int4 kernels only run for a quantized model. - Use cuda-version as the matrix key, as cuda.yml and the wheel workflow do. - The tests read the matrix key from gpu-arch-version, check that the jobs use the train (gpu-arch-version, CUDA\v path, bin\x64), that every model runs on the image's train, and that the Windows cudart is fetched.
|
Addressed in a2e42ab:
Also:
|
|
@claude review effort=max |
|
Claude finished @Gasoonjia's task in 32m 19s —— View job Request Changes: the CUDA 13.4 int4 row added in a2e42ab ( Detailed reviewPR Review: #25307SummaryThe PR runs the Windows CUDA export, e2e and runtime-test jobs on both 13.2 and 13.4. The export job fetches the 13.4 toolkits itself when the image's nvcc doesn't match, and
InfrastructureThe runtime job installs torch before it puts the row's toolkit on The two jobs set up
Low confidence: I couldn't read test-infra's Related: the e2e TestingThe 13.4 int4 row never calls The new comment says this row "runs the int4 kernels the non-quantized model does not". It doesn't:
So no 13.4 job runs
Specific Comments
RecommendationRequest Changes. The Testing finding leaves open the coverage gap that the row was added to close, and the comment says the gap is closed. Swapping the row or rewording the comment resolves it. The Infrastructure finding is low confidence, and the rest is optional. Review steps
Not run: |
- The 13.4 int4 row is Voxtral Realtime quantized-int4-tile-packed, the quant that lowers to _weight_int4pack_mm; quantized-int4-weight-only lowers to a dequantize plus matmul and never ran the int4mm shim. It already runs on 13.2 on Windows, so the row differs only in its train. - unittest-cuda-runtime-windows puts the toolkit on PATH before setup, as the e2e job does, so setup sees the row's nvcc. - The tests check every train has a tile-packed row, check the PATH line for both Windows jobs and that it precedes setup, and lose the duplicated assertion.
|
Addressed in 1c3cfb3:
|
|
Addressed in fe17538:
|
|
@claude review effort=max |
|
Claude finished @Gasoonjia's task in 23m 51s —— View job Request Changes: the new CUDA 13.4 Voxtral Realtime tile-packed export fails at the MinGW link on every run, because the torch 2.15 AOTI wrapper calls Detailed reviewPR Review: #25307SummaryThe PR runs the Windows CUDA export, e2e and runtime jobs on both 13.2 and 13.4. The image lacks the 13.4 toolkits, so the export fetches them with Two problems:
All three changed files were reviewed. When this review was written, two runs hadn't finished: the current head's run (44618301389) and the 13.4 dinov2 e2e in 44611792996. Infrastructure1. The 13.4 Voxtral Realtime tile-packed row can't link under torch 2.15 (blocker). The row is at Cause:
This is a torch gap, not a CUDA 13.4 one: in the same run, the 13.2 copy of the row passed on torch 2.14.0. An earlier round of this review suggested this row and missed the gap. Consequences:
Fix:
Both steps are needed: per Otherwise, drop the 13.4 Voxtral rows and the per-train tile-packed assertion ( Fix this → · Fix this (drop the row) → 2. The PR adds
It doesn't add them to the runtime job's gate (
So a PR that only touches Specific Comments
RecommendationRequest Changes.
Review steps
|
…artswith - unittest-cuda-runtime-windows also runs on install_requirements.py and install_utils.py: it builds through install_executorch.py, and setup.py enables CUDA from SUPPORTED_CUDA_VERSIONS. - The name test checks the train comes first, as its comment says. - Reword the e2e matrix comment.
|
Addressed in fec953f:
|
|
@claude review effort=max |
|
Claude finished @Gasoonjia's task in 29m 26s —— View job Request Changes, for the landing order only: the new 13.4 Voxtral Realtime tile-packed row has failed its export in every run so far and can't pass until #25284 lands. The rest is two test gaps, one low-confidence cleanup and some comment wording. Detailed reviewPR Review: #25307SummaryThe PR runs cuda-windows.yml's export, e2e and runtime jobs on every train in
Nothing in the diff breaks a job at fec953f. But one of the new 13.4 rows can't pass until #25284 lands, two of the new tests miss the regressions they guard against, and three comments need rewording. What this review did not cover:
InfrastructureLand after #25284. Until then the new 13.4 Voxtral Realtime row stays red, and each run keeps about 21 GB of artifacts (
TestingThe check-name test passes for the name the rename was meant to fix (
Code QualityLow confidence: the two copies of the toolkit switch differ in
Specific Comments
RecommendationRequest Changes, only for the landing order. Rebase onto #25284 and show the 13.4 Voxtral Realtime export and e2e passing. Nothing else blocks. |
…LIBRARY_PATH - The test renders each row's check name from the template and requires the names to be unique and at most 100 characters. An explicit name stops GitHub appending matrix values, so a prefix check alone let six rows share a name. - _matrix_cuda_versions adds include values to the axis and removes exclude values, so an exclude that drops a train fails the coverage test. - Drop the LD_LIBRARY_PATH export from the toolkit switch, matching the wheel pipeline's copy: nvcc finds its own libraries, and the export links nothing against the Linux runtime. - Comment wording: bin\x64 goes ahead of bin on PATH; only the export has its own torch; the gate test is named per row.
|
Addressed in a874e5e:
Landing order: agreed, and unchanged. The 13.4 Voxtral Realtime export failed again in run 44624449486, on the same single |
|
@claude review effort=max |
|
Claude finished @Gasoonjia's task in 28m 50s —— View job Request Changes, only for the landing order you already agreed to: the 13.4 Voxtral Realtime tile-packed export still fails without #25284's Detailed reviewPR Review: #25307SummaryThis PR adds CUDA 13.4 rows to the Windows CUDA export, e2e and runtime jobs, puts the CUDA version in each job name, and adds static tests for the matrices. The landing order you agreed to is the only blocker. a874e5e also adds a minor regression to the test's matrix helper. Infrastructure
Testing
Specific Comments
RecommendationRequest Changes. Only the landing order blocks this, and your plan covers it: rebase onto #25284, get both 13.4 rows green, then update the description and Test plan. Send the follow-ups with the rebase push. Any push touching Review steps
|
Follow-up to #25255. Land after #25284. The 13.4 rows install
CU134_TORCH_PACKAGES, a torch nightly from 2026-09-21 that leaves download.pytorch.org around 2026-11-20. #25284 replaces it with the torch 2.15.0 release, so this PR is rebased onto it once it lands and the 13.4 rows are checked again on the released torch.Why
PyTorch publishes CUDA 13.2 and 13.4 wheels (torch 2.15.0 is on
whl/cu132andwhl/cu134, not onwhl/cu130), and ExecuTorch'sSUPPORTED_CUDA_VERSIONSis(13, 2), (13, 4). But every job incuda-windows.ymlran 13.2 only, because theexecutorch-ubuntu-22.04-cuda-windowsimage carries one train's toolkits. The only 13.4 Windows cross-compile was in the nightly wheel pipeline, so a pull request that broke it could land green.What
cuda-version, the same key ascuda.ymlandbuild-wheels-cuda-windows.yml).install_cuda_redist.py, as the wheel pipeline does. The image is unchanged.install_executorch.shpicked a torch built for that train.quantized-int4-tile-packed, the only quant here that lowers to_weight_int4pack_mmand so runs theint4mmshim built by that train's nvcc. It already runs on 13.2.build-wheels-cuda-windows.yml. One row's failed export, say an expired cu134 torch pin, no longer skips the other rows; each row fails on its own missing artifact. A skipped export (fork PR) still skips e2e.unittest-cuda-runtime-windows: runs on both trains. The Windows GPU runners already carry thev13.4toolkit.CUDA_HOME\bin\x64;CUDA_HOME\binfirst onPATH, before setup, astest_cuda_runtime_windows.ps1and the wheel e2e script do..ci/scripts/wheel/install_cuda_redist.py, and oninstall_requirements.pyandinstall_utils.py, which hold the torch pins andSUPPORTED_CUDA_VERSIONS.<job>-<cuda-version> (<model>, <quant>). GitHub cuts generated names at 100 characters, which gave the 13.2 and 13.4 Voxtral Realtime rows the same name.test_cuda_workflow.py: reads each job's train from the matrix key itsgpu-arch-versionnames, and checks that:gpu-arch-version, theCUDA\vpath,bin\x64before setup);WINDOWS_CUDA_HOME;Test plan
pytest .ci/scripts/tests/test_cuda_workflow.py .ci/scripts/tests/test_cu134_dependencies.py .ci/scripts/tests/test_filter_cuda_matrix.py: 54 passed.== 'success';v13.2orgpu-arch-version: "13.2";bin\x64on PATH (e2e or runtime);.ci/scripts/testshas the same 36 environment failures/errors (missing local deps or env vars) with and without this change.install_cuda_redist.py --train 13.4 --platform windows-x86_64 --components cuda_cudartgivesbin/x64/cudart64_13.dllandlib/x64/cudart.lib. Those are the pathstorch.utils.cpp_extension(WINDOWS_CUDA_HOME/lib/x64) and Inductor's MinGW import-library step (bin/x64) read.unittest-cuda-runtime-windowstrains passed. The 13.4 runtime job built withv13.4\bin\nvcc.exe.