Skip to content

Cover CUDA 13.4 in the Windows CUDA pull request jobs - #25307

Open
Gasoonjia wants to merge 7 commits into
mainfrom
gasoonjia/windows-cuda-13.4
Open

Gasoonjia wants to merge 7 commits into
mainfrom
gasoonjia/windows-cuda-13.4

Conversation

@Gasoonjia

@Gasoonjia Gasoonjia commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #25255. Land after #25284. The 13.4 rows install CU134_TORCH_PACKAGES, a torch nightly from 2026-09-21 that leaves download.pytorch.org around 2026-11-20. #25284 replaces it with the torch 2.15.0 release, so this PR is rebased onto it once it lands and the 13.4 rows are checked again on the released torch.

Why

PyTorch publishes CUDA 13.2 and 13.4 wheels (torch 2.15.0 is on whl/cu132 and whl/cu134, not on whl/cu130), and ExecuTorch's SUPPORTED_CUDA_VERSIONS is (13, 2), (13, 4). But every job in cuda-windows.yml ran 13.2 only, because the executorch-ubuntu-22.04-cuda-windows image carries one train's toolkits. The only 13.4 Windows cross-compile was in the nightly wheel pipeline, so a pull request that broke it could land green.

What

  • Model export and e2e: each row names its train (cuda-version, the same key as cuda.yml and build-wheels-cuda-windows.yml).
    • If the row's train differs from the image's nvcc, the export fetches that train's Linux nvcc and Windows cudart with install_cuda_redist.py, as the wheel pipeline does. The image is unchanged.
    • The export then asserts that install_executorch.sh picked a torch built for that train.
    • Artifact names carry the train.
  • 13.4 rows: each train installs its own toolkit and its own torch, so 13.4 runs two models; all six rows (four models) keep running on 13.2:
    • dinov2, the smallest model;
    • Voxtral Realtime quantized-int4-tile-packed, the only quant here that lowers to _weight_int4pack_mm and so runs the int4mm shim built by that train's nvcc. It already runs on 13.2.
  • e2e gate: e2e is not gated on the export matrix succeeding, mirroring build-wheels-cuda-windows.yml. One row's failed export, say an expired cu134 torch pin, no longer skips the other rows; each row fails on its own missing artifact. A skipped export (fork PR) still skips e2e.
  • unittest-cuda-runtime-windows: runs on both trains. The Windows GPU runners already carry the v13.4 toolkit.
  • PATH: both Windows jobs put CUDA_HOME\bin\x64;CUDA_HOME\bin first on PATH, before setup, as test_cuda_runtime_windows.ps1 and the wheel e2e script do.
  • Path filters: the workflow triggers on .ci/scripts/wheel/install_cuda_redist.py, and on install_requirements.py and install_utils.py, which hold the torch pins and SUPPORTED_CUDA_VERSIONS.
  • Check names: export and e2e jobs are named <job>-<cuda-version> (<model>, <quant>). GitHub cuts generated names at 100 characters, which gave the 13.2 and 13.4 Voxtral Realtime rows the same name.
  • test_cuda_workflow.py: reads each job's train from the matrix key its gpu-arch-version names, and checks that:
    • each Windows job covers every supported train and uses it (gpu-arch-version, the CUDA\v path, bin\x64 before setup);
    • export and e2e list the same rows;
    • every train has a tile-packed int4 row;
    • every model runs on the image's own train;
    • the export fetches both the Linux toolkit and the Windows cudart and exports WINDOWS_CUDA_HOME;
    • e2e doesn't gate on the export matrix's overall success.

Test plan

  • pytest .ci/scripts/tests/test_cuda_workflow.py .ci/scripts/tests/test_cu134_dependencies.py .ci/scripts/tests/test_filter_cuda_matrix.py: 54 passed.
  • Reintroduced each guarded regression locally, and each one fails a test:
    • e2e gate back to == 'success';
    • runtime pinned to v13.2 or gpu-arch-version: "13.2";
    • no bin\x64 on PATH (e2e or runtime);
    • no Windows cudart fetch;
    • image bumped to 13.4;
    • the 13.4 int4 row switched back to weight-only.
  • Full .ci/scripts/tests has the same 36 environment failures/errors (missing local deps or env vars) with and without this change.
  • install_cuda_redist.py --train 13.4 --platform windows-x86_64 --components cuda_cudart gives bin/x64/cudart64_13.dll and lib/x64/cudart.lib. Those are the paths torch.utils.cpp_extension (WINDOWS_CUDA_HOME/lib/x64) and Inductor's MinGW import-library step (bin/x64) read.
  • In this PR's CI, the 13.4 dinov2 export and both unittest-cuda-runtime-windows trains passed. The 13.4 runtime job built with v13.4\bin\nvcc.exe.

PyTorch publishes CUDA 13.2 and 13.4 wheels, and ExecuTorch supports both,
but every job in cuda-windows.yml ran 13.2 only, because the cross-compile
image carries one train's toolkits. The 13.4 Windows build was only exercised
by the nightly wheel pipeline, so a pull request that broke it would land
green.

The model jobs now take the train from their matrix. A train other than the
image's fetches its own Linux nvcc and Windows cudart with
install_cuda_redist.py, the way the wheel pipeline does, so the image is
unchanged. 13.4 runs on dinov2, the smallest model, since the toolkit is what
differs between trains; the six models keep running on 13.2. The CUDA
runtime unit tests run on both trains. The Windows GPU runners already carry
the v13.4 toolkit.

test_cuda_workflow.py now checks that these jobs cover every supported train,
that export and e2e list the same rows, and that the export installs the
toolkits for a train the image does not carry.
@pytorch-bot

pytorch-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25307

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure, 4 Unclassified Failures

As of commit a874e5e with merge base 5e70618 (image):

NEW FAILURE - The following job has failed:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

FLAKY - The following job failed but was likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 19, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 28m 19s —— View job


Request Changes: the new CUDA 13.4 export row shares the e2e job's matrix-wide result == 'success' gate, so if only the 13.4 export fails, all six 13.2 Windows e2e rows are skipped too. That can happen soon: the cu134 nightly torch pin from 2026-09-21 reaches download.pytorch.org's ~60-day retention about now. The other findings are smaller gaps in path filters, Windows PATH setup, the 13.4 scope comment, and what the new tests check.

Detailed review

PR Review: #25307

Summary

The PR adds a CUDA 13.4 dinov2 row to the Windows export and e2e matrices, runs unittest-cuda-runtime-windows on both trains, and extends test_cuda_workflow.py to check train coverage. When the image's nvcc doesn't match, the new row fetches the 13.4 Linux nvcc and Windows cudart with install_cuda_redist.py. Problems found:

  • The new row can skip every e2e row.
  • The script it now depends on is outside the workflow's path filters.
  • The 13.4 e2e row may load the 13.2 CUDA runtime (low confidence).
  • The "only the toolkit differs" scope claim isn't accurate.
  • The new tests check which trains the matrices list, not that the jobs use them.

Infrastructure

1. The new 13.4 export row can skip all seven e2e rows (.github/workflows/cuda-windows.yml:199)

needs.export-model-cuda-windows-artifact.result is one result for the whole export matrix. If any export leg fails, test-model-cuda-windows-e2e skips every row, including the 13.2 rows whose artifacts uploaded fine. The gate isn't new, but before this PR every row shared one train and one torch. The 13.4 row adds failure modes the other rows don't have:

  • Toolkit downloads: lines 152-159 download two toolkits, and _fetch in .ci/scripts/wheel/install_cuda_redist.py:55-59 has no retry.
  • Expiring torch pin: the row installs torch==2.15.0.dev20260921+cu134. install_requirements.py:30-34 says this pin expires because download.pytorch.org keeps nightlies for about 60 days, which runs out around 2026-11-20. From then on, every run loses all Windows e2e coverage until someone bumps the pin.

build-wheels-cuda-windows.yml:253-258 stopped gating on the matrix result for this exact reason. Suggested gate: !cancelled() && needs.export-model-cuda-windows-artifact.result != 'skipped' && (...). The skipped check still covers the fork-PR case described in the comment at lines 192-196, and only rows whose own export failed fail at download. Update that comment too. Fix this →

2. .ci/scripts/wheel/install_cuda_redist.py isn't in this workflow's path filters

The export job now runs this script for the 13.4 row, but the script is missing from pull_request.paths (lines 15-26) and from the contains() lists at lines 59-69 and 198-208. A PR that changes only this script, for example _RELEASES["13.4"] at install_cuda_redist.py:36, gets no 13.4 coverage:

  • build-wheels-cuda-windows.yml does trigger on .ci/**, but its PR matrix is cu132 only (.github/scripts/filter_cuda_matrix.py:76).
  • On push, this job runs only on sampled full runs.

Fix this →

3. (Low confidence) The 13.4 e2e row may load the 13.2 CUDA runtime

Line 269 prepends only $env:CUDA_HOME\bin. The 13.x Windows toolkits keep their DLLs in bin\x64, which is why test_cuda_runtime_windows.ps1:27 and wheel/test_cuda_windows_e2e.ps1:59 prepend bin\x64;bin. test_model_e2e_windows.ps1 doesn't add it either.

Every 13.x train names its runtime cudart64_13.dll. So which copy the runner and the AOTI DLL load depends on the order of the image's system PATH, unless dinov2's post-build copy (examples/models/dinov2/CMakeLists.txt:88-99) picks up cudart. I couldn't confirm whether TARGET_RUNTIME_DLLS includes it.

If the 13.2 copy wins, the row still passes, because -ExpectedCudaVersion (line 278) checks only nvcc. Prepending bin\x64 the way the sibling scripts do settles it. Fix this →

Testing

4. The 13.4 row covers less than its comment says (lines 109-110 and 244-245)

The comment and the PR description say only the toolkit differs between trains. The Linux export also installs a different torch per train (install_requirements.py:33-37, 104-130; test_cu134_dependencies.py:72, 114):

  • 13.2: torch==2.14.0
  • 13.4: torch==2.15.0.dev20260921+cu134

The model also decides which backend kernels run. So on 13.4, neither the int4 export path under that torch nor backends/cuda/runtime/shims/int4mm.cu built by the 13.4 nvcc ever runs, and unittest-cuda-runtime-windows doesn't cover int4 either.

Either add one quantized 13.4 row, for example nvidia/parakeet-tdt with quantized-int4-weight-only, or narrow the comment. "The other CUDA train" also assumes there are exactly two trains. Fix this →

5. The new tests check which trains the matrices list, not that the jobs use them

test_windows_jobs_cover_every_supported_train compares only matrix values. If line 305 goes back to "13.2" or line 314 goes back to v13.2, every test still passes, and unittest-cuda-runtime-windows-13.4 quietly runs 13.2. Two changes:

  • Check the wiring: for the three Windows jobs, mirror line 55's gpu-arch-version == "${{ matrix.cuda }}", and assert that the e2e and runtime scripts use CUDA\v${{ matrix.cuda }}.
  • Stop guessing the key: _matrix_cuda_versions (lines 28-32) should read the key named in the gpu-arch-version expression. Today any include wins, which breaks in two ways:
    • Moving test-model-cuda-e2e (cuda.yml:1043) onto a cuda-version axis, as test-cuda-builds uses, raises KeyError on its pybind include rows.
    • A fromJSON(...) matrix, the pattern at build-wheels-cuda-windows.yml:170, gets split into single characters.

Fix this →

6. test_windows_export_image_matches_the_export_job no longer checks what its name says

It used to assert gpu-arch-version == image_version. Line 114 now asserts only that the image's version appears somewhere in the matrix, which line 109 plus the coverage test already guarantee. Two regressions now pass:

  • Bumping .ci/docker/build.sh:91 to CUDA_VERSION=13.4 passes, while the six 13.2 rows quietly start downloading two toolkits on every run.
  • Deleting the Windows cudart fetch (lines 158-160) passes the substring checks at lines 116-117, because the --train substring also matches the Linux install at line 152. The 13.4 export would then link against the image's 13.2 Windows cudart.

Assert that the non-dinov2 rows use image_version, and check for --platform windows-x86_64 and the WINDOWS_CUDA_HOME export. Fix this →

Specific Comments

  • .github/workflows/cuda-windows.yml:82 (every row): the matrix key here is cuda, but cuda.yml:155 and build-wheels-cuda-windows.yml:170 use cuda-version. Using one name would also let _matrix_cuda_versions drop a fallback.
  • .ci/scripts/tests/test_cuda_workflow.py:73: for version in versions rebinds version from line 66.
  • .ci/scripts/tests/test_cuda_workflow.py:93-99: the generator rebinds name from the loop at line 83. Two direct lookups would be simpler.
  • .ci/scripts/wheel/install_cuda_redist.py:10, 33-35: these lines say the script's only consumers are the wheel e2e jobs, which take their matrix from filter_cuda_matrix.py. cuda-windows.yml now uses it too, with a hard-coded matrix.

Recommendation

Request Changes. Finding 1 is a small if: change, and the expiring cu134 pin makes it likely to trigger soon. Finding 2 adds one path in three places. The rest are up to the author.

Review steps
  • Gather context (diff, related workflows, scripts)
  • Run three independent reviewers
  • Consolidate findings
  • Fact-check findings
  • Post review

- test-model-cuda-windows-e2e no longer gates on the export matrix's overall
  result, so one train failing its export (say an expired torch pin) does not
  skip every row. Each row fails on its own missing artifact; a skipped export
  (fork pull request) still skips this.
- The e2e job puts CUDA_HOME\bin\x64 ahead of bin, as the runtime and wheel
  e2e scripts do, so the 13.4 row loads the 13.4 runtime.
- Trigger on install_cuda_redist.py, which the export now runs.
- Add parakeet-tdt int4 on 13.4: each train also installs its own torch, and
  the int4 kernels only run for a quantized model.
- Use cuda-version as the matrix key, as cuda.yml and the wheel workflow do.
- The tests read the matrix key from gpu-arch-version, check that the jobs use
  the train (gpu-arch-version, CUDA\v path, bin\x64), that every model runs on
  the image's train, and that the Windows cudart is fetched.
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Addressed in a2e42ab:

  1. e2e gate: now !cancelled() && needs.export-model-cuda-windows-artifact.result != 'skipped', as in build-wheels-cuda-windows.yml. A failed 13.4 export no longer skips the 13.2 rows; each row fails on its own missing artifact. A fork PR, where the export is skipped, still skips this job.
  2. Path filters: .ci/scripts/wheel/install_cuda_redist.py added to pull_request.paths and to both contains() lists.
  3. PATH: the e2e job puts CUDA_HOME\bin\x64 ahead of bin, as test_cuda_runtime_windows.ps1 and test_cuda_windows_e2e.ps1 do.
  4. 13.4 scope: added nvidia/parakeet-tdt quantized-int4-weight-only on 13.4, since each train installs its own torch and only a quantized model runs the int4 kernels. Also rewrote the comment so it doesn't assume exactly two trains.
  5. Tests check the wiring:
    • _matrix_cuda_versions reads the key named in gpu-arch-version.
    • The Windows jobs must use ${{ matrix.cuda-version }} for gpu-arch-version and the CUDA\v path.
    • The e2e PATH must include bin\x64.
    • Every model must run on the image's train.
    • The export must fetch the Windows cudart and export WINDOWS_CUDA_HOME.
  6. New checks fail on the regressions they guard: I reintroduced each one locally (gate back to == 'success', runtime pinned to v13.2, gpu-arch-version: "13.2", no bin\x64, no Windows cudart fetch, image bumped to 13.4), and every one makes a test fail.

Also:

  • The matrix key is now cuda-version, matching cuda.yml and the wheel workflow.
  • The version/name rebinding in the tests is gone.
  • install_cuda_redist.py's comments now name cuda-windows.yml as a consumer.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 32m 19s —— View job


Request Changes: the CUDA 13.4 int4 row added in a2e42ab (parakeet-tdt quantized-int4-weight-only, which my previous review wrongly suggested) never calls _weight_int4pack_mm, so no int4 kernel runs on 13.4 even though the new matrix comment says one does; the other findings are minor.

Detailed review

PR Review: #25307

Summary

The PR runs the Windows CUDA export, e2e and runtime-test jobs on both 13.2 and 13.4. The export job fetches the 13.4 toolkits itself when the image's nvcc doesn't match, and test_cuda_workflow.py now checks that wiring. Problems found in a2e42ab:

  • The new 13.4 int4 row doesn't run the int4 kernel it was added to cover.
  • The runtime job may install torch for the wrong train (low confidence).
  • One assertion is duplicated, one comment is worded wrongly, and the PR description doesn't match the diff.

Infrastructure

The runtime job installs torch before it puts the row's toolkit on PATH (.github/workflows/cuda-windows.yml:328-330)

The two jobs set up PATH differently:

  • e2e: puts CUDA_HOME\bin\x64;CUDA_HOME\bin first on PATH before running setup-windows.ps1 (:283).
  • runtime: sets only CUDA_HOME and CUDA_PATH. .ci/scripts/test_cuda_runtime_windows.ps1:27 fixes PATH, but only after setup has run.

_selected_nvcc checks PATH before CUDA_PATH (install_utils.py:213-220). So on a row whose train isn't the runner's default nvcc, install_executorch.bat installs the other train's torch. The C++ tests are then built with the row's nvcc (test_cuda_runtime_windows.ps1:24-27, 45-46), while find_package_torch() (backends/cuda/CMakeLists.txt:214) finds the other train's torch.

Low confidence: I couldn't read test-infra's windows_job.yml to check whether gpu-arch-version already reorders PATH. Copying line 283 into this job settles it either way.

Related: the e2e PATH assertion at .ci/scripts/tests/test_cuda_workflow.py:111-115 is repeated word for word at :116-120. Pointing the second copy at this job would test the fix. Fix this →

Testing

The 13.4 int4 row never calls _weight_int4pack_mm (.github/workflows/cuda-windows.yml:111-121, :253-263)

The new comment says this row "runs the int4 kernels the non-quantized model does not". It doesn't:

  • quantized-int4-weight-only passes --qlinear_encoder 4w with no packing format (.ci/scripts/export_model_artifact.sh:303).
  • extension/llm/export/quantize.py:50-54 turns that into IntxWeightOnlyConfig(weight_dtype=torch.int4), not Int4WeightOnlyConfig. The encoder's linears become IntxUnpackedToInt8Tensor, which lowers to a dequantize followed by a plain matmul.
  • The Triton int4 path (pack_int4_weights_for_aoti) is used only by Voxtral Realtime, and only on ROCm (examples/models/voxtral_realtime/export_voxtral_rt.py:758).

So no 13.4 job runs backends/cuda/runtime/shims/int4mm.cu (compiled by the 13.4 nvcc) or any Triton int4 kernel. .ci/scripts/test_cuda_runtime_windows.ps1:32-42 has no int4 test either. My previous review gave this exact row as its example, and that example was wrong.

quantized-int4-tile-packed is the quant that selects Int4WeightOnlyConfig with tile_packed_to_4d (export_model_artifact.sh:296), which lowers to _weight_int4pack_mm. Options:

  • nvidia/parakeet-tdt quantized-int4-tile-packed on 13.4. It runs on Linux (.github/workflows/cuda.yml:991-994 excludes only parakeet's weight-only row) but has no Windows row yet. .ci/scripts/tests/test_cuda_workflow.py:147-154 then needs a 13.2 row for it too.
  • mistralai/Voxtral-Mini-4B-Realtime-2602 quantized-int4-tile-packed vr-offline on 13.4. It already runs on Windows with 13.2 and passes that test as it is, but it is a much larger export.
  • Keep the row and reword both comments. Say it covers an int4 weight-only export under the 13.4 torch, not the int4 kernels.

Fix this →

Specific Comments

  • .github/workflows/cuda-windows.yml:199: the comment says "Not gated on the export's result", but line 207 does check that result (!= 'skipped'). "Not gated on the export succeeding" says what is meant.
  • PR description: it becomes the squash commit message but still describes 494823e:
    • The matrix key is cuda-version, not cuda.
    • The "13.4 scope" bullet says 13.4 runs on dinov2 only, and still says only the toolkit differs between trains. The workflow comment no longer makes that claim.
    • It says the jobs "together" cover every train; the test now requires each job to cover every train.
    • "53 passed" is from before a2e42ab changed the tests.

Recommendation

Request Changes. The Testing finding leaves open the coverage gap that the row was added to close, and the comment says the gap is closed. Swapping the row or rewording the comment resolves it. The Infrastructure finding is low confidence, and the rest is optional.

Review steps
  • Gather context (diff, a2e42ab, related scripts)
  • Run three independent reviewers
  • Consolidate findings
  • Fact-check findings
  • Post review

Not run: python and pytest, which this action's --allowedTools blocks, and fetching test-infra's windows_job.yml, which needs WebFetch. This review comes from reading the code. This PR's CI was still running when it was posted.

- The 13.4 int4 row is Voxtral Realtime quantized-int4-tile-packed, the quant
  that lowers to _weight_int4pack_mm; quantized-int4-weight-only lowers to a
  dequantize plus matmul and never ran the int4mm shim. It already runs on
  13.2 on Windows, so the row differs only in its train.
- unittest-cuda-runtime-windows puts the toolkit on PATH before setup, as the
  e2e job does, so setup sees the row's nvcc.
- The tests check every train has a tile-packed row, check the PATH line for
  both Windows jobs and that it precedes setup, and lose the duplicated
  assertion.
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Addressed in 1c3cfb3:

  • The 13.4 int4 row didn't run an int4 kernel: confirmed. quantized-int4-weight-only passes no packing format, so quantize.py picks IntxWeightOnlyConfig, which lowers to dequantize plus matmul. The 13.4 int4 row is now Voxtral Realtime quantized-int4-tile-packed (vr-offline). That is the quant that selects Int4WeightOnlyConfig(tile_packed_to_4d) and lowers to _weight_int4pack_mm, so it runs the int4mm.cu shim built by the 13.4 nvcc. It already runs on Windows on 13.2 (its export takes ~29 min), so the 13.4 row differs only in its train. I picked it over parakeet tile-packed, which has no Windows row yet. A test now requires a tile-packed row on every train.
  • Runtime job's PATH: the runtime job now puts CUDA_HOME\bin\x64;CUDA_HOME\bin first on PATH before setup-windows.ps1, like e2e. The test checks this for both jobs and that the line comes before setup. In practice this job installs CPU torch (torch==2.14.0 from whl/test/cpu on the previous head), and its 13.4 run already built with v13.4\bin\nvcc.exe. So nothing was mis-built, but the two jobs now agree.
  • Duplicated assertion: removed.
  • Gate comment: now says "not gated on the export succeeding".
  • PR description: rewritten to match the current diff (cuda-version key, the two 13.4 rows, per-job coverage, 54 tests).

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Addressed in fe17538:

  • Land after Use torch 2.15.0 for every CUDA train #25284: agreed. The description now says so up front, in place of "independent of either": the 13.4 rows install the 2026-09-21 cu134 nightly, which expires around 2026-11-20, and Use torch 2.15.0 for every CUDA train #25284 replaces it with the 2.15.0 release. Once Use torch 2.15.0 for every CUDA train #25284 lands I'll rebase this PR onto it and re-check the 13.4 rows on the released torch.
  • Check names: confirmed from the checks list, where both 13.2 and 13.4 Voxtral Realtime exports render as ...quantized-int4-tile... / linux-job. Both jobs now name themselves <job>-${{ matrix.cuda-version }} (${{ matrix.model_name }}, ${{ matrix.quant }}). With the runner suffix, the longest rendered names are 111 characters (export) and 106 (e2e). Every row is still unique after the 100-character cut, because the CUDA version comes before the cut point. test_windows_jobs_cover_every_supported_train now asserts that each name starts with <job>-${{ matrix.cuda-version }}.
  • Path filters: install_requirements.py and install_utils.py are added to pull_request.paths and to both contains() lists.
  • e2e comments: the e2e matrix comment now points at the export's rationale, and gpu-arch-version is described as the export's CUDA version, not its toolkit.
  • PR description: "six rows (four models)".

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 23m 51s —— View job


Request Changes: the new CUDA 13.4 Voxtral Realtime tile-packed export fails at the MinGW link on every run, because the torch 2.15 AOTI wrapper calls aoti_torch_dtype_element_size and aoti_cuda_shims neither defines nor exports it. The shim has to land first, or the row has to go.

Detailed review

PR Review: #25307

Summary

The PR runs the Windows CUDA export, e2e and runtime jobs on both 13.2 and 13.4. The image lacks the 13.4 toolkits, so the export fetches them with install_cuda_redist.py.

Two problems:

  • One new row fails on every run, and while it fails, artifact cleanup never runs.
  • The runtime job ignores two of the new trigger paths.

All three changed files were reviewed. When this review was written, two runs hadn't finished: the current head's run (44618301389) and the 13.4 dinov2 e2e in 44611792996.

Infrastructure

1. The 13.4 Voxtral Realtime tile-packed row can't link under torch 2.15 (blocker).

The row is at cuda-windows.yml:122-126, mirrored in e2e at :265-269. In run 44611792996, its export (job 135870453102) ran with nvcc 13.4.59 and torch==2.15.0.dev20260921+cu134. It failed at the final x86_64-w64-mingw32-g++ link of the wrapper against aoti_cuda_shims.lib:

torch/csrc/inductor/aoti_runtime/utils.h:490: undefined reference to `aoti_torch_dtype_element_size'

Cause:

  • The torch 2.15 wrapper's assert_alignment calls aoti_torch_dtype_element_size.
  • It is only defined in the ETensor common shims (backends/aoti/common_shims.cpp:218).
  • aoti_cuda_shims.dll is built from the SlimTensor shims (common_shims_slim.h/.cpp), which don't have it.
  • The checked-in backends/cuda/runtime/aoti_cuda_shims.lib doesn't export it either.

This is a torch gap, not a CUDA 13.4 one: in the same run, the 13.2 copy of the row passed on torch 2.14.0. An earlier round of this review suggested this row and missed the gap.

Consequences:

Fix:

  1. Declare and define aoti_torch_dtype_element_size in common_shims_slim.h/.cpp. The slim c10::elementSize that slim_tensor.h:108 uses covers it.
  2. Regenerate aoti_cuda_shims.lib with the recipe in aoti_cuda_shims.lib.md.

Both steps are needed: per aoti_cuda_shims.lib.md:34-36, a name the DLL doesn't define links and then fails to load. Making the change in this PR lets the 13.4 row test it.

Otherwise, drop the 13.4 Voxtral rows and the per-train tile-packed assertion (test_cuda_workflow.py:123-132) until the shim lands.

Fix this → · Fix this (drop the row) →

2. unittest-cuda-runtime-windows ignores the two new install paths (minor).

The PR adds install_requirements.py and install_utils.py to:

  • the workflow's paths (:23-24);
  • the export gate;
  • the e2e gate (:221-222).

It doesn't add them to the runtime job's gate (:307-315), even though that job builds through both files:

  • setup-windows.ps1:34 calls install_executorch.bat, which runs install_executorch.py, and that imports both files (:18, :22).
  • setup.py auto-enables the CUDA build when nvcc's version is in install_utils.SUPPORTED_CUDA_VERSIONS.

So a PR that only touches install_utils.py, for example to edit that list, runs export and e2e but skips both runtime trains.

Fix this →

Specific Comments

  • .github/workflows/cuda-windows.yml:260: "The export matrix's rows, which runs one model per row; see there for why." is ungrammatical. Suggested wording: "Same rows as the export matrix; see there for why."
  • .ci/scripts/tests/test_cuda_workflow.py:102-103: the comment says the train goes first, but assertIn passes wherever the train appears in the name. assertTrue(job["name"].startswith(...)) would check what the comment says.

Recommendation

Request Changes.

  • Finding 1 makes a new row fail on every run, and each run keeps ~18 GB of artifacts until it's fixed. Add the shim and regenerate the .lib, or drop the row.
  • Finding 2 needs two more lines in the runtime job's gate.

Review steps

  • Load /pr-review and its checklist
  • Diff vs origin/main (base 5e70618): 3 files, +201/−32, all reviewed
  • Three independent reviewers (all three flagged finding 1)
  • Consolidate, then fact-check against the code, the checked-in .lib and the CI logs
  • Run pytest .ci/scripts/tests/test_cuda_workflow.py: not run, because the command needs approval in this job
  • Read Use torch 2.15.0 for every CUDA train #25284: not read, because fetching it wasn't permitted

…artswith

- unittest-cuda-runtime-windows also runs on install_requirements.py and
  install_utils.py: it builds through install_executorch.py, and setup.py
  enables CUDA from SUPPORTED_CUDA_VERSIONS.
- The name test checks the train comes first, as its comment says.
- Reword the e2e matrix comment.
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Addressed in fec953f:

  • 13.4 Voxtral Realtime link failure: confirmed. Job 135870453102 has exactly one undefined reference to 'aoti_torch_dtype_element_size', on torch==2.15.0.dev20260921+cu134 with nvcc 13.4. Use torch 2.15.0 for every CUDA train #25284 already adds that shim to common_shims_slim.h/.cpp and regenerates aoti_cuda_shims.lib with the export. On Use torch 2.15.0 for every CUDA train #25284, the same Windows tile-packed export passed on torch 2.15.0+cu132 (job 135869521002), so the 13.2 row won't regress either. I'm keeping the row rather than dropping it. This PR already has to land after Use torch 2.15.0 for every CUDA train #25284, as the description says, and once rebased on it the row tests the shim on 13.4. Until Use torch 2.15.0 for every CUDA train #25284 lands, this row is expected to be red.
  • Runtime job paths: install_requirements.py and install_utils.py are added to unittest-cuda-runtime-windows's if:.
  • Name test: now startswith(f"{name}-${{ matrix.cuda-version }}").
  • e2e comment: now "Same rows as the export matrix; see there for why."

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 29m 26s —— View job


Request Changes, for the landing order only: the new 13.4 Voxtral Realtime tile-packed row has failed its export in every run so far and can't pass until #25284 lands. The rest is two test gaps, one low-confidence cleanup and some comment wording.

Detailed review

PR Review: #25307

Summary

The PR runs cuda-windows.yml's export, e2e and runtime jobs on every train in SUPPORTED_CUDA_VERSIONS:

  • the 13.4 rows fetch their own toolkits with install_cuda_redist.py;
  • each check name now includes the train;
  • the e2e gate is per row;
  • test_cuda_workflow.py gets tests for the new wiring.

Nothing in the diff breaks a job at fec953f. But one of the new 13.4 rows can't pass until #25284 lands, two of the new tests miss the regressions they guard against, and three comments need rewording.

What this review did not cover:

  • test_cuda_workflow.py was not run; pytest needs approval in this job.
  • test-infra's linux_job_v3.yml and windows_job.yml were not read; they aren't in the checkout.
  • Use torch 2.15.0 for every CUDA train #25284 was not read; it can't be fetched here.

Infrastructure

Land after #25284. Until then the new 13.4 Voxtral Realtime row stays red, and each run keeps about 21 GB of artifacts (cuda-windows.yml:122-126, :265-269, :351)

  • The row's export failed in both earlier runs, and each time its e2e then failed at the download:
  • The current run 44624449486 at fec953f hasn't finished that row. Nothing in fec953f supplies aoti_torch_dtype_element_size, so it should fail the same way.
  • At fec953f neither 13.4 e2e row has run yet. The 13.4 dinov2 e2e passed once, before the rename (job 135894134749). The 13.4 Voxtral e2e has never got past the download.
  • A failed e2e row skips delete-model-cuda-windows-artifacts, as happened in run 44611792996. That keeps the run's exports for the repository retention window, about 21 GB by the job's own comment. The skip is intentional, but while this row is red every CUDA-touching push and main commit pays for it. The landing order is enforced only by the description.
  • The cu134 pin these rows install is 59 days old today: torch==2.15.0.dev20260921+cu134 at install_requirements.py:34. Its comment gives download.pytorch.org about 60 days of retention. Once the pin ages out, both 13.4 exports fail at install_executorch.sh whatever Use torch 2.15.0 for every CUDA train #25284 does, so check that it still resolves after the rebase.
  • Fix:
    1. Rebase onto Use torch 2.15.0 for every CUDA train #25284 and get both 13.4 rows green before merging.
    2. Then update the description: drop the "Land after Use torch 2.15.0 for every CUDA train #25284" and pin-expiry preamble, and add the 13.4 Voxtral result to the Test plan. The plan currently leaves out the failure.

Testing

The check-name test passes for the name the rename was meant to fix (test_cuda_workflow.py:102-106)

  • It checks only the prefix, so name: export-model-cuda-windows-artifact-${{ matrix.cuda-version }} passes.
  • GitHub stops appending matrix values once a name contains an expression; see unittest-cuda-runtime-windows-13.2 in run 44624449486. With that name, all six 13.2 exports would share the check name export-model-cuda-windows-artifact-13.2 / linux-job.
  • Rows that differ only in mode would collide the same way, because mode isn't in the name.
  • The 100-character cap isn't checked either:
    • The longest name is 99 characters: export-model-cuda-windows-artifact-13.4 (Voxtral-Mini-4B-Realtime-2602, quantized-int4-tile-packed).
    • Run 44611792996 shows what happens to longer names: both Voxtral Realtime exports became …quantized-int4-tile... / linux-job.
  • Fix: substitute each row's values into the name template, then assert that the rendered names are unique and at most 100 characters. Fix this →

_matrix_cuda_versions ignores exclude, and also ignores include when the train is an axis (test_cuda_workflow.py:36-39)

  • exclude: [{cuda-version: "13.4"}] on unittest-cuda-runtime-windows drops the 13.4 runtime job, but test_windows_jobs_cover_every_supported_train still passes.
  • include: [{cuda-version: "12.8"}] next to the axis adds a 12.8 job that test_every_cuda_job_uses_a_supported_version never checks.
  • No workflow does either today. cuda.yml's only exclude: (line 935) is on a job with a fixed gpu-arch-version.
  • Fix: take the axis values, add the include values for the key, and remove the exclude values. Fix this →

Code Quality

Low confidence: the two copies of the toolkit switch differ in LD_LIBRARY_PATH (cuda-windows.yml:169 vs build-wheels-cuda-windows.yml:199-209)

  • The blocks are otherwise identical line for line, but only this one prepends the redist lib64 to LD_LIBRARY_PATH.
  • If it's needed, for example to keep the image's 13.2 libraries from shadowing the ones the cu134 torch expects, the wheel pipeline needs it too.
  • If it isn't needed, it's one more way the 13.4 export differs from the 13.2 one.
  • Fix: drop it, or add a one-line comment saying what needs it and add the same export to the wheel pipeline. Fix this →

Specific Comments

  • .github/workflows/cuda-windows.yml:115-117:
    • The "so" links the per-train toolkit and torch to the choice of two models. But they're the reason each train needs rows at all, and the subset is a separate choice.
    • "Its own torch" also holds only for the export. On Windows, determine_torch_url always picks the CPU index (install_utils.py:368-372).
    • Suggested wording: "Each train installs its own toolkit and, for the export, its own torch, so each needs rows of its own; the other trains get only the smallest model and the tile-packed int4 model, …"
  • .ci/scripts/tests/test_cuda_workflow.py:116: "keep their DLLs in bin\x64, ahead of bin" reads as if the DLLs sit ahead of bin. The point is that bin\x64 goes ahead of bin on PATH.
  • .ci/scripts/tests/test_cuda_workflow.py:137: the gate is per row, not per train. test_one_failed_export_row_does_not_skip_the_others would match the comment at cuda-windows.yml:204-208.

Fix this →

Recommendation

Request Changes, only for the landing order. Rebase onto #25284 and show the 13.4 Voxtral Realtime export and e2e passing. Nothing else blocks.

…LIBRARY_PATH

- The test renders each row's check name from the template and requires the
  names to be unique and at most 100 characters. An explicit name stops GitHub
  appending matrix values, so a prefix check alone let six rows share a name.
- _matrix_cuda_versions adds include values to the axis and removes exclude
  values, so an exclude that drops a train fails the coverage test.
- Drop the LD_LIBRARY_PATH export from the toolkit switch, matching the wheel
  pipeline's copy: nvcc finds its own libraries, and the export links nothing
  against the Linux runtime.
- Comment wording: bin\x64 goes ahead of bin on PATH; only the export has its
  own torch; the gate test is named per row.
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Addressed in a874e5e:

  • Check-name test: the test now renders each include row's name from the template and asserts that the names are unique and at most 100 characters. Checked by reintroducing each regression:
    • name: export-model-cuda-windows-artifact-${{ matrix.cuda-version }} (six 13.2 rows collide) fails;
    • a name over 100 characters fails;
    • mode isn't in the name, but no two rows differ only by mode, and the uniqueness check would catch a pair that did.
  • _matrix_cuda_versions: it now takes the axis values, adds the include values for the key and removes the exclude values. exclude: [{cuda-version: "13.4"}] on the runtime job now fails test_windows_jobs_cover_every_supported_train.
  • LD_LIBRARY_PATH: dropped, to match build-wheels-cuda-windows.yml. nvcc finds its own libraries, and the export links only against the Windows cudart, so nothing loads the redist's Linux lib64.
  • Comments:
    • the matrix comment uses your wording ("…and, for the export, its own torch, so each needs rows of its own…");
    • the test comment now says bin\x64 goes ahead of bin on PATH;
    • the gate test is renamed test_one_failed_export_row_does_not_skip_the_others.

Landing order: agreed, and unchanged. The 13.4 Voxtral Realtime export failed again in run 44624449486, on the same single undefined reference to 'aoti_torch_dtype_element_size'. Once #25284 lands I'll rebase onto it, which also drops the cu134 nightly pin you flagged. I'll get both 13.4 rows green and update the description and Test plan before asking to merge.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 28m 50s —— View job


Request Changes, only for the landing order you already agreed to: the 13.4 Voxtral Realtime tile-packed export still fails without #25284's aoti_torch_dtype_element_size shim, so its Windows e2e has never run, and the rest is a minor test-helper regression from a874e5e plus some wording.

Detailed review

PR Review: #25307

Summary

This PR adds CUDA 13.4 rows to the Windows CUDA export, e2e and runtime jobs, puts the CUDA version in each job name, and adds static tests for the matrices. The landing order you agreed to is the only blocker. a874e5e also adds a minor regression to the test's matrix helper.

Infrastructure

  • Blocker, still open: land after Use torch 2.15.0 for every CUDA train #25284, then rebase.
    • The 13.4 Voxtral Realtime export failed at "Run script" in all three finished runs: 44611792996, 44618301389 and 44624449486. Its e2e row failed at the download each time.
    • So the 13.4 tile-packed row has never reached Windows (cuda-windows.yml:123-127, :265-269), though its whole purpose is to run the int4mm shim there. Only the 13.4 dinov2 rows have passed end to end: 135894134749 and 135912213245.
    • This is time-sensitive. The PR description says the cu134 nightly pinned at install_requirements.py:30-37 leaves download.pytorch.org around 2026-11-20, which is tomorrow. After that, both 13.4 exports will fail at install_executorch.sh on this branch until the rebase.
    • Storage question: delete-model-cuda-windows-artifacts has no if: (:343-351), so it is skipped whenever any row fails. Each of those runs keeps most of the ~21 GB its comment estimates for the whole retention window, and re-running can't fix this failure.
      • Could you delete those runs' artifacts from their summary pages?
      • Run 44632959026 will most likely add another set, since nothing in a874e5e adds the shim.

Testing

  • Minor regression from a874e5e: _matrix_cuda_versions (test_cuda_workflow.py:36-39) can drop versions that GitHub runs.

    • Line 38 treats a version as excluded whenever any exclude row names it. GitHub only drops combinations that match every key of the row, so {cuda-version: "13.4", os: windows} would make the helper skip 13.4 everywhere.
    • Line 39 also filters the include values, but GitHub applies include after exclude.
    • Both cases make test_every_cuda_job_uses_a_supported_version silently skip a version that runs.
    • fec953f's isinstance assert is gone. A ${{ fromJSON(...) }} axis is now split into characters and fails as '$' not found in [...].
    • No current job hits any of these. The cause is the round-5 review's "axis + include − exclude" suggestion, which was too loose. A fix:
      axis = matrix.get(key, [])
      assert isinstance(axis, list), axis
      excluded = {row[key] for row in matrix.get("exclude", []) if row.keys() == {key}}
      versions = [version for version in axis if version not in excluded]
      return versions + [row[key] for row in matrix.get("include", []) if key in row]
      It ignores multi-key excludes. The worst that does is make the helper check a version that never runs; it never skips one that does.

    Fix this →

Specific Comments

  • .ci/scripts/tests/test_cuda_workflow.py:114: say "job names" here. GitHub cuts the job's own name, then appends / linux-job or / windows-job. Run 44632959026 shows 111-character check names in full.
  • .ci/scripts/tests/test_cuda_workflow.py:119-120: the old name: export-model-cuda-windows-artifact was explicit too, and GitHub still appended the values. What stops it is a name that uses a matrix expression.
  • .ci/scripts/wheel/install_cuda_redist.py:10-11: the paragraph wasn't rewrapped, so line 11 is about 109 characters while the rest are about 90.
  • .github/workflows/cuda-windows.yml:176: "Verify pre-installed dependencies" is no longer accurate for the 13.4 rows, whose nvcc and WINDOWS_CUDA_HOME come from the block just above.

Fix this →

Recommendation

Request Changes. Only the landing order blocks this, and your plan covers it: rebase onto #25284, get both 13.4 rows green, then update the description and Test plan. Send the follow-ups with the rebase push. Any push touching cuda-windows.yml before then keeps another set of exports.

Review steps
  • Three independent reviews of a874e5e against base 5e70618, consolidated and fact-checked
  • Checked job results in runs 44611792996, 44618301389, 44624449486 and 44632959026
  • Not run: .ci/scripts/tests/test_cuda_workflow.py, because pytest needs approval in this mode
  • Not read: test-infra's linux_job_v3.yml and windows_job.yml, which aren't in the checkout
  • Not read: Use torch 2.15.0 for every CUDA train #25284, because only git is available here. Its contents come from this PR's description and discussion.
  • Not finished: run 44632959026 was still running, including its 13.4 Voxtral export (135944433371)

This branch was previously deployed

1 inactive deployment
cadence — a874e5ec Deployed Nov 19, 2026 by Gasoonjia via hifi-op-test / hifi4 #39538
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant