Skip to content

Use torch 2.15.0 for every CUDA train - #25284

Open
Gasoonjia wants to merge 9 commits into
mainfrom
gasoonjia/torch-2.15
Open

Gasoonjia wants to merge 9 commits into
mainfrom
gasoonjia/torch-2.15

Conversation

@Gasoonjia

@Gasoonjia Gasoonjia commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #25255 (drop CUDA 13.0).

Why

The cu134 train installs a torch 2.15.0.dev20260921 nightly. Its oldest wheels are a few days from expiring off the nightly index, and once they go, every cu134 job fails to resolve. cu132 and CPU are still on torch 2.14.0. torch 2.15.0 is released (2026-10-28) and is on the test index for cpu, cu132 and cu134, so every train can use one release.

What

One set of pins (install_requirements.py, torch_pin.py)

  • torch 2.15.0, torchvision 0.29.2 and torchaudio 2.11.1 for CPU, cu132 and cu134, read from torch_pin.py. torchvision 0.29.2 accepts torch>=2.14.0 and torchaudio 2.11.1 accepts torch>=2.11. All three wheels are on test/{cpu,cu132,cu134}.
  • CU134_TORCH_PACKAGES, cu134_requirements() and the cu134 special cases in install_requirements.py are removed.
  • ROCm stays on torch 2.14.0 (ROCM_TORCH_VERSION) because the rocm7.2 index has no 2.15.0 wheel.
  • EXECUTORCH_USE_RELEASE_CPU_DEPS still pairs released torch with torchao 0.18.0. I checked locally that 0.18.0 loads and quantize_ runs on torch 2.15.0+cpu.

FLA kernels (backends/cuda/triton/kernels/chunk_gated_delta_rule.py)
torch 2.15 requires triton 3.9. Every flash-linear-attention release before 0.6.0 fails to import under triton 3.9 (ValueError: key contains names that are not kernel arguments: STAGE), which broke test-cuda-shims on the first version of #25255. The wrapper is ported to the fla-core 0.6.0 kernel signatures:

  • the gating kernels now use exp2, so the chunk cumsum is scaled by RCP_LN2;
  • the h and o kernels take HV and STATE_V_FIRST, and fold the V-block index into grid axis 0;
  • the cumsum kernel uses a flattened grid and takes USE_GRAPH instead of HEAD_FIRST;
  • the kkt and w/u kernels take HV.

CI and the qwen3.5-MoE requirements install fla-core==0.6.0 rather than the flash-linear-attention meta package, which now pins transformers>=5.6.2.

Windows CUDA shims (backends/aoti/common_shims_slim.*, aoti_cuda_shims.lib)
torch 2.15's AOTI runtime calls aoti_torch_dtype_element_size from assert_alignment. The Windows DLL is built from the SlimTensor shims, which did not define it, so a model compiled with 2.15 could not load. The shim is added there, and the import library is regenerated per aoti_cuda_shims.lib.md with mingw-w64 binutils 2.40. Rebuilding the old export list with 2.40 reproduces the checked-in file byte for byte, and the new library differs only by this one export. The .md now notes that binutils 2.46 numbers import hints differently and does not reproduce the file.

Pin readers

  • The ESP-IDF generator reads TORCH_VERSION and TORCH_URL_BASE by importing install_requirements.py instead of sed-scraping its text. torch_pin.py is added to its stamp inputs, to test_version_stamp.sh and to the test-espressif path filter, and the negative test now corrupts torch_pin.py.
  • windows-msvc.yml reads the torch version from torch_pin.py.
  • Every workflow path filter and changed-files check that names install_requirements.py now also names torch_pin.py.

Test plan

  • pytest .ci/scripts/tests/test_install_dependencies.py .ci/scripts/tests/test_cuda_workflow.py .ci/scripts/tests/test_filter_cuda_matrix.py: 50 passed. The ROCm test now runs each ROCm script's pin-read snippet and checks its output, and the source-build test checks that the core install also pins torchao==TORCHAO_NIGHTLY_VERSION.
  • examples/espressif/test_version_stamp.sh: 51 OK, 0 FAIL, including the new TORCH_VERSION must be X.Y.Z case.
  • In a venv with torch 2.15.0+cpu, triton 3.9.0 and fla-core 0.6.0, chunk_gated_delta_rule.py imports and registers torch.ops.triton.chunk_gated_delta_rule. A script also checked each of the 6 launched kernels: the keyword arguments passed match the 0.6.0 kernel parameters, minus the ones set by autotune configs or defaults, with no missing, extra or conflicting names. None of the 0.6.0 autotuners set the pre/post hooks that wrap_triton rejects.
  • Built common_shims_slim.cpp and ran a small check of aoti_torch_dtype_element_size for every dtype constant, plus ComplexFloat, which returns 0. The new gtest cases cover the same.
  • Not verified locally (no GPU or Windows here), left to CI: the numerics of the ported kernels (test_chunk_gated_delta_rule.py compares against FLA's naive reference and FLA's own chunk implementation) and the Windows CUDA e2e load.

@pytorch-bot

pytorch-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25284

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 Pending, 2 Unrelated Failures, 4 Unclassified Failures

As of commit 32f947b with merge base 679e49e (image):

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

FLAKY - The following job failed but was likely due to flakiness present on trunk:

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 19, 2026
@Gasoonjia
Gasoonjia deployed to upload-benchmark-results November 19, 2026 11:05 — with GitHub Actions Active
Base automatically changed from gasoonjia/drop-cuda-13.0 to main November 19, 2026 18:01
@github-actions github-actions Bot added the ciflow/docker Trigger docker-builds to rebuild and push the CI images for this PR label Nov 19, 2026
The cu134 train ran a torch 2.15 nightly whose oldest wheels are days from
expiring off the nightly index, while cu132 and CPU used torch 2.14.0.
torch 2.15.0 is released and on the test index for cpu, cu132 and cu134, so
every train now installs the same torch, torchvision 0.29.2 and torchaudio
2.11.1, read from torch_pin.py.

ROCm keeps torch 2.14.0 because rocm7.2 has no 2.15.0 wheel yet.

The ESP-IDF generator reads the pin through install_requirements.py instead
of scraping its text, and every workflow that triggers on
install_requirements.py also triggers on torch_pin.py.
torch 2.15 requires triton 3.9, and every flash-linear-attention release
before 0.6.0 fails to import under it (an autotune key names a removed
kernel argument). 0.6.0 changed the kernels this op launches:

- the gating kernels use exp2, so the chunk cumsum is scaled by 1/ln 2;
- the h and o kernels take HV and STATE_V_FIRST and flatten V blocks into
  grid axis 0;
- the cumsum kernel flattens its grid and takes USE_GRAPH instead of
  HEAD_FIRST;
- the kkt and w/u kernels take HV.

Only fla-core is installed: it holds the kernels, and the
flash-linear-attention meta package now pins transformers>=5.6.2.
torch 2.15's AOTI runtime checks input alignment through
aoti_torch_dtype_element_size. The ETensor shims had it, but the Windows
DLL is built from the SlimTensor shims, so a model compiled with 2.15 could
not load. Add it there and to the import library.

The import library was regenerated per aoti_cuda_shims.lib.md with
mingw-w64 binutils 2.40, which rebuilds the old export list byte for byte;
the new one adds exactly this name.
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 31m 9s —— View job


Needs Discussion: nothing blocks moving every CUDA train to torch 2.15.0 and the FLA op to fla-core 0.6.0, but a leftover pre-0.6.0 FLA install now breaks importing the CUDA backend, and nobody has yet confirmed that torchvision 0.29.2 and torchaudio 2.11.1 work with torch 2.15.0.

Detailed review of #25284

Summary

This PR pins torch 2.15.0 from torch_pin.py for CPU and every CUDA train, while ROCm stays on 2.14.0. It also ports chunk_gated_delta_rule to fla-core 0.6.0 for triton 3.9, and adds aoti_torch_dtype_element_size to the SlimTensor shims and the Windows import library.

There are no blockers. The two points to settle are under Recommendation.

What I could not verify:

  • the regenerated aoti_cuda_shims.lib, because it is binary;
  • fla-core 0.6.0's kernel internals, because its source could not be fetched from this job;
  • whether the needed wheels exist on the package indexes.

The pull, trunk and Test CUDA Builds runs were still queued when I checked.

Code Quality

  • The install hint fla-core>=0.6.0 is wrong (backends/cuda/triton/kernels/chunk_gated_delta_rule.py:26, backends/cuda/tests/test_chunk_gated_delta_rule.py:10).
    • Pasted into a shell, the unquoted >=0.6.0 is a redirect. It installs the latest fla-core and writes a file named =0.6.0.
    • The wrapper passes 0.6.0's private kernel arguments (HV, STATE_V_FIRST, USE_GRAPH). A later fla-core can break it even with the operator quoted.
    • CI and examples/models/qwen3_5_moe/requirements.txt both pin ==0.6.0. Use pip install "fla-core==0.6.0".
    • Fix this →
  • The ESP-IDF stamp builds its own torch requirement (examples/espressif/build_idf_component.sh:264).
    • It writes torch==$TORCH_VERSION itself, so the requirement format now lives in two places: here and in TORCH_PACKAGE (install_requirements.py:111).
    • Suppose TORCH_PACKAGE goes back to a nightly pin, torch=={TORCH_VERSION}.{NIGHTLY_VERSION}, with TORCH_URL_BASE on /whl/nightly. The X.Y.Z check still passes, and the component quietly declares a torch==X.Y.Z that the nightly index does not have. The text scrape this PR removes would have failed loudly.
    • Fix: export the requirement string from install_requirements.py and print that here.
    • Fix this →
  • The new shim narrows the dtype before validating it (backends/aoti/common_shims_slim.cpp:188).
    • slim::c10::ScalarType is enum class : int8_t, so the cast truncates the value before isValidScalarType sees it.
    • aoti_torch_dtype_element_size(262) therefore returns 4 (Float), not the 0 that common_shims_slim.h:96 documents for an unsupported code.
    • AOTI only passes real dtype codes, so this is minor. Range-check dtype before the cast.
    • Fix this →

Infrastructure

  • Two workflows do not trigger on torch_pin.py.
    • .github/workflows/windows-msvc.yml:95 now reads the torch version from torch_pin.py. That file is missing from pull_request.paths (lines 11-14) and from the build-windows-arm64 if: (lines 65-69).
    • In cuda.yml, neither pull_request.paths (lines 24-50) nor any job if: lists torch_pin.py or install_requirements.py. That was already true on main, but after this PR torch_pin.py chooses the torch for every CUDA train.
    • So a PR that only bumps TORCH_VERSION runs neither the arm64 install nor the CUDA workflow. On main, they only run on sampled full runs.
    • This PR is covered only because it also touches .ci/scripts/** and backends/cuda/**.
    • Fix this →
  • Low confidence: torchvision and torchaudio may not match torch 2.15.0. I could not reach the package indexes, so this needs a check on your side.
    • TORCHVISION_VERSION = "0.29.2" (install_requirements.py:38) belongs to the torch 2.14 series; 0.30.x is the series for 2.15. Even if pip accepts the pair, that does not show torchvision._C, which was built against 2.14, loads on 2.15.0.
    • TORCHAUDIO_VERSION = "2.11.1" now also applies below Python 3.15, where main deliberately used 2.11.0.
    • Please confirm that install_executorch.sh --example on cpu, cu132 and cu134 keeps torch at 2.15.0 and can import torchvision.ops and torchaudio. Check with both the oldest and the newest supported Python.

Testing

  • The FLA test runs a single shape (backends/cuda/tests/test_chunk_gated_delta_rule.py:45).
    • Every test uses B, T, H, K, V = 1, 128, 4, 64, 64.
    • This PR rewrites the wrapper's grid math. The V blocks are folded into axis 0 of the h and o grids, and the cumsum grid becomes (NT * B * H,).
    • With B = 1 the B factor has no effect. With V = 64, cdiv(V, BV) can be 1, depending on which BV the autotuner picks. A grid that dropped either factor would still pass.
    • The test also never uses Qwen3.5's K = V = 128 (examples/models/qwen3_5_moe/model.py:43-44), or a T that is not a multiple of 64.
    • Add shapes such as (1, 128, 4, 128, 128) and (2, 100, 4, 128, 128) to test_eager_fla_configs.
    • Fix this →
  • The ROCm script test asserts a temporary state (.ci/scripts/tests/test_install_dependencies.py:204).
    • assertNotEqual(ROCM_TORCH_VERSION, TORCH_VERSION) will fail with no explanation once rocm7.2 gets 2.15.0 and the two pins are aligned.
    • Replace it with a skipTest that says the test needs the pins to differ.
    • Fix this →

Backward Compatibility

  • An old FLA install now breaks importing the CUDA backend (backends/cuda/triton/kernels/__init__.py:49).
    • The guard only catches ImportError. Per the PR description, FLA releases before 0.6.0 fail under triton 3.9 with ValueError: key contains names that are not kernel arguments: STAGE.
    • cuda_backend.py:61 imports this package. Take any environment that still has the old pin flash-linear-attention==0.4.2, from the previous qwen3_5_moe requirements and cuda.yml. Re-running install_executorch.sh brings in torch 2.15 and triton 3.9, and import executorch.backends.cuda.cuda_backend then fails.
    • That breaks every CUDA lowering, not just Qwen3.5.
    • Fix: check importlib.metadata.version("fla-core") before importing FLA, and raise an ImportError that names the required version. PackageNotFoundError is a subclass of ImportError, so the existing except already skips the op when fla-core is missing.
    • Fix this →

Specific Comments

  • backends/cuda/runtime/aoti_cuda_shims.lib.md:30 says binutils 2.40 and 2.41 reproduce the checked-in file byte for byte. The commit message only reports checking 2.40. Fix this →
  • install_requirements.py:36-39: the rocm7.2 comment sits directly above TORCHVISION_VERSION and TORCHAUDIO_VERSION, so those read like ROCm pins.
    • The commit message says the set is "read from torch_pin.py", but only torch is. The next TORCH_VERSION bump has to change these two by hand.
    • Moving them into torch_pin.py would clash with the weekly bump, which rewrites that file. A blank line plus a note that they must move with TORCH_VERSION is enough.
    • Fix this →

Recommendation

Needs Discussion. Two things should be settled before merging:

  1. Whether an old FLA install should stop CudaBackend from importing. The fix is a few lines.
  2. Whether the torchvision and torchaudio pins load on torch 2.15.0. The queued pull and trunk runs may answer this.

Everything else is a minor follow-up.

- chunk_gated_delta_rule raises ImportError when fla-core is older than 0.6.0,
  so the kernels package skips the op instead of older FLA's ValueError under
  triton 3.9 breaking import of the whole CUDA backend.
- aoti_torch_dtype_element_size range-checks the code before the int8_t cast.
- The FLA test also runs B=2, K=V=128 and T=100.
- Quote the fla-core install hint and pin it to 0.6.0.
- windows-msvc.yml triggers on torch_pin.py.
- The ROCm pin test skips rather than fails once the pins align.
- aoti_cuda_shims.lib.md names binutils 2.42, which was checked, not 2.41.
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 29m 34s —— View job


No blockers; six minor findings (two CI install lines that bypass their source of truth, two test gaps, a missing fla-core error that drops the install hint, and a possibly dropped transformers dependency), so approve once the queued CUDA GPU and Windows E2E jobs pass.

Detailed review of #25284 at 9b09091

Summary

This PR moves the CPU and CUDA installs to torch 2.15.0, read from torch_pin.py. ROCm stays on 2.14.0 through ROCM_TORCH_VERSION. It also ports chunk_gated_delta_rule to fla-core 0.6.0, because older FLA releases fail to import under triton 3.9.

I found no blockers. The findings below are minor, and two of them are low confidence.

Not verified:

  • the regenerated backends/cuda/runtime/aoti_cuda_shims.lib, which is a binary
  • fla-core 0.6.0's private kernel signatures and autotune configs (fla-core is not installed on this runner)
  • that every pinned wheel exists on the test and CUDA indexes

CI at the time of writing: 23 passed, 0 failed, 17 pending. The pending runs include Test CUDA Builds (still queued, and the only job that runs the FLA kernels on a GPU), Test CUDA Windows Export and E2E, Windows MSVC Build and Test ROCm AOTI.

Code Quality

  • backends/cuda/triton/kernels/chunk_gated_delta_rule.py:37: if fla-core is not installed, importlib.metadata.version("fla-core") raises PackageNotFoundError. That error is an ImportError, so the package __init__ still skips the kernel. The direct submodule imports this PR adds hit it too: examples/models/qwen3_5_moe/export.py:1320, inference.py:148 and the two Qwen tests. There it surfaces as No package metadata was found for fla-core, without the pip install "fla-core==0.6.0" hint. Catch importlib.metadata.PackageNotFoundError and raise the same ImportError with the hint.
    Fix this →

Infrastructure

  • .github/workflows/cuda.yml:630-632: fla-core is now pinned in two places: inline here, and in examples/models/qwen3_5_moe/requirements.txt:1, which .ci/scripts/export_model_artifact.sh:609 already installs from. If the next FLA bump only touches requirements.txt, the GPU kernel and Qwen tests keep testing the old version. The comment also says the install is for the kernel tests, but the Qwen tests at :648 now import the submodule directly and need it too. Use pip install -r examples/models/qwen3_5_moe/requirements.txt, and mention the Qwen tests in the comment.
    Fix this →

  • .github/workflows/windows-msvc.yml:98-99: the version now comes from torch_pin.py, but the index is still hard-coded to https://download.pytorch.org/whl/test/cpu.

    • When TORCH_URL_BASE (install_requirements.py:29) moves off whl/test, this arm64 job keeps installing from the test channel. examples/espressif/build_idf_component.sh:256 in this PR avoids that by reading both TORCH_URL_BASE and TORCH_VERSION from install_requirements.
    • This step also lacks $PSNativeCommandUseErrorActionPreference = $true, which the x64 job sets at line 60. If python -c fails, $torchVersion is empty and the step fails at pip on torch==, which hides the real import error.

    Fix this →

Testing

  • examples/espressif/build_idf_component.sh:256-264: the generator now stamps torch==$TORCH_VERSION from torch_pin.py, but no test checks the stamped value.

    • .ci/scripts/test_espressif_idf_component.sh:112 only matches torch==\d+\.\d+\.\d+\S*, which is looser than the generator's new X.Y.Z check. Line :114 checks the index against the real TORCH_URL_BASE, but nothing does the same for the version.
    • test_version_stamp.sh only covers the malformed-pin error.

    A stamp that drifts from the pin therefore still passes. Import TORCH_VERSION next to TORCH_URL_BASE at :110 and require the exact torch_requirement: torch=={TORCH_VERSION} line.
    Fix this →

  • backends/cuda/tests/test_chunk_gated_delta_rule.py:47-55 (low confidence: I couldn't read fla-core 0.6.0's autotune configs): the comment's "several V blocks" and "K != V catches a grid sized with the wrong head dimension" depend on the BV that autotune picks.

    • If chunk_fwd_kernel_o picks BV=128, every shape here (V ≤ 128) has a single V block.
    • For (2, 100, 4, 64, 128), cdiv(K, BV) == cdiv(V, BV) == 1, so a grid_o sized from K would still pass.

    A shape such as (2, 100, 4, 64, 256) gives at least two V blocks, and different K and V block counts, for any BV ≤ 128. _validate_inputs only limits K, so this shape is allowed.
    Fix this →

Backward Compatibility

  • examples/models/qwen3_5_moe/requirements.txt:1 (low confidence: I couldn't check the wheel metadata offline): as I understand it, flash-linear-attention is FLA's layers/models package and depends on transformers, while fla-core contains only the ops. If that's right, pip install -r requirements.txt no longer installs transformers.

    • run.py:118 imports transformers whenever it gets a tokenizer and a prompt.
    • Neither the README prerequisites (lines 19-25) nor the default install_executorch.sh (without --example) install it.

    Add transformers to requirements.txt or to the README prerequisites.
    Fix this →

Specific Comments

  • examples/models/qwen3_5_moe/export.py:1319, inference.py:147: "missing or too old" is inaccurate, because the guard rejects anything that isn't 0.6.x, including newer releases.
  • .ci/scripts/test-rocm-aoti.sh:141: still says "the optional flash-linear-attention package". The dependency is now fla-core.
  • .ci/scripts/test-rocm-aoti.sh:25, test-rocm-voxtral.sh:36: the shell TORCH_VERSION now holds ROCM_TORCH_VERSION, which is easy to mistake for torch_pin.TORCH_VERSION when grepping. Consider renaming it ROCM_TORCH_VERSION.

Fix the first two →

Recommendation

Approve. None of the findings block. The FLA port on a GPU and the regenerated Windows import library are only exercised by Test CUDA Builds and Test CUDA Windows Export and E2E, which hadn't finished, so merge once both are green.

Review steps
  • Collect the PR context and the diff against origin/main
  • Run three independent reviewers over every changed file
  • Merge duplicate findings
  • Re-read the cited code for each finding
  • Post the review

- A missing fla-core raises the same ImportError with the install hint,
  instead of PackageNotFoundError's bare message.
- cuda.yml installs fla-core from qwen3_5_moe/requirements.txt, the pin the
  Qwen3.5 export already uses, and says the Qwen3.5 tests need it too.
- windows-msvc.yml's arm64 step reads the torch index from install_requirements
  along with the version, fails on a bad import, and triggers on
  install_requirements.py.
- The ESP-IDF release check requires torch==TORCH_VERSION exactly.
- The K != V grid shape uses V = 256, which gives several V blocks and a V
  block count unlike K's for every BV fla-core 0.6.0 autotunes (at most 128).
- The Qwen3.5 README lists transformers for run.py: flash-linear-attention
  depended on it, fla-core does not.
- Comment wording (not 0.6.x; fla-core in test-rocm-aoti.sh).
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Addressed in 9dc9285:

  • Missing fla-core: PackageNotFoundError now raises the same ImportError with the hint (found none: pip install "fla-core==0.6.0"). With metadata hidden, the message is that one. 0.4.2 and a fake 0.7rc1 still give the hint, and 0.6.0 still imports.
  • fla-core pinned in two places: cuda.yml now runs pip install -r examples/models/qwen3_5_moe/requirements.txt. The comment says the Qwen3.5 tests need it too.
  • windows-msvc.yml arm64: the step reads TORCH_URL_BASE from install_requirements together with TORCH_VERSION. It sets $PSNativeCommandUseErrorActionPreference = $true and $ErrorActionPreference = 'Stop', and triggers on install_requirements.py.
  • ESP-IDF stamp: the release check now requires exactly torch_requirement: torch=={TORCH_VERSION}, imported from torch_pin.
  • K != V shape: changed to (2, 100, 4, 64, 256). fla-core 0.6.0 autotunes BV in {32, 64} for chunk_delta_h and up to 128 for chunk_o. So V = 256 always gives several V blocks, and a V block count different from K = 64's.
  • transformers: confirmed. flash-linear-attention==0.4.2 requires transformers, and fla-core==0.6.0 only needs einops plus torch/triton. install_executorch.sh installs requirements-examples.txt (transformers == 5.11.0) unless run with --minimal. So I didn't add a second, unpinned copy to requirements.txt. The README now lists it as a prerequisite for run.py and says where it comes from.
  • Wording: "missing or not 0.6.x", and test-rocm-aoti.sh now says fla-core.

Not changed: renaming the ROCm scripts' shell TORCH_VERSION. It's local to those two scripts and is read straight from ROCM_TORCH_VERSION two lines above.

CI on 9b09091: the test-vulkan-* jobs now pass after merging main (#24000's wmma configs).

  • Infra failures, unrelated to this PR:
    • the macOS jobs fail at Setup miniconda with CondaHTTPError: HTTP 429;
    • test-moshi-linux and test-arm-backend-ethos-u hit HuggingFace 429s;
    • the Android emulator times out booting.
  • Test CUDA Builds (the FLA kernel tests) and the Windows e2e are still running.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 42m 26s —— View job


Approve: there are no blockers at 9dc9285 and only two minor findings, but two CI jobs still need to pass before merge: rerun the parakeet non-quantized Windows export (a cut-off nvidia-cusparselt-cu13 download failed it, and that skipped the Windows CUDA e2e), and wait for unittest-cuda (kernel).

Detailed review of #25284

Summary

This PR moves CPU and every CUDA train to torch 2.15.0, read from torch_pin.py; ROCm stays on 2.14.0. It also ports chunk_gated_delta_rule to fla-core 0.6.0 for triton 3.9, and adds aoti_torch_dtype_element_size to the SlimTensor shims and aoti_cuda_shims.lib. 9dc9285 addresses the previous round. Two minor findings remain: bumping the torch pin doesn't run the Windows CUDA jobs, and the torchao source-build test lost a check.

CI at 9dc9285:

  • Windows export failure. export-model-cuda-windows-artifact (nvidia, parakeet-tdt, non-quantized) failed while installing torch==2.15.0. pip stopped downloading nvidia-cusparselt-cu13==0.8.1 at 73.4 of 170.1 MB, then reported THESE PACKAGES DO NOT MATCH THE HASHES. The other five exports installed the same pins without trouble, so this looks like a network flake.
  • Skipped Windows e2e. Because that export failed, test-model-cuda-windows-e2e was skipped. The new .lib has no Windows run on this commit.
  • Kernel test still queued. unittest-cuda (kernel) is the only job that runs the ported FLA kernels on a GPU. On d6e9b57 it failed before any test ran (conda-forge 403).
  • Still running: ROCm AOTI, pull and trunk.

Not verified:

  • I confirmed the new aoti_cuda_shims.lib names aoti_torch_dtype_element_size, but I didn't compare its full export list with main's.
  • fla-core 0.6.0 isn't installed here, so I couldn't check the new kernel arguments against its source.

Infrastructure

  • Bumping the torch pin doesn't run the Windows CUDA jobs, but aoti_cuda_shims.lib has to change with it (minor)
    • Why it matters: backends/cuda/runtime/aoti_cuda_shims.lib.md:8-9 says to regenerate the .lib "whenever the PyTorch pin moves". This PR is an example: torch 2.15 needed aoti_torch_dtype_element_size added to it.
    • What the PR did: it added torch_pin.py everywhere install_requirements.py already triggered a workflow, including the Linux and aarch64 CUDA wheel builds.
    • What it missed: two Windows CUDA workflows never listed install_requirements.py, so neither got torch_pin.py:
      • .github/workflows/build-wheels-cuda-windows.yml:6-22 (paths)
      • .github/workflows/cuda-windows.yml:15-26 (paths), and the changed-files if: blocks of the export job (59-69) and the e2e job (165-175). Those if: blocks also filter pushes to main.
    • Failure case: a later bump changes only torch_pin.py and the domain pins and forgets the .lib. No Windows CUDA job runs, either on the PR or on main outside a sampled full run. The missing shim shows up on the next PR that touches backends/cuda. In this PR the jobs ran only because files under backends/cuda/** and .ci/** changed.
    • Suggestion: this could go in the cuda.yml trigger follow-up you already planned.
    • Fix this →

Testing

  • test_explicit_torchao_source_build_is_kept no longer checks that later installs leave the source-built torchao alone (minor)
    • Where: .ci/scripts/tests/test_install_dependencies.py:384-390.
    • What changed: the cu134 version of this test asserted that no command pinned torchao==. commands[0] now pins the nightly wheel on purpose, so that loop became a positive check on commands[0] alone.
    • The gap: nothing checks commands[2:] (pip install ., the domain libraries and requirements-examples.txt). These run after the third-party/ao build in commands[1].
    • Failure case: if one of those steps starts pinning torchao==, pip replaces the source build and the test still passes.
    • Fix: self.assertFalse(any(arg.startswith("torchao==") for command in commands[2:] for arg in command)).
    • Fix this →

Recommendation

Approve. Neither finding blocks the merge. Before merging:

  • rerun the failed Windows export, so test-model-cuda-windows-e2e runs against the new .lib;
  • wait for unittest-cuda (kernel) to pass.

  • Load the review skill and the changes since the last round
  • Run independent reviewers
  • Merge duplicate findings and re-read the cited code
  • Check CI status
  • Post the review

- test_explicit_torchao_source_build_is_kept again checks that nothing after
  the third-party/ao build pins torchao, which would replace the source build.
- build-wheels-cuda-windows.yml triggers on torch_pin.py and
  install_requirements.py: aoti_cuda_shims.lib has to move with the torch pin,
  so a pin-only bump must run the Windows CUDA wheel jobs.
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Addressed in 32f947b:

  • torchao source build: test_explicit_torchao_source_build_is_kept again asserts that nothing after the third-party/ao build (commands[2:]) pins torchao==.
  • Windows CUDA triggers on the pin: build-wheels-cuda-windows.yml now triggers on torch_pin.py and install_requirements.py. The same triggers for cuda-windows.yml (paths plus the export, e2e and runtime if: lists) are in Cover CUDA 13.4 in the Windows CUDA pull request jobs #25307, which already reworks those lists.

CI on 9dc9285 (that commit changes only a test and a workflow trigger):

  • unittest-cuda (kernel): fla-core-0.6.0 is installed and all 5 test_chunk_gated_delta_rule tests pass, test_eager_fla_configs included with all four GRID_SHAPES. The job's only failures are the same 5 test_conv_implicit_gemm OutOfResources: shared memory cases that fail on main (6840285).
  • Windows export + e2e (rerun after a truncated nvidia-cusparselt-cu13 download failed one export): all 6 exports and all 6 test-model-cuda-windows-e2e rows pass against the regenerated aoti_cuda_shims.lib. delete-model-cuda-windows-artifacts then ran.
  • Remaining red is infra or also fails on main:
    • macOS Setup miniconda HTTP 429;
    • HuggingFace 429 in test-moshi-linux and test-arm-backend-ethos-u;
    • Android emulator boot timeout;
    • ROCm gfx950 missing pandas;
    • test-models-cuda cancelled at its 90-minute timeout, as on main.
  • Still queued: Qwen3.5-35B-A3B and gemma-4-31B are waiting for an A100 runner, as they are on main right now.

This branch was successfully deployed

1 active (outdated) and 1 inactive deployments
cadence — 32f947bf Deployed Nov 20, 2026 by Gasoonjia via hifi-op-test / hifi4 #39564
upload-benchmark-results — abdec96c Deployed Nov 19, 2026 by Gasoonjia via upload-benchmark-results #22801
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/docker Trigger docker-builds to rebuild and push the CI images for this PR CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant