Repository navigation
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25284
Note: Links to docs will display an error until the docs builds have been completed. ❌ 6 Pending, 2 Unrelated Failures, 4 Unclassified FailuresAs of commit 32f947b with merge base 679e49e ( UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
FLAKY - The following job failed but was likely due to flakiness present on trunk:
BROKEN TRUNK - The following job failed but was present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
The cu134 train ran a torch 2.15 nightly whose oldest wheels are days from expiring off the nightly index, while cu132 and CPU used torch 2.14.0. torch 2.15.0 is released and on the test index for cpu, cu132 and cu134, so every train now installs the same torch, torchvision 0.29.2 and torchaudio 2.11.1, read from torch_pin.py. ROCm keeps torch 2.14.0 because rocm7.2 has no 2.15.0 wheel yet. The ESP-IDF generator reads the pin through install_requirements.py instead of scraping its text, and every workflow that triggers on install_requirements.py also triggers on torch_pin.py.
torch 2.15 requires triton 3.9, and every flash-linear-attention release before 0.6.0 fails to import under it (an autotune key names a removed kernel argument). 0.6.0 changed the kernels this op launches: - the gating kernels use exp2, so the chunk cumsum is scaled by 1/ln 2; - the h and o kernels take HV and STATE_V_FIRST and flatten V blocks into grid axis 0; - the cumsum kernel flattens its grid and takes USE_GRAPH instead of HEAD_FIRST; - the kkt and w/u kernels take HV. Only fla-core is installed: it holds the kernels, and the flash-linear-attention meta package now pins transformers>=5.6.2.
torch 2.15's AOTI runtime checks input alignment through aoti_torch_dtype_element_size. The ETensor shims had it, but the Windows DLL is built from the SlimTensor shims, so a model compiled with 2.15 could not load. Add it there and to the import library. The import library was regenerated per aoti_cuda_shims.lib.md with mingw-w64 binutils 2.40, which rebuilds the old export list byte for byte; the new one adds exactly this name.
abdec96 to
12b76fc
Compare
This PR needs a
|
|
@claude review effort=max |
|
Claude finished @Gasoonjia's task in 31m 9s —— View job Needs Discussion: nothing blocks moving every CUDA train to torch 2.15.0 and the FLA op to fla-core 0.6.0, but a leftover pre-0.6.0 FLA install now breaks importing the CUDA backend, and nobody has yet confirmed that torchvision 0.29.2 and torchaudio 2.11.1 work with torch 2.15.0. Detailed review of #25284SummaryThis PR pins torch 2.15.0 from There are no blockers. The two points to settle are under Recommendation. What I could not verify:
The Code Quality
Infrastructure
Testing
Backward Compatibility
Specific Comments
RecommendationNeeds Discussion. Two things should be settled before merging:
Everything else is a minor follow-up. |
- chunk_gated_delta_rule raises ImportError when fla-core is older than 0.6.0, so the kernels package skips the op instead of older FLA's ValueError under triton 3.9 breaking import of the whole CUDA backend. - aoti_torch_dtype_element_size range-checks the code before the int8_t cast. - The FLA test also runs B=2, K=V=128 and T=100. - Quote the fla-core install hint and pin it to 0.6.0. - windows-msvc.yml triggers on torch_pin.py. - The ROCm pin test skips rather than fails once the pins align. - aoti_cuda_shims.lib.md names binutils 2.42, which was checked, not 2.41.
|
@claude review effort=max |
|
Claude finished @Gasoonjia's task in 29m 34s —— View job No blockers; six minor findings (two CI install lines that bypass their source of truth, two test gaps, a missing fla-core error that drops the install hint, and a possibly dropped Detailed review of #25284 at
|
- A missing fla-core raises the same ImportError with the install hint, instead of PackageNotFoundError's bare message. - cuda.yml installs fla-core from qwen3_5_moe/requirements.txt, the pin the Qwen3.5 export already uses, and says the Qwen3.5 tests need it too. - windows-msvc.yml's arm64 step reads the torch index from install_requirements along with the version, fails on a bad import, and triggers on install_requirements.py. - The ESP-IDF release check requires torch==TORCH_VERSION exactly. - The K != V grid shape uses V = 256, which gives several V blocks and a V block count unlike K's for every BV fla-core 0.6.0 autotunes (at most 128). - The Qwen3.5 README lists transformers for run.py: flash-linear-attention depended on it, fla-core does not. - Comment wording (not 0.6.x; fla-core in test-rocm-aoti.sh).
|
Addressed in 9dc9285:
Not changed: renaming the ROCm scripts' shell CI on 9b09091: the
|
|
@claude review effort=max |
|
Claude finished @Gasoonjia's task in 42m 26s —— View job Approve: there are no blockers at Detailed review of #25284SummaryThis PR moves CPU and every CUDA train to torch 2.15.0, read from CI at
Not verified:
Infrastructure
Testing
RecommendationApprove. Neither finding blocks the merge. Before merging:
|
- test_explicit_torchao_source_build_is_kept again checks that nothing after the third-party/ao build pins torchao, which would replace the source build. - build-wheels-cuda-windows.yml triggers on torch_pin.py and install_requirements.py: aoti_cuda_shims.lib has to move with the torch pin, so a pin-only bump must run the Windows CUDA wheel jobs.
|
Addressed in 32f947b:
CI on 9dc9285 (that commit changes only a test and a workflow trigger):
|
Follow-up to #25255 (drop CUDA 13.0).
Why
The cu134 train installs a torch
2.15.0.dev20260921nightly. Its oldest wheels are a few days from expiring off the nightly index, and once they go, every cu134 job fails to resolve. cu132 and CPU are still on torch 2.14.0. torch 2.15.0 is released (2026-10-28) and is on the test index for cpu, cu132 and cu134, so every train can use one release.What
One set of pins (
install_requirements.py,torch_pin.py)torch_pin.py. torchvision 0.29.2 acceptstorch>=2.14.0and torchaudio 2.11.1 acceptstorch>=2.11. All three wheels are on test/{cpu,cu132,cu134}.CU134_TORCH_PACKAGES,cu134_requirements()and the cu134 special cases ininstall_requirements.pyare removed.ROCM_TORCH_VERSION) because the rocm7.2 index has no 2.15.0 wheel.EXECUTORCH_USE_RELEASE_CPU_DEPSstill pairs released torch with torchao 0.18.0. I checked locally that 0.18.0 loads andquantize_runs on torch 2.15.0+cpu.FLA kernels (
backends/cuda/triton/kernels/chunk_gated_delta_rule.py)torch 2.15 requires triton 3.9. Every flash-linear-attention release before 0.6.0 fails to import under triton 3.9 (
ValueError: key contains names that are not kernel arguments: STAGE), which broketest-cuda-shimson the first version of #25255. The wrapper is ported to the fla-core 0.6.0 kernel signatures:exp2, so the chunk cumsum is scaled byRCP_LN2;HVandSTATE_V_FIRST, and fold the V-block index into grid axis 0;USE_GRAPHinstead ofHEAD_FIRST;HV.CI and the qwen3.5-MoE requirements install
fla-core==0.6.0rather than theflash-linear-attentionmeta package, which now pinstransformers>=5.6.2.Windows CUDA shims (
backends/aoti/common_shims_slim.*,aoti_cuda_shims.lib)torch 2.15's AOTI runtime calls
aoti_torch_dtype_element_sizefromassert_alignment. The Windows DLL is built from the SlimTensor shims, which did not define it, so a model compiled with 2.15 could not load. The shim is added there, and the import library is regenerated peraoti_cuda_shims.lib.mdwith mingw-w64 binutils 2.40. Rebuilding the old export list with 2.40 reproduces the checked-in file byte for byte, and the new library differs only by this one export. The.mdnow notes that binutils 2.46 numbers import hints differently and does not reproduce the file.Pin readers
TORCH_VERSIONandTORCH_URL_BASEby importinginstall_requirements.pyinstead of sed-scraping its text.torch_pin.pyis added to its stamp inputs, totest_version_stamp.shand to thetest-espressifpath filter, and the negative test now corruptstorch_pin.py.windows-msvc.ymlreads the torch version fromtorch_pin.py.install_requirements.pynow also namestorch_pin.py.Test plan
pytest .ci/scripts/tests/test_install_dependencies.py .ci/scripts/tests/test_cuda_workflow.py .ci/scripts/tests/test_filter_cuda_matrix.py: 50 passed. The ROCm test now runs each ROCm script's pin-read snippet and checks its output, and the source-build test checks that the core install also pinstorchao==TORCHAO_NIGHTLY_VERSION.examples/espressif/test_version_stamp.sh: 51 OK, 0 FAIL, including the newTORCH_VERSION must be X.Y.Zcase.chunk_gated_delta_rule.pyimports and registerstorch.ops.triton.chunk_gated_delta_rule. A script also checked each of the 6 launched kernels: the keyword arguments passed match the 0.6.0 kernel parameters, minus the ones set by autotune configs or defaults, with no missing, extra or conflicting names. None of the 0.6.0 autotuners set the pre/post hooks thatwrap_tritonrejects.common_shims_slim.cppand ran a small check ofaoti_torch_dtype_element_sizefor every dtype constant, plus ComplexFloat, which returns 0. The new gtest cases cover the same.test_chunk_gated_delta_rule.pycompares against FLA's naive reference and FLA's own chunk implementation) and the Windows CUDA e2e load.