Repository navigation
CUDA backend: add a PTE device-id remap tool - #25315
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25315
Note: Links to docs will display an error until the docs builds have been completed. ❌ 18 Pending, 1 Unrelated Failure, 1 Unclassified FailureAs of commit fcf50da with merge base 679e49e ( UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:
BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
143ecbe to
be54df6
Compare
|
@claude /pr-review |
|
Claude finished @Gasoonjia's task in 16m 30s —— View job Request Changes: the remap tool rewrites PTEs with different serializer settings than export, can silently skip PR Review: #25315SummaryAdds
All 8 changed files were reviewed. Running Python wasn't permitted in this run, so these findings come from reading the code, and the new tests were not re-run. Code Quality
Testing
API Design
Backward Compatibility
RecommendationRequest Changes What blocks approval:
|
be54df6 to
060a508
Compare
|
@claude thanks — addressed all of these in the latest push:
|
|
@claude /pr-review |
|
Claude finished @Gasoonjia's task in 3m 18s —— View job Request Changes: nearly all of the earlier review is fixed, but the new tests never run in CI, and the README edit moves merge instructions under the new remap heading. The multi-device warning also leaves out the runtime limitation that makes the dense default risky, and a few smaller gaps remain. PR Review: #25315 (re-review)SummaryThe PR adds I checked the fixes you listed in the code and they hold:
What's left is below. Python wasn't allowed in this run, so I didn't run the 12 tests; everything here comes from reading the code. Infrastructure
API Design
Code Quality
Testing
Backward Compatibility
RecommendationRequest Changes To unblock:
The remaining items are minor. |
|
@claude sixth-round items addressed:
Left as a follow-up, deliberately: a one-step |
|
@claude /pr-review |
|
Claude finished @Gasoonjia's task in 27m 47s —— View job Approve: the sixth-round fixes are in the code and nothing blocks merging; what's left is minor: PR Review: #25315 (seventh pass)SummaryAdds Code Quality
Testing
API Design
RecommendationApprove: nothing blocks merging. The in-place |
|
Claude finished @Gasoonjia's task in 3m 43s —— View job Approve: I checked each sixth-round fix in the code and nothing blocks merging. Three minor gaps remain: shortening the docstring removed the only user-facing note that re-serialization resets alignment, a failed output write still ends in a raw traceback, and a non-integer PR Review: #25315 (seventh pass)SummaryThis PR adds The sixth-round fixes are all in the code:
All 8 changed files were covered by three independent reviewers, and I re-read every cited line. Running Python wasn't allowed in this job, so I didn't re-run the tests. API Design
Code Quality
Testing
RecommendationApprove Nothing blocks merging. The cheapest fixes before landing are the README alignment sentence and catching |
278f0cc to
f200b3e
Compare
|
@claude seventh-round items addressed:
21 tests pass locally. The |
|
@claude /pr-review |
|
Claude finished @Gasoonjia's task in 27m 3s —— View job Approve: nothing blocks; five minor findings remain: a dead PR Review: #25315 (eighth pass)SummaryThe PR adds
Code Quality
Infrastructure
Testing
API Design
RecommendationApprove Nothing blocks. All five findings are minor (dead code, documentation, Buck dependency hygiene and test assertions) and can land here or in a follow-up. |
|
Claude finished @Gasoonjia's task in 3m 20s —— View job Approve: the seventh-round fixes are all in place and nothing blocks merging. Two minor gaps remain: a serialization failure during the final write still ends in a raw traceback, and a PR Review: #25315 (eighth pass)SummaryThis PR adds Three independent reviewers covered all 8 changed files, and I re-read every cited line. Running pytest wasn't allowed in this job, so I didn't re-run the 21 tests. The checkout is shallow, so I reviewed Code Quality
RecommendationApprove Nothing blocks merging. The write-stage exception handling is a small change worth making before landing; the rest is minor. |
|
@claude eighth-round items addressed:
22 tests pass locally. |
|
@claude /pr-review |
|
Claude finished @Gasoonjia's task in 26m 48s —— View job Approve: nothing blocks this ninth pass; the main remaining issue is that PR Review: #25315 (ninth pass)SummaryThe PR adds How this pass was done:
Code Quality
Infrastructure
Testing
API Design
RecommendationApprove. Nothing blocks merging, and all six findings are minor. The |
|
Claude finished @Gasoonjia's task in 43s —— View job Approve: all four eighth-round fixes are in the code and I found nothing that blocks merging; there are two optional nits in the new device-type pre-check. Checking the eighth-round fixes for #25315
Checked
Nits (optional)
I didn't run the tests: this job isn't allowed to run Python, so everything above comes from reading the code. The 22 passing tests are from your local run. |
|
@claude both optional nits are in as well (type names derive from |
|
@claude /pr-review |
|
Claude finished @Gasoonjia's task in 25m 36s —— View job Approve: nothing blocks, but PR Review: #25315SummaryThis PR adds
API Design
Infrastructure
Testing
Code Quality
RecommendationApprove. Nothing blocks. The |
|
Claude finished @Gasoonjia's task in 21s —— View job Approve: both optional nits from the ninth pass are in the code, and nothing else is outstanding from my side. Checking the ninth-pass nits for #25315
Checked
A small gap that doesn't need a change: the test asserts only that the I didn't run the tests, because this job can't run Python. The checks above come from reading the code. |
A CUDA program records the devices from its target_device compile spec in its planned memory table, in tensor placement info, and in each delegate's target_device compile spec itself. All three must agree, so a program exported with an explicit non-zero target_device (cuda:3, say) does not load on a machine whose GPUs have other indices, and the loader has no way to move it. scripts/remap_pte_devices.py rewrites all three places in the file with one mapping, dense by default (3, 5, 7 become 0, 1, 2, keeping their order, with a warning unless the result lands entirely on cuda:0) or explicit (--mapping 3=0,5=0,7=0 collapses everything onto cuda:0). The output is an ordinary PTE, so the runtime needs no remapping support of its own. The companion .ptd is untouched: tensor data carries no device ids. A mapping that does not cover a recorded device is refused, so a file is never rewritten only in part. The tool starts a new backends/cuda/scripts directory for standalone CUDA backend tools, alongside the library code. Its CPU-only tests join pytest.ini testpaths and a Buck python_unittest, so serializer and schema changes run them without a GPU. Test Plan: - backends/cuda/tests/test_remap_pte_devices.py (also registered as a Buck python_unittest): 22 tests covering collection from all three places across multiple execution plans, dense and explicit mappings, coverage and validation errors, bare/uppercase/non-CUDA/ malformed target_device specs, shared compile specs, serialize round trips with named data, identity-remap byte equality, failed-write atomicity, the not-entirely-cuda:0 warning, an exported program (DeviceAwarePartitioner, cuda:3) remapped to equal a native cuda:0 export, and the CLI dry-run, input-error and write paths. All pass locally.
|
Fixed the remaining direct-dependency lint issue in Local checks: repository-configured flake8 and Black pass on the new Python files; the 22-test suite was already green on the prior head (this amend only changes |
|
@claude /pr-review |
|
Claude finished @Gasoonjia's task in 33m 9s —— View job Approve: nothing blocks; the only commit since the ninth pass fixes the Buck deps, so three earlier findings are still open (led by PR Review: #25315 (tenth pass)
SummaryAdds
Code Quality
Testing
API Design
RecommendationApprove. Nothing here blocks:
|
Why
A CUDA program records the devices from its
target_devicecompile spec in three places in the PTE: the planned-memory table (non_const_buffer_device), tensorextra_tensor_info, and each delegate'starget_devicespec itself. All three must agree. A program exported with an explicit non-zerotarget_device(cuda:3, say) therefore does not load on a machine whose GPUs have other indices — the loader tries to allocate planned buffers on a device that does not exist (see #24887 discussion). Leavingtarget_deviceat itscuda:0default avoids this; compiling on another GPU does not by itself record that GPU.Earlier I tried solving this at load time with a
target_deviceruntime option (#25310, closed). That spreads device knowledge into the generic loaders, which only allocate ExecuTorch-owned buffers and should not know any backend's options. This PR replaces that approach with an offline tool, in the spirit ofmerge_ptes.py: rewrite the file once, and the runtime runs it unchanged.What
backends/cuda/scripts/remap_pte_devices.py, which starts a newbackends/cuda/scripts/package for standalone CUDA backend tools (merge_ptes.pymoves into it in the stacked follow-up #25323):python -m executorch.backends.cuda.scripts.remap_pte_devices \ --input-pte model.pte --output-pte model_dense.pte # 3,5,7 -> 0,1,2 python -m executorch.backends.cuda.scripts.remap_pte_devices \ --input-pte model.pte --output-pte model_cuda0.pte --mapping 3=0,5=0,7=0 python -m executorch.backends.cuda.scripts.remap_pte_devices \ --input-pte model.pte --dry-run # show the mapping onlytarget_devicespecs — parsed with export's own_parse_device_spec_value, soCUDA:3and barecudabehave exactly as at export) and applies one mapping to all three places, so they stay consistent. Collection and rewriting share one walk over the device sites, deduplicated by object identity for specs several delegates share.0..n-1range, keeping them distinct and in order. The tool warns whenever the result is not entirely oncuda:0: every remapped device must exist on the target machine, and the CUDA runtime currently runs every delegate on the current device (CUDA backend: run each delegate on its target device (#24887) #24890).--mapping OLD=NEW,...overrides the default and may collapse several devices ontocuda:0. Distinct-preserving mappings load like a native export for the new devices; collapsing mappings stay correct but keep device→host→device bounces and per-device input copies a native single-device export would have elided or shared.0..=127, so a file is never rewritten only in part and a mapping meant for another file fails loudly. Re-serialization usesextract_delegate_segments=True, matching export; like merge_ptes it re-serializes with the default alignments, so exports made with non-default alignments may be repacked (contents unchanged). The companion.ptdneeds no changes: tensor data carries no device ids.pytest.initestpaths and a Buckpython_unittest, so serializer/schema changes run them without a GPU.Tests
backends/cuda/tests/test_remap_pte_devices.pybuilds a program in memory with devices spread across all three places and several execution plans (planned buffers on cuda:3/cuda:5, a tensor on cuda:7, a delegate spec on cuda:9, a second plan on cuda:11) and checks collection, dense and explicit mappings (including collapsing to one device), bare/uppercase/non-CUDA/malformed specs, shared-spec rewriting, coverage/repeated-source/out-of-range errors, serialize round trips with named data, identity-remap byte equality, failed-write atomicity, the not-entirely-cuda:0 warning, the CLI dry-run (printed mapping), CLI input errors asSystemExit(2), and the CLI write path with--mapping. It also exports a small program withDeviceAwarePartitioner("cuda:3")and checks the remapped program equals a nativecuda:0export. All 22 tests pass locally (CPU-only environment; no GPU needed since the tool only edits metadata).