Repository navigation
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25310
Note: Links to docs will display an error until the docs builds have been completed. ❌ 103 New Failures, 2 Unrelated Failures, 73 Unclassified FailuresAs of commit b18c24a with merge base d52a96d ( NEW FAILURES - The following jobs have failed:
UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
FLAKY - The following jobs failed but were likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
…device load option Stacked on #24890, which runs each delegate on the device its program was lowered for. Reviewers there asked for the opposite direction too: a program exported on whichever GPU was free (cuda:3, say) does not load on a single-GPU box as cuda:0, and a library cannot ask its host process to set CUDA_VISIBLE_DEVICES before CUDA starts. Passing target_device as a load option (backend_options {"CudaBackend": {"target_device": "cuda:0"}}) now remaps the whole method at load: the loaders move every planned device buffer to the named device, and the delegate creates its container, weights and launches there. Memory planning does not change; all buffers move together. The remap is deliberately narrow. It only applies when the method was lowered for a single device: a method whose planned buffers span devices fails in the loaders, and a method whose delegates were lowered for different devices fails in the delegate registry. A remap to a device the process cannot see fails at load. Without the option, loading behaves exactly as in #24890.
a52c093 to
b18c24a
Compare
|
Closing in favor of #25315. This load-time remap spreads CUDA device knowledge into the generic loaders ( #24890 (the base of this PR) is unaffected and still proceeding: running each delegate on its exported device is the right default; remapping to different devices is now a file-level choice made before deployment instead of a load-time one. |
## Why A CUDA program records the devices from its `target_device` compile spec in three places in the PTE: the planned-memory table (`non_const_buffer_device`), tensor `extra_tensor_info`, and each delegate's `target_device` spec itself. All three must agree. A program exported with an explicit non-zero `target_device` (`cuda:3`, say) therefore does not load on a machine whose GPUs have other indices — the loader tries to allocate planned buffers on a device that does not exist (see #24887 discussion). Leaving `target_device` at its `cuda:0` default avoids this; compiling on another GPU does not by itself record that GPU. Earlier I tried solving this at load time with a `target_device` runtime option (#25310, closed). That spreads device knowledge into the generic loaders, which only allocate ExecuTorch-owned buffers and should not know any backend's options. This PR replaces that approach with an offline tool, in the spirit of `merge_ptes.py`: rewrite the file once, and the runtime runs it unchanged. ## What `backends/cuda/scripts/remap_pte_devices.py`, which starts a new `backends/cuda/scripts/` package for standalone CUDA backend tools (`merge_ptes.py` moves into it in the stacked follow-up #25323): ```bash python -m executorch.backends.cuda.scripts.remap_pte_devices \ --input-pte model.pte --output-pte model_dense.pte # 3,5,7 -> 0,1,2 python -m executorch.backends.cuda.scripts.remap_pte_devices \ --input-pte model.pte --output-pte model_cuda0.pte --mapping 3=0,5=0,7=0 python -m executorch.backends.cuda.scripts.remap_pte_devices \ --input-pte model.pte --dry-run # show the mapping only ``` - Collects every CUDA device index recorded in the program (planned buffers, tensor infos, delegate `target_device` specs — parsed with export's own `_parse_device_spec_value`, so `CUDA:3` and bare `cuda` behave exactly as at export) and applies one mapping to all three places, so they stay consistent. Collection and rewriting share one walk over the device sites, deduplicated by object identity for specs several delegates share. - Default mapping compresses the recorded ids to a dense `0..n-1` range, keeping them distinct and in order. The tool warns whenever the result is not entirely on `cuda:0`: every remapped device must exist on the target machine, and the CUDA runtime currently runs every delegate on the current device (#24890). `--mapping OLD=NEW,...` overrides the default and may collapse several devices onto `cuda:0`. Distinct-preserving mappings load like a native export for the new devices; collapsing mappings stay correct but keep device→host→device bounces and per-device input copies a native single-device export would have elided or shared. - Refuses a mapping that does not cover a recorded device, names one the program does not record, or names a target outside the stored byte range `0..=127`, so a file is never rewritten only in part and a mapping meant for another file fails loudly. Re-serialization uses `extract_delegate_segments=True`, matching export; like merge_ptes it re-serializes with the default alignments, so exports made with non-default alignments may be repacked (contents unchanged). The companion `.ptd` needs no changes: tensor data carries no device ids. - No runtime changes. The CPU-only tests are wired into `pytest.ini` testpaths and a Buck `python_unittest`, so serializer/schema changes run them without a GPU. ## Tests `backends/cuda/tests/test_remap_pte_devices.py` builds a program in memory with devices spread across all three places and several execution plans (planned buffers on cuda:3/cuda:5, a tensor on cuda:7, a delegate spec on cuda:9, a second plan on cuda:11) and checks collection, dense and explicit mappings (including collapsing to one device), bare/uppercase/non-CUDA/malformed specs, shared-spec rewriting, coverage/repeated-source/out-of-range errors, serialize round trips with named data, identity-remap byte equality, failed-write atomicity, the not-entirely-cuda:0 warning, the CLI dry-run (printed mapping), CLI input errors as `SystemExit(2)`, and the CLI write path with `--mapping`. It also exports a small program with `DeviceAwarePartitioner("cuda:3")` and checks the remapped program equals a native `cuda:0` export. All 22 tests pass locally (CPU-only environment; no GPU needed since the tool only edits metadata).
Stacked on #24890 — review/merge that first; this PR's diff is against its branch.
Why
#24890 makes a delegate run on the device its program was lowered for. In review, @shoumikhin raised the deploy side (#24887 discussion): a program exported on a shared multi-GPU box on whichever GPU was free (
cuda:3, say) will not load on a robot (Thor/Orin) that has a singlecuda:0, and a library such as lerobot-export cannot ask its host process (a notebook, a multi-GPU job, train-then-export in one process) to setCUDA_VISIBLE_DEVICESbefore CUDA starts. The ask: an opt-in 1-to-1 device remap at load time.What
Pass
target_deviceas a load option — the compile spec's key, reused as a runtime spec:In C++ the same option goes through
LoadBackendOptionsMapunderCudaBackend. With it set:extension/modulefor C++,extension/pybindingsfor Python) allocate the method's planned device buffers on the named device instead of the planned one. Sizes and layout do not change; every buffer moves together.handle->device_index, so this is one assignment at init).Rules
The remap only applies when the method was lowered for a single device, and fails loudly otherwise:
cuda[:N], or not a string): load fails.Tests
backends/cuda/tests/test_target_device.py(the multi-GPU job added in #24890 runs it):test_load_option_remaps_to_the_named_device: export for cuda:0 → remap to cuda:1 and export for cuda:1 → remap to cuda:0, each with the exported device current; outputs match eager and the caller's current device is unchanged.test_load_option_remap_to_a_missing_device_fails: remap to the first device index the process does not have; loading must fail.Needs two CUDA devices, like the rest of the file. The C++
Moduleloader shares the planned-buffer logic with the Python loader; its remap path is covered by code inspection here and would benefit from a native multi-GPU test once one exists forModuledevice buffers.