Skip to content

CUDA backend: remap a method to another device at load with a target_device load option - #25310

Closed
Gasoonjia wants to merge 1 commit into
fix/cuda-target-devicefrom
fix/cuda-load-device-override
Closed

Gasoonjia wants to merge 1 commit into
fix/cuda-target-devicefrom
fix/cuda-load-device-override

Conversation

@Gasoonjia

Copy link
Copy Markdown
Contributor

Stacked on #24890 — review/merge that first; this PR's diff is against its branch.

Why

#24890 makes a delegate run on the device its program was lowered for. In review, @shoumikhin raised the deploy side (#24887 discussion): a program exported on a shared multi-GPU box on whichever GPU was free (cuda:3, say) will not load on a robot (Thor/Orin) that has a single cuda:0, and a library such as lerobot-export cannot ask its host process (a notebook, a multi-GPU job, train-then-export in one process) to set CUDA_VISIBLE_DEVICES before CUDA starts. The ask: an opt-in 1-to-1 device remap at load time.

What

Pass target_device as a load option — the compile spec's key, reused as a runtime spec:

Runtime.get().load_program(
    "model.pte",
    data_path="aoti_cuda_blob.ptd",
    backend_options={"CudaBackend": {"target_device": "cuda:0"}},
)

In C++ the same option goes through LoadBackendOptionsMap under CudaBackend. With it set:

  • The loaders (extension/module for C++, extension/pybindings for Python) allocate the method's planned device buffers on the named device instead of the planned one. Sizes and layout do not change; every buffer moves together.
  • The CUDA delegate parses the same key from its runtime specs and creates its AOTI container, weights, KV state and launches on the named device (CUDA backend: run each delegate on its target device (#24887) #24890 centralized all of that on handle->device_index, so this is one assignment at init).

Rules

The remap only applies when the method was lowered for a single device, and fails loudly otherwise:

  • Planned buffers spanning more than one CUDA device (or on a non-CUDA device): load fails in the loaders.
  • Delegates of one method lowered for different devices: load fails in the delegate (a per-method registry keyed by named data map + method name tracks the exported device of live remapped delegates).
  • Remap target the process cannot see: load fails at init.
  • Malformed value (not cuda[:N], or not a string): load fails.
  • No option: behavior is exactly CUDA backend: run each delegate on its target device (#24887) #24890's.

Tests

backends/cuda/tests/test_target_device.py (the multi-GPU job added in #24890 runs it):

  • test_load_option_remaps_to_the_named_device: export for cuda:0 → remap to cuda:1 and export for cuda:1 → remap to cuda:0, each with the exported device current; outputs match eager and the caller's current device is unchanged.
  • test_load_option_remap_to_a_missing_device_fails: remap to the first device index the process does not have; loading must fail.

Needs two CUDA devices, like the rest of the file. The C++ Module loader shares the planned-buffer logic with the Python loader; its remap path is covered by code inspection here and would benefit from a native multi-GPU test once one exists for Module device buffers.

@pytorch-bot

pytorch-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25310

Note: Links to docs will display an error until the docs builds have been completed.

❌ 103 New Failures, 2 Unrelated Failures, 73 Unclassified Failures

As of commit b18c24a with merge base d52a96d (image):

NEW FAILURES - The following jobs have failed:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 19, 2026
…device load option

Stacked on #24890, which runs each delegate on the device its program
was lowered for. Reviewers there asked for the opposite direction too:
a program exported on whichever GPU was free (cuda:3, say) does not
load on a single-GPU box as cuda:0, and a library cannot ask its
host process to set CUDA_VISIBLE_DEVICES before CUDA starts.

Passing target_device as a load option (backend_options
{"CudaBackend": {"target_device": "cuda:0"}}) now remaps the whole
method at load: the loaders move every planned device buffer to the
named device, and the delegate creates its container, weights and
launches there. Memory planning does not change; all buffers move
together.

The remap is deliberately narrow. It only applies when the method was
lowered for a single device: a method whose planned buffers span
devices fails in the loaders, and a method whose delegates were
lowered for different devices fails in the delegate registry. A remap
to a device the process cannot see fails at load. Without the option,
loading behaves exactly as in #24890.
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Closing in favor of #25315.

This load-time remap spreads CUDA device knowledge into the generic loaders (extension/module, extension/pybindings), which only allocate ExecuTorch-owned planned buffers and should not know any backend's option names. The replacement is an offline tool in the spirit of merge_ptes.py: remap_pte_devices.py rewrites the device ids recorded in the PTE itself (planned-memory table, tensor placement info, and delegate target_device specs) with one mapping — dense by default, so 3, 5, 7 become 0, 1, 2 — and the runtime then loads the output unchanged, with no remapping support of its own.

#24890 (the base of this PR) is unaffected and still proceeding: running each delegate on its exported device is the right default; remapping to different devices is now a file-level choice made before deployment instead of a load-time one.

@Gasoonjia Gasoonjia closed this Nov 19, 2026
Gasoonjia added a commit that referenced this pull request Nov 20, 2026
## Why

A CUDA program records the devices from its `target_device` compile spec
in three places in the PTE: the planned-memory table
(`non_const_buffer_device`), tensor `extra_tensor_info`, and each
delegate's `target_device` spec itself. All three must agree. A program
exported with an explicit non-zero `target_device` (`cuda:3`, say)
therefore does not load on a machine whose GPUs have other indices — the
loader tries to allocate planned buffers on a device that does not exist
(see #24887 discussion). Leaving `target_device` at
its `cuda:0` default avoids this; compiling on another GPU does not by
itself record that GPU.

Earlier I tried solving this at load time with a `target_device` runtime
option (#25310, closed). That spreads device knowledge into the generic
loaders, which only allocate ExecuTorch-owned buffers and should not
know any backend's options. This PR replaces that approach with an
offline tool, in the spirit of `merge_ptes.py`: rewrite the file once,
and the runtime runs it unchanged.

## What

`backends/cuda/scripts/remap_pte_devices.py`, which starts a new
`backends/cuda/scripts/` package for standalone CUDA backend tools
(`merge_ptes.py` moves into it in the stacked follow-up #25323):

```bash
python -m executorch.backends.cuda.scripts.remap_pte_devices \
    --input-pte model.pte --output-pte model_dense.pte            # 3,5,7 -> 0,1,2
python -m executorch.backends.cuda.scripts.remap_pte_devices \
    --input-pte model.pte --output-pte model_cuda0.pte --mapping 3=0,5=0,7=0
python -m executorch.backends.cuda.scripts.remap_pte_devices \
    --input-pte model.pte --dry-run                               # show the mapping only
```

- Collects every CUDA device index recorded in the program (planned
buffers, tensor infos, delegate `target_device` specs — parsed with
export's own `_parse_device_spec_value`, so `CUDA:3` and bare `cuda`
behave exactly as at export) and applies one mapping to all three
places, so they stay consistent. Collection and rewriting share one walk
over the device sites, deduplicated by object identity for specs several
delegates share.
- Default mapping compresses the recorded ids to a dense `0..n-1` range,
keeping them distinct and in order. The tool warns whenever the result
is not entirely on `cuda:0`: every remapped device must exist on the
target machine, and the CUDA runtime currently runs every delegate on
the current device (#24890). `--mapping OLD=NEW,...` overrides the
default and may collapse several devices onto `cuda:0`.
Distinct-preserving mappings load like a native export for the new
devices; collapsing mappings stay correct but keep device→host→device
bounces and per-device input copies a native single-device export would
have elided or shared.
- Refuses a mapping that does not cover a recorded device, names one the
program does not record, or names a target outside the stored byte range
`0..=127`, so a file is never rewritten only in part and a mapping meant
for another file fails loudly. Re-serialization uses
`extract_delegate_segments=True`, matching export; like merge_ptes it
re-serializes with the default alignments, so exports made with
non-default alignments may be repacked (contents unchanged). The
companion `.ptd` needs no changes: tensor data carries no device ids.
- No runtime changes. The CPU-only tests are wired into `pytest.ini`
testpaths and a Buck `python_unittest`, so serializer/schema changes run
them without a GPU.

## Tests

`backends/cuda/tests/test_remap_pte_devices.py` builds a program in
memory with devices spread across all three places and several execution
plans (planned buffers on cuda:3/cuda:5, a tensor on cuda:7, a delegate
spec on cuda:9, a second plan on cuda:11) and checks collection, dense
and explicit mappings (including collapsing to one device),
bare/uppercase/non-CUDA/malformed specs, shared-spec rewriting,
coverage/repeated-source/out-of-range errors, serialize round trips with
named data, identity-remap byte equality, failed-write atomicity, the
not-entirely-cuda:0 warning, the CLI dry-run (printed mapping), CLI
input errors as `SystemExit(2)`, and the CLI write path with
`--mapping`. It also exports a small program with
`DeviceAwarePartitioner("cuda:3")` and checks the remapped program
equals a native `cuda:0` export. All 22 tests pass locally (CPU-only
environment; no GPU needed since the tool only edits metadata).

This branch had an error being deployed

1 failed and 1 inactive deployments
upload-benchmark-results — b18c24a3 Deployed Nov 19, 2026 by Gasoonjia via upload-benchmark-results #22821
cadence — b18c24a3 Deployed Nov 19, 2026 by Gasoonjia via hifi-op-test / hifi4 #39510
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant