Skip to content

CUDA backend: add a PTE device-id remap tool - #25315

Merged
Gasoonjia merged 1 commit into
mainfrom
cuda-remap-pte-devices
Nov 20, 2026
Merged

Gasoonjia merged 1 commit into
mainfrom
cuda-remap-pte-devices

Conversation

@Gasoonjia

@Gasoonjia Gasoonjia commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Why

A CUDA program records the devices from its target_device compile spec in three places in the PTE: the planned-memory table (non_const_buffer_device), tensor extra_tensor_info, and each delegate's target_device spec itself. All three must agree. A program exported with an explicit non-zero target_device (cuda:3, say) therefore does not load on a machine whose GPUs have other indices — the loader tries to allocate planned buffers on a device that does not exist (see #24887 discussion). Leaving target_device at its cuda:0 default avoids this; compiling on another GPU does not by itself record that GPU.

Earlier I tried solving this at load time with a target_device runtime option (#25310, closed). That spreads device knowledge into the generic loaders, which only allocate ExecuTorch-owned buffers and should not know any backend's options. This PR replaces that approach with an offline tool, in the spirit of merge_ptes.py: rewrite the file once, and the runtime runs it unchanged.

What

backends/cuda/scripts/remap_pte_devices.py, which starts a new backends/cuda/scripts/ package for standalone CUDA backend tools (merge_ptes.py moves into it in the stacked follow-up #25323):

python -m executorch.backends.cuda.scripts.remap_pte_devices \
    --input-pte model.pte --output-pte model_dense.pte            # 3,5,7 -> 0,1,2
python -m executorch.backends.cuda.scripts.remap_pte_devices \
    --input-pte model.pte --output-pte model_cuda0.pte --mapping 3=0,5=0,7=0
python -m executorch.backends.cuda.scripts.remap_pte_devices \
    --input-pte model.pte --dry-run                               # show the mapping only
  • Collects every CUDA device index recorded in the program (planned buffers, tensor infos, delegate target_device specs — parsed with export's own _parse_device_spec_value, so CUDA:3 and bare cuda behave exactly as at export) and applies one mapping to all three places, so they stay consistent. Collection and rewriting share one walk over the device sites, deduplicated by object identity for specs several delegates share.
  • Default mapping compresses the recorded ids to a dense 0..n-1 range, keeping them distinct and in order. The tool warns whenever the result is not entirely on cuda:0: every remapped device must exist on the target machine, and the CUDA runtime currently runs every delegate on the current device (CUDA backend: run each delegate on its target device (#24887) #24890). --mapping OLD=NEW,... overrides the default and may collapse several devices onto cuda:0. Distinct-preserving mappings load like a native export for the new devices; collapsing mappings stay correct but keep device→host→device bounces and per-device input copies a native single-device export would have elided or shared.
  • Refuses a mapping that does not cover a recorded device, names one the program does not record, or names a target outside the stored byte range 0..=127, so a file is never rewritten only in part and a mapping meant for another file fails loudly. Re-serialization uses extract_delegate_segments=True, matching export; like merge_ptes it re-serializes with the default alignments, so exports made with non-default alignments may be repacked (contents unchanged). The companion .ptd needs no changes: tensor data carries no device ids.
  • No runtime changes. The CPU-only tests are wired into pytest.ini testpaths and a Buck python_unittest, so serializer/schema changes run them without a GPU.

Tests

backends/cuda/tests/test_remap_pte_devices.py builds a program in memory with devices spread across all three places and several execution plans (planned buffers on cuda:3/cuda:5, a tensor on cuda:7, a delegate spec on cuda:9, a second plan on cuda:11) and checks collection, dense and explicit mappings (including collapsing to one device), bare/uppercase/non-CUDA/malformed specs, shared-spec rewriting, coverage/repeated-source/out-of-range errors, serialize round trips with named data, identity-remap byte equality, failed-write atomicity, the not-entirely-cuda:0 warning, the CLI dry-run (printed mapping), CLI input errors as SystemExit(2), and the CLI write path with --mapping. It also exports a small program with DeviceAwarePartitioner("cuda:3") and checks the remapped program equals a native cuda:0 export. All 22 tests pass locally (CPU-only environment; no GPU needed since the tool only edits metadata).

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 19, 2026
@pytorch-bot

pytorch-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25315

Note: Links to docs will display an error until the docs builds have been completed.

❌ 18 Pending, 1 Unrelated Failure, 1 Unclassified Failure

As of commit fcf50da with merge base 679e49e (image):

UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 16m 30s —— View job


Request Changes: the remap tool rewrites PTEs with different serializer settings than export, can silently skip target_device specs that export accepts, describes runtime device selection incorrectly (so its dense default is unsafe for programs that record several devices), and moves merge_ptes with no compatibility path.

PR Review: #25315

Summary

Adds backends/cuda/scripts/remap_pte_devices.py, an offline tool that rewrites CUDA device indices in three places: planned buffers, tensor infos, and delegate target_device specs. The PR also moves merge_ptes.py into the same scripts/ package. Problems found:

  • The output is re-serialized with settings that differ from export.
  • A target_device value that export accepts can be skipped without an error.
  • The docstring and the dense default rely on runtime behavior that doesn't exist.
  • Mapping validation runs only in the CLI parser.
  • The merge_ptes move breaks its documented command.

All 8 changed files were reviewed. Running Python wasn't permitted in this run, so these findings come from reading the code, and the new tests were not re-run.

Code Quality

  • Re-serialization changes the file layout.
    • remap_pte_devices.py:205 and :263 call serialize_pte_binary(pte_file) with the default extract_delegate_segments=False (exir/_serialize/_program.py:386).
    • deserialize_pte_binary has already turned every segment-backed delegate into inline data (_restore_delegates, _program.py:567-596). Export defaults to extract_delegate_segments=True (exir/capture/_config.py:78).
    • Result: every delegate blob of a normally exported PTE moves into the flatbuffer. It stays resident for the life of the program and counts toward the flatbuffer size limit. This contradicts "leaves everything else alone … loads exactly as if it had been exported" (lines 23-26).
    • The sibling tool passes extract_delegate_segments=True (scripts/merge_ptes.py:645).
    • Impact: small for a CUDA-only program, because the CUDA processed blob is just the AOTI metadata manifest. For a program that also uses another backend, it is that backend's whole blob.
    • Fix this →
  • target_device parsing differs from export's and fails silently.
    • _parse_target_device (remap_pte_devices.py:57, :70-76) uses a case-sensitive regex that allows no whitespace after the colon. Values it can't parse are skipped without an error (:97-99, :174-176).
    • The pass that derives placement from the spec, _parse_device_spec_value (exir/passes/propagate_device_pass.py:56-78), lower-cases and strips the value. CudaPartitioner passes user specs through unchanged (cuda_partitioner.py:43-53).
    • Example: with CompileSpec("target_device", b"CUDA:3"), export puts the buffers and tensors on cuda:3. The tool rewrites those to cuda:0, leaves the spec as CUDA:3, and reports success. That is the partial rewrite the docstring says can't happen.
    • Fix: reuse TARGET_DEVICE_COMPILE_SPEC_KEY and _parse_device_spec_value from propagate_device_pass (cuda_partitioner.py:13 already imports the key), filter to DeviceType.CUDA, and let parse errors raise.
    • Fix this →
  • main() re-implements remap_pte_bytes.
    • remap_pte_devices.py:238-244 repeats the deserialize → collect → reject-empty → dense-default → remap sequence from :198-204. That gives two serialize call sites to keep in sync, so the serializer fix above has to be made twice.
    • A missing --output-pte (:235) raises a ValueError traceback; parser.error would print usage instead.
    • Fix this →
  • Temp-file handling (remap_pte_devices.py:257-264).
    • NamedTemporaryFile(delete=False) creates the file with mode 0600, so after os.replace the output .pte is readable only by its owner, regardless of umask. That breaks deploy flows where a different user loads the model.
    • If serialization raises, the hidden .<name>.XXXX file is left in the output directory.
    • merge_ptes.py:722-727 has the same pattern.
  • The new tool isn't in the README. examples/cuda/README.md documents merge_ptes, and the PR edits that line, but it doesn't mention the remap tool. A user hitting the single-GPU case from CUDA backend: a program crashes with an illegal memory access when CUDA device 1 is current #24887 would look there.

Testing

  • Test gaps in backends/cuda/tests/test_remap_pte_devices.py:
    • The fixture program (:45-99) has an empty inline delegate and no segments, named data, constants or mutable data.
    • The round-trip test (:144-148) checks only the set of collected device indices, so nothing verifies that non-device content survives.
    • The non-dry-run CLI path (mkdir, temp file and os.replace at remap_pte_devices.py:256-265) never runs.
    • --mapping is never passed through the CLI.
    • There are no tests for a bare b"cuda" spec or for a program with no CUDA ids.
    • assertRaises(ValueError) at :137 and :141 would also pass on an unrelated ValueError; use assertRaisesRegex.
    • Fix this →

API Design

  • The docstring misstates how the runtime picks a device, and the dense default depends on that claim.
    • remap_pte_devices.py:14-15 says target_device "tells the delegate which device to create its container and run on". No C++ in the repo reads that spec; the only mention is a comment at backends/cuda/runtime/cuda_backend.cpp:762.
    • The container is created with "cuda" and no index (cuda_backend.cpp:588). SM-variant selection and the weight cache both use cudaGetDevice (cuda_backend.cpp:470-474, cuda_weight_cache.cpp:608-609). So every CUDA delegate runs on whatever device is current at init.
    • Consequence: for a program that records several devices, the advertised dense default (3,5,7 → 0,1,2) still names cuda:1 and cuda:2. On a single-GPU target it still fails to load. On a multi-GPU target the delegates run on the current device against buffers that live on other devices.
    • Fix: correct the docstring. When more than one device is recorded and no --mapping is given, fail or at least warn.
    • Fix this →
  • Mapping validation lives only in the CLI parser.
    • parse_device_mapping (remap_pte_devices.py:123-125) rejects negative and repeated targets. The public remap_program_devices and remap_pte_bytes accept anything: remap_pte_bytes(data, {3: -1}) writes device_index=-1 and b"cuda:-1".
    • Repeated source ids ("3=0,3=1", :120) silently become {3: 1}.
    • The rule against two sources sharing a target also blocks 3=0,5=0. Given the previous point, that is the only multi-device remap the current runtime can execute on one GPU, and the PR gives no reason for the rule. Either drop it, or document why it exists and enforce it in remap_program_devices.
    • Fix this →

Backward Compatibility

  • Moving merge_ptes breaks its documented entry points.
    • backends/cuda/merge_ptes.py moves to backends/cuda/scripts/merge_ptes.py with no shim. That breaks the previously documented python -m executorch.backends.cuda.merge_ptes and any from executorch.backends.cuda.merge_ptes import ….
    • The Buck target keeps the name :merge_ptes but its module path changes, so importers of the old module fail at import time, not at build time.
    • The move is unrelated to the PR's stated purpose and isn't mentioned in the description. Leave a forwarding module at the old path, or split the move into its own PR with a release notes: label.
    • The description's commands (python -m executorch.backends.cuda.remap_pte_devices, "backends/cuda/remap_pte_devices.py") name a module that doesn't exist; it lives under scripts/. Correct them before the description becomes the commit message.
    • Fix this →

Recommendation

Request Changes

What blocks approval:

  • The serializer flag and the target_device parser are small, concrete correctness fixes.
  • The merge_ptes move needs a compatibility path or an explicit announcement.
  • The docstring, the dense default and the rule against shared targets need a decision on how multi-device programs should be handled, since the runtime binds every CUDA delegate to the current device.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude thanks — addressed all of these in the latest push:

  • Serializer settings: both serialize_pte_binary calls now pass extract_delegate_segments=True, matching export and merge_ptes; the round-trip test now serializes with segments and named data and checks the named data and tensor content survive.
  • target_device parsing: replaced the local regex with TARGET_DEVICE_COMPILE_SPEC_KEY + _parse_device_spec_value from propagate_device_pass (plus the BUCK dep). Only non-CUDA specs are skipped; malformed values now raise instead of being silently left behind. Added tests for CUDA:3 and bare cuda.
  • Docstring and the dense default: corrected — the spec is described as the device the delegate was lowered for, with the multi-device caveats spelled out and CUDA backend: run each delegate on its target device (#24887) #24890 referenced for delegate-side device selection. The dense default stays (it is the requested behavior for same-shape multi-GPU targets), but the CLI now warns when it is applied to a program recording several devices and points at --mapping to collapse them.
  • Mapping validation: target non-negativity is now enforced in remap_program_devices itself, parse_device_mapping rejects repeated sources, and the distinct-targets rule is dropped — collapsing (3=0,5=0) is exactly the single-GPU case.
  • Shared helper + CLI errors: main() and remap_pte_bytes share _remap_pte_file; a missing --output-pte now goes through parser.error. The output file also keeps the input's permissions, and the temp file is removed if writing fails.
  • merge_ptes compatibility: a forwarding module stays at backends/cuda/merge_ptes.py re-exporting scripts/merge_ptes.py, so python -m executorch.backends.cuda.merge_ptes and existing imports keep working. PR description commands corrected to the scripts/ paths.
  • Tests/readme: added the non-dry-run CLI write path, --mapping through the CLI, bare-spec and no-CUDA-ids cases, assertRaisesRegex, and a README section for the new tool. 12 tests pass locally.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 3m 18s —— View job


Request Changes: nearly all of the earlier review is fixed, but the new tests never run in CI, and the README edit moves merge instructions under the new remap heading. The multi-device warning also leaves out the runtime limitation that makes the dense default risky, and a few smaller gaps remain.

PR Review: #25315 (re-review)

Summary

The PR adds backends/cuda/scripts/remap_pte_devices.py, which rewrites CUDA device ids in the planned-buffer table, tensor infos and delegate target_device specs, and moves merge_ptes.py into scripts/ with a shim at the old path.

I checked the fixes you listed in the code and they hold:

  • Both serialize calls pass extract_delegate_segments=True.
  • Spec parsing goes through _parse_device_spec_value.
  • Negative targets are rejected in remap_program_devices.
  • Repeated sources are rejected.
  • main() and remap_pte_bytes share _remap_pte_file.
  • A missing --output-pte goes through parser.error.
  • The output keeps the input's permissions, and the temp file is cleaned up on failure.
  • scripts/merge_ptes.py is byte-identical to the old file.

What's left is below. Python wasn't allowed in this run, so I didn't run the 12 tests; everything here comes from reading the code.

Infrastructure

  • test_remap_pte_devices.py doesn't run in CI.
    • The CUDA unit-test step at .github/workflows/cuda.yml:800-803 lists only test_cuda_weight_metadata.py and test_merge_ptes.py.
    • The new file isn't registered anywhere else either: there's no BUCK test target and no pytest.ini testpath.
    • All 12 tests, including the CLI write path and --mapping tests, would only run when someone runs them by hand. Add the file to that pytest invocation.
    • The PR already edits this workflow. I can't change files under .github/workflows, so this one is yours to make.

API Design

  • The multi-device warning doesn't say why the dense default is unsafe (remap_pte_devices.py:277-287).
    • The warning only says every remapped device must exist on the target.
    • The underlying problem: the CUDA runtime never reads target_device. The AOTI container is created with a bare "cuda" (cuda_backend.cpp:588), and device lookups use cudaGetDevice. Meanwhile, planned buffers are allocated on their recorded non_const_buffer_device.
    • Example: a program recording 3 and 5, dense-mapped to 0 and 1, runs on a 2-GPU box. Every delegate runs on the current device, but some of its buffers live on cuda:1.
    • The docstring (:29-31) states the CUDA backend: run each delegate on its target device (#24887) #24890 precondition, but the warning, which is the only thing these users see, and the README (examples/cuda/README.md:72-86) do not.
    • Fix: say in the warning and the README that, until CUDA backend: run each delegate on its target device (#24887) #24890 lands, distinct devices are only safe if each delegate runs on its recorded device, and recommend collapsing.
    • Fix this →
  • The docstring still says the tool "leaves the rest of the file's layout alone" (remap_pte_devices.py:23-26).
    • The PTE is fully re-serialized with default alignments (segment_alignment=128, no constant_tensor_alignment or delegate_alignment).
    • A PTE exported with a non-default ExecutorchBackendConfig alignment comes out with a different layout.
    • merge_ptes behaves the same way, so the claim needs rewording, not a code change.

Code Quality

  • The README insertion breaks up the merge section (examples/cuda/README.md:72-86).
    • The new ## Run an export on different GPUs… heading sits between the merge description and "Export every regular input with PTX disabled:", the cuda_include_ptx snippet, and the fallback/ELF paragraphs.
    • Those paragraphs are merge instructions, but they now appear under the remap heading. Move the new section after them.
    • Separately, --mapping 3=0,5=0 follows the "3, 5, 7 become 0, 1, 2" example in both the README and the docstring (:28-32). Applied to that program, the mapping is refused because cuda:7 isn't covered. Use 3=0,5=0,7=0.
    • Fix this →
  • Bad --mapping input or an uncovered/CPU-only program gives a raw traceback (remap_pte_devices.py:131, :134, :266-269).
    • --mapping 3=x fails inside int() with a message that doesn't name the bad entry.
    • The ValueErrors from _remap_pte_file escape main() as tracebacks, while the missing --output-pte case now goes through parser.error.
    • Fix: catch ValueError in main() and pass it to parser.error.
  • Targets above 127 aren't rejected up front (remap_pte_devices.py:149).
    • device_index is a flatbuffer byte (int8) in schema/program.fbs:97,442, so --mapping 3=200 passes validation and fails or wraps later, during serialization.
    • Low confidence on the exact failure mode. Checking 0 <= target <= 127 next to the negative check would settle it.

Testing

  • The round-trip test doesn't exercise delegate segments. The fixture's delegate is BackendDelegateInlineData(b"") (test_remap_pte_devices.py:97). _program.py only moves non-empty inline data into a segment, so extract_delegate_segments=True is never exercised and the "segments survive" claim isn't tested. Give the fixture a non-empty delegate blob, then assert it comes back as a segment with the same bytes.
  • The bare/uppercase spec test (:152-163) only checks that index 0 is collected. It never remaps a bare cuda spec, so the rewrite of cuda to cuda:N is untested. The comment about "cuda:3's planned device" doesn't describe what the test does.

Backward Compatibility

  • The shim only partly preserves the old module path for Buck users (backends/cuda/BUCK:152-177).
    • The existing :merge_ptes target now ships only scripts/merge_ptes.py; the old path lives in the new :merge_ptes_shim target.
    • A Buck dependent of //executorch/backends/cuda:merge_ptes that imports executorch.backends.cuda.merge_ptes fails at import time.
    • Fix: put the shim in :merge_ptes's srcs, or add the scripts module to the existing target.
    • I couldn't see internal dependents, so the impact is unconfirmed.
    • Smaller issue: import * with no __all__ doesn't re-export the _-prefixed helpers, and patches applied through the old path don't reach the real code.

Recommendation

Request Changes

To unblock:

The remaining items are minor.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude sixth-round items addressed:

  • main() now reports a missing/unreadable --input-pte through parser.error (OSError caught alongside ValueError), rejects --dry-run combined with --output-pte instead of silently ignoring it, and attributes the warning correctly on both paths (warn_stacklevel threaded through). Tests cover all three new CLI exits.
  • write_atomically flushes and os.fsyncs before the rename, matching measured_routing.py.
  • Unrecorded mapping sources are now rejected, so target validation and coverage agree (a mapping meant for another file fails loudly); added a test.
  • The module docstring is trimmed to the essentials, pointing at the README for the long version.
  • :remap_pte_devices now lists //executorch/exir/backend:compile_spec_schema directly.

Left as a follow-up, deliberately: a one-step --mapping all=0 shorthand. The warning already prints a ready-to-paste collapse mapping, and I would rather not grow the CLI surface in the same PR that introduces the tool.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 27m 47s —— View job


Approve: the sixth-round fixes are in the code and nothing blocks merging; what's left is minor: remap_program_devices rewrites CompileSpec objects that an emitted program shares with its lowered modules, the trimmed docstring now points to a README that lacks the caveats it dropped, and there are a few small CLI and test gaps.

PR Review: #25315 (seventh pass)

Summary

Adds backends/cuda/scripts/remap_pte_devices.py, which rewrites the CUDA device ids a PTE records (planned-memory devices, tensor extra_tensor_info and delegate target_device specs) using one mapping. It also adds an atomic-write helper, 20 tests, Buck targets and a README section. Every sixth-round fix is in the code. Three independent reviewers covered all 8 changed files, and every line cited below was re-read. Python isn't allowed in this job, so the tests weren't re-run. All findings are minor.

Code Quality

  • backends/cuda/scripts/remap_pte_devices.py:229: the if old in indices filter in targets = {new for old, new in report.mapping if old in indices} never drops anything. remap_program_devices already rejects any source the program doesn't record (:184-189), yet the filter reads as if such sources could get this far. targets = set(mapping.values()) gives the same result.
  • remap_pte_devices.py:314-320: write_atomically runs outside the try at :295-302. A write-side OSError therefore still ends in a traceback instead of the exit-2 parser.error that input-side errors now get. Two ways to hit it:
    • --output-pte names an existing directory, so os.replace raises IsADirectoryError;
    • the output directory can't be created or written to.

Fix this →

Testing

  • backends/cuda/tests/test_remap_pte_devices.py:358-365: the --mapping "" case only checks for exit code 2. It would still pass if the "device mapping is empty" check (remap_pte_devices.py:153-154) were deleted, because {} then fails with "does not cover cuda device(s) [3, 5, 7, 9]" instead. Assert the stderr message, or drop the redundant check.
  • test_remap_pte_devices.py:417-421: the permission check compares the output's mode with the input's, but the input gets its mode from the process umask. Under a 077 umask the input is 0600, which is also NamedTemporaryFile's default. The check would then pass even with the os.chmod in _atomic_output.py:46 removed. Call input_pte.chmod(0o644) first.

Fix this →

API Design

  • remap_program_devices also rewrites specs that are owned outside the program (remap_pte_devices.py:120-122, :174-176). The function is public, and its comment says it handles "an emitted in-memory program", but it sets CompileSpec.value in place. That leaks for these reasons:

    • In an emitted program, each BackendDelegate.compile_specs is the lowered module's own list (exir/emit/_emitter.py:1553).
    • to_backend copies only the list, not the specs (exir/backend/backend_api.py:117).
    • CudaPartitioner passes a list that already has a target_device through unchanged (backends/cuda/cuda_partitioner.py:43-53). So the specs are likely the caller's own objects as well.

    After remap_program_devices(manager.executorch_program, {3: 0}), manager's lowered modules also say cuda:0, and a later export that reuses the same partitioner or spec list would quietly target cuda:0. The id() dedup only matters for in-memory programs, because a deserialized program never shares specs. So the case the dedup was added for is exactly the case where the rewrite leaks. The CLI and remap_pte_bytes deserialize fresh objects and aren't affected. Either copy each delegate's compile_specs before rewriting, or document that the function takes a deserialized program and drop the in-memory comment and the dedup.

    Fix this →

  • The docstring defers to a README that lacks what it dropped (remap_pte_devices.py:18-19, examples/cuda/README.md:84-105). That README section is now the tool's only long-form documentation, and it leaves out three things:

    • Re-serialization. The output is fully re-serialized with the default segment, constant and delegate alignments (exir/_serialize/_program.py:383-390), and deserialization resets named-data alignment to 1 (:645). An export made with non-default alignments therefore comes back repacked. This caveat now appears only in the PR description. It is not in the README, --help or the commit message.
    • The cost of a collapse. The README calls collapsing onto cuda:0 "the remap that runs everywhere today", but doesn't say that the bounces and per-device input copies remain. It also doesn't say that each original device keeps its own planned buffer (exir/memory_planning.py:1388-1391), so a collapsed program can need more cuda:0 planned memory than a native single-device export.
    • --dry-run. It shows the recorded ids before anything is written, which are the keys a mapping must cover for the tool to accept it.

    Fix this →

Recommendation

Approve: nothing blocks merging. The in-place CompileSpec rewrite and the README gap are worth fixing before landing; the rest can go in this PR or a follow-up.

• [`cuda-remap-pte-devices`](https://github.com/pytorch/executorch/tree/cuda-remap-pte-devices)

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 3m 43s —— View job


Approve: I checked each sixth-round fix in the code and nothing blocks merging. Three minor gaps remain: shortening the docstring removed the only user-facing note that re-serialization resets alignment, a failed output write still ends in a raw traceback, and a non-integer --mapping entry gets a vague error.

PR Review: #25315 (seventh pass)

Summary

This PR adds backends/cuda/scripts/remap_pte_devices.py. It rewrites the CUDA device ids a PTE records (planned buffers, tensor extra_tensor_info, delegate target_device specs) with one mapping, and adds a shared atomic writer, Buck targets, CPU-run tests and a README section.

The sixth-round fixes are all in the code:

  • OSError/ValueError on the read path go through parser.error.
  • --dry-run with --output-pte is rejected.
  • warn_stacklevel is passed through.
  • The writer calls fsync.
  • Unrecorded mapping sources are rejected.
  • The compile_spec_schema dep is in place.

All 8 changed files were covered by three independent reviewers, and I re-read every cited line. Running Python wasn't allowed in this job, so I didn't re-run the tests. origin/main has no merge base in this shallow checkout, so I reviewed HEAD~1..HEAD (the single PR commit). The --mapping all=0 shorthand and the GPU load test are deferred as you said, so they aren't raised again. The same goes for merge_ptes adopting the writer, which happens in #25323.

API Design

  • Trimming the docstring removed the only user-facing alignment caveat (remap_pte_devices.py:7-20; examples/cuda/README.md).
    • Both write paths re-serialize with the default alignments (:258, :316-318). Deserialization also resets named-data alignment to 1 (exir/_serialize/_program.py:645).
    • So a PTE exported with, say, ExecutorchBackendConfig(segment_alignment=4096) for page-aligned mmap comes back 128-aligned, and the tool says nothing about it.
    • The previous docstring said so. After the trim, neither the docstring nor the README section it now points to mentions alignment; only the PR description does.
    • Current CUDA exports use the defaults, so nothing breaks today. But the README tells users to run this tool before deploying, so the caveat belongs there.
    • All three reviewers raised this.
    • Fix this →

Code Quality

  • main() still has raw-traceback paths (remap_pte_devices.py:295-320).
    • Output-side errors: write_atomically runs outside the try (:314). Any of these gives a traceback instead of exit 2:
      • --output-pte /readonly_dir/x.pte;
      • an output path that is an existing directory;
      • a full disk.
    • Corrupt input: an input that passes the header checks but has a corrupt flatbuffer can raise subprocess.CalledProcessError from _run_flatc, which calls subprocess.run(..., check=True) (exir/_serialize/_flatbuffer.py:345). That is neither OSError nor ValueError. Low confidence on whether the default deserialize path reaches flatc.
    • Two reviewers raised this.
    • Fix this →
  • Non-integer mapping entries give a vague error (remap_pte_devices.py:149, :152).
    • For --mapping cuda:3=0 (the docs spell devices as cuda:N) or 3=0=1, the CLI prints invalid literal for int() with base 10: 'cuda:3'. The message names neither the entry nor the expected form, unlike the OLD=NEW message three lines above.
    • Fix: wrap both int() calls and re-raise with the same "not of the form OLD=NEW" message.
    • All three reviewers raised this.
  • Dead filter (remap_pte_devices.py:229).
    • remap_program_devices now rejects both missing and unrecorded sources, so report.mapping's keys always equal indices.
    • That makes if old in indices dead code; use targets = set(mapping.values()).

Testing

  • The writer's failure contract is untested (backends/cuda/tests/test_remap_pte_devices.py).
    • Nothing makes write raise and then checks that no .model.pte.* temp file is left and that an existing output is untouched. That is the guarantee _atomic_output.py:9-13 gives as its reason for existing.
    • Deleting the finally cleanup (_atomic_output.py:49-51) would pass every test.
    • Two reviewers raised this.

Recommendation

Approve

Nothing blocks merging. The cheapest fixes before landing are the README alignment sentence and catching OSError around the write.

• [`cuda-remap-pte-devices`](https://github.com/pytorch/executorch/tree/cuda-remap-pte-devices)

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude seventh-round items addressed:

  • The alignment caveat is back in user-facing docs: one sentence in the README section and one in the module docstring (default alignments on re-serialization; named-data alignment reset to 1).
  • Output-side OSErrors from write_atomically now go through parser.error, as does a flatc CalledProcessError from a corrupt flatbuffer on the read path; --dry-run with --output-pte is rejected instead of silently ignored. Tests cover an output path that is a directory.
  • Non-integer mapping entries (cuda:3=0, 3=0=1) now fail with the same "not of the form OLD=NEW" message, tested via parse_device_mapping.
  • The dead old in indices filter is gone — unrecorded sources are rejected up front, so targets is just the mapping's values.
  • The writer's failure contract is tested directly: a write that raises mid-stream leaves the previous output byte-identical and no temp file behind.

21 tests pass locally. The --mapping all=0 shorthand stays out, as before: the warning prints a ready-to-paste collapse mapping.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 27m 3s —— View job


Approve: nothing blocks; five minor findings remain: a dead CalledProcessError catch, an unreachable parser branch, a README that doesn't say what collapsing onto cuda:0 costs, two missing direct Buck test deps, and CLI error tests that check only the exit code.

PR Review: #25315 (eighth pass)

Summary

The PR adds remap_pte_devices, a CLI and library that applies one mapping to the CUDA device ids a PTE records, in three places: planned-memory devices, tensor extra_tensor_info, and delegate target_device specs. It also adds an atomic-write helper, tests, Buck targets and a README section.

  • Coverage: three reviewers covered all 8 changed files, and I re-checked every finding below against the code.
  • Tests not run: Python can't run in this job.
  • Diff: the checkout is shallow, so I reviewed HEAD~1..HEAD, which is the PR's single commit f200b3e.
  • Not re-raised: the deliberate deferrals (dense default with a warning, --mapping all=0, a GPU load-and-run test, the merge_ptes move and writer adoption in CUDA backend: move merge_ptes into backends/cuda/scripts #25323).

Code Quality

  • The subprocess.CalledProcessError catch at backends/cuda/scripts/remap_pte_devices.py:309 is dead, and so is import subprocess at :35. The seventh-round reply describes it as catching a flatc failure on a corrupt flatbuffer, but neither path runs flatc:

    • deserialize_pte_binary parses with _flatbuffer_to_program (exir/_serialize/_program.py:734).
    • serialize_pte_binary builds with _program_to_flatbuffer (:501).
    • Both come from _flatbuffer_program.py, which uses the pure-Python flatbuffers package and never calls _run_flatc.

    Inputs the reader rejects with something other than ValueError or OSError still print a traceback. A truncated file or a .ptd passed by mistake likely raises struct.error or IndexError, but I didn't run it.

    Fix: drop the exception and the import. If a clean message for non-PTE input matters, catch the reader's errors around deserialize_pte_binary only.
    Fix this →

  • The if not separator branch in parse_device_mapping (:154-157) can never run. When the entry has no =, partition leaves new as "". int(new) then raises first, and the except at :150 reports the same message. Delete the branch.

    The only malformed entry tested is cuda:3=0 (backends/cuda/tests/test_remap_pte_devices.py:144). Adding 3 and 3=0=1 would cover the missing and the doubled separator.
    Fix this →

Infrastructure

  • The test_remap_pte_devices target (backends/cuda/tests/targets.bzl:592-600) is missing direct deps for two of its imports.

    • The test imports executorch.backends.cuda.scripts._atomic_output (test_remap_pte_devices.py:15), but //executorch/backends/cuda:scripts_atomic_output is not in the deps. Today it resolves only through :remap_pte_devices (backends/cuda/BUCK:187).
    • It imports executorch.exir.scalar_type (:31), but //executorch/exir:scalar_type (exir/BUCK:52) is not in the deps. Today it resolves only through //executorch/exir:schema (exir/BUCK:64).

    This target already lists compile_spec_schema directly for the same reason. The visibility of :scripts_atomic_output already covers the tests package.
    Fix this →

Testing

  • The six cases in test_cli_input_errors_exit_like_argparse_errors (backends/cuda/tests/test_remap_pte_devices.py:373-425) assert only that the exit code is 2. Every parser.error and every argparse usage error exits with 2. A case that starts failing for a different reason still passes. For example, if a flag is renamed, argparse rejects the old one as an unrecognized argument and the test stays green.

    The directory case (:411-419) passes directory itself as --output-pte. write_atomically therefore puts its temp file in Path(directory).parent, the system temp dir, where nothing checks that it is removed after os.replace fails. test_failed_atomic_write_keeps_existing_output only covers a failure inside write.

    Fix:

    • Capture stderr with redirect_stderr and assert each case's message: "required unless --dry-run", "device mapping is empty", "does not cover" and "not written with --dry-run". For the two OSError cases, assert the offending path.
    • For the directory case, pass a subdirectory of directory and assert that no .<subdir>.* file is left next to it.
      Fix this →

API Design

  • The README calls collapsing onto cuda:0 "the remap that runs everywhere today" but doesn't say what it costs (examples/cuda/README.md:101-104). The module docstring sends readers there for "the longer discussion" (remap_pte_devices.py:16-21), but the README says less than the docstring does:

    • Copies: the docstring mentions the device→host→device copies and input copies that a collapse keeps. The README leaves them out.
    • Memory: neither one mentions it. Memory planning gives each (device_type, device_index) its own buffer (exir/memory_planning.py:1388-1391, :1485-1487), and the remap keeps those buffers. After --mapping 3=0,5=0,7=0, the program holds one planned cuda:0 buffer per original device, and their sizes add. A native cuda:0 export plans the same tensors in one arena and can reuse memory across their lifetimes.
    • Layout: the layout sentence at :105-108 leaves out that output is always written with extract_delegate_segments=True (remap_pte_devices.py:266, :326). Inline delegate data therefore comes back as segments.

    Fix: state these costs in the README, note that re-exporting with the default target_device avoids both, and keep the docstring's pointer accurate.
    Fix this →

Recommendation

Approve

Nothing blocks. All five findings are minor (dead code, documentation, Buck dependency hygiene and test assertions) and can land here or in a follow-up.

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 3m 20s —— View job


Approve: the seventh-round fixes are all in place and nothing blocks merging. Two minor gaps remain: a serialization failure during the final write still ends in a raw traceback, and a target_device spec naming an unknown device type aborts the whole remap instead of being skipped as the docstring says.

PR Review: #25315 (eighth pass)

Summary

This PR adds backends/cuda/scripts/remap_pte_devices.py. The tool rewrites the CUDA device ids a PTE records (planned buffers, tensor extra_tensor_info, delegate target_device specs) using one mapping. It also adds a shared atomic writer, Buck targets, tests that run on CPU, and a README section.

Three independent reviewers covered all 8 changed files, and I re-read every cited line. Running pytest wasn't allowed in this job, so I didn't re-run the 21 tests. The checkout is shallow, so I reviewed HEAD~1..HEAD. I left out a few things you already decided or deferred: --mapping all=0, the GPU load test, merge_ptes adopting the writer (#25323), and the dense default.

Code Quality

  • A serialization failure during the write still prints a raw traceback (remap_pte_devices.py:322-331).
    • The first try (:301-309) catches (OSError, ValueError, subprocess.CalledProcessError). But serialize_pte_binary runs inside the write_atomically lambda, in the second try, and that one catches only OSError.
    • Serialization can raise ValueError. It can also fall back to the JSON+flatc path (exir/_serialize/_program.py:499), which can raise CalledProcessError. Either error escapes as a traceback after the "CUDA device mapping / changed" lines have printed.
    • --dry-run never serializes, so it can report success for an input whose real write then fails.
    • Fix: serialize inside the first try, which also makes --dry-run exercise serialization, and pass the resulting bytes to write_atomically. The temp-file cleanup already works either way.
    • All three reviewers reported this.
    • Fix this →
  • A spec naming an unknown device type aborts the whole remap (remap_pte_devices.py:77-87, :107-110).
    • The _parse_target_device docstring says a spec that names a non-CUDA device "returns None … not this tool's to move".
    • That only holds for CPU. _parse_device_spec_value raises for any type outside schema.DeviceType (exir/passes/propagate_device_pass.py:74-77), and that enum has only CPU and CUDA today.
    • _cuda_device_sites checks the key on every delegate, whatever its backend id. So a PTE that also carries another backend's delegate with, say, target_device=b"npu:0" fails with "Unknown device type" and can't be remapped at all.
    • No in-tree backend does this today, so the problem is latent. Either skip specs on delegates whose id isn't CudaBackend, or reword the docstring to say unknown types raise.
  • Dead branch in parse_device_mapping (remap_pte_devices.py:147-157).
    • With no =, partition returns new == "". int("") then raises inside the try, so if not separator: can never fire. Drop it.

Recommendation

Approve

Nothing blocks merging. The write-stage exception handling is a small change worth making before landing; the rest is minor.

• [`cuda-remap-pte-devices`](https://github.com/pytorch/executorch/tree/cuda-remap-pte-devices)

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude eighth-round items addressed:

  • Serialization now runs inside the first try in main(), so ValueError/CalledProcessError from it go through parser.error and --dry-run exercises serialization; the writer lambda only writes the pre-serialized result.
  • Unknown device types in a target_device spec (some other backend's delegate sharing the key, e.g. npu:0) are now treated as non-CUDA and skipped, instead of aborting via the shared parser; malformed CUDA values (cuda:three) still raise, and the test for that now uses one. I went with the type-name pre-check rather than filtering by delegate id, since other partitioners legitimately emit CUDA target_device specs under their own backend ids (the exported-program test's demo backend is one).
  • The dead if not separator branch is gone.
  • The alignment caveat is back in both the README and a shortened form in the docstring.

22 tests pass locally.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 26m 48s —— View job


Approve: nothing blocks this ninth pass; the main remaining issue is that remap_program_devices, given an in-memory program, rewrites CompileSpec objects that the caller's lowered modules and partitioner still hold, and five smaller findings are in the details.

PR Review: #25315 (ninth pass)

Summary

The PR adds backends/cuda/scripts/remap_pte_devices.py, a library and CLI that rewrites the CUDA device ids a PTE records (planned-memory devices, tensor extra_tensor_info, and delegate target_device specs) with one mapping, along with Buck and pytest wiring, tests, and a README section. There are no blockers; all six findings below are minor.

How this pass was done:

  • Three independent reviewers covered all 8 changed files, and every finding was checked against the code.
  • The 22 tests were not re-run, because this job has no environment that can run them.
  • The checkout is shallow and has no merge base with main, so the reviewed diff is HEAD~1..HEAD, the single PR commit.
  • Points settled in earlier rounds are not raised again: --mapping all=0, a GPU load test, merge_ptes and the shared writer (CUDA backend: move merge_ptes into backends/cuda/scripts #25323), and the dense default.

Code Quality

  • _parse_target_device hand-copies the DeviceType names (backends/cuda/scripts/remap_pte_devices.py:86-94).
    • The ("cpu", "cuda") tuple exists so that _parse_device_spec_value is never called on another backend's value. The DeviceType.CUDA check after it exists only to drop the CPU case the tuple let through.
    • Comparing the prefix with "cuda" alone gives the same result for every well-formed spec and removes the second check.
    • It also matches the docstring, which says only malformed CUDA values raise. Today a malformed cpu:x spec, which the tool never rewrites, aborts the whole remap.
    • Fix this →
  • main() re-implements remap_pte_bytes (remap_pte_devices.py:306-315).
    • Both deserialize, call _remap_pte_file, and serialize with extract_delegate_segments=True.
    • The warn_stacklevel parameter of _remap_pte_file (:227, :252) exists only so that each copy points the warning at the right frame.
    • If main() called remap_pte_bytes(args.input_pte.read_bytes(), mapping) and wrote the result with write_atomically, the parameter could go, and --dry-run would still serialize.
    • The cost is one extra in-memory copy of the output (the bytes() of the Cord). If avoiding that copy is the reason for the duplicate, the comment at :312-314 should say so.
    • Fix this →

Infrastructure

  • The test target is missing two direct deps (backends/cuda/tests/targets.bzl:592-600).
    • test_remap_pte_devices.py imports executorch.backends.cuda.scripts._atomic_output (line 15) and executorch.exir.scalar_type (line 31).
    • The deps list neither //executorch/backends/cuda:scripts_atomic_output nor //executorch/exir:scalar_type. The test gets both only transitively, through :remap_pte_devices and //executorch/exir:schema.
    • Fix this →

Testing

  • The only real export is single-device, and it is compared as a Program, not as bytes (backends/cuda/tests/test_remap_pte_devices.py:310-342).
    • Multi-device is untested. The docstring's claim that distinct-preserving mappings load like a native export (remap_pte_devices.py:16-17) is never checked for more than one device.
    • More than one device is where the propagation pass makes per-device choices: _reusable_copy matches on device_index (exir/passes/propagate_device_pass.py:253), and round-trip removal compares indices (:761).
    • Exporting two delegates on cuda:3 and cuda:5, then comparing the dense remap with a native cuda:0/cuda:1 export, would cover it.
    • PerAddDeviceDelegatePartitioner (exir/tests/test_propagate_device_pass.py:129) already builds one delegate per device. It could move into exir/backend/test/device_util.py, which the test already depends on.
    • The byte-identity test only round-trips its own fixture. The test at :262-267 uses a fixture written by the same serialize_pte_binary call the tool makes, so it cannot notice the tool's hard-coded settings drifting from what to_executorch writes. An identity remap of export_for("cuda:3") would catch that.
    • Fix this →

API Design

  • remap_program_devices rewrites CompileSpec objects that the caller's export state still holds (backends/cuda/scripts/remap_pte_devices.py:132, :185-187). This was raised in an earlier pass and is still open.
    • The function is public. Per the comment at :185-186, it is meant to accept an emitted in-memory program.
    • Why the objects are shared:
      • In such a program, BackendDelegate.compile_specs is the lowered module's own list (exir/emit/_emitter.py:1553).
      • That list holds the partitioner's DelegationSpec objects, because to_backend only shallow-copies it (exir/backend/backend_api.py:117).
      • CompileSpec is a mutable dataclass.
    • Effect: remap_program_devices(manager.executorch_program, {3: 0}) also sets target_device to cuda:0 in the lowered modules of manager.exported_program() and in the partitioner itself. A later export with that partitioner then targets cuda:0, with no error.
    • The id() de-dup handles the aliasing inside the program, but not outside it.
    • Fix: give each delegate fresh CompileSpec copies before rewriting, instead of assigning spec.value, so the rewrite stays inside the program. Otherwise, document that the program must share nothing with live export state, as a deserialized program does.
    • Fix this →
  • The documented contract is wrong in one place and incomplete in another.
    • Wrong: the module docstring (remap_pte_devices.py:9-10) says that, with the cuda:0 default, only an explicit non-zero spec records an id. In fact a default export still records cuda:0, and --mapping 0=1 is how such an export moves to another GPU.
    • Incomplete: the --mapping help (:286-288) and the README section (examples/cuda/README.md:98-100) never say that the mapping must list exactly the ids the file records. remap_program_devices rejects both missing and extra sources (:189-200), so --mapping 3=0, meant to move only device 3, fails.
    • Fix this →

Recommendation

Approve. Nothing blocks merging, and all six findings are minor. The CompileSpec mutation affects only in-memory callers of the public function. The CLI and remap_pte_bytes always work on a freshly deserialized program, so they are not affected.

• [`cuda-remap-pte-devices`](https://github.com/pytorch/executorch/tree/cuda-remap-pte-devices)

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 43s —— View job


Approve: all four eighth-round fixes are in the code and I found nothing that blocks merging; there are two optional nits in the new device-type pre-check.

Checking the eighth-round fixes for #25315
  • Read the updated remap_pte_devices.py, _atomic_output.py, the README and the tests
  • Check each claimed fix
  • Post the result

Checked

  • Serialization in the first try: this holds. serialize_pte_binary(..., extract_delegate_segments=True) now runs inside the try at remap_pte_devices.py:306-317, together with the read, the parse and the remap. All three exception types go through parser.error. write_atomically gets serialized.write_to_file, so --dry-run now runs serialization too.
  • Unknown device types: this holds. _parse_target_device (:86-90) returns None for any type other than cpu or cuda before it calls the shared parser. A malformed CUDA value such as cuda:three still raises from int() inside _parse_device_spec_value. The tests cover both npu:0 (test_other_backends_target_device_spec_is_ignored) and cuda:three. I agree with checking the type name instead of the delegate id: export's demo backend legitimately emits CUDA specs under its own backend id.
  • Dead branch: this holds. parse_device_mapping has no if not separator any more, and an entry without = fails through int("") with the "not of the form OLD=NEW" message.
  • Alignment caveat: this holds. It is in the README (examples/cuda/README.md:106-108) and in the docstring (:18-20).
  • I also checked the warning location. With warn_stacklevel=2 from main() and the default of 3 through remap_pte_bytes, the warning points at the caller in both cases.

Nits (optional)

  • Hard-coded type list (remap_pte_devices.py:86-89). The pre-check lists ("cpu", "cuda") by hand, while the shared parser matches against schema.DeviceType. If a new member is added to that enum, specs naming it would be treated as "unrecognized" here. That still returns None, but for the wrong reason. Building the set from the enum keeps the two in sync:
    if type_name not in {dt.name.lower() for dt in DeviceType}:
  • Non-UTF-8 spec values (remap_pte_devices.py:86). value.decode("utf-8") runs before the type check. If another backend stores a non-UTF-8 value under target_device, the remap still aborts with UnicodeDecodeError. That is the case the pre-check exists to tolerate. No in-tree backend does this, so the risk is low. decode("utf-8", errors="replace") in the pre-check would cover it.

I didn't run the tests: this job isn't allowed to run Python, so everything above comes from reading the code. The 22 passing tests are from your local run.

• [`cuda-remap-pte-devices`](https://github.com/pytorch/executorch/tree/cuda-remap-pte-devices)

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude both optional nits are in as well (type names derive from DeviceType, and the pre-check decodes with errors="replace", covered by a non-UTF-8 spec in the ignored-specs test).

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 25m 36s —— View job


Approve: nothing blocks, but remap_program_devices rewrites caller-owned CompileSpec objects when given an emitted in-memory program, write_atomically copies the input's whole mode, and some tests, the README and one Buck target have small gaps.

PR Review: #25315

Summary

This PR adds backends/cuda/scripts/remap_pte_devices.py, a CLI and library that rewrites the CUDA device ids a PTE records (planned-buffer devices, tensor extra_tensor_info, and delegate target_device specs), together with an atomic-write helper, tests, Buck targets and a README section.

  • Three reviewers covered all 8 changed files. They found nothing blocking; the problems are listed below.
  • I did not re-run the tests, because running Python needs approval in this job.
  • The checkout is shallow, so the diff is HEAD~1..HEAD, which is the PR's single commit.
  • Items already deferred or decided in the thread are left out.

API Design

  • remap_program_devices rewrites CompileSpec objects that the program shares with its caller (backends/cuda/scripts/remap_pte_devices.py:130, :183-185).

    • Why they are shared: the comment at :183-185 treats emitted in-memory programs as a supported input. In such a program, a delegate's compile_specs is lowered_module.compile_specs (exir/emit/_emitter.py:1553). Those objects come from the partitioner's DelegationSpec (exir/backend/backend_api.py:503, :583, :623). CudaPartitioner passes the user's list straight through when it already has a target_device (backends/cuda/cuda_partitioner.py:43-53).
    • Effect: remap_program_devices(manager.executorch_program, {3: 0}) also rewrites the user's spec list and the partitioner. A later export that reuses either one then targets cuda:0 without any warning.
    • It also does nothing to the saved output: manager.buffer and write_to_file keep the old ids, because ExecutorchProgramManager serializes when it is built (exir/program/_program.py:1979, :2052-2063).
    • Not affected: the CLI and remap_pte_bytes, which work on freshly deserialized objects.
    • Fix, option 1: before rewriting, give each delegate its own copies: delegate.compile_specs = [CompileSpec(s.key, s.value) for s in delegate.compile_specs]. This also removes the sharing that the id() dedup at :185 exists for. Also document that an in-memory program has to be serialized again.
    • Fix, option 2: document that the function takes deserialized programs only, and drop the in-memory comment.
    • This was raised in two earlier passes and has no reply yet.

    Fix this →

  • The README doesn't say how to find the ids that --mapping must name (examples/cuda/README.md:98-100). remap_program_devices rejects a mapping that misses a recorded id or names an extra one (remap_pte_devices.py:187-198). The README shows --mapping 3=0,5=0,7=0 without mentioning that rule. It also never mentions --dry-run, which prints the recorded ids without writing anything.

    Fix this →

Infrastructure

  • The test_remap_pte_devices target doesn't list :scripts_atomic_output in its deps, though the test imports it (backends/cuda/tests/targets.bzl:592-600). The test imports executorch.backends.cuda.scripts._atomic_output directly (test_remap_pte_devices.py:15), but that import only resolves through the deps of :remap_pte_devices.

    • Fix: add //executorch/backends/cuda:scripts_atomic_output to the test's deps. Its visibility (//executorch/backends/cuda/...) already covers the tests package.

    Fix this →

Testing

  • test_exported_program_round_trip checks less than its comment says (backends/cuda/tests/test_remap_pte_devices.py:345-348). deserialize_pte_binary moves named and mutable data out of .program and clears program.segments (exir/_serialize/_program.py:693-718). Comparing only .program therefore can't see a device-dependent difference in that data or in the layout.

    • Fix: assert remapped == export_for("cuda:0"), or at least compare the whole PTEFiles.
    • The byte comparison should pass for this model, since the tool's serializer arguments match the ExecutorchBackendConfig defaults. I did not run it.

    Fix this →

  • test_cli_input_errors_exit_like_argparse_errors passes no matter which error fires (:392-444). It only checks code == 2.

    • Example: if parse_device_mapping("") stopped raising, the --mapping "" case would still exit with 2, through the "does not cover" error.
    • Directory case: it passes the test's temp directory itself as --output-pte (:435). write_atomically therefore puts its temp file in the system temp dir, where the test can't see whether it was cleaned up.
    • Fix: assert each case's stderr message. For the directory case, use a subdirectory as the output and check that nothing is left behind.

    Fix this →

  • test_second_execution_plan_is_rewritten_too can't tell whether the second plan's tensor was visited (:305-314). That tensor keeps index 7, the same as the first plan's tensor. The result is checked only through collect_cuda_device_indices, which uses the same walk under test.

    • Fix: give that tensor its own index and assert the second plan's fields directly.

    Fix this →

Code Quality

  • write_atomically copies the input's whole mode (backends/cuda/scripts/_atomic_output.py:46). stat.S_IMODE keeps the write bits and the setuid/setgid/sticky bits, though the docstring only asks for the output to stay readable.

    • Effect: a read-only input (0444) gives a read-only output, which a later write can't overwrite in place.
    • Windows (suspected, not verified): os.replace onto that read-only output may also fail.
    • Fix: use (mode_from.stat().st_mode & 0o444) | stat.S_IWUSR, and make the mode check in test_cli_writes_remapped_pte_with_mapping (:469-472) compare read bits only.
    • This was raised in an earlier pass and is still open.

    Fix this →

  • The CLI reports its main caveat through warnings, which needs a private warn_stacklevel knob (backends/cuda/scripts/remap_pte_devices.py:225, :250, :309).

    • Effect: on the command line the caveat prints as .../remap_pte_devices.py:309: UserWarning: ..., followed by the echoed source line. -W ignore or PYTHONWARNINGS=ignore hide it completely.
    • Fix: have _remap_pte_file return the caveat text. remap_pte_bytes then passes it to warnings.warn(..., stacklevel=2), and main() prints it to stderr.
    • This keeps the library warning that an earlier pass asked for, and removes the knob.

    Fix this →

Recommendation

Approve. Nothing blocks.

The CompileSpec aliasing is the one item to settle before merge, either by copying the specs or by narrowing the documented contract. It only affects in-memory callers, because the CLI and remap_pte_bytes work on fresh objects. The other items can be follow-ups.

• Branch: [`cuda-remap-pte-devices`](https://github.com/pytorch/executorch/tree/cuda-remap-pte-devices)

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 21s —— View job


Approve: both optional nits from the ninth pass are in the code, and nothing else is outstanding from my side.

Checking the ninth-pass nits for #25315
  • Read the updated _parse_target_device and the ignored-specs test
  • Check both fixes
  • Post the result

Checked

  • Type names now come from DeviceType. backends/cuda/scripts/remap_pte_devices.py:87 checks against {dt.name.lower() for dt in DeviceType}. If a new member is added to the schema enum, specs naming it go to the shared parser instead of being treated as "unrecognized".
  • Non-UTF-8 values are tolerated. :86 decodes with errors="replace" before the type check. A spec such as b"\xff:0" decodes to a type name that isn't in the enum, so it returns None before reaching _parse_device_spec_value.
  • Test coverage. test_remap_pte_devices.py:196-204 adds an OtherBackend delegate with b"\xff:0" alongside the npu:0 one. The test checks that collection still returns {3, 5, 7, 9} and that the remap goes through without touching the foreign specs.

A small gap that doesn't need a change: the test asserts only that the npu:0 spec keeps its value, not the \xff:0 one. Since a successful remap already means the undecodable spec was skipped, this is fine as it is.

I didn't run the tests, because this job can't run Python. The checks above come from reading the code.

• [`cuda-remap-pte-devices`](https://github.com/pytorch/executorch/tree/cuda-remap-pte-devices)

A CUDA program records the devices from its target_device compile
spec in its planned memory table, in tensor placement info, and in
each delegate's target_device compile spec itself. All three must
agree, so a program exported with an explicit non-zero target_device
(cuda:3, say) does not load on a machine whose GPUs have other
indices, and the loader has no way to move it.

scripts/remap_pte_devices.py rewrites all three places in the file
with one mapping, dense by default (3, 5, 7 become 0, 1, 2, keeping
their order, with a warning unless the result lands entirely on
cuda:0) or explicit (--mapping 3=0,5=0,7=0 collapses everything
onto cuda:0). The output is an ordinary PTE, so the runtime needs
no remapping support of its own. The companion .ptd is untouched:
tensor data carries no device ids. A mapping that does not cover a
recorded device is refused, so a file is never rewritten only in
part.

The tool starts a new backends/cuda/scripts directory for standalone
CUDA backend tools, alongside the library code. Its CPU-only tests
join pytest.ini testpaths and a Buck python_unittest, so serializer
and schema changes run them without a GPU.

Test Plan:
- backends/cuda/tests/test_remap_pte_devices.py (also registered as
  a Buck python_unittest): 22 tests covering collection from all
  three places across multiple execution plans, dense and explicit
  mappings, coverage and validation errors, bare/uppercase/non-CUDA/
  malformed target_device specs, shared compile specs, serialize
  round trips with named data, identity-remap byte equality, failed-write
  atomicity, the not-entirely-cuda:0 warning, an exported program
  (DeviceAwarePartitioner, cuda:3) remapped to equal a native cuda:0
  export, and the CLI dry-run, input-error and write paths. All pass
  locally.
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Fixed the remaining direct-dependency lint issue in test_remap_pte_devices: the Buck test target now directly depends on :scripts_atomic_output and //executorch/exir:scalar_type, matching the modules the test imports directly.

Local checks: repository-configured flake8 and Black pass on the new Python files; the 22-test suite was already green on the prior head (this amend only changes targets.bzl).

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 33m 9s —— View job


Approve: nothing blocks; the only commit since the ninth pass fixes the Buck deps, so three earlier findings are still open (led by remap_program_devices rewriting the partitioner's own CompileSpec objects), and this pass adds three small new ones.

PR Review: #25315 (tenth pass)
  • Gather PR context and the diff (shallow checkout, so the diff is HEAD~1..HEAD, the single PR commit fcf50da)
  • Three independent reviews (all 8 changed files covered)
  • Consolidate findings (8 candidates)
  • Fact-check against the code and earlier passes (6 kept)
  • Post the review

Summary

Adds backends/cuda/scripts/remap_pte_devices.py, a CLI and library that applies one mapping to every CUDA device id a PTE records (planned-buffer devices, tensor extra_tensor_info, delegate target_device specs), plus an atomic writer, tests, a README section and Buck/pytest wiring. Nothing blocks. The only change since the ninth pass is the targets.bzl deps fix, so the three findings marked Still open were raised before and haven't been changed or answered. If one is intended as is, a one-line reply will keep it out of later passes. The other three are new.

  • Three independent reviewers covered all 8 changed files. The checkout is shallow, so the diff is HEAD~1..HEAD (the single PR commit fcf50da).
  • I didn't re-run the 22 tests, because this job isn't allowed to run Python.
  • Not re-raised: the items settled or deferred earlier (dense default, --mapping all=0, a GPU load test, the merge_ptes move to CUDA backend: move merge_ptes into backends/cuda/scripts #25323, the type-name pre-check), and two style and coverage points already raised twice (main() repeating remap_pte_bytes, and CLI error tests that assert only exit code 2).

Code Quality

  • Still open (ninth pass): write_atomically copies the input's whole mode (backends/cuda/scripts/_atomic_output.py:46).
    • A read-only input, such as a 0444 build output, gives a read-only output, and the setuid, setgid and sticky bits come along too. The docstring (:11-13) only asks for the output to stay readable.
    • The mode test doesn't guard the chmod either (backends/cuda/tests/test_remap_pte_devices.py:469-472). The input comes from write_bytes, so under umask 077 both files are 0600 and the test passes with :46 deleted.
    • Fix: copy only the read bits and add owner write, (stat.S_IMODE(...) & 0o444) | stat.S_IWUSR, and chmod the test input to 0o444 first. Fix this →
  • Still open (eighth pass): the subprocess.CalledProcessError catch can never fire (backends/cuda/scripts/remap_pte_devices.py:314, with import subprocess at :35).
    • Moving serialization into the try didn't change this. serialize_pte_binary builds the flatbuffer in Python (exir/_serialize/_program.py:499-505), and deserialize_pte_binary parses it with _flatbuffer_to_program. Only _flatc_compile and _flatc_decompile run flatc (exir/_serialize/_flatbuffer.py:398, :426).
    • An earlier pass of this review said a corrupt flatbuffer raises CalledProcessError from _run_flatc. That was wrong.
    • A corrupt file most likely fails inside the flatbuffers runtime instead (struct.error or similar on a bad offset). Nothing catches that, so the result is still a traceback. This point is low confidence, because I couldn't run it.
    • Fix: drop the catch and the import. If corrupt input should exit with 2, a test with a corrupted PTE will show which exception to catch. Fix this →
  • The module docstring says a default export records no device id (remap_pte_devices.py:9-10, which reads "default cuda:0, so only an explicit non-zero spec records one").
    • A default export records cuda:0 in all three places. CudaPartitioner adds b"cuda:0" when no spec is given (backends/cuda/cuda_partitioner.py:46-52), and test_exported_program_round_trip relies on the native cuda:0 export recording 0.
    • The README gets this right. Fix: "only an explicit non-zero spec records a non-zero id". Fix this →
  • remap_program_devices checks each site's type twice (remap_pte_devices.py:128-133, :200-212).
    • The single-use _set_site_device_index checks isinstance(site, CompileSpec) to write the new id. The loop then checks the type again to bump a string-keyed dict, which is unpacked into the report.
    • Fix: one isinstance check in the loop plus a Counter keyed by type(site) replaces both the helper and the dict. Fix this →

Testing

  • The exported-program check covers only what an Add of two inputs records (backends/cuda/tests/test_remap_pte_devices.py:316-348).
    • The comment at :345-346 says an id the tool doesn't know about would show up as a difference from a native cuda:0 export.
    • But the model has no parameters or buffers. Constants that feed the delegate (exir/passes/propagate_device_pass.py:458-471) and mutable buffers never go through the tool end to end.
    • The comparison also covers only .program, and deserialization has already moved constant offsets, mutable-data segments, named data and segments out of it (exir/_serialize/_program.py:690-716). The ninth pass raised this part.
    • Fix: give the model a parameter and an in-place-updated buffer that reach the delegate through torch.add, because DeviceAwarePartitioner only delegates aten.add.Tensor (exir/backend/test/device_util.py:35-41). Then compare the remapped bytes with the native export's bytes, or at least the whole deserialized PTEFile. Fix this →

API Design

  • Still open (ninth pass): remap_program_devices rewrites CompileSpec objects it doesn't own (remap_pte_devices.py:130, :183-185).
    • The function is public, and the comment at :183-184 invites callers to pass an emitted in-memory program.
    • In that program the specs are the partitioner's own objects. to_backend copies only the list (exir/backend/backend_api.py:117), and the emitter passes that list through (exir/emit/_emitter.py:1553).
    • So after remap_program_devices(manager.executorch_program, {3: 0}), the next export with the same partitioner or spec list targets cuda:0.
    • The remap also never reaches manager.buffer or write_to_file, because the manager serialized when it was built (exir/program/_program.py:1979, :2052-2063).
    • Fix: make the function private and point in-memory users at remap_pte_bytes(manager.buffer), or reply that the in-place contract is intended. Fix this →

Recommendation

Approve. Nothing here blocks:

  • For a valid PTE, the only finding that changes what the CLI writes is the mode copy.
  • The CompileSpec rewrite affects only Python callers that pass an in-memory program.
  • The rest are dead code, a wrong docstring line, a simplification and test coverage.
· Branch: [`cuda-remap-pte-devices`](https://github.com/pytorch/executorch/tree/cuda-remap-pte-devices)

This branch was successfully deployed

1 active deployment
cadence — fcf50dae Deployed Nov 20, 2026 by Gasoonjia via hifi-op-test / hifi4 #39582
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. release notes: devtools Changes to dev tooling, for example the debugger & profiler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants