Repository navigation
Make save raise on failure; let save move file-backed data into the output instead of copying it - #25099
Make save raise on failure; let save move file-backed data into the output instead of copying it#25099shoumikhin wants to merge 2 commits into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25099
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit e689a97 with merge base cd12c4d ( BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@claude review effort=max |
|
Claude finished @shoumikhin's task in 28m 14s —— View job Request changes: the first write to reach a shared PR Review: #25099SummaryThe PR adds a The main problem is the design:
How this was checked: by reading the code and the CPython stdlib sources. The job wasn't allowed to run Python, so nothing was executed. pytorch/TensorRT#5045 was not reviewed. Backward Compatibility
Code Quality
Testing
Performance
Specific Comments
RecommendationRequest Changes What blocks approval:
|
9fab878 to
fcfc152
Compare
|
@claude review effort=max |
Review of PR #25099Request changes: after a consuming save, a second or retried PR Review: #25099SummaryThe PR adds an opt-in The main problems:
Python commands aren't permitted in this job, so the tests weren't run. Everything below comes from reading the code. Code Quality
API Design
Backward Compatibility
Testing
Performance
Specific Comments
RecommendationRequest Changes. Four findings can each lose data that only a full re-export brings back, and each has a small fix:
The Windows test failures will also block CI.
|
fcfc152 to
a97f7bf
Compare
|
@claude review effort=max |
|
Claude finished @shoumikhin's task in 40m 34s —— View job PR Review: #25099
Request Changes: a write after a consuming write is only caught after the output has been opened with Detailed reviewSummaryThe PR adds
Code Quality
Testing
API Design
Performance
Specific Comments
RecommendationRequest Changes Two things block this PR:
The silent copy and the sync per chunk are open to discussion. But on a large model they decide whether |
a97f7bf to
4c8247d
Compare
|
@claude review effort=max |
|
Claude finished @shoumikhin's task in 37m 13s —— View job Review of PR #25099Request Changes: if a consuming Detailed reviewPR Review: #25099SummaryThe PR adds Request Changes, for two reasons:
Points the previous review raised that are still open are marked (still open). I couldn't run the tests because running Python in this job needs approval, so every finding comes from reading the code. Code Quality
Testing
API Design
Performance
Specific Comments
Fix this → (all three) RecommendationRequest Changes. The killed-save problem under Code Quality blocks approval. Because the move writes in place, an interrupted consuming save becomes a Review steps
|
…tead of copying them `compile_and_save` writes each engine to a temporary file as soon as it is built. The ExecuTorch save then copies that file into the `.pte` (or the `.ptd`), and the file is deleted only when the call returns. So while the program is written, every engine is on disk twice. For a model whose engine is tens of gigabytes, the copy needs that much more free disk, and an export can fail on a device that has room for the result. The backend now hands each of these engine files to ExecuTorch as data that owns the file (`FileBackedData(path, cleanup=True)`); it is deleted after the save anyway. Inside `compile_and_save`, the save then writes the program and the `.ptd` with `consume=True`, so ExecuTorch moves each engine into the program: it copies it back to front and truncates the file as it goes, and each engine is on disk once. ExecuTorch does this only when the temporary directory, where the engine files are, is on the same file system as the program; otherwise it copies. With `generate_etrecord`, the save copies the engines instead. The ETRecord reads the engines after the program is written, so the save order stays the same as before, and the program is kept when the ETRecord can't be written. Every ExecuTorch save now passes `consume` (as `False` outside `compile_and_save`), so this needs the `consume` argument from pytorch/executorch#25099. It lands after that change, together with an ExecuTorch pin that has it, which also raises the minimum ExecuTorch version. Test plan: - `tests/py/dynamo/executorch/test_api.py`, CPU, with the lowering stubbed: - New `test_compile_and_save_moves_the_engines`: inside `compile_and_save`'s offload block, the program and the `.ptd` are written with `consume=True`. It fails without this change. - New `test_compile_and_save_keeps_the_program_when_the_etrecord_fails`: in the same block with an ETRecord, the program and the `.ptd` are written first, without `consume`, and the `.pte` is kept when the ETRecord save raises. - New `test_save_executorch_keeps_the_program_when_the_etrecord_fails`: the same for a plain save. - New `test_save_executorch_does_not_consume`: a plain save writes without `consume`. - `tests/py/dynamo/executorch/test_compile_and_save.py` on an A100: - The streaming test now also checks that each engine reaches ExecuTorch as data that owns its file. - New `test_engines_move_into_the_program_without_a_second_copy` checks that each engine file is empty after the program is written. - New `test_an_etrecord_makes_the_save_copy_the_engines`: with an ETRecord, the engine files keep their bytes, and both the `.pte` and the ETRecord are written.
|
@claude review |
|
Claude finished @shoumikhin's task in 5m 49s —— View job Needs Discussion: I found no data-loss bug at f3d4581, but PR Review: #25099SummaryThe PR makes API Design
Code Quality
RecommendationNeeds Discussion No correctness blocker is left. Before merging, the open design questions are:
The rename race is a small fix that should go in with this PR.
|
…tead of copying them `compile_and_save` writes each engine to a temporary file as soon as it is built. The ExecuTorch save then copies that file into the `.pte` (or the `.ptd`), and the file is deleted only when the call returns. So while the program is written, every engine is on disk twice. For a model whose engine is tens of gigabytes, the copy needs that much more free disk, and an export can fail on a device that has room for the result. The backend now hands each of these engine files to ExecuTorch as data that owns the file (`FileBackedData(path, cleanup=True)`); it is deleted after the save anyway. Inside `compile_and_save`, the program is then written with ExecuTorch's `save(file_path, consume=True)` and the `.ptd` with `write_tensor_data_to_file(..., consume=True)`, so ExecuTorch moves each engine into the program: it copies it back to front and truncates the file as it goes, so the engines are on disk once (plus one chunk) instead of twice, on a file system with sparse files. ExecuTorch moves data only into a new file, and only when the temporary directory, where the engine files are, is on the same file system as the program; otherwise, as when exporting again onto an existing file, it copies. Its `save` takes only a `.pte` path, so a program saved under another name is written as before, with a copy; the `.ptd` still moves. ExecuTorch is an optional extra, and releases before pytorch/executorch#25099, such as 1.6.0, have no `consume` argument. Both writes check whether ExecuTorch's method takes it, through one small helper, and copy as before when it doesn't. That fallback writes the program through `open` and `write_to_file`, as before this change, so a failed write still raises. The move is slower than a copy, because ExecuTorch syncs each chunk to disk before the engine file gives it up: the save takes 2.5 to 9 times as long as a copy (measured on btrfs). The `compile_and_save` docstring says so, with the conditions for the move. If a move stops partway, ExecuTorch keeps its partial output, `<name>.partial`, and its error note says which part of an engine is still in its file. The engine files are temporary and deleted when the offload block ends, as before, so the partial output can't be completed. When some engine file shrank and some still has bytes, the block adds a note that the engine files are deleted, so an output that isn't complete has to be deleted (any partial file ExecuTorch's note names) and exported again. (ExecuTorch never replaces a partial output, since another save may still be writing it; while it is there, a save to that path copies and its warning names it.) An engine file the cleanup can't delete is now logged as a warning, so it can't hide the error that ended the block, and a save that succeeded doesn't fail over it. With `generate_etrecord`, the save copies the engines instead. The ETRecord is written after the program and reads the engines, so a move would leave it nothing to read. The program is still written first, as before. The `save` docstring now says that `compile_and_save` can write an ETRecord. The move needs the `consume` argument of `save` and `write_tensor_data_to_file` from pytorch/executorch#25099; with an ExecuTorch without it, `compile_and_save` copies as before. Only the move passes `consume`, so every other save calls ExecuTorch exactly as before. Test plan: - `tests/py/dynamo/executorch/test_api.py`, CPU, with the lowering stubbed: - New `test_compile_and_save_moves_the_engines`: inside `compile_and_save`'s offload block, the program and the `.ptd` are written with `consume=True`. It fails without this change. - New `test_compile_and_save_moves_only_the_ptd_into_a_path_without_pte`: the same block with a `.bin` path writes the program without `consume` and the `.ptd` with it. - New `test_compile_and_save_copies_with_an_executorch_without_consume`: the same block, with an ExecuTorch whose `save` and `write_tensor_data_to_file` take no `consume`, writes the program through `write_to_file` and the `.ptd` without `consume`. It fails without the check. - New `test_offload_block_deletes_the_engine_files_of_a_failed_move`: when the block ends with an error, every engine file is deleted, and the note to export again is added only when the move stopped partway: inside an engine, or between engines, but not before any move and not when every engine moved. - New `test_offload_block_survives_an_engine_file_it_cannot_check`: an engine file whose size can't be read and that can't be deleted doesn't replace the error that ended the block, and the other file is deleted. - New `test_offload_block_warns_about_an_engine_file_it_cannot_delete`: after a successful block, such a file is logged as a warning instead of raising. - New `test_compile_and_save_keeps_the_program_when_the_etrecord_fails`: in the same block with an ETRecord, the program and the `.ptd` are written first, without `consume`, and the `.pte` is kept when the ETRecord save raises. - New `test_save_executorch_keeps_the_program_when_the_etrecord_fails`: the same for a plain save. - New `test_save_executorch_does_not_consume`: a plain save writes without `consume`. The stubbed ExecuTorch raises TypeError for any `consume` other than `True`, as releases without it do, so this test fails if a plain save passes `consume=False`. - `tests/py/dynamo/executorch/test_compile_and_save.py` on an A100: - The streaming test now also checks that each engine reaches ExecuTorch as data that owns its file. - New `test_engines_move_into_the_program_without_a_second_copy` checks that ExecuTorch copied no engine and that both are in the `.pte`, or in the `.ptd` with `external_engine_data`. It is skipped with an ExecuTorch whose `save` takes no `consume`. - New `test_an_etrecord_makes_the_save_copy_the_engines`: with an ETRecord, the engine files keep their bytes, and both the `.pte` and the ETRecord are written.
…tead of copying them `compile_and_save` writes each engine to a temporary file as soon as it is built. The ExecuTorch save then copies that file into the `.pte` (or the `.ptd`), and the file is deleted only when the call returns. So while the program is written, every engine is on disk twice. For a model whose engine is tens of gigabytes, the copy needs that much more free disk, and an export can fail on a device that has room for the result. The backend now hands each of these engine files to ExecuTorch as data that owns the file (`FileBackedData(path, cleanup=True)`); it is deleted after the save anyway. Inside `compile_and_save`, the program is then written with ExecuTorch's `save(file_path, consume=True)` and the `.ptd` with `write_tensor_data_to_file(..., consume=True)`, so ExecuTorch moves each engine into the program: it copies it back to front and truncates the file as it goes, so the engines are on disk once (plus one chunk) instead of twice, on a file system with sparse files. ExecuTorch moves data only into a new file, and only when the temporary directory, where the engine files are, is on the same file system as the program; otherwise, as when exporting again onto an existing file, it copies. Its `save` takes only a `.pte` path, so a program saved under another name is written as before, with a copy; the `.ptd` still moves. ExecuTorch is an optional extra, and releases before pytorch/executorch#25099, such as 1.6.0, have no `consume` argument. Both writes check whether ExecuTorch's method takes it, through one small helper, and copy as before when it doesn't. That fallback writes the program through `open` and `write_to_file`, as before this change, so a failed write still raises. The move is slower than a copy, because ExecuTorch syncs each chunk to disk before the engine file gives it up: the save takes 2.5 to 9 times as long as a copy (measured on btrfs). The `compile_and_save` docstring says so, with the conditions for the move. If a move stops partway, ExecuTorch keeps its partial output, `<name>.partial`, and its error note says which part of an engine is still in its file. The engine files are temporary and deleted when the offload block ends, as before, so the partial output can't be completed. When some engine file shrank and some still has bytes, the block adds a note that the engine files are deleted, so an output that isn't complete has to be deleted (any partial file ExecuTorch's note names) and exported again. (ExecuTorch never replaces a partial output, since another save may still be writing it; while it is there, a save to that path copies and its warning names it.) An engine file the cleanup can't delete is now logged as a warning, so it can't hide the error that ended the block, and a save that succeeded doesn't fail over it. With `generate_etrecord`, the save copies the engines instead. The ETRecord is written after the program and reads the engines, so a move would leave it nothing to read. The program is still written first, as before. The `save` docstring now says that `compile_and_save` can write an ETRecord. The move needs the `consume` argument of `save` and `write_tensor_data_to_file` from pytorch/executorch#25099; with an ExecuTorch without it, `compile_and_save` copies as before. Only the move passes `consume`, so every other save calls ExecuTorch exactly as before. Test plan: - `tests/py/dynamo/executorch/test_api.py`, CPU, with the lowering stubbed: - New `test_compile_and_save_moves_the_engines`: inside `compile_and_save`'s offload block, the program and the `.ptd` are written with `consume=True`. It fails without this change. - New `test_compile_and_save_moves_only_the_ptd_into_a_path_without_pte`: the same block with a `.bin` path writes the program without `consume` and the `.ptd` with it. - New `test_compile_and_save_copies_with_an_executorch_without_consume`: the same block, with an ExecuTorch whose `save` and `write_tensor_data_to_file` take no `consume`, writes the program through `write_to_file` and the `.ptd` without `consume`. It fails without the check. - New `test_offload_block_deletes_the_engine_files_of_a_failed_move`: when the block ends with an error, every engine file is deleted, and the note to export again is added only when the move stopped partway: inside an engine, or between engines, but not before any move and not when every engine moved. - New `test_offload_block_survives_an_engine_file_it_cannot_check`: an engine file whose size can't be read and that can't be deleted doesn't replace the error that ended the block, and the other file is deleted. - New `test_offload_block_warns_about_an_engine_file_it_cannot_delete`: after a successful block, such a file is logged as a warning instead of raising. - New `test_compile_and_save_keeps_the_program_when_the_etrecord_fails`: in the same block with an ETRecord, the program and the `.ptd` are written first, without `consume`, and the `.pte` is kept when the ETRecord save raises. - New `test_save_executorch_keeps_the_program_when_the_etrecord_fails`: the same for a plain save. - New `test_save_executorch_does_not_consume`: a plain save writes without `consume`. The stubbed ExecuTorch raises TypeError for any `consume` other than `True`, as releases without it do, so this test fails if a plain save passes `consume=False`. - `tests/py/dynamo/executorch/test_compile_and_save.py` on an A100: - The streaming test now also checks that each engine reaches ExecuTorch as data that owns its file. - New `test_engines_move_into_the_program_without_a_second_copy` checks that ExecuTorch copied no engine and that both are in the `.pte`, or in the `.ptd` with `external_engine_data`. It is skipped with an ExecuTorch whose `save` takes no `consume`. - New `test_an_etrecord_makes_the_save_copy_the_engines`: with an ETRecord, the engine files keep their bytes, and both the `.pte` and the ETRecord are written. Signed-off-by: Anthony Shoumikhin <[email protected]>
…tead of copying them `compile_and_save` writes each engine to a temporary file as soon as it is built. The ExecuTorch save then copies that file into the `.pte` (or the `.ptd`), and the file is deleted only when the call returns. So while the program is written, every engine is on disk twice. For a model whose engine is tens of gigabytes, the copy needs that much more free disk, and an export can fail on a device that has room for the result. The backend now hands each of these engine files to ExecuTorch as data that owns the file (`FileBackedData(path, cleanup=True)`); it is deleted after the save anyway. Inside `compile_and_save`, the program is then written with ExecuTorch's `save(file_path, consume=True)` and the `.ptd` with `write_tensor_data_to_file(..., consume=True)`, so ExecuTorch moves each engine into the program: it copies it back to front and truncates the file as it goes, so the engines are on disk once (plus one chunk) instead of twice, on a file system with sparse files. ExecuTorch moves other data that owns its file the same way, such as the CUDA backend's weights. It moves data only into a new file, and only when the temporary directory, where the engine files are, is on the same file system as the program; otherwise, as when exporting again onto an existing file, it copies. Its `save` takes only a `.pte` path, so a program saved under another name is written as before, with a copy; the `.ptd` still moves. ExecuTorch is an optional extra, and releases before pytorch/executorch#25099, such as 1.6.0, have no `consume` argument. Both writes check whether ExecuTorch's method takes it, through one small helper, and copy as before when it doesn't. That fallback writes the program through `open` and `write_to_file`, as before this change, so a failed write still raises. The move is slower than a copy, because ExecuTorch syncs each chunk to disk before the engine file gives it up: the save takes 2.5 to 9 times as long as a copy (measured on btrfs). The `compile_and_save` docstring says so, with the conditions for the move. If a move stops partway, ExecuTorch keeps its partial output, `<name>.partial`, which holds the moved data, while the rest is still in an engine file. The engine files are temporary and are deleted when `compile_and_save` returns, so that output can never be finished. While it is there, ExecuTorch copies on every later save to the same path, and a successful retry leaves it on disk next to the new program. So the consuming writes now run inside `moving_engines_into`: when the save fails after some engine data left its file and some did not, it deletes each `.partial` file this save created, and a note on the error says to export again. A partial file that was there before the save is never touched, and one whose engines all moved is left with ExecuTorch's own note, since it may be complete. Afterwards it empties the plan of each engine the save moved, in part or in full: that plan maps an emptied file, so a later read of it through its module, such as a second save in the block, killed the process with a bus error. Now the plan is empty and that read raises. An engine file the cleanup can't delete is logged as a warning, so it can't hide the error that ended the block, and a save that succeeded doesn't fail over it. The program path goes through `os.fspath` first, so a `pathlib.Path` still works. With `generate_etrecord`, the save copies the engines instead. The ETRecord is written after the program and reads the engines, so a move would leave it nothing to read. The program is still written first, as before. The docs now say why `compile_and_save` can write an ETRecord when a plain save of a program with TensorRT engines can't: it keeps each engine as a byte buffer read from its file. The move needs the `consume` argument of `save` and `write_tensor_data_to_file` from pytorch/executorch#25099; with an ExecuTorch without it, `compile_and_save` copies as before. Only the move passes `consume`, so every other save calls ExecuTorch exactly as before. Test plan: - `tests/py/dynamo/executorch/test_api.py`, CPU, with the lowering stubbed: - New `test_save_in_the_offload_block_moves_the_engines`: inside `compile_and_save`'s offload block, the program and the `.ptd` are written with `consume=True`. The path is a `pathlib.Path`. - New `test_save_in_the_offload_block_moves_only_the_ptd_into_a_path_without_pte`: the same block with a `.bin` path writes the program without `consume` and the `.ptd` with it. - New `test_save_in_the_offload_block_copies_with_an_executorch_without_consume`: with an ExecuTorch whose writes take no `consume`, the program goes through `write_to_file` and the `.ptd` is written without `consume`. - New `test_a_failed_move_in_the_offload_block_deletes_its_partial_output`: a consuming `.pte` or `.ptd` write that stops inside an engine leaves no partial file, and the error says to export again. - New `test_a_failed_move_deletes_a_partial_output_it_cannot_finish`: the partial file is deleted only when the move stopped partway (inside an engine or between engines), not before any move and not when every engine moved; a partial file from before the save is kept; the engine files are deleted in every case. - New `test_a_save_empties_the_plans_of_the_engines_it_moved`: after a save, a moved engine's plan is empty and no longer found as an engine file, the other one is untouched, and a second save passes over the moved one. - New `test_offload_block_deletes_the_engine_files_when_an_engine_write_fails`, plus the warning and unreadable-size cases for engine files the cleanup can't check or delete. - New `test_save_in_the_offload_block_keeps_the_program_when_the_etrecord_fails` and `test_save_executorch_does_not_consume`. - `tests/py/dynamo/executorch/test_compile_and_save.py` on an A100 with TensorRT 11.4, with ExecuTorch 1.6 and with ExecuTorch from pytorch/executorch#25099: - The streaming test also checks that each engine reaches ExecuTorch as data that owns its file. - New `test_engines_move_into_the_program_without_a_second_copy`, new `test_an_etrecord_makes_the_save_copy_the_engines`, and new `test_a_failed_move_leaves_no_partial_output` (the disk fills after the first engine moved; no partial file is left, and exporting again moves instead of copying). They need an ExecuTorch that moves data. The skip checks for `Cord.file_backed_buffers` as well as `consume`, so an ExecuTorch that renames `consume` fails them instead of skipping them. Signed-off-by: Anthony Shoumikhin <[email protected]>
`ExecutorchProgramManager.save` caught any error from the write, logged it and returned normally. Both `save_pte_program` helpers (the shared one in `extension/export_util` and the Cadence one) did the same, and the shared one then returned the path of a file that might be empty or cut short. A caller had no way to notice, short of reading the log. Now the error reaches the caller. `save` still logs the path when it succeeds. Both `save_pte_program` helpers write the `.pte` through `save` instead of repeating its code, so all of them behave the same way. Behavior change for callers: - `save` and `save_pte_program` raise on a failed write, where they used to log and return. - `save_pte_program` needs an object with `save`, as its type hint already says (`ExecutorchProgramManager`). The legacy `ExecutorchProgram` from the deprecated `exir.capture(...).to_edge().to_executorch()` path has no `save`, so passing it now raises `AttributeError`. Such code can call the legacy object's own `write_to_file`. Test plan: `exir/program/test/test_program.py`. New tests check that `save` raises when the write fails, and that `save_pte_program` writes the `.pte` and `.ptd` files, returns the path, and raises when either write fails. They fail on the code before this change. The Cadence helper has no test target, so it was checked by hand.
`FileBackedData` keeps a large blob, such as a delegate's compiled engine, in a file until the program is written. Writing it copies that file into the `.pte` or `.ptd`, so the blob is on disk twice until the caller deletes the file. For an engine of tens of gigabytes, that copy alone can make an export run out of disk. This adds a keyword-only `consume=True` to `ExecutorchProgramManager.save`, `write_tensor_data_to_file` and both `save_pte_program` helpers. With it, data that owns its file (`cleanup=True`, as from `move_from`) moves into the output: - The output is written to a new temporary file next to `path`. Everything except the moved data, including data that can't move, is written first, so a full disk shows up before any source changes. - Each moved buffer is then copied from its end back to its start, one chunk at a time. After each chunk is synced, it is cut off the source file. On file systems with sparse files, peak disk use is the data once plus one chunk. - An existing file at `path` is removed before the move, so a rerun into the same output also moves. At the end the temporary file is hard-linked to `path` and removed, so a file that appears at `path` during the move is not overwritten. - The cost is time, because of the sync per chunk. Data is copied instead, with a warning, when it does not own its file, the file is a link, has another name or can't be written, the program uses the same file more than once, or `path` is a link or a device. If a save fails after data moved, a note on the error names the file that holds it. Other behavior changes: - Reading a `FileBackedData` whose file changed size, for example because it was moved, raises instead of returning short data. So does writing a program again after a consuming save. - `save` and `write_tensor_data_to_file` raise `ValueError` when the output is the file that holds the data to write, instead of emptying it. - `copy.deepcopy` of a `FileBackedData` returns the same object, so a program deep-copied for an ETRecord can still move its data. Before, the copy could not be read once the original was collected. `write_to_file`, which takes a file the caller opened, always copies, as before. Test plan: `exir/_serialize/test/test_cord.py` and `exir/program/test/test_program.py` on Linux x86_64, including a save through `to_edge_transform_and_lower` that moves backend data into a `.pte` and a `.ptd`, a full disk before and during the move, a file created at the output during the move, a rerun into an existing output, and each copy case. Each new test fails when the code it covers is removed.
f3d4581 to
e689a97
Compare
|
@claude review |
|
Claude finished @shoumikhin's task in 4m 53s —— View job Needs Discussion: I found no data-loss bug in the move at 905c4bc+2 (HEAD), but the PR Review: #25099SummaryThe PR makes API Design
Backward Compatibility
Code Quality
Testing
Performance
RecommendationNeeds Discussion. Nothing blocks on correctness of the move itself. Fix the ETRecord docstring before merge, since it tells users to do something that still fails. Decide whether replacing the output's inode (and dropping its other hard links) is acceptable for
|
|
@claude review effort=max |
|
Claude finished @shoumikhin's task in 53m 28s —— View job Request Changes: a consuming PR Review: #25099SummaryThis PR makes Code Quality
Testing
API Design
Thread Safety
Performance
Specific Comments
RecommendationRequest Changes The partial-failure state is the blocker. The missing |
1. Make
saveandsave_pte_programraise when a write failsExecutorchProgramManager.saveand bothsave_pte_programhelpers logged a failed write and returned, so the caller could get a path to an empty or cut-short file. Now they raise, and both helpers write the.ptethroughsave.The legacy
ExecutorchProgramhas nosave, sosave_pte_programnow raisesAttributeErrorfor it.2. Let
savemove file-backed data into the output instead of copying itFileBackedDatakeeps a large blob, such as a delegate's compiled engine, in a file until the program is written. Writing copies it into the.pteor.ptd, so the blob is on disk twice. For a large engine, that can fill the disk.This adds a keyword-only
consume=Truetosave,write_tensor_data_to_fileand bothsave_pte_programhelpers:Data that owns its file (as from
FileBackedData.move_from) then moves:pathis removed first, so a rerun into the same output also moves. At the end the temporary file is hard-linked topath, so a file created there during the move is not overwritten.Data that can't move safely is copied, with a warning. If a save fails after data moved, a note on the error names the file that holds it. The sync per chunk makes the save slower.
Other behavior changes:
FileBackedDatawhose file changed size raises instead of returning short data.ValueErrorinstead of emptying it.copy.deepcopyof aFileBackedDatareturns the same object. Before, the copy could not be read once the original was collected.Test plan
test_cord.pyandtest_program.pyon Linux x86_64, for each commit. New tests cover a real lowering that moves backend data into a.pteand a.ptd, a full disk before and during the move, a file created at the output during the move, a rerun into an existing output, and each copy case. Each fails when the code it covers is removed.