Repository navigation
Conversation
A `pre-built` build step resolved its module (a local path, or a URL through the package cache) and copied it to the build's scratch `out.wasm`. The build operation then read that file back and wrote the same bytes to the artifact store. The scratch copy had no reader but the build itself, and on a host whose disk is memory (dfinity/imcp2) it cost the module's size in RAM for every canister being built at once. A build step now says where it left the module: `Module::AtOutput`, as before, or `Module::Bytes`, the module itself. The pre-built step reads its source through `FileSystem` and hands the bytes back; the operation saves them straight to the artifact store. Only when another step follows does it write them to the output path first, since that step looks for the module at `ICP_WASM_OUTPUT_PATH` and may change it there. The source file and the shared cache entry are only ever read. Writes of the module's bytes to disk while building one canister whose only step is `pre-built`: source before after local path scratch + artifact (2) artifact (1) URL, cache miss cache + scratch + artifact (3) cache + artifact (2) URL, already cached scratch + artifact (2) artifact (1) Copies on disk at the build's peak, counting the source or cache entry: 3 before, 2 after. A `pre-built` step followed by a script step is unchanged: the scratch copy is what the script works on. Install and deploy read the artifact into memory and write nothing more. The artifact store's own copy stays: it is what `canister install`, `deploy` and `project bundle` read, possibly in a later process, and it is a snapshot of the build rather than a pointer at a file that can change. Sharing it with the source (a hard link, or a reference) would change that contract and is not attempted here. A `pre-built` step whose file cannot be read still fails as that step, now with `failed to read wasm file '<path>'` where it said `failed to copy wasm to output file`. Through a confined filesystem (`--untrusted`) the read is refused for the same paths the copy was. `Builder::new` and `wasm::Fetch` are unchanged, so a consumer that supplies its own `Fetch` to the library's `Builder` needs no change; one that implements `Build` itself returns `Module::AtOutput` to keep what it did.
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
There are no active review issues blocking approval.
No open issues 🎉
What changed in this PR
This PR avoids an unnecessary temporary WASM copy in the canister build pipeline, reducing disk use and memory use on memory-backed filesystems. It preserves the output-file workflow when a later build step needs the module.
Changes:
- Introduces
Moduleso build steps can return bytes or leave their output in a file. - Saves returned bytes directly to the artifact store unless another step follows.
- Adds unit and integration coverage for step ordering, source-file preservation, and read failures.
| File | Description |
|---|---|
crates/icp-project/src/operations/build.rs |
Handles returned bytes, defers temporary writes, and tests step sequencing. |
crates/icp-project/src/canister/build/prebuilt.rs |
Returns module bytes instead of copying; adds read-error tests. |
crates/icp-project/src/canister/build/mod.rs |
Defines Module and updates the build interface and implementations. |
crates/icp-cli/tests/build_adapter_tests.rs |
Verifies a following script modifies the build output without changing the source. |
CHANGELOG.md |
Documents reduced copying and the updated read-error message. |
Review effort: Balanced
…atch update: `Cargo.lock` now resolves wasmtime at 49.0.7 instead of 49.0.5, which is inside the patched range (>=49.0.6, <50) the advisories name. I could not run cargo-deny locally because it is not installed, so the advisory scan itself is unconfirmed until CI re-runs. Invariant: no wasmtime crate in `Cargo.lock` may resolve to a version with an open RustSec advisory. It has to hold for every crate released in lockstep with wasmtime, not just the `wasmtime` package the log flags, so all of them moved together. What changed (only `Cargo.lock`, uncommitted in the worktree): - `wasmtime`, `wasmtime-wasi`, `wasmtime-wasi-io`, `wasmtime-environ` and the `wasmtime-internal-*` crates: 49.0.5 -> 49.0.7 - `pulley-interpreter` and `pulley-macros`: 49.0.5 -> 49.0.7 - the `cranelift-*` crates: 0.136.5 -> 0.136.7 The pulley and cranelift bumps are the ones cargo required to move wasmtime; they are not `wasmtime-*` crates by name, so flagging them against the "wasmtime siblings only" constraint. Nothing else in the lockfile moved, and no entry remains at 49.0.5 or 0.136.5. No `Cargo.toml`, `deny.toml` or source file was touched. The workspace requirement stays `"49.0.5"`, which 49.0.7 satisfies. Verification: - `cargo check --locked -p icp-sync-plugin -p icp-extension` passes, so the lockfile is consistent with the manifests. - `cargo test --locked -p icp-sync-plugin -p icp-extension` passes (55 and 20 tests, 0 failed). The CI log also shows three `advisory-not-detected` warnings for existing zstd ignores in `deny.toml`. They are warnings, not the cause of the failure, and I left them alone
raymondk
marked this pull request as draft
November 20, 2026 03:50
This reverts commit 1b76a77. Handing the module's bytes back and writing them once to the artifact store still left a copy per canister. The approach is replaced by referencing the pre-built file by path, with no copy at all, in the commit that follows. The tree is back to the pre-change base plus the lockfile-only wasmtime 49.0.7 update.
A `pre-built` build step resolved its module (a local path, or a URL
through the package cache), copied it to the build's scratch `out.wasm`,
and the build operation then read that back and wrote the same bytes to
`.icp/cache/artifacts/<canister>`. Two copies per canister of a file
that already exists, and on a host whose disk is memory (dfinity/imcp2)
each of them is RAM: fifty canisters built from one module held it fifty
times over.
A pre-built module is now the build's artifact where it lies. Nothing is
copied:
- A build step says where it left the module: `Module::AtOutput`, as
before, or `Module::Borrowed { path, sha256 }`, a file the build does
not own. The pre-built step returns the resolved file and the SHA-256
of what it read there, through the project's `FileSystem`.
- The build operation hands a borrowed module to the artifact store with
the new `store_artifact::Access::link`, which records the path and the
hash instead of bytes. The host store keeps that as a small JSON file,
`<canister>%link`, beside where a saved module would be; a canister
has one or the other, and no encoded canister name can collide with
the link's.
- `Access::lookup` resolves the link: it reads the file from where it is
and hands it over only if it still hashes to what was linked. Install,
deploy, the candid compatibility check, `canister call`'s offline
candid lookup and `project bundle` all read the artifact through
`lookup`, so none of them changes, and `--untrusted` builds as before.
- A module is copied only when another build step follows the pre-built
one: that step finds it at `ICP_WASM_OUTPUT_PATH` and may change it,
so it gets a private file and the borrowed one is left as it was.
Disk writes of the module's bytes while building one canister whose only
step is `pre-built`:
source before after
local path scratch + artifact (2) none (0)
URL, cache miss cache + scratch + artifact (3) cache (1)
URL, already cached scratch + artifact (2) none (0)
Copies of the module on disk after the build, counting the source or
cache entry: 2 per canister before, 1 in all after, however many
canisters name it. A `pre-built` step followed by a script step is
unchanged (one scratch copy, one saved artifact).
What this gives up is the snapshot. The referenced file is its owner's:
the user's own, or a shared cache entry, never written, moved or removed
by the build. It can change or disappear between `icp build` and a later
`icp canister install`. The store then refuses the lookup, naming the
file and asking for the canister to be built again
(`LookupLinkedChanged`, `LookupLinkedRead`), rather than install a
module other than the one that was built or verified; `install_many`
passes those two errors through instead of reporting a missing artifact.
`icp deploy` and `icp project bundle` build first and are unaffected.
The same check means a link record cannot be used to hand out a file
whose contents its writer did not already know.
A `pre-built` step whose file cannot be read still fails as that step,
now with `failed to read wasm file '<path>'` where it said `failed to
copy wasm to output file`.
`Builder::new` and `wasm::Fetch` are unchanged. `Build::build` now
returns `Module`, and `store_artifact::Access` gains `link`: a consumer
implementing either itself returns `Module::AtOutput` or adds the
method; one using the library's `Builder` and `ArtifactStore` needs no
change.
| if link.exists() { | ||
| let Link { path, sha256 } = | ||
| json::load(&link).map_err(|e| lookup_store(name, e))?; | ||
| let wasm = read(&path).map_err(|e| LookupArtifactError::LookupLinkedRead { |
Comment on lines
+324
to
+325
| remove_if_present(&store.artifact_by_name(name)) | ||
| .map_err(|e| save_store(name, e))?; |
Collaborator
Author
|
Superseded by #887, which keeps each distinct module once in the artifact store instead of linking to the pre-built source file. The scratch copy this PR removed is not addressed there and can follow separately. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Intent
The cli build step processing has some inefficiencies. When the step is a pre build the wasm gets copied twice which is not necessary. Look at the codebase and find an elegant way to fix this - the goal would be to not waste disk space which translates to memory on the current mcp server.
I'm not sure I like the approach - it makes sense to copy the file for a build script step but for the a prebuilt wasm, the path to the prebuilt wasm can be reused without a single copy this way many canisters can use the same prebuilt wasm and point to the same file
Context: "pre build" is the prebuilt build step (crates/icp-project/src/canister/build/prebuilt.rs), which resolves/fetches the wasm (crates/icp-project/src/canister/wasm.rs) and then copies it to the step's output path; the surrounding build pipeline (crates/icp-project/src/canister/build/mod.rs and its callers) then stores the build output again. The MCP server referred to is dfinity/imcp2, which runs these builds where disk is memory-backed, so each extra copy of a wasm costs RAM.
What Changed
pre-builtbuild step no longer copies its WASM to the step output and then again into.icp/cache/artifacts.Build::buildnow returns aModule(AtOutputorBorrowed { path, sha256 }), and the build operation records a borrowed module through a newstore_artifact::Access::link, which writes a small<name>%linkJSON file (path and SHA-256) in place of the module. Canisters built from the same WASM share the one file. The module is still copied to the output path when another build step follows thepre-builtone, since that step may change it.lookupreads a linked module from where it lies and checks it against the recorded SHA-256. If the file has changed or been removed sinceicp build,icp canister installfails with an error that names the file and asks for a rebuild, instead of the generic "Could not find build artifact". Apre-builtstep whose file cannot be read now fails withfailed to read wasm file '<path>'(previouslyfailed to copy wasm to output file).pre-builtchanging only the build's copy). UpdatesCHANGELOG.mdand the build and project-model docs.Cargo.lockmoves wasmtime from 49.0.5 to 49.0.7, with the cranelift and pulley crates that move with it, to clear the RustSec advisories.Risk Assessment
.icpdepend on files outside it.Testing
Built
icpfrom the target commit and drove it by hand in a disposable HOME/ICP_HOME with throwaway projects: shared local pre-built, URL pre-built from a localhost HTTP server, pre-built followed by a script step, switching a canister between script and pre-built, sha256 pin, changed/removed file after build, bundle, and a live deploy/install/call on a managed local network. Also ran the sixpre_built_pathintegration tests inbuild_adapter_tests. All passed; the first URL attempt failed only because my fixture read the server port too early, and passed once re-driven. No UI surface, so evidence is CLI transcripts. The disposable workdir was removed, the network stopped, and the worktree is clean.icp buildon a project with two canisters that name the same local pre-built wasm (synthetic project, 205794-byte vendored wasm):.icpholds no copy, both canisters record the same file, and t….icpcontains onlyfirst%linkandsecond%link(139 bytes each) plus an empty lock; both nameshared/module.wasmwith sha256 17a05e36...; source hash identica…icp buildon two canisters with the sameurlpre-built (localhost HTTP server): the project holds no copy and both canisters point at the single cached download under ICP_HOME%linkfiles (216 bytes) namehome/icp/pkg/wasms/17a05e36.../module.wasm; that is the only wasm-sized file under ICP_HOMEicp deployto a managed local network for two canisters sharing one pre-built wasm, then call each: both install from the shared file and answergreetreturns("Hello, first!")and("Hello, second!");canister statusmodule hash 0x17a05e36... eq…icp buildand then a separateicp canister install --mode reinstallon the local network: install reads the module through the recorded pathCanister second installed successfully, thengreetreturns("Hello, again!")icp build, then runicp canister install: it is refused, naming the file and asking for a rebuildhas changed since ... build the canister again, thencan no longer be read; build the canister again) and live-transcript.txt S2 (same refusal agains…icp buildwhen the pre-built file is missing: the build fails and names the filefailed to read wasm file '.../s3/module.wasm'$ICP_WASM_OUTPUT_PATH: the artifact is a modified copy and the user's source file is untouchedmy-canisteris 205802 bytes (source 205794 + 8) ending inappended; source sha256 unchangedicp buildeach time: the artifacts directory holds exactly one of the copied module or the link, never bothmy-canister(205794 bytes) ->my-canister%link(139 bytes) ->my-canister(205794 bytes)icp buildwith a wrongsha256on a local pre-built, then with the right one: the wrong pin fails and leaves nothing behind, the right pin builds with a link onlychecksum mismatch, expected: 000..., project contains onlyicp.yaml; after correcting, exit 0 and onlymy-canister%linkicp project bundleon the project with two canisters sharing a pre-built wasm: the bundle contains the module onceicp.yamland a singlecanisters/17a05e36....wasmof 205794 bytespre_built_pathintegration tests inbuild_adapter_testspass against the built binarycargo test --locked -p icp-cli --test build_adapter_tests pre_built_path— 6 passed, 0 failedEvidence: Offline scenarios transcript (shared local pre-built, changed/removed file, script after pre-built, switching, sha256 pin, bundle)
Evidence: URL source and live local-network transcript (deploy, call, status, reinstall, refusal)
Evidence: Targeted integration test output
Evidence: Driver script for the offline scenarios
Evidence: Driver script for the URL and live network scenarios
Evidence: Two canisters, one pre-built wasm: what `icp build` leaves in `.icp`
Evidence: Live deploy from the shared pre-built, and refusal after the file changes
Pipeline
Updates from git push no-mistakes
✅ **Intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
crates/icp-project/src/store_artifact.rs:110- Simplification: the SHA-256 pin on a linked module is not required by the intent, which asks only that "the path to the prebuilt wasm can be reused without a single copy". The change adds a hash recorded at build and a refusal at lookup when the file has changed. This is new user-visible behaviour:icp canister installnow fails with "has changed since … build the canister again" after the user edits or replaces their own wasm. It also costs a full read of the module into memory on every pre-built step just to hash it (crates/icp-project/src/canister/build/prebuilt.rs:45-49), and a local source withsha256:set is read and hashed twice (crates/icp-app/src/wasm.rs:99 and again in the step). Parts of this component:Borrowed.sha256(crates/icp-project/src/canister/build/mod.rs:57),verify_linkedandLookupLinkedChanged(store_artifact.rs:95-127),Link.sha256(store_artifact.rs:198), the error mapping in crates/icp-project/src/operations/install.rs:354-364. The narrower form is to record only the path and have lookup read whatever is there. Whether a changed file should be refused or installed as-is is a product decision; please confirm you want the pin.crates/icp-project/src/store_artifact.rs:307-linkrefuses a canister whose encoded name plus%linkexceeds 255 bytes (encoded name of 251–255 bytes). Such a canister with a pre-built last step built before this change (viasave) and now failsicp buildwith SaveNameTooLong. It is only reachable with ~250-byte canister names, so this is noted and not blocking; the testa_name_with_no_room_for_a_link_is_refused_onepins it as intended.crates/icp-project/src/operations/build.rs:105- Tradeoff inherent to the requested design:.icp/cache/artifactsis no longer self-contained for pre-built canisters. The link stores an absolute path to the user's file or to the global package cache entry for aurlsource. Soicp buildon one machine or CI job followed byicp canister installon another (carrying only the project and.icp), or after moving the project or clearing the package cache, now fails with the "can no longer be read; build the canister again" error.icp deployandicp project bundlerebuild first (deploy.rs:239, bundle.rs:495) and are unaffected. The CHANGELOG and docs describe this accurately. Also,store_artifact::Accessgained a requiredlinkmethod andBuild::buildchanged its return type; both are breaking for external implementors of the publishedicp-projectcrate (such as imcp2), which CLAUDE.md permits.✅ **Test** - passed
✅ No issues found.
icp buildon a project with two canisters that name the same local pre-built wasm (synthetic project, 205794-byte vendored wasm):.icpholds no copy, both canisters record the same file, and t….icpcontains onlyfirst%linkandsecond%link(139 bytes each) plus an empty lock; both nameshared/module.wasmwith sha256 17a05e36...; source hash identica…icp buildon two canisters with the sameurlpre-built (localhost HTTP server): the project holds no copy and both canisters point at the single cached download under ICP_HOME%linkfiles (216 bytes) namehome/icp/pkg/wasms/17a05e36.../module.wasm; that is the only wasm-sized file under ICP_HOMEicp deployto a managed local network for two canisters sharing one pre-built wasm, then call each: both install from the shared file and answergreetreturns("Hello, first!")and("Hello, second!");canister statusmodule hash 0x17a05e36... eq…icp buildand then a separateicp canister install --mode reinstallon the local network: install reads the module through the recorded pathCanister second installed successfully, thengreetreturns("Hello, again!")icp build, then runicp canister install: it is refused, naming the file and asking for a rebuildhas changed since ... build the canister again, thencan no longer be read; build the canister again) and live-transcript.txt S2 (same refusal agains…icp buildwhen the pre-built file is missing: the build fails and names the filefailed to read wasm file '.../s3/module.wasm'$ICP_WASM_OUTPUT_PATH: the artifact is a modified copy and the user's source file is untouchedmy-canisteris 205802 bytes (source 205794 + 8) ending inappended; source sha256 unchangedicp buildeach time: the artifacts directory holds exactly one of the copied module or the link, never bothmy-canister(205794 bytes) ->my-canister%link(139 bytes) ->my-canister(205794 bytes)icp buildwith a wrongsha256on a local pre-built, then with the right one: the wrong pin fails and leaves nothing behind, the right pin builds with a link onlychecksum mismatch, expected: 000..., project contains onlyicp.yaml; after correcting, exit 0 and onlymy-canister%linkicp project bundleon the project with two canisters sharing a pre-built wasm: the bundle contains the module onceicp.yamland a singlecanisters/17a05e36....wasmof 205794 bytespre_built_pathintegration tests inbuild_adapter_testspass against the built binarycargo test --locked -p icp-cli --test build_adapter_tests pre_built_path— 6 passed, 0 failedcargo build --bin icp --locked(exit 0)bash drive.sh:icp buildon two canisters sharing one local pre-built wasm, then listing.icpand reading the%linkrecordsicp buildthenicp canister installafter rewriting and after deleting the pre-built file;icp buildwith the file missingicp buildwith apre-builtstep followed by ascriptstep appending to$ICP_WASM_OUTPUT_PATHicp buildthree times while switching one canister script -> pre-built -> script, listing.icp/cache/artifactseach timeicp buildwith a wrong and then correctsha256on a local pre-builticp project bundle --outputon the shared pre-built project, thentar tzvfbash drive2.sh:icp buildon two canisters with the sameurlpre-built served by a localhostpython3 -m http.servericp network start -d,icp cycles mint,icp deploy,icp canister call ... greet,icp canister status,icp build+icp canister install --mode reinstall, then install after changing the shared file,icp network stopcargo test --locked -p icp-cli --test build_adapter_tests pre_built_path(6 passed)✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.