Skip to content

feat(build): reference a pre-built WASM by path instead of copying it - #883

Closed
raymondk wants to merge 4 commits into
mainfrom
fm/icp-cli-prebuilt-copy-p6
Closed

raymondk wants to merge 4 commits into
mainfrom
fm/icp-cli-prebuilt-copy-p6

Conversation

@raymondk

@raymondk raymondk commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Intent

The cli build step processing has some inefficiencies. When the step is a pre build the wasm gets copied twice which is not necessary. Look at the codebase and find an elegant way to fix this - the goal would be to not waste disk space which translates to memory on the current mcp server.

I'm not sure I like the approach - it makes sense to copy the file for a build script step but for the a prebuilt wasm, the path to the prebuilt wasm can be reused without a single copy this way many canisters can use the same prebuilt wasm and point to the same file

Context: "pre build" is the prebuilt build step (crates/icp-project/src/canister/build/prebuilt.rs), which resolves/fetches the wasm (crates/icp-project/src/canister/wasm.rs) and then copies it to the step's output path; the surrounding build pipeline (crates/icp-project/src/canister/build/mod.rs and its callers) then stores the build output again. The MCP server referred to is dfinity/imcp2, which runs these builds where disk is memory-backed, so each extra copy of a wasm costs RAM.

What Changed

  • A pre-built build step no longer copies its WASM to the step output and then again into .icp/cache/artifacts. Build::build now returns a Module (AtOutput or Borrowed { path, sha256 }), and the build operation records a borrowed module through a new store_artifact::Access::link, which writes a small <name>%link JSON file (path and SHA-256) in place of the module. Canisters built from the same WASM share the one file. The module is still copied to the output path when another build step follows the pre-built one, since that step may change it.
  • lookup reads a linked module from where it lies and checks it against the recorded SHA-256. If the file has changed or been removed since icp build, icp canister install fails with an error that names the file and asks for a rebuild, instead of the generic "Could not find build artifact". A pre-built step whose file cannot be read now fails with failed to read wasm file '<path>' (previously failed to copy wasm to output file).
  • Adds unit tests for the pre-built step, the build operation and the artifact store, plus three CLI integration tests (no copy left in the project, a changed or removed module refused at install, a script step after pre-built changing only the build's copy). Updates CHANGELOG.md and the build and project-model docs. Cargo.lock moves wasmtime from 49.0.5 to 49.0.7, with the cranelift and pulley crates that move with it, to clear the RustSec advisories.

Risk Assessment

⚠️ Medium: The change meets the intent (no copy for a final pre-built step, shared file across canisters) and I found no correctness defect in the traced paths, but it changes the install path's user-visible behaviour with a hash-pin refusal the intent did not ask for and makes .icp depend on files outside it.

Testing

Built icp from the target commit and drove it by hand in a disposable HOME/ICP_HOME with throwaway projects: shared local pre-built, URL pre-built from a localhost HTTP server, pre-built followed by a script step, switching a canister between script and pre-built, sha256 pin, changed/removed file after build, bundle, and a live deploy/install/call on a managed local network. Also ran the six pre_built_path integration tests in build_adapter_tests. All passed; the first URL attempt failed only because my fixture read the server port too early, and passed once re-driven. No UI surface, so evidence is CLI transcripts. The disposable workdir was removed, the network stopped, and the worktree is clean.

  • Live validation: ✅ go - 11 of 11 scenarios driven live against the product
Scenario Result Live Evidence
Run icp build on a project with two canisters that name the same local pre-built wasm (synthetic project, 205794-byte vendored wasm): .icp holds no copy, both canisters record the same file, and t… ✅ pass live offline-transcript.txt section S1: .icp contains only first%link and second%link (139 bytes each) plus an empty lock; both name shared/module.wasm with sha256 17a05e36...; source hash identica…
Run icp build on two canisters with the same url pre-built (localhost HTTP server): the project holds no copy and both canisters point at the single cached download under ICP_HOME ✅ pass live live-transcript.txt section S7: both %link files (216 bytes) name home/icp/pkg/wasms/17a05e36.../module.wasm; that is the only wasm-sized file under ICP_HOME
Run icp deploy to a managed local network for two canisters sharing one pre-built wasm, then call each: both install from the shared file and answer ✅ pass live live-transcript.txt section S2: deploy exit 0; artifacts dir holds only two 139-byte links; greet returns (&#34;Hello, first!&#34;) and (&#34;Hello, second!&#34;); canister status module hash 0x17a05e36... eq…
Run icp build and then a separate icp canister install --mode reinstall on the local network: install reads the module through the recorded path ✅ pass live live-transcript.txt section S2: Canister second installed successfully, then greet returns (&#34;Hello, again!&#34;)
Adversarial: change or delete the pre-built file after icp build, then run icp canister install: it is refused, naming the file and asking for a rebuild ✅ pass live offline-transcript.txt section S3 (exit 1 with has changed since ... build the canister again, then can no longer be read; build the canister again) and live-transcript.txt S2 (same refusal agains…
Run icp build when the pre-built file is missing: the build fails and names the file ✅ pass live offline-transcript.txt section S3: exit 1 with failed to read wasm file &#39;.../s3/module.wasm&#39;
Adversarial: put a script step after the pre-built step that appends to $ICP_WASM_OUTPUT_PATH: the artifact is a modified copy and the user's source file is untouched ✅ pass live offline-transcript.txt section S4: artifact my-canister is 205802 bytes (source 205794 + 8) ending in appended; source sha256 unchanged
Switch one canister from a script build to pre-built and back, running icp build each time: the artifacts directory holds exactly one of the copied module or the link, never both ✅ pass live offline-transcript.txt section S5: my-canister (205794 bytes) -> my-canister%link (139 bytes) -> my-canister (205794 bytes)
Run icp build with a wrong sha256 on a local pre-built, then with the right one: the wrong pin fails and leaves nothing behind, the right pin builds with a link only ✅ pass live offline-transcript.txt section S6: exit 1 with checksum mismatch, expected: 000..., project contains only icp.yaml; after correcting, exit 0 and only my-canister%link
Run icp project bundle on the project with two canisters sharing a pre-built wasm: the bundle contains the module once ✅ pass live offline-transcript.txt section S8: exit 0; tarball lists icp.yaml and a single canisters/17a05e36....wasm of 205794 bytes
Existing and new pre_built_path integration tests in build_adapter_tests pass against the built binary ✅ pass live integration-tests.txt: cargo test --locked -p icp-cli --test build_adapter_tests pre_built_path — 6 passed, 0 failed
Evidence: Offline scenarios transcript (shared local pre-built, changed/removed file, script after pre-built, switching, sha256 pin, bundle)
module: /tmp/icp-prebuilt-IpUWav/shared/module.wasm  205794 bytes  sha256 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a

===== S1: two canisters, one local pre-built wasm =====
canisters:
  - name: first
    build:
      steps:
        - type: pre-built
          path: '/tmp/icp-prebuilt-IpUWav/shared/module.wasm'
  - name: second
    build:
      steps:
        - type: pre-built
          path: '/tmp/icp-prebuilt-IpUWav/shared/module.wasm'

$ {run-worktree}/target/debug/icp build
Building canisters:
Canisters built successfully
[exit 0]

--- files under /tmp/icp-prebuilt-IpUWav/s1/.icp (bytes  path)
0  ./cache/artifacts/.lock
139  ./cache/artifacts/first%link
139  ./cache/artifacts/second%link
--- /tmp/icp-prebuilt-IpUWav/s1/.icp/cache/artifacts/first%link
{
  "path": "/tmp/icp-prebuilt-IpUWav/shared/module.wasm",
  "sha256": "17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a"
}
--- /tmp/icp-prebuilt-IpUWav/s1/.icp/cache/artifacts/second%link
{
  "path": "/tmp/icp-prebuilt-IpUWav/shared/module.wasm",
  "sha256": "17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a"
}
source sha256 after build: 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a (before: 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a)
leftover scratch dirs in TMPDIR mentioning icp: 0

===== S3: changed / removed after build -> install refused =====

$ {run-worktree}/target/debug/icp build my-canister
Building canisters:
Canisters built successfully
[exit 0]

--- files under /tmp/icp-prebuilt-IpUWav/s3/.icp (bytes  path)
0  ./cache/artifacts/.lock
135  ./cache/artifacts/my-canister%link

$ {run-worktree}/target/debug/icp canister install my-canister
Error: canister 'my-canister' was built from the module at '/tmp/icp-prebuilt-IpUWav/s3/module.wasm', which has changed since: its sha256 was 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a and is now 8ae80e081080361453a8de47dabb27f305a013756dee0a79662a0a728dbdef23; build the canister again
[exit 1]

$ {run-worktree}/target/debug/icp canister install my-canister
Error: canister 'my-canister' was built from the module at '/tmp/icp-prebuilt-IpUWav/s3/module.wasm', which can no longer be read; build the canister again

Caused by:
    0: Filesystem operation failed at /tmp/icp-prebuilt-IpUWav/s3/module.wasm
    1: No such file or directory (os error 2)
[exit 1]
--- build with the file missing

$ {run-worktree}/target/debug/icp build my-canister
Building canisters:
ERR ----- Failed to build canister 'my-canister' -----
ERR 'failed to read wasm file '/tmp/icp-prebuilt-IpUWav/s3/module.wasm''
ERR [my-canister] Build output:
ERR [my-canister] Building: step 1 of 1 (pre-built):
ERR [my-canister] path: module.wasm, sha: n/a:
ERR [my-canister] <no output>
Error: Canister(s) ["my-canister"] failed to build.
[exit 1]

===== S4: pre-built then script step -> copied, source untouched =====

$ {run-worktree}/target/debug/icp build my-canister
Building canisters:
Canisters built successfully
[exit 0]

--- files under /tmp/icp-prebuilt-IpUWav/s4/.icp (bytes  path)
0  ./cache/artifacts/.lock
205802  ./cache/artifacts/my-canister
artifact tail: appended
artifact size 205802 = source 205794 + 8
source sha256 after: 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a (before: 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a)

===== S5: switching a canister between script and pre-built =====

$ {run-worktree}/target/debug/icp build
Building canisters:
Canisters built successfully
[exit 0]

--- files under /tmp/icp-prebuilt-IpUWav/s5/.icp/cache/artifacts (bytes  path)
0  ./.lock
205794  ./my-canister

$ {run-worktree}/target/debug/icp build
Building canisters:
Canisters built successfully
[exit 0]

--- files under /tmp/icp-prebuilt-IpUWav/s5/.icp/cache/artifacts (bytes  path)
0  ./.lock
139  ./my-canister%link

$ {run-worktree}/target/debug/icp build
Building canisters:
Canisters built successfully
[exit 0]

--- files under /tmp/icp-prebuilt-IpUWav/s5/.icp/cache/artifacts (bytes  path)
0  ./.lock
205794  ./my-canister

===== S6: sha256 pin on a local pre-built =====

$ {run-worktree}/target/debug/icp build
Building canisters:
ERR ----- Failed to build canister 'my-canister' -----
ERR 'checksum mismatch, expected: 0000000000000000000000000000000000000000000000000000000000000000, actual: 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a'
ERR [my-canister] Build output:
ERR [my-canister] Building: step 1 of 1 (pre-built):
ERR [my-canister] path: /tmp/icp-prebuilt-IpUWav/shared/module.wasm, sha: 0000000000000000000000000000000000000000000000000000000000000000:
ERR [my-canister] > Reading wasm: /tmp/icp-prebuilt-IpUWav/shared/module.wasm
ERR [my-canister] > Verifying checksum
Error: Canister(s) ["my-canister"] failed to build.
[exit 1]

--- files under /tmp/icp-prebuilt-IpUWav/s6 (bytes  path)
228  ./icp.yaml

$ {run-worktree}/target/debug/icp build
Building canisters:
Canisters built successfully
[exit 0]

--- files under /tmp/icp-prebuilt-IpUWav/s6/.icp (bytes  path)
0  ./cache/artifacts/.lock
139  ./cache/artifacts/my-canister%link

===== S7: url pre-built shared by two canisters =====
http server on 

$ {run-worktree}/target/debug/icp build
Building canisters:
ERR ----- Failed to build canister 'first' -----
ERR 'failed to fetch wasm file'
ERR [first] Build output:
ERR [first] Building: step 1 of 1 (pre-built):
ERR [first] url: http://127.0.0.1:/canister.wasm, sha: 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a:
ERR [first] > Fetching wasm: http://127.0.0.1/canister.wasm
ERR ----- Failed to build canister 'second' -----
ERR 'failed to fetch wasm file'
ERR [second] Build output:
ERR [second] Building: step 1 of 1 (pre-built):
ERR [second] url: http://127.0.0.1:/canister.wasm, sha: 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a:
ERR [second] > Fetching wasm: http://127.0.0.1/canister.wasm
Error: Canister(s) ["first", "second"] failed to build.
[exit 1]

--- files under /tmp/icp-prebuilt-IpUWav/s7/.icp (bytes  path)
~/.no-mistakes/evidence/01M7VJA59B9Q27TPW4A50DY3TR/drive.sh: line 15: cd: /tmp/icp-prebuilt-IpUWav/s7/.icp: No such file or directory
--- /tmp/icp-prebuilt-IpUWav/s7/.icp/cache/artifacts/*
cat: '/tmp/icp-prebuilt-IpUWav/s7/.icp/cache/artifacts/*': No such file or directory

--- wasm-sized files under ICP_HOME:
--- http requests served:

===== S8: bundle from a pre-built =====

$ {run-worktree}/target/debug/icp project bundle --output /tmp/icp-prebuilt-IpUWav/s1.tar.gz
[exit 0]
-rw-r--r-- 0/0             581 1970-01-01 00:00 icp.yaml
-rw-r--r-- 0/0          205794 1970-01-01 00:00 canisters/17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a.wasm
DONE-OFFLINE
Evidence: URL source and live local-network transcript (deploy, call, status, reinstall, refusal)
===== S7: url pre-built shared by two canisters =====
http server on 38701

$ {run-worktree}/target/debug/icp build
Building canisters:
Canisters built successfully
[exit 0]

--- files under /tmp/icp-prebuilt-IpUWav/s7/.icp (bytes  path)
0  ./cache/artifacts/.lock
216  ./cache/artifacts/first%link
216  ./cache/artifacts/second%link
--- /tmp/icp-prebuilt-IpUWav/s7/.icp/cache/artifacts/first%link
{
  "path": "/tmp/icp-prebuilt-IpUWav/home/icp/pkg/wasms/17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a/module.wasm",
  "sha256": "17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a"
}
--- /tmp/icp-prebuilt-IpUWav/s7/.icp/cache/artifacts/second%link
{
  "path": "/tmp/icp-prebuilt-IpUWav/home/icp/pkg/wasms/17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a/module.wasm",
  "sha256": "17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a"
}
--- wasm-sized files under ICP_HOME:
205794  /tmp/icp-prebuilt-IpUWav/home/icp/pkg/wasms/17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a/module.wasm
--- http requests served:
Serving HTTP on 127.0.0.1 port 38701 (http://127.0.0.1:38701/) ...
127.0.0.1 - - [20/Nov/2026 19:40:10] "GET /canister.wasm HTTP/1.1" 200 -
127.0.0.1 - - [20/Nov/2026 19:40:10] "GET /canister.wasm HTTP/1.1" 200 -

===== S2: live deploy to a local network from a shared pre-built =====

$ {run-worktree}/target/debug/icp network start random-network -d
For background mode, network output will be redirected:
  stdout: /tmp/icp-prebuilt-IpUWav/s2/.icp/cache/networks/random-network/network-launcher/stdout.log
  stderr: /tmp/icp-prebuilt-IpUWav/s2/.icp/cache/networks/random-network/network-launcher/stderr.log
Network started on port 38507
Seeding ICP and cycles account balances
To stop the network, run `icp network stop`
[exit 0]

$ {run-worktree}/target/debug/icp network ping random-network --wait-healthy
{
  "replica_health_status": "healthy",
  "root_key": [48, 129, 130, 48, 29, 6, 13, 43, 6, 1, 4, 1, 130, 220, 124, 5, 3, 1, 2, 1, 6, 12, 43, 6, 1, 4, 1, 130, 220, 124, 5, 3, 2, 1, 3, 97, 0, 165, 135, 172, 39, 136, 79, 35, 90, 145, 221, 216, 105, 39, 238, 193, 30, 9, 135, 36, 23, 35, 27, 65, 171, 143, 185, 86, 5, 198, 51, 175, 151, 2, 132, 156, 10, 214, 146, 150, 19, 57, 43, 57, 199, 21, 252, 35, 44, 10, 13, 142, 176, 10, 30, 169, 25, 133, 179, 228, 64, 188, 157, 11, 120, 165, 135, 47, 150, 218, 231, 83, 243, 149, 205, 102, 2, 81, 106, 213, 135, 72, 214, 152, 149, 137, 118, 198, 194, 91, 134, 197, 90, 163, 10, 58, 249]
}
[exit 0]

$ {run-worktree}/target/debug/icp cycles mint --cycles 10000000000000 --environment random-environment
Minted 10_000_000_000_000 cycles to your account, new balance: 170_141_183_460_469_231_731_687_293_715_684_105_727 cycles.
[exit 0]

$ {run-worktree}/target/debug/icp deploy --environment random-environment
Building canisters:
Creating canisters:
Setting environment variables:
Applying canister settings:
Checking compatibility:
Installing canisters:
No canisters have sync steps configured
Created canister first with ID 4caro-hl777-77775-aaaba-cai
Created canister second with ID 4fbx2-kt777-77775-aaabq-cai
Deployed canisters:

Backends (Candid UI):
  first: http://4mc4g-43777-77775-aaaaa-cai.localhost:38507/?id=4caro-hl777-77775-aaaba-cai
  second: http://4mc4g-43777-77775-aaaaa-cai.localhost:38507/?id=4fbx2-kt777-77775-aaabq-cai
[exit 0]

--- files under /tmp/icp-prebuilt-IpUWav/s2/.icp/cache/artifacts (bytes  path)
0  ./.lock
139  ./first%link
139  ./second%link

$ {run-worktree}/target/debug/icp canister call --environment random-environment first greet ("first")
("Hello, first!")
[exit 0]

$ {run-worktree}/target/debug/icp canister call --environment random-environment second greet ("second")
("Hello, second!")
[exit 0]
--- installed module hash vs source (17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a)

$ {run-worktree}/target/debug/icp canister status --environment random-environment first
Canister Id: 4caro-hl777-77775-aaaba-cai
Canister Name: first
Canister Status Report:
  Status: Running
  Controllers:
    controller: 2vxsx-fae
  Compute allocation: 0
  Memory allocation: 0
  Freezing threshold: 2_592_000
  Reserved cycles limit: 5_000_000_000_000
  Wasm memory limit: 3_221_225_472
  Wasm memory threshold: 0
  Log memory limit: 4_096
  Log visibility: Controllers
  Snapshot visibility: Controllers
  Status visibility: Controllers
  Environment Variables:
    Name: PUBLIC_CANISTER_ID:first, Value: 4caro-hl777-77775-aaaba-cai
    Name: PUBLIC_CANISTER_ID:second, Value: 4fbx2-kt777-77775-aaabq-cai
  Module hash: 0x17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a
  Memory size: 2_382_268
  Cycles: 1_498_934_144_537
  Reserved cycles: 0
  Idle cycles burned per day: 924_862_280
  Query stats:
    Calls: 0
    Instructions: 0
    Req payload bytes: 0
    Res payload bytes: 0
[exit 0]
--- separate build then install (reinstall) reads the linked file

$ {run-worktree}/target/debug/icp build
Building canisters:
Canisters built successfully
[exit 0]

$ {run-worktree}/target/debug/icp canister install --environment random-environment --mode reinstall -y second
Canister second installed successfully
[exit 0]

$ {run-worktree}/target/debug/icp canister call --environment random-environment second greet ("again")
("Hello, again!")
[exit 0]
--- change the shared file, install is refused before reaching the network

$ {run-worktree}/target/debug/icp canister install --environment random-environment --mode reinstall -y second
Error: canister 'second' was built from the module at '/tmp/icp-prebuilt-IpUWav/shared/module.wasm', which has changed since: its sha256 was 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a and is now 1df91fecc906ace7d3d6ef065aefe6681e739193a905f6c1b9c2f7f5a9bbc861; build the canister again
[exit 1]

$ {run-worktree}/target/debug/icp network stop random-network
Stopping background network (PID: 895970)...
Network stopped successfully
[exit 0]
DONE-LIVE
Evidence: Targeted integration test output
   Compiling tokio v1.54.1
   Compiling nix v0.31.3
   Compiling cryptoki-sys v0.5.0
   Compiling predicates-core v1.0.10
   Compiling float-cmp v0.10.0
   Compiling libloading v0.8.9
   Compiling difflib v0.4.0
   Compiling assert_cmd v2.2.2
   Compiling normalize-line-endings v0.3.0
   Compiling termtree v0.5.1
   Compiling secrecy v0.10.3
   Compiling serial_test_derive v3.5.0
   Compiling wait-timeout v0.2.1
   Compiling fslock v0.2.1
   Compiling test-tag v0.1.6
   Compiling send_ctrlc v0.6.0
   Compiling predicates-tree v1.0.13
   Compiling predicates v3.1.4
   Compiling serial_test v3.5.0
   Compiling cryptoki v0.12.1
   Compiling ctrlc v3.5.2
   Compiling candid_parser v0.3.2
   Compiling tokio-util v0.7.19
   Compiling tower v0.5.3
   Compiling tokio-rustls v0.26.5
   Compiling icp-events v0.1.0 ({run-worktree}/crates/icp-events)
   Compiling async-scoped v0.9.0
   Compiling wasmtime-wasi v49.0.7
   Compiling backoff v0.4.0
   Compiling async-dropper-simple v0.2.6
   Compiling async-dropper v0.3.1
   Compiling tower-http v0.6.11
   Compiling h2 v0.4.19
   Compiling icp-project v0.1.0 ({run-worktree}/crates/icp-project)
   Compiling hyper v1.11.1
   Compiling hyper-util v0.1.20
   Compiling hyper-rustls v0.27.10
   Compiling hyperlocal v0.9.1
   Compiling httptest v0.16.4
   Compiling axum v0.8.9
   Compiling reqwest v0.13.5
   Compiling bollard v0.20.2
   Compiling ic-agent v0.49.2
   Compiling axoasset v2.0.1
   Compiling axoupdater v0.10.2
   Compiling ic-utils v0.49.2
   Compiling ic-identity-hsm v0.49.2
   Compiling icp-sync-plugin v0.1.0 ({run-worktree}/crates/icp-sync-plugin)
   Compiling icp-extension v1.8.0 ({run-worktree}/crates/icp-extension)
   Compiling icp-app v1.8.0 ({run-worktree}/crates/icp-app)
   Compiling icp-cli v1.8.0 ({run-worktree}/crates/icp-cli)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 49.90s
     Running tests/build_adapter_tests.rs (target/debug/deps/build_adapter_tests-fcbf9e6c29a00f6c)

running 6 tests
test build_adapter_pre_built_path_is_not_copied_into_the_project ... ok
test build_adapter_pre_built_path_valid_checksum ... ok
test build_adapter_pre_built_path_then_script_changes_the_output_alone ... ok
test build_adapter_pre_built_path ... ok
test build_adapter_pre_built_path_invalid_checksum ... ok
test build_adapter_pre_built_path_changed_after_build_is_not_installed ... ok

test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.48s
Evidence: Driver script for the offline scenarios
#!/usr/bin/env bash
# Drives the real `icp` binary against disposable projects under $W.
set -u
WT={run-worktree}
ICP=$WT/target/debug/icp
W=$(mktemp -d /tmp/icp-prebuilt-XXXXXX)
echo "$W" > /tmp/icp-prebuilt-workdir
export HOME=$W/home ICP_HOME=$W/home/icp XDG_CONFIG_HOME=$W/home/.config XDG_DATA_HOME=$W/home/.local/share XDG_CACHE_HOME=$W/home/.cache
export ICP_CLI_KEYRING_MOCK_DIR=$W/keyring ICP_TELEMETRY_DISABLED=1 ICP_CLI_TEST_NO_TELEMETRY_UPLOAD=1 NO_COLOR=1
mkdir -p "$HOME" "$ICP_CLI_KEYRING_MOCK_DIR" "$W/shared"
cp $WT/crates/icp-cli/tests/assets/example_icp_mo.wasm "$W/shared/module.wasm"
SRC=$W/shared/module.wasm
SRCSUM=$(sha256sum "$SRC" | cut -d' ' -f1)
run() { echo; echo "\$ $*"; "$@"; local rc=$?; echo "[exit $rc]"; return $rc; }
tree() { echo; echo "--- files under $1 (bytes  path)"; (cd "$1" && find . -type f -printf '%s  %p\n' | sort -k2); }
echo "module: $SRC  $(stat -c %s "$SRC") bytes  sha256 $SRCSUM"

echo; echo "===== S1: two canisters, one local pre-built wasm ====="
P=$W/s1; mkdir -p $P; cd $P
cat > icp.yaml <<Y
canisters:
  - name: first
    build:
      steps:
        - type: pre-built
          path: '$SRC'
  - name: second
    build:
      steps:
        - type: pre-built
          path: '$SRC'
Y
cat icp.yaml
run $ICP build
tree $P/.icp
for f in $P/.icp/cache/artifacts/*; do echo "--- $f"; cat "$f"; echo; done
echo "source sha256 after build: $(sha256sum "$SRC" | cut -d' ' -f1) (before: $SRCSUM)"
echo "leftover scratch dirs in TMPDIR mentioning icp: $(ls /tmp | grep -c '^\.tmp' )"

echo; echo "===== S3: changed / removed after build -> install refused ====="
P=$W/s3; mkdir -p $P; cd $P; cp "$SRC" module.wasm
cat > icp.yaml <<Y
canisters:
  - name: my-canister
    build:
      steps:
        - type: pre-built
          path: module.wasm
Y
run $ICP build my-canister
tree $P/.icp
printf 'another module' > module.wasm
run $ICP canister install my-canister
rm module.wasm
run $ICP canister install my-canister
echo "--- build with the file missing"
run $ICP build my-canister

echo; echo "===== S4: pre-built then script step -> copied, source untouched ====="
P=$W/s4; mkdir -p $P; cd $P
cat > icp.yaml <<Y
canisters:
  - name: my-canister
    build:
      steps:
        - type: pre-built
          path: '$SRC'
        - type: script
          command: printf appended >> "\$ICP_WASM_OUTPUT_PATH"
Y
run $ICP build my-canister
tree $P/.icp
echo "artifact tail: $(tail -c 8 .icp/cache/artifacts/my-canister)"
echo "artifact size $(stat -c %s .icp/cache/artifacts/my-canister) = source $(stat -c %s "$SRC") + 8"
echo "source sha256 after: $(sha256sum "$SRC" | cut -d' ' -f1) (before: $SRCSUM)"

echo; echo "===== S5: switching a canister between script and pre-built ====="
P=$W/s5; mkdir -p $P; cd $P
cat > icp.yaml <<Y
canisters:
  - name: my-canister
    build:
      steps:
        - type: script
          command: cp '$SRC' "\$ICP_WASM_OUTPUT_PATH"
Y
run $ICP build
tree $P/.icp/cache/artifacts
cat > icp.yaml <<Y
canisters:
  - name: my-canister
    build:
      steps:
        - type: pre-built
          path: '$SRC'
Y
run $ICP build
tree $P/.icp/cache/artifacts
cat > icp.yaml <<Y
canisters:
  - name: my-canister
    build:
      steps:
        - type: script
          command: cp '$SRC' "\$ICP_WASM_OUTPUT_PATH"
Y
run $ICP build
tree $P/.icp/cache/artifacts

echo; echo "===== S6: sha256 pin on a local pre-built ====="
P=$W/s6; mkdir -p $P; cd $P
cat > icp.yaml <<Y
canisters:
  - name: my-canister
    build:
      steps:
        - type: pre-built
          path: '$SRC'
          sha256: 0000000000000000000000000000000000000000000000000000000000000000
Y
run $ICP build
tree $P 
sed -i "s/sha256: 0\{64\}/sha256: $SRCSUM/" icp.yaml
run $ICP build
tree $P/.icp

echo; echo "===== S7: url pre-built shared by two canisters ====="
mkdir -p $W/www; cp "$SRC" $W/www/canister.wasm
(cd $W/www && exec python3 -I -m http.server 0 --bind 127.0.0.1 > $W/http.log 2>&1) &
HP=$!; sleep 1.5
PORT=$(grep -o 'port [0-9]*' $W/http.log | cut -d' ' -f2)
echo "http server on $PORT"
P=$W/s7; mkdir -p $P; cd $P
cat > icp.yaml <<Y
canisters:
  - name: first
    build:
      steps:
        - type: pre-built
          url: http://127.0.0.1:$PORT/canister.wasm
          sha256: $SRCSUM
  - name: second
    build:
      steps:
        - type: pre-built
          url: http://127.0.0.1:$PORT/canister.wasm
          sha256: $SRCSUM
Y
run $ICP build
tree $P/.icp
for f in $P/.icp/cache/artifacts/*; do echo "--- $f"; cat "$f"; echo; done
echo "--- wasm-sized files under ICP_HOME:"; find $ICP_HOME -type f -size +100k -printf '%s  %p\n'
kill $HP; wait $HP 2>/dev/null
echo "--- http requests served:"; cat $W/http.log

echo; echo "===== S8: bundle from a pre-built ====="
cd $W/s1
run $ICP project bundle --output $W/s1.tar.gz
tar tzvf $W/s1.tar.gz
echo DONE-OFFLINE
Evidence: Driver script for the URL and live network scenarios
#!/usr/bin/env bash
# URL source and live local-network scenarios, same disposable workdir as drive.sh.
set -u
WT={run-worktree}
ICP=$WT/target/debug/icp
W=$(cat /tmp/icp-prebuilt-workdir)
export HOME=$W/home ICP_HOME=$W/home/icp XDG_CONFIG_HOME=$W/home/.config XDG_DATA_HOME=$W/home/.local/share XDG_CACHE_HOME=$W/home/.cache
export ICP_CLI_KEYRING_MOCK_DIR=$W/keyring ICP_TELEMETRY_DISABLED=1 ICP_CLI_TEST_NO_TELEMETRY_UPLOAD=1 NO_COLOR=1
SRC=$W/shared/module.wasm
SRCSUM=$(sha256sum "$SRC" | cut -d' ' -f1)
run() { echo; echo "\$ $*"; "$@"; local rc=$?; echo "[exit $rc]"; return $rc; }
tree() { echo; echo "--- files under $1 (bytes  path)"; (cd "$1" && find . -type f -printf '%s  %p\n' | sort -k2); }

echo "===== S7: url pre-built shared by two canisters ====="
mkdir -p $W/www; cp "$SRC" $W/www/canister.wasm
(cd $W/www && exec python3 -I -u -m http.server 0 --bind 127.0.0.1 > $W/http.log 2>&1) &
HP=$!
for i in $(seq 50); do PORT=$(grep -o 'port [0-9]*' $W/http.log | head -1 | cut -d' ' -f2); [ -n "$PORT" ] && break; sleep 0.2; done
echo "http server on $PORT"
P=$W/s7; rm -rf $P; mkdir -p $P; cd $P
cat > icp.yaml <<Y
canisters:
  - name: first
    build:
      steps:
        - type: pre-built
          url: http://127.0.0.1:$PORT/canister.wasm
          sha256: $SRCSUM
  - name: second
    build:
      steps:
        - type: pre-built
          url: http://127.0.0.1:$PORT/canister.wasm
          sha256: $SRCSUM
Y
run $ICP build
tree $P/.icp
for f in $P/.icp/cache/artifacts/*; do echo "--- $f"; cat "$f"; echo; done
echo "--- wasm-sized files under ICP_HOME:"; find $ICP_HOME -type f -size +100k -printf '%s  %p\n'
kill $HP; wait $HP 2>/dev/null
echo "--- http requests served:"; cat $W/http.log

echo; echo "===== S2: live deploy to a local network from a shared pre-built ====="
P=$W/s2; mkdir -p $P; cd $P
cat > icp.yaml <<Y
canisters:
  - name: first
    build:
      steps:
        - type: pre-built
          path: '$SRC'
  - name: second
    build:
      steps:
        - type: pre-built
          path: '$SRC'
networks:
  - name: random-network
    mode: managed
    gateway:
      port: 0
environments:
  - name: random-environment
    network: random-network
Y
run $ICP network start random-network -d
run $ICP network ping random-network --wait-healthy
run $ICP cycles mint --cycles 10000000000000 --environment random-environment
run $ICP deploy --environment random-environment
tree $P/.icp/cache/artifacts
run $ICP canister call --environment random-environment first greet '("first")'
run $ICP canister call --environment random-environment second greet '("second")'
echo "--- installed module hash vs source ($SRCSUM)"
run $ICP canister status --environment random-environment first
echo "--- separate build then install (reinstall) reads the linked file"
run $ICP build
run $ICP canister install --environment random-environment --mode reinstall -y second
run $ICP canister call --environment random-environment second greet '("again")'
echo "--- change the shared file, install is refused before reaching the network"
cp "$SRC" $W/shared/module.bak; printf 'x' >> "$SRC"
run $ICP canister install --environment random-environment --mode reinstall -y second
cp $W/shared/module.bak "$SRC"
run $ICP network stop random-network
echo DONE-LIVE
Evidence: Two canisters, one pre-built wasm: what `icp build` leaves in `.icp`
module: shared/module.wasm 205794 bytes sha256 17a05e36...be7a
$ icp build
Canisters built successfully
--- files under s1/.icp (bytes path)
0 ./cache/artifacts/.lock
139 ./cache/artifacts/first%link
139 ./cache/artifacts/second%link
--- first%link
{ "path": "/tmp/icp-prebuilt-IpUWav/shared/module.wasm", "sha256": "17a05e36...be7a" }
--- second%link
{ "path": "/tmp/icp-prebuilt-IpUWav/shared/module.wasm", "sha256": "17a05e36...be7a" }
Evidence: Live deploy from the shared pre-built, and refusal after the file changes
$ icp deploy --environment random-environment
Created canister first with ID 4caro-hl777-77775-aaaba-cai
Created canister second with ID 4fbx2-kt777-77775-aaabq-cai
$ icp canister call --environment random-environment first greet ("first")
("Hello, first!")
$ icp canister status ... first
Module hash: 0x17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a
$ icp canister install --environment random-environment --mode reinstall -y second (after appending a byte to the shared file)
Error: canister 'second' was built from the module at '/tmp/icp-prebuilt-IpUWav/shared/module.wasm', which has changed since: its sha256 was 17a05e36... and is now 1df91fec...; build the canister again
[exit 1]

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 3 issues (1 warning, 2 infos)
  • ⚠️ crates/icp-project/src/store_artifact.rs:110 - Simplification: the SHA-256 pin on a linked module is not required by the intent, which asks only that "the path to the prebuilt wasm can be reused without a single copy". The change adds a hash recorded at build and a refusal at lookup when the file has changed. This is new user-visible behaviour: icp canister install now fails with "has changed since … build the canister again" after the user edits or replaces their own wasm. It also costs a full read of the module into memory on every pre-built step just to hash it (crates/icp-project/src/canister/build/prebuilt.rs:45-49), and a local source with sha256: set is read and hashed twice (crates/icp-app/src/wasm.rs:99 and again in the step). Parts of this component: Borrowed.sha256 (crates/icp-project/src/canister/build/mod.rs:57), verify_linked and LookupLinkedChanged (store_artifact.rs:95-127), Link.sha256 (store_artifact.rs:198), the error mapping in crates/icp-project/src/operations/install.rs:354-364. The narrower form is to record only the path and have lookup read whatever is there. Whether a changed file should be refused or installed as-is is a product decision; please confirm you want the pin.
  • ℹ️ crates/icp-project/src/store_artifact.rs:307 - link refuses a canister whose encoded name plus %link exceeds 255 bytes (encoded name of 251–255 bytes). Such a canister with a pre-built last step built before this change (via save) and now fails icp build with SaveNameTooLong. It is only reachable with ~250-byte canister names, so this is noted and not blocking; the test a_name_with_no_room_for_a_link_is_refused_one pins it as intended.
  • ℹ️ crates/icp-project/src/operations/build.rs:105 - Tradeoff inherent to the requested design: .icp/cache/artifacts is no longer self-contained for pre-built canisters. The link stores an absolute path to the user's file or to the global package cache entry for a url source. So icp build on one machine or CI job followed by icp canister install on another (carrying only the project and .icp), or after moving the project or clearing the package cache, now fails with the "can no longer be read; build the canister again" error. icp deploy and icp project bundle rebuild first (deploy.rs:239, bundle.rs:495) and are unaffected. The CHANGELOG and docs describe this accurately. Also, store_artifact::Access gained a required link method and Build::build changed its return type; both are breaking for external implementors of the published icp-project crate (such as imcp2), which CLAUDE.md permits.
✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 11 of 11 scenarios driven live against the product
Scenario Result Live Evidence
Run icp build on a project with two canisters that name the same local pre-built wasm (synthetic project, 205794-byte vendored wasm): .icp holds no copy, both canisters record the same file, and t… ✅ pass live offline-transcript.txt section S1: .icp contains only first%link and second%link (139 bytes each) plus an empty lock; both name shared/module.wasm with sha256 17a05e36...; source hash identica…
Run icp build on two canisters with the same url pre-built (localhost HTTP server): the project holds no copy and both canisters point at the single cached download under ICP_HOME ✅ pass live live-transcript.txt section S7: both %link files (216 bytes) name home/icp/pkg/wasms/17a05e36.../module.wasm; that is the only wasm-sized file under ICP_HOME
Run icp deploy to a managed local network for two canisters sharing one pre-built wasm, then call each: both install from the shared file and answer ✅ pass live live-transcript.txt section S2: deploy exit 0; artifacts dir holds only two 139-byte links; greet returns (&#34;Hello, first!&#34;) and (&#34;Hello, second!&#34;); canister status module hash 0x17a05e36... eq…
Run icp build and then a separate icp canister install --mode reinstall on the local network: install reads the module through the recorded path ✅ pass live live-transcript.txt section S2: Canister second installed successfully, then greet returns (&#34;Hello, again!&#34;)
Adversarial: change or delete the pre-built file after icp build, then run icp canister install: it is refused, naming the file and asking for a rebuild ✅ pass live offline-transcript.txt section S3 (exit 1 with has changed since ... build the canister again, then can no longer be read; build the canister again) and live-transcript.txt S2 (same refusal agains…
Run icp build when the pre-built file is missing: the build fails and names the file ✅ pass live offline-transcript.txt section S3: exit 1 with failed to read wasm file &#39;.../s3/module.wasm&#39;
Adversarial: put a script step after the pre-built step that appends to $ICP_WASM_OUTPUT_PATH: the artifact is a modified copy and the user's source file is untouched ✅ pass live offline-transcript.txt section S4: artifact my-canister is 205802 bytes (source 205794 + 8) ending in appended; source sha256 unchanged
Switch one canister from a script build to pre-built and back, running icp build each time: the artifacts directory holds exactly one of the copied module or the link, never both ✅ pass live offline-transcript.txt section S5: my-canister (205794 bytes) -> my-canister%link (139 bytes) -> my-canister (205794 bytes)
Run icp build with a wrong sha256 on a local pre-built, then with the right one: the wrong pin fails and leaves nothing behind, the right pin builds with a link only ✅ pass live offline-transcript.txt section S6: exit 1 with checksum mismatch, expected: 000..., project contains only icp.yaml; after correcting, exit 0 and only my-canister%link
Run icp project bundle on the project with two canisters sharing a pre-built wasm: the bundle contains the module once ✅ pass live offline-transcript.txt section S8: exit 0; tarball lists icp.yaml and a single canisters/17a05e36....wasm of 205794 bytes
Existing and new pre_built_path integration tests in build_adapter_tests pass against the built binary ✅ pass live integration-tests.txt: cargo test --locked -p icp-cli --test build_adapter_tests pre_built_path — 6 passed, 0 failed
  • cargo build --bin icp --locked (exit 0)
  • bash drive.sh: icp build on two canisters sharing one local pre-built wasm, then listing .icp and reading the %link records
  • icp build then icp canister install after rewriting and after deleting the pre-built file; icp build with the file missing
  • icp build with a pre-built step followed by a script step appending to $ICP_WASM_OUTPUT_PATH
  • icp build three times while switching one canister script -> pre-built -> script, listing .icp/cache/artifacts each time
  • icp build with a wrong and then correct sha256 on a local pre-built
  • icp project bundle --output on the shared pre-built project, then tar tzvf
  • bash drive2.sh: icp build on two canisters with the same url pre-built served by a localhost python3 -m http.server
  • icp network start -d, icp cycles mint, icp deploy, icp canister call ... greet, icp canister status, icp build + icp canister install --mode reinstall, then install after changing the shared file, icp network stop
  • cargo test --locked -p icp-cli --test build_adapter_tests pre_built_path (6 passed)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

A `pre-built` build step resolved its module (a local path, or a URL
through the package cache) and copied it to the build's scratch
`out.wasm`. The build operation then read that file back and wrote the
same bytes to the artifact store. The scratch copy had no reader but the
build itself, and on a host whose disk is memory (dfinity/imcp2) it cost
the module's size in RAM for every canister being built at once.

A build step now says where it left the module: `Module::AtOutput`, as
before, or `Module::Bytes`, the module itself. The pre-built step reads
its source through `FileSystem` and hands the bytes back; the operation
saves them straight to the artifact store. Only when another step
follows does it write them to the output path first, since that step
looks for the module at `ICP_WASM_OUTPUT_PATH` and may change it there.
The source file and the shared cache entry are only ever read.

Writes of the module's bytes to disk while building one canister whose
only step is `pre-built`:

  source                 before                          after
  local path             scratch + artifact (2)          artifact (1)
  URL, cache miss        cache + scratch + artifact (3)  cache + artifact (2)
  URL, already cached    scratch + artifact (2)          artifact (1)

Copies on disk at the build's peak, counting the source or cache entry:
3 before, 2 after. A `pre-built` step followed by a script step is
unchanged: the scratch copy is what the script works on. Install and
deploy read the artifact into memory and write nothing more.

The artifact store's own copy stays: it is what `canister install`,
`deploy` and `project bundle` read, possibly in a later process, and it
is a snapshot of the build rather than a pointer at a file that can
change. Sharing it with the source (a hard link, or a reference) would
change that contract and is not attempted here.

A `pre-built` step whose file cannot be read still fails as that step,
now with `failed to read wasm file '<path>'` where it said `failed to
copy wasm to output file`. Through a confined filesystem (`--untrusted`)
the read is refused for the same paths the copy was.

`Builder::new` and `wasm::Fetch` are unchanged, so a consumer that
supplies its own `Fetch` to the library's `Builder` needs no change; one
that implements `Build` itself returns `Module::AtOutput` to keep what it
did.
@raymondk
raymondk requested a review from a team as a code owner November 20, 2026 02:15
Copilot AI balanced review requested due to automatic review settings November 20, 2026 02:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues blocking approval.

No open issues 🎉

What changed in this PR

This PR avoids an unnecessary temporary WASM copy in the canister build pipeline, reducing disk use and memory use on memory-backed filesystems. It preserves the output-file workflow when a later build step needs the module.

Changes:

  • Introduces Module so build steps can return bytes or leave their output in a file.
  • Saves returned bytes directly to the artifact store unless another step follows.
  • Adds unit and integration coverage for step ordering, source-file preservation, and read failures.
File Description
crates/​icp-project/​src/​operations/​build.rs Handles returned bytes, defers temporary writes, and tests step sequencing.
crates/​icp-project/​src/​canister/​build/​prebuilt.rs Returns module bytes instead of copying; adds read-error tests.
crates/​icp-project/​src/​canister/​build/​mod.rs Defines Module and updates the build interface and implementations.
crates/​icp-cli/​tests/​build_adapter_tests.rs Verifies a following script modifies the build output without changing the source.
CHANGELOG.md Documents reduced copying and the updated read-error message.

Review effort: Balanced

…atch update: `Cargo.lock` now resolves wasmtime at 49.0.7 instead of 49.0.5, which is inside the patched range (>=49.0.6, <50) the advisories name. I could not run cargo-deny locally because it is not installed, so the advisory scan itself is unconfirmed until CI re-runs. Invariant: no wasmtime crate in `Cargo.lock` may resolve to a version with an open RustSec advisory. It has to hold for every crate released in lockstep with wasmtime, not just the `wasmtime` package the log flags, so all of them moved together. What changed (only `Cargo.lock`, uncommitted in the worktree): - `wasmtime`, `wasmtime-wasi`, `wasmtime-wasi-io`, `wasmtime-environ` and the `wasmtime-internal-*` crates: 49.0.5 -> 49.0.7 - `pulley-interpreter` and `pulley-macros`: 49.0.5 -> 49.0.7 - the `cranelift-*` crates: 0.136.5 -> 0.136.7 The pulley and cranelift bumps are the ones cargo required to move wasmtime; they are not `wasmtime-*` crates by name, so flagging them against the "wasmtime siblings only" constraint. Nothing else in the lockfile moved, and no entry remains at 49.0.5 or 0.136.5. No `Cargo.toml`, `deny.toml` or source file was touched. The workspace requirement stays `"49.0.5"`, which 49.0.7 satisfies. Verification: - `cargo check --locked -p icp-sync-plugin -p icp-extension` passes, so the lockfile is consistent with the manifests. - `cargo test --locked -p icp-sync-plugin -p icp-extension` passes (55 and 20 tests, 0 failed). The CI log also shows three `advisory-not-detected` warnings for existing zstd ignores in `deny.toml`. They are warnings, not the cause of the failure, and I left them alone
Copilot AI balanced review requested due to automatic review settings November 20, 2026 02:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the build-pipeline change that reduces temporary disk usage.

No open issues 🎉

Review effort: Balanced

@raymondk
raymondk marked this pull request as draft November 20, 2026 03:50
This reverts commit 1b76a77.

Handing the module's bytes back and writing them once to the artifact
store still left a copy per canister. The approach is replaced by
referencing the pre-built file by path, with no copy at all, in the
commit that follows. The tree is back to the pre-change base plus the
lockfile-only wasmtime 49.0.7 update.
A `pre-built` build step resolved its module (a local path, or a URL
through the package cache), copied it to the build's scratch `out.wasm`,
and the build operation then read that back and wrote the same bytes to
`.icp/cache/artifacts/<canister>`. Two copies per canister of a file
that already exists, and on a host whose disk is memory (dfinity/imcp2)
each of them is RAM: fifty canisters built from one module held it fifty
times over.

A pre-built module is now the build's artifact where it lies. Nothing is
copied:

- A build step says where it left the module: `Module::AtOutput`, as
  before, or `Module::Borrowed { path, sha256 }`, a file the build does
  not own. The pre-built step returns the resolved file and the SHA-256
  of what it read there, through the project's `FileSystem`.
- The build operation hands a borrowed module to the artifact store with
  the new `store_artifact::Access::link`, which records the path and the
  hash instead of bytes. The host store keeps that as a small JSON file,
  `<canister>%link`, beside where a saved module would be; a canister
  has one or the other, and no encoded canister name can collide with
  the link's.
- `Access::lookup` resolves the link: it reads the file from where it is
  and hands it over only if it still hashes to what was linked. Install,
  deploy, the candid compatibility check, `canister call`'s offline
  candid lookup and `project bundle` all read the artifact through
  `lookup`, so none of them changes, and `--untrusted` builds as before.
- A module is copied only when another build step follows the pre-built
  one: that step finds it at `ICP_WASM_OUTPUT_PATH` and may change it,
  so it gets a private file and the borrowed one is left as it was.

Disk writes of the module's bytes while building one canister whose only
step is `pre-built`:

  source                 before                          after
  local path             scratch + artifact (2)          none (0)
  URL, cache miss        cache + scratch + artifact (3)  cache (1)
  URL, already cached    scratch + artifact (2)          none (0)

Copies of the module on disk after the build, counting the source or
cache entry: 2 per canister before, 1 in all after, however many
canisters name it. A `pre-built` step followed by a script step is
unchanged (one scratch copy, one saved artifact).

What this gives up is the snapshot. The referenced file is its owner's:
the user's own, or a shared cache entry, never written, moved or removed
by the build. It can change or disappear between `icp build` and a later
`icp canister install`. The store then refuses the lookup, naming the
file and asking for the canister to be built again
(`LookupLinkedChanged`, `LookupLinkedRead`), rather than install a
module other than the one that was built or verified; `install_many`
passes those two errors through instead of reporting a missing artifact.
`icp deploy` and `icp project bundle` build first and are unaffected.
The same check means a link record cannot be used to hand out a file
whose contents its writer did not already know.

A `pre-built` step whose file cannot be read still fails as that step,
now with `failed to read wasm file '<path>'` where it said `failed to
copy wasm to output file`.

`Builder::new` and `wasm::Fetch` are unchanged. `Build::build` now
returns `Module`, and `store_artifact::Access` gains `link`: a consumer
implementing either itself returns `Module::AtOutput` or adds the
method; one using the library's `Builder` and `ArtifactStore` needs no
change.
Copilot AI balanced review requested due to automatic review settings November 20, 2026 19:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Linked-artifact reads bypass filesystem confinement, allowing out-of-root reads or indefinite blocking, while replacing an artifact can delete its own source and leave a dangling link.

2 open issues

Review effort: Balanced

if link.exists() {
let Link { path, sha256 } =
json::load(&link).map_err(|e| lookup_store(name, e))?;
let wasm = read(&path).map_err(|e| LookupArtifactError::LookupLinkedRead {
Comment on lines +324 to +325
remove_if_present(&store.artifact_by_name(name))
.map_err(|e| save_store(name, e))?;
@raymondk raymondk changed the title fix(build): store a pre-built module without a scratch copy feat(build): reference a pre-built WASM by path instead of copying it Nov 20, 2026
@raymondk

Copy link
Copy Markdown
Collaborator Author

Superseded by #887, which keeps each distinct module once in the artifact store instead of linking to the pre-built source file. The scratch copy this PR removed is not addressed there and can follow separately.

@raymondk raymondk closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants