Repository navigation
📨 fix: Deliver Lane Cleanup and Quarantine Receipts When a Worker Run Stops - #402
Conversation
…tops A concurrent worker run that stopped after a lane's settlement committed dropped the cleanup acknowledgement and the quarantine receipt, because both were bound to the run's abort signal. Code API offers only unsettled work on reconnect, so nothing re-sent them and the lane's fence stayed until reset, also blocking its parent checkout. One failed registration heartbeat was enough to stop every slot this way. The cleanup acknowledgement for a committed settlement and the quarantine receipt are now delivered with their bounded retries regardless of the run stopping. A quarantine receipt whose delivery stays uncertain is kept and delivered after the next registration, before any slot leases. A concurrent run absorbs one transient heartbeat failure before stopping its slots. Uncertain outcomes still quarantine; the receipt now also reaches Code API.
|
@codex review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. 👍 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review — final review. Please check that receipts for already-settled assignments survive a stopping run without weakening quarantine of uncertain outcomes, and that the one-retry heartbeat stays bounded. |
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Summary
Follow-up to #401. That PR stopped the Code API from waiting on a lost cleanup acknowledgement after a rejected lane settlement. This PR fixes the worker so the acknowledgement, or a quarantine receipt, is not lost in the first place. It also covers fulfilled results, which #401 does not.
The breakage
A linked-worktree lane (and, the same way, any root under concurrent workspace slots) holds a server-side fence for each assignment. After a settlement, Code API releases that fence only when the worker either acknowledges local cleanup (
workspace-cleanup) or reports the lane as quarantined (quarantine). Until then, the lane refuses new work, and so does the checkout above it (WORKSPACE_QUARANTINED).On the worker, both of those calls were bound to the concurrent run's abort signal. If the run stopped after a settlement had committed but before the call went out,
reportWorkspaceOwnershipgave up without sending anything. The fallback quarantine receipt then gave up the same way. On reconnect, Code API offers only unsettled assignments again, so nothing ever re-sent the call. The fence stayed, with no TTL, until an operator reset it.The trigger
Any failure that stops the concurrent run does this to every slot. The most likely one is a single failed registration heartbeat: inside
runConcurrentthe heartbeat had no retry and stopped all slots on its first transient error.On the hosted bridge, an
execute_commandon a lane was rejected by the worker. One second later a sibling lane's in-flight command was quarantined. That pattern matches a run stop followed by the worker's one-second reconnect. The rejected lane's fence then held its plain assignment id for hours, the checkout above it refused every request, and a request queued behind it on the same lane failed only when its slot reservation expired 65 s later.The fix (
packages/code/src/worker.ts)WORKER_FENCED,WORKER_QUARANTINED) still stop immediately. The two callers that already retried transient failures without limit keep that behavior (Number.POSITIVE_INFINITY).Uncertain outcomes still quarantine. An interrupted command still quarantines its local guard and is never settled as fulfilled. What changes is that its quarantine receipt now actually reaches Code API when the run is stopping, instead of leaving a plain fence behind.
Compatibility
There is no protocol change and no opt-in flag. These are the same endpoints and bodies, which every Code API since concurrent workspace slots already accepts. A late
workspace-cleanupafter #401 is idempotent. A latequarantineis still checked against the assignment's receipt and the workspace's reset epoch, so it cannot fence a root that was reset after the assignment.Trade-offs
Testing
packages/code/src/worker-receipts.test.tsdrives the realBridgeWorker.runagainst a fixture bridge that records every control call:read_fileand a committedexecute_commandresult still sendworkspace-cleanupwhen the run stops right after the settlement.quarantinereceipt is delivered.main. The sixth, "two consecutive heartbeat failures still stop every slot", passes onmaintoo and guards against the retry becoming unbounded. All 6 pass with this change; I ran the file three times.packages/codenpm test: 1000 pass, 9 fail. The same 9 fail onmainlocally (programmatic executor and storage-permission tests that depend on the environment).src/bridge/concurrent-worker.test.ts,src/bridge/instructions.test.ts): 8 pass.tsc -p packages/code/tsconfig.json --noEmitis clean.