Skip to content

fix: idle Factory polls read only job documents that changed - #2361

Closed
santoshkumarradha wants to merge 13 commits into
app/v3from
fix/c61-floor-reads
Closed

santoshkumarradha wants to merge 13 commits into
app/v3from
fix/c61-floor-reads

Conversation

@santoshkumarradha

@santoshkumarradha santoshkumarradha commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Idle Factory polls no longer read every job document on each look. Held: do not merge until the owner has reviewed before-and-after numbers and authorized it. Stacked on #2358; retarget after that lands.

What changed

With #2358 applied, the largest remaining engine work on a real install with 355 jobs is store.List. A read-only sample of the idle engine shows 135 samples in List: lease Pull 82, job settlement 35, runner capacity 12, OverLimit 11. Each look decoded every document.

  • List now stats every file on every look and decodes only files whose identity, size or time changed. It returns a deep copy, so callers, migration and output retention never touch the kept document.
  • Lease grants and execution gates still read the job under its lock.
  • List reads every document again at least once a minute, so a decision never rests on file times for longer than that.

Measurements

Read counts on a floor of 300 jobs, unchanged floor inside the 60 s bound, from the tests:

Route Before After, unchanged After, one job changed
Pull 900 0 1
OverLimit 900 0 1
Settle and capacity 1800 0 1

Real app, Mac Studio (M3 Ultra), 355 jobs, 3 job chats open, about 4 minutes after start. Two runs each, same desktop build, same session, including the 60 s re-read. Not the laptop. Before is 56d8b5acb (#2358 without its re-read); after is 4aa44a98f (this PR and #2358's re-read). The later merges of app/v3 were not re-measured.

Before After
Engine CPU used at about 3:26 16.0, 15.5 s 14.1, 14.4 s
Idle, three 20 s windows 8.9, 8.9, 3.7% and 8.5, 8.9, 3.6% 7.3, 8.0, 3.5% and 7.5, 8.7, 3.5%
Memory (RSS) 240, 241 MB 254, 258 MB
Key to paint, p50 / p99 24.8 / 40.2, 24.6 / 38.6 ms 24.7 / 40.6, 24.8 / 39.7 ms

The gain on this machine is small: about 10% less engine CPU for about 14-17 MB more memory. Typing does not change. The target is the laptop, where this work is a larger share.

Laptop before and after: not measured yet. It needs an owner-approved install. This PR stays draft until those numbers are here.

Correctness tests

Case Test
Another process's Stop, Start, budget change, create, delete, take, release, rename TestPullAndCapacitySeeExternalFloorChanges, TestSettleSeesExternalStopStartAndBudget
Fresh authority after selection TestPullRechecksLockedJobAfterListing, TestSettleRechecksStopAndBudgetAfterListing; the lease fix itself landed separately in #2362
Ownership, retention, migration TestListOwnsMutableDocuments, TestListReappliesOutputRetention, TestListReappliesRepositoryMigration
Replacement with the same size and time TestListSeesRenameWithSameSizeAndTime
Looks racing writes, under -race TestFloorLooksRaceExternalWrites

Limits

  • The kept documents add memory, since they hold the output that retention trims from what callers see. Measured in the pending rows.
  • Every look still stats every file. Only decoding is avoided.
  • Every look still trusts file identity between full reads. An in-place rewrite that keeps the same inode, size and modification time can wait up to 60 s. List reads every document again at least once a minute, the bound, tested by TestListSeesInPlaceSameTimeEditWithinItsBound. Grants still read under the lock.

🤖 Generated with Claude Code

santoshkumarradha and others added 13 commits November 4, 2026 13:43
List stats every numeric job file on every call and copies its cached JSON graph before migration, retention or caller mutation. Locked reads and writes bypass the cache.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Keep file-identity reuse and detached List results. Prove external changes, retention, migration and concurrent looks. Derive thread-only authority under the item lock so a Stop during selection cannot restart job steps.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
An Update callback holding an item lock could deadlock against a List migrating
that item. Published indexes are never changed, so a scan now reads a snapshot
and the lock guards only the pointer.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
File identity cannot see an in-place rewrite that keeps its size and time, and a
Stop or budget must not rest on file times with no limit.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>

# Conflicts:
#	internal/factory/lease/lease.go
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Base automatically changed from fix/c61-stop-watch-cpu to app/v3 November 11, 2026 17:42
@AbirAbbas

Copy link
Copy Markdown
Collaborator

Closing because this is already on dev. Every behaviour this PR adds landed through other PRs after app/v3 reached dev (#2561, #2597):

The branch stays as it is, so nothing is lost.

@AbirAbbas AbirAbbas closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants