[AbirAbbas]

What this adds

The Factory keeps a durable record of job outcomes and serves it from a cursor, so an app that was not connected when a job merged, finished, failed or started needing a person can still learn of it.

• Factory: at each store write a job outcome is recorded: merged, finished (with how), failed (code, step) or needs_you (need, reason), with the job's own time for that fact. The newest 2,000 are kept in job-outcomes.json beside the live record and served by GET /v1/job-events?after=&limit=&wait= for the jobs the caller may see. The live record and /v1/events are unchanged.
• App wire: factory.events_since {factory, after?, limit?, follow?} answers {factory, supported, events, cursor, reset, more}; with follow the engine pushes later outcomes as factory.event. Hello feature factory.job_events, capability job_events. A Factory older than the route answers supported: false.

Who calls it

The desktop app's Work place (codeaf-desk crates/core/src/engine/factory_events.rs) already calls this behind the factory.job_events hello feature, to wake a workstream's agent when its jobs change. No engine on dev announces the feature today, so that path is switched off.

Where it comes from

This carries #2416, which merged only into workstreams/main. That branch's integration PR (#2306) is closed and will not merge, so the change never reached dev. Adapted to dev as it is now:

• A person's stop is not an outcome. On dev a job a person stopped waits in Backlog and is no longer needs_you (factory.JobState), so stopping a job records nothing. The two "stop enters needs-you" cases of TestStoredOutcomesFromInPlaceNestedUpdates now assert that.
• Visibility is read from the saved job. The follow-up on workstreams/main (0190d57) is folded in: the cursor checks a job's repository with Store.GetSaved, so reading it looks up no author access on GitHub and writes no job. TestJobEventsVisibilityUsesSavedAuthorAccess covers it and fails (3 lookups) with the hydrating read.
• Left out: filling an outcome's group from the job's group. A job has no Group field on dev until the create-extras PR (factory.job_create: issue choice, labels, starting budget, retry key, split_from; factory.job_context_set #2617) lands; the wire field stays and is empty until then. The desk does not route on it.
• The capability flag and the hello line sit on lines no other carried Factory PR touches (factory.job_create: issue choice, labels, starting budget, retry key, split_from; factory.job_context_set #2617, factory.repo_issues, and a GitHub rate limit on a repository check answers retry #2618, Factory: a keyed create that wrote nothing runs again, announced as keyed_create; a rate-limited poll says so #2619); the four merge together cleanly and build.

Tests

CI on this head (87a503a): check, light gate, touched codeaf, touched rest and windows native tests pass.

Locally (macOS), make pr-ready did not pass as one command. The PR is marked ready on the strength of CI and the comparison with dev below:

• The light gate passed.
• The touched packages other than cmd/codeaf passed fresh:

  ok  	github.com/Agent-Field/codeaf/internal/appwire	2.156s
  ok  	github.com/Agent-Field/codeaf/internal/factory/api	0.476s
  ok  	github.com/Agent-Field/codeaf/internal/factory/live	12.271s
  ok  	github.com/Agent-Field/codeaf/internal/factory/serve	80.000s
  ok  	github.com/Agent-Field/codeaf/internal/factory/store	9.509s
  

• The cmd/codeaf leg was refused inside pr-ready by the one-heavy-suite lock (other sessions were running suites on the machine), so it was run by itself once the lock was free (make test PKGS=./cmd/codeaf TEST_FLAGS='-count=1 -p 2' SHARDS=8). 15 tests failed. None is attributable to this change:
  • 6 passed when run again alone: TestAppTakeC1UnavailableModelUsesLocalDefaultOnce, TestDoReturnsAfterCancelledAccountingAndItsWriterStop, TestFactoryPersonStopInFlightWakeAndPersonChat, TestFactoryStartBadRepositoryFile, TestHostedTakeBackReachesAcquireWithoutSharedUploadsOrReleaseDelay, TestProcessCloseJoinsTheUsageWriter.
  • 9 fail the same way alone on a clean origin/dev (ffeeb78) on the same Mac: TestAppFactoryPlanFilesAnswer, TestFactoryCommandStepRetryAfterOutOfAttempts, TestFactoryMeasuredOnlyPublication, TestFactoryRetryActionsDoNotReportStopped, TestFactoryStepRetryWaitsForPreviousLeaseCleanup, TestFactoryWriteUsesJobBaseAcrossAttempts, TestMemoryPauseResumeAfterTrailingProgress, TestMemoryPauseRetainsPendingForResume, TestMemoryPausedJobLifecycle (for example write is not green: exit 127: the Factory's command steps do not find their shell on this Mac).

The change's own tests pass locally: the ten TestAppFactoryJobEvents* tests in cmd/codeaf, TestFactoryJobEvents* in internal/appwire, TestJobEvents* in internal/factory/serve, and the outcome tests in internal/factory/live and internal/factory/store.

An engine built from this branch with #2617 and #2618 merged in announces factory.job_events in hello and answers job_events: true from factory.capabilities (captured from codeaf app in an empty home, no provider key).

Changes after review

After this PR was marked ready, it was tested by hand on a built engine (a real codeaf factory serve, codeaf runner join and codeaf app) and reviewed in three passes. Every defect found is listed below. Each was fixed on this branch with a test that fails before the fix and passes after it.

The head is now 9d21962 (it was 87a503a). It adds 17 commits (fixes, their tests and their docs) and one merge of dev. The merge was needed for #2618's rate-limit handling (fix 12). It also brings in #2609, #2612, #2613, #2614 and #2615, which are already on dev. A conflict in meaning was resolved in the merge commit 73fb26c (see fix 12). The "Tests" section above describes the earlier head, 87a503a.

What factory.job and the wire now say differently

• needs_you.since is when the wait began. It stays put while the job waits with the same need and question.
  • A wait with no clock of its own used since = the job's last change, so it moved with every rename, priority, budget or note. These waits are a start held for approval (ask_first), a take-over, an AgentField run's approval, a Stop undone at a gate or question, a failed merge, and a pull request with requested changes or a conflict.
  • A failed publish used since = the landed run's end. On a Factory service it used the current step's time until the runner's done arrived. It now uses the time the publish failed, so a publish that is retried and refused again waits from its own refusal.
  • A job parked at its budget on a Factory service used since = the arrival of the runner's latest report. It now keeps the time the budget question was first asked.
• Changing the assignee of a running or waiting job, or unassigning it, leaves it in Backlog. The job has no job-level failure and no question (waiting_since is absent), and it offers Run as new; the run_as_new door accepts it. The step the change cut still shows the stop. Before, a reassigned job waited in Needs you as failed, "Stopped by you", with the cut run's question still on it. An unassigned job kept the question and never offered Run as new.
• Answers that stop a job leave it in Backlog, "Stopped by you", as Stop does. These are a cancel to a step out of attempts, and a runner's loop stopping on a person's answer (a no to the budget, or a send back with no earlier step to go back to). Before, once a Start had cleared an earlier stop, they left the job in Needs you as failed. On a Factory service they always did.
• GET /v1/job-events can answer HTTP 503 "try again" (the engine's engine error) when the first outcome it would return belongs to a repository GitHub could not answer for just now. That covers a 5xx, a failed connection or a spent rate limit. A page ends before such an outcome with more: true.
• The engine's follower backs off on a repeated reset. When a reset leaves its cursor where it was, it waits 2 seconds, doubling to a minute. The Factory closes a gap on the read that finds it.
• New store field Item.NeedsAt (fix 10). It is not on the wire and is listed in api.Lost.

Pass 1: found by hand on 87a503a

1. A wait was recorded again on every write. A job waiting at its plan gate recorded a new needs_you on every later write: priority, budget, rename, and each runner report. A paused decision did the same.
   • Cause: Store.update compared the new job with live.OutcomeBasis, a partial copy of the prior job. The copy dropped each phase's kind, round and name, and the paused-recovery question, so the prior wait read as a different need (approve_step against approve_plan) or a different reason. The copy was needed on workstreams/main. On dev, since Carry the open app/v3 pull requests that integrate cleanly onto dev (16 of 22) #2597, the update callback already gets a decoded copy.
   • Fix: OutcomeBasis is removed. Store.update passes the whole prior job, as Store.Save already did.
   • Test: TestStoredNeedsYouOncePerWait (store).
2. A person's stop was recorded as failed (code stopped).
   • Roads: Stop at a gate or in a step, the local Runner.Stop, a step's own Stop, a no to the stop after a step, and a cancel out of attempts. Close and Reject recorded failed beside their finished, and so did the reset before a reassignment.
   • Cause: every road that ends a run on a person's word writes the stop as a failure coded stopped, and live.Outcomes recorded any new failure. The PR's "explicit stop is quiet" test never wrote that failure.
   • Fix, in its final form after pass 2: a failed outcome follows factory.JobState. A Backlog job records nothing. A job in Needs you always records it. A done job records any code except stopped.
   • Tests: TestStoredPersonStopIsNoOutcome (store); TestServiceStopDoorsRecordNoFailedOutcome (lease, through the real doors); TestLoopPersonStopsRecordNoOutcome and TestLoopStepFailureRecordsOneFailedOutcome (run); TestFailedOutcomeFollowsTheJobState (live).
3. A refused publish on a Factory service was recorded twice as publish_failed.
   • Cause: the first record came at the runner's report, read with the current step's clock while the service's copy was still running. The second came at the runner's done, read with the landed run's end. factory.job's needs_you.since moved the same way.
   • Fix: NeedsSince answers a failed publish's own time, PublishStatus.At.
   • Tests: TestStoredPublishFailureOnce (store); TestJobEventsRecordEachRefusedPublishOnce (serve, the real worker HTTP road, including a retry that is refused again).

Found by review of pass 1, fixed in pass 2

4. Some stops showed as failed but recorded nothing. The pass-1 stop rule skipped any stopped failure, but several roads write that failure without marking the job as stopped by its person. On them factory.job showed Needs you / failed "Stopped by you" while the events recorded nothing.
   • Roads: the reset before a change of assignee or an unassign; a cancel out of attempts when no runner holds the job; the runner loop's own stops on a person's answer; and, on a Factory service, every stop the runner made. The served copy learns of such a stop from the runner's report, which does not carry the person-stop mark.
   • Fix, road by road:
     • A cancel when no runner holds the job, and the loop's stops on a person's answer, now mark the stop and take the ask down, so the job waits in Backlog.
     • reportStep marks a reported step failure coded stopped as the person's stop on the served copy.
     • The reassignment reset drops the job-level stop failure it has just written, so the job is handed on in Backlog. The cut step keeps the failure.
     • failed follows JobState, as in fix 2.
   • Tests:
     • TestJobEventsStopRoadsAgreeWithFactoryJob (serve), with one row per road: stop at a gate and mid-step, a step's own stop, close, reject, a cancel when no runner holds the job, reassign running and at a gate, unassign, and a tool_error reported by a runner.
     • TestJobEventsRunnerMadeStopsAgreeWithFactoryJob (serve): a cancel out of attempts and a no to the budget, each reported by the runner as the worker reports it.
     • TestLoopAnswersThatStopLeaveTheJobInBacklog (run).
5. A parked budget was recorded again on every runner report. A job parked at its budget on a Factory service recorded a new budget needs_you each time.
   • Cause: the worker sends the parked budget question again, with no clock, on every report that carries messages, usage or acks, and reportQuestion stamped it with the report's arrival.
   • Fix: when the same ask still stands and the re-sent question has no clock, reportQuestion keeps the first time and writes no second ask line.
   • Test: TestJobEventsRecordAParkedBudgetOnce (serve).
6. factory.job and the events disagreed about a person's stop. factory.job keeps failure {code: stopped} on a Backlog job and on a closed or rejected job, so a client forming occurrences from a full read keyed it as a failed job.
   • Fix: no wire change. PROTOCOL now states the rule a full read follows:
     • a failure is a failed job only on needs_you, or on done with a code other than stopped;
     • a wait counts only on needs_you with no failure.
   • Test: every row of the serve stop table in fix 4 checks that each key a full read forms by this rule is a recorded outcome. The matching desk change is codeaf-desk guard: the goroutine sweep reads the defers, and every bare spawn in the guarded tree runs under guard.Go #414.
7. The run-package stop tests ran only on the runner's mirror. Those tests drive the runner's own store, which in production is the runner's mirror and is never served. The serve worker-road tests in fix 4 cover the served copy.

Also fixed in pass 2

8. The follower read back to back during a gap. The engine's follower read again at once after a reset that left its cursor where it was, because a Factory answers reset immediately while a gap lasts, until its next commit. In the test it made 3,238 reads in 300 ms.
   • Fix: the follower waits 2 seconds, doubling to a minute, while that repeats. The Factory also closes the gap on the read that finds it (live.HealOutcomes), so a gap resets a reader once.
   • Tests: TestAppFactoryJobEventsGapResetWaits (cmd/codeaf), TestJobEventsGapResetsOnce (serve), TestHealOutcomesClosesOnlyAGap (live).
9. A signed-in reader lost an outcome when GitHub did not answer for the job's repository (a 5xx or a failed connection). The outcome was decided as hidden and the cursor moved past it.
   • Fix: repoAllowed also says whether its answer is known. A page ends before an undecided outcome with more: true, and a read that would start at one answers 503.
   • Test: TestJobEventsWaitOnAnUndecidedRepository (serve).

Found by review of pass 2, fixed in pass 3

10. A wait with no clock of its own was still recorded again on every write, and factory.job's needs_you.since moved with it, because NeedsSince fell back to the job's last change.
    • Roads: intake ask_first, an Undo of a Stop at a gate or question, control take_over, and an AgentField run's approval wait. A search found three more: a failed merge, and a shipped pull request with changes requested or with a conflict.
    • Fix: one clock kept by the store instead of one per road, so no road can forget it. A new field, Item.NeedsAt, is set by factory.NoteWait on every store write (store.update and Store.Save):
      • The write that begins a wait stamps it. A wait begins when the job enters Needs you, or when its need, question kind or question changes.
      • A continuing wait keeps it.
      • A job that waits on nobody has none.
      • A waiting record written by an older binary adopts the since it already showed, so an upgrade records nothing new.
    • NeedsSince reads NeedsAt after the job's own clocks (failure, publish, landed, ask, step) and before the last change, so a wait that already had a clock keeps exactly the since it had. NeedsAt is store bookkeeping: it is left out of the action-revision comparison and named in api.Lost.
    • Tests:
      • TestJobEventsRecordAClocklessWaitOnce (serve), through the real doors and reports. Its rows are ask_first, stop then undo at the plan gate, take_over, and an AgentField waiting run. Each row then takes four stamped writes.
      • TestStoredClocklessWaitOncePerWait (store): a failed merge, changes requested, a conflict, a take-over.
      • TestStoredWaitClockIsEachWaits (store): an old record, a new wait, a job that waits on nobody.
11. A job handed on to Backlog kept its old question. A change of assignee or an unassign of a job waiting at a gate or out of attempts left the cut run's question standing, so factory.job showed waiting_since on a Backlog job, which a full read would key as a wait.
    • Cause: the reset clears the question and question kind but never the ask, and neither assign nor unassign cleared it.
    • Fix: the reset takes the ask down when it cuts a run. unassignJob also clears it, because no reset cuts a run that has already failed.
    • Tests: new rows in TestJobEventsStopRoadsAgreeWithFactoryJob: reassign out of attempts; unassign at the plan gate; unassign out of attempts; unassign a failed job that still asks. Every row now also asserts no waiting_since and no needs_you unless the job is in Needs you, and the table's full-read model includes the waiting_since fallback.
12. A GitHub rate limit was read as a definite "hidden" for a job's repository and cached, so a signed-in reader whose token was rate-limited lost those outcomes for the cache's life.
    • The fix needed factory.repo_issues, and a GitHub rate limit on a repository check answers retry #2618 from dev, so dev was merged in, with no rebase and no force. factory.repo_issues, and a GitHub rate limit on a repository check answers retry #2618 makes a spent limit (any 429, or a 403 with X-RateLimit-Remaining: 0 or Retry-After) return GitHub's retry time. It never caches the limit as a denial, and it remembers the limit per token.
    • The merge had no textual conflict but did not compile: repoAllowed now returns (allow, known), and factory.repo_issues, and a GitHub rate limit on a repository check answers retry #2618's remembered-limit branch returned a bare false. In the merge commit that branch returns undecided, because a limit says nothing about the repository.
    • The job-events route keeps its 503. factory.repo_issues, and a GitHub rate limit on a repository check answers retry #2618's 429 with retry_at is meant for routes that name a repository.
    • Test: TestJobEventsWaitOutAGitHubRateLimit (serve), with a spent primary limit and a 429 secondary limit. It fails if the remembered-limit branch answers known. factory.repo_issues, and a GitHub rate limit on a repository check answers retry #2618's own tests pass on the merged tree.
13. A change of assignee that cut a run lost Run as new. After fix 4 the job is New with no failure, so RunEnded was false and the run_as_new door refused it.
    • Fix: factory.CanRunAsNew also accepts a run that a reset cut: state New, the stream ended, and the current step's failure coded stopped. RunEnded is unchanged, so lease reset decisions and brief wording do not move.
    • Consequence: an unassign that cuts a running or waiting job now also offers Run as new, which it did not before this PR. An unassigned failed job still offers nothing.
    • Test: TestJobEventsAHandedOnRunOffersRunAsNew (serve): reassign a running job, reassign at the plan gate, and unassign a running job. A job that never ran must not offer it.

Known limits left

• A 403 secondary rate limit that carries neither Retry-After nor X-RateLimit-Remaining: 0 is still read as a denial and cached. That is factory.repo_issues, and a GitHub rate limit on a repository check answers retry #2618's rule, which judges by headers only.
• markRunnerLost leaves a standing ask on a failed job in Needs you, so factory.job can show waiting_since on a failed job. It is not a Backlog road, and a retried step may need the ask. The full-read rule ignores waiting_since on a failed job.
• A job created already finished or waiting records nothing, as before.
• A job already waiting at publish_failed when the engine is upgraded shows a new needs_you.since once, so the desk may wake once more for it.
• A no to an approval with no earlier step, on a held job, reads Backlog (stopped by its person) on the served copy, but not Rejected. That predates this PR.

Tests on this head (9d21962)

• Each fix's test above failed before its fix and passes after it. Pass 2's verifier also reverted each non-test hunk of that pass on its own and saw its proving test fail.
• Whole packages, go test -p 2, all ok: the internal/factory root, live, store, lease, api, run, serve and worker. go build ./... and go vet ./internal/factory/... are clean.
• cmd/codeaf by name: ^TestAppFactoryJobEvents, 11 of 11 pass.
• make test-quick exits 0. make test-packed-manual manual-gates is ok (the manual changed). make changelog-check reports every entry well formed.
• Hand runs on a built engine at b2aee5d (after pass 1) and c7ee922 (after pass 2). The app had no provider key, and model spend was about $0.10 in all. They showed:
  • one needs_you per plan-gate wait across later writes;
  • no outcome for a stop at a gate, mid-step, after a step, or while waiting to retry;
  • one publish_failed per refusal, including after retry_publish;
  • one budget outcome across eight runner reports while parked;
  • only finished for a Close;
  • one failed: machine_lost for a runner killed with SIGKILL;
  • unassigning a running job leaves it in Backlog with no outcome;
  • the follow stream pushed each stored outcome once, in order, across service restarts;
  • a gap reset the follower once.
• Fixes 10 to 13 were not run by hand. They are covered through the real serve doors and worker reports.

🤖 Generated with Claude Code