Skip to content

Factory: a keyed create that wrote nothing runs again, announced as keyed_create; a rate-limited poll says so - #2619

Merged
AbirAbbas merged 6 commits into
devfrom
fix/factory-github-retryable
Nov 13, 2026
Merged

AbirAbbas merged 6 commits into
devfrom
fix/factory-github-retryable

Conversation

@AbirAbbas

@AbirAbbas AbirAbbas commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

What this fixes

  • A keyed job create that wrote nothing runs again; one that wrote something is replayed. The Factory's idempotency cache replayed the first answer under an Idempotency-Key for 24 hours. That included server errors and refusals given before anything was written, so a resend after a transient failure, or after the person fixed a refusal, could never succeed. A job create (POST /v1/jobs, POST /v1/issues) now starts with the X-Codeaf-Phase: pre-mutation marker and clears it just before its first write: the GitHub issue request once it has a connection, and the job's own make. An answer still marked is not kept, so the same key runs the create again. An answer given after that, a 5xx or 408 included, is replayed, so a resend does not make a second job or issue. Rows an older build stored carry no marker and are replayed. Worker and runner routes keep their rule: a 5xx, 408 or 429 is never kept. Two limits remain. The row is keyed by the exact Authorization and Cookie, so a resend under a new bearer or session cookie runs as a new create. And it is saved only after the create commits, so a Factory that stops in between runs the resend again.
  • The Factory says so. GET /v1/features (factory.features) lists keyed_create, with or without a store. An engine can then tell this Factory from an older one, which keeps every keyed answer for 24 hours, refusals and server errors included, and send a create key only here. factory.job_create: issue choice, labels, starting budget, retry key, split_from; factory.job_context_set #2617 does exactly that.
  • A rate-limited source poll says so. TroubleWords reads "rate limited" for any GitHub 429 and for a 403 that carries a retry time; a 403 without one still reads "not reachable".

Where it comes from

This carries the poll words of #2439 and #2460, which merged only into workstreams/main. It replaces #2439's keyed-replay rule: #2439 stopped keeping every 5xx, 408 and 429, so a create that failed after making its job or issue ran again and made a second one. That branch's integration PR (#2306) is closed and will not merge. The visibility-check halves of both are in #2618, because they answer in the repository issues read's shape.

Both faults are still on dev. With only the tests brought over:

--- FAIL: TestAKeyedCreateThatWroteNothingRunsAgain
    keyed_create_test.go:305: resend after the cause passed = 502 {..."error":"GitHub isn't answering; retry this job change"...}
    keyed_create_test.go:305: resend after the cause passed = 422 {..."error":"pipeline not found"...}
--- FAIL: TestAKeyedServerErrorRunsAgainOnRetry
    idempotency_transient_test.go:130: attempt 0 = 503 "retry this answer", header="retry this answer" phase="" calls=1; want 503 "retry this answer" phase="pre-mutation" calls=1
--- FAIL: TestTheFactorySaysItRunsAKeyedCreateThatWroteNothingAgain
    keyed_create_feature_test.go:31: features = [group_policy waits_on], want "keyed_create" listed
--- FAIL: TestTroubleWordsNameASpentRateLimit
    poll_limit_test.go:32: trouble words = "not reachable", want "rate limited"

TestAKeyedCreateAnsweredAfterItsWriteIsReplayed passes on dev, because dev keeps every answer. It fails on #2439's rule, which made a second issue or job.

Tests

Locally (macOS), on this head (c937532):

  • go test -count=1 -p 2 ./internal/factory/serve/ ./internal/factory/api/: both ok (internal/factory/serve in 58.7s).
  • In cmd/codeaf, by name with -p 2: TestADeskGoAheadStartsTheCoveredJobAfterItsParentMerges and the four TestAppFactoryJobAction* tests: ok.
  • make test-quick: exit 0.
  • make changelog-check: 795 entries, all well formed.
  • make test-packed-manual manual-gates: exit 0.
  • internal/factory/github (the poll words) is unchanged since 3489017, where make pr-ready ran it fresh: ok in 40.2s. That run kept gh off PATH: on this Mac TestAHangingGHNeverHoldsTheReadsThatAskForTheToken fails on a clean origin/dev too when the real gh is on PATH (a gh that answers at once: "" via ""). It is not touched by this change.

Merged with #2617 (f0996a4) on dev 6616e85, in a throwaway worktree: go build ./... ok; internal/factory/serve, api, run, store and internal/appwire ok; 41 named cmd/codeaf tests pass. One of them was a throwaway end-to-end test: the real merged Factory gets create_key and the key, a pre-write 501 under the key runs again, and then the same key replays one job. The same server answering features as an older Factory does gets create_key: false, a keyed create answers factory_too_old, and the older Factory never sees the key. make test-packed-manual manual-gates, make changelog-check (803 entries) and make test-quick exit 0.

CI on this head (c937532): check, light gate, touched codeaf, touched rest, touched session, touched tui3, touched packages and windows native tests pass.

Changes after review

Round 1. The first version carried #2439's rule as it was: no 5xx, 408 or 429 answer under a key was kept. Review found that a job create can fail after it has made its GitHub issue or its job: a lost GitHub answer, a failed store write, a failed start. The same-key resend then ran again and made a second issue or job. The fix (9b590fe) reuses the Factory's existing X-Codeaf-Phase: pre-mutation marker. POST /v1/jobs and POST /v1/issues start marked. newJob clears the marker just before seam.New or seam.NewJob, and the GitHub issue request clears it once GitHub may have acted: a failure after the request got a connection, or any answer but GitHub's own 4xx refusal. Only a marked answer is not kept. The in-memory capture had dropped the marker the handler cleared, so a keyed answer always looked pre-write; it now copies it, and a replay never claims nothing was written. Worker and runner routes keep dev's rule. The PROTOCOL paragraph, a new manual section and the change entry describe the rule.

Pass 2. Review then found three more things.

  • With factory.job_create: issue choice, labels, starting budget, retry key, split_from; factory.job_context_set #2617, the engine sent a fixed per-draft key to Factories of any age. A Factory older than this PR keeps every keyed answer, refusals included, so a draft refused as too old got the same refusal back for 24 hours, even after the Factory was updated, because the old row carries no marker. Rather than guess which old rows to drop, the Factory now announces the rule: GET /v1/features lists keyed_create (new api.FeatureKeyedCreate, 878ecd9). factory.job_create: issue choice, labels, starting budget, retry key, split_from; factory.job_context_set #2617 sends a key only to a Factory that lists it, so no engine sends a key to a Factory that would keep a pre-write answer. internal/factory/api/keyed_create.go is the same file byte for byte on factory.job_create: issue choice, labels, starting budget, retry key, split_from; factory.job_context_set #2617, so the two merge in either order.
  • TestKeyedSuccessAndRefusalsStillReplay claimed that a /v1/jobs refusal replays for 24 hours. It passed only because it skipped the route's marker; a real create refusal is marked and not kept. It is replaced by TestAKeptKeyedAnswerReplaysForADay, which goes through the marker and covers a create's success and an unmarked verb's 422 and 409, in memory and on disk. A create refusal the person fixed now runs again on the real handler (TestAKeyedCreateThatWroteNothingRunsAgain/a_job_create_refusal_the_person_fixed). The marker clear before seam.NewJob is now proven too: a new case makes the job through NewJob, and it fails when that clear is deleted (0a97315).
  • The docs said a same-key resend never makes a second job or issue. They now state the two limits the code has: the exact Authorization and Cookie in the row's key, and the row saved only after the create commits (c937532).

The title and this description were rewritten to match.

🤖 Generated with Claude Code

AbirAbbas and others added 2 commits November 12, 2026 12:13
…a rate-limited poll says so

Carried from #2439 and #2460 on workstreams/main (closed integration PR #2306).
The idempotency cache no longer keeps or replays a try-again answer (5xx, 408,
429), including rows an older build stored, so a keyed retry after a transient
failure can succeed. A source poll that GitHub answers 429, or 403 with a
retry time, reads "rate limited" instead of "not reachable".

Co-Authored-By: Claude Mythos 5.2 <[email protected]>
Co-Authored-By: Claude Mythos 5.2 <[email protected]>
AbirAbbas and others added 4 commits November 12, 2026 18:28
The keyed-answer rule this branch carried from #2439 stopped keeping every
5xx, 408 and 429. A job create that failed after it had made its job, or
after its GitHub issue request had gone out, then ran again on the same-key
resend the engine calls safe after an uncertain answer, and made a second
job or a second GitHub issue.

The rule is now: a keyed answer is replayed unless its handler said it
committed nothing. The handler says so with the existing phase marker
(X-Codeaf-Phase: pre-mutation). POST /v1/jobs and /v1/issues now start
marked. newJob clears the marker immediately before seam.New/NewJob, and
githubJobJSON clears it once GitHub may have acted: the request got a
connection, or GitHub answered anything but its own 4xx refusal. An answer
still marked (a refusal, the visibility check, a sign-in or read failure, a
GitHub refusal, or GitHub unreachable before the request was sent) is not
kept, so a resend runs again once the cause passes. Every answer after a
write is replayed, a 5xx included. The capture writer now receives the
marker from the real writer, so clearPhase reaches it, and a replay never
carries it. Worker and runner routes keep dev's rule.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The Factory now says it applies the keyed create rule: a keyed POST
/v1/jobs or /v1/issues answered before its first write is not kept, so
the same key runs again. An older Factory keeps every keyed answer for a
day, refusals and server errors included, so an engine must be able to
tell the two apart before it sends a create key. The word is additive
and lives in the request path's rule, so a host without a store lists it
too. The constant sits in its own file, identical on #2617, so the two
branches merge without a conflict.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…'s clear is proven

TestKeyedSuccessAndRefusalsStillReplay called idempotent without the
marker ServeHTTP sets, so it claimed a /v1/jobs 422 replays for a day,
which the real route no longer does. It becomes TestAKeptKeyedAnswerReplaysForADay:
a create's success and an unmarked verb's refusal, both through the
marker. TestAKeyedCreateThatWroteNothingRunsAgain gains a real /v1/jobs
refusal the person fixed. A create through the seam's NewJob whose start
failed must be replayed, and an unkeyed one made through NewJob carries
no marker: both fail if the clearPhase before seam.NewJob is deleted.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The docs said a resend with the same key never makes a second job or
issue. The replay row is saved only after the create commits, so a
Factory that stops in between runs the resend again, and the row is
keyed by the exact Authorization and Cookie headers, so a resend under
a new bearer or session cookie is a new create. PROTOCOL.md, the manual
and the change entry now say both.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
AbirAbbas added a commit that referenced this pull request Nov 13, 2026
The create text promised that the Factory replays the first response for
24 hours, so a same-key resend after an uncertain answer is safe. #2619,
which merges with this, changes which keyed answers are replayed: one given
before the create wrote anything runs again. PROTOCOL.md, the chat manual
and the change entry now state only what is true on either branch: within
24 hours a same-key resend never makes a second job or issue, a different
key makes a separate job, and a still-uncertain resend means read the
queue. The change entry says this must merge with #2619, which carries the
replay rule itself.

Context attachments: the docs said markdown must be non-empty, but the
engine and the Factory both refuse whitespace alone ("context markdown
must be non-empty"). The docs now say so, and the Factory's refusal table
pins the blank case the engine's table already pinned.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
AbirAbbas added a commit that referenced this pull request Nov 13, 2026
…ed_create

A Factory older than #2619 keeps every keyed answer for 24 hours, a
refusal or a 5xx included, so the desk's fixed per-draft key would
replay a transient failure or a fixed refusal for a day and the draft
could not launch. The engine now asks the Factory's GET /v1/features on
every create with a key and sends the key only when it lists
keyed_create; to any other Factory a keyed create answers conflict
factory_too_old and no create is sent. A create without a key is
unchanged and asks nothing.

factory.capabilities takes an optional factory: create_key is true only
in an answer naming a Factory that lists the word, and false without
one, so an app that reads capabilities once sends no key. On its own
this branch never sees the word and never sends a key. The word's
constant sits in its own file, identical on #2619.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@AbirAbbas AbirAbbas changed the title Factory: a keyed request is never answered from a cached 5xx or 429; a rate-limited poll says so Factory: a keyed create that wrote nothing runs again, announced as keyed_create; a rate-limited poll says so Nov 13, 2026
@AbirAbbas
AbirAbbas merged commit 9d3afc6 into dev Nov 13, 2026
10 checks passed
@AbirAbbas
AbirAbbas deleted the fix/factory-github-retryable branch November 13, 2026 02:22
AbirAbbas added a commit that referenced this pull request Nov 13, 2026
… split_from; factory.job_context_set (#2617)

* factory.job_create: issue choice, labels, starting budget, idempotency key; factory.job_context_set

Carried from #2415 on workstreams/main (closed integration PR #2306), adapted to
dev: the Factory capability struct, wire Job and Item keep dev's go-ahead
fields; the new job's group, milestone and context sit beside them. The job
outcome's group is left out here because job outcomes are a separate change.

Co-Authored-By: Claude Mythos 5.2 <[email protected]>

* factory.job_create: split_from records the job a new job was split from

Carried from #2429 on workstreams/main (closed integration PR #2306). The
parent check reads the saved job, as the follow-up there made it, so checking
a split asks GitHub nothing about the parent's author and never writes it.

Co-Authored-By: Claude Mythos 5.2 <[email protected]>

* The too-old refusal reason has a file of its own

factory.repo_issues names the same reason; both changes add this one file
unchanged, so they merge in either order.

Co-Authored-By: Claude Mythos 5.2 <[email protected]>

* changes: entry for this pull request

Co-Authored-By: Claude Mythos 5.2 <[email protected]>

* factory.job_create: a hidden repository's 404 on /v1/issues stays not_found

Every 404 or 405 on POST /v1/issues was reported as factory_too_old, even
the current Factory's own JSON 404 "repository not found" from newJob's
visibility check. A person was told to update a Factory that was current,
and the real cause, a repository their GitHub cannot see, was lost.

A 404 there is now too old only when its body is the mux's
"factory endpoint not found" or not the Factory's JSON, the same body test
the context route already used (now one helper, missingRouteBody). A 405
stays too old: every Factory that serves /v1/issues takes POST there (GET
and POST landed together in #2561). Any other 404 keeps the not_found
mapping and the Factory's words. The "issue 404" case of
TestAppFactoryCreateOlderFactoryRefusals now sends the body an older
Factory really sends; its JSON "not found" was the broad match this fixes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs: a keyed create claims only what holds with or without #2619

The create text promised that the Factory replays the first response for
24 hours, so a same-key resend after an uncertain answer is safe. #2619,
which merges with this, changes which keyed answers are replayed: one given
before the create wrote anything runs again. PROTOCOL.md, the chat manual
and the change entry now state only what is true on either branch: within
24 hours a same-key resend never makes a second job or issue, a different
key makes a separate job, and a still-uncertain resend means read the
queue. The change entry says this must merge with #2619, which carries the
replay rule itself.

Context attachments: the docs said markdown must be non-empty, but the
engine and the Factory both refuse whitespace alone ("context markdown
must be non-empty"). The docs now say so, and the Factory's refusal table
pins the blank case the engine's table already pinned.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Factory: a create's $5 budget is kept when the rules are read after the clone

settledCap lowers a job's budget to its repository file's smaller
default_budget when the start had to take DefaultJobBudget ($5) because the
repository was not here yet. It told the two apart by amount alone, so a
person who created a job with budget 5 had it lowered to the file's default
(say $2) once the runner had cloned, although the docs promise the create's
budget is kept on an immediate or later start.

The create's stamp now marks its budget as chosen (Item.CapChosen), Run as
new keeps that mark with the amount, the worker wire carries it in the job's
desk section (cap_chosen), and settledCap leaves a chosen budget alone. A
budget a start took is still lowered as before, and jobs made without a
budget are unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* manual: the create-extras section fits its size again

The two corrections to "Creating a job with an issue, group, milestone,
budget and retry key" (the 404 that stays not_found and the keyed resend
wording) took the section past the manual's 2,000-character guide. Same
claims, fewer words.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* factory.job_create: a create key goes only to a Factory that says keyed_create

A Factory older than #2619 keeps every keyed answer for 24 hours, a
refusal or a 5xx included, so the desk's fixed per-draft key would
replay a transient failure or a fixed refusal for a day and the draft
could not launch. The engine now asks the Factory's GET /v1/features on
every create with a key and sends the key only when it lists
keyed_create; to any other Factory a keyed create answers conflict
factory_too_old and no create is sent. A create without a key is
unchanged and asks nothing.

factory.capabilities takes an optional factory: create_key is true only
in an answer naming a Factory that lists the word, and false without
one, so an app that reads capabilities once sends no key. On its own
this branch never sees the word and never sends a key. The word's
constant sits in its own file, identical on #2619.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs: a keyed create's replay has two limits

A resend with the same key was said never to make a second job or
issue. The Factory saves its answer only after the create commits, so a
Factory that stops in between runs the resend again, and it matches a
resend by the exact Authorization and Cookie headers, so one sent under
a new bearer or session cookie runs as a new create. PROTOCOL.md, the
manual and the change entry now say both.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* manual: a non-JSON 404 on /v1/issues is a Factory too old too

The create-extras section named two signs of a Factory without
/v1/issues, a 405 and a 404 saying "factory endpoint not found", and
left out a 404 whose body is not the Factory's JSON, which the engine
(missingRouteBody) and PROTOCOL.md also count as too old.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Mythos 5.2 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant