Skip to content

new(exec, metrics): BUILD-LOG refusal rows name the guard that refused - #813

Open
MendixMau wants to merge 3 commits into
masterfrom
triage/buildlog-guard-name
Open

MendixMau wants to merge 3 commits into
masterfrom
triage/buildlog-guard-name

Conversation

@MendixMau

Copy link
Copy Markdown
Owner

Direct lane

There is no issue. This comes from the over-engineering audit of 2026-11-20, which found that in the one field metrics file we have (#804), 13 of 136 execs were "blocked or refused", and no row said which guard stopped them. Until rows name the guard, nobody can measure a guard's catch rate against its false-refusal rate, and so nobody can decide which guards to keep.

What changed and why (one paragraph): project-bin/exec.sh's log_build takes an optional third argument, the guard id.

  • The token. Every refusal, rollback and blocked row now carries · guard=<id> right after its detail text. That covers all 25 call sites with status 🚫 / ✗ refused / ❌ / ↩.
  • The ids. plan-rule, stop-row-scan, row-group-gate, lock-studio-pro, commit-first, uncommitted-model, marketplace-gate, mxcli-check, mxbuild-gate, patch-exit, hash-verify.
  • Row shape. The token sits inside the detail cell, so a row still has 5 cells and every column reader is unchanged.
  • Metrics line. project-bin/metrics-trend.sh adds one line under "outcomes": blocked by guard: mxbuild-gate 15, plan-rule 7, …, unnamed N. Rows from an older exec.sh count as unnamed. bin/harvest-learnings.sh already pastes this summary into the inbox metrics file, so the next field harvest reports per-guard counts.
  • Unchanged: verdicts, exit codes, on-screen refusal text, and the metrics-ledger TSV schema.

Readers checked. The token appears only in cell 6, before steps:, and no guard id contains applied, PARTIAL, UNVERIFIED, dirty model, cr: CR- or steps:. Readers that parse rows:

ready.sh and token-burn.sh don't read the BUILD-LOG.

Impact

Row Answer
Process none. Nothing new blocks.
Dependencies project-bin/exec.sh: one parameter expansion in log_build plus a trailing argument on 25 calls. project-bin/metrics-trend.sh now also reads docs/BUILD-LOG.md (written by exec.sh's log_build on every exec). Projects pick both up via sync-project.sh --upgrade-bin. Any open PR that adds a log_build "🚫 …" call must pass a guard id, or the new static check fails.
Context cost none: no routed or always-loaded file touched
Behaviour n/a: scripts only
Runtime ~0. ${3:+ · guard=$3} starts no process. test-exec-process-budget.sh counts 132 starts on both master and the branch.

Time cost: none per exec · 0 extra process starts (132 → 132) · can force rework: n · offset: n/a

Field evidence: the audit's count from #804. The row shape was checked against the captured tests/wave2/fixtures/verify-446/BUILD-LOG.captured.md, which the new fixture appends verbatim. The next harvest is the first field run of the new token.

Fixture: tests/wave2/test-buildlog-guard-name.sh [exec.sh]

  • A (static): all 25 stop-row calls name a known guard, and no applied or PARTIAL row names one.
  • B: two real refusals run end to end on a stub project (pre-exec mxcli check, uncommitted model). Each row carries guard=<id>; the result cell, exit 1 and the 5-cell shape are unchanged.
  • C: metrics-trend over B's ledger and BUILD-LOG, plus the captured applied row, gives blocked by guard: mxcli-check 1, uncommitted-model 1, unnamed 2. --since filters, and a missing BUILD-LOG is reported rather than skipped.
result
master 10 passed, 5 failed
branch 15 passed, 0 failed

Existing fixtures that target the changed files, all green on the branch:

  • exec-check-refusal 10/0
  • exec-crash-rollback 13/0
  • exec-gate-log 24/0
  • exec-gate-retry 52/0
  • exec-hash-verify 49/0
  • exec-result-line 72/0
  • exec-step-times 7/0
  • exec-gate-kind 32/0
  • exec-lint-autorestore 17/0
  • exec-lint-diff 9/0
  • exec-subjects 6/0
  • exec-change-class 21/0
  • exec-cr-brief-warn 19/0
  • exec-lock-watch 125/0
  • exec-process-budget 4/0
  • bug07-08 48/0
  • describe-baseline 33/0
  • entry-mode-plan-rule 47/0
  • mode-contract 31/0
  • restatement-guard 58/0
  • mdl-scan 53/0
  • metrics-ledger 62/0
  • harvest-learnings 27/0

check-scripts, check-portability, the leak guard and check-pr-discipline are clean.

  • No client data anywhere in the diff
  • Size cap: 2 files, 93 lines changed (excluding the fixture and CHANGELOG)
  • Test tier: T1 (scoped fixtures plus end-to-end refusals on a stub project)
  • Changed instrument: row shape checked against the captured BUILD-LOG. The token doesn't depend on layout. Bash parameter expansion only.
  • CHANGELOG.md line appended, crediting the audit and the field metrics (contrib/inbox: learnings from a requirements-driven full build #804)

Not done, by design: the Studio Pro lock and exec-lock refusals write no BUILD-LOG row today. Adding rows would change behaviour, so they stay uncounted.

Review: pending independent review


Generated by Claude Code

Every refusal / rollback / blocked row exec.sh writes (25 log_build call sites)
now carries " · guard=<id>" right after its detail text, inside the detail
cell: column readers (status, build-log-read, script-owners, done-drift-check,
row-groups, doctor steps:, progress-report, obligation-check, ci-gates,
metrics-ledger classify) are unchanged. metrics-trend.sh adds a
"blocked by guard:" count line from docs/BUILD-LOG.md, rows from an older
exec.sh counted as unnamed, so the harvested metrics file reports it.
Same verdicts, exit codes and refusal text; no new process per exec
(process budget 132 starts on master and branch).

Fixture tests/wave2/test-buildlog-guard-name.sh: master 10 passed 5 failed,
branch 15 passed 0 failed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
@MendixMau

Copy link
Copy Markdown
Owner Author

Review: MERGE-AFTER-FIX. mxbuild-gate covers 11 different stops: toolchain missing, pre-existing errors, errors the script introduced, and mxbuild crashes. A per-guard count therefore cannot separate catches from false refusals, which is this PR's stated purpose.

Value: The mechanism is sound and minimal:

  • ${3:+ · guard=$3} at project-bin/exec.sh:726;
  • 25 stop calls tagged;
  • one counting block in project-bin/metrics-trend.sh:127-146.

The largest bucket misses the goal. mxbuild-gate tags these stops:

Stop exec.sh lines What it is
✗ refused … mxbuild: missing :1743 environment problem
🚫 blocked PRE-EXISTING :2592 the model was already broken
❌ gate failed … rolled back :2582/2595/2598 true catches
gate could not run :2621-2629 mxbuild crash

Also, patch-exit (:2234/2236) is the script failing, not a guard. Ids become history once written, so the split must happen before merge.

Regression risk: Low. I checked every BUILD-LOG reader myself, not from the PR body:

  • status.sh:101
  • script-owners.sh:177
  • progress-report.py:1346-1362
  • done-drift-check.sh:48
  • doctor.sh:1107+ (the token sits before steps:)
  • build-log-read.sh
  • gate-check.sh:3204
  • the external parseExecRows

No verdict, exit code or on-screen text changed. The code is bash 3.2-safe and works in both layouts.

Correctness: Fixtures:

  • new fixture 15/0
  • exec-process-budget 4/0 (132 → 132 starts)
  • exec-gate-log 24/0
  • exec-check-refusal 10/0
  • exec-crash-rollback 13/0
  • exec-hash-verify 49/0
  • metrics-ledger 62/0

The input is real exec runs plus captured rows. Minor: metrics-trend.sh:138 matches a bare guard= anywhere on the line, so detail text that contains guard= would be counted.

Context cost: 65568 → 65568 (measured).

Time cost: 0 process starts per exec.

Exact fixes required:

  1. Split mxbuild-gate:

    • mxbuild-missing (:1743)
    • pre-existing-errors (:2592)
    • mxbuild-crash (:2621/2624/2629)
    • keep mxbuild-gate for rollbacks the script itself caused (:2554/2582/2595/2598/2660/2662)

    Update KNOWN in the fixture and the id list in the exec.sh header and CHANGELOG.

  2. Either mark patch-exit as "not a guard" in the header, or exclude it from "blocked by guard".

  3. Match · guard=([a-z0-9-]+) in metrics-trend.sh:138.


Generated by Claude Code

…ch-exit is not a guard

Independent review of #813 (MERGE-AFTER-FIX):
- mxbuild-gate covered four different stops. Now: mxbuild-missing (toolchain absent),
  pre-existing-errors (model already broken), mxbuild-crash (gate could not run), and
  mxbuild-gate only for errors the script introduced (the real catches).
- patch-exit is called out as not a guard; metrics-trend.sh counts it on a separate
  "script failed" line instead of "blocked by guard".
- metrics-trend.sh matches " · guard=<id>" only, so a bare guard= quoted in detail text
  is not counted.
Fixture: real mxbuild-missing refusal run, per-site id checks for the split, a derived
row with a bare guard= in detail text, and a patch-exit row (5 of 22 fail on the
pre-fix head, 22/22 pass).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
@MendixMau

Copy link
Copy Markdown
Owner Author

Review fixes pushed in f41b78c:

  1. mxbuild-gate is split into four ids:

    • mxbuild-missing (:1743)
    • pre-existing-errors (:2592)
    • mxbuild-crash (:2621/2624/2629)
    • mxbuild-gate (errors the script itself introduced)

    The header now explains each id.

  2. patch-exit is marked "not a guard". metrics-trend reports it on its own line: script failed (--patch exit, not a guard): N.

  3. The guard match is anchored on · guard=.

Fixture results:

  • On this commit: 22/0. On the previous head: 17/5; it counted a bare guard= in detail text as a block.
  • exec-process-budget: 4/0 (132 starts).
  • exec-gate-log: 24/0.
  • exec-crash-rollback: 13/0.
  • metrics-ledger: 62/0.

Needs a re-review.


Generated by Claude Code

@MendixMau

Copy link
Copy Markdown
Owner Author

Review: MERGE-AFTER-FIX — the split landed at the right lines, but the two --patch rows at project-bin/exec.sh:2664,2666 are still tagged mxbuild-gate ("the real catches"). Those rows are only reached when the gate was skipped, crashed or unverified. My earlier fix list wrongly put them in the catches group.
Value: Fixes 1–3 landed:

  • mxbuild-missing at :1747.
  • pre-existing-errors at :2596.
  • mxbuild-crash at :2625,2628,2633.
  • mxbuild-gate kept for script-introduced CE at :2558,2586,2599,2602.
  • The header at :688-692 explains each id. patch-exit is counted apart as "script failed" (metrics-trend.sh:142-148).
  • The match is anchored on · guard= (:141).

The remaining gap is the --patch branch (:2657). It runs only when GATE_STATE != pass and RESTORE_TRIED=0. A failure with CE codes restores first (RESTORE_TRIED=1), so this branch only sees:

  • skipped (:2647);
  • unverified (:2455,2468);
  • fail from a crash (:2609; :2633 defers the --patch crash row here).

A --patch mxbuild crash is therefore counted as a catch. That mixes the buckets this PR exists to separate.
Regression risk: Low. Rows keep 5 cells, there is no exit-code or verdict change, and process starts are 132 (= master).
Correctness: Fixtures on head:

Fixture Passed / failed
test-buildlog-guard-name 22/0
test-exec-process-budget 4/0, starts=132
test-exec-gate-log 24/0
test-metrics-ledger 62/0

Context cost: 65,568 → 65,568 (measured).
Time cost: 0 process starts per exec.
CI / merge: checks was in progress, authoring-audit passed. Merges cleanly on origin/master; the CHANGELOG line conflicts with sibling PRs, which is trivial.
Exact fixes required:

  1. At exec.sh:2664,2666, tag by GATE_STATE: skipped → mxbuild-missing; fail or unverified → mxbuild-crash. Never mxbuild-gate.
  2. Add one fixture check that a --patch crash row carries guard=mxbuild-crash.

Generated by Claude Code

…, not mxbuild-gate (#813 review)

The --patch rollback branch only runs when the gate was skipped, crashed or
unverified, so its rows were miscounted as catches. Tag by GATE_STATE;
fixture D checks both rows statically and runs a --patch crash end to end.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
@MendixMau

Copy link
Copy Markdown
Owner Author

Review: MERGE — re-review of dc784df: the --patch rollback rows (project-bin/exec.sh:2663-2676) now tag by GATE_STATE: skipped → mxbuild-missing, and fail or unverified → mxbuild-crash. mxbuild-gate is left only on the script-introduced CE rows (:2558,2586,2599,2602).
Value: Every required fix is in.

  • Earlier round: the split, patch-exit counted as "script failed", and the anchored · guard= match.
  • This round: the remaining mis-tag. I checked the claim that "skipped is static-only". It holds. With mxbuild or java missing, :1729-1750 always refuses in --patch mode: the ALLOW_UNVERIFIED bypass at :1730 excludes PATCH_MODE=1, and that branch exits 1 with mxbuild-missing. So the skipped arm at :2663/:2670 can only be reached if the toolchain vanishes between :1729 and :2393. It is defensive, and right to keep.
  • Minor: unverified (errors file unparseable, or 0 problems with a non-zero exit) shares mxbuild-crash. The header calls it "gate could not run", which is accurate enough.
    Regression risk: Low. Rows keep 5 cells, and exit codes and verdicts are unchanged. Process starts are 132, the same as master.
    Correctness: Section D checks both arms statically and runs a real --patch crash end to end, asserting the row reads guard=mxbuild-crash, the rollback, and exit 1. Results in a detached worktree:
Fixture Passed / failed
test-buildlog-guard-name 27/0
test-exec-process-budget 4/0
test-exec-gate-log 24/0
test-exec-hash-verify 49/0
test-metrics-ledger 62/0

Context cost: 65,568 → 65,568 (measured).
Time cost: 0 process starts per exec.
CI / merge: checks is still in progress; authoring-audit passed. Merges cleanly with origin/master. Stacked on sibling PRs, only CHANGELOG.md conflicts.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants