diff --git a/CHANGELOG.md b/CHANGELOG.md index e9e148df..d3ff974b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m `docs/RELEASE-PROCESS.md` ยง8. ## Unreleased +- new(exec, metrics): **refusal rows name the guard that refused**: every ๐Ÿšซ / โœ— refused / โŒ / โ†ฉ row `bin/exec.sh` writes to `docs/BUILD-LOG.md` (27 call sites) now carries a stable token ` ยท guard=` right after its detail text (`plan-rule`, `stop-row-scan`, `row-group-gate`, `lock-studio-pro`, `commit-first`, `uncommitted-model`, `marketplace-gate`, `mxcli-check`, `mxbuild-missing` (toolchain absent), `pre-existing-errors` (model already broken), `mxbuild-crash` (gate could not run, including a `--patch` rolled back after a crash or unverified gate), `mxbuild-gate` (errors the script introduced โ€” the real catches), `hash-verify`; `patch-exit` marks a `--patch` script that exited non-zero and is not a guard), inside the detail cell so every column reader is unchanged; `bin/metrics-trend.sh` (and so the harvested metrics inbox file) adds a `blocked by guard: mxbuild-gate 15, plan-rule 7, โ€ฆ` line, matched on the ` ยท guard=` separator only (a bare `guard=` quoted in detail text is not counted), rows from an older exec.sh counted as `unnamed`, and `patch-exit` rows on a separate `script failed` line. Why: a field metrics file had 13 of 136 execs blocked or refused and no way to tell which guard, so no guard's catch rate could be weighed against its false refusals. Same verdicts, exit codes and refusal text; no new process per exec. Fixture `tests/wave2/test-buildlog-guard-name.sh`. โ€” overengineering audit 2026-11-20 (field metrics, #804); mxbuild split and anchoring from independent review of #813 - new(doctor, bug ledger): **a guard for mxcli's silent MxBuild fallback**: when the project's Mendix version is not in `~/.mxcli/mxbuild/` (and no Studio Pro of that version exists), `mxcli docker build` / `docker check` fall back to the newest cached MxBuild and print its path but no version warning (`run --local` is exact-or-refuse); `bin/doctor.sh` now warns, naming the model version, the cached ones and `./mxcli setup mxbuild -p .mpr` (`mxtk_mxbuild_cache_gap`, warning only, never changes a gate). Ledger entry `BUG-DRAFT-mxbuild-silent-version-fallback`; upstream issue not filed. โ€” a four-app Mendix build (field run), independent review - fix(gate-check): **a fresh clone no longer FAILs Stage 3 as "stale render"**: `blueprint.html` / `process-map.html` freshness used `[ render -ot source ]`, a nanosecond mtime compare, and a checkout stamps files in directory order (`process-map.html` lands before `process-model.json`), so an unedited project read stale on every clone (reproduced: 200-wireframe checkout, 10 of 10 clones). New `bin/lib/render-fresh.sh` keeps `-ot` as the first test (a pass never flips, no git call) and, only where `-ot` would FAIL, asks git once per render (one `log`, one `status`) whether the source was committed after the render's commit; measured 0.19 s for 200 sources all `-ot`-stale on a 300-commit history (a per-source git version cost 12.9 s). Limit: a depth-1 clone cannot see a genuinely stale render. Field run: a four-app Mendix build checked out on a second machine. โ€” a four-app Mendix build (field run), independent review - fix(test-stack-up, e2e-harness-base): **the app port follows the app's configured port, and a multi-app start is no longer pinned to 8080**: with no `APP_PORTS`, `test-stack-up.sh` puts `RUN_LOCAL_APP_PORT`, then `APP_PORT` from the `.docker/.env` beside the `.mpr` (`apps//.docker/.env`; the project-root one as fallback) first, ahead of the old fixed `8080 8081 8084` tail. A fresh `run --local` start for a model under `apps/` with no handshake and no configured port used to fall back to 8080 for every app (`--app-port 8080`); it now exits 4 naming `RUN_LOCAL_APP_PORT`. `e2e-harness-base.md` states that a PAD/Docker runtime listens on 8080 *inside* the container only. Fixtures T28โ€“T32 added. Field run: a four-app Mendix build on 8081โ€“8084 whose fresh start failed with every app pinned to 8080. โ€” a four-app Mendix build (field run), independent review diff --git a/project-bin/exec.sh b/project-bin/exec.sh index c4a721a7..cb0fbc27 100755 --- a/project-bin/exec.sh +++ b/project-bin/exec.sh @@ -682,7 +682,19 @@ for _ea in "${MXTK_ROOT:-}/bin/exec-approval.sh" "$_TKR/bin/exec-approval.sh"; d break done -log_build() { # $1=status $2=detail +# $3 (refusal, rollback and blocked rows only) names the guard that stopped the write, as a +# stable token " ยท guard=" right after the detail text: plan-rule ยท stop-row-scan ยท +# row-group-gate ยท lock-studio-pro ยท commit-first ยท uncommitted-model ยท marketplace-gate ยท +# mxcli-check ยท mxbuild-missing (toolchain absent: environment, not a catch) ยท +# pre-existing-errors (model already broken, not by this script) ยท mxbuild-crash (gate could +# not run) ยท mxbuild-gate (errors this script introduced: the real catches) ยท hash-verify. +# patch-exit is NOT a guard: the --patch script itself exited non-zero, and metrics-trend.sh +# counts it apart, as "script failed". WHY (overengineering audit, +# 2026-11-20): 13 of 136 field execs were "blocked or refused" and no row said which guard, +# so nobody could weigh a guard's catches against its false refusals. Counted per guard by +# project-bin/metrics-trend.sh. Inside the detail cell, so the row keeps its 5 cells and every +# column reader is unchanged; a parameter expansion, so no process start. +log_build() { # $1=status $2=detail [$3=guard id] [ -d "${BUILD_LOG%/*}" ] || mkdir -p "${BUILD_LOG%/*}" [ -f "$BUILD_LOG" ] || cat > "$BUILD_LOG" <<'HDR' # Build log โ€” auto-appended by bin/exec.sh @@ -715,7 +727,7 @@ HDR2 # The gate cell is read from GATE_STATE rather than passed in, so no caller # can omit it โ€” a blank cell in this table reads as "fine", which is the exact # false-green the gate exists to prevent. - _detail="$2${CHANGE_CLASS:+ ยท change: $CHANGE_CLASS}${CR_ID:+ ยท cr: $CR_ID}${POS_NOTE:+ ยท $POS_NOTE}${SCAN_NOTE:+ ยท $SCAN_NOTE}${ROWS_NOTE:+ ยท $ROWS_NOTE}${MKT_NOTE:+ ยท $MKT_NOTE}${RESTATE_NOTE:+ ยท $RESTATE_NOTE}${GATE_NOTE:+ ยท $GATE_NOTE}${MXB_LOG_NOTE:+ ยท $MXB_LOG_NOTE}${HV_NOTE:+ ยท $HV_NOTE}${SCOPE_NOTE:+ ยท $SCOPE_NOTE}$_EXEC_APPROVAL_SUFFIX${STEP_TIMES:+ ยท steps: $STEP_TIMES}" + _detail="$2${3:+ ยท guard=$3}${CHANGE_CLASS:+ ยท change: $CHANGE_CLASS}${CR_ID:+ ยท cr: $CR_ID}${POS_NOTE:+ ยท $POS_NOTE}${SCAN_NOTE:+ ยท $SCAN_NOTE}${ROWS_NOTE:+ ยท $ROWS_NOTE}${MKT_NOTE:+ ยท $MKT_NOTE}${RESTATE_NOTE:+ ยท $RESTATE_NOTE}${GATE_NOTE:+ ยท $GATE_NOTE}${MXB_LOG_NOTE:+ ยท $MXB_LOG_NOTE}${HV_NOTE:+ ยท $HV_NOTE}${SCOPE_NOTE:+ ยท $SCOPE_NOTE}$_EXEC_APPROVAL_SUFFIX${STEP_TIMES:+ ยท steps: $STEP_TIMES}" if [ -n "${EPOCHSECONDS:-}" ]; then printf -v _when '%(%Y-%m-%dT%H:%M:%S%z)T' -1; else _when="$(date '+%Y-%m-%dT%H:%M:%S%z')"; fi printf '| %s | `%s` | %s | %s | %s |\n' \ "$_when" "$LOG_NAME" \ @@ -822,7 +834,7 @@ if [ -f "$_XD/_position.sh" ]; then echo "โœ— Plan rule (Interview mode: auto): $1, and the pre-build steps are not done โ€” $_GC $PROJECT_ROOT build-ready has not passed. NOTHING was written." echo " Remedy: run stages 1-4, then $_GC $PROJECT_ROOT build-ready; or $_GC $PROJECT_ROOT --adopt 5 --reason \"...\" โ€” under auto that is a โœ‹ human-pending decision, and Stage 6 FAILs until a person confirms it." echo " Already building past Stage 5? One build-ready pass clears it: its record is kept per clone (.mpr-snapshots/), so each fresh clone runs it once." - log_build "๐Ÿšซ blocked" "plan rule (auto): $1, build-ready not passed and no adoption โ€” nothing written" + log_build "๐Ÿšซ blocked" "plan rule (auto): $1, build-ready not passed and no adoption โ€” nothing written" plan-rule exit 1 } if [ "$POS_ENTRY" = greenfield-retired ]; then @@ -855,7 +867,7 @@ if [ -f "$_XD/_position.sh" ]; then echo "โœ— Plan rule: Entry mode $POS_ENTRY, and Stage 4 has not passed or been adopted โ€” no build plan, no module briefs. NOTHING was written." echo " Remedy: run stages 1-4 (conversion-runbook.md), or record why not: $_GC $PROJECT_ROOT --adopt 5 --reason \"...\" (it shows in the register)." echo " This is the one refusal this clone will ever get: every later write goes on, its BUILD-LOG row tagged \"no plan\"." - log_build "๐Ÿšซ blocked" "plan rule: Entry mode $POS_ENTRY, no build plan/briefs and Stage 4 not adopted โ€” refused once, nothing written" + log_build "๐Ÿšซ blocked" "plan rule: Entry mode $POS_ENTRY, no build plan/briefs and Stage 4 not adopted โ€” refused once, nothing written" plan-rule exit 1 fi fi @@ -905,7 +917,7 @@ if [ "$PATCH_MODE" != "1" ]; then echo "โœ— STOP-row scan: refusing exec on a blocking rule. NOTHING was locked, snapshotted or written." echo " Each line above names its learned-mdl-preflight.md row and the rewrite. Fix the script." echo " Only the user may override, with a reason: MXTK_SCAN_OVERRIDE=\"\" ./bin/exec.sh $SCRIPT" - log_build "๐Ÿšซ blocked" "STOP-row scan: $(printf '%s\n' "$_scan_out" | grep -c ' \[block\]$') blocking hit(s), $(scan_ids block) โ€” nothing written" + log_build "๐Ÿšซ blocked" "STOP-row scan: $(printf '%s\n' "$_scan_out" | grep -c ' \[block\]$') blocking hit(s), $(scan_ids block) โ€” nothing written" stop-row-scan exit 1 ;; *) echo "โš  STOP-row scan OVERRIDDEN by MXTK_SCAN_OVERRIDE: ${MXTK_SCAN_OVERRIDE}" >&2 @@ -938,7 +950,7 @@ if [ "$PATCH_MODE" != "1" ] && [ -x "$PROJECT_ROOT/bin/row-groups.sh" ]; then result_line refused no skipped 1 echo "โœ— Row-group gate: refusing exec. NOTHING was locked, snapshotted or written." echo " See the groups: bin/row-groups.sh. Override with a reason: MXTK_ROWS_OVERRIDE=\"\" ./bin/exec.sh $SCRIPT" - log_build "๐Ÿšซ blocked" "row-group gate: $(printf '%s' "$_rg_out" | head -1 | tr '|' '/' | cut -c1-200) โ€” nothing written" + log_build "๐Ÿšซ blocked" "row-group gate: $(printf '%s' "$_rg_out" | head -1 | tr '|' '/' | cut -c1-200) โ€” nothing written" row-group-gate exit 1 fi echo "โš  Row-group gate OVERRIDDEN: ${MXTK_ROWS_OVERRIDE}" >&2 @@ -1090,7 +1102,7 @@ commit_model_first() { echo "โœ— --commit-first refused: Studio Pro holds $NAME (PID $SP_PID). Its in-memory model is not" echo " what is on disk, so a commit now would record a state nobody is looking at." echo " โ†’ Save and close the project in Studio Pro, then re-run. Nothing was committed or executed." - log_build "๐Ÿšซ blocked" "--commit-first refused: Studio Pro holds the model" + log_build "๐Ÿšซ blocked" "--commit-first refused: Studio Pro holds the model" lock-studio-pro return 1 fi echo "โ†’ --commit-first: committing ONLY these model paths (nothing else is staged or committed):" @@ -1106,7 +1118,7 @@ commit_model_first() { IFS="$_IFS"; set +f result_line refused no skipped 1 echo "โœ— --commit-first: git add of the model paths failed โ€” nothing was committed or executed." - log_build "๐Ÿšซ blocked" "--commit-first: git add failed" + log_build "๐Ÿšซ blocked" "--commit-first: git add failed" commit-first return 1 fi if ! _cout="$(git commit -q -m "model: commit pending model changes before exec.sh $LOG_NAME" \ @@ -1118,7 +1130,7 @@ commit_model_first() { echo "โœ— --commit-first: git commit was refused โ€” nothing was executed. The model paths are left staged." echo " If the model-stamp hook refused it: ./bin/verify-model.sh (one mxbuild, stamps a clean model)," echo " then re-run with --commit-first." - log_build "๐Ÿšซ blocked" "--commit-first: git commit refused" + log_build "๐Ÿšซ blocked" "--commit-first: git commit refused" commit-first return 1 fi IFS="$_IFS"; set +f @@ -1143,7 +1155,7 @@ if [ -n "$MPR_DIRTY" ]; then # A refusal here is otherwise invisible in docs/BUILD-LOG.md โ€” nothing runs # after this exit, so the row has to be written before it, same shape as # every other blocked-exec row this script writes. - log_build "๐Ÿšซ blocked" "uncommitted model changes โ€” refusing exec to prevent snapshot regression" + log_build "๐Ÿšซ blocked" "uncommitted model changes โ€” refusing exec to prevent snapshot regression" uncommitted-model exit 1 fi echo " (FORCE_EXEC set โ€” proceeding despite uncommitted changes)" @@ -1313,7 +1325,7 @@ if [ "$PATCH_MODE" != "1" ] && [ -f "$SCRIPT" ] && [ -n "$_MKT_LIB" ] \ echo " or record it as deliberately not installed: bin/gate-check.sh --waive marketplace/ --reason \"...\"" echo " The PAT the install needs: skills/learned-constants-and-secrets.md ยง \"Creating a PAT for Marketplace installs\"." echo " Override with a reason: MXTK_MKT_OVERRIDE=\"\" ./bin/exec.sh $SCRIPT" - log_build "๐Ÿšซ blocked" "marketplace install gate: $(printf '%s' "$_mkt_nt" | tr '|' '/' | cut -c1-200) โ€” nothing written" + log_build "๐Ÿšซ blocked" "marketplace install gate: $(printf '%s' "$_mkt_nt" | tr '|' '/' | cut -c1-200) โ€” nothing written" marketplace-gate exit 1 fi echo "โš  Marketplace install gate OVERRIDDEN: ${MXTK_MKT_OVERRIDE}" >&2 @@ -1622,7 +1634,7 @@ elif [ "${SKIP_CHECK:-0}" != "1" ] && [ -n "$PMXCLI" ]; then echo "" echo " โœ— mxcli check failed โ€” refusing to exec. NOTHING was written to the model." echo " Fix the script, or re-run with SKIP_CHECK=1 if you know why the parser is wrong." - log_build "๐Ÿšซ blocked" "mxcli check failed pre-exec โ€” nothing written to the model${_why:+; check: $_why}" + log_build "๐Ÿšซ blocked" "mxcli check failed pre-exec โ€” nothing written to the model${_why:+; check: $_why}" mxcli-check exit 1 fi rm -f "$_co" @@ -1732,7 +1744,7 @@ if [ ! -x "$MXBUILD" ] || [ ! -x "$JAVA_EXE" ]; then echo " To write anyway, unverified: ALLOW_UNVERIFIED=1 ./bin/exec.sh $SCRIPT โ€” the commit hook will then" >&2 echo " refuse the result until ./bin/verify-model.sh has run the gate over it." >&2 fi - log_build "โœ— refused" "mxbuild gate cannot run (mxbuild: $([ -x "$MXBUILD" ] && echo ok || echo missing), java: $([ -x "$JAVA_EXE" ] && echo ok || echo missing)); nothing written" + log_build "โœ— refused" "mxbuild gate cannot run (mxbuild: $([ -x "$MXBUILD" ] && echo ok || echo missing), java: $([ -x "$JAVA_EXE" ] && echo ok || echo missing)); nothing written" mxbuild-missing exit 1 fi fi @@ -2223,9 +2235,9 @@ if [ "$PATCH_MODE" = "1" ]; then restore_snapshot if [ -x ./bin/model-stamp.sh ]; then ./bin/model-stamp.sh clear >/dev/null 2>&1 || true; fi if [ "$RESTORED" -eq 1 ]; then - log_build "โŒ patch failed" "script exit $EXEC_STATUS โ€” rolled back, gate not run" + log_build "โŒ patch failed" "script exit $EXEC_STATUS โ€” rolled back, gate not run" patch-exit else - log_build "โŒ patch failed" "script exit $EXEC_STATUS โ€” NOT rolled back (restore did not run)" + log_build "โŒ patch failed" "script exit $EXEC_STATUS โ€” NOT rolled back (restore did not run)" patch-exit fi rm -f "$HV_BASE" if [ "$RESTORED" -eq 1 ]; then result_line restored no skipped 1; else result_line failed unknown skipped 1; fi @@ -2313,9 +2325,9 @@ EOF_HV if [ -x ./bin/model-stamp.sh ]; then ./bin/model-stamp.sh clear >/dev/null 2>&1 || true; fi rm -f "$HV_BASE" "$HV_POST" if [ "$RESTORED" -eq 1 ]; then - log_build "โŒ patch refused" "$_hv_detail โ€” rolled back, gate not run" + log_build "โŒ patch refused" "$_hv_detail โ€” rolled back, gate not run" hash-verify else - log_build "โŒ patch refused" "$_hv_detail โ€” NOT rolled back (restore did not run)" + log_build "โŒ patch refused" "$_hv_detail โ€” NOT rolled back (restore did not run)" hash-verify fi if [ "$RESTORED" -eq 1 ]; then result_line restored no skipped 1; else result_line failed unknown skipped 1; fi restore_close @@ -2543,7 +2555,7 @@ EOF # output and call its error PRE-EXISTING. Say what happened instead. echo "" echo " โœ— The model still holds this script's changes and fails with [$CE_CODES]." - log_build "โŒ gate failed" "$CE_COUNT error(s): $CE_CODES โ€” NOT rolled back (restore did not run)" + log_build "โŒ gate failed" "$CE_COUNT error(s): $CE_CODES โ€” NOT rolled back (restore did not run)" mxbuild-gate else # โ”€โ”€ Whose error is it? โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ # The model is now back at its pre-exec state. If it STILL fails, the @@ -2571,7 +2583,7 @@ EOF echo " โ†’ Undone: this script added $NEW_COUNT new error(s) [$NEW_CODES]. Fix those and re-run." echo " The model is back to its $BASE_COUNT earlier error(s) [$BASE_CODES]. Those did not cause this; a script" echo " that adds nothing new is kept even while they remain." - log_build "โŒ gate failed" "$NEW_COUNT new error(s): $NEW_CODES โ€” script rolled back; $BASE_COUNT pre-existing remain" + log_build "โŒ gate failed" "$NEW_COUNT new error(s): $NEW_CODES โ€” script rolled back; $BASE_COUNT pre-existing remain" mxbuild-gate elif [ "$BASE_COUNT" != "0" ] && [ "$BASE_COUNT" != "?" ]; then echo " โš  PRE-EXISTING: the restored model already fails with $BASE_COUNT error(s) [$BASE_CODES]." echo " This script is NOT the cause. Nothing will exec until that is cleared." @@ -2581,13 +2593,13 @@ EOF echo " CE0066 security overview โ†’ update entity access" echo " CE0463 right-click a widget โ†’ Update all widgets" ;; esac - log_build "๐Ÿšซ blocked" "PRE-EXISTING $BASE_CODES โ€” not caused by this script; needs Studio Pro" + log_build "๐Ÿšซ blocked" "PRE-EXISTING $BASE_CODES โ€” not caused by this script; needs Studio Pro" pre-existing-errors elif [ "$BASE_COUNT" = "?" ]; then echo " โ†’ The restored model could not be measured, so whose error it is stays open: [$CE_CODES]." - log_build "โŒ gate failed" "$CE_COUNT error(s): $CE_CODES โ€” script rolled back; restored model not measured" + log_build "โŒ gate failed" "$CE_COUNT error(s): $CE_CODES โ€” script rolled back; restored model not measured" mxbuild-gate else echo " โ†’ The restored model builds clean, so this script introduced [$CE_CODES]." - log_build "โŒ gate failed" "$CE_COUNT error(s): $CE_CODES โ€” script rolled back" + log_build "โŒ gate failed" "$CE_COUNT error(s): $CE_CODES โ€” script rolled back" mxbuild-gate fi fi rm -f "$ERRORS_FILE" @@ -2610,15 +2622,15 @@ EOF echo " โœ— mxbuild ran on this model before the write and crashed after it โ€” rolling the write back." restore_snapshot if [ "$RESTORED" -eq 1 ]; then - log_build "โŒ gate could not run" "mxbuild exit $MXBUILD_EXIT on the written model (ran on the baseline) โ€” script rolled back${MXB_WHY:+; mxbuild: $MXB_WHY}" + log_build "โŒ gate could not run" "mxbuild exit $MXBUILD_EXIT on the written model (ran on the baseline) โ€” script rolled back${MXB_WHY:+; mxbuild: $MXB_WHY}" mxbuild-crash else echo " STOP: the model may not load. Restore it before the next exec: ./bin/restore-mpr.sh " - log_build "โŒ gate could not run" "mxbuild exit $MXBUILD_EXIT on the written model โ€” NOT rolled back (restore did not run)" + log_build "โŒ gate could not run" "mxbuild exit $MXBUILD_EXIT on the written model โ€” NOT rolled back (restore did not run)" mxbuild-crash fi else echo " โ†’ Snapshot preserved. Open in SP to verify manually before proceeding." # --patch logs its own row below, after rolling back; one row per run. - [ "$PATCH_MODE" = "1" ] || log_build "โŒ gate could not run" "mxbuild exit $MXBUILD_EXIT" + [ "$PATCH_MODE" = "1" ] || log_build "โŒ gate could not run" "mxbuild exit $MXBUILD_EXIT" mxbuild-crash fi else GATE_STATE="pass"; GATE_ERRORS=0 @@ -2648,10 +2660,19 @@ if [ "$PATCH_MODE" = "1" ] && [ "$GATE_STATE" != "pass" ] && [ "$RESTORE_TRIED" echo " โœ— --patch: gate $GATE_STATE โ€” nothing verified the patch, so it is rolled back." restore_snapshot if [ -x ./bin/model-stamp.sh ]; then ./bin/model-stamp.sh clear >/dev/null 2>&1 || true; fi + # Not a catch (review of #813): this branch only sees a gate that never ran (skipped -> + # mxbuild-missing) or ran and could not verify (fail = a crash, unverified -> mxbuild-crash). + # Never mxbuild-gate. Literal ids, so the static guard-name check can read them. if [ "$RESTORED" -eq 1 ]; then - log_build "โ†ฉ patch rolled back" "gate $GATE_STATE${MXB_WHY:+ (mxbuild: $MXB_WHY)} โ€” an unverified patch is not kept" + if [ "$GATE_STATE" = "skipped" ]; then + log_build "โ†ฉ patch rolled back" "gate $GATE_STATE โ€” an unverified patch is not kept" mxbuild-missing + else + log_build "โ†ฉ patch rolled back" "gate $GATE_STATE${MXB_WHY:+ (mxbuild: $MXB_WHY)} โ€” an unverified patch is not kept" mxbuild-crash + fi + elif [ "$GATE_STATE" = "skipped" ]; then + log_build "โŒ patch unverified" "gate $GATE_STATE โ€” NOT rolled back (restore did not run)" mxbuild-missing else - log_build "โŒ patch unverified" "gate $GATE_STATE โ€” NOT rolled back (restore did not run)" + log_build "โŒ patch unverified" "gate $GATE_STATE โ€” NOT rolled back (restore did not run)" mxbuild-crash fi if [ "$RESTORED" -eq 1 ]; then result_line restored no no 1; else result_line failed unknown no 1; fi restore_close diff --git a/project-bin/metrics-trend.sh b/project-bin/metrics-trend.sh index e6331840..ab9e1567 100755 --- a/project-bin/metrics-trend.sh +++ b/project-bin/metrics-trend.sh @@ -5,6 +5,10 @@ # through metrics-ledger.sh, plus one module-close row per module from bin/module-close.sh and # any score-only rows from `metrics-ledger.sh record`) and prints: # - outcomes: execs, applied share, gate failures, restores, lint rises +# - blocked by guard: refusal / rollback / blocked rows per guard, read from the guard= +# token exec.sh's log_build puts in docs/BUILD-LOG.md beside the ledger (the ledger itself +# has no guard column; MXTK_BUILD_LOG overrides the path). Rows written before the token +# existed count as "unnamed", so the denominator stays visible. # - speed: median and worst elapsed, first-half vs second-half median # - lint: total findings first โ†’ last (only rows where lint ran) # - score: `mxcli report` overall and per category, first โ†’ last, with the delta, from every @@ -63,9 +67,9 @@ if [ ! -f "$LEDGER" ]; then exit 2 fi -"$PY" - "$LEDGER" "$MODE" "$SINCE" <<'PYEOF' -import csv, statistics, sys -ledger, mode, since = sys.argv[1:4] +"$PY" - "$LEDGER" "$MODE" "$SINCE" "${MXTK_BUILD_LOG:-}" <<'PYEOF' +import csv, os, re, statistics, sys +ledger, mode, since, build_log = sys.argv[1:5] rows = [r for r in csv.DictReader(open(ledger), delimiter="\t")] if since: rows = [r for r in rows if r["ts"][:10] >= since] @@ -119,6 +123,31 @@ print(" applied, other %s" % pct(len(applied) - len(clean) - len(li print(" gate failed / rolled back %s" % pct(len(gate_failed))) print(" blocked / refused %s" % pct(len(blocked))) print(" snapshot restores %d" % len(restored)) + +# Per guard, from docs/BUILD-LOG.md (/.claude/loop/metrics-ledger.tsv -> /docs/). +# A stopped row is one whose result cell exec.sh writes for a refusal, rollback or blocked write. +if not build_log: + build_log = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(ledger)))), "docs", "BUILD-LOG.md") +STOP = ("blocked", "refused", "gate failed", "gate could not run", "rolled back", "patch failed", "patch unverified") +if os.path.isfile(build_log): + per, unnamed, failed = {}, 0, 0 + for line in open(build_log, encoding="utf-8", errors="replace"): + m = re.match(r"^\|\s*(\d{4}-\d\d-\d\d)", line) + if not m or (since and m.group(1) < since): continue + c = line.strip().strip("|").split("|") + if len(c) < 5 or not any(k in c[3].lower() for k in STOP): continue + # Anchored on exec.sh's own " ยท guard=" separator: detail text quoting mxcli or a CE + # message can carry a bare "guard=" and must not be counted. + g = re.search(r" ยท guard=([a-z0-9-]+)", line) + if not g: unnamed += 1 + elif g.group(1) == "patch-exit": failed += 1 # the script failed; no guard stopped it + else: per[g.group(1)] = per.get(g.group(1), 0) + 1 + named = ", ".join("%s %d" % kv for kv in sorted(per.items(), key=lambda kv: (-kv[1], kv[0]))) + if unnamed: named = (named + ", " if named else "") + "unnamed %d" % unnamed + print(" blocked by guard: %s" % (named or "none")) + if failed: print(" script failed (--patch exit, not a guard): %d" % failed) +else: + print(" blocked by guard: not counted (no docs/BUILD-LOG.md beside the ledger)") print() if el: half = len(el) // 2 diff --git a/tests/wave2/test-buildlog-guard-name.sh b/tests/wave2/test-buildlog-guard-name.sh new file mode 100644 index 00000000..3430ab0a --- /dev/null +++ b/tests/wave2/test-buildlog-guard-name.sh @@ -0,0 +1,205 @@ +#!/usr/bin/env bash +# test-buildlog-guard-name.sh - every refusal / rollback / blocked BUILD-LOG row names its guard. +# +# usage: bash tests/wave2/test-buildlog-guard-name.sh [path-to-exec.sh] +# (metrics-trend.sh and metrics-ledger.sh are read from the same directory) +# +# WHY (overengineering audit, 2026-11-20). A field metrics file had 13 of 136 execs "blocked or +# refused" and no row said which guard refused, so nobody could weigh a guard's catches against +# its false refusals. exec.sh's log_build now takes the guard id as $3 and writes " ยท guard=" +# right after the detail text; metrics-trend.sh counts those per guard. +# +# A static, over the subject: every log_build call whose status is a stop (๐Ÿšซ, โœ— refused, โŒ, +# โ†ฉ) passes a guard id from the known set; the denominator (27 calls on 2026-11-20) is +# printed, and a call with none fails. No applied/PARTIAL row passes one. +# B three real refusals, run end to end on a stub project (pre-exec `mxcli check`, uncommitted +# model, mxbuild absent): the row carries guard=; result cell, exit code and 5-cell +# shape unchanged. mxbuild-missing is its own id, apart from mxbuild-gate (review of #813: +# an absent toolchain is the environment, not a catch). +# A also checks the split: pre-existing errors and gate crashes carry their own ids, and +# patch-exit (the script failing, not a guard) stays out of "blocked by guard". +# C metrics-trend.sh over the ledger + BUILD-LOG exactly as B's runs wrote them (not +# hand-written), plus the captured verify-446 applied row and the same refusals with their +# token stripped (the shape a pre-change exec.sh writes): "blocked by guard:" names each +# guard with its count and counts the old rows as unnamed; --since filters it. +# D --patch rollback rows (review of #813): that branch only sees a gate that was skipped, +# crashed or unverified, so it never says mxbuild-gate. Static over both rows, and one end +# to end run: baseline mxbuild ok, mxbuild crashes after the patch -> guard=mxbuild-crash. +# (skipped -> mxbuild-missing is static only: an absent toolchain refuses --patch before +# any write, at the mxbuild-missing refusal B already runs.) +# +# Synthetic project, stub mxcli. Nothing real runs. +set -uo pipefail + +TOOLKIT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +SUBJECT="${1:-$TOOLKIT/project-bin/exec.sh}" +case "$SUBJECT" in /*) ;; *) SUBJECT="$PWD/$SUBJECT" ;; esac +SUBDIR="$(dirname "$SUBJECT")" +TREND="$SUBDIR/metrics-trend.sh"; [ -f "$TREND" ] || TREND="$TOOLKIT/project-bin/metrics-trend.sh" +LEDGER_SH="$SUBDIR/metrics-ledger.sh"; [ -f "$LEDGER_SH" ] || LEDGER_SH="$TOOLKIT/project-bin/metrics-ledger.sh" +CAPTURED="$TOOLKIT/tests/wave2/fixtures/verify-446/BUILD-LOG.captured.md" +WORK="$(cd "$(mktemp -d "${TMPDIR:-/tmp}/bl-guard.XXXXXX")" && pwd -P)" || exit 2 +trap 'rm -rf "$WORK"' EXIT +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); echo " ok - $1"; } +bad() { FAIL=$((FAIL+1)); echo " FAIL - $1${2:+: $2}"; } +echo "== subject: $SUBJECT" +echo "== trend: $TREND" + +# ---- A: static ----------------------------------------------------------------------------- +KNOWN="plan-rule stop-row-scan row-group-gate lock-studio-pro commit-first uncommitted-model marketplace-gate mxcli-check mxbuild-missing pre-existing-errors mxbuild-crash mxbuild-gate patch-exit hash-verify" +n_stop=0; n_named=0; n_bad_applied=0; missing="" +while IFS= read -r line; do + st="${line#*log_build \"}"; st="${st%%\"*}" + last="${line##*\" }"; case "$line" in *'"') last="" ;; esac + case "$st" in + "๐Ÿšซ"*|"โœ— refused"*|"โŒ"*|"โ†ฉ"*) + n_stop=$((n_stop+1)) + case " $KNOWN " in *" $last "*) [ -n "$last" ] && n_named=$((n_named+1)) || missing="$missing|$line" ;; + *) missing="$missing|$line" ;; esac ;; + *) [ -n "$last" ] && n_bad_applied=$((n_bad_applied+1)) ;; + esac +done < <(grep -E '^[^#]*log_build "' "$SUBJECT" | grep -v '^ *log_build() ') +echo " (stop-row log_build calls: $n_stop, naming a known guard: $n_named)" +[ "$n_stop" -ge 20 ] && ok "A: found the stop-row calls ($n_stop)" || bad "A: found the stop-row calls" "only $n_stop" +[ "$n_named" = "$n_stop" ] && ok "A: all $n_stop stop rows name a known guard" \ + || bad "A: all stop rows name a known guard" "$((n_stop - n_named)) without:$(printf '%s' "$missing" | cut -c1-300)" +[ "$n_bad_applied" = 0 ] && ok "A: no applied/PARTIAL row names a guard" || bad "A: applied/PARTIAL rows naming a guard" "$n_bad_applied" + +# ---- B: real refusals ---------------------------------------------------------------------- +P="$WORK/proj" +mkdir -p "$P/bin" "$P/mprcontents" "$P/mdlsource" +printf 'not a real model\n' > "$P/Fixture.mpr" +printf 'bson\n' > "$P/mprcontents/a.mxunit" +printf 'CREATE MODULE "Nope";\n' > "$P/mdlsource/test.mdl" +cp "$TOOLKIT/project-bin/_common.sh" "$TOOLKIT/project-bin/restore-mpr.sh" "$P/bin/" +cp "$SUBJECT" "$P/bin/exec.sh" +cp "$LEDGER_SH" "$P/bin/metrics-ledger.sh"; cp "$TREND" "$P/bin/metrics-trend.sh" +printf '#!/usr/bin/env bash\necho "Snapshot saved: none"\n' > "$P/bin/snapshot-mpr.sh" +cat > "$P/mxcli" <<'MXCLI' +#!/usr/bin/env bash +case "${1:-}" in + check) [ -n "${CHECK_OK:-}" ] && exit 0 + echo "error MDL-PAGEARG01: show page with two parameters" >&2; exit 1 ;; + exec) echo "EXEC REACHED" >> "$EXEC_N"; exit 0 ;; +esac +exit 0 +MXCLI +chmod +x "$P/bin/"*.sh "$P/mxcli" +( cd "$P" && git init -q . && printf '.mpr-snapshots/\ndocs/\n.claude/\n' > .gitignore && git add -A \ + && git -c user.email=t@t -c user.name=t commit -qm fixture ) >/dev/null 2>&1 + +run() { # -> RC, ROW + ( cd "$P" && MXBUILD_PATH="$WORK/no-mxbuild" EXEC_N="$WORK/n" SP_RESTART=0 MXTK_NO_INSTALL=1 MXTK_NO_SCORE=1 TMPDIR="$WORK" \ + ./bin/exec.sh mdlsource/test.mdl ) >"$WORK/out" 2>&1 + RC=$? + ROW="$(grep '^| ' "$P/docs/BUILD-LOG.md" 2>/dev/null | tail -1)" +} +cells() { printf '%s' "$1" | sed 's/\\|//g' | tr -cd '|' | wc -c | tr -d ' '; } + +run # 1: pre-exec mxcli check refuses +case "$ROW" in *"| ๐Ÿšซ blocked | mxcli check failed pre-exec โ€” nothing written to the model; check: "*" two parameters ยท guard=mxcli-check ยท change: "*) + ok "B: mxcli check refusal row carries guard=mxcli-check after the detail text" ;; + *) bad "B: mxcli check refusal row carries guard=mxcli-check" "$ROW" ;; esac +case "$ROW" in *"check: error MDL-PAGEARG01"*) ok "B: check reason still in the row" ;; *) bad "B: check reason still in the row" "$ROW" ;; esac +[ "$RC" = 1 ] && ok "B: exit code still 1" || bad "B: exit code still 1" "$RC" +[ "$(cells "$ROW")" = 6 ] && ok "B: row keeps 5 cells" || bad "B: row keeps 5 cells" "$ROW" + +printf 'bson changed\n' > "$P/mprcontents/a.mxunit" +run # 2: uncommitted model refuses (before check) +case "$ROW" in *"| ๐Ÿšซ blocked | uncommitted model changes โ€” refusing exec to prevent snapshot regression ยท guard=uncommitted-model"*) + ok "B: uncommitted-model row carries guard=uncommitted-model" ;; + *) bad "B: uncommitted-model row carries guard=uncommitted-model" "$ROW" ;; esac +[ "$RC" = 1 ] && ok "B: exit code still 1" || bad "B: exit code still 1" "$RC" +[ ! -f "$WORK/n" ] && ok "B: exec never reached" || bad "B: exec reached" + +( cd "$P" && git checkout -q -- mprcontents ) +export CHECK_OK=1 +run # 3: mxcli check passes, mxbuild absent -> refused before any write +unset CHECK_OK +case "$ROW" in *"| โœ— refused | mxbuild gate cannot run (mxbuild: missing"*"nothing written ยท guard=mxbuild-missing"*) + ok "B: absent mxbuild row carries guard=mxbuild-missing, not mxbuild-gate" ;; + *) bad "B: absent mxbuild row carries guard=mxbuild-missing" "$ROW" ;; esac +[ "$RC" = 1 ] && ok "B: exit code still 1" || bad "B: exit code still 1" "$RC" +[ ! -f "$WORK/n" ] && ok "B: exec never reached" || bad "B: exec reached" + +# A, the split (review of #813): each mxbuild stop names what stopped it. +gid() { grep -F "$1" "$SUBJECT" | grep 'log_build' | sed 's/.*" \([a-z0-9-]*\)$/\1/' | sort -u | tr '\n' ' '; } +[ "$(gid 'PRE-EXISTING $BASE_CODES')" = "pre-existing-errors " ] && ok "A: pre-existing errors -> pre-existing-errors" \ + || bad "A: pre-existing errors id" "$(gid 'PRE-EXISTING $BASE_CODES')" +[ "$(gid '"โŒ gate could not run"')" = "mxbuild-crash " ] && ok "A: gate could not run -> mxbuild-crash (all sites)" \ + || bad "A: gate could not run id" "$(gid '"โŒ gate could not run"')" +[ "$(gid '"โŒ gate failed"')" = "mxbuild-gate " ] && ok "A: script-introduced errors -> mxbuild-gate (all sites)" \ + || bad "A: gate failed id" "$(gid '"โŒ gate failed"')" + +# ---- C: metrics-trend per-guard line ------------------------------------------------------- +L="$P/.claude/loop/metrics-ledger.tsv" +[ "$(grep -c . "$L" 2>/dev/null)" = 4 ] && ok "C: the three refusals landed in the ledger" || bad "C: ledger rows" "$(cat "$L" 2>/dev/null)" +# The captured applied row, and the same two refusals as a pre-change exec.sh wrote them. +grep '^| 2026-' "$CAPTURED" >> "$P/docs/BUILD-LOG.md" +grep '๐Ÿšซ' "$P/docs/BUILD-LOG.md" | sed 's/ ยท guard=[a-z0-9-]*//' >> "$P/docs/BUILD-LOG.md" +# A token-less old row whose detail text quotes a bare "guard=" (mxcli or CE text) stays unnamed, +# and a --patch script failure is counted apart from the guards. Both derived from B's real row. +b="$(grep 'guard=mxcli-check' "$P/docs/BUILD-LOG.md" | head -1)" +printf '%s\n' "$b" | sed 's/ ยท guard=mxcli-check/ (mxcli: guard=row-group-gate)/' >> "$P/docs/BUILD-LOG.md" +printf '%s\n' "$b" | sed 's/| ๐Ÿšซ blocked | mxcli check failed pre-exec/| โŒ patch failed | script exit 3 โ€” rolled back, gate not run/; s/guard=mxcli-check/guard=patch-exit/' >> "$P/docs/BUILD-LOG.md" +s="$(bash "$P/bin/metrics-trend.sh" 2>&1)" +case "$s" in *"blocked by guard: mxbuild-missing 1, mxcli-check 1, uncommitted-model 1, unnamed 3"*) ok "C: per-guard counts, old rows (and a bare guard= in detail text) as unnamed" ;; + *) bad "C: per-guard counts" "$(printf '%s' "$s" | tr '\n' '~')" ;; esac +case "$s" in *"script failed (--patch exit, not a guard): 1"*) ok "C: patch-exit counted as script failed, not a guard" ;; + *) bad "C: patch-exit counted apart" "$(printf '%s' "$s" | tr '\n' '~')" ;; esac +case "$s" in *"blocked / refused 3 (100%)"*) ok "C: ledger outcome counts unchanged" ;; + *) bad "C: ledger outcome counts unchanged" "$(printf '%s' "$s" | tr '\n' '~')" ;; esac +s2="$(bash "$P/bin/metrics-trend.sh" --since 2999-01-01 2>&1)" +case "$s2" in *"blocked by guard"*) bad "C: --since 2999 printed a guard line" "$s2" ;; *) ok "C: --since filters (no rows, no line)" ;; esac +rm -f "$P/docs/BUILD-LOG.md" +s3="$(bash "$P/bin/metrics-trend.sh" 2>&1)" +case "$s3" in *"blocked by guard: not counted (no docs/BUILD-LOG.md beside the ledger)"*) ok "C: missing BUILD-LOG said, not silent" ;; + *) bad "C: missing BUILD-LOG said" "$(printf '%s' "$s3" | tr '\n' '~')" ;; esac + +# ---- D: --patch rollback rows ------------------------------------------------------------- +[ "$(gid '"โ†ฉ patch rolled back"')" = "mxbuild-crash mxbuild-missing " ] && ok "D: --patch rolled-back rows -> mxbuild-crash / mxbuild-missing, never mxbuild-gate" \ + || bad "D: --patch rolled-back ids" "$(gid '"โ†ฉ patch rolled back"')" +[ "$(gid '"โŒ patch unverified"')" = "mxbuild-crash mxbuild-missing " ] && ok "D: --patch unverified rows -> mxbuild-crash / mxbuild-missing, never mxbuild-gate" \ + || bad "D: --patch unverified ids" "$(gid '"โŒ patch unverified"')" +Q="$WORK/patchproj" +mkdir -p "$Q/bin" "$Q/mprcontents" "$WORK/jdk/bin" +printf 'not a real model\n' > "$Q/Fixture.mpr" +printf 'bson\n' > "$Q/mprcontents/a.mxunit" +cp "$TOOLKIT/project-bin/_common.sh" "$TOOLKIT/project-bin/restore-mpr.sh" "$Q/bin/" +cp "$SUBJECT" "$Q/bin/exec.sh" +cat > "$Q/bin/snapshot-mpr.sh" <<'SNAP' +#!/usr/bin/env bash +set -e +cd "$(dirname "$0")/.." +D=".mpr-snapshots/$(date +%Y%m%d-%H%M%S)-$$-$RANDOM" +mkdir -p "$D"; cp Fixture.mpr "$D/"; cp -r mprcontents "$D/" +echo "Snapshot saved: $D" +SNAP +printf '#!/usr/bin/env bash\necho "written by the patch" >> "$1"\nexit 0\n' > "$Q/patch.sh" +printf '#!/usr/bin/env bash\nexit 0\n' > "$Q/mxcli" +# mxbuild: call 1 (baseline) ok, call 2 (after the patch) crashes with no errors file. +cat > "$WORK/mxbuild" <<'MXB' +#!/usr/bin/env bash +n=$(( $(cat "$MXB_N" 2>/dev/null || echo 0) + 1 )); echo "$n" > "$MXB_N" +if [ "$n" -ge 2 ]; then echo "Unhandled exception: loader died" >&2; exit 137; fi +for a in "$@"; do case "$a" in --write-errors=*) printf '{"problems":[]}' > "${a#--write-errors=}" ;; esac; done +exit 0 +MXB +printf '#!/usr/bin/env bash\nexit 0\n' > "$WORK/jdk/bin/java" +chmod +x "$Q/bin/"*.sh "$Q/patch.sh" "$Q/mxcli" "$WORK/mxbuild" "$WORK/jdk/bin/java" +( cd "$Q" && git init -q . && printf '.mpr-snapshots/\ndocs/\n.claude/\n' > .gitignore && git add -A \ + && git -c user.email=t@t -c user.name=t commit -qm fixture ) >/dev/null 2>&1 +mkdir -p "$WORK/tq" +( cd "$Q" && TMPDIR="$WORK/tq" SP_RESTART=0 MXTK_NO_INSTALL=1 MXTK_NO_SCORE=1 MXB_N="$WORK/mxb-n" \ + MXBUILD_PATH="$WORK/mxbuild" JAVA_HOME="$WORK/jdk" ./bin/exec.sh --patch patch.sh ) >"$WORK/out.patch" 2>&1 +RC=$? +ROW="$(grep '^| ' "$Q/docs/BUILD-LOG.md" 2>/dev/null | tail -1)" +case "$ROW" in *"| โ†ฉ patch rolled back | gate fail"*"ยท guard=mxbuild-crash"*) ok "D: --patch crash row carries guard=mxbuild-crash" ;; + *) bad "D: --patch crash row carries guard=mxbuild-crash" "${ROW:-} ~ $(tail -5 "$WORK/out.patch" | tr '\n' '~')" ;; esac +grep -q 'written by the patch' "$Q/Fixture.mpr" && bad "D: crashed --patch write was kept" || ok "D: crashed --patch write rolled back" +[ "$RC" = 1 ] && ok "D: exit code 1" || bad "D: exit code 1" "$RC" + +echo "" +echo "== $PASS passed, $FAIL failed" +[ "$FAIL" -eq 0 ]