Skip to content

e2e harness follow-ups: design-audit stale artifact + controlbar, per-project restart log, exec '…' literals (live Mendix 11.8 run) - #827

Open
MendixMau wants to merge 3 commits into
masterfrom
fix/e2e-live-run-followups
Open

MendixMau wants to merge 3 commits into
masterfrom
fix/e2e-live-run-followups

Conversation

@MendixMau

Copy link
Copy Markdown
Owner

Direct lane

What changed and why (one paragraph): These are the follow-up findings from the same field run as the "e2e harness: works out of the box on a fresh scaffold" PR: a real Mendix 11.8 app under mxcli run --local. This PR has three commits:

  1. design-audit leaves no stale artifact. skills/e2e-harness-base.md named only Playwright. Followed literally in a container with a global Playwright, npm i -D playwright shadowed the global copy with a version whose browser was not downloaded. design-audit then crashed at chromium.launch (exit 1) and wrote nothing, so the previous run's design-audit.json stayed in place and read as current. Now a browser that does not launch gives a FAULT row per rung-7 check, in both the app sweep and the positive controls; the artifact is rewritten and the exit code is 2. Any other crash removes the previous artifact and says so, so report-normalize reads fault. The doc now says to probe for a working Playwright before installing one, and to npm i -D axe-core.
  2. design-audit's #891 guard names the unnamed control bar. The guard matched only controlbar <name>. mxcli writes controlbar { with no name, so the page FAULTed correctly but the message said "controlbars: none". The verdict is unchanged.
  3. app-restart and exec.sh. The app-restart.sh log moves from ${TMPDIR:-/tmp}/<mpr name>-run-local.log to <project>/.claude/loop/app-restart/run-local.log. The old path was shared by every model with the same name. The new file is self-gitignored and created only when written; LOGDIR still overrides. In exec.sh's watch-compat and change-class lines, a blanked string literal now reads '…' instead of ''. Before, create or modify demo user '' ( Password: '', … read as an empty user name. Literals are still never printed, so passwords stay masked.

Impact

Row Answer
Process none. No gate or step added. design-audit's exit code on a browser that does not launch moves from 1 (crash) to 2 (instrument fault), which verify-module already reads as "not measured".
Dependencies project-bin/exec.sh: display-only shown() added to the shared awk lexer; the '' used by the watch classifier's regex is untouched. project-bin/app-restart.sh: the log path changes; test-stack-up.sh still writes its own ${LOGDIR:-$TMPDIR}/<project>-run-local.log (not changed here). design-audit.js, skills/e2e-harness-base.md (on-demand). No new artifact reads. Independent of the fresh-scaffold PR, except that both add lines at the top of ## Unreleased in CHANGELOG.md.
Context cost none: e2e-harness-base.md is on-demand. render-routing.sh --check gives baseline 65,568 / 80,000 on both base and head.
Behaviour NOT RUN: e2e-harness-base.md is on-demand, and the change is a two-bullet install-step correction measured in the field run below — maintainer to accept or request A/B
Runtime n/a. Time cost: none · 0 s · can force rework: n · offset: maintainer OK

Fixtures, master vs branch

Fixture master branch
tests/wave2/test-design-audit-stale-artifact.sh (new; Playwright's launch message verbatim from the field log) 1/5 6/6
tests/wave2/test-design-audit-controlbar.sh (new; captured describe --format elk of the field page) 4/5 5/5
tests/wave2/test-app-restart.sh (extended) 120/123 123/123
tests/wave2/test-exec-lock-watch.sh (extended) 126/128 128/128

Existing fixtures for the changed files are green on the branch: test-design-audit-a11y-evidence 9/9, test-design-audit-css-corpus 7/7, test-design-audit-nav-map 26/26, test-exec-change-class 21/21, test-module-close-cadence 67/67.

Field evidence. A blank Mendix 11.8.0 app was scaffolded with init-project.sh and booted with mxcli run --local --db-type hsqldb (no Docker). Master @1a56c3a1 and this branch ran against the same running app:

Run master branch
design-audit --page <overview page>, then the same with PLAYWRIGHT_BROWSERS_PATH=/nonexistent design-audit crashed: browserType.launch: Executable doesn't exist …, exit 1; artifact md5 unchanged (6fdc6ce6a2b5 → 6fdc6ce6a2b5) exit 2, INSTRUMENT FAULT (12); artifact rewritten (778ea70cb93a → 116d2107a3b5); a11y/overflow/structure rows are fault — browser did not launch: …
design-audit on a DataGrid2 page with New/Edit buttons controlbars: none controlbars: (unnamed, mdlSource line 4)
bin/app-restart.sh (LOGDIR unset) ✓ started (log: $TMPDIR/ProbeApp-run-local.log), 59 s ✓ started (log: <project>/.claude/loop/app-restart/run-local.log), 75 s; nothing written to $TMPDIR
exec.sh --watch-compat on the app's demo-user script create or modify demo user '' ( Password: '', Enti... create or modify demo user '…' ( Password: '…', Enti...; the password appears in neither output
  • No client data anywhere in the diff (leak guard passes)
  • Size cap: 10 files; 95 changed lines excluding fixtures, the capture and CHANGELOG.md
  • Test tier: T2. Scoped fixtures, then a field run against a live Mendix 11.8 runtime.
  • Golden input captured (elk describe, Playwright's launch error). Both layouts: paths go through the existing REC_DIR and nearRoot roots. Platforms: plain awk (gsub with a string pattern; checked on mawk), mkdir -p only. check-portability.sh did not complete inside the 300 s budget in this container: NOT RUN to completion.
  • CHANGELOG.md line per commit, credited to the field run.

Review: pending independent review


Generated by Claude Code

…s axe-core

e2e-harness-base.md named only Playwright. Followed literally in a container with a global
Playwright, `npm i -D playwright` shadowed it with a version whose browser was not downloaded;
design-audit then crashed at chromium.launch (exit 1) and wrote nothing, so the previous run's
artifacts/design-audit.json stayed in place and read as current.

A browser that does not launch is now a FAULT row per rung-7 check (the artifact is rewritten,
exit 2), in the app sweep and in the positive controls. Any other crash removes the previous
run's artifact and says so, so report-normalize reads design-audit as `fault`. The doc says to
probe for a working Playwright before installing one, and to install axe-core.

Fixture: tests/wave2/test-design-audit-stale-artifact.sh (master 1/5, branch 6/6), with
Playwright's launch message captured verbatim from the field log.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
readCompleteness() matched only `controlbar <name>`, but mxcli writes a data grid's control bar
unnamed. On a real DataGrid2 page with New/Edit buttons the guard FAULTed the page correctly
(the buttons are unread) while saying "controlbars: none". Unnamed control bars are now listed
as "(unnamed, mdlSource line N)"; the verdict is unchanged.

Fixture: tests/wave2/test-design-audit-controlbar.sh over a captured `describe --format elk`
of the field app's Tasks.Task_Overview (Mendix 11.8.0) — master 4/5, branch 5/5.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
…literal

app-restart.sh logged to ${TMPDIR:-/tmp}/<mpr name>-run-local.log, shared by every project
whose model has the same name (several field apps were all ProbeApp.mpr). The log is now
<project>/.claude/loop/app-restart/run-local.log, beside start.cmd, self-gitignored and created
only when something is written; LOGDIR still overrides.

exec.sh's watch-compat and change-class lines print a statement head with string literals
blanked to '' ("create or modify demo user '' ( Password: '', ..."), which reads as an empty
name. A blanked literal now shows as '…'; the literal itself is still never printed, so a demo
user's password stays masked.

Fixtures: test-app-restart.sh (master 120/123, branch 123/123), test-exec-lock-watch.sh
(master 126/128, branch 128/128).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
@MendixMau

Copy link
Copy Markdown
Owner Author

Review: MERGE — no blocking findings. A stale design-audit artifact can no longer read as current, and '…' prints no literal (probed with 4 shapes, 0 leaks).
Value: It fixes all three findings as reported, with nothing overbuilt:

  • A browser that does not launch now gives FAULT rows, exit 2, and a rewritten artifact (design-audit.js:772-796, 1051-1058).
  • A crash removes the stale artifact (:1294-1306).
  • The guard names the unnamed controlbar { (:286).
  • The restart log is per project (app-restart.sh:307-309), and exec prints '…' (exec.sh:247).

Regression risk:

  • Browser fails to launch: verify-module used to record INFO (rc 1). It now records FAULT, counted in FAULTED (rc 2). That is honest, and journeys and monkey fail in that environment anyway.
  • Control bar regex: any page with a control bar is now partial. With mxcli 11.8 its buttons are already unread, so the verdict does not change. It changes only for an empty controlbar {}, or if a future mxcli carries the bar's children, which is the intent stated at :270-273.
  • Old log path: nothing reads it. $LOG is read only by app-restart's own _logsz (:214) and messages. test-stack-up.sh:185 keeps its own log, and fixtures/project-facts/bin/app-restart.sh is a frozen copy. .gitignore matches mark_owned (:221). The path is built from PROJECT_ROOT, so both layouts work.
  • Minor: the log accumulates in the project with no rotation.
  • awk: string-pattern gsub works on mawk and BWK.

Correctness:

  • Leaks: shown() touches only '' pairs in text that is already blanked. I probed a doubled-quote password, a multi-line password, a caption and a microflow literal: 0 leaks. Cosmetic only: 'a''b' → '…''…'.
  • Crash handler: it could delete a fresh artifact if write()'s console tail threw. Unlikely.
  • Fixtures, detached worktree: stale-artifact 6/6, controlbar 5/5, app-restart 123/123, exec-lock-watch 128/128, plus 8 existing fixtures (a11y, css-corpus, nav-map, app-readiness, exec-change-class, close-cadence, mdl-scan, normalize-two-tree), all green.
  • Golden input: captured.
  • Conflicts: none with master. CHANGELOG conflicts with e2e harness: works out of the box on a fresh scaffold (proven on a live Mendix 11.8 app) #826 (trivial).

Live, Mendix 11.8, base vs head:

  • controlbars: none → (unnamed, mdlSource line 4), exit 2 on both.
  • With a missing browser: base exit 1 and the artifact md5 unchanged. Head exit 2 and the artifact rewritten.
  • app-restart logs to <project>/.claude/loop/app-restart/run-local.log, serves after 56 s, and writes nothing to $TMPDIR.
  • On the demo-user script, watch-compat prints '…'.

Context cost: 65,568 → 65,568 / 80,000, measured. e2e-harness-base.md is loaded on demand.
Time cost: none per build row.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants