Skip to content

e2e harness: works out of the box on a fresh scaffold (proven on a live Mendix 11.8 app) - #826

Open
MendixMau wants to merge 4 commits into
masterfrom
fix/e2e-fresh-scaffold
Open

MendixMau wants to merge 4 commits into
masterfrom
fix/e2e-fresh-scaffold

Conversation

@MendixMau

@MendixMau MendixMau commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Direct lane

What changed and why (one paragraph): A fresh scaffold's tests/e2e/project.config.js is a byte copy of project.config.template.js. The template carried one sample app's CONFIGURED values: 8 menu captions, a probe page, two users, a 3-act walkthrough, 14 wireframe rows and that app's widget names in READY_SELECTORS. On a blank Mendix 11.8 app scaffolded with init-project.sh, the first runs measured nothing: monkey reached 0 of 8 targets, and full-app-walkthrough.js died at its first seed with an OQL error for an entity the app never had. This PR has three commits:

  1. The template ships empty. Every CONFIGURED block is empty and marked FILL ME, with its shape in a comment. node tests/e2e/project.config.js marks each empty block and names the instrument it stops. The walkthrough's NOT CONFIGURED message names the blocks to fill. page-audit advice prints <no exemplar: set exemplars.listPage …> instead of another app's page. install-tests.sh lists the blocks. A project's existing project.config.js is never rewritten, as before.
  2. Walkthrough act order. For an act with no handoffAssert (the first act, or one with no role change), the widget checks are now read after its leading nav steps. Before this fix, a first act that opens its page in step 1 (the template example's own shape) failed every check against the home page. After a non-empty handoffAssert the checks are read straight away, as on master, even when the act's own steps open with a nav elsewhere. The docs say checks describe the act's landing page, so the engine changed, not the docs.
  3. Monkey on an empty NAV. An empty NAV stops before the browser with SETUP NOT CONFIGURED: NAV … and exits 2. The message names --skip-monkey for an app with no menu to fuzz. exitCode() treats 0 targets as not measured. 0 of N>0 was already exit 2 (monkey.js: the sample NAV in project.config.js is never flagged, and a run that reached 0 of N targets still exits 0 with only info rows #679). Without this commit, commit 1 would open a "reached 0 of 0, exit 0" hole.

A second PR (fix/e2e-live-run-followups) carries the design-audit, app-restart and exec findings from the same field run.

Impact

Row Answer
Process No new gate or human step. A fresh project now gets an explicit NOT CONFIGURED (exit 2) from monkey and the walkthrough until it fills NAV, ROLES and WALKTHROUGH. Before, it got a 0-target run or an OQL fault. Existing projects are unchanged: their project.config.js is never synced.
Dependencies Touches project.config.template.js, config.js (comment only), monkey.js, full-app-walkthrough.js, page-audit-rules.js and bin/install-tests.sh (message text). Fixtures that copy the template as project.config.js were re-run, all green (list below). config.js TEMPLATE_NAV is kept so that configs installed earlier are still recognised. No new artifact reads. Independent of the follow-up PR, except that both add lines at the top of ## Unreleased in CHANGELOG.md (trivial conflict for whichever merges second).
Context cost none: no routed or always-loaded file touched (baseline 65,568 / 80,000 on base and head).
Behaviour NOT RUN: agents read the template's comments (the FILL ME blocks and the shape example), but no A/B was run. The text is a shape description, and its first-run effect was measured in the field run below — maintainer to accept or request A/B
Runtime n/a. The NOT CONFIGURED exits happen before a browser launches, so they are faster than the runs they replace. Time cost: none · 0 s added · can force rework: n · offset: monkey/walkthrough no longer spend a login and a run measuring nothing

Fixtures, master vs branch

Fixture master branch
tests/wave2/test-e2e-config-neutral.sh (new) 1/16 16/16
tests/wave2/test-walkthrough-act-order.sh (new; plus the review's handoff-act-opening-with-a-nav case) 3/5 5/5 (head 0ecdc7e before the review fix: 4/5)
tests/wave2/test-monkey-template-nav.sh (extended; master monkey.js with the branch template) 9/13 13/13 (0ecdc7e: 12/13, no --skip-monkey)

Existing fixtures that target the changed files or copy the template are green on the branch: test-e2e-visible-wait 8/8, test-navto-missing-item ok, test-monkey-left-app 22/22, test-monkey-oracle 14/14, monkey.selftest ok, the 11 test-journey-* fixtures that copy the template (all pass), test-named-controls 37/37, test-oql-rows 15/15, test-otel-capture-until 4/4, test-report-normalize-two-tree 19/19, test-report-seeds-668 9/9. test-verify-module-446 is 21/22 on both master and the branch (V3 playwright rows=1: an environment difference with a global Playwright, not caused by this PR).

Field evidence. A blank Mendix 11.8.0 app (mxcli run --local --db-type hsqldb, Playwright and Chromium, no Docker) was scaffolded with init-project.sh and booted for real, cold boot 195 s. Master @1a56c3a1 and this branch ran against the same running app:

Run master branch
monkey, freshly installed config exit 2: "NAV is the unedited sample" (another app's captions) exit 2: SETUP NOT CONFIGURED: NAV … is empty + how to fill it
full-app-walkthrough, freshly installed config exit 1, seed fails with OQL error: Can't find entity … (the sample app's entity), findings.json written exit 2 NOT CONFIGURED — … Fill the WALKTHROUGH and ROLES blocks, no findings.json
node project.config.js, fresh NAV 8, roles 2, walkthrough 3 acts (another app's) ← FILL ME on NAV, PAGE_WIREFRAME, roles, walkthrough
walkthrough, configured, act 1 = nav first ✗ raise:widget:dgTasks … not visible, ✗ raise:widget:btnNew, then ✓ raise:Tasks opens ✓ raise:Tasks opens, ✓ raise:widget:dgTasks, ✓ raise:widget:btnNew
monkey, configured NAV, seed 42, 30 rounds reached 3 of 3, exit 0 reached 3 of 3, exit 0 (no regression)
review fix: handoff act (handoffAssert → SideBar, checks: [dgSideBar], step 1 = nav to TopBar) 0ecdc7e: ✗ see:widget:dgSideBar … not visible (read on TopBar) 340afd7: ✓ see:widget:dgSideBar, then ✓ TopBar overview opens first

The remaining walkthrough fail on both sides is the field config's own single-module pre-flight, which is by design.

  • No client data anywhere in the diff. The leak guard passes. The template's old sample values (module names, two person-like usernames, project codenames in three comments) are removed. Similar provenance codenames remain in other e2e engine comments and are out of scope here.
  • Size cap: 10 files, but 399+/299− with fixtures and CHANGELOG excluded comes to about 430 lines. 249 of those are deletions of the sample values from the template. I am flagging this rather than splitting further: the three commits depend on each other (commit 3's hole exists only after commit 1).
  • Test tier: T2. Scoped fixtures, then a field run against a live Mendix 11.8 runtime.
  • Golden input: the template is installed by the real install-tests.sh. The walkthrough fixture uses a stub helpers.js (no browser) around the real config.js and walkthrough. Both layouts: unchanged root discovery, no path logic touched. Platforms: Node only; the one shell change is echo text. check-portability.sh did not complete inside the 300 s budget in this container: NOT RUN to completion.
  • CHANGELOG.md line per commit, credited to the field run.

Review: pending independent review

https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC


Generated by Claude Code

…ot another app's values

The installed tests/e2e/project.config.js is the template, and the template carried one sample
app's CONFIGURED values: NAV, PROBE_PAGE, two users in ROLES, a 3-act WALKTHROUGH, 14
PAGE_WIREFRAME rows, project widget names in READY_SELECTORS. On a fresh scaffold of a real
Mendix 11.8 app, monkey reached 0 of 8 targets and full-app-walkthrough died at its first seed
with an OQL error for an entity the app never had.

Every CONFIGURED block now ships empty with a FILL ME marker and its shape in a comment.
`node project.config.js` marks each empty block and names the instrument it stops. The
walkthrough's NOT CONFIGURED message names the blocks to fill; page-audit advice prints a
"no exemplar set" placeholder instead of a page; install-tests.sh lists the blocks.

Fixture: tests/wave2/test-e2e-config-neutral.sh (master 1/16, branch 16/16).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
… steps

The act loop evaluated act.checks before act.steps. The first act has no handoffAssert to
navigate for it, so a first act whose step 1 is the nav to its page read every check on the
home page and failed them, one line before its own nav passed (field run on a real Mendix
11.8 app: "x raise:widget:dgTasks not visible", then "v raise:Tasks opens").

An act's leading `nav` steps are now its arrival: they run first, then the widget checks, then
the remaining steps. A fatal arrival leaves the checks NOT REACHED. Acts that arrive through a
handoffAssert are unchanged.

Fixture: tests/wave2/test-walkthrough-act-order.sh (master 2/4, branch 4/4).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
…t measured

With the template's NAV now empty, a fresh scaffold's monkey signed in, reported "reached 0 of
0 target(s)" and exited 0: exitCode() treated zero targets as a clean run. (0 of N > 0 already
exits 2 since #679.) An empty NAV now stops before the browser with a SETUP NOT CONFIGURED line
that says how to fill it, and exitCode() returns 2 for zero targets. The old sample NAV is
still recognised (config.js TEMPLATE_NAV) for configs installed before the template went empty.

Fixture: tests/wave2/test-monkey-template-nav.sh extended (master monkey.js 9/13, branch 13/13).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
@MendixMau

Copy link
Copy Markdown
Owner Author

Review: MERGE-AFTER-FIX — the act-order change flips PASS→FAIL for a handoff act whose steps start with a nav. "Handoff acts are unchanged" does not hold for that shape.
Value: It fixes the fresh-scaffold problem as reported. The template's CONFIGURED blocks ship empty. Monkey (monkey.js:312-317) and the walkthrough (full-app-walkthrough.js:74-82) stop with NOT CONFIGURED, exit 2. page-audit-rules.js:40-42 no longer names another app's page or throws when there are no exemplars. Nothing is overbuilt.
Regression risk: full-app-walkthrough.js:555-569 applies splitArrival to every act. Take an act that arrives on page X through handoffAssert, has checks for X, and opens its steps with a nav to Y. Its checks are now read on Y.

  • Reproduced with the PR's stub fixture plus one case: base dgQueue ok, head missing.
  • Live on Mendix 11.8, handoffAssert→SideBar, checks:['dgSideBar'], step 1 nav TopBar: base ✓, head ✗.

Other configured shapes are unaffected or improve. On the field config live, raise checks go ✗✗ → ✓✓, see is identical, and monkey (seed 42) reached 3/3 with exit 0 on both. A fresh template gives exit 2 from both tools, and no findings.json.

Monkey runs under verify-module as kind info, but _report_result maps any rc 2 to FAULT (*:2), FAULTED+1). A project that deliberately has NAV = {} therefore goes from PASS ("0 of 0") to INSTRUMENT FAULT. That is honest, but it needs a named escape. Old sample-NAV configs already exit 2 (#679). project.config.js is never rewritten, and no path logic changed, so both layouts are unaffected.
Correctness:

  • The act-order fixture covers a handoff act only when its steps do not open with a nav.
  • Detached worktree: config-neutral 16/16, act-order 4/4, monkey-template-nav 13/13, and 20 existing fixtures pass. verify-module-446 is 21/22 on base and on head (V3, environmental).
  • No conflict with master. CHANGELOG conflicts with e2e harness follow-ups: design-audit stale artifact + controlbar, per-project restart log, exec '…' literals (live Mendix 11.8 run) #827 (trivial).
  • Minor: the Behaviour row should be NOT RUN with a reason, not n/a, because agents read the template's comments.
    Context cost: 65,568 → 65,568 / 80,000, measured on base and head.
    Time cost: none. Both NOT CONFIGURED exits happen before the browser starts.
    Exact fixes required:
  1. full-app-walkthrough.js act loop: when the act ran a non-empty handoffAssert, read checks right after it, as before. Use splitArrival only for acts without one. Update the template comment to match.
  2. test-walkthrough-act-order.sh: add a handoff act whose steps open with a nav elsewhere, and assert that its checks are read on the handoffAssert page.
  3. monkey.js:313: the SETUP line names --skip-monkey for an app with no menu to fuzz.

Generated by Claude Code

… checks after the assert

Independent review: the nav-first arrival applied to every act, so a handoff act that arrives
on page X through its handoffAssert and opens its steps with a nav to Y had its X checks read
on Y (reproduced live on Mendix 11.8: see:widget:dgSideBar passed on master, failed on the
previous head). After a non-empty handoffAssert the checks are read straight away again;
leading-nav arrival applies only to acts without one. The template comment says so.

Monkey's NOT CONFIGURED line names --skip-monkey for an app with no menu to fuzz.

Fixtures: test-walkthrough-act-order.sh gains that case (master 3/5, previous head 4/5,
branch 5/5); test-monkey-template-nav.sh asserts --skip-monkey (previous head 12/13, 13/13).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
@MendixMau

Copy link
Copy Markdown
Owner Author

Review: MERGE — re-review at 340afd7. All three earlier fixes are in. A handoff act now reads its checks on the handoffAssert page, even when its steps start with a nav, and this is verified live on Mendix 11.8.
Value: Fixes the fresh-scaffold defect as reported.

  • The template's CONFIGURED blocks ship empty.
  • monkey (monkey.js:312-317) and the walkthrough (full-app-walkthrough.js:74-82) stop with NOT CONFIGURED and exit 2.
  • page-audit advice (page-audit-rules.js:40-42) no longer names another app's page.
  • Nothing is overbuilt.

Regression risk:

  • Handoff acts. full-app-walkthrough.js:525-526 sets arrivedByHandoff when the role changes and handoffAssert is non-empty. That act's checks are read straight after the assert, as on master (:561). Only acts without one use nav-first.
    • Live, same app, head: a handoff act whose step 1 is a nav elsewhere reports ✓ see:widget:dgSideBar. The previous head reported ✗, and master ✓.
    • The field config's first-act checks go ✗✗ (master) → ✓✓.
    • Monkey (seed 42, live) reached 3/3, exit 0.
  • Residual change: a same-role act whose steps open with a nav now reads its checks after that nav. Before, they were read on the previous act's last page, which only matched by accident. Direction: FAIL→PASS.
  • Empty NAV under verify-module: monkey still exits 2, which verify-module records as FAULT. The SETUP line now names --skip-monkey (monkey.js:313-315), and the fixture asserts it.
  • Existing projects and layouts: project.config.js is never rewritten, configs with the old sample NAV already exit 2 (monkey.js: the sample NAV in project.config.js is never flagged, and a run that reached 0 of N targets still exits 0 with only info rows #679), and no path logic changed.

Correctness:

Context cost: 65,568 → 65,568 / 80,000. The fix commits touch no routed or always-loaded file, so the earlier measurement stands.
Time cost: none. The NOT CONFIGURED exits happen before the browser starts, and the act order adds no step.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants