Skip to content

tests: fixtures no longer write .leakguard-deny into the toolkit clone - #829

Open
MendixMau wants to merge 2 commits into
masterfrom
tests/leakguard-deny-fixtures
Open

MendixMau wants to merge 2 commits into
masterfrom
tests/leakguard-deny-fixtures

Conversation

@MendixMau

@MendixMau MendixMau commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Direct lane

What changed and why (one paragraph): Four wave2 fixtures scaffold throwaway projects through bin/init-project.sh without setting MXTK_LEAKGUARD_DENYFILE. These are test-mode-contract.sh (:165, :173), test-ci-gates.sh (:163), test-gitignore-crlf.sh (:71, :76, :82) and test-must-haves-advisory.sh (:27, :31, :40). run-all.sh exports the variable for every fixture. A scoped run, which is how CLAUDE.md asks fixtures to be run, gets no such cover. In that case init-project.sh (~:631) appended the test project names to the clone's real .leakguard-deny: \bhand\b, \bc6\b, \binit-present\b, \binit-missing\b, \binit-fresh\b, \bfresh\b, \bold\b and \bmid\b. The pre-commit leak guard then flagged nearly every file for whoever committed next in that clone, and this hit several sessions on 2026-11-19. Each fixture now exports MXTK_LEAKGUARD_DENYFILE="$WORK/leakguard-deny" right after creating its scratch dir, which the existing trap removes. As a guard against the next fixture that forgets, tests/wave2/run-all.sh now treats the scratch MXTK_LEAKGUARD_DENYFILE it already exports as a sentinel. It empties the file before each fixture and FAILs, by name, any fixture that leaves it non-empty. Such a fixture relies on run-all's cover and would write the clone's real denylist when run alone. A fixture that exports its own path overrides the sentinel and never trips it. The full grep of tests/ found no other caller: every other fixture that runs init-project.sh already sets the variable.

Option not taken (kept minimal): init-project.sh could refuse to write the denylist into $TOOLKIT_ROOT when the project root is under $TMPDIR. That would also cover fixtures written later, but it changes a shipped script's behaviour, and a real project can live under a temp path. Left for the maintainer to decide.

Impact

Row Answer
Process none: test fixtures only; no gate, step or artifact added
Dependencies no bin/ or project-bin/ files touched. tests/wave2/run-all.sh gains the sentinel; no new tools
Context cost none: no routed or always-loaded file touched
Behaviour n/a: scripts only
Runtime ~0: one rm -f and one [ -s ] per fixture in run-all.sh

Time cost: per suite run · ~0 s (one rm and one test per fixture) · can force rework: n · offset: replaces the 250 s re-run fixture from the first revision

Proof (run-all.sh has no filter, so the full suite was not run). A harness at scratchpad/sentinel-harness.sh builds a symlink farm of the branch worktree. Its tests/wave2/ holds only the branch's run-all.sh, fixtures/ and one fixture, and the harness runs run-all.sh there with MXTK_LEAKGUARD_DENYFILE unset:

Run Result
Branch run-all.sh + master's test-mode-contract.sh FAIL test-mode-contract.sh scaffolds via init-project.sh without exporting its own MXTK_LEAKGUARD_DENYFILE (... �hand�), failing 1, rc=1
Branch run-all.sh + branch test-mode-contract.sh ok, failing 0, rc=0
Earlier, from the first revision: each of the four changed fixtures run alone in a toolkit copy with the variable unset rc=0, no root .leakguard-deny (the independent review re-ran this too)
bash bin/check-scripts.sh 145/145 shell, 30/30 Node parse
Leak guard vs origin/master no client data in 1463 tracked files

Revision 2 follows the independent review (MERGE-AFTER-FIX): it removes test-fixture-denylist-isolation.sh and adds the run-all.sh sentinel.

Field evidence: observed in this toolkit's own shared clone on 2026-11-19. After scoped fixture runs, the clone's root denylist held \bhand\b and \bc6\b, and the pre-commit leak guard blocked commits from several sessions.

  • No client data anywhere in the diff
  • Size cap: 6 files, ~30 lines net
  • Test tier reached: T1 (the scoped fixture runs above)
  • For a new/changed instrument: n/a (test fixtures only, no instrument under bin/)
  • For a new skill: n/a
  • CHANGELOG.md line appended in this PR, crediting the source
  • New bug entries: n/a

Review: pending independent review

https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC


Generated by Claude Code

test-mode-contract.sh, test-ci-gates.sh, test-gitignore-crlf.sh and
test-must-haves-advisory.sh scaffolded through init-project.sh without
MXTK_LEAKGUARD_DENYFILE. run-all.sh sets it, but a scoped run of any of
them alone appended its project names (\bhand\b, \bc6\b, \bold\b, ...)
to the clone's real .leakguard-deny, and the pre-commit leak guard then
flagged nearly every file for the next committer.

Each now exports MXTK_LEAKGUARD_DENYFILE into its own scratch dir. New
fixture test-fixture-denylist-isolation.sh runs them in a copy of the
toolkit with the variable unset and asserts the root denylist is
unchanged: 4 FAIL against master, 4 ok on this branch.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
@MendixMau

Copy link
Copy Markdown
Owner Author

Review: MERGE-AFTER-FIX — the four one-line exports are right and complete; the new 250 s fixture re-runs fixtures CI already runs, guards only those four, and a zero-cost run-all.sh sentinel gives the same signal for every fixture.

Value: Fixes the issue as reported. bin/init-project.sh:631 is the only writer of $TOOLKIT_ROOT/.leakguard-deny (no bin/ or project-bin script calls init-project). A grep of all of tests/ (wave2, run-tests.sh, behaviour, bench, retests, advisory) and pipelines/ finds exactly these four fixtures calling it without MXTK_LEAKGUARD_DENYFILE (test-mode-contract.sh:169,177, test-ci-gates.sh, test-gitignore-crlf.sh, test-must-haves-advisory.sh:31,35,44). Every other match is a comment or a grep, and the ~30 other callers export their own path. Overbuilt: test-fixture-denylist-isolation.sh hard-codes the four names (:24), so it is a regression test for them only and cannot catch the next fixture that forgets.

Regression risk: None for projects, since only tests/ changed and nothing ships via sync-project.sh. Run alone in a detached worktree with the variable unset, the four fixtures pass: mode-contract 32 s rc=0, ci-gates 81 s PASS=22 FAIL=0, gitignore-crlf 49 s rc=0, must-haves 86 s rc=0. No root .leakguard-deny appeared. Portability: tar --null -T - works with bsdtar and GNU tar, and git ls-files -co is portable. Edge case: under set -o pipefail (:17), a tracked file deleted mid-edit in a shared clone makes tar fail, and the fixture exits 2 "could not copy" (:33).

Correctness: The new fixture works. Narrowed to mode-contract, it gave PASS=1 on the branch (27 s) and caught \bhand\b with master's init-project.sh (FAIL=1). It always runs in a copy, so the clone stays clean. Its pass condition ignores the fixtures' own rc, as documented. No golden input is involved.

Context cost: none. Baseline 65568 → 65568 (budget 80000), render-routing.sh --check on base and head.

Time cost: 0 per build row (tests only). CI: +~250 s per PR on a wave2 suite of ~5 min, re-running four fixtures run-all.sh already runs. The "maintainer OK" offset is the builder's own claim, and I can't check it.

Exact fixes required:

  1. Drop tests/wave2/test-fixture-denylist-isolation.sh from the wave2 glob. Either delete it or move it to tests/retests/ as an on-demand check.
  2. Replace it with a sentinel in tests/wave2/run-all.sh. Keep the line-25 export as SENTINEL. rm -f "$SENTINEL" before each fixture. After each fixture, [ -s "$SENTINEL" ] → FAIL $name scaffolds via init-project.sh without exporting its own MXTK_LEAKGUARD_DENYFILE (run alone it writes $ROOT_DENY) and FAILED+1. A fixture that exports its own path overrides the sentinel, so the sentinel is written only by fixtures that rely on run-all's cover. Verified: master's test-mode-contract.sh writes \bhand\b to the sentinel, and the branch version leaves it empty. This adds ~0 s and covers every current and future fixture.
  3. Update the CHANGELOG line to match. No static grep fixture: a probe of non-comment init-project.sh invocations gave both false positives (test-lint-delivery.sh:153) and misses (direct "$TOOLKIT/bin/init-project.sh" calls), so it needs an allowlist and still guesses.

No conflicts with origin/master.


Generated by Claude Code

Review of #829 (MERGE-AFTER-FIX): the 250 s fixture re-ran four fixtures
and guarded only those. run-all.sh now empties its exported scratch
MXTK_LEAKGUARD_DENYFILE before each fixture and FAILs, by name, any
fixture that leaves it non-empty -- i.e. one that relies on run-all's
cover and would write the clone's real .leakguard-deny when run alone.
Covers every current and future fixture at ~0 s.

Proved with a one-fixture symlink-farm harness of run-all.sh: master's
test-mode-contract.sh trips it (\bhand\b), the branch's does not.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
@MendixMau

Copy link
Copy Markdown
Owner Author

Review: MERGE — re-review of 47ebe4b: the 250 s fixture is gone, and the run-all.sh sentinel (tests/wave2/run-all.sh:30,80-87) catches master's leaking test-mode-contract.sh at ~0 s while passing fixed and own-export fixtures.

Value: Fixes the issue as reported. bin/init-project.sh:631 is the only writer of the clone's .leakguard-deny. The four fixtures that called it without MXTK_LEAKGUARD_DENYFILE are the only offenders in tests/ and pipelines/; every other caller exports its own path. Each of the four now exports a scratch path. The sentinel covers every current and future wave2 fixture, which the deleted copy-and-rerun fixture could not.

Regression risk: None for projects: only tests/ changed and nothing ships via sync-project.sh. Harness run (branch run-all.sh, a symlink farm holding one fixture):

  • master's test-mode-contract.sh: sentinel FAIL naming \bhand\b, failing 1, rc=1.
  • branch's test-mode-contract.sh: ok, rc=0.
  • test-surface-column.sh, which exports its own path: 13 ok, rc=0 (no false FAIL).

No farm or worktree root denylist was created. Fixtures that export their own path use mxtk-fixture-denylist.$$ or $WORK/..., so they cannot collide with run-all's mxtk-runall-denylist.$$. Exit codes are unchanged: a sentinel hit adds to FAILED, and the existing [ "$FAILED" -eq 0 ] turns it into rc=1. Portable: only rm -f, [ -s ], grep, tr and printf.

Correctness: Correct. Non-blocking points:

  1. On a sentinel hit with fixture rc=0, the fixture prints a FAIL line followed by an "ok" line. That reads as contradictory, though the total and rc are right.
  2. A fixture with rc≠0 that also leaks counts twice in failing N.
  3. A fixture that inherits the variable and deletes the file in its own trap would hide a leak. No current fixture does that; all that trap-rm it export their own first.

Context cost: none. Baseline 65568 → 65568 words (budget 80000), from bin/render-routing.sh --check on base and head in the first pass. No routed file was touched since.

Time cost: 0 per build row. In CI, one rm and one test -s per fixture, about 0 s, replacing the earlier +250 s.

No conflicts with origin/master.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants