Skip to content

fix(file-bugs): stop redaction garbling ordinary English (#759) - #815

Open
MendixMau wants to merge 3 commits into
masterfrom
triage/759-file-bugs-redaction
Open

MendixMau wants to merge 3 commits into
masterfrom
triage/759-file-bugs-redaction

Conversation

@MendixMau

Copy link
Copy Markdown
Owner

Fixes #759

Note: the bodies of #759 and #760 look swapped (see the triage comment). This PR fixes #759's title, file-bugs.sh redaction garbling ordinary English. The reproduction uses the file-bugs text from #760's body.

Direct lane

What changed and why (one paragraph): Two separate causes, each reproduced on origin/master with the reported text. The garbled title came out byte for byte as reported: journey<redacted>runner: 'row' never<redacted>matches<redacted>….

(1) A project named .

  • Cause: bin/init-project.sh . took basename . as the project name and wrote \b.\b into the toolkit clone's .leakguard-deny. file-bugs.py reads that file, so every space, hyphen and one-letter word was redacted.
  • Fix in init-project.sh: resolve the folder (cd && pwd) before basename; write the name regex-escaped; dedupe on that exact line; refuse a name under 3 characters, with a NOTE.
  • Fix in file-bugs.py: drop, with a warning, any deny pattern that matches "", " " or a-b c. This repairs clones that already hold the bad line.

(2) Attribute names inside ordinary words

  • Cause: "Workaround" became WorkaAttrN (attribute Round), and the weak attribute Used matched the prose word "used".
  • Fix: a new attr_pat(), for attributes only.
    • A distinctive name matches case-insensitively, but only where an identifier segment starts.
    • A weak name matches as a whole word or segment, in its model spelling only.
    • ACT_Round_Save, $w/Used and myRound are still redacted.

Deviation from the triage's (c): applying sub=False to every attribute breaks two existing section M checks (Kundennummer2, kundennummerold), so attr_pat keeps them. Known limit: the whole word "round" in prose is still redacted for an attribute named Round; only "Workaround" is fixed.

Impact

Row Answer
Process Nothing blocks. init-project.sh no longer registers a name under 3 characters (it prints a NOTE). file-bugs.sh warns when it ignores a match-everything deny line. A report refused only because of such a line is no longer refused.
Dependencies bin/init-project.sh (leak-guard registration block and PROJECT_NAME), bin/lib/file-bugs.py, the fixture, CHANGELOG. PROJECT_NAME also feeds the PROJECT.md title: for init-project.sh . it is now the folder name instead of .. POSIX tools only.
Context cost none: no routed or always-loaded file touched
Behaviour n/a: scripts only
Runtime Negligible.

Time cost: per filing · three regex probes per deny pattern · can force rework: n · offset: removes the unreadable, always-refused reports in every clone that ran init-project.sh .

Field evidence: reported by thanarat (toolkit v2026.10.19-134, macOS). Reproduced here on master with the exact reported title and body. Master's init-project.sh . was confirmed to write \b.\b.

Fixture: tests/wave2/test-file-bugs.sh, new section S with 6 checks.

master branch
test-file-bugs.sh 219 pass, 5 fail (all in S) 224 pass, 0 fail
test-guide-reopen.sh — 10/0
test-init-autocompact.sh — 15/0

One branch run made in parallel with the master run showed one unrelated, load-dependent FAIL in section P. It passed on the sequential re-run (224/0).

Review: pending independent review


Generated by Claude Code

Two causes, both reproduced with the reported text:
1. `init-project.sh .` took basename "." as the project name and wrote
   `\b.\b` to the toolkit clone's denylist, which file-bugs.py reads: every
   space and hyphen became <redacted> and the scan refused the report.
   init-project.sh now resolves the folder first, writes the name
   regex-escaped, dedupes on that exact line, and refuses a name under 3
   characters. file-bugs.py ignores, with a warning, any deny pattern that
   matches empty text, a space or "a-b c", which repairs existing clones.
2. Attribute names matched inside ordinary words: Round turned "Workaround"
   into "WorkaAttrN", and Used redacted "used". Attributes now match only
   where an identifier segment starts; a short or common-word one only in
   its model spelling. ACT_Round_Save, $w/Used, Kundennummer2 and
   kundennummerold are still redacted.

tests/wave2/test-file-bugs.sh section S: master 219 pass / 5 fail, branch
224 pass / 0 fail.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
@MendixMau

Copy link
Copy Markdown
Owner Author

Review: MERGE-AFTER-FIX. The new "drop match-everything deny lines" logic fails open. A deny line with a trailing | (Acmecorp|Initrode|) is now dropped with only a warning, and the report posts with both names in the clear (reproduced).

Value: Fixes #759 as titled; the body is #760's, as triage noted. Both causes are real and both are fixed:

  • bin/init-project.sh:168 resolves the folder before taking its name.
  • :645-650 escapes and dedupes the name, and refuses names under 3 characters.
  • bin/lib/file-bugs.py:197-209 adds attr_pat, so "Workaround" and "used" survive. ACT_Round_Save, $o/Used and myRound are still redacted.

Existing clones that hold \b.\b are repaired for file-bugs only. check-no-client-data.sh:90 still flags every file until the user deletes that line, and the warning tells them to.

Regression risk:

  • Security. file-bugs.py:327-332 drops any pattern that matches "", " " or a-b c, with only a warning. I probed it with a project deny file of Globex\b plus Acmecorp|Initrode|:

    • master: everything is garbled and the scan refuses, so it fails closed;
    • branch: the output is 'Acmecorp and Initrode staff, <redacted> too', scan() returns [], and the report posts.

    The same happens with (Acme)? or Acme|. The check-no-client-data.sh leak guard itself is untouched, so this is no bypass of it.

  • Attribute coverage. Compared with master, two cases now leak: a distinctive attribute inside a lowercase compound (prekundennummer), and an all-caps weak attribute (USED). Hosts, emails, paths, and module and entity names are still redacted.

  • init-project.sh. The sed is GNU/BSD-safe and the code is bash 3.2-safe. The project name doesn't depend on the layout.

Correctness: test-file-bugs 224/0 on one run. Another run hit a flaky, unrelated #646 check once. test-guide-reopen 10/0 and test-init-autocompact 15/0. Section S uses the reported title verbatim and runs the real init-project.sh.

Context cost: 65568 → 65568 (measured).

Time cost: none per build row.

Exact fixes required:

  1. Fail closed in file-bugs.py add(). Auto-drop a probe-matching line only when it holds no name, i.e. no [A-Za-z0-9]{3,} run (\b.\b, \b\b). Every other probe-matching line goes to bad_deny, which refuses posting, --links and --json. Add a section S check: Acmecorp|Initrode| is refused, or Acmecorp never appears in the body.
  2. Document the two known coverage losses (prekundennummer, USED) in the attr_pat docstring and on the CHANGELOG line.

Generated by Claude Code

…s a name

Independent review of #815 (MERGE-AFTER-FIX):
- add() drops a probe-matching deny line only when it holds no name (no run of 3+
  letters/digits: `\b.\b`, `\b\b`). Any other probe-matching line (`Acmecorp|Initrode|`,
  `(Acmecorp)?`) goes to bad_deny, so posting, --links and --json are refused until it
  is fixed, as for an invalid line. Before, it was dropped with a warning and the names
  posted in clear.
- attr_pat docstring and CHANGELOG state the accepted limit: a distinctive attribute in a
  lowercase compound (prekundennummer) and an all-caps weak one (USED) are no longer redacted.
Fixture S: two new refusal checks (fail on the pre-fix head, 225/2) and a check that the
nameless line is still dropped with a warning; 227/0 on the branch.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
@MendixMau

Copy link
Copy Markdown
Owner Author

Review fixes pushed in d8137fa:

  1. Fails closed. A deny line that matches ordinary text is now auto-dropped only if it holds no run of 3 or more letters or digits (\b.\b). Any other such line, for example Acmecorp|Initrode| or (Acmecorp)?, goes to bad_deny, so posting, --links and --json are refused until it is fixed. Section S has three new checks for this.
  2. Coverage limits documented. The attr_pat docstring and the CHANGELOG line now state that prekundennummer and USED are no longer redacted.

Fixture results:

Fixture Result
test-file-bugs 227/0 (225/2 on the previous head)
guide-reopen 10/0
init-autocompact 15/0

Needs a re-review.


Generated by Claude Code

@MendixMau

Copy link
Copy Markdown
Owner Author

Review: MERGE-AFTER-FIX — the fail-closed branch works for ASCII names, but the "holds a name" test is [A-Za-z0-9]{3,} (bin/lib/file-bugs.py:332). So Öko| is dropped with only a warning and the name posts in clear (reproduced).
Value: Both earlier fixes landed: the refusal in add() (:330-340) and the documented limits in the attr_pat docstring (:203-206) and the CHANGELOG. I probed Redactor directly with Globex plus one bad line.

Result Deny lines
Refused (bad_deny) (Acmecorp)?, Acmecorp|Initrode|, Ac[m]ecorp|, (?:Acmecorp)*, Acmecorp|$, Acmecorp|(?=), Acmecorp|\s*
Dropped with a warning, no name lost A|, .*, ^, \w*, \b\b, \b.\b
Skipped empty line
Leak Öko| (no 3-run of ASCII letters/digits), and likewise any short name with a non-ASCII letter

A legitimate list (Acmecorp, \bInitrode\b, Öko) still redacts all three, own_deny is 3, and scan() is clean.
Regression risk: Unchanged from the first review. A bad line in the toolkit clone's shared .leakguard-deny now refuses filing for every project, which is the intended fail-closed behaviour.
Correctness: test-file-bugs 227/0 (run sequentially), test-guide-reopen 10/0. My earlier fix spec named the ASCII class; the gap is in that spec.
Context cost: 65,568 → 65,568 (measured).
Time cost: none per build row.
CI / merge: checks was in progress, authoring-audit passed. Merges cleanly on origin/master; the CHANGELOG line conflicts with sibling PRs, which is trivial.
Exact fixes required:

  1. At file-bugs.py:332, use re.search(r"[^\W_]{3,}", pat), which is Unicode-aware. \b.\b, \b\b, \w* and A| still have no such run.
  2. Add an S check that Öko| is refused.

Generated by Claude Code

…refused (#815 review)

[A-Za-z0-9]{3,} missed names with non-ASCII letters: Öko| was dropped with a
warning and the name posted in clear. Use [^\W_]{3,}; fixture S checks Öko|.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HXpQSjXefSNicGGqP4aiSC
@MendixMau

Copy link
Copy Markdown
Owner Author

Review: MERGE — re-review of 3379387: the name test is now Unicode-aware ([^\W_]{3,}, bin/lib/file-bugs.py:332), so Öko| is refused instead of being dropped and posted in clear.
Value: The required fix landed, with a new S check. I re-ran my bad-deny probe against Redactor, with Globex plus one bad line each:

Result Deny lines
Refused (bad_deny) Öko|, (Acmecorp)?, Acmecorp|Initrode|, Ac[m]ecorp|, (?:Acmecorp)*, Acmecorp|$, Acmecorp|(?=), Acmecorp|\s*
Dropped with a warning, no name lost A|, .*, ^, \w*, \b\b, \b.\b
Skipped empty line

No name leaks through a match-everything line any more. A legitimate list (Acmecorp, \bInitrode\b, Öko) still redacts all three, own_deny is 3, and scan() is clean.
Regression risk: Unchanged from the first review. A bad line in the shared toolkit .leakguard-deny refuses filing everywhere, which is the intended fail-closed behaviour. Existing \b.\b lines are still only warned about.
Correctness: test-file-bugs 228/0 (run sequentially, and it includes the new Öko| check). test-guide-reopen 10/0.
Context cost: 65,568 → 65,568 (measured).
Time cost: none per build row.
CI / merge: checks is still in progress; authoring-audit passed. Merges cleanly with origin/master. Stacked on sibling PRs, only CHANGELOG.md conflicts.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

file-bugs.sh: redaction replaces spaces and hyphens in ordinary English text, which makes the issue unreadable

2 participants