Skip to content

MAINT: Make shell command scorer helpers keyword-only - #4322

Open
Aman (its-amann) wants to merge 4 commits into
microsoft:mainfrom
its-amann:maint/shell-scorer-keyword-only-4216
Open

Aman (its-amann) wants to merge 4 commits into
microsoft:mainfrom
its-amann:maint/shell-scorer-keyword-only-4216

Conversation

@its-amann

Copy link
Copy Markdown
Contributor

Description

Relates to #4216. Makes the six private pattern-building helpers in shell_command_output_scorer keyword-only and updates their callers.

  • Removes the six migrated entries from the keyword-only baseline.
  • Adds signature coverage alongside the existing scorer behavior tests.
  • Keeps the public scorer API, regex callbacks and matching behavior unchanged.

Roman Lutz (@romanlutz) this covers the module assigned to me.

Copilot AI balanced review requested due to automatic review settings November 15, 2026 18:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The review reports no active issues with the keyword-only migration of the six private scorer helpers.

No open issues 🎉

What changed in this PR

This PR advances #4216 by migrating six private pattern-building helpers in ShellCommandOutputScorer to keyword-only arguments, without changing the public scorer API or regex matching behavior.

Changes:

  • Makes all six helpers keyword-only and updates their callers.
  • Adds parameterized signature coverage alongside existing behavior tests.
  • Removes the six migrated entries from the keyword-only baseline.
File Description
tests/​unit/​score/​regex/​test_shell_command_output_scorer.py Adds signature checks for all six helpers.
pyrit/​score/​true_false/​regex/​shell_command_output_scorer.py Converts helper signatures and calls to keyword-only arguments.
build_scripts/​keyword_only_baseline.txt Removes the six migrated helper exemptions.

Review effort: Balanced

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through the diff, and the keyword-only changes look consistent:

  1. All six converted helpers (_quoted_option, _download_command, _curl_option, _curl_number_option, _curl_mode_then, _shell_options) have their call sites updated in this PR, and the six entries removed from keyword_only_baseline.txt are exactly those.
  2. One thing to confirm. _DOWNLOAD was restructured from a single f-string into + concatenation, and several rf"..." strings were split. In a module that builds very large regexes, it is easy to change the pattern by accident (a missing | or parenthesis). Could you confirm the compiled _DOWNLOAD pattern string is byte-identical to the one on main (for example by comparing the two strings once locally)? If not already covered, a test pinning the pattern, or a few representative commands, would guard against this kind of refactor.

…shell-scorer-keyword-only-4216

# Conflicts:
#	build_scripts/keyword_only_baseline.txt
The narrow-viewport test measured the baseline and attack success-rate
tiles immediately after opening the comparison, racing the layout
settling into one row (seen as a 54px y offset that passed on retry,
failing CI under --fail-on-flaky-tests). Wrap the measurement in
expect().toPass() like the other E2E specs so it waits for the settled
layout instead of asserting a single snapshot.

Co-Authored-By: Claude Fable 5 <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants