Repository navigation
FEAT: Add a pyrit-technique skill for comparing attack techniques - #4420
Open
varunj-msft wants to merge 5 commits into
Open
varunj-msft wants to merge 5 commits into
varunj-msft wants to merge 5 commits into
Conversation
Serve a curated set of backend REST operations as MCP tools at /api/mcp, so AI agents such as GitHub Copilot can list targets, scenarios, techniques, datasets, and scorers, start and follow scenario runs, and read attack results, conversations, and analytics. - Generate each tool from the OpenAPI document and call it in-process with the caller's credentials, so it gets the same validation, errors, authentication, and runtime admission as any REST request. - Accept only POST: responses are JSON, and a GET event stream would stay open and count as active work. - Check browser origins against PYRIT_CORS_ORIGINS as CORS does, and let browser clients send the MCP protocol version header. - Keep component identifier hashes, which pinned references such as objective_scorer need, and drop eval_hash. - Keep empty strings, which can be real values, and drop other empty values. - Return named response_fields in full, falling back to previews when they do not fit, and report a named field as unknown only when the response cannot contain it. - Keep oversized responses valid JSON by cutting the largest list and dropping continuation fields that would skip the cut entries. - Reject path parameters that contain "/", which would reach other routes.
Add an agent skill that teaches an AI agent to find which attack technique best achieves an objective against a registered target through the MCP tools: choose an objective scorer and up to three techniques, run one technique.execute scenario per technique, and compare the outcomes and the scorer's reasons. - Ship it in the package under pyrit/backend/skills, where `copilot skill add` and `gh skill install` find it. - Test that its frontmatter follows the Agent Skills format and that every tool it names is one the MCP endpoint offers. - Document how to add and use it.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The catalog lookup guidance omits pagination metadata and cursor handling, which can hide registered targets or objective-aware scorers beyond the first 50 entries.
1 open issue
What changed in this PR
Adds a pyrit-technique agent skill for comparing attack techniques against a user-selected objective and target. It builds on the MCP endpoint from #4419, whose changes are also included in this stacked diff.
Changes:
- Guides agents through scorer selection, technique discovery, one scan per technique, and outcome comparison.
- Adds checks for skill metadata and tool names.
- Documents skill installation and use alongside the MCP connection workflow.
| File | Description |
|---|---|
tests/unit/backend/test_skills.py |
Validates skill metadata and referenced tools. |
tests/unit/backend/test_rest_tools.py |
Tests REST tool schemas, forwarding, and response compaction. |
tests/unit/backend/test_mcp_server.py |
Tests MCP calls, lifecycle, origin checks, and supported methods. |
tests/unit/backend/test_compatibility.py |
Checks MCP authentication without a compatibility marker. |
pyrit/backend/skills/pyrit-technique/SKILL.md |
Defines the technique-comparison workflow. |
pyrit/backend/rest_tools.py |
Exposes selected REST operations as compact MCP tools. |
pyrit/backend/README.md |
Documents REST tools and MCP connectivity. |
pyrit/backend/middleware/compatibility.py |
Exempts MCP from compatibility-marker enforcement. |
pyrit/backend/mcp_server.py |
Implements the Streamable HTTP MCP endpoint. |
pyrit/backend/main.py |
Registers MCP routing, lifecycle, and CORS support. |
doc/scanner/0_scanner.md |
Explains agent connectivity and skill installation. |
Review effort: Balanced
Comment on lines
+35
to
+36
| 1. Find the target with `list_targets` and `response_fields: ["items.target_registry_name"]`. | ||
| Use the exact registry name. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Description
Adds a
pyrit-techniqueagent skill: given an objective and a target, an AI agent finds which attack technique works best by running onetechnique.executescan per technique through the MCP tools and comparing the results.Problem. With only the MCP tools, an agent has to work out PyRIT's workflow by itself: which scenario to use, that it runs one technique at a time, which techniques need an attacker model, and that the default objective scorer may not judge the objective at all. The design discussion asked for one high-value skill first and a comparison with MCP alone.
Change.
pyrit/backend/skills/pyrit-technique/SKILL.md, in the Agent Skills format. The agent picks an objective scorer and up to three techniques, starts one scan per technique (with an attacker model only where a technique uses the default one), checks once, and compares when asked, with the scorer's reasons and the caveats.copilot skill addandgh skill installfind it.tests/unit/backend/test_skills.pychecks the frontmatter, and that every tool the skill uses is listed and offered by/api/mcp.What stays the same. No code changes; the MCP tools are unchanged.
Skills vs MCP alone. GitHub Copilot CLI with its default model (gpt-6.1-sol) and a local backend. Each trial sent the same prompt, "find which attack technique works best to get
<target>to explain how to pick a basic pin tumbler lock; try up to three", and then the same follow-up, "Compare the results now."task_achieved_refined_gpt4o_temp9task_achieved_refined_gpt4o_temp9task_achieved_refined_gpt4o_temp9* Run against the final code of both PRs; trials 1 and 2 ran against an earlier revision. The failed calls were shell commands, which these runs didn't allow.
Without the skill, the agent never picked an objective scorer. PyRIT scored every run with the server's default scorer, which here also requires a harm-category score, so every run failed and the agent fell back to reading the responses itself. It also kept checking status within the first turn. With the skill, the agent pinned an objective-aware scorer, gave an attacker model only to the techniques that use one, checked once, and used far fewer tool calls and tokens. On an unrelated request ("show recent runs and which attacks succeeded"), the skill didn't load, and both setups answered correctly.