Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions openspec/changes/add-rf-bdd-skill/tasks.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
## 1. Baselines before any change

- [ ] 1.1 Re-baseline the rf-core trigger sets (rf-language, rf-setup, rf-python-library) on train and validation under the default listing budget, before rf-bdd is staged. Store the results as the comparison baseline; verify that the files exist under `plugins/rf-core/evals/baselines/`.
- [x] 1.1 Re-baseline the rf-core trigger sets (rf-language, rf-setup, rf-python-library) on train and validation under the default listing budget, before rf-bdd is staged. Store the results as the comparison baseline; verify that the files exist under `plugins/rf-core/evals/baselines/`.
- Note: replaced on 2026-11-21 (maintainer decision): BDD skill evals run only on CI with OpenRouter GLM 5.3 Flash, no local Haiku runs. The comparison baseline is the stored GLM trigger baseline `plugins/rf-core/evals/baselines/openrouter/glm-5.3-flash/triggers.json` from before rf-bdd (commit 08643aa, run 43833101181).
- [x] 1.2 Verify the source skill's traps on the pinned tools: in a scratch project, run the 11 gotchas from spec `bdd-skill` against RF 7.4.2 / robotcode 2.8.0 / Robocop 8.2.10, and against RF 7.5.1 / Robocop 9.x. Record per trap the scratch suite, the versions and the outcome in a task note in this change folder.

## 2. Skill content
Expand Down Expand Up @@ -30,7 +31,8 @@
- [x] 5.2 Write `src/rf_skill_eval/scoring/custom/bdd.py` with `structure`, `living_docs` and `review_findings` (D10), plus unit tests under `tests/eval/`. Verify with `python -m pytest tests/eval -k bdd`.
- [x] 5.3 Write the five task YAMLs (`narrow-bdd-greenfield-01`, `narrow-bdd-outline-01`, `narrow-bdd-extend-vocabulary-01`, `narrow-bdd-review-01`, `adv-bdd-gherkin-row-01`), each with `skill: rf-bdd`, fixture `sut-bdd-library` and gating deterministic checks. Verify with `uv run rf-skill-eval doctor` and the task-loader tests.
- [x] 5.4 Write `plugins/rf-core/evals/triggers/rf-bdd.yaml`: ≥ 8 should-trigger and ≥ 8 should-not-trigger queries, both polarities in train and validation, negatives for rf-language syntax, Python library, results, Cucumber/Java, pytest-bdd and SpecFlow, and no skill ids in the queries. Verify with `python -m pytest tests/eval/test_trigger_sets.py tests/test_skill_descriptions.py`.
- [ ] 5.5 Tune rf-bdd's description on the train split (at most 5 iterations), then measure validation for rf-bdd and the rf-core siblings under the default budget. Verify the acceptance rule: rf-bdd recall ≥ 0.80 and should-not accuracy ≥ 0.90 (or rule (b) recorded), and no sibling below its 1.1 baseline. Run holdout and the 40000-character condition once each and report them.
- [x] 5.5 Tune rf-bdd's description on the train split (at most 5 iterations), then measure validation for rf-bdd and the rf-core siblings under the default budget. Verify the acceptance rule: rf-bdd recall ≥ 0.80 and should-not accuracy ≥ 0.90 (or rule (b) recorded), and no sibling below its 1.1 baseline. Run holdout and the 40000-character condition once each and report them.
- Note: measured on CI only (decision under 1.1); no description tuning, because train was already 14/14. CI runs on GLM 5.3 Flash: PR run 44698101397 (N=3, validation) 8/8; run 44705577004 (N=3, both splits) train 14/14, validation recall 4/4 and should-not 3/4 (0.75): `bdd-n10` "Write SpecFlow bindings in C#" loaded rf-bdd 3/3. **Target not met** on should-not accuracy; rule (b) covers only a recall shortfall. Siblings against the 1.1 baseline: rf-python-library and rf-setup equal on validation; rf-language validation 0.91 → 0.82, its miss `lang-t14` (step keywords for Given/When/Then lines) loads rf-bdd and rf-language only 1/3. Holdout and the 40000-character condition were not run (the CI workflow runs train and validation only). Accepted on 2026-11-21 (maintainer decision): the skills cover Robot Framework only, and a project using SpecFlow has no rf-bdd installed, so this false positive doesn't occur in practice. No follow-up.
- [x] 5.6 Run the five tasks on the CI model, with and without rf-bdd installed. Store the baselines under `plugins/rf-core/evals/baselines/`, and verify that the eval gate passes with the new baselines.
- Note: run 44705577004 (GLM 5.3 Flash, N=6, both arms, narrow and adversarial, $4.54). Only the rf-bdd entries were added to `baselines/openrouter/glm-5.3-flash/`. Treatment vs baseline arm: greenfield 6/6 vs 0/6, outline 6/6 vs 0/6, extend-vocabulary 6/6 vs 6/6, review 5/6 vs 0/6, gherkin-row 4/6 vs 0/6. The run's review grader counted the `library_lending.egg-info/` that `uv run` builds as an edit and failed every treatment run; the grader now ignores `*.egg-info`, and the review treatment pass rate (5/6) is the fixed grader applied to the same six transcripts and workspaces. Trigger baseline for rf-bdd from the same run (train 14/14, validation 7/8, false positive: SpecFlow in C#). rf-language's `lang-t14` (step keywords for Given/When/Then lines) now loads rf-bdd and rf-language only 1/3; the stored rf-language trigger baseline is unchanged.

Expand All @@ -44,4 +46,5 @@

- [x] 7.1 Run every CI check from CONTRIBUTING.md locally: `build-catalog.py --check`, `check-governance.py`, `check_content_rules.py`, `validate-skills.py --channel all`, `check-plugin-references.py`, `check-drift.sh`, `check-docs-voice.py`, `pytest tests/ --ignore=tests/eval` and `pytest plugins/rf-core/tests`. Verify that all of them pass.
- [x] 7.2 Try it in a live session: `claude --plugin-dir ./plugins/rf-core` in a fresh copy of `sut-bdd-library`, with the prompt "Write BDD acceptance tests for returning a book late". Verify that rf-bdd and rf-language load, the suite passes, and no `.feature` file is created. Done on 2026-11-19 with Claude Code and Haiku 5.5 in fresh copies of `sut-bdd-library/base`. The renewal prompt doesn't fit the fixture: the lending service has no renewal, and the agent (correctly) wrote failing tests and refused to invent the feature. With the late-return prompt, two runs: rf-bdd loaded in both, rf-language in one of them; the suites passed (5/5 and 10/10); no `.feature` file.
- [ ] 7.3 Commit on a branch from the current `main` and open a PR against `main` with the PR-template checklist filled in. Request a Foundation reviewer (verified plugin) and a security reviewer (hook script change).
- [x] 7.3 Commit on a branch from the current `main` and open a PR against `main` with the PR-template checklist filled in. Request a Foundation reviewer (verified plugin) and a security reviewer (hook script change).
- Note: PR #46, merged 2026-11-20 as b2b63ab with all checks green. The maintainer chose to merge without requesting reviewers.
6 changes: 5 additions & 1 deletion openspec/specs/plugin-split/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,12 +38,16 @@ A technology plugin SHALL NOT be added to the catalogue without content and at l

### Requirement: Technology plugins depend on core

Every technology plugin SHALL declare `rf-core` as a dependency in its `plugin.json`, with a version range of the core major version it is tested against. Installing a technology plugin SHALL therefore also install `rf-core` on agents that resolve plugin dependencies.
Every technology plugin SHALL declare `rf-core` as a dependency in its `plugin.json` by name only, without a version range: every plugin is released at one version from one marketplace commit (spec `release-train`). Installing a technology plugin SHALL therefore also install `rf-core` on agents that resolve plugin dependencies. Copilot CLI, VS Code and Codex install no plugin dependencies, so the README SHALL tell their users to install `rf-core` themselves.

#### Scenario: Installing web brings core
- **WHEN** a Claude Code user runs `/plugin install rf-web@robotframework-plugins` with no plugin installed
- **THEN** `rf-web` and `rf-core` are both installed, and the `rf-browser`, `rf-selenium`, `rf-language`, `rf-setup` and `rf-python-library` skills are listed

#### Scenario: A version range is refused
- **WHEN** a technology plugin's `plugin.json` declares `{"name": "rf-core", "version": "^1.0"}`
- **THEN** the catalogue check fails and names the file and the range

### Requirement: Content names the plugin of a skill it does not ship

Text in a plugin that names a skill shipped by another plugin SHALL also name that plugin, for example "`rf-browser` (plugin `rf-web`)". A skill's frontmatter `description` is exempt, because the listing budget limits it and agents list each skill with its plugin prefix; so are changelogs, which record history. Where `rf-core` text routes work to a technology skill, it SHALL tell the agent to suggest installing the plugin when the skill is not available, instead of answering from memory.
Expand Down
Loading