diff --git a/openspec/changes/add-rf-bdd-skill/tasks.md b/openspec/changes/add-rf-bdd-skill/tasks.md index 4088a1d..1f04425 100644 --- a/openspec/changes/add-rf-bdd-skill/tasks.md +++ b/openspec/changes/add-rf-bdd-skill/tasks.md @@ -1,6 +1,7 @@ ## 1. Baselines before any change -- [ ] 1.1 Re-baseline the rf-core trigger sets (rf-language, rf-setup, rf-python-library) on train and validation under the default listing budget, before rf-bdd is staged. Store the results as the comparison baseline; verify that the files exist under `plugins/rf-core/evals/baselines/`. +- [x] 1.1 Re-baseline the rf-core trigger sets (rf-language, rf-setup, rf-python-library) on train and validation under the default listing budget, before rf-bdd is staged. Store the results as the comparison baseline; verify that the files exist under `plugins/rf-core/evals/baselines/`. + - Note: replaced on 2026-11-21 (maintainer decision): BDD skill evals run only on CI with OpenRouter GLM 5.3 Flash, no local Haiku runs. The comparison baseline is the stored GLM trigger baseline `plugins/rf-core/evals/baselines/openrouter/glm-5.3-flash/triggers.json` from before rf-bdd (commit 08643aa, run 43833101181). - [x] 1.2 Verify the source skill's traps on the pinned tools: in a scratch project, run the 11 gotchas from spec `bdd-skill` against RF 7.4.2 / robotcode 2.8.0 / Robocop 8.2.10, and against RF 7.5.1 / Robocop 9.x. Record per trap the scratch suite, the versions and the outcome in a task note in this change folder. ## 2. Skill content @@ -30,7 +31,8 @@ - [x] 5.2 Write `src/rf_skill_eval/scoring/custom/bdd.py` with `structure`, `living_docs` and `review_findings` (D10), plus unit tests under `tests/eval/`. Verify with `python -m pytest tests/eval -k bdd`. - [x] 5.3 Write the five task YAMLs (`narrow-bdd-greenfield-01`, `narrow-bdd-outline-01`, `narrow-bdd-extend-vocabulary-01`, `narrow-bdd-review-01`, `adv-bdd-gherkin-row-01`), each with `skill: rf-bdd`, fixture `sut-bdd-library` and gating deterministic checks. Verify with `uv run rf-skill-eval doctor` and the task-loader tests. - [x] 5.4 Write `plugins/rf-core/evals/triggers/rf-bdd.yaml`: ≥ 8 should-trigger and ≥ 8 should-not-trigger queries, both polarities in train and validation, negatives for rf-language syntax, Python library, results, Cucumber/Java, pytest-bdd and SpecFlow, and no skill ids in the queries. Verify with `python -m pytest tests/eval/test_trigger_sets.py tests/test_skill_descriptions.py`. -- [ ] 5.5 Tune rf-bdd's description on the train split (at most 5 iterations), then measure validation for rf-bdd and the rf-core siblings under the default budget. Verify the acceptance rule: rf-bdd recall ≥ 0.80 and should-not accuracy ≥ 0.90 (or rule (b) recorded), and no sibling below its 1.1 baseline. Run holdout and the 40000-character condition once each and report them. +- [x] 5.5 Tune rf-bdd's description on the train split (at most 5 iterations), then measure validation for rf-bdd and the rf-core siblings under the default budget. Verify the acceptance rule: rf-bdd recall ≥ 0.80 and should-not accuracy ≥ 0.90 (or rule (b) recorded), and no sibling below its 1.1 baseline. Run holdout and the 40000-character condition once each and report them. + - Note: measured on CI only (decision under 1.1); no description tuning, because train was already 14/14. CI runs on GLM 5.3 Flash: PR run 44698101397 (N=3, validation) 8/8; run 44705577004 (N=3, both splits) train 14/14, validation recall 4/4 and should-not 3/4 (0.75): `bdd-n10` "Write SpecFlow bindings in C#" loaded rf-bdd 3/3. **Target not met** on should-not accuracy; rule (b) covers only a recall shortfall. Siblings against the 1.1 baseline: rf-python-library and rf-setup equal on validation; rf-language validation 0.91 → 0.82, its miss `lang-t14` (step keywords for Given/When/Then lines) loads rf-bdd and rf-language only 1/3. Holdout and the 40000-character condition were not run (the CI workflow runs train and validation only). Accepted on 2026-11-21 (maintainer decision): the skills cover Robot Framework only, and a project using SpecFlow has no rf-bdd installed, so this false positive doesn't occur in practice. No follow-up. - [x] 5.6 Run the five tasks on the CI model, with and without rf-bdd installed. Store the baselines under `plugins/rf-core/evals/baselines/`, and verify that the eval gate passes with the new baselines. - Note: run 44705577004 (GLM 5.3 Flash, N=6, both arms, narrow and adversarial, $4.54). Only the rf-bdd entries were added to `baselines/openrouter/glm-5.3-flash/`. Treatment vs baseline arm: greenfield 6/6 vs 0/6, outline 6/6 vs 0/6, extend-vocabulary 6/6 vs 6/6, review 5/6 vs 0/6, gherkin-row 4/6 vs 0/6. The run's review grader counted the `library_lending.egg-info/` that `uv run` builds as an edit and failed every treatment run; the grader now ignores `*.egg-info`, and the review treatment pass rate (5/6) is the fixed grader applied to the same six transcripts and workspaces. Trigger baseline for rf-bdd from the same run (train 14/14, validation 7/8, false positive: SpecFlow in C#). rf-language's `lang-t14` (step keywords for Given/When/Then lines) now loads rf-bdd and rf-language only 1/3; the stored rf-language trigger baseline is unchanged. @@ -44,4 +46,5 @@ - [x] 7.1 Run every CI check from CONTRIBUTING.md locally: `build-catalog.py --check`, `check-governance.py`, `check_content_rules.py`, `validate-skills.py --channel all`, `check-plugin-references.py`, `check-drift.sh`, `check-docs-voice.py`, `pytest tests/ --ignore=tests/eval` and `pytest plugins/rf-core/tests`. Verify that all of them pass. - [x] 7.2 Try it in a live session: `claude --plugin-dir ./plugins/rf-core` in a fresh copy of `sut-bdd-library`, with the prompt "Write BDD acceptance tests for returning a book late". Verify that rf-bdd and rf-language load, the suite passes, and no `.feature` file is created. Done on 2026-11-19 with Claude Code and Haiku 5.5 in fresh copies of `sut-bdd-library/base`. The renewal prompt doesn't fit the fixture: the lending service has no renewal, and the agent (correctly) wrote failing tests and refused to invent the feature. With the late-return prompt, two runs: rf-bdd loaded in both, rf-language in one of them; the suites passed (5/5 and 10/10); no `.feature` file. -- [ ] 7.3 Commit on a branch from the current `main` and open a PR against `main` with the PR-template checklist filled in. Request a Foundation reviewer (verified plugin) and a security reviewer (hook script change). +- [x] 7.3 Commit on a branch from the current `main` and open a PR against `main` with the PR-template checklist filled in. Request a Foundation reviewer (verified plugin) and a security reviewer (hook script change). + - Note: PR #46, merged 2026-11-20 as b2b63ab with all checks green. The maintainer chose to merge without requesting reviewers. diff --git a/openspec/specs/plugin-split/spec.md b/openspec/specs/plugin-split/spec.md index 53df027..dba9416 100644 --- a/openspec/specs/plugin-split/spec.md +++ b/openspec/specs/plugin-split/spec.md @@ -38,12 +38,16 @@ A technology plugin SHALL NOT be added to the catalogue without content and at l ### Requirement: Technology plugins depend on core -Every technology plugin SHALL declare `rf-core` as a dependency in its `plugin.json`, with a version range of the core major version it is tested against. Installing a technology plugin SHALL therefore also install `rf-core` on agents that resolve plugin dependencies. +Every technology plugin SHALL declare `rf-core` as a dependency in its `plugin.json` by name only, without a version range: every plugin is released at one version from one marketplace commit (spec `release-train`). Installing a technology plugin SHALL therefore also install `rf-core` on agents that resolve plugin dependencies. Copilot CLI, VS Code and Codex install no plugin dependencies, so the README SHALL tell their users to install `rf-core` themselves. #### Scenario: Installing web brings core - **WHEN** a Claude Code user runs `/plugin install rf-web@robotframework-plugins` with no plugin installed - **THEN** `rf-web` and `rf-core` are both installed, and the `rf-browser`, `rf-selenium`, `rf-language`, `rf-setup` and `rf-python-library` skills are listed +#### Scenario: A version range is refused +- **WHEN** a technology plugin's `plugin.json` declares `{"name": "rf-core", "version": "^1.0"}` +- **THEN** the catalogue check fails and names the file and the range + ### Requirement: Content names the plugin of a skill it does not ship Text in a plugin that names a skill shipped by another plugin SHALL also name that plugin, for example "`rf-browser` (plugin `rf-web`)". A skill's frontmatter `description` is exempt, because the listing budget limits it and agents list each skill with its plugin prefix; so are changelogs, which record history. Where `rf-core` text routes work to a technology skill, it SHALL tell the agent to suggest installing the plugin when the skill is not available, instead of answering from memory.