Skip to content

feat(rf-core): add the rf-bdd skill - #46

Merged
manykarim merged 2 commits into
mainfrom
change/add-rf-bdd-skill-v5
Nov 20, 2026
Merged

manykarim merged 2 commits into
mainfrom
change/add-rf-bdd-skill-v5

Conversation

@manykarim

Copy link
Copy Markdown
Collaborator

Summary

Adds rf-bdd to rf-core: writing and reviewing behaviour-driven specifications in native .robot and .resource files (OpenSpec change add-rf-bdd-skill).

  • Skill: SKILL.md (169 lines) with 11 house rules, Gherkin input, 11 gotchas; 7 references; assets/robocop-bdd.toml; two runnable example projects (resource strategy, World library) on a fake ATM service.
  • Verified: the examples analyse, dry-run and pass on RF 7.4.2 and 7.5.1 (and Python 3.8 with RF 7.3.2), and lint clean on Robocop 8.2.10 and 9.4. All 11 gotchas were reproduced on both version sets (trap-verification.md).
  • rf-language: points to rf-bdd; the casing gotcha follows the project's Robocop configuration; corrects "a comma-separated --select is rejected" to "selects nothing".
  • Routing: the context hook and rf-test-architect (plus Codex/OpenCode variants) route BDD work to rf-bdd.
  • Evals: sut-bdd-library fixture (base, steps, review), a BDD grader with 30 tests and golden solutions, five tasks, a trigger set (11 positive, 11 negative, 4 holdout). One-line cli.py preflight fix: tasks are selected by fixture root (sut-bdd-library/base → sut-bdd-library).
  • CHANGELOG ## Unreleased, README and rf-core README updated; skill count 10.

Not in this PR

Tasks 1.1, 5.5 and 5.6 (live trigger baseline and tuning, CI-model task baselines) follow in a separate PR. A local spot check with Haiku 5.5 loaded rf-bdd in 3 of 3 runs; the late-return suites passed (5/5, 10/10) with no .feature files.

Test plan

  • Full pytest locally: all pass except the 4 Browser tests that need Playwright
  • build-catalog --check, governance, content rules, validate-skills, plugin references, drift, docs voice, security scan
  • openspec validate --all --strict
  • CI

AI assistance

Written with Claude Code (Claude Opus 5.5); the maintainer reviewed the change and directed the scope. Reference porting and the eval fixture were done by subagents and checked by the tests above.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP

Adds rf-bdd for behaviour-driven specifications in native .robot files:
house rules, Gherkin mapping, step vocabularies, scenario state, a
Robocop configuration, and two runnable example projects (resource and
World-library strategies), verified on RF 7.4.2 and 7.5.1 and on
Robocop 8.2.10 and 9.4.

- rf-language points to rf-bdd; its casing gotcha follows the project's
  Robocop configuration; a comma-separated --select selects nothing
  (it is not rejected).
- The context hook and rf-test-architect route BDD work to rf-bdd.
- Evals: the sut-bdd-library fixture (base, steps, review variants),
  a BDD grader with tests and golden solutions, five tasks and a
  trigger set. The preflight selects tasks by fixture root.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Skill Evaluation: rf-mobile (narrow, treatment, N=3, z-ai/glm-5.3-flash)

Model: z-ai/glm-5.3-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.35
Harness estimate (OpenRouter list prices) 0.04
Credit remaining on the key 33.88275571

Gate

gate: PASS (1 task(s) checked)

rf-skill-eval report

Runs: 3 · pass: 3 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.04

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-appium-login-01 rf-appium narrow treatment 3 100% 100% 0 405475 11426 21.3 140.5 0.0143 (est.) no
narrow-appium-login-01 rf-appium narrow baseline (stored) 3 100% 100% 0 126666 2566 10.3 30.9 0.0043 (est.) –
narrow-appium-login-01 rf-appium narrow Δ T−B +0 pp +278809 +8860 +11.0 +109.7 +0.0100

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-appium 1 100% 100% +0 pp +278809 +11.0 +0.0100

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-appium-login-01 loaded_rf_appium_skill: 3/3

Adversarial tier (reported only, never gates)

None.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Skill Evaluation: rf-desktop (narrow, treatment, N=3, z-ai/glm-5.3-flash)

Model: z-ai/glm-5.3-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.33
Harness estimate (OpenRouter list prices) 0.04
Credit remaining on the key 33.914124709999996

Gate

gate: PASS (1 task(s) checked)

rf-skill-eval report

Runs: 3 · pass: 3 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.04 · turn-limit stops: 1

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-platynui-calculator-01 rf-platynui narrow treatment 3 100% 100% 0 423590 10337 20.0 (1 at limit) 113.5 0.0142 (est.) no
narrow-platynui-calculator-01 rf-platynui narrow baseline (stored) 3 100% 100% 0 361178 4420 19.7 (1 at limit) 109.6 0.0116 (est.) –
narrow-platynui-calculator-01 rf-platynui narrow Δ T−B +0 pp +62411 +5917 +0.3 +3.9 +0.0026

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-platynui 1 100% 100% +0 pp +62411 +0.3 +0.0026

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-platynui-calculator-01 loaded_rf_platynui_skill: 3/3

Adversarial tier (reported only, never gates)

None.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Skill Evaluation: rf-web (narrow, treatment, N=3, z-ai/glm-5.3-flash)

Model: z-ai/glm-5.3-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.27
Harness estimate (OpenRouter list prices) 0.04
Credit remaining on the key 33.970856547

Gate

gate: PASS (2 task(s) checked)

rf-skill-eval report

Runs: 6 · pass: 6 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.04

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-browser-login-01 rf-browser narrow treatment 3 100% 100% 0 192667 1191 13.0 26.3 0.0063 (est.) no
narrow-browser-login-01 rf-browser narrow baseline (stored) 6 100% 100% 0 70371 1454 6.7 22.1 0.0024 (est.) –
narrow-browser-login-01 rf-browser narrow Δ T−B +0 pp +122297 -262 +6.3 +4.2 +0.0039
narrow-selenium-login-01 rf-selenium narrow treatment 3 100% 100% 0 157839 3389 11.0 42.5 0.0055 (est.) no
narrow-selenium-login-01 rf-selenium narrow baseline (stored) 6 100% 100% 0 72006 801 8.0 12.7 0.0024 (est.) –
narrow-selenium-login-01 rf-selenium narrow Δ T−B +0 pp +85833 +2588 +3.0 +29.8 +0.0031

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-browser 1 100% 100% +0 pp +122297 +6.3 +0.0039
rf-selenium 1 100% 100% +0 pp +85833 +3.0 +0.0031

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-browser-login-01 loaded_rf_browser_skill: 3/3
  • narrow-selenium-login-01 loaded_rf_selenium_skill: 3/3

Adversarial tier (reported only, never gates)

None.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Skill Evaluation: rf-api (narrow, treatment, N=3, z-ai/glm-5.3-flash)

Model: z-ai/glm-5.3-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.31
Harness estimate (OpenRouter list prices) 0.06
Credit remaining on the key 33.930031853

Gate

gate: PASS (2 task(s) checked)

rf-skill-eval report

Runs: 6 · pass: 6 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.06

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-requests-users-01 rf-requests narrow treatment 3 100% 100% 0 292669 2294 18.0 34.0 0.0094 (est.) no
narrow-requests-users-01 rf-requests narrow baseline (stored) 6 100% 100% 0 78154 836 8.0 12.3 0.0025 (est.) –
narrow-requests-users-01 rf-requests narrow Δ T−B +0 pp +214515 +1458 +10.0 +21.8 +0.0069
narrow-restinstance-schema-01 rf-restinstance narrow treatment 3 100% 100% 0 385337 3293 22.0 57.8 0.0121 (est.) no
narrow-restinstance-schema-01 rf-restinstance narrow baseline (stored) 6 100% 100% 0 142111 1992 11.8 25.9 0.0047 (est.) –
narrow-restinstance-schema-01 rf-restinstance narrow Δ T−B +0 pp +243226 +1301 +10.2 +31.9 +0.0074

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-requests 1 100% 100% +0 pp +214515 +10.0 +0.0069
rf-restinstance 1 100% 100% +0 pp +243226 +10.2 +0.0074

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-requests-users-01 loaded_rf_requests_skill: 3/3
  • narrow-restinstance-schema-01 loaded_rf_restinstance_skill: 3/3

Adversarial tier (reported only, never gates)

None.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Skill Evaluation: rf-core (narrow, treatment, N=3, z-ai/glm-5.3-flash)

Model: z-ai/glm-5.3-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 1.17
Harness estimate (OpenRouter list prices) 0.55
Credit remaining on the key 33.074202584000005

Gate

gate: PASS (16 task(s) checked)

Trigger gate

gate: PASS (1 task(s) checked)

rf-skill-eval trigger report

Model z-ai/glm-5.3-flash · skill-listing budget default · a query is triggered when its trigger rate (loads / runs) is >= its set's threshold. TP/FP/TN/FN per split; train and validation are reported separately (tune on train only).

Skill Train TP/FP/TN/FN Train P Train R Train Acc Val TP/FP/TN/FN Val P Val R Val Acc Stored Val Acc
rf-bdd – – – – 4/0/4/0 100% 100% 100% –

Description visibility

Sessions whose skill listing showed the skill's description (the rest listed it by name only). Own counts the sessions of the skill's own queries, All every session in this result.

Skill Own sessions All sessions
rf-appium – 24/24
rf-bdd 24/24 24/24
rf-browser – 24/24
rf-language – 24/24
rf-platynui – 24/24
rf-python-library – 24/24
rf-requests – 24/24
rf-restinstance – 24/24
rf-selenium – 24/24
rf-setup – 24/24

Failing queries

None.

rf-skill-eval report

Runs: 48 · pass: 46 · fail: 2 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.53 · turn-limit stops: 1

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-bdd-extend-vocabulary-01 rf-bdd narrow treatment 3 100% 100% 0 621929 19062 25.3 186.9 0.0210 (est.) no
narrow-bdd-extend-vocabulary-01 rf-bdd narrow baseline (stored) 6 100% 100% 0 390570 10806 21.3 128.9 0.0129 (est.) –
narrow-bdd-extend-vocabulary-01 rf-bdd narrow Δ T−B +0 pp +231360 +8256 +4.0 +58.0 +0.0081
narrow-bdd-greenfield-01 rf-bdd narrow treatment 3 100% 100% 0 1182543 18904 34.3 206.1 0.0376 (est.) no
narrow-bdd-greenfield-01 rf-bdd narrow baseline (stored) 6 0% 0% 0 627902 16739 27.2 124.5 0.0206 (est.) –
narrow-bdd-greenfield-01 rf-bdd narrow Δ T−B +100 pp +554641 +2165 +7.2 +81.6 +0.0170
narrow-bdd-outline-01 rf-bdd narrow treatment 3 100% 100% 0 1255642 24448 41.0 261.1 0.0404 (est.) no
narrow-bdd-outline-01 rf-bdd narrow baseline (stored) 6 0% 0% 0 440259 15134 20.2 104.3 0.0148 (est.) –
narrow-bdd-outline-01 rf-bdd narrow Δ T−B +100 pp +815382 +9314 +20.8 +156.8 +0.0256
narrow-bdd-review-01 rf-bdd narrow treatment 3 33% 33% 0 275902 7275 16.7 87.7 0.0092 (est.) yes
narrow-bdd-review-01 rf-bdd narrow baseline (stored) 6 0% 0% 0 66729 3116 6.0 31.2 0.0024 (est.) –
narrow-bdd-review-01 rf-bdd narrow Δ T−B +33 pp +209172 +4159 +10.7 +56.5 +0.0068
narrow-language-bdd-01 rf-language narrow treatment 3 100% 100% 0 228628 1469 15.3 29.5 0.0072 (est.) no
narrow-language-bdd-01 rf-language narrow baseline (stored) 6 100% 100% 0 79812 848 7.5 13.0 0.0026 (est.) –
narrow-language-bdd-01 rf-language narrow Δ T−B +0 pp +148817 +621 +7.8 +16.5 +0.0046
narrow-language-data-driven-01 rf-language narrow treatment 3 100% 100% 0 214542 3824 14.3 51.2 0.0070 (est.) no
narrow-language-data-driven-01 rf-language narrow baseline (stored) 6 83% 83% 0 114378 1513 10.0 24.1 0.0038 (est.) –
narrow-language-data-driven-01 rf-language narrow Δ T−B +17 pp +100164 +2310 +4.3 +27.1 +0.0032
narrow-language-embedded-01 rf-language narrow treatment 3 100% 100% 0 123949 3061 10.0 41.0 0.0043 (est.) no
narrow-language-embedded-01 rf-language narrow baseline (stored) 6 100% 100% 0 106643 2098 9.2 25.9 0.0037 (est.) –
narrow-language-embedded-01 rf-language narrow Δ T−B +0 pp +17305 +964 +0.8 +15.1 +0.0006
narrow-language-env-varfiles-01 rf-language narrow treatment 3 100% 100% 0 485383 7580 22.7 (1 at limit) 99.6 0.0154 (est.) no
narrow-language-env-varfiles-01 rf-language narrow baseline (stored) 6 100% 100% 0 193964 5167 17.2 (1 at limit) 52.4 0.0066 (est.) –
narrow-language-env-varfiles-01 rf-language narrow Δ T−B +0 pp +291419 +2413 +5.5 +47.2 +0.0088
narrow-language-suite-setup-tags-01 rf-language narrow treatment 3 100% 100% 0 281882 2380 20.3 32.3 0.0089 (est.) no
narrow-language-suite-setup-tags-01 rf-language narrow baseline (stored) 6 100% 100% 0 112928 1551 13.2 21.0 0.0037 (est.) –
narrow-language-suite-setup-tags-01 rf-language narrow Δ T−B +0 pp +168954 +829 +7.2 +11.3 +0.0052
narrow-non-rf-control-01 plugin narrow treatment 3 100% 100% 0 37119 186 2.3 4.3 0.0013 (est.) no
narrow-non-rf-control-01 plugin narrow baseline (stored) 6 100% 100% 0 26311 95 2.0 3.2 0.0009 (est.) –
narrow-non-rf-control-01 plugin narrow Δ T−B +0 pp +10808 +91 +0.3 +1.1 +0.0004
narrow-python-library-flaky-listener-01 rf-python-library narrow treatment 3 100% 100% 0 216697 2701 15.0 25.4 0.0069 (est.) no
narrow-python-library-flaky-listener-01 rf-python-library narrow baseline (stored) 6 100% 100% 0 139834 3160 11.5 34.2 0.0046 (est.) –
narrow-python-library-flaky-listener-01 rf-python-library narrow Δ T−B +0 pp +76863 -459 +3.5 -8.8 +0.0023
narrow-python-library-inventory-scope-01 rf-python-library narrow treatment 3 100% 100% 0 185542 2010 12.7 29.6 0.0061 (est.) no
narrow-python-library-inventory-scope-01 rf-python-library narrow baseline (stored) 6 100% 100% 0 246142 3910 16.5 (1 at limit) 49.6 0.0082 (est.) –
narrow-python-library-inventory-scope-01 rf-python-library narrow Δ T−B +0 pp -60600 -1900 -3.8 -20.0 -0.0021
narrow-python-library-mode-literal-01 rf-python-library narrow treatment 3 100% 100% 0 229144 4518 14.7 52.6 0.0077 (est.) no
narrow-python-library-mode-literal-01 rf-python-library narrow baseline (stored) 6 100% 100% 0 128381 2979 10.3 34.9 0.0044 (est.) –
narrow-python-library-mode-literal-01 rf-python-library narrow Δ T−B +0 pp +100763 +1539 +4.3 +17.8 +0.0033
narrow-rf-injection-positive-01 plugin narrow treatment 3 100% 100% 0 89361 1057 6.3 17.2 0.0030 (est.) no
narrow-rf-injection-positive-01 plugin narrow baseline (stored) 6 100% 100% 0 64147 1338 4.8 15.4 0.0022 (est.) –
narrow-rf-injection-positive-01 plugin narrow Δ T−B +0 pp +25214 -280 +1.5 +1.8 +0.0008
narrow-rf-mcp-execute-01 plugin narrow treatment 3 100% 100% 0 306810 3038 15.0 43.3 0.0098 (est.) no
narrow-rf-mcp-execute-01 plugin narrow baseline (stored) 6 100% 100% 0 145726 933 8.5 17.4 0.0049 (est.) –
narrow-rf-mcp-execute-01 plugin narrow Δ T−B +0 pp +161083 +2105 +6.5 +25.8 +0.0049
narrow-setup-requests-01 rf-setup narrow treatment 3 100% 100% 0 141550 2611 11.0 31.7 0.0047 (est.) no
narrow-setup-requests-01 rf-setup narrow baseline (stored) 6 100% 100% 0 96690 839 8.5 18.0 0.0032 (est.) –
narrow-setup-requests-01 rf-setup narrow Δ T−B +0 pp +44860 +1772 +2.5 +13.7 +0.0015

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-bdd 4 83% 25% +58 pp +452639 +10.7 +0.0144
rf-language 5 100% 97% +3 pp +145332 +5.1 +0.0045
rf-python-library 3 100% 100% +0 pp +39008 +1.3 +0.0012
rf-setup 1 100% 100% +0 pp +44860 +2.5 +0.0015

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-bdd-extend-vocabulary-01 loaded_rf_bdd_skill: 3/3
  • narrow-bdd-greenfield-01 loaded_rf_bdd_skill: 3/3
  • narrow-bdd-outline-01 loaded_rf_bdd_skill: 3/3
  • narrow-bdd-review-01 loaded_rf_bdd_skill: 3/3
  • narrow-language-bdd-01 loaded_rf_language_skill: 3/3
  • narrow-language-data-driven-01 loaded_rf_language_skill: 3/3
  • narrow-language-embedded-01 loaded_rf_language_skill: 3/3
  • narrow-language-env-varfiles-01 loaded_rf_language_skill: 3/3
  • narrow-language-suite-setup-tags-01 loaded_rf_language_skill: 3/3
  • narrow-non-rf-control-01 bounded_tool_calls: 3/3
  • narrow-non-rf-control-01 no_libdoc_invocation: 3/3
  • narrow-non-rf-control-01 no_rf_mcp_calls: 3/3
  • narrow-python-library-flaky-listener-01 loaded_rf_python_library_skill: 3/3
  • narrow-python-library-inventory-scope-01 loaded_rf_python_library_skill: 3/3
  • narrow-python-library-mode-literal-01 loaded_rf_python_library_skill: 3/3
  • narrow-rf-injection-positive-01 leveraged_rf_lookup: 3/3
  • narrow-rf-mcp-execute-01 rf_mcp_before_write: 3/3
  • narrow-rf-mcp-execute-01 rf_mcp_called_at_least_twice: 3/3
  • narrow-rf-mcp-execute-01 rf_mcp_had_successful_result: 3/3
  • narrow-setup-requests-01 used_uv_add: 3/3

Adversarial tier (reported only, never gates)

None.

…77004)

Adds only the rf-bdd entries (narrow, adversarial, triggers) from the
candidate of run 44705577004: N=6, both arms, turn factor 1.5, $4.54 of
OpenRouter spend. Existing entries are unchanged.

The review grader counted the library_lending.egg-info/ that `uv run`
builds as an edit, so every treatment run of narrow-bdd-review-01
failed. It now ignores *.egg-info; the stored treatment pass rate
(5/6) is the fixed grader applied to the same six runs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
@manykarim
manykarim merged commit b2b63ab into main Nov 20, 2026
33 checks passed
@manykarim
manykarim deleted the change/add-rf-bdd-skill-v5 branch November 21, 2026 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant