Skip to content

fix(rf-desktop): fix-desktop-eval-turn-overhead (OpenSpec change) - #4

Draft
manykarim wants to merge 3 commits into
mainfrom
change/fix-desktop-eval-turn-overhead
Draft

fix(rf-desktop): fix-desktop-eval-turn-overhead (OpenSpec change)#4
manykarim wants to merge 3 commits into
mainfrom
change/fix-desktop-eval-turn-overhead

Conversation

@manykarim

@manykarim manykarim commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds the OpenSpec change fix-desktop-eval-turn-overhead. It explains why rf-desktop's skill arm scores below the baseline arm on the OpenRouter tier, and plans the fix. It also ticks the verified openrouter-ci-eval-tier tasks.

This PR holds only the planning artifacts for now. The implementation (tasks 2.3 to 11) lands on this branch next, followed by one re-baseline. It stays a draft until then.

Why

Running the 63 suites from the eval sessions against real Windows Calculator (German Windows 11, PlatynUI 0.12.0.dev330, 2026-11-12) showed:

  • The sut-platynui fixture README gives wrong locators. PlatynUI exposes the UIA AutomationId as @Id; @AutomationId matches nothing, and @Name is localised. All 63 suites failed as written. With the names translated, only 5 suites passed, all from the skill arm. The skill's own example passed both times.
  • The static gate can't see locator mistakes. And min_calls fails 7 correct skill-arm suites that wrap Pointer Click in a user keyword.
  • rf-core hooks add turns the baseline arm never pays: the formatting-drift hint, and the Stop reminder to run tests that can't run there.

Planned changes

  • @Id locators in the fixture README.
  • A platynui_locators static check.
  • keywords_resolve counting library calls through user keywords.
  • Formatting drift becomes opt-in (RF_AGENTSKILLS_FORMAT_CHECK).
  • The Stop reminder is skipped after a run.
  • The robotcode doc keyword line moves up in rf-platynui.
  • A manual Windows live check.
  • One re-baseline on both tiers.

openrouter-ci-eval-tier

  • Ticks 5.2, 5.3, 6.1 and 6.2, with the runs that verified them.
  • 3.2 stays open: list prices came out about 51% above the real OpenRouter charge on the full run ($12.58 against $8.34).

Test plan

  • openspec validate --strict passes for both changes
  • scripts/check-docs-voice.py passes
  • Implementation tasks and re-baseline (tracked in tasks.md)

🤖 Generated with Claude Code

https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP

manykarim and others added 2 commits November 12, 2026 15:34
@manykarim @claude
docs(openspec): fix-desktop-eval-turn-overhead; openrouter tier verified
edd986c 
Adds the fix-desktop-eval-turn-overhead change: the sut-platynui README
gives locators that don't work on a real desktop (checked on Windows 11
with PlatynUI 0.12.0.dev330), keywords_resolve fails suites that wrap
library keywords in user keywords, and rf-core hooks add turns the
baseline arm never pays.

Ticks openrouter-ci-eval-tier tasks 5.2, 5.3, 6.1 and 6.2 with the runs
that verified them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
@manykarim @claude
fix(rf-desktop): working fixture locators, locator check, less hook o…
12205bb 
…verhead

- sut-platynui README: @id locators (verified on Windows 11), language-independent window
- keywords_resolve: min_calls counts library calls reached through user keywords
- new custom.platynui:locators_ok check, gating on the narrow task and in the
  partial fallback (which now takes a list of checks)
- rf-core: formatting drift opt-in (RF_AGENTSKILLS_FORMAT_CHECK); Stop reminder
  skipped once the suite ran after the last write
- rf-platynui: robotcode doc keyword line in Library Import
- scripts/desktop-live-check.py and docs/ci/desktop-live-check.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
@github-actions

github-actions Bot commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Skill Evaluation: rf-mobile (narrow, treatment, N=3, deepseek/deepseek-v4.1-flash)

Model: deepseek/deepseek-v4.1-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

Key spend unavailable (spend file missing). Harness estimate: $0.00.

Gate

gate did not run

no report (no runs recorded)

@github-actions

github-actions Bot commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Skill Evaluation: rf-desktop (narrow, treatment, N=3, deepseek/deepseek-v4.1-flash)

Model: deepseek/deepseek-v4.1-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.03
Harness estimate (OpenRouter list prices) 0.03
Credit remaining on the key 18.556299542

Gate

gate: PASS (1 task(s) checked)

rf-skill-eval report

Runs: 3 · pass: 1 · fail: 2 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.03

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-platynui-calculator-01 rf-platynui narrow treatment 3 33% 33% 0 90568 2500 6.3 10.6 0.0108 (est.) yes
narrow-platynui-calculator-01 rf-platynui narrow baseline (stored) 6 100% 100% 0 508077 14447 19.8 (3 at limit) 109.0 0.0315 (est.) –
narrow-platynui-calculator-01 rf-platynui narrow Δ T−B -67 pp -417510 -11947 -13.5 -98.4 -0.0207

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-platynui 1 33% 100% -67 pp -417510 -13.5 -0.0207

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-platynui-calculator-01 loaded_rf_platynui_skill: 1/3

Adversarial tier (reported only, never gates)

None.

@github-actions

github-actions Bot commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Skill Evaluation: rf-web (narrow, treatment, N=3, deepseek/deepseek-v4.1-flash)

Model: deepseek/deepseek-v4.1-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.04
Harness estimate (OpenRouter list prices) 0.01
Credit remaining on the key 18.543094226

Gate

gate: FAIL (2 task(s) checked)
- [pass-rate] narrow-selenium-login-01: treatment pass rate 0.00 (N=3) vs baseline 1.00 (N=6); Fisher p=0.012 < 0.05

rf-skill-eval report

Runs: 6 · pass: 0 · fail: 6 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.01

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-browser-login-01 rf-browser narrow treatment 3 0% 0% 0 44460 775 5.7 8.8 0.0037 (est.) no
narrow-browser-login-01 rf-browser narrow baseline (stored) 6 100% 100% 0 259818 4250 18.3 41.5 0.0221 (est.) –
narrow-browser-login-01 rf-browser narrow Δ T−B -100 pp -215357 -3474 -12.7 -32.7 -0.0184
narrow-selenium-login-01 rf-selenium narrow treatment 3 0% 0% 0 0 0 1.0 0.4 0.0000 (est.) no
narrow-selenium-login-01 rf-selenium narrow baseline (stored) 6 100% 100% 0 121834 2829 11.0 26.1 0.0120 (est.) –
narrow-selenium-login-01 rf-selenium narrow Δ T−B -100 pp -121834 -2829 -10.0 -25.8 -0.0120

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-browser 1 0% 100% -100 pp -215357 -12.7 -0.0184
rf-selenium 1 0% 100% -100 pp -121834 -10.0 -0.0120

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-browser-login-01 loaded_rf_browser_skill: 1/3
  • narrow-selenium-login-01 loaded_rf_selenium_skill: 0/3

Adversarial tier (reported only, never gates)

None.

@github-actions

github-actions Bot commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Skill Evaluation: rf-api (narrow, treatment, N=3, deepseek/deepseek-v4.1-flash)

Model: deepseek/deepseek-v4.1-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.04
Harness estimate (OpenRouter list prices) 0.02
Credit remaining on the key 18.543094226

Gate

gate: FAIL (2 task(s) checked)
- [pass-rate] narrow-requests-users-01: treatment pass rate 0.00 (N=3) vs baseline 1.00 (N=6); Fisher p=0.012 < 0.05
- [pass-rate] narrow-restinstance-schema-01: treatment pass rate 0.00 (N=3) vs baseline 1.00 (N=6); Fisher p=0.012 < 0.05

rf-skill-eval report

Runs: 6 · pass: 0 · fail: 6 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.02

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-requests-users-01 rf-requests narrow treatment 3 0% 0% 0 33026 426 7.3 4.3 0.0046 (est.) no
narrow-requests-users-01 rf-requests narrow baseline (stored) 6 100% 100% 0 97830 1588 10.7 15.8 0.0099 (est.) –
narrow-requests-users-01 rf-requests narrow Δ T−B -100 pp -64804 -1162 -3.3 -11.5 -0.0053
narrow-restinstance-schema-01 rf-restinstance narrow treatment 3 0% 0% 0 4367 43 2.0 0.9 0.0014 (est.) no
narrow-restinstance-schema-01 rf-restinstance narrow baseline (stored) 6 100% 100% 0 204062 3312 14.7 28.1 0.0119 (est.) –
narrow-restinstance-schema-01 rf-restinstance narrow Δ T−B -100 pp -199696 -3268 -12.7 -27.2 -0.0105

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-requests 1 0% 100% -100 pp -64804 -3.3 -0.0053
rf-restinstance 1 0% 100% -100 pp -199696 -12.7 -0.0105

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-requests-users-01 loaded_rf_requests_skill: 2/3
  • narrow-restinstance-schema-01 loaded_rf_restinstance_skill: 1/3

Adversarial tier (reported only, never gates)

None.

@github-actions

github-actions Bot commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Skill Evaluation: rf-core (narrow, treatment, N=3, deepseek/deepseek-v4.1-flash)

Model: deepseek/deepseek-v4.1-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.05
Harness estimate (OpenRouter list prices) 0.04
Credit remaining on the key 18.527648231

Gate

gate: FAIL (12 task(s) checked)
- [pass-rate] narrow-language-bdd-01: treatment pass rate 0.00 (N=3) vs baseline 1.00 (N=6); Fisher p=0.012 < 0.05
- [pass-rate] narrow-language-data-driven-01: treatment pass rate 0.00 (N=3) vs baseline 1.00 (N=6); Fisher p=0.012 < 0.05
- [pass-rate] narrow-language-embedded-01: treatment pass rate 0.00 (N=3) vs baseline 1.00 (N=6); Fisher p=0.012 < 0.05
- [pass-rate] narrow-language-env-varfiles-01: treatment pass rate 0.00 (N=3) vs baseline 0.83 (N=6); Fisher p=0.048 < 0.05
- [pass-rate] narrow-language-suite-setup-tags-01: treatment pass rate 0.00 (N=3) vs baseline 1.00 (N=6); Fisher p=0.012 < 0.05
- [pass-rate] narrow-non-rf-control-01: treatment pass rate 0.00 (N=3) vs baseline 1.00 (N=6); Fisher p=0.012 < 0.05
- [pass-rate] narrow-python-library-flaky-listener-01: treatment pass rate 0.00 (N=3) vs baseline 1.00 (N=6); Fisher p=0.012 < 0.05
- [pass-rate] narrow-python-library-inventory-scope-01: treatment pass rate 0.00 (N=3) vs baseline 1.00 (N=6); Fisher p=0.012 < 0.05
- [pass-rate] narrow-python-library-mode-literal-01: treatment pass rate 0.00 (N=3) vs baseline 1.00 (N=6); Fisher p=0.012 < 0.05
- [pass-rate] narrow-setup-requests-01: treatment pass rate 0.00 (N=3) vs baseline 1.00 (N=6); Fisher p=0.012 < 0.05

rf-skill-eval report

Runs: 36 · pass: 2 · fail: 34 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.04

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-language-bdd-01 rf-language narrow treatment 3 0% 0% 0 4399 44 2.0 1.4 0.0014 (est.) no
narrow-language-bdd-01 rf-language narrow baseline (stored) 6 100% 100% 0 113239 2040 11.7 14.1 0.0087 (est.) –
narrow-language-bdd-01 rf-language narrow Δ T−B -100 pp -108840 -1996 -9.7 -12.7 -0.0073
narrow-language-data-driven-01 rf-language narrow treatment 3 0% 0% 0 4330 48 2.0 1.0 0.0013 (est.) no
narrow-language-data-driven-01 rf-language narrow baseline (stored) 6 67% 67% 0 175617 4578 15.5 29.9 0.0127 (est.) –
narrow-language-data-driven-01 rf-language narrow Δ T−B -67 pp -171287 -4530 -13.5 -28.9 -0.0114
narrow-language-embedded-01 rf-language narrow treatment 3 0% 0% 0 0 0 1.0 0.4 0.0000 (est.) no
narrow-language-embedded-01 rf-language narrow baseline (stored) 6 100% 100% 0 171208 4690 14.8 36.7 0.0152 (est.) –
narrow-language-embedded-01 rf-language narrow Δ T−B -100 pp -171208 -4690 -13.8 -36.3 -0.0152
narrow-language-env-varfiles-01 rf-language narrow treatment 3 0% 0% 0 0 0 1.0 0.4 0.0000 (est.) no
narrow-language-env-varfiles-01 rf-language narrow baseline (stored) 6 100% 100% 0 266775 6067 22.0 56.7 0.0158 (est.) –
narrow-language-env-varfiles-01 rf-language narrow Δ T−B -100 pp -266775 -6067 -21.0 -56.3 -0.0158
narrow-language-suite-setup-tags-01 rf-language narrow treatment 3 0% 0% 0 0 0 1.0 0.4 0.0000 (est.) no
narrow-language-suite-setup-tags-01 rf-language narrow baseline (stored) 6 100% 100% 0 183760 4722 19.0 40.2 0.0141 (est.) –
narrow-language-suite-setup-tags-01 rf-language narrow Δ T−B -100 pp -183760 -4722 -18.0 -39.8 -0.0141
narrow-non-rf-control-01 plugin narrow treatment 3 0% 0% 0 0 0 1.0 0.4 0.0000 (est.) no
narrow-non-rf-control-01 plugin narrow baseline (stored) 6 100% 100% 0 27066 204 2.0 3.0 0.0047 (est.) –
narrow-non-rf-control-01 plugin narrow Δ T−B -100 pp -27066 -204 -1.0 -2.6 -0.0047
narrow-python-library-flaky-listener-01 rf-python-library narrow treatment 3 0% 0% 0 0 0 1.0 0.4 0.0000 (est.) no
narrow-python-library-flaky-listener-01 rf-python-library narrow baseline (stored) 6 100% 100% 0 572084 13255 23.7 (4 at limit) 78.0 0.0356 (est.) –
narrow-python-library-flaky-listener-01 rf-python-library narrow Δ T−B -100 pp -572084 -13255 -22.7 -77.6 -0.0356
narrow-python-library-inventory-scope-01 rf-python-library narrow treatment 3 0% 0% 0 0 0 1.0 0.5 0.0000 (est.) no
narrow-python-library-inventory-scope-01 rf-python-library narrow baseline (stored) 6 100% 100% 0 394652 12663 22.8 (1 at limit) 102.2 0.0273 (est.) –
narrow-python-library-inventory-scope-01 rf-python-library narrow Δ T−B -100 pp -394652 -12663 -21.8 -101.7 -0.0273
narrow-python-library-mode-literal-01 rf-python-library narrow treatment 3 0% 0% 0 0 0 1.0 0.5 0.0000 (est.) no
narrow-python-library-mode-literal-01 rf-python-library narrow baseline (stored) 6 100% 100% 0 219376 6140 15.0 (1 at limit) 48.0 0.0148 (est.) –
narrow-python-library-mode-literal-01 rf-python-library narrow Δ T−B -100 pp -219376 -6140 -14.0 -47.5 -0.0148
narrow-rf-injection-positive-01 plugin narrow treatment 3 33% 33% 0 79543 1827 5.7 16.9 0.0047 (est.) yes
narrow-rf-injection-positive-01 plugin narrow baseline (stored) 6 100% 100% 0 190699 3077 14.2 21.9 0.0150 (est.) –
narrow-rf-injection-positive-01 plugin narrow Δ T−B -67 pp -111156 -1250 -8.5 -5.0 -0.0103
narrow-rf-mcp-execute-01 plugin narrow treatment 3 33% 33% 0 120427 1288 7.0 13.4 0.0058 (est.) yes
narrow-rf-mcp-execute-01 plugin narrow baseline (stored) 6 100% 100% 0 205244 1904 10.8 21.5 0.0139 (est.) –
narrow-rf-mcp-execute-01 plugin narrow Δ T−B -67 pp -84817 -615 -3.8 -8.1 -0.0081
narrow-setup-requests-01 rf-setup narrow treatment 3 0% 0% 0 0 0 1.0 0.3 0.0000 (est.) no
narrow-setup-requests-01 rf-setup narrow baseline (stored) 6 100% 100% 0 127650 2196 11.7 23.4 0.0119 (est.) –
narrow-setup-requests-01 rf-setup narrow Δ T−B -100 pp -127650 -2196 -10.7 -23.1 -0.0119

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-language 5 0% 93% -93 pp -180374 -15.2 -0.0128
rf-python-library 3 0% 100% -100 pp -395371 -19.5 -0.0259
rf-setup 1 0% 100% -100 pp -127650 -10.7 -0.0119

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-language-bdd-01 loaded_rf_language_skill: 1/3
  • narrow-language-data-driven-01 loaded_rf_language_skill: 1/3
  • narrow-language-embedded-01 loaded_rf_language_skill: 0/3
  • narrow-language-env-varfiles-01 loaded_rf_language_skill: 0/3
  • narrow-language-suite-setup-tags-01 loaded_rf_language_skill: 0/3
  • narrow-non-rf-control-01 bounded_tool_calls: 0/3
  • narrow-non-rf-control-01 no_libdoc_invocation: 3/3
  • narrow-non-rf-control-01 no_rf_mcp_calls: 3/3
  • narrow-python-library-flaky-listener-01 loaded_rf_python_library_skill: 0/3
  • narrow-python-library-inventory-scope-01 loaded_rf_python_library_skill: 0/3
  • narrow-python-library-mode-literal-01 loaded_rf_python_library_skill: 0/3
  • narrow-rf-injection-positive-01 leveraged_rf_lookup: 1/3
  • narrow-rf-mcp-execute-01 rf_mcp_before_write: 1/3
  • narrow-rf-mcp-execute-01 rf_mcp_called_at_least_twice: 1/3
  • narrow-rf-mcp-execute-01 rf_mcp_had_successful_result: 1/3
  • narrow-setup-requests-01 used_uv_add: 0/3

Adversarial tier (reported only, never gates)

None.

@manykarim @claude
chore(eval): OpenRouter baselines for all plugins from run 43263363544
ea7ee26 
Re-baseline after the fixture, grader and hook changes of this PR
(N=6, both arms, all tiers, DeepSeek V4.1 Flash, turn factor 1.5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@manykarim