fix(rf-desktop): fix-desktop-eval-turn-overhead (OpenSpec change)

bundles/rf-agentskills/hooks/README.md

@@ -21,7 +21,7 @@ rather than written-and-broken.
 | Event | Script | Always fires? | Cost when no-op | Cost when active | Hook timeout |
 |---|---|---|---|---|---|
 | `SessionStart` | `scripts/check_rf_environment.mjs` | yes | — | ~0.5–1 s (import probes) | 15 s |
-| `PostToolUse` (matcher: `Write\|Edit`) | `scripts/validate_robot.mjs` | only on Write/Edit | ~30ms (file extension check) | ~1–1.5 s on a 500-line file once robotcode's cache is warm (`robotcode analyze code`, Robocop check and format run in parallel); the first run in a project ~4 s; +0.4 s or more with the opt-in dry run | 60 s |
+| `PostToolUse` (matcher: `Write\|Edit`) | `scripts/validate_robot.mjs` | only on Write/Edit | ~30ms (file extension check) | ~1–1.5 s on a 500-line file once robotcode's cache is warm (`robotcode analyze code` and Robocop check run in parallel, plus the formatter when enabled); the first run in a project ~4 s; +0.4 s or more with the opt-in dry run | 60 s |
 | `PostToolUse` (matcher: `Bash`) | `scripts/rf_error_hints.mjs` | only on Bash | ~30 ms (text scan) | ~30 ms | 10 s |
 | `UserPromptSubmit` | `scripts/maybe_inject_rf_context.mjs` | always invoked, conditional injection | ~30ms (regex over prompt) | same (≤ 450-character text) | 15 s |
 | `Stop` | `scripts/report_pending_errors.mjs` | always invoked, reports only files whose validation found errors or unresolved names that are still there | ~30 ms (reads the session record) | one `validate_robot.mjs` run per pending file | 60 s |
@@ -106,9 +106,12 @@ Fires from `PostToolUse` after a Write/Edit of a `.robot`/`.resource` file.
     gets no `VAR` hint), and so does the project's Robocop config.
   - If Robocop is missing, times out (10 s), or prints nothing parsable while
     failing, the tier is silent — a tool failure is never a finding.
-- **Tier 2 — formatting drift:** runs `robocop format --no-cache --check --diff`.
-  Purely informational — surfaces the proposed reformat as `additionalContext`
-  (exit 0). Formatting differences **never** cause exit 2.
+- **Tier 2 — formatting drift (opt-in):** with `RF_AGENTSKILLS_FORMAT_CHECK=1`,
+  runs `robocop format --no-cache --check --diff`. Purely informational —
+  surfaces the proposed reformat as `additionalContext` (exit 0). Formatting
+  differences **never** cause exit 2. Off by default: shown a diff, the agent
+  reformats and re-checks a file that is otherwise correct, which cost one to
+  three turns per written file in the evals.
 - **Opt-in per-file dry run:** with `RF_AGENTSKILLS_FILE_DRYRUN=1`, a written
   `.robot` suite (not `.resource`, not `__init__.robot`) is also run through
   `robot --dryrun --output NONE --report NONE --log NONE` from the project
@@ -128,6 +131,7 @@ exit-2 error diagnostic is written. Robocop is always called with
 |---|---|---|
 | `RF_AGENTSKILLS_DEPRECATION_CHECK` | `warn` (default), `off` | `off` drops all deprecation output. Any other value is treated as `warn`; no value makes a deprecation block (exit 2), and the variable never affects error findings. |
 | `RF_AGENTSKILLS_FILE_DRYRUN` | unset (default), `1`/`true`/`yes`/`on` | Enables the per-file dry run above. |
+| `RF_AGENTSKILLS_FORMAT_CHECK` | unset (default), `1`/`true`/`yes`/`on` | Enables the formatting-drift tier above. |
 | `RF_AGENTSKILLS_PROJECT_VALIDATION` | unset (default), `1`/`true`/`yes`/`on` | Enables the project-wide Stop tier below. |
 
 #### Opting out per rule
@@ -335,8 +339,12 @@ event produces no output; it always exits 0.
   or `"…something.resource"` (with bounded regex — `notes.robotic.md`
   does not count). Streaming chunking keeps memory bounded on
   long sessions.
+- Also notes the last Bash `"command"` that runs Robot Framework (`robot`,
+  `python -m robot`, `robotcode … robot`, dry runs included). If that run
+  comes after the last `.robot`/`.resource` file path, the session already
+  ran its suite and the hook exits without output.
 - On hit: emits `additionalContext` reminding the user to run
-  the affected tests (`uv run robotcode robot -bl "<longname>"`), read the
+  the affected tests where they can run (`uv run robotcode robot -bl "<longname>"`), read the
   results with `uv run robotcode results summary --failed`, and open
   `results/report.html`.
 - On miss: stdout stays empty.

bundles/rf-agentskills/scripts/maybe_remind_robot_tests.mjs

@@ -5,7 +5,9 @@
 // transcript for any tool result that wrote a .robot or .resource file,
 // and only emits an additionalContext reminder ("run robot, inspect
 // report.html") when at least one such file was touched. Sessions that
-// produced only non-RF artifacts get no injection.
+// produced only non-RF artifacts get no injection, and neither do sessions
+// that ran Robot Framework (`robot`, `python -m robot`, `robotcode robot`,
+// dry runs included) after their last .robot/.resource write.
 //
 // Cross-platform port of maybe_remind_robot_tests.sh. Reads the
 // transcript via streaming-ish chunking so very long sessions don't
@@ -55,13 +57,25 @@ if (sessionId) {
 // Pattern: `"file_path"  :  "...path.robot"` (or .resource). Matches the
 // `grep -E` regex from the bash original. Whitespace optional between
 // quotes and colon.
-const FILE_PATH_RE = /"file_path"\s*:\s*"[^"]*\.(?:robot|resource)"/i;
+const FILE_PATH_RE = /"file_path"\s*:\s*"[^"]*\.(?:robot|resource)"/gi;
+// A Bash tool call's `"command": "…"` (JSON string, escapes kept).
+const COMMAND_RE = /"command"\s*:\s*"((?:[^"\\]|\\.)*)"/g;
+// Robot Framework run at a command position: start, after ; & | ( or a
+// `uv/poetry/pipenv run`, optionally with a path (`.venv/bin/robot`).
+const RF_RUN_RE = new RegExp(
+  String.raw`(?:^|[;&|(]\s*|\b(?:uv|poetry|pipenv)\s+run\s+)(?:[\w.~/\\-]*[/\\])?` +
+    String.raw`(?:python3?(?:\.exe)?\s+-m\s+robot(?=\s|$)|robotcode(?:\.exe)?(?:\s+\S+)*?\s+robot\b|robot(?:\.exe)?(?=\s|$))`,
+);
+const CARRY = 4096;
 
-function transcriptMentionsRfFile(path) {
+// Offsets of the last .robot/.resource file_path and the last Robot
+// Framework run command in the transcript (-1 when absent).
+function lastWriteAndRun(path) {
   // The transcript is a JSONL file that can be tens of MB on long
   // sessions. Read in 64 KB chunks rather than slurping it all.
   const size = (() => { try { return statSync(path).size; } catch { return 0; } })();
-  if (!size) return false;
+  const found = { write: -1, run: -1 };
+  if (!size) return found;
   const BUF = Buffer.alloc(65536);
   const fd = openSync(path, "r");
   let pos = 0;
@@ -71,27 +85,33 @@ function transcriptMentionsRfFile(path) {
       const n = readSync(fd, BUF, 0, BUF.length, pos);
       if (n <= 0) break;
       // Stitch the boundary with the previous chunk so we don't miss a
-      // match split across the seam.
+      // match split across the seam (offsets are approximate, in chars).
       const chunk = carry + BUF.slice(0, n).toString("utf-8");
-      if (FILE_PATH_RE.test(chunk)) return true;
-      // Keep the last ~256 bytes as carry to bridge boundary matches.
-      carry = chunk.length > 256 ? chunk.slice(-256) : chunk;
+      const base = pos - carry.length;
+      for (const m of chunk.matchAll(FILE_PATH_RE)) found.write = Math.max(found.write, base + m.index);
+      for (const m of chunk.matchAll(COMMAND_RE)) {
+        if (RF_RUN_RE.test(m[1])) found.run = Math.max(found.run, base + m.index);
+      }
+      carry = chunk.length > CARRY ? chunk.slice(-CARRY) : chunk;
       pos += n;
     }
   } finally {
     try { closeSync(fd); } catch {}
   }
-  return false;
+  return found;
 }
 
-if (!transcriptMentionsRfFile(transcript)) process.exit(0);
+const { write: lastWrite, run: lastRun } = lastWriteAndRun(transcript);
+if (lastWrite < 0) process.exit(0);
+// The suite already ran (or dry-ran) after the last write: nothing to remind.
+if (lastRun > lastWrite) process.exit(0);
 
 const payload = {
   hookSpecificOutput: {
     hookEventName: "Stop",
     additionalContext: [
       "Robot Framework artifacts detected this session. Recommended next steps:",
-      '\n  1. Run the affected tests: uv run robotcode robot -bl "<Suite.Test>" (or a whole directory)',
+      '\n  1. Run the affected tests where they can run: uv run robotcode robot -bl "<Suite.Test>" (or a whole directory)',
       "\n  2. Read the results: uv run robotcode results summary --failed, then robotcode results log --failed",
       " (not using uv: run robotcode from the project's .venv; see rf-setup)",
       "\n  3. Open results/report.html for the rendered report.",

bundles/rf-agentskills/scripts/validate_robot.mjs

@@ -35,8 +35,10 @@
 //     any other value (including a legacy `block`) means `warn`.
 //     No parsable line from a failing Robocop run = tool failure = silence.
 //
-//   Tier 2 (formatting drift): `robocop format --check --diff`. Informational
-//     additionalContext only.
+//   Tier 2 (opt-in, RF_AGENTSKILLS_FORMAT_CHECK=1, formatting drift):
+//     `robocop format --check --diff`. Informational additionalContext only.
+//     Off by default: a formatting diff leads the agent to reformat and
+//     re-check a file that is otherwise correct.
 //
 //   Tier 3 (opt-in, RF_AGENTSKILLS_FILE_DRYRUN=1): `robot --dryrun` on the
 //     edited suite (.robot only, not __init__.robot). Advisory context only.
@@ -451,7 +453,7 @@ const advisory = [];
 const robocopPy = STOP_RECHECK ? null : findInterpreterWith("robocop", cwd);
 const analyzeRun = startAnalyze();
 const robocopRun = robocopPy ? startRobocop(robocopPy) : Promise.resolve(null);
-const formatRun = robocopPy
+const formatRun = robocopPy && isTruthy(process.env.RF_AGENTSKILLS_FORMAT_CHECK)
   ? spawnAsync(robocopPy, ["-m", "robocop", "format", "--no-cache", "--check", "--diff", "--no-overwrite", filePath],
     ROBOCOP_TIMEOUT_MS)
   : Promise.resolve(null);
@@ -482,7 +484,7 @@ if (robocopPy) {
       if (text) advisory.push(text);
     }
 
-    // ── Tier 2: formatting drift (informational only) ──
+    // ── Tier 2 (opt-in): formatting drift (informational only) ──
     const fmt = await formatRun;
     if (fmt && !fmt.error && fmt.status !== 0 && fmt.status !== null && (fmt.stdout ?? "").trim()) {
       let diff = fmt.stdout.trim();

bundles/rf-agentskills/skills/rf-platynui/SKILL.md

@@ -74,6 +74,8 @@ Library    PlatynUI.BareMetal    auto_activate=${True}    pointer_profile=${POIN
 
 Library scope is **SUITE**. The native runtime is created lazily on first keyword use (so `libdoc` and import work without a desktop).
 
+**Check a keyword** before you use it: `robotcode doc keyword PlatynUI.BareMetal "<Keyword>"` prints its arguments, and `robotcode doc keywords PlatynUI.BareMetal "<text>"` searches by name. Use them instead of parsing a libdoc file or reading the library source.
+
 ## Essential Concepts
 
 ### The normalized desktop UI model

docs/ci/desktop-live-check.md

@@ -0,0 +1,110 @@
+# rf-desktop live check
+
+The CI runner has no desktop, so the eval grades rf-desktop's PlatynUI suites statically: `keywords_resolve` checks that every keyword exists, and `platynui_locators` rejects locators known to fail on a real desktop. Neither runs a suite. The live check does: on a Windows desktop, it runs the suites one eval run wrote, and the skill's example, against the real Calculator, and reports how many pass per tier and arm.
+
+Run it by hand:
+
+- before a release of `rf-desktop`;
+- when the `sut-platynui` fixture README, the PlatynUI tasks or the locator guidance in `rf-platynui` change;
+- when the static and the live picture may differ, for example after a PlatynUI upgrade.
+
+It isn't part of CI, and it never writes baselines.
+
+## Set up the Windows machine
+
+You need Windows 10 or 11 with the Calculator app, a logged-in and unlocked desktop session, Python 3.12, uv and git.
+
+```bat
+git clone https://github.com/MarketSquare/robotframework-plugins C:\workspace\robotframework-plugins
+cd /d C:\workspace\robotframework-plugins
+uv venv --python 3.12 .venv-desktop
+uv pip install --python .venv-desktop --prerelease allow robotframework==7.5.1 robotframework-PlatynUI==0.12.0.dev330 platynui-cli==0.12.0.dev330
+```
+
+These are the versions of the committed libdoc spec (`eval/fixtures/sut-platynui/specs/`). Check that PlatynUI sees Calculator: start it, then run
+
+```bat
+.venv-desktop\Scripts\platynui-cli.exe query "//control:Text[@Id='CalculatorResults']"
+```
+
+It prints the result display twice (once through the desktop tree, once through the application tree). If it prints nothing, Calculator isn't ready yet; a cold start can take more than 4 seconds.
+
+## Get a run's artifacts
+
+Download the full-run artifact of an eval run on any machine with `gh`, and copy it to the Windows machine:
+
+```bash
+gh run download <run-id> -R MarketSquare/robotframework-plugins -n skill-eval-full-rf-desktop-<run-id> -D live-check/<run-id>
+```
+
+The artifacts hold model-written suites. Treat them as untrusted: keep them in their own directory and run the script with `python -I` from the repository, not from inside the artifact directory.
+
+## Run it
+
+List what would run first. This collects the `tests/calculator.robot` of every PlatynUI session, plus the skill's example, and screens each one:
+
+```bat
+.venv-desktop\Scripts\python.exe -I scripts\desktop-live-check.py C:\workspace\live-check\<run-id> --list
+```
+
+Screening skips a suite, with a reason, when it:
+
+- imports a library other than `PlatynUI.BareMetal`, `BuiltIn`, `Collections` or `String` (the skill's example may also import `Process`, to start Calculator);
+- imports a variable file, or a resource outside its session's workspace;
+- calls `Evaluate`, `Run Process`, `Start Process` (except the example), `Call Method` or an `Import …` keyword;
+- has text that could run Python in an expression, such as `${{ }}`, `__`, `import` or `subprocess`.
+
+Then run it in the desktop session:
+
+```bat
+.venv-desktop\Scripts\python.exe -I scripts\desktop-live-check.py C:\workspace\live-check\<run-id> --out C:\workspace\live-check\<run-id>-results
+```
+
+For each suite it closes Calculator, starts a fresh one (the example starts its own), waits until the result display answers a query, and runs `robot` with a 150-second limit (`--timeout`). Expect about half a minute per suite, since a locator that matches nothing waits 30 seconds before it fails; the 64 suites of 2026-11-12 took 32 minutes. Don't use the mouse or keyboard meanwhile: PlatynUI moves the real pointer.
+
+### Over SSH
+
+An SSH session can't see the desktop, so UI Automation finds nothing. Run the check through a scheduled task that runs in the logged-in session (`/it`). Create the task once, as the logged-in user:
+
+```bat
+schtasks /create /tn rf-desktop-live-check /tr C:\workspace\live-check\run.cmd /sc once /st 00:00 /sd 01.01.2030 /it /ru %USERNAME% /f
+```
+
+`C:\workspace\live-check\run.cmd` runs the check and marks the end of the output:
+
+```bat
+@echo off
+cd /d C:\workspace\robotframework-plugins
+.venv-desktop\Scripts\python.exe -I scripts\desktop-live-check.py C:\workspace\live-check\current --out C:\workspace\live-check\current-results > C:\workspace\live-check\out.txt 2>&1
+echo DONE>> C:\workspace\live-check\out.txt
+```
+
+Then, over SSH: copy the artifacts to `C:\workspace\live-check\current`, delete the old `out.txt`, start the task with `schtasks /run /tn rf-desktop-live-check`, and wait until `out.txt` ends with `DONE`. The `/sd` date only keeps the task from running by itself; use the date format of the machine's locale.
+
+## Read the report
+
+```
+display language: de_DE
+
+tier     arm        suites passed skipped
+example  example         1      1       0
+narrow   baseline        3      0       0
+narrow   treatment       5      1       0
+...
+
+most common failures:
+   15  ElementNotFoundError: No UiNode found for UiNodeDescriptor query Q within timeout of N seconds.
+```
+
+- **display language** matters: `@Name` values follow it (a German desktop shows `Rechner`, `Eins`, `Gleich`). A suite that finds controls by English names passes on an English desktop only. Compare runs on the same language, or note it next to the numbers.
+- **suites** counts the sessions that wrote the suite; a session that wrote nothing isn't listed.
+- **passed** counts suites whose every test passed against the real Calculator.
+- **most common failures** groups failure messages with quoted values and numbers replaced by `Q` and `N`.
+
+Each suite's `output.xml`, `log.html` and `report.html` are under `--out`, in a folder named after the tier and session.
+
+Put the per-arm pass counts next to the static ones (the candidate baseline's `keywords_resolve` and `platynui_locators` verdicts) in the PR that changes the fixture, the tasks or the skill. If a suite passes live and fails the static gate, or the other way round, look at it: either the static check needs a rule, or the live run hit something the eval can't see.
+
+## Example results
+
+On 2026-11-12, on German Windows 11 with PlatynUI 0.12.0.dev330, the 63 suites of runs 42887127109, 42993062842 and 43167186288 all failed as written: they used the old README's `@Name='Calculator'` window and `@AutomationId` locators. The skill's example passed. Those results led to the `@Id` locators in the fixture README and the `platynui_locators` check.

docs/ci/usage.md

@@ -543,6 +543,8 @@ to prefer which.
 
 - [local-testing.md](local-testing.md) — pre-push checklist and
   troubleshooting.
+- [desktop-live-check.md](desktop-live-check.md) — running rf-desktop's
+  eval suites against the real Calculator on Windows.
 - [faq.md](faq.md) — common questions, rate limits, rotation.
 - [architecture/ddd-design.md](architecture/ddd-design.md) — bounded
   contexts and module layout.

eval/fixtures/sut-platynui/README.md

@@ -11,10 +11,17 @@ The standard desktop calculator (Windows Calculator / GNOME Calculator).
 
 | Element | XPath (PlatynUI desktop tree) |
 |---|---|
-| Main window | `//control:Window[@Name='Calculator']` |
-| Digit buttons | `//control:Button[@Name='One']` … `//control:Button[@Name='Nine']` |
-| Plus / Equals | `//control:Button[@Name='Plus']`, `//control:Button[@Name='Equals']` |
-| Result display | `//control:Text[@AutomationId='CalculatorResults']` |
+| Main window | `//control:Window[@Name='Calculator' or @Name='Rechner']` |
+| Digit buttons | `//control:Button[@Id='num1Button']` … `//control:Button[@Id='num9Button']` |
+| Plus / Equals | `//control:Button[@Id='plusButton']`, `//control:Button[@Id='equalButton']` |
+| Result display | `//control:Text[@Id='CalculatorResults']` |
+
+PlatynUI exposes the UIA AutomationId as `@Id`, as `rf-platynui` teaches; a
+plain `@AutomationId` matches nothing. `@Name` values follow the display
+language (a German desktop shows `Rechner`, `Eins`, `Gleich`), so the window
+locator lists both names and the controls use `@Id`. The result display's
+`Name` holds the shown value, such as "Display is 3" (localised). Verified with
+`platynui-cli query` on Windows 11 on 2026-11-12.
 
 `tests/example.robot` is a stub that resolves cleanly. Real execution stays a
 manual, local activity on a desktop machine.

openspec/changes/fix-desktop-eval-turn-overhead/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-11-12