You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit debdd0e
Browse filesBrowse the repository at this point in the historyBrowse files
fix(desktop): accept common computer calls and let input reach attached sheets (#9093)
* fix(desktop): accept coordinate input without a window, one-axis scrolls and targeted keys
Mirrors the worker contract. Coordinate clicks, scrolls and drags without a windowId
target the window the snapshot selected; scroll deltas default to 0; press_key with an
elementId focuses that element first. Each shape was rejected before reaching the app
in benchmark runs (about one call per task).
* fix(desktop): let input reach sheets attached to the observed window
Go to Folder and out-of-process open and save panels are sheets: the app reports
them as its focused window, and a panel's contents report no window at all. Input
checks now treat a sheet as the window it is attached to, and a click on windowless
panel content is allowed only inside the app's focused sheet.
* fix(desktop): keep coordinate input on the selected window and confirm targeted focus
- Coordinate clicks and scrolls without a windowId use the snapshot's selected
window through every input check; element targets keep their own window
- press_key with an elementId sends the key only once that element holds focus
- The service observes the selected window after a call without one
- A scroll needs at least one delta, matching the worker contract
* fix(desktop): observe the focused window after an untargeted key
guard snapshot.windowFrames[windowId]== frame else{throwComputerError("stale_window","Window moved or changed; observe it again.")}
508
542
letx=tryrequired(end ? p.toX : p.x, end ?"toX":"x"); lety=tryrequired(end ? p.toY : p.y, end ?"toY":"y")
@@ -579,6 +613,7 @@ final class Driver {
579
613
}
580
614
guardlet selectedWindowId = p.windowId ?? focusedID ?? windows.first?["windowId"]as?Stringelse{throwComputerError("app_window_unavailable","No accessible app window is available. Open a window, then observe the app again.")}
581
615
guard snapshot.windowFrames[selectedWindowId]!=nilelse{throwComputerError("window_unavailable","Selected window does not belong to the app.")}
throwComputerError("stale_window","The editor's window moved or is not an observed window; observe again.")
682
706
}
683
-
guardlet current =attribute(root, kAXFocusedUIElementAttribute),holdsFocus(current)else{throwComputerError("focus_changed","The exact editor is no longer focused; remaining input was not dispatched. Observe again.")}
707
+
guardlet current =attribute(root, kAXFocusedUIElementAttribute),holdsFocus(current, element)else{throwComputerError("focus_changed","The exact editor is no longer focused; remaining input was not dispatched. Observe again.")}
guard focused else{throwComputerError("focus_changed","The element did not take keyboard focus; no key was sent. Observe again.")}
771
+
}
733
772
// Keys go to the app's key window: the observed window, or a dialog, panel or popover above it,
734
773
// but never another document window the snapshot did not observe.
735
774
guardlet rawWindow =attribute(root, kAXFocusedWindowAttribute),CFGetTypeID(rawWindow)==AXUIElementGetTypeID()else{throwComputerError("focus_failed","The app has no focused window to receive keys; click a window or call activate_app, then observe again.")}
The helper observes up to 2000 accessibility nodes (depth 64, queue 4000, traversal deadline 8 seconds, each AX call timeout 250ms). Selected-window and focused-element ancestry are prioritized; menu traversal is deferred and capped at 80 nodes. Electron accessibility is enabled using its documented `AXManualAccessibility` attribute. The helper observes 100 windows and 1000 apps. Requests are bounded to 128 KiB during ingestion. Text/value input is capped at 32000 UTF-16 units. State references are per-process, per-app, expire after 60 seconds, and are consumed by every action that reaches the app; a request that fails before sending anything reports `dispatchState: "not_started"` and keeps them valid. Restarts invalidate all references. Screen captures exclude window shadows, preserve the selected window’s coordinate geometry, and are capped to 1600 pixels on the longest side and 8 MiB encoded. Complete JSON replies are capped at 16 MiB: oversized observations omit screenshots first, then truncate accessibility nodes, while preserving action outcomes. Screenshots are optional and failure is reported in `screenshotError` without discarding accessibility state. macOS 26 uses the screenshot-specific capture API; the macOS 14/15 API path includes only the selected window in a display filter and crops its exact frame. That older path requires the window to fit within one display and reports `capture_geometry_unavailable` otherwise.
12
12
13
-
Coordinates in action inputs are window-local points; node frames are global screen points. Window bounds must still match the observed snapshot. App PID and launch time must match. Secure fields and descendants are unavailable as action targets; secure values and labels are suppressed. Sim's own apps and System Settings are protected targets. `get_app_state` can open an installed bundle ID without activating it. `list_apps` discovers installed apps under `/Applications`, `/System/Applications`, and `~/Applications` (bounded to 5000 filesystem entries and depth 3), merging running regular apps; installed apps omit `pid`.
13
+
Coordinates in action inputs are window-local points; node frames are global screen points. Coordinate input without a `windowId` targets the window the snapshot selected; `press_key` with an `elementId` focuses that element first. Window bounds must still match the observed snapshot. App PID and launch time must match. Secure fields and descendants are unavailable as action targets; secure values and labels are suppressed. Sim's own apps and System Settings are protected targets. `get_app_state` can open an installed bundle ID without activating it. `list_apps` discovers installed apps under `/Applications`, `/System/Applications`, and `~/Applications` (bounded to 5000 filesystem entries and depth 3), merging running regular apps; installed apps omit `pid`.
14
14
15
-
AXPress and AXSetValue are direct semantic operations. Keyboard events target the app PID. Cmd+A uses the focused editor's writable accessibility selection range where supported. Coordinate mouse input requires the target app in front; `activate_app` makes that focus change explicit and confirms it. Click/drag/scroll validate the foreground process, unchanged window geometry and the app that receives the point; the topmost window-list entry is not the click target because Notification Center and the Dock keep full-screen click-through windows. A press that times out while the app opens a sheet or modal (`cannotComplete`) counts as dispatched. A menu-item press that leaves its menu open with the item highlighted (Calendar on macOS 26) is completed with Return. Open menus and their submenus are observed first, and menu item labels carry their shortcuts (for example ⇧⌘N). Some apps ignore background keyboard events. `dispatched:true,verified:false` means only dispatch succeeded; the agent must observe again to establish the outcome. No foreground activation fallback is hidden inside the helper. Native key codes currently assume the US layout for shortcuts; literal text uses Unicode events. Unsupported shortcuts fail explicitly.
15
+
AXPress and AXSetValue are direct semantic operations. Keyboard events target the app PID. Cmd+A uses the focused editor's writable accessibility selection range where supported. Coordinate mouse input requires the target app in front; `activate_app` makes that focus change explicit and confirms it. Click/drag/scroll validate the foreground process, unchanged window geometry and the app that receives the point; the topmost window-list entry is not the click target because Notification Center and the Dock keep full-screen click-through windows. A press that times out while the app opens a sheet or modal (`cannotComplete`) counts as dispatched. A menu-item press that leaves its menu open with the item highlighted (Calendar on macOS 26) is completed with Return. Sheets, including Go to Folder and out-of-process open and save panels whose contents report no window, count as the window they are attached to. Open menus and their submenus are observed first, and menu item labels carry their shortcuts (for example ⇧⌘N). Some apps ignore background keyboard events. `dispatched:true,verified:false` means only dispatch succeeded; the agent must observe again to establish the outcome. No foreground activation fallback is hidden inside the helper. Native key codes currently assume the US layout for shortcuts; literal text uses Unicode events. Unsupported shortcuts fail explicitly.
16
16
17
17
`type_text` and `input_sequence` target any observed element that can take keyboard focus (text fields, terminals, combo boxes, web editors). `input_sequence` executes 1–32 key/text steps against one such element. It validates every step before dispatch and checks the same editor, window, and secure-input ancestry between steps. Enter can continue into another step when the same editor and window retain focus. Actual focus loss, Tab, or navigation shortcuts end the sequence before later input. Explicit `activateFirst: true` activates the target app inside the batch before input; it never enables automatic refocusing between steps. Partial execution reports completed steps and an error; callers must observe before retrying. Every mutation is followed by fresh state under the same authorization and queue entry; `observeAfter` optionally requests a screenshot. Observation failure is reported separately so the mutation is never replayed. The model receives a compact, bounded projection of the native snapshot; omitted nodes are reported explicitly.
0 commit comments