fix(desktop): let input reach sheets attached to the observed window

Go to Folder and out-of-process open and save panels are sheets: the app reports
them as its focused window, and a panel's contents report no window at all. Input
checks now treat a sheet as the window it is attached to, and a click on windowless
panel content is allowed only inside the app's focused sheet.

Author: waleedlatif1

apps/desktop/native/computer-use/ComputerUse.swift

@@ -425,14 +425,34 @@ func requireForegroundTarget(pid: pid_t, windowID: String, window: AXUIElement,
             throw ComputerError("window_occluded", "Another of this app's windows covers the target point; observe that window and target it instead.")
         }
     } else if ![kAXMenuRole, kAXMenuItemRole, kAXMenuBarRole, kAXMenuBarItemRole].contains(attribute(hit, kAXRoleAttribute) as? String ?? "") {
-        throw ComputerError("window_occluded", "Cannot confirm which window receives this point; observe again before retrying.")
+        // Out-of-process panels (open and save) expose their contents without a window; they are
+        // reachable only inside the app's focused sheet, which must belong to the observed window.
+        let root = AXUIElementCreateApplication(pid)
+        guard let rawSheet = attribute(root, kAXFocusedWindowAttribute), CFGetTypeID(rawSheet) == AXUIElementGetTypeID(),
+              case let sheet = unsafeBitCast(rawSheet, to: AXUIElement.self), attribute(sheet, kAXRoleAttribute) as? String == kAXSheetRole,
+              elementRect(sheet)?.contains(point) == true, acceptsInput(sheet, observed: [(window, frame)]) else {
+            throw ComputerError("window_occluded", "Cannot confirm which window receives this point; observe again before retrying.")
+        }
+    }
+}
+/// The window a sheet is attached to: its parent, or for an out-of-process panel that reports no
+/// parent, the window it reports as its own.
+func sheetOwner(_ sheet: AXUIElement) -> AXUIElement? {
+    for name in [kAXParentAttribute, kAXWindowAttribute] {
+        if let raw = attribute(sheet, name), CFGetTypeID(raw) == AXUIElementGetTypeID() {
+            let owner = unsafeBitCast(raw, to: AXUIElement.self)
+            if attribute(owner, kAXRoleAttribute) as? String == kAXWindowRole { return owner }
+        }
     }
+    return nil
 }
 /// Whether input aimed at an observed window may go to `window`: the observed window itself, or one of
 /// the app's dialogs or floating panels above it. Another document window, or one whose kind cannot be
 /// read, is refused.
 func acceptsInput(_ window: AXUIElement, observed: [(element: AXUIElement, frame: CGRect)]) -> Bool {
     if observed.contains(where: { CFEqual($0.element, window) && elementRect(window) == $0.frame }) { return true }
+    // A sheet (Go to Folder, an open or save panel) belongs to the window it is attached to.
+    if attribute(window, kAXRoleAttribute) as? String == kAXSheetRole { return sheetOwner(window).map { acceptsInput($0, observed: observed) } ?? false }
     let auxiliary: Set<String> = [kAXDialogSubrole, kAXSystemDialogSubrole, kAXFloatingWindowSubrole, kAXSystemFloatingWindowSubrole]
     return (attribute(window, kAXSubroleAttribute) as? String).map(auxiliary.contains) ?? false
 }

apps/desktop/native/computer-use/README.md

@@ -12,7 +12,7 @@ The helper observes up to 2000 accessibility nodes (depth 64, queue 4000, traver
 
 Coordinates in action inputs are window-local points; node frames are global screen points. Coordinate input without a `windowId` targets the window the snapshot selected; `press_key` with an `elementId` focuses that element first. Window bounds must still match the observed snapshot. App PID and launch time must match. Secure fields and descendants are unavailable as action targets; secure values and labels are suppressed. Sim's own apps and System Settings are protected targets. `get_app_state` can open an installed bundle ID without activating it. `list_apps` discovers installed apps under `/Applications`, `/System/Applications`, and `~/Applications` (bounded to 5000 filesystem entries and depth 3), merging running regular apps; installed apps omit `pid`.
 
-AXPress and AXSetValue are direct semantic operations. Keyboard events target the app PID. Cmd+A uses the focused editor's writable accessibility selection range where supported. Coordinate mouse input requires the target app in front; `activate_app` makes that focus change explicit and confirms it. Click/drag/scroll validate the foreground process, unchanged window geometry and the app that receives the point; the topmost window-list entry is not the click target because Notification Center and the Dock keep full-screen click-through windows. A press that times out while the app opens a sheet or modal (`cannotComplete`) counts as dispatched. A menu-item press that leaves its menu open with the item highlighted (Calendar on macOS 26) is completed with Return. Open menus and their submenus are observed first, and menu item labels carry their shortcuts (for example ⇧⌘N). Some apps ignore background keyboard events. `dispatched:true,verified:false` means only dispatch succeeded; the agent must observe again to establish the outcome. No foreground activation fallback is hidden inside the helper. Native key codes currently assume the US layout for shortcuts; literal text uses Unicode events. Unsupported shortcuts fail explicitly.
+AXPress and AXSetValue are direct semantic operations. Keyboard events target the app PID. Cmd+A uses the focused editor's writable accessibility selection range where supported. Coordinate mouse input requires the target app in front; `activate_app` makes that focus change explicit and confirms it. Click/drag/scroll validate the foreground process, unchanged window geometry and the app that receives the point; the topmost window-list entry is not the click target because Notification Center and the Dock keep full-screen click-through windows. A press that times out while the app opens a sheet or modal (`cannotComplete`) counts as dispatched. A menu-item press that leaves its menu open with the item highlighted (Calendar on macOS 26) is completed with Return. Sheets, including Go to Folder and out-of-process open and save panels whose contents report no window, count as the window they are attached to. Open menus and their submenus are observed first, and menu item labels carry their shortcuts (for example ⇧⌘N). Some apps ignore background keyboard events. `dispatched:true,verified:false` means only dispatch succeeded; the agent must observe again to establish the outcome. No foreground activation fallback is hidden inside the helper. Native key codes currently assume the US layout for shortcuts; literal text uses Unicode events. Unsupported shortcuts fail explicitly.
 
 `type_text` and `input_sequence` target any observed element that can take keyboard focus (text fields, terminals, combo boxes, web editors). `input_sequence` executes 1–32 key/text steps against one such element. It validates every step before dispatch and checks the same editor, window, and secure-input ancestry between steps. Enter can continue into another step when the same editor and window retain focus. Actual focus loss, Tab, or navigation shortcuts end the sequence before later input. Explicit `activateFirst: true` activates the target app inside the batch before input; it never enables automatic refocusing between steps. Partial execution reports completed steps and an error; callers must observe before retrying. Every mutation is followed by fresh state under the same authorization and queue entry; `observeAfter` optionally requests a screenshot. Observation failure is reported separately so the mutation is never replayed. The model receives a compact, bounded projection of the native snapshot; omitted nodes are reported explicitly.