feat(cua-driver): add target-scoped post-action rediscovery

[injaneity]

why

#2238 describes a stale-target failure: an action starts against window a, opens a sheet, dialog, popover, or Open/Save panel b, and leaves the caller bound to a. later snapshots and actions then target the blocked parent instead of the new interaction surface.

cua-driver already noticed some new macos windows, but its global detector could not prove that a desktop change belonged to the action. it also kept the result in diagnostic prose, so callers received neither a stable window address nor a machine-readable rebind instruction.

this pr resolves the structured-rediscovery part of the issue. after an action opens a target-owned surface, the result identifies the observed candidates and recommends rebind. when exactly one candidate is owner-verified, the result includes its (pid, window_id) so the caller can take a fresh window-scoped snapshot without activating or raising it.

design

retained from cua-driver

this keeps cua-driver's native action routes, focus-suppression leases, shared ActionExecutionRecord, closed ActionResult, and harness-owned escalation policy. it also retains the useful contract work from #2746: typed window_change metadata, an explicit rebind escalation, and generated rust, python, typescript, and manifest bindings.

it removes the unsafe parts of the old path: the desktop-global WindowChangeDetector, seven per-tool diagnostic suffix mutations, the dead observation-skip hook, topology transport through legacy json, and window changes as evidence that an actuator succeeded.

adapted from pi-computer-use

this adapts the target-root observation model from injaneity/pi-computer-use commit 022a280a377065c95736cc15f684bf1fad46479e:

• snapshot the addressed process's accessibility roots before dispatch;
• run the native actuator;
• use a cheap target-window signature only to end a 400 ms wait early;
• run the authoritative accessibility diff once, with bounded catch-up when the signal beats the tree; and
• rebind directly when the AX diff yields one owner-verified logical root, and fail closed when several roots appear.

the persistent ax-observer ring is intentionally not copied because the source commit established that sheet creation emits no reliable notification; it adds lifecycle state without improving correctness here. cua-driver instead merges AXSheets with AXChildren, keeps its focus-suppression lease across dispatch and observation, and resolves owners only after the authoritative ax diff. unresolved appeared roots retry WindowServer ownership independently; an incomplete owner set can return known candidates but never an exact target.

combined result

macos action decorator
  → target-process ax snapshot
  → native actuator
  → target-process ax diff
  → windowserver ownership proof
  → shared candidate resolver
  → typed action record
  → closed action result

observation and action accounting remain separate. a surface change can add window_change and escalation, but it cannot promote effect to confirmed; confirmation still requires value readback. ambiguous candidates are returned without an invented exact target and can be correlated with one fresh list_windows call.

issue coverage

this pr covers the structured rebind rung from #2238:

• return stable candidate pids, window ids, titles, app names, and foreground-change state;
• recommend rebind, with an exact target only when ownership and modal/focused state make it unambiguous;
• preserve the human's foreground app, z-order, cursor position, and input isolation; and
• add the canonical TextEdit Open-panel case, then re-address the returned panel window.

it does not implement the issue's later privacy-sensitive desktop-frame fallback or inert-resnapshot backstop. those remain separate because this structured path recovers without widening capture scope. this pr therefore uses Refs, not an issue-closing keyword.

diff shape

the runtime production diff is +899/-829 (net +70). the new 411-line observer replaces the deleted 530-line global detector, while affected macos tool files remove 203 lines and add 79. most apparent growth is required generated contract output: +1103/-157 across the manifest and python/typescript bindings.

logical-root correction

host investigation found that AppKit can expose two Open-panel WindowServer records: an AX-addressable same-process TextEdit proxy and an AX-empty XPC duplicate. pi-computer-use keeps the target-scoped AX identity rather than requiring the foreign service pid. this branch now does the same: it preserves the appeared AX root's exact WindowServer identity, removes internal modal/focus eligibility state, and rebinds whenever exactly one owner-verified logical root appears. ambiguity still omits the exact target.

on the host probe, the returned TextEdit proxy exposed 62 Open-panel AX elements; the XPC duplicate exposed none. a direct exact-head action returned escalation.target: rebind, the proxy (pid, window_id), and a successful fresh scoped snapshot without changing the foreground app.

compatibility and validation

macos emits the new optional topology. windows and linux emit none rather than substituting a global heuristic. generated sdk consumers gain optional window_change plus the rebind and surface_changed enum values.

local validation passed:

• cargo check --workspace --all-targets
• cargo test -p cua-driver-contract --lib — 33 passed
• cargo test -p cua-driver-core --lib — 587 passed at rebased head f5ea38210932db1389c199bc768891a21b9782e0
• cargo test -p platform-macos --lib — 338 passed, 2 ignored at the same sha
• pre-rebase direct host TextEdit Open-panel probe at a2d513b93ac6122ad4dd8639e3e1ec7c88bca5cf — one typed rebind to an AX-addressable logical root with 62 elements and unchanged foreground
• pre-rebase complete local macos contributor matrix on an integration of a2d513b93ac6122ad4dd8639e3e1ec7c88bca5cf with #2907 — 140 of 158 cells passed; macos-textedit-open-panel-background-rebind passed all five declared oracles. the 18 unrelated failures were host input-isolation, cursor, desktop-coordinate, and video-finalization failures, so this is diagnostic evidence rather than canonical acceptance
• contract and uniffi generation checks
• typescript build and python binding compile check
• rustfmt, shell syntax, and diff checks

before this pull request is ready, the final candidate still needs the canonical macos Lume matrix. the local host can run the complete matrix when the foreground-sentinel fix in #2907 is included, but its active desktop is not an acceptance environment: cursor movement, global-coordinate delivery, and concurrent recordings caused unrelated failures. maintainers must also confirm whether this additive public contract requires a separate rfc.

Salvaged from #2746.

Refs #2238.

stacked scheduling follow-up

The stacked draft #3391 replaces duplicate native action admission with one scheduler and canonical lease plan, ensuring this pull request's before/action/after decorator remains inside the same desktop action lease. It does not add another observation owner or change the public tool contract.

canonical macos lume evidence — 2026-09-04

• tested source: b131ca4f8e22650a8ed992df228f659b93eb9640 (the current PR head, not a merge commit).
• command: libs/cua-driver/tests/runners/macos-lume/run-all.sh, full unfiltered matrix; no retry or single-cell selector.
• run id: 20260904T181935Z-11114.
• environment: disposable public macos-tahoe-cua:26.5.2 worker, macOS 26.5.2 build 25F84, SIP disabled, stable local certificate, app-owned grants seeded with the repository helper, and a logged-in guest Terminal session.
• results: 158/158 passed, including 150 delivered actions and 8 expected refusals; 0 failed and 0 skipped.
• macos-textedit-open-panel-background-rebind: passed AX state, focus, z-order, cursor, and no-leaked-input checks.
• failures.json: failure_count=0, preflight_failed=false, report_failed=false, no failed lanes or video failures.
• daemon verified restored to standard mode; worker stopped and retained.
• evidence retrieved to the maintainer host: ~/workspaces/cua-pr-3373-b131ca4f/artifacts/cua-driver/vm/lume_192.168.64.2/20260904T184538Z/ (local evidence, not a public download link).
• standalone installed Chrome/Edge matrix was not run.

This certifies only the SHA above. The requested rebase and review fixes remain outstanding; this evidence must not be presented as certification of a later candidate.

[injaneity]

successfully ran the lume e2e test

[f-trycua]

@injaneity thanks — the target-scoped structure here is a real improvement over the global detector. I did a blast-radius pass; four questions gate merge for me, two are nice-to-have, and the items at the end are maintainer-side so I'll resolve them myself.

Merge-gating

1. Lume matrix evidence. Re "successfully ran the lume e2e test": which exact SHA did that certify, and was it the full macos-lume/run-all.sh matrix or just the new background_open_panel_returns_a_typed_rebind row? Please record the tested SHA and row results in the PR description. The branch is also a few commits behind main now — after rebasing, the certified SHA needs to match the final candidate.

2. Consumers of the removed surface.

• _skip_window_change_detection is deleted; its comment said embedded interactive clients used it to skip observation entirely. Is that embedding path still live anywhere (pi-computer-use, an interactive host)? If so, those clients silently start paying up to 400 ms per input event.
• ActionEvidenceKind::WindowChange removal: any known consumer matching on it, or persisted payloads carrying "kind": "window_change"? Deserializing those now fails under deny_unknown_fields.

3. Focus-suppression divergence — intentional? delivery_mode: "foreground" without window_id previously got SuppressTarget suppression; the decorator now decides on delivery_mode alone, so that case gets none (decorator check in post_action_observer.rs vs fg = is_foreground() && window_id.is_some() in click.rs). Deliberate, or a side effect of centralizing? Either way a focused test or comment would pin it.

4. Ordering vs #3391. If this lands and #3391 lags, can the decorator's suppression lease and the existing admission path double-lease in the interim? If yes, what's the observable failure mode, and is it benign?

Nice-to-have

5. Windows/Linux follow-through. Does #2238 stay open to track equivalent observers on Windows (modal dialogs) and Linux (GTK/Qt file choosers)? A cheap per-platform test asserting window_change stays absent would also cover the cross-platform coverage expectation in AGENTS.md.

6. Why 400 ms? Tuned empirically on the host probe, or inherited from pi-computer-use? A one-line comment on OBSERVATION_TIMEOUT would help future tuning.

Maintainer-side (I'll handle these)

• Title: this isn't purely additive — ActionEvidenceKind::WindowChange leaves the public contract — so I plan to retitle to feat(cua-driver)!: ... unless question 2 turns up a reason to keep the variant instead.
• RFC: the description asks maintainers to rule on whether the contract change needs one; I'll answer in-thread once question 2 is settled.
• Re-review: my earlier approval was pinned to bf85705, head is now b131ca4 — I'll re-review the delta (mainly the logical-root correction) after the rebase.

[injaneity]

thanks for the detailed review. i’m taking the follow-up here in this pr, not in a separate implementation.

i added the exact lume SHA, full-matrix command, 158 passing rows, refusal counts, environment, and local evidence location to the description. the run was the full unfiltered matrix, not just the new TextEdit row. it certifies b131ca4f8e22650a8ed992df228f659b93eb9640 only.

i’m treating the removed embedded opt-out, old evidence payload compatibility, foreground-without-window suppression, and interim lease ordering as unresolved until checked and covered. the current repository search finds the old opt-out helper/test but no caller on main; that alone does not establish that external embedding clients are unused. i’ll avoid claiming that it is safe to remove solely on that search.

the requested rebase and any runtime/contract corrections will require a new exact-candidate run. the earlier approval and old certification should not be used to waive your re-review. #2238 remains open; this pr uses Refs rather than a closing keyword.

[f-trycua]

I am withdrawing my earlier approval because the current diff has two release blockers:

1. It removes the public UniFFI ActionEvidenceKind::WindowChange enum variant while the PR is titled as an additive non-breaking feature. Existing generated SDK consumers can depend on that variant. Preserve backward compatibility or explicitly treat and title this as a breaking release with a migration path.
2. Every affected macOS action now performs a post-action observation window with a 400 ms wait plus catch-up reads. That is a broad latency/behavior change across click, drag, key, text, scroll, value, and zoom, not a narrow topology annotation. The old 158/158 Lume evidence predates the requested rebase, and the branch now conflicts with main; it cannot certify the current candidate.

Please rebase without dropping landed contributor work, settle the public contract/release semantics, measure action latency before and after on representative fast and no-new-window paths, and rerun the full canonical macOS matrix at the final exact SHA. The typed rebind direction is useful, but the present conflicted and breaking candidate is not ready.

[f-trycua]

I am withdrawing my earlier approval because the current diff has two release blockers:

1. It removes the public UniFFI ActionEvidenceKind::WindowChange enum variant while the PR is titled as an additive non-breaking feature. Existing generated SDK consumers can depend on that variant. Preserve backward compatibility or explicitly treat and title this as a breaking release with a migration path.
2. Every affected macOS action now performs a post-action observation window with a 400 ms wait plus catch-up reads. That is a broad latency/behavior change across click, drag, key, text, scroll, value, and zoom, not a narrow topology annotation. The old 158/158 Lume evidence predates the requested rebase, and the branch now conflicts with main; it cannot certify the current candidate.

Please rebase without dropping landed contributor work, settle the public contract/release semantics, measure action latency before and after on representative fast and no-new-window paths, and rerun the full canonical macOS matrix at the final exact SHA. The typed rebind direction is useful, but the present conflicted and breaking candidate is not ready.

[kvnloo]

I think there is one source-level contract bug beyond the already-recorded release/latency blockers: a surface rebind currently hides the actuator's own escalation.

The internal API explicitly says:

/// Attach observer facts without changing the actuator's effect or
/// overwriting its internal escalation account.
pub fn observe_surface_delta(&mut self, delta: ActionSurfaceDelta) {
    self.surface_delta = Some(delta);
}

But public_result() projects escalation as:

escalation: projection
    .surface_delta
    .as_ref()
    .map(|delta| ActionEscalation {
        target: Rebind,
        reason: SurfaceChanged,
        ...
    })
    .or_else(|| projection.escalation.map(...))

So whenever a surface delta exists, the Rebind advice wins and the actuator escalation is dropped from the public result.

That can lose information from a completely different axis. For example:

• an actuator returns suspected_noop with a pixel/foreground retry recommendation;
• the same call also observes one new target-owned sheet/window;
• the caller receives only rebind, not the actuator's delivery/effect recovery advice.

The action effect itself remains truthful, but the recovery plan no longer is. This seems especially risky because topology and actuation are deliberately independent in this PR: “a new surface appeared” does not prove that rebinding is the only or highest-priority next step.

I would keep those facts structurally separate rather than choose one silently. The smallest contract shape may be to put the exact optional rebind target on window_change itself (for example window_change.rebind) and leave ActionResult.escalation owned by the actuator. If the public contract must retain escalation.target = rebind, then it needs an explicit precedence rule and a way not to discard an existing actuator escalation; a singular field cannot faithfully represent two independent recommendations.

A focused regression can construct:

1. ActionEffect::SuspectedNoop with an existing RetryWithPixelTarget escalation;
2. attach one exact surface_delta.rebind;
3. project the public result;
4. require the actuator recovery information to remain observable and the topology/rebind fact to remain observable.

This is the recovery lesson I'd preserve here: fresh topology tells the caller where to observe next; it must not erase what the actuator already proved about how the previous action failed.