Skip to content

feat(extension): answer remote gateway UI requests outside the tool queue - #296

Open
lyingbug wants to merge 1 commit into
Tencent:mainfrom
lyingbug:feat/ui-task-preview
Open

lyingbug wants to merge 1 commit into
Tencent:mainfrom
lyingbug:feat/ui-task-preview

Conversation

@lyingbug

Copy link
Copy Markdown
Contributor

Problem

A gateway that runs browser tasks for a user renders its own view of the running task: a small preview of the page the agent is working on, and a control that brings that tab to the front so the user can take over.

Neither can be built on the tool RPCs. ToolQueues::dispatch serializes work per session, so a tool.screenshot issued for a preview waits behind whatever the agent is doing; while request_help waits for the user, it waits for the whole human-help timeout. The result is that the preview goes dark exactly when the user most wants to look — during a long navigation, and while the agent is asking for help.

The DSH plugin's own PiP overlay works around this from the other side: ObservationService.acquireForeground cancels the pending thumbnail and kills an active bsk screenshot as soon as model-facing work starts, and only schedules a fresh frame after it ends.

Change

Two optional requests are answered on the socket before WSTransport sees the frame, so they never enter the tool dispatcher:

{"id":"ui-1","method":"ui.task_preview","params":{"session_id":""}}
{"id":"ui-2","method":"ui.task_focus","params":{"session_id":""}}
  • ui.task_preview returns image_base64, format, tab_id, title and captured_at.
  • ui.task_focus activates the task's own tab and raises its window.

Neither starts a session, and the channel is attached only to authenticated remote sockets — an extension connected to a local daemon answers unknown_method as before, so a gateway can feature-detect.

Constraints kept deliberately tight:

  • Only a tab the task owns is captured or focused. Window membership alone never qualifies, so an unowned tab inside the Agent Window is skipped and another owned tab is used instead.
  • Authorization, the document revision and the debugger attachment identity are re-checked after the capture and after the rescale; a frame that outlived its authorization is discarded rather than returned.
  • The preview acquires the same explicit background-execution claim the tools use before reading a page that is not in the foreground.
  • Captures are coalesced per task and at most one runs per tab. A local deadline cannot cancel a CDP command, so the fence is released only when Chrome answers; a poll that arrives meanwhile is refused instead of queueing another capture behind a stuck one. The caller's wait is bounded at 3s.
  • UI frames keep the extension's control and help overlays visible: hiding and restoring them on every poll makes them flicker in the user's browser. Tool screenshots still suppress them for unobstructed page content.
  • session.stop drains an in-flight capture before it detaches the debugger and closes tabs. No session state, turn-end detach or extra lifecycle callback is introduced.
  • Frames are bounded to 640 pixels wide, encoded in the extension, because a viewport capture on a HiDPI display comes back several times larger than a preview needs.

docs/remote-extension-connection.md documents the contract and drops the sentence that said there is no preview or focus side protocol.

Validation

  • pnpm --filter @browser-skill/extension test: 1874 passed, 103 skipped. 12 of those are new, covering the channel (routing, invalid params, failure reporting, late replies on a closed socket) and the preview (overlay retention, coalescing, the HiDPI bound, the in-flight refusal and its recovery, stop draining a stuck capture, a failed stop restoring previews, unowned tabs, and discarding frames after authorization or attachment changes).
  • pnpm --filter @browser-skill/extension compile passes.
  • Not exercised against a live browser in this branch.

Notes

The names are a proposal. If you would rather namespace these differently, or serve them from the daemon's remote server instead of the extension socket, say which shape you want and I will rework it.

This is the capability WeKnora has been carrying downstream; the version here is self-contained and free of anything specific to that deployment.

…ueue

A gateway that runs tasks for a user renders its own view of the running task:
a preview of the page the agent is working on, and a control that brings that
tab to the front. Neither can be served by the tool RPCs, because those are
serialized per session, so both go dark exactly when the user wants to look —
during a long navigation, or while request_help waits for an answer.

Answer two optional requests, ui.task_preview and ui.task_focus, on the socket
before the transport sees them. They never enter the tool dispatcher, never
start a session, and exist only on authenticated remote sockets.

The preview captures the task's own tab, keeps the control and help overlays
visible so a periodic poll does not make them flicker, and returns a JPEG frame
bounded to 640 pixels wide. Captures are coalesced per task, at most one runs
per tab, a poll that arrives while Chrome still holds one is refused rather
than queued behind it, and authorization, the document revision and the
debugger identity are re-checked before a frame is returned. session.stop
drains an in-flight capture before it detaches the debugger and closes tabs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant