Conversation
…ueue A gateway that runs tasks for a user renders its own view of the running task: a preview of the page the agent is working on, and a control that brings that tab to the front. Neither can be served by the tool RPCs, because those are serialized per session, so both go dark exactly when the user wants to look — during a long navigation, or while request_help waits for an answer. Answer two optional requests, ui.task_preview and ui.task_focus, on the socket before the transport sees them. They never enter the tool dispatcher, never start a session, and exist only on authenticated remote sockets. The preview captures the task's own tab, keeps the control and help overlays visible so a periodic poll does not make them flicker, and returns a JPEG frame bounded to 640 pixels wide. Captures are coalesced per task, at most one runs per tab, a poll that arrives while Chrome still holds one is refused rather than queued behind it, and authorization, the document revision and the debugger identity are re-checked before a frame is returned. session.stop drains an in-flight capture before it detaches the debugger and closes tabs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A gateway that runs browser tasks for a user renders its own view of the running task: a small preview of the page the agent is working on, and a control that brings that tab to the front so the user can take over.
Neither can be built on the tool RPCs.
ToolQueues::dispatchserializes work per session, so atool.screenshotissued for a preview waits behind whatever the agent is doing; whilerequest_helpwaits for the user, it waits for the whole human-help timeout. The result is that the preview goes dark exactly when the user most wants to look — during a long navigation, and while the agent is asking for help.The DSH plugin's own PiP overlay works around this from the other side:
ObservationService.acquireForegroundcancels the pending thumbnail and kills an activebsk screenshotas soon as model-facing work starts, and only schedules a fresh frame after it ends.Change
Two optional requests are answered on the socket before
WSTransportsees the frame, so they never enter the tool dispatcher:{"id":"ui-1","method":"ui.task_preview","params":{"session_id":"…"}} {"id":"ui-2","method":"ui.task_focus","params":{"session_id":"…"}}ui.task_previewreturnsimage_base64,format,tab_id,titleandcaptured_at.ui.task_focusactivates the task's own tab and raises its window.Neither starts a session, and the channel is attached only to authenticated remote sockets — an extension connected to a local daemon answers
unknown_methodas before, so a gateway can feature-detect.Constraints kept deliberately tight:
session.stopdrains an in-flight capture before it detaches the debugger and closes tabs. No session state, turn-end detach or extra lifecycle callback is introduced.docs/remote-extension-connection.mddocuments the contract and drops the sentence that said there is no preview or focus side protocol.Validation
pnpm --filter @browser-skill/extension test: 1874 passed, 103 skipped. 12 of those are new, covering the channel (routing, invalid params, failure reporting, late replies on a closed socket) and the preview (overlay retention, coalescing, the HiDPI bound, the in-flight refusal and its recovery, stop draining a stuck capture, a failed stop restoring previews, unowned tabs, and discarding frames after authorization or attachment changes).pnpm --filter @browser-skill/extension compilepasses.Notes
The names are a proposal. If you would rather namespace these differently, or serve them from the daemon's remote server instead of the extension socket, say which shape you want and I will rework it.
This is the capability WeKnora has been carrying downstream; the version here is self-contained and free of anything specific to that deployment.