diff --git a/README.i18n.yaml b/README.i18n.yaml index 8a63c95eb..ef8cc28fb 100644 --- a/README.i18n.yaml +++ b/README.i18n.yaml @@ -1,5 +1,5 @@ # Bilingual consistency record for the repository README files. # README.md is the default English page; README.zh.md is its Chinese counterpart. # Values are git blob hashes from the last confirmed-consistent review. -README.md: 1163cc9c9a7332a95f8b2448461243f0813ffdf2 -README.zh.md: 6ce7a8314f10dcd45316906f79991e0604b0814c +README.md: 767a4e02b360e6c2b28ee54d79067b5f64e4520d +README.zh.md: 04405f75c75884a3682defb784c26d0e0c541129 diff --git a/README.md b/README.md index 3f743c5b9..767a4e02b 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ Connect [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) to t `dsh` is DeepSeek AI's open-source, plugin-based agent harness. This repository provides a companion browser bridge plugin and Chrome/Firefox MV3 extension as one standalone pnpm workspace. -Browser operation remains text-only: pages become structured text with a numbered inventory of interactive elements, and the model addresses those elements by number. dsh 0.2.0 multimodal chat is separate from that page channel—the side panel accepts PNG, JPEG, WebP, and GIF attachments when the host advertises image support, while browser tools still never capture screenshots. +Browser operation is text-first: pages become structured text with a numbered inventory of interactive elements, and the model addresses those elements by number. `browser_screenshot` adds the one image path — the extension captures the visible area, the bridge writes those bytes to a local file, and the model receives that path instead of pixels. dsh 0.2.0 multimodal chat is separate from that page channel: the side panel accepts PNG, JPEG, WebP, and GIF attachments when the host advertises image support. > [!IMPORTANT] > The workspace pins dsh 0.2.0-rc.2, the minimum supported runtime. Older DSH releases are not supported. @@ -59,6 +59,7 @@ The paired Playwright / extension duration ratio was **1.24** (95% CI **1.16–1 | Follow tab | `browser_follow_tab` | Bind later browser tools to a tab returned by `browser_list_tabs` without activating it | | Close tab | `browser_close_tab` | Close a tab returned by `browser_list_tabs` | | Read region | `browser_get_text` | Lazy-loaded or partial page text | +| Capture the view | `browser_screenshot` | Save the visible area as a local image and return its path for `read_image`; use it for visual content (captcha, chart, layout) | | Wait for stability | `browser_wait` | Page-load and render-settle detection | | Send images | `session.prompt` / `session.attachment` | Host-capability-gated image drafts, image-only prompts, and durable history previews | | Quote a selection | side panel composer | Text you highlight in the page appears in the composer and is sent with your next message as fenced, attributed page content | @@ -76,7 +77,7 @@ scripts/install.ps1 ## Why this design - **Your real browser, not a headless copy**: the model works in the page you already have open, retaining logins, sessions, and cookies. -- **A text-first page interface**: numbered controls, stable IDs across snapshots, delta updates, and masked sensitive values make pages operable without screenshots; user-attached chat images use dsh's separate multimodal message path. +- **A text-first page interface**: numbered controls, stable IDs across snapshots, delta updates, and masked sensitive values make pages operable as text; `browser_screenshot` is the deliberate exception, and its pixels are saved to a local file rather than entering the conversation. User-attached chat images use dsh's separate multimodal message path. - **Pointing instead of describing**: highlight the passage you mean and the side panel quotes it, so "explain this" needs no page tour. The quote is captured only while a panel is open, and nothing is sent until you send the message. - **A narrow privacy boundary**: passwords and payment-card values are always rendered as `••••` and never leave the page. - **A guarded bridge**: authenticated handshakes protect remote connections, privileged gateway methods reject non-loopback callers, and the extension binds tools to one user-controlled tab. @@ -185,7 +186,7 @@ If you encounter `cache.hydratePrepared is not a function`, update the repositor - The bridge path sits outside the `/api` trust boundary and performs its own bearer-token authentication. - Local Chrome extension origins retain zero-configuration loopback access; Firefox origins are per-install UUIDs and must present the bearer token. - Privileged gateway methods such as `settings.*`, `credentials.*`, and `host.open*` reject non-loopback sources. -- The browser-page pipeline is text-only and never captures screenshots; explicitly attached chat images use dsh's durable attachment service. Password and payment-card values never leave the page. +- The browser-page pipeline returns text, and its single image route (`browser_screenshot`) writes each capture to a local temp file so the model is handed a path rather than pixels; explicitly attached chat images use dsh's durable attachment service. Password and payment-card values never leave the page. - When work begins, the assistant binds to the active tab (at prompt submission, or at the first direct browser-tool call). If you switch tabs manually, later browser actions pause and the side panel asks whether the assistant should continue on the original tab or follow the new one. Choosing the original tab permits background operation; the extension never silently retargets or changes your visible tab. Closing the controlled tab also pauses tools until you explicitly select the current page. - Text you highlight is captured only while a side panel is open and page sharing is not `off`, and never from password or payment-card fields. It stays inside the extension until you send the message, is dropped when you dismiss it or its page navigates or closes, and reaches the model inside the same untrusted-content boundary as page snapshots — including its source title and URL, which the page also controls. - Page-authored text is wrapped as untrusted input. The default `auto` mode reads only the controlled tab without an extra prompt; privacy-sensitive users can select `ask` for per-read confirmation or `off` to block reads entirely. In `ask` mode, the read dialog can allow one read or persistently switch back to `auto`; this can be reversed in Settings. Read page text is sent to the selected model. diff --git a/README.zh.md b/README.zh.md index fc8fe4d45..04405f75c 100644 --- a/README.zh.md +++ b/README.zh.md @@ -8,7 +8,7 @@ `dsh` 是由 DeepSeek AI 开发的开源、插件化 agent harness（智能体框架）。本仓库将配套的浏览器桥插件与 Chrome/Firefox MV3 扩展组成一个独立的 pnpm workspace。 -浏览器操作仍采用纯文本设计：页面会转换为结构化文本和带编号的交互元素清单，模型通过编号定位元素。dsh 0.2.0 的多模态对话走独立通道——宿主声明图片能力时，侧栏可发送 PNG、JPEG、WebP 和 GIF；浏览器工具本身仍不会截取页面截图。 +浏览器操作以文本为主：页面会转换为结构化文本和带编号的交互元素清单，模型通过编号定位元素。`browser_screenshot` 是唯一的图片通道——扩展截取可见区域，桥接层把这些字节写成本地文件，模型拿到的是文件路径而不是像素。dsh 0.2.0 的多模态对话走独立通道：宿主声明图片能力时，侧栏可发送 PNG、JPEG、WebP 和 GIF。 > [!IMPORTANT] > 当前工作区固定使用 dsh 0.2.0-rc.2，也是最低支持版本；不再支持旧版 DSH。 @@ -59,6 +59,7 @@ Playwright / 扩展的配对耗时比为 **1.24**（95% CI **1.16–1.34**）： | 跟随标签页 | `browser_follow_tab` | 将后续浏览器工具绑定到 `browser_list_tabs` 返回的标签页，而不激活该标签页 | | 关闭标签页 | `browser_close_tab` | 关闭 `browser_list_tabs` 返回的标签页 | | 读取区域 | `browser_get_text` | 懒加载内容 / 局部文本 | +| 截取画面 | `browser_screenshot` | 把可见区域存成本地图片并返回路径供 `read_image` 查看；页面含义靠视觉（验证码、图表、排版）时用它 | | 等待稳定 | `browser_wait` | 页面加载与渲染稳定检测 | | 发送图片 | `session.prompt` / `session.attachment` | 按宿主能力启用图片草稿、纯图片消息和持久历史预览 | | 引用选中内容 | 侧栏输入框 | 在页面里划选的文字会出现在输入框，随下一条消息一起发送，并带上来源与不可信内容边界 | @@ -76,7 +77,7 @@ scripts/install.ps1 ## 为什么这样设计 - **使用你的真实浏览器，而不是无头副本**：模型操作你已经打开的页面，登录态、会话和 Cookie 均会保留。 -- **纯文本页面接口**：编号控件、跨快照稳定 ID、delta 更新和敏感值掩码，使模型无需截图也能操作页面；用户主动添加的对话图片走 dsh 独立的多模态消息通道。 +- **文本优先的页面接口**：编号控件、跨快照稳定 ID、delta 更新和敏感值掩码，使模型以文本方式操作页面；`browser_screenshot` 是刻意的例外，其像素会写成本地文件而不会进入对话。用户主动添加的对话图片走 dsh 独立的多模态消息通道。 - **用「指」代替「描述」**：直接划选你要问的那段文字，侧栏会把它引用下来，说「解释这个」不必再描述整页内容。只有侧栏打开时才会捕获，并且在你发送消息之前不会离开浏览器。 - **收窄隐私边界**：密码和支付卡字段始终显示为 `••••`，字段值不会离开页面。 - **受保护的桥连接**：远程连接使用认证握手，特权网关方法拒绝非回环调用方，扩展把工具绑定到一个由用户控制的标签页。 @@ -185,7 +186,7 @@ pnpm --filter dsh-browser-extension run test - 桥路径在 `/api` 信任栅栏之外，自带 bearer token 认证。 - Chrome 扩展的本地 Origin 保留零配置回环访问；Firefox Origin 是每次安装生成的 UUID，必须携带 bearer token。 - 特权网关方法（`settings.*`/`credentials.*`/`host.open*`）对非回环来源一律拒绝。 -- 单活动连接；浏览器页面管线为纯文本且不截图；用户主动添加的对话图片交给 dsh 持久附件服务，密码和卡号值永不回传。 +- 单活动连接；浏览器页面管线以文本为主，其唯一的图片通道（`browser_screenshot`）把每次截图写入本地临时文件，模型拿到的是路径而不是像素；用户主动添加的对话图片交给 dsh 持久附件服务，密码和卡号值永不回传。 - 助手开始工作时会绑定当时的活动标签页（提交提示时绑定；直接调用浏览器工具时则在首次调用绑定）。用户手动切页后，后续浏览器操作会暂停，侧栏会询问让助手继续原页面还是跟随新页面；选择原页面后允许在后台继续，但扩展绝不静默改绑或切换用户正在看的页面。受控标签页关闭后也会暂停，直到用户显式选择当前页。 - 只有在侧栏打开、且页面共享不是「关闭」时才会捕获划选内容，密码和卡号字段永不读取。内容在发送之前始终留在扩展内部；移除、页面跳转或标签页关闭都会丢弃它；发送时与页面快照一样包在不可信内容边界内，来源标题和 URL 同样由页面提供，因此也放在边界之内。 - 网页文字会标记为不可信输入。默认「自动共享」只按需读取受控标签页且不额外弹窗；对隐私敏感时可选择「每次询问」，或用「关闭」完全阻断读取。在「每次询问」模式下，读取弹窗可以仅允许一次，也可以持久切回自动读取；之后仍可在设置中关闭。读取的页面文字会发送给当前选择的模型。 diff --git a/extensions/dsh-browser/README.i18n.yaml b/extensions/dsh-browser/README.i18n.yaml index c591a0b25..22445ab1f 100644 --- a/extensions/dsh-browser/README.i18n.yaml +++ b/extensions/dsh-browser/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write dsh-browser/extensions/dsh-browser/README.md -README.md: 6b6ea0b302a340fca44d6f461ebf84a7b5419c89 -README.zh.md: 36f25412f7e1431aa89a8ac41ef045c9b7676790 +README.md: 30d41256b315c6e93490e1fca400a04678075f3c +README.zh.md: 3bd1765041e2d85f8bd1cf367071cd0eba0e621f diff --git a/extensions/dsh-browser/README.md b/extensions/dsh-browser/README.md index 711efd03a..30d41256b 100644 --- a/extensions/dsh-browser/README.md +++ b/extensions/dsh-browser/README.md @@ -4,7 +4,7 @@ English | [中文](README.zh.md) The **browser-operation end** of dsh: the model reads and operates the browser page you have open — extract content, click elements, fill forms, scroll, and navigate, all in the real page with your login state preserved. The side panel is the conversation entry. -**Two explicit channels**: browser pages are still rendered as structured text (a numbered interactive-element inventory), so browser tools never take screenshots. Separately, a dsh 0.1.2 host can advertise multimodal image limits; the side panel then accepts PNG, JPEG, WebP, and GIF attachments and renders their durable history references. +**Text-first page interface, one image route**: browser pages are rendered as structured text (a numbered interactive-element inventory) and operated by number. `browser_screenshot` is the exception: it captures the visible area, the bridge saves those bytes to a local file, and the model is given that path rather than image data. Separately, a dsh 0.1.2 host can advertise multimodal image limits; the side panel then accepts PNG, JPEG, WebP, and GIF attachments and renders their durable history references. ## What the model can do @@ -20,6 +20,7 @@ The **browser-operation end** of dsh: the model reads and operates the browser p | Follow tab | `browser_follow_tab` | Bind later browser tools to a listed tab without activating it | | Close tab | `browser_close_tab` | Close a listed tab | | Read region | `browser_get_text` | Lazy-loaded content / partial text | +| Capture the view | `browser_screenshot` | Saves the visible area as a local image file and returns its path for `read_image`. Use it when the meaning is visual (captcha, chart, layout). `format` is `jpeg` (default) or `png`, with an optional `quality` of 1–100 | | Wait | `browser_wait` | Page load and render-settle detection | | Chat with images | `session.prompt` / `session.attachment` | Host-gated image selection, image-only sends, and durable history previews | | Quote what you highlight | side panel composer | The text you select in the page becomes a quote in the composer and rides along with your next message | @@ -101,13 +102,14 @@ The recommended zero-configuration command does not require Git or a local clone 3. **Use it**: open a normal `http://` or `https://` page and click the DeepSeek whale icon. Both builds auto-discover local dsh. Chrome loopback connections need no address or token; Firefox must be given the token from `~/.dsh/ext-bridge-token` because a `moz-extension://` UUID is not an add-on identity. Chat directly or click "Read page" first. -Pages that were already open before extension installation or reload are instrumented automatically on the first action, so they do not require a manual refresh. Browser-internal and protected pages such as `chrome://` and the Chrome Web Store expose only tab metadata and browser-level HTTP(S) navigation, back, forward, and reload; their DOM cannot be read or operated. +Pages that were already open before extension installation or reload are instrumented automatically on the first action, so they do not require a manual refresh. Browser-internal and protected pages such as `chrome://` and the Chrome Web Store expose only tab metadata, browser-level HTTP(S) navigation, back, forward, reload, and `browser_screenshot` (the capture is taken by the browser, not by the page); their DOM cannot be read or operated. For extension-only development, load `extensions/dsh-browser/dist/` from `chrome://extensions`, or run `build:firefox` and load `extensions/dsh-browser/dist-firefox/manifest.json` from `about:debugging#/runtime/this-firefox`. Rebuild and reload after code changes. -## Why browser operation stays text-only +## Why browser operation is text-first - **Snapshot as the view**: the model's entire view of the page is structured text (title/URL/main/numbered elements/forms), budgeted at 32k chars by default (plugin-configurable, negotiated to the extension via `hello.ok`). +- **One deliberate image route**: `browser_screenshot` photographs the visible area — through `chrome.tabs.captureVisibleTab` first, and through the DevTools protocol (`Page.captureScreenshot`) whenever that call is refused, so the answer says which route produced the image. The extension returns the pixels as base64 beside its text, the bridge writes them to a file under the OS temp directory, and the model receives the path. Image bytes never enter the conversation, and the fallback needs the optional `debugger` permission, which the side panel Settings can grant or withhold. - **Page text is untrusted input**: snapshots and targeted text reads are enclosed in a fresh nonce-bound trust marker and explicitly tell the model never to treat page-authored commands as instructions. This is defense in depth; extension-side action approval is the enforcement boundary. - **Stable numbering**: element numbers persist across snapshots (WeakMap + `data-dsh-el`), so the model can say "click 7"; a large page change explicitly reports "numbers reindexed". - **Delta mode**: `browser_snapshot({delta:true})` returns only changed element numbers, saving tokens. @@ -119,14 +121,14 @@ For extension-only development, load `extensions/dsh-browser/dist/` from `chrome ## Permissions -Chrome uses `sidePanel`; Firefox uses `sidebar_action`. Both request `storage` (settings and recent-session continuity), `notifications` (optional reminders for approvals received while the panel is closed), `tabs` + `activeTab` + `scripting` (observe tab changes and inject/message the explicitly controlled tab, including lazy recovery for pages opened before install), `webNavigation` (enumerate and bind messages to that tab's frame documents), `alarms` (background keepalive), and `http/https` (content-script injection on normal pages). Firefox's AMO manifest declares the browsing activity, website content/activity, and personal communications that the add-on sends to the configured dsh/model service. The extension never changes the visible tab or silently follows a manual switch; background operation happens only after the user chooses to stay on the original tab. +Chrome uses `sidePanel`; Firefox uses `sidebar_action`. Both request `storage` (settings and recent-session continuity), `notifications` (optional reminders for approvals received while the panel is closed), `tabs` + `activeTab` + `scripting` (observe tab changes and inject/message the explicitly controlled tab, including lazy recovery for pages opened before install), `webNavigation` (enumerate and bind messages to that tab's frame documents), `alarms` (background keepalive), and `http/https` (content-script injection on normal pages). They also declare `debugger` as an **optional** permission: it is never requested at install time, the browser's "being controlled" banner appears only while a capture runs, and Settings offers an explicit button that calls `chrome.permissions.request`. Firefox's AMO manifest declares the browsing activity, website content/activity, and personal communications that the add-on sends to the configured dsh/model service. The extension never changes the visible tab or silently follows a manual switch; background operation happens only after the user chooses to stay on the original tab. ## Known limitations - Only one extension connection at a time. An unopened browser profile never claims it; if another open panel replaces a live connection, the replaced client yields instead of starting a reconnect fight. - Tab affinity is global to that extension connection rather than per chat session. - Accessible cross-origin iframes are snapshotted and operated with stable `(frame, index)` addresses. Restricted or short-lived frames are reported as unavailable without failing the whole page snapshot. -- Captcha/image-only controls cannot be handled — the tool result reports "elements with no accessible name" and asks the user to complete that step manually. +- Captcha/image-only controls are reported as "elements with no accessible name". `browser_screenshot` lets the model see them, but the user still has to solve them. - No automatic token rotation. - Synthetic `browser_press` events do not trigger browser-native default actions such as Tab focus movement, arrow-key scrolling, or Enter activation; use manual input when a workflow depends on those defaults. - `browser_wait` considers page load plus a fixed quiet window, but does not observe continuously changing DOM state; a live-updating SPA may be reported as stable. diff --git a/extensions/dsh-browser/README.zh.md b/extensions/dsh-browser/README.zh.md index 20f886825..3bd176504 100644 --- a/extensions/dsh-browser/README.zh.md +++ b/extensions/dsh-browser/README.zh.md @@ -4,7 +4,7 @@ dsh 的**浏览器操作端**：让模型直接读取并操作你在浏览器里打开的页面——抓取内容、点击元素、填写表单、滚动与导航，全部在真实页面执行、登录态保留。侧边栏面板是与模型对话的入口。 -**两条明确分离的通道**：浏览器页面仍以结构化文本呈现（带编号的交互元素清单），浏览器工具不会截图；另一方面，dsh 0.1.2 宿主可声明多模态图片限制，侧栏据此接收 PNG、JPEG、WebP 和 GIF，并渲染会话中的持久图片附件。 +**文本优先的页面接口 + 一条图片通道**：浏览器页面以结构化文本呈现（带编号的交互元素清单）并按编号操作；`browser_screenshot` 是例外——它截取可见区域，桥接层把字节写成本地文件，模型拿到的是文件路径而不是图片数据。另一方面，dsh 0.1.2 宿主可声明多模态图片限制，侧栏据此接收 PNG、JPEG、WebP 和 GIF，并渲染会话中的持久图片附件。 ## 模型能做什么 @@ -20,6 +20,7 @@ dsh 的**浏览器操作端**：让模型直接读取并操作你在浏览器里 | 跟随标签页 | `browser_follow_tab` | 将后续浏览器工具绑定到已列出的标签页，而不激活该标签页 | | 关闭标签页 | `browser_close_tab` | 关闭已列出的标签页 | | 读区域 | `browser_get_text` | 懒加载内容 / 局部文本 | +| 截取画面 | `browser_screenshot` | 把可见区域存成本地图片文件并返回路径，供 `read_image` 查看。页面含义靠视觉（验证码、图表、排版）时用它。`format` 取 `jpeg`（默认）或 `png`，可选 `quality` 1–100 | | 等待 | `browser_wait` | 页面加载与渲染稳定检测 | | 图片对话 | `session.prompt` / `session.attachment` | 按宿主能力启用图片选择、纯图片发送和持久历史预览 | | 引用你划选的内容 | 侧栏输入框 | 你在页面里选中的文字会变成输入框里的引用，随下一条消息一起发送 | @@ -101,13 +102,14 @@ pnpm --filter dsh-browser-extension run test 3. **开始使用**：打开普通的 `http://` 或 `https://` 页面，点击 DeepSeek 鲸鱼图标打开侧边栏。两个构建都会自动探测本机 dsh。Chrome 回环连接无需地址或 Token；Firefox 的 `moz-extension://` UUID 不能证明扩展身份，必须在设置中填入 `~/.dsh/ext-bridge-token`。可以直接对话，或先点「读取页面」。 -页面即使在扩展安装或重载之前已经打开，也会在第一次操作时自动补加载内容脚本，无需手动刷新。`chrome://`、Chrome Web Store 等浏览器内置或受保护页面只提供标签页元数据，以及浏览器级 HTTP(S) 导航、后退、前进和刷新；不能读取或操作其 DOM。 +页面即使在扩展安装或重载之前已经打开，也会在第一次操作时自动补加载内容脚本，无需手动刷新。`chrome://`、Chrome Web Store 等浏览器内置或受保护页面只提供标签页元数据，以及浏览器级 HTTP(S) 导航、后退、前进、刷新和 `browser_screenshot`（截图由浏览器而非页面完成）；不能读取或操作其 DOM。 如果只开发扩展，Chrome 从 `chrome://extensions` 加载 `extensions/dsh-browser/dist/`；Firefox 运行 `build:firefox` 后，从 `about:debugging#/runtime/this-firefox` 加载 `extensions/dsh-browser/dist-firefox/manifest.json`。代码更新后需重新构建并重新加载。 -## 为什么浏览器操作仍采用纯文本 +## 为什么浏览器操作以文本为主 - **快照即视图**：模型对页面的全部认知 = 结构化文本（标题/URL/正文/编号元素/表单），默认预算 32k 字符（插件可配，经 `hello.ok` 协商给扩展）。 +- **一条刻意的图片通道**：`browser_screenshot` 拍摄可见区域——先走 `chrome.tabs.captureVisibleTab`，一旦该调用被拒绝就改走 DevTools 协议（`Page.captureScreenshot`），因此结果会说明图片是哪条路径产出的。扩展把像素以 base64 随文本一起返回，桥接层写入系统临时目录下的文件，模型只拿到路径。图片字节不会进入对话；回退路径需要可选的 `debugger` 权限，可在侧栏设置中显式授予或拒绝。 - **页面文字是不可信输入**：快照和局部文本读取会放进带随机 nonce 的信任边界，并明确要求模型不得把网页中的命令当成指令。这只是纵深防御；扩展侧的操作审批才是强制安全边界。 - **稳定编号**：元素编号跨快照保持（WeakMap + `data-dsh-el`），模型可以说"点 7 号"；页面大改时显式提示"编号已重排"。 - **delta 模式**：`browser_snapshot({delta:true})` 只返回变化元素的编号，省 token。 @@ -119,14 +121,14 @@ pnpm --filter dsh-browser-extension run test ## 权限说明 -Chrome 使用 `sidePanel`，Firefox 使用 `sidebar_action`。两者都申请 `storage`（设置与最近会话续接）、`notifications`（侧栏关闭时可选的审批提醒）、`tabs` + `activeTab` + `scripting`（观察切页，并向用户显式选择的受控标签页注入/发消息；安装前已打开的页面也会按需补注入）、`webNavigation`（枚举该标签页中的 frame，并把消息绑定到具体文档）、`alarms`（后台保活）和 `http/https`（内容脚本注入普通网页）。Firefox AMO manifest 如实声明扩展会把浏览活动、网页内容/操作和对话内容发送给用户配置的 dsh/模型服务。扩展绝不改变用户正在看的标签页，也不会静默跟随手动切页；只有用户选择继续原页面后，助手才会在后台操作。 +Chrome 使用 `sidePanel`，Firefox 使用 `sidebar_action`。两者都申请 `storage`（设置与最近会话续接）、`notifications`（侧栏关闭时可选的审批提醒）、`tabs` + `activeTab` + `scripting`（观察切页，并向用户显式选择的受控标签页注入/发消息；安装前已打开的页面也会按需补注入）、`webNavigation`（枚举该标签页中的 frame，并把消息绑定到具体文档）、`alarms`（后台保活）和 `http/https`（内容脚本注入普通网页）。两者还把 `debugger` 声明为**可选权限**：安装时不会申请，浏览器「正在受控」横幅只在截图进行中出现，设置页提供一个显式按钮调用 `chrome.permissions.request`。Firefox AMO manifest 如实声明扩展会把浏览活动、网页内容/操作和对话内容发送给用户配置的 dsh/模型服务。扩展绝不改变用户正在看的标签页，也不会静默跟随手动切页；只有用户选择继续原页面后，助手才会在后台操作。 ## 已知限制 - 同时只有一个扩展连接桥。未打开侧栏的浏览器 Profile 不会抢占连接；另一个已打开的侧栏顶替连接后，被替换的一端会主动让权，不再反复重连互踢。 - 标签页绑定属于整个扩展连接，而不是单个对话会话。 - 可访问的跨源 iframe 会进入快照，并通过稳定的 `(frame, index)` 地址执行操作；受保护或已销毁的 frame 会标记为不可访问，不影响整页快照。 -- 验证码/纯图片按钮无法处理——工具结果会标注"存在无文本可访问名的元素"，提示用户手动完成该步。 +- 验证码/纯图片按钮会标注"存在无文本可访问名的元素"；`browser_screenshot` 能让模型看见它们，但仍需用户自己作答。 - 令牌无自动轮换。 - `browser_press` 的合成按键不触发浏览器原生默认行为（Tab 焦点移动、方向键、Enter 激活等），仅用于框架内的键盘事件；依赖原生行为的场景请手动操作。 - `browser_wait` 以加载完成 + 固定静默窗口为准，不观察持续 DOM 更新（连续刷新的 SPA 可能被报为稳定）。 diff --git a/extensions/dsh-browser/manifest.firefox.json b/extensions/dsh-browser/manifest.firefox.json index 8d6dda156..f56db6e43 100644 --- a/extensions/dsh-browser/manifest.firefox.json +++ b/extensions/dsh-browser/manifest.firefox.json @@ -27,6 +27,9 @@ "alarms", "notifications" ], + "optional_permissions": [ + "debugger" + ], "host_permissions": [ "http://*/*", "https://*/*" diff --git a/extensions/dsh-browser/manifest.json b/extensions/dsh-browser/manifest.json index d1311c27b..ccc972a86 100644 --- a/extensions/dsh-browser/manifest.json +++ b/extensions/dsh-browser/manifest.json @@ -15,6 +15,9 @@ "alarms", "notifications" ], + "optional_permissions": [ + "debugger" + ], "host_permissions": [ "http://*/*", "https://*/*" diff --git a/extensions/dsh-browser/src/background/authorization.ts b/extensions/dsh-browser/src/background/authorization.ts index d263a0587..3a86f1740 100644 --- a/extensions/dsh-browser/src/background/authorization.ts +++ b/extensions/dsh-browser/src/background/authorization.ts @@ -5,7 +5,7 @@ import type { TabFrame } from './frames.ts' import type { ApprovalPrompt } from '../security/approval.ts' import { getUiLocale, type UiLocale } from '../i18n.ts' -const PAGE_READS = new Set(['browser_snapshot', 'browser_get_text']) +const PAGE_READS = new Set(['browser_snapshot', 'browser_get_text', 'browser_screenshot']) const STATE_CHANGING_ACTIONS = new Set([ 'browser_click', 'browser_type', @@ -32,9 +32,7 @@ export function approvalPromptForCall( return { kind: 'read', action: call.name, - summary: call.name === 'browser_snapshot' - ? localized(locale, 'Read the current page and accessible iframes', '读取当前页面及可访问 iframe') - : localized(locale, 'Read text from the specified area of the current page', '读取当前页面的指定文本区域'), + summary: readSummary(call, locale), origins: uniqueOrigins(targetFrames, frames), canTrust: false, } @@ -106,6 +104,20 @@ export function originFromUrl(value: string): string | undefined { } } +/** + * Reader-facing purpose of one page-reading tool, shown in the `ask`-mode read + * prompt. A screenshot is a read: it carries page pixels, not just text. + */ +function readSummary(call: ToolCall, locale: UiLocale): string { + if (call.name === 'browser_get_text') { + return localized(locale, 'Read text from the specified area of the current page', '读取当前页面的指定文本区域') + } + if (call.name === 'browser_screenshot') { + return localized(locale, 'Capture the visible area of the current page as an image', '截取当前页面的可见区域为图片') + } + return localized(locale, 'Read the current page and accessible iframes', '读取当前页面及可访问 iframe') +} + function summarizeAction(call: ToolCall, locale: UiLocale): string { const frame = typeof call.args.frame === 'number' && call.args.frame !== 0 ? localized(locale, `, iframe ${call.args.frame}`, `，iframe ${call.args.frame}`) diff --git a/extensions/dsh-browser/src/background/tools.ts b/extensions/dsh-browser/src/background/tools.ts index 0fc3069a1..5055ec3e0 100644 --- a/extensions/dsh-browser/src/background/tools.ts +++ b/extensions/dsh-browser/src/background/tools.ts @@ -1,6 +1,7 @@ /** * Tool dispatch: executes `tool.call` frames in an explicitly selected tab via - * the content script and answers with the text-only result. + * the content script or the DevTools protocol and answers with text, carrying + * captured image bytes alongside it when a screenshot was requested. * * The background service owns tab-affinity policy. Direct callers may omit a * target for backward-compatible active-tab dispatch in isolated tests. @@ -254,6 +255,162 @@ async function dispatchTabNativeTool( return { ok: true, result: { text, navigationPending: true } } } +/** + * Give the compositor one frame to present a tab that was just brought forward. + * `captureVisibleTab` photographs whatever is painted, so shooting immediately + * after `tabs.update` can still catch the previous tab. + */ +async function settleAfterActivation(): Promise { + await new Promise((resolve) => setTimeout(resolve, 120)) +} + +/** Build the answer for a viewport capture, naming the route that produced it. */ +function viewportScreenshotAnswer(base64: string, format: string, route: string, note: string): ToolAnswer { + if (base64 === '') return unavailable('The browser returned an empty screenshot.') + // `format` arrives as the bare subtype ('jpeg' / 'png'): the data URL's own + // `image/...` prefix is stripped before it reaches here. + const bare = format.replace(/^image\//, '') + return { + ok: true, + result: { + text: `Captured the visible area of the controlled tab${note} as image/${bare} ${route}.`, + screenshot: { mime: `image/${bare}`, base64 }, + }, + } +} + +/** + * Capture the visible area of one tab and hand the pixels back to the bridge. + * + * `captureVisibleTab` always photographs the active tab of a window, so a + * background target is brought to the front first — otherwise the model would + * receive a picture of whatever the user happened to be looking at instead. + * + * That call also needs the extension to hold `` or `activeTab` for the + * photographed tab, which is exactly the permission a hosted build can withhold + * from the user's site-access choice; when it is, the call fails with "Either the + * '' or 'activeTab' permission is required". The DevTools protocol is + * therefore the fallback for any failure of that call, and it needs no + * activation at all: its pixels come from the renderer. + */ +async function dispatchScreenshot( + tabId: number, + windowId: number | undefined, + call: ToolCall, + signal?: AbortSignal, +): Promise { + if (isCancelled(call, signal)) return cancelled() + const format = call.args.format === 'png' ? 'png' : 'jpeg' + const requestedQuality = typeof call.args.quality === 'number' ? call.args.quality : 85 + const quality = Math.min(100, Math.max(1, Math.round(requestedQuality))) + try { + const tab = await chrome.tabs.get(tabId) + let activated = false + if (!tab.active) { + await chrome.tabs.update(tabId, { active: true }) + activated = true + // Give the compositor one frame to present the newly activated tab. + await settleAfterActivation() + } + if (isCancelled(call, signal)) return cancelled() + const dataUrl = await chrome.tabs.captureVisibleTab( + windowId ?? tab.windowId, + format === 'png' ? { format: 'png' } : { format: 'jpeg', quality }, + ) + const match = /^data:([^;,]+);base64,(.*)$/s.exec(dataUrl) + if (match === null) return unavailable('The browser returned a screenshot in an unexpected format.') + const note = activated ? ' (the tab was brought to the front first)' : '' + return viewportScreenshotAnswer(match[2] ?? '', match[1] ?? format, 'via chrome.tabs.captureVisibleTab', note) + } catch (error: unknown) { + const detail = error instanceof Error ? error.message : String(error) + return await captureViewportThroughDebugger(tabId, format, quality, detail) + } +} + +/** + * Fallback viewport capture, used when `captureVisibleTab` fails for any reason. + * It waits one frame first so a tab that was just activated has presented, then + * shoots through the protocol. + */ +async function captureViewportThroughDebugger( + tabId: number, + format: string, + quality: number, + captureDetail: string, +): Promise { + return await runWithDebugger(tabId, async (target) => { + try { + await settleAfterActivation() + const answer = await chrome.debugger.sendCommand(target, 'Page.captureScreenshot', { + format, + ...format === 'jpeg' ? { quality } : {}, + fromSurface: true, + }) as { data?: string } + const base64 = typeof answer?.data === 'string' ? answer.data : '' + if (base64 === '') { + return unavailable(`The screenshot could not be taken: ${captureDetail}`) + } + return viewportScreenshotAnswer( + base64, + format, + `through the debugger, because captureVisibleTab was refused: ${captureDetail}`, + '', + ) + } catch (error: unknown) { + const fallbackDetail = error instanceof Error ? error.message : String(error) + return unavailable( + `The screenshot could not be taken: ${captureDetail} (the debugger fallback failed too: ${fallbackDetail})`, + ) + } + }) +} + +/** + * Run one DevTools-protocol task against a tab, attaching first and always + * detaching afterwards. + * + * MV3 gives the extension no way to read renderer pixels or synthesise trusted + * input, so the debugger is the only route to either. Attaching is what raises + * the browser's "being controlled" banner, which is why a session is never held + * open between calls — detaching in `finally` is what takes the banner back down. + * + * `debugger` is an optional permission, so the user has to grant it once from + * the side panel before any protocol route can run; when it is missing the + * answer names the exact switch that grants it. + */ +async function runWithDebugger( + tabId: number, + run: (target: chrome.debugger.Debuggee) => Promise, +): Promise { + let granted = false + try { + granted = await chrome.permissions.contains({ permissions: ['debugger'] }) + } catch { + granted = false + } + if (!granted) { + return unavailable( + 'This needs the "debugger" permission, which this extension has not been granted. Open the side panel Settings and turn on "Screenshot capture (debugger)" to allow it.', + ) + } + const target = { tabId } + try { + await chrome.debugger.attach(target, '1.3') + } catch (error: unknown) { + const detail = error instanceof Error ? error.message : String(error) + return unavailable(`Could not attach the debugger: ${detail}. DevTools open on this tab will block it.`) + } + try { + return await run(target) + } finally { + try { + await chrome.debugger.detach(target) + } catch { + // Already detached; nothing left to release. + } + } +} + function cancelled(): ToolAnswer { return { ok: false, error: { code: 'bridge-closed', message: 'The browser tool call was cancelled.' } } } @@ -689,7 +846,7 @@ export async function dispatchToolCall( // Privacy boundary: with sharing off, no page content may leave the page. if (!tabManagement.unrestrictedAccess && sharePageContent === 'off' - && (call.name === 'browser_snapshot' || call.name === 'browser_get_text')) { + && (call.name === 'browser_snapshot' || call.name === 'browser_get_text' || call.name === 'browser_screenshot')) { return { ok: false, error: { code: 'action-failed', message: 'Page content sharing is disabled in Settings > Page content sharing.' } } } const tab = targetTab ?? (await chrome.tabs.query({ active: true, lastFocusedWindow: true }))[0] @@ -703,7 +860,9 @@ export async function dispatchToolCall( if (targetStillAllowed?.() === false) return targetChanged() const frameError = validateFrameTarget(call, frames) if (frameError !== undefined) return frameError - if (!isInjectablePage(tab.url) && !TAB_NATIVE_TOOLS.has(call.name)) { + // A viewport capture is taken by the browser itself rather than by the page, + // so it needs no injectable document and stays available on protected pages. + if (!isInjectablePage(tab.url) && !TAB_NATIVE_TOOLS.has(call.name) && call.name !== 'browser_screenshot') { return unavailable('The current page DOM is protected by the browser. Only snapshot metadata, navigate, back, forward, and reload are available on this page.') } const targetError = validateElementTarget(call, tab.id, frames) @@ -733,6 +892,12 @@ export async function dispatchToolCall( const refreshedTargetError = validateElementTarget(call, tab.id, executionFrames) if (refreshedTargetError !== undefined) return refreshedTargetError } + // A screenshot is taken by the browser or the renderer rather than by the + // page, so it never goes through the content script. The plain tab capture + // comes first and the DevTools protocol is the fallback. + if (call.name === 'browser_screenshot') { + return await dispatchScreenshot(tab.id, tab.windowId, call, signal) + } if (!isInjectablePage(tab.url)) { return await dispatchTabNativeTool(tab.id, tab.url, tab.title, tab.windowId, call, effectiveBudget, signal, targetStillAllowed, tabManagement.commitAction) ?? unavailable('The current page DOM is protected by the browser.') diff --git a/extensions/dsh-browser/src/panel/App.tsx b/extensions/dsh-browser/src/panel/App.tsx index 6d3d93602..2f8d2a799 100644 --- a/extensions/dsh-browser/src/panel/App.tsx +++ b/extensions/dsh-browser/src/panel/App.tsx @@ -647,6 +647,8 @@ export function App(): React.JSX.Element { const [approvalQueue, setApprovalQueue] = useState([]) const [tabAffinity, setTabAffinity] = useState(null) const [trustedOriginInput, setTrustedOriginInput] = useState('') + const [debuggerGranted, setDebuggerGranted] = useState(null) + const [permissionNotice, setPermissionNotice] = useState(null) const [error, setError] = useState(null) const [showSessionPicker, setShowSessionPicker] = useState(false) const [loadingSessions, setLoadingSessions] = useState(false) @@ -1633,6 +1635,40 @@ export function App(): React.JSX.Element { } } + /** + * Ask for the optional `debugger` permission behind the screenshot fallback. + * + * It is a user-gesture-only prompt, which is why the request lives on this + * explicit button rather than inside tool dispatch, and why nothing else in + * the panel ever calls it. + */ + async function grantDebuggerPermission(): Promise { + try { + const granted = await chrome.permissions.request({ permissions: ['debugger'] }) + setDebuggerGranted(granted) + setPermissionNotice(granted ? copy.settings.screenshotGranted : copy.settings.screenshotDenied) + } catch (cause) { + setPermissionNotice(copy.settings.screenshotGrantFailed(cause instanceof Error ? cause.message : String(cause))) + } + } + + /** Reflect the stored grant state every time the settings view is opened. */ + useEffect(() => { + if (!showSettings) return + let cancelled = false + const read = async (): Promise => { + try { + const granted = await chrome.permissions.contains({ permissions: ['debugger'] }) + if (!cancelled) setDebuggerGranted(granted) + } catch { + // Without the API there is no grant to report, and the button stays usable. + if (!cancelled) setDebuggerGranted(false) + } + } + void read() + return () => { cancelled = true } + }, [showSettings]) + /** Load relay profiles from the llm-pi-ai settings namespace once per settings visit. */ useEffect(() => { if (!showSettings || relayLoaded) return @@ -2024,6 +2060,18 @@ export function App(): React.JSX.Element {