Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion extensions/dsh-browser/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ The **browser-operation end** of dsh: the model reads and operates the browser p
| Close tab | `browser_close_tab` | Close a listed tab |
| Read region | `browser_get_text` | Lazy-loaded content / partial text |
| Wait | `browser_wait` | Page load and render-settle detection |
| Inspect elements / CSS | `browser_devtools_elements` / `browser_devtools_set_element_style` / `browser_devtools_set_element_attribute` | High privilege: computed styles, box model, matched rules, live CSS and attribute edits for debugging |
| Run page JavaScript | `browser_console_eval` / `browser_console_logs` | High privilege: evaluate in the page's own context and read recorded console output |
| Capture network | `browser_devtools_network` / `browser_devtools_list_requests` / `browser_devtools_get_request` / `browser_devtools_request_body` | High privilege: captured request list, per-request detail, and bodies (credential headers redacted by default) |
| Chat with images | `session.prompt` / `session.attachment` | Host-gated image selection, image-only sends, and durable history previews |
| Quote what you highlight | side panel composer | The text you select in the page becomes a quote in the composer and rides along with your next message |

Expand Down Expand Up @@ -117,13 +120,24 @@ For extension-only development, load `extensions/dsh-browser/dist/` from `chrome
- **Unrestricted control is explicit**: **Allow unrestricted browser control** becomes active only after the setting is saved successfully, then lets page reads, page actions, and tab list/follow/close operations run without confirmation. Calls capture their access mode when received, so enabling unrestricted control never retroactively elevates an existing restricted call. Disabling it takes effect immediately, cancels calls that have not dispatched an action, waits for already-dispatched browser operations to settle, and only then saves the restrictive setting. A rapid re-enable remains restricted until that revocation finishes, and concurrent saves persist in request order. Protected-page DOM content remains inaccessible.
- **Conversation continuity**: reopening the panel resumes the most recently active browser conversation by default, falling back to the latest non-empty durable session before creating a new one. This can be disabled in Settings.

## Developer tools (opt-in, high privilege)

The developer-tools group drives the controlled tab through the Chrome debugger protocol (`chrome.debugger`), so it can inspect and edit page elements and CSS, run JavaScript in the page's own context, and capture network traffic — including response bodies, which page-level interception cannot read reliably.

- **Off by default**: every developer-tools call raises its own approval dialog. The dialog has its own heading and can allow once, trust the group for the current side-panel session, or deny. The blanket **Allow unrestricted browser control** switch and the persistent origin allowlist never cover this group.
- **Turn it on deliberately**: **Allow developer tools** in Settings is the only way to run these calls without a per-call prompt. Turning it off detaches the debugger immediately and cancels in-flight devtools calls; the session-scoped grant is cleared the moment the setting changes.
- **One debugger per tab**: Chrome refuses a second debugger, so the browser's own DevTools cannot stay open on the same tab while capture is attached. The tool result explains this and tells you to close DevTools on that tab.
- **Data handling**: capture is in memory only, bounded to 200 requests and 500 console entries, and resets on attach and on every top-level navigation. `Authorization`, `Cookie`, `Set-Cookie`, and similar header values are shown as `[redacted]` unless a call explicitly opts in, and bodies of authentication-shaped URLs are redacted by default.
- **Platform**: Chrome only. The Firefox build does not request the `debugger` permission, so the host never registers the group there.

## Permissions

Chrome uses `sidePanel`; Firefox uses `sidebar_action`. Both request `storage` (settings and recent-session continuity), `notifications` (optional reminders for approvals received while the panel is closed), `tabs` + `activeTab` + `scripting` (observe tab changes and inject/message the explicitly controlled tab, including lazy recovery for pages opened before install), `webNavigation` (enumerate and bind messages to that tab's frame documents), `alarms` (background keepalive), and `http/https` (content-script injection on normal pages). Firefox's AMO manifest declares the browsing activity, website content/activity, and personal communications that the add-on sends to the configured dsh/model service. The extension never changes the visible tab or silently follows a manual switch; background operation happens only after the user chooses to stay on the original tab.
Chrome uses `sidePanel`; Firefox uses `sidebar_action`. Both request `storage` (settings and recent-session continuity), `notifications` (optional reminders for approvals received while the panel is closed), `tabs` + `activeTab` + `scripting` (observe tab changes and inject/message the explicitly controlled tab, including lazy recovery for pages opened before install), `webNavigation` (enumerate and bind messages to that tab's frame documents), `alarms` (background keepalive), and `http/https` (content-script injection on normal pages). Chrome additionally requests `debugger`, which is what the opt-in developer-tools group needs; the Firefox manifest omits it, so the group stays unavailable there. Firefox's AMO manifest declares the browsing activity, website content/activity, and personal communications that the add-on sends to the configured dsh/model service. The extension never changes the visible tab or silently follows a manual switch; background operation happens only after the user chooses to stay on the original tab.

## Known limitations

- Only one extension connection at a time. An unopened browser profile never claims it; if another open panel replaces a live connection, the replaced client yields instead of starting a reconnect fight.
- Developer tools are Chrome-only, need the user's setting or a per-call approval, and cannot attach while the browser's own DevTools is open on the same tab. Attaching shows Chrome's "being debugged" banner for that tab until the call finishes or the setting is turned off.
- Tab affinity is global to that extension connection rather than per chat session.
- Accessible cross-origin iframes are snapshotted and operated with stable `(frame, index)` addresses. Restricted or short-lived frames are reported as unavailable without failing the whole page snapshot.
- Captcha/image-only controls cannot be handled — the tool result reports "elements with no accessible name" and asks the user to complete that step manually.
Expand Down
16 changes: 15 additions & 1 deletion extensions/dsh-browser/README.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ dsh 的**浏览器操作端**:让模型直接读取并操作你在浏览器里
| 关闭标签页 | `browser_close_tab` | 关闭已列出的标签页 |
| 读区域 | `browser_get_text` | 懒加载内容 / 局部文本 |
| 等待 | `browser_wait` | 页面加载与渲染稳定检测 |
| 查看元素与 CSS | `browser_devtools_elements` / `browser_devtools_set_element_style` / `browser_devtools_set_element_attribute` | 高权限:计算样式、盒模型、命中规则,并可现场修改 CSS 与属性来调试 |
| 执行页面 JS | `browser_console_eval` / `browser_console_logs` | 高权限:在页面自身上下文执行代码,并读取记录的控制台输出 |
| 网络抓包 | `browser_devtools_network` / `browser_devtools_list_requests` / `browser_devtools_get_request` / `browser_devtools_request_body` | 高权限:已捕获请求列表、单条详情与正文(凭据类请求头默认脱敏) |
| 图片对话 | `session.prompt` / `session.attachment` | 按宿主能力启用图片选择、纯图片发送和持久历史预览 |
| 引用你划选的内容 | 侧栏输入框 | 你在页面里选中的文字会变成输入框里的引用,随下一条消息一起发送 |

Expand Down Expand Up @@ -117,13 +120,24 @@ pnpm --filter dsh-browser-extension run test
- **完全控制需显式开启**:**允许模型完全控制浏览器**只有设置保存成功后才会生效,启用后页面读取、页面操作以及标签页列出/跟随/关闭均无需确认。调用在收到时固定其访问模式,因此开启完全控制不会追溯提升已经开始的受限调用。关闭会立即生效:取消尚未下发操作的调用,等待已经下发到浏览器的操作完成,再保存限制设置。快速重新开启也会在旧权限撤销完成前保持受限,并发保存会按请求顺序落盘。受保护页面的 DOM 内容始终不可访问。
- **会话续接**:重新打开侧栏时默认恢复最近活跃的浏览器会话;若该会话不可用,则恢复最新的非空持久会话,最后才创建新会话。可在设置中关闭。

## 开发者工具(需显式开启,高权限)

开发者工具组通过 Chrome 调试协议(`chrome.debugger`)驱动受控标签页,因此可以查看并修改页面元素与 CSS、在页面自身上下文执行 JavaScript、以及抓取网络请求(含页面层拦截拿不可靠的响应正文)。

- **默认关闭**:每次开发者工具调用都会单独弹出审批。该弹窗有独立标题,可选择仅允许这一次、本次会话信任该组、或拒绝。**允许模型完全控制浏览器**开关与持久化 origin 白名单**都不会**覆盖这一组。
- **需要显式开启**:只有设置里的**允许开发者工具**开启后,这些调用才会免除逐次确认。关闭会立即断开调试器并中断进行中的开发者工具调用;设置一变,本次会话的临时信任也同时清空。
- **每个标签页只允许一个调试器**:Chrome 会拒绝第二个调试器,因此附加期间浏览器自带的开发者工具无法在同一标签页打开。工具返回会说明原因并提示先关闭该标签页的 DevTools。
- **数据处理**:抓包只存在内存中,上限为 200 条请求与 500 条控制台记录,附加时与每次顶层导航时都会重置。`Authorization`、`Cookie`、`Set-Cookie` 等请求头默认显示为 `[redacted]`,除非某次调用显式选择;疑似认证 URL 的正文默认脱敏。
- **平台**:仅 Chrome。Firefox 构建不申请 `debugger` 权限,因此宿主也不会在该平台注册这一组工具。

## 权限说明

Chrome 使用 `sidePanel`,Firefox 使用 `sidebar_action`。两者都申请 `storage`(设置与最近会话续接)、`notifications`(侧栏关闭时可选的审批提醒)、`tabs` + `activeTab` + `scripting`(观察切页,并向用户显式选择的受控标签页注入/发消息;安装前已打开的页面也会按需补注入)、`webNavigation`(枚举该标签页中的 frame,并把消息绑定到具体文档)、`alarms`(后台保活)和 `http/https`(内容脚本注入普通网页)。Firefox AMO manifest 如实声明扩展会把浏览活动、网页内容/操作和对话内容发送给用户配置的 dsh/模型服务。扩展绝不改变用户正在看的标签页,也不会静默跟随手动切页;只有用户选择继续原页面后,助手才会在后台操作。
Chrome 使用 `sidePanel`,Firefox 使用 `sidebar_action`。两者都申请 `storage`(设置与最近会话续接)、`notifications`(侧栏关闭时可选的审批提醒)、`tabs` + `activeTab` + `scripting`(观察切页,并向用户显式选择的受控标签页注入/发消息;安装前已打开的页面也会按需补注入)、`webNavigation`(枚举该标签页中的 frame,并把消息绑定到具体文档)、`alarms`(后台保活)和 `http/https`(内容脚本注入普通网页)。Chrome 额外申请 `debugger`,这是可选的开发者工具组所需;Firefox manifest 不申请该权限,因此该组在 Firefox 上始终不可用。Firefox AMO manifest 如实声明扩展会把浏览活动、网页内容/操作和对话内容发送给用户配置的 dsh/模型服务。扩展绝不改变用户正在看的标签页,也不会静默跟随手动切页;只有用户选择继续原页面后,助手才会在后台操作。

## 已知限制

- 同时只有一个扩展连接桥。未打开侧栏的浏览器 Profile 不会抢占连接;另一个已打开的侧栏顶替连接后,被替换的一端会主动让权,不再反复重连互踢。
- 开发者工具仅 Chrome 可用,需要用户开启设置或逐次审批,且在同一标签页开着浏览器自带 DevTools 时无法附加。附加期间该标签页会显示 Chrome 的“正在调试此浏览器”提示,直到调用结束或设置关闭。
- 标签页绑定属于整个扩展连接,而不是单个对话会话。
- 可访问的跨源 iframe 会进入快照,并通过稳定的 `(frame, index)` 地址执行操作;受保护或已销毁的 frame 会标记为不可访问,不影响整页快照。
- 验证码/纯图片按钮无法处理——工具结果会标注"存在无文本可访问名的元素",提示用户手动完成该步。
Expand Down
3 changes: 2 additions & 1 deletion extensions/dsh-browser/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@
"scripting",
"webNavigation",
"alarms",
"notifications"
"notifications",
"debugger"
],
"host_permissions": [
"http://*/*",
Expand Down
Loading