Skip to content

feat(browser): add a viewport screenshot tool and sidebar image paste - #117

Closed
graydovee wants to merge 4 commits into
omdsh-dev:mainfrom
graydovee:feat/screenshot
Closed

graydovee wants to merge 4 commits into
omdsh-dev:mainfrom
graydovee:feat/screenshot

Conversation

@graydovee

Copy link
Copy Markdown

概述 / Summary

Adds browser_screenshot: it captures the visible viewport of the controlled tab and returns the image as an image content block, so vision-capable models read it directly; text-only models get the runtime's placeholder projection and are unaffected. A screenshot photographs whatever is actually on the user's screen — passwords and private messages included — so every call requires an approval prompt in the side panel. The sidebar also gains paste and drag-and-drop image intake. Related to #75.

改动内容 / Changes

Screenshot tool:

  • chrome.tabs.captureVisibleTab on the controlled tab, with png / jpeg format and quality options
  • Pixels go into the attachment store and come back as an image content block; the persisted tool result keeps metadata only — base64 never enters the session file. A png over the byte limit retries once as jpeg
  • Only the active controlled tab of its window is captured; a controlled tab that is not active, or a non-http/https page, is rejected with an explanatory error

Approval:

  • Every screenshot call prompts in the side panel: "Capture a screenshot of the current page (may contain sensitive content)"; denial aborts the capture
  • The prompt is independent of the "page content sharing" setting, which still governs only the text-reading tools; the sole skip path is "allow full browser control" in settings
  • Host permission is <all_urls>: captureVisibleTab accepts only <all_urls> or a gesture-activated activeTab; origin-scoped http/https permissions are not accepted

Sidebar:

  • ⌘V image paste in the composer and drag-and-drop onto the composer box, with the same validation as file upload (count, bytes, dimensions, media types)

测试 / Testing

  • pnpm run check:runtime / typecheck / build / test:smoke
  • pnpm run test: bridge 157/157, extension 411/411
  • Verified in Chrome: the capture succeeds and the model describes the image accurately; the approval prompt appears and denial blocks the capture; ⌘V paste sends an image correctly

影响范围 / Impact

  • Viewport only; full-page capture is out of scope
  • Host permission is <all_urls> (the admission requirement of captureVisibleTab; origin-scoped permissions are not accepted)
  • Wire protocol unchanged; no new dependencies

Checklist

  • 已在相关平台实测通过 / Tested on the relevant platforms
  • 已通过 pnpm run build / bash -n scripts/install.sh 等基础检查 / Passed basic checks
  • 文档已同步更新(如需要)/ Docs updated if needed
  • 不包含无关改动 / No unrelated changes included

扩展侧用 chrome.tabs.captureVisibleTab 截取受控标签页的可视区域,
返回 base64 与媒体类型;宿主侧把像素存入附件存储,以图片内容块
返回给模型,持久化结果只保留元数据。授权与 browser_snapshot 同为
读取档;png 超过字节上限时自动降级为 jpeg 重试一次。
输入框增加 onPaste、composer-box 增加 onDrop:截图后可直接 ⌘V
粘贴或拖入图片,复用现有上传校验(数量、字节、尺寸、媒体类型);
宿主不支持图片输入时给出与上传按钮相同的提示。
chrome.tabs.captureVisibleTab 只接受 <all_urls> 或用户手势激活的
activeTab,域名级 http/https host 权限不满足要求,实测报
"Either the '<all_urls>' or 'activeTab' permission is required"。
两个平台的清单统一改为 <all_urls>。
截图拍到的是用户屏幕上实际可见的内容,可能包含密码、私信等
敏感信息,比读文本更敏感:不再跟随"页面内容共享"设置,每次
调用都弹审批;只有"完全控制浏览器"开启时跳过。
@graydovee

Copy link
Copy Markdown
Author

注意到 #116 也实现了截图能力。对比了两边的实现,主要差异如下:

图片交付:#116 将像素写入本地文件、向模型返回路径,页面通道保持纯文本优先;当前 PR 将图片存入附件存储后以内容块直接进入模型上下文。由此当前 PR 每次截图少一次工具调用——#116 的模型拿到路径后,需再调用读文件工具才能看到图片;连续多张截图时该差异会成倍放大。纯文本模型由运行时投影为占位符,不受影响。

权限路线:#116 采用 optional debugger 与 CDP 回退,权限授予经由设置页的用户手势完成;当前 PR 采用 <all_urls> 主机权限。一个实测发现:captureVisibleTab 不接受域名级 http/https 权限,会报 Either the '<all_urls>' or 'activeTab' permission is required;清单中的 activeTab 亦不构成豁免(仅在用户手势触发时临时生效)。#116 的 CDP 回退不依赖该权限,但每次截图需附加调试器,伴随“正在被调试”横幅、与浏览器自带 DevTools 互斥、Firefox 上不可用。

审批:当前 PR 对每次调用弹出侧边栏审批,仅“完全控制浏览器”可跳过。

受保护页面:chrome:// 等页面上所有工具均被扩展拒绝执行;#116 单独放行截图(captureVisibleTab 由浏览器在渲染层直接拍摄,无需向页面注入脚本);当前 PR 在这些页面拒绝截图并说明原因。

附带改动:当前 PR 另包含侧边栏粘贴、拖拽添加图片。

@graydovee graydovee closed this Oct 16, 2026
@graydovee
graydovee deleted the feat/screenshot branch October 16, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant