Repository navigation
Conversation
扩展侧用 chrome.tabs.captureVisibleTab 截取受控标签页的可视区域, 返回 base64 与媒体类型;宿主侧把像素存入附件存储,以图片内容块 返回给模型,持久化结果只保留元数据。授权与 browser_snapshot 同为 读取档;png 超过字节上限时自动降级为 jpeg 重试一次。
输入框增加 onPaste、composer-box 增加 onDrop:截图后可直接 ⌘V 粘贴或拖入图片,复用现有上传校验(数量、字节、尺寸、媒体类型); 宿主不支持图片输入时给出与上传按钮相同的提示。
chrome.tabs.captureVisibleTab 只接受 <all_urls> 或用户手势激活的 activeTab,域名级 http/https host 权限不满足要求,实测报 "Either the '<all_urls>' or 'activeTab' permission is required"。 两个平台的清单统一改为 <all_urls>。
截图拍到的是用户屏幕上实际可见的内容,可能包含密码、私信等 敏感信息,比读文本更敏感:不再跟随"页面内容共享"设置,每次 调用都弹审批;只有"完全控制浏览器"开启时跳过。
Author
|
注意到 #116 也实现了截图能力。对比了两边的实现,主要差异如下: 图片交付:#116 将像素写入本地文件、向模型返回路径,页面通道保持纯文本优先;当前 PR 将图片存入附件存储后以内容块直接进入模型上下文。由此当前 PR 每次截图少一次工具调用——#116 的模型拿到路径后,需再调用读文件工具才能看到图片;连续多张截图时该差异会成倍放大。纯文本模型由运行时投影为占位符,不受影响。 权限路线:#116 采用 optional 审批:当前 PR 对每次调用弹出侧边栏审批,仅“完全控制浏览器”可跳过。 受保护页面:chrome:// 等页面上所有工具均被扩展拒绝执行;#116 单独放行截图( 附带改动:当前 PR 另包含侧边栏粘贴、拖拽添加图片。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概述 / Summary
Adds
browser_screenshot: it captures the visible viewport of the controlled tab and returns the image as an image content block, so vision-capable models read it directly; text-only models get the runtime's placeholder projection and are unaffected. A screenshot photographs whatever is actually on the user's screen — passwords and private messages included — so every call requires an approval prompt in the side panel. The sidebar also gains paste and drag-and-drop image intake. Related to #75.改动内容 / Changes
Screenshot tool:
chrome.tabs.captureVisibleTabon the controlled tab, with png / jpeg format and quality optionsApproval:
<all_urls>:captureVisibleTabaccepts only<all_urls>or a gesture-activated activeTab; origin-scoped http/https permissions are not acceptedSidebar:
测试 / Testing
pnpm run check:runtime/typecheck/build/test:smokepnpm run test: bridge 157/157, extension 411/411影响范围 / Impact
<all_urls>(the admission requirement ofcaptureVisibleTab; origin-scoped permissions are not accepted)Checklist
pnpm run build/bash -n scripts/install.sh等基础检查 / Passed basic checks