Skip to content

feat: add optional browser annotations with screenshot and element context - #796

Open
agentone666-ship-it wants to merge 5 commits into
dataelement:mainfrom
agentone666-ship-it:codex/browser-annotations-plugin
Open

agentone666-ship-it wants to merge 5 commits into
dataelement:mainfrom
agentone666-ship-it:codex/browser-annotations-plugin

Conversation

@agentone666-ship-it

@agentone666-ship-it agentone666-ship-it commented Oct 27, 2026 •

Copy link
Copy Markdown

Behavior

Adds an optional browser beside the conversation: native sidebar tabs, URL/search navigation, element/region annotations with anchored input, and hold-Space browsing. The composer shows a screenshot review popover. One send carries real image blocks, element/DOM/style evidence and comments; sent messages show a readable summary with optional detailed review.

The plugin is not installed/enabled by default. The only host patch addition is data-dsh-composer-primary="send" | "stop" on the native primary button. Existing native handlers and slot contracts stay intact; a narrow, documented adapter observes that marker, preserves Stop and disables annotation submission when the marker is unavailable. The marker can be removed when upstream exposes an equivalent public composer submission API.

Implementation

  • packages/dsh-page-review 0.4.2 owns its build/lockfile, public sidebar/accessory/chat-node contributions and Cordis lifecycle cleanup.
  • Private Puppeteer code is bundled into a standalone CJS backend with third-party license notices. React remains host-owned. The archive launches without a runtime dependency install and passes the normal Desktop generation closure check.
  • Native file/session reference chips use the source owner's codec. Draft revisions and attachments survive failed preparation/admission. Explicit rejection allows a repaired packet; uncertain transport retries retain their request identity.
  • Successful admission removes submitted attachments using the native keep-only contract, while preserving draft edits and attachments added during sending. Regressions model that contract rather than merely asserting a callback call.
  • Sent-message screenshots use the host's authorized content factory/image loader. Full model metadata remains durable while ordinary chat stays readable.

PDF/Word/Markdown preview annotation UI and side-chat/session-mail work are outside this PR. Current main lacks the incoming prototype's legacy preview integration contract; compatible packet/location helpers remain tested.

Validation

  • Optional plugin: clean npm ci, build, 82/82 tests passed, zero skipped/cancelled. Includes real Chrome lifecycle/reconnection, extracted archive startup, license coverage, reference serialization, readable message projection, admission retry behavior and concurrent attachment retirement. Final 0.4.2 run: 43.5s.
  • Normal Desktop generation install/update exercised through the 0.4.2 packed archive. Running generation client bytes match the build.
  • Real macOS Desktop 0.12.1 / Harness 0.2.1-rc.2 verified: generic new-tab chooser; native tabs; element/region annotations; hold-Space click/release; composer and sent screenshot popovers; annotation-only and combined draft + native reference + image + annotation sends; successful input cleanup; running queue and native Stop; dark/light appearance; narrow window; Chinese/English labels; native plugin disable/re-enable and single submission after reload. The locale picker itself did not expand during the UI attempt, so English was selected through the isolated profile configuration.
  • Clean root install/postinstall patch replay passed earlier. Final root typecheck and Electron bundling passed. git diff --check is clean.
  • Root full regression and final build are not all green. Two-worker run: 2203 passed / 1 DOCX timeout; serial run: 2195 passed / 9 Office timeouts. A later isolated DOCX check also timed out. Repeated full build and the latest npm test pretest stopped in unchanged Office template verification. No assertions or deadlines were weakened. After unlock, the direct complete Vitest run (using the unchanged earlier template cache after that pretest failure) finished with 2186 passed / 18 failed, 219 files, in 957.2s. Failures were in Office runtime and PPT import/template suites, including timeouts and startup/progress assertions. Those files are unchanged by this PR, but a matching upstream baseline is not established; they are not certified as pre-existing. See the validation record for evidence boundaries.

The deterministic local model validates transport and queue mechanics, not a commercial model's visual reasoning. Windows/Linux desktop behavior remains unverified; the optional-package CI matrix is defined, but GitHub reported no check results at the last inspection. Ready for review with these limitations disclosed, not an all-green repository or release certification.

Details: docs/browser-annotations-validation.md.

@agentone666-ship-it
agentone666-ship-it marked this pull request as ready for review October 27, 2026 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant