Skip to content

Commit 5712053

Browse files
committed
refactor(extension): consolidate page capture in the chat composer
1 parent 9c1800a commit 5712053

20 files changed

Lines changed: 473 additions & 172 deletions

File tree

‎.github/workflows/browser-extension.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ on:
1515
- 'apps/sim/lib/api/contracts/mothership-chat-context.ts'
1616
- 'apps/sim/proxy.ts'
1717
- 'apps/sim/next.config.ts'
18+
- 'apps/sim/package.json'
1819
- 'packages/emcn/**'
1920
- 'bun.lock'
2021
workflow_dispatch:

‎apps/extension/DESIGN.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,11 @@ and workspace authorization. A conversation can continue in the full app.
1111

1212
The user signs in in a normal Sim tab, chooses a workspace, opens or creates a
1313
conversation, and explicitly attaches the current page or highlighted text.
14-
Pending snapshots sit beside the composer as compact removable attachments. Opening one
14+
The composer’s + menu offers Current page and Selected text. Pending snapshots sit
15+
inside the composer as compact removable chips alongside file attachments. Opening one
1516
shows the captured text, source, capture time, and conversation-retention disclosure.
1617
Partial captures remain labeled in the attachment row. Site access lives in the
17-
extension options menu; successful delivery adds no persistent status banner.
18+
composer’s Site access action, which opens an extension-owned dialog; successful delivery adds no persistent status banner.
1819
They are immutable: navigating to another page cannot silently change an attached
1920
source. Removing an attachment, changing workspace, or starting a new conversation
2021
clears pending page context. Several explicitly attached pages can support comparison.

‎apps/extension/README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ bun run --cwd apps/extension build
1313

1414
In Chrome, open `chrome://extensions`, enable Developer mode, choose **Load unpacked**, and select `apps/extension/dist`. Copy the extension ID into the Sim server's `BROWSER_EXTENSION_IDS` configuration. It accepts comma-separated Chrome extension IDs. Restart the app after changing that configuration. Unconfigured IDs cannot embed the app.
1515

16-
Click the Sim toolbar icon to open chat and attach the current page. **Attach page** adds a fresh snapshot. Attach up to three pages or selections to summarize, extract facts, or compare sources. To quote a passage, select it and choose **Ask Sim about selection** from the context menu. Review or remove the attachment before sending. The **Extension options** menu keeps site permissions out of the chat: **Allow access to [site]** grants access only to that site; **Remove access to [site]** revokes it. Persistent site permission never enables background capture.
16+
Click the Sim toolbar icon to open chat and attach the current page. **+ → Current page** adds a fresh snapshot; **Selected text** attaches the highlighted passage. Attachments appear as chips inside the composer. Attach up to three pages or selections to summarize, extract facts, or compare sources. To quote a passage, select it and choose **Ask Sim about selection** from the context menu. Review or remove the attachment before sending. **+ → Site access** opens the extension-owned permission dialog: **Allow access to [site]** grants access only to that site; **Remove access to [site]** revokes it. Persistent site permission never enables background capture.
1717

1818
For local development:
1919

@@ -32,7 +32,7 @@ SIM_EXTENSION_DEV=1 SIM_EXTENSION_ORIGIN=http://localhost:3000 bun run --cwd app
3232
- Source URLs omit credentials, query parameters, and fragments. Human-readable text can still contain sensitive information; it is the user-selected material they review and send into their conversation.
3333
- Snapshots stay in memory until handed to Sim and acknowledged or expired after 30 seconds. Extension storage contains only a short-lived intent with a tab/document identity and a SHA-256 URL digest, never raw URLs, captured text, or credentials. Closing a tab or navigating invalidates its pending intent. Sim applies its normal conversation retention once a message is sent.
3434
- No browser history, bookmarks, cookie access, debugger access, remote extension JavaScript, arbitrary network proxy, or background content collection is requested. Incognito use is disabled.
35-
- The iframe bridge checks both the exact configured origin and the exact frame, then validates versioned messages. The hosted page may acknowledge readiness and receipt; it cannot request privileged browser actions through the bridge.
35+
- The iframe bridge checks both the exact configured origin and the exact frame, then validates versioned messages. The hosted composer may request only page capture, selection capture, or the native site-access dialog. Requests require the current conversation nonce and Chrome transient user activation; arbitrary URLs, tabs, script execution, and network operations are never accepted. Permission grants still require a click in the extension-owned dialog.
3636
- Captures and queued toolbar intents are scoped to the receiving conversation. Changing conversation, workspace, or account discards an unfinished handoff. Captured attachments remain explicit snapshots when the source tab changes.
3737
- The complete hosted document, including portaled source dialogs and menus, is blocked from session replay. Clipboard writing is delegated only to the configured Sim origin. Clipboard reading and microphone capture are not delegated; voice input remains available in the main app.
3838

@@ -44,7 +44,7 @@ EXTENSION_REPORT_PATH=test-results/extension-report.json bun run --cwd apps/exte
4444
bun run --cwd apps/extension build
4545
```
4646

47-
The E2E suite uses real Chromium, the packaged extension, and a disposable HTTP app fixture. It pregrants only the fixture page origin in a temporary package to exercise extraction without automating the browser's permission dialog. It checks HttpOnly session-cookie transport, origin/frame spoofing, sensitive-field exclusion, source URL minimization, selected excerpts, UTF-8 limits, stale-document handoffs, denied host access, and absence of page bytes in extension storage. The cookie check uses a fixture, not Better Auth or SSO. JSON results and a screenshot are written beside the caller-supplied report path. The E2E build points `dist` at its temporary fixture origin; rebuild normally before loading or distributing it.
47+
The E2E suite uses real Chromium, the packaged extension, and a disposable HTTP app fixture. It pregrants only the fixture page origin in a temporary package to exercise extraction without automating the browser's permission dialog. It checks HttpOnly session-cookie transport, origin/frame spoofing, sensitive-field exclusion, source URL minimization, selected excerpts, UTF-8 limits, stale-document handoffs, denied host access, and absence of page bytes in extension storage. The cookie check uses a fixture, not Better Auth or SSO. JSON results and a screenshot are written beside the caller-supplied report path. The E2E suite builds into a temporary directory and leaves shipping `dist` untouched.
4848

4949
Release verification must additionally exercise the native toolbar and shortcut, Chrome's real grant/revoke prompt, sign-in/SSO and expiry against the running Sim app, side-panel closing/reopening, workspace changes, narrow widths, and the real Mothership streaming response. The fixture suite does not establish those integrations. The design document records launch scope and release criteria.
5050

@@ -58,7 +58,7 @@ SIM_EXTENSION_KEY="$PUBLIC_EXTENSION_KEY" \
5858
bun run --cwd apps/sim test:browser-extension:e2e
5959
```
6060

61-
Configure the matching extension ID in the running app. The suite creates its own accounts/workspaces and removes them afterward. It verifies real authentication, workspace denial, page limits, navigation, source links, clipboard writes, actual replay-recorder exclusion, themes, widths, and sign-out. CI provisions this environment automatically. Set `EXTENSION_APP_E2E_LIVE_CHAT=1` when the app is connected to a working local Mothership instance to additionally verify live rendering, saved page context, queue editing, and drafts through chat admission. Provider credentials belong only in that server's environment. Reports and screenshots are written beside the supplied report path. Native toolbar/shortcut/context-menu and permission-prompt checks remain manual release checks.
61+
Configure the matching extension ID in the running app. The suite builds into a temporary directory, creates its own accounts/workspaces, and removes them afterward; it leaves the shipping `dist` untouched. It verifies real authentication, workspace denial, page limits, navigation, source links, clipboard writes, actual replay-recorder exclusion, themes, widths, and sign-out. CI provisions this isolated environment with `next dev`; the main checks workflow separately validates the production build. This keeps the disposable extension origin and identity independent of the shared app build. Set `EXTENSION_APP_E2E_LIVE_CHAT=1` when the app is connected to a working local Mothership instance to additionally verify live rendering, saved page context, queue editing, and drafts through chat admission. Provider credentials belong only in that server's environment. Reports and screenshots are written beside the supplied report path. Native toolbar/shortcut/context-menu and permission-prompt checks remain manual release checks.
6262

6363
## Distribution
6464

‎apps/extension/scripts/build.ts‎

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import { createPublicKey } from 'node:crypto'
12
import { cp, mkdir, readFile, rm, writeFile } from 'node:fs/promises'
23
import { dirname, resolve } from 'node:path'
34
import { fileURLToPath } from 'node:url'
@@ -22,11 +23,26 @@ if (
2223
)
2324
}
2425
const key = process.env.SIM_EXTENSION_KEY
25-
if (key && (!/^[A-Za-z0-9+/]+={0,2}$/.test(key) || key.length > 4_096))
26-
throw new Error('SIM_EXTENSION_KEY must be a base64 public key.')
26+
if (key) {
27+
try {
28+
if (!/^[A-Za-z0-9+/]+={0,2}$/.test(key) || key.length > 4_096)
29+
throw new Error('Invalid encoding')
30+
const parsed = createPublicKey({ key: Buffer.from(key, 'base64'), format: 'der', type: 'spki' })
31+
if (
32+
parsed.asymmetricKeyType !== 'rsa' ||
33+
parsed.export({ format: 'der', type: 'spki' }).toString('base64') !== key
34+
)
35+
throw new Error('Invalid public key')
36+
} catch {
37+
throw new Error('SIM_EXTENSION_KEY must be a base64 RSA public key in SPKI format.')
38+
}
39+
}
2740

28-
const dist = resolve(root, 'dist')
29-
await rm(dist, { recursive: true, force: true })
41+
const outputArgument = process.argv.find((argument) => argument.startsWith('--outdir='))
42+
const outputDirectory = outputArgument?.slice('--outdir='.length)
43+
if (outputArgument && !outputDirectory) throw new Error('--outdir requires a directory.')
44+
const dist = resolve(root, outputDirectory ?? 'dist')
45+
if (!outputArgument) await rm(dist, { recursive: true, force: true })
3046
await mkdir(dist, { recursive: true })
3147
await build({
3248
absWorkingDir: root,

‎apps/extension/scripts/test-e2e.ts‎

Lines changed: 149 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,11 @@
11
import { spawnSync } from 'node:child_process'
2-
import { cp, mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
2+
import { generateKeyPairSync } from 'node:crypto'
3+
import { existsSync } from 'node:fs'
4+
import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
35
import { tmpdir } from 'node:os'
46
import { dirname, resolve } from 'node:path'
57
import { fileURLToPath } from 'node:url'
6-
import { type BrowserContext, chromium, type Frame, type Page } from '@playwright/test'
8+
import { type BrowserContext, chromium, expect, type Frame, type Page } from '@playwright/test'
79
import { pageSnapshotSchema } from '@sim/browser-extension-protocol'
810
import { createLogger } from '@sim/logger'
911
import { getErrorMessage } from '@sim/utils/errors'
@@ -83,15 +85,32 @@ async function check(name: string, run: () => Promise<void>) {
8385
}
8486

8587
try {
88+
await check('The build rejects private key material before creating output', async () => {
89+
const { privateKey } = generateKeyPairSync('rsa', { modulusLength: 2048 })
90+
const rejectedOutput = resolve(temporary, 'rejected-key')
91+
const result = spawnSync(process.execPath, ['scripts/build.ts', `--outdir=${rejectedOutput}`], {
92+
cwd: root,
93+
env: {
94+
...process.env,
95+
SIM_EXTENSION_KEY: privateKey.export({ type: 'pkcs8', format: 'der' }).toString('base64'),
96+
},
97+
encoding: 'utf8',
98+
})
99+
assert(result.status !== 0, 'The build accepted private key material')
100+
assert(
101+
result.stderr.includes('RSA public key in SPKI format'),
102+
'The build failed for an unrelated reason'
103+
)
104+
assert(!existsSync(rejectedOutput), 'Invalid key material reached build output')
105+
})
86106
const origin = `http://127.0.0.1:${fixture.port}`
87107
const pageOrigin = `http://localhost:${fixture.port}`
88-
const built = spawnSync(process.execPath, ['scripts/build.ts'], {
108+
const built = spawnSync(process.execPath, ['scripts/build.ts', `--outdir=${extensionPath}`], {
89109
cwd: root,
90110
env: { ...process.env, SIM_EXTENSION_ORIGIN: origin, SIM_EXTENSION_DEV: '1' },
91111
encoding: 'utf8',
92112
})
93113
assert(built.status === 0, `Extension build failed: ${built.stderr}`)
94-
await cp(resolve(root, 'dist'), extensionPath, { recursive: true })
95114
const manifest = JSON.parse(await readFile(resolve(extensionPath, 'manifest.json'), 'utf8'))
96115
// The test package pregrants its disposable fixture site, equivalent to an accepted site prompt.
97116
manifest.host_permissions.push(`${pageOrigin}/*`)
@@ -109,7 +128,7 @@ try {
109128
await signIn.close()
110129
const panel: Page = await context.newPage()
111130
await panel.goto(`chrome-extension://${extensionId}/sidepanel.html`)
112-
await panel.getByRole('button', { name: 'Attach page', exact: true }).waitFor()
131+
await panel.getByTitle('Sim chat').waitFor()
113132
const frame: Frame = panel.frames()[1] ?? (await panel.waitForEvent('frameattached'))
114133
const app = panel.frames().find((candidate) => candidate.url().startsWith(origin)) ?? frame
115134
assert(app, 'Hosted frame did not load')
@@ -123,18 +142,38 @@ try {
123142
const page = await context.newPage()
124143
await page.goto(`${pageOrigin}/ticket?access_token=URL_SECRET#secret`)
125144
await page.bringToFront()
126-
const capture = (name = 'Attach page') =>
127-
panel
128-
.getByRole('button', { name, exact: true })
129-
.evaluate((button) => (button as HTMLButtonElement).click())
145+
let receiverReady = false
146+
const capture = async (name = 'Attach page') => {
147+
if (!receiverReady) {
148+
const intent = await pendingIntent()
149+
await worker.evaluate(async (intent) => {
150+
await chrome.storage.session.set({ [`captureIntent:${intent.windowId}:initial`]: intent })
151+
}, intent)
152+
return
153+
}
154+
await app.evaluate(
155+
({ sessionId, action, origin }) => {
156+
parent.postMessage(
157+
{ channel: 'sim-extension', version: 1, type: 'action', sessionId, action },
158+
origin
159+
)
160+
},
161+
{
162+
sessionId: receiverSessionId,
163+
action: name === 'Selection' ? 'capture-selection' : 'capture-page',
164+
origin: `chrome-extension://${extensionId}`,
165+
}
166+
)
167+
}
130168
const snapshots = async () =>
131169
pageSnapshotSchema.array().parse(JSON.parse(await app.locator('#received').innerText()))
132170
let receiverSessionId = generateId()
133171
await app.evaluate((sessionId) => {
134172
document.body.dataset.sessionId = sessionId
135173
}, receiverSessionId)
136-
const ready = () =>
137-
app.evaluate(
174+
const ready = () => {
175+
receiverReady = true
176+
return app.evaluate(
138177
({ targetOrigin, sessionId }) => {
139178
document.body.dataset.sessionId = sessionId
140179
parent.postMessage(
@@ -144,6 +183,7 @@ try {
144183
},
145184
{ targetOrigin: `chrome-extension://${extensionId}`, sessionId: receiverSessionId }
146185
)
186+
}
147187
const pendingIntent = () =>
148188
worker.evaluate(async () => {
149189
const [tab] = await chrome.tabs.query({ active: true, currentWindow: true })
@@ -210,8 +250,6 @@ try {
210250
await page.evaluate(() => {
211251
document.body.textContent = 'SECOND_CAPTURE_SENTINEL'
212252
})
213-
await capture()
214-
await panel.getByRole('status').filter({ hasText: 'previous capture' }).waitFor()
215253
await ready()
216254
await panel.waitForFunction(() => document.querySelector('[role="status"]')?.textContent === '')
217255
assert(
@@ -475,6 +513,104 @@ try {
475513
assert((await snapshots()).length === 5, 'Unvalidated selection ancestors were traversed')
476514
await page.reload()
477515
})
516+
await check('Unicode and escaped text fit within the complete snapshot budget', async () => {
517+
await page.goto(`${pageOrigin}/${'a'.repeat(4_000)}`)
518+
await page.evaluate(() => {
519+
document.title = '"'.repeat(512)
520+
document.body.textContent = '界'.repeat(45_000) + '"'.repeat(5_000)
521+
})
522+
const before = (await snapshots()).length
523+
await capture()
524+
await app.waitForFunction(
525+
(count) =>
526+
JSON.parse(document.getElementById('received')?.textContent ?? '[]').length === count,
527+
before + 1
528+
)
529+
const snapshot = pageSnapshotSchema.parse((await snapshots()).at(-1))
530+
assert(snapshot.truncated, 'Oversized serialized text was not marked partial')
531+
assert(
532+
snapshot.text.startsWith('界') && snapshot.text.includes('"'),
533+
'Capture lost the expected page content'
534+
)
535+
})
536+
await check(
537+
'Composer capture requests require a current user gesture and conversation',
538+
async () => {
539+
const before = (await snapshots()).length
540+
await panel.evaluate(() => {
541+
let queries = 0
542+
let actions = 0
543+
const originalQuery = chrome.tabs.query.bind(chrome.tabs)
544+
chrome.tabs.query = ((
545+
query: chrome.tabs.QueryInfo,
546+
callback?: (tabs: chrome.tabs.Tab[]) => void
547+
) => {
548+
queries++
549+
return callback ? originalQuery(query, callback) : originalQuery(query)
550+
}) as typeof chrome.tabs.query
551+
window.addEventListener('message', (event: MessageEvent<unknown>) => {
552+
if (
553+
event.source !== document.querySelector('iframe')?.contentWindow ||
554+
typeof event.data !== 'object' ||
555+
event.data === null ||
556+
!('type' in event.data) ||
557+
event.data.type !== 'action'
558+
)
559+
return
560+
document.body.dataset.actionQueries = String(queries)
561+
document.body.dataset.observedActions = String(++actions)
562+
})
563+
})
564+
const parentCdp = await panel.context().newCDPSession(panel)
565+
const cdp = await panel.context().newCDPSession(app)
566+
await expect
567+
.poll(
568+
async () => {
569+
const result = await parentCdp.send('Runtime.evaluate', {
570+
expression: 'navigator.userActivation.isActive',
571+
userGesture: false,
572+
})
573+
return result.result.value
574+
},
575+
{ timeout: 10_000 }
576+
)
577+
.toBe(false)
578+
const action = {
579+
channel: 'sim-extension',
580+
version: 1,
581+
type: 'action',
582+
sessionId: receiverSessionId,
583+
action: 'capture-page',
584+
}
585+
await cdp.send('Runtime.evaluate', {
586+
expression: `parent.postMessage(${JSON.stringify(action)}, ${JSON.stringify(`chrome-extension://${extensionId}`)})`,
587+
userGesture: false,
588+
})
589+
await expect(panel.locator('body')).toHaveAttribute('data-observed-actions', '1')
590+
await expect(panel.locator('body')).toHaveAttribute('data-action-queries', '0')
591+
assert((await snapshots()).length === before, 'Captured without a user gesture')
592+
await app.evaluate(({ action, targetOrigin }) => parent.postMessage(action, targetOrigin), {
593+
action: { ...action, sessionId: generateId() },
594+
targetOrigin: `chrome-extension://${extensionId}`,
595+
})
596+
await expect(panel.locator('body')).toHaveAttribute('data-observed-actions', '2')
597+
await expect(panel.locator('body')).toHaveAttribute('data-action-queries', '0')
598+
assert((await snapshots()).length === before, 'Captured for an obsolete conversation')
599+
await capture()
600+
await app.waitForFunction(
601+
(count) =>
602+
JSON.parse(document.getElementById('received')?.textContent ?? '[]').length === count,
603+
before + 1
604+
)
605+
await expect(panel.locator('body')).toHaveAttribute('data-observed-actions', '3')
606+
assert(
607+
Number(await panel.locator('body').getAttribute('data-action-queries')) > 0,
608+
'Allowed capture did not reach the browser API'
609+
)
610+
await cdp.detach()
611+
await parentCdp.detach()
612+
}
613+
)
478614
await mkdir(dirname(output), { recursive: true })
479615
await panel.screenshot({ path: output.replace(/\.json$/, '.png'), fullPage: true })
480616
} catch (error) {

0 commit comments

Comments
 (0)