diff --git a/.github/CI.md b/.github/CI.md index 9fbeb94e472..61fffc8d844 100644 --- a/.github/CI.md +++ b/.github/CI.md @@ -24,7 +24,7 @@ SEO depends on workspace UI and server code: landing pages reuse workspace prici Affected type-checks follow the package graph too; repository audits and lint run whenever the checks workflow runs. Workspace-only unit jobs use a free GitHub runner and do not start the app test shards. -Database integration and lightweight HTTP suites run on free standard GitHub runners with provider-specific dependency setup. A same-source integration comparison of 16 jobs per provider (32 total) had a slowest shard of 6m 04s on 2-vCPU Blacksmith versus 6m 55s on free GitHub, with identical coverage. GitHub trades the observed 51 seconds for zero runner compute cost. This is one paired sample, not a long-run percentile estimate. These explicitly selected GitHub jobs do not follow `CI_PROVIDER`. +Database integration uses 4-vCPU Blacksmith runners with sticky dependency disks, following `CI_PROVIDER` and falling back to standard GitHub runners with matching setup. A prior same-source comparison of 16 jobs per provider (32 total) had a slowest shard of 6m 04s on 2-vCPU Blacksmith versus 6m 55s on GitHub, with identical coverage. This is one paired sample, not a long-run percentile estimate. Evaluate runner changes against complete coverage and the overall CI gate, since a faster shard does not shorten a run when another job finishes later. Lightweight HTTP suites retain free standard GitHub runners and do not follow `CI_PROVIDER`. GitHub runners install dependencies directly rather than transferring Bun and `node_modules` archives. In a same-source paired comparison, those restores took 26.2 seconds before an 82-millisecond install, while a clean install took 12.42 seconds. This is one paired observation; retain the policy only while repeated CI timings support it. Blacksmith keeps its sticky dependency disks. GitHub Turbo output caches remain enabled for jobs that request them, with a commit-specific key and a task/event/fork-scoped restore prefix, so a rerun can reuse an exact entry without creating another archive. Push jobs never restore pull-request entries. @@ -34,10 +34,12 @@ Every HTTP and browser group consumes the same production build in an isolated j Desktop inbox protocol, assistant actions UI, and organization search UI run as three parallel HTTP jobs with isolated databases, apps, and reports. They retain one acceptance task boundary, so changing any of their harnesses selects all three jobs. Desktop inbox needs no browser. The UI jobs and browser extension install Chromium while their app starts, waiting for both before executing assertions; browser-install failures still fail the job and preserve the install log. No acceptance assertions are removed by the split. -Browser extension acceptance keeps native streaming and first-open capture checks on real timers. Its access-revalidation scenarios then reopen the persistent browser profile and install Playwright's clock before the app mounts. Advancing the production polling timers preserves real HTTP and database authorization checks without waiting through each polling interval. Capture timestamps originate in Chromium's isolated world, so the harness restores wall time after each advance. Live-provider runs retain native timing throughout. +Browser extension acceptance runs native interaction and access-revalidation groups in parallel jobs with isolated databases, apps, browser profiles, and reports. Both retain the same task boundary, so an affected extension harness selects both groups. Native streaming and first-open capture checks retain real timers. Access checks install Playwright's clock before the app mounts: advancing production polling timers preserves real HTTP and database authorization checks without waiting through each polling interval. Capture timestamps originate in Chromium's isolated world, so the harness restores wall time after each advance. The local `browser-extension` group still runs the complete suite; live-provider runs require this complete mode and retain native timing throughout. Integration shard weights come from successful reports for both push and migration provisioning paths. Refresh them with `scripts/update-integration-durations.ts`, then check the new partition against a separate complete run before adopting it. Weights only change scheduling; discovery still owns coverage. The current refresh uses runs [41348786822](https://github.com/simstudioai/sim/actions/runs/41348786822) and [41377145857](https://github.com/simstudioai/sim/actions/runs/41377145857), with [41377810718](https://github.com/simstudioai/sim/actions/runs/41377810718) held out for validation. +Desktop pickup checks overlap their real grace periods using distinct calls and waiters under fixed authorization and parent-run fixtures. Recovery exhaustion checks keep worker responses, injected faults, and observed requests scoped to each stream before overlapping their real backoff schedules. Their immediate-takeover checks remain serial. File isolation and serial execution between integration files remain unchanged; these suites still verify actual timeout, lease, retry-floor, and late-claim behavior against PostgreSQL and Redis. + SCIM's production app uses a `sim.ai` fixture hostname mapped to loopback, since `NEXT_PUBLIC_FORCE_HOSTED` is ignored in production. Readiness, HTTP tests, and internal API requests still connect to loopback. The SCIM suite verifies that ending an Enterprise subscription rejects both owner administration and an existing bearer token, and that restoring it restores access. ## Coverage removed or consolidated diff --git a/.github/ci-suites.json b/.github/ci-suites.json index c8ad7304328..df2dc16de3a 100644 --- a/.github/ci-suites.json +++ b/.github/ci-suites.json @@ -25,7 +25,11 @@ "task": "ci:e2e-desktop-inbox" }, { - "group": "browser-extension", + "group": "browser-extension-native", + "task": "ci:e2e-browser-extension" + }, + { + "group": "browser-extension-access", "task": "ci:e2e-browser-extension" }, { diff --git a/.github/scripts/http-e2e.sh b/.github/scripts/http-e2e.sh index dce1ca6d394..cb86412ab48 100755 --- a/.github/scripts/http-e2e.sh +++ b/.github/scripts/http-e2e.sh @@ -3,7 +3,7 @@ # CI groups, which reuse the production build from `e2e-build`. # The job extracts it into apps/sim/.next before running these suites. # -# Usage: http-e2e.sh (run from apps/sim) +# Usage: http-e2e.sh (run from apps/sim) # # The job provides DATABASE_URL, BETTER_AUTH_SECRET and ENCRYPTION_KEY; each group sets the rest of # its app's environment here. Reports and server logs land in $RUNNER_TEMP/e2e. @@ -20,7 +20,7 @@ # telemetry flush runs detached and still writes .next/dev). set -euo pipefail -group=${1:?usage: http-e2e.sh } +group=${1:?usage: http-e2e.sh } report_dir="$RUNNER_TEMP/e2e" ready_timeout_seconds=600 mkdir -p "$report_dir" @@ -264,13 +264,18 @@ case "$group" in esac ;; - browser-extension) + browser-extension|browser-extension-native|browser-extension-access) + extension_group=all + case "$group" in + browser-extension-native) extension_group=native ;; + browser-extension-access) extension_group=access ;; + esac export REDIS_URL=redis://127.0.0.1:6379 export NEXT_PUBLIC_FORCE_HOSTED=false export KNOWLEDGE_MEMBER_ACCESS=true CREDENTIAL_GROUPS=true ACCESS_CONTROL_ENABLED=true export NEXT_PUBLIC_ACCESS_CONTROL_ENABLED=true export API_ENCRYPTION_KEY="$ENCRYPTION_KEY" - start_browser_install browser-extension chromium + start_browser_install "$group" chromium extension_identity=$(bun - <<'JS' import { generateKeyPairSync, createHash } from 'node:crypto' const { publicKey } = generateKeyPairSync('rsa', { modulusLength: 2048 }) @@ -282,11 +287,12 @@ JS ) export SIM_EXTENSION_KEY="${extension_identity%%$'\n'*}" export BROWSER_EXTENSION_IDS="${extension_identity#*$'\n'}" - start_app browser-extension 3007 'browser extension' record-http-status + start_app "$group" 3007 'browser extension' record-http-status wait_for_browsers EXTENSION_APP_E2E_BASE_URL="$local_app_url" \ EXTENSION_APP_E2E_DATABASE_URL="$DATABASE_URL" \ - EXTENSION_APP_E2E_REPORT_PATH="$report_dir/browser-extension/report.json" \ + EXTENSION_APP_E2E_GROUP="$extension_group" \ + EXTENSION_APP_E2E_REPORT_PATH="$report_dir/$group/report.json" \ bun run test:browser-extension:e2e ;; diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index ae181e80978..53c50e749d8 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -76,7 +76,7 @@ jobs: name: integration (${{ matrix.provision }}, ${{ matrix.shard == 'db' && 'db' || format('{0}/8', matrix.shard) }}) needs: changes if: needs.changes.outputs.integration == 'true' - runs-on: ubuntu-24.04 + runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }} timeout-minutes: 20 strategy: fail-fast: false @@ -135,7 +135,7 @@ jobs: - name: Setup workspace uses: ./.github/actions/setup with: - provider: github + provider: ${{ vars.CI_PROVIDER }} - name: Provision a fresh database through the supported command working-directory: packages/db @@ -219,7 +219,7 @@ jobs: --health-retries 10 # Desktop inbox, browser extension, and Project Files use Redis. redis: - image: ${{ (matrix.group == 'desktop-inbox' || matrix.group == 'assistant-actions' || matrix.group == 'organization-search' || matrix.group == 'browser-extension' || matrix.group == 'project-files') && 'redis:8.2-alpine' || '' }} + image: ${{ (matrix.group == 'desktop-inbox' || matrix.group == 'assistant-actions' || matrix.group == 'organization-search' || startsWith(matrix.group, 'browser-extension-') || matrix.group == 'project-files') && 'redis:8.2-alpine' || '' }} ports: - 6379:6379 options: >- @@ -298,7 +298,7 @@ jobs: name: e2e-reports-${{ matrix.group }} path: | ${{ runner.temp }}/e2e/**/*.json - ${{ runner.temp }}/e2e/browser-extension/**/*.png + ${{ runner.temp }}/e2e/browser-extension*/**/*.png ${{ runner.temp }}/e2e/*-ui-report.json.artifacts/*.png if-no-files-found: ignore retention-days: 7 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c2d2c3a5e66..b032f2f7a72 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -8,8 +8,8 @@ name: ci # A repo variable, not a committed value: during a Blacksmith outage there is no # working CI to merge a switchover through. Only unset/'blacksmith' selects # Blacksmith; anything unrecognized selects GitHub so a typo can't queue jobs -# against the provider you're escaping. Paid build/browser jobs and their cache actions share -# this predicate. Task planning, the aggregate gate, HTTP acceptance and integration tests use free GitHub runners. +# against the provider you're escaping. Integration, build and browser jobs and their cache +# actions share this predicate. Task planning, the aggregate gate and HTTP acceptance use free GitHub runners. # # GitHub mode is break-glass, not a peer — cold layers, slower runs. The app image # is the one job on a paid larger runner: next build needs ~32 GB and OOM-kills diff --git a/apps/desktop/e2e/chat-panel.spec.ts b/apps/desktop/e2e/chat-panel.spec.ts index 446f791db24..bb41187fa1d 100644 --- a/apps/desktop/e2e/chat-panel.spec.ts +++ b/apps/desktop/e2e/chat-panel.spec.ts @@ -114,8 +114,6 @@ test('chat panel sizes survive navigation, chat switches, collapse, and layout c const shell = app const page = await shell.firstWindow() const inputSession = await page.context().newCDPSession(page) - // Playwright otherwise keeps the document focused when its native window loses focus. - await inputSession.send('Emulation.setFocusEmulationEnabled', { enabled: false }) page.on('pageerror', (error) => errors.push(error.message)) page.on('crash', () => { rendererCrashed = true @@ -270,6 +268,8 @@ test('chat panel sizes survive navigation, chat switches, collapse, and layout c await divider.evaluate((element) => element.releasePointerCapture(1)) await page.mouse.move(rect.x + 100, rect.y + 100) } else if (interruption === 'blur') { + // Reloads restore Playwright's focus emulation; native blur must reach the document. + await inputSession.send('Emulation.setFocusEmulationEnabled', { enabled: false }) await shell.evaluate(({ BrowserWindow }) => BrowserWindow.getAllWindows()[0].blur()) await expect.poll(() => page.evaluate(() => document.hasFocus())).toBe(false) } else if (interruption === 'chat switch') { diff --git a/apps/desktop/package.json b/apps/desktop/package.json index 94e75d5bf65..3fa0036c03f 100644 --- a/apps/desktop/package.json +++ b/apps/desktop/package.json @@ -50,7 +50,7 @@ }, "devDependencies": { "@electron/fuses": "1.8.0", - "@playwright/test": "1.61.1", + "@playwright/test": "1.64.1", "@sim/emcn": "workspace:*", "@sim/tsconfig": "workspace:*", "@types/micromatch": "4.0.10", diff --git a/apps/docs/package.json b/apps/docs/package.json index 530221569f7..12c471f9ac2 100644 --- a/apps/docs/package.json +++ b/apps/docs/package.json @@ -67,7 +67,7 @@ "vfile": "6.0.3" }, "devDependencies": { - "@playwright/test": "1.61.1", + "@playwright/test": "1.64.1", "@sim/tsconfig": "workspace:*", "@tailwindcss/postcss": "^4.3.3", "@types/node": "24.19.2", diff --git a/apps/extension/package.json b/apps/extension/package.json index e92c11c5047..e4fa5911ca2 100644 --- a/apps/extension/package.json +++ b/apps/extension/package.json @@ -24,7 +24,7 @@ "zod": "4.3.6" }, "devDependencies": { - "@playwright/test": "1.61.1", + "@playwright/test": "1.64.1", "@sim/tsconfig": "workspace:*", "@types/bun": "1.4.3", "@types/chrome": "0.3.6", diff --git a/apps/sim/lib/mothership/request/session/recovery-storm.integration.ts b/apps/sim/lib/mothership/request/session/recovery-storm.integration.ts index 2319dad203a..6d9ebcbdea1 100644 --- a/apps/sim/lib/mothership/request/session/recovery-storm.integration.ts +++ b/apps/sim/lib/mothership/request/session/recovery-storm.integration.ts @@ -10,45 +10,68 @@ import { deleteWorkspaceFixture, insertWorkspaceFixture } from '@sim/db/testing/ import { authMock, authMockFns } from '@sim/testing/mocks/auth.mock' import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest' -const { redisUrl, inheritedEnv, worker, faults } = await vi.hoisted(async () => { +const { redisUrl, inheritedEnv, worker } = await vi.hoisted(async () => { const { readTestRedisUrl } = await import('@sim/db/testing/test-infrastructure') + const { toRecord } = await import('@sim/utils/object') const { createServer } = await import('node:http') - const faults = { - /** Which leased appends fail, and how; `undefined` lets every append through. */ - append: undefined as - | undefined - | { - frames: 'any_tool' | 'tool_result' - effect: 'throw' | 'throw_once' | 'lose_lease' - }, - /** Fails a recovering controller's read of the whole replay ring. */ - recoveryRead: false, + interface RunFixture { + mode: 'frames' | 'json500' | 'drop' + frames: unknown[] + posts: { path: string; at: number }[] + append?: { + frames: 'any_tool' | 'tool_result' + effect: 'throw' | 'throw_once' | 'lose_lease' + } + recoveryRead: boolean } const worker = { - posts: [] as Array<{ path: string; at: number }>, - mode: 'frames' as 'frames' | 'json500' | 'drop', - frames: [] as unknown[], + runs: new Map(), + unexpectedRequests: [] as { path: string; reason: string }[], } const server = createServer(async (request, response) => { - await new Promise((resolve) => request.on('end', resolve).resume()) - worker.posts.push({ path: request.url ?? '', at: Date.now() }) + let body: Record + try { + let raw = '' + request.setEncoding('utf8') + for await (const chunk of request) raw += chunk + body = toRecord(JSON.parse(raw)) + } catch { + worker.unexpectedRequests.push({ path: request.url ?? '', reason: 'malformed_body' }) + response.writeHead(404, { 'content-type': 'application/json' }) + response.end(JSON.stringify({ error: 'Run not found' })) + return + } + const fixture = typeof body.messageId === 'string' ? worker.runs.get(body.messageId) : undefined + if ( + !fixture || + (request.url !== '/api/mothership' && request.url !== '/api/streams/explicit-abort') + ) { + worker.unexpectedRequests.push({ + path: request.url ?? '', + reason: fixture ? 'unknown_path' : 'unknown_stream', + }) + response.writeHead(404, { 'content-type': 'application/json' }) + response.end(JSON.stringify({ error: 'Run not found' })) + return + } + fixture.posts.push({ path: request.url, at: Date.now() }) if (request.url === '/api/streams/explicit-abort') { response.writeHead(200, { 'content-type': 'application/json' }) response.end(JSON.stringify({ settled: true })) return } - if (worker.mode === 'json500') { + if (fixture.mode === 'json500') { response.writeHead(500, { 'content-type': 'application/json' }) response.end(JSON.stringify({ error: 'internal' })) return } response.writeHead(200, { 'content-type': 'text/event-stream' }) - if (worker.mode === 'drop') { - response.write(`data: ${JSON.stringify(worker.frames[0])}\n\n`) + if (fixture.mode === 'drop') { + response.write(`data: ${JSON.stringify(fixture.frames[0])}\n\n`) setTimeout(() => response.socket?.destroy(), 20) return } - for (const frame of worker.frames) response.write(`data: ${JSON.stringify(frame)}\n\n`) + for (const frame of fixture.frames) response.write(`data: ${JSON.stringify(frame)}\n\n`) response.end('data: [DONE]\n\n') }) await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)) @@ -61,7 +84,7 @@ const { redisUrl, inheritedEnv, worker, faults } = await vi.hoisted(async () => /** The real Redis module and worker URL resolution read these at import. */ process.env.REDIS_URL = url process.env.SIM_AGENT_API_URL = `http://127.0.0.1:${port}` - return { redisUrl: url, inheritedEnv, worker: Object.assign(worker, { server }), faults } + return { redisUrl: url, inheritedEnv, worker: Object.assign(worker, { server }) } }) vi.mock('@/lib/auth', () => authMock) @@ -71,8 +94,15 @@ vi.mock('@/lib/mothership/request/session/buffer', async (importOriginal) => { return { ...actual, appendEvents: async (...args: Parameters) => { - const [envelopes, , lease] = args - const fault = faults.append + const [envelopes, scope, lease] = args + if (envelopes.length === 0) return actual.appendEvents(...args) + const streamId = scope?.streamId ?? envelopes[0]?.stream.streamId + const fixture = streamId ? worker.runs.get(streamId) : undefined + if (!fixture) { + worker.unexpectedRequests.push({ path: 'appendEvents', reason: 'unknown_append_stream' }) + throw new Error('Append has no registered recovery fixture') + } + const fault = fixture.append const hit = lease && fault && @@ -85,7 +115,7 @@ vi.mock('@/lib/mothership/request/session/buffer', async (importOriginal) => { if (hit && fault.effect === 'throw') throw new Error('simulated Redis write failure') /** A Redis blip that outlasts the append retries, after which Redis is healthy again. */ if (hit && fault.effect === 'throw_once') { - faults.append = undefined + fixture.append = undefined throw new Error('simulated transient Redis write failure') } /** The lock expires under a live controller, and nobody else holds it. */ @@ -93,8 +123,13 @@ vi.mock('@/lib/mothership/request/session/buffer', async (importOriginal) => { return actual.appendEvents(...args) }, readEvents: async (...args: Parameters) => { - const [, afterCursor] = args - if (faults.recoveryRead && afterCursor === '0') + const [streamId, afterCursor] = args + const fixture = worker.runs.get(streamId) + if (!fixture) { + worker.unexpectedRequests.push({ path: 'readEvents', reason: 'unknown_read_stream' }) + throw new Error('Read has no registered recovery fixture') + } + if (fixture.recoveryRead && afterCursor === '0') throw new Error('simulated Redis read failure') return actual.readEvents(...args) }, @@ -141,6 +176,7 @@ async function orphanedRun( options: { status?: 'active' | 'paused_waiting_for_tool' recoveryBackoff?: Record + mode?: 'frames' | 'json500' | 'drop' } = {} ) { const chatId = generateId() @@ -180,6 +216,13 @@ async function orphanedRun( ...(options.recoveryBackoff ? { recoveryBackoff: options.recoveryBackoff } : {}), }, }) + const fixture: NonNullable> = { + mode: options.mode ?? 'frames', + frames: [], + posts: [], + recoveryRead: false, + } + worker.runs.set(streamId, fixture) await appendEvents( [1, 2].map((seq) => createEvent({ @@ -201,7 +244,7 @@ async function orphanedRun( stream: { streamId, chatId }, payload, }) - worker.frames = [ + fixture.frames = [ frame(1, 'tool', { phase: 'call', toolCallId: 'go-call', @@ -221,7 +264,7 @@ async function orphanedRun( }), frame(3, 'complete', { status: 'complete', textLength: 14 }), ] - return { chatId, streamId, runId } + return { chatId, streamId, runId, fixture } } async function storedRun(runId: string) { @@ -245,8 +288,12 @@ async function storedRun(runId: string) { * lock released, or `windowMs` passes, and returns the worker requests made meanwhile. */ async function reconnect( - { chatId, streamId, runId }: { chatId: string; streamId: string; runId: string }, - { tails = 1, windowMs }: { tails?: number; windowMs: number } + { chatId, streamId, runId, fixture }: Awaited>, + { + tails = 1, + windowMs, + waitForLeaseRelease = true, + }: { tails?: number; windowMs: number; waitForLeaseRelease?: boolean } ) { const startedAt = Date.now() const controllers = Array.from({ length: tails }, () => new AbortController()) @@ -270,8 +317,9 @@ async function reconnect( } for (const controller of controllers) controller.abort() await Promise.allSettled(drains) - for (let i = 0; i < 50 && (await redis().get(chatStreamLockKey(chatId))); i++) await sleep(100) - const requests = worker.posts.filter((post) => post.at >= startedAt) + if (waitForLeaseRelease) + for (let i = 0; i < 50 && (await redis().get(chatStreamLockKey(chatId))); i++) await sleep(100) + const requests = fixture.posts.filter((post) => post.at >= startedAt) return { startedAt, turns: requests.filter((post) => post.path === '/api/mothership'), @@ -281,10 +329,12 @@ async function reconnect( afterAll(async () => { await new Promise((resolve) => worker.server.close(() => resolve())) + worker.runs.clear() for (const [key, value] of Object.entries(inheritedEnv)) { if (value === undefined) delete process.env[key] else process.env[key] = value } + expect(worker.unexpectedRequests).toEqual([]) }) describe.runIf(Boolean(redisUrl))('reconnecting to an orphaned Chat run', () => { @@ -332,7 +382,6 @@ describe.runIf(Boolean(redisUrl))('reconnecting to an orphaned Chat run', () => it.each(['active', 'paused_waiting_for_tool'] as const)( 'takes over an orphaned %s run at once and completes it', async (status) => { - worker.mode = 'frames' const run = await orphanedRun({ status }) const { startedAt, turns } = await reconnect(run, { windowMs: 10_000 }) @@ -347,9 +396,8 @@ describe.runIf(Boolean(redisUrl))('reconnecting to an orphaned Chat run', () => ) it('hands off a turn whose append fails once, and the next controller completes it', async () => { - worker.mode = 'frames' const run = await orphanedRun() - faults.append = { frames: 'any_tool', effect: 'throw_once' } + run.fixture.append = { frames: 'any_tool', effect: 'throw_once' } try { const { turns, stops } = await reconnect(run, { windowMs: 15_000 }) @@ -360,11 +408,11 @@ describe.runIf(Boolean(redisUrl))('reconnecting to an orphaned Chat run', () => recoveryBackoff: { attempts: 2 }, }) } finally { - faults.append = undefined + run.fixture.append = undefined } }, 60_000) - it.each([ + it.concurrent.for([ { frames: 'any_tool', effect: 'throw', tails: 1 }, { frames: 'any_tool', effect: 'throw', tails: 3 }, { frames: 'tool_result', effect: 'throw', tails: 1 }, @@ -372,10 +420,10 @@ describe.runIf(Boolean(redisUrl))('reconnecting to an orphaned Chat run', () => { frames: 'any_tool', effect: 'lose_lease', tails: 3 }, ] as const)( 'gives up on a run whose recovered controllers keep failing ($effect on $frames, $tails tails)', - async ({ frames, effect, tails }) => { - worker.mode = 'frames' + { timeout: 120_000 }, + async ({ frames, effect, tails }, { expect }) => { const run = await orphanedRun() - faults.append = { frames, effect } + run.fixture.append = { frames, effect } try { const { turns, stops } = await reconnect(run, { tails, windowMs: 60_000 }) @@ -391,14 +439,12 @@ describe.runIf(Boolean(redisUrl))('reconnecting to an orphaned Chat run', () => recoveryBackoff: { attempts: MAX_RECOVERY_ATTEMPTS + 1 }, }) } finally { - faults.append = undefined + run.fixture.append = undefined } - }, - 120_000 + } ) it('waits out the backoff of a recent takeover, then takes over and completes the run', async () => { - worker.mode = 'frames' const now = Date.now() const notBefore = now + 3_000 const run = await orphanedRun({ @@ -415,7 +461,6 @@ describe.runIf(Boolean(redisUrl))('reconnecting to an orphaned Chat run', () => }, 30_000) it('starts a fresh budget once the previous takeover is older than the reset window', async () => { - worker.mode = 'frames' const claimedAt = Date.now() - RECOVERY_BUDGET_RESET_MS - 1_000 const run = await orphanedRun({ recoveryBackoff: { attempts: MAX_RECOVERY_ATTEMPTS, claimedAt, notBefore: claimedAt }, @@ -431,14 +476,13 @@ describe.runIf(Boolean(redisUrl))('reconnecting to an orphaned Chat run', () => }, 30_000) it('leaves the run untouched when a takeover fails before it starts, then ends it once exhausted', async () => { - worker.mode = 'frames' const claimedAt = Date.now() - 1_000 const recoveryBackoff = { attempts: MAX_RECOVERY_ATTEMPTS, claimedAt, notBefore: claimedAt } const run = await orphanedRun({ recoveryBackoff }) const { controllerToken } = (await storedRun(run.runId)).requestContext as { controllerToken: string } - faults.recoveryRead = true + run.fixture.recoveryRead = true try { const { turns } = await reconnect(run, { windowMs: 2_000 }) @@ -449,7 +493,7 @@ describe.runIf(Boolean(redisUrl))('reconnecting to an orphaned Chat run', () => recoveryBackoff, }) } finally { - faults.recoveryRead = false + run.fixture.recoveryRead = false } const { turns, stops } = await reconnect(run, { windowMs: 10_000 }) @@ -464,17 +508,21 @@ describe.runIf(Boolean(redisUrl))('reconnecting to an orphaned Chat run', () => }, 30_000) it('never takes over a parked run whose controller still holds the chat lock', async () => { - worker.mode = 'frames' const run = await orphanedRun({ status: 'paused_waiting_for_tool' }) const liveController = `${run.streamId}\n${generateId()}` await redis().set(chatStreamLockKey(run.chatId), liveController, 'EX', 60) try { - const { turns } = await reconnect(run, { tails: 3, windowMs: 3_000 }) + const { turns } = await reconnect(run, { + tails: 3, + windowMs: 3_000, + waitForLeaseRelease: false, + }) expect(turns).toHaveLength(0) const stored = await storedRun(run.runId) expect(stored.status).toBe('paused_waiting_for_tool') expect(stored.recoveryBackoff).toBeUndefined() + expect(await redis().get(chatStreamLockKey(run.chatId))).toBe(liveController) } finally { await redis().del(chatStreamLockKey(run.chatId)) } @@ -483,16 +531,11 @@ describe.runIf(Boolean(redisUrl))('reconnecting to an orphaned Chat run', () => it.each(['json500', 'drop'] as const)( 'ends the run after its bounded retries when the worker fails every leg (%s)', async (mode) => { - worker.mode = mode - const run = await orphanedRun() - try { - const { turns } = await reconnect(run, { windowMs: 20_000 }) + const run = await orphanedRun({ mode }) + const { turns } = await reconnect(run, { windowMs: 20_000 }) - expect(turns).toHaveLength(4) - expect((await storedRun(run.runId)).status).toBe('error') - } finally { - worker.mode = 'frames' - } + expect(turns).toHaveLength(4) + expect((await storedRun(run.runId)).status).toBe('error') }, 60_000 ) diff --git a/apps/sim/lib/mothership/tools/client/desktop-tool-pickup.integration.ts b/apps/sim/lib/mothership/tools/client/desktop-tool-pickup.integration.ts index 13a2c6120da..e0d9701fbaf 100644 --- a/apps/sim/lib/mothership/tools/client/desktop-tool-pickup.integration.ts +++ b/apps/sim/lib/mothership/tools/client/desktop-tool-pickup.integration.ts @@ -5,10 +5,11 @@ * hanging until a watchdog calls it "hung". Runs against real PostgreSQL and Redis: the call is * persisted and dispatched by the production stream handlers, the desktop claims it through the * authorize route and reports through the confirm route. + * Cases own separate calls and waiters; the shared authorization and parent-run fixtures stay fixed. */ import { deleteWorkspaceFixture, insertWorkspaceFixture } from '@sim/db/testing/workspace-fixtures' import { authMock, authMockFns } from '@sim/testing/mocks/auth.mock' -import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest' +import { afterAll, beforeAll, describe, it, vi } from 'vitest' const { redisUrl, inheritedEnv } = await vi.hoisted(async () => { const { readTestRedisUrl } = await import('@sim/db/testing/test-infrastructure') @@ -69,7 +70,7 @@ afterAll(async () => { } }) -describe.runIf(Boolean(redisUrl))('a desktop tool call nobody picks up', () => { +describe.runIf(Boolean(redisUrl)).concurrent('a desktop tool call nobody picks up', () => { const userId = generateId() const workspaceId = generateId() const chatId = generateId() @@ -181,12 +182,13 @@ describe.runIf(Boolean(redisUrl))('a desktop tool call nobody picks up', () => { await db.delete(user).where(eq(user.id, userId)) }) - it.each([ + it.for([ ['browser_snapshot', {}], ['terminal', { operation: 'run', args: { command: 'ls' } }], - ])( + ] as const)( 'tells the agent a %s call never started, and refuses a late pickup', - async (toolName, args) => { + { timeout: TURN_WAIT_MS + 10_000 }, + async ([toolName, args], { expect }) => { const startedAt = Date.now() const { toolCallId, answer } = await agentCalls(toolName, args) @@ -203,13 +205,12 @@ describe.runIf(Boolean(redisUrl))('a desktop tool call nobody picks up', () => { .where(eq(copilotAsyncToolCalls.toolCallId, toolCallId)) expect(row.status).toBe('failed') expect((await desktopClaims(toolCallId)).ok).toBe(false) - }, - TURN_WAIT_MS + 10_000 + } ) it( 'keeps waiting for a call the desktop picked up in time, past the pickup grace', - async () => { + async ({ expect }) => { const { toolCallId, answer } = await agentCalls('browser_snapshot', {}) expect((await desktopClaims(toolCallId)).status).toBe(200) @@ -229,7 +230,7 @@ describe.runIf(Boolean(redisUrl))('a desktop tool call nobody picks up', () => { it( 'never lets a stale "not started" report end a call the desktop is running', - async () => { + async ({ expect }) => { const { toolCallId, answer } = await agentCalls('terminal', { operation: 'run', args: { command: 'bun run test' }, @@ -258,7 +259,7 @@ describe.runIf(Boolean(redisUrl))('a desktop tool call nobody picks up', () => { it( 'tells the agent a local read nobody picked up never started, for a desktop that claims reads', - async () => { + async ({ expect }) => { const startedAt = Date.now() const { toolCallId, answer } = await agentCalls( 'read_local_file', @@ -277,7 +278,7 @@ describe.runIf(Boolean(redisUrl))('a desktop tool call nobody picks up', () => { it( 'lets a desktop that claimed a local read keep reading it, past the pickup grace', - async () => { + async ({ expect }) => { const { toolCallId, answer } = await agentCalls( 'grep', { path: 'user-local/Project--mount-1', pattern: 'TODO' }, @@ -302,7 +303,7 @@ describe.runIf(Boolean(redisUrl))('a desktop tool call nobody picks up', () => { it( 'reads as before for a desktop that does not claim local reads', - async () => { + async ({ expect }) => { const { toolCallId, answer } = await agentCalls('read_local_file', { path: '/Users/me/notes.txt', }) diff --git a/apps/sim/package.json b/apps/sim/package.json index 430e0c3451d..afbc2dc9748 100644 --- a/apps/sim/package.json +++ b/apps/sim/package.json @@ -292,7 +292,7 @@ "devDependencies": { "@next/env": "16.4.1", "@opentelemetry/context-async-hooks": "2.10.0", - "@playwright/test": "1.61.1", + "@playwright/test": "1.64.1", "@sim/testing": "workspace:*", "@sim/tsconfig": "workspace:*", "@tailwindcss/postcss": "^4.3.3", diff --git a/apps/sim/scripts/test-browser-extension-e2e.ts b/apps/sim/scripts/test-browser-extension-e2e.ts index 40549684f25..2b9fb7b9dab 100644 --- a/apps/sim/scripts/test-browser-extension-e2e.ts +++ b/apps/sim/scripts/test-browser-extension-e2e.ts @@ -41,6 +41,9 @@ const base = new URL(required('EXTENSION_APP_E2E_BASE_URL')) const database = assertDisposableTestDatabaseUrl(required('EXTENSION_APP_E2E_DATABASE_URL')) const report = resolve(required('EXTENSION_APP_E2E_REPORT_PATH')) const liveChat = process.env.EXTENSION_APP_E2E_LIVE_CHAT === '1' +const group = process.env.EXTENSION_APP_E2E_GROUP ?? 'all' +assert(group === 'all' || group === 'native' || group === 'access', 'Unknown extension E2E group') +assert(!liveChat || group === 'all', 'Live-provider acceptance requires the complete suite') const loopback = ['127.0.0.1', 'localhost', '[::1]'] assert( loopback.includes(base.hostname) && @@ -112,7 +115,8 @@ await once(fixture, 'listening') const fixtureAddress = fixture.address() assert(fixtureAddress && typeof fixtureAddress !== 'string') -async function check(name: string, run: () => Promise) { +async function check(owner: 'native' | 'access', name: string, run: () => Promise) { + if (group !== 'all' && group !== owner) return const started = performance.now() try { await run() @@ -128,6 +132,9 @@ async function check(name: string, run: () => Promise) { throw error } } +const checkNative = (name: string, run: () => Promise) => check('native', name, run) +const checkAccess = (name: string, run: () => Promise) => check('access', name, run) + async function screenshot(name: string) { if (!panel) return for (const frame of panel.frames()) { @@ -255,6 +262,24 @@ async function seed() { }) } +async function seedViewerOrganization() { + assert(userId) + await sql`insert into organization (id, name, slug, created_at) values (${organizationId}, 'Extension QA organization', ${organizationId}, now())` + await sql`insert into member (id, user_id, organization_id, role) values (${generateId()}, ${userId}, ${organizationId}, 'owner')` +} +async function seedOtherOrganization() { + assert(userId) + await sql`insert into organization (id, name, slug, created_at) values (${otherOrganizationId}, 'Other QA organization', ${otherOrganizationId}, now())` + await sql`insert into member (id, user_id, organization_id, role) values (${generateId()}, ${otherUserId}, ${otherOrganizationId}, 'owner')` + await sql`insert into copilot_chats (id, user_id, organization_id, type, title, config) values (${otherOrganizationChatId}, ${userId}, ${otherOrganizationId}, 'mothership', 'Other organization history', ${sql.json({ conversationMode: 'agent' })})` +} +async function seedHostedWorkspace() { + assert(userId) + await sql`insert into project (id, name, owner_id, organization_id) values (${hostProjectId}, 'Hosted QA project', ${otherUserId}, ${otherOrganizationId})` + await sql`insert into workspace (id, project_id, organization_id, name, owner_id, billed_account_user_id) values (${hostedWorkspaceId}, ${hostProjectId}, ${otherOrganizationId}, 'Hosted QA workspace', ${otherUserId}, ${otherUserId})` + await sql`insert into permissions (id, user_id, entity_type, entity_id, permission_type) values (${generateId()}, ${userId}, 'workspace', ${hostedWorkspaceId}, 'admin')` +} + try { await mkdir(dirname(report), { recursive: true }) const extensionPath = resolve(temporary, 'extension') @@ -351,7 +376,7 @@ try { await frame().evaluate((destination) => window.location.assign(destination), url) await navigation } - await check('Signed-out users get the real top-level Sim sign-in flow', async () => { + await checkNative('Signed-out users get the real top-level Sim sign-in flow', async () => { await expect(frame().getByRole('link', { name: 'Sign in', exact: true })).toBeVisible() await screenshot('signed-out') const link = frame().getByRole('link', { name: 'Sign in', exact: true }) @@ -362,7 +387,7 @@ try { assert.equal(new URL(login.url()).origin, base.origin) await login.close() }) - await check('A real Better Auth session reaches the extension iframe', async () => { + const signInAndSeed = async () => { const signup = await browserContext.request.post(`${base.origin}/api/auth/sign-up/email`, { headers: { origin: base.origin }, data: { @@ -377,6 +402,9 @@ try { assert.equal(typeof id, 'string') userId = String(id) await seed() + } + await checkNative('A real Better Auth session reaches the extension iframe', async () => { + await signInAndSeed() await shell.reload() await expect( frame().getByRole('link', { name: 'Extension QA alpha', exact: true }) @@ -397,6 +425,7 @@ try { timeout: 120_000, }) }) + if (group === 'access') await signInAndSeed() let source = await context.newPage() await source.goto(`${fixtureOrigin}/page?private=QUERY_SECRET#FRAGMENT_SECRET`) const observeCaptureReceipts = () => @@ -454,7 +483,7 @@ try { frame().getByRole('button', { name: 'Remove extension-notes.txt', exact: true }) ).toBeEnabled() } - await check( + await checkNative( 'Capture, preview, removal, and the three-page cap use the real composer', async () => { await capture() @@ -487,7 +516,7 @@ try { ) } ) - await check( + await checkNative( 'Toolbar and authenticated chat align at 320/390 pixels in light and dark themes', async () => { for (const theme of ['light', 'dark'] as const) { @@ -564,31 +593,34 @@ try { } } ) - await check('Workspace and conversation navigation clear pending page context', async () => { - await capture() - await expect(attachments()).toHaveCount(1) - await frame().getByRole('button', { name: 'Extension QA alpha', exact: true }).click() - await screenshot('workspace-menu') - await expect( - frame().getByRole('menuitem', { name: 'Extension QA alpha', exact: true }) - ).toBeDisabled() - await frame().getByRole('menuitem', { name: 'Extension QA beta', exact: true }).click() - await expect( - frame().getByRole('button', { name: 'Extension QA beta', exact: true }) - ).toBeVisible() - await expect(attachments()).toHaveCount(0) - await navigate(`${base.origin}/extension/${alphaId}`) - await expect(frame().getByRole('textbox', { name: 'Message', exact: true })).toBeVisible() - await capture() - await frame().getByRole('button', { name: 'Chat history' }).click() - await screenshot('chat-history') - await frame().getByRole('menu').press('End') - await screenshot('chat-history-end') - await frame().getByRole('menuitem', { name: 'Extension QA history' }).click() - await expect(attachments()).toHaveCount(0) - await expect(frame().getByText('QA history response.', { exact: false })).toBeVisible() - }) - await check( + await checkNative( + 'Workspace and conversation navigation clear pending page context', + async () => { + await capture() + await expect(attachments()).toHaveCount(1) + await frame().getByRole('button', { name: 'Extension QA alpha', exact: true }).click() + await screenshot('workspace-menu') + await expect( + frame().getByRole('menuitem', { name: 'Extension QA alpha', exact: true }) + ).toBeDisabled() + await frame().getByRole('menuitem', { name: 'Extension QA beta', exact: true }).click() + await expect( + frame().getByRole('button', { name: 'Extension QA beta', exact: true }) + ).toBeVisible() + await expect(attachments()).toHaveCount(0) + await navigate(`${base.origin}/extension/${alphaId}`) + await expect(frame().getByRole('textbox', { name: 'Message', exact: true })).toBeVisible() + await capture() + await frame().getByRole('button', { name: 'Chat history' }).click() + await screenshot('chat-history') + await frame().getByRole('menu').press('End') + await screenshot('chat-history-end') + await frame().getByRole('menuitem', { name: 'Extension QA history' }).click() + await expect(attachments()).toHaveCount(0) + await expect(frame().getByText('QA history response.', { exact: false })).toBeVisible() + } + ) + await checkNative( 'Page chips protect draft history and support page-only Enter at the HTTP boundary', async () => { const editor = frame().getByRole('textbox', { name: 'Message', exact: true }) @@ -647,7 +679,7 @@ try { contentType: 'application/javascript', }) ) - await check('Saved page previews stay out of session replay in the full app', async () => { + await checkNative('Saved page previews stay out of session replay in the full app', async () => { const opened = browserContext.waitForEvent('page') await frame().getByRole('link', { name: 'Open in Sim', exact: true }).click() const app = await opened @@ -678,7 +710,7 @@ try { await app.close() } }) - await check('Sources and resource actions preserve their real destinations', async () => { + await checkNative('Sources and resource actions preserve their real destinations', async () => { await frame().getByRole('button', { name: '1 source', exact: true }).click() await frame().getByRole('button', { name: 'View all sources' }).click() const dialog = frame().getByRole('dialog', { name: 'Cited sources' }) @@ -717,7 +749,7 @@ try { await frame().getByRole('button', { name: 'New chat', exact: true }).click() await expect(frame().getByRole('textbox', { name: 'Message', exact: true })).toHaveValue('') }) - await check( + await checkNative( 'The extension blocks microphone capture even when the Sim origin has permission', async () => { await browserContext.grantPermissions(['microphone'], { origin: base.origin }) @@ -735,7 +767,7 @@ try { await browserContext.clearPermissions() } ) - await check( + await checkNative( 'Files and pages share the attachment row, remove independently, and reach the submission boundary together', async () => { await capture() @@ -787,7 +819,7 @@ try { await expect(attachments()).toHaveCount(0) } ) - await check( + await checkNative( 'Selected text attaches only the selected excerpt through the shared plus menu', async () => { await source.evaluate(() => { @@ -1046,7 +1078,7 @@ try { assert.equal(toRecord(submitted[0]).workspaceId, scope.organizationId ? undefined : alphaId) } if (liveChat) { - await check( + await checkNative( 'Workspace live admission preserves queued pages, files, and the active draft', () => verifyLiveChat({ @@ -1056,9 +1088,9 @@ try { draftKey: `extension:${userId}:${alphaId}:new`, }) ) - } else + } else if (group !== 'access') checks.push({ name: 'Live Mothership send and admission', status: 'skipped', durationMs: 0 }) - await check( + await checkNative( 'Another user’s existing workspace fails closed and returns to an authorized workspace', async () => { await navigate(`${base.origin}/extension/${deniedId}`) @@ -1069,12 +1101,10 @@ try { await expect(frame().getByRole('textbox', { name: 'Message', exact: true })).toBeVisible() } ) - await check( + await checkNative( 'Organization default landing binds page-only submissions to the organization', async () => { - assert(userId) - await sql`insert into organization (id, name, slug, created_at) values (${organizationId}, 'Extension QA organization', ${organizationId}, now())` - await sql`insert into member (id, user_id, organization_id, role) values (${generateId()}, ${userId}, ${organizationId}, 'owner')` + await seedViewerOrganization() await navigate(`${base.origin}/extension`) await expect .poll(() => new URL(frame().url()).pathname) @@ -1107,9 +1137,7 @@ try { } finally { await shell.unroute('**/api/mothership/chat') } - await sql`insert into organization (id, name, slug, created_at) values (${otherOrganizationId}, 'Other QA organization', ${otherOrganizationId}, now())` - await sql`insert into member (id, user_id, organization_id, role) values (${generateId()}, ${otherUserId}, ${otherOrganizationId}, 'owner')` - await sql`insert into copilot_chats (id, user_id, organization_id, type, title, config) values (${otherOrganizationChatId}, ${userId}, ${otherOrganizationId}, 'mothership', 'Other organization history', ${sql.json({ conversationMode: 'agent' })})` + await seedOtherOrganization() await navigate( `${base.origin}/extension/o/${otherOrganizationId}?chat=${otherOrganizationChatId}` ) @@ -1122,11 +1150,7 @@ try { await screenshot('organization-wrong-scope-history') } ) - await check('A shared workspace returns to the viewer’s own organization chat', async () => { - assert(userId) - await sql`insert into project (id, name, owner_id, organization_id) values (${hostProjectId}, 'Hosted QA project', ${otherUserId}, ${otherOrganizationId})` - await sql`insert into workspace (id, project_id, organization_id, name, owner_id, billed_account_user_id) values (${hostedWorkspaceId}, ${hostProjectId}, ${otherOrganizationId}, 'Hosted QA workspace', ${otherUserId}, ${otherUserId})` - await sql`insert into permissions (id, user_id, entity_type, entity_id, permission_type) values (${generateId()}, ${userId}, 'workspace', ${hostedWorkspaceId}, 'admin')` + const setActiveOrganization = async () => { const active = await browserContext.request.post( `${base.origin}/api/auth/organization/set-active`, { @@ -1135,22 +1159,33 @@ try { } ) assert.equal(active.status(), 200) - await navigate(`${base.origin}/extension`) - await expect.poll(() => new URL(frame().url()).pathname).toBe(`/extension/o/${organizationId}`) - await navigate(`${base.origin}/extension/${hostedWorkspaceId}`) - await expect(frame().getByRole('textbox', { name: 'Message', exact: true })).toBeVisible() - await capture() - await frame() - .getByRole('textbox', { name: 'Message', exact: true }) - .fill('Workspace-specific draft') - await frame().getByRole('button', { name: 'Hosted QA workspace', exact: true }).click() - await frame().getByRole('menuitem', { name: 'Organization chat', exact: true }).click() - await expect.poll(() => new URL(frame().url()).pathname).toBe(`/extension/o/${organizationId}`) - await expect(frame().getByRole('textbox', { name: 'Ask Sim', exact: true })).toHaveValue('') - await expect(attachments()).toHaveCount(0) - await screenshot('shared-workspace-organization-navigation') - }) - await check( + } + await checkNative( + 'A shared workspace returns to the viewer’s own organization chat', + async () => { + await seedHostedWorkspace() + await setActiveOrganization() + await navigate(`${base.origin}/extension`) + await expect + .poll(() => new URL(frame().url()).pathname) + .toBe(`/extension/o/${organizationId}`) + await navigate(`${base.origin}/extension/${hostedWorkspaceId}`) + await expect(frame().getByRole('textbox', { name: 'Message', exact: true })).toBeVisible() + await capture() + await frame() + .getByRole('textbox', { name: 'Message', exact: true }) + .fill('Workspace-specific draft') + await frame().getByRole('button', { name: 'Hosted QA workspace', exact: true }).click() + await frame().getByRole('menuitem', { name: 'Organization chat', exact: true }).click() + await expect + .poll(() => new URL(frame().url()).pathname) + .toBe(`/extension/o/${organizationId}`) + await expect(frame().getByRole('textbox', { name: 'Ask Sim', exact: true })).toHaveValue('') + await expect(attachments()).toHaveCount(0) + await screenshot('shared-workspace-organization-navigation') + } + ) + await checkNative( 'A first-open toolbar capture survives authenticated organization landing', async () => { await source.bringToFront() @@ -1189,7 +1224,7 @@ try { } ) if (liveChat) { - await check( + await checkNative( 'Organization live admission preserves queued pages, files, and the admitted conversation mode', () => verifyLiveChat({ @@ -1201,7 +1236,13 @@ try { }) ) } - if (!liveChat) { + if (group === 'access') { + await seedViewerOrganization() + await seedOtherOrganization() + await seedHostedWorkspace() + await setActiveOrganization() + } + if (group !== 'native' && !liveChat) { // Install timers before app code in a fresh context; live provider checks keep native timing. await browserContext.close() context = await launchBrowserContext() @@ -1247,7 +1288,7 @@ try { await shell.clock.setSystemTime(Date.now()) } for (const allowRequests of [false, true]) { - await check( + await checkAccess( `Workspace Chat policy revocation and regrant work with access requests ${allowRequests ? 'enabled' : 'disabled'}`, async () => { assert(userId) @@ -1310,7 +1351,7 @@ try { } ) } - await check( + await checkAccess( 'Organization permission changes switch a new chat to Assistant without losing its draft', async () => { assert(userId) @@ -1387,7 +1428,7 @@ try { } else await shell.unroute('**/api/mothership/chat') } ) - await check( + await checkAccess( 'Revoking organization membership removes the live composer and pending captures', async () => { assert(userId) @@ -1415,7 +1456,7 @@ try { await expect(frame().getByRole('textbox', { name: 'Message', exact: true })).toBeVisible() } ) - await check( + await checkAccess( 'Revoking workspace access removes the live composer and pending captures', async () => { assert(userId) @@ -1460,7 +1501,7 @@ try { [404, 'A missing workspace-list refetch removes cached choices'], [400, 'An invalid workspace-list request removes cached choices'], ] as const) { - await check(name, async () => { + await checkAccess(name, async () => { await navigate(`${base.origin}/extension`) const choices = frame().getByRole('navigation', { name: 'Workspaces', exact: true }) await expect(choices).toBeVisible() @@ -1521,7 +1562,7 @@ try { } }) } - await check('An initial workspace-list 401 opens sign-in recovery', async () => { + await checkAccess('An initial workspace-list 401 opens sign-in recovery', async () => { await browserContext.route(workspaceListUrl, async (route) => { injectedRequests.add(route.request()) await route.fulfill({ status: 401, json: { error: 'QA workspace-list session expired' } }) @@ -1537,7 +1578,7 @@ try { await browserContext.unroute(workspaceListUrl) } }) - await check( + await checkAccess( 'Transient session, workspace, and permission-policy refetch failures preserve the mounted composer and page draft', async () => { for (const [path, attempts, scope] of [ @@ -1587,7 +1628,7 @@ try { await navigate(`${base.origin}/extension/${alphaId}`) } ) - await check( + await checkAccess( 'Explicit session, workspace, and permission-policy failures clear captured context after refetch', async () => { for (const [path, status, scope] of [ @@ -1634,7 +1675,7 @@ try { await expect(frame().getByRole('textbox', { name: 'Message', exact: true })).toBeVisible() } ) - await check('An initial permission-policy outage keeps chat inaccessible', async () => { + await checkAccess('An initial permission-policy outage keeps chat inaccessible', async () => { const endpoint = `${base.origin}/api/permission-groups/user*` await browserContext.route(endpoint, async (route) => { injectedRequests.add(route.request()) @@ -1651,20 +1692,27 @@ try { await navigate(`${base.origin}/extension/${alphaId}`) await expect(frame().getByRole('textbox', { name: 'Message', exact: true })).toBeVisible() }) - await check('Signing out removes authenticated chat and captured page context', async () => { - await capture() - const response = await browserContext.request.post(`${base.origin}/api/auth/sign-out`, { - headers: { origin: base.origin }, - data: {}, - }) - assert.equal(response.status(), 200) - await shell.clock.setFixedTime(Date.now() + 360_000) - await reconnect() - await expect(frame().getByRole('link', { name: 'Sign in', exact: true })).toBeVisible() - await screenshot('expired-session') - await expect(attachments()).toHaveCount(0) - await expect(frame().getByRole('textbox', { name: 'Message', exact: true })).toHaveCount(0) - }) + await checkAccess( + 'Signing out removes authenticated chat and captured page context', + async () => { + await capture() + const response = await browserContext.request.post(`${base.origin}/api/auth/sign-out`, { + headers: { origin: base.origin }, + data: {}, + }) + assert.equal(response.status(), 200) + await shell.clock.setFixedTime(Date.now() + 360_000) + await reconnect() + await expect(frame().getByRole('link', { name: 'Sign in', exact: true })).toBeVisible() + await screenshot('expired-session') + await expect(attachments()).toHaveCount(0) + await expect(frame().getByRole('textbox', { name: 'Message', exact: true })).toHaveCount(0) + } + ) + assert( + checks.some((entry) => entry.status === 'passed'), + 'The selected group ran no acceptance checks' + ) assert.deepEqual(browserErrors, [], 'Browser raised uncaught application errors') assert.deepEqual( httpErrors.filter((entry) => entry.status >= 500 && !entry.injected), @@ -1704,7 +1752,7 @@ try { for (const path of uploadedPaths) await rm(path, { force: true }) fixture.closeAllConnections() fixture.close() - await check('Disposable database fixtures are removed', async () => { + const cleanupFixtures = async () => { if (!userId) return const owner = userId await sql.begin(async (tx) => { @@ -1719,7 +1767,19 @@ try { }) const remaining = await sql`select id from "user" where id=${owner}` assert.equal(remaining.length, 0) - }).catch((error) => { + } + const cleanup = + group === 'access' + ? cleanupFixtures() + : checkNative('Disposable database fixtures are removed', cleanupFixtures) + await cleanup.catch((error) => { + if (group === 'access') + checks.push({ + name: 'Fixture cleanup', + status: 'failed', + durationMs: 0, + error: getErrorMessage(error), + }) logger.error('Fixture cleanup failed', { error: getErrorMessage(error) }) process.exitCode = 1 }) @@ -1729,6 +1789,7 @@ try { JSON.stringify( { suite: 'browser-extension-real-app', + group, liveChat, checks, screenshots, diff --git a/bun.lock b/bun.lock index 0758e450d2a..a10ed775525 100644 --- a/bun.lock +++ b/bun.lock @@ -67,7 +67,7 @@ }, "devDependencies": { "@electron/fuses": "1.8.0", - "@playwright/test": "1.61.1", + "@playwright/test": "1.64.1", "@sim/emcn": "workspace:*", "@sim/tsconfig": "workspace:*", "@types/micromatch": "4.0.10", @@ -131,7 +131,7 @@ }, "devDependencies": { "@modelcontextprotocol/client": "2.3.1", - "@playwright/test": "1.61.1", + "@playwright/test": "1.64.1", "@sim/tsconfig": "workspace:*", "@tailwindcss/postcss": "^4.3.3", "@types/hast": "3.0.5", @@ -162,7 +162,7 @@ "zod": "4.3.6", }, "devDependencies": { - "@playwright/test": "1.61.1", + "@playwright/test": "1.64.1", "@sim/tsconfig": "workspace:*", "@types/bun": "1.4.3", "@types/chrome": "0.3.6", @@ -449,7 +449,7 @@ "devDependencies": { "@next/env": "16.4.1", "@opentelemetry/context-async-hooks": "2.10.0", - "@playwright/test": "1.61.1", + "@playwright/test": "1.64.1", "@sim/testing": "workspace:*", "@sim/tsconfig": "workspace:*", "@tailwindcss/postcss": "^4.3.3", @@ -1794,7 +1794,7 @@ "@pinojs/redact": ["@pinojs/redact@0.4.0", "", {}, "sha512-k2ENnmBugE/rzQfEcdWHcCY+/FM3VLzH9cYEsbdsoqrvzAKRhUZeRNhAZvB8OitQJ1TBed3yqWtdjzS6wJKBwg=="], - "@playwright/test": ["@playwright/test@1.61.1", "", { "dependencies": { "playwright": "1.61.1" }, "bin": { "playwright": "cli.js" } }, "sha512-8nKv6+0RJSL9FE4jYOEGXnPeM/Hg12qZpmqzZjRh3qM0Y7c3z1mrOTfFLids72RDQYVh9WpLEfR5WdpNX4fkig=="], + "@playwright/test": ["@playwright/test@1.64.1", "", { "dependencies": { "playwright": "1.64.1" }, "bin": { "playwright": "cli.js" } }, "sha512-IaYGml/nSOiZWROfPtuIL7neuu9qDOZ3TNvNljUIrhDMxiGTgPjzj/g9p9RxpS2tSepPDRY+UwAaEeiKtUvIAg=="], "@posthog/browser-common": ["@posthog/browser-common@0.9.4", "", { "dependencies": { "@posthog/core": "^1.57.4", "@posthog/types": "^1.415.2" } }, "sha512-QlG0WZFG9iTp9cmXTypB4cgnxBAD3GImrj9AwmA84uh8uy5vIVgVLjoTTdOAW29XL+kh/eLn/5ZtBQt2JQfR+w=="], @@ -3986,9 +3986,9 @@ "platform": ["platform@1.3.6", "", {}, "sha512-fnWVljUchTro6RiCFvCXBbNhJc2NijN7oIQxbwsyL0buWJPG85v81ehlHI9fXrJsMNgTofEoWIQeClKpgxFLrg=="], - "playwright": ["playwright@1.61.1", "", { "dependencies": { "playwright-core": "1.61.1" }, "optionalDependencies": { "fsevents": "2.3.2" }, "bin": { "playwright": "cli.js" } }, "sha512-DWnY5o3YbLWK4GovuAVwpqL+1VwGNdUGrRr++8j8PtQQzvAVZUIMjKQ90fY689sEJZJBbZVw1rXaOKSTitkzPQ=="], + "playwright": ["playwright@1.64.1", "", { "dependencies": { "playwright-core": "1.64.1" }, "bin": { "playwright": "cli.js" } }, "sha512-HCrVhHr5QQuwYoswEUGTb7gGp522yflbv/ST8nF4sFWGuLjQCt17Q/EubpGGSNKYn1tsTCPZxqHhHCfVURu7UA=="], - "playwright-core": ["playwright-core@1.61.1", "", { "bin": { "playwright-core": "cli.js" } }, "sha512-h7Qlt6m4REp25qvIdvbDtVmD4LqVXfpRxhORv9L0jzETM05p4fuPJ3dKyuSXQxDSbXnmS79HAgi9589lGSpLkg=="], + "playwright-core": ["playwright-core@1.64.1", "", { "bin": { "playwright-core": "cli.js" } }, "sha512-BC4s8y6SEVA6Q1rSxZTto8hI1KUI3xXlHGZgvGqP4IqPdI/a6FX9H29IDNz7vKo9l21b6LfYosScPL6M3xxyDQ=="], "plist": ["plist@3.1.0", "", { "dependencies": { "@xmldom/xmldom": "^0.8.8", "base64-js": "^1.5.1", "xmlbuilder": "^15.1.1" } }, "sha512-uysumyrvkUX0rX/dEVqt8gC3sTBzd4zoWfLeS29nb53imdaXVvLINYXTI2GNqzaMuvacNx4uJQ8+b3zXR0pkgQ=="], @@ -5338,8 +5338,6 @@ "pkijs/tslib": ["tslib@2.8.1", "", {}, "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w=="], - "playwright/fsevents": ["fsevents@2.3.2", "", { "os": "darwin" }, "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA=="], - "plist/xmlbuilder": ["xmlbuilder@15.1.1", "", {}, "sha512-yMqGBqtXyeN1e3TGYvgNgDVZ3j84W4cwkOXQswghol6APgZWaff9lnbvN7MHYJOiXsvGPXtjTYJEiC9J2wv9Eg=="], "posthog-node/@posthog/core": ["@posthog/core@1.24.4", "", { "dependencies": { "cross-spawn": "^7.0.6" } }, "sha512-S+TolwBHSSJz7WWtgaELQWQqXviSm3uf1e+qorWUts0bZcgPwWzhnmhCUZAhvn0NVpTQHDJ3epv+hHbPLl5dHg=="], diff --git a/scripts/ci-changes.test.ts b/scripts/ci-changes.test.ts index 626167cfa95..fb48aa93ff1 100644 --- a/scripts/ci-changes.test.ts +++ b/scripts/ci-changes.test.ts @@ -147,7 +147,8 @@ describe('CI task attribution through real Turborepo', () => { expect(plan.fallback).toBeUndefined() expect(groups(plan)).toEqual([ 'assistant-actions', - 'browser-extension', + 'browser-extension-access', + 'browser-extension-native', 'desktop-inbox', 'mobile', 'organization-search', @@ -237,7 +238,6 @@ describe('CI task attribution through real Turborepo', () => { }) it.each([ ['test-mobile-e2e.ts', 'mobile'], - ['test-browser-extension-e2e.ts', 'browser-extension'], ['test-project-file-sharing-e2e.ts', 'project-files'], ['test-scim-e2e.ts', 'scim'], ['test-landing-seo-e2e.ts', 'seo'], @@ -254,12 +254,26 @@ describe('CI task attribution through real Turborepo', () => { e2e_build: true, }) }) + it('selects both isolated extension groups for an extension harness change', () => { + const plan = classifyFiles(['apps/sim/scripts/test-browser-extension-e2e.ts']) + expect(plan.fallback).toBeUndefined() + expect(groups(plan)).toEqual(['browser-extension-access', 'browser-extension-native']) + expect(plan.flags).toMatchObject({ + http: true, + production: false, + integration: false, + test: false, + build: false, + e2e_build: true, + }) + }) it('propagates a transitive security dependency to every affected boundary', () => { const plan = classifyFiles(['packages/security/src/index.ts']) expect(plan.fallback).toBeUndefined() expect(groups(plan)).toEqual([ 'assistant-actions', - 'browser-extension', + 'browser-extension-access', + 'browser-extension-native', 'cli', 'desktop-inbox', 'mobile', diff --git a/vitest.integration-durations.json b/vitest.integration-durations.json index 99b0ef0519f..aeb9bf4ea2b 100644 --- a/vitest.integration-durations.json +++ b/vitest.integration-durations.json @@ -206,7 +206,7 @@ "apps/sim/lib/mothership/request/lifecycle/file-owner-protocol.integration.ts": 2.2, "apps/sim/lib/mothership/request/lifecycle/unattended.integration.ts": 13.6, "apps/sim/lib/mothership/request/session/buffer-ttl.integration.ts": 12.4, - "apps/sim/lib/mothership/request/session/recovery-storm.integration.ts": 259.1, + "apps/sim/lib/mothership/request/session/recovery-storm.integration.ts": 100, "apps/sim/lib/mothership/request/session/replay-budget.integration.ts": 73.6, "apps/sim/lib/mothership/request/session/replay-gap.integration.ts": 36.9, "apps/sim/lib/mothership/request/session/stream-recovery.integration.ts": 13, @@ -215,7 +215,7 @@ "apps/sim/lib/mothership/tools/client/desktop-tool-authorization.integration.ts": 13.7, "apps/sim/lib/mothership/tools/client/desktop-tool-chat-view-lease.integration.ts": 12.3, "apps/sim/lib/mothership/tools/client/desktop-tool-executor.integration.ts": 5.6, - "apps/sim/lib/mothership/tools/client/desktop-tool-pickup.integration.ts": 89.4, + "apps/sim/lib/mothership/tools/client/desktop-tool-pickup.integration.ts": 30, "apps/sim/lib/mothership/tools/server/media/write-target-precheck.integration.ts": 1.7, "apps/sim/lib/oauth/credential-revocation.integration.ts": 4.8, "apps/sim/lib/oauth/shopify.integration.ts": 3.1,