Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/CI.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ SEO depends on workspace UI and server code: landing pages reuse workspace prici

Affected type-checks follow the package graph too; repository audits and lint run whenever the checks workflow runs. Workspace-only unit jobs use a free GitHub runner and do not start the app test shards.

Database integration and lightweight HTTP suites run on free standard GitHub runners with provider-specific dependency setup. A same-source integration comparison of 16 jobs per provider (32 total) had a slowest shard of 6m 04s on 2-vCPU Blacksmith versus 6m 55s on free GitHub, with identical coverage. GitHub trades the observed 51 seconds for zero runner compute cost. This is one paired sample, not a long-run percentile estimate. These explicitly selected GitHub jobs do not follow `CI_PROVIDER`.
Database integration uses 4-vCPU Blacksmith runners with sticky dependency disks, following `CI_PROVIDER` and falling back to standard GitHub runners with matching setup. A prior same-source comparison of 16 jobs per provider (32 total) had a slowest shard of 6m 04s on 2-vCPU Blacksmith versus 6m 55s on GitHub, with identical coverage. This is one paired sample, not a long-run percentile estimate. Evaluate runner changes against complete coverage and the overall CI gate, since a faster shard does not shorten a run when another job finishes later. Lightweight HTTP suites retain free standard GitHub runners and do not follow `CI_PROVIDER`.

GitHub runners install dependencies directly rather than transferring Bun and `node_modules` archives. In a same-source paired comparison, those restores took 26.2 seconds before an 82-millisecond install, while a clean install took 12.42 seconds. This is one paired observation; retain the policy only while repeated CI timings support it. Blacksmith keeps its sticky dependency disks. GitHub Turbo output caches remain enabled for jobs that request them, with a commit-specific key and a task/event/fork-scoped restore prefix, so a rerun can reuse an exact entry without creating another archive. Push jobs never restore pull-request entries.

Expand All @@ -34,10 +34,12 @@ Every HTTP and browser group consumes the same production build in an isolated j

Desktop inbox protocol, assistant actions UI, and organization search UI run as three parallel HTTP jobs with isolated databases, apps, and reports. They retain one acceptance task boundary, so changing any of their harnesses selects all three jobs. Desktop inbox needs no browser. The UI jobs and browser extension install Chromium while their app starts, waiting for both before executing assertions; browser-install failures still fail the job and preserve the install log. No acceptance assertions are removed by the split.

Browser extension acceptance keeps native streaming and first-open capture checks on real timers. Its access-revalidation scenarios then reopen the persistent browser profile and install Playwright's clock before the app mounts. Advancing the production polling timers preserves real HTTP and database authorization checks without waiting through each polling interval. Capture timestamps originate in Chromium's isolated world, so the harness restores wall time after each advance. Live-provider runs retain native timing throughout.
Browser extension acceptance runs native interaction and access-revalidation groups in parallel jobs with isolated databases, apps, browser profiles, and reports. Both retain the same task boundary, so an affected extension harness selects both groups. Native streaming and first-open capture checks retain real timers. Access checks install Playwright's clock before the app mounts: advancing production polling timers preserves real HTTP and database authorization checks without waiting through each polling interval. Capture timestamps originate in Chromium's isolated world, so the harness restores wall time after each advance. The local `browser-extension` group still runs the complete suite; live-provider runs require this complete mode and retain native timing throughout.

Integration shard weights come from successful reports for both push and migration provisioning paths. Refresh them with `scripts/update-integration-durations.ts`, then check the new partition against a separate complete run before adopting it. Weights only change scheduling; discovery still owns coverage. The current refresh uses runs [41348786822](https://github.com/simstudioai/sim/actions/runs/41348786822) and [41377145857](https://github.com/simstudioai/sim/actions/runs/41377145857), with [41377810718](https://github.com/simstudioai/sim/actions/runs/41377810718) held out for validation.

Desktop pickup checks overlap their real grace periods using distinct calls and waiters under fixed authorization and parent-run fixtures. Recovery exhaustion checks keep worker responses, injected faults, and observed requests scoped to each stream before overlapping their real backoff schedules. Their immediate-takeover checks remain serial. File isolation and serial execution between integration files remain unchanged; these suites still verify actual timeout, lease, retry-floor, and late-claim behavior against PostgreSQL and Redis.

SCIM's production app uses a `sim.ai` fixture hostname mapped to loopback, since `NEXT_PUBLIC_FORCE_HOSTED` is ignored in production. Readiness, HTTP tests, and internal API requests still connect to loopback. The SCIM suite verifies that ending an Enterprise subscription rejects both owner administration and an existing bearer token, and that restoring it restores access.

## Coverage removed or consolidated
Expand Down
6 changes: 5 additions & 1 deletion .github/ci-suites.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,11 @@
"task": "ci:e2e-desktop-inbox"
},
{
"group": "browser-extension",
"group": "browser-extension-native",
"task": "ci:e2e-browser-extension"
},
{
"group": "browser-extension-access",
"task": "ci:e2e-browser-extension"
},
{
Expand Down
18 changes: 12 additions & 6 deletions .github/scripts/http-e2e.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# CI groups, which reuse the production build from `e2e-build`.
# The job extracts it into apps/sim/.next before running these suites.
#
# Usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|project-files|mobile|seo> (run from apps/sim)
# Usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|browser-extension-native|browser-extension-access|project-files|mobile|seo> (run from apps/sim)
#
# The job provides DATABASE_URL, BETTER_AUTH_SECRET and ENCRYPTION_KEY; each group sets the rest of
# its app's environment here. Reports and server logs land in $RUNNER_TEMP/e2e.
Expand All @@ -20,7 +20,7 @@
# telemetry flush runs detached and still writes .next/dev).
set -euo pipefail

group=${1:?usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|project-files|mobile|seo>}
group=${1:?usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|browser-extension-native|browser-extension-access|project-files|mobile|seo>}
report_dir="$RUNNER_TEMP/e2e"
ready_timeout_seconds=600
mkdir -p "$report_dir"
Expand Down Expand Up @@ -264,13 +264,18 @@ case "$group" in
esac
;;

browser-extension)
browser-extension|browser-extension-native|browser-extension-access)
extension_group=all
case "$group" in
browser-extension-native) extension_group=native ;;
browser-extension-access) extension_group=access ;;
esac
export REDIS_URL=redis://127.0.0.1:6379
export NEXT_PUBLIC_FORCE_HOSTED=false
export KNOWLEDGE_MEMBER_ACCESS=true CREDENTIAL_GROUPS=true ACCESS_CONTROL_ENABLED=true
export NEXT_PUBLIC_ACCESS_CONTROL_ENABLED=true
export API_ENCRYPTION_KEY="$ENCRYPTION_KEY"
start_browser_install browser-extension chromium
start_browser_install "$group" chromium
extension_identity=$(bun - <<'JS'
import { generateKeyPairSync, createHash } from 'node:crypto'
const { publicKey } = generateKeyPairSync('rsa', { modulusLength: 2048 })
Expand All @@ -282,11 +287,12 @@ JS
)
export SIM_EXTENSION_KEY="${extension_identity%%$'\n'*}"
export BROWSER_EXTENSION_IDS="${extension_identity#*$'\n'}"
start_app browser-extension 3007 'browser extension' record-http-status
start_app "$group" 3007 'browser extension' record-http-status
wait_for_browsers
EXTENSION_APP_E2E_BASE_URL="$local_app_url" \
EXTENSION_APP_E2E_DATABASE_URL="$DATABASE_URL" \
EXTENSION_APP_E2E_REPORT_PATH="$report_dir/browser-extension/report.json" \
EXTENSION_APP_E2E_GROUP="$extension_group" \
EXTENSION_APP_E2E_REPORT_PATH="$report_dir/$group/report.json" \
bun run test:browser-extension:e2e
;;

Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ jobs:
name: integration (${{ matrix.provision }}, ${{ matrix.shard == 'db' && 'db' || format('{0}/8', matrix.shard) }})
needs: changes
if: needs.changes.outputs.integration == 'true'
runs-on: ubuntu-24.04
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}
timeout-minutes: 20
strategy:
fail-fast: false
Expand Down Expand Up @@ -135,7 +135,7 @@ jobs:
- name: Setup workspace
uses: ./.github/actions/setup
with:
provider: github
provider: ${{ vars.CI_PROVIDER }}

- name: Provision a fresh database through the supported command
working-directory: packages/db
Expand Down Expand Up @@ -219,7 +219,7 @@ jobs:
--health-retries 10
# Desktop inbox, browser extension, and Project Files use Redis.
redis:
image: ${{ (matrix.group == 'desktop-inbox' || matrix.group == 'assistant-actions' || matrix.group == 'organization-search' || matrix.group == 'browser-extension' || matrix.group == 'project-files') && 'redis:8.2-alpine' || '' }}
image: ${{ (matrix.group == 'desktop-inbox' || matrix.group == 'assistant-actions' || matrix.group == 'organization-search' || startsWith(matrix.group, 'browser-extension-') || matrix.group == 'project-files') && 'redis:8.2-alpine' || '' }}
ports:
- 6379:6379
options: >-
Expand Down Expand Up @@ -298,7 +298,7 @@ jobs:
name: e2e-reports-${{ matrix.group }}
path: |
${{ runner.temp }}/e2e/**/*.json
${{ runner.temp }}/e2e/browser-extension/**/*.png
${{ runner.temp }}/e2e/browser-extension*/**/*.png
${{ runner.temp }}/e2e/*-ui-report.json.artifacts/*.png
if-no-files-found: ignore
retention-days: 7
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ name: ci
# A repo variable, not a committed value: during a Blacksmith outage there is no
# working CI to merge a switchover through. Only unset/'blacksmith' selects
# Blacksmith; anything unrecognized selects GitHub so a typo can't queue jobs
# against the provider you're escaping. Paid build/browser jobs and their cache actions share
# this predicate. Task planning, the aggregate gate, HTTP acceptance and integration tests use free GitHub runners.
# against the provider you're escaping. Integration, build and browser jobs and their cache
# actions share this predicate. Task planning, the aggregate gate and HTTP acceptance use free GitHub runners.
#
# GitHub mode is break-glass, not a peer — cold layers, slower runs. The app image
# is the one job on a paid larger runner: next build needs ~32 GB and OOM-kills
Expand Down
4 changes: 2 additions & 2 deletions apps/desktop/e2e/chat-panel.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -114,8 +114,6 @@ test('chat panel sizes survive navigation, chat switches, collapse, and layout c
const shell = app
const page = await shell.firstWindow()
const inputSession = await page.context().newCDPSession(page)
// Playwright otherwise keeps the document focused when its native window loses focus.
await inputSession.send('Emulation.setFocusEmulationEnabled', { enabled: false })
page.on('pageerror', (error) => errors.push(error.message))
page.on('crash', () => {
rendererCrashed = true
Expand Down Expand Up @@ -270,6 +268,8 @@ test('chat panel sizes survive navigation, chat switches, collapse, and layout c
await divider.evaluate((element) => element.releasePointerCapture(1))
await page.mouse.move(rect.x + 100, rect.y + 100)
} else if (interruption === 'blur') {
// Reloads restore Playwright's focus emulation; native blur must reach the document.
await inputSession.send('Emulation.setFocusEmulationEnabled', { enabled: false })
await shell.evaluate(({ BrowserWindow }) => BrowserWindow.getAllWindows()[0].blur())
await expect.poll(() => page.evaluate(() => document.hasFocus())).toBe(false)
} else if (interruption === 'chat switch') {
Expand Down
2 changes: 1 addition & 1 deletion apps/desktop/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
},
"devDependencies": {
"@electron/fuses": "1.8.0",
"@playwright/test": "1.61.1",
"@playwright/test": "1.64.1",
"@sim/emcn": "workspace:*",
"@sim/tsconfig": "workspace:*",
"@types/micromatch": "4.0.10",
Expand Down
2 changes: 1 addition & 1 deletion apps/docs/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@
"vfile": "6.0.3"
},
"devDependencies": {
"@playwright/test": "1.61.1",
"@playwright/test": "1.64.1",
"@sim/tsconfig": "workspace:*",
"@tailwindcss/postcss": "^4.3.3",
"@types/node": "24.19.2",
Expand Down
2 changes: 1 addition & 1 deletion apps/extension/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"zod": "4.3.6"
},
"devDependencies": {
"@playwright/test": "1.61.1",
"@playwright/test": "1.64.1",
"@sim/tsconfig": "workspace:*",
"@types/bun": "1.4.3",
"@types/chrome": "0.3.6",
Expand Down
Loading
Loading