Skip to content

Commit 974dd6c

Browse files
authored
improvement(ci): shorten the acceptance critical path (#9556)
* improvement(ci): refresh integration shard duration weights * improvement(ci): parallelize acceptance suites and avoid slow dependency restores * improvement(ci): advance extension access polling in acceptance tests * chore(ci): document acceptance timing and shard refresh * fix(ci): settle extension screenshots after advancing browser timers * fix(ci): finish finite animations in clock-controlled extension checks * fix(ci): preserve access revalidation budgets with virtual timers
1 parent 2da47b9 commit 974dd6c

8 files changed

Lines changed: 544 additions & 392 deletions

File tree

‎.github/CI.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,20 @@ SEO depends on workspace UI and server code: landing pages reuse workspace prici
2424

2525
Affected type-checks follow the package graph too; repository audits and lint run whenever the checks workflow runs. Workspace-only unit jobs use a free GitHub runner and do not start the app test shards.
2626

27-
Database integration and lightweight HTTP suites run on free standard GitHub runners with matching GitHub cache setup. A same-source integration comparison of 16 jobs per provider (32 total) had a slowest shard of 6m 04s on 2-vCPU Blacksmith versus 6m 55s on free GitHub, with identical coverage. GitHub trades the observed 51 seconds for zero runner compute cost. This is one paired sample, not a long-run percentile estimate. These explicitly selected GitHub jobs do not follow `CI_PROVIDER`.
27+
Database integration and lightweight HTTP suites run on free standard GitHub runners with provider-specific dependency setup. A same-source integration comparison of 16 jobs per provider (32 total) had a slowest shard of 6m 04s on 2-vCPU Blacksmith versus 6m 55s on free GitHub, with identical coverage. GitHub trades the observed 51 seconds for zero runner compute cost. This is one paired sample, not a long-run percentile estimate. These explicitly selected GitHub jobs do not follow `CI_PROVIDER`.
28+
29+
GitHub runners install dependencies directly rather than transferring Bun and `node_modules` archives. In a same-source paired comparison, those restores took 26.2 seconds before an 82-millisecond install, while a clean install took 12.42 seconds. This is one paired observation; retain the policy only while repeated CI timings support it. Blacksmith keeps its sticky dependency disks. GitHub Turbo output caches remain enabled for jobs that request them, with a commit-specific key and a task/event/fork-scoped restore prefix, so a rerun can reuse an exact entry without creating another archive. Push jobs never restore pull-request entries.
2830

2931
Memory-heavy builds and browser jobs retain appropriately sized Blacksmith runners. Electron uses 4-vCPU Blacksmith: both shards passed within 14 seconds of the 8-vCPU control; the free GitHub rest shard was slower and its lease run stalled.
3032

3133
Every HTTP and browser group consumes the same production build in an isolated job/database. Electron also uses that build and skips its development-route warm-up. Local HTTP and Electron development runs retain their dev-server behavior. Build inputs exclude test-only files so a harness edit can reuse an unchanged production build.
3234

35+
Desktop inbox protocol, assistant actions UI, and organization search UI run as three parallel HTTP jobs with isolated databases, apps, and reports. They retain one acceptance task boundary, so changing any of their harnesses selects all three jobs. Desktop inbox needs no browser. The UI jobs and browser extension install Chromium while their app starts, waiting for both before executing assertions; browser-install failures still fail the job and preserve the install log. No acceptance assertions are removed by the split.
36+
37+
Browser extension acceptance keeps native streaming and first-open capture checks on real timers. Its access-revalidation scenarios then reopen the persistent browser profile and install Playwright's clock before the app mounts. Advancing the production polling timers preserves real HTTP and database authorization checks without waiting through each polling interval. Capture timestamps originate in Chromium's isolated world, so the harness restores wall time after each advance. Live-provider runs retain native timing throughout.
38+
39+
Integration shard weights come from successful reports for both push and migration provisioning paths. Refresh them with `scripts/update-integration-durations.ts`, then check the new partition against a separate complete run before adopting it. Weights only change scheduling; discovery still owns coverage. The current refresh uses runs [41348786822](https://github.com/simstudioai/sim/actions/runs/41348786822) and [41377145857](https://github.com/simstudioai/sim/actions/runs/41377145857), with [41377810718](https://github.com/simstudioai/sim/actions/runs/41377810718) held out for validation.
40+
3341
SCIM's production app uses a `sim.ai` fixture hostname mapped to loopback, since `NEXT_PUBLIC_FORCE_HOSTED` is ignored in production. Readiness, HTTP tests, and internal API requests still connect to loopback. The SCIM suite verifies that ending an Enterprise subscription rejects both owner administration and an existing bearer token, and that restoring it restores access.
3442

3543
## Coverage removed or consolidated

‎.github/actions/setup/action.yml‎

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
name: setup
2-
description: Install the pinned Bun and Node toolchain, mount the dependency (and optionally Turbo) caches, and install workspace dependencies.
2+
description: Install the pinned toolchain and workspace dependencies, using sticky dependency disks on Blacksmith and optional Turbo output caching on either provider.
33

44
inputs:
55
provider:
@@ -19,12 +19,13 @@ inputs:
1919
required: false
2020
default: ''
2121

22-
# Cache keys are scoped by event name, and fork PRs get their own namespace on
22+
# Sticky disk keys are scoped by event name, and fork PRs get their own namespace on
2323
# top: untrusted fork runs must never share a cache with push runs (whose caches
2424
# feed production image builds) or with trusted internal-PR runs.
2525
#
26-
# node_modules also keys on the lockfile hash: a sticky disk is a mutable volume,
27-
# and `bun install --frozen-lockfile` adds what the lockfile needs without
26+
# GitHub runners install directly: restoring both dependency archives takes longer
27+
# than a clean Bun install. node_modules on sticky disks keys on the lockfile hash:
28+
# `bun install --frozen-lockfile` adds what the lockfile needs without
2829
# pruning what it dropped, so branches on different lockfiles were contaminating
2930
# each other (a stale @next/swc 16.2.6 outlived the 16.2.11 bump). The bun and
3031
# Turbo caches are content/hash-addressed, so they stay shared — that is what
@@ -45,27 +46,37 @@ runs:
4546
registry-url: ${{ inputs.registry-url }}
4647

4748
- name: Mount Bun cache
49+
if: inputs.provider == '' || inputs.provider == 'blacksmith'
4850
uses: ./.github/actions/cache
4951
with:
5052
provider: ${{ inputs.provider }}
5153
key: ${{ github.repository }}-bun-cache-${{ github.event_name }}${{ github.event.pull_request.head.repo.fork && '-fork' || '' }}
5254
path: ~/.bun/install/cache
5355

5456
- name: Mount node_modules
57+
if: inputs.provider == '' || inputs.provider == 'blacksmith'
5558
uses: ./.github/actions/cache
5659
with:
5760
provider: ${{ inputs.provider }}
5861
key: ${{ github.repository }}-node-modules-${{ github.event_name }}${{ github.event.pull_request.head.repo.fork && '-fork' || '' }}-${{ hashFiles('bun.lock') }}
5962
path: ./node_modules
6063

6164
- name: Mount Turbo cache
62-
if: inputs.turbo-cache-key != ''
65+
if: inputs.turbo-cache-key != '' && (inputs.provider == '' || inputs.provider == 'blacksmith')
6366
uses: ./.github/actions/cache
6467
with:
6568
provider: ${{ inputs.provider }}
6669
key: ${{ github.repository }}-${{ inputs.turbo-cache-key }}-${{ github.event_name }}${{ github.event.pull_request.head.repo.fork && '-fork' || '' }}
6770
path: ./.turbo
6871

72+
- name: Restore and save Turbo cache
73+
if: inputs.turbo-cache-key != '' && inputs.provider != '' && inputs.provider != 'blacksmith'
74+
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v5
75+
with:
76+
key: ${{ github.repository }}-${{ inputs.turbo-cache-key }}-${{ github.event_name }}${{ github.event.pull_request.head.repo.fork && '-fork' || '' }}-${{ github.sha }}
77+
restore-keys: ${{ github.repository }}-${{ inputs.turbo-cache-key }}-${{ github.event_name }}${{ github.event.pull_request.head.repo.fork && '-fork' || '' }}-
78+
path: ./.turbo
79+
6980
# Bun retries a failed request itself, but a registry connection that stalls
7081
# without failing leaves `bun install` waiting until the job's own timeout
7182
# cancels it, the whole budget spent on one hung step. Healthy installs finish

‎.github/ci-suites.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,14 @@
1616
"group": "desktop-inbox",
1717
"task": "ci:e2e-desktop-inbox"
1818
},
19+
{
20+
"group": "assistant-actions",
21+
"task": "ci:e2e-desktop-inbox"
22+
},
23+
{
24+
"group": "organization-search",
25+
"task": "ci:e2e-desktop-inbox"
26+
},
1927
{
2028
"group": "browser-extension",
2129
"task": "ci:e2e-browser-extension"

‎.github/scripts/http-e2e.sh‎

Lines changed: 52 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# CI groups, which reuse the production build from `e2e-build`.
44
# The job extracts it into apps/sim/.next before running these suites.
55
#
6-
# Usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|browser-extension|project-files|mobile|seo> (run from apps/sim)
6+
# Usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|project-files|mobile|seo> (run from apps/sim)
77
#
88
# The job provides DATABASE_URL, BETTER_AUTH_SECRET and ENCRYPTION_KEY; each group sets the rest of
99
# its app's environment here. Reports and server logs land in $RUNNER_TEMP/e2e.
@@ -20,7 +20,7 @@
2020
# telemetry flush runs detached and still writes .next/dev).
2121
set -euo pipefail
2222

23-
group=${1:?usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|browser-extension|project-files|mobile|seo>}
23+
group=${1:?usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|project-files|mobile|seo>}
2424
report_dir="$RUNNER_TEMP/e2e"
2525
ready_timeout_seconds=600
2626
mkdir -p "$report_dir"
@@ -71,6 +71,24 @@ finish() {
7171
}
7272
trap finish EXIT
7373

74+
start_browser_install() {
75+
local name=$1
76+
shift
77+
browsers_log="$report_dir/$name-playwright-install.log"
78+
setsid --wait bunx --no-install playwright install --with-deps "$@" > "$browsers_log" 2>&1 &
79+
browsers_pid=$!
80+
}
81+
82+
wait_for_browsers() {
83+
if ! wait "$browsers_pid"; then
84+
browsers_pid=''
85+
tail -n 100 "$browsers_log"
86+
echo "::error::Playwright browser install failed."
87+
exit 1
88+
fi
89+
browsers_pid=''
90+
}
91+
7492
# start_app <name> <port> <label> [record-http-status]
7593
start_app() {
7694
local name=$1 port=$2 label=$3
@@ -209,34 +227,41 @@ case "$group" in
209227
bun run test:workflow-stop-after:e2e
210228
;;
211229

212-
# The desktop background executor's protocol: device registration, the SSE doorbell over Redis
213-
# pub/sub, presence, leased claims, Stop and isolation.
214-
desktop-inbox)
230+
desktop-inbox|assistant-actions|organization-search)
215231
export REDIS_URL=redis://127.0.0.1:6379
216232
export NEXT_PUBLIC_FORCE_HOSTED=false
217233
export COPILOT_TOOL_PERMISSIONS_ENABLED=true
218234
export KNOWLEDGE_MEMBER_ACCESS=true CREDENTIAL_GROUPS=true ACCESS_CONTROL_ENABLED=true
219235
export NEXT_PUBLIC_ACCESS_CONTROL_ENABLED=true NEXT_PUBLIC_ORGANIZATIONS_ENABLED=true
220-
bunx --no-install playwright install --with-deps chromium
236+
if [ "$group" != desktop-inbox ]; then start_browser_install "$group" chromium; fi
221237
export INTERNAL_API_SECRET=desktop-inbox-http-ci-local-secret-at-least-32-characters
222238
export DB_TX_TRIPWIRE=throw
223-
start_app desktop-inbox 3019 'desktop executor' record-http-status
224-
DESKTOP_INBOX_E2E_BASE_URL="$NEXT_PUBLIC_APP_URL" \
225-
DESKTOP_INBOX_E2E_DATABASE_URL="$DATABASE_URL" \
226-
DESKTOP_INBOX_E2E_REDIS_URL="$REDIS_URL" \
227-
DESKTOP_INBOX_E2E_AUTH_SECRET="$BETTER_AUTH_SECRET" \
228-
DESKTOP_INBOX_E2E_REPORT_PATH="$report_dir/desktop-inbox-http-report.json" \
229-
bun run test:desktop-inbox:e2e
230-
ASSISTANT_ACTIONS_E2E_BASE_URL="$local_app_url" \
231-
ASSISTANT_ACTIONS_E2E_DATABASE_URL="$DATABASE_URL" \
232-
ASSISTANT_ACTIONS_E2E_AUTH_SECRET="$BETTER_AUTH_SECRET" \
233-
ASSISTANT_ACTIONS_E2E_REPORT_PATH="$report_dir/assistant-actions-ui-report.json" \
234-
bun --no-env-file scripts/test-assistant-actions-e2e.ts
235-
ORGANIZATION_SEARCH_E2E_BASE_URL="$local_app_url" \
236-
ORGANIZATION_SEARCH_E2E_DATABASE_URL="$DATABASE_URL" \
237-
ORGANIZATION_SEARCH_E2E_AUTH_SECRET="$BETTER_AUTH_SECRET" \
238-
ORGANIZATION_SEARCH_E2E_REPORT_PATH="$report_dir/organization-search-ui-report.json" \
239-
bun --no-env-file scripts/test-organization-search-e2e.ts
239+
start_app "$group" 3019 "$group" record-http-status
240+
if [ "$group" != desktop-inbox ]; then wait_for_browsers; fi
241+
case "$group" in
242+
desktop-inbox)
243+
DESKTOP_INBOX_E2E_BASE_URL="$NEXT_PUBLIC_APP_URL" \
244+
DESKTOP_INBOX_E2E_DATABASE_URL="$DATABASE_URL" \
245+
DESKTOP_INBOX_E2E_REDIS_URL="$REDIS_URL" \
246+
DESKTOP_INBOX_E2E_AUTH_SECRET="$BETTER_AUTH_SECRET" \
247+
DESKTOP_INBOX_E2E_REPORT_PATH="$report_dir/desktop-inbox-http-report.json" \
248+
bun run test:desktop-inbox:e2e
249+
;;
250+
assistant-actions)
251+
ASSISTANT_ACTIONS_E2E_BASE_URL="$local_app_url" \
252+
ASSISTANT_ACTIONS_E2E_DATABASE_URL="$DATABASE_URL" \
253+
ASSISTANT_ACTIONS_E2E_AUTH_SECRET="$BETTER_AUTH_SECRET" \
254+
ASSISTANT_ACTIONS_E2E_REPORT_PATH="$report_dir/assistant-actions-ui-report.json" \
255+
bun --no-env-file scripts/test-assistant-actions-e2e.ts
256+
;;
257+
organization-search)
258+
ORGANIZATION_SEARCH_E2E_BASE_URL="$local_app_url" \
259+
ORGANIZATION_SEARCH_E2E_DATABASE_URL="$DATABASE_URL" \
260+
ORGANIZATION_SEARCH_E2E_AUTH_SECRET="$BETTER_AUTH_SECRET" \
261+
ORGANIZATION_SEARCH_E2E_REPORT_PATH="$report_dir/organization-search-ui-report.json" \
262+
bun --no-env-file scripts/test-organization-search-e2e.ts
263+
;;
264+
esac
240265
;;
241266

242267
browser-extension)
@@ -245,7 +270,7 @@ case "$group" in
245270
export KNOWLEDGE_MEMBER_ACCESS=true CREDENTIAL_GROUPS=true ACCESS_CONTROL_ENABLED=true
246271
export NEXT_PUBLIC_ACCESS_CONTROL_ENABLED=true
247272
export API_ENCRYPTION_KEY="$ENCRYPTION_KEY"
248-
bunx --no-install playwright install --with-deps chromium
273+
start_browser_install browser-extension chromium
249274
extension_identity=$(bun - <<'JS'
250275
import { generateKeyPairSync, createHash } from 'node:crypto'
251276
const { publicKey } = generateKeyPairSync('rsa', { modulusLength: 2048 })
@@ -258,6 +283,7 @@ JS
258283
export SIM_EXTENSION_KEY="${extension_identity%%$'\n'*}"
259284
export BROWSER_EXTENSION_IDS="${extension_identity#*$'\n'}"
260285
start_app browser-extension 3007 'browser extension' record-http-status
286+
wait_for_browsers
261287
EXTENSION_APP_E2E_BASE_URL="$local_app_url" \
262288
EXTENSION_APP_E2E_DATABASE_URL="$DATABASE_URL" \
263289
EXTENSION_APP_E2E_REPORT_PATH="$report_dir/browser-extension/report.json" \
@@ -280,19 +306,10 @@ JS
280306
export INTERNAL_API_SECRET=mobile-http-ci-local-secret-at-least-32-characters
281307
export NEXT_PUBLIC_SOCKET_URL=http://127.0.0.1:3025 SOCKET_SERVER_URL=http://127.0.0.1:3025
282308
read -ra browsers <<< "${MOBILE_E2E_BROWSER:-chromium webkit}"
283-
# The browsers install while the app boots, in their own process group so finish() can stop them.
284-
browsers_log="$report_dir/mobile-playwright-install.log"
285-
setsid --wait bunx --no-install playwright install --with-deps "${browsers[@]}" > "$browsers_log" 2>&1 &
286-
browsers_pid=$!
309+
start_browser_install mobile "${browsers[@]}"
287310
start_app mobile 3024 'mobile browser' record-http-status
288311
start_realtime mobile 3025
289-
if ! wait "$browsers_pid"; then
290-
browsers_pid=''
291-
tail -n 100 "$browsers_log"
292-
echo "::error::Playwright browser install failed."
293-
exit 1
294-
fi
295-
browsers_pid=''
312+
wait_for_browsers
296313
MOBILE_E2E_BASE_URL="$NEXT_PUBLIC_APP_URL" \
297314
MOBILE_E2E_DATABASE_URL="$DATABASE_URL" \
298315
MOBILE_E2E_AUTH_SECRET="$BETTER_AUTH_SECRET" \

‎.github/workflows/checks.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -219,7 +219,7 @@ jobs:
219219
--health-retries 10
220220
# Desktop inbox, browser extension, and Project Files use Redis.
221221
redis:
222-
image: ${{ (matrix.group == 'desktop-inbox' || matrix.group == 'browser-extension' || matrix.group == 'project-files') && 'redis:8.2-alpine' || '' }}
222+
image: ${{ (matrix.group == 'desktop-inbox' || matrix.group == 'assistant-actions' || matrix.group == 'organization-search' || matrix.group == 'browser-extension' || matrix.group == 'project-files') && 'redis:8.2-alpine' || '' }}
223223
ports:
224224
- 6379:6379
225225
options: >-

0 commit comments

Comments
 (0)