Skip to content

test(e2e): replay recorded weather app project end to end in CI - #81

Merged
TheGreatAxios merged 1 commit into
mainfrom
cl-11938-replay-a-whole-builder-project-from-recorded-model-turns-in
Nov 2, 2026
Merged

TheGreatAxios merged 1 commit into
mainfrom
cl-11938-replay-a-whole-builder-project-from-recorded-model-turns-in

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Replays a full Builder project end to end against a real Hub, Postgres, sidecar and workspace sandbox without calling a live model provider.

  • test/e2e/support/recorded-model.ts serves an OpenAI-compatible /v1/chat/completions SSE endpoint backed by recorded exchanges under test/fixtures/pipeline/. Each request matches its expect.json structurally by run kind, ordered tool declarations, user/tool turn shapes and required tool-result substrings, captures runtime values (approvalId, appUrl) from tool results and substitutes {{bind:<key>}} placeholders in streamed tool call arguments. Any divergence fails with the scenario, stage, run, exchange index and re-record instructions.
  • test/e2e/pipeline.test.ts drives the animated weather app through the planning documents (problem_brief, constraints, approaches v1, reviewer revision to approaches v2, product_requirements), the three-screen HTML design (check_design in headless Chrome), build_plan, the coder (app_status, Playwright browser_* checks, bun test, git commit) and the built app (v1 from starter setup and v2 after the coder commits, fetched via GET /api/projects/:id/app, extracted and verified with bun test), asserting each stage's Hub state and the ordered runs and artifacts sequences against manifest.json.
  • apps/hub/src/builds/posix-sandbox.ts, apps/hub/src/builds/sidecar-tools.ts and apps/hub/src/live-app/host.ts pass --conditions=intx-src to the sandboxed tool child, keep TMPDIR pointed at a writable directory inside wrapped commands while denying writes to /tmp/claude, allow the Linux proxy bridge Unix sockets in allowRead, relay the dev app over a Unix socket across Linux sandbox network namespaces (createAppRelay with bwrap mount/PID isolation), enable allowAllUnixSockets on Linux for sidecar-owned browser/relay helpers after wrapping the agent tool child, and prefix Linux sandbox wrappers with exec so kill terminates the bubblewrap namespace owner directly.
  • apps/hub/src/middleware/correlation.ts attaches requestContext as contextLocalStorage when LogTape was initialized with the default sink before setupHubLogging().
  • .github/workflows/ci.yml keeps e2e on the fast tests (custom-provider, oauth-model-login, specialist-end-notices, recorded-model) and runs test/e2e/pipeline.test.ts in its own e2e-pipeline job (continue-on-error: true).
  • test/AGENTS.md documents the test layout, how recorded replay works and how to update or re-record fixtures when prompts, tools or starter files change.

Verified with bun run format:check, bun run lint, bun run typecheck, bun run test, bun --conditions=intx-src test ./test/e2e/support/recorded-model.test.ts, bun --conditions=intx-src test ./test/e2e/pipeline.test.ts, bun run pins:verify and bun run vendor:verify.

@linear-code

linear-code Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

CL-11938

CL-11848

CL-11875

@TheGreatAxios
TheGreatAxios force-pushed the cl-11938-replay-a-whole-builder-project-from-recorded-model-turns-in branch 3 times, most recently from cc9d3a6 to 36a1530 Compare November 2, 2026 05:40
@TheGreatAxios
TheGreatAxios marked this pull request as ready for review November 2, 2026 05:40
@TheGreatAxios
TheGreatAxios force-pushed the cl-11938-replay-a-whole-builder-project-from-recorded-model-turns-in branch 2 times, most recently from a6d0ed6 to 4b1f787 Compare November 2, 2026 05:54

@corbits-triage corbits-triage Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Triage requests changes:

  • Conventions: added code matches a code pattern
    • apps/hub/src/builds/posix-sandbox.ts:435 empty-list-fallback: Invented fallback `?? []`; handle the missing value instead.
    • apps/hub/src/builds/posix-sandbox.ts:527-541 nested-ternary: Nested ternary; use if statements or a lookup.
    • apps/hub/src/builds/posix-sandbox.ts:784-845 iife: Immediately invoked function; name the function.
    • apps/hub/src/builds/posix-sandbox.ts:784-845 promise-chain: Promise chain; use async/await with try/catch.
    • apps/hub/src/builds/sidecar-tools.ts:225-300 promise-chain: Promise chain; use async/await with try/catch.
    • test/e2e/support/extract.ts:118 empty-list-fallback: Invented fallback `?? []`; handle the missing value instead.
    • workflows/coder/check/check-app.ts:576-581 promise-chain: Promise chain; use async/await with try/catch.

@TheGreatAxios
TheGreatAxios force-pushed the cl-11938-replay-a-whole-builder-project-from-recorded-model-turns-in branch from 4b1f787 to bcd5608 Compare November 2, 2026 06:32

@corbits-triage corbits-triage Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Triage requests changes:

  • Conventions: added code matches a code pattern
    • apps/hub/src/builds/posix-sandbox.ts:567-586 nested-ternary: Nested ternary; use if statements or a lookup.
    • apps/hub/src/builds/posix-sandbox.ts:829-890 iife: Immediately invoked function; name the function.
    • apps/hub/src/builds/posix-sandbox.ts:829-890 promise-chain: Promise chain; use async/await with try/catch.
    • apps/hub/src/builds/sidecar-tools.ts:225-300 promise-chain: Promise chain; use async/await with try/catch.
    • test/e2e/support/extract.ts:118 empty-list-fallback: Invented fallback `?? []`; handle the missing value instead.
    • workflows/coder/check/check-app.ts:576-581 promise-chain: Promise chain; use async/await with try/catch.

@TheGreatAxios
TheGreatAxios force-pushed the cl-11938-replay-a-whole-builder-project-from-recorded-model-turns-in branch from bcd5608 to ecb10db Compare November 2, 2026 06:53
@corbits-triage

corbits-triage Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Triage: In progress

Triage is waiting for CI to finish before it decides. This comment shows the verdict once it has.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant