Playwright end-to-end tests that run against a local PlaceOS stack. There is no
"which environment am I pointed at today" mode: assertLocalOnly() refuses any
non-loopback host before a browser launches. A suite that creates, mutates and
deletes real data must never be one typo away from doing it to a deployment.
Currently wired up for workplace. Other apps follow the same shape.
What is covered, and what is not:
../E2E_USER_STORIES.md. Adding a feature or fixing a bug means adding a row there and a spec, in the same PR — that document is what stops the suite decaying.
# 1. an isolated backend, seeded and ready (see "The stack" below)
e2e/stack/up.sh # --fresh to destroy its volumes first
# 2. the tests
export E2E_BACKEND_URL=https://localhost:9443
bunx playwright test --config apps/workplace/playwright.config.ts # everything
bunx playwright test --config apps/workplace/playwright.config.ts --project=local # real backend
bunx playwright test --config apps/workplace/playwright.config.ts --project=mock # no backend
bunx playwright test --config apps/workplace/playwright.config.ts --project=settings # Workplace settings, real backend
bunx playwright test --config apps/workplace/playwright.config.ts --project=settings-browser # Workplace settings, no backend
bunx playwright test --config apps/workplace/playwright.config.ts --ui # watch it| Project | Specs | Needs the stack | CI (e2e-advisory.yml) |
|---|---|---|---|
local |
apps/workplace/e2e/local/ |
yes | every run |
mock |
apps/workplace/e2e/*.spec.ts |
no | every run |
settings |
apps/workplace/e2e/settings/, except the browser-only ones |
yes | nightly and "Run workflow" only |
settings-browser |
the browser-only settings specs (apps/workplace/e2e/settings/**/*.browser.spec.ts) |
no | every run, on a GitHub-hosted runner |
local and settings depend on preflight, which checks the stack is up. Add
--no-deps to skip it.
Playwright starts the Angular dev server itself
(nx serve workplace --configuration=e2e --port=4214) and points its API proxy
at the stack. The e2e configuration turns Sentry off, so nothing reaches the
network.
If a run dies with "Executable doesn't exist", the workspace's pinned Playwright
needs its browser: bunx playwright install chromium.
To target a PlaceOS/local stack instead, set E2E_BACKEND_URL=https://localhost:8443
and run bunx tsx e2e/support/seed.ts once. Note that seeding mutates that stack
(it adds an OAuth app, a tenant, test users, and makes login_url absolute), so
prefer the isolated stack if someone else is using yours.
e2e/stack/ is a self-contained PlaceOS deployment for testing — not
PlaceOS/local. It runs as its own compose project (placeos-e2e) with its own
network, volumes and secrets, on shifted ports (9443/9080), so it coexists with a
developer's own stack and cannot disturb it.
e2e/stack/up.sh # bring up + seed (reuses volumes)
e2e/stack/up.sh --fresh # destroy volumes first, driver binaries included — a genuine cold start
e2e/stack/up.sh --pull # pull the current images first (what CI does every run)
e2e/stack/up.sh --pull-only # pull and stop; CI runs this as its own bounded step
e2e/stack/down.sh # stop (--volumes to wipe; driver binaries are kept)It is trimmed to what the e2e path exercises: postgres, elasticsearch, redis, search-ingest, frontend-loader, auth, rest-api, staff-api, core, source, mosquitto, nginx, init. Dropped from PlaceOS/local: edge, triggers, dispatch, influx, chronograf, minio and the loki/grafana profile.
The calendar and directory are backed by the PlaceOS Microsoft 365 sandbox
tenant. Set E2E_O365_TENANT, E2E_O365_CLIENT_ID and E2E_O365_CLIENT_SECRET
in e2e/.env or the environment before up.sh (CI takes them from the
repository variables and secret of the same names); the seed stops with an error
naming whichever is missing. It writes app-only credentials to the tenant row,
creates a local admin whose address is a mailbox there (roleFor('calendar')),
and /api/staff/v1/events and /api/staff/v1/people answer from the tenant.
Only the seed reads the variables: specs ask the stack (calendarBacked in
e2e/support/calendar/calendar.api.ts). See e2e/support/calendar/calendar.env.ts.
core runs real drivers from PlaceOS/drivers, pinned to a commit in
e2e/support/drivers/drivers.env.ts. It compiles nothing itself: binaries come
from the PlaceOS build farm (build.placeos.run), which builds a commit it has not
seen for the CPU architecture on first request and serves a download after that.
seed.ts creates the driver rows and waits for core to hold the binaries, so a
spec never waits on the farm. If the drivers do not load, the seed logs a warning
and continues: only the specs that need a room status fail. To test a drivers branch against the suite, set
E2E_DRIVERS_URI, E2E_DRIVERS_BRANCH and E2E_DRIVERS_COMMIT before up.sh.
Two things a cold start taught us that a long-lived stack hides:
frontend-loaderis not optional, even though the SPA is served by the dev server. On startup it clonesPlaceOS/www-coreinto the sharedwwwvolume, and that is where the platform's/loginpage lives. Without it nginx serves a bare 404 at/login.up.shgates on/loginreturning 200 before seeding. (This is the one point where bring-up needs network access to GitHub.)- A fresh authority gets a RELATIVE
login_url(/login?continue={{url}}), and ts-client resolves that against the authority'sdomaincolumn — a bare host with no port — so on any non-default port it navigates to a dead URL (http://localhost/login) and the app dead-ends on a browser error page.seed.tspatcheslogin_urlto an absolute URL. A long-lived stack usually has an absolute value already, which is why this never shows up locally.
Stagehand reads module state from MQTT, so the stack carries that path:
driver --> redis --> source --> mosquitto:1883 (source publishes, API key auth)
mosquitto:9001 <-- nginx /api/mqtt/ <-- browser (bearer auth)
sourcepublishes to Broker rows, nothing else.e2e/support/mqtt/mqtt.seed.tscreates one formosquitto:1883, signed in with an API key minted as Support (rest-api only lets support users publish).up.shruns it.- mosquitto asks rest-api about every client and topic, as a deployment does. The routes
are
/api/engine/v2/mqtt/userand/mqtt/access. PlaceOS/local's config namesmqtt_user/mqtt_access, which 404, so every client is refused there;e2e/stack/mosquitto.confhas the working names. - A subscription is sent everything retained under it at once, and mosquitto drops what
does not fit the client's queue.
mosquitto.confsetsmax_queued_messages 10000, as deployments now do; the default of 1000 cut wide subscriptions short on large sites. - Restarting mosquitto empties it (no persistence). source before v1.22.1 (so
placeos-2.2610.1) neither republishes after reconnecting nor retries a broker it could not reach at startup, so restart source after mosquitto. v1.22.1 (PlaceOS/source#65) does both. - Test against a newer source than the release:
E2E_SOURCE_IMAGE=placeos/source:nightly, or build it (docker build -t placeos/source:local .in the source clone), thenE2E_SOURCE_IMAGE=placeos/source:local e2e/stack/up.sh, or recreate just that service withLOG_LEVEL=info E2E_SOURCE_IMAGE=placeos/source:local docker compose -p placeos-e2e -f e2e/stack/docker-compose.yml up -d --no-deps --force-recreate source.LOG_LEVEL=infoshows source'sreconciled retained stateandswept retained statelines. - Assert the broker against the truth, not a value.
expectedTopics(stagehand.api.ts) builds every topic a system should have from its module order, names, zones and each module's own state;retainedForSystem(broker-view.ts) is what a fresh client is handed. The lifecycle specs require them to be equal, so a stale, missing or misplaced topic all fail the same way. - Do not run a Crystal spec suite in Docker alongside the stack on an 8 GB Docker VM: the compiler is OOM-killed, and takes Elasticsearch with it. Stop the stack first.
- Specs read the broker as well as the page.
e2e/support/mqtt/broker-view.tsconnects the way stagehand does:BrokerViewfollows topics live,retainedSnapshotis what a client connecting now is handed. Stagehand re-checks alerts over REST, so a right-looking page does not mean the broker is right.
The specs drive state with drivers/place/e2e/state_harness.cr, pinned in
e2e/support/drivers/drivers.env.ts (HARNESS_*). It is a UDP device, so it reports
connected: true without a peer; set_connected, set_status and clear_status change what
it reports. Create a harness module with running: true rather than creating then starting it:
core drops a start that arrives before it has loaded the module (No protocol manager), and
the module stays running with no process behind it.
If the build farm cannot build for this machine's architecture (arm64 answered 502 for every build on 2026-10-14 while amd64 worked), run core as amd64 under emulation:
DOCKER_DEFAULT_PLATFORM=linux/amd64 docker compose -p placeos-e2e -f e2e/stack/docker-compose.yml \
up -d --force-recreate --no-deps coree2e/ shared engine, all apps
stack/ isolated PlaceOS deployment (compose + up.sh/down.sh)
support/
load-env.ts dotenv side-effect loader
env.ts config + per-worker roles + assertLocalOnly()
auth.ts headless PKCE mint -> storageState
login.ts drives the REAL login form
fixtures.ts worker-scoped auth fixtures — import test/expect from here
background.ts pauses the animated page background in a browser context
teardown.ts tells teardown errors from app errors (closedByTeardown, jsonBody)
api.ts engine/staff-api helpers + asset sweep
flows.ts multi-step UI flows (bookDeskViaUI)
visitor/ visitor-invite coverage, kept out of the files above
visitor.env.ts per-worker visitor addresses + slot table
visitor.api.ts guest + visitor-booking helpers and sweeps
visitor.settings.ts per-context app settings presets
invite-form.page.ts locators for the invite page (`visitor-flow-new`)
your-bookings.page.ts locators for the schedule + details modal
visitor.flows.ts multi-step UI flows (inviteVisitorViaUI)
room/ room-booking coverage, kept out of the files above
room.env.ts per-worker rooms + slot table
room.seed.ts creates the rooms as engine Systems (needs ADMIN)
room.api.ts room-booking helpers, sweeps, tryRoomBooking
room.settings.ts per-context app settings presets (use_bookings mode)
meeting-flow.page.ts locators for the `meeting-flow-new` booking flow
schedule.page.ts extends visitor/your-bookings.page.ts for event cards
room.flows.ts multi-step UI flows (bookRoomViaUI)
mqtt/ the MQTT path stagehand reads
mqtt.seed.ts the Broker row source publishes to
broker-view.ts an MQTT client that sees what stagehand sees
stagehand/ stagehand coverage
stagehand.env.ts app URL and timings
stagehand.seed.ts OAuth app for the stagehand origin + the harness driver
stagehand.api.ts systems and harness modules
alerts.page.ts the hot list
fixtures.ts admin session, bearer, harness driver, zones
repro/ standalone reproducers for confirmed backend bugs
preflight.ts "is the stack up?" — fails in 1s, not 90
preflight.setup.ts setup project: `local` and `settings` depend on it
seed.ts idempotent API-driven seeding
.env.example copy to .env to override anything
apps/workplace/
playwright.config.ts projects: preflight, local, settings, settings-browser, mock
e2e/
local/*.spec.ts real-backend specs
settings/ Workplace settings specs (`settings`, `settings-browser`)
*.spec.ts mock-mode specs (no backend)
Under nx serve the app and the backend are different origins, and conflating
them is the fastest way to a mysteriously unauthenticated SPA:
E2E_BACKEND_URL |
https://localhost:8443 — the stack (nginx TLS entrypoint) |
E2E_APP_URL |
http://localhost:4214 — the dev server serving the SPA |
The dev server proxies /api, /auth, /control through to the backend, so the
browser only ever sees E2E_APP_URL — and that is the origin localStorage (and
therefore the token) belongs to.
The real login flow is the primary path. login.ts drives it: unauthenticated
visit → redirect to the authority's login page (a cross-origin hop, with continue
carrying the return URL) → credentials → back into the app. Use it for anything
whose subject is authentication. apps/workplace/e2e/local/login.spec.ts is the
worked example, and it guards the exact class of regression that has bitten
production: scope surviving a refresh, rotation, and rest-api still accepting the
refreshed token.
Injected tokens are a speed optimisation, not the default way in. The worker
fixtures mint a bearer over PKCE and write a storageState seeding
localStorage[${cid}_access_token], so a spec that merely needs to be logged in
skips ~5s of form driving. Every spec gets this automatically. Specs that test auth
opt out:
test.use({ storageState: undefined });Things that will bite you:
- After a real login, the access token is NOT in localStorage. ts-client only
persists it when the device is trusted —
_storeTokenDetailsgates thesetItemonisTrusted()(ts-client/src/auth/functions.ts:1042); otherwise the bearer lives in memory and only${cid}_expires_atis written. So "the token is in localStorage" is true of a seeded state and false of a real login. Assert on the token exchange or on/users/current, never on the key. cidis derived from where the app is served. ts-client computesredirect_uri = ${location.origin}${route}oauth-resp.htmlat runtime, so the dev server and a stack-deployed build are different clients and each needs its own registered application.seed.tsregisters the dev-server one._expires_atmust be in the future in a seeded state, or ts-client treats the token as stale and triggers a re-auth redirect mid-test.- Do not inject
sessionStorage['ENGINE.auth.params']. The app deletes it on init; re-adding a stalecodemakes ts-client try to re-exchange it and auth breaks. - nginx gates static assets behind an HMAC
verifiedcookie. Irrelevant while the dev server serves the SPA, but a suite pointed at a stack-served build (https://localhost:8443/workplace/) will 302 to/auth/loginunless the cookie is carried across.mintTokenalready captures it.
The suite runs fullyParallel across E2E_WORKERS (default 4). We own the whole
stack, so isolation is a seeding problem rather than a reason to serialise:
seed.ts provisions one non-admin identity per worker (e2e-staff-0@place.tech …)
and fixtures.ts mints per worker, keyed on parallelIndex.
Fixtures available:
| Fixture | What |
|---|---|
page |
authenticated as this worker's admin (the default storageState) |
staffPage |
a page authenticated as this worker's non-admin |
staffApi |
an APIRequestContext as this worker's non-admin |
Assert permission-gated behaviour as staffPage/staffApi. A sys_admin
bypasses the checks, so the same assertion made as admin passes whether or not the
permission logic works.
Raise E2E_WORKERS and re-run seed.ts to scale out; keep the config's workers
at or below it.
| Step | Why |
|---|---|
OAuth application for the dev-server redirect_uri |
ts-client's client_id is Md5(redirect_uri); authorize refuses without a matching uid. |
An absolute authority login_url |
A relative one resolves without the port and dead-ends the login redirect (see "The stack"). |
| A tenant for the backend domain | staff-api rejects every /bookings and /events call with "domain does not have a tenant configured" until one exists. |
| One non-admin user per worker | Permission gating needs a non-admin; parallel mutation needs distinct identities. |
seed.ts polls for the authority rather than reading it once: /domains is
served from Elasticsearch, so on a cold stack the row exists in Postgres before
the API can see it. Failing fast there is the single most likely way a CI run
breaks, and the error looks nothing like the cause.
It bootstraps through the backoffice application, which init always creates —
otherwise registering an OAuth app would require a token that requires an OAuth app.
staff-api only dereferences tenant credentials when it instantiates a PlaceCalendar client, which happens on the calendar-backed routes and nowhere else:
GET /bookings?type=desk|locker|parking|visitor|room -> PlaceOS only
GET /calendars, GET /events, GET /people -> the sandbox tenant
Room specs that set app.events.use_bookings = true save an ordinary room
booking through /bookings (see ROOM_E2E_HANDOVER.md),
so they exercise the PlaceOS side of rooms, not the calendar path.
.github/workflows/e2e-advisory.yml — runs hourly at :40 when develop has a new
commit since the previous run, nightly at 01:10 UTC on every day, and via Run
workflow on any branch. There is deliberately no pull_request trigger (see the table
below — it is a security decision, not an oversight), so the hourly run is the thing that
covers a merge. It brings up the isolated stack, seeds it,
runs the suite, writes a step summary, and uploads the HTML report plus (on failure) traces,
videos and backend container logs.
There are no path filters, on purpose — most real changes touch libs/**, which workplace
depends on, and the genuinely dangerous ones (bun.lock, tsconfig.base.json, config/) are
the easiest to leave off an include-list. A filter that is 95% right silently skips the run
that mattered.
There is deliberately no push trigger: there is one self-hosted runner binding fixed host
ports, so runs serialise, and a run per push built a queue on busy days. The hourly check
(job changes) runs on a GitHub-hosted runner, so an hour with no new commit costs the
self-hosted runner nothing.
It is advisory and must stay that way for now — do not add it to branch protection or
required status checks until the suite has a track record. See the CI section of
../E2E_USER_STORIES.md for the bar it needs to clear first.
It runs on a self-hosted macOS runner (label placeos-e2e). Setup, the network
requirements, and the macOS-specific gotchas are in
stack/SELF_HOSTED_RUNNER.md.
CI-specific choices worth knowing:
E2E_WORKERS: 6 |
Higher than the local default of 4. The runner also hosts ~10 containers, a dev server and the browsers. Before the specs paused the animated page background (support/background.ts), four workers used all the CPU and six were no faster. Now six workers run both test steps in about 19 minutes, against 23 minutes with four. Seeding reads the same value, so set it once for the job. |
E2E_WEBSERVER_TIMEOUT_MS: '900000' (settings-browser job only) |
The config's webServer timeout, 180000 ms unless set. The GitHub-hosted runner starts nx serve workplace cold with no Nx cache, which can take longer than the default. |
No pull_request trigger |
This repo is public and the runner is self-hosted, so a fork PR could run arbitrary code on an internal machine. Triggers are limited to ones requiring write access. Do not add it back without moving to ubuntu-latest. |
| Reclaim step | A self-hosted machine is not a fresh VM; a previous aborted run can leave the stack up or port 4214 held. |
No vm.max_map_count bump |
Needed on GitHub-hosted Linux, meaningless on macOS — the value lives inside Colima's VM, which already sets it to 1048576. Restore it if reverting to ubuntu-latest. |
| Network access to GitHub | up.sh clones PlaceOS/www-core into the www volume once — that is where the platform /login page comes from. |
| Network access to the build farm | core fetches driver binaries from build.placeos.run and the S3 bucket it hands out links to. A commit the farm has not built for the runner's architecture is compiled there on the first run that asks. The binaries stay in the placeos-e2e-core-drivers volume between runs, so later runs do not ask again. |
| Network access to Docker Hub, every run | up.sh --pull-only runs first as its own step, bounded to 20 minutes and advisory, so latest tracks the registry rather than the day the runner was set up. Each image pulls on its own; one that fails or stalls leaves the stack on the image already present, and "Record backend inputs" shows which. SELF_HOSTED_RUNNER.md lists the hosts. |
Learned the hard way while building the desk specs. Each of these produced a failure that looked like something else entirely.
GET /bookingsis scoped to the CALLER. Listing as an admin does not show other users' bookings, so "I checked for leaks as admin and found none" proves nothing. Verify against the database, or list as the owning user. Deleting is caller-scoped too.- The list query parameter is
type, notbooking_type. The latter is the model field and appears in the controller'sPARAMS, which makes the source misleading; the request 422s withmissing required parameter 'type'. - Desks come from Zone METADATA, not systems —
metadata.desks.details[]on a level zone.groups: []leaves them unrestricted; a populatedgroupsgates them behind membership. - A desk is exclusive for its time range. Each worker books its own desk, and
each spec sweeps that desk before booking (
releaseAsset). A post-testfinallyalone is not enough: if a run dies mid-flight the leftover holds the desk, and because these are all-day bookings it holds it for the rest of the day. Every later run then fails as "desk not offered" or a 422 with an emptyfailuresarray — nothing that points at the real cause. POST /bookingsoccasionally 500s withDB::ConnectionLostunder concurrent writes — roughly 1 run in 8 at 4 workers, on stock connection settings (which PlaceOS/local ships too). This is a staff-api defect, not a rig artifact; sizing the connection pool in compose was tried and did not demonstrably help. CI'sretries: 2absorbs it and reports the test asflaky, which is the right outcome: the gate stays usable and the instability stays visible. Worth fixing in staff-api on its own merits.- Calendar routes reach the internet.
/calendars,/eventsand/peoplecalllogin.microsoftonline.comand Graph for the sandbox tenant, so a run needs those hosts to be reachable.
- One spec owns its own data. Create it, assert it, delete it in a
finally. UseuniqueTitle()so concurrent workers cannot collide. - Sweep before you create, not just after. Post-test cleanup handles the happy path; a pre-test sweep is what lets a spec recover from a previous crashed run.
- Never blindly toggle a control — ensure the state you want. Toggling a checkbox that happened to start in the target state silently inverts your intent, and the flow then dead-ends in a way that reads as a selector bug.
- Assert through the API, not only UI text. A green screen reflecting stale state is worse than a red test.
- Every bugfix PR adds a regression spec carrying the ticket ID, and a
REG-*row inE2E_USER_STORIES.md. A regression spec that was never seen red is a guess, not a guard. - Red-check new assertions. Break the expected value once and confirm the test fails for the reason you think. An assertion that never observed real data is worse than no assertion.
- "Nothing is shown" needs a control. An empty state is also what a page shows before its data loads. Hold a second item that the page can only show when it has loaded, and assert on that.
- A workaround for an app defect goes in the report. A helper that retries past
a known defect adds an annotation to the test each time the retry fires:
test.info().annotations.push({ type: 'NEW-F5', description }). The type is the id of the finding inE2E_USER_STORIES.md. Aconsole.warnalone is not read on a green run, and the defect can then get worse with no sign. - Do not make a booking of TODAY for the user of another worker. The home page
shows the first booking a user has today, so the home specs delete every booking
their user has today (
clearToday). Book a later day. - A context you create yourself calls
pauseBackground(context)before its first page. Thecontextfixture does it for you. The animated page background costs headless Chromium about 5 CPU-s per page load, and on the CPU-bound CI runner that is most of the run time. - The app can still have requests in flight when a test ends. A route handler
that calls
route.fetch(), or a response body kept for later, then fails with a closed-target error. Wrap such a handler inignoreClosed, and read a kept body withjsonBody(response). The settings audits drop closed-target errors that are recorded after their teardown starts. .auth/and.envare gitignored — those tokens are impersonation-grade.