Skip to content

Latest commit

 

History

History
430 lines (365 loc) · 26.2 KB

README.md

File metadata and controls

430 lines (365 loc) · 26.2 KB

PlaceOS e2e — local backend only

Playwright end-to-end tests that run against a local PlaceOS stack. There is no "which environment am I pointed at today" mode: assertLocalOnly() refuses any non-loopback host before a browser launches. A suite that creates, mutates and deletes real data must never be one typo away from doing it to a deployment.

Currently wired up for workplace. Other apps follow the same shape.

What is covered, and what is not: ../E2E_USER_STORIES.md. Adding a feature or fixing a bug means adding a row there and a spec, in the same PR — that document is what stops the suite decaying.

Run it

# 1. an isolated backend, seeded and ready (see "The stack" below)
e2e/stack/up.sh              # --fresh to destroy its volumes first

# 2. the tests
export E2E_BACKEND_URL=https://localhost:9443
bunx playwright test --config apps/workplace/playwright.config.ts                    # everything
bunx playwright test --config apps/workplace/playwright.config.ts --project=local    # real backend
bunx playwright test --config apps/workplace/playwright.config.ts --project=mock     # no backend
bunx playwright test --config apps/workplace/playwright.config.ts --project=settings          # Workplace settings, real backend
bunx playwright test --config apps/workplace/playwright.config.ts --project=settings-browser  # Workplace settings, no backend
bunx playwright test --config apps/workplace/playwright.config.ts --ui               # watch it
Project Specs Needs the stack CI (e2e-advisory.yml)
local apps/workplace/e2e/local/ yes every run
mock apps/workplace/e2e/*.spec.ts no every run
settings apps/workplace/e2e/settings/, except the browser-only ones yes nightly and "Run workflow" only
settings-browser the browser-only settings specs (apps/workplace/e2e/settings/**/*.browser.spec.ts) no every run, on a GitHub-hosted runner

local and settings depend on preflight, which checks the stack is up. Add --no-deps to skip it.

Playwright starts the Angular dev server itself (nx serve workplace --configuration=e2e --port=4214) and points its API proxy at the stack. The e2e configuration turns Sentry off, so nothing reaches the network.

If a run dies with "Executable doesn't exist", the workspace's pinned Playwright needs its browser: bunx playwright install chromium.

To target a PlaceOS/local stack instead, set E2E_BACKEND_URL=https://localhost:8443 and run bunx tsx e2e/support/seed.ts once. Note that seeding mutates that stack (it adds an OAuth app, a tenant, test users, and makes login_url absolute), so prefer the isolated stack if someone else is using yours.

The stack

e2e/stack/ is a self-contained PlaceOS deployment for testing — not PlaceOS/local. It runs as its own compose project (placeos-e2e) with its own network, volumes and secrets, on shifted ports (9443/9080), so it coexists with a developer's own stack and cannot disturb it.

e2e/stack/up.sh              # bring up + seed (reuses volumes)
e2e/stack/up.sh --fresh      # destroy volumes first, driver binaries included — a genuine cold start
e2e/stack/up.sh --pull       # pull the current images first (what CI does every run)
e2e/stack/up.sh --pull-only  # pull and stop; CI runs this as its own bounded step
e2e/stack/down.sh            # stop        (--volumes to wipe; driver binaries are kept)

It is trimmed to what the e2e path exercises: postgres, elasticsearch, redis, search-ingest, frontend-loader, auth, rest-api, staff-api, core, source, mosquitto, nginx, init. Dropped from PlaceOS/local: edge, triggers, dispatch, influx, chronograf, minio and the loki/grafana profile.

The calendar and directory are backed by the PlaceOS Microsoft 365 sandbox tenant. Set E2E_O365_TENANT, E2E_O365_CLIENT_ID and E2E_O365_CLIENT_SECRET in e2e/.env or the environment before up.sh (CI takes them from the repository variables and secret of the same names); the seed stops with an error naming whichever is missing. It writes app-only credentials to the tenant row, creates a local admin whose address is a mailbox there (roleFor('calendar')), and /api/staff/v1/events and /api/staff/v1/people answer from the tenant. Only the seed reads the variables: specs ask the stack (calendarBacked in e2e/support/calendar/calendar.api.ts). See e2e/support/calendar/calendar.env.ts.

core runs real drivers from PlaceOS/drivers, pinned to a commit in e2e/support/drivers/drivers.env.ts. It compiles nothing itself: binaries come from the PlaceOS build farm (build.placeos.run), which builds a commit it has not seen for the CPU architecture on first request and serves a download after that. seed.ts creates the driver rows and waits for core to hold the binaries, so a spec never waits on the farm. If the drivers do not load, the seed logs a warning and continues: only the specs that need a room status fail. To test a drivers branch against the suite, set E2E_DRIVERS_URI, E2E_DRIVERS_BRANCH and E2E_DRIVERS_COMMIT before up.sh.

Two things a cold start taught us that a long-lived stack hides:

  • frontend-loader is not optional, even though the SPA is served by the dev server. On startup it clones PlaceOS/www-core into the shared www volume, and that is where the platform's /login page lives. Without it nginx serves a bare 404 at /login. up.sh gates on /login returning 200 before seeding. (This is the one point where bring-up needs network access to GitHub.)
  • A fresh authority gets a RELATIVE login_url (/login?continue={{url}}), and ts-client resolves that against the authority's domain column — a bare host with no port — so on any non-default port it navigates to a dead URL (http://localhost/login) and the app dead-ends on a browser error page. seed.ts patches login_url to an absolute URL. A long-lived stack usually has an absolute value already, which is why this never shows up locally.

MQTT

Stagehand reads module state from MQTT, so the stack carries that path:

driver --> redis --> source --> mosquitto:1883      (source publishes, API key auth)
                                 mosquitto:9001  <-- nginx /api/mqtt/ <-- browser (bearer auth)
  • source publishes to Broker rows, nothing else. e2e/support/mqtt/mqtt.seed.ts creates one for mosquitto:1883, signed in with an API key minted as Support (rest-api only lets support users publish). up.sh runs it.
  • mosquitto asks rest-api about every client and topic, as a deployment does. The routes are /api/engine/v2/mqtt/user and /mqtt/access. PlaceOS/local's config names mqtt_user/mqtt_access, which 404, so every client is refused there; e2e/stack/mosquitto.conf has the working names.
  • A subscription is sent everything retained under it at once, and mosquitto drops what does not fit the client's queue. mosquitto.conf sets max_queued_messages 10000, as deployments now do; the default of 1000 cut wide subscriptions short on large sites.
  • Restarting mosquitto empties it (no persistence). source before v1.22.1 (so placeos-2.2610.1) neither republishes after reconnecting nor retries a broker it could not reach at startup, so restart source after mosquitto. v1.22.1 (PlaceOS/source#65) does both.
  • Test against a newer source than the release: E2E_SOURCE_IMAGE=placeos/source:nightly, or build it (docker build -t placeos/source:local . in the source clone), then E2E_SOURCE_IMAGE=placeos/source:local e2e/stack/up.sh, or recreate just that service with LOG_LEVEL=info E2E_SOURCE_IMAGE=placeos/source:local docker compose -p placeos-e2e -f e2e/stack/docker-compose.yml up -d --no-deps --force-recreate source. LOG_LEVEL=info shows source's reconciled retained state and swept retained state lines.
  • Assert the broker against the truth, not a value. expectedTopics (stagehand.api.ts) builds every topic a system should have from its module order, names, zones and each module's own state; retainedForSystem (broker-view.ts) is what a fresh client is handed. The lifecycle specs require them to be equal, so a stale, missing or misplaced topic all fail the same way.
  • Do not run a Crystal spec suite in Docker alongside the stack on an 8 GB Docker VM: the compiler is OOM-killed, and takes Elasticsearch with it. Stop the stack first.
  • Specs read the broker as well as the page. e2e/support/mqtt/broker-view.ts connects the way stagehand does: BrokerView follows topics live, retainedSnapshot is what a client connecting now is handed. Stagehand re-checks alerts over REST, so a right-looking page does not mean the broker is right.

The specs drive state with drivers/place/e2e/state_harness.cr, pinned in e2e/support/drivers/drivers.env.ts (HARNESS_*). It is a UDP device, so it reports connected: true without a peer; set_connected, set_status and clear_status change what it reports. Create a harness module with running: true rather than creating then starting it: core drops a start that arrives before it has loaded the module (No protocol manager), and the module stays running with no process behind it.

If the build farm cannot build for this machine's architecture (arm64 answered 502 for every build on 2026-10-14 while amd64 worked), run core as amd64 under emulation:

DOCKER_DEFAULT_PLATFORM=linux/amd64 docker compose -p placeos-e2e -f e2e/stack/docker-compose.yml \
    up -d --force-recreate --no-deps core

Layout

e2e/                          shared engine, all apps
  stack/                      isolated PlaceOS deployment (compose + up.sh/down.sh)
  support/
    load-env.ts                 dotenv side-effect loader
    env.ts                      config + per-worker roles + assertLocalOnly()
    auth.ts                     headless PKCE mint -> storageState
    login.ts                    drives the REAL login form
    fixtures.ts                 worker-scoped auth fixtures — import test/expect from here
    background.ts               pauses the animated page background in a browser context
    teardown.ts                 tells teardown errors from app errors (closedByTeardown, jsonBody)
    api.ts                      engine/staff-api helpers + asset sweep
    flows.ts                    multi-step UI flows (bookDeskViaUI)
    visitor/                    visitor-invite coverage, kept out of the files above
      visitor.env.ts              per-worker visitor addresses + slot table
      visitor.api.ts              guest + visitor-booking helpers and sweeps
      visitor.settings.ts         per-context app settings presets
      invite-form.page.ts         locators for the invite page (`visitor-flow-new`)
      your-bookings.page.ts       locators for the schedule + details modal
      visitor.flows.ts            multi-step UI flows (inviteVisitorViaUI)
    room/                     room-booking coverage, kept out of the files above
      room.env.ts                 per-worker rooms + slot table
      room.seed.ts                creates the rooms as engine Systems (needs ADMIN)
      room.api.ts                 room-booking helpers, sweeps, tryRoomBooking
      room.settings.ts            per-context app settings presets (use_bookings mode)
      meeting-flow.page.ts        locators for the `meeting-flow-new` booking flow
      schedule.page.ts            extends visitor/your-bookings.page.ts for event cards
      room.flows.ts               multi-step UI flows (bookRoomViaUI)
    mqtt/                       the MQTT path stagehand reads
      mqtt.seed.ts                the Broker row source publishes to
      broker-view.ts              an MQTT client that sees what stagehand sees
    stagehand/                  stagehand coverage
      stagehand.env.ts            app URL and timings
      stagehand.seed.ts           OAuth app for the stagehand origin + the harness driver
      stagehand.api.ts            systems and harness modules
      alerts.page.ts              the hot list
      fixtures.ts                 admin session, bearer, harness driver, zones
    repro/                      standalone reproducers for confirmed backend bugs
    preflight.ts                "is the stack up?" — fails in 1s, not 90
    preflight.setup.ts          setup project: `local` and `settings` depend on it
    seed.ts                     idempotent API-driven seeding
  .env.example                  copy to .env to override anything
apps/workplace/
  playwright.config.ts        projects: preflight, local, settings, settings-browser, mock
  e2e/
    local/*.spec.ts             real-backend specs
    settings/                   Workplace settings specs (`settings`, `settings-browser`)
    *.spec.ts                   mock-mode specs (no backend)

Two origins, not one

Under nx serve the app and the backend are different origins, and conflating them is the fastest way to a mysteriously unauthenticated SPA:

E2E_BACKEND_URL https://localhost:8443 — the stack (nginx TLS entrypoint)
E2E_APP_URL http://localhost:4214 — the dev server serving the SPA

The dev server proxies /api, /auth, /control through to the backend, so the browser only ever sees E2E_APP_URL — and that is the origin localStorage (and therefore the token) belongs to.

Auth: two paths, and when to use which

The real login flow is the primary path. login.ts drives it: unauthenticated visit → redirect to the authority's login page (a cross-origin hop, with continue carrying the return URL) → credentials → back into the app. Use it for anything whose subject is authentication. apps/workplace/e2e/local/login.spec.ts is the worked example, and it guards the exact class of regression that has bitten production: scope surviving a refresh, rotation, and rest-api still accepting the refreshed token.

Injected tokens are a speed optimisation, not the default way in. The worker fixtures mint a bearer over PKCE and write a storageState seeding localStorage[${cid}_access_token], so a spec that merely needs to be logged in skips ~5s of form driving. Every spec gets this automatically. Specs that test auth opt out:

test.use({ storageState: undefined });

Things that will bite you:

  • After a real login, the access token is NOT in localStorage. ts-client only persists it when the device is trusted — _storeTokenDetails gates the setItem on isTrusted() (ts-client/src/auth/functions.ts:1042); otherwise the bearer lives in memory and only ${cid}_expires_at is written. So "the token is in localStorage" is true of a seeded state and false of a real login. Assert on the token exchange or on /users/current, never on the key.
  • cid is derived from where the app is served. ts-client computes redirect_uri = ${location.origin}${route}oauth-resp.html at runtime, so the dev server and a stack-deployed build are different clients and each needs its own registered application. seed.ts registers the dev-server one.
  • _expires_at must be in the future in a seeded state, or ts-client treats the token as stale and triggers a re-auth redirect mid-test.
  • Do not inject sessionStorage['ENGINE.auth.params']. The app deletes it on init; re-adding a stale code makes ts-client try to re-exchange it and auth breaks.
  • nginx gates static assets behind an HMAC verified cookie. Irrelevant while the dev server serves the SPA, but a suite pointed at a stack-served build (https://localhost:8443/workplace/) will 302 to /auth/login unless the cookie is carried across. mintToken already captures it.

Parallelism

The suite runs fullyParallel across E2E_WORKERS (default 4). We own the whole stack, so isolation is a seeding problem rather than a reason to serialise: seed.ts provisions one non-admin identity per worker (e2e-staff-0@place.tech …) and fixtures.ts mints per worker, keyed on parallelIndex.

Fixtures available:

Fixture What
page authenticated as this worker's admin (the default storageState)
staffPage a page authenticated as this worker's non-admin
staffApi an APIRequestContext as this worker's non-admin

Assert permission-gated behaviour as staffPage/staffApi. A sys_admin bypasses the checks, so the same assertion made as admin passes whether or not the permission logic works.

Raise E2E_WORKERS and re-run seed.ts to scale out; keep the config's workers at or below it.

What seed.ts sets up, and why

Step Why
OAuth application for the dev-server redirect_uri ts-client's client_id is Md5(redirect_uri); authorize refuses without a matching uid.
An absolute authority login_url A relative one resolves without the port and dead-ends the login redirect (see "The stack").
A tenant for the backend domain staff-api rejects every /bookings and /events call with "domain does not have a tenant configured" until one exists.
One non-admin user per worker Permission gating needs a non-admin; parallel mutation needs distinct identities.

seed.ts polls for the authority rather than reading it once: /domains is served from Elasticsearch, so on a cold stack the row exists in Postgres before the API can see it. Failing fast there is the single most likely way a CI run breaks, and the error looks nothing like the cause.

It bootstraps through the backoffice application, which init always creates — otherwise registering an OAuth app would require a token that requires an OAuth app.

Which routes call Microsoft 365

staff-api only dereferences tenant credentials when it instantiates a PlaceCalendar client, which happens on the calendar-backed routes and nowhere else:

GET /bookings?type=desk|locker|parking|visitor|room   -> PlaceOS only
GET /calendars, GET /events, GET /people              -> the sandbox tenant

Room specs that set app.events.use_bookings = true save an ordinary room booking through /bookings (see ROOM_E2E_HANDOVER.md), so they exercise the PlaceOS side of rooms, not the calendar path.

CI

.github/workflows/e2e-advisory.yml — runs hourly at :40 when develop has a new commit since the previous run, nightly at 01:10 UTC on every day, and via Run workflow on any branch. There is deliberately no pull_request trigger (see the table below — it is a security decision, not an oversight), so the hourly run is the thing that covers a merge. It brings up the isolated stack, seeds it, runs the suite, writes a step summary, and uploads the HTML report plus (on failure) traces, videos and backend container logs.

There are no path filters, on purpose — most real changes touch libs/**, which workplace depends on, and the genuinely dangerous ones (bun.lock, tsconfig.base.json, config/) are the easiest to leave off an include-list. A filter that is 95% right silently skips the run that mattered.

There is deliberately no push trigger: there is one self-hosted runner binding fixed host ports, so runs serialise, and a run per push built a queue on busy days. The hourly check (job changes) runs on a GitHub-hosted runner, so an hour with no new commit costs the self-hosted runner nothing.

It is advisory and must stay that way for now — do not add it to branch protection or required status checks until the suite has a track record. See the CI section of ../E2E_USER_STORIES.md for the bar it needs to clear first.

It runs on a self-hosted macOS runner (label placeos-e2e). Setup, the network requirements, and the macOS-specific gotchas are in stack/SELF_HOSTED_RUNNER.md.

CI-specific choices worth knowing:

E2E_WORKERS: 6 Higher than the local default of 4. The runner also hosts ~10 containers, a dev server and the browsers. Before the specs paused the animated page background (support/background.ts), four workers used all the CPU and six were no faster. Now six workers run both test steps in about 19 minutes, against 23 minutes with four. Seeding reads the same value, so set it once for the job.
E2E_WEBSERVER_TIMEOUT_MS: '900000' (settings-browser job only) The config's webServer timeout, 180000 ms unless set. The GitHub-hosted runner starts nx serve workplace cold with no Nx cache, which can take longer than the default.
No pull_request trigger This repo is public and the runner is self-hosted, so a fork PR could run arbitrary code on an internal machine. Triggers are limited to ones requiring write access. Do not add it back without moving to ubuntu-latest.
Reclaim step A self-hosted machine is not a fresh VM; a previous aborted run can leave the stack up or port 4214 held.
No vm.max_map_count bump Needed on GitHub-hosted Linux, meaningless on macOS — the value lives inside Colima's VM, which already sets it to 1048576. Restore it if reverting to ubuntu-latest.
Network access to GitHub up.sh clones PlaceOS/www-core into the www volume once — that is where the platform /login page comes from.
Network access to the build farm core fetches driver binaries from build.placeos.run and the S3 bucket it hands out links to. A commit the farm has not built for the runner's architecture is compiled there on the first run that asks. The binaries stay in the placeos-e2e-core-drivers volume between runs, so later runs do not ask again.
Network access to Docker Hub, every run up.sh --pull-only runs first as its own step, bounded to 20 minutes and advisory, so latest tracks the registry rather than the day the runner was set up. Each image pulls on its own; one that fails or stalls leaves the stack on the image already present, and "Record backend inputs" shows which. SELF_HOSTED_RUNNER.md lists the hosts.

Booking specs: what the backend actually does

Learned the hard way while building the desk specs. Each of these produced a failure that looked like something else entirely.

  • GET /bookings is scoped to the CALLER. Listing as an admin does not show other users' bookings, so "I checked for leaks as admin and found none" proves nothing. Verify against the database, or list as the owning user. Deleting is caller-scoped too.
  • The list query parameter is type, not booking_type. The latter is the model field and appears in the controller's PARAMS, which makes the source misleading; the request 422s with missing required parameter 'type'.
  • Desks come from Zone METADATA, not systems — metadata.desks.details[] on a level zone. groups: [] leaves them unrestricted; a populated groups gates them behind membership.
  • A desk is exclusive for its time range. Each worker books its own desk, and each spec sweeps that desk before booking (releaseAsset). A post-test finally alone is not enough: if a run dies mid-flight the leftover holds the desk, and because these are all-day bookings it holds it for the rest of the day. Every later run then fails as "desk not offered" or a 422 with an empty failures array — nothing that points at the real cause.
  • POST /bookings occasionally 500s with DB::ConnectionLost under concurrent writes — roughly 1 run in 8 at 4 workers, on stock connection settings (which PlaceOS/local ships too). This is a staff-api defect, not a rig artifact; sizing the connection pool in compose was tried and did not demonstrably help. CI's retries: 2 absorbs it and reports the test as flaky, which is the right outcome: the gate stays usable and the instability stays visible. Worth fixing in staff-api on its own merits.
  • Calendar routes reach the internet. /calendars, /events and /people call login.microsoftonline.com and Graph for the sandbox tenant, so a run needs those hosts to be reachable.

Conventions

  • One spec owns its own data. Create it, assert it, delete it in a finally. Use uniqueTitle() so concurrent workers cannot collide.
  • Sweep before you create, not just after. Post-test cleanup handles the happy path; a pre-test sweep is what lets a spec recover from a previous crashed run.
  • Never blindly toggle a control — ensure the state you want. Toggling a checkbox that happened to start in the target state silently inverts your intent, and the flow then dead-ends in a way that reads as a selector bug.
  • Assert through the API, not only UI text. A green screen reflecting stale state is worse than a red test.
  • Every bugfix PR adds a regression spec carrying the ticket ID, and a REG-* row in E2E_USER_STORIES.md. A regression spec that was never seen red is a guess, not a guard.
  • Red-check new assertions. Break the expected value once and confirm the test fails for the reason you think. An assertion that never observed real data is worse than no assertion.
  • "Nothing is shown" needs a control. An empty state is also what a page shows before its data loads. Hold a second item that the page can only show when it has loaded, and assert on that.
  • A workaround for an app defect goes in the report. A helper that retries past a known defect adds an annotation to the test each time the retry fires: test.info().annotations.push({ type: 'NEW-F5', description }). The type is the id of the finding in E2E_USER_STORIES.md. A console.warn alone is not read on a green run, and the defect can then get worse with no sign.
  • Do not make a booking of TODAY for the user of another worker. The home page shows the first booking a user has today, so the home specs delete every booking their user has today (clearToday). Book a later day.
  • A context you create yourself calls pauseBackground(context) before its first page. The context fixture does it for you. The animated page background costs headless Chromium about 5 CPU-s per page load, and on the CPU-bound CI runner that is most of the run time.
  • The app can still have requests in flight when a test ends. A route handler that calls route.fetch(), or a response body kept for later, then fails with a closed-target error. Wrap such a handler in ignoreClosed, and read a kept body with jsonBody(response). The settings audits drop closed-target errors that are recorded after their teardown starts.
  • .auth/ and .env are gitignored — those tokens are impersonation-grade.