Skip to content

README.md

Latest commit

 

History

History
378 lines (269 loc) · 15.2 KB

README.md

File metadata and controls

378 lines (269 loc) · 15.2 KB
Raw

Ghost End-To-End Test Suite

This top-level workspace is Ghost's browser end-to-end test suite. It runs automated browser tests against a complete, running Ghost instance to verify critical user journeys across packages and applications. A package's own Playwright suite is an acceptance suite, not an E2E one — see the testing guide for how the layers differ.

Quick Start

Prerequisites

  • Docker and Docker Compose installed
  • Node.js installed (pnpm is managed via corepack — run corepack enable pnpm first)

Running Tests

To run the test, within this e2e folder run:

# Install dependencies
pnpm

# All tests
pnpm test

Dev Environment Mode (Recommended for Development)

If GHOST_E2E_MODE is unset, the e2e shell entrypoints auto-select:

  • dev when the local admin dev server is reachable on http://127.0.0.1:5174
  • build otherwise

To use dev mode, start pnpm dev before running tests:

# Terminal 1: Start dev environment (from repository root)
pnpm dev

# Terminal 2: Run e2e tests (from e2e folder)
pnpm test

If infra is already running, pnpm infra:up is safe to run again. For dev-mode test runs, infra:up also ensures required local Ghost/gateway dev images exist. If you want to force a mode, set GHOST_E2E_MODE=dev or GHOST_E2E_MODE=build explicitly.

Analytics Development Flow

When working on analytics locally, use:

# Terminal 1 (repo root)
pnpm dev:analytics

# Terminal 2
pnpm test:analytics

E2E test scripts automatically sync Tinybird tokens when Tinybird is running.

Build Mode (Prebuilt Image)

Use build mode when you don’t want to run dev servers. It uses a prebuilt Ghost image and serves public assets from /content/files.

# From repository root
pnpm build
pnpm --filter @tryghost/e2e build:apps
GHOST_E2E_BASE_IMAGE=<ghost-image> pnpm --filter @tryghost/e2e build:docker
GHOST_E2E_MODE=build pnpm --filter @tryghost/e2e infra:up

# Run tests
GHOST_E2E_MODE=build GHOST_E2E_IMAGE=ghost-e2e:local pnpm --filter @tryghost/e2e test

Build-mode E2E infra uses tmpfs-backed MySQL storage by default so database snapshot restore cycles stay fast and isolated from local development data. Set GHOST_E2E_MYSQL_TMPFS=false to use the normal Docker volume instead, or GHOST_E2E_MYSQL_TMPFS_SIZE=4g to adjust the tmpfs size.

Set GHOST_E2E_TINYBIRD_SLIM=true to swap the Tinybird service for the distilled slim image (ghcr.io/tryghost/tinybird-local-slim): ~0.7GB pulled and ~2.4GB on disk, against ~2.1GB and ~6.9GB for upstream. CI enables it so the analytics jobs fit inside the runner disk budget. Override the image/tag with GHOST_E2E_TINYBIRD_SLIM_IMAGE. Local dev (compose.dev.analytics.yaml) always uses the upstream image.

The slim image's GHCR package is internal, so a pull can legitimately fail — most often on a PR from a public fork, whose token cannot read it. infra-up.sh warns and falls back to the upstream image rather than failing the run. CI leaves the flag off for cross-repo PRs so those runs skip the doomed pull entirely.

For a CI-like local preflight (pulls Playwright + gateway images and starts infra), run:

pnpm --filter @tryghost/e2e preflight:build

Running Specific Tests

# Run the Admin sign-in test
pnpm test tests/admin/signin.test.ts

# Matching a pattern
pnpm test --grep "homepage"

# With browser visible (for debugging)
pnpm test --debug

Tests Development

See Writing Browser E2E Tests for the canonical conventions, Page Object pattern, locator priority, waiting patterns, and worked examples. This README covers the local workspace, infrastructure, fixtures, and commands.

The test suite is organized into separate directories for different areas/functions:

Current Test Suites

  • tests/public/ - Public-facing site tests (homepage, posts, etc.)
  • tests/admin/ - Ghost admin panel tests (login, content creation, settings)
  • tests/portal/ - Portal member journey tests

We can decide whether to add additional sub-folders as we add more tests.

Project folder structure can be seen below:

e2e/
├── tests/                      # All the tests
│   ├── public/                 # Public site tests
│   │   └── member-signup.test.ts
│   ├── admin/                  # Admin site tests
│   │   └── signin.test.ts
│   ├── portal/                 # Portal tests
│   ├── global.setup.ts         # Global setup script
│   └── global.teardown.ts      # Global teardown script
├── helpers/                    # All helpers that support the tests, utilities, fixtures, page objects etc.
│   ├── playwright/             # Playwright specific helpers
│   │   └── fixture.ts          # Playwright fixtures
│   ├── pages/                  # Page Object Models, grouped by area
│   │   ├── base-page.ts        # Base class for all page objects
│   │   └── admin/              # e.g. login-page.ts, admin-page.ts
│   ├── environment/            # Ghost container/database lifecycle
│   ├── services/               # Test doubles (fake Stripe, Mailgun, etc.)
│   └── utils/                  # Shared utilities
├── data-factory/               # Test data factories (see its own README)
├── visual-regression/          # Screenshot baseline suite (separate config)
├── scripts/                    # Infra and runner shell scripts
├── playwright.config.mjs       # Playwright configuration
├── eslint.config.js            # Lint config for the workspace
├── package.json                # Dependencies and scripts
└── tsconfig.json               # TypeScript configuration and path aliases

Global Setup and Teardown

Tests use Project Dependencies to define special tests as global setup and teardown tests:

  • Global Setup: tests/global.setup.ts - runs once before all tests
  • Global Teardown: tests/global.teardown.ts - runs once after all tests

Playwright Fixtures

Playwright Fixtures are defined in helpers/playwright/fixture.ts and provide reusable test setup/teardown logic.

The fixtures a test usually reaches for:

  • page - browser page against this test's Ghost instance
  • pageWithAuthenticatedUser - the same, already signed in to Ghost Admin
  • ghostAccountOwner - the owner account's credentials
  • ghostInstance - the running instance: baseUrl, database, port, siteUuid, containerId, instanceId
  • resolvedIsolation - 'per-file' | 'per-test' for the current test
  • resetEnvironment() - force an environment recycle (see the escape hatch below)

The fixture resolves isolation mode per test file:

  • Default: per-file isolation (one Ghost environment cycle per file)
  • Opt-in per-test: call usePerTestIsolation() from @/helpers/playwright/isolation at the root of the file
  • Forced per-test: any run with fullyParallel: true

Test Isolation

Test isolation is still automatic, but no longer always per-test.

Infrastructure (MySQL, Redis, Mailpit, Tinybird) must already be running before tests start. Use pnpm dev or pnpm --filter @tryghost/e2e infra:up.

Global setup (tests/global.setup.ts) does:

  • Cleans up e2e containers and test databases
  • Creates a base database, starts Ghost, waits for health, snapshots the DB

Per-file mode (helpers/playwright/fixture.ts) does:

  • Clones a new database from snapshot at file boundary
  • Restarts Ghost with the new database and waits for readiness
  • Reuses that environment for tests in the file

Per-test mode (helpers/playwright/fixture.ts) does:

  • Clones a new database from snapshot for each test
  • Restarts Ghost with the new database and waits for readiness

Environment identity for per-file reuse:

  • config participates in the environment identity.
  • labs participates in the environment identity.
  • If either changes between tests in the same file, the shared per-file Ghost environment is recycled before reuse.
  • stripeEnabled does not participate in per-file reuse. It always forces per-test isolation because Ghost must boot against a per-test fake Stripe server.

Fixture option behavior:

  • config: use for boot-time Ghost config that should get a fresh environment when it changes.
  • labs: use for labs flags that should get a fresh environment when they change.
  • stripeEnabled: use for Stripe-backed tests; this always runs each test with a fully isolated Ghost environment.

Escape hatch:

  • resetEnvironment() is supported only in beforeEach hooks for per-file tests.
  • Use it only before resolving stateful fixtures such as baseURL, page, pageWithAuthenticatedUser, or ghostAccountOwner.
  • Safe hook pattern: test.beforeEach(async ({resetEnvironment}) => { ... })
  • Unsupported pattern: calling resetEnvironment() after page or an authenticated session has already been created.
  • ESLint catches the obvious misuse cases, but the runtime guard in the fixture remains the hard safety check.

Opting into per-test isolation:

  • Use usePerTestIsolation() from @/helpers/playwright/isolation at the root of the file.
  • This configures both Playwright parallel mode and the fixture isolation in one call.

Global teardown (tests/global.teardown.ts) does:

  • Cleans up e2e containers and test databases (infra services stay running)

Modes:

  • Dev mode: Ghost mounts source code and proxies assets to host dev servers
  • Build mode: Ghost uses a prebuilt image and serves assets from /content/files

Best Practices

  1. Use Page Objects for reusable UI structure and interactions. Direct semantic locators are fine for small, one-off assertions where a Page Object would not improve reuse or readability.
  2. Add meaningful assertions beyond just page loads. Keep assertions in tests.
  3. Prefer semantic locators, such as getByRole() and getByLabel(). Use stable test IDs when semantic locators are unavailable. Avoid selectors coupled to styling or DOM position.
  4. Clean up test data when tests modify Ghost state
  5. Group related tests in describe blocks
  6. Do not use should to describe test scenarios

CI Integration

Tests run automatically in GitHub Actions on every PR and commit to main.

CI Process

  1. Setup: Ubuntu runner with Node.js and Docker
  2. Build Assets: Build server/admin assets and public app UMD bundles
  3. Build E2E Image: pnpm --filter @tryghost/e2e build:docker (layers public apps into /content/files)
  4. Prepare E2E Runtime: Pull Playwright/gateway images in parallel, start infra, and sync Tinybird state (pnpm --filter @tryghost/e2e preflight:build)
  5. Test Execution: Run Playwright E2E tests inside the official Playwright container
  6. Artifacts: Upload Playwright traces and reports on failure

Available Scripts

Within the e2e directory:

# Run all tests
pnpm test

# Start/stop test infra (MySQL/Redis/Mailpit/Tinybird)
pnpm infra:up
pnpm infra:down

# CI-like preflight for build mode (pulls images + starts infra)
pnpm preflight:build

# Debug failed tests (keeps containers)
PRESERVE_ENV=true pnpm test

# Check the fake Stripe server against captured Stripe responses (no infra, ~1s)
pnpm test:fixtures

# Put a Stripe test account into the state fixtures are captured from
pnpm stripe:provision

# Re-capture Stripe fixtures from test mode (needs STRIPE_SECRET_KEY)
pnpm stripe:fixtures

# Re-measure the checkout limits the fake server enforces
pnpm stripe:probe

# Run TypeScript type checking
pnpm test:types

# Lint code and tests
pnpm lint

# Build (for utilities)
pnpm build
pnpm dev           # Watch mode for TypeScript compilation

Stripe fixtures

The fake Stripe server in helpers/services/stripe/ hand-builds the objects Stripe would return. Those shapes were originally written from the docs rather than from Stripe, so nothing checked them against the real API.

helpers/services/stripe/fixtures/ holds responses captured from Stripe test mode at API version 2020-08-27, the version ghost/core pins. pnpm test:fixtures asserts the builders against them, and needs no Ghost, no Docker and no browser.

Two failures are worth catching. A builder emitting a key Stripe does not return means the fake describes an API that does not exist. A builder omitting a key Ghost reads is worse, because it is silent: the property access yields undefined, the branch behind it never runs, and the suite stays green.

The same suite checks that the fake refuses requests Stripe refuses. Those constraints were measured, not read from the docs, because the docs and Stripe's published OpenAPI spec each disagree with the API on at least one of them.

Re-capturing

STRIPE_SECRET_KEY=sk_test_... pnpm stripe:provision   # once per account
STRIPE_SECRET_KEY=sk_test_... pnpm stripe:fixtures

Test-mode keys only; a live key is refused.

stripe:provision puts the account into the state fixtures are captured from: a tier product, its Monthly, Yearly and Complimentary prices, and a coupon. The nicknames are the point, because Ghost's own code matches on them. Capturing runs it first, so the two commands are only separate when you want to inspect what an account holds.

Both are idempotent and reuse what is already there, since Stripe cannot delete a product that has prices.

Capturing a completed checkout

A completed checkout cannot be captured with the above, because Stripe blocks automating its hosted payment page. pnpm stripe:fixtures:checkout does everything either side of the payment and asks for one card entry:

STRIPE_SECRET_KEY=sk_test_... pnpm stripe:fixtures:checkout

It prints a Checkout URL, waits, then captures the completed session once you have paid with 4242 4242 4242 4242. Shipping address, tax ID and custom field collection are all requested on the same session, so one payment captures every shape the API cannot give us. Fill all of them in, or the fields come back null and the fixture answers nothing.

The event envelope is deliberately not captured. An Event is an immutable snapshot rendered at the account's default API version when it was created, and fetching it with a pinned client does not re-render it. Ghost pins its webhook endpoint to the same version its client uses, so what Ghost receives and what the API can hand back are different renderings: at Stripe's current default the shipping address moves to collected_information.shipping_details, which Ghost never sees. Ghost reads only event.type and event.data.object, so the envelope carries nothing worth pinning.

The same difference applies to stripe listen, which pnpm dev:stripe --listen uses: it renders events at the account default too. The default pnpm dev:stripe lets Ghost register its own pinned endpoint, so it receives the payloads production receives (see Development setup).

Resolving issues

Test Failures

  1. Screenshots: Playwright captures screenshots on failure
  2. Traces: Available in test-results/ directory
  3. Debug Mode: Run with pnpm test --debug or pnpm test --ui to see browser
  4. Verbose Logging: Check CI logs for detailed error information