Agent Browser Validation

A config-driven browser automation harness that validates an agent-facing web application from two directions simultaneously: backend API contracts and real browser user journeys.

---

Objectives

This system is built to answer two questions on every run:

1. What are the flows? — What user journeys and API contracts does the application support, and are they all passing?
2. How long does a flow take? — Is the application responding within acceptable time bounds, or is it slow even when it appears functional?

Both questions must be answered together. A flow that passes but exceeds timing thresholds indicates a degraded environment. A fast environment where flows fail indicates a broken one. Only when all covered flows pass and all timing bounds are met is the application considered healthy.

---

Concept Of Operations

The harness is built around the two objectives above. On every run it must produce:

• a complete inventory of flows — which passed, which failed, and at which step
• a timing record for every flow and API check — against defined target and maximum bounds

To answer what are the flows, the config encodes each API contract and each user journey as a named, ordered sequence of assertions. The suite runs all of them and reports pass/fail per flow, per step. The config is the authoritative list of what the system is expected to do.

To answer how long does a flow take, every check is timed and compared against thresholds. A flow that passes but breaches its timing bound is not a clean pass — it is a degraded result.

Operating model:

1. Define the target environment in config/validation.json.
2. Encode every required backend contract in apiChecks.
3. Encode every required user journey in flows.
4. Run the suite — locally, pre-release, or in CI.
5. Read the output: flow inventory first, timing second, then triage any failures.

Three intended operating modes:

Mode	Purpose
Local	Build and iterate on flows against a dev or staging URL
Pre-release	Regression gate before promoting a build
CI smoke	Continuous health check against a known deployed environment

The code stays stable. The environment URL, API assertions, selectors, credentials, and user journeys are updated in config as the application evolves.

---

Output And Success Criteria

Each run produces a quantitative result set:

• total API checks executed
• total browser flows executed
• pass count and fail count
• pass rate
• failure location down to the individual flow step
• failure artifacts: screenshot, trace, and video when a browser step fails

The primary output is the Playwright test summary printed at the end of each run.

Release thresholds

Metric	Threshold
API pass rate	100%
Critical flow pass rate	100%
Non-critical flow pass rate	>= 95%
Unhandled browser errors	0
Broken selectors in critical flows	0
Consecutive clean CI runs before release	3

Runtime thresholds

Metric	Target	Maximum
Single API check	<= 2s	<= 5s
Single browser flow	<= 30s	<= 60s
Full smoke suite	<= 10 min	—

Decision rules

• Healthy — all critical API checks pass, all critical browser flows pass, no timing threshold exceeds its maximum.
• Degraded — pass rate is high but one or more checks exceed runtime thresholds or non-critical flows fail.
• Not release-ready — any critical check fails, or failures are flaky across repeated runs.

Interpreting results

• 100% on critical checks means the covered backend contracts and covered user journeys are functioning.
• Anything below 100% on critical coverage means there is a verified regression in the tested scope.
• Classify every check as critical or non-critical in your operating process and require at least 3 consecutive clean CI runs before calling the environment stable.

To extend quantitative confidence beyond defaults, add: latency assertions, retry-rate tracking, visual snapshot comparisons, larger flow coverage by feature area, or scheduled repeated runs to measure flake rate over time.

---

Stack

• Node.js
• Playwright
• JSON-based validation spec

---

Project Layout

.
|-- README.md
|-- package.json
|-- playwright.config.js
|-- config/
|   `-- validation.example.json
|-- src/
|   |-- loadValidationConfig.js
|   `-- runFlowStep.js
`-- tests/
    `-- validate.spec.js

---

Setup

Install dependencies:

npm install

Install Playwright browsers:

npx playwright install

Create the runtime config:

cp config/validation.example.json config/validation.json

Edit config/validation.json with real values:

• baseUrl — the environment under test
• apiChecks — HTTP endpoints that must return the expected status and JSON payload
• flows — browser journeys to execute end-to-end

---

Config Format

{
  "baseUrl": "https://your-app.example",
  "apiChecks": [
    {
      "name": "Health endpoint",
      "url": "/api/health",
      "method": "GET",
      "expectedStatus": 200,
      "expectJson": [
        {
          "path": "status",
          "equals": "ok"
        }
      ]
    }
  ],
  "flows": [
    {
      "name": "User can log in",
      "steps": [
        { "action": "goto", "url": "/login" },
        { "action": "fill", "selector": "[name='email']", "value": "user@example.com" },
        { "action": "fill", "selector": "[name='password']", "valueFromEnv": "E2E_PASSWORD" },
        { "action": "click", "selector": "button[type='submit']" },
        { "action": "waitForURL", "urlContains": "/dashboard" }
      ]
    }
  ]
}

---

Supported Browser Actions

Each flow step executes in order. Supported action values:

Action	Key parameters
goto	url — resolved against baseUrl if relative
click	selector
fill	selector, value or valueFromEnv
press	selector, key
waitForURL	urlContains
waitForSelector	selector
assertVisible	selector
assertText	selector, text
assertURL	matches (regex string)

---

Running The Validation

# Headless
npm test

# Visible browser
npm run test:headed

# Playwright UI mode
npm run test:ui

# Custom config file
VALIDATION_CONFIG_PATH=config/staging.validation.json npm test

# Inject secrets
E2E_PASSWORD='your-secret' npm test

---

Current Status

The harness is ready. The environment-specific URL, API checks, and user flows still need to be supplied in config/validation.json.