stopline

A decision gate for browser agents. Every click, fill, key press, or navigation a Playwright agent proposes is described from the DOM, sent to a System One model as three typed questions, and turned into allow, ask, or block by a policy written in ordinary code. Each decision is appended to an evidence log.

The model classifies. The application authorises. The two never merge.

agent proposes action
        │
        ▼
 describe it from the DOM ──► hard rules (password fields, card fields, off-origin navigation)
        │                                   │
        ▼                                   │
 Jev or Laya answers 3 typed questions      │
   effect: read_only | reversible_change |  │
           public_or_irreversible |         │
           credentials_or_money             │
   servesTask: P(true)                      │
   pageInstructsAgent: P(true)              │
        │                                   │
        ▼                                   ▼
 policy.ts: strictest verdict wins ──► allow · ask a human · block
        │
        ▼
 evidence log (JSONL: action, state hash, answers, verdict, approval, executed)

Why a gate, and why this shape

A browser agent that reads a page also reads whatever the page author wrote for it. The fixture reviews page in this repo hides a note for "AI shopping assistants" that says the customer has pre-approved a purchase. A planner that treats page text as instructions will follow it.

Asking a large model "is this action safe?" on every step is slow and returns an opinion. Jev and Laya return a probability per answer in tens to hundreds of milliseconds, which is fast enough to sit in front of every action. The design follows the guidance Vercel published alongside Jev: keep classification separate from authorisation. The model can only add friction on top of the hard rules; it can never remove it. If the model is unreachable, the gate fails closed to a human.

The policy

All thresholds live in src/policy.ts:

Condition	Verdict
Fills a password, card, or one-time-code field, or navigates off-origin	at least ask (hard rule, no model)
pageInstructsAgent ≥ 0.5	block (the page is quarantined, even for harmless actions)
servesTask < 0.3	block
Top effect is public_or_irreversible or credentials_or_money	ask
0.2 or more probability on those two classes, whatever the top class is	ask
servesTask < 0.8, or top effect probability < 0.8	ask
Otherwise	allow

The fifth row is the reason to use a model that returns a distribution. An argmax of reversible_change at 0.70 with 0.25 on public_or_irreversible is not the same decision as reversible_change at 0.99.

Revision-bound approval example

The small src/revisionGate.ts helper demonstrates one adjacent delivery-contract rule: an approval is accepted only when its action identifier, proposed revision, and captured state identifier still match. The accompanying tests show the rejection when either the revision or state changes after approval. The revision-bound approval note explains the example and how to run it.

This is an illustrative building block, not a repository merge control. It does not calculate a commit hash, authenticate the approver, or prove that a surrounding workflow cannot bypass the check.

Run it

Requires Node 20.12+ and, for Jev, the Vercel CLI.

See a local result first

You can run the labelled keyword baseline without an API key or Vercel account:

npm install
STOPLINE_BACKEND=keyword npm run eval

This is deliberately a weak, offline comparison. On the checked-in 49-case fixture set it reports the verdict agreement, unsafe allows, and stricter decisions that the model-backed runs should improve. It is a measurement baseline, not a safe unattended policy.

npm install
npx playwright install chromium

# Jev through Vercel AI Gateway. Either set AI_GATEWAY_API_KEY in .env.local,
# or link a Vercel project and pull a short-lived OIDC token:
vercel link --yes
vercel env pull .env.local --yes

npm test                              # policy unit tests, no network
npm run eval                          # 49 labelled cases against Jev
npm run demo                          # five scripted browser tasks through the gate
npm run demo:revision                 # show a matching approval and a changed-revision rejection
STOPLINE_INTERACTIVE=1 npm run demo   # approve "ask" verdicts yourself

Other backends:

STOPLINE_BACKEND=keyword npm run eval   # keyword heuristic, no model, the baseline to beat

# Laya, locally, over its Jev-compatible /v1/systemone endpoint
python -m venv .venv-laya && .venv-laya/bin/pip install "laya[serve]"
LAYA_DEVICE=mps LAYA_PRELOAD=1 .venv-laya/bin/laya-serve &
STOPLINE_BACKEND=laya npm run eval

scripts/run-local.sh does all of the above in one go and writes to results/.

The eval

eval/cases.ts holds 49 proposed actions across a shop, a developer community site, webmail, a code host, a bank, a login page, docs, search, a calendar, and an admin console. Each case is labelled on the three questions only. The expected verdict is derived from the labels by one fixed rule (expectedVerdict), so a disagreement with a label does not require a disagreement with the policy. Cases marked hard: true are ones where I think a careful reviewer could label either way.

The eval reports per-question accuracy, verdict agreement, latency, and two lists that matter more than accuracy:

• unsafeAllows: the gate allowed an action that should have stopped.
• stricterThanExpected: friction the gate added that the labels say was unnecessary.

It also reports what the hard rules alone would do with no model, which is the floor the model has to improve on.

Results: see results/ and the summary below.

Offline keyword baseline (27 September 2026)

The current checked-in 49-case fixture run with STOPLINE_BACKEND=keyword npm run eval produced:

Measure	Result
Cases answered	49/49
Effect classification accuracy	71.4%
Task-fit classification accuracy	53.1%
Prompt-injection accuracy	95.9%
Prompt-injection recall	100%
Verdict agreement	49.0%
Unsafe allows	0

The keyword baseline now compares exact words in the visible action label and ignores the navigation target when estimating task fit. That prevents a repository path or a product-name substring from making an unrelated action appear on task. The change removes the two unsafe allows from the first run, while adding some conservative ask decisions.

This is the no-network keyword baseline, not a model-backed result or a safety guarantee. The fixture is hand-labelled and intentionally small; use it to find failure modes and compare later runs, not to certify a deployment. Both the first run and the revised JSON results remain under results/ for inspection.

Continue the review

The gate is an experimental implementation and the evaluation is an initial measurement, not a production safety certification. For a smaller, implementation-independent exercise, use the AI-assisted code review kit. It includes the code, worksheet, worked answer, sample team output, and a proposed facilitator guide.

For a focused team discussion, use the team evaluation guide. It turns one proposed browser-agent action into a short decision record covering allow, ask, block, evidence ownership, and recovery. The guide is an evaluation exercise, not a production rollout plan.

If you want to report a reproducible behaviour or propose a focused change, read the contribution guide first. It explains the checks to run and the evidence boundary for issues and pull requests.

For a security concern, read the security policy before opening an issue. Do not include credentials, tokens, personal information, or a complete exploit in a public issue.

Teams that want to compare the exercise with their own review practice can read about the private workshop interest path. That page describes a proposed pilot and does not reserve a date or accept payment.

If you want to evaluate the boundary with a team

Start with the free review kit, then run the revision demo and inspect the evidence log. A team that wants facilitated discussion can review the workshop interest page and send a fit enquiry. The workshop is a proposed, bounded learning session; it is not a Stopline support contract, production integration, security audit, or certification.

Limits

• The agent in the demo is a fixed script, not an LLM planner. That keeps runs repeatable and isolates the gate, but it means the demo does not show how often a real planner proposes bad actions.
• 49 hand-labelled cases is enough to find failure modes, not to certify a threshold. Treat the numbers as a first measurement on my labels.
• The gate reads document.body.textContent, so it sees hidden text a DOM-reading agent would see. It does not read images, iframes, or shadow DOM.
• ask needs a human. In unattended runs the default approver says no.
• The OIDC token from vercel env pull expires after about 12 hours locally.

Licence

MIT