Launch Stealth Bench V2: 100 domains, 60 seconds, 12 browser configurations

[MagMueller]

Replace Stealth Bench V1 with a reproducible 100-domain, 60-second protected-site access challenge. A deterministic Playwright probe opens each URL; a separate Luna judge evaluates page text and screenshots at 20 and 60 seconds. There is no executor LLM or self-reported success. The same frozen cohort was evaluated twice across 12 browser configurations, retaining all 2,400 scheduled attempts.

Configuration	Repeat 1	Repeat 2	Confirmed access	Unresolved / 200
Browser Use Cloud + solver	89%	92%	90.5%	0
Browser Use Cloud, solver off	87%	81%	84%	0
Anchor	85%	79%	82%	0
Kernel	79%	82%	80.5%	0
Browserbase standard	80%	77%	78.5%	1
Browserless	77%	78%	77.5%	17
Steel	74%	75%	74.5%	1
Hyperbrowser	73%	74%	73.5%	2
Chromium headful + US proxy	15%	13%	14%	0
Chromium headful	10%	10%	10%	0
Chromium headless	3%	3%	3%	0
Chromium headless + US proxy	2%	2%	2%	0

The headline denominator is all 200 attempts per configuration. Unresolved outcomes have null per-task scores and are never relabeled bot blocks. Browserless's possible-access upper bound is 86%; chart hatching shows recorded unresolved outcomes. Original observations, 97 one-time missing-measurement retries, effective outcomes, source hashes, requested settings, privacy checks and separate judge-sensitivity bounds are included.

Adds a pinned standalone runner, exact task/judge documentation and light/dark PNG/SVG charts. The encrypted dataset contains 100 distinct starting registered domains: 97 homepages and three generic public routes, with no queries, fragments, credentials or customer identifiers. Raw captures, customer complaints, private traces, session IDs and egress IPs remain private. Removes Stealth V1's dataset, results, plots and legacy CLI selection; BU Bench V1 and V2 remain available.

This cohort was historically selected for Cloud versus headful-plus-proxy differences, then frozen before this provider comparison. It is a regression challenge, not representative web-wide access or a vendor-balanced ranking. Browser versions, network exits, native solvers and the Cloud content viewport differ. Browserbase Verified requires an Enterprise plan and has no score. The methodology documents the corrected Browserless configuration and acquisition retry history.

A visual audit covered 61 distinct attempts, including a fixed random matched sample of 40 captures. It identified blank-page and age-overlay judgment ambiguities. Original judgments are preserved; the separate sensitivity analysis places Cloud at 88.5–90.5% and headful-plus-proxy at 13.5–16% for the identified cases. These bounds are not confidence intervals or a full human relabeling.

Validation: 78 unittest cases (three optional skips), Ruff, locked runner help, chart aggregate validation/rendering and light/dark visual review, exact dataset/task coverage, original/retry lineage, source/lock hashes and public-field privacy checks. The private companion implementation is in new-eval-platform PR #33. No merge or deployment is part of this PR.

---

Summary by cubic

Replaces Stealth Bench V1 with V2, a reproducible 100-domain, 60-second protected-site access challenge. A deterministic Playwright probe opens each URL; an identity-blinded Luna judge scores page text and screenshots at 20 and 60 seconds, replacing the executor LLM and self-reported success.

New Features

• Publishes 2,400 scheduled attempts (100 domains × 2 repetitions × 12 configurations) plus a fresh 700-attempt seven-provider confirmation, with provenance, retry lineage, judge-sensitivity bounds, and light/dark PNG/SVG charts.
• Adds a clearly marked equal-scale broken-axis main chart that omits 20–65% while retaining all configurations; the full 0–100% view stays available.
• Cloud with solver leads at 90.5% pooled; stock headless Chromium gets 3%. Unresolved attempts keep null scores and are never relabeled as bot blocks; Browserless's possible-access upper bound is 86%.

Refactors

• Removes the V1 dataset, results, plots, and --benchmark Stealth_Bench_V1; BU Bench V1 and V2 keep their paths.
• Access judging now needs OPENAI_API_KEY; the V1 Gemini binary judge is gone.
• CI validates and renders published V2 results and lints the new modules; targeted ruff ignores keep probe/judge snapshots byte-identical for provenance.

The cohort was historically selected for Cloud versus headful-plus-proxy differences and is a regression challenge, not a representative or vendor-balanced benchmark. The private companion implementation ships separately in new-eval-platform PR #33.

^{Written for commit 74b4271. Summary will update on new commits.}

Independent provider confirmation

All seven managed providers were rerun on the same frozen 100 tasks: 700 additional original attempts and nine one-time missing-measurement retries. Browserbase uses standard mode; no Enterprise feature or plan upgrade was used. Each provider used concurrency five and the unchanged 60-second access-v3 protocol, with task-order seed 20260930. Original two-repeat results are preserved byte-for-byte.

Provider	Original two-run average	Confirmation	Unresolved / 100
Browser Use Cloud	90.5%	90%	1
Anchor	82%	85%	0
Kernel	80.5%	83%	0
Browserless	77.5%	83%	3
Browserbase standard	78.5%	78%	0
Steel	74.5%	74%	0
Hyperbrowser	73.5%	72%	1

The new outcomes, originals, retries, provenance, privacy/integrity audit and separate sensitivity bounds are in stealth_bench/official_results/confirmation-20260925. All 100 tasks remain in each denominator. Possible-access upper bounds are Cloud 91%, Hyperbrowser 73%, and Browserless 86%. A fixed sample of 21 screenshots was reviewed: 19 supported the recorded classifications; blank-page and age-overlay ambiguities are documented without changing verdicts. The sensitivity check applies across all providers and is not a confidence interval.

Added exact 700-attempt coverage/retry-lineage checks and CI rendering for the new chart. Current public validation: 78 unittest cases (three optional local skips), Ruff, chart aggregate checks, source/data/privacy audits and desktop/mobile report QA. This remains a selected-cohort product comparison, with differences in browser/network/solver profiles and concurrency documented. No merge or deployment.

The launch chart now includes all Chromium controls with a visibly marked 20–65% axis break; both visible segments use the same scale, and the full 0–100% chart remains linked. Original scores and charts are unchanged. The new renderer passed the six Stealth tests, aggregate checks, light/dark visual review and hosted CI.