pmanko · Oct 30, 2026
diff --git a/README.md b/README.md
@@ -50,35 +50,29 @@ node install.mjs --target /path/to/host-repo
 
 ## Requirements
 - `evidence-bundle`: `ffmpeg` (webm→mp4) and `zip` on PATH.
-- Local review runner: Node.js 20+ and Git. Acceptance tests also need Python 3. Optional model review: Codex CLI. OpenELIS execution: Lima 2+, virtualization and sufficient build resources.
+- Local review runner: Node.js 20+ and Git. Acceptance tests also need Python 3. Optional model review: authenticated Codex or Claude CLI. Generic check execution: Docker. OpenELIS execution: Lima 2+, virtualization and sufficient build resources.
 
 ## License
 MIT — see [LICENSE](LICENSE).
 
-## Local review harness (v0.4)
+## Local review harness (v0.5)
 
-**Implementation roadmap:** follow the [staged checklist](docs/ogc-1560-review-harness/roadmap.md). Stage 1 is in progress; the runner now distinguishes required evidence from execution settings.
-
-Run a review of an existing PR:
+Review an existing PR without checking it out in an application worktree:
 
 ```sh
-node review/cli.mjs review --pr https://github.com/OWNER/REPO/pull/NUMBER --level L1
+node review/cli.mjs review --pr https://github.com/OWNER/REPO/pull/NUMBER --level L2
 ```
 
-Choose L1 for deterministic checks or L2 for semantic review with explicit intent,
-ownership, consumer and test assessments. L3–L5 require verification capabilities
-that are still pending; they return partial before startup or model spend. Existing
-declared OpenELIS checks remain usable with L1/L2.
+| Level | What it does |
+| --- | --- |
+| L1 | Selected mechanical checks, zero model calls |
+| L2 | Intent, ownership, callers/consumers and test adequacy review |
+| L3 | Focused executed tests, applicable mutation/revert proof and bounded requested experiments |
+| L4 | Running integration, loaded artifact identity, selected persistence/lifecycle scenarios |
+| L5 | Full-stack selected journeys, saved outcomes and rendered comparison with product designs |
 
-The independent development profile defaults to one inexpensive Sol session.
-Override the model explicitly; there is no core Sol/Luna allowlist. Reviewers inspect
-pinned source on demand without a source-byte cap. No automatic model escalation occurs.
+Levels select evidence, not model size. Codex and Claude share restricted source/evidence tools; provider/model, scope, environment and spending settings are independent. The development profile uses inexpensive Sol settings. Missing required evidence stays partial; no automatic model upgrade occurs. Claude image conformance is not yet claimed, so it cannot run L5.
 
-The [runner guide](docs/review-runner.md) includes installation, focused reviews,
-limits, repository configuration and executable examples. Run `npm test` for
-no-credit acceptance tests. The [implementation record](docs/ogc-1560-review-harness/implementation-status.md)
-separates tested capabilities from remaining integrations and quality evaluation.
-The [specification](docs/ogc-1560-review-harness/specification.md) defines the target contract; the checklist tracks implementation. Automatic publication, Claude integration and the model-quality pilot
-are later work.
+Use the [runner guide](docs/review-runner.md) for commands and prerequisites, the [execution guide](docs/ogc-1560-review-harness/execution-guide.md) for check/probe/runtime inputs, and the [evaluation guide](docs/ogc-1560-review-harness/evaluation-guide.md) for repeated baseline comparisons. Run `npm test` and `npm run lint` without model credits.
 
-See the [execution guide](docs/ogc-1560-review-harness/execution-guide.md) for isolated mechanical commands, accepted decisions and re-review history.
+The [checklist](docs/ogc-1560-review-harness/roadmap.md) separates implementation from acceptance. Real OpenELIS runtime/save/readback and rendered comparison have run; formal product sign-off is separate from completed design comparison. Quality labels and held-out adjudication require independent maintainers. The protected manual CI dispatcher is source-only; publication is a separate explicit command. The reviewer never repairs or merges.
diff --git a/docs/ogc-1560-review-harness/automation-boundary.md b/docs/ogc-1560-review-harness/automation-boundary.md
@@ -12,4 +12,4 @@ References validated against current official documentation on 30 October 2026:
 - [Claude CLI](https://code.claude.com/docs/en/cli-reference): restricted tools, explicit MCP configuration, structured output and noninteractive permission policy.
 - [GitHub workflow security](https://docs.github.com/en/actions/reference/security/secure-use): keep untrusted candidate execution away from privileged workflow credentials.
 
-Remaining acceptance is reported in the roadmap: reviewed configuration for each deployment, approved product-design demonstration, independent quality labels/adjudication and broader adversarial evaluation. These are not silently counted as passed by unit tests.
+Remaining acceptance is reported in the roadmap: reviewed configuration for each deployment, independent quality labels/adjudication and broader adversarial evaluation. These are not silently counted as passed by unit tests.
diff --git a/docs/ogc-1560-review-harness/evaluation-guide.md b/docs/ogc-1560-review-harness/evaluation-guide.md
@@ -0,0 +1,26 @@
+# Evaluating review quality
+
+The evaluation runner compares the harness procedure with a plain-review prompt using the same provider/model, source/evidence broker, selected checks, execution environment, time allowance and session count. Trial order alternates to reduce ordering effects. It does not use another model to invent a quality score. Labels and adjudication stay outside the reviewer snapshots and prompts.
+
+1. Prepare a versioned corpus with opaque case IDs, development/held-out splits, requirements and complete before/after file maps. Optional `review` settings supply the same level, checks/probes/scenarios/workflows/designs to both arms. The included seed corpus is source-only: its runtime/design cases inspect verifier/view code, not actual deployment or rendered acceptance. Use collected L4/L5 cases before claiming those evaluation dimensions.
+2. Prepare labels separately: defects per case, author, approval status and independent approver. The shipped labels are **provisional implementation fixtures**, not independent ground truth. Have a maintainer inspect and approve them before held-out execution.
+3. Set provider, model, effort, seconds, repeated trials and acceptance thresholds before freezing. There is no automatic model escalation.
+4. Freeze and run the development split; retain all partial/error outcomes. Improve the procedure using development cases only. Reversion or relabeling requires a new corpus version, not editing a frozen run.
+5. A named independent adjudicator classifies each non-refuted finding as true positive, false positive or non-defect suggestion, identifies matched defect IDs, and records harmful repair proposals and rationale. Review suggestions as well as blocking findings. Fill every run row, including those with no findings; missing adjudication remains unscored.
+6. Score against the frozen labels. The report includes misses, false positives, false-alarm case rate, harmful proposals, completion, repeatability, elapsed time and recorded provider usage. Unknown monetary cost remains unknown. `qualityAccepted` requires independently approved held-out labels, complete adjudication and the frozen thresholds. A passing development run cannot establish it.
+
+```bash
+node review/evaluation/cli.mjs freeze \
+  --corpus examples/evaluation/corpus.json --labels examples/evaluation/labels.json \
+  --settings examples/evaluation/settings.json --out /durable/evaluation-freeze.json
+node review/evaluation/cli.mjs run \
+  --freeze /durable/evaluation-freeze.json \
+  --corpus examples/evaluation/corpus.json --labels examples/evaluation/labels.json \
+  --split development --out /durable/development-run
+node review/evaluation/cli.mjs score \
+  --evaluation /durable/development-run/evaluation.json \
+  --labels examples/evaluation/labels.json \
+  --adjudication /durable/completed-adjudication.json --out /durable/scores.json
+```
+
+Output destinations must be new; previous evidence is never overwritten. Per-run records survive interruption. Each batch retains private corpus and label copies beside the frozen settings, outside the reviewer snapshots. Local CLI invocations run as operator-authorized development work; no reviews are posted. The seed corpus covers approved removal, weak/wrong-layer tests, consumer mismatch, stale-runtime checks, design divergence in view code, valid refactoring and injected instructions. These demonstrate the runner and scoring mechanics, not broad review reliability.
diff --git a/docs/ogc-1560-review-harness/execution-guide.md b/docs/ogc-1560-review-harness/execution-guide.md
@@ -47,7 +47,7 @@ The L3 reviewer maps requirements to appropriate test layers, tests, observable
 
 L4 uses `--scenarios` (behavior, category, applicable, reason, checkId) plus loaded runtime identity. OpenELIS collects image IDs and deployed Java class bytes itself. Other containers declare a `--runtime-check` whose `runtime` JSON artifact contains head, components (name, method, expected, observed), configuration and dependencies. Checkout HEAD or a login response alone cannot establish this identity.
 
-L5 additionally uses `--workflows` (id, requirement, checkId, designId or visualInapplicableReason) and `--designs` (id, file, sha256, source, revision, authority, approvalStatus). Designs may be draft for comparison, but only approved references support acceptance. A scenario declares artifacts by relative path and kind: http, readback, trace, screenshot, video, measurement, runtime or text. Fresh artifacts are copied out, bound to the candidate and checked for transport integrity. Screenshots and design images are attached to the reviewer. Missing media/readback, wrong loaded artifacts or visible deviations remain explicit in the report.
+L5 additionally uses `--workflows` (id, requirement, checkId, designId or visualInapplicableReason) and `--designs` (id, file, sha256, source, revision, authority, approvalStatus). Designs may be draft for comparison, but reference status is recorded without blocking comparison; formal sign-off is separate. A scenario declares artifacts by relative path and kind: http, readback, trace, screenshot, video, measurement, runtime or text. Fresh artifacts are copied out, bound to the candidate and checked for transport integrity. Screenshots and design images are attached to the reviewer. Missing media/readback, wrong loaded artifacts or visible deviations remain explicit in the report.
 
 The OpenELIS example `examples/openelis/label-preset-workflow.mjs` uses synthetic state in the disposable worker. Run the browser container as the worker UID/GID so subsequent checks can remove old evidence. The example records browser errors, screenshots, persisted readback and a trace, and restores the original preset. Setup exceptions are JUnit errors, not successful mutation detections. See `validation-v06.json` for the actual run and its limits. Artifacts are limited to 20 MiB each and 40 MiB per check; oversized evidence is reported as missing, never silently accepted.
 
@@ -64,3 +64,5 @@ Use `--adapter claude --model haiku --reasoning-effort low` for an inexpensive C
 `code-qa publish --result result.json --finding ID --verified ID` previews selected actionable findings. `--verified` is the operator's attestation, not a model-generated field. Add `--authorize-publication` only after reviewing that concrete preview and obtaining publication authorization. The publisher checks the live base/head, rechecks cited evidence, suppresses current-head duplicates, and posts a COMMENT review tied to the reviewed commit. It never approves or merges. GitHub can change immediately after the final check; the explicit commit ID preserves historical identity if that race occurs. Re-review dispositions remain private; this command does not resolve other reviewers' threads.
 
 The manual `Trusted review dispatch` workflow is restricted to trusted main, a protected `code-qa-review` environment, read-only GitHub permissions and source-only L1/L2. Configure that environment's reviewers and API secret before use. It never checks out candidate source or runs its code with secrets. L3–L5 use the local CLI with isolated execution; this hosted profile does not pretend to provision Lima. Public-fork event triggers are deliberately absent. The dispatcher is not live until this stack is reviewed and merged by its owner.
+
+OpenELIS preparation follows [the repository profile](../../examples/openelis/REVIEW.md): find matching `openelis-work` sources and published mocks before asking the user. `code-qa designs` returns ranked pinned references. Draft status is provenance, not a review-completion prerequisite.
diff --git a/docs/ogc-1560-review-harness/goals.md b/docs/ogc-1560-review-harness/goals.md
@@ -25,7 +25,7 @@ Captured 22 October 2026, clarified 30 October 2026, from the user's requests in
 
 ## Current scope
 
-The user has authorized implementing, testing and pushing the harness, with OpenELIS as its first isolated environment and one initial AI reviewer. Version 0.4 delivers the first Stage 1 contract/report changes; the stage remains in progress and the broader review methods are unfinished. Follow the [implementation checklist](roadmap.md), including acceptance evidence at every stage. Do not change application behavior, install enforcement hooks, merge PRs or infer review reliability from smoke tests.
+The user has authorized implementing, testing and pushing the harness, with OpenELIS as its first isolated environment and one initial AI reviewer. Version 0.5 implements the review methods and interfaces; the roadmap distinguishes execution evidence from outstanding independent evaluation and deployment acceptance. Follow the [implementation checklist](roadmap.md), including acceptance evidence at every stage. Do not change application behavior, install enforcement hooks, merge PRs or infer review reliability from smoke tests.
 
 ## Decisions still open
 

diff --git a/docs/ogc-1560-review-harness/implementation-status.md b/docs/ogc-1560-review-harness/implementation-status.md
@@ -1,5 +1,17 @@
 # Implementation and validation
 
+## Version 0.5 — 30 October 2026: implemented methods and explicit acceptance limits
+
+The current implementation supports evidence-based L1–L5, isolated test/probe execution, loaded-runtime verification, rendered product/design comparisons, Codex and Claude source review, managed-result exchange, explicit publication, and paired quality evaluation. Use the [execution guide](execution-guide.md), [evaluation guide](evaluation-guide.md), and [OpenELIS preparation profile](../../examples/openelis/REVIEW.md). The [checklist](roadmap.md) and [current evidence record](validation-v07.json) distinguish implemented tooling from independent quality acceptance. Entries below are historical.
+
+The OpenELIS profile now discovers `openelis-work` repository mockups and their published previews before asking the user. It records revision/provenance; Draft is context, not an approval gate. Designs define visible product intent, never engineering ownership. A reviewed comparison can complete with deviations; that does not mean the product matches the design.
+
+Executed demonstrations include an actual OpenELIS mutation with unchanged JUnit tests, a passing-per-layer/failing-assembled contract fixture, real loaded image/class identity, and a browser edit/save with persisted readback and rendered screenshots/trace. Codex and Claude inspected source through the restricted broker. Claude returned valid but partial semantic reviews; rendered-image conformance is not claimed for it.
+
+The paired development evaluation ran eight reviews using identical settings across plain/harness arms: four complete, four partial. Its labels are provisional, with a potential valid-case requirement ambiguity documented for adjudication. No accuracy, false-positive rate, or superiority claim is justified yet. Maintainers must approve independent labels and adjudicate held-out trials, including real runtime/design cases, before broader quality claims. The evaluator supports those steps; it does not manufacture independent approval.
+
+Manual hosted CI is implemented but requires merging and protected-environment configuration by the owner. Public-fork event automation remains disabled pending deployment security acceptance. Provider executables are trusted; the source/evidence broker restricts agent tools, not arbitrary provider software at the operating-system level. Nothing was merged.
+
 ## Version 0.4 — 30 October 2026: Stage 1 in progress
 
 The first implementation slice separates evidence obligations from execution settings. L1/L2 remain usable; L3–L5 return explicit partial results before startup or AI spend because their required verification executors are pending. Reports now preserve coverage, unanswered questions, counterevidence and actionable/suggestion/refuted dispositions. The core model allowlist is removed; a development profile keeps default testing inexpensive. The test-quality procedure is loaded by the reviewer.