feat(review): execute test-strength probes and bounded falsifying experiments

docs/ogc-1560-review-harness/execution-guide.md

@@ -26,3 +26,19 @@ Use `--decisions decisions.json` to designate accepted intent explicitly. Each e
 Pass `--history /previous/run/history.json`. The reviewer reassesses each prior finding as `still-present`, `resolved`, `superseded`, `declined` or `unverified`, retaining original and current evidence. Missing reassessment remains unverified. Evidence is checked against the current source; old report text is not fresh verification. `history.json` is private history, including resolved items; it is not a public comment payload. Nothing is published automatically.
 
 Reviewers can record a consequential approach objection with its problem, reachable path, supported simpler alternative and counterevidence. Suggestions stay separate from actionable defects. These are reviewer judgments; a valid source quotation alone does not prove the conclusion.
+
+## L3: executed tests and test strength
+
+Use `--probes probes.json` with the container executor. A probe declares `id`, `behavior`, `checkId`, `fixedTests`, production `path`, one exact `before`/`after` replacement, `expectedCase` and `expectedFailure`. The executor runs corrected → mutant → restored copies, keeps test source fixed and retains fresh JUnit reports. Assertion failure must match the expected case and diagnostic. Outcomes distinguish detected, survived, uncovered, invalid, setup-error, timeout and explicitly inapplicable probes. A valid removal may supply `inapplicableReason` instead of a mutation. A compile failure is not proof that tests detect the behavior.
+
+The L3 reviewer maps requirements to appropriate test layers, tests, observable assertions and executed check/probe IDs. Scope and gap status are explicit: unrelated behavior does not silently expand a targeted review. Semantic judgments about adequacy remain review judgments; execution and source identity checks cannot prove them automatically.
+
+`examples/java-unit-image` demonstrates a Java 21 executor with the repository's existing JUnit tests. Supply JUnit/Hamcrest jars in its build context; the image contains no repository or credentials. The small launcher records existing tests, without inventing replacement assertions. This example is appropriate for dependency-free units, not framework/persistence integration.
+
+## Requested experiments
+
+`--experiments catalog.json --max-rounds 1` reserves a second reviewer call within the same total review-time allowance. The catalog uses the same check format. The reviewer may request a catalog ID and explain what it would falsify; it cannot supply arbitrary execution commands. The executor runs the check in the isolated environment and supplies the result for reconsideration. Missing authorization, repeated requests and exhausted allowances remain explicit incomplete evidence. Default is zero extra rounds. No automatic model escalation or repair.
+
+## Dependencies and stack sources
+
+`--sources sources.json` attaches only explicitly selected repositories. Each source has a safe `id`, repository identity, absolute local `repo`, full pinned `revision`, role (`dependency`, `consumer`, `stack`) and `required` boolean. Optional `parent` validates stack ancestry; `pinPath` checks the primary candidate's gitlink. Missing required source or mismatched ancestry/pins is a coverage gap, not a code finding. Reviewers inspect these sources read-only and cite repository/revision with `sourceId`. `--history-depth N` explicitly includes needed ancestry; default is the selected revisions only. These checks do not merge or rewrite upstream branches.

docs/ogc-1560-review-harness/implementation-status.md

@@ -127,3 +127,7 @@ This version uses a fixed authentication smoke, not complete OpenELIS/browser su
 ## Context and isolated commands follow-up
 
 Added explicit accepted-decision provenance, paginated review-thread collection, private re-review history and generic container execution. Regression: 55 passing tests plus skill validation. Real L1 fixture: two passing behavior tests, zero AI calls, image identity recorded, cleanup removed the container. This is implementation and executor evidence; it does not establish higher-level review quality or completion of Stages 2–5. See [execution guide](execution-guide.md).
+
+## Test-strength implementation
+
+Pinned source/stack context, explicit behavior maps, isolated mutation probes, typed JUnit outcomes and a bounded catalog-only experiment loop are implemented. All 62 regression tests pass. The actual OpenELIS BoundedDecimal tests passed, detected a production mutant at the expected assertion and passed restored. A live Sol review exposed schema/scope issues now covered in the protocol; its historical partial result remains partial. See [validation-v05.json](validation-v05.json). Full runtime/product acceptance is still unverified.

docs/ogc-1560-review-harness/roadmap.md

@@ -2,7 +2,7 @@
 
 Updated 30 October 2026. This is the implementation roadmap following the requirements audit of code-qa v0.3 at `853a24d557cc451bce89ca8743fcf52a807de6fb`.
 
-**Status: Stage 1 is in progress; Stages 2–5 are pending.** Version 0.4 delivers the evidence contract and initial report/model-policy changes. Only checked items below are complete. See [validation-v04.json](validation-v04.json); this is not full-stage or review-quality acceptance.
+**Status: Stages 1–2 are implemented in part and under acceptance validation; Stage 3 implementation is in progress; Stages 4–5 remain pending.** Version 0.4 delivers the evidence contract and initial report/model-policy changes. Only checked items below are complete. See [validation-v04.json](validation-v04.json); this is not full-stage or review-quality acceptance.
 
 This checklist supersedes the earlier milestone sequence and the interpretation of L1–L5 as effort presets. See the [specification](specification.md), [goals](goals.md), [historical implementation evidence](implementation-status.md), and [20-gap audit](gap-inventory.json). Gap IDs below are stable links to that audit.
 
@@ -100,34 +100,36 @@ Scope is independent of level. L1 may execute existing deterministic probes; L2
 
 ### G05 — Inspect necessary dependencies, consumers and history
 
-- [ ] Add trusted collection of necessary pinned dependency/consumer source and relevant ancestry.
+- [x] Add trusted collection of necessary pinned dependency/consumer source and relevant ancestry.
 - [ ] **Acceptance:** Attach needed dependencies/consumers as pinned read-only sources through trusted collection. Permit bounded ancestry inspection when a claim needs history. Every source citation identifies repository and revision. A missing required dependency produces a question/coverage gap, not an invented conclusion. Keep unrelated source out of the default review.
 
 ### G06 — Review the assembled stack and both sides of contracts
 
-- [ ] Represent ordered PR stacks, dependency revisions and the assembled candidate.
+- [x] Represent ordered PR stacks, dependency revisions and the assembled candidate.
 - [ ] **Acceptance:** Resolve ordered stack/dependency revisions and the assembled state without merging upstream. Compare emitted and consumed contracts and relevant configuration. A fixture with two individually passing layers that fail together must be detected and attributed to the owning layer. Report stale ancestry/pins separately from code findings.
 
 ### G07 — Connect test adequacy and test-layer choice to the runner
 
-- [ ] Wire right-layer test assessment into review and produce a behavior-to-test map.
+- [x] Wire right-layer test assessment into review and produce a behavior-to-test map.
 - [ ] **Acceptance:** For changed behavior, record the requirement, appropriate test layer, selected test, observable assertion and remaining gap. Pure calculations may use unit tests; persistence/framework behavior needs a real integration path; rendered state needs rendering; a user journey needs the running product. Detect a mock-only test that misses the actual regression, and accept legitimate removal of obsolete tests.
 
 ### G08 — Execute meaningful mutation and revert probes
 
-- [ ] Implement isolated production-mutation/revert probes with explicit expected assertions and outcome classification.
+- [x] Implement isolated production-mutation/revert probes with explicit expected assertions and outcome classification.
 - [ ] **Acceptance:** Predeclare the behavior, fixed tests, production mutation/revert and expected assertion. Show corrected pass → meaningful faulty variant fails the intended assertion → restored pass, in a disposable workspace. Distinguish survived, uncovered, equivalent/inapplicable, invalid mutation, setup/compile error, timeout and real detection. Cancellation discards mutated state. Refactors/removals may be explicitly inapplicable; never manufacture a red result.
 
 ### G09 — Let the reviewer request a falsifying experiment
 
-- [ ] Add a bounded reviewer-requested verification loop controlled by the executor.
+- [x] Add a bounded reviewer-requested verification loop controlled by the executor.
 - [ ] **Acceptance:** Support a bounded inspect → request check → isolated execution → reconsider sequence. The coordinator validates scope, authority, capabilities and execution limits; the model never receives unrestricted execution credentials. If a check cannot run, preserve the exact question/omission. Test a suspected defect that the requested experiment refutes.
 
 ### G10 — Represent actual outcomes and evidence types
 
 - [ ] Support typed artifacts and distinguish test cases, assertion failures and infrastructure errors.
 - [ ] **Acceptance:** Report test cases as test cases unless assertion counts are known. Preserve assertion failures versus infrastructure errors and links to raw reports. Support typed evidence such as command output, HTTP exchanges, persisted-state readback, browser traces, screenshots/videos and measurements. Validate artifact identity and make necessary evidence available to the reviewer; reuse the existing bundler where useful.
 
+**Implementation evidence:** [validation-v05.json](validation-v05.json) records 62 passing regression tests and real corrected → faulty → restored runs for both a decimal fixture and OpenELIS `BoundedDecimalTest`. The latter executed the three existing JUnit tests unchanged in Java 21. Live model runs remain recorded as partial until the final scope/gap contract demonstration passes. Typed JUnit outcomes/raw reports are implemented; browser/media artifacts are Stage 3 work.
+
 ### Stage 2 exit
 
 - [ ] **Working demonstration and negative controls:** One real OpenELIS change demonstrates useful tests and a meaningful mutation; a weak test survives the mutant and is reported; a setup error is inconclusive; a valid refactor is accepted; an incompatible stack contract is caught.

docs/ogc-1560-review-harness/validation-v05.json

@@ -0,0 +1,12 @@
+{
+  "date": "2026-10-30",
+  "scope": "Stage 2 implementation and controlled validation; full roadmap acceptance remains open",
+  "deterministicTests": 62,
+  "realProbeRuns": [
+    {"run":"34581962-070a-4aa4-84b2-f06421d62fb0","repository":"synthetic decimal boundary fixture","head":"36d2eb43ed7b3de1a47c52cbebd9476c6a5dbf06","baseline":"pass","mutant":"specific precision assertion failed","restored":"pass","cleanup":"removed"},
+    {"run":"d8795553-2c63-4daa-a8e8-6253b0e74866","repository":"DIGI-UW/OpenELIS-Global-2","head":"37c4bc496f79feb047b2b3a68ead2b081b231571","test":"BoundedDecimalTest","testCases":3,"baseline":"pass","mutant":"acceptsTheNumbersAnalyzersReport failed on 7.5","restored":"pass","cleanup":"removed"}
+  ],
+  "negativeControls": ["weak assertion survives mutant","compile error is inconclusive","timeout is inconclusive","wrong assertion cannot establish detection","fixed-test mutation rejected","stale dependency pin and wrong ancestry expose gaps","undeclared reviewer commands never execute"],
+  "liveReviewer": {"model":"gpt-6.1-sol","effort":"low","status":"partial","reason":"Initial live runs exposed strict schema and explicit scope/gap protocol issues; retain those results rather than relabel them as complete"},
+  "unverified": ["held-out review quality","full OpenELIS runtime and product acceptance","unattended fork isolation","second real reviewer adapter"]
+}

examples/java-unit-image/Dockerfile

@@ -0,0 +1,5 @@
+FROM maven:3.9-eclipse-temurin-21
+RUN apt-get update && apt-get install -y --no-install-recommends python3 && rm -rf /var/lib/apt/lists/*
+COPY junit.jar hamcrest.jar /opt/code-qa/
+COPY JUnitEvidence.java /opt/code-qa/
+RUN javac -cp /opt/code-qa/junit.jar /opt/code-qa/JUnitEvidence.java

examples/java-unit-image/JUnitEvidence.java

@@ -0,0 +1,33 @@
+import java.nio.file.*;
+import java.util.*;
+import javax.xml.parsers.*;
+import javax.xml.transform.*;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+import org.junit.runner.*;
+import org.junit.runner.notification.*;
+import org.w3c.dom.*;
+
+/** Records the existing JUnit tests; it adds no replacement assertions. */
+public class JUnitEvidence {
+  public static void main(String[] args) throws Exception {
+    Document doc = DocumentBuilderFactory.newInstance().newDocumentBuilder().newDocument();
+    Element suite = doc.createElement("testsuite"); doc.appendChild(suite);
+    Map<Description,Element> cases = new LinkedHashMap<>();
+    JUnitCore runner = new JUnitCore();
+    runner.addListener(new RunListener() {
+      public void testStarted(Description d) {
+        Element c=doc.createElement("testcase");c.setAttribute("classname",d.getClassName());c.setAttribute("name",d.getMethodName());suite.appendChild(c);cases.put(d,c);
+      }
+      public void testFailure(Failure f) {
+        Element c=cases.get(f.getDescription());
+        if(c==null){testStarted(f.getDescription());c=cases.get(f.getDescription());}
+        Element e=doc.createElement(f.getException() instanceof AssertionError ? "failure":"error");
+        e.setAttribute("message",String.valueOf(f.getMessage()));e.setTextContent(f.getTrace());c.appendChild(e);
+      }
+    });
+    org.junit.runner.Result result=runner.run(Class.forName(args[0]));
+    TransformerFactory.newInstance().newTransformer().transform(new DOMSource(doc),new StreamResult(args[1]));
+    System.exit(result.wasSuccessful()?0:1);
+  }
+}

review/checks.mjs

@@ -16,6 +16,7 @@ export function validateChecks(checks) {
 }
 export function validateCheckResult(result, spec, head) {
   if (result.id !== spec.id || result.head !== head || result.kind !== spec.kind) throw new Error('Check evidence identity mismatch');
+  if (result.status === 'error' && ['infrastructure-error','timeout','uncovered'].includes(result.outcome) && Number.isInteger(result.executed) && result.executed >= 0 && Array.isArray(result.assertions)) return result;
   if (!['pass', 'fail'].includes(result.status) || !Number.isInteger(result.executed) || result.executed < 1 ||
       !Array.isArray(result.assertions) || result.assertions.length !== result.executed ||
       result.assertions.some(a => typeof a.name !== 'string' || !a.name || typeof a.passed !== 'boolean')) throw new Error('No valid executed check assertions');

review/cli.mjs

@@ -25,17 +25,17 @@ Options:
   --application pr|targeted --question TEXT --lens TEXT
   --path PATH (repeatable) --context FILE (repeatable)
   --requirements FILE --config FILE --environment none|openelis|container --image IMAGE
-  --decisions FILE --history FILE
+  --decisions FILE --history FILE --sources FILE --history-depth N --probes FILE --experiments FILE --max-rounds 0..2
   --profile development --adapter codex --model MODEL_ID --billing-mode subscription|api|unknown
   --review-seconds N --reasoning-effort low|medium|high
   --startup-seconds N --checks FILE
 Config/check files are explicit trusted operator inputs, not PR instructions.
 L1: no AI. L2: semantic review with explicit coverage and questions.
 L3: test-strength proof. L4: running integration. L5: product/design acceptance.
-L3-L5 evidence executors are pending; these runs return partial before AI/startup.
-L1/L2 can run declared checks in OpenELIS (kind: test|scenario, argv, reports,
+L3 needs declared isolated probes and a behavior-to-test map. L4/L5 remain pending.
+L1/L2 can run declared checks in container/OpenELIS (kind: test|scenario|command, argv, reports,
 timeoutSeconds); those checks do not imply completion of a higher level.
-The development profile uses one inexpensive Sol session; --model overrides it.
+The development profile uses one inexpensive Sol session by default; --max-rounds explicitly reserves up to two reconsideration rounds within the same time allowance; --model overrides it.
 No automatic retry or model upgrade. Provider-specific model availability still applies.
 --config overrides the repository's .code-qa.json read from the base revision.
 Legacy --rigor mechanical|standard aliases map to L1/L3.
@@ -48,7 +48,7 @@ try {
     out:{type:'string'}, plan:{type:'string'}, run:{type:'string'}, level:{type:'string'}, lens:{type:'string'},
     application:{type:'string'}, question:{type:'string'}, purpose:{type:'string'}, requirements:{type:'string'},
     path:{type:'string',multiple:true}, context:{type:'string',multiple:true},
-    image:{type:'string'},decisions:{type:'string'},history:{type:'string'},config:{type:'string'}, checks:{type:'string'}, environment:{type:'string'}, rigor:{type:'string'},
+    sources:{type:'string'},'history-depth':{type:'string'},probes:{type:'string'},experiments:{type:'string'},'max-rounds':{type:'string'},image:{type:'string'},decisions:{type:'string'},history:{type:'string'},config:{type:'string'}, checks:{type:'string'}, environment:{type:'string'}, rigor:{type:'string'},
     profile:{type:'string'}, adapter:{type:'string'}, model:{type:'string'}, 'billing-mode':{type:'string'}, 'reasoning-effort':{type:'string'},
     'review-seconds':{type:'string'},'startup-seconds':{type:'string'},
   }});
@@ -69,6 +69,8 @@ try {
     const baseRequirements = await configurationRequirements(snap,config);
     const requirements = [baseRequirements, v.requirements ? await readFile(v.requirements,'utf8') : ''].filter(Boolean).join('\n\n');
     plan = await createPlan({...source,...neighbors,application:v.application,question:v.question,purpose:v.purpose,
+      sources:v.sources ? JSON.parse(await readFile(v.sources,'utf8')) : config.sources ?? [],historyDepth:Number(v['history-depth'] ?? config.historyDepth ?? 1),
+      probes:v.probes ? JSON.parse(await readFile(v.probes,'utf8')) : config.probes ?? [],experiments:v.experiments ? JSON.parse(await readFile(v.experiments,'utf8')) : config.experiments ?? [],maxRounds:Number(v['max-rounds'] ?? config.maxRounds ?? 0),
       decisions:v.decisions ? JSON.parse(await readFile(v.decisions,'utf8')) : config.decisions ?? [], history:v.history ? JSON.parse(await readFile(v.history,'utf8')) : null,image:v.image ?? config.image,
       requirements,paths:v.path,environment:v.environment ?? config.environment,level:v.level,rigor:v.rigor,lens:v.lens,
       profile:v.profile ?? config.profile,adapter:v.adapter ?? (resolveLevel(v).level === 'L1' ? undefined : config.adapter),model:v.model ?? (resolveLevel(v).level === 'L1' ? undefined : config.model),billingMode:v['billing-mode'],reasoningEffort:v['reasoning-effort'], checks,