fix(strix): distinguish recovered provider faults from inconsistent terminal scan evidence

[seonghobae]

Fresh exact canary

ContextualWisdomLab/Orgmetra#64@818950960c4fe0a4e7b026bd5499fe9cca1002be produced a new required Strix terminal failure in run 34131638369, job 101787200568.

The job is not a startup/queue/sandbox failure. Exact-head admission, trusted source materialization, contextual-orchestrator sidecar provisioning, Strix installation, and model/base preparation all succeeded. Run Strix (quick) ran from 2026-09-07T15:52:49Z to 18:52:48Z, then the artifact was uploaded.

Artifact strix-reports id 10030120846 is exact-head bound and has GitHub digest sha256:e61c95771ab6c37f7a60431f01eff4ada91adbcfbd2f6eaf79a803aacd7f9d5c. Independent local hashing of the downloaded ZIP matched that digest.

Inside that immutable artifact:

• run.json (sha256:3fab449a5fcc1d4507c163d58109c6d414b786dddc92f950e9f00beea73e6166) records status="completed", scan_results.scan_completed=true, scan_results.success=true, and 191 LLM requests.
• findings.sarif (sha256:32434df418c1d319e32cb5c174697ccd70bff77b93b74459fec9632c762225ab) contains an empty results array.
• penetration_test_report.md (sha256:037d479f18d3b8cbdaeed06eb95e8814536eff98333681be8ee6d5c41fed1566) says the penetration test completed with no high-impact vulnerabilities, but its Technical Analysis also says it identified "several low-impact issues", including potential SQL injection and employment-version-validation issues. Those claims are not represented as SARIF findings.
• gate-console.log (sha256:97b3bcb9752ce00efab0ae4d76d2c03d1952d3d152ad22faf8fb2a66a6206a2d) contains a real recovered openai.InternalServerError/HTTP 500 during the scan, then later prints Penetration test completed and Vulnerabilities 0, followed by Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed., Strix run emitted provider infrastructure or failure-signal output; failing closed., and finally STRIX_PROVIDER_UNAVAILABLE: contextual-orchestrator/orchestrator/free exhausted.

No rerun, elapsed-time cap, external provider fallback, manual success status, or bypass was used.

Source contradiction on protected main

Protected .github main@78a4937c684a54ca8e415822c913742f41c6efc4 currently documents in scripts/ci/strix_quick_gate.sh that the sticky infrastructure flag prevents incomplete-evidence bypass, but that a successful Strix exit ignores the flag because the scanner produced a complete result set. The same script then makes strict mode (STRIX_FAIL_ON_PROVIDER_SIGNAL=1) return failure for rc=0 whenever the newest report contains a warning/fatal/denied/timeout or provider failure signal. The required workflow explicitly sets STRIX_FAIL_ON_PROVIDER_SIGNAL: "1" and forbids external fallback because orchestrator/free owns discovery/failover.

The fresh Orgmetra canary therefore exposes two different questions that the current verdict collapses into STRIX_PROVIDER_UNAVAILABLE:

1. Recovered provider transport faults: a model request failed during a long scan, but Strix later emitted status=completed, scan_completed=true, and success=true.
2. Terminal evidence consistency: the terminal narrative itself names low-impact potential issues while SARIF contains no findings. That is an evidence-integrity concern independent of provider availability.

The current gate cannot tell the operator which of these actually invalidated the scan. In this canary it also says the gateway was exhausted even though the same artifact contains a terminal completed scan after the recovered fault. Do not treat that diagnostic as proof that the gateway had no usable route at termination.

Safety boundary

This issue does not propose converting recovered provider errors into automatic success. The Orgmetra artifact is not acceptable GREEN evidence as-is because the terminal narrative/SARIF representation is internally inconsistent. It also does not propose an elapsed-time timeout, direct provider/model fallback, orchestrator/auto, paid fallback, or weakening #891 fail-closed behavior.

The owner repair should preserve orchestrator/free and fail closed, but classify the failure causally.

RED acceptance

A deterministic fixture reproduces all of the following at once:

• Strix process exits 0 and terminal structured metadata says completed/success;
• a bounded/recovered provider exception exists earlier in the same run;
• terminal report/SARIF disagree about whether findings exist;
• current strict gate emits the generic provider-unavailable/exhausted verdict without distinguishing recovered transport telemetry from terminal evidence inconsistency.

Also retain fixtures where provider failure actually prevents terminal scan evidence; those must remain STRIX_PROVIDER_UNAVAILABLE/non-passing.

GREEN acceptance

1. Define a typed terminal-evidence state machine for at least: complete-consistent scan, incomplete/provider-unavailable scan, and complete-but-internally-inconsistent scan.
2. Recovered provider faults remain observable and auditable, but do not by themselves prove gateway exhaustion after a later terminal scan receipt exists.
3. A completed scan whose narrative/run metadata/SARIF disagree about findings fails closed as an evidence-integrity verdict, not a fabricated provider-exhaustion diagnosis.
4. Missing/partial/skipped/cancelled/neutral/provider-exhausted evidence remains non-passing per [Security] Fail closed when Strix produces no authoritative scan evidence #891.
5. Preserve orchestrator/free + gateway token only; no provider/model/group hard-code or external/paid fallback.
6. Preserve model timeout default-null semantics. Do not classify a healthy reasoning/stream/tool-call solely by elapsed time; Strix run ended "cancelled" at 6h with no verdict: 909 provider attempts, 92% on one route, 0 exhaustion events, request identity absent from evidence #2000's attempt/identity investigation stays separate.
7. Add exact artifact fixtures for recovered provider fault + complete consistent evidence, recovered provider fault + inconsistent terminal evidence, and genuine incomplete/provider-unavailable evidence.
8. After protected merge, rerun unchanged Orgmetra#64@818950960c4fe0a4e7b026bd5499fe9cca1002be and require a typed terminal verdict whose diagnosis matches the emitted artifact. Do not reuse predecessor evidence.

Related owner work

• [Security] Fail closed when Strix produces no authoritative scan evidence #891: fail closed when no authoritative Strix evidence exists.
• fix(strix): bare Fatal/Denied/Warn/Warning match lacks provider-context guard, causing false-positive fail-closed on clean scans #1291: false positives from bare warning/fatal/denied tokens; this canary is broader because it contains a real provider 500 plus a later terminal scan.
• Strix run ended "cancelled" at 6h with no verdict: 909 provider attempts, 92% on one route, 0 exhaustion events, request identity absent from evidence #2000: long-run request identity/exhaustion evidence; do not solve that by adding a wall-clock model timeout.
• [CO owner] eliminate orchestrator/free provider-family SPOF before gateway release #1915/Review sidecar probes and replays spend the per-key free-tier rate budget the review itself needs #1948: orchestrator/free provider-family availability and free-tier pressure.

[seonghobae]

Fresh downstream RED from ContextualWisdomLab/fast-mlsirm#1773@fa3d06a1a55ecc91e561603d1daf9f2c4de2c348 sharpens this issue beyond the existing Orgmetra inconsistent-terminal-evidence canary.

Exact required Strix run/job: 34284217353 / 102256794796. Exact retained artifact: strix-reports id 10080622334, GitHub digest sha256:feb402d0062244db0455c203395ae3b04ab95132f835de70db44f9d94bae620f.

I downloaded and inspected that immutable artifact rather than classifying from the check conclusion alone:

• run.json: top-level status="completed"; scan_results.scan_completed=true; scan_results.success=true.
• findings.sarif: valid SARIF 2.1.0 with results=[].
• penetration_test_report.md: repeatedly states no actionable vulnerabilities in the declared two-file PR scope; unlike the original Orgmetra canary, the terminal narrative observed here is consistent with zero SARIF findings.
• strix.log contains exactly one WARNING-level record: strix.tools.web_search.tool: web_search invoked without PERPLEXITY_API_KEY configured. No ERROR/CRITICAL/FATAL record is present in the report log.
• gate-attempts/001-orchestrator_free-rc0.log also contains Strix's ordinary MODEL QUALITY WARNING banner because openai/orchestrator/free is not one of Strix's hard-coded recommended model names.
• Nevertheless the scanner reaches Penetration test completed and Vulnerabilities 0, after which the central gate emits Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed, then Strix run emitted provider infrastructure or failure-signal output, and finally fabricates STRIX_PROVIDER_UNAVAILABLE: contextual-orchestrator/orchestrator/free exhausted.

The gateway was demonstrably not exhausted at admission: the same job's sidecar preflight selected 24 free candidates, found 6 ready routes, and the gateway chat/completions preflight completed on attempt 1 with finish_reason=stop; priced selected routes remained 0. The scan then ran for roughly 36 minutes and produced terminal structured evidence.

This adds a distinct RED class to #2026: complete + internally consistent terminal evidence plus an auxiliary-tool / model-quality warning is currently collapsed into provider exhaustion. A provider-exhausted verdict is causally false here even if policy ultimately decides the missing auxiliary web-search capability should keep the security gate non-passing.

Please preserve fail-closed semantics but add an exact fixture/state for this shape. At minimum, terminal classification must distinguish (a) genuine gateway/provider exhaustion, (b) completed evidence with recovered provider telemetry, (c) completed evidence with auxiliary capability unavailable (here PERPLEXITY_API_KEY), and (d) internally inconsistent report/SARIF evidence. Do not silence unknown warnings globally and do not convert this artifact to GREEN merely because the process exited 0; the point is to stop diagnosing an unrelated auxiliary/tool warning as orchestrator/free exhausted and to bind the final verdict to typed terminal evidence.

Owner boundary: no fast-mlsirm workflow/provider fallback was added, and orchestrator/free remains the only model route.

[seonghobae]

Fresh exact Orgmetra canary adds the complete-but-internally-inconsistent class already required by this issue, and shows why the terminal verdict must not be collapsed into provider exhaustion.

Consumer: ContextualWisdomLab/Orgmetra#63@d88800a5ca3ca15df332e8def5e25064c46e4005.
Required Strix run/job: 34294884985 / 102289341649.
Immutable strix-reports artifact: id 10083683583, GitHub digest sha256:0936587636b5689136fb28221d55b84982fa8241950d087cb52818cb2299887f.

The job passed exact-head admission, trusted source materialization, CO sidecar provisioning, Strix installation, and orchestrator/free model/base preparation. Run Strix (quick) then ran from 2026-09-09T00:38:00Z to 01:02:16Z and emitted terminal artifacts. run.json says status=completed, scan_results.scan_completed=true, and scan_results.success=true; its executive/technical summaries say no high-impact vulnerabilities. In the same terminal artifact set, however, vulnerabilities.json and SARIF contain one HIGH CWE-863 result, and the console says Vulnerabilities HIGH: 1 (Total: 1). This is terminal evidence inconsistency, independent of earlier provider/tool telemetry.

The gate then emits Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed, reports exit code 2, and finally diagnoses STRIX_PROVIDER_UNAVAILABLE: contextual-orchestrator/orchestrator/free exhausted. That diagnosis is causally wrong for a run that reached terminal structured evidence with 204 LLM requests; the artifact is non-passing because its terminal evidence conflicts, not because terminal evidence is absent.

Please add this exact fixture shape to the typed terminal-evidence state machine: completed/success metadata + narrative claiming no high-impact finding + SARIF/vulnerability ledger containing a HIGH finding. Preserve fail-closed semantics. Do not turn this run GREEN, suppress the finding, add a direct provider/model fallback, or classify a long-running reasoning/tool path by elapsed time. The required output is a typed evidence-integrity verdict distinct from genuine CO/provider exhaustion.

[seonghobae]

Related validation evidence from PR #2089 exact head 693e618b6c11cc4e1286734940a1c3b21a50d074: the new Strix location validator passes trusted-state 2 for both ../outside.py and a file reached through a symlinked ancestor inside the scan tree (scan/link -> target, link/secret.py). This means the current validator checks only the final file and does not bind finding paths to the trusted scan-tree root. Finding and reproduction are recorded on PR #2089: #2089 (comment).