feat(bin): add three-valued availability observation - #89
Open
sbracewell64 wants to merge 26 commits into
Open
Conversation
sbracewell64
changed the title
sbracewell64
changed the title
Two facts collapsed into one recorded availability state: "the probe could not run" and "the probe ran and established the candidate is unavailable". A routed model that is entitled and live was recorded unreachable, the single-candidate pool it sits in was refused, and the refusal carried an empty reason. Three defects produced that, all repaired here. fm-model-verify.sh merged its own probe schema into state/model-health.json, which fm-route-lib.sh owns as a negative-only hold register keyed on an entry's mere presence. Every probed model became a permanent hold, including models whose probe had positively reported them reachable. Holds now go through that library's supported writer in its own closed vocabulary, and what a probe observed goes to state/model-observation.json. The result record wrote absent numeric fields as consecutive tabs. Tab is an IFS whitespace character, so read folded the run into one delimiter and the reason arrived in the rc field while the reason field arrived empty. No field is emitted empty now, and a reader that printed nothing says so. claude had no probe arm at all, so every claude-routed model recorded a could-not-observe forever. The arm uses the command shape config/models.json already records as how that model's identity was established. bin/fm-availability-lib.sh is the new owner of what a probe result means: AVAILABLE, UNAVAILABLE, UNOBSERVABLE, bound by a total bijection onto fm-verify-lib.sh's PASS/FAIL/NO_VERIFIER_RAN and consumed through that library's own exhaustiveness rule rather than a second copy of it. An unmapped shape reaches UNOBSERVABLE by the default arm. Routing stays fail-closed and is not loosened. UNOBSERVABLE excludes a candidate, with the reader, the failure class, the evidence and the affected routes named, so the refusal points at the reader to repair rather than at a hold that releasing would not fix. Only an attempted-and-failed observation excludes; a model nobody probed has no entry. A positive observation never admits a candidate and never releases a hold, so a stale positive cannot override a fresh negative. The enrichment step now carries every exclusion the decision found rather than recomputing from a subset. A foreign hold entry is reported with the supported release command rather than reinterpreted, so no candidate is silently re-admitted. Runtime evidence as a second source of a positive observation is NOT built. Task metadata and the wake ledger record the bare harness alias rather than the registry key, the registry itself records that the alias binding moves on harness updates, and no record carries a per-invocation served-at timestamp, so identity, freshness and scope all fail. The reasoning is recorded in the library header so it is not rediscovered. The recurrence probe tests the mechanism rather than the symptom: it reintroduces each half of the permitting mechanism in a controlled copy and requires the probe to fail against it, failing loudly if a mutation did not change behaviour. A live opt-in guard covers the harness-dependent probe command shapes, reports an absent harness explicitly, and refuses a pass that checked nothing.
…vation An independent design review of the candidate found eight landing-blocking defects, two of which reintroduced the permitting behaviour this lane exists to remove. This repairs all of them. The observation record is validated on every read. It previously recovered only from unparseable JSON, so a file that was valid JSON with a wrong or absent schema succeeded as an EMPTY exclusion set: every recorded could-not-observe disappeared and every candidate it excluded became eligible again. Parseability is not validity, and the schema, entry shape, closed observation vocabulary and tooling-gap fields are now all checked, with the reason for a refusal printed so the record can be repaired rather than guessed at. An established unavailability is enforced independently of its hold. The two records are written by two calls, and an invariant that holds only when both writes succeed is not an invariant: a lost hold write left a measured negative fact in the observation record while routing called the candidate eligible. The fail-closed hold is now written first, and routing excludes on the observation itself when no hold accompanies it. So that an explicit override still means something, the supported release retires the UNAVAILABLE observation it overrides - never an UNOBSERVABLE one, because releasing repairs nothing about a broken reader. The claude probe runs with nothing of this machine attached to it: an empty working directory, no user, project or local settings and therefore no hooks, no MCP servers, a tool deny list, no session persistence and no inherited subagent config. The neighbouring pi path has said --no-tools --no-session since it was written; this arm passed only --strict-mcp-config, which addresses MCP and nothing else. A broken reader now files its own repair work. The tooling-gap block recorded backlog_item: null always, which closed broken-reader-to-evidence and left evidence-to-repair to whoever read the record. The item is filed through the backlog this home already uses, under an id derived from the candidate so repeated sweeps converge instead of duplicating, and confirmed through the same open-item reader a TOOLING_GAP dispatch is certified against. A null now always travels with a status saying why there is no item. Production routes probe results through the canonical exhaustive consumer rather than a case whose default arm was AVAILABLE, so an empty or unexpected observation can no longer reach the permissive branch; the gap block's reason code comes from its owner instead of a literal; a refusal names every exclusion that applies instead of stating that no hold is involved when one is recorded; and probe evidence is sanitized where provider text first enters the fleet's records, bounding length and removing terminal control sequences and credential-shaped strings before either a record or an operator's terminal sees it. Reverts two attestation documentation commits that entered this branch from the validation pipeline and belong to no part of this lane.
…controls One case per defect the independent design review found, each red for the mechanism rather than for the symptom. Eight observation records that all PARSE - wrong schema, absent schema, empty object, models as an array, an observation outside the closed vocabulary, an UNOBSERVABLE entry with no tooling-gap block, an entry that is not an object - must each refuse rather than read as an empty exclusion set, with a valid record alongside them as the control that keeps this validation rather than a blanket refusal. An established unavailability keeps excluding when its hold is removed. The supported release retires the observation it overrides and can never clear a broken reader. A tooling gap files a repair item the certification reader accepts, converges on the same item across sweeps, and records why when it cannot file one. A refusal names both exclusions when both apply. Probe evidence is bounded, stripped of terminal control sequences and redacted. The probe's isolation is observed rather than assumed: a fake harness records the working directory, argument vector and environment it was launched with, and the case runs it from a project carrying instructions, a settings hook and a secret, so the boundary is measured for free rather than through a live request. The recurrence control grows from two mutants to five. Each new one reintroduces a permitting mechanism this change removed - a reader that recovers a corrupt record into an empty exclusion set, eligibility computed from the hold alone, and a two-branch consumer whose default arm is the favourable one - and the probe must turn red against it. The exhaustive-consumer control injects the same fault on both sides and differs only in the consumer, so it isolates the exhaustiveness rule rather than the map. Every test home now carries its own backlog config, so a case that files a repair item cannot reach the operator's real backlog. Eligibility is matched as a whole line rather than as a substring, because a terminal report names the candidates it refuses and a substring check is true of both answers.
… contract Re-establishes the probe command-shape rows against the real binaries after the claude arm gained its isolation flags: claude/opus answers AVAILABLE in 3s and openai-codex/gpt-5.6-sol in 7s, both server round trips, which is the timing discriminator that separates a client-side configuration error from a provider answer. The live guard now reports that latency on every run, so the number is re-established by the same command rather than measured by hand. Adds what a reader has to satisfy before the record is read at all, why an established unavailability is enforced independently of its hold and what the supported release therefore retires, how probe evidence is sanitized, and how the probe's isolation boundary is observed without a live request. The recurrence table grows from three mutants to six. The three new ones are fail-open defects found in this change itself, controlled the same way rather than merely fixed. States the sweep-timeout decision plainly rather than leaving it implicit: a probe killed by the ceiling records a could-not-observe because an unfinished probe with no result at all is the same collapse one level down, and the consequence - a slow provider excluding its candidate until the next sweep - is the fail-closed direction with the ceiling or the provider as the repair.
The sweep now files the reader-repair item itself, so the diagnostics playbook tells firstmate which item to dispatch rather than to file one, and the reported TOOLING_GAP line carries that id - or UNFILED with the recorded reason when the backlog backend could not be used, which is the only case that still needs a hand-filed item. Corrects two statements that the repairs made inaccurate: the observation record has one owning library rather than one writing script, since the supported release also retires an entry an operator explicitly overrides, and the spawn chokepoint refuses an UNAVAILABLE observation no hold accompanies as well as an UNOBSERVABLE one.
sbracewell64
force-pushed
the
fm/availability-observation-three-valued
branch
from
August 13, 2026 19:30
3f71bc7 to
16065de
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Make availability observation distinguish observed unavailable from could not observe, preserve fail-closed routing, emit repairable TOOLING_GAP backlog evidence, repair the Claude reader, and satisfy the required negative and red-capable recurrence controls plus retrospective findings F1-F8. For this CI repair, make Behavior portable serial 2 deterministic without changing production behavior, keep changes minimal and cross-platform, verify locally, and publish a head-bound no-mistakes attestation for the exact new commit.
What Changed
AVAILABLE,UNAVAILABLE, andUNOBSERVABLE), including repaired Claude probing, validated records, and repairableTOOLING_GAPbacklog evidence.Risk Assessment
✅ Low: The three-valued observation change consistently preserves fail-closed routing, validates durable evidence, repairs the Claude probe boundary, and includes deterministic recurrence coverage without a substantiated source defect.
Testing
Targeted validation exercised the real availability sweep/routing/backlog paths with fake provider readers, all seven red-capable recurrence controls, and the portable-serial-2 remote trace test twice; all behavior checks passed and reviewer transcripts were captured, but exact-head attestation verification failed because no no-mistakes notes ref is present.
Evidence: Availability behavior transcript
Evidence: Portable serial shard 2 selection
Evidence: Remote trace deterministic run 1
Evidence: Remote trace deterministic run 2
Evidence: Exact-head attestation verification
fm-attest: not attested (no-attestation-ref) This repository holds no refs/notes/no-mistakes, so no attestation was published or fetched.Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
bin/fm-attest.sh verify --head ...refused withno-attestation-ref. The behavioral changes pass locally, but the authoritative acceptance criteria are not fully satisfied until that exact head is validated and itsrefs/notes/no-mistakesattestation is published/fetched.git diff --stat 09200e86f88f575d2f20f59a7996d204a00ae7d8..1738be23e378b1c186bac3b2bdf73c9ef5121b60and inspected the focused verification documentation/testbash tests/fm-availability-observation.test.shbin/fm-test-run.sh --list --lane portable-serial-2of4bin/fm-test-run.sh --json /tmp/no-mistakes-evidence/01KZWS1XXX2BH7RF9AXVSN9B8F/remote-trace-run-1.json tests/fm-remote-secondmate-trace-context.test.shRepeated the remote trace test asremote-trace-run-2.jsonto exercise deterministic cleanup twicebin/fm-attest.sh verify --head 1738be23e378b1c186bac3b2bdf73c9ef5121b60✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.