Close the second-party review's nine findings and eight variants of them - #1
Open
Jamarian Payne (SpookyGroup) wants to merge 10 commits into
Open
Jamarian Payne (SpookyGroup) wants to merge 10 commits into
Jamarian Payne (SpookyGroup) wants to merge 10 commits into
Conversation
…review
R1 Confused deputy: learn_fact() now stores evidence (non-retrievable
candidate + server-recorded origin) even for identities holding
PUBLISH_MEMORY. publish=True is the explicit opt-out. New agent_path_v1
benchmark with a negative control (default 0/5, publish control 3/5).
R2 assemble_context honours query, task_type and max_tokens: lexical
word-stem relevance and mild recency re-rank; admission is token-budgeted.
Guardrails are always included and exempt from the budget.
R3 Authorized corrections register a contradiction: node identity and
dependency edges survive a republish, dependents receive grief. Session-step
success/failure is now weak operational evidence (0.25x, trust ceiling 0.75
< the 0.77 maximum-stakes bar).
R4 Producer-declared trust ceilings (never raises): failed episodes keep
outcome-derived trust, promoted skills are capped at 0.5.
R5 validate_skill replays triggers against held-out episodes (precision,
recall, lift over the no-skill baseline). shadow_runs counts distinct
episodes, so repeated calls cannot manufacture runs. Says plainly that it
is not outcome improvement.
R6 Restatement check strips zero-width/format characters and punctuation and
compares by similarity. The same zero-width gap hid protected terms and
prefixes from policy matching; fixed in _normalize.
R7 Faith is earned by confirmation (cap 0.6 < sacred 0.92), eroded by
contradiction.
R8 get_context_nodes returns detached copies; MemoryStore docstring no longer
claims an encapsulation it does not provide.
R9 Console context endpoint formats through a request-local adapter instead of
mutating the shared gateway and raising on every provider format.
Tests: 127 -> 179. New guards were confirmed to fail with the fix reverted.
Docs corrected where claims outran code, including a stale README statement
that NOE-F-026 was still open. Ledger entries follow in the next commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…party review Nine findings, all fixed by 115e126, each with regression tests. The review is preserved verbatim in evidence/second-party-review-2026-09-19.md with the pre-repair verification of every claim. Classified first-party on purpose: the reviewer's identity and repository access are not independently established. Recorded honestly in the pack: - the policy-matching half of NOE-F-034 was NOT in the review; found in repair - the 0.999 trust figure (NOE-F-032) was derived, not observed - the R9 console failure was confirmed by reading, not executed live - seven new limitations, including that provenance does not yet propagate through derived nodes (NOE-L-016) Also corrects a stale README claim that NOE-F-026 was still open, and adds a dated update to REVIEW-REQUEST.md without altering its pinned figures. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…finds them The first repair fixed the reported instances. A proactive variant hunt found eight more routes to the same failures; each is fixed and regression-tested, including a structural guard so the class cannot silently reopen. - Promotion / quarantine release rewrote only `value`, but providers also emit a skill's objective/method/constraints and (formerly) an episode's reflection. Every non-reviewed field is now reset on promotion and preserved in audit metadata; skill status returns to PROPOSED. - The autopsy wrote raw task text and tool error output into ACTIVE episodes, and the Claude adapter emitted the whole narrative. Episode `value` is now templated from system data only (closed pattern vocabulary, sanitised tokens); the raw narrative is kept, bounded, for audit and never rendered. Skill candidate descriptions and missed-opportunity text are sanitised at source. - A skill could be written already PROMOTED, and promote_skill trusted shadow_runs/score/lift set on the object. Direct writes of PROMOTED are refused; promotion recomputes its evidence from the store. - Candidate and quarantined facts earned or lost trust from session signals, so trust could be farmed before promotion. Only published facts take part. - The token estimate used raw length; JSON/XML escaping emits up to 6x more, so non-ASCII text slipped past the budget. Estimate now bounds escaped output. - Policy matching survives combining marks, variation selectors, blank fillers, common Cyrillic/Greek look-alikes, and spacing/punctuation splits. - set_profile / set_project_state (always-loaded classes) published directly, exactly as learn_fact did. Evidence by default; publish=True is explicit. - The benchmark scored only node.value. It now scores the text providers render, and agent_path_v1 grows to 8 cases (default 0/8, publish control 6/8). New structural guard: test_provider_emission_matches_the_reviewed_allowlist fails whenever a provider begins emitting a field nobody reviewed. Each hardened module was reverted alone to confirm its tests fail without it. Tests: 179 -> 228. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Eight variants of the review's flaw classes, all fixed by 7df9919: promotion scrub, episode narrative channel, skill publication gate and self-reported evidence, candidate trust farming, escape-aware token budget, wider policy folding, profile/project-state publication, and the benchmark's node.value-only scoring. NOE-L-016 (provenance not propagated through derived nodes) is marked resolved with a status transition rather than deleted, per ledger law: it was logged as a limitation in the first repair and is fixed in this one. NOE-L-014 is narrowed to what the folding now covers. NOE-L-021 records that the emitted-field allowlist is enforced by a test rather than by types. evidence/variant-hunt-2026-09-21.md records the method (emission, producer, self-report and measurement audits), the emission table, and the per-module negative controls. First-party; not independent review. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The earlier repair made faith EARNABLE from confirmations. For a zero-trust design that was backwards: relief from grief must not be something a node, or an agent acting on it, can buy. Operational confirmations are exactly the kind of signal an agent can influence, so an attacker could have farmed faith to make a poisoned node resist purging. - Faith is policy. TrustGate.faith_for(node) returns the operator's base_faith (guardrails: the sacred constant). Every consumer reads it there, never from the stored value, so a tampered value grants nothing. - Nothing a session does moves faith: the growth and loss added last round are deleted (net fewer lines). - Tripwire: any stored faith that differs from policy, higher or lower, is treated as tampering. GriefCascade force-purges the node through the ordinary cascade path (skipping faith resistance and seppuku criteria), so dependents are notified, and records the event in tamper_log. Sacred nodes are logged and left in place. - GriefCascade.cusp() is a read-only readout of the distance from a cascade (per-node and aggregate) for tuning. The benign benchmark reports its minimum node margin; today that is 0.900, i.e. the system is nowhere near the cusp. - base_faith is a constructor parameter: published numbers are defaults, deployed values are operator-set. Tests that mutated stored faith to obtain resistance now configure policy instead; one asserted grief < 0.9, which a purge satisfied by zeroing grief, and is now non-vacuous. Each changed module was reverted alone to confirm its tests fail (grief_cascade 9, trust_gate 15). Tests: 228 -> 241. Benchmark unchanged (0/13, 0/8, agent path 0/8 vs 6/8). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Records that the earlier repair of NOE-F-035 (making faith earnable) was wrong for a zero-trust design and is replaced by static policy faith plus a tamper tripwire (6919d6e). Per ledger law NOE-F-035 keeps its entry and gains a history event pointing at NOE-F-046 rather than being edited. Two new limitations recorded honestly: the tripwire covers faith only (no keyed seal over other governance fields), and the operating point is untuned - the benign corpus leaves a node margin of 0.900, so the intended cusp has not been approached. The cusp() readout must be kept away from the agent. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Cloud review of PR #1 raised two findings, both verified against source, both pre-existing in master. Hunting their classes turned up two more real routes. - Replacing a published node kept its id and dependency edges only inside contradiction handling, so an identical or empty-value republish, and any replacement that landed QUARANTINED, silently dropped the graph the cascade walks. The quarantine path also overwrote the published value with a non-retrievable one. Identity and edges now carry over in write() for every replacement, and a replacement that policy would quarantine is refused so the published value is preserved. - write() returns True for "stored" as well as "published". promote_skill logged and returned success when the skill was stored as a candidate (or quarantined). It now refuses up front when the identity cannot publish, and verifies the skill is retrievable afterwards. process_patterns no longer reports drafts whose write was refused; end_session warns when its episode was not stored. MemoryStore gains can_publish() and is_retrievable(). - Variant: republishing an identical value reset grief to 0, letting anyone with publish authority launder the circuit breaker. Grief, grief state and the confirmation record now carry over when the claim is unchanged. - Variant: republishing over a purged node revived it as ACTIVE. A purged key is now refused; a new key is the way back. Tests: 241 -> 257. Each changed module reverted alone to confirm its tests fail (store 13, skill_forge 3, gateway 1). Benchmark unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two findings from the first ultrareview of PR #1, both verified against source and fixed in a75f664, plus a third route found while hunting their classes (republishing laundered grief and revived purged keys). The report is preserved as received, including its truncation, in evidence/ultrareview-1-2026-09-21.md. Classed first-party: the reviewer's independence from the author of the fixes is not established. NOE-L-024 records that write() still returns a boolean meaning "stored", not "published"; known callers now check is_retrievable(), but the API shape still permits the mistake. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
MemoryStore.write() returned (True, reason) both for a node that went ACTIVE and for one held as a candidate or quarantined. Callers that needed publication read the boolean and were wrong (promote_skill, found by the ultrareview). Fixing the callers left the trap in the API; this removes it. WriteResult carries an explicit outcome (PUBLISHED, CANDIDATE, QUARANTINED, REFUSED) with .stored and .published. It deliberately has no truth value and cannot be unpacked, so `if store.write(...)` and `ok, why = store.write(...)` raise instead of guessing. The write() family (write, write_fact, write_episode, write_profile, write_project_state) and the gateway's learn_fact, set_profile and set_project_state return it. promote_candidate, release_quarantined and write_guardrail keep (bool, str): True there means the effect happened. Because the type only raises on paths that execute, a static guard scans the package for the mistake (boolean, subscript or unpack of a write-family call), and is itself tested against each offending form and each accepted form. 63 existing call sites converted; two monkeypatches and one getattr-style call by hand. Benchmark harness and friction adjusted at their boundary; the naive baseline store keeps its own tuple. Quickstart and README show .stored and .published. Tests: 257 -> 276. Reverting each changed module fails its tests (store 8, skill_forge 1, gateway 2). Benchmark unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Records the return-type repair (9009047). NOE-L-024 is marked resolved with a status transition rather than deleted, per ledger law. NOE-L-025 is the honest residual: the static guard recognises calls by attribute name, so aliasing or getattr-style calls are only caught at runtime. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Repairs all nine findings from a second-party adversarial review (NOE-F-029 to 037) and eight variants of them found by a deliberate follow-up hunt (NOE-F-038 to 045), each with regression tests. Suite: 127 -> 241 passing. Benchmark headline unchanged (0/13 attacks, 0/8 false positives); the new agent-path corpus is 0/8 by default versus 6/8 in its control arm.
Part 1 - the nine reported findings
learn_fact()went livepublish=Trueis the explicit opt-outassemble_contextignored query/task_type/max_tokensregister_correction; operational evidence weighted 0.25, trust ceiling 0.75write()overwrote producer-declared trusttrust_ceilingPart 2 - variants, hunted before a second reviewer could find them
The first repair fixed the reported instances. The hunt asked, for each fix, what a hostile reader would try next, using four audits: which fields providers emit vs what review covers, which code paths write nodes, where decisions rest on self-reported values, and whether the benchmark could even see each attack.
value, but providers also emit skillobjective/method/constraintsand episodereflectionvaluetemplated from system data only; raw narrative kept for audit, never renderedPROMOTED;promote_skilltrusted self-reported evidencePROMOTEDwrites refused; promotion recomputes its own evidenceset_profile/set_project_statepublished directly (always-loaded classes)learn_factnode.valueA structural guard (
test_provider_emission_matches_the_reviewed_allowlist) fails whenever a provider starts emitting a field nobody reviewed, so this class cannot silently reopen.Worth reviewing
publish=Truecontrol arm reproduces pre-repair behaviour and lands 6/8, including the profile, project-state and episode-narrative routes. A containment result is only evidence if that control loses.set_profilepublishing).README.mdnow has a table of exactly what can reach the model and what gates each field.Honest limits
Test plan
python -m pytest-> 228 passedpython benchmarks/harness.py-> 0/13, 0/8; agent path 0/8 (control 6/8)examples/quickstart.pyruns end to endtests/test_failure_ledger.pycontract tests pass (45 entries, 21 limitations)Evidence:
evidence/second-party-review-2026-09-19.md(review, verbatim, with pre-repair verification) andevidence/variant-hunt-2026-09-21.md(method, emission table, negative controls).🤖 Generated with Claude Code
Update: faith is a static damper with a tamper tripwire (NOE-F-046)
Design review by the repository owner showed that the repair for the review's "faith is decorative" finding was wrong. Faith is not something a node earns; it is a system-set damper on grief intake that holds the system a controlled distance from a cascade. Making it earnable let relief be bought with signals an agent can influence.
TrustGate.faith_for()is the only source of faith (operatorbase_faith; guardrails 0.92). The stored value is never used for relief.GriefCascade._faith_tripwireforce-purges a tampered node through the normal cascade path (no faith resistance, no seppuku criteria), notifies its dependents, and logs totamper_log. Sacred nodes are logged and left in place.GriefCascade.cusp()is a read-only readout of distance from a cascade; the benign benchmark reports it. Today it reads 0.900, so the operating point is untuned (NOE-L-023).Tests: 241 passing. Negative controls: grief_cascade reverted -> 9 fail, trust_gate reverted -> 15 fail.
Update: first cloud ultrareview (NOE-F-047 to 049)
The ultrareview of this PR returned two findings, both labeled pre-existing in
masterand both verified against source before any change:write()for every replacement, and a replacement that would be quarantined is refused so the published memory survives. (NOE-F-047)promote_skillreported success whenwrite()stored a candidate.write()returnsTruefor "stored" as well as "published".promote_skillnow refuses up front for an identity withoutpublish_memoryand verifies the skill is retrievable afterwards;process_patternsandend_sessionno longer ignore refused writes. (NOE-F-048)Hunting those classes found a third route, confirmed by probe before repair: republishing an identical value reset grief to 0 (laundering the circuit breaker), and republishing a purged key revived it as ACTIVE. Both are fixed (NOE-F-049).
Tests: 257 passing. Each changed module reverted alone fails its tests (store 13, forge 3, gateway 1). The report is preserved as received in
evidence/ultrareview-1-2026-09-21.md; it is classed first-party because the reviewer's independence is not established.Update:
write()can no longer be misread as "published" (NOE-F-050, resolves NOE-L-024)MemoryStore.write()returned(True, reason)both for a node that went ACTIVE and for one held as a candidate or quarantined. That ambiguity caused thepromote_skilldefect (NOE-F-048); fixing the callers left the trap in the API.WriteResultcarries an explicit outcome (PUBLISHED,CANDIDATE,QUARANTINED,REFUSED) with.storedand.published.if store.write(...)andok, why = store.write(...)raise instead of guessing. A caller must say which they mean.write,write_fact,write_episode,write_profile,write_project_state, and the gateway'slearn_fact,set_profile,set_project_state.promote_candidate,release_quarantinedandwrite_guardrailare unchanged (Truethere means the effect happened).63 existing call sites converted. Tests: 276 passing. Reverting each changed module fails its tests.