feat: pre-flight spend gate (--max-spend-usd, exit 4) — DSE-1514 round 4 of 4 - #66
Draft
ernestprovo23 wants to merge 10 commits into
Draft
feat: pre-flight spend gate (--max-spend-usd, exit 4) — DSE-1514 round 4 of 4#66ernestprovo23 wants to merge 10 commits into
ernestprovo23 wants to merge 10 commits into
Conversation
Member
Author
Round 5 — CSO read-only review (whole DSE-1514 branch at
|
…1514) Widens several pre-existing test doubles for call_model/call_model_stream (test_cache.py, test_council.py, test_keyleak_audit.py, test_output_contract_plumbing.py, test_streaming.py, test_streaming_verdict.py, test_secret_safety_matrix.py, test_integration_verdict.py, test_cli.py) to accept **kwargs, since fan_out/adjudicate/extract_verdict now always pass max_output_tokens (None when unset) through to call_model/call_model_stream -- matching the pattern the shared patch_call_model fixture already used. Also adds the Round 3 review's end-to-end reservation-basis test (tests/test_pricing_receipts.py): with a real max_output_tokens set, a member call that fails with no usage prices as cost_basis == "reservation", and the run-level ceiling still sums correctly alongside a usage-priced sibling receipt. Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
…r call (DSE-1514) Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
…the call plan; table-driven plan_calls; one spend-gate chokepoint (DSE-1514 review) Fix A (Critical) -- adversarial input bounds. Council.plan_calls now emits the byte-worst-case adversarial shape: 1 proposer call (upstream=0) plus N-1 critic calls (upstream=1 each, embedding the proposal's answer via CRITIC_SYSTEM + critic_user), then a judge call per keyed chain candidate (upstream=N, embedding the proposal and every critique via JUDGE_SYSTEM + judge_user). Total member calls remains exactly N for every k (proposer attempts + critics), so the mode's call-count arithmetic (N + C) is unchanged -- only the per-call phase/upstream attribution corrects. Byte-lower-bound regression (Fix A tests, tests/test_spend_plan.py): building the REAL messages for critique/synthesis/debate-round-2/elite critique+revision/verdict-extraction+repair with worst-case-length upstream text exposed three additional pre-existing undercounts in the shipped Task 10 template arithmetic, now fixed: - The "synthesis" phase's template allowance was the bare system prompt only; council._synth_user_content is now a reusable extraction of Council._synthesize's real user-content builder, and plan_calls measures it with real member placeholders (conclave.council._placeholder_answers). - verdict_synthesis.VERDICT_CONTRACT_BYTES compact-encoded the extraction schema while the real prompt embeds it indent=2 (larger), and neither counted the fixed wrapper prose or per-member answer labels. plan_calls now measures the verdict_extraction/verdict_repair templates by calling verdict_synthesis._build_messages/_repair_instruction directly with real member placeholders; the now-unused VERDICT_CONTRACT_BYTES/ VERDICT_TEMPLATE_PROBE module constants are removed. - Elite's "revision" phase undercounted upstream at 2N: every reviser's own initial answer is embedded TWICE in the real prompt (once standalone as "original answer", once again inside the anonymized initial panel -- modes._elite_revision_messages_for). Upstream is now 2N+1. The one existing assertion this changes is documented below. - Debate round >=2 and elite critique/revision templates now measure their real per-member label overhead via placeholder answers instead of an empty-list/empty-string approximation. Fix B (Important) -- table-driven plan_calls. Council._plan_table returns a declarative list of _PhaseSpec rows (phase, targets, template, upstream, message_count, contract) built from N/C/R/V; Council.plan_calls is now one small expansion loop instead of two parallel if/elif chains. Per-mode call counts are unchanged: raw N, vote N, synthesize N+C+2CV, debate N*R+C, adversarial N+C, elite 3N+C+2CV. Fix C (Important) -- one gate chokepoint. New Council._gate_live_run is the single pre-flight spend-gate call site per entry point: _cached_run and ask_stream each call it exactly once, after the cache-hit decision and before the first provider call, replacing four call sites (two per entry point) of _enforce_spend_cap with one per entry point. Behaviour is unchanged: a cache hit never reserves; a live run always reserves first. Minors: - config.py: fixed the max_output_tokens docstring entry's indentation (was nested three spaces under converge_threshold's continuation). - tests/test_cli.py: the over-budget refusal test now asserts the council seam made zero calls explicitly (calls == []), not just that a tripwire assertion never fired. - New test: --json + an over-budget refusal emits nothing on stdout (result.stdout == "") and the refusal message on stderr, exit 4. The `ask` command docstring's exit-4 note already covers this; verified this round rather than re-documented, since the mode dispatch that would build the JSON payload never runs when the gate raises first. Pre-existing assertion adjusted: tests/test_spend_plan.py:: test_downstream_phases_declare_their_upstream_dependencies -- the "revision" phase's upstream_output_call_count assertion changes from 6 (2N) to 7 (2N+1), per the elite-revision duplication fix above. 919 passed (909 baseline + 10 new). ruff check/format clean. Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
…y on the priced manifest (DSE-1514) Task 13 as planned: - tests/test_manifest_all_modes.py: test_every_mode_prices_its_manifest (parametrized over synthesize/raw/debate/adversarial/vote/elite) proves pricing runs at the same _cached_run chokepoint the manifest-on-every- result invariant runs at; test_elite_prices_every_one_of_its_3n_plus_ receipts checks every one of Elite's per-phase receipts is priced. - tests/test_secret_safety_matrix.py: test_pricing_fields_never_un_verify_ the_stamp and test_pricing_warnings_are_a_closed_vocabulary_in_the_code (static source scan of every warnings.append call). Plus, per the Round 4 review brief: - New src/conclave/council.py constant PRICING_WARNING_VOCABULARY: the same five identifiers _price_manifest ever appends, now importable so a test can assert against it directly instead of re-deriving the set. - test_a_fully_populated_priced_manifest_still_stamps_verified: a REAL run (not a hand-built manifest) whose manifest carries a stale-snapshot warning AND an unpriced model/receipt AND a real ceiling on the priced receipt simultaneously still stamps secret_safety VERIFIED. - test_pricing_warnings_stay_within_the_closed_vocabulary (parametrized over all six modes): drives four warning-producing shapes (no snapshot, fully priced but stale, an unpriced chain candidate, every call failing with no output cap) through each mode and asserts every pricing_warnings value ever produced is a member of PRICING_WARNING_VOCABULARY -- the dynamic complement to the static source-scan guard. - The adversarial-critic byte bound is already covered by Commit A's tests/test_spend_plan.py (Fix A); not duplicated here. 935 passed (919 after Commit A + 16 new). ruff check/format clean. Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
…g, config example (DSE-1514) Task 14 as planned, plus the coordinator's Round 4 correction: - README.md: new "Cost ceilings and spend gate" section. Ceiling-vs-estimate distinction, the all-or-nothing rule, the three exact refusal messages (no output cap / no priced rate / over cap) with exit code 4, the output cap as a --max-spend-usd prerequisite, the omitted-model note (groq and deepseek, tracked in DSE-1537), the adversarial worst-case shape (1 proposer + N-1 critics; judge embeds all N prior outputs), and an exit-code table (0/1/2/3/4). - docs/PRODUCT_DESIGN_DOCUMENT.md §4a: extended the ModelHarnessManifest field list with the six new run-level pricing fields and receipt-level cost_ceiling_usd/cost_basis; rewrote the "No invented pricing" bullet's second sentence to distinguish a ceiling from an estimate; new "Cost ceilings, never estimates (v1.4)" subsection covering all-or-nothing, never-a-substitute-rate (with the DSE-1537 omitted-model note), priced- last ordering, the pre-flight gate's three exact messages, and the adversarial byte-worst-case shape. §9: one paragraph noting H1/H4 now have a real cost denominator instead of a guess. - CHANGELOG.md [Unreleased]: Added bullets for bounded cost ceilings, the dated vendor-cited snapshot (with the DSE-1537 omission note), the --max-output-tokens cap, and the --max-spend-usd gate. Changed bullets for CACHE_FORMAT_VERSION 4 -> 5 and generation_settings gaining max_output_tokens when set. Not changed bullet for estimated_cost staying None permanently. (The widened test-double kwargs are not user-visible and are correctly omitted.) - DOCUMENTATION_INDEX.md: linked the plan beside the DSE-1512 entry. - config.example.yml: commented max_output_tokens example. - SYSTEM_CONTEXT_DIAGRAM.md: unchanged -- verified it does not enumerate manifest fields (grep for estimated_cost found nothing), matching the plan's conditional instruction. - docs/plans/2026-09-03-bounded-cost-receipts.md: corrected Task 10's worked example and the header "Worst-case call plan per mode" table's adversarial row to state the shape actually implemented in Fix A (1 proposer call with upstream=0, N-1 critic calls with upstream=1 each, one judge call per keyed chain candidate with upstream=N), each marked "CORRECTED 2026-09-04 (Round 4 review, Fix A)" / "Byte shape, corrected 2026-09-04 (Round 4 review, Fix A)" so the history is visible. The call- COUNT arithmetic (N + C) was already correct and is unchanged. 935 passed, coverage 92.03% (>= 75% floor). ruff check/format clean, including the Markdown Python fenced blocks in README/CHANGELOG/docs. Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
…non-positive), never render locals, price streaming cache hits, harden the snapshot loader (DSE-1514 review) Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
…document the fourth refusal and the gate's scope (DSE-1514 QA C1/M1-M5)
QA C1 (Round 4 half): _price_manifest re-gains reservation pricing for
usage-less receipts, now phase-aware. A usage-less receipt's phase maps
onto the SAME _PhaseSpec row Council.plan_calls would build for it
(Council._reservation_row_for_phase): "member" for an untagged
raw/synthesize/vote member call; "initial"/"critique"/"revision" for
elite; "synthesis"/"judge"/"debate_final" for the adjudication row;
"verdict_extraction"/"verdict_repair" for the verdict rows;
"round-{n}" for a debate round, rebuilt with the right round number so
its upstream count (0 for round 1, every member for round 2+) is
exact. An adversarial proposal/critique receipt (phase=None on both
shapes, so not distinguishable post hoc) finds no "member" row in
adversarial's table and stays unpriced -- never silently priced as the
wrong shape. No cap, or no matching row: unpriced (A1's rule).
Extracted the shared arithmetic into _PhaseSpec.to_planned_call (used
by both plan_calls and the reservation lookup) and Council._reserve_call
(used by both _reserve_plan and the reservation lookup), so the
template-bytes/framing/reservation formula is written exactly once.
Re-instated test_a_failed_call_with_no_usage_is_priced_as_a_reservation_when_capped
with the phase-aware expectation, and added a parametrized regression
(tests/test_spend_plan.py) proving the reservation for EVERY phase
that can be usage-less prices at or above the real worst-case message
for that phase -- the assertion that makes the flat-constant bug
impossible to re-introduce.
QA M1: README + PDD Sec4a now document all four refusal messages
(added "cannot bound spend: price snapshot unavailable"), each exit
code 4.
QA M2: Council.__init__'s max_spend_usd docstring and README's
spend-gate section now state exactly where the gate lives (ask/
ask_stream and the mode wrappers) and which primitives bypass it
(fan_out, synthesize_blocks, adjudicate, verdict_synthesis.extract_verdict,
modes.run_* called directly).
QA M3: src/conclave/evals/pricing.py's reserve_call_cost hoists
upstream_output_token_ceilings into a tuple once, before either use --
a one-shot iterable (e.g. a generator) used to be exhausted by
reserve_cost's own internal tuple(...) and then silently record an
empty tuple on CallReservation.
QA M4: README's worked example ceiling date corrected to 2026-09-04,
the shipped snapshot's actual capture date.
QA M5: widened test_pricing_warnings_are_a_closed_vocabulary_in_the_code
to also scan the missing-snapshot path's pricing_warnings = [...]
list-literal assignment, matching what the module's own comment on
PRICING_WARNING_VOCABULARY has always claimed it covers.
Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
ernestprovo23
force-pushed
the
feat/dse-1514-bounded-cost-receipts
branch
from
September 4, 2026 22:30
7663eea to
4420274
Compare
…-1514 QA I2) `_build_debate_manifest` (added for QA C2) built its per-round receipts and the debate manifest's `generation_settings` without `max_output_tokens`, so a capped debate run attested to calls made without a cap while `fan_out` had actually applied one. Every other mode recorded it. Route both through the same `_generation_settings()` / `receipt_from_answer(max_output_tokens=)` path the other builders use, and pin it with a per-mode regression test (fails on debate only without this change). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
Member
Author
Round 5 — QA adversarial re-verification (Opus, read-only, verified by execution)Scope: stack sanity ( Closed by execution
New finding, fixed here
Verdict after the fix: no open findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes DSE-1514 — Round 4 of 4: the pre-flight spend gate. Before the first provider call, the full worst-case call plan for the selected mode is reserved at ceiling rates; over the cap, or unboundable, the run is refused with exit code 4.
max_output_tokens— config,Council(...),--max-output-tokens; threaded to every call (members, adjudication ladder, verdict extraction incl. the repair retry, streaming); recorded ingeneration_settingsonly when set. An output cap is a prerequisite for a ceiling: nothing is inferred.Council.plan_calls(mode, …)— worst-case call plan derived frommodes.py: rawN· voteN· synthesizeN + C + 2CV· debateN·R + C· adversarialN + C· elite3N + C + 2CV(C = keyed chain candidates, V = verdict extraction). Input bound = UTF-8 bytes of the known prompt + template + framing allowances + Σ(upstream output caps) × bytes-per-token for calls that embed prior outputs.--max-spend-usd(Decimal, never float) → reserve viareserve_cost→ refuse before any provider call:SpendCapExceeded— reserved total, cap, call count → exit 4cannot bound spend: no output cap (set --max-output-tokens or config max_output_tokens)→ exit 4cannot bound spend: no priced rate for <model_id> in snapshot <digest> (<date>)→ exit 4Changed (deliberate)
CACHE_FORMAT_VERSION4 → 5 (Round 3); the output cap is part of cache identity.**kwargs(call sites now always passmax_output_tokens=); one pre-existing test uses two members instead of one so the verdict-extraction calls it asserts on are actually made (the N<2 gate).Verification
max_output_tokensfromgeneration_settingswhile the calls were made with the cap; fixed in8e83cf6via the shared_generation_settings()path, pinned by a per-mode regression test. Round 5 QA fixes: usage-less receipts are priced from the phase-aware pre-flight bound (the plan-table row for the receipt's phase) when an output cap is set, otherwise unpriced — never the old flat constants (which under-bounded upstream-embedding phases 3–9×); a parametrized test asserts every phase's reservation ≥ the honest ceiling for its real worst-case bytes; the fourth refusal message documented; the gate's scope for library callers documented;evals.pricingone-shot-iterable robustness restored. Round 5 CSO findings (cap validation: NaN/Infinity/non-positive rejected;pretty_exceptions_show_locals=False; streaming cache-hit re-priced; snapshot loader never raises) fixed in7663eea; thejudgephase added to the byte-lower-bound suite.docs/plans/2026-09-03-bounded-cost-receipts.mdTasks 9–14. Round 5 QA adversarial + CSO read-only review summaries in the PR comments.🤖 Generated with Claude Code
https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH