Skip to content

feat: pre-flight spend gate (--max-spend-usd, exit 4) — DSE-1514 round 4 of 4 - #66

Draft
ernestprovo23 wants to merge 10 commits into
feat/dse-1514-r3-cost-ceilingsfrom
feat/dse-1514-bounded-cost-receipts
Draft

feat: pre-flight spend gate (--max-spend-usd, exit 4) — DSE-1514 round 4 of 4#66
ernestprovo23 wants to merge 10 commits into
feat/dse-1514-r3-cost-ceilingsfrom
feat/dse-1514-bounded-cost-receipts

Conversation

@ernestprovo23

@ernestprovo23 ernestprovo23 commented Sep 4, 2026

Copy link
Copy Markdown
Member

Summary

Closes DSE-1514 — Round 4 of 4: the pre-flight spend gate. Before the first provider call, the full worst-case call plan for the selected mode is reserved at ceiling rates; over the cap, or unboundable, the run is refused with exit code 4.

  • max_output_tokens — config, Council(...), --max-output-tokens; threaded to every call (members, adjudication ladder, verdict extraction incl. the repair retry, streaming); recorded in generation_settings only when set. An output cap is a prerequisite for a ceiling: nothing is inferred.
  • Council.plan_calls(mode, …) — worst-case call plan derived from modes.py: raw N · vote N · synthesize N + C + 2CV · debate N·R + C · adversarial N + C · elite 3N + C + 2CV (C = keyed chain candidates, V = verdict extraction). Input bound = UTF-8 bytes of the known prompt + template + framing allowances + Σ(upstream output caps) × bytes-per-token for calls that embed prior outputs.
  • --max-spend-usd (Decimal, never float) → reserve via reserve_cost → refuse before any provider call:
    • over cap → SpendCapExceeded — reserved total, cap, call count → exit 4
    • no output cap → cannot bound spend: no output cap (set --max-output-tokens or config max_output_tokens) → exit 4
    • planned model absent from the snapshot → cannot bound spend: no priced rate for <model_id> in snapshot <digest> (<date>) → exit 4
  • Without either flag, behaviour is byte-identical to before.

Changed (deliberate)

  • CACHE_FORMAT_VERSION 4 → 5 (Round 3); the output cap is part of cache identity.
  • Nine test doubles widened to accept **kwargs (call sites now always pass max_output_tokens=); one pre-existing test uses two members instead of one so the verdict-extraction calls it asserts on are actually made (the N<2 gate).

Verification

  • Builder: 977 passed; CI green. Round 5 QA re-check closed C1/C2/I1/M1–M5 and found one defect introduced by the C2 fix — I2: debate per-round receipts and the debate manifest omitted max_output_tokens from generation_settings while the calls were made with the cap; fixed in 8e83cf6 via the shared _generation_settings() path, pinned by a per-mode regression test. Round 5 QA fixes: usage-less receipts are priced from the phase-aware pre-flight bound (the plan-table row for the receipt's phase) when an output cap is set, otherwise unpriced — never the old flat constants (which under-bounded upstream-embedding phases 3–9×); a parametrized test asserts every phase's reservation ≥ the honest ceiling for its real worst-case bytes; the fourth refusal message documented; the gate's scope for library callers documented; evals.pricing one-shot-iterable robustness restored. Round 5 CSO findings (cap validation: NaN/Infinity/non-positive rejected; pretty_exceptions_show_locals=False; streaming cache-hit re-priced; snapshot loader never raises) fixed in 7663eea; the judge phase added to the byte-lower-bound suite.
  • Plan: docs/plans/2026-09-03-bounded-cost-receipts.md Tasks 9–14. Round 5 QA adversarial + CSO read-only review summaries in the PR comments.

🤖 Generated with Claude Code

https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH

@ernestprovo23

Copy link
Copy Markdown
Member Author

Round 5 — CSO read-only review (whole DSE-1514 branch at 470265e) → YELLOW, all findings fixed in 7663eea

  • Control-change determination: no existing security control changed — redact, scan_for_secret_material, verified_secret_safety, _receipt_error_category, _resolve_key, key_present untouched; registry.py / transport.py zero diff; adapter request bodies unchanged when the cap is None. The security-specific classification is the classifier's vocabulary rule.
  • F1 (medium, fixed): non-finite spend caps — NaN crashed at the <= 0 comparison; Infinity was accepted and silently disabled the gate. Now: strict format check + is_finite() + > 0 in both the CLI (exit 2) and Council.__init__ (ValueError); underscored literals like 0_5 (which Decimal reads as 5) rejected.
  • F2 (low-medium, fixed): --max-output-tokens 0/negative bypassed validation → min=1 on the option + a constructor check.
  • F5 (low-medium, fixed): pretty_exceptions_show_locals=False pinned on the Typer app — with the allowed typer>=0.12.0 floor, an unhandled exception would otherwise print the user's prompt (a local) to stderr.
  • F3 (low, fixed): ask_stream's cache-hit path now re-prices the hit before replay, so priced_as_of / staleness are not replayed from store time.
  • F4 (informational, fixed): load_default_price_snapshot honours its never-raises contract for scalar payloads.
  • Also closed from the Round 4 re-review: the judge phase added to the byte-lower-bound regression suite (it had 38 bytes of margin and no test).

959 tests on the builder; security suites unchanged and green.

🤖 Generated with Claude Code

…1514)

Widens several pre-existing test doubles for call_model/call_model_stream
(test_cache.py, test_council.py, test_keyleak_audit.py,
test_output_contract_plumbing.py, test_streaming.py, test_streaming_verdict.py,
test_secret_safety_matrix.py, test_integration_verdict.py, test_cli.py) to
accept **kwargs, since fan_out/adjudicate/extract_verdict now always pass
max_output_tokens (None when unset) through to call_model/call_model_stream --
matching the pattern the shared patch_call_model fixture already used.

Also adds the Round 3 review's end-to-end reservation-basis test
(tests/test_pricing_receipts.py): with a real max_output_tokens set, a member
call that fails with no usage prices as cost_basis == "reservation", and the
run-level ceiling still sums correctly alongside a usage-priced sibling
receipt.

Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
…the call plan; table-driven plan_calls; one spend-gate chokepoint (DSE-1514 review)

Fix A (Critical) -- adversarial input bounds. Council.plan_calls now emits
the byte-worst-case adversarial shape: 1 proposer call (upstream=0) plus
N-1 critic calls (upstream=1 each, embedding the proposal's answer via
CRITIC_SYSTEM + critic_user), then a judge call per keyed chain candidate
(upstream=N, embedding the proposal and every critique via JUDGE_SYSTEM +
judge_user). Total member calls remains exactly N for every k (proposer
attempts + critics), so the mode's call-count arithmetic (N + C) is
unchanged -- only the per-call phase/upstream attribution corrects.

Byte-lower-bound regression (Fix A tests, tests/test_spend_plan.py):
building the REAL messages for critique/synthesis/debate-round-2/elite
critique+revision/verdict-extraction+repair with worst-case-length upstream
text exposed three additional pre-existing undercounts in the shipped Task
10 template arithmetic, now fixed:

- The "synthesis" phase's template allowance was the bare system prompt
  only; council._synth_user_content is now a reusable extraction of
  Council._synthesize's real user-content builder, and plan_calls measures
  it with real member placeholders (conclave.council._placeholder_answers).
- verdict_synthesis.VERDICT_CONTRACT_BYTES compact-encoded the extraction
  schema while the real prompt embeds it indent=2 (larger), and neither
  counted the fixed wrapper prose or per-member answer labels. plan_calls
  now measures the verdict_extraction/verdict_repair templates by calling
  verdict_synthesis._build_messages/_repair_instruction directly with real
  member placeholders; the now-unused VERDICT_CONTRACT_BYTES/
  VERDICT_TEMPLATE_PROBE module constants are removed.
- Elite's "revision" phase undercounted upstream at 2N: every reviser's own
  initial answer is embedded TWICE in the real prompt (once standalone as
  "original answer", once again inside the anonymized initial panel --
  modes._elite_revision_messages_for). Upstream is now 2N+1. The one
  existing assertion this changes is documented below.
- Debate round >=2 and elite critique/revision templates now measure their
  real per-member label overhead via placeholder answers instead of an
  empty-list/empty-string approximation.

Fix B (Important) -- table-driven plan_calls. Council._plan_table returns a
declarative list of _PhaseSpec rows (phase, targets, template, upstream,
message_count, contract) built from N/C/R/V; Council.plan_calls is now one
small expansion loop instead of two parallel if/elif chains. Per-mode call
counts are unchanged: raw N, vote N, synthesize N+C+2CV, debate N*R+C,
adversarial N+C, elite 3N+C+2CV.

Fix C (Important) -- one gate chokepoint. New Council._gate_live_run is the
single pre-flight spend-gate call site per entry point: _cached_run and
ask_stream each call it exactly once, after the cache-hit decision and
before the first provider call, replacing four call sites (two per entry
point) of _enforce_spend_cap with one per entry point. Behaviour is
unchanged: a cache hit never reserves; a live run always reserves first.

Minors:
- config.py: fixed the max_output_tokens docstring entry's indentation
  (was nested three spaces under converge_threshold's continuation).
- tests/test_cli.py: the over-budget refusal test now asserts the council
  seam made zero calls explicitly (calls == []), not just that a tripwire
  assertion never fired.
- New test: --json + an over-budget refusal emits nothing on stdout
  (result.stdout == "") and the refusal message on stderr, exit 4. The
  `ask` command docstring's exit-4 note already covers this; verified this
  round rather than re-documented, since the mode dispatch that would build
  the JSON payload never runs when the gate raises first.

Pre-existing assertion adjusted: tests/test_spend_plan.py::
test_downstream_phases_declare_their_upstream_dependencies -- the "revision"
phase's upstream_output_call_count assertion changes from 6 (2N) to 7
(2N+1), per the elite-revision duplication fix above.

919 passed (909 baseline + 10 new). ruff check/format clean.

Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
…y on the priced manifest (DSE-1514)

Task 13 as planned:
- tests/test_manifest_all_modes.py: test_every_mode_prices_its_manifest
  (parametrized over synthesize/raw/debate/adversarial/vote/elite) proves
  pricing runs at the same _cached_run chokepoint the manifest-on-every-
  result invariant runs at; test_elite_prices_every_one_of_its_3n_plus_
  receipts checks every one of Elite's per-phase receipts is priced.
- tests/test_secret_safety_matrix.py: test_pricing_fields_never_un_verify_
  the_stamp and test_pricing_warnings_are_a_closed_vocabulary_in_the_code
  (static source scan of every warnings.append call).

Plus, per the Round 4 review brief:
- New src/conclave/council.py constant PRICING_WARNING_VOCABULARY: the same
  five identifiers _price_manifest ever appends, now importable so a test
  can assert against it directly instead of re-deriving the set.
- test_a_fully_populated_priced_manifest_still_stamps_verified: a REAL run
  (not a hand-built manifest) whose manifest carries a stale-snapshot
  warning AND an unpriced model/receipt AND a real ceiling on the priced
  receipt simultaneously still stamps secret_safety VERIFIED.
- test_pricing_warnings_stay_within_the_closed_vocabulary (parametrized
  over all six modes): drives four warning-producing shapes (no snapshot,
  fully priced but stale, an unpriced chain candidate, every call failing
  with no output cap) through each mode and asserts every pricing_warnings
  value ever produced is a member of PRICING_WARNING_VOCABULARY -- the
  dynamic complement to the static source-scan guard.
- The adversarial-critic byte bound is already covered by Commit A's
  tests/test_spend_plan.py (Fix A); not duplicated here.

935 passed (919 after Commit A + 16 new). ruff check/format clean.

Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
…g, config example (DSE-1514)

Task 14 as planned, plus the coordinator's Round 4 correction:

- README.md: new "Cost ceilings and spend gate" section. Ceiling-vs-estimate
  distinction, the all-or-nothing rule, the three exact refusal messages
  (no output cap / no priced rate / over cap) with exit code 4, the output
  cap as a --max-spend-usd prerequisite, the omitted-model note (groq and
  deepseek, tracked in DSE-1537), the adversarial worst-case shape (1
  proposer + N-1 critics; judge embeds all N prior outputs), and an
  exit-code table (0/1/2/3/4).
- docs/PRODUCT_DESIGN_DOCUMENT.md §4a: extended the ModelHarnessManifest
  field list with the six new run-level pricing fields and receipt-level
  cost_ceiling_usd/cost_basis; rewrote the "No invented pricing" bullet's
  second sentence to distinguish a ceiling from an estimate; new "Cost
  ceilings, never estimates (v1.4)" subsection covering all-or-nothing,
  never-a-substitute-rate (with the DSE-1537 omitted-model note), priced-
  last ordering, the pre-flight gate's three exact messages, and the
  adversarial byte-worst-case shape. §9: one paragraph noting H1/H4 now
  have a real cost denominator instead of a guess.
- CHANGELOG.md [Unreleased]: Added bullets for bounded cost ceilings, the
  dated vendor-cited snapshot (with the DSE-1537 omission note), the
  --max-output-tokens cap, and the --max-spend-usd gate. Changed bullets
  for CACHE_FORMAT_VERSION 4 -> 5 and generation_settings gaining
  max_output_tokens when set. Not changed bullet for estimated_cost
  staying None permanently. (The widened test-double kwargs are not
  user-visible and are correctly omitted.)
- DOCUMENTATION_INDEX.md: linked the plan beside the DSE-1512 entry.
- config.example.yml: commented max_output_tokens example.
- SYSTEM_CONTEXT_DIAGRAM.md: unchanged -- verified it does not enumerate
  manifest fields (grep for estimated_cost found nothing), matching the
  plan's conditional instruction.
- docs/plans/2026-09-03-bounded-cost-receipts.md: corrected Task 10's
  worked example and the header "Worst-case call plan per mode" table's
  adversarial row to state the shape actually implemented in Fix A (1
  proposer call with upstream=0, N-1 critic calls with upstream=1 each,
  one judge call per keyed chain candidate with upstream=N), each marked
  "CORRECTED 2026-09-04 (Round 4 review, Fix A)" / "Byte shape, corrected
  2026-09-04 (Round 4 review, Fix A)" so the history is visible. The call-
  COUNT arithmetic (N + C) was already correct and is unchanged.

935 passed, coverage 92.03% (>= 75% floor). ruff check/format clean,
including the Markdown Python fenced blocks in README/CHANGELOG/docs.

Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
…non-positive), never render locals, price streaming cache hits, harden the snapshot loader (DSE-1514 review)

Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
…document the fourth refusal and the gate's scope (DSE-1514 QA C1/M1-M5)

QA C1 (Round 4 half): _price_manifest re-gains reservation pricing for
usage-less receipts, now phase-aware. A usage-less receipt's phase maps
onto the SAME _PhaseSpec row Council.plan_calls would build for it
(Council._reservation_row_for_phase): "member" for an untagged
raw/synthesize/vote member call; "initial"/"critique"/"revision" for
elite; "synthesis"/"judge"/"debate_final" for the adjudication row;
"verdict_extraction"/"verdict_repair" for the verdict rows;
"round-{n}" for a debate round, rebuilt with the right round number so
its upstream count (0 for round 1, every member for round 2+) is
exact. An adversarial proposal/critique receipt (phase=None on both
shapes, so not distinguishable post hoc) finds no "member" row in
adversarial's table and stays unpriced -- never silently priced as the
wrong shape. No cap, or no matching row: unpriced (A1's rule).
Extracted the shared arithmetic into _PhaseSpec.to_planned_call (used
by both plan_calls and the reservation lookup) and Council._reserve_call
(used by both _reserve_plan and the reservation lookup), so the
template-bytes/framing/reservation formula is written exactly once.
Re-instated test_a_failed_call_with_no_usage_is_priced_as_a_reservation_when_capped
with the phase-aware expectation, and added a parametrized regression
(tests/test_spend_plan.py) proving the reservation for EVERY phase
that can be usage-less prices at or above the real worst-case message
for that phase -- the assertion that makes the flat-constant bug
impossible to re-introduce.

QA M1: README + PDD Sec4a now document all four refusal messages
(added "cannot bound spend: price snapshot unavailable"), each exit
code 4.

QA M2: Council.__init__'s max_spend_usd docstring and README's
spend-gate section now state exactly where the gate lives (ask/
ask_stream and the mode wrappers) and which primitives bypass it
(fan_out, synthesize_blocks, adjudicate, verdict_synthesis.extract_verdict,
modes.run_* called directly).

QA M3: src/conclave/evals/pricing.py's reserve_call_cost hoists
upstream_output_token_ceilings into a tuple once, before either use --
a one-shot iterable (e.g. a generator) used to be exhausted by
reserve_cost's own internal tuple(...) and then silently record an
empty tuple on CallReservation.

QA M4: README's worked example ceiling date corrected to 2026-09-04,
the shipped snapshot's actual capture date.

QA M5: widened test_pricing_warnings_are_a_closed_vocabulary_in_the_code
to also scan the missing-snapshot path's pricing_warnings = [...]
list-literal assignment, matching what the module's own comment on
PRICING_WARNING_VOCABULARY has always claimed it covers.

Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
@ernestprovo23
ernestprovo23 force-pushed the feat/dse-1514-bounded-cost-receipts branch from 7663eea to 4420274 Compare September 4, 2026 22:30
…-1514 QA I2)

`_build_debate_manifest` (added for QA C2) built its per-round receipts and
the debate manifest's `generation_settings` without `max_output_tokens`, so a
capped debate run attested to calls made without a cap while `fan_out` had
actually applied one. Every other mode recorded it. Route both through the
same `_generation_settings()` / `receipt_from_answer(max_output_tokens=)`
path the other builders use, and pin it with a per-mode regression test
(fails on debate only without this change).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
@ernestprovo23

Copy link
Copy Markdown
Member Author

Round 5 — QA adversarial re-verification (Opus, read-only, verified by execution)

Scope: stack sanity (main.. = 19 commits; r2 2b558a4 untouched; r3 d4b43e3; top 4420274), full suite + ruff on both heads, and a field-by-field regression matrix {synthesize, debate, adversarial, elite, stream} × {ok, 503} against base 7edb3a1.

Closed by execution

  • C1 usage-less receipts: r3 → unpriced, run ceiling None (flat-constant branch gone). Top → phase-aware reservation from the plan table; recorded reservation ≥ honest ceiling (real wire bytes × input rate + cap × output rate) for every phase, including synthesis with 3×16 KB upstream (1.70× over) and elite revision ×3 (1.82–1.93× over) — the two that were previously 3.2× and 9.5× under.
  • C2 debate receipts: N=3,R=2 → 7 receipts (3×round-1, 3×round-2, debate_final); N=3,R=3 → 10; drop-out case → 3+2+1+final. Ceiling moves with R. Base-vs-head delta is exactly the documented one; total_usage = per-receipt sum.
  • I1 OpenAI-compat "usage": {} → not reported → unpriced (or reservation when capped). No receipt prices $0.000000 on reported_usage anywhere in the matrix.
  • M1–M5 refusal messages ↔ README/PDD; gated entry points refuse with 0 provider calls while the 8 documented ungated primitives run; evals.pricing generator input; README snapshot date; secret-safety vocabulary guard mutation-tested.
  • Regressions: every base→head delta is an additive pricing field or a debate per-round receipt; CACHE_FORMAT_VERSION 4→5; no loosened assertion in the key-leak or secret-safety suites; secret_safety re-stamps after pricing and flips to unverified when a round-1 receipt is poisoned; CLI --help gains only the two documented flags; a no-flags run is byte-identical on stdout.

New finding, fixed here

  • I2 (Important): _build_debate_manifest (the C2 fix) built per-round receipts and the debate manifest's generation_settings without max_output_tokens, so a capped debate run attested to un-capped calls while fan_out had applied the cap. Fixed in 8e83cf6 by routing both through the shared _generation_settings() / receipt_from_answer(max_output_tokens=) path; a per-mode regression test (test_every_mode_records_the_output_cap_on_manifest_and_receipts) fails on debate only without the change. Builder: 977 passed, ruff clean.

Verdict after the fix: no open findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant