Skip to content

feat: cost ceilings on receipts from a dated, vendor-cited price snapshot — DSE-1514 round 3 of 4 - #65

Draft
ernestprovo23 wants to merge 6 commits into
feat/dse-1514-r2-pricing-modulefrom
feat/dse-1514-r3-cost-ceilings
Draft

feat: cost ceilings on receipts from a dated, vendor-cited price snapshot — DSE-1514 round 3 of 4#65
ernestprovo23 wants to merge 6 commits into
feat/dse-1514-r2-pricing-modulefrom
feat/dse-1514-r3-cost-ceilings

Conversation

@ernestprovo23

@ernestprovo23 ernestprovo23 commented Sep 4, 2026

Copy link
Copy Markdown
Member

Summary

Part of DSE-1514 (bounded cost receipts) — Round 3 of 4: cost ceilings on receipts and the run manifest, from a dated, vendor-cited price snapshot. Builds on the Round 2 refactor (conclave.pricing).

Ceilings, never estimates. estimated_cost stays None forever. Every number this PR emits is a falsifiable upper bound: reported usage × the snapshot's ceiling rate, Decimal only, ROUND_CEILING, quantized to a micro-cent, stamped with the snapshot digest and date.

  • PriceSnapshot (frozen, Decimal-only rates, source_url per entry, digest namespace conclave_product_prices_v1, staleness threshold 90 days).
  • src/conclave/data/prices-2026-09-04.json + load_default_price_snapshot() (shipped in the wheel). 7 of 9 default models priced from official vendor pages, rounded up, sources cited; groq/llama-3.3-70b-versatile (enterprise-only since Aug 2026) and deepseek/deepseek-chat (id retired 2026-07-24) are deliberately omitted → always unpriced (follow-up: DSE-1537).
  • Receipt fields cost_ceiling_usd / cost_basis (reported_usage | reservation); manifest cost_ceiling_usd, price_snapshot_digest, priced_as_of, unpriced_models, unpriced_receipts, pricing_warnings (fixed literals only — the secret_safety stamp stays provable).
  • Council._price_manifest — runs last on every path (all _cached_run branches, both streaming done yields). All-or-nothing: one unpriceable receipt → run ceiling None with the reasons populated. Never a partial sum.
  • Snapshot digest + output cap join cache identity; CACHE_FORMAT_VERSION 4 → 5 (old entries miss safely).

Verification

  • Review spot-checked four prices against the live vendor pages (gpt-4.1, claude-sonnet-4-6, gemini-2.5-pro, mistral-large): all match; none below the published rate.
  • Builder: 878 passed; CI green. Round 5 QA fixes in the last commit: a receipt without reported usage (incl. all-zero usage on a success) is unpriced and the run ceiling is None — never a low number; debate manifests carry one receipt per answer per round (round-{n}) before debate_final, so the ceiling covers every call made.
  • Plan: docs/plans/2026-09-03-bounded-cost-receipts.md Tasks 4–8.

Not in this PR

Round 4 (max_output_tokens, plan_calls, --max-spend-usd, exit code 4) follows.

🤖 Generated with Claude Code

https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH

…ebate receipts cover every round; zero usage is not a cost (DSE-1514 QA C1/C2/I1)

QA C1 (Round 3 half): _price_manifest priced a usage-less receipt via a
flat reservation (prompt_template_token_allowance=4096,
upstream_output_token_ceilings=()) that ignores which phase the call
belongs to, so synthesis/judge/verdict/elite-revision receipts (which
embed upstream output) came out 3-9x low while unpriced_receipts stayed
0 and the run ceiling looked complete. usage is None on a SUCCESSFUL
call too (openai_compat.py returns None when a provider omits usage,
including some streams). This round has no phase-aware call plan to
reserve from, so the honest rule is: a receipt without reported usage
is unpriced (cost_ceiling_usd=None, cost_basis=None, counted in
unpriced_receipts, "unpriced_receipts_present" warning), and the run
ceiling is None (all-or-nothing). Removed the dead reservation branch
and its two flat constants from _price_manifest; a phase-aware
reservation returns in Round 4.

QA I1: an all-zero reported TokenUsage (usage: {} coerced to
TokenUsage(0, 0, 0)) is now treated as "no usage reported" via a new
_usage_is_reported() helper, never priced as $0.000000 -- that would
be a false floor, not a bound.

QA C2 (Critical): debate's manifest receipts were built from
result.answers (the final round only), so rounds 1..R-1 and any
dropped-out member had no receipt and the run ceiling covered a strict
subset of the calls actually made (4 of 7 at N=3/R=2) while reporting
complete. Added Council._build_debate_manifest, which builds one
receipt per answer per round, phase-stamped "round-{n}" (matching
_plan_table's existing debate-round phase naming so a Round 4
phase-aware reservation can map a receipt onto its plan row by plain
equality), in round order, before the debate_final consolidation
receipt is appended. providers_called/model_ids stay full resolved
membership, unchanged.

Claude-Session: https://claude.ai/code/session_01K1dHPjZ1bZcE2GnX3KMMSH
@ernestprovo23

Copy link
Copy Markdown
Member Author

Round 5 — QA / CSO status for this PR's scope (Round 3)

  • CSO read-only review: YELLOW → fixed (7663eea, verified by execution): snapshot loader never raises; streaming cache hit re-priced by _price_manifest; the cap-validation and pretty_exceptions_show_locals items land in Round 4 (feat: pre-flight spend gate (--max-spend-usd, exit 4) — DSE-1514 round 4 of 4 #66).
  • QA adversarial review + re-check (verified by execution):
    • C1 a receipt without reported usage was priced from flat constants while the run claimed a complete ceiling (3–9× under for upstream-embedding phases). In this PR's scope the fix is: no reported usage → receipt unpriced, run ceiling None, unpriced_receipts_present + no_output_cap_configured warnings. Re-check: synthesize / elite-revision / member usage=None all → None ✓; the flat-reservation branch is gone from _price_manifest.
    • C2 debate manifests carried only final-round receipts. Now one receipt per answer per round (round-{n}) before debate_final; N=3,R=2 → 7 receipts, R=3 → 10, drop-outs → 3+2+1+final; the ceiling moves with R ✓.
    • I1 all-zero usage on a success is treated as no usage, not $0 ✓.
    • Regression matrix vs main (7edb3a1): every delta is an additive pricing field or a debate per-round receipt; no loosened assertion in the key-leak / secret-safety suites; secret_safety re-stamps after pricing and flips to unverified when a round-1 receipt is poisoned ✓.
  • Head d4b43e3: builder 878 passed, ruff clean; CI green. Full re-check write-up on feat: pre-flight spend gate (--max-spend-usd, exit 4) — DSE-1514 round 4 of 4 #66.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant