Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,46 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
exhausted. Independent of `degraded`; the cache never stores a `true` run.
- A declared chain widens which vendors may receive the prompt (see README › Synthesizer
failover › Confidentiality).
- **Bounded cost ceilings on the manifest (DSE-1514).** Every receipt now carries
`cost_ceiling_usd` (exact `Decimal`, `ROUND_CEILING`) and `cost_basis` (`reported_usage` or
`reservation`), and the manifest carries a run-level `cost_ceiling_usd`,
`price_snapshot_digest`, `priced_as_of`, `unpriced_models`, `unpriced_receipts`, and a bounded
`pricing_warnings` list. A ceiling is a falsifiable claim — *"this run cost no more than $X,
priced against snapshot `<digest>` dated `<date>`"* — not an estimate. `estimated_cost` is
untouched and stays `None`. **All-or-nothing:** one unpriced model or one unpriceable receipt
leaves the run ceiling `None` rather than emitting a partial sum.
- **Dated, vendor-cited price snapshot.** `src/conclave/data/prices-<date>.json` ships in the
wheel. Every entry cites the vendor page its rate was read from; rates are rounded **up**; a
model whose list price could not be verified is **omitted** (unpriced), never guessed —
`groq/llama-3.3-70b-versatile` and `deepseek/deepseek-chat` are currently omitted this way,
tracked for re-pricing in DSE-1537. Nothing is fetched at runtime. A snapshot older than 90
days adds a `price_snapshot_stale` warning and still prices at exactly its recorded rates.
- **`--max-output-tokens` / `max_output_tokens:`.** A hard output ceiling threaded to every call
a council makes — members, synthesizer, judge, verdict extraction and its repair retry, and
both streaming paths — and recorded in `generation_settings` when set.
- **`--max-spend-usd` pre-flight spend gate.** Enumerates the worst-case call plan for the
selected mode (`raw` `N`, `synthesize` `N+C+2C`, `vote` `N`, `debate` `N*R+C`, `adversarial`
`N+C`, `elite` `3N+C+2C`, where `C` is the count of *keyed* synthesizer-chain candidates), and
the adversarial byte-worst-case is 1 proposer succeeding immediately + N-1 critics (each
embedding the proposal's answer) with the judge embedding the proposal and every critique.
Reserves each call pessimistically at ceiling rates and **refuses before the first provider
call** with the reserved total, the cap, and the call count. New exit code **4**. A plan that
cannot be bounded — no output cap, no snapshot, or an unpriced model — refuses with a distinct
message rather than guessing.
- **Cap validation (DSE-1514 review).** `--max-spend-usd` and `Council(max_spend_usd=...)` now
reject `NaN` (every spelling: `NaN`, `-NaN`, `sNaN`, case-insensitive), `Infinity`/`inf`
(signed), and PEP-515 underscore literals (`0_5` reads as `5`, not `0.5`) — previously `NaN`
crashed uncaught at the cap comparison and `Infinity` silently disabled the gate. The CLI
applies a strict format allow-list before `Decimal(...)` ever runs (usage error, exit `2`,
names `--max-spend-usd`); `Council.__init__` enforces `is_finite()` and `> 0` independently for
library callers. `1e999999` is a deliberate exception: it is finite, just enormous, and stays
accepted. `--max-output-tokens 0`/negative now fails the same way (`min=1` on the CLI option,
plus a `ValueError` in `Council.__init__`) instead of reaching a provider as `max_tokens: 0` or
crashing inside `pricing.py`.
- `typer.Typer(..., pretty_exceptions_show_locals=False)` is now explicit rather than relying on
the installed typer version's default: `typer>=0.12.0` (this package's own floor) defaults that
flag `True`, which would render local variables — including the user's prompt — into an
unhandled exception's stderr traceback.

### Changed

Expand All @@ -59,6 +99,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
not.
- `ProviderError` and `TransportError` accept keyword-only `category` (and `http_status`);
positional construction is unchanged.
- **Cache format version `4` → `5` (DSE-1514).** Identity now additionally carries the
price-snapshot rate fingerprint and `max_output_tokens`; old entries miss safely.
- `generation_settings` (on receipts and the manifest) gains `max_output_tokens` when a cap is
configured, so an integer token cap round-trips as an integer, not a float. An uncapped run's
`generation_settings` is byte-identical to before this change.

### Not changed (deliberately)

Expand All @@ -67,6 +112,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
is decided by whether it ever answered.
- Member-level failover (members already degrade gracefully), transport-level retries, and
the substring-derived `ReceiptErrorCategory` on receipts.
- **`estimated_cost` stays `None` everywhere, permanently (DSE-1514).** It is never assigned,
never summed into, never renamed. A ceiling (`cost_ceiling_usd`) is a different, falsifiable
claim from an estimate, and the two must never be conflated in one field.

## [1.3.0] - 2026-08-01

Expand Down
1 change: 1 addition & 0 deletions DOCUMENTATION_INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ the canonical authority spec on top of those.
| **H1 Live Runner Plan** | [`docs/plans/2026-07-18-h1-live-evaluation-runner.md`](docs/plans/2026-07-18-h1-live-evaluation-runner.md) | Exact TDD tasks for the six live conditions, dry-run estimator, replay fixtures, CLI gate, and correctness-only paid smoke. |
| **Durable JSON Output Design** | [`docs/plans/2026-07-21-durable-json-output-design.md`](docs/plans/2026-07-21-durable-json-output-design.md) | Opt-in atomic user-private result persistence for long buffered council runs and detached supervisors. |
| **Adjudication Succession** | [`docs/plans/2026-09-03-adjudication-succession.md`](docs/plans/2026-09-03-adjudication-succession.md) | DSE-1512 adjudication succession implementation plan (typed failure categories, synthesizer chain, succession ledger). |
| **Bounded Cost Receipts** | [`docs/plans/2026-09-03-bounded-cost-receipts.md`](docs/plans/2026-09-03-bounded-cost-receipts.md) | DSE-1514 implementation plan: dated price snapshot + `cost_ceiling_usd` on every receipt/manifest (all-or-nothing, never an estimate), `max_output_tokens`, and the pre-flight `--max-spend-usd` spend gate with exit code `4`. |

---

Expand Down
73 changes: 73 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -537,6 +537,79 @@ not touch the network. Entries live under `$XDG_CACHE_HOME/conclave` (else
`~/.cache/conclave`); a corrupt or unreadable entry is treated as a miss and
never crashes a run.

## Cost ceilings and spend gate

A council run has always reported *tokens*. It now also reports *dollars* — as a
**ceiling**, never an estimate.

* An **estimate** is a guess. A wrong number inside an audit receipt is worse than
no number, which is why `estimated_cost` is `None` and always will be.
* A **ceiling** is a falsifiable claim: *"this run cost no more than $0.0412,
priced against snapshot `sha256:...` dated 2026-09-04."* You can check it
against your invoice.

```bash
conclave ask "should we migrate?" --mode elite --max-output-tokens 4000 --max-spend-usd 0.40
```

Before the first provider call, conclave enumerates the mode's worst-case call
plan (`Council.plan_calls`), prices every call at ceiling rates from a dated
snapshot committed to this repo, and **refuses** — exit code `4`, nothing ran,
nothing was spent — if the total exceeds your cap. `--max-output-tokens` (or
config `max_output_tokens`) is a **prerequisite**, not an option: output is the
only unbounded term in a call's cost, so a cap on it is what makes a dollar
ceiling possible at all. Both caps must be finite positive numbers: `--max-spend-usd`
rejects every spelling of `NaN`/`Infinity` and PEP-515 underscore literals
(`0_5` reads as `5`, not `0.5`) with exit code `2` before ever constructing a
`Decimal`, and `--max-output-tokens` requires a value of at least `1`. The
four refusal messages, verbatim (every one maps to exit code `4`):

| Condition | Message |
|---|---|
| `--max-spend-usd` with no output cap | `cannot bound spend: no output cap (set --max-output-tokens or config max_output_tokens)` |
| No price snapshot could be loaded at all | `cannot bound spend: price snapshot unavailable` |
| A planned call's model has no snapshot entry | `cannot bound spend: no priced rate for <model_id> in snapshot <digest> (<date>)` |
| The priced plan exceeds the cap | `refusing to run: reserved <X> USD for <N> calls exceeds the cap of <Y> USD` |

It never falls back to a similar model's rate to dodge either unboundable
message — inventing a number to get past the gate would defeat the gate.

**Where the gate lives.** The spend cap is enforced at exactly one chokepoint —
`ask`/`ask_stream` and the mode wrappers (`debate`, `adversarial`, `vote`,
`elite`, their `_sync` variants, and the CLI) — so a caller reaching directly
into a lower-level primitive (`Council.fan_out`, `Council.synthesize_blocks`,
`Council.adjudicate`, `verdict_synthesis.extract_verdict`, or a `conclave.modes`
`run_*` function called without going through the matching `Council` method)
makes real provider calls without the cap ever being consulted.

**All-or-nothing.** The prices are a hand-verified, dated file (`src/conclave/data/prices-*.json`),
not a live feed. A model whose published price could not be verified is simply
absent — which makes it unpriced, and makes the whole run's ceiling `None` with
`manifest.unpriced_models` naming it. A partial sum would read exactly like a
complete one, so there is no partial sum: one unpriced model or one unpriceable
receipt nulls the entire run-level `cost_ceiling_usd`. Two of the nine default
models are currently omitted from the snapshot for exactly this reason —
`groq/llama-3.3-70b-versatile` (moved to an Enterprise-only "Contact Sales"
tier) and `deepseek/deepseek-chat` (retired, its replacement is a different
model id conclave does not resolve) — re-pricing them is tracked in DSE-1537.

**The adversarial worst case.** `run_adversarial` tries members as proposer in
council order until one produces a usable answer, then fans the rest out as
critics; a real run makes exactly `N` member calls no matter how many proposer
attempts fail. The byte-worst-case plan is therefore **1 proposer succeeding
immediately + N-1 critics**, since every critic call embeds the proposal's full
answer text (the more critics, the more upstream bytes) — and the judge call
embeds the proposal *and every critique*, so its input bound covers all `N`
prior outputs.

| Exit code | Meaning |
|---|---|
| `0` | clean run |
| `1` | no usable answers |
| `2` | usage/config error |
| `3` | degraded — it ran, the judge/synthesizer step failed |
| `4` | **refused — nothing ran, nothing was spent** |

## Test

```bash
Expand Down
8 changes: 8 additions & 0 deletions config.example.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,11 @@ cache: false
# `--converge` / `--no-converge`. A high value (e.g. 0.95) only stops on
# near-identical successive answers.
# converge_threshold: 0.95

# Optional hard ceiling on OUTPUT tokens for every call a council makes -- members,
# synthesizer, judge, verdict extraction and its repair retry, and both streaming
# paths (OFF by default, unset). Override per invocation with --max-output-tokens.
# This is also the precondition for --max-spend-usd: a run whose output is
# unbounded cannot have its spend bounded, so the spend gate refuses (exit 4)
# rather than inventing a number.
# max_output_tokens: 4000
73 changes: 69 additions & 4 deletions docs/PRODUCT_DESIGN_DOCUMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -272,18 +272,23 @@ and cache hits (synthesize/raw builds its own richer one earlier). Pinned by
`providers_considered/called/skipped`
(each skip a `ProviderSkip{name, reason}`), `model_ids`, `generation_settings`, `receipts` (each
a `ProviderExecutionReceipt{phase, attempt, outcome, name, provider, model_id,
generation_settings, latency_ms, usage, error_category, schema_valid, versions}`),
generation_settings, latency_ms, usage, error_category, schema_valid, versions, cost_ceiling_usd,
cost_basis}`),
`total_latency_ms`, `total_usage`, `schema_valid`,
`redacted_errors`, `adjudication_succession` (the per-role succession ledger: candidate,
attempt index, outcome, bounded failure category, HTTP status; never free text), and
verdict-provenance slots (`verdict_extraction: VerdictExtraction{model_id,
attempt index, outcome, bounded failure category, HTTP status; never free text), verdict-provenance
slots (`verdict_extraction: VerdictExtraction{model_id,
prompt_version}` — the execution-trace hook — plus `verdict_type`, `consensus_method`,
`verdict_absent_reason`). Two deliberate honesty choices:
`verdict_absent_reason`), and the run-level pricing fields (v1.4, DSE-1514): `cost_ceiling_usd`,
`price_snapshot_digest`, `priced_as_of`, `unpriced_models`, `unpriced_receipts`,
`pricing_warnings` — see "Cost ceilings, never estimates" below. Two deliberate honesty choices:

For buffered Elite, every attempted call becomes a receipt: `initial`, `critique`, `revision`, `synthesis`, `verdict_extraction`, and `verdict_repair` when repair is attempted. Receipts carry phase, attempt/outcome, provider/model identity, latency, available usage/cost, bounded error category, and prompt/schema/protocol versions; totals are recomputed from this complete ledger. Incomplete runs retain only calls actually attempted.

- **No invented pricing.** Unknown per-call or aggregate `estimated_cost` stays `None`; a total is
computed only when every actual call has trustworthy priced data. Usage is recorded when reported.
`estimated_cost` is a *guess* and stays `None` forever; `cost_ceiling_usd` (v1.4, below) is a
*falsifiable claim* computed from exact rates, and the two are never conflated in one field.
- **Proven secret-safety.** `secret_safety` defaults to `unverified`, promoted to
`verified_no_secrets` **only** after `scan_for_secret_material()` proves the serialized manifest
free of forbidden substrings (`sk-`, `bearer`, `authorization`, `api_key`, `x-api-key`). Key
Expand All @@ -309,6 +314,61 @@ by whether the candidate EVER answered across its initial call and same-model re
not by whichever attempt happened to run last: a candidate that answered on either attempt is
terminal for the role even if its other attempt hit an unrelated infrastructure error.

### Cost ceilings, never estimates (v1.4)

`manifest.py` has always said `estimated_cost` stays `None` because "a wrong number
inside an audit receipt is worse than no number." That stands. What changed is that a
*different* claim is now available: not an estimate but a **ceiling** —
`cost_ceiling_usd`, computed with exact `Decimal` rates and `ROUND_CEILING` against a
dated, content-digested, vendor-cited snapshot committed to the repo
(`src/conclave/data/prices-<date>.json`), and always accompanied by
`price_snapshot_digest` and `priced_as_of` so it is checkable rather than trusted. The
two live in separate fields on purpose; conflating them would let a guess inherit a
ceiling's credibility.

Three rules keep the ceiling honest. **All-or-nothing:** any model in the run absent
from the snapshot, or any receipt that cannot be bounded, leaves the run-level ceiling
`None` with `unpriced_models` / `unpriced_receipts` naming why — a partial sum is
indistinguishable from a complete one and is the exact failure mode this design
prevents. Scope note: `unpriced_models` covers the models that actually ran (`model_ids`
plus every receipt's model), not members skipped for a missing key, which made no call
and cannot be billed. **Never a substitute rate:** an absent model is unpriced; a
similar model's rate is never borrowed, and a stale snapshot (older than
`PRICE_SNAPSHOT_MAX_AGE_DAYS = 90`) warns (`pricing_warnings`, bounded identifiers
only — see `conclave.council.PRICING_WARNING_VOCABULARY`) but still prices at the rates
it actually records. Two of the nine `registry.DEFAULT_MODELS` are currently omitted
from the shipped snapshot for exactly this reason (`groq/llama-3.3-70b-versatile`,
moved to an Enterprise-only tier; `deepseek/deepseek-chat`, retired with no successor
sharing that model id) — re-pricing them is tracked in DSE-1537. **Priced last:**
`Council._price_manifest` runs after `_ensure_manifest` and after the final receipt
append, so it can never miss the synthesis or verdict-repair calls; the snapshot's rate
digest joins cache identity (`CACHE_FORMAT_VERSION` `4` → `5`) so a hit can never serve
a ceiling that was never true of those rates.

The pre-flight `--max-spend-usd` gate is the same arithmetic run forward, and requires
`--max-output-tokens` (or config `max_output_tokens`) as a precondition: output is the
only unbounded term in a call's cost, so an uncapped run cannot be bounded in dollars.
`Council.plan_calls` enumerates the worst-case plan — `raw` `N`, `synthesize`
`N + C + 2CV`, `vote` `N`, `debate` `N*R + C`, `adversarial` `N + C`, `elite`
`3N + C + 2CV`, with `C` the number of **keyed** synthesizer-chain candidates and the
verdict's repair retry always counted — bounds each call's input by UTF-8 bytes (plus
the sum of upstream output caps times `max_output_bytes_per_token` for calls that embed
a prior model's output), and refuses before the first call with one of four exact
messages: `cannot bound spend: no output cap (set --max-output-tokens or config
max_output_tokens)`; `cannot bound spend: price snapshot unavailable` (no snapshot could
be loaded at all); `cannot bound spend: no priced rate for <model_id> in snapshot
<digest> (<date>)`; or `refusing to run: reserved <X> USD for <N> calls exceeds the cap
of <Y> USD`. All four exit CLI code `4`. Refusing is the designed outcome for an
unbounded plan: inventing a number to get past the gate would defeat the gate.

**Adversarial's byte-worst-case shape (DSE-1514 review, Fix A):** `run_adversarial`
tries members as proposer in council order until one succeeds, then fans the rest out
as critics — a real run always makes exactly `N` member calls, whatever the split. The
byte-worst case is **1 proposer succeeding immediately + N-1 critics**, since every
critic call embeds the proposal's full answer text (upstream=1 each) and maximizing
the critic count maximizes that embedded-output byte total; the judge call embeds the
proposal *and* every critique, so its upstream bound is `N` regardless of the split.

---

## 5. Provider Support Matrix
Expand Down Expand Up @@ -450,6 +510,11 @@ paid execution requires `--execute`, exact `--approve-spend-usd 10.00`, and an o
never serialize that key; frozen `max_output_bytes_per_token` attestations bound inserted UTF-8
bytes. One call is in flight, reservations persist first, and resume never repeats interrupted cells. The smoke proves correctness only—not efficiency or decision quality—and remains not decision eligible.

H1's budget-matched ablations and H4's quality-per-dollar question now have a real
denominator rather than a guess: every run carries a `cost_ceiling_usd` with its
snapshot digest and capture date (§4a), so *"is Elite worth `3N + 2` calls?"* is
answerable **at** the decision instead of after the invoice.

The canonical roadmap is
[`docs/plans/2026-07-17-decision-quality-roadmap.md`](plans/2026-07-17-decision-quality-roadmap.md):
**H0** closes Elite correctness and wording gaps before merge; **H1** runs budget-matched,
Expand Down
Loading