Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
f004491
docs(plan): DSE-1512 adjudication succession implementation plan
ernestprovo23 Sep 3, 2026
6d44a58
feat(models): type provider failure categories at the raise site (DSE…
ernestprovo23 Sep 3, 2026
0edce4c
feat(providers): carry typed failure_category/http_status on ModelAns…
ernestprovo23 Sep 3, 2026
669b1ec
docs(plan): ruff-format the Python blocks in the DSE-1512 plan
ernestprovo23 Sep 3, 2026
eca804a
fix(transport): carry http_status on streaming TransportError (DSE-1512)
ernestprovo23 Sep 3, 2026
8e9aad4
feat(config): synthesizer_chain ordered failover ladder (DSE-1512)
ernestprovo23 Sep 3, 2026
bbd1a37
feat(council): adjudicate() succession seam + manifest ledger (DSE-1512)
ernestprovo23 Sep 3, 2026
5b9272c
refactor(council): attempt builder + typed unkeyed synthetic answer (…
ernestprovo23 Sep 3, 2026
62d05a0
feat(council,modes): route all adjudication roles through the success…
ernestprovo23 Sep 3, 2026
6ede6db
refactor(council,modes): share the chain keyed-check and adjudicate-a…
ernestprovo23 Sep 3, 2026
720526b
feat(verdict): fail verdict extraction over on infra errors (DSE-1512)
ernestprovo23 Sep 3, 2026
ea09bfd
refactor(council): single _classify_outcome for every adjudication ro…
ernestprovo23 Sep 3, 2026
c29010c
feat(streaming): synthesis succession before the first delta (DSE-1512)
ernestprovo23 Sep 3, 2026
964a2a5
feat(cache): chain in identity; never cache a run whose primary faile…
ernestprovo23 Sep 3, 2026
d65ca78
refactor(models): CouncilResult.primary_failed_over computed field (D…
ernestprovo23 Sep 3, 2026
ab47f60
docs(cli): synthesizer chain surface, failover note, changelog, READM…
ernestprovo23 Sep 3, 2026
0e18a18
fix(council,verdict): content failures never fail over; cache guard o…
ernestprovo23 Sep 3, 2026
618c26a
docs: precise cache rule, verdict-extraction rule, chain confidential…
ernestprovo23 Sep 3, 2026
3ce3f55
docs(modes): module docstring names the adjudication path, not synthe…
ernestprovo23 Sep 3, 2026
da6f73f
fix(models): uniform primary_failed_over — an unkeyed primary is an i…
ernestprovo23 Sep 3, 2026
b2d7006
docs(cli,changelog): name primary_failed_over; uniform-rule wording i…
ernestprovo23 Sep 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,67 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- **Adjudication succession (DSE-1512).** The synthesizer, debate consolidator, adversarial
judge, and verdict extractor can now fail over along an operator-declared ladder:
`--synthesizer claude>grok>gemini` on the CLI, `synthesizer_chain: [claude, grok, gemini]` in
`~/.conclave/config.yml`, or `Council(synthesizer="claude>grok")` / a list in the library.
Candidates are tried strictly in declared order — no scoring, no health tracking, no routing.
Failover fires **only on infrastructure failures** (`unkeyed`, `unresolved`, `auth` 401/403,
`quota` 402/429, `unavailable` 5xx, `timeout`, `transport`). A candidate that *answered* —
even unusably (`bad_request`, `malformed_response`) — is terminal for that role, so
adjudication can never shop for a result. A run adjudicated by a successor is a clean run
(exit `0`); `degraded` / exit `3` now means the whole chain was exhausted. With no chain
configured, behaviour is unchanged.
- **Typed failure categories.** `ModelAnswer.failure_category` and `ModelAnswer.http_status`
are derived at the raise site (`TransportError.category`, `ProviderError.category` /
`http_status`, `conclave.models.categorize_http_status`), never by inspecting error text.
`ModelAnswer.error` strings are byte-for-byte unchanged.
- **Succession ledger on the manifest.** `ModelHarnessManifest.adjudication_succession`
records every candidate attempt per role (`synthesis`, `debate_final`, `judge`,
`verdict_extraction`) with `outcome` (`success` / `failed_over` / `exhausted` /
`terminal_failure` / `skipped_unkeyed`), `failure_category`, and `http_status` — bounded
values only, no free text, so the `secret_safety` stamp stays provably clean.
`result.synthesizer` / `result.adversarial.judge` now name the candidate that actually
adjudicated.
- **Streaming parity.** `--stream` synthesis walks the same ladder but fails over only
**before the first token** is emitted; a failure after output has started is terminal
regardless of category (tokens cannot be un-shown).
- `VerdictSynthesisResult.failure_category` / `http_status` (set on the extraction-failed
path only) and public `REASON_TOO_FEW` / `REASON_OPEN_ENDED` / `REASON_EXTRACTION_FAILED`.
- `CouncilResult.primary_failed_over` (computed field, present in `model_dump(mode="json")`
and `--json`): `true` when, for any role, the primary adjudicator did not itself adjudicate
for an infrastructure reason (no key, auth, quota, 5xx, timeout, network) or the ladder was
exhausted. Independent of `degraded`; the cache never stores a `true` run.
- A declared chain widens which vendors may receive the prompt (see README › Synthesizer
failover › Confidentiality).

### Changed

- `debate` and `adversarial` manifests now carry a receipt for the final-consolidation /
judge call, which they previously omitted; `total_latency_ms`, `total_usage`, and
`redacted_errors` for those modes include it. Every real call now has a receipt, matching
the Elite contract.
- **Cache.** The full synthesizer chain is part of cache identity; `CACHE_FORMAT_VERSION`
`3` → `4` (old entries miss safely). A run whose primary adjudicator did not adjudicate for
an infrastructure reason — no key, auth, quota, 5xx, timeout, network — or whose ladder was
exhausted, is **never stored** (buffered or `--stream`); a cache hit must never pin a result
the primary did not produce, nor replay an outage after it ends. `terminal_failure` runs (the
model answered) remain cacheable. Chain-of-one consequence: a degraded run whose sole
synthesizer had no key or errored for an infrastructure reason used to be cached and now is
not.
- `ProviderError` and `TransportError` accept keyword-only `category` (and `http_status`);
positional construction is unchanged.

### Not changed (deliberately)

- Verdict extraction's same-model repair retry is still attempted after an infrastructure
error; its outcome can never turn a content failure into a failover — the candidate's fate
is decided by whether it ever answered.
- Member-level failover (members already degrade gracefully), transport-level retries, and
the substring-derived `ReceiptErrorCategory` on receipts.

## [1.3.0] - 2026-08-01

### Added
Expand Down
1 change: 1 addition & 0 deletions DOCUMENTATION_INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ the canonical authority spec on top of those.
| **H1 Live Runner Design** | [`docs/plans/2026-07-18-h1-live-evaluation-runner-design.md`](docs/plans/2026-07-18-h1-live-evaluation-runner-design.md) | Sequential paid-exploratory execution, hash-bound UTF-8 estimates, USD 10 cap, authenticated checkpoints, and no-repeat resume. |
| **H1 Live Runner Plan** | [`docs/plans/2026-07-18-h1-live-evaluation-runner.md`](docs/plans/2026-07-18-h1-live-evaluation-runner.md) | Exact TDD tasks for the six live conditions, dry-run estimator, replay fixtures, CLI gate, and correctness-only paid smoke. |
| **Durable JSON Output Design** | [`docs/plans/2026-07-21-durable-json-output-design.md`](docs/plans/2026-07-21-durable-json-output-design.md) | Opt-in atomic user-private result persistence for long buffered council runs and detached supervisors. |
| **Adjudication Succession** | [`docs/plans/2026-09-03-adjudication-succession.md`](docs/plans/2026-09-03-adjudication-succession.md) | DSE-1512 adjudication succession implementation plan (typed failure categories, synthesizer chain, succession ledger). |

---

Expand Down
60 changes: 60 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -433,6 +433,65 @@ can compare it across runs to detect that the synthesis wording changed, instead
of silently attributing the shift to model drift. The test suite pins both the
prompt text and the version, so changing one without the other fails CI.

### Synthesizer failover

One vendor's outage should not strip a run of its synthesis, judge verdict, or structured
verdict. Declare an ordered ladder and conclave tries the next candidate **only when the
previous one failed for an infrastructure reason**:

```bash
conclave ask "Is a service mesh worth it for 8 services?" \
-c grok,gemini,claude,perplexity --synthesizer "claude>grok>gemini"
```

```yaml
# ~/.conclave/config.yml
synthesizer: claude
synthesizer_chain: [claude, grok, gemini] # optional; empty means just `synthesizer`
```

| Failure category | Trigger | Next candidate tried? |
|---|---|---|
| `unkeyed` / `unresolved` | no API key in the environment / unknown provider | yes (no network request is made) |
| `auth` | HTTP 401 / 403 | yes |
| `quota` | HTTP 402 / 429 | yes |
| `unavailable` | HTTP 5xx | yes |
| `timeout` / `transport` | deadline, DNS, connection | yes |
| `bad_request` | other HTTP 4xx | **no** — the request was wrong, not the vendor |
| `malformed_response` | 2xx with unusable content | **no** — the model answered |

The rule is deliberately narrow: a model that answered is never second-guessed by another
vendor, so a ladder cannot be used to shop for a verdict. The order is yours; conclave adds
no scoring or health tracking. With `--stream`, failover happens only before the first token
is shown.

Verdict extraction attempts every candidate rather than pre-skipping unkeyed ones (those
attempts fail before any network request), so its receipts stay complete; a candidate that
produced any response — even unusable JSON — is terminal for that role.

**Confidentiality.** A chain widens which vendors receive your prompt: on an infrastructure
failure the same prompt and council answers are sent to the next declared candidate. Declare
only vendors you are willing to have see the prompt.

Every attempt is recorded on the manifest so the receipt answers *who adjudicated, and why
not the primary?*:

```json
"adjudication_succession": [
{"role": "synthesis", "candidate": "claude", "model_id": "anthropic/claude-sonnet-4-6",
"attempt_index": 1, "outcome": "failed_over", "failure_category": "quota", "http_status": 429},
{"role": "synthesis", "candidate": "grok", "model_id": "xai/grok-4.3",
"attempt_index": 2, "outcome": "success", "failure_category": null, "http_status": null}
]
```

`result.synthesizer` names the candidate that actually adjudicated. A run adjudicated by a
successor exits `0`; exit `3` (`degraded`) now means the whole ladder was exhausted. A run
whose primary adjudicator did not adjudicate for an infrastructure reason — no key, auth,
quota, 5xx, timeout, network — or whose ladder was exhausted, is never written to the result
cache (buffered or `--stream`) — a cache hit must never pin a result the primary did not
produce, nor replay an outage after it ends.

## Config (optional)

Create `~/.conclave/config.yml` to add models, define named councils, and set a
Expand All @@ -446,6 +505,7 @@ councils:
default: [grok, gemini, claude, perplexity]
fast: [grok, perplexity]
synthesizer: claude
synthesizer_chain: [claude, grok] # optional: ordered failover ladder
```

Then: `conclave ask "..." --council fast`.
Expand Down
4 changes: 4 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -240,3 +240,7 @@ is available, and credit you in the advisory unless you ask to remain anonymous.
- conclave is a council aggregator, not a security control. The *content* a model
returns is not adjudicated for safety; that is out of scope. A leak of the
user's own credentials is the security boundary we defend.
- Synthesizer failover (`synthesizer_chain`, v1.4): on an infrastructure failure the
same prompt and council answers are sent to the next candidate the operator
declared. Failover never fires on a content failure and never adds data; it only
widens which of the operator's own declared vendors receive the prompt.
4 changes: 4 additions & 0 deletions config.example.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ councils:
# the answers into a scored, auditable verdict on `result.verdict`/`result.manifest`.
# There is no config switch for it -- opt out in code via Council(extract_verdict=False).
synthesizer: claude
# synthesizer_chain: [claude, grok] # optional: ordered failover ladder (DSE-1512).
# Tried in order; a candidate advances to the next ONLY on an infrastructure
# failure (auth/quota/5xx/timeout/network/no-key). Empty (the default) means
# "just `synthesizer`" -- a chain of one, identical to today's behavior.

# Optional result cache (OFF by default). When true, an identical repeat run is
# served from an on-disk cache instead of re-calling the providers -- handy for
Expand Down
28 changes: 25 additions & 3 deletions docs/PRODUCT_DESIGN_DOCUMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
- **Repo:** `/Users/ernestprovo/dev/conclave/`
- **License:** MIT
- **Author:** Data Science & Engineering Experts, Inc. (DSE)
- **Last updated:** 2026-07-18
- **Last updated:** 2026-09-03

---

Expand Down Expand Up @@ -73,7 +73,7 @@ Helicone is in §11).
| **The skeptical engineer** | Senior dev / architect making a consequential technical call | A fast second/third opinion across models, with raw per-model answers visible so they can judge disagreement themselves. Uses the CLI ad hoc. |
| **The library integrator** | Developer building a tool that needs multi-model input at *design/eval time* | `from conclave import Council`, structured `CouncilResult` (latency, token usage, per-model errors), partial-failure resilience. The primary downstream example is **mcp-warden** (see §10). |
| **The researcher / evaluator** | Someone comparing model behavior on a prompt set | Deterministic structure around answers, JSON output (`--json`) for downstream analysis, per-model latency and token accounting. |
| **The cost-conscious power user** | Heavy LLM user who already pays each provider directly | BYO-keys with **no markup** and **no third party seeing the prompt**. conclave is a thin local orchestrator over the user's own accounts. |
| **The cost-conscious power user** | Heavy LLM user who already pays each provider directly | BYO-keys with **no markup** and **no third party seeing the prompt** (a declared synthesizer chain can send the prompt to the next declared vendor on an infrastructure failure — §4a). conclave is a thin local orchestrator over the user's own accounts. |

Non-personas (*not* who we build for): teams wanting a hosted multi-agent SaaS, or anyone
needing a deterministic runtime adjudicator (Non-Goals §8, mcp-warden boundary §10).
Expand Down Expand Up @@ -274,7 +274,9 @@ and cache hits (synthesize/raw builds its own richer one earlier). Pinned by
a `ProviderExecutionReceipt{phase, attempt, outcome, name, provider, model_id,
generation_settings, latency_ms, usage, error_category, schema_valid, versions}`),
`total_latency_ms`, `total_usage`, `schema_valid`,
`redacted_errors`, and verdict-provenance slots (`verdict_extraction: VerdictExtraction{model_id,
`redacted_errors`, `adjudication_succession` (the per-role succession ledger: candidate,
attempt index, outcome, bounded failure category, HTTP status; never free text), and
verdict-provenance slots (`verdict_extraction: VerdictExtraction{model_id,
prompt_version}` — the execution-trace hook — plus `verdict_type`, `consensus_method`,
`verdict_absent_reason`). Two deliberate honesty choices:

Expand All @@ -287,6 +289,26 @@ For buffered Elite, every attempted call becomes a receipt: `initial`, `critique
free of forbidden substrings (`sk-`, `bearer`, `authorization`, `api_key`, `x-api-key`). Key
*values* never appear; errors are redacted upstream and re-redacted on construction.

### Adjudication succession (v1.4)

The synthesizer / judge / verdict-extractor identity is an ordered ladder
(`synthesizer_chain`; a chain of one is the v1.3 behaviour). The seam `Council.adjudicate`
walks it under one rule, shared by every role including streaming synthesis and verdict
extraction: **advance only on an infrastructure failure** (`unkeyed`, `unresolved`, `auth`,
`quota`, `unavailable`, `timeout`, `transport`); any candidate that answered — including a
malformed answer — is terminal. The rule is narrow on purpose: allowing a second vendor to
re-adjudicate after a content failure would let a run shop for its verdict and would break
reproducibility. Failure categories are typed at the raise site (`TransportError.category`,
`ProviderError.category`), never inferred from error text. The ledger carries bounded
categories and an integer HTTP status only, so `secret_safety` remains provable. A successor
adjudication is a clean run; a run whose primary adjudicator did not adjudicate for an
infrastructure reason — no key, auth, quota, 5xx, timeout, network — or whose ladder was
exhausted, is never cached — a cache hit must never pin a result the primary did not produce,
nor replay an outage after it ends. For verdict extraction specifically, the failure category is decided
by whether the candidate EVER answered across its initial call and same-model repair retry,
not by whichever attempt happened to run last: a candidate that answered on either attempt is
terminal for the role even if its other attempt hit an unrelated infrastructure error.

---

## 5. Provider Support Matrix
Expand Down
Loading