diff --git a/CHANGELOG.md b/CHANGELOG.md index e5f1ca0..638c500 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,67 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- **Adjudication succession (DSE-1512).** The synthesizer, debate consolidator, adversarial + judge, and verdict extractor can now fail over along an operator-declared ladder: + `--synthesizer claude>grok>gemini` on the CLI, `synthesizer_chain: [claude, grok, gemini]` in + `~/.conclave/config.yml`, or `Council(synthesizer="claude>grok")` / a list in the library. + Candidates are tried strictly in declared order — no scoring, no health tracking, no routing. + Failover fires **only on infrastructure failures** (`unkeyed`, `unresolved`, `auth` 401/403, + `quota` 402/429, `unavailable` 5xx, `timeout`, `transport`). A candidate that *answered* — + even unusably (`bad_request`, `malformed_response`) — is terminal for that role, so + adjudication can never shop for a result. A run adjudicated by a successor is a clean run + (exit `0`); `degraded` / exit `3` now means the whole chain was exhausted. With no chain + configured, behaviour is unchanged. +- **Typed failure categories.** `ModelAnswer.failure_category` and `ModelAnswer.http_status` + are derived at the raise site (`TransportError.category`, `ProviderError.category` / + `http_status`, `conclave.models.categorize_http_status`), never by inspecting error text. + `ModelAnswer.error` strings are byte-for-byte unchanged. +- **Succession ledger on the manifest.** `ModelHarnessManifest.adjudication_succession` + records every candidate attempt per role (`synthesis`, `debate_final`, `judge`, + `verdict_extraction`) with `outcome` (`success` / `failed_over` / `exhausted` / + `terminal_failure` / `skipped_unkeyed`), `failure_category`, and `http_status` — bounded + values only, no free text, so the `secret_safety` stamp stays provably clean. + `result.synthesizer` / `result.adversarial.judge` now name the candidate that actually + adjudicated. +- **Streaming parity.** `--stream` synthesis walks the same ladder but fails over only + **before the first token** is emitted; a failure after output has started is terminal + regardless of category (tokens cannot be un-shown). +- `VerdictSynthesisResult.failure_category` / `http_status` (set on the extraction-failed + path only) and public `REASON_TOO_FEW` / `REASON_OPEN_ENDED` / `REASON_EXTRACTION_FAILED`. +- `CouncilResult.primary_failed_over` (computed field, present in `model_dump(mode="json")` + and `--json`): `true` when, for any role, the primary adjudicator did not itself adjudicate + for an infrastructure reason (no key, auth, quota, 5xx, timeout, network) or the ladder was + exhausted. Independent of `degraded`; the cache never stores a `true` run. +- A declared chain widens which vendors may receive the prompt (see README › Synthesizer + failover › Confidentiality). + +### Changed + +- `debate` and `adversarial` manifests now carry a receipt for the final-consolidation / + judge call, which they previously omitted; `total_latency_ms`, `total_usage`, and + `redacted_errors` for those modes include it. Every real call now has a receipt, matching + the Elite contract. +- **Cache.** The full synthesizer chain is part of cache identity; `CACHE_FORMAT_VERSION` + `3` → `4` (old entries miss safely). A run whose primary adjudicator did not adjudicate for + an infrastructure reason — no key, auth, quota, 5xx, timeout, network — or whose ladder was + exhausted, is **never stored** (buffered or `--stream`); a cache hit must never pin a result + the primary did not produce, nor replay an outage after it ends. `terminal_failure` runs (the + model answered) remain cacheable. Chain-of-one consequence: a degraded run whose sole + synthesizer had no key or errored for an infrastructure reason used to be cached and now is + not. +- `ProviderError` and `TransportError` accept keyword-only `category` (and `http_status`); + positional construction is unchanged. + +### Not changed (deliberately) + +- Verdict extraction's same-model repair retry is still attempted after an infrastructure + error; its outcome can never turn a content failure into a failover — the candidate's fate + is decided by whether it ever answered. +- Member-level failover (members already degrade gracefully), transport-level retries, and + the substring-derived `ReceiptErrorCategory` on receipts. + ## [1.3.0] - 2026-08-01 ### Added diff --git a/DOCUMENTATION_INDEX.md b/DOCUMENTATION_INDEX.md index d120be2..69e9284 100644 --- a/DOCUMENTATION_INDEX.md +++ b/DOCUMENTATION_INDEX.md @@ -37,6 +37,7 @@ the canonical authority spec on top of those. | **H1 Live Runner Design** | [`docs/plans/2026-07-18-h1-live-evaluation-runner-design.md`](docs/plans/2026-07-18-h1-live-evaluation-runner-design.md) | Sequential paid-exploratory execution, hash-bound UTF-8 estimates, USD 10 cap, authenticated checkpoints, and no-repeat resume. | | **H1 Live Runner Plan** | [`docs/plans/2026-07-18-h1-live-evaluation-runner.md`](docs/plans/2026-07-18-h1-live-evaluation-runner.md) | Exact TDD tasks for the six live conditions, dry-run estimator, replay fixtures, CLI gate, and correctness-only paid smoke. | | **Durable JSON Output Design** | [`docs/plans/2026-07-21-durable-json-output-design.md`](docs/plans/2026-07-21-durable-json-output-design.md) | Opt-in atomic user-private result persistence for long buffered council runs and detached supervisors. | +| **Adjudication Succession** | [`docs/plans/2026-09-03-adjudication-succession.md`](docs/plans/2026-09-03-adjudication-succession.md) | DSE-1512 adjudication succession implementation plan (typed failure categories, synthesizer chain, succession ledger). | --- diff --git a/README.md b/README.md index 02b776a..0e547d2 100644 --- a/README.md +++ b/README.md @@ -433,6 +433,65 @@ can compare it across runs to detect that the synthesis wording changed, instead of silently attributing the shift to model drift. The test suite pins both the prompt text and the version, so changing one without the other fails CI. +### Synthesizer failover + +One vendor's outage should not strip a run of its synthesis, judge verdict, or structured +verdict. Declare an ordered ladder and conclave tries the next candidate **only when the +previous one failed for an infrastructure reason**: + +```bash +conclave ask "Is a service mesh worth it for 8 services?" \ + -c grok,gemini,claude,perplexity --synthesizer "claude>grok>gemini" +``` + +```yaml +# ~/.conclave/config.yml +synthesizer: claude +synthesizer_chain: [claude, grok, gemini] # optional; empty means just `synthesizer` +``` + +| Failure category | Trigger | Next candidate tried? | +|---|---|---| +| `unkeyed` / `unresolved` | no API key in the environment / unknown provider | yes (no network request is made) | +| `auth` | HTTP 401 / 403 | yes | +| `quota` | HTTP 402 / 429 | yes | +| `unavailable` | HTTP 5xx | yes | +| `timeout` / `transport` | deadline, DNS, connection | yes | +| `bad_request` | other HTTP 4xx | **no** — the request was wrong, not the vendor | +| `malformed_response` | 2xx with unusable content | **no** — the model answered | + +The rule is deliberately narrow: a model that answered is never second-guessed by another +vendor, so a ladder cannot be used to shop for a verdict. The order is yours; conclave adds +no scoring or health tracking. With `--stream`, failover happens only before the first token +is shown. + +Verdict extraction attempts every candidate rather than pre-skipping unkeyed ones (those +attempts fail before any network request), so its receipts stay complete; a candidate that +produced any response — even unusable JSON — is terminal for that role. + +**Confidentiality.** A chain widens which vendors receive your prompt: on an infrastructure +failure the same prompt and council answers are sent to the next declared candidate. Declare +only vendors you are willing to have see the prompt. + +Every attempt is recorded on the manifest so the receipt answers *who adjudicated, and why +not the primary?*: + +```json +"adjudication_succession": [ + {"role": "synthesis", "candidate": "claude", "model_id": "anthropic/claude-sonnet-4-6", + "attempt_index": 1, "outcome": "failed_over", "failure_category": "quota", "http_status": 429}, + {"role": "synthesis", "candidate": "grok", "model_id": "xai/grok-4.3", + "attempt_index": 2, "outcome": "success", "failure_category": null, "http_status": null} +] +``` + +`result.synthesizer` names the candidate that actually adjudicated. A run adjudicated by a +successor exits `0`; exit `3` (`degraded`) now means the whole ladder was exhausted. A run +whose primary adjudicator did not adjudicate for an infrastructure reason — no key, auth, +quota, 5xx, timeout, network — or whose ladder was exhausted, is never written to the result +cache (buffered or `--stream`) — a cache hit must never pin a result the primary did not +produce, nor replay an outage after it ends. + ## Config (optional) Create `~/.conclave/config.yml` to add models, define named councils, and set a @@ -446,6 +505,7 @@ councils: default: [grok, gemini, claude, perplexity] fast: [grok, perplexity] synthesizer: claude +synthesizer_chain: [claude, grok] # optional: ordered failover ladder ``` Then: `conclave ask "..." --council fast`. diff --git a/SECURITY.md b/SECURITY.md index 946d860..f280f02 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -240,3 +240,7 @@ is available, and credit you in the advisory unless you ask to remain anonymous. - conclave is a council aggregator, not a security control. The *content* a model returns is not adjudicated for safety; that is out of scope. A leak of the user's own credentials is the security boundary we defend. +- Synthesizer failover (`synthesizer_chain`, v1.4): on an infrastructure failure the + same prompt and council answers are sent to the next candidate the operator + declared. Failover never fires on a content failure and never adds data; it only + widens which of the operator's own declared vendors receive the prompt. diff --git a/config.example.yml b/config.example.yml index 644b1c1..5d52f68 100644 --- a/config.example.yml +++ b/config.example.yml @@ -22,6 +22,10 @@ councils: # the answers into a scored, auditable verdict on `result.verdict`/`result.manifest`. # There is no config switch for it -- opt out in code via Council(extract_verdict=False). synthesizer: claude +# synthesizer_chain: [claude, grok] # optional: ordered failover ladder (DSE-1512). +# Tried in order; a candidate advances to the next ONLY on an infrastructure +# failure (auth/quota/5xx/timeout/network/no-key). Empty (the default) means +# "just `synthesizer`" -- a chain of one, identical to today's behavior. # Optional result cache (OFF by default). When true, an identical repeat run is # served from an on-disk cache instead of re-calling the providers -- handy for diff --git a/docs/PRODUCT_DESIGN_DOCUMENT.md b/docs/PRODUCT_DESIGN_DOCUMENT.md index f968a5e..1853bff 100644 --- a/docs/PRODUCT_DESIGN_DOCUMENT.md +++ b/docs/PRODUCT_DESIGN_DOCUMENT.md @@ -14,7 +14,7 @@ - **Repo:** `/Users/ernestprovo/dev/conclave/` - **License:** MIT - **Author:** Data Science & Engineering Experts, Inc. (DSE) -- **Last updated:** 2026-07-18 +- **Last updated:** 2026-09-03 --- @@ -73,7 +73,7 @@ Helicone is in §11). | **The skeptical engineer** | Senior dev / architect making a consequential technical call | A fast second/third opinion across models, with raw per-model answers visible so they can judge disagreement themselves. Uses the CLI ad hoc. | | **The library integrator** | Developer building a tool that needs multi-model input at *design/eval time* | `from conclave import Council`, structured `CouncilResult` (latency, token usage, per-model errors), partial-failure resilience. The primary downstream example is **mcp-warden** (see §10). | | **The researcher / evaluator** | Someone comparing model behavior on a prompt set | Deterministic structure around answers, JSON output (`--json`) for downstream analysis, per-model latency and token accounting. | -| **The cost-conscious power user** | Heavy LLM user who already pays each provider directly | BYO-keys with **no markup** and **no third party seeing the prompt**. conclave is a thin local orchestrator over the user's own accounts. | +| **The cost-conscious power user** | Heavy LLM user who already pays each provider directly | BYO-keys with **no markup** and **no third party seeing the prompt** (a declared synthesizer chain can send the prompt to the next declared vendor on an infrastructure failure — §4a). conclave is a thin local orchestrator over the user's own accounts. | Non-personas (*not* who we build for): teams wanting a hosted multi-agent SaaS, or anyone needing a deterministic runtime adjudicator (Non-Goals §8, mcp-warden boundary §10). @@ -274,7 +274,9 @@ and cache hits (synthesize/raw builds its own richer one earlier). Pinned by a `ProviderExecutionReceipt{phase, attempt, outcome, name, provider, model_id, generation_settings, latency_ms, usage, error_category, schema_valid, versions}`), `total_latency_ms`, `total_usage`, `schema_valid`, -`redacted_errors`, and verdict-provenance slots (`verdict_extraction: VerdictExtraction{model_id, +`redacted_errors`, `adjudication_succession` (the per-role succession ledger: candidate, +attempt index, outcome, bounded failure category, HTTP status; never free text), and +verdict-provenance slots (`verdict_extraction: VerdictExtraction{model_id, prompt_version}` — the execution-trace hook — plus `verdict_type`, `consensus_method`, `verdict_absent_reason`). Two deliberate honesty choices: @@ -287,6 +289,26 @@ For buffered Elite, every attempted call becomes a receipt: `initial`, `critique free of forbidden substrings (`sk-`, `bearer`, `authorization`, `api_key`, `x-api-key`). Key *values* never appear; errors are redacted upstream and re-redacted on construction. +### Adjudication succession (v1.4) + +The synthesizer / judge / verdict-extractor identity is an ordered ladder +(`synthesizer_chain`; a chain of one is the v1.3 behaviour). The seam `Council.adjudicate` +walks it under one rule, shared by every role including streaming synthesis and verdict +extraction: **advance only on an infrastructure failure** (`unkeyed`, `unresolved`, `auth`, +`quota`, `unavailable`, `timeout`, `transport`); any candidate that answered — including a +malformed answer — is terminal. The rule is narrow on purpose: allowing a second vendor to +re-adjudicate after a content failure would let a run shop for its verdict and would break +reproducibility. Failure categories are typed at the raise site (`TransportError.category`, +`ProviderError.category`), never inferred from error text. The ledger carries bounded +categories and an integer HTTP status only, so `secret_safety` remains provable. A successor +adjudication is a clean run; a run whose primary adjudicator did not adjudicate for an +infrastructure reason — no key, auth, quota, 5xx, timeout, network — or whose ladder was +exhausted, is never cached — a cache hit must never pin a result the primary did not produce, +nor replay an outage after it ends. For verdict extraction specifically, the failure category is decided +by whether the candidate EVER answered across its initial call and same-model repair retry, +not by whichever attempt happened to run last: a candidate that answered on either attempt is +terminal for the role even if its other attempt hit an unrelated infrastructure error. + --- ## 5. Provider Support Matrix diff --git a/docs/plans/2026-09-03-adjudication-succession.md b/docs/plans/2026-09-03-adjudication-succession.md new file mode 100644 index 0000000..6904ddf --- /dev/null +++ b/docs/plans/2026-09-03-adjudication-succession.md @@ -0,0 +1,1174 @@ +# Adjudication Succession Implementation Plan (DSE-1512) + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Let the judge/synthesizer/verdict-extractor fail over, in operator-declared order, to the next candidate on *infrastructure* failures only — so one vendor's outage no longer strips a council run of its adjudication — and record every attempt in the manifest. + +**Architecture:** (1) Type the failure category at the raise site (transport + adapters) and thread it onto `ModelAnswer.failure_category` — no substring matching. (2) Replace the single `synthesizer` identity with an ordered `synthesizer_chain` (chain-of-one == today). (3) One new seam, `Council.adjudicate(role, …)`, walks the chain with a strict failover rule and returns the answer plus a per-attempt ledger; every adjudication role (synthesis, debate final, adversarial judge, verdict extraction, streaming synthesis) routes through it. (4) `ModelHarnessManifest.adjudication_succession` records the ledger with **no free text** (categories + HTTP status only) so the secret-safety stamp stays provably clean. (5) The full chain joins cache identity; a result adjudicated by a successor is never stored in the cache. + +**Tech Stack:** Python 3.11+, pydantic v2, httpx (transport), typer (CLI), pytest + pytest-asyncio (offline, mocked seams — see `tests/conftest.py`). + +**Execution locality:** edit + git on the laptop worktree `~/dev/worktrees/conclave-dse-1512`; run every test/lint on the builder via +`~/.claude/scripts/builder-run.sh conclave-dse-1512 ''` (it rsyncs first). Never run pytest/ruff on the laptop. +Shorthand used below: `BR='~/.claude/scripts/builder-run.sh conclave-dse-1512'`. + +--- + +## Ground rules (read before Task 1) + +- **Additive only.** `ModelAnswer.error` text must be byte-identical to today for every existing test. New fields default to `None`/empty. Do not rename anything. +- **Chain of one == v1.3.0.** With no chain configured, every code path must behave exactly as it does now. The suite at `origin/main` is the regression oracle — run it after every task. +- **Failover fires on infrastructure failure only.** A valid-but-unwelcome or malformed answer is terminal. Never let adjudication shop for a result. +- **The manifest carries no free text for succession.** `AdjudicationAttempt` has categories and an HTTP status, never an error string. `scan_for_secret_material()` forbids the substrings `sk-`, `bearer`, `authorization`, `api_key`, `x-api-key` in the serialized manifest — a provider error body saying "Missing Authorization header" would un-verify the stamp. That is why the ticket's `redacted_reason` field is **replaced** by `failure_category` + `http_status`. +- **Do not touch** `redact()`, `scan_for_secret_material()`, `_receipt_error_category()`, or any credential path. Touching them re-classifies the PR as security-specific. +- **TDD.** Failing test → run → minimal code → run → commit. Commit after each task with the message given. +- **Do not modify the existing repair-retry behaviour in `extract_verdict`** (it still retries once on the same model even after an infra error). Out of scope; note it as a follow-up in the PR body. + +Existing seams you will rely on: + +| Seam | Where | Note | +|---|---|---| +| Member/synth call | `conclave.council.call_model` | patched by `patch_call_model` fixture | +| Verdict call | `conclave.verdict_synthesis.call_model` | autouse offline stub in conftest | +| Transport | `conclave.transport.post_json` | patched in `tests/test_providers.py` for end-to-end `call_model` | +| Key presence | `conclave.registry.key_present` | env-var name check; unknown providers return `True` | + +--- + +### Task 1: Typed failure categories (models + transport + adapters) + +**Files:** +- Modify: `src/conclave/models.py` (add near `TokenUsage`) +- Modify: `src/conclave/transport.py:94-140,184,192,281,306,310` +- Modify: `src/conclave/adapters/base.py:240-247` +- Modify: `src/conclave/adapters/__init__.py:96` +- Modify: `src/conclave/adapters/openai_compat.py:215-218` +- Modify: `src/conclave/adapters/anthropic.py:201-204` +- Modify: `src/conclave/adapters/gemini.py:344-345` +- Test: `tests/test_failure_category.py` (new) + +**Step 1: Write the failing tests** + +```python +# tests/test_failure_category.py +"""Typed failure categories are derived from status codes / exception types (DSE-1512).""" + +from __future__ import annotations + +import httpx +import pytest + +from conclave import transport +from conclave.adapters import ProviderError, resolve_adapter +from conclave.adapters.anthropic import AnthropicAdapter +from conclave.adapters.gemini import GeminiAdapter +from conclave.adapters.openai_compat import OpenAICompatAdapter +from conclave.config import ConclaveConfig +from conclave.models import FAILOVER_CATEGORIES, categorize_http_status + + +@pytest.mark.parametrize( + ("status", "expected"), + [ + (401, "auth"), + (403, "auth"), + (402, "quota"), + (429, "quota"), + (408, "timeout"), + (500, "unavailable"), + (502, "unavailable"), + (503, "unavailable"), + (529, "unavailable"), + (400, "bad_request"), + (404, "bad_request"), + (422, "bad_request"), + ], +) +def test_categorize_http_status(status, expected): + assert categorize_http_status(status) == expected + + +def test_failover_set_is_infrastructure_only(): + assert FAILOVER_CATEGORIES == frozenset( + {"unkeyed", "unresolved", "auth", "quota", "unavailable", "timeout", "transport"} + ) + assert "bad_request" not in FAILOVER_CATEGORIES + assert "malformed_response" not in FAILOVER_CATEGORIES + assert "unexpected" not in FAILOVER_CATEGORIES + + +def test_provider_error_defaults_to_malformed_response(): + err = ProviderError("x: empty response") + assert err.category == "malformed_response" + assert err.http_status is None + + +def test_provider_error_carries_status_category(): + err = ProviderError("x: HTTP 429: slow down", category="quota", http_status=429) + assert err.category == "quota" + assert err.http_status == 429 + # message is still redacted on construction (existing contract) + assert "sk-" not in str(ProviderError("leak sk-abc123def456ghi789", category="auth")) + + +def test_transport_error_category(): + assert ( + transport.TransportError("request timed out after 5s", category="timeout").category + == "timeout" + ) + assert transport.TransportError("network error: ConnectError").category == "transport" + + +@pytest.mark.parametrize( + "adapter", [OpenAICompatAdapter("openai"), AnthropicAdapter(), GeminiAdapter()] +) +def test_adapters_type_non_2xx(adapter): + with pytest.raises(ProviderError) as info: + adapter.parse_response(401, {"error": {"message": "bad key"}}) + assert info.value.category == "auth" + assert info.value.http_status == 401 + with pytest.raises(ProviderError) as info: + adapter.parse_response(503, {"error": {"message": "down"}}) + assert info.value.category == "unavailable" + + +def test_adapter_malformed_is_not_failover(): + with pytest.raises(ProviderError) as info: + OpenAICompatAdapter("openai").parse_response(200, {"choices": []}) + assert info.value.category == "malformed_response" + assert info.value.category not in FAILOVER_CATEGORIES + + +def test_unresolved_provider_is_typed(): + with pytest.raises(ProviderError) as info: + resolve_adapter("nope/model", ConclaveConfig()) + assert info.value.category == "unresolved" + + +async def test_post_json_timeout_is_typed(monkeypatch): + class _Client: + is_closed = False + + async def post(self, *a, **k): + raise httpx.ReadTimeout("slow") + + monkeypatch.setattr(transport, "_client", _Client()) + with pytest.raises(transport.TransportError) as info: + await transport.post_json("https://x", {}, {}, 1.0) + assert info.value.category == "timeout" + + +async def test_post_json_network_is_typed(monkeypatch): + class _Client: + is_closed = False + + async def post(self, *a, **k): + raise httpx.ConnectError("refused") + + monkeypatch.setattr(transport, "_client", _Client()) + with pytest.raises(transport.TransportError) as info: + await transport.post_json("https://x", {}, {}, 1.0) + assert info.value.category == "transport" +``` + +Check the adapter constructor signatures before running (`grep -n "class .*Adapter\|def __init__" src/conclave/adapters/*.py`) and adjust the parametrize instantiation to match — the registry in `adapters/__init__.py` shows how each is constructed. + +**Step 2: Run to verify it fails** + +Run: `$BR '.venv/bin/python -m pytest -q tests/test_failure_category.py'` +Expected: FAIL — `ImportError: cannot import name 'FAILOVER_CATEGORIES'` + +**Step 3: Implement** + +`src/conclave/models.py` — add after `TokenUsage`: + +```python +# DSE-1512 — typed failure categories. Derived at the RAISE SITE from the HTTP +# status or exception type, never by inspecting a rendered error string. The +# adjudication ladder (Council.adjudicate) fails over ONLY on the categories in +# FAILOVER_CATEGORIES: infrastructure failures where no model ever produced an +# answer. A model that answered (even malformed) is terminal for that role. +FailureCategory = Literal[ + "unkeyed", # env var absent -- no call made + "unresolved", # unknown provider prefix -- no call made + "auth", # 401 / 403 + "quota", # 402 / 429 + "unavailable", # 5xx + "timeout", # 408 or transport deadline + "transport", # DNS / connection / other httpx network error + "bad_request", # other 4xx -- the request was wrong, not the vendor + "malformed_response", # 2xx with an unusable payload / empty content + "unexpected", # anything else -- never failed over +] + +FAILOVER_CATEGORIES: frozenset[str] = frozenset( + {"unkeyed", "unresolved", "auth", "quota", "unavailable", "timeout", "transport"} +) + + +def categorize_http_status(status: int) -> FailureCategory: + """Map a non-2xx HTTP status to a :data:`FailureCategory` (pure, no I/O).""" + if status in (401, 403): + return "auth" + if status in (402, 429): + return "quota" + if status == 408: + return "timeout" + if 500 <= status <= 599: + return "unavailable" + if 400 <= status <= 499: + return "bad_request" + return "malformed_response" +``` + +Add `from typing import Literal` if not already imported. Add two fields to `ModelAnswer` (after `warnings`), and document them in the class docstring: + +```python + failure_category: FailureCategory | None = None + http_status: int | None = None +``` + +`src/conclave/transport.py`: + +```python +from .models import FailureCategory, categorize_http_status # add import + + +class TransportError(Exception): + """...(keep docstring)...""" + + def __init__(self, message: str, *, category: FailureCategory = "transport") -> None: + super().__init__(message) + self.category: FailureCategory = category + + +def _raise_transport_error(message: str, category: FailureCategory = "transport") -> NoReturn: + raise TransportError(message, category=category) from None +``` + +- Both `httpx.TimeoutException` sites: `_raise_transport_error(f"request timed out after {timeout:.0f}s", "timeout")`. +- Both `httpx.HTTPError` sites: unchanged call (default `"transport"`). +- `stream_sse` non-2xx: `raise TransportError(f"HTTP {response.status_code}: {detail}", category=categorize_http_status(response.status_code))`. + +`src/conclave/adapters/base.py`: + +```python +from ..models import FailureCategory, TokenUsage # extend the existing import + + +class ProviderError(Exception): + """...(keep docstring; add:) ``category``/``http_status`` are typed at the raise + site (DSE-1512) so failover never depends on the message text.""" + + def __init__( + self, + message: str, + *, + category: FailureCategory = "malformed_response", + http_status: int | None = None, + ) -> None: + super().__init__(redact(message)) + self.category: FailureCategory = category + self.http_status = http_status +``` + +Adapters — only the three buffered non-2xx raise sites change (stream-frame in-band errors carry `status 200` and stay `malformed_response` by design): + +```python +# openai_compat.py parse_response + if status < 200 or status >= 300: + raise ProviderError( + status_error(self.prefix, status, payload, secondary_keys=("type",)), + category=categorize_http_status(status), + http_status=status, + ) +``` +Same shape in `anthropic.py` (secondary_keys `("type",)`) and `gemini.py` (secondary_keys `("status",)`). Import `categorize_http_status` from `..models` in each. + +`src/conclave/adapters/__init__.py:96` — the unknown-provider raise gets `category="unresolved"`. + +**Step 4: Run to verify it passes** + +Run: `$BR '.venv/bin/python -m pytest -q tests/test_failure_category.py tests/test_adapters.py tests/test_transport.py tests/test_providers.py'` +Expected: PASS (all) + +**Step 5: Full regression + commit** + +Run: `$BR` (full suite) — Expected: same pass count as baseline, 0 failures. +Run: `$BR '.venv/bin/ruff check . && .venv/bin/ruff format --check .'` + +```bash +git add src/conclave/models.py src/conclave/transport.py src/conclave/adapters tests/test_failure_category.py +git commit -m "feat(models): type provider failure categories at the raise site (DSE-1512)" +``` + +--- + +### Task 2: Thread the category onto `ModelAnswer` in `call_model` / `call_model_stream` + +**Files:** +- Modify: `src/conclave/providers.py:169-260` (`call_model`), `:300-409` (`call_model_stream`) +- Test: `tests/test_providers.py` (append) + +**Step 1: Failing tests** (append to `tests/test_providers.py`; follow the file's existing `post_json` patching style) + +```python +async def test_call_model_types_unkeyed(monkeypatch, clear_keys): + ans = await call_model("grok", "xai/grok-4.3", [{"role": "user", "content": "hi"}]) + assert ans.error and ans.failure_category == "unkeyed" and ans.http_status is None + + +async def test_call_model_types_http_status(monkeypatch, patch_transport): + patch_transport(status=402, body={"error": {"message": "insufficient credit"}}) + ans = await call_model( + "claude", "anthropic/claude-sonnet-4-6", [{"role": "user", "content": "hi"}] + ) + assert ans.error and ans.failure_category == "quota" and ans.http_status == 402 + + +async def test_call_model_types_timeout(monkeypatch, patch_transport_raise): + patch_transport_raise( + transport.TransportError("request timed out after 1s", category="timeout") + ) + ans = await call_model( + "claude", "anthropic/claude-sonnet-4-6", [{"role": "user", "content": "hi"}] + ) + assert ans.failure_category == "timeout" + + +async def test_call_model_error_text_unchanged(monkeypatch, patch_transport): + """Additive-only guarantee: the error STRING is byte-identical to before.""" + patch_transport(status=401, body={"error": {"message": "bad key"}}) + ans = await call_model( + "claude", "anthropic/claude-sonnet-4-6", [{"role": "user", "content": "hi"}] + ) + assert ans.error == "anthropic: HTTP 401: bad key" + + +async def test_call_model_unresolved_is_typed(): + ans = await call_model( + "x", "nope/model", [{"role": "user", "content": "hi"}], config=ConclaveConfig() + ) + assert ans.failure_category == "unresolved" +``` + +`clear_keys` exists in `tests/test_council.py`'s fixtures; if it is module-local, copy the pattern (monkeypatch.delenv each `*_API_KEY`) into a local fixture. Write `patch_transport`/`patch_transport_raise` as small local fixtures that `monkeypatch.setattr(transport, "post_json", fake)` — mirror the existing helpers in `tests/test_providers.py`. Set a dummy `ANTHROPIC_API_KEY` in those tests so the unkeyed branch is not taken. + +**Step 2: Run** — `$BR '.venv/bin/python -m pytest -q tests/test_providers.py -k "types or unchanged"'` → FAIL (`failure_category` is `None`). + +**Step 3: Implement** — in `call_model`, each early return / except branch adds the category (error strings untouched): + +```python + except ProviderError as exc: # unresolved adapter + ... error=str(exc), failure_category=exc.category) + if api_key is None: + ... error=msg, failure_category="unkeyed") + except (ProviderError, TransportError) as exc: + ... error=message, + failure_category=exc.category, + http_status=getattr(exc, "http_status", None)) + except Exception as exc: # noqa: BLE001 + ... error=message, failure_category="unexpected") +``` + +Apply the identical four-way mapping to every `yield ModelAnswer(... error=...)` in `call_model_stream`. The "empty response (no streamed content)" yield gets `failure_category="malformed_response"`. + +**Step 4: Run** — same command → PASS. Then `$BR` full suite → 0 failures. + +**Step 5: Commit** +```bash +git add src/conclave/providers.py tests/test_providers.py +git commit -m "feat(providers): carry typed failure_category/http_status on ModelAnswer (DSE-1512)" +``` + +--- + +### Task 3: `synthesizer_chain` config + parsing + `Council` resolution + +**Files:** +- Modify: `src/conclave/config.py:57-95,190-236` +- Modify: `src/conclave/council.py:154-192` +- Modify: `config.example.yml` +- Test: `tests/test_registry_config.py` (append), `tests/test_council.py` (append) + +**Step 1: Failing tests** + +```python +# tests/test_registry_config.py (append) +from conclave.config import parse_synthesizer_chain, _load_config_uncached + + +def test_parse_synthesizer_chain_splits_and_dedupes(): + assert parse_synthesizer_chain("claude>grok > gemini>claude") == ["claude", "grok", "gemini"] + assert parse_synthesizer_chain("claude") == ["claude"] + assert parse_synthesizer_chain(" ") == [] + + +def test_config_synthesizer_chain_from_yaml(tmp_path): + p = tmp_path / "c.yml" + p.write_text("synthesizer: claude\nsynthesizer_chain: [claude, grok]\n") + cfg = _load_config_uncached(p) + assert cfg.synthesizer == "claude" + assert cfg.synthesizer_chain == ["claude", "grok"] + + +def test_config_synthesizer_chain_accepts_arrow_string(tmp_path): + p = tmp_path / "c.yml" + p.write_text("synthesizer_chain: 'claude>grok'\n") + assert _load_config_uncached(p).synthesizer_chain == ["claude", "grok"] + + +def test_config_synthesizer_chain_bad_value_is_empty(tmp_path): + p = tmp_path / "c.yml" + p.write_text("synthesizer_chain: 42\n") + assert _load_config_uncached(p).synthesizer_chain == [] +``` + +```python +# tests/test_council.py (append) +def test_council_chain_defaults_to_single_synthesizer(): + c = Council(models=["grok"], config=ConclaveConfig(synthesizer="claude")) + assert c.synthesizer_chain == ["claude"] and c.synthesizer == "claude" + + +def test_council_chain_from_constructor_string(): + c = Council(models=["grok"], synthesizer="claude>grok", config=ConclaveConfig()) + assert c.synthesizer_chain == ["claude", "grok"] and c.synthesizer == "claude" + + +def test_council_chain_from_constructor_list(): + c = Council(models=["grok"], synthesizer=["gemini", "grok"], config=ConclaveConfig()) + assert c.synthesizer_chain == ["gemini", "grok"] and c.synthesizer == "gemini" + + +def test_council_chain_from_config_overrides_scalar(): + cfg = ConclaveConfig(synthesizer="claude", synthesizer_chain=["grok", "gemini"]) + c = Council(models=["claude"], config=cfg) + assert c.synthesizer_chain == ["grok", "gemini"] and c.synthesizer == "grok" + + +def test_council_constructor_arg_beats_config_chain(): + cfg = ConclaveConfig(synthesizer_chain=["grok", "gemini"]) + c = Council(models=["claude"], synthesizer="claude", config=cfg) + assert c.synthesizer_chain == ["claude"] +``` + +**Step 2: Run** — `$BR '.venv/bin/python -m pytest -q tests/test_registry_config.py tests/test_council.py -k chain'` → FAIL. + +**Step 3: Implement** + +`config.py`: +```python +def parse_synthesizer_chain(spec: str) -> list[str]: + """Split ``"a>b>c"`` into an ordered, de-duplicated candidate list.""" + seen: list[str] = [] + for part in spec.split(">"): + name = part.strip() + if name and name not in seen: + seen.append(name) + return seen + + +def _coerce_chain(value: Any) -> list[str]: + if value is None: + return [] + if isinstance(value, str): + return parse_synthesizer_chain(value) + if isinstance(value, list) and all(isinstance(v, str) for v in value): + return parse_synthesizer_chain(">".join(value)) + logger.warning("synthesizer_chain %r is not a list of names; ignoring", value) + return [] +``` +Add `synthesizer_chain: list[str] = Field(default_factory=list)` to `ConclaveConfig` (docstring: ordered failover ladder; empty means "just `synthesizer`"). In `_load_config_uncached`: `synthesizer_chain=_coerce_chain(raw.get("synthesizer_chain"))`. Update the module docstring example. + +`council.py` `__init__`: change the annotation to `synthesizer: str | Sequence[str] | None = None` and resolve: +```python + self.synthesizer_chain = self._resolve_chain(synthesizer, self.config) + # Back-compat: the primary candidate keeps the historic attribute. + self.synthesizer = self.synthesizer_chain[0] + + @staticmethod + def _resolve_chain(spec: str | Sequence[str] | None, config: ConclaveConfig) -> list[str]: + """constructor arg → config.synthesizer_chain → [config.synthesizer].""" + if isinstance(spec, str): + chain = parse_synthesizer_chain(spec) + elif spec is not None: + chain = parse_synthesizer_chain(">".join(spec)) + else: + chain = list(config.synthesizer_chain) + return chain or [config.synthesizer] +``` +Document the ladder in the class docstring `synthesizer:` entry. Add `synthesizer_chain: [claude, grok]` (commented) to `config.example.yml`. + +**Step 4: Run** → PASS; `$BR` full → 0 failures. + +**Step 5: Commit** — `git commit -m "feat(config): synthesizer_chain ordered failover ladder (DSE-1512)"` + +--- + +### Task 4: Manifest ledger type + `Council.adjudicate` seam + +**Files:** +- Modify: `src/conclave/manifest.py` (new `AdjudicationAttempt`, new manifest field) +- Modify: `src/conclave/council.py` (new `adjudicate`, `_record_adjudication`; `synthesize_blocks` becomes a wrapper) +- Test: `tests/test_adjudication.py` (new) + +**Step 1: Failing tests** + +```python +# tests/test_adjudication.py +"""Council.adjudicate walks the synthesizer chain with the infra-only failover rule.""" + +from __future__ import annotations + +import pytest + +from conclave.config import ConclaveConfig +from conclave.council import Council +from conclave.manifest import AdjudicationAttempt, ModelHarnessManifest +from conclave.models import CouncilResult, ModelAnswer +from conclave import council as council_mod + +CFG = ConclaveConfig(models={"claude": "anthropic/c", "grok": "xai/g", "gemini": "gemini/m"}) + + +def _fail(name, model_id, category, status=None): + return ModelAnswer( + name=name, + model_id=model_id, + error=f"{name} failed", + failure_category=category, + http_status=status, + ) + + +def _ok(name, model_id): + return ModelAnswer( + name=name, model_id=model_id, answer=f"{name} says yes", answer_id=f"{name}-1" + ) + + +def _install(monkeypatch, script: dict[str, ModelAnswer]): + calls = [] + + async def fake(name, model_id, messages, **kw): + calls.append(name) + return script[name] + + monkeypatch.setattr(council_mod, "call_model", fake) + return calls + + +@pytest.fixture +def keys(monkeypatch): + for var in ("ANTHROPIC_API_KEY", "XAI_API_KEY", "GEMINI_API_KEY"): + monkeypatch.setenv(var, "dummy") + + +async def test_chain_of_one_success(monkeypatch, keys): + calls = _install(monkeypatch, {"claude": _ok("claude", "anthropic/c")}) + c = Council(models=["grok"], synthesizer="claude", config=CFG) + out = await c.adjudicate("synthesis", "sys", "user") + assert out.answer.ok and out.name == "claude" and calls == ["claude"] + assert [a.outcome for a in out.attempts] == ["success"] + + +async def test_auth_failure_advances(monkeypatch, keys): + calls = _install( + monkeypatch, + {"claude": _fail("claude", "anthropic/c", "auth", 401), "grok": _ok("grok", "xai/g")}, + ) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG) + out = await c.adjudicate("synthesis", "sys", "user") + assert out.answer.ok and out.name == "grok" and out.model_id == "xai/g" + assert calls == ["claude", "grok"] + assert [(a.candidate, a.outcome, a.failure_category, a.http_status) for a in out.attempts] == [ + ("claude", "failed_over", "auth", 401), + ("grok", "success", None, None), + ] + + +async def test_bad_request_is_terminal(monkeypatch, keys): + calls = _install( + monkeypatch, + { + "claude": _fail("claude", "anthropic/c", "bad_request", 400), + "grok": _ok("grok", "xai/g"), + }, + ) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG) + out = await c.adjudicate("synthesis", "sys", "user") + assert not out.answer.ok and out.name == "claude" + assert calls == ["claude"] # grok was never consulted + assert [a.outcome for a in out.attempts] == ["terminal_failure"] + + +async def test_malformed_is_terminal(monkeypatch, keys): + calls = _install( + monkeypatch, + { + "claude": _fail("claude", "anthropic/c", "malformed_response"), + "grok": _ok("grok", "xai/g"), + }, + ) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG) + out = await c.adjudicate("judge", "sys", "user") + assert calls == ["claude"] and out.attempts[0].outcome == "terminal_failure" + + +async def test_unkeyed_candidate_is_skipped_without_call(monkeypatch): + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + monkeypatch.setenv("XAI_API_KEY", "dummy") + calls = _install(monkeypatch, {"grok": _ok("grok", "xai/g")}) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG) + out = await c.adjudicate("synthesis", "sys", "user") + assert calls == ["grok"] + assert [a.outcome for a in out.attempts] == ["skipped_unkeyed", "success"] + + +async def test_chain_exhausted(monkeypatch, keys): + calls = _install( + monkeypatch, + { + "claude": _fail("claude", "anthropic/c", "quota", 429), + "grok": _fail("grok", "xai/g", "unavailable", 503), + }, + ) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG) + out = await c.adjudicate("synthesis", "sys", "user") + assert calls == ["claude", "grok"] + assert not out.answer.ok and out.name == "grok" + assert [a.outcome for a in out.attempts] == ["failed_over", "exhausted"] + + +async def test_all_unkeyed_returns_no_answer(monkeypatch): + for var in ("ANTHROPIC_API_KEY", "XAI_API_KEY"): + monkeypatch.delenv(var, raising=False) + calls = _install(monkeypatch, {}) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG) + out = await c.adjudicate("synthesis", "sys", "user") + assert out.answer is None and calls == [] + assert [a.outcome for a in out.attempts] == ["skipped_unkeyed", "skipped_unkeyed"] + + +def test_record_adjudication_appends_ledger_and_receipts(): + c = Council(models=["grok"], synthesizer="claude", config=CFG) + result = CouncilResult( + prompt="p", + manifest=ModelHarnessManifest(request_id="r", conclave_version="t", mode="synthesize"), + ) + attempts = [ + AdjudicationAttempt( + role="synthesis", + candidate="claude", + model_id="anthropic/c", + attempt_index=1, + outcome="failed_over", + failure_category="auth", + http_status=401, + ), + AdjudicationAttempt( + role="synthesis", candidate="grok", model_id="xai/g", attempt_index=2, outcome="success" + ), + ] + called = [_fail("claude", "anthropic/c", "auth", 401), _ok("grok", "xai/g")] + c._record_adjudication(result, attempts, called, phase="synthesis") + m = result.manifest + assert m.adjudication_succession == attempts + assert [(r.phase, r.attempt, r.outcome) for r in m.receipts] == [ + ("synthesis", 1, "failed"), + ("synthesis", 2, "success"), + ] + assert m.secret_safety == "verified_no_secrets" + + +def test_ledger_has_no_free_text_fields(): + fields = set(AdjudicationAttempt.model_fields) + assert fields == { + "role", + "candidate", + "model_id", + "attempt_index", + "outcome", + "failure_category", + "http_status", + } +``` + +**Step 2: Run** — `$BR '.venv/bin/python -m pytest -q tests/test_adjudication.py'` → FAIL (`ImportError: AdjudicationAttempt`). + +**Step 3: Implement** + +`manifest.py` — add before `ProviderExecutionReceipt`: +```python +AdjudicationRole = Literal["synthesis", "debate_final", "judge", "verdict_extraction"] +AdjudicationAttemptOutcome = Literal[ + "success", # this candidate adjudicated + "failed_over", # infra failure; the next candidate was tried + "exhausted", # infra failure on the LAST candidate; nothing left to try + "terminal_failure", # the candidate answered unusably; failover refused by rule + "skipped_unkeyed", # no API key in the environment; no call made +] + + +class AdjudicationAttempt(BaseModel): + """One step of the synthesizer/judge succession ladder (DSE-1512). + + Deliberately carries NO free text: only bounded categories and an HTTP + status. A raw provider error string could contain words the secret-safety + scan forbids (e.g. "authorization"), which would un-verify the manifest. + """ + + role: AdjudicationRole + candidate: str + model_id: str + attempt_index: int = Field(ge=1) + outcome: AdjudicationAttemptOutcome + failure_category: str | None = None + http_status: int | None = None +``` +Add to `ModelHarnessManifest` (after `redacted_errors`, documented in the docstring): +```python + adjudication_succession: list[AdjudicationAttempt] = Field(default_factory=list) +``` + +`council.py` — a small dataclass + the seam: +```python +@dataclass +class AdjudicationOutcome: + """Return value of :meth:`Council.adjudicate`.""" + + answer: ( + ModelAnswer | None + ) # success, or the terminal/exhausted failure; None if no candidate could be called + attempts: list[AdjudicationAttempt] + called: list[ModelAnswer] # every real call, in order (for receipts) + + @property + def name(self) -> str | None: + return self.answer.name if self.answer is not None else None + + @property + def model_id(self) -> str | None: + return self.answer.model_id if self.answer is not None else None +``` +```python +async def adjudicate( + self, role: AdjudicationRole, system_prompt: str, user_content: str +) -> AdjudicationOutcome: + """Walk ``synthesizer_chain`` for one adjudication role (DSE-1512). + + Rule: a candidate is tried in declared order; an unkeyed candidate is + skipped without a call; a call that fails with a category in + :data:`FAILOVER_CATEGORIES` advances to the next candidate; ANY other + failure is terminal for the role (a model that answered is never + second-guessed by another vendor -- that would let adjudication shop for + a result). No scoring, no health tracking: the order is the operator's. + """ + messages = [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_content}, + ] + attempts: list[AdjudicationAttempt] = [] + called: list[ModelAnswer] = [] + last_failure: ModelAnswer | None = None + chain = self.synthesizer_chain + for index, candidate in enumerate(chain, start=1): + model_id = self.config.resolve_model_id(candidate) + if not key_present(model_id): + attempts.append( + AdjudicationAttempt( + role=role, + candidate=candidate, + model_id=model_id, + attempt_index=index, + outcome="skipped_unkeyed", + failure_category="unkeyed", + ) + ) + continue + answer = await call_model( + candidate, + model_id, + messages, + config=self.config, + temperature=self.temperature, + timeout=self.timeout, + ) + called.append(answer) + if answer.ok: + attempts.append( + AdjudicationAttempt( + role=role, + candidate=candidate, + model_id=model_id, + attempt_index=index, + outcome="success", + ) + ) + return AdjudicationOutcome(answer=answer, attempts=attempts, called=called) + category = answer.failure_category + if category in FAILOVER_CATEGORIES: + is_last = index == len(chain) + attempts.append( + AdjudicationAttempt( + role=role, + candidate=candidate, + model_id=model_id, + attempt_index=index, + outcome="exhausted" if is_last else "failed_over", + failure_category=category, + http_status=answer.http_status, + ) + ) + last_failure = answer + if not is_last: + logger.warning( + "%s: '%s' failed (%s); trying next candidate", role, candidate, category + ) + continue + attempts.append( + AdjudicationAttempt( + role=role, + candidate=candidate, + model_id=model_id, + attempt_index=index, + outcome="terminal_failure", + failure_category=category, + http_status=answer.http_status, + ) + ) + return AdjudicationOutcome(answer=answer, attempts=attempts, called=called) + return AdjudicationOutcome(answer=last_failure, attempts=attempts, called=called) + + +def _record_adjudication( + self, + result: CouncilResult, + attempts: list[AdjudicationAttempt], + called: list[ModelAnswer], + *, + phase: str, + record_receipts: bool = True, + protocol_version: str | None = None, + prompt_version: str | None = SYNTHESIS_PROMPT_VERSION, +) -> None: + """Append the succession ledger (always) and one receipt per real call.""" + if result.manifest is None: + return + result.manifest.adjudication_succession.extend(attempts) + receipts = ( + [ + receipt_from_answer( + answer, + temperature=self.temperature, + timeout=self.timeout, + phase=phase, + attempt=i, + protocol_version=protocol_version, + prompt_version=prompt_version, + ) + for i, answer in enumerate(called, start=1) + ] + if record_receipts + else [] + ) + if receipts: + result.manifest.receipts.extend(receipts) + self._recompute_manifest_accounting(result.manifest) # re-stamps secret_safety +``` +`synthesize_blocks` becomes a compatibility wrapper: `out = await self.adjudicate("synthesis", ...)`; return `out.answer` or a `ModelAnswer(name=self.synthesizer, model_id=resolve(self.synthesizer), error="no candidate in synthesizer chain has an API key")`. Import `FAILOVER_CATEGORIES` from `.models` and `AdjudicationAttempt, AdjudicationRole` from `.manifest`; `from dataclasses import dataclass`. + +Attempt numbering for receipts: `attempt=i` where `i` counts *real calls*, matching how `extract_verdict` numbers its attempts. + +**Step 4: Run** → PASS; `$BR` full → 0 failures (nothing calls `adjudicate` yet). + +**Step 5: Commit** — `git commit -m "feat(council): adjudicate() succession seam + manifest ledger (DSE-1512)"` + +--- + +### Task 5: Route synthesis, debate final, adversarial judge, and Elite through the seam + +**Files:** +- Modify: `src/conclave/council.py` (`_synthesize`, `_ask_uncached`, `elite`) +- Modify: `src/conclave/modes.py` (`_debate_synthesize`, `run_debate`, `_adversarial_judge`, `run_adversarial`) +- Test: `tests/test_council.py`, `tests/test_modes.py`, `tests/test_manifest_all_modes.py` (append) + +**Step 1: Failing tests** — one per mode; reuse the `_install`/`_fail`/`_ok` helpers by moving them into `tests/conftest.py` as `make_failed_answer(name, model_id, category, status=None)` / `make_ok_answer(name, model_id)` and a `patch_council_call_model(monkeypatch, script)` helper. Members need a script entry too (they go through the same council seam): give them `_ok`. + +```python +async def test_synthesize_mode_fails_over_and_is_not_degraded(monkeypatch, keys): + patch_council_call_model(monkeypatch, {"gemini": ok("gemini","gemini/m"), "claude": fail("claude","anthropic/c","quota",402), "grok": ok("grok","xai/g")}) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG, extract_verdict=False) + r = await c.ask("q") + assert r.synthesis == "grok says yes" and r.synthesis_error is None and r.degraded is False + assert (r.synthesizer, r.synthesizer_model_id) == ("grok", "xai/g") + ledger = r.manifest.adjudication_succession + assert [(a.role, a.candidate, a.outcome) for a in ledger] == [("synthesis","claude","failed_over"), ("synthesis","grok","success")] + assert [(x.phase, x.attempt) for x in r.manifest.receipts if x.phase == "synthesis"] == [("synthesis",1), ("synthesis",2)] + assert r.manifest.secret_safety == "verified_no_secrets" + +async def test_synthesize_mode_exhausted_is_degraded(monkeypatch, keys): + ... claude quota, grok unavailable → r.synthesis is None, r.synthesis_error == "grok failed", r.degraded is True, + ledger outcomes == ["failed_over", "exhausted"] + +async def test_synthesize_chain_of_one_message_unchanged(monkeypatch, clear_keys, patch_call_model): + # exact string from the existing test_synthesizer_without_key_returns_raw must still hold + ... + +async def test_synthesize_chain_all_unkeyed_message(monkeypatch): + # no keys for claude or grok, gemini keyed: + assert r.synthesis_error == "synthesizer chain [claude, grok] has no API key for any candidate; returning raw answers only" + assert [a.outcome for a in r.manifest.adjudication_succession] == ["skipped_unkeyed","skipped_unkeyed"] + +async def test_debate_final_fails_over(monkeypatch, keys): + # 2 members ok, chain claude(auth)>grok(ok), rounds=1 → r.synthesis from grok, ledger role == "debate_final" + +async def test_adversarial_judge_fails_over(monkeypatch, keys): + # proposer+critic ok, chain claude(503)>grok(ok) → r.adversarial.verdict from grok, + # (r.adversarial.judge, r.adversarial.judge_model_id) == ("grok","xai/g"), r.degraded False, ledger role == "judge" + +async def test_adversarial_judge_terminal_does_not_fail_over(monkeypatch, keys): + # claude bad_request → adv.verdict_error set, grok never called, r.degraded True + +async def test_elite_synthesis_fails_over(monkeypatch, keys): + # 3 members ok through all phases (script ok for gemini/claude/grok as MEMBERS), chain "openai>grok" + # with openai 429 → r.synthesis from grok, ledger role "synthesis", manifest has elite receipts + 2 synthesis receipts +``` +Note for Elite: the synthesizer candidates must not collide with member names in the script dict unless you want the member to fail too — use distinct names (`openai`, `mistral`) for the chain and give them `models` entries in `CFG`. + +**Step 2: Run** → FAIL. + +**Step 3: Implement** + +`Council._synthesize`: +```python + usable = result.successful_answers + if not usable: + result.synthesis_error = "no successful member answers to synthesize"; ...; return None + + keyed = [c for c in self.synthesizer_chain if key_present(self.config.resolve_model_id(c))] + result.synthesizer = self.synthesizer + result.synthesizer_model_id = self.config.resolve_model_id(self.synthesizer) + if not keyed: + if len(self.synthesizer_chain) == 1: + result.synthesis_error = (f"synthesizer '{self.synthesizer}' ({result.synthesizer_model_id}) has no API key; returning raw answers only") + else: + result.synthesis_error = (f"synthesizer chain [{', '.join(self.synthesizer_chain)}] has no API key for any candidate; returning raw answers only") + logger.warning(result.synthesis_error) + self._record_adjudication(result, self._skipped_attempts("synthesis"), [], phase="synthesis") + return None + + ...build blocks/user_content exactly as today... + outcome = await self.adjudicate("synthesis", _SYNTH_SYSTEM, user_content) + result.synthesizer, result.synthesizer_model_id = outcome.name, outcome.model_id + answer = outcome.answer + if answer is not None and answer.ok: + result.synthesis = answer.answer + elif answer is not None: + result.synthesis_error = answer.error + self._record_adjudication(result, outcome.attempts, outcome.called, phase="synthesis", + protocol_version=(ELITE_PROTOCOL_VERSION if result.mode == "elite" else None)) + return answer +``` +`_skipped_attempts(role)` builds one `skipped_unkeyed` attempt per chain candidate. In `_ask_uncached` **delete** the manual `_append_manifest_receipts([...synthesis receipt...])` block — `_record_adjudication` now owns the synthesis receipts (otherwise they double). Same deletion in `elite()`, and move `self._ensure_manifest(result, "elite")` to **before** `self._synthesize(result)` so the ledger has a manifest to land on. + +`modes.run_debate`: insert `council._ensure_manifest(result, "debate")` immediately before `await _debate_synthesize(council, result)`. `_debate_synthesize` mirrors `_synthesize` (keyed check with the chain-of-one message preserved: `"...has no API key; returning final-round answers only"`), then `outcome = await council.adjudicate("debate_final", prompts.DEBATE_FINAL_SYSTEM, user_content)` and `council._record_adjudication(result, outcome.attempts, outcome.called, phase="debate_final", prompt_version=None)`. + +`modes.run_adversarial`: insert `council._ensure_manifest(result, "adversarial")` immediately before `await _adversarial_judge(council, prompt, adv)`. `_adversarial_judge(council, prompt, adv, result)` — add the `result` parameter so it can record; keep the proposal-failed and unkeyed short-circuits (chain-aware wording as above, `"...returning proposal and critiques only"`); then `outcome = await council.adjudicate("judge", prompts.JUDGE_SYSTEM, user_content)`; `adv.judge, adv.judge_model_id = outcome.name, outcome.model_id`; verdict / verdict_error from the answer; `council._record_adjudication(result, outcome.attempts, outcome.called, phase="judge", prompt_version=None)`. + +`_degrade_to_synthesize` needs no change (`_synthesize` returns before consulting anyone when there are no usable answers). + +**Step 4: Run** the new tests → PASS. Then `$BR` full. Expect `tests/test_manifest_all_modes.py` to flag that `debate`/`adversarial` manifests now carry a judge/synthesizer receipt they previously lacked. That is a **deliberate, documented improvement** (H0 principle: every real call gets a receipt). Update those assertions (count + phase) — do not weaken them — and add a line to the CHANGELOG entry in Task 9. + +**Step 5: Commit** — `git commit -m "feat(council,modes): route all adjudication roles through the succession seam (DSE-1512)"` + +--- + +### Task 6: Verdict extraction succession + +**Files:** +- Modify: `src/conclave/verdict_synthesis.py` (`VerdictSynthesisResult`, tail of `extract_verdict`) +- Modify: `src/conclave/council.py` (`_apply_verdict`) +- Test: `tests/test_council_verdict.py` (append) + +**Step 1: Failing tests** — drive the verdict seam (`conclave.verdict_synthesis.call_model`) with a handler that branches on `name`: primary returns an errored `ModelAnswer(failure_category="auth", http_status=401)` for both the initial and repair calls; successor returns valid extraction JSON (copy the fixture JSON `test_council_verdict.py` already uses). + +```python +async def test_verdict_extraction_fails_over_to_successor(...): + r = await Council(models=[...], synthesizer="claude>grok", config=CFG).ask("Should we X?") + assert r.verdict is not None + assert r.manifest.verdict_extraction.model_id == "xai/g" + ledger = [a for a in r.manifest.adjudication_succession if a.role == "verdict_extraction"] + assert [(a.candidate, a.outcome, a.failure_category) for a in ledger] == [("claude","failed_over","auth"), ("grok","success",None)] + phases = [(x.phase, x.name) for x in r.manifest.receipts if x.phase.startswith("verdict")] + assert phases == [("verdict_extraction","claude"), ("verdict_repair","claude"), ("verdict_extraction","grok")] + +async def test_verdict_extraction_schema_failure_is_terminal(...): + # primary returns prose twice (schema_invalid), successor would return valid JSON → successor NOT called, + # r.verdict is None, verdict_absent_reason == "verdict extraction failed schema validation", + # ledger == [("claude","terminal_failure","malformed_response")] +``` + +**Step 2: Run** → FAIL. + +**Step 3: Implement** + +`VerdictSynthesisResult` gains: +```python + failure_category: str | None = None + http_status: int | None = None +``` +populated only on the `_REASON_EXTRACTION_FAILED` return: take them from the **last** attempt that errored (`retry` if the repair ran and `retry.error`, else `answer` if `answer.error`); when the last attempt *responded* but failed validation, set `failure_category="malformed_response"`. + +`Council._apply_verdict` — replace the single call with a chain walk (verdict extraction receipts are already produced per attempt by `extract_verdict`; append them per candidate exactly as today): +```python + chain = self.synthesizer_chain + attempts: list[AdjudicationAttempt] = [] + vsr = None + for index, candidate in enumerate(chain, start=1): + model_id = self.config.resolve_model_id(candidate) + if not key_present(model_id): + attempts.append(skipped_unkeyed attempt); continue + vsr = await extract_verdict_fn(result.prompt, result.answers, synthesizer_name=candidate, + synthesizer_model_id=model_id, config=self.config, + temperature=self.temperature, timeout=self.timeout, + protocol_version=(ELITE_PROTOCOL_VERSION if result.mode == "elite" else None)) + if record_receipts: + self._append_manifest_receipts(result, vsr.attempt_receipts) + if vsr.verdict_absent_reason == _REASON_EXTRACTION_FAILED and vsr.failure_category in FAILOVER_CATEGORIES: + is_last = index == len(chain) + attempts.append(failed_over / exhausted attempt with vsr.failure_category, vsr.http_status) + continue + attempts.append(success attempt if vsr.verdict is not None or vsr.verdict_absent_reason in (N<2, open-ended) + else terminal_failure attempt with failure_category=vsr.failure_category) + break + if vsr is None: # every candidate unkeyed -- today's behaviour was to call anyway and fail; keep verdict absent + ...set result.manifest.verdict_absent_reason = "verdict extractor has no API key" ; record attempts; return + ...hoist vsr fields exactly as today... + if result.manifest is not None: + result.manifest.adjudication_succession.extend(attempts) + ...existing provenance writes + re-stamp... +``` +Import `_REASON_EXTRACTION_FAILED` from `verdict_synthesis` (it is module-private today; export it as `REASON_EXTRACTION_FAILED` and keep the old name as an alias). Note the N<2 gate makes no call and must count as `success` for the ledger only when it is the *first* keyed candidate (it will always be — the gate is answer-driven, not model-driven); simplest: if `vsr.attempt_receipts` is empty, record the attempt as `success` and break. + +**Step 4: Run** → PASS; `$BR` full → 0 failures. + +**Step 5: Commit** — `git commit -m "feat(verdict): fail verdict extraction over on infra errors (DSE-1512)"` + +--- + +### Task 7: Streaming parity + +**Files:** +- Modify: `src/conclave/streaming.py:252-322` +- Test: `tests/test_streaming.py` (append) + +**Step 1: Failing tests** — patch `conclave.streaming.call_model_stream` with an async generator that, for the primary, yields **only** a final errored `ModelAnswer(failure_category="quota", http_status=429)` (no deltas), and for the successor yields deltas then an ok answer. + +```python +async def test_stream_synthesis_fails_over_before_first_delta(...): + events = [e async for e in council.ask_stream("q")] + deltas = [e.text for e in events if e.type == "synthesis_delta"] + done = [e for e in events if e.type == "synthesis_done"] + assert "".join(deltas) == "grok says yes" and len(done) == 1 and done[0].name == "grok" + result = events[-1].result + assert result.synthesis == "grok says yes" and result.degraded is False + assert [(a.role, a.candidate, a.outcome) for a in result.manifest.adjudication_succession][:2] == [("synthesis","claude","failed_over"), ("synthesis","grok","success")] + +async def test_stream_synthesis_does_not_fail_over_after_deltas(...): + # primary yields one delta THEN an errored answer with category "unavailable" → terminal; successor never consulted; + # result.synthesis_error set; ledger outcome "terminal_failure" +``` + +**Step 2: Run** → FAIL. + +**Step 3: Implement** `_stream_synthesis`: keep the usable/keyed short-circuits (chain-aware, same messages as `_synthesize`); then loop the chain; per candidate stream; track `emitted = False`; set `emitted = True` on the first delta; on the final answer: ok → success/break; error with `failure_category in FAILOVER_CATEGORIES and not emitted` → `failed_over` (or `exhausted` on the last), continue; else → `terminal_failure`, break. Yield a single `synthesis_done` for the last consulted candidate. Set `result.synthesizer`/`synthesizer_model_id` to that candidate. Record via `council._record_adjudication(result, attempts, called, phase="synthesis", record_receipts=False)` — streaming keeps its documented "no synthesis receipt" contract; the ledger is not a receipt. + +**Step 4: Run** → PASS; `$BR` full → 0 failures. + +**Step 5: Commit** — `git commit -m "feat(streaming): synthesis succession before the first delta (DSE-1512)"` + +--- + +### Task 8: Cache identity + no-store on succession + +**Files:** +- Modify: `src/conclave/cache.py:59,148-296` +- Modify: `src/conclave/council.py` (`_cache_key`, `_cached_run`) +- Test: `tests/test_cache.py` (append) + +**Step 1: Failing tests** + +```python +def test_identity_includes_full_chain(): + a = make_key(prompt="p", mode="synthesize", members=[("g","xai/g")], synthesizer="claude", synthesizer_model_id="anthropic/c", synthesizer_chain=[("claude","anthropic/c")], temperature=0.7) + b = make_key(..., synthesizer_chain=[("claude","anthropic/c"), ("grok","xai/g")]) + assert a != b + +def test_cache_format_version_bumped(): + assert CACHE_FORMAT_VERSION == "4" + +async def test_result_adjudicated_by_successor_is_not_stored(monkeypatch, tmp_path, keys): + # cache on, claude auth-fails, grok succeeds → run twice; second run must call the providers again + # (cache_mod.store was skipped); assert both results have cached is False and the counting fake saw 2x calls +``` + +**Step 2: Run** → FAIL. + +**Step 3: Implement** — `build_identity`/`make_key` gain `synthesizer_chain: list[tuple[str, str]] | None = None` and write `"synthesizer_chain": [[n, m] ...]` (keep the existing `"synthesizer"` key). `CACHE_FORMAT_VERSION = "4"`. `Council._cache_key` passes `[(c, self.config.resolve_model_id(c)) for c in self.synthesizer_chain]` and includes every chain prefix in `used_prefixes`. In `_cached_run`, before `cache_mod.store`, skip the store when the run's succession ledger records that the **primary** candidate itself failed for an infrastructure reason — not just "a successor adjudicated". That is a strictly wider condition: it also covers a chain of one whose sole candidate exhausted the ladder (no successor to speak of), which the narrower "any `failed_over` attempt" check below would miss entirely: +```python + if result.primary_failed_over: + logger.info( + "not caching %s run: primary adjudicator failed for an infrastructure reason", + mode, + ) + return result +``` +`primary_failed_over` is `True` iff, for any role, the attempt at `attempt_index == 1` (the chain's declared primary) has `outcome` in `{"failed_over", "exhausted", "skipped_unkeyed"}` — i.e. the primary did not itself adjudicate, whether because of a live infrastructure failure or because it had no key. This was promoted to a `CouncilResult.primary_failed_over` computed field in the Unit F review (after this task originally landed it as a private `_primary_failed_over(result)` module function), and further review-corrected to this uniform, index-1-based rule: the original two-outcome version (`"failed_over"` **or** `"exhausted"` anywhere in the ledger) left `"skipped_unkeyed"` uncounted, so an unkeyed primary was reported inconsistently depending on which adjudication roles ran and whether verdict extraction (which deliberately calls an unkeyed candidate and records `"failed_over"`/`"exhausted"` instead of `"skipped_unkeyed"`) happened to be enabled. A `"terminal_failure"` entry at index 1 (a candidate answered, just not usably) does **not** skip the store — re-running would not produce a different, better answer, so it stays cacheable exactly as before. Consequence for a chain of one: a degraded run whose sole synthesizer had no key or errored for an infrastructure reason used to be cached under v1.3.0 and is not anymore. + +**Step 4: Run** → PASS; `$BR` full → 0 failures. + +**Step 5: Commit** — `git commit -m "feat(cache): chain in identity; never cache a run whose primary failed over (DSE-1512)"` + +--- + +### Task 9: CLI surface + docs + changelog + +**Files:** +- Modify: `src/conclave/cli.py:497-500` (help), `:741` (`providers` footer) +- Modify: `README.md` (synthesizer section), `docs/PRODUCT_DESIGN_DOCUMENT.md` §4a (manifest table + a short "Adjudication succession" paragraph), `CHANGELOG.md` (Unreleased), `DOCUMENTATION_INDEX.md` (link this plan), `SYSTEM_CONTEXT_DIAGRAM.md` (only if it enumerates manifest fields) +- Test: `tests/test_cli.py` (append) + +**Step 1: Failing tests** +```python +def test_cli_synthesizer_chain_parses_and_exits_zero_on_successor(...): + # patch council seam: claude 402, grok ok; run `ask q -c gemini -s "claude>grok" --json` + assert result.exit_code == 0 and payload["degraded"] is False and payload["synthesizer"] == "grok" + assert payload["manifest"]["adjudication_succession"][0]["outcome"] == "failed_over" + +def test_cli_chain_exhausted_exits_degraded(...): + assert result.exit_code == cli._DEGRADED_EXIT_CODE +``` + +**Step 2: Run** → FAIL only if the CLI mangles the `>`; otherwise these pass immediately — that is fine, keep them as the contract. + +**Step 3: Implement** — `--synthesizer` help: `"Synthesizer/judge model name, or an ordered failover ladder 'claude>grok>gemini' (DSE-1512): the next candidate is tried only on auth/quota/5xx/timeout/network failures."` `conclave providers` footer prints `synthesizer chain: a > b` when `cfg.synthesizer_chain` is non-empty. Docs: + +- **CHANGELOG `[Unreleased]` → Added:** "Adjudication succession (DSE-1512)" — chain config/CLI, the infra-only rule, `manifest.adjudication_succession`, per-role application, cache no-store rule, `ModelAnswer.failure_category`/`http_status`; **Changed:** debate/adversarial manifests now carry the judge/synthesizer receipt(s) they previously omitted; cache format version 3 → 4 (old entries miss safely); **Not changed:** the verdict repair retry still runs on the same model after an infra error (follow-up). +- **README:** a "Synthesizer failover" subsection with the CLI form, the YAML form, the rule table (which categories advance), and one sentence on reading `manifest.adjudication_succession`. +- **PDD §4a:** add `adjudication_succession` to the manifest description; add the failover rule as a design decision (why content failures never fail over: reproducibility). +- **DOCUMENTATION_INDEX.md:** add `docs/plans/2026-09-03-adjudication-succession.md`. + +**Step 4: Run** `$BR` full + ruff. **Step 5: Commit** — `git commit -m "docs(cli): synthesizer chain surface, changelog, PDD §4a (DSE-1512)"` + +--- + +### Task 10: Ship + +1. `git push -u origin feat/dse-1512-adjudication-succession` (from the laptop). +2. `gh pr create --title "feat: adjudication succession — judge/synthesizer failover on infrastructure errors (DSE-1512)" --body-file ` — body: summary, the rule table, the receipt-completeness change, the follow-up note, `Closes DSE-1512`, the required attribution footer. +3. Wait for `Test` (3.11/3.12/3.13), `ruff`, `pip-audit`, `Gitleaks` — all green. +4. `python3 ~/.claude/scripts/release_control.py classify` on the complete diff → must be `routine-non-security`. +5. `python3 ~/.claude/scripts/release_control.py merge --repo DataScience-EngineeringExperts/conclave --pr --head-sha <40> --method squash`. +6. Linear DSE-1512 → Done with the merge SHA. diff --git a/src/conclave/adapters/__init__.py b/src/conclave/adapters/__init__.py index 3ddf380..114aa71 100644 --- a/src/conclave/adapters/__init__.py +++ b/src/conclave/adapters/__init__.py @@ -75,7 +75,9 @@ def resolve_adapter(model_id: str, config: ConclaveConfig | None = None) -> Prov Raises: ProviderError: When the prefix is unknown and no custom endpoint declares - it. The message names the prefix and the remedy. + it. The message names the prefix and the remedy. ``category`` is + ``"unresolved"`` (DSE-1512) -- no call was ever made, so a failover + ladder treats it the same as any other infrastructure failure. """ prefix = provider_prefix(model_id) @@ -96,5 +98,6 @@ def resolve_adapter(model_id: str, config: ConclaveConfig | None = None) -> Prov raise ProviderError( f"unknown provider '{prefix}' for model '{model_id}': no built-in adapter " "and no custom OpenAI-compatible endpoint declared in config " - "(add it under 'endpoints:' in ~/.conclave/config.yml)" + "(add it under 'endpoints:' in ~/.conclave/config.yml)", + category="unresolved", ) diff --git a/src/conclave/adapters/anthropic.py b/src/conclave/adapters/anthropic.py index df6afaf..7c64c16 100644 --- a/src/conclave/adapters/anthropic.py +++ b/src/conclave/adapters/anthropic.py @@ -30,7 +30,7 @@ import json from ..logging import get_logger -from ..models import TokenUsage +from ..models import TokenUsage, categorize_http_status from ..provider_catalog import capabilities_for from ..registry import PROVIDER_ENV_VARS from .base import OutputContract, ProviderError, SSEDelta, status_error @@ -200,7 +200,9 @@ def parse_response(self, status: int, payload: object) -> tuple[str, TokenUsage """ if status < 200 or status >= 300: raise ProviderError( - status_error("anthropic", status, payload, secondary_keys=("type",)) + status_error("anthropic", status, payload, secondary_keys=("type",)), + category=categorize_http_status(status), + http_status=status, ) if not isinstance(payload, dict): raise ProviderError(f"anthropic: non-JSON response body (status {status})") diff --git a/src/conclave/adapters/base.py b/src/conclave/adapters/base.py index 35bb952..e2d19b1 100644 --- a/src/conclave/adapters/base.py +++ b/src/conclave/adapters/base.py @@ -30,7 +30,7 @@ from pydantic import BaseModel, Field -from ..models import TokenUsage +from ..models import FailureCategory, TokenUsage from ..registry import PROVIDER_ENV_VARS # Matches "Bearer sk-abc123" / "Bearer xai-..." auth headers echoed into errors. @@ -241,11 +241,24 @@ class ProviderError(Exception): """A provider-side failure: non-2xx status or a malformed/empty payload. The message passed in is redacted on construction, so the stored message is - always safe to place in ``ModelAnswer.error`` and to log. + always safe to place in ``ModelAnswer.error`` and to log. ``category`` / + ``http_status`` are typed at the raise site (DSE-1512) so failover never + depends on the message text. ``category`` defaults to + ``"malformed_response"`` -- the shape every existing raise site in this + package had before this field existed (a 2xx response with an unusable + payload); the non-2xx raise sites pass an explicit status-derived category. """ - def __init__(self, message: str) -> None: + def __init__( + self, + message: str, + *, + category: FailureCategory = "malformed_response", + http_status: int | None = None, + ) -> None: super().__init__(redact(message)) + self.category: FailureCategory = category + self.http_status = http_status @runtime_checkable diff --git a/src/conclave/adapters/gemini.py b/src/conclave/adapters/gemini.py index 35d0b39..c5b629e 100644 --- a/src/conclave/adapters/gemini.py +++ b/src/conclave/adapters/gemini.py @@ -38,7 +38,7 @@ import json import warnings -from ..models import TokenUsage +from ..models import TokenUsage, categorize_http_status from ..provider_catalog import capabilities_for from ..registry import PROVIDER_ENV_VARS from .base import OutputContract, ProviderError, SSEDelta, status_error @@ -342,7 +342,11 @@ def build_request( def parse_response(self, status: int, payload: object) -> tuple[str, TokenUsage | None]: """Concatenate the first candidate's text parts. See base protocol.""" if status < 200 or status >= 300: - raise ProviderError(status_error("gemini", status, payload, secondary_keys=("status",))) + raise ProviderError( + status_error("gemini", status, payload, secondary_keys=("status",)), + category=categorize_http_status(status), + http_status=status, + ) if not isinstance(payload, dict): raise ProviderError(f"gemini: non-JSON response body (status {status})") diff --git a/src/conclave/adapters/openai_compat.py b/src/conclave/adapters/openai_compat.py index 6ec9c04..3d826dd 100644 --- a/src/conclave/adapters/openai_compat.py +++ b/src/conclave/adapters/openai_compat.py @@ -26,7 +26,7 @@ import json from ..logging import get_logger -from ..models import TokenUsage +from ..models import TokenUsage, categorize_http_status from ..provider_catalog import capabilities_for from .base import OutputContract, ProviderError, SSEDelta, status_error @@ -214,7 +214,9 @@ def parse_response(self, status: int, payload: object) -> tuple[str, TokenUsage """ if status < 200 or status >= 300: raise ProviderError( - status_error(self.prefix, status, payload, secondary_keys=("type",)) + status_error(self.prefix, status, payload, secondary_keys=("type",)), + category=categorize_http_status(status), + http_status=status, ) if not isinstance(payload, dict): raise ProviderError(f"{self.prefix}: non-JSON response body (status {status})") diff --git a/src/conclave/cache.py b/src/conclave/cache.py index 94a3a17..ebcf106 100644 --- a/src/conclave/cache.py +++ b/src/conclave/cache.py @@ -6,7 +6,10 @@ stored payload are derived solely from the exact prompt content, the ordered council member friendly-names + resolved model ids, the run mode, the synthesizer/judge identity, and the mode parameters that affect output. No environment variable is -read here; no key value reaches the key string or the on-disk artifact. +read here; no key value reaches the key string or the on-disk artifact. Identity +also carries the full ordered synthesizer/judge failover chain (DSE-1512), not +just the primary candidate, so changing any candidate anywhere in the ladder +invalidates a prior entry -- see :func:`build_identity`. Storage ======= @@ -38,7 +41,7 @@ import hashlib import json import os -from collections.abc import Mapping +from collections.abc import Mapping, Sequence from pathlib import Path from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit @@ -56,7 +59,9 @@ # Bumped if the cache-key composition or stored schema changes incompatibly, so # old entries simply miss instead of being mis-served against new code. -CACHE_FORMAT_VERSION = "3" +# v4 (DSE-1512): identity now carries the full ordered synthesizer/judge chain, +# not just the primary candidate. +CACHE_FORMAT_VERSION = "4" _SECRET_QUERY_PARTS = ( "authorization", "auth", @@ -161,6 +166,7 @@ def build_identity( extract_verdict: bool = True, endpoint_urls: Mapping[str, str] | None = None, source_bundle_digest: str | None = None, + synthesizer_chain: Sequence[tuple[str, str]] | None = None, cache_format_version: str = CACHE_FORMAT_VERSION, protocol_version: str = ELITE_PROTOCOL_VERSION, synthesis_prompt_version: str = SYNTHESIS_PROMPT_VERSION, @@ -173,7 +179,19 @@ def build_identity( Raw endpoint URLs and source bundle values never enter the returned document; only sanitized one-way fingerprints do. API keys are not accepted and no environment value is read here. + + ``synthesizer_chain`` (DSE-1512) is the full ordered ladder of + ``(friendly_name, resolved_model_id)`` candidates tried for the + synthesizer/judge role, not just the primary. When omitted it defaults to a + chain of one built from ``synthesizer``/``synthesizer_model_id``, so a + direct caller that never passes it still gets a stable, meaningful value + and existing single-candidate callers are unaffected. """ + chain = ( + list(synthesizer_chain) + if synthesizer_chain is not None + else [(synthesizer, synthesizer_model_id)] + ) payload: dict[str, object] = { "versions": { "cache_format": cache_format_version, @@ -188,6 +206,9 @@ def build_identity( # Pairs as lists so JSON round-trips; order preserved deliberately. "members": [[name, model_id] for name, model_id in members], "synthesizer": [synthesizer, synthesizer_model_id], + # The full ordered failover ladder (DSE-1512); a chain of one is + # byte-for-byte equivalent to the legacy "synthesizer" pair above. + "synthesizer_chain": [[name, model_id] for name, model_id in chain], "generation": {"temperature": temperature, "timeout": timeout}, "extract_verdict": extract_verdict, "endpoint_fingerprints": { @@ -229,6 +250,7 @@ def make_key( extract_verdict: bool = True, endpoint_urls: Mapping[str, str] | None = None, source_bundle_digest: str | None = None, + synthesizer_chain: Sequence[tuple[str, str]] | None = None, cache_format_version: str = CACHE_FORMAT_VERSION, protocol_version: str = ELITE_PROTOCOL_VERSION, synthesis_prompt_version: str = SYNTHESIS_PROMPT_VERSION, @@ -244,7 +266,8 @@ def make_key( * run mode, * ordered ``(friendly_name, resolved_model_id)`` member pairs (order matters -- see module docstring), - * synthesizer/judge friendly name + resolved model id, + * synthesizer/judge friendly name + resolved model id, AND the full ordered + ``synthesizer_chain`` failover ladder (DSE-1512), * generation settings and mode parameters, * protocol/prompt/schema/cache-format versions, * verdict extraction behavior, custom endpoint routing, and an optional @@ -263,6 +286,12 @@ def make_key( converge_threshold: Debate early-stop threshold (included only for ``debate``). A converged run and a fixed-rounds run over otherwise identical inputs must not collide, so this is part of the key. + synthesizer_chain: The full ordered ``(friendly_name, resolved_model_id)`` + failover ladder for the synthesizer/judge role (DSE-1512). Two runs + with the same primary but a different successor ladder must not + collide, since a later run over the same prompt could fail over + differently. Defaults to a chain of one built from + ``synthesizer``/``synthesizer_model_id`` when omitted. Returns: A 64-char lowercase hex SHA-256 digest. Contains zero key material. @@ -282,6 +311,7 @@ def make_key( extract_verdict=extract_verdict, endpoint_urls=endpoint_urls, source_bundle_digest=source_bundle_digest, + synthesizer_chain=synthesizer_chain, cache_format_version=cache_format_version, protocol_version=protocol_version, synthesis_prompt_version=synthesis_prompt_version, diff --git a/src/conclave/cli.py b/src/conclave/cli.py index f76ca22..381cddd 100644 --- a/src/conclave/cli.py +++ b/src/conclave/cli.py @@ -418,6 +418,68 @@ def on_event(event: StreamEvent) -> None: return result +def _format_adjudication_candidate(attempt) -> str: + """Format one succession-ledger attempt for the failover note (DSE-1512). + + A terminal ``"success"`` attempt is shown bare (just the candidate name) -- + it is the answer that shipped, not a failure worth annotating. An + ``"exhausted"`` attempt (the last candidate, still an infrastructure + failure) is marked ``(exhausted)`` with no category/status, matching the + ledger's own semantics: it is the chain running out, not one more + diagnosable failure. ``"failed_over"`` and a terminal ``"terminal_failure"`` + ending (a successor answered, just not usably) both show the candidate's + bounded failure category, plus the HTTP status when one was recorded. + """ + if attempt.outcome == "success": + return attempt.candidate + if attempt.outcome == "exhausted": + return f"{attempt.candidate} (exhausted)" + if attempt.failure_category: + detail = attempt.failure_category + if attempt.http_status: + detail += f", HTTP {attempt.http_status}" + return f"{attempt.candidate} ({detail})" + return attempt.candidate + + +def _render_failover_note(result: CouncilResult) -> None: + """Print one dim stderr line per adjudication role that failed over (DSE-1512). + + A no-op when the run has no manifest, or when no role's succession ledger + contains a ``"failed_over"`` attempt -- so a chain-of-one run (or a chain + that never needed to advance) prints nothing, keeping the human output + byte-identical to v1.3.0. Skipped-unkeyed attempts are omitted from the + line entirely; they are not part of the story of "who tried and failed". + + Roles are rendered in the order they first appear in + ``manifest.adjudication_succession`` (synthesis before verdict_extraction + for a synthesize-mode run, for example), one line each: + ``adjudication failover: : ([, HTTP ]) → → ``. + Never called on the ``--json`` path -- the ledger is already in the + JSON payload's ``manifest``. + """ + manifest = result.manifest + if manifest is None: + return + + roles_seen: list[str] = [] + by_role: dict[str, list] = {} + for attempt in manifest.adjudication_succession: + if attempt.role not in by_role: + by_role[attempt.role] = [] + roles_seen.append(attempt.role) + by_role[attempt.role].append(attempt) + + for role in roles_seen: + attempts = by_role[role] + if not any(a.outcome == "failed_over" for a in attempts): + continue + segments = [ + _format_adjudication_candidate(a) for a in attempts if a.outcome != "skipped_unkeyed" + ] + err_console.print(f"[dim]adjudication failover: {role}: {' → '.join(segments)}[/dim]") + + # Mode name -> human renderer. JSON output bypasses this via model_dump. _RENDERERS = { "synthesize": _render_human, @@ -490,7 +552,14 @@ def ask( help="Run mode: synthesize | raw | debate | adversarial | vote | elite.", ), synthesizer: str | None = typer.Option( - None, "--synthesizer", "-s", help="Override the synthesizer/judge model name." + None, + "--synthesizer", + "-s", + help=( + "Synthesizer/judge model name, or an ordered failover ladder " + "'claude>grok>gemini': the next candidate is tried only on missing-key, " + "unknown-provider, auth, quota, 5xx, timeout, or network failures (DSE-1512)." + ), ), rounds: int = typer.Option( 2, "--rounds", "-r", help="Maximum number of debate rounds (debate mode only).", min=1 @@ -566,7 +635,10 @@ def ask( * 0 -- the run produced at least one usable member answer, the judge/synthesizer step (when attempted) succeeded, and for Elite, - ``decision_readiness`` is ``ready``. + ``decision_readiness`` is ``ready``. This now also covers a run + adjudicated by a successor after a ``--synthesizer``/``synthesizer_chain`` + failover (DSE-1512): a successor answering is a clean pass, not a + degraded one -- see ``CouncilResult.primary_failed_over`` below. * 1 -- the run produced zero usable member answers (e.g. no council member had an API key, or every member failed), or an Elite result is missing or has ``decision_readiness`` ``not_ready``/``indeterminate``. Under ``--json`` @@ -584,7 +656,19 @@ def ask( credit failure took out the judge while 4/5 members still answered, and the run exited 0 with ``adversarial.verdict`` silently ``null``). Not raised for Elite (its own readiness gate above already covers a failed - synthesis/verdict step with exit code 1). + synthesis/verdict step with exit code 1). With a synthesizer chain + (DSE-1512), this now also means the WHOLE chain was exhausted -- every + candidate failed for an infrastructure reason -- rather than just the + lone synthesizer failing. + + ``--json`` also carries the top-level ``"primary_failed_over"`` field + (DSE-1512, additive): ``true`` when, for any role, the declared primary + adjudicator did not itself adjudicate for an infrastructure reason (no key, + or an infrastructure failure) -- whether a successor then answered (exit 0), + the chain was exhausted (exit 3), or a chain of one simply had no key + (exit 3) -- see ``CouncilResult.primary_failed_over``. The human render path prints one + dim ``adjudication failover: ...`` line per role that failed over; a + chain-of-one run that never fails over prints nothing extra. """ mode_lower = mode.lower() if mode_lower not in _VALID_MODES: @@ -626,6 +710,7 @@ def ask( # applies identically. if stream and not as_json: result = _stream_to_terminal(c, prompt, synthesize=(mode_lower == "synthesize")) + _render_failover_note(result) if not result.successful_answers: err_console.print( "[red]No usable council answers. Run 'conclave providers' to check keys.[/red]" @@ -690,6 +775,7 @@ def ask( err_console.print("[red]Elite decision not ready: missing result[/red]") raise typer.Exit(code=1) _render_elite(result) + _render_failover_note(result) if not result.elite.completed: reason = result.elite.failure_reason or ", ".join(result.elite.readiness_reasons) err_console.print(f"[red]Elite protocol incomplete: {reason}[/red]") @@ -712,6 +798,7 @@ def ask( raise typer.Exit(code=1) _RENDERERS[result.mode](result) + _render_failover_note(result) if json_output_failed: raise typer.Exit(code=1) if result.degraded: @@ -739,6 +826,9 @@ def providers() -> None: console.print(table) console.print(f"[dim]synthesizer default: {cfg.synthesizer} · conclave {__version__}[/dim]") + if cfg.synthesizer_chain: + chain = " > ".join(cfg.synthesizer_chain) + console.print(f"[dim]synthesizer chain: {chain}[/dim]") def _builtin_default_note() -> str: diff --git a/src/conclave/config.py b/src/conclave/config.py index eef2716..5d81474 100644 --- a/src/conclave/config.py +++ b/src/conclave/config.py @@ -11,6 +11,7 @@ default: [grok, claude, perplexity] fast: [grok, perplexity] synthesizer: claude + synthesizer_chain: [claude, grok] # optional: ordered failover ladder endpoints: # optional: custom OpenAI-compatible providers together: completions_url: https://api.together.xyz/v1/chat/completions @@ -61,6 +62,15 @@ class ConclaveConfig(BaseModel): models: friendly name -> provider model id. councils: named lists of friendly names. synthesizer: friendly name of the default synthesizer model. + synthesizer_chain: ordered failover ladder for the synthesizer / judge / + verdict-extractor role (DSE-1512). Empty (the default) means "just + ``synthesizer``" -- a chain of one, identical to the historic + behavior. When non-empty, :class:`conclave.council.Council` tries + each candidate in order and only advances to the next one on an + INFRASTRUCTURE failure (auth/quota/5xx/timeout/network/no-key -- + see :data:`conclave.models.FAILOVER_CATEGORIES`); any other failure + (a model that answered, even malformed, or a bad request) is + terminal for the role rather than triggering another vendor's call. endpoints: prefix -> custom OpenAI-compatible endpoint declaration. cache: opt-in result cache (off by default). When ``True`` an identical repeat run is served from the on-disk cache (see :mod:`conclave.cache`) @@ -78,6 +88,7 @@ class ConclaveConfig(BaseModel): models: dict[str, str] = Field(default_factory=dict) councils: dict[str, list[str]] = Field(default_factory=dict) synthesizer: str = DEFAULT_SYNTHESIZER + synthesizer_chain: list[str] = Field(default_factory=list) endpoints: dict[str, CustomEndpoint] = Field(default_factory=dict) cache: bool = False converge_threshold: float | None = None @@ -101,6 +112,45 @@ def resolve_council(self, name_or_csv: str) -> list[str]: return [part.strip() for part in name_or_csv.split(",") if part.strip()] +def parse_synthesizer_chain(spec: str) -> list[str]: + """Split ``"a>b>c"`` into an ordered, de-duplicated candidate list. + + Whitespace around each name is stripped; empty segments (a leading/trailing + ``>`` or a blank string) are dropped; a repeated name keeps only its first + (highest-priority) position. + + Args: + spec: An arrow-delimited chain spec, e.g. ``"claude>grok>gemini"``. + + Returns: + The ordered, de-duplicated list of names (``[]`` for a blank ``spec``). + """ + seen: list[str] = [] + for part in spec.split(">"): + name = part.strip() + if name and name not in seen: + seen.append(name) + return seen + + +def _coerce_chain(value: Any) -> list[str]: + """Coerce a config ``synthesizer_chain`` value to an ordered name list, or []. + + Accepts a YAML list of strings or an arrow-delimited string (mirroring + :func:`parse_synthesizer_chain`). Any other shape degrades to ``[]`` with a + warning, matching this module's resilient-loading convention (a bad config + field never crashes a run, it just disables the optional feature). + """ + if value is None: + return [] + if isinstance(value, str): + return parse_synthesizer_chain(value) + if isinstance(value, list) and all(isinstance(v, str) for v in value): + return parse_synthesizer_chain(">".join(value)) + logger.warning("synthesizer_chain %r is not a list of names; ignoring", value) + return [] + + def _read_yaml(path: Path) -> dict[str, Any]: """Read a YAML file into a dict, returning {} on absence or parse error.""" if not path.exists(): @@ -189,6 +239,7 @@ def _load_config_uncached(path: Path) -> ConclaveConfig: councils.setdefault("default", list(DEFAULT_MODELS.keys())) synthesizer = raw.get("synthesizer", DEFAULT_SYNTHESIZER) + synthesizer_chain = _coerce_chain(raw.get("synthesizer_chain")) endpoints = { prefix: CustomEndpoint(**spec) @@ -207,6 +258,7 @@ def _load_config_uncached(path: Path) -> ConclaveConfig: models=merged_models, councils=councils, synthesizer=synthesizer, + synthesizer_chain=synthesizer_chain, endpoints=endpoints, cache=cache, converge_threshold=converge_threshold, diff --git a/src/conclave/council.py b/src/conclave/council.py index 3206315..05a1416 100644 --- a/src/conclave/council.py +++ b/src/conclave/council.py @@ -52,25 +52,40 @@ from __future__ import annotations import asyncio -from collections.abc import AsyncIterator, Awaitable, Callable +from collections.abc import AsyncIterator, Awaitable, Callable, Sequence +from dataclasses import dataclass +from typing import TYPE_CHECKING from uuid import uuid4 from . import cache as cache_mod from . import transport from .adapters.base import redact -from .config import ConclaveConfig, load_config +from .config import ConclaveConfig, load_config, parse_synthesizer_chain from .logging import get_logger from .manifest import ( + AdjudicationAttempt, + AdjudicationAttemptOutcome, + AdjudicationRole, ModelHarnessManifest, ProviderExecutionReceipt, ProviderSkip, verified_secret_safety, ) -from .models import ELITE_PROTOCOL_VERSION, CouncilResult, ModelAnswer, StreamEvent, TokenUsage +from .models import ( + ELITE_PROTOCOL_VERSION, + FAILOVER_CATEGORIES, + CouncilResult, + ModelAnswer, + StreamEvent, + TokenUsage, +) from .prompts import ELITE_PROMPT_VERSION, SYNTHESIS_PROMPT_VERSION from .providers import call_model, receipt_from_answer from .registry import key_present +if TYPE_CHECKING: # avoid an import cycle at runtime; only needed for typing + from .verdict_synthesis import VerdictSynthesisResult + logger = get_logger("council") # A per-member message-list factory: given a (friendly_name, model_id) member, @@ -98,13 +113,44 @@ __all__ = ["Council", "SYNTHESIS_PROMPT_VERSION"] +@dataclass +class AdjudicationOutcome: + """Return value of :meth:`Council.adjudicate`. + + ``answer`` is the successful answer, or the terminal / exhausted failure, or + ``None`` when no candidate could be called at all (every one unkeyed). + ``called`` lists every REAL call in order (for receipts); ``attempts`` is the + full ledger including skipped candidates. + """ + + answer: ModelAnswer | None + attempts: list[AdjudicationAttempt] + called: list[ModelAnswer] + + @property + def name(self) -> str | None: + return self.answer.name if self.answer is not None else None + + @property + def model_id(self) -> str | None: + return self.answer.model_id if self.answer is not None else None + + class Council: """A council of foundation models with an optional synthesizer. Args: models: Friendly names (or raw provider-prefixed model ids) of council members. - synthesizer: Friendly name of the synthesizer model. If ``None``, the - config default is used. + synthesizer: Friendly name of the synthesizer model, an ordered chain + spec (``"claude>grok>gemini"``), or a list of names + (``["claude", "grok"]``). If ``None``, ``config.synthesizer_chain`` + is used when set, else ``config.synthesizer`` (a chain of one). + Whichever form resolves, ``self.synthesizer`` keeps its historic + meaning: the primary (first) candidate. The full ordered ladder is + ``self.synthesizer_chain`` (DSE-1512); :meth:`adjudicate` walks it + and advances past a candidate only on an infrastructure failure + (see :data:`conclave.models.FAILOVER_CATEGORIES`) -- any other + failure is terminal for the role. config: Pre-loaded config; if ``None``, loaded from disk + defaults. temperature: Sampling temperature for member calls. timeout: Per-call timeout in seconds. @@ -154,7 +200,7 @@ class Council: def __init__( self, models: list[str], - synthesizer: str | None = None, + synthesizer: str | Sequence[str] | None = None, config: ConclaveConfig | None = None, temperature: float = 0.7, timeout: float = 120.0, @@ -165,7 +211,9 @@ def __init__( ) -> None: self.config = config or load_config() self.requested_models = list(models) - self.synthesizer = synthesizer or self.config.synthesizer + self.synthesizer_chain = self._resolve_chain(synthesizer, self.config) + # Back-compat: the primary candidate keeps the historic attribute. + self.synthesizer = self.synthesizer_chain[0] self.temperature = temperature self.timeout = timeout # Explicit override wins; otherwise defer to config (off by default). @@ -190,6 +238,31 @@ def __init__( if not allow_transport_debug_logging: transport.guard_transport_logging() + @staticmethod + def _resolve_chain(spec: str | Sequence[str] | None, config: ConclaveConfig) -> list[str]: + """Resolve the synthesizer ladder (DSE-1512). + + Precedence, highest first: the constructor ``synthesizer=`` arg (string + chain spec or list of names) -> ``config.synthesizer_chain`` -> a chain + of one built from ``config.synthesizer``. This mirrors the pre-existing + scalar-``synthesizer`` precedence documented in the class docstring, just + extended to an ordered list -- a chain of one behaves exactly like today. + + Args: + spec: The constructor's ``synthesizer`` argument, or ``None``. + config: The resolved council config. + + Returns: + A non-empty ordered list of candidate friendly names. + """ + if isinstance(spec, str): + chain = parse_synthesizer_chain(spec) + elif spec is not None: + chain = parse_synthesizer_chain(">".join(spec)) + else: + chain = list(config.synthesizer_chain) + return chain or [config.synthesizer] + def _available_members(self) -> tuple[list[tuple[str, str]], list[str]]: """Partition requested members into (available, skipped-for-no-key). @@ -221,17 +294,21 @@ def _cache_key( ) -> str: """Build the cache key for a run from the resolved, secret-free identity. - Uses the *resolved* member ids and the synthesizer/judge identity so two - runs collide only when they would genuinely produce equivalent output. - Members that would be skipped for a missing key are excluded -- a cache - entry reflects the council that actually ran, so a key reappearing later + Uses the *resolved* member ids and the FULL resolved synthesizer/judge + chain (DSE-1512, not just the primary) so two runs collide only when + they would genuinely produce equivalent output: changing any successor + candidate in the ladder invalidates a prior entry, since a later run + over the same prompt could fail over to a different model. Members + that would be skipped for a missing key are excluded -- a cache entry + reflects the council that actually ran, so a key reappearing later produces the same membership. No environment value is read here. """ members, _skipped = self._available_members() - synth_id = self.config.resolve_model_id(self.synthesizer) + chain_pairs = [(c, self.config.resolve_model_id(c)) for c in self.synthesizer_chain] + synth_id = chain_pairs[0][1] used_prefixes = { model_id.split("/", 1)[0] - for _name, model_id in [*members, (self.synthesizer, synth_id)] + for _name, model_id in [*members, *chain_pairs] if "/" in model_id } return cache_mod.make_key( @@ -240,6 +317,7 @@ def _cache_key( members=members, synthesizer=self.synthesizer, synthesizer_model_id=synth_id, + synthesizer_chain=chain_pairs, temperature=self.temperature, timeout=self.timeout, rounds=rounds, @@ -273,8 +351,9 @@ async def _cached_run( On a hit the cached :class:`CouncilResult` is returned with ``cached=True`` and the providers are not called. On a miss (or when caching is off) the - live ``run`` executes; a successful live run is stored best-effort. Cache - read/write failures never propagate -- they degrade to a normal live run. + live ``run`` executes; a successful live run is stored best-effort, EXCEPT + for the no-store rule below. Cache read/write failures never propagate -- + they degrade to a normal live run. This is the single chokepoint every mode funnels through, so it is also where the manifest-on-every-result invariant is enforced: each returned @@ -282,6 +361,26 @@ async def _cached_run( synthesize/raw path, which builds its own richer manifest in :meth:`_ask_uncached`; a fill for ``debate``/``adversarial``/``vote`` and for a cache hit stored before the manifest existed). + + **No-store when the primary did not adjudicate (DSE-1512).** A cache + hit must never pin a result the chain's primary adjudicator did not + produce, and it must never replay an infrastructure outage -- including + a still-missing key -- after it has cleared. So a live result is NOT + written to the cache when + :attr:`~conclave.models.CouncilResult.primary_failed_over` is ``True``. + See that property for the exact rule; in one sentence: the primary + adjudicator of some role did not itself produce the answer, for an + infrastructure reason (no key, auth, quota, 5xx, timeout, network) or + because the whole ladder was exhausted. The uncached result is still + returned to THIS caller unchanged; only the write to disk is skipped, + so the next identical ``ask`` gets a fresh chance at a healthy primary + rather than a pinned failure or a successor's answer served under the + primary's name. This is a deliberate, narrow behavior change from + v1.3.0: a chain-of-one run whose sole synthesizer had no key or + errored for an infrastructure reason used to be cached and now is not. + A ``"terminal_failure"`` ledger entry (the model answered, just not + usably) is unaffected and remains cacheable exactly as before -- + re-running would not produce a different, better answer. """ if not self.cache_enabled: result = await run() @@ -304,6 +403,13 @@ async def _cached_run( result = await run() self._ensure_manifest(result, mode) + if result.primary_failed_over: + logger.info( + "not caching %s run (%s): primary adjudicator failed for an infrastructure reason", + mode, + key[:12], + ) + return result cache_mod.store(key, result) return result @@ -605,28 +711,15 @@ async def _ask_uncached(self, prompt: str, synthesize: bool = True) -> CouncilRe if synthesize: # Prose synthesis first, then the structured verdict over the SAME - # answers. ``_apply_verdict`` runs after the manifest exists so it can - # populate the manifest's verdict-provenance slots; it is skipped in - # raw mode (no synthesizer call) and is opt-out via the constructor - # flag (resolved inside the helper). The no-members early return above - # never reaches here, so a memberless run carries no verdict. - synthesis_answer = await self._synthesize(result) - if synthesis_answer is not None: - self._append_manifest_receipts( - result, - [ - receipt_from_answer( - synthesis_answer, - temperature=self.temperature, - timeout=self.timeout, - phase="synthesis", - protocol_version=( - ELITE_PROTOCOL_VERSION if result.mode == "elite" else None - ), - prompt_version=SYNTHESIS_PROMPT_VERSION, - ) - ], - ) + # answers. ``_synthesize`` records its own succession ledger + one + # receipt per real call via ``_record_adjudication`` (DSE-1512), so + # no manual receipt append is needed here. ``_apply_verdict`` runs + # after the manifest exists so it can populate the manifest's + # verdict-provenance slots; it is skipped in raw mode (no synthesizer + # call) and is opt-out via the constructor flag (resolved inside the + # helper). The no-members early return above never reaches here, so + # a memberless run carries no verdict. + await self._synthesize(result) await self._apply_verdict(result) return result @@ -648,7 +741,14 @@ async def ask_stream(self, prompt: str, synthesize: bool = True) -> AsyncIterato synthesis was cached) followed by the matching ``*_done`` events and the terminal ``done`` (with ``result.cached is True``). The providers are not called. On a cache **miss**, the live stream runs and, on completion, the - assembled result is stored so a later ``--stream`` or buffered run hits. + assembled result is stored so a later ``--stream`` or buffered run hits + -- UNLESS :attr:`~conclave.models.CouncilResult.primary_failed_over` is + ``True``: the primary adjudicator of some role did not itself produce + the answer (see that property for the exact rule, including a missing + key), mirroring :meth:`_cached_run`'s no-store rule exactly -- a cache + hit must never pin a result the primary did not produce, nor replay an + infrastructure outage after it has cleared. The run is still returned + to this caller unchanged; only the write to disk is skipped. Args: prompt: The user prompt to fan out. @@ -670,14 +770,23 @@ async def ask_stream(self, prompt: str, synthesize: bool = True) -> AsyncIterato yield event return - # Live miss: stream, capture the terminal result, then store it. + # Live miss: stream, capture the terminal result, then store it + # (no-store on primary infrastructure failure -- see the docstring). final: CouncilResult | None = None async for event in stream_ask(self, prompt, synthesize=synthesize): if event.type == "done" and event.result is not None: final = event.result yield event if final is not None: - cache_mod.store(key, final) + if final.primary_failed_over: + logger.info( + "not caching %s run (%s): primary adjudicator failed for an " + "infrastructure reason", + mode, + key[:12], + ) + else: + cache_mod.store(key, final) return async for event in stream_ask(self, prompt, synthesize=synthesize): @@ -733,13 +842,15 @@ def _replay_cached(result: CouncilResult) -> list[StreamEvent]: return events async def _synthesize(self, result: CouncilResult) -> ModelAnswer | None: - """Run the synthesizer over the successful answers, mutating ``result``. + """Run the synthesizer chain over the successful answers, mutating ``result``. This is the buffered (non-streaming) synthesize path; the streaming counterpart :func:`conclave.streaming._stream_synthesis` mirrors it - short-circuit for short-circuit. The synthesizer model is - ``self.synthesizer`` (resolved per the precedence documented in the module - docstring: constructor arg, else config, else the ``"claude"`` default). + short-circuit for short-circuit (unchanged by DSE-1512 -- streaming keeps + the single-candidate synthesizer path). The synthesizer identity is + ``self.synthesizer`` (the chain's primary, resolved per the precedence + documented in the module docstring); the full ordered ladder tried is + ``self.synthesizer_chain`` (:meth:`adjudicate` walks it). Every degraded outcome is made observable on ``result`` -- none is silent. On success ``result.synthesis`` holds the merged answer; on any @@ -747,17 +858,28 @@ async def _synthesize(self, result: CouncilResult) -> ModelAnswer | None: ``result.synthesis_error`` carries the reason: * **no usable answers** -- every member failed/was skipped, so there is - nothing to merge; - * **synthesizer unkeyed** -- ``self.synthesizer``'s API key is absent, so - the raw member answers are returned with an explanatory error; - * **synthesizer call failed** -- the synthesizer provider errored, and its - error text is surfaced verbatim. + nothing to merge (no adjudication attempt is made; the ledger stays + untouched for this run); + * **no chain candidate keyed** -- every candidate in + ``self.synthesizer_chain`` has no API key, so the raw member answers + are returned with an explanatory error (a chain of one keeps the + historic single-model wording byte-for-byte); the ledger records one + ``skipped_unkeyed`` attempt per candidate via + :meth:`_skipped_attempts`; + * **chain exhausted or terminal** -- every keyed candidate failed + (:meth:`adjudicate`'s succession rule), or the answering candidate's + failure is terminal for the role; the error text of the resolved + answer is surfaced verbatim. The synthesizer identity (``synthesizer`` / ``synthesizer_model_id``) is - recorded on ``result`` before the key check so a consumer can see *which* - model was selected even when it could not run. The prompt used is the - versioned :data:`_SYNTH_SYSTEM`; the version tag already lives on - ``result.prompt_version``. + recorded on ``result`` before the key check (as the chain's primary) and + again after adjudication (as whichever candidate actually answered), so + a consumer can see which model was selected even when it could not run, + and which one actually produced the synthesis when the chain failed + over. The prompt used is the versioned :data:`_SYNTH_SYSTEM`; the + version tag already lives on ``result.prompt_version``. Every real call + this method makes is recorded via :meth:`_record_adjudication`, which + appends the succession ledger and one execution receipt per call. """ usable = result.successful_answers if not usable: @@ -765,16 +887,17 @@ async def _synthesize(self, result: CouncilResult) -> ModelAnswer | None: logger.warning(result.synthesis_error) return None - synth_id = self.config.resolve_model_id(self.synthesizer) + primary_id = self.config.resolve_model_id(self.synthesizer) result.synthesizer = self.synthesizer - result.synthesizer_model_id = synth_id - - if not key_present(synth_id): - result.synthesis_error = ( - f"synthesizer '{self.synthesizer}' ({synth_id}) has no API key; " - "returning raw answers only" + result.synthesizer_model_id = primary_id + + err = self._chain_unkeyed_error("synthesizer", "returning raw answers only") + if err is not None: + result.synthesis_error = err + logger.warning(err) + self._record_adjudication( + result, self._skipped_attempts("synthesis"), [], phase="synthesis" ) - logger.warning(result.synthesis_error) return None blocks = "\n\n".join( @@ -787,11 +910,21 @@ async def _synthesize(self, result: CouncilResult) -> ModelAnswer | None: f"Council answers:\n\n{blocks}\n\n" "Now produce the consolidated answer." ) - answer = await self.synthesize_blocks(_SYNTH_SYSTEM, user_content) - if answer.ok: - result.synthesis = answer.answer - else: - result.synthesis_error = answer.error + outcome = await self._adjudicate_and_record( + result, + "synthesis", + _SYNTH_SYSTEM, + user_content, + phase="synthesis", + protocol_version=(ELITE_PROTOCOL_VERSION if result.mode == "elite" else None), + ) + answer = outcome.answer + if answer is not None: + result.synthesizer, result.synthesizer_model_id = outcome.name, outcome.model_id + if answer.ok: + result.synthesis = answer.answer + else: + result.synthesis_error = answer.error return answer async def _apply_verdict( @@ -827,22 +960,69 @@ async def _apply_verdict( retry on a malformed response) distinct from the prose synthesis call -- the documented cost of the default-on verdict. - ``extract_verdict`` owns the N<2 gate (it returns ``verdict=None`` with the - reason ``"fewer than 2 responding members"`` and makes NO LLM call in that - case), so this method delegates unconditionally rather than duplicating the - responder-counting logic; that keeps a single code path and lets the - manifest carry the N<2 reason. ``extract_verdict`` never raises, and this - method only assigns already-secret-free objects afterward, so no defensive - try/except is needed. + **Chain walk (DSE-1512).** Unlike every other adjudication role, this one + cannot simply call :meth:`adjudicate`: verdict extraction is not a single + call -- :func:`conclave.verdict_synthesis.extract_verdict` makes the + initial structured call PLUS one same-model repair retry, validates the + JSON, and computes consensus, all in one invocation. So the chain walk for + this role lives here: ``extract_verdict`` is called once per + ``self.synthesizer_chain`` candidate, and the outcome is classified from + what it reports (``vsr.verdict_absent_reason`` / + ``vsr.failure_category`` / ``vsr.http_status``) rather than from a raw + :class:`~conclave.models.ModelAnswer`. A candidate whose failure category + is in :data:`~conclave.models.FAILOVER_CATEGORIES` advances the chain + (``"failed_over"``, or ``"exhausted"`` on the last candidate); any other + failure -- including ``"malformed_response"`` (the candidate answered, but + not usably) -- is terminal for the role, exactly like every other + adjudication role's rule: a model that answered is never second-guessed by + another vendor. + + **Unkeyed candidates are NOT pre-skipped for this role** -- a deliberate + difference from :meth:`adjudicate`. Today, with a chain of one and an + unkeyed synthesizer, ``_apply_verdict`` still calls ``extract_verdict``, + which makes two ``call_model`` invocations that return instantly with the + "no API key" error (no network), yielding two failed + ``verdict_extraction``/``verdict_repair`` receipts and + ``verdict_absent_reason == "verdict extraction failed schema + validation"``. Preserving that byte-for-byte (the chain-of-one rule) means + letting ``extract_verdict`` run for every candidate rather than + pre-filtering by :func:`conclave.registry.key_present` first: an unkeyed + candidate's ``vsr.failure_category`` comes back ``"unkeyed"``, which IS in + ``FAILOVER_CATEGORIES``, so it becomes ``"failed_over"`` (or + ``"exhausted"`` on the last candidate) -- matching the receipts exactly, + since the calls did happen and did fail. The ``"skipped_unkeyed"`` outcome + (used by every other role via :meth:`_skipped_attempts`) is therefore + never produced for this role. + + This distinction is invisible to the cache. Whether an unkeyed primary + lands as ``"skipped_unkeyed"`` (every other role) or as + ``"failed_over"``/``"exhausted"`` (this role) does not change the + no-store outcome, because + :attr:`~conclave.models.CouncilResult.primary_failed_over` treats all + three the same at the primary's ``attempt_index == 1`` -- see that + property for the uniform rule. + + The N<2 responder gate is unaffected: ``extract_verdict`` returns + immediately with NO call made (``vsr.attempt_receipts == []``) regardless + of which candidate is asked, since the gate counts responding MEMBERS, not + chain candidates. The chain walk stops after the first candidate in that + case (asking a second candidate would just repeat the same no-call + no-op), so N<2 never contributes a ``verdict_extraction`` ledger entry. + + ``extract_verdict`` never raises, and this method only assigns + already-secret-free objects afterward, so no defensive try/except is + needed. When ``result.manifest`` exists its verdict-provenance slots are populated - (extractor identity + prompt version, absent reason, consensus method, - verdict type) and the manifest's ``secret_safety`` stamp is RE-RUN over the + from the LAST candidate consulted (extractor identity + prompt version, + absent reason, consensus method, verdict type), the full succession ledger + is appended, and the manifest's ``secret_safety`` stamp is RE-RUN over the final content: the stamp was first computed in :meth:`_build_manifest` before these fields existed, so re-stamping keeps the VERIFIED claim honest over the manifest a consumer actually receives. The new fields (a resolved model id, a prompt-version string, the ``verdict_type``/``consensus_method`` - literals) are provably key-free, so the stamp stays VERIFIED. + literals, and the ledger's bounded categories) are provably key-free, so + the stamp stays VERIFIED. Args: result: The in-progress :class:`CouncilResult` (answers + manifest @@ -854,23 +1034,96 @@ async def _apply_verdict( # Lazy import mirrors this module's deferred-import style (``modes`` / # ``streaming`` are imported inside methods) and sidesteps any import-cycle # risk between council and the verdict engine. + from .verdict_synthesis import REASON_EXTRACTION_FAILED from .verdict_synthesis import extract_verdict as extract_verdict_fn - synthesizer_name = self.synthesizer - synth_id = self.config.resolve_model_id(self.synthesizer) - vsr = await extract_verdict_fn( - result.prompt, - result.answers, - synthesizer_name=synthesizer_name, - synthesizer_model_id=synth_id, - config=self.config, - temperature=self.temperature, - timeout=self.timeout, - protocol_version=(ELITE_PROTOCOL_VERSION if result.mode == "elite" else None), - ) + protocol_version = ELITE_PROTOCOL_VERSION if result.mode == "elite" else None + chain = self.synthesizer_chain + attempts: list[AdjudicationAttempt] = [] + vsr: VerdictSynthesisResult | None = None + # Running count of verdict receipts already appended across PRIOR chain + # candidates in this call (QA review M2): each candidate's own + # ``attempt_receipts`` restarts at 1 (extract_verdict has no visibility + # into the chain), so without an offset a successor's receipts collide + # with the primary's ("attempt=1" appears twice). Renumbering here keeps + # ``attempt`` monotonically increasing across the whole verdict- + # extraction sequence on the manifest; a chain of one has offset 0 and + # is byte-for-byte unchanged. + verdict_receipts_so_far = 0 + for index, candidate in enumerate(chain, start=1): + model_id = self.config.resolve_model_id(candidate) + vsr = await extract_verdict_fn( + result.prompt, + result.answers, + synthesizer_name=candidate, + synthesizer_model_id=model_id, + config=self.config, + temperature=self.temperature, + timeout=self.timeout, + protocol_version=protocol_version, + ) + renumbered_receipts = [ + receipt.model_copy(update={"attempt": verdict_receipts_so_far + offset}) + for offset, receipt in enumerate(vsr.attempt_receipts, start=1) + ] + if record_receipts: + self._append_manifest_receipts(result, renumbered_receipts) + verdict_receipts_so_far += len(renumbered_receipts) + if not vsr.attempt_receipts: + # N<2 gate: no call was made for this candidate (or any other -- + # the responder count does not depend on which candidate is + # asked), so there is nothing to adjudicate. Stop here rather + # than repeating the same no-op for every remaining candidate. + break + + def _attempt( + outcome: str, + *, + failure_category: str | None = None, + http_status: int | None = None, + _candidate: str = candidate, + _model_id: str = model_id, + _index: int = index, + ) -> AdjudicationAttempt: + """Build one ledger entry for the candidate/index of this iteration. + + The loop variables are bound as default-argument values so the + closure captures THIS iteration's ``candidate``/``model_id``/ + ``index`` rather than whatever they are when the loop ends + (flake8-bugbear B023) -- mirrors :meth:`adjudicate`'s ``_attempt``. + """ + return AdjudicationAttempt( + role="verdict_extraction", + candidate=_candidate, + model_id=_model_id, + attempt_index=_index, + outcome=outcome, + failure_category=failure_category, + http_status=http_status, + ) - if record_receipts: - self._append_manifest_receipts(result, vsr.attempt_receipts) + failed = vsr.verdict_absent_reason == REASON_EXTRACTION_FAILED + is_last = index == len(chain) + outcome = self._classify_outcome( + failed=failed, category=vsr.failure_category, is_last=is_last + ) + attempts.append( + _attempt( + outcome, + failure_category=None if outcome == "success" else vsr.failure_category, + http_status=None if outcome == "success" else vsr.http_status, + ) + ) + if outcome == "failed_over": + logger.warning( + "verdict_extraction: '%s' failed (%s); trying next candidate", + candidate, + vsr.failure_category, + ) + continue + if outcome == "exhausted": + continue + break result.verdict = vsr.verdict if vsr.verdict is not None: @@ -883,42 +1136,324 @@ async def _apply_verdict( result.minority_reports = vsr.verdict.minority_reports if result.manifest is not None: + result.manifest.adjudication_succession.extend(attempts) result.manifest.verdict_extraction = vsr.extraction result.manifest.verdict_absent_reason = vsr.verdict_absent_reason result.manifest.consensus_method = vsr.verdict.consensus_method if vsr.verdict else None result.manifest.verdict_type = vsr.verdict.verdict_type if vsr.verdict else None # Re-stamp over the now-complete manifest so the VERIFIED claim covers - # the verdict-provenance fields just written (they are key-free). + # the verdict-provenance fields (including the new ledger entries) + # just written (they are key-free). result.manifest.secret_safety = verified_secret_safety(result.manifest) - async def synthesize_blocks(self, system_prompt: str, user_content: str) -> ModelAnswer: - """Call the synthesizer model with an arbitrary system + user message. + @staticmethod + def _classify_outcome( + *, failed: bool, category: str | None, is_last: bool + ) -> AdjudicationAttemptOutcome: + """The single failover rule, shared by every adjudication role (DSE-1512 review). + + An infrastructure failure (``category in FAILOVER_CATEGORIES``) advances + the chain -- ``"failed_over"``, or ``"exhausted"`` when there is no next + candidate to try. Any other failure is terminal for the role: a model + that answered (even unusably) is never second-guessed by another vendor. + A non-failure is always ``"success"``. - Shared by synthesize mode, debate's final consolidation, and the - adversarial judge so the synthesizer call path (and its error capture) - is written once. Callers are responsible for checking ``key_present`` - on the synthesizer beforehand when they need a distinct no-key message; - this method still returns a ``ModelAnswer.error`` if the call fails. + Args: + failed: Whether this attempt failed. + category: The attempt's typed failure category, or ``None``. Callers + that must force a terminal outcome regardless of the real + category (e.g. streaming's post-first-delta rule) pass ``None`` + here while still recording the true category on the ledger entry. + is_last: Whether this is the last candidate in the chain. + + Returns: + The bounded :data:`~conclave.manifest.AdjudicationAttemptOutcome`. + """ + if not failed: + return "success" + if category in FAILOVER_CATEGORIES: + return "exhausted" if is_last else "failed_over" + return "terminal_failure" + + async def adjudicate( + self, role: AdjudicationRole, system_prompt: str, user_content: str + ) -> AdjudicationOutcome: + """Walk ``synthesizer_chain`` for one adjudication role (DSE-1512). + + Rule: candidates are tried in declared order; an unkeyed candidate is + skipped without a call; a call that fails with a category in + :data:`FAILOVER_CATEGORIES` advances to the next candidate; ANY other + failure is terminal for the role -- a model that answered is never + second-guessed by another vendor, which would let adjudication shop for + a result. No scoring, no health tracking: the order is the operator's. Args: + role: Which adjudication role this call serves (recorded on every + :class:`~conclave.manifest.AdjudicationAttempt`). system_prompt: System instruction for the synthesizer/judge. user_content: The user-role content (prompt + answers/critiques). Returns: - A :class:`ModelAnswer` from the synthesizer model. + An :class:`AdjudicationOutcome` carrying the resolved answer (or + ``None`` when every candidate was unkeyed), the full attempt + ledger, and the list of real calls made. """ - synth_id = self.config.resolve_model_id(self.synthesizer) messages = [ {"role": "system", "content": system_prompt}, {"role": "user", "content": user_content}, ] - return await call_model( - self.synthesizer, - synth_id, - messages, - config=self.config, - temperature=self.temperature, - timeout=self.timeout, + attempts: list[AdjudicationAttempt] = [] + called: list[ModelAnswer] = [] + last_failure: ModelAnswer | None = None + chain = self.synthesizer_chain + for index, candidate in enumerate(chain, start=1): + model_id = self.config.resolve_model_id(candidate) + + def _attempt( + outcome: str, + *, + failure_category: str | None = None, + http_status: int | None = None, + _candidate: str = candidate, + _model_id: str = model_id, + _index: int = index, + ) -> AdjudicationAttempt: + """Build one ledger entry for the candidate/index of this iteration. + + The loop variables are bound as default-argument values so the + closure captures THIS iteration's ``candidate``/``model_id``/ + ``index`` rather than whatever they are when the loop ends + (flake8-bugbear B023). + """ + return AdjudicationAttempt( + role=role, + candidate=_candidate, + model_id=_model_id, + attempt_index=_index, + outcome=outcome, + failure_category=failure_category, + http_status=http_status, + ) + + if not key_present(model_id): + attempts.append(_attempt("skipped_unkeyed", failure_category="unkeyed")) + continue + answer = await call_model( + candidate, + model_id, + messages, + config=self.config, + temperature=self.temperature, + timeout=self.timeout, + ) + called.append(answer) + is_last = index == len(chain) + category = answer.failure_category + outcome = self._classify_outcome( + failed=not answer.ok, category=category, is_last=is_last + ) + attempts.append( + _attempt( + outcome, + failure_category=None if outcome == "success" else category, + http_status=None if outcome == "success" else answer.http_status, + ) + ) + if outcome == "success": + return AdjudicationOutcome(answer=answer, attempts=attempts, called=called) + if outcome == "terminal_failure": + return AdjudicationOutcome(answer=answer, attempts=attempts, called=called) + # "failed_over" or "exhausted": advance the chain (or fall through to + # the final return below when this was the last candidate). + last_failure = answer + if outcome == "failed_over": + logger.warning( + "%s: '%s' failed (%s); trying next candidate", role, candidate, category + ) + return AdjudicationOutcome(answer=last_failure, attempts=attempts, called=called) + + def _skipped_attempts(self, role: AdjudicationRole) -> list[AdjudicationAttempt]: + """Build the succession ledger for a chain where NO candidate is keyed. + + Used by every adjudication call site (synthesis, debate's final + consolidation, the adversarial judge) when it short-circuits on the + "no candidate has an API key" branch -- :meth:`adjudicate` is never + called (there is nothing to call), but the ledger must still record + that every chain candidate was considered and skipped, one + ``skipped_unkeyed`` :class:`~conclave.manifest.AdjudicationAttempt` per + candidate in chain order. + + Args: + role: Which adjudication role this skip ledger serves. + + Returns: + One ``skipped_unkeyed`` attempt per candidate in + ``self.synthesizer_chain``, in chain order. + """ + return [ + AdjudicationAttempt( + role=role, + candidate=candidate, + model_id=self.config.resolve_model_id(candidate), + attempt_index=index, + outcome="skipped_unkeyed", + failure_category="unkeyed", + ) + for index, candidate in enumerate(self.synthesizer_chain, start=1) + ] + + def _chain_unkeyed_error(self, actor: str, suffix: str) -> str | None: + """Return the no-key error for the whole chain, or ``None`` if any candidate is keyed. + + The shared keyed-check for every adjudication call site (synthesis, + debate's final consolidation, the adversarial judge) that needs a + distinct "no candidate has a key" message before calling + :meth:`adjudicate` (DSE-1512 review, Unit C). ``actor`` is the role + noun used in the message (``"synthesizer"`` / ``"judge"``); ``suffix`` + is the mode-specific tail (``"returning raw answers only"``, + ``"returning final-round answers only"``, ``"returning proposal and + critiques only"``). A chain of one keeps the historic single-candidate + wording verbatim byte-for-byte. + + Args: + actor: The role noun for the message. + suffix: The mode-specific tail clause. + + Returns: + The formatted error string when every chain candidate is unkeyed, + else ``None``. + """ + keyed = [c for c in self.synthesizer_chain if key_present(self.config.resolve_model_id(c))] + if keyed: + return None + if len(self.synthesizer_chain) == 1: + primary_id = self.config.resolve_model_id(self.synthesizer) + return f"{actor} '{self.synthesizer}' ({primary_id}) has no API key; {suffix}" + names = ", ".join(self.synthesizer_chain) + return f"{actor} chain [{names}] has no API key for any candidate; {suffix}" + + def _record_adjudication( + self, + result: CouncilResult, + attempts: list[AdjudicationAttempt], + called: list[ModelAnswer], + *, + phase: str, + record_receipts: bool = True, + protocol_version: str | None = None, + prompt_version: str | None = SYNTHESIS_PROMPT_VERSION, + ) -> None: + """Append the succession ledger (always) and one receipt per real call. + + No-op when the result has no manifest yet. ``_recompute_manifest_accounting`` + re-derives totals and re-stamps ``secret_safety`` over the new content. + + Args: + result: The in-progress result. Mutated in place. + attempts: The full attempt ledger from :meth:`adjudicate`, including + skipped-unkeyed candidates. + called: The real calls made, in order (from + :attr:`AdjudicationOutcome.called`). + phase: The manifest receipt phase to stamp on each call. + record_receipts: When ``False``, skip appending receipts (the ledger + is still recorded); used by callers that record receipts + elsewhere. + protocol_version: Optional protocol version to stamp on receipts. + prompt_version: Prompt version to stamp on receipts; defaults to the + synthesis prompt version. + """ + if result.manifest is None: + return + result.manifest.adjudication_succession.extend(attempts) + if record_receipts: + result.manifest.receipts.extend( + receipt_from_answer( + answer, + temperature=self.temperature, + timeout=self.timeout, + phase=phase, + attempt=index, + protocol_version=protocol_version, + prompt_version=prompt_version, + ) + for index, answer in enumerate(called, start=1) + ) + self._recompute_manifest_accounting(result.manifest) + + async def _adjudicate_and_record( + self, + result: CouncilResult, + role: AdjudicationRole, + system_prompt: str, + user_content: str, + *, + phase: str, + protocol_version: str | None = None, + ) -> AdjudicationOutcome: + """``adjudicate`` then ``_record_adjudication`` -- the common tail of every role. + + Shared by :meth:`_synthesize`, :func:`conclave.modes._debate_synthesize`, + and :func:`conclave.modes._adversarial_judge` (DSE-1512 review, Unit C) + so the "call the chain, then land the ledger + receipts" sequence is + written exactly once. + + Args: + result: The in-progress result. Mutated in place via + :meth:`_record_adjudication`. + role: Which adjudication role this call serves. + system_prompt: System instruction for the synthesizer/judge. + user_content: The user-role content (prompt + answers/critiques). + phase: The manifest receipt phase to stamp on each real call. + protocol_version: Optional protocol version to stamp on receipts. + + Returns: + The :class:`AdjudicationOutcome` from :meth:`adjudicate`. + """ + outcome = await self.adjudicate(role, system_prompt, user_content) + self._record_adjudication( + result, outcome.attempts, outcome.called, phase=phase, protocol_version=protocol_version + ) + return outcome + + async def synthesize_blocks(self, system_prompt: str, user_content: str) -> ModelAnswer: + """Call the synthesizer chain with an arbitrary system + user message. + + A compatibility wrapper kept for external/library callers that want one + synthesizer answer without building a full :class:`CouncilResult` (QA + review M4). It walks ``synthesizer_chain`` via :meth:`adjudicate` -- + with a chain of one (the default) this is byte-for-byte the historic + single-call behavior -- and returns whichever :class:`ModelAnswer` the + walk resolves to, or a synthetic ``ModelAnswer.error`` when every + candidate was unkeyed. Callers are responsible for checking + ``key_present`` beforehand when they need a distinct no-key message. + + **No caller inside this package uses this method any more.** + :meth:`_synthesize`, :func:`conclave.modes._debate_synthesize`, and + :func:`conclave.modes._adversarial_judge` all go through + :meth:`_adjudicate_and_record` instead, because THIS method records + neither the succession ledger nor any receipts -- it has no + :class:`CouncilResult` to attach them to. A caller that needs the audit + trail (the manifest's ``adjudication_succession`` + per-call receipts) + must go through :meth:`Council.ask` / :meth:`debate` / :meth:`adversarial` + instead; this method remains for a caller that genuinely only wants the + bare answer. No behavior change from calling :meth:`adjudicate` directly. + + Args: + system_prompt: System instruction for the synthesizer/judge. + user_content: The user-role content (prompt + answers/critiques). + + Returns: + A :class:`ModelAnswer` from the synthesizer chain. + """ + outcome = await self.adjudicate("synthesis", system_prompt, user_content) + if outcome.answer is not None: + return outcome.answer + synth_id = self.config.resolve_model_id(self.synthesizer) + return ModelAnswer( + name=self.synthesizer, + model_id=synth_id, + error="no candidate in synthesizer chain has an API key", + failure_category="unkeyed", ) async def debate( @@ -983,22 +1518,15 @@ async def elite(self, prompt: str) -> CouncilResult: async def run() -> CouncilResult: result = await run_elite(self, prompt) if result.elite is not None and result.elite.completed: - synthesis_answer = await self._synthesize(result) + # Ensure the manifest exists BEFORE synthesizing so + # ``_synthesize``'s ``_record_adjudication`` call has somewhere + # to land the succession ledger and per-call receipts + # (DSE-1512). ``_ensure_manifest`` only flattens the already- + # collected phase artifacts (initial/critique/revision), so + # building it before synthesis runs is safe -- synthesis is not + # one of those phases. self._ensure_manifest(result, "elite") - if synthesis_answer is not None: - self._append_manifest_receipts( - result, - [ - receipt_from_answer( - synthesis_answer, - temperature=self.temperature, - timeout=self.timeout, - phase="synthesis", - protocol_version=ELITE_PROTOCOL_VERSION, - prompt_version=SYNTHESIS_PROMPT_VERSION, - ) - ], - ) + await self._synthesize(result) if result.synthesis is None: result.elite.decision_readiness = "not_ready" result.elite.readiness_reasons = ["synthesis.failed"] diff --git a/src/conclave/manifest.py b/src/conclave/manifest.py index 0bd3f4b..833e6f4 100644 --- a/src/conclave/manifest.py +++ b/src/conclave/manifest.py @@ -29,7 +29,9 @@ runs the other way (``models`` imports the manifest types and calls ``model_rebuild()``) so there is no cycle — the same no-cycle pattern :mod:`conclave.verdict` uses. It DOES import :class:`~conclave.models.TokenUsage` -(a leaf type with no back-edge to the manifest) for the usage fields. +and the :data:`~conclave.models.FailureCategory` literal (both leaf types with +no back-edge to the manifest) for the usage fields and the bounded +``failure_category`` on :class:`AdjudicationAttempt`, respectively. """ from __future__ import annotations @@ -38,7 +40,7 @@ from pydantic import BaseModel, Field -from .models import TokenUsage +from .models import FailureCategory, TokenUsage # ``secret_safety`` status literals. UNVERIFIED is the safe default (a manifest is # untrusted until the self-scan proves it clean); VERIFIED is stamped only by @@ -99,6 +101,45 @@ class VerdictExtraction(BaseModel): prompt_version: str | None = None +AdjudicationRole = Literal["synthesis", "debate_final", "judge", "verdict_extraction"] +AdjudicationAttemptOutcome = Literal[ + "success", # this candidate adjudicated + "failed_over", # infra failure; the ladder advanced to the next candidate + # (which may itself have been skipped for a missing key) + "exhausted", # infra failure on the LAST candidate; nothing left to try + "terminal_failure", # the candidate answered unusably; failover refused by rule + "skipped_unkeyed", # no API key in the environment; no call made +] + + +class AdjudicationAttempt(BaseModel): + """One step of the synthesizer/judge succession ladder (DSE-1512). + + Deliberately carries NO free text: only bounded categories and an HTTP + status. A raw provider error string could contain words the secret-safety + scan forbids (e.g. "authorization"), which would un-verify the manifest. + + Attributes: + role: Which adjudication role this attempt served. + candidate: Friendly name of the model tried at this step. + model_id: Resolved provider-prefixed model id. + attempt_index: One-based position of this candidate in the chain. + outcome: Bounded outcome of the attempt -- see + :data:`AdjudicationAttemptOutcome`. + failure_category: The typed :data:`conclave.models.FailureCategory` of + the failure, or ``None`` on ``"success"``. + http_status: The HTTP status that produced the failure, when known. + """ + + role: AdjudicationRole + candidate: str + model_id: str + attempt_index: int = Field(ge=1) + outcome: AdjudicationAttemptOutcome + failure_category: FailureCategory | None = None + http_status: int | None = None + + class ProviderExecutionReceipt(BaseModel): """A per-call execution record for one council member that was CALLED. @@ -192,6 +233,11 @@ class ModelHarnessManifest(BaseModel): verdict_absent_reason: Why ``result.verdict`` is ``None`` (open-ended generation, N<2, or structured-extraction failure), or ``None`` when a verdict is present / not yet computed (DD-2 ripple, filled by CAC-05). + adjudication_succession: The full synthesizer/judge/verdict-extractor + succession ledger (DSE-1512) -- one :class:`AdjudicationAttempt` per + candidate tried across every adjudication role in this run, + including skipped-unkeyed candidates. Empty for a run that never + called :meth:`conclave.council.Council.adjudicate`. """ # REQUIRED identity. @@ -229,6 +275,9 @@ class ModelHarnessManifest(BaseModel): consensus_method: str | None = None verdict_absent_reason: str | None = None + # Synthesizer/judge/verdict-extractor succession ledger (DSE-1512). + adjudication_succession: list[AdjudicationAttempt] = Field(default_factory=list) + def scan_for_secret_material(manifest: ModelHarnessManifest) -> bool: """Return True when the serialized manifest is CLEAN of key material. diff --git a/src/conclave/models.py b/src/conclave/models.py index fbcdaeb..e2c911b 100644 --- a/src/conclave/models.py +++ b/src/conclave/models.py @@ -115,6 +115,54 @@ class TokenUsage(BaseModel): total_tokens: int = Field(default=0, ge=0) +# DSE-1512 — typed failure categories. Derived at the RAISE SITE from the HTTP +# status or exception type, never by inspecting a rendered error string. The +# adjudication ladder (Council.adjudicate) fails over ONLY on the categories in +# FAILOVER_CATEGORIES: infrastructure failures where no model ever produced an +# answer. A model that answered (even malformed) is terminal for that role. +FailureCategory = Literal[ + "unkeyed", # env var absent -- no call made + "unresolved", # unknown provider prefix -- no call made + "auth", # 401 / 403 + "quota", # 402 / 429 + "unavailable", # 5xx + "timeout", # 408 or transport deadline + "transport", # DNS / connection / other httpx network error + "bad_request", # other 4xx -- the request was wrong, not the vendor + "malformed_response", # 2xx with an unusable payload / empty content + "unexpected", # anything else -- never failed over +] + +FAILOVER_CATEGORIES: frozenset[str] = frozenset( + {"unkeyed", "unresolved", "auth", "quota", "unavailable", "timeout", "transport"} +) + + +def categorize_http_status(status: int) -> FailureCategory: + """Map a non-2xx HTTP status to a :data:`FailureCategory` (pure, no I/O). + + Every 4xx/5xx status is bounded by one of the categories above. Anything + OUTSIDE that range (1xx, 2xx, 3xx, or >= 600 -- none of which this + function is meant to be called with, but a defensive raise site could) + maps to ``"unexpected"`` (QA review M3), never ``"malformed_response"``: + that category is reserved for a 2xx response with an unusable PAYLOAD, a + distinct failure mode this function never sees. Neither ``"unexpected"`` + nor ``"malformed_response"`` is in :data:`FAILOVER_CATEGORIES`, so this + never affects failover either way. + """ + if status in (401, 403): + return "auth" + if status in (402, 429): + return "quota" + if status == 408: + return "timeout" + if 500 <= status <= 599: + return "unavailable" + if 400 <= status <= 499: + return "bad_request" + return "unexpected" + + class ModelAnswer(BaseModel): """One council member's response (or failure). @@ -132,6 +180,15 @@ class ModelAnswer(BaseModel): warnings: Non-fatal notes about this answer (e.g. structured-output repair applied). Empty by default. Distinct from ``error``, which marks the whole call as failed. + failure_category: Typed classification of ``error`` (DSE-1512), derived + at the raise site from the HTTP status or exception type -- never by + inspecting ``error`` text. ``None`` on success and on any answer + collected before this field existed. See :data:`FailureCategory` and + :data:`FAILOVER_CATEGORIES`. + http_status: The HTTP status code that produced ``error``, when the + failure came from a response (as opposed to a pre-call or transport + failure). ``None`` on success and whenever no HTTP response was + received. """ name: str @@ -142,6 +199,8 @@ class ModelAnswer(BaseModel): error: str | None = None answer_id: str | None = None warnings: list[str] = Field(default_factory=list) + failure_category: FailureCategory | None = None + http_status: int | None = None @property def ok(self) -> bool: @@ -257,6 +316,15 @@ def successful_critiques(self) -> list[ModelAnswer]: return [c for c in self.critiques if c.ok] +# DSE-1512 review — the uniform primary_failed_over rule. An outcome at the +# PRIMARY's attempt (attempt_index == 1) in this set means the primary did not +# itself adjudicate for an infrastructure reason: a missing key +# ("skipped_unkeyed") is treated exactly like a live failover or an exhausted +# ladder, because in every case no key/content answer ever came back from the +# primary. See CouncilResult.primary_failed_over for the full rule. +_PRIMARY_INFRA_OUTCOMES: frozenset[str] = frozenset({"failed_over", "exhausted", "skipped_unkeyed"}) + + class VoteResult(BaseModel): """The tally from a constrained-choice vote run. @@ -360,6 +428,16 @@ class CouncilResult(BaseModel): so a caller cannot mistake a partial run for a clean pass. See the property docstring below for the exact rule and the CLI's exit-code contract (:func:`conclave.cli.ask`) that keys off it. + primary_failed_over: Computed field (DSE-1512, review-uniform rule) -- + ``True`` when the primary adjudicator of at least one role did not + itself adjudicate for an infrastructure reason (no key, auth, + quota, 5xx, timeout, network) or the ladder was exhausted, whether + or not a successor then answered. Independent of ``degraded``: a + successor adjudication is ``primary_failed_over=True, + degraded=False``. A ``terminal_failure`` primary (the model + answered, just not usably) is ``False``. See the property + docstring below for the exact rule and why the cache never stores + such a run, buffered or streamed. """ prompt: str @@ -447,6 +525,75 @@ def degraded(self) -> bool: return True return False + @computed_field # type: ignore[prop-decorator] + @property + def primary_failed_over(self) -> bool: + """True when the primary adjudicator did not adjudicate, for any role (DSE-1512, uniform rule). + + ``True`` iff, for any adjudication role recorded on + ``self.manifest.adjudication_succession`` (synthesis, debate's final + consolidation, the adversarial judge, or verdict extraction), the + attempt at ``attempt_index == 1`` -- the chain's declared primary -- + has ``outcome`` in :data:`_PRIMARY_INFRA_OUTCOMES`: + + * ``"failed_over"`` -- the primary failed for an infrastructure reason + (auth/quota/5xx/timeout/network/unresolved, see + :data:`FAILOVER_CATEGORIES`) and a successor then answered; + * ``"exhausted"`` -- the primary failed the same way and NOTHING in + the chain answered (the ladder was exhausted); + * ``"skipped_unkeyed"`` -- the primary had no API key, so no call was + ever made for it. A missing key is an infrastructure reason like any + other: the primary simply did not adjudicate. + + This one condition subsumes every case the previous, two-outcome rule + handled separately: a successor can only have adjudicated + (``attempt_index > 1``, ``outcome == "success"``) if the primary at + index 1 was ``"failed_over"`` or ``"skipped_unkeyed"``; a fully + exhausted ladder always has index 1 in this set too. A role with no + ledger entries (it never ran for this result -- e.g. ``vote``/``raw`` + modes, or a role this run never reached) contributes nothing. + + ``False`` when the primary's index-1 attempt is ``"success"`` (it + adjudicated) or ``"terminal_failure"`` (it answered, just not usably + -- a content failure, not an infrastructure one, so re-running the + same model would not produce a different, better answer). Also + ``False`` for any run with no manifest. + + This closes a real gap from the prior two-outcome rule: an unkeyed + primary was recorded as ``"skipped_unkeyed"`` by every role except + verdict extraction (which deliberately calls the unkeyed candidate + anyway and records ``"failed_over"``/``"exhausted"`` -- see + :meth:`conclave.council.Council._apply_verdict`), so whether an + unkeyed-primary run reported ``True`` used to depend on which roles + happened to run and on ``extract_verdict``, rather than being uniform. + + Independent of :attr:`degraded`: a run adjudicated by a successor is a + clean run (``primary_failed_over=True, degraded=False``), while a run + where the whole chain was exhausted is both + (``primary_failed_over=True, degraded=True``). The two flags answer + different questions -- "did the judge/synthesis step ultimately + produce a usable result?" (``degraded``) versus "did the primary + candidate itself need to be replaced or exhausted?" + (``primary_failed_over``). + + A run for which this is ``True`` is never written to the result cache, + whether the run was buffered or streamed via ``--stream`` (see + :meth:`conclave.council.Council._cached_run` and + :meth:`conclave.council.Council.ask_stream`): a cache hit must never + pin a result the primary did not produce, nor replay an + infrastructure outage -- including a still-missing key -- after it + has cleared. Included as a top-level key in ``model_dump(mode="json")`` + output (a Pydantic ``computed_field``), so a scripted consumer can + check it directly instead of walking the manifest's succession + ledger. + """ + if self.manifest is None: + return False + return any( + a.attempt_index == 1 and a.outcome in _PRIMARY_INFRA_OUTCOMES + for a in self.manifest.adjudication_succession + ) + # Late import (see the note near the top): ``manifest`` imports ``TokenUsage`` # from this module, so it can only be imported once the leaf types above exist. diff --git a/src/conclave/modes.py b/src/conclave/modes.py index 133d1f0..248d3cb 100644 --- a/src/conclave/modes.py +++ b/src/conclave/modes.py @@ -1,8 +1,10 @@ """Deliberation modes: multi-round debate and adversarial propose/refute/verdict. Both modes are built on :meth:`conclave.council.Council.fan_out` (the single -concurrency + partial-failure primitive) and :meth:`Council.synthesize_blocks` -(the single synthesizer call path). Keeping the logic here keeps ``council.py`` +concurrency + partial-failure primitive) and +:meth:`Council._adjudicate_and_record` (the single adjudication path: it walks +the synthesizer chain via :meth:`Council.adjudicate` and records the succession +ledger + receipts, DSE-1512). Keeping the logic here keeps ``council.py`` focused on the v0.1 surface while the deliberation algorithms live on their own. Prompt wording lives in :mod:`conclave.prompts`. @@ -37,7 +39,6 @@ VoteResult, derive_phase_answer_id, ) -from .registry import key_present if TYPE_CHECKING: # avoid a circular import at runtime; only needed for typing from .council import Council @@ -383,6 +384,13 @@ async def run_debate( if result.rounds: result.answers = list(result.rounds[-1].answers) + # Attach the manifest BEFORE the final consolidation so + # ``_debate_synthesize``'s ``_record_adjudication`` call (DSE-1512) has + # somewhere to land the succession ledger and per-call receipts. + # ``result.answers`` already mirrors the final round, so the manifest built + # here is identical to the one ``_cached_run`` would otherwise build after + # this function returns. + council._ensure_manifest(result, "debate") await _debate_synthesize(council, result) return result @@ -489,33 +497,52 @@ def messages_for(name: str, _model_id: str) -> list[dict[str, str]]: async def _debate_synthesize(council: Council, result: CouncilResult) -> None: - """Consolidate the final round's surviving answers via the synthesizer.""" + """Consolidate the final round's surviving answers via the synthesizer chain. + + Mirrors :meth:`conclave.council.Council._synthesize` short-circuit for + short-circuit (DSE-1512): the "no surviving answers" gate is debate-specific + (there is nothing else to consolidate), but the keyed-chain check + (:meth:`Council._chain_unkeyed_error`) and the adjudicate-then-record tail + (:meth:`Council._adjudicate_and_record`) are the SAME shared methods + :meth:`Council._synthesize` and :func:`_adversarial_judge` call (DSE-1512 + review, Unit C), so the chain-of-one/chain wording and the + succession-ledger bookkeeping can never drift between the three roles. + ``result.manifest`` is guaranteed non-``None`` here -- :func:`run_debate` + calls :meth:`Council._ensure_manifest` immediately before this function. + """ final = result.rounds[-1].successful_answers if result.rounds else [] if not final: result.synthesis_error = "no surviving member answers to synthesize" logger.warning(result.synthesis_error) return - synth_id = council.config.resolve_model_id(council.synthesizer) + primary_id = council.config.resolve_model_id(council.synthesizer) result.synthesizer = council.synthesizer - result.synthesizer_model_id = synth_id - if not key_present(synth_id): - result.synthesis_error = ( - f"synthesizer '{council.synthesizer}' ({synth_id}) has no API key; " - "returning final-round answers only" + result.synthesizer_model_id = primary_id + + err = council._chain_unkeyed_error("synthesizer", "returning final-round answers only") + if err is not None: + result.synthesis_error = err + logger.warning(err) + council._record_adjudication( + result, council._skipped_attempts("debate_final"), [], phase="debate_final" ) - logger.warning(result.synthesis_error) return blocks = "\n\n".join( f"### Final answer from {a.name} ({a.model_id})\n{a.answer}" for a in final ) user_content = prompts.debate_final_user(result.prompt, len(result.rounds), blocks) - answer = await council.synthesize_blocks(prompts.DEBATE_FINAL_SYSTEM, user_content) - if answer.ok: - result.synthesis = answer.answer - else: - result.synthesis_error = answer.error + outcome = await council._adjudicate_and_record( + result, "debate_final", prompts.DEBATE_FINAL_SYSTEM, user_content, phase="debate_final" + ) + answer = outcome.answer + if answer is not None: + result.synthesizer, result.synthesizer_model_id = outcome.name, outcome.model_id + if answer.ok: + result.synthesis = answer.answer + else: + result.synthesis_error = answer.error async def run_adversarial( @@ -600,7 +627,13 @@ async def run_adversarial( result.answers.extend(adv.critiques) # Step 4: the judge weighs proposal vs critiques and issues a verdict. - await _adversarial_judge(council, prompt, adv) + # Attach the manifest first so ``_adversarial_judge``'s ``_record_adjudication`` + # call (DSE-1512) has somewhere to land the succession ledger and per-call + # receipts. ``result.answers`` already holds the proposal attempt(s) and + # critiques, so the manifest built here is identical to the one + # ``_cached_run`` would otherwise build after this function returns. + council._ensure_manifest(result, "adversarial") + await _adversarial_judge(council, prompt, adv, result) result.adversarial = adv result.synthesis = adv.verdict result.synthesis_error = adv.verdict_error @@ -678,11 +711,27 @@ def _proposer_order(members: list[tuple[str, str]], requested: str) -> list[tupl return [requested_member, *rest] -async def _adversarial_judge(council: Council, prompt: str, adv: AdversarialResult) -> None: - """Run the judge over the proposal + critiques, mutating ``adv``.""" - judge_id = council.config.resolve_model_id(council.synthesizer) +async def _adversarial_judge( + council: Council, prompt: str, adv: AdversarialResult, result: CouncilResult +) -> None: + """Run the judge chain over the proposal + critiques, mutating ``adv``. + + Mirrors :meth:`conclave.council.Council._synthesize` short-circuit for + short-circuit (DSE-1512): the "proposal failed" gate is adversarial-specific + (nothing to judge without a proposal), but the keyed-chain check + (:meth:`Council._chain_unkeyed_error`) and the adjudicate-then-record tail + (:meth:`Council._adjudicate_and_record`) are the SAME shared methods + :meth:`Council._synthesize` and :func:`_debate_synthesize` call (DSE-1512 + review, Unit C), so the chain-of-one/chain wording and the + succession-ledger bookkeeping can never drift between the three roles. + ``result`` (the enclosing :class:`CouncilResult`) is threaded through only + to reach its manifest -- callers guarantee ``result.manifest is not None`` + before calling this (:func:`run_adversarial` calls + :meth:`Council._ensure_manifest` immediately before this function). + """ + primary_id = council.config.resolve_model_id(council.synthesizer) adv.judge = council.synthesizer - adv.judge_model_id = judge_id + adv.judge_model_id = primary_id if not adv.proposal.ok: adv.verdict_error = ( @@ -690,12 +739,12 @@ async def _adversarial_judge(council: Council, prompt: str, adv: AdversarialResu ) logger.warning(adv.verdict_error) return - if not key_present(judge_id): - adv.verdict_error = ( - f"judge '{council.synthesizer}' ({judge_id}) has no API key; " - "returning proposal and critiques only" - ) - logger.warning(adv.verdict_error) + + err = council._chain_unkeyed_error("judge", "returning proposal and critiques only") + if err is not None: + adv.verdict_error = err + logger.warning(err) + council._record_adjudication(result, council._skipped_attempts("judge"), [], phase="judge") return usable_critiques = adv.successful_critiques @@ -709,8 +758,13 @@ async def _adversarial_judge(council: Council, prompt: str, adv: AdversarialResu user_content = prompts.judge_user( prompt, adv.proposer, adv.proposal.answer or "", critique_blocks ) - answer = await council.synthesize_blocks(prompts.JUDGE_SYSTEM, user_content) - if answer.ok: - adv.verdict = answer.answer - else: - adv.verdict_error = answer.error + outcome = await council._adjudicate_and_record( + result, "judge", prompts.JUDGE_SYSTEM, user_content, phase="judge" + ) + answer = outcome.answer + if answer is not None: + adv.judge, adv.judge_model_id = outcome.name, outcome.model_id + if answer.ok: + adv.verdict = answer.answer + else: + adv.verdict_error = answer.error diff --git a/src/conclave/providers.py b/src/conclave/providers.py index 7bb52a9..92ebb48 100644 --- a/src/conclave/providers.py +++ b/src/conclave/providers.py @@ -176,7 +176,13 @@ async def call_model( except ProviderError as exc: latency = time.perf_counter() - started logger.warning("%s (%s) unresolved: %s", name, model_id, exc) - return ModelAnswer(name=name, model_id=model_id, latency_s=latency, error=str(exc)) + return ModelAnswer( + name=name, + model_id=model_id, + latency_s=latency, + error=str(exc), + failure_category=exc.category, + ) api_key = _resolve_key(adapter) if api_key is None: @@ -184,7 +190,13 @@ async def call_model( names = " or ".join(adapter.env_vars) or "(none)" msg = f"no API key in environment (set {names})" logger.warning("%s (%s) %s", name, model_id, msg) - return ModelAnswer(name=name, model_id=model_id, latency_s=latency, error=msg) + return ModelAnswer( + name=name, + model_id=model_id, + latency_s=latency, + error=msg, + failure_category="unkeyed", + ) try: url, headers, body = adapter.build_request( @@ -214,12 +226,25 @@ async def call_model( # transport message and any composed string. message = redact(str(exc)) logger.warning("%s (%s) failed: %s", name, model_id, message) - return ModelAnswer(name=name, model_id=model_id, latency_s=latency, error=message) + return ModelAnswer( + name=name, + model_id=model_id, + latency_s=latency, + error=message, + failure_category=exc.category, + http_status=getattr(exc, "http_status", None), + ) except Exception as exc: # noqa: BLE001 -- never let an unexpected raise kill the run latency = time.perf_counter() - started message = redact(f"{type(exc).__name__}: {exc}") logger.warning("%s (%s) unexpected error: %s", name, model_id, message) - return ModelAnswer(name=name, model_id=model_id, latency_s=latency, error=message) + return ModelAnswer( + name=name, + model_id=model_id, + latency_s=latency, + error=message, + failure_category="unexpected", + ) def _merge_usage(acc: TokenUsage | None, frame: TokenUsage | None) -> TokenUsage | None: @@ -309,7 +334,13 @@ async def call_model_stream( except ProviderError as exc: latency = time.perf_counter() - started logger.warning("%s (%s) unresolved: %s", name, model_id, exc) - yield ModelAnswer(name=name, model_id=model_id, latency_s=latency, error=str(exc)) + yield ModelAnswer( + name=name, + model_id=model_id, + latency_s=latency, + error=str(exc), + failure_category=exc.category, + ) return # Providers without a streaming path degrade to a single-chunk render so the @@ -336,7 +367,13 @@ async def call_model_stream( names = " or ".join(adapter.env_vars) or "(none)" msg = f"no API key in environment (set {names})" logger.warning("%s (%s) %s", name, model_id, msg) - yield ModelAnswer(name=name, model_id=model_id, latency_s=latency, error=msg) + yield ModelAnswer( + name=name, + model_id=model_id, + latency_s=latency, + error=msg, + failure_category="unkeyed", + ) return parts: list[str] = [] @@ -370,6 +407,7 @@ async def call_model_stream( model_id=model_id, latency_s=latency, error=f"{adapter.prefix}: empty response (no streamed content)", + failure_category="malformed_response", ) return logger.info("%s (%s) streamed ok in %.2fs", name, model_id, latency) @@ -394,6 +432,8 @@ async def call_model_stream( latency_s=latency, usage=usage, error=message, + failure_category=exc.category, + http_status=getattr(exc, "http_status", None), ) except Exception as exc: # noqa: BLE001 -- never let an unexpected raise kill the run latency = time.perf_counter() - started @@ -406,4 +446,5 @@ async def call_model_stream( latency_s=latency, usage=usage, error=message, + failure_category="unexpected", ) diff --git a/src/conclave/streaming.py b/src/conclave/streaming.py index e140537..e28dee3 100644 --- a/src/conclave/streaming.py +++ b/src/conclave/streaming.py @@ -7,13 +7,28 @@ * fans the prompt out to every available member **concurrently**, interleaving each member's incremental text into one flat :class:`conclave.models.StreamEvent` sequence (``member_delta`` / ``member_done``), -* optionally streams the synthesizer over the successful answers +* optionally streams the synthesizer **chain** (:attr:`Council.synthesizer_chain`, + DSE-1512) over the successful answers, walked by :func:`_stream_synthesis` (``synthesis_delta`` / ``synthesis_done``), and * emits a terminal ``done`` event carrying the fully-assembled :class:`conclave.models.CouncilResult` whose shape is **byte-for-byte identical** to the non-streaming :meth:`Council.ask` result -- so downstream consumers (and the cache) are unaffected. +**Synthesis succession before the first delta (DSE-1512, Task 7).** +:func:`_stream_synthesis` walks ``synthesizer_chain`` exactly like +:meth:`Council.adjudicate`, but with one addition specific to a live token +stream: a candidate may fail over to the next candidate ONLY while it has not +yet emitted a single ``synthesis_delta``. Once a candidate's tokens have been +shown to the caller they cannot be un-emitted, so any failure after the first +delta is terminal for the role regardless of its failure category -- a stream +that already started is a stream the caller committed to. The full succession +(including the forced-terminal case) is recorded on +``result.manifest.adjudication_succession`` via :meth:`Council._record_adjudication` +exactly like every other adjudication role; the streaming receipt contract is +unchanged (no ``phase="synthesis"`` receipts on this path -- see the comment +above the ``_apply_verdict(result, record_receipts=False)`` call below). + The terminal ``done`` result also carries the auditable :class:`conclave.manifest.ModelHarnessManifest` (CAC-04) and the structured verdict (CAC-05/CAC-06) -- making the "byte-for-byte identical to non-streaming" @@ -51,6 +66,7 @@ from .adapters.base import redact from .logging import get_logger +from .manifest import AdjudicationAttempt from .models import CouncilResult, ModelAnswer, StreamEvent from .providers import call_model_stream from .registry import key_present @@ -237,27 +253,59 @@ async def stream_ask( # stream completes so it can populate the now-existing manifest's # verdict-provenance slots. It is opt-out via the constructor flag and a # no-op when disabled, never raises, and only attaches secret-free content. + # _stream_synthesis walks the synthesizer chain (DSE-1512) and always + # records the full succession ledger via Council._record_adjudication -- + # see that method's docstring for the before-first-delta failover rule. async for event in _stream_synthesis(council, result): yield event - # Streaming synthesis has its own token transport and does not yet emit a - # synthesis receipt. Preserve the established streaming manifest as a - # member-call ledger rather than appending only half of the downstream - # call sequence. Complete receipt capture is currently the buffered/Elite - # contract. + # Streaming synthesis has its own token transport and does not append a + # synthesis receipt on this path (record_receipts=False on both the + # succession ledger call inside _stream_synthesis and here) -- the ledger + # is NOT a receipt and is always recorded regardless. Preserve the + # established streaming manifest as a member-call receipt ledger rather + # than appending only half of the downstream call sequence. Complete + # receipt capture is currently the buffered/Elite contract. await council._apply_verdict(result, record_receipts=False) yield StreamEvent(type="done", result=result) async def _stream_synthesis(council: Council, result: CouncilResult) -> AsyncIterator[StreamEvent]: - """Stream the synthesizer over ``result``'s successful answers, mutating it. - - Mirrors :meth:`Council._synthesize` (no-usable-answers and no-key short - circuits set ``synthesis_error`` exactly the same way), but streams the - synthesizer's tokens as ``synthesis_delta`` events and finishes with a - ``synthesis_done`` event. On any short circuit nothing is yielded (there is - no live token stream) -- the reason lands on ``result.synthesis_error`` and - is visible in the terminal ``done`` event. + """Walk the synthesizer chain over ``result``'s successful answers, mutating it. + + The streaming counterpart of :meth:`Council._synthesize` (DSE-1512): the two + short circuits (no usable member answers; no chain candidate keyed) set + ``result.synthesis_error`` with byte-for-byte identical wording and, in the + no-key case, record the same ``skipped_unkeyed``-per-candidate ledger via + :meth:`Council._skipped_attempts` -- a chain of one with an unkeyed + synthesizer therefore still yields NO events at all, matching the historic + single-candidate streaming behavior exactly. On either short circuit nothing + is yielded (there is no live token stream); the reason lands on + ``result.synthesis_error``, visible in the terminal ``done`` event. + + Past the short circuits this walks ``council.synthesizer_chain`` exactly + like :meth:`Council.adjudicate`, with one addition specific to a live + stream: **the failover boundary is the first emitted delta.** A candidate + that fails before yielding any ``synthesis_delta`` is a pure infrastructure + failure and may fail over per :meth:`Council._classify_outcome` (the same + rule ``adjudicate`` uses); a candidate that already streamed live tokens to + the caller cannot be silently retried elsewhere -- those tokens were already + shown -- so ANY failure after the first delta is terminal for the role + regardless of its failure category. This is implemented by passing + ``category=None`` into the classifier for a post-delta failure (masking it + out of :data:`~conclave.models.FAILOVER_CATEGORIES` membership) while the + ledger entry still records the REAL category for audit purposes. + + Every real call this generator drives -- across every candidate tried -- is + recorded via :meth:`Council._record_adjudication`, which appends the full + succession ledger to ``result.manifest.adjudication_succession``. Per the + streaming receipt contract (see the comment above the + ``_apply_verdict(result, record_receipts=False)`` call in :func:`stream_ask`), + NO ``phase="synthesis"`` receipts are appended on this path -- the ledger is + not a receipt and is recorded regardless (``record_receipts=False`` here + only suppresses receipts, never the ledger). At most one ``synthesis_done`` + is ever yielded, for whichever candidate the walk finally resolves to + (success, terminal failure, or chain exhaustion). """ from .council import _SYNTH_SYSTEM @@ -267,16 +315,20 @@ async def _stream_synthesis(council: Council, result: CouncilResult) -> AsyncIte logger.warning(result.synthesis_error) return - synth_id = council.config.resolve_model_id(council.synthesizer) result.synthesizer = council.synthesizer - result.synthesizer_model_id = synth_id - - if not key_present(synth_id): - result.synthesis_error = ( - f"synthesizer '{council.synthesizer}' ({synth_id}) has no API key; " - "returning raw answers only" + result.synthesizer_model_id = council.config.resolve_model_id(council.synthesizer) + + no_key = council._chain_unkeyed_error("synthesizer", "returning raw answers only") + if no_key is not None: + result.synthesis_error = no_key + logger.warning(no_key) + council._record_adjudication( + result, + council._skipped_attempts("synthesis"), + [], + phase="synthesis", + record_receipts=False, ) - logger.warning(result.synthesis_error) return blocks = "\n\n".join(f"### Answer from {a.name} ({a.model_id})\n{a.answer}" for a in usable) @@ -290,33 +342,87 @@ async def _stream_synthesis(council: Council, result: CouncilResult) -> AsyncIte {"role": "user", "content": user_content}, ] + attempts: list[AdjudicationAttempt] = [] + called: list[ModelAnswer] = [] + chain = council.synthesizer_chain final: ModelAnswer | None = None - async for item in call_model_stream( - council.synthesizer, - synth_id, - messages, - temperature=council.temperature, - timeout=council.timeout, - config=council.config, - ): - if isinstance(item, ModelAnswer): - final = item - else: - yield StreamEvent( - type="synthesis_delta", - name=council.synthesizer, - model_id=synth_id, - text=item, + for index, candidate in enumerate(chain, start=1): + model_id = council.config.resolve_model_id(candidate) + is_last = index == len(chain) + + if not key_present(model_id): + attempts.append( + AdjudicationAttempt( + role="synthesis", + candidate=candidate, + model_id=model_id, + attempt_index=index, + outcome="skipped_unkeyed", + failure_category="unkeyed", + ) ) + continue + + emitted = False + candidate_final: ModelAnswer | None = None + async for item in call_model_stream( + candidate, + model_id, + messages, + temperature=council.temperature, + timeout=council.timeout, + config=council.config, + ): + if isinstance(item, ModelAnswer): + candidate_final = item + else: + emitted = True + yield StreamEvent( + type="synthesis_delta", name=candidate, model_id=model_id, text=item + ) + + if candidate_final is None: + # Defensive: call_model_stream's yield contract always ends with a + # final ModelAnswer. This should never happen in practice. + break + + called.append(candidate_final) + final = candidate_final + failed = not candidate_final.ok + category = candidate_final.failure_category + # A stream that already emitted tokens cannot be retried elsewhere: mask + # the category out of the classifier (forcing "terminal_failure") while + # the ledger entry below still records the REAL category for audit. + outcome = council._classify_outcome( + failed=failed, category=None if emitted else category, is_last=is_last + ) + attempts.append( + AdjudicationAttempt( + role="synthesis", + candidate=candidate, + model_id=model_id, + attempt_index=index, + outcome=outcome, + failure_category=category if failed else None, + http_status=candidate_final.http_status if failed else None, + ) + ) + if outcome == "failed_over": + logger.warning( + "synthesis: '%s' failed (%s) before any output; trying next candidate", + candidate, + category, + ) + continue + break - if final is not None and final.ok: - result.synthesis = final.answer - elif final is not None: - result.synthesis_error = final.error if final is not None: + result.synthesizer, result.synthesizer_model_id = final.name, final.model_id + if final.ok: + result.synthesis = final.answer + else: + result.synthesis_error = final.error yield StreamEvent( - type="synthesis_done", - name=council.synthesizer, - model_id=synth_id, - answer=final, + type="synthesis_done", name=final.name, model_id=final.model_id, answer=final ) + council._record_adjudication(result, attempts, called, phase="synthesis", record_receipts=False) diff --git a/src/conclave/transport.py b/src/conclave/transport.py index ea16f36..70b11db 100644 --- a/src/conclave/transport.py +++ b/src/conclave/transport.py @@ -20,6 +20,7 @@ import httpx from .logging import get_logger +from .models import FailureCategory, categorize_http_status logger = get_logger("transport") @@ -98,6 +99,10 @@ class TransportError(Exception): it into a non-raising ``ModelAnswer.error``. The message is built from the exception type only -- never from request headers -- so it carries no secret. + ``category`` is typed at the raise site (DSE-1512, :data:`conclave.models.FailureCategory`) + so a caller can decide "retry a different provider or stop" from a typed + attribute instead of substring-matching the message. + KEY-LEAK NOTE (audit RANK 1/5): the raise sites route through :func:`_raise_transport_error` (``raise ... from None``) and a boundary clear, so the surfaced TransportError retains **no** reference to the underlying httpx @@ -108,10 +113,25 @@ class TransportError(Exception): chain, or a direct ``err.__context__`` attribute walk. Dropping the chain is deliberate -- the message already names the failure kind, so no diagnostic value is lost. + + ``http_status`` (DSE-1512) carries the HTTP status when the failure came from + a non-2xx response (the streaming path); it stays ``None`` for a network/ + timeout failure, which never produced a response. """ + def __init__( + self, + message: str, + *, + category: FailureCategory = "transport", + http_status: int | None = None, + ) -> None: + super().__init__(message) + self.category: FailureCategory = category + self.http_status: int | None = http_status + -def _raise_transport_error(message: str) -> NoReturn: +def _raise_transport_error(message: str, category: FailureCategory = "transport") -> NoReturn: """Raise a :class:`TransportError` that retains no link to the httpx exception. KEY-LEAK NOTE (audit RANK 1/5). The httpx exception active when this is called @@ -133,7 +153,7 @@ def _raise_transport_error(message: str) -> NoReturn: transport raise sites. The message names only the failure kind, so dropping the chain loses no diagnostic value. """ - raise TransportError(message) from None + raise TransportError(message, category=category) from None def _get_client() -> httpx.AsyncClient: @@ -166,7 +186,9 @@ async def post_json( Raises: TransportError: On any network-level failure (timeout, connection error, or other ``httpx.HTTPError``). The message names only the failure - kind and never echoes the headers, so no key can leak. The underlying + kind and never echoes the headers, so no key can leak. ``category`` + is ``"timeout"`` for a timeout and ``"transport"`` for any other + ``httpx.HTTPError`` (DSE-1512). The underlying httpx exception is deliberately dropped from the cause chain (``__cause__`` and ``__context__`` both cleared) so its header-bearing ``.request`` cannot leak the key via the surfaced error's traceback, @@ -182,7 +204,7 @@ async def post_json( try: response = await client.post(url, headers=headers, json=json_body, timeout=timeout) except httpx.TimeoutException: - _raise_transport_error(f"request timed out after {timeout:.0f}s") + _raise_transport_error(f"request timed out after {timeout:.0f}s", "timeout") except httpx.HTTPError as exc: # Use the exception class NAME, not str(exc): httpx error strings can # include the request URL but never headers, yet we stay conservative. @@ -246,7 +268,11 @@ async def stream_sse( Raises: TransportError: On any network-level failure (timeout, connection error) or a non-2xx streaming status. The message names only the - failure kind / HTTP status and never echoes the headers. The + failure kind / HTTP status and never echoes the headers. + ``category`` (DSE-1512) is ``"timeout"`` for a timeout, + ``"transport"`` for any other network error, and + :func:`conclave.models.categorize_http_status` of the status for a + non-2xx response. The underlying httpx exception is dropped from the cause chain (``__cause__`` and ``__context__`` both cleared) so its header-bearing ``.request`` cannot leak the key via the surfaced error's traceback, @@ -278,7 +304,11 @@ async def stream_sse( # on ModelAnswer.error or is logged. No streamed text delta is # emitted on this path (deltas carry only parsed answer content), # so the only surface for this string is that redacted final answer. - raise TransportError(f"HTTP {response.status_code}: {detail}") + raise TransportError( + f"HTTP {response.status_code}: {detail}", + category=categorize_http_status(response.status_code), + http_status=response.status_code, + ) event_name = "" data_lines: list[str] = [] @@ -307,7 +337,7 @@ async def stream_sse( # Map to TransportError with the chain dropped (audit RANK 1/5). The # streaming httpx exception also carries ``.request.headers`` with the # live auth value; _raise_transport_error raises ``from None``. - _raise_transport_error(f"request timed out after {timeout:.0f}s") + _raise_transport_error(f"request timed out after {timeout:.0f}s", "timeout") except httpx.HTTPError as exc: # Drop the httpx exception from the cause chain so its header-bearing # ``.request`` cannot leak the key (audit RANK 1/5). diff --git a/src/conclave/verdict_synthesis.py b/src/conclave/verdict_synthesis.py index 526a154..2187661 100644 --- a/src/conclave/verdict_synthesis.py +++ b/src/conclave/verdict_synthesis.py @@ -58,7 +58,24 @@ again. If it still fails (or the model returned an error / empty answer), the verdict is absent (``verdict=None``) with a recorded reason — it NEVER raises. The extractor's identity + prompt version is recorded as provenance on EVERY path -(success and all three absent paths). +(success and all three absent paths). This repair-retry behavior is unchanged +by DSE-1512: the retry is always made against the SAME model that made the +initial call, even when that call failed for an infrastructure reason -- only +:meth:`conclave.council.Council._apply_verdict`'s outer chain walk (one +``extract_verdict`` call per candidate) advances to a different model. + +On the ``REASON_EXTRACTION_FAILED`` path, :class:`VerdictSynthesisResult` also +carries ``failure_category``/``http_status`` (DSE-1512, review-corrected) so a +caller can tell an infrastructure failure (eligible for chain failover) apart +from a model that answered unusably (terminal for the role) without +re-deriving it from the receipts. The category is decided by whether the +candidate EVER answered across its attempts, not by which attempt happened to +run last: a candidate that answered on either the initial call or the repair +retry is terminal (``"malformed_response"``) even if its OTHER attempt failed +for an infrastructure reason -- a content failure can never be laundered into +a failover by an unrelated infra hiccup on the repair. Only when every attempt +made actually errored does the category reflect the last errored attempt's +real :class:`~conclave.models.FailureCategory`. """ from __future__ import annotations @@ -71,7 +88,7 @@ from .adapters.base import OutputContract, redact from .logging import get_logger from .manifest import ProviderExecutionReceipt, VerdictExtraction -from .models import ModelAnswer +from .models import FailureCategory, ModelAnswer from .providers import call_model, receipt_from_answer from .verdict import ( VERDICT_EXTRACTION_PROMPT_VERSION, @@ -83,6 +100,9 @@ ) __all__ = [ + "REASON_EXTRACTION_FAILED", + "REASON_OPEN_ENDED", + "REASON_TOO_FEW", "VERDICT_EXTRACTION_PROMPT_VERSION", "VERDICT_REPAIR_ERROR_DETAIL_MAX_BYTES", "VerdictSynthesisResult", @@ -102,6 +122,15 @@ _REASON_OPEN_ENDED = "open-ended prompt (no decision/review to adjudicate)" _REASON_EXTRACTION_FAILED = "verdict extraction failed schema validation" +# Public aliases (DSE-1512). ``council.py``'s ``_apply_verdict`` (and any other +# out-of-module reader) imports these rather than reaching for a +# leading-underscore "private" symbol. The private names above remain the +# canonical definitions and every existing internal use/import of them is +# unchanged -- these are additive aliases, not a rename. +REASON_TOO_FEW = _REASON_TOO_FEW +REASON_OPEN_ENDED = _REASON_OPEN_ENDED +REASON_EXTRACTION_FAILED = _REASON_EXTRACTION_FAILED + def _bounded_repair_error(detail: object) -> str: scrubbed = redact(str(detail)) @@ -168,12 +197,32 @@ class VerdictSynthesisResult(BaseModel): schema validation"``), or ``None`` when a verdict is present. attempt_receipts: One secret-free receipt for each actual extraction or repair call. The N<2 gate makes no call and therefore yields none. + failure_category: Populated ONLY when ``verdict_absent_reason == + REASON_EXTRACTION_FAILED`` (DSE-1512, review-corrected), so + :meth:`conclave.council.Council._apply_verdict` can apply the same + chain-failover rule every other adjudication role uses. ``None`` on + every other path (success, N<2, open-ended). Decided by whether the + candidate EVER answered (see :func:`_extraction_failure_category`): + when every attempt made errored, this is the LAST errored attempt's + :attr:`~conclave.models.ModelAnswer.failure_category` verbatim, + which may or may not be in :data:`conclave.models.FAILOVER_CATEGORIES`; + when any attempt answered (even unusably), this is the fixed + literal ``"malformed_response"`` -- never a category the model + itself could have produced, and never in ``FAILOVER_CATEGORIES``, + so a candidate that responded even once is always terminal for the + role regardless of what its other attempt did. + http_status: The HTTP status of the same attempt ``failure_category`` + was derived from, when the failure came from an HTTP response; + ``None`` otherwise (including the ``"malformed_response"`` case, + which has no HTTP failure). """ verdict: CouncilVerdict | None = None extraction: VerdictExtraction verdict_absent_reason: str | None = Field(default=None) attempt_receipts: list[ProviderExecutionReceipt] = Field(default_factory=list) + failure_category: FailureCategory | None = None + http_status: int | None = None def _verdict_attempt_receipt( @@ -211,6 +260,50 @@ def _verdict_attempt_receipt( ) +def _extraction_failure_category( + initial: ModelAnswer, retry: ModelAnswer | None +) -> tuple[FailureCategory | None, int | None]: + """Classify the terminal (repair-exhausted) extraction failure (DSE-1512 review). + + The category is decided by whether the candidate EVER answered across its + attempts -- NOT by which attempt happened to run last. This closes a real + bug: the initial call can answer (200 + unparsable prose) while the SAME- + MODEL repair retry then hits an unrelated infrastructure error (e.g. a + 429 mid-repair); classifying from "the last attempt" alone would read that + as an infrastructure failure and wrongly fail the role over to the next + chain candidate even though the candidate demonstrably answered. + + * If EITHER ``initial`` or ``retry`` answered (``error is None``), the + candidate answered -- return the fixed literal ``"malformed_response"`` + (never in :data:`conclave.models.FAILOVER_CATEGORIES`, so it is always + terminal for the role: a model that answered is never second-guessed by + another vendor, regardless of what its other attempt did). + * Only when EVERY attempt made actually errored does this return the LAST + errored attempt's typed :attr:`~conclave.models.ModelAnswer.failure_category` + / :attr:`~conclave.models.ModelAnswer.http_status` verbatim (an + infrastructure failure, which may or may not be in + ``FAILOVER_CATEGORIES``). + + :meth:`conclave.council.Council._apply_verdict` reads this pair to decide + whether the verdict-extraction role fails over to the next chain candidate. + + Args: + initial: The initial extraction attempt. + retry: The repair retry, or ``None`` when no repair was attempted + (the initial call already validated, or errored and no retry + occurred). + + Returns: + ``(failure_category, http_status)``, either fully populated (an infra + failure on every attempt) or ``("malformed_response", None)`` (the + candidate answered on at least one attempt). + """ + if initial.error is None or (retry is not None and retry.error is None): + return "malformed_response", None + last_attempt = retry if retry is not None else initial + return last_attempt.failure_category, last_attempt.http_status + + def _responding(member_answers: list[ModelAnswer]) -> list[ModelAnswer]: """Return the members that produced a non-empty answer, in order. @@ -573,6 +666,7 @@ async def extract_verdict( protocol_version=protocol_version, ) ] + retry: ModelAnswer | None = None if extraction is None: repair_messages = messages + [ { @@ -611,11 +705,14 @@ async def extract_verdict( if extraction is None: # Repair exhausted — degrade gracefully (DD-2), never raise. logger.warning("verdict extraction failed schema validation after repair: %s", errors) + failure_category, http_status = _extraction_failure_category(answer, retry) return VerdictSynthesisResult( verdict=None, extraction=extraction_provenance, verdict_absent_reason=_REASON_EXTRACTION_FAILED, attempt_receipts=attempt_receipts, + failure_category=failure_category, + http_status=http_status, ) # Step 4 — open-ended prompt → synthesis-only, no verdict (DD-2). diff --git a/tests/conftest.py b/tests/conftest.py index 3f9349f..077abdd 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -188,6 +188,86 @@ def clear_keys(monkeypatch) -> None: monkeypatch.delenv(var, raising=False) +@pytest.fixture +def keys(monkeypatch) -> None: + """Set every DSE-1512 adjudication-chain provider key to a dummy value. + + Covers the five friendly names used across the ``Council.adjudicate`` + succession tests (``claude``/``grok``/``gemini``/``openai``/``mistral``), + so a chain like ``"openai>mistral"`` (used by the Elite synthesis-failover + tests) is fully keyed without each test hand-rolling the env vars. + """ + for var in ( + "ANTHROPIC_API_KEY", + "XAI_API_KEY", + "GEMINI_API_KEY", + "OPENAI_API_KEY", + "MISTRAL_API_KEY", + ): + monkeypatch.setenv(var, "dummy") + + +def make_failed_answer( + name: str, model_id: str, category: str, status: int | None = None +) -> ModelAnswer: + """Build a failed :class:`~conclave.models.ModelAnswer` with a typed category. + + Shared by the ``Council.adjudicate`` succession tests + (``tests/test_adjudication.py``, ``tests/test_council.py``, + ``tests/test_modes.py``, ``tests/test_elite_mode.py``) so every test drives + the failover rule from the same typed-failure shape rather than raising a + bare exception (which would carry no ``failure_category``). + """ + return ModelAnswer( + name=name, + model_id=model_id, + error=f"{name} failed", + failure_category=category, + http_status=status, + ) + + +def make_ok_answer(name: str, model_id: str) -> ModelAnswer: + """Build a successful :class:`~conclave.models.ModelAnswer` for a member/candidate.""" + return ModelAnswer( + name=name, model_id=model_id, answer=f"{name} says yes", answer_id=f"{name}-1" + ) + + +def install_council_script(monkeypatch, script: dict[str, ModelAnswer]) -> list[str]: + """Patch the council ``call_model`` seam to return a fixed answer per name. + + Unlike :func:`make_response`/``patch_call_model`` (which drive a handler + keyed by model id + messages), this seam is keyed by the friendly + ``name`` and returns the SAME :class:`~conclave.models.ModelAnswer` object + every time that name is called -- exactly what the adjudication-succession + tests need, since a council member and an adjudication-chain candidate are + called through the identical ``conclave.council.call_model`` seam (a test + exercising a member fan-out plus a chain failover needs a script entry for + every name involved, members included). Only the council seam is patched + (not the verdict-extraction seam); callers exercising those tests pass + ``extract_verdict=False`` to keep the verdict path out of the picture. + + Args: + monkeypatch: The pytest monkeypatch fixture. + script: Friendly name -> the fixed :class:`ModelAnswer` to return for + every call to that name. + + Returns: + The call log: one friendly name appended per call, in call order. + """ + import conclave.council as council_mod + + calls: list[str] = [] + + async def fake(name, model_id, messages, **kwargs): + calls.append(name) + return script[name] + + monkeypatch.setattr(council_mod, "call_model", fake) + return calls + + @pytest.fixture def conclave_caplog(caplog): """caplog that reliably captures the non-propagating ``conclave`` logger. diff --git a/tests/test_adjudication.py b/tests/test_adjudication.py new file mode 100644 index 0000000..673b704 --- /dev/null +++ b/tests/test_adjudication.py @@ -0,0 +1,185 @@ +"""Council.adjudicate walks the synthesizer chain with the infra-only failover rule.""" + +from __future__ import annotations + +import pytest +from pydantic import ValidationError + +from conclave.config import ConclaveConfig +from conclave.council import Council +from conclave.manifest import AdjudicationAttempt, ModelHarnessManifest +from conclave.models import CouncilResult +from tests.conftest import install_council_script, make_failed_answer, make_ok_answer + +CFG = ConclaveConfig(models={"claude": "anthropic/c", "grok": "xai/g", "gemini": "gemini/m"}) + + +async def test_chain_of_one_success(monkeypatch, keys): + calls = install_council_script(monkeypatch, {"claude": make_ok_answer("claude", "anthropic/c")}) + c = Council(models=["grok"], synthesizer="claude", config=CFG) + out = await c.adjudicate("synthesis", "sys", "user") + assert out.answer.ok and out.name == "claude" and calls == ["claude"] + assert [a.outcome for a in out.attempts] == ["success"] + + +async def test_auth_failure_advances(monkeypatch, keys): + calls = install_council_script( + monkeypatch, + { + "claude": make_failed_answer("claude", "anthropic/c", "auth", 401), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG) + out = await c.adjudicate("synthesis", "sys", "user") + assert out.answer.ok and out.name == "grok" and out.model_id == "xai/g" + assert calls == ["claude", "grok"] + assert [(a.candidate, a.outcome, a.failure_category, a.http_status) for a in out.attempts] == [ + ("claude", "failed_over", "auth", 401), + ("grok", "success", None, None), + ] + + +async def test_bad_request_is_terminal(monkeypatch, keys): + calls = install_council_script( + monkeypatch, + { + "claude": make_failed_answer("claude", "anthropic/c", "bad_request", 400), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG) + out = await c.adjudicate("synthesis", "sys", "user") + assert not out.answer.ok and out.name == "claude" + assert calls == ["claude"] # grok was never consulted + assert [a.outcome for a in out.attempts] == ["terminal_failure"] + + +async def test_malformed_is_terminal(monkeypatch, keys): + calls = install_council_script( + monkeypatch, + { + "claude": make_failed_answer("claude", "anthropic/c", "malformed_response"), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG) + out = await c.adjudicate("judge", "sys", "user") + assert calls == ["claude"] and out.attempts[0].outcome == "terminal_failure" + + +async def test_unkeyed_candidate_is_skipped_without_call(monkeypatch): + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + monkeypatch.setenv("XAI_API_KEY", "dummy") + calls = install_council_script(monkeypatch, {"grok": make_ok_answer("grok", "xai/g")}) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG) + out = await c.adjudicate("synthesis", "sys", "user") + assert calls == ["grok"] + assert [a.outcome for a in out.attempts] == ["skipped_unkeyed", "success"] + + +async def test_chain_exhausted(monkeypatch, keys): + calls = install_council_script( + monkeypatch, + { + "claude": make_failed_answer("claude", "anthropic/c", "quota", 429), + "grok": make_failed_answer("grok", "xai/g", "unavailable", 503), + }, + ) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG) + out = await c.adjudicate("synthesis", "sys", "user") + assert calls == ["claude", "grok"] + assert not out.answer.ok and out.name == "grok" + assert [a.outcome for a in out.attempts] == ["failed_over", "exhausted"] + + +async def test_all_unkeyed_returns_no_answer(monkeypatch): + for var in ("ANTHROPIC_API_KEY", "XAI_API_KEY"): + monkeypatch.delenv(var, raising=False) + calls = install_council_script(monkeypatch, {}) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG) + out = await c.adjudicate("synthesis", "sys", "user") + assert out.answer is None and calls == [] + assert [a.outcome for a in out.attempts] == ["skipped_unkeyed", "skipped_unkeyed"] + + +async def test_synthesize_blocks_all_unkeyed_synthetic_answer_is_typed(monkeypatch): + """The synthetic error answer for an all-unkeyed chain is typed 'unkeyed'. + + ``synthesize_blocks`` fabricates a ``ModelAnswer`` when no chain candidate + has a key. It must carry ``failure_category="unkeyed"`` like every other + unkeyed outcome, not the pre-DSE-1512 untyped default of ``None``. + """ + for var in ("ANTHROPIC_API_KEY", "XAI_API_KEY"): + monkeypatch.delenv(var, raising=False) + install_council_script(monkeypatch, {}) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG) + answer = await c.synthesize_blocks("sys", "user") + assert not answer.ok + assert answer.failure_category == "unkeyed" + + +def test_record_adjudication_appends_ledger_and_receipts(): + c = Council(models=["grok"], synthesizer="claude", config=CFG) + result = CouncilResult( + prompt="p", + manifest=ModelHarnessManifest(request_id="r", conclave_version="t", mode="synthesize"), + ) + attempts = [ + AdjudicationAttempt( + role="synthesis", + candidate="claude", + model_id="anthropic/c", + attempt_index=1, + outcome="failed_over", + failure_category="auth", + http_status=401, + ), + AdjudicationAttempt( + role="synthesis", candidate="grok", model_id="xai/g", attempt_index=2, outcome="success" + ), + ] + called = [ + make_failed_answer("claude", "anthropic/c", "auth", 401), + make_ok_answer("grok", "xai/g"), + ] + c._record_adjudication(result, attempts, called, phase="synthesis") + m = result.manifest + assert m.adjudication_succession == attempts + assert [(r.phase, r.attempt, r.outcome) for r in m.receipts] == [ + ("synthesis", 1, "failed"), + ("synthesis", 2, "success"), + ] + assert m.secret_safety == "verified_no_secrets" + + +def test_ledger_has_no_free_text_fields(): + fields = set(AdjudicationAttempt.model_fields) + assert fields == { + "role", + "candidate", + "model_id", + "attempt_index", + "outcome", + "failure_category", + "http_status", + } + + +def test_failure_category_is_a_bounded_literal(): + """AdjudicationAttempt.failure_category rejects an arbitrary string (CSO finding, A4). + + A raw provider error string could smuggle a word the secret-safety scan + forbids (e.g. "Authorization"); a bounded :data:`conclave.models.FailureCategory` + literal makes that structurally impossible rather than relying on every + call site to remember to pass a bounded category. + """ + with pytest.raises(ValidationError): + AdjudicationAttempt( + role="synthesis", + candidate="claude", + model_id="anthropic/c", + attempt_index=1, + outcome="terminal_failure", + failure_category="Missing Authorization header", + ) diff --git a/tests/test_cache.py b/tests/test_cache.py index 612c29c..082f553 100644 --- a/tests/test_cache.py +++ b/tests/test_cache.py @@ -27,8 +27,9 @@ from conclave import Council from conclave import cache as cache_mod from conclave.config import ConclaveConfig, CustomEndpoint -from conclave.models import ModelAnswer -from tests.conftest import make_response +from conclave.manifest import AdjudicationAttempt, ModelHarnessManifest +from conclave.models import CouncilResult, ModelAnswer +from tests.conftest import install_council_script, make_failed_answer, make_ok_answer, make_response @pytest.fixture @@ -53,6 +54,18 @@ def _config(cache: bool = False) -> ConclaveConfig: ) +def _chain_config() -> ConclaveConfig: + """A deterministic config for the DSE-1512 adjudication-succession cache tests. + + Mirrors ``tests/test_adjudication.py``'s ``CFG`` so the chain candidate + friendly names/model ids line up with :func:`tests.conftest.install_council_script`. + """ + return ConclaveConfig( + models={"claude": "anthropic/c", "grok": "xai/g", "gemini": "gemini/m"}, + cache=True, + ) + + def _set_keys(monkeypatch) -> None: """Set every provider key to a dummy non-empty value.""" for var in ("XAI_API_KEY", "GEMINI_API_KEY", "ANTHROPIC_API_KEY", "PERPLEXITY_API_KEY"): @@ -662,3 +675,416 @@ def handler(model, messages, **kwargs): # Two distinct cache files exist. files = list(cache_home.glob("*.json")) assert len(files) == 2 + + +# --------------------------------------------------------------------------- # +# DSE-1512: chain identity + no-store on successor/exhausted adjudication +# --------------------------------------------------------------------------- # + + +def test_identity_includes_full_chain(): + """A different successor ladder invalidates a prior entry.""" + base = dict( + prompt="p", + mode="synthesize", + members=[("g", "xai/g")], + synthesizer="claude", + synthesizer_model_id="anthropic/c", + temperature=0.7, + ) + one = cache_mod.make_key(**base, synthesizer_chain=[("claude", "anthropic/c")]) + two = cache_mod.make_key( + **base, synthesizer_chain=[("claude", "anthropic/c"), ("grok", "xai/g")] + ) + assert one != two + + doc = cache_mod.build_identity( + **base, synthesizer_chain=[("claude", "anthropic/c"), ("grok", "xai/g")] + ) + assert doc["synthesizer"] == ["claude", "anthropic/c"] # legacy key kept + assert doc["synthesizer_chain"] == [["claude", "anthropic/c"], ["grok", "xai/g"]] + + +def test_identity_chain_defaults_to_primary_when_omitted(): + """A direct caller that never passes ``synthesizer_chain`` still gets a stable value.""" + base = dict( + prompt="p", + mode="synthesize", + members=[("g", "xai/g")], + synthesizer="claude", + synthesizer_model_id="anthropic/c", + temperature=0.7, + ) + omitted = cache_mod.make_key(**base) + explicit = cache_mod.make_key(**base, synthesizer_chain=[("claude", "anthropic/c")]) + assert omitted == explicit + + doc = cache_mod.build_identity(**base) + assert doc["synthesizer_chain"] == [["claude", "anthropic/c"]] + + +def test_cache_format_version_bumped(): + assert cache_mod.CACHE_FORMAT_VERSION == "4" + + +async def test_result_adjudicated_by_successor_is_not_stored(monkeypatch, keys, cache_home): + """A run where the primary failed over to a successor is never persisted. + + gemini (member) ok, claude (primary) quota 429 -> failed over, grok + (successor) ok -> success. Because the primary failed for an + infrastructure reason, the run must not be cached: a second identical + ``ask`` re-calls every provider rather than replaying the successor's + answer from cache. + """ + calls = install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "claude": make_failed_answer("claude", "anthropic/c", "quota", 429), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + c = Council( + models=["gemini"], + synthesizer="claude>grok", + config=_chain_config(), + cache=True, + extract_verdict=False, + ) + r1 = await c.ask("q") + r2 = await c.ask("q") + + assert r1.cached is False and r2.cached is False # second run was NOT served from cache + assert calls.count("gemini") == 2 and calls.count("grok") == 2 + assert not list(cache_home.glob("*.json")) # nothing was ever written + assert [a.outcome for a in r2.manifest.adjudication_succession] == [ + "failed_over", + "success", + ] + + +async def test_result_adjudicated_by_primary_is_stored(monkeypatch, keys, cache_home): + """A run where the primary answers cleanly is cached like any other run.""" + calls = install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "claude": make_ok_answer("claude", "anthropic/c"), + }, + ) + c = Council( + models=["gemini"], + synthesizer="claude>grok", + config=_chain_config(), + cache=True, + extract_verdict=False, + ) + r1 = await c.ask("q") + r2 = await c.ask("q") + + assert r1.cached is False + assert r2.cached is True + assert calls.count("gemini") == 1 and calls.count("claude") == 1 # not re-called + assert [a.outcome for a in r2.manifest.adjudication_succession] == ["success"] + + +async def test_exhausted_run_is_not_stored(monkeypatch, keys, cache_home): + """A chain-exhausted run (every candidate failed for an infra reason) is not cached. + + This is a deliberate, narrow behavior change from v1.3.0 (see + ``Council._cached_run``'s docstring): caching a run whose whole chain was + down means an operator would get the same failure back from cache after + the outage clears. + """ + calls = install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "claude": make_failed_answer("claude", "anthropic/c", "quota", 429), + "grok": make_failed_answer("grok", "xai/g", "unavailable", 503), + }, + ) + c = Council( + models=["gemini"], + synthesizer="claude>grok", + config=_chain_config(), + cache=True, + extract_verdict=False, + ) + r1 = await c.ask("q") + r2 = await c.ask("q") + + assert r1.cached is False and r1.degraded is True + assert r2.cached is False and r2.degraded is True + assert calls.count("gemini") == 2 # ran again on the second call, not from cache + assert not list(cache_home.glob("*.json")) + assert [a.outcome for a in r2.manifest.adjudication_succession] == [ + "failed_over", + "exhausted", + ] + + +async def test_terminal_failure_run_is_stored(monkeypatch, keys, cache_home): + """A terminal-failure run (the model answered, just unusably) is still cached. + + claude returns a bad_request (400): the request was wrong, not an + infrastructure problem, so the chain never fails over and the degraded + result is cached exactly like any other content failure (unchanged from + v1.3.0). + """ + calls = install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "claude": make_failed_answer("claude", "anthropic/c", "bad_request", 400), + }, + ) + c = Council( + models=["gemini"], + synthesizer="claude>grok", + config=_chain_config(), + cache=True, + extract_verdict=False, + ) + r1 = await c.ask("q") + r2 = await c.ask("q") + + assert r1.cached is False and r1.degraded is True + assert r2.cached is True and r2.degraded is True + assert calls.count("claude") == 1 # only the first, live run called claude + assert [a.outcome for a in r2.manifest.adjudication_succession] == ["terminal_failure"] + + +async def test_chain_of_one_unkeyed_run_is_not_stored(monkeypatch, cache_home): + """A chain-of-one unkeyed synthesizer IS primary_failed_over (DSE-1512 review, + uniform rule) and is therefore NOT cached -- the ledger's lone + ``skipped_unkeyed`` entry at attempt_index 1 means the primary never + adjudicated, even though there is no successor to have adjudicated instead. + Runs with ``extract_verdict=False`` (below) and the default + ``extract_verdict=True`` (the second test) must agree: the rule does not + depend on which roles ran. + """ + monkeypatch.setenv("GEMINI_API_KEY", "dummy") + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + calls = install_council_script(monkeypatch, {"gemini": make_ok_answer("gemini", "gemini/m")}) + c = Council( + models=["gemini"], + synthesizer="claude", + config=_chain_config(), + cache=True, + extract_verdict=False, + ) + r1 = await c.ask("q") + r2 = await c.ask("q") + + assert r1.primary_failed_over is True + assert r1.cached is False and r2.cached is False + assert not list(cache_home.glob("*.json")) # nothing was ever written + assert calls.count("gemini") == 2 # second run re-called the providers, not cached + + +async def test_chain_of_one_unkeyed_run_is_not_stored_with_verdict_extraction( + monkeypatch, cache_home +): + """Same as above with the default ``extract_verdict=True``.""" + monkeypatch.setenv("GEMINI_API_KEY", "dummy") + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + calls = install_council_script(monkeypatch, {"gemini": make_ok_answer("gemini", "gemini/m")}) + c = Council( + models=["gemini"], + synthesizer="claude", + config=_chain_config(), + cache=True, + ) + r1 = await c.ask("q") + r2 = await c.ask("q") + + assert r1.primary_failed_over is True + assert r1.cached is False and r2.cached is False + assert not list(cache_home.glob("*.json")) + assert calls.count("gemini") == 2 + + +# --------------------------------------------------------------------------- # +# CouncilResult.primary_failed_over truth table (DSE-1512 review, uniform rule). +# Constructs a ModelHarnessManifest directly -- no council run -- to pin the +# computed field's rule independent of how any particular role's ledger gets +# built. +# --------------------------------------------------------------------------- # + + +def _attempt(role: str, outcome: str, index: int) -> AdjudicationAttempt: + """Build one succession-ledger entry for the truth-table test below.""" + return AdjudicationAttempt( + role=role, + candidate="claude", + model_id="anthropic/c", + attempt_index=index, + outcome=outcome, + ) + + +def _manifest_with(attempts: list[AdjudicationAttempt]) -> ModelHarnessManifest: + return ModelHarnessManifest( + request_id="r", + conclave_version="v", + mode="synthesize", + adjudication_succession=attempts, + ) + + +@pytest.mark.parametrize( + ("attempts", "expected"), + [ + pytest.param([], False, id="empty_ledger"), + pytest.param( + [_attempt("synthesis", "terminal_failure", 1)], False, id="terminal_failure_at_1" + ), + pytest.param( + [_attempt("synthesis", "skipped_unkeyed", 1)], True, id="skipped_unkeyed_at_1" + ), + pytest.param( + [_attempt("synthesis", "skipped_unkeyed", 1), _attempt("synthesis", "success", 2)], + True, + id="skipped_unkeyed_then_success", + ), + pytest.param( + [ + _attempt("synthesis", "skipped_unkeyed", 1), + _attempt("synthesis", "terminal_failure", 2), + ], + True, + id="skipped_unkeyed_then_terminal_failure", + ), + pytest.param( + [ + _attempt("synthesis", "failed_over", 1), + _attempt("synthesis", "skipped_unkeyed", 2), + ], + True, + id="failed_over_then_skipped_unkeyed", + ), + pytest.param([_attempt("synthesis", "success", 1)], False, id="success_at_1"), + pytest.param([_attempt("synthesis", "exhausted", 1)], True, id="exhausted_at_1"), + pytest.param( + [_attempt("synthesis", "success", 1), _attempt("judge", "failed_over", 1)], + True, + id="second_role_index_1_failed_over", + ), + ], +) +def test_primary_failed_over_truth_table(attempts, expected): + """primary_failed_over depends only on each role's attempt_index==1 outcome, + uniformly across every role, independent of whether a council run ever + actually happened. + """ + result = CouncilResult(prompt="p", manifest=_manifest_with(attempts)) + assert result.primary_failed_over is expected + + +# --------------------------------------------------------------------------- # +# DSE-1512 review, Unit A2: ask_stream's cache store must honor the same +# no-store-on-primary-failover rule as the buffered path (_cached_run). +# --------------------------------------------------------------------------- # + + +def _stream_chain_config() -> ConclaveConfig: + """Mirrors ``_chain_config`` with a friendly-name roster wide enough for + both the sole council member (``gemini``) and the ``claude>grok`` chain. + """ + return ConclaveConfig( + models={"claude": "anthropic/c", "grok": "xai/g", "gemini": "gemini/m"}, + cache=True, + ) + + +def _install_stream_script(monkeypatch, script: dict[str, list]) -> list[str]: + """Patch the streaming ``call_model_stream`` seam with a per-name script. + + Mirrors ``tests/test_streaming.py``'s helper of the same name: council + members and synthesizer-chain candidates share this one seam, so a script + entry is needed for every name a test's run touches. + """ + import conclave.streaming as streaming_mod + + calls: list[str] = [] + + async def fake_stream(name, model_id, messages, *, temperature=0.7, timeout=120.0, config=None): + calls.append(name) + for item in script[name]: + yield item + + monkeypatch.setattr(streaming_mod, "call_model_stream", fake_stream) + return calls + + +async def test_stream_successor_run_is_not_stored(monkeypatch, keys, cache_home): + """A stream whose primary synthesizer fails over is never persisted. + + claude yields only an errored final (429, no deltas) so it fails over + before any output; grok then streams cleanly. Because the primary failed + for an infrastructure reason, the run must not reach the result cache -- + a subsequent buffered ``ask`` re-calls every provider rather than + replaying the successor's answer from a stale entry. + """ + _install_stream_script( + monkeypatch, + { + "gemini": ["gemini ", "says yes", make_ok_answer("gemini", "gemini/m")], + "claude": [make_failed_answer("claude", "anthropic/c", "quota", 429)], + "grok": ["grok ", "says yes", make_ok_answer("grok", "xai/g")], + }, + ) + c = Council( + models=["gemini"], + synthesizer="claude>grok", + config=_stream_chain_config(), + cache=True, + extract_verdict=False, + ) + + events = [e async for e in c.ask_stream("q")] + result = events[-1].result + assert result.cached is False + assert result.primary_failed_over is True + assert not list(cache_home.glob("*.json")) # nothing was ever written + + # A subsequent buffered ask is NOT served from cache: every provider is + # called again rather than replaying the successor's answer. + buffered_calls = install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "claude": make_failed_answer("claude", "anthropic/c", "quota", 429), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + r2 = await c.ask("q") + assert r2.cached is False + assert buffered_calls == ["gemini", "claude", "grok"] + + +async def test_stream_primary_run_is_stored(monkeypatch, keys, cache_home): + """A stream whose primary synthesizer succeeds is cached like any other run.""" + _install_stream_script( + monkeypatch, + { + "gemini": ["gemini ", "says yes", make_ok_answer("gemini", "gemini/m")], + "claude": ["claude ", "says yes", make_ok_answer("claude", "anthropic/c")], + }, + ) + c = Council( + models=["gemini"], + synthesizer="claude>grok", + config=_stream_chain_config(), + cache=True, + extract_verdict=False, + ) + + first = [e async for e in c.ask_stream("q")] + assert first[-1].result.cached is False + assert first[-1].result.primary_failed_over is False + assert len(list(cache_home.glob("*.json"))) == 1 + + r2 = await c.ask("q") + assert r2.cached is True diff --git a/tests/test_cli.py b/tests/test_cli.py index f78fb02..5d42bce 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -33,7 +33,7 @@ from conclave.config import ConclaveConfig from conclave.models import CouncilResult, EliteResult, ModelAnswer from conclave.verdict import CouncilVerdict -from tests.conftest import make_response +from tests.conftest import install_council_script, make_failed_answer, make_ok_answer, make_response runner = CliRunner() @@ -894,3 +894,114 @@ def test_providers_command_lists_new_first_class_providers(monkeypatch, tmp_path # No secret VALUE ever appears (only the env-var NAME does). for val in secrets.values(): assert val not in result.output + + +# --------------------------------------------------------------------------- # +# Synthesizer chain / adjudication succession CLI surface (DSE-1512, Task 9). +# --------------------------------------------------------------------------- # + + +def test_cli_synthesizer_chain_successor_exits_zero_json(monkeypatch, patch_cli_config, keys): + """A successor adjudication is a clean run: exit 0, degraded false. + + ``--json`` gains ``primary_failed_over: true`` (additive) and ``synthesizer`` + names the candidate that actually adjudicated, not the declared primary. + """ + install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/gemini-2.5-pro"), + "claude": make_failed_answer("claude", "anthropic/claude-sonnet-4-6", "quota", 402), + "grok": make_ok_answer("grok", "xai/grok-4.3"), + }, + ) + result = runner.invoke( + cli.app, + ["ask", "hello", "--council", "gemini", "--synthesizer", "claude>grok", "--json"], + ) + + assert result.exit_code == 0 + payload = json.loads(result.stdout) + assert payload["degraded"] is False + assert payload["primary_failed_over"] is True + assert payload["synthesizer"] == "grok" + ledger = payload["manifest"]["adjudication_succession"] + assert [(a["candidate"], a["outcome"]) for a in ledger if a["role"] == "synthesis"] == [ + ("claude", "failed_over"), + ("grok", "success"), + ] + + +def test_cli_synthesizer_chain_exhausted_exits_degraded(monkeypatch, patch_cli_config, keys): + """Every candidate failing for an infrastructure reason is degraded, not clean.""" + install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/gemini-2.5-pro"), + "claude": make_failed_answer("claude", "anthropic/claude-sonnet-4-6", "quota", 402), + "grok": make_failed_answer("grok", "xai/grok-4.3", "unavailable", 503), + }, + ) + result = runner.invoke( + cli.app, + ["ask", "hello", "--council", "gemini", "--synthesizer", "claude>grok", "--json"], + ) + + assert result.exit_code == cli._DEGRADED_EXIT_CODE + payload = json.loads(result.stdout) + assert payload["degraded"] is True + assert payload["primary_failed_over"] is True + ledger = payload["manifest"]["adjudication_succession"] + assert [(a["candidate"], a["outcome"]) for a in ledger if a["role"] == "synthesis"] == [ + ("claude", "failed_over"), + ("grok", "exhausted"), + ] + + +def test_cli_human_output_prints_failover_note(monkeypatch, patch_cli_config, keys): + """The human render path prints one dim failover note per role that failed over.""" + install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/gemini-2.5-pro"), + "claude": make_failed_answer("claude", "anthropic/claude-sonnet-4-6", "quota", 402), + "grok": make_ok_answer("grok", "xai/grok-4.3"), + }, + ) + result = runner.invoke( + cli.app, + ["ask", "hello", "--council", "gemini", "--synthesizer", "claude>grok"], + ) + + assert result.exit_code == 0 + assert "adjudication failover: synthesis: claude (quota, HTTP 402) → grok" in result.output + + +def test_cli_human_output_unchanged_without_failover( + monkeypatch, patch_cli_config, patch_call_model +): + """A chain-of-one clean run prints no failover note (byte-identical to v1.3.0).""" + _all_keys(monkeypatch) + + def handler(model, messages, **kwargs): + return make_response(f"answer from {model}") + + patch_call_model(handler) + result = runner.invoke(cli.app, ["ask", "hello", "--council", "grok,gemini"]) + + assert result.exit_code == 0 + assert "adjudication failover" not in result.output + + +def test_cli_providers_footer_shows_chain(monkeypatch, tmp_path): + """`conclave providers` shows the configured synthesizer chain when set.""" + config_path = tmp_path / "config.yml" + config_path.write_text("synthesizer_chain: [claude, grok]\n", encoding="utf-8") + monkeypatch.setenv("CONCLAVE_CONFIG", str(config_path)) + from conclave.config import clear_config_cache + + clear_config_cache() + + result = runner.invoke(cli.app, ["providers"]) + assert result.exit_code == 0 + assert "synthesizer chain: claude > grok" in result.output diff --git a/tests/test_council.py b/tests/test_council.py index ba55114..076f779 100644 --- a/tests/test_council.py +++ b/tests/test_council.py @@ -12,7 +12,7 @@ from conclave import Council from conclave.config import ConclaveConfig -from tests.conftest import make_response +from tests.conftest import install_council_script, make_failed_answer, make_ok_answer, make_response def _all_keys(monkeypatch) -> None: @@ -324,3 +324,239 @@ async def fake_post(url, headers, json_body, timeout): assert reads["n"] <= 1, f"expected at most one disk read for the run, got {reads['n']}" config_mod.clear_config_cache() + + +# --------------------------------------------------------------------------- # +# synthesizer_chain resolution (DSE-1512) +# --------------------------------------------------------------------------- # + + +def test_council_chain_defaults_to_single_synthesizer(): + c = Council(models=["grok"], config=ConclaveConfig(synthesizer="claude")) + assert c.synthesizer_chain == ["claude"] and c.synthesizer == "claude" + + +def test_council_chain_from_constructor_string(): + c = Council(models=["grok"], synthesizer="claude>grok", config=ConclaveConfig()) + assert c.synthesizer_chain == ["claude", "grok"] and c.synthesizer == "claude" + + +def test_council_chain_from_constructor_list(): + c = Council(models=["grok"], synthesizer=["gemini", "grok"], config=ConclaveConfig()) + assert c.synthesizer_chain == ["gemini", "grok"] and c.synthesizer == "gemini" + + +def test_council_chain_from_config_overrides_scalar(): + cfg = ConclaveConfig(synthesizer="claude", synthesizer_chain=["grok", "gemini"]) + c = Council(models=["claude"], config=cfg) + assert c.synthesizer_chain == ["grok", "gemini"] and c.synthesizer == "grok" + + +def test_council_constructor_arg_beats_config_chain(): + cfg = ConclaveConfig(synthesizer_chain=["grok", "gemini"]) + c = Council(models=["claude"], synthesizer="claude", config=cfg) + assert c.synthesizer_chain == ["claude"] + + +# --------------------------------------------------------------------------- # +# prose synthesis routed through the adjudication succession seam (DSE-1512, task 5) +# --------------------------------------------------------------------------- # + +CFG = ConclaveConfig( + models={ + "claude": "anthropic/c", + "grok": "xai/g", + "gemini": "gemini/m", + "openai": "openai/o", + "mistral": "mistral/m", + } +) + + +async def test_synthesize_mode_fails_over_and_is_not_degraded(monkeypatch, keys): + calls = install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "claude": make_failed_answer("claude", "anthropic/c", "quota", 402), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG, extract_verdict=False) + r = await c.ask("q") + assert r.synthesis == "grok says yes" and r.synthesis_error is None and r.degraded is False + assert (r.synthesizer, r.synthesizer_model_id) == ("grok", "xai/g") + ledger = r.manifest.adjudication_succession + assert [(a.role, a.candidate, a.outcome) for a in ledger] == [ + ("synthesis", "claude", "failed_over"), + ("synthesis", "grok", "success"), + ] + assert [ + (x.phase, x.attempt, x.name) for x in r.manifest.receipts if x.phase == "synthesis" + ] == [ + ("synthesis", 1, "claude"), + ("synthesis", 2, "grok"), + ] + assert r.manifest.secret_safety == "verified_no_secrets" + assert calls == ["gemini", "claude", "grok"] + + +async def test_synthesize_mode_exhausted_is_degraded(monkeypatch, keys): + calls = install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "claude": make_failed_answer("claude", "anthropic/c", "quota", 429), + "grok": make_failed_answer("grok", "xai/g", "unavailable", 503), + }, + ) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG, extract_verdict=False) + r = await c.ask("q") + assert r.synthesis is None + assert r.synthesis_error == "grok failed" + assert r.degraded is True + ledger = r.manifest.adjudication_succession + assert [a.outcome for a in ledger] == ["failed_over", "exhausted"] + assert (r.synthesizer, r.synthesizer_model_id) == ("grok", "xai/g") + assert calls == ["gemini", "claude", "grok"] + + +async def test_synthesize_chain_of_one_no_key_message_unchanged(monkeypatch): + monkeypatch.setenv("GEMINI_API_KEY", "dummy") + calls = install_council_script(monkeypatch, {"gemini": make_ok_answer("gemini", "gemini/m")}) + c = Council(models=["gemini"], synthesizer="claude", config=CFG, extract_verdict=False) + r = await c.ask("q") + assert r.synthesis_error == ( + "synthesizer 'claude' (anthropic/c) has no API key; returning raw answers only" + ) + ledger = r.manifest.adjudication_succession + assert [(a.role, a.candidate, a.outcome) for a in ledger] == [ + ("synthesis", "claude", "skipped_unkeyed") + ] + assert [x for x in r.manifest.receipts if x.phase == "synthesis"] == [] + assert calls == ["gemini"] + + +async def test_synthesize_chain_all_unkeyed_message(monkeypatch): + monkeypatch.setenv("GEMINI_API_KEY", "dummy") + calls = install_council_script(monkeypatch, {"gemini": make_ok_answer("gemini", "gemini/m")}) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG, extract_verdict=False) + r = await c.ask("q") + assert r.synthesis_error == ( + "synthesizer chain [claude, grok] has no API key for any candidate; " + "returning raw answers only" + ) + ledger = r.manifest.adjudication_succession + assert [a.outcome for a in ledger] == ["skipped_unkeyed", "skipped_unkeyed"] + assert r.degraded is True + assert calls == ["gemini"] + + +# --------------------------------------------------------------------------- # +# CouncilResult.primary_failed_over computed field (DSE-1512 review, Unit F) +# --------------------------------------------------------------------------- # + + +async def test_successor_run_is_primary_failed_over_but_not_degraded(monkeypatch, keys): + install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "claude": make_failed_answer("claude", "anthropic/c", "quota", 402), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG, extract_verdict=False) + r = await c.ask("q") + assert r.primary_failed_over is True + assert r.degraded is False + assert "primary_failed_over" in r.model_dump(mode="json") + + +async def test_chain_of_one_clean_run_is_not_primary_failed_over(monkeypatch, keys): + install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "claude": make_ok_answer("claude", "anthropic/c"), + }, + ) + c = Council(models=["gemini"], synthesizer="claude", config=CFG, extract_verdict=False) + r = await c.ask("q") + assert r.primary_failed_over is False + assert "primary_failed_over" in r.model_dump(mode="json") + + +async def test_terminal_failure_primary_is_not_primary_failed_over(monkeypatch, keys): + """A primary that answered unusably (``terminal_failure`` at index 1) is NOT + primary_failed_over -- it adjudicated, just badly, so failover never fires and + the run must stay cacheable exactly like any other content failure. + """ + install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "claude": make_failed_answer("claude", "anthropic/c", "bad_request", 400), + }, + ) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG, extract_verdict=False) + r = await c.ask("q") + assert r.primary_failed_over is False + assert r.degraded is True + assert [a.outcome for a in r.manifest.adjudication_succession] == ["terminal_failure"] + + +async def test_successor_after_unkeyed_primary_is_primary_failed_over(monkeypatch): + """DSE-1512 review, Unit A3: a keyed successor after a SKIPPED (not failed) primary + still counts as primary_failed_over -- no candidate ever errored on a live call. + """ + monkeypatch.setenv("GEMINI_API_KEY", "dummy") + monkeypatch.setenv("XAI_API_KEY", "dummy") + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + c = Council(models=["gemini"], synthesizer="claude>grok", config=CFG, extract_verdict=False) + r = await c.ask("q") + assert r.primary_failed_over is True + assert r.degraded is False + ledger = r.manifest.adjudication_succession + assert [(a.candidate, a.outcome, a.attempt_index) for a in ledger] == [ + ("claude", "skipped_unkeyed", 1), + ("grok", "success", 2), + ] + + +async def test_chain_of_one_unkeyed_is_primary_failed_over(monkeypatch): + """A chain-of-one unkeyed synthesizer IS primary_failed_over (DSE-1512 review, + uniform rule): the primary's attempt_index==1 outcome is "skipped_unkeyed" -- + it never adjudicated, for an infrastructure reason (no key) -- exactly like a + live failover or an exhausted ladder, even though there is no successor to + have adjudicated instead. + """ + monkeypatch.setenv("GEMINI_API_KEY", "dummy") + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + install_council_script(monkeypatch, {"gemini": make_ok_answer("gemini", "gemini/m")}) + c = Council(models=["gemini"], synthesizer="claude", config=CFG, extract_verdict=False) + r = await c.ask("q") + assert r.primary_failed_over is True + assert r.degraded is True + assert [a.outcome for a in r.manifest.adjudication_succession] == ["skipped_unkeyed"] + + +async def test_chain_of_one_unkeyed_is_primary_failed_over_with_verdict_extraction(monkeypatch): + """Same as above with the default ``extract_verdict=True``: the rule must not + depend on which roles ran or whether verdict extraction's separate + unkeyed-candidate handling (see ``Council._apply_verdict``) happened to run. + """ + monkeypatch.setenv("GEMINI_API_KEY", "dummy") + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + install_council_script(monkeypatch, {"gemini": make_ok_answer("gemini", "gemini/m")}) + c = Council(models=["gemini"], synthesizer="claude", config=CFG) + r = await c.ask("q") + assert r.primary_failed_over is True + assert r.degraded is True diff --git a/tests/test_council_verdict.py b/tests/test_council_verdict.py index 1c7bded..677b329 100644 --- a/tests/test_council_verdict.py +++ b/tests/test_council_verdict.py @@ -28,7 +28,7 @@ _REASON_OPEN_ENDED, _REASON_TOO_FEW, ) -from tests.conftest import make_response +from tests.conftest import install_council_script, make_ok_answer, make_response # The synthesizer/extractor resolved id for the "claude" friendly name below. _SYNTH_MODEL_ID = "anthropic/claude-sonnet-4-6" @@ -432,3 +432,421 @@ def handler(model, messages, **kwargs): asyncio.run(council._apply_verdict(result)) assert result.verdict is None assert result.manifest.verdict_extraction.model_id is None + + +# --------------------------------------------------------------------------- # +# 9. Verdict-extraction succession (DSE-1512, Task 6): ``_apply_verdict`` walks +# the synthesizer chain, calling ``extract_verdict`` once per candidate and +# classifying from what it reports. +# --------------------------------------------------------------------------- # + +# A council seam config wide enough for every candidate used below (members +# gemini/openai, chain candidates claude/grok). The verdict-extraction seam is +# patched per-test with a bespoke fake (keyed by ``name``, mirroring the +# ``config_spy``/``fake_call_model`` pattern in ``test_council.py`` / +# ``test_manifest_all_modes.py``) so each test can script a distinct +# infra-failure/success sequence per chain candidate independently of the +# council seam. +_SUCCESSION_CFG = ConclaveConfig( + models={ + "gemini": "gemini/gm", + "openai": "openai/o", + "claude": "anthropic/c", + "grok": "xai/g", + } +) + + +def _verdict_receipts(result: CouncilResult) -> list[tuple[str, str, str]]: + """Return ``(phase, name, outcome)`` for every verdict-related receipt, in order.""" + return [ + (r.phase, r.name, r.outcome) + for r in result.manifest.receipts + if r.phase and r.phase.startswith("verdict") + ] + + +def _verdict_ledger(result: CouncilResult) -> list: + """Return the ``verdict_extraction`` slice of the adjudication succession ledger.""" + return [a for a in result.manifest.adjudication_succession if a.role == "verdict_extraction"] + + +async def test_verdict_extraction_fails_over_to_successor(monkeypatch, keys): + """An infra failure on the primary extractor fails over to the next candidate. + + The repair retry (same-model, per ``extract_verdict``'s own unchanged + behavior) also fails for claude, so claude contributes TWO failed receipts + before the chain advances to grok, which succeeds on its first call. + """ + install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/gm"), + "openai": make_ok_answer("openai", "openai/o"), + "claude": make_ok_answer("claude", "anthropic/c"), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + + async def verdict_seam(name, model_id, messages, **kwargs): + if name == "claude": + return ModelAnswer( + name=name, + model_id=model_id, + error="claude failed", + failure_category="auth", + http_status=401, + ) + return ModelAnswer( + name=name, model_id=model_id, answer=_extraction_json(members=("gemini", "openai")) + ) + + monkeypatch.setattr("conclave.verdict_synthesis.call_model", verdict_seam) + + r = await Council( + models=["gemini", "openai"], synthesizer="claude>grok", config=_SUCCESSION_CFG + ).ask("Should we X?") + + assert r.verdict is not None + assert r.manifest.verdict_extraction.model_id == "xai/g" + ledger = _verdict_ledger(r) + assert [(a.candidate, a.outcome, a.failure_category, a.http_status) for a in ledger] == [ + ("claude", "failed_over", "auth", 401), + ("grok", "success", None, None), + ] + assert _verdict_receipts(r) == [ + ("verdict_extraction", "claude", "failed"), + ("verdict_repair", "claude", "failed"), + ("verdict_extraction", "grok", "success"), + ] + # QA review M2: attempt numbers stay monotonic ACROSS candidates rather + # than each candidate's receipts restarting at 1 (which would collide). + assert [ + (receipt.phase, receipt.attempt, receipt.name) + for receipt in r.manifest.receipts + if receipt.phase and receipt.phase.startswith("verdict") + ] == [ + ("verdict_extraction", 1, "claude"), + ("verdict_repair", 2, "claude"), + ("verdict_extraction", 3, "grok"), + ] + assert r.manifest.secret_safety == SECRET_SAFETY_VERIFIED + + +async def test_verdict_extraction_schema_failure_is_terminal(monkeypatch, keys): + """A candidate that ANSWERS unusably is terminal for the role -- no failover. + + claude returns prose (not JSON) on both the initial call and the repair + retry; grok is never consulted -- confirmed via a call log keyed by name. + """ + install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/gm"), + "openai": make_ok_answer("openai", "openai/o"), + "claude": make_ok_answer("claude", "anthropic/c"), + }, + ) + verdict_calls: list[str] = [] + + async def verdict_seam(name, model_id, messages, **kwargs): + verdict_calls.append(name) + if name == "grok": + raise AssertionError("grok must not be consulted after a terminal failure") + return ModelAnswer(name=name, model_id=model_id, answer="not json") + + monkeypatch.setattr("conclave.verdict_synthesis.call_model", verdict_seam) + + r = await Council( + models=["gemini", "openai"], synthesizer="claude>grok", config=_SUCCESSION_CFG + ).ask("Should we X?") + + assert verdict_calls == ["claude", "claude"] + assert r.verdict is None + assert r.manifest.verdict_absent_reason == _REASON_EXTRACTION_FAILED + ledger = _verdict_ledger(r) + assert [(a.candidate, a.outcome, a.failure_category, a.http_status) for a in ledger] == [ + ("claude", "terminal_failure", "malformed_response", None), + ] + assert _verdict_receipts(r) == [ + ("verdict_extraction", "claude", "schema_invalid"), + ("verdict_repair", "claude", "schema_invalid"), + ] + assert r.manifest.secret_safety == SECRET_SAFETY_VERIFIED + + +async def test_verdict_extraction_chain_of_one_unkeyed_unchanged(monkeypatch): + """Chain of one + unkeyed synthesizer: today's receipt shape is preserved. + + Restores the REAL ``conclave.providers.call_model`` on the verdict seam + (instead of an offline fake) so the "no API key" short-circuit -- which + makes no network call -- runs exactly as it does in production: two + failed receipts (initial + repair), both unkeyed, no schema check ever + reached. The new ledger entry is additive. + """ + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + monkeypatch.setenv("GEMINI_API_KEY", "dummy") + monkeypatch.setenv("OPENAI_API_KEY", "dummy") + install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/gm"), + "openai": make_ok_answer("openai", "openai/o"), + }, + ) + import conclave.providers as providers_mod + + monkeypatch.setattr("conclave.verdict_synthesis.call_model", providers_mod.call_model) + + r = await Council( + models=["gemini", "openai"], synthesizer="claude", config=_SUCCESSION_CFG + ).ask("Should we X?") + + assert r.verdict is None + assert r.manifest.verdict_absent_reason == _REASON_EXTRACTION_FAILED + ledger = _verdict_ledger(r) + assert [(a.candidate, a.outcome, a.failure_category, a.http_status) for a in ledger] == [ + ("claude", "exhausted", "unkeyed", None), + ] + verdict_receipts = _verdict_receipts(r) + assert [phase for phase, _name, _outcome in verdict_receipts] == [ + "verdict_extraction", + "verdict_repair", + ] + assert all(outcome == "failed" for _phase, _name, outcome in verdict_receipts) + + +async def test_verdict_extraction_exhausted(monkeypatch, keys): + """Every chain candidate fails an infra error -> the chain is exhausted.""" + install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/gm"), + "openai": make_ok_answer("openai", "openai/o"), + "claude": make_ok_answer("claude", "anthropic/c"), + }, + ) + + async def verdict_seam(name, model_id, messages, **kwargs): + if name == "claude": + return ModelAnswer( + name=name, + model_id=model_id, + error="claude quota exceeded", + failure_category="quota", + http_status=429, + ) + return ModelAnswer( + name=name, + model_id=model_id, + error="grok unavailable", + failure_category="unavailable", + http_status=503, + ) + + monkeypatch.setattr("conclave.verdict_synthesis.call_model", verdict_seam) + + r = await Council( + models=["gemini", "openai"], synthesizer="claude>grok", config=_SUCCESSION_CFG + ).ask("Should we X?") + + assert r.verdict is None + assert r.manifest.verdict_absent_reason == _REASON_EXTRACTION_FAILED + ledger = _verdict_ledger(r) + assert [a.outcome for a in ledger] == ["failed_over", "exhausted"] + assert [(a.candidate, a.failure_category, a.http_status) for a in ledger] == [ + ("claude", "quota", 429), + ("grok", "unavailable", 503), + ] + assert r.manifest.verdict_extraction.model_id == "xai/g" + assert _verdict_receipts(r) == [ + ("verdict_extraction", "claude", "failed"), + ("verdict_repair", "claude", "failed"), + ("verdict_extraction", "grok", "failed"), + ("verdict_repair", "grok", "failed"), + ] + assert r.manifest.secret_safety == SECRET_SAFETY_VERIFIED + + +async def test_verdict_extraction_answered_then_repair_infra_is_terminal(monkeypatch, keys): + """A candidate that answered on EITHER attempt is terminal, even if the other errored. + + claude's initial call answers with prose (200, unparsable as JSON); its + same-model repair retry then hits an unrelated infrastructure error + (429). Because claude answered at least once, the category must be the + fixed terminal ``"malformed_response"`` -- NOT the repair's infra + category -- so grok is never consulted (QA review A1). + """ + install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/gm"), + "openai": make_ok_answer("openai", "openai/o"), + "claude": make_ok_answer("claude", "anthropic/c"), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + verdict_calls: list[str] = [] + + async def verdict_seam(name, model_id, messages, **kwargs): + verdict_calls.append(name) + if name == "grok": + raise AssertionError("grok must not be consulted after a terminal failure") + if verdict_calls.count("claude") == 1: + # Initial call: answered, but the prose isn't parseable JSON. + return ModelAnswer(name=name, model_id=model_id, answer="not json") + # Repair retry: an unrelated infrastructure failure. + return ModelAnswer( + name=name, + model_id=model_id, + error="claude quota exceeded", + failure_category="quota", + http_status=429, + ) + + monkeypatch.setattr("conclave.verdict_synthesis.call_model", verdict_seam) + + r = await Council( + models=["gemini", "openai"], synthesizer="claude>grok", config=_SUCCESSION_CFG + ).ask("Should we X?") + + assert verdict_calls == ["claude", "claude"] + assert r.verdict is None + assert r.manifest.verdict_absent_reason == _REASON_EXTRACTION_FAILED + ledger = _verdict_ledger(r) + assert [(a.candidate, a.outcome, a.failure_category, a.http_status) for a in ledger] == [ + ("claude", "terminal_failure", "malformed_response", None), + ] + assert r.manifest.secret_safety == SECRET_SAFETY_VERIFIED + + +async def test_verdict_extraction_infra_then_answered_is_terminal(monkeypatch, keys): + """The mirror of the above: infra failure first, then an answer on repair. + + claude's initial call fails infra-side (503); its repair retry then + answers unusably. Because claude answered on the repair, the category is + still the fixed terminal ``"malformed_response"`` regardless of the + initial's infra category, and grok is never consulted. + """ + install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/gm"), + "openai": make_ok_answer("openai", "openai/o"), + "claude": make_ok_answer("claude", "anthropic/c"), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + verdict_calls: list[str] = [] + + async def verdict_seam(name, model_id, messages, **kwargs): + verdict_calls.append(name) + if name == "grok": + raise AssertionError("grok must not be consulted after a terminal failure") + if verdict_calls.count("claude") == 1: + return ModelAnswer( + name=name, + model_id=model_id, + error="claude unavailable", + failure_category="unavailable", + http_status=503, + ) + return ModelAnswer(name=name, model_id=model_id, answer="not json") + + monkeypatch.setattr("conclave.verdict_synthesis.call_model", verdict_seam) + + r = await Council( + models=["gemini", "openai"], synthesizer="claude>grok", config=_SUCCESSION_CFG + ).ask("Should we X?") + + assert verdict_calls == ["claude", "claude"] + assert r.verdict is None + assert r.manifest.verdict_absent_reason == _REASON_EXTRACTION_FAILED + ledger = _verdict_ledger(r) + assert [(a.candidate, a.outcome, a.failure_category, a.http_status) for a in ledger] == [ + ("claude", "terminal_failure", "malformed_response", None), + ] + assert r.manifest.secret_safety == SECRET_SAFETY_VERIFIED + + +async def test_verdict_extraction_both_attempts_infra_fails_over(monkeypatch, keys): + """Both attempts errored -> the LAST errored attempt's category wins and the + chain advances (the ``extract_verdict_fails_over_to_successor`` case, made + explicit for both attempts erroring with DIFFERENT categories). + """ + install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/gm"), + "openai": make_ok_answer("openai", "openai/o"), + "claude": make_ok_answer("claude", "anthropic/c"), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + verdict_calls: list[str] = [] + + async def verdict_seam(name, model_id, messages, **kwargs): + verdict_calls.append(name) + if name != "claude": + return ModelAnswer( + name=name, model_id=model_id, answer=_extraction_json(members=("gemini", "openai")) + ) + if verdict_calls.count("claude") == 1: + return ModelAnswer( + name=name, + model_id=model_id, + error="claude unavailable", + failure_category="unavailable", + http_status=503, + ) + return ModelAnswer( + name=name, + model_id=model_id, + error="claude quota exceeded", + failure_category="quota", + http_status=429, + ) + + monkeypatch.setattr("conclave.verdict_synthesis.call_model", verdict_seam) + + r = await Council( + models=["gemini", "openai"], synthesizer="claude>grok", config=_SUCCESSION_CFG + ).ask("Should we X?") + + assert r.verdict is not None + ledger = _verdict_ledger(r) + assert [(a.candidate, a.outcome, a.failure_category, a.http_status) for a in ledger] == [ + ("claude", "failed_over", "quota", 429), + ("grok", "success", None, None), + ] + + +async def test_verdict_extraction_n_lt_2_records_no_ledger(monkeypatch, keys): + """N<2 responders -> no extraction call at all, no verdict_extraction ledger entries.""" + install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/gm"), + "openai": ModelAnswer( + name="openai", model_id="openai/o", error="openai failed", failure_category="auth" + ), + "claude": make_ok_answer("claude", "anthropic/c"), + }, + ) + + verdict_calls: list[str] = [] + + async def verdict_seam(name, model_id, messages, **kwargs): + verdict_calls.append(name) + raise AssertionError("verdict extraction must not run with <2 responders") + + monkeypatch.setattr("conclave.verdict_synthesis.call_model", verdict_seam) + + r = await Council( + models=["gemini", "openai"], synthesizer="claude>grok", config=_SUCCESSION_CFG + ).ask("Should we X?") + + assert verdict_calls == [] + assert r.verdict is None + assert r.manifest.verdict_absent_reason == _REASON_TOO_FEW + assert _verdict_ledger(r) == [] diff --git a/tests/test_elite_mode.py b/tests/test_elite_mode.py index 39184cf..1e87b90 100644 --- a/tests/test_elite_mode.py +++ b/tests/test_elite_mode.py @@ -22,7 +22,7 @@ elite_critic_user, elite_revision_user, ) -from tests.conftest import make_response +from tests.conftest import install_council_script, make_failed_answer, make_ok_answer, make_response def _all_keys(monkeypatch) -> None: @@ -602,3 +602,61 @@ def handler(model_id, messages): assert result.elite.completed is True assert result.elite.decision_readiness == "indeterminate" assert result.elite.readiness_reasons == ["adjudication.open_ended"] + + +# --------------------------------------------------------------------------- # +# elite synthesis routed through the adjudication succession seam (DSE-1512, task 5) +# --------------------------------------------------------------------------- # + +CFG = ConclaveConfig( + models={ + "claude": "anthropic/c", + "grok": "xai/g", + "gemini": "gemini/m", + "openai": "openai/o", + "mistral": "mistral/m", + } +) + + +async def test_elite_synthesis_fails_over(monkeypatch, keys): + """A failed primary synthesizer fails over to the next chain candidate. + + Three members (gemini, claude, grok) answer identically across the + initial/critique/revision phases -- ``install_council_script`` returns the + same canned answer for every call to a given name, which is fine here + since the elite gate only requires >= 3 successful responders per phase, + not distinct text. The synthesizer chain "openai>mistral" fails over from + a quota-limited openai to a healthy mistral. + """ + calls = install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "claude": make_ok_answer("claude", "anthropic/c"), + "grok": make_ok_answer("grok", "xai/g"), + "openai": make_failed_answer("openai", "openai/o", "quota", 429), + "mistral": make_ok_answer("mistral", "mistral/m"), + }, + ) + c = Council( + models=["gemini", "claude", "grok"], + synthesizer="openai>mistral", + config=CFG, + extract_verdict=False, + ) + r = await c.elite("q") + + assert r.elite.completed is True + assert r.synthesis == "mistral says yes" + ledger = r.manifest.adjudication_succession + assert [(a.role, a.candidate, a.outcome) for a in ledger] == [ + ("synthesis", "openai", "failed_over"), + ("synthesis", "mistral", "success"), + ] + synth_receipts = [x for x in r.manifest.receipts if x.phase == "synthesis"] + assert [(x.attempt, x.name) for x in synth_receipts] == [(1, "openai"), (2, "mistral")] + assert all(x.protocol_version == ELITE_PROTOCOL_VERSION for x in synth_receipts) + assert r.elite.decision_readiness == "indeterminate" + assert r.elite.readiness_reasons == ["adjudication.disabled"] + assert "openai" in calls and "mistral" in calls diff --git a/tests/test_failure_category.py b/tests/test_failure_category.py new file mode 100644 index 0000000..569c627 --- /dev/null +++ b/tests/test_failure_category.py @@ -0,0 +1,132 @@ +"""Typed failure categories are derived from status codes / exception types (DSE-1512).""" + +from __future__ import annotations + +import httpx +import pytest + +from conclave import transport +from conclave.adapters import ProviderError, resolve_adapter +from conclave.adapters.anthropic import AnthropicAdapter +from conclave.adapters.gemini import GeminiAdapter +from conclave.adapters.openai_compat import OpenAICompatAdapter +from conclave.config import ConclaveConfig +from conclave.models import FAILOVER_CATEGORIES, categorize_http_status + + +def _openai_adapter() -> OpenAICompatAdapter: + return OpenAICompatAdapter( + prefix="openai", + completions_url="https://api.openai.com/v1/chat/completions", + env_vars=("OPENAI_API_KEY",), + ) + + +@pytest.mark.parametrize( + ("status", "expected"), + [ + (401, "auth"), + (403, "auth"), + (402, "quota"), + (429, "quota"), + (408, "timeout"), + (500, "unavailable"), + (502, "unavailable"), + (503, "unavailable"), + (529, "unavailable"), + (400, "bad_request"), + (404, "bad_request"), + (422, "bad_request"), + (302, "unexpected"), + (101, "unexpected"), + ], +) +def test_categorize_http_status(status, expected): + assert categorize_http_status(status) == expected + + +def test_failover_set_is_infrastructure_only(): + assert FAILOVER_CATEGORIES == frozenset( + {"unkeyed", "unresolved", "auth", "quota", "unavailable", "timeout", "transport"} + ) + assert "bad_request" not in FAILOVER_CATEGORIES + assert "malformed_response" not in FAILOVER_CATEGORIES + assert "unexpected" not in FAILOVER_CATEGORIES + + +def test_provider_error_defaults_to_malformed_response(): + err = ProviderError("x: empty response") + assert err.category == "malformed_response" + assert err.http_status is None + + +def test_provider_error_carries_status_category(): + err = ProviderError("x: HTTP 429: slow down", category="quota", http_status=429) + assert err.category == "quota" + assert err.http_status == 429 + # message is still redacted on construction (existing contract) + assert "sk-" not in str(ProviderError("leak sk-abc123def456ghi789", category="auth")) + + +def test_transport_error_category(): + assert ( + transport.TransportError("request timed out after 5s", category="timeout").category + == "timeout" + ) + assert transport.TransportError("network error: ConnectError").category == "transport" + + +@pytest.mark.parametrize("adapter", [_openai_adapter(), AnthropicAdapter(), GeminiAdapter()]) +def test_adapters_type_non_2xx(adapter): + with pytest.raises(ProviderError) as info: + adapter.parse_response(401, {"error": {"message": "bad key"}}) + assert info.value.category == "auth" + assert info.value.http_status == 401 + with pytest.raises(ProviderError) as info: + adapter.parse_response(503, {"error": {"message": "down"}}) + assert info.value.category == "unavailable" + + +def test_adapter_malformed_is_not_failover(): + with pytest.raises(ProviderError) as info: + _openai_adapter().parse_response(200, {"choices": []}) + assert info.value.category == "malformed_response" + assert info.value.category not in FAILOVER_CATEGORIES + + +def test_unresolved_provider_is_typed(): + with pytest.raises(ProviderError) as info: + resolve_adapter("nope/model", ConclaveConfig()) + assert info.value.category == "unresolved" + + +async def test_post_json_timeout_is_typed(monkeypatch): + class _Client: + is_closed = False + + async def post(self, *a, **k): + raise httpx.ReadTimeout("slow") + + monkeypatch.setattr(transport, "_client", _Client()) + with pytest.raises(transport.TransportError) as info: + await transport.post_json("https://x", {}, {}, 1.0) + assert info.value.category == "timeout" + + +async def test_post_json_network_is_typed(monkeypatch): + class _Client: + is_closed = False + + async def post(self, *a, **k): + raise httpx.ConnectError("refused") + + monkeypatch.setattr(transport, "_client", _Client()) + with pytest.raises(transport.TransportError) as info: + await transport.post_json("https://x", {}, {}, 1.0) + assert info.value.category == "transport" + + +def test_transport_error_carries_http_status(): + err = transport.TransportError("HTTP 503: x", category="unavailable", http_status=503) + assert err.http_status == 503 + assert transport.TransportError("network error: ConnectError").http_status is None diff --git a/tests/test_modes.py b/tests/test_modes.py index 2305bb5..9df2a4c 100644 --- a/tests/test_modes.py +++ b/tests/test_modes.py @@ -14,7 +14,7 @@ from conclave import AdversarialResult, Council, DebateRound from conclave.config import ConclaveConfig -from tests.conftest import make_response +from tests.conftest import install_council_script, make_failed_answer, make_ok_answer, make_response def _all_keys(monkeypatch) -> None: @@ -798,3 +798,193 @@ async def _inner(): council.adversarial_sync("hi") asyncio.run(_inner()) + + +# --------------------------------------------------------------------------- # +# debate final consolidation + adversarial judge routed through the +# adjudication succession seam (DSE-1512, task 5) +# --------------------------------------------------------------------------- # + +CFG = ConclaveConfig( + models={ + "claude": "anthropic/c", + "grok": "xai/g", + "gemini": "gemini/m", + "openai": "openai/o", + "mistral": "mistral/m", + } +) + + +async def test_debate_final_fails_over(monkeypatch, keys): + calls = install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "openai": make_ok_answer("openai", "openai/o"), + "claude": make_failed_answer("claude", "anthropic/c", "auth", 401), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + c = Council( + models=["gemini", "openai"], synthesizer="claude>grok", config=CFG, extract_verdict=False + ) + r = await c.debate("q", rounds=1) + assert r.synthesis == "grok says yes" + assert (r.synthesizer, r.synthesizer_model_id) == ("grok", "xai/g") + assert r.degraded is False + ledger = r.manifest.adjudication_succession + assert [(a.role, a.candidate, a.outcome) for a in ledger] == [ + ("debate_final", "claude", "failed_over"), + ("debate_final", "grok", "success"), + ] + assert [ + (x.phase, x.attempt, x.name) for x in r.manifest.receipts if x.phase == "debate_final" + ] == [ + ("debate_final", 1, "claude"), + ("debate_final", 2, "grok"), + ] + assert calls == ["gemini", "openai", "claude", "grok"] + + +async def test_adversarial_judge_fails_over(monkeypatch, keys): + calls = install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "openai": make_ok_answer("openai", "openai/o"), + "claude": make_failed_answer("claude", "anthropic/c", "unavailable", 503), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + c = Council(models=["gemini", "openai"], synthesizer="claude>grok", config=CFG) + r = await c.adversarial("q") + adv = r.adversarial + assert adv.verdict == "grok says yes" + assert (adv.judge, adv.judge_model_id) == ("grok", "xai/g") + assert r.synthesis == "grok says yes" + assert r.degraded is False + ledger = r.manifest.adjudication_succession + assert [(a.role, a.candidate, a.outcome) for a in ledger] == [ + ("judge", "claude", "failed_over"), + ("judge", "grok", "success"), + ] + assert [(x.phase, x.attempt, x.name) for x in r.manifest.receipts if x.phase == "judge"] == [ + ("judge", 1, "claude"), + ("judge", 2, "grok"), + ] + assert calls[0] == "gemini" # default proposer is the first requested member + assert "grok" in calls + + +async def test_adversarial_judge_terminal_does_not_fail_over(monkeypatch, keys): + calls = install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "openai": make_ok_answer("openai", "openai/o"), + "claude": make_failed_answer("claude", "anthropic/c", "bad_request", 400), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + c = Council(models=["gemini", "openai"], synthesizer="claude>grok", config=CFG) + r = await c.adversarial("q") + adv = r.adversarial + assert adv.verdict is None + assert adv.verdict_error == "claude failed" + assert r.degraded is True + assert "grok" not in calls + ledger = r.manifest.adjudication_succession + assert [(a.role, a.candidate, a.outcome) for a in ledger] == [ + ("judge", "claude", "terminal_failure") + ] + + +async def test_debate_chain_of_one_no_key_message_unchanged(monkeypatch): + monkeypatch.setenv("GEMINI_API_KEY", "dummy") + install_council_script(monkeypatch, {"gemini": make_ok_answer("gemini", "gemini/m")}) + c = Council(models=["gemini"], synthesizer="claude", config=CFG, extract_verdict=False) + r = await c.debate("q", rounds=1) + assert r.synthesis_error == ( + "synthesizer 'claude' (anthropic/c) has no API key; returning final-round answers only" + ) + + +async def test_adversarial_judge_chain_of_one_no_key_message_unchanged(monkeypatch): + monkeypatch.setenv("GEMINI_API_KEY", "dummy") + install_council_script(monkeypatch, {"gemini": make_ok_answer("gemini", "gemini/m")}) + c = Council(models=["gemini"], synthesizer="claude", config=CFG, extract_verdict=False) + r = await c.adversarial("q") + adv = r.adversarial + assert adv.verdict_error == ( + "judge 'claude' (anthropic/c) has no API key; returning proposal and critiques only" + ) + + +# --------------------------------------------------------------------------- # +# primary_failed_over via a successor after a skipped-unkeyed primary +# (DSE-1512 review, Unit A3) -- and proof the second run is not served from +# cache, mirroring tests/test_council.py's synthesize-mode counterpart. +# --------------------------------------------------------------------------- # + + +async def test_debate_successor_after_unkeyed_primary_is_primary_failed_over(monkeypatch, tmp_path): + """A debate whose final consolidator's primary was skipped for a missing key, + with a keyed successor, counts as primary_failed_over -- and is not cached. + """ + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path)) + monkeypatch.setenv("GEMINI_API_KEY", "dummy") + monkeypatch.setenv("XAI_API_KEY", "dummy") + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + calls = install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + c = Council( + models=["gemini"], + synthesizer="claude>grok", + config=CFG, + extract_verdict=False, + cache=True, + ) + r1 = await c.debate("q", rounds=1) + assert r1.primary_failed_over is True + assert r1.cached is False + + r2 = await c.debate("q", rounds=1) + assert r2.cached is False # not served from cache -- ran again + assert calls.count("gemini") == 2 + + +async def test_adversarial_successor_after_unkeyed_primary_is_primary_failed_over( + monkeypatch, tmp_path +): + """Same for the adversarial judge: a successor after a skipped-unkeyed primary.""" + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path)) + monkeypatch.setenv("GEMINI_API_KEY", "dummy") + monkeypatch.setenv("XAI_API_KEY", "dummy") + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + calls = install_council_script( + monkeypatch, + { + "gemini": make_ok_answer("gemini", "gemini/m"), + "grok": make_ok_answer("grok", "xai/g"), + }, + ) + c = Council( + models=["gemini"], + synthesizer="claude>grok", + config=CFG, + extract_verdict=False, + cache=True, + ) + r1 = await c.adversarial("q") + assert r1.primary_failed_over is True + assert r1.cached is False + + r2 = await c.adversarial("q") + assert r2.cached is False # not served from cache -- ran again + assert calls.count("gemini") == 2 diff --git a/tests/test_providers.py b/tests/test_providers.py index a85fa67..6021bff 100644 --- a/tests/test_providers.py +++ b/tests/test_providers.py @@ -20,6 +20,7 @@ import conclave.config as config_mod import conclave.providers as providers_mod +from conclave import transport from conclave.adapters import ProviderError, resolve_adapter from conclave.adapters.anthropic import AnthropicAdapter from conclave.adapters.base import redact @@ -519,3 +520,73 @@ def test_config_cache_invalidates_on_file_change(tmp_path): second = load_config(path=config_file) assert second.synthesizer == "gemini" + + +# --------------------------------------------------------------------------- # +# call_model carries a typed failure_category / http_status (DSE-1512) +# --------------------------------------------------------------------------- # + + +async def test_call_model_types_unkeyed(monkeypatch): + """No key in env -> failure_category is the typed 'unkeyed', no HTTP status.""" + monkeypatch.delenv("XAI_API_KEY", raising=False) + monkeypatch.setenv("CONCLAVE_CONFIG", "/nonexistent/conclave.yml") + + ans = await call_model("grok", "xai/grok-4.3", [{"role": "user", "content": "hi"}]) + assert ans.error and ans.failure_category == "unkeyed" and ans.http_status is None + + +async def test_call_model_types_http_status(monkeypatch): + """A non-2xx provider response carries its status-derived category + status.""" + monkeypatch.setenv("ANTHROPIC_API_KEY", "dummy") + monkeypatch.setenv("CONCLAVE_CONFIG", "/nonexistent/conclave.yml") + + async def fake_post(url, headers, body, timeout): + return 402, {"error": {"message": "insufficient credit"}} + + monkeypatch.setattr(transport, "post_json", fake_post) + + ans = await call_model( + "claude", "anthropic/claude-sonnet-4-6", [{"role": "user", "content": "hi"}] + ) + assert ans.error and ans.failure_category == "quota" and ans.http_status == 402 + + +async def test_call_model_types_timeout(monkeypatch): + """A typed transport timeout propagates its category onto the ModelAnswer.""" + monkeypatch.setenv("ANTHROPIC_API_KEY", "dummy") + monkeypatch.setenv("CONCLAVE_CONFIG", "/nonexistent/conclave.yml") + + async def fake_post(url, headers, body, timeout): + raise transport.TransportError("request timed out after 1s", category="timeout") + + monkeypatch.setattr(transport, "post_json", fake_post) + + ans = await call_model( + "claude", "anthropic/claude-sonnet-4-6", [{"role": "user", "content": "hi"}] + ) + assert ans.failure_category == "timeout" + + +async def test_call_model_error_text_unchanged(monkeypatch): + """Additive-only guarantee: the error STRING is byte-identical to before.""" + monkeypatch.setenv("ANTHROPIC_API_KEY", "dummy") + monkeypatch.setenv("CONCLAVE_CONFIG", "/nonexistent/conclave.yml") + + async def fake_post(url, headers, body, timeout): + return 401, {"error": {"message": "bad key"}} + + monkeypatch.setattr(transport, "post_json", fake_post) + + ans = await call_model( + "claude", "anthropic/claude-sonnet-4-6", [{"role": "user", "content": "hi"}] + ) + assert ans.error == "anthropic: HTTP 401: bad key" + + +async def test_call_model_unresolved_is_typed(): + """An unresolved provider prefix carries the typed 'unresolved' category.""" + ans = await call_model( + "x", "nope/model", [{"role": "user", "content": "hi"}], config=ConclaveConfig() + ) + assert ans.failure_category == "unresolved" diff --git a/tests/test_registry_config.py b/tests/test_registry_config.py index d23622e..48c437d 100644 --- a/tests/test_registry_config.py +++ b/tests/test_registry_config.py @@ -4,7 +4,7 @@ import pytest -from conclave.config import load_config +from conclave.config import _load_config_uncached, load_config, parse_synthesizer_chain from conclave.registry import ( DEFAULT_MODELS, NATIVE_PROVIDERS, @@ -201,3 +201,34 @@ def test_consistency_check_detects_url_mismatch(monkeypatch): with pytest.raises(RegistryError, match="URL drift"): _assert_metadata_consistent() + + +# --------------------------------------------------------------------------- # +# synthesizer_chain ordered failover ladder (DSE-1512) +# --------------------------------------------------------------------------- # + + +def test_parse_synthesizer_chain_splits_and_dedupes(): + assert parse_synthesizer_chain("claude>grok > gemini>claude") == ["claude", "grok", "gemini"] + assert parse_synthesizer_chain("claude") == ["claude"] + assert parse_synthesizer_chain(" ") == [] + + +def test_config_synthesizer_chain_from_yaml(tmp_path): + p = tmp_path / "c.yml" + p.write_text("synthesizer: claude\nsynthesizer_chain: [claude, grok]\n") + cfg = _load_config_uncached(p) + assert cfg.synthesizer == "claude" + assert cfg.synthesizer_chain == ["claude", "grok"] + + +def test_config_synthesizer_chain_accepts_arrow_string(tmp_path): + p = tmp_path / "c.yml" + p.write_text("synthesizer_chain: 'claude>grok'\n") + assert _load_config_uncached(p).synthesizer_chain == ["claude", "grok"] + + +def test_config_synthesizer_chain_bad_value_is_empty(tmp_path): + p = tmp_path / "c.yml" + p.write_text("synthesizer_chain: 42\n") + assert _load_config_uncached(p).synthesizer_chain == [] diff --git a/tests/test_streaming.py b/tests/test_streaming.py index 89e5c23..035ad42 100644 --- a/tests/test_streaming.py +++ b/tests/test_streaming.py @@ -27,6 +27,7 @@ from conclave.config import ConclaveConfig from conclave.models import ModelAnswer, StreamEvent from conclave.providers import call_model, call_model_stream +from tests.conftest import make_failed_answer, make_ok_answer runner = CliRunner() @@ -575,3 +576,229 @@ def test_stream_event_done_carries_full_result_shape(): dumped = ev.model_dump(mode="json") assert dumped["type"] == "done" assert dumped["result"]["prompt"] == "p" + + +# --------------------------------------------------------------------------- # +# Streaming synthesis succession (DSE-1512, Task 7) +# --------------------------------------------------------------------------- # + +SYNTH_CFG = ConclaveConfig(models={"claude": "anthropic/c", "grok": "xai/g", "gemini": "gemini/m"}) + + +def _install_stream_script(monkeypatch, script: dict[str, list]) -> list[str]: + """Patch the streaming ``call_model_stream`` seam with a per-name script. + + ``script[name]`` is the exact item list ``call_model_stream`` would yield + for that friendly name: zero or more ``str`` deltas, then exactly one + trailing :class:`~conclave.models.ModelAnswer` -- mirroring the real yield + contract. Council members stream through this SAME seam as the synthesizer + chain, so every member name needs a script entry too. + + Returns: + The call log: one friendly name appended per invocation, in call order. + """ + import conclave.streaming as streaming_mod + + calls: list[str] = [] + + async def fake_stream(name, model_id, messages, *, temperature=0.7, timeout=120.0, config=None): + calls.append(name) + for item in script[name]: + yield item + + monkeypatch.setattr(streaming_mod, "call_model_stream", fake_stream) + return calls + + +def _ledger(result): + """Flatten a result's adjudication ledger to comparable tuples.""" + return [ + (a.role, a.candidate, a.outcome, a.failure_category, a.http_status) + for a in result.manifest.adjudication_succession + ] + + +async def test_stream_synthesis_fails_over_before_first_delta(monkeypatch, keys): + """A synthesizer candidate that fails with no deltas emitted fails over cleanly.""" + calls = _install_stream_script( + monkeypatch, + { + "gemini": ["gem", "ini ok", make_ok_answer("gemini", "gemini/m")], + "claude": [make_failed_answer("claude", "anthropic/c", "quota", 429)], + "grok": ["grok ", "says yes", make_ok_answer("grok", "xai/g")], + }, + ) + council = Council( + models=["gemini"], synthesizer="claude>grok", config=SYNTH_CFG, extract_verdict=False + ) + events = [e async for e in council.ask_stream("q")] + + deltas = [e.text for e in events if e.type == "synthesis_delta"] + done = [e for e in events if e.type == "synthesis_done"] + assert "".join(deltas) == "grok says yes" + assert len(done) == 1 + assert done[0].name == "grok" and done[0].model_id == "xai/g" and done[0].answer.ok + + result = events[-1].result + assert result.synthesis == "grok says yes" and result.synthesis_error is None + assert result.degraded is False + assert (result.synthesizer, result.synthesizer_model_id) == ("grok", "xai/g") + assert _ledger(result) == [ + ("synthesis", "claude", "failed_over", "quota", 429), + ("synthesis", "grok", "success", None, None), + ] + # Streaming receipt contract unchanged: no synthesis-phase receipts. + assert not [r for r in result.manifest.receipts if r.phase == "synthesis"] + assert result.manifest.secret_safety == "verified_no_secrets" + assert calls == ["gemini", "claude", "grok"] + + +async def test_stream_synthesis_does_not_fail_over_after_deltas(monkeypatch, keys): + """A post-first-delta failure is terminal even though its category is infra-shaped.""" + calls = _install_stream_script( + monkeypatch, + { + "gemini": [make_ok_answer("gemini", "gemini/m")], + "claude": ["partial", make_failed_answer("claude", "anthropic/c", "unavailable", 503)], + "grok": [make_ok_answer("grok", "xai/g")], + }, + ) + council = Council( + models=["gemini"], synthesizer="claude>grok", config=SYNTH_CFG, extract_verdict=False + ) + events = [e async for e in council.ask_stream("q")] + + deltas = [e for e in events if e.type == "synthesis_delta"] + done = [e for e in events if e.type == "synthesis_done"] + assert len(deltas) == 1 and deltas[0].text == "partial" + assert len(done) == 1 + assert done[0].name == "claude" and not done[0].answer.ok + + result = events[-1].result + assert result.synthesis is None + assert result.synthesis_error == "claude failed" + assert result.degraded is True + assert _ledger(result) == [("synthesis", "claude", "terminal_failure", "unavailable", 503)] + assert calls == ["gemini", "claude"] # grok never invoked + + +async def test_stream_synthesis_terminal_category_does_not_fail_over(monkeypatch, keys): + """A terminal (non-failover) category never advances the chain, deltas or not.""" + calls = _install_stream_script( + monkeypatch, + { + "gemini": [make_ok_answer("gemini", "gemini/m")], + "claude": [make_failed_answer("claude", "anthropic/c", "bad_request", 400)], + "grok": [make_ok_answer("grok", "xai/g")], + }, + ) + council = Council( + models=["gemini"], synthesizer="claude>grok", config=SYNTH_CFG, extract_verdict=False + ) + events = [e async for e in council.ask_stream("q")] + + result = events[-1].result + assert _ledger(result) == [("synthesis", "claude", "terminal_failure", "bad_request", 400)] + assert result.degraded is True + assert calls == ["gemini", "claude"] # grok never invoked + + +async def test_stream_synthesis_chain_exhausted(monkeypatch, keys): + """Every keyed candidate fails infra-side -> chain exhausted, no deltas at all.""" + calls = _install_stream_script( + monkeypatch, + { + "gemini": [make_ok_answer("gemini", "gemini/m")], + "claude": [make_failed_answer("claude", "anthropic/c", "quota", 429)], + "grok": [make_failed_answer("grok", "xai/g", "unavailable", 503)], + }, + ) + council = Council( + models=["gemini"], synthesizer="claude>grok", config=SYNTH_CFG, extract_verdict=False + ) + events = [e async for e in council.ask_stream("q")] + + assert not [e for e in events if e.type == "synthesis_delta"] + done = [e for e in events if e.type == "synthesis_done"] + assert len(done) == 1 + assert done[0].name == "grok" and not done[0].answer.ok + + result = events[-1].result + assert result.synthesis_error == "grok failed" + assert result.degraded is True + assert [a.outcome for a in result.manifest.adjudication_succession] == [ + "failed_over", + "exhausted", + ] + assert (result.synthesizer, result.synthesizer_model_id) == ("grok", "xai/g") + assert calls == ["gemini", "claude", "grok"] + + +async def test_stream_synthesis_skips_unkeyed_candidate(monkeypatch): + """An unkeyed chain candidate is skipped without ever opening a stream for it.""" + monkeypatch.setenv("GEMINI_API_KEY", "dummy") + monkeypatch.setenv("XAI_API_KEY", "dummy") + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + calls = _install_stream_script( + monkeypatch, + { + "gemini": [make_ok_answer("gemini", "gemini/m")], + "grok": [make_ok_answer("grok", "xai/g")], + }, + ) + council = Council( + models=["gemini"], synthesizer="claude>grok", config=SYNTH_CFG, extract_verdict=False + ) + events = [e async for e in council.ask_stream("q")] + + result = events[-1].result + assert _ledger(result) == [ + ("synthesis", "claude", "skipped_unkeyed", "unkeyed", None), + ("synthesis", "grok", "success", None, None), + ] + assert calls == ["gemini", "grok"] # claude never called + + +async def test_stream_synthesis_chain_of_one_no_key_message_unchanged(monkeypatch): + """A chain-of-one unkeyed synthesizer keeps today's wording and event shape.""" + monkeypatch.setenv("GEMINI_API_KEY", "dummy") + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + calls = _install_stream_script(monkeypatch, {"gemini": [make_ok_answer("gemini", "gemini/m")]}) + council = Council( + models=["gemini"], synthesizer="claude", config=SYNTH_CFG, extract_verdict=False + ) + events = [e async for e in council.ask_stream("q")] + + types = [e.type for e in events] + assert "synthesis_delta" not in types and "synthesis_done" not in types + + result = events[-1].result + assert ( + result.synthesis_error + == "synthesizer 'claude' (anthropic/c) has no API key; returning raw answers only" + ) + assert _ledger(result) == [("synthesis", "claude", "skipped_unkeyed", "unkeyed", None)] + assert calls == ["gemini"] + + +async def test_stream_synthesis_chain_of_one_success_records_ledger(monkeypatch, keys): + """A chain-of-one success streams exactly like today and records one ledger entry.""" + calls = _install_stream_script( + monkeypatch, + { + "gemini": [make_ok_answer("gemini", "gemini/m")], + "claude": ["claude ", "says yes", make_ok_answer("claude", "anthropic/c")], + }, + ) + council = Council( + models=["gemini"], synthesizer="claude", config=SYNTH_CFG, extract_verdict=False + ) + events = [e async for e in council.ask_stream("q")] + + types = [e.type for e in events] + assert types == ["member_done", "synthesis_delta", "synthesis_delta", "synthesis_done", "done"] + + result = events[-1].result + assert result.synthesis == "claude says yes" + assert _ledger(result) == [("synthesis", "claude", "success", None, None)] + assert calls == ["gemini", "claude"]