Hand off exhausted tool runs to an evidence-only final answer

[Maya-Kid]

Post-merge synchronization (2026-09-22)

Rebased onto dev@7cfeedfeb11c3a1c42d5091c5941a51afaaeb161 after #848, #851, #849, #850 and #852 merged. Current head: b4646386be142d2e502c6e88eaf01e9817f497ec.

Retained the four merged regression modules in rustfmt order: fallback, persistence, registry, sources. Both terminal and stream test modules remain. The evidence-only handoff runtime and the #858 fixture adjustments from the previous head are unchanged. The unified PostgreSQL Chat CI step and ADR placement remain as agreed.

Validation on combined tree d06f4ad0199a382635a6bbf94210b9421cbeba66 (675 tracked Linux inputs hash-matched):

• Rust fmt, strict workspace/all-target Clippy and workspace build passed. Workspace tests: 1018 passed, 0 failed, 5 ignored. The two delivery/process-kill parent tests and busy-deferral test additionally passed on their own exclusive database; crash_child is invoked by its parent. The existing public-network RSS test remains ignored.
• The exact unified Chat CI command listed and executed 47 passing tests, including all 37 introduced tests from Hand off exhausted tool runs to an evidence-only final answer #845/Keep older chat handles from unregistering newer generations #848/Publish citation sources added by document reads #849/Keep fallback retrieval failures distinct from empty results #850/Preserve exact chat outcomes across stream closure and reattachment #851/Do not report fallback chat completion when saving fails #852/Assert a chat turn ends in exactly one earned terminal #858. All four terminal-table rows ran. Seven deliberately omitted groups each produced a green Cargo result but were rejected by the independent executed-test checker (exit 1); the complete run passed (exit 0).
• Frontend: 129 module tests, 27 browser subtests (29 including parent tests) and build/typecheck/guard passed.
• The five actual-backend browser scenarios from the previous validation were not repeated this time: the complete frontend and frozen finalization/LLM implementation are unchanged. This pass reruns the deterministic browser and real-PostgreSQL Rust regressions. No new paid-model accuracy claim is made.

Earlier evidence below refers to its stated historical heads.

---

When a tool run exhausts its gathering budget, withdrawing tool definitions can still leave the model emitting DSML in ordinary response content. The original real endpoint did this even with explicit tool_choice: none.

This change hands off before the old seventh provider request: the six-turn gathering loop stops through Rig's completion hook, then one dedicated answer call receives the completed evidence as untrusted JSON. It has no tool definitions, protocol-role history, or gathering preamble. Ordinary early answers and the existing empty-reply retry keep their short paths.

The answer call preserves tool-result identities, bytes and failure status, conversation context, current source IDs and temporal/plan distinctions. Prior citation IDs stay in a separate namespace. Invalid candidates may receive one tool-free format repair, with unchanged evidence and no rejected text. Both attempts share a 120-second deadline; input and answer text are bounded at 1 MiB. Auth, billing, rate, transport, content-filter and storage failures do not grant a retry. The buffered answer is published only after the assistant save succeeds.

Without the pre-existing gathering compatibility retries, this is at most 6 gathering + 1 answer + 1 optional repair. DSML remains a publication guard and is never executed as a tool.

Historical validation before the maintenance rebase, on the hash-checked, formatted tree eafd85f (real-model runtime 0ce696f; the original head 02e6aa3 changed documentation only):

• 997 Rust tests passed, 0 failed, 1 pre-existing live-network test ignored; strict clippy, formatting and build passed. 116 frontend tests passed.
• Five actual mutations compiled and failed runtime assertions, covering old-history requests, routing-prompt contamination, dropped evidence, a third attempt, and ignored save errors.
• Six captured failures already contained DSML in upstream delta.content; replay through the actual parser reproduced that content exactly.
• Original 30-question Linux rerun: current H needed six recoveries (first-pass 24/30, final 30/30); selected P2 achieved first-pass 30/30, no DSML or repair, 88/88 required fact slots, and 30/30 body/source persistence matches. This is not a claim of universally correct answers.
• Tradeoff: median end-to-end latency was 24.15 s vs H's 21.30 s; reported completion tokens increased. The change removes a failure-prone protocol boundary, not all model latency or factual errors.

Detailed historical run evidence is recorded below, separately from the maintenance revalidation.

Closes #844.

Historical real-model validation (before maintenance/rebase)

The following run identifiers are historical branch snapshots, not the rebased head. No real-model rerun is implied by the documentation and CI cleanup.

Tested versions

Run	PR implementation	Stable-main backport used on Linux
H: current passive recovery	86f7ba547ebc0ba40860d13b2bfd09962dac0181	560f0b752d100538a2f72aacc713061f9e58c539
P: proactive answer handoff	9114f9c	7b67133565870c10af84628e7e557be50544531e
P2: same handoff, focused answer policy	0ce696f9238462d98695ef241741fa63da6b5070	4d7e8bc6cc06e281e771b151091376b497b2319f

P2 is the selected implementation. The dev branch includes upstream
ea0557ba466979449a93b7060ca42a2cf46e2b96; the release backport keeps stable main's
migration set. This change introduces no migration. Subsequent documentation-only commits
must not be confused with a new real-model run.

Tests ran on an isolated Linux copy, with the original 30 questions and existing gold
(88 required fact slots), unchanged corpus and model configuration. Gold was never provided
to the answering model. The configured upstream was api.deepseek.com; both request and
response used the name deepseek-flash. The provider's internal model revision is unknown.
A capture proxy forwarded requests and upstream SSE without retaining authorization headers.
Raw business evidence, credentials and model answers are deliberately not committed here.

First failing boundary

Six H responses already contained DSML in upstream choices[].delta.content, with no
structured tool calls and a stop finish. Replaying those six raw responses through the
actual utopia-llm parser produced exactly the same content, zero tool calls and the original
finish reason. The adapter did not turn a valid tool call into prose in these samples.

Withdrawing tool definitions while retaining protocol-role tool history is insufficient for
this endpoint. This identifies an application-side trigger and the upstream-content boundary;
it does not establish the provider's internal reason for generating that text.

Fixed-evidence ablation

Three captured failures, three repeats each. W0 uses the captured original final request;
W1 adds the original tools and explicit tool_choice: none; W2 uses H's captured recovery;
W3 keeps W2's evidence data but substitutes the dedicated answer system. W3c appends the
focused-answer instruction now used by P2. Assertions checked that every original tool
result remained in the evidence payload. No new retrieval or gold was added.

Variant	DSML responses	Required slots answered	Median latency	Reported completion tokens, total
W0: tool protocol history	5/9	16/42	2.452 s	5,399
W1: same + explicit none	5/9	18/42	2.029 s	5,776
W2: existing recovery	0/9	42/42	7.696 s	13,747
W3: dedicated answer policy	0/9	42/42	18.710 s	35,801
W3c: focused answer policy	0/9	42/42	13.979 s	23,069

All returned HTTP 200. W0/W1's shorter times include fast invalid outputs. W2 and W3's zero
failures do not establish a reliability difference. Cache hits and stochastic generation
also differ; latency is not a controlled estimate of prompt-only cost.

Fresh end-to-end runs

Each column is a new run of all 30 questions. It is not the fixed-evidence experiment above.
The historical 29/30 result belongs to an older PR head (58d67dd / release 341044f),
not H. The previously paused partial run is not counted as a completed evaluation.

Measure	H	P	P2 selected
First answer structurally clean	24/30	30/30	30/30
Final answer clean after permitted repair	30/30	30/30	30/30
Raw upstream DSML responses	6	0	0
DSML published / stored	0 / 0	0 / 0	0 / 0
Required fact slots answered	88/88	88/88	88/88
Original eight failures: required slots	28/28	28/28	28/28
Original fifteen capped controls: required slots	38/38	38/38	38/38
Citation syntax and numbers resolve	29/30	30/30	30/30
SSE body equals saved body	30/30	30/30	30/30
Final SSE sources equal saved sources	30/30	30/30	30/30
All chat HTTP requests	239	233	235
Included gathering shape retries	30	30	30
Tool-bearing model rounds	173	173	175
Executed tool calls	394	379	384
Cases reaching six tool rounds	27	24	27
Median end-to-end seconds	21.30	32.70	24.15
Nearest-rank p95 seconds	25.8	41.4	33.9
Reported input tokens	2,885,842	2,964,421	2,967,814
Reported completion tokens	88,212	159,206	112,815
Included reasoning tokens	42,819	120,225	84,237

The existing compatibility retry received Thinking mode does not support this tool_choice
on the first gathering request in each conversation. Those 30 physical requests are counted,
not hidden behind the logical budget. P2 made 27 dedicated answer calls, with no candidate
repair; three conversations answered before the boundary. H made 27 old boundary calls and
six recovery calls; three answered early. Tool counts differ because retrieval was fresh.

Core facts were checked against the original gold and the actual evidence returned in each
run, including numbers, units and plan/report/verified distinctions. Core-slot completeness
and resolvable citation numbers are not a claim that every additional sentence is correct.
Separate factual/attribution findings are tracked privately, outside this DSML change.
One H answer used full-width citation brackets that the existing frontend does not recognize;
this PR does not change citation parsing.

The reason to select P2 is the provider-before-I/O answer handoff: it avoids sending a known
failure-prone final request and retains all required evidence, without more tool execution.
It is not a latency improvement over H: median latency is 2.85 seconds higher and reported
completion tokens increase. P2 reduces the initial P prototype's excess output and latency.
No dollar-cost claim is made from token counts. One frozen set is not a universal guarantee.

Deterministic and combined checks

On the formatted dev tree eafd85f (the same runtime implementation as 0ce696f),
with real PostgreSQL fixtures and PDF extraction enabled:

• cargo fmt --all --check, strict workspace/all-target clippy, locked workspace build: pass.
• cargo test --locked --workspace: 997 passed, 0 failed, 1 ignored. The ignored test is
  the pre-existing live public-HTTPS RSS acceptance test requiring httpbingo.org.
• Frontend: 116 passed; typecheck/style guard/production build pass.
• Exact stable backport: 29 chat tests, 36 LLM tests, and 82 frontend tests pass;
  its release image starts against migration 69 and passes the original-admin login.

Representative production-path assertions:

Contract	Tests / evidence
Old seventh request never goes out	budget_finalization_accepts_an_answer_and_protocol_explanations; outgoing request has two messages, no protocol roles/tools/routing preamble
Early empty/narration retries consume budget; greeting stays short	early_retry_budget_and_no_evidence_short_path_remain_bounded, existing empty-reply regressions
Multiple calls per turn retained; UTF-16 offsets stable	parallel_tool_results_and_utf16_step_positions_survive_handoff (12 results)
No tools execute after handoff	budget_finalization_refuses_structured_calls; bounded-recovery request/step counts
At most one repair with unchanged evidence	finalization_recovers_once_from_existing_evidence_without_tools, unsuccessful_recovery_never_loops_or_reopens_tools
Real errors cannot spoof handoff	gathering_errors_cannot_spoof_the_private_handoff
Failures, prior citations, identity, injection text remain data	evidence_and_citations_are_data_not_protocol_messages, prior_citations_are_separate_and_failed_unknown_results_are_not_absence, failed_tool_observation_is_not_reported_as_empty_knowledge
Full serialized input / output / total deadline bounded	oversized-context/text tests, the_total_deadline_covers_candidate_repair
Nonrepairable statuses and finishes do not retry	HTTP 400/401/402/403/422/429, content-filter and unknown-finish matrix
Split DSML and narration prefixes rejected; explanations preserved	budget-finalization split/prose/fence/explicit-example controls
EOF, UTF-8, missing/unknown finish semantics preserved	utopia-llm bytewise/unfinished-stream/finish-reason tests and adapter tests
Save before publication; no done or model retry on DB error	save_failure_is_an_error_without_publishing_the_buffered_answer_or_retrying uses an actual PostgreSQL trigger failure
Source mapping, disconnect, concurrency	final-sources, disconnect/reattach, concurrent-chat tests plus all 30 SSE/DB comparisons

Five mutations were actually executed in an isolated copy at 9114f9c, each compiling and
then failing a runtime assertion (not merely failing compilation): restore the old seventh
request; add the routing preamble back; drop the last evidence result; permit a third answer;
ignore assistant persistence failure. Restoring the source passed the targeted suite.
The final focused-policy and early-budget changes subsequently passed the full checks above.
All 506 tracked build-input files were hash-compared with that checked Linux tree.

Reproduction and privacy

The private archive retains per-run questions, source/gold review, outgoing JSON, upstream
SSE and chunk offsets, parsed-turn comparison, delivered SSE, persisted records, usage,
image identities and mutation logs. The corpus hash was identical between staging and
production before/after the runs. Runtime account/model configuration was unchanged except
for the isolated capture proxy URL. This report publishes aggregate findings only.

[WaylandYang]

Read it. The shape is right and the care shows in the places that usually get missed.

Things I specifically checked and liked:

• The detector is narrow. It only runs at the budget-finalization turn, only when the question does not mention DSML, and only tests starts_with against the first 80 characters with whitespace and both pipe spellings stripped. Prose that mentions the syntax, or a fenced example partway through an answer, is untouched. Inspecting the assembled turn rather than deltas is the right call — chunk boundaries were the obvious way to get this wrong.
• The recovery output goes through the same gate. finalization_error plus the stop finish-reason check run on the recovered text too, so a second round of markup fails instead of being published. That closes the hole the fix would otherwise open.
• finalization_rejected is set only by a rejected model candidate, not by an auth, credit, transport or database error. That matches the reasoning behind NOT_RETRYABLE in error.rs — a retry after a billing failure just says the same thing again seven minutes later.
• The recovery request is injection-aware by construction: evidence is passed as JSON data with the system prompt saying it is untrusted, tools are absent, and the comment about tool_choice being the field a compatibility retry could flip to auto shows you looked at how the escape would actually happen. evidence_and_citations_are_data_not_protocol_messages asserting the rejected plan text is absent from the payload is the right assertion to have written.

Two things.

1. The description contradicts the record and the code. The PR says the fix "deliberately fails honestly at the exhausted boundary rather than requesting a retry with no remaining budget", while 0042 says "the route permits exactly one additional physical request" and chat_finalization::recover makes it. I can see the intended distinction — the logical budget of seven model calls is unchanged and the extra call carries no tools — but as written the summary says the opposite of what ships, and the description is what a reviewer reads first. Please reword it.

2. The fix has not met the endpoint that caused it. #844 came out of a 30-question real-model evaluation, and your validation is explicit that these are deterministic scripted-upstream regressions rather than a rerun of it. That is the honest disclosure and I would rather have it than a vague claim — but it does leave the end-to-end rate unknown, and this endpoint has a history of ignoring what it is told. Rerunning that evaluation against the patched build, and reporting how many of the thirty now finalize cleanly, is what would close it. Not a blocker for merging a guard that is strictly better than publishing markup as an answer; worth doing before anyone concludes the problem is gone.

CI is still running the backend and migrations jobs.

[Maya-Kid]

Thanks for the review. I’m pausing further guard changes while I investigate why DSML is entering the response body in the first place. Relying only on interception rules is not a good long-term solution.

The full real-model rerun you suggested was useful: 29 of the 30 questions finalized cleanly, but one response put narration before a bare DSML block and bypassed the initial guard. That reinforces the need to understand the underlying failure rather than keep expanding detection patterns.

I’m now focusing on the raw final-turn request and response, including the tool-history format, endpoint behavior, and adapter handling, to identify where the protocol text enters ordinary content. I’ll report the findings before proposing further changes. The PR description has also been corrected to distinguish the unchanged logical tool-runner budget from the one additional tool-free recovery request.

[WaylandYang]

Both points from my earlier review are answered, and the second one better than I asked.

You reran the 30 questions against the real endpoint with the original gold (88 fact slots), unchanged corpus and configuration, gold withheld from the answering model, across three implementation variants. That is the thing that was missing, and finding that six H responses carried DSML in choices[].delta.content with no structured tool calls and a stop finish — then replaying those six raw responses through the actual utopia-llm parser to confirm the adapter was not turning a valid tool call into prose — is a cleaner attribution than I expected anyone to produce. The description contradiction is gone too; the new title and the handoff description say the same thing the code does.

The validation report is in the wrong place, though.

docs/decisions/0042-evidence-finalization-validation.md is a second file numbered 0042, beside 0042-the-chat-loop-is-a-runner-with-hooks.md. The conventions make the number a stable anchor for references — two files carrying it means "0042" no longer names one thing. The PR also does not add the record to either index table in docs/decisions/README.md, so it is unindexed as well as ambiguous.

There is a sharper reason than the convention. The report's version table cites 86f7ba54, 9114f9c and 0ce696f9 — commits on this PR branch. This repository squash-merges to dev, so those SHAs stop existing the moment this lands, and the record would permanently cite three commits nobody can resolve.

What I would do instead:

• The durable conclusions go into 0042 itself, which this PR already amends by 61 lines: that the endpoint emits DSML in delta.content with a stop finish and no structured calls, that the adapter was cleared by replay, and that withdrawing tool definitions while retaining protocol-role history is insufficient. Those are findings about the system that outlive the PR.
• The run-by-run comparison table belongs in this PR description, where branch SHAs are still meaningful and where a reader looking for "how was this validated" will actually look.

If some of it really is decision material rather than evidence — the choice of P2 over H and P looks like it might be — then it wants its own number, which is 0052 now that 0049, 0050 and 0051 are taken by #839, #840 and #841, plus a row in both index tables.

Separately, this PR is one of five appending a step to ci.yml; see my note on #852 for why that should collapse into one step and which PRs conflict as a result.

[Maya-Kid]

Completed the requested placement cleanup. The second 0042-evidence-finalization-validation.md is removed. The original ADR 0042 retains the durable findings: DSML was already in upstream delta.content with a stop finish and no structured calls; parser replay reproduced it; withdrawing tool definitions while keeping protocol-role history was insufficient. It now links to this PR for the run-by-run evidence.

The full historical comparison tables and branch-specific SHAs have moved into this PR description, explicitly separated from the maintenance revalidation. No new ADR number was introduced. The evidence-only handoff implementation and all test files are unchanged; this pass only changes documentation and the shared CI step.

Rebased onto dev@79caca04 and pushed head 24fb77b. The CI consolidation and positive/negative test-coverage checks are reported on #852; full integration validation and its exact SHA are in the updated description. No new real-model run is claimed by this cleanup.

[Maya-Kid]

Rebased and pushed as 1b33fd8f9caff47212bacd63729701a1913de3e7 on dev 474b904. The evidence-only handoff runtime remains unchanged; ADR placement and unified Chat CI remain as agreed.

For integration with #858, the two pending DSML table rows now exercise the existing six-tool-turn finalization boundary, rather than stopping after one tool turn. The narration fixture puts the DSML marker on a following bare control line. This is a test-only adjustment to the frozen boundary, not an expansion of early-answer or inline-marker policy. Both terminal tests remain, and all four table rows executed without pending skips.

The updated description records the complete Linux validation and exact executed-test coverage checks; the real-model run tables remain explicitly historical.

[Maya-Kid]

Completed the post-merge rebase onto dev@7cfeedf. Current head b4646386be142d2e502c6e88eaf01e9817f497ec. Kept all four sibling regression modules in the rustfmt order you specified (fallback, persistence, registry, sources), and retained both terminal/stream modules. The evidence-only handoff and LLM implementation are unchanged.

Combined Linux tree d06f4ad0199a382635a6bbf94210b9421cbeba66: fmt, strict Clippy and build pass; workspace 1,018 pass, 5 ignored. Delivery/process-kill parent tests and busy-deferral additionally pass on an exclusive database. The exact Chat CI command executes 47 tests, including all 37 introduced tests; all four terminal-table rows run. Seven deliberately omitted groups are each rejected by the independent executed-test coverage check despite green Cargo exits. Frontend 129 module tests and 27 browser subtests pass.

The description now records this head and validation scope. No runtime-policy expansion or new model evaluation is included.