Never auto-merge a similarity-proposed pair on the batch verdict alone; tell the adjudicator the names differ - #889
Conversation
|
Two more bench runs with the trace recording (runs 5 and 6), and what they settled.
Still no false merge. The traces of the twelve second looks answer the budget question directly. After the six lookups every one of them asked for the two lookups the prompt still lists, Refused once, it still concludes; refused twice, it never gets the turn. The model works down the menu the prompt offers, and that menu is eight lookups long (facts ×2, passages ×2, ledger ×2, namesakes, consequences), not six. Third commit: Runs 7 and 8 on that build are in progress; the numbers to watch are the count of "looked but did not conclude" and whether Unrelated but recorded: two of twelve bench runs ended with zero typed entities after extraction, which makes every anchor |
…e; tell the adjudicator the names differ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
On the identity bench, roughly one second look in three ended as "the agent looked but did not conclude": six lookups, then two more turns that never reached decide or defer. The trace did not say what those two turns were, so it now records a refused lookup (with the tool and arguments the model asked for) and a turn that only spoke (its first 200 characters). Turns that hit the limit are budgeted separately from lookups, and the limit message says to answer with decide or defer. Whether the budget itself is too small is what the next bench run is for. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
Two bench runs with the trace recording showed what the inconclusive second looks were doing after six lookups: asking for the two lookups the prompt still lists, namesakes and consequences, and being refused both. A look that was refused once still concluded on its last turn; one refused twice never did. Six was two short of the menu the prompt offers; eight is that menu. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
11b2763 to
51beb99
Compare
|
Runs 7 and 8 on the
Nine second looks, none ended as "looked but did not conclude". P2 split in two orders is extraction emitting |
|
Review (posted as a comment because GitHub refuses a formal review on a PR under the same account). The direction is right and the bench write-up is convincing, but the title's "never" does not hold yet. Two paths still apply a batch
Suggest: escalate ( Branch is clean against dev (behind, no conflicts), CI green, DCO fine. |
…un, under governance too The batch verdict on a name-vector pair could still merge on two paths: when look_again returned None (loop budget spent, model error) the adjudicator fell through to put_verdict and apply_verdict, and the cached verdict then skipped the second look on every later encounter; and with governance on, wants_second_look had no similarity rule, so a 0.85 same on an unrelated-shaped pair merged on the batch alone. Now batch_verdict_may_apply says when a batch verdict may land without the second look (never for a similarity-proposed same), and both paths escalate as second_look_unavailable instead, without caching. The name_vector prefix lives once, in utopia_core::review_reasons; the store writes it, the server reads it, and the extractor takes only the cosine. The stale six-lookups comment follows MAX_STEPS. The design note and 0041's status line say what the rule is now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
|
Applied the review as maintainer edits so the title holds, rebased onto current dev.
Locally: core, extract (83), adjudication, governance and store resolution tests pass; clippy clean. Merging once CI agrees, after #906. |
1 participant
Follow-up to #877, from the identity bench rather than from a review.
What the bench showed
Two runs each of
devbefore and after #877 onscripts/bench/identity.mjs(DeepSeek-V3.2 extraction, bge-m3 embeddings, forward and reverse ingest order). Channel 2 fixed the case it was built for: with the short form arriving first,海洋探测器1号and海探1were split in bothdevruns and merged in both branch runs, from aname_vectorpair with cosine 0.78. But one branch run also produced three false merges, all from a single batch verdict: the adjudicator merged the bare张伟(engineering) into财务部总监张伟(finance) at confidence 0.85, over the 0.8 auto-merge line. That pair only exists because channel 2 proposed it; the literal channel never puts two different strings together, sodevkept the namesakes apart partly by luck.The batch step is the wrong place to settle such a pair. It sees two names and a few facts, and the identity rules it reads say a dropped qualifier is the same thing, which is exactly wrong when the qualifier is what tells two namesakes apart.
What this changes
Two small things, both scoped to pairs whose review reason is
name_vector|…:AdjudicationPairgainsproposed_because; for a similarity-proposed pair both the batch prompt and the second-look prompt carry aWhy paired:line stating that the names are similar but not the same string, so a dropped qualifier is not evidence and the facts are. Same-name pairs are unchanged and carry no such line.sameon such a pair is never applied directly.needs_second_lookroutes it to the existing tool-using second look regardless of confidence; that look can read each side's facts and passages before deciding, and its result goes through the same 0.8 gate as before.differentandunsurefollow the existing rules: keeping two records apart is the safe direction, and the second look already handles the unsettled ones."Proposed because similar" and "proposed because identical" are two strengths of evidence; they no longer share one auto-merge line. The threshold number itself is not touched: the model's confidences are not calibrated, so moving 0.8 to 0.9 would only shift which lucky verdicts get through.
A third, smaller change came out of reading the bench traces (second commit): the second look's trace now records a lookup it refused for being over budget and a turn that only spoke, and turns that hit the limit are budgeted separately from lookups. See "What it costs" for why.
Measured
Same bench, same models, two runs per configuration.
P/R/F1are pairwise over the anchors the run resolved;missingis anchors the run could not locate (type binding varies between runs, so runs are comparable in kind, not in F1 to the second decimal).What the traces say, pair by pair:
张伟 ~ 财务部总监张伟was routed to the second look, which kept them apart at 0.95 after reading both sides' passages ("one is 工程部工程师张伟, the other 财务部总监张伟"). No run of this PR produced a false merge; precision is 1.00 in all seven runs that resolved anything.海探1 ~ 海洋探测器1号was merged by the second look in run1 forward and both orders of run3 (0.95 to 0.98, "the source passage says 海洋探测器1号（简称'海探1'）"). In run2 (both orders) and run4 forward it stayedpendingbecause the second look ended as "looked but did not conclude". In run4 reverse the pair was never proposed: the embedding endpoint failed once and that batch fell back to literal recall, which is Recall a mention by its name vector so a short form or another script meets its entity #877's designed fallback and unrelated to this PR.MAX_STEPS6) the pattern was identical: one lookup per turn, six lookups, then two turns that never reacheddecideordefer. The trace recording added in the second commit showed what those turns were: requests for the two lookups the prompt still lists,namesakesandconsequences, refused as over budget. Refused once, the look still concluded on its last turn; refused twice, it never got the turn. The prompt's menu is eight lookups long, so the third commit setsMAX_STEPSto 8. Runs 7 and 8 on that build: nine second looks, none inconclusive;海探1 ~ 海洋探测器1号merged by the second look in all four orders (0.95, 0.98, 0.99, 0.95, each citing the passage that says 海洋探测器1号 is abbreviated 海探1); the twounsureoutcomes are proper deferrals that carry a question for a person. Cost per second look went from 8 chat calls to 9 or 10.MAX_STEPS8 orders is not this PR's. In those runs extraction emitted海洋探测器2号and海探2as two entities from the same document. Channel 2 looks for neighbours when a mention is resolved, and a sibling from the same document has no name vector yet at that moment, so no pair was ever proposed and neither adjudicator saw it. That is Recall a mention by its name vector so a short form or another script meets its entity #877's resolve-at-mention timing, which cut 3 (resolve after parse) is for.dev: the cross-script name pairs have bge-m3 cosine around 0.50, below channel 2's 0.60 floor, so no pair is ever proposed for them. Cut 3 territory, not this PR's.What it costs
One extra tool-using round for every similarity-proposed pair the batch step calls
same: 2 to 4 pairs per 12 documents on the bench, 9 or 10 chat calls each with prompts of 2k to 4k tokens. Measured on the server's usage log, all second looks together are 5 to 6 percent of the pipeline's DeepSeek tokens for a run, and the guard's share of those is smaller still. Nothing changes for pairs from the literal channel. A second look takes 40 to 60 seconds of wall time in the background job, so a short-form merge lands a minute later than #877 applied it, with the agent's trace attached to the review.Verified
cargo test -p utopia-extract75/0 (prompt carries the line for a name-vector pair and not for a same-name pair; onlyname_vector|reasons become a note);cargo test -p utopia-server(same → second look regardless of confidence; different unchanged; same-name pairs keep the old rule); clippy clean; fmt clean.🤖 Generated with Claude Code