Skip to content

feat(workhub): add opt-in Jev assisted routing - #5562

Merged
ARE404 merged 9 commits into
apache:mainfrom
ARE404:feat/jev-routing
Sep 23, 2026
Merged

ARE404 merged 9 commits into
apache:mainfrom
ARE404:feat/jev-routing

Conversation

@ARE404

@ARE404 ARE404 commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Adds opt-in TypeSafe Jev decisions to WorkHub, configured under General → Advanced settings. The default remains disabled; there is no separate tool-model setting or change to conversation, execution, or title models.

Jev classifies intent before selecting from existing bounded opaque candidates. A valid decision constrains subsequent coordination actions; it is not merely advisory. The existing coordination model/tool execution remains in place. Valid uncertainty asks for clarification; unavailable credentials, privacy mode, malformed responses, and provider failures preserve the existing routing path. Candidate-set binding and the existing Action Gate remain authoritative. The API key lives in the Runtime Host credential vault and only a masked projection reaches settings.

Validation:

  • 65 targeted codec, routing, coordination, Host settings, and generation tests passed.
  • All 33 execution-composition tests passed, including the default production WorkHub path. Absent Jev decisions omit the optional field so durable admission preserves the existing path.
  • Real policy/vault persistence test passed.
  • Core, Runtime Host, Desktop main and renderer builds; renderer typecheck; strict architecture comparison against main passed.
  • Temporary native Electron acceptance passed: save key, enable, reopen, narrow layout, remove key and disable.

Live TypeSafe verification passed with synthetic data through the production Jev routing adapter: create_new, delegate_existing with the expected candidateRef/candidateSetId, and answer_here (0.85–1.44 s). Credentials were supplied only in memory. Repository lint and formatting checks also pass.

Visual comparison

Historical Electron screenshots (1200 × 900), isolated synthetic profile. The current consent text additionally discloses recent conversation. The before image uses the base GeneralSettingsPage source; after shows explicit Jev opt-in with a synthetic key.

Before After: Jev enabled
General settings before Jev General settings with Jev enabled

Default collapsed state.

Small live routing comparison

Twelve fixed synthetic cases, using the production split Intent/Recall adapters; this is not a full default WorkHub coordination/tool-loop benchmark.

Configuration Expected outcomes Median latency
Jev jev-1.13.0 10/12 0.80 s
DPSK deepseek-v4-pro, diagnostic 2048-token output budget 10/12 5.25 s

The unchanged DPSK split-adapter budget (80/160 tokens) produced no decision JSON in 12/12 cases: its reasoning exhausted the intent budget. This is an adapter/budget incompatibility, not evidence of inferior model accuracy. An explicit thinking-off probe did not alter the wire options. The larger-budget control changes only the test transport, not product settings.

Jev unnecessarily clarified two continuations. DPSK's diagnostic run bound an ambiguous request to a specific candidate and exhausted its reasoning budget on one other request. One pass, synthetic cases, and provider/cache variation limit conclusions.

Full report, inputs, raw measurements, and replay script.

Correctness review follow-up

The chosen scope keeps opt-in two-stage Jev routing and the saved-key/off switch. Jev supplies the existing structured routing decision; this PR does not replace the coordination/tool execution loop or claim an end-to-end quality/cost advantage from the small synthetic comparison. Valid decisions constrain actions; errors return the unbound path.

  • Compatibility epoch 182 → 183 for policy snapshots, mutation, and credential scope, preserving main's executor compatibility changes.
  • Host action operations bypass model routing; missing decisions remain absent in durable admission. Recovery checks execution availability before routing egress.
  • Policy, outbound admission, and credentials are refreshed before both Intent and Recall. The deadline and cancellation cover preflight reads, candidates, response parsing, and waiting for transport cleanup; Host preparation is bounded too.
  • Fixed diagnostic tags make fallback visible without logging credentials, payloads, or raw provider errors. Strict probability validation is retained (0.999 is already within tolerance). Intent keys are checked against the core union.
  • Consent copy now includes recent conversation; localized copy and the Settings action guard use existing shared modules. Historical evidence is archived under docs/archive; the replay harness lives in scripts/compare-jev-routing.mjs.

Validation: full workspace build and typecheck; lint, formatting, UI inventory and renderer architecture checks; execution-composition, Jev adapter, protocol, and selected WorkHub/recovery root tests passed. Tests include privacy/disablement between requests, aborting hung reads and fetches, the real 8-second deadline, unavailable/empty candidates, action admission, and undefined → unbound durable admission.

Candidate-resolution failure intentionally falls back in the Jev adapter. Stale selected candidate sets still reject at the existing Action Gate; they are not rebound automatically.

Opt-in scope and follow-up review

The chosen product scope is a user-facing, default-off Jev setting under General → Advanced. Users can keep a saved key while temporarily disabling Jev. Intent and Recall remain separate calls followed by the existing coordination and Action Gate path; this does not replace the coordination model or establish a project-wide routing decision.

  • One 8-second production preparation budget covers policy/transcript reads, Intent, candidate resolution, Recall and transport cleanup. A slow Intent can exhaust the remaining budget; the turn then falls back to the existing unbound path. Production deadline failures now report timeout; explicit caller cancellation remains silent.
  • Explicit routing-model injection remains an experiment seam and bypasses the production Jev policy/deadline wrapper. The code now documents that injected implementations own those protections.
  • The existing failure asymmetry is retained and documented: transcript/preparation failures and injected-model throws bind the admission to clarify-only; errors handled inside the Jev adapter return no decision and preserve the legacy unbound path.
  • Probability validation is retained: the official Choice answer contract specifies every criteria option and probabilities summing to 1. The decoder allows ±0.01 rounding tolerance.

Validation: Runtime Host build and the Jev/execution-composition regression suites passed locally.

Rebased onto main c7d205a42 at head a78791cdc. Full build, typecheck, lint, formatting and renderer architecture checks passed; 172 focused regression tests passed, including WorkHub result delivery and recovery. Remote CI must validate the rewritten head.

@github-actions github-actions Bot added effort/L Under 1000 readable lines effort/XXL Over 2500 readable lines and removed effort/L Under 1000 readable lines labels Sep 21, 2026

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ARE404 — the engineering here is careful: bounded projections, fail-open semantics, strict response decoding, a clean vault lifecycle, and unusually honest benchmark reporting. Reviewed head e5e126eff (+4,570/−30 across 32 files; ~410 of those are production lines).

Verified clean

  • The | undefined contract holds end to end: absent decisions preserve today's unbound path at every consumer (root-turn-coordinator.ts:1577, readActiveWorkHubRoutingRequest, durable normalization).
  • Candidate binding is sound: opaque whc_<hash> refs re-validated against a fresh candidate set at act time; stale sets reject rather than silently rebind.
  • Credential lifecycle is closed: write-only masked projection, absent from policy snapshots and durable decisions, clear→disable ordering is fail-closed, export strips it.
  • Egress is bounded (≤2K-char message, ≤8 transcript turns at ≤600 chars, ≤32 opaque candidates — no session IDs, paths, or turn IDs) and the response decoder is strict.
  • Every internal failure is fail-open: disabled, missing key, incognito, and provider errors all return undefined into the existing path.

Design — needs a decision before the mechanics

This activates a deliberately dormant seam. dependencies.workHubRoutingModel is the experiment seam: nothing injects it in production, and the #5152 acceptance record (docs/workhub-native-acceptance-2026-09-12.zh-CN.md) notes that as intentional pending the comparative production-path evidence #3492 asks for. This PR makes createJevRoutingModel the production default behind a user toggle (execution-composition.ts:2093) — shipping the activation decision as a preference rather than settling it in the tracker. The direction question there is still open (the simpler one-session-plus-tools baseline proposed on 9/12 has no reply yet). Whichever way it goes, this wants an explicit call on #3492 or a dedicated issue first.

The evidence doesn't yet carry a production integration. The shipped comparison is 12 synthetic cases: 10/12 vs 10/12 — parity, with the latency delta confounded by the starved 80/160-token budget the report itself flags. What the data does establish is a cheaper problem: the existing split adapter starves reasoning models (0/12 at production budget on deepseek-v4-pro). If routing quality is the motivation, fixing that budget in the existing adapter is the smallest change. If the motivation is evaluating Jev, compare-routing.mjs already proves the zero-plumbing path — env key + stub stores + the injection seam — so the settings/vault/policy lane (~240 of the ~410 production lines) isn't needed for that. Shipping it commits us to a persisted policy field, a vault scope, a wire mutation, and a hardcoded vendor endpoint one week after the vendor came out of stealth.

"Advisory" understates the semantics. A returned decision is binding: routingDecisionAllowsProposal (workhub-coordination-coordinator.ts:751,1145-1164) rejects every proposal that deviates, and a clarify decision rejects all actions. With the toggle on, the service pre-empts the coordination model's judgment on every WorkHub message — fail-open on errors, but not advisory on decisions. Fine if that is the chosen direction; it should be stated plainly.

Findings

P1 — missing compatibility epoch. set_jev, the jev credential locator, and RuntimePolicy.jev all cross the wire; every prior additive policy change bumped the epoch (#5219→146, #5164→145, #3704, #2684). Once any host persists jev — even {enabled:false} — it rides every runtime.policy.query snapshot, and older clients' strict exactRecord decode fails on the unknown key: toggling the feature on and back off permanently breaks policy reads for not-yet-upgraded peers. Bump RUNTIME_HOST_COMPATIBILITY_EPOCH (176→177 on current main).

P2 — per-PR evidence committed to the tree. docs/pr-5562/ (README + ~3.5K lines of result JSON) and .maka-shots/pr-5562/*.png are run logs and screenshots; docs/README.md keeps those on the PR thread or in docs/archive/. compare-routing.mjs is the durable part — scripts/ if we keep it.

Line-local items are inline — an operation:'action' admission poison (P2), disclosure copy that omits the transcript payload (P2), the Intent→Recall incognito window (P2), and a few smaller notes.

Decision for you — is the enabled bit worth a policy field? "Key configured" already disables the feature at three layers (exportCredentialMaterial → undefined, the Switch disabled without a saved key, clearing the key force-disables). Dropping policy.jev/set_jev/AppSettings.jev.enabled removes ~80-100 lines including the codec and mutation; the only lost capability is "key saved but temporarily off".

Remaining notes: resolveCandidates failing inside Jev yields an unbound turn while the same failure in the host adapter yields a bound clarify that rejects every action — worth a sentence either way; a stale candidate set after a bound delegate_existing wedges the turn into clarify (pre-existing contract, now default-exercised); startRecoveredMessages invokes the prep before the availability check (root-turn-coordinator.ts:1511), so a doomed recovery still egresses; JEV_COPY belongs in renderer/locales/ with its 34 siblings; worth adding tests for abort-mid-fetch, resolveCandidates throwing, empty candidates, and an end-to-end → undefined → unbound admission.

Next step

I'd hold at the design question: settle on #3492 (or a dedicated issue) whether a vendor-specific routing backend is the direction and whether the evidence bar #5152 set is met — before reviewing the mechanics further. If "evaluate first" wins, the smallest honest shape is jev-routing-model.ts + its test + the harness, deferring the settings/vault/policy lane until the comparison earns it. The adapter itself is in good shape; happy to re-review whichever form lands.

中文

谢谢 @ARE404——工程做得相当细致:有界投影、fail-open 语义、严格的响应解码、干净的密钥生命周期、少见的诚实基准报告。评审基于 head e5e126eff(+4,570/−30,32 个文件;其中约 410 行生产代码)。

已验证干净:| undefined 契约端到端成立(缺席决策在所有消费方都保留现有无绑定路径);候选绑定可靠(opaque ref 在执行时对全新候选集重验,陈旧集合直接拒绝);凭据生命周期闭环(只写掩码投影、不进 policy 快照与持久化决策、先删 key 再置 disabled 的 fail-closed 顺序、导出脱敏);外发载荷有界且响应解码严格;所有内部失败 fail-open 回现有路径。

设计层需先定案:

  1. 本 PR 激活了一条被刻意休眠的缝。 dependencies.workHubRoutingModel 是实验缝:生产从未注入,docs/workhub-native-acceptance-2026-09-12.zh-CN.md 明确记录这是 #5152 的既定设计——在等 #3492 要求的对照生产路径证据。本 PR 把 createJevRoutingModel 接成生产默认(execution-composition.ts:2093),等于把激活决策变成了用户设置项,而没有先在 tracker 定案。那里的方向问题仍开着(9/12 提出的"一个会话加工具"简化基线尚无回复)。无论结论如何,都应先在 #3492 或专门 issue 拍板。

  2. 证据还撑不起一个生产集成。 所附对比是 12 个合成 case:10/12 对 10/12 打平,延迟差异又被报告自己标注的 80/160 token 预算饿死所混淆。数据真正确立的是一个更便宜的问题:现有 split adapter 的预算饿死推理模型(deepseek-v4-pro 在生产预算下 0/12)。若动机是路由质量,修该预算是更小改动;若动机是评估 Jev,compare-routing.mjs 已证明零管线路径可行(env key + stub stores + 注入缝),那么设置/vault/policy 整层(约 410 行生产代码里的 ~240 行)并非必需。带着持久化 policy 字段、vault scope、wire mutation 和硬编码供应商端点上线,等于在供应商出 stealth 一周时就签下长期义务。

  3. "advisory" 轻看了语义。 返回的决定是绑定:routingDecisionAllowsProposal(workhub-coordination-coordinator.ts:751,1145-1164)拒绝一切偏离提案,clarify 决定则拒绝所有动作。开关打开时,该服务对每条 WorkHub 消息前置裁决协调模型的判断——错误时 fail-open,但决策上不是"建议"。若这是选定的方向没问题,但应当明说。

发现

  • P1 — 缺兼容性 epoch。 set_jev、jev credential locator、RuntimePolicy.jev 都过 wire;历史上每次同类变更都 bump epoch(#5219→146、#5164→145、#3704、#2684)。一旦任何 Host 持久化过 jev——哪怕 {enabled:false}——它会随每个 runtime.policy.query 快照下行,旧 client 的 strict exactRecord 解码直接失败:即开了又关也会永久破坏未升级对端的 policy 读取。请 bump RUNTIME_HOST_COMPATIBILITY_EPOCH(当前 main 为 176→177)。
  • P2 — 每 PR 证据入库。 docs/pr-5562/(README + ~3.5K 行结果 JSON)与 .maka-shots/pr-5562/*.png 属 run log 与截图;docs/README.md 约定这类内容放 PR 线程或 docs/archive/。compare-routing.mjs 是其中真正耐久的部分,若保留应去 scripts/。
  • 行内条目见行内评论——operation:'action' admission 中毒(P2)、披露文案漏掉 transcript 载荷(P2)、Intent→Recall 之间的 incognito 窗口(P2),以及若干小项。
  • 留给你拍板——enabled 位是否值得一个 policy 字段? "已配置 key" 本身已能在三层关掉功能(exportCredentialMaterial→undefined、无存 key 时 Switch 禁用、清 key 强制关)。删掉 policy.jev/set_jev/AppSettings.jev.enabled 约省 80-100 行(含 codec 与 mutation);唯一失去的能力是"存着 key 但暂时关"。

其余备注:Jev 内 resolveCandidates 失败得到无绑定 turn,而 host adapter 同样失败得到拒绝一切动作的绑定 clarify——值得写一句;绑定的 delegate_existing 遇上候选集变陈旧会把该 turn 卡成 clarify(既有契约,现在默认被走到);startRecoveredMessages 在可用性检查前就调 prep(root-turn-coordinator.ts:1511),注定失败的恢复也会外发;JEV_COPY 应进 renderer/locales/(与 34 个兄弟文件一致);值得补的测试:abort-mid-fetch、resolveCandidates 抛错、空候选集、端到端 → undefined → 无绑定 admission。

下一步

建议停在设计问题上:先在 #3492(或专门 issue)定案——供应商专属路由后端是否为方向、#5152 立的证据门槛是否已达到——再继续评实现。若结论是"先评估",最小诚实的形状是 jev-routing-model.ts + 测试 + harness,设置/vault/policy 层留到对比证明 Jev 值得时再加。adapter 本身状态不错;无论哪种形态落地我都愿意复审。

@@ -1568,15 +1568,13 @@ export class RootTurnCoordinator implements HostedExecutionAuthority {
if (execution.kind !== 'workhub_coordination' || !this.prepareWorkHubRoutingDecision) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prepareFreshWorkHubExecution now attaches a routing decision to every workhub_coordination execution, but operation:'action' descriptors are forbidden from carrying one — normalizeRootExecutionDescriptor (agent-run-store-contract.ts:1143) throws on operation !== undefined && routingDecision !== undefined, and a throw inside admitRootTurn poisons the singleton coordination session for the host epoch. Reachability caveat: runWorkHubCoordinationOperation has no in-repo caller yet, so this is a latent path — but this PR is what arms it (on main the callback was absent without an injected model). One-line guard: || execution.operation === 'action'.

中文 `prepareFreshWorkHubExecution` 现在给每个 `workhub_coordination` 执行都附 routing decision,但 `operation:'action'` 描述符禁止携带它——`normalizeRootExecutionDescriptor`(agent-run-store-contract.ts:1143)会对 `operation !== undefined && routingDecision !== undefined` 抛错,而 `admitRootTurn` 内的抛错会让单例协调会话在整个 Host epoch 中毒。可达性说明:`runWorkHubCoordinationOperation` 目前仓内无调用方,是潜伏路径——但正是本 PR 把它武装起来的(main 上无注入模型时回调根本不存在)。一行守卫即可:`|| execution.operation === 'action'`。

criteria[candidate.candidateRef] =
'This candidate is the clear best match for the requested work.';
if (state.candidates.length === 0) return { kind: 'routing', disposition: 'clarify' };
const target = await choose(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No policy re-check between the Intent and Recall fetches: enabling incognito (or disabling Jev) while the first request is in flight still sends the second request — user text plus candidate summaries — to api.typesafe.ai after the user's explicit privacy signal. Re-read the snapshot (or call resolveHostOutboundExecution again) before this second choose.

中文 Intent 与 Recall 两次 fetch 之间没有复查策略:第一发请求在途时用户开启 incognito(或关闭 Jev),第二发仍会把 userText + 候选摘要发给 api.typesafe.ai——发生在用户明确表达隐私意图之后。建议在这个 `choose` 前重读 snapshot(或再调一次 `resolveHostOutboundExecution`)。

const transport = (input.createTransport ?? createProxiedFetchTransport)(
toRuntimePolicyProxy(outbound.networkProxy, outbound.secretMaterial.networkProxy?.secret),
);
const signal = AbortSignal.any([request.abortSignal, AbortSignal.timeout(8_000)]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The three store reads above run before the 8s signal exists and honor request.abortSignal only via the one-shot .aborted check at :53 — a hung store read makes decide hang forever inside sessionAdmission.run, which also serializes claimWorkHubAction. Worth racing the reads against the combined signal.

中文 上面三个 store 读取在 8s 信号创建之前执行,只靠 :53 的一次性 `.aborted` 检查响应中止——store 读挂死会让 `decide` 在 `sessionAdmission.run` 里永远挂住,而这条 admission lane 同时串行着 `claimWorkHubAction`。建议让这几个读取与合并信号竞速。

} finally {
await transport.close();
}
} catch {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

catch { return undefined } makes a permanently failing endpoint (401, schema change, proxy drop) indistinguishable from "disabled" — a silent no-op forever. The jev_request_failed/jev_invalid_response tags already exist and carry no secrets; emitting or counting them keeps the failure visible without logging payloads.

中文 `catch { return undefined }` 让永久性故障(401、schema 变更、代理掉线)与"未启用"不可区分——永远静默无操作。`jev_request_failed`/`jev_invalid_response` 这两个 tag 已经存在且不含机密;emit 或计数一下就能让故障可见而不记录载荷。

) {
throw new Error('jev_invalid_response');
}
return Math.min(answer.confidence, probabilities[choice] as number) >= 0.82 ? choice : 'unclear';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This requires probabilities to cover exactly the criteria set and sum to 1±0.01 — a provider returning 0.999-total or an extra key throws → undefined → silent disablement. Real jev-1.13.0 responses satisfy it today, but a contract drift here turns into a permanent silent fallback with no signal (see the catch note below). Worth either a small tolerance relaxation or making the rejection diagnosable.

中文 这里要求 `probabilities` 精确覆盖 criteria 键集且总和 1±0.01——供应商返回 0.999 或多一个键就会抛错 → `undefined` → 静默禁用。当前真实 `jev-1.13.0` 响应满足该契约,但一旦契约漂移就变成无信号的永久静默回退(见下面 catch 那条)。值得放宽一点容忍度,或让拒绝可诊断。

import { createProxiedFetchTransport } from '@maka/runtime/network/scoped-fetch-transport';
import { toRuntimePolicyProxy } from './runtime-policy-proxy.js';

const INTENTS: Record<string, string> = {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

INTENTS re-declares core's intent taxonomy as unchecked keys, and line 82's choice as 'discuss'|'execute'|'create'|'continue' bypasses decodeWorkHubIntent — a new intent kind added in workhub-routing.ts would silently be unemittable here. Typing the map keys (or validating the constructed assessment) against the core union keeps the two in lockstep; the description strings themselves are legitimately local to the API's criteria format.

中文 `INTENTS` 把 core 的意图分类重新声明为未校验的键,且 :82 的 `choice as ...` cast 绕过了 `decodeWorkHubIntent`——`workhub-routing.ts` 日后新增意图类型时这里会静默无法发出。把 map 的键类型对齐 core union(或对构造出的 assessment 过一遍校验)能让两边同步;描述字符串本身作为 API 的 `criteria` 格式保留在本地是合理的。

},
en: {
advanced: 'Advanced settings', title: 'Jev assisted decisions',
help: 'Use TypeSafe Jev for WorkHub intent classification and work routing. Sends the current message and candidate work summaries. Conversation, execution, and titles keep their original models.',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The consent copy says "current message and candidate work summaries", but the Intent request also carries up to 8 recent WorkHub transcript messages (≤600 chars each — projectWorkHubIntentModelInput, workhub-routing.ts:62-76). For an opt-in egress feature the disclosure should name it, e.g. "…the current message, recent conversation, and candidate work summaries".

中文 同意文案只写了"当前消息与候选工作摘要",但 Intent 请求还会携带最多 8 条近期 WorkHub transcript 消息(每条 ≤600 字符——`projectWorkHubIntentModelInput`,workhub-routing.ts:62-76)。opt-in 外发功能的披露应当点明,例如"……当前消息、近期对话与候选工作摘要"。

const copy = JEV_COPY[locale];
const [key, setKey] = useState('');
const [saving, setSaving] = useState(false);
const pending = useRef(false);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Settings writers conventionally use useKeyedActionGuard/useActionGuard (settings/use-action-guard.ts; siblings at general-settings-page.tsx:518, web-search-settings-page.tsx:75) rather than a hand-rolled pending ref + mounted guard.

中文 settings 写入方的惯例是 `useKeyedActionGuard`/`useActionGuard`(`settings/use-action-guard.ts`;兄弟用法见 `general-settings-page.tsx:518`、`web-search-settings-page.tsx:75`),而不是手写 `pending` ref + mounted 守卫。

under the License.
-->

# Jev settings and routing comparison — PR #5562

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs/README.md keeps PR run logs and screenshots on the PR thread or in docs/archive/ — a committed docs/pr-NNNN/ directory is new precedent. The README + result JSONs can move to the PR body/attachments; compare-routing.mjs is the durable part and belongs under scripts/ if kept.

中文 `docs/README.md` 约定 PR 的 run log 与截图放 PR 线程或 `docs/archive/`——入库的 `docs/pr-NNNN/` 目录是新先例。README 与结果 JSON 可移到 PR 正文/附件;`compare-routing.mjs` 是其中耐久的部分,若保留应放 `scripts/`。

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 661053247. The rework is thorough — every finding from the last round is fixed, and several fixes went beyond the letter of the comments.

Verified fixes:

Prior finding Status
P1 missing compat-epoch bump Fixed — epoch 178 with a documented reason and a pin test (protocol.test.ts)
P2 operation:'action' admission throw Fixed — excluded at root-turn-coordinator.ts:1568, and prepareFreshWorkHubExecution now runs after runtimeHostExecutionUnavailableReason, so recovery/unavailable turns no longer pay the Jev call
P2 incognito mid-flight Fixed — ask() re-reads the policy snapshot before each provider request; covered by a mid-flight mutation test
P2 consent copy Fixed — now discloses "recent conversation" alongside message and candidates
P2 committed evidence Fixed — evaluation moved to docs/archive/jev-routing-evaluation-2026-09-22/ (correct archive location) and the harness to scripts/compare-jev-routing.mjs. One correction to my earlier comment: .maka-shots/ is actually the sanctioned committed location for PR screenshots (the docs/README.md rule only covers docs/); keeping them is a convention choice, not a violation
P3 INTENTS re-declared taxonomy + unchecked cast Fixed — IntentChoice is derived from WorkHubIntentAssessment and runs through decodeWorkHubIntent
P3 silent permanent failure Fixed — reportFailure emits fixed tags; nothing sensitive crosses the boundary
P3 unbounded pre-flight store reads Fixed — all reads race the signal via readDuringBackendCreation
P3 hand-rolled save guard Fixed — useActionGuard
P3 inline JEV_COPY Fixed — moved to locales/settings-jev-copy.ts with a UiCatalog shape check

Remaining findings — all P3, none blocking. One inline comment on the timeout tag; the rest in brief:

  • The single 8s budget spans both asks plus resolveCandidates and transport close — a slow Intent starves Recall into undefined. Safe degradation either way; if a per-request budget was intended, move the timeout inside ask().
  • The injected-model path (dependencies.workHubRoutingModel) bypasses the gate entirely — no policy check, no signal wrap, and readMessagesAfter inside prepareRoutingDecision is unraced. That's the intended experiment seam today; worth one comment so a future production injection doesn't silently lose the hang protection.
  • Each ask() builds a fresh proxied transport — two TLS handshakes per decision, no keep-alive between Intent and Recall. Correct for mid-flight proxy changes; just a small per-turn cost.
  • response.json() SyntaxError and fetch TypeError classify as 'unavailable' rather than invalid_response/request_failed — tag granularity only.
  • Undocumented fail-asymmetry in workhub-coordination-coordinator.ts:1006-1010 (pre-existing, made reachable by this feature): any throw before or inside the routing model — transcript read failure, injected-model throw — returns a bound clarify decision persisted on the admission, which blocks all actFromTurn proposals for the turn's lifetime via routingDecisionAllowsProposal. A transient store hiccup at admission locks the turn into clarify-only, while a provider outage inside the model returns undefined → unbound → proposals unconstrained. Both directions are defensible (fail-closed vs degrade-to-legacy), but the asymmetry deserves a comment or an explicit choice.

Still open — product decisions, unchanged from last round: whether this ships as a user-facing opt-in (current shape) or stays an experiment-seam adapter (the injection point existed before this PR with zero policy/UI surface); whether jev.enabled earns its policy field given the UI already forbids enabled ∧ ¬key and clearing the key force-disables — note webSearch has the identical enabled+key split, so keeping is also the consistent answer; and whether the strict probabilities contract (exact key-set + sum 1±0.01) matches a TypeSafe guarantee — if not contractual, validating only the consumed fields would survive benign provider drift.

Nothing in this head blocks on correctness grounds. The remaining gate is the product question from the first review — worth settling in #3492 or a dedicated issue rather than in this diff.

中文

在 661053247 复审通过实现层面:上轮全部 P1/P2 修复并验证成立,且多处修法超出原评论(如把 availability 检查挪到 Jev 调用之前,顺带消除了恢复路径的外发顺序问题)。一处更正:.maka-shots/ 实际是 PR 截图的合规提交位置,docs/README.md 只禁止 docs/ 下的截图——archive 迁移后文档面已完全合规。

剩余均为 P3:单条 8s 预算覆盖两次请求+候选解析+连接关闭(慢 Intent 会饿死 Recall,安全降级);注入缝完全旁路门禁(实验用途属预期,值得一行注释防止未来生产注入失去挂起保护);每个 ask 新建 transport(两次 TLS 握手,代理中途变更下是正确取舍);json()/网络错误的标签归到 unavailable 颗粒度偏粗。另两条行内:生产接线下 timeout 标签实际不可达(组合层 8s 先到 → 走 preparation_unavailable 而非 timeout);transcript 读取失败绑死 clarify 与模型失败放行之间的失败不对称未文档化。

设计问题不变:ship 为 opt-in 功能 vs 留在实验缝(注入点本已存在);jev.enabled 字段是否值得(UI 已禁 enabled ∧ ¬key,清 key 强制关——但 webSearch 同样是 enabled+key 双轨,保留也自洽);probabilities 严格契约是否有 TypeSafe 保证背书。代码层面已无阻塞项,剩下的闸是产品决策——建议在 #3492 或专门 issue 定案,而不是在这个 diff 里。


Re-review generated with AI assistance (Devin); all fixes verified against head 661053247.

candidateSetId,
);
} catch (error) {
if (request.abortSignal.aborted) return undefined;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 — trigger ② (failure path): the timeout tag is unreachable through the production wiring. In production request.abortSignal is the coordinator's any([compositionSignal, 15s]), and the composition's 8s timer starts strictly earlier than this adapter's own 8s (:63) — so whenever a timeout fires, request.abortSignal.aborted is already true here and we return before reaching the classifier. The incident still gets logged, but under the composition's preparation_unavailable tag instead of timeout. The tag only fires on direct calls (tests/benchmark). If the distinction matters, classify on request.abortSignal.reason (a TimeoutError DOMException vs the caller's reason) rather than .aborted; otherwise this is just a mislabeled warn.

中文

timeout 标签在生产接线下不可达:组合层 8s 定时器总是先于本层触发,届时 request.abortSignal.aborted 已为真而提前返回——事件仍会以 preparation_unavailable 记录,只是标签不对。只有直调路径(测试/benchmark)能打出 timeout。要精确的话按 request.abortSignal.reason 分类,否则这只是 warn 标签误名。

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at e271bc794 — everything from both rounds is now resolved. The diagnostics-layering comments document the intent precisely (composition owns the shared deadline and its tags; the adapter's own 8s covers direct callers), the signal.aborted → 'timeout' fix at the composition catch makes the tag reachable, and the fail-asymmetry seam is now documented as deliberate. Strict-decoder + per-ask policy refresh + bounded reads all hold.

One housekeeping item: this now conflicts with main — protocol/index.ts (your 178 vs main's 181 after #5224) plus execution-composition.ts overlap. Rebase and re-bump to epoch ≥182.

On the earlier design discussion: the opt-in shape, per-ask consent checks, and fail-open fallback are a reasonable interpretation of the experiment seam for a shipped feature; whether WorkHub wants a provider-specific route long-term stays a tracker-level call, not a blocker for this diff.

中文

e271bc794 复审通过:两轮 findings 全部修复或已确认设计意图——组合层 catch 现在能正确打出 timeout 标签,诊断分层、注入缝旁路、共享预算、失败不对称都以注释写明意图。剩余事项:与 main 冲突(epoch 178 vs main 181,加 composition 重叠),rebase 时重提到 ≥182。设计层的"是否长期留供应商特定路由"归 tracker 讨论,不阻塞本 diff。


Re-review generated with AI assistance (Devin); verified against head e271bc794.

@ARE404
ARE404 merged commit e6e35ae into apache:main Sep 23, 2026
1 check passed
@ARE404
ARE404 deleted the feat/jev-routing branch September 23, 2026 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/XXL Over 2500 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants