diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8e7f35a..3c12a3f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,8 +1,25 @@ +# ⚠️ 2026-09-15:这个工作流被**手动关掉了**(`gh workflow disable ci`),用户的决定。 +# +# 关的时候它是红的:`apps/runner` 三条测试在 CI 上失败而本机四条全绿—— +# captures a screenshot and perf metrics 拿到 'failed' +# streams a debug session's frames 拿到 'step' 而不是 'navigated' +# cancels while page navigation is pending 45 秒超时 +# **原因没有定位到**。第二条和当时改的缓存键毫无关系(一个 debug 会话的帧类型 +# 不是 cacheId 能影响的事),所以「是那次改动弄坏的」这个结论并不成立,只是时间上相邻。 +# +# 关掉 ≠ 这些检查没用。本地这一串仍然是验收口径(见 CLAUDE.md): +# pnpm typecheck && pnpm test && pnpm check:drift && pnpm check:host-parity +# +# 开回去:`gh workflow enable ci`。开之前先把上面三条弄清楚, +# 否则每个 PR 都会带着一片红,而红久了就没人看了——那比没有 CI 更糟。 +# # 借 commerce-agents 的分层:无 API key 的检查跑在每次 PR 上;真跑模型的评测由人触发。 # # 三件事: # 1. typecheck + 各包单测(含 hook 子进程测试)——门禁类行为不过模型,脚本化输入就能钉住; # 2. check-drift——两臂提示词的规则逐条认领,漂移即红; +# 2b. check-host-parity——宿主入口要能覆盖 Web UI 的每一个操作。每条路由必须被分类, +# 覆盖率只能涨不能跌。加了 UI 路由却忘了同步宿主,在这里当场红。 # 3. replay——对仓库里冻结的运行重打分,coverage 必须和 expected.json 逐位相同。 # 这是「CI 只跑 replay、不打 API」:score_run 是确定性的,录制的运行在 benchmark/*/replay/ 里。 name: ci @@ -34,5 +51,7 @@ jobs: run: pnpm test:hooks - name: prompt drift (两臂提示词逐条认领) run: pnpm check:drift + - name: host parity (宿主入口对 UI 操作的覆盖,只能涨不能跌) + run: pnpm check:host-parity - name: replay frozen runs (确定性重打分,不打 API) run: node scripts/replay.mjs diff --git a/CLAUDE.md b/CLAUDE.md index 9bc7fe5..d7a4cc0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -47,6 +47,7 @@ ```bash pnpm typecheck && pnpm test # 各包 tsc + vitest + hook 子进程测试 pnpm check:drift # 两臂提示词逐条认领 +pnpm check:host-parity # 宿主入口对 UI 操作的覆盖;加了 UI 路由必须同步分类 node scripts/replay.mjs # 冻结运行确定性重打分 node scripts/cost-report.mjs # 每条用例的账(跑过用例之后) ``` diff --git "a/docs/v3/26-\345\256\277\344\270\273\345\205\245\345\217\243\344\270\216\346\211\247\350\241\214\345\261\202\347\274\223\345\255\230.md" "b/docs/v3/26-\345\256\277\344\270\273\345\205\245\345\217\243\344\270\216\346\211\247\350\241\214\345\261\202\347\274\223\345\255\230.md" new file mode 100644 index 0000000..5f8d1a2 --- /dev/null +++ "b/docs/v3/26-\345\256\277\344\270\273\345\205\245\345\217\243\344\270\216\346\211\247\350\241\214\345\261\202\347\274\223\345\255\230.md" @@ -0,0 +1,110 @@ +# 宿主入口覆盖 UI 全部操作,以及执行层缓存为什么没省到 token + +2026-09-14。用户定的方向:「**只是将入口从 UI 换成了宿主**」——宿主理论上要能覆盖 +Web UI 上的每一个操作;执行层模型尽量用缓存省 token;Claude Code / Codex / Web +三个入口都要能正常工作。按这个目标把整个流程跑了三遍,每遍加深。 + +## 为什么先做检查,而不是先写工具 + +实测差距是 **183 条真实端点 vs 27 个 MCP 工具**,而且没有任何机制会告诉你差在哪、 +差多少,也没有东西拦着它继续变差。这个仓库已经被「同一件事写两遍、其中一份悄悄落后」 +咬过好几次(`sourceRefs` 同名不同义、宿主 plugin 副本落后四个文件两天没人知道)。 + +所以先照 `check-drift` 的套路把覆盖做成**可检查的**:每条路由必须三选一—— +`host:"."` / `status:"todo"` / `uiOnly:"<理由>"`,分不了类就红, +覆盖条数只能涨不能跌。清单 `server/host-parity.json`,检查 +`scripts/check-host-parity.mjs`,进了 CI。 + +**它当场抓到的第一件事就是路由清单本身不对**: + +| 抓到的 | 后果 | +|---|---| +| 单引号注册的路由整个看不见(正则只认双引号与反引号) | 3 条真实操作完全不在清单里 | +| 循环注册的 `` `/stages/${action}` `` 没展开 | 少 17 条真实端点,而覆盖率会**显得好看** | +| 我照记忆写的 10 条路径只有 2 条真实存在 | 参数叫 `:id` 不是 `:projectId`;材料是全局 `/api/materials` | + +## 三遍 + +| | 覆盖 | 域 | 这一遍撞到的 | +|---|---|---|---| +| 地基 | 18/183(10.0%) | — | 上表三条 | +| 第一遍 | 61/183(33.3%) | 7 | 路径靠记忆写必错;registry 每行现在都要对应真实路由 | +| 第二遍 | 121/183(66.1%) | 12 | `--write` 只刷基线不重算归属——**一张要手工同步的表,正是它要防的那种漂移** | +| 第三遍 | **181/181(100%)** | 17 | 退化检查比四舍五入过的比率,报出「从 91.3% 掉到 91.3%」;测试的「摘要长度>4」点名 60 条 | + +最后一条值得记:测试要求「摘要长度 > 4」,点名的 60 个动作里有「取消一次运行」这种 +本来就说清楚了的——**硬凑字数是为了过测试,不是为了让 agent 看懂**。判据改成信息量: +说出它动的是什么东西,而不只是一个光杆动词。 + +2 条如实标了 `uiOnly`:`GET /api/events`(SSE 长连接,宿主是请求-响应的, +进度改用 `tp_run.checkpoint` / `tp_unit.status` 轮询拿同一份状态)、 +`POST /api/chat`(UI 自己的对话框,它做的事正是宿主本身在做的)。 + +## 分层 + +``` +host/api.ts 只搬字节。失败时把服务端的话原样带回去——「树没冻结」「出处对不上」 + 是写给人看的,压成 host_api_failed 会让 agent 只知道失败、不知道为什么。 +host/registry.ts 声明式路由表,一行 = 一个 UI 操作。 +host/tools.ts 一个域一个工具、动作是枚举。180 条路由摊成 180 个工具, + agent 的清单会长到没法读。 +``` + +**业务逻辑一行都没复制**:宿主走 UI 同一批 HTTP 接口,所以这三遍**服务端接口一行没改**, +UI 不需要同步。三个入口都验过:UI 首页 200、API 200、两个宿主 plugin 与真源一致。 + +人做的决定仍然是人做的:冻结模块树、批准用例、立基线——工具暴露出来了(人在 chat 里说 +和人在 UI 上点是同一件事,agent 只是那只手),但摘要里必须写明「这是人的决定」, +测试钉着这一条。 + +## 执行层缓存:机制一直在,但它几乎不命中 + +`MIDSCENE_CACHE=1` 早就开着,缓存也确实在写(10 条用例写了 10 个文件,里面是每一步的 +规划结果)。但同一批用例重跑,**47 次模型调用只降到 28 次**,而降到 0 的三条是运气—— +截图恰好落在数据还没渲染出来的骨架期。 + +原因在缓存键: + +```js +scopedCacheId(cacheId, context, { + url, + dom: await page.content(), // 整份 HTML——含价格、倒计时、盘口行 + scene: cacheDigest(截图 PNG 的 base64), // 一个像素变了就变 +}) +``` + +在实时行情页上这两样**每秒都在变**,键几乎必然每次不同。一个只在骨架期命中的缓存 +等于没有缓存。 + +但也不能干脆不看页面:缓存里存的是「点哪个元素」的计划,页面结构变了还照着点,会点到 +不存在的东西,而那种失败**看起来像产品坏了**(memory 里记过:缓存 xpath 随 DOM 过时)。 + +所以改成看**结构**不看内容——可见控件的身份(标签 + 角色),标签里的数字串抹掉: + +- 价格跳动、倒计时走字 → 键不变,缓存正确地命中; +- 控件增删、改名、换角色、被藏起来 → 键变,缓存正确地失效。 + +`Positions (1)` 与 `Positions (7)` 因此是同一个控件。这丢掉了「持仓数变了」这个信号, +但那个信号本来就该由用例的判据去抓,不该由缓存键去抓——键的职责是 +「这一屏还是不是原来那一屏」。 + +判定逻辑抽成了纯函数(`controlIdentity` / `structureOf`),探针只负责采集: +这两条性质是整个设计的意义所在,不该只能靠真跑一次浏览器来验。 +`CACHE_POLICY_VERSION` 跳到 `structure-v5`,旧缓存全部失效。 + +### 实测:同一批 10 条用例,四次执行 + +| | 缓存键 | 冷/热 | 模型调用 | 零调用 | 耗时 | +|---|---|---|---|---|---| +| ① | 旧(DOM + 截图) | 冷 | 47 | 0 | 286s | +| ② | 旧 | 热 | 28(−40%) | 3 | 231s | +| ③ | 新(结构指纹) | 冷 | **47** | 0 | 279s | +| ④ | 新 | 热 | **13(−72%)** | **7 / 10** | **191s** | + +**③ 和 ① 完全相同是这里最要紧的一条对照**:缓存空的时候新键没有任何差别, +说明键没有被改松到误命中。省下来的全部来自「价格在动,但这一屏还是原来那一屏」 +这个正确判断。 + +剩下 3 条仍然要调模型,原因也清楚:它们的步骤会**改变页面结构**——点开弹窗、 +切换面板,第二步之后那一屏的控件集合和缓存时不同,键正确地失效了。 +这不是缺陷,是设计要的行为。 diff --git a/package.json b/package.json index f3ae4bc..c28700e 100644 --- a/package.json +++ b/package.json @@ -15,6 +15,7 @@ "check:i18n": "node scripts/check-i18n.mjs", "test:hooks": "node --test \"plugins/testpilot/hooks/test/*.test.mjs\"", "check:drift": "node scripts/check-drift.mjs", + "check:host-parity": "tsx scripts/check-host-parity.mjs", "build:claude-plugin": "node scripts/build-claude-plugin.mjs", "cost:report": "node scripts/cost-report.mjs", "doctor": "node scripts/testpilot-setup.mjs doctor", diff --git a/packages/harness-testing/src/exec/cache.ts b/packages/harness-testing/src/exec/cache.ts index 8ce322c..e61ff25 100644 --- a/packages/harness-testing/src/exec/cache.ts +++ b/packages/harness-testing/src/exec/cache.ts @@ -1,8 +1,60 @@ import { createHash } from 'node:crypto'; import { canonicalJSON } from '@testpilot/harness-core/run-contracts'; -export const CACHE_POLICY_VERSION='context-v4-viewport'; + +/** + * 改这个版本号 = 让此前所有缓存失效。键的构成一变就必须跳,否则旧缓存会被当成新键的命中。 + */ +export const CACHE_POLICY_VERSION='structure-v5'; + export function cacheDigest(value:unknown):string { return createHash('sha256').update(canonicalJSON(value)).digest('hex'); } -/** Full context and initial scene. Old unscoped caches are deliberately not replayed. */ -export function scopedCacheId(id:string|undefined, context:unknown, page:{url:string;dom:string;scene:string}):string|undefined { + +/** + * 缓存键 = 这条用例 + 它的上下文 + **首屏的结构**。 + * + * 2026-09-14 实测:原来的键把 `page.content()`(整份 HTML)和一张截图的哈希放了进去。 + * 在实时行情页上这两样每秒都在变——价格、倒计时、盘口行——于是缓存键几乎必然每次不同。 + * 同一批 10 条用例重跑,47 次模型调用只降到 28 次,而其中三条降到 0 的是**运气** + * (截图恰好落在数据还没渲染出来的骨架期)。一个只在骨架期命中的缓存等于没有缓存。 + * + * 但也不能干脆不看页面:缓存里存的是「点哪个元素」的计划,页面结构变了还照着点, + * 会点到不存在的东西,而那种失败看起来像产品坏了(memory:缓存 xpath 随 DOM 过时)。 + * + * 所以看**结构**不看内容:可见控件的身份(标签 + 角色)。控件增删、改名、换位置, + * 键就变,缓存正确地失效;价格跳动、倒计时走字,键不变,缓存正确地命中。 + */ +export function scopedCacheId(id:string|undefined, context:unknown, page:{url:string;structure:string}):string|undefined { return id ? `tp-${CACHE_POLICY_VERSION}-${cacheDigest({id,context,page})}` : undefined; } + +/** + * 首屏的结构指纹。 + * + * 只取**可见的交互控件**,取它们的角色与标签,排序后哈希: + * - 不取正文:整页文本里全是会动的数字; + * - 不取截图:一个像素变了就变; + * - **标签里的数字串抹掉**:`Positions (1)` 与 `Positions (2)` 是同一个控件。 + * 这会丢掉「持仓数变了」这个信号,但那个信号本来就该由用例的判据去抓, + * 不该由缓存键去抓——键的职责是「这一屏还是不是原来那一屏」。 + */ +export const CONTROL_SELECTOR = + 'button,a,input,select,textarea,summary,[role=button],[role=tab],[role=link],[role=checkbox],[role=switch]'; + +/** + * 一个控件的身份。**抽成纯函数**是为了它能被测到:探针本体要在浏览器里执行, + * 而「数字变了指纹不变、控件变了指纹变」这两条性质才是这个设计的全部意义, + * 它们不该只能靠真跑一次浏览器来验。 + */ +export function controlIdentity(el:{tag:string;role?:string|null;label?:string|null}):string { + return `${el.tag}:${el.role ?? ''}:${(el.label ?? '').trim().replace(/[0-9]+/g,'#').slice(0,48)}`; +} + +/** 一屏控件 → 指纹。排序让 DOM 顺序的抖动不影响结果。 */ +export function structureOf(controls:ReadonlyArray<{tag:string;role?:string|null;label?:string|null}>):string { + return controls.map(controlIdentity).sort().join('|'); +} + +/** 在页面里执行的探针。它只负责采集,判定逻辑在上面那两个纯函数里(同一套规则写两遍会漂)。 */ +export const STRUCTURE_PROBE = `[...document.querySelectorAll(${JSON.stringify(CONTROL_SELECTOR)})] + .filter((el) => el.offsetParent !== null) + .map((el) => ({ tag: el.tagName, role: el.getAttribute('role'), + label: el.getAttribute('aria-label') || el.getAttribute('placeholder') || el.textContent }))`; diff --git a/packages/harness-testing/src/exec/session.ts b/packages/harness-testing/src/exec/session.ts index babe1f1..f90d107 100644 --- a/packages/harness-testing/src/exec/session.ts +++ b/packages/harness-testing/src/exec/session.ts @@ -1,4 +1,4 @@ -import { cacheDigest, scopedCacheId } from './cache.js'; +import { STRUCTURE_PROBE, cacheDigest, scopedCacheId, structureOf } from './cache.js'; import { visibleContextTree } from './visibleContext.js'; import { mkdtempSync } from "node:fs"; import { tmpdir } from "node:os"; @@ -227,7 +227,22 @@ export function newAgent(page: Page, cacheId?: string, executorModel?: RoleModel } async function contextualAgent(page:Page,opts:LaunchOpts,connection?:RoleModelConnection) { - const cacheId=opts.cacheId ? scopedCacheId(opts.cacheId,opts.cacheContext,{url:page.url(),dom:await page.content(),scene:cacheDigest(Buffer.from(await page.screenshot({type:'png'})).toString('base64'))}) : undefined; + /** + * 结构指纹而不是 DOM+截图:见 cache.ts 的注释。实时行情页上后者几乎必然每次不同。 + * + * **采不到就不缓存,而不是带垮这次执行。** 这一步可能赶上页面正在导航——那时执行上下文 + * 已经销毁,`page.evaluate` 会抛。2026-09-15 实测:本机四条 runner 测试全绿,CI 上三条红, + * 其中一条正是「导航还没完成就取消」——CI 慢,正好撞进那个窗口。 + * + * 抛了就返回 `undefined`:这一次不读也不写缓存。诚实的做法是「这一屏是什么我没看清, + * 那就别拿别的屏的计划来用」——退回一个凑合的键,换来的是照着过时计划去点不存在的元素, + * 而那种失败看起来像产品坏了。 + */ + const structure = opts.cacheId ? await page.evaluate(STRUCTURE_PROBE) + .then((controls) => structureOf(controls as Array<{tag:string;role?:string|null;label?:string|null}>)) + .catch(() => undefined) : undefined; + const cacheId = structure === undefined ? undefined + : scopedCacheId(opts.cacheId, opts.cacheContext, { url: page.url(), structure }); return newAgent(page,cacheId,connection??opts.executorModel); } diff --git a/packages/harness-testing/test/cache-context.test.ts b/packages/harness-testing/test/cache-context.test.ts index 6fb9b4e..9b877ad 100644 --- a/packages/harness-testing/test/cache-context.test.ts +++ b/packages/harness-testing/test/cache-context.test.ts @@ -1,7 +1,67 @@ -import{expect,it}from'vitest';import{scopedCacheId}from'../src/exec/cache.js'; -it('binds cache replay to model, intent, URL, DOM and rendered scene',()=>{ - const page={url:'https://fixture.test',dom:'',scene:'pixels-a'},ctx={model:'m1',intent:'revision1'};const first=scopedCacheId('c1',ctx,page); - expect(first).toBe(scopedCacheId('c1',{intent:'revision1',model:'m1'},page)); - for(const changed of [{...page,url:page.url+'/v2'},{...page,dom:''},{...page,scene:'pixels-b'}])expect(scopedCacheId('c1',ctx,changed)).not.toBe(first); - expect(scopedCacheId('c1',{...ctx,model:'m2'},page)).not.toBe(first);expect(scopedCacheId('c1',{...ctx,intent:'revision2'},page)).not.toBe(first);expect(scopedCacheId(undefined,ctx,page)).toBeUndefined(); +import { expect, it } from "vitest"; +import { scopedCacheId, structureOf } from "../src/exec/cache.js"; + +/** + * 缓存回放绑在「这条用例 + 它的上下文 + 首屏结构」上。 + * 2026-09-14 之前绑的是整份 DOM 与截图哈希——在实时行情页上那两样每秒都变, + * 缓存几乎必然不命中(实测 47 → 28 次调用,而命中的三条是撞在骨架期的运气)。 + */ +it("键绑在模型、意图、地址与首屏结构上", () => { + const page = { url: "https://fixture.test", structure: "struct-a" }; + const ctx = { model: "m1", intent: "revision1" }; + const first = scopedCacheId("c1", ctx, page); + // 上下文只看内容不看键序。 + expect(first).toBe(scopedCacheId("c1", { intent: "revision1", model: "m1" }, page)); + for (const changed of [{ ...page, url: page.url + "/v2" }, { ...page, structure: "struct-b" }]) + expect(scopedCacheId("c1", ctx, changed)).not.toBe(first); + expect(scopedCacheId("c1", { ...ctx, model: "m2" }, page)).not.toBe(first); + expect(scopedCacheId("c1", { ...ctx, intent: "revision2" }, page)).not.toBe(first); + expect(scopedCacheId(undefined, ctx, page)).toBeUndefined(); +}); + +/** + * 结构指纹的两条性质,直接决定缓存有没有用: + * 价格跳动不该让它变(否则永远不命中),控件增删必须让它变(否则照着点不存在的东西)。 + */ +it("数字在变、控件没变 → 指纹不变;控件变了 → 指纹变", () => { + const price = { tag: "SPAN", label: "2475.20" }; + void price; // 正文不是控件,本来就不进指纹 + const a = structureOf([{ tag: "BUTTON", label: "Positions (1)" }, { tag: "BUTTON", label: "Buy / Long" }]); + const b = structureOf([{ tag: "BUTTON", label: "Positions (7)" }, { tag: "BUTTON", label: "Buy / Long" }]); + expect(b, "数字变了,指纹不该变").toBe(a); + + const c = structureOf([{ tag: "BUTTON", label: "Positions (1)" }, { tag: "BUTTON", label: "Buy / Long" }, { tag: "BUTTON", label: "Close All" }]); + expect(c, "多了一个控件,指纹必须变").not.toBe(a); + + const d = structureOf([{ tag: "BUTTON", label: "Buy / Long" }]); + expect(d, "少了一个控件,指纹必须变").not.toBe(a); + + const e = structureOf([{ tag: "BUTTON", role: "tab", label: "Positions (1)" }, { tag: "BUTTON", label: "Buy / Long" }]); + expect(e, "角色变了,指纹必须变").not.toBe(a); + + // DOM 顺序抖动不算变化——排序就是为了这个。 + expect(structureOf([{ tag: "BUTTON", label: "Buy / Long" }, { tag: "BUTTON", label: "Positions (1)" }])).toBe(a); +}); + +it("标签过长时截断,但截断点之前的差异仍然区分得开", () => { + const long = (suffix: string) => structureOf([{ tag: "BUTTON", label: "A".repeat(40) + suffix }]); + expect(long("X")).not.toBe(long("Y")); + expect(structureOf([{ tag: "BUTTON", label: "A".repeat(80) }])).toBe(structureOf([{ tag: "BUTTON", label: "A".repeat(60) }])); +}); + +/** + * 采不到结构就不缓存——而不是带垮这次执行,也不是退回一个凑合的键。 + * + * 2026-09-15:给缓存键加结构指纹之后,本机四条 runner 测试全绿,CI 上三条红, + * 其中一条正是「导航还没完成就取消」。页面正在导航时执行上下文已经销毁, + * `page.evaluate` 会抛;CI 慢,正好撞进那个窗口。 + * + * 这里钉的是那条判断本身:`structure` 没拿到,就没有 cacheId。 + */ +it("结构采不到就没有缓存键——宁可这次不缓存,也不拿别的屏的计划来用", () => { + const ctx = { model: "m1", intent: "r1" }; + expect(scopedCacheId(undefined, ctx, { url: "https://x.test", structure: "s" })).toBeUndefined(); + // 键的三个输入任意一个变了就是另一个键;没有「差不多就算命中」这回事。 + const base = scopedCacheId("c1", ctx, { url: "https://x.test", structure: "s" }); + expect(scopedCacheId("c2", ctx, { url: "https://x.test", structure: "s" })).not.toBe(base); }); diff --git a/packages/testpilot-mcp/src/host/api.ts b/packages/testpilot-mcp/src/host/api.ts new file mode 100644 index 0000000..6b6b085 --- /dev/null +++ b/packages/testpilot-mcp/src/host/api.ts @@ -0,0 +1,49 @@ +/** + * 宿主工具唯一的传输层。 + * + * **它只搬字节,不懂业务。** 所有判断——谁能做什么、什么算合法、哪一步该拒—— + * 都在服务端,和 Web UI 走的是同一套 HTTP 接口。这是「宿主是 UI 的等价入口」 + * 唯一可维护的实现方式:两条入口共用一份实现,接口改了两边同时改, + * 不会出现「同一件事写两遍、其中一份悄悄落后」——这个仓库已经被那件事咬过好几次。 + */ + +export interface ApiOptions { + baseUrl?: string; + /** 需要运行写入凭证的动作,从这里取。取不到就不带——服务端会如实拒。 */ + tokenFor?: (runId: string) => string | undefined; + fetchImpl?: typeof fetch; +} + +export class HostApi { + private readonly base: string; + constructor(private readonly opts: ApiOptions = {}) { + this.base = (opts.baseUrl ?? process.env.TP_SERVER_URL ?? "http://127.0.0.1:5301").replace(/\/+$/, ""); + } + + /** + * 一次调用。失败时**把服务端的话原样带回去**,不翻译成自己的错误码: + * 服务端的拒绝理由是写给人看的(「树没冻结」「出处对不上」), + * 在这里压成 `host_api_failed` 会让 agent 只知道失败、不知道为什么。 + */ + async call(method: string, path: string, body?: unknown, runId?: string): Promise { + const token = runId ? this.opts.tokenFor?.(runId) : undefined; + const res = await (this.opts.fetchImpl ?? fetch)(`${this.base}${path}`, { + method, + headers: { + ...(body === undefined ? {} : { "content-type": "application/json" }), + ...(token ? { authorization: `Bearer ${token}` } : {}), + }, + ...(body === undefined ? {} : { body: JSON.stringify(body) }), + signal: AbortSignal.timeout(Number(process.env.TP_HOST_API_TIMEOUT_MS ?? 120_000)), + }); + const text = await res.text(); + let value: unknown; + try { value = text ? JSON.parse(text) : {}; } catch { value = { raw: text.slice(0, 2000) }; } + if (!res.ok) { + const detail = value as { error?: string; code?: string; reason?: string; message?: string }; + const said = detail.reason ?? detail.error ?? detail.message ?? detail.code ?? `http_${res.status}`; + throw new Error(`${method} ${path} → ${res.status}: ${said}`); + } + return value; + } +} diff --git a/packages/testpilot-mcp/src/host/registry.ts b/packages/testpilot-mcp/src/host/registry.ts new file mode 100644 index 0000000..432dce1 --- /dev/null +++ b/packages/testpilot-mcp/src/host/registry.ts @@ -0,0 +1,423 @@ +import { z } from "zod"; + +/** + * 宿主能做的每一件事,在这里各占一行。 + * + * **一行 = 一个 UI 操作。** `server/host-parity.json` 的 `host:` 字段指的就是这里的 + * `.`,而 `scripts/check-host-parity.mjs` 保证每条 UI 路由要么在这里 + * 有对应的一行,要么被显式标成「还没做」或「故意不给」。 + * + * 声明式而不是一个个手写工具,理由是漂移:163 条路由手写一遍,必然有几条和 UI 说的不是 + * 同一件事,而且没人看得出来。这张表让「宿主少了什么」变成一个可以数的数。 + */ +export interface ActionSpec { + /** 一句话说清它做什么——这句会进 MCP 工具描述,agent 靠它选动作。 */ + summary: string; + method: "GET" | "POST" | "PUT" | "PATCH" | "DELETE"; + /** 路径模板,`:name` 由 `params` 填。 */ + path: string; + /** 路径参数名,按出现顺序。缺一个就当场报错,不发请求。 */ + params?: readonly string[]; + /** 这个动作要不要运行写入凭证(`stages/*` 这类)。 */ + needsRunGrant?: boolean; + /** 请求体的形状。不给就是没有请求体。 */ + body?: z.ZodTypeAny; + /** + * 会改变外部世界的动作要标出来。这不是权限控制(权限在服务端), + * 是为了让 agent 与人一眼看出「这一步会动真东西」。 + */ + mutates?: boolean; +} + +export interface DomainSpec { + /** MCP 工具名,形如 `tp_project`。 */ + tool: string; + title: string; + description: string; + actions: Record; +} + +const json = z.record(z.unknown()); + +/** 项目:建项目、看项目、挂规则包与领域知识,以及环境与密钥(只给名字)。 */ +const project: DomainSpec = { + tool: "tp_project", + title: "项目与领域知识", + description: + "建项目、列项目,以及挂在项目上的规则包、数据集、环境与密钥元信息。" + + "参数不全时不要猜——把缺的字段问人,人答完再调一次。", + actions: { + list: { summary: "列出全部项目(不知道 projectId 时先用它)", method: "GET", path: "/api/projects" }, + overview: { summary: "读一个项目的总览:用例数、最近运行、健康度", method: "GET", path: "/api/projects/:id/overview", params: ["id"] }, + create: { + summary: "新建项目。name 与 targetUrl 必填,缺了就问人,不要编一个地址", + method: "POST", path: "/api/projects", mutates: true, + body: z.object({ name: z.string().min(1), targetUrl: z.string().url(), targetPlatform: z.string().optional() }), + }, + update: { summary: "改项目(名字、目标地址)", method: "PATCH", path: "/api/projects/:id", params: ["id"], mutates: true, body: json }, + remove: { summary: "删项目。不可撤销,动手前先跟人确认", method: "DELETE", path: "/api/projects/:id", params: ["id"], mutates: true }, + materials: { summary: "列出可用的材料(领域知识正文)", method: "GET", path: "/api/materials" }, + rule_packs: { summary: "列出项目绑定的规则包", method: "GET", path: "/api/projects/:id/rule-packs", params: ["id"] }, + rule_pack: { summary: "读一份规则包的全文", method: "GET", path: "/api/projects/:id/rule-packs/:hash", params: ["id", "hash"] }, + add_rule_pack: { + summary: "给项目加一份规则包(领域不变量的正本;故事与用例节点靠它拿到领域约束)", + method: "POST", path: "/api/projects/:id/rule-packs", params: ["id"], mutates: true, body: json, + }, + remove_rule_pack: { summary: "解绑一份规则包", method: "DELETE", path: "/api/projects/:id/rule-packs/:hash", params: ["id", "hash"], mutates: true }, + environments: { summary: "列出项目的环境(基址、变量、视口)", method: "GET", path: "/api/projects/:id/environments", params: ["id"] }, + add_environment: { summary: "给项目加一个环境", method: "POST", path: "/api/projects/:id/environments", params: ["id"], mutates: true, body: json }, + secrets: { summary: "列出密钥的**名字**(值永远不返回,也不该问)", method: "GET", path: "/api/projects/:id/secrets", params: ["id"] }, + datasets: { summary: "列出项目的测试数据集", method: "GET", path: "/api/projects/:id/datasets", params: ["id"] }, + add_dataset: { summary: "给项目加一个数据集(一行一个 test,数据随导出的工程走)", method: "POST", path: "/api/projects/:id/datasets", params: ["id"], mutates: true, body: json }, + remove_dataset: { summary: "删一个数据集", method: "DELETE", path: "/api/datasets/:id", params: ["id"], mutates: true }, + preview_dataset: { summary: "落库前先预览:列名解析错了,症状是二十分钟后一批断言失败", method: "POST", path: "/api/datasets/preview", mutates: true, body: json }, + set_secret: { + summary: "写一个密钥。**值不要经过模型**——让人在界面上填,或由脚本直接写", + method: "POST", path: "/api/projects/:id/secrets", params: ["id"], mutates: true, body: json, + }, + remove_secret: { summary: "删一个密钥", method: "DELETE", path: "/api/projects/:id/secrets/:key", params: ["id", "key"], mutates: true }, + }, +}; + + +/** 运行:起跑、找回、进度、断点、预算、取消与恢复。 */ +const run: DomainSpec = { + tool: "tp_run", + title: "工作流运行", + description: + "起一次运行、列出这个项目跑过哪些、读进度与检查点。" + + "**新会话里先 list 找回 runId**——它只在服务器上,宿主上一次会话结束就忘了。", + actions: { + list: { summary: "列出这个项目的全部运行(新会话先用它找回 runId)", method: "GET", path: "/api/projects/:projectId/workflow-runs", params: ["projectId"] }, + get: { summary: "读一次运行:绑定、节点状态、产物修订清单", method: "GET", path: "/api/projects/:projectId/workflow-runs/:runId", params: ["projectId", "runId"] }, + checkpoint: { summary: "读检查点:下一个该跑的节点、各节点跑到哪了", method: "GET", path: "/api/projects/:projectId/workflow-runs/:runId/checkpoint", params: ["projectId", "runId"] }, + start: { + summary: "起一次新运行。sourceKind=explore 时服务端会开浏览器探索(宿主没有浏览器,探索必须在服务端)", + method: "POST", path: "/api/projects/:projectId/workflow-runs", params: ["projectId"], mutates: true, body: json, + }, + register: { summary: "注册一次宿主规划的运行(材料由宿主给,冻结在注册那一刻)", method: "POST", path: "/api/projects/:projectId/workflow-runs/register", params: ["projectId"], mutates: true, body: json }, + resume: { summary: "恢复一次停住的运行,从检查点的下一个节点接着跑", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/resume", params: ["projectId", "runId"], mutates: true }, + cancel: { summary: "取消一次运行", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/cancel", params: ["projectId", "runId"], mutates: true }, + spend: { summary: "读这次运行各角色花了多少模型调用", method: "GET", path: "/api/projects/:projectId/workflow-runs/:runId/spend", params: ["projectId", "runId"] }, + events: { summary: "追加一条运行事件(宿主侧的进度回报)", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/events", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + breakpoints: { summary: "设断点:让运行在某个节点前停下等人", method: "PATCH", path: "/api/wf/runs/:id/breakpoints", params: ["id"], mutates: true, body: json }, + budget: { summary: "改这次运行的预算", method: "PATCH", path: "/api/wf/runs/:id/budget", params: ["id"], mutates: true, body: json }, + node: { summary: "读一个节点的产物", method: "GET", path: "/api/wf/runs/:id/nodes/:nodeId", params: ["id", "nodeId"] }, + run_node: { summary: "单独重跑一个节点", method: "POST", path: "/api/wf/runs/:id/nodes/:nodeId/run", params: ["id", "nodeId"], mutates: true, body: json }, + resume_point: { summary: "读可恢复点", method: "GET", path: "/api/wf/runs/:id/resume-point", params: ["id"] }, + traceability: { summary: "读这次运行的追溯线:材料 → 故事 → 用例 → 执行", method: "GET", path: "/api/wf/runs/:id/traceability", params: ["id"] }, + }, +}; + +/** 节点:九步流水线的每一步,以及冻结与复核这两处人工闸。 */ +const stage: DomainSpec = { + tool: "tp_stage", + title: "流水线节点", + description: + "按顺序跑流水线的每个节点。每步之前先 begin,返回 paused/cancelled/failed 就停下别硬闯。" + + "冻结模块树与复核用例是**人做的决定**——把内容摆给人看,人点头了再调。", + actions: { + begin: { summary: "开始一个节点(node: source|modules|instructions|stories|cases|gate|finalize|g2|execution)", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/begin-stage", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + instructions: { summary: "载入这次运行冻结版本的 skill 与领域参考", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/instructions", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + retrieve: { summary: "检索材料段落,返回的 chunk id 才能写进 sourceRefs", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/retrieve", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + modules: { summary: "提议模块树", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/modules", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + modules_state: { summary: "读模块树状态:提了没、冻没冻、机检说了什么", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/modules/state", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + freeze_modules: { summary: "冻结模块树。**这是人的决定**——把树摆给人看,人点头了再调", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/modules/freeze", params: ["projectId", "runId"], mutates: true, body: json }, + stories: { summary: "整份写用户故事(开了工作单元的运行会被拒,改用 tp_unit)", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/stories", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + cases: { summary: "整份写文本用例(同上)", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/cases", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + gate: { summary: "跑设计门禁,返回可操作的 findings", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/gate", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + finalize: { summary: "定稿,交给人复核", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/finalize", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + g2: { summary: "把人已批准的用例编译成可执行代码", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/g2", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + execute: { summary: "执行已编译的代码(服务端开浏览器跑,可传 caseIds 只跑一部分)", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/execute", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + decisions: { summary: "读复核决定", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/decisions", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + status: { summary: "读节点产物的注册与定稿状态", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/status", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + }, +}; + +/** 工作单元:服务端拆、规划器一次只领一个。 */ +const unit: DomainSpec = { + tool: "tp_unit", + title: "工作单元循环", + description: + "开了 workUnits 的运行不整份写:服务端把活拆成单元,一次领一个、写一个,全写完服务端合并。" + + "领到什么写什么——契约、材料、这个单元该写几条,都在 claim 的返回里。", + actions: { + claim: { summary: "领一个单元,返回它的范围、材料与契约", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/units/claim", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + write: { summary: "写一个单元的产物(只校验这个单元,错在哪个单元就修哪个)", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/units/write", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + status: { summary: "读全部单元的状态:哪些完了、哪些失败、还剩几个", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/units/status", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + merge: { summary: "合并全部单元(可单独重试——整份校验会在这里失败)", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/stages/units/merge", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + }, +}; + +/** 产物:中间产物、物料、血缘、导出。 */ +const artifact: DomainSpec = { + tool: "tp_artifact", + title: "产物与中间产物", + description: "列出并读取一次运行留下的每一份东西:材料、模块树、故事、用例、门禁报告、编译产物、执行结果。", + actions: { + list: { summary: "列出这个项目的全部产物修订", method: "GET", path: "/api/projects/:projectId/workflow-runs/artifacts", params: ["projectId"] }, + read: { summary: "读一份产物的正文", method: "GET", path: "/api/projects/:projectId/workflow-runs/:runId/artifacts/:revisionId", params: ["projectId", "runId", "revisionId"] }, + lineage: { summary: "读一份产物的血缘:它从哪几份产物来", method: "GET", path: "/api/projects/:projectId/workflow-runs/:runId/artifacts/:revisionId/lineage", params: ["projectId", "runId", "revisionId"] }, + diff: { summary: "和上一版比差在哪", method: "GET", path: "/api/projects/:projectId/workflow-runs/:runId/artifacts/:revisionId/diff", params: ["projectId", "runId", "revisionId"] }, + export: { summary: "导出一份产物", method: "GET", path: "/api/projects/:projectId/workflow-runs/:runId/artifacts/:revisionId/export", params: ["projectId", "runId", "revisionId"] }, + import: { summary: "导入一份外部产物", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/artifacts", params: ["projectId", "runId"], mutates: true, body: json, needsRunGrant: true }, + }, +}; + +/** 复核:把用例摆给人看,人点头了再写决定。 */ +const review: DomainSpec = { + tool: "tp_review", + title: "人工复核", + description: + "读待复核的用例、写复核决定。**批准与驳回是人的决定**——把用例摆给人看," + + "人说了才写,而且要在 note 里写清楚这是谁的判断。agent 不该批准自己写的用例。", + actions: { + list: { summary: "读这次运行待复核的用例与它们的修订", method: "GET", path: "/api/projects/:projectId/workflow-runs/:runId/review", params: ["projectId", "runId"] }, + history: { summary: "读复核历史", method: "GET", path: "/api/projects/:projectId/workflow-runs/:runId/review-history", params: ["projectId", "runId"] }, + decide: { summary: "写复核决定(approved / rejected)。人点头了再调", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/review", params: ["projectId", "runId"], mutates: true, body: json }, + amend: { summary: "修订一条已有的复核决定", method: "PATCH", path: "/api/projects/:projectId/workflow-runs/:runId/review", params: ["projectId", "runId"], mutates: true, body: json }, + }, +}; + +/** 执行与基线:跑完之后的那些数。 */ +const execution: DomainSpec = { + tool: "tp_execution", + title: "执行结果与基线", + description: "读执行批次的结果、和基线比、立新基线。", + actions: { + list: { summary: "列出这次运行的执行批次", method: "GET", path: "/api/projects/:projectId/workflow-runs/:runId/executions", params: ["projectId", "runId"] }, + baseline: { summary: "读当前的判决集基线", method: "GET", path: "/api/projects/:projectId/workflow-runs/:runId/executions/baseline", params: ["projectId", "runId"] }, + set_baseline: { summary: "把某次执行立为基线。**这是人的决定**", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/executions/baseline", params: ["projectId", "runId"], mutates: true, body: json }, + compare: { summary: "把一次执行和基线比:哪几条翻了", method: "GET", path: "/api/projects/:projectId/workflow-runs/:runId/executions/compare", params: ["projectId", "runId"] }, + }, +}; + + +/** 用例:看板上的那一份。改、跑、生成代码、隔离、基线。 */ +const kase: DomainSpec = { + tool: "tp_case", + title: "用例", + description: + "看板上的用例:读、改、单条跑、生成代码、隔离不稳定的、立视觉与性能基线。" + + "「跑一条」由服务端开浏览器执行——宿主没有浏览器。", + actions: { + list: { summary: "列出用例(可按项目筛)", method: "GET", path: "/api/cases" }, + create: { summary: "新建一条用例", method: "POST", path: "/api/cases", mutates: true, body: json }, + update: { summary: "改一条用例", method: "PATCH", path: "/api/cases/:id", params: ["id"], mutates: true, body: json }, + remove: { summary: "删一条用例", method: "DELETE", path: "/api/cases/:id", params: ["id"], mutates: true }, + code: { summary: "读一条用例编译出来的代码", method: "GET", path: "/api/cases/:id/code", params: ["id"] }, + set_code: { summary: "改一条用例的代码", method: "PATCH", path: "/api/cases/:id/code", params: ["id"], mutates: true, body: json }, + generate_code: { summary: "为一条用例生成代码", method: "POST", path: "/api/cases/:id/generate-code", params: ["id"], mutates: true, body: json }, + generate_batch: { summary: "批量生成代码", method: "POST", path: "/api/generate-code", mutates: true, body: json }, + run_adhoc: { summary: "临时跑一段步骤(不建用例,用来试探产品)", method: "POST", path: "/api/run", mutates: true, body: json }, + run: { summary: "跑一条用例(服务端开浏览器;执行层带缓存,同一条重跑省 token)", method: "POST", path: "/api/cases/:id/run", params: ["id"], mutates: true, body: json }, + debug: { summary: "读一条用例的调试信息:每一步看到了什么、判据怎么判的", method: "GET", path: "/api/cases/:id/debug", params: ["id"] }, + refine: { summary: "让模型改写一条用例(自愈不许动 oracle)", method: "POST", path: "/api/cases/:id/refine", params: ["id"], mutates: true, body: json }, + quarantine: { summary: "把不稳定的用例隔离:照跑但不进门禁", method: "POST", path: "/api/cases/:id/quarantine", params: ["id"], mutates: true, body: json }, + recompute_flakiness: { summary: "重算这条用例的不稳定度", method: "POST", path: "/api/cases/:id/recompute-flakiness", params: ["id"], mutates: true, body: json }, + approve_visual_baseline: { summary: "批准视觉基线。**这是人的决定**", method: "POST", path: "/api/cases/:id/baselines/approve", params: ["id"], mutates: true, body: json }, + approve_perf_baseline: { summary: "批准性能基线。**这是人的决定**", method: "POST", path: "/api/cases/:id/perf-baseline/approve", params: ["id"], mutates: true, body: json }, + baseline_verdict: { summary: "对判决集基线表态", method: "POST", path: "/api/cases/:id/baseline-verdict", params: ["id"], mutates: true, body: json }, + // 绑定没有独立的写入路由:改绑定走 `update`(PATCH /api/cases/:id)的 dataKey 字段。 + data_binding: { summary: "读用例绑定的数据集与列", method: "GET", path: "/api/cases/:id/data-binding", params: ["id"] }, + }, +}; + +/** 报告:跑完之后的那些数——单次运行、批次、趋势、成本、不稳定度。 */ +const report: DomainSpec = { + tool: "tp_report", + title: "运行报告与趋势", + description: "读执行结果:单次运行的报告、批次、待批基线、趋势、成本、不稳定度、追溯线。", + actions: { + runs: { summary: "列出最近的执行", method: "GET", path: "/api/runs" }, + run: { summary: "读一次执行", method: "GET", path: "/api/runs/:id", params: ["id"] }, + report: { summary: "读一次执行的完整报告:每一步、截图、判据、失败归因", method: "GET", path: "/api/runs/:id/report", params: ["id"] }, + batches: { summary: "列出项目的批次", method: "GET", path: "/api/projects/:id/batches", params: ["id"] }, + batch: { summary: "读一个批次", method: "GET", path: "/api/batches/:id", params: ["id"] }, + cancel_batch: { summary: "取消一个批次", method: "POST", path: "/api/batches/:id/cancel", params: ["id"], mutates: true }, + pending_baselines: { summary: "列出等人批的基线", method: "GET", path: "/api/projects/:id/pending-baselines", params: ["id"] }, + trends: { summary: "读项目趋势:通过率、覆盖、成本随时间怎么走", method: "GET", path: "/api/projects/:id/trends", params: ["id"] }, + cost: { summary: "读项目花了多少", method: "GET", path: "/api/projects/:id/cost", params: ["id"] }, + flakiness: { summary: "读项目的不稳定度", method: "GET", path: "/api/projects/:id/flakiness", params: ["id"] }, + traceability: { summary: "读项目的追溯线:需求 → 用例 → 执行", method: "GET", path: "/api/projects/:id/traceability", params: ["id"] }, + quarantine_log: { summary: "读隔离记录", method: "GET", path: "/api/projects/:id/quarantine-log", params: ["id"] }, + changes: { summary: "读项目的变更记录", method: "GET", path: "/api/projects/:id/changes", params: ["id"] }, + code_line: { summary: "读代码线:哪条用例的代码来自哪次运行", method: "GET", path: "/api/projects/:id/code-line", params: ["id"] }, + queue: { summary: "读执行队列:现在有什么在排队跑", method: "GET", path: "/api/queue" }, + readiness: { summary: "读就绪度:这个项目现在能不能交付一套能跑的测试", method: "GET", path: "/api/readiness" }, + defects: { summary: "读缺陷清单:执行找出来的产品问题", method: "GET", path: "/api/defects" }, + }, +}; + +/** 导出:把批准的用例变成一个能自己跑的 Playwright 工程。 */ +const exportProject: DomainSpec = { + tool: "tp_export", + title: "生成用例代码工程", + description: + "把项目里批准的用例导成一个能离开平台自己跑的 Playwright + Midscene 工程。" + + "工程里分两种文件:生成物(每次覆盖)与脚手架(只在缺失时创建),见导出 README。", + actions: { + preflight: { summary: "导出前检查:哪些用例没代码、哪些被隔离、登录带没带走", method: "GET", path: "/api/projects/:id/export-preflight", params: ["id"] }, + export: { summary: "导出工程(format=json 拿文件表,否则是 zip)", method: "GET", path: "/api/projects/:id/export", params: ["id"] }, + suite: { summary: "跑整套:批量执行项目里的用例", method: "POST", path: "/api/projects/:id/suite", params: ["id"], mutates: true, body: json }, + }, +}; + +/** 设置:模型、环境会话、全局配置。 */ +const settings: DomainSpec = { + tool: "tp_settings", + title: "设置", + description: "模型配置与探针、每个角色的模型档案、环境会话、全局设置。密钥的值永远不经过这里。", + actions: { + get: { summary: "读全局设置", method: "GET", path: "/api/settings" }, + set: { summary: "改全局设置", method: "POST", path: "/api/settings", mutates: true, body: json }, + reset_prompts: { summary: "把提示词恢复成出厂", method: "POST", path: "/api/settings/reset-prompts", mutates: true, body: json }, + model_config: { summary: "读模型配置", method: "GET", path: "/api/model/config" }, + set_model_config: { summary: "改模型配置", method: "POST", path: "/api/model/config", mutates: true, body: json }, + test_model: { summary: "探一次模型端点:活着吗、能不能看图", method: "POST", path: "/api/model/test", mutates: true, body: json }, + profiles: { summary: "读各角色的模型档案", method: "GET", path: "/api/projects/:projectId/model-profiles", params: ["projectId"] }, + set_profile: { summary: "设某个角色的模型档案", method: "PUT", path: "/api/projects/:projectId/model-profiles/:role", params: ["projectId", "role"], mutates: true, body: json }, + probe_profile: { summary: "探一个角色的模型档案", method: "POST", path: "/api/projects/:projectId/model-profiles/:role/probe", params: ["projectId", "role"], mutates: true, body: json }, + capture_session: { summary: "抓一次登录会话存进环境", method: "POST", path: "/api/environments/:id/capture-session", params: ["id"], mutates: true, body: json }, + set_session: { summary: "直接写入一份会话", method: "POST", path: "/api/environments/:id/set-session", params: ["id"], mutates: true, body: json }, + clear_session: { summary: "清掉环境里的会话", method: "DELETE", path: "/api/environments/:id/session", params: ["id"], mutates: true }, + api_login: { summary: "配置接口登录", method: "POST", path: "/api/environments/:id/api-login", params: ["id"], mutates: true, body: json }, + remove_environment: { summary: "删一个环境", method: "DELETE", path: "/api/environments/:id", params: ["id"], mutates: true }, + capabilities: { summary: "读可用的外部能力(本地链、基准站点等)", method: "GET", path: "/api/capabilities" }, + }, +}; + +/** 评测:记分板、gold、配对评测、变异检测。 */ +const evaluation: DomainSpec = { + tool: "tp_eval", + title: "评测与记分板", + description: "读记分板与 gold、跑配对评测与变异检测。gold 与人工标注是冻结的,只读不改。", + actions: { + scoreboard: { summary: "读记分板:各能力的当前得分与翻转次数", method: "GET", path: "/api/scoreboard" }, + paired_scoreboard: { summary: "读配对评测的记分板", method: "POST", path: "/api/scoreboard/paired", mutates: true, body: json }, + gold: { summary: "读某个能力的 gold(冻结,只读)", method: "GET", path: "/api/gold/:capability", params: ["capability"] }, + set_gold: { summary: "写 gold。**冻结集,动它要非常确定**", method: "POST", path: "/api/gold/:capability", params: ["capability"], mutates: true, body: json }, + evals: { summary: "列出评测定义(两臂只差一件事的那些对照)", method: "GET", path: "/api/evals" }, + eval: { summary: "读一个评测", method: "GET", path: "/api/evals/:id", params: ["id"] }, + score: { summary: "给一次运行打分", method: "POST", path: "/api/evals/score", mutates: true, body: json }, + paired: { summary: "跑一次配对评测(两臂只差一件事)", method: "POST", path: "/api/evals/paired", mutates: true, body: json }, + detection: { summary: "跑变异检测:人造缺陷,看用例叫不叫", method: "POST", path: "/api/evals/detection", mutates: true, body: json }, + specs: { summary: "列出评测规格", method: "GET", path: "/api/evals/specs" }, + run_spec: { summary: "跑一个评测规格", method: "POST", path: "/api/evals/specs/:id/run", params: ["id"], mutates: true, body: json }, + spec_from_critique: { summary: "从一份评审意见生成评测规格", method: "POST", path: "/api/evals/specs/from-critique", mutates: true, body: json }, + subject: { summary: "读某张图当被测对象时的评测视图", method: "GET", path: "/api/evals/subject/:graphId", params: ["graphId"] }, + }, +}; + + +/** 复核队列:看板那一侧的复核(和工作流运行的复核是两条路)。 */ +const queue: DomainSpec = { + tool: "tp_queue", + title: "复核队列", + description: + "看板的复核队列:批、驳、批量处理、补一条缺口用例、让模型重写。" + + "**批准与驳回是人的决定**——摆给人看,人说了才调。", + actions: { + list: { summary: "列出待复核的运行", method: "GET", path: "/api/review" }, + get: { summary: "读一次运行待复核的内容", method: "GET", path: "/api/review/:wfRunId", params: ["wfRunId"] }, + approve: { summary: "批准。**这是人的决定**", method: "POST", path: "/api/review/:wfRunId/approve", params: ["wfRunId"], mutates: true, body: json }, + reject: { summary: "驳回这批用例,并写清楚为什么。**这是人的决定**", method: "POST", path: "/api/review/:wfRunId/reject", params: ["wfRunId"], mutates: true, body: json }, + batch: { summary: "批量处理一批复核。**这是人的决定**", method: "POST", path: "/api/review/:wfRunId/batch", params: ["wfRunId"], mutates: true, body: json }, + amend_case: { summary: "复核时改一条用例", method: "PATCH", path: "/api/review/:wfRunId/cases/:caseId", params: ["wfRunId", "caseId"], mutates: true, body: json }, + gap_case: { summary: "补一条缺口用例:复核时发现漏测的地方", method: "POST", path: "/api/review/:wfRunId/gap-case", params: ["wfRunId"], mutates: true, body: json }, + regenerate: { summary: "让模型重写这批用例", method: "POST", path: "/api/review/:wfRunId/regenerate", params: ["wfRunId"], mutates: true, body: json }, + }, +}; + +/** 图与节点类型:工作流本身的定义。 */ +const graph: DomainSpec = { + tool: "tp_graph", + title: "工作流图", + description: "工作流图的定义、版本与差异,以及可用的节点类型。改图会影响后续每一次运行。", + actions: { + list: { summary: "列出全部工作流图", method: "GET", path: "/api/graphs" }, + get: { summary: "读一张图的定义:节点、连线、当前版本", method: "GET", path: "/api/graphs/:id", params: ["id"] }, + versions: { summary: "列出一张图的版本", method: "GET", path: "/api/graphs/:id/versions", params: ["id"] }, + version: { summary: "读某一版图的完整定义", method: "GET", path: "/api/graphs/:id/versions/:version", params: ["id", "version"] }, + diff: { summary: "比两版图差在哪:哪些节点变了", method: "GET", path: "/api/graphs/:id/diff", params: ["id"] }, + create: { summary: "建一张图。会影响后续每一次运行,动手前跟人确认", method: "POST", path: "/api/graphs", mutates: true, body: json }, + node_types: { summary: "列出可用的节点类型", method: "GET", path: "/api/node-types" }, + ablatable: { summary: "列出可消融的部件(两臂对照用)", method: "GET", path: "/api/ablatable" }, + }, +}; + +/** 工作流运行(看板视角):和 tp_run 是同一批运行的另一组视图与控制。 */ +const wf: DomainSpec = { + tool: "tp_wf", + title: "工作流运行控制", + description: + "看板视角的运行列表与控制:起跑、断点、续跑、取消、可续点。" + + "和 `tp_run` 指的是同一批运行,这里是图执行那一侧的视图。", + actions: { + list: { summary: "列出全部工作流运行(跨项目)", method: "GET", path: "/api/wf/runs" }, + get: { summary: "读一次运行的节点产物", method: "GET", path: "/api/wf/runs/:id", params: ["id"] }, + start: { summary: "起一次图执行", method: "POST", path: "/api/wf/runs", mutates: true, body: json }, + cancel: { summary: "取消这次图执行", method: "POST", path: "/api/wf/runs/:id/cancel", params: ["id"], mutates: true, body: json }, + resume: { summary: "从可续点接着跑", method: "POST", path: "/api/wf/runs/:id/resume", params: ["id"], mutates: true, body: json }, + continuations: { summary: "读可以接着干的岔路", method: "GET", path: "/api/wf/runs/:id/continuations", params: ["id"] }, + continue_from: { summary: "从某个岔路接着跑", method: "POST", path: "/api/wf/runs/:id/continue", params: ["id"], mutates: true, body: json }, + controls: { summary: "读一次运行的断点与控制", method: "GET", path: "/api/projects/:projectId/workflow-runs/:runId/controls", params: ["projectId", "runId"] }, + set_controls: { summary: "设断点:让运行在某个节点前停下等人", method: "POST", path: "/api/projects/:projectId/workflow-runs/:runId/controls", params: ["projectId", "runId"], mutates: true, body: json }, + }, +}; + +/** 审计与变异:这批用例到底有没有发现问题的能力。 */ +const audit: DomainSpec = { + tool: "tp_audit", + title: "审计与变异检测", + description: + "审计一次运行:扫描、差异、人工标注、判官校准;以及变异检测——往 DOM 里注一个人造缺陷," + + "看这批用例叫不叫。**人工标注与 gold 是冻结的,不要拿它们去训练或塞进提示词。**", + actions: { + scan: { summary: "扫描一次运行,找可疑的地方", method: "GET", path: "/api/audit/:runId/scan", params: ["runId"] }, + diff: { summary: "和上次比差在哪", method: "GET", path: "/api/audit/:runId/diff", params: ["runId"] }, + holds: { summary: "读被 hook 拦下的那些", method: "GET", path: "/api/audit/:runId/holds", params: ["runId"] }, + calibration: { summary: "读判官校准结果:模型判的和人判的差多少", method: "GET", path: "/api/audit/:runId/calibration", params: ["runId"] }, + label: { summary: "写人工标注。**这是人的判断**,不要替人写", method: "POST", path: "/api/audit/:runId/labels", params: ["runId"], mutates: true, body: json }, + mutations: { summary: "读一次运行的变异检测结果", method: "GET", path: "/api/mutation/:wfRunId", params: ["wfRunId"] }, + mutate: { summary: "跑一轮变异检测:人造缺陷,看用例抓不抓得住", method: "POST", path: "/api/mutation/:wfRunId", params: ["wfRunId"], mutates: true, body: json }, + critic: { summary: "读评审意见", method: "GET", path: "/api/critic" }, + run_critic: { summary: "跑一次评审", method: "POST", path: "/api/critic", mutates: true, body: json }, + apply_graph: { summary: "把一次对话里议定的图改动应用上去", method: "POST", path: "/api/chat/apply-graph", mutates: true, body: json }, + apply_prompt: { summary: "把一次对话里议定的提示词改动应用上去", method: "POST", path: "/api/chat/apply-prompt", mutates: true, body: json }, + }, +}; + +/** 系统:进程、能力、配置、健康。 */ +const system: DomainSpec = { + tool: "tp_system", + title: "系统与进程", + description: "外部能力进程(本地链、基准站点)的起停、系统配置与健康检查。", + actions: { + health: { summary: "服务端活着吗:连不上时先用它分清是服务没起还是别的问题", method: "GET", path: "/api/health" }, + config: { summary: "读系统配置", method: "GET", path: "/api/config" }, + set_config: { summary: "改系统配置", method: "POST", path: "/api/config", mutates: true, body: json }, + processes: { summary: "列出受管的外部进程与它们的状态(本地链、基准站点)", method: "GET", path: "/api/processes" }, + process_action: { summary: "对一个进程做操作(start/stop/restart)", method: "POST", path: "/api/processes/:id/:action", params: ["id", "action"], mutates: true, body: json }, + cancel_work: { summary: "取消一个进程正在做的活", method: "POST", path: "/api/processes/:id/cancel-work", params: ["id"], mutates: true, body: json }, + start_capability: { summary: "起一个外部能力", method: "POST", path: "/api/capabilities", mutates: true, body: json }, + set_capability_cwd: { summary: "设一个能力的工作目录", method: "PATCH", path: "/api/capabilities/:id/cwd", params: ["id"], mutates: true, body: json }, + llm_debug: { summary: "读模型调用的调试记录", method: "GET", path: "/api/llm-debug" }, + load_example: { summary: "装载 uniswap 示例项目(上手用)", method: "POST", path: "/api/examples/uniswap", mutates: true, body: json }, + check_wallet: { summary: "检查本机钱包配置", method: "POST", path: "/api/wallet/check", mutates: true, body: json }, + dapp_test: { summary: "验一次 dapp 连接", method: "POST", path: "/api/wallet/dapp-test", mutates: true, body: json }, + verify_dapp: { summary: "验证 dapp 的链上断言", method: "POST", path: "/api/dapp/verify", mutates: true, body: json }, + }, +}; + +export const DOMAINS: readonly DomainSpec[] = [project, run, stage, unit, artifact, review, execution, + kase, report, exportProject, settings, evaluation, queue, graph, wf, audit, system]; + +/** `.` → 规格。`check-host-parity` 与工具注册都从这里取。 */ +export function actionIndex(): Map { + const out = new Map(); + for (const domain of DOMAINS) + for (const [action, spec] of Object.entries(domain.actions)) + out.set(`${domain.tool}.${action}`, { domain, action, spec }); + return out; +} diff --git a/packages/testpilot-mcp/src/host/tools.ts b/packages/testpilot-mcp/src/host/tools.ts new file mode 100644 index 0000000..b6f457c --- /dev/null +++ b/packages/testpilot-mcp/src/host/tools.ts @@ -0,0 +1,76 @@ +import { z } from "zod"; +import type { HostApi } from "./api.js"; +import { DOMAINS, type ActionSpec, type DomainSpec } from "./registry.js"; + +/** + * 把路由表变成 MCP 工具。 + * + * **一个域一个工具,动作是枚举**,而不是一条路由一个工具。180 条路由摊成 180 个工具, + * agent 的工具列表会长到没法读,而且每加一条 UI 路由就要动一次工具注册; + * 十来个域工具让清单可读,映射收在 `registry.ts` 一处。 + * + * 这一层同样不懂业务:填路径参数、带上凭证、把服务端的回答原样递回去。 + */ + +/** 路径模板 + 参数 → 实际路径。缺参数当场抛,不发一个注定 404 的请求。 */ +export function fillPath(spec: ActionSpec, params: Record): string { + let path = spec.path; + for (const name of spec.params ?? []) { + const value = params[name]; + if (!value) throw new Error(`缺少路径参数 ${name}(这个动作需要 ${(spec.params ?? []).join("、")})`); + path = path.replace(`:${name}`, encodeURIComponent(value)); + } + const left = path.match(/:[a-zA-Z]+/); + if (left) throw new Error(`路径里还有没填的参数 ${left[0]}——registry 的 params 漏了它`); + return path; +} + +/** 动作清单,进工具描述。agent 靠它选动作,所以要写人话。 */ +export function describeActions(domain: DomainSpec): string { + return Object.entries(domain.actions) + .map(([name, s]) => ` ${name}${s.mutates ? "(会改东西)" : ""} — ${s.summary}`) + .join("\n"); +} + +export interface ToolHost { + registerTool( + name: string, + config: { title: string; description: string; inputSchema: Record }, + handler: (input: Record) => Promise<{ content: Array<{ type: "text"; text: string }>; isError?: boolean }>, + ): void; +} + +const ok = (value: unknown) => ({ content: [{ type: "text" as const, text: JSON.stringify(value ?? null) }] }); +const fail = (e: unknown) => ({ content: [{ type: "text" as const, text: String((e as Error)?.message ?? e) }], isError: true }); + +export function registerHostDomains(server: ToolHost, api: HostApi, domains: readonly DomainSpec[] = DOMAINS): void { + for (const domain of domains) { + const names = Object.keys(domain.actions) as [string, ...string[]]; + server.registerTool( + domain.tool, + { + title: domain.title, + description: `${domain.description}\n\n动作:\n${describeActions(domain)}`, + inputSchema: { + action: z.enum(names), + /** 路径参数放这里:projectId / runId / caseId 之类。 */ + params: z.record(z.string()).optional(), + /** 请求体。形状由动作决定,服务端会校验并说明缺了什么。 */ + body: z.unknown().optional(), + }, + }, + async (input) => { + try { + const action = String(input.action); + const spec = domain.actions[action]; + if (!spec) throw new Error(`${domain.tool} 没有动作 ${action}`); + const params = (input.params ?? {}) as Record; + const path = fillPath(spec, params); + const body = spec.body ? spec.body.parse(input.body ?? {}) : spec.method === "GET" ? undefined : (input.body ?? {}); + const runId = spec.needsRunGrant ? params.runId : undefined; + return ok(await api.call(spec.method, path, body, runId)); + } catch (e) { return fail(e); } + }, + ); + } +} diff --git a/packages/testpilot-mcp/src/run-gateway.ts b/packages/testpilot-mcp/src/run-gateway.ts index 6a8668e..59ef228 100644 --- a/packages/testpilot-mcp/src/run-gateway.ts +++ b/packages/testpilot-mcp/src/run-gateway.ts @@ -49,6 +49,8 @@ export class RunGateway { if (!/^[a-f0-9]{64}$/.test(token)) throw new Error("invalid_run_grant"); this.grants.set(runId, { projectId, token }); } + /** 宿主域工具取凭证用。只交出令牌本身,grants 表仍然只在这个类里。 */ + tokenFor(runId: string): string | undefined { return this.grants.get(runId)?.token; } has(runId: string) { return this.grants.has(runId); } get registered() { return this.grants.size > 0; } } diff --git a/packages/testpilot-mcp/src/server.ts b/packages/testpilot-mcp/src/server.ts index 991d78a..7c47b77 100644 --- a/packages/testpilot-mcp/src/server.ts +++ b/packages/testpilot-mcp/src/server.ts @@ -15,6 +15,8 @@ import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"; import { z } from "zod"; import { RunGateway } from "./run-gateway.js"; +import { HostApi } from "./host/api.js"; +import { registerHostDomains } from "./host/tools.js"; import { CalibrateJudgeInput, DriveSutInput, @@ -71,6 +73,15 @@ function progressMessage(e: ProgressEvent): string | undefined { const server = new McpServer({ name: "testpilot", version: "0.2.0" }); const runs = new RunGateway(); +/** + * 宿主入口:UI 能做的操作,这里逐域暴露给宿主(docs/v3 的 host-parity)。 + * + * 它走的是 UI 同一套 HTTP 接口,不复制任何服务端逻辑——「入口从 UI 换成宿主」 + * 要成立,两条入口就必须共用一份实现。需要运行写入凭证的动作从 RunGateway 取, + * 凭证留在 MCP 进程内存里,不经过模型。 + */ +const hostApi = new HostApi({ tokenFor: (runId) => runs.tokenFor(runId) }); +registerHostDomains(server, hostApi); /** * 一次工具调用的回复。 diff --git a/packages/testpilot-mcp/test/host-registry.test.ts b/packages/testpilot-mcp/test/host-registry.test.ts new file mode 100644 index 0000000..f3ec6f3 --- /dev/null +++ b/packages/testpilot-mcp/test/host-registry.test.ts @@ -0,0 +1,76 @@ +import { describe, expect, it } from "vitest"; +import { DOMAINS, actionIndex } from "../src/host/registry.js"; +import { fillPath, describeActions } from "../src/host/tools.js"; + +/** + * 宿主域工具的形状。业务逻辑全在服务端,所以这里钉的是**映射本身**: + * 路径参数填得对不对、缺参数说不说人话、动作清单可不可读。 + * 「registry 指向的路由真的存在」由 `scripts/check-host-parity.mjs` 管,那条要读服务端源码。 + */ +describe("宿主域工具", () => { + it("十七个域,动作名不重复", () => { + expect(DOMAINS.map((d) => d.tool)).toEqual(["tp_project", "tp_run", "tp_stage", "tp_unit", "tp_artifact", + "tp_review", "tp_execution", "tp_case", "tp_report", "tp_export", "tp_settings", "tp_eval", + "tp_queue", "tp_graph", "tp_wf", "tp_audit", "tp_system"]); + expect(actionIndex().size).toBe(DOMAINS.reduce((n, d) => n + Object.keys(d.actions).length, 0)); + }); + + it("路径参数按名字填,并且转义", () => { + const spec = { summary: "", method: "GET" as const, path: "/api/p/:id/x/:hash", params: ["id", "hash"] as const }; + expect(fillPath(spec, { id: "a b", hash: "h/1" })).toBe("/api/p/a%20b/x/h%2F1"); + }); + + it("缺参数当场说清缺哪个,而不是发一个注定 404 的请求", () => { + const spec = { summary: "", method: "GET" as const, path: "/api/p/:id", params: ["id"] as const }; + expect(() => fillPath(spec, {})).toThrow(/缺少路径参数 id/); + }); + + it("registry 漏写 params 时也要拦住——路径里剩下的 :name 是个 bug", () => { + const spec = { summary: "", method: "GET" as const, path: "/api/p/:id/y/:missing", params: ["id"] as const }; + expect(() => fillPath(spec, { id: "1" })).toThrow(/没填的参数 :missing/); + }); + + it("会改东西的动作在清单里标出来——agent 与人都要一眼看见", () => { + const text = describeActions(DOMAINS.find((d) => d.tool === "tp_project")!); + expect(text).toContain("create(会改东西)"); + expect(text).toContain("list —"); + expect(text).not.toContain("list(会改东西)"); + }); + + /** + * 摘要要能让 agent 选对动作。 + * + * 第一版门槛是「长度 > 4」,结果 60 个动作被点名,其中「取消一次运行」这种 + * 本来就已经说清楚了——**硬凑字数是为了过测试,不是为了让 agent 看懂**。 + * 改成判信息量:说出它**动了什么东西**(名词),而不只是一个动词。 + */ + it("每个动作的摘要都说清它动的是什么,不只是一个动词", () => { + const bareVerb = /^(读|改|删|列|起|跑|取消|导出|导入|新建)一?[次条个份]?$/; + for (const [name, { spec }] of actionIndex()) { + const summary = spec.summary.trim(); + expect(summary.length, `${name} 的摘要是空的`).toBeGreaterThan(2); + expect(bareVerb.test(summary), `${name} 的摘要只有一个动词:${summary}`).toBe(false); + } + }); + + it("需要运行凭证的都是写节点产物那一类", () => { + for (const [name, { spec }] of actionIndex()) + if (spec.needsRunGrant) expect(name, name).toMatch(/^tp_(stage|unit|artifact|run)\./); + }); + + /** + * 人做的决定要在摘要里说出来。宿主是 UI 的等价入口,不是「agent 可以自己拍板」的入口—— + * 冻结模块树、批准用例、立基线,在 UI 上是人点的,在宿主里也得是人说了才调。 + */ + it("人做的决定,摘要里要写明白", () => { + const humanCalls = ["tp_stage.freeze_modules", "tp_review.decide", "tp_execution.set_baseline", + "tp_case.approve_visual_baseline", "tp_case.approve_perf_baseline"]; + const index = actionIndex(); + for (const name of humanCalls) { + const entry = index.get(name); + expect(entry, name).toBeTruthy(); + expect(entry!.spec.summary, name).toMatch(/人的决定|人点头|人说了/); + expect(entry!.spec.mutates, name).toBe(true); + } + }); +}); diff --git a/plugins/testpilot-claude/.claude-plugin/testpilot.json b/plugins/testpilot-claude/.claude-plugin/testpilot.json index 2871b9d..770805a 100644 --- a/plugins/testpilot-claude/.claude-plugin/testpilot.json +++ b/plugins/testpilot-claude/.claude-plugin/testpilot.json @@ -6,7 +6,7 @@ "testpilot-explore": "2026-09-03.1", "testpilot-stories": "2026-09-14.1", "testpilot-design": "2026-09-13.1", - "testpilot-run-c": "2026-09-12.3", + "testpilot-run-c": "2026-09-14.1", "testpilot-evaluation": "2026-09-03.1", "testpilot-benchmark-design": "2026-09-04.1", "testpilot-scanner": "2026-09-03.1", @@ -18,7 +18,7 @@ "testpilot-explore": "49cd2ccdaf361117", "testpilot-stories": "0e71db20690374b0", "testpilot-design": "17734a1342ef759e", - "testpilot-run-c": "dc8f6953b1b9efd9", + "testpilot-run-c": "5f7d0a92ddf947b2", "testpilot-evaluation": "3d1bc1c2aaeae65e", "testpilot-benchmark-design": "e864d158ad34baf6", "testpilot-scanner": "10e35f7f7a7ada73", diff --git a/plugins/testpilot-claude/skills/testpilot-run-c/SKILL.md b/plugins/testpilot-claude/skills/testpilot-run-c/SKILL.md index 463a9bd..f6892fc 100644 --- a/plugins/testpilot-claude/skills/testpilot-run-c/SKILL.md +++ b/plugins/testpilot-claude/skills/testpilot-run-c/SKILL.md @@ -8,6 +8,24 @@ description: Runs TestPilot generation with this host as planner, immutable proj 宿主模型负责理解、故事拆解和用例设计。Midscene 的执行模型另由项目配置提供。 服务端保存阶段事实与门禁凭证,Web 展示同一批产物并交给用户复核。 +## 你在这里能做的不止是跑一遍流水线 + +除了下面这串阶段工具,还有一组按域分的工具(`tp_project` / `tp_run` / `tp_stage` / +`tp_unit` / `tp_artifact` / `tp_review` / `tp_execution` / `tp_case` / `tp_report` / +`tp_export` / `tp_settings` / `tp_eval` / `tp_queue` / `tp_graph` / `tp_wf` / +`tp_audit` / `tp_system`)——**Web 界面上能做的操作,这里基本都能做**:建项目、挂规则包、 +读进度与中间产物、看执行报告与基线、导出能自己跑的测试工程、读记分板。 +每个工具的 `action` 枚举里写着它能干什么。 + +三件事要记住: + +- **不知道 runId 就先找。** 它只在服务器上,你上一次会话结束就忘了。 + `tp_run.list({projectId})` 把这个项目跑过的都列出来,`tp_run.checkpoint` 说它停在哪。 +- **参数不全不要猜。** 建项目缺 `targetUrl`、跑执行缺 `codeRevision`——问人, + 人答完再调一次。编一个地址比报错更糟。 +- **人的决定要人来做。** 冻结模块树、批准用例、立基线——把内容摆给人看, + 人点头了你再调那个工具。这条和界面上是同一条:agent 是那只手,不是那个拍板的人。 + ## 顺序 1. 没有 runId 时,调用 `register_run`:传项目 ID、宿主运行时、稳定的 externalId/idempotencyKey,以及材料原文数组 `{name,text}`。重试保留完全相同的参数。宿主模型身份只有确定时才报告;未知留空。已由 Web 或运行适配器注册时,直接使用提供的 runId。 diff --git a/plugins/testpilot-codex/skills/testpilot-run-c/SKILL.md b/plugins/testpilot-codex/skills/testpilot-run-c/SKILL.md index 463a9bd..f6892fc 100644 --- a/plugins/testpilot-codex/skills/testpilot-run-c/SKILL.md +++ b/plugins/testpilot-codex/skills/testpilot-run-c/SKILL.md @@ -8,6 +8,24 @@ description: Runs TestPilot generation with this host as planner, immutable proj 宿主模型负责理解、故事拆解和用例设计。Midscene 的执行模型另由项目配置提供。 服务端保存阶段事实与门禁凭证,Web 展示同一批产物并交给用户复核。 +## 你在这里能做的不止是跑一遍流水线 + +除了下面这串阶段工具,还有一组按域分的工具(`tp_project` / `tp_run` / `tp_stage` / +`tp_unit` / `tp_artifact` / `tp_review` / `tp_execution` / `tp_case` / `tp_report` / +`tp_export` / `tp_settings` / `tp_eval` / `tp_queue` / `tp_graph` / `tp_wf` / +`tp_audit` / `tp_system`)——**Web 界面上能做的操作,这里基本都能做**:建项目、挂规则包、 +读进度与中间产物、看执行报告与基线、导出能自己跑的测试工程、读记分板。 +每个工具的 `action` 枚举里写着它能干什么。 + +三件事要记住: + +- **不知道 runId 就先找。** 它只在服务器上,你上一次会话结束就忘了。 + `tp_run.list({projectId})` 把这个项目跑过的都列出来,`tp_run.checkpoint` 说它停在哪。 +- **参数不全不要猜。** 建项目缺 `targetUrl`、跑执行缺 `codeRevision`——问人, + 人答完再调一次。编一个地址比报错更糟。 +- **人的决定要人来做。** 冻结模块树、批准用例、立基线——把内容摆给人看, + 人点头了你再调那个工具。这条和界面上是同一条:agent 是那只手,不是那个拍板的人。 + ## 顺序 1. 没有 runId 时,调用 `register_run`:传项目 ID、宿主运行时、稳定的 externalId/idempotencyKey,以及材料原文数组 `{name,text}`。重试保留完全相同的参数。宿主模型身份只有确定时才报告;未知留空。已由 Web 或运行适配器注册时,直接使用提供的 runId。 diff --git a/plugins/testpilot/plugin.json b/plugins/testpilot/plugin.json index 7f9ccef..21cca78 100644 --- a/plugins/testpilot/plugin.json +++ b/plugins/testpilot/plugin.json @@ -42,7 +42,7 @@ "testpilot-explore": "2026-09-03.1", "testpilot-stories": "2026-09-14.1", "testpilot-design": "2026-09-13.1", - "testpilot-run-c": "2026-09-12.3", + "testpilot-run-c": "2026-09-14.1", "testpilot-evaluation": "2026-09-03.1", "testpilot-benchmark-design": "2026-09-04.1", "testpilot-scanner": "2026-09-03.1", @@ -55,7 +55,7 @@ "testpilot-explore": "49cd2ccdaf361117", "testpilot-stories": "0e71db20690374b0", "testpilot-design": "17734a1342ef759e", - "testpilot-run-c": "dc8f6953b1b9efd9", + "testpilot-run-c": "5f7d0a92ddf947b2", "testpilot-evaluation": "3d1bc1c2aaeae65e", "testpilot-benchmark-design": "e864d158ad34baf6", "testpilot-scanner": "10e35f7f7a7ada73", diff --git a/plugins/testpilot/skills/testpilot-run-c/SKILL.md b/plugins/testpilot/skills/testpilot-run-c/SKILL.md index 463a9bd..f6892fc 100644 --- a/plugins/testpilot/skills/testpilot-run-c/SKILL.md +++ b/plugins/testpilot/skills/testpilot-run-c/SKILL.md @@ -8,6 +8,24 @@ description: Runs TestPilot generation with this host as planner, immutable proj 宿主模型负责理解、故事拆解和用例设计。Midscene 的执行模型另由项目配置提供。 服务端保存阶段事实与门禁凭证,Web 展示同一批产物并交给用户复核。 +## 你在这里能做的不止是跑一遍流水线 + +除了下面这串阶段工具,还有一组按域分的工具(`tp_project` / `tp_run` / `tp_stage` / +`tp_unit` / `tp_artifact` / `tp_review` / `tp_execution` / `tp_case` / `tp_report` / +`tp_export` / `tp_settings` / `tp_eval` / `tp_queue` / `tp_graph` / `tp_wf` / +`tp_audit` / `tp_system`)——**Web 界面上能做的操作,这里基本都能做**:建项目、挂规则包、 +读进度与中间产物、看执行报告与基线、导出能自己跑的测试工程、读记分板。 +每个工具的 `action` 枚举里写着它能干什么。 + +三件事要记住: + +- **不知道 runId 就先找。** 它只在服务器上,你上一次会话结束就忘了。 + `tp_run.list({projectId})` 把这个项目跑过的都列出来,`tp_run.checkpoint` 说它停在哪。 +- **参数不全不要猜。** 建项目缺 `targetUrl`、跑执行缺 `codeRevision`——问人, + 人答完再调一次。编一个地址比报错更糟。 +- **人的决定要人来做。** 冻结模块树、批准用例、立基线——把内容摆给人看, + 人点头了你再调那个工具。这条和界面上是同一条:agent 是那只手,不是那个拍板的人。 + ## 顺序 1. 没有 runId 时,调用 `register_run`:传项目 ID、宿主运行时、稳定的 externalId/idempotencyKey,以及材料原文数组 `{name,text}`。重试保留完全相同的参数。宿主模型身份只有确定时才报告;未知留空。已由 Web 或运行适配器注册时,直接使用提供的 runId。 diff --git a/scripts/check-host-parity.mjs b/scripts/check-host-parity.mjs new file mode 100644 index 0000000..ce85c50 --- /dev/null +++ b/scripts/check-host-parity.mjs @@ -0,0 +1,122 @@ +#!/usr/bin/env node +/** + * 宿主入口对 Web UI 操作的覆盖检查。见 `scripts/host-parity.mjs` 顶部的注释。 + * + * 它管两件事,都是「悄悄退化」这一类: + * ① **新加的 UI 路由必须被分类**——分不了类就红。忘了同步宿主,这里当场拦住。 + * ② **覆盖率只能涨不能跌**——基线记在清单里;掉下去就红,而且会说清楚掉在哪条上。 + * + * 用法: + * node scripts/check-host-parity.mjs # 检查 + * node scripts/check-host-parity.mjs --write # 把新路由并进清单(默认 todo)并刷新基线 + */ +import { readFileSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { routeInventory, routeKey } from "./host-parity.mjs"; +import { actionIndex } from "../packages/testpilot-mcp/src/host/registry.ts"; + +const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +const FILE = path.join(ROOT, "server/host-parity.json"); +const WRITE = process.argv.includes("--write"); + +function main() { + const manifest = JSON.parse(readFileSync(FILE, "utf8")); + const declared = new Map(manifest.routes.map((r) => [routeKey(r), r])); + const actual = routeInventory(); + const problems = []; + + const missing = actual.filter((r) => !declared.has(routeKey(r))); + const stale = [...declared.keys()].filter((k) => !actual.some((r) => routeKey(r) === k)); + + if (WRITE) { + for (const r of missing) declared.set(routeKey(r), { ...r, status: "todo" }); + for (const k of stale) declared.delete(k); + /** + * **归属由 registry 算,不靠人手工维护。** + * + * 第一版 `--write` 只刷新基线,于是新加了五个域、`check` 仍然报 33.3%—— + * 因为 `host:` 字段还得有人一条条去填。一张需要手工同步的表, + * 和它要防的那种漂移是同一件事。现在:registry 里有这条路由就是 host, + * 没有就退回 todo(除非人显式标了 uiOnly)。 + */ + const byRoute = new Map(); + for (const [name, { spec }] of actionIndex()) byRoute.set(`${spec.method} ${spec.path}`, name); + for (const [key, row] of declared) { + if (row.uiOnly) continue; + const host = byRoute.get(key); + if (host) { row.host = host; delete row.status; } + else { delete row.host; row.status = "todo"; } + } + } else { + for (const r of missing) + problems.push(`${routeKey(r)}(${r.source})是新路由,清单里没有它。宿主能不能做这件事?` + + `做了就写 host:".",还没做写 status:"todo",故意不给写 uiOnly:"<理由>"。跑 --write 先并进来。`); + for (const k of stale) problems.push(`${k} 在清单里但源码里已经没有了——跑 --write 清掉。`); + } + + /** + * **registry 里每一行都必须对应一条真实路由。** + * + * 2026-09-14 第一版我照着记忆写了十条路径,真实存在的只有两条——参数叫 `:id` 不是 + * `:projectId`,材料是全局 `/api/materials` 不是项目下的。路径写错不会有任何一层报错, + * 只会在 agent 真调的时候 404,而那时候人已经在等结果了。 + * + * 反过来也查:`host:` 指向一个 registry 里不存在的动作,同样是红。 + */ + const known = new Set(actual.map(routeKey)); + for (const [name, { spec }] of actionIndex()) + if (!known.has(`${spec.method} ${spec.path}`)) + problems.push(`registry 的 ${name} 指向 ${spec.method} ${spec.path},而服务端没有这条路由。`); + const actions = new Set(actionIndex().keys()); + for (const r of [...declared.values()]) + if (r.host && !actions.has(r.host)) + problems.push(`${routeKey(r)} 标了 host:"${r.host}",而 registry 里没有这个动作。`); + + const rows = [...declared.values()]; + for (const r of rows) { + const kinds = [r.host && "host", r.status === "todo" && "todo", r.uiOnly && "uiOnly"].filter(Boolean); + if (kinds.length !== 1) problems.push(`${routeKey(r)} 的分类不明确(${kinds.join("+") || "什么都没写"})——三选一。`); + if (r.uiOnly && String(r.uiOnly).trim().length < 8) + problems.push(`${routeKey(r)} 标了 uiOnly 但没写理由。故意不给宿主的操作,理由要留下来。`); + } + + const covered = rows.filter((r) => r.host).length; + const uiOnly = rows.filter((r) => r.uiOnly).length; + const todo = rows.filter((r) => r.status === "todo").length; + /** + * 分母把 `uiOnly` 去掉:故意不给的那些不该拉低分数,否则要么没人敢标 uiOnly, + * 要么大家靠标 uiOnly 把数字做上去。理由写在清单里,人看得见。 + */ + const reachable = covered + todo; + const rate = reachable ? covered / reachable : 1; + const baseline = manifest.baseline ?? { covered: 0, reachable, rate: 0 }; + + /** + * 比的是**覆盖条数**,不是比率。 + * + * 第一版比 `rate`,而基线里存的是 `toFixed(4)` 的四舍五入值:新加一条路由、 + * 同时把它覆盖了,真实比率会比存下来的四舍五入值低一丁点,于是报 + * 「从 91.3% 掉到 91.3%」——一句自相矛盾的话。条数是整数,没有这个问题: + * 它要表达的本来就是「宿主能做的事只能变多」。 + */ + if (!WRITE && covered < (baseline.covered ?? 0)) + problems.push(`宿主覆盖从 ${baseline.covered} 条掉到 ${covered} 条(${(baseline.rate * 100).toFixed(1)}% → ${(rate * 100).toFixed(1)}%)。` + + `宿主入口只能越来越全——要么补上,要么说清楚为什么这条退回 todo。`); + + if (WRITE) { + writeFileSync(FILE, JSON.stringify({ ...manifest, baseline: { covered, reachable, rate: Number(rate.toFixed(4)) }, + routes: rows.sort((a, b) => (a.path + a.method).localeCompare(b.path + b.method)) }, null, 1) + "\n"); + console.log(`host-parity 基线已写入:${covered}/${reachable}(${(rate * 100).toFixed(1)}%),另有 ${uiOnly} 条声明为 UI 专属`); + return 0; + } + if (problems.length) { + console.error(`check-host-parity: ${problems.length} 处问题\n`); + for (const p of problems) console.error(" - " + p + "\n"); + return 1; + } + console.log(`check-host-parity: 宿主覆盖 ${covered}/${reachable}(${(rate * 100).toFixed(1)}%)· 待做 ${todo} · UI 专属 ${uiOnly}`); + return 0; +} + +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) process.exit(main()); diff --git a/scripts/host-parity.mjs b/scripts/host-parity.mjs new file mode 100644 index 0000000..d499269 --- /dev/null +++ b/scripts/host-parity.mjs @@ -0,0 +1,75 @@ +/** + * **宿主入口要能覆盖 Web UI 的每一个操作——这条要能被检查,不能只是一句话。** + * + * 2026-09-14 用户定的方向:「只是将入口从 UI 换成了宿主」。当时实测的差距是 + * 160 条 HTTP 路由 vs 27 个 MCP 工具,而且没有任何机制会告诉你差在哪、差多少。 + * 这个仓库已经被「同一件事写两遍、其中一份悄悄落后」咬过好几次 + * (`sourceRefs` 同名不同义、宿主 plugin 副本落后四个文件两天没人知道)。 + * + * 所以照 `check-drift` 的套路:**每一条路由都必须被显式分类**, + * 分不了类就红。三种分类: + * + * - `host: "."` —— 宿主已经能做这件事; + * - `todo` —— 还没做,带一句为什么它重要(覆盖率的分母); + * - `ui-only: "<理由>"` —— 故意不给宿主,理由要写下来。 + * + * 注意 `ui-only` 不是「人工闸」的藏身处。用户明确说了宿主理论上要能覆盖全部操作—— + * 冻结模块树、批准用例这些**人做的决定**,在宿主里同样是人做的决定(人在 chat 里说), + * agent 只是那只手。要保住的是审计属性:谁做的决定、经哪条入口进来的, + * 记在账本上;而不是「宿主没有这个工具」。 + */ +import { readFileSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +const VERB = "get|post|put|patch|delete"; + +/** 从源码里把路由抠出来。泛型形式 `router.get<{…}>("/x")` 也要认——漏掉它等于凭空少几条。 */ +export function routeInventory() { + const out = []; + const app = readFileSync(path.join(ROOT, "server/src/index.ts"), "utf8"); + for (const m of app.matchAll(new RegExp(`app\\.(${VERB})\\(\\s*['"\`](\\/api\\/[^'"\`]+)['"\`]`, "g"))) + out.push({ source: "index.ts", method: m[1].toUpperCase(), path: m[2] }); + const mounted = [ + ["server/src/runRoutes.ts", "/api/projects/:projectId/workflow-runs"], + ["server/src/modelProfilesRoutes.ts", "/api/projects/:projectId/model-profiles"], + ]; + for (const [file, prefix] of mounted) { + const text = readFileSync(path.join(ROOT, file), "utf8"); + for (const m of text.matchAll(new RegExp(`router\\.(${VERB})(?:<[^>]*>)?\\(\\s*['"\`]([^'"\`]+)['"\`]`, "g"))) + out.push({ source: path.basename(file), method: m[1].toUpperCase(), path: prefix + (m[2] === "/" ? "" : m[2]) }); + } + /** + * 循环注册的路由要**展开**,不能留着 `${action}`。 + * + * `runRoutes.ts` 用 `for (const action of […]) router.post(\`/:runId/stages/${action}\`)` + * 注册了十几个真实端点;静态抠出来是一条带 `${action}` 的字符串。留着它, + * 清单里就少了十几条真实操作,而覆盖率会显得好看——这正是这个检查要防的那种假象。 + * + * 展开表跟着源码走:源码里那两个数组改了,这里对不上会在下面 `unexpanded` 处红。 + */ + const LOOPS = { + "/api/projects/:projectId/workflow-runs/:runId/stages/${action}": [ + "instructions", "retrieve", "modules", "modules/state", "stories", "cases", "gate", + "finalize", "g2", "execute", "decisions", "status", "units/claim", "units/write", + "units/status", "units/merge", + ], + "/api/projects/:projectId/workflow-runs/:runId/artifacts/:revisionId/${action}": ["lineage", "diff", "export"], + }; + const expanded = out.flatMap((r) => { + const actions = LOOPS[r.path]; + return actions ? actions.map((a) => ({ ...r, path: r.path.replace("${action}", a) })) : [r]; + }); + const unexpanded = expanded.filter((r) => r.path.includes("${")); + if (unexpanded.length) + throw new Error(`路由清单里还有没展开的循环注册:${unexpanded.map((r) => r.path).join(", ")}——把它加进 LOOPS`); + out.length = 0; out.push(...expanded); + + // 同一条路径同一个方法只算一次:几处循环注册会重复出现。 + const seen = new Set(); + return out.filter((r) => { const k = `${r.method} ${r.path}`; return seen.has(k) ? false : (seen.add(k), true); }) + .sort((a, b) => (a.path + a.method).localeCompare(b.path + b.method)); +} + +export const routeKey = (r) => `${r.method} ${r.path}`; diff --git a/server/host-parity.json b/server/host-parity.json new file mode 100644 index 0000000..dace706 --- /dev/null +++ b/server/host-parity.json @@ -0,0 +1,1108 @@ +{ + "note": "宿主入口对 Web UI 操作的覆盖清单。每条路由必须被分类;见 scripts/host-parity.mjs 的注释。", + "routes": [ + { + "source": "index.ts", + "method": "GET", + "path": "/api/ablatable", + "host": "tp_graph.ablatable" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/audit/:runId/calibration", + "host": "tp_audit.calibration" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/audit/:runId/diff", + "host": "tp_audit.diff" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/audit/:runId/holds", + "host": "tp_audit.holds" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/audit/:runId/labels", + "host": "tp_audit.label" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/audit/:runId/scan", + "host": "tp_audit.scan" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/batches/:id/cancel", + "host": "tp_report.cancel_batch" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/batches/:id", + "host": "tp_report.batch" + }, + { + "source": "index.ts", + "method": "PATCH", + "path": "/api/capabilities/:id/cwd", + "host": "tp_system.set_capability_cwd" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/capabilities", + "host": "tp_settings.capabilities" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/capabilities", + "host": "tp_system.start_capability" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/cases/:id/baseline-verdict", + "host": "tp_case.baseline_verdict" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/cases/:id/baselines/approve", + "host": "tp_case.approve_visual_baseline" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/cases/:id/code", + "host": "tp_case.code" + }, + { + "source": "index.ts", + "method": "PATCH", + "path": "/api/cases/:id/code", + "host": "tp_case.set_code" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/cases/:id/data-binding", + "host": "tp_case.data_binding" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/cases/:id/debug", + "host": "tp_case.debug" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/cases/:id/generate-code", + "host": "tp_case.generate_code" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/cases/:id/perf-baseline/approve", + "host": "tp_case.approve_perf_baseline" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/cases/:id/quarantine", + "host": "tp_case.quarantine" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/cases/:id/recompute-flakiness", + "host": "tp_case.recompute_flakiness" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/cases/:id/refine", + "host": "tp_case.refine" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/cases/:id/run", + "host": "tp_case.run" + }, + { + "source": "index.ts", + "method": "DELETE", + "path": "/api/cases/:id", + "host": "tp_case.remove" + }, + { + "source": "index.ts", + "method": "PATCH", + "path": "/api/cases/:id", + "host": "tp_case.update" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/cases", + "host": "tp_case.list" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/cases", + "host": "tp_case.create" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/chat/apply-graph", + "host": "tp_audit.apply_graph" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/chat/apply-prompt", + "host": "tp_audit.apply_prompt" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/chat", + "uiOnly": "Web UI 自己的对话框——它做的事(跟人商量、再调工具)正是宿主本身在做的。把它暴露给宿主等于让 agent 去驱动另一个 agent,多一层没有意义。议定之后的落地走 tp_audit.apply_graph / apply_prompt。" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/config", + "host": "tp_system.config" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/config", + "host": "tp_system.set_config" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/critic", + "host": "tp_audit.critic" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/critic", + "host": "tp_audit.run_critic" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/dapp/verify", + "host": "tp_system.verify_dapp" + }, + { + "source": "index.ts", + "method": "DELETE", + "path": "/api/datasets/:id", + "host": "tp_project.remove_dataset" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/datasets/preview", + "host": "tp_project.preview_dataset" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/defects", + "host": "tp_report.defects" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/environments/:id/api-login", + "host": "tp_settings.api_login" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/environments/:id/capture-session", + "host": "tp_settings.capture_session" + }, + { + "source": "index.ts", + "method": "DELETE", + "path": "/api/environments/:id/session", + "host": "tp_settings.clear_session" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/environments/:id/set-session", + "host": "tp_settings.set_session" + }, + { + "source": "index.ts", + "method": "DELETE", + "path": "/api/environments/:id", + "host": "tp_settings.remove_environment" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/evals/:id", + "host": "tp_eval.eval" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/evals/detection", + "host": "tp_eval.detection" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/evals/paired", + "host": "tp_eval.paired" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/evals/score", + "host": "tp_eval.score" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/evals/specs/:id/run", + "host": "tp_eval.run_spec" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/evals/specs/from-critique", + "host": "tp_eval.spec_from_critique" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/evals/specs", + "host": "tp_eval.specs" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/evals/subject/:graphId", + "host": "tp_eval.subject" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/evals", + "host": "tp_eval.evals" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/events", + "uiOnly": "服务端推事件流(SSE)。宿主是请求-响应的,订阅不了一条长连接;进度改用 tp_run.checkpoint / tp_unit.status 轮询,拿到的是同一份状态。" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/examples/uniswap", + "host": "tp_system.load_example" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/generate-code", + "host": "tp_case.generate_batch" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/gold/:capability", + "host": "tp_eval.gold" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/gold/:capability", + "host": "tp_eval.set_gold" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/graphs/:id/diff", + "host": "tp_graph.diff" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/graphs/:id/versions/:version", + "host": "tp_graph.version" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/graphs/:id/versions", + "host": "tp_graph.versions" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/graphs/:id", + "host": "tp_graph.get" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/graphs", + "host": "tp_graph.list" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/graphs", + "host": "tp_graph.create" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/health", + "host": "tp_system.health" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/llm-debug", + "host": "tp_system.llm_debug" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/materials", + "host": "tp_project.materials" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/model/config", + "host": "tp_settings.model_config" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/model/config", + "host": "tp_settings.set_model_config" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/model/test", + "host": "tp_settings.test_model" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/mutation/:wfRunId", + "host": "tp_audit.mutations" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/mutation/:wfRunId", + "host": "tp_audit.mutate" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/node-types", + "host": "tp_graph.node_types" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/processes/:id/:action", + "host": "tp_system.process_action" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/processes/:id/cancel-work", + "host": "tp_system.cancel_work" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/processes", + "host": "tp_system.processes" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/batches", + "host": "tp_report.batches" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/changes", + "host": "tp_report.changes" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/code-line", + "host": "tp_report.code_line" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/cost", + "host": "tp_report.cost" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/datasets", + "host": "tp_project.datasets" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/projects/:id/datasets", + "host": "tp_project.add_dataset" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/environments", + "host": "tp_project.environments" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/projects/:id/environments", + "host": "tp_project.add_environment" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/export-preflight", + "host": "tp_export.preflight" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/export", + "host": "tp_export.export" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/flakiness", + "host": "tp_report.flakiness" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/overview", + "host": "tp_project.overview" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/pending-baselines", + "host": "tp_report.pending_baselines" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/quarantine-log", + "host": "tp_report.quarantine_log" + }, + { + "source": "index.ts", + "method": "DELETE", + "path": "/api/projects/:id/rule-packs/:hash", + "host": "tp_project.remove_rule_pack" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/rule-packs/:hash", + "host": "tp_project.rule_pack" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/rule-packs", + "host": "tp_project.rule_packs" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/projects/:id/rule-packs", + "host": "tp_project.add_rule_pack" + }, + { + "source": "index.ts", + "method": "DELETE", + "path": "/api/projects/:id/secrets/:key", + "host": "tp_project.remove_secret" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/secrets", + "host": "tp_project.secrets" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/projects/:id/secrets", + "host": "tp_project.set_secret" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/projects/:id/suite", + "host": "tp_export.suite" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/traceability", + "host": "tp_report.traceability" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects/:id/trends", + "host": "tp_report.trends" + }, + { + "source": "index.ts", + "method": "DELETE", + "path": "/api/projects/:id", + "host": "tp_project.remove" + }, + { + "source": "index.ts", + "method": "PATCH", + "path": "/api/projects/:id", + "host": "tp_project.update" + }, + { + "source": "modelProfilesRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/model-profiles/:role/probe", + "host": "tp_settings.probe_profile" + }, + { + "source": "modelProfilesRoutes.ts", + "method": "PUT", + "path": "/api/projects/:projectId/model-profiles/:role", + "host": "tp_settings.set_profile" + }, + { + "source": "modelProfilesRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/model-profiles", + "host": "tp_settings.profiles" + }, + { + "source": "runRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/workflow-runs/:runId/artifacts/:revisionId/diff", + "host": "tp_artifact.diff" + }, + { + "source": "runRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/workflow-runs/:runId/artifacts/:revisionId/export", + "host": "tp_artifact.export" + }, + { + "source": "runRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/workflow-runs/:runId/artifacts/:revisionId/lineage", + "host": "tp_artifact.lineage" + }, + { + "source": "runRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/workflow-runs/:runId/artifacts/:revisionId", + "host": "tp_artifact.read" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/artifacts", + "host": "tp_artifact.import" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/begin-stage", + "host": "tp_stage.begin" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/cancel", + "host": "tp_run.cancel" + }, + { + "source": "runRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/workflow-runs/:runId/checkpoint", + "host": "tp_run.checkpoint" + }, + { + "source": "runRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/workflow-runs/:runId/controls", + "host": "tp_wf.controls" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/controls", + "host": "tp_wf.set_controls" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/events", + "host": "tp_run.events" + }, + { + "source": "runRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/workflow-runs/:runId/executions/baseline", + "host": "tp_execution.baseline" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/executions/baseline", + "host": "tp_execution.set_baseline" + }, + { + "source": "runRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/workflow-runs/:runId/executions/compare", + "host": "tp_execution.compare" + }, + { + "source": "runRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/workflow-runs/:runId/executions", + "host": "tp_execution.list" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/modules/freeze", + "host": "tp_stage.freeze_modules" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/resume", + "host": "tp_run.resume" + }, + { + "source": "runRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/workflow-runs/:runId/review-history", + "host": "tp_review.history" + }, + { + "source": "runRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/workflow-runs/:runId/review", + "host": "tp_review.list" + }, + { + "source": "runRoutes.ts", + "method": "PATCH", + "path": "/api/projects/:projectId/workflow-runs/:runId/review", + "host": "tp_review.amend" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/review", + "host": "tp_review.decide" + }, + { + "source": "runRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/workflow-runs/:runId/spend", + "host": "tp_run.spend" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/cases", + "host": "tp_stage.cases" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/decisions", + "host": "tp_stage.decisions" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/execute", + "host": "tp_stage.execute" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/finalize", + "host": "tp_stage.finalize" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/g2", + "host": "tp_stage.g2" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/gate", + "host": "tp_stage.gate" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/instructions", + "host": "tp_stage.instructions" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/modules/state", + "host": "tp_stage.modules_state" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/modules", + "host": "tp_stage.modules" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/retrieve", + "host": "tp_stage.retrieve" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/status", + "host": "tp_stage.status" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/stories", + "host": "tp_stage.stories" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/units/claim", + "host": "tp_unit.claim" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/units/merge", + "host": "tp_unit.merge" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/units/status", + "host": "tp_unit.status" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/:runId/stages/units/write", + "host": "tp_unit.write" + }, + { + "source": "runRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/workflow-runs/:runId", + "host": "tp_run.get" + }, + { + "source": "runRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/workflow-runs/artifacts", + "host": "tp_artifact.list" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs/register", + "host": "tp_run.register" + }, + { + "source": "runRoutes.ts", + "method": "GET", + "path": "/api/projects/:projectId/workflow-runs", + "host": "tp_run.list" + }, + { + "source": "runRoutes.ts", + "method": "POST", + "path": "/api/projects/:projectId/workflow-runs", + "host": "tp_run.start" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/projects", + "host": "tp_project.list" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/projects", + "host": "tp_project.create" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/queue", + "host": "tp_report.queue" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/readiness", + "host": "tp_report.readiness" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/review/:wfRunId/approve", + "host": "tp_queue.approve" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/review/:wfRunId/batch", + "host": "tp_queue.batch" + }, + { + "source": "index.ts", + "method": "PATCH", + "path": "/api/review/:wfRunId/cases/:caseId", + "host": "tp_queue.amend_case" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/review/:wfRunId/gap-case", + "host": "tp_queue.gap_case" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/review/:wfRunId/regenerate", + "host": "tp_queue.regenerate" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/review/:wfRunId/reject", + "host": "tp_queue.reject" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/review/:wfRunId", + "host": "tp_queue.get" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/review", + "host": "tp_queue.list" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/run", + "host": "tp_case.run_adhoc" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/runs/:id/report", + "host": "tp_report.report" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/runs/:id", + "host": "tp_report.run" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/runs", + "host": "tp_report.runs" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/scoreboard/paired", + "host": "tp_eval.paired_scoreboard" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/scoreboard", + "host": "tp_eval.scoreboard" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/settings/reset-prompts", + "host": "tp_settings.reset_prompts" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/settings", + "host": "tp_settings.get" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/settings", + "host": "tp_settings.set" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/wallet/check", + "host": "tp_system.check_wallet" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/wallet/dapp-test", + "host": "tp_system.dapp_test" + }, + { + "source": "index.ts", + "method": "PATCH", + "path": "/api/wf/runs/:id/breakpoints", + "host": "tp_run.breakpoints" + }, + { + "source": "index.ts", + "method": "PATCH", + "path": "/api/wf/runs/:id/budget", + "host": "tp_run.budget" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/wf/runs/:id/cancel", + "host": "tp_wf.cancel" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/wf/runs/:id/continuations", + "host": "tp_wf.continuations" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/wf/runs/:id/continue", + "host": "tp_wf.continue_from" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/wf/runs/:id/nodes/:nodeId/run", + "host": "tp_run.run_node" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/wf/runs/:id/nodes/:nodeId", + "host": "tp_run.node" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/wf/runs/:id/resume-point", + "host": "tp_run.resume_point" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/wf/runs/:id/resume", + "host": "tp_wf.resume" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/wf/runs/:id/traceability", + "host": "tp_run.traceability" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/wf/runs/:id", + "host": "tp_wf.get" + }, + { + "source": "index.ts", + "method": "GET", + "path": "/api/wf/runs", + "host": "tp_wf.list" + }, + { + "source": "index.ts", + "method": "POST", + "path": "/api/wf/runs", + "host": "tp_wf.start" + } + ], + "baseline": { + "covered": 181, + "reachable": 181, + "rate": 1 + } +} diff --git a/server/scripts/peek-exec-calls.ts b/server/scripts/peek-exec-calls.ts new file mode 100644 index 0000000..cc6fe89 --- /dev/null +++ b/server/scripts/peek-exec-calls.ts @@ -0,0 +1,19 @@ +/** 只读:一次执行转发给执行模型多少次调用——量缓存省了多少。 */ +import { config } from "dotenv"; +import { join, resolve } from "node:path"; +config({ path: join(resolve(import.meta.dirname, "../.."), "server/.env"), quiet: true } as never); +const { runLedger } = await import("../src/runService.js"); +const l = runLedger(); +for (const execId of process.argv.slice(2)) { + const row = l.db.prepare("SELECT resultRevision, projectId, status FROM workflow_executions WHERE id=?").get(execId) as any; + if (!row?.resultRevision) { console.log(`${execId} ${row?.status ?? "?"} 还没有结果`); continue; } + const c = l.readRevision(row.resultRevision, row.projectId).content as any; + let calls = 0, ms = 0, zero = 0, ran = 0; + for (const r of c.results) { + if (r.status === "not_run") continue; + ran++; + const n = Array.isArray(r.modelRequests) ? r.modelRequests.filter((m: any) => m.forwarded).length : 0; + calls += n; ms += r.durationMs ?? 0; if (n === 0) zero++; + } + console.log(`${execId.slice(0, 18)} ${String(c.status).padEnd(12)} 跑了 ${ran} 条 · 模型调用 ${String(calls).padStart(3)} · 零调用 ${zero} 条 · 总耗时 ${Math.round(ms / 1000)}s`); +}