Skip to content

Commit 6fd7818

Browse files
committed
H4 切片 12:审稿结论进 experiment-summary 契约 / 结果页 / 论文材料——实验代码与检验脚本两处独立审稿的结论接到用户面前(可选字段、投影清洗、结果页两条审稿条目、论文材料稳健性 / 局限性句)
切片 10 / 11 把三个沙盒消费方的产物都送进了生成者-评审者环,但审稿结论只落库不进契约: 结果页看不到、论文材料不提、旧消费者对 review 键一无所知。本刀把实验 / 检验两处审稿结论 接出来;清洗审稿(dataset-profile)留给后续切片。 契约(packages/contracts) - experiment-summary.v1 加两个可选字段:顶层 review(实验代码审稿)与 robustness_report.review (检验脚本审稿),均 oneOf null | $defs/review_report;新 $defs review_report(九键必填、 additionalProperties=false:executed / verdict / rounds / findings / blockers / summary / stalemate / rerun_consistent / reason)、review_finding(id / severity blocker|major|minor / location / issue / fix_hint)、review_verdict(accept|reject)。不碰 required / title, check_compat 加性 OK。 - fixtures:valid experiment-summary.5(实验审稿通过 1 轮 1 条 minor + 检验审稿僵持 2 轮 1 blocker)、 invalid experiment-summary.bad-review-severity(severity=critical)。validate 16 schema / 61 fixture。 - Py(ReviewReport / ReviewFinding / ReviewVerdict / Severity,extra=forbid)与 TS (review?: null | ReviewReport)重生成;OpenAPI 基线在隔离 worktree(HEAD + 本刀补丁)里导出 (+159 行,只多 Review* 组件与两处 review 字段),避免把工作区其他未提交接口冻进基线。 投影(backend/api/omm_api/stage_outputs.py) - 新增 _review_report:白名单九键,剔 llm_calls / rerun.metrics / rerun.diff;rerun 折成 rerun_consistent 三态(未复跑 → null);findings 清洗(非 dict 丢、空 issue 丢、severity 越界归 minor、大小写归一、缺 id 按序补 R{n}),blockers 按清洗后重算而不信节点自报;executed=false 时 verdict null / findings [] / summary "",保留 rounds / reason;无该键(审稿环之前的运行)→ null。 - _robustness_report 返回多一键 review;_experiment_summary 传顶层 review。 结果页(apps/web/src/integration) - experiment-notes.ts 新增 describeReview 四态纯函数(absent / skipped / accepted / stalemate, 僵持只保留 blocker 行;复跑三态)。 - stage-content.ts renderExperimentsPanel 稳健性小节复跑逐项之后插两条 noteItem: 「实验代码独立审稿」「检验脚本独立审稿」——通过 = 绿勾「通过|N 轮|K 条意见|复跑一致 — summary。 意见:[次要] location:issue」;僵持 = 红叉「未通过|N 轮|K 条阻断性意见未解决|复跑… — reason。 未解决意见:[阻断] …」;未派出 = 黄叹号带节点原因;旧运行无该键则不出现(渲染与改前一致)。 - en-US +15 词条(审稿人原话不翻)。 论文材料(agents/skills) - review.py 新增 review_material(review, subject):未执行空串;通过「{subject}经独立审稿通过 (N 轮,K 条意见;确定性复跑核对一致 / 不一致(可复现性存疑)/ 未复跑核对)。」;僵持 「{subject}经独立审稿 N 轮后仍有 K 条阻断性意见未解决(reason;复跑…),须在模型检验与 局限性部分如实说明:」+ blocker 逐条。 - nodes.py:_validation_material 在稳健性段末追加检验脚本审稿句;新增 _experiment_material, 只在实验审稿僵持时追加(通过句已由实验节点经 verdict_summary_text 写进 experiment_summary); PaperWritingNode 的 experiment_summary 材料改走 _experiment_material。 测试 - backend/api test_stage_outputs 11 → 13(未执行 / 旧运行 null;剔过程字段 + 清洗 findings + 重算 blockers;全链断言 experiment_summary.review 与 robustness.review 等于 REVIEW_OUTPUT 投影 且无 llm_calls / rerun)。 - agents/skills 163 → 165(review_material 四态;论文材料双消费方:实验僵持只列 blocker、 检验通过一句进稳健性段、实验通过材料不变)。 - web node --test +4(describeReview 四例);npm run check / build 通过。 - 浏览器走查(CDP,拦截 /stage-outputs 换 fixture 变体)14/14、0 console error:双审稿 / 未派出 / 无 review 键(= 修改前对照)/ en-US / demo;条目三行截断可展开。 - 隔离 worktree(HEAD + 本刀暂存补丁):agents 五包 + worker 526、backend/api 319 passed / 2 skipped, 合计 845(上刀 841);contracts validate / compat / generate_python --check / export_openapi --check 全过;web check / build 通过;contracts check 在 Windows CRLF 检出下报 stale 为已知假阳性 (重生成后 --check 通过、--ignore-cr-at-eol 为空、tsc 0)。主树 SQLite 383 / 2 skipped、 真 PG 384 / 1 skipped。 已知局限 - 清洗审稿(cleaning.review)仍只落库并经 G2 impact 可见,未进 dataset-profile 契约 / 数据页 / 论文。 - run.log 进度行不变;沙盒只捕获新建文件的既有局限不变。
1 parent e6f0a40 commit 6fd7818

16 files changed

Lines changed: 1022 additions & 12 deletions

File tree

agents/skills/src/omm_agent_skills/nodes.py

Lines changed: 25 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@
7474
normalize_verdict,
7575
rerun_material,
7676
review_feedback,
77+
review_material,
7778
reviewer_tool_brief,
7879
verdict_summary_text,
7980
)
@@ -3570,14 +3571,30 @@ def _emit_progress(services: NodeServices, payload: dict[str, Any]) -> None:
35703571
pass
35713572

35723573

3574+
def _experiment_material(experiment: Mapping[str, Any]) -> str:
3575+
"""论文的实验材料:节点的实验过程摘要 + 独立审稿僵持时未解决的阻断性意见。
3576+
3577+
通过的审稿结论节点已写进 ``experiment_summary``(一行「独立审稿通过…」),这里
3578+
不重复;僵持才追加——用户在 G3 选了接受,未解决的意见也必须进论文的局限性。
3579+
"""
3580+
summary = str(experiment.get("experiment_summary") or "无")
3581+
review = experiment.get("review")
3582+
if isinstance(review, Mapping) and review.get("stalemate"):
3583+
text = review_material(review, "实验代码")
3584+
if text:
3585+
summary = f"{summary}\n{text}"
3586+
return summary
3587+
3588+
35733589
def _validation_material(
35743590
validation: Mapping[str, Any], review_decisions: Mapping[str, str]
35753591
) -> str:
3576-
"""论文的检验材料:评审判读 + 沙盒复跑的稳健性结论 + G3 决策台账。
3592+
"""论文的检验材料:评审判读 + 沙盒复跑的稳健性结论 + 检验脚本审稿结论 + G3 决策台账。
35773593
3578-
稳健性一句话由验证节点按标记行数字生成(不是模型转述);用户在 G3 选了
3579-
「接受并记录局限」时把这条纪律写进材料——未通过的检查项必须进论文的局限性,
3580-
不允许因为用户点了接受就把它们淡化掉。
3594+
稳健性一句话由验证节点按标记行数字生成(不是模型转述);检验脚本的独立审稿
3595+
(§8.4)通过与僵持都写——检验章要说得出「检验代码本身经过核查」,僵持时未解决
3596+
的意见逐条进局限性;用户在 G3 选了「接受并记录局限」时把这条纪律写进材料——
3597+
未通过的检查项必须进论文的局限性,不允许因为用户点了接受就把它们淡化掉。
35813598
"""
35823599
summary = str(validation.get("validation_summary") or "无")
35833600
robustness = validation.get("robustness")
@@ -3588,6 +3605,9 @@ def _validation_material(
35883605
coverage_text = _assumption_coverage_text(robustness.get("assumption_coverage"))
35893606
if coverage_text:
35903607
summary = f"{summary}\n{coverage_text}"
3608+
review_text = review_material(robustness.get("review"), "稳健性检验脚本")
3609+
if review_text:
3610+
summary = f"{summary}\n{review_text}"
35913611
if review_decisions.get(TaskState.VALIDATING.value) == G3_ACCEPT_OPTION_ID:
35923612
summary += (
35933613
"\n用户已在结果采用闸门确认「接受并记录局限」:未通过的检查项必须在"
@@ -3661,7 +3681,7 @@ def build_variables(self, ctx: NodeContext) -> dict[str, Any]:
36613681
plan_assumptions(planning, plan.get("id"))
36623682
),
36633683
"model_symbols": symbol_material(plan_symbols(planning, plan.get("id"))),
3664-
"experiment_summary": str(experiment.get("experiment_summary") or "无"),
3684+
"experiment_summary": _experiment_material(experiment),
36653685
"validation_summary": _validation_material(validation, ctx.review_decisions),
36663686
"frozen_numbers": render_frozen_numbers(build_frozen_numbers(ctx.prior_outputs)),
36673687
}

agents/skills/src/omm_agent_skills/review.py

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -311,6 +311,34 @@ def review_feedback(
311311
return "\n\n".join(parts)
312312

313313

314+
def review_material(review: Any, subject: str) -> str:
315+
"""论文材料里的一段独立审稿结论;``subject`` 是被审对象(「实验代码」/「稳健性检验脚本」)。
316+
317+
未执行 → 空串(论文不声称审过);通过 → 一句话(轮数、意见数、复跑核对三态);
318+
僵持 → 未解决的阻断性意见逐条列出,并点明须进模型检验与局限性——用户在闸门
319+
选了接受也不能让它们从论文里消失。数字只来自节点记录,不是模型转述。
320+
"""
321+
if not isinstance(review, Mapping) or not review.get("executed"):
322+
return ""
323+
rounds = int(review.get("rounds") or 0)
324+
findings = [entry for entry in review.get("findings") or [] if isinstance(entry, Mapping)]
325+
rerun = review.get("rerun") if isinstance(review.get("rerun"), Mapping) else {}
326+
if not rerun.get("executed"):
327+
rerun_text = "未复跑核对"
328+
elif rerun.get("consistent"):
329+
rerun_text = "确定性复跑核对一致"
330+
else:
331+
rerun_text = "确定性复跑核对不一致(可复现性存疑)"
332+
if review.get("stalemate"):
333+
blockers = [entry for entry in findings if entry.get("severity") == "blocker"]
334+
head = (
335+
f"{subject}经独立审稿 {rounds} 轮后仍有 {len(blockers)} 条阻断性意见未解决"
336+
f"({review.get('reason') or '僵持'}{rerun_text}),须在模型检验与局限性部分如实说明:"
337+
)
338+
return head + "\n" + findings_material(blockers)
339+
return f"{subject}经独立审稿通过({rounds} 轮,{len(findings)} 条意见;{rerun_text})。"
340+
341+
314342
def verdict_summary_text(review: Mapping[str, Any]) -> str:
315343
"""面向用户的一句话审稿结论(进度旁路 / 进度叙述用)。"""
316344
if not review.get("executed"):

agents/skills/tests/test_nodes.py

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3222,6 +3222,59 @@ def test_paper_material_carries_robustness_and_g3_decision(registry):
32223222
)
32233223

32243224

3225+
def test_paper_material_carries_review_verdicts_of_both_sandbox_consumers(registry):
3226+
"""论文材料:检验脚本的审稿结论进检验材料(通过与僵持都写);实验代码审稿只在
3227+
僵持时把未解决的阻断性意见追加进实验材料(通过句节点已写进 experiment_summary)。"""
3228+
prior = paper_prior()
3229+
prior[TaskState.EXPERIMENTING.value] = {
3230+
**prior[TaskState.EXPERIMENTING.value],
3231+
"review": {
3232+
"executed": True, "rounds": 2, "verdict": "reject", "blockers": 1,
3233+
"findings": [
3234+
{"id": "R1", "severity": "blocker", "location": "experiment.py:split", "issue": "训练 / 评估切分泄漏", "fix_hint": "按时间切分"},
3235+
{"id": "R2", "severity": "minor", "location": "", "issue": "命名", "fix_hint": ""},
3236+
],
3237+
"summary": "存在泄漏", "rerun": {"executed": True, "consistent": True},
3238+
"stalemate": True, "reason": "R2 运行预算已尽,无法按审稿意见修复",
3239+
},
3240+
}
3241+
prior[TaskState.VALIDATING.value] = {
3242+
**VALIDATION_OK,
3243+
"robustness": {
3244+
"executed": True,
3245+
"status": "passed",
3246+
"summary_text": "沙盒复跑稳健性检查 3 项,通过 3 项,全部达标。",
3247+
"review": {
3248+
"executed": True, "rounds": 1, "verdict": "accept", "blockers": 0,
3249+
"findings": [], "summary": "检验口径与实验一致",
3250+
"rerun": {"executed": True, "consistent": True}, "stalemate": False, "reason": "",
3251+
},
3252+
},
3253+
}
3254+
ctx = make_ctx(TaskState.PAPER_WRITING, prior=prior)
3255+
3256+
variables = PaperWritingNode(registry).build_variables(ctx)
3257+
3258+
experiment_material = variables["experiment_summary"]
3259+
assert experiment_material.startswith(prior[TaskState.EXPERIMENTING.value]["experiment_summary"])
3260+
assert "实验代码经独立审稿 2 轮后仍有 1 条阻断性意见未解决" in experiment_material
3261+
assert "[R1|blocker] experiment.py:split:训练 / 评估切分泄漏(修法:按时间切分)" in experiment_material
3262+
assert "[R2|minor]" not in experiment_material, "非阻断意见不进论文的局限性清单"
3263+
assert "须在模型检验与局限性部分如实说明" in experiment_material
3264+
validation_material = variables["validation_summary"]
3265+
assert validation_material.endswith(
3266+
"稳健性检验脚本经独立审稿通过(1 轮,0 条意见;确定性复跑核对一致)。"
3267+
)
3268+
3269+
# 实验审稿通过(未僵持):实验材料保持节点输出原样——通过句已在 experiment_summary 里
3270+
prior[TaskState.EXPERIMENTING.value]["review"] = {
3271+
"executed": True, "rounds": 1, "verdict": "accept", "blockers": 0, "findings": [],
3272+
"summary": "忠实于方案", "rerun": {"executed": True, "consistent": True}, "stalemate": False, "reason": "",
3273+
}
3274+
accepted = PaperWritingNode(registry).build_variables(make_ctx(TaskState.PAPER_WRITING, prior=prior))
3275+
assert accepted["experiment_summary"] == prior[TaskState.EXPERIMENTING.value]["experiment_summary"]
3276+
3277+
32253278
# -- 假设表的下游消费:实验任务卡 / 判读 / 稳健性检验 / 论文材料 ----------------------
32263279

32273280

agents/skills/tests/test_review.py

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
findings_material,
2323
rerun_material,
2424
review_feedback,
25+
review_material,
2526
verdict_summary_text,
2627
)
2728

@@ -233,3 +234,38 @@ def test_verdict_summary_text_covers_accept_stalemate_and_skipped():
233234
"stalemate": True, "reason": "审稿 2 轮后仍有阻断性意见未解决",
234235
})
235236
assert text == "独立审稿 2 轮后仍有 1 条阻断性意见未解决(审稿 2 轮后仍有阻断性意见未解决),交结果采用闸门裁定"
237+
238+
239+
def test_review_material_writes_paper_facts_only_when_reviewed():
240+
"""论文材料:未审稿不声称审过;通过一句话(轮数 / 意见数 / 复跑三态);僵持把
241+
未解决的阻断性意见逐条列出并点明须进局限性。"""
242+
assert review_material(None, "实验代码") == ""
243+
assert review_material({"executed": False, "reason": "未配置子代理监督者"}, "实验代码") == ""
244+
245+
accepted = review_material({
246+
"executed": True, "rounds": 1, "verdict": "accept",
247+
"findings": [{"id": "R1", "severity": "minor", "issue": "命名"}],
248+
"blockers": 0, "rerun": {"executed": True, "consistent": True}, "stalemate": False,
249+
}, "实验代码")
250+
assert accepted == "实验代码经独立审稿通过(1 轮,1 条意见;确定性复跑核对一致)。"
251+
unrerun = review_material({
252+
"executed": True, "rounds": 1, "verdict": "accept", "findings": [], "blockers": 0,
253+
"rerun": {"executed": False, "reason": "剩余预算不足以复跑核对"}, "stalemate": False,
254+
}, "稳健性检验脚本")
255+
assert unrerun == "稳健性检验脚本经独立审稿通过(1 轮,0 条意见;未复跑核对)。"
256+
257+
stalemate = review_material({
258+
"executed": True, "rounds": 2, "verdict": "reject",
259+
"findings": [
260+
{"id": "R1", "severity": "blocker", "location": "perturb()", "issue": "评估集未同步扰动", "fix_hint": "同步扰动"},
261+
{"id": "R2", "severity": "minor", "issue": "阈值来源未说明"},
262+
],
263+
"blockers": 1, "rerun": {"executed": True, "consistent": False},
264+
"stalemate": True, "reason": "审稿 2 轮后仍有阻断性意见未解决",
265+
}, "稳健性检验脚本")
266+
assert stalemate == (
267+
"稳健性检验脚本经独立审稿 2 轮后仍有 1 条阻断性意见未解决"
268+
"(审稿 2 轮后仍有阻断性意见未解决;确定性复跑核对不一致(可复现性存疑)),须在模型检验与局限性部分如实说明:\n"
269+
"[R1|blocker] perturb():评估集未同步扰动(修法:同步扰动)"
270+
)
271+
assert "R2" not in stalemate, "非阻断意见不进论文的局限性清单"

apps/web/src/i18n/en-US.ts

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -802,6 +802,22 @@ const EXPERIMENT_STAGE: Record<string, string> = {
802802
"未通过": "Failed",
803803
"实测": "Observed",
804804
"阈值": "Threshold",
805+
// 独立审稿条目(stage-content:experiment-summary.review / robustness.review,H4 切片 12)
806+
"实验代码独立审稿": "Independent review of experiment code",
807+
"实验代码独立审稿未执行": "Independent review of experiment code not executed",
808+
"检验脚本独立审稿": "Independent review of validation script",
809+
"检验脚本独立审稿未执行": "Independent review of validation script not executed",
810+
"轮": "round(s)",
811+
"条意见": "finding(s)",
812+
"条阻断性意见未解决": "unresolved blocker(s)",
813+
"复跑一致": "Rerun consistent",
814+
"复跑不一致": "Rerun inconsistent",
815+
"未复跑": "No rerun",
816+
"意见": "Findings",
817+
"未解决意见": "Unresolved findings",
818+
"阻断": "Blocker",
819+
"主要": "Major",
820+
"次要": "Minor",
805821
"在 5 个不同随机种子下波动较小,最大标准差 1.52%。":
806822
"Across five random seeds the spread stays small, with a maximum standard deviation of 1.52%.",
807823
"跨时段与区域验证均优于基线,整体性能稳定。":

apps/web/src/integration/experiment-notes.test.mjs

Lines changed: 82 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ const source = await readFile(new URL("./experiment-notes.ts", import.meta.url),
88
const { outputText } = ts.transpileModule(source, {
99
compilerOptions: { module: ts.ModuleKind.ESNext, target: ts.ScriptTarget.ES2022 },
1010
});
11-
const { describeRobustness, formatMetricValue } = await import(
11+
const { describeReview, describeRobustness, formatMetricValue } = await import(
1212
`data:text/javascript;charset=utf-8,${encodeURIComponent(outputText)}`
1313
);
1414

@@ -95,6 +95,87 @@ test("absent field (runs before sandboxing / sim nodes) renders nothing about th
9595
assert.deepEqual(describeRobustness(undefined), { kind: "absent" });
9696
});
9797

98+
// ── 独立审稿(契约 review / robustness.review,fixture experiment-summary.5) ──
99+
100+
/** fixture.5 的实验代码审稿:一轮通过,一条 minor 意见,复跑一致。 */
101+
function acceptedReview() {
102+
return {
103+
executed: true,
104+
verdict: "accept",
105+
rounds: 1,
106+
findings: [
107+
{ id: "R1", severity: "minor", location: "experiment.py:12", issue: "随机种子写死在脚本里,建议改为常量集中管理", fix_hint: "抽成 SEED 常量" },
108+
],
109+
blockers: 0,
110+
summary: "实现忠实于方案 A,指标口径与方案一致,可复现",
111+
stalemate: false,
112+
rerun_consistent: true,
113+
reason: "",
114+
};
115+
}
116+
117+
/** fixture.5 的检验脚本审稿:两轮后僵持,一条 blocker + 一条 minor。 */
118+
function stalemateReview() {
119+
return {
120+
executed: true,
121+
verdict: "reject",
122+
rounds: 2,
123+
findings: [
124+
{ id: "R1", severity: "blocker", location: "robustness.py:perturb()", issue: "扰动只作用在训练集,评估集未同步扰动,敏感性数值偏乐观", fix_hint: "扰动后重新切分并同时评估" },
125+
{ id: "R2", severity: "minor", location: "", issue: "阈值 0.2 未说明来源", fix_hint: "" },
126+
],
127+
blockers: 1,
128+
summary: "扰动实现有缺陷,敏感性结论不能采信",
129+
stalemate: true,
130+
rerun_consistent: true,
131+
reason: "审稿 2 轮后仍有阻断性意见未解决",
132+
};
133+
}
134+
135+
test("accepted review: rounds, every finding (location-prefixed), rerun state, reviewer summary", () => {
136+
assert.deepEqual(describeReview(acceptedReview()), {
137+
kind: "accepted",
138+
rounds: 1,
139+
findings: [
140+
{ severity: "minor", severityLabel: "次要", text: "experiment.py:12:随机种子写死在脚本里,建议改为常量集中管理" },
141+
],
142+
summary: "实现忠实于方案 A,指标口径与方案一致,可复现",
143+
rerun: "consistent",
144+
});
145+
});
146+
147+
test("stalemate review: only blockers are listed (they are what G3 and the paper must carry)", () => {
148+
assert.deepEqual(describeReview(stalemateReview()), {
149+
kind: "stalemate",
150+
rounds: 2,
151+
blockers: 1,
152+
findings: [
153+
{ severity: "blocker", severityLabel: "阻断", text: "robustness.py:perturb():扰动只作用在训练集,评估集未同步扰动,敏感性数值偏乐观" },
154+
],
155+
summary: "扰动实现有缺陷,敏感性结论不能采信",
156+
reason: "审稿 2 轮后仍有阻断性意见未解决",
157+
rerun: "consistent",
158+
});
159+
});
160+
161+
test("rerun state: null → not_run, false → inconsistent; blank location keeps the bare issue", () => {
162+
const noRerun = describeReview({ ...acceptedReview(), rerun_consistent: null, findings: [
163+
{ id: "R1", severity: "major", location: " ", issue: "只报了 rmse", fix_hint: "" },
164+
] });
165+
assert.equal(noRerun.rerun, "not_run");
166+
assert.deepEqual(noRerun.findings, [{ severity: "major", severityLabel: "主要", text: "只报了 rmse" }]);
167+
assert.equal(describeReview({ ...stalemateReview(), rerun_consistent: false }).rerun, "inconsistent");
168+
});
169+
170+
test("skipped review surfaces the node's reason; absent field renders nothing", () => {
171+
assert.deepEqual(
172+
describeReview({ executed: false, verdict: null, rounds: 0, findings: [], blockers: 0, summary: "", stalemate: false, rerun_consistent: null, reason: "未配置子代理监督者,跳过独立审稿" }),
173+
{ kind: "skipped", reason: "未配置子代理监督者,跳过独立审稿" },
174+
);
175+
assert.deepEqual(describeReview(null), { kind: "absent" });
176+
assert.deepEqual(describeReview(undefined), { kind: "absent" });
177+
});
178+
98179
test("formatMetricValue: thousands separators, bounded decimals, non-numbers untouched", () => {
99180
assert.equal(formatMetricValue(0.123456), "0.1235");
100181
assert.equal(formatMetricValue(1234.5678), "1,234.57");

0 commit comments

Comments
 (0)