Skip to content

fix(codex): share resume sessions across benchmark planning and execution - #5527

Merged
huangruiteng merged 5 commits into
mainfrom
codex/benchmark-session-continuity
Oct 3, 2026
Merged

huangruiteng merged 5 commits into
mainfrom
codex/benchmark-session-continuity

Conversation

@huangruiteng

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

Benchmark resume previously restarted heartbeat/planning execs and lost continuity when a Turn selected a new Todo. Planning, heartbeat and ordinary Codex Turn execution now share one Goal/Agent conversation and resume its exact native ID.

  • Outcome basis / source: maintainer request for continuous benchmark sessions.
  • Before → after: separate planning/wake/Todo conversations → one agent-scoped binding, proven through native Codex history readback and two independently settled Todos.
  • Intended base: main.

Author Declaration

  • Written by: model_agent — OpenAI Codex (GPT-6 family).
  • Specification: the request above; the context contract is documented in docs/reference/protocols/loopx-turn-v0.md#cross-iteration-context-policy. The engineering boundary follows the existing long-horizon harness research RFC §11.
Criterion Disposition Owner / evidence
Planning, heartbeat and Turn resume share the same conversation across Todos implemented Codex session persistence; benchmark/tests/test_resume_sessions.py, two-Todo public Turn CLI integration
Canonical resume, with no old-name alias; ordinary Codex exec defaults to agent scope implemented Turn policy/parser and shared runner; defaults and rejected-name tests
Preserve current task authority and Goal lifetime; refuse incompatible binding/profile or unexpected native ID implemented Existing Goal admission and Todo settlement owners; lifetime, profile, fork and timeout cases

The scope vocabulary is local to conversation binding, distinct from settlement binding. The context values extend the existing Turn policy owner. Python remains the existing Codex transport/persistence adapter; Goal and settlement decisions retain their existing typed owners. Extracting persistence from the oversized exec adapter gives both runner paths one store and updates its active private imports.

Scope And Continuation

Complete within this scope: heartbeat/Turn resume includes planning and cross-Todo execution, timeout ID retention and fail-closed reuse. fresh remains the benchmark default. Managed operation-equipped app-server sessions and other providers retain their existing Todo/approval contracts.

This qualifies the runtime seam, not matched-budget quality gains. No scored benchmark jobs or leaderboard submissions were launched; E3/E4 research acceptance remains separate. No successor is needed for this request.

Validation

  • Tested revision: 438af594b402f116abab83874ced5eec6b98e671.
  • Run state: finished.
  • Input classes: synthetic, public_fixture.
Check kind Result Public-safe evidence / limitation
integration passed Broader benchmark/Turn/recovery/operation/lifetime suite: 369 passed; 2 explicitly gated online managed-operation release checks unactivated. Final runtime-focused suite: 145 passed.
real_entrypoint passed Native Codex CLI 0.160.0 against a disposable scripted Responses endpoint: planning → heartbeat retain one ID and prior history. Resume suite: 10 passed. Public Turn CLI completes and independently validates two Todos with separate settlements and the planning session.
regression_parity passed Fresh controls, Goal/Agent isolation, stale lifetime, corrupt/profile-mismatched bindings, unexpected fork and timeout. Managed-operation approval/handoff and lock ordering retain existing behavior.
integration passed Optional Ark adapter's canonical-context fixture and SDK suite: 27 passed.
static passed Changed-path Ruff, diff checks, semantic advisory/full-tree drift smoke, four-file scoped mypy. Full follow-import mypy has 0 new diagnostic messages versus base.
static failed Full follow-import mypy remains non-green on both base/head (4377/4374 errors); existing repository type debt is outside this slice.
manual passed All selected premerge canaries/risk smokes and repository checks passed. Premerge retains benchmark_sensitive maintainer-review hold; no self-merge. Public/private scan excludes raw trajectories, credentials, local paths and generated state.

Coverage includes the actual native exec transport and public CLI settlement path. Production provider behavior, online operation release qualification and scored benchmark efficacy were not claimed.

Frontend / Visual Evidence

  • UI impact: none. Entry points are the existing Turn CLI and research runner; no new frontend capability, visual surface or settings owner.
  • Before / after / viewports: N/A.

Signed-off-by: huangruiteng <huangrt01@163.com>
Signed-off-by: huangruiteng <huangrt01@163.com>
Signed-off-by: huangruiteng <huangrt01@163.com>
Signed-off-by: huangruiteng <huangrt01@163.com>
Signed-off-by: huangruiteng <huangrt01@163.com>

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent | gpt-6.1-sol | OpenAI | runtime_reported | reasoning_effort=xhigh

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

精确 head:fc42b84cedb21a56c15ca3b4af5eab0407829cf8。无阻塞发现;本次维护者已明确授权合并此 runtime/benchmark PR。

动机

同一 Agent 连续处理多个任务时,规划会话与执行会话分离,每次换 Todo 又丢失已有对话上下文。

在真实 CLI 的两个 Todo 对照中,旧版启动两个会话;新版第二个 Todo 续接原会话,两个 Todo 仍分别校验、分别结算。

规划、心跳与 Codex exec 可保持同一原生会话 ID;失配时拒绝静默换会话,显式 fresh 可恢复。

本次不调整 benchmark 评分、预算或任务语义,不扩大工具权限,也不改变 managed operation 与其他宿主的 Todo 会话边界。

改动思路

复用现有 Codex 原生会话描述符,而不是给 benchmark 另建状态。规划与心跳先观察原生 ID,普通 exec 再按 Goal/Agent 读取同一绑定。会话存储只承载上下文,不承担任务选择、权限、租约、验证或结算。每次执行仍采用当前 Todo 的签名输入与独立校验。managed operation 的 app-server、Chat/Lark 审批及其他宿主保留原 Todo 合同;Python 位于已有 provider 传输边界,类型化效果 owner 没有搬迁。

具体改动

基准为 e9e0f1bb30dc29260799f07e73dab69ab8f31e95 的 docs/reference/protocols/loopx-turn-v0.md;按接受前的合同核对 TurnEnvelope 独立校验/写回链和 Session eligibility 只作恢复元数据两项,均有实际 CLI 证据。上下文名与 Codex exec 范围是本次明确披露的有意变化,不用新文档反证旧合同。

关键代码讲解

  • _session_plan 通过 SessionBindingScope 比较会话身份;只改变上下文 key,交易 key 与 Todo 结算身份保持独立。
  • select_codex_cli_session 复用 Goal lifetime fence;agent 文件损坏时拒绝静默新建。codex_session_profile_digest 将工作区、home/config、可执行文件、模型、effort、sandbox 与 MCP 绑定到会话;失配要求显式 fresh。
  • BenchmarkSessionWake 被现有 planning/heartbeat worker 实际调用;stdout 必须确认一个原生 ID,timeout 保留 ID 但不声明成功,意外 fork 不替换已保存绑定。
  • CLI parser、run-once 与 journal retry 投影同一 scope;operation host/handoff 改用抽出的既有存储 owner,原审批 key 与锁顺序保留。Ark/DSH、Chat/Lark 测试随规范名更新;benchmark README/RUNTIME、Turn 与 DSH 文档披露上下文选择。

修复补齐了共享会话构造边界的类型标注,并整合已合入的两处摘要消费清单修复。旧 Todo 文件保持原 key;不会自动导入 agent scope。CLI 默认从 resume-if-available 改为 resume,Codex exec 从 Todo 改为 Agent;benchmark 默认仍为 fresh。旧显式名称需更新;需要隔离可使用 --session-scope todo 或 --iteration-context fresh。

对主干的风险

主要风险是把旧 Todo 完成或审批当作下一任务的授权。实际 CLI 对照证明两个 Todo 有不同 turn key/写回,只有新版第二次续接上下文;profile、Goal lifetime、损坏文件、另一个 Goal/Agent、超时及意外原生 ID 均有反例。managed operation 与 Chat/Lark 的原审批及接收者边界保留。历史上下文可能影响独立题目,fresh/todo 提供明确隔离;不宣称 benchmark 得分或付费模型质量改善。

父提交定向测试 278 项通过、15 项按平台/在线条件跳过;原生用例初次并行运行在未改动的进程清理路径出现 PermissionError,保留失败记录;隔离复跑通过,最终 head 的 10 项续接测试全部通过,包含实际 Codex CLI 0.159.2 与本地脚本化 Responses 端点的历史读回。四文件 mypy、Ruff、diff、公私边界、语义 advisory 与完整语义检查通过。用户已接受风险验证,不等待完整 CI 矩阵;这里不把跳过或合成模型响应写成在线质量通过。

语义与 CI 对齐

复用现有 Turn context 合同并规范化 resume 值,新增 provider 局部 scope 枚举;精确相等匹配,没有子串规则。错误与恢复是机器执行约束,文档明确其后果。benchmark-sensitive 自动分类仍要求维护者决定;本次明确授权覆盖这一合并决定,评分、预算、数据与 benchmark job 均未改变。

我的整体评价

APPROVE。long_horizon 和 user_experience 均改善:跨任务保留历史,失败恢复可解释,独立结果仍按 Todo 验证。范围与问题匹配;面向后续改动的简化已通过抽出一个存储 owner 落地,进一步框架化无必要。exact-scope CQS 已验证有效;风险 quick premerge 的直接检查通过,保留 benchmark-sensitive 维护者 hold,并由本次明确授权作合并决定;合并立即前复核 head/readiness。本次授权是此 PR 的维护者合并决定。

English verdict: APPROVE — fc42b84cedb21a56c15ca3b4af5eab0407829cf8. Shared Codex exec continuity preserves independent Todo validation and settlement. Real CLI baseline/head comparison, native loopback history, final-head 10 tests, scoped static/semantic checks passed; gated tests and the initial cleanup failure remain disclosed. Maintainer explicitly authorizes this merge and risk-based release qualification.

@huangruiteng
huangruiteng merged commit c52b0ae into main Oct 3, 2026
8 checks passed
@huangruiteng
huangruiteng deleted the codex/benchmark-session-continuity branch October 3, 2026 12:33

This branch was successfully deployed

1 active deployment
pypi — fc42b84c Deployed Oct 3, 2026 by huangruiteng via publish-pypi #4905
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant