Skip to content

fix(capability): retain scoped trial feedback in improvement advice - #6106

Merged
huangruiteng merged 1 commit into
mainfrom
codex/capability-trial-feedback-20261010
Oct 10, 2026
Merged

huangruiteng merged 1 commit into
mainfrom
codex/capability-trial-feedback-20261010

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

The existing improvement-intent planner ignores trial feedback: a caller can report that a trial failed and receive the same trial proposal again. Consume optional scoped feedback through the existing agent-context --phase before_plan entry so matching failures/no evidence continue useful work, successes request original-owner review, and changed inputs invalidate old feedback.

  • Outcome basis: a reproduced advisory consumer gap in the M1 backend introduced by feat(capabilities): bounded Goal improvement intent M1 backend preview #5493.
  • Before → after: matched real CLI development vectors at the fixed baseline satisfy 1/4 feedback expectations, versus 4/4 with this change. These are deterministic contract expectations, not agent-task effectiveness measurements.
  • Intended base: main; original configuration, admission, receipts and rollback retain their owners.

Author Declaration

  • Written by: model_agent, GPT-6 family, OpenAI.

Implemented against

Specification: docs/architecture/rfcs/goal-scoped-capability-portfolio-v0.md, Runtime injection and direct-first invariants, at baseline c2b17b3e1f59d0f226d0e50d566804baf58b1a93; existing improvement-intent README at that revision. Both RFC mirrors now qualify this bounded backend slice.

Criterion Disposition Symbol / path Validation
Direct capabilities keep their existing route implemented planCapabilityImprovement Direct-first, zero-budget, invalid-feedback cases
Owner refs and revisions delimit advice implemented proposal trial_basis_digest, optional trial_feedback Scope/revision/ref drift tests; real CLI readback
Success is not adoption, utility or authority implemented inspect_trial_outcome, execution_authorized: false Typed owner tests and context projection
Optional advice fails open; default off is isolated implemented existing context-provider boundary Failure/recovery, registry no-mutation and off parity
Durable portfolio adoption and composition lifecycle out_of_scope existing RFC milestones No new store, DSL, installer or lifecycle owner

Self-check: read the owning implementation and relevant RFC/README; independently reviewed the complete diff and corrected caller-supplied-feedback wording. Digest identity is not receipt authentication, and omitting feedback retains legacy advice.

Scope And Continuation

This finishes a reversible advice-consumer slice. Original owners still need authenticated outcome/effect review and real caller adoption before qualification. Composition is justified only by actual dependencies or joint acceptance; this PR adds no generic composer. App/Lark companion qualification from the existing backend preview remains open. No production enablement, upgrade, merge or release is included.

Validation

  • Tested revision: 2e34a7b6275421c5511d4096ac5729330cb70859 (runtime source frozen before the independent vectors; later edits were tests/docs only).
  • Run state: finished.
  • Input classes: synthetic.
Check kind Result Public-safe evidence / limitation
unit passed node --experimental-strip-types --test tests/control_plane_ts/goal_capability_organization.test.ts tests/control_plane_ts/agent_context.test.ts — 26 passed; owner decisions, off parity, authority and bounded projection
real_entrypoint passed LOOPX_USAGE_PING=0 uv run --extra test python -m pytest tests/capabilities/test_capability_improvement.py tests/control_plane/test_agent_context.py -q — 22 passed; actual CLI configuration/readback, supplied feedback, failure/recovery and registry immutability
regression_parity passed One attempt per arm, fixed baseline/head and inputs: 4 development CLI expectations, baseline 1/4 and head 4/4; no model or external provider calls in this comparison
manual failed Four independently authored synthetic contract vectors after source freeze: baseline 3/4, head 3/4. One head expectation asked for a proposal with stale success feedback, while the frozen policy requests owner review. Preserved disagreement; no tuning or repeat-to-green. Different failing cases between arms; no held-out uplift claim
static passed uv run python scripts/generate_semantic_inventory.py --changed-from HEAD against the pre-commit baseline: no supported vocabulary carrier findings; dynamic literals remain outside this advisory scan
static passed uv run --extra test loopx check, seven-path privacy scan and git diff --check
real_backend not_applicable Advice has no external-provider execution or authenticated receipt verifier

Coverage: deterministic software behavior and the existing CLI consumer only. The matched pair took about 8.16 seconds total including setup; this is not a performance benchmark. Overall development/retrieval/review token and currency costs are unknown. No task improvement, transfer, cost reduction, authenticated owner outcome, financial utility or complete product qualification is established. The independently authored vectors were used once after the runtime freeze and not for candidate search.

Frontend / Visual Evidence

UI impact: none. No frontend interaction or layout changes; before/after, states/viewports and visual data are not applicable. Existing optional guidance remains bounded in the shared context projection.

Type of Change

  • Bug fix
  • Documentation update
  • Test update

LoopX Area

  • Capability or extension (providers, adapters, skills)

Technical Direction

Existing Goal capability improvement and portfolio RFC; narrow feedback adoption in its current typed owner.

Shared-authority RFC fixture impact

N/A: this does not claim shared Goal Authority or migration promotion. No provider conformance, runtime routing or compatibility projection changes.

Boundary Checklist

  • Diff and public text contain no private state, credentials, raw traces, private links or local paths.
  • No duplicated benchmark program or parallel authority.
  • Scoped to the reproduced advice-consumer defect.
  • UI impact marked none.
  • Commit includes a DCO Signed-off-by trailer.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent, gpt-6.1-sol, OpenAI, runtime_reported, xhigh

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机

完成工具试用后,使用既有有界能力规划 CLI 的研究者或其它调用者。原入口忽略已提交的失败反馈,再次提议相同试用,可能造成无效重复工作。改后收到同一输入 basis 的失败反馈时继续当前有效工作,并保留结果引用。 真实 CLI 已验证匹配失败与缺证可细化建议,成功只请求原 owner 评审,独立新候选仍可使用。 本次未交付回执认证、持久化禁止重试、安装、组合语言或任务/金融效用。 原 owner 自动产生反馈、认证资格与真实领域采用仍待原有责任方验收。

改动思路

既有可选规划 provider 拥有这项建议,无须新增状态 authority 或执行机制。 本 PR 只完成既有 before_plan 入口对调用者提交反馈的消费。 保留既有 Goal 配置与准入,直接能力优先;只在收到同 basis 的反馈时细化建议。doing nothing 会继续丢失反馈,通用 composer 或持久化禁止重试会引入多余状态,因此选择原 owner 的有界 consumer。摘要绑定声明,不验证真实性,缺失反馈保持原行为;记录、退役、回滚仍由原 owner 处理。

具体改动

规范:loopx/capabilities/goal_capability_organization/README.md,revision c2b17b3e1f59d0f226d0e50d566804baf58b1a93。按改动前的契约判断,不用本 PR 的规范修订反证实现。

  • Progressive disclosure:implemented。直接能力和有界观察保持,反馈不授予执行权。
  • Original owner and configuration:implemented。原 Goal registry/typed owner 保持,实际 CLI 观察不修改配置。
  • Delivery and qualification boundary:implemented。后端 preview 范围保持,未来 adoption、composition 和产品资格并未关闭。

关键代码讲解

planCapabilityImprovement(loopx/control_plane/capabilities/goal_capability_organization.ts:46)先保留 off/direct/零预算路径,再遍历最多八个候选,计算 scope、版本与原引用的 SHA256 identity。匹配失败/缺证返回继续,成功与 stale 返回 owner review;独立候选仍能被选择。

improvementContextProvider(同文件:114)从既有 before_plan 收到 scope,将声明反馈及其 provenance 投影到原 facts。Python 仅做已有 CLI IO,未新增 adapter、flag、receipt writer 或调度器。两份测试覆盖确定性语义及真实 CLI 的配置/恢复/冷读;四份中英契约同步说明缺失反馈兼容、digest 不认证、默认关闭与尚未交付范围。

对主干的风险

最强反例是调用者省略反馈、改名 revision 或伪造成功,所有测试仍可能通过。该边界没有认证或禁止重试能力,docs 明确这一限制;success 不等于 utility,建议不授予权限。非法反馈只丢失可选贡献,新的 owner 观察能恢复;直接路径优先且关闭状态完全隔离。原配置可关闭/清除,历史证据不被改写。

本地 26 Node、22 Python 检查覆盖改变的 invariant;独立同预算 CLI 对照从 1/4 到 4/4。另四个源码冻结后的独立合成向量两边均 3/4,head 的一个 stale 预期分歧保留,没有调试到绿或宣称 held-out uplift。base/head 关闭反事实均无贡献且无 registry 写入。可见 CI 仍排队,仅作为单独诊断,不用来认证上述证据,也不尝试合并。

语义与 CI 对齐

新增词汇限于既有 advisory owner,原 default off、来源授权、独立工作、配额和 settlement 不变。语义扫描未发现支持范围内的新 carrier,动态字面量另经人工检查;README/RFC 同步限定收到反馈才抑制相同建议,避免把提示写成机器义务。

我的整体评价

exact head 2e34a7b6275421c5511d4096ac5729330cb70859,无阻塞发现。结论为 justified_increment:工程 consumer 缺口已通过真实入口验证,剩余自动生产反馈、认证资格与域内采用保留。long_horizon 与 user_experience 为 preserved:建议可失败,独立有效工作与直接路径继续,不增加账户/安装或强制确认步骤。单 owner 的小范围循环与可选输入是 proportionate,保留旧输入兼容,未建立平行版本/结果 authority。任务迁移、完整开发/评估成本、生产采用和金融收益尚未证明;root/child token 与金额未知。这些限制及 CI 合并状态与当前工程评审分开。

English verdict: APPROVE - 2e34a7b; scoped caller-declared trial feedback is retained through the existing advice owner, with direct/off/fail-open parity and no authority grant. 26 Node and 22 Python checks passed; real CLI development comparison 1/4→4/4, independent synthetic vectors 3/4→3/4 with disagreement retained. No task utility or merge authorization claimed.

@huangruiteng
huangruiteng merged commit 30d2440 into main Oct 10, 2026
13 checks passed
@huangruiteng
huangruiteng deleted the codex/capability-trial-feedback-20261010 branch October 10, 2026 10:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants