From 15f3772b6e9d2ad6d34a94694c321b9760867e37 Mon Sep 17 00:00:00 2001 From: forrestlinfeng Date: Fri, 23 Oct 2026 09:32:28 +0800 Subject: [PATCH 1/6] Keep Evolver out of the model context: report at turn end, distill at session end The plugin no longer registers evolver_* tools or the capability-evolver skill, and the injected strategy ends with "Apply them where they fit." instead of asking the agent to search, report or distill. Reuse is reported by the turn-end hook (now also as aborted when a session goes away or is killed mid-turn); a verified turn leaves a candidate on disk, distilled once at session end from a digest of the turn through the host model. /evolver-* commands run in the plugin and answer in the UI. Measured on 60 tasks x 2 attempts with real Hub recall (dsh-local-bench, layer 4): the form without tools or skill used the fewest context tokens per step, and the stock plugin loaded the skill in 62/120 trials and reported by hand in 53. Co-Authored-By: Claude Opus 5.5 --- README.md | 14 +- README.zh.md | 4 +- assets/commands/distill.md | 19 --- assets/commands/evolve.md | 12 +- assets/commands/review.md | 15 -- assets/commands/run.md | 22 --- assets/commands/search.md | 20 --- assets/commands/solidify.md | 17 --- assets/commands/status.md | 48 ------ assets/commands/sync.md | 15 -- assets/skills/capability-evolver/SKILL.md | 100 ------------- src/commands.js | 108 ++++++++++---- src/index.js | 136 +++++++++++------ src/injected-assets.js | 45 +++++- src/prime.js | 3 +- src/session-distill.js | 174 ++++++++++++++++++++++ src/skill.js | 41 ----- src/turn-record.js | 2 +- test/applied.test.js | 2 +- test/apply.test.js | 74 ++++----- test/injected-assets.test.js | 4 +- test/instructions.test.js | 24 +-- test/prime.test.js | 6 +- test/reporting.test.js | 30 +--- test/session-distill.test.js | 167 +++++++++++++++++++++ test/skill.test.js | 11 -- 26 files changed, 617 insertions(+), 496 deletions(-) delete mode 100644 assets/commands/distill.md delete mode 100644 assets/commands/review.md delete mode 100644 assets/commands/run.md delete mode 100644 assets/commands/search.md delete mode 100644 assets/commands/solidify.md delete mode 100644 assets/commands/status.md delete mode 100644 assets/commands/sync.md delete mode 100644 assets/skills/capability-evolver/SKILL.md create mode 100644 src/session-distill.js delete mode 100644 src/skill.js create mode 100644 test/session-distill.test.js delete mode 100644 test/skill.test.js diff --git a/README.md b/README.md index c9d4f6c..bb64c67 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ Evolver lets your dsh agent evolve itself. Every success becomes a shortcut for - **Survival of the fittest.** Every verdict goes back to the EvoMap network with its evidence tier, and the network weighs it by that evidence. Strategies that work get recalled more often; ones proven wrong sink. - **One agent's mistake, every agent's detour.** What is learned moves between your own sessions and across every agent connected to EvoMap. -Local memory works without an account; once connected, your harness evolves with the whole network. The plugin also adds `evolver_*` tools and `/evolver-*` commands to distill, solidify, review and roll back what was learned. +Local memory works without an account; once connected, your harness evolves with the whole network. Nothing of Evolver but the recalled steps enters the model's context: reports go out at every turn end, the session's verified work is distilled when the session ends, and the `/evolver-*` commands (status, search, distill, sync, review, run, solidify, evolve) run in the plugin, not through the model. ## Install @@ -102,6 +102,16 @@ Network recall needs `@evomap/evolver` 2.0.39 or newer. The evidence on reuse re reaches the Hub only through a Proxy newer than 2.0.42; an older one drops it, and the Hub counts those reports as unevidenced. +## Session-end distillation + +A turn that ended completed, used tools, passed its last check and reused no injected strategy leaves a +candidate in `~/.evolver/state/distill-pending.json`; a later verified turn replaces it, and a next prompt +that reads as a correction drops it. When the session ends, one call to the session's own model writes a +Gene from a digest of that turn (task, commands, touched paths, checks, final answer — never the whole +history) and sends it to the Proxy's `POST /conversation/distill` with `persist: true`, `publish: false` +and its own token cost as `distill_usage`. A killed session's candidate is distilled at the next session +start. `/evolver-distill` does the same at once. + ## Reuse reports Every strategy injected into a turn is reported through the Proxy's @@ -113,7 +123,7 @@ weighs each report by its evidence: | Turn end | the turn ends | `dsh::` | `tier` `L1` when the turn ran a test, build, lint or type check, else `none`; `applied`; `check` `{kind, passed, fail_then_pass}` of the last check; `aborted`; `stage: immediate` | | Correction | the next prompt says the answer did not hold, or approves it | `…:correction` | `tier: L3`, `applied`, `stage: correction` | | Retention | the next session in the same project finds the turn's edits kept or reverted | `…:retention` | `tier: L2`, `applied`, `stage: retention` | -| `evolver_asset_reuse_result` | the model reports on its own | the model's own | `tier: L5`, `applied`, `stage: immediate` | +| Aborted | the session goes away, or is killed, before the turn ends (a killed session's reports go out at the next session start) | `dsh::` | `tier: none`, `applied: partial`, `aborted: true`, `stage: immediate` | - **`applied`** compares the strategy's numbered steps with the turn's shell commands, touched paths, the code its edits wrote (200,000 characters a turn, kept in memory only) diff --git a/README.zh.md b/README.zh.md index 38dcac3..75b892e 100644 --- a/README.zh.md +++ b/README.zh.md @@ -17,7 +17,7 @@ Evolver 让你的 dsh 智能体自己进化。每一次成功都会变成下一 - **优胜劣汰。** 每份复用结论都带着证据等级回报 EvoMap 网络，网络按证据为它加权。有效的策略越来越常被召回，被证伪的策略逐渐沉下去。 - **一个智能体踩过的坑，整个网络都绕开。** 经验在你的每个会话之间流转，也在所有接入 EvoMap 的智能体之间流转。 -不注册账号也能用本地记忆；接入网络后，你的 harness 就和整个网络一起进化。插件还提供 `evolver_*` 工具和 `/evolver-*` 命令，用来提炼、固化、审阅和回滚学到的经验。 +不注册账号也能用本地记忆；接入网络后，你的 harness 就和整个网络一起进化。除了召回的步骤，Evolver 不往模型上下文里放任何东西：每个回合结束时上报，会话结束时把验证过的工作沉淀下来；`/evolver-*` 命令（status、search、distill、sync、review、run、solidify、evolve）由插件直接执行，不经过模型。 ## 安装 @@ -97,7 +97,7 @@ dsh plugin --profile dsh-tui add -w @evomap/dsh-evolver@latest | 回合结束 | 回合结束时 | `dsh:<会话>:<回合>` | 本回合跑过测试、构建、lint 或类型检查时 `tier` 为 `L1`，否则为 `none`；`applied`；最后一次检查的 `check` `{kind, passed, fail_then_pass}`；`aborted`；`stage: immediate` | | 追问补报 | 下一条提示词表示结果不对，或表示认可 | `…:correction` | `tier: L3`、`applied`、`stage: correction` | | 留存回访 | 同一项目的下一个会话发现本回合的改动被保留或被撤销 | `…:retention` | `tier: L2`、`applied`、`stage: retention` | -| `evolver_asset_reuse_result` | 模型自己上报 | 模型自带 | `tier: L5`、`applied`、`stage: immediate` | +| 中止 | 回合结束前会话消失或被强杀（被强杀会话的上报在下次会话开始时发出） | `dsh::` | `tier: none`、`applied: partial`、`aborted: true`、`stage: immediate` | - **`applied`** 把策略的编号步骤与本回合的 shell 命令、涉及的路径、编辑写入的代码（每回合最多 20 万字符，只留在内存里）和最终回答比对：半数以上步骤出现记 `used`，少于半数记 `partial`，一条都没有记 `unused`。回合没留下可比对的内容，或策略没有编号步骤时，按 tier `none` 加 `applied: partial` 上报，两边都不计权重，reason 里写 `applied: unknown`。 - **检查**指运行测试、构建、lint 或类型检查的 `bash` 命令。结果报错或输出里有失败信息都算失败，因为经 `tail` 管道的失败运行退出码仍是 0。判断 `fail_then_pass` 时会先去掉输出管道（`| tail -9`）和重定向（`2>&1`）再比较命令。 diff --git a/assets/commands/distill.md b/assets/commands/distill.md deleted file mode 100644 index f46997b..0000000 --- a/assets/commands/distill.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -description: Distill a reusable skill/gene from this session's verified work. -argument-hint: "[what to distill]" ---- - -Distill what this session proved into a reusable asset. - -Prefer `evolver_distill_conversation`. Supply a concrete summary, reproducible strategy, -validation commands that actually passed, relevant artifacts, and signals. Persistence is -on by default; set `publish: true` only when the user explicitly asked to share the result. -Never include secrets, credentials, private paths, or an unverified claim. - -Use the final `Invocation arguments (verbatim JSON string)` as additional scope for what to -distill. If the Proxy is unavailable and the full CLI is installed, use the installed -`evolver` executable (or `npx -y @evomap/evolver`) with its `distill` subcommand and the -same decoded input. - -Tell the user what was distilled, the evidence attached, and whether it stayed local or was -submitted for review. diff --git a/assets/commands/evolve.md b/assets/commands/evolve.md index 58da080..159fd4f 100644 --- a/assets/commands/evolve.md +++ b/assets/commands/evolve.md @@ -7,15 +7,11 @@ Take a deliberate evolution step for the current task. 1. **Recall.** Look at the evolution memory this plugin injected behind the prompt, or read the tail of the full engine's memory graph — `~/.evolver/memory/evolution/memory_graph.jsonl`, or the project's `memory/evolution/memory_graph.jsonl` when it exists. Summarize any recent - outcome — success or failure — relevant to what we're working on. Search the network - too, with `evolver_search_assets`, when the task looks like something others have hit. + outcome — success or failure — relevant to what we're working on. 2. **Reflect.** Given the current diff and task state, say in one or two lines: what worked, what didn't, and what the durable lesson is. -3. **Record.** Reuse of an injected strategy is reported automatically at every turn end. - If the lesson is worth sharing, distill it with - `evolver_distill_conversation` (see `/evolver-distill`). To run the full engine now, - use `/evolver-run`. - -Keep this lightweight — it is an explicit checkpoint, not a ceremony on every turn. +Reuse of an injected strategy is reported at every turn end, and the session's verified work is +distilled when the session ends; `/evolver-distill` distills it now. Keep this lightweight — it is an +explicit checkpoint, not a ceremony on every turn. diff --git a/assets/commands/review.md b/assets/commands/review.md deleted file mode 100644 index 88d22dd..0000000 --- a/assets/commands/review.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -description: Review Evolver's pending evolved changes, then approve (solidify) or reject (roll back). -argument-hint: "[--approve | --reject]" ---- - -Review the changes Evolver currently has **pending solidify** in this repository. - -1. Show `git status --short` and `git diff HEAD` so the user sees the proposed edits. -2. Decode the final `Invocation arguments (verbatim JSON string)` value. -3. Use the installed `evolver` executable, or `npx -y @evomap/evolver` when absent: - - `--approve` or an explicit request to accept: invoke `review --approve`. - - `--reject` or an explicit request to discard: invoke `review --reject`. - - Empty or ambiguous input: summarize the pending diff and ask the user to choose before - running either state-changing command. -4. Report the final state and resulting git status. diff --git a/assets/commands/run.md b/assets/commands/run.md deleted file mode 100644 index 793a7c2..0000000 --- a/assets/commands/run.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -description: Run one Evolver self-evolution cycle on the current repo (collect signals → select/mutate genes → propose changes). -argument-hint: "[--loop] [--dry-run] [--strategy=balanced|innovate|harden|repair-only]" ---- - -Run one Evolver cycle in the **current git repository**. - -The final `Invocation arguments (verbatim JSON string)` value contains the exact user -suffix. Decode that JSON string, preserve its argument order, and never drop a protective -flag such as `--dry-run` or invent a flag the user did not provide. - -1. Confirm `git rev-parse --is-inside-work-tree` succeeds. Otherwise explain that Evolver - requires git and stop. -2. Use the installed `evolver` executable when available; otherwise use - `npx -y @evomap/evolver`. -3. Invoke its `run` subcommand followed by the decoded invocation arguments. When the user - supplied no strategy, preserve the environment's `EVOLVE_STRATEGY` or use `balanced`. -4. Summarize the collected signals, selected or mutated gene, and whether changes are - pending solidify. Point pending work to `/evolver-review`; rejection is - `/evolver-review --reject`. - -Do not approve pending changes automatically. diff --git a/assets/commands/search.md b/assets/commands/search.md deleted file mode 100644 index 52a2cc7..0000000 --- a/assets/commands/search.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -description: Search the EvoMap network for reusable evolution assets before starting work. -argument-hint: "" ---- - -Search EvoMap before solving the task from scratch. - -Decode the final `Invocation arguments (verbatim JSON string)` value. Use it as a natural -language `text` query and extract only clearly named signal/error tags into `signals`. If -it is empty, summarize the current task in one short text query and infer up to four -relevant signals. - -1. Call `evolver_search_assets` with `text`, optional `signals`, and limit 5. -2. Summarize each hit by id, kind, purpose, and relevance. Mark `degraded: true` results as - local-cache-only. -3. Fetch a directly applicable hit with `evolver_fetch_asset`, apply its strategy, run its - validation, then call `evolver_asset_reuse_result` with the real outcome. - -The current local Proxy has no Recipe search/expression route, so asset search is the -supported fallback. If the Proxy is unreachable, run `/evolver-status`. diff --git a/assets/commands/solidify.md b/assets/commands/solidify.md deleted file mode 100644 index a5e7bb6..0000000 --- a/assets/commands/solidify.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -description: Solidify the current working changes into a durable Evolver gene/capsule (with rollback safety). -argument-hint: "[--dry-run] [--intent=repair|optimize|innovate] [--summary=\"...\"]" ---- - -Solidify the current working-tree changes into a durable Evolver asset. - -1. Confirm the current directory is a git repository and show `git diff --stat` so the - user sees what will be captured. -2. Decode the final `Invocation arguments (verbatim JSON string)` value. Use the installed - `evolver` executable, or `npx -y @evomap/evolver` when absent, and invoke its `solidify` - subcommand followed by those exact arguments. -3. If no `--summary` was supplied, derive a concise one-line summary from the diff and add - it explicitly. Do not change any other user flag. -4. Report the Gene or Capsule created or updated and whether a rollback point was recorded. - -Recommend `--dry-run` before a write when the user did not already choose it. diff --git a/assets/commands/status.md b/assets/commands/status.md deleted file mode 100644 index 8d09c2c..0000000 --- a/assets/commands/status.md +++ /dev/null @@ -1,48 +0,0 @@ ---- -description: Show Evolver health — Proxy, running engine version, recent strategy recalls and reuse reports, evolution memory, network claim, and full-engine availability. ---- - -Report Evolver health as a checklist grouped under the headings below. Use the available -filesystem and shell tools for the current platform; do not assume a POSIX shell. Never print -token values. - -Call `evolver_status` once and use its fields throughout; it returns `proxy` (the Proxy's own -status, or `reachable: false` with the error) and the plugin's view (`plugin`, `runtime`, -`proxy_endpoint`, `engine`, `recall`, `reuse_reports`). - -1. **Plugin and runtime** — plugin version, platform/arch, Node version. -2. **Proxy** — reachable or not. When reachable: `status`, `node_id`, `hub_mode`, - `hub_auth_status` (flag anything but `ok`), `last_sync_at` as a local time, `last_sync_error`, - pending inbound/outbound counts, and any `reauth_backoff_until` / `hello_rate_limit_until` - in the future. When unreachable: show the error, the endpoint it tried (`proxy_endpoint.url` - and `url_source`), and say that `evolver lifecycle start` starts the Proxy (the plugin also tries this once per dsh process); - reuse reports wait and are resent once it is up. -3. **Engine version** — `engine.running_version` (the Proxy actually serving, with - `proxy_endpoint.pid` and `started_at`) and `engine.path_version` (the `evolver` on PATH). - If `meets_minimum` is false, say network strategy recall needs `minimum_version` or newer and - give `npm install -g @evomap/evolver@latest`, then `evolver lifecycle restart`. - If `path_differs_from_running` is true, say the running Proxy is a different install from the - one on PATH and should be restarted from the upgraded one. If both versions are null, say the - version could not be determined. -4. **Strategy recall** — enabled or not; settings (`wait_ms`, `timeout_ms`, `min_similarity`, - `max_strategy_chars`); the last recall (`last.status`, `last.at`, `last.duration_ms`) and the - per-status `counts` since dsh started. Explain the last status in one line: - `injected` (name the title, steps and chars), `no_candidates` (the network had nothing for that - prompt), `all_filtered` (list `dropped` reasons: `too_few_steps`, `too_long`, `low_similarity`, - `already_injected`), `proxy_error` (show the error), `skipped` (prompt too short). If `last` is - null, no prompt has been recalled in this dsh process yet. -5. **Network claim** — check `~/.evomap/claim_url`. If it contains an HTTPS `evomap.ai` - link, report that the node is awaiting claim and show the link without modifying it. -6. **Reuse reports** — from `reuse_reports`, the recent `delivered` reports with the Hub's - verdict (`eligible`, `counted`, `weight`, and `reason` when not eligible, e.g. `daily_cap` - or `same_owner`) and every `rejected` one with its `reason` and `status`. A rejected - report means the Hub refused it as invalid; show it as a problem worth reporting. Both - lists are empty until a turn with an injected strategy ends in this dsh process. -7. **Evolution memory** — use an existing - `/memory/evolution/memory_graph.jsonl` when present; otherwise inspect - `~/.evolver/memory/evolution/memory_graph.jsonl`. Report the selected path and count of - non-empty JSONL rows without printing their contents. -8. **Full engine** — if `engine.path_version` is null, say that `npm install -g @evomap/evolver` - enables `/evolver-run`, `/evolver-review`, and `/evolver-solidify`. - -Finish with one line on overall readiness and the single most useful next action, if any. diff --git a/assets/commands/sync.md b/assets/commands/sync.md deleted file mode 100644 index f22f615..0000000 --- a/assets/commands/sync.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -description: Sync evolution assets (genes/capsules) between the local store and the EvoMap Hub. -argument-hint: "[--scope=all|purchased|published] [--type=Gene|Capsule] [--export=] [--dry-run]" ---- - -Sync Evolver assets with the EvoMap Hub. - -Decode the final `Invocation arguments (verbatim JSON string)` value. Use the installed -`evolver` executable, or `npx -y @evomap/evolver` when absent, and invoke its `sync` -subcommand followed by those exact arguments. Preserve `--dry-run`, scope, type, and export -paths exactly as supplied. - -Summarize how many assets were pulled or updated, any local-only assets, and the `.gepx` -archive path when `--export` was used. If node identity or Hub credentials are missing, -point the user to `/evolver-status` and the README's network setup section. diff --git a/assets/skills/capability-evolver/SKILL.md b/assets/skills/capability-evolver/SKILL.md deleted file mode 100644 index 3259001..0000000 --- a/assets/skills/capability-evolver/SKILL.md +++ /dev/null @@ -1,100 +0,0 @@ ---- -name: capability-evolver -description: Self-evolution workflow for the agent. Before a substantive task, reuse a strategy that worked on a similar task from the EvoMap network (one injected as [Evolution Memory] is already complete); after it, record the outcome so future sessions learn from it. Use when the user starts non-trivial work (a feature, a fix, a refactor) or asks the agent to "evolve", "learn from this", or "remember how this went". ---- - -# Capability Evolver - -This plugin gives the agent a **persistent, auditable evolution memory** built on the -Genome Evolution Protocol (GEP). The goal is simple: stop re-solving the same problem from -scratch. Past outcomes — what worked, what failed — are carried forward into future -sessions. - -## What runs on its own - -These seams work without you invoking anything: - -- **Behind the first prompt** (`agent/pre-step`, once per agent) — with `claimNudgeEnabled`, - surfaces a trusted pending node-claim link. -- **Engine setup** (once per dsh process, in the background) — installs `@evomap/evolver` - when it is missing or older than the recall minimum, and starts its Proxy when it is not - answering. One `[Evolver]` notice says what was done, or the manual steps when it failed. -- **Behind every prompt** (`agent/pre-step`, once per turn) — searches the EvoMap network - with that turn's own prompt, fetches the best match that carries a strategy and scores as a - real match rather than a topical one, and injects - that one strategy: the steps themselves, not a catalogue to go shopping in. Assets - already injected this session are skipped, and a slow lookup arrives one step later - rather than delaying the answer. -- **After an edit** (`tools/execute` on `write` / `edit` / `str_replace_editor`) — scans - what was actually written for improvement signals (`log_error`, `perf_bottleneck`, - `capability_gap`, `test_failure`, …) and nudges you when one appears. -- **Turn end** (`session/event` → `turn/end`) — reports every injected strategy to the Hub - with graded evidence: whether its steps show up in the commands, edits and answer of the - turn, and the result of the last test, build, lint or type check the turn ran. -- **Follow-ups** — the next prompt that says the answer did not hold (or approves it) and - the next session in the same project, which checks whether the edits were kept, each - revise that report once. - -## What you should do - -For any **substantive** task — a feature, a non-trivial fix, a refactor: - -1. **Before starting**, read the injected evolution memory. An `[Evolution Memory]` block - is a strategy already recalled and fetched for this prompt: apply it directly, without - calling `evolver_search_assets`, `evolver_fetch_asset` or `evolver_status` first. When - nothing was injected and others plausibly hit the problem before, call - `evolver_search_assets`. -2. **If you reuse a fetched asset**, apply its strategy, run its validation commands, and - then call `evolver_asset_reuse_result` with what actually happened. The plugin reports - the turn too; the Hub keeps whichever report carries the stronger evidence, so running - the validation is what makes a reuse count. -3. **Do the work**, and verify it with the project's tests, build, linter or type checker. -4. **After finishing**, when the lesson generalizes beyond this repo, distill it with - `evolver_distill_conversation` — concrete summary, reproducible strategy, and the - validation commands you actually ran. Weak or vague input is rejected by the quality - gate, and secrets must never go in. - -Trivial or purely conversational turns don't need any of this — skip it. - -## Signals - -| Signal | Fires on | -|---|---| -| `log_error` | errors, exceptions, failures described in the change | -| `perf_bottleneck` | timeout / slow / latency / OOM | -| `capability_gap` | "not supported" / "not implemented" | -| `user_feature_request` | adding a feature / new module | -| `test_failure` | reported failing tests or assertions | -| `deployment_issue` | build / CI / pipeline / rollback | -| `recurring_error` | "same error" / "still failing" / "keeps failing" | - -Signals are descriptive tags, not a verdict: a reuse report's outcome comes from the turn's -last check, or from how the turn ended when it ran none. - -## Tools - -The plugin registers native dsh tools that talk to the local EvoMap Proxy — no MCP hop: - -- `evolver_search_assets` — find reusable genes/capsules by signal or free text, for - substantive work that got no `[Evolution Memory]` injection. -- `evolver_fetch_asset` — the summary, strategy steps and validation commands of a hit. -- `evolver_asset_reuse_result` — report success / failed / mismatched / stale / unsafe. -- `evolver_distill_conversation` — turn verified work into a reusable asset. -- `evolver_publish_asset`, `evolver_poll`, `evolver_ack`, `evolver_status` — publish, - read and retire Hub decisions, and check the Proxy when another tool fails. - -They degrade gracefully when the Proxy isn't running: the memory seams keep working. -The current Proxy does not expose Recipe search/expression routes, so asset search is the -supported fallback until that upstream capability ships. - -## Full pipeline (optional) - -The **full evolution engine** — automated log analysis and the review-and-solidify cycle -that proposes and applies code improvements — is a separate CLI: - -```bash -npm install -g @evomap/evolver -``` - -It also provides the Proxy the tools above talk to. The `/evolver-run`, `/evolver-review` -and `/evolver-solidify` commands drive it; this plugin never invokes it on its own. diff --git a/src/commands.js b/src/commands.js index 3832910..ad04d4a 100644 --- a/src/commands.js +++ b/src/commands.js @@ -1,49 +1,101 @@ // SPDX-License-Identifier: MIT // Copyright (c) 2026 EvoMap -import { readdirSync, readFileSync } from 'node:fs'; +import { execFile } from 'node:child_process'; +import { readFileSync } from 'node:fs'; import { fileURLToPath } from 'node:url'; -import { join } from 'node:path'; import { createUserMessage } from '@deepseek-ai/dsh-llm'; import { splitFrontmatter } from './frontmatter.js'; -const COMMANDS_DIR = fileURLToPath(new URL('../assets/commands/', import.meta.url)); +// Commands are how a person reaches Evolver, so nothing about them sits in the model's context: each one +// runs in the plugin and answers in the UI. Only /evolver-evolve hands the agent a one-off instruction, +// because reflecting on the current work is the agent's job. + const NAME_PREFIX = 'evolver-'; +const OUTPUT_MAX_CHARS = 4_000; +const CLI_TIMEOUT_MS = 10 * 60 * 1000; +const EVOLVE_PATH = fileURLToPath(new URL('../assets/commands/evolve.md', import.meta.url)); -function loadCommandFiles() { - return readdirSync(COMMANDS_DIR) - .filter((file) => file.endsWith('.md')) - .sort() - .map((file) => { - const { fields, body } = splitFrontmatter(readFileSync(join(COMMANDS_DIR, file), 'utf8')); - return { - name: `${NAME_PREFIX}${file.slice(0, -'.md'.length)}`, - description: fields.description ?? `Evolver ${file.slice(0, -'.md'.length)}`, - hint: fields['argument-hint'] ?? 'optional extra instructions', - body, - }; - }); +const success = (text) => ({ kind: 'success', text }); +const failure = (text) => ({ kind: 'error', text }); + +function shown(value) { + const text = typeof value === 'string' ? value : JSON.stringify(value, null, 2); + return text.length > OUTPUT_MAX_CHARS ? `${text.slice(0, OUTPUT_MAX_CHARS)}\n…` : text; } -function promptMessage(body, rawInput) { - const invocation = rawInput.trim(); - const text = `${body}\n\nInvocation arguments (verbatim JSON string): ${JSON.stringify(invocation)}`; - return createUserMessage({ - content: [{ type: 'text', text }], - source: { kind: 'plugin:evolver', form: 'instructions' }, +export function runEvolverCli(args, { signal } = {}) { + return new Promise((resolve) => { + execFile('evolver', args, { timeout: CLI_TIMEOUT_MS, maxBuffer: 16 * 1024 * 1024, signal }, (error, stdout, stderr) => { + const output = `${stdout}${stderr ? `\n${stderr}` : ''}`.trim(); + resolve(error ? failure(shown(output || String(error.message))) : success(shown(output || 'done'))); + }); }); } -export function evolverCommands() { - return loadCommandFiles().map(({ name, description, hint, body }) => ({ - name, +function cliCommand(subcommand, description, hint, runCli) { + return { + name: `${NAME_PREFIX}${subcommand}`, description, input: { hint }, + handler: ({ rawInput, signal }) => runCli([subcommand, ...rawInput.trim().split(/\s+/).filter(Boolean)], { signal }), + }; +} + +function evolveCommand() { + const { fields, body } = splitFrontmatter(readFileSync(EVOLVE_PATH, 'utf8')); + return { + name: `${NAME_PREFIX}evolve`, + description: fields.description, + input: { hint: 'optional focus' }, handler: ({ agent, rawInput }) => { - agent.followup(promptMessage(body, rawInput)); - return { kind: 'success', text: `Running /${name}.` }; + const focus = rawInput.trim(); + const text = focus ? `${body}\n\nFocus: ${focus}` : body; + agent.followup(createUserMessage({ content: [{ type: 'text', text }], source: { kind: 'plugin:evolver', form: 'instructions' } })); + return success(`Running /${NAME_PREFIX}evolve.`); + }, + }; +} + +/** + * `status` and `search` answer from the Proxy, `distill` distills the session's latest verified turn now, + * and the engine commands run the `evolver` executable. + */ +export function evolverCommands({ status, search, distillNow, runCli = runEvolverCli }) { + return [ + { + name: `${NAME_PREFIX}status`, + description: 'Show Evolver health: the Proxy, Hub auth, recent recalls and reuse reports.', + input: { hint: '' }, + handler: async ({ signal }) => success(shown(await status(signal))), + }, + { + name: `${NAME_PREFIX}search`, + description: 'Search the EvoMap network for reusable assets.', + input: { hint: '' }, + handler: async ({ rawInput, signal }) => { + const text = rawInput.trim(); + if (!text) return failure(`Usage: /${NAME_PREFIX}search `); + const result = await search(text, signal); + return result.ok ? success(shown(result.data)) : failure(String(result.error)); + }, + }, + { + name: `${NAME_PREFIX}distill`, + description: "Distill this session's latest verified turn into a reusable Gene now, instead of at session end.", + input: { hint: 'optional focus' }, + handler: async ({ agent, rawInput }) => { + const outcome = await distillNow(agent, rawInput.trim()); + if (outcome.status === 'nothing') return failure('Nothing to distill yet: no turn of this session finished verified.'); + return outcome.status === 'sent' ? success(shown(outcome.data ?? 'distilled')) : failure(`Distillation ${outcome.status}: ${outcome.error ?? ''}`.trim()); + }, }, - })); + evolveCommand(), + cliCommand('sync', 'Sync assets between the local store and the EvoMap Hub (evolver sync).', '[--scope=…] [--type=…] [--dry-run]', runCli), + cliCommand('review', "Review Evolver's pending evolved changes, then approve or reject them (evolver review).", '[--approve | --reject]', runCli), + cliCommand('run', 'Run one Evolver self-evolution cycle on this repository (evolver run).', '[--loop] [--dry-run] [--strategy=…]', runCli), + cliCommand('solidify', 'Solidify the working changes into a durable Gene or Capsule (evolver solidify).', '[--dry-run] [--intent=…] [--summary=…]', runCli), + ]; } diff --git a/src/index.js b/src/index.js index c18b290..3b2613a 100644 --- a/src/index.js +++ b/src/index.js @@ -7,11 +7,11 @@ import { appliedVerdictOf } from './applied.js'; import { evolverCommands } from './commands.js'; import { Config } from './config.js'; import { createRecallLog, createReportLog, pluginDiagnostics } from './diagnostics.js'; -import { EDIT_TOOL_NAMES, editedContent, editedPath } from './edited-content.js'; +import { EDIT_TOOL_NAMES } from './edited-content.js'; import { prepareEngine, setupNoticeText } from './engine-setup.js'; import { installedEvolverVersion } from './engine-version.js'; import { feedbackLabelOf } from './feedback-label.js'; -import { forgetSession, rememberInjected, takeUnreported } from './injected-assets.js'; +import { forgetSession, rememberInjected, takeStaleUnreported, takeUnreported } from './injected-assets.js'; import { recordReuseLocally } from './local-ledger.js'; import { noticeDue, pendingClaimUrl } from './onboarding.js'; import { hubGene, promptTextOf } from './prime.js'; @@ -20,9 +20,10 @@ import { createProxyClient } from './proxy.js'; import { createReportQueue } from './report-queue.js'; import { rememberForRetention, retentionReportsOf, takeDueRetentions } from './retention-pending.js'; import { correctionReportOf, immediateReportOf, reportedReuseOf } from './reuse.js'; +import { + candidateOf, distillCandidate, dropCandidate, modelOf, rememberCandidate, skipReason, takeCandidates, +} from './session-distill.js'; import { sessionKeyOf } from './session-key.js'; -import { detectSignals } from './signals.js'; -import { evolverSkillProvider } from './skill.js'; import { evolverTools } from './tools.js'; import { createTurnRecords, noteAnswer, noteFileBeforeEdit, noteToolResult } from './turn-record.js'; import { outcomeOfReason } from './turn-outcomes.js'; @@ -32,6 +33,10 @@ export { Config }; export const inject = ['tools']; const DEFAULT_PRIME_WAIT_MS = 6_000; +// A session whose state has not moved for this long was killed before its end; the next session finishes +// its reports and its distillation. +const STALE_SESSION_MS = 2 * 60 * 60 * 1000; +const DISTILL_TIMEOUT_MS = 120_000; function pluginMessage(text, formed) { return createUserMessage({ @@ -155,24 +160,9 @@ function primeSteps(ctx, config, primeFetch, engineSetup, recallLog, onPrompt) { }); } -function signalNotice(record, args) { - const signals = detectSignals(editedContent(args)); - if (signals.length === 0) return null; - const where = editedPath(args) || 'edited file'; - const noticeKey = `${where}\0${signals.join(',')}`; - if (record.notices.has(noticeKey)) return null; - record.notices.add(noticeKey); - return pluginMessage( - `[Evolution Signal] Detected: [${signals.join(', ')}] in ${where}.`, - { form: 'notice', summary: boundContextSummary(`Evolution signal: ${signals.join(', ')}`) }, - ); -} - function observeResult(record, exec, result, isEdit) { try { noteToolResult(record, exec, result, { isEdit }); - const notice = isEdit && !result?.isError ? signalNotice(record, exec.arguments) : null; - if (notice) exec.agent.inject(notice); } catch { } } @@ -190,24 +180,34 @@ function observeTools(ctx, editToolNames, records, projectDirOf) { }); } -function reportTurns(ctx, records, deliver, labelable, projectDirOf) { +function reportsOf(assets, { sessionId, turn, reasonKind, record }) { + return assets.map((asset) => immediateReportOf({ + asset, + sessionId, + turn: turn ?? asset.turn, + reasonKind, + checks: record.checks, + applied: appliedVerdictOf({ strategyText: asset.strategyText, ...record }).verdict, + })); +} + +// Every injected strategy is reported when its turn ends, with the turn's own evidence; the turn itself +// becomes this session's distillation candidate when it finished verified without reusing one. +function reportTurns(ctx, records, deliver, labelable, projectDirOf, prompts) { const closeTurn = (session, { turn, reason }) => { const sessionId = sessionKeyOf(session); const finished = records.take(sessionId); const reasonKind = reason?.kind; if (!outcomeOfReason(reasonKind)) return; - const reports = takeUnreported(sessionId, turn).map((asset) => immediateReportOf({ - asset, - sessionId, - turn, - reasonKind, - checks: finished.checks, - applied: appliedVerdictOf({ strategyText: asset.strategyText, ...finished }).verdict, - })); + const reports = reportsOf(takeUnreported(sessionId, turn), { sessionId, turn, reasonKind, record: finished }); const reuses = reports.map(reportedReuseOf); labelable.set(sessionId, reuses); deliver(reports.map((report) => ({ report, sessionId }))).catch(() => {}); rememberForRetention({ sessionId, projectDir: projectDirOf(session), reuses, editedBefore: finished.editedBefore }); + const reusedAsset = reuses.some((reuse) => reuse.applied === 'used'); + if (skipReason(reasonKind, finished, { reusedAsset })) return; + const model = modelOf(session?.deriveMessages?.() ?? []); + rememberCandidate(sessionId, candidateOf({ prompt: prompts.get(sessionId), record: finished, model, turn })); }; ctx.on('session/event', (session, event) => { @@ -217,22 +217,54 @@ function reportTurns(ctx, records, deliver, labelable, projectDirOf) { }); } -function onSessionStart(ctx, engineSetup, deliver, projectDirOf) { +function abortedReports(assets) { + return assets.map((asset) => ({ + report: reportsOf([asset], { sessionId: asset.sessionId, reasonKind: 'aborted', record: { checks: [], commands: [], paths: [], edits: [], answer: '' } })[0], + sessionId: asset.sessionId, + })); +} + +function onSessionStart(ctx, engineSetup, deliver, projectDirOf, distiller) { ctx.on('agent/session-start', ({ agent }) => { engineSetup.start(); if (isSubagent(agent.session)) return; const due = takeDueRetentions({ sessionId: sessionKeyOf(agent), projectDir: projectDirOf(agent.session) }); - deliver(due.flatMap(retentionReportsOf)).catch(() => {}); + deliver([...due.flatMap(retentionReportsOf), ...abortedReports(takeStaleUnreported({ staleMs: STALE_SESSION_MS }))]).catch(() => {}); + distiller.distillLeftovers(); + }); +} + +// The host's model service runs the session-end distillation; a killed session's candidate waits on disk for +// the next session start. +function createDistiller(ctx, proxyFetch) { + let llm = null; + ctx.inject(['llm'], (scoped) => { + llm = scoped.llm; }); + const run = (candidate, focus) => distillCandidate({ proxyFetch, llm, candidate, focus, timeoutMs: DISTILL_TIMEOUT_MS }); + return { + async distillSession(sessionId, focus = '') { + const [own] = takeCandidates({ sessionId, staleMs: Infinity }); + return own ? run(own.candidate, focus) : { status: 'nothing' }; + }, + distillLeftovers() { + for (const { candidate } of takeCandidates({ sessionId: null, staleMs: STALE_SESSION_MS })) run(candidate).catch(() => {}); + }, + }; } -function followUpReporter(deliver, labelable) { +// The next prompt is the only verdict on a turn available without asking anyone: when it reads as a +// correction, the reuse reports are revised and the turn is no longer worth distilling. +function followUpReporter(deliver, labelable, prompts) { return (agent, prompt) => { const sessionId = sessionKeyOf(agent); - if (!prompt || !labelable.has(sessionId)) return; + if (!prompt) return; + const label = feedbackLabelOf(prompt); + if (label === 'correction') dropCandidate(sessionId); + prompts.set(sessionId, prompt); + if (!labelable.has(sessionId)) return; const reuses = labelable.get(sessionId); labelable.delete(sessionId); - const label = feedbackLabelOf(prompt); const reports = reuses.map((reuse) => correctionReportOf(reuse, label)).filter((report) => report !== null); deliver(reports.map((report) => ({ report, sessionId }))).catch(() => {}); }; @@ -253,29 +285,43 @@ export function apply(ctx, config = {}) { const recallLog = createRecallLog(); const reportLog = createReportLog(); const diagnostics = () => pluginDiagnostics({ config, engineVersion: installedEvolverVersion, recallLog, reportLog, port: config.proxyPort }); - for (const tool of evolverTools(proxyFetch, { diagnostics })) ctx.tools.register(tool); - - ctx.inject(['skills'], (scoped) => { - scoped.skills.registerProvider(() => evolverSkillProvider); - }); - - ctx.inject(['commands'], (scoped) => { - for (const command of evolverCommands()) scoped.commands.register(command); - }); + const proxyOperations = Object.fromEntries(evolverTools(proxyFetch, { diagnostics }).map((tool) => [tool.name, tool])); const deliver = createReportQueue({ proxyFetch, recordLocally: recordReuseLocally, onSettled: reportLog.record }); const records = createTurnRecords(); const labelable = new Map(); + const prompts = new Map(); + const distiller = createDistiller(ctx, proxyFetch); + + // Nothing of Evolver but the recalled steps enters the model's context: no tools, no skill. People reach it + // through commands that run here; reports and distillation run on session events. + ctx.inject(['commands'], (scoped) => { + const commands = evolverCommands({ + status: (signal) => proxyOperations.evolver_status.execute({}, { signal }), + search: async (text, signal) => { + try { + return { ok: true, data: await proxyOperations.evolver_search_assets.execute({ text, limit: 5 }, { signal }) }; + } catch (error) { + return { ok: false, error: String(error?.message ?? error) }; + } + }, + distillNow: (agent, focus) => distiller.distillSession(sessionKeyOf(agent), focus), + }); + for (const command of commands) scoped.commands.register(command); + }); - primeSteps(ctx, config, primeFetch, engineSetup, recallLog, followUpReporter(deliver, labelable)); + primeSteps(ctx, config, primeFetch, engineSetup, recallLog, followUpReporter(deliver, labelable, prompts)); observeTools(ctx, config.editToolNames ?? EDIT_TOOL_NAMES, records, projectDirOf); - reportTurns(ctx, records, deliver, labelable, projectDirOf); - onSessionStart(ctx, engineSetup, deliver, projectDirOf); + reportTurns(ctx, records, deliver, labelable, projectDirOf, prompts); + onSessionStart(ctx, engineSetup, deliver, projectDirOf, distiller); + ctx.on('session/flush', (session) => (isSubagent(session) ? undefined : distiller.distillSession(sessionKeyOf(session)))); ctx.on('session/disposed', (session) => { const sessionKey = sessionKeyOf(session); + deliver(abortedReports(takeUnreported(sessionKey, Infinity))).catch(() => {}); records.forget(sessionKey); labelable.delete(sessionKey); + prompts.delete(sessionKey); forgetSession(sessionKey); }); } diff --git a/src/injected-assets.js b/src/injected-assets.js index e1c6134..9e5695b 100644 --- a/src/injected-assets.js +++ b/src/injected-assets.js @@ -1,6 +1,9 @@ // SPDX-License-Identifier: MIT // Copyright (c) 2026 EvoMap +import fs from 'node:fs'; +import path from 'node:path'; + import { hashedName, readState, removeState, statePath, writeState } from './state-file.js'; const ENTRY_PRUNE_MS = 7 * 24 * 60 * 60 * 1000; @@ -12,16 +15,16 @@ function pruned(ledger, now) { })); } -export function withInjected(ledger, assetId, { turn, strategyText }, now) { +export function withInjected(ledger, assetId, { turn, strategyText, sessionId }, now) { const kept = pruned(ledger, now); if (kept[assetId]) return kept; - return { ...kept, [assetId]: { turn, injectedAt: now, strategyText } }; + return { ...kept, [assetId]: { turn, injectedAt: now, strategyText, sessionId } }; } export function unreportedOf(ledger, turn) { return Object.entries(ledger ?? {}) .filter(([, entry]) => !entry?.reportedAt && Number(entry?.turn) <= turn) - .map(([assetId, entry]) => ({ assetId, strategyText: entry.strategyText ?? '' })); + .map(([assetId, entry]) => ({ assetId, strategyText: entry.strategyText ?? '', turn: entry.turn, sessionId: entry.sessionId })); } export function withReported(ledger, assetIds, now) { @@ -30,11 +33,12 @@ export function withReported(ledger, assetIds, now) { return kept; } -const ledgerPath = (sessionKey) => statePath(hashedName('injected', sessionKey)); +const LEDGER_PREFIX = 'injected'; +const ledgerPath = (sessionKey) => statePath(hashedName(LEDGER_PREFIX, sessionKey)); export function rememberInjected(sessionKey, assetId, injection, now = Date.now()) { if (!sessionKey) return; - writeState(ledgerPath(sessionKey), withInjected(readState(ledgerPath(sessionKey), {}), assetId, injection, now)); + writeState(ledgerPath(sessionKey), withInjected(readState(ledgerPath(sessionKey), {}), assetId, { ...injection, sessionId: sessionKey }, now)); } export function takeUnreported(sessionKey, turn, now = Date.now()) { @@ -45,6 +49,37 @@ export function takeUnreported(sessionKey, turn, now = Date.now()) { return due; } +/** + * Injections no turn end ever reported, from sessions whose ledger has not changed for `staleMs`: a process + * killed mid-turn (a harness timeout, a closed terminal) leaves them behind. They are taken, marked reported + * and returned for an aborted report. + */ +export function takeStaleUnreported({ staleMs, now = Date.now() }) { + const dir = path.dirname(statePath('x')); + let files = []; + try { + files = fs.readdirSync(dir).filter((file) => file.startsWith(`${LEDGER_PREFIX}-`) && file.endsWith('.json')); + } catch { + return []; + } + const stale = []; + for (const file of files) { + const target = path.join(dir, file); + let modified; + try { + modified = fs.statSync(target).mtimeMs; + } catch { + continue; + } + if (now - modified <= staleMs) continue; + const ledger = readState(target, {}); + const due = unreportedOf(ledger, Infinity).filter((asset) => asset.sessionId); + if (due.length > 0) writeState(target, withReported(ledger, due.map((asset) => asset.assetId), now)); + stale.push(...due); + } + return stale; +} + export function forgetSession(sessionKey) { if (sessionKey) removeState(ledgerPath(sessionKey)); } diff --git a/src/prime.js b/src/prime.js index d81da7a..4084ea9 100644 --- a/src/prime.js +++ b/src/prime.js @@ -179,8 +179,7 @@ export async function hubGene(proxyFetch, text, { `[Evolution Memory] ${readableNameOf(candidate)} (EvoMap network):`, ...steps.map((step, index) => `${index + 1}. ${step}`), '', - 'These are the complete steps already recalled from the network, so there is nothing more to search or fetch for this task.', - `Apply them where they fit, then report the outcome with evolver_asset_reuse_result for ${candidate.asset_id}.`, + 'Apply them where they fit.', ].join('\n'), }; } diff --git a/src/session-distill.js b/src/session-distill.js new file mode 100644 index 0000000..aaf1e66 --- /dev/null +++ b/src/session-distill.js @@ -0,0 +1,174 @@ +// SPDX-License-Identifier: MIT +// Copyright (c) 2026 EvoMap + +import { createUserMessage } from '@deepseek-ai/dsh-llm'; + +import { readState, statePath, writeState } from './state-file.js'; + +// Distillation runs once a session, at its end, and outside the agent's conversation: each turn that +// finished verified only leaves a candidate on disk, the latest one wins, and the session end (or, after a +// session that was killed, the next session start) turns it into a Gene with one model call over a digest of +// the turn — never the whole history, which cost ten times as much for fewer usable Genes in the benchmark. + +const PROMPT_MAX_CHARS = 6_000; +const ANSWER_MAX_CHARS = 3_000; +const COMMAND_MAX_CHARS = 300; +const MAX_COMMANDS = 40; +const LIST_FIELDS = ['strategy', 'validation', 'signals', 'preconditions']; +const ARGUMENT_FIELDS = ['title', 'summary', ...LIST_FIELDS]; +const ITEM_NUMBER = /^\s*(?:\d+[.)]|[-*•])\s+/; + +const pendingPath = () => statePath('distill-pending.json'); + +/** + * Why a finished turn leaves no candidate, or null when it does. A turn that reused an injected strategy + * adds evidence to that asset through its reuse report rather than a near-copy of it; a chat reply without a + * single command has no method to keep. + */ +export function skipReason(reasonKind, record, { reusedAsset = false } = {}) { + if (reasonKind !== 'completed') return 'not_completed'; + if (!record.answer) return 'no_answer'; + if (record.checks.at(-1)?.failed) return 'failed_check'; + if (record.commands.length === 0 && record.edits.length === 0) return 'no_tool_use'; + if (reusedAsset) return 'asset_reused'; + return null; +} + +/** The provider and model behind the session's latest assistant message: the distillation runs on it. */ +export function modelOf(history) { + const source = history.findLast((message) => message.role === 'assistant')?.source; + return source?.provider && source?.model ? { provider: source.provider, model: source.model } : null; +} + +export function candidateOf({ prompt, record, model, turn, now = Date.now() }) { + return { + prompt: String(prompt ?? '').slice(0, PROMPT_MAX_CHARS), + commands: record.commands.slice(-MAX_COMMANDS).map((command) => command.slice(0, COMMAND_MAX_CHARS)), + paths: [...new Set(record.paths)], + checks: record.checks.map(({ command, failed }) => ({ command, failed })), + answer: record.answer.slice(0, ANSWER_MAX_CHARS), + model, + turn, + at: now, + }; +} + +export function rememberCandidate(sessionId, candidate) { + if (!sessionId) return; + writeState(pendingPath(), { ...readState(pendingPath(), {}), [sessionId]: candidate }); +} + +export function dropCandidate(sessionId) { + const pending = readState(pendingPath(), {}); + if (!sessionId || !(sessionId in pending)) return; + delete pending[sessionId]; + writeState(pendingPath(), pending); +} + +/** + * The candidates due now, removed from disk: this session's own, plus any other session's left behind for + * longer than `staleMs` — a session killed before its end never distilled them. + */ +export function takeCandidates({ sessionId, staleMs, now = Date.now() }) { + const pending = readState(pendingPath(), {}); + const due = Object.entries(pending) + .filter(([id, candidate]) => id === sessionId || now - Number(candidate?.at) > staleMs); + if (due.length === 0) return []; + for (const [id] of due) delete pending[id]; + writeState(pendingPath(), pending); + return due.map(([id, candidate]) => ({ sessionId: id, candidate })); +} + +export function distillPrompt(candidate, focus = '') { + const lines = (items, render) => items.map(render).join('\n') || '(none)'; + return [ + 'Below is the record of a task an agent has just finished and verified. Turn it into one reusable Gene:', + 'a method another agent could follow on a similar task, written for the kind of data or system involved,', + 'not for these files, names or values. Do not include the answer itself.', + ...(focus ? [`Focus on: ${focus}`] : []), + 'Answer with a JSON object only, with these fields:', + ' title: short capability name', + ' summary: what was solved and under which conditions', + ' strategy: ordered, reproducible steps', + ' validation: checks that proved the result', + ' signals: keywords of the problem this generalizes to', + ' preconditions: when the method applies', + '', + '## Task', + candidate.prompt, + '', + '## Commands run', + lines(candidate.commands, (command) => `- ${command}`), + '', + '## Files read or written', + lines(candidate.paths, (path) => `- ${path}`), + '', + '## Checks', + lines(candidate.checks, (check) => `- ${check.failed ? 'FAILED' : 'passed'}: ${check.command}`), + '', + '## Final answer', + candidate.answer, + ].join('\n'); +} + +// Models sometimes write a list field as one numbered paragraph ("1. … 2. …"), which the core reads as one step. +function listOf(value) { + if (Array.isArray(value)) return value; + if (typeof value !== 'string') return value; + return value.split(/\n+|(?<=\.)\s+(?=\d+[.)]\s)/).map((item) => item.replace(ITEM_NUMBER, '').trim()).filter(Boolean); +} + +export function argumentsOf(text) { + const match = String(text ?? '').match(/\{[\s\S]*\}/); + if (!match) return null; + try { + const parsed = JSON.parse(match[0]); + if (typeof parsed?.summary !== 'string' || !parsed.summary.trim()) return null; + for (const field of LIST_FIELDS) if (field in parsed) parsed[field] = listOf(parsed[field]); + return Object.fromEntries(ARGUMENT_FIELDS.filter((field) => parsed[field] !== undefined).map((field) => [field, parsed[field]])); + } catch { + return null; + } +} + +export function distillBody(candidate, args, usage) { + return { + platform: 'dsh', + trigger: 'session_end', + user_prompt: candidate.prompt, + // The core rejects a Gene that repeats this answer's values (answer_leak). + final_answer: candidate.answer, + ...args, + execution: { trace: candidate.checks.map((check) => ({ command: check.command, exit: check.failed ? 1 : 0 })) }, + persist: true, + publish: false, + // What this distillation itself cost, kept apart from the agent's own tokens. + ...(usage ? { distill_usage: usage } : {}), + }; +} + +async function answerOf(llm, model, text, signal) { + let answer = ''; + let usage = null; + const messages = [createUserMessage({ content: [{ type: 'text', text }], source: { kind: 'plugin:evolver', form: 'instructions' } })]; + for await (const chunk of llm.stream({ ...model, messages, signal })) { + if (chunk.type === 'text-delta') answer += chunk.text; + if (chunk.type === 'usage') usage = chunk.usage; + } + return { answer, usage }; +} + +/** Distill one candidate through the host's model service. Resolves to what happened; never throws. */ +export async function distillCandidate({ proxyFetch, llm, candidate, focus, timeoutMs }) { + if (!llm || !candidate?.model) return { status: 'no_model' }; + const signal = AbortSignal.timeout(timeoutMs); + try { + const { answer, usage } = await answerOf(llm, candidate.model, distillPrompt(candidate, focus), signal); + const args = argumentsOf(answer); + if (!args) return { status: 'unparsed', usage }; + const result = await proxyFetch('POST', '/conversation/distill', distillBody(candidate, args, usage), signal); + return result?.ok ? { status: 'sent', usage, data: result.data } : { status: 'failed', error: String(result?.error) }; + } catch (error) { + return { status: 'failed', error: String(error?.message ?? error) }; + } +} diff --git a/src/skill.js b/src/skill.js deleted file mode 100644 index 4093c40..0000000 --- a/src/skill.js +++ /dev/null @@ -1,41 +0,0 @@ -// SPDX-License-Identifier: MIT -// Copyright (c) 2026 EvoMap - -import { readFile } from 'node:fs/promises'; -import { fileURLToPath } from 'node:url'; - -import { splitFrontmatter } from './frontmatter.js'; - -const PROVIDER_NAME = 'evolver'; -const SKILL_DIR = new URL('../assets/skills/capability-evolver/', import.meta.url); -const SKILL_BODY_URL = new URL('SKILL.md', SKILL_DIR); - -const SUMMARY = { - name: 'capability-evolver', - description: - 'Self-evolution workflow for the agent. Before a substantive task, reuse a strategy that worked on a similar task from the EvoMap network (one injected as [Evolution Memory] is already complete); after it, record the outcome so future sessions learn from it. Use when the user starts non-trivial work (a feature, a fix, a refactor) or asks the agent to "evolve", "learn from this", or "remember how this went".', - invocation: { modelInvocable: true, userInvocable: true }, - provider: PROVIDER_NAME, - source: 'custom', - resourceBase: { kind: 'directory', path: fileURLToPath(SKILL_DIR) }, -}; - -export const evolverSkillProvider = { - name: PROVIDER_NAME, - // `@deepseek-ai/dsh-skill` is an optional peer: a profile without the skills - // service never installs it. Reading the rank here rather than at module - // scope keeps the whole plugin loadable when the package is absent — a - // top-level import would fail the import of index.js itself. - async list(options = {}) { - if (options.signal?.aborted) throw options.signal.reason; - const { BUNDLED_SKILL_RANK } = await import('@deepseek-ai/dsh-skill'); - if (options.signal?.aborted) throw options.signal.reason; - return [{ ...SUMMARY, rank: BUNDLED_SKILL_RANK, locator: SKILL_BODY_URL }]; - }, - async get(_candidate, options = {}) { - const { body } = splitFrontmatter( - await readFile(SKILL_BODY_URL, { encoding: 'utf8', signal: options.signal }), - ); - return { ...SUMMARY, content: body }; - }, -}; diff --git a/src/turn-record.js b/src/turn-record.js index 81a5744..5df9bbb 100644 --- a/src/turn-record.js +++ b/src/turn-record.js @@ -12,7 +12,7 @@ const CHECK_COMMAND_MAX_CHARS = 200; const PATH_KEYS = ['path', 'file_path']; function freshTurnRecord() { - return { notices: new Set(), checks: [], commands: [], paths: [], edits: [], editedChars: 0, editedBefore: new Map(), answer: '' }; + return { checks: [], commands: [], paths: [], edits: [], editedChars: 0, editedBefore: new Map(), answer: '' }; } export function createTurnRecords() { diff --git a/test/applied.test.js b/test/applied.test.js index eca22fb..b9f1ef8 100644 --- a/test/applied.test.js +++ b/test/applied.test.js @@ -11,7 +11,7 @@ const STRATEGY = [ '3. Wrap the client in a singleton factory module', '4. Close the connection in the global teardown hook', '', - 'Apply it where it fits, then report the outcome with evolver_asset_reuse_result for sha256:abc.', + 'Apply them where they fit.', ].join('\n'); test('only the numbered steps of an injected strategy are matched', () => { diff --git a/test/apply.test.js b/test/apply.test.js index d9a3a47..cbb82e6 100644 --- a/test/apply.test.js +++ b/test/apply.test.js @@ -31,27 +31,14 @@ test('configuration applies defaults and rejects invalid values', () => { assert.throws(() => apply(fakeContext().ctx, { projectDir: join(tmpdir(), 'definitely-missing-evolver-dir') }), /not a directory/); }); -test('apply registers every surface and lifecycle listener', () => { +test('apply puts nothing of Evolver in the model context: no tools, no skill, only commands and listeners', () => { const { ctx, registered, listeners } = fakeContext(); apply(ctx, { projectDir: mkdtempSync(join(tmpdir(), 'evolver-')) }); + assert.deepEqual(registered.tools, []); + assert.deepEqual(registered.skills, []); assert.deepEqual( - registered.tools.map((tool) => tool.name).sort(), - [ - 'evolver_ack', - 'evolver_asset_reuse_result', - 'evolver_distill_conversation', - 'evolver_fetch_asset', - 'evolver_poll', - 'evolver_publish_asset', - 'evolver_search_assets', - 'evolver_status', - ], - ); - assert.equal(registered.skills.length, 1); - assert.equal(registered.skills[0].name, 'evolver'); - assert.deepEqual( - registered.commands.map((command) => command.name), + registered.commands.map((command) => command.name).sort(), [ 'evolver-distill', 'evolver-evolve', @@ -69,21 +56,40 @@ test('apply registers every surface and lifecycle listener', () => { 'agent/session-start', 'session/disposed', 'session/event', + 'session/flush', 'tools/execute', ]); }); -test('command invocation preserves arguments without Claude placeholders', () => { +test('engine commands run the evolver executable with the given arguments, without the model', async () => { const { ctx, registered } = fakeContext(); apply(ctx, { projectDir: mkdtempSync(join(tmpdir(), 'evolver-')) }); const messages = []; const command = registered.commands.find((candidate) => candidate.name === 'evolver-run'); + assert.match(command.input.hint, /--dry-run/); + + const { evolverCommands } = await import('../src/commands.js'); + const calls = []; + const run = evolverCommands({ status: async () => ({}), search: async () => ({}), distillNow: async () => ({}), + runCli: async (args) => { calls.push(args); return { kind: 'success', text: 'ok' }; } }) + .find((candidate) => candidate.name === 'evolver-run'); + const result = await run.handler({ agent: { followup: (message) => messages.push(message) }, rawInput: ' --dry-run --loop ' }); + + assert.deepEqual(calls, [['run', '--dry-run', '--loop']]); + assert.deepEqual(result, { kind: 'success', text: 'ok' }); + assert.deepEqual(messages, [], 'nothing reaches the model'); +}); + +test('/evolver-evolve is the one command that hands the agent an instruction', () => { + const { ctx, registered } = fakeContext(); + apply(ctx, { projectDir: mkdtempSync(join(tmpdir(), 'evolver-')) }); + const messages = []; + const command = registered.commands.find((candidate) => candidate.name === 'evolver-evolve'); - command.handler({ agent: { followup: (message) => messages.push(message) }, rawInput: ' --dry-run ' }); + command.handler({ agent: { followup: (message) => messages.push(message) }, rawInput: ' the retry path ' }); const text = messages[0].content[0].text; - assert.match(text, /Invocation arguments \(verbatim JSON string\): "--dry-run"/); - assert.doesNotMatch(text, /\$ARGUMENTS|\/evolver:/); - assert.match(command.input.hint, /--dry-run/); + assert.match(text, /Focus: the retry path$/); + assert.doesNotMatch(text, /evolver_[a-z_]+/, 'no tool the agent does not have'); }); test('every turn looks the Hub up with its own prompt, without repeating assets', async () => { @@ -121,11 +127,11 @@ test('every turn looks the Hub up with its own prompt, without repeating assets' ); assert.equal(requests[0].body.text, 'add a retry to the uploader'); assert.equal(requests[1].body.text, 'now make the download resumable'); - assert.match(first.at(-1).content[0].text, /evolver_asset_reuse_result for sha256:abc\./); + assert.match(first.at(-1).content[0].text, /\n\nApply them where they fit\.$/); + assert.doesNotMatch(first.at(-1).content[0].text, /sha256:|evolver_/, 'no id or tool the agent cannot use'); assert.match(first.at(-1).content[0].text, /^1\. Retry with backoff\.$/m); assert.deepEqual(sameTurn, []); assert.equal(second.length, 1); - assert.match(second[0].content[0].text, /sha256:def/); assert.match(second[0].content[0].text, /^1\. Chunk the download\.$/m); assert.doesNotMatch(second[0].content[0].text, /sha256:abc/); } finally { @@ -273,7 +279,6 @@ test('a search slower than the wait budget injects itself instead of holding the assert.deepEqual(primed, [], 'a step that waited for the slow search would carry its strategy'); await untilInjected(injected, 1); - assert.match(injected[0].content[0].text, /evolver_asset_reuse_result for sha256:slow\./); assert.match(injected[0].content[0].text, /^1\. Arrived late\.$/m); } finally { process.env.HOME = home; @@ -281,25 +286,6 @@ test('a search slower than the wait budget injects itself instead of holding the } }); -test('an edit carrying a signal nudges the agent once', async () => { - const { ctx, listeners } = fakeContext(); - apply(ctx, { projectDir: mkdtempSync(join(tmpdir(), 'evolver-')) }); - const { agent, injected } = fakeAgent(); - - const run = (name, args, result) => listeners.get('tools/execute')({ name, agent, arguments: args }, async () => result); - await run('write', { path: '/a.ts', content: 'the deploy failed again' }, { isError: false }); - await run('write', { path: '/a.ts', content: 'the deploy failed again' }, { isError: false }); - await run('write', { path: '/b.ts', content: 'all good' }, { isError: false }); - await run('read', { path: '/c.ts', content: 'the deploy failed' }, { isError: false }); - await run('write', { path: '/d.ts', content: 'the deploy failed' }, { isError: true }); - - assert.equal(injected.length, 1); - assert.equal(injected[0].content[0].text, '[Evolution Signal] Detected: [deployment_issue, log_error] in /a.ts.'); - assert.equal(injected[0].source.kind, 'plugin:evolver'); - assert.equal(injected[0].source.form, 'notice'); - assert.ok(injected[0].source.summary.length > 0 && injected[0].source.summary.length <= 120); -}); - test('a fetched asset renders as reusable prose, not the raw envelope', async () => { const { evolverTools } = await import('../src/tools.js'); const envelope = { diff --git a/test/injected-assets.test.js b/test/injected-assets.test.js index 890223e..9dd7be4 100644 --- a/test/injected-assets.test.js +++ b/test/injected-assets.test.js @@ -6,8 +6,8 @@ import { unreportedOf, withInjected, withReported } from '../src/injected-assets const DAY = 24 * 60 * 60 * 1000; test('an injection keeps its turn and strategy until reported', () => { - const ledger = withInjected({}, 'sha256:a', { turn: 2, strategyText: '1. Step' }, 0); - assert.deepEqual(unreportedOf(ledger, 2), [{ assetId: 'sha256:a', strategyText: '1. Step' }]); + const ledger = withInjected({}, 'sha256:a', { turn: 2, strategyText: '1. Step', sessionId: 'session-a' }, 0); + assert.deepEqual(unreportedOf(ledger, 2), [{ assetId: 'sha256:a', strategyText: '1. Step', turn: 2, sessionId: 'session-a' }]); assert.deepEqual(unreportedOf(withReported(ledger, ['sha256:a'], 1), 2), []); }); diff --git a/test/instructions.test.js b/test/instructions.test.js index 727e0bc..d97107c 100644 --- a/test/instructions.test.js +++ b/test/instructions.test.js @@ -5,23 +5,11 @@ import { join } from 'node:path'; import { test } from 'node:test'; const root = fileURLToPath(new URL('../', import.meta.url)); -const commandsDir = join(root, 'assets', 'commands'); -function commandBodies() { - return readdirSync(commandsDir) - .filter((file) => file.endsWith('.md')) - .map((file) => readFileSync(join(commandsDir, file), 'utf8')) - .join('\n'); -} - -test('bundled instructions use DSH command and invocation semantics', () => { - const commands = commandBodies(); - const skill = readFileSync(join(root, 'assets', 'skills', 'capability-evolver', 'SKILL.md'), 'utf8'); - - assert.doesNotMatch(commands, /\$ARGUMENTS|\/evolver:|User input:/); - assert.match(commands, /Invocation arguments \(verbatim JSON string\)/); - assert.doesNotMatch(skill, /`SessionStart`|`PostToolUse`|`Stop`|bundles a lightweight MCP bridge/); - assert.match(skill, /agent\/pre-step/); - assert.match(skill, /evolver_asset_reuse_result/); - assert.match(skill, /evolver_ack/); +test('the one bundled instruction names no tool the agent no longer has', () => { + const files = readdirSync(join(root, 'assets', 'commands')); + assert.deepEqual(files, ['evolve.md']); + const evolve = readFileSync(join(root, 'assets', 'commands', 'evolve.md'), 'utf8'); + assert.doesNotMatch(evolve, /evolver_[a-z_]+|\$ARGUMENTS|\/evolver:/); + assert.match(evolve, /\/evolver-distill/); }); diff --git a/test/prime.test.js b/test/prime.test.js index 57c0863..50f3e0b 100644 --- a/test/prime.test.js +++ b/test/prime.test.js @@ -51,8 +51,8 @@ test('one call recalls by text and injects that asset\'s strategy alone', async assert.equal(calls[0].signal, controller.signal); assert.deepEqual(ids, ['sha256:abc']); assert.match(text, /\[Evolution Memory\] Retry the upload with backoff\. \(EvoMap network\)/); - assert.match(text, /nothing more to search or fetch/); - assert.match(text, /evolver_asset_reuse_result for sha256:abc\./); + assert.match(text, /\n\nApply them where they fit\.$/); + assert.doesNotMatch(text, /sha256:|evolver_|search or fetch/, 'nothing that sends the agent to a tool'); assert.match(text, /^1\. Measure the failure rate first\.$/m); assert.match(text, /^4\. Alert on the cap\.$/m); assert.doesNotMatch(text, /npm test/); @@ -81,7 +81,6 @@ test('an asset already injected this session is passed over', async () => { assert.deepEqual(ids, ['sha256:next']); assert.doesNotMatch(text, /sha256:abc|Measure the failure rate/); assert.match(text, /\[Evolution Memory\] Capsule \(EvoMap network\)/); - assert.match(text, /evolver_asset_reuse_result for sha256:next\./); assert.match(text, /^1\. Drain the queue first\.$/m); }); @@ -171,7 +170,6 @@ test('a named gene is announced by its name, and its hash only where it gets use assert.match(text, /^\[Evolution Memory\] Chinese Social Media Writing Template \(EvoMap network\):$/m); assert.doesNotMatch(text.split('\n')[0], /sha256:/, 'the hash is not what the header is for'); - assert.match(text, /evolver_asset_reuse_result for sha256:named\./); assert.doesNotMatch(text, /no business being in front of the model|Also long/); }); diff --git a/test/reporting.test.js b/test/reporting.test.js index a1dc77f..f17356d 100644 --- a/test/reporting.test.js +++ b/test/reporting.test.js @@ -90,10 +90,7 @@ test('a recorded dsh session reports each turn with the evidence it left', async reason: 'Injected by Evolver into dsh turn 2; applied: unknown; no check ran.', evidence: { tier: 'none', applied: 'partial', aborted: false, stage: 'immediate' }, }); - assert.deepEqual( - injected.map((message) => message.content[0].text), - ['[Evolution Signal] Detected: [user_feature_request] in /home/dev/fair-play/README.md.'], - ); + assert.deepEqual(injected, [], 'edits add nothing to the model context'); await primedBy(listeners, agent, '不对，我问的是推理小说里的意思', 3); await untilCount(proxy.requests, (request) => request.path === '/asset/reuse-result', 3); @@ -239,7 +236,7 @@ test('a report the Hub refuses is settled, shown in evolver_status and never res await new Promise((resolve) => { setTimeout(resolve, 150); }); assert.equal(reportsIn(proxy.requests).length, 1); - const status = await registered.tools.find((tool) => tool.name === 'evolver_status').execute({}, {}); + const status = JSON.parse((await registered.commands.find((command) => command.name === 'evolver-status').handler({})).text); assert.deepEqual(status.reuse_reports.rejected.map(({ task_id: taskId, reason, status: code }) => ({ taskId, reason, code })), [ { taskId: 'dsh:session-refused:1', reason: 'invalid_evidence', code: 400 }, ]); @@ -266,7 +263,7 @@ test('a report the Hub could not take yet is resent at the next turn end and wri const [first, second] = reportsIn(proxy.requests); assert.deepEqual(second, first); - const status = await registered.tools.find((tool) => tool.name === 'evolver_status').execute({}, {}); + const status = JSON.parse((await registered.commands.find((command) => command.name === 'evolver-status').handler({})).text); assert.deepEqual(status.reuse_reports.delivered.map(({ eligible, counted, weight }) => ({ eligible, counted, weight })), [ { eligible: true, counted: true, weight: 0.6 }, ]); @@ -288,26 +285,6 @@ test('a report with no evidence never reaches the local ledger', async (t) => { assert.equal(existsSync(join(process.env.HOME, '.evomap', 'evolution', 'root_events.jsonl')), false); }); -test("the model's own report carries L5 evidence, passes the Hub verdict through, and leaves the automatic one in place", async (t) => { - const { proxy, listeners, registered } = await harness(t, 'evolver-self-report', hubAnswering({ - assets: () => [{ asset_id: 'sha256:owned', strategy: STEPS }], - reuse: (body) => ({ recorded: true, eligible: true, counted: body.evidence.tier !== 'L5', weight: body.evidence.tier === 'L5' ? 0.1 : 0 }), - })); - const { agent } = fakeAgent({ sessionId: 'session-owned' }); - await primedBy(listeners, agent, 'add a retry to the uploader', 1); - - const tool = registered.tools.find((candidate) => candidate.name === 'evolver_asset_reuse_result'); - const receipt = await tool.execute({ asset_id: 'sha256:owned', outcome: 'mismatched', reason: 'not a fit' }, {}); - assert.deepEqual(receipt, { recorded: true, eligible: true, counted: false, weight: 0.1 }); - endTurn(listeners, agent, 1); - await untilCount(proxy.requests, (request) => request.path === '/asset/reuse-result', 2); - - const [own, automatic] = reportsIn(proxy.requests); - assert.deepEqual(own.evidence, { tier: 'L5', applied: 'unused', stage: 'immediate' }); - assert.equal(automatic.task_id, 'dsh:session-owned:1'); - assert.equal(automatic.evidence.tier, 'none'); -}); - test('an injected asset is reported from disk after the process that injected it is gone', async (t) => { const { proxy, projectDir, listeners } = await harness(t, 'evolver-durable', oneAsset('sha256:durable')); const { agent } = fakeAgent({ sessionId: 'session-durable', cwd: projectDir }); @@ -350,3 +327,4 @@ test('subagent turns and synthesized turn ends report nothing', async (t) => { await new Promise((resolve) => { setTimeout(resolve, 150); }); assert.deepEqual(reportsIn(proxy.requests), []); }); + diff --git a/test/session-distill.test.js b/test/session-distill.test.js new file mode 100644 index 0000000..1b84615 --- /dev/null +++ b/test/session-distill.test.js @@ -0,0 +1,167 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; + +import { Config, apply } from '../src/index.js'; +import { + argumentsOf, candidateOf, distillCandidate, distillPrompt, rememberCandidate, skipReason, takeCandidates, +} from '../src/session-distill.js'; +import { fakeAgent, fakeContext, freshHome, primedBy, projectDirectory, startProxy, untilCount, untilRequest } from './support/host.js'; + +const MODEL = { provider: 'gateway', model: 'deepseek-flash' }; +const GENE = { title: 'Retry uploads', summary: 'Retry a flaky upload with backoff.', strategy: '1. Measure. 2. Back off.', signals: ['retry'] }; + +function record(overrides = {}) { + return { checks: [], commands: ['npm test'], paths: ['src/up.js'], edits: [], answer: 'Done: retries added.', ...overrides }; +} + +function fakeLlm(answer = JSON.stringify(GENE), calls = []) { + return { + calls, + async *stream(request) { + calls.push(request); + yield { type: 'text-delta', text: answer }; + yield { type: 'usage', usage: { inputTokens: 900, outputTokens: 120 } }; + }, + }; +} + +function session(id) { + return { id, header: {}, deriveMessages: () => [{ role: 'assistant', source: MODEL }] }; +} + +test('only a verified, completed turn that used tools and reused no strategy becomes a candidate', () => { + assert.equal(skipReason('completed', record()), null); + assert.equal(skipReason('aborted', record()), 'not_completed'); + assert.equal(skipReason('completed', record({ answer: '' })), 'no_answer'); + assert.equal(skipReason('completed', record({ checks: [{ command: 'npm test', failed: true }] })), 'failed_check'); + assert.equal(skipReason('completed', record({ commands: [] })), 'no_tool_use'); + assert.equal(skipReason('completed', record(), { reusedAsset: true }), 'asset_reused'); +}); + +test('the prompt is a digest of the turn, never its whole history, and asks for no answer values', () => { + const candidate = candidateOf({ prompt: 'add a retry to the uploader', record: record({ checks: [{ command: 'npm test', failed: false }] }), model: MODEL, turn: 1 }); + const prompt = distillPrompt(candidate, 'the backoff'); + assert.match(prompt, /## Task\nadd a retry to the uploader/); + assert.match(prompt, /- npm test/); + assert.match(prompt, /- passed: npm test/); + assert.match(prompt, /Focus on: the backoff/); + assert.match(prompt, /Do not include the answer itself/); +}); + +test('a list the model wrote as one numbered paragraph becomes steps', () => { + assert.deepEqual(argumentsOf(JSON.stringify(GENE)).strategy, ['Measure.', 'Back off.']); + assert.equal(argumentsOf('no json here'), null); + assert.equal(argumentsOf('{"title":"no summary"}'), null); +}); + +test('a candidate is distilled through the host model and sent to the Proxy with its own cost', async () => { + const sent = []; + const proxyFetch = async (method, path, body) => { sent.push({ path, body }); return { ok: true, data: { id: 'gene_x' } }; }; + const candidate = candidateOf({ prompt: 'add a retry', record: record(), model: MODEL, turn: 1 }); + const llm = fakeLlm(); + const outcome = await distillCandidate({ proxyFetch, llm, candidate, timeoutMs: 1_000 }); + + assert.equal(outcome.status, 'sent'); + assert.deepEqual(llm.calls[0].provider, 'gateway'); + assert.equal(sent[0].path, '/conversation/distill'); + assert.equal(sent[0].body.trigger, 'session_end'); + assert.equal(sent[0].body.persist, true); + assert.equal(sent[0].body.publish, false); + assert.deepEqual(sent[0].body.distill_usage, { inputTokens: 900, outputTokens: 120 }); + assert.equal((await distillCandidate({ proxyFetch, llm, candidate: { ...candidate, model: null }, timeoutMs: 1_000 })).status, 'no_model'); +}); + +test("a session's candidate is taken once; another session's only once it has gone stale", (t) => { + t.after(freshHome('evolver-candidates-')); + rememberCandidate('live', { at: 1_000 }); + rememberCandidate('killed', { at: 0 }); + assert.deepEqual(takeCandidates({ sessionId: null, staleMs: 5_000, now: 2_000 }), []); + assert.deepEqual(takeCandidates({ sessionId: 'live', staleMs: Infinity, now: 2_000 }).map((due) => due.sessionId), ['live']); + assert.deepEqual(takeCandidates({ sessionId: 'live', staleMs: Infinity, now: 2_000 }), []); + assert.deepEqual(takeCandidates({ sessionId: null, staleMs: 5_000, now: 10_000 }).map((due) => due.sessionId), ['killed']); +}); + +async function harness(t, prefix, answerOf) { + t.after(freshHome(`${prefix}-home-`)); + const proxy = await startProxy(answerOf); + t.after(() => proxy.close()); + const { ctx, listeners } = fakeContext(); + ctx.llm = fakeLlm(); + apply(ctx, Config({ projectDir: projectDirectory(`${prefix}-`), proxyPort: proxy.port })); + return { proxy, listeners, llm: ctx.llm }; +} + +const proxyAnswer = (assets = []) => (path) => (path === '/asset/fetch' ? { assets } : { recorded: true }); + +async function verifiedTurn(listeners, agent, turn = 1) { + await primedBy(listeners, agent, 'add a retry to the uploader', turn); + await listeners.get('tools/execute')({ name: 'bash', agent, arguments: { command: 'npm test' } }, async () => ({ isError: false, content: [{ type: 'text', text: 'ℹ pass 2\nℹ fail 0' }] })); + listeners.get('session/event')(agent.session, { type: 'assistant/message', data: { turn, step: 1, message: { role: 'assistant', content: [{ type: 'text', text: 'Retries added.' }] } } }); + listeners.get('session/event')(agent.session, { type: 'turn/end', data: { turn, reason: { kind: 'completed' } } }); +} + +test('a verified turn is distilled once, when the session ends, outside the conversation', async (t) => { + const { proxy, listeners, llm } = await harness(t, 'evolver-session-end', proxyAnswer()); + const agent = { ...fakeAgent({ sessionId: 'session-end' }).agent, session: session('session-end') }; + + await verifiedTurn(listeners, agent, 1); + assert.equal(llm.calls.length, 0, 'nothing runs at turn end'); + await listeners.get('session/flush')(agent.session); + await untilRequest(proxy.requests, '/conversation/distill'); + await listeners.get('session/flush')(agent.session); + + assert.equal(llm.calls.length, 1); + assert.equal(proxy.requests.filter((request) => request.path === '/conversation/distill').length, 1); +}); + +test('a turn the next prompt corrects is not distilled', async (t) => { + const { proxy, listeners, llm } = await harness(t, 'evolver-corrected', proxyAnswer()); + const agent = { ...fakeAgent({ sessionId: 'session-corrected' }).agent, session: session('session-corrected') }; + + await verifiedTurn(listeners, agent, 1); + await primedBy(listeners, agent, '不对，这样不行', 2); + await listeners.get('session/flush')(agent.session); + + assert.equal(llm.calls.length, 0); + assert.equal(proxy.requests.filter((request) => request.path === '/conversation/distill').length, 0); +}); + +test('a strategy whose turn never ended is reported failed when the session goes away', async (t) => { + const { proxy, listeners } = await harness(t, 'evolver-disposed', proxyAnswer([{ asset_id: 'sha256:cut', strategy: ['a.', 'b.', 'c.', 'd.'] }])); + const agent = { ...fakeAgent({ sessionId: 'session-cut' }).agent, session: session('session-cut') }; + + await primedBy(listeners, agent, 'add a retry to the uploader', 1); + listeners.get('session/disposed')(agent.session); + await untilCount(proxy.requests, (request) => request.path === '/asset/reuse-result', 1); + + const [report] = proxy.requests.filter((request) => request.path === '/asset/reuse-result').map((request) => request.body); + assert.equal(report.asset_id, 'sha256:cut'); + assert.equal(report.outcome, 'failed'); + assert.equal(report.task_id, 'dsh:session-cut:1'); + assert.equal(report.evidence.aborted, true); +}); + +test("the next session reports a killed session's open strategies and distills its leftover candidate", async (t) => { + const { proxy, listeners, llm } = await harness(t, 'evolver-killed', proxyAnswer()); + const { rememberInjected } = await import('../src/injected-assets.js'); + const { statePath } = await import('../src/state-file.js'); + const { readdirSync, utimesSync } = await import('node:fs'); + const { dirname, join } = await import('node:path'); + const old = (Date.now() - 3 * 60 * 60 * 1000) / 1000; + + rememberInjected('session-killed', 'sha256:orphan', { turn: 4, strategyText: '1. a.' }); + const dir = dirname(statePath('x')); + for (const file of readdirSync(dir).filter((name) => name.startsWith('injected-'))) utimesSync(join(dir, file), old, old); + rememberCandidate('session-killed', { ...candidateOf({ prompt: 'p', record: record(), model: MODEL, turn: 4 }), at: Date.now() - 3 * 60 * 60 * 1000 }); + + const agent = { ...fakeAgent({ sessionId: 'session-next' }).agent, session: session('session-next') }; + listeners.get('agent/session-start')({ agent }); + await untilRequest(proxy.requests, '/asset/reuse-result'); + await untilRequest(proxy.requests, '/conversation/distill'); + + const [report] = proxy.requests.filter((request) => request.path === '/asset/reuse-result').map((request) => request.body); + assert.equal(report.asset_id, 'sha256:orphan'); + assert.equal(report.task_id, 'dsh:session-killed:4'); + assert.equal(report.outcome, 'failed'); + assert.equal(llm.calls.length, 1); +}); diff --git a/test/skill.test.js b/test/skill.test.js deleted file mode 100644 index 7b29539..0000000 --- a/test/skill.test.js +++ /dev/null @@ -1,11 +0,0 @@ -import assert from 'node:assert/strict'; -import { test } from 'node:test'; - -import { evolverSkillProvider } from '../src/skill.js'; - -test('skill loading observes caller cancellation', async () => { - await assert.rejects( - () => evolverSkillProvider.get({}, { signal: AbortSignal.abort(new Error('cancelled')) }), - /cancelled|aborted/i, - ); -}); From 883c43a8068c775b3b3370ff84533017915ab14f Mon Sep 17 00:00:00 2001 From: forrestlinfeng Date: Fri, 23 Oct 2026 09:34:24 +0800 Subject: [PATCH 2/6] Offer the capability-evolver skill again, behind skillEnabled (off by default) The skill is rewritten for the hook-only design: it explains what runs on its own and points to the /evolver-* commands, names no tool, and no longer tells the agent to load it before every substantive task. Off by default, because its catalog line rides every turn and its body every later step. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- README.zh.md | 2 +- assets/skills/capability-evolver/SKILL.md | 30 +++++++++++++++++ src/config.js | 2 ++ src/index.js | 10 ++++-- src/skill.js | 41 +++++++++++++++++++++++ test/apply.test.js | 12 +++++++ test/skill.test.js | 21 ++++++++++++ 8 files changed, 116 insertions(+), 4 deletions(-) create mode 100644 assets/skills/capability-evolver/SKILL.md create mode 100644 src/skill.js create mode 100644 test/skill.test.js diff --git a/README.md b/README.md index bb64c67..0d073a8 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ Evolver lets your dsh agent evolve itself. Every success becomes a shortcut for - **Survival of the fittest.** Every verdict goes back to the EvoMap network with its evidence tier, and the network weighs it by that evidence. Strategies that work get recalled more often; ones proven wrong sink. - **One agent's mistake, every agent's detour.** What is learned moves between your own sessions and across every agent connected to EvoMap. -Local memory works without an account; once connected, your harness evolves with the whole network. Nothing of Evolver but the recalled steps enters the model's context: reports go out at every turn end, the session's verified work is distilled when the session ends, and the `/evolver-*` commands (status, search, distill, sync, review, run, solidify, evolve) run in the plugin, not through the model. +Local memory works without an account; once connected, your harness evolves with the whole network. Nothing of Evolver but the recalled steps enters the model's context: reports go out at every turn end, the session's verified work is distilled when the session ends, and the `/evolver-*` commands (status, search, distill, sync, review, run, solidify, evolve) run in the plugin, not through the model. Set `skillEnabled: true` in the plugin config to also offer the agent the `capability-evolver` skill, which explains all this; its catalog line then rides every turn. ## Install diff --git a/README.zh.md b/README.zh.md index 75b892e..db432d2 100644 --- a/README.zh.md +++ b/README.zh.md @@ -17,7 +17,7 @@ Evolver 让你的 dsh 智能体自己进化。每一次成功都会变成下一 - **优胜劣汰。** 每份复用结论都带着证据等级回报 EvoMap 网络，网络按证据为它加权。有效的策略越来越常被召回，被证伪的策略逐渐沉下去。 - **一个智能体踩过的坑，整个网络都绕开。** 经验在你的每个会话之间流转，也在所有接入 EvoMap 的智能体之间流转。 -不注册账号也能用本地记忆；接入网络后，你的 harness 就和整个网络一起进化。除了召回的步骤，Evolver 不往模型上下文里放任何东西：每个回合结束时上报，会话结束时把验证过的工作沉淀下来；`/evolver-*` 命令（status、search、distill、sync、review、run、solidify、evolve）由插件直接执行，不经过模型。 +不注册账号也能用本地记忆；接入网络后，你的 harness 就和整个网络一起进化。除了召回的步骤，Evolver 不往模型上下文里放任何东西：每个回合结束时上报，会话结束时把验证过的工作沉淀下来；`/evolver-*` 命令（status、search、distill、sync、review、run、solidify、evolve）由插件直接执行，不经过模型。插件配置里设 `skillEnabled: true` 可再给 agent 提供说明这些机制的 `capability-evolver` 技能，代价是技能目录那一行每回合都会进上下文。 ## 安装 diff --git a/assets/skills/capability-evolver/SKILL.md b/assets/skills/capability-evolver/SKILL.md new file mode 100644 index 0000000..67cedc9 --- /dev/null +++ b/assets/skills/capability-evolver/SKILL.md @@ -0,0 +1,30 @@ +--- +name: capability-evolver +description: How Evolver's evolution memory works in this session and how to reach it. Use when the user asks about Evolver or EvoMap, an [Evolution Memory] block, or asks the agent to "evolve", "learn from this", or "remember how this went". An injected [Evolution Memory] block needs none of this: apply it. +--- + +# Capability Evolver + +This plugin gives the agent a **persistent, auditable evolution memory** built on the +Genome Evolution Protocol (GEP): past outcomes — what worked, what failed — are carried +forward into future sessions. All of it runs on its own; nothing here asks you to call a tool. + +## What runs on its own + +- **Behind every prompt** (`agent/pre-step`, once per turn) — recalls the strategy that best + fits the turn's prompt from the local history and the EvoMap network and injects its steps + as an `[Evolution Memory]` block. Apply them where they fit. +- **Turn end** — reports every injected strategy with the turn's evidence: whether its steps + show up in the commands, edits and answer, and the result of the last test, build, lint or + type check. A turn that never ends (the session goes away or is killed) is reported as aborted. +- **Follow-ups** — the next prompt that says the answer did not hold (or approves it), and the + next session in the same project, which checks whether the edits were kept, revise that report. +- **Session end** — the latest turn that finished verified, used tools and reused no injected + strategy is distilled into a reusable Gene, outside this conversation. + +## Commands the user can run + +`/evolver-status`, `/evolver-search `, `/evolver-distill [focus]` (distill now instead of +at session end), `/evolver-sync`, `/evolver-review`, `/evolver-run`, `/evolver-solidify`, and +`/evolver-evolve` for a deliberate checkpoint. They run in the plugin; point the user to them +rather than reproducing them by hand. diff --git a/src/config.js b/src/config.js index 5a714e7..1191955 100644 --- a/src/config.js +++ b/src/config.js @@ -16,6 +16,8 @@ export const Config = Schema.object({ assetPrimeMinSimilarity: Schema.number().min(0).max(1).default(0.3), assetPrimeWaitMs: Schema.number().step(1).min(1).max(MAX_TIMER_MS).default(6_000), assetPrimeTimeoutMs: Schema.number().step(1).min(1).max(MAX_TIMER_MS).default(8_000), + // Off by default: the skill's catalog line rides every turn and its body, once loaded, every later step. + skillEnabled: Schema.boolean().default(false), claimNudgeEnabled: Schema.boolean().default(false), claimNudgeTtlMs: Schema.number().step(1).min(1).max(MAX_TIMER_MS).default(12 * 60 * 60 * 1000), }); diff --git a/src/index.js b/src/index.js index 3b2613a..42a7d30 100644 --- a/src/index.js +++ b/src/index.js @@ -24,6 +24,7 @@ import { candidateOf, distillCandidate, dropCandidate, modelOf, rememberCandidate, skipReason, takeCandidates, } from './session-distill.js'; import { sessionKeyOf } from './session-key.js'; +import { evolverSkillProvider } from './skill.js'; import { evolverTools } from './tools.js'; import { createTurnRecords, noteAnswer, noteFileBeforeEdit, noteToolResult } from './turn-record.js'; import { outcomeOfReason } from './turn-outcomes.js'; @@ -293,8 +294,13 @@ export function apply(ctx, config = {}) { const prompts = new Map(); const distiller = createDistiller(ctx, proxyFetch); - // Nothing of Evolver but the recalled steps enters the model's context: no tools, no skill. People reach it - // through commands that run here; reports and distillation run on session events. + // Nothing of Evolver but the recalled steps enters the model's context: no tools, and the skill only when + // configured. People reach it through commands that run here; reports and distillation run on session events. + if (config.skillEnabled) { + ctx.inject(['skills'], (scoped) => { + scoped.skills.registerProvider(() => evolverSkillProvider); + }); + } ctx.inject(['commands'], (scoped) => { const commands = evolverCommands({ status: (signal) => proxyOperations.evolver_status.execute({}, { signal }), diff --git a/src/skill.js b/src/skill.js new file mode 100644 index 0000000..ba1d13e --- /dev/null +++ b/src/skill.js @@ -0,0 +1,41 @@ +// SPDX-License-Identifier: MIT +// Copyright (c) 2026 EvoMap + +import { readFile } from 'node:fs/promises'; +import { fileURLToPath } from 'node:url'; + +import { splitFrontmatter } from './frontmatter.js'; + +const PROVIDER_NAME = 'evolver'; +const SKILL_DIR = new URL('../assets/skills/capability-evolver/', import.meta.url); +const SKILL_BODY_URL = new URL('SKILL.md', SKILL_DIR); + +const SUMMARY = { + name: 'capability-evolver', + description: + 'How Evolver\'s evolution memory works in this session and how to reach it. Use when the user asks about Evolver or EvoMap, an [Evolution Memory] block, or asks the agent to "evolve", "learn from this", or "remember how this went". An injected [Evolution Memory] block needs none of this: apply it.', + invocation: { modelInvocable: true, userInvocable: true }, + provider: PROVIDER_NAME, + source: 'custom', + resourceBase: { kind: 'directory', path: fileURLToPath(SKILL_DIR) }, +}; + +export const evolverSkillProvider = { + name: PROVIDER_NAME, + // `@deepseek-ai/dsh-skill` is an optional peer: a profile without the skills + // service never installs it. Reading the rank here rather than at module + // scope keeps the whole plugin loadable when the package is absent — a + // top-level import would fail the import of index.js itself. + async list(options = {}) { + if (options.signal?.aborted) throw options.signal.reason; + const { BUNDLED_SKILL_RANK } = await import('@deepseek-ai/dsh-skill'); + if (options.signal?.aborted) throw options.signal.reason; + return [{ ...SUMMARY, rank: BUNDLED_SKILL_RANK, locator: SKILL_BODY_URL }]; + }, + async get(_candidate, options = {}) { + const { body } = splitFrontmatter( + await readFile(SKILL_BODY_URL, { encoding: 'utf8', signal: options.signal }), + ); + return { ...SUMMARY, content: body }; + }, +}; diff --git a/test/apply.test.js b/test/apply.test.js index cbb82e6..4bdffe9 100644 --- a/test/apply.test.js +++ b/test/apply.test.js @@ -20,6 +20,7 @@ test('configuration applies defaults and rejects invalid values', () => { assert.deepEqual(configured.editToolNames, ['write', 'edit', 'str_replace_editor']); assert.equal(configured.proxyTimeoutMs, 8_000); assert.equal(configured.claimNudgeEnabled, false); + assert.equal(configured.skillEnabled, false); assert.equal(configured.assetPrimeEnabled, true); assert.equal(configured.assetPrimeWaitMs, 6_000); assert.ok( @@ -61,6 +62,17 @@ test('apply puts nothing of Evolver in the model context: no tools, no skill, on ]); }); +test('the capability-evolver skill is registered only when configured', () => { + const off = fakeContext(); + apply(off.ctx, Config({ projectDir: mkdtempSync(join(tmpdir(), 'evolver-')) })); + assert.deepEqual(off.registered.skills, []); + + const on = fakeContext(); + apply(on.ctx, Config({ projectDir: mkdtempSync(join(tmpdir(), 'evolver-')), skillEnabled: true })); + assert.equal(on.registered.skills.length, 1); + assert.deepEqual(on.registered.tools, []); +}); + test('engine commands run the evolver executable with the given arguments, without the model', async () => { const { ctx, registered } = fakeContext(); apply(ctx, { projectDir: mkdtempSync(join(tmpdir(), 'evolver-')) }); diff --git a/test/skill.test.js b/test/skill.test.js new file mode 100644 index 0000000..1d27d75 --- /dev/null +++ b/test/skill.test.js @@ -0,0 +1,21 @@ +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +import { test } from 'node:test'; + +import { splitFrontmatter } from '../src/frontmatter.js'; + +import { evolverSkillProvider } from '../src/skill.js'; + +test('skill loading observes caller cancellation', async () => { + await assert.rejects( + () => evolverSkillProvider.get({}, { signal: AbortSignal.abort(new Error('cancelled')) }), + /cancelled|aborted/i, + ); +}); + +test('the catalog summary is the description SKILL.md declares, and names no tool', async () => { + const markdown = readFileSync(new URL('../assets/skills/capability-evolver/SKILL.md', import.meta.url), 'utf8'); + const [summary] = await evolverSkillProvider.list(); + assert.equal(summary.description, splitFrontmatter(markdown).fields.description); + assert.doesNotMatch(markdown, /evolver_[a-z_]+/); +}); From e5baf2b9d86df77ef4807d6d13370a70e2588d74 Mon Sep 17 00:00:00 2001 From: forrestlinfeng Date: Fri, 23 Oct 2026 10:04:09 +0800 Subject: [PATCH 3/6] Keep distillation candidates until distilled; take over only dead sessions Addresses the Sentinel review of #51: - A candidate is no longer removed before the model call: it is settled only once distilled, and a failed try (no model, timeout, unparsed answer, Proxy error) keeps it for the next flush or session start, up to three tries. One distillation per session runs at a time. - A session counts as killed only when the dsh process that wrote its ledger or candidate is gone (signal-0 probe on the recorded pid), not when it has been idle for a while; entries without a pid are left alone. - Keys and tokens are stripped from the prompt, commands, checks and answer before a candidate is written. - Command arguments keep quoted words together; a missing `evolver` executable falls back to `npx -y @evomap/evolver` again. Co-Authored-By: Claude Opus 5.5 --- README.md | 7 +-- src/commands.js | 42 ++++++++++++++-- src/index.js | 33 ++++++++----- src/injected-assets.js | 35 +++++++------- src/process-owner.js | 18 +++++++ src/session-distill.js | 44 ++++++++++++----- test/apply.test.js | 13 +++++ test/session-distill.test.js | 93 ++++++++++++++++++++++++++++-------- 8 files changed, 214 insertions(+), 71 deletions(-) create mode 100644 src/process-owner.js diff --git a/README.md b/README.md index 0d073a8..00652f7 100644 --- a/README.md +++ b/README.md @@ -109,8 +109,9 @@ candidate in `~/.evolver/state/distill-pending.json`; a later verified turn repl that reads as a correction drops it. When the session ends, one call to the session's own model writes a Gene from a digest of that turn (task, commands, touched paths, checks, final answer — never the whole history) and sends it to the Proxy's `POST /conversation/distill` with `persist: true`, `publish: false` -and its own token cost as `distill_usage`. A killed session's candidate is distilled at the next session -start. `/evolver-distill` does the same at once. +and its own token cost as `distill_usage`. Keys and tokens are stripped from the candidate first. A +candidate stays on disk until it is distilled (three failed tries give it up); a candidate whose dsh +process is gone (a killed session) is distilled at the next session start, while an idle session keeps its own. `/evolver-distill` does the same at once. ## Reuse reports @@ -123,7 +124,7 @@ weighs each report by its evidence: | Turn end | the turn ends | `dsh::` | `tier` `L1` when the turn ran a test, build, lint or type check, else `none`; `applied`; `check` `{kind, passed, fail_then_pass}` of the last check; `aborted`; `stage: immediate` | | Correction | the next prompt says the answer did not hold, or approves it | `…:correction` | `tier: L3`, `applied`, `stage: correction` | | Retention | the next session in the same project finds the turn's edits kept or reverted | `…:retention` | `tier: L2`, `applied`, `stage: retention` | -| Aborted | the session goes away, or is killed, before the turn ends (a killed session's reports go out at the next session start) | `dsh::` | `tier: none`, `applied: partial`, `aborted: true`, `stage: immediate` | +| Aborted | the session goes away, or is killed, before the turn ends (a killed session's go out at the next session start, once its dsh process is gone) | `dsh::` | `tier: none`, `applied: partial`, `aborted: true`, `stage: immediate` | - **`applied`** compares the strategy's numbered steps with the turn's shell commands, touched paths, the code its edits wrote (200,000 characters a turn, kept in memory only) diff --git a/src/commands.js b/src/commands.js index ad04d4a..a2782e5 100644 --- a/src/commands.js +++ b/src/commands.js @@ -26,21 +26,53 @@ function shown(value) { return text.length > OUTPUT_MAX_CHARS ? `${text.slice(0, OUTPUT_MAX_CHARS)}\n…` : text; } -export function runEvolverCli(args, { signal } = {}) { +// Quotes group words the way a shell would (`--summary="two words"` stays one argument); nothing is expanded. +export function argumentsOf(rawInput) { + const args = []; + let current = ''; + let quote = null; + let started = false; + for (const char of rawInput) { + if (quote) { + if (char === quote) quote = null; + else current += char; + } else if (char === '"' || char === "'") { + quote = char; + started = true; + } else if (/\s/.test(char)) { + if (started) args.push(current); + current = ''; + started = false; + } else { + current += char; + started = true; + } + } + if (started) args.push(current); + return args; +} + +function execute(file, args, signal) { return new Promise((resolve) => { - execFile('evolver', args, { timeout: CLI_TIMEOUT_MS, maxBuffer: 16 * 1024 * 1024, signal }, (error, stdout, stderr) => { - const output = `${stdout}${stderr ? `\n${stderr}` : ''}`.trim(); - resolve(error ? failure(shown(output || String(error.message))) : success(shown(output || 'done'))); + execFile(file, args, { timeout: CLI_TIMEOUT_MS, maxBuffer: 16 * 1024 * 1024, signal }, (error, stdout, stderr) => { + resolve({ error, output: `${stdout ?? ''}${stderr ? `\n${stderr}` : ''}`.trim() }); }); }); } +// The installed `evolver` when there is one, else the published package through npx. +export async function runEvolverCli(args, { signal, run = execute } = {}) { + let { error, output } = await run('evolver', args, signal); + if (error?.code === 'ENOENT') ({ error, output } = await run('npx', ['-y', '@evomap/evolver', ...args], signal)); + return error ? failure(shown(output || String(error.message))) : success(shown(output || 'done')); +} + function cliCommand(subcommand, description, hint, runCli) { return { name: `${NAME_PREFIX}${subcommand}`, description, input: { hint }, - handler: ({ rawInput, signal }) => runCli([subcommand, ...rawInput.trim().split(/\s+/).filter(Boolean)], { signal }), + handler: ({ rawInput, signal }) => runCli([subcommand, ...argumentsOf(rawInput)], { signal }), }; } diff --git a/src/index.js b/src/index.js index 42a7d30..cd2d1d0 100644 --- a/src/index.js +++ b/src/index.js @@ -11,7 +11,7 @@ import { EDIT_TOOL_NAMES } from './edited-content.js'; import { prepareEngine, setupNoticeText } from './engine-setup.js'; import { installedEvolverVersion } from './engine-version.js'; import { feedbackLabelOf } from './feedback-label.js'; -import { forgetSession, rememberInjected, takeStaleUnreported, takeUnreported } from './injected-assets.js'; +import { forgetSession, rememberInjected, takeOrphanedUnreported, takeUnreported } from './injected-assets.js'; import { recordReuseLocally } from './local-ledger.js'; import { noticeDue, pendingClaimUrl } from './onboarding.js'; import { hubGene, promptTextOf } from './prime.js'; @@ -21,7 +21,7 @@ import { createReportQueue } from './report-queue.js'; import { rememberForRetention, retentionReportsOf, takeDueRetentions } from './retention-pending.js'; import { correctionReportOf, immediateReportOf, reportedReuseOf } from './reuse.js'; import { - candidateOf, distillCandidate, dropCandidate, modelOf, rememberCandidate, skipReason, takeCandidates, + candidateOf, distillCandidate, dropCandidate, dueCandidates, modelOf, rememberCandidate, settleCandidate, skipReason, } from './session-distill.js'; import { sessionKeyOf } from './session-key.js'; import { evolverSkillProvider } from './skill.js'; @@ -34,9 +34,6 @@ export { Config }; export const inject = ['tools']; const DEFAULT_PRIME_WAIT_MS = 6_000; -// A session whose state has not moved for this long was killed before its end; the next session finishes -// its reports and its distillation. -const STALE_SESSION_MS = 2 * 60 * 60 * 1000; const DISTILL_TIMEOUT_MS = 120_000; function pluginMessage(text, formed) { @@ -230,26 +227,38 @@ function onSessionStart(ctx, engineSetup, deliver, projectDirOf, distiller) { engineSetup.start(); if (isSubagent(agent.session)) return; const due = takeDueRetentions({ sessionId: sessionKeyOf(agent), projectDir: projectDirOf(agent.session) }); - deliver([...due.flatMap(retentionReportsOf), ...abortedReports(takeStaleUnreported({ staleMs: STALE_SESSION_MS }))]).catch(() => {}); + deliver([...due.flatMap(retentionReportsOf), ...abortedReports(takeOrphanedUnreported())]).catch(() => {}); distiller.distillLeftovers(); }); } -// The host's model service runs the session-end distillation; a killed session's candidate waits on disk for -// the next session start. +// The host's model service runs the session-end distillation; a candidate stays on disk until distilled (or +// given up), and a killed session's waits for the next session start. function createDistiller(ctx, proxyFetch) { let llm = null; ctx.inject(['llm'], (scoped) => { llm = scoped.llm; }); - const run = (candidate, focus) => distillCandidate({ proxyFetch, llm, candidate, focus, timeoutMs: DISTILL_TIMEOUT_MS }); + const run = async ({ sessionId, candidate }, focus) => { + const outcome = await distillCandidate({ proxyFetch, llm, candidate, focus, timeoutMs: DISTILL_TIMEOUT_MS }); + settleCandidate(sessionId, candidate, { sent: outcome.status === 'sent' }); + return outcome; + }; + // One distillation per session at a time: a second flush while the first runs shares its outcome. + const inFlight = new Map(); + const once = (due, focus) => { + if (!inFlight.has(due.sessionId)) { + inFlight.set(due.sessionId, run(due, focus).finally(() => inFlight.delete(due.sessionId))); + } + return inFlight.get(due.sessionId); + }; return { async distillSession(sessionId, focus = '') { - const [own] = takeCandidates({ sessionId, staleMs: Infinity }); - return own ? run(own.candidate, focus) : { status: 'nothing' }; + const [own] = dueCandidates({ sessionId }); + return own ? once(own, focus) : { status: 'nothing' }; }, distillLeftovers() { - for (const { candidate } of takeCandidates({ sessionId: null, staleMs: STALE_SESSION_MS })) run(candidate).catch(() => {}); + for (const due of dueCandidates({ sessionId: null })) once(due).catch(() => {}); }, }; } diff --git a/src/injected-assets.js b/src/injected-assets.js index 9e5695b..4940e33 100644 --- a/src/injected-assets.js +++ b/src/injected-assets.js @@ -4,6 +4,7 @@ import fs from 'node:fs'; import path from 'node:path'; +import { ownerIsGone } from './process-owner.js'; import { hashedName, readState, removeState, statePath, writeState } from './state-file.js'; const ENTRY_PRUNE_MS = 7 * 24 * 60 * 60 * 1000; @@ -15,10 +16,10 @@ function pruned(ledger, now) { })); } -export function withInjected(ledger, assetId, { turn, strategyText, sessionId }, now) { +export function withInjected(ledger, assetId, { turn, strategyText, sessionId, pid }, now) { const kept = pruned(ledger, now); if (kept[assetId]) return kept; - return { ...kept, [assetId]: { turn, injectedAt: now, strategyText, sessionId } }; + return { ...kept, [assetId]: { turn, injectedAt: now, strategyText, sessionId, pid } }; } export function unreportedOf(ledger, turn) { @@ -38,7 +39,7 @@ const ledgerPath = (sessionKey) => statePath(hashedName(LEDGER_PREFIX, sessionKe export function rememberInjected(sessionKey, assetId, injection, now = Date.now()) { if (!sessionKey) return; - writeState(ledgerPath(sessionKey), withInjected(readState(ledgerPath(sessionKey), {}), assetId, { ...injection, sessionId: sessionKey }, now)); + writeState(ledgerPath(sessionKey), withInjected(readState(ledgerPath(sessionKey), {}), assetId, { ...injection, sessionId: sessionKey, pid: process.pid }, now)); } export function takeUnreported(sessionKey, turn, now = Date.now()) { @@ -50,11 +51,11 @@ export function takeUnreported(sessionKey, turn, now = Date.now()) { } /** - * Injections no turn end ever reported, from sessions whose ledger has not changed for `staleMs`: a process - * killed mid-turn (a harness timeout, a closed terminal) leaves them behind. They are taken, marked reported - * and returned for an aborted report. + * Injections no turn end ever reported, from sessions whose dsh process is gone: a process killed mid-turn (a + * harness timeout, a closed terminal) leaves them behind. They are taken, marked reported and returned for an + * aborted report. A session that is merely idle keeps its open injections. */ -export function takeStaleUnreported({ staleMs, now = Date.now() }) { +export function takeOrphanedUnreported({ isGone = ownerIsGone, now = Date.now() } = {}) { const dir = path.dirname(statePath('x')); let files = []; try { @@ -62,22 +63,18 @@ export function takeStaleUnreported({ staleMs, now = Date.now() }) { } catch { return []; } - const stale = []; + const orphaned = []; for (const file of files) { const target = path.join(dir, file); - let modified; - try { - modified = fs.statSync(target).mtimeMs; - } catch { - continue; - } - if (now - modified <= staleMs) continue; const ledger = readState(target, {}); - const due = unreportedOf(ledger, Infinity).filter((asset) => asset.sessionId); - if (due.length > 0) writeState(target, withReported(ledger, due.map((asset) => asset.assetId), now)); - stale.push(...due); + const due = Object.entries(ledger) + .filter(([, entry]) => !entry?.reportedAt && entry?.sessionId && isGone(entry.pid)) + .map(([assetId, entry]) => ({ assetId, strategyText: entry.strategyText ?? '', turn: entry.turn, sessionId: entry.sessionId })); + if (due.length === 0) continue; + writeState(target, withReported(ledger, due.map((asset) => asset.assetId), now)); + orphaned.push(...due); } - return stale; + return orphaned; } export function forgetSession(sessionKey) { diff --git a/src/process-owner.js b/src/process-owner.js new file mode 100644 index 0000000..9a4b35c --- /dev/null +++ b/src/process-owner.js @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: MIT +// Copyright (c) 2026 EvoMap + +/** + * Whether the dsh process that wrote a state entry has exited. Signal 0 only probes: ESRCH means no such + * process; EPERM means it exists under another user. An entry without a pid (written by an older version) + * is never taken for a dead owner's. + */ +export function ownerIsGone(pid) { + if (!Number.isInteger(pid) || pid <= 0) return false; + if (pid === process.pid) return false; + try { + process.kill(pid, 0); + return false; + } catch (error) { + return error?.code === 'ESRCH'; + } +} diff --git a/src/session-distill.js b/src/session-distill.js index aaf1e66..f8ed894 100644 --- a/src/session-distill.js +++ b/src/session-distill.js @@ -3,12 +3,15 @@ import { createUserMessage } from '@deepseek-ai/dsh-llm'; +import { ownerIsGone } from './process-owner.js'; +import { withoutSecrets } from './redact.js'; import { readState, statePath, writeState } from './state-file.js'; // Distillation runs once a session, at its end, and outside the agent's conversation: each turn that // finished verified only leaves a candidate on disk, the latest one wins, and the session end (or, after a // session that was killed, the next session start) turns it into a Gene with one model call over a digest of // the turn — never the whole history, which cost ten times as much for fewer usable Genes in the benchmark. +// A candidate leaves the disk only once distilled or given up after MAX_ATTEMPTS failed tries. const PROMPT_MAX_CHARS = 6_000; const ANSWER_MAX_CHARS = 3_000; @@ -17,6 +20,7 @@ const MAX_COMMANDS = 40; const LIST_FIELDS = ['strategy', 'validation', 'signals', 'preconditions']; const ARGUMENT_FIELDS = ['title', 'summary', ...LIST_FIELDS]; const ITEM_NUMBER = /^\s*(?:\d+[.)]|[-*•])\s+/; +const MAX_ATTEMPTS = 3; const pendingPath = () => statePath('distill-pending.json'); @@ -40,15 +44,18 @@ export function modelOf(history) { return source?.provider && source?.model ? { provider: source.provider, model: source.model } : null; } -export function candidateOf({ prompt, record, model, turn, now = Date.now() }) { +// Keys and tokens are stripped before anything is written to disk or sent to a model or the Proxy. +export function candidateOf({ prompt, record, model, turn, pid = process.pid, now = Date.now() }) { return { - prompt: String(prompt ?? '').slice(0, PROMPT_MAX_CHARS), - commands: record.commands.slice(-MAX_COMMANDS).map((command) => command.slice(0, COMMAND_MAX_CHARS)), + prompt: withoutSecrets(String(prompt ?? '')).slice(0, PROMPT_MAX_CHARS), + commands: record.commands.slice(-MAX_COMMANDS).map((command) => withoutSecrets(command).slice(0, COMMAND_MAX_CHARS)), paths: [...new Set(record.paths)], - checks: record.checks.map(({ command, failed }) => ({ command, failed })), - answer: record.answer.slice(0, ANSWER_MAX_CHARS), + checks: record.checks.map(({ command, failed }) => ({ command: withoutSecrets(command), failed })), + answer: withoutSecrets(record.answer).slice(0, ANSWER_MAX_CHARS), model, turn, + pid, + attempts: 0, at: now, }; } @@ -66,17 +73,28 @@ export function dropCandidate(sessionId) { } /** - * The candidates due now, removed from disk: this session's own, plus any other session's left behind for - * longer than `staleMs` — a session killed before its end never distilled them. + * The candidates due now, left on disk until settled: this session's own, or (sessionId null) those whose + * owning process is gone — a session killed before its end never distilled them. An idle session whose + * process still runs keeps its candidate however long it waits. */ -export function takeCandidates({ sessionId, staleMs, now = Date.now() }) { +export function dueCandidates({ sessionId, isGone = ownerIsGone }) { + return Object.entries(readState(pendingPath(), {})) + .filter(([id, candidate]) => (sessionId ? id === sessionId : isGone(candidate?.pid))) + .map(([id, candidate]) => ({ sessionId: id, candidate })); +} + +/** + * Removes a candidate once it was distilled, or counts a failed try and gives up after MAX_ATTEMPTS. A + * newer candidate the session wrote meanwhile is left alone. + */ +export function settleCandidate(sessionId, candidate, { sent }) { const pending = readState(pendingPath(), {}); - const due = Object.entries(pending) - .filter(([id, candidate]) => id === sessionId || now - Number(candidate?.at) > staleMs); - if (due.length === 0) return []; - for (const [id] of due) delete pending[id]; + const current = pending[sessionId]; + if (!current || current.at !== candidate.at) return; + const attempts = (current.attempts ?? 0) + 1; + if (sent || attempts >= MAX_ATTEMPTS) delete pending[sessionId]; + else pending[sessionId] = { ...current, attempts }; writeState(pendingPath(), pending); - return due.map(([id, candidate]) => ({ sessionId: id, candidate })); } export function distillPrompt(candidate, focus = '') { diff --git a/test/apply.test.js b/test/apply.test.js index 4bdffe9..2a1b335 100644 --- a/test/apply.test.js +++ b/test/apply.test.js @@ -92,6 +92,19 @@ test('engine commands run the evolver executable with the given arguments, witho assert.deepEqual(messages, [], 'nothing reaches the model'); }); +test('command arguments keep quoted words together, and a missing evolver falls back to npx', async () => { + const { argumentsOf, runEvolverCli } = await import('../src/commands.js'); + assert.deepEqual(argumentsOf(' --dry-run --summary="two words" '), ['--dry-run', '--summary=two words']); + + const calls = []; + const run = async (file, args) => { + calls.push([file, ...args]); + return file === 'evolver' ? { error: Object.assign(new Error('spawn evolver ENOENT'), { code: 'ENOENT' }) } : { error: null, output: 'synced' }; + }; + assert.deepEqual(await runEvolverCli(['sync', '--dry-run'], { run }), { kind: 'success', text: 'synced' }); + assert.deepEqual(calls, [['evolver', 'sync', '--dry-run'], ['npx', '-y', '@evomap/evolver', 'sync', '--dry-run']]); +}); + test('/evolver-evolve is the one command that hands the agent an instruction', () => { const { ctx, registered } = fakeContext(); apply(ctx, { projectDir: mkdtempSync(join(tmpdir(), 'evolver-')) }); diff --git a/test/session-distill.test.js b/test/session-distill.test.js index 1b84615..a4cb01c 100644 --- a/test/session-distill.test.js +++ b/test/session-distill.test.js @@ -1,9 +1,10 @@ import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; import { test } from 'node:test'; import { Config, apply } from '../src/index.js'; import { - argumentsOf, candidateOf, distillCandidate, distillPrompt, rememberCandidate, skipReason, takeCandidates, + argumentsOf, candidateOf, distillCandidate, distillPrompt, dueCandidates, rememberCandidate, settleCandidate, skipReason, } from '../src/session-distill.js'; import { fakeAgent, fakeContext, freshHome, primedBy, projectDirectory, startProxy, untilCount, untilRequest } from './support/host.js'; @@ -14,12 +15,13 @@ function record(overrides = {}) { return { checks: [], commands: ['npm test'], paths: ['src/up.js'], edits: [], answer: 'Done: retries added.', ...overrides }; } -function fakeLlm(answer = JSON.stringify(GENE), calls = []) { +function fakeLlm(fixed = JSON.stringify(GENE), calls = []) { return { calls, + answer: undefined, async *stream(request) { calls.push(request); - yield { type: 'text-delta', text: answer }; + yield { type: 'text-delta', text: this.answer ?? fixed }; yield { type: 'usage', usage: { inputTokens: 900, outputTokens: 120 } }; }, }; @@ -71,14 +73,55 @@ test('a candidate is distilled through the host model and sent to the Proxy with assert.equal((await distillCandidate({ proxyFetch, llm, candidate: { ...candidate, model: null }, timeoutMs: 1_000 })).status, 'no_model'); }); -test("a session's candidate is taken once; another session's only once it has gone stale", (t) => { +test("a session's candidate is due to it; another session's only once that session's process is gone", (t) => { t.after(freshHome('evolver-candidates-')); - rememberCandidate('live', { at: 1_000 }); - rememberCandidate('killed', { at: 0 }); - assert.deepEqual(takeCandidates({ sessionId: null, staleMs: 5_000, now: 2_000 }), []); - assert.deepEqual(takeCandidates({ sessionId: 'live', staleMs: Infinity, now: 2_000 }).map((due) => due.sessionId), ['live']); - assert.deepEqual(takeCandidates({ sessionId: 'live', staleMs: Infinity, now: 2_000 }), []); - assert.deepEqual(takeCandidates({ sessionId: null, staleMs: 5_000, now: 10_000 }).map((due) => due.sessionId), ['killed']); + rememberCandidate('live', { pid: 111, at: 1 }); + rememberCandidate('killed', { pid: 222, at: 2 }); + const isGone = (pid) => pid === 222; + assert.deepEqual(dueCandidates({ sessionId: 'live', isGone }).map((due) => due.sessionId), ['live']); + assert.deepEqual(dueCandidates({ sessionId: null, isGone }).map((due) => due.sessionId), ['killed']); +}); + +test('a candidate leaves the disk only once distilled, or after three failed tries', (t) => { + t.after(freshHome('evolver-settle-')); + const candidate = { pid: 1, at: 5, attempts: 0 }; + rememberCandidate('s', candidate); + settleCandidate('s', candidate, { sent: false }); + settleCandidate('s', candidate, { sent: false }); + assert.equal(dueCandidates({ sessionId: 's' })[0].candidate.attempts, 2, 'kept for a retry'); + settleCandidate('s', candidate, { sent: false }); + assert.deepEqual(dueCandidates({ sessionId: 's' }), [], 'given up'); + + rememberCandidate('s', candidate); + rememberCandidate('s', { ...candidate, at: 9 }); + settleCandidate('s', candidate, { sent: true }); + assert.equal(dueCandidates({ sessionId: 's' })[0].candidate.at, 9, 'a newer candidate written meanwhile stays'); +}); + +test('keys and tokens never reach the candidate', () => { + const secret = 'sk-evomap-0123456789abcdef0123456789abcdef'; + const candidate = candidateOf({ + prompt: `use ${secret}`, model: MODEL, turn: 1, + record: record({ commands: [`curl -H "Authorization: Bearer ${secret}" x`], answer: `done with ${secret}` }), + }); + assert.doesNotMatch(JSON.stringify(candidate), new RegExp(secret)); +}); + +function deadPid() { + const child = spawnSync(process.execPath, ['-e', 'process.stdout.write(String(process.pid))']); + return Number(child.stdout.toString()); +} + +test('an idle session keeps its open strategies: only a gone process counts as killed', async (t) => { + t.after(freshHome('evolver-idle-')); + const { takeOrphanedUnreported } = await import('../src/injected-assets.js'); + const { statePath, writeState, hashedName } = await import('../src/state-file.js'); + writeState(statePath(hashedName('injected', 'idle')), { 'sha256:a': { turn: 1, injectedAt: 0, strategyText: '', sessionId: 'idle', pid: process.pid } }); + writeState(statePath(hashedName('injected', 'old')), { 'sha256:b': { turn: 1, injectedAt: 0, strategyText: '', sessionId: 'old' } }); + assert.deepEqual(takeOrphanedUnreported(), []); + writeState(statePath(hashedName('injected', 'gone')), { 'sha256:c': { turn: 2, injectedAt: 0, strategyText: '', sessionId: 'gone', pid: deadPid() } }); + assert.deepEqual(takeOrphanedUnreported().map((asset) => asset.assetId), ['sha256:c']); + assert.deepEqual(takeOrphanedUnreported(), [], 'reported once'); }); async function harness(t, prefix, answerOf) { @@ -100,6 +143,20 @@ async function verifiedTurn(listeners, agent, turn = 1) { listeners.get('session/event')(agent.session, { type: 'turn/end', data: { turn, reason: { kind: 'completed' } } }); } +test('a failed distillation keeps the candidate for the next try', async (t) => { + const { proxy, listeners, llm } = await harness(t, 'evolver-retry', proxyAnswer()); + const agent = { ...fakeAgent({ sessionId: 'session-retry' }).agent, session: session('session-retry') }; + llm.answer = 'not json'; + + await verifiedTurn(listeners, agent, 1); + await listeners.get('session/flush')(agent.session); + assert.equal(dueCandidates({ sessionId: 'session-retry' }).length, 1); + llm.answer = undefined; + await listeners.get('session/flush')(agent.session); + await untilRequest(proxy.requests, '/conversation/distill'); + assert.deepEqual(dueCandidates({ sessionId: 'session-retry' }), []); +}); + test('a verified turn is distilled once, when the session ends, outside the conversation', async (t) => { const { proxy, listeners, llm } = await harness(t, 'evolver-session-end', proxyAnswer()); const agent = { ...fakeAgent({ sessionId: 'session-end' }).agent, session: session('session-end') }; @@ -143,16 +200,14 @@ test('a strategy whose turn never ended is reported failed when the session goes test("the next session reports a killed session's open strategies and distills its leftover candidate", async (t) => { const { proxy, listeners, llm } = await harness(t, 'evolver-killed', proxyAnswer()); - const { rememberInjected } = await import('../src/injected-assets.js'); const { statePath } = await import('../src/state-file.js'); - const { readdirSync, utimesSync } = await import('node:fs'); - const { dirname, join } = await import('node:path'); - const old = (Date.now() - 3 * 60 * 60 * 1000) / 1000; - - rememberInjected('session-killed', 'sha256:orphan', { turn: 4, strategyText: '1. a.' }); - const dir = dirname(statePath('x')); - for (const file of readdirSync(dir).filter((name) => name.startsWith('injected-'))) utimesSync(join(dir, file), old, old); - rememberCandidate('session-killed', { ...candidateOf({ prompt: 'p', record: record(), model: MODEL, turn: 4 }), at: Date.now() - 3 * 60 * 60 * 1000 }); + + const { writeState, hashedName } = await import('../src/state-file.js'); + const dead = deadPid(); + writeState(statePath(hashedName('injected', 'session-killed')), { + 'sha256:orphan': { turn: 4, injectedAt: Date.now(), strategyText: '1. a.', sessionId: 'session-killed', pid: dead }, + }); + rememberCandidate('session-killed', candidateOf({ prompt: 'p', record: record(), model: MODEL, turn: 4, pid: dead })); const agent = { ...fakeAgent({ sessionId: 'session-next' }).agent, session: session('session-next') }; listeners.get('agent/session-start')({ agent }); From 7dc84d077405037483a56586495a8c5d1484c29f Mon Sep 17 00:00:00 2001 From: forrestlinfeng Date: Fri, 23 Oct 2026 14:29:29 +0800 Subject: [PATCH 4/6] Lock state writes, claim leftover candidates, redact paths and focus Addresses the second Sentinel review of #51: - `updateState()` reads, changes and writes a state file under an exclusive lock whose dead owner is taken over, so two dsh processes sharing `~/.evolver/state` no longer overwrite each other. The candidate file and the injection ledgers go through it; four parallel writers keep all 60 entries, against 17 unlocked. - A session that ends marks its candidate closed, so a distillation that failed at its end is tried again at the next session start instead of waiting for the host process to die. Leftovers are claimed under the lock by one process at a time, and a failed try releases the claim. - Touched paths and a `/evolver-distill` focus go through `withoutSecrets()` like the rest of the digest. - Every ledger entry is stamped with the live session and pid on each injection, so entries written before this version, or by the process a resumed session ran in, are neither orphaned nor unclaimable. - A cancelled or timed-out CLI command takes down the process group it leads, and the npx fallback is held to `@evomap/evolver@^2.0.39`, the range this plugin is built against. Co-Authored-By: Claude Opus 5 --- README.md | 8 ++-- src/commands.js | 49 +++++++++++++++++++-- src/index.js | 9 ++-- src/injected-assets.js | 39 +++++++++++------ src/session-distill.js | 82 ++++++++++++++++++++++++++---------- src/state-file.js | 51 ++++++++++++++++++++++ test/apply.test.js | 6 ++- test/injected-assets.test.js | 11 +++++ test/session-distill.test.js | 46 +++++++++++++++++--- test/state-file.test.js | 42 ++++++++++++++++++ 10 files changed, 287 insertions(+), 56 deletions(-) create mode 100644 test/state-file.test.js diff --git a/README.md b/README.md index 00652f7..5e52fc6 100644 --- a/README.md +++ b/README.md @@ -109,9 +109,11 @@ candidate in `~/.evolver/state/distill-pending.json`; a later verified turn repl that reads as a correction drops it. When the session ends, one call to the session's own model writes a Gene from a digest of that turn (task, commands, touched paths, checks, final answer — never the whole history) and sends it to the Proxy's `POST /conversation/distill` with `persist: true`, `publish: false` -and its own token cost as `distill_usage`. Keys and tokens are stripped from the candidate first. A -candidate stays on disk until it is distilled (three failed tries give it up); a candidate whose dsh -process is gone (a killed session) is distilled at the next session start, while an idle session keeps its own. `/evolver-distill` does the same at once. +and its own token cost as `distill_usage`. Keys and tokens are stripped from the candidate first, paths and +a manual focus included. A candidate stays on disk until it is distilled (three failed tries give it up); +one whose session is over — ended with a failed try, or killed with its dsh process — is claimed by the next +session that starts, one process at a time, while an idle session keeps its own. `/evolver-distill` does the +same at once. ## Reuse reports diff --git a/src/commands.js b/src/commands.js index a2782e5..104a10e 100644 --- a/src/commands.js +++ b/src/commands.js @@ -7,6 +7,7 @@ import { fileURLToPath } from 'node:url'; import { createUserMessage } from '@deepseek-ai/dsh-llm'; +import { MIN_EVOLVER_VERSION } from './engine-version.js'; import { splitFrontmatter } from './frontmatter.js'; // Commands are how a person reaches Evolver, so nothing about them sits in the model's context: each one @@ -16,6 +17,9 @@ import { splitFrontmatter } from './frontmatter.js'; const NAME_PREFIX = 'evolver-'; const OUTPUT_MAX_CHARS = 4_000; const CLI_TIMEOUT_MS = 10 * 60 * 1000; +const KILL_GRACE_MS = 5_000; +// The fallback runs whatever npm resolves, so it is held to the range this plugin is built against. +const CLI_PACKAGE = `@evomap/evolver@^${MIN_EVOLVER_VERSION}`; const EVOLVE_PATH = fileURLToPath(new URL('../assets/commands/evolve.md', import.meta.url)); const success = (text) => ({ kind: 'success', text }); @@ -26,14 +30,24 @@ function shown(value) { return text.length > OUTPUT_MAX_CHARS ? `${text.slice(0, OUTPUT_MAX_CHARS)}\n…` : text; } -// Quotes group words the way a shell would (`--summary="two words"` stays one argument); nothing is expanded. +/** + * Quotes group words the way a shell would (`--summary="two words"` stays one argument) and a backslash + * escapes the character after it, so a quote can be part of a value; nothing else is expanded. + */ export function argumentsOf(rawInput) { const args = []; let current = ''; let quote = null; let started = false; + let escaped = false; for (const char of rawInput) { - if (quote) { + if (escaped) { + current += char; + escaped = false; + } else if (char === '\\' && quote !== "'") { + escaped = true; + started = true; + } else if (quote) { if (char === quote) quote = null; else current += char; } else if (char === '"' || char === "'") { @@ -52,18 +66,45 @@ export function argumentsOf(rawInput) { return args; } +// The CLI starts the Proxy and workers of its own, so a cancelled or timed-out run takes down the whole +// process group it leads rather than only the process this plugin started. Windows has no group to signal. +const grouped = process.platform !== 'win32'; + +function endTree(child) { + const target = grouped ? -child.pid : child.pid; + const signalTree = (name) => { + try { + process.kill(target, name); + } catch { + } + }; + signalTree('SIGTERM'); + setTimeout(() => signalTree('SIGKILL'), KILL_GRACE_MS).unref?.(); +} + function execute(file, args, signal) { return new Promise((resolve) => { - execFile(file, args, { timeout: CLI_TIMEOUT_MS, maxBuffer: 16 * 1024 * 1024, signal }, (error, stdout, stderr) => { + let timer = null; + const child = execFile(file, args, { maxBuffer: 16 * 1024 * 1024, detached: grouped }, (error, stdout, stderr) => { + clearTimeout(timer); + signal?.removeEventListener('abort', onAbort); resolve({ error, output: `${stdout ?? ''}${stderr ? `\n${stderr}` : ''}`.trim() }); }); + function onAbort() { + endTree(child); + } + if (!child.pid) return; + timer = setTimeout(onAbort, CLI_TIMEOUT_MS); + timer.unref?.(); + if (signal?.aborted) onAbort(); + else signal?.addEventListener('abort', onAbort, { once: true }); }); } // The installed `evolver` when there is one, else the published package through npx. export async function runEvolverCli(args, { signal, run = execute } = {}) { let { error, output } = await run('evolver', args, signal); - if (error?.code === 'ENOENT') ({ error, output } = await run('npx', ['-y', '@evomap/evolver', ...args], signal)); + if (error?.code === 'ENOENT') ({ error, output } = await run('npx', ['-y', CLI_PACKAGE, ...args], signal)); return error ? failure(shown(output || String(error.message))) : success(shown(output || 'done')); } diff --git a/src/index.js b/src/index.js index cd2d1d0..74d6ffd 100644 --- a/src/index.js +++ b/src/index.js @@ -21,7 +21,8 @@ import { createReportQueue } from './report-queue.js'; import { rememberForRetention, retentionReportsOf, takeDueRetentions } from './retention-pending.js'; import { correctionReportOf, immediateReportOf, reportedReuseOf } from './reuse.js'; import { - candidateOf, distillCandidate, dropCandidate, dueCandidates, modelOf, rememberCandidate, settleCandidate, skipReason, + candidateOf, claimLeftoverCandidates, closeCandidate, distillCandidate, dropCandidate, dueCandidates, modelOf, + rememberCandidate, settleCandidate, skipReason, } from './session-distill.js'; import { sessionKeyOf } from './session-key.js'; import { evolverSkillProvider } from './skill.js'; @@ -233,7 +234,8 @@ function onSessionStart(ctx, engineSetup, deliver, projectDirOf, distiller) { } // The host's model service runs the session-end distillation; a candidate stays on disk until distilled (or -// given up), and a killed session's waits for the next session start. +// given up), and one whose session is over — ended with a failed try, or killed with its process — is +// claimed and retried by the next session that starts. function createDistiller(ctx, proxyFetch) { let llm = null; ctx.inject(['llm'], (scoped) => { @@ -258,7 +260,7 @@ function createDistiller(ctx, proxyFetch) { return own ? once(own, focus) : { status: 'nothing' }; }, distillLeftovers() { - for (const due of dueCandidates({ sessionId: null })) once(due).catch(() => {}); + for (const due of claimLeftoverCandidates()) once(due).catch(() => {}); }, }; } @@ -334,6 +336,7 @@ export function apply(ctx, config = {}) { ctx.on('session/disposed', (session) => { const sessionKey = sessionKeyOf(session); deliver(abortedReports(takeUnreported(sessionKey, Infinity))).catch(() => {}); + closeCandidate(sessionKey); records.forget(sessionKey); labelable.delete(sessionKey); prompts.delete(sessionKey); diff --git a/src/injected-assets.js b/src/injected-assets.js index 4940e33..0261f97 100644 --- a/src/injected-assets.js +++ b/src/injected-assets.js @@ -5,7 +5,7 @@ import fs from 'node:fs'; import path from 'node:path'; import { ownerIsGone } from './process-owner.js'; -import { hashedName, readState, removeState, statePath, writeState } from './state-file.js'; +import { hashedName, removeState, statePath, updateState } from './state-file.js'; const ENTRY_PRUNE_MS = 7 * 24 * 60 * 60 * 1000; @@ -16,8 +16,16 @@ function pruned(ledger, now) { })); } +/** + * Every entry in a ledger belongs to the session it is named after, so each injection also stamps the + * entries already there with the live owner: an entry written before this version, or by the process a + * resumed session ran in, would otherwise read as an unclaimable or orphaned one. + */ +const owned = (ledger, sessionId, pid) => + Object.fromEntries(Object.entries(ledger).map(([id, entry]) => [id, { ...entry, sessionId, pid }])); + export function withInjected(ledger, assetId, { turn, strategyText, sessionId, pid }, now) { - const kept = pruned(ledger, now); + const kept = owned(pruned(ledger, now), sessionId, pid); if (kept[assetId]) return kept; return { ...kept, [assetId]: { turn, injectedAt: now, strategyText, sessionId, pid } }; } @@ -39,14 +47,17 @@ const ledgerPath = (sessionKey) => statePath(hashedName(LEDGER_PREFIX, sessionKe export function rememberInjected(sessionKey, assetId, injection, now = Date.now()) { if (!sessionKey) return; - writeState(ledgerPath(sessionKey), withInjected(readState(ledgerPath(sessionKey), {}), assetId, { ...injection, sessionId: sessionKey, pid: process.pid }, now)); + updateState(ledgerPath(sessionKey), {}, (ledger) => + withInjected(ledger, assetId, { ...injection, sessionId: sessionKey, pid: process.pid }, now)); } export function takeUnreported(sessionKey, turn, now = Date.now()) { if (!sessionKey) return []; - const ledger = readState(ledgerPath(sessionKey), {}); - const due = unreportedOf(ledger, turn); - if (due.length > 0) writeState(ledgerPath(sessionKey), withReported(ledger, due.map((asset) => asset.assetId), now)); + let due = []; + updateState(ledgerPath(sessionKey), {}, (ledger) => { + due = unreportedOf(ledger, turn); + return due.length > 0 ? withReported(ledger, due.map((asset) => asset.assetId), now) : undefined; + }); return due; } @@ -65,14 +76,14 @@ export function takeOrphanedUnreported({ isGone = ownerIsGone, now = Date.now() } const orphaned = []; for (const file of files) { - const target = path.join(dir, file); - const ledger = readState(target, {}); - const due = Object.entries(ledger) - .filter(([, entry]) => !entry?.reportedAt && entry?.sessionId && isGone(entry.pid)) - .map(([assetId, entry]) => ({ assetId, strategyText: entry.strategyText ?? '', turn: entry.turn, sessionId: entry.sessionId })); - if (due.length === 0) continue; - writeState(target, withReported(ledger, due.map((asset) => asset.assetId), now)); - orphaned.push(...due); + updateState(path.join(dir, file), {}, (ledger) => { + const due = Object.entries(ledger) + .filter(([, entry]) => !entry?.reportedAt && entry?.sessionId && isGone(entry.pid)) + .map(([assetId, entry]) => ({ assetId, strategyText: entry.strategyText ?? '', turn: entry.turn, sessionId: entry.sessionId })); + if (due.length === 0) return undefined; + orphaned.push(...due); + return withReported(ledger, due.map((asset) => asset.assetId), now); + }); } return orphaned; } diff --git a/src/session-distill.js b/src/session-distill.js index f8ed894..de3e5f0 100644 --- a/src/session-distill.js +++ b/src/session-distill.js @@ -5,13 +5,14 @@ import { createUserMessage } from '@deepseek-ai/dsh-llm'; import { ownerIsGone } from './process-owner.js'; import { withoutSecrets } from './redact.js'; -import { readState, statePath, writeState } from './state-file.js'; +import { readState, statePath, updateState } from './state-file.js'; // Distillation runs once a session, at its end, and outside the agent's conversation: each turn that // finished verified only leaves a candidate on disk, the latest one wins, and the session end (or, after a // session that was killed, the next session start) turns it into a Gene with one model call over a digest of // the turn — never the whole history, which cost ten times as much for fewer usable Genes in the benchmark. -// A candidate leaves the disk only once distilled or given up after MAX_ATTEMPTS failed tries. +// A candidate leaves the disk only once distilled or given up after MAX_ATTEMPTS failed tries; a try that +// failed is claimed again by the next session that starts. const PROMPT_MAX_CHARS = 6_000; const ANSWER_MAX_CHARS = 3_000; @@ -49,7 +50,7 @@ export function candidateOf({ prompt, record, model, turn, pid = process.pid, no return { prompt: withoutSecrets(String(prompt ?? '')).slice(0, PROMPT_MAX_CHARS), commands: record.commands.slice(-MAX_COMMANDS).map((command) => withoutSecrets(command).slice(0, COMMAND_MAX_CHARS)), - paths: [...new Set(record.paths)], + paths: [...new Set(record.paths)].map((path) => withoutSecrets(path)), checks: record.checks.map(({ command, failed }) => ({ command: withoutSecrets(command), failed })), answer: withoutSecrets(record.answer).slice(0, ANSWER_MAX_CHARS), model, @@ -62,25 +63,55 @@ export function candidateOf({ prompt, record, model, turn, pid = process.pid, no export function rememberCandidate(sessionId, candidate) { if (!sessionId) return; - writeState(pendingPath(), { ...readState(pendingPath(), {}), [sessionId]: candidate }); + updateState(pendingPath(), {}, (pending) => ({ ...pending, [sessionId]: candidate })); } export function dropCandidate(sessionId) { - const pending = readState(pendingPath(), {}); - if (!sessionId || !(sessionId in pending)) return; - delete pending[sessionId]; - writeState(pendingPath(), pending); + if (!sessionId) return; + updateState(pendingPath(), {}, (pending) => { + if (!(sessionId in pending)) return undefined; + const { [sessionId]: dropped, ...rest } = pending; + return rest; + }); +} + +/** A session that ended keeps its candidate due, so a distillation that failed at its end is tried again. */ +export function closeCandidate(sessionId) { + if (!sessionId) return; + updateState(pendingPath(), {}, (pending) => (pending[sessionId] + ? { ...pending, [sessionId]: { ...pending[sessionId], closed: true } } + : undefined)); +} + +// A candidate is up for distillation again once its session is over and no live process is working on it. +function leftover(candidate, isGone) { + const over = Boolean(candidate?.closed) || isGone(candidate?.pid); + return over && (!candidate?.claimedBy || isGone(candidate.claimedBy)); +} + +/** This session's own candidate, if it has one. Left on disk until settled. */ +export function dueCandidates({ sessionId }) { + const candidate = readState(pendingPath(), {})[sessionId]; + return candidate ? [{ sessionId, candidate }] : []; } /** - * The candidates due now, left on disk until settled: this session's own, or (sessionId null) those whose - * owning process is gone — a session killed before its end never distilled them. An idle session whose - * process still runs keeps its candidate however long it waits. + * The candidates of sessions that are over — ended, or killed with their process — claimed for this process + * under the state lock, so two hosts starting at once cannot distill the same one twice. An idle session + * whose process still runs keeps its candidate however long it waits. */ -export function dueCandidates({ sessionId, isGone = ownerIsGone }) { - return Object.entries(readState(pendingPath(), {})) - .filter(([id, candidate]) => (sessionId ? id === sessionId : isGone(candidate?.pid))) - .map(([id, candidate]) => ({ sessionId: id, candidate })); +export function claimLeftoverCandidates({ isGone = ownerIsGone, pid = process.pid } = {}) { + const claimed = []; + updateState(pendingPath(), {}, (pending) => { + const next = { ...pending }; + for (const [sessionId, candidate] of Object.entries(pending)) { + if (!leftover(candidate, isGone) || candidate.claimedBy === pid) continue; + next[sessionId] = { ...candidate, claimedBy: pid }; + claimed.push({ sessionId, candidate: next[sessionId] }); + } + return claimed.length > 0 ? next : undefined; + }); + return claimed; } /** @@ -88,22 +119,27 @@ export function dueCandidates({ sessionId, isGone = ownerIsGone }) { * newer candidate the session wrote meanwhile is left alone. */ export function settleCandidate(sessionId, candidate, { sent }) { - const pending = readState(pendingPath(), {}); - const current = pending[sessionId]; - if (!current || current.at !== candidate.at) return; - const attempts = (current.attempts ?? 0) + 1; - if (sent || attempts >= MAX_ATTEMPTS) delete pending[sessionId]; - else pending[sessionId] = { ...current, attempts }; - writeState(pendingPath(), pending); + updateState(pendingPath(), {}, (pending) => { + const current = pending[sessionId]; + if (!current || current.at !== candidate.at) return undefined; + const attempts = (current.attempts ?? 0) + 1; + if (sent || attempts >= MAX_ATTEMPTS) { + const { [sessionId]: settled, ...rest } = pending; + return rest; + } + // The claim is released, so the next session start can try this candidate again. + return { ...pending, [sessionId]: { ...current, attempts, claimedBy: null } }; + }); } export function distillPrompt(candidate, focus = '') { const lines = (items, render) => items.map(render).join('\n') || '(none)'; + const focusText = withoutSecrets(String(focus ?? '')); return [ 'Below is the record of a task an agent has just finished and verified. Turn it into one reusable Gene:', 'a method another agent could follow on a similar task, written for the kind of data or system involved,', 'not for these files, names or values. Do not include the answer itself.', - ...(focus ? [`Focus on: ${focus}`] : []), + ...(focusText ? [`Focus on: ${focusText}`] : []), 'Answer with a JSON object only, with these fields:', ' title: short capability name', ' summary: what was solved and under which conditions', diff --git a/src/state-file.js b/src/state-file.js index 90b5a6a..09e1e04 100644 --- a/src/state-file.js +++ b/src/state-file.js @@ -6,6 +6,8 @@ import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; +import { ownerIsGone } from './process-owner.js'; + export function statePath(name) { return path.join(os.homedir(), '.evolver', 'state', name); } @@ -39,6 +41,55 @@ export function writeState(target, value) { } } +const LOCK_WAIT_MS = 2_000; +const LOCK_RETRY_MS = 20; + +// Atomics.wait is the only sleep that holds a synchronous read-modify-write together. +function pause(ms) { + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); +} + +function abandoned(lockPath) { + try { + return ownerIsGone(Number(fs.readFileSync(lockPath, 'utf8'))); + } catch { + return false; + } +} + +function acquire(lockPath, deadline) { + for (;;) { + try { + fs.mkdirSync(path.dirname(lockPath), { recursive: true, mode: 0o700 }); + fs.writeFileSync(lockPath, String(process.pid), { mode: 0o600, flag: 'wx' }); + return true; + } catch (error) { + if (error?.code !== 'EEXIST') return false; + if (abandoned(lockPath)) fs.rmSync(lockPath, { force: true }); + else if (Date.now() >= deadline) return false; + else pause(LOCK_RETRY_MS); + } + } +} + +/** + * Reads a state file, applies `mutate` and writes the result back while holding an exclusive lock, so two + * dsh processes sharing `~/.evolver/state` cannot overwrite each other's entries. A lock whose owner died is + * taken over; a lock that cannot be had within LOCK_WAIT_MS is given up on and the update runs unguarded, + * which is still the behaviour of an unlocked write and never blocks a session. + */ +export function updateState(target, fallback, mutate) { + const lockPath = `${target}.lock`; + const locked = acquire(lockPath, Date.now() + LOCK_WAIT_MS); + try { + const next = mutate(readState(target, fallback)); + if (next !== undefined) writeState(target, next); + return next; + } finally { + if (locked) removeState(lockPath); + } +} + export function removeState(target) { try { fs.rmSync(target, { force: true }); diff --git a/test/apply.test.js b/test/apply.test.js index 2a1b335..8f9b64f 100644 --- a/test/apply.test.js +++ b/test/apply.test.js @@ -92,9 +92,11 @@ test('engine commands run the evolver executable with the given arguments, witho assert.deepEqual(messages, [], 'nothing reaches the model'); }); -test('command arguments keep quoted words together, and a missing evolver falls back to npx', async () => { +test('command arguments keep quoted words together, and a missing evolver falls back to a pinned npx', async () => { const { argumentsOf, runEvolverCli } = await import('../src/commands.js'); + const { MIN_EVOLVER_VERSION } = await import('../src/engine-version.js'); assert.deepEqual(argumentsOf(' --dry-run --summary="two words" '), ['--dry-run', '--summary=two words']); + assert.deepEqual(argumentsOf('--summary="a \\"quoted\\" word" --intent=a\\ b'), ['--summary=a "quoted" word', '--intent=a b']); const calls = []; const run = async (file, args) => { @@ -102,7 +104,7 @@ test('command arguments keep quoted words together, and a missing evolver falls return file === 'evolver' ? { error: Object.assign(new Error('spawn evolver ENOENT'), { code: 'ENOENT' }) } : { error: null, output: 'synced' }; }; assert.deepEqual(await runEvolverCli(['sync', '--dry-run'], { run }), { kind: 'success', text: 'synced' }); - assert.deepEqual(calls, [['evolver', 'sync', '--dry-run'], ['npx', '-y', '@evomap/evolver', 'sync', '--dry-run']]); + assert.deepEqual(calls, [['evolver', 'sync', '--dry-run'], ['npx', '-y', `@evomap/evolver@^${MIN_EVOLVER_VERSION}`, 'sync', '--dry-run']]); }); test('/evolver-evolve is the one command that hands the agent an instruction', () => { diff --git a/test/injected-assets.test.js b/test/injected-assets.test.js index 9dd7be4..dfa7392 100644 --- a/test/injected-assets.test.js +++ b/test/injected-assets.test.js @@ -16,6 +16,17 @@ test('an asset is recorded once per session', () => { assert.equal(withInjected(first, 'sha256:a', { turn: 3, strategyText: 'two' }, 1)['sha256:a'].turn, 1); }); +test('an entry written before this version is stamped with the live owner, so an orphan scan can claim it', () => { + const legacy = { 'sha256:old': { turn: 1, injectedAt: 0, strategyText: 'one' } }; + const ledger = withInjected(legacy, 'sha256:new', { turn: 2, strategyText: 'two', sessionId: 'session-a', pid: 42 }, 1); + assert.deepEqual(ledger['sha256:old'], { turn: 1, injectedAt: 0, strategyText: 'one', sessionId: 'session-a', pid: 42 }); +}); + +test('a session that resumed in another process owns the entries it already had', () => { + const first = withInjected({}, 'sha256:a', { turn: 1, strategyText: 'one', sessionId: 'session-a', pid: 1 }, 0); + assert.equal(withInjected(first, 'sha256:a', { turn: 3, strategyText: 'two', sessionId: 'session-a', pid: 2 }, 1)['sha256:a'].pid, 2); +}); + test('a strategy that lands after its turn ended is reported at the next turn end', () => { const ledger = withInjected({}, 'sha256:a', { turn: 2, strategyText: '' }, 0); assert.deepEqual(unreportedOf(ledger, 1), []); diff --git a/test/session-distill.test.js b/test/session-distill.test.js index a4cb01c..7899044 100644 --- a/test/session-distill.test.js +++ b/test/session-distill.test.js @@ -4,7 +4,8 @@ import { test } from 'node:test'; import { Config, apply } from '../src/index.js'; import { - argumentsOf, candidateOf, distillCandidate, distillPrompt, dueCandidates, rememberCandidate, settleCandidate, skipReason, + argumentsOf, candidateOf, claimLeftoverCandidates, closeCandidate, distillCandidate, distillPrompt, dueCandidates, + rememberCandidate, settleCandidate, skipReason, } from '../src/session-distill.js'; import { fakeAgent, fakeContext, freshHome, primedBy, projectDirectory, startProxy, untilCount, untilRequest } from './support/host.js'; @@ -73,18 +74,31 @@ test('a candidate is distilled through the host model and sent to the Proxy with assert.equal((await distillCandidate({ proxyFetch, llm, candidate: { ...candidate, model: null }, timeoutMs: 1_000 })).status, 'no_model'); }); -test("a session's candidate is due to it; another session's only once that session's process is gone", (t) => { +test("a session's candidate is due to it; another session's only once that session is over", (t) => { t.after(freshHome('evolver-candidates-')); rememberCandidate('live', { pid: 111, at: 1 }); rememberCandidate('killed', { pid: 222, at: 2 }); + rememberCandidate('ended', { pid: 111, at: 3 }); + closeCandidate('ended'); const isGone = (pid) => pid === 222; - assert.deepEqual(dueCandidates({ sessionId: 'live', isGone }).map((due) => due.sessionId), ['live']); - assert.deepEqual(dueCandidates({ sessionId: null, isGone }).map((due) => due.sessionId), ['killed']); + assert.deepEqual(dueCandidates({ sessionId: 'live' }).map((due) => due.sessionId), ['live']); + assert.deepEqual(claimLeftoverCandidates({ isGone, pid: 7 }).map((due) => due.sessionId), ['killed', 'ended']); +}); + +test('a leftover candidate is claimed by one process only, and again once the claim dies', (t) => { + t.after(freshHome('evolver-claim-')); + const isGone = (pid) => pid !== 111; + rememberCandidate('killed', { pid: 222, at: 1 }); + + assert.deepEqual(claimLeftoverCandidates({ isGone, pid: 111 }).map((due) => due.sessionId), ['killed']); + assert.deepEqual(claimLeftoverCandidates({ isGone, pid: 333 }), [], 'a live claim is left alone'); + settleCandidate('killed', { at: 1 }, { sent: false }); + assert.deepEqual(claimLeftoverCandidates({ isGone, pid: 333 }).map((due) => due.candidate.attempts), [1], 'a failed try is claimable again'); }); test('a candidate leaves the disk only once distilled, or after three failed tries', (t) => { t.after(freshHome('evolver-settle-')); - const candidate = { pid: 1, at: 5, attempts: 0 }; + const candidate = { pid: process.pid, at: 5, attempts: 0 }; rememberCandidate('s', candidate); settleCandidate('s', candidate, { sent: false }); settleCandidate('s', candidate, { sent: false }); @@ -98,13 +112,14 @@ test('a candidate leaves the disk only once distilled, or after three failed tri assert.equal(dueCandidates({ sessionId: 's' })[0].candidate.at, 9, 'a newer candidate written meanwhile stays'); }); -test('keys and tokens never reach the candidate', () => { +test('keys and tokens never reach the candidate, nor the prompt a manual focus writes', () => { const secret = 'sk-evomap-0123456789abcdef0123456789abcdef'; const candidate = candidateOf({ prompt: `use ${secret}`, model: MODEL, turn: 1, - record: record({ commands: [`curl -H "Authorization: Bearer ${secret}" x`], answer: `done with ${secret}` }), + record: record({ commands: [`curl -H "Authorization: Bearer ${secret}" x`], answer: `done with ${secret}`, paths: [`keys/${secret}.pem`] }), }); assert.doesNotMatch(JSON.stringify(candidate), new RegExp(secret)); + assert.doesNotMatch(distillPrompt(candidate, `Authorization: Bearer ${secret}`), new RegExp(secret)); }); function deadPid() { @@ -157,6 +172,23 @@ test('a failed distillation keeps the candidate for the next try', async (t) => assert.deepEqual(dueCandidates({ sessionId: 'session-retry' }), []); }); +test('a distillation that failed at the end of a session is claimed and tried again at the next session start', async (t) => { + const { proxy, listeners, llm } = await harness(t, 'evolver-reclaim', proxyAnswer()); + const agent = { ...fakeAgent({ sessionId: 'session-over' }).agent, session: session('session-over') }; + llm.answer = 'not json'; + + await verifiedTurn(listeners, agent, 1); + await listeners.get('session/flush')(agent.session); + listeners.get('session/disposed')(agent.session); + assert.equal(proxy.requests.filter((request) => request.path === '/conversation/distill').length, 0); + + llm.answer = undefined; + const next = { ...fakeAgent({ sessionId: 'session-after' }).agent, session: session('session-after') }; + listeners.get('agent/session-start')({ agent: next }); + await untilRequest(proxy.requests, '/conversation/distill'); + assert.deepEqual(dueCandidates({ sessionId: 'session-over' }), []); +}); + test('a verified turn is distilled once, when the session ends, outside the conversation', async (t) => { const { proxy, listeners, llm } = await harness(t, 'evolver-session-end', proxyAnswer()); const agent = { ...fakeAgent({ sessionId: 'session-end' }).agent, session: session('session-end') }; diff --git a/test/state-file.test.js b/test/state-file.test.js new file mode 100644 index 0000000..f6780b9 --- /dev/null +++ b/test/state-file.test.js @@ -0,0 +1,42 @@ +import assert from 'node:assert/strict'; +import { spawn } from 'node:child_process'; +import { test } from 'node:test'; + +import { readState, statePath, updateState } from '../src/state-file.js'; +import { freshHome } from './support/host.js'; + +const MODULE = new URL('../src/state-file.js', import.meta.url).href; +const WRITERS = 4; +const PER_WRITER = 15; + +const CHILD = ` + const { statePath, updateState } = await import(process.argv[1]); + for (const key of process.argv.slice(2)) { + updateState(statePath('shared.json'), {}, (state) => ({ ...state, [key]: true })); + } +`; + +function writer(keys) { + return new Promise((resolve) => { + const child = spawn(process.execPath, ['--input-type=module', '-e', CHILD, MODULE, ...keys], { stdio: 'ignore' }); + child.on('exit', resolve); + }); +} + +test('an update is read, changed and written under a lock, so parallel writers keep every entry', async (t) => { + t.after(freshHome('evolver-state-lock-')); + const keysOf = (writerIndex) => Array.from({ length: PER_WRITER }, (unused, index) => `w${writerIndex}-${index}`); + + await Promise.all(Array.from({ length: WRITERS }, (unused, index) => writer(keysOf(index)))); + + const written = readState(statePath('shared.json'), {}); + assert.equal(Object.keys(written).length, WRITERS * PER_WRITER); +}); + +test('an update that changes nothing leaves the file alone and releases its lock', (t) => { + t.after(freshHome('evolver-state-noop-')); + updateState(statePath('kept.json'), {}, () => ({ a: 1 })); + updateState(statePath('kept.json'), {}, () => undefined); + assert.deepEqual(readState(statePath('kept.json'), {}), { a: 1 }); + assert.deepEqual(updateState(statePath('kept.json'), {}, (state) => ({ ...state, b: 2 })), { a: 1, b: 2 }); +}); From 20f20d9d4f8c8b7c5cd6aa1af78cc2a9f88919f4 Mon Sep 17 00:00:00 2001 From: forrestlinfeng Date: Fri, 23 Oct 2026 18:56:46 +0800 Subject: [PATCH 5/6] Stop a corrected turn's running distillation; reclaim only the dead lock A correction now also aborts a distillation already running for the session, and every distillation asks once more whether its candidate is still the one on disk before it writes to the Proxy, so a turn judged not worth keeping is never persisted. A lock carries a token unique to its acquisition. A dead owner's lock is moved aside before it is deleted and put back when what was moved is not the lock that was inspected, and a process releases a lock only while it still holds its own token, so reclaiming a stale lock can no longer delete one another process just took. Co-Authored-By: Claude Opus 5.5 --- src/index.js | 24 ++++++++++++------ src/session-distill.js | 16 +++++++++--- src/state-file.js | 48 ++++++++++++++++++++++++++++-------- test/session-distill.test.js | 43 +++++++++++++++++++++++++++++++- test/state-file.test.js | 22 ++++++++++++++++- 5 files changed, 130 insertions(+), 23 deletions(-) diff --git a/src/index.js b/src/index.js index 74d6ffd..fc01373 100644 --- a/src/index.js +++ b/src/index.js @@ -22,7 +22,7 @@ import { rememberForRetention, retentionReportsOf, takeDueRetentions } from './r import { correctionReportOf, immediateReportOf, reportedReuseOf } from './reuse.js'; import { candidateOf, claimLeftoverCandidates, closeCandidate, distillCandidate, dropCandidate, dueCandidates, modelOf, - rememberCandidate, settleCandidate, skipReason, + isCurrentCandidate, rememberCandidate, settleCandidate, skipReason, } from './session-distill.js'; import { sessionKeyOf } from './session-key.js'; import { evolverSkillProvider } from './skill.js'; @@ -241,8 +241,9 @@ function createDistiller(ctx, proxyFetch) { ctx.inject(['llm'], (scoped) => { llm = scoped.llm; }); - const run = async ({ sessionId, candidate }, focus) => { - const outcome = await distillCandidate({ proxyFetch, llm, candidate, focus, timeoutMs: DISTILL_TIMEOUT_MS }); + const run = async ({ sessionId, candidate }, focus, cancel) => { + const isCurrent = () => isCurrentCandidate(sessionId, candidate); + const outcome = await distillCandidate({ proxyFetch, llm, candidate, focus, timeoutMs: DISTILL_TIMEOUT_MS, cancel, isCurrent }); settleCandidate(sessionId, candidate, { sent: outcome.status === 'sent' }); return outcome; }; @@ -250,9 +251,11 @@ function createDistiller(ctx, proxyFetch) { const inFlight = new Map(); const once = (due, focus) => { if (!inFlight.has(due.sessionId)) { - inFlight.set(due.sessionId, run(due, focus).finally(() => inFlight.delete(due.sessionId))); + const controller = new AbortController(); + const outcome = run(due, focus, controller.signal).finally(() => inFlight.delete(due.sessionId)); + inFlight.set(due.sessionId, { outcome, controller }); } - return inFlight.get(due.sessionId); + return inFlight.get(due.sessionId).outcome; }; return { async distillSession(sessionId, focus = '') { @@ -262,17 +265,22 @@ function createDistiller(ctx, proxyFetch) { distillLeftovers() { for (const due of claimLeftoverCandidates()) once(due).catch(() => {}); }, + /** The session's turn no longer deserves a Gene: its candidate goes, and a distillation already running stops. */ + forget(sessionId) { + dropCandidate(sessionId); + inFlight.get(sessionId)?.controller.abort(); + }, }; } // The next prompt is the only verdict on a turn available without asking anyone: when it reads as a // correction, the reuse reports are revised and the turn is no longer worth distilling. -function followUpReporter(deliver, labelable, prompts) { +function followUpReporter(deliver, labelable, prompts, distiller) { return (agent, prompt) => { const sessionId = sessionKeyOf(agent); if (!prompt) return; const label = feedbackLabelOf(prompt); - if (label === 'correction') dropCandidate(sessionId); + if (label === 'correction') distiller.forget(sessionId); prompts.set(sessionId, prompt); if (!labelable.has(sessionId)) return; const reuses = labelable.get(sessionId); @@ -327,7 +335,7 @@ export function apply(ctx, config = {}) { for (const command of commands) scoped.commands.register(command); }); - primeSteps(ctx, config, primeFetch, engineSetup, recallLog, followUpReporter(deliver, labelable, prompts)); + primeSteps(ctx, config, primeFetch, engineSetup, recallLog, followUpReporter(deliver, labelable, prompts, distiller)); observeTools(ctx, config.editToolNames ?? EDIT_TOOL_NAMES, records, projectDirOf); reportTurns(ctx, records, deliver, labelable, projectDirOf, prompts); onSessionStart(ctx, engineSetup, deliver, projectDirOf, distiller); diff --git a/src/session-distill.js b/src/session-distill.js index de3e5f0..fd2578e 100644 --- a/src/session-distill.js +++ b/src/session-distill.js @@ -75,6 +75,11 @@ export function dropCandidate(sessionId) { }); } +/** Whether this is still the session's candidate on disk: not dropped by a correction, nor replaced by a newer one. */ +export function isCurrentCandidate(sessionId, candidate) { + return readState(pendingPath(), {})[sessionId]?.at === candidate.at; +} + /** A session that ended keeps its candidate due, so a distillation that failed at its end is tried again. */ export function closeCandidate(sessionId) { if (!sessionId) return; @@ -212,14 +217,19 @@ async function answerOf(llm, model, text, signal) { return { answer, usage }; } -/** Distill one candidate through the host's model service. Resolves to what happened; never throws. */ -export async function distillCandidate({ proxyFetch, llm, candidate, focus, timeoutMs }) { +/** + * Distill one candidate through the host's model service. `cancel` stops it and `isCurrent` is asked once more + * before anything is persisted, since the candidate may have been dropped while the model answered. Resolves to + * what happened; never throws. + */ +export async function distillCandidate({ proxyFetch, llm, candidate, focus, timeoutMs, cancel, isCurrent = () => true }) { if (!llm || !candidate?.model) return { status: 'no_model' }; - const signal = AbortSignal.timeout(timeoutMs); + const signal = AbortSignal.any([AbortSignal.timeout(timeoutMs), ...(cancel ? [cancel] : [])]); try { const { answer, usage } = await answerOf(llm, candidate.model, distillPrompt(candidate, focus), signal); const args = argumentsOf(answer); if (!args) return { status: 'unparsed', usage }; + if (cancel?.aborted || !isCurrent()) return { status: 'dropped', usage }; const result = await proxyFetch('POST', '/conversation/distill', distillBody(candidate, args, usage), signal); return result?.ok ? { status: 'sent', usage, data: result.data } : { status: 'failed', error: String(result?.error) }; } catch (error) { diff --git a/src/state-file.js b/src/state-file.js index 09e1e04..d6e5ba8 100644 --- a/src/state-file.js +++ b/src/state-file.js @@ -49,29 +49,57 @@ function pause(ms) { Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); } -function abandoned(lockPath) { +function contentOf(file) { try { - return ownerIsGone(Number(fs.readFileSync(lockPath, 'utf8'))); + return fs.readFileSync(file, 'utf8'); } catch { - return false; + return null; } } +// A lock holds its owner's pid and a token unique to that acquisition, so a lock is told apart from a newer one +// the same process, or another, took after it. +const ownerPidOf = (token) => Number(token.split(':')[0]); + +// Another process may reclaim the same dead lock and a third take a fresh one before this removal happens, so the +// lock is first moved aside, and only what was inspected as dead is deleted; anything else is put back. +function reclaim(lockPath, deadToken) { + const aside = `${lockPath}.${crypto.randomBytes(6).toString('hex')}.reclaimed`; + try { + fs.renameSync(lockPath, aside); + } catch { + return; + } + if (contentOf(aside) !== deadToken) { + try { + fs.linkSync(aside, lockPath); + } catch { + } + } + removeState(aside); +} + function acquire(lockPath, deadline) { + const token = `${process.pid}:${crypto.randomBytes(8).toString('hex')}`; for (;;) { try { fs.mkdirSync(path.dirname(lockPath), { recursive: true, mode: 0o700 }); - fs.writeFileSync(lockPath, String(process.pid), { mode: 0o600, flag: 'wx' }); - return true; + fs.writeFileSync(lockPath, token, { mode: 0o600, flag: 'wx' }); + return token; } catch (error) { - if (error?.code !== 'EEXIST') return false; - if (abandoned(lockPath)) fs.rmSync(lockPath, { force: true }); - else if (Date.now() >= deadline) return false; + if (error?.code !== 'EEXIST') return null; + const held = contentOf(lockPath); + if (held !== null && ownerIsGone(ownerPidOf(held))) reclaim(lockPath, held); + else if (Date.now() >= deadline) return null; else pause(LOCK_RETRY_MS); } } } +function release(lockPath, token) { + if (contentOf(lockPath) === token) removeState(lockPath); +} + /** * Reads a state file, applies `mutate` and writes the result back while holding an exclusive lock, so two * dsh processes sharing `~/.evolver/state` cannot overwrite each other's entries. A lock whose owner died is @@ -80,13 +108,13 @@ function acquire(lockPath, deadline) { */ export function updateState(target, fallback, mutate) { const lockPath = `${target}.lock`; - const locked = acquire(lockPath, Date.now() + LOCK_WAIT_MS); + const token = acquire(lockPath, Date.now() + LOCK_WAIT_MS); try { const next = mutate(readState(target, fallback)); if (next !== undefined) writeState(target, next); return next; } finally { - if (locked) removeState(lockPath); + if (token) release(lockPath, token); } } diff --git a/test/session-distill.test.js b/test/session-distill.test.js index 7899044..dd1c029 100644 --- a/test/session-distill.test.js +++ b/test/session-distill.test.js @@ -5,7 +5,7 @@ import { test } from 'node:test'; import { Config, apply } from '../src/index.js'; import { argumentsOf, candidateOf, claimLeftoverCandidates, closeCandidate, distillCandidate, distillPrompt, dueCandidates, - rememberCandidate, settleCandidate, skipReason, + isCurrentCandidate, rememberCandidate, settleCandidate, skipReason, } from '../src/session-distill.js'; import { fakeAgent, fakeContext, freshHome, primedBy, projectDirectory, startProxy, untilCount, untilRequest } from './support/host.js'; @@ -20,8 +20,10 @@ function fakeLlm(fixed = JSON.stringify(GENE), calls = []) { return { calls, answer: undefined, + hold: undefined, async *stream(request) { calls.push(request); + await this.hold; yield { type: 'text-delta', text: this.answer ?? fixed }; yield { type: 'usage', usage: { inputTokens: 900, outputTokens: 120 } }; }, @@ -74,6 +76,28 @@ test('a candidate is distilled through the host model and sent to the Proxy with assert.equal((await distillCandidate({ proxyFetch, llm, candidate: { ...candidate, model: null }, timeoutMs: 1_000 })).status, 'no_model'); }); +test('a candidate dropped or cancelled while the model answers is never sent to the Proxy', async (t) => { + t.after(freshHome('evolver-distill-dropped-')); + const sent = []; + const proxyFetch = async (method, path) => { sent.push(path); return { ok: true }; }; + const candidate = candidateOf({ prompt: 'add a retry', record: record(), model: MODEL, turn: 1 }); + rememberCandidate('s', candidate); + assert.equal(isCurrentCandidate('s', candidate), true); + rememberCandidate('s', { ...candidate, at: candidate.at + 1 }); + assert.equal(isCurrentCandidate('s', candidate), false); + + const replaced = await distillCandidate({ proxyFetch, llm: fakeLlm(), candidate, timeoutMs: 1_000, isCurrent: () => isCurrentCandidate('s', candidate) }); + const controller = new AbortController(); + const llm = fakeLlm(); + llm.hold = new Promise((resolve) => setTimeout(resolve, 20)); + const cancelled = distillCandidate({ proxyFetch, llm, candidate, timeoutMs: 1_000, cancel: controller.signal }); + controller.abort(); + + assert.equal(replaced.status, 'dropped'); + assert.equal((await cancelled).status, 'dropped'); + assert.deepEqual(sent, []); +}); + test("a session's candidate is due to it; another session's only once that session is over", (t) => { t.after(freshHome('evolver-candidates-')); rememberCandidate('live', { pid: 111, at: 1 }); @@ -215,6 +239,23 @@ test('a turn the next prompt corrects is not distilled', async (t) => { assert.equal(proxy.requests.filter((request) => request.path === '/conversation/distill').length, 0); }); +test('a correction that arrives while the turn is being distilled stops that distillation', async (t) => { + const { proxy, listeners, llm } = await harness(t, 'evolver-corrected-late', proxyAnswer()); + const agent = { ...fakeAgent({ sessionId: 'session-late' }).agent, session: session('session-late') }; + let release; + llm.hold = new Promise((resolve) => { release = resolve; }); + + await verifiedTurn(listeners, agent, 1); + const flushed = listeners.get('session/flush')(agent.session); + await untilCount(llm.calls, () => true, 1); + await primedBy(listeners, agent, '不对，这样不行', 2); + release(); + + assert.equal((await flushed).status, 'dropped'); + assert.equal(proxy.requests.filter((request) => request.path === '/conversation/distill').length, 0); + assert.deepEqual(dueCandidates({ sessionId: 'session-late' }), []); +}); + test('a strategy whose turn never ended is reported failed when the session goes away', async (t) => { const { proxy, listeners } = await harness(t, 'evolver-disposed', proxyAnswer([{ asset_id: 'sha256:cut', strategy: ['a.', 'b.', 'c.', 'd.'] }])); const agent = { ...fakeAgent({ sessionId: 'session-cut' }).agent, session: session('session-cut') }; diff --git a/test/state-file.test.js b/test/state-file.test.js index f6780b9..b0094ff 100644 --- a/test/state-file.test.js +++ b/test/state-file.test.js @@ -1,5 +1,6 @@ import assert from 'node:assert/strict'; -import { spawn } from 'node:child_process'; +import { spawn, spawnSync } from 'node:child_process'; +import fs from 'node:fs'; import { test } from 'node:test'; import { readState, statePath, updateState } from '../src/state-file.js'; @@ -40,3 +41,22 @@ test('an update that changes nothing leaves the file alone and releases its lock assert.deepEqual(readState(statePath('kept.json'), {}), { a: 1 }); assert.deepEqual(updateState(statePath('kept.json'), {}, (state) => ({ ...state, b: 2 })), { a: 1, b: 2 }); }); + +test("a dead owner's lock is taken over, and a lock someone else holds by the time of release is left to them", (t) => { + t.after(freshHome('evolver-state-stale-')); + const target = statePath('stale.json'); + const lockPath = `${target}.lock`; + const deadPid = spawnSync(process.execPath, ['-e', '']).pid; + fs.mkdirSync(statePath(''), { recursive: true }); + fs.writeFileSync(lockPath, `${deadPid}:dead`); + + updateState(target, {}, () => ({ a: 1 })); + assert.equal(fs.existsSync(lockPath), false, "the dead owner's lock is gone after the update"); + + updateState(target, {}, (state) => { + fs.writeFileSync(lockPath, `${process.pid}:someone-else`); + return { ...state, b: 2 }; + }); + assert.equal(fs.readFileSync(lockPath, 'utf8'), `${process.pid}:someone-else`); + assert.deepEqual(readState(target, {}), { a: 1, b: 2 }); +}); From 0a824f3b1f2080887256e27c06236dddb3bdc168 Mon Sep 17 00:00:00 2001 From: forrestlinfeng Date: Fri, 23 Oct 2026 19:07:56 +0800 Subject: [PATCH 6/6] Wait for the retried candidate to settle, not just for its request The candidate leaves the disk once the Proxy answers, a moment after the request arrives; asserting right after the request raced the settle on slower CI runners. Co-Authored-By: Claude Opus 5.5 --- test/session-distill.test.js | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/test/session-distill.test.js b/test/session-distill.test.js index dd1c029..8e2fd6a 100644 --- a/test/session-distill.test.js +++ b/test/session-distill.test.js @@ -30,6 +30,15 @@ function fakeLlm(fixed = JSON.stringify(GENE), calls = []) { }; } +// A candidate is settled once the Proxy has answered, a moment after its request arrived. +async function untilSettled(sessionId, deadlineMs = 15_000) { + const start = Date.now(); + while (dueCandidates({ sessionId }).length > 0) { + if (Date.now() - start > deadlineMs) assert.fail(`the candidate of ${sessionId} was never settled`); + await new Promise((resolve) => { setTimeout(resolve, 10); }); + } +} + function session(id) { return { id, header: {}, deriveMessages: () => [{ role: 'assistant', source: MODEL }] }; } @@ -210,7 +219,7 @@ test('a distillation that failed at the end of a session is claimed and tried ag const next = { ...fakeAgent({ sessionId: 'session-after' }).agent, session: session('session-after') }; listeners.get('agent/session-start')({ agent: next }); await untilRequest(proxy.requests, '/conversation/distill'); - assert.deepEqual(dueCandidates({ sessionId: 'session-over' }), []); + await untilSettled('session-over'); }); test('a verified turn is distilled once, when the session ends, outside the conversation', async (t) => {