diff --git a/README.md b/README.md index c9d4f6c..5e52fc6 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ Evolver lets your dsh agent evolve itself. Every success becomes a shortcut for - **Survival of the fittest.** Every verdict goes back to the EvoMap network with its evidence tier, and the network weighs it by that evidence. Strategies that work get recalled more often; ones proven wrong sink. - **One agent's mistake, every agent's detour.** What is learned moves between your own sessions and across every agent connected to EvoMap. -Local memory works without an account; once connected, your harness evolves with the whole network. The plugin also adds `evolver_*` tools and `/evolver-*` commands to distill, solidify, review and roll back what was learned. +Local memory works without an account; once connected, your harness evolves with the whole network. Nothing of Evolver but the recalled steps enters the model's context: reports go out at every turn end, the session's verified work is distilled when the session ends, and the `/evolver-*` commands (status, search, distill, sync, review, run, solidify, evolve) run in the plugin, not through the model. Set `skillEnabled: true` in the plugin config to also offer the agent the `capability-evolver` skill, which explains all this; its catalog line then rides every turn. ## Install @@ -102,6 +102,19 @@ Network recall needs `@evomap/evolver` 2.0.39 or newer. The evidence on reuse re reaches the Hub only through a Proxy newer than 2.0.42; an older one drops it, and the Hub counts those reports as unevidenced. +## Session-end distillation + +A turn that ended completed, used tools, passed its last check and reused no injected strategy leaves a +candidate in `~/.evolver/state/distill-pending.json`; a later verified turn replaces it, and a next prompt +that reads as a correction drops it. When the session ends, one call to the session's own model writes a +Gene from a digest of that turn (task, commands, touched paths, checks, final answer — never the whole +history) and sends it to the Proxy's `POST /conversation/distill` with `persist: true`, `publish: false` +and its own token cost as `distill_usage`. Keys and tokens are stripped from the candidate first, paths and +a manual focus included. A candidate stays on disk until it is distilled (three failed tries give it up); +one whose session is over — ended with a failed try, or killed with its dsh process — is claimed by the next +session that starts, one process at a time, while an idle session keeps its own. `/evolver-distill` does the +same at once. + ## Reuse reports Every strategy injected into a turn is reported through the Proxy's @@ -113,7 +126,7 @@ weighs each report by its evidence: | Turn end | the turn ends | `dsh::` | `tier` `L1` when the turn ran a test, build, lint or type check, else `none`; `applied`; `check` `{kind, passed, fail_then_pass}` of the last check; `aborted`; `stage: immediate` | | Correction | the next prompt says the answer did not hold, or approves it | `…:correction` | `tier: L3`, `applied`, `stage: correction` | | Retention | the next session in the same project finds the turn's edits kept or reverted | `…:retention` | `tier: L2`, `applied`, `stage: retention` | -| `evolver_asset_reuse_result` | the model reports on its own | the model's own | `tier: L5`, `applied`, `stage: immediate` | +| Aborted | the session goes away, or is killed, before the turn ends (a killed session's go out at the next session start, once its dsh process is gone) | `dsh::` | `tier: none`, `applied: partial`, `aborted: true`, `stage: immediate` | - **`applied`** compares the strategy's numbered steps with the turn's shell commands, touched paths, the code its edits wrote (200,000 characters a turn, kept in memory only) diff --git a/README.zh.md b/README.zh.md index 38dcac3..db432d2 100644 --- a/README.zh.md +++ b/README.zh.md @@ -17,7 +17,7 @@ Evolver 让你的 dsh 智能体自己进化。每一次成功都会变成下一 - **优胜劣汰。** 每份复用结论都带着证据等级回报 EvoMap 网络，网络按证据为它加权。有效的策略越来越常被召回，被证伪的策略逐渐沉下去。 - **一个智能体踩过的坑，整个网络都绕开。** 经验在你的每个会话之间流转，也在所有接入 EvoMap 的智能体之间流转。 -不注册账号也能用本地记忆；接入网络后，你的 harness 就和整个网络一起进化。插件还提供 `evolver_*` 工具和 `/evolver-*` 命令，用来提炼、固化、审阅和回滚学到的经验。 +不注册账号也能用本地记忆；接入网络后，你的 harness 就和整个网络一起进化。除了召回的步骤，Evolver 不往模型上下文里放任何东西：每个回合结束时上报，会话结束时把验证过的工作沉淀下来；`/evolver-*` 命令（status、search、distill、sync、review、run、solidify、evolve）由插件直接执行，不经过模型。插件配置里设 `skillEnabled: true` 可再给 agent 提供说明这些机制的 `capability-evolver` 技能，代价是技能目录那一行每回合都会进上下文。 ## 安装 @@ -97,7 +97,7 @@ dsh plugin --profile dsh-tui add -w @evomap/dsh-evolver@latest | 回合结束 | 回合结束时 | `dsh:<会话>:<回合>` | 本回合跑过测试、构建、lint 或类型检查时 `tier` 为 `L1`，否则为 `none`；`applied`；最后一次检查的 `check` `{kind, passed, fail_then_pass}`；`aborted`；`stage: immediate` | | 追问补报 | 下一条提示词表示结果不对，或表示认可 | `…:correction` | `tier: L3`、`applied`、`stage: correction` | | 留存回访 | 同一项目的下一个会话发现本回合的改动被保留或被撤销 | `…:retention` | `tier: L2`、`applied`、`stage: retention` | -| `evolver_asset_reuse_result` | 模型自己上报 | 模型自带 | `tier: L5`、`applied`、`stage: immediate` | +| 中止 | 回合结束前会话消失或被强杀（被强杀会话的上报在下次会话开始时发出） | `dsh::` | `tier: none`、`applied: partial`、`aborted: true`、`stage: immediate` | - **`applied`** 把策略的编号步骤与本回合的 shell 命令、涉及的路径、编辑写入的代码（每回合最多 20 万字符，只留在内存里）和最终回答比对：半数以上步骤出现记 `used`，少于半数记 `partial`，一条都没有记 `unused`。回合没留下可比对的内容，或策略没有编号步骤时，按 tier `none` 加 `applied: partial` 上报，两边都不计权重，reason 里写 `applied: unknown`。 - **检查**指运行测试、构建、lint 或类型检查的 `bash` 命令。结果报错或输出里有失败信息都算失败，因为经 `tail` 管道的失败运行退出码仍是 0。判断 `fail_then_pass` 时会先去掉输出管道（`| tail -9`）和重定向（`2>&1`）再比较命令。 diff --git a/assets/commands/distill.md b/assets/commands/distill.md deleted file mode 100644 index f46997b..0000000 --- a/assets/commands/distill.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -description: Distill a reusable skill/gene from this session's verified work. -argument-hint: "[what to distill]" ---- - -Distill what this session proved into a reusable asset. - -Prefer `evolver_distill_conversation`. Supply a concrete summary, reproducible strategy, -validation commands that actually passed, relevant artifacts, and signals. Persistence is -on by default; set `publish: true` only when the user explicitly asked to share the result. -Never include secrets, credentials, private paths, or an unverified claim. - -Use the final `Invocation arguments (verbatim JSON string)` as additional scope for what to -distill. If the Proxy is unavailable and the full CLI is installed, use the installed -`evolver` executable (or `npx -y @evomap/evolver`) with its `distill` subcommand and the -same decoded input. - -Tell the user what was distilled, the evidence attached, and whether it stayed local or was -submitted for review. diff --git a/assets/commands/evolve.md b/assets/commands/evolve.md index 58da080..159fd4f 100644 --- a/assets/commands/evolve.md +++ b/assets/commands/evolve.md @@ -7,15 +7,11 @@ Take a deliberate evolution step for the current task. 1. **Recall.** Look at the evolution memory this plugin injected behind the prompt, or read the tail of the full engine's memory graph — `~/.evolver/memory/evolution/memory_graph.jsonl`, or the project's `memory/evolution/memory_graph.jsonl` when it exists. Summarize any recent - outcome — success or failure — relevant to what we're working on. Search the network - too, with `evolver_search_assets`, when the task looks like something others have hit. + outcome — success or failure — relevant to what we're working on. 2. **Reflect.** Given the current diff and task state, say in one or two lines: what worked, what didn't, and what the durable lesson is. -3. **Record.** Reuse of an injected strategy is reported automatically at every turn end. - If the lesson is worth sharing, distill it with - `evolver_distill_conversation` (see `/evolver-distill`). To run the full engine now, - use `/evolver-run`. - -Keep this lightweight — it is an explicit checkpoint, not a ceremony on every turn. +Reuse of an injected strategy is reported at every turn end, and the session's verified work is +distilled when the session ends; `/evolver-distill` distills it now. Keep this lightweight — it is an +explicit checkpoint, not a ceremony on every turn. diff --git a/assets/commands/review.md b/assets/commands/review.md deleted file mode 100644 index 88d22dd..0000000 --- a/assets/commands/review.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -description: Review Evolver's pending evolved changes, then approve (solidify) or reject (roll back). -argument-hint: "[--approve | --reject]" ---- - -Review the changes Evolver currently has **pending solidify** in this repository. - -1. Show `git status --short` and `git diff HEAD` so the user sees the proposed edits. -2. Decode the final `Invocation arguments (verbatim JSON string)` value. -3. Use the installed `evolver` executable, or `npx -y @evomap/evolver` when absent: - - `--approve` or an explicit request to accept: invoke `review --approve`. - - `--reject` or an explicit request to discard: invoke `review --reject`. - - Empty or ambiguous input: summarize the pending diff and ask the user to choose before - running either state-changing command. -4. Report the final state and resulting git status. diff --git a/assets/commands/run.md b/assets/commands/run.md deleted file mode 100644 index 793a7c2..0000000 --- a/assets/commands/run.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -description: Run one Evolver self-evolution cycle on the current repo (collect signals → select/mutate genes → propose changes). -argument-hint: "[--loop] [--dry-run] [--strategy=balanced|innovate|harden|repair-only]" ---- - -Run one Evolver cycle in the **current git repository**. - -The final `Invocation arguments (verbatim JSON string)` value contains the exact user -suffix. Decode that JSON string, preserve its argument order, and never drop a protective -flag such as `--dry-run` or invent a flag the user did not provide. - -1. Confirm `git rev-parse --is-inside-work-tree` succeeds. Otherwise explain that Evolver - requires git and stop. -2. Use the installed `evolver` executable when available; otherwise use - `npx -y @evomap/evolver`. -3. Invoke its `run` subcommand followed by the decoded invocation arguments. When the user - supplied no strategy, preserve the environment's `EVOLVE_STRATEGY` or use `balanced`. -4. Summarize the collected signals, selected or mutated gene, and whether changes are - pending solidify. Point pending work to `/evolver-review`; rejection is - `/evolver-review --reject`. - -Do not approve pending changes automatically. diff --git a/assets/commands/search.md b/assets/commands/search.md deleted file mode 100644 index 52a2cc7..0000000 --- a/assets/commands/search.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -description: Search the EvoMap network for reusable evolution assets before starting work. -argument-hint: "" ---- - -Search EvoMap before solving the task from scratch. - -Decode the final `Invocation arguments (verbatim JSON string)` value. Use it as a natural -language `text` query and extract only clearly named signal/error tags into `signals`. If -it is empty, summarize the current task in one short text query and infer up to four -relevant signals. - -1. Call `evolver_search_assets` with `text`, optional `signals`, and limit 5. -2. Summarize each hit by id, kind, purpose, and relevance. Mark `degraded: true` results as - local-cache-only. -3. Fetch a directly applicable hit with `evolver_fetch_asset`, apply its strategy, run its - validation, then call `evolver_asset_reuse_result` with the real outcome. - -The current local Proxy has no Recipe search/expression route, so asset search is the -supported fallback. If the Proxy is unreachable, run `/evolver-status`. diff --git a/assets/commands/solidify.md b/assets/commands/solidify.md deleted file mode 100644 index a5e7bb6..0000000 --- a/assets/commands/solidify.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -description: Solidify the current working changes into a durable Evolver gene/capsule (with rollback safety). -argument-hint: "[--dry-run] [--intent=repair|optimize|innovate] [--summary=\"...\"]" ---- - -Solidify the current working-tree changes into a durable Evolver asset. - -1. Confirm the current directory is a git repository and show `git diff --stat` so the - user sees what will be captured. -2. Decode the final `Invocation arguments (verbatim JSON string)` value. Use the installed - `evolver` executable, or `npx -y @evomap/evolver` when absent, and invoke its `solidify` - subcommand followed by those exact arguments. -3. If no `--summary` was supplied, derive a concise one-line summary from the diff and add - it explicitly. Do not change any other user flag. -4. Report the Gene or Capsule created or updated and whether a rollback point was recorded. - -Recommend `--dry-run` before a write when the user did not already choose it. diff --git a/assets/commands/status.md b/assets/commands/status.md deleted file mode 100644 index 8d09c2c..0000000 --- a/assets/commands/status.md +++ /dev/null @@ -1,48 +0,0 @@ ---- -description: Show Evolver health — Proxy, running engine version, recent strategy recalls and reuse reports, evolution memory, network claim, and full-engine availability. ---- - -Report Evolver health as a checklist grouped under the headings below. Use the available -filesystem and shell tools for the current platform; do not assume a POSIX shell. Never print -token values. - -Call `evolver_status` once and use its fields throughout; it returns `proxy` (the Proxy's own -status, or `reachable: false` with the error) and the plugin's view (`plugin`, `runtime`, -`proxy_endpoint`, `engine`, `recall`, `reuse_reports`). - -1. **Plugin and runtime** — plugin version, platform/arch, Node version. -2. **Proxy** — reachable or not. When reachable: `status`, `node_id`, `hub_mode`, - `hub_auth_status` (flag anything but `ok`), `last_sync_at` as a local time, `last_sync_error`, - pending inbound/outbound counts, and any `reauth_backoff_until` / `hello_rate_limit_until` - in the future. When unreachable: show the error, the endpoint it tried (`proxy_endpoint.url` - and `url_source`), and say that `evolver lifecycle start` starts the Proxy (the plugin also tries this once per dsh process); - reuse reports wait and are resent once it is up. -3. **Engine version** — `engine.running_version` (the Proxy actually serving, with - `proxy_endpoint.pid` and `started_at`) and `engine.path_version` (the `evolver` on PATH). - If `meets_minimum` is false, say network strategy recall needs `minimum_version` or newer and - give `npm install -g @evomap/evolver@latest`, then `evolver lifecycle restart`. - If `path_differs_from_running` is true, say the running Proxy is a different install from the - one on PATH and should be restarted from the upgraded one. If both versions are null, say the - version could not be determined. -4. **Strategy recall** — enabled or not; settings (`wait_ms`, `timeout_ms`, `min_similarity`, - `max_strategy_chars`); the last recall (`last.status`, `last.at`, `last.duration_ms`) and the - per-status `counts` since dsh started. Explain the last status in one line: - `injected` (name the title, steps and chars), `no_candidates` (the network had nothing for that - prompt), `all_filtered` (list `dropped` reasons: `too_few_steps`, `too_long`, `low_similarity`, - `already_injected`), `proxy_error` (show the error), `skipped` (prompt too short). If `last` is - null, no prompt has been recalled in this dsh process yet. -5. **Network claim** — check `~/.evomap/claim_url`. If it contains an HTTPS `evomap.ai` - link, report that the node is awaiting claim and show the link without modifying it. -6. **Reuse reports** — from `reuse_reports`, the recent `delivered` reports with the Hub's - verdict (`eligible`, `counted`, `weight`, and `reason` when not eligible, e.g. `daily_cap` - or `same_owner`) and every `rejected` one with its `reason` and `status`. A rejected - report means the Hub refused it as invalid; show it as a problem worth reporting. Both - lists are empty until a turn with an injected strategy ends in this dsh process. -7. **Evolution memory** — use an existing - `/memory/evolution/memory_graph.jsonl` when present; otherwise inspect - `~/.evolver/memory/evolution/memory_graph.jsonl`. Report the selected path and count of - non-empty JSONL rows without printing their contents. -8. **Full engine** — if `engine.path_version` is null, say that `npm install -g @evomap/evolver` - enables `/evolver-run`, `/evolver-review`, and `/evolver-solidify`. - -Finish with one line on overall readiness and the single most useful next action, if any. diff --git a/assets/commands/sync.md b/assets/commands/sync.md deleted file mode 100644 index f22f615..0000000 --- a/assets/commands/sync.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -description: Sync evolution assets (genes/capsules) between the local store and the EvoMap Hub. -argument-hint: "[--scope=all|purchased|published] [--type=Gene|Capsule] [--export=] [--dry-run]" ---- - -Sync Evolver assets with the EvoMap Hub. - -Decode the final `Invocation arguments (verbatim JSON string)` value. Use the installed -`evolver` executable, or `npx -y @evomap/evolver` when absent, and invoke its `sync` -subcommand followed by those exact arguments. Preserve `--dry-run`, scope, type, and export -paths exactly as supplied. - -Summarize how many assets were pulled or updated, any local-only assets, and the `.gepx` -archive path when `--export` was used. If node identity or Hub credentials are missing, -point the user to `/evolver-status` and the README's network setup section. diff --git a/assets/skills/capability-evolver/SKILL.md b/assets/skills/capability-evolver/SKILL.md index 3259001..67cedc9 100644 --- a/assets/skills/capability-evolver/SKILL.md +++ b/assets/skills/capability-evolver/SKILL.md @@ -1,100 +1,30 @@ --- name: capability-evolver -description: Self-evolution workflow for the agent. Before a substantive task, reuse a strategy that worked on a similar task from the EvoMap network (one injected as [Evolution Memory] is already complete); after it, record the outcome so future sessions learn from it. Use when the user starts non-trivial work (a feature, a fix, a refactor) or asks the agent to "evolve", "learn from this", or "remember how this went". +description: How Evolver's evolution memory works in this session and how to reach it. Use when the user asks about Evolver or EvoMap, an [Evolution Memory] block, or asks the agent to "evolve", "learn from this", or "remember how this went". An injected [Evolution Memory] block needs none of this: apply it. --- # Capability Evolver This plugin gives the agent a **persistent, auditable evolution memory** built on the -Genome Evolution Protocol (GEP). The goal is simple: stop re-solving the same problem from -scratch. Past outcomes — what worked, what failed — are carried forward into future -sessions. +Genome Evolution Protocol (GEP): past outcomes — what worked, what failed — are carried +forward into future sessions. All of it runs on its own; nothing here asks you to call a tool. ## What runs on its own -These seams work without you invoking anything: - -- **Behind the first prompt** (`agent/pre-step`, once per agent) — with `claimNudgeEnabled`, - surfaces a trusted pending node-claim link. -- **Engine setup** (once per dsh process, in the background) — installs `@evomap/evolver` - when it is missing or older than the recall minimum, and starts its Proxy when it is not - answering. One `[Evolver]` notice says what was done, or the manual steps when it failed. -- **Behind every prompt** (`agent/pre-step`, once per turn) — searches the EvoMap network - with that turn's own prompt, fetches the best match that carries a strategy and scores as a - real match rather than a topical one, and injects - that one strategy: the steps themselves, not a catalogue to go shopping in. Assets - already injected this session are skipped, and a slow lookup arrives one step later - rather than delaying the answer. -- **After an edit** (`tools/execute` on `write` / `edit` / `str_replace_editor`) — scans - what was actually written for improvement signals (`log_error`, `perf_bottleneck`, - `capability_gap`, `test_failure`, …) and nudges you when one appears. -- **Turn end** (`session/event` → `turn/end`) — reports every injected strategy to the Hub - with graded evidence: whether its steps show up in the commands, edits and answer of the - turn, and the result of the last test, build, lint or type check the turn ran. -- **Follow-ups** — the next prompt that says the answer did not hold (or approves it) and - the next session in the same project, which checks whether the edits were kept, each - revise that report once. - -## What you should do - -For any **substantive** task — a feature, a non-trivial fix, a refactor: - -1. **Before starting**, read the injected evolution memory. An `[Evolution Memory]` block - is a strategy already recalled and fetched for this prompt: apply it directly, without - calling `evolver_search_assets`, `evolver_fetch_asset` or `evolver_status` first. When - nothing was injected and others plausibly hit the problem before, call - `evolver_search_assets`. -2. **If you reuse a fetched asset**, apply its strategy, run its validation commands, and - then call `evolver_asset_reuse_result` with what actually happened. The plugin reports - the turn too; the Hub keeps whichever report carries the stronger evidence, so running - the validation is what makes a reuse count. -3. **Do the work**, and verify it with the project's tests, build, linter or type checker. -4. **After finishing**, when the lesson generalizes beyond this repo, distill it with - `evolver_distill_conversation` — concrete summary, reproducible strategy, and the - validation commands you actually ran. Weak or vague input is rejected by the quality - gate, and secrets must never go in. - -Trivial or purely conversational turns don't need any of this — skip it. - -## Signals - -| Signal | Fires on | -|---|---| -| `log_error` | errors, exceptions, failures described in the change | -| `perf_bottleneck` | timeout / slow / latency / OOM | -| `capability_gap` | "not supported" / "not implemented" | -| `user_feature_request` | adding a feature / new module | -| `test_failure` | reported failing tests or assertions | -| `deployment_issue` | build / CI / pipeline / rollback | -| `recurring_error` | "same error" / "still failing" / "keeps failing" | - -Signals are descriptive tags, not a verdict: a reuse report's outcome comes from the turn's -last check, or from how the turn ended when it ran none. - -## Tools - -The plugin registers native dsh tools that talk to the local EvoMap Proxy — no MCP hop: - -- `evolver_search_assets` — find reusable genes/capsules by signal or free text, for - substantive work that got no `[Evolution Memory]` injection. -- `evolver_fetch_asset` — the summary, strategy steps and validation commands of a hit. -- `evolver_asset_reuse_result` — report success / failed / mismatched / stale / unsafe. -- `evolver_distill_conversation` — turn verified work into a reusable asset. -- `evolver_publish_asset`, `evolver_poll`, `evolver_ack`, `evolver_status` — publish, - read and retire Hub decisions, and check the Proxy when another tool fails. - -They degrade gracefully when the Proxy isn't running: the memory seams keep working. -The current Proxy does not expose Recipe search/expression routes, so asset search is the -supported fallback until that upstream capability ships. - -## Full pipeline (optional) - -The **full evolution engine** — automated log analysis and the review-and-solidify cycle -that proposes and applies code improvements — is a separate CLI: - -```bash -npm install -g @evomap/evolver -``` - -It also provides the Proxy the tools above talk to. The `/evolver-run`, `/evolver-review` -and `/evolver-solidify` commands drive it; this plugin never invokes it on its own. +- **Behind every prompt** (`agent/pre-step`, once per turn) — recalls the strategy that best + fits the turn's prompt from the local history and the EvoMap network and injects its steps + as an `[Evolution Memory]` block. Apply them where they fit. +- **Turn end** — reports every injected strategy with the turn's evidence: whether its steps + show up in the commands, edits and answer, and the result of the last test, build, lint or + type check. A turn that never ends (the session goes away or is killed) is reported as aborted. +- **Follow-ups** — the next prompt that says the answer did not hold (or approves it), and the + next session in the same project, which checks whether the edits were kept, revise that report. +- **Session end** — the latest turn that finished verified, used tools and reused no injected + strategy is distilled into a reusable Gene, outside this conversation. + +## Commands the user can run + +`/evolver-status`, `/evolver-search `, `/evolver-distill [focus]` (distill now instead of +at session end), `/evolver-sync`, `/evolver-review`, `/evolver-run`, `/evolver-solidify`, and +`/evolver-evolve` for a deliberate checkpoint. They run in the plugin; point the user to them +rather than reproducing them by hand. diff --git a/src/commands.js b/src/commands.js index 3832910..104a10e 100644 --- a/src/commands.js +++ b/src/commands.js @@ -1,49 +1,174 @@ // SPDX-License-Identifier: MIT // Copyright (c) 2026 EvoMap -import { readdirSync, readFileSync } from 'node:fs'; +import { execFile } from 'node:child_process'; +import { readFileSync } from 'node:fs'; import { fileURLToPath } from 'node:url'; -import { join } from 'node:path'; import { createUserMessage } from '@deepseek-ai/dsh-llm'; +import { MIN_EVOLVER_VERSION } from './engine-version.js'; import { splitFrontmatter } from './frontmatter.js'; -const COMMANDS_DIR = fileURLToPath(new URL('../assets/commands/', import.meta.url)); +// Commands are how a person reaches Evolver, so nothing about them sits in the model's context: each one +// runs in the plugin and answers in the UI. Only /evolver-evolve hands the agent a one-off instruction, +// because reflecting on the current work is the agent's job. + const NAME_PREFIX = 'evolver-'; +const OUTPUT_MAX_CHARS = 4_000; +const CLI_TIMEOUT_MS = 10 * 60 * 1000; +const KILL_GRACE_MS = 5_000; +// The fallback runs whatever npm resolves, so it is held to the range this plugin is built against. +const CLI_PACKAGE = `@evomap/evolver@^${MIN_EVOLVER_VERSION}`; +const EVOLVE_PATH = fileURLToPath(new URL('../assets/commands/evolve.md', import.meta.url)); -function loadCommandFiles() { - return readdirSync(COMMANDS_DIR) - .filter((file) => file.endsWith('.md')) - .sort() - .map((file) => { - const { fields, body } = splitFrontmatter(readFileSync(join(COMMANDS_DIR, file), 'utf8')); - return { - name: `${NAME_PREFIX}${file.slice(0, -'.md'.length)}`, - description: fields.description ?? `Evolver ${file.slice(0, -'.md'.length)}`, - hint: fields['argument-hint'] ?? 'optional extra instructions', - body, - }; - }); +const success = (text) => ({ kind: 'success', text }); +const failure = (text) => ({ kind: 'error', text }); + +function shown(value) { + const text = typeof value === 'string' ? value : JSON.stringify(value, null, 2); + return text.length > OUTPUT_MAX_CHARS ? `${text.slice(0, OUTPUT_MAX_CHARS)}\n…` : text; +} + +/** + * Quotes group words the way a shell would (`--summary="two words"` stays one argument) and a backslash + * escapes the character after it, so a quote can be part of a value; nothing else is expanded. + */ +export function argumentsOf(rawInput) { + const args = []; + let current = ''; + let quote = null; + let started = false; + let escaped = false; + for (const char of rawInput) { + if (escaped) { + current += char; + escaped = false; + } else if (char === '\\' && quote !== "'") { + escaped = true; + started = true; + } else if (quote) { + if (char === quote) quote = null; + else current += char; + } else if (char === '"' || char === "'") { + quote = char; + started = true; + } else if (/\s/.test(char)) { + if (started) args.push(current); + current = ''; + started = false; + } else { + current += char; + started = true; + } + } + if (started) args.push(current); + return args; } -function promptMessage(body, rawInput) { - const invocation = rawInput.trim(); - const text = `${body}\n\nInvocation arguments (verbatim JSON string): ${JSON.stringify(invocation)}`; - return createUserMessage({ - content: [{ type: 'text', text }], - source: { kind: 'plugin:evolver', form: 'instructions' }, +// The CLI starts the Proxy and workers of its own, so a cancelled or timed-out run takes down the whole +// process group it leads rather than only the process this plugin started. Windows has no group to signal. +const grouped = process.platform !== 'win32'; + +function endTree(child) { + const target = grouped ? -child.pid : child.pid; + const signalTree = (name) => { + try { + process.kill(target, name); + } catch { + } + }; + signalTree('SIGTERM'); + setTimeout(() => signalTree('SIGKILL'), KILL_GRACE_MS).unref?.(); +} + +function execute(file, args, signal) { + return new Promise((resolve) => { + let timer = null; + const child = execFile(file, args, { maxBuffer: 16 * 1024 * 1024, detached: grouped }, (error, stdout, stderr) => { + clearTimeout(timer); + signal?.removeEventListener('abort', onAbort); + resolve({ error, output: `${stdout ?? ''}${stderr ? `\n${stderr}` : ''}`.trim() }); + }); + function onAbort() { + endTree(child); + } + if (!child.pid) return; + timer = setTimeout(onAbort, CLI_TIMEOUT_MS); + timer.unref?.(); + if (signal?.aborted) onAbort(); + else signal?.addEventListener('abort', onAbort, { once: true }); }); } -export function evolverCommands() { - return loadCommandFiles().map(({ name, description, hint, body }) => ({ - name, +// The installed `evolver` when there is one, else the published package through npx. +export async function runEvolverCli(args, { signal, run = execute } = {}) { + let { error, output } = await run('evolver', args, signal); + if (error?.code === 'ENOENT') ({ error, output } = await run('npx', ['-y', CLI_PACKAGE, ...args], signal)); + return error ? failure(shown(output || String(error.message))) : success(shown(output || 'done')); +} + +function cliCommand(subcommand, description, hint, runCli) { + return { + name: `${NAME_PREFIX}${subcommand}`, description, input: { hint }, + handler: ({ rawInput, signal }) => runCli([subcommand, ...argumentsOf(rawInput)], { signal }), + }; +} + +function evolveCommand() { + const { fields, body } = splitFrontmatter(readFileSync(EVOLVE_PATH, 'utf8')); + return { + name: `${NAME_PREFIX}evolve`, + description: fields.description, + input: { hint: 'optional focus' }, handler: ({ agent, rawInput }) => { - agent.followup(promptMessage(body, rawInput)); - return { kind: 'success', text: `Running /${name}.` }; + const focus = rawInput.trim(); + const text = focus ? `${body}\n\nFocus: ${focus}` : body; + agent.followup(createUserMessage({ content: [{ type: 'text', text }], source: { kind: 'plugin:evolver', form: 'instructions' } })); + return success(`Running /${NAME_PREFIX}evolve.`); + }, + }; +} + +/** + * `status` and `search` answer from the Proxy, `distill` distills the session's latest verified turn now, + * and the engine commands run the `evolver` executable. + */ +export function evolverCommands({ status, search, distillNow, runCli = runEvolverCli }) { + return [ + { + name: `${NAME_PREFIX}status`, + description: 'Show Evolver health: the Proxy, Hub auth, recent recalls and reuse reports.', + input: { hint: '' }, + handler: async ({ signal }) => success(shown(await status(signal))), + }, + { + name: `${NAME_PREFIX}search`, + description: 'Search the EvoMap network for reusable assets.', + input: { hint: '' }, + handler: async ({ rawInput, signal }) => { + const text = rawInput.trim(); + if (!text) return failure(`Usage: /${NAME_PREFIX}search `); + const result = await search(text, signal); + return result.ok ? success(shown(result.data)) : failure(String(result.error)); + }, + }, + { + name: `${NAME_PREFIX}distill`, + description: "Distill this session's latest verified turn into a reusable Gene now, instead of at session end.", + input: { hint: 'optional focus' }, + handler: async ({ agent, rawInput }) => { + const outcome = await distillNow(agent, rawInput.trim()); + if (outcome.status === 'nothing') return failure('Nothing to distill yet: no turn of this session finished verified.'); + return outcome.status === 'sent' ? success(shown(outcome.data ?? 'distilled')) : failure(`Distillation ${outcome.status}: ${outcome.error ?? ''}`.trim()); + }, }, - })); + evolveCommand(), + cliCommand('sync', 'Sync assets between the local store and the EvoMap Hub (evolver sync).', '[--scope=…] [--type=…] [--dry-run]', runCli), + cliCommand('review', "Review Evolver's pending evolved changes, then approve or reject them (evolver review).", '[--approve | --reject]', runCli), + cliCommand('run', 'Run one Evolver self-evolution cycle on this repository (evolver run).', '[--loop] [--dry-run] [--strategy=…]', runCli), + cliCommand('solidify', 'Solidify the working changes into a durable Gene or Capsule (evolver solidify).', '[--dry-run] [--intent=…] [--summary=…]', runCli), + ]; } diff --git a/src/config.js b/src/config.js index 5a714e7..1191955 100644 --- a/src/config.js +++ b/src/config.js @@ -16,6 +16,8 @@ export const Config = Schema.object({ assetPrimeMinSimilarity: Schema.number().min(0).max(1).default(0.3), assetPrimeWaitMs: Schema.number().step(1).min(1).max(MAX_TIMER_MS).default(6_000), assetPrimeTimeoutMs: Schema.number().step(1).min(1).max(MAX_TIMER_MS).default(8_000), + // Off by default: the skill's catalog line rides every turn and its body, once loaded, every later step. + skillEnabled: Schema.boolean().default(false), claimNudgeEnabled: Schema.boolean().default(false), claimNudgeTtlMs: Schema.number().step(1).min(1).max(MAX_TIMER_MS).default(12 * 60 * 60 * 1000), }); diff --git a/src/index.js b/src/index.js index c18b290..fc01373 100644 --- a/src/index.js +++ b/src/index.js @@ -7,11 +7,11 @@ import { appliedVerdictOf } from './applied.js'; import { evolverCommands } from './commands.js'; import { Config } from './config.js'; import { createRecallLog, createReportLog, pluginDiagnostics } from './diagnostics.js'; -import { EDIT_TOOL_NAMES, editedContent, editedPath } from './edited-content.js'; +import { EDIT_TOOL_NAMES } from './edited-content.js'; import { prepareEngine, setupNoticeText } from './engine-setup.js'; import { installedEvolverVersion } from './engine-version.js'; import { feedbackLabelOf } from './feedback-label.js'; -import { forgetSession, rememberInjected, takeUnreported } from './injected-assets.js'; +import { forgetSession, rememberInjected, takeOrphanedUnreported, takeUnreported } from './injected-assets.js'; import { recordReuseLocally } from './local-ledger.js'; import { noticeDue, pendingClaimUrl } from './onboarding.js'; import { hubGene, promptTextOf } from './prime.js'; @@ -20,8 +20,11 @@ import { createProxyClient } from './proxy.js'; import { createReportQueue } from './report-queue.js'; import { rememberForRetention, retentionReportsOf, takeDueRetentions } from './retention-pending.js'; import { correctionReportOf, immediateReportOf, reportedReuseOf } from './reuse.js'; +import { + candidateOf, claimLeftoverCandidates, closeCandidate, distillCandidate, dropCandidate, dueCandidates, modelOf, + isCurrentCandidate, rememberCandidate, settleCandidate, skipReason, +} from './session-distill.js'; import { sessionKeyOf } from './session-key.js'; -import { detectSignals } from './signals.js'; import { evolverSkillProvider } from './skill.js'; import { evolverTools } from './tools.js'; import { createTurnRecords, noteAnswer, noteFileBeforeEdit, noteToolResult } from './turn-record.js'; @@ -32,6 +35,7 @@ export { Config }; export const inject = ['tools']; const DEFAULT_PRIME_WAIT_MS = 6_000; +const DISTILL_TIMEOUT_MS = 120_000; function pluginMessage(text, formed) { return createUserMessage({ @@ -155,24 +159,9 @@ function primeSteps(ctx, config, primeFetch, engineSetup, recallLog, onPrompt) { }); } -function signalNotice(record, args) { - const signals = detectSignals(editedContent(args)); - if (signals.length === 0) return null; - const where = editedPath(args) || 'edited file'; - const noticeKey = `${where}\0${signals.join(',')}`; - if (record.notices.has(noticeKey)) return null; - record.notices.add(noticeKey); - return pluginMessage( - `[Evolution Signal] Detected: [${signals.join(', ')}] in ${where}.`, - { form: 'notice', summary: boundContextSummary(`Evolution signal: ${signals.join(', ')}`) }, - ); -} - function observeResult(record, exec, result, isEdit) { try { noteToolResult(record, exec, result, { isEdit }); - const notice = isEdit && !result?.isError ? signalNotice(record, exec.arguments) : null; - if (notice) exec.agent.inject(notice); } catch { } } @@ -190,24 +179,34 @@ function observeTools(ctx, editToolNames, records, projectDirOf) { }); } -function reportTurns(ctx, records, deliver, labelable, projectDirOf) { +function reportsOf(assets, { sessionId, turn, reasonKind, record }) { + return assets.map((asset) => immediateReportOf({ + asset, + sessionId, + turn: turn ?? asset.turn, + reasonKind, + checks: record.checks, + applied: appliedVerdictOf({ strategyText: asset.strategyText, ...record }).verdict, + })); +} + +// Every injected strategy is reported when its turn ends, with the turn's own evidence; the turn itself +// becomes this session's distillation candidate when it finished verified without reusing one. +function reportTurns(ctx, records, deliver, labelable, projectDirOf, prompts) { const closeTurn = (session, { turn, reason }) => { const sessionId = sessionKeyOf(session); const finished = records.take(sessionId); const reasonKind = reason?.kind; if (!outcomeOfReason(reasonKind)) return; - const reports = takeUnreported(sessionId, turn).map((asset) => immediateReportOf({ - asset, - sessionId, - turn, - reasonKind, - checks: finished.checks, - applied: appliedVerdictOf({ strategyText: asset.strategyText, ...finished }).verdict, - })); + const reports = reportsOf(takeUnreported(sessionId, turn), { sessionId, turn, reasonKind, record: finished }); const reuses = reports.map(reportedReuseOf); labelable.set(sessionId, reuses); deliver(reports.map((report) => ({ report, sessionId }))).catch(() => {}); rememberForRetention({ sessionId, projectDir: projectDirOf(session), reuses, editedBefore: finished.editedBefore }); + const reusedAsset = reuses.some((reuse) => reuse.applied === 'used'); + if (skipReason(reasonKind, finished, { reusedAsset })) return; + const model = modelOf(session?.deriveMessages?.() ?? []); + rememberCandidate(sessionId, candidateOf({ prompt: prompts.get(sessionId), record: finished, model, turn })); }; ctx.on('session/event', (session, event) => { @@ -217,22 +216,75 @@ function reportTurns(ctx, records, deliver, labelable, projectDirOf) { }); } -function onSessionStart(ctx, engineSetup, deliver, projectDirOf) { +function abortedReports(assets) { + return assets.map((asset) => ({ + report: reportsOf([asset], { sessionId: asset.sessionId, reasonKind: 'aborted', record: { checks: [], commands: [], paths: [], edits: [], answer: '' } })[0], + sessionId: asset.sessionId, + })); +} + +function onSessionStart(ctx, engineSetup, deliver, projectDirOf, distiller) { ctx.on('agent/session-start', ({ agent }) => { engineSetup.start(); if (isSubagent(agent.session)) return; const due = takeDueRetentions({ sessionId: sessionKeyOf(agent), projectDir: projectDirOf(agent.session) }); - deliver(due.flatMap(retentionReportsOf)).catch(() => {}); + deliver([...due.flatMap(retentionReportsOf), ...abortedReports(takeOrphanedUnreported())]).catch(() => {}); + distiller.distillLeftovers(); }); } -function followUpReporter(deliver, labelable) { +// The host's model service runs the session-end distillation; a candidate stays on disk until distilled (or +// given up), and one whose session is over — ended with a failed try, or killed with its process — is +// claimed and retried by the next session that starts. +function createDistiller(ctx, proxyFetch) { + let llm = null; + ctx.inject(['llm'], (scoped) => { + llm = scoped.llm; + }); + const run = async ({ sessionId, candidate }, focus, cancel) => { + const isCurrent = () => isCurrentCandidate(sessionId, candidate); + const outcome = await distillCandidate({ proxyFetch, llm, candidate, focus, timeoutMs: DISTILL_TIMEOUT_MS, cancel, isCurrent }); + settleCandidate(sessionId, candidate, { sent: outcome.status === 'sent' }); + return outcome; + }; + // One distillation per session at a time: a second flush while the first runs shares its outcome. + const inFlight = new Map(); + const once = (due, focus) => { + if (!inFlight.has(due.sessionId)) { + const controller = new AbortController(); + const outcome = run(due, focus, controller.signal).finally(() => inFlight.delete(due.sessionId)); + inFlight.set(due.sessionId, { outcome, controller }); + } + return inFlight.get(due.sessionId).outcome; + }; + return { + async distillSession(sessionId, focus = '') { + const [own] = dueCandidates({ sessionId }); + return own ? once(own, focus) : { status: 'nothing' }; + }, + distillLeftovers() { + for (const due of claimLeftoverCandidates()) once(due).catch(() => {}); + }, + /** The session's turn no longer deserves a Gene: its candidate goes, and a distillation already running stops. */ + forget(sessionId) { + dropCandidate(sessionId); + inFlight.get(sessionId)?.controller.abort(); + }, + }; +} + +// The next prompt is the only verdict on a turn available without asking anyone: when it reads as a +// correction, the reuse reports are revised and the turn is no longer worth distilling. +function followUpReporter(deliver, labelable, prompts, distiller) { return (agent, prompt) => { const sessionId = sessionKeyOf(agent); - if (!prompt || !labelable.has(sessionId)) return; + if (!prompt) return; + const label = feedbackLabelOf(prompt); + if (label === 'correction') distiller.forget(sessionId); + prompts.set(sessionId, prompt); + if (!labelable.has(sessionId)) return; const reuses = labelable.get(sessionId); labelable.delete(sessionId); - const label = feedbackLabelOf(prompt); const reports = reuses.map((reuse) => correctionReportOf(reuse, label)).filter((report) => report !== null); deliver(reports.map((report) => ({ report, sessionId }))).catch(() => {}); }; @@ -253,29 +305,49 @@ export function apply(ctx, config = {}) { const recallLog = createRecallLog(); const reportLog = createReportLog(); const diagnostics = () => pluginDiagnostics({ config, engineVersion: installedEvolverVersion, recallLog, reportLog, port: config.proxyPort }); - for (const tool of evolverTools(proxyFetch, { diagnostics })) ctx.tools.register(tool); - - ctx.inject(['skills'], (scoped) => { - scoped.skills.registerProvider(() => evolverSkillProvider); - }); - - ctx.inject(['commands'], (scoped) => { - for (const command of evolverCommands()) scoped.commands.register(command); - }); + const proxyOperations = Object.fromEntries(evolverTools(proxyFetch, { diagnostics }).map((tool) => [tool.name, tool])); const deliver = createReportQueue({ proxyFetch, recordLocally: recordReuseLocally, onSettled: reportLog.record }); const records = createTurnRecords(); const labelable = new Map(); + const prompts = new Map(); + const distiller = createDistiller(ctx, proxyFetch); + + // Nothing of Evolver but the recalled steps enters the model's context: no tools, and the skill only when + // configured. People reach it through commands that run here; reports and distillation run on session events. + if (config.skillEnabled) { + ctx.inject(['skills'], (scoped) => { + scoped.skills.registerProvider(() => evolverSkillProvider); + }); + } + ctx.inject(['commands'], (scoped) => { + const commands = evolverCommands({ + status: (signal) => proxyOperations.evolver_status.execute({}, { signal }), + search: async (text, signal) => { + try { + return { ok: true, data: await proxyOperations.evolver_search_assets.execute({ text, limit: 5 }, { signal }) }; + } catch (error) { + return { ok: false, error: String(error?.message ?? error) }; + } + }, + distillNow: (agent, focus) => distiller.distillSession(sessionKeyOf(agent), focus), + }); + for (const command of commands) scoped.commands.register(command); + }); - primeSteps(ctx, config, primeFetch, engineSetup, recallLog, followUpReporter(deliver, labelable)); + primeSteps(ctx, config, primeFetch, engineSetup, recallLog, followUpReporter(deliver, labelable, prompts, distiller)); observeTools(ctx, config.editToolNames ?? EDIT_TOOL_NAMES, records, projectDirOf); - reportTurns(ctx, records, deliver, labelable, projectDirOf); - onSessionStart(ctx, engineSetup, deliver, projectDirOf); + reportTurns(ctx, records, deliver, labelable, projectDirOf, prompts); + onSessionStart(ctx, engineSetup, deliver, projectDirOf, distiller); + ctx.on('session/flush', (session) => (isSubagent(session) ? undefined : distiller.distillSession(sessionKeyOf(session)))); ctx.on('session/disposed', (session) => { const sessionKey = sessionKeyOf(session); + deliver(abortedReports(takeUnreported(sessionKey, Infinity))).catch(() => {}); + closeCandidate(sessionKey); records.forget(sessionKey); labelable.delete(sessionKey); + prompts.delete(sessionKey); forgetSession(sessionKey); }); } diff --git a/src/injected-assets.js b/src/injected-assets.js index e1c6134..0261f97 100644 --- a/src/injected-assets.js +++ b/src/injected-assets.js @@ -1,7 +1,11 @@ // SPDX-License-Identifier: MIT // Copyright (c) 2026 EvoMap -import { hashedName, readState, removeState, statePath, writeState } from './state-file.js'; +import fs from 'node:fs'; +import path from 'node:path'; + +import { ownerIsGone } from './process-owner.js'; +import { hashedName, removeState, statePath, updateState } from './state-file.js'; const ENTRY_PRUNE_MS = 7 * 24 * 60 * 60 * 1000; @@ -12,16 +16,24 @@ function pruned(ledger, now) { })); } -export function withInjected(ledger, assetId, { turn, strategyText }, now) { - const kept = pruned(ledger, now); +/** + * Every entry in a ledger belongs to the session it is named after, so each injection also stamps the + * entries already there with the live owner: an entry written before this version, or by the process a + * resumed session ran in, would otherwise read as an unclaimable or orphaned one. + */ +const owned = (ledger, sessionId, pid) => + Object.fromEntries(Object.entries(ledger).map(([id, entry]) => [id, { ...entry, sessionId, pid }])); + +export function withInjected(ledger, assetId, { turn, strategyText, sessionId, pid }, now) { + const kept = owned(pruned(ledger, now), sessionId, pid); if (kept[assetId]) return kept; - return { ...kept, [assetId]: { turn, injectedAt: now, strategyText } }; + return { ...kept, [assetId]: { turn, injectedAt: now, strategyText, sessionId, pid } }; } export function unreportedOf(ledger, turn) { return Object.entries(ledger ?? {}) .filter(([, entry]) => !entry?.reportedAt && Number(entry?.turn) <= turn) - .map(([assetId, entry]) => ({ assetId, strategyText: entry.strategyText ?? '' })); + .map(([assetId, entry]) => ({ assetId, strategyText: entry.strategyText ?? '', turn: entry.turn, sessionId: entry.sessionId })); } export function withReported(ledger, assetIds, now) { @@ -30,21 +42,52 @@ export function withReported(ledger, assetIds, now) { return kept; } -const ledgerPath = (sessionKey) => statePath(hashedName('injected', sessionKey)); +const LEDGER_PREFIX = 'injected'; +const ledgerPath = (sessionKey) => statePath(hashedName(LEDGER_PREFIX, sessionKey)); export function rememberInjected(sessionKey, assetId, injection, now = Date.now()) { if (!sessionKey) return; - writeState(ledgerPath(sessionKey), withInjected(readState(ledgerPath(sessionKey), {}), assetId, injection, now)); + updateState(ledgerPath(sessionKey), {}, (ledger) => + withInjected(ledger, assetId, { ...injection, sessionId: sessionKey, pid: process.pid }, now)); } export function takeUnreported(sessionKey, turn, now = Date.now()) { if (!sessionKey) return []; - const ledger = readState(ledgerPath(sessionKey), {}); - const due = unreportedOf(ledger, turn); - if (due.length > 0) writeState(ledgerPath(sessionKey), withReported(ledger, due.map((asset) => asset.assetId), now)); + let due = []; + updateState(ledgerPath(sessionKey), {}, (ledger) => { + due = unreportedOf(ledger, turn); + return due.length > 0 ? withReported(ledger, due.map((asset) => asset.assetId), now) : undefined; + }); return due; } +/** + * Injections no turn end ever reported, from sessions whose dsh process is gone: a process killed mid-turn (a + * harness timeout, a closed terminal) leaves them behind. They are taken, marked reported and returned for an + * aborted report. A session that is merely idle keeps its open injections. + */ +export function takeOrphanedUnreported({ isGone = ownerIsGone, now = Date.now() } = {}) { + const dir = path.dirname(statePath('x')); + let files = []; + try { + files = fs.readdirSync(dir).filter((file) => file.startsWith(`${LEDGER_PREFIX}-`) && file.endsWith('.json')); + } catch { + return []; + } + const orphaned = []; + for (const file of files) { + updateState(path.join(dir, file), {}, (ledger) => { + const due = Object.entries(ledger) + .filter(([, entry]) => !entry?.reportedAt && entry?.sessionId && isGone(entry.pid)) + .map(([assetId, entry]) => ({ assetId, strategyText: entry.strategyText ?? '', turn: entry.turn, sessionId: entry.sessionId })); + if (due.length === 0) return undefined; + orphaned.push(...due); + return withReported(ledger, due.map((asset) => asset.assetId), now); + }); + } + return orphaned; +} + export function forgetSession(sessionKey) { if (sessionKey) removeState(ledgerPath(sessionKey)); } diff --git a/src/prime.js b/src/prime.js index d81da7a..4084ea9 100644 --- a/src/prime.js +++ b/src/prime.js @@ -179,8 +179,7 @@ export async function hubGene(proxyFetch, text, { `[Evolution Memory] ${readableNameOf(candidate)} (EvoMap network):`, ...steps.map((step, index) => `${index + 1}. ${step}`), '', - 'These are the complete steps already recalled from the network, so there is nothing more to search or fetch for this task.', - `Apply them where they fit, then report the outcome with evolver_asset_reuse_result for ${candidate.asset_id}.`, + 'Apply them where they fit.', ].join('\n'), }; } diff --git a/src/process-owner.js b/src/process-owner.js new file mode 100644 index 0000000..9a4b35c --- /dev/null +++ b/src/process-owner.js @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: MIT +// Copyright (c) 2026 EvoMap + +/** + * Whether the dsh process that wrote a state entry has exited. Signal 0 only probes: ESRCH means no such + * process; EPERM means it exists under another user. An entry without a pid (written by an older version) + * is never taken for a dead owner's. + */ +export function ownerIsGone(pid) { + if (!Number.isInteger(pid) || pid <= 0) return false; + if (pid === process.pid) return false; + try { + process.kill(pid, 0); + return false; + } catch (error) { + return error?.code === 'ESRCH'; + } +} diff --git a/src/session-distill.js b/src/session-distill.js new file mode 100644 index 0000000..fd2578e --- /dev/null +++ b/src/session-distill.js @@ -0,0 +1,238 @@ +// SPDX-License-Identifier: MIT +// Copyright (c) 2026 EvoMap + +import { createUserMessage } from '@deepseek-ai/dsh-llm'; + +import { ownerIsGone } from './process-owner.js'; +import { withoutSecrets } from './redact.js'; +import { readState, statePath, updateState } from './state-file.js'; + +// Distillation runs once a session, at its end, and outside the agent's conversation: each turn that +// finished verified only leaves a candidate on disk, the latest one wins, and the session end (or, after a +// session that was killed, the next session start) turns it into a Gene with one model call over a digest of +// the turn — never the whole history, which cost ten times as much for fewer usable Genes in the benchmark. +// A candidate leaves the disk only once distilled or given up after MAX_ATTEMPTS failed tries; a try that +// failed is claimed again by the next session that starts. + +const PROMPT_MAX_CHARS = 6_000; +const ANSWER_MAX_CHARS = 3_000; +const COMMAND_MAX_CHARS = 300; +const MAX_COMMANDS = 40; +const LIST_FIELDS = ['strategy', 'validation', 'signals', 'preconditions']; +const ARGUMENT_FIELDS = ['title', 'summary', ...LIST_FIELDS]; +const ITEM_NUMBER = /^\s*(?:\d+[.)]|[-*•])\s+/; +const MAX_ATTEMPTS = 3; + +const pendingPath = () => statePath('distill-pending.json'); + +/** + * Why a finished turn leaves no candidate, or null when it does. A turn that reused an injected strategy + * adds evidence to that asset through its reuse report rather than a near-copy of it; a chat reply without a + * single command has no method to keep. + */ +export function skipReason(reasonKind, record, { reusedAsset = false } = {}) { + if (reasonKind !== 'completed') return 'not_completed'; + if (!record.answer) return 'no_answer'; + if (record.checks.at(-1)?.failed) return 'failed_check'; + if (record.commands.length === 0 && record.edits.length === 0) return 'no_tool_use'; + if (reusedAsset) return 'asset_reused'; + return null; +} + +/** The provider and model behind the session's latest assistant message: the distillation runs on it. */ +export function modelOf(history) { + const source = history.findLast((message) => message.role === 'assistant')?.source; + return source?.provider && source?.model ? { provider: source.provider, model: source.model } : null; +} + +// Keys and tokens are stripped before anything is written to disk or sent to a model or the Proxy. +export function candidateOf({ prompt, record, model, turn, pid = process.pid, now = Date.now() }) { + return { + prompt: withoutSecrets(String(prompt ?? '')).slice(0, PROMPT_MAX_CHARS), + commands: record.commands.slice(-MAX_COMMANDS).map((command) => withoutSecrets(command).slice(0, COMMAND_MAX_CHARS)), + paths: [...new Set(record.paths)].map((path) => withoutSecrets(path)), + checks: record.checks.map(({ command, failed }) => ({ command: withoutSecrets(command), failed })), + answer: withoutSecrets(record.answer).slice(0, ANSWER_MAX_CHARS), + model, + turn, + pid, + attempts: 0, + at: now, + }; +} + +export function rememberCandidate(sessionId, candidate) { + if (!sessionId) return; + updateState(pendingPath(), {}, (pending) => ({ ...pending, [sessionId]: candidate })); +} + +export function dropCandidate(sessionId) { + if (!sessionId) return; + updateState(pendingPath(), {}, (pending) => { + if (!(sessionId in pending)) return undefined; + const { [sessionId]: dropped, ...rest } = pending; + return rest; + }); +} + +/** Whether this is still the session's candidate on disk: not dropped by a correction, nor replaced by a newer one. */ +export function isCurrentCandidate(sessionId, candidate) { + return readState(pendingPath(), {})[sessionId]?.at === candidate.at; +} + +/** A session that ended keeps its candidate due, so a distillation that failed at its end is tried again. */ +export function closeCandidate(sessionId) { + if (!sessionId) return; + updateState(pendingPath(), {}, (pending) => (pending[sessionId] + ? { ...pending, [sessionId]: { ...pending[sessionId], closed: true } } + : undefined)); +} + +// A candidate is up for distillation again once its session is over and no live process is working on it. +function leftover(candidate, isGone) { + const over = Boolean(candidate?.closed) || isGone(candidate?.pid); + return over && (!candidate?.claimedBy || isGone(candidate.claimedBy)); +} + +/** This session's own candidate, if it has one. Left on disk until settled. */ +export function dueCandidates({ sessionId }) { + const candidate = readState(pendingPath(), {})[sessionId]; + return candidate ? [{ sessionId, candidate }] : []; +} + +/** + * The candidates of sessions that are over — ended, or killed with their process — claimed for this process + * under the state lock, so two hosts starting at once cannot distill the same one twice. An idle session + * whose process still runs keeps its candidate however long it waits. + */ +export function claimLeftoverCandidates({ isGone = ownerIsGone, pid = process.pid } = {}) { + const claimed = []; + updateState(pendingPath(), {}, (pending) => { + const next = { ...pending }; + for (const [sessionId, candidate] of Object.entries(pending)) { + if (!leftover(candidate, isGone) || candidate.claimedBy === pid) continue; + next[sessionId] = { ...candidate, claimedBy: pid }; + claimed.push({ sessionId, candidate: next[sessionId] }); + } + return claimed.length > 0 ? next : undefined; + }); + return claimed; +} + +/** + * Removes a candidate once it was distilled, or counts a failed try and gives up after MAX_ATTEMPTS. A + * newer candidate the session wrote meanwhile is left alone. + */ +export function settleCandidate(sessionId, candidate, { sent }) { + updateState(pendingPath(), {}, (pending) => { + const current = pending[sessionId]; + if (!current || current.at !== candidate.at) return undefined; + const attempts = (current.attempts ?? 0) + 1; + if (sent || attempts >= MAX_ATTEMPTS) { + const { [sessionId]: settled, ...rest } = pending; + return rest; + } + // The claim is released, so the next session start can try this candidate again. + return { ...pending, [sessionId]: { ...current, attempts, claimedBy: null } }; + }); +} + +export function distillPrompt(candidate, focus = '') { + const lines = (items, render) => items.map(render).join('\n') || '(none)'; + const focusText = withoutSecrets(String(focus ?? '')); + return [ + 'Below is the record of a task an agent has just finished and verified. Turn it into one reusable Gene:', + 'a method another agent could follow on a similar task, written for the kind of data or system involved,', + 'not for these files, names or values. Do not include the answer itself.', + ...(focusText ? [`Focus on: ${focusText}`] : []), + 'Answer with a JSON object only, with these fields:', + ' title: short capability name', + ' summary: what was solved and under which conditions', + ' strategy: ordered, reproducible steps', + ' validation: checks that proved the result', + ' signals: keywords of the problem this generalizes to', + ' preconditions: when the method applies', + '', + '## Task', + candidate.prompt, + '', + '## Commands run', + lines(candidate.commands, (command) => `- ${command}`), + '', + '## Files read or written', + lines(candidate.paths, (path) => `- ${path}`), + '', + '## Checks', + lines(candidate.checks, (check) => `- ${check.failed ? 'FAILED' : 'passed'}: ${check.command}`), + '', + '## Final answer', + candidate.answer, + ].join('\n'); +} + +// Models sometimes write a list field as one numbered paragraph ("1. … 2. …"), which the core reads as one step. +function listOf(value) { + if (Array.isArray(value)) return value; + if (typeof value !== 'string') return value; + return value.split(/\n+|(?<=\.)\s+(?=\d+[.)]\s)/).map((item) => item.replace(ITEM_NUMBER, '').trim()).filter(Boolean); +} + +export function argumentsOf(text) { + const match = String(text ?? '').match(/\{[\s\S]*\}/); + if (!match) return null; + try { + const parsed = JSON.parse(match[0]); + if (typeof parsed?.summary !== 'string' || !parsed.summary.trim()) return null; + for (const field of LIST_FIELDS) if (field in parsed) parsed[field] = listOf(parsed[field]); + return Object.fromEntries(ARGUMENT_FIELDS.filter((field) => parsed[field] !== undefined).map((field) => [field, parsed[field]])); + } catch { + return null; + } +} + +export function distillBody(candidate, args, usage) { + return { + platform: 'dsh', + trigger: 'session_end', + user_prompt: candidate.prompt, + // The core rejects a Gene that repeats this answer's values (answer_leak). + final_answer: candidate.answer, + ...args, + execution: { trace: candidate.checks.map((check) => ({ command: check.command, exit: check.failed ? 1 : 0 })) }, + persist: true, + publish: false, + // What this distillation itself cost, kept apart from the agent's own tokens. + ...(usage ? { distill_usage: usage } : {}), + }; +} + +async function answerOf(llm, model, text, signal) { + let answer = ''; + let usage = null; + const messages = [createUserMessage({ content: [{ type: 'text', text }], source: { kind: 'plugin:evolver', form: 'instructions' } })]; + for await (const chunk of llm.stream({ ...model, messages, signal })) { + if (chunk.type === 'text-delta') answer += chunk.text; + if (chunk.type === 'usage') usage = chunk.usage; + } + return { answer, usage }; +} + +/** + * Distill one candidate through the host's model service. `cancel` stops it and `isCurrent` is asked once more + * before anything is persisted, since the candidate may have been dropped while the model answered. Resolves to + * what happened; never throws. + */ +export async function distillCandidate({ proxyFetch, llm, candidate, focus, timeoutMs, cancel, isCurrent = () => true }) { + if (!llm || !candidate?.model) return { status: 'no_model' }; + const signal = AbortSignal.any([AbortSignal.timeout(timeoutMs), ...(cancel ? [cancel] : [])]); + try { + const { answer, usage } = await answerOf(llm, candidate.model, distillPrompt(candidate, focus), signal); + const args = argumentsOf(answer); + if (!args) return { status: 'unparsed', usage }; + if (cancel?.aborted || !isCurrent()) return { status: 'dropped', usage }; + const result = await proxyFetch('POST', '/conversation/distill', distillBody(candidate, args, usage), signal); + return result?.ok ? { status: 'sent', usage, data: result.data } : { status: 'failed', error: String(result?.error) }; + } catch (error) { + return { status: 'failed', error: String(error?.message ?? error) }; + } +} diff --git a/src/skill.js b/src/skill.js index 4093c40..ba1d13e 100644 --- a/src/skill.js +++ b/src/skill.js @@ -13,7 +13,7 @@ const SKILL_BODY_URL = new URL('SKILL.md', SKILL_DIR); const SUMMARY = { name: 'capability-evolver', description: - 'Self-evolution workflow for the agent. Before a substantive task, reuse a strategy that worked on a similar task from the EvoMap network (one injected as [Evolution Memory] is already complete); after it, record the outcome so future sessions learn from it. Use when the user starts non-trivial work (a feature, a fix, a refactor) or asks the agent to "evolve", "learn from this", or "remember how this went".', + 'How Evolver\'s evolution memory works in this session and how to reach it. Use when the user asks about Evolver or EvoMap, an [Evolution Memory] block, or asks the agent to "evolve", "learn from this", or "remember how this went". An injected [Evolution Memory] block needs none of this: apply it.', invocation: { modelInvocable: true, userInvocable: true }, provider: PROVIDER_NAME, source: 'custom', diff --git a/src/state-file.js b/src/state-file.js index 90b5a6a..d6e5ba8 100644 --- a/src/state-file.js +++ b/src/state-file.js @@ -6,6 +6,8 @@ import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; +import { ownerIsGone } from './process-owner.js'; + export function statePath(name) { return path.join(os.homedir(), '.evolver', 'state', name); } @@ -39,6 +41,83 @@ export function writeState(target, value) { } } +const LOCK_WAIT_MS = 2_000; +const LOCK_RETRY_MS = 20; + +// Atomics.wait is the only sleep that holds a synchronous read-modify-write together. +function pause(ms) { + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); +} + +function contentOf(file) { + try { + return fs.readFileSync(file, 'utf8'); + } catch { + return null; + } +} + +// A lock holds its owner's pid and a token unique to that acquisition, so a lock is told apart from a newer one +// the same process, or another, took after it. +const ownerPidOf = (token) => Number(token.split(':')[0]); + +// Another process may reclaim the same dead lock and a third take a fresh one before this removal happens, so the +// lock is first moved aside, and only what was inspected as dead is deleted; anything else is put back. +function reclaim(lockPath, deadToken) { + const aside = `${lockPath}.${crypto.randomBytes(6).toString('hex')}.reclaimed`; + try { + fs.renameSync(lockPath, aside); + } catch { + return; + } + if (contentOf(aside) !== deadToken) { + try { + fs.linkSync(aside, lockPath); + } catch { + } + } + removeState(aside); +} + +function acquire(lockPath, deadline) { + const token = `${process.pid}:${crypto.randomBytes(8).toString('hex')}`; + for (;;) { + try { + fs.mkdirSync(path.dirname(lockPath), { recursive: true, mode: 0o700 }); + fs.writeFileSync(lockPath, token, { mode: 0o600, flag: 'wx' }); + return token; + } catch (error) { + if (error?.code !== 'EEXIST') return null; + const held = contentOf(lockPath); + if (held !== null && ownerIsGone(ownerPidOf(held))) reclaim(lockPath, held); + else if (Date.now() >= deadline) return null; + else pause(LOCK_RETRY_MS); + } + } +} + +function release(lockPath, token) { + if (contentOf(lockPath) === token) removeState(lockPath); +} + +/** + * Reads a state file, applies `mutate` and writes the result back while holding an exclusive lock, so two + * dsh processes sharing `~/.evolver/state` cannot overwrite each other's entries. A lock whose owner died is + * taken over; a lock that cannot be had within LOCK_WAIT_MS is given up on and the update runs unguarded, + * which is still the behaviour of an unlocked write and never blocks a session. + */ +export function updateState(target, fallback, mutate) { + const lockPath = `${target}.lock`; + const token = acquire(lockPath, Date.now() + LOCK_WAIT_MS); + try { + const next = mutate(readState(target, fallback)); + if (next !== undefined) writeState(target, next); + return next; + } finally { + if (token) release(lockPath, token); + } +} + export function removeState(target) { try { fs.rmSync(target, { force: true }); diff --git a/src/turn-record.js b/src/turn-record.js index 81a5744..5df9bbb 100644 --- a/src/turn-record.js +++ b/src/turn-record.js @@ -12,7 +12,7 @@ const CHECK_COMMAND_MAX_CHARS = 200; const PATH_KEYS = ['path', 'file_path']; function freshTurnRecord() { - return { notices: new Set(), checks: [], commands: [], paths: [], edits: [], editedChars: 0, editedBefore: new Map(), answer: '' }; + return { checks: [], commands: [], paths: [], edits: [], editedChars: 0, editedBefore: new Map(), answer: '' }; } export function createTurnRecords() { diff --git a/test/applied.test.js b/test/applied.test.js index eca22fb..b9f1ef8 100644 --- a/test/applied.test.js +++ b/test/applied.test.js @@ -11,7 +11,7 @@ const STRATEGY = [ '3. Wrap the client in a singleton factory module', '4. Close the connection in the global teardown hook', '', - 'Apply it where it fits, then report the outcome with evolver_asset_reuse_result for sha256:abc.', + 'Apply them where they fit.', ].join('\n'); test('only the numbered steps of an injected strategy are matched', () => { diff --git a/test/apply.test.js b/test/apply.test.js index d9a3a47..8f9b64f 100644 --- a/test/apply.test.js +++ b/test/apply.test.js @@ -20,6 +20,7 @@ test('configuration applies defaults and rejects invalid values', () => { assert.deepEqual(configured.editToolNames, ['write', 'edit', 'str_replace_editor']); assert.equal(configured.proxyTimeoutMs, 8_000); assert.equal(configured.claimNudgeEnabled, false); + assert.equal(configured.skillEnabled, false); assert.equal(configured.assetPrimeEnabled, true); assert.equal(configured.assetPrimeWaitMs, 6_000); assert.ok( @@ -31,27 +32,14 @@ test('configuration applies defaults and rejects invalid values', () => { assert.throws(() => apply(fakeContext().ctx, { projectDir: join(tmpdir(), 'definitely-missing-evolver-dir') }), /not a directory/); }); -test('apply registers every surface and lifecycle listener', () => { +test('apply puts nothing of Evolver in the model context: no tools, no skill, only commands and listeners', () => { const { ctx, registered, listeners } = fakeContext(); apply(ctx, { projectDir: mkdtempSync(join(tmpdir(), 'evolver-')) }); + assert.deepEqual(registered.tools, []); + assert.deepEqual(registered.skills, []); assert.deepEqual( - registered.tools.map((tool) => tool.name).sort(), - [ - 'evolver_ack', - 'evolver_asset_reuse_result', - 'evolver_distill_conversation', - 'evolver_fetch_asset', - 'evolver_poll', - 'evolver_publish_asset', - 'evolver_search_assets', - 'evolver_status', - ], - ); - assert.equal(registered.skills.length, 1); - assert.equal(registered.skills[0].name, 'evolver'); - assert.deepEqual( - registered.commands.map((command) => command.name), + registered.commands.map((command) => command.name).sort(), [ 'evolver-distill', 'evolver-evolve', @@ -69,21 +57,66 @@ test('apply registers every surface and lifecycle listener', () => { 'agent/session-start', 'session/disposed', 'session/event', + 'session/flush', 'tools/execute', ]); }); -test('command invocation preserves arguments without Claude placeholders', () => { +test('the capability-evolver skill is registered only when configured', () => { + const off = fakeContext(); + apply(off.ctx, Config({ projectDir: mkdtempSync(join(tmpdir(), 'evolver-')) })); + assert.deepEqual(off.registered.skills, []); + + const on = fakeContext(); + apply(on.ctx, Config({ projectDir: mkdtempSync(join(tmpdir(), 'evolver-')), skillEnabled: true })); + assert.equal(on.registered.skills.length, 1); + assert.deepEqual(on.registered.tools, []); +}); + +test('engine commands run the evolver executable with the given arguments, without the model', async () => { const { ctx, registered } = fakeContext(); apply(ctx, { projectDir: mkdtempSync(join(tmpdir(), 'evolver-')) }); const messages = []; const command = registered.commands.find((candidate) => candidate.name === 'evolver-run'); + assert.match(command.input.hint, /--dry-run/); + + const { evolverCommands } = await import('../src/commands.js'); + const calls = []; + const run = evolverCommands({ status: async () => ({}), search: async () => ({}), distillNow: async () => ({}), + runCli: async (args) => { calls.push(args); return { kind: 'success', text: 'ok' }; } }) + .find((candidate) => candidate.name === 'evolver-run'); + const result = await run.handler({ agent: { followup: (message) => messages.push(message) }, rawInput: ' --dry-run --loop ' }); + + assert.deepEqual(calls, [['run', '--dry-run', '--loop']]); + assert.deepEqual(result, { kind: 'success', text: 'ok' }); + assert.deepEqual(messages, [], 'nothing reaches the model'); +}); + +test('command arguments keep quoted words together, and a missing evolver falls back to a pinned npx', async () => { + const { argumentsOf, runEvolverCli } = await import('../src/commands.js'); + const { MIN_EVOLVER_VERSION } = await import('../src/engine-version.js'); + assert.deepEqual(argumentsOf(' --dry-run --summary="two words" '), ['--dry-run', '--summary=two words']); + assert.deepEqual(argumentsOf('--summary="a \\"quoted\\" word" --intent=a\\ b'), ['--summary=a "quoted" word', '--intent=a b']); - command.handler({ agent: { followup: (message) => messages.push(message) }, rawInput: ' --dry-run ' }); + const calls = []; + const run = async (file, args) => { + calls.push([file, ...args]); + return file === 'evolver' ? { error: Object.assign(new Error('spawn evolver ENOENT'), { code: 'ENOENT' }) } : { error: null, output: 'synced' }; + }; + assert.deepEqual(await runEvolverCli(['sync', '--dry-run'], { run }), { kind: 'success', text: 'synced' }); + assert.deepEqual(calls, [['evolver', 'sync', '--dry-run'], ['npx', '-y', `@evomap/evolver@^${MIN_EVOLVER_VERSION}`, 'sync', '--dry-run']]); +}); + +test('/evolver-evolve is the one command that hands the agent an instruction', () => { + const { ctx, registered } = fakeContext(); + apply(ctx, { projectDir: mkdtempSync(join(tmpdir(), 'evolver-')) }); + const messages = []; + const command = registered.commands.find((candidate) => candidate.name === 'evolver-evolve'); + + command.handler({ agent: { followup: (message) => messages.push(message) }, rawInput: ' the retry path ' }); const text = messages[0].content[0].text; - assert.match(text, /Invocation arguments \(verbatim JSON string\): "--dry-run"/); - assert.doesNotMatch(text, /\$ARGUMENTS|\/evolver:/); - assert.match(command.input.hint, /--dry-run/); + assert.match(text, /Focus: the retry path$/); + assert.doesNotMatch(text, /evolver_[a-z_]+/, 'no tool the agent does not have'); }); test('every turn looks the Hub up with its own prompt, without repeating assets', async () => { @@ -121,11 +154,11 @@ test('every turn looks the Hub up with its own prompt, without repeating assets' ); assert.equal(requests[0].body.text, 'add a retry to the uploader'); assert.equal(requests[1].body.text, 'now make the download resumable'); - assert.match(first.at(-1).content[0].text, /evolver_asset_reuse_result for sha256:abc\./); + assert.match(first.at(-1).content[0].text, /\n\nApply them where they fit\.$/); + assert.doesNotMatch(first.at(-1).content[0].text, /sha256:|evolver_/, 'no id or tool the agent cannot use'); assert.match(first.at(-1).content[0].text, /^1\. Retry with backoff\.$/m); assert.deepEqual(sameTurn, []); assert.equal(second.length, 1); - assert.match(second[0].content[0].text, /sha256:def/); assert.match(second[0].content[0].text, /^1\. Chunk the download\.$/m); assert.doesNotMatch(second[0].content[0].text, /sha256:abc/); } finally { @@ -273,7 +306,6 @@ test('a search slower than the wait budget injects itself instead of holding the assert.deepEqual(primed, [], 'a step that waited for the slow search would carry its strategy'); await untilInjected(injected, 1); - assert.match(injected[0].content[0].text, /evolver_asset_reuse_result for sha256:slow\./); assert.match(injected[0].content[0].text, /^1\. Arrived late\.$/m); } finally { process.env.HOME = home; @@ -281,25 +313,6 @@ test('a search slower than the wait budget injects itself instead of holding the } }); -test('an edit carrying a signal nudges the agent once', async () => { - const { ctx, listeners } = fakeContext(); - apply(ctx, { projectDir: mkdtempSync(join(tmpdir(), 'evolver-')) }); - const { agent, injected } = fakeAgent(); - - const run = (name, args, result) => listeners.get('tools/execute')({ name, agent, arguments: args }, async () => result); - await run('write', { path: '/a.ts', content: 'the deploy failed again' }, { isError: false }); - await run('write', { path: '/a.ts', content: 'the deploy failed again' }, { isError: false }); - await run('write', { path: '/b.ts', content: 'all good' }, { isError: false }); - await run('read', { path: '/c.ts', content: 'the deploy failed' }, { isError: false }); - await run('write', { path: '/d.ts', content: 'the deploy failed' }, { isError: true }); - - assert.equal(injected.length, 1); - assert.equal(injected[0].content[0].text, '[Evolution Signal] Detected: [deployment_issue, log_error] in /a.ts.'); - assert.equal(injected[0].source.kind, 'plugin:evolver'); - assert.equal(injected[0].source.form, 'notice'); - assert.ok(injected[0].source.summary.length > 0 && injected[0].source.summary.length <= 120); -}); - test('a fetched asset renders as reusable prose, not the raw envelope', async () => { const { evolverTools } = await import('../src/tools.js'); const envelope = { diff --git a/test/injected-assets.test.js b/test/injected-assets.test.js index 890223e..dfa7392 100644 --- a/test/injected-assets.test.js +++ b/test/injected-assets.test.js @@ -6,8 +6,8 @@ import { unreportedOf, withInjected, withReported } from '../src/injected-assets const DAY = 24 * 60 * 60 * 1000; test('an injection keeps its turn and strategy until reported', () => { - const ledger = withInjected({}, 'sha256:a', { turn: 2, strategyText: '1. Step' }, 0); - assert.deepEqual(unreportedOf(ledger, 2), [{ assetId: 'sha256:a', strategyText: '1. Step' }]); + const ledger = withInjected({}, 'sha256:a', { turn: 2, strategyText: '1. Step', sessionId: 'session-a' }, 0); + assert.deepEqual(unreportedOf(ledger, 2), [{ assetId: 'sha256:a', strategyText: '1. Step', turn: 2, sessionId: 'session-a' }]); assert.deepEqual(unreportedOf(withReported(ledger, ['sha256:a'], 1), 2), []); }); @@ -16,6 +16,17 @@ test('an asset is recorded once per session', () => { assert.equal(withInjected(first, 'sha256:a', { turn: 3, strategyText: 'two' }, 1)['sha256:a'].turn, 1); }); +test('an entry written before this version is stamped with the live owner, so an orphan scan can claim it', () => { + const legacy = { 'sha256:old': { turn: 1, injectedAt: 0, strategyText: 'one' } }; + const ledger = withInjected(legacy, 'sha256:new', { turn: 2, strategyText: 'two', sessionId: 'session-a', pid: 42 }, 1); + assert.deepEqual(ledger['sha256:old'], { turn: 1, injectedAt: 0, strategyText: 'one', sessionId: 'session-a', pid: 42 }); +}); + +test('a session that resumed in another process owns the entries it already had', () => { + const first = withInjected({}, 'sha256:a', { turn: 1, strategyText: 'one', sessionId: 'session-a', pid: 1 }, 0); + assert.equal(withInjected(first, 'sha256:a', { turn: 3, strategyText: 'two', sessionId: 'session-a', pid: 2 }, 1)['sha256:a'].pid, 2); +}); + test('a strategy that lands after its turn ended is reported at the next turn end', () => { const ledger = withInjected({}, 'sha256:a', { turn: 2, strategyText: '' }, 0); assert.deepEqual(unreportedOf(ledger, 1), []); diff --git a/test/instructions.test.js b/test/instructions.test.js index 727e0bc..d97107c 100644 --- a/test/instructions.test.js +++ b/test/instructions.test.js @@ -5,23 +5,11 @@ import { join } from 'node:path'; import { test } from 'node:test'; const root = fileURLToPath(new URL('../', import.meta.url)); -const commandsDir = join(root, 'assets', 'commands'); -function commandBodies() { - return readdirSync(commandsDir) - .filter((file) => file.endsWith('.md')) - .map((file) => readFileSync(join(commandsDir, file), 'utf8')) - .join('\n'); -} - -test('bundled instructions use DSH command and invocation semantics', () => { - const commands = commandBodies(); - const skill = readFileSync(join(root, 'assets', 'skills', 'capability-evolver', 'SKILL.md'), 'utf8'); - - assert.doesNotMatch(commands, /\$ARGUMENTS|\/evolver:|User input:/); - assert.match(commands, /Invocation arguments \(verbatim JSON string\)/); - assert.doesNotMatch(skill, /`SessionStart`|`PostToolUse`|`Stop`|bundles a lightweight MCP bridge/); - assert.match(skill, /agent\/pre-step/); - assert.match(skill, /evolver_asset_reuse_result/); - assert.match(skill, /evolver_ack/); +test('the one bundled instruction names no tool the agent no longer has', () => { + const files = readdirSync(join(root, 'assets', 'commands')); + assert.deepEqual(files, ['evolve.md']); + const evolve = readFileSync(join(root, 'assets', 'commands', 'evolve.md'), 'utf8'); + assert.doesNotMatch(evolve, /evolver_[a-z_]+|\$ARGUMENTS|\/evolver:/); + assert.match(evolve, /\/evolver-distill/); }); diff --git a/test/prime.test.js b/test/prime.test.js index 57c0863..50f3e0b 100644 --- a/test/prime.test.js +++ b/test/prime.test.js @@ -51,8 +51,8 @@ test('one call recalls by text and injects that asset\'s strategy alone', async assert.equal(calls[0].signal, controller.signal); assert.deepEqual(ids, ['sha256:abc']); assert.match(text, /\[Evolution Memory\] Retry the upload with backoff\. \(EvoMap network\)/); - assert.match(text, /nothing more to search or fetch/); - assert.match(text, /evolver_asset_reuse_result for sha256:abc\./); + assert.match(text, /\n\nApply them where they fit\.$/); + assert.doesNotMatch(text, /sha256:|evolver_|search or fetch/, 'nothing that sends the agent to a tool'); assert.match(text, /^1\. Measure the failure rate first\.$/m); assert.match(text, /^4\. Alert on the cap\.$/m); assert.doesNotMatch(text, /npm test/); @@ -81,7 +81,6 @@ test('an asset already injected this session is passed over', async () => { assert.deepEqual(ids, ['sha256:next']); assert.doesNotMatch(text, /sha256:abc|Measure the failure rate/); assert.match(text, /\[Evolution Memory\] Capsule \(EvoMap network\)/); - assert.match(text, /evolver_asset_reuse_result for sha256:next\./); assert.match(text, /^1\. Drain the queue first\.$/m); }); @@ -171,7 +170,6 @@ test('a named gene is announced by its name, and its hash only where it gets use assert.match(text, /^\[Evolution Memory\] Chinese Social Media Writing Template \(EvoMap network\):$/m); assert.doesNotMatch(text.split('\n')[0], /sha256:/, 'the hash is not what the header is for'); - assert.match(text, /evolver_asset_reuse_result for sha256:named\./); assert.doesNotMatch(text, /no business being in front of the model|Also long/); }); diff --git a/test/reporting.test.js b/test/reporting.test.js index a1dc77f..f17356d 100644 --- a/test/reporting.test.js +++ b/test/reporting.test.js @@ -90,10 +90,7 @@ test('a recorded dsh session reports each turn with the evidence it left', async reason: 'Injected by Evolver into dsh turn 2; applied: unknown; no check ran.', evidence: { tier: 'none', applied: 'partial', aborted: false, stage: 'immediate' }, }); - assert.deepEqual( - injected.map((message) => message.content[0].text), - ['[Evolution Signal] Detected: [user_feature_request] in /home/dev/fair-play/README.md.'], - ); + assert.deepEqual(injected, [], 'edits add nothing to the model context'); await primedBy(listeners, agent, '不对，我问的是推理小说里的意思', 3); await untilCount(proxy.requests, (request) => request.path === '/asset/reuse-result', 3); @@ -239,7 +236,7 @@ test('a report the Hub refuses is settled, shown in evolver_status and never res await new Promise((resolve) => { setTimeout(resolve, 150); }); assert.equal(reportsIn(proxy.requests).length, 1); - const status = await registered.tools.find((tool) => tool.name === 'evolver_status').execute({}, {}); + const status = JSON.parse((await registered.commands.find((command) => command.name === 'evolver-status').handler({})).text); assert.deepEqual(status.reuse_reports.rejected.map(({ task_id: taskId, reason, status: code }) => ({ taskId, reason, code })), [ { taskId: 'dsh:session-refused:1', reason: 'invalid_evidence', code: 400 }, ]); @@ -266,7 +263,7 @@ test('a report the Hub could not take yet is resent at the next turn end and wri const [first, second] = reportsIn(proxy.requests); assert.deepEqual(second, first); - const status = await registered.tools.find((tool) => tool.name === 'evolver_status').execute({}, {}); + const status = JSON.parse((await registered.commands.find((command) => command.name === 'evolver-status').handler({})).text); assert.deepEqual(status.reuse_reports.delivered.map(({ eligible, counted, weight }) => ({ eligible, counted, weight })), [ { eligible: true, counted: true, weight: 0.6 }, ]); @@ -288,26 +285,6 @@ test('a report with no evidence never reaches the local ledger', async (t) => { assert.equal(existsSync(join(process.env.HOME, '.evomap', 'evolution', 'root_events.jsonl')), false); }); -test("the model's own report carries L5 evidence, passes the Hub verdict through, and leaves the automatic one in place", async (t) => { - const { proxy, listeners, registered } = await harness(t, 'evolver-self-report', hubAnswering({ - assets: () => [{ asset_id: 'sha256:owned', strategy: STEPS }], - reuse: (body) => ({ recorded: true, eligible: true, counted: body.evidence.tier !== 'L5', weight: body.evidence.tier === 'L5' ? 0.1 : 0 }), - })); - const { agent } = fakeAgent({ sessionId: 'session-owned' }); - await primedBy(listeners, agent, 'add a retry to the uploader', 1); - - const tool = registered.tools.find((candidate) => candidate.name === 'evolver_asset_reuse_result'); - const receipt = await tool.execute({ asset_id: 'sha256:owned', outcome: 'mismatched', reason: 'not a fit' }, {}); - assert.deepEqual(receipt, { recorded: true, eligible: true, counted: false, weight: 0.1 }); - endTurn(listeners, agent, 1); - await untilCount(proxy.requests, (request) => request.path === '/asset/reuse-result', 2); - - const [own, automatic] = reportsIn(proxy.requests); - assert.deepEqual(own.evidence, { tier: 'L5', applied: 'unused', stage: 'immediate' }); - assert.equal(automatic.task_id, 'dsh:session-owned:1'); - assert.equal(automatic.evidence.tier, 'none'); -}); - test('an injected asset is reported from disk after the process that injected it is gone', async (t) => { const { proxy, projectDir, listeners } = await harness(t, 'evolver-durable', oneAsset('sha256:durable')); const { agent } = fakeAgent({ sessionId: 'session-durable', cwd: projectDir }); @@ -350,3 +327,4 @@ test('subagent turns and synthesized turn ends report nothing', async (t) => { await new Promise((resolve) => { setTimeout(resolve, 150); }); assert.deepEqual(reportsIn(proxy.requests), []); }); + diff --git a/test/session-distill.test.js b/test/session-distill.test.js new file mode 100644 index 0000000..8e2fd6a --- /dev/null +++ b/test/session-distill.test.js @@ -0,0 +1,304 @@ +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { test } from 'node:test'; + +import { Config, apply } from '../src/index.js'; +import { + argumentsOf, candidateOf, claimLeftoverCandidates, closeCandidate, distillCandidate, distillPrompt, dueCandidates, + isCurrentCandidate, rememberCandidate, settleCandidate, skipReason, +} from '../src/session-distill.js'; +import { fakeAgent, fakeContext, freshHome, primedBy, projectDirectory, startProxy, untilCount, untilRequest } from './support/host.js'; + +const MODEL = { provider: 'gateway', model: 'deepseek-flash' }; +const GENE = { title: 'Retry uploads', summary: 'Retry a flaky upload with backoff.', strategy: '1. Measure. 2. Back off.', signals: ['retry'] }; + +function record(overrides = {}) { + return { checks: [], commands: ['npm test'], paths: ['src/up.js'], edits: [], answer: 'Done: retries added.', ...overrides }; +} + +function fakeLlm(fixed = JSON.stringify(GENE), calls = []) { + return { + calls, + answer: undefined, + hold: undefined, + async *stream(request) { + calls.push(request); + await this.hold; + yield { type: 'text-delta', text: this.answer ?? fixed }; + yield { type: 'usage', usage: { inputTokens: 900, outputTokens: 120 } }; + }, + }; +} + +// A candidate is settled once the Proxy has answered, a moment after its request arrived. +async function untilSettled(sessionId, deadlineMs = 15_000) { + const start = Date.now(); + while (dueCandidates({ sessionId }).length > 0) { + if (Date.now() - start > deadlineMs) assert.fail(`the candidate of ${sessionId} was never settled`); + await new Promise((resolve) => { setTimeout(resolve, 10); }); + } +} + +function session(id) { + return { id, header: {}, deriveMessages: () => [{ role: 'assistant', source: MODEL }] }; +} + +test('only a verified, completed turn that used tools and reused no strategy becomes a candidate', () => { + assert.equal(skipReason('completed', record()), null); + assert.equal(skipReason('aborted', record()), 'not_completed'); + assert.equal(skipReason('completed', record({ answer: '' })), 'no_answer'); + assert.equal(skipReason('completed', record({ checks: [{ command: 'npm test', failed: true }] })), 'failed_check'); + assert.equal(skipReason('completed', record({ commands: [] })), 'no_tool_use'); + assert.equal(skipReason('completed', record(), { reusedAsset: true }), 'asset_reused'); +}); + +test('the prompt is a digest of the turn, never its whole history, and asks for no answer values', () => { + const candidate = candidateOf({ prompt: 'add a retry to the uploader', record: record({ checks: [{ command: 'npm test', failed: false }] }), model: MODEL, turn: 1 }); + const prompt = distillPrompt(candidate, 'the backoff'); + assert.match(prompt, /## Task\nadd a retry to the uploader/); + assert.match(prompt, /- npm test/); + assert.match(prompt, /- passed: npm test/); + assert.match(prompt, /Focus on: the backoff/); + assert.match(prompt, /Do not include the answer itself/); +}); + +test('a list the model wrote as one numbered paragraph becomes steps', () => { + assert.deepEqual(argumentsOf(JSON.stringify(GENE)).strategy, ['Measure.', 'Back off.']); + assert.equal(argumentsOf('no json here'), null); + assert.equal(argumentsOf('{"title":"no summary"}'), null); +}); + +test('a candidate is distilled through the host model and sent to the Proxy with its own cost', async () => { + const sent = []; + const proxyFetch = async (method, path, body) => { sent.push({ path, body }); return { ok: true, data: { id: 'gene_x' } }; }; + const candidate = candidateOf({ prompt: 'add a retry', record: record(), model: MODEL, turn: 1 }); + const llm = fakeLlm(); + const outcome = await distillCandidate({ proxyFetch, llm, candidate, timeoutMs: 1_000 }); + + assert.equal(outcome.status, 'sent'); + assert.deepEqual(llm.calls[0].provider, 'gateway'); + assert.equal(sent[0].path, '/conversation/distill'); + assert.equal(sent[0].body.trigger, 'session_end'); + assert.equal(sent[0].body.persist, true); + assert.equal(sent[0].body.publish, false); + assert.deepEqual(sent[0].body.distill_usage, { inputTokens: 900, outputTokens: 120 }); + assert.equal((await distillCandidate({ proxyFetch, llm, candidate: { ...candidate, model: null }, timeoutMs: 1_000 })).status, 'no_model'); +}); + +test('a candidate dropped or cancelled while the model answers is never sent to the Proxy', async (t) => { + t.after(freshHome('evolver-distill-dropped-')); + const sent = []; + const proxyFetch = async (method, path) => { sent.push(path); return { ok: true }; }; + const candidate = candidateOf({ prompt: 'add a retry', record: record(), model: MODEL, turn: 1 }); + rememberCandidate('s', candidate); + assert.equal(isCurrentCandidate('s', candidate), true); + rememberCandidate('s', { ...candidate, at: candidate.at + 1 }); + assert.equal(isCurrentCandidate('s', candidate), false); + + const replaced = await distillCandidate({ proxyFetch, llm: fakeLlm(), candidate, timeoutMs: 1_000, isCurrent: () => isCurrentCandidate('s', candidate) }); + const controller = new AbortController(); + const llm = fakeLlm(); + llm.hold = new Promise((resolve) => setTimeout(resolve, 20)); + const cancelled = distillCandidate({ proxyFetch, llm, candidate, timeoutMs: 1_000, cancel: controller.signal }); + controller.abort(); + + assert.equal(replaced.status, 'dropped'); + assert.equal((await cancelled).status, 'dropped'); + assert.deepEqual(sent, []); +}); + +test("a session's candidate is due to it; another session's only once that session is over", (t) => { + t.after(freshHome('evolver-candidates-')); + rememberCandidate('live', { pid: 111, at: 1 }); + rememberCandidate('killed', { pid: 222, at: 2 }); + rememberCandidate('ended', { pid: 111, at: 3 }); + closeCandidate('ended'); + const isGone = (pid) => pid === 222; + assert.deepEqual(dueCandidates({ sessionId: 'live' }).map((due) => due.sessionId), ['live']); + assert.deepEqual(claimLeftoverCandidates({ isGone, pid: 7 }).map((due) => due.sessionId), ['killed', 'ended']); +}); + +test('a leftover candidate is claimed by one process only, and again once the claim dies', (t) => { + t.after(freshHome('evolver-claim-')); + const isGone = (pid) => pid !== 111; + rememberCandidate('killed', { pid: 222, at: 1 }); + + assert.deepEqual(claimLeftoverCandidates({ isGone, pid: 111 }).map((due) => due.sessionId), ['killed']); + assert.deepEqual(claimLeftoverCandidates({ isGone, pid: 333 }), [], 'a live claim is left alone'); + settleCandidate('killed', { at: 1 }, { sent: false }); + assert.deepEqual(claimLeftoverCandidates({ isGone, pid: 333 }).map((due) => due.candidate.attempts), [1], 'a failed try is claimable again'); +}); + +test('a candidate leaves the disk only once distilled, or after three failed tries', (t) => { + t.after(freshHome('evolver-settle-')); + const candidate = { pid: process.pid, at: 5, attempts: 0 }; + rememberCandidate('s', candidate); + settleCandidate('s', candidate, { sent: false }); + settleCandidate('s', candidate, { sent: false }); + assert.equal(dueCandidates({ sessionId: 's' })[0].candidate.attempts, 2, 'kept for a retry'); + settleCandidate('s', candidate, { sent: false }); + assert.deepEqual(dueCandidates({ sessionId: 's' }), [], 'given up'); + + rememberCandidate('s', candidate); + rememberCandidate('s', { ...candidate, at: 9 }); + settleCandidate('s', candidate, { sent: true }); + assert.equal(dueCandidates({ sessionId: 's' })[0].candidate.at, 9, 'a newer candidate written meanwhile stays'); +}); + +test('keys and tokens never reach the candidate, nor the prompt a manual focus writes', () => { + const secret = 'sk-evomap-0123456789abcdef0123456789abcdef'; + const candidate = candidateOf({ + prompt: `use ${secret}`, model: MODEL, turn: 1, + record: record({ commands: [`curl -H "Authorization: Bearer ${secret}" x`], answer: `done with ${secret}`, paths: [`keys/${secret}.pem`] }), + }); + assert.doesNotMatch(JSON.stringify(candidate), new RegExp(secret)); + assert.doesNotMatch(distillPrompt(candidate, `Authorization: Bearer ${secret}`), new RegExp(secret)); +}); + +function deadPid() { + const child = spawnSync(process.execPath, ['-e', 'process.stdout.write(String(process.pid))']); + return Number(child.stdout.toString()); +} + +test('an idle session keeps its open strategies: only a gone process counts as killed', async (t) => { + t.after(freshHome('evolver-idle-')); + const { takeOrphanedUnreported } = await import('../src/injected-assets.js'); + const { statePath, writeState, hashedName } = await import('../src/state-file.js'); + writeState(statePath(hashedName('injected', 'idle')), { 'sha256:a': { turn: 1, injectedAt: 0, strategyText: '', sessionId: 'idle', pid: process.pid } }); + writeState(statePath(hashedName('injected', 'old')), { 'sha256:b': { turn: 1, injectedAt: 0, strategyText: '', sessionId: 'old' } }); + assert.deepEqual(takeOrphanedUnreported(), []); + writeState(statePath(hashedName('injected', 'gone')), { 'sha256:c': { turn: 2, injectedAt: 0, strategyText: '', sessionId: 'gone', pid: deadPid() } }); + assert.deepEqual(takeOrphanedUnreported().map((asset) => asset.assetId), ['sha256:c']); + assert.deepEqual(takeOrphanedUnreported(), [], 'reported once'); +}); + +async function harness(t, prefix, answerOf) { + t.after(freshHome(`${prefix}-home-`)); + const proxy = await startProxy(answerOf); + t.after(() => proxy.close()); + const { ctx, listeners } = fakeContext(); + ctx.llm = fakeLlm(); + apply(ctx, Config({ projectDir: projectDirectory(`${prefix}-`), proxyPort: proxy.port })); + return { proxy, listeners, llm: ctx.llm }; +} + +const proxyAnswer = (assets = []) => (path) => (path === '/asset/fetch' ? { assets } : { recorded: true }); + +async function verifiedTurn(listeners, agent, turn = 1) { + await primedBy(listeners, agent, 'add a retry to the uploader', turn); + await listeners.get('tools/execute')({ name: 'bash', agent, arguments: { command: 'npm test' } }, async () => ({ isError: false, content: [{ type: 'text', text: 'ℹ pass 2\nℹ fail 0' }] })); + listeners.get('session/event')(agent.session, { type: 'assistant/message', data: { turn, step: 1, message: { role: 'assistant', content: [{ type: 'text', text: 'Retries added.' }] } } }); + listeners.get('session/event')(agent.session, { type: 'turn/end', data: { turn, reason: { kind: 'completed' } } }); +} + +test('a failed distillation keeps the candidate for the next try', async (t) => { + const { proxy, listeners, llm } = await harness(t, 'evolver-retry', proxyAnswer()); + const agent = { ...fakeAgent({ sessionId: 'session-retry' }).agent, session: session('session-retry') }; + llm.answer = 'not json'; + + await verifiedTurn(listeners, agent, 1); + await listeners.get('session/flush')(agent.session); + assert.equal(dueCandidates({ sessionId: 'session-retry' }).length, 1); + llm.answer = undefined; + await listeners.get('session/flush')(agent.session); + await untilRequest(proxy.requests, '/conversation/distill'); + assert.deepEqual(dueCandidates({ sessionId: 'session-retry' }), []); +}); + +test('a distillation that failed at the end of a session is claimed and tried again at the next session start', async (t) => { + const { proxy, listeners, llm } = await harness(t, 'evolver-reclaim', proxyAnswer()); + const agent = { ...fakeAgent({ sessionId: 'session-over' }).agent, session: session('session-over') }; + llm.answer = 'not json'; + + await verifiedTurn(listeners, agent, 1); + await listeners.get('session/flush')(agent.session); + listeners.get('session/disposed')(agent.session); + assert.equal(proxy.requests.filter((request) => request.path === '/conversation/distill').length, 0); + + llm.answer = undefined; + const next = { ...fakeAgent({ sessionId: 'session-after' }).agent, session: session('session-after') }; + listeners.get('agent/session-start')({ agent: next }); + await untilRequest(proxy.requests, '/conversation/distill'); + await untilSettled('session-over'); +}); + +test('a verified turn is distilled once, when the session ends, outside the conversation', async (t) => { + const { proxy, listeners, llm } = await harness(t, 'evolver-session-end', proxyAnswer()); + const agent = { ...fakeAgent({ sessionId: 'session-end' }).agent, session: session('session-end') }; + + await verifiedTurn(listeners, agent, 1); + assert.equal(llm.calls.length, 0, 'nothing runs at turn end'); + await listeners.get('session/flush')(agent.session); + await untilRequest(proxy.requests, '/conversation/distill'); + await listeners.get('session/flush')(agent.session); + + assert.equal(llm.calls.length, 1); + assert.equal(proxy.requests.filter((request) => request.path === '/conversation/distill').length, 1); +}); + +test('a turn the next prompt corrects is not distilled', async (t) => { + const { proxy, listeners, llm } = await harness(t, 'evolver-corrected', proxyAnswer()); + const agent = { ...fakeAgent({ sessionId: 'session-corrected' }).agent, session: session('session-corrected') }; + + await verifiedTurn(listeners, agent, 1); + await primedBy(listeners, agent, '不对，这样不行', 2); + await listeners.get('session/flush')(agent.session); + + assert.equal(llm.calls.length, 0); + assert.equal(proxy.requests.filter((request) => request.path === '/conversation/distill').length, 0); +}); + +test('a correction that arrives while the turn is being distilled stops that distillation', async (t) => { + const { proxy, listeners, llm } = await harness(t, 'evolver-corrected-late', proxyAnswer()); + const agent = { ...fakeAgent({ sessionId: 'session-late' }).agent, session: session('session-late') }; + let release; + llm.hold = new Promise((resolve) => { release = resolve; }); + + await verifiedTurn(listeners, agent, 1); + const flushed = listeners.get('session/flush')(agent.session); + await untilCount(llm.calls, () => true, 1); + await primedBy(listeners, agent, '不对，这样不行', 2); + release(); + + assert.equal((await flushed).status, 'dropped'); + assert.equal(proxy.requests.filter((request) => request.path === '/conversation/distill').length, 0); + assert.deepEqual(dueCandidates({ sessionId: 'session-late' }), []); +}); + +test('a strategy whose turn never ended is reported failed when the session goes away', async (t) => { + const { proxy, listeners } = await harness(t, 'evolver-disposed', proxyAnswer([{ asset_id: 'sha256:cut', strategy: ['a.', 'b.', 'c.', 'd.'] }])); + const agent = { ...fakeAgent({ sessionId: 'session-cut' }).agent, session: session('session-cut') }; + + await primedBy(listeners, agent, 'add a retry to the uploader', 1); + listeners.get('session/disposed')(agent.session); + await untilCount(proxy.requests, (request) => request.path === '/asset/reuse-result', 1); + + const [report] = proxy.requests.filter((request) => request.path === '/asset/reuse-result').map((request) => request.body); + assert.equal(report.asset_id, 'sha256:cut'); + assert.equal(report.outcome, 'failed'); + assert.equal(report.task_id, 'dsh:session-cut:1'); + assert.equal(report.evidence.aborted, true); +}); + +test("the next session reports a killed session's open strategies and distills its leftover candidate", async (t) => { + const { proxy, listeners, llm } = await harness(t, 'evolver-killed', proxyAnswer()); + const { statePath } = await import('../src/state-file.js'); + + const { writeState, hashedName } = await import('../src/state-file.js'); + const dead = deadPid(); + writeState(statePath(hashedName('injected', 'session-killed')), { + 'sha256:orphan': { turn: 4, injectedAt: Date.now(), strategyText: '1. a.', sessionId: 'session-killed', pid: dead }, + }); + rememberCandidate('session-killed', candidateOf({ prompt: 'p', record: record(), model: MODEL, turn: 4, pid: dead })); + + const agent = { ...fakeAgent({ sessionId: 'session-next' }).agent, session: session('session-next') }; + listeners.get('agent/session-start')({ agent }); + await untilRequest(proxy.requests, '/asset/reuse-result'); + await untilRequest(proxy.requests, '/conversation/distill'); + + const [report] = proxy.requests.filter((request) => request.path === '/asset/reuse-result').map((request) => request.body); + assert.equal(report.asset_id, 'sha256:orphan'); + assert.equal(report.task_id, 'dsh:session-killed:4'); + assert.equal(report.outcome, 'failed'); + assert.equal(llm.calls.length, 1); +}); diff --git a/test/skill.test.js b/test/skill.test.js index 7b29539..1d27d75 100644 --- a/test/skill.test.js +++ b/test/skill.test.js @@ -1,6 +1,9 @@ import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; import { test } from 'node:test'; +import { splitFrontmatter } from '../src/frontmatter.js'; + import { evolverSkillProvider } from '../src/skill.js'; test('skill loading observes caller cancellation', async () => { @@ -9,3 +12,10 @@ test('skill loading observes caller cancellation', async () => { /cancelled|aborted/i, ); }); + +test('the catalog summary is the description SKILL.md declares, and names no tool', async () => { + const markdown = readFileSync(new URL('../assets/skills/capability-evolver/SKILL.md', import.meta.url), 'utf8'); + const [summary] = await evolverSkillProvider.list(); + assert.equal(summary.description, splitFrontmatter(markdown).fields.description); + assert.doesNotMatch(markdown, /evolver_[a-z_]+/); +}); diff --git a/test/state-file.test.js b/test/state-file.test.js new file mode 100644 index 0000000..b0094ff --- /dev/null +++ b/test/state-file.test.js @@ -0,0 +1,62 @@ +import assert from 'node:assert/strict'; +import { spawn, spawnSync } from 'node:child_process'; +import fs from 'node:fs'; +import { test } from 'node:test'; + +import { readState, statePath, updateState } from '../src/state-file.js'; +import { freshHome } from './support/host.js'; + +const MODULE = new URL('../src/state-file.js', import.meta.url).href; +const WRITERS = 4; +const PER_WRITER = 15; + +const CHILD = ` + const { statePath, updateState } = await import(process.argv[1]); + for (const key of process.argv.slice(2)) { + updateState(statePath('shared.json'), {}, (state) => ({ ...state, [key]: true })); + } +`; + +function writer(keys) { + return new Promise((resolve) => { + const child = spawn(process.execPath, ['--input-type=module', '-e', CHILD, MODULE, ...keys], { stdio: 'ignore' }); + child.on('exit', resolve); + }); +} + +test('an update is read, changed and written under a lock, so parallel writers keep every entry', async (t) => { + t.after(freshHome('evolver-state-lock-')); + const keysOf = (writerIndex) => Array.from({ length: PER_WRITER }, (unused, index) => `w${writerIndex}-${index}`); + + await Promise.all(Array.from({ length: WRITERS }, (unused, index) => writer(keysOf(index)))); + + const written = readState(statePath('shared.json'), {}); + assert.equal(Object.keys(written).length, WRITERS * PER_WRITER); +}); + +test('an update that changes nothing leaves the file alone and releases its lock', (t) => { + t.after(freshHome('evolver-state-noop-')); + updateState(statePath('kept.json'), {}, () => ({ a: 1 })); + updateState(statePath('kept.json'), {}, () => undefined); + assert.deepEqual(readState(statePath('kept.json'), {}), { a: 1 }); + assert.deepEqual(updateState(statePath('kept.json'), {}, (state) => ({ ...state, b: 2 })), { a: 1, b: 2 }); +}); + +test("a dead owner's lock is taken over, and a lock someone else holds by the time of release is left to them", (t) => { + t.after(freshHome('evolver-state-stale-')); + const target = statePath('stale.json'); + const lockPath = `${target}.lock`; + const deadPid = spawnSync(process.execPath, ['-e', '']).pid; + fs.mkdirSync(statePath(''), { recursive: true }); + fs.writeFileSync(lockPath, `${deadPid}:dead`); + + updateState(target, {}, () => ({ a: 1 })); + assert.equal(fs.existsSync(lockPath), false, "the dead owner's lock is gone after the update"); + + updateState(target, {}, (state) => { + fs.writeFileSync(lockPath, `${process.pid}:someone-else`); + return { ...state, b: 2 }; + }); + assert.equal(fs.readFileSync(lockPath, 'utf8'), `${process.pid}:someone-else`); + assert.deepEqual(readState(target, {}), { a: 1, b: 2 }); +});