Evolve Agent v0.3: evaluation-driven evolution

[DCLXAI]

Summary

Adds Evolve Agent v0.1 → v0.3 as an isolated TypeScript package under evolve-agent/, without changing the existing AGENTR application runtime.

The project uses GPT-5.6 Sol through the OpenAI Responses API and targets a narrower architectural axis than OpenClaw or Hermes Agent: verifiable execution plus measured, signed, reversible adaptation.

This remains a Draft PR. It does not claim ecosystem, integration, adoption, reliability-history, or production-maturity parity with OpenClaw or Hermes.

v0.1 — Evidence-gated kernel

• bounded autonomous task loop with turn, tool, token, and wall-time budgets
• durable checkpoints and resumable provider/network interruptions
• separate final-answer verifier pass
• current-Episode evidence contract and rejection of fabricated evidence IDs
• content-addressed artifacts and append-only SHA-256 hash-chained Episode ledger
• exact expiring HMAC capabilities bound to normalized tool arguments
• explicit approval for protected actions
• workspace traversal and symlink-escape protection
• evidence-aware memory
• candidate → evaluation → canary → explicit promotion → rollback Skill lifecycle
• no automatic Skill promotion

v0.2 — Hardened execution

• executor interface and registry
• Docker-first, fail-closed process execution
• local host execution disabled by default
• exact repository@sha256:<digest> image allowlist and --pull never
• network=none by default; unsafe built-in network modes rejected
• read-only root filesystem and read-only workspace by default
• numeric non-root UID:GID, cap-drop ALL, and no-new-privileges
• memory, swap, CPU, PID, tmpfs, file-descriptor, timeout, and output ceilings
• short-lived file Secret broker with allowlisted names and output redaction
• execution receipts with command, policy, sandbox, and result hashes
• Episode lease, heartbeat, duplicate-run rejection, and stale-lock recovery
• authority state required outside the mounted task workspace

v0.3 — Evaluation-Driven Evolution

v0.3 replaces manually entered Skill scores with a real baseline-versus-candidate evaluation pipeline.

A Skill can no longer be promoted because it appears plausible, worked once, or was evaluated on the same Episodes that created it. It must pass independent replay fixtures, survive non-intervening shadow canaries, and remain within a signed production envelope after promotion.

Replay fixtures and leakage control

• content-addressed replay fixtures captured only from clean committed Episodes
• deterministic train/validation/holdout split support
• supporting/training Episodes excluded from evaluation fixtures
• training provenance freezes when evaluation begins; later Episodes remain eligible as independent holdout material
• fixture integrity hash and deterministic fixture ID
• source Episode, task, budgets, model-proposed tool trace, evidence records, artifact hashes, baseline score, usage, duration, and active Skill IDs retained
• rejected arguments, policy denial, approval denial, terminal failure, incomplete trace, missing artifacts, and provenance mismatch fail closed

Paired baseline-versus-candidate execution

• baseline and candidate receive the same task, tools, observations, budgets, evidence world, and verifier
• candidate differs only by inclusion of the candidate Skill
• execution order alternates to reduce ordering bias
• exact model-proposal tool sequence and raw-argument hash matching
• executed-argument evidence provenance verification
• early final answers, extra tools, trace changes, fabricated evidence, and citation-contract violations fail closed

Measured promotion gates

The evaluation engine measures:

• success rate
• independent verifier score
• input, output, and total tokens
• turns and tool calls
• duration
• trace integrity
• safety violations
• candidate wins, losses, ties, and new failures
• deterministic paired-bootstrap confidence intervals

Default offline gates require:

• at least three unique independent matching fixtures
• complete baseline/candidate pairs
• zero new failures
• no success-rate regression
• verifier-score regression no worse than the configured tolerance
• lower confidence bounds inside non-regression limits
• token and tool-call usage inside configured budgets
• no trace or safety regression
• measurable quality or efficiency improvement

Failed offline or canary evaluation moves the Skill to quarantined.

Signed evaluation authority

• Ed25519-signed offline, canary, and production-monitor reports
• private evaluation key stored with mode 0600
• signed payload hash, report ID, and public-key fingerprint
• report tampering and authority mismatch rejection
• promotion authorization binds the Skill ID, Skill fingerprint, offline report, canary report, policy hash, and signing-key fingerprint
• SkillStore re-verifies signed reports itself, preventing callers from bypassing gates through the library API

Shadow canary and automatic rollback

• candidate Skills replay production Episodes counterfactually in shadow
• candidate output is measured and discarded
• the answer already delivered to the user is never replaced by shadow evaluation
• passing signed offline and canary reports are both required for explicit promotion
• promoted Skills record attributed production outcomes
• a rolling monitor compares live quality and cost with the signed canary envelope
• success, verifier-score, or token regression triggers a signed monitor report and automatic rollback

CLI additions

evolve-agent evaluations fixtures capture <episode-id>
evolve-agent evaluations fixtures import <path>
evolve-agent evaluations fixtures list
evolve-agent evaluations run <skill-id>
evolve-agent evaluations shadow <skill-id> <episode-id>
evolve-agent evaluations canary <skill-id>
evolve-agent evaluations monitor <skill-id>
evolve-agent evaluations reports list
evolve-agent evaluations reports show <report-id>
evolve-agent evaluations verify <report-id>

The existing hardened-execution and Skill lifecycle commands remain available.

Validation

The reviewed source snapshot passed three transport-integrity layers before the clean commit was created:

• each of nine source fragments matched its expected SHA-256
• the reconstructed base64 stream matched SHA-256 f5b12ba25b320ae4c1c797865bb6b64bdd6acc1773cfa071908833144df80e41
• the decoded tar.xz archive matched SHA-256 87423bcd1b4bc0c324cb9372a607f23122e55a5a9c06306cc1745883cf95b767

Package checks:

• npm run typecheck — passed
• npm test — 38/38 passed
• npm run build — passed
• npm pack --dry-run — passed
• CLI help/version contract — passed
• CLI doctor fail-closed path — passed

GitHub Actions:

• evolve-agent-ci on Node.js 22 — passed
• evolve-agent-ci on Node.js 24 — passed
• existing AGENTR compile check — passed
• existing AGENTR test suite — passed
• existing AGENTR API Docker image build — passed
• existing AGENTR Console Docker image build — passed

The 38-test suite covers v0.1/v0.2 invariants plus fixture capture and tamper detection, supporting-Episode leakage exclusion, paired trace replay, trace mismatch rejection, evaluation metrics and bootstrap gates, new-failure rejection, Ed25519 signing and tamper rejection, direct-store promotion authority, shadow non-intervention, production regression rollback, runtime active-Skill attribution, optional fixture capture, and frozen training provenance.

Important limitations

• No live GPT-5.6 Sol request was made in the build environment; the Responses API adapter and evaluation paths use deterministic providers for testability.
• No live Evolve Docker executor run was made against a real Docker daemon in this build; Docker policy and runtime integration remain covered through deterministic fake runners.
• The local Ed25519 file key is not hardware-backed attestation and has no remote quorum or rotation protocol yet.
• Replay fixtures control the model/tool evidence world but do not reproduce arbitrary live external side effects.
• v0.3 does not establish causal proof that a Skill alone produced an improvement.
• Large-sample sequential tests, evaluator calibration, and multiple-hypothesis correction are not implemented yet.
• Skill promotion remains intentionally explicit; only regression rollback is automatic.
• Distributed multi-agent lease coordination and ACP/EDL quorum evidence receipts remain future work.

Next defensible milestones

1. live GPT-5.6 Sol repeated-run variance matrix
2. rootless Docker and Docker Desktop qualification
3. adversarial and mutation-generated replay fixtures
4. evaluator calibration, sequential tests, and multiple-comparison control
5. signed executor-policy and image provenance
6. remote or hardware-backed evaluation signing
7. lease-based multi-agent work graph
8. ACP/EDL Episode binding and quorum evidence receipts

Scope and repository safety

• all new runtime code remains under evolve-agent/
• existing AGENTR application code is unchanged
• final v0.3 is one clean commit on top of the clean v0.2 commit
• PR remains Draft and unmerged
• package version: 0.3.0
• license: MIT