From 871973807be717dd45f9ff63d594f988dc584585 Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Sun, 13 Sep 2026 01:33:34 +0300 Subject: [PATCH 01/22] feat(memory): add V1 migration audit and deprecation guidance --- cli/knowledge.md | 6 +- .../commands/__tests__/memory-command.test.ts | 143 ++++++- cli/src/commands/memory-command.ts | 119 +++++- cli/src/data/__tests__/slash-commands.test.ts | 2 +- cli/src/data/slash-commands.ts | 2 +- common/knowledge.md | 4 +- common/src/types/session-state.ts | 6 +- common/src/types/task-memory.ts | 20 + sdk/CHANGELOG.md | 3 + sdk/src/__tests__/memory-v2-contract.test.ts | 21 + sdk/src/index.ts | 4 + .../memory-v2/__tests__/v1-migration.test.ts | 374 +++++++++++++++++- sdk/src/services/memory-v2/v1-migration.ts | 367 ++++++++++++++++- sdk/src/services/task-memory-store.ts | 17 + 14 files changed, 1069 insertions(+), 19 deletions(-) diff --git a/cli/knowledge.md b/cli/knowledge.md index ca1c6ef006..8f86aba071 100644 --- a/cli/knowledge.md +++ b/cli/knowledge.md @@ -16,14 +16,14 @@ - The SQLite repository projects the latest `coverage.recorded` event per `(taskId, dimension)` into the `currentCoverage` retrieval category, filtered by exact workspace revision/snapshot freshness and optional `taskId` narrowing. - The `query_index` tool handler (`packages/agent-runtime/src/tools/handlers/tool/query-index.ts`) records its results into `agentState.discoveryCoverage` via `recordDiscoveryResult` with a bounded (4000-char) question string; recording is wrapped in try/catch so a coverage failure can never break the tool call. - When testing tool handlers that import a function to spy on, prefer `spyOn(namespaceImport, 'fn')` over `mock.module()` (repo convention in `docs/testing.md`/`CONTRIBUTING.md`); a relative `mock.module` specifier resolves from the test file's directory and silently misses the module the subject under test imports when the two live at different depths. -- Memory V2 authority is selected by `OPENBUFF_MEMORY_AUTHORITY` via `getMemoryAuthoritySelection` (`cli/src/utils/env.ts`); the default is `sqlite-v2-opt-in` (fail-closed — no V1 fallback), with `json-v1` as an explicit opt-out escape hatch and `shadow-v2` as an explicit shadow mode. The provider (`cli/src/services/memory-v2/provider.ts`) keys its shadow-vs-fail-closed behavior off `authority.effective`, so flipping the default flips which degradation path default runs take. +- Memory V2 authority is selected by `OPENBUFF_MEMORY_AUTHORITY` via `getMemoryAuthoritySelection` (`cli/src/utils/env.ts`); the default is `sqlite-v2-opt-in` (fail-closed — no V1 fallback), with `json-v1` as an explicit opt-out escape hatch and `shadow-v2` as an explicit shadow mode. Release N keeps `json-v1` and `shadow-v2` supported but deprecates both compatibility modes; `sqlite-v2-opt-in` is the default and replacement. Authority selection, degradation, and fallback behavior are unchanged in this release, and no removal date/version is implied. ## Slash Commands and Plan Mode - Durable planning is entered through `mode:plan`; the standalone `/plan` command is intentionally absent from `COMMAND_REGISTRY` and `SLASH_COMMANDS` so there is one plan-entry path. - Keep the durable-plan quartet registered: `/resume-plan` (`rp`), `/update-plan` (`up`), `/plan-status` (`ps`), and `/lessons` (`lesson`). These commands operate on `.agents/sessions//` artifacts and fall back to the plan-session picker when no target is provided. - `/plans` (`plan-ls`) lists the artifact-bearing sessions returned by `listPlanSessions()` and never prompts the agent, while `/plan-use` (`plan-active`, `use-plan`) writes the `ACTIVE_SESSION` pointer file under `.agents/` and only accepts a bare slug or `.agents/sessions/` — the resolved directory must be exactly one segment under `.agents/sessions/` because the pointer stores bare slugs, so nested paths, non-session paths, missing directories, and artifact-less directories are all rejected before any write. Both share `formatPlanSessionListRow` / `PLAN_SESSIONS_DIR_PREFIX` from `cli/src/commands/plan-artifacts.ts` so the rendered box and the text report cannot drift. -- `/memory` (alias `/mem`, `cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts`) inspects the persisted cross-session task memory for the current project (`.openbuff/memory/task-memory.json`). `/memory status` (the default) reports the record's revision and age, its goal and per-list counts, and how much of its evidence still verifies against disk, listing up to five stale paths; `/memory prune` drops evidence that no longer verifies. Both subcommands are move-aware via `collectWorkspaceMoves` + `WorkspaceJournalService` so a renamed file's evidence rebinds rather than being reported stale and deleted. +- `/memory` (alias `/mem`, `cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts`) inspects the persisted cross-session task memory for the current project (`.openbuff/memory/task-memory.json`). `/memory status` (the default) reports the record's revision and age, its goal and per-list counts, and how much of its evidence still verifies against disk, listing up to five stale paths; `/memory prune` drops evidence that no longer verifies. Both subcommands are move-aware via `collectWorkspaceMoves` + `WorkspaceJournalService` so a renamed file's evidence rebinds rather than being reported stale and deleted. `/memory audit-migration` is dedicated and read-only: it passes the currently loaded V1 record and leased V2 repository to the SDK `auditTaskMemoryV1Migration` API, never uses the CLI canonical-inventory marker scan, and never appends. An `exact` audit verifies source revision/checksum, marker, imported body, and provenance, but is described as lossless only when omissions, truncations, and warnings are all zero; all other audit outcomes are rendered distinctly with bounded reason text. - Slash-command descriptions should stay model-agnostic under BYOK/local mode. Use wording such as "configured reviewer" rather than naming hosted models. ## Import Guidelines @@ -919,7 +919,7 @@ Streaming markdown renders as plain text until the message or agent finishes. Th - _Knowledge refresh 2026-08-23: add `/memory` (alias `/mem`) slash command; staleness guard touch._ -- _Knowledge refresh 2026-09-12: Memory V2 implementation across `cli/src/services/memory-v2/` (SQLite kernel with WAL/append-only triggers/canonical projections/project binding, contained file I/O with O_NOFOLLOW/proc-fd anchoring, lease-based provider with opt-in fail-closed), `/memory` command expanded to 18 subcommands (authority/diagnose/query/inspect/consolidate/repair/revalidate/correct/forget/pin/export/import plus existing status/prune) with sanitized error boundaries and preview-by-default mutators, `cli/src/utils/codebuff-client.ts` gained `ManagedOpenbuffClient` with lease-based lifecycle and `memoryV2ClientConfigFromProvider` wiring, `cli/src/utils/env.ts` gained `getMemoryAuthoritySelection` for the `OPENBUFF_MEMORY_AUTHORITY` env var, `cli/src/types/chat.ts` gained `MemoryReportContentBlock` and `v2Lines` additive fields, and `cli/src/components/renderers/memory-box.tsx` gained the `report` state renderer with insert-command buttons._ +- _Knowledge refresh 2026-09-12: Memory V2 implementation across `cli/src/services/memory-v2/` (SQLite kernel with WAL/append-only triggers/canonical projections/project binding, contained file I/O with O_NOFOLLOW/proc-fd anchoring, lease-based provider with opt-in fail-closed), `/memory` expanded with authority/diagnose/query/inspect/consolidate/repair/revalidate/correct/forget/pin/export/import and the read-only SDK-backed `audit-migration` command plus existing status/prune, all with sanitized error boundaries and preview-by-default mutators; `cli/src/utils/codebuff-client.ts` gained `ManagedOpenbuffClient` with lease-based lifecycle and `memoryV2ClientConfigFromProvider` wiring, `cli/src/utils/env.ts` gained `getMemoryAuthoritySelection` for the `OPENBUFF_MEMORY_AUTHORITY` env var, `cli/src/types/chat.ts` gained `MemoryReportContentBlock` and `v2Lines` additive fields, and `cli/src/components/renderers/memory-box.tsx` gained the `report` state renderer with insert-command buttons._ - _Knowledge refresh 2026-08-31: live compaction status rendering (`context_compaction_status` consumption, run-correlated pending/settled pairing, replayed-pending-as-interrupted) in `cli/src/utils/sdk-event-handlers.ts` and `cli/src/components/renderers/compaction-box.tsx`._ diff --git a/cli/src/commands/__tests__/memory-command.test.ts b/cli/src/commands/__tests__/memory-command.test.ts index 06d7562133..f422dfccfc 100644 --- a/cli/src/commands/__tests__/memory-command.test.ts +++ b/cli/src/commands/__tests__/memory-command.test.ts @@ -333,10 +333,10 @@ describe('/memory command', () => { const { deps, calls } = createDeps({ memory: makeMemory() }) expect(await handleMemoryCommand('wat', deps)).toContain( - 'Usage: /memory [status|authority|diagnose|query ', + 'Usage: /memory [status|authority|diagnose|audit-migration|query ', ) expect(await handleMemoryCommand('PRUNE-ish', deps)).toContain( - 'Usage: /memory [status|authority|diagnose|query ', + 'Usage: /memory [status|authority|diagnose|audit-migration|query ', ) expect(calls.prune).toBe(0) }) @@ -350,6 +350,9 @@ describe('/memory command', () => { const result = await handleMemoryCommand('authority', deps) expect(result).toContain('json-v1, shadow-v2, sqlite-v2-opt-in') expect(result).toContain('Requested: sqlite-v2-opt-in; effective: json-v1') + expect(result).toContain('Release N') + expect(result).toContain('json-v1 and shadow-v2 remain supported but are deprecated') + expect(result).toContain('sqlite-v2-opt-in is the default and replacement') expect(result).toContain('reset/restart') }) @@ -486,6 +489,119 @@ describe('/memory blocks', () => { } }) + test('audit-migration renders an exact lossless SDK outcome with success tone', async () => { + const memory = makeMemory() + const { deps, calls } = createDeps({ memory }) + const repository = { append: async () => { throw new Error('must not append') } } + let request: unknown + let releases = 0 + deps.auditTaskMemoryV1Migration = async (value) => { + request = value + return { + outcome: 'exact', revision: 3, checksum: 'full-source-checksum', identity: 'v1:3:id', + markerEventId: 'marker', repositoryLastEventId: 'tail', importedTaskId: 'task', + importedObservationIds: ['observation-1', 'observation-2'], omittedFields: 0, warnings: [], + } as unknown as Awaited> + } + deps.getMemoryV2 = async () => ({ + status: 'available', requestedAuthority: 'sqlite-v2-opt-in', effectiveAuthority: 'sqlite-v2-opt-in', + projectId: 'project-1', operator: {}, repository, release: async () => { releases++ }, + }) as unknown as Awaited>> + + const block = await handleMemoryCommandBlocks('audit-migration', deps) + + if (block.state !== 'report') throw new Error('expected report') + expect(block.tone).toBe('success') + expect(block.lines.join('\n')).toContain('Outcome: exact') + expect(block.lines.join('\n')).toContain('Source revision: 3') + expect(block.lines.join('\n')).toContain('Source checksum: full-source-checksum') + expect(block.lines.join('\n')).toContain('Marker verification: exact; imported body/provenance verification: exact') + expect(block.lines.join('\n')).toContain('Imported observations: 2') + expect(block.lines.join('\n')).toContain('Omitted fields: 0') + expect(block.lines.join('\n')).toContain('Truncated fields: 0') + expect(block.lines.join('\n')).toContain('Warnings: 0 (none)') + expect(block.lines.join('\n')).toContain('Lossless migration evidence: yes') + expect(block.lines.join('\n')).toContain('No writes were performed') + expect(request).toEqual({ memory, projectId: 'project-1', repository }) + expect(calls.prune).toBe(0) + expect(releases).toBe(1) + }) + + test('audit-migration warns when an exact outcome documents loss', async () => { + const { deps } = createDeps({ memory: makeMemory() }) + deps.auditTaskMemoryV1Migration = async () => ({ + outcome: 'exact', revision: 3, checksum: 'checksum', identity: 'v1:3:id', + markerEventId: 'marker', repositoryLastEventId: 'tail', importedTaskId: 'task', + importedObservationIds: [], omittedFields: 2, truncatedFields: 1, + warnings: ['text-truncated', 'goal-excluded'], + }) as unknown as Awaited> + deps.getMemoryV2 = async () => ({ + status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', + projectId: 'project-1', operator: {}, repository: {}, release: async () => {}, + }) as unknown as Awaited>> + + const block = await handleMemoryCommandBlocks('audit-migration', deps) + + if (block.state !== 'report') throw new Error('expected report') + expect(block.tone).toBe('warning') + expect(block.lines.join('\n')).toContain('Omitted fields: 2') + expect(block.lines.join('\n')).toContain('Truncated fields: 1') + expect(block.lines.join('\n')).toContain('Warnings: 2 (text-truncated, goal-excluded)') + expect(block.lines.join('\n')).toContain('Lossless migration evidence: no') + }) + + test('audit-migration renders every non-exact outcome distinctly and sanitizes failures', async () => { + const { deps } = createDeps({ memory: makeMemory() }) + const outcomes = [ + { outcome: 'no-record' }, + { outcome: 'not-migrated', revision: 3, checksum: 'checksum' }, + { outcome: 'incomplete', reason: 'reservation-only', revision: 3, checksum: 'checksum' }, + { outcome: 'mismatch', reason: 'imported-body-mismatch', revision: 3, checksum: 'checksum' }, + { outcome: 'rejected', reason: 'repository-rejected' }, + { outcome: 'failed', reason: 'repository-failed' }, + ] as const + let index = 0 + let releases = 0 + deps.auditTaskMemoryV1Migration = async () => outcomes[index++] as unknown as Awaited> + deps.getMemoryV2 = async () => ({ + status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', + projectId: 'project-1', operator: {}, repository: {}, release: async () => { releases++ }, + }) as unknown as Awaited>> + + const rendered = [] + for (let i = 0; i < outcomes.length; i++) { + const block = await handleMemoryCommandBlocks('audit-migration', deps) + if (block.state !== 'report') throw new Error('expected report') + rendered.push(block.lines.join('\n')) + } + expect(rendered[0]).toContain('Outcome: no-record') + expect(rendered[0]).toContain('No currently loaded V1') + expect(rendered[1]).toContain('Outcome: not-migrated') + expect(rendered[2]).toContain('Outcome: incomplete') + expect(rendered[3]).toContain('Outcome: mismatch') + expect(rendered[4]).toContain('Outcome: rejected') + expect(rendered[5]).toContain('Outcome: failed') + expect(rendered.join('\n')).not.toMatch(/\.sqlite|SELECT|\/secret/i) + expect(releases).toBe(outcomes.length) + }) + + test('audit-migration releases the provider lease once when the injected audit throws', async () => { + const { deps } = createDeps({ memory: makeMemory() }) + let releases = 0 + deps.auditTaskMemoryV1Migration = async () => { throw new Error('/secret/store.sqlite SELECT token') } + deps.getMemoryV2 = async () => ({ + status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', + projectId: 'project-1', operator: {}, repository: {}, release: async () => { releases++ }, + }) as unknown as Awaited>> + + const block = await handleMemoryCommandBlocks('audit-migration', deps) + + expect(releases).toBe(1) + if (block.state !== 'report') throw new Error('expected report') + expect(block.tone).toBe('error') + expect(block.lines.join('\n')).not.toMatch(/secret|sqlite|SELECT|token/i) + }) + test('query and diagnose render bounded generic reports', async () => { const { deps } = createDeps({ memory: makeMemory() }) deps.getMemoryV2 = async () => ({ @@ -504,7 +620,11 @@ describe('/memory blocks', () => { expect(query.state).toBe('report') expect(diagnose.state).toBe('report') if (query.state === 'report') expect(query.lines[0]).toContain('Tasks: 0') - if (diagnose.state === 'report') expect(diagnose.lines.join('\n')).toContain('Kernel: healthy') + if (diagnose.state === 'report') { + expect(diagnose.lines.join('\n')).toContain('Kernel: healthy') + expect(diagnose.lines.join('\n')).toContain('Release N') + expect(diagnose.lines.join('\n')).toContain('json-v1 and shadow-v2 remain supported but are deprecated') + } }) test('inspect and diagnose use project-scoped canonical export inventories', async () => { @@ -985,11 +1105,26 @@ describe('/memory blocks', () => { expect(block.type).toBe('memory') expect(block.state).toBe('report') - if (block.state === 'report') expect(block.lines.join('\n')).toContain('Usage: /memory [status|authority|diagnose|query ') + if (block.state === 'report') expect(block.lines.join('\n')).toContain('Usage: /memory [status|authority|diagnose|audit-migration|query ') expect(calls.prune).toBe(0) expect(calls.moves).toBe(0) }) + test('V2 status lines include Release N authority deprecation guidance', async () => { + const memory = makeMemory() + const { deps } = createDeps({ memory, reconciled: memory }) + deps.getMemoryV2 = async () => ({ + status: 'unavailable', requestedAuthority: 'shadow-v2', effectiveAuthority: 'json-v1', + degradation: 'V1 JSON memory is authoritative; SQLite was not opened.', retryable: false, + }) + + const output = await handleMemoryCommand('status', deps) + + expect(output).toContain('Release N') + expect(output).toContain('json-v1 and shadow-v2 remain supported but are deprecated') + expect(output).toContain('sqlite-v2-opt-in is the default and replacement') + }) + test('status forwards journal moves so evidence rebinds', async () => { const memory = makeMemory({ evidence: [makeEvidence({ id: 'ev-moved', path: 'old.ts' })], diff --git a/cli/src/commands/memory-command.ts b/cli/src/commands/memory-command.ts index 8fe693922d..b7e8ac93f3 100644 --- a/cli/src/commands/memory-command.ts +++ b/cli/src/commands/memory-command.ts @@ -1,10 +1,11 @@ /** - * `/memory` — inspect and prune persisted cross-session task memory for the - * current project root. Read-only by default; `prune` drops stale evidence. + * `/memory` — inspect, audit, and prune persisted cross-session task memory for + * the current project root. Read-only by default; `prune` drops stale evidence. */ import { createHash } from 'node:crypto' import { + auditTaskMemoryV1Migration, collectWorkspaceMoves, getHarnessStateDir, loadPersistedTaskMemory, @@ -27,13 +28,18 @@ import { import { getProjectMemoryV2Provider } from '../services/memory-v2/provider' import { formatAge, pluralizeEntries } from '../utils/format-helpers' -import type { TaskMemoryPruneOutcome, WorkspaceMoveRecord } from '@openbuff/sdk' +import type { + TaskMemoryPruneOutcome, + V1MigrationAuditOutcome, + WorkspaceMoveRecord, +} from '@openbuff/sdk' export type MemoryCommandDeps = { getRootDir: () => string loadPersistedTaskMemory: typeof loadPersistedTaskMemory reconcileTaskMemoryEvidence: typeof reconcileTaskMemoryEvidence pruneStaleTaskMemoryEvidence: typeof pruneStaleTaskMemoryEvidence + auditTaskMemoryV1Migration: typeof auditTaskMemoryV1Migration /** * Journal-recorded file moves for this project. Both subcommands reconcile * evidence, and reconciliation without moves reports a renamed file's @@ -70,6 +76,7 @@ const defaultDeps: MemoryCommandDeps = { loadPersistedTaskMemory, reconcileTaskMemoryEvidence, pruneStaleTaskMemoryEvidence, + auditTaskMemoryV1Migration, getWorkspaceMoves: loadWorkspaceMoves, getMemoryV2: getProjectMemoryV2Provider, } @@ -196,7 +203,8 @@ function memoryBlockToString( } } -const MEMORY_USAGE = 'Usage: /memory [status|authority|diagnose|query |inspect [eventId]|consolidate [--apply] [--task ]|repair [--apply]|revalidate [--apply]|correct [--apply]|forget [--apply]|pin [--apply]|export [--format json|markdown] [--include-stale]|import [--apply]|prune]' +const MEMORY_USAGE = 'Usage: /memory [status|authority|diagnose|audit-migration|query |inspect [eventId]|consolidate [--apply] [--task ]|repair [--apply]|revalidate [--apply]|correct [--apply]|forget [--apply]|pin [--apply]|export [--format json|markdown] [--include-stale]|import [--apply]|prune]' +const RELEASE_N_AUTHORITY_WARNING = 'Release N: json-v1 and shadow-v2 remain supported but are deprecated; sqlite-v2-opt-in is the default and replacement.' const CLI_SESSION_ID = 'memory-cli' const EXPORT_MAX_BYTES = 8 * 1024 * 1024 @@ -242,6 +250,93 @@ function deterministicId(prefix: string, value: string): string { return `${prefix}:${createHash('sha256').update(value).digest('hex').slice(0, 24)}` } +const AUDIT_INCOMPLETE_REASONS: Record< + Extract['reason'], + string +> = { + 'reservation-only': 'Only the migration reservation was found.', + 'legacy-marker-unverifiable': 'The legacy migration marker cannot verify this source record.', + 'missing-imported-task': 'The imported task body is missing.', + 'missing-imported-observations': 'One or more imported observations are missing.', + 'imported-body-mismatch': 'The imported body or provenance does not match the source marker.', + 'revision-conflict': 'Conflicting migration evidence exists for this source revision.', + 'checksum-conflict': 'Conflicting migration evidence exists for this source checksum.', +} + +const AUDIT_FAILURE_REASONS: Record< + Extract['reason'], + string +> = { + 'checksum-mismatch': 'The V1 source record failed checksum validation.', + 'repository-rejected': 'The audit read was rejected safely.', + 'repository-failed': 'The audit read could not be completed safely.', + 'invalid-export': 'The audit could not validate the exported migration evidence.', + 'wrong-project': 'The exported migration evidence belongs to another project.', + 'pagination-invalid': 'The exported migration evidence did not form a valid bounded sequence.', + 'page-limit-exceeded': 'The migration evidence exceeded the bounded audit scan.', +} + +function renderMigrationAudit( + outcome: V1MigrationAuditOutcome, +): import('../types/chat').MemoryContentBlock { + if (outcome.outcome === 'no-record') { + return report('Memory V1 migration audit', [ + 'Outcome: no-record.', + 'No currently loaded V1 task-memory record is available to audit.', + 'No writes were performed.', + ], 'secondary') + } + if (outcome.outcome === 'exact') { + const truncatedFields = outcome.truncatedFields ?? 0 + const warningCount = outcome.warnings.length + const lossless = outcome.omittedFields === 0 && truncatedFields === 0 && warningCount === 0 + return report('Memory V1 migration audit', [ + 'Outcome: exact.', + `Source revision: ${outcome.revision}.`, + `Source checksum: ${outcome.checksum}.`, + 'Marker verification: exact; imported body/provenance verification: exact.', + `Imported observations: ${outcome.importedObservationIds.length}.`, + `Omitted fields: ${outcome.omittedFields}.`, + `Truncated fields: ${truncatedFields}.`, + `Warnings: ${warningCount}${warningCount ? ` (${outcome.warnings.slice(0, 100).join(', ')})` : ' (none)'}.`, + `Lossless migration evidence: ${lossless ? 'yes' : 'no'}.`, + 'No writes were performed.', + ], lossless ? 'success' : 'warning') + } + if (outcome.outcome === 'not-migrated') { + return report('Memory V1 migration audit', [ + 'Outcome: not-migrated.', + `Source revision: ${outcome.revision}.`, + `Source checksum: ${outcome.checksum}.`, + 'No matching migration marker was found in the bounded V2 audit.', + 'No writes were performed.', + ], 'warning') + } + if (outcome.outcome === 'incomplete' || outcome.outcome === 'mismatch') { + return report('Memory V1 migration audit', [ + `Outcome: ${outcome.outcome}.`, + `Source revision: ${outcome.revision}.`, + `Source checksum: ${outcome.checksum}.`, + AUDIT_INCOMPLETE_REASONS[outcome.reason], + 'No writes were performed.', + ], outcome.outcome === 'mismatch' ? 'error' : 'warning') + } + if (outcome.outcome === 'rejected' || outcome.outcome === 'failed') { + return report('Memory V1 migration audit', [ + `Outcome: ${outcome.outcome}.`, + AUDIT_FAILURE_REASONS[outcome.reason], + ...(outcome.revision === undefined ? [] : [`Source revision: ${outcome.revision}.`]), + ...(outcome.checksum === undefined ? [] : [`Source checksum: ${outcome.checksum}.`]), + 'No writes were performed.', + ], outcome.outcome === 'failed' ? 'error' : 'warning') + } + return report( + 'Memory V1 migration audit', + ['The audit returned an unsupported outcome.', 'No writes were performed.'], + 'error', + ) +} + async function getV2(deps: MemoryCommandDeps) { return deps.getMemoryV2 ? deps.getMemoryV2(deps.getRootDir()) @@ -263,23 +358,34 @@ function parseArgs(rawArgs: string): { command: string; args: string[]; apply: b async function runV2Command(rawArgs: string, deps: MemoryCommandDeps): Promise { const parsed = parseArgs(rawArgs) - if (!['authority', 'diagnose', 'query', 'inspect', 'consolidate', 'repair', 'revalidate', 'correct', 'forget', 'pin', 'export', 'import'].includes(parsed.command)) return commandError(MEMORY_USAGE) + if (!['authority', 'diagnose', 'audit-migration', 'query', 'inspect', 'consolidate', 'repair', 'revalidate', 'correct', 'forget', 'pin', 'export', 'import'].includes(parsed.command)) return commandError(MEMORY_USAGE) const v2 = await getV2(deps) try { if (parsed.command === 'authority') { return report('Memory authority', [ 'Valid values: json-v1, shadow-v2, sqlite-v2-opt-in.', + RELEASE_N_AUTHORITY_WARNING, `Requested: ${v2.requestedAuthority}; effective: ${v2.effectiveAuthority}.`, 'Switch via OPENBUFF_MEMORY_AUTHORITY or the SDK authority option, then reset/restart the client.', ]) } if (v2.status === 'unavailable') return report('Memory V2 unavailable', [ `Requested authority: ${v2.requestedAuthority}; active authority: ${v2.effectiveAuthority}.`, + RELEASE_N_AUTHORITY_WARNING, safeOperationMessage(v2.degradation), ], 'warning') const scope = { schemaVersion: 2 as const, projectId: v2.projectId, sessionId: MemorySessionIdSchema.parse(CLI_SESSION_ID) } const now = new Date().toISOString() + if (parsed.command === 'audit-migration') { + const memory = await deps.loadPersistedTaskMemory({ rootDir: deps.getRootDir() }) + return renderMigrationAudit(await deps.auditTaskMemoryV1Migration({ + memory, + projectId: v2.projectId, + repository: v2.repository, + })) + } + if (parsed.command === 'diagnose') { const [health, kernel, capabilities, inventory] = await Promise.all([ v2.repository.health({ schemaVersion: 2, projectId: v2.projectId }), @@ -290,6 +396,7 @@ async function runV2Command(rawArgs: string, deps: MemoryCommandDeps): Promise { expect(command).toBeDefined() expect(command!.label).toBe('memory') expect(command!.description).toBe( - 'Inspect, query, diagnose, export, or maintain project memory', + 'Inspect, query, audit migrations, export, or maintain project memory', ) expect(command!.aliases).toEqual(['mem']) // Stateful command: must not fire without an explicit leading slash. diff --git a/cli/src/data/slash-commands.ts b/cli/src/data/slash-commands.ts index 0de1de0973..02876d5390 100644 --- a/cli/src/data/slash-commands.ts +++ b/cli/src/data/slash-commands.ts @@ -59,7 +59,7 @@ const ALL_SLASH_COMMANDS: SlashCommand[] = [ { id: 'memory', label: 'memory', - description: 'Inspect, query, diagnose, export, or maintain project memory', + description: 'Inspect, query, audit migrations, export, or maintain project memory', aliases: ['mem'], }, { diff --git a/common/knowledge.md b/common/knowledge.md index c04ea59fe6..da14ecf86d 100644 --- a/common/knowledge.md +++ b/common/knowledge.md @@ -26,11 +26,11 @@ This package contains code shared across the Openbuff monorepo, especially the l - **Provider config**: Shared validation and contracts for OpenAI-compatible, Anthropic-compatible, and other BYOK providers - **Workspace state**: `common/src/types/workspace-state.ts` tracks monotonically increasing local workspace revisions and bounded change records so SDK/runtime tools can tie mutation receipts, review bundles, and snapshot-aware validation to a stable `workspace.v1..` identifier. - **Task memory**: `common/src/types/task-memory.ts` defines bounded Zod schemas for durable requirements, evidence, blockers, validation, and review notes that survive long task execution without leaking unbounded context. `TASK_MEMORY_LIST_CAPS` is exported alongside the schema as the single source of truth for per-list array caps, so consumers (notably the SDK task-memory store that merges a finished run's memory into the persisted record) cannot hand-mirror limits and drift from what the schema enforces. -- **Published task-memory API (`@openbuff/sdk`)**: the SDK publishes `loadPersistedTaskMemory`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, and `codebuffFsToNodePromises`, together with the complete type closure their signatures name — `TaskMemoryV1`, `TaskMemoryDraftV1`, `TaskMemoryEvidenceV1` (re-exported from `common/src/types/task-memory.ts`) plus `TaskMemoryStoreFs`, `WorkspaceMoveRecord`, and `TaskMemoryPruneOutcome`. External consumers must be able to _name_ every parameter/return type of a published function, so adding a task-memory export means publishing its types from `sdk/src/index.ts` in the same change. +- **Published task-memory API (`@openbuff/sdk`)**: the SDK publishes `loadPersistedTaskMemory`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, `auditTaskMemoryV1Migration`, and `codebuffFsToNodePromises`, together with the complete type closure their signatures name — `TaskMemoryV1`, `TaskMemoryDraftV1`, `TaskMemoryEvidenceV1` (re-exported from `common/src/types/task-memory.ts`) plus `TaskMemoryStoreFs`, `WorkspaceMoveRecord`, `TaskMemoryPruneOutcome`, `V1MigrationAuditReader`, and `V1MigrationAuditOutcome`. The audit API is additive and read-only: it verifies a loaded V1 source revision/checksum against the V2 migration marker, imported body, and provenance without appending. `exact` is lossless only when `omittedFields`, `truncatedFields` (default zero), and warnings are all zero; bounded outcomes separately represent no record, not migrated, incomplete, mismatch, rejected, and failed. External consumers must be able to _name_ every parameter/return type of a published function, so adding a task-memory export means publishing its types from `sdk/src/index.ts` in the same change. - **task-memory.json writers**: `.openbuff/memory/task-memory.json` has two writers — `saveMergedTaskMemory` (end of run) and `pruneStaleTaskMemoryEvidence` (`/memory prune`) — and its `revision` must stay monotonic and unique across both. Uniqueness is enforced by serializing each writer's load→revision→commit section on a shared lock: an in-process promise chain keyed by the resolved record path, plus an exclusive-create `.lock` file for other processes that is reclaimed after a staleness timeout. Without it a save landing between prune's revision check and its rename, or two saves loading the same record, published one revision twice with different payloads. Evidence reconciliation deliberately stays outside the locked section because it hashes every evidence file. The lock is advisory — adapters that ignore exclusive-create, or a lock not obtained within the bounded attempt budget, degrade to running unlocked — so both writers keep their own guards: saves re-read the on-disk record and merge against whichever of it and the caller's hydrated `priorMemory` is newer, emitting one past both, and prune reports `status: 'failed', reason: 'concurrent-write'` when the record moved off the revision it loaded. - **Pruned evidence must not come back**: a revision guard prevents revision reuse, not content resurrection. `/memory prune` runs from inside a live session whose `runMemory.evidence` still holds the entries it dropped, so `saveMergedTaskMemory` also filters the run's contribution: when the on-disk record is the merge base, every evidence id the caller's hydrated snapshot carries but that record no longer does was deliberately removed by the other writer and is dropped from the run's evidence. Evidence the run recorded itself is untouched. - **Prune honors the move contract**: `pruneStaleTaskMemoryEvidence` takes the same optional `workspaceMoves` as `reconcileTaskMemoryEvidence`, because prune DELETES what reconciles stale. Callers that can see journal-recorded moves must pass them, or evidence bound to a renamed file reconciles stale and is permanently lost instead of rebinding to its destination as hydration documents. `collectWorkspaceMoves` is published from `@openbuff/sdk` for exactly that reason: non-run callers need the same journal-derived move set hydration uses. -- **`/memory` CLI command** (`cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts` as `memory`, alias `mem`): `/memory status` summarizes the persisted record and its fresh/stale evidence; `/memory prune` drops stale evidence. Both subcommands read journal moves via `collectWorkspaceMoves` + `WorkspaceJournalService` and pass them into reconciliation/prune, so a renamed file's evidence rebinds instead of being reported stale and deleted; a journal read that fails surfaces as a command failure rather than a blind prune. Prune reports the store's `TaskMemoryPruneOutcome` faithfully — `no-record` (nothing persisted yet), `pruned` (including `removed: 0`, "nothing to prune"), and `failed` with its reason (schema reject, lost revision race, unwritable target or rename-less filesystem adapter). A failed write is never phrased as an absent record or as "nothing to prune"; it names the cause and reports the stale entries still present. +- **`/memory` CLI command** (`cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts` as `memory`, alias `mem`): `/memory status` summarizes the persisted record and its fresh/stale evidence; `/memory prune` drops stale evidence; `/memory audit-migration` invokes the SDK audit against the currently loaded V1 record and leased V2 repository without mutation. Both status/prune subcommands read journal moves via `collectWorkspaceMoves` + `WorkspaceJournalService` and pass them into reconciliation/prune, so a renamed file's evidence rebinds instead of being reported stale and deleted; a journal read that fails surfaces as a command failure rather than a blind prune. Prune reports the store's `TaskMemoryPruneOutcome` faithfully — `no-record` (nothing persisted yet), `pruned` (including `removed: 0`, "nothing to prune"), and `failed` with its reason (schema reject, lost revision race, unwritable target or rename-less filesystem adapter). A failed write is never phrased as an absent record or as "nothing to prune"; it names the cause and reports the stale entries still present. Release N keeps `json-v1` and `shadow-v2` supported but deprecates them in favor of the default/replacement `sqlite-v2-opt-in`; authority selection and compatibility/fallback behavior remain unchanged, with no removal date implied and any future removal contingent on a later decision plus successful exact audits. - **Stable hash (`common/src/util/stable-hash.ts`)**: canonical FNV-1a 32-bit string hash rendered as padded 8-hex. It is the one implementation shared by agent-runtime `commitTaskMemory` checksums, the SDK task-memory store's persisted-record checksum, and the `git_status` change-gate fingerprint; byte-format vectors (`'' -> 811c9dc5`) are pinned by tests because the same digest must round-trip across packages. - **Gate telemetry sink (`common/src/util/gate-telemetry.ts`)**: append-only JSONL sink for base2's `emitGateTelemetry` payloads under `/.openbuff/telemetry/base2-gate.jsonl`, so per-round gate evidence outlives a single turn's `console.info` and questions like "what share of repair rounds were nit-driven" become mechanically answerable. Best-effort by contract: it never throws (not even when `logger.warn` itself throws), reports failure through its boolean return, and keeps owner-only modes (`0o700` directory, `0o600` file) like the SDK task-memory store. Lines are size-bounded in three escalating steps — array-item caps, per-field byte caps, then an event-name-only marker — and `truncated` / `truncatedFields` / `droppedPayloadKeys` are written by the SINK, so a payload field of one of those names is dropped rather than merged and cannot forge the truncation signal. Rotation keeps at most one `.1` generation and deliberately accepts two documented races; `GATE_TELEMETRY_MAX_BYTES` is the authoritative note for both. `errorCode` in `common/src/util/error.ts` is the shared helper it keys `ENOENT` on, and `common/src/testing/gate-telemetry-fixtures.ts` owns the temp-root helpers plus the platform skip policy shared with the agent-runtime suite (a skip rule that drifts between the two silently stops covering one). - **On-demand guide fallback (`common/src/util/guides.ts`)**: base2's progressive prompt disclosure replaces each verbose advisory section with a compact pointer naming a workspace-relative `agents/guides/*.md` path, which `read_files` resolves against the EMBEDDER's project root — and no publish pipeline ships the guides, so in any workspace other than this repo the pointer read fails and the model loses the whole section. `FALLBACK_GUIDES` is the single copy of those paths (base2's `GUIDE_PATHS` is an alias of it and its `GuidePath` union is derived from it); the table lives here rather than beside base2's pointers because `packages/agent-runtime` must not import from `agents/`. `findMissingGuides` returns `[]` for a falsy or non-string root — "unknown root" must never mean "everything is missing", or every prompt formatted without a real workspace regrows by six full sections — and `formatGuideFallbackSection` returns `''` for a guide that is present, so the resolved in-repo prompt stays byte-identical. Recovery is PER POINTER and clause-keyed: a mode that deliberately omits a pointer (plan mode is read-only and emits neither the git-discipline pointer nor its body) never recovers that body, and `BROAD_AUDIT_FALLBACK_SECTIONS` keys the one clause-parameterized body by finalize clause so a guide-less plan-mode prompt cannot receive the contradictory implementation clause. diff --git a/common/src/types/session-state.ts b/common/src/types/session-state.ts index b76e427782..40b6e369b4 100644 --- a/common/src/types/session-state.ts +++ b/common/src/types/session-state.ts @@ -396,7 +396,11 @@ export type AgentState = { childRunId?: string receipt?: AgentReceipt }> - /** Typed operational memory compiled into each model request independently of chat summaries. */ + /** + * Typed operational memory compiled into each model request independently of chat summaries. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ taskMemory?: TaskMemoryV1 /** Opt-in runtime-neutral Memory V2 lifecycle state. */ memoryV2?: MemoryRuntimeStateV2 diff --git a/common/src/types/task-memory.ts b/common/src/types/task-memory.ts index 803684d32b..1613035899 100644 --- a/common/src/types/task-memory.ts +++ b/common/src/types/task-memory.ts @@ -7,6 +7,8 @@ const boundedText = (max: number) => z.string().max(max) * consumers (e.g. the SDK task-memory store) merge against the exact * limits this schema enforces instead of hand-mirroring them and * drifting silently. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export const TASK_MEMORY_LIST_CAPS = { requirements: 64, @@ -20,6 +22,9 @@ export const TASK_MEMORY_LIST_CAPS = { evidence: 256, } as const +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export const taskMemoryEvidenceV1Schema = z.object({ id: z.string().min(1).max(160), kind: z.enum([ @@ -43,6 +48,9 @@ export const taskMemoryEvidenceV1Schema = z.object({ stale: z.boolean().optional(), }) +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export const taskMemoryDraftV1Schema = z.object({ schemaVersion: z.literal(1), goal: boundedText(8_000).default(''), @@ -87,12 +95,24 @@ export const taskMemoryDraftV1Schema = z.object({ workspaceSnapshotId: boundedText(256).optional(), }) +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export const taskMemoryV1Schema = taskMemoryDraftV1Schema.extend({ revision: z.number().int().nonnegative(), updatedAt: z.number().int().nonnegative(), checksum: z.string().min(1).max(64), }) +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export type TaskMemoryEvidenceV1 = z.infer +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export type TaskMemoryDraftV1 = z.infer +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export type TaskMemoryV1 = z.infer diff --git a/sdk/CHANGELOG.md b/sdk/CHANGELOG.md index 3bd7e5788e..47683953ed 100644 --- a/sdk/CHANGELOG.md +++ b/sdk/CHANGELOG.md @@ -6,6 +6,9 @@ All notable changes to the @openbuff/sdk package will be documented in this file ### Added +- New additive, read-only `auditTaskMemoryV1Migration` API and `V1MigrationAuditOutcome` type inspect the currently loaded V1 record against a V2 repository without appending or mutating. An `exact` result verifies the source revision/checksum, migration marker, imported task/observation bodies, and migration provenance; it is lossless only when `omittedFields`, `truncatedFields` (defaulting to zero when absent), and `warnings` are all zero. Other outcomes distinguish no record, not migrated, incomplete, mismatch, rejected, and failed states using bounded reason codes rather than repository error text. +- Release N deprecation guidance: `json-v1` and `shadow-v2` remain supported compatibility modes but are deprecated; `sqlite-v2-opt-in` is the default and replacement. This release does not remove V1 APIs or persistence, migration import, authority selection, shadow behavior, or fallback behavior. Any later removal requires a separate release decision and successful exact migration audits. + - New published task-memory API for durable cross-session task state: `loadPersistedTaskMemory`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, and `codebuffFsToNodePromises`, backed by `.openbuff/memory/task-memory.json` under the project root. The complete type closure those signatures name is published alongside them so consumers can name every parameter and return type: `TaskMemoryV1`, `TaskMemoryDraftV1`, and `TaskMemoryEvidenceV1` (re-exported from `@codebuff/common/types/task-memory`), plus `TaskMemoryStoreFs`, `WorkspaceMoveRecord`, and `TaskMemoryPruneOutcome`. - `pruneStaleTaskMemoryEvidence` returns a discriminated `TaskMemoryPruneOutcome` rather than a nullable count pair, so "there is no record" (`status: 'no-record'`) is distinguishable from "the prune could not be committed" (`status: 'failed'`, with `reason: 'invalid-record' | 'concurrent-write' | 'write-failed'` and the `removed`/`remaining` counts the prune would have written). A successful prune is `status: 'pruned'`, including the no-op `removed: 0` case for a fully fresh record. - `task-memory.json` has two writers, so `revision` stays monotonic and unique across both. `saveMergedTaskMemory` re-reads the on-disk record and merges against whichever of it and the caller's `priorMemory` is newer, emitting one revision past both; pruning refuses to write when the record advanced while it was reconciling. A session that hydrated a pre-prune record therefore cannot resurrect pruned evidence under an already-published revision. diff --git a/sdk/src/__tests__/memory-v2-contract.test.ts b/sdk/src/__tests__/memory-v2-contract.test.ts index d491968426..5befad1161 100644 --- a/sdk/src/__tests__/memory-v2-contract.test.ts +++ b/sdk/src/__tests__/memory-v2-contract.test.ts @@ -16,12 +16,16 @@ import type { MemoryRetrievalRequest, MemoryVerifyOutcome, MemoryVerifyRequest, + V1MigrationAuditOutcome, + V1MigrationAuditReader, } from '../index' import { + auditTaskMemoryV1Migration, MemoryAppendRequestSchema, MemoryEventDraftSchema, MemoryEventEnvelopeSchema, MemoryRetrievalRequestSchema, + ProjectIdSchema, } from '../index' const timestamp = '2026-09-10T19:41:53.753Z' @@ -151,6 +155,23 @@ class FakeMemoryRepositoryV2 implements MemoryRepositoryV2 { } describe('MemoryRepositoryV2 public contract', () => { + test('the V1 migration audit accepts an export-only reader', async () => { + let exportCalls = 0 + const reader: V1MigrationAuditReader = { + async export() { + exportCalls++ + return { outcome: 'page', events: [], nextAfterEventId: null } + }, + } + const outcome: V1MigrationAuditOutcome = await auditTaskMemoryV1Migration({ + projectId: ProjectIdSchema.parse('project:demo'), + repository: reader, + }) + + expect(outcome).toEqual({ outcome: 'no-record' }) + expect(exportCalls).toBe(0) + }) + test('supports a compile-time and runtime fake implementation', async () => { const event = MemoryEventDraftSchema.parse({ schemaVersion: 2, diff --git a/sdk/src/index.ts b/sdk/src/index.ts index 35bad25500..50d0a7732e 100644 --- a/sdk/src/index.ts +++ b/sdk/src/index.ts @@ -107,11 +107,15 @@ export type { export type * from './services/memory-v2/types' export { MemoryV2Coordinator } from './services/memory-v2/coordinator' export { + auditTaskMemoryV1Migration, getV1MigrationIdentity, importTaskMemoryV1, } from './services/memory-v2/v1-migration' export type { + V1MigrationAuditOutcome, + V1MigrationAuditReader, V1MigrationOutcome, + V1MigrationSourceItemCounts, V1MigrationWarningCode, } from './services/memory-v2/v1-migration' export { MemoryV2OperatorService } from './services/memory-v2/operator-service' diff --git a/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts b/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts index 2b0bbece4e..6484f8ac1d 100644 --- a/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts +++ b/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts @@ -2,8 +2,11 @@ import { describe, expect, test } from 'bun:test' import { MemoryAppendRequestSchema, + MemoryEventEnvelopeSchema, + MemoryEventIdSchema, MemorySessionIdSchema, ProjectIdSchema, + TaskIdSchema, type MemoryEventEnvelope, } from '@codebuff/common/types/memory-v2' import { @@ -12,7 +15,12 @@ import { } from '@codebuff/common/types/task-memory' import { stableHash } from '@codebuff/common/util/stable-hash' -import { getV1MigrationIdentity, importTaskMemoryV1 } from '../v1-migration' +import { + auditTaskMemoryV1Migration, + getV1MigrationIdentity, + importTaskMemoryV1, + type V1MigrationAuditReader, +} from '../v1-migration' import type { MemoryRepositoryV2 } from '../types' const projectId = ProjectIdSchema.parse('project:migration-test') @@ -56,6 +64,7 @@ class Repository implements MemoryRepositoryV2 { events = new Map() sequence = 0 appendCalls = 0 + exportCalls = 0 failAppendCall: number | undefined failAppendAsConflict = false appendRequests: Array> = [] @@ -122,6 +131,7 @@ class Repository implements MemoryRepositoryV2 { } async export(input: Parameters[0]) { + this.exportCalls++ const events = [...this.events.values()] const afterIndex = input.afterEventId ? events.findIndex((event) => event.eventId === input.afterEventId) @@ -152,6 +162,14 @@ class Repository implements MemoryRepositoryV2 { const run = (repository: Repository, value?: TaskMemoryV1) => importTaskMemoryV1({ memory: value, projectId, sessionId, repository }) +const audit = (repository: V1MigrationAuditReader, value?: TaskMemoryV1) => + auditTaskMemoryV1Migration({ memory: value, projectId, repository }) + +const cloneEvent = ( + event: MemoryEventEnvelope, + overrides: Partial, +): MemoryEventEnvelope => MemoryEventEnvelopeSchema.parse({ ...event, ...overrides }) + const migrationReservations = (repository: Repository) => [...repository.events.values()].filter( (event) => event.eventType === 'migration.v1.reserved', @@ -493,3 +511,357 @@ describe('V1 memory migration', () => { } }) }) + +describe('V1 memory migration audit', () => { + test('no record and checksum mismatch make no repository calls', async () => { + const repository = new Repository() + expect(await audit(repository)).toEqual({ outcome: 'no-record' }) + expect(await audit(repository, memory({ checksum: 'invalid' }))).toMatchObject({ + outcome: 'rejected', + reason: 'checksum-mismatch', + }) + expect(repository.exportCalls).toBe(0) + expect(repository.appendCalls).toBe(0) + }) + + test('reports not-migrated from a complete empty scan', async () => { + const repository = new Repository() + const source = memory() + expect(await audit(repository, source)).toEqual({ + outcome: 'not-migrated', + revision: source.revision, + checksum: source.checksum, + }) + expect(repository.appendCalls).toBe(0) + }) + + test('certifies an imported body without writes and preserves lossy metadata', async () => { + const repository = new Repository() + const source = memory({ historicalSummary: 'h'.repeat(1_100) }) + const imported = await run(repository, source) + expect(imported.outcome).toBe('imported') + const appendCalls = repository.appendCalls + + const outcome = await audit(repository, source) + expect(outcome).toMatchObject({ + outcome: 'exact', + revision: source.revision, + checksum: source.checksum, + identity: getV1MigrationIdentity({ + projectId, + revision: source.revision, + checksum: source.checksum, + }), + repositoryLastEventId: [...repository.events.values()].at(-1)!.eventId, + warnings: expect.arrayContaining([ + 'goal-excluded', + 'legacy-evidence-unverified', + 'text-truncated', + ]), + truncatedFields: 1, + }) + if (outcome.outcome === 'exact' && imported.outcome === 'imported') { + expect(outcome.markerEventId).toBe([...migrationMarkers(repository)][0]!.eventId) + expect(outcome.importedTaskId).toBe(imported.importedTaskId) + expect(outcome.importedObservationIds).toEqual(imported.importedObservationIds) + expect(outcome.omittedFields).toBe(imported.omittedFields) + expect(outcome.sourceItemCounts).toEqual(imported.sourceItemCounts) + } + expect(repository.appendCalls).toBe(appendCalls) + }) + + test('reports a reservation-only partial import as incomplete', async () => { + const repository = new Repository() + const source = memory() + repository.failAppendCall = 2 + expect(await run(repository, source)).toMatchObject({ outcome: 'failed' }) + + expect(await audit(repository, source)).toMatchObject({ + outcome: 'incomplete', + reason: 'reservation-only', + repositoryLastEventId: migrationReservations(repository)[0]!.eventId, + }) + }) + + test('reports same-revision checksum ownership as a mismatch', async () => { + const repository = new Repository() + const first = memory() + expect((await run(repository, first)).outcome).toBe('imported') + const changed = memory({ + revision: first.revision, + updatedAt: first.updatedAt + 1, + decisions: [...first.decisions, 'changed'], + }) + + expect(await audit(repository, changed)).toMatchObject({ + outcome: 'mismatch', + reason: 'checksum-conflict', + checksum: changed.checksum, + }) + }) + + test('detects missing imported task and observation bodies', async () => { + const missingTaskRepository = new Repository() + const source = memory() + const taskImport = await run(missingTaskRepository, source) + expect(taskImport.outcome).toBe('imported') + const taskEvent = [...missingTaskRepository.events.values()].find( + (event) => event.eventType === 'task.created', + )! + missingTaskRepository.events.delete(taskEvent.eventId) + expect(await audit(missingTaskRepository, source)).toMatchObject({ + outcome: 'incomplete', + reason: 'missing-imported-task', + }) + + const repository = new Repository() + const imported = await run(repository, source) + expect(imported.outcome).toBe('imported') + if (imported.outcome !== 'imported') return + const removed = [...repository.events.values()].find( + (event) => + event.eventType === 'observation.recorded' && + event.payload.observation.observationId === imported.importedObservationIds[0], + )! + repository.events.delete(removed.eventId) + + expect(await audit(repository, source)).toMatchObject({ + outcome: 'incomplete', + reason: 'missing-imported-observations', + }) + }) + + test('detects mismatched imported observation provenance', async () => { + const repository = new Repository() + const source = memory() + const imported = await run(repository, source) + expect(imported.outcome).toBe('imported') + if (imported.outcome !== 'imported') return + const original = [...repository.events.values()].find( + (event) => + event.eventType === 'observation.recorded' && + event.payload.observation.observationId === imported.importedObservationIds[0], + )! + if (original.eventType !== 'observation.recorded') return + const mutated = MemoryEventEnvelopeSchema.parse({ + ...original, + payload: { + ...original.payload, + observation: { + ...original.payload.observation, + provenance: { + ...original.payload.observation.provenance, + metadata: { + ...original.payload.observation.provenance?.metadata, + checksum: 'different', + }, + }, + }, + }, + }) + repository.events.set(original.eventId, mutated) + + expect(await audit(repository, source)).toMatchObject({ + outcome: 'mismatch', + reason: 'imported-body-mismatch', + }) + }) + + test('treats a legacy marker without source proof as unverifiable', async () => { + const repository = new Repository() + const source = memory() + expect((await run(repository, source)).outcome).toBe('imported') + const marker = migrationMarkers(repository)[0]! + if (marker.eventType !== 'migration.v1.imported') return + const { + sourceRevision: _sourceRevision, + sourceChecksum: _sourceChecksum, + ...legacyPayload + } = marker.payload + repository.events.set( + marker.eventId, + MemoryEventEnvelopeSchema.parse({ ...marker, payload: legacyPayload }), + ) + + expect(await audit(repository, source)).toMatchObject({ + outcome: 'incomplete', + reason: 'legacy-marker-unverifiable', + markerEventId: marker.eventId, + }) + }) + + test('maps repository failures and fails closed on malformed or wrong-project output', async () => { + const source = memory() + const repository = new Repository() + await run(repository, source) + const event = [...repository.events.values()][0]! + const otherProjectEvent = cloneEvent(event, { + projectId: ProjectIdSchema.parse('project:other'), + }) + const cases: Array<{ + reader: V1MigrationAuditReader + expected: { outcome: string; reason: string } + }> = [ + { + reader: { + async export() { + return { + outcome: 'rejected', + error: { code: 'invalid-request', message: 'no', retryable: false }, + } + }, + }, + expected: { outcome: 'rejected', reason: 'repository-rejected' }, + }, + { + reader: { + async export() { + return { + outcome: 'failed', + error: { code: 'unavailable', message: 'no', retryable: true }, + } + }, + }, + expected: { outcome: 'failed', reason: 'repository-failed' }, + }, + { + reader: { + async export(): Promise { + throw new Error('private failure') + }, + }, + expected: { outcome: 'failed', reason: 'repository-failed' }, + }, + { + reader: { + async export() { + return { outcome: 'page', events: 'invalid', nextAfterEventId: null } + }, + } as unknown as V1MigrationAuditReader, + expected: { outcome: 'failed', reason: 'invalid-export' }, + }, + { + reader: { + async export() { + return { outcome: 'page', events: [otherProjectEvent], nextAfterEventId: null } + }, + }, + expected: { outcome: 'failed', reason: 'wrong-project' }, + }, + ] + + for (const { reader, expected } of cases) { + expect(await audit(reader, source)).toMatchObject(expected) + } + }) + + test('rejects empty, non-tail, and repeated pagination cursors', async () => { + const source = memory() + const repository = new Repository() + await run(repository, source) + const first = [...repository.events.values()][0]! + const second = cloneEvent(first, { + eventId: MemoryEventIdSchema.parse('event:audit-second'), + sequence: first.sequence + 1, + }) + const cursor = first.eventId + const readers: V1MigrationAuditReader[] = [ + { + async export() { + return { outcome: 'page', events: [], nextAfterEventId: cursor } + }, + }, + { + async export() { + return { + outcome: 'page', + events: [first], + nextAfterEventId: second.eventId, + } + }, + }, + { + async export(request) { + return request.afterEventId + ? { outcome: 'page', events: [second], nextAfterEventId: cursor } + : { outcome: 'page', events: [first], nextAfterEventId: cursor } + }, + }, + { + async export(request) { + return request.afterEventId + ? { outcome: 'page', events: [first], nextAfterEventId: null } + : { outcome: 'page', events: [first], nextAfterEventId: cursor } + }, + }, + ] + for (const reader of readers) { + expect(await audit(reader, source)).toMatchObject({ + outcome: 'failed', + reason: 'pagination-invalid', + }) + } + }) + + test('fails rather than certifying a scan with a tenth continuation page', async () => { + const source = memory() + const repository = new Repository() + await run(repository, source) + const template = [...repository.events.values()][0]! + let page = 0 + const reader: V1MigrationAuditReader = { + async export() { + const event = cloneEvent(template, { + eventId: MemoryEventIdSchema.parse(`event:audit-page-${page}`), + sequence: page + 1, + }) + page++ + return { outcome: 'page', events: [event], nextAfterEventId: event.eventId } + }, + } + + expect(await audit(reader, source)).toMatchObject({ + outcome: 'failed', + reason: 'page-limit-exceeded', + }) + expect(page).toBe(10) + }) + + test('allows a full final page to be followed by an empty terminal page', async () => { + const source = memory() + const repository = new Repository() + expect((await run(repository, source)).outcome).toBe('imported') + const canonical = [...repository.events.values()] + const template = canonical.find( + (event) => event.eventType === 'task.created', + )! + const padding = Array.from({ length: 1_000 - canonical.length }, (_, index) => + cloneEvent(template, { + eventId: MemoryEventIdSchema.parse(`event:audit-padding-${index}`), + sequence: canonical.length + index + 1, + sessionId: MemorySessionIdSchema.parse(`session:audit-padding-${index}`), + payload: { + ...template.payload, + taskId: TaskIdSchema.parse(`task:audit-padding-${index}`), + }, + }), + ) + const events = [...canonical, ...padding] + let calls = 0 + const reader: V1MigrationAuditReader = { + async export(request) { + calls++ + return request.afterEventId + ? { outcome: 'page', events: [], nextAfterEventId: null } + : { + outcome: 'page', + events, + nextAfterEventId: events.at(-1)!.eventId, + } + }, + } + + expect(await audit(reader, source)).toMatchObject({ outcome: 'exact' }) + expect(calls).toBe(2) + }) +}) diff --git a/sdk/src/services/memory-v2/v1-migration.ts b/sdk/src/services/memory-v2/v1-migration.ts index 6cd8d187a9..f4d19c7c18 100644 --- a/sdk/src/services/memory-v2/v1-migration.ts +++ b/sdk/src/services/memory-v2/v1-migration.ts @@ -2,9 +2,11 @@ import { MemoryAppendOutcomeSchema, MemoryAppendRequestSchema, MemoryEventIdSchema, + MemoryExportOutcomeSchema, ObservationIdSchema, TaskIdSchema, type MemoryEventDraft, + type MemoryEventEnvelope, type MemoryEventId, type MemorySessionId, type ObservationId, @@ -41,7 +43,7 @@ const MIGRATION_CATEGORIES = [ ] as const type V1MigrationCategory = (typeof MIGRATION_CATEGORIES)[number] -type V1MigrationSourceItemCounts = Record +export type V1MigrationSourceItemCounts = Record function newSourceItemCounts(): V1MigrationSourceItemCounts { return Object.fromEntries( @@ -60,6 +62,9 @@ function markerSourceItemCounts( return normalized } +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export type V1MigrationWarningCode = | 'goal-excluded' | 'observation-cap-reached' @@ -69,6 +74,9 @@ export type V1MigrationWarningCode = | 'empty-field-omitted' | 'text-truncated' +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export type V1MigrationOutcome = | { outcome: 'no-record' } | { @@ -105,6 +113,9 @@ const boundedText = ( const hashToken = (value: string): string => stableHash(value).replace(/[^A-Za-z0-9._:-]/g, '').slice(0, 96) +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export function getV1MigrationIdentity(params: { projectId: ProjectId revision: number @@ -275,6 +286,360 @@ function outcomeFromMarker(params: { } } +export type V1MigrationAuditReader = Pick + +export type V1MigrationAuditOutcome = + | { outcome: 'no-record' } + | { + outcome: 'not-migrated' + revision: number + checksum: string + repositoryLastEventId?: MemoryEventId + } + | { + outcome: 'exact' + revision: number + checksum: string + identity: string + markerEventId: MemoryEventId + repositoryLastEventId: MemoryEventId + importedTaskId: TaskId + importedObservationIds: ObservationId[] + omittedFields: number + warnings: V1MigrationWarningCode[] + sourceItemCounts?: V1MigrationSourceItemCounts + truncatedFields?: number + } + | { + outcome: 'incomplete' | 'mismatch' + reason: + | 'reservation-only' + | 'legacy-marker-unverifiable' + | 'missing-imported-task' + | 'missing-imported-observations' + | 'imported-body-mismatch' + | 'revision-conflict' + | 'checksum-conflict' + revision: number + checksum: string + repositoryLastEventId?: MemoryEventId + markerEventId?: MemoryEventId + missingEventIds?: MemoryEventId[] + } + | { + outcome: 'rejected' | 'failed' + reason: + | 'checksum-mismatch' + | 'repository-rejected' + | 'repository-failed' + | 'invalid-export' + | 'wrong-project' + | 'pagination-invalid' + | 'page-limit-exceeded' + revision?: number + checksum?: string + } + +type V1MigrationAuditScan = + | { + outcome: 'complete' + events: MemoryEventEnvelope[] + repositoryLastEventId?: MemoryEventId + } + | Extract + +type ImportedMarkerEvent = Extract< + MemoryEventEnvelope, + { eventType: 'migration.v1.imported' } +> +type RecordedObservationEvent = Extract< + MemoryEventEnvelope, + { eventType: 'observation.recorded' } +> + +async function scanV1MigrationAuditEvents( + repository: V1MigrationAuditReader, + projectId: ProjectId, +): Promise { + const events: MemoryEventEnvelope[] = [] + const eventIds = new Set() + const cursors = new Set() + let afterEventId: MemoryEventId | undefined + let repositoryLastEventId: MemoryEventId | undefined + + for (let page = 0; page < MAX_EXPORT_PAGES; page++) { + let raw: unknown + try { + raw = await repository.export({ + schemaVersion: 2, + projectId, + ...(afterEventId ? { afterEventId } : {}), + limit: 1_000, + }) + } catch { + return { outcome: 'failed', reason: 'repository-failed' } + } + const parsed = MemoryExportOutcomeSchema.safeParse(raw) + if (!parsed.success) return { outcome: 'failed', reason: 'invalid-export' } + const exported = parsed.data + if (exported.outcome !== 'page') { + return { + outcome: exported.outcome, + reason: + exported.outcome === 'rejected' + ? 'repository-rejected' + : 'repository-failed', + } + } + + for (const event of exported.events) { + if (event.projectId !== projectId) { + return { outcome: 'failed', reason: 'wrong-project' } + } + if (eventIds.has(event.eventId)) { + return { outcome: 'failed', reason: 'pagination-invalid' } + } + eventIds.add(event.eventId) + events.push(event) + repositoryLastEventId = event.eventId + } + + const nextAfterEventId = exported.nextAfterEventId ?? undefined + if (!nextAfterEventId) { + return { outcome: 'complete', events, repositoryLastEventId } + } + const pageLastEventId = exported.events.at(-1)?.eventId + if ( + !pageLastEventId || + nextAfterEventId !== pageLastEventId || + nextAfterEventId === afterEventId || + cursors.has(nextAfterEventId) + ) { + return { outcome: 'failed', reason: 'pagination-invalid' } + } + cursors.add(nextAfterEventId) + afterEventId = nextAfterEventId + } + + return { outcome: 'failed', reason: 'page-limit-exceeded' } +} + +function auditMigrationMarkerBody(params: { + marker: ImportedMarkerEvent + events: MemoryEventEnvelope[] + revision: number + checksum: string + identity: string + repositoryLastEventId: MemoryEventId +}): V1MigrationAuditOutcome { + const { + marker, + events, + revision, + checksum, + identity, + repositoryLastEventId, + } = params + const markerResult = (outcome: 'incomplete' | 'mismatch', reason: + | 'missing-imported-task' + | 'missing-imported-observations' + | 'imported-body-mismatch'): V1MigrationAuditOutcome => ({ + outcome, + reason, + revision, + checksum, + repositoryLastEventId, + markerEventId: marker.eventId, + }) + + if (marker.payload.legacyRecordKey !== identity) { + return markerResult('mismatch', 'imported-body-mismatch') + } + const taskExists = events.some( + (event) => + event.eventType === 'task.created' && + event.payload.taskId === marker.payload.importedTaskId, + ) + if (!taskExists) return markerResult('incomplete', 'missing-imported-task') + if ( + new Set(marker.payload.importedObservationIds).size !== + marker.payload.importedObservationIds.length + ) { + return markerResult('mismatch', 'imported-body-mismatch') + } + + const observations = new Map() + for (const event of events) { + if (event.eventType !== 'observation.recorded') continue + const observationId = event.payload.observation.observationId + const matching = observations.get(observationId) ?? [] + matching.push(event) + observations.set(observationId, matching) + } + + let missing = false + for (const observationId of marker.payload.importedObservationIds) { + const matching = observations.get(observationId) ?? [] + if (matching.length === 0) { + missing = true + continue + } + if (matching.length !== 1) { + return markerResult('mismatch', 'imported-body-mismatch') + } + const observation = matching[0]!.payload.observation + if ( + observation.taskId !== marker.payload.importedTaskId || + observation.provenance?.origin !== 'migration' || + observation.provenance.metadata?.revision !== revision || + observation.provenance.metadata?.checksum !== checksum || + !observation.tags.includes('legacy-v1') + ) { + return markerResult('mismatch', 'imported-body-mismatch') + } + } + if (missing) { + return markerResult('incomplete', 'missing-imported-observations') + } + + return { + outcome: 'exact', + revision, + checksum, + identity, + markerEventId: marker.eventId, + repositoryLastEventId, + importedTaskId: marker.payload.importedTaskId, + importedObservationIds: [...marker.payload.importedObservationIds], + omittedFields: marker.payload.omittedFields ?? 0, + warnings: marker.payload.warnings.filter(isV1MigrationWarningCode), + ...(marker.payload.sourceItemCounts + ? { + sourceItemCounts: markerSourceItemCounts( + marker.payload.sourceItemCounts, + ), + } + : {}), + ...(marker.payload.truncatedFields === undefined + ? {} + : { truncatedFields: marker.payload.truncatedFields }), + } +} + +export async function auditTaskMemoryV1Migration(params: { + memory?: TaskMemoryV1 + projectId: ProjectId + repository: V1MigrationAuditReader +}): Promise { + const { memory, projectId, repository } = params + if (!memory) return { outcome: 'no-record' } + + const { revision, updatedAt, checksum, ...candidateDraft } = memory + const parsedDraft = taskMemoryDraftV1Schema.safeParse(candidateDraft) + if (!parsedDraft.success) { + return { outcome: 'rejected', reason: 'checksum-mismatch', revision, checksum } + } + const recomputed = stableHash( + JSON.stringify({ revision, updatedAt, memory: parsedDraft.data }), + ) + if (recomputed !== checksum) { + return { outcome: 'rejected', reason: 'checksum-mismatch', revision, checksum } + } + + const scanned = await scanV1MigrationAuditEvents(repository, projectId) + if (scanned.outcome !== 'complete') { + return { ...scanned, revision, checksum } + } + const identity = getV1MigrationIdentity({ projectId, revision, checksum }) + const matchingReservations = scanned.events.filter( + (event) => + event.eventType === 'migration.v1.reserved' && + event.payload.sourceRevision === revision && + event.payload.sourceChecksum === checksum && + event.payload.migrationId === identity, + ) + const conflictingReservation = scanned.events.some( + (event) => + event.eventType === 'migration.v1.reserved' && + event.payload.sourceRevision === revision && + (event.payload.sourceChecksum !== checksum || + event.payload.migrationId !== identity), + ) + const exactMarkers = scanned.events.filter( + (event): event is ImportedMarkerEvent => + event.eventType === 'migration.v1.imported' && + event.payload.sourceRevision === revision && + event.payload.sourceChecksum === checksum, + ) + const conflictingMarker = scanned.events.some( + (event) => + event.eventType === 'migration.v1.imported' && + event.payload.sourceRevision === revision && + event.payload.sourceChecksum !== undefined && + event.payload.sourceChecksum !== checksum, + ) + const legacyMarkers = scanned.events.filter( + (event): event is ImportedMarkerEvent => + event.eventType === 'migration.v1.imported' && + event.payload.legacyRecordKey === identity && + (event.payload.sourceRevision === undefined || + event.payload.sourceChecksum === undefined), + ) + const unverifiableRevisionMarker = scanned.events.some( + (event) => + event.eventType === 'migration.v1.imported' && + event.payload.sourceRevision === revision && + event.payload.sourceChecksum === undefined, + ) + const auditEvidence = { + revision, + checksum, + ...(scanned.repositoryLastEventId + ? { repositoryLastEventId: scanned.repositoryLastEventId } + : {}), + } + + if (conflictingReservation || conflictingMarker) { + return { outcome: 'mismatch', reason: 'checksum-conflict', ...auditEvidence } + } + if ( + exactMarkers.length > 1 || + matchingReservations.length > 1 || + legacyMarkers.length > 1 || + (unverifiableRevisionMarker && exactMarkers.length > 0) + ) { + return { outcome: 'mismatch', reason: 'revision-conflict', ...auditEvidence } + } + if (exactMarkers.length === 1) { + return auditMigrationMarkerBody({ + marker: exactMarkers[0]!, + events: scanned.events, + revision, + checksum, + identity, + repositoryLastEventId: scanned.repositoryLastEventId!, + }) + } + if (legacyMarkers.length === 1) { + return { + outcome: 'incomplete', + reason: 'legacy-marker-unverifiable', + markerEventId: legacyMarkers[0]!.eventId, + ...auditEvidence, + } + } + if (unverifiableRevisionMarker) { + return { outcome: 'mismatch', reason: 'revision-conflict', ...auditEvidence } + } + if (matchingReservations.length === 1) { + return { outcome: 'incomplete', reason: 'reservation-only', ...auditEvidence } + } + return { outcome: 'not-migrated', ...auditEvidence } +} + +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export async function importTaskMemoryV1(params: { memory?: TaskMemoryV1 projectId: ProjectId diff --git a/sdk/src/services/task-memory-store.ts b/sdk/src/services/task-memory-store.ts index 44adaadd6a..91221c7bbb 100644 --- a/sdk/src/services/task-memory-store.ts +++ b/sdk/src/services/task-memory-store.ts @@ -16,6 +16,9 @@ import type { TaskMemoryV1, } from '@codebuff/common/types/task-memory' +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export interface WorkspaceMoveRecord { from: string to: string @@ -29,6 +32,8 @@ type FsModule = typeof nodeFsPromises * them as the full `FsModule` would let callers invoke anything else (e.g. * `copyFile`) and hit a silent runtime `undefined` instead of a compile * error. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export interface TaskMemoryStoreFs { mkdir( @@ -106,6 +111,8 @@ const LOCK_STALE_MS = 10_000 * Load and schema-validate the persisted task memory for a project root, * re-verifying the stored checksum against the loaded payload. Missing, * corrupt, or checksum-mismatched data yields undefined; never throws. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export async function loadPersistedTaskMemory(params: { rootDir: string @@ -286,6 +293,8 @@ async function isPathInsideRoot( * Re-evaluate each evidence item against current disk state. Missing or * changed files mark the entry stale; a matching workspace move rebinds the * path to its destination before evaluating. Entries are never deleted. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export async function reconcileTaskMemoryEvidence(params: { memory: TaskMemoryV1 @@ -393,6 +402,8 @@ function collectDroppedEvidenceIds( * longer carries but the caller's hydrated snapshot still does is treated as * deliberately dropped by the other writer and filtered out of the run's * still-hydrated `evidence` instead of being merged back in. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export async function saveMergedTaskMemory(params: { rootDir: string @@ -699,6 +710,8 @@ async function acquireRecordLock( * user as "nothing to prune". On `failed`, `removed`/`remaining` describe the * prune that WOULD have been written, so the record still holds `removed` * stale entries. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export type TaskMemoryPruneOutcome = | { status: 'pruned'; removed: number; remaining: number } @@ -734,6 +747,8 @@ export type TaskMemoryPruneOutcome = * that can see workspace moves must pass them, or evidence bound to a renamed * file reconciles stale and is permanently deleted instead of rebinding to * its destination. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export async function pruneStaleTaskMemoryEvidence(params: { rootDir: string @@ -841,6 +856,8 @@ export async function pruneStaleTaskMemoryEvidence(params: { * rather than a non-atomic write. A native `open` on the host (real node * fs) is forwarded so oversized evidence hashing streams leading bytes * instead of buffering the file. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export function codebuffFsToNodePromises( codebuffFs: CodebuffFileSystem, From aa60102c2025d83ef01d8bd0cfc640681516b35b Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Sun, 13 Sep 2026 01:58:45 +0300 Subject: [PATCH 02/22] docs(memory): define V1 removal readiness gates --- cli/knowledge.md | 4 +- common/knowledge.md | 4 +- docs/memory-v1-removal-readiness.md | 243 ++++++++++++++++++++++++++++ sdk/CHANGELOG.md | 4 +- 4 files changed, 249 insertions(+), 6 deletions(-) create mode 100644 docs/memory-v1-removal-readiness.md diff --git a/cli/knowledge.md b/cli/knowledge.md index 8f86aba071..26ee700b14 100644 --- a/cli/knowledge.md +++ b/cli/knowledge.md @@ -16,14 +16,14 @@ - The SQLite repository projects the latest `coverage.recorded` event per `(taskId, dimension)` into the `currentCoverage` retrieval category, filtered by exact workspace revision/snapshot freshness and optional `taskId` narrowing. - The `query_index` tool handler (`packages/agent-runtime/src/tools/handlers/tool/query-index.ts`) records its results into `agentState.discoveryCoverage` via `recordDiscoveryResult` with a bounded (4000-char) question string; recording is wrapped in try/catch so a coverage failure can never break the tool call. - When testing tool handlers that import a function to spy on, prefer `spyOn(namespaceImport, 'fn')` over `mock.module()` (repo convention in `docs/testing.md`/`CONTRIBUTING.md`); a relative `mock.module` specifier resolves from the test file's directory and silently misses the module the subject under test imports when the two live at different depths. -- Memory V2 authority is selected by `OPENBUFF_MEMORY_AUTHORITY` via `getMemoryAuthoritySelection` (`cli/src/utils/env.ts`); the default is `sqlite-v2-opt-in` (fail-closed — no V1 fallback), with `json-v1` as an explicit opt-out escape hatch and `shadow-v2` as an explicit shadow mode. Release N keeps `json-v1` and `shadow-v2` supported but deprecates both compatibility modes; `sqlite-v2-opt-in` is the default and replacement. Authority selection, degradation, and fallback behavior are unchanged in this release, and no removal date/version is implied. +- Memory V2 authority is selected by `OPENBUFF_MEMORY_AUTHORITY` via `getMemoryAuthoritySelection` (`cli/src/utils/env.ts`); the default is `sqlite-v2-opt-in` (fail-closed — no V1 fallback), with `json-v1` as an explicit opt-out escape hatch and `shadow-v2` as an explicit shadow mode. Release N keeps `json-v1` and `shadow-v2` supported but deprecates both compatibility modes; `sqlite-v2-opt-in` is the default and replacement. Authority selection, degradation, and fallback behavior are unchanged in this release, no removal date/version is implied, and migration is not claimed complete. Follow the normative [Memory V1 removal readiness plan](../docs/memory-v1-removal-readiness.md) before any later removal decision. ## Slash Commands and Plan Mode - Durable planning is entered through `mode:plan`; the standalone `/plan` command is intentionally absent from `COMMAND_REGISTRY` and `SLASH_COMMANDS` so there is one plan-entry path. - Keep the durable-plan quartet registered: `/resume-plan` (`rp`), `/update-plan` (`up`), `/plan-status` (`ps`), and `/lessons` (`lesson`). These commands operate on `.agents/sessions//` artifacts and fall back to the plan-session picker when no target is provided. - `/plans` (`plan-ls`) lists the artifact-bearing sessions returned by `listPlanSessions()` and never prompts the agent, while `/plan-use` (`plan-active`, `use-plan`) writes the `ACTIVE_SESSION` pointer file under `.agents/` and only accepts a bare slug or `.agents/sessions/` — the resolved directory must be exactly one segment under `.agents/sessions/` because the pointer stores bare slugs, so nested paths, non-session paths, missing directories, and artifact-less directories are all rejected before any write. Both share `formatPlanSessionListRow` / `PLAN_SESSIONS_DIR_PREFIX` from `cli/src/commands/plan-artifacts.ts` so the rendered box and the text report cannot drift. -- `/memory` (alias `/mem`, `cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts`) inspects the persisted cross-session task memory for the current project (`.openbuff/memory/task-memory.json`). `/memory status` (the default) reports the record's revision and age, its goal and per-list counts, and how much of its evidence still verifies against disk, listing up to five stale paths; `/memory prune` drops evidence that no longer verifies. Both subcommands are move-aware via `collectWorkspaceMoves` + `WorkspaceJournalService` so a renamed file's evidence rebinds rather than being reported stale and deleted. `/memory audit-migration` is dedicated and read-only: it passes the currently loaded V1 record and leased V2 repository to the SDK `auditTaskMemoryV1Migration` API, never uses the CLI canonical-inventory marker scan, and never appends. An `exact` audit verifies source revision/checksum, marker, imported body, and provenance, but is described as lossless only when omissions, truncations, and warnings are all zero; all other audit outcomes are rendered distinctly with bounded reason text. +- `/memory` (alias `/mem`, `cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts`) inspects the persisted cross-session task memory for the current project (`.openbuff/memory/task-memory.json`). `/memory status` (the default) reports the record's revision and age, its goal and per-list counts, and how much of its evidence still verifies against disk, listing up to five stale paths; `/memory prune` drops evidence that no longer verifies. Both subcommands are move-aware via `collectWorkspaceMoves` + `WorkspaceJournalService` so a renamed file's evidence rebinds rather than being reported stale and deleted. `/memory audit-migration` is dedicated and read-only: it passes the currently loaded V1 record and leased V2 repository to the SDK `auditTaskMemoryV1Migration` API, never uses the CLI canonical-inventory marker scan, and never appends. An `exact` audit verifies source revision/checksum, marker identity, imported task existence, imported observation IDs, and migration provenance, not full source-derived body equality; it is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`. `not-migrated` means removal is not ready and is not a product defect; all outcomes are rendered distinctly with bounded reason text. - Slash-command descriptions should stay model-agnostic under BYOK/local mode. Use wording such as "configured reviewer" rather than naming hosted models. ## Import Guidelines diff --git a/common/knowledge.md b/common/knowledge.md index da14ecf86d..79c268e2e1 100644 --- a/common/knowledge.md +++ b/common/knowledge.md @@ -26,11 +26,11 @@ This package contains code shared across the Openbuff monorepo, especially the l - **Provider config**: Shared validation and contracts for OpenAI-compatible, Anthropic-compatible, and other BYOK providers - **Workspace state**: `common/src/types/workspace-state.ts` tracks monotonically increasing local workspace revisions and bounded change records so SDK/runtime tools can tie mutation receipts, review bundles, and snapshot-aware validation to a stable `workspace.v1..` identifier. - **Task memory**: `common/src/types/task-memory.ts` defines bounded Zod schemas for durable requirements, evidence, blockers, validation, and review notes that survive long task execution without leaking unbounded context. `TASK_MEMORY_LIST_CAPS` is exported alongside the schema as the single source of truth for per-list array caps, so consumers (notably the SDK task-memory store that merges a finished run's memory into the persisted record) cannot hand-mirror limits and drift from what the schema enforces. -- **Published task-memory API (`@openbuff/sdk`)**: the SDK publishes `loadPersistedTaskMemory`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, `auditTaskMemoryV1Migration`, and `codebuffFsToNodePromises`, together with the complete type closure their signatures name — `TaskMemoryV1`, `TaskMemoryDraftV1`, `TaskMemoryEvidenceV1` (re-exported from `common/src/types/task-memory.ts`) plus `TaskMemoryStoreFs`, `WorkspaceMoveRecord`, `TaskMemoryPruneOutcome`, `V1MigrationAuditReader`, and `V1MigrationAuditOutcome`. The audit API is additive and read-only: it verifies a loaded V1 source revision/checksum against the V2 migration marker, imported body, and provenance without appending. `exact` is lossless only when `omittedFields`, `truncatedFields` (default zero), and warnings are all zero; bounded outcomes separately represent no record, not migrated, incomplete, mismatch, rejected, and failed. External consumers must be able to _name_ every parameter/return type of a published function, so adding a task-memory export means publishing its types from `sdk/src/index.ts` in the same change. +- **Published task-memory API (`@openbuff/sdk`)**: the SDK publishes `loadPersistedTaskMemory`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, `auditTaskMemoryV1Migration`, and `codebuffFsToNodePromises`, together with the complete type closure their signatures name — `TaskMemoryV1`, `TaskMemoryDraftV1`, `TaskMemoryEvidenceV1` (re-exported from `common/src/types/task-memory.ts`) plus `TaskMemoryStoreFs`, `WorkspaceMoveRecord`, `TaskMemoryPruneOutcome`, `V1MigrationAuditReader`, and `V1MigrationAuditOutcome`. The audit API is additive and read-only: it verifies source revision/checksum, marker identity, imported task existence, imported observation IDs, and migration provenance without appending; it does not reconstruct and compare every source-derived body field. `exact` is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`; `not-migrated` means removal is not ready and is not a defect. Bounded outcomes separately represent no record, not migrated, incomplete, mismatch, rejected, and failed. Keep the audit API/types as transitional read-only contracts through rollback, and follow the normative [Memory V1 removal readiness plan](../docs/memory-v1-removal-readiness.md) before any later removal decision. External consumers must be able to _name_ every parameter/return type of a published function, so adding a task-memory export means publishing its types from `sdk/src/index.ts` in the same change. - **task-memory.json writers**: `.openbuff/memory/task-memory.json` has two writers — `saveMergedTaskMemory` (end of run) and `pruneStaleTaskMemoryEvidence` (`/memory prune`) — and its `revision` must stay monotonic and unique across both. Uniqueness is enforced by serializing each writer's load→revision→commit section on a shared lock: an in-process promise chain keyed by the resolved record path, plus an exclusive-create `.lock` file for other processes that is reclaimed after a staleness timeout. Without it a save landing between prune's revision check and its rename, or two saves loading the same record, published one revision twice with different payloads. Evidence reconciliation deliberately stays outside the locked section because it hashes every evidence file. The lock is advisory — adapters that ignore exclusive-create, or a lock not obtained within the bounded attempt budget, degrade to running unlocked — so both writers keep their own guards: saves re-read the on-disk record and merge against whichever of it and the caller's hydrated `priorMemory` is newer, emitting one past both, and prune reports `status: 'failed', reason: 'concurrent-write'` when the record moved off the revision it loaded. - **Pruned evidence must not come back**: a revision guard prevents revision reuse, not content resurrection. `/memory prune` runs from inside a live session whose `runMemory.evidence` still holds the entries it dropped, so `saveMergedTaskMemory` also filters the run's contribution: when the on-disk record is the merge base, every evidence id the caller's hydrated snapshot carries but that record no longer does was deliberately removed by the other writer and is dropped from the run's evidence. Evidence the run recorded itself is untouched. - **Prune honors the move contract**: `pruneStaleTaskMemoryEvidence` takes the same optional `workspaceMoves` as `reconcileTaskMemoryEvidence`, because prune DELETES what reconciles stale. Callers that can see journal-recorded moves must pass them, or evidence bound to a renamed file reconciles stale and is permanently lost instead of rebinding to its destination as hydration documents. `collectWorkspaceMoves` is published from `@openbuff/sdk` for exactly that reason: non-run callers need the same journal-derived move set hydration uses. -- **`/memory` CLI command** (`cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts` as `memory`, alias `mem`): `/memory status` summarizes the persisted record and its fresh/stale evidence; `/memory prune` drops stale evidence; `/memory audit-migration` invokes the SDK audit against the currently loaded V1 record and leased V2 repository without mutation. Both status/prune subcommands read journal moves via `collectWorkspaceMoves` + `WorkspaceJournalService` and pass them into reconciliation/prune, so a renamed file's evidence rebinds instead of being reported stale and deleted; a journal read that fails surfaces as a command failure rather than a blind prune. Prune reports the store's `TaskMemoryPruneOutcome` faithfully — `no-record` (nothing persisted yet), `pruned` (including `removed: 0`, "nothing to prune"), and `failed` with its reason (schema reject, lost revision race, unwritable target or rename-less filesystem adapter). A failed write is never phrased as an absent record or as "nothing to prune"; it names the cause and reports the stale entries still present. Release N keeps `json-v1` and `shadow-v2` supported but deprecates them in favor of the default/replacement `sqlite-v2-opt-in`; authority selection and compatibility/fallback behavior remain unchanged, with no removal date implied and any future removal contingent on a later decision plus successful exact audits. +- **`/memory` CLI command** (`cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts` as `memory`, alias `mem`): `/memory status` summarizes the persisted record and its fresh/stale evidence; `/memory prune` drops stale evidence; `/memory audit-migration` invokes the SDK audit against the currently loaded V1 record and leased V2 repository without mutation. Both status/prune subcommands read journal moves via `collectWorkspaceMoves` + `WorkspaceJournalService` and pass them into reconciliation/prune, so a renamed file's evidence rebinds instead of being reported stale and deleted; a journal read that fails surfaces as a command failure rather than a blind prune. Prune reports the store's `TaskMemoryPruneOutcome` faithfully — `no-record` (nothing persisted yet), `pruned` (including `removed: 0`, "nothing to prune"), and `failed` with its reason (schema reject, lost revision race, unwritable target or rename-less filesystem adapter). A failed write is never phrased as an absent record or as "nothing to prune"; it names the cause and reports the stale entries still present. Release N keeps `json-v1` and `shadow-v2` supported but deprecates them in favor of the default/replacement `sqlite-v2-opt-in`; authority selection and compatibility/fallback behavior remain unchanged, with no removal date implied and migration not claimed complete. Any future removal requires a later decision after the canonical readiness gates pass, not merely a current `exact` audit. - **Stable hash (`common/src/util/stable-hash.ts`)**: canonical FNV-1a 32-bit string hash rendered as padded 8-hex. It is the one implementation shared by agent-runtime `commitTaskMemory` checksums, the SDK task-memory store's persisted-record checksum, and the `git_status` change-gate fingerprint; byte-format vectors (`'' -> 811c9dc5`) are pinned by tests because the same digest must round-trip across packages. - **Gate telemetry sink (`common/src/util/gate-telemetry.ts`)**: append-only JSONL sink for base2's `emitGateTelemetry` payloads under `/.openbuff/telemetry/base2-gate.jsonl`, so per-round gate evidence outlives a single turn's `console.info` and questions like "what share of repair rounds were nit-driven" become mechanically answerable. Best-effort by contract: it never throws (not even when `logger.warn` itself throws), reports failure through its boolean return, and keeps owner-only modes (`0o700` directory, `0o600` file) like the SDK task-memory store. Lines are size-bounded in three escalating steps — array-item caps, per-field byte caps, then an event-name-only marker — and `truncated` / `truncatedFields` / `droppedPayloadKeys` are written by the SINK, so a payload field of one of those names is dropped rather than merged and cannot forge the truncation signal. Rotation keeps at most one `.1` generation and deliberately accepts two documented races; `GATE_TELEMETRY_MAX_BYTES` is the authoritative note for both. `errorCode` in `common/src/util/error.ts` is the shared helper it keys `ENOENT` on, and `common/src/testing/gate-telemetry-fixtures.ts` owns the temp-root helpers plus the platform skip policy shared with the agent-runtime suite (a skip rule that drifts between the two silently stops covering one). - **On-demand guide fallback (`common/src/util/guides.ts`)**: base2's progressive prompt disclosure replaces each verbose advisory section with a compact pointer naming a workspace-relative `agents/guides/*.md` path, which `read_files` resolves against the EMBEDDER's project root — and no publish pipeline ships the guides, so in any workspace other than this repo the pointer read fails and the model loses the whole section. `FALLBACK_GUIDES` is the single copy of those paths (base2's `GUIDE_PATHS` is an alias of it and its `GuidePath` union is derived from it); the table lives here rather than beside base2's pointers because `packages/agent-runtime` must not import from `agents/`. `findMissingGuides` returns `[]` for a falsy or non-string root — "unknown root" must never mean "everything is missing", or every prompt formatted without a real workspace regrows by six full sections — and `formatGuideFallbackSection` returns `''` for a guide that is present, so the resolved in-repo prompt stays byte-identical. Recovery is PER POINTER and clause-keyed: a mode that deliberately omits a pointer (plan mode is read-only and emits neither the git-discipline pointer nor its body) never recovers that body, and `BROAD_AUDIT_FALLBACK_SECTIONS` keys the one clause-parameterized body by finalize clause so a guide-less plan-mode prompt cannot receive the contradictory implementation clause. diff --git a/docs/memory-v1-removal-readiness.md b/docs/memory-v1-removal-readiness.md new file mode 100644 index 0000000000..c3bc2c0b01 --- /dev/null +++ b/docs/memory-v1-removal-readiness.md @@ -0,0 +1,243 @@ +# Memory V1 Removal Readiness Plan + +## Non-claims and current status + +This is the canonical readiness plan for a later Memory V1 removal decision. It is not removal approval. + +- Release N retains V1 compatibility. `json-v1`, `shadow-v2`, V1 persistence, migration, fallback behavior, and the published V1 APIs remain supported as currently documented. +- No removal date or version is set. Removal, if any, requires a later release decision after every gate below passes. +- Migration is not claimed complete. Current evidence is insufficient to authorize removal. +- `/memory audit-migration` outcome `not-migrated` means removal is **not ready** for that project. It reports migration state and is not a product defect. +- Outcome `exact` currently means marker identity, imported task existence, imported observation IDs, and migration provenance were verified. It does not mean every source-derived task or observation body field was reconstructed and compared. +- Even `exact` is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`. Warnings are never waived. + +Normative terms such as **MUST**, **MUST NOT**, and **SHOULD** define acceptance criteria for a later removal decision. + +## Readiness definition + +Memory V1 is removal-ready only when all in-scope projects and supported deployment paths have mechanically reviewable evidence that removing V1 will not lose data, misstate authority, strand mixed-version clients, or prevent rollback. Before any V1 behavior or API is removed: + +1. Every source blocker below **MUST** be closed in implementation and tests. +2. Every in-scope project **MUST** have the per-project evidence bundle below. A project with `not-migrated`, lossy `exact`, an ambiguous `no-record`, incomplete evidence, or a changed V1 revision/checksum is a no-go. +3. The compatibility window and every staged rollout gate **MUST** complete without an unresolved rollback trigger. +4. Storage, restore, mixed-version, public-consumer, operator, telemetry/support, and ownership criteria **MUST** be signed off. +5. `auditTaskMemoryV1Migration`, `V1MigrationAuditReader`, `V1MigrationAuditOutcome`, and the related audit types **MUST** remain supported transitional read-only contracts until the rollback window closes, unless an equivalent supported diagnostic is available first. + +## Current removal-readiness blockers + +These blockers are grounded in the current source and make the present answer **not ready**: + +1. **Body equality is not proved.** `auditMigrationMarkerBody` in `sdk/src/services/memory-v2/v1-migration.ts` verifies marker identity, imported task existence, imported observation IDs, task association, migration provenance, and the `legacy-v1` tag. It does not reconstruct the import from the V1 source and compare every source-derived task/observation body field. A future audit **MUST** perform that stronger deterministic comparison, with tests that tamper with each imported body field and prove the audit fails closed, before `exact` can authorize removal. +2. **`no-record` is ambiguous.** `loadPersistedTaskMemory` in `sdk/src/services/task-memory-store.ts` returns `undefined` for absent, invalid, unreadable, and checksum-failed records. Therefore `no-record` alone cannot prove that no V1 data exists. Readiness requires a future non-mutating storage inspection result that distinguishes at least `absent`, `valid`, `invalid`, and `unreadable`; only independently verified `absent` can satisfy absence evidence. +3. **The importer deliberately loses or excludes some data.** `importTaskMemoryV1` warns or omits data including a non-empty goal (`goal-excluded`) and legacy evidence (`legacy-evidence-unverified`), and can omit stale/unsafe/empty/capped data or truncate text. Any warning, omission, or truncation is lossy and cannot authorize removal. Common legitimate warning cases do not create an exception; warnings **MUST NOT** be waived. +4. **Interactive audit is bounded.** `scanV1MigrationAuditEvents` scans at most 10 export pages. `/memory audit-migration` therefore cannot be the sole release-evidence path for repositories beyond that bound. Release evidence requires a full or resumable, schema-validated audit that validates pagination and reaches the canonical end of every in-scope repository. +5. **Post-V1 authority failure semantics are undecided.** Before removal, behavior for legacy or invalid authority values and for V2-open/reset failure **MUST** be explicitly specified, tested, and communicated. It **MUST** fail closed and must not silently report successful V2 authority when canonical V2 storage did not open. The decision must cover `OPENBUFF_MEMORY_AUTHORITY`, SDK authority inputs, provider result types, CLI reporting, and persisted/session authority state. + +## Per-project evidence bundle + +For each project in the release scope, retain a reviewable bundle outside this canonical policy document. Do not paste local machine paths, local source revisions/checksums, or artifact hashes into this file. The bundle **MUST** contain: + +- project identity and evidence collection time, tool/release identity, and the declared release-scope owner; +- the non-mutating V1 inspection result (`valid` or independently proven `absent`; never inferred from `no-record`); +- for a valid record, the checksum-verified V1 backup and its recorded revision/checksum in the controlled evidence system; +- import outcome and a fresh full/resumable audit against that same V1 revision/checksum; +- an `exact` result from the future full-body-comparison audit, with `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`; +- a validated canonical V2 export, including validated manifest/event schemas, project identity, pagination continuity, and terminal cursor/end-of-export evidence; +- restore-drill evidence from a disposable copy, plus read/query verification after restore; +- mixed-version test results, authority/degradation diagnostics, and operator acknowledgement of any retry or remediation; +- evidence-review signoff and an explicit pass/fail disposition for every checklist item. + +Any stale, partial, unparsable, mismatched, `not-migrated`, `incomplete`, `mismatch`, `rejected`, or `failed` evidence blocks that project. + +## Compatibility window + +Release N is the deprecation release and keeps compatibility. During the compatibility window: + +- V1 read/write persistence, import, shadow/fallback behavior, authority values, CLI guidance, and public SDK/common contracts remain available. +- Deprecation notices direct operators to collect evidence; they do not claim migration completion or imply a removal date/version. +- A prior compatible artifact and configuration **MUST** remain available and tested for rollback. +- The window ends only through a later release decision after all gates and signoffs pass. It is not ended merely by low observed V1 use or by an `exact` result from the current bounded audit. + +## Staged rollout gates + +Each gate is blocking and advances only with recorded evidence and owner approval. No numeric threshold or fixed hold duration is defined here; release owners must judge the scoped evidence and unresolved risk rather than retrofit an arbitrary percentage. + +### Gate 0: blocker closure + +- [ ] Full source-derived task/observation body comparison is implemented and tampered-body tests fail closed. +- [ ] Non-mutating V1 inspection distinguishes `absent`, `valid`, `invalid`, and `unreadable`. +- [ ] A full/resumable validated audit supports stores beyond the interactive 10-page bound. +- [ ] Post-V1 authority and V2-open/reset failure semantics are approved and fail closed. +- [ ] Lossless evaluation uses the exact zero-omission/zero-truncation/zero-warning predicate. + +### Gate 1: evidence tooling and rehearsal + +- [ ] Evidence tooling is read-only except for an explicit import operation and produces the complete per-project bundle. +- [ ] Backup, canonical V2 export, disposable restore, re-import, and rollback procedures pass in supported storage environments. +- [ ] Unit, contract, integration, storage, consumer, and release validation below pass. + +### Gate 2: compatibility-window observation + +- [ ] All in-scope projects have fresh passing bundles. +- [ ] Telemetry and support review finds no unresolved V1 dependency, data-loss signal, authority misreporting, or rollback blocker. +- [ ] Mixed-version operation and rollback remain supported with both stores preserved. + +### Gate 3: reversible removal candidate + +- [ ] A candidate artifact can disable the proposed V1 surfaces without deleting either store or preventing re-enable/rollback. +- [ ] Upgrade, downgrade, mixed-version, invalid-authority, V2-open failure, and restored-data scenarios pass. +- [ ] Public API and CLI removal inventories have consumer migration evidence and release-note review. + +### Gate 4: later removal decision + +- [ ] Every go/no-go item is checked and required owners have signed off. +- [ ] No rollback trigger is open. +- [ ] The decision explicitly identifies the surfaces approved for removal; unlisted V1 behavior remains supported. +- [ ] The rollback window and criteria are declared for that release while transitional diagnostics remain supported. + +## Public SDK/common removal inventory + +A later decision **MUST** explicitly inventory and assess, rather than implicitly delete, at least: + +- SDK task-memory functions: `loadPersistedTaskMemory`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, and `codebuffFsToNodePromises`; +- SDK migration functions: `importTaskMemoryV1`, `getV1MigrationIdentity`, and the transitional read-only `auditTaskMemoryV1Migration`; +- SDK types: `TaskMemoryStoreFs`, `WorkspaceMoveRecord`, `TaskMemoryPruneOutcome`, `V1MigrationOutcome`, `V1MigrationWarningCode`, `V1MigrationSourceItemCounts`, `V1MigrationAuditReader`, and `V1MigrationAuditOutcome`; +- common schemas/constants/types: `TASK_MEMORY_LIST_CAPS`, `taskMemoryEvidenceV1Schema`, `taskMemoryDraftV1Schema`, `taskMemoryV1Schema`, `TaskMemoryEvidenceV1`, `TaskMemoryDraftV1`, and `TaskMemoryV1`; +- common session/checkpoint contracts that carry V1 import, compatibility-shadow, parity, warning, or authority state, including `MemoryV1ImportWarningCode`, `MemoryV1ImportState`, and `MemoryAuthorityStateV2`; +- persisted `.openbuff/memory/task-memory.json` semantics and any runtime hydration, reconciliation, save, prune, shadow, fallback, or migration callers. + +For each item, record public/internal status, known consumers, replacement, compatibility impact, test coverage, documentation, and rollback behavior. Audit contracts follow the longer transitional lifetime stated above. + +## CLI/env/provider removal inventory + +A later decision **MUST** separately inventory: + +- `/memory status`, `/memory prune`, and `/memory audit-migration`, including all outcome and warning guidance; +- `OPENBUFF_MEMORY_AUTHORITY`, `getMemoryAuthoritySelection`, and authority values `json-v1`, `shadow-v2`, and `sqlite-v2-opt-in`; +- invalid/legacy authority handling, the default authority, and SDK-supplied authority options; +- `ProjectMemoryV2Provider`, `MemoryV2ProviderResult`, effective/requested authority reporting, and storage-unavailable/reset-during-open degradation; +- CLI status/diagnose/authority output, V1 compatibility-shadow reporting, V2 export/import, and operator recovery instructions; +- packaged configuration, environment documentation, release notes, scripts, and external automation that set or parse these values. + +Removal readiness requires tested replacements and fail-closed reporting for every inventoried path. + +## Storage, backup, and no-deletion rules + +Before a project can pass readiness: + +- Create and retain a checksum-verified V1 backup before import or authority changes. +- Produce and validate a canonical V2 export as described in the evidence bundle. +- Perform a restore drill on a disposable copy and validate restored reads/queries; never rehearse destructively on the only copy. +- Preserve both V1 and V2 stores, their backups, and the prior compatible artifact/configuration throughout rollout and the rollback window. +- **MUST NOT** automatically delete V1 data, automatically clean up either store, or migrate in place. Import/copy to V2 and verify independently. +- Destructive cleanup, if ever proposed after the rollback window, requires a separate explicit decision, fresh backups, operator-visible scope, and its own rollback analysis. It is not authorized by this plan. + +## Mixed-version and rollback criteria + +Older compatible clients may continue to revise V1 while newer clients use V2. Therefore: + +- After any V1 revision or checksum change, the prior migration evidence is stale: re-import and collect a fresh lossless full audit before proceeding. +- Mixed-version tests **MUST** cover old-writer/new-reader, new-writer/old-reader, concurrent/sequential use, upgrade, downgrade, and interrupted V2 open/import. +- Rollback **MUST** restore the prior compatible artifact and configuration without deleting or overwriting either store. Both stores remain intact for diagnosis and a later retry. +- A rollback is complete only when authority reporting matches actual storage use, the V1 backup still verifies, V1 operation is usable where expected, and no post-backup V1 revision is silently discarded. +- `auditTaskMemoryV1Migration` and its audit types remain available through the rollback window unless an equivalent supported read-only diagnostic exists. + +## Telemetry and support review + +Use privacy-appropriate, bounded signals already available to the product; do not add project contents, local paths, checksums, or raw repository errors to telemetry. Review by release scope and supported deployment path: + +- requested versus effective authority, fallback/degradation, V2 open/reset failures, and invalid/legacy authority selection; +- migration outcomes and warning/omission/truncation presence, without treating `not-migrated` as a defect; +- audit incompleteness/mismatch/failure and page-bound encounters; +- restore, rollback, data-loss, stale-evidence, and mixed-version support cases; +- external consumer reports concerning deprecated SDK/common/CLI contracts. + +No numeric threshold, support-volume limit, percentage, or hold duration is prescribed here. Owners must document the reviewed evidence, unresolved cases, and why the observed record is sufficient for the scoped decision. Any credible unresolved data loss, false authority claim, inability to restore, or required V1 dependency is a no-go. + +## Decision ownership and signoffs + +Roles may be filled according to the release process; this plan does not name individuals. The release record **MUST** include approval from: + +- the Memory implementation owner, for blocker closure and semantic correctness; +- the storage/reliability owner, for backup, canonical export, restore, mixed-version, and rollback evidence; +- SDK/common and CLI maintainers, for public contract and operator inventory; +- test/release engineering, for the validation matrix and artifact reproducibility; +- support/operations, for migration guidance, observed cases, and rollback readiness; +- the release decision owner, for final scope, accepted residual risk, rollback window, and go/no-go disposition. + +A signoff identifies its evidence bundle and scope; silence or a general release approval is not a Memory V1 removal signoff. + +## Rollback triggers + +Stop rollout and restore the prior compatible artifact/configuration when any of the following occurs: + +- V1 inspection is ambiguous, invalid, unreadable, or disagrees with the retained backup; +- audit is not fresh and lossless, any warning/omission/truncation appears, or a source-derived body tamper is not detected; +- a V1 revision/checksum changes without re-import and fresh audit; +- canonical V2 export, restore, pagination, integrity, or query validation fails; +- requested and effective authority diverge without an explicit fail-closed degradation, or V2-open failure is reported as successful V2 authority; +- a supported mixed-version, upgrade, downgrade, or consumer path requires a removed surface; +- either store, backup, prior artifact/configuration, or transitional diagnostic is unavailable during the rollback window; +- telemetry/support review identifies credible data loss, inaccessible memory, or an unresolved rollback blocker. + +## Validation matrix + +### Unit and contract + +- Import mapping and lossless predicate tests, including every warning/omission/truncation code. +- Full reconstruction/equality tests for every source-derived task and observation field; tamper each field and require fail-closed non-`exact` results. +- V1 inspector tests for `absent`, `valid`, `invalid`, `unreadable`, and checksum failure. +- Audit outcome, schema, project-binding, duplicate-ID, pagination, terminal-cursor, and resumability tests. +- Authority parsing/provider-result tests for current, legacy, invalid, unavailable, reset, and thrown-open paths; reported authority must equal actual authority. +- Public type/export contract tests, retaining transitional audit contracts. + +### Integration and storage + +- Checksum-verified V1 backup, import, full canonical V2 export, disposable restore, and restored query/read validation. +- Stores larger than the interactive 10-page audit bound, malformed pages, interrupted/resumed scans, and repository failure. +- Filesystem permission/unreadable/invalid/checksum-failed cases without mutation. +- Both stores preserved across upgrade, downgrade, rollback, failed import, and failed V2 open. +- V1 revision/checksum change followed by mandatory re-import and fresh lossless audit. + +### Consumer and release + +- Published SDK/common API inventory and representative external-consumer compile/contract validation. +- CLI commands, environment/config parsing, diagnostics, operator messages, package upgrade/downgrade, and prior-artifact rollback. +- Release artifact smoke tests for each supported deployment/storage path. +- Release notes state Release N compatibility, no set removal date/version, no migration-complete claim, and the later-decision requirement. +- Evidence bundles and signoffs are reviewable without embedding local paths or secret/raw project content in committed documentation. + +## Operator migration guidance + +1. Keep a known-compatible artifact and the current authority configuration available. +2. Inspect V1 non-mutatingly. If the result is `invalid` or `unreadable`, stop and repair access/record handling; do not interpret it as absence. Until the future inspector exists, treat `no-record` as inconclusive. +3. Create and verify a V1 backup. Do not migrate in place or delete the source. +4. Import to V2 using the supported path, then run the full/resumable audit. `/memory audit-migration` remains useful interactively but its 10-page bound cannot be sole evidence for larger stores. +5. Stop on `not-migrated`, `incomplete`, `mismatch`, `rejected`, or `failed`. `not-migrated` means readiness work remains; it is not itself a product defect. +6. For `exact`, require `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`, plus the future full source-derived body comparison. Do not waive `goal-excluded`, `legacy-evidence-unverified`, or any other warning. +7. Validate and retain a canonical V2 export, then restore and test it on a disposable copy. +8. If V1 revision/checksum changes, re-import and repeat the fresh lossless audit and export validation. +9. Preserve both stores until the declared rollback window closes. On a trigger, restore the prior artifact/configuration; do not delete either store. + +## Explicit go/no-go checklist + +A later removal decision is **GO** only when every applicable item is checked: + +- [ ] This is a later release decision; Release N was compatibility-only, and the proposed scope/date is explicit in that later record. +- [ ] Migration is not assumed complete; every in-scope project has a fresh per-project evidence bundle. +- [ ] All five current blockers are closed with reviewed implementation and tests. +- [ ] Every valid V1 record has full-body `exact` evidence and the exact lossless predicate passes with no warnings waived. +- [ ] Every claimed absent V1 record is independently inspected as `absent`; no decision relies on `no-record` alone. +- [ ] No project has `not-migrated`, stale, incomplete, mismatched, rejected, failed, lossy, invalid, unreadable, or partial evidence. +- [ ] Repositories beyond 10 pages were audited through the full/resumable validated path. +- [ ] Checksum-verified V1 backups, validated canonical V2 exports, and disposable restore drills pass. +- [ ] Both stores and a prior compatible artifact/configuration will remain intact through rollback; no automatic deletion or migrate-in-place is enabled. +- [ ] Every V1 revision/checksum change was followed by re-import and a fresh lossless audit. +- [ ] Mixed-version, authority, V2-open failure, upgrade/downgrade, rollback, and consumer validation pass fail-closed. +- [ ] Public SDK/common and CLI/env/provider inventories have replacements, consumer evidence, documentation, and rollback coverage. +- [ ] Transitional `auditTaskMemoryV1Migration` contracts remain supported through rollback or an equivalent diagnostic is available. +- [ ] Telemetry/support review has no unresolved removal blocker and invents no threshold to hide one. +- [ ] Required role-based signoffs reference the scoped evidence. +- [ ] No rollback trigger is open. + +Any unchecked item is **NO-GO**. A no-go preserves current compatibility and sets no removal date/version. diff --git a/sdk/CHANGELOG.md b/sdk/CHANGELOG.md index 47683953ed..238023117a 100644 --- a/sdk/CHANGELOG.md +++ b/sdk/CHANGELOG.md @@ -6,8 +6,8 @@ All notable changes to the @openbuff/sdk package will be documented in this file ### Added -- New additive, read-only `auditTaskMemoryV1Migration` API and `V1MigrationAuditOutcome` type inspect the currently loaded V1 record against a V2 repository without appending or mutating. An `exact` result verifies the source revision/checksum, migration marker, imported task/observation bodies, and migration provenance; it is lossless only when `omittedFields`, `truncatedFields` (defaulting to zero when absent), and `warnings` are all zero. Other outcomes distinguish no record, not migrated, incomplete, mismatch, rejected, and failed states using bounded reason codes rather than repository error text. -- Release N deprecation guidance: `json-v1` and `shadow-v2` remain supported compatibility modes but are deprecated; `sqlite-v2-opt-in` is the default and replacement. This release does not remove V1 APIs or persistence, migration import, authority selection, shadow behavior, or fallback behavior. Any later removal requires a separate release decision and successful exact migration audits. +- New additive, read-only `auditTaskMemoryV1Migration` API and `V1MigrationAuditOutcome` type inspect the currently loaded V1 record against a V2 repository without appending or mutating. An `exact` result verifies the source revision/checksum, marker identity, imported task existence, imported observation IDs, and migration provenance; it does not reconstruct and compare every source-derived body field. It is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`. Other outcomes distinguish no record, not migrated, incomplete, mismatch, rejected, and failed states using bounded reason codes rather than repository error text. +- Release N deprecation guidance: `json-v1` and `shadow-v2` remain supported compatibility modes but are deprecated; `sqlite-v2-opt-in` is the default and replacement. This release does not remove V1 APIs or persistence, migration import, authority selection, shadow behavior, or fallback behavior, sets no removal date/version, and does not claim migration is complete. Any removal requires a later release decision after the normative gates in the [Memory V1 removal readiness plan](../docs/memory-v1-removal-readiness.md) pass; `not-migrated` means removal is not ready, not that the product is defective. - New published task-memory API for durable cross-session task state: `loadPersistedTaskMemory`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, and `codebuffFsToNodePromises`, backed by `.openbuff/memory/task-memory.json` under the project root. The complete type closure those signatures name is published alongside them so consumers can name every parameter and return type: `TaskMemoryV1`, `TaskMemoryDraftV1`, and `TaskMemoryEvidenceV1` (re-exported from `@codebuff/common/types/task-memory`), plus `TaskMemoryStoreFs`, `WorkspaceMoveRecord`, and `TaskMemoryPruneOutcome`. - `pruneStaleTaskMemoryEvidence` returns a discriminated `TaskMemoryPruneOutcome` rather than a nullable count pair, so "there is no record" (`status: 'no-record'`) is distinguishable from "the prune could not be committed" (`status: 'failed'`, with `reason: 'invalid-record' | 'concurrent-write' | 'write-failed'` and the `removed`/`remaining` counts the prune would have written). A successful prune is `status: 'pruned'`, including the no-op `removed: 0` case for a fully fresh record. From 2946c98c61635f622468f24c456965fa7f9ca9a2 Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Sun, 13 Sep 2026 17:48:46 +0300 Subject: [PATCH 03/22] feat(memory): harden V1 evidence handling and migration audit Deprecate json-v1 and shadow-v2 in favor of sqlite-v2-opt-in with explicit no-removal-date guidance, and make /memory audit-migration read-only and inspection-first so absent, invalid, and unreadable V1 records are reported distinctly before any V2 acquisition. Reconcile V1 evidence against descriptor-bound versioned hashes with containment checks and fail-closed, token-owned cross-process locking so pruning never acts on unverifiable or inherited stale verdicts. Validate V1->V2 migration deterministically by reconstructing every source-derived task and observation event and comparing full normalized headers and payloads, closing the first two Gate 0 removal-readiness blockers documented in docs/memory-v1-removal-readiness.md. --- cli/knowledge.md | 2 +- .../commands/__tests__/memory-command.test.ts | 652 ++++++-- cli/src/commands/memory-command.ts | 1058 +++++++++---- common/knowledge.md | 4 +- docs/memory-v1-removal-readiness.md | 22 +- sdk/CHANGELOG.md | 3 +- sdk/src/__tests__/memory-v2-contract.test.ts | 110 +- sdk/src/__tests__/task-memory-store.test.ts | 664 ++++++++- sdk/src/index.ts | 2 + .../memory-v2/__tests__/v1-migration.test.ts | 707 ++++++++- sdk/src/services/memory-v2/v1-migration.ts | 1324 ++++++++++++----- sdk/src/services/task-memory-store.ts | 617 +++++--- 12 files changed, 4001 insertions(+), 1164 deletions(-) diff --git a/cli/knowledge.md b/cli/knowledge.md index 26ee700b14..92ece01b49 100644 --- a/cli/knowledge.md +++ b/cli/knowledge.md @@ -23,7 +23,7 @@ - Durable planning is entered through `mode:plan`; the standalone `/plan` command is intentionally absent from `COMMAND_REGISTRY` and `SLASH_COMMANDS` so there is one plan-entry path. - Keep the durable-plan quartet registered: `/resume-plan` (`rp`), `/update-plan` (`up`), `/plan-status` (`ps`), and `/lessons` (`lesson`). These commands operate on `.agents/sessions//` artifacts and fall back to the plan-session picker when no target is provided. - `/plans` (`plan-ls`) lists the artifact-bearing sessions returned by `listPlanSessions()` and never prompts the agent, while `/plan-use` (`plan-active`, `use-plan`) writes the `ACTIVE_SESSION` pointer file under `.agents/` and only accepts a bare slug or `.agents/sessions/` — the resolved directory must be exactly one segment under `.agents/sessions/` because the pointer stores bare slugs, so nested paths, non-session paths, missing directories, and artifact-less directories are all rejected before any write. Both share `formatPlanSessionListRow` / `PLAN_SESSIONS_DIR_PREFIX` from `cli/src/commands/plan-artifacts.ts` so the rendered box and the text report cannot drift. -- `/memory` (alias `/mem`, `cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts`) inspects the persisted cross-session task memory for the current project (`.openbuff/memory/task-memory.json`). `/memory status` (the default) reports the record's revision and age, its goal and per-list counts, and how much of its evidence still verifies against disk, listing up to five stale paths; `/memory prune` drops evidence that no longer verifies. Both subcommands are move-aware via `collectWorkspaceMoves` + `WorkspaceJournalService` so a renamed file's evidence rebinds rather than being reported stale and deleted. `/memory audit-migration` is dedicated and read-only: it passes the currently loaded V1 record and leased V2 repository to the SDK `auditTaskMemoryV1Migration` API, never uses the CLI canonical-inventory marker scan, and never appends. An `exact` audit verifies source revision/checksum, marker identity, imported task existence, imported observation IDs, and migration provenance, not full source-derived body equality; it is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`. `not-migrated` means removal is not ready and is not a product defect; all outcomes are rendered distinctly with bounded reason text. +- `/memory` (alias `/mem`, `cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts`) inspects the persisted cross-session task memory for the current project (`.openbuff/memory/task-memory.json`). `/memory status` (the default) reports the record's revision and age, its goal and per-list counts, and how much of its evidence still verifies against disk, listing up to five stale paths; `/memory prune` drops evidence that no longer verifies. Both subcommands are move-aware via `collectWorkspaceMoves` + `WorkspaceJournalService` so a renamed file's evidence rebinds rather than being reported stale and deleted. `/memory audit-migration` is dedicated and read-only: before acquiring V2 it calls `inspectPersistedTaskMemoryV1`, blocks absent/invalid/unreadable states with bounded distinct reports, and passes the exact checksum-verified memory to the SDK audit without a second read. An `exact` audit reconstructs and compares every deterministic source-derived task/observation header and payload plus marker metadata; it is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`. The legacy loader still collapses non-valid inspection states to `undefined` for compatibility. `not-migrated` means removal is not ready and is not a product defect; all outcomes are rendered distinctly with bounded reason text. - Slash-command descriptions should stay model-agnostic under BYOK/local mode. Use wording such as "configured reviewer" rather than naming hosted models. ## Import Guidelines diff --git a/cli/src/commands/__tests__/memory-command.test.ts b/cli/src/commands/__tests__/memory-command.test.ts index f422dfccfc..7390d03736 100644 --- a/cli/src/commands/__tests__/memory-command.test.ts +++ b/cli/src/commands/__tests__/memory-command.test.ts @@ -89,6 +89,10 @@ function createDeps(options: { if (options.loadThrows) throw new Error('load exploded') return options.memory }, + inspectPersistedTaskMemoryV1: async () => + options.memory + ? { status: 'valid' as const, memory: options.memory } + : { status: 'absent' as const }, reconcileTaskMemoryEvidence: async (params: { workspaceMoves?: WorkspaceMoveRecord[] }) => { @@ -122,6 +126,22 @@ describe('/memory command', () => { expect(result).toContain('first successful run') }) + test('legacy typed dependency injectors may omit the V1 inspector', async () => { + const { deps } = createDeps({ memory: undefined }) + const { + inspectPersistedTaskMemoryV1: _newInspector, + ...legacyDependencies + } = deps + const compatibleDependencies: MemoryCommandDeps = legacyDependencies + + const result = await handleMemoryCommand( + 'audit-migration', + compatibleDependencies, + ) + + expect(result).toContain('Outcome: absent') + }) + test('status summarizes counts and lists stale evidence paths', async () => { const memory = makeMemory({ updatedAt: Date.now() - 2 * 60 * 60 * 1_000, @@ -344,14 +364,19 @@ describe('/memory command', () => { test('authority reports valid values and current selection without mutation', async () => { const { deps } = createDeps({ memory: makeMemory() }) deps.getMemoryV2 = async () => ({ - status: 'unavailable', requestedAuthority: 'sqlite-v2-opt-in', - effectiveAuthority: 'json-v1', degradation: 'fallback', retryable: true, + status: 'unavailable', + requestedAuthority: 'sqlite-v2-opt-in', + effectiveAuthority: 'json-v1', + degradation: 'fallback', + retryable: true, }) const result = await handleMemoryCommand('authority', deps) expect(result).toContain('json-v1, shadow-v2, sqlite-v2-opt-in') expect(result).toContain('Requested: sqlite-v2-opt-in; effective: json-v1') expect(result).toContain('Release N') - expect(result).toContain('json-v1 and shadow-v2 remain supported but are deprecated') + expect(result).toContain( + 'json-v1 and shadow-v2 remain supported but are deprecated', + ) expect(result).toContain('sqlite-v2-opt-in is the default and replacement') expect(result).toContain('reset/restart') }) @@ -393,13 +418,27 @@ describe('/memory blocks', () => { return { outcome: 'preview', plannedEvents: [] } } const observation = { - observationId: 'observation-1', taskId: 'task-1', kind: 'discovery', summary: 'old', detail: 'old', - confidence: 1, selectors: [], evidence: [], tags: [], observedAt: '2025-01-01T00:00:00.000Z', + observationId: 'observation-1', + taskId: 'task-1', + kind: 'discovery', + summary: 'old', + detail: 'old', + confidence: 1, + selectors: [], + evidence: [], + tags: [], + observedAt: '2025-01-01T00:00:00.000Z', } const observationEvent = MemoryEventEnvelopeSchema.parse({ - schemaVersion: 2, eventSchemaVersion: 1, eventType: 'observation.recorded', - eventId: 'observation-event-1', projectId: 'project-1', sessionId: 'session-1', sequence: 1, - occurredAt: '2025-01-01T00:00:00.000Z', payload: { payloadSchemaVersion: 1, observation }, + schemaVersion: 2, + eventSchemaVersion: 1, + eventType: 'observation.recorded', + eventId: 'observation-event-1', + projectId: 'project-1', + sessionId: 'session-1', + sequence: 1, + occurredAt: '2025-01-01T00:00:00.000Z', + payload: { payloadSchemaVersion: 1, observation }, }) const operator = { consolidate: invoke('consolidate'), @@ -407,11 +446,24 @@ describe('/memory blocks', () => { revalidate: invoke('revalidate'), correct: invoke('correct'), } - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', operator, - repository: { export: async () => ({ outcome: 'page', events: [observationEvent], nextAfterEventId: null }) }, - release: async () => {}, - }) as unknown as Awaited>> + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator, + repository: { + export: async () => ({ + outcome: 'page', + events: [observationEvent], + nextAfterEventId: null, + }), + }, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > const commands = [ 'consolidate', @@ -422,23 +474,36 @@ describe('/memory blocks', () => { ] for (const command of commands) { const preview = await handleMemoryCommandBlocks(command, deps) - const applied = await handleMemoryCommandBlocks(`${command} --apply`, deps) + const applied = await handleMemoryCommandBlocks( + `${command} --apply`, + deps, + ) expect(preview.state).toBe('report') expect(applied.state).toBe('report') const previewCall = calls.at(-2)?.request as { mode?: string } const applyCall = calls.at(-1)?.request as { mode?: string } expect(previewCall.mode).toBe('preview') expect(applyCall.mode).toBe('apply') - if (preview.state === 'report') expect(preview.insertCommands?.[0]?.command).toContain('--apply') + if (preview.state === 'report') + expect(preview.insertCommands?.[0]?.command).toContain('--apply') } }) test('failed rejected and busy mutators render error tone with sanitized retryability', async () => { const { deps } = createDeps({ memory: makeMemory() }) const outcomes = [ - { outcome: 'failed', error: { message: '/secret/store.sqlite locked', retryable: true } }, - { outcome: 'rejected', error: { message: 'Request was rejected.', retryable: false } }, - { outcome: 'busy', error: { message: 'Try again shortly.', retryable: true } }, + { + outcome: 'failed', + error: { message: '/secret/store.sqlite locked', retryable: true }, + }, + { + outcome: 'rejected', + error: { message: 'Request was rejected.', retryable: false }, + }, + { + outcome: 'busy', + error: { message: 'Try again shortly.', retryable: true }, + }, ] let index = 0 const observationEvent = MemoryEventEnvelopeSchema.parse({ @@ -466,18 +531,24 @@ describe('/memory blocks', () => { }, }, }) - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', - operator: { correct: async () => outcomes[index++]! }, - repository: { - export: async () => ({ - outcome: 'page', - events: [observationEvent], - nextAfterEventId: null, - }), - }, - release: async () => {}, - }) as unknown as Awaited>> + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: { correct: async () => outcomes[index++]! }, + repository: { + export: async () => ({ + outcome: 'page', + events: [observationEvent], + nextAfterEventId: null, + }), + }, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > for (const command of ['pin one', 'pin one', 'pin one']) { const block = await handleMemoryCommandBlocks(command, deps) @@ -489,24 +560,137 @@ describe('/memory blocks', () => { } }) + test('audit-migration blocks every non-valid inspection before provider acquisition', async () => { + const cases = [ + [{ status: 'absent' as const }, 'Outcome: absent', 'secondary'], + [ + { status: 'invalid' as const, reason: 'malformed-json' as const }, + 'malformed JSON', + 'warning', + ], + [ + { status: 'invalid' as const, reason: 'schema-invalid' as const }, + 'schema validation', + 'warning', + ], + [ + { status: 'invalid' as const, reason: 'checksum-mismatch' as const }, + 'checksum validation', + 'warning', + ], + [ + { status: 'unreadable' as const, reason: 'read-failed' as const }, + 'could not be read safely', + 'error', + ], + ] as const + for (const [inspection, expected, tone] of cases) { + const { deps } = createDeps({ memory: makeMemory() }) + let providers = 0 + let audits = 0 + deps.inspectPersistedTaskMemoryV1 = async () => inspection + deps.getMemoryV2 = async () => { + providers++ + throw new Error('/private/provider') + } + deps.auditTaskMemoryV1Migration = async () => { + audits++ + return { outcome: 'no-record' } + } + const block = await handleMemoryCommandBlocks('audit-migration', deps) + if (block.state !== 'report') throw new Error('expected report') + expect(block.tone).toBe(tone) + expect(block.lines.join('\n')).toContain(expected) + expect(block.lines.join('\n')).not.toMatch(/private|provider/i) + expect(providers).toBe(0) + expect(audits).toBe(0) + } + }) + + test('audit-migration sanitizes an inspector throw without opening V2', async () => { + const { deps } = createDeps({ memory: makeMemory() }) + let providers = 0 + deps.inspectPersistedTaskMemoryV1 = async () => { + throw new Error('/secret/v1.json contents') + } + deps.getMemoryV2 = async () => { + providers++ + throw new Error('unused') + } + const block = await handleMemoryCommandBlocks('audit-migration', deps) + if (block.state !== 'report') throw new Error('expected report') + expect(block.tone).toBe('error') + expect(block.lines.join('\n')).not.toMatch(/secret|v1\.json|contents/i) + expect(providers).toBe(0) + }) + + test('audit-migration preserves valid unavailable authority behavior without audit or release', async () => { + const memory = makeMemory() + const { deps } = createDeps({ memory }) + let audits = 0 + let releases = 0 + deps.getMemoryV2 = async () => ({ + status: 'unavailable', + requestedAuthority: 'sqlite-v2-opt-in', + effectiveAuthority: 'sqlite-v2-opt-in', + degradation: + 'Memory V2 storage is unavailable; V1 remains disabled under opt-in authority.', + retryable: true, + }) + deps.auditTaskMemoryV1Migration = async () => { + audits++ + return { outcome: 'no-record' } + } + const block = await handleMemoryCommandBlocks('audit-migration', deps) + if (block.state !== 'report') throw new Error('expected report') + expect(block.lines.join('\n')).toContain( + 'Requested authority: sqlite-v2-opt-in; active authority: sqlite-v2-opt-in', + ) + expect(audits).toBe(0) + expect(releases).toBe(0) + }) + test('audit-migration renders an exact lossless SDK outcome with success tone', async () => { const memory = makeMemory() const { deps, calls } = createDeps({ memory }) - const repository = { append: async () => { throw new Error('must not append') } } + const repository = { + append: async () => { + throw new Error('must not append') + }, + } let request: unknown let releases = 0 deps.auditTaskMemoryV1Migration = async (value) => { request = value return { - outcome: 'exact', revision: 3, checksum: 'full-source-checksum', identity: 'v1:3:id', - markerEventId: 'marker', repositoryLastEventId: 'tail', importedTaskId: 'task', - importedObservationIds: ['observation-1', 'observation-2'], omittedFields: 0, warnings: [], - } as unknown as Awaited> + outcome: 'exact', + revision: 3, + checksum: 'full-source-checksum', + identity: 'v1:3:id', + markerEventId: 'marker', + repositoryLastEventId: 'tail', + importedTaskId: 'task', + importedObservationIds: ['observation-1', 'observation-2'], + omittedFields: 0, + warnings: [], + } as unknown as Awaited< + ReturnType + > } - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'sqlite-v2-opt-in', effectiveAuthority: 'sqlite-v2-opt-in', - projectId: 'project-1', operator: {}, repository, release: async () => { releases++ }, - }) as unknown as Awaited>> + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'sqlite-v2-opt-in', + effectiveAuthority: 'sqlite-v2-opt-in', + projectId: 'project-1', + operator: {}, + repository, + release: async () => { + releases++ + }, + }) as unknown as Awaited< + ReturnType> + > const block = await handleMemoryCommandBlocks('audit-migration', deps) @@ -514,8 +698,12 @@ describe('/memory blocks', () => { expect(block.tone).toBe('success') expect(block.lines.join('\n')).toContain('Outcome: exact') expect(block.lines.join('\n')).toContain('Source revision: 3') - expect(block.lines.join('\n')).toContain('Source checksum: full-source-checksum') - expect(block.lines.join('\n')).toContain('Marker verification: exact; imported body/provenance verification: exact') + expect(block.lines.join('\n')).toContain( + 'Source checksum: full-source-checksum', + ) + expect(block.lines.join('\n')).toContain( + 'Marker verification and full deterministic source-derived task/observation body equality: exact', + ) expect(block.lines.join('\n')).toContain('Imported observations: 2') expect(block.lines.join('\n')).toContain('Omitted fields: 0') expect(block.lines.join('\n')).toContain('Truncated fields: 0') @@ -529,16 +717,34 @@ describe('/memory blocks', () => { test('audit-migration warns when an exact outcome documents loss', async () => { const { deps } = createDeps({ memory: makeMemory() }) - deps.auditTaskMemoryV1Migration = async () => ({ - outcome: 'exact', revision: 3, checksum: 'checksum', identity: 'v1:3:id', - markerEventId: 'marker', repositoryLastEventId: 'tail', importedTaskId: 'task', - importedObservationIds: [], omittedFields: 2, truncatedFields: 1, - warnings: ['text-truncated', 'goal-excluded'], - }) as unknown as Awaited> - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', - projectId: 'project-1', operator: {}, repository: {}, release: async () => {}, - }) as unknown as Awaited>> + deps.auditTaskMemoryV1Migration = async () => + ({ + outcome: 'exact', + revision: 3, + checksum: 'checksum', + identity: 'v1:3:id', + markerEventId: 'marker', + repositoryLastEventId: 'tail', + importedTaskId: 'task', + importedObservationIds: [], + omittedFields: 2, + truncatedFields: 1, + warnings: ['text-truncated', 'goal-excluded'], + }) as unknown as Awaited< + ReturnType + > + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: {}, + repository: {}, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > const block = await handleMemoryCommandBlocks('audit-migration', deps) @@ -546,7 +752,9 @@ describe('/memory blocks', () => { expect(block.tone).toBe('warning') expect(block.lines.join('\n')).toContain('Omitted fields: 2') expect(block.lines.join('\n')).toContain('Truncated fields: 1') - expect(block.lines.join('\n')).toContain('Warnings: 2 (text-truncated, goal-excluded)') + expect(block.lines.join('\n')).toContain( + 'Warnings: 2 (text-truncated, goal-excluded)', + ) expect(block.lines.join('\n')).toContain('Lossless migration evidence: no') }) @@ -555,18 +763,41 @@ describe('/memory blocks', () => { const outcomes = [ { outcome: 'no-record' }, { outcome: 'not-migrated', revision: 3, checksum: 'checksum' }, - { outcome: 'incomplete', reason: 'reservation-only', revision: 3, checksum: 'checksum' }, - { outcome: 'mismatch', reason: 'imported-body-mismatch', revision: 3, checksum: 'checksum' }, + { + outcome: 'incomplete', + reason: 'reservation-only', + revision: 3, + checksum: 'checksum', + }, + { + outcome: 'mismatch', + reason: 'imported-body-mismatch', + revision: 3, + checksum: 'checksum', + }, { outcome: 'rejected', reason: 'repository-rejected' }, { outcome: 'failed', reason: 'repository-failed' }, ] as const let index = 0 let releases = 0 - deps.auditTaskMemoryV1Migration = async () => outcomes[index++] as unknown as Awaited> - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', - projectId: 'project-1', operator: {}, repository: {}, release: async () => { releases++ }, - }) as unknown as Awaited>> + deps.auditTaskMemoryV1Migration = async () => + outcomes[index++] as unknown as Awaited< + ReturnType + > + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: {}, + repository: {}, + release: async () => { + releases++ + }, + }) as unknown as Awaited< + ReturnType> + > const rendered = [] for (let i = 0; i < outcomes.length; i++) { @@ -588,11 +819,23 @@ describe('/memory blocks', () => { test('audit-migration releases the provider lease once when the injected audit throws', async () => { const { deps } = createDeps({ memory: makeMemory() }) let releases = 0 - deps.auditTaskMemoryV1Migration = async () => { throw new Error('/secret/store.sqlite SELECT token') } - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', - projectId: 'project-1', operator: {}, repository: {}, release: async () => { releases++ }, - }) as unknown as Awaited>> + deps.auditTaskMemoryV1Migration = async () => { + throw new Error('/secret/store.sqlite SELECT token') + } + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: {}, + repository: {}, + release: async () => { + releases++ + }, + }) as unknown as Awaited< + ReturnType> + > const block = await handleMemoryCommandBlocks('audit-migration', deps) @@ -604,18 +847,55 @@ describe('/memory blocks', () => { test('query and diagnose render bounded generic reports', async () => { const { deps } = createDeps({ memory: makeMemory() }) - deps.getMemoryV2 = async () => ({ - status: 'available', projectId: 'project-1', operator: {}, - repository: { - query: async () => ({ outcome: 'result', result: { matchedTasks: [], verifiedKnowledge: [], reusableDiscovery: [], rereadRequired: [], historicalContext: [], degradation: { state: 'none' } } }), - health: async () => ({ status: 'healthy', authority: { kind: 'authoritative' }, backend: { backendId: 'bun-sqlite-memory-v2', capabilities: ['query'] }, issues: [] }), - kernelHealth: async () => ({ status: 'healthy', schemaVersion: 2, projectionCursor: 0 }), - getCapabilities: async () => ({ status: 'ok', capabilities: [{ name: 'query', available: true }] }), - export: async () => ({ outcome: 'page', events: [], nextAfterEventId: null }), - }, - release: async () => {}, - }) as unknown as Awaited>> - const query = await handleMemoryCommandBlocks('query deterministic lookup', deps) + deps.getMemoryV2 = async () => + ({ + status: 'available', + projectId: 'project-1', + operator: {}, + repository: { + query: async () => ({ + outcome: 'result', + result: { + matchedTasks: [], + verifiedKnowledge: [], + reusableDiscovery: [], + rereadRequired: [], + historicalContext: [], + degradation: { state: 'none' }, + }, + }), + health: async () => ({ + status: 'healthy', + authority: { kind: 'authoritative' }, + backend: { + backendId: 'bun-sqlite-memory-v2', + capabilities: ['query'], + }, + issues: [], + }), + kernelHealth: async () => ({ + status: 'healthy', + schemaVersion: 2, + projectionCursor: 0, + }), + getCapabilities: async () => ({ + status: 'ok', + capabilities: [{ name: 'query', available: true }], + }), + export: async () => ({ + outcome: 'page', + events: [], + nextAfterEventId: null, + }), + }, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > + const query = await handleMemoryCommandBlocks( + 'query deterministic lookup', + deps, + ) const diagnose = await handleMemoryCommandBlocks('diagnose', deps) expect(query.state).toBe('report') expect(diagnose.state).toBe('report') @@ -623,7 +903,9 @@ describe('/memory blocks', () => { if (diagnose.state === 'report') { expect(diagnose.lines.join('\n')).toContain('Kernel: healthy') expect(diagnose.lines.join('\n')).toContain('Release N') - expect(diagnose.lines.join('\n')).toContain('json-v1 and shadow-v2 remain supported but are deprecated') + expect(diagnose.lines.join('\n')).toContain( + 'json-v1 and shadow-v2 remain supported but are deprecated', + ) } }) @@ -631,17 +913,39 @@ describe('/memory blocks', () => { const { deps } = createDeps({ memory: makeMemory() }) let exports = 0 let lowLevelLists = 0 - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', operator: {}, - repository: { - export: async () => { exports++; return { outcome: 'page', events: [], nextAfterEventId: null } }, - listEvents: async () => { lowLevelLists++; return { status: 'ok', events: [] } }, - health: async () => ({ status: 'healthy', authority: { kind: 'authoritative' }, backend: { backendId: 'bun-sqlite-memory-v2', capabilities: [] }, issues: [] }), - kernelHealth: async () => ({ status: 'healthy', schemaVersion: 2, projectionCursor: 0 }), - getCapabilities: async () => ({ status: 'ok', capabilities: [] }), - }, - release: async () => {}, - }) as unknown as Awaited>> + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: {}, + repository: { + export: async () => { + exports++ + return { outcome: 'page', events: [], nextAfterEventId: null } + }, + listEvents: async () => { + lowLevelLists++ + return { status: 'ok', events: [] } + }, + health: async () => ({ + status: 'healthy', + authority: { kind: 'authoritative' }, + backend: { backendId: 'bun-sqlite-memory-v2', capabilities: [] }, + issues: [], + }), + kernelHealth: async () => ({ + status: 'healthy', + schemaVersion: 2, + projectionCursor: 0, + }), + getCapabilities: async () => ({ status: 'ok', capabilities: [] }), + }, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > await handleMemoryCommandBlocks('inspect', deps) await handleMemoryCommandBlocks('diagnose', deps) expect(exports).toBe(2) @@ -655,12 +959,28 @@ describe('/memory blocks', () => { symlinkSync(outside, join(root, '.openbuff', 'memory')) const { deps } = createDeps({ memory: makeMemory() }) deps.getRootDir = () => root - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', - operator: { exportManifest: async () => ({ outcome: 'exported', manifest: { checksum: 'sha256:1234567890abcdef', canonicalEventCount: 0, events: [], warnings: [] } }) }, - repository: {}, - release: async () => {}, - }) as unknown as Awaited>> + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: { + exportManifest: async () => ({ + outcome: 'exported', + manifest: { + checksum: 'sha256:1234567890abcdef', + canonicalEventCount: 0, + events: [], + warnings: [], + }, + }), + }, + repository: {}, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > const block = await handleMemoryCommandBlocks('export', deps) if (block.state !== 'report') throw new Error('expected report') expect(block.tone).toBe('error') @@ -680,13 +1000,36 @@ describe('/memory blocks', () => { const { deps } = createDeps({ memory: makeMemory() }) deps.getRootDir = () => root let imports = 0 - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', - operator: { importManifest: async () => { imports++; return { outcome: 'imported' } } }, - repository: { export: async () => ({ outcome: 'page', events: [{ eventId: 'existing' }], nextAfterEventId: null }) }, - release: async () => {}, - }) as unknown as Awaited>> - for (const path of ['/absolute.json', '../escape.json', 'imports/link.json', 'imports/oversized.json', 'imports/malformed.json']) { + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: { + importManifest: async () => { + imports++ + return { outcome: 'imported' } + }, + }, + repository: { + export: async () => ({ + outcome: 'page', + events: [{ eventId: 'existing' }], + nextAfterEventId: null, + }), + }, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > + for (const path of [ + '/absolute.json', + '../escape.json', + 'imports/link.json', + 'imports/oversized.json', + 'imports/malformed.json', + ]) { const block = await handleMemoryCommandBlocks(`import ${path}`, deps) if (block.state !== 'report') throw new Error('expected report') expect(block.tone).toBe('error') @@ -697,21 +1040,54 @@ describe('/memory blocks', () => { test('available command leases release exactly once on success and throw', async () => { const { deps } = createDeps({ memory: makeMemory() }) let releases = 0 - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', - operator: {}, - repository: { - query: async () => ({ outcome: 'result', result: { matchedTasks: [], verifiedKnowledge: [], reusableDiscovery: [], rereadRequired: [], historicalContext: [], degradation: { state: 'none' } } }), - }, - release: async () => { releases++ }, - }) as unknown as Awaited>> - expect((await handleMemoryCommandBlocks('query safe', deps)).state).toBe('report') + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: {}, + repository: { + query: async () => ({ + outcome: 'result', + result: { + matchedTasks: [], + verifiedKnowledge: [], + reusableDiscovery: [], + rereadRequired: [], + historicalContext: [], + degradation: { state: 'none' }, + }, + }), + }, + release: async () => { + releases++ + }, + }) as unknown as Awaited< + ReturnType> + > + expect((await handleMemoryCommandBlocks('query safe', deps)).state).toBe( + 'report', + ) expect(releases).toBe(1) - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', operator: {}, - repository: { query: async () => { throw new Error('/secret/store.sqlite SELECT token') } }, - release: async () => { releases++ }, - }) as unknown as Awaited>> + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: {}, + repository: { + query: async () => { + throw new Error('/secret/store.sqlite SELECT token') + }, + }, + release: async () => { + releases++ + }, + }) as unknown as Awaited< + ReturnType> + > const failed = await handleMemoryCommandBlocks('query safe', deps) expect(releases).toBe(2) if (failed.state !== 'report') throw new Error('expected report') @@ -721,21 +1097,37 @@ describe('/memory blocks', () => { test('pagination rejects repeated and empty cursors without leaking details', async () => { const { deps } = createDeps({ memory: makeMemory() }) let calls = 0 - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', operator: {}, - repository: { export: async () => { calls++; return { outcome: 'page', events: [], nextAfterEventId: 'same' } } }, - release: async () => {}, - }) as unknown as Awaited>> + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: {}, + repository: { + export: async () => { + calls++ + return { outcome: 'page', events: [], nextAfterEventId: 'same' } + }, + }, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > const block = await handleMemoryCommandBlocks('inspect target', deps) if (block.state !== 'report') throw new Error('expected report') expect(block.tone).toBe('error') expect(calls).toBe(1) - expect(block.lines.join('\n')).toContain('operation could not be completed safely') + expect(block.lines.join('\n')).toContain( + 'operation could not be completed safely', + ) }) test('sanitized boundary hides arbitrary provider failures', async () => { const { deps } = createDeps({ memory: makeMemory() }) - deps.getMemoryV2 = async () => { throw new Error('/home/private/store.sqlite SELECT api_key=secret') } + deps.getMemoryV2 = async () => { + throw new Error('/home/private/store.sqlite SELECT api_key=secret') + } const block = await handleMemoryCommandBlocks('diagnose', deps) if (block.state !== 'report') throw new Error('expected report') const output = block.lines.join('\n') @@ -1105,7 +1497,10 @@ describe('/memory blocks', () => { expect(block.type).toBe('memory') expect(block.state).toBe('report') - if (block.state === 'report') expect(block.lines.join('\n')).toContain('Usage: /memory [status|authority|diagnose|audit-migration|query ') + if (block.state === 'report') + expect(block.lines.join('\n')).toContain( + 'Usage: /memory [status|authority|diagnose|audit-migration|query ', + ) expect(calls.prune).toBe(0) expect(calls.moves).toBe(0) }) @@ -1114,14 +1509,19 @@ describe('/memory blocks', () => { const memory = makeMemory() const { deps } = createDeps({ memory, reconciled: memory }) deps.getMemoryV2 = async () => ({ - status: 'unavailable', requestedAuthority: 'shadow-v2', effectiveAuthority: 'json-v1', - degradation: 'V1 JSON memory is authoritative; SQLite was not opened.', retryable: false, + status: 'unavailable', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'json-v1', + degradation: 'V1 JSON memory is authoritative; SQLite was not opened.', + retryable: false, }) const output = await handleMemoryCommand('status', deps) expect(output).toContain('Release N') - expect(output).toContain('json-v1 and shadow-v2 remain supported but are deprecated') + expect(output).toContain( + 'json-v1 and shadow-v2 remain supported but are deprecated', + ) expect(output).toContain('sqlite-v2-opt-in is the default and replacement') }) diff --git a/cli/src/commands/memory-command.ts b/cli/src/commands/memory-command.ts index b7e8ac93f3..9df4bee2a6 100644 --- a/cli/src/commands/memory-command.ts +++ b/cli/src/commands/memory-command.ts @@ -8,6 +8,7 @@ import { auditTaskMemoryV1Migration, collectWorkspaceMoves, getHarnessStateDir, + inspectPersistedTaskMemoryV1, loadPersistedTaskMemory, pruneStaleTaskMemoryEvidence, reconcileTaskMemoryEvidence, @@ -30,6 +31,7 @@ import { formatAge, pluralizeEntries } from '../utils/format-helpers' import type { TaskMemoryPruneOutcome, + TaskMemoryV1Inspection, V1MigrationAuditOutcome, WorkspaceMoveRecord, } from '@openbuff/sdk' @@ -37,6 +39,7 @@ import type { export type MemoryCommandDeps = { getRootDir: () => string loadPersistedTaskMemory: typeof loadPersistedTaskMemory + inspectPersistedTaskMemoryV1?: typeof inspectPersistedTaskMemoryV1 reconcileTaskMemoryEvidence: typeof reconcileTaskMemoryEvidence pruneStaleTaskMemoryEvidence: typeof pruneStaleTaskMemoryEvidence auditTaskMemoryV1Migration: typeof auditTaskMemoryV1Migration @@ -74,6 +77,7 @@ async function loadWorkspaceMoves( const defaultDeps: MemoryCommandDeps = { getRootDir: getProjectRoot, loadPersistedTaskMemory, + inspectPersistedTaskMemoryV1, reconcileTaskMemoryEvidence, pruneStaleTaskMemoryEvidence, auditTaskMemoryV1Migration, @@ -203,21 +207,38 @@ function memoryBlockToString( } } -const MEMORY_USAGE = 'Usage: /memory [status|authority|diagnose|audit-migration|query |inspect [eventId]|consolidate [--apply] [--task ]|repair [--apply]|revalidate [--apply]|correct [--apply]|forget [--apply]|pin [--apply]|export [--format json|markdown] [--include-stale]|import [--apply]|prune]' -const RELEASE_N_AUTHORITY_WARNING = 'Release N: json-v1 and shadow-v2 remain supported but are deprecated; sqlite-v2-opt-in is the default and replacement.' +const MEMORY_USAGE = + 'Usage: /memory [status|authority|diagnose|audit-migration|query |inspect [eventId]|consolidate [--apply] [--task ]|repair [--apply]|revalidate [--apply]|correct [--apply]|forget [--apply]|pin [--apply]|export [--format json|markdown] [--include-stale]|import [--apply]|prune]' +const RELEASE_N_AUTHORITY_WARNING = + 'Release N: json-v1 and shadow-v2 remain supported but are deprecated; sqlite-v2-opt-in is the default and replacement.' const CLI_SESSION_ID = 'memory-cli' const EXPORT_MAX_BYTES = 8 * 1024 * 1024 -function report(title: string, lines: string[], tone: import('../types/chat').MemoryReportTone = 'secondary', insertCommands?: Array<{ label: string; command: string }>): import('../types/chat').MemoryContentBlock { - return { type: 'memory', state: 'report', title, tone, lines: lines.slice(0, 100), ...(insertCommands?.length ? { insertCommands } : {}) } +function report( + title: string, + lines: string[], + tone: import('../types/chat').MemoryReportTone = 'secondary', + insertCommands?: Array<{ label: string; command: string }>, +): import('../types/chat').MemoryContentBlock { + return { + type: 'memory', + state: 'report', + title, + tone, + lines: lines.slice(0, 100), + ...(insertCommands?.length ? { insertCommands } : {}), + } } -function commandError(message: string): import('../types/chat').MemoryContentBlock { +function commandError( + message: string, +): import('../types/chat').MemoryContentBlock { return report('Memory V2', [message.slice(0, 1_024)], 'error') } function safeOperationMessage(value: unknown): string { - if (typeof value !== 'string' || value.length === 0) return 'The operation could not be completed.' + if (typeof value !== 'string' || value.length === 0) + return 'The operation could not be completed.' const allowed = [ 'Canonical inventory is temporarily unavailable.', 'Canonical inventory failed validation.', @@ -228,22 +249,34 @@ function safeOperationMessage(value: unknown): string { 'V1 JSON memory is authoritative; SQLite was not opened.', 'Invalid memory authority; using V1 memory only.', ] - return allowed.includes(value) ? value : 'The operation could not be completed safely.' + return allowed.includes(value) + ? value + : 'The operation could not be completed safely.' } -function operationLines(outcome: { outcome: string; error?: { message?: unknown; retryable?: unknown } }, extra: string[] = []): string[] { +function operationLines( + outcome: { + outcome: string + error?: { message?: unknown; retryable?: unknown } + }, + extra: string[] = [], +): string[] { return [ `Outcome: ${outcome.outcome}.`, ...extra, - ...(outcome.error ? [ - `Error: ${safeOperationMessage(outcome.error.message)}`, - `Retryable: ${outcome.error.retryable === true ? 'yes' : 'no'}.`, - ] : []), + ...(outcome.error + ? [ + `Error: ${safeOperationMessage(outcome.error.message)}`, + `Retryable: ${outcome.error.retryable === true ? 'yes' : 'no'}.`, + ] + : []), ] } function isFailureOutcome(outcome: { outcome: string }): boolean { - return ['failed', 'rejected', 'busy', 'integrity-mismatch'].includes(outcome.outcome) + return ['failed', 'rejected', 'busy', 'integrity-mismatch'].includes( + outcome.outcome, + ) } function deterministicId(prefix: string, value: string): string { @@ -251,84 +284,124 @@ function deterministicId(prefix: string, value: string): string { } const AUDIT_INCOMPLETE_REASONS: Record< - Extract['reason'], + Extract< + V1MigrationAuditOutcome, + { outcome: 'incomplete' | 'mismatch' } + >['reason'], string > = { 'reservation-only': 'Only the migration reservation was found.', - 'legacy-marker-unverifiable': 'The legacy migration marker cannot verify this source record.', + 'legacy-marker-unverifiable': + 'The legacy migration marker cannot verify this source record.', 'missing-imported-task': 'The imported task body is missing.', - 'missing-imported-observations': 'One or more imported observations are missing.', - 'imported-body-mismatch': 'The imported body or provenance does not match the source marker.', - 'revision-conflict': 'Conflicting migration evidence exists for this source revision.', - 'checksum-conflict': 'Conflicting migration evidence exists for this source checksum.', + 'missing-imported-observations': + 'One or more imported observations are missing.', + 'imported-body-mismatch': + 'The imported body or provenance does not match the source marker.', + 'revision-conflict': + 'Conflicting migration evidence exists for this source revision.', + 'checksum-conflict': + 'Conflicting migration evidence exists for this source checksum.', } const AUDIT_FAILURE_REASONS: Record< - Extract['reason'], + Extract< + V1MigrationAuditOutcome, + { outcome: 'rejected' | 'failed' } + >['reason'], string > = { 'checksum-mismatch': 'The V1 source record failed checksum validation.', 'repository-rejected': 'The audit read was rejected safely.', 'repository-failed': 'The audit read could not be completed safely.', - 'invalid-export': 'The audit could not validate the exported migration evidence.', - 'wrong-project': 'The exported migration evidence belongs to another project.', - 'pagination-invalid': 'The exported migration evidence did not form a valid bounded sequence.', - 'page-limit-exceeded': 'The migration evidence exceeded the bounded audit scan.', + 'invalid-export': + 'The audit could not validate the exported migration evidence.', + 'wrong-project': + 'The exported migration evidence belongs to another project.', + 'pagination-invalid': + 'The exported migration evidence did not form a valid bounded sequence.', + 'page-limit-exceeded': + 'The migration evidence exceeded the bounded audit scan.', } function renderMigrationAudit( outcome: V1MigrationAuditOutcome, ): import('../types/chat').MemoryContentBlock { if (outcome.outcome === 'no-record') { - return report('Memory V1 migration audit', [ - 'Outcome: no-record.', - 'No currently loaded V1 task-memory record is available to audit.', - 'No writes were performed.', - ], 'secondary') + return report( + 'Memory V1 migration audit', + [ + 'Outcome: no-record.', + 'No currently loaded V1 task-memory record is available to audit.', + 'No writes were performed.', + ], + 'secondary', + ) } if (outcome.outcome === 'exact') { const truncatedFields = outcome.truncatedFields ?? 0 const warningCount = outcome.warnings.length - const lossless = outcome.omittedFields === 0 && truncatedFields === 0 && warningCount === 0 - return report('Memory V1 migration audit', [ - 'Outcome: exact.', - `Source revision: ${outcome.revision}.`, - `Source checksum: ${outcome.checksum}.`, - 'Marker verification: exact; imported body/provenance verification: exact.', - `Imported observations: ${outcome.importedObservationIds.length}.`, - `Omitted fields: ${outcome.omittedFields}.`, - `Truncated fields: ${truncatedFields}.`, - `Warnings: ${warningCount}${warningCount ? ` (${outcome.warnings.slice(0, 100).join(', ')})` : ' (none)'}.`, - `Lossless migration evidence: ${lossless ? 'yes' : 'no'}.`, - 'No writes were performed.', - ], lossless ? 'success' : 'warning') + const lossless = + outcome.omittedFields === 0 && truncatedFields === 0 && warningCount === 0 + return report( + 'Memory V1 migration audit', + [ + 'Outcome: exact.', + `Source revision: ${outcome.revision}.`, + `Source checksum: ${outcome.checksum}.`, + 'Marker verification and full deterministic source-derived task/observation body equality: exact.', + `Imported observations: ${outcome.importedObservationIds.length}.`, + `Omitted fields: ${outcome.omittedFields}.`, + `Truncated fields: ${truncatedFields}.`, + `Warnings: ${warningCount}${warningCount ? ` (${outcome.warnings.slice(0, 100).join(', ')})` : ' (none)'}.`, + `Lossless migration evidence: ${lossless ? 'yes' : 'no'}.`, + 'No writes were performed.', + ], + lossless ? 'success' : 'warning', + ) } if (outcome.outcome === 'not-migrated') { - return report('Memory V1 migration audit', [ - 'Outcome: not-migrated.', - `Source revision: ${outcome.revision}.`, - `Source checksum: ${outcome.checksum}.`, - 'No matching migration marker was found in the bounded V2 audit.', - 'No writes were performed.', - ], 'warning') + return report( + 'Memory V1 migration audit', + [ + 'Outcome: not-migrated.', + `Source revision: ${outcome.revision}.`, + `Source checksum: ${outcome.checksum}.`, + 'No matching migration marker was found in the bounded V2 audit.', + 'No writes were performed.', + ], + 'warning', + ) } if (outcome.outcome === 'incomplete' || outcome.outcome === 'mismatch') { - return report('Memory V1 migration audit', [ - `Outcome: ${outcome.outcome}.`, - `Source revision: ${outcome.revision}.`, - `Source checksum: ${outcome.checksum}.`, - AUDIT_INCOMPLETE_REASONS[outcome.reason], - 'No writes were performed.', - ], outcome.outcome === 'mismatch' ? 'error' : 'warning') + return report( + 'Memory V1 migration audit', + [ + `Outcome: ${outcome.outcome}.`, + `Source revision: ${outcome.revision}.`, + `Source checksum: ${outcome.checksum}.`, + AUDIT_INCOMPLETE_REASONS[outcome.reason], + 'No writes were performed.', + ], + outcome.outcome === 'mismatch' ? 'error' : 'warning', + ) } if (outcome.outcome === 'rejected' || outcome.outcome === 'failed') { - return report('Memory V1 migration audit', [ - `Outcome: ${outcome.outcome}.`, - AUDIT_FAILURE_REASONS[outcome.reason], - ...(outcome.revision === undefined ? [] : [`Source revision: ${outcome.revision}.`]), - ...(outcome.checksum === undefined ? [] : [`Source checksum: ${outcome.checksum}.`]), - 'No writes were performed.', - ], outcome.outcome === 'failed' ? 'error' : 'warning') + return report( + 'Memory V1 migration audit', + [ + `Outcome: ${outcome.outcome}.`, + AUDIT_FAILURE_REASONS[outcome.reason], + ...(outcome.revision === undefined + ? [] + : [`Source revision: ${outcome.revision}.`]), + ...(outcome.checksum === undefined + ? [] + : [`Source checksum: ${outcome.checksum}.`]), + 'No writes were performed.', + ], + outcome.outcome === 'failed' ? 'error' : 'warning', + ) } return report( 'Memory V1 migration audit', @@ -337,6 +410,51 @@ function renderMigrationAudit( ) } +const V1_INSPECTION_INVALID_REASONS: Record< + Extract['reason'], + string +> = { + 'malformed-json': 'The persisted V1 record is malformed JSON.', + 'schema-invalid': 'The persisted V1 record failed schema validation.', + 'checksum-mismatch': 'The persisted V1 record failed checksum validation.', +} + +function renderV1InspectionBlock( + inspection: Exclude, +): import('../types/chat').MemoryContentBlock { + if (inspection.status === 'absent') { + return report( + 'Memory V1 migration audit', + [ + 'Outcome: absent.', + 'Independent read-only inspection verified that no persisted V1 record exists.', + 'Memory V2 was not opened. No writes were performed.', + ], + 'secondary', + ) + } + if (inspection.status === 'invalid') { + return report( + 'Memory V1 migration audit', + [ + 'Outcome: invalid.', + V1_INSPECTION_INVALID_REASONS[inspection.reason], + 'Audit blocked; Memory V2 was not opened. No writes were performed.', + ], + 'warning', + ) + } + return report( + 'Memory V1 migration audit', + [ + 'Outcome: unreadable.', + 'The persisted V1 record could not be read safely.', + 'Audit blocked; Memory V2 was not opened. No writes were performed.', + ], + 'error', + ) +} + async function getV2(deps: MemoryCommandDeps) { return deps.getMemoryV2 ? deps.getMemoryV2(deps.getRootDir()) @@ -349,165 +467,416 @@ async function getV2(deps: MemoryCommandDeps) { } } -function parseArgs(rawArgs: string): { command: string; args: string[]; apply: boolean } { +function parseArgs(rawArgs: string): { + command: string + args: string[] + apply: boolean +} { const tokens = rawArgs.trim().split(/\s+/).filter(Boolean) const command = (tokens.shift() ?? 'status').toLowerCase() const apply = tokens.includes('--apply') return { command, args: tokens.filter((token) => token !== '--apply'), apply } } -async function runV2Command(rawArgs: string, deps: MemoryCommandDeps): Promise { +async function runV2Command( + rawArgs: string, + deps: MemoryCommandDeps, +): Promise { const parsed = parseArgs(rawArgs) - if (!['authority', 'diagnose', 'audit-migration', 'query', 'inspect', 'consolidate', 'repair', 'revalidate', 'correct', 'forget', 'pin', 'export', 'import'].includes(parsed.command)) return commandError(MEMORY_USAGE) + if ( + ![ + 'authority', + 'diagnose', + 'audit-migration', + 'query', + 'inspect', + 'consolidate', + 'repair', + 'revalidate', + 'correct', + 'forget', + 'pin', + 'export', + 'import', + ].includes(parsed.command) + ) + return commandError(MEMORY_USAGE) + let inspectedMemory: + | Extract + | undefined + if (parsed.command === 'audit-migration') { + const inspect = + deps.inspectPersistedTaskMemoryV1 ?? inspectPersistedTaskMemoryV1 + const inspected = await inspect({ rootDir: deps.getRootDir() }) + if (inspected.status !== 'valid') return renderV1InspectionBlock(inspected) + inspectedMemory = inspected + } const v2 = await getV2(deps) try { if (parsed.command === 'authority') { - return report('Memory authority', [ - 'Valid values: json-v1, shadow-v2, sqlite-v2-opt-in.', - RELEASE_N_AUTHORITY_WARNING, - `Requested: ${v2.requestedAuthority}; effective: ${v2.effectiveAuthority}.`, - 'Switch via OPENBUFF_MEMORY_AUTHORITY or the SDK authority option, then reset/restart the client.', - ]) - } - if (v2.status === 'unavailable') return report('Memory V2 unavailable', [ - `Requested authority: ${v2.requestedAuthority}; active authority: ${v2.effectiveAuthority}.`, - RELEASE_N_AUTHORITY_WARNING, - safeOperationMessage(v2.degradation), - ], 'warning') - const scope = { schemaVersion: 2 as const, projectId: v2.projectId, sessionId: MemorySessionIdSchema.parse(CLI_SESSION_ID) } - const now = new Date().toISOString() - - if (parsed.command === 'audit-migration') { - const memory = await deps.loadPersistedTaskMemory({ rootDir: deps.getRootDir() }) - return renderMigrationAudit(await deps.auditTaskMemoryV1Migration({ - memory, + return report('Memory authority', [ + 'Valid values: json-v1, shadow-v2, sqlite-v2-opt-in.', + RELEASE_N_AUTHORITY_WARNING, + `Requested: ${v2.requestedAuthority}; effective: ${v2.effectiveAuthority}.`, + 'Switch via OPENBUFF_MEMORY_AUTHORITY or the SDK authority option, then reset/restart the client.', + ]) + } + if (v2.status === 'unavailable') + return report( + 'Memory V2 unavailable', + [ + `Requested authority: ${v2.requestedAuthority}; active authority: ${v2.effectiveAuthority}.`, + RELEASE_N_AUTHORITY_WARNING, + safeOperationMessage(v2.degradation), + ], + 'warning', + ) + const scope = { + schemaVersion: 2 as const, projectId: v2.projectId, - repository: v2.repository, - })) - } + sessionId: MemorySessionIdSchema.parse(CLI_SESSION_ID), + } + const now = new Date().toISOString() - if (parsed.command === 'diagnose') { - const [health, kernel, capabilities, inventory] = await Promise.all([ - v2.repository.health({ schemaVersion: 2, projectId: v2.projectId }), - v2.repository.kernelHealth(), - v2.repository.getCapabilities(), - canonicalInventory(v2, 1_000), - ]) - const events = inventory.events - return report('Memory V2 diagnosis', [ - `Requested authority: ${v2.requestedAuthority}; active authority: ${v2.effectiveAuthority}.`, - RELEASE_N_AUTHORITY_WARNING, - 'V1 compatibility shadow: available when task-memory.json is hydrated and remains normally persisted.', - `Health: ${health.status}; repository authority: ${health.authority.kind}.`, - `Backend: ${health.backend.backendId}; capabilities: ${health.backend.capabilities.join(', ')}.`, - `Kernel: ${kernel.status}; schema: ${kernel.schemaVersion ?? 'unknown'}; projection cursor: ${kernel.projectionCursor ?? 'unknown'}.`, - `Integrity: ${health.issues.length ? health.issues.map(safeOperationMessage).join('; ') : 'ok'}.`, - `Events inspected: ${inventory.error ? 'unavailable' : events.length}.`, - !inventory.error - ? (() => { - const marker = [...events].reverse().find((event) => event.eventType === 'migration.v1.imported') - const payload = marker ? V1MigrationPayloadSchema.safeParse(marker.payload) : undefined - return payload?.success - ? `Last V1 import: revision ${payload.data.sourceRevision ?? 'unknown'}; checksum ${payload.data.sourceChecksum?.slice(0, 24) ?? 'unknown'}; identity ${payload.data.legacyRecordKey.slice(0, 64)}.` - : 'Last V1 import: unavailable.' - })() - : 'Last V1 import: unavailable.', - 'Last parity: unavailable in this command session.', - capabilities.status === 'ok' ? `Kernel capabilities: ${capabilities.capabilities.filter(({ available }) => available).map(({ name }) => name).join(', ')}.` : 'Kernel capabilities unavailable.', - health.status === 'healthy' ? 'Recovery guidance: none required.' : 'Recovery guidance: run /memory repair for a non-destructive projection preview.', - ], health.status === 'healthy' ? 'success' : 'warning') - } + if (parsed.command === 'audit-migration') { + return renderMigrationAudit( + await deps.auditTaskMemoryV1Migration({ + memory: inspectedMemory!.memory, + projectId: v2.projectId, + repository: v2.repository, + }), + ) + } - if (parsed.command === 'query') { - const text = parsed.args.join(' ').trim() - if (!text) return commandError(MEMORY_USAGE) - const outcome = await v2.repository.query({ ...scope, queryId: QueryIdSchema.parse(deterministicId('query', text)), query: text, selectors: [], artifactKinds: [], includeHistorical: false, maxResultsPerCategory: 10 }) - if (outcome.outcome !== 'result') return commandError(`Memory query failed: ${safeOperationMessage(outcome.error.message)} Retryable: ${outcome.error.retryable ? 'yes' : 'no'}.`) - const result = outcome.result - const lines = [ - `Tasks: ${result.matchedTasks.length}; verified: ${result.verifiedKnowledge.length}; discoveries: ${result.reusableDiscovery.length}; reread: ${result.rereadRequired.length}; historical: ${result.historicalContext.length}.`, - ...result.matchedTasks.map((item) => `Task ${item.taskId} [${item.score.toFixed(2)}]: ${item.title}.`), - ...result.verifiedKnowledge.map((item) => `Verified ${item.observation.observationId} [${item.score.toFixed(2)}]: ${item.observation.summary}. Reasons: ${item.reasons.map(({ code }) => code).join(', ')}.`), - ...result.reusableDiscovery.map((item) => `Discovery ${item.observation.observationId} [${item.score.toFixed(2)}]: ${item.observation.summary}.`), - ...result.rereadRequired.map((item) => `Reread ${item.observationId}: ${'path' in item.selector ? item.selector.path : item.selector.uri} (${item.reason}).`), - `Degradation: ${result.degradation.state === 'none' ? 'none' : result.degradation.reasons.map(({ detail }) => detail).join('; ')}.`, - ] - return report('Memory V2 query', lines, result.rereadRequired.length ? 'warning' : 'success') - } + if (parsed.command === 'diagnose') { + const [health, kernel, capabilities, inventory] = await Promise.all([ + v2.repository.health({ schemaVersion: 2, projectId: v2.projectId }), + v2.repository.kernelHealth(), + v2.repository.getCapabilities(), + canonicalInventory(v2, 1_000), + ]) + const events = inventory.events + return report( + 'Memory V2 diagnosis', + [ + `Requested authority: ${v2.requestedAuthority}; active authority: ${v2.effectiveAuthority}.`, + RELEASE_N_AUTHORITY_WARNING, + 'V1 compatibility shadow: available when task-memory.json is hydrated and remains normally persisted.', + `Health: ${health.status}; repository authority: ${health.authority.kind}.`, + `Backend: ${health.backend.backendId}; capabilities: ${health.backend.capabilities.join(', ')}.`, + `Kernel: ${kernel.status}; schema: ${kernel.schemaVersion ?? 'unknown'}; projection cursor: ${kernel.projectionCursor ?? 'unknown'}.`, + `Integrity: ${health.issues.length ? health.issues.map(safeOperationMessage).join('; ') : 'ok'}.`, + `Events inspected: ${inventory.error ? 'unavailable' : events.length}.`, + !inventory.error + ? (() => { + const marker = [...events] + .reverse() + .find((event) => event.eventType === 'migration.v1.imported') + const payload = marker + ? V1MigrationPayloadSchema.safeParse(marker.payload) + : undefined + return payload?.success + ? `Last V1 import: revision ${payload.data.sourceRevision ?? 'unknown'}; checksum ${payload.data.sourceChecksum?.slice(0, 24) ?? 'unknown'}; identity ${payload.data.legacyRecordKey.slice(0, 64)}.` + : 'Last V1 import: unavailable.' + })() + : 'Last V1 import: unavailable.', + 'Last parity: unavailable in this command session.', + capabilities.status === 'ok' + ? `Kernel capabilities: ${capabilities.capabilities + .filter(({ available }) => available) + .map(({ name }) => name) + .join(', ')}.` + : 'Kernel capabilities unavailable.', + health.status === 'healthy' + ? 'Recovery guidance: none required.' + : 'Recovery guidance: run /memory repair for a non-destructive projection preview.', + ], + health.status === 'healthy' ? 'success' : 'warning', + ) + } - if (parsed.command === 'inspect') { - const inventory = await canonicalInventory(v2, parsed.args[0] ? 10_000 : 100) - if (inventory.error) return commandError(`Memory inspect failed: ${safeOperationMessage(inventory.error.message)}`) - const selected = parsed.args[0] ? inventory.events.filter(({ eventId }) => eventId === parsed.args[0]) : inventory.events - if (parsed.args[0] && selected.length === 0) return commandError('The requested canonical event was not found in the bounded inventory.') - return report('Memory V2 canonical events', selected.slice(0, 100).map((event) => `#${event.sequence} ${event.eventId} · ${event.eventType} · ${event.occurredAt}`)) - } + if (parsed.command === 'query') { + const text = parsed.args.join(' ').trim() + if (!text) return commandError(MEMORY_USAGE) + const outcome = await v2.repository.query({ + ...scope, + queryId: QueryIdSchema.parse(deterministicId('query', text)), + query: text, + selectors: [], + artifactKinds: [], + includeHistorical: false, + maxResultsPerCategory: 10, + }) + if (outcome.outcome !== 'result') + return commandError( + `Memory query failed: ${safeOperationMessage(outcome.error.message)} Retryable: ${outcome.error.retryable ? 'yes' : 'no'}.`, + ) + const result = outcome.result + const lines = [ + `Tasks: ${result.matchedTasks.length}; verified: ${result.verifiedKnowledge.length}; discoveries: ${result.reusableDiscovery.length}; reread: ${result.rereadRequired.length}; historical: ${result.historicalContext.length}.`, + ...result.matchedTasks.map( + (item) => + `Task ${item.taskId} [${item.score.toFixed(2)}]: ${item.title}.`, + ), + ...result.verifiedKnowledge.map( + (item) => + `Verified ${item.observation.observationId} [${item.score.toFixed(2)}]: ${item.observation.summary}. Reasons: ${item.reasons.map(({ code }) => code).join(', ')}.`, + ), + ...result.reusableDiscovery.map( + (item) => + `Discovery ${item.observation.observationId} [${item.score.toFixed(2)}]: ${item.observation.summary}.`, + ), + ...result.rereadRequired.map( + (item) => + `Reread ${item.observationId}: ${'path' in item.selector ? item.selector.path : item.selector.uri} (${item.reason}).`, + ), + `Degradation: ${result.degradation.state === 'none' ? 'none' : result.degradation.reasons.map(({ detail }) => detail).join('; ')}.`, + ] + return report( + 'Memory V2 query', + lines, + result.rereadRequired.length ? 'warning' : 'success', + ) + } - if (parsed.command === 'consolidate') { - const taskIndex = parsed.args.indexOf('--task') - const taskId = taskIndex >= 0 ? parsed.args[taskIndex + 1] : undefined - if (taskIndex >= 0 && !taskId) return commandError(MEMORY_USAGE) - const outcome = await v2.operator.consolidate({ ...scope, ...(taskId ? { taskId } : {}), policyVersion: 'cli-v1', mode: parsed.apply ? 'apply' : 'preview', occurredAt: now, maxGroups: 5 }) - const lines = operationLines(outcome, ['Canonical events are append-only; consolidation does not delete source events.']) - const command = `/memory consolidate${taskId ? ` --task ${taskId}` : ''} --apply` - return report('Memory V2 consolidation', lines, isFailureOutcome(outcome) ? 'error' : 'secondary', parsed.apply ? undefined : [{ label: 'Insert apply command', command }]) - } + if (parsed.command === 'inspect') { + const inventory = await canonicalInventory( + v2, + parsed.args[0] ? 10_000 : 100, + ) + if (inventory.error) + return commandError( + `Memory inspect failed: ${safeOperationMessage(inventory.error.message)}`, + ) + const selected = parsed.args[0] + ? inventory.events.filter(({ eventId }) => eventId === parsed.args[0]) + : inventory.events + if (parsed.args[0] && selected.length === 0) + return commandError( + 'The requested canonical event was not found in the bounded inventory.', + ) + return report( + 'Memory V2 canonical events', + selected + .slice(0, 100) + .map( + (event) => + `#${event.sequence} ${event.eventId} · ${event.eventType} · ${event.occurredAt}`, + ), + ) + } - if (parsed.command === 'repair') { - if (parsed.args.includes('--reset-corrupt')) return commandError('Destructive reset-corrupt is not supported in this release.') - const outcome = await v2.operator.repair({ ...scope, mode: parsed.apply ? 'apply' : 'preview', rebuildId: deterministicId('repair', String(v2.projectId)), projectionNames: ['tasks', 'sessions', 'artifacts', 'claims', 'evidence', 'discoveries'] }) - return report('Memory V2 projection repair', operationLines(outcome, ['Canonical events are never deleted or reset.']), isFailureOutcome(outcome) ? 'error' : 'secondary', parsed.apply ? undefined : [{ label: 'Insert apply command', command: '/memory repair --apply' }]) - } + if (parsed.command === 'consolidate') { + const taskIndex = parsed.args.indexOf('--task') + const taskId = taskIndex >= 0 ? parsed.args[taskIndex + 1] : undefined + if (taskIndex >= 0 && !taskId) return commandError(MEMORY_USAGE) + const outcome = await v2.operator.consolidate({ + ...scope, + ...(taskId ? { taskId } : {}), + policyVersion: 'cli-v1', + mode: parsed.apply ? 'apply' : 'preview', + occurredAt: now, + maxGroups: 5, + }) + const lines = operationLines(outcome, [ + 'Canonical events are append-only; consolidation does not delete source events.', + ]) + const command = `/memory consolidate${taskId ? ` --task ${taskId}` : ''} --apply` + return report( + 'Memory V2 consolidation', + lines, + isFailureOutcome(outcome) ? 'error' : 'secondary', + parsed.apply ? undefined : [{ label: 'Insert apply command', command }], + ) + } - if (parsed.command === 'revalidate') { - if (parsed.args.length !== 2) return commandError(MEMORY_USAGE) - const observationId = parsed.args[0]! - const requestedPath = parsed.args[1]! - const pathParts = requestedPath.split('/') - if (!requestedPath || requestedPath.startsWith('/') || requestedPath.includes('\\') || pathParts.some((part) => !part || part === '.' || part === '..')) { - return commandError('Revalidation requires a contained project-relative path.') + if (parsed.command === 'repair') { + if (parsed.args.includes('--reset-corrupt')) + return commandError( + 'Destructive reset-corrupt is not supported in this release.', + ) + const outcome = await v2.operator.repair({ + ...scope, + mode: parsed.apply ? 'apply' : 'preview', + rebuildId: deterministicId('repair', String(v2.projectId)), + projectionNames: [ + 'tasks', + 'sessions', + 'artifacts', + 'claims', + 'evidence', + 'discoveries', + ], + }) + return report( + 'Memory V2 projection repair', + operationLines(outcome, [ + 'Canonical events are never deleted or reset.', + ]), + isFailureOutcome(outcome) ? 'error' : 'secondary', + parsed.apply + ? undefined + : [ + { + label: 'Insert apply command', + command: '/memory repair --apply', + }, + ], + ) } - const taskId = await lookupObservationTaskId(v2, [observationId]) - if (!taskId) return commandError('The observation task could not be resolved.') - const action = parsed.apply - ? { kind: 'verify' as const, observationId, selector: { kind: 'file' as const, path: requestedPath }, observedDigest: readContainedProjectFile(deps.getRootDir(), requestedPath, EXPORT_MAX_BYTES).digest } - : { kind: 'verify' as const, observationId, selector: { kind: 'file' as const, path: requestedPath } } - const outcome = await v2.operator.revalidate({ ...scope, taskId, mode: parsed.apply ? 'apply' : 'preview', actions: [action] }) - return report('Memory V2 revalidation', operationLines(outcome, [parsed.apply ? 'Live file verification completed.' : 'Selector validated; live file verification is pending apply.']), isFailureOutcome(outcome) ? 'error' : 'secondary', parsed.apply ? undefined : [{ label: 'Insert apply command', command: `/memory revalidate ${parsed.args[0]} ${parsed.args[1]} --apply` }]) - } - if (['correct', 'forget', 'pin'].includes(parsed.command)) { - const observationId = parsed.args[0] - if (!observationId) return commandError(MEMORY_USAGE) - let action: unknown - if (parsed.command === 'forget') { - action = { kind: 'forget', observationIds: parsed.args, reason: 'user-request', requestedBy: 'cli-user', evidenceDisposition: 'remove-references' } - } else if (parsed.command === 'pin') { - if (parsed.args.length !== 1) return commandError(MEMORY_USAGE) - action = { kind: 'pin', observationId, reason: 'Pinned by user from the CLI', pinnedBy: 'cli-user' } - } else { - const replacement = parsed.args.slice(1).join(' ').trim() - if (!replacement) return commandError(MEMORY_USAGE) - const inventory = await canonicalInventory(v2, 10_000) - if (inventory.error) return commandError(`Correction lookup failed: ${safeOperationMessage(inventory.error.message)}`) - const source = inventory.events.find((event) => event.eventType === 'observation.recorded' && event.payload.observation.observationId === observationId) - if (!source || source.eventType !== 'observation.recorded') return commandError('The source observation could not be found; correction was not created.') - const original = source.payload.observation - action = { kind: 'correct', observationId, reason: 'User correction from the CLI', correction: { ...original, observationId: deterministicId('observation', `${observationId}:${replacement}`), summary: replacement.slice(0, 1_024), detail: replacement.slice(0, 16_384), evidence: original.evidence.map(({ excerpt: _excerpt, ...evidence }) => evidence), observedAt: now } } + if (parsed.command === 'revalidate') { + if (parsed.args.length !== 2) return commandError(MEMORY_USAGE) + const observationId = parsed.args[0]! + const requestedPath = parsed.args[1]! + const pathParts = requestedPath.split('/') + if ( + !requestedPath || + requestedPath.startsWith('/') || + requestedPath.includes('\\') || + pathParts.some((part) => !part || part === '.' || part === '..') + ) { + return commandError( + 'Revalidation requires a contained project-relative path.', + ) + } + const taskId = await lookupObservationTaskId(v2, [observationId]) + if (!taskId) + return commandError('The observation task could not be resolved.') + const action = parsed.apply + ? { + kind: 'verify' as const, + observationId, + selector: { kind: 'file' as const, path: requestedPath }, + observedDigest: readContainedProjectFile( + deps.getRootDir(), + requestedPath, + EXPORT_MAX_BYTES, + ).digest, + } + : { + kind: 'verify' as const, + observationId, + selector: { kind: 'file' as const, path: requestedPath }, + } + const outcome = await v2.operator.revalidate({ + ...scope, + taskId, + mode: parsed.apply ? 'apply' : 'preview', + actions: [action], + }) + return report( + 'Memory V2 revalidation', + operationLines(outcome, [ + parsed.apply + ? 'Live file verification completed.' + : 'Selector validated; live file verification is pending apply.', + ]), + isFailureOutcome(outcome) ? 'error' : 'secondary', + parsed.apply + ? undefined + : [ + { + label: 'Insert apply command', + command: `/memory revalidate ${parsed.args[0]} ${parsed.args[1]} --apply`, + }, + ], + ) } - const targetObservationIds = parsed.command === 'forget' ? parsed.args : [observationId] - const taskId = parsed.command === 'correct' && typeof action === 'object' && action !== null && 'correction' in action - ? (action as { correction: { taskId: string } }).correction.taskId - : await lookupObservationTaskId(v2, targetObservationIds) - if (!taskId) return commandError('The observation task could not be resolved.') - const outcome = await v2.operator.correct({ ...scope, taskId, mode: parsed.apply ? 'apply' : 'preview', occurredAt: now, action }) - const applyCommand = `/memory ${parsed.command} ${parsed.args.join(' ')} --apply` - return report(`Memory V2 ${parsed.command}`, operationLines(outcome, ['Preview is the default; apply appends canonical lifecycle events.']), isFailureOutcome(outcome) ? 'error' : 'secondary', parsed.apply ? undefined : [{ label: 'Insert apply command', command: applyCommand }]) - } - if (parsed.command === 'export') return runExport(v2, parsed.args, deps.getRootDir()) - if (parsed.command === 'import') return runImport(v2, parsed.args, parsed.apply, deps.getRootDir()) - return commandError(MEMORY_USAGE) + if (['correct', 'forget', 'pin'].includes(parsed.command)) { + const observationId = parsed.args[0] + if (!observationId) return commandError(MEMORY_USAGE) + let action: unknown + if (parsed.command === 'forget') { + action = { + kind: 'forget', + observationIds: parsed.args, + reason: 'user-request', + requestedBy: 'cli-user', + evidenceDisposition: 'remove-references', + } + } else if (parsed.command === 'pin') { + if (parsed.args.length !== 1) return commandError(MEMORY_USAGE) + action = { + kind: 'pin', + observationId, + reason: 'Pinned by user from the CLI', + pinnedBy: 'cli-user', + } + } else { + const replacement = parsed.args.slice(1).join(' ').trim() + if (!replacement) return commandError(MEMORY_USAGE) + const inventory = await canonicalInventory(v2, 10_000) + if (inventory.error) + return commandError( + `Correction lookup failed: ${safeOperationMessage(inventory.error.message)}`, + ) + const source = inventory.events.find( + (event) => + event.eventType === 'observation.recorded' && + event.payload.observation.observationId === observationId, + ) + if (!source || source.eventType !== 'observation.recorded') + return commandError( + 'The source observation could not be found; correction was not created.', + ) + const original = source.payload.observation + action = { + kind: 'correct', + observationId, + reason: 'User correction from the CLI', + correction: { + ...original, + observationId: deterministicId( + 'observation', + `${observationId}:${replacement}`, + ), + summary: replacement.slice(0, 1_024), + detail: replacement.slice(0, 16_384), + evidence: original.evidence.map( + ({ excerpt: _excerpt, ...evidence }) => evidence, + ), + observedAt: now, + }, + } + } + const targetObservationIds = + parsed.command === 'forget' ? parsed.args : [observationId] + const taskId = + parsed.command === 'correct' && + typeof action === 'object' && + action !== null && + 'correction' in action + ? (action as { correction: { taskId: string } }).correction.taskId + : await lookupObservationTaskId(v2, targetObservationIds) + if (!taskId) + return commandError('The observation task could not be resolved.') + const outcome = await v2.operator.correct({ + ...scope, + taskId, + mode: parsed.apply ? 'apply' : 'preview', + occurredAt: now, + action, + }) + const applyCommand = `/memory ${parsed.command} ${parsed.args.join(' ')} --apply` + return report( + `Memory V2 ${parsed.command}`, + operationLines(outcome, [ + 'Preview is the default; apply appends canonical lifecycle events.', + ]), + isFailureOutcome(outcome) ? 'error' : 'secondary', + parsed.apply + ? undefined + : [{ label: 'Insert apply command', command: applyCommand }], + ) + } + + if (parsed.command === 'export') + return runExport(v2, parsed.args, deps.getRootDir()) + if (parsed.command === 'import') + return runImport(v2, parsed.args, parsed.apply, deps.getRootDir()) + return commandError(MEMORY_USAGE) } finally { if (v2.status === 'available') await v2.release?.() } @@ -522,13 +891,22 @@ async function lookupObservationTaskId( const wanted = new Set(observationIds) const taskIds = new Set() for (const event of inventory.events) { - if (event.eventType === 'observation.recorded' && wanted.has(event.payload.observation.observationId)) { + if ( + event.eventType === 'observation.recorded' && + wanted.has(event.payload.observation.observationId) + ) { taskIds.add(event.payload.observation.taskId) wanted.delete(event.payload.observation.observationId) - } else if (event.eventType === 'claim.consolidated' && wanted.has(event.payload.canonicalObservation.observationId)) { + } else if ( + event.eventType === 'claim.consolidated' && + wanted.has(event.payload.canonicalObservation.observationId) + ) { taskIds.add(event.payload.canonicalObservation.taskId) wanted.delete(event.payload.canonicalObservation.observationId) - } else if (event.eventType === 'claim.corrected' && wanted.has(event.payload.correction.observationId)) { + } else if ( + event.eventType === 'claim.corrected' && + wanted.has(event.payload.correction.observationId) + ) { taskIds.add(event.payload.correction.taskId) wanted.delete(event.payload.correction.observationId) } @@ -543,50 +921,103 @@ async function runExport( ): Promise { const formatIndex = args.indexOf('--format') const format = formatIndex >= 0 ? args[formatIndex + 1] : 'json' - if (format !== 'json' && format !== 'markdown') return commandError('Export format must be json or markdown.') - const outcome = await v2.operator.exportManifest({ schemaVersion: 2, projectId: v2.projectId, generatedAt: new Date().toISOString(), includeStale: args.includes('--include-stale'), rendering: format === 'markdown' ? 'json-and-markdown' : 'json' }) - if (outcome.outcome !== 'exported') return commandError(`Memory export failed: ${safeOperationMessage(outcome.error.message)} Retryable: ${outcome.error.retryable ? 'yes' : 'no'}.`) + if (format !== 'json' && format !== 'markdown') + return commandError('Export format must be json or markdown.') + const outcome = await v2.operator.exportManifest({ + schemaVersion: 2, + projectId: v2.projectId, + generatedAt: new Date().toISOString(), + includeStale: args.includes('--include-stale'), + rendering: format === 'markdown' ? 'json-and-markdown' : 'json', + }) + if (outcome.outcome !== 'exported') + return commandError( + `Memory export failed: ${safeOperationMessage(outcome.error.message)} Retryable: ${outcome.error.retryable ? 'yes' : 'no'}.`, + ) const stem = `memory-v2-${outcome.manifest.checksum.slice(7, 23)}` const jsonName = `${stem}.json` const json = `${JSON.stringify(outcome.manifest, null, 2)}\n` - if (Buffer.byteLength(json) > EXPORT_MAX_BYTES) return commandError('Memory export exceeds the bounded file size.') - const directory = createContainedProjectDirectory(root, '.openbuff/memory/exports') + if (Buffer.byteLength(json) > EXPORT_MAX_BYTES) + return commandError('Memory export exceeds the bounded file size.') + const directory = createContainedProjectDirectory( + root, + '.openbuff/memory/exports', + ) let jsonCreated = false try { try { directory.writeExclusive(jsonName, json) jsonCreated = true } catch (error) { - if (!(error instanceof ContainedFileIoError) || error.code !== 'exists') throw error - const existing = readContainedProjectFile(root, `.openbuff/memory/exports/${jsonName}`, EXPORT_MAX_BYTES).text + if (!(error instanceof ContainedFileIoError) || error.code !== 'exists') + throw error + const existing = readContainedProjectFile( + root, + `.openbuff/memory/exports/${jsonName}`, + EXPORT_MAX_BYTES, + ).text if (existing !== json) throw error } const paths = [`.openbuff/memory/exports/${jsonName}`] if (format === 'markdown') { - if (!outcome.markdown || Buffer.byteLength(outcome.markdown) > EXPORT_MAX_BYTES) throw new ContainedFileIoError('too-large') + if ( + !outcome.markdown || + Buffer.byteLength(outcome.markdown) > EXPORT_MAX_BYTES + ) + throw new ContainedFileIoError('too-large') const markdownName = `${stem}.md` try { directory.writeExclusive(markdownName, outcome.markdown) } catch (error) { - if (!(error instanceof ContainedFileIoError) || error.code !== 'exists') throw error - const existingMarkdown = readContainedProjectFile(root, `.openbuff/memory/exports/${markdownName}`, EXPORT_MAX_BYTES).text + if (!(error instanceof ContainedFileIoError) || error.code !== 'exists') + throw error + const existingMarkdown = readContainedProjectFile( + root, + `.openbuff/memory/exports/${markdownName}`, + EXPORT_MAX_BYTES, + ).text if (existingMarkdown !== outcome.markdown) throw error } paths.push(`.openbuff/memory/exports/${markdownName}`) } - return report('Memory V2 export', [`Created: ${paths.join(', ')}.`, `Checksum: ${outcome.manifest.checksum}.`, `Canonical events: ${outcome.manifest.canonicalEventCount}; exported: ${outcome.manifest.events.length}.`, `Warnings: ${outcome.manifest.warnings.length ? outcome.manifest.warnings.slice(0, 20).map(safeOperationMessage).join('; ') : 'none'}.`], 'success') + return report( + 'Memory V2 export', + [ + `Created: ${paths.join(', ')}.`, + `Checksum: ${outcome.manifest.checksum}.`, + `Canonical events: ${outcome.manifest.canonicalEventCount}; exported: ${outcome.manifest.events.length}.`, + `Warnings: ${outcome.manifest.warnings.length ? outcome.manifest.warnings.slice(0, 20).map(safeOperationMessage).join('; ') : 'none'}.`, + ], + 'success', + ) } catch { if (jsonCreated) directory.remove(jsonName) - return commandError('Memory export failed: local output could not be created safely.') + return commandError( + 'Memory export failed: local output could not be created safely.', + ) } finally { directory.close() } } function stableManifestJson(value: unknown): string { - if (value === null || typeof value === 'boolean' || typeof value === 'number' || typeof value === 'string') return JSON.stringify(value) - if (Array.isArray(value)) return `[${value.map(stableManifestJson).join(',')}]` - if (typeof value === 'object') return `{${Object.keys(value as Record).sort().map((key) => `${JSON.stringify(key)}:${stableManifestJson((value as Record)[key])}`).join(',')}}` + if ( + value === null || + typeof value === 'boolean' || + typeof value === 'number' || + typeof value === 'string' + ) + return JSON.stringify(value) + if (Array.isArray(value)) + return `[${value.map(stableManifestJson).join(',')}]` + if (typeof value === 'object') + return `{${Object.keys(value as Record) + .sort() + .map( + (key) => + `${JSON.stringify(key)}:${stableManifestJson((value as Record)[key])}`, + ) + .join(',')}}` return JSON.stringify(null) } @@ -597,22 +1028,72 @@ async function runImport( root: string, ): Promise { if (args.length !== 1) return commandError(MEMORY_USAGE) - if (!args[0]!.toLowerCase().endsWith('.json')) return commandError('Memory import rejected: Import requires a project-relative JSON path.') + if (!args[0]!.toLowerCase().endsWith('.json')) + return commandError( + 'Memory import rejected: Import requires a project-relative JSON path.', + ) try { - const manifest = MemoryExportManifestV2Schema.parse(JSON.parse(readContainedProjectFile(root, args[0]!, EXPORT_MAX_BYTES).text)) - if (manifest.projectId !== v2.projectId) return commandError('Memory import rejected: Manifest project does not match the current project.') + const manifest = MemoryExportManifestV2Schema.parse( + JSON.parse( + readContainedProjectFile(root, args[0]!, EXPORT_MAX_BYTES).text, + ), + ) + if (manifest.projectId !== v2.projectId) + return commandError( + 'Memory import rejected: Manifest project does not match the current project.', + ) const { checksum: _checksum, ...checksumManifest } = manifest const checksum = `sha256:${createHash('sha256').update(stableManifestJson(checksumManifest)).digest('hex')}` - if (checksum !== manifest.checksum) return commandError('Memory import rejected: Manifest checksum is invalid.') - if (!apply) return report('Memory V2 import preview', [`Validated ${args[0]}.`, `Events: ${manifest.events.length}; checksum: ${manifest.checksum}.`, 'No writes were performed.'], 'secondary', [{ label: 'Insert apply command', command: `/memory import ${args[0]} --apply` }]) - const outcome = await v2.operator.importManifest({ schemaVersion: 2, projectId: v2.projectId, manifest, rebuildId: deterministicId('import', manifest.checksum), projectionNames: ['tasks', 'sessions', 'artifacts', 'claims', 'evidence', 'discoveries'], pageSize: 100 }) - return report('Memory V2 import', operationLines(outcome, [`Source: ${args[0]}.`]), outcome.outcome === 'imported' || outcome.outcome === 'no-op' ? 'success' : 'error') + if (checksum !== manifest.checksum) + return commandError( + 'Memory import rejected: Manifest checksum is invalid.', + ) + if (!apply) + return report( + 'Memory V2 import preview', + [ + `Validated ${args[0]}.`, + `Events: ${manifest.events.length}; checksum: ${manifest.checksum}.`, + 'No writes were performed.', + ], + 'secondary', + [ + { + label: 'Insert apply command', + command: `/memory import ${args[0]} --apply`, + }, + ], + ) + const outcome = await v2.operator.importManifest({ + schemaVersion: 2, + projectId: v2.projectId, + manifest, + rebuildId: deterministicId('import', manifest.checksum), + projectionNames: [ + 'tasks', + 'sessions', + 'artifacts', + 'claims', + 'evidence', + 'discoveries', + ], + pageSize: 100, + }) + return report( + 'Memory V2 import', + operationLines(outcome, [`Source: ${args[0]}.`]), + outcome.outcome === 'imported' || outcome.outcome === 'no-op' + ? 'success' + : 'error', + ) } catch (error) { - const message = error instanceof ContainedFileIoError && error.code === 'invalid-path' - ? 'Import requires a project-relative JSON path.' - : error instanceof ContainedFileIoError && ['missing', 'not-regular', 'too-large'].includes(error.code) - ? 'Import file is missing, not regular, or too large.' - : 'The input could not be validated safely.' + const message = + error instanceof ContainedFileIoError && error.code === 'invalid-path' + ? 'Import requires a project-relative JSON path.' + : error instanceof ContainedFileIoError && + ['missing', 'not-regular', 'too-large'].includes(error.code) + ? 'Import file is missing, not regular, or too large.' + : 'The input could not be validated safely.' return commandError(`Memory import rejected: ${message}`) } } @@ -620,39 +1101,67 @@ async function runImport( async function canonicalInventory( v2: Extract>, { status: 'available' }>, maximum: number, -): Promise<{ events: import('@openbuff/sdk').MemoryEventEnvelope[]; error?: { message: string } }> { +): Promise<{ + events: import('@openbuff/sdk').MemoryEventEnvelope[] + error?: { message: string } +}> { const events: import('@openbuff/sdk').MemoryEventEnvelope[] = [] const cursors = new Set() let afterEventId: import('@openbuff/sdk').MemoryEventId | undefined const pageBound = Math.min(100, Math.ceil(maximum / 1_000) + 1) - for (let pageIndex = 0; pageIndex < pageBound && events.length < maximum; pageIndex++) { + for ( + let pageIndex = 0; + pageIndex < pageBound && events.length < maximum; + pageIndex++ + ) { const page = await v2.repository.export({ schemaVersion: 2, projectId: v2.projectId, ...(afterEventId ? { afterEventId } : {}), limit: Math.min(1_000, maximum - events.length), }) - if (page.outcome !== 'page') return { events: [], error: { message: 'Canonical inventory is temporarily unavailable.' } } + if (page.outcome !== 'page') + return { + events: [], + error: { message: 'Canonical inventory is temporarily unavailable.' }, + } const parsedEvents: import('@openbuff/sdk').MemoryEventEnvelope[] = [] for (const candidate of page.events) { const parsed = MemoryEventEnvelopeSchema.safeParse(candidate) - if (!parsed.success || parsed.data.projectId !== v2.projectId) return { events: [], error: { message: 'Canonical inventory failed validation.' } } + if (!parsed.success || parsed.data.projectId !== v2.projectId) + return { + events: [], + error: { message: 'Canonical inventory failed validation.' }, + } parsedEvents.push(parsed.data) } - if (page.nextAfterEventId && parsedEvents.length === 0) return { events: [], error: { message: 'Canonical inventory pagination did not advance.' } } + if (page.nextAfterEventId && parsedEvents.length === 0) + return { + events: [], + error: { message: 'Canonical inventory pagination did not advance.' }, + } if (page.nextAfterEventId) { if ( page.nextAfterEventId === afterEventId || cursors.has(page.nextAfterEventId) || page.nextAfterEventId !== parsedEvents.at(-1)?.eventId - ) return { events: [], error: { message: 'Canonical inventory pagination did not advance.' } } + ) + return { + events: [], + error: { message: 'Canonical inventory pagination did not advance.' }, + } cursors.add(page.nextAfterEventId) } events.push(...parsedEvents) if (!page.nextAfterEventId) return { events } afterEventId = page.nextAfterEventId } - return events.length >= maximum ? { events } : { events: [], error: { message: 'Canonical inventory exceeded its page bound.' } } + return events.length >= maximum + ? { events } + : { + events: [], + error: { message: 'Canonical inventory exceeded its page bound.' }, + } } export async function handleMemoryCommandBlocks( @@ -665,7 +1174,9 @@ export async function handleMemoryCommandBlocks( if (parsed.command === 'status') return await runStatusBlock(deps) return await runV2Command(rawArgs, deps) } catch { - return commandError('The memory operation could not be completed safely. Please retry.') + return commandError( + 'The memory operation could not be completed safely. Please retry.', + ) } } @@ -685,43 +1196,46 @@ async function runStatusBlock( if (deps.getMemoryV2) { const v2 = await getV2(deps) try { - if (v2.status === 'available') { - const [health, kernel, inventory] = await Promise.all([ - v2.repository.health({ schemaVersion: 2, projectId: v2.projectId }), - v2.repository.kernelHealth(), - canonicalInventory(v2, 1_000), - ]) - const migration = !inventory.error - ? [...inventory.events].reverse().find((event) => event.eventType === 'migration.v1.imported') - : undefined - const migrationPayload = migration - ? V1MigrationPayloadSchema.safeParse(migration.payload) - : undefined - v2Lines = [ - `Memory authority: requested ${v2.requestedAuthority}; active ${v2.effectiveAuthority}.`, - RELEASE_N_AUTHORITY_WARNING, - `V1 compatibility shadow: ${context ? 'available' : 'not yet available'}.`, - `Memory V2: ${health.status}; repository authority ${health.authority.kind}; backend ${health.backend.backendId}.`, - migrationPayload?.success - ? `Last V1 import: revision ${migrationPayload.data.sourceRevision ?? 'unknown'}; checksum ${migrationPayload.data.sourceChecksum?.slice(0, 24) ?? 'unknown'}; identity ${migrationPayload.data.legacyRecordKey.slice(0, 64)}.` - : 'Last V1 import: unavailable.', - 'Last parity: unavailable in this command session.', - `Schema ${kernel.schemaVersion ?? 'unknown'}; capabilities ${health.backend.capabilities.join(', ')}; events ${inventory.error ? 'unavailable' : inventory.events.length}; projection ${kernel.projectionCursor ?? 'unknown'}.`, - `Degradation: ${health.issues.length ? health.issues.map(safeOperationMessage).join('; ') : 'none'}.`, - ] - } else { - v2Lines = [ - `Memory authority: requested ${v2.requestedAuthority}; active ${v2.effectiveAuthority}.`, - RELEASE_N_AUTHORITY_WARNING, - `V1 compatibility shadow: ${context ? 'available' : 'not yet available'}.`, - `Memory V2: ${safeOperationMessage(v2.degradation)}`, - ] - } + if (v2.status === 'available') { + const [health, kernel, inventory] = await Promise.all([ + v2.repository.health({ schemaVersion: 2, projectId: v2.projectId }), + v2.repository.kernelHealth(), + canonicalInventory(v2, 1_000), + ]) + const migration = !inventory.error + ? [...inventory.events] + .reverse() + .find((event) => event.eventType === 'migration.v1.imported') + : undefined + const migrationPayload = migration + ? V1MigrationPayloadSchema.safeParse(migration.payload) + : undefined + v2Lines = [ + `Memory authority: requested ${v2.requestedAuthority}; active ${v2.effectiveAuthority}.`, + RELEASE_N_AUTHORITY_WARNING, + `V1 compatibility shadow: ${context ? 'available' : 'not yet available'}.`, + `Memory V2: ${health.status}; repository authority ${health.authority.kind}; backend ${health.backend.backendId}.`, + migrationPayload?.success + ? `Last V1 import: revision ${migrationPayload.data.sourceRevision ?? 'unknown'}; checksum ${migrationPayload.data.sourceChecksum?.slice(0, 24) ?? 'unknown'}; identity ${migrationPayload.data.legacyRecordKey.slice(0, 64)}.` + : 'Last V1 import: unavailable.', + 'Last parity: unavailable in this command session.', + `Schema ${kernel.schemaVersion ?? 'unknown'}; capabilities ${health.backend.capabilities.join(', ')}; events ${inventory.error ? 'unavailable' : inventory.events.length}; projection ${kernel.projectionCursor ?? 'unknown'}.`, + `Degradation: ${health.issues.length ? health.issues.map(safeOperationMessage).join('; ') : 'none'}.`, + ] + } else { + v2Lines = [ + `Memory authority: requested ${v2.requestedAuthority}; active ${v2.effectiveAuthority}.`, + RELEASE_N_AUTHORITY_WARNING, + `V1 compatibility shadow: ${context ? 'available' : 'not yet available'}.`, + `Memory V2: ${safeOperationMessage(v2.degradation)}`, + ] + } } finally { if (v2.status === 'available') await v2.release?.() } } - if (!context) return { type: 'memory', state: 'empty', ...(v2Lines ? { v2Lines } : {}) } + if (!context) + return { type: 'memory', state: 'empty', ...(v2Lines ? { v2Lines } : {}) } return { type: 'memory', state: 'status', diff --git a/common/knowledge.md b/common/knowledge.md index 79c268e2e1..2945cb09a4 100644 --- a/common/knowledge.md +++ b/common/knowledge.md @@ -26,8 +26,8 @@ This package contains code shared across the Openbuff monorepo, especially the l - **Provider config**: Shared validation and contracts for OpenAI-compatible, Anthropic-compatible, and other BYOK providers - **Workspace state**: `common/src/types/workspace-state.ts` tracks monotonically increasing local workspace revisions and bounded change records so SDK/runtime tools can tie mutation receipts, review bundles, and snapshot-aware validation to a stable `workspace.v1..` identifier. - **Task memory**: `common/src/types/task-memory.ts` defines bounded Zod schemas for durable requirements, evidence, blockers, validation, and review notes that survive long task execution without leaking unbounded context. `TASK_MEMORY_LIST_CAPS` is exported alongside the schema as the single source of truth for per-list array caps, so consumers (notably the SDK task-memory store that merges a finished run's memory into the persisted record) cannot hand-mirror limits and drift from what the schema enforces. -- **Published task-memory API (`@openbuff/sdk`)**: the SDK publishes `loadPersistedTaskMemory`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, `auditTaskMemoryV1Migration`, and `codebuffFsToNodePromises`, together with the complete type closure their signatures name — `TaskMemoryV1`, `TaskMemoryDraftV1`, `TaskMemoryEvidenceV1` (re-exported from `common/src/types/task-memory.ts`) plus `TaskMemoryStoreFs`, `WorkspaceMoveRecord`, `TaskMemoryPruneOutcome`, `V1MigrationAuditReader`, and `V1MigrationAuditOutcome`. The audit API is additive and read-only: it verifies source revision/checksum, marker identity, imported task existence, imported observation IDs, and migration provenance without appending; it does not reconstruct and compare every source-derived body field. `exact` is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`; `not-migrated` means removal is not ready and is not a defect. Bounded outcomes separately represent no record, not migrated, incomplete, mismatch, rejected, and failed. Keep the audit API/types as transitional read-only contracts through rollback, and follow the normative [Memory V1 removal readiness plan](../docs/memory-v1-removal-readiness.md) before any later removal decision. External consumers must be able to _name_ every parameter/return type of a published function, so adding a task-memory export means publishing its types from `sdk/src/index.ts` in the same change. -- **task-memory.json writers**: `.openbuff/memory/task-memory.json` has two writers — `saveMergedTaskMemory` (end of run) and `pruneStaleTaskMemoryEvidence` (`/memory prune`) — and its `revision` must stay monotonic and unique across both. Uniqueness is enforced by serializing each writer's load→revision→commit section on a shared lock: an in-process promise chain keyed by the resolved record path, plus an exclusive-create `.lock` file for other processes that is reclaimed after a staleness timeout. Without it a save landing between prune's revision check and its rename, or two saves loading the same record, published one revision twice with different payloads. Evidence reconciliation deliberately stays outside the locked section because it hashes every evidence file. The lock is advisory — adapters that ignore exclusive-create, or a lock not obtained within the bounded attempt budget, degrade to running unlocked — so both writers keep their own guards: saves re-read the on-disk record and merge against whichever of it and the caller's hydrated `priorMemory` is newer, emitting one past both, and prune reports `status: 'failed', reason: 'concurrent-write'` when the record moved off the revision it loaded. +- **Published task-memory API (`@openbuff/sdk`)**: the SDK publishes `loadPersistedTaskMemory`, `inspectPersistedTaskMemoryV1`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, `auditTaskMemoryV1Migration`, and `codebuffFsToNodePromises`, together with the complete type closure their signatures name — `TaskMemoryV1`, `TaskMemoryDraftV1`, `TaskMemoryEvidenceV1` (re-exported from `common/src/types/task-memory.ts`) plus `TaskMemoryStoreFs`, `TaskMemoryV1Inspection`, `WorkspaceMoveRecord`, `TaskMemoryPruneOutcome`, `V1MigrationAuditReader`, and `V1MigrationAuditOutcome`. The inspector is additive/read-only and distinguishes absent, valid, invalid, and unreadable records; the legacy loader delegates to it but still collapses all non-valid states to `undefined`. The audit reconstructs every deterministic source-derived task/observation event using the exact marker session and compares full normalized headers/payloads plus marker metadata without appending. `exact` is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`; `not-migrated` means removal is not ready and is not a defect. Keep the audit API/types as transitional read-only contracts through rollback, and follow the normative [Memory V1 removal readiness plan](../docs/memory-v1-removal-readiness.md) before any later removal decision. External consumers must be able to _name_ every parameter/return type of a published function, so adding a task-memory export means publishing its types from `sdk/src/index.ts` in the same change. +- **task-memory.json writers**: `.openbuff/memory/task-memory.json` has two writers — `saveMergedTaskMemory` (end of run) and `pruneStaleTaskMemoryEvidence` (`/memory prune`) — and its `revision` must stay monotonic and unique across both. Uniqueness is enforced by serializing each writer's load→revision→commit section on a shared lock: an in-process promise chain keyed by the resolved record path, plus an explicit atomic-create `.lock` file for other processes. A stale same-host lock is reclaimed only after its recorded owner PID is proven dead; live locks are never stolen. Without it a save landing between prune's revision check and its rename, or two saves loading the same record, published one revision twice with different payloads. Evidence reconciliation deliberately stays outside the locked section because it hashes every evidence file. The cross-process lock is mandatory and ownership-token protected: contention or adapters without the explicit atomic-create capability fail closed without writing, acquisition verifies the token it wrote, and cleanup unlinks only the caller's own token. Large evidence hashes are tagged `sha256-whole:`; unversioned legacy one-megabyte-prefix hashes are validated against both legacy and whole digests, then backfilled only after a match, while unknown/unverifiable formats are preserved so prune cannot delete them — while both writers keep their own guards: saves re-read the on-disk record and merge against whichever of it and the caller's hydrated `priorMemory` is newer, emitting one past both, and prune reports `status: 'failed', reason: 'concurrent-write'` when the record moved off the revision it loaded. - **Pruned evidence must not come back**: a revision guard prevents revision reuse, not content resurrection. `/memory prune` runs from inside a live session whose `runMemory.evidence` still holds the entries it dropped, so `saveMergedTaskMemory` also filters the run's contribution: when the on-disk record is the merge base, every evidence id the caller's hydrated snapshot carries but that record no longer does was deliberately removed by the other writer and is dropped from the run's evidence. Evidence the run recorded itself is untouched. - **Prune honors the move contract**: `pruneStaleTaskMemoryEvidence` takes the same optional `workspaceMoves` as `reconcileTaskMemoryEvidence`, because prune DELETES what reconciles stale. Callers that can see journal-recorded moves must pass them, or evidence bound to a renamed file reconciles stale and is permanently lost instead of rebinding to its destination as hydration documents. `collectWorkspaceMoves` is published from `@openbuff/sdk` for exactly that reason: non-run callers need the same journal-derived move set hydration uses. - **`/memory` CLI command** (`cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts` as `memory`, alias `mem`): `/memory status` summarizes the persisted record and its fresh/stale evidence; `/memory prune` drops stale evidence; `/memory audit-migration` invokes the SDK audit against the currently loaded V1 record and leased V2 repository without mutation. Both status/prune subcommands read journal moves via `collectWorkspaceMoves` + `WorkspaceJournalService` and pass them into reconciliation/prune, so a renamed file's evidence rebinds instead of being reported stale and deleted; a journal read that fails surfaces as a command failure rather than a blind prune. Prune reports the store's `TaskMemoryPruneOutcome` faithfully — `no-record` (nothing persisted yet), `pruned` (including `removed: 0`, "nothing to prune"), and `failed` with its reason (schema reject, lost revision race, unwritable target or rename-less filesystem adapter). A failed write is never phrased as an absent record or as "nothing to prune"; it names the cause and reports the stale entries still present. Release N keeps `json-v1` and `shadow-v2` supported but deprecates them in favor of the default/replacement `sqlite-v2-opt-in`; authority selection and compatibility/fallback behavior remain unchanged, with no removal date implied and migration not claimed complete. Any future removal requires a later decision after the canonical readiness gates pass, not merely a current `exact` audit. diff --git a/docs/memory-v1-removal-readiness.md b/docs/memory-v1-removal-readiness.md index c3bc2c0b01..ad09f6f24a 100644 --- a/docs/memory-v1-removal-readiness.md +++ b/docs/memory-v1-removal-readiness.md @@ -8,7 +8,7 @@ This is the canonical readiness plan for a later Memory V1 removal decision. It - No removal date or version is set. Removal, if any, requires a later release decision after every gate below passes. - Migration is not claimed complete. Current evidence is insufficient to authorize removal. - `/memory audit-migration` outcome `not-migrated` means removal is **not ready** for that project. It reports migration state and is not a product defect. -- Outcome `exact` currently means marker identity, imported task existence, imported observation IDs, and migration provenance were verified. It does not mean every source-derived task or observation body field was reconstructed and compared. +- Outcome `exact` means every deterministic source-derived task/observation body field and header, plus marker metadata, was reconstructed from the checksum-verified source and exact marker session and compared after schema normalization. - Even `exact` is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`. Warnings are never waived. Normative terms such as **MUST**, **MUST NOT**, and **SHOULD** define acceptance criteria for a later removal decision. @@ -27,8 +27,8 @@ Memory V1 is removal-ready only when all in-scope projects and supported deploym These blockers are grounded in the current source and make the present answer **not ready**: -1. **Body equality is not proved.** `auditMigrationMarkerBody` in `sdk/src/services/memory-v2/v1-migration.ts` verifies marker identity, imported task existence, imported observation IDs, task association, migration provenance, and the `legacy-v1` tag. It does not reconstruct the import from the V1 source and compare every source-derived task/observation body field. A future audit **MUST** perform that stronger deterministic comparison, with tests that tamper with each imported body field and prove the audit fails closed, before `exact` can authorize removal. -2. **`no-record` is ambiguous.** `loadPersistedTaskMemory` in `sdk/src/services/task-memory-store.ts` returns `undefined` for absent, invalid, unreadable, and checksum-failed records. Therefore `no-record` alone cannot prove that no V1 data exists. Readiness requires a future non-mutating storage inspection result that distinguishes at least `absent`, `valid`, `invalid`, and `unreadable`; only independently verified `absent` can satisfy absence evidence. +1. **Full source-derived body equality is implemented.** `auditMigrationMarkerBody` reconstructs deterministic task/observation drafts from the V1 source and marker session, compares full normalized headers and payloads, and fails closed on missing, duplicate, reordered, substituted, or tampered deterministic evidence. This Gate 0 blocker is complete. +2. **Four-state read-only inspection is implemented.** `inspectPersistedTaskMemoryV1` distinguishes `absent`, checksum-verified `valid`, bounded-reason `invalid`, and `unreadable`; only `ENOENT` is absence. The legacy loader continues collapsing non-valid states for compatibility. Verified inspector `absent` is independent absence evidence. This Gate 0 blocker is complete. 3. **The importer deliberately loses or excludes some data.** `importTaskMemoryV1` warns or omits data including a non-empty goal (`goal-excluded`) and legacy evidence (`legacy-evidence-unverified`), and can omit stale/unsafe/empty/capped data or truncate text. Any warning, omission, or truncation is lossy and cannot authorize removal. Common legitimate warning cases do not create an exception; warnings **MUST NOT** be waived. 4. **Interactive audit is bounded.** `scanV1MigrationAuditEvents` scans at most 10 export pages. `/memory audit-migration` therefore cannot be the sole release-evidence path for repositories beyond that bound. Release evidence requires a full or resumable, schema-validated audit that validates pagination and reaches the canonical end of every in-scope repository. 5. **Post-V1 authority failure semantics are undecided.** Before removal, behavior for legacy or invalid authority values and for V2-open/reset failure **MUST** be explicitly specified, tested, and communicated. It **MUST** fail closed and must not silently report successful V2 authority when canonical V2 storage did not open. The decision must cover `OPENBUFF_MEMORY_AUTHORITY`, SDK authority inputs, provider result types, CLI reporting, and persisted/session authority state. @@ -64,8 +64,8 @@ Each gate is blocking and advances only with recorded evidence and owner approva ### Gate 0: blocker closure -- [ ] Full source-derived task/observation body comparison is implemented and tampered-body tests fail closed. -- [ ] Non-mutating V1 inspection distinguishes `absent`, `valid`, `invalid`, and `unreadable`. +- [x] Full source-derived task/observation body comparison is implemented and tampered-body tests fail closed. +- [x] Non-mutating V1 inspection distinguishes `absent`, `valid`, `invalid`, and `unreadable`. - [ ] A full/resumable validated audit supports stores beyond the interactive 10-page bound. - [ ] Post-V1 authority and V2-open/reset failure semantics are approved and fail closed. - [ ] Lossless evaluation uses the exact zero-omission/zero-truncation/zero-warning predicate. @@ -99,9 +99,11 @@ Each gate is blocking and advances only with recorded evidence and owner approva A later decision **MUST** explicitly inventory and assess, rather than implicitly delete, at least: -- SDK task-memory functions: `loadPersistedTaskMemory`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, and `codebuffFsToNodePromises`; +- SDK task-memory functions: `inspectPersistedTaskMemoryV1`, `loadPersistedTaskMemory`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, and `codebuffFsToNodePromises`; + - SDK migration functions: `importTaskMemoryV1`, `getV1MigrationIdentity`, and the transitional read-only `auditTaskMemoryV1Migration`; -- SDK types: `TaskMemoryStoreFs`, `WorkspaceMoveRecord`, `TaskMemoryPruneOutcome`, `V1MigrationOutcome`, `V1MigrationWarningCode`, `V1MigrationSourceItemCounts`, `V1MigrationAuditReader`, and `V1MigrationAuditOutcome`; +- SDK types: `TaskMemoryStoreFs`, `TaskMemoryV1Inspection`, `WorkspaceMoveRecord`, `TaskMemoryPruneOutcome`, `V1MigrationOutcome`, `V1MigrationWarningCode`, `V1MigrationSourceItemCounts`, `V1MigrationAuditReader`, and `V1MigrationAuditOutcome`; + - common schemas/constants/types: `TASK_MEMORY_LIST_CAPS`, `taskMemoryEvidenceV1Schema`, `taskMemoryDraftV1Schema`, `taskMemoryV1Schema`, `TaskMemoryEvidenceV1`, `TaskMemoryDraftV1`, and `TaskMemoryV1`; - common session/checkpoint contracts that carry V1 import, compatibility-shadow, parity, warning, or authority state, including `MemoryV1ImportWarningCode`, `MemoryV1ImportState`, and `MemoryAuthorityStateV2`; - persisted `.openbuff/memory/task-memory.json` semantics and any runtime hydration, reconciliation, save, prune, shadow, fallback, or migration callers. @@ -210,11 +212,11 @@ Stop rollout and restore the prior compatible artifact/configuration when any of ## Operator migration guidance 1. Keep a known-compatible artifact and the current authority configuration available. -2. Inspect V1 non-mutatingly. If the result is `invalid` or `unreadable`, stop and repair access/record handling; do not interpret it as absence. Until the future inspector exists, treat `no-record` as inconclusive. +2. Inspect V1 non-mutatingly with `inspectPersistedTaskMemoryV1`. If the result is `invalid` or `unreadable`, stop and repair access/record handling; do not interpret it as absence. Only a verified `absent` result is independent absence evidence. 3. Create and verify a V1 backup. Do not migrate in place or delete the source. 4. Import to V2 using the supported path, then run the full/resumable audit. `/memory audit-migration` remains useful interactively but its 10-page bound cannot be sole evidence for larger stores. 5. Stop on `not-migrated`, `incomplete`, `mismatch`, `rejected`, or `failed`. `not-migrated` means readiness work remains; it is not itself a product defect. -6. For `exact`, require `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`, plus the future full source-derived body comparison. Do not waive `goal-excluded`, `legacy-evidence-unverified`, or any other warning. +6. For `exact`, require `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`; full deterministic source-derived body equality is already part of `exact`. Do not waive `goal-excluded`, `legacy-evidence-unverified`, or any other warning. 7. Validate and retain a canonical V2 export, then restore and test it on a disposable copy. 8. If V1 revision/checksum changes, re-import and repeat the fresh lossless audit and export validation. 9. Preserve both stores until the declared rollback window closes. On a trigger, restore the prior artifact/configuration; do not delete either store. @@ -225,7 +227,7 @@ A later removal decision is **GO** only when every applicable item is checked: - [ ] This is a later release decision; Release N was compatibility-only, and the proposed scope/date is explicit in that later record. - [ ] Migration is not assumed complete; every in-scope project has a fresh per-project evidence bundle. -- [ ] All five current blockers are closed with reviewed implementation and tests. +- [ ] All remaining current blockers are closed with reviewed implementation and tests. - [ ] Every valid V1 record has full-body `exact` evidence and the exact lossless predicate passes with no warnings waived. - [ ] Every claimed absent V1 record is independently inspected as `absent`; no decision relies on `no-record` alone. - [ ] No project has `not-migrated`, stale, incomplete, mismatched, rejected, failed, lossy, invalid, unreadable, or partial evidence. diff --git a/sdk/CHANGELOG.md b/sdk/CHANGELOG.md index 238023117a..0835f9580f 100644 --- a/sdk/CHANGELOG.md +++ b/sdk/CHANGELOG.md @@ -6,7 +6,8 @@ All notable changes to the @openbuff/sdk package will be documented in this file ### Added -- New additive, read-only `auditTaskMemoryV1Migration` API and `V1MigrationAuditOutcome` type inspect the currently loaded V1 record against a V2 repository without appending or mutating. An `exact` result verifies the source revision/checksum, marker identity, imported task existence, imported observation IDs, and migration provenance; it does not reconstruct and compare every source-derived body field. It is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`. Other outcomes distinguish no record, not migrated, incomplete, mismatch, rejected, and failed states using bounded reason codes rather than repository error text. +- New additive, read-only `auditTaskMemoryV1Migration` API and `V1MigrationAuditOutcome` type inspect a checksum-verified V1 record against a V2 repository without appending or mutating. An `exact` result reconstructs every deterministic source-derived task/observation event from the source and the marker session, then compares full schema-normalized headers and payloads (excluding only envelope `sequence`) plus all marker metadata. It is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`. Other outcomes distinguish no record, not migrated, incomplete, mismatch, rejected, and failed states using bounded reason codes rather than repository error text. +- New additive, read-only `inspectPersistedTaskMemoryV1` API and `TaskMemoryV1Inspection` type distinguish `absent`, checksum-verified `valid`, `invalid` (malformed JSON, schema invalid, or checksum mismatch), and `unreadable` records without exposing paths, contents, validation issues, or raw errors. The legacy `loadPersistedTaskMemory` signature and behavior are unchanged and now delegate to the inspector, collapsing every non-valid result to `undefined` for compatibility. - Release N deprecation guidance: `json-v1` and `shadow-v2` remain supported compatibility modes but are deprecated; `sqlite-v2-opt-in` is the default and replacement. This release does not remove V1 APIs or persistence, migration import, authority selection, shadow behavior, or fallback behavior, sets no removal date/version, and does not claim migration is complete. Any removal requires a later release decision after the normative gates in the [Memory V1 removal readiness plan](../docs/memory-v1-removal-readiness.md) pass; `not-migrated` means removal is not ready, not that the product is defective. - New published task-memory API for durable cross-session task state: `loadPersistedTaskMemory`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, and `codebuffFsToNodePromises`, backed by `.openbuff/memory/task-memory.json` under the project root. The complete type closure those signatures name is published alongside them so consumers can name every parameter and return type: `TaskMemoryV1`, `TaskMemoryDraftV1`, and `TaskMemoryEvidenceV1` (re-exported from `@codebuff/common/types/task-memory`), plus `TaskMemoryStoreFs`, `WorkspaceMoveRecord`, and `TaskMemoryPruneOutcome`. diff --git a/sdk/src/__tests__/memory-v2-contract.test.ts b/sdk/src/__tests__/memory-v2-contract.test.ts index 5befad1161..8a5d8b7c55 100644 --- a/sdk/src/__tests__/memory-v2-contract.test.ts +++ b/sdk/src/__tests__/memory-v2-contract.test.ts @@ -16,11 +16,14 @@ import type { MemoryRetrievalRequest, MemoryVerifyOutcome, MemoryVerifyRequest, + TaskMemoryV1, + TaskMemoryV1Inspection, V1MigrationAuditOutcome, V1MigrationAuditReader, } from '../index' import { auditTaskMemoryV1Migration, + inspectPersistedTaskMemoryV1, MemoryAppendRequestSchema, MemoryEventDraftSchema, MemoryEventEnvelopeSchema, @@ -45,26 +48,38 @@ class FakeMemoryRepositoryV2 implements MemoryRepositoryV2 { }, } } - const projectEvents = this.events.filter((event) => event.projectId === request.projectId) + const projectEvents = this.events.filter( + (event) => event.projectId === request.projectId, + ) const currentLastEventId = projectEvents.at(-1)?.eventId - const expectedTail = request.expectedTail ?? ( - request.expectedLastEventId === undefined + const expectedTail = + request.expectedTail ?? + (request.expectedLastEventId === undefined ? { kind: 'any' as const } - : { kind: 'event' as const, eventId: request.expectedLastEventId } - ) - const tailMatches = expectedTail.kind === 'any' - || (expectedTail.kind === 'empty' && currentLastEventId === undefined) - || (expectedTail.kind === 'event' && expectedTail.eventId === currentLastEventId) + : { kind: 'event' as const, eventId: request.expectedLastEventId }) + const tailMatches = + expectedTail.kind === 'any' || + (expectedTail.kind === 'empty' && currentLastEventId === undefined) || + (expectedTail.kind === 'event' && + expectedTail.eventId === currentLastEventId) if (!tailMatches) { return { outcome: 'rejected', - error: { code: 'conflict', message: 'The memory store changed.', retryable: true }, + error: { + code: 'conflict', + message: 'The memory store changed.', + retryable: true, + }, } } const stagedDrafts = new Map(this.drafts) const pending: MemoryEventEnvelope[] = [] - const entries: Array<{ eventId: MemoryEventDraft['eventId']; sequence: number; duplicate: boolean }> = [] + const entries: Array<{ + eventId: MemoryEventDraft['eventId'] + sequence: number + duplicate: boolean + }> = [] for (const draft of request.events) { const existingDraft = stagedDrafts.get(draft.eventId) const existingEvent = [...this.events, ...pending].find( @@ -74,10 +89,18 @@ class FakeMemoryRepositoryV2 implements MemoryRepositoryV2 { if (JSON.stringify(existingDraft) !== JSON.stringify(draft)) { return { outcome: 'rejected', - error: { code: 'conflict', message: 'The event ID already exists.', retryable: false }, + error: { + code: 'conflict', + message: 'The event ID already exists.', + retryable: false, + }, } } - entries.push({ eventId: draft.eventId, sequence: existingEvent.sequence, duplicate: true }) + entries.push({ + eventId: draft.eventId, + sequence: existingEvent.sequence, + duplicate: true, + }) continue } @@ -87,7 +110,11 @@ class FakeMemoryRepositoryV2 implements MemoryRepositoryV2 { }) stagedDrafts.set(draft.eventId, draft) pending.push(event) - entries.push({ eventId: draft.eventId, sequence: event.sequence, duplicate: false }) + entries.push({ + eventId: draft.eventId, + sequence: event.sequence, + duplicate: false, + }) } this.events.push(...pending) @@ -121,12 +148,20 @@ class FakeMemoryRepositoryV2 implements MemoryRepositoryV2 { async verify(_request: MemoryVerifyRequest): Promise { return { outcome: 'failed', - error: { code: 'not-found', message: 'Observation not found', retryable: false }, + error: { + code: 'not-found', + message: 'Observation not found', + retryable: false, + }, } } async rebuild(request: MemoryRebuildRequest): Promise { - return { outcome: 'rebuilt', rebuildId: request.rebuildId, processedEvents: this.events.length } + return { + outcome: 'rebuilt', + rebuildId: request.rebuildId, + processedEvents: this.events.length, + } } async health(_request: MemoryHealthRequest): Promise { @@ -139,7 +174,14 @@ class FakeMemoryRepositoryV2 implements MemoryRepositoryV2 { backendId: 'fake', kind: 'in-memory', persistence: 'ephemeral', - capabilities: ['append', 'query', 'verify', 'rebuild', 'health', 'export'], + capabilities: [ + 'append', + 'query', + 'verify', + 'rebuild', + 'health', + 'export', + ], }, issues: [], } @@ -155,6 +197,32 @@ class FakeMemoryRepositoryV2 implements MemoryRepositoryV2 { } describe('MemoryRepositoryV2 public contract', () => { + test('exports the four-state V1 inspector with a nameable exhaustive union', async () => { + const result: TaskMemoryV1Inspection = await inspectPersistedTaskMemoryV1({ + rootDir: '/absent-contract-record', + fs: { + readFile: async () => { + throw Object.assign(new Error('missing'), { code: 'ENOENT' }) + }, + } as never, + }) + const describe = (inspection: TaskMemoryV1Inspection): string => { + switch (inspection.status) { + case 'absent': + return 'absent' + case 'valid': { + const memory: TaskMemoryV1 = inspection.memory + return String(memory.revision) + } + case 'invalid': + return inspection.reason + case 'unreadable': + return inspection.reason + } + } + expect(describe(result)).toBe('absent') + }) + test('the V1 migration audit accepts an export-only reader', async () => { let exportCalls = 0 const reader: V1MigrationAuditReader = { @@ -208,10 +276,12 @@ describe('MemoryRepositoryV2 public contract', () => { const repository: MemoryRepositoryV2 = new FakeMemoryRepositoryV2() const appendOutcome = await repository.append(appendRequest) - const staleEmpty = await repository.append(MemoryAppendRequestSchema.parse({ - ...appendRequest, - expectedTail: { kind: 'empty' }, - })) + const staleEmpty = await repository.append( + MemoryAppendRequestSchema.parse({ + ...appendRequest, + expectedTail: { kind: 'empty' }, + }), + ) const duplicateOutcome = await repository.append(appendRequest) const queryOutcome = await repository.query(queryRequest) const exportOutcome = await repository.export({ diff --git a/sdk/src/__tests__/task-memory-store.test.ts b/sdk/src/__tests__/task-memory-store.test.ts index 50093250cd..6b9aceb390 100644 --- a/sdk/src/__tests__/task-memory-store.test.ts +++ b/sdk/src/__tests__/task-memory-store.test.ts @@ -6,6 +6,8 @@ import { readFile, rm, stat, + symlink, + utimes, writeFile, } from 'node:fs/promises' import * as nodeFsPromises from 'node:fs/promises' @@ -20,6 +22,7 @@ import { stableHash } from '@codebuff/common/util/stable-hash' import { collectWorkspaceMoves, persistRunTaskMemory } from '../run' import { codebuffFsToNodePromises, + inspectPersistedTaskMemoryV1, loadPersistedTaskMemory, pruneStaleTaskMemoryEvidence, reconcileTaskMemoryEvidence, @@ -83,6 +86,77 @@ describe('task-memory-store', () => { await rm(rootDir, { recursive: true, force: true }) }) + test('inspector distinguishes absent, valid, invalid, and unreadable without writes', async () => { + expect(await inspectPersistedTaskMemoryV1({ rootDir })).toEqual({ + status: 'absent', + }) + + const saved = await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory(), + }) + if (!saved) throw new Error('expected saved memory') + expect(await inspectPersistedTaskMemoryV1({ rootDir })).toEqual({ + status: 'valid', + memory: saved, + }) + const memoryPath = path.join( + rootDir, + '.openbuff', + 'memory', + 'task-memory.json', + ) + await writeFile(memoryPath, '{private malformed contents') + expect(await inspectPersistedTaskMemoryV1({ rootDir })).toEqual({ + status: 'invalid', + reason: 'malformed-json', + }) + await writeFile(memoryPath, JSON.stringify({ schemaVersion: 1 })) + expect(await inspectPersistedTaskMemoryV1({ rootDir })).toEqual({ + status: 'invalid', + reason: 'schema-invalid', + }) + await writeFile(memoryPath, JSON.stringify({ ...saved, checksum: 'wrong' })) + expect(await inspectPersistedTaskMemoryV1({ rootDir })).toEqual({ + status: 'invalid', + reason: 'checksum-mismatch', + }) + + let writeCalls = 0 + const readOnlyFs = { + readFile: async () => { + throw Object.assign(new Error('/private/path denied'), { + code: 'EACCES', + }) + }, + mkdir: async () => { + writeCalls++ + }, + rename: async () => { + writeCalls++ + }, + stat: async () => { + throw new Error('unused') + }, + unlink: async () => { + writeCalls++ + }, + writeFile: async () => { + writeCalls++ + }, + } as unknown as import('../services/task-memory-store').TaskMemoryStoreFs + await expect( + inspectPersistedTaskMemoryV1({ rootDir, fs: readOnlyFs }), + ).resolves.toEqual({ + status: 'unreadable', + reason: 'read-failed', + }) + expect(writeCalls).toBe(0) + expect( + await loadPersistedTaskMemory({ rootDir, fs: readOnlyFs }), + ).toBeUndefined() + }) + test('AC1: save then load+reconcile verifies fresh evidence', async () => { await writeFile(path.join(rootDir, 'a.ts'), 'export const a = 1') const memory = makeMemory({ @@ -332,6 +406,139 @@ describe('task-memory-store', () => { } }) + test('ancestor symlink escapes go stale without reading outside the project', async () => { + if (process.platform === 'win32') return + const outsideDir = await mkdtemp( + path.join(tmpdir(), 'task-memory-symlink-outside-'), + ) + try { + const outsideContents = 'outside through ancestor link' + await writeFile(path.join(outsideDir, 'secret.ts'), outsideContents) + await symlink(outsideDir, path.join(rootDir, 'linked')) + + const readFileCalls: string[] = [] + const spyingFs = new Proxy(nodeFsPromises, { + get(target, prop) { + if (prop === 'readFile') { + return async ( + ...args: Parameters + ) => { + readFileCalls.push(String(args[0])) + return nodeFsPromises.readFile(...args) + } + } + return Reflect.get(target, prop) + }, + }) + const reconciled = await reconcileTaskMemoryEvidence({ + memory: makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-ancestor-link', + path: 'linked/secret.ts', + freshnessHash: sha256(outsideContents), + }), + ], + }), + rootDir, + fs: spyingFs, + }) + + expect(reconciled.evidence[0]!.stale).toBe(true) + expect(readFileCalls).toEqual([]) + } finally { + await rm(outsideDir, { recursive: true, force: true }) + } + }) + + test('ancestor replacement after realpath cannot redirect descriptor hashing outside', async () => { + if (process.platform === 'win32') return + const outsideDir = await mkdtemp( + path.join(tmpdir(), 'task-memory-race-outside-'), + ) + try { + await mkdir(path.join(rootDir, 'inside')) + await writeFile(path.join(rootDir, 'inside', 'secret.ts'), 'inside body') + await writeFile(path.join(outsideDir, 'secret.ts'), 'outside secret') + const linkedPath = path.join(rootDir, 'linked') + await symlink(path.join(rootDir, 'inside'), linkedPath) + const candidate = path.join(linkedPath, 'secret.ts') + let replaced = false + const racingFs = new Proxy(nodeFsPromises, { + get(target, prop) { + if (prop === 'realpath') { + return async (requested: string) => { + const resolved = await nodeFsPromises.realpath(requested) + if ( + !replaced && + path.resolve(requested) === path.resolve(candidate) + ) { + replaced = true + await rm(linkedPath) + await symlink(outsideDir, linkedPath) + } + return resolved + } + } + return Reflect.get(target, prop) + }, + }) + + const reconciled = await reconcileTaskMemoryEvidence({ + memory: makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-raced-link', + path: 'linked/secret.ts', + freshnessHash: sha256('outside secret'), + }), + ], + }), + rootDir, + fs: racingFs, + }) + + expect(replaced).toBe(true) + expect(reconciled.evidence[0]!.stale).toBe(true) + } finally { + await rm(outsideDir, { recursive: true, force: true }) + } + }) + + test('small evidence uses bounded descriptor reads rather than pathname readFile', async () => { + const contents = 'small stable body' + await writeFile(path.join(rootDir, 'small.ts'), contents) + let readFileCalls = 0 + const boundedFs = new Proxy(nodeFsPromises, { + get(target, prop) { + if (prop === 'readFile') { + return async () => { + readFileCalls++ + return Buffer.alloc(2_000_000) + } + } + return Reflect.get(target, prop) + }, + }) + + const reconciled = await reconcileTaskMemoryEvidence({ + memory: makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-small-bounded', + path: 'small.ts', + freshnessHash: sha256(contents), + }), + ], + }), + rootDir, + fs: boundedFs, + }) + + expect(reconciled.evidence[0]!.stale).toBe(false) + expect(readFileCalls).toBe(0) + }) + test('saveMergedTaskMemory never throws on schema-invalid run memory', async () => { const saved = await saveMergedTaskMemory({ rootDir, @@ -363,19 +570,27 @@ describe('task-memory-store', () => { test('concurrent saves serialize: unique tmp names, distinct monotonic revisions', async () => { const writtenPaths: string[] = [] - const spyingFs = new Proxy(nodeFsPromises, { - get(target, prop) { - if (prop === 'writeFile') { - return async ( - ...args: Parameters - ) => { - writtenPaths.push(String(args[0])) - return nodeFsPromises.writeFile(...args) + const spyingFs = { + ...new Proxy(nodeFsPromises, { + get(target, prop) { + if (prop === 'writeFile') { + return async ( + ...args: Parameters + ) => { + writtenPaths.push(String(args[0])) + return nodeFsPromises.writeFile(...args) + } } - } - return Reflect.get(target, prop) + return Reflect.get(target, prop) + }, + }), + createFileExclusive: async ( + file: Parameters[0], + data: Parameters[1], + ) => { + await writeFile(file, data, { flag: 'wx', mode: 0o600 }) }, - }) + } const [savedA, savedB] = await Promise.all([ saveMergedTaskMemory({ @@ -420,6 +635,98 @@ describe('task-memory-store', () => { expect([...reloaded!.decisions].sort()).toEqual(['Save A', 'Save B']) }) + test('an old live lock fails closed without stealing or committing', async () => { + const initial = await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ decisions: ['Initial'] }), + }) + const memoryPath = path.join( + rootDir, + '.openbuff', + 'memory', + 'task-memory.json', + ) + const lockPath = `${memoryPath}.lock` + const owner = `${JSON.stringify({ + token: 'external-owner', + pid: process.pid, + createdAt: Date.now() - 60_000, + })}\n` + await writeFile(lockPath, owner, { flag: 'wx', mode: 0o600 }) + const old = new Date(Date.now() - 60_000) + await utimes(lockPath, old, old) + + const blocked = await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ decisions: ['Must not commit'] }), + }) + + expect(blocked).toBeUndefined() + expect(await readFile(lockPath, 'utf8')).toBe(owner) + const reloaded = await loadPersistedTaskMemory({ rootDir }) + expect(reloaded?.revision).toBe(initial?.revision) + expect(reloaded?.decisions).toEqual(['Initial']) + await rm(lockPath) + }) + + test('an old orphaned lock fails closed instead of risking replacement-owner deletion', async () => { + const memoryPath = path.join( + rootDir, + '.openbuff', + 'memory', + 'task-memory.json', + ) + await mkdir(path.dirname(memoryPath), { recursive: true }) + const lockPath = `${memoryPath}.lock` + await writeFile( + lockPath, + `${JSON.stringify({ + token: 'crashed-owner', + pid: 2_147_483_647, + createdAt: Date.now() - 60_000, + })}\n`, + { flag: 'wx', mode: 0o600 }, + ) + const old = new Date(Date.now() - 60_000) + await utimes(lockPath, old, old) + + const saved = await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ decisions: ['Recovered'] }), + }) + + expect(saved).toBeUndefined() + expect(await readFile(lockPath, 'utf8')).toContain('crashed-owner') + }) + + test('adapter that ignores wx cannot enter persistence without explicit exclusive create', async () => { + let renameCalls = 0 + const unsafeAdapter = { + mkdir: async () => {}, + readFile: async () => { + throw Object.assign(new Error('missing'), { code: 'ENOENT' }) + }, + rename: async () => { + renameCalls++ + }, + stat: async () => { + throw Object.assign(new Error('missing'), { code: 'ENOENT' }) + }, + unlink: async () => {}, + // Deliberately accepts and ignores the wx option. + writeFile: async () => {}, + } as unknown as import('../services/task-memory-store').TaskMemoryStoreFs + + expect( + await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory(), + fs: unsafeAdapter, + }), + ).toBeUndefined() + expect(renameCalls).toBe(0) + }) + test('missing renameFile capability degrades to a skipped save with no tmp litter', async () => { // Adapter without the optional renameFile capability (only the members // the store consumes): mirrors virtual-fs hosts that cannot rename @@ -445,9 +752,11 @@ describe('task-memory-store', () => { expect(await readdir(memoryDir)).toEqual([]) }) - test('oversized evidence hashing streams leading bytes instead of buffering the whole file', async () => { + test('large evidence uses a streamed whole-content hash and detects tail changes', async () => { const prefix = 'b'.repeat(1_000_000) - await writeFile(path.join(rootDir, 'huge.ts'), `${prefix}tail-beyond-cap`) + const original = `${prefix}original-tail` + const filePath = path.join(rootDir, 'huge.ts') + await writeFile(filePath, original) const readFileCalls: string[] = [] const spyingFs = new Proxy(nodeFsPromises, { @@ -463,33 +772,37 @@ describe('task-memory-store', () => { return Reflect.get(target, prop) }, }) - const memory = makeMemory({ evidence: [ makeEvidence({ id: 'ev-huge', path: 'huge.ts', - freshnessHash: sha256(prefix), + freshnessHash: sha256(original), }), ], }) + const reconciled = await reconcileTaskMemoryEvidence({ memory, rootDir, fs: spyingFs, }) - // Digest still honors the leading-bytes contract AND the whole-file - // buffered read was never taken on the default node fs path. expect(reconciled.evidence[0]!.stale).toBe(false) expect(readFileCalls).toEqual([]) + + await writeFile(filePath, `${prefix}changed-tail`) + const changed = await reconcileTaskMemoryEvidence({ + memory, + rootDir, + fs: spyingFs, + }) + expect(changed.evidence[0]!.stale).toBe(true) }) - test('codebuffFsToNodePromises forwards host open so adapter-backed hashing streams', async () => { + test('codebuffFsToNodePromises forwards host open for whole-content streaming', async () => { const prefix = 'c'.repeat(1_000_000) - await writeFile( - path.join(rootDir, 'huge-adapter.ts'), - `${prefix}tail-beyond-cap`, - ) + const contents = `${prefix}tail-beyond-cap` + await writeFile(path.join(rootDir, 'huge-adapter.ts'), contents) // Host carrying `open` beyond its published type — what a spread of // fs.promises / createNodeFileSystem() looks like — plus a readFile spy. @@ -517,15 +830,15 @@ describe('task-memory-store', () => { makeEvidence({ id: 'ev-huge-adapter', path: 'huge-adapter.ts', - freshnessHash: sha256(prefix), + freshnessHash: sha256(contents), }), ], }), rootDir, fs: storeFs, }) - // Digest honors the leading-bytes contract AND the buffered whole-file - // read was never taken on the adapter path either. + // The canonical whole-content digest is produced without a buffered + // read on the adapter path. expect(reconciled.evidence[0]!.stale).toBe(false) expect(readFileCalls).toEqual([]) }) @@ -562,10 +875,155 @@ describe('task-memory-store', () => { rootDir, fs: codebuffFsToNodePromises(codebuffFs), }) - // Fail-closed: without a partial-read primitive the multi-GB body is - // never buffered; the unverified entry goes stale instead. + // Fail closed without declaring the evidence stale: inability to verify is + // not proof that the file changed, and prune must preserve it. expect(readFileCalls).toEqual([]) - expect(reconciled.evidence[0]!.stale).toBe(true) + expect(reconciled.evidence[0]!.stale).not.toBe(true) + }) + + test('legacy large-file prefix hashes validate and backfill to versioned whole hashes', async () => { + const prefix = 'p'.repeat(1_000_000) + const contents = `${prefix}legacy-tail` + await writeFile(path.join(rootDir, 'legacy-large.ts'), contents) + const legacyMemory = makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-legacy-large', + path: 'legacy-large.ts', + freshnessHash: sha256(prefix), + }), + ], + }) + + const reconciled = await reconcileTaskMemoryEvidence({ + memory: legacyMemory, + rootDir, + }) + expect(reconciled.evidence[0]).toMatchObject({ + stale: false, + freshnessHash: `sha256-whole:${sha256(contents)}`, + }) + + await saveMergedTaskMemory({ rootDir, runMemory: legacyMemory }) + expect(await pruneStaleTaskMemoryEvidence({ rootDir })).toEqual({ + status: 'pruned', + removed: 0, + remaining: 1, + }) + const persisted = await loadPersistedTaskMemory({ rootDir }) + expect(persisted?.evidence[0]?.freshnessHash).toBe( + `sha256-whole:${sha256(contents)}`, + ) + }) + + test('prune preserves stale large evidence when an unversioned hash does not match', async () => { + const currentContents = `${'c'.repeat(1_000_000)}current-tail` + await writeFile(path.join(rootDir, 'legacy-algorithm.ts'), currentContents) + await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-unversioned-mismatch', + path: 'legacy-algorithm.ts', + // Above the historical boundary, a bare digest cannot say whether + // this mismatch reflects changed content or a legacy prefix + // algorithm. Even an inherited stale verdict is therefore + // insufficient authority for destructive pruning. + freshnessHash: sha256(`${'l'.repeat(1_000_000)}legacy-tail`), + stale: true, + }), + ], + }), + }) + + const persistedBeforePrune = (await loadPersistedTaskMemory({ rootDir }))! + const reconciled = await reconcileTaskMemoryEvidence({ + memory: persistedBeforePrune, + rootDir, + }) + expect(reconciled.evidence[0]?.stale).toBe(true) + + // Reconciliation truthfully reports the mismatch, but prune must not use + // an ambiguous unversioned algorithm as sole authority to delete a present + // file's evidence. + expect(await pruneStaleTaskMemoryEvidence({ rootDir })).toEqual({ + status: 'pruned', + removed: 0, + remaining: 1, + }) + expect((await loadPersistedTaskMemory({ rootDir }))?.evidence[0]?.id).toBe( + 'ev-unversioned-mismatch', + ) + }) + + test('unknown and malformed hash formats cannot turn inherited stale flags into prune authority', async () => { + await writeFile(path.join(rootDir, 'unknown-format.ts'), 'contents') + await writeFile(path.join(rootDir, 'malformed-version.ts'), 'contents') + await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-unknown-format', + path: 'unknown-format.ts', + freshnessHash: 'future-digest-format:value', + stale: true, + }), + makeEvidence({ + id: 'ev-malformed-version', + path: 'malformed-version.ts', + freshnessHash: 'sha256-whole:not-a-valid-digest', + stale: true, + }), + { + id: 'ev-pathless-legacy', + kind: 'read', + summary: 'Legacy evidence without a path', + freshnessHash: 'legacy-unknown-format', + stale: true, + }, + ], + }), + }) + + expect(await pruneStaleTaskMemoryEvidence({ rootDir })).toEqual({ + status: 'pruned', + removed: 0, + remaining: 3, + }) + expect( + (await loadPersistedTaskMemory({ rootDir }))?.evidence.map( + (item) => item.id, + ), + ).toEqual([ + 'ev-unknown-format', + 'ev-malformed-version', + 'ev-pathless-legacy', + ]) + }) + + test('an explicit versioned mismatch authorizes pruning changed present evidence', async () => { + await writeFile(path.join(rootDir, 'changed-versioned.ts'), 'new contents') + await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-versioned-mismatch', + path: 'changed-versioned.ts', + freshnessHash: `sha256-whole:${sha256('old contents')}`, + }), + ], + }), + }) + + expect(await pruneStaleTaskMemoryEvidence({ rootDir })).toEqual({ + status: 'pruned', + removed: 1, + remaining: 0, + }) + expect((await loadPersistedTaskMemory({ rootDir }))?.evidence).toEqual([]) }) test('batched reconciliation preserves order and per-item verdicts past one chunk', async () => { @@ -580,10 +1038,10 @@ describe('task-memory-store', () => { makeEvidence({ id: `ev-${index}`, path: name, - // Odd indexes carry a mismatching hash and must flip stale even - // when they land in later batches. + // Odd indexes carry a valid but mismatching whole-content hash and + // must flip stale even when they land in later batches. ...(index % 2 === 1 - ? { freshnessHash: 'mismatched' } + ? { freshnessHash: '0'.repeat(64) } : { freshnessHash: sha256(contents) }), }), ) @@ -600,29 +1058,20 @@ describe('task-memory-store', () => { }) }) - test('AC7: oversized evidence files hash only their leading bytes', async () => { - const prefix = 'a'.repeat(1_000_000) + test('AC7: the streaming boundary retains the whole-content hash contract', async () => { + const contents = 'a'.repeat(1_000_001) const memory = makeMemory({ evidence: [ makeEvidence({ id: 'ev-big', path: 'big.ts', - freshnessHash: sha256(prefix), + freshnessHash: sha256(contents), }), ], }) - await writeFile(path.join(rootDir, 'big.ts'), prefix) + await writeFile(path.join(rootDir, 'big.ts'), contents) const reconciled = await reconcileTaskMemoryEvidence({ memory, rootDir }) expect(reconciled.evidence[0]!.stale).toBe(false) - - // Documented trade-off: mutations beyond the size cap do not flip - // staleness because only the leading bytes feed the digest. - await writeFile(path.join(rootDir, 'big.ts'), `${prefix}tail-changed`) - const afterBeyondCap = await reconcileTaskMemoryEvidence({ - memory, - rootDir, - }) - expect(afterBeyondCap.evidence[0]!.stale).toBe(false) }) }) @@ -878,6 +1327,15 @@ describe('run integration gates', () => { files.delete(oldPath) files.set(newPath, contents) }, + createFileExclusive: async ( + filePath: string, + data: string | Buffer, + ) => { + if (files.has(filePath)) { + throw Object.assign(new Error('exists'), { code: 'EEXIST' }) + } + files.set(filePath, String(data)) + }, stat: async () => ({ size: 0, mode: 0o600 }), unlink: async (filePath: string) => { files.delete(filePath) @@ -1151,17 +1609,57 @@ describe('pruneStaleTaskMemoryEvidence', () => { ), ).toEqual(['ev-moved']) - // Without them the same record reconciles stale and the entry is lost — - // which is exactly why callers must pass the moves they know about. + // The successful compatibility backfill persists the rebound path and + // versioned whole-content hash. A later prune no longer needs the journal + // move to preserve the same evidence. expect(await pruneStaleTaskMemoryEvidence({ rootDir })).toEqual({ status: 'pruned', - removed: 1, - remaining: 0, + removed: 0, + remaining: 1, + }) + expect( + (await loadPersistedTaskMemory({ rootDir }))?.evidence[0], + ).toMatchObject({ + id: 'ev-moved', + path: 'nested/new.ts', + freshnessHash: `sha256-whole:${sha256('moved body')}`, }) - expect((await loadPersistedTaskMemory({ rootDir }))?.evidence).toEqual([]) }) - test('returns zero removals for a fully fresh record without rewriting', async () => { + test('pruning treats evidence through an escaping ancestor symlink as stale', async () => { + if (process.platform === 'win32') return + const outsideDir = await mkdtemp( + path.join(tmpdir(), 'task-memory-prune-outside-'), + ) + try { + const outsideContents = 'outside prune target' + await writeFile(path.join(outsideDir, 'secret.ts'), outsideContents) + await symlink(outsideDir, path.join(rootDir, 'linked')) + await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-prune-link', + path: 'linked/secret.ts', + freshnessHash: sha256(outsideContents), + }), + ], + }), + }) + + expect(await pruneStaleTaskMemoryEvidence({ rootDir })).toEqual({ + status: 'pruned', + removed: 1, + remaining: 0, + }) + expect((await loadPersistedTaskMemory({ rootDir }))?.evidence).toEqual([]) + } finally { + await rm(outsideDir, { recursive: true, force: true }) + } + }) + + test('returns zero removals for a fully fresh versioned record without rewriting', async () => { await writeFile(path.join(rootDir, 'ok.ts'), 'ok') const saved = await saveMergedTaskMemory({ rootDir, @@ -1170,7 +1668,7 @@ describe('pruneStaleTaskMemoryEvidence', () => { makeEvidence({ id: 'ev-ok', path: 'ok.ts', - freshnessHash: sha256('ok'), + freshnessHash: `sha256-whole:${sha256('ok')}`, }), ], }), @@ -1199,6 +1697,14 @@ describe('pruneStaleTaskMemoryEvidence', () => { mkdir: nodeFsPromises.mkdir.bind(nodeFsPromises), readFile: nodeFsPromises.readFile.bind(nodeFsPromises), stat: nodeFsPromises.stat.bind(nodeFsPromises), + realpath: nodeFsPromises.realpath.bind(nodeFsPromises), + createFileExclusive: async ( + file: Parameters[0], + data: Parameters[1], + ) => { + await writeFile(file, data, { flag: 'wx', mode: 0o600 }) + }, + open: nodeFsPromises.open.bind(nodeFsPromises), unlink: nodeFsPromises.unlink.bind(nodeFsPromises), writeFile: nodeFsPromises.writeFile.bind(nodeFsPromises), } as unknown as CodebuffFileSystem @@ -1235,28 +1741,48 @@ describe('pruneStaleTaskMemoryEvidence', () => { }) const before = (await loadPersistedTaskMemory({ rootDir }))! - // Simulate a concurrent save landing while reconciliation hashes evidence: - // the stat call for the (missing) evidence file is the reconcile step, so - // advance the record from there. + // Simulate a concurrent save landing after prune reads its initial record. + // Intercept only task-memory.json (never the ownership-token lock file), + // return the original bytes to prune, and publish the newer revision before + // prune's guarded reload inside the lock. + const memoryPath = path.resolve( + rootDir, + '.openbuff', + 'memory', + 'task-memory.json', + ) let advanced = false - const racingFs: typeof nodeFsPromises = new Proxy(nodeFsPromises, { - get(target, prop) { - if (prop === 'stat') { - return async (...args: Parameters) => { - if (!advanced) { - advanced = true - await saveMergedTaskMemory({ - rootDir, - runMemory: makeMemory({ decisions: ['Concurrent save'] }), - priorMemory: before, - }) + const racingFs = { + ...new Proxy(nodeFsPromises, { + get(target, prop) { + if (prop === 'readFile') { + return async ( + ...args: Parameters + ) => { + const requestedPath = path.resolve(String(args[0])) + if (!advanced && requestedPath === memoryPath) { + const original = await nodeFsPromises.readFile(...args) + advanced = true + await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ decisions: ['Concurrent save'] }), + priorMemory: before, + }) + return original + } + return nodeFsPromises.readFile(...args) } - return nodeFsPromises.stat(...args) } - } - return Reflect.get(target, prop) + return Reflect.get(target, prop) + }, + }), + createFileExclusive: async ( + file: Parameters[0], + data: Parameters[1], + ) => { + await writeFile(file, data, { flag: 'wx', mode: 0o600 }) }, - }) + } as import('../services/task-memory-store').TaskMemoryStoreFs const result = await pruneStaleTaskMemoryEvidence({ rootDir, diff --git a/sdk/src/index.ts b/sdk/src/index.ts index 50d0a7732e..bbcddeb410 100644 --- a/sdk/src/index.ts +++ b/sdk/src/index.ts @@ -71,6 +71,7 @@ export type { AgentDefinition } from '@codebuff/common/templates/initial-agents- export type { ToolName } from '@codebuff/common/tools/constants' export { codebuffFsToNodePromises, + inspectPersistedTaskMemoryV1, loadPersistedTaskMemory, pruneStaleTaskMemoryEvidence, reconcileTaskMemoryEvidence, @@ -79,6 +80,7 @@ export { export type { TaskMemoryPruneOutcome, TaskMemoryStoreFs, + TaskMemoryV1Inspection, WorkspaceMoveRecord, } from './services/task-memory-store' // The persisted-record type closure the task-memory API above is typed with. diff --git a/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts b/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts index 6484f8ac1d..a97770e36d 100644 --- a/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts +++ b/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts @@ -40,9 +40,27 @@ const defaultDraft = { nextActions: ['next'], historicalSummary: 'history', evidence: [ - { id: 'fresh', kind: 'read' as const, summary: 'fresh', path: 'src/live.ts', stale: false }, - { id: 'stale', kind: 'read' as const, summary: 'stale', path: 'src/stale.ts', stale: true }, - { id: 'private', kind: 'read' as const, summary: 'private', path: '.env', stale: false }, + { + id: 'fresh', + kind: 'read' as const, + summary: 'fresh', + path: 'src/live.ts', + stale: false, + }, + { + id: 'stale', + kind: 'read' as const, + summary: 'stale', + path: 'src/stale.ts', + stale: true, + }, + { + id: 'private', + kind: 'read' as const, + summary: 'private', + path: '.env', + stale: false, + }, ], } @@ -53,10 +71,15 @@ function memory(overrides: Partial = {}): TaskMemoryV1 { checksum: checksumOverride, ...draftOverrides } = overrides - const draft = taskMemoryDraftV1Schema.parse({ ...defaultDraft, ...draftOverrides }) + const draft = taskMemoryDraftV1Schema.parse({ + ...defaultDraft, + ...draftOverrides, + }) const checksum = checksumOverride ?? - stableHash(JSON.stringify({ revision, updatedAt: sourceUpdatedAt, memory: draft })) + stableHash( + JSON.stringify({ revision, updatedAt: sourceUpdatedAt, memory: draft }), + ) return { ...draft, revision, updatedAt: sourceUpdatedAt, checksum } } @@ -67,6 +90,7 @@ class Repository implements MemoryRepositoryV2 { exportCalls = 0 failAppendCall: number | undefined failAppendAsConflict = false + beforeFailedAppend?: (repository: Repository) => void appendRequests: Array> = [] async append(input: Parameters[0]) { @@ -74,35 +98,53 @@ class Repository implements MemoryRepositoryV2 { this.appendCalls++ this.appendRequests.push(request) if (this.appendCalls === this.failAppendCall) { + this.beforeFailedAppend?.(this) return this.failAppendAsConflict ? { outcome: 'rejected' as const, - error: { code: 'conflict' as const, message: 'cursor conflict', retryable: true }, + error: { + code: 'conflict' as const, + message: 'cursor conflict', + retryable: true, + }, } : { outcome: 'failed' as const, - error: { code: 'unavailable' as const, message: 'offline', retryable: true }, + error: { + code: 'unavailable' as const, + message: 'offline', + retryable: true, + }, } } const canonicalTail = [...this.events.values()].at(-1)?.eventId - const expectedTail = request.expectedTail ?? ( - request.expectedLastEventId === undefined + const expectedTail = + request.expectedTail ?? + (request.expectedLastEventId === undefined ? { kind: 'any' as const } - : { kind: 'event' as const, eventId: request.expectedLastEventId } - ) - const tailMatches = expectedTail.kind === 'any' - || (expectedTail.kind === 'empty' && canonicalTail === undefined) - || (expectedTail.kind === 'event' && expectedTail.eventId === canonicalTail) + : { kind: 'event' as const, eventId: request.expectedLastEventId }) + const tailMatches = + expectedTail.kind === 'any' || + (expectedTail.kind === 'empty' && canonicalTail === undefined) || + (expectedTail.kind === 'event' && expectedTail.eventId === canonicalTail) if (!tailMatches) { return { outcome: 'rejected' as const, - error: { code: 'conflict' as const, message: 'cursor conflict', retryable: true }, + error: { + code: 'conflict' as const, + message: 'cursor conflict', + retryable: true, + }, } } const staged = new Map(this.events) - const entries: Array<{ eventId: MemoryEventEnvelope['eventId']; sequence: number; duplicate: boolean }> = [] + const entries: Array<{ + eventId: MemoryEventEnvelope['eventId'] + sequence: number + duplicate: boolean + }> = [] let sequence = this.sequence for (const event of request.events) { const existing = staged.get(event.eventId) @@ -111,10 +153,18 @@ class Repository implements MemoryRepositoryV2 { if (JSON.stringify(existingDraft) !== JSON.stringify(event)) { return { outcome: 'rejected' as const, - error: { code: 'conflict' as const, message: 'event content conflict', retryable: false }, + error: { + code: 'conflict' as const, + message: 'event content conflict', + retryable: false, + }, } } - entries.push({ eventId: event.eventId, sequence: existing.sequence, duplicate: true }) + entries.push({ + eventId: event.eventId, + sequence: existing.sequence, + duplicate: true, + }) continue } const envelope = { ...event, sequence: ++sequence } as MemoryEventEnvelope @@ -168,7 +218,8 @@ const audit = (repository: V1MigrationAuditReader, value?: TaskMemoryV1) => const cloneEvent = ( event: MemoryEventEnvelope, overrides: Partial, -): MemoryEventEnvelope => MemoryEventEnvelopeSchema.parse({ ...event, ...overrides }) +): MemoryEventEnvelope => + MemoryEventEnvelopeSchema.parse({ ...event, ...overrides }) const migrationReservations = (repository: Repository) => [...repository.events.values()].filter( @@ -215,7 +266,9 @@ describe('V1 memory migration', () => { ]), ) expect(JSON.stringify(events)).not.toContain(source.goal) - expect(events.some((event) => event.eventType === 'evidence.verified')).toBe(false) + expect( + events.some((event) => event.eventType === 'evidence.verified'), + ).toBe(false) expect(JSON.stringify(events)).not.toContain('src/stale.ts') expect(JSON.stringify(events)).not.toContain('.env') @@ -246,7 +299,10 @@ describe('V1 memory migration', () => { const marker = migrationMarkers(repository)[0] expect(marker?.eventType).toBe('migration.v1.imported') - if (marker?.eventType === 'migration.v1.imported' && outcome.outcome === 'imported') { + if ( + marker?.eventType === 'migration.v1.imported' && + outcome.outcome === 'imported' + ) { expect(marker.payload.sourceItemCounts).toEqual(outcome.sourceItemCounts) expect(marker.payload.truncatedFields).toBe(0) expect(marker.payload.omittedFields).toBe(outcome.omittedFields) @@ -276,18 +332,228 @@ describe('V1 memory migration', () => { const repeated = await run(repository, first) expect(repeated.outcome).toBe('no-op') if (repeated.outcome === 'no-op' && imported.outcome === 'imported') { - expect(repeated.importedObservationIds).toEqual(imported.importedObservationIds) + expect(repeated.importedObservationIds).toEqual( + imported.importedObservationIds, + ) + expect(repeated.sourceItemCounts).toEqual(imported.sourceItemCounts) + expect(repeated.truncatedFields).toBe(imported.truncatedFields) + expect(repeated.lastEventId).toBe( + [...repository.events.values()].at(-1)!.eventId, + ) + } + expect(repository.events.size).toBe(eventCount) + }) + + test('binds an ordinary exact-repeat no-op to the marker whose complete body was audited', async () => { + const repository = new Repository() + const source = memory() + const imported = await run(repository, source) + expect(imported.outcome).toBe('imported') + const canonicalMarker = migrationMarkers(repository)[0]! + if ( + canonicalMarker.eventType !== 'migration.v1.imported' || + imported.outcome !== 'imported' + ) + return + + const forgedMarker = MemoryEventEnvelopeSchema.parse({ + ...canonicalMarker, + eventId: 'event:later-forged-exact-source-marker', + payload: { + ...canonicalMarker.payload, + importedTaskId: 'task:forged', + importedObservationIds: [], + omittedFields: 99, + sourceItemCounts: { requirements: 99 }, + truncatedFields: 99, + warnings: [], + }, + sequence: repository.sequence + 1, + }) + repository.events.set(forgedMarker.eventId, forgedMarker) + repository.sequence = forgedMarker.sequence + const eventCount = repository.events.size + + const repeated = await run(repository, source) + + expect(repeated.outcome).toBe('no-op') + if (repeated.outcome === 'no-op') { + expect(repeated.importedTaskId).toBe(imported.importedTaskId) + expect(repeated.importedObservationIds).toEqual( + imported.importedObservationIds, + ) + expect(repeated.omittedFields).toBe(imported.omittedFields) expect(repeated.sourceItemCounts).toEqual(imported.sourceItemCounts) expect(repeated.truncatedFields).toBe(imported.truncatedFields) - expect(repeated.lastEventId).toBe([...repository.events.values()].at(-1)!.eventId) + expect(repeated.importedTaskId).not.toBe('task:forged') } expect(repository.events.size).toBe(eventCount) }) + test('repairs missing body evidence despite a matching persisted marker', async () => { + const repository = new Repository() + const source = memory() + expect((await run(repository, source)).outcome).toBe('imported') + const missing = [...repository.events.values()].find( + (event) => event.eventType === 'task.created', + )! + repository.events.delete(missing.eventId) + + expect(await run(repository, source)).toMatchObject({ outcome: 'imported' }) + expect(await audit(repository, source)).toMatchObject({ outcome: 'exact' }) + const eventCount = repository.events.size + expect(await run(repository, source)).toMatchObject({ outcome: 'no-op' }) + expect(repository.events.size).toBe(eventCount) + expect(migrationMarkers(repository)).toHaveLength(2) + }) + + test('repairs a tampered marker even when its deterministic body remains complete', async () => { + const repository = new Repository() + const source = memory() + expect((await run(repository, source)).outcome).toBe('imported') + const marker = migrationMarkers(repository)[0]! + if (marker.eventType !== 'migration.v1.imported') return + repository.events.set( + marker.eventId, + MemoryEventEnvelopeSchema.parse({ + ...marker, + payload: { ...marker.payload, omittedFields: 99 }, + }), + ) + + const repaired = await run(repository, source) + + expect(repaired).toMatchObject({ outcome: 'imported' }) + expect(await audit(repository, source)).toMatchObject({ outcome: 'exact' }) + expect(migrationMarkers(repository)).toHaveLength(2) + }) + + test('repairs incompatible deterministic body evidence with new event IDs', async () => { + const repository = new Repository() + const source = memory() + expect((await run(repository, source)).outcome).toBe('imported') + const original = [...repository.events.values()].find( + (event) => event.eventType === 'observation.recorded', + )! + if (original.eventType !== 'observation.recorded') return + repository.events.set( + original.eventId, + MemoryEventEnvelopeSchema.parse({ + ...original, + payload: { + ...original.payload, + observation: { + ...original.payload.observation, + summary: 'older incompatible representation', + }, + }, + }), + ) + + expect(await run(repository, source)).toMatchObject({ outcome: 'imported' }) + expect(await audit(repository, source)).toMatchObject({ outcome: 'exact' }) + expect(migrationMarkers(repository)).toHaveLength(2) + }) + + test('fails closed when a resumed deterministic event ID has corrupt content', async () => { + const repository = new Repository() + const source = memory() + repository.failAppendCall = 3 + expect(await run(repository, source)).toMatchObject({ outcome: 'failed' }) + const task = [...repository.events.values()].find( + (event) => event.eventType === 'task.created', + )! + repository.events.set( + task.eventId, + MemoryEventEnvelopeSchema.parse({ + ...task, + payload: { ...task.payload, title: 'non-migration data' }, + }), + ) + repository.failAppendCall = undefined + const eventCount = repository.events.size + + expect(await run(repository, source)).toMatchObject({ + outcome: 'failed', + reason: 'repository-failed', + }) + expect(repository.events.size).toBe(eventCount) + expect(migrationMarkers(repository)).toHaveLength(0) + }) + + test('does not accept an incomplete exact marker discovered during conflict recovery', async () => { + const repository = new Repository() + const source = memory() + const concurrent = new Repository() + expect((await run(concurrent, source)).outcome).toBe('imported') + const concurrentMarker = migrationMarkers(concurrent)[0]! + repository.failAppendCall = 1 + repository.failAppendAsConflict = true + repository.beforeFailedAppend = (target) => { + target.events.set(concurrentMarker.eventId, concurrentMarker) + target.sequence = concurrentMarker.sequence + } + + expect(await run(repository, source)).toMatchObject({ + outcome: 'failed', + reason: 'repository-failed', + }) + expect( + [...repository.events.values()].filter( + (event) => + event.eventType === 'task.created' || + event.eventType === 'observation.recorded', + ), + ).toHaveLength(0) + }) + + test('binds conflict recovery no-op fields to the validated complete marker', async () => { + const repository = new Repository() + const source = memory() + const concurrent = new Repository() + const imported = await run(concurrent, source) + expect(imported.outcome).toBe('imported') + const canonicalMarker = migrationMarkers(concurrent)[0]! + if (canonicalMarker.eventType !== 'migration.v1.imported') return + const forgedMarker = MemoryEventEnvelopeSchema.parse({ + ...canonicalMarker, + eventId: 'event:concurrent-forged-marker', + payload: { + ...canonicalMarker.payload, + importedTaskId: 'task:forged', + importedObservationIds: [], + }, + sequence: concurrent.sequence + 1, + }) + + repository.failAppendCall = 1 + repository.failAppendAsConflict = true + repository.beforeFailedAppend = (target) => { + for (const event of concurrent.events.values()) { + target.events.set(event.eventId, event) + } + target.events.set(forgedMarker.eventId, forgedMarker) + target.sequence = forgedMarker.sequence + } + + const recovered = await run(repository, source) + expect(recovered.outcome).toBe('no-op') + if (recovered.outcome === 'no-op' && imported.outcome === 'imported') { + expect(recovered.importedTaskId).toBe(imported.importedTaskId) + expect(recovered.importedObservationIds).toEqual( + imported.importedObservationIds, + ) + expect(recovered.importedTaskId).not.toBe('task:forged') + } + }) + test('records all bounded text truncation in the marker and outcome', async () => { const repository = new Repository() const source = memory({ - requirements: ['x'.repeat(1_100), ...Array.from({ length: 17 }, () => 'y'.repeat(1_000))], + requirements: [ + 'x'.repeat(1_100), + ...Array.from({ length: 17 }, () => 'y'.repeat(1_000)), + ], historicalSummary: 'h'.repeat(1_100), evidence: [ { @@ -319,7 +585,9 @@ describe('V1 memory migration', () => { } for (const event of repository.events.values()) { if (event.eventType === 'observation.recorded') { - expect(event.payload.observation.detail.length).toBeLessThanOrEqual(16_384) + expect(event.payload.observation.detail.length).toBeLessThanOrEqual( + 16_384, + ) } } }) @@ -338,7 +606,9 @@ describe('V1 memory migration', () => { expect(forgotten).toHaveLength(first.importedObservationIds.length) expect( forgotten.flatMap((event) => - event.eventType === 'claim.forgotten' ? event.payload.observationIds : [], + event.eventType === 'claim.forgotten' + ? event.payload.observationIds + : [], ), ).toEqual(first.importedObservationIds) for (const event of forgotten) { @@ -367,6 +637,108 @@ describe('V1 memory migration', () => { } }) + test('does not retire claims from a lower-revision marker with forged observation references', async () => { + const repository = new Repository() + expect((await run(repository, memory())).outcome).toBe('imported') + const marker = migrationMarkers(repository)[0]! + if (marker.eventType !== 'migration.v1.imported') return + repository.events.set( + marker.eventId, + MemoryEventEnvelopeSchema.parse({ + ...marker, + payload: { + ...marker.payload, + importedObservationIds: ['observation:user-owned'], + }, + }), + ) + + expect((await run(repository, memory({ revision: 4 }))).outcome).toBe( + 'imported', + ) + expect( + [...repository.events.values()].filter( + (event) => event.eventType === 'claim.forgotten', + ), + ).toEqual([]) + }) + + test('does not retire claims from a lower-revision marker without deterministic migration identity', async () => { + const repository = new Repository() + expect((await run(repository, memory())).outcome).toBe('imported') + const marker = migrationMarkers(repository)[0]! + if (marker.eventType !== 'migration.v1.imported') return + repository.events.set( + marker.eventId, + MemoryEventEnvelopeSchema.parse({ + ...marker, + payload: { + ...marker.payload, + legacyRecordKey: 'v1:forged-lower-revision-marker', + }, + }), + ) + + expect((await run(repository, memory({ revision: 4 }))).outcome).toBe( + 'imported', + ) + expect( + [...repository.events.values()].filter( + (event) => event.eventType === 'claim.forgotten', + ), + ).toEqual([]) + }) + + test('does not retire claims when a lower-revision marker has no migration reservation', async () => { + const repository = new Repository() + expect((await run(repository, memory())).outcome).toBe('imported') + const reservation = migrationReservations(repository)[0]! + repository.events.delete(reservation.eventId) + + expect((await run(repository, memory({ revision: 4 }))).outcome).toBe( + 'imported', + ) + expect( + [...repository.events.values()].filter( + (event) => event.eventType === 'claim.forgotten', + ), + ).toEqual([]) + }) + + test('does not retire claims when a referenced observation is not migration-owned', async () => { + const repository = new Repository() + expect((await run(repository, memory())).outcome).toBe('imported') + const observation = [...repository.events.values()].find( + (event) => event.eventType === 'observation.recorded', + )! + if (observation.eventType !== 'observation.recorded') return + repository.events.set( + observation.eventId, + MemoryEventEnvelopeSchema.parse({ + ...observation, + payload: { + ...observation.payload, + observation: { + ...observation.payload.observation, + provenance: { + ...observation.payload.observation.provenance, + recordedBy: 'user-authored', + }, + }, + }, + }), + ) + + expect((await run(repository, memory({ revision: 4 }))).outcome).toBe( + 'imported', + ) + expect( + [...repository.events.values()].filter( + (event) => event.eventType === 'claim.forgotten', + ), + ).toEqual([]) + }) + test('reports imported when retry commits a marker after the body already committed', async () => { const repository = new Repository() const source = memory() @@ -383,9 +755,13 @@ describe('V1 memory migration', () => { const recovered = await run(repository, source) expect(recovered).toMatchObject({ outcome: 'imported' }) if (recovered.outcome === 'imported') { - expect(recovered.lastEventId).toBe([...repository.events.values()].at(-1)!.eventId) + expect(recovered.lastEventId).toBe( + [...repository.events.values()].at(-1)!.eventId, + ) } - expect([...repository.events.keys()].slice(0, bodyIds.length)).toEqual(bodyIds) + expect([...repository.events.keys()].slice(0, bodyIds.length)).toEqual( + bodyIds, + ) expect(new Set(repository.events.keys()).size).toBe(repository.events.size) expect(migrationMarkers(repository)).toHaveLength(1) }) @@ -438,9 +814,13 @@ describe('V1 memory migration', () => { expect(recovered).toEqual(imported) expect([...recovering.events.keys()]).toEqual([...clean.events.keys()]) expect( - [...recovering.events.values()].map(({ sequence: _sequence, ...event }) => event), + [...recovering.events.values()].map( + ({ sequence: _sequence, ...event }) => event, + ), ).toEqual( - [...clean.events.values()].map(({ sequence: _sequence, ...event }) => event), + [...clean.events.values()].map( + ({ sequence: _sequence, ...event }) => event, + ), ) }) @@ -450,7 +830,11 @@ describe('V1 memory migration', () => { const first = memory() expect(await run(repository, first)).toMatchObject({ outcome: 'failed' }) expect(migrationReservations(repository)).toHaveLength(1) - expect([...repository.events.values()].filter((event) => event.eventType !== 'migration.v1.reserved')).toHaveLength(0) + expect( + [...repository.events.values()].filter( + (event) => event.eventType !== 'migration.v1.reserved', + ), + ).toHaveLength(0) repository.failAppendCall = undefined const changed = memory({ @@ -462,7 +846,11 @@ describe('V1 memory migration', () => { outcome: 'rejected', reason: 'checksum-mismatch', }) - expect([...repository.events.values()].filter((event) => event.eventType !== 'migration.v1.reserved')).toHaveLength(0) + expect( + [...repository.events.values()].filter( + (event) => event.eventType !== 'migration.v1.reserved', + ), + ).toHaveLength(0) expect(await run(repository, first)).toMatchObject({ outcome: 'imported' }) }) @@ -490,7 +878,8 @@ describe('V1 memory migration', () => { expect(outcome.outcome).toBe('imported') expect(repository.appendRequests).toHaveLength(4) - const [reservationPage, firstPage, secondPage, markerPage] = repository.appendRequests + const [reservationPage, firstPage, secondPage, markerPage] = + repository.appendRequests expect(reservationPage.events).toHaveLength(1) expect(reservationPage.events[0]!.eventType).toBe('migration.v1.reserved') expect(reservationPage.expectedTail).toEqual({ kind: 'empty' }) @@ -516,7 +905,9 @@ describe('V1 memory migration audit', () => { test('no record and checksum mismatch make no repository calls', async () => { const repository = new Repository() expect(await audit(repository)).toEqual({ outcome: 'no-record' }) - expect(await audit(repository, memory({ checksum: 'invalid' }))).toMatchObject({ + expect( + await audit(repository, memory({ checksum: 'invalid' })), + ).toMatchObject({ outcome: 'rejected', reason: 'checksum-mismatch', }) @@ -561,9 +952,13 @@ describe('V1 memory migration audit', () => { truncatedFields: 1, }) if (outcome.outcome === 'exact' && imported.outcome === 'imported') { - expect(outcome.markerEventId).toBe([...migrationMarkers(repository)][0]!.eventId) + expect(outcome.markerEventId).toBe( + [...migrationMarkers(repository)][0]!.eventId, + ) expect(outcome.importedTaskId).toBe(imported.importedTaskId) - expect(outcome.importedObservationIds).toEqual(imported.importedObservationIds) + expect(outcome.importedObservationIds).toEqual( + imported.importedObservationIds, + ) expect(outcome.omittedFields).toBe(imported.omittedFields) expect(outcome.sourceItemCounts).toEqual(imported.sourceItemCounts) } @@ -621,7 +1016,8 @@ describe('V1 memory migration audit', () => { const removed = [...repository.events.values()].find( (event) => event.eventType === 'observation.recorded' && - event.payload.observation.observationId === imported.importedObservationIds[0], + event.payload.observation.observationId === + imported.importedObservationIds[0], )! repository.events.delete(removed.eventId) @@ -640,7 +1036,8 @@ describe('V1 memory migration audit', () => { const original = [...repository.events.values()].find( (event) => event.eventType === 'observation.recorded' && - event.payload.observation.observationId === imported.importedObservationIds[0], + event.payload.observation.observationId === + imported.importedObservationIds[0], )! if (original.eventType !== 'observation.recorded') return const mutated = MemoryEventEnvelopeSchema.parse({ @@ -667,6 +1064,192 @@ describe('V1 memory migration audit', () => { }) }) + test('detects full deterministic task, observation, header, and marker tampering', async () => { + const source = memory() + const assertTamper = async ( + select: (repository: Repository) => MemoryEventEnvelope, + mutate: (event: MemoryEventEnvelope) => unknown, + ) => { + const repository = new Repository() + expect((await run(repository, source)).outcome).toBe('imported') + const original = select(repository) + const replacement = MemoryEventEnvelopeSchema.parse(mutate(original)) + repository.events.set(original.eventId, replacement) + expect(await audit(repository, source)).toMatchObject({ + outcome: 'mismatch', + reason: 'imported-body-mismatch', + }) + } + const task = (repository: Repository) => + [...repository.events.values()].find( + (event) => event.eventType === 'task.created', + )! + const observation = (repository: Repository) => + [...repository.events.values()].find( + (event) => event.eventType === 'observation.recorded', + )! + const marker = (repository: Repository) => migrationMarkers(repository)[0]! + + for (const payload of [ + { title: 'tampered' }, + { objective: 'tampered' }, + { initialStatus: 'created' as const, taskId: 'task:tampered' }, + { payloadSchemaVersion: 1 as const, title: 'tampered' }, + ]) { + await assertTamper(task, (event) => ({ + ...event, + payload: { ...event.payload, ...payload }, + })) + } + for (const header of [ + { sessionId: 'session:tampered' }, + { occurredAt: '2027-01-01T00:00:00.000Z' }, + { projectId: 'project:tampered' }, + ]) { + await assertTamper(task, (event) => ({ ...event, ...header })) + } + + const observationMutations: Array< + (value: Record) => Record + > = [ + (value) => ({ ...value, observationId: 'observation:tampered' }), + (value) => ({ ...value, taskId: 'task:tampered' }), + (value) => ({ ...value, kind: 'fact' }), + (value) => ({ ...value, summary: 'tampered' }), + (value) => ({ ...value, detail: 'tampered' }), + (value) => ({ ...value, confidence: 0.5 }), + (value) => ({ + ...value, + evidence: [ + { + artifact: { + artifactId: 'artifact:tampered', + location: 'src/tampered.ts', + classification: { + kind: 'source', + generated: false, + sensitivity: 'internal', + labels: [], + }, + }, + selector: { kind: 'file', path: 'src/tampered.ts' }, + provenance: { + origin: 'migration', + recordedBy: 'tampered', + sourceEventIds: [], + sourceSessionId: sessionId, + metadata: {}, + }, + capturedAt: '2026-09-10T19:41:53.753Z', + }, + ], + }), + (value) => ({ + ...value, + selectors: [{ kind: 'file', path: 'tampered.ts' }], + }), + (value) => ({ + ...value, + provenance: { ...(value.provenance as object), origin: 'derived' }, + }), + (value) => ({ + ...value, + provenance: { ...(value.provenance as object), recordedBy: 'tampered' }, + }), + (value) => ({ + ...value, + provenance: { + ...(value.provenance as object), + sourceEventIds: ['event:tampered'], + }, + }), + (value) => ({ + ...value, + provenance: { + ...(value.provenance as object), + sourceSessionId: 'session:tampered', + }, + }), + (value) => ({ + ...value, + provenance: { + ...(value.provenance as object), + metadata: { + category: 'tampered', + revision: 3, + checksum: source.checksum, + }, + }, + }), + (value) => ({ ...value, tags: ['tampered'] }), + (value) => ({ ...value, observedAt: '2027-01-01T00:00:00.000Z' }), + ] + for (const mutate of observationMutations) { + await assertTamper(observation, (event) => { + if (event.eventType !== 'observation.recorded') return event + return { + ...event, + payload: { + ...event.payload, + observation: mutate( + event.payload.observation as unknown as Record, + ), + }, + } + }) + } + + for (const payload of [ + { importedTaskId: 'task:tampered' }, + { importedObservationIds: [] }, + { importedObservationIds: ['observation:tampered'] }, + { omittedFields: 99 }, + { warnings: [] }, + { sourceItemCounts: { requirements: 99 } }, + { truncatedFields: 99 }, + { legacyRecordKey: 'tampered' }, + ]) { + await assertTamper(marker, (event) => ({ + ...event, + payload: { ...event.payload, ...payload }, + })) + } + }) + + test('allows unrelated events but fails duplicate deterministic IDs', async () => { + const source = memory() + const repository = new Repository() + expect((await run(repository, source)).outcome).toBe('imported') + const template = [...repository.events.values()].find( + (event) => event.eventType === 'task.created', + )! + repository.events.set( + 'event:unrelated', + MemoryEventEnvelopeSchema.parse({ + ...template, + eventId: 'event:unrelated', + payload: { ...template.payload, taskId: 'task:unrelated' }, + sequence: repository.sequence + 1, + }), + ) + expect(await audit(repository, source)).toMatchObject({ outcome: 'exact' }) + + const events = [...repository.events.values()] + const reader: V1MigrationAuditReader = { + async export() { + return { + outcome: 'page', + events: [...events, template], + nextAfterEventId: null, + } + }, + } + expect(await audit(reader, source)).toMatchObject({ + outcome: 'mismatch', + reason: 'imported-body-mismatch', + }) + }) + test('treats a legacy marker without source proof as unverifiable', async () => { const repository = new Repository() const source = memory() @@ -707,7 +1290,11 @@ describe('V1 memory migration audit', () => { async export() { return { outcome: 'rejected', - error: { code: 'invalid-request', message: 'no', retryable: false }, + error: { + code: 'invalid-request', + message: 'no', + retryable: false, + }, } }, }, @@ -735,7 +1322,11 @@ describe('V1 memory migration audit', () => { { reader: { async export() { - return { outcome: 'page', events: 'invalid', nextAfterEventId: null } + return { + outcome: 'page', + events: 'invalid', + nextAfterEventId: null, + } }, } as unknown as V1MigrationAuditReader, expected: { outcome: 'failed', reason: 'invalid-export' }, @@ -743,7 +1334,11 @@ describe('V1 memory migration audit', () => { { reader: { async export() { - return { outcome: 'page', events: [otherProjectEvent], nextAfterEventId: null } + return { + outcome: 'page', + events: [otherProjectEvent], + nextAfterEventId: null, + } }, }, expected: { outcome: 'failed', reason: 'wrong-project' }, @@ -816,7 +1411,11 @@ describe('V1 memory migration audit', () => { sequence: page + 1, }) page++ - return { outcome: 'page', events: [event], nextAfterEventId: event.eventId } + return { + outcome: 'page', + events: [event], + nextAfterEventId: event.eventId, + } }, } @@ -835,16 +1434,20 @@ describe('V1 memory migration audit', () => { const template = canonical.find( (event) => event.eventType === 'task.created', )! - const padding = Array.from({ length: 1_000 - canonical.length }, (_, index) => - cloneEvent(template, { - eventId: MemoryEventIdSchema.parse(`event:audit-padding-${index}`), - sequence: canonical.length + index + 1, - sessionId: MemorySessionIdSchema.parse(`session:audit-padding-${index}`), - payload: { - ...template.payload, - taskId: TaskIdSchema.parse(`task:audit-padding-${index}`), - }, - }), + const padding = Array.from( + { length: 1_000 - canonical.length }, + (_, index) => + cloneEvent(template, { + eventId: MemoryEventIdSchema.parse(`event:audit-padding-${index}`), + sequence: canonical.length + index + 1, + sessionId: MemorySessionIdSchema.parse( + `session:audit-padding-${index}`, + ), + payload: { + ...template.payload, + taskId: TaskIdSchema.parse(`task:audit-padding-${index}`), + }, + }), ) const events = [...canonical, ...padding] let calls = 0 diff --git a/sdk/src/services/memory-v2/v1-migration.ts b/sdk/src/services/memory-v2/v1-migration.ts index f4d19c7c18..6966333177 100644 --- a/sdk/src/services/memory-v2/v1-migration.ts +++ b/sdk/src/services/memory-v2/v1-migration.ts @@ -1,6 +1,7 @@ import { MemoryAppendOutcomeSchema, MemoryAppendRequestSchema, + MemoryEventDraftSchema, MemoryEventIdSchema, MemoryExportOutcomeSchema, ObservationIdSchema, @@ -111,7 +112,9 @@ const boundedText = ( } const hashToken = (value: string): string => - stableHash(value).replace(/[^A-Za-z0-9._:-]/g, '').slice(0, 96) + stableHash(value) + .replace(/[^A-Za-z0-9._:-]/g, '') + .slice(0, 96) /** * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. @@ -135,11 +138,15 @@ function getV1MigrationBodyIdentity(params: { } function reservationEventId(identity: string): MemoryEventId { - return MemoryEventIdSchema.parse(`event:migration:reservation:${hashToken(identity)}`) + return MemoryEventIdSchema.parse( + `event:migration:reservation:${hashToken(identity)}`, + ) } function eventId(identity: string, index: number) { - return MemoryEventIdSchema.parse(`event:migration:${hashToken(`${identity}:${index}`)}`) + return MemoryEventIdSchema.parse( + `event:migration:${hashToken(`${identity}:${index}`)}`, + ) } type MigrationMarkerMetadata = { @@ -152,18 +159,25 @@ type MigrationMarkerMetadata = { truncatedFields?: number } +type ImportedMarkerEvent = Extract< + MemoryEventEnvelope, + { eventType: 'migration.v1.imported' } +> + type MigrationMarkerLookup = | { state: 'none' lastEventId?: MemoryEventId prior?: MigrationMarkerMetadata reserved: boolean + events: MemoryEventEnvelope[] } | { state: 'conflict' lastEventId?: MemoryEventId prior?: MigrationMarkerMetadata reserved: boolean + events: MemoryEventEnvelope[] } | { state: 'exact' @@ -171,9 +185,12 @@ type MigrationMarkerLookup = marker: MigrationMarkerMetadata prior?: MigrationMarkerMetadata reserved: boolean + events: MemoryEventEnvelope[] } -function isV1MigrationWarningCode(value: string): value is V1MigrationWarningCode { +function isV1MigrationWarningCode( + value: string, +): value is V1MigrationWarningCode { switch (value) { case 'goal-excluded': case 'observation-cap-reached': @@ -188,6 +205,183 @@ function isV1MigrationWarningCode(value: string): value is V1MigrationWarningCod } } +function markerMetadata(event: ImportedMarkerEvent): MigrationMarkerMetadata { + return { + eventId: event.eventId, + importedTaskId: event.payload.importedTaskId, + importedObservationIds: [...event.payload.importedObservationIds], + omittedFields: event.payload.omittedFields ?? 0, + warnings: event.payload.warnings.filter(isV1MigrationWarningCode), + ...(event.payload.sourceItemCounts + ? { + sourceItemCounts: markerSourceItemCounts( + event.payload.sourceItemCounts, + ), + } + : {}), + ...(event.payload.truncatedFields === undefined + ? {} + : { truncatedFields: event.payload.truncatedFields }), + } +} + +function isV1MigrationCategory(value: unknown): value is V1MigrationCategory { + return ( + typeof value === 'string' && + (MIGRATION_CATEGORIES as readonly string[]).includes(value) + ) +} + +/** + * Return cleanup authority only for a canonical marker whose reservation and + * complete referenced body have the deterministic shape emitted by this + * migration. Repair markers and migration-looking provenance deliberately do + * not authorize destructive retirement: without the old V1 source they cannot + * be distinguished from user-authored events strongly enough to delete data. + */ +function validatedMigrationOwnedMarkerMetadata(params: { + marker: ImportedMarkerEvent + events: MemoryEventEnvelope[] + projectId: ProjectId +}): MigrationMarkerMetadata | undefined { + const { marker, events, projectId } = params + const { sourceRevision, sourceChecksum, legacyRecordKey, sourceItemCounts } = + marker.payload + if ( + sourceRevision === undefined || + sourceChecksum === undefined || + sourceItemCounts === undefined + ) + return undefined + + const identity = getV1MigrationIdentity({ + projectId, + revision: sourceRevision, + checksum: sourceChecksum, + }) + const bodyIdentity = getV1MigrationBodyIdentity({ + projectId, + revision: sourceRevision, + checksum: sourceChecksum, + }) + const expectedTaskId = TaskIdFor(bodyIdentity) + const expectedObservationIds = marker.payload.importedObservationIds.map( + (_, index) => ObservationIdFor(bodyIdentity, index), + ) + if ( + legacyRecordKey !== identity || + marker.projectId !== projectId || + marker.eventId !== eventId(identity, MAX_OBSERVATIONS + 1) || + marker.payload.payloadSchemaVersion !== 1 || + marker.payload.sourceSchemaVersion !== 1 || + marker.payload.importedTaskId !== expectedTaskId || + expectedObservationIds.some( + (observationId, index) => + observationId !== marker.payload.importedObservationIds[index], + ) + ) + return undefined + + const reservations = events.filter( + (event) => event.eventId === reservationEventId(identity), + ) + if (reservations.length !== 1) return undefined + const reservation = reservations[0]! + if ( + reservation.eventType !== 'migration.v1.reserved' || + reservation.projectId !== projectId || + reservation.sessionId !== marker.sessionId || + reservation.occurredAt !== marker.occurredAt || + reservation.payload.payloadSchemaVersion !== 1 || + reservation.payload.migrationId !== identity || + reservation.payload.sourceRevision !== sourceRevision || + reservation.payload.sourceChecksum !== sourceChecksum + ) + return undefined + + const expectedTaskEventId = eventId(bodyIdentity, 0) + const taskEvents = events.filter( + (event) => event.eventId === expectedTaskEventId, + ) + if (taskEvents.length !== 1) return undefined + const task = taskEvents[0]! + if ( + task.eventType !== 'task.created' || + task.projectId !== projectId || + task.sessionId !== marker.sessionId || + task.occurredAt !== marker.occurredAt || + task.payload.payloadSchemaVersion !== 1 || + task.payload.taskId !== expectedTaskId || + task.payload.title !== + `Imported legacy task memory revision ${sourceRevision}` || + task.payload.objective !== + 'Preserve bounded legacy operational memory without importing its goal.' || + task.payload.initialStatus !== 'created' + ) + return undefined + + let previousCategoryIndex = -1 + const categoryOccurrences = new Map() + for (const [index, observationId] of expectedObservationIds.entries()) { + const expectedEventId = eventId(bodyIdentity, index + 1) + const matching = events.filter((event) => event.eventId === expectedEventId) + if (matching.length !== 1) return undefined + const event = matching[0]! + if (event.eventType !== 'observation.recorded') return undefined + const observation = event.payload.observation + const category = observation.provenance?.metadata.category + if (!isV1MigrationCategory(category)) return undefined + const categoryIndex = MIGRATION_CATEGORIES.indexOf(category) + const occurrences = (categoryOccurrences.get(category) ?? 0) + 1 + categoryOccurrences.set(category, occurrences) + const expectedSummary = + category === 'path-evidence' + ? 'Legacy path evidence requires reread' + : category === 'historical-summary' + ? 'Legacy historical summary' + : `Legacy ${category} items (${sourceItemCounts[category] ?? 0})` + if ( + categoryIndex < previousCategoryIndex || + (occurrences > 1 && category !== 'path-evidence') || + (sourceItemCounts[category] ?? 0) < occurrences || + event.projectId !== projectId || + event.sessionId !== marker.sessionId || + event.occurredAt !== marker.occurredAt || + event.payload.payloadSchemaVersion !== 1 || + observation.observationId !== observationId || + observation.taskId !== expectedTaskId || + observation.kind !== + (category === 'blockers' ? 'warning' : 'discovery') || + observation.summary !== expectedSummary || + observation.detail.length > MAX_AGGREGATE_DETAIL || + observation.confidence !== 0.25 || + observation.evidence.length !== 0 || + observation.observedAt !== marker.occurredAt || + observation.provenance?.origin !== 'migration' || + observation.provenance.recordedBy !== 'sdk-memory-v1-import' || + observation.provenance.sourceEventIds.length !== 0 || + observation.provenance.sourceSessionId !== marker.sessionId || + observation.provenance.metadata.revision !== sourceRevision || + observation.provenance.metadata.checksum !== sourceChecksum || + JSON.stringify(observation.tags) !== + JSON.stringify(['legacy-v1', category, 'unverified']) + ) + return undefined + const selectors = observation.selectors ?? [] + if (category === 'path-evidence') { + if (selectors.length !== 1 || selectors[0]?.kind !== 'file') + return undefined + const decision = classifyMemoryArtifactPath(selectors[0].path) + if (!decision.allowed || !decision.normalizedPath) return undefined + } else if (selectors.length !== 0) { + return undefined + } + previousCategoryIndex = categoryIndex + } + + return markerMetadata(marker) +} + async function findMigrationMarker( repository: MemoryRepositoryV2, projectId: ProjectId, @@ -199,7 +393,8 @@ async function findMigrationMarker( let exact: MigrationMarkerMetadata | undefined let conflict = false let reserved = false - let prior: MigrationMarkerMetadata | undefined + const events: MemoryEventEnvelope[] = [] + const priorCandidates: ImportedMarkerEvent[] = [] for (let page = 0; page < MAX_EXPORT_PAGES; page++) { const outcome = await repository.export({ @@ -208,52 +403,55 @@ async function findMigrationMarker( ...(afterEventId ? { afterEventId } : {}), limit: 1_000, }) - if (outcome.outcome !== 'page') throw new Error('Migration marker lookup failed') + if (outcome.outcome !== 'page') + throw new Error('Migration marker lookup failed') for (const event of outcome.events) { + events.push(event) lastEventId = event.eventId if (event.eventType === 'migration.v1.reserved') { if (event.payload.sourceRevision === revision) { if ( - event.payload.migrationId === getV1MigrationIdentity({ projectId, revision, checksum }) && + event.payload.migrationId === + getV1MigrationIdentity({ projectId, revision, checksum }) && event.payload.sourceChecksum === checksum - ) reserved = true + ) + reserved = true else conflict = true } continue } if (event.eventType !== 'migration.v1.imported') continue - const marker: MigrationMarkerMetadata = { - eventId: event.eventId, - importedTaskId: event.payload.importedTaskId, - importedObservationIds: [...event.payload.importedObservationIds], - omittedFields: event.payload.omittedFields ?? 0, - warnings: event.payload.warnings.filter(isV1MigrationWarningCode), - ...(event.payload.sourceItemCounts - ? { - sourceItemCounts: markerSourceItemCounts( - event.payload.sourceItemCounts, - ), - } - : {}), - ...(event.payload.truncatedFields === undefined - ? {} - : { truncatedFields: event.payload.truncatedFields }), - } const sourceRevision = event.payload.sourceRevision if (sourceRevision === revision) { - if (event.payload.sourceChecksum === checksum) exact = marker + if (event.payload.sourceChecksum === checksum) + exact = markerMetadata(event) else conflict = true } else if (sourceRevision !== undefined && sourceRevision < revision) { - prior = marker + priorCandidates.push(event) } } if (!outcome.nextAfterEventId) { - if (conflict) return { state: 'conflict', lastEventId, prior, reserved } - if (exact && lastEventId) return { state: 'exact', lastEventId, marker: exact, prior, reserved } - return { state: 'none', lastEventId, prior, reserved } + const prior = [...priorCandidates] + .reverse() + .map((marker) => + validatedMigrationOwnedMarkerMetadata({ marker, events, projectId }), + ) + .find((metadata) => metadata !== undefined) + if (conflict) + return { state: 'conflict', lastEventId, prior, reserved, events } + if (exact && lastEventId) + return { + state: 'exact', + lastEventId, + marker: exact, + prior, + reserved, + events, + } + return { state: 'none', lastEventId, prior, reserved, events } } afterEventId = outcome.nextAfterEventId lastEventId = outcome.nextAfterEventId @@ -279,13 +477,294 @@ function outcomeFromMarker(params: { omittedFields: marker.omittedFields, warnings: marker.warnings, lastEventId, - ...(marker.sourceItemCounts ? { sourceItemCounts: marker.sourceItemCounts } : {}), + ...(marker.sourceItemCounts + ? { sourceItemCounts: marker.sourceItemCounts } + : {}), ...(marker.truncatedFields === undefined ? {} : { truncatedFields: marker.truncatedFields }), } } +type V1MigrationBuild = { + identity: string + representationIdentity: string + bodyIdentity: string + occurredAt: string + importedTaskId: TaskId + importedObservationIds: ObservationId[] + omittedFields: number + warnings: V1MigrationWarningCode[] + sourceItemCounts: V1MigrationSourceItemCounts + truncatedFields: number + bodyDrafts: MemoryEventDraft[] + markerDraft: MemoryEventDraft +} + +function normalizedEventDraft(event: MemoryEventEnvelope): MemoryEventDraft { + const { sequence: _sequence, ...draft } = event + return MemoryEventDraftSchema.parse(draft) +} + +function equalEventDraft( + expected: MemoryEventDraft, + actual: MemoryEventEnvelope, +): boolean { + return ( + JSON.stringify(MemoryEventDraftSchema.parse(expected)) === + JSON.stringify(normalizedEventDraft(actual)) + ) +} + +function validatedCompleteBuildMarker( + lookup: Extract, + build: V1MigrationBuild, +): MigrationMarkerMetadata | undefined { + const markerEvents = lookup.events.filter( + (event): event is ImportedMarkerEvent => + event.eventId === build.markerDraft.eventId && + event.eventType === 'migration.v1.imported', + ) + if ( + markerEvents.length !== 1 || + !equalEventDraft(build.markerDraft, markerEvents[0]!) + ) + return undefined + + const completeBody = build.bodyDrafts.every((draft) => { + const matching = lookup.events.filter( + (event) => event.eventId === draft.eventId, + ) + return matching.length === 1 && equalEventDraft(draft, matching[0]!) + }) + return completeBody ? markerMetadata(markerEvents[0]!) : undefined +} + +/** Pure source-to-events transform shared by import and audit. */ +function buildTaskMemoryV1Migration(params: { + memory: TaskMemoryV1 + projectId: ProjectId + sessionId: MemorySessionId + representationIdentity?: string +}): V1MigrationBuild { + const { memory, projectId, sessionId } = params + const { revision, updatedAt, checksum } = memory + const identity = getV1MigrationIdentity({ projectId, revision, checksum }) + const representationIdentity = params.representationIdentity ?? identity + const canonicalBodyIdentity = getV1MigrationBodyIdentity({ + projectId, + revision, + checksum, + }) + const bodyIdentity = + representationIdentity === identity + ? canonicalBodyIdentity + : `${canonicalBodyIdentity}:${hashToken(representationIdentity)}` + const occurredAt = new Date(updatedAt).toISOString() + const importedTaskId = TaskIdFor(bodyIdentity) + const warnings = new Set() + let omittedFields = 0 + let truncatedFields = 0 + if (bounded(memory.goal)) warnings.add('goal-excluded') + + const sourceItemCounts = newSourceItemCounts() + const recordTruncation = () => { + truncatedFields++ + omittedFields++ + warnings.add('text-truncated') + } + const observations: Array<{ + category: V1MigrationCategory + summary: string + detail: string + selector?: { kind: 'file'; path: string } + }> = [] + const addObservation = ( + observation: (typeof observations)[number], + ): boolean => { + if (observations.length >= MAX_OBSERVATIONS) { + omittedFields++ + warnings.add('observation-cap-reached') + return false + } + observations.push(observation) + return true + } + const categories = [ + ['requirements', memory.requirements], + ['decisions', memory.decisions], + ['files-inspected', memory.filesInspected], + ['edits-made', memory.editsMade], + ['validation-results', memory.validationResults], + ['review-receipts', memory.reviewReceipts], + ['blockers', memory.blockers], + ['next-actions', memory.nextActions], + ] as const + for (const [category, values] of categories) { + const nonEmpty: string[] = [] + for (const value of values) { + const item = boundedText(String(value)) + if (item.truncated) recordTruncation() + if (item.value) nonEmpty.push(item.value) + } + if (nonEmpty.length === 0) continue + const aggregate = boundedText(nonEmpty.join('\n'), MAX_AGGREGATE_DETAIL) + if (aggregate.truncated) recordTruncation() + sourceItemCounts[category] += nonEmpty.length + addObservation({ + category, + summary: `Legacy ${category} items (${nonEmpty.length})`, + detail: aggregate.value, + }) + } + + const historical = boundedText(memory.historicalSummary) + if (historical.truncated) recordTruncation() + if (historical.value) { + sourceItemCounts['historical-summary']++ + addObservation({ + category: 'historical-summary', + summary: 'Legacy historical summary', + detail: historical.value, + }) + } + + for (const evidence of memory.evidence) { + warnings.add('legacy-evidence-unverified') + if (evidence.stale === true) { + omittedFields++ + warnings.add('stale-evidence-omitted') + continue + } + if (typeof evidence.path !== 'string') { + omittedFields++ + warnings.add('unsafe-path-omitted') + continue + } + const decision = classifyMemoryArtifactPath(evidence.path) + if (!decision.allowed || !decision.normalizedPath) { + omittedFields++ + warnings.add('unsafe-path-omitted') + continue + } + const summary = boundedText( + evidence.summary || 'Legacy path discovery requires live verification.', + ) + if (summary.truncated) recordTruncation() + sourceItemCounts['path-evidence']++ + addObservation({ + category: 'path-evidence', + summary: 'Legacy path evidence requires reread', + detail: summary.value, + selector: { kind: 'file', path: decision.normalizedPath }, + }) + } + + const bodyDrafts: MemoryEventDraft[] = [ + withMigrationEventId( + createMemoryEventDraft({ + projectId, + sessionId, + userInputId: bodyIdentity, + occurredAt, + eventType: 'task.created', + payload: { + payloadSchemaVersion: 1, + taskId: importedTaskId, + title: `Imported legacy task memory revision ${revision}`, + objective: + 'Preserve bounded legacy operational memory without importing its goal.', + initialStatus: 'created', + }, + }), + bodyIdentity, + 0, + ), + ] + const importedObservationIds: ObservationId[] = [] + observations.forEach((item, index) => { + const observationId = ObservationIdFor(bodyIdentity, index) + importedObservationIds.push(observationId) + bodyDrafts.push( + withMigrationEventId( + createMemoryEventDraft({ + projectId, + sessionId, + userInputId: bodyIdentity, + sourceIndex: index + 1, + occurredAt, + eventType: 'observation.recorded', + payload: { + payloadSchemaVersion: 1, + observation: { + observationId, + taskId: importedTaskId, + kind: item.category === 'blockers' ? 'warning' : 'discovery', + summary: item.summary, + detail: item.detail, + confidence: 0.25, + evidence: [], + ...(item.selector ? { selectors: [item.selector] } : {}), + provenance: { + origin: 'migration', + recordedBy: 'sdk-memory-v1-import', + sourceEventIds: [], + sourceSessionId: sessionId, + metadata: { category: item.category, revision, checksum }, + }, + tags: ['legacy-v1', item.category, 'unverified'], + observedAt: occurredAt, + }, + }, + }), + bodyIdentity, + index + 1, + ), + ) + }) + + const warningList = [...warnings].slice(0, 100) + const markerDraft = withMigrationEventId( + createMemoryEventDraft({ + projectId, + sessionId, + userInputId: representationIdentity, + sourceIndex: MAX_OBSERVATIONS + 1, + occurredAt, + eventType: 'migration.v1.imported', + payload: { + payloadSchemaVersion: 1, + sourceSchemaVersion: 1, + legacyRecordKey: representationIdentity, + importedTaskId, + importedObservationIds, + sourceRevision: revision, + sourceChecksum: checksum, + omittedFields, + sourceItemCounts, + truncatedFields, + warnings: warningList, + }, + }), + representationIdentity, + MAX_OBSERVATIONS + 1, + ) + return { + identity, + representationIdentity, + bodyIdentity, + occurredAt, + importedTaskId, + importedObservationIds, + omittedFields, + warnings: warningList, + sourceItemCounts, + truncatedFields, + bodyDrafts, + markerDraft, + } +} + export type V1MigrationAuditReader = Pick export type V1MigrationAuditOutcome = @@ -344,25 +823,20 @@ type V1MigrationAuditScan = | { outcome: 'complete' events: MemoryEventEnvelope[] + duplicateEventIds: MemoryEventId[] + wrongProjectEventIds: MemoryEventId[] repositoryLastEventId?: MemoryEventId } | Extract -type ImportedMarkerEvent = Extract< - MemoryEventEnvelope, - { eventType: 'migration.v1.imported' } -> -type RecordedObservationEvent = Extract< - MemoryEventEnvelope, - { eventType: 'observation.recorded' } -> - async function scanV1MigrationAuditEvents( repository: V1MigrationAuditReader, projectId: ProjectId, ): Promise { const events: MemoryEventEnvelope[] = [] const eventIds = new Set() + const duplicateEventIds = new Set() + const wrongProjectEventIds = new Set() const cursors = new Set() let afterEventId: MemoryEventId | undefined let repositoryLastEventId: MemoryEventId | undefined @@ -393,20 +867,22 @@ async function scanV1MigrationAuditEvents( } for (const event of exported.events) { - if (event.projectId !== projectId) { - return { outcome: 'failed', reason: 'wrong-project' } - } - if (eventIds.has(event.eventId)) { - return { outcome: 'failed', reason: 'pagination-invalid' } - } - eventIds.add(event.eventId) + if (event.projectId !== projectId) wrongProjectEventIds.add(event.eventId) + if (eventIds.has(event.eventId)) duplicateEventIds.add(event.eventId) + else eventIds.add(event.eventId) events.push(event) repositoryLastEventId = event.eventId } const nextAfterEventId = exported.nextAfterEventId ?? undefined if (!nextAfterEventId) { - return { outcome: 'complete', events, repositoryLastEventId } + return { + outcome: 'complete', + events, + duplicateEventIds: [...duplicateEventIds], + wrongProjectEventIds: [...wrongProjectEventIds], + repositoryLastEventId, + } } const pageLastEventId = exported.events.at(-1)?.eventId if ( @@ -427,102 +903,89 @@ async function scanV1MigrationAuditEvents( function auditMigrationMarkerBody(params: { marker: ImportedMarkerEvent events: MemoryEventEnvelope[] - revision: number - checksum: string - identity: string + memory: TaskMemoryV1 + projectId: ProjectId repositoryLastEventId: MemoryEventId }): V1MigrationAuditOutcome { - const { - marker, - events, - revision, - checksum, - identity, - repositoryLastEventId, - } = params - const markerResult = (outcome: 'incomplete' | 'mismatch', reason: - | 'missing-imported-task' - | 'missing-imported-observations' - | 'imported-body-mismatch'): V1MigrationAuditOutcome => ({ + const { marker, events, memory, projectId, repositoryLastEventId } = params + const { revision, checksum } = memory + const build = buildTaskMemoryV1Migration({ + memory, + projectId, + sessionId: marker.sessionId, + representationIdentity: marker.payload.legacyRecordKey, + }) + const markerResult = ( + outcome: 'incomplete' | 'mismatch', + reason: + | 'missing-imported-task' + | 'missing-imported-observations' + | 'imported-body-mismatch', + missingEventIds?: MemoryEventId[], + ): V1MigrationAuditOutcome => ({ outcome, reason, revision, checksum, repositoryLastEventId, markerEventId: marker.eventId, + ...(missingEventIds?.length + ? { missingEventIds: missingEventIds.slice(0, MAX_OBSERVATIONS) } + : {}), }) - - if (marker.payload.legacyRecordKey !== identity) { + if (!equalEventDraft(build.markerDraft, marker)) { return markerResult('mismatch', 'imported-body-mismatch') } - const taskExists = events.some( - (event) => - event.eventType === 'task.created' && - event.payload.taskId === marker.payload.importedTaskId, - ) - if (!taskExists) return markerResult('incomplete', 'missing-imported-task') + const byEventId = new Map() + for (const event of events) { + const matching = byEventId.get(event.eventId) ?? [] + matching.push(event) + byEventId.set(event.eventId, matching) + } + const expectedTask = build.bodyDrafts[0]! + const taskMatches = byEventId.get(expectedTask.eventId) ?? [] + if (taskMatches.length === 0) { + return markerResult('incomplete', 'missing-imported-task') + } if ( - new Set(marker.payload.importedObservationIds).size !== - marker.payload.importedObservationIds.length + taskMatches.length !== 1 || + !equalEventDraft(expectedTask, taskMatches[0]!) ) { return markerResult('mismatch', 'imported-body-mismatch') } - const observations = new Map() - for (const event of events) { - if (event.eventType !== 'observation.recorded') continue - const observationId = event.payload.observation.observationId - const matching = observations.get(observationId) ?? [] - matching.push(event) - observations.set(observationId, matching) - } - - let missing = false - for (const observationId of marker.payload.importedObservationIds) { - const matching = observations.get(observationId) ?? [] + const missingObservationIds: MemoryEventId[] = [] + for (const expected of build.bodyDrafts.slice(1)) { + const matching = byEventId.get(expected.eventId) ?? [] if (matching.length === 0) { - missing = true + missingObservationIds.push(expected.eventId) continue } - if (matching.length !== 1) { - return markerResult('mismatch', 'imported-body-mismatch') - } - const observation = matching[0]!.payload.observation - if ( - observation.taskId !== marker.payload.importedTaskId || - observation.provenance?.origin !== 'migration' || - observation.provenance.metadata?.revision !== revision || - observation.provenance.metadata?.checksum !== checksum || - !observation.tags.includes('legacy-v1') - ) { + if (matching.length !== 1 || !equalEventDraft(expected, matching[0]!)) { return markerResult('mismatch', 'imported-body-mismatch') } } - if (missing) { - return markerResult('incomplete', 'missing-imported-observations') + if (missingObservationIds.length) { + return markerResult( + 'incomplete', + 'missing-imported-observations', + missingObservationIds, + ) } return { outcome: 'exact', revision, checksum, - identity, + identity: build.identity, markerEventId: marker.eventId, repositoryLastEventId, - importedTaskId: marker.payload.importedTaskId, - importedObservationIds: [...marker.payload.importedObservationIds], - omittedFields: marker.payload.omittedFields ?? 0, - warnings: marker.payload.warnings.filter(isV1MigrationWarningCode), - ...(marker.payload.sourceItemCounts - ? { - sourceItemCounts: markerSourceItemCounts( - marker.payload.sourceItemCounts, - ), - } - : {}), - ...(marker.payload.truncatedFields === undefined - ? {} - : { truncatedFields: marker.payload.truncatedFields }), + importedTaskId: build.importedTaskId, + importedObservationIds: build.importedObservationIds, + omittedFields: build.omittedFields, + warnings: build.warnings, + sourceItemCounts: build.sourceItemCounts, + truncatedFields: build.truncatedFields, } } @@ -537,13 +1000,23 @@ export async function auditTaskMemoryV1Migration(params: { const { revision, updatedAt, checksum, ...candidateDraft } = memory const parsedDraft = taskMemoryDraftV1Schema.safeParse(candidateDraft) if (!parsedDraft.success) { - return { outcome: 'rejected', reason: 'checksum-mismatch', revision, checksum } + return { + outcome: 'rejected', + reason: 'checksum-mismatch', + revision, + checksum, + } } const recomputed = stableHash( JSON.stringify({ revision, updatedAt, memory: parsedDraft.data }), ) if (recomputed !== checksum) { - return { outcome: 'rejected', reason: 'checksum-mismatch', revision, checksum } + return { + outcome: 'rejected', + reason: 'checksum-mismatch', + revision, + checksum, + } } const scanned = await scanV1MigrationAuditEvents(repository, projectId) @@ -599,26 +1072,92 @@ export async function auditTaskMemoryV1Migration(params: { : {}), } + if (exactMarkers.length > 0 && scanned.wrongProjectEventIds.length > 0) { + const build = buildTaskMemoryV1Migration({ + memory, + projectId, + sessionId: exactMarkers[0]!.sessionId, + }) + const expectedIds = new Set([ + build.markerDraft.eventId, + ...build.bodyDrafts.map((draft) => draft.eventId), + ]) + if ( + scanned.wrongProjectEventIds.some((eventId) => expectedIds.has(eventId)) + ) { + return { + outcome: 'mismatch', + reason: 'imported-body-mismatch', + markerEventId: exactMarkers[0]!.eventId, + ...auditEvidence, + } + } + return { outcome: 'failed', reason: 'wrong-project', revision, checksum } + } + if (scanned.wrongProjectEventIds.length > 0) { + return { outcome: 'failed', reason: 'wrong-project', revision, checksum } + } + if (scanned.duplicateEventIds.length > 0) { + if (exactMarkers.length > 0) { + const build = buildTaskMemoryV1Migration({ + memory, + projectId, + sessionId: exactMarkers[0]!.sessionId, + }) + const expectedIds = new Set([ + build.markerDraft.eventId, + ...build.bodyDrafts.map((draft) => draft.eventId), + ]) + if ( + scanned.duplicateEventIds.some((eventId) => expectedIds.has(eventId)) + ) { + return { + outcome: 'mismatch', + reason: 'imported-body-mismatch', + markerEventId: exactMarkers[0]!.eventId, + ...auditEvidence, + } + } + } + return { + outcome: 'failed', + reason: 'pagination-invalid', + revision, + checksum, + } + } if (conflictingReservation || conflictingMarker) { - return { outcome: 'mismatch', reason: 'checksum-conflict', ...auditEvidence } + return { + outcome: 'mismatch', + reason: 'checksum-conflict', + ...auditEvidence, + } } if ( - exactMarkers.length > 1 || matchingReservations.length > 1 || legacyMarkers.length > 1 || (unverifiableRevisionMarker && exactMarkers.length > 0) ) { - return { outcome: 'mismatch', reason: 'revision-conflict', ...auditEvidence } + return { + outcome: 'mismatch', + reason: 'revision-conflict', + ...auditEvidence, + } } - if (exactMarkers.length === 1) { - return auditMigrationMarkerBody({ - marker: exactMarkers[0]!, - events: scanned.events, - revision, - checksum, - identity, - repositoryLastEventId: scanned.repositoryLastEventId!, - }) + if (exactMarkers.length > 0) { + let latestFailure: V1MigrationAuditOutcome | undefined + for (const marker of [...exactMarkers].reverse()) { + const audited = auditMigrationMarkerBody({ + marker, + events: scanned.events, + memory, + projectId, + repositoryLastEventId: scanned.repositoryLastEventId!, + }) + if (audited.outcome === 'exact') return audited + latestFailure ??= audited + } + return latestFailure! } if (legacyMarkers.length === 1) { return { @@ -629,10 +1168,18 @@ export async function auditTaskMemoryV1Migration(params: { } } if (unverifiableRevisionMarker) { - return { outcome: 'mismatch', reason: 'revision-conflict', ...auditEvidence } + return { + outcome: 'mismatch', + reason: 'revision-conflict', + ...auditEvidence, + } } if (matchingReservations.length === 1) { - return { outcome: 'incomplete', reason: 'reservation-only', ...auditEvidence } + return { + outcome: 'incomplete', + reason: 'reservation-only', + ...auditEvidence, + } } return { outcome: 'not-migrated', ...auditEvidence } } @@ -652,201 +1199,110 @@ export async function importTaskMemoryV1(params: { const { revision, updatedAt, checksum, ...candidateDraft } = memory const parsedDraft = taskMemoryDraftV1Schema.safeParse(candidateDraft) if (!parsedDraft.success) { - return { outcome: 'rejected', reason: 'checksum-mismatch', revision, checksum } - } - const recomputed = stableHash( - JSON.stringify({ revision, updatedAt, memory: parsedDraft.data }), - ) - if (recomputed !== checksum) { - return { outcome: 'rejected', reason: 'checksum-mismatch', revision, checksum } - } - - const identity = getV1MigrationIdentity({ projectId, revision, checksum }) - const bodyIdentity = getV1MigrationBodyIdentity({ projectId, revision, checksum }) - let lookup: MigrationMarkerLookup - try { - lookup = await findMigrationMarker(repository, projectId, revision, checksum) - } catch { - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } - } - if (lookup.state === 'conflict') { - return { outcome: 'rejected', reason: 'checksum-mismatch', revision, checksum } - } - if (lookup.state === 'exact') { - return outcomeFromMarker({ - marker: lookup.marker, + return { + outcome: 'rejected', + reason: 'checksum-mismatch', revision, checksum, - identity, - lastEventId: lookup.lastEventId, - }) - } - - let expectedTail = lookup.lastEventId - ? { kind: 'event' as const, eventId: lookup.lastEventId } - : { kind: 'empty' as const } - const occurredAt = new Date(updatedAt).toISOString() - const importedTaskId = TaskIdFor(bodyIdentity) - const warnings = new Set() - let omittedFields = 0 - let truncatedFields = 0 - if (bounded(memory.goal)) warnings.add('goal-excluded') - - const sourceItemCounts = newSourceItemCounts() - const recordTruncation = () => { - truncatedFields++ - omittedFields++ - warnings.add('text-truncated') - } - - const observations: Array<{ - category: V1MigrationCategory - summary: string - detail: string - selector?: { kind: 'file'; path: string } - }> = [] - const addObservation = (observation: (typeof observations)[number]): boolean => { - if (observations.length >= MAX_OBSERVATIONS) { - omittedFields++ - warnings.add('observation-cap-reached') - return false - } - observations.push(observation) - return true + } } - const categories = [ - ['requirements', memory.requirements], - ['decisions', memory.decisions], - ['files-inspected', memory.filesInspected], - ['edits-made', memory.editsMade], - ['validation-results', memory.validationResults], - ['review-receipts', memory.reviewReceipts], - ['blockers', memory.blockers], - ['next-actions', memory.nextActions], - ] as const - for (const [category, values] of categories) { - const nonEmpty: string[] = [] - for (const value of values) { - const item = boundedText(String(value)) - if (item.truncated) recordTruncation() - if (item.value) nonEmpty.push(item.value) + const recomputed = stableHash( + JSON.stringify({ revision, updatedAt, memory: parsedDraft.data }), + ) + if (recomputed !== checksum) { + return { + outcome: 'rejected', + reason: 'checksum-mismatch', + revision, + checksum, } - if (nonEmpty.length === 0) continue - const aggregate = boundedText(nonEmpty.join('\n'), MAX_AGGREGATE_DETAIL) - if (aggregate.truncated) recordTruncation() - sourceItemCounts[category] += nonEmpty.length - addObservation({ - category, - summary: `Legacy ${category} items (${nonEmpty.length})`, - detail: aggregate.value, - }) } - const historical = boundedText(memory.historicalSummary) - if (historical.truncated) recordTruncation() - if (historical.value) { - sourceItemCounts['historical-summary']++ - addObservation({ - category: 'historical-summary', - summary: 'Legacy historical summary', - detail: historical.value, - }) + let build = buildTaskMemoryV1Migration({ memory, projectId, sessionId }) + const identity = build.identity + let lookup: MigrationMarkerLookup + try { + lookup = await findMigrationMarker( + repository, + projectId, + revision, + checksum, + ) + } catch { + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } - - for (const evidence of memory.evidence) { - warnings.add('legacy-evidence-unverified') - if (evidence.stale === true) { - omittedFields++ - warnings.add('stale-evidence-omitted') - continue + if (lookup.state === 'conflict') { + return { + outcome: 'rejected', + reason: 'checksum-mismatch', + revision, + checksum, } - if (typeof evidence.path !== 'string') { - omittedFields++ - warnings.add('unsafe-path-omitted') - continue + } + if (lookup.state === 'exact') { + const audited = await auditTaskMemoryV1Migration({ + memory, + projectId, + repository, + }) + if (audited.outcome === 'exact') { + return outcomeFromMarker({ + marker: { + eventId: audited.markerEventId, + importedTaskId: audited.importedTaskId, + importedObservationIds: audited.importedObservationIds, + omittedFields: audited.omittedFields, + warnings: audited.warnings, + ...(audited.sourceItemCounts + ? { sourceItemCounts: audited.sourceItemCounts } + : {}), + ...(audited.truncatedFields === undefined + ? {} + : { truncatedFields: audited.truncatedFields }), + }, + revision, + checksum, + identity, + lastEventId: lookup.lastEventId, + }) } - const decision = classifyMemoryArtifactPath(evidence.path) - if (!decision.allowed || !decision.normalizedPath) { - omittedFields++ - warnings.add('unsafe-path-omitted') - continue + if (audited.outcome === 'rejected' || audited.outcome === 'failed') { + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } - const summary = boundedText( - evidence.summary || 'Legacy path discovery requires live verification.', - ) - if (summary.truncated) recordTruncation() - sourceItemCounts['path-evidence']++ - addObservation({ - category: 'path-evidence', - summary: 'Legacy path evidence requires reread', - detail: summary.value, - selector: { kind: 'file', path: decision.normalizedPath }, + const representationIdentity = `v1-repair:${hashToken( + `${identity}:${lookup.marker.eventId}:representation-2`, + )}` + build = buildTaskMemoryV1Migration({ + memory, + projectId, + sessionId, + representationIdentity, }) } - const drafts: MemoryEventDraft[] = [] - drafts.push( - withMigrationEventId( - createMemoryEventDraft({ - projectId, - sessionId, - userInputId: bodyIdentity, - occurredAt, - eventType: 'task.created', - payload: { - payloadSchemaVersion: 1, - taskId: importedTaskId, - title: `Imported legacy task memory revision ${revision}`, - objective: 'Preserve bounded legacy operational memory without importing its goal.', - initialStatus: 'created', - }, - }), - bodyIdentity, - 0, - ), - ) - - const importedObservationIds: ObservationId[] = [] - observations.forEach((item, index) => { - const observationId = ObservationIdFor(bodyIdentity, index) - importedObservationIds.push(observationId) - drafts.push( - withMigrationEventId( - createMemoryEventDraft({ - projectId, - sessionId, - userInputId: bodyIdentity, - sourceIndex: index + 1, - occurredAt, - eventType: 'observation.recorded', - payload: { - payloadSchemaVersion: 1, - observation: { - observationId, - taskId: importedTaskId, - kind: item.category === 'blockers' ? 'warning' : 'discovery', - summary: item.summary, - detail: item.detail, - confidence: 0.25, - evidence: [], - ...(item.selector ? { selectors: [item.selector] } : {}), - provenance: { - origin: 'migration', - recordedBy: 'sdk-memory-v1-import', - sourceEventIds: [], - sourceSessionId: sessionId, - metadata: { category: item.category, revision, checksum }, - }, - tags: ['legacy-v1', item.category, 'unverified'], - observedAt: occurredAt, - }, - }, - }), - bodyIdentity, - index + 1, - ), - ) - }) + const { + bodyIdentity, + occurredAt, + importedTaskId, + importedObservationIds, + omittedFields, + sourceItemCounts, + truncatedFields, + } = build + let expectedTail = lookup.lastEventId + ? { kind: 'event' as const, eventId: lookup.lastEventId } + : { kind: 'empty' as const } + const drafts = [...build.bodyDrafts] if (lookup.prior) { const currentObservationIds = new Set(importedObservationIds) @@ -897,38 +1353,88 @@ export async function importTaskMemoryV1(params: { eventId: reservationEventId(identity), } - const reloadOwnership = async (): Promise => { + const reloadOwnership = async (): Promise< + MigrationMarkerLookup | V1MigrationOutcome + > => { try { - const reloaded = await findMigrationMarker(repository, projectId, revision, checksum) + const reloaded = await findMigrationMarker( + repository, + projectId, + revision, + checksum, + ) if (reloaded.state === 'conflict') { - return { outcome: 'rejected', reason: 'checksum-mismatch', revision, checksum } + return { + outcome: 'rejected', + reason: 'checksum-mismatch', + revision, + checksum, + } } return reloaded } catch { - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } } if (!lookup.reserved) { try { const reserved = MemoryAppendOutcomeSchema.parse( - await repository.append(MemoryAppendRequestSchema.parse({ - schemaVersion: 2, - projectId, - expectedTail, - events: [reservation], - })), + await repository.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId, + expectedTail, + events: [reservation], + }), + ), ) if (reserved.outcome === 'appended') { expectedTail = { kind: 'event', eventId: reserved.lastEventId } - lookup = { ...lookup, reserved: true, lastEventId: reserved.lastEventId } - } else if (reserved.error.code === 'conflict' && reserved.error.retryable) { + lookup = { + ...lookup, + reserved: true, + lastEventId: reserved.lastEventId, + events: [ + ...lookup.events, + { ...reservation, sequence: reserved.entries[0]!.sequence }, + ], + } + } else if ( + reserved.error.code === 'conflict' && + reserved.error.retryable + ) { const reloaded = await reloadOwnership() if ('outcome' in reloaded) return reloaded if (reloaded.state === 'exact') { - return outcomeFromMarker({ marker: reloaded.marker, revision, checksum, identity, lastEventId: reloaded.lastEventId }) + const validatedMarker = validatedCompleteBuildMarker(reloaded, build) + if (!validatedMarker) + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } + return outcomeFromMarker({ + marker: validatedMarker, + revision, + checksum, + identity, + lastEventId: reloaded.lastEventId, + }) } - if (!reloaded.reserved) return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + if (!reloaded.reserved) + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } lookup = reloaded expectedTail = reloaded.lastEventId ? { kind: 'event', eventId: reloaded.lastEventId } @@ -936,29 +1442,62 @@ export async function importTaskMemoryV1(params: { } else { return { outcome: reserved.outcome === 'rejected' ? 'rejected' : 'failed', - reason: reserved.outcome === 'rejected' ? 'repository-rejected' : 'repository-failed', + reason: + reserved.outcome === 'rejected' + ? 'repository-rejected' + : 'repository-failed', revision, checksum, } } } catch { - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } } - const revalidateAfterConflict = async (): Promise => { + const revalidateAfterConflict = async (): Promise< + MigrationMarkerLookup | V1MigrationOutcome + > => { let revalidated: MigrationMarkerLookup try { - revalidated = await findMigrationMarker(repository, projectId, revision, checksum) + revalidated = await findMigrationMarker( + repository, + projectId, + revision, + checksum, + ) } catch { - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } if (revalidated.state === 'conflict') { - return { outcome: 'rejected', reason: 'checksum-mismatch', revision, checksum } + return { + outcome: 'rejected', + reason: 'checksum-mismatch', + revision, + checksum, + } } if (revalidated.state === 'exact') { + const validatedMarker = validatedCompleteBuildMarker(revalidated, build) + if (!validatedMarker) + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } return outcomeFromMarker({ - marker: revalidated.marker, + marker: validatedMarker, revision, checksum, identity, @@ -972,7 +1511,7 @@ export async function importTaskMemoryV1(params: { let recoveries = 0 while (true) { const remainingDrafts = drafts.slice(offset, offset + APPEND_PAGE_SIZE) - const existingIds = new Map() + const existingEvents = new Map() try { let afterEventId: MemoryEventId | undefined for (let page = 0; page < MAX_EXPORT_PAGES; page++) { @@ -982,15 +1521,40 @@ export async function importTaskMemoryV1(params: { ...(afterEventId ? { afterEventId } : {}), limit: 1_000, }) - if (exported.outcome !== 'page') throw new Error('Migration progress lookup failed') - for (const event of exported.events) existingIds.set(event.eventId, event.eventId) + if (exported.outcome !== 'page') + throw new Error('Migration progress lookup failed') + for (const event of exported.events) { + const matching = existingEvents.get(event.eventId) ?? [] + matching.push(event) + existingEvents.set(event.eventId, matching) + } if (!exported.nextAfterEventId) break afterEventId = exported.nextAfterEventId } } catch { - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } + } + for (const draft of remainingDrafts) { + const matching = existingEvents.get(draft.eventId) ?? [] + if ( + matching.length > 0 && + (matching.length !== 1 || !equalEventDraft(draft, matching[0]!)) + ) + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } - const events = remainingDrafts.filter((draft) => !existingIds.has(draft.eventId)) + const events = remainingDrafts.filter( + (draft) => !existingEvents.has(draft.eventId), + ) if (events.length === 0) break let outcome try { @@ -1005,13 +1569,22 @@ export async function importTaskMemoryV1(params: { ), ) } catch { - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } if (outcome.outcome === 'appended') { expectedTail = { kind: 'event', eventId: outcome.lastEventId } break } - if (outcome.error.code === 'conflict' && outcome.error.retryable && recoveries < 2) { + if ( + outcome.error.code === 'conflict' && + outcome.error.retryable && + recoveries < 2 + ) { recoveries++ const recovered = await revalidateAfterConflict() if ('outcome' in recovered) return recovered @@ -1022,39 +1595,17 @@ export async function importTaskMemoryV1(params: { } return { outcome: outcome.outcome === 'rejected' ? 'rejected' : 'failed', - reason: outcome.outcome === 'rejected' ? 'repository-rejected' : 'repository-failed', + reason: + outcome.outcome === 'rejected' + ? 'repository-rejected' + : 'repository-failed', revision, checksum, } } } - const warningList = [...warnings].slice(0, 100) - const marker = withMigrationEventId( - createMemoryEventDraft({ - projectId, - sessionId, - userInputId: identity, - sourceIndex: MAX_OBSERVATIONS + 1, - occurredAt, - eventType: 'migration.v1.imported', - payload: { - payloadSchemaVersion: 1, - sourceSchemaVersion: 1, - legacyRecordKey: identity, - importedTaskId, - importedObservationIds, - sourceRevision: revision, - sourceChecksum: checksum, - omittedFields, - sourceItemCounts, - truncatedFields, - warnings: warningList, - }, - }), - identity, - MAX_OBSERVATIONS + 1, - ) + const marker = build.markerDraft try { const markerOutcome = MemoryAppendOutcomeSchema.parse( await repository.append( @@ -1067,10 +1618,18 @@ export async function importTaskMemoryV1(params: { ), ) if (markerOutcome.outcome !== 'appended') { - if (markerOutcome.error.code === 'conflict' && markerOutcome.error.retryable) { + if ( + markerOutcome.error.code === 'conflict' && + markerOutcome.error.retryable + ) { const recovered = await revalidateAfterConflict() if ('outcome' in recovered) return recovered - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } return { outcome: markerOutcome.outcome === 'rejected' ? 'rejected' : 'failed', @@ -1083,6 +1642,18 @@ export async function importTaskMemoryV1(params: { } } const markerDuplicate = markerOutcome.entries[0]?.duplicate === true + const completed = await reloadOwnership() + if ( + 'outcome' in completed || + completed.state !== 'exact' || + !validatedCompleteBuildMarker(completed, build) + ) + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } return { outcome: markerDuplicate ? 'no-op' : 'imported', revision, @@ -1091,13 +1662,18 @@ export async function importTaskMemoryV1(params: { importedTaskId, importedObservationIds, omittedFields, - warnings: warningList, - lastEventId: markerOutcome.lastEventId, + warnings: build.warnings, + lastEventId: completed.lastEventId, sourceItemCounts, truncatedFields, } } catch { - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } } @@ -1106,7 +1682,9 @@ function TaskIdFor(identity: string): TaskId { } function ObservationIdFor(identity: string, index: number): ObservationId { - return ObservationIdSchema.parse(`observation:migration:${hashToken(`${identity}:${index}`)}`) + return ObservationIdSchema.parse( + `observation:migration:${hashToken(`${identity}:${index}`)}`, + ) } function withMigrationEventId( diff --git a/sdk/src/services/task-memory-store.ts b/sdk/src/services/task-memory-store.ts index 91221c7bbb..d5b7d782c8 100644 --- a/sdk/src/services/task-memory-store.ts +++ b/sdk/src/services/task-memory-store.ts @@ -1,5 +1,5 @@ import { createHash, randomUUID } from 'node:crypto' -import type { Stats } from 'node:fs' +import type { PathLike, Stats } from 'node:fs' import * as nodeFsPromises from 'node:fs/promises' import path from 'node:path' @@ -8,9 +8,13 @@ import { taskMemoryDraftV1Schema, taskMemoryV1Schema, } from '@codebuff/common/types/task-memory' +import { errorCode } from '@codebuff/common/util/error' import { stableHash } from '@codebuff/common/util/stable-hash' -import type { CodebuffFileSystem } from '@codebuff/common/types/filesystem' +import type { + CodebuffFileContent, + CodebuffFileSystem, +} from '@codebuff/common/types/filesystem' import type { TaskMemoryEvidenceV1, TaskMemoryV1, @@ -42,10 +46,15 @@ export interface TaskMemoryStoreFs { ): Promise readFile(file: Parameters[0]): Promise /** - * Partial-read primitive; exposing it enables streamed hashing of - * oversized evidence files. Stores without it never buffer past - * MAX_EVIDENCE_HASH_BYTES: hashFile treats an oversized target as - * unverifiable (stale) rather than reading it whole. + * Explicit atomic-create capability used for cross-process locking. Merely + * accepting node's `wx` write option is not sufficient: virtual adapters + * have historically ignored it while reporting a successful write. + */ + createFileExclusive?(file: PathLike, data: CodebuffFileContent): Promise + /** + * Partial-read primitive used to hash large evidence files incrementally. + * Stores without it treat large targets as unverifiable rather than change + * the digest contract or buffer an unbounded body. */ open?: FsModule['open'] rename( @@ -53,9 +62,7 @@ export interface TaskMemoryStoreFs { newPath: Parameters[1], ): Promise stat(path: Parameters[0]): Promise - lstat?(path: Parameters[0]): Promise realpath?(path: Parameters[0]): Promise - readlink?(path: Parameters[0]): Promise unlink(path: Parameters[0]): Promise writeFile( file: Parameters[0], @@ -77,14 +84,14 @@ function truncateToCap(values: T[], cap: number): T[] { return values.length > cap ? values.slice(-cap) : values } -/** - * Upper bound for hashing a single evidence target during reconciliation - * (same spirit as MAX_DISCOVERED_PROJECT_READ_BYTES in run-state.ts). - * Freshness digests only need a stable prefix; capping bounds memory and - * cold-cache read cost per evidence item. Known trade-off: mutations beyond - * the cap do not flip staleness. - */ -const MAX_EVIDENCE_HASH_BYTES = 1_000_000 +/** Files above this size are hashed incrementally instead of buffered whole. */ +const EVIDENCE_STREAM_THRESHOLD_BYTES = 1_000_000 +const WHOLE_CONTENT_HASH_PREFIX = 'sha256-whole:' +const LEGACY_PREFIX_HASH_PREFIX = 'sha256-prefix-1m:' +const SHA256_HEX = /^[0-9a-f]{64}$/ + +/** Fixed working buffer for whole-content hashing of large evidence files. */ +const EVIDENCE_HASH_BUFFER_BYTES = 64 * 1024 /** Evidence reads processed per Promise.all batch during reconciliation. */ const EVIDENCE_HASH_CONCURRENCY = 16 @@ -96,21 +103,88 @@ const EVIDENCE_HASH_CONCURRENCY = 16 */ const IN_PROCESS_MEMORY_LOCKS = new Map>() -/** Bounded wait for another process's lock: 50 * 20ms ≈ 1s, then degrade. */ +/** Bounded wait for another process's lock: failure skips the write. */ const LOCK_ACQUIRE_ATTEMPTS = 50 const LOCK_RETRY_DELAY_MS = 20 +const DEFAULT_TASK_MEMORY_FS: TaskMemoryStoreFs = { + ...nodeFsPromises, + createFileExclusive: async (file, data) => { + await nodeFsPromises.writeFile(file, data, { flag: 'wx', mode: 0o600 }) + }, +} + +function taskMemoryFs(fs: TaskMemoryStoreFs | undefined): TaskMemoryStoreFs { + if (!fs) return DEFAULT_TASK_MEMORY_FS + // Direct node:fs/promises adapters have native exclusive-create semantics. + // Virtual adapters must instead opt in through createFileExclusive. + if (fs === nodeFsPromises && typeof fs.createFileExclusive !== 'function') { + return { + ...fs, + createFileExclusive: DEFAULT_TASK_MEMORY_FS.createFileExclusive, + } + } + return fs +} + +export type TaskMemoryV1Inspection = + | { status: 'absent' } + | { status: 'valid'; memory: TaskMemoryV1 } + | { + status: 'invalid' + reason: 'malformed-json' | 'schema-invalid' | 'checksum-mismatch' + } + | { status: 'unreadable'; reason: 'read-failed' } + /** - * A lock older than this is treated as abandoned by a crashed writer and - * reclaimed. Locked sections are short (one load plus one rename), so a - * lock this old cannot belong to a live writer. + * Inspect the persisted V1 record without mutating storage. The bounded result + * deliberately exposes no path, contents, validation issues, or raw read + * error. Only ENOENT proves absence; every other read failure is unreadable. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ -const LOCK_STALE_MS = 10_000 +export async function inspectPersistedTaskMemoryV1(params: { + rootDir: string + fs?: Pick +}): Promise { + const fs = params.fs ?? nodeFsPromises + let raw: Buffer + try { + raw = await fs.readFile(getMemoryFilePath(params.rootDir)) + } catch (error) { + return errorCode(error) === 'ENOENT' + ? { status: 'absent' } + : { status: 'unreadable', reason: 'read-failed' } + } + + let candidate: unknown + try { + candidate = JSON.parse(raw.toString('utf8')) + } catch { + return { status: 'invalid', reason: 'malformed-json' } + } + + const parsed = taskMemoryV1Schema.safeParse(candidate) + if (!parsed.success) return { status: 'invalid', reason: 'schema-invalid' } + const parsedDraft = taskMemoryDraftV1Schema.safeParse(parsed.data) + if (!parsedDraft.success) + return { status: 'invalid', reason: 'schema-invalid' } + const expectedChecksum = stableHash( + JSON.stringify({ + revision: parsed.data.revision, + updatedAt: parsed.data.updatedAt, + memory: parsedDraft.data, + }), + ) + if (expectedChecksum !== parsed.data.checksum) { + return { status: 'invalid', reason: 'checksum-mismatch' } + } + return { status: 'valid', memory: parsed.data } +} /** - * Load and schema-validate the persisted task memory for a project root, - * re-verifying the stored checksum against the loaded payload. Missing, - * corrupt, or checksum-mismatched data yields undefined; never throws. + * Load and schema-validate persisted task memory. Compatibility callers keep + * the historical behavior: every non-valid inspection collapses to undefined. * * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ @@ -118,37 +192,8 @@ export async function loadPersistedTaskMemory(params: { rootDir: string fs?: TaskMemoryStoreFs }): Promise { - const fs = params.fs ?? nodeFsPromises - try { - const raw = await fs.readFile(getMemoryFilePath(params.rootDir)) - const parsed = taskMemoryV1Schema.safeParse( - JSON.parse(raw.toString('utf8')), - ) - if (!parsed.success) return undefined - // A corrupted-but-schema-valid file must not hydrate silently: recompute - // the checksum over the same draft-shaped payload saveMergedTaskMemory - // hashed (memory WITHOUT the revision/updatedAt/checksum envelope) and - // reject mismatches. - // - // Compatibility: records written before checksum enforcement are also - // rejected here (fail-closed). No pre-checksum format was ever deployed - // — this store shipped alongside enforcement — so tolerate-and-upgrade - // would only weaken the corruption guard above. Revisit only if a - // deployed legacy format ever materializes. - const parsedDraft = taskMemoryDraftV1Schema.safeParse(parsed.data) - if (!parsedDraft.success) return undefined - const expectedChecksum = stableHash( - JSON.stringify({ - revision: parsed.data.revision, - updatedAt: parsed.data.updatedAt, - memory: parsedDraft.data, - }), - ) - if (expectedChecksum !== parsed.data.checksum) return undefined - return parsed.data - } catch { - return undefined - } + const inspected = await inspectPersistedTaskMemoryV1(params) + return inspected.status === 'valid' ? inspected.memory : undefined } // Evidence paths and journal destinations are stored with forward slashes; @@ -188,54 +233,137 @@ function resolveMoveTarget( return visited.size > 1 ? current : relativePath } +type EvidenceFileHashes = + | { status: 'hashed'; whole: string; legacyPrefix?: string } + | { status: 'missing' } + | { status: 'outside' } + | { status: 'unverifiable' } + +/** + * Open, contain, and hash one stable file descriptor. Opening before canonical + * validation closes the check/use gap: even if an ancestor pathname is + * replaced while realpath runs, descriptor identity must still match the + * contained canonical target before any bytes are read. Every size uses the + * same fixed buffer, and reads stop at the descriptor's initial size so growth + * cannot turn a small-file stat into an unbounded allocation or read. + */ async function hashFile( fs: TaskMemoryStoreFs, + rootDir: string, absolutePath: string, -): Promise { +): Promise { + if (!isPathInsideRootLexical(rootDir, absolutePath)) { + return { status: 'outside' } + } + if (typeof fs.open !== 'function' || typeof fs.realpath !== 'function') { + return { status: 'unverifiable' } + } + + let handle: Awaited> | undefined try { - const stats = await fs.stat(absolutePath) - if (stats.size > MAX_EVIDENCE_HASH_BYTES) { - if (typeof fs.open !== 'function') { - // Oversized target with no partial-read primitive: refuse to buffer - // a multi-GB body just to feed the digest. Returning undefined marks - // the entry stale (unverifiable) instead of risking OOM at session - // start. - return undefined - } - // Read only the leading bytes so a multi-GB evidence file never gets - // buffered whole just to feed the digest. - const handle = await fs.open(absolutePath, 'r') - try { - const leading = Buffer.alloc(MAX_EVIDENCE_HASH_BYTES) - const { bytesRead } = await handle.read( - leading, - 0, - MAX_EVIDENCE_HASH_BYTES, - 0, + handle = await fs.open(absolutePath, 'r') + const realRoot = await fs.realpath(rootDir) + const realCandidate = await fs.realpath(absolutePath) + if (!isPathInsideRootLexical(realRoot, realCandidate)) { + return { status: 'outside' } + } + + const [openedStats, canonicalStats] = await Promise.all([ + handle.stat(), + fs.stat(realCandidate), + ]) + if ( + !openedStats.isFile() || + openedStats.dev !== canonicalStats.dev || + openedStats.ino !== canonicalStats.ino || + !Number.isSafeInteger(openedStats.size) || + openedStats.size < 0 + ) { + return { status: 'unverifiable' } + } + + const wholeDigest = createHash('sha256') + const legacyPrefixDigest = createHash('sha256') + const buffer = Buffer.allocUnsafe(EVIDENCE_HASH_BUFFER_BYTES) + let position = 0 + let legacyBytes = 0 + while (position < openedStats.size) { + const length = Math.min(buffer.length, openedStats.size - position) + const { bytesRead } = await handle.read(buffer, 0, length, position) + if (bytesRead === 0) return { status: 'unverifiable' } + const bytes = buffer.subarray(0, bytesRead) + wholeDigest.update(bytes) + if (legacyBytes < EVIDENCE_STREAM_THRESHOLD_BYTES) { + const prefixBytes = Math.min( + bytesRead, + EVIDENCE_STREAM_THRESHOLD_BYTES - legacyBytes, ) - return createHash('sha256') - .update(leading.subarray(0, bytesRead)) - .digest('hex') - } finally { - try { - await handle.close() - } catch { - // Ignore: the digest is already computed. - } + legacyPrefixDigest.update(bytes.subarray(0, prefixBytes)) + legacyBytes += prefixBytes } + position += bytesRead } - let contents = await fs.readFile(absolutePath) - if (contents.length > MAX_EVIDENCE_HASH_BYTES) { - // Growth raced past the cap between stat and read: still digest only - // the leading bytes. - contents = contents.subarray(0, MAX_EVIDENCE_HASH_BYTES) + + const finalStats = await handle.stat() + if ( + finalStats.dev !== openedStats.dev || + finalStats.ino !== openedStats.ino || + finalStats.size !== openedStats.size || + finalStats.mtimeMs !== openedStats.mtimeMs || + finalStats.ctimeMs !== openedStats.ctimeMs + ) { + return { status: 'unverifiable' } } - return createHash('sha256').update(contents).digest('hex') - } catch { - return undefined + return { + status: 'hashed', + whole: wholeDigest.digest('hex'), + legacyPrefix: legacyPrefixDigest.digest('hex'), + } + } catch (error) { + return handle === undefined && errorCode(error) === 'ENOENT' + ? { status: 'missing' } + : { status: 'unverifiable' } + } finally { + await handle?.close().catch(() => {}) } } +function reconcileFreshnessHash( + recorded: string | undefined, + hashes: Extract, +): { status: 'fresh' | 'stale' | 'unverifiable'; freshnessHash?: string } { + if (recorded === undefined) return { status: 'fresh' } + + let expected: string + let matches: boolean + if (recorded.startsWith(WHOLE_CONTENT_HASH_PREFIX)) { + expected = recorded.slice(WHOLE_CONTENT_HASH_PREFIX.length) + if (!SHA256_HEX.test(expected)) return { status: 'unverifiable' } + matches = expected === hashes.whole + } else if (recorded.startsWith(LEGACY_PREFIX_HASH_PREFIX)) { + expected = recorded.slice(LEGACY_PREFIX_HASH_PREFIX.length) + if (!SHA256_HEX.test(expected)) return { status: 'unverifiable' } + // For files at or below the historical prefix limit, the prefix digest is + // identical to the whole-content digest and hashFile need not produce both. + matches = expected === (hashes.legacyPrefix ?? hashes.whole) + } else if (SHA256_HEX.test(recorded)) { + // Unversioned records may be either the historical one-megabyte prefix + // digest or the newer whole-content digest. A match can be upgraded. A + // mismatch is still reported stale by reconciliation, but prune separately + // refuses to delete a present file on this ambiguous evidence alone. + matches = recorded === hashes.whole || recorded === hashes.legacyPrefix + } else { + return { status: 'unverifiable' } + } + + return matches + ? { + status: 'fresh', + freshnessHash: `${WHOLE_CONTENT_HASH_PREFIX}${hashes.whole}`, + } + : { status: 'stale' } +} + function isPathInsideRootLexical( rootDir: string, candidatePath: string, @@ -252,43 +380,6 @@ function isPathInsideRootLexical( ) } -/** - * True when candidatePath resolves inside rootDir. Evidence paths originate - * from persisted state and journal destinations, so anything resolving - * outside the project root is treated as untrusted and never read. - * - * Lexical check is followed by a symlink-escape guard when the filesystem - * exposes lstat/realpath: a symlink planted inside rootDir that points - * outside is treated as outside and never hashed. Exposure on adapters - * without those primitives remains limited to feeding outside bytes into the - * freshness digest (contents are never surfaced to callers). Degrades to - * lexical-only when those primitives are unavailable or the target is - * missing (hashFile will then mark stale). - */ -async function isPathInsideRoot( - rootDir: string, - candidatePath: string, - fs: TaskMemoryStoreFs, -): Promise { - if (!isPathInsideRootLexical(rootDir, candidatePath)) return false - if (typeof fs.lstat !== 'function' || typeof fs.realpath !== 'function') { - return true - } - try { - const lst = await fs.lstat(candidatePath) - const isSymlink = - typeof (lst as unknown as { isSymbolicLink?: () => boolean }) - .isSymbolicLink === 'function' - ? (lst as unknown as { isSymbolicLink: () => boolean }).isSymbolicLink() - : false - if (!isSymlink) return true - const real = await fs.realpath(candidatePath) - return isPathInsideRootLexical(rootDir, real) - } catch { - return true - } -} - /** * Re-evaluate each evidence item against current disk state. Missing or * changed files mark the entry stale; a matching workspace move rebinds the @@ -302,7 +393,7 @@ export async function reconcileTaskMemoryEvidence(params: { fs?: TaskMemoryStoreFs workspaceMoves?: WorkspaceMoveRecord[] }): Promise { - const fs = params.fs ?? nodeFsPromises + const fs = taskMemoryFs(params.fs) const reconcileItem = async ( item: TaskMemoryEvidenceV1, ): Promise => { @@ -311,24 +402,43 @@ export async function reconcileTaskMemoryEvidence(params: { } const boundPath = resolveMoveTarget(item.path, params.workspaceMoves) const absolutePath = path.join(params.rootDir, boundPath) - if (!(await isPathInsideRoot(params.rootDir, absolutePath, fs))) { - // evidence.path comes from persisted state and journal destinations; - // refuse to read (or adopt) any path resolving outside the root, - // including symlink-escape when the filesystem exposes lstat/realpath. + const hashes = await hashFile(fs, params.rootDir, absolutePath) + if (hashes.status === 'outside') { + // A proven lexical or descriptor-validated canonical escape is stale and + // no bytes from the opened target are read. return { ...item, stale: true } } - const digest = await hashFile(fs, absolutePath) - const isFresh = - digest !== undefined && - (item.freshnessHash === undefined || digest === item.freshnessHash) - return isFresh - ? { ...item, path: boundPath, stale: false, verifiedAt: Date.now() } + if (hashes.status === 'missing') { + // ENOENT while opening is positive evidence that the file is gone. + return { ...item, path: boundPath, stale: true } + } + if (hashes.status === 'unverifiable') { + // Capability/read failures and unknown hash formats are not evidence + // that the file changed. Preserve the prior verdict so prune cannot + // destroy evidence merely because this adapter cannot validate it. + return { ...item, path: boundPath } + } + const freshness = reconcileFreshnessHash(item.freshnessHash, hashes) + if (freshness.status === 'unverifiable') { + // Unknown formats and read-capability failures cannot establish a new + // verdict. Preserve the prior state; prune treats them non-destructively. + return { ...item, path: boundPath } + } + return freshness.status === 'fresh' + ? { + ...item, + path: boundPath, + ...(freshness.freshnessHash + ? { freshnessHash: freshness.freshnessHash } + : {}), + stale: false, + verifiedAt: Date.now(), + } : { ...item, path: boundPath, stale: true } } - // Evidence lists are capped (LIST_CAPS.evidence), but each item may read - // up to MAX_EVIDENCE_HASH_BYTES, so hashing every entry concurrently - // spiked ~256MB of transient buffers on a cold cache. Chunks keep reads - // pipelined while bounding peak memory; output order follows input. + // Evidence lists are capped (LIST_CAPS.evidence), and large files use a + // fixed streaming buffer. Chunks bound the number of simultaneous file + // descriptors and buffered small-file reads while preserving output order. const evidence: TaskMemoryEvidenceV1[] = [] for ( let start = 0; @@ -413,7 +523,7 @@ export async function saveMergedTaskMemory(params: { }): Promise { const { runMemory } = params if (!runMemory) return undefined - const fs = params.fs ?? nodeFsPromises + const fs = taskMemoryFs(params.fs) // Load, revision derivation and commit run as one serialized section: the // revision this save reads from disk must not also be publishable by the // other writer (see withMemoryFileLock). @@ -587,64 +697,44 @@ async function writeRecordAtomically( /** * Serialize the load→revision→commit section shared by both task-memory - * writers. - * - * `saveMergedTaskMemory` and `pruneStaleTaskMemoryEvidence` each read the - * record, derive the next `revision` from what they read, and commit by - * rename. With no mutual exclusion those windows interleave — a save landing - * between prune's revision check and its rename, or two saves loading the - * same record — and both writers publish the SAME revision with different - * payloads, which is exactly what the record's monotonic-and-unique revision - * contract forbids. - * - * Two layers, because neither alone covers the writers: - * - an in-process promise chain keyed by the resolved record path, which - * handles the common case (a CLI `/memory prune` and an SDK save in one - * process) without needing any filesystem support; - * - an exclusive-create (`wx`) lock file for other processes sharing the - * project, reclaimed after {@link LOCK_STALE_MS} so a crashed writer cannot - * block later ones forever. + * writers. The in-process chain covers callers in this process, while an + * exclusive-create lock covers other processes. * - * The cross-process layer is advisory: an adapter that ignores the `flag` - * option, or a lock that cannot be taken within the bounded attempt budget, - * degrades to running the section unlocked rather than dropping the write. - * Both callers keep their own on-disk re-read (save) and revision guard - * (prune), so the degraded path is exactly as safe as before this lock - * existed. - * - * Evidence reconciliation deliberately stays OUTSIDE the section: it hashes - * every evidence file, so holding the lock across it would stall the other - * writer, and a run finishing mid-prune (which reconciliation's own IO can - * trigger) would deadlock against it. + * Cross-process exclusion is mandatory: failure to acquire the lock returns + * `undefined` and the write is skipped. Acquisition requires an explicit + * atomic-create capability, rather than trusting an adapter's handling of + * `writeFile(..., { flag: 'wx' })`. An old same-host lock is reclaimed only + * after its recorded owner process is proven dead; age alone never steals a + * live writer's lock. Automatic stale-lock deletion is deliberately avoided: + * portable filesystem APIs cannot atomically compare ownership and unlink, so + * reclaiming could remove a replacement owner's lock. The unique token is + * checked before release. */ async function withMemoryFileLock( fs: TaskMemoryStoreFs, filePath: string, section: () => Promise, -): Promise { +): Promise { const key = path.resolve(filePath) const previous = IN_PROCESS_MEMORY_LOCKS.get(key) ?? Promise.resolve() - const run = async (): Promise => { - // The lock file sits beside the record, so its directory must exist first. - // A failing mkdir is not fatal here: the section's own write path creates - // the directory and reports the failure through its normal outcome. + const run = async (): Promise => { await fs.mkdir(path.dirname(filePath), { recursive: true }).catch(() => {}) const lockPath = `${filePath}.lock` - const locked = await acquireRecordLock(fs, lockPath) + const token = await acquireRecordLock(fs, lockPath) + if (!token) return undefined try { return await section() } finally { - if (locked) { - await fs.unlink(lockPath).catch(() => { - // Ignore: a reclaimed or already-removed lock is not this writer's - // problem, and the section has already committed. - }) + const currentToken = await fs + .readFile(lockPath) + .then((contents) => contents.toString('utf8')) + .catch(() => undefined) + if (currentToken === token) { + await fs.unlink(lockPath).catch(() => {}) } } } const result = previous.then(run, run) - // Keep the chain alive across failures: one rejected section must not - // poison every later writer in this process. const settled = result.then( () => {}, () => {}, @@ -658,47 +748,41 @@ async function withMemoryFileLock( return result } +type RecordLockPayload = { + token: string + pid: number + createdAt: number +} + /** - * Take the cross-process lock file, or report that this writer is proceeding - * without it. Returns true only when the lock was created here, so the caller - * never unlinks a lock it does not hold. + * Take the cross-process lock and return its ownership bytes. Contention is + * bounded, but exhaustion and adapters without explicit atomic create fail + * closed: callers never enter the mutation section without proven exclusion. */ async function acquireRecordLock( fs: TaskMemoryStoreFs, lockPath: string, -): Promise { - let missingLockObservations = 0 +): Promise { + if (typeof fs.createFileExclusive !== 'function') return undefined + const payload: RecordLockPayload = { + token: randomUUID(), + pid: process.pid, + createdAt: Date.now(), + } + const token = `${JSON.stringify(payload)}\n` for (let attempt = 0; attempt < LOCK_ACQUIRE_ATTEMPTS; attempt += 1) { try { - await fs.writeFile(lockPath, `${process.pid}\n`, { - flag: 'wx', - mode: 0o600, - }) - return true + await fs.createFileExclusive(lockPath, token) + const observed = await fs.readFile(lockPath) + if (observed.toString('utf8') !== token) return undefined + return token } catch { - const stats = await fs.stat(lockPath).catch(() => undefined) - if (!stats) { - // Nothing holds the lock, so the rejection was not contention: this - // adapter does not honor exclusive-create. Give up after a second - // observation (which absorbs a holder releasing between the write and - // this stat) instead of burning the whole budget on every write. - missingLockObservations += 1 - if (missingLockObservations >= 2) return false - continue + if (attempt + 1 < LOCK_ACQUIRE_ATTEMPTS) { + await new Promise((resolve) => setTimeout(resolve, LOCK_RETRY_DELAY_MS)) } - missingLockObservations = 0 - // Reclaim a lock abandoned by a crashed writer. A stat without a usable - // mtime keeps waiting rather than stealing a possibly-live lock. - if (Date.now() - stats.mtimeMs > LOCK_STALE_MS) { - await fs.unlink(lockPath).catch(() => { - // Ignore: another writer may have reclaimed it first. - }) - continue - } - await new Promise((resolve) => setTimeout(resolve, LOCK_RETRY_DELAY_MS)) } } - return false + return undefined } /** @@ -723,6 +807,53 @@ export type TaskMemoryPruneOutcome = remaining: number } +/** + * Decide whether the current disk state supplies format-compatible authority + * to remove one reconciled item. A persisted or inherited `stale` flag is + * never sufficient by itself: older writers used unversioned hashes, and + * future writers may introduce formats this process cannot interpret. + * + * Missing and escaping targets are independently destructive-safe. For a + * present contained target, only a syntactically valid, explicitly versioned + * digest can prove a content mismatch. Bare and unknown digests are always + * preserved, as are pathless items and any item whose current bytes cannot be + * read. This deliberately repeats the bounded hash after reconciliation so a + * transient read failure cannot turn an inherited stale verdict into deletion + * authority. + */ +async function canPruneReconciledEvidence(params: { + item: TaskMemoryEvidenceV1 + recordedHash: string | undefined + rootDir: string + fs: TaskMemoryStoreFs +}): Promise { + const { item, recordedHash, rootDir, fs } = params + if (item.stale !== true || !item.path) return false + + const absolutePath = path.join(rootDir, item.path) + const hashes = await hashFile(fs, rootDir, absolutePath) + if (hashes.status === 'missing' || hashes.status === 'outside') return true + if (hashes.status !== 'hashed' || recordedHash === undefined) return false + + let expected: string + let algorithm: 'whole' | 'legacy-prefix' + if (recordedHash.startsWith(WHOLE_CONTENT_HASH_PREFIX)) { + expected = recordedHash.slice(WHOLE_CONTENT_HASH_PREFIX.length) + algorithm = 'whole' + } else if (recordedHash.startsWith(LEGACY_PREFIX_HASH_PREFIX)) { + expected = recordedHash.slice(LEGACY_PREFIX_HASH_PREFIX.length) + algorithm = 'legacy-prefix' + } else { + // Unversioned and unknown formats cannot authorize destructive pruning. + return false + } + if (!SHA256_HEX.test(expected)) return false + + const actual = + algorithm === 'whole' ? hashes.whole : (hashes.legacyPrefix ?? hashes.whole) + return expected !== actual +} + /** * Drop stale evidence from the persisted record and rewrite it atomically * (revision bumped past the loaded record, checksum recomputed over the same @@ -762,7 +893,7 @@ export async function pruneStaleTaskMemoryEvidence(params: { */ workspaceMoves?: WorkspaceMoveRecord[] }): Promise { - const fs = params.fs ?? nodeFsPromises + const fs = taskMemoryFs(params.fs) const persisted = await loadPersistedTaskMemory({ rootDir: params.rootDir, fs, @@ -777,10 +908,28 @@ export async function pruneStaleTaskMemoryEvidence(params: { fs, workspaceMoves: params.workspaceMoves, }) - const kept = memory.evidence.filter((item) => item.stale !== true) + const persistedById = new Map( + persisted.evidence.map((item) => [item.id, item] as const), + ) + const kept: TaskMemoryEvidenceV1[] = [] + for (const item of memory.evidence) { + const removalSafe = await canPruneReconciledEvidence({ + item, + recordedHash: persistedById.get(item.id)?.freshnessHash, + rootDir: params.rootDir, + fs, + }) + if (!removalSafe) kept.push(item) + } const removed = memory.evidence.length - kept.length const remaining = kept.length - if (removed === 0) return { status: 'pruned', removed: 0, remaining } + const needsHashBackfill = memory.evidence.some( + (item, index) => + item.freshnessHash !== persisted.evidence[index]?.freshnessHash, + ) + if (removed === 0 && !needsHashBackfill) { + return { status: 'pruned', removed: 0, remaining } + } const parsedDraft = taskMemoryDraftV1Schema.safeParse({ ...memory, @@ -792,7 +941,7 @@ export async function pruneStaleTaskMemoryEvidence(params: { // Serialized section: the revision check and the commit must not straddle // another writer's commit (reconciliation above is deliberately outside it). - return withMemoryFileLock( + const outcome = await withMemoryFileLock( fs, getMemoryFilePath(params.rootDir), async (): Promise => { @@ -842,6 +991,9 @@ export async function pruneStaleTaskMemoryEvidence(params: { return { status: 'pruned', removed, remaining } }, ) + return ( + outcome ?? { status: 'failed', reason: 'write-failed', removed, remaining } + ) } /** @@ -854,8 +1006,8 @@ export async function pruneStaleTaskMemoryEvidence(params: { * require the optional `renameFile` capability; without it, persistence * degrades to a skipped save (saveMergedTaskMemory returns undefined) * rather than a non-atomic write. A native `open` on the host (real node - * fs) is forwarded so oversized evidence hashing streams leading bytes - * instead of buffering the file. + * fs) is forwarded so large evidence files receive the same whole-content + * digest as buffered files without unbounded memory use. * * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ @@ -871,18 +1023,15 @@ export function codebuffFsToNodePromises( } // CodebuffFileSystem's published type omits `open`, but node-fs-backed // hosts (spreads of fs.promises, createNodeFileSystem()) still carry it at - // runtime. Detect and forward it so oversized evidence hashing streams - // leading bytes instead of taking the buffered fallback. + // runtime. Detect and forward it for bounded whole-content streaming. const maybeOpen = (codebuffFs as { open?: FsModule['open'] }).open const open: TaskMemoryStoreFs['open'] = typeof maybeOpen === 'function' ? (file, flags, mode) => maybeOpen(file, flags, mode) : undefined - const maybeLstat = (codebuffFs as { lstat?: FsModule['stat'] }).lstat - const lstat: TaskMemoryStoreFs['lstat'] = - typeof maybeLstat === 'function' - ? (p) => - (maybeLstat as unknown as (p: string) => Promise)(p as string) + const createFileExclusive: TaskMemoryStoreFs['createFileExclusive'] = + typeof codebuffFs.createFileExclusive === 'function' + ? (file, data) => codebuffFs.createFileExclusive!(file, data) : undefined const maybeRealpath = ( codebuffFs as { realpath?: (p: string) => Promise } @@ -891,13 +1040,6 @@ export function codebuffFsToNodePromises( typeof maybeRealpath === 'function' ? (p) => maybeRealpath(p as string) : undefined - const maybeReadlink = ( - codebuffFs as { readlink?: (p: string) => Promise } - ).readlink - const readlink: TaskMemoryStoreFs['readlink'] = - typeof maybeReadlink === 'function' - ? (p) => maybeReadlink(p as string) - : undefined return { // Discard recursive mkdir's first-created-path result; the store only // needs completion, and TaskMemoryStoreFs declares Promise. @@ -905,11 +1047,10 @@ export function codebuffFsToNodePromises( await codebuffFs.mkdir(path, options) }, readFile: (file) => codebuffFs.readFile(file) as Promise, + createFileExclusive, rename, open, - lstat, realpath, - readlink, stat: (path) => codebuffFs.stat(path), unlink: (path) => codebuffFs.unlink(path), writeFile: (file, data, options) => From 722c510fc35eb3113f6bca6db45df6dec6d82b28 Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Sun, 13 Sep 2026 19:31:20 +0300 Subject: [PATCH 04/22] test(memory): fix coordinator test contract + refactor marker ownership validation The V1->V2 import now runs before the query-id check and fails closed against a stateless stub, so update the stale coordinator expectation to the deterministic 'migration-failed' reason while keeping the no-context/no-parity safety invariant. Split validatedMigrationOwnedMarkerMetadata's inline check chain into focused boolean predicate helpers; semantics are unchanged. --- .../memory-v2/__tests__/coordinator.test.ts | 638 +++++++++++++----- sdk/src/services/memory-v2/v1-migration.ts | 371 +++++++--- 2 files changed, 745 insertions(+), 264 deletions(-) diff --git a/sdk/src/services/memory-v2/__tests__/coordinator.test.ts b/sdk/src/services/memory-v2/__tests__/coordinator.test.ts index 5d753ae3c2..0efddef95b 100644 --- a/sdk/src/services/memory-v2/__tests__/coordinator.test.ts +++ b/sdk/src/services/memory-v2/__tests__/coordinator.test.ts @@ -50,7 +50,9 @@ class RepositoryStub implements MemoryRepositoryV2 { queryDegradation: MemoryRetrievalResult['degradation'] = { state: 'none' } appendGate: ((requestIndex: number) => Promise) | undefined - async append(request: Parameters[0]): Promise { + async append( + request: Parameters[0], + ): Promise { const parsed = MemoryAppendRequestSchema.parse(request) this.inFlight++ this.maxInFlight = Math.max(this.maxInFlight, this.inFlight) @@ -89,13 +91,17 @@ class RepositoryStub implements MemoryRepositoryV2 { request: MemoryRetrievalRequest, ): Promise> { this.queryRequests.push(request) - if (this.queryFailure === 'throw') throw new Error('sensitive backend detail') + if (this.queryFailure === 'throw') + throw new Error('sensitive backend detail') if (this.queryFailure === 'invalid') return { malformed: 'result' } as never if (this.queryFailure === 'rejected' || this.queryFailure === 'failed') { return MemoryQueryOutcomeSchema.parse({ outcome: this.queryFailure, error: { - code: this.queryFailure === 'rejected' ? 'invalid-request' : 'unavailable', + code: + this.queryFailure === 'rejected' + ? 'invalid-request' + : 'unavailable', message: 'sensitive backend detail', retryable: this.queryFailure === 'failed', }, @@ -135,17 +141,19 @@ class RepositoryStub implements MemoryRepositoryV2 { if (scripted !== undefined) return scripted as MemoryExportOutcome type ExportPage = Extract const events = this.exportTailEventId - ? ([{ - schemaVersion: 2, - eventSchemaVersion: 1, - eventType: 'session.started', - eventId: this.exportTailEventId, - projectId, - sessionId: 'session:export', - occurredAt: generatedAt, - sequence: 1, - payload: { payloadSchemaVersion: 1, startedAt: generatedAt }, - }] as unknown as ExportPage['events']) + ? ([ + { + schemaVersion: 2, + eventSchemaVersion: 1, + eventType: 'session.started', + eventId: this.exportTailEventId, + projectId, + sessionId: 'session:export', + occurredAt: generatedAt, + sequence: 1, + payload: { payloadSchemaVersion: 1, startedAt: generatedAt }, + }, + ] as unknown as ExportPage['events']) : [] return { outcome: 'page' as const, events, nextAfterEventId: null } } @@ -190,7 +198,9 @@ const taskMemory = (requirements: string[] = ['sensitive requirement']) => { ...draft, revision, updatedAt, - checksum: stableHash(JSON.stringify({ revision, updatedAt, memory: draft })), + checksum: stableHash( + JSON.stringify({ revision, updatedAt, memory: draft }), + ), } } @@ -201,24 +211,25 @@ const migrationOutcome = (params: { importedObservationCount: number sourceItemCounts?: unknown truncatedFields?: unknown -}): V1MigrationOutcome => ({ - outcome: 'imported', - revision: 7, - checksum: 'checksum:parity', - identity: 'identity:parity', - importedObservationIds: Array.from( - { length: params.importedObservationCount }, - (_, index) => `observation:parity-${index}`, - ), - omittedFields: [], - warnings: [], - ...(params.sourceItemCounts !== undefined - ? { sourceItemCounts: params.sourceItemCounts } - : {}), - ...(params.truncatedFields !== undefined - ? { truncatedFields: params.truncatedFields } - : {}), -} as unknown as V1MigrationOutcome) +}): V1MigrationOutcome => + ({ + outcome: 'imported', + revision: 7, + checksum: 'checksum:parity', + identity: 'identity:parity', + importedObservationIds: Array.from( + { length: params.importedObservationCount }, + (_, index) => `observation:parity-${index}`, + ), + omittedFields: [], + warnings: [], + ...(params.sourceItemCounts !== undefined + ? { sourceItemCounts: params.sourceItemCounts } + : {}), + ...(params.truncatedFields !== undefined + ? { truncatedFields: params.truncatedFields } + : {}), + }) as unknown as V1MigrationOutcome const emptyRetrievalResult = (): MemoryRetrievalResult => { const outcome = MemoryQueryOutcomeSchema.parse({ @@ -243,7 +254,10 @@ const emptyRetrievalResult = (): MemoryRetrievalResult => { describe('Memory V2 event factory', () => { test('derives stable, source-isolated IDs from trusted identity fields', () => { - const sessionId = deriveMemorySessionId({ projectId, userInputId: 'input:1' }) + const sessionId = deriveMemorySessionId({ + projectId, + userInputId: 'input:1', + }) const input = { projectId, sessionId, @@ -263,11 +277,22 @@ describe('MemoryV2Coordinator lifecycle', () => { test('explicit authority wins over legacy mode and json-v1 performs no V2 operations', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator({ ...config(repository, 'inject'), authority: 'json-v1' }) - await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:json-v1', query: 'none' }) + const coordinator = new MemoryV2Coordinator({ + ...config(repository, 'inject'), + authority: 'json-v1', + }) + await coordinator.prepareTurn({ + agentState: state, + trustedUserInputId: 'input:json-v1', + query: 'none', + }) expect(repository.requests).toHaveLength(0) expect(repository.queryRequests).toHaveLength(0) - expect(state.memoryAuthority).toMatchObject({ requested: 'json-v1', active: 'json-v1', fallbackOccurred: false }) + expect(state.memoryAuthority).toMatchObject({ + requested: 'json-v1', + active: 'json-v1', + fallbackOccurred: false, + }) }) test('invalid runtime authority safely selects json-v1 and performs no V2 work', async () => { @@ -321,7 +346,11 @@ describe('MemoryV2Coordinator lifecycle', () => { test('isolates fresh tasks and reuses an interrupted persisted turn', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:1', @@ -329,7 +358,11 @@ describe('MemoryV2Coordinator lifecycle', () => { }) const first = structuredClone(state.memoryV2!) - const fresh = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const fresh = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await fresh.prepareTurn({ agentState: state, trustedUserInputId: 'input:2', @@ -341,7 +374,11 @@ describe('MemoryV2Coordinator lifecycle', () => { state.memoryV2 = first const beforeResume = allEvents(repository).length const queriesBeforeResume = repository.queryRequests.length - const resumed = new MemoryV2Coordinator(config(repository), undefined, () => '2027-01-01T00:00:00.000Z') + const resumed = new MemoryV2Coordinator( + config(repository), + undefined, + () => '2027-01-01T00:00:00.000Z', + ) await resumed.prepareTurn({ agentState: state, trustedUserInputId: 'different-runtime-input', @@ -367,7 +404,11 @@ describe('MemoryV2Coordinator lifecycle', () => { revision: 9, snapshotId: 'snapshot:9', } - const inject = new MemoryV2Coordinator(config(repository, 'inject'), undefined, () => generatedAt) + const inject = new MemoryV2Coordinator( + config(repository, 'inject'), + undefined, + () => generatedAt, + ) await inject.prepareTurn({ agentState: state, trustedUserInputId: 'input:inject', @@ -387,7 +428,11 @@ describe('MemoryV2Coordinator lifecycle', () => { }) expect(state.memoryAuthority?.active).toBe('sqlite-v2-opt-in') - const shadow = new MemoryV2Coordinator(config(repository, 'shadow'), undefined, () => generatedAt) + const shadow = new MemoryV2Coordinator( + config(repository, 'shadow'), + undefined, + () => generatedAt, + ) await shadow.prepareTurn({ agentState: state, trustedUserInputId: 'input:shadow', @@ -396,7 +441,11 @@ describe('MemoryV2Coordinator lifecycle', () => { expect(state.memoryV2Context).toBeUndefined() expect(state.memoryAuthority?.active).toBe('json-v1') - const successful = new MemoryV2Coordinator(config(repository, 'inject'), undefined, () => generatedAt) + const successful = new MemoryV2Coordinator( + config(repository, 'inject'), + undefined, + () => generatedAt, + ) await successful.prepareTurn({ agentState: state, trustedUserInputId: 'input:successful', @@ -404,7 +453,11 @@ describe('MemoryV2Coordinator lifecycle', () => { }) expect(state.memoryV2Context).toBeDefined() repository.queryFailure = 'failed' - const failed = new MemoryV2Coordinator(config(repository, 'inject'), undefined, () => generatedAt) + const failed = new MemoryV2Coordinator( + config(repository, 'inject'), + undefined, + () => generatedAt, + ) await failed.prepareTurn({ agentState: state, trustedUserInputId: 'input:failed', @@ -424,35 +477,48 @@ describe('MemoryV2Coordinator lifecycle', () => { ['failed', 'query-failed'], ['throw', 'query-threw'], ['invalid', 'query-invalid-result'], - ] as const)('bounds %s query failures and keeps opt-in fail-closed', async (failure, reason) => { - const repository = new RepositoryStub() - repository.queryFailure = failure - const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository, 'inject'), undefined, () => generatedAt) + ] as const)( + 'bounds %s query failures and keeps opt-in fail-closed', + async (failure, reason) => { + const repository = new RepositoryStub() + repository.queryFailure = failure + const state = getInitialAgentState() + const coordinator = new MemoryV2Coordinator( + config(repository, 'inject'), + undefined, + () => generatedAt, + ) - await expect( - coordinator.prepareTurn({ - agentState: state, - trustedUserInputId: `input:${failure}`, - query: 'sensitive query text', - }), - ).resolves.toBeUndefined() + await expect( + coordinator.prepareTurn({ + agentState: state, + trustedUserInputId: `input:${failure}`, + query: 'sensitive query text', + }), + ).resolves.toBeUndefined() - expect(state.memoryV2Context).toBeUndefined() - expect(state.memoryAuthority).toMatchObject({ - requested: 'sqlite-v2-opt-in', - active: 'sqlite-v2-opt-in', - fallbackOccurred: false, - reason, - }) - expect(JSON.stringify(state.memoryAuthority)).not.toContain('sensitive backend detail') - }) + expect(state.memoryV2Context).toBeUndefined() + expect(state.memoryAuthority).toMatchObject({ + requested: 'sqlite-v2-opt-in', + active: 'sqlite-v2-opt-in', + fallbackOccurred: false, + reason, + }) + expect(JSON.stringify(state.memoryAuthority)).not.toContain( + 'sensitive backend detail', + ) + }, + ) test('rejects a mismatched query project and keeps opt-in fail-closed', async () => { const repository = new RepositoryStub() repository.queryProjectId = ProjectIdSchema.parse('project:other') const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository, 'inject'), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository, 'inject'), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, @@ -473,14 +539,20 @@ describe('MemoryV2Coordinator lifecycle', () => { const repository = new RepositoryStub() repository.queryDegradation = { state: 'degraded', - reasons: [{ - code: 'resource-budget', - detail: 'The bounded scan stopped at its payload budget.', - retryable: true, - }], + reasons: [ + { + code: 'resource-budget', + detail: 'The bounded scan stopped at its payload budget.', + retryable: true, + }, + ], } const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository, 'inject'), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository, 'inject'), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, @@ -503,7 +575,11 @@ describe('MemoryV2Coordinator lifecycle', () => { const repository = new RepositoryStub() repository.queryFailure = failure const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository, 'shadow'), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository, 'shadow'), + undefined, + () => generatedAt, + ) await expect( coordinator.prepareTurn({ agentState: state, @@ -520,9 +596,21 @@ describe('MemoryV2Coordinator lifecycle', () => { ) test.each([ - { classification: 'match', sourceItemCounts: { requirements: 1, evidence: 1 }, imported: 2 }, - { classification: 'v1-ahead', sourceItemCounts: { requirements: 2, evidence: 1 }, imported: 2 }, - { classification: 'v2-ahead', sourceItemCounts: { requirements: 1 }, imported: 2 }, + { + classification: 'match', + sourceItemCounts: { requirements: 1, evidence: 1 }, + imported: 2, + }, + { + classification: 'v1-ahead', + sourceItemCounts: { requirements: 2, evidence: 1 }, + imported: 2, + }, + { + classification: 'v2-ahead', + sourceItemCounts: { requirements: 1 }, + imported: 2, + }, { classification: 'unavailable', sourceItemCounts: undefined, imported: 2 }, ] as const)( 'computes $classification parity from migration source item counts', @@ -549,26 +637,42 @@ describe('MemoryV2Coordinator lifecycle', () => { ) test('extracts summed source counts and one direct safe truncated-field count', () => { - expect(extractV1MigrationExtras(migrationOutcome({ - importedObservationCount: 0, - sourceItemCounts: { requirements: 2, evidence: [1, 2] }, - truncatedFields: 4, - }))).toMatchObject({ sourceItemCount: 5, truncatedFields: 4 }) - - expect(extractV1MigrationExtras(migrationOutcome({ - importedObservationCount: 0, - truncatedFields: [4], - })).truncatedFields).toBeUndefined() - expect(extractV1MigrationExtras(migrationOutcome({ - importedObservationCount: 0, - truncatedFields: Number.MAX_SAFE_INTEGER + 1, - })).truncatedFields).toBeUndefined() + expect( + extractV1MigrationExtras( + migrationOutcome({ + importedObservationCount: 0, + sourceItemCounts: { requirements: 2, evidence: [1, 2] }, + truncatedFields: 4, + }), + ), + ).toMatchObject({ sourceItemCount: 5, truncatedFields: 4 }) + + expect( + extractV1MigrationExtras( + migrationOutcome({ + importedObservationCount: 0, + truncatedFields: [4], + }), + ).truncatedFields, + ).toBeUndefined() + expect( + extractV1MigrationExtras( + migrationOutcome({ + importedObservationCount: 0, + truncatedFields: Number.MAX_SAFE_INTEGER + 1, + }), + ).truncatedFields, + ).toBeUndefined() }) test('rebases retryable append conflicts and retries the same deterministic batch at most twice', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:conflict-retry', @@ -604,17 +708,27 @@ describe('MemoryV2Coordinator lifecycle', () => { kind: 'event', eventId: repository.exportTailEventId, }) - expect(attempts.every((attempt) => attempt.expectedTail !== undefined)).toBe(true) - expect(attempts.every((attempt) => attempt.expectedLastEventId === undefined)).toBe(true) + expect( + attempts.every((attempt) => attempt.expectedTail !== undefined), + ).toBe(true) + expect( + attempts.every((attempt) => attempt.expectedLastEventId === undefined), + ).toBe(true) expect(attempts[1]!.events).toEqual(attempts[0]!.events) expect(attempts[2]!.events).toEqual(attempts[0]!.events) - expect(state.memoryV2!.lastEventId).toBe(attempts[2]!.events.at(-1)!.eventId) + expect(state.memoryV2!.lastEventId).toBe( + attempts[2]!.events.at(-1)!.eventId, + ) }) test('serializes concurrent captures in invocation order with exact cursor chaining through terminal append', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:gated', @@ -631,12 +745,20 @@ describe('MemoryV2Coordinator lifecycle', () => { } const first = coordinator.recordToolObservation({ - toolName: 'get_build_targets', callId: 'call:first', userInputId: 'input:gated', - input: { files: ['src/first.ts'] }, output: [{ type: 'json', value: { targets: ['first'] } }], native: true, + toolName: 'get_build_targets', + callId: 'call:first', + userInputId: 'input:gated', + input: { files: ['src/first.ts'] }, + output: [{ type: 'json', value: { targets: ['first'] } }], + native: true, }) const second = coordinator.recordToolObservation({ - toolName: 'get_affected_tests', callId: 'call:second', userInputId: 'input:gated', - input: { files: ['src/second.ts'] }, output: [{ type: 'json', value: { tests: ['second.test.ts'] } }], native: true, + toolName: 'get_affected_tests', + callId: 'call:second', + userInputId: 'input:gated', + input: { files: ['src/second.ts'] }, + output: [{ type: 'json', value: { tests: ['second.test.ts'] } }], + native: true, }) const terminal = coordinator.finishTurn({ agentState: state, @@ -678,7 +800,11 @@ describe('MemoryV2Coordinator lifecycle', () => { test('serializes concurrent captures, advances cursors, and finishes once', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:1', @@ -715,15 +841,29 @@ describe('MemoryV2Coordinator lifecycle', () => { expect(repository.maxInFlight).toBe(1) expect(allEvents(repository).length).toBe(count) expect(state.memoryV2!.turn.status).toBe('completed') - expect(state.memoryV2!.lastEventId).toBe(repository.requests.at(-1)!.events.at(-1)!.eventId) - expect(repository.requests.every((request) => request.expectedTail !== undefined)).toBe(true) - expect(repository.requests.every((request) => request.expectedLastEventId === undefined)).toBe(true) + expect(state.memoryV2!.lastEventId).toBe( + repository.requests.at(-1)!.events.at(-1)!.eventId, + ) + expect( + repository.requests.every( + (request) => request.expectedTail !== undefined, + ), + ).toBe(true) + expect( + repository.requests.every( + (request) => request.expectedLastEventId === undefined, + ), + ).toBe(true) }) test('parks a failed terminal append and later commits the unchanged decision', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:terminal-retry', @@ -785,28 +925,39 @@ describe('MemoryV2Coordinator lifecycle', () => { signal: AbortSignal.abort(), expected: 'cancelled' as const, }, - ])('records $name terminal lifecycle status', async ({ output, signal, expected }) => { - const repository = new RepositoryStub() - const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) - await coordinator.prepareTurn({ - agentState: state, - trustedUserInputId: `input:${expected}`, - query: expected, - }) - await coordinator.finishTurn({ agentState: state, output, signal }) + ])( + 'records $name terminal lifecycle status', + async ({ output, signal, expected }) => { + const repository = new RepositoryStub() + const state = getInitialAgentState() + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) + await coordinator.prepareTurn({ + agentState: state, + trustedUserInputId: `input:${expected}`, + query: expected, + }) + await coordinator.finishTurn({ agentState: state, output, signal }) - expect(state.memoryV2!.turn.status).toBe(expected) - expect(allEvents(repository).at(-1)).toMatchObject({ - eventType: 'session.ended', - payload: { status: expected }, - }) - }) + expect(state.memoryV2!.turn.status).toBe(expected) + expect(allEvents(repository).at(-1)).toMatchObject({ + eventType: 'session.ended', + payload: { status: expected }, + }) + }, + ) test('captures only recognized successful native structured metadata with bounded paths', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:1', @@ -829,9 +980,9 @@ describe('MemoryV2Coordinator lifecycle', () => { { kind: 'file', path: 'src/good.ts' }, ]) expect(observationEvent.payload.observation.evidence).toEqual([]) - expect(observationEvent.payload.observation.provenance?.metadata.outputDigest).toMatch( - /^sha256:[a-f0-9]{64}$/, - ) + expect( + observationEvent.payload.observation.provenance?.metadata.outputDigest, + ).toMatch(/^sha256:[a-f0-9]{64}$/) } for (const attempt of [ @@ -868,32 +1019,64 @@ describe('MemoryV2Coordinator lifecycle', () => { test('filters private memory, generated, dependency, binary, secret, log, and build paths', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) - await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:policy', query: 'capture' }) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) + await coordinator.prepareTurn({ + agentState: state, + trustedUserInputId: 'input:policy', + query: 'capture', + }) const before = allEvents(repository).length await coordinator.recordToolObservation({ - toolName: 'code_search', callId: 'call:policy', userInputId: 'input:policy', - input: { files: ['.openbuff/memory/export.json', 'node_modules/pkg/index.js', 'generated/client.ts', 'assets/logo.png', '.env', 'logs/agent.log', 'dist/index.js'] }, - output: [{ type: 'json', value: { matches: [] } }], native: true, + toolName: 'code_search', + callId: 'call:policy', + userInputId: 'input:policy', + input: { + files: [ + '.openbuff/memory/export.json', + 'node_modules/pkg/index.js', + 'generated/client.ts', + 'assets/logo.png', + '.env', + 'logs/agent.log', + 'dist/index.js', + ], + }, + output: [{ type: 'json', value: { matches: [] } }], + native: true, }) expect(allEvents(repository)).toHaveLength(before) await coordinator.recordToolObservation({ - toolName: 'code_search', callId: 'call:mixed-policy', userInputId: 'input:policy', + toolName: 'code_search', + callId: 'call:mixed-policy', + userInputId: 'input:policy', input: { files: ['src/good.ts', '.openbuff/backups/memory.json'] }, - output: [{ type: 'json', value: { matches: [] } }], native: true, + output: [{ type: 'json', value: { matches: [] } }], + native: true, }) const event = allEvents(repository).at(-1)! expect(event.eventType).toBe('observation.recorded') if (event.eventType === 'observation.recorded') { - expect(event.payload.observation.selectors).toEqual([{ kind: 'file', path: 'src/good.ts' }]) - expect(event.payload.observation.detail).toContain('1 project-relative path') + expect(event.payload.observation.selectors).toEqual([ + { kind: 'file', path: 'src/good.ts' }, + ]) + expect(event.payload.observation.detail).toContain( + '1 project-relative path', + ) } }) test('captures only confirmed mutation actions', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:1', @@ -921,7 +1104,8 @@ describe('MemoryV2Coordinator lifecycle', () => { path: 'src/new.ts', outcome: 'applied', beforeHash: null, - afterHash: 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', + afterHash: + 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', }, ], authorityTier: 'portable_path', @@ -942,7 +1126,8 @@ describe('MemoryV2Coordinator lifecycle', () => { path: 'src/new.ts', status: 'committed', beforeHash: null, - afterHash: 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', + afterHash: + 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', }, ], finalHashes: { @@ -964,7 +1149,8 @@ describe('MemoryV2Coordinator lifecycle', () => { action: 'create', path: 'src/new.ts', beforeHash: null, - afterHash: 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', + afterHash: + 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', }, ]) } @@ -984,23 +1170,44 @@ describe('MemoryV2Coordinator lifecycle', () => { version: 1, operationId: 'operation:private', outcome: 'applied', - actions: [{ - actionId: 'action:private', index: 0, action: 'create', - path: '.openbuff/memory/export.json', outcome: 'applied', beforeHash: null, - afterHash: 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', - }], + actions: [ + { + actionId: 'action:private', + index: 0, + action: 'create', + path: '.openbuff/memory/export.json', + outcome: 'applied', + beforeHash: null, + afterHash: + 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', + }, + ], authorityTier: 'portable_path', receiptId: 'receipt:private', authorityReceipt: { - kind: 'commit_receipt', version: 1, receiptId: 'receipt:private', - operationId: 'operation:private', callId: 'call:private-write', - authorityTier: 'portable_path', status: 'committed', - actions: [{ - actionId: 'action:private', index: 0, action: 'create', - path: '.openbuff/memory/export.json', status: 'committed', beforeHash: null, - afterHash: 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', - }], - finalHashes: { '.openbuff/memory/export.json': 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' }, + kind: 'commit_receipt', + version: 1, + receiptId: 'receipt:private', + operationId: 'operation:private', + callId: 'call:private-write', + authorityTier: 'portable_path', + status: 'committed', + actions: [ + { + actionId: 'action:private', + index: 0, + action: 'create', + path: '.openbuff/memory/export.json', + status: 'committed', + beforeHash: null, + afterHash: + 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', + }, + ], + finalHashes: { + '.openbuff/memory/export.json': + 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', + }, }, errors: [], freshCapabilities: [], @@ -1018,24 +1225,38 @@ describe('MemoryV2Coordinator lifecycle', () => { const logger = { warn: (fields: unknown) => warnings.push(fields) } const shadowState = getInitialAgentState() - const shadow = new MemoryV2Coordinator(config(repository, 'shadow'), logger, () => generatedAt) - await expect(shadow.prepareTurn({ - agentState: shadowState, - trustedUserInputId: 'input:shadow-failure', - query: 'continue despite failure', - })).resolves.toBeUndefined() + const shadow = new MemoryV2Coordinator( + config(repository, 'shadow'), + logger, + () => generatedAt, + ) + await expect( + shadow.prepareTurn({ + agentState: shadowState, + trustedUserInputId: 'input:shadow-failure', + query: 'continue despite failure', + }), + ).resolves.toBeUndefined() expect(shadowState.memoryAuthority).toMatchObject({ - requested: 'shadow-v2', active: 'json-v1', fallbackOccurred: false, + requested: 'shadow-v2', + active: 'json-v1', + fallbackOccurred: false, reason: 'lifecycle-append-failed', }) const optInState = getInitialAgentState() - const optIn = new MemoryV2Coordinator(config(repository, 'inject'), logger, () => generatedAt) - await expect(optIn.prepareTurn({ - agentState: optInState, - trustedUserInputId: 'input:opt-in-failure', - query: 'fall back', - })).resolves.toBeUndefined() + const optIn = new MemoryV2Coordinator( + config(repository, 'inject'), + logger, + () => generatedAt, + ) + await expect( + optIn.prepareTurn({ + agentState: optInState, + trustedUserInputId: 'input:opt-in-failure', + query: 'fall back', + }), + ).resolves.toBeUndefined() expect(optInState.memoryV2!.lastEventId).toBeUndefined() expect(optInState.memoryV2Context).toBeUndefined() expect(optInState.memoryAuthority).toMatchObject({ @@ -1056,7 +1277,11 @@ describe('MemoryV2Coordinator lifecycle', () => { }) const state = getInitialAgentState() state.taskMemory = taskMemory() - const coordinator = new MemoryV2Coordinator(config(repository, 'inject'), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository, 'inject'), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, @@ -1068,22 +1293,37 @@ describe('MemoryV2Coordinator lifecycle', () => { expect(state.memoryAuthority?.parity).toBeUndefined() expect(state.memoryAuthority).toMatchObject({ active: 'sqlite-v2-opt-in', - reason: 'query-invalid-result', + reason: 'migration-failed', }) }) test('replays persisted terminal data before a new lifecycle', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const first = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) - await first.prepareTurn({ agentState: state, trustedUserInputId: 'input:first', query: 'first' }) + const first = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) + await first.prepareTurn({ + agentState: state, + trustedUserInputId: 'input:first', + query: 'first', + }) repository.failNextAppend = true - await first.finishTurn({ agentState: state, output: { type: 'structuredOutput', value: {} } }) + await first.finishTurn({ + agentState: state, + output: { type: 'structuredOutput', value: {} }, + }) const failedBatch = structuredClone(repository.requests.at(-1)!.events) const checkpoint = structuredClone(state) const before = repository.requests.length - const second = new MemoryV2Coordinator(config(repository), undefined, () => '2027-01-01T00:00:00.000Z') + const second = new MemoryV2Coordinator( + config(repository), + undefined, + () => '2027-01-01T00:00:00.000Z', + ) await second.prepareTurn({ agentState: checkpoint, trustedUserInputId: 'input:second', @@ -1091,14 +1331,20 @@ describe('MemoryV2Coordinator lifecycle', () => { }) expect(repository.requests[before]!.events).toEqual(failedBatch) - expect(repository.requests[before + 1]!.events[0]!.eventType).toBe('session.started') + expect(repository.requests[before + 1]!.events[0]!.eventType).toBe( + 'session.started', + ) expect(checkpoint.memoryV2!.pendingTerminal).toBeUndefined() }) test('emits coverage.recorded events from evaluate_audit_coverage results', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:coverage', @@ -1110,13 +1356,28 @@ describe('MemoryV2Coordinator lifecycle', () => { callId: 'call:coverage', userInputId: 'input:coverage', input: {}, - output: [{ type: 'json', value: { status: 'complete', features: [{ feature: 'auth' }] } }], + output: [ + { + type: 'json', + value: { status: 'complete', features: [{ feature: 'auth' }] }, + }, + ], native: true, - workspaceState: { schemaVersion: 1 as const, revision: 5, snapshotId: 'snap:5', updatedAt: Date.now(), changes: [] }, + workspaceState: { + schemaVersion: 1 as const, + revision: 5, + snapshotId: 'snap:5', + updatedAt: Date.now(), + changes: [], + }, }) const newEvents = allEvents(repository).slice(before) - const observationEvents = newEvents.filter((e) => e.eventType === 'observation.recorded') - const coverageEvents = newEvents.filter((e) => e.eventType === 'coverage.recorded') + const observationEvents = newEvents.filter( + (e) => e.eventType === 'observation.recorded', + ) + const coverageEvents = newEvents.filter( + (e) => e.eventType === 'coverage.recorded', + ) expect(observationEvents).toHaveLength(1) expect(coverageEvents).toHaveLength(1) const coveragePayload = coverageEvents[0]!.payload @@ -1133,7 +1394,10 @@ describe('MemoryV2Coordinator lifecycle', () => { test('fails closed when conflict tail export cannot produce a valid page', async () => { const failures: Array = [ - { outcome: 'failed', error: { code: 'unavailable', message: 'no', retryable: true } }, + { + outcome: 'failed', + error: { code: 'unavailable', message: 'no', retryable: true }, + }, { malformed: true }, new Error('export threw'), { outcome: 'page', events: [], nextAfterEventId: 'event:next' }, @@ -1141,14 +1405,26 @@ describe('MemoryV2Coordinator lifecycle', () => { for (const scripted of failures) { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) - await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:tail-failure', query: 'tail' }) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) + await coordinator.prepareTurn({ + agentState: state, + trustedUserInputId: 'input:tail-failure', + query: 'tail', + }) repository.appendConflictsRemaining = 1 repository.exportScript = [scripted] const before = repository.requests.length await coordinator.recordToolObservation({ - toolName: 'get_build_targets', callId: 'call:tail-failure', userInputId: 'input:tail-failure', - input: { files: ['src/a.ts'] }, output: [{ type: 'json', value: { targets: ['sdk'] } }], native: true, + toolName: 'get_build_targets', + callId: 'call:tail-failure', + userInputId: 'input:tail-failure', + input: { files: ['src/a.ts'] }, + output: [{ type: 'json', value: { targets: ['sdk'] } }], + native: true, }) expect(repository.requests.slice(before)).toHaveLength(1) expect(repository.requests.at(-1)!.expectedTail).toBeDefined() diff --git a/sdk/src/services/memory-v2/v1-migration.ts b/sdk/src/services/memory-v2/v1-migration.ts index 6966333177..e2ff977420 100644 --- a/sdk/src/services/memory-v2/v1-migration.ts +++ b/sdk/src/services/memory-v2/v1-migration.ts @@ -232,44 +232,53 @@ function isV1MigrationCategory(value: unknown): value is V1MigrationCategory { ) } +type ImportedObservationEvent = Extract< + MemoryEventEnvelope, + { eventType: 'observation.recorded' } +> + +type CanonicalMarkerSourceItemCounts = NonNullable< + ImportedMarkerEvent['payload']['sourceItemCounts'] +> + /** - * Return cleanup authority only for a canonical marker whose reservation and - * complete referenced body have the deterministic shape emitted by this - * migration. Repair markers and migration-looking provenance deliberately do - * not authorize destructive retirement: without the old V1 source they cannot - * be distinguished from user-authored events strongly enough to delete data. + * The marker payload must carry the V1 source fields every remaining + * ownership check depends on. */ -function validatedMigrationOwnedMarkerMetadata(params: { - marker: ImportedMarkerEvent - events: MemoryEventEnvelope[] - projectId: ProjectId -}): MigrationMarkerMetadata | undefined { - const { marker, events, projectId } = params - const { sourceRevision, sourceChecksum, legacyRecordKey, sourceItemCounts } = - marker.payload - if ( - sourceRevision === undefined || - sourceChecksum === undefined || - sourceItemCounts === undefined +function hasCanonicalMarkerSourceFields( + payload: ImportedMarkerEvent['payload'], +): payload is ImportedMarkerEvent['payload'] & { + sourceRevision: number + sourceChecksum: string + sourceItemCounts: CanonicalMarkerSourceItemCounts +} { + return ( + payload.sourceRevision !== undefined && + payload.sourceChecksum !== undefined && + payload.sourceItemCounts !== undefined ) - return undefined +} - const identity = getV1MigrationIdentity({ - projectId, - revision: sourceRevision, - checksum: sourceChecksum, - }) - const bodyIdentity = getV1MigrationBodyIdentity({ +/** + * The marker payload must reproduce the deterministic identities, payload + * schema versions, and observation id sequence emitted by this migration. + */ +function hasCanonicalMarkerPayloadShape(params: { + marker: ImportedMarkerEvent + projectId: ProjectId + identity: string + expectedTaskId: TaskId + expectedObservationIds: ObservationId[] +}): boolean { + const { + marker, projectId, - revision: sourceRevision, - checksum: sourceChecksum, - }) - const expectedTaskId = TaskIdFor(bodyIdentity) - const expectedObservationIds = marker.payload.importedObservationIds.map( - (_, index) => ObservationIdFor(bodyIdentity, index), - ) - if ( - legacyRecordKey !== identity || + identity, + expectedTaskId, + expectedObservationIds, + } = params + return !( + marker.payload.legacyRecordKey !== identity || marker.projectId !== projectId || marker.eventId !== eventId(identity, MAX_OBSERVATIONS + 1) || marker.payload.payloadSchemaVersion !== 1 || @@ -280,14 +289,34 @@ function validatedMigrationOwnedMarkerMetadata(params: { observationId !== marker.payload.importedObservationIds[index], ) ) - return undefined +} +/** + * Exactly one canonical reservation event must exist, matching the marker + * envelope and the V1 source fields. + */ +function hasValidMigrationReservation(params: { + marker: ImportedMarkerEvent + events: MemoryEventEnvelope[] + projectId: ProjectId + identity: string + sourceRevision: number + sourceChecksum: string +}): boolean { + const { + marker, + events, + projectId, + identity, + sourceRevision, + sourceChecksum, + } = params const reservations = events.filter( (event) => event.eventId === reservationEventId(identity), ) - if (reservations.length !== 1) return undefined + if (reservations.length !== 1) return false const reservation = reservations[0]! - if ( + return !( reservation.eventType !== 'migration.v1.reserved' || reservation.projectId !== projectId || reservation.sessionId !== marker.sessionId || @@ -297,15 +326,35 @@ function validatedMigrationOwnedMarkerMetadata(params: { reservation.payload.sourceRevision !== sourceRevision || reservation.payload.sourceChecksum !== sourceChecksum ) - return undefined +} +/** + * Exactly one canonical task.created event must exist for the imported task, + * matching the marker envelope and the deterministic import text. + */ +function hasValidImportedTaskEvent(params: { + marker: ImportedMarkerEvent + events: MemoryEventEnvelope[] + projectId: ProjectId + bodyIdentity: string + expectedTaskId: TaskId + sourceRevision: number +}): boolean { + const { + marker, + events, + projectId, + bodyIdentity, + expectedTaskId, + sourceRevision, + } = params const expectedTaskEventId = eventId(bodyIdentity, 0) const taskEvents = events.filter( (event) => event.eventId === expectedTaskEventId, ) - if (taskEvents.length !== 1) return undefined + if (taskEvents.length !== 1) return false const task = taskEvents[0]! - if ( + return !( task.eventType !== 'task.created' || task.projectId !== projectId || task.sessionId !== marker.sessionId || @@ -318,66 +367,222 @@ function validatedMigrationOwnedMarkerMetadata(params: { 'Preserve bounded legacy operational memory without importing its goal.' || task.payload.initialStatus !== 'created' ) - return undefined +} + +/** + * A single imported observation event must match the deterministic body shape + * for its category, within the source item counts and canonical ordering. + */ +function hasValidImportedObservationEvent(params: { + marker: ImportedMarkerEvent + event: ImportedObservationEvent + projectId: ProjectId + expectedTaskId: TaskId + observationId: ObservationId + category: V1MigrationCategory + categoryIndex: number + occurrences: number + previousCategoryIndex: number + sourceRevision: number + sourceChecksum: string + sourceItemCounts: CanonicalMarkerSourceItemCounts +}): boolean { + const { + marker, + event, + projectId, + expectedTaskId, + observationId, + category, + categoryIndex, + occurrences, + previousCategoryIndex, + sourceRevision, + sourceChecksum, + sourceItemCounts, + } = params + const observation = event.payload.observation + const expectedSummary = + category === 'path-evidence' + ? 'Legacy path evidence requires reread' + : category === 'historical-summary' + ? 'Legacy historical summary' + : `Legacy ${category} items (${sourceItemCounts[category] ?? 0})` + if ( + categoryIndex < previousCategoryIndex || + (occurrences > 1 && category !== 'path-evidence') || + (sourceItemCounts[category] ?? 0) < occurrences || + event.projectId !== projectId || + event.sessionId !== marker.sessionId || + event.occurredAt !== marker.occurredAt || + event.payload.payloadSchemaVersion !== 1 || + observation.observationId !== observationId || + observation.taskId !== expectedTaskId || + observation.kind !== (category === 'blockers' ? 'warning' : 'discovery') || + observation.summary !== expectedSummary || + observation.detail.length > MAX_AGGREGATE_DETAIL || + observation.confidence !== 0.25 || + observation.evidence.length !== 0 || + observation.observedAt !== marker.occurredAt || + observation.provenance?.origin !== 'migration' || + observation.provenance.recordedBy !== 'sdk-memory-v1-import' || + observation.provenance.sourceEventIds.length !== 0 || + observation.provenance.sourceSessionId !== marker.sessionId || + observation.provenance.metadata.revision !== sourceRevision || + observation.provenance.metadata.checksum !== sourceChecksum || + JSON.stringify(observation.tags) !== + JSON.stringify(['legacy-v1', category, 'unverified']) + ) + return false + const selectors = observation.selectors ?? [] + if (category === 'path-evidence') { + if (selectors.length !== 1 || selectors[0]?.kind !== 'file') return false + const decision = classifyMemoryArtifactPath(selectors[0].path) + if (!decision.allowed || !decision.normalizedPath) return false + } else if (selectors.length !== 0) { + return false + } + return true +} +/** + * Every expected observation event must exist exactly once, in canonical + * category order and within the per-category source item counts. + */ +function hasValidImportedObservationEvents(params: { + marker: ImportedMarkerEvent + events: MemoryEventEnvelope[] + projectId: ProjectId + bodyIdentity: string + expectedTaskId: TaskId + expectedObservationIds: ObservationId[] + sourceRevision: number + sourceChecksum: string + sourceItemCounts: CanonicalMarkerSourceItemCounts +}): boolean { + const { + marker, + events, + projectId, + bodyIdentity, + expectedTaskId, + expectedObservationIds, + sourceRevision, + sourceChecksum, + sourceItemCounts, + } = params let previousCategoryIndex = -1 const categoryOccurrences = new Map() for (const [index, observationId] of expectedObservationIds.entries()) { const expectedEventId = eventId(bodyIdentity, index + 1) const matching = events.filter((event) => event.eventId === expectedEventId) - if (matching.length !== 1) return undefined + if (matching.length !== 1) return false const event = matching[0]! - if (event.eventType !== 'observation.recorded') return undefined + if (event.eventType !== 'observation.recorded') return false const observation = event.payload.observation const category = observation.provenance?.metadata.category - if (!isV1MigrationCategory(category)) return undefined + if (!isV1MigrationCategory(category)) return false const categoryIndex = MIGRATION_CATEGORIES.indexOf(category) const occurrences = (categoryOccurrences.get(category) ?? 0) + 1 categoryOccurrences.set(category, occurrences) - const expectedSummary = - category === 'path-evidence' - ? 'Legacy path evidence requires reread' - : category === 'historical-summary' - ? 'Legacy historical summary' - : `Legacy ${category} items (${sourceItemCounts[category] ?? 0})` if ( - categoryIndex < previousCategoryIndex || - (occurrences > 1 && category !== 'path-evidence') || - (sourceItemCounts[category] ?? 0) < occurrences || - event.projectId !== projectId || - event.sessionId !== marker.sessionId || - event.occurredAt !== marker.occurredAt || - event.payload.payloadSchemaVersion !== 1 || - observation.observationId !== observationId || - observation.taskId !== expectedTaskId || - observation.kind !== - (category === 'blockers' ? 'warning' : 'discovery') || - observation.summary !== expectedSummary || - observation.detail.length > MAX_AGGREGATE_DETAIL || - observation.confidence !== 0.25 || - observation.evidence.length !== 0 || - observation.observedAt !== marker.occurredAt || - observation.provenance?.origin !== 'migration' || - observation.provenance.recordedBy !== 'sdk-memory-v1-import' || - observation.provenance.sourceEventIds.length !== 0 || - observation.provenance.sourceSessionId !== marker.sessionId || - observation.provenance.metadata.revision !== sourceRevision || - observation.provenance.metadata.checksum !== sourceChecksum || - JSON.stringify(observation.tags) !== - JSON.stringify(['legacy-v1', category, 'unverified']) + !hasValidImportedObservationEvent({ + marker, + event, + projectId, + expectedTaskId, + observationId, + category, + categoryIndex, + occurrences, + previousCategoryIndex, + sourceRevision, + sourceChecksum, + sourceItemCounts, + }) ) - return undefined - const selectors = observation.selectors ?? [] - if (category === 'path-evidence') { - if (selectors.length !== 1 || selectors[0]?.kind !== 'file') - return undefined - const decision = classifyMemoryArtifactPath(selectors[0].path) - if (!decision.allowed || !decision.normalizedPath) return undefined - } else if (selectors.length !== 0) { - return undefined - } + return false previousCategoryIndex = categoryIndex } + return true +} + +/** + * Return cleanup authority only for a canonical marker whose reservation and + * complete referenced body have the deterministic shape emitted by this + * migration. Repair markers and migration-looking provenance deliberately do + * not authorize destructive retirement: without the old V1 source they cannot + * be distinguished from user-authored events strongly enough to delete data. + */ +function validatedMigrationOwnedMarkerMetadata(params: { + marker: ImportedMarkerEvent + events: MemoryEventEnvelope[] + projectId: ProjectId +}): MigrationMarkerMetadata | undefined { + const { marker, events, projectId } = params + if (!hasCanonicalMarkerSourceFields(marker.payload)) return undefined + const { sourceRevision, sourceChecksum, sourceItemCounts } = marker.payload + + const identity = getV1MigrationIdentity({ + projectId, + revision: sourceRevision, + checksum: sourceChecksum, + }) + const bodyIdentity = getV1MigrationBodyIdentity({ + projectId, + revision: sourceRevision, + checksum: sourceChecksum, + }) + const expectedTaskId = TaskIdFor(bodyIdentity) + const expectedObservationIds = marker.payload.importedObservationIds.map( + (_, index) => ObservationIdFor(bodyIdentity, index), + ) + if ( + !hasCanonicalMarkerPayloadShape({ + marker, + projectId, + identity, + expectedTaskId, + expectedObservationIds, + }) + ) + return undefined + if ( + !hasValidMigrationReservation({ + marker, + events, + projectId, + identity, + sourceRevision, + sourceChecksum, + }) + ) + return undefined + if ( + !hasValidImportedTaskEvent({ + marker, + events, + projectId, + bodyIdentity, + expectedTaskId, + sourceRevision, + }) + ) + return undefined + if ( + !hasValidImportedObservationEvents({ + marker, + events, + projectId, + bodyIdentity, + expectedTaskId, + expectedObservationIds, + sourceRevision, + sourceChecksum, + sourceItemCounts, + }) + ) + return undefined return markerMetadata(marker) } From eb1e79e46385fde6853928a45a5f256998e72586 Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Sun, 13 Sep 2026 20:16:16 +0300 Subject: [PATCH 05/22] fix(memory): compare migration drafts order-insensitively The Bun SQLite backend persists event payloads via stableJson (recursive key sort) and export() re-parses from that sorted JSON, so record-valued marker fields like sourceItemCounts round-trip in sorted key order while the in-memory draft keeps insertion order. equalEventDraft's naive JSON.stringify comparison therefore always reported imported-body-mismatch on the real provider. Canonicalize both sides (recursive key sort, arrays keep order) so comparison is by content, letting import return 'imported' and audit report 'exact'. Adds a regression test mimicking the stableJson round-trip, hoists the loop-invariant getV1MigrationIdentity out of findMigrationMarker's reservation loop, and documents hashToken truncation collision tolerance. --- .../memory-v2/__tests__/v1-migration.test.ts | 47 +++++++++++++++++++ sdk/src/services/memory-v2/v1-migration.ts | 41 ++++++++++++++-- 2 files changed, 84 insertions(+), 4 deletions(-) diff --git a/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts b/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts index a97770e36d..eb1e884176 100644 --- a/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts +++ b/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts @@ -965,6 +965,53 @@ describe('V1 memory migration audit', () => { expect(repository.appendCalls).toBe(appendCalls) }) + test('certifies exact after a JSON round-trip that sorts record-valued payload keys', async () => { + // Regression: the Bun SQLite repository persists payloads via stableJson, + // which recursively sorts object keys, and export() re-parses from that + // sorted JSON. The marker's sourceItemCounts (a z.record) therefore comes + // back in alphabetical order while the in-memory draft keeps insertion + // order. equalEventDraft must compare content, not key order, or the real + // provider always reports imported-body-mismatch. This subclass mimics + // that round-trip by re-parsing each exported event with sorted keys. + class SortedJsonRepository extends Repository { + override async export( + input: Parameters[0], + ) { + const page = await super.export(input) + if (page.outcome !== 'page') return page + return { + ...page, + events: page.events.map((event) => + MemoryEventEnvelopeSchema.parse( + JSON.parse( + JSON.stringify(event, (_key, value) => + value !== null && + typeof value === 'object' && + !Array.isArray(value) + ? Object.fromEntries( + Object.entries(value as Record).sort( + ([a], [b]) => (a < b ? -1 : a > b ? 1 : 0), + ), + ) + : value, + ), + ), + ), + ), + } + } + } + + const repository = new SortedJsonRepository() + const source = memory({ requirements: ['req-1'], decisions: ['dec-1'] }) + expect((await run(repository, source)).outcome).toBe('imported') + expect(await audit(repository, source)).toMatchObject({ + outcome: 'exact', + revision: source.revision, + checksum: source.checksum, + }) + }) + test('reports a reservation-only partial import as incomplete', async () => { const repository = new Repository() const source = memory() diff --git a/sdk/src/services/memory-v2/v1-migration.ts b/sdk/src/services/memory-v2/v1-migration.ts index e2ff977420..405e36288f 100644 --- a/sdk/src/services/memory-v2/v1-migration.ts +++ b/sdk/src/services/memory-v2/v1-migration.ts @@ -111,6 +111,13 @@ const boundedText = ( return { value: trimmed.slice(0, limit), truncated: trimmed.length > limit } } +/** + * stableHash is FNV-1a 32-bit -> a fixed 8-hex-char token. The sanitizing + * regex strips nothing from that output, so the slice(0, 96) cap is a no-op + * safeguard. Collision tolerance is governed by the 32-bit hash space + * (birthday-bound collisions become non-negligible around ~2^16 distinct + * tokens), not by the 96-char cap, which never further shortens the token. + */ const hashToken = (value: string): string => stableHash(value) .replace(/[^A-Za-z0-9._:-]/g, '') @@ -600,6 +607,9 @@ async function findMigrationMarker( let reserved = false const events: MemoryEventEnvelope[] = [] const priorCandidates: ImportedMarkerEvent[] = [] + // Loop-invariant: projectId, revision, and checksum are function params, so + // compute the migration identity once instead of per reservation event. + const identity = getV1MigrationIdentity({ projectId, revision, checksum }) for (let page = 0; page < MAX_EXPORT_PAGES; page++) { const outcome = await repository.export({ @@ -617,8 +627,7 @@ async function findMigrationMarker( if (event.eventType === 'migration.v1.reserved') { if (event.payload.sourceRevision === revision) { if ( - event.payload.migrationId === - getV1MigrationIdentity({ projectId, revision, checksum }) && + event.payload.migrationId === identity && event.payload.sourceChecksum === checksum ) reserved = true @@ -711,13 +720,37 @@ function normalizedEventDraft(event: MemoryEventEnvelope): MemoryEventDraft { return MemoryEventDraftSchema.parse(draft) } +/** + * Canonicalize a value for order-insensitive comparison: plain-object keys are + * emitted in sorted order (matching the recursive key-sort the Bun SQLite + * repository applies via stableJson before persisting), arrays keep their + * order, and primitives pass through. Without this, a record-valued payload + * field (e.g. sourceItemCounts) survives a stableJson round-trip with + * alphabetically sorted keys while the in-memory draft keeps insertion order, + * so a naive JSON.stringify comparison would always report a mismatch on the + * real provider even though both sides carry identical content. + */ +function canonicalizeForCompare(value: unknown): unknown { + if (Array.isArray(value)) return value.map(canonicalizeForCompare) + if (value !== null && typeof value === 'object') { + const record = value as Record + const sorted: Record = {} + for (const key of Object.keys(record).sort()) { + sorted[key] = canonicalizeForCompare(record[key]) + } + return sorted + } + return value +} + function equalEventDraft( expected: MemoryEventDraft, actual: MemoryEventEnvelope, ): boolean { return ( - JSON.stringify(MemoryEventDraftSchema.parse(expected)) === - JSON.stringify(normalizedEventDraft(actual)) + JSON.stringify( + canonicalizeForCompare(MemoryEventDraftSchema.parse(expected)), + ) === JSON.stringify(canonicalizeForCompare(normalizedEventDraft(actual))) ) } From 99affafee84d5a3c959de6865576a3af50cd440d Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Sun, 13 Sep 2026 20:59:23 +0300 Subject: [PATCH 06/22] test(memory): real-SQLite import+audit round-trip for key-order Seed a V1 task-memory record, import it into V2 through the real Bun SQLite backend against a temp project, and assert the migration audit certifies an exact result with matching sourceItemCounts. This exercises the canonicalizeForCompare key-order round-trip end to end: the backend persists event payloads via key-sorted stableJson, so the re-read marker arrives with sorted record keys while the in-memory draft keeps insertion order. --- .../memory-v2-sqlite-roundtrip.test.ts | 151 ++++++++++++++++++ 1 file changed, 151 insertions(+) create mode 100644 sdk/src/__tests__/memory-v2-sqlite-roundtrip.test.ts diff --git a/sdk/src/__tests__/memory-v2-sqlite-roundtrip.test.ts b/sdk/src/__tests__/memory-v2-sqlite-roundtrip.test.ts new file mode 100644 index 0000000000..046a3fe07c --- /dev/null +++ b/sdk/src/__tests__/memory-v2-sqlite-roundtrip.test.ts @@ -0,0 +1,151 @@ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import path from 'node:path' + +import { afterEach, describe, expect, test } from 'bun:test' + +import { + MemorySessionIdSchema, + ProjectIdSchema, +} from '@codebuff/common/types/memory-v2' + +import { + auditTaskMemoryV1Migration, + importTaskMemoryV1, +} from '../services/memory-v2/v1-migration' +import { + loadPersistedTaskMemory, + saveMergedTaskMemory, +} from '../services/task-memory-store' + +import type { TaskMemoryV1 } from '@codebuff/common/types/task-memory' + +// Package placement: this test lives in the sdk package (not beside the cli +// backend test) because the real Bun SQLite backend needs nothing beyond the +// plain `bun test` runner — `bun:sqlite` is built into the Bun runtime and the +// existing backend test (cli/src/services/memory-v2/__tests__/bun-sqlite-memory-repository.test.ts) +// runs with no special preload or flags, exactly like the sdk suite +// (`bun --cwd sdk test`). The backend is imported through the same +// workspace-relative source path the cli backend test itself uses for its own +// imports, so the sdk runner opens the real store unchanged. The migration +// (`importTaskMemoryV1`/`auditTaskMemoryV1Migration`) and V1 store +// (`saveMergedTaskMemory`/`loadPersistedTaskMemory`) functions are the exact +// modules re-exported from the `@openbuff/sdk` package root (sdk/src/index.ts); +// they are imported from source here like every other sdk test, because the +// `@openbuff/sdk` specifier resolves to the built dist/, which plain +// `bun test` does not build first. +import { openBunSQLiteMemoryRepository } from '../../../cli/src/services/memory-v2/bun-sqlite-memory-repository' + +const projectId = ProjectIdSchema.parse('project:sqlite-roundtrip') +const sessionId = MemorySessionIdSchema.parse('memory-cli') + +// Non-empty categories whose marker sourceItemCounts keys are NOT in +// alphabetical order (`decisions` < `path-evidence` < `requirements`). The +// Bun SQLite repository persists event payloads via key-sorted stableJson and +// export() re-parses them, so the re-read marker arrives with sorted record +// keys while the in-memory draft keeps insertion order. Without the +// canonicalizeForCompare fix in equalEventDraft, the audit below mismatches +// on exactly this key-order round-trip and reports imported-body-mismatch. +const runMemory: TaskMemoryV1 = { + schemaVersion: 1, + goal: 'Excluded from the import; retained only in the persisted V1 record.', + requirements: ['req-1'], + decisions: ['dec-1'], + filesInspected: [], + editsMade: [], + validationResults: [], + reviewReceipts: [], + blockers: [], + nextActions: [], + historicalSummary: '', + evidence: [ + { + id: 'ev-1', + kind: 'read', + summary: 'round-trip path evidence', + path: 'src/roundtrip.ts', + stale: false, + }, + ], + revision: 0, + updatedAt: 0, + checksum: 'seeded-by-save', +} + +describe('V1→V2 migration against the real Bun SQLite backend', () => { + let tempRoot: string | undefined + + afterEach(async () => { + if (tempRoot !== undefined) { + rmSync(tempRoot, { recursive: true, force: true }) + tempRoot = undefined + } + }) + + test('import + audit round-trip the marker sourceItemCounts through real SQLite storage', async () => { + tempRoot = mkdtempSync(path.join(tmpdir(), 'openbuff-memory-v2-roundtrip-')) + const rootDir = tempRoot + + // Seed a valid V1 record through the real store and load it back, so the + // persisted checksum/revision round-trip is exercised too. + const saved = await saveMergedTaskMemory({ rootDir, runMemory }) + expect(saved).toBeDefined() + const memory = await loadPersistedTaskMemory({ rootDir }) + expect(memory).toBeDefined() + if (!memory) throw new Error('expected persisted V1 task memory') + expect(memory.requirements).toEqual(['req-1']) + expect(memory.decisions).toEqual(['dec-1']) + expect(memory.evidence).toHaveLength(1) + + // Open the REAL backend: creates /.openbuff/memory/memory-v2.sqlite. + const opened = await openBunSQLiteMemoryRepository({ + repositoryRoot: rootDir, + }) + if (opened.status !== 'ok') { + throw new Error( + `expected the real backend to open, got ${opened.error.kind}: ${opened.error.message}`, + ) + } + const repository = opened.repository + try { + expect(repository.databasePath).toBe( + path.join(rootDir, '.openbuff', 'memory', 'memory-v2.sqlite'), + ) + + const imported = await importTaskMemoryV1({ + memory, + projectId, + sessionId, + repository, + }) + expect(imported.outcome).toBe('imported') + if (imported.outcome !== 'imported') return + expect(imported.sourceItemCounts).toEqual({ + blockers: 0, + decisions: 1, + 'edits-made': 0, + 'files-inspected': 0, + 'historical-summary': 0, + 'next-actions': 0, + 'path-evidence': 1, + requirements: 1, + 'review-receipts': 0, + 'validation-results': 0, + }) + + const audit = await auditTaskMemoryV1Migration({ + memory, + projectId, + repository, + }) + expect(audit.outcome).toBe('exact') + if (audit.outcome !== 'exact') return + // The key assertion: the audit re-derived its counts from the + // SQLite-reparsed marker envelope (sorted record keys) and still + // certifies an exact match against the in-memory import outcome. + expect(audit.sourceItemCounts).toEqual(imported.sourceItemCounts) + } finally { + await repository.close() + } + }) +}) From d9c4c091c1f2ab69ceb1f559dedd0177fc56e6b0 Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Sun, 13 Sep 2026 21:39:50 +0300 Subject: [PATCH 07/22] test(memory): move SQLite migration round-trip test to cli package The sdk-side test reached into cli internals via a cross-package relative path to import the Bun SQLite backend, which a reviewer advisory flagged as fragile. Colocating the test with the existing cli backend test makes the backend import package-local; the migration and V1-store modules are still exercised from sdk source. --- ...mory-v1-migration-sqlite-roundtrip.test.ts | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) rename sdk/src/__tests__/memory-v2-sqlite-roundtrip.test.ts => cli/src/services/memory-v2/__tests__/memory-v1-migration-sqlite-roundtrip.test.ts (79%) diff --git a/sdk/src/__tests__/memory-v2-sqlite-roundtrip.test.ts b/cli/src/services/memory-v2/__tests__/memory-v1-migration-sqlite-roundtrip.test.ts similarity index 79% rename from sdk/src/__tests__/memory-v2-sqlite-roundtrip.test.ts rename to cli/src/services/memory-v2/__tests__/memory-v1-migration-sqlite-roundtrip.test.ts index 046a3fe07c..8c4bac6857 100644 --- a/sdk/src/__tests__/memory-v2-sqlite-roundtrip.test.ts +++ b/cli/src/services/memory-v2/__tests__/memory-v1-migration-sqlite-roundtrip.test.ts @@ -12,29 +12,29 @@ import { import { auditTaskMemoryV1Migration, importTaskMemoryV1, -} from '../services/memory-v2/v1-migration' +} from '../../../../../sdk/src/services/memory-v2/v1-migration' import { loadPersistedTaskMemory, saveMergedTaskMemory, -} from '../services/task-memory-store' +} from '../../../../../sdk/src/services/task-memory-store' import type { TaskMemoryV1 } from '@codebuff/common/types/task-memory' -// Package placement: this test lives in the sdk package (not beside the cli -// backend test) because the real Bun SQLite backend needs nothing beyond the -// plain `bun test` runner — `bun:sqlite` is built into the Bun runtime and the -// existing backend test (cli/src/services/memory-v2/__tests__/bun-sqlite-memory-repository.test.ts) -// runs with no special preload or flags, exactly like the sdk suite -// (`bun --cwd sdk test`). The backend is imported through the same -// workspace-relative source path the cli backend test itself uses for its own -// imports, so the sdk runner opens the real store unchanged. The migration -// (`importTaskMemoryV1`/`auditTaskMemoryV1Migration`) and V1 store -// (`saveMergedTaskMemory`/`loadPersistedTaskMemory`) functions are the exact -// modules re-exported from the `@openbuff/sdk` package root (sdk/src/index.ts); -// they are imported from source here like every other sdk test, because the +// Package placement: this test lives in the cli package, beside the existing +// real-backend test (bun-sqlite-memory-repository.test.ts in this directory), +// so the Bun SQLite backend is imported through a normal package-local +// relative path — a reviewer advisory flagged the former sdk-side location's +// cross-package relative import of this cli internal as fragile, since the sdk +// suite would break if the cli backend file moved. The backend needs nothing +// beyond the plain `bun test` runner: `bun:sqlite` is built into the Bun +// runtime and the sibling backend test runs with no special preload or flags. +// The migration (`importTaskMemoryV1`/`auditTaskMemoryV1Migration`) and V1 +// store (`saveMergedTaskMemory`/`loadPersistedTaskMemory`) functions are the +// exact modules re-exported from the `@openbuff/sdk` package root +// (sdk/src/index.ts); they are imported from SDK source here because the // `@openbuff/sdk` specifier resolves to the built dist/, which plain // `bun test` does not build first. -import { openBunSQLiteMemoryRepository } from '../../../cli/src/services/memory-v2/bun-sqlite-memory-repository' +import { openBunSQLiteMemoryRepository } from '../bun-sqlite-memory-repository' const projectId = ProjectIdSchema.parse('project:sqlite-roundtrip') const sessionId = MemorySessionIdSchema.parse('memory-cli') From ce3a178b5810b37c4226fcaf634b247621c0badf Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Sun, 13 Sep 2026 21:48:32 +0300 Subject: [PATCH 08/22] docs(memory): note order-insensitive migration audit + SQLite round-trip coverage --- cli/knowledge.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cli/knowledge.md b/cli/knowledge.md index 92ece01b49..1e9bfaf5ce 100644 --- a/cli/knowledge.md +++ b/cli/knowledge.md @@ -23,7 +23,7 @@ - Durable planning is entered through `mode:plan`; the standalone `/plan` command is intentionally absent from `COMMAND_REGISTRY` and `SLASH_COMMANDS` so there is one plan-entry path. - Keep the durable-plan quartet registered: `/resume-plan` (`rp`), `/update-plan` (`up`), `/plan-status` (`ps`), and `/lessons` (`lesson`). These commands operate on `.agents/sessions//` artifacts and fall back to the plan-session picker when no target is provided. - `/plans` (`plan-ls`) lists the artifact-bearing sessions returned by `listPlanSessions()` and never prompts the agent, while `/plan-use` (`plan-active`, `use-plan`) writes the `ACTIVE_SESSION` pointer file under `.agents/` and only accepts a bare slug or `.agents/sessions/` — the resolved directory must be exactly one segment under `.agents/sessions/` because the pointer stores bare slugs, so nested paths, non-session paths, missing directories, and artifact-less directories are all rejected before any write. Both share `formatPlanSessionListRow` / `PLAN_SESSIONS_DIR_PREFIX` from `cli/src/commands/plan-artifacts.ts` so the rendered box and the text report cannot drift. -- `/memory` (alias `/mem`, `cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts`) inspects the persisted cross-session task memory for the current project (`.openbuff/memory/task-memory.json`). `/memory status` (the default) reports the record's revision and age, its goal and per-list counts, and how much of its evidence still verifies against disk, listing up to five stale paths; `/memory prune` drops evidence that no longer verifies. Both subcommands are move-aware via `collectWorkspaceMoves` + `WorkspaceJournalService` so a renamed file's evidence rebinds rather than being reported stale and deleted. `/memory audit-migration` is dedicated and read-only: before acquiring V2 it calls `inspectPersistedTaskMemoryV1`, blocks absent/invalid/unreadable states with bounded distinct reports, and passes the exact checksum-verified memory to the SDK audit without a second read. An `exact` audit reconstructs and compares every deterministic source-derived task/observation header and payload plus marker metadata; it is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`. The legacy loader still collapses non-valid inspection states to `undefined` for compatibility. `not-migrated` means removal is not ready and is not a product defect; all outcomes are rendered distinctly with bounded reason text. +- `/memory` (alias `/mem`, `cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts`) inspects the persisted cross-session task memory for the current project (`.openbuff/memory/task-memory.json`). `/memory status` (the default) reports the record's revision and age, its goal and per-list counts, and how much of its evidence still verifies against disk, listing up to five stale paths; `/memory prune` drops evidence that no longer verifies. Both subcommands are move-aware via `collectWorkspaceMoves` + `WorkspaceJournalService` so a renamed file's evidence rebinds rather than being reported stale and deleted. `/memory audit-migration` is dedicated and read-only: before acquiring V2 it calls `inspectPersistedTaskMemoryV1`, blocks absent/invalid/unreadable states with bounded distinct reports, and passes the exact checksum-verified memory to the SDK audit without a second read. An `exact` audit reconstructs and compares every deterministic source-derived task/observation header and payload plus marker metadata; it is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`. The legacy loader still collapses non-valid inspection states to `undefined` for compatibility. `not-migrated` means removal is not ready and is not a product defect; all outcomes are rendered distinctly with bounded reason text. The audit's exactness check is order-insensitive (canonicalized before comparison) because the SQLite backend persists payloads via key-sorted `stableJson` and re-parses them on export; this is covered end-to-end by `cli/src/services/memory-v2/__tests__/memory-v1-migration-sqlite-roundtrip.test.ts`. - Slash-command descriptions should stay model-agnostic under BYOK/local mode. Use wording such as "configured reviewer" rather than naming hosted models. ## Import Guidelines From cfb126d15b266bda95f6c4fe5e4961382ce92c99 Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Sun, 13 Sep 2026 23:00:08 +0300 Subject: [PATCH 09/22] feat(memory): add SQLite secure-open gate and bounded projection replay Race-free WAL/SHM open is provably impossible in pure JS with bun:sqlite, so add an opt-in requireSecureOpen gate that fails closed with a typed non-retryable unsupported-open error and performs zero SQLite mutation, plus an honest openPosture visibility field on the default open path. Bound projection replay at MAX_REPLAY_EVENTS = 10000 so rebuildProjections stops at the cap, parks the cursor at the last replayed sequence, and reports truncated: true instead of falsely claiming the canonical tail; v1-to-v2 migrate passes Number.MAX_SAFE_INTEGER so migration replay stays complete. Also sync STATUS.md plan tracking for both changes. --- .../dynamic-cross-session-memory/STATUS.md | 35 +- .../bun-sqlite-memory-repository.test.ts | 1671 ++++++++++---- .../memory-v2/bun-sqlite-memory-repository.ts | 2052 +++++++++++++---- 3 files changed, 2880 insertions(+), 878 deletions(-) diff --git a/.agents/sessions/dynamic-cross-session-memory/STATUS.md b/.agents/sessions/dynamic-cross-session-memory/STATUS.md index afff8c0635..54e5956b11 100644 --- a/.agents/sessions/dynamic-cross-session-memory/STATUS.md +++ b/.agents/sessions/dynamic-cross-session-memory/STATUS.md @@ -1,8 +1,8 @@ # STATUS — Dynamic Cross-Session Memory V2 Repair -Status: ready for implementation; source work is incomplete and reviewer-blocked. -Current phase: R0 — contract freeze. -Current task: MEM2-R0-T1 Freeze and align canonical contracts. +Status: implementation largely complete; final gate is MEM2-R1-T2 (race-resistant SQLite open). +Current phase: R1 — SQLite kernel / storage security. +Current task: MEM2-R1-T2 Resolve race-resistant SQLite open support (typed-unsupported fallback; full closure needs native-addon authorization). ## Implemented and locally validated before this plan refresh @@ -138,3 +138,32 @@ All declared workflow items complete. - reliability-reviewer: 4 findings repaired (export retry, bounded finishTurn, generation-gated observations, leaf error masking) **Discovery coverage vertical slice** shipped: taskId + workspaceSnapshotId in schema, coordinator, spawn wiring, and 3 new tests. + + +## Plan tracking re-verified against live worktree — 2026-09-12 + +Re-verified the durable plan's stale tracking against the live worktree. Focused suites are green: MEM2-R3-T1 migration/operator/coordinator 96/96; MEM2-R1-T1 SQLite kernel 44/44. The only remaining gate is MEM2-R1-T2 (race-resistant SQLite open). Verified fact: race-free WAL/SHM open is provably impossible in pure JS with bun:sqlite (SQLite opens the -wal/-shm sidecars by derived pathname internally; bun:sqlite accepts a path string only — no fd, no dirfd-relative open). R1-T2 therefore resolves via its typed-unsupported fail-closed fallback (report a typed unavailable outcome, perform no SQLite mutation where race-resistance cannot be proven); full race closure is deferred pending explicit native-addon/dependency authorization (SPEC decision #9, option C). + + +## R1-T2 typed-unsupported secure-open + R7 integration matrix — 2026-09-13T19:33:12.321Z + +MEM2-R1-T2 resolved via its typed-unsupported fail-closed fallback. Verified fact: race-free SQLite open is provably impossible in pure JS with bun:sqlite (path-only constructor; -wal/-shm sidecars open by derived pathname internally, no fd/dirfd support). Implemented a hybrid: a strict `requireSecureOpen` opt-in gate in `BunSQLiteMemoryRepository.open()` that fails closed with a typed non-retryable `unsupported-open` error and performs zero SQLite mutation, plus a default-on honest `openPosture: 'pathname-best-effort-unverified-open'` visibility field on the open result and `kernelHealth`. Default open path and all pre-existing pathname hardening are unchanged. No dependency added (SPEC #9 respected); full WAL/SHM race closure deferred pending native-addon authorization (option C). + +Security review: LOOKS_GOOD, 0 findings (strict gate prevents all mutation; no path/SQL/secret leak; no false security claim; default path unregressed). + +R7 integration matrix re-run against this state (all green): +- SDK Memory V2: 117/117 (coordinator, v1-migration, operator-service, contract, run-cancellation) +- CLI Memory V2: 201/201 across 9 files (SQLite repo 47 incl. 3 new strict-secure-open tests, contained-file-io, provider, roundtrip, memory-command, codebuff-client, env, slash-commands, memory-box) +- common Memory V2 contracts: 42/42 +- agent-runtime: 39/39 (task-memory, memory-v2-context) +- Monorepo typecheck: 11/11 packages pass + + + +## Projection replay cap + R2 verification — 2026-09-13T19:53:39.176Z + +Addressed the gate advisory that rebuildProjections/replayProjections had no event cap. Added `MAX_REPLAY_EVENTS = 10_000` (consistent with `MAX_QUERY_EVENTS`); `replayProjections(database, maxEvents)` now stops the paged replay loop at the cap, and on truncation sets the projection cursor to the last replayed sequence and returns `truncated: true` rather than throwing or falsely claiming the canonical tail. `rebuildProjections()` surfaces `truncated: boolean` on its ok-result. v1→v2 `migrate()` passes `Number.MAX_SAFE_INTEGER` so migration replay stays complete; rollback-on-failure unchanged. Backward-compatible and additive. + +R2 (SDK run/coordinator reliability) verified green: coordinator + run-cancellation 53/53. + +Validation: SQLite focused suite 49/49 (2 new cap tests + 1 updated), V1→V2 round-trip 1/1, cli typecheck clean, Prettier clean. diff --git a/cli/src/services/memory-v2/__tests__/bun-sqlite-memory-repository.test.ts b/cli/src/services/memory-v2/__tests__/bun-sqlite-memory-repository.test.ts index a074ac111a..7fa85da6f3 100644 --- a/cli/src/services/memory-v2/__tests__/bun-sqlite-memory-repository.test.ts +++ b/cli/src/services/memory-v2/__tests__/bun-sqlite-memory-repository.test.ts @@ -1,6 +1,15 @@ import { afterEach, describe, expect, test } from 'bun:test' import { Database } from 'bun:sqlite' -import { chmodSync, mkdirSync, mkdtempSync, readFileSync, statSync, symlinkSync, writeFileSync } from 'node:fs' +import { + chmodSync, + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + statSync, + symlinkSync, + writeFileSync, +} from 'node:fs' import { tmpdir } from 'node:os' import { join } from 'node:path' @@ -25,6 +34,8 @@ import { import { BunSQLiteMemoryRepository, + SQLITE_OPEN_POSTURE, + openBunSQLiteMemoryRepository, type MemoryV2EventInput, } from '../bun-sqlite-memory-repository' @@ -49,10 +60,7 @@ function event( } } -function draft( - eventId: string, - projectId = 'project-1', -): MemoryEventDraft { +function draft(eventId: string, projectId = 'project-1'): MemoryEventDraft { return MemoryEventDraftSchema.parse({ schemaVersion: 2, eventSchemaVersion: 1, @@ -90,22 +98,38 @@ function canonicalDraft( }) } -function evidenceFixture(path = 'src/example.ts', digest = 'sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa') { +function evidenceFixture( + path = 'src/example.ts', + digest = 'sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', +) { return { artifact: { artifactId: `artifact-${path}`, location: path, - classification: { kind: 'source' as const, generated: false, sensitivity: 'internal' as const, labels: [] }, + classification: { + kind: 'source' as const, + generated: false, + sensitivity: 'internal' as const, + labels: [], + }, }, selector: { kind: 'file' as const, path }, - provenance: { origin: 'repository' as const, recordedBy: 'test', sourceEventIds: [], metadata: {} }, + provenance: { + origin: 'repository' as const, + recordedBy: 'test', + sourceEventIds: [], + metadata: {}, + }, capturedAt: '2025-01-02T03:04:05.000Z', contentDigest: digest, excerpt: 'deterministic evidence', } } -function observationFixture(observationId: string, evidence = [evidenceFixture()]) { +function observationFixture( + observationId: string, + evidence = [evidenceFixture()], +) { return { observationId, taskId: 'task-1', @@ -115,7 +139,12 @@ function observationFixture(observationId: string, evidence = [evidenceFixture() confidence: 0.9, evidence, selectors: evidence.map(({ selector }) => selector), - provenance: { origin: 'repository' as const, recordedBy: 'test', sourceEventIds: [], metadata: {} }, + provenance: { + origin: 'repository' as const, + recordedBy: 'test', + sourceEventIds: [], + metadata: {}, + }, tags: ['deterministic'], observedAt: '2025-01-02T03:04:05.000Z', } @@ -145,30 +174,40 @@ const V1_SCHEMA = ` PRAGMA user_version = 1; ` -function insertStoredDraft(database: Database, value: MemoryEventDraft, payloadOverride?: unknown): void { - database.query( - `INSERT INTO memory_events ( +function insertStoredDraft( + database: Database, + value: MemoryEventDraft, + payloadOverride?: unknown, +): void { + database + .query( + `INSERT INTO memory_events ( event_id, idempotency_key, event_type, occurred_at, payload_json, metadata_json, task_id, session_id, artifact_id ) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, NULL)`, - ).run( - value.eventId, - `event:${value.eventId}`, - value.eventType, - value.occurredAt, - JSON.stringify(payloadOverride ?? value.payload), - JSON.stringify({ - schemaVersion: value.schemaVersion, - eventSchemaVersion: value.eventSchemaVersion, - projectId: value.projectId, - sessionId: value.sessionId, - }), - 'taskId' in value.payload ? value.payload.taskId : null, - value.sessionId, - ) + ) + .run( + value.eventId, + `event:${value.eventId}`, + value.eventType, + value.occurredAt, + JSON.stringify(payloadOverride ?? value.payload), + JSON.stringify({ + schemaVersion: value.schemaVersion, + eventSchemaVersion: value.eventSchemaVersion, + projectId: value.projectId, + sessionId: value.sessionId, + }), + 'taskId' in value.payload ? value.payload.taskId : null, + value.sessionId, + ) } -function createV1Fixture(root: string, value: MemoryEventDraft, payloadOverride?: unknown): string { +function createV1Fixture( + root: string, + value: MemoryEventDraft, + payloadOverride?: unknown, +): string { const memory = join(root, '.openbuff', 'memory') mkdirSync(memory, { recursive: true }) const path = join(memory, 'memory-v2.sqlite') @@ -182,14 +221,25 @@ function createV1Fixture(root: string, value: MemoryEventDraft, payloadOverride? return path } -function rewriteStoredProject(path: string, eventId: string, projectId: unknown): void { +function rewriteStoredProject( + path: string, + eventId: string, + projectId: unknown, +): void { const database = new Database(path) try { database.exec('DROP TRIGGER memory_events_no_update') - const metadata = projectId === undefined - ? { schemaVersion: 2, eventSchemaVersion: 1, sessionId: 'session-1' } - : { schemaVersion: 2, eventSchemaVersion: 1, projectId, sessionId: 'session-1' } - database.query('UPDATE memory_events SET metadata_json = ?1 WHERE event_id = ?2') + const metadata = + projectId === undefined + ? { schemaVersion: 2, eventSchemaVersion: 1, sessionId: 'session-1' } + : { + schemaVersion: 2, + eventSchemaVersion: 1, + projectId, + sessionId: 'session-1', + } + database + .query('UPDATE memory_events SET metadata_json = ?1 WHERE event_id = ?2') .run(JSON.stringify(metadata), eventId) database.exec(`CREATE TRIGGER memory_events_no_update BEFORE UPDATE ON memory_events BEGIN SELECT RAISE(ABORT, 'canonical memory events are append only'); END;`) @@ -202,8 +252,14 @@ async function openResult(root: string) { return BunSQLiteMemoryRepository.open({ repositoryRoot: root }) } -async function open(root: string, busyTimeoutMs?: number): Promise { - const result = await BunSQLiteMemoryRepository.open({ repositoryRoot: root, busyTimeoutMs }) +async function open( + root: string, + busyTimeoutMs?: number, +): Promise { + const result = await BunSQLiteMemoryRepository.open({ + repositoryRoot: root, + busyTimeoutMs, + }) if (result.status === 'error') throw new Error(result.error.message) repositories.push(result.repository) return result.repository @@ -216,12 +272,22 @@ function projectBindingState(repository: BunSQLiteMemoryRepository): { } { const database = new Database(repository.databasePath) try { - const events = database.query('SELECT COUNT(*) AS count FROM memory_events').get() as { count: number } - const tasks = database.query('SELECT COUNT(*) AS count FROM memory_tasks').get() as { count: number } + const events = database + .query('SELECT COUNT(*) AS count FROM memory_events') + .get() as { count: number } + const tasks = database + .query('SELECT COUNT(*) AS count FROM memory_tasks') + .get() as { count: number } const binding = database - .query("SELECT value FROM memory_projection_metadata WHERE key = 'project_id'") + .query( + "SELECT value FROM memory_projection_metadata WHERE key = 'project_id'", + ) .get() as { value: string } | null - return { events: events.count, tasks: tasks.count, projectId: binding?.value ?? null } + return { + events: events.count, + tasks: tasks.count, + projectId: binding?.value ?? null, + } } finally { database.close() } @@ -240,22 +306,30 @@ describe('BunSQLiteMemoryRepository', () => { const database = new Database(databasePath) try { - const version = database.query('PRAGMA user_version').get() as { user_version: number } + const version = database.query('PRAGMA user_version').get() as { + user_version: number + } expect(version.user_version).toBe(2) - const names = (database - .query("SELECT name FROM sqlite_master WHERE type = 'table' ORDER BY name") - .all() as Array<{ name: string }>).map(({ name }) => name) - expect(names).toEqual(expect.arrayContaining([ - 'memory_events', - 'memory_tasks', - 'memory_sessions', - 'memory_artifacts', - 'memory_claims', - 'memory_evidence', - 'memory_discoveries', - 'memory_projection_metadata', - 'memory_store_capabilities', - ])) + const names = ( + database + .query( + "SELECT name FROM sqlite_master WHERE type = 'table' ORDER BY name", + ) + .all() as Array<{ name: string }> + ).map(({ name }) => name) + expect(names).toEqual( + expect.arrayContaining([ + 'memory_events', + 'memory_tasks', + 'memory_sessions', + 'memory_artifacts', + 'memory_claims', + 'memory_evidence', + 'memory_discoveries', + 'memory_projection_metadata', + 'memory_store_capabilities', + ]), + ) } finally { database.close() } @@ -281,7 +355,10 @@ describe('BunSQLiteMemoryRepository', () => { const listed = await repository.listEvents() expect(listed.status).toBe('ok') if (listed.status === 'ok') { - expect(listed.events.map(({ eventId }) => eventId)).toEqual(['one', 'two']) + expect(listed.events.map(({ eventId }) => eventId)).toEqual([ + 'one', + 'two', + ]) } }) @@ -296,7 +373,9 @@ describe('BunSQLiteMemoryRepository', () => { const empty = await repository.listEvents() expect(empty).toEqual({ status: 'ok', events: [] }) - expect((await repository.appendEvents([event('original')])).status).toBe('ok') + expect((await repository.appendEvents([event('original')])).status).toBe( + 'ok', + ) const conflict = await repository.appendEvents([ event('different-id', { idempotencyKey: 'key-original' }), ]) @@ -307,7 +386,8 @@ describe('BunSQLiteMemoryRepository', () => { event('original', { idempotencyKey: 'different-key' }), ]) expect(sameIdConflict.status).toBe('error') - if (sameIdConflict.status === 'error') expect(sameIdConflict.error.kind).toBe('invalid') + if (sameIdConflict.status === 'error') + expect(sameIdConflict.error.kind).toBe('invalid') expect(conflict.status).toBe('error') if (conflict.status === 'error') expect(conflict.error.kind).toBe('invalid') }) @@ -315,29 +395,51 @@ describe('BunSQLiteMemoryRepository', () => { test('rejects same-request event and idempotency collisions before binding or mutation', async () => { for (const batch of [ [event('same'), event('same')], - [event('same-id'), event('same-id', { payload: { taskId: 'task-1', value: 'different' } })], - [event('first-key'), event('second-key', { idempotencyKey: 'key-first-key' })], + [ + event('same-id'), + event('same-id', { payload: { taskId: 'task-1', value: 'different' } }), + ], + [ + event('first-key'), + event('second-key', { idempotencyKey: 'key-first-key' }), + ], ]) { const repository = await open(temporaryRepository()) - const rejected = await repository.appendEvents(batch.map((entry) => ({ - ...entry, - metadata: { projectId: 'project-1', schemaVersion: 2, eventSchemaVersion: 1 }, - }))) - expect(rejected).toMatchObject({ status: 'error', error: { kind: 'invalid', retryable: false } }) - expect(projectBindingState(repository)).toEqual({ events: 0, tasks: 0, projectId: null }) + const rejected = await repository.appendEvents( + batch.map((entry) => ({ + ...entry, + metadata: { + projectId: 'project-1', + schemaVersion: 2, + eventSchemaVersion: 1, + }, + })), + ) + expect(rejected).toMatchObject({ + status: 'error', + error: { kind: 'invalid', retryable: false }, + }) + expect(projectBindingState(repository)).toEqual({ + events: 0, + tasks: 0, + projectId: null, + }) } }) test('persists canonical events across close and reopen', async () => { const root = temporaryRepository() const repository = await open(root) - expect((await repository.appendEvents([event('persisted')])).status).toBe('ok') + expect((await repository.appendEvents([event('persisted')])).status).toBe( + 'ok', + ) await repository.close() const reopened = await open(root) const listed = await reopened.listEvents() expect(listed.status).toBe('ok') - if (listed.status === 'ok') expect(listed.events[0]?.eventId).toBe('persisted') + if (listed.status === 'ok') + expect(listed.events[0]?.eventId).toBe('persisted') }) test('uses owner-only directory and SQLite file permissions', async () => { @@ -346,7 +448,10 @@ describe('BunSQLiteMemoryRepository', () => { expect(statSync(join(root, '.openbuff')).mode & 0o777).toBe(0o700) expect(statSync(join(root, '.openbuff', 'memory')).mode & 0o777).toBe(0o700) expect(statSync(repository.databasePath).mode & 0o777).toBe(0o600) - for (const sibling of [`${repository.databasePath}-wal`, `${repository.databasePath}-shm`]) { + for (const sibling of [ + `${repository.databasePath}-wal`, + `${repository.databasePath}-shm`, + ]) { try { expect(statSync(sibling).mode & 0o777).toBe(0o600) } catch { @@ -379,7 +484,9 @@ describe('BunSQLiteMemoryRepository', () => { expect(result.status).toBe('ok') if (result.status !== 'ok') return const fts = result.capabilities.find(({ name }) => name === 'fts5') - const fallback = result.capabilities.find(({ name }) => name === 'lexical_fallback') + const fallback = result.capabilities.find( + ({ name }) => name === 'lexical_fallback', + ) expect(fts).toBeDefined() expect(fts?.available || fts?.fallback === 'lexical-scan-v1').toBe(true) expect(fallback).toEqual({ @@ -409,17 +516,107 @@ describe('BunSQLiteMemoryRepository', () => { const database = new Database(repository.databasePath) try { database.exec('DELETE FROM memory_tasks; DELETE FROM memory_claims;') - database.query("UPDATE memory_projection_metadata SET value = '0' WHERE key = 'cursor'").run() + database + .query( + "UPDATE memory_projection_metadata SET value = '0' WHERE key = 'cursor'", + ) + .run() } finally { database.close() } const rebuilt = await repository.rebuildProjections() - expect(rebuilt).toEqual({ status: 'ok', cursor: 3, projectedEvents: 3 }) + expect(rebuilt).toEqual({ + status: 'ok', + cursor: 3, + projectedEvents: 3, + truncated: false, + }) const after = await repository.getProjectionSnapshot() expect(after).toEqual(before) }) + test('rebuilds a store under the replay cap with cursor at the tail and no truncation', async () => { + const repository = await open(temporaryRepository()) + await repository.appendEvents([ + event('under-cap-one'), + event('under-cap-two'), + ]) + const listed = await repository.listEvents() + expect(listed.status).toBe('ok') + const tail = listed.status === 'ok' ? listed.events.at(-1)!.sequence : -1 + expect(tail).toBe(2) + const rebuilt = await repository.rebuildProjections() + expect(rebuilt.status).toBe('ok') + if (rebuilt.status !== 'ok') return + expect(rebuilt.truncated).toBe(false) + expect(rebuilt.projectedEvents).toBe(2) + expect(rebuilt.cursor).toBe(tail) + expect(rebuilt.cursor).toBe(2) + }) + + test('truncates a rebuild beyond the replay event budget and signals degradation without claiming the tail', async () => { + const repository = await open(temporaryRepository()) + const database = new Database(repository.databasePath) + try { + const insert = database.query( + `INSERT INTO memory_events ( + event_id, idempotency_key, event_type, occurred_at, payload_json, + metadata_json, task_id, session_id, artifact_id + ) VALUES (?1, ?2, 'task.updated', '2025-01-02T03:04:05.000Z', ?3, '{}', 'task-1', 'session-1', NULL)`, + ) + database.exec('BEGIN IMMEDIATE') + for (let index = 0; index < 10_250; index++) { + insert.run( + `replay-cap-${index}`, + `key-replay-cap-${index}`, + JSON.stringify({ taskId: 'task-1', value: index }), + ) + } + database.exec('COMMIT') + } finally { + database.close() + } + + const rebuilt = await repository.rebuildProjections() + expect(rebuilt.status).toBe('ok') + if (rebuilt.status !== 'ok') return + // Deterministic degradation signal: the replay budget bounded the rebuild. + expect(rebuilt.truncated).toBe(true) + expect(rebuilt.projectedEvents).toBe(10_000) + // The cursor must NOT falsely claim the real tail (10_250). + expect(rebuilt.cursor).toBe(10_000) + expect(rebuilt.cursor).not.toBe(10_250) + + const snapshot = await repository.getProjectionSnapshot() + expect(snapshot.status).toBe('ok') + if (snapshot.status !== 'ok') return + expect(snapshot.cursor).toBe(10_000) + + const verification = new Database(repository.databasePath) + try { + const tail = ( + verification + .query( + 'SELECT COALESCE(MAX(sequence), 0) AS sequence FROM memory_events', + ) + .get() as { sequence: number } + ).sequence + const cursor = ( + verification + .query( + "SELECT value FROM memory_projection_metadata WHERE key = 'cursor'", + ) + .get() as { value: string } + ).value + expect(tail).toBe(10_250) + expect(cursor).toBe('10000') + expect(cursor).not.toBe(String(tail)) + } finally { + verification.close() + } + }) + test('reduces canonical lifecycle chains and rebuilds the exact normalized state', async () => { const repository = await open(temporaryRepository()) const sourceOne = observationFixture('source-one') @@ -428,29 +625,66 @@ describe('BunSQLiteMemoryRepository', () => { const events = [ draft('canonical-task'), canonicalDraft('task.transitioned', 'task-transitioned', { - payloadSchemaVersion: 1, taskId: 'task-canonical-task', fromStatus: 'created', toStatus: 'completed', reason: 'done', + payloadSchemaVersion: 1, + taskId: 'task-canonical-task', + fromStatus: 'created', + toStatus: 'completed', + reason: 'done', + }), + canonicalDraft( + 'session.started', + 'session-started', + { + payloadSchemaVersion: 1, + startedAt: '2025-01-02T03:04:05.000Z', + }, + 'canonical-session', + ), + canonicalDraft( + 'session.ended', + 'session-ended', + { + payloadSchemaVersion: 1, + status: 'completed', + endedAt: '2025-01-02T04:04:05.000Z', + }, + 'canonical-session', + ), + canonicalDraft('observation.recorded', 'observation-one', { + payloadSchemaVersion: 1, + observation: sourceOne, + }), + canonicalDraft('observation.recorded', 'observation-two', { + payloadSchemaVersion: 1, + observation: sourceTwo, }), - canonicalDraft('session.started', 'session-started', { - payloadSchemaVersion: 1, startedAt: '2025-01-02T03:04:05.000Z', - }, 'canonical-session'), - canonicalDraft('session.ended', 'session-ended', { - payloadSchemaVersion: 1, status: 'completed', endedAt: '2025-01-02T04:04:05.000Z', - }, 'canonical-session'), - canonicalDraft('observation.recorded', 'observation-one', { payloadSchemaVersion: 1, observation: sourceOne }), - canonicalDraft('observation.recorded', 'observation-two', { payloadSchemaVersion: 1, observation: sourceTwo }), canonicalDraft('evidence.attached', 'evidence-attached', { - payloadSchemaVersion: 1, observationId: 'source-one', evidence: sourceOne.evidence, + payloadSchemaVersion: 1, + observationId: 'source-one', + evidence: sourceOne.evidence, }), canonicalDraft('claim.pinned', 'claim-pinned', { - payloadSchemaVersion: 1, observationId: 'source-one', reason: 'keep', pinnedBy: 'test', pinnedAt: '2025-01-02T03:04:05.000Z', + payloadSchemaVersion: 1, + observationId: 'source-one', + reason: 'keep', + pinnedBy: 'test', + pinnedAt: '2025-01-02T03:04:05.000Z', }), canonicalDraft('evidence.verified', 'evidence-verified', { - payloadSchemaVersion: 1, observationId: 'source-one', selector: sourceOne.evidence[0]!.selector, - verifier: 'test', verifiedAt: '2025-01-02T03:05:05.000Z', observedDigest: sourceOne.evidence[0]!.contentDigest, + payloadSchemaVersion: 1, + observationId: 'source-one', + selector: sourceOne.evidence[0]!.selector, + verifier: 'test', + verifiedAt: '2025-01-02T03:05:05.000Z', + observedDigest: sourceOne.evidence[0]!.contentDigest, }), canonicalDraft('evidence.invalidated', 'evidence-invalidated', { - payloadSchemaVersion: 1, observationId: 'source-one', selector: sourceOne.evidence[0]!.selector, - reason: 'changed', detail: 'changed', invalidatedAt: '2025-01-02T03:06:05.000Z', + payloadSchemaVersion: 1, + observationId: 'source-one', + selector: sourceOne.evidence[0]!.selector, + reason: 'changed', + detail: 'changed', + invalidatedAt: '2025-01-02T03:06:05.000Z', }), canonicalDraft('evidence.rebound', 'evidence-rebound', { payloadSchemaVersion: 1, @@ -460,58 +694,95 @@ describe('BunSQLiteMemoryRepository', () => { reason: 'moved', }), canonicalDraft('claim.corrected', 'claim-corrected', { - payloadSchemaVersion: 1, observationId: 'source-two', correction: observationFixture('corrected', []), reason: 'corrected', + payloadSchemaVersion: 1, + observationId: 'source-two', + correction: observationFixture('corrected', []), + reason: 'corrected', }), canonicalDraft('claim.superseded', 'claim-superseded', { - payloadSchemaVersion: 1, observationId: 'corrected', supersededByObservationId: 'canonical', reason: 'newer', + payloadSchemaVersion: 1, + observationId: 'corrected', + supersededByObservationId: 'canonical', + reason: 'newer', }), canonicalDraft('claim.forgotten', 'claim-forgotten', { - payloadSchemaVersion: 1, observationIds: ['corrected'], reason: 'duplicate', requestedBy: 'test', evidenceDisposition: 'retain-artifacts', + payloadSchemaVersion: 1, + observationIds: ['corrected'], + reason: 'duplicate', + requestedBy: 'test', + evidenceDisposition: 'retain-artifacts', }), canonicalDraft('claim.consolidated', 'claim-consolidated', { - payloadSchemaVersion: 1, sourceObservationIds: ['source-one', 'source-two'], canonicalObservation: canonical, reason: 'merge', + payloadSchemaVersion: 1, + sourceObservationIds: ['source-one', 'source-two'], + canonicalObservation: canonical, + reason: 'merge', }), ] - const appended = await repository.append(MemoryAppendRequestSchema.parse({ - schemaVersion: 2, projectId: 'project-1', events, - })) + const appended = await repository.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-1', + events, + }), + ) expect(appended.outcome).toBe('appended') const before = await repository.getProjectionSnapshot() expect(before.status).toBe('ok') if (before.status !== 'ok') return expect(before.tasks[0]?.state).toMatchObject({ - title: 'Task canonical-task', objective: 'Exercise the public Memory V2 port.', status: 'completed', + title: 'Task canonical-task', + objective: 'Exercise the public Memory V2 port.', + status: 'completed', }) expect(before.sessions[0]).toMatchObject({ - entityId: 'canonical-session', state: { - sessionId: 'canonical-session', status: 'completed', - startedAt: '2025-01-02T03:04:05.000Z', endedAt: '2025-01-02T04:04:05.000Z', + entityId: 'canonical-session', + state: { + sessionId: 'canonical-session', + status: 'completed', + startedAt: '2025-01-02T03:04:05.000Z', + endedAt: '2025-01-02T04:04:05.000Z', }, }) - const attachedEvidence = before.evidence.find(({ state }) => ( - state as { evidence?: { selector?: { path?: string } } } - ).evidence?.selector?.path === 'src/example.ts') + const attachedEvidence = before.evidence.find( + ({ state }) => + (state as { evidence?: { selector?: { path?: string } } }).evidence + ?.selector?.path === 'src/example.ts', + ) expect(attachedEvidence?.state).toMatchObject({ - lifecycle: 'attached', evidence: sourceOne.evidence[0], freshness: {}, + lifecycle: 'attached', + evidence: sourceOne.evidence[0], + freshness: {}, }) - const reboundEvidence = before.evidence.find(({ state }) => ( - state as { evidence?: { selector?: { path?: string } } } - ).evidence?.selector?.path !== 'src/example.ts') + const reboundEvidence = before.evidence.find( + ({ state }) => + (state as { evidence?: { selector?: { path?: string } } }).evidence + ?.selector?.path !== 'src/example.ts', + ) expect(reboundEvidence?.state).toMatchObject({ - lifecycle: 'rebound', freshness: {}, + lifecycle: 'rebound', + freshness: {}, }) - expect(before.discoveries.find(({ entityId }) => entityId === 'source-one')?.state) - .toMatchObject({ lifecycle: 'superseded' }) - expect(before.claims.find(({ entityId }) => entityId === 'source-one')?.state) - .toMatchObject({ lifecycle: 'superseded' }) - expect(before.claims.find(({ entityId }) => entityId === 'canonical')?.state) - .toMatchObject({ lifecycle: 'consolidated', observationId: 'canonical' }) + expect( + before.discoveries.find(({ entityId }) => entityId === 'source-one') + ?.state, + ).toMatchObject({ lifecycle: 'superseded' }) + expect( + before.claims.find(({ entityId }) => entityId === 'source-one')?.state, + ).toMatchObject({ lifecycle: 'superseded' }) + expect( + before.claims.find(({ entityId }) => entityId === 'canonical')?.state, + ).toMatchObject({ lifecycle: 'consolidated', observationId: 'canonical' }) const database = new Database(repository.databasePath) try { database.exec(`DELETE FROM memory_tasks; DELETE FROM memory_sessions; DELETE FROM memory_artifacts; DELETE FROM memory_claims; DELETE FROM memory_evidence; DELETE FROM memory_discoveries;`) - database.query("UPDATE memory_projection_metadata SET value = '0' WHERE key = 'cursor'").run() + database + .query( + "UPDATE memory_projection_metadata SET value = '0' WHERE key = 'cursor'", + ) + .run() } finally { database.close() } @@ -541,19 +812,30 @@ describe('BunSQLiteMemoryRepository', () => { const closed = await repository.kernelHealth() expect(closed.status).toBe('unavailable') expect(closed.failure).toEqual({ - kind: 'closed', message: 'The memory store is closed.', retryable: false, + kind: 'closed', + message: 'The memory store is closed.', + retryable: false, }) const closedAppend = await repository.appendEvents([event('after-close')]) - expect(closedAppend).toMatchObject({ status: 'error', error: { kind: 'closed', retryable: false } }) - if (closedAppend.status === 'error') expect(closedAppend.error.message).not.toContain(repository.databasePath) + expect(closedAppend).toMatchObject({ + status: 'error', + error: { kind: 'closed', retryable: false }, + }) + if (closedAppend.status === 'error') + expect(closedAppend.error.message).not.toContain(repository.databasePath) }) test('classifies a generic local filesystem failure as bounded nonretryable I/O', async () => { const root = temporaryRepository() const fileRoot = join(root, 'not-a-directory') writeFileSync(fileRoot, 'occupied') - const opened = await BunSQLiteMemoryRepository.open({ repositoryRoot: fileRoot }) - expect(opened).toMatchObject({ status: 'error', error: { kind: 'io', retryable: false } }) + const opened = await BunSQLiteMemoryRepository.open({ + repositoryRoot: fileRoot, + }) + expect(opened).toMatchObject({ + status: 'error', + error: { kind: 'io', retryable: false }, + }) if (opened.status === 'error') { expect(opened.error.message.length).toBeLessThan(256) expect(opened.error.message).not.toContain(root) @@ -567,28 +849,53 @@ describe('BunSQLiteMemoryRepository', () => { const readSemanticState = () => { const database = new Database(path, { readonly: true }) try { - const canonicalEvents = database.query( - `SELECT sequence, event_id AS eventId, idempotency_key AS idempotencyKey + const canonicalEvents = database + .query( + `SELECT sequence, event_id AS eventId, idempotency_key AS idempotencyKey FROM memory_events ORDER BY sequence`, - ).all() as Array<{ sequence: number; eventId: string; idempotencyKey: string }> - const canonicalEventCount = (database.query( - 'SELECT COUNT(*) AS count FROM memory_events', - ).get() as { count: number }).count - const binding = database.query( - "SELECT value FROM memory_projection_metadata WHERE key = 'project_id'", - ).get() as { value: string } | null - const capabilities = database.query( - 'SELECT name, available, fallback, value FROM memory_store_capabilities ORDER BY name', - ).all() as Array<{ name: string; available: number; fallback: string | null; value: string }> - const projectionCursor = (database.query( - "SELECT value FROM memory_projection_metadata WHERE key = 'cursor'", - ).get() as { value: string }).value - const userVersion = (database.query( - 'PRAGMA user_version', - ).get() as { user_version: number }).user_version - const quickCheck = (database.query( - 'PRAGMA quick_check(1)', - ).get() as { quick_check: string }).quick_check + ) + .all() as Array<{ + sequence: number + eventId: string + idempotencyKey: string + }> + const canonicalEventCount = ( + database + .query('SELECT COUNT(*) AS count FROM memory_events') + .get() as { count: number } + ).count + const binding = database + .query( + "SELECT value FROM memory_projection_metadata WHERE key = 'project_id'", + ) + .get() as { value: string } | null + const capabilities = database + .query( + 'SELECT name, available, fallback, value FROM memory_store_capabilities ORDER BY name', + ) + .all() as Array<{ + name: string + available: number + fallback: string | null + value: string + }> + const projectionCursor = ( + database + .query( + "SELECT value FROM memory_projection_metadata WHERE key = 'cursor'", + ) + .get() as { value: string } + ).value + const userVersion = ( + database.query('PRAGMA user_version').get() as { + user_version: number + } + ).user_version + const quickCheck = ( + database.query('PRAGMA quick_check(1)').get() as { + quick_check: string + } + ).quick_check return { canonicalEvents, canonicalEventCount, @@ -608,7 +915,9 @@ describe('BunSQLiteMemoryRepository', () => { expect(snapshot.status).toBe('ok') if (snapshot.status === 'ok') { expect(snapshot.cursor).toBe(1) - expect(snapshot.tasks.map(({ entityId }) => entityId)).toEqual(['task-v1-task']) + expect(snapshot.tasks.map(({ entityId }) => entityId)).toEqual([ + 'task-v1-task', + ]) } await repository.close() const afterMigration = readFileSync(path) @@ -628,7 +937,9 @@ describe('BunSQLiteMemoryRepository', () => { await reopened.close() const reopenedState = readSemanticState() expect(reopenedSnapshot).toEqual(snapshot) - expect(reopenedState.canonicalEventCount).toBe(migratedState.canonicalEventCount) + expect(reopenedState.canonicalEventCount).toBe( + migratedState.canonicalEventCount, + ) expect(reopenedState.canonicalEvents).toEqual(migratedState.canonicalEvents) expect(reopenedState.projectBinding).toBe(migratedState.projectBinding) expect(reopenedState.capabilities).toEqual(migratedState.capabilities) @@ -654,29 +965,76 @@ describe('BunSQLiteMemoryRepository', () => { database.close() } } else { - rewriteStoredProject(path, first.eventId, mode === 'missing' ? undefined : 'not a valid project') + rewriteStoredProject( + path, + first.eventId, + mode === 'missing' ? undefined : 'not a valid project', + ) } const before = readFileSync(path) const opened = await openResult(root) - expect(opened).toMatchObject({ status: 'error', error: { kind: 'incompatible', retryable: false } }) + expect(opened).toMatchObject({ + status: 'error', + error: { kind: 'incompatible', retryable: false }, + }) expect(readFileSync(path)).toEqual(before) } }) test('rolls back schema-v1 DDL, binding, cursor, and projections when replay fails', async () => { const root = temporaryRepository() - const path = createV1Fixture(root, draft('bad-v1'), { payloadSchemaVersion: 1, taskId: 'task-bad-v1' }) + const path = createV1Fixture(root, draft('bad-v1'), { + payloadSchemaVersion: 1, + taskId: 'task-bad-v1', + }) const before = readFileSync(path) const opened = await openResult(root) - expect(opened).toMatchObject({ status: 'error', error: { kind: 'incompatible', retryable: false } }) + expect(opened).toMatchObject({ + status: 'error', + error: { kind: 'incompatible', retryable: false }, + }) expect(readFileSync(path)).toEqual(before) const database = new Database(path, { readonly: true }) try { - expect((database.query('PRAGMA user_version').get() as { user_version: number }).user_version).toBe(1) - expect((database.query("SELECT COUNT(*) AS count FROM sqlite_master WHERE type = 'table' AND name = 'memory_tasks'").get() as { count: number }).count).toBe(0) - expect(database.query("SELECT value FROM memory_projection_metadata WHERE key = 'project_id'").get()).toBeNull() - expect((database.query("SELECT value FROM memory_projection_metadata WHERE key = 'cursor'").get() as { value: string }).value).toBe('0') - expect((database.query('SELECT COUNT(*) AS count FROM memory_events').get() as { count: number }).count).toBe(1) + expect( + ( + database.query('PRAGMA user_version').get() as { + user_version: number + } + ).user_version, + ).toBe(1) + expect( + ( + database + .query( + "SELECT COUNT(*) AS count FROM sqlite_master WHERE type = 'table' AND name = 'memory_tasks'", + ) + .get() as { count: number } + ).count, + ).toBe(0) + expect( + database + .query( + "SELECT value FROM memory_projection_metadata WHERE key = 'project_id'", + ) + .get(), + ).toBeNull() + expect( + ( + database + .query( + "SELECT value FROM memory_projection_metadata WHERE key = 'cursor'", + ) + .get() as { value: string } + ).value, + ).toBe('0') + expect( + ( + database + .query('SELECT COUNT(*) AS count FROM memory_events') + .get() as { count: number } + ).count, + ).toBe(1) } finally { database.close() } @@ -739,13 +1097,15 @@ describe('BunSQLiteMemoryRepository', () => { databasePath: join(root, '..', 'escaped.sqlite'), }) expect(escaped.status).toBe('error') - if (escaped.status === 'error') expect(escaped.error.kind).toBe('incompatible') + if (escaped.status === 'error') + expect(escaped.error.kind).toBe('incompatible') const repository = await open(root) expect(await repository.search()).toEqual({ status: 'unsupported', capability: 'semantic-search', - message: 'Memory V2 semantic-search is not implemented by the Bun SQLite kernel.', + message: + 'Memory V2 semantic-search is not implemented by the Bun SQLite kernel.', }) }) @@ -760,11 +1120,13 @@ describe('BunSQLiteMemoryRepository', () => { const parsedAppend = MemoryAppendOutcomeSchema.parse(appended) expect(parsedAppend.outcome).toBe('appended') if (parsedAppend.outcome !== 'appended') return - expect(parsedAppend.entries.map(({ eventId, sequence, duplicate }) => ({ - eventId: String(eventId), - sequence, - duplicate, - }))).toEqual([ + expect( + parsedAppend.entries.map(({ eventId, sequence, duplicate }) => ({ + eventId: String(eventId), + sequence, + duplicate, + })), + ).toEqual([ { eventId: 'public-one', sequence: 1, duplicate: false }, { eventId: 'public-two', sequence: 2, duplicate: false }, ]) @@ -778,11 +1140,15 @@ describe('BunSQLiteMemoryRepository', () => { ) expect(duplicate.outcome).toBe('appended') if (duplicate.outcome !== 'appended') return - expect(duplicate.entries.map(({ eventId, sequence, duplicate: isDuplicate }) => ({ - eventId: String(eventId), - sequence, - duplicate: isDuplicate, - }))).toEqual([{ eventId: 'public-one', sequence: 1, duplicate: true }]) + expect( + duplicate.entries.map( + ({ eventId, sequence, duplicate: isDuplicate }) => ({ + eventId: String(eventId), + sequence, + duplicate: isDuplicate, + }), + ), + ).toEqual([{ eventId: 'public-one', sequence: 1, duplicate: true }]) expect(String(duplicate.lastEventId)).toBe('public-two') const exported = await repository.export( @@ -876,106 +1242,183 @@ describe('BunSQLiteMemoryRepository', () => { test('enforces empty, event, any, legacy, and equivalent dual tail preconditions atomically', async () => { const repository = await open(temporaryRepository()) - const append = (value: Record) => repository.append( - MemoryAppendRequestSchema.parse({ schemaVersion: 2, projectId: 'project-1', ...value }), - ) - expect((await append({ expectedTail: { kind: 'empty' }, events: [draft('tail-one')] })).outcome).toBe('appended') + const append = (value: Record) => + repository.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-1', + ...value, + }), + ) + expect( + ( + await append({ + expectedTail: { kind: 'empty' }, + events: [draft('tail-one')], + }) + ).outcome, + ).toBe('appended') for (const request of [ { expectedTail: { kind: 'empty' }, events: [draft('tail-empty-stale')] }, - { expectedTail: { kind: 'event', eventId: 'missing' }, events: [draft('tail-event-stale')] }, + { + expectedTail: { kind: 'event', eventId: 'missing' }, + events: [draft('tail-event-stale')], + }, { expectedLastEventId: 'missing', events: [draft('tail-legacy-stale')] }, ]) { const outcome = await append(request) - expect(outcome).toMatchObject({ outcome: 'rejected', error: { code: 'conflict', retryable: true } }) + expect(outcome).toMatchObject({ + outcome: 'rejected', + error: { code: 'conflict', retryable: true }, + }) } - expect((await append({ expectedTail: { kind: 'any' }, events: [draft('tail-any')] })).outcome).toBe('appended') - expect((await append({ events: [draft('tail-omitted')] })).outcome).toBe('appended') + expect( + ( + await append({ + expectedTail: { kind: 'any' }, + events: [draft('tail-any')], + }) + ).outcome, + ).toBe('appended') + expect((await append({ events: [draft('tail-omitted')] })).outcome).toBe( + 'appended', + ) const tail = 'tail-omitted' - expect((await append({ - expectedTail: { kind: 'event', eventId: tail }, - expectedLastEventId: tail, - events: [draft('tail-dual')], - })).outcome).toBe('appended') - expect(MemoryAppendRequestSchema.safeParse({ - schemaVersion: 2, - projectId: 'project-1', - expectedTail: { kind: 'empty' }, - expectedLastEventId: 'tail-dual', - events: [draft('tail-invalid-dual')], - }).success).toBe(false) + expect( + ( + await append({ + expectedTail: { kind: 'event', eventId: tail }, + expectedLastEventId: tail, + events: [draft('tail-dual')], + }) + ).outcome, + ).toBe('appended') + expect( + MemoryAppendRequestSchema.safeParse({ + schemaVersion: 2, + projectId: 'project-1', + expectedTail: { kind: 'empty' }, + expectedLastEventId: 'tail-dual', + events: [draft('tail-invalid-dual')], + }).success, + ).toBe(false) }) test('keeps deterministic event ID content conflicts hard under a matching tail', async () => { const repository = await open(temporaryRepository()) - const first = await repository.append(MemoryAppendRequestSchema.parse({ - schemaVersion: 2, - projectId: 'project-1', - expectedTail: { kind: 'empty' }, - events: [draft('same-id')], - })) + const first = await repository.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-1', + expectedTail: { kind: 'empty' }, + events: [draft('same-id')], + }), + ) expect(first.outcome).toBe('appended') const conflicting = draft('same-id') - if (conflicting.eventType !== 'task.created') throw new Error('invalid fixture') + if (conflicting.eventType !== 'task.created') + throw new Error('invalid fixture') conflicting.payload.title = 'Different deterministic content' - const outcome = await repository.append(MemoryAppendRequestSchema.parse({ - schemaVersion: 2, - projectId: 'project-1', - expectedTail: { kind: 'event', eventId: 'same-id' }, - events: [conflicting], - })) - expect(outcome).toMatchObject({ outcome: 'rejected', error: { retryable: false } }) + const outcome = await repository.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-1', + expectedTail: { kind: 'event', eventId: 'same-id' }, + events: [conflicting], + }), + ) + expect(outcome).toMatchObject({ + outcome: 'rejected', + error: { retryable: false }, + }) }) test('binds the database to the first public project before any foreign mutation', async () => { const repository = await open(temporaryRepository()) - const first = await repository.append(MemoryAppendRequestSchema.parse({ - schemaVersion: 2, - projectId: 'project-1', - events: [draft('bound')], - })) + const first = await repository.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-1', + events: [draft('bound')], + }), + ) expect(first.outcome).toBe('appended') - const foreignAppend = await repository.append(MemoryAppendRequestSchema.parse({ - schemaVersion: 2, - projectId: 'project-2', - events: [draft('foreign', 'project-2')], - })) + const foreignAppend = await repository.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-2', + events: [draft('foreign', 'project-2')], + }), + ) expect(foreignAppend.outcome).toBe('rejected') - const unscopedAppend = await repository.appendEvents([event('unscoped-after-binding')]) + const unscopedAppend = await repository.appendEvents([ + event('unscoped-after-binding'), + ]) expect(unscopedAppend.status).toBe('error') - if (unscopedAppend.status === 'error') expect(unscopedAppend.error.kind).toBe('invalid') - const foreignQuery = await repository.query(MemoryRetrievalRequestSchema.parse({ - schemaVersion: 2, queryId: 'foreign-query', projectId: 'project-2', sessionId: 'session-1', - query: 'Task', selectors: [], artifactKinds: [], includeHistorical: false, maxResultsPerCategory: 10, - })) + if (unscopedAppend.status === 'error') + expect(unscopedAppend.error.kind).toBe('invalid') + const foreignQuery = await repository.query( + MemoryRetrievalRequestSchema.parse({ + schemaVersion: 2, + queryId: 'foreign-query', + projectId: 'project-2', + sessionId: 'session-1', + query: 'Task', + selectors: [], + artifactKinds: [], + includeHistorical: false, + maxResultsPerCategory: 10, + }), + ) expect(foreignQuery.outcome).toBe('rejected') - const foreignExport = await repository.export(MemoryExportRequestSchema.parse({ - schemaVersion: 2, projectId: 'project-2', limit: 10, - })) + const foreignExport = await repository.export( + MemoryExportRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-2', + limit: 10, + }), + ) expect(foreignExport.outcome).toBe('rejected') - const foreignRebuild = await repository.rebuild(MemoryRebuildRequestSchema.parse({ - schemaVersion: 2, projectId: 'project-2', rebuildId: 'foreign-rebuild', projectionNames: ['tasks'], - })) + const foreignRebuild = await repository.rebuild( + MemoryRebuildRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-2', + rebuildId: 'foreign-rebuild', + projectionNames: ['tasks'], + }), + ) expect(foreignRebuild.outcome).toBe('rejected') - const foreignVerify = await repository.verify(MemoryVerifyRequestSchema.parse({ - schemaVersion: 2, - projectId: 'project-2', - sessionId: 'session-1', - action: { - kind: 'verify', - observationId: 'observation-1', - selector: { kind: 'file', path: 'src/example.ts' }, - observedDigest: 'sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', - }, - })) + const foreignVerify = await repository.verify( + MemoryVerifyRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-2', + sessionId: 'session-1', + action: { + kind: 'verify', + observationId: 'observation-1', + selector: { kind: 'file', path: 'src/example.ts' }, + observedDigest: + 'sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', + }, + }), + ) expect(foreignVerify.outcome).toBe('rejected') const inventory = await repository.listEvents({ limit: 10 }) - expect(inventory.status === 'ok' ? inventory.events.map(({ eventId }) => eventId) : []).toEqual(['bound']) + expect( + inventory.status === 'ok' + ? inventory.events.map(({ eventId }) => eventId) + : [], + ).toEqual(['bound']) const projection = await repository.getProjectionSnapshot() - expect(projection.status === 'ok' ? projection.tasks.map(({ entityId }) => entityId) : []).toEqual(['task-bound']) + expect( + projection.status === 'ok' + ? projection.tasks.map(({ entityId }) => entityId) + : [], + ).toEqual(['task-bound']) }) test('retrieves idempotent verification retries after more than one thousand events', async () => { @@ -984,20 +1427,34 @@ describe('BunSQLiteMemoryRepository', () => { schemaVersion: 2, projectId: 'project-1', sessionId: 'session-1', - action: { kind: 'verify', observationId: 'observation-1', selector: { kind: 'file', path: 'src/example.ts' }, observedDigest: 'sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' }, + action: { + kind: 'verify', + observationId: 'observation-1', + selector: { kind: 'file', path: 'src/example.ts' }, + observedDigest: + 'sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', + }, }) - const first = MemoryVerifyOutcomeSchema.parse(await repository.verify(request)) + const first = MemoryVerifyOutcomeSchema.parse( + await repository.verify(request), + ) expect(first.outcome).toBe('recorded') - const fillers = Array.from({ length: 1_001 }, (_, index) => draft(`after-verify-${index}`)) + const fillers = Array.from({ length: 1_001 }, (_, index) => + draft(`after-verify-${index}`), + ) for (let offset = 0; offset < fillers.length; offset += 100) { - const appended = await repository.append(MemoryAppendRequestSchema.parse({ - schemaVersion: 2, - projectId: 'project-1', - events: fillers.slice(offset, offset + 100), - })) + const appended = await repository.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-1', + events: fillers.slice(offset, offset + 100), + }), + ) expect(appended.outcome).toBe('appended') } - const retry = MemoryVerifyOutcomeSchema.parse(await repository.verify(request)) + const retry = MemoryVerifyOutcomeSchema.parse( + await repository.verify(request), + ) expect(retry.outcome).toBe('recorded') if (first.outcome === 'recorded' && retry.outcome === 'recorded') { expect(retry.event.eventId).toBe(first.event.eventId) @@ -1011,7 +1468,10 @@ describe('BunSQLiteMemoryRepository', () => { const database = new Database(target, { create: true }) database.close() symlinkSync(target, join(root, 'linked.sqlite')) - const opened = await BunSQLiteMemoryRepository.open({ repositoryRoot: root, databasePath: 'linked.sqlite' }) + const opened = await BunSQLiteMemoryRepository.open({ + repositoryRoot: root, + databasePath: 'linked.sqlite', + }) expect(opened.status).toBe('error') if (opened.status === 'error') { expect(opened.error.kind).toBe('incompatible') @@ -1031,7 +1491,10 @@ describe('BunSQLiteMemoryRepository', () => { } symlinkSync(join(root, 'missing-target'), `${path}${suffix}`) const opened = await openResult(root) - expect(opened).toMatchObject({ status: 'error', error: { kind: 'incompatible', retryable: false } }) + expect(opened).toMatchObject({ + status: 'error', + error: { kind: 'incompatible', retryable: false }, + }) } }) @@ -1062,7 +1525,8 @@ describe('BunSQLiteMemoryRepository', () => { kind: 'verify', observationId: 'observation-1', selector: { kind: 'file', path: 'src/example.ts' }, - observedDigest: 'sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', + observedDigest: + 'sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', }, }), ) @@ -1076,11 +1540,16 @@ describe('BunSQLiteMemoryRepository', () => { schemaVersion: 2, projectId: 'project-1', sessionId: 'session-1', - action: { kind: 'verify', observationId: 'observation-1', selector: { kind: 'file', path: 'src/example.ts' } }, + action: { + kind: 'verify', + observationId: 'observation-1', + selector: { kind: 'file', path: 'src/example.ts' }, + }, }) const rejected = await repository.verify(request) expect(rejected.outcome).toBe('rejected') - if (rejected.outcome === 'rejected') expect(rejected.error.code).toBe('invalid-request') + if (rejected.outcome === 'rejected') + expect(rejected.error.code).toBe('invalid-request') const listed = await repository.listEvents() expect(listed.status === 'ok' ? listed.events : []).toEqual([]) }) @@ -1093,10 +1562,20 @@ describe('BunSQLiteMemoryRepository', () => { sessionId: 'session-1', workspaceRevision: 1, workspaceSnapshotId: 'snapshot-1', - action: { kind: 'verify', observationId: 'observation-1', selector: { kind: 'file', path: 'src/example.ts' }, observedDigest: 'sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' }, + action: { + kind: 'verify', + observationId: 'observation-1', + selector: { kind: 'file', path: 'src/example.ts' }, + observedDigest: + 'sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', + }, }) - const first = MemoryVerifyOutcomeSchema.parse(await repository.verify(request)) - const second = MemoryVerifyOutcomeSchema.parse(await repository.verify(request)) + const first = MemoryVerifyOutcomeSchema.parse( + await repository.verify(request), + ) + const second = MemoryVerifyOutcomeSchema.parse( + await repository.verify(request), + ) expect(first.outcome).toBe('recorded') expect(second.outcome).toBe('recorded') if (first.outcome === 'recorded' && second.outcome === 'recorded') { @@ -1104,7 +1583,9 @@ describe('BunSQLiteMemoryRepository', () => { expect(second.event.sequence).toBe(first.event.sequence) expect(first.event.eventType).toBe('evidence.verified') if (first.event.eventType === 'evidence.verified') { - expect(first.event.payload.observedDigest).toBe('sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa') + expect(first.event.payload.observedDigest).toBe( + 'sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', + ) expect(first.event.payload.workspaceRevision).toBe(1) expect(first.event.payload.workspaceSnapshotId).toBe('snapshot-1') } @@ -1117,9 +1598,14 @@ describe('BunSQLiteMemoryRepository', () => { const repository = await open(temporaryRepository()) const evidence = evidenceFixture() const observation = observationFixture('freshness', [evidence]) - const recorded = canonicalDraft('observation.recorded', 'freshness-observation', { - payloadSchemaVersion: 1, observation, - }) + const recorded = canonicalDraft( + 'observation.recorded', + 'freshness-observation', + { + payloadSchemaVersion: 1, + observation, + }, + ) const verified = canonicalDraft('evidence.verified', 'freshness-verified', { payloadSchemaVersion: 1, observationId: 'freshness', @@ -1128,58 +1614,125 @@ describe('BunSQLiteMemoryRepository', () => { verifiedAt: '2025-01-02T03:05:05.000Z', observedDigest: evidence.contentDigest, }) - expect((await repository.append(MemoryAppendRequestSchema.parse({ - schemaVersion: 2, projectId: 'project-1', events: [recorded, verified], - }))).outcome).toBe('appended') - - const query = async (queryId: string, context: Record = {}) => repository.query( - MemoryRetrievalRequestSchema.parse({ - schemaVersion: 2, queryId, projectId: 'project-1', sessionId: 'session-1', - query: 'Discovery freshness', selectors: [], artifactKinds: [], includeHistorical: false, - maxResultsPerCategory: 10, ...context, - }), - ) + expect( + ( + await repository.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-1', + events: [recorded, verified], + }), + ) + ).outcome, + ).toBe('appended') + + const query = async ( + queryId: string, + context: Record = {}, + ) => + repository.query( + MemoryRetrievalRequestSchema.parse({ + schemaVersion: 2, + queryId, + projectId: 'project-1', + sessionId: 'session-1', + query: 'Discovery freshness', + selectors: [], + artifactKinds: [], + includeHistorical: false, + maxResultsPerCategory: 10, + ...context, + }), + ) const noContext = await query('fresh-none') - expect(noContext.outcome === 'result' ? noContext.result.verifiedKnowledge.length : 0).toBe(1) + expect( + noContext.outcome === 'result' + ? noContext.result.verifiedKnowledge.length + : 0, + ).toBe(1) for (const [eventId, observedDigest] of [ - ['freshness-mismatch', 'sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb'], + [ + 'freshness-mismatch', + 'sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb', + ], ['freshness-missing', undefined], ] as const) { - expect((await repository.append(MemoryAppendRequestSchema.parse({ - schemaVersion: 2, - projectId: 'project-1', - events: [canonicalDraft('evidence.verified', eventId, { - payloadSchemaVersion: 1, - observationId: 'freshness', selector: evidence.selector, verifier: 'test', - verifiedAt: '2025-01-02T03:05:30.000Z', - ...(observedDigest ? { observedDigest } : {}), - })], - }))).outcome).toBe('appended') + expect( + ( + await repository.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-1', + events: [ + canonicalDraft('evidence.verified', eventId, { + payloadSchemaVersion: 1, + observationId: 'freshness', + selector: evidence.selector, + verifier: 'test', + verifiedAt: '2025-01-02T03:05:30.000Z', + ...(observedDigest ? { observedDigest } : {}), + }), + ], + }), + ) + ).outcome, + ).toBe('appended') const result = await query(`${eventId}-query`) - expect(result.outcome === 'result' ? result.result.verifiedKnowledge : []).toEqual([]) - expect(result.outcome === 'result' ? result.result.rereadRequired.length : 0).toBe(1) + expect( + result.outcome === 'result' ? result.result.verifiedKnowledge : [], + ).toEqual([]) + expect( + result.outcome === 'result' ? result.result.rereadRequired.length : 0, + ).toBe(1) } - expect((await repository.append(MemoryAppendRequestSchema.parse({ - schemaVersion: 2, - projectId: 'project-1', - events: [canonicalDraft('evidence.verified', 'freshness-context', { - payloadSchemaVersion: 1, - observationId: 'freshness', selector: evidence.selector, verifier: 'test', - verifiedAt: '2025-01-02T03:06:05.000Z', observedDigest: evidence.contentDigest, - workspaceRevision: 7, workspaceSnapshotId: 'snapshot-7', - })], - }))).outcome).toBe('appended') - const exact = await query('fresh-exact', { workspaceRevision: 7, workspaceSnapshotId: 'snapshot-7' }) - expect(exact.outcome === 'result' ? exact.result.verifiedKnowledge.length : 0).toBe(1) + expect( + ( + await repository.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-1', + events: [ + canonicalDraft('evidence.verified', 'freshness-context', { + payloadSchemaVersion: 1, + observationId: 'freshness', + selector: evidence.selector, + verifier: 'test', + verifiedAt: '2025-01-02T03:06:05.000Z', + observedDigest: evidence.contentDigest, + workspaceRevision: 7, + workspaceSnapshotId: 'snapshot-7', + }), + ], + }), + ) + ).outcome, + ).toBe('appended') + const exact = await query('fresh-exact', { + workspaceRevision: 7, + workspaceSnapshotId: 'snapshot-7', + }) + expect( + exact.outcome === 'result' ? exact.result.verifiedKnowledge.length : 0, + ).toBe(1) for (const [queryId, context] of [ - ['fresh-revision-mismatch', { workspaceRevision: 8, workspaceSnapshotId: 'snapshot-7' }], - ['fresh-snapshot-mismatch', { workspaceRevision: 7, workspaceSnapshotId: 'snapshot-8' }], + [ + 'fresh-revision-mismatch', + { workspaceRevision: 8, workspaceSnapshotId: 'snapshot-7' }, + ], + [ + 'fresh-snapshot-mismatch', + { workspaceRevision: 7, workspaceSnapshotId: 'snapshot-8' }, + ], ['fresh-pair-revision-only', { workspaceRevision: 7 }], ['fresh-pair-snapshot-only', { workspaceSnapshotId: 'snapshot-7' }], ] as const) { const result = await query(queryId, context) - expect(result.outcome === 'result' ? result.result.verifiedKnowledge : []).toEqual([]) - expect(result.outcome === 'result' ? result.result.rereadRequired.length : 0).toBe(1) + expect( + result.outcome === 'result' ? result.result.verifiedKnowledge : [], + ).toEqual([]) + expect( + result.outcome === 'result' ? result.result.rereadRequired.length : 0, + ).toBe(1) } }) @@ -1204,61 +1757,100 @@ describe('BunSQLiteMemoryRepository', () => { expect(queried.result.verifiedKnowledge).toEqual([]) expect(queried.result.rereadRequired).toEqual([]) } - const health = await repository.health(MemoryHealthRequestSchema.parse({ schemaVersion: 2, projectId: 'project-1' })) - expect(health.backend.capabilities).toEqual(expect.arrayContaining(['query', 'verify'])) + const health = await repository.health( + MemoryHealthRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-1', + }), + ) + expect(health.backend.capabilities).toEqual( + expect.arrayContaining(['query', 'verify']), + ) }) test('returns a partial query when the aggregate payload budget is reached', async () => { const repository = await open(temporaryRepository()) const large = 'x'.repeat(1024 * 1024) for (let index = 0; index < 9; index++) { - const appended = await repository.appendEvents([event(`large-${index}`, { - payload: { taskId: 'task-1', value: large }, - metadata: { projectId: 'project-1', schemaVersion: 2, eventSchemaVersion: 1 }, - })]) + const appended = await repository.appendEvents([ + event(`large-${index}`, { + payload: { taskId: 'task-1', value: large }, + metadata: { + projectId: 'project-1', + schemaVersion: 2, + eventSchemaVersion: 1, + }, + }), + ]) expect(appended.status).toBe('ok') } - const queried = await repository.query(MemoryRetrievalRequestSchema.parse({ - schemaVersion: 2, queryId: 'budget-query', projectId: 'project-1', sessionId: 'session-1', - query: 'large', selectors: [], artifactKinds: [], includeHistorical: false, maxResultsPerCategory: 10, - })) + const queried = await repository.query( + MemoryRetrievalRequestSchema.parse({ + schemaVersion: 2, + queryId: 'budget-query', + projectId: 'project-1', + sessionId: 'session-1', + query: 'large', + selectors: [], + artifactKinds: [], + includeHistorical: false, + maxResultsPerCategory: 10, + }), + ) expect(queried.outcome).toBe('result') if (queried.outcome === 'result') { expect(queried.result.matchedTasks).toEqual([]) expect(queried.result.degradation.state).toBe('degraded') if (queried.result.degradation.state === 'degraded') { - expect(queried.result.degradation.reasons.map(({ code }) => code)).toContain('resource-budget') + expect( + queried.result.degradation.reasons.map(({ code }) => code), + ).toContain('resource-budget') } } }) test('rejects malformed rows that claim a recognized canonical event type', async () => { const repository = await open(temporaryRepository()) - const appended = await repository.appendEvents([event('malformed-canonical', { - eventType: 'task.created', - payload: { taskId: 'task-1' }, - metadata: { - projectId: 'project-1', - schemaVersion: 2, - eventSchemaVersion: 1, - sessionId: 'session-1', - }, - })]) + const appended = await repository.appendEvents([ + event('malformed-canonical', { + eventType: 'task.created', + payload: { taskId: 'task-1' }, + metadata: { + projectId: 'project-1', + schemaVersion: 2, + eventSchemaVersion: 1, + sessionId: 'session-1', + }, + }), + ]) expect(appended.status).toBe('ok') - const queried = await repository.query(MemoryRetrievalRequestSchema.parse({ - schemaVersion: 2, queryId: 'malformed-query', projectId: 'project-1', sessionId: 'session-1', - query: 'task', selectors: [], artifactKinds: [], includeHistorical: false, maxResultsPerCategory: 10, - })) + const queried = await repository.query( + MemoryRetrievalRequestSchema.parse({ + schemaVersion: 2, + queryId: 'malformed-query', + projectId: 'project-1', + sessionId: 'session-1', + query: 'task', + selectors: [], + artifactKinds: [], + includeHistorical: false, + maxResultsPerCategory: 10, + }), + ) expect(queried.outcome).toBe('failed') }) test('omits absent optional metadata fields in low-level storage', async () => { const repository = await open(temporaryRepository()) - expect((await repository.appendEvents([event('metadata')])).status).toBe('ok') + expect((await repository.appendEvents([event('metadata')])).status).toBe( + 'ok', + ) const database = new Database(repository.databasePath) try { - const row = database.query('SELECT metadata_json FROM memory_events').get() as { + const row = database + .query('SELECT metadata_json FROM memory_events') + .get() as { metadata_json: string } expect(row.metadata_json).toBe('{}') @@ -1269,64 +1861,144 @@ describe('BunSQLiteMemoryRepository', () => { test('allows an exact lost-response retry despite a stale tail but rejects a mixed retry', async () => { const repository = await open(temporaryRepository()) - const append = (events: MemoryEventDraft[], expectedTail: Record) => repository.append( - MemoryAppendRequestSchema.parse({ schemaVersion: 2, projectId: 'project-1', events, expectedTail }), - ) - expect((await append([draft('retry-one')], { kind: 'empty' })).outcome).toBe('appended') - expect((await repository.append(MemoryAppendRequestSchema.parse({ - schemaVersion: 2, projectId: 'project-1', events: [draft('later')], - }))).outcome).toBe('appended') + const append = ( + events: MemoryEventDraft[], + expectedTail: Record, + ) => + repository.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-1', + events, + expectedTail, + }), + ) + expect( + (await append([draft('retry-one')], { kind: 'empty' })).outcome, + ).toBe('appended') + expect( + ( + await repository.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-1', + events: [draft('later')], + }), + ) + ).outcome, + ).toBe('appended') const retry = await append([draft('retry-one')], { kind: 'empty' }) - expect(retry).toMatchObject({ outcome: 'appended', entries: [{ duplicate: true }] }) - const mixed = await append([draft('retry-one'), draft('new-in-mixed')], { kind: 'empty' }) - expect(mixed).toMatchObject({ outcome: 'rejected', error: { code: 'conflict' } }) + expect(retry).toMatchObject({ + outcome: 'appended', + entries: [{ duplicate: true }], + }) + const mixed = await append([draft('retry-one'), draft('new-in-mixed')], { + kind: 'empty', + }) + expect(mixed).toMatchObject({ + outcome: 'rejected', + error: { code: 'conflict' }, + }) const listed = await repository.listEvents() - expect(listed.status === 'ok' ? listed.events.map(({ eventId }) => eventId) : []).toEqual(['retry-one', 'later']) + expect( + listed.status === 'ok' ? listed.events.map(({ eventId }) => eventId) : [], + ).toEqual(['retry-one', 'later']) }) test('binds a valid project-scoped low-level V2 append immediately', async () => { const repository = await open(temporaryRepository()) - const appended = await repository.appendEvents([event('canonical-low-level', { - metadata: { projectId: 'project-1', schemaVersion: 2, eventSchemaVersion: 1 }, - })]) + const appended = await repository.appendEvents([ + event('canonical-low-level', { + metadata: { + projectId: 'project-1', + schemaVersion: 2, + eventSchemaVersion: 1, + }, + }), + ]) expect(appended.status).toBe('ok') - expect(projectBindingState(repository)).toEqual({ events: 1, tasks: 0, projectId: 'project-1' }) + expect(projectBindingState(repository)).toEqual({ + events: 1, + tasks: 0, + projectId: 'project-1', + }) }) test('keeps legacy writes unbound until canonical binding and then rejects unscoped writes', async () => { const repository = await open(temporaryRepository()) - expect((await repository.appendEvents([event('legacy-before-binding')])).status).toBe('ok') + expect( + (await repository.appendEvents([event('legacy-before-binding')])).status, + ).toBe('ok') expect(projectBindingState(repository).projectId).toBeNull() - expect((await repository.appendEvents([event('canonical-after-legacy', { - metadata: { projectId: 'project-1', schemaVersion: 2, eventSchemaVersion: 1 }, - })])).status).toBe('ok') + expect( + ( + await repository.appendEvents([ + event('canonical-after-legacy', { + metadata: { + projectId: 'project-1', + schemaVersion: 2, + eventSchemaVersion: 1, + }, + }), + ]) + ).status, + ).toBe('ok') expect(projectBindingState(repository).projectId).toBe('project-1') - const unscoped = await repository.appendEvents([event('legacy-after-binding')]) - expect(unscoped).toMatchObject({ status: 'error', error: { kind: 'invalid' } }) - expect(projectBindingState(repository)).toEqual({ events: 2, tasks: 1, projectId: 'project-1' }) + const unscoped = await repository.appendEvents([ + event('legacy-after-binding'), + ]) + expect(unscoped).toMatchObject({ + status: 'error', + error: { kind: 'invalid' }, + }) + expect(projectBindingState(repository)).toEqual({ + events: 2, + tasks: 1, + projectId: 'project-1', + }) }) test('rejects malformed and mixed low-level project identities atomically', async () => { const batches: MemoryV2EventInput[][] = [ - [event('missing-project', { - metadata: { schemaVersion: 2, eventSchemaVersion: 1 }, - })], - [event('invalid-project', { - metadata: { projectId: 'not a valid project', schemaVersion: 2, eventSchemaVersion: 1 }, - })], + [ + event('missing-project', { + metadata: { schemaVersion: 2, eventSchemaVersion: 1 }, + }), + ], + [ + event('invalid-project', { + metadata: { + projectId: 'not a valid project', + schemaVersion: 2, + eventSchemaVersion: 1, + }, + }), + ], [ event('mixed-project-one', { - metadata: { projectId: 'project-1', schemaVersion: 2, eventSchemaVersion: 1 }, + metadata: { + projectId: 'project-1', + schemaVersion: 2, + eventSchemaVersion: 1, + }, }), event('mixed-project-two', { - metadata: { projectId: 'project-2', schemaVersion: 2, eventSchemaVersion: 1 }, + metadata: { + projectId: 'project-2', + schemaVersion: 2, + eventSchemaVersion: 1, + }, }), ], [ event('valid-before-missing', { - metadata: { projectId: 'project-1', schemaVersion: 2, eventSchemaVersion: 1 }, + metadata: { + projectId: 'project-1', + schemaVersion: 2, + eventSchemaVersion: 1, + }, }), event('missing-after-valid', { metadata: { schemaVersion: 2, eventSchemaVersion: 1 }, @@ -1334,7 +2006,11 @@ describe('BunSQLiteMemoryRepository', () => { ], [ event('canonical-before-unscoped', { - metadata: { projectId: 'project-1', schemaVersion: 2, eventSchemaVersion: 1 }, + metadata: { + projectId: 'project-1', + schemaVersion: 2, + eventSchemaVersion: 1, + }, }), event('unscoped-legacy-in-canonical-batch'), ], @@ -1343,28 +2019,68 @@ describe('BunSQLiteMemoryRepository', () => { for (const batch of batches) { const repository = await open(temporaryRepository()) const rejected = await repository.appendEvents(batch) - expect(rejected).toMatchObject({ status: 'error', error: { kind: 'invalid' } }) - expect(projectBindingState(repository)).toEqual({ events: 0, tasks: 0, projectId: null }) + expect(rejected).toMatchObject({ + status: 'error', + error: { kind: 'invalid' }, + }) + expect(projectBindingState(repository)).toEqual({ + events: 0, + tasks: 0, + projectId: null, + }) } }) test('uses UTF-8 byte admission and does not parse an excluded oversized newest row', async () => { const repository = await open(temporaryRepository()) - expect((await repository.appendEvents([event('valid-small', { - metadata: { projectId: 'project-1', schemaVersion: 2, eventSchemaVersion: 1 }, - })])).status).toBe('ok') - expect((await repository.appendEvents([event('oversized-malformed', { - eventType: 'task.created', - payload: { taskId: 'task-1', value: '😀'.repeat(2_100_000) }, - metadata: { projectId: 'project-1', schemaVersion: 2, eventSchemaVersion: 1, sessionId: 'session-1' }, - })])).status).toBe('ok') - const queried = await repository.query(MemoryRetrievalRequestSchema.parse({ - schemaVersion: 2, queryId: 'oversized-query', projectId: 'project-1', sessionId: 'session-1', - query: 'valid', selectors: [], artifactKinds: [], includeHistorical: false, maxResultsPerCategory: 10, - })) + expect( + ( + await repository.appendEvents([ + event('valid-small', { + metadata: { + projectId: 'project-1', + schemaVersion: 2, + eventSchemaVersion: 1, + }, + }), + ]) + ).status, + ).toBe('ok') + expect( + ( + await repository.appendEvents([ + event('oversized-malformed', { + eventType: 'task.created', + payload: { taskId: 'task-1', value: '😀'.repeat(2_100_000) }, + metadata: { + projectId: 'project-1', + schemaVersion: 2, + eventSchemaVersion: 1, + sessionId: 'session-1', + }, + }), + ]) + ).status, + ).toBe('ok') + const queried = await repository.query( + MemoryRetrievalRequestSchema.parse({ + schemaVersion: 2, + queryId: 'oversized-query', + projectId: 'project-1', + sessionId: 'session-1', + query: 'valid', + selectors: [], + artifactKinds: [], + includeHistorical: false, + maxResultsPerCategory: 10, + }), + ) expect(queried.outcome).toBe('result') if (queried.outcome === 'result') { - expect(queried.result.degradation).toMatchObject({ state: 'degraded', reasons: [{ code: 'resource-budget' }] }) + expect(queried.result.degradation).toMatchObject({ + state: 'degraded', + reasons: [{ code: 'resource-budget' }], + }) } }) @@ -1381,33 +2097,66 @@ describe('BunSQLiteMemoryRepository', () => { ) database.exec('BEGIN IMMEDIATE') const metadata = JSON.stringify({ - schemaVersion: 2, eventSchemaVersion: 1, projectId: 'project-1', sessionId: 'session-1', + schemaVersion: 2, + eventSchemaVersion: 1, + projectId: 'project-1', + sessionId: 'session-1', }) insert.run( - 'cap-oldest', 'key-cap-oldest', 'task.created', + 'cap-oldest', + 'key-cap-oldest', + 'task.created', JSON.stringify({ - payloadSchemaVersion: 1, taskId: 'task-oldest', title: 'Excluded oldest', - objective: 'This semantic candidate must be outside the newest-first admission set.', initialStatus: 'created', + payloadSchemaVersion: 1, + taskId: 'task-oldest', + title: 'Excluded oldest', + objective: + 'This semantic candidate must be outside the newest-first admission set.', + initialStatus: 'created', }), - metadata, 'task-oldest', + metadata, + 'task-oldest', ) for (let index = 0; index < 10_000; index++) { - insert.run(`cap-${index}`, `key-cap-${index}`, 'unsupported.low-level', '{}', metadata, null) + insert.run( + `cap-${index}`, + `key-cap-${index}`, + 'unsupported.low-level', + '{}', + metadata, + null, + ) } - database.query("INSERT INTO memory_projection_metadata(key, value) VALUES ('project_id', 'project-1')").run() + database + .query( + "INSERT INTO memory_projection_metadata(key, value) VALUES ('project_id', 'project-1')", + ) + .run() database.exec('COMMIT') } finally { database.close() } - const queried = await repository.query(MemoryRetrievalRequestSchema.parse({ - schemaVersion: 2, queryId: 'cap-query', projectId: 'project-1', sessionId: 'session-1', - query: 'unsupported', selectors: [], artifactKinds: [], includeHistorical: false, maxResultsPerCategory: 10, - })) + const queried = await repository.query( + MemoryRetrievalRequestSchema.parse({ + schemaVersion: 2, + queryId: 'cap-query', + projectId: 'project-1', + sessionId: 'session-1', + query: 'unsupported', + selectors: [], + artifactKinds: [], + includeHistorical: false, + maxResultsPerCategory: 10, + }), + ) expect(queried.outcome).toBe('result') if (queried.outcome === 'result') { expect(queried.result.matchedTasks).toEqual([]) expect(queried.result.verifiedKnowledge).toEqual([]) - expect(queried.result.degradation).toMatchObject({ state: 'degraded', reasons: [{ code: 'result-cap-reached' }] }) + expect(queried.result.degradation).toMatchObject({ + state: 'degraded', + reasons: [{ code: 'result-cap-reached' }], + }) } }) @@ -1426,7 +2175,10 @@ describe('BunSQLiteMemoryRepository', () => { database.close() const before = readFileSync(path) const opened = await openResult(root) - expect(opened).toMatchObject({ status: 'error', error: { kind: 'incompatible', retryable: false } }) + expect(opened).toMatchObject({ + status: 'error', + error: { kind: 'incompatible', retryable: false }, + }) expect(readFileSync(path)).toEqual(before) } }) @@ -1439,8 +2191,13 @@ describe('BunSQLiteMemoryRepository', () => { const database = new Database(path, { create: true }) database.close() mkdirSync(`${path}-wal`) - const opened = await BunSQLiteMemoryRepository.open({ repositoryRoot: root }) - expect(opened).toMatchObject({ status: 'error', error: { kind: 'incompatible' } }) + const opened = await BunSQLiteMemoryRepository.open({ + repositoryRoot: root, + }) + expect(opened).toMatchObject({ + status: 'error', + error: { kind: 'incompatible' }, + }) }) test('canonical rows cannot be updated or deleted', async () => { @@ -1448,10 +2205,12 @@ describe('BunSQLiteMemoryRepository', () => { await repository.appendEvents([event('immutable')]) const database = new Database(repository.databasePath) try { - expect(() => database.exec("UPDATE memory_events SET event_type = 'changed'")) - .toThrow('canonical memory events are append only') - expect(() => database.exec('DELETE FROM memory_events')) - .toThrow('canonical memory events are append only') + expect(() => + database.exec("UPDATE memory_events SET event_type = 'changed'"), + ).toThrow('canonical memory events are append only') + expect(() => database.exec('DELETE FROM memory_events')).toThrow( + 'canonical memory events are append only', + ) } finally { database.close() } @@ -1470,36 +2229,132 @@ describe('BunSQLiteMemoryRepository', () => { workspaceRevision: 3, workspaceSnapshotId: 'snapshot-3', }) - expect((await repository.append(MemoryAppendRequestSchema.parse({ - schemaVersion: 2, - projectId: 'project-1', - events: [ - coverage('coverage-tests-old', 'tests', 'partial'), - coverage('coverage-tests-new', 'tests', 'covered'), - coverage('coverage-risk', 'risk', 'partial'), - ], - }))).outcome).toBe('appended') - - const matching = await repository.query(MemoryRetrievalRequestSchema.parse({ - schemaVersion: 2, queryId: 'coverage-match', projectId: 'project-1', sessionId: 'session-1', - query: 'coverage', selectors: [], artifactKinds: [], includeHistorical: false, - maxResultsPerCategory: 10, workspaceRevision: 3, workspaceSnapshotId: 'snapshot-3', - })) + expect( + ( + await repository.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId: 'project-1', + events: [ + coverage('coverage-tests-old', 'tests', 'partial'), + coverage('coverage-tests-new', 'tests', 'covered'), + coverage('coverage-risk', 'risk', 'partial'), + ], + }), + ) + ).outcome, + ).toBe('appended') + + const matching = await repository.query( + MemoryRetrievalRequestSchema.parse({ + schemaVersion: 2, + queryId: 'coverage-match', + projectId: 'project-1', + sessionId: 'session-1', + query: 'coverage', + selectors: [], + artifactKinds: [], + includeHistorical: false, + maxResultsPerCategory: 10, + workspaceRevision: 3, + workspaceSnapshotId: 'snapshot-3', + }), + ) expect(matching.outcome).toBe('result') if (matching.outcome !== 'result') return const taskId = TaskIdSchema.parse('task-1') expect(matching.result.currentCoverage).toEqual([ - { dimension: 'risk', state: 'partial', taskId, notes: 'risk-partial-notes', workspaceRevision: 3, workspaceSnapshotId: 'snapshot-3' }, - { dimension: 'tests', state: 'covered', taskId, notes: 'tests-covered-notes', workspaceRevision: 3, workspaceSnapshotId: 'snapshot-3' }, + { + dimension: 'risk', + state: 'partial', + taskId, + notes: 'risk-partial-notes', + workspaceRevision: 3, + workspaceSnapshotId: 'snapshot-3', + }, + { + dimension: 'tests', + state: 'covered', + taskId, + notes: 'tests-covered-notes', + workspaceRevision: 3, + workspaceSnapshotId: 'snapshot-3', + }, ]) - const mismatched = await repository.query(MemoryRetrievalRequestSchema.parse({ - schemaVersion: 2, queryId: 'coverage-mismatch', projectId: 'project-1', sessionId: 'session-1', - query: 'coverage', selectors: [], artifactKinds: [], includeHistorical: false, - maxResultsPerCategory: 10, workspaceRevision: 4, workspaceSnapshotId: 'snapshot-3', - })) + const mismatched = await repository.query( + MemoryRetrievalRequestSchema.parse({ + schemaVersion: 2, + queryId: 'coverage-mismatch', + projectId: 'project-1', + sessionId: 'session-1', + query: 'coverage', + selectors: [], + artifactKinds: [], + includeHistorical: false, + maxResultsPerCategory: 10, + workspaceRevision: 4, + workspaceSnapshotId: 'snapshot-3', + }), + ) expect(mismatched.outcome).toBe('result') if (mismatched.outcome !== 'result') return expect(mismatched.result.currentCoverage).toEqual([]) }) }) + +describe('BunSQLiteMemoryRepository strict secure-open gate', () => { + test('fails closed with a typed non-retryable unsupported-open error and performs no SQLite mutation', async () => { + const root = temporaryRepository() + const result = await openBunSQLiteMemoryRepository({ + repositoryRoot: root, + requireSecureOpen: true, + }) + expect(result.status).toBe('error') + if (result.status !== 'error') return + expect(result.error.kind).toBe('unsupported-open') + expect(result.error.retryable).toBe(false) + expect(result.error.message).toContain( + 'bun:sqlite opens the database and its -wal/-shm sidecars by pathname', + ) + expect(result.error.message).toContain('fail-closed') + expect(result.error.message).not.toContain(root) + expect(existsSync(join(root, '.openbuff'))).toBe(false) + }) + + test('leaves a pre-existing store byte-identical when the strict open is refused', async () => { + const root = temporaryRepository() + const repository = await open(root) + expect( + (await repository.appendEvents([event('pre-existing')])).status, + ).toBe('ok') + await repository.close() + const databasePath = join(root, '.openbuff', 'memory', 'memory-v2.sqlite') + const before = readFileSync(databasePath) + + const result = await openBunSQLiteMemoryRepository({ + repositoryRoot: root, + requireSecureOpen: true, + }) + expect(result).toMatchObject({ + status: 'error', + error: { kind: 'unsupported-open', retryable: false }, + }) + expect(readFileSync(databasePath)).toEqual(before) + }) + + test('opens by default and reports the honest best-effort open posture on the result and in kernel health', async () => { + const root = temporaryRepository() + const result = await openBunSQLiteMemoryRepository({ repositoryRoot: root }) + expect(result.status).toBe('ok') + if (result.status !== 'ok') return + repositories.push(result.repository) + expect(result.openPosture).toBe(SQLITE_OPEN_POSTURE) + expect(SQLITE_OPEN_POSTURE).toBe('pathname-best-effort-unverified-open') + const health = await result.repository.kernelHealth() + expect(health.openPosture).toBe(SQLITE_OPEN_POSTURE) + expect( + existsSync(join(root, '.openbuff', 'memory', 'memory-v2.sqlite')), + ).toBe(true) + }) +}) diff --git a/cli/src/services/memory-v2/bun-sqlite-memory-repository.ts b/cli/src/services/memory-v2/bun-sqlite-memory-repository.ts index 94bb271403..ea26b763f5 100644 --- a/cli/src/services/memory-v2/bun-sqlite-memory-repository.ts +++ b/cli/src/services/memory-v2/bun-sqlite-memory-repository.ts @@ -1,6 +1,14 @@ import { Database } from 'bun:sqlite' import { createHash } from 'node:crypto' -import { chmodSync, existsSync, lstatSync, mkdirSync, realpathSync, statSync, type Stats } from 'node:fs' +import { + chmodSync, + existsSync, + lstatSync, + mkdirSync, + realpathSync, + statSync, + type Stats, +} from 'node:fs' import { dirname, isAbsolute, join, relative, resolve } from 'node:path' import { @@ -48,6 +56,7 @@ const MAX_BUSY_TIMEOUT_MS = 10_000 const PAGE_SIZE = 250 const MAX_QUERY_EVENTS = 10_000 const MAX_QUERY_PAYLOAD_BYTES = 8 * 1024 * 1024 +const MAX_REPLAY_EVENTS = 10_000 const CANONICAL_EVENT_TYPES: ReadonlySet = new Set([ 'task.created', 'task.transitioned', @@ -77,6 +86,7 @@ export type MemoryV2FailureKind = | 'incompatible' | 'invalid' | 'io' + | 'unsupported-open' export interface MemoryV2Failure { kind: MemoryV2FailureKind @@ -88,6 +98,19 @@ export type MemoryV2Result = | ({ status: 'ok' } & T) | { status: 'error'; error: MemoryV2Failure } +/** + * The honest posture of a bun:sqlite store open. bun:sqlite accepts only a + * path string, and SQLite opens the main database plus its -wal/-shm sidecars + * by derived pathname inside its own VFS. The pathname hardening in this + * module is therefore best-effort defense-in-depth: it cannot prove that the + * files SQLite opened are the files that were validated beforehand. + */ +export type SQLiteOpenPosture = 'pathname-best-effort-unverified-open' + +/** The single posture every bun:sqlite open has; see SQLiteOpenPosture. */ +export const SQLITE_OPEN_POSTURE: SQLiteOpenPosture = + 'pathname-best-effort-unverified-open' + export interface MemoryV2EventInput { eventId: string idempotencyKey: string @@ -132,6 +155,8 @@ export interface MemoryV2Health { synchronous: string | null projectionCursor: number | null capabilities: MemoryV2Capability[] + /** Honest posture of the underlying open; never a proven-safe claim. */ + openPosture?: SQLiteOpenPosture failure?: MemoryV2Failure } @@ -139,10 +164,23 @@ export interface BunSQLiteMemoryRepositoryOptions { repositoryRoot?: string databasePath?: string busyTimeoutMs?: number + /** + * Strict opt-in gate. When true, open() refuses every bun:sqlite store + * because this driver offers no descriptor-relative or no-follow/beneath + * open (see SQLITE_OPEN_POSTURE): it returns a typed, non-retryable + * 'unsupported-open' failure and performs no SQLite or filesystem mutation. + * Defaults to false, which keeps the store usable with the pathname + * hardening below as best-effort defense-in-depth. + */ + requireSecureOpen?: boolean } export type BunSQLiteMemoryRepositoryOpenResult = - | { status: 'ok'; repository: BunSQLiteMemoryRepository } + | { + status: 'ok' + repository: BunSQLiteMemoryRepository + openPosture: SQLiteOpenPosture + } | { status: 'error'; error: MemoryV2Failure } export interface ProjectionRow { @@ -213,7 +251,9 @@ interface AppendGuard { interface ProjectionDefinition { table: string payloadId: string - explicitId: keyof Pick | null + explicitId: + | keyof Pick + | null } interface FileIdentity { @@ -242,11 +282,27 @@ interface QueryScanResult { const PROJECTIONS: Record = { task: { table: 'memory_tasks', payloadId: 'taskId', explicitId: 'taskId' }, - session: { table: 'memory_sessions', payloadId: 'sessionId', explicitId: 'sessionId' }, - artifact: { table: 'memory_artifacts', payloadId: 'artifactId', explicitId: 'artifactId' }, + session: { + table: 'memory_sessions', + payloadId: 'sessionId', + explicitId: 'sessionId', + }, + artifact: { + table: 'memory_artifacts', + payloadId: 'artifactId', + explicitId: 'artifactId', + }, claim: { table: 'memory_claims', payloadId: 'claimId', explicitId: null }, - evidence: { table: 'memory_evidence', payloadId: 'evidenceId', explicitId: null }, - discovery: { table: 'memory_discoveries', payloadId: 'discoveryId', explicitId: null }, + evidence: { + table: 'memory_evidence', + payloadId: 'evidenceId', + explicitId: null, + }, + discovery: { + table: 'memory_discoveries', + payloadId: 'discoveryId', + explicitId: null, + }, } const PROJECTION_TABLES = Object.values(PROJECTIONS).map(({ table }) => table) @@ -278,24 +334,41 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { let database: Database | undefined try { + // Strict mode refuses every bun:sqlite open before any filesystem or + // SQLite mutation: no descriptor-relative/no-follow open exists for + // this driver, so the requested guarantee cannot be met. + if (options.requireSecureOpen === true) { + throw new MemoryV2StorageError(insecureOpenUnavailableFailure()) + } const preparedPath = prepareDatabasePath(options) const timeout = boundedBusyTimeout(options.busyTimeoutMs) preflightExistingDatabase(preparedPath) - database = new Database(preparedPath.databasePath, { create: true, strict: true }) + database = new Database(preparedPath.databasePath, { + create: true, + strict: true, + }) verifyOpenedDatabasePath(preparedPath) database.exec(`PRAGMA busy_timeout = ${timeout}`) database.exec('PRAGMA foreign_keys = ON') migrate(database) secureDatabaseFiles(preparedPath) database.exec('PRAGMA synchronous = NORMAL') - const journalMode = readPragmaString(database, 'PRAGMA journal_mode = WAL', 'journal_mode') + const journalMode = readPragmaString( + database, + 'PRAGMA journal_mode = WAL', + 'journal_mode', + ) secureDatabaseFiles(preparedPath) recordRuntimeCapabilities(database, journalMode) secureDatabaseFiles(preparedPath) return { status: 'ok', - repository: new BunSQLiteMemoryRepository(database, preparedPath.databasePath), + repository: new BunSQLiteMemoryRepository( + database, + preparedPath.databasePath, + ), + openPosture: SQLITE_OPEN_POSTURE, } } catch (error) { if (database) { @@ -325,26 +398,44 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { async append(request: MemoryAppendRequest): Promise { const parsed = MemoryAppendRequestSchema.safeParse(request) - if (!parsed.success) return rejectedOutcome('The memory append request is invalid.') - if (parsed.data.events.some((event) => event.projectId !== parsed.data.projectId)) { - return rejectedOutcome('Every event must belong to the requested project.') + if (!parsed.success) + return rejectedOutcome('The memory append request is invalid.') + if ( + parsed.data.events.some( + (event) => event.projectId !== parsed.data.projectId, + ) + ) { + return rejectedOutcome( + 'Every event must belong to the requested project.', + ) } const result = this.appendKernel(parsed.data.events, { projectId: parsed.data.projectId, - expectedTail: parsed.data.expectedTail ?? ( - parsed.data.expectedLastEventId === undefined + expectedTail: + parsed.data.expectedTail ?? + (parsed.data.expectedLastEventId === undefined ? undefined - : { kind: 'event', eventId: parsed.data.expectedLastEventId } - ), + : { kind: 'event', eventId: parsed.data.expectedLastEventId }), }) if (result.status === 'error') return appendFailureOutcome(result.error) - if (!result.lastEventId || result.events.length !== parsed.data.events.length) { - return failedOutcome('internal', 'The memory store did not return the appended events.', false) + if ( + !result.lastEventId || + result.events.length !== parsed.data.events.length + ) { + return failedOutcome( + 'internal', + 'The memory store did not return the appended events.', + false, + ) } const lastEventId = MemoryEventIdSchema.safeParse(result.lastEventId) if (!lastEventId.success) { - return failedOutcome('internal', 'The memory store returned an invalid last event ID.', false) + return failedOutcome( + 'internal', + 'The memory store returned an invalid last event ID.', + false, + ) } return { outcome: 'appended', @@ -358,7 +449,11 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { } private appendKernel( - inputs: readonly (MemoryV2EventInput | MemoryEventDraft | MemoryEventEnvelope)[], + inputs: readonly ( + | MemoryV2EventInput + | MemoryEventDraft + | MemoryEventEnvelope + )[], guard?: AppendGuard, ): MemoryV2Result { const unavailable = this.requireOpen() @@ -385,12 +480,16 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { const eventIds = new Set() const idempotencyKeys = new Set() for (const event of events) { - if (eventIds.has(event.eventId) || idempotencyKeys.has(event.idempotencyKey)) { + if ( + eventIds.has(event.eventId) || + idempotencyKeys.has(event.idempotencyKey) + ) { return { status: 'error', error: { kind: 'invalid', - message: 'An append request cannot contain duplicate event IDs or idempotency keys.', + message: + 'An append request cannot contain duplicate event IDs or idempotency keys.', retryable: false, }, } @@ -407,17 +506,24 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { if (!parsedProjectId.success) { return { status: 'error', - error: invalidEventFailure(new Error( - 'Events that claim schemaVersion 2 must include a valid projectId.', - )), + error: invalidEventFailure( + new Error( + 'Events that claim schemaVersion 2 must include a valid projectId.', + ), + ), } } - if (canonicalProjectId !== null && canonicalProjectId !== parsedProjectId.data) { + if ( + canonicalProjectId !== null && + canonicalProjectId !== parsedProjectId.data + ) { return { status: 'error', - error: invalidEventFailure(new Error( - 'Every canonical event in a batch must belong to the same project.', - )), + error: invalidEventFailure( + new Error( + 'Every canonical event in a batch must belong to the same project.', + ), + ), } } canonicalProjectId = parsedProjectId.data @@ -426,24 +532,32 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { try { const append = this.database.transaction((prepared: PreparedEvent[]) => { const boundProjectId = this.readBoundProjectId() - if (boundProjectId !== null && !ProjectIdSchema.safeParse(boundProjectId).success) { + if ( + boundProjectId !== null && + !ProjectIdSchema.safeParse(boundProjectId).success + ) { throw invalidProjectStoreError() } - const existingCanonicalProjects = this.database.query( - `SELECT DISTINCT json_extract(metadata_json, '$.projectId') AS project_id, + const existingCanonicalProjects = this.database + .query( + `SELECT DISTINCT json_extract(metadata_json, '$.projectId') AS project_id, json_type(metadata_json, '$.projectId') AS project_type FROM memory_events WHERE json_extract(metadata_json, '$.schemaVersion') = 2 LIMIT 2`, - ).all() as Array<{ project_id: unknown; project_type: string | null }> + ) + .all() as Array<{ project_id: unknown; project_type: string | null }> let existingCanonicalProjectId: string | null = null for (const row of existingCanonicalProjects) { - const parsedProjectId = row.project_type === 'text' - ? ProjectIdSchema.safeParse(row.project_id) - : null + const parsedProjectId = + row.project_type === 'text' + ? ProjectIdSchema.safeParse(row.project_id) + : null if (!parsedProjectId?.success) throw invalidProjectStoreError() - if (existingCanonicalProjectId !== null - && existingCanonicalProjectId !== parsedProjectId.data) { + if ( + existingCanonicalProjectId !== null && + existingCanonicalProjectId !== parsedProjectId.data + ) { throw invalidProjectStoreError() } existingCanonicalProjectId = parsedProjectId.data @@ -463,8 +577,10 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { retryable: false, }) } - if (projectId !== null - && prepared.some((event) => eventProjectId(event) !== projectId)) { + if ( + projectId !== null && + prepared.some((event) => eventProjectId(event) !== projectId) + ) { throw new MemoryV2StorageError({ kind: 'invalid', message: 'Every event must belong to the requested project.', @@ -486,25 +602,34 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { .all(event.eventId, event.idempotencyKey) as EventRow[] if (existing.length === 0) { hasNewEvent = true - } else if (existing.length !== 1 || !isIdempotentMatch(existing[0], event)) { + } else if ( + existing.length !== 1 || + !isIdempotentMatch(existing[0], event) + ) { throw new MemoryV2StorageError({ kind: 'invalid', - message: 'The event ID or idempotency key conflicts with an existing event.', + message: + 'The event ID or idempotency key conflicts with an existing event.', retryable: false, }) } else { - results.push({ eventId: existing[0].event_id, sequence: existing[0].sequence, duplicate: true }) + results.push({ + eventId: existing[0].event_id, + sequence: existing[0].sequence, + duplicate: true, + }) } } const currentLastEventId = guard ? this.readLastEventIdForProject(guard.projectId) : this.readLastEventId() - const staleTail = guard?.expectedTail?.kind === 'empty' - ? currentLastEventId !== null - : guard?.expectedTail?.kind === 'event' - ? guard.expectedTail.eventId !== currentLastEventId - : false + const staleTail = + guard?.expectedTail?.kind === 'empty' + ? currentLastEventId !== null + : guard?.expectedTail?.kind === 'event' + ? guard.expectedTail.eventId !== currentLastEventId + : false if (hasNewEvent && staleTail) { throw new MemoryV2StorageError({ kind: 'conflict', @@ -525,7 +650,11 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { ) .get(event.eventId) as EventRow | null if (existing) { - results.push({ eventId: existing.event_id, sequence: existing.sequence, duplicate: true }) + results.push({ + eventId: existing.event_id, + sequence: existing.sequence, + duplicate: true, + }) continue } const inserted = this.database @@ -536,8 +665,15 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { ) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)`, ) .run( - event.eventId, event.idempotencyKey, event.eventType, event.occurredAt, - event.payloadJson, event.metadataJson, event.taskId, event.sessionId, event.artifactId, + event.eventId, + event.idempotencyKey, + event.eventType, + event.occurredAt, + event.payloadJson, + event.metadataJson, + event.taskId, + event.sessionId, + event.artifactId, ) const sequence = Number(inserted.lastInsertRowid) applyProjection(this.database, event, sequence) @@ -546,25 +682,33 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { } } - const appendedCount = results.filter(({ duplicate }) => !duplicate).length + const appendedCount = results.filter( + ({ duplicate }) => !duplicate, + ).length const last = guard - ? this.database + ? (this.database .query( `SELECT sequence, event_id FROM memory_events WHERE json_extract(metadata_json, '$.projectId') = ?1 ORDER BY sequence DESC LIMIT 1`, ) - .get(guard.projectId) as { sequence: number; event_id: string } | null - : this.database - .query('SELECT sequence, event_id FROM memory_events ORDER BY sequence DESC LIMIT 1') - .get() as { sequence: number; event_id: string } | null + .get(guard.projectId) as { + sequence: number + event_id: string + } | null) + : (this.database + .query( + 'SELECT sequence, event_id FROM memory_events ORDER BY sequence DESC LIMIT 1', + ) + .get() as { sequence: number; event_id: string } | null) return { events: results, appendedCount, duplicateCount: results.length - appendedCount, - lastSequence: results.length > 0 - ? Math.max(...results.map(({ sequence }) => sequence)) - : last?.sequence ?? 0, + lastSequence: + results.length > 0 + ? Math.max(...results.map(({ sequence }) => sequence)) + : (last?.sequence ?? 0), lastEventId: last?.event_id ?? null, } }) @@ -574,15 +718,20 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { } } - async listEvents(options: { - afterSequence?: number - limit?: number - } = {}): Promise> { + async listEvents( + options: { + afterSequence?: number + limit?: number + } = {}, + ): Promise> { const unavailable = this.requireOpen() if (unavailable) return unavailable const afterSequence = validNonNegativeInteger(options.afterSequence ?? 0) - const limit = Math.min(validPositiveInteger(options.limit ?? PAGE_SIZE), 1_000) + const limit = Math.min( + validPositiveInteger(options.limit ?? PAGE_SIZE), + 1_000, + ) try { const rows = this.database .query( @@ -600,19 +749,25 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { } } - async *iterateEvents(options: { - afterSequence?: number - limit?: number - } = {}): AsyncGenerator { + async *iterateEvents( + options: { + afterSequence?: number + limit?: number + } = {}, + ): AsyncGenerator { let cursor = validNonNegativeInteger(options.afterSequence ?? 0) - let remaining = Math.min(validPositiveInteger(options.limit ?? 1_000), 10_000) + let remaining = Math.min( + validPositiveInteger(options.limit ?? 1_000), + 10_000, + ) while (remaining > 0) { const result = await this.listEvents({ afterSequence: cursor, limit: Math.min(PAGE_SIZE, remaining), }) - if (result.status === 'error') throw new MemoryV2StorageError(result.error) + if (result.status === 'error') + throw new MemoryV2StorageError(result.error) if (result.events.length === 0) return for (const event of result.events) { yield event @@ -622,16 +777,20 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { } } - async rebuildProjections(): Promise> { + async rebuildProjections(): Promise< + MemoryV2Result<{ + cursor: number + projectedEvents: number + truncated: boolean + }> + > { const unavailable = this.requireOpen() if (unavailable) return unavailable try { const rebuild = this.database.transaction(() => { - for (const table of PROJECTION_TABLES) this.database.exec(`DELETE FROM ${table}`) + for (const table of PROJECTION_TABLES) + this.database.exec(`DELETE FROM ${table}`) setProjectionCursor(this.database, 0) return replayProjections(this.database) }) @@ -641,7 +800,9 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { } } - async getProjectionSnapshot(): Promise> { + async getProjectionSnapshot(): Promise< + MemoryV2Result + > { const unavailable = this.requireOpen() if (unavailable) return unavailable @@ -661,7 +822,9 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { } } - async getCapabilities(): Promise> { + async getCapabilities(): Promise< + MemoryV2Result<{ capabilities: MemoryV2Capability[] }> + > { const unavailable = this.requireOpen() if (unavailable) return unavailable try { @@ -673,7 +836,8 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { async query(request: MemoryRetrievalRequest): Promise { const parsed = MemoryRetrievalRequestSchema.safeParse(request) - if (!parsed.success) return rejectedOutcome('The memory query request is invalid.') + if (!parsed.success) + return rejectedOutcome('The memory query request is invalid.') const unavailable = this.requireOpen() if (unavailable) return operationFailureOutcome(unavailable.error) @@ -703,10 +867,17 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { async verify(request: MemoryVerifyRequest): Promise { const parsed = MemoryVerifyRequestSchema.safeParse(request) - if (!parsed.success) return rejectedOutcome('The memory verify request is invalid.') + if (!parsed.success) + return rejectedOutcome('The memory verify request is invalid.') const action = parsed.data.action - if (action.kind === 'verify' && 'path' in action.selector && !action.observedDigest) { - return rejectedOutcome('Path-backed evidence verification requires an observed digest.') + if ( + action.kind === 'verify' && + 'path' in action.selector && + !action.observedDigest + ) { + return rejectedOutcome( + 'Path-backed evidence verification requires an observed digest.', + ) } const identity = stableJson({ projectId: parsed.data.projectId, @@ -716,39 +887,49 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { action, }) const hash = createHash('sha256').update(identity).digest('hex') - const timestamp = new Date(Date.UTC(2020, 0, 1) + (Number.parseInt(hash.slice(0, 8), 16) * 1_000)).toISOString() - const eventType = action.kind === 'verify' - ? 'evidence.verified' - : action.kind === 'invalidate' - ? 'evidence.invalidated' - : 'evidence.rebound' - const payload = action.kind === 'verify' - ? { - payloadSchemaVersion: 1 as const, - observationId: action.observationId, - selector: action.selector, - verifier: 'bun-sqlite-memory-v2', - verifiedAt: timestamp, - ...(action.observedDigest ? { observedDigest: action.observedDigest } : {}), - ...(parsed.data.workspaceRevision ? { workspaceRevision: parsed.data.workspaceRevision } : {}), - ...(parsed.data.workspaceSnapshotId ? { workspaceSnapshotId: parsed.data.workspaceSnapshotId } : {}), - } - : action.kind === 'invalidate' + const timestamp = new Date( + Date.UTC(2020, 0, 1) + Number.parseInt(hash.slice(0, 8), 16) * 1_000, + ).toISOString() + const eventType = + action.kind === 'verify' + ? 'evidence.verified' + : action.kind === 'invalidate' + ? 'evidence.invalidated' + : 'evidence.rebound' + const payload = + action.kind === 'verify' ? { payloadSchemaVersion: 1 as const, observationId: action.observationId, selector: action.selector, - reason: action.reason, - detail: action.detail, - invalidatedAt: timestamp, - } - : { - payloadSchemaVersion: 1 as const, - observationId: action.observationId, - previousSelector: action.previousSelector, - evidence: action.evidence, - reason: action.reason, + verifier: 'bun-sqlite-memory-v2', + verifiedAt: timestamp, + ...(action.observedDigest + ? { observedDigest: action.observedDigest } + : {}), + ...(parsed.data.workspaceRevision + ? { workspaceRevision: parsed.data.workspaceRevision } + : {}), + ...(parsed.data.workspaceSnapshotId + ? { workspaceSnapshotId: parsed.data.workspaceSnapshotId } + : {}), } + : action.kind === 'invalidate' + ? { + payloadSchemaVersion: 1 as const, + observationId: action.observationId, + selector: action.selector, + reason: action.reason, + detail: action.detail, + invalidatedAt: timestamp, + } + : { + payloadSchemaVersion: 1 as const, + observationId: action.observationId, + previousSelector: action.previousSelector, + evidence: action.evidence, + reason: action.reason, + } const draft = MemoryEventDraftSchema.parse({ schemaVersion: 2, eventSchemaVersion: 1, @@ -759,12 +940,15 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { occurredAt: timestamp, payload, }) - const appended = await this.append(MemoryAppendRequestSchema.parse({ - schemaVersion: 2, - projectId: parsed.data.projectId, - events: [draft], - })) - if (appended.outcome !== 'appended') return MemoryVerifyOutcomeSchema.parse(appended) + const appended = await this.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId: parsed.data.projectId, + events: [draft], + }), + ) + if (appended.outcome !== 'appended') + return MemoryVerifyOutcomeSchema.parse(appended) try { const row = this.database .query( @@ -776,16 +960,27 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { LIMIT 1`, ) .get(draft.eventId, parsed.data.projectId) as EventRow | null - if (!row) return failedOutcome('internal', 'The committed verification event was not found.', false) - return MemoryVerifyOutcomeSchema.parse({ outcome: 'recorded', event: envelopeFromRow(row) }) + if (!row) + return failedOutcome( + 'internal', + 'The committed verification event was not found.', + false, + ) + return MemoryVerifyOutcomeSchema.parse({ + outcome: 'recorded', + event: envelopeFromRow(row), + }) } catch (error) { - return MemoryVerifyOutcomeSchema.parse(operationFailureOutcome(classifyStorageError(error))) + return MemoryVerifyOutcomeSchema.parse( + operationFailureOutcome(classifyStorageError(error)), + ) } } async rebuild(request: MemoryRebuildRequest): Promise { const parsed = MemoryRebuildRequestSchema.safeParse(request) - if (!parsed.success) return rejectedOutcome('The memory rebuild request is invalid.') + if (!parsed.success) + return rejectedOutcome('The memory rebuild request is invalid.') if (parsed.data.fromEventId !== undefined) { return rejectedOutcome('Partial projection rebuilds are not supported.') } @@ -806,22 +1001,42 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { async health(request: MemoryHealthRequest): Promise { const parsed = MemoryHealthRequestSchema.safeParse(request) if (!parsed.success) { - return commonHealth('unavailable', ['The memory health request is invalid.']) + return commonHealth('unavailable', [ + 'The memory health request is invalid.', + ]) } - if (this.closed) return commonHealth('unavailable', ['The memory store is closed.']) + if (this.closed) + return commonHealth('unavailable', ['The memory store is closed.']) try { - const schemaVersion = readPragmaNumber(this.database, 'PRAGMA user_version', 'user_version') - const check = readPragmaString(this.database, 'PRAGMA quick_check(1)', 'quick_check') - const journalMode = readPragmaString(this.database, 'PRAGMA journal_mode', 'journal_mode') + const schemaVersion = readPragmaNumber( + this.database, + 'PRAGMA user_version', + 'user_version', + ) + const check = readPragmaString( + this.database, + 'PRAGMA quick_check(1)', + 'quick_check', + ) + const journalMode = readPragmaString( + this.database, + 'PRAGMA journal_mode', + 'journal_mode', + ) const issues: string[] = [] - if (schemaVersion !== SCHEMA_VERSION) issues.push('The memory store schema is incompatible.') - if (check !== 'ok') issues.push('The memory store failed its integrity check.') - if (journalMode.toLowerCase() !== 'wal') issues.push('The memory store is not using WAL mode.') + if (schemaVersion !== SCHEMA_VERSION) + issues.push('The memory store schema is incompatible.') + if (check !== 'ok') + issues.push('The memory store failed its integrity check.') + if (journalMode.toLowerCase() !== 'wal') + issues.push('The memory store is not using WAL mode.') return commonHealth( schemaVersion !== SCHEMA_VERSION || check !== 'ok' ? 'unavailable' - : issues.length > 0 ? 'degraded' : 'healthy', + : issues.length > 0 + ? 'degraded' + : 'healthy', issues, ) } catch (error) { @@ -831,7 +1046,8 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { async export(request: MemoryExportRequest): Promise { const parsed = MemoryExportRequestSchema.safeParse(request) - if (!parsed.success) return rejectedOutcome('The memory export request is invalid.') + if (!parsed.success) + return rejectedOutcome('The memory export request is invalid.') const unavailable = this.requireOpen() if (unavailable) return operationFailureOutcome(unavailable.error) @@ -845,8 +1061,14 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { WHERE event_id = ?1 AND json_extract(metadata_json, '$.projectId') = ?2`, ) - .get(parsed.data.afterEventId, parsed.data.projectId) as { sequence: number } | null - if (!cursor) return rejectedOutcome('The export cursor was not found.', 'not-found') + .get(parsed.data.afterEventId, parsed.data.projectId) as { + sequence: number + } | null + if (!cursor) + return rejectedOutcome( + 'The export cursor was not found.', + 'not-found', + ) afterSequence = cursor.sequence } const rows = this.database @@ -859,14 +1081,20 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { ORDER BY sequence LIMIT ?3`, ) - .all(afterSequence, parsed.data.projectId, parsed.data.limit) as EventRow[] + .all( + afterSequence, + parsed.data.projectId, + parsed.data.limit, + ) as EventRow[] const events = rows.map(envelopeFromRow) return { outcome: 'page', events, - nextAfterEventId: rows.length === parsed.data.limit - ? rows[rows.length - 1]?.event_id as MemoryEventEnvelope['eventId'] - : null, + nextAfterEventId: + rows.length === parsed.data.limit + ? (rows[rows.length - 1] + ?.event_id as MemoryEventEnvelope['eventId']) + : null, } } catch (error) { const failure = classifyStorageError(error) @@ -886,30 +1114,51 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { } try { - const schemaVersion = readPragmaNumber(this.database, 'PRAGMA user_version', 'user_version') + const schemaVersion = readPragmaNumber( + this.database, + 'PRAGMA user_version', + 'user_version', + ) if (schemaVersion !== SCHEMA_VERSION) { - return unavailableHealth({ - kind: 'incompatible', - message: 'The memory store schema is incompatible with this CLI.', - retryable: false, - }, schemaVersion) + return unavailableHealth( + { + kind: 'incompatible', + message: 'The memory store schema is incompatible with this CLI.', + retryable: false, + }, + schemaVersion, + ) } - const check = readPragmaString(this.database, 'PRAGMA quick_check(1)', 'quick_check') + const check = readPragmaString( + this.database, + 'PRAGMA quick_check(1)', + 'quick_check', + ) if (check !== 'ok') { - return unavailableHealth({ - kind: 'corrupt', - message: 'The memory store failed its integrity check.', - retryable: false, - }, schemaVersion) + return unavailableHealth( + { + kind: 'corrupt', + message: 'The memory store failed its integrity check.', + retryable: false, + }, + schemaVersion, + ) } - const journalMode = readPragmaString(this.database, 'PRAGMA journal_mode', 'journal_mode') - const synchronous = String(readPragmaNumber(this.database, 'PRAGMA synchronous', 'synchronous')) + const journalMode = readPragmaString( + this.database, + 'PRAGMA journal_mode', + 'journal_mode', + ) + const synchronous = String( + readPragmaNumber(this.database, 'PRAGMA synchronous', 'synchronous'), + ) return { status: journalMode.toLowerCase() === 'wal' ? 'healthy' : 'degraded', schemaVersion, journalMode, synchronous, + openPosture: SQLITE_OPEN_POSTURE, projectionCursor: this.readProjectionCursor(), capabilities: readCapabilities(this.database), } @@ -954,13 +1203,16 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { if (bound !== null && bound !== projectId) { throw new MemoryV2StorageError({ kind: 'invalid', - message: 'This memory database is already bound to a different project.', + message: + 'This memory database is already bound to a different project.', retryable: false, }) } if (bound === null) { this.database - .query("INSERT INTO memory_projection_metadata(key, value) VALUES ('project_id', ?1)") + .query( + "INSERT INTO memory_projection_metadata(key, value) VALUES ('project_id', ?1)", + ) .run(projectId) } } @@ -974,7 +1226,8 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { if (eventful) { throw new MemoryV2StorageError({ kind: 'invalid', - message: 'This eventful memory database is not bound to a valid project.', + message: + 'This eventful memory database is not bound to a valid project.', retryable: false, }) } @@ -991,7 +1244,9 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { private readBoundProjectId(): string | null { const row = this.database - .query("SELECT value FROM memory_projection_metadata WHERE key = 'project_id'") + .query( + "SELECT value FROM memory_projection_metadata WHERE key = 'project_id'", + ) .get() as { value: string } | null return row?.value ?? null } @@ -1009,14 +1264,18 @@ export class BunSQLiteMemoryRepository implements MemoryRepositoryV2 { private readLastEventId(): string | null { const row = this.database - .query('SELECT event_id FROM memory_events ORDER BY sequence DESC LIMIT 1') + .query( + 'SELECT event_id FROM memory_events ORDER BY sequence DESC LIMIT 1', + ) .get() as { event_id: string } | null return row?.event_id ?? null } private readProjectionCursor(): number { const row = this.database - .query("SELECT value FROM memory_projection_metadata WHERE key = 'cursor'") + .query( + "SELECT value FROM memory_projection_metadata WHERE key = 'cursor'", + ) .get() as { value: string } | null return row ? Number(row.value) : 0 } @@ -1028,10 +1287,14 @@ export async function openBunSQLiteMemoryRepository( return BunSQLiteMemoryRepository.open(options) } -function prepareDatabasePath(options: BunSQLiteMemoryRepositoryOptions): PreparedDatabasePath { +function prepareDatabasePath( + options: BunSQLiteMemoryRepositoryOptions, +): PreparedDatabasePath { const root = resolve(options.repositoryRoot ?? process.cwd()) const requested = options.databasePath ?? DEFAULT_DATABASE_PATH - const databasePath = isAbsolute(requested) ? resolve(requested) : resolve(root, requested) + const databasePath = isAbsolute(requested) + ? resolve(requested) + : resolve(root, requested) if (!isContainedPath(root, databasePath)) throw containedDatabaseError() mkdirSync(root, { recursive: true, mode: 0o700 }) @@ -1049,7 +1312,10 @@ function prepareDatabasePath(options: BunSQLiteMemoryRepositoryOptions): Prepare const realParent = realpathSync(parent) if (!isContainedPath(realRoot, realParent)) throw containedDatabaseError() - const entries = databasePaths(databasePath).map((path) => ({ path, entry: lstatExisting(path) })) + const entries = databasePaths(databasePath).map((path) => ({ + path, + entry: lstatExisting(path), + })) for (const { entry } of entries) if (entry) assertSafeFileEntry(entry) const existingIdentity = entries[0]!.entry ? fileIdentity(databasePath) : null return { databasePath, realRoot, realParent, existingIdentity } @@ -1060,12 +1326,14 @@ function databasePaths(databasePath: string): string[] { } function assertOwned(entry: Stats): void { - if (typeof process.getuid === 'function' && entry.uid !== process.getuid()) throw unsafeFilesystemError() + if (typeof process.getuid === 'function' && entry.uid !== process.getuid()) + throw unsafeFilesystemError() } function assertSafeDirectory(path: string): void { const entry = lstatSync(path) - if (entry.isSymbolicLink() || !entry.isDirectory()) throw unsafeFilesystemError() + if (entry.isSymbolicLink() || !entry.isDirectory()) + throw unsafeFilesystemError() assertOwned(entry) } @@ -1073,8 +1341,13 @@ function lstatExisting(path: string): Stats | null { try { return lstatSync(path) } catch (error) { - if (typeof error === 'object' && error !== null && 'code' in error - && (error as { code: unknown }).code === 'ENOENT') return null + if ( + typeof error === 'object' && + error !== null && + 'code' in error && + (error as { code: unknown }).code === 'ENOENT' + ) + return null throw error } } @@ -1097,13 +1370,19 @@ function fileIdentity(path: string): FileIdentity { function verifyOpenedDatabasePath(prepared: PreparedDatabasePath): void { const realDatabase = realpathSync(prepared.databasePath) - if (!isContainedPath(prepared.realRoot, realDatabase) - || !isContainedPath(prepared.realParent, realDatabase) - || realpathSync(dirname(prepared.databasePath)) !== prepared.realParent) throw containedDatabaseError() + if ( + !isContainedPath(prepared.realRoot, realDatabase) || + !isContainedPath(prepared.realParent, realDatabase) || + realpathSync(dirname(prepared.databasePath)) !== prepared.realParent + ) + throw containedDatabaseError() assertSafeFile(prepared.databasePath) if (prepared.existingIdentity) { const opened = fileIdentity(prepared.databasePath) - if (opened.dev !== prepared.existingIdentity.dev || opened.ino !== prepared.existingIdentity.ino) { + if ( + opened.dev !== prepared.existingIdentity.dev || + opened.ino !== prepared.existingIdentity.ino + ) { throw unsafeFilesystemError() } } @@ -1131,12 +1410,25 @@ function preflightExistingDatabase(prepared: PreparedDatabasePath): void { } let database: Database | undefined try { - database = new Database(prepared.databasePath, { readonly: true, strict: true }) - const version = readPragmaNumber(database, 'PRAGMA user_version', 'user_version') - if (readPragmaString(database, 'PRAGMA quick_check(1)', 'quick_check') !== 'ok') throw corruptStoreError() + database = new Database(prepared.databasePath, { + readonly: true, + strict: true, + }) + const version = readPragmaNumber( + database, + 'PRAGMA user_version', + 'user_version', + ) + if ( + readPragmaString(database, 'PRAGMA quick_check(1)', 'quick_check') !== + 'ok' + ) + throw corruptStoreError() if (version > SCHEMA_VERSION) { throw new MemoryV2StorageError({ - kind: 'incompatible', message: 'The memory store was created by a newer, incompatible CLI.', retryable: false, + kind: 'incompatible', + message: 'The memory store was created by a newer, incompatible CLI.', + retryable: false, }) } if (version === 1 || version === 2) validateSchemaShape(database, version) @@ -1147,74 +1439,133 @@ function preflightExistingDatabase(prepared: PreparedDatabasePath): void { } function validateSchemaShape(database: Database, version: number): void { - const requiredTables = ['memory_events', 'memory_projection_metadata', 'memory_store_capabilities'] + const requiredTables = [ + 'memory_events', + 'memory_projection_metadata', + 'memory_store_capabilities', + ] if (version === 2) requiredTables.push(...PROJECTION_TABLES) - const tables = database.query( - `SELECT name FROM sqlite_master WHERE type = 'table' AND name IN (${requiredTables.map(() => '?').join(',')})`, - ).all(...requiredTables) as Array<{ name: string }> + const tables = database + .query( + `SELECT name FROM sqlite_master WHERE type = 'table' AND name IN (${requiredTables.map(() => '?').join(',')})`, + ) + .all(...requiredTables) as Array<{ name: string }> if (tables.length !== requiredTables.length) throw incompatibleSchemaError() assertTableColumns(database, 'memory_events', [ - 'sequence', 'event_id', 'idempotency_key', 'event_type', 'occurred_at', - 'payload_json', 'metadata_json', 'task_id', 'session_id', 'artifact_id', + 'sequence', + 'event_id', + 'idempotency_key', + 'event_type', + 'occurred_at', + 'payload_json', + 'metadata_json', + 'task_id', + 'session_id', + 'artifact_id', ]) assertTableColumns(database, 'memory_projection_metadata', ['key', 'value']) - assertTableColumns(database, 'memory_store_capabilities', ['name', 'available', 'fallback', 'value']) + assertTableColumns(database, 'memory_store_capabilities', [ + 'name', + 'available', + 'fallback', + 'value', + ]) if (version === 2) { for (const table of PROJECTION_TABLES) { assertTableColumns(database, table, [ - 'entity_id', 'task_id', 'session_id', 'state_json', 'source_sequence', 'updated_at', + 'entity_id', + 'task_id', + 'session_id', + 'state_json', + 'source_sequence', + 'updated_at', ]) } } - const triggers = database.query( - `SELECT name, sql FROM sqlite_master + const triggers = database + .query( + `SELECT name, sql FROM sqlite_master WHERE type = 'trigger' AND name IN ('memory_events_no_update', 'memory_events_no_delete')`, - ).all() as Array<{ name: string; sql: string | null }> + ) + .all() as Array<{ name: string; sql: string | null }> if (triggers.length !== 2) throw incompatibleSchemaError() for (const [name, operation] of [ ['memory_events_no_update', 'update'], ['memory_events_no_delete', 'delete'], ] as const) { - const sql = triggers.find((trigger) => trigger.name === name)?.sql - ?.toLowerCase().replace(/\s+/g, ' ') - if (!sql?.includes(`before ${operation} on memory_events`) - || !sql.includes("raise(abort, 'canonical memory events are append only')")) { + const sql = triggers + .find((trigger) => trigger.name === name) + ?.sql?.toLowerCase() + .replace(/\s+/g, ' ') + if ( + !sql?.includes(`before ${operation} on memory_events`) || + !sql.includes("raise(abort, 'canonical memory events are append only')") + ) { throw incompatibleSchemaError() } } } -function assertTableColumns(database: Database, table: string, required: readonly string[]): void { - const columns = database.query(`PRAGMA table_info(${table})`).all() as Array<{ name: string }> +function assertTableColumns( + database: Database, + table: string, + required: readonly string[], +): void { + const columns = database.query(`PRAGMA table_info(${table})`).all() as Array<{ + name: string + }> const names = new Set(columns.map(({ name }) => name)) if (required.some((name) => !names.has(name))) throw incompatibleSchemaError() } function unsafeFilesystemError(): MemoryV2StorageError { return new MemoryV2StorageError({ - kind: 'incompatible', message: 'The memory store contains an unsafe filesystem entry.', retryable: false, + kind: 'incompatible', + message: 'The memory store contains an unsafe filesystem entry.', + retryable: false, }) } function incompatibleSchemaError(): MemoryV2StorageError { return new MemoryV2StorageError({ - kind: 'incompatible', message: 'The memory store schema is incompatible with this CLI.', retryable: false, + kind: 'incompatible', + message: 'The memory store schema is incompatible with this CLI.', + retryable: false, }) } function corruptStoreError(): MemoryV2StorageError { return new MemoryV2StorageError({ - kind: 'corrupt', message: 'The memory store failed its integrity check.', retryable: false, + kind: 'corrupt', + message: 'The memory store failed its integrity check.', + retryable: false, }) } +/** + * The typed outcome of a refused strict open. bun:sqlite opens the database + * and its -wal/-shm sidecars by pathname and cannot accept a descriptor, so a + * descriptor-relative/no-follow open cannot be proven for any path and the + * store is left unopened. + */ +function insecureOpenUnavailableFailure(): MemoryV2Failure { + return { + kind: 'unsupported-open', + message: + 'SQLite secure open is not available: bun:sqlite opens the database and its -wal/-shm sidecars by pathname; a descriptor-relative/no-follow open cannot be proven, so the store is left unopened (fail-closed).', + retryable: false, + } +} + function isContainedPath(parent: string, child: string): boolean { const relation = relative(parent, child) - return relation !== '..' - && !relation.startsWith(`..${process.platform === 'win32' ? '\\' : '/'}`) - && !isAbsolute(relation) + return ( + relation !== '..' && + !relation.startsWith(`..${process.platform === 'win32' ? '\\' : '/'}`) && + !isAbsolute(relation) + ) } function containedDatabaseError(): MemoryV2StorageError { @@ -1238,7 +1589,11 @@ function boundedBusyTimeout(value: number | undefined): number { } function migrate(database: Database): void { - const version = readPragmaNumber(database, 'PRAGMA user_version', 'user_version') + const version = readPragmaNumber( + database, + 'PRAGMA user_version', + 'user_version', + ) if (version > SCHEMA_VERSION) throw incompatibleSchemaError() database.exec('BEGIN IMMEDIATE') @@ -1246,8 +1601,13 @@ function migrate(database: Database): void { if (version < 1) database.exec(MIGRATION_1) if (version < 2) database.exec(MIGRATION_2) inferAndBindProject(database) - if (version === 1) replayProjections(database) - if (version !== SCHEMA_VERSION) database.exec(`PRAGMA user_version = ${SCHEMA_VERSION}`) + // A v1->v2 migration replay must stay correct-and-complete: truncating it + // would leave the projections partially applied while the migration commits, + // so it intentionally replays fully (no total-event budget). Failure still + // rolls the whole migration back. + if (version === 1) replayProjections(database, Number.MAX_SAFE_INTEGER) + if (version !== SCHEMA_VERSION) + database.exec(`PRAGMA user_version = ${SCHEMA_VERSION}`) database.exec('COMMIT') } catch (error) { database.exec('ROLLBACK') @@ -1256,49 +1616,76 @@ function migrate(database: Database): void { } function inferAndBindProject(database: Database): void { - const bound = database.query( - "SELECT value FROM memory_projection_metadata WHERE key = 'project_id'", - ).get() as { value: string } | null + const bound = database + .query( + "SELECT value FROM memory_projection_metadata WHERE key = 'project_id'", + ) + .get() as { value: string } | null const parsedBound = bound ? ProjectIdSchema.safeParse(bound.value) : null if (parsedBound && !parsedBound.success) throw invalidProjectStoreError() - const count = (database.query('SELECT COUNT(*) AS count FROM memory_events').get() as { count: number }).count + const count = ( + database.query('SELECT COUNT(*) AS count FROM memory_events').get() as { + count: number + } + ).count if (count === 0) return - const rows = database.query( - `SELECT json_extract(metadata_json, '$.projectId') AS project_id, + const rows = database + .query( + `SELECT json_extract(metadata_json, '$.projectId') AS project_id, json_type(metadata_json, '$.projectId') AS project_type FROM memory_events WHERE json_extract(metadata_json, '$.schemaVersion') = 2`, - ).all() as Array<{ project_id: unknown; project_type: string | null }> + ) + .all() as Array<{ project_id: unknown; project_type: string | null }> if (rows.length === 0) return const projects = new Set() for (const row of rows) { - const parsedProjectId = row.project_type === 'text' - ? ProjectIdSchema.safeParse(row.project_id) - : null + const parsedProjectId = + row.project_type === 'text' + ? ProjectIdSchema.safeParse(row.project_id) + : null if (!parsedProjectId?.success) throw invalidProjectStoreError() projects.add(parsedProjectId.data) if (projects.size > 1) throw invalidProjectStoreError() } const projectId = [...projects][0] if (!projectId) throw invalidProjectStoreError() - if (parsedBound?.success && parsedBound.data !== projectId) throw invalidProjectStoreError() - if (!bound) database.query( - "INSERT INTO memory_projection_metadata(key, value) VALUES ('project_id', ?1)", - ).run(projectId) + if (parsedBound?.success && parsedBound.data !== projectId) + throw invalidProjectStoreError() + if (!bound) + database + .query( + "INSERT INTO memory_projection_metadata(key, value) VALUES ('project_id', ?1)", + ) + .run(projectId) } function invalidProjectStoreError(): MemoryV2StorageError { return new MemoryV2StorageError({ - kind: 'incompatible', message: 'The memory store contains invalid or mixed project identities.', retryable: false, + kind: 'incompatible', + message: 'The memory store contains invalid or mixed project identities.', + retryable: false, }) } -function replayProjections(database: Database): { cursor: number; projectedEvents: number } { +function replayProjections( + database: Database, + maxEvents: number = MAX_REPLAY_EVENTS, +): { + cursor: number + projectedEvents: number + truncated: boolean +} { let cursor = 0 let projectedEvents = 0 + let truncated = false while (true) { + if (projectedEvents >= maxEvents) { + truncated = true + break + } const rows = database .query( `SELECT sequence, event_id, idempotency_key, event_type, occurred_at, @@ -1308,7 +1695,10 @@ function replayProjections(database: Database): { cursor: number; projectedEvent ORDER BY sequence LIMIT ?2`, ) - .all(cursor, PAGE_SIZE) as EventRow[] + .all( + cursor, + Math.min(PAGE_SIZE, maxEvents - projectedEvents), + ) as EventRow[] if (rows.length === 0) break for (const row of rows) { validateCanonicalProjectionRow(row) @@ -1317,12 +1707,22 @@ function replayProjections(database: Database): { cursor: number; projectedEvent projectedEvents += 1 } } - const tail = (database - .query('SELECT COALESCE(MAX(sequence), 0) AS sequence FROM memory_events') - .get() as { sequence: number }).sequence - if (cursor !== tail) throw new Error('Projection replay did not reach the canonical tail.') + const tail = ( + database + .query('SELECT COALESCE(MAX(sequence), 0) AS sequence FROM memory_events') + .get() as { sequence: number } + ).sequence + if (truncated && cursor < tail) { + // The replay budget was exhausted before reaching the canonical tail. Record + // the last replayed sequence as the cursor (not the tail, which would falsely + // claim the projections are complete) and surface the truncation signal. + setProjectionCursor(database, cursor) + return { cursor, projectedEvents, truncated: true } + } + if (cursor !== tail) + throw new Error('Projection replay did not reach the canonical tail.') setProjectionCursor(database, tail) - return { cursor: tail, projectedEvents } + return { cursor: tail, projectedEvents, truncated: false } } const MIGRATION_1 = ` @@ -1384,10 +1784,15 @@ const MIGRATION_2 = ` ) WITHOUT ROWID; ` -function recordRuntimeCapabilities(database: Database, journalMode: string): void { +function recordRuntimeCapabilities( + database: Database, + journalMode: string, +): void { let fts5 = false try { - database.exec('CREATE VIRTUAL TABLE temp.memory_v2_fts_probe USING fts5(value)') + database.exec( + 'CREATE VIRTUAL TABLE temp.memory_v2_fts_probe USING fts5(value)', + ) database.exec('DROP TABLE temp.memory_v2_fts_probe') fts5 = true } catch { @@ -1405,8 +1810,18 @@ function recordRuntimeCapabilities(database: Database, journalMode: string): voi OR fallback IS NOT excluded.fallback OR value IS NOT excluded.value`, ) - write.run('fts5', fts5 ? 1 : 0, fts5 ? null : 'lexical-scan-v1', fts5 ? 'fts5' : 'lexical-scan-v1') - write.run('journal_mode', journalMode.toLowerCase() === 'wal' ? 1 : 0, journalMode.toLowerCase() === 'wal' ? null : journalMode, journalMode) + write.run( + 'fts5', + fts5 ? 1 : 0, + fts5 ? null : 'lexical-scan-v1', + fts5 ? 'fts5' : 'lexical-scan-v1', + ) + write.run( + 'journal_mode', + journalMode.toLowerCase() === 'wal' ? 1 : 0, + journalMode.toLowerCase() === 'wal' ? null : journalMode, + journalMode, + ) write.run('lexical_fallback', 1, null, 'unicode-codepoint-order-v1') write.run('query', 1, null, 'bounded-lexical-v1') write.run('verify', 1, null, 'append-only-v1') @@ -1415,7 +1830,8 @@ function recordRuntimeCapabilities(database: Database, journalMode: string): voi function prepareEvent( input: MemoryV2EventInput | MemoryEventDraft | MemoryEventEnvelope, ): PreparedEvent { - if (!input || typeof input !== 'object') throw new Error('Event must be an object.') + if (!input || typeof input !== 'object') + throw new Error('Event must be an object.') const value = input as unknown as Record const eventId = requiredString(value.eventId ?? value.id, 'eventId', 512) const isEnvelope = value.schemaVersion === 2 && value.eventSchemaVersion === 1 @@ -1424,19 +1840,31 @@ function prepareEvent( 'idempotencyKey', 512, ) - const eventType = requiredString(value.eventType ?? value.type, 'eventType', 256) - const occurredAt = requiredString(value.occurredAt ?? value.timestamp, 'occurredAt', 64) - if (Number.isNaN(Date.parse(occurredAt))) throw new Error('occurredAt must be an ISO timestamp.') + const eventType = requiredString( + value.eventType ?? value.type, + 'eventType', + 256, + ) + const occurredAt = requiredString( + value.occurredAt ?? value.timestamp, + 'occurredAt', + 64, + ) + if (Number.isNaN(Date.parse(occurredAt))) + throw new Error('occurredAt must be an ISO timestamp.') const payload = value.payload ?? {} - const payloadRecord = payload && typeof payload === 'object' - ? payload as Record - : {} - const metadata = value.metadata ?? compactObject({ - schemaVersion: value.schemaVersion, - eventSchemaVersion: value.eventSchemaVersion, - projectId: value.projectId, - sessionId: value.sessionId, - }) + const payloadRecord = + payload && typeof payload === 'object' + ? (payload as Record) + : {} + const metadata = + value.metadata ?? + compactObject({ + schemaVersion: value.schemaVersion, + eventSchemaVersion: value.eventSchemaVersion, + projectId: value.projectId, + sessionId: value.sessionId, + }) return { eventId, @@ -1449,20 +1877,30 @@ function prepareEvent( metadataJson: stableJson(metadata), taskId: optionalString(value.taskId ?? payloadRecord.taskId, 'taskId', 512), sessionId: optionalString(value.sessionId, 'sessionId', 512), - artifactId: optionalString(value.artifactId ?? payloadRecord.artifactId, 'artifactId', 512), + artifactId: optionalString( + value.artifactId ?? payloadRecord.artifactId, + 'artifactId', + 512, + ), } } -function compactObject(record: Record): Record { - return Object.fromEntries(Object.entries(record).filter(([, value]) => value !== undefined)) +function compactObject( + record: Record, +): Record { + return Object.fromEntries( + Object.entries(record).filter(([, value]) => value !== undefined), + ) } function stableJson(value: unknown): string { const seen = new Set() const normalize = (item: unknown): unknown => { - if (item === null || typeof item === 'string' || typeof item === 'boolean') return item + if (item === null || typeof item === 'string' || typeof item === 'boolean') + return item if (typeof item === 'number') { - if (!Number.isFinite(item)) throw new Error('JSON numbers must be finite.') + if (!Number.isFinite(item)) + throw new Error('JSON numbers must be finite.') return item } if (Array.isArray(item)) { @@ -1488,96 +1926,260 @@ function stableJson(value: unknown): string { return JSON.stringify(normalize(value)) } -function applyProjection(database: Database, event: PreparedEvent, sequence: number): void { - const payload = event.payload && typeof event.payload === 'object' - ? event.payload as Record - : {} - const canonical = event.metadata && typeof event.metadata === 'object' - && (event.metadata as Record).schemaVersion === 2 +function applyProjection( + database: Database, + event: PreparedEvent, + sequence: number, +): void { + const payload = + event.payload && typeof event.payload === 'object' + ? (event.payload as Record) + : {} + const canonical = + event.metadata && + typeof event.metadata === 'object' && + (event.metadata as Record).schemaVersion === 2 switch (event.eventType) { case 'task.created': - upsertProjection(database, 'memory_tasks', projectionId(payload.taskId), event, sequence, payload.taskId, { - ...payload, - status: payload.initialStatus, - }) + upsertProjection( + database, + 'memory_tasks', + projectionId(payload.taskId), + event, + sequence, + payload.taskId, + { + ...payload, + status: payload.initialStatus, + }, + ) return case 'task.transitioned': { const taskId = projectionId(payload.taskId) - upsertProjection(database, 'memory_tasks', taskId, event, sequence, payload.taskId, { - ...readProjectionState(database, 'memory_tasks', taskId), - ...payload, - status: payload.toStatus, - }) + upsertProjection( + database, + 'memory_tasks', + taskId, + event, + sequence, + payload.taskId, + { + ...readProjectionState(database, 'memory_tasks', taskId), + ...payload, + status: payload.toStatus, + }, + ) return } case 'session.started': case 'session.ended': { const sessionId = event.sessionId - upsertProjection(database, 'memory_sessions', sessionId, event, sequence, payload.taskId, { - ...readProjectionState(database, 'memory_sessions', sessionId), - ...payload, + upsertProjection( + database, + 'memory_sessions', sessionId, - status: event.eventType === 'session.ended' ? payload.status : 'active', - }) + event, + sequence, + payload.taskId, + { + ...readProjectionState(database, 'memory_sessions', sessionId), + ...payload, + sessionId, + status: + event.eventType === 'session.ended' ? payload.status : 'active', + }, + ) return } case 'artifact.classified': - upsertProjection(database, 'memory_artifacts', projectionId(payload.artifactId ?? nestedValue(payload.artifact, 'artifactId')), event, sequence, payload.taskId, payload) + upsertProjection( + database, + 'memory_artifacts', + projectionId( + payload.artifactId ?? nestedValue(payload.artifact, 'artifactId'), + ), + event, + sequence, + payload.taskId, + payload, + ) return case 'observation.recorded': { const observation = objectValue(payload.observation) const observationId = projectionId(observation.observationId) - upsertProjection(database, 'memory_discoveries', observationId, event, sequence, observation.taskId, observation) - upsertClaim(database, event, sequence, observationId, 'canonical', observation) + upsertProjection( + database, + 'memory_discoveries', + observationId, + event, + sequence, + observation.taskId, + observation, + ) + upsertClaim( + database, + event, + sequence, + observationId, + 'canonical', + observation, + ) return } case 'claim.consolidated': { for (const sourceId of stringArray(payload.sourceObservationIds)) { - markProjectionLifecycle(database, 'memory_discoveries', sourceId, event, sequence, 'superseded') - markProjectionLifecycle(database, 'memory_claims', sourceId, event, sequence, 'superseded') + markProjectionLifecycle( + database, + 'memory_discoveries', + sourceId, + event, + sequence, + 'superseded', + ) + markProjectionLifecycle( + database, + 'memory_claims', + sourceId, + event, + sequence, + 'superseded', + ) } const observation = objectValue(payload.canonicalObservation) const observationId = projectionId(observation.observationId) - upsertProjection(database, 'memory_discoveries', observationId, event, sequence, observation.taskId, observation) - upsertClaim(database, event, sequence, observationId, 'consolidated', observation) + upsertProjection( + database, + 'memory_discoveries', + observationId, + event, + sequence, + observation.taskId, + observation, + ) + upsertClaim( + database, + event, + sequence, + observationId, + 'consolidated', + observation, + ) return } case 'claim.corrected': - upsertClaim(database, event, sequence, projectionId(payload.observationId), 'corrected', payload) - upsertClaim(database, event, sequence, projectionId(nestedValue(payload.correction, 'observationId')), 'canonical', objectValue(payload.correction)) + upsertClaim( + database, + event, + sequence, + projectionId(payload.observationId), + 'corrected', + payload, + ) + upsertClaim( + database, + event, + sequence, + projectionId(nestedValue(payload.correction, 'observationId')), + 'canonical', + objectValue(payload.correction), + ) return case 'claim.superseded': - upsertClaim(database, event, sequence, projectionId(payload.observationId), 'superseded', payload) + upsertClaim( + database, + event, + sequence, + projectionId(payload.observationId), + 'superseded', + payload, + ) return case 'claim.forgotten': for (const observationId of stringArray(payload.observationIds)) { - upsertClaim(database, event, sequence, observationId, 'forgotten', payload) + upsertClaim( + database, + event, + sequence, + observationId, + 'forgotten', + payload, + ) } return case 'claim.pinned': - upsertClaim(database, event, sequence, projectionId(payload.observationId), 'pinned', payload) + upsertClaim( + database, + event, + sequence, + projectionId(payload.observationId), + 'pinned', + payload, + ) return case 'evidence.attached': for (const evidence of objectArray(payload.evidence)) { - upsertEvidence(database, event, sequence, projectionId(payload.observationId), evidence, 'attached') + upsertEvidence( + database, + event, + sequence, + projectionId(payload.observationId), + evidence, + 'attached', + ) } return case 'coverage.recorded': - upsertProjection(database, 'memory_discoveries', projectionId(payload.taskId + ':' + payload.dimension), event, sequence, payload.taskId, payload) + upsertProjection( + database, + 'memory_discoveries', + projectionId(payload.taskId + ':' + payload.dimension), + event, + sequence, + payload.taskId, + payload, + ) return case 'evidence.verified': - upsertEvidence(database, event, sequence, projectionId(payload.observationId), payload, 'verified') + upsertEvidence( + database, + event, + sequence, + projectionId(payload.observationId), + payload, + 'verified', + ) return case 'evidence.invalidated': - upsertEvidence(database, event, sequence, projectionId(payload.observationId), payload, 'invalidated') + upsertEvidence( + database, + event, + sequence, + projectionId(payload.observationId), + payload, + 'invalidated', + ) return case 'evidence.rebound': { const observationId = projectionId(payload.observationId) const previousSelector = objectValue(payload.previousSelector) const replacement = objectValue(payload.evidence) - upsertEvidence(database, event, sequence, observationId, { selector: previousSelector }, 'rebound') - upsertEvidence(database, event, sequence, observationId, replacement, 'attached') + upsertEvidence( + database, + event, + sequence, + observationId, + { selector: previousSelector }, + 'rebound', + ) + upsertEvidence( + database, + event, + sequence, + observationId, + replacement, + 'attached', + ) return } default: @@ -1588,19 +2190,46 @@ function applyProjection(database: Database, event: PreparedEvent, sequence: num const definition = PROJECTIONS[kind] if (!definition) return const explicit = definition.explicitId ? event[definition.explicitId] : null - const candidate = projectionId(explicit ?? payload[definition.payloadId] ?? payload.id) - upsertProjection(database, definition.table, candidate, event, sequence, event.taskId, payload) + const candidate = projectionId( + explicit ?? payload[definition.payloadId] ?? payload.id, + ) + upsertProjection( + database, + definition.table, + candidate, + event, + sequence, + event.taskId, + payload, + ) } -function readProjectionState(database: Database, table: string, entityId: string | null): Record { +function readProjectionState( + database: Database, + table: string, + entityId: string | null, +): Record { if (!entityId) return {} - const row = database.query(`SELECT state_json FROM ${table} WHERE entity_id = ?1`).get(entityId) as { state_json: string } | null + const row = database + .query(`SELECT state_json FROM ${table} WHERE entity_id = ?1`) + .get(entityId) as { state_json: string } | null return row ? objectValue(JSON.parse(row.state_json) as unknown) : {} } -function markProjectionLifecycle(database: Database, table: string, entityId: string, event: PreparedEvent, sequence: number, lifecycle: string): void { +function markProjectionLifecycle( + database: Database, + table: string, + entityId: string, + event: PreparedEvent, + sequence: number, + lifecycle: string, +): void { const prior = readProjectionState(database, table, entityId) - if (Object.keys(prior).length > 0) upsertProjection(database, table, entityId, event, sequence, prior.taskId, { ...prior, lifecycle }) + if (Object.keys(prior).length > 0) + upsertProjection(database, table, entityId, event, sequence, prior.taskId, { + ...prior, + lifecycle, + }) } function upsertClaim( @@ -1612,12 +2241,20 @@ function upsertClaim( value: Record, ): void { const prior = readProjectionState(database, 'memory_claims', observationId) - upsertProjection(database, 'memory_claims', observationId, event, sequence, value.taskId, { - ...prior, - lifecycle, + upsertProjection( + database, + 'memory_claims', observationId, - value: { ...objectValue(prior.value), ...value }, - }) + event, + sequence, + value.taskId, + { + ...prior, + lifecycle, + observationId, + value: { ...objectValue(prior.value), ...value }, + }, + ) } function upsertEvidence( @@ -1632,16 +2269,27 @@ function upsertEvidence( if (!observationId || Object.keys(selector).length === 0) return const entityId = `${observationId}:${stableJson(selector)}` const prior = readProjectionState(database, 'memory_evidence', entityId) - const evidence = Object.keys(objectValue(prior.evidence)).length > 0 - ? objectValue(prior.evidence) - : lifecycle === 'attached' ? value : {} - upsertProjection(database, 'memory_evidence', entityId, event, sequence, value.taskId, { - ...prior, - lifecycle, - observationId, - evidence, - freshness: lifecycle === 'attached' ? {} : value, - }) + const evidence = + Object.keys(objectValue(prior.evidence)).length > 0 + ? objectValue(prior.evidence) + : lifecycle === 'attached' + ? value + : {} + upsertProjection( + database, + 'memory_evidence', + entityId, + event, + sequence, + value.taskId, + { + ...prior, + lifecycle, + observationId, + evidence, + freshness: lifecycle === 'attached' ? {} : value, + }, + ) } function upsertProjection( @@ -1666,16 +2314,25 @@ function upsertProjection( source_sequence = excluded.source_sequence, updated_at = excluded.updated_at`, ) - .run(entityId, projectionId(taskId) ?? event.taskId, event.sessionId, stableJson(state), sequence, event.occurredAt) + .run( + entityId, + projectionId(taskId) ?? event.taskId, + event.sessionId, + stableJson(state), + sequence, + event.occurredAt, + ) } function projectionId(value: unknown): string | null { - return typeof value === 'string' && value.length > 0 && value.length <= 512 ? value : null + return typeof value === 'string' && value.length > 0 && value.length <= 512 + ? value + : null } function objectValue(value: unknown): Record { return value && typeof value === 'object' && !Array.isArray(value) - ? value as Record + ? (value as Record) : {} } @@ -1688,30 +2345,40 @@ function objectArray(value: unknown): Record[] { } function stringArray(value: unknown): string[] { - return Array.isArray(value) ? value.map(projectionId).filter((item): item is string => item !== null) : [] + return Array.isArray(value) + ? value.map(projectionId).filter((item): item is string => item !== null) + : [] } function setProjectionCursor(database: Database, sequence: number): void { database - .query("UPDATE memory_projection_metadata SET value = ?1 WHERE key = 'cursor'") + .query( + "UPDATE memory_projection_metadata SET value = ?1 WHERE key = 'cursor'", + ) .run(String(sequence)) } function isIdempotentMatch(row: EventRow, event: PreparedEvent): boolean { - return row.event_id === event.eventId - && row.idempotency_key === event.idempotencyKey - && row.event_type === event.eventType - && row.occurred_at === event.occurredAt - && row.payload_json === event.payloadJson - && row.metadata_json === event.metadataJson - && row.task_id === event.taskId - && row.session_id === event.sessionId - && row.artifact_id === event.artifactId + return ( + row.event_id === event.eventId && + row.idempotency_key === event.idempotencyKey && + row.event_type === event.eventType && + row.occurred_at === event.occurredAt && + row.payload_json === event.payloadJson && + row.metadata_json === event.metadataJson && + row.task_id === event.taskId && + row.session_id === event.sessionId && + row.artifact_id === event.artifactId + ) } function validateCanonicalProjectionRow(row: EventRow): void { const metadata = objectValue(JSON.parse(row.metadata_json) as unknown) - if (metadata.schemaVersion !== 2 || !CANONICAL_EVENT_TYPES.has(row.event_type)) return + if ( + metadata.schemaVersion !== 2 || + !CANONICAL_EVENT_TYPES.has(row.event_type) + ) + return try { envelopeFromRow(row) } catch { @@ -1784,20 +2451,23 @@ function storedEventFromRow(row: EventRow): MemoryV2StoredEvent { } } -function readProjectionRows(database: Database, table: string): ProjectionRow[] { +function readProjectionRows( + database: Database, + table: string, +): ProjectionRow[] { const rows = database .query( `SELECT entity_id, task_id, session_id, state_json, source_sequence, updated_at FROM ${table} ORDER BY entity_id`, ) .all() as Array<{ - entity_id: string - task_id: string | null - session_id: string | null - state_json: string - source_sequence: number - updated_at: string - }> + entity_id: string + task_id: string | null + session_id: string | null + state_json: string + source_sequence: number + updated_at: string + }> return rows.map((row) => ({ entityId: row.entity_id, taskId: row.task_id, @@ -1809,24 +2479,30 @@ function readProjectionRows(database: Database, table: string): ProjectionRow[] } function selectorKey(selector: MemorySelector): string { - if (selector.kind === 'uri-fragment') return `${selector.uri.toLowerCase()}#${selector.fragment.toLowerCase()}` + if (selector.kind === 'uri-fragment') + return `${selector.uri.toLowerCase()}#${selector.fragment.toLowerCase()}` const base = selector.path.replaceAll('\\', '/').toLowerCase() - if (selector.kind === 'symbol') return `${base}#${selector.symbol.toLowerCase()}` - if (selector.kind === 'json-pointer') return `${base}#${selector.pointer.toLowerCase()}` - if (selector.kind === 'line-range') return `${base}:${selector.startLine}-${selector.endLine}` + if (selector.kind === 'symbol') + return `${base}#${selector.symbol.toLowerCase()}` + if (selector.kind === 'json-pointer') + return `${base}#${selector.pointer.toLowerCase()}` + if (selector.kind === 'line-range') + return `${base}:${selector.startLine}-${selector.endLine}` return base } function scanQueryRows(database: Database, projectId: string): QueryScanResult { - const admission = database.query( - `SELECT sequence, + const admission = database + .query( + `SELECT sequence, length(CAST(payload_json AS BLOB)) AS payload_bytes, length(CAST(metadata_json AS BLOB)) AS metadata_bytes FROM memory_events WHERE json_extract(metadata_json, '$.projectId') = ?1 ORDER BY sequence DESC LIMIT ?2`, - ).all(projectId, MAX_QUERY_EVENTS + 1) as QueryAdmissionRow[] + ) + .all(projectId, MAX_QUERY_EVENTS + 1) as QueryAdmissionRow[] const sequences: number[] = [] let bytes = 0 let payloadBudgetReached = false @@ -1840,23 +2516,43 @@ function scanQueryRows(database: Database, projectId: string): QueryScanResult { bytes += rowBytes } if (sequences.length === 0) { - return { rows: [], eventCapReached: admission.length > MAX_QUERY_EVENTS, payloadBudgetReached } + return { + rows: [], + eventCapReached: admission.length > MAX_QUERY_EVENTS, + payloadBudgetReached, + } } - const rows = database.query( - `SELECT sequence, event_id, idempotency_key, event_type, occurred_at, + const rows = database + .query( + `SELECT sequence, event_id, idempotency_key, event_type, occurred_at, payload_json, metadata_json, task_id, session_id, artifact_id FROM memory_events WHERE sequence IN (${sequences.map(() => '?').join(',')}) ORDER BY sequence DESC`, - ).all(...sequences) as EventRow[] - return { rows, eventCapReached: admission.length > MAX_QUERY_EVENTS, payloadBudgetReached } + ) + .all(...sequences) as EventRow[] + return { + rows, + eventCapReached: admission.length > MAX_QUERY_EVENTS, + payloadBudgetReached, + } } function lexicalTokens(value: string): Set { - return new Set(value.toLowerCase().split(/[^a-z0-9._:/-]+/).filter((token) => token.length > 1).slice(0, 128)) + return new Set( + value + .toLowerCase() + .split(/[^a-z0-9._:/-]+/) + .filter((token) => token.length > 1) + .slice(0, 128), + ) } -function reason(code: RankingReason['code'], contribution: number, detail: string): RankingReason { +function reason( + code: RankingReason['code'], + contribution: number, + detail: string, +): RankingReason { return { code, contribution, detail } } @@ -1874,16 +2570,32 @@ function buildLexicalResult( superseded: boolean corrected: boolean pinned: boolean - freshness: Map + freshness: Map< + string, + { + state: 'verified' | 'invalid' + at: string + reason?: string + observedDigest?: string + workspaceRevision?: number + workspaceSnapshotId?: string + } + > + } + type TaskState = { + taskId: string + title: string + objective: string + status: + | 'created' + | 'active' + | 'blocked' + | 'completed' + | 'failed' + | 'cancelled' + sourceSequence: number + eventId: string } - type TaskState = { taskId: string; title: string; objective: string; status: 'created' | 'active' | 'blocked' | 'completed' | 'failed' | 'cancelled'; sourceSequence: number; eventId: string } const observations = new Map() const tasks = new Map() @@ -1899,7 +2611,13 @@ function buildLexicalResult( }) } else if (event.eventType === 'task.transitioned') { const task = tasks.get(event.payload.taskId) - if (task) tasks.set(task.taskId, { ...task, status: event.payload.toStatus, sourceSequence: event.sequence, eventId: event.eventId }) + if (task) + tasks.set(task.taskId, { + ...task, + status: event.payload.toStatus, + sourceSequence: event.sequence, + eventId: event.eventId, + }) } else if (event.eventType === 'observation.recorded') { observations.set(event.payload.observation.observationId, { observation: event.payload.observation, @@ -1913,7 +2631,14 @@ function buildLexicalResult( }) } else if (event.eventType === 'evidence.attached') { const state = observations.get(event.payload.observationId) - if (state) state.observation = { ...state.observation, evidence: [...state.observation.evidence, ...event.payload.evidence].slice(0, 32) } + if (state) + state.observation = { + ...state.observation, + evidence: [ + ...state.observation.evidence, + ...event.payload.evidence, + ].slice(0, 32), + } } else if (event.eventType === 'claim.consolidated') { for (const id of event.payload.sourceObservationIds) { const source = observations.get(id) @@ -1964,150 +2689,452 @@ function buildLexicalResult( }) } else if (event.eventType === 'evidence.invalidated') { const state = observations.get(event.payload.observationId) - state?.freshness.set(selectorKey(event.payload.selector), { state: 'invalid', at: event.payload.invalidatedAt, reason: event.payload.reason }) + state?.freshness.set(selectorKey(event.payload.selector), { + state: 'invalid', + at: event.payload.invalidatedAt, + reason: event.payload.reason, + }) } else if (event.eventType === 'evidence.rebound') { const state = observations.get(event.payload.observationId) if (state) { - state.observation = { ...state.observation, evidence: [...state.observation.evidence.filter((e) => selectorKey(e.selector) !== selectorKey(event.payload.previousSelector)), event.payload.evidence] } + state.observation = { + ...state.observation, + evidence: [ + ...state.observation.evidence.filter( + (e) => + selectorKey(e.selector) !== + selectorKey(event.payload.previousSelector), + ), + event.payload.evidence, + ], + } state.freshness.delete(selectorKey(event.payload.previousSelector)) } } } - const queryTokens = lexicalTokens([request.query, ...request.selectors.map(selectorKey)].join(' ')) + const queryTokens = lexicalTokens( + [request.query, ...request.selectors.map(selectorKey)].join(' '), + ) const requestedSelectors = new Set(request.selectors.map(selectorKey)) - const contextMatches = (evidence: MemoryObservation['evidence'][number], freshness: ObservationState['freshness'] extends Map ? F : never) => { + const contextMatches = ( + evidence: MemoryObservation['evidence'][number], + freshness: ObservationState['freshness'] extends Map + ? F + : never, + ) => { if (freshness.state !== 'verified') return false - if ('path' in evidence.selector && freshness.observedDigest !== evidence.contentDigest) return false - const requestHasContext = request.workspaceRevision !== undefined || request.workspaceSnapshotId !== undefined - const verificationHasContext = freshness.workspaceRevision !== undefined || freshness.workspaceSnapshotId !== undefined + if ( + 'path' in evidence.selector && + freshness.observedDigest !== evidence.contentDigest + ) + return false + const requestHasContext = + request.workspaceRevision !== undefined || + request.workspaceSnapshotId !== undefined + const verificationHasContext = + freshness.workspaceRevision !== undefined || + freshness.workspaceSnapshotId !== undefined if (requestHasContext || verificationHasContext) { - return request.workspaceRevision === freshness.workspaceRevision - && request.workspaceSnapshotId === freshness.workspaceSnapshotId + return ( + request.workspaceRevision === freshness.workspaceRevision && + request.workspaceSnapshotId === freshness.workspaceSnapshotId + ) } return true } - const rank = (text: string, taskId: string | undefined, selectors: MemorySelector[], verified: boolean, pinned: boolean, sourceSequence: number) => { + const rank = ( + text: string, + taskId: string | undefined, + selectors: MemorySelector[], + verified: boolean, + pinned: boolean, + sourceSequence: number, + ) => { const candidateTokens = lexicalTokens(text) let tokenMatches = 0 - for (const token of queryTokens) if (candidateTokens.has(token)) tokenMatches++ - const selectorMatch = selectors.some((selector) => requestedSelectors.has(selectorKey(selector))) + for (const token of queryTokens) + if (candidateTokens.has(token)) tokenMatches++ + const selectorMatch = selectors.some((selector) => + requestedSelectors.has(selectorKey(selector)), + ) const taskMatch = request.taskId !== undefined && request.taskId === taskId const reasons: RankingReason[] = [] - if (taskMatch) reasons.push(reason('task-match', 0.25, 'The task ID exactly matches the requested task.')) - if (selectorMatch) reasons.push(reason('selector-match', 0.25, 'A selector exactly matches the request.')) - if (tokenMatches > 0) reasons.push(reason('semantic-match', Math.min(0.3, tokenMatches * 0.06), `${tokenMatches} lexical token match(es).`)) - if (verified) reasons.push(reason('verified-evidence', 0.15, 'The observation has current verified evidence.')) - if (pinned) reasons.push(reason('reusability', 0.04, 'The observation is pinned.')) - reasons.push(reason('recency', 0.01, `Canonical source sequence ${sourceSequence}.`)) - return { score: Math.min(1, reasons.reduce((sum, item) => sum + Math.max(0, item.contribution), 0)), reasons, exact: Number(taskMatch) + Number(selectorMatch), tokenMatches, verified: Number(verified), pinned: Number(pinned), sourceSequence } - } - const compare = ; id: string }>(left: T, right: T) => - right.ranking.exact - left.ranking.exact - || right.ranking.tokenMatches - left.ranking.tokenMatches - || right.ranking.verified - left.ranking.verified - || right.ranking.pinned - left.ranking.pinned - || right.ranking.sourceSequence - left.ranking.sourceSequence - || compareUnicodeCodePoints(left.id, right.id) + if (taskMatch) + reasons.push( + reason( + 'task-match', + 0.25, + 'The task ID exactly matches the requested task.', + ), + ) + if (selectorMatch) + reasons.push( + reason( + 'selector-match', + 0.25, + 'A selector exactly matches the request.', + ), + ) + if (tokenMatches > 0) + reasons.push( + reason( + 'semantic-match', + Math.min(0.3, tokenMatches * 0.06), + `${tokenMatches} lexical token match(es).`, + ), + ) + if (verified) + reasons.push( + reason( + 'verified-evidence', + 0.15, + 'The observation has current verified evidence.', + ), + ) + if (pinned) + reasons.push(reason('reusability', 0.04, 'The observation is pinned.')) + reasons.push( + reason('recency', 0.01, `Canonical source sequence ${sourceSequence}.`), + ) + return { + score: Math.min( + 1, + reasons.reduce((sum, item) => sum + Math.max(0, item.contribution), 0), + ), + reasons, + exact: Number(taskMatch) + Number(selectorMatch), + tokenMatches, + verified: Number(verified), + pinned: Number(pinned), + sourceSequence, + } + } + const compare = ; id: string }>( + left: T, + right: T, + ) => + right.ranking.exact - left.ranking.exact || + right.ranking.tokenMatches - left.ranking.tokenMatches || + right.ranking.verified - left.ranking.verified || + right.ranking.pinned - left.ranking.pinned || + right.ranking.sourceSequence - left.ranking.sourceSequence || + compareUnicodeCodePoints(left.id, right.id) const taskCandidates = [...tasks.values()] .filter((task) => !request.taskId || task.taskId === request.taskId) - .map((task) => ({ id: task.taskId, task, ranking: rank(`${task.taskId} ${task.title} ${task.objective}`, task.taskId, [], false, false, task.sourceSequence) })) + .map((task) => ({ + id: task.taskId, + task, + ranking: rank( + `${task.taskId} ${task.title} ${task.objective}`, + task.taskId, + [], + false, + false, + task.sourceSequence, + ), + })) .filter((item) => item.ranking.tokenMatches > 0 || item.ranking.exact > 0) .sort(compare) const observationCandidates = [...observations.values()] .map((state) => { - const selectors = state.observation.selectors ?? state.observation.evidence.map(({ selector }) => selector) + const selectors = + state.observation.selectors ?? + state.observation.evidence.map(({ selector }) => selector) const verified = state.observation.evidence.some((evidence) => { const freshness = state.freshness.get(selectorKey(evidence.selector)) return freshness !== undefined && contextMatches(evidence, freshness) }) - return { id: state.observation.observationId, state, selectors, ranking: rank(`${state.observation.taskId} ${state.observation.kind} ${state.observation.summary} ${state.observation.detail} ${selectors.map(selectorKey).join(' ')}`, state.observation.taskId, selectors, verified, state.pinned, state.sourceSequence) } + return { + id: state.observation.observationId, + state, + selectors, + ranking: rank( + `${state.observation.taskId} ${state.observation.kind} ${state.observation.summary} ${state.observation.detail} ${selectors.map(selectorKey).join(' ')}`, + state.observation.taskId, + selectors, + verified, + state.pinned, + state.sourceSequence, + ), + } }) - .filter((item) => (!request.taskId || item.state.observation.taskId === request.taskId) - && (request.artifactKinds.length === 0 || item.state.observation.evidence.some(({ artifact }) => request.artifactKinds.includes(artifact.classification.kind))) - && (item.ranking.tokenMatches > 0 || item.ranking.exact > 0 || item.state.pinned)) + .filter( + (item) => + (!request.taskId || item.state.observation.taskId === request.taskId) && + (request.artifactKinds.length === 0 || + item.state.observation.evidence.some(({ artifact }) => + request.artifactKinds.includes(artifact.classification.kind), + )) && + (item.ranking.tokenMatches > 0 || + item.ranking.exact > 0 || + item.state.pinned), + ) .sort(compare) const limit = request.maxResultsPerCategory - const verifiedCandidates = observationCandidates.filter(({ state }) => !state.forgotten && !state.superseded && !state.corrected && state.observation.evidence.some((evidence) => { - const freshness = state.freshness.get(selectorKey(evidence.selector)) - return freshness !== undefined && contextMatches(evidence, freshness) - })) - const reusableCandidates = observationCandidates.filter(({ state }) => !state.forgotten && !state.superseded && !state.corrected && state.observation.kind === 'discovery') - const rereadCandidates = observationCandidates.filter(({ state }) => !state.forgotten && !state.superseded && !state.corrected).flatMap(({ state, selectors, ranking }) => selectors.map((selector) => { - const freshness = state.freshness.get(selectorKey(selector)) - const evidence = state.observation.evidence.find((item) => selectorKey(item.selector) === selectorKey(selector)) - if (freshness && evidence && contextMatches(evidence, freshness)) return null - const rereadReason = freshness?.reason === 'missing' ? 'missing' : freshness?.reason === 'expired' ? 'expired' : freshness?.state === 'invalid' ? 'changed' : 'never-verified' - return { observationId: state.observation.observationId, selector, reason: rereadReason as 'never-verified' | 'changed' | 'missing' | 'expired', detail: freshness?.state === 'invalid' ? 'The latest evidence state is invalidated; reread before use.' : 'This selector requires verification in the current workspace context.', score: ranking.score, reasons: [...ranking.reasons, reason('stale-evidence', -0.2, 'Current evidence is unavailable.')].slice(0, 16) } - }).filter((item): item is NonNullable => item !== null)) + const verifiedCandidates = observationCandidates.filter( + ({ state }) => + !state.forgotten && + !state.superseded && + !state.corrected && + state.observation.evidence.some((evidence) => { + const freshness = state.freshness.get(selectorKey(evidence.selector)) + return freshness !== undefined && contextMatches(evidence, freshness) + }), + ) + const reusableCandidates = observationCandidates.filter( + ({ state }) => + !state.forgotten && + !state.superseded && + !state.corrected && + state.observation.kind === 'discovery', + ) + const rereadCandidates = observationCandidates + .filter( + ({ state }) => !state.forgotten && !state.superseded && !state.corrected, + ) + .flatMap(({ state, selectors, ranking }) => + selectors + .map((selector) => { + const freshness = state.freshness.get(selectorKey(selector)) + const evidence = state.observation.evidence.find( + (item) => selectorKey(item.selector) === selectorKey(selector), + ) + if (freshness && evidence && contextMatches(evidence, freshness)) + return null + const rereadReason = + freshness?.reason === 'missing' + ? 'missing' + : freshness?.reason === 'expired' + ? 'expired' + : freshness?.state === 'invalid' + ? 'changed' + : 'never-verified' + return { + observationId: state.observation.observationId, + selector, + reason: rereadReason as + | 'never-verified' + | 'changed' + | 'missing' + | 'expired', + detail: + freshness?.state === 'invalid' + ? 'The latest evidence state is invalidated; reread before use.' + : 'This selector requires verification in the current workspace context.', + score: ranking.score, + reasons: [ + ...ranking.reasons, + reason( + 'stale-evidence', + -0.2, + 'Current evidence is unavailable.', + ), + ].slice(0, 16), + } + }) + .filter((item): item is NonNullable => item !== null), + ) const historicalCandidates = request.includeHistorical - ? observationCandidates.filter(({ state }) => state.forgotten || state.superseded || state.corrected) + ? observationCandidates.filter( + ({ state }) => state.forgotten || state.superseded || state.corrected, + ) : [] - const matchedTasks = taskCandidates.slice(0, limit).map(({ task, ranking }) => ({ taskId: task.taskId, title: task.title, status: task.status, summary: task.objective, score: ranking.score, reasons: ranking.reasons })) - const verifiedKnowledge = verifiedCandidates.slice(0, limit).map(({ state, ranking }) => { - const verifiedSelectors = [...state.freshness].filter(([key, freshness]) => { - const evidence = state.observation.evidence.find((item) => selectorKey(item.selector) === key) - return evidence !== undefined && contextMatches(evidence, freshness) + const matchedTasks = taskCandidates + .slice(0, limit) + .map(({ task, ranking }) => ({ + taskId: task.taskId, + title: task.title, + status: task.status, + summary: task.objective, + score: ranking.score, + reasons: ranking.reasons, + })) + const verifiedKnowledge = verifiedCandidates + .slice(0, limit) + .map(({ state, ranking }) => { + const verifiedSelectors = [...state.freshness].filter( + ([key, freshness]) => { + const evidence = state.observation.evidence.find( + (item) => selectorKey(item.selector) === key, + ) + return evidence !== undefined && contextMatches(evidence, freshness) + }, + ) + const verifiedEvidence = verifiedSelectors + .map(([key]) => + state.observation.evidence.find( + (evidence) => selectorKey(evidence.selector) === key, + ), + ) + .filter( + (evidence): evidence is MemoryObservation['evidence'][number] => + evidence !== undefined, + ) + const verifiedAt = verifiedSelectors + .map(([, freshness]) => freshness.at) + .sort(compareUnicodeCodePoints) + .at(-1)! + return { + observation: state.observation, + verifiedEvidence, + verifiedAt, + score: ranking.score, + reasons: ranking.reasons, + } }) - const verifiedEvidence = verifiedSelectors.map(([key]) => state.observation.evidence.find((evidence) => selectorKey(evidence.selector) === key)).filter((evidence): evidence is MemoryObservation['evidence'][number] => evidence !== undefined) - const verifiedAt = verifiedSelectors.map(([, freshness]) => freshness.at).sort(compareUnicodeCodePoints).at(-1)! - return { observation: state.observation, verifiedEvidence, verifiedAt, score: ranking.score, reasons: ranking.reasons } - }).filter(({ verifiedEvidence }) => verifiedEvidence.length > 0) - const reusableDiscovery = reusableCandidates.slice(0, limit).map(({ state, ranking }) => ({ observation: state.observation, reuseGuidance: state.pinned ? 'Pinned discovery; verify selectors before reuse.' : 'Verify selectors before reusing this discovery.', score: ranking.score, reasons: ranking.reasons })) + .filter(({ verifiedEvidence }) => verifiedEvidence.length > 0) + const reusableDiscovery = reusableCandidates + .slice(0, limit) + .map(({ state, ranking }) => ({ + observation: state.observation, + reuseGuidance: state.pinned + ? 'Pinned discovery; verify selectors before reuse.' + : 'Verify selectors before reusing this discovery.', + score: ranking.score, + reasons: ranking.reasons, + })) const rereadRequired = rereadCandidates.slice(0, limit) - const historicalContext = historicalCandidates.slice(0, limit).map(({ state, ranking }) => ({ taskId: state.observation.taskId, summary: state.observation.summary, eventIds: [state.sourceEventId], score: ranking.score, reasons: [...ranking.reasons, reason('historical-only', -0.1, 'This observation is historical only.')].slice(0, 16) })) - const latestCoverageByKey = new Map['payload']; sequence: number }>() + const historicalContext = historicalCandidates + .slice(0, limit) + .map(({ state, ranking }) => ({ + taskId: state.observation.taskId, + summary: state.observation.summary, + eventIds: [state.sourceEventId], + score: ranking.score, + reasons: [ + ...ranking.reasons, + reason('historical-only', -0.1, 'This observation is historical only.'), + ].slice(0, 16), + })) + const latestCoverageByKey = new Map< + string, + { + payload: Extract< + MemoryEventEnvelope, + { eventType: 'coverage.recorded' } + >['payload'] + sequence: number + } + >() for (const event of events) { if (event.eventType !== 'coverage.recorded') continue const key = `${event.payload.taskId}:${event.payload.dimension}` const prior = latestCoverageByKey.get(key) if (!prior || event.sequence > prior.sequence) { - latestCoverageByKey.set(key, { payload: event.payload, sequence: event.sequence }) + latestCoverageByKey.set(key, { + payload: event.payload, + sequence: event.sequence, + }) } } const currentCoverage = [...latestCoverageByKey.entries()] .filter(([, entry]) => { - if (request.taskId !== undefined && entry.payload.taskId !== request.taskId) return false - const requestHasContext = request.workspaceRevision !== undefined || request.workspaceSnapshotId !== undefined - const payloadHasContext = entry.payload.workspaceRevision !== undefined || entry.payload.workspaceSnapshotId !== undefined + if ( + request.taskId !== undefined && + entry.payload.taskId !== request.taskId + ) + return false + const requestHasContext = + request.workspaceRevision !== undefined || + request.workspaceSnapshotId !== undefined + const payloadHasContext = + entry.payload.workspaceRevision !== undefined || + entry.payload.workspaceSnapshotId !== undefined if (requestHasContext || payloadHasContext) { - return request.workspaceRevision === entry.payload.workspaceRevision - && request.workspaceSnapshotId === entry.payload.workspaceSnapshotId + return ( + request.workspaceRevision === entry.payload.workspaceRevision && + request.workspaceSnapshotId === entry.payload.workspaceSnapshotId + ) } return true }) - .sort(([leftKey], [rightKey]) => compareUnicodeCodePoints(leftKey, rightKey)) + .sort(([leftKey], [rightKey]) => + compareUnicodeCodePoints(leftKey, rightKey), + ) .slice(0, 5) .map(([, entry]) => { - const item: { dimension: typeof entry.payload.dimension; state: typeof entry.payload.state; taskId: typeof entry.payload.taskId; notes?: string; workspaceRevision?: number; workspaceSnapshotId?: string } = { + const item: { + dimension: typeof entry.payload.dimension + state: typeof entry.payload.state + taskId: typeof entry.payload.taskId + notes?: string + workspaceRevision?: number + workspaceSnapshotId?: string + } = { dimension: entry.payload.dimension, state: entry.payload.state, taskId: entry.payload.taskId, notes: entry.payload.notes.slice(0, 1024), } - if (entry.payload.workspaceRevision !== undefined) item.workspaceRevision = entry.payload.workspaceRevision - if (entry.payload.workspaceSnapshotId !== undefined) item.workspaceSnapshotId = entry.payload.workspaceSnapshotId + if (entry.payload.workspaceRevision !== undefined) + item.workspaceRevision = entry.payload.workspaceRevision + if (entry.payload.workspaceSnapshotId !== undefined) + item.workspaceSnapshotId = entry.payload.workspaceSnapshotId return item }) - const categories = { matchedTasks, verifiedKnowledge, reusableDiscovery, rereadRequired, historicalContext } + const categories = { + matchedTasks, + verifiedKnowledge, + reusableDiscovery, + rereadRequired, + historicalContext, + } const rankingReasons = [ - ...matchedTasks.map((value) => ({ category: 'matchedTasks' as const, targetId: value.taskId, reasons: value.reasons })), - ...verifiedKnowledge.map((value) => ({ category: 'verifiedKnowledge' as const, targetId: value.observation.observationId, reasons: value.reasons })), - ...reusableDiscovery.map((value) => ({ category: 'reusableDiscovery' as const, targetId: value.observation.observationId, reasons: value.reasons })), - ...rereadRequired.map((value) => ({ category: 'rereadRequired' as const, targetId: value.observationId, reasons: value.reasons })), - ...historicalContext.map((value) => ({ category: 'historicalContext' as const, targetId: value.taskId ?? value.eventIds[0]!, reasons: value.reasons })), + ...matchedTasks.map((value) => ({ + category: 'matchedTasks' as const, + targetId: value.taskId, + reasons: value.reasons, + })), + ...verifiedKnowledge.map((value) => ({ + category: 'verifiedKnowledge' as const, + targetId: value.observation.observationId, + reasons: value.reasons, + })), + ...reusableDiscovery.map((value) => ({ + category: 'reusableDiscovery' as const, + targetId: value.observation.observationId, + reasons: value.reasons, + })), + ...rereadRequired.map((value) => ({ + category: 'rereadRequired' as const, + targetId: value.observationId, + reasons: value.reasons, + })), + ...historicalContext.map((value) => ({ + category: 'historicalContext' as const, + targetId: value.taskId ?? value.eventIds[0]!, + reasons: value.reasons, + })), ] - const resultCapReached = eventCapReached - || taskCandidates.length > limit - || verifiedCandidates.length > limit - || reusableCandidates.length > limit - || rereadCandidates.length > limit - || historicalCandidates.length > limit + const resultCapReached = + eventCapReached || + taskCandidates.length > limit || + verifiedCandidates.length > limit || + reusableCandidates.length > limit || + rereadCandidates.length > limit || + historicalCandidates.length > limit const degradationReasons = [ - ...(resultCapReached ? [{ code: 'result-cap-reached' as const, detail: 'The deterministic result or event cap was reached.', retryable: false }] : []), - ...(payloadBudgetReached ? [{ code: 'resource-budget' as const, detail: 'The deterministic query resource budget was reached.', retryable: false }] : []), + ...(resultCapReached + ? [ + { + code: 'result-cap-reached' as const, + detail: 'The deterministic result or event cap was reached.', + retryable: false, + }, + ] + : []), + ...(payloadBudgetReached + ? [ + { + code: 'resource-budget' as const, + detail: 'The deterministic query resource budget was reached.', + retryable: false, + }, + ] + : []), ] return MemoryRetrievalResultSchema.parse({ schemaVersion: 2, @@ -2116,14 +3143,20 @@ function buildLexicalResult( generatedAt: events.at(-1)?.occurredAt ?? '1970-01-01T00:00:00.000Z', ...categories, currentCoverage, - degradation: degradationReasons.length > 0 ? { state: 'degraded', reasons: degradationReasons } : { state: 'none' }, + degradation: + degradationReasons.length > 0 + ? { state: 'degraded', reasons: degradationReasons } + : { state: 'none' }, rankingReasons: rankingReasons.slice(0, limit * 5), }) } function compareUnicodeCodePoints(left: string, right: string): number { const leftPoints = Array.from(left, (character) => character.codePointAt(0)!) - const rightPoints = Array.from(right, (character) => character.codePointAt(0)!) + const rightPoints = Array.from( + right, + (character) => character.codePointAt(0)!, + ) const length = Math.min(leftPoints.length, rightPoints.length) for (let index = 0; index < length; index++) { const difference = leftPoints[index]! - rightPoints[index]! @@ -2134,48 +3167,80 @@ function compareUnicodeCodePoints(left: string, right: string): number { function readCapabilities(database: Database): MemoryV2Capability[] { const rows = database - .query('SELECT name, available, fallback, value FROM memory_store_capabilities ORDER BY name') - .all() as Array<{ name: string; available: number; fallback: string | null; value: string }> + .query( + 'SELECT name, available, fallback, value FROM memory_store_capabilities ORDER BY name', + ) + .all() as Array<{ + name: string + available: number + fallback: string | null + value: string + }> return rows.map((row) => ({ ...row, available: row.available === 1 })) } -function readPragmaString(database: Database, sql: string, key: string): string { +function readPragmaString( + database: Database, + sql: string, + key: string, +): string { const row = database.query(sql).get() as Record | null const value = row?.[key] ?? (row ? Object.values(row)[0] : undefined) - if (typeof value !== 'string') throw new Error('Unexpected SQLite pragma response.') + if (typeof value !== 'string') + throw new Error('Unexpected SQLite pragma response.') return value } -function readPragmaNumber(database: Database, sql: string, key: string): number { +function readPragmaNumber( + database: Database, + sql: string, + key: string, +): number { const row = database.query(sql).get() as Record | null const value = row?.[key] ?? (row ? Object.values(row)[0] : undefined) - if (typeof value !== 'number') throw new Error('Unexpected SQLite pragma response.') + if (typeof value !== 'number') + throw new Error('Unexpected SQLite pragma response.') return value } function requiredString(value: unknown, name: string, max: number): string { - if (typeof value !== 'string' || value.length === 0 || value.length > max || value.includes('\0')) { + if ( + typeof value !== 'string' || + value.length === 0 || + value.length > max || + value.includes('\0') + ) { throw new Error(`${name} must be a non-empty bounded string.`) } return value } -function optionalString(value: unknown, name: string, max: number): string | null { +function optionalString( + value: unknown, + name: string, + max: number, +): string | null { if (value === undefined || value === null) return null return requiredString(value, name, max) } function validNonNegativeInteger(value: number): number { - if (!Number.isSafeInteger(value) || value < 0) throw new MemoryV2StorageError({ - kind: 'invalid', message: 'The event cursor must be a non-negative integer.', retryable: false, - }) + if (!Number.isSafeInteger(value) || value < 0) + throw new MemoryV2StorageError({ + kind: 'invalid', + message: 'The event cursor must be a non-negative integer.', + retryable: false, + }) return value } function validPositiveInteger(value: number): number { - if (!Number.isSafeInteger(value) || value < 1) throw new MemoryV2StorageError({ - kind: 'invalid', message: 'The event limit must be a positive integer.', retryable: false, - }) + if (!Number.isSafeInteger(value) || value < 1) + throw new MemoryV2StorageError({ + kind: 'invalid', + message: 'The event limit must be a positive integer.', + retryable: false, + }) return value } @@ -2202,11 +3267,15 @@ function failedOutcome( return { outcome: 'failed', error: operationError(code, message, retryable) } } -function operationFailureOutcome( - failure: MemoryV2Failure, -): { outcome: 'failed'; error: MemoryOperationError } { +function operationFailureOutcome(failure: MemoryV2Failure): { + outcome: 'failed' + error: MemoryOperationError +} { return failedOutcome( - failure.kind === 'busy' || failure.kind === 'closed' || failure.kind === 'io' + failure.kind === 'busy' || + failure.kind === 'closed' || + failure.kind === 'io' || + failure.kind === 'unsupported-open' ? 'unavailable' : 'internal', failure.message, @@ -2233,14 +3302,30 @@ function commonHealth( schemaVersion: 2, status, checkedAt: new Date().toISOString(), - authority: status === 'unavailable' - ? { kind: 'unavailable', writable: false, reason: issues[0] ?? 'The memory store is unavailable.' } - : { kind: 'authoritative', writable: true, source: 'Bun SQLite Memory V2' }, + authority: + status === 'unavailable' + ? { + kind: 'unavailable', + writable: false, + reason: issues[0] ?? 'The memory store is unavailable.', + } + : { + kind: 'authoritative', + writable: true, + source: 'Bun SQLite Memory V2', + }, backend: { backendId: 'bun-sqlite-memory-v2', kind: 'local-persistent', persistence: 'durable', - capabilities: ['append', 'query', 'verify', 'rebuild', 'health', 'export'], + capabilities: [ + 'append', + 'query', + 'verify', + 'rebuild', + 'health', + 'export', + ], }, issues, } @@ -2256,26 +3341,59 @@ function invalidEventFailure(error: unknown): MemoryV2Failure { function classifyStorageError(error: unknown): MemoryV2Failure { if (error instanceof MemoryV2StorageError) return error.failure - const code = typeof error === 'object' && error !== null && 'code' in error - ? String((error as { code: unknown }).code).toUpperCase() - : '' + const code = + typeof error === 'object' && error !== null && 'code' in error + ? String((error as { code: unknown }).code).toUpperCase() + : '' const message = error instanceof Error ? error.message.toUpperCase() : '' const signature = `${code} ${message}` - if (signature.includes('SQLITE_BUSY') || signature.includes('SQLITE_LOCKED') - || signature.includes('DATABASE IS LOCKED') || code === '5' || code === '6') { - return { kind: 'busy', message: 'The memory store is busy; retry shortly.', retryable: true } + if ( + signature.includes('SQLITE_BUSY') || + signature.includes('SQLITE_LOCKED') || + signature.includes('DATABASE IS LOCKED') || + code === '5' || + code === '6' + ) { + return { + kind: 'busy', + message: 'The memory store is busy; retry shortly.', + retryable: true, + } } - if (signature.includes('SQLITE_CORRUPT') || signature.includes('SQLITE_NOTADB') || signature.includes('NOT A DATABASE') || signature.includes('MALFORMED')) { - return { kind: 'corrupt', message: 'The memory store is corrupt or unreadable.', retryable: false } + if ( + signature.includes('SQLITE_CORRUPT') || + signature.includes('SQLITE_NOTADB') || + signature.includes('NOT A DATABASE') || + signature.includes('MALFORMED') + ) { + return { + kind: 'corrupt', + message: 'The memory store is corrupt or unreadable.', + retryable: false, + } } - if (signature.includes('SQLITE_SCHEMA') || signature.includes('SQLITE_MISMATCH')) { - return { kind: 'incompatible', message: 'The memory store schema is incompatible with this CLI.', retryable: false } + if ( + signature.includes('SQLITE_SCHEMA') || + signature.includes('SQLITE_MISMATCH') + ) { + return { + kind: 'incompatible', + message: 'The memory store schema is incompatible with this CLI.', + retryable: false, + } + } + return { + kind: 'io', + message: 'The memory store could not complete a local I/O operation.', + retryable: false, } - return { kind: 'io', message: 'The memory store could not complete a local I/O operation.', retryable: false } } -function unavailableHealth(failure: MemoryV2Failure, schemaVersion: number | null = null): MemoryV2Health { +function unavailableHealth( + failure: MemoryV2Failure, + schemaVersion: number | null = null, +): MemoryV2Health { return { status: 'unavailable', schemaVersion, From 9c13f250d5bbce41cad590ce18d32351cc1d48e3 Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Mon, 14 Sep 2026 00:18:17 +0300 Subject: [PATCH 10/22] chore: updating the relevant docs --- .../sessions/dynamic-cross-session-memory/STATUS.md | 12 +++++++++--- cli/knowledge.md | 2 +- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/.agents/sessions/dynamic-cross-session-memory/STATUS.md b/.agents/sessions/dynamic-cross-session-memory/STATUS.md index 54e5956b11..ddceddfa13 100644 --- a/.agents/sessions/dynamic-cross-session-memory/STATUS.md +++ b/.agents/sessions/dynamic-cross-session-memory/STATUS.md @@ -1,8 +1,8 @@ # STATUS — Dynamic Cross-Session Memory V2 Repair -Status: implementation largely complete; final gate is MEM2-R1-T2 (race-resistant SQLite open). -Current phase: R1 — SQLite kernel / storage security. -Current task: MEM2-R1-T2 Resolve race-resistant SQLite open support (typed-unsupported fallback; full closure needs native-addon authorization). +Status: implementation complete; MEM2-R1-T2 resolved (typed-unsupported secure-open fallback) and validated green. +Current phase: R7 — integration / finalization. +Current task: MEM2-R7 — obtain stable exact-snapshot reviews; full race-free WAL/SHM open deferred pending native-addon authorization (option C). ## Implemented and locally validated before this plan refresh @@ -167,3 +167,9 @@ Addressed the gate advisory that rebuildProjections/replayProjections had no eve R2 (SDK run/coordinator reliability) verified green: coordinator + run-cancellation 53/53. Validation: SQLite focused suite 49/49 (2 new cap tests + 1 updated), V1→V2 round-trip 1/1, cli typecheck clean, Prettier clean. + + +## projectId index advisory evaluated — no change warranted — 2026-09-13T20:17:06.358Z + +Evaluated the reviewer advisory that `scanQueryRows`/`readLastEventIdForProject` use unindexed `json_extract` projectId filters. Benchmarked at the current 10k-event cap (50 iterations each): 250-row filtered query mean 0.815ms unindexed vs 0.671ms with an expression index (within noise); tail query 0.009ms vs 0.008ms. EXPLAIN QUERY PLAN confirms the expression index is used when present, but the absolute cost is already sub-millisecond at the cap. Decision: no schema/index change now; revisit only if the event cap grows materially. Recorded as a data-backed no-change decision. + diff --git a/cli/knowledge.md b/cli/knowledge.md index 1e9bfaf5ce..7d8dacb95a 100644 --- a/cli/knowledge.md +++ b/cli/knowledge.md @@ -11,7 +11,7 @@ ## Memory V2 and Discovery Coverage -- Cross-session task memory lives in `.openbuff/memory/task-memory.json` (gitignored local state) and is backed by an append-only event store; the Bun SQLite repository (`cli/src/services/memory-v2/bun-sqlite-memory-repository.ts`) implements append/query/verify/rebuild/health/export with schema-validated outcomes, hardened local storage (contained DB path, symlink/ownership rejection, `0700`/`0600` perms), and generic fail-closed error classification that never leaks paths. +- Cross-session task memory lives in `.openbuff/memory/task-memory.json` (gitignored local state) and is backed by an append-only event store; the Bun SQLite repository (`cli/src/services/memory-v2/bun-sqlite-memory-repository.ts`) implements append/query/verify/rebuild/health/export with schema-validated outcomes, hardened local storage (contained DB path, symlink/ownership rejection, `0700`/`0600` perms), and generic fail-closed error classification that never leaks paths. `BunSQLiteMemoryRepository.open()` accepts a strict opt-in `requireSecureOpen`: when true it fails closed with a typed non-retryable `unsupported-open` error and performs zero SQLite/filesystem mutation, because bun:sqlite opens the DB and its `-wal`/`-shm` sidecars by derived pathname only (no fd/dirfd), so a race-free open is provably impossible in pure JS — the default open is unchanged and honestly reports `openPosture: 'pathname-best-effort-unverified-open'` on the open result and in `kernelHealth`, never claiming race resistance. Projection replay is bounded by `MAX_REPLAY_EVENTS` (10,000) with existing `PAGE_SIZE` (250) batching; on truncation `replayProjections` sets the projection cursor to the last replayed sequence and returns `truncated: true` (an honest cursor, never falsely claiming the canonical tail), `rebuildProjections()` surfaces `truncated: boolean`, and the v1→v2 migrate passes `Number.MAX_SAFE_INTEGER` so migration replay stays complete. - `evaluate_audit_coverage` tool results are recorded as `coverage.recorded` events by the Memory V2 coordinator (`sdk/src/services/memory-v2/coordinator.ts`), bound to the current workspace revision/snapshot so stale coverage is never reused across edits. - The SQLite repository projects the latest `coverage.recorded` event per `(taskId, dimension)` into the `currentCoverage` retrieval category, filtered by exact workspace revision/snapshot freshness and optional `taskId` narrowing. - The `query_index` tool handler (`packages/agent-runtime/src/tools/handlers/tool/query-index.ts`) records its results into `agentState.discoveryCoverage` via `recordDiscoveryResult` with a bounded (4000-char) question string; recording is wrapped in try/catch so a coverage failure can never break the tool call. From afd2292d89ba51b3f379e77d389d99e20afff52a Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Mon, 14 Sep 2026 00:57:06 +0300 Subject: [PATCH 11/22] fix(gate): resolve deleted-before-snapshot files to missing marker MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A pending gate file deleted before any snapshot captured its bytes re-triggered specialist review forever: readGateFileContentMarker returned an unreadable:* marker for a nonexistent path, but deleted-file handling only recognized the exact missing marker, so the deletion was never classified, per-file credit failed closed, and the specialist's assigned-file-unreadable blocker was never cleared. Now an lstatSync probe maps true nonexistence (ENOENT) to the missing marker while dangling symlinks stay present and other failures stay unreadable (fail closed), and a turn-start prune drops open reviewer findings whose entire file set is missing — along with their verbatim blockers — before the owed-set rehydration can re-arm the reviewer family. --- agents/__tests__/base2.test.ts | 281 ++++++++++++++++++++++++++++++++- agents/base2/base2.ts | 89 ++++++++++- 2 files changed, 363 insertions(+), 7 deletions(-) diff --git a/agents/__tests__/base2.test.ts b/agents/__tests__/base2.test.ts index 9a381bf6bd..2458205ed4 100644 --- a/agents/__tests__/base2.test.ts +++ b/agents/__tests__/base2.test.ts @@ -10397,13 +10397,17 @@ describe('base2 reviewer re-review round ledger', () => { return String(reviewCall.input.agents[0].prompt) } - function codeReviewerFinding(text: string, index: number) { + // `files` must name REAL on-disk paths: the turn-start prune drops findings + // whose every file resolves to the `missing` content marker, so a virtual + // path here would prune the seeded findings before the review packet is + // built and silently empty the ledger under test. + function codeReviewerFinding(text: string, index: number, files: string[]) { return { id: `RF-${index + 1}-0000000${index}`, gateId: 'code-reviewer:prior-snapshot', text, status: 'open' as const, - files: ['src/a.ts'], + files, snapshotFingerprint: 'prior-snapshot', reviewer: 'code-reviewer' as const, createdAt: '2025-01-01T00:00:00.000Z', @@ -10452,10 +10456,13 @@ describe('base2 reviewer re-review round ledger', () => { const gateFile = normalizeGateFilePath(join(tmpDir, 'a.ts')) writeFileSync(join(tmpDir, 'a.ts'), 'export const value = 1\n') const codeFindings = [ - codeReviewerFinding('NON_BLOCKING: Tighten the early-return guard.', 0), + codeReviewerFinding('NON_BLOCKING: Tighten the early-return guard.', 0, [ + gateFile, + ]), codeReviewerFinding( 'BLOCKING: [code-reviewer:tests:missing-case] Add a case for the empty payload.', 1, + [gateFile], ), ] const securityFinding = { @@ -10514,7 +10521,9 @@ describe('base2 reviewer re-review round ledger', () => { const openReviewerFindings = Array.from( { length: 14 }, (_unused, index) => - codeReviewerFinding(`NON_BLOCKING: Finding number ${index}.`, index), + codeReviewerFinding(`NON_BLOCKING: Finding number ${index}.`, index, [ + gateFile, + ]), ) const prompt = driveSeededStateToReviewPrompt( gateFile, @@ -12895,3 +12904,267 @@ describe('base2 EXECUTE_PLAN gate-issued plan-task receipts', () => { ) }) }) + +describe('base2 deleted-before-first-snapshot gate files', () => { + // Regression for the live scripts/perf-probe-tmp.ts loop: a pending gate + // file that is DELETED before any snapshot captured its bytes spawned a + // specialist that could only return `BLOCKING: ...assigned-file-unreadable...`, + // and the open finding that review recorded was never cleared — every turn + // rehydrated it into an owed revalidation, which evicted the specialist's + // credit and re-spawned it forever. Deletion now resolves to the `missing` + // content marker (attested-by-absence, so a `missing`-keyed credit stays + // fresh), and open findings whose files are ALL missing are pruned at turn + // start, before the owed-set rehydration can re-arm the reviewer family. + test('prunes stale unreadable findings for a deleted-never-snapshotted file and never re-spawns its specialist', () => { + const tmpDir = makeProjectTempDir('base2-deleted-before-snapshot-') + try { + // The parent directory exists; only the leaf file is gone (never + // created, never tracked, never committed — exactly the live bug). + mkdirSync(join(tmpDir, 'scripts'), { recursive: true }) + const deletedFile = normalizeGateFilePath( + join(tmpDir, 'scripts', 'perf-probe-tmp.ts'), + ) + const staleBlocker = + 'BLOCKING: performance-specialist assigned-file-unreadable: scripts/perf-probe-tmp.ts' + const base2 = createBase2('default') + const agentState = { + agentId: 'base2-custom', + base2ActiveWork: { + changedFiles: [deletedFile], + touchedFiles: [deletedFile], + pendingGateFiles: [deletedFile], + currentPhase: 'awaiting_validation', + latestWorkSummary: '', + openReviewerBlockers: [staleBlocker], + // The stale finding the live bug could never clear: its only file is + // deleted, so every fresh review returned the same unreadable + // blocker and the finding was re-created each time. + openReviewerFindings: [ + { + id: 'RF-1-deadbeef', + gateId: 'performance-specialist:prior-snapshot', + text: staleBlocker, + status: 'open' as const, + files: [deletedFile], + snapshotFingerprint: 'prior-snapshot', + reviewer: 'performance-specialist', + createdAt: '2025-01-01T00:00:00.000Z', + }, + ], + lastValidationSummary: '', + nextRequiredAction: '', + lastPinnedStateMessage: '', + gatePassedFiles: [], + gatePassedPendingFiles: [], + gatePassedReviewerVerdict: '', + gatePassedValidationSummary: '', + gatePassedFingerprint: '', + lastReviewerGateSkipReason: '', + reviewReceipts: [], + owedReviewerRevalidations: [], + testWriterGateDone: true, + docWriterGateDone: true, + securityReviewGateDone: true, + preEditSecurityReviewDone: true, + // The specialist already passed once against the deleted bytes, so + // its per-file credit marker is the stable `missing` marker. Before + // the fix the marker was not `missing`, so credit freshness treated + // it as stale on every sweep — the other half of the loop. + specialistReviewGatesDone: ['performance-specialist'], + specialistReviewGateFingerprints: { + 'performance-specialist': buildFingerprint( + [{ file: deletedFile, contentMarker: 'missing' }], + '', + ), + }, + specialistReviewFileMarkers: { + 'performance-specialist': { [deletedFile]: 'missing' }, + }, + auxGatesLastPendingFiles: [deletedFile], + }, + } + const gen = base2.handleSteps!({ + agentState, + prompt: 'Please finish the pending performance finding.', + params: {}, + } as any) + + expect(gen.next().value).toMatchObject({ toolName: 'git_status' }) + // Turn start, BEFORE the owed-set rehydration: the all-missing finding + // and its verbatim blocker are pruned, so nothing re-arms a + // performance-specialist revalidation from stale serialized state. + const turnStartWork = (agentState as any).base2ActiveWork + expect(turnStartWork.openReviewerFindings).toEqual([]) + expect(turnStartWork.openReviewerBlockers).toEqual([]) + expect(turnStartWork.owedReviewerRevalidations).toEqual([]) + expect(turnStartWork.requiredReviewerRevalidation).toBeUndefined() + + // The file is untracked and deleted, so git status is clean. + expect(gen.next(feedJson({ status: '' })).value).toMatchObject({ + toolName: 'spawn_agent_inline', + input: { agent_type: 'context-pruner' }, + }) + const maybePinned = gen.next().value + if (maybePinned !== 'STEP') { + expect(maybePinned).toMatchObject({ toolName: 'add_message' }) + expect(gen.next().value).toBe('STEP') + } + expect(gen.next(finishStepWithToolResult({})).value).toMatchObject({ + toolName: 'git_status', + }) + // No specialist spawn: the fresh `missing`-marker credit routes nothing. + // Validation hooks run next for the still-pending (deleted) file. + const hooksCall = gen.next(feedJson({ status: '' })) + expect(hooksCall.value).toMatchObject({ + toolName: 'run_file_change_hooks', + input: { files: [deletedFile] }, + }) + expect(gen.next(feedJson([])).value).toMatchObject({ + toolName: 'git_status', + }) + const reviewCall = gen.next(feedJson({ status: '' })).value as any + expect(reviewCall).toMatchObject({ + toolName: 'spawn_agents', + input: { agents: [{ agent_type: 'code-reviewer' }] }, + }) + const spawnedAgentTypes = ( + reviewCall.input.agents as Array<{ agent_type: string }> + ).map((agent) => agent.agent_type) + expect(spawnedAgentTypes).not.toContain('performance-specialist') + // The snapshot binds the deleted file through the `missing` marker... + const reviewPrompt = String(reviewCall.input.agents[0].prompt) + expect(reviewPrompt).toContain(`${deletedFile}\tmissing`) + const snapshotFingerprint = + reviewPrompt.match( + /Snapshot fingerprint \(echo exactly\): ([^\n]+)/, + )?.[1] ?? '' + // ...so the reviewer attests-by-absence: reviewedFiles legitimately + // omits the deleted file and the review still passes. + expect( + gen.next({ + toolResult: [ + { + type: 'json', + value: [ + { + schemaVersion: 1, + verdict: 'LOOKS_GOOD', + snapshotFingerprint, + reviewedFiles: [], + findings: [], + coverage: 'covered', + dimensions: {}, + requirementCoverage: [], + }, + ], + }, + ], + } as any).value, + ).toMatchObject({ toolName: 'git_status' }) + const gatePassed = gen.next(feedJson({ status: '' })) + expect(gatePassed.value).toMatchObject({ + toolName: 'add_message', + input: { role: 'user' }, + }) + expect((gatePassed.value as any).input.content).toMatch( + /reviewer gate passed with LOOKS_GOOD/i, + ) + const finalWork = (agentState as any).base2ActiveWork + expect(finalWork.currentPhase).toBe('final_response_allowed') + expect(finalWork.openReviewerBlockers).toEqual([]) + expect(finalWork.openReviewerFindings).toEqual([]) + // The deletion is credited as a stable gate-passed state, so later turns + // do not re-arm on it either. + expect(finalWork.gatePassedFiles).toEqual([deletedFile]) + expect(finalWork.specialistReviewGatesDone).toContain( + 'performance-specialist', + ) + } finally { + rmSync(tmpDir, { recursive: true, force: true }) + } + }) + + test('keeps open findings that name any still-existing or unreadable-but-present file', () => { + const tmpDir = makeProjectTempDir('base2-prune-missing-findings-keep-') + try { + const presentFile = join(tmpDir, 'exists.ts') + writeFileSync(presentFile, 'export const here = 1\n') + const presentGateFile = normalizeGateFilePath(presentFile) + const missingGateFile = normalizeGateFilePath(join(tmpDir, 'gone.ts')) + // A present-but-not-a-file path (a directory): the fail-closed + // `unreadable:not-a-file` marker, which must never be read as deleted. + mkdirSync(join(tmpDir, 'adir.ts'), { recursive: true }) + const directoryGateFile = normalizeGateFilePath(join(tmpDir, 'adir.ts')) + const finding = (id: string, text: string, files: string[]) => ({ + id, + gateId: 'code-reviewer:prior-snapshot', + text, + status: 'open' as const, + files, + snapshotFingerprint: 'prior-snapshot', + reviewer: 'code-reviewer' as const, + createdAt: '2025-01-01T00:00:00.000Z', + }) + const prunedText = + 'BLOCKING: assigned-file-unreadable for the deleted probe.' + const keptTexts = [ + 'BLOCKING: still-present file issue.', + 'BLOCKING: mixed deleted-and-present file set issue.', + 'BLOCKING: legacy finding with no files.', + 'BLOCKING: present-but-unreadable directory path issue.', + ] + const base2 = createBase2('default') + const agentState = { + agentId: 'base2-custom', + base2ActiveWork: { + changedFiles: [presentGateFile], + touchedFiles: [presentGateFile], + pendingGateFiles: [presentGateFile], + currentPhase: 'repair_loop', + latestWorkSummary: '', + openReviewerBlockers: [prunedText, ...keptTexts], + openReviewerFindings: [ + // All files missing -> pruned (the loop-breaking case). + finding('find-missing-only', prunedText, [missingGateFile]), + // Still exists -> kept. + finding('find-present', keptTexts[0], [presentGateFile]), + // ANY still-existing file -> kept. + finding('find-mixed', keptTexts[1], [ + missingGateFile, + presentGateFile, + ]), + // No file list at all -> kept (fail closed). + finding('find-no-files', keptTexts[2], []), + // Present but not a regular file -> kept (fail closed). + finding('find-directory', keptTexts[3], [directoryGateFile]), + ], + lastValidationSummary: '', + nextRequiredAction: '', + lastPinnedStateMessage: '', + }, + } + const gen = base2.handleSteps!({ + agentState, + prompt: 'Finish the previous response.', + params: {}, + } as any) + + // The prune runs during turn-start hydration, before the first yield. + expect(gen.next().value).toMatchObject({ toolName: 'git_status' }) + const activeWork = (agentState as any).base2ActiveWork + expect( + (activeWork.openReviewerFindings as Array<{ id: string }>).map( + (entry) => entry.id, + ), + ).toEqual([ + 'find-present', + 'find-mixed', + 'find-no-files', + 'find-directory', + ]) + expect(activeWork.openReviewerBlockers).toEqual(keptTexts) + } finally { + rmSync(tmpDir, { recursive: true, force: true }) + } + }) +}) diff --git a/agents/base2/base2.ts b/agents/base2/base2.ts index 4f29604e88..81dcbe6f5a 100644 --- a/agents/base2/base2.ts +++ b/agents/base2/base2.ts @@ -1202,6 +1202,58 @@ ${guideSections} // serialized state lacks this field, which is what makes the // condonedFindingTexts fallback below conditional on it being empty. activeWorkState.condonedFindingKeys ??= [] + // Deleted-file finding prune. A pending gate file DELETED before any + // snapshot captured its bytes resolves to the `missing` content marker + // (readGateFileContentMarker) and is attested-by-absence, so an open + // finding whose ENTIRE file set is now missing can never be cleared by a + // fresh matching review: the reviewer cannot read a deleted file and + // keeps returning `BLOCKING: ...assigned-file-unreadable...`, which + // re-creates the finding and re-spawns its specialist forever (the + // scripts/perf-probe-tmp.ts loop). Remove such findings — and their + // verbatim blocker strings, matched by the same text-containment rule + // mergeReviewerFindings uses to keep blockers and findings in sync — + // BEFORE the owed-set rehydration below so a pruned finding's reviewer + // family is not rehydrated into another re-review. A finding naming ANY + // still-existing file is kept untouched: genuinely + // unreadable-but-present files (permissions, EISDIR, symlink escape) + // produce `unreadable:*` markers, never `missing`, so fail-closed + // re-review for them is preserved. + { + const openFindings = activeWorkState.openReviewerFindings ?? [] + const missingFileFindings = openFindings.filter((finding) => { + const findingFiles = Array.isArray(finding.files) + ? finding.files.filter( + (file): file is string => + typeof file === 'string' && file.length > 0, + ) + : [] + return ( + findingFiles.length > 0 && + findingFiles.every( + (file) => readGateFileContentMarker(file) === 'missing', + ) + ) + }) + if (missingFileFindings.length > 0) { + const prunedFindingSet = new Set(missingFileFindings) + activeWorkState.openReviewerFindings = openFindings.filter( + (finding) => !prunedFindingSet.has(finding), + ) + const prunedFindingTexts = missingFileFindings + .map((finding) => finding.text) + .filter( + (text): text is string => + typeof text === 'string' && text.length > 0, + ) + activeWorkState.openReviewerBlockers = ( + activeWorkState.openReviewerBlockers ?? [] + ).filter( + (blocker) => + !prunedFindingTexts.some((text) => blocker.includes(text)), + ) + markActiveWorkStateChanged() + } + } if (activeWorkState.openReviewerFindings.length > 0) { // Rehydrate the owed set from EVERY open finding, not just findings[0]: // serialized state can carry open findings from several reviewers and @@ -10602,9 +10654,12 @@ function hashGateSnapshotDetails(details: string): string { * Resolve a normalized gate file path against process.cwd() and return * a deterministic content marker for fingerprinting. Regular files are * hashed in fixed-size chunks; symlink markers additionally bind the link - * path to bytes read from its resolved target. Never throws: scope, read, - * or stat failures become `unreadable:` markers so stale credit - * fails closed. + * path to bytes read from its resolved target. A path that does not exist + * on disk returns the exact marker `missing` (attested-by-absence — the + * same marker a snapshotted-then-deleted file gets from the ENOENT catch + * below, now also produced for a file deleted before its first snapshot). + * Never throws: every other scope, read, or stat failure becomes an + * `unreadable:` marker so stale credit fails closed. */ function readGateFileContentMarker(normalizedPath: string): string { if (!normalizedPath) return 'unreadable:empty-path' @@ -10653,6 +10708,34 @@ function hashGateSnapshotDetails(details: string): string { return 'unreadable:outside-project' } try { + // True nonexistence is attested-by-absence, NOT an unreadable file: + // a pending gate file deleted before any snapshot captured its bytes + // must resolve to the existing `missing` marker so + // collectDeletedFilesFromSnapshotDetails recognizes the deletion and + // per-file credit treats it as stable (a never-snapshotted deletion + // otherwise re-triggers specialist review forever). Probe existence + // BEFORE the component walk; the walk's lstatSync (or the open below) + // still throws ENOENT for the TOCTOU case — deleted between this + // probe and the read — and the catch below maps ENOENT to `missing` + // too. A path that EXISTS (regular file, directory, or symlink) falls + // through unchanged, so present-but-not-a-file stays + // `unreadable:not-a-file` and an escaping symlink stays + // `unreadable:outside-project-symlink`: only true absence yields + // `missing`. + // Use lstatSync (not existsSync) for the probe: existsSync follows a + // dangling symlink to its nonexistent target and would wrongly report + // `missing`, whereas lstatSync sees the link entry itself as present + // (ENOENT only when the path truly does not exist). + try { + fs.lstatSync(absolutePath) + } catch (probeError) { + if ((probeError as NodeJS.ErrnoException).code === 'ENOENT') { + return 'missing' + } + // A present path that cannot be stated (permissions, etc.) stays + // fail-closed as unreadable rather than being mistaken for absent. + return 'unreadable:lstat-failed' + } const pathSegments = projectRelativePath .split(path.sep) .filter(Boolean) From acbc8ce092a23b4ba6cb0d3b8bdf3f9106b79aa5 Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Mon, 14 Sep 2026 01:15:17 +0300 Subject: [PATCH 12/22] test(gate): create aux-triple fixture on disk so seeded findings survive the missing-file prune The committed gate fix resolves open reviewer findings whose files no longer exist on disk to a missing marker and prunes them. Two e2e fixture tests seeded openReviewerFindings with a virtual path (cli/src/auth/token-store.ts) that never existed, so the turn-start prune dropped the finding and the owed specialist never rehydrated, failing the pre-push hook. Create the fixture file before handleSteps runs and remove only what the fixture created in afterEach (rmdirSync refuses non-empty dirs, so real repo content is never deleted). --- agents/e2e/gate-aux-ordering.e2e.test.ts | 28 +++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/agents/e2e/gate-aux-ordering.e2e.test.ts b/agents/e2e/gate-aux-ordering.e2e.test.ts index a4d48e36e8..409ad9e633 100644 --- a/agents/e2e/gate-aux-ordering.e2e.test.ts +++ b/agents/e2e/gate-aux-ordering.e2e.test.ts @@ -1,4 +1,4 @@ -import { mkdirSync, rmSync, writeFileSync } from 'node:fs' +import { mkdirSync, rmdirSync, rmSync, writeFileSync } from 'node:fs' import { afterEach, describe, expect, test } from 'bun:test' @@ -256,6 +256,18 @@ describe('base2 pre-reviewer aux gate ordering e2e', () => { // them even when an assertion fails mid-test (not only on the happy path). afterEach(() => { rmSync(SPECIALIST_SCRATCH_ROOT, { recursive: true, force: true }) + // The owed-specialist tests below create AUX_TRIPLE_FILE on disk so their + // seeded openReviewerFindings entry survives the turn-start missing-file + // prune. Remove only what the fixture created: the file, plus the + // cli/src/auth and cli/src dirs ONLY when empty (rmdirSync refuses + // non-empty dirs, so a pre-existing real repo tree is never deleted). + rmSync(AUX_TRIPLE_FILE, { force: true }) + try { + rmdirSync('cli/src/auth') + rmdirSync('cli/src') + } catch { + // Absent or non-empty (a real repo path): leave it in place. + } }) test('fires test-writer -> doc-writer -> security-reviewer before validation hooks + code-reviewer, then does not re-spawn', () => { @@ -638,6 +650,13 @@ describe('base2 pre-reviewer aux gate ordering e2e', () => { }) test('a coverage-complete routed specialist review with a matching gate fingerprint does not block the gate', () => { + // The seeded openReviewerFindings entry below names AUX_TRIPLE_FILE; the + // turn-start prune drops open findings whose files are all missing on + // disk. Create the fixture file BEFORE handleSteps runs so the finding + // survives the prune and the owed specialist rehydrates (mirrors the + // SPECIALIST_FILE pattern; the describe afterEach removes it). + mkdirSync('cli/src/auth', { recursive: true }) + writeFileSync(AUX_TRIPLE_FILE, 'export const token = 1\n') const base2 = createBase2('default') // Seed a reliability-reviewer owed marker (the aux-triple fixture // cli/src/auth/token-store.ts routes no specialist on its own, so the @@ -1523,6 +1542,13 @@ describe('base2 pre-reviewer aux gate ordering e2e', () => { }) test('revalidates an owed specialist reviewer as aux-owned across turns before the final code-reviewer', () => { + // The seeded openReviewerFindings entry below names AUX_TRIPLE_FILE; the + // turn-start prune drops open findings whose files are all missing on + // disk. Create the fixture file BEFORE handleSteps runs so the finding + // survives the prune and the owed specialist marker rehydrates (mirrors + // the SPECIALIST_FILE pattern; the describe afterEach removes it). + mkdirSync('cli/src/auth', { recursive: true }) + writeFileSync(AUX_TRIPLE_FILE, 'export const token = 1\n') const base2 = createBase2('default') // Seed the turn so the marker is already owed to a specialist, simulating // a prior-turn blocking reliability-reviewer finding. The other aux gates From bb38cba163ef209d858453303bb7781995830f29 Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Mon, 14 Sep 2026 02:20:53 +0300 Subject: [PATCH 13/22] fix(gate): retire stale owed-reviewer reference on prune Share the 'missing' content marker via a single in-handleSteps GATE_FILE_MISSING_CONTENT_MARKER constant so the deletion-semantics producer and consumers cannot drift, and when the turn-start deleted-file prune empties openReviewerFindings also remove a pruned-and-unbacked family from owedReviewerRevalidations and re-mirror requiredReviewerRevalidation from the filtered owed[0]. A family still backed by a remaining open finding stays owed (fail closed). Adds regression coverage including a partial-prune case and syncs STATUS.md. --- .../dynamic-cross-session-memory/STATUS.md | 8 ++ agents/__tests__/base2.test.ts | 122 +++++++++++++++++- agents/base2/base2.ts | 94 +++++++++++--- 3 files changed, 207 insertions(+), 17 deletions(-) diff --git a/.agents/sessions/dynamic-cross-session-memory/STATUS.md b/.agents/sessions/dynamic-cross-session-memory/STATUS.md index ddceddfa13..abbdb532fa 100644 --- a/.agents/sessions/dynamic-cross-session-memory/STATUS.md +++ b/.agents/sessions/dynamic-cross-session-memory/STATUS.md @@ -173,3 +173,11 @@ Validation: SQLite focused suite 49/49 (2 new cap tests + 1 updated), V1→V2 ro Evaluated the reviewer advisory that `scanQueryRows`/`readLastEventIdForProject` use unindexed `json_extract` projectId filters. Benchmarked at the current 10k-event cap (50 iterations each): 250-row filtered query mean 0.815ms unindexed vs 0.671ms with an expression index (within noise); tail query 0.009ms vs 0.008ms. EXPLAIN QUERY PLAN confirms the expression index is used when present, but the absolute cost is already sub-millisecond at the cap. Decision: no schema/index change now; revisit only if the event cap grows materially. Recorded as a data-backed no-change decision. + + +## R4/R6 verification + phantom-file gate fix + 'missing' marker constant — 2026-09-13T22:50:29.563Z + +Re-verified the memory-v2 plan's remaining focused suites in the current tree: R4 provider/client authority (provider + env) 26/26; R6 coverage/prompt safety (agent-runtime task-memory + memory-v2-context + loop-agent-steps) 84/84; memory-retention eval 6/6. R1-T1 (SQLite 44/44), R1-T2 (typed-unsupported secure-open), R2 (coordinator 53/53), R3-T1 (migration/operator 96/96) all confirmed green. Only R7 finalization (stable exact-snapshot reviews) remains. + +Also shipped the phantom-file gate fix (commits afd2292d8 + acbc8ce09): a pending gate file deleted before its first snapshot now resolves to the `missing` content marker (attested-by-absence), and open findings whose files are all missing are pruned at turn start — closing the scripts/perf-probe-tmp.ts review loop. Follow-up hardening: extracted the `'missing'` sentinel into a single in-handleSteps constant `GATE_FILE_MISSING_CONTENT_MARKER` shared by readGateFileContentMarker, collectDeletedFilesFromSnapshotDetails, the turn-start prune, and isCreditableContentMarker. Pure refactor; agents typecheck clean and gate/parity/serialization suites 255/255. + diff --git a/agents/__tests__/base2.test.ts b/agents/__tests__/base2.test.ts index 2458205ed4..ecf49ba96e 100644 --- a/agents/__tests__/base2.test.ts +++ b/agents/__tests__/base2.test.ts @@ -12961,7 +12961,11 @@ describe('base2 deleted-before-first-snapshot gate files', () => { gatePassedFingerprint: '', lastReviewerGateSkipReason: '', reviewReceipts: [], - owedReviewerRevalidations: [], + // The stale owed-set entry the prune must retire: both the legacy + // scalar and the list point at the pruned reviewer family, so a + // missing clear would leave a dead revalidation reference behind. + requiredReviewerRevalidation: 'performance-specialist', + owedReviewerRevalidations: ['performance-specialist'], testWriterGateDone: true, docWriterGateDone: true, securityReviewGateDone: true, @@ -12992,7 +12996,9 @@ describe('base2 deleted-before-first-snapshot gate files', () => { expect(gen.next().value).toMatchObject({ toolName: 'git_status' }) // Turn start, BEFORE the owed-set rehydration: the all-missing finding // and its verbatim blocker are pruned, so nothing re-arms a - // performance-specialist revalidation from stale serialized state. + // performance-specialist revalidation from stale serialized state. The + // prune also retires the stale owed-set entry that pointed at the + // pruned family: both the legacy scalar and the list are cleared. const turnStartWork = (agentState as any).base2ActiveWork expect(turnStartWork.openReviewerFindings).toEqual([]) expect(turnStartWork.openReviewerBlockers).toEqual([]) @@ -13167,4 +13173,116 @@ describe('base2 deleted-before-first-snapshot gate files', () => { rmSync(tmpDir, { recursive: true, force: true }) } }) + + // Companion to the prune: a family is un-owed only when it was actually + // pruned AND no REMAINING open finding still backs it. Seed one prunable + // specialist finding (deleted file) plus a prunable and a surviving + // code-reviewer finding so the prune fires WITHOUT emptying the ledger: the + // specialist family is shed from the owed list, the still-backed + // code-reviewer family stays owed (fail closed), and the legacy scalar — + // seeded on the pruned family — is rewritten to the filtered owed[0] exactly + // as the rehydration block derives it. + test('a partial prune sheds only the pruned-and-unbacked family from the owed set and re-mirrors the scalar', () => { + const tmpDir = makeProjectTempDir('base2-prune-owed-partial-') + try { + const presentFile = join(tmpDir, 'exists.ts') + writeFileSync(presentFile, 'export const here = 1\n') + const presentGateFile = normalizeGateFilePath(presentFile) + const missingGateFile = normalizeGateFilePath(join(tmpDir, 'gone.ts')) + const prunedSpecialistText = + 'BLOCKING: performance-specialist assigned-file-unreadable for the deleted probe.' + const prunedCodeText = + 'BLOCKING: code-reviewer finding on the deleted file.' + const keptCodeText = + 'BLOCKING: code-reviewer finding on the surviving file.' + const finding = ( + id: string, + text: string, + files: string[], + reviewer: 'code-reviewer' | 'performance-specialist', + ) => ({ + id, + gateId: `${reviewer}:prior-snapshot`, + text, + status: 'open' as const, + files, + snapshotFingerprint: 'prior-snapshot', + reviewer, + createdAt: '2025-01-01T00:00:00.000Z', + }) + const base2 = createBase2('default') + const agentState = { + agentId: 'base2-custom', + base2ActiveWork: { + changedFiles: [presentGateFile], + touchedFiles: [presentGateFile], + pendingGateFiles: [presentGateFile], + currentPhase: 'repair_loop', + latestWorkSummary: '', + openReviewerBlockers: [ + prunedSpecialistText, + prunedCodeText, + keptCodeText, + ], + openReviewerFindings: [ + // All files missing -> pruned; the ONLY specialist finding. + finding( + 'find-specialist-missing', + prunedSpecialistText, + [missingGateFile], + 'performance-specialist', + ), + // All files missing -> pruned, but the code-reviewer family stays + // backed by the surviving finding below. + finding( + 'find-code-missing', + prunedCodeText, + [missingGateFile], + 'code-reviewer', + ), + // Still exists -> kept; keeps the code-reviewer family owed. + finding( + 'find-code-present', + keptCodeText, + [presentGateFile], + 'code-reviewer', + ), + ], + // The legacy scalar seeded on the PRUNED family's head: after the + // filter it must be re-mirrored to the surviving owed[0]. + requiredReviewerRevalidation: 'performance-specialist', + owedReviewerRevalidations: [ + 'performance-specialist', + 'code-reviewer', + ], + lastValidationSummary: '', + nextRequiredAction: '', + lastPinnedStateMessage: '', + }, + } + const gen = base2.handleSteps!({ + agentState, + prompt: 'Finish the previous response.', + params: {}, + } as any) + + // The prune runs during turn-start hydration, before the first yield. + expect(gen.next().value).toMatchObject({ toolName: 'git_status' }) + const activeWork = (agentState as any).base2ActiveWork + // The specialist family was pruned and is backed by no remaining + // finding, so it is shed from the owed list; the code-reviewer family + // was also pruned but stays owed because find-code-present still backs + // it. The scalar is rewritten from the filtered list's first entry. + expect(activeWork.owedReviewerRevalidations).toEqual(['code-reviewer']) + expect(activeWork.requiredReviewerRevalidation).toBe('code-reviewer') + expect( + (activeWork.openReviewerFindings as Array<{ id: string }>).map( + (entry) => entry.id, + ), + ).toEqual(['find-code-present']) + expect(activeWork.openReviewerBlockers).toEqual([keptCodeText]) + } finally { + rmSync(tmpDir, { recursive: true, force: true }) + } + }) }) diff --git a/agents/base2/base2.ts b/agents/base2/base2.ts index 81dcbe6f5a..c7fd3b07d7 100644 --- a/agents/base2/base2.ts +++ b/agents/base2/base2.ts @@ -1012,6 +1012,26 @@ ${guideSections} 'policy', ] const SECURITY_SENSITIVE_NAME_SUBSTRINGS = ['secret', 'token', 'apikey'] + // The sentinel content marker for a gate file that was deleted (true + // nonexistence — see the ENOENT probe in readGateFileContentMarker). + // Single source of truth for every deletion-semantics site so the + // producer and its consumers cannot drift: readGateFileContentMarker + // (both ENOENT returns), collectDeletedFilesFromSnapshotDetails, the + // turn-start open-finding prune, and isCreditableContentMarker. + // Declared INSIDE the handleSteps body rather than at module scope for + // the same serialization reason as the globs above: handleSteps is + // serialized via .toString() and reconstructed with new Function(...), + // so a module-scope binding would be undefined in the reconstructed + // body, while this in-body const is serialized along with the function. + // It must stay ABOVE the deleted-file finding prune in source order: + // `const` bindings are not hoisted (temporal dead zone), and the prune + // executes during the generator's initial top-to-bottom pass, so a + // lower declaration would throw a ReferenceError at turn start. The + // VALUE must remain exactly 'missing': it is persisted in serialized + // gate state (the per-file marker ledgers) and compared against markers + // recomputed from the live filesystem, so a changed value would silently + // evict all persisted deletion credit. + const GATE_FILE_MISSING_CONTENT_MARKER = 'missing' const runReviewerGate = runValidationGate const reviewerAgentType = 'code-reviewer' const MAX_REVIEWER_NO_VERDICT_RETRIES = 1 @@ -1204,7 +1224,8 @@ ${guideSections} activeWorkState.condonedFindingKeys ??= [] // Deleted-file finding prune. A pending gate file DELETED before any // snapshot captured its bytes resolves to the `missing` content marker - // (readGateFileContentMarker) and is attested-by-absence, so an open + // (readGateFileContentMarker's GATE_FILE_MISSING_CONTENT_MARKER) and is + // attested-by-absence, so an open // finding whose ENTIRE file set is now missing can never be cleared by a // fresh matching review: the reviewer cannot read a deleted file and // keeps returning `BLOCKING: ...assigned-file-unreadable...`, which @@ -1230,7 +1251,9 @@ ${guideSections} return ( findingFiles.length > 0 && findingFiles.every( - (file) => readGateFileContentMarker(file) === 'missing', + (file) => + readGateFileContentMarker(file) === + GATE_FILE_MISSING_CONTENT_MARKER, ) ) }) @@ -1251,6 +1274,42 @@ ${guideSections} (blocker) => !prunedFindingTexts.some((text) => blocker.includes(text)), ) + // Retire the owed-set reference the prune would otherwise strand. A + // family whose every open finding was just dropped can never be + // re-attested — the reviewer cannot read a deleted file — so leaving + // it in owedReviewerRevalidations would keep re-arming its aux block + // forever after the ledger emptied. Remove ONLY a family that was + // actually pruned AND is backed by no remaining open finding: a + // pruned family with a surviving finding stays owed (fail closed), + // and a family that was never pruned is untouched. The legacy scalar + // is owed[0]'s mirror (exactly how the rehydration block derives + // it), so a stale one is rewritten from the filtered list rather + // than left pointing at the pruned family. + const prunedFamilies = new Set( + missingFileFindings.map((finding) => + reviewerFamilyFromFinding(finding), + ), + ) + const remainingOwed = new Set( + activeWorkState.openReviewerFindings.map((finding) => + reviewerFamilyFromFinding(finding), + ), + ) + activeWorkState.owedReviewerRevalidations = ( + activeWorkState.owedReviewerRevalidations ?? [] + ).filter( + (family) => + !prunedFamilies.has(family) || remainingOwed.has(family), + ) + const staleOwedScalar = activeWorkState.requiredReviewerRevalidation + if ( + staleOwedScalar !== undefined && + prunedFamilies.has(staleOwedScalar) && + !remainingOwed.has(staleOwedScalar) + ) { + activeWorkState.requiredReviewerRevalidation = + activeWorkState.owedReviewerRevalidations[0] ?? undefined + } markActiveWorkStateChanged() } } @@ -9575,7 +9634,10 @@ function hashGateSnapshotDetails(details: string): string { // non-attestable markers (unreadable:, missing-crypto, etc.) remain // excluded so they can never grant durable gate credit. function isCreditableContentMarker(value: string): boolean { - return isAttestableContentMarker(value) || value === 'missing' + return ( + isAttestableContentMarker(value) || + value === GATE_FILE_MISSING_CONTENT_MARKER + ) } function hasFreshGateFingerprintForPendingFiles( @@ -10626,13 +10688,14 @@ function hashGateSnapshotDetails(details: string): string { } // Deleted-file extraction from files-v4 snapshot details. A pending file - // whose content marker is exactly `missing` was deleted in the changeset - // and cannot be read by the reviewer, so it is attested-by-absence and - // excluded from the reviewedFiles requirement. Only exact `missing` - // markers count: `unreadable:` is a present-but-unreadable file - // that must still be attested (fail closed). Self-contained inline - // helper (handleSteps is serialized via .toString() + new Function(...), - // so it must not reference module-scope imports). + // whose content marker is exactly `missing` — the shared + // GATE_FILE_MISSING_CONTENT_MARKER sentinel — was deleted in the + // changeset and cannot be read by the reviewer, so it is + // attested-by-absence and excluded from the reviewedFiles requirement. + // Only exact `missing` markers count: `unreadable:` is a + // present-but-unreadable file that must still be attested (fail closed). + // Self-contained inline helper (handleSteps is serialized via .toString() + // + new Function(...), so it must not reference module-scope imports). function collectDeletedFilesFromSnapshotDetails( details: string, ): string[] { @@ -10643,7 +10706,7 @@ function hashGateSnapshotDetails(details: string): string { if (line === '--') break const tabIndex = line.indexOf('\t') if (tabIndex <= 0) continue - if (line.slice(tabIndex + 1) === 'missing') { + if (line.slice(tabIndex + 1) === GATE_FILE_MISSING_CONTENT_MARKER) { deletedFiles.push(line.slice(0, tabIndex)) } } @@ -10655,8 +10718,9 @@ function hashGateSnapshotDetails(details: string): string { * a deterministic content marker for fingerprinting. Regular files are * hashed in fixed-size chunks; symlink markers additionally bind the link * path to bytes read from its resolved target. A path that does not exist - * on disk returns the exact marker `missing` (attested-by-absence — the - * same marker a snapshotted-then-deleted file gets from the ENOENT catch + * on disk returns the exact marker `missing` + * (GATE_FILE_MISSING_CONTENT_MARKER; attested-by-absence — the same + * marker a snapshotted-then-deleted file gets from the ENOENT catch * below, now also produced for a file deleted before its first snapshot). * Never throws: every other scope, read, or stat failure becomes an * `unreadable:` marker so stale credit fails closed. @@ -10730,7 +10794,7 @@ function hashGateSnapshotDetails(details: string): string { fs.lstatSync(absolutePath) } catch (probeError) { if ((probeError as NodeJS.ErrnoException).code === 'ENOENT') { - return 'missing' + return GATE_FILE_MISSING_CONTENT_MARKER } // A present path that cannot be stated (permissions, etc.) stays // fail-closed as unreadable rather than being mistaken for absent. @@ -10814,7 +10878,7 @@ function hashGateSnapshotDetails(details: string): string { err && typeof err === 'object' && 'code' in err ? String((err as { code?: unknown }).code ?? 'unknown') : 'unknown' - if (code === 'ENOENT') return 'missing' + if (code === 'ENOENT') return GATE_FILE_MISSING_CONTENT_MARKER return `unreadable:${code}` } } From 36ca9e0f30f3ff8b783f05b1c4949c9eb2928d61 Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Mon, 14 Sep 2026 02:42:42 +0300 Subject: [PATCH 14/22] test(gate): align reviewer-marker parity mirror with missing-file production behavior The pre-push parity guard failed because the test-local gateFileMarker mirror and the production-oracle loader had drifted from readGateFileContentMarker, which returns the 'missing' content marker for nonexistent paths. Add the early lstatSync existence probe to the mirror so it attests missing leaf/intermediate paths like production, and hoist the GATE_FILE_MISSING_CONTENT_MARKER declaration into the synthetic new Function scope in loadProductionGateFileContentMarker so the extracted oracle no longer throws a ReferenceError on the ENOENT probe. Validated: parity suite 19/19, full agents suite 1184 pass / 0 fail, agents typecheck clean. --- .../dynamic-cross-session-memory/STATUS.md | 6 +++ .../e2e/reviewer-spawn-conditions.e2e.test.ts | 41 ++++++++++++++++++- 2 files changed, 46 insertions(+), 1 deletion(-) diff --git a/.agents/sessions/dynamic-cross-session-memory/STATUS.md b/.agents/sessions/dynamic-cross-session-memory/STATUS.md index abbdb532fa..ef434eea00 100644 --- a/.agents/sessions/dynamic-cross-session-memory/STATUS.md +++ b/.agents/sessions/dynamic-cross-session-memory/STATUS.md @@ -181,3 +181,9 @@ Re-verified the memory-v2 plan's remaining focused suites in the current tree: R Also shipped the phantom-file gate fix (commits afd2292d8 + acbc8ce09): a pending gate file deleted before its first snapshot now resolves to the `missing` content marker (attested-by-absence), and open findings whose files are all missing are pruned at turn start — closing the scripts/perf-probe-tmp.ts review loop. Follow-up hardening: extracted the `'missing'` sentinel into a single in-handleSteps constant `GATE_FILE_MISSING_CONTENT_MARKER` shared by readGateFileContentMarker, collectDeletedFilesFromSnapshotDetails, the turn-start prune, and isCreditableContentMarker. Pure refactor; agents typecheck clean and gate/parity/serialization suites 255/255. + + +## Parity-mirror fix for missing-marker (pre-push green) — 2026-09-13T23:36:05.557Z + +Fixed the pre-push hook failure that blocked the gate-improvement push: the test-local `gateFileMarker` mirror in `agents/e2e/reviewer-spawn-conditions.e2e.test.ts` had drifted from production `readGateFileContentMarker` (which now returns `'missing'` for a nonexistent path). Root-caused with a debugger: the parity oracle extracts only the `readGateFileContentMarker` function body, so the `GATE_FILE_MISSING_CONTENT_MARKER` const (declared earlier in `handleSteps`) was unbound in the synthetic `new Function` scope, making the ENOENT probe throw a ReferenceError surfaced as `unreadable:unknown`. Fixed two ways: (1) added the early `lstatSync` existence probe to the test mirror so it returns `missing` for nonexistent paths like production, and (2) updated `loadProductionGateFileContentMarker` to hoist the `GATE_FILE_MISSING_CONTENT_MARKER` declaration into the synthetic eval scope (following the `specialist-router-parity.test.ts` hoisted-constant precedent), preserving the const's single-source-of-truth and the drift-safety property. Validated: parity suite 19/19, full agents suite 1184 pass / 0 fail, agents typecheck clean. + diff --git a/agents/e2e/reviewer-spawn-conditions.e2e.test.ts b/agents/e2e/reviewer-spawn-conditions.e2e.test.ts index 4c09bed4b9..3e679468a3 100644 --- a/agents/e2e/reviewer-spawn-conditions.e2e.test.ts +++ b/agents/e2e/reviewer-spawn-conditions.e2e.test.ts @@ -133,6 +133,20 @@ function gateFileMarker(path: string): string { ) { return 'unreadable:outside-project' } + // Existence probe BEFORE the component walk (mirrors production): a + // nonexistent absolute path — missing leaf OR missing intermediate + // directory — is attested-by-absence ('missing') without ever reaching + // realpathSync/readFileSync; a present-but-unstatable path stays + // fail-closed. lstatSync (not existsSync) so a dangling symlink's link + // entry still counts as present. + try { + lstatSync(absolutePath) + } catch (probeError) { + if ((probeError as { code?: unknown }).code === 'ENOENT') { + return 'missing' + } + return 'unreadable:lstat-failed' + } const pathSegments = projectRelativePath.split(sep).filter(Boolean) const symlinkParts: string[] = [] let entryPath = cwd @@ -220,8 +234,33 @@ function loadProductionGateFileContentMarker(): (path: string) => string { base2JavaScript, 'readGateFileContentMarker', ) + // readGateFileContentMarker references the hoisted in-handleSteps const + // GATE_FILE_MISSING_CONTENT_MARKER (the 'missing' deletion sentinel). That + // binding sits OUTSIDE the extracted function span, so the standalone + // `new Function` reconstruction would leave it undefined and the ENOENT + // probe would throw a ReferenceError (surfacing as `unreadable:unknown` + // instead of `missing`). Slice the hoisted declaration from the transpiled + // source and prepend it to the evaluated scope, exactly as + // specialist-router-parity.test.ts does for hoisted constants. The value is + // pulled verbatim from production source, so the parity property is kept. + const hoistStart = base2JavaScript.indexOf( + 'const GATE_FILE_MISSING_CONTENT_MARKER', + ) + const inlineFnStart = base2JavaScript.indexOf( + 'function readGateFileContentMarker(', + ) + if (hoistStart < 0 || inlineFnStart < 0 || hoistStart > inlineFnStart) { + throw new Error( + 'Unable to find hoisted GATE_FILE_MISSING_CONTENT_MARKER before readGateFileContentMarker', + ) + } + const hoistEnd = base2JavaScript.indexOf(';', hoistStart) + if (hoistEnd < 0) { + throw new Error('Unable to find the end of the GATE_FILE_MISSING_CONTENT_MARKER declaration') + } + const hoistedConstSource = base2JavaScript.slice(hoistStart, hoistEnd + 1) const fn = new Function( - `"use strict";\n${helperSource}\nreturn readGateFileContentMarker`, + `"use strict";\n${hoistedConstSource}\n${helperSource}\nreturn readGateFileContentMarker`, ) as () => (path: string) => string return fn() } From cf5d8f41748e723f58982618f35a795be9b214cd Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Mon, 14 Sep 2026 09:43:29 +0300 Subject: [PATCH 15/22] test(gate): robust hoisted-const slice and visible symlink-skip logging in parity guard Reviewer-advisory hardening for the reviewer-spawn-conditions parity e2e: (1) loadProductionGateFileContentMarker now locates the GATE_FILE_MISSING_CONTENT_MARKER declaration end by scanning for the equals sign, the initializer's opening quote, its matching closing quote, then the terminating semicolon, so a semicolon inside the string literal can no longer mis-slice the hoisted const. (2) The external, internal, and mid-path POSIX symlink catch blocks now console.warn which sub-case was skipped (with the error code) instead of silently returning, making sandbox EPERM skips visible. Also appends the R7-T2 package-wide validation record to the dynamic-cross-session-memory STATUS doc. Validated: parity suite 19/19, full agents suite 1184 pass / 0 fail, agents typecheck clean. --- .../dynamic-cross-session-memory/STATUS.md | 6 +++ .../e2e/reviewer-spawn-conditions.e2e.test.ts | 49 +++++++++++++++++-- 2 files changed, 51 insertions(+), 4 deletions(-) diff --git a/.agents/sessions/dynamic-cross-session-memory/STATUS.md b/.agents/sessions/dynamic-cross-session-memory/STATUS.md index ef434eea00..dee28e7037 100644 --- a/.agents/sessions/dynamic-cross-session-memory/STATUS.md +++ b/.agents/sessions/dynamic-cross-session-memory/STATUS.md @@ -187,3 +187,9 @@ Also shipped the phantom-file gate fix (commits afd2292d8 + acbc8ce09): a pendin Fixed the pre-push hook failure that blocked the gate-improvement push: the test-local `gateFileMarker` mirror in `agents/e2e/reviewer-spawn-conditions.e2e.test.ts` had drifted from production `readGateFileContentMarker` (which now returns `'missing'` for a nonexistent path). Root-caused with a debugger: the parity oracle extracts only the `readGateFileContentMarker` function body, so the `GATE_FILE_MISSING_CONTENT_MARKER` const (declared earlier in `handleSteps`) was unbound in the synthetic `new Function` scope, making the ENOENT probe throw a ReferenceError surfaced as `unreadable:unknown`. Fixed two ways: (1) added the early `lstatSync` existence probe to the test mirror so it returns `missing` for nonexistent paths like production, and (2) updated `loadProductionGateFileContentMarker` to hoist the `GATE_FILE_MISSING_CONTENT_MARKER` declaration into the synthetic eval scope (following the `specialist-router-parity.test.ts` hoisted-constant precedent), preserving the const's single-source-of-truth and the drift-safety property. Validated: parity suite 19/19, full agents suite 1184 pass / 0 fail, agents typecheck clean. + + +## R7-T2 package-wide validation green — 2026-09-14T06:32:15.052Z + +Re-ran MEM2-R7-T2 (package-wide validation + artifact smoke) on the current tree to produce a fresh green baseline before R7-T3. Results: monorepo typecheck 11/11; common 1245 pass; agent-runtime 1618 pass; sdk 1398 pass / 1 skip; cli 3093 pass / 15 skip / 2 fail; evals memory-retention 6/6. SDK build (ESM/CJS/types) and `smoke-test:dist` (CJS require + tree-sitter) passed; CLI binary build + `--version` probe passed. The 2 CLI failures are the known flaky `StatusBar` React-act tests (`renders the status label...` and `hides the scroll control...`), which pass 3/3 in isolation, were untouched by this work, and are unrelated to memory-v2 — a pre-existing flake, not an R7 blocker. R7-T2 acceptance met. + diff --git a/agents/e2e/reviewer-spawn-conditions.e2e.test.ts b/agents/e2e/reviewer-spawn-conditions.e2e.test.ts index 3e679468a3..dbc7a89447 100644 --- a/agents/e2e/reviewer-spawn-conditions.e2e.test.ts +++ b/agents/e2e/reviewer-spawn-conditions.e2e.test.ts @@ -254,7 +254,24 @@ function loadProductionGateFileContentMarker(): (path: string) => string { 'Unable to find hoisted GATE_FILE_MISSING_CONTENT_MARKER before readGateFileContentMarker', ) } - const hoistEnd = base2JavaScript.indexOf(';', hoistStart) + // End-of-declaration robust to a `;` inside the string-literal initializer: + // find the `=`, the initializer's opening quote (either quote style), its + // matching closing quote, then the terminating `;`. A bare indexOf(';') + // could mis-slice if the literal ever contained a `;`. + const equalsIndex = base2JavaScript.indexOf('=', hoistStart) + const openQuoteOffset = + equalsIndex < 0 ? -1 : base2JavaScript.slice(equalsIndex).search(/['"]/) + const openQuoteIndex = + openQuoteOffset < 0 ? -1 : equalsIndex + openQuoteOffset + const closeQuoteIndex = + openQuoteIndex < 0 + ? -1 + : base2JavaScript.indexOf( + base2JavaScript.charAt(openQuoteIndex), + openQuoteIndex + 1, + ) + const hoistEnd = + closeQuoteIndex < 0 ? -1 : base2JavaScript.indexOf(';', closeQuoteIndex) if (hoistEnd < 0) { throw new Error('Unable to find the end of the GATE_FILE_MISSING_CONTENT_MARKER declaration') } @@ -1265,7 +1282,15 @@ describe('base2 reviewer spawn conditions e2e', () => { const symlinkAbsolute = join(tempDir, 'fixture.ts') try { symlinkSync(target, symlinkAbsolute, 'file') - } catch { + } catch (symlinkError) { + // POSIX symlink creation can still fail (e.g. EPERM under a + // sandboxed filesystem); log the skip so it is visible, then return. + const code = String( + (symlinkError as { code?: unknown }).code ?? 'unknown', + ) + console.warn( + `parity guard: skipping external-symlink sub-case (symlinkSync failed, code=${code})`, + ) return } const symlinkPath = relative(process.cwd(), symlinkAbsolute).replace( @@ -1295,7 +1320,15 @@ describe('base2 reviewer spawn conditions e2e', () => { const internalSymlinkAbsolute = join(linkDir, 'link.ts') try { symlinkSync(internalTarget, internalSymlinkAbsolute, 'file') - } catch { + } catch (symlinkError) { + // POSIX symlink creation can still fail (e.g. EPERM under a + // sandboxed filesystem); log the skip so it is visible, then return. + const code = String( + (symlinkError as { code?: unknown }).code ?? 'unknown', + ) + console.warn( + `parity guard: skipping internal-symlink sub-case (symlinkSync failed, code=${code})`, + ) return } const internalSymlinkPath = relative( @@ -1328,7 +1361,15 @@ describe('base2 reviewer spawn conditions e2e', () => { try { symlinkSync(realDir, aliasDir, 'dir') midSymlinkAbsolute = join(aliasDir, 'sub', 'file.ts') - } catch { + } catch (symlinkError) { + // POSIX symlink creation can still fail (e.g. EPERM under a + // sandboxed filesystem); log the skip so it is visible, then return. + const code = String( + (symlinkError as { code?: unknown }).code ?? 'unknown', + ) + console.warn( + `parity guard: skipping mid-path-symlink sub-case (symlinkSync failed, code=${code})`, + ) return } const midSymlinkPath = relative( From f62d761c590cb065b3bd379df180aa19ba47e67a Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Mon, 14 Sep 2026 10:23:39 +0300 Subject: [PATCH 16/22] test(gate,cli): fix flaky status-bar act and fail-fast slice guard Wrap the OpenTUI testRender mount and renderOnce frame in a dev/prod-safe act() shim and set IS_REACT_ACT_ENVIRONMENT so reconciler updates are captured deterministically, eliminating intermittent not-wrapped-in-act failures. Assert the hoisted GATE_FILE_MISSING_CONTENT_MARKER slice keeps declaration shape before eval so a base2.ts layout reorder throws a clear error instead of a confusing ReferenceError or silent sentinel mismatch. --- .../e2e/reviewer-spawn-conditions.e2e.test.ts | 23 +++++++++++ .../components/__tests__/status-bar.test.tsx | 39 ++++++++++++++++--- 2 files changed, 57 insertions(+), 5 deletions(-) diff --git a/agents/e2e/reviewer-spawn-conditions.e2e.test.ts b/agents/e2e/reviewer-spawn-conditions.e2e.test.ts index dbc7a89447..df68388036 100644 --- a/agents/e2e/reviewer-spawn-conditions.e2e.test.ts +++ b/agents/e2e/reviewer-spawn-conditions.e2e.test.ts @@ -276,6 +276,29 @@ function loadProductionGateFileContentMarker(): (path: string) => string { throw new Error('Unable to find the end of the GATE_FILE_MISSING_CONTENT_MARKER declaration') } const hoistedConstSource = base2JavaScript.slice(hoistStart, hoistEnd + 1) + // Fail-fast structural assertion on the slice BEFORE evaluating: it must be + // exactly the hoisted const declaration — starting with the const name, + // terminated by `;`, and containing exactly one `=` (its initializer). The + // check is quote-style agnostic because the transpiled initializer may use + // either quote. If base2.ts is ever reordered so the const lands after + // readGateFileContentMarker (caught by the hoistStart/inlineFnStart + // ordering guard above) or the const is moved inside the function body + // (caught here when the slice loses declaration shape), these guards throw + // a clear error instead of the `new Function` eval failing with a confusing + // ReferenceError that the parity assertion would only surface as + // `unreadable:unknown` — or worse, a silently wrong sentinel. + const sliceEqualsCount = hoistedConstSource.split('=').length - 1 + if ( + !hoistedConstSource.startsWith( + 'const GATE_FILE_MISSING_CONTENT_MARKER', + ) || + !hoistedConstSource.endsWith(';') || + sliceEqualsCount !== 1 + ) { + throw new Error( + `loadProductionGateFileContentMarker: hoisted GATE_FILE_MISSING_CONTENT_MARKER slice is malformed (layout changed?): ${JSON.stringify(hoistedConstSource.slice(0, 120))}`, + ) + } const fn = new Function( `"use strict";\n${hoistedConstSource}\n${helperSource}\nreturn readGateFileContentMarker`, ) as () => (path: string) => string diff --git a/cli/src/components/__tests__/status-bar.test.tsx b/cli/src/components/__tests__/status-bar.test.tsx index 3979b8b5c3..add9095b80 100644 --- a/cli/src/components/__tests__/status-bar.test.tsx +++ b/cli/src/components/__tests__/status-bar.test.tsx @@ -17,13 +17,42 @@ initializeThemeStore() */ const renderTest = process.env.NODE_ENV === 'production' ? test.skip : test +/** + * Dev/prod act mismatch: `bun test` resolves the dev React build (which warns + * on any update "not wrapped in act(...)") while `@opentui/react/test-utils` + * resolves production React, whose `act` is a throwing stub — so testRender's + * internal act-wrapping silently no-ops and reconciler updates escape capture + * under load. Only an explicit outer act can be trusted; this shim is safe + * under both builds, using React's real act when available (dev) and a + * microtask-flushing passthrough otherwise (production or a missing export). + */ +const actPassthrough = async (callback: () => Promise): Promise => { + await callback() + await Promise.resolve() +} + +const act: (callback: () => Promise) => Promise = + process.env.NODE_ENV === 'production' + ? actPassthrough + : ((React as any).act ?? actPassthrough) + +// React's dev build only honors act() when this flag is set before the first +// render, so set it once at module scope. +;(globalThis as any).IS_REACT_ACT_ENVIRONMENT = true + const renderFrame = async (node: React.ReactNode): Promise => { const { testRender } = await import('@opentui/react/test-utils') - const setup = await testRender( - {node}, - { width: 100, height: 40 }, - ) - await setup.renderOnce() + let setup!: Awaited> + await act(async () => { + setup = await testRender( + {node}, + { width: 100, height: 40 }, + ) + }) + await act(async () => { + await setup.renderOnce() + await Promise.resolve() + }) const frame: string = setup.captureCharFrame() setup.renderer.destroy() return frame From 543bae880bbcaedd1d5d30aeb4201704af1ede87 Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Mon, 14 Sep 2026 11:23:05 +0300 Subject: [PATCH 17/22] docs(cli): document OpenTUI reconciler act wrapping convention in knowledge.md --- cli/knowledge.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/cli/knowledge.md b/cli/knowledge.md index 7d8dacb95a..3bb7631628 100644 --- a/cli/knowledge.md +++ b/cli/knowledge.md @@ -26,6 +26,10 @@ - `/memory` (alias `/mem`, `cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts`) inspects the persisted cross-session task memory for the current project (`.openbuff/memory/task-memory.json`). `/memory status` (the default) reports the record's revision and age, its goal and per-list counts, and how much of its evidence still verifies against disk, listing up to five stale paths; `/memory prune` drops evidence that no longer verifies. Both subcommands are move-aware via `collectWorkspaceMoves` + `WorkspaceJournalService` so a renamed file's evidence rebinds rather than being reported stale and deleted. `/memory audit-migration` is dedicated and read-only: before acquiring V2 it calls `inspectPersistedTaskMemoryV1`, blocks absent/invalid/unreadable states with bounded distinct reports, and passes the exact checksum-verified memory to the SDK audit without a second read. An `exact` audit reconstructs and compares every deterministic source-derived task/observation header and payload plus marker metadata; it is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`. The legacy loader still collapses non-valid inspection states to `undefined` for compatibility. `not-migrated` means removal is not ready and is not a product defect; all outcomes are rendered distinctly with bounded reason text. The audit's exactness check is order-insensitive (canonicalized before comparison) because the SQLite backend persists payloads via key-sorted `stableJson` and re-parses them on export; this is covered end-to-end by `cli/src/services/memory-v2/__tests__/memory-v1-migration-sqlite-roundtrip.test.ts`. - Slash-command descriptions should stay model-agnostic under BYOK/local mode. Use wording such as "configured reviewer" rather than naming hosted models. +## Test Conventions + +- OpenTUI reconciler tests (e.g. `status-bar.test.tsx`) must wrap `testRender` and `renderOnce` calls in an explicit `act()` using the dev/prod-safe shim defined at module scope in the test file. `@opentui/react/test-utils` resolves production React whose built-in `act` is a throwing stub, so the renderer's internal act-wrapping silently no-ops under load. Set `globalThis.IS_REACT_ACT_ENVIRONMENT = true` at module scope before the first render. + ## Import Guidelines **Never use dynamic `await import()` calls.** Always use static imports at the top of the file. From 765f5f8f35ef1df6e51067b13ec883cfe909dfb8 Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Mon, 14 Sep 2026 13:22:06 +0300 Subject: [PATCH 18/22] docs(memory-v2): record R7-T3 harness-limitation blocker and finalize R7 durable artifacts Mark MEM2-R7-T3 BLOCKED (not done) because a committed clean tree mints no gate-assigned v3 fingerprint, so reviewer-family specialists cannot attest the same frozen bundle; record the repro and the proposed committed-surface fix. Refresh PLAN/STATUS headers and current-task pointers, and capture the R7-T4 evidence-substituted disposition plus the specialist-token/clean-tree lesson. --- .../dynamic-cross-session-memory/LESSONS.md | 9 +++++++ .../dynamic-cross-session-memory/PLAN.md | 5 ++-- .../dynamic-cross-session-memory/STATUS.md | 24 +++++++++++++++++-- 3 files changed, 34 insertions(+), 4 deletions(-) diff --git a/.agents/sessions/dynamic-cross-session-memory/LESSONS.md b/.agents/sessions/dynamic-cross-session-memory/LESSONS.md index 0f23b4b04c..82129dba4d 100644 --- a/.agents/sessions/dynamic-cross-session-memory/LESSONS.md +++ b/.agents/sessions/dynamic-cross-session-memory/LESSONS.md @@ -86,3 +86,12 @@ Raw SQLite database bytes are not a valid reopen-idempotence contract under WAL - Export retry idempotency requires EEXIST handling that compares existing content rather than failing deterministically on identical checksums. - finishTurn must be bounded against the run abort signal; an unbounded await on a hung SQLite store can delay run termination indefinitely. - Post-abort tool observations need a generation gate; isCurrent(undefined) returning true unconditionally allows commits after invalidate. + + + +## Specialist-token / clean-tree limitation — 2026-09-14T09:12:00.000Z + +- Reviewer-family specialists (`compatibility-reviewer`, `migration-reviewer`, `reliability-reviewer`, `performance-specialist`, `accessibility-reviewer`, `ux-visual-reviewer`, `dependency-reviewer`, `product-reviewer`, `evaluator`) require `params.snapshot_id` matching `^v3:[a-f0-9]{64}$`. That token is `hashGateSnapshotDetails(files-v4 details of PENDING files)` = `'v3:' + sha256(details)`. With a CLEAN worktree there are no pending files, so the fingerprint is a constant and `gate-state.ts` documents it "never mints a receipt, for any kind" (`'no-diff'` = constant by construction). +- Consequence: R7-T3-style "one frozen bundle, every specialist carries the SAME gate-assigned token" is UNREPRESENTABLE on a fully-committed clean tree. It is a harness-design limitation, not a flaky failure. Only `security-reviewer` (looser `changed_files` + `snapshot_fingerprint` contract) can run post-commit. +- Anti-pattern to avoid: manufacturing a throwaway dirty edit purely to mint a token is evidence theater — the receipts attest the synthetic delta, not the shipped artifact, and go stale on revert. Never launder a same-shaped token into false credit. +- Correct disposition: mark such a task BLOCKED (not done) with a repro; proceed with dependent finalization only under an explicitly-labeled evidence-substituted basis; file the fix (a `committed-surface` snapshot mode minting `v3:<64hex>` deterministically from the committed tree at HEAD, gated to clean worktree + non-empty fileset) as the real unblocker. diff --git a/.agents/sessions/dynamic-cross-session-memory/PLAN.md b/.agents/sessions/dynamic-cross-session-memory/PLAN.md index 95aa02a408..6ac24dd3b1 100644 --- a/.agents/sessions/dynamic-cross-session-memory/PLAN.md +++ b/.agents/sessions/dynamic-cross-session-memory/PLAN.md @@ -1,9 +1,9 @@ # PLAN — Dynamic Cross-Session Memory V2 Repair Session: `dynamic-cross-session-memory` -Status: ready for implementation +Status: implementation complete; R7-T2 green; R7-T3 blocked by harness limitation (see STATUS.md). - + ## Execution rules @@ -95,6 +95,7 @@ Status: ready for implementation - Depends on: MEM2-R7-T2 - Build one fresh bundle and freeze mutations while security, compatibility, migration, reliability, and final code review run. - Acceptance: every reviewer returns a structured non-blocking verdict with the matching fingerprint; quota/protocol failures are retried only against a fresh stable bundle and are never counted as approval. + - Status: BLOCKED by harness design — a fully-committed clean tree mints no gate-assigned `v3:<64-hex>` token, so reviewer-family specialists cannot be spawned against HEAD. See STATUS.md for the repro and the proposed `committed-surface` fingerprint mode. - Validate: record fingerprint and receipt IDs in `STATUS.md`. - [ ] MEM2-R7-T4 Finalize durable artifacts diff --git a/.agents/sessions/dynamic-cross-session-memory/STATUS.md b/.agents/sessions/dynamic-cross-session-memory/STATUS.md index dee28e7037..3553b5e81f 100644 --- a/.agents/sessions/dynamic-cross-session-memory/STATUS.md +++ b/.agents/sessions/dynamic-cross-session-memory/STATUS.md @@ -1,8 +1,8 @@ # STATUS — Dynamic Cross-Session Memory V2 Repair -Status: implementation complete; MEM2-R1-T2 resolved (typed-unsupported secure-open fallback) and validated green. +Status: implementation complete; MEM2-R1-T2 resolved (typed-unsupported secure-open fallback); R7-T2 green; R7-T3 BLOCKED by harness limitation (see R7 gate entry at end). Current phase: R7 — integration / finalization. -Current task: MEM2-R7 — obtain stable exact-snapshot reviews; full race-free WAL/SHM open deferred pending native-addon authorization (option C). +Current task: none active. R7-T3 is blocked pending the proposed `committed-surface` harness fix; R7-T4 durable-artifact finalization is dispositioned below. ## Implemented and locally validated before this plan refresh @@ -193,3 +193,23 @@ Fixed the pre-push hook failure that blocked the gate-improvement push: the test Re-ran MEM2-R7-T2 (package-wide validation + artifact smoke) on the current tree to produce a fresh green baseline before R7-T3. Results: monorepo typecheck 11/11; common 1245 pass; agent-runtime 1618 pass; sdk 1398 pass / 1 skip; cli 3093 pass / 15 skip / 2 fail; evals memory-retention 6/6. SDK build (ESM/CJS/types) and `smoke-test:dist` (CJS require + tree-sitter) passed; CLI binary build + `--version` probe passed. The 2 CLI failures are the known flaky `StatusBar` React-act tests (`renders the status label...` and `hides the scroll control...`), which pass 3/3 in isolation, were untouched by this work, and are unrelated to memory-v2 — a pre-existing flake, not an R7 blocker. R7-T2 acceptance met. + + +## R7-T3 BLOCKED (harness limitation) + R7-T4 disposition — 2026-09-14T09:12:00.000Z + +**R7-T3 "obtain stable exact-snapshot reviews" — status: BLOCKED (not done).** + +The memory-v2 feature is fully committed at HEAD `543bae880` with a clean worktree, so there is no pending gate-file set to fingerprint. Reviewer-family specialists (`compatibility-reviewer`, `migration-reviewer`, `reliability-reviewer`) require `params.snapshot_id` matching `^v3:[a-f0-9]{64}$`, and that token is minted by `hashGateSnapshotDetails(details)` = `'v3:' + sha256(files-v4 details of PENDING files)` (`agents/base2/gate-fingerprint.ts`, `isAttestableSnapshotFingerprint`). With no pending files that fingerprint is a constant, and `agents/base2/gate-state.ts` documents that a `'no-diff'` fingerprint "is a CONSTANT by construction" and that a non-attestable fingerprint "never mints a receipt, for any kind." So R7-T3's own rule — "every specialist spawn must carry the SAME gate-assigned `params.snapshot_id`" — is **unsatisfiable on a committed clean tree by construction**. It is not a flaky failure and not user error. + +**Repro:** (1) `get_change_review_bundle` → `files=["cli/knowledge.md"]` and a bare-hex `snapshotId` that is evidence-only (changes every call, explicitly non-reusable per the params contract). (2) A reviewer-family spawn with `params.snapshot_id` = that bare hex / the truncated display form `v3:131ae03adb957` is rejected as "invalid params" (no legacy-format bypass exists). Only `security-reviewer` uses the looser `changed_files` + `snapshot_fingerprint` contract, which is why it alone ran. + +**Equivalent-evidence package (NOT a frozen-bundle signoff):** +- R7-T2 green package-wide baseline at `543bae880` (above). +- `security-reviewer`: NON_BLOCKING with 2 low advisories (both accepted) — valid under its looser contract; run against the current bundle. +- Prior migration / compatibility / reliability receipts from **earlier dirty-tree gates over these exact files** — they attest to their edit snapshots, not to HEAD-as-shipped. Explicitly labeled as such. +- **No NEW frozen bundle is representable post-commit.** + +**Proposed fix (the real unblocker):** add a `committed-surface` snapshot mode that mints `v3:<64hex>` deterministically from the committed tree at HEAD (enumerate reviewable files at HEAD with content markers; hash via the existing `buildGateSnapshotDetails(files, '')` + `hashGateSnapshotDetails`), gated to be usable only when the worktree is clean and the fileset is non-empty. This makes R7-T3 mechanically satisfiable for post-commit review without weakening the dirty-tree contract — an additive evidence *kind* (like the existing `'reviewed-diff'` vs `'no-diff'`), not a new trust model. **Do not mark R7-T3 done until this lands and a real bundle is frozen.** + +**R7-T4 "finalize durable artifacts" — status: PROCEED, explicitly evidence-substituted.** +PLAN/STATUS/LESSONS now reflect the actual worktree, the R1-T2 supported/disabled platform decision, the R7-T2 receipts, and the deferred default-cutover/native-helper work; the PLAN current-task pointer is cleared. This is **explicitly NOT a frozen-bundle signoff** — it is an equivalent-evidence disposition pending the `committed-surface` fix. Do not claim "all four specialists attested the same snapshot": one did (security); three did not and could not. From 64a934832b1ceb7edd638746a78b54c942a9685a Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Tue, 15 Sep 2026 01:47:48 +0300 Subject: [PATCH 19/22] fix(spawn): omit-for-manual snapshot_id and calm spawn error noise Reviewer-family specialist schemas hard-required params.snapshot_id, but the gate-minted v3 token is only obtainable by runtime-owned programmatic spawns, so agents following caller-facing guidance wrongly spawned reviewer specialists and hit an unfixable validation wall. Make snapshot_id optional with the v3 pattern enforced only when supplied, state the two-mode (runtime-owned vs manual omit) contract consistently across the routing guide, prompt sections, tool description, regenerated type sources, and docs, and branch the recovery hints on whether the caller supplied the key (security-reviewer stays the documented exception). Also emit concise autoRecovering userMessage events on the spawn_agents partial-failure and spawn_agent_inline pre-validation paths so the CLI shows a one-liner instead of the raw multi-KB validation wall; the full contract still reaches the agent's message history. --- .agents/types/tools.ts | 2 +- .../base2-writer-spawn-rules.test.ts | 112 ++++++++- agents/base2/base2.ts | 2 +- agents/guides/specialist-routing.md | 18 +- agents/specialists/create-specialist.ts | 18 +- agents/types/tools.ts | 2 +- .../initial-agent-type-sources.generated.ts | 2 +- common/src/constants/prompt-sections.ts | 8 +- .../initial-agents-dir/types/tools.ts | 2 +- .../__tests__/spawn-agents-schema.test.ts | 13 ++ common/src/tools/params/tool/spawn-agents.ts | 4 +- docs/agents-and-tools.md | 26 +-- .../spawn-agents-permissions.test.ts | 76 ++++++- .../__tests__/tool-validation-error.test.ts | 214 +++++++++++++++++- .../tools/handlers/tool/spawn-agent-utils.ts | 20 +- .../agent-runtime/src/tools/tool-executor.ts | 21 +- 16 files changed, 469 insertions(+), 71 deletions(-) diff --git a/.agents/types/tools.ts b/.agents/types/tools.ts index 32cd35f189..24a5052ac8 100644 --- a/.agents/types/tools.ts +++ b/.agents/types/tools.ts @@ -979,7 +979,7 @@ export interface SpawnAgentsParams { push?: boolean /** Remote used for fetch/push (git-committer) */ remote?: string - /** Assigned gate snapshot fingerprint (reviewer specialists) */ + /** Optional gate-assigned snapshot token (reviewer specialists). Runtime-owned spawns pass the gate-assigned v3:… token; manual spawns omit this key entirely. */ snapshot_id?: string /** Changed file paths to review (security-reviewer) */ changed_files?: string[] diff --git a/agents/__tests__/base2-writer-spawn-rules.test.ts b/agents/__tests__/base2-writer-spawn-rules.test.ts index b5cc95c1e4..994f1d062b 100644 --- a/agents/__tests__/base2-writer-spawn-rules.test.ts +++ b/agents/__tests__/base2-writer-spawn-rules.test.ts @@ -4,6 +4,7 @@ import { describe, expect, test } from 'bun:test' import { createBase2 } from '../base2/base2' import { isTestCoverageReviewerFinding } from '../base2/gate-reviewer' +import { specialistRoutingSection } from '../base2/quality-prompt-section' import { createReviewer } from '../reviewer/code-reviewer' import { createSpecialist } from '../specialists/create-specialist' import { editReceipt } from './helpers/base2-step-fixtures' @@ -797,7 +798,7 @@ describe('editor / repair-editor / test-writer cohesion', () => { expect(specialist.toolNames).toContain('set_output') }) - test('non-advisory createSpecialist requires attestable v3 snapshot_id pattern', () => { + test('non-advisory createSpecialist keeps attestable v3 snapshot_id pattern when supplied but omits the key from required', () => { const specialist = createSpecialist({ id: 'compatibility-reviewer', displayName: 'Compatibility Reviewer', @@ -814,6 +815,113 @@ describe('editor / repair-editor / test-writer cohesion', () => { expect(paramsSchema.properties?.snapshot_id?.pattern).toBe( '^v3:[a-f0-9]{64}$', ) - expect(paramsSchema.required).toContain('snapshot_id') + // snapshot_id is optional: manual spawns supply only params.files and + // still validate; the v3 pattern is enforced when the key IS supplied. + expect(paramsSchema.required).not.toContain('snapshot_id') + }) + + test('non-advisory createSpecialist instructionsPrompt treats an absent snapshot_id as the omit-for-manual contract, not a stale-snapshot failure', () => { + const specialist = createSpecialist({ + id: 'migration-reviewer', + displayName: 'Migration Reviewer', + purpose: 'Review schema and data migrations.', + focus: ['Migration safety'], + }) + const instructions = specialist.instructionsPrompt + // The omitted-key path is the legalized manual contract: an absent + // snapshot_id must no longer be listed among the stale-snapshot BLOCKING + // triggers (that would make every manual-omit spawn self-defeating). + expect(instructions).not.toContain('missing/empty snapshot_id') + expect(instructions).toContain( + 'not live-bundle drift during review and not an absent snapshot_id on a manual spawn', + ) + // Manual-omit directive: echo an empty snapshotFingerprint rather than + // minting a token the caller cannot have. + expect(instructions).toContain( + 'emit snapshotFingerprint as the empty string', + ) + expect(instructions).toContain('never invent a v3 token') + // Protocol-failure rubric: an absent snapshot_id on a manual spawn is not + // a protocol failure. + expect(instructions).toContain( + 'An absent snapshot_id on a manual spawn is the documented omit-for-manual contract, not a protocol failure', + ) + // The spawner-facing prompt keeps the omit-for-manual directive. + expect(specialist.spawnerPrompt).toContain( + 'manual spawns omit params.snapshot_id entirely', + ) + }) +}) + +describe('caller-facing omit-for-manual guidance surfaces (RF-1-55de1919)', () => { + // Every surface a manual spawner might consult must state the omit-for-manual + // contract: no caller-facing guidance may tell an agent to require, hunt for, + // or supply a gate-owned snapshot token it cannot obtain. + const base2Source = readFileSync( + new URL('../base2/base2.ts', import.meta.url), + 'utf8', + ) + const routingGuide = readFileSync( + new URL('../guides/specialist-routing.md', import.meta.url), + 'utf8', + ) + + test('specialist-routing guide states the omit-for-manual contract (no manual token requirement)', () => { + // Runtime-owned vs manual split is stated explicitly in the Params Contract. + expect(routingGuide).toContain( + 'manual spawns omit `params.snapshot_id` entirely', + ) + expect(routingGuide).toContain( + 'Manual/advisory reviewer-family spawns must OMIT `params.snapshot_id` entirely.', + ) + // security-reviewer keeps its schema-required fingerprint on manual + // spawns; the guide must document that exception so the manual pre-edit + // security-review path stays usable under the omit-for-manual contract. + expect(routingGuide).toContain( + '`security-reviewer` is the documented exception', + ) + expect(routingGuide).toContain( + 'its schema still requires `params.changed_files` + `params.snapshot_fingerprint` on manual spawns', + ) + // The stale pre-migration claim (snapshot-scoped reviewers require the + // gate-owned token) must not survive in any form. + expect(routingGuide).not.toMatch( + /reviewers? require[^\n]{0,120}snapshot_id/i, + ) + }) + + test('specialistRoutingSection prompt section states the omit-for-manual contract', () => { + expect(specialistRoutingSection).toContain( + 'omitted entirely for manual spawns', + ) + expect(specialistRoutingSection).toContain( + 'For manual/advisory reviewer-family spawns omit `params.snapshot_id`', + ) + // The section must state the security-reviewer exception so a manual + // caller knows the schema-required fingerprint is still passed on + // manual spawns (with a caller-supplied stable value). + expect(specialistRoutingSection).toContain( + 'security-reviewer is the documented exception', + ) + expect(specialistRoutingSection).toContain( + 'its schema still requires `params.changed_files` + `params.snapshot_fingerprint` on manual spawns too', + ) + expect(specialistRoutingSection).not.toMatch( + /reviewer-family specialists require params\.snapshot_id/i, + ) + }) + + test('base2 specialistRoutingPointer degraded clause states the omit-for-manual contract', () => { + expect(base2Source).toContain( + 'never include `snapshot_id` in a manually authored reviewer-family spawn', + ) + expect(base2Source).toContain( + 'the gate-assigned token is only available to runtime-owned spawns', + ) + // The degraded clause keeps the security-reviewer schema requirement + // visible so a manual pre-edit security spawn still supplies both keys. + expect(base2Source).toContain( + 'security-reviewer still requires `changed_files` + `snapshot_fingerprint` on manual spawns', + ) }) }) diff --git a/agents/base2/base2.ts b/agents/base2/base2.ts index c7fd3b07d7..a7c0521711 100644 --- a/agents/base2/base2.ts +++ b/agents/base2/base2.ts @@ -65,7 +65,7 @@ const BROAD_AUDIT_POINTER_TAILS: Record = { ' In plan mode, do not implement — translate the findings into the durable plan packet instead.', } const specialistRoutingPointer = - 'Choosing a specialist agent → read_files `agents/guides/specialist-routing.md`. If that guide is unavailable, route only on a crossed risk boundary (architecture, requirements, performance, reliability, migration, compatibility, accessibility, dependencies), pass the gate-assigned `params.snapshot_id`, and never substitute a specialist for the runtime-owned final gate.' + 'Choosing a specialist agent → read_files `agents/guides/specialist-routing.md`. If that guide is unavailable, route only on a crossed risk boundary (architecture, requirements, performance, reliability, migration, compatibility, accessibility, dependencies), never include `snapshot_id` in a manually authored reviewer-family spawn (the gate-assigned token is only available to runtime-owned spawns; put scoped files in params and the question in the prompt; security-reviewer still requires `changed_files` + `snapshot_fingerprint` on manual spawns), and never substitute a specialist for the runtime-owned final gate.' const gitDisciplinePointer = 'Before any git commit/branch/push → read_files `agents/guides/git-discipline.md`. If that guide is unavailable, apply the standard git rules: delegate to `git-committer` with `params.owned_paths`, commit only after GATE: PASSED, never push or alter git config unless explicitly asked, and never commit secrets.' // The named guide is advisory routing only (when to ask for a pre-edit diff --git a/agents/guides/specialist-routing.md b/agents/guides/specialist-routing.md index 0f265f7616..353244b169 100644 --- a/agents/guides/specialist-routing.md +++ b/agents/guides/specialist-routing.md @@ -53,26 +53,28 @@ Ownership and timing — Final Gate always runs last; specialist gates are scope ## Params Contract -Pass the exact params contract or the spawn fails. Do not substitute the bare hex `snapshotId` from `get_change_review_bundle` — reviewer-family requires the opaque `v3:<64-hex>` token from the parent gate. +`params.snapshot_id` is optional in the reviewer-family schema, but only one mode is valid per spawn. Runtime-owned programmatic spawns pass the gate-assigned opaque `v3:<64-hex>` token; manual spawns omit `params.snapshot_id` entirely. Never substitute the bare hex `snapshotId` from `get_change_review_bundle` — when the key is supplied it must be the opaque `v3:<64-hex>` token from the parent gate. | Specialist family | Required `params` | On mismatch | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | ----------------------------------------------------------------------- | -| Reviewer-family (`product-reviewer`, `performance-specialist`, `reliability-reviewer`, `migration-reviewer`, `compatibility-reviewer`, `accessibility-reviewer`, `ux-visual-reviewer`, `dependency-reviewer`, `evaluator`) | `params.snapshot_id` = `v3:<64-hex>` (opaque gate token) | Spawn fails: missing or wrong key, or bare hex instead of `v3:<64-hex>` | -| `security-reviewer` (exception) | `params.changed_files` + `params.snapshot_fingerprint` | Spawn fails; does not accept `params.snapshot_id` | +| Reviewer-family (`product-reviewer`, `performance-specialist`, `reliability-reviewer`, `migration-reviewer`, `compatibility-reviewer`, `accessibility-reviewer`, `ux-visual-reviewer`, `dependency-reviewer`, `evaluator`) | `params.snapshot_id` = `v3:<64-hex>` (opaque gate token) for runtime-owned spawns; omitted entirely for manual spawns | Spawn fails: when supplied, a wrong value or bare hex instead of `v3:<64-hex>` fails; manual spawns that supply any value fail | +| `security-reviewer` (exception) | `params.changed_files` + `params.snapshot_fingerprint` (required on manual spawns too) | Spawn fails; does not accept `params.snapshot_id`; omitting the schema-required `snapshot_fingerprint` also fails | + +Manual/advisory reviewer-family spawns must OMIT `params.snapshot_id` entirely. The gate-assigned `v3:<64-hex>` token is only minted for runtime-owned programmatic spawns; a prompt-authored spawn call cannot obtain one, and display fingerprints in gate blocks or telemetry are truncated 16-char prefixes that never satisfy `^v3:[a-f0-9]{64}$`. No caller-facing guidance may tell a manual reviewer-family spawner to require, hunt for, or supply a gate-owned token: manual caller guidance always states the omit-for-manual contract (put the scoped file list in `params.files` and the review question in the prompt). `security-reviewer` is the documented exception: its schema still requires `params.changed_files` + `params.snapshot_fingerprint` on manual spawns, so a manual caller passes both keys and supplies as `snapshot_fingerprint` the stable fingerprint value it wants echoed exactly — the schema imposes no `^v3:` pattern on that key, so the caller never needs a gate-owned token; omitting the key fails the spawn. ## Example spawns ```text -# reviewer-family (advisory pre-edit) — requires gate token +# reviewer-family (advisory pre-edit) — no token is passed on manual spawns spawn product-reviewer - params.snapshot_id: "v3:<64-hex>" # opaque token from parent gate, not bare hex snapshotId + params.files: ["src/example.ts"] # snapshot_id omitted entirely on manual spawns; the gate-assigned v3 token is only minted for runtime-owned spawns ``` ```text -# security-reviewer (exception) — requires files + fingerprint +# security-reviewer (exception) — the schema-required fingerprint is passed on manual spawns too spawn security-reviewer params.changed_files: ["src/auth/login.ts", "src/auth/session.ts"] - params.snapshot_fingerprint: "" # never snapshot_id + params.snapshot_fingerprint: "" # schema-required on every spawn; no v3 pattern is imposed, so a manual caller supplies its own stable value ``` ```text @@ -85,7 +87,7 @@ spawn_agents [ ## Compaction recovery -After `context-pruner` / compaction the prior bundle hex is stale. Recompute the gate fingerprint from the fresh `get_change_review_bundle` and re-derive `v3:<64-hex>` before any manual specialist spawn. Do not reuse a stale bundle hex or a pre-compaction `snapshot_id` — the gate will reject it and the finding will not attest to the current pending set. +After `context-pruner` / compaction do not re-derive or re-mint any `v3:<64-hex>` token for a manual specialist spawn — the gate-assigned token is only available to runtime-owned programmatic spawns, and hand-rolled recomputation never attests. Do not manually re-spawn reviewer-family specialists: wait for the runtime-owned Final Gate result. A manual reviewer-family spawn, when explicitly requested, must omit `params.snapshot_id` entirely; never pass a stale bundle hex, a stale pre-compaction `snapshot_id`, or a truncated display prefix. `security-reviewer` keeps its schema-required `params.snapshot_fingerprint` on manual spawns; supply a stable value to echo rather than re-deriving a gate-owned token. ## Sequential vs parallel diff --git a/agents/specialists/create-specialist.ts b/agents/specialists/create-specialist.ts index ed9abc69d6..1ef22749d7 100644 --- a/agents/specialists/create-specialist.ts +++ b/agents/specialists/create-specialist.ts @@ -83,7 +83,7 @@ export function createSpecialist( displayName: config.displayName, spawnerPrompt: config.advisory ? config.purpose - : `${config.purpose} Requires params.snapshot_id with the assigned gate snapshot fingerprint for this spawn.`, + : `${config.purpose} When supplied, params.snapshot_id must be the assigned gate snapshot fingerprint (gate-assigned opaque v3 token) for this spawn; manual spawns omit params.snapshot_id entirely.`, inputSchema: { prompt: { type: 'string', @@ -101,13 +101,15 @@ export function createSpecialist( snapshot_id: { type: 'string', maxLength: 512, - // Non-advisory spawns require a gate-assigned v3 token. Advisory - // may omit the field; do not apply the pattern when optional so an - // empty advisory value is not forced into a hard schema reject. + // snapshot_id is OPTIONAL: manual spawns omit the key entirely and + // validate; runtime-owned spawns pass the gate-assigned v3 token. + // The pattern is enforced only when the key is supplied. Do not + // apply the pattern when advisory so an empty advisory value is + // not forced into a hard schema reject. ...(config.advisory ? {} : { pattern: '^v3:[a-f0-9]{64}$' }), description: config.advisory ? 'Optional assigned gate snapshot fingerprint for this spawn — when present, the gate-assigned opaque v3:… token from the parent gate (not bare hex from get_change_review_bundle.snapshotId). Echo it exactly as snapshotFingerprint.' - : 'Required assigned gate snapshot fingerprint for this spawn — gate-assigned opaque v3:<64-hex> token from the parent gate (params.snapshot_id / specialistCreditFingerprint), not bare hex from get_change_review_bundle.snapshotId. Echo it exactly as snapshotFingerprint; do not invent a different value.', + : 'Optional assigned gate snapshot fingerprint for this spawn — when supplied, the gate-assigned opaque v3:<64-hex> token from the parent gate (params.snapshot_id / specialistCreditFingerprint), not bare hex from get_change_review_bundle.snapshotId. Manual spawns omit this key entirely; runtime-owned spawns echo it exactly as snapshotFingerprint.', }, command: { type: 'string', @@ -116,7 +118,7 @@ export function createSpecialist( 'Optional bounded diagnostic command for terminal-enabled specialists.', }, }, - required: config.advisory ? [] : ['snapshot_id'], + required: [], }, }, outputMode: 'structured_output', @@ -237,7 +239,7 @@ export function createSpecialist( instructionsPrompt: [ config.advisory ? 'Read the exact current sources and task state. A snapshot_id is optional for pre-edit advisory work; when supplied, it is the assigned gate snapshot fingerprint for this spawn — echo it exactly as snapshotFingerprint and do not invent a different value or re-validate against a live bundle that may have moved.' - : 'params.snapshot_id is the authoritative assigned snapshot for this review spawn (opaque v3:… gate-assigned token from the parent gate — not the bare hex snapshotId from get_change_review_bundle). Echo that exact value as snapshotFingerprint. You may use get_change_review_bundle as read-only evidence (file list/diff/empty-tree check); if a fresh call returns a different bare id, keep reviewing against params.snapshot_id and echo params.snapshot_id — do not emit stale-snapshot solely because the live bundle moved. List the exact normalized project-relative paths you read. Stale-snapshot BLOCKING is only for: missing/empty snapshot_id, inventing a different fingerprint, or inability to read the assigned files — not live-bundle drift during review.', + : 'When supplied, params.snapshot_id is the authoritative assigned snapshot for this review spawn (opaque v3:… gate-assigned token from the parent gate — not the bare hex snapshotId from get_change_review_bundle). Echo that exact value as snapshotFingerprint. When params.snapshot_id is absent, this is a manual spawn following the omit-for-manual contract (the gate-assigned token exists only for runtime-owned spawns): review the files in params.files and the prompt scope, emit snapshotFingerprint as the empty string, and never invent a v3 token. You may use get_change_review_bundle as read-only evidence (file list/diff/empty-tree check); if a fresh call returns a different bare id, keep reviewing against params.snapshot_id and echo params.snapshot_id — do not emit stale-snapshot solely because the live bundle moved. List the exact normalized project-relative paths you read. Stale-snapshot BLOCKING is only for: inventing a different fingerprint or inability to read the assigned files — not live-bundle drift during review and not an absent snapshot_id on a manual spawn.', config.advisory ? 'Return family=advisory. Your output is design/coordination evidence; do not invent a blocking gate verdict and do not mutate files or external systems.' : 'Return family=reviewer. Any material issue requiring a code or contract change is BLOCKING. Only enumerate requirementCoverage for in-scope review requirements this specialist can judge from source/diff evidence; omit parent/orchestrator workflow (git rewrite, validation runs, commit/push, CI green). Never mark those missing/uncertain.', @@ -246,7 +248,7 @@ export function createSpecialist( config.terminal ? 'Use only the tools exposed for this specialist. run_terminal_command is available only for the optional bounded diagnostic command; do not call a basher agent.' : 'Use only the tools exposed for this specialist. Do not call basher or run terminal validation; if runtime evidence is required, report the exact missing evidence for the parent to collect.', - `Use these exact dimension keys: ${dimensionKeys.join(', ')}. Every finding ID must be stable and formatted ${config.id}::; include severity, concrete evidence, and an actionable correction. Only emit findings that require a concrete code or contract change; do not emit informational observations about intended or documented behavior (e.g. 'this is the intended scope, not a defect'). Deleting mocks, fixtures, test doubles, stubs, or other test-only scaffolding is intended cleanup, not a defect: do not emit any finding (of any severity) solely because such a file was deleted; only flag a deletion when production source or a genuine public contract was removed without cleaning up its references. Keep the result compact: at most ${MAX_FINDINGS} findings and ${MAX_EVIDENCE_ITEMS} evidence items per finding. Snapshot/file-attestation protocol failures (missing/empty snapshot_id, invented fingerprint, or inability to read assigned files) are not source findings; report a stale-snapshot finding and do not invent a repair. Do not treat live get_change_review_bundle drift as stale-snapshot. Call set_output with a JSON object directly; never JSON.stringify the object or wrap it in a string. Return the required structured output and do not modify files.`, + `Use these exact dimension keys: ${dimensionKeys.join(', ')}. Every finding ID must be stable and formatted ${config.id}::; include severity, concrete evidence, and an actionable correction. Only emit findings that require a concrete code or contract change; do not emit informational observations about intended or documented behavior (e.g. 'this is the intended scope, not a defect'). Deleting mocks, fixtures, test doubles, stubs, or other test-only scaffolding is intended cleanup, not a defect: do not emit any finding (of any severity) solely because such a file was deleted; only flag a deletion when production source or a genuine public contract was removed without cleaning up its references. Keep the result compact: at most ${MAX_FINDINGS} findings and ${MAX_EVIDENCE_ITEMS} evidence items per finding. Snapshot/file-attestation protocol failures (an invented fingerprint or inability to read assigned files) are not source findings; report a stale-snapshot finding and do not invent a repair. An absent snapshot_id on a manual spawn is the documented omit-for-manual contract, not a protocol failure. Do not treat live get_change_review_bundle drift as stale-snapshot. Call set_output with a JSON object directly; never JSON.stringify the object or wrap it in a string. Return the required structured output and do not modify files.`, ].join('\n'), } } diff --git a/agents/types/tools.ts b/agents/types/tools.ts index 32cd35f189..24a5052ac8 100644 --- a/agents/types/tools.ts +++ b/agents/types/tools.ts @@ -979,7 +979,7 @@ export interface SpawnAgentsParams { push?: boolean /** Remote used for fetch/push (git-committer) */ remote?: string - /** Assigned gate snapshot fingerprint (reviewer specialists) */ + /** Optional gate-assigned snapshot token (reviewer specialists). Runtime-owned spawns pass the gate-assigned v3:… token; manual spawns omit this key entirely. */ snapshot_id?: string /** Changed file paths to review (security-reviewer) */ changed_files?: string[] diff --git a/cli/src/data/initial-agent-type-sources.generated.ts b/cli/src/data/initial-agent-type-sources.generated.ts index 761860bf8f..be027cae4d 100644 --- a/cli/src/data/initial-agent-type-sources.generated.ts +++ b/cli/src/data/initial-agent-type-sources.generated.ts @@ -6,6 +6,6 @@ export const agentDefinitionSource = "/**\n * Openbuff Agent Type Definitions\n *\n * This file provides TypeScript type definitions for creating custom Openbuff agents.\n * Import these types in your agent files to get full type safety and IntelliSense.\n *\n * Usage in .agents/your-agent.ts:\n * import { AgentDefinition, ToolName, ModelName } from './types/agent-definition'\n *\n * const definition: AgentDefinition = {\n * // ... your agent configuration with full type safety ...\n * }\n *\n * export default definition\n */\n\n// ============================================================================\n// Agent Definition and Utility Types\n// ============================================================================\n\nexport interface AgentDefinition {\n /** Unique identifier for this agent. Must contain only lowercase letters, numbers, and hyphens, e.g. 'code-reviewer' */\n id: string\n\n /** Version string (if not provided, will default to '0.0.1' and be bumped on each publish) */\n version?: string\n\n /** Publisher ID for the agent. Must be provided if you want to publish the agent. */\n publisher?: string\n\n /** Human-readable name for the agent */\n displayName: string\n\n /**\n * AI model to use for this agent. Can be any model in OpenRouter: https://openrouter.ai/models\n *\n * Optional: if omitted, the model is resolved entirely from the user's openbuff.json via\n * `agents[agentId]` or `defaultModel`. An error is thrown at runtime if neither is configured.\n */\n model?: ModelName\n\n /** Maximum subagent nesting depth. Defaults to the runtime limit. */\n maxSpawnDepth?: number\n\n /**\n * https://openrouter.ai/docs/use-cases/reasoning-tokens\n * One of `max_tokens` or `effort` is required.\n * If `exclude` is true, reasoning will be removed from the response. Default is false.\n */\n reasoningOptions?: {\n enabled?: boolean\n exclude?: boolean\n } & (\n | {\n max_tokens: number\n }\n | {\n effort: 'high' | 'medium' | 'low' | 'minimal' | 'none'\n }\n )\n\n /**\n * Provider routing options for OpenRouter.\n * Controls which providers to use and fallback behavior.\n * See https://openrouter.ai/docs/features/provider-routing\n */\n providerOptions?: {\n /**\n * List of provider slugs to try in order (e.g. [\"anthropic\", \"openai\"])\n */\n order?: string[]\n /**\n * Whether to allow backup providers when primary is unavailable (default: true)\n */\n allow_fallbacks?: boolean\n /**\n * Only use providers that support all parameters in your request (default: false)\n */\n require_parameters?: boolean\n /**\n * Control whether to use providers that may store data\n */\n data_collection?: 'allow' | 'deny'\n /**\n * List of provider slugs to allow for this request\n */\n only?: string[]\n /**\n * List of provider slugs to skip for this request\n */\n ignore?: string[]\n /**\n * List of quantization levels to filter by (e.g. [\"int4\", \"int8\"])\n */\n quantizations?: Array<\n | 'int4'\n | 'int8'\n | 'fp4'\n | 'fp6'\n | 'fp8'\n | 'fp16'\n | 'bf16'\n | 'fp32'\n | 'unknown'\n >\n /**\n * Sort providers by price, throughput, or latency\n */\n sort?: 'price' | 'throughput' | 'latency'\n /**\n * Maximum pricing you want to pay for this request\n */\n max_price?: {\n prompt?: number | string\n completion?: number | string\n image?: number | string\n audio?: number | string\n request?: number | string\n }\n }\n\n /**\n * Optional per-run cost cap in US cents. When set, the agent runtime\n * enforces this as a hard spend ceiling — the turn ends if cumulative\n * creditsUsed exceeds it. Useful for BYOK configurations to guard\n * against runaway spend. Undefined = no cap.\n */\n maxCostCents?: number\n\n /**\n * Optional per-step input token cap. When set, the agent runtime ends\n * the turn if a single step's total input tokens exceed this threshold.\n * Undefined = no cap.\n */\n maxTokensPerTurn?: number\n\n // ============================================================================\n // Tools and Subagents\n // ============================================================================\n\n /** MCP servers by name. Names cannot contain `/`. */\n mcpServers?: Record\n\n /**\n * Tools this agent can use.\n *\n * By default, all tools are available from any specified MCP server. In\n * order to limit the tools from a specific MCP server, add the tool name(s)\n * in the format `'mcpServerName/toolName1'`, `'mcpServerName/toolName2'`,\n * etc.\n */\n toolNames?: (ToolName | (string & {}))[]\n\n /** Tools callable only from `handleSteps`; these are hidden from the model. */\n programmaticToolNames?: (ToolName | (string & {}))[]\n /**\n * Controls whether every spawnable agent is exposed as a separate native\n * tool (`direct`) or only through the generic `spawn_agents` tool\n * (`generic`). Defaults to `direct` for compatibility.\n */\n spawnableAgentToolMode?: 'direct' | 'generic'\n\n /** Enforced shell capability for this agent. Defaults to workspace-write. */\n terminalPermissionProfile?:\n | 'read-only'\n | 'librarian-read-only'\n | 'git-commit'\n | 'dependency-mutation'\n | 'validation-diagnosis'\n | 'tmux-test'\n | 'workspace-write'\n | 'full-access'\n /** Runtime-enforced project-relative glob allowlists for filesystem tools. */\n filesystemScope?: {\n read?: string[]\n write?: string[]\n }\n programmaticConfig?: Record\n\n /** Other agents this agent can spawn, like 'openbuff/file-picker@0.0.1'.\n *\n * Use the fully qualified agent id from the agent store, including publisher and version, for example: 'openbuff/file-picker@0.0.1'\n * (publisher and version are required!)\n *\n * Or, use the agent id from a local agent file in your .agents directory: 'file-picker'.\n */\n spawnableAgents?: string[]\n\n // ============================================================================\n // Input and Output\n // ============================================================================\n\n /** The input schema required to spawn the agent. Provide a prompt string and/or a params object or none.\n * 80% of the time you want just a prompt string with a description:\n * inputSchema: {\n * prompt: { type: 'string', description: 'A description of what info would be helpful to the agent' }\n * }\n */\n inputSchema?: {\n prompt?: { type: 'string'; description?: string }\n params?: JsonObjectSchema\n }\n\n /** How the agent should output a response to its parent (defaults to 'last_message')\n *\n * last_message: The last message from the agent, typically after using tools.\n *\n * all_messages: All messages from the agent, including tool calls and results.\n *\n * structured_output: Make the agent output a JSON object. Can be used with outputSchema or without if you want freeform json output.\n */\n outputMode?: 'last_message' | 'all_messages' | 'structured_output'\n\n /** JSON schema for structured output (when outputMode is 'structured_output') */\n outputSchema?: JsonObjectSchema\n\n // ============================================================================\n // Prompts\n // ============================================================================\n\n /** Prompt for when and why to spawn this agent. Include the main purpose and use cases.\n *\n * This field is key if the agent is intended to be spawned by other agents. */\n spawnerPrompt?: string\n\n /** Whether to include conversation history from the parent agent in context.\n *\n * Defaults to false.\n * Use this when the agent needs to know all the previous messages in the conversation.\n */\n includeMessageHistory?: boolean\n /** Bounded parent-history transfer policy. Defaults from includeMessageHistory. */\n messageHistoryMode?: 'none' | 'pinned' | 'full'\n /** Explicit capability for inline history-editor agents. Defaults to false. */\n propagateMessageHistoryChanges?: boolean\n\n /** Whether to append model reasoning chunks to this agent's message history.\n *\n * Defaults to false for better prompt-cache stability. Enable only when an\n * agent explicitly needs its hidden reasoning replayed on later turns.\n */\n includeReasoningInMessageHistory?: boolean\n\n /** Whether to inherit the parent agent's system prompt instead of using this agent's own systemPrompt.\n *\n * Defaults to false.\n * Use this when you want to enable prompt caching by preserving the same system prompt prefix.\n * Cannot be used together with the systemPrompt field.\n */\n inheritParentSystemPrompt?: boolean\n\n /** Background information for the agent. Fairly optional. Prefer using instructionsPrompt for agent instructions. */\n systemPrompt?: string\n\n /** Instructions for the agent.\n *\n * IMPORTANT: Updating this prompt is the best way to shape the agent's behavior.\n * This prompt is inserted after each user input. */\n instructionsPrompt?: string\n\n /** Prompt inserted at each agent step.\n *\n * Powerful for changing the agent's behavior, but usually not necessary for smart models.\n * Prefer instructionsPrompt for most instructions. */\n stepPrompt?: string\n\n // ============================================================================\n // Handle Steps\n // ============================================================================\n\n /** Programmatically step the agent forward and run tools.\n *\n * You can either yield:\n * - A tool call object with toolName and input properties.\n * - 'STEP' to run agent's model and generate one assistant message.\n * - 'STEP_ALL' to run the agent's model until it uses the end_turn tool or stops includes no tool calls in a message.\n *\n * Or use 'return' to end the turn.\n *\n * Example 1:\n * function* handleSteps({ agentState, prompt, params, logger }) {\n * logger.info('Starting file read process')\n * const { toolResult } = yield {\n * toolName: 'read_files',\n * input: { paths: ['file1.txt', 'file2.txt'] }\n * }\n * yield 'STEP_ALL'\n *\n * // Optionally do a post-processing step here...\n * logger.info('Files read successfully, setting output')\n * yield {\n * toolName: 'set_output',\n * input: {\n * output: 'The files were read successfully.',\n * },\n * }\n * }\n *\n * Example 2:\n * handleSteps: function* ({ agentState, prompt, params, logger }) {\n * while (true) {\n * logger.debug('Spawning thinker agent')\n * yield {\n * toolName: 'spawn_agents',\n * input: {\n * agents: [\n * {\n * agent_type: 'thinker',\n * prompt: 'Think deeply about the user request',\n * },\n * ],\n * },\n * }\n * const { stepsComplete } = yield 'STEP'\n * if (stepsComplete) break\n * }\n * }\n */\n handleSteps?: (context: AgentStepContext) => Generator<\n ToolCall | 'STEP' | 'STEP_ALL' | StepText | GenerateN,\n void,\n {\n agentState: AgentState\n toolResult: ToolResultOutput[] | undefined\n stepsComplete: boolean\n nResponses?: string[]\n }\n >\n}\n\n// ============================================================================\n// Supporting Types\n// ============================================================================\n\nexport interface AgentState {\n agentId: string\n runId: string\n parentId: string | undefined\n\n /** The agent's conversation history: messages from the user and the assistant. */\n messageHistory: Message[]\n\n /** The last value set by the set_output tool. This is a plain object or undefined if not set. */\n output: Record | undefined\n\n /** The system prompt for this agent. */\n systemPrompt: string\n\n /** The tool definitions for this agent. */\n toolDefinitions: Record<\n string,\n { description: string | undefined; inputSchema: {} }\n >\n\n /**\n * The token count from the Anthropic API.\n * This is updated on every agent step via the /api/v1/token-count endpoint.\n */\n contextTokenCount: number\n\n /** Context window resolved from the active model/provider, when known. */\n contextWindowTokens?: number\n\n /** Runtime-owned orchestrator state preserved independently of messages. */\n base2ActiveWork?: Record\n}\n\n/**\n * Context provided to handleSteps generator function\n */\nexport interface AgentStepContext {\n agentState: AgentState\n prompt?: string\n params?: Record\n logger: Logger\n config?: Record\n}\n\nexport type StepText = { type: 'STEP_TEXT'; text: string }\nexport type GenerateN = { type: 'GENERATE_N'; n: number }\n\n/**\n * Tool call object for handleSteps generator\n */\nexport type ToolCall = {\n [K in T]: {\n toolName: K\n input: GetToolParams\n includeToolCall?: boolean\n }\n}[T]\n\n// ============================================================================\n// Available Tools\n// ============================================================================\n\n/**\n * File operation tools\n */\nexport type FileEditingTools = 'read_files' | 'write_file' | 'str_replace'\n\n/**\n * Code analysis tools\n */\nexport type CodeAnalysisTools = 'code_search' | 'find_files' | 'read_files'\n\n/**\n * Terminal and system tools\n */\nexport type TerminalTools = 'run_terminal_command' | 'code_search'\n\n/**\n * Web and browser tools\n */\nexport type WebTools = 'web_search' | 'read_docs'\n\n/**\n * Agent management tools\n */\nexport type AgentTools = 'spawn_agents'\n\n/**\n * Output and control tools\n */\nexport type OutputTools = 'set_output'\n\n// ============================================================================\n// Available Models (see: https://openrouter.ai/models)\n// ============================================================================\n\n/**\n * AI models available for agents. Pick from our selection of recommended models or choose any model in OpenRouter.\n *\n * See available models at https://openrouter.ai/models\n */\nexport type ModelName =\n // Recommended Models\n\n // OpenAI\n | 'openai/gpt-5.5'\n | 'openai/gpt-5.4'\n | 'openai/gpt-5.4-mini'\n | 'openai/gpt-5.4-nano'\n | 'openai/gpt-5.3'\n | 'openai/gpt-5.3-codex'\n | 'openai/gpt-5.2'\n | 'openai/gpt-5.2-chat-latest'\n | 'openai/gpt-5.1'\n | 'openai/gpt-5.1-chat'\n\n // Anthropic\n | 'anthropic/claude-sonnet-4.6'\n | 'anthropic/claude-opus-4.7'\n | 'anthropic/claude-opus-4.6'\n | 'anthropic/claude-opus-4.5'\n | 'anthropic/claude-haiku-4.5'\n | 'anthropic/claude-sonnet-4.5'\n | 'anthropic/claude-opus-4.1'\n\n // Gemini\n | 'google/gemini-3.1-pro-preview'\n | 'google/gemini-3-pro-preview'\n | 'google/gemini-3-flash-preview'\n | 'google/gemini-3.1-flash-lite-preview'\n | 'google/gemini-2.5-pro'\n | 'google/gemini-2.5-flash'\n | 'google/gemini-2.5-flash-lite'\n\n // X-AI\n | 'x-ai/grok-4-fast'\n | 'x-ai/grok-4.1-fast'\n | 'x-ai/grok-code-fast-1'\n\n // Qwen\n | 'qwen/qwen3-max'\n | 'qwen/qwen3-coder-plus'\n | 'qwen/qwen3-coder'\n | 'qwen/qwen3-coder:nitro'\n | 'qwen/qwen3-coder-flash'\n | 'qwen/qwen3-235b-a22b-2507'\n | 'qwen/qwen3-235b-a22b-2507:nitro'\n | 'qwen/qwen3-235b-a22b-thinking-2507'\n | 'qwen/qwen3-235b-a22b-thinking-2507:nitro'\n | 'qwen/qwen3-30b-a3b'\n | 'qwen/qwen3-30b-a3b:nitro'\n\n // DeepSeek\n | 'deepseek/deepseek-v4-pro'\n | 'deepseek-v4-pro'\n | 'deepseek/deepseek-v4-flash'\n | 'deepseek-v4-flash'\n | 'deepseek/deepseek-chat-v3-0324'\n | 'deepseek/deepseek-chat-v3-0324:nitro'\n | 'deepseek/deepseek-r1-0528'\n | 'deepseek/deepseek-r1-0528:nitro'\n\n // Other open source models\n | 'moonshotai/kimi-k2'\n | 'moonshotai/kimi-k2:nitro'\n | 'moonshotai/kimi-k2.6'\n | 'z-ai/glm-5'\n | 'z-ai/glm-5.1'\n | 'z-ai/glm-4.6'\n | 'z-ai/glm-4.6:nitro'\n | 'z-ai/glm-4.7'\n | 'z-ai/glm-4.7:nitro'\n | 'z-ai/glm-4.7-flash'\n | 'z-ai/glm-4.7-flash:nitro'\n | 'minimax/minimax-m2.5'\n | 'minimax/minimax-m2.7'\n | (string & {})\n\nimport type { ToolName, GetToolParams } from './tools'\nimport type {\n Message,\n ToolResultOutput,\n JsonObjectSchema,\n MCPConfig,\n Logger,\n} from './util-types'\n\nexport type { ToolName, GetToolParams }\n" -export const toolsSource = "/**\n * Union type of all available tool names\n */\nexport type ToolName =\n | 'add_message'\n | 'ask_user'\n | 'check_background_agent'\n | 'check_job'\n | 'code_search'\n | 'end_turn'\n | 'edit_transaction'\n | 'edit_3d_asset'\n | 'find_files'\n | 'find_files_matching_content'\n | 'git_status'\n | 'git_branch'\n | 'get_task'\n | 'get_change_review_bundle'\n | 'inspect_workspace'\n | 'inspect_environment'\n | 'inspect_3d_asset'\n | 'get_affected_tests'\n | 'get_build_targets'\n | 'inspect_codebase_structure'\n | 'inspect_feature_completeness'\n | 'evaluate_audit_coverage'\n | 'glob'\n | 'kill_job'\n | 'list_directory'\n | 'list_jobs'\n | 'lookup_agent_info'\n | 'query_index'\n | 'read_docs'\n | 'read_files'\n | 'read_image'\n | 'render_3d_preview'\n | 'read_logs'\n | 'read_outline'\n | 'read_subtree'\n | 'replace_range'\n | 'rewrite_symbol'\n | 'render_ui'\n | 'run_file_change_hooks'\n | 'run_targeted_validation'\n | 'run_terminal_command'\n | 'set_messages'\n | 'set_output'\n | 'skill'\n | 'spawn_agents'\n | 'str_replace'\n | 'suggest_followups'\n | 'task_completed'\n | 'think_deeply'\n | 'update_plan_status'\n | 'web_search'\n | 'write_file'\n | 'write_audit_findings'\n | 'write_todos'\n\n/**\n * Map of tool names to their parameter types\n */\nexport interface ToolParamsMap {\n add_message: AddMessageParams\n ask_user: AskUserParams\n check_background_agent: CheckBackgroundAgentParams\n check_job: CheckJobParams\n code_search: CodeSearchParams\n end_turn: EndTurnParams\n edit_transaction: EditTransactionParams\n edit_3d_asset: Edit3dAssetParams\n find_files: FindFilesParams\n find_files_matching_content: FindFilesMatchingContentParams\n git_status: GitStatusParams\n git_branch: GitBranchParams\n get_task: GetTaskParams\n get_change_review_bundle: GetChangeReviewBundleParams\n inspect_workspace: InspectWorkspaceParams\n inspect_environment: InspectEnvironmentParams\n inspect_3d_asset: Inspect3dAssetParams\n get_affected_tests: GetAffectedTestsParams\n get_build_targets: GetBuildTargetsParams\n inspect_codebase_structure: InspectCodebaseStructureParams\n inspect_feature_completeness: InspectFeatureCompletenessParams\n evaluate_audit_coverage: EvaluateAuditCoverageParams\n glob: GlobParams\n kill_job: KillJobParams\n list_directory: ListDirectoryParams\n list_jobs: ListJobsParams\n lookup_agent_info: LookupAgentInfoParams\n query_index: QueryIndexParams\n read_docs: ReadDocsParams\n read_files: ReadFilesParams\n read_image: ReadImageParams\n render_3d_preview: Render3dPreviewParams\n read_logs: ReadLogsParams\n read_outline: ReadOutlineParams\n read_subtree: ReadSubtreeParams\n replace_range: ReplaceRangeParams\n rewrite_symbol: RewriteSymbolParams\n render_ui: RenderUiParams\n run_file_change_hooks: RunFileChangeHooksParams\n run_targeted_validation: RunTargetedValidationParams\n run_terminal_command: RunTerminalCommandParams\n set_messages: SetMessagesParams\n set_output: SetOutputParams\n skill: SkillParams\n spawn_agents: SpawnAgentsParams\n str_replace: StrReplaceParams\n suggest_followups: SuggestFollowupsParams\n task_completed: TaskCompletedParams\n think_deeply: ThinkDeeplyParams\n update_plan_status: UpdatePlanStatusParams\n web_search: WebSearchParams\n write_file: WriteFileParams\n write_audit_findings: WriteAuditFindingsParams\n write_todos: WriteTodosParams\n}\n\n/**\n * Add a new message to the conversation history. To be used for complex requests that can't be solved in a single step, as you may forget what happened!\n */\nexport interface AddMessageParams {\n role: 'user' | 'assistant'\n content: string\n}\n\n/**\n * Ask the user a list of multiple choice questions. Each question must have at least 2 options. The agent execution will pause until the user submits their answers.\n */\nexport interface AskUserParams {\n /** List of multiple choice questions to ask the user */\n questions: {\n /** The question to ask the user */\n question: string\n /** Optional short display label. Values longer than 18 Unicode code points are truncated instead of rejecting the question. */\n header?: string\n /** Array of answer options with label and optional description. */\n options: {\n /** The display text for this option */\n label: string\n /** Explanation shown when option is focused */\n description?: string\n }[]\n /** If true, allows selecting multiple options (checkbox). If false, single selection only (radio). */\n multiSelect?: boolean\n /** Validation rules for \"Other\" text input */\n validation?: {\n /** Maximum length for \"Other\" text input */\n maxLength?: number\n /** Minimum length for \"Other\" text input */\n minLength?: number\n /** Regex pattern for \"Other\" text input */\n pattern?: string\n /** Custom error message when pattern fails */\n patternError?: string\n }\n }[]\n}\n\n/**\n * Join/wait on a background agent turn started by spawn_agents({ background: true }): returns the sequenced agent_chunk events produced since the cursor plus the unified job state. Use it to observe a long-running background agent without blocking the turn.\n */\nexport interface CheckBackgroundAgentParams {\n /** The jobId returned by spawn_agents({ background: true }) for the background agent turn. */\n jobId: string\n /** Optional sequence cursor from a prior response. Polling is idempotent for an explicit cursor; nextCursor can be supplied on the next call. */\n cursor?: number\n /** Optional substring to wait for in the new streamed chunks before returning (follow mode). Returns early as soon as it appears in any chunk payload. Useful for waiting until a background agent emits a specific milestone (e.g. a tool_result or a text marker). */\n wait_for?: string\n /** Max seconds to wait for new chunks / the wait_for pattern. 0 (default) returns immediately with whatever new chunks exist (poll mode); >0 blocks up to this long (follow mode). */\n timeout_seconds?: number\n /** When true, explicitly cancel the running background agent before returning its final status. Defaults to false. */\n cancel?: boolean\n}\n\n/**\n * Join/wait on a background job started by run_terminal_command: returns the sequenced output events produced since the last check plus the unified job state and exit code. Use it to observe a long-running process without blocking the turn. To watch an arbitrary log file, start a `tail -f ` BACKGROUND job and check_job it with a wait_for pattern.\n */\nexport interface CheckJobParams {\n /** The jobId returned by run_terminal_command with process_type: BACKGROUND. */\n jobId: string\n /** Optional substring to wait for in the new output before returning (follow mode). Returns early as soon as it appears (e.g. \"Listening on\" / \"compiled successfully\"). */\n wait_for?: string\n /** Max seconds to wait for new output / the wait_for pattern. 0 (default) returns immediately with whatever new output exists (poll mode); >0 blocks up to this long (follow mode). */\n timeout_seconds?: number\n /** Follow mode only: SIGTERM the job on follow-timeout. Poll mode never kills. Default false. */\n kill_on_timeout?: boolean\n}\n\n/**\n * Search for string patterns in the project's files. This tool uses ripgrep (rg), a fast line-oriented search tool. Use this tool only when read_files is not sufficient to find the files you need.\n */\nexport interface CodeSearchParams {\n /** The pattern to search for. */\n pattern: string\n /** Optional safe ripgrep flags as one string or argv tokens (e.g., \"-i -g *.ts -A 2\" or [\"-i\", \"-g\", \"*.ts\", \"-A\", \"2\"]). Allowed: -i/--ignore-case, -S/--smart-case, -s/--case-sensitive, -w/--word-regexp, -F/--fixed-strings, -U/--multiline, --multiline-dotall, -g/--glob, -t/--type, -T/--type-not, plus context -A/-B/-C (and long forms). JSON quotes delimit the string; do not embed another quote pair around the entire expression. Line numbers are automatic; -n/--line-number are ignored. Output-shape flags such as -c/--count, --count-matches, -l, -v/--invert-match, -r/--replace, --exec, and -z/--null are rejected. */\n flags?: string | string[]\n /** Optional working directory or single file to search within, relative to the project root or absolute. Absolute paths may be outside the project. A directory becomes ripgrep's cwd and scopes the search under that path (plus existing blessed hidden dirs when no paths are given); a file scopes the search to that file only (process cwd = project root when the file is under the project, else the file's parent). Defaults to searching the entire project root. */\n cwd?: string\n /** Optional list of file and/or directory paths to search (relative to the project root, or absolute). When non-empty, ripgrep searches only these targets instead of the whole cwd tree (and does not auto-expand hidden dirs). Can be combined with a file cwd. */\n paths?: string[]\n /** Maximum number of results to return per file. Defaults to 15. There is also a global limit of 250 results across all files. */\n maxResults?: number\n}\n\n/**\n * End your turn, regardless of any new tool results that might be coming. This will allow the user to type another prompt.\n */\nexport interface EndTurnParams {}\n\n/**\n * Parameters for edit_transaction tool\n */\nexport interface EditTransactionParams {\n edits: (\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'str_replace'\n replacements: {\n oldString: string\n newString: string\n allowMultiple?: boolean\n occurrenceIndex?: number\n /** Optional authenticated cap.v3 readCapability copied verbatim from the matching fresh read_files editAnchor. */\n basedOnRead?: string\n /** For deletion replacements only (newString is empty): treat a missing oldString as an already-applied no-op. Use only for explicit idempotent cleanup retries, never for ordinary edits. When every requested change resolves to such a no-op - every replacement of a standalone str_replace call, or every edit of an edit_transaction - the call succeeds with zero file changes and the skip messages rather than failing. When combined with occurrenceIndex, a partially-applied cleanup also skips: fewer remaining exact occurrences than the requested index means that occurrence is treated as already applied. Only valid when newString is empty; both the input and provider schemas reject any other combination. */\n skipIfMissing?: boolean\n }[]\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n /** A structured edit dispatched by operation kind. */\n type: 'structured'\n /** Structured edit operation to apply to this file. */\n operation:\n | {\n /** Deterministic text insertion. */\n kind: 'insert_text'\n /** 1-indexed insertion position. */\n position: {\n /** 1-indexed target line. */\n line: number\n /** 1-indexed target column. */\n column: number\n }\n text: string\n }\n | {\n /** Language-aware import insertion. */\n kind: 'insert_import'\n /** Complete language-native import statement to add, e.g. \"import { foo } from 'bar'\", \"from app import value\", or \"use crate::value\". */\n importStatement: string\n }\n | {\n /** Language-aware import removal. */\n kind: 'remove_import'\n /** Complete language-native import statement to remove. Required unless moduleSpecifier is provided. */\n importStatement?: string\n /** Module specifier to remove imports from, e.g. \"react\" or \"./helper\". */\n moduleSpecifier?: string\n }\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'create'\n /** Exact bytes to write to the new file. */\n content: string\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'delete'\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'move'\n /** New project-relative path. The destination must be absent. */\n destinationPath: string\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'replace_range'\n readCapability: string\n startLine?: number\n endLine?: number\n occurrence?: {\n match: string\n occurrence?: number\n }\n newContent: string\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'rewrite_symbol'\n symbol: string\n content: string\n occurrence?: number\n /** Optional cap.v3 copied from the matching read_files symbol slice. It authorizes exactly the symbol and its contiguous preceding comment block. */\n readCapability?: string\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'patch'\n diff: string\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'write_file'\n content: string\n /** Optional whole-file-covering cap.v3 from a fresh complete whole-file read. Only a full-file capability with a hash matching current content may authorize overwrite; partial ranges never authorize write_file. */\n basedOnRead?: string\n }\n )[]\n}\n\n/**\n * Parameters for edit_3d_asset tool\n */\nexport interface Edit3dAssetParams {\n /** Project-relative .blend path. */\n path: string\n /** Exact source hash returned by inspect_3d_asset. */\n source_hash: string\n operations: (\n | {\n type: 'rename_object'\n object: string\n new_name: string\n }\n | {\n type: 'set_object_transform'\n object: string\n location?: any[]\n rotation_degrees?: any[]\n scale?: any[]\n }\n | {\n type: 'set_render_resolution'\n width: number\n height: number\n percentage?: number\n }\n | {\n type: 'set_frame_range'\n start: number\n end: number\n }\n )[]\n}\n\n/**\n * Find several files related to a brief natural language description of the files or the name of a function or class you are looking for.\n */\nexport interface FindFilesParams {\n /** A brief natural language description of the files or the name of a function or class you are looking for. It's also helpful to mention a directory or two to look within. */\n prompt: string\n}\n\n/**\n * List unique file paths whose content matches a pattern, with optional symbol grouping. Built on top of ripgrep (rg).\n */\nexport interface FindFilesMatchingContentParams {\n /** Regex pattern (ripgrep syntax) to match file content against. */\n pattern: string\n /** Optional safe ripgrep flags as one string or argv tokens. Allowed: -i/--ignore-case, -S/--smart-case, -s/--case-sensitive, -w/--word-regexp, -F/--fixed-strings, -U/--multiline, --multiline-dotall, -g/--glob, -t/--type, -T/--type-not. Examples: \"-g *.ts -g *.tsx\" or [\"-g\", \"*.ts\", \"-g\", \"*.tsx\"]. Do not quote the entire expression inside the JSON string. Output-shape flags such as -c/--count, --count-matches, -l, -v/--invert-match, context -A/-B/-C, -r/--replace, --exec, and -z/--null are rejected (this tool forces -l or --json itself). Redundant -n/--line-number inputs are ignored. */\n flags?: string | string[]\n /** Optional working directory or single file to search within, relative to the project root or absolute. Absolute paths may be outside the project. A directory becomes ripgrep's cwd and scopes the search under that path (plus existing blessed hidden dirs); a file scopes the search to that file only (process cwd = project root when the file is under the project, else the file's parent). Defaults to the project root. */\n cwd?: string\n /** Maximum number of unique files to return. Defaults to 100. */\n maxFiles?: number\n /** When true, also return the names of the top-level symbols (functions, classes, methods, exports, constants) that contain each match, plus the per-file match count. Symbol extraction is heuristic and works best for JS/TS/Python/Go/Rust source files; languages without a recognized declaration shape produce an empty symbols list. */\n groupBySymbol?: boolean\n /** Maximum seconds to let ripgrep run before returning partial results. Defaults to 15. */\n timeoutSeconds?: number\n}\n\n/**\n * Read-only git status and (optionally) diff for the current project.\n */\nexport interface GitStatusParams {\n /** When true, also return the unified diff of uncommitted changes. */\n include_diff?: boolean\n /** When true with include_diff, returns the staged diff instead of unstaged. */\n staged?: boolean\n /** Optional path to scope status/diff to (relative to project root). */\n path?: string\n /** Maximum characters of diff output to return. Defaults to 40,000. */\n max_chars?: number\n}\n\n/**\n * Create a new git branch, optionally switching to it. Refuses to branch when the working tree is dirty unless `allow_dirty` is true.\n */\nexport interface GitBranchParams {\n /** Name of the branch to create. Must start with an alphanumeric character and contain only [a-zA-Z0-9._/-]. */\n branch_name: string\n /** When true (default), create AND switch to the branch (`git checkout -b`). When false, only create the branch (`git branch`), leaving the current branch checked out. */\n switch?: boolean\n /** When true, skip the dirty-tree refusal check. Defaults to false — the tool refuses to branch when the working tree has uncommitted changes. */\n allow_dirty?: boolean\n}\n\n/**\n * Parameters for get_task tool\n */\nexport interface GetTaskParams {\n /** Optional plan session slug. Defaults to .agents/ACTIVE_SESSION. */\n session?: string\n}\n\n/**\n * Parameters for get_change_review_bundle tool\n */\nexport interface GetChangeReviewBundleParams {\n max_chars?: number\n}\n\n/**\n * Inspect the current repository/worktree identity and Git state without modifying it.\n */\nexport interface InspectWorkspaceParams {}\n\n/**\n * Parameters for inspect_environment tool\n */\nexport interface InspectEnvironmentParams {}\n\n/**\n * Parameters for inspect_3d_asset tool\n */\nexport interface Inspect3dAssetParams {\n /** Project-relative 3D asset path. */\n path: string\n}\n\n/**\n * Parameters for get_affected_tests tool\n */\nexport interface GetAffectedTestsParams {\n files: string[]\n}\n\n/**\n * Parameters for get_build_targets tool\n */\nexport interface GetBuildTargetsParams {\n files: string[]\n}\n\n/**\n * Parameters for inspect_codebase_structure tool\n */\nexport interface InspectCodebaseStructureParams {\n scope?: string[]\n}\n\n/**\n * Parameters for inspect_feature_completeness tool\n */\nexport interface InspectFeatureCompletenessParams {\n feature: string\n snapshot_id: string\n scope?: string[]\n}\n\n/**\n * Parameters for evaluate_audit_coverage tool\n */\nexport interface EvaluateAuditCoverageParams {\n snapshot_id: string\n structural_receipts: {\n schema_version: 1\n snapshot_id: string\n shard_id: string\n subsystem_ids: string[]\n files: string[]\n domains: (\n | 'security'\n | 'correctness'\n | 'state-mutation'\n | 'error-handling'\n | 'performance'\n | 'dependency-hygiene'\n | 'test-coverage'\n | 'api-contract'\n )[]\n }[]\n features: {\n schema_version: 1\n snapshot_id: string\n feature: string\n evidence_kind: 'heuristic' | 'verified'\n evidence: {\n entrypoints: string[]\n implementation: string[]\n consumers: string[]\n tests: string[]\n docs: string[]\n failure_states: string[]\n }\n }[]\n out_of_scope?: {\n id: string\n reason: string\n }[]\n scope?: string[]\n}\n\n/**\n * Search for files matching a glob pattern. Returns matching file paths sorted by modification time (newest first, then path for deterministic ties).\n */\nexport interface GlobParams {\n /** Glob pattern to match files against (e.g., *.js, src/glob/*.ts, glob/test/glob/*.go). */\n pattern: string\n /** Optional working directory or file path, relative to project root. If a directory, the glob pattern is matched against paths relative to this cwd, while returned files remain project-relative. If a file path, the pattern is matched against that file only (full path or basename). If not provided, searches from project root. */\n cwd?: string\n}\n\n/**\n * Cancel a background job started by run_terminal_command.\n */\nexport interface KillJobParams {\n /** The jobId returned by run_terminal_command with process_type: BACKGROUND. */\n jobId: string\n /** Signal to send. Defaults to SIGTERM; use SIGKILL only if graceful termination fails. */\n signal?: 'SIGTERM' | 'SIGKILL'\n}\n\n/**\n * List files and directories in the specified path. Returns separate arrays of file names and directory names.\n */\nexport interface ListDirectoryParams {\n /** Directory path to list, relative to the project root. */\n path: string\n}\n\n/**\n * List this run's background jobs (shell processes and background agents, running and settled) with statuses, bucketed pending process/log output relative to the last check_job consumer cursor (agents usually show pending: 'none'), and a gap flag.\n */\nexport interface ListJobsParams {}\n\n/**\n * Retrieve information about an agent by ID\n */\nexport interface LookupAgentInfoParams {\n /** Agent ID (short local or full published format) */\n agentId: string\n}\n\n/**\n * Query the local codebase graph index to find relevant files ranked by symbol names, imports, headings, paths, doc concepts, and graph relationships. The index is built automatically on startup.\n */\nexport interface QueryIndexParams {\n /** Natural language query or keyword terms describing the files you are looking for. Optional for graph modes when from/to paths are provided. For example: \"authentication\", \"database migrations\", \"editor mutation logic\", \"React components\". */\n query?: string\n /** Maximum number of results to return. Defaults to 20. */\n limit?: number\n /** Optional list of file extensions to filter results (without dot). E.g. [\"ts\", \"tsx\"] for TypeScript only. */\n fileTypes?: string[]\n /** Optional normalized project-relative directory prefixes. Results outside every prefix are excluded before ranking/limiting. */\n pathPrefixes?: string[]\n /** search|explain|neighbors|path|commands|references — see tool description. */\n mode?: 'search' | 'neighbors' | 'path' | 'explain' | 'commands' | 'references'\n /** Optional source file path for neighbors, path, and references modes. */\n from?: string\n /** Optional target file path for path mode. Also used as the seed file for references mode when from is omitted or not indexed. */\n to?: string\n}\n\n/**\n * Fetch up-to-date documentation for libraries and frameworks using Context7 API.\n */\nexport interface ReadDocsParams {\n /** The library or framework name (e.g., \"Next.js\", \"MongoDB\", \"React\"). Use the official name as it appears in documentation if possible. Only public libraries available in Context7's database are supported, so small or private libraries may not be available. */\n libraryTitle: string\n /** Specific topic to focus on (e.g., \"routing\", \"hooks\", \"authentication\") */\n topic: string\n /** Optional maximum number of tokens to return. Defaults to 10000. Values less than 10000 are automatically increased to 10000. */\n max_tokens?: number\n}\n\n/**\n * Read multiple files from disk and return their contents. Use this tool to read as many files as would be helpful to answer the user's request.\n */\nexport interface ReadFilesParams {\n /** Whole-file paths to read. Complete results include editAnchor.readCapability for follow-up edits. */\n paths?: string[]\n /** 1-indexed inclusive line ranges. Sole `paths` entry infers missing path. */\n ranges?: {\n /** Project-relative file path. */\n path: string\n /** 1-indexed inclusive start line. Defaults to 1. */\n startLine?: number\n /** 1-indexed inclusive end line. Defaults to the last line. */\n endLine?: number\n }[]\n /** Contiguous line windows; each complete window mints a scoped cap.v3 editAnchor. */\n windows?: {\n /** File path to read in contiguous line windows, relative to the project root. */\n path: string\n /** Lines per window. Defaults to 400, capped at 5000. */\n windowSize?: number\n /** 1-indexed window number to return. Omit to get the window manifest (totalLines, windowSize, windowCount) plus the first window. */\n window?: number\n }[]\n /** Literal-anchored context blocks with a scoped cap.v3 editAnchor per block. */\n around?: {\n /** File path to read a content-anchored block from, relative to the project root. */\n path: string\n /** Exact literal string to anchor on. Robust to line-number drift. */\n match: string\n /** 1-indexed occurrence of `match` to anchor on. Defaults to 1. */\n occurrence?: number\n /** Lines of context to include on each side of the match, clamped at file boundaries. Defaults to 40, capped at 2000. */\n contextLines?: number\n }[]\n /** Nth top-level symbol by name (rewrite_symbol occurrence semantics); prefer batch `symbols` when possible. */\n symbol?: {\n /** File path to extract a symbol slice from, relative to the project root. */\n path: string\n /** Top-level symbol name (function, class, interface, method) to pull, as shown by read_outline. */\n name: string\n /** When multiple top-level symbols share this name, the 1-indexed one to return. Defaults to 1. Matches rewrite_symbol occurrence semantics. */\n occurrence?: number\n }[]\n /** Named symbol slices with editAnchors; prefer over full reads when names are known. */\n symbols?: {\n /** Project-relative file path. */\n path: string\n /** Symbol names to slice. */\n names: string[]\n }[]\n}\n\n/**\n * Read image files from disk and return them as model-visible image media.\n */\nexport interface ReadImageParams {\n /** List of image file paths to read. */\n paths: string[]\n}\n\n/**\n * Parameters for render_3d_preview tool\n */\nexport interface Render3dPreviewParams {\n /** Project-relative 3D asset path. */\n path: string\n views?: ('camera' | 'perspective' | 'front' | 'side' | 'top')[]\n mode?: 'material' | 'clay' | 'wireframe'\n width?: number\n height?: number\n}\n\n/**\n * Read the last N lines from a log/text file or background job log without starting a background tail process.\n */\nexport interface ReadLogsParams {\n /** Path to the log file, relative to the project root unless absolute. Required unless jobId is provided. */\n path?: string\n /** Background job id returned by run_terminal_command(process_type: BACKGROUND). When provided, reads the job log file directly. */\n jobId?: string\n /** Number of trailing lines to read. Defaults to 200. */\n lines?: number\n /** Maximum characters to return. Defaults to 20,000. */\n max_chars?: number\n}\n\n/**\n * Generate an outline of imports, exports, classes, methods, and function signatures in a source file without reading the entire implementation.\n */\nexport interface ReadOutlineParams {\n /** File path to generate the AST-like outline for, relative to the project root. */\n path: string\n}\n\n/**\n * Read one or more directory subtrees (as a blob including subdirectories, file names, and parsed variables within each source file) or return parsed variable names for files. If no paths are provided, returns the entire project tree.\n */\nexport interface ReadSubtreeParams {\n /** List of paths to directories or files. Relative to the project root. If omitted, the entire project tree is used. */\n paths?: string[]\n /** Maximum token budget for the subtree blob; the tree will be truncated to fit within this budget by first dropping file variables and then removing the most-nested files and directories. */\n maxTokens?: number\n}\n\n/**\n * Replace all of, a contained sub-range of, or the Nth literal occurrence inside content observed through one fresh cap.v3 read capability.\n */\nexport interface ReplaceRangeParams {\n /** The path to the file to edit. */\n path: string\n /** Copy the cap.v3 readCapability verbatim from the matching fresh read_files editAnchor. The token supplies the observed line bounds and content hash. */\n readCapability: string\n /** Optional 1-indexed target start within the capability-covered range. Omit with endLine to replace the complete observed range. */\n startLine?: number\n /** Optional 1-indexed target end within the capability-covered range. Omit with startLine to replace the complete observed range. */\n endLine?: number\n /** Optional occurrence targeting: replace the 1-indexed occurrence (default 1) of the exact literal match found inside the capability-authorized range. Mutually exclusive with startLine/endLine. */\n occurrence?: {\n match: string\n occurrence?: number\n }\n /** Complete replacement content for the selected line range. */\n newContent: string\n}\n\n/**\n * Replace a whole symbol's definition by name using the file's syntax tree, without copying its current text. Resolves the exact AST range and applies it through the safe str_replace path (atomic, anchored).\n */\nexport interface RewriteSymbolParams {\n /** File path containing the symbol, relative to the project root. */\n path: string\n /** Name of the function/class/method/type/interface to replace (as shown by read_outline). */\n symbol: string\n /** The complete new source for the symbol, replacing its entire current definition (e.g. the whole function including its signature and body). Provide REAL newlines/tabs in the string — literal backslash-n (\\n) and backslash-t (\\t) sequences are not interpreted and will be written verbatim into the file. This matches str_replace. */\n content: string\n /** When multiple top-level symbols share this name, the 1-indexed one to replace. */\n occurrence?: number\n /** Optional cap.v3 copied from the matching read_files symbol slice. Under strict read-before-edit this authorizes exactly the symbol and its contiguous preceding comment block. */\n readCapability?: string\n}\n\n/**\n * Render a small interactive UI widget in the Openbuff CLI. Currently supports a button that opens a link.\n */\nexport interface RenderUiParams {\n /** The UI widget to render. */\n widget: {\n /** Widget type. Currently, the only supported widget is button. */\n type: 'button'\n /** Short button label shown to the user. */\n text: string\n /** The http:// or https:// URL to open when the user clicks the button. */\n link: string\n /** Theme-aware color treatment. Use primary for the main action and secondary for lower-emphasis actions. */\n variant?: 'primary' | 'secondary'\n }\n}\n\n/**\n * Parameters for run_file_change_hooks tool\n */\nexport interface RunFileChangeHooksParams {\n /** List of file paths that were changed and should trigger file change hooks */\n files: string[]\n}\n\n/**\n * Parameters for run_targeted_validation tool\n */\nexport interface RunTargetedValidationParams {\n snapshot_id: string\n files: string[]\n artifact_kinds?: string[]\n}\n\n/**\n * Execute a CLI command from the **project root** (different from the user's cwd).\n */\nexport interface RunTerminalCommandParams {\n /** CLI command valid for user's OS. */\n command: string\n /** SYNC (default) for finite commands that exit: waits and returns output. BACKGROUND only for long-running or never-exiting processes (dev servers, watchers, log tails): starts a detached job and returns a jobId immediately so the turn is not blocked. Live job_update already drives the user UI; use check_job for agent-side readiness/exitCode/join, not solely for user progress. */\n process_type?: 'SYNC' | 'BACKGROUND'\n /** For BACKGROUND commands only: keep the job running if the owning request is cancelled. Defaults to false. */\n detach?: boolean\n /** The working directory to run the command in. Default is the project root. */\n cwd?: string\n /** Wall-clock bound in seconds for SYNC commands. Omit or use -1 for no timeout (the default). Does not apply to BACKGROUND commands. */\n timeout_seconds?: number\n /** Runtime-managed background job owner; agents must omit. */\n owner?: {\n clientSessionId: string\n rootRunId: string\n parentRunId: string\n parentAgentId: string\n }\n}\n\n/**\n * Atomically replace conversation history and, when supplied, commit a validated structured task-memory revision.\n */\nexport interface SetMessagesParams {\n messages: any\n taskMemory?: {\n schemaVersion: 1\n goal?: string\n requirements?: string[]\n decisions?: string[]\n filesInspected?: string[]\n editsMade?: string[]\n validationResults?: string[]\n reviewReceipts?: string[]\n blockers?: string[]\n nextActions?: string[]\n historicalSummary?: string\n evidence?: {\n id: string\n kind:\n | 'requirement'\n | 'decision'\n | 'read'\n | 'edit'\n | 'validation'\n | 'review'\n | 'blocker'\n | 'handoff'\n | 'note'\n summary: string\n source?: string\n path?: string\n freshnessHash?: string\n workspaceRevision?: number\n verifiedAt?: number\n supersedes?: string[]\n stale?: boolean\n }[]\n workspaceRevision?: number\n workspaceSnapshotId?: string\n }\n expectedTaskMemoryRevision?: number\n}\n\n/**\n * JSON object to set as the agent output. The shape of the parameters are specified dynamically further down in the conversation. This completely replaces any previous output. If the agent was spawned, this value will be passed back to its parent. If the agent has an outputSchema defined, the output will be validated against it.\n */\nexport interface SetOutputParams {\n data?: Record\n [key: string]: any\n}\n\n/**\n * Load a skill by name to get its full instructions. Skills provide reusable behaviors and instructions.\n */\nexport interface SkillParams {\n /** The name of the skill to load */\n name: string\n}\n\n/**\n * Spawn up to 12 agents and send a prompt and/or parameters to each of them. These agents will run in parallel. Note that that means they will run independently. Split larger work into bounded waves. If you need to run agents sequentially, use spawn_agents with one agent at a time instead.\n */\nexport interface SpawnAgentsParams {\n agents: {\n /** Agent to spawn. Must be a name from the live \"You can spawn the following agents\" catalog (hyphenated ids; underscores accepted). */\n agent_type: string\n /** Prompt to send to the agent */\n prompt?: string\n /** If true, return jobId immediately and run as in-process coroutine; poll with check_background_agent. Defaults to false (blocking). Cannot outlive this CLI session. */\n background?: boolean\n /** Optional structured handoff; additive — non-consumers still get prompt/params. */\n handoff?:\n | {\n schemaVersion: 1\n taskId: string\n role:\n | 'orchestrator'\n | 'explorer'\n | 'thinker'\n | 'editor'\n | 'repair-editor'\n | 'test-writer'\n | 'doc-writer'\n | 'dependency-manager'\n | 'debugger'\n | 'validator'\n | 'reviewer'\n | 'security-reviewer'\n | 'committer'\n | 'synthesizer'\n | 'specialist'\n | 'general'\n objective: string\n requirements: {\n id: string\n text: string\n required: boolean\n }[]\n acceptanceCriteria: {\n id: string\n behavior: string\n verification: string\n }[]\n context:\n | {\n path: string\n symbols: string[]\n reason: string\n confidence: 'confirmed' | 'inferred' | 'unknown'\n freshnessHash?: string\n workspaceRevision?: number\n }[]\n | Record\n | string\n currentBehavior?: string\n desiredBehavior?: string\n invariants?: string[]\n nonGoals: string[]\n risks?: string[]\n unknowns?: string[]\n findings: {\n id: string\n text: string\n files: string[]\n snapshotFingerprint: string\n }[]\n permissions: {\n readablePaths: string[]\n writablePaths: string[]\n allowedTools: string[]\n }\n workspaceRevision?: number\n workspaceSnapshotId?: string\n summary?: string\n artifacts?: string[]\n successCriteria?: string[]\n constraints?: string[]\n }\n | Record\n /** Parameters object for the agent */\n params?: {\n /** Terminal command to run (basher, tmux-cli) */\n command?: string\n /** What information from the command output is desired (basher) */\n what_to_summarize?: string\n /** Timeout for command in seconds. Omit or -1 for no timeout (default). */\n timeout_seconds?: number\n /** Save full command output to a /tmp log and extract failure lines for long SYNC command output (basher) */\n save_full_log?: boolean\n /** grep -E failure extraction pattern used with save_full_log (basher) */\n failure_pattern?: string\n /** Maximum extracted failure lines to return with save_full_log (basher) */\n max_failure_lines?: number\n /** Relevant file paths to read (general-agent) */\n filePaths?: string[]\n /** Relevant directory paths to inventory (general-agent) */\n directoryPaths?: string[]\n /** Directories to search within (file-picker) */\n directories?: string[]\n /** Starting URL to navigate to (browser-use) */\n url?: string\n /** Exact task-owned paths eligible for staging (git-committer) */\n owned_paths?: string[]\n /** Optional branch to create or switch to (git-committer) */\n branch_name?: string\n /** Create and switch to branch_name when true (git-committer) */\n branch_switch?: boolean\n /** Allow branch create/switch on a dirty worktree (git-committer) */\n allow_dirty_branch?: boolean\n /** Push the resulting feature branch when authorized (git-committer) */\n push?: boolean\n /** Remote used for fetch/push (git-committer) */\n remote?: string\n /** Assigned gate snapshot fingerprint (reviewer specialists) */\n snapshot_id?: string\n /** Changed file paths to review (security-reviewer) */\n changed_files?: string[]\n /** Opaque snapshot token to echo (security-reviewer) */\n snapshot_fingerprint?: string\n /** Package manager selected from repository manifests (dependency-manager) */\n manager?: string\n /** Dependency operation: add, remove, sync, restore, or update (dependency-manager) */\n operation?: string\n /** Exact package specifications (dependency-manager) */\n packages?: string[]\n /** Optional workspace selector (dependency-manager) */\n workspace?: string\n /** GitHub repository URL to clone (librarian) */\n repoUrl?: string\n /** Retain the owned /tmp clone after completion (librarian) */\n retainClone?: boolean\n /** Optional search or path patterns */\n patterns?: string[]\n /** Exact files in scope (reviewer specialists) */\n files?: string[]\n /** Optional agent-specific prompts */\n prompts?: string[]\n [key: string]: any\n }\n }[]\n}\n\n/**\n * Parameters for str_replace tool\n */\nexport interface StrReplaceParams {\n /** The file to edit. */\n path: string\n atomic?: boolean\n replacements: {\n oldString: string\n newString: string\n allowMultiple?: boolean\n occurrenceIndex?: number\n /** Optional authenticated cap.v3 readCapability copied verbatim from the matching fresh read_files editAnchor. */\n basedOnRead?: string\n /** For deletion replacements only (newString is empty): treat a missing oldString as an already-applied no-op. Use only for explicit idempotent cleanup retries, never for ordinary edits. When every requested change resolves to such a no-op - every replacement of a standalone str_replace call, or every edit of an edit_transaction - the call succeeds with zero file changes and the skip messages rather than failing. When combined with occurrenceIndex, a partially-applied cleanup also skips: fewer remaining exact occurrences than the requested index means that occurrence is treated as already applied. Only valid when newString is empty; both the input and provider schemas reject any other combination. */\n skipIfMissing?: boolean\n }[]\n}\n\n/**\n * Suggest clickable followup prompts to the user. Each followup becomes a card the user can click to send that prompt.\n */\nexport interface SuggestFollowupsParams {\n /** List of suggested followup prompts the user can click to send */\n followups: {\n /** The full prompt text to send as a user message when clicked */\n prompt: string\n /** Short display label for the card (defaults to truncated prompt if not provided) */\n label?: string\n }[]\n}\n\n/**\n * Signal that the task is complete. Use this tool when:\n- The user's request is completely fulfilled\n- You need clarification from the user before continuing\n- You are stuck or need help from the user to continue\n\nThis tool explicitly marks the end of your work on the current task.\n */\nexport interface TaskCompletedParams {}\n\n/**\n * Deeply consider complex tasks by brainstorming approaches and tradeoffs step-by-step.\n */\nexport interface ThinkDeeplyParams {\n /** Detailed step-by-step analysis. Initially keep each step concise (max ~5-7 words per step). */\n thought: string\n}\n\n/**\n * Parameters for update_plan_status tool\n */\nexport interface UpdatePlanStatusParams {\n /** Artifact path. Must be `.agents/sessions//PLAN.md`, `.agents/sessions//STATUS.md`, or `.agents/sessions//LESSONS.md`. Absolute paths and `..` traversal are rejected. Editing PLAN.md is permitted only for tri-state task toggles (not full overwrites). */\n path: string\n /** Targeted updates applied in order. Each entry rewrites at most one matching checklist line; unmatched updates fall through to `append`. */\n updates?: {\n /** Stable task ID at the start of a checklist line (for example `P2-T3`). Preferred over substring matching. */\n taskId?: string\n /** Substring of the existing task/checklist line to match (case-insensitive). The first matching `- [ ]`/`-[x]`/`-[~]`/`-[/]`/`-[!]` line in the artifact will be updated in place. */\n task?: string\n /** When provided, sets the checkbox state of the matched line (true -> `[x]`, false -> `[ ]`). Ignored when `status` is also provided. */\n completed?: boolean\n /** Explicit tri-state task status. When provided, overrides `completed`. Transitions a task to `in_progress` (`[~]`), `done` (`[x]`), `cancelled` (`[/]`), `blocked` (`[!]`), or back to `pending` (`[ ]`). */\n status?: 'pending' | 'in_progress' | 'done' | 'cancelled' | 'blocked'\n /** Optional short note to append to the matched line in parentheses. Preserves any existing trailing text on the line. */\n note?: string\n }[]\n /** Optional delimited entry appended at the end of the artifact (used when there is no matching task line for the change being recorded). */\n append?: {\n /** Short heading for an appended entry. Used to form a clearly delimited block (`## — `). */\n heading: string\n /** Markdown body for the appended entry. Written verbatim under the heading. */\n body: string\n }\n /** Optional session-level status transition. When provided, `.agents/sessions//STATE.json` is created or updated to reflect the new lifecycle status. */\n sessionStatus?:\n | 'draft'\n | 'ready'\n | 'active'\n | 'executing'\n | 'validating'\n | 'reviewing'\n | 'blocked'\n | 'paused'\n | 'completed'\n | 'archived'\n /** Optional current-task pointer written as a `` annotation in PLAN.md. Pass an empty string or omit to clear the pointer. Only takes effect when path targets PLAN.md. */\n currentTask?: string\n /** Optional STATE.json compare-and-swap revision. The update fails without writing when the current revision differs. */\n expectedRevision?: number\n /** Validation or review evidence associated with a stable task ID. Completing a PLAN task requires a passed validation checkpoint with receiptIds. */\n checkpoint?: {\n taskId: string\n phase: 'validation' | 'review'\n passed: boolean\n summary?: string\n receiptIds?: string[]\n }\n}\n\n/**\n * Search the web for current information, or fetch the content of a specific URL.\n */\nexport interface WebSearchParams {\n /** The search query to find relevant web content. Required unless url is provided. */\n query?: string\n /** A specific URL to fetch and read the full text content of. When provided, fetches this page directly instead of searching. Useful for reading documentation, GitHub READMEs, blog posts, or any public web page. */\n url?: string\n /** Search depth - 'standard' for quick results, 'deep' for more comprehensive search. Default is 'standard'. Ignored when url is provided. */\n depth?: 'standard' | 'deep'\n /** When fetching a URL, also extract and return links found on the page. Enables navigation by letting you see what pages are linked. Default: true. */\n include_links?: boolean\n /** Maximum number of links to extract when include_links is true. Default: 40. */\n max_links?: number\n}\n\n/**\n * Create or overwrite a file with the given content.\n */\nexport interface WriteFileParams {\n /** Path to the file relative to the **project root** */\n path: string\n /** What the change is intended to do in only one sentence. */\n instructions: string\n /** Complete file content to write to the file. */\n content: string\n /** Optional whole-file-covering cap.v3 from a fresh complete whole-file read (paths or full-file range). Only a capability that covers the entire current file (startLine=1 through the current line count) with a hash matching current content may authorize overwrite; partial range capabilities never authorize write_file. */\n basedOnRead?: string\n}\n\n/**\n * Parameters for write_audit_findings tool\n */\nexport interface WriteAuditFindingsParams {\n /** Existing durable audit session slug under .agents/sessions/. Accepts only a short identifier token: 1 to 100 characters of letters, digits, dot, underscore, or dash, and neither `.` nor `..` on its own. */\n sessionSlug: string\n /** Unique shard identifier used as the findings filename. Accepts only a short identifier token: 1 to 100 characters of letters, digits, dot, underscore, or dash, and neither `.` nor `..` on its own. */\n shardId: string\n /** Exact snapshotId returned by inspect_codebase_structure, such as its 64-character sha256 digest. Required for a directly composable structuralReceipt; omitted only for legacy callers. Accepts only a short identifier token: 1 to 100 characters of letters, digits, dot, underscore, or dash, and neither `.` nor `..` on its own. When snapshotId and coverage.domains are both present the call receives a structuralReceipt, so coverage.subsystemIds and coverage.files must each name at least one entry: evaluate_audit_coverage rejects a receipt whose subsystem_ids or files list is empty. */\n snapshotId?: string\n /** Each findings entry rejects control and Unicode format characters in title, risk, fix, and evidence — NUL, any other control character, and the U+2028/U+2029 line separators — while still accepting tabs and line breaks in that prose. findings[].path is a location rather than prose, so it must be a single-line value with none of those characters and no tabs or line breaks; it is trimmed, and the trimmed value is the one rendered into the finding heading. */\n findings: {\n severity: 'CRITICAL' | 'HIGH' | 'MEDIUM' | 'LOW'\n domain:\n | 'security'\n | 'correctness'\n | 'state-mutation'\n | 'error-handling'\n | 'performance'\n | 'dependency-hygiene'\n | 'test-coverage'\n | 'api-contract'\n | 'api-abi'\n path: string\n line?: number\n title: string\n risk: string\n fix: string\n evidence: string\n }[]\n /** Every coverage list must name each entry at most once: a repeated file, subsystemId, featureId, or domain is rejected rather than counted twice. Entries are compared after trimming surrounding whitespace, and the trimmed value is what reaches the artifact and the receipt, so two spellings that differ only in whitespace are the same entry. Every coverage files, subsystemIds, and featureIds entry must be a single-line value: tabs, carriage returns, newlines, NUL, any other control or Unicode format character, and the U+2028/U+2029 line separators are rejected. Entries are trimmed, and the trimmed value is the one uniqueness is judged on. */\n coverage: {\n /** See the coverage description for the single-line hygiene rule, which applies to this list too. See the coverage description for the uniqueness rule, which applies to this list too. */\n subsystemIds: string[]\n /** See the coverage description for the single-line hygiene rule, which applies to this list too. See the coverage description for the uniqueness rule, which applies to this list too. */\n featureIds: string[]\n /** See the coverage description for the single-line hygiene rule, which applies to this list too. See the coverage description for the uniqueness rule, which applies to this list too. */\n files: string[]\n /** coverage.domains accepts canonical domain ids only, so use api-contract there: the legacy api-abi alias is accepted only in findings[].domain. When coverage.domains is present it must name at least one domain: an empty list is rejected rather than treated as an omitted field. See the coverage description for the uniqueness rule, which applies to this list too. */\n domains?: (\n | 'security'\n | 'correctness'\n | 'state-mutation'\n | 'error-handling'\n | 'performance'\n | 'dependency-hygiene'\n | 'test-coverage'\n | 'api-contract'\n )[]\n }\n /** Set noIssuesFound=true exactly when findings is empty and false whenever findings is non-empty; any other combination is rejected. */\n noIssuesFound?: boolean\n}\n\n/**\n * Write a todo list to track tasks for multi-step implementations. Use this frequently to maintain an updated step-by-step plan.\n */\nexport interface WriteTodosParams {\n /** List of todos with their completion status. Add ALL of the applicable tasks to the list, so you don't forget to do anything. Try to order the todos the same way you will complete them. Do not mark todos as completed if you have not completed them yet! */\n todos: {\n /** Description of the task */\n task: string\n /** Whether the task is completed */\n completed: boolean\n }[]\n}\n\n/**\n * Get parameters type for a specific tool\n */\nexport type GetToolParams = ToolParamsMap[T]\n" +export const toolsSource = "/**\n * Union type of all available tool names\n */\nexport type ToolName =\n | 'add_message'\n | 'ask_user'\n | 'check_background_agent'\n | 'check_job'\n | 'code_search'\n | 'end_turn'\n | 'edit_transaction'\n | 'edit_3d_asset'\n | 'find_files'\n | 'find_files_matching_content'\n | 'git_status'\n | 'git_branch'\n | 'get_task'\n | 'get_change_review_bundle'\n | 'inspect_workspace'\n | 'inspect_environment'\n | 'inspect_3d_asset'\n | 'get_affected_tests'\n | 'get_build_targets'\n | 'inspect_codebase_structure'\n | 'inspect_feature_completeness'\n | 'evaluate_audit_coverage'\n | 'glob'\n | 'kill_job'\n | 'list_directory'\n | 'list_jobs'\n | 'lookup_agent_info'\n | 'query_index'\n | 'read_docs'\n | 'read_files'\n | 'read_image'\n | 'render_3d_preview'\n | 'read_logs'\n | 'read_outline'\n | 'read_subtree'\n | 'replace_range'\n | 'rewrite_symbol'\n | 'render_ui'\n | 'run_file_change_hooks'\n | 'run_targeted_validation'\n | 'run_terminal_command'\n | 'set_messages'\n | 'set_output'\n | 'skill'\n | 'spawn_agents'\n | 'str_replace'\n | 'suggest_followups'\n | 'task_completed'\n | 'think_deeply'\n | 'update_plan_status'\n | 'web_search'\n | 'write_file'\n | 'write_audit_findings'\n | 'write_todos'\n\n/**\n * Map of tool names to their parameter types\n */\nexport interface ToolParamsMap {\n add_message: AddMessageParams\n ask_user: AskUserParams\n check_background_agent: CheckBackgroundAgentParams\n check_job: CheckJobParams\n code_search: CodeSearchParams\n end_turn: EndTurnParams\n edit_transaction: EditTransactionParams\n edit_3d_asset: Edit3dAssetParams\n find_files: FindFilesParams\n find_files_matching_content: FindFilesMatchingContentParams\n git_status: GitStatusParams\n git_branch: GitBranchParams\n get_task: GetTaskParams\n get_change_review_bundle: GetChangeReviewBundleParams\n inspect_workspace: InspectWorkspaceParams\n inspect_environment: InspectEnvironmentParams\n inspect_3d_asset: Inspect3dAssetParams\n get_affected_tests: GetAffectedTestsParams\n get_build_targets: GetBuildTargetsParams\n inspect_codebase_structure: InspectCodebaseStructureParams\n inspect_feature_completeness: InspectFeatureCompletenessParams\n evaluate_audit_coverage: EvaluateAuditCoverageParams\n glob: GlobParams\n kill_job: KillJobParams\n list_directory: ListDirectoryParams\n list_jobs: ListJobsParams\n lookup_agent_info: LookupAgentInfoParams\n query_index: QueryIndexParams\n read_docs: ReadDocsParams\n read_files: ReadFilesParams\n read_image: ReadImageParams\n render_3d_preview: Render3dPreviewParams\n read_logs: ReadLogsParams\n read_outline: ReadOutlineParams\n read_subtree: ReadSubtreeParams\n replace_range: ReplaceRangeParams\n rewrite_symbol: RewriteSymbolParams\n render_ui: RenderUiParams\n run_file_change_hooks: RunFileChangeHooksParams\n run_targeted_validation: RunTargetedValidationParams\n run_terminal_command: RunTerminalCommandParams\n set_messages: SetMessagesParams\n set_output: SetOutputParams\n skill: SkillParams\n spawn_agents: SpawnAgentsParams\n str_replace: StrReplaceParams\n suggest_followups: SuggestFollowupsParams\n task_completed: TaskCompletedParams\n think_deeply: ThinkDeeplyParams\n update_plan_status: UpdatePlanStatusParams\n web_search: WebSearchParams\n write_file: WriteFileParams\n write_audit_findings: WriteAuditFindingsParams\n write_todos: WriteTodosParams\n}\n\n/**\n * Add a new message to the conversation history. To be used for complex requests that can't be solved in a single step, as you may forget what happened!\n */\nexport interface AddMessageParams {\n role: 'user' | 'assistant'\n content: string\n}\n\n/**\n * Ask the user a list of multiple choice questions. Each question must have at least 2 options. The agent execution will pause until the user submits their answers.\n */\nexport interface AskUserParams {\n /** List of multiple choice questions to ask the user */\n questions: {\n /** The question to ask the user */\n question: string\n /** Optional short display label. Values longer than 18 Unicode code points are truncated instead of rejecting the question. */\n header?: string\n /** Array of answer options with label and optional description. */\n options: {\n /** The display text for this option */\n label: string\n /** Explanation shown when option is focused */\n description?: string\n }[]\n /** If true, allows selecting multiple options (checkbox). If false, single selection only (radio). */\n multiSelect?: boolean\n /** Validation rules for \"Other\" text input */\n validation?: {\n /** Maximum length for \"Other\" text input */\n maxLength?: number\n /** Minimum length for \"Other\" text input */\n minLength?: number\n /** Regex pattern for \"Other\" text input */\n pattern?: string\n /** Custom error message when pattern fails */\n patternError?: string\n }\n }[]\n}\n\n/**\n * Join/wait on a background agent turn started by spawn_agents({ background: true }): returns the sequenced agent_chunk events produced since the cursor plus the unified job state. Use it to observe a long-running background agent without blocking the turn.\n */\nexport interface CheckBackgroundAgentParams {\n /** The jobId returned by spawn_agents({ background: true }) for the background agent turn. */\n jobId: string\n /** Optional sequence cursor from a prior response. Polling is idempotent for an explicit cursor; nextCursor can be supplied on the next call. */\n cursor?: number\n /** Optional substring to wait for in the new streamed chunks before returning (follow mode). Returns early as soon as it appears in any chunk payload. Useful for waiting until a background agent emits a specific milestone (e.g. a tool_result or a text marker). */\n wait_for?: string\n /** Max seconds to wait for new chunks / the wait_for pattern. 0 (default) returns immediately with whatever new chunks exist (poll mode); >0 blocks up to this long (follow mode). */\n timeout_seconds?: number\n /** When true, explicitly cancel the running background agent before returning its final status. Defaults to false. */\n cancel?: boolean\n}\n\n/**\n * Join/wait on a background job started by run_terminal_command: returns the sequenced output events produced since the last check plus the unified job state and exit code. Use it to observe a long-running process without blocking the turn. To watch an arbitrary log file, start a `tail -f ` BACKGROUND job and check_job it with a wait_for pattern.\n */\nexport interface CheckJobParams {\n /** The jobId returned by run_terminal_command with process_type: BACKGROUND. */\n jobId: string\n /** Optional substring to wait for in the new output before returning (follow mode). Returns early as soon as it appears (e.g. \"Listening on\" / \"compiled successfully\"). */\n wait_for?: string\n /** Max seconds to wait for new output / the wait_for pattern. 0 (default) returns immediately with whatever new output exists (poll mode); >0 blocks up to this long (follow mode). */\n timeout_seconds?: number\n /** Follow mode only: SIGTERM the job on follow-timeout. Poll mode never kills. Default false. */\n kill_on_timeout?: boolean\n}\n\n/**\n * Search for string patterns in the project's files. This tool uses ripgrep (rg), a fast line-oriented search tool. Use this tool only when read_files is not sufficient to find the files you need.\n */\nexport interface CodeSearchParams {\n /** The pattern to search for. */\n pattern: string\n /** Optional safe ripgrep flags as one string or argv tokens (e.g., \"-i -g *.ts -A 2\" or [\"-i\", \"-g\", \"*.ts\", \"-A\", \"2\"]). Allowed: -i/--ignore-case, -S/--smart-case, -s/--case-sensitive, -w/--word-regexp, -F/--fixed-strings, -U/--multiline, --multiline-dotall, -g/--glob, -t/--type, -T/--type-not, plus context -A/-B/-C (and long forms). JSON quotes delimit the string; do not embed another quote pair around the entire expression. Line numbers are automatic; -n/--line-number are ignored. Output-shape flags such as -c/--count, --count-matches, -l, -v/--invert-match, -r/--replace, --exec, and -z/--null are rejected. */\n flags?: string | string[]\n /** Optional working directory or single file to search within, relative to the project root or absolute. Absolute paths may be outside the project. A directory becomes ripgrep's cwd and scopes the search under that path (plus existing blessed hidden dirs when no paths are given); a file scopes the search to that file only (process cwd = project root when the file is under the project, else the file's parent). Defaults to searching the entire project root. */\n cwd?: string\n /** Optional list of file and/or directory paths to search (relative to the project root, or absolute). When non-empty, ripgrep searches only these targets instead of the whole cwd tree (and does not auto-expand hidden dirs). Can be combined with a file cwd. */\n paths?: string[]\n /** Maximum number of results to return per file. Defaults to 15. There is also a global limit of 250 results across all files. */\n maxResults?: number\n}\n\n/**\n * End your turn, regardless of any new tool results that might be coming. This will allow the user to type another prompt.\n */\nexport interface EndTurnParams {}\n\n/**\n * Parameters for edit_transaction tool\n */\nexport interface EditTransactionParams {\n edits: (\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'str_replace'\n replacements: {\n oldString: string\n newString: string\n allowMultiple?: boolean\n occurrenceIndex?: number\n /** Optional authenticated cap.v3 readCapability copied verbatim from the matching fresh read_files editAnchor. */\n basedOnRead?: string\n /** For deletion replacements only (newString is empty): treat a missing oldString as an already-applied no-op. Use only for explicit idempotent cleanup retries, never for ordinary edits. When every requested change resolves to such a no-op - every replacement of a standalone str_replace call, or every edit of an edit_transaction - the call succeeds with zero file changes and the skip messages rather than failing. When combined with occurrenceIndex, a partially-applied cleanup also skips: fewer remaining exact occurrences than the requested index means that occurrence is treated as already applied. Only valid when newString is empty; both the input and provider schemas reject any other combination. */\n skipIfMissing?: boolean\n }[]\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n /** A structured edit dispatched by operation kind. */\n type: 'structured'\n /** Structured edit operation to apply to this file. */\n operation:\n | {\n /** Deterministic text insertion. */\n kind: 'insert_text'\n /** 1-indexed insertion position. */\n position: {\n /** 1-indexed target line. */\n line: number\n /** 1-indexed target column. */\n column: number\n }\n text: string\n }\n | {\n /** Language-aware import insertion. */\n kind: 'insert_import'\n /** Complete language-native import statement to add, e.g. \"import { foo } from 'bar'\", \"from app import value\", or \"use crate::value\". */\n importStatement: string\n }\n | {\n /** Language-aware import removal. */\n kind: 'remove_import'\n /** Complete language-native import statement to remove. Required unless moduleSpecifier is provided. */\n importStatement?: string\n /** Module specifier to remove imports from, e.g. \"react\" or \"./helper\". */\n moduleSpecifier?: string\n }\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'create'\n /** Exact bytes to write to the new file. */\n content: string\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'delete'\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'move'\n /** New project-relative path. The destination must be absent. */\n destinationPath: string\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'replace_range'\n readCapability: string\n startLine?: number\n endLine?: number\n occurrence?: {\n match: string\n occurrence?: number\n }\n newContent: string\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'rewrite_symbol'\n symbol: string\n content: string\n occurrence?: number\n /** Optional cap.v3 copied from the matching read_files symbol slice. It authorizes exactly the symbol and its contiguous preceding comment block. */\n readCapability?: string\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'patch'\n diff: string\n }\n | {\n /** Optional stable edit identifier echoed in diagnostics. */\n id?: string\n /** The file to edit. */\n path: string\n type: 'write_file'\n content: string\n /** Optional whole-file-covering cap.v3 from a fresh complete whole-file read. Only a full-file capability with a hash matching current content may authorize overwrite; partial ranges never authorize write_file. */\n basedOnRead?: string\n }\n )[]\n}\n\n/**\n * Parameters for edit_3d_asset tool\n */\nexport interface Edit3dAssetParams {\n /** Project-relative .blend path. */\n path: string\n /** Exact source hash returned by inspect_3d_asset. */\n source_hash: string\n operations: (\n | {\n type: 'rename_object'\n object: string\n new_name: string\n }\n | {\n type: 'set_object_transform'\n object: string\n location?: any[]\n rotation_degrees?: any[]\n scale?: any[]\n }\n | {\n type: 'set_render_resolution'\n width: number\n height: number\n percentage?: number\n }\n | {\n type: 'set_frame_range'\n start: number\n end: number\n }\n )[]\n}\n\n/**\n * Find several files related to a brief natural language description of the files or the name of a function or class you are looking for.\n */\nexport interface FindFilesParams {\n /** A brief natural language description of the files or the name of a function or class you are looking for. It's also helpful to mention a directory or two to look within. */\n prompt: string\n}\n\n/**\n * List unique file paths whose content matches a pattern, with optional symbol grouping. Built on top of ripgrep (rg).\n */\nexport interface FindFilesMatchingContentParams {\n /** Regex pattern (ripgrep syntax) to match file content against. */\n pattern: string\n /** Optional safe ripgrep flags as one string or argv tokens. Allowed: -i/--ignore-case, -S/--smart-case, -s/--case-sensitive, -w/--word-regexp, -F/--fixed-strings, -U/--multiline, --multiline-dotall, -g/--glob, -t/--type, -T/--type-not. Examples: \"-g *.ts -g *.tsx\" or [\"-g\", \"*.ts\", \"-g\", \"*.tsx\"]. Do not quote the entire expression inside the JSON string. Output-shape flags such as -c/--count, --count-matches, -l, -v/--invert-match, context -A/-B/-C, -r/--replace, --exec, and -z/--null are rejected (this tool forces -l or --json itself). Redundant -n/--line-number inputs are ignored. */\n flags?: string | string[]\n /** Optional working directory or single file to search within, relative to the project root or absolute. Absolute paths may be outside the project. A directory becomes ripgrep's cwd and scopes the search under that path (plus existing blessed hidden dirs); a file scopes the search to that file only (process cwd = project root when the file is under the project, else the file's parent). Defaults to the project root. */\n cwd?: string\n /** Maximum number of unique files to return. Defaults to 100. */\n maxFiles?: number\n /** When true, also return the names of the top-level symbols (functions, classes, methods, exports, constants) that contain each match, plus the per-file match count. Symbol extraction is heuristic and works best for JS/TS/Python/Go/Rust source files; languages without a recognized declaration shape produce an empty symbols list. */\n groupBySymbol?: boolean\n /** Maximum seconds to let ripgrep run before returning partial results. Defaults to 15. */\n timeoutSeconds?: number\n}\n\n/**\n * Read-only git status and (optionally) diff for the current project.\n */\nexport interface GitStatusParams {\n /** When true, also return the unified diff of uncommitted changes. */\n include_diff?: boolean\n /** When true with include_diff, returns the staged diff instead of unstaged. */\n staged?: boolean\n /** Optional path to scope status/diff to (relative to project root). */\n path?: string\n /** Maximum characters of diff output to return. Defaults to 40,000. */\n max_chars?: number\n}\n\n/**\n * Create a new git branch, optionally switching to it. Refuses to branch when the working tree is dirty unless `allow_dirty` is true.\n */\nexport interface GitBranchParams {\n /** Name of the branch to create. Must start with an alphanumeric character and contain only [a-zA-Z0-9._/-]. */\n branch_name: string\n /** When true (default), create AND switch to the branch (`git checkout -b`). When false, only create the branch (`git branch`), leaving the current branch checked out. */\n switch?: boolean\n /** When true, skip the dirty-tree refusal check. Defaults to false — the tool refuses to branch when the working tree has uncommitted changes. */\n allow_dirty?: boolean\n}\n\n/**\n * Parameters for get_task tool\n */\nexport interface GetTaskParams {\n /** Optional plan session slug. Defaults to .agents/ACTIVE_SESSION. */\n session?: string\n}\n\n/**\n * Parameters for get_change_review_bundle tool\n */\nexport interface GetChangeReviewBundleParams {\n max_chars?: number\n}\n\n/**\n * Inspect the current repository/worktree identity and Git state without modifying it.\n */\nexport interface InspectWorkspaceParams {}\n\n/**\n * Parameters for inspect_environment tool\n */\nexport interface InspectEnvironmentParams {}\n\n/**\n * Parameters for inspect_3d_asset tool\n */\nexport interface Inspect3dAssetParams {\n /** Project-relative 3D asset path. */\n path: string\n}\n\n/**\n * Parameters for get_affected_tests tool\n */\nexport interface GetAffectedTestsParams {\n files: string[]\n}\n\n/**\n * Parameters for get_build_targets tool\n */\nexport interface GetBuildTargetsParams {\n files: string[]\n}\n\n/**\n * Parameters for inspect_codebase_structure tool\n */\nexport interface InspectCodebaseStructureParams {\n scope?: string[]\n}\n\n/**\n * Parameters for inspect_feature_completeness tool\n */\nexport interface InspectFeatureCompletenessParams {\n feature: string\n snapshot_id: string\n scope?: string[]\n}\n\n/**\n * Parameters for evaluate_audit_coverage tool\n */\nexport interface EvaluateAuditCoverageParams {\n snapshot_id: string\n structural_receipts: {\n schema_version: 1\n snapshot_id: string\n shard_id: string\n subsystem_ids: string[]\n files: string[]\n domains: (\n | 'security'\n | 'correctness'\n | 'state-mutation'\n | 'error-handling'\n | 'performance'\n | 'dependency-hygiene'\n | 'test-coverage'\n | 'api-contract'\n )[]\n }[]\n features: {\n schema_version: 1\n snapshot_id: string\n feature: string\n evidence_kind: 'heuristic' | 'verified'\n evidence: {\n entrypoints: string[]\n implementation: string[]\n consumers: string[]\n tests: string[]\n docs: string[]\n failure_states: string[]\n }\n }[]\n out_of_scope?: {\n id: string\n reason: string\n }[]\n scope?: string[]\n}\n\n/**\n * Search for files matching a glob pattern. Returns matching file paths sorted by modification time (newest first, then path for deterministic ties).\n */\nexport interface GlobParams {\n /** Glob pattern to match files against (e.g., *.js, src/glob/*.ts, glob/test/glob/*.go). */\n pattern: string\n /** Optional working directory or file path, relative to project root. If a directory, the glob pattern is matched against paths relative to this cwd, while returned files remain project-relative. If a file path, the pattern is matched against that file only (full path or basename). If not provided, searches from project root. */\n cwd?: string\n}\n\n/**\n * Cancel a background job started by run_terminal_command.\n */\nexport interface KillJobParams {\n /** The jobId returned by run_terminal_command with process_type: BACKGROUND. */\n jobId: string\n /** Signal to send. Defaults to SIGTERM; use SIGKILL only if graceful termination fails. */\n signal?: 'SIGTERM' | 'SIGKILL'\n}\n\n/**\n * List files and directories in the specified path. Returns separate arrays of file names and directory names.\n */\nexport interface ListDirectoryParams {\n /** Directory path to list, relative to the project root. */\n path: string\n}\n\n/**\n * List this run's background jobs (shell processes and background agents, running and settled) with statuses, bucketed pending process/log output relative to the last check_job consumer cursor (agents usually show pending: 'none'), and a gap flag.\n */\nexport interface ListJobsParams {}\n\n/**\n * Retrieve information about an agent by ID\n */\nexport interface LookupAgentInfoParams {\n /** Agent ID (short local or full published format) */\n agentId: string\n}\n\n/**\n * Query the local codebase graph index to find relevant files ranked by symbol names, imports, headings, paths, doc concepts, and graph relationships. The index is built automatically on startup.\n */\nexport interface QueryIndexParams {\n /** Natural language query or keyword terms describing the files you are looking for. Optional for graph modes when from/to paths are provided. For example: \"authentication\", \"database migrations\", \"editor mutation logic\", \"React components\". */\n query?: string\n /** Maximum number of results to return. Defaults to 20. */\n limit?: number\n /** Optional list of file extensions to filter results (without dot). E.g. [\"ts\", \"tsx\"] for TypeScript only. */\n fileTypes?: string[]\n /** Optional normalized project-relative directory prefixes. Results outside every prefix are excluded before ranking/limiting. */\n pathPrefixes?: string[]\n /** search|explain|neighbors|path|commands|references — see tool description. */\n mode?: 'search' | 'neighbors' | 'path' | 'explain' | 'commands' | 'references'\n /** Optional source file path for neighbors, path, and references modes. */\n from?: string\n /** Optional target file path for path mode. Also used as the seed file for references mode when from is omitted or not indexed. */\n to?: string\n}\n\n/**\n * Fetch up-to-date documentation for libraries and frameworks using Context7 API.\n */\nexport interface ReadDocsParams {\n /** The library or framework name (e.g., \"Next.js\", \"MongoDB\", \"React\"). Use the official name as it appears in documentation if possible. Only public libraries available in Context7's database are supported, so small or private libraries may not be available. */\n libraryTitle: string\n /** Specific topic to focus on (e.g., \"routing\", \"hooks\", \"authentication\") */\n topic: string\n /** Optional maximum number of tokens to return. Defaults to 10000. Values less than 10000 are automatically increased to 10000. */\n max_tokens?: number\n}\n\n/**\n * Read multiple files from disk and return their contents. Use this tool to read as many files as would be helpful to answer the user's request.\n */\nexport interface ReadFilesParams {\n /** Whole-file paths to read. Complete results include editAnchor.readCapability for follow-up edits. */\n paths?: string[]\n /** 1-indexed inclusive line ranges. Sole `paths` entry infers missing path. */\n ranges?: {\n /** Project-relative file path. */\n path: string\n /** 1-indexed inclusive start line. Defaults to 1. */\n startLine?: number\n /** 1-indexed inclusive end line. Defaults to the last line. */\n endLine?: number\n }[]\n /** Contiguous line windows; each complete window mints a scoped cap.v3 editAnchor. */\n windows?: {\n /** File path to read in contiguous line windows, relative to the project root. */\n path: string\n /** Lines per window. Defaults to 400, capped at 5000. */\n windowSize?: number\n /** 1-indexed window number to return. Omit to get the window manifest (totalLines, windowSize, windowCount) plus the first window. */\n window?: number\n }[]\n /** Literal-anchored context blocks with a scoped cap.v3 editAnchor per block. */\n around?: {\n /** File path to read a content-anchored block from, relative to the project root. */\n path: string\n /** Exact literal string to anchor on. Robust to line-number drift. */\n match: string\n /** 1-indexed occurrence of `match` to anchor on. Defaults to 1. */\n occurrence?: number\n /** Lines of context to include on each side of the match, clamped at file boundaries. Defaults to 40, capped at 2000. */\n contextLines?: number\n }[]\n /** Nth top-level symbol by name (rewrite_symbol occurrence semantics); prefer batch `symbols` when possible. */\n symbol?: {\n /** File path to extract a symbol slice from, relative to the project root. */\n path: string\n /** Top-level symbol name (function, class, interface, method) to pull, as shown by read_outline. */\n name: string\n /** When multiple top-level symbols share this name, the 1-indexed one to return. Defaults to 1. Matches rewrite_symbol occurrence semantics. */\n occurrence?: number\n }[]\n /** Named symbol slices with editAnchors; prefer over full reads when names are known. */\n symbols?: {\n /** Project-relative file path. */\n path: string\n /** Symbol names to slice. */\n names: string[]\n }[]\n}\n\n/**\n * Read image files from disk and return them as model-visible image media.\n */\nexport interface ReadImageParams {\n /** List of image file paths to read. */\n paths: string[]\n}\n\n/**\n * Parameters for render_3d_preview tool\n */\nexport interface Render3dPreviewParams {\n /** Project-relative 3D asset path. */\n path: string\n views?: ('camera' | 'perspective' | 'front' | 'side' | 'top')[]\n mode?: 'material' | 'clay' | 'wireframe'\n width?: number\n height?: number\n}\n\n/**\n * Read the last N lines from a log/text file or background job log without starting a background tail process.\n */\nexport interface ReadLogsParams {\n /** Path to the log file, relative to the project root unless absolute. Required unless jobId is provided. */\n path?: string\n /** Background job id returned by run_terminal_command(process_type: BACKGROUND). When provided, reads the job log file directly. */\n jobId?: string\n /** Number of trailing lines to read. Defaults to 200. */\n lines?: number\n /** Maximum characters to return. Defaults to 20,000. */\n max_chars?: number\n}\n\n/**\n * Generate an outline of imports, exports, classes, methods, and function signatures in a source file without reading the entire implementation.\n */\nexport interface ReadOutlineParams {\n /** File path to generate the AST-like outline for, relative to the project root. */\n path: string\n}\n\n/**\n * Read one or more directory subtrees (as a blob including subdirectories, file names, and parsed variables within each source file) or return parsed variable names for files. If no paths are provided, returns the entire project tree.\n */\nexport interface ReadSubtreeParams {\n /** List of paths to directories or files. Relative to the project root. If omitted, the entire project tree is used. */\n paths?: string[]\n /** Maximum token budget for the subtree blob; the tree will be truncated to fit within this budget by first dropping file variables and then removing the most-nested files and directories. */\n maxTokens?: number\n}\n\n/**\n * Replace all of, a contained sub-range of, or the Nth literal occurrence inside content observed through one fresh cap.v3 read capability.\n */\nexport interface ReplaceRangeParams {\n /** The path to the file to edit. */\n path: string\n /** Copy the cap.v3 readCapability verbatim from the matching fresh read_files editAnchor. The token supplies the observed line bounds and content hash. */\n readCapability: string\n /** Optional 1-indexed target start within the capability-covered range. Omit with endLine to replace the complete observed range. */\n startLine?: number\n /** Optional 1-indexed target end within the capability-covered range. Omit with startLine to replace the complete observed range. */\n endLine?: number\n /** Optional occurrence targeting: replace the 1-indexed occurrence (default 1) of the exact literal match found inside the capability-authorized range. Mutually exclusive with startLine/endLine. */\n occurrence?: {\n match: string\n occurrence?: number\n }\n /** Complete replacement content for the selected line range. */\n newContent: string\n}\n\n/**\n * Replace a whole symbol's definition by name using the file's syntax tree, without copying its current text. Resolves the exact AST range and applies it through the safe str_replace path (atomic, anchored).\n */\nexport interface RewriteSymbolParams {\n /** File path containing the symbol, relative to the project root. */\n path: string\n /** Name of the function/class/method/type/interface to replace (as shown by read_outline). */\n symbol: string\n /** The complete new source for the symbol, replacing its entire current definition (e.g. the whole function including its signature and body). Provide REAL newlines/tabs in the string — literal backslash-n (\\n) and backslash-t (\\t) sequences are not interpreted and will be written verbatim into the file. This matches str_replace. */\n content: string\n /** When multiple top-level symbols share this name, the 1-indexed one to replace. */\n occurrence?: number\n /** Optional cap.v3 copied from the matching read_files symbol slice. Under strict read-before-edit this authorizes exactly the symbol and its contiguous preceding comment block. */\n readCapability?: string\n}\n\n/**\n * Render a small interactive UI widget in the Openbuff CLI. Currently supports a button that opens a link.\n */\nexport interface RenderUiParams {\n /** The UI widget to render. */\n widget: {\n /** Widget type. Currently, the only supported widget is button. */\n type: 'button'\n /** Short button label shown to the user. */\n text: string\n /** The http:// or https:// URL to open when the user clicks the button. */\n link: string\n /** Theme-aware color treatment. Use primary for the main action and secondary for lower-emphasis actions. */\n variant?: 'primary' | 'secondary'\n }\n}\n\n/**\n * Parameters for run_file_change_hooks tool\n */\nexport interface RunFileChangeHooksParams {\n /** List of file paths that were changed and should trigger file change hooks */\n files: string[]\n}\n\n/**\n * Parameters for run_targeted_validation tool\n */\nexport interface RunTargetedValidationParams {\n snapshot_id: string\n files: string[]\n artifact_kinds?: string[]\n}\n\n/**\n * Execute a CLI command from the **project root** (different from the user's cwd).\n */\nexport interface RunTerminalCommandParams {\n /** CLI command valid for user's OS. */\n command: string\n /** SYNC (default) for finite commands that exit: waits and returns output. BACKGROUND only for long-running or never-exiting processes (dev servers, watchers, log tails): starts a detached job and returns a jobId immediately so the turn is not blocked. Live job_update already drives the user UI; use check_job for agent-side readiness/exitCode/join, not solely for user progress. */\n process_type?: 'SYNC' | 'BACKGROUND'\n /** For BACKGROUND commands only: keep the job running if the owning request is cancelled. Defaults to false. */\n detach?: boolean\n /** The working directory to run the command in. Default is the project root. */\n cwd?: string\n /** Wall-clock bound in seconds for SYNC commands. Omit or use -1 for no timeout (the default). Does not apply to BACKGROUND commands. */\n timeout_seconds?: number\n /** Runtime-managed background job owner; agents must omit. */\n owner?: {\n clientSessionId: string\n rootRunId: string\n parentRunId: string\n parentAgentId: string\n }\n}\n\n/**\n * Atomically replace conversation history and, when supplied, commit a validated structured task-memory revision.\n */\nexport interface SetMessagesParams {\n messages: any\n taskMemory?: {\n schemaVersion: 1\n goal?: string\n requirements?: string[]\n decisions?: string[]\n filesInspected?: string[]\n editsMade?: string[]\n validationResults?: string[]\n reviewReceipts?: string[]\n blockers?: string[]\n nextActions?: string[]\n historicalSummary?: string\n evidence?: {\n id: string\n kind:\n | 'requirement'\n | 'decision'\n | 'read'\n | 'edit'\n | 'validation'\n | 'review'\n | 'blocker'\n | 'handoff'\n | 'note'\n summary: string\n source?: string\n path?: string\n freshnessHash?: string\n workspaceRevision?: number\n verifiedAt?: number\n supersedes?: string[]\n stale?: boolean\n }[]\n workspaceRevision?: number\n workspaceSnapshotId?: string\n }\n expectedTaskMemoryRevision?: number\n}\n\n/**\n * JSON object to set as the agent output. The shape of the parameters are specified dynamically further down in the conversation. This completely replaces any previous output. If the agent was spawned, this value will be passed back to its parent. If the agent has an outputSchema defined, the output will be validated against it.\n */\nexport interface SetOutputParams {\n data?: Record\n [key: string]: any\n}\n\n/**\n * Load a skill by name to get its full instructions. Skills provide reusable behaviors and instructions.\n */\nexport interface SkillParams {\n /** The name of the skill to load */\n name: string\n}\n\n/**\n * Spawn up to 12 agents and send a prompt and/or parameters to each of them. These agents will run in parallel. Note that that means they will run independently. Split larger work into bounded waves. If you need to run agents sequentially, use spawn_agents with one agent at a time instead.\n */\nexport interface SpawnAgentsParams {\n agents: {\n /** Agent to spawn. Must be a name from the live \"You can spawn the following agents\" catalog (hyphenated ids; underscores accepted). */\n agent_type: string\n /** Prompt to send to the agent */\n prompt?: string\n /** If true, return jobId immediately and run as in-process coroutine; poll with check_background_agent. Defaults to false (blocking). Cannot outlive this CLI session. */\n background?: boolean\n /** Optional structured handoff; additive — non-consumers still get prompt/params. */\n handoff?:\n | {\n schemaVersion: 1\n taskId: string\n role:\n | 'orchestrator'\n | 'explorer'\n | 'thinker'\n | 'editor'\n | 'repair-editor'\n | 'test-writer'\n | 'doc-writer'\n | 'dependency-manager'\n | 'debugger'\n | 'validator'\n | 'reviewer'\n | 'security-reviewer'\n | 'committer'\n | 'synthesizer'\n | 'specialist'\n | 'general'\n objective: string\n requirements: {\n id: string\n text: string\n required: boolean\n }[]\n acceptanceCriteria: {\n id: string\n behavior: string\n verification: string\n }[]\n context:\n | {\n path: string\n symbols: string[]\n reason: string\n confidence: 'confirmed' | 'inferred' | 'unknown'\n freshnessHash?: string\n workspaceRevision?: number\n }[]\n | Record\n | string\n currentBehavior?: string\n desiredBehavior?: string\n invariants?: string[]\n nonGoals: string[]\n risks?: string[]\n unknowns?: string[]\n findings: {\n id: string\n text: string\n files: string[]\n snapshotFingerprint: string\n }[]\n permissions: {\n readablePaths: string[]\n writablePaths: string[]\n allowedTools: string[]\n }\n workspaceRevision?: number\n workspaceSnapshotId?: string\n summary?: string\n artifacts?: string[]\n successCriteria?: string[]\n constraints?: string[]\n }\n | Record\n /** Parameters object for the agent */\n params?: {\n /** Terminal command to run (basher, tmux-cli) */\n command?: string\n /** What information from the command output is desired (basher) */\n what_to_summarize?: string\n /** Timeout for command in seconds. Omit or -1 for no timeout (default). */\n timeout_seconds?: number\n /** Save full command output to a /tmp log and extract failure lines for long SYNC command output (basher) */\n save_full_log?: boolean\n /** grep -E failure extraction pattern used with save_full_log (basher) */\n failure_pattern?: string\n /** Maximum extracted failure lines to return with save_full_log (basher) */\n max_failure_lines?: number\n /** Relevant file paths to read (general-agent) */\n filePaths?: string[]\n /** Relevant directory paths to inventory (general-agent) */\n directoryPaths?: string[]\n /** Directories to search within (file-picker) */\n directories?: string[]\n /** Starting URL to navigate to (browser-use) */\n url?: string\n /** Exact task-owned paths eligible for staging (git-committer) */\n owned_paths?: string[]\n /** Optional branch to create or switch to (git-committer) */\n branch_name?: string\n /** Create and switch to branch_name when true (git-committer) */\n branch_switch?: boolean\n /** Allow branch create/switch on a dirty worktree (git-committer) */\n allow_dirty_branch?: boolean\n /** Push the resulting feature branch when authorized (git-committer) */\n push?: boolean\n /** Remote used for fetch/push (git-committer) */\n remote?: string\n /** Optional gate-assigned snapshot token (reviewer specialists). Runtime-owned spawns pass the gate-assigned v3:… token; manual spawns omit this key entirely. */\n snapshot_id?: string\n /** Changed file paths to review (security-reviewer) */\n changed_files?: string[]\n /** Opaque snapshot token to echo (security-reviewer) */\n snapshot_fingerprint?: string\n /** Package manager selected from repository manifests (dependency-manager) */\n manager?: string\n /** Dependency operation: add, remove, sync, restore, or update (dependency-manager) */\n operation?: string\n /** Exact package specifications (dependency-manager) */\n packages?: string[]\n /** Optional workspace selector (dependency-manager) */\n workspace?: string\n /** GitHub repository URL to clone (librarian) */\n repoUrl?: string\n /** Retain the owned /tmp clone after completion (librarian) */\n retainClone?: boolean\n /** Optional search or path patterns */\n patterns?: string[]\n /** Exact files in scope (reviewer specialists) */\n files?: string[]\n /** Optional agent-specific prompts */\n prompts?: string[]\n [key: string]: any\n }\n }[]\n}\n\n/**\n * Parameters for str_replace tool\n */\nexport interface StrReplaceParams {\n /** The file to edit. */\n path: string\n atomic?: boolean\n replacements: {\n oldString: string\n newString: string\n allowMultiple?: boolean\n occurrenceIndex?: number\n /** Optional authenticated cap.v3 readCapability copied verbatim from the matching fresh read_files editAnchor. */\n basedOnRead?: string\n /** For deletion replacements only (newString is empty): treat a missing oldString as an already-applied no-op. Use only for explicit idempotent cleanup retries, never for ordinary edits. When every requested change resolves to such a no-op - every replacement of a standalone str_replace call, or every edit of an edit_transaction - the call succeeds with zero file changes and the skip messages rather than failing. When combined with occurrenceIndex, a partially-applied cleanup also skips: fewer remaining exact occurrences than the requested index means that occurrence is treated as already applied. Only valid when newString is empty; both the input and provider schemas reject any other combination. */\n skipIfMissing?: boolean\n }[]\n}\n\n/**\n * Suggest clickable followup prompts to the user. Each followup becomes a card the user can click to send that prompt.\n */\nexport interface SuggestFollowupsParams {\n /** List of suggested followup prompts the user can click to send */\n followups: {\n /** The full prompt text to send as a user message when clicked */\n prompt: string\n /** Short display label for the card (defaults to truncated prompt if not provided) */\n label?: string\n }[]\n}\n\n/**\n * Signal that the task is complete. Use this tool when:\n- The user's request is completely fulfilled\n- You need clarification from the user before continuing\n- You are stuck or need help from the user to continue\n\nThis tool explicitly marks the end of your work on the current task.\n */\nexport interface TaskCompletedParams {}\n\n/**\n * Deeply consider complex tasks by brainstorming approaches and tradeoffs step-by-step.\n */\nexport interface ThinkDeeplyParams {\n /** Detailed step-by-step analysis. Initially keep each step concise (max ~5-7 words per step). */\n thought: string\n}\n\n/**\n * Parameters for update_plan_status tool\n */\nexport interface UpdatePlanStatusParams {\n /** Artifact path. Must be `.agents/sessions//PLAN.md`, `.agents/sessions//STATUS.md`, or `.agents/sessions//LESSONS.md`. Absolute paths and `..` traversal are rejected. Editing PLAN.md is permitted only for tri-state task toggles (not full overwrites). */\n path: string\n /** Targeted updates applied in order. Each entry rewrites at most one matching checklist line; unmatched updates fall through to `append`. */\n updates?: {\n /** Stable task ID at the start of a checklist line (for example `P2-T3`). Preferred over substring matching. */\n taskId?: string\n /** Substring of the existing task/checklist line to match (case-insensitive). The first matching `- [ ]`/`-[x]`/`-[~]`/`-[/]`/`-[!]` line in the artifact will be updated in place. */\n task?: string\n /** When provided, sets the checkbox state of the matched line (true -> `[x]`, false -> `[ ]`). Ignored when `status` is also provided. */\n completed?: boolean\n /** Explicit tri-state task status. When provided, overrides `completed`. Transitions a task to `in_progress` (`[~]`), `done` (`[x]`), `cancelled` (`[/]`), `blocked` (`[!]`), or back to `pending` (`[ ]`). */\n status?: 'pending' | 'in_progress' | 'done' | 'cancelled' | 'blocked'\n /** Optional short note to append to the matched line in parentheses. Preserves any existing trailing text on the line. */\n note?: string\n }[]\n /** Optional delimited entry appended at the end of the artifact (used when there is no matching task line for the change being recorded). */\n append?: {\n /** Short heading for an appended entry. Used to form a clearly delimited block (`## — `). */\n heading: string\n /** Markdown body for the appended entry. Written verbatim under the heading. */\n body: string\n }\n /** Optional session-level status transition. When provided, `.agents/sessions//STATE.json` is created or updated to reflect the new lifecycle status. */\n sessionStatus?:\n | 'draft'\n | 'ready'\n | 'active'\n | 'executing'\n | 'validating'\n | 'reviewing'\n | 'blocked'\n | 'paused'\n | 'completed'\n | 'archived'\n /** Optional current-task pointer written as a `` annotation in PLAN.md. Pass an empty string or omit to clear the pointer. Only takes effect when path targets PLAN.md. */\n currentTask?: string\n /** Optional STATE.json compare-and-swap revision. The update fails without writing when the current revision differs. */\n expectedRevision?: number\n /** Validation or review evidence associated with a stable task ID. Completing a PLAN task requires a passed validation checkpoint with receiptIds. */\n checkpoint?: {\n taskId: string\n phase: 'validation' | 'review'\n passed: boolean\n summary?: string\n receiptIds?: string[]\n }\n}\n\n/**\n * Search the web for current information, or fetch the content of a specific URL.\n */\nexport interface WebSearchParams {\n /** The search query to find relevant web content. Required unless url is provided. */\n query?: string\n /** A specific URL to fetch and read the full text content of. When provided, fetches this page directly instead of searching. Useful for reading documentation, GitHub READMEs, blog posts, or any public web page. */\n url?: string\n /** Search depth - 'standard' for quick results, 'deep' for more comprehensive search. Default is 'standard'. Ignored when url is provided. */\n depth?: 'standard' | 'deep'\n /** When fetching a URL, also extract and return links found on the page. Enables navigation by letting you see what pages are linked. Default: true. */\n include_links?: boolean\n /** Maximum number of links to extract when include_links is true. Default: 40. */\n max_links?: number\n}\n\n/**\n * Create or overwrite a file with the given content.\n */\nexport interface WriteFileParams {\n /** Path to the file relative to the **project root** */\n path: string\n /** What the change is intended to do in only one sentence. */\n instructions: string\n /** Complete file content to write to the file. */\n content: string\n /** Optional whole-file-covering cap.v3 from a fresh complete whole-file read (paths or full-file range). Only a capability that covers the entire current file (startLine=1 through the current line count) with a hash matching current content may authorize overwrite; partial range capabilities never authorize write_file. */\n basedOnRead?: string\n}\n\n/**\n * Parameters for write_audit_findings tool\n */\nexport interface WriteAuditFindingsParams {\n /** Existing durable audit session slug under .agents/sessions/. Accepts only a short identifier token: 1 to 100 characters of letters, digits, dot, underscore, or dash, and neither `.` nor `..` on its own. */\n sessionSlug: string\n /** Unique shard identifier used as the findings filename. Accepts only a short identifier token: 1 to 100 characters of letters, digits, dot, underscore, or dash, and neither `.` nor `..` on its own. */\n shardId: string\n /** Exact snapshotId returned by inspect_codebase_structure, such as its 64-character sha256 digest. Required for a directly composable structuralReceipt; omitted only for legacy callers. Accepts only a short identifier token: 1 to 100 characters of letters, digits, dot, underscore, or dash, and neither `.` nor `..` on its own. When snapshotId and coverage.domains are both present the call receives a structuralReceipt, so coverage.subsystemIds and coverage.files must each name at least one entry: evaluate_audit_coverage rejects a receipt whose subsystem_ids or files list is empty. */\n snapshotId?: string\n /** Each findings entry rejects control and Unicode format characters in title, risk, fix, and evidence — NUL, any other control character, and the U+2028/U+2029 line separators — while still accepting tabs and line breaks in that prose. findings[].path is a location rather than prose, so it must be a single-line value with none of those characters and no tabs or line breaks; it is trimmed, and the trimmed value is the one rendered into the finding heading. */\n findings: {\n severity: 'CRITICAL' | 'HIGH' | 'MEDIUM' | 'LOW'\n domain:\n | 'security'\n | 'correctness'\n | 'state-mutation'\n | 'error-handling'\n | 'performance'\n | 'dependency-hygiene'\n | 'test-coverage'\n | 'api-contract'\n | 'api-abi'\n path: string\n line?: number\n title: string\n risk: string\n fix: string\n evidence: string\n }[]\n /** Every coverage list must name each entry at most once: a repeated file, subsystemId, featureId, or domain is rejected rather than counted twice. Entries are compared after trimming surrounding whitespace, and the trimmed value is what reaches the artifact and the receipt, so two spellings that differ only in whitespace are the same entry. Every coverage files, subsystemIds, and featureIds entry must be a single-line value: tabs, carriage returns, newlines, NUL, any other control or Unicode format character, and the U+2028/U+2029 line separators are rejected. Entries are trimmed, and the trimmed value is the one uniqueness is judged on. */\n coverage: {\n /** See the coverage description for the single-line hygiene rule, which applies to this list too. See the coverage description for the uniqueness rule, which applies to this list too. */\n subsystemIds: string[]\n /** See the coverage description for the single-line hygiene rule, which applies to this list too. See the coverage description for the uniqueness rule, which applies to this list too. */\n featureIds: string[]\n /** See the coverage description for the single-line hygiene rule, which applies to this list too. See the coverage description for the uniqueness rule, which applies to this list too. */\n files: string[]\n /** coverage.domains accepts canonical domain ids only, so use api-contract there: the legacy api-abi alias is accepted only in findings[].domain. When coverage.domains is present it must name at least one domain: an empty list is rejected rather than treated as an omitted field. See the coverage description for the uniqueness rule, which applies to this list too. */\n domains?: (\n | 'security'\n | 'correctness'\n | 'state-mutation'\n | 'error-handling'\n | 'performance'\n | 'dependency-hygiene'\n | 'test-coverage'\n | 'api-contract'\n )[]\n }\n /** Set noIssuesFound=true exactly when findings is empty and false whenever findings is non-empty; any other combination is rejected. */\n noIssuesFound?: boolean\n}\n\n/**\n * Write a todo list to track tasks for multi-step implementations. Use this frequently to maintain an updated step-by-step plan.\n */\nexport interface WriteTodosParams {\n /** List of todos with their completion status. Add ALL of the applicable tasks to the list, so you don't forget to do anything. Try to order the todos the same way you will complete them. Do not mark todos as completed if you have not completed them yet! */\n todos: {\n /** Description of the task */\n task: string\n /** Whether the task is completed */\n completed: boolean\n }[]\n}\n\n/**\n * Get parameters type for a specific tool\n */\nexport type GetToolParams = ToolParamsMap[T]\n" export const utilTypesSource = "// ===== JSON Types =====\nexport type JSONValue =\n | null\n | string\n | number\n | boolean\n | JSONObject\n | JSONArray\n\nexport type JSONObject = { [key: string]: JSONValue }\n\nexport type JSONArray = JSONValue[]\n\n/**\n * JSON Schema definition (for prompt schema or output schema)\n */\nexport type JsonSchema = {\n type?:\n | 'object'\n | 'array'\n | 'string'\n | 'number'\n | 'boolean'\n | 'null'\n | 'integer'\n description?: string\n properties?: Record\n required?: string[]\n enum?: Array\n [k: string]: unknown\n}\nexport type JsonObjectSchema = JsonSchema & { type: 'object' }\n\n// ===== Data Content Types =====\nexport type DataContent = string | Uint8Array | ArrayBuffer | Buffer\n\n// ===== Provider Metadata Types =====\nexport type ProviderMetadata = Record>\n\n// ===== Content Part Types =====\nexport type TextPart = {\n type: 'text'\n text: string\n providerOptions?: ProviderMetadata\n}\n\nexport type ImagePart = {\n type: 'image'\n image: DataContent\n mediaType?: string\n providerOptions?: ProviderMetadata\n}\n\nexport type FilePart = {\n type: 'file'\n data: DataContent\n filename?: string\n mediaType: string\n providerOptions?: ProviderMetadata\n}\n\nexport type ReasoningPart = {\n type: 'reasoning'\n text: string\n providerOptions?: ProviderMetadata\n}\n\nexport type ToolCallPart = {\n type: 'tool-call'\n toolCallId: string\n toolName: string\n input: Record\n providerOptions?: ProviderMetadata\n providerExecuted?: boolean\n}\n\nexport type ToolResultOutput =\n | {\n type: 'json'\n value: JSONValue\n }\n | {\n type: 'media'\n data: string\n mediaType: string\n }\n\n// ===== Message Types =====\nexport type AuxiliaryMessageData = {\n providerOptions?: ProviderMetadata\n tags?: string[]\n\n /** @deprecated Use tags instead. */\n timeToLive?: 'agentStep' | 'userPrompt'\n /** @deprecated Use tags instead. */\n keepDuringTruncation?: boolean\n /** @deprecated Use tags instead. */\n keepLastTags?: string[]\n}\n\nexport type SystemMessage = {\n role: 'system'\n content: TextPart[]\n} & AuxiliaryMessageData\n\nexport type UserMessage = {\n role: 'user'\n content: (TextPart | ImagePart | FilePart)[]\n} & AuxiliaryMessageData\n\nexport type AssistantMessage = {\n role: 'assistant'\n content: (TextPart | ReasoningPart | ToolCallPart)[]\n} & AuxiliaryMessageData\n\nexport type ToolMessage = {\n role: 'tool'\n toolCallId: string\n toolName: string\n content: ToolResultOutput[]\n} & AuxiliaryMessageData\n\nexport type Message =\n | SystemMessage\n | UserMessage\n | AssistantMessage\n | ToolMessage\n\n// ===== MCP Server Types =====\n\n/**\n * MCP server configuration for stdio-based servers.\n *\n * Environment variables in `env` can be:\n * - A plain string value (hardcoded, e.g., `'production'`)\n * - A `$VAR_NAME` reference to read from local environment (e.g., `'$NOTION_TOKEN'`)\n *\n * The `$VAR_NAME` syntax reads from `process.env.VAR_NAME` at agent load time.\n * This keeps secrets out of your agent definitions - store them in `.env.local` instead.\n *\n * @example\n * ```typescript\n * env: {\n * // Read NOTION_TOKEN from local .env file\n * NOTION_TOKEN: '$NOTION_TOKEN',\n * // Read MY_API_KEY from local env, pass as API_KEY to MCP server\n * API_KEY: '$MY_API_KEY',\n * // Hardcoded value (non-secret)\n * NODE_ENV: 'production',\n * }\n * ```\n */\nexport type MCPConfig =\n | {\n type?: 'stdio'\n command: string\n args?: string[]\n env?: Record\n }\n | {\n type?: 'http' | 'sse'\n url: string\n params?: Record\n headers?: Record\n }\n\n// ============================================================================\n// Logger Interface\n// ============================================================================\nexport interface Logger {\n debug: (data: any, msg?: string) => void\n info: (data: any, msg?: string) => void\n warn: (data: any, msg?: string) => void\n error: (data: any, msg?: string) => void\n}\n" diff --git a/common/src/constants/prompt-sections.ts b/common/src/constants/prompt-sections.ts index eb6c6a03a8..2a53a78727 100644 --- a/common/src/constants/prompt-sections.ts +++ b/common/src/constants/prompt-sections.ts @@ -165,14 +165,16 @@ Gather the exact source and snapshot evidence before spawning. Advisory speciali | Agent family | Required \`params\` | Rejected | Notes | |---|---|---|---| -| Reviewer-family (\`product-reviewer\`, \`performance-specialist\`, \`reliability-reviewer\`, \`migration-reviewer\`, \`compatibility-reviewer\`, \`accessibility-reviewer\`, \`ux-visual-reviewer\`, \`dependency-reviewer\`, \`evaluator\`) | \`params.snapshot_id\` = gate-assigned opaque \`v3:<64-hex>\` token from the parent gate | bare hex or missing token | Spawning with the wrong or missing snapshot key fails the spawn | -| \`security-reviewer\` (exception) | \`params.changed_files\` + \`params.snapshot_fingerprint\` | \`params.snapshot_id\` | Rejects \`snapshot_id\`; requires file list + fingerprint only | +| Reviewer-family (\`product-reviewer\`, \`performance-specialist\`, \`reliability-reviewer\`, \`migration-reviewer\`, \`compatibility-reviewer\`, \`accessibility-reviewer\`, \`ux-visual-reviewer\`, \`dependency-reviewer\`, \`evaluator\`) | \`params.snapshot_id\` = gate-assigned opaque \`v3:<64-hex>\` token from the parent gate for runtime-owned spawns; omitted entirely for manual spawns | When supplied, bare hex or a wrong value fails the spawn; manual spawns must omit the key | +| \`security-reviewer\` (exception) | \`params.changed_files\` + \`params.snapshot_fingerprint\` on every spawn, manual included | \`params.snapshot_id\` | Rejects \`snapshot_id\`; the unchanged schema hard-requires the file list + fingerprint and imposes no \`v3:\` pattern on \`snapshot_fingerprint\` | Bare hex \`snapshotId\` from \`get_change_review_bundle\` is evidence-only — do not use it as \`params.snapshot_id\`. +Manual spawns must not attempt to pass attestation tokens. \`params.snapshot_id\` is minted only for runtime-owned programmatic spawns; a prompt-authored spawn_agents/spawn_agent_inline call cannot obtain a valid token, and fingerprints shown in gate blocks or telemetry are truncated 16-char display prefixes that fail the \`v3:<64-hex>\` pattern. For manual/advisory reviewer-family spawns omit \`params.snapshot_id\`: put the scoped file list in \`params.files\` and the review question in the prompt. security-reviewer is the documented exception: its schema still requires \`params.changed_files\` + \`params.snapshot_fingerprint\` on manual spawns too, so a manual caller passes both keys and supplies as \`snapshot_fingerprint\` the stable fingerprint value it wants echoed exactly (the schema imposes no \`v3:\` pattern on that key); omitting the key fails the spawn. + ## Compaction recovery -If compaction drops GATE state, re-derive from the runtime's pinned GATE line / \`pendingGateFiles\` and do not manually re-spawn reviewer-family specialists — wait for the runtime-owned Final Gate result. +If compaction drops GATE state, re-derive from the runtime's pinned GATE line / \`pendingGateFiles\` and do not manually re-spawn reviewer-family specialists — wait for the runtime-owned Final Gate result. Never re-derive or re-mint a \`v3:<64-hex>\` token for a manual specialist spawn: the gate-assigned token is runtime-owned, and a manual spawn, when explicitly requested, must omit \`params.snapshot_id\` entirely. ## Sequential vs parallel diff --git a/common/src/templates/initial-agents-dir/types/tools.ts b/common/src/templates/initial-agents-dir/types/tools.ts index 32cd35f189..24a5052ac8 100644 --- a/common/src/templates/initial-agents-dir/types/tools.ts +++ b/common/src/templates/initial-agents-dir/types/tools.ts @@ -979,7 +979,7 @@ export interface SpawnAgentsParams { push?: boolean /** Remote used for fetch/push (git-committer) */ remote?: string - /** Assigned gate snapshot fingerprint (reviewer specialists) */ + /** Optional gate-assigned snapshot token (reviewer specialists). Runtime-owned spawns pass the gate-assigned v3:… token; manual spawns omit this key entirely. */ snapshot_id?: string /** Changed file paths to review (security-reviewer) */ changed_files?: string[] diff --git a/common/src/tools/__tests__/spawn-agents-schema.test.ts b/common/src/tools/__tests__/spawn-agents-schema.test.ts index a57d5a1328..a71a897333 100644 --- a/common/src/tools/__tests__/spawn-agents-schema.test.ts +++ b/common/src/tools/__tests__/spawn-agents-schema.test.ts @@ -56,6 +56,19 @@ describe('spawn_agents handoff schema', () => { expect(result.success).toBe(true) }) + it('rejects a versioned handoff that does not satisfy the canonical AgentHandoff schema', () => { + const result = spawnAgentsParams.inputSchema.safeParse({ + agents: [ + { + agent_type: 'editor', + handoff: { schemaVersion: 1, objective: 'x' }, + }, + ], + }) + + expect(result.success).toBe(false) + }) + it('repairs double-stringified lists and stringified agent entries', () => { const entry = { agent_type: 'file-picker', diff --git a/common/src/tools/params/tool/spawn-agents.ts b/common/src/tools/params/tool/spawn-agents.ts index 1d805e15e7..3594eb9809 100644 --- a/common/src/tools/params/tool/spawn-agents.ts +++ b/common/src/tools/params/tool/spawn-agents.ts @@ -165,7 +165,7 @@ const spawnAgentEntryFields = { .string() .optional() .describe( - 'Assigned gate snapshot fingerprint (reviewer specialists)', + 'Optional gate-assigned snapshot token (reviewer specialists). Runtime-owned spawns pass the gate-assigned v3:… token; manual spawns omit this key entirely.', ), changed_files: z .array(z.string()) @@ -291,7 +291,7 @@ Spawn agents in parallel (up to batch max). Pass \`agents\` as a real array of o - **\`agent_type\` must be a name from the live "You can spawn the following agents" catalog** (hyphenated ids; underscores accepted). It is an agent name (e.g. basher, file-picker, general-agent), **not a tool name** (read_files, str_replace, …). Call tools directly; do not wrap them in spawn_agents. - Prefer spawn_agents over single-agent tool aliases so multiple agents can run in parallel. Same nested \`prompt\` + \`params\` schema either way. -- Include required agent params (e.g. basher \`command\`, git-committer \`owned_paths\`, librarian \`repoUrl\`, dependency-manager \`manager\`+\`operation\`, security-reviewer \`changed_files\`+\`snapshot_fingerprint\`, reviewer specialists \`snapshot_id\`, repair-editor versioned \`handoff\`). Agent-specific fields go in \`params\`, not only the prompt. +- Include required agent params (e.g. basher \`command\`, git-committer \`owned_paths\`, librarian \`repoUrl\`, dependency-manager \`manager\`+\`operation\`, security-reviewer \`changed_files\`+\`snapshot_fingerprint\`, repair-editor versioned \`handoff\`). Reviewer-family \`snapshot_id\` is NOT a manual param: manual spawns omit it entirely (the gate-assigned \`v3:\` token is minted only for runtime-owned spawns); put scoped files in \`params.files\` and the question in the prompt. Agent-specific fields go in \`params\`, not only the prompt. - \`background: true\` returns a jobId immediately; poll with check_background_agent. Example: diff --git a/docs/agents-and-tools.md b/docs/agents-and-tools.md index 4ac72d7ac3..026a909e05 100644 --- a/docs/agents-and-tools.md +++ b/docs/agents-and-tools.md @@ -1284,31 +1284,21 @@ Input fields: - `params` (object, optional) — parameters object for the child agent. Direct agent schemas also accept a stringified JSON object for `params` and parse it before validation; malformed JSON, arrays, and objects that - do not match the child agent's schema still fail validation. -- `handoff` (object, optional) — structured handoff payload forwarded to - the child spawn entry. -- `background` (boolean, optional) — launches the child as a background job. - `spawn_agents.agents` also performs bounded repair for one- or double-stringified arrays and stringified object entries. Malformed or truncated JSON remains rejected; the runtime never fabricates an empty agent entry or silently drops required parameters. Stringified `params` and `handoff` objects are decoded at their envelope boundary only; legitimate nested string values such as shell commands remain strings. Basher requires -`params.command`, and snapshot-scoped reviewers require the exact current -gate-owned `v3:…` `params.snapshot_id` / `snapshot_fingerprint` from the parent -gate (not bare `get_change_review_bundle.snapshotId` hex, which is evidence-only). - -Example: - -```json -{ - "prompt": "Run pwd", - "params": { "command": "pwd" } -} -``` +`params.command`. Reviewer-family specialists accept `params.snapshot_id` only +on runtime-owned programmatic spawns, where the parent gate mints the exact +current opaque `v3:…` token (never bare `get_change_review_bundle.snapshotId` +hex, which is evidence-only); manual/advisory prompt-authored spawns must omit +`params.snapshot_id` entirely — put the scoped file list in `params.files` and +the review question in the prompt. Only `security-reviewer` accepts +`params.snapshot_fingerprint` (with `params.changed_files`), and manual spawns +omit that key too. -Equivalent tolerated form when a provider serializes nested params as a string: ```json diff --git a/packages/agent-runtime/src/__tests__/spawn-agents-permissions.test.ts b/packages/agent-runtime/src/__tests__/spawn-agents-permissions.test.ts index 97325978c3..efcea66af0 100644 --- a/packages/agent-runtime/src/__tests__/spawn-agents-permissions.test.ts +++ b/packages/agent-runtime/src/__tests__/spawn-agents-permissions.test.ts @@ -752,8 +752,13 @@ describe('editor implementation brief validation', () => { const compatibilityTemplate = { id: 'compatibility-reviewer', - inputSchema: { params: z.object({ snapshot_id: z.string().min(1) }) }, + inputSchema: { + params: z.object({ + snapshot_id: z.string().regex(/^v3:[a-f0-9]{64}$/), + }), + }, } as unknown as AgentTemplate + // No snapshot_id key supplied: a single omit-and-wait directive. expect(() => validateAgentInput( compatibilityTemplate, @@ -761,6 +766,33 @@ describe('editor implementation brief validation', () => { 'Review compatibility.', {}, ), + ).toThrow('manual spawns omit `params.snapshot_id` entirely') + try { + validateAgentInput( + compatibilityTemplate, + 'compatibility-reviewer', + 'Review compatibility.', + {}, + ) + } catch (error) { + const message = error instanceof Error ? error.message : String(error) + expect(message).toContain('params.files') + expect(message).toContain('wait for the gate') + expect(message).not.toContain('set params.snapshot_id') + expect(message).not.toContain('gate-assigned opaque v3:') + expect(message).not.toMatch( + /exact current snapshot fingerprint from get_change_review_bundle/i, + ) + } + + // Supplied-but-invalid snapshot_id: a single no-self-minting directive. + expect(() => + validateAgentInput( + compatibilityTemplate, + 'compatibility-reviewer', + 'Review compatibility.', + { snapshot_id: 'v3:' + 'a'.repeat(63) }, + ), ).toThrow( // Gate-assigned opaque v3 token — bare bundle hex is evidence-only. 'gate-assigned opaque v3:', @@ -770,19 +802,55 @@ describe('editor implementation brief validation', () => { compatibilityTemplate, 'compatibility-reviewer', 'Review compatibility.', - {}, + { snapshot_id: 'v3:' + 'a'.repeat(63) }, ) } catch (error) { const message = error instanceof Error ? error.message : String(error) - expect(message).toContain('"snapshot_id": "v3:<64-hex>"') - expect(message).toContain('specialistCreditFingerprint') + expect(message).toContain('the supplied params.snapshot_id is invalid') expect(message).toContain('evidence-only') + // No-self-minting: the hint never names a caller-side recompute path. + expect(message).not.toMatch(/hashGateSnapshotDetails/i) + expect(message).not.toMatch(/recompute|re-mint/i) + // A manual caller that supplied an invalid token gets the + // omit-for-manual contract, not a recipe for sourcing a replacement. + expect(message).toContain('omit params.snapshot_id entirely') + expect(message).toContain('wait for the runtime-owned gate') expect(message).not.toMatch( /exact current snapshot fingerprint from get_change_review_bundle/i, ) } }) + it('accepts a manual security-reviewer spawn with both schema-required keys (omit-for-manual exception)', () => { + // security-reviewer is the documented exception to the omit-for-manual + // contract: its schema hard-requires changed_files + snapshot_fingerprint + // on manual spawns too, and imposes no v3 pattern on the fingerprint, so + // the manual pre-edit security-review path stays usable with a + // caller-supplied stable value. + const securityReviewerTemplate = { + id: 'security-reviewer', + inputSchema: { + params: z + .object({ + changed_files: z.array(z.string()), + snapshot_fingerprint: z.string(), + }) + .strict(), + }, + } as unknown as AgentTemplate + expect(() => + validateAgentInput( + securityReviewerTemplate, + 'security-reviewer', + 'Review the auth change.', + { + changed_files: ['src/auth/login.ts'], + snapshot_fingerprint: 'pre-edit-review-fingerprint', + }, + ), + ).not.toThrow() + }) + it('accepts a concrete prose brief with actionable target files', () => { expect(() => validateAgentInput( diff --git a/packages/agent-runtime/src/__tests__/tool-validation-error.test.ts b/packages/agent-runtime/src/__tests__/tool-validation-error.test.ts index 3f1c62d112..481187362b 100644 --- a/packages/agent-runtime/src/__tests__/tool-validation-error.test.ts +++ b/packages/agent-runtime/src/__tests__/tool-validation-error.test.ts @@ -2166,6 +2166,8 @@ describe('tool validation error handling', () => { expect(typeof errorEvents[0].userMessage).toBe('string') expect((errorEvents[0].userMessage ?? '').length).toBeGreaterThan(0) expect(errorEvents[0].userMessage).not.toContain('Raw validation issues') + expect(errorEvents[0].userMessage).not.toContain('Exact params contract') + expect(errorEvents[0].autoRecovering).toBe(true) // Verify hadToolCallError is true so the agent loop continues expect(result.hadToolCallError).toBe(true) @@ -2269,10 +2271,14 @@ describe('tool validation error handling', () => { ...testAgentTemplate, id: 'reliability-reviewer', inputSchema: { - params: z.object({ snapshot_id: z.string() }), + params: z.object({ + snapshot_id: z.string().regex(/^v3:[a-f0-9]{64}$/), + }), }, } + // No snapshot_id key supplied: a single directive to omit the key and + // wait for the runtime-owned gate — never 'set params.snapshot_id'. let message = '' try { validateAgentInput( @@ -2286,16 +2292,67 @@ describe('tool validation error handling', () => { } expect(message).toContain('Missing required: snapshot_id') - expect(message).toContain('params.snapshot_id') - expect(message).toContain('"agent_type": "reliability-reviewer"') - expect(message).toContain('"snapshot_id": "v3:<64-hex>"') - expect(message).toContain('gate-assigned opaque v3:') - expect(message).toContain('specialistCreditFingerprint') - expect(message).toContain('evidence-only') - expect(message).toContain('will fail attestation') + expect(message).toContain('manual spawns omit `params.snapshot_id` entirely') + expect(message).toContain('params.files') + expect(message).toContain('wait for the gate') + expect(message).not.toContain('set params.snapshot_id') + expect(message).not.toContain('gate-assigned opaque v3:') // Must not tell the caller to source the attestation token from the bare // bundle snapshotId (evidence-only). expect(message).not.toMatch(/fingerprint from get_change_review_bundle/i) + + // Supplied-but-invalid: one directive — the token is minted only for + // runtime-owned spawns, so a manual caller omits the key and waits for + // the gate (no self-minting, no invented replacement). + let suppliedMessage = '' + try { + validateAgentInput( + reliabilityReviewer, + 'reliability-reviewer', + undefined, + { snapshot_id: 'v3:' + 'a'.repeat(63) }, + ) + } catch (error) { + suppliedMessage = error instanceof Error ? error.message : String(error) + } + + expect(suppliedMessage).toContain( + 'the supplied params.snapshot_id is invalid', + ) + expect(suppliedMessage).toContain('gate-assigned opaque v3:') + expect(suppliedMessage).toContain('evidence-only') + expect(suppliedMessage).toContain('will fail attestation') + expect(suppliedMessage).toContain( + 'never use a truncated 16-char display prefix', + ) + // No-self-minting: the hint never names a caller-side recompute path. + expect(suppliedMessage).not.toMatch(/hashGateSnapshotDetails/i) + expect(suppliedMessage).not.toMatch(/recompute|re-mint/i) + // A manual caller that supplied an invalid token is directed to the + // omit-for-manual contract and the runtime-owned gate. + expect(suppliedMessage).toContain('omit params.snapshot_id entirely') + expect(suppliedMessage).toContain('wait for the runtime-owned gate') + // Must not tell the caller to source the attestation token from the bare + // bundle snapshotId (evidence-only). + expect(suppliedMessage).not.toMatch( + /fingerprint from get_change_review_bundle/i, + ) + }) + + it('spawn_agents tool description stops listing reviewer snapshot_id as a required manual param', async () => { + const { spawnAgentsParams } = await import( + '@codebuff/common/tools/params/tool/spawn-agents' + ) + const description = spawnAgentsParams.description + // The live tool description must not tell callers to supply snapshot_id + // for reviewer specialists: manual spawns omit the key entirely (the + // gate-assigned v3 token is minted only for runtime-owned spawns). + expect(description).not.toContain('reviewer specialists `snapshot_id`') + expect(description).toContain('manual spawns omit it entirely') + expect(description).toContain('`params.files`') + // A manual caller-facing surface must never instruct callers to hunt for a + // gate-assigned v3 token they cannot obtain (omit-for-manual contract). + expect(description).not.toMatch(/reviewer specialists? require/i) }) it('validateAgentInput accepts attestable v3 snapshot_id and rejects bare hex', async () => { @@ -2422,10 +2479,64 @@ describe('tool validation error handling', () => { 'Exact params contract (from the child agent schema)', ) expect(message).toContain( - 'replace params.snapshot_id with params.snapshot_fingerprint', + 'Replace params.snapshot_id with params.snapshot_fingerprint', ) expect(message).toContain('Retain params.changed_files') expect(message).toContain('Preserve params field names exactly.') + // The hint must state the documented exception (schema-required + // fingerprint on manual spawns too) rather than the reviewer-family + // omit-both directive that deterministically fails this schema. + expect(message).toContain( + 'documented exception to the omit-for-manual contract', + ) + expect(message).toContain('imposes no v3: pattern on that key') + expect(message).not.toContain('omit `params.snapshot_id` entirely') + }) + + it('accepts a manual security-reviewer spawn supplying both schema-required keys', async () => { + const { validateAgentInput } = + await import('../tools/handlers/tool/spawn-agent-utils') + const securityReviewer = { + ...testAgentTemplate, + id: 'security-reviewer', + inputSchema: { + params: z + .object({ + changed_files: z.array(z.string()), + snapshot_fingerprint: z.string(), + }) + .strict(), + }, + } + + // The manual pre-edit path advertised in agents/guides/security-review.md: + // the caller passes both schema-required keys and its own stable + // fingerprint value (no v3 pattern is imposed on snapshot_fingerprint, so + // no unobtainable gate-owned token is needed). + expect(() => + validateAgentInput(securityReviewer, 'security-reviewer', undefined, { + changed_files: ['src/auth/login.ts'], + snapshot_fingerprint: 'pre-edit-review-fingerprint', + }), + ).not.toThrow() + + // Omitting the schema-required fingerprint still fails, and the recovery + // hint must direct the caller to add the key (the exception), never to + // the reviewer-family omit-both contract. + let missingMessage = '' + try { + validateAgentInput(securityReviewer, 'security-reviewer', undefined, { + changed_files: ['src/auth/login.ts'], + }) + } catch (error) { + missingMessage = error instanceof Error ? error.message : String(error) + } + expect(missingMessage).toContain('Missing required: snapshot_fingerprint') + expect(missingMessage).toContain( + 'documented exception to the omit-for-manual contract', + ) + expect(missingMessage).toContain('add params.snapshot_fingerprint') + expect(missingMessage).not.toContain('omit `params.snapshot_id` entirely') }) it('publishes a structured failure result when Basher is missing command', async () => { @@ -2520,6 +2631,91 @@ describe('tool validation error handling', () => { ) }) + it('emits a calm userMessage on partial spawn failures', async () => { + // One valid + one invalid agent entry exercises the PARTIAL spawn_agents + // failure path in executeToolCall. The detailed `message` for the agent is + // unchanged; the CLI-facing `userMessage` is the calm one-liner and the + // event is marked autoRecovering so the CLI suppresses the raw wall. + const parent: AgentTemplate = { + ...testAgentTemplate, + toolNames: ['spawn_agents', 'end_turn'], + spawnableAgents: ['basher'], + } + const basher: AgentTemplate = { + ...testAgentTemplate, + id: 'basher', + inputSchema: { params: z.object({ command: z.string().min(1) }) }, + toolNames: ['run_terminal_command'], + spawnableAgents: [], + } + const partialSpawn: StreamChunk = { + type: 'tool-call', + toolName: 'spawn_agents', + toolCallId: 'basher-partial-invalid-tool-call-id', + input: { + agents: [ + { + agent_type: 'basher', + prompt: 'Run the tests', + params: { command: 'bun test' }, + }, + { agent_type: 'basher', params: {} }, + ], + }, + } + async function* mockStream() { + yield partialSpawn + return promptSuccess('mock-message-id') + } + const responseChunks: (string | PrintModeEvent)[] = [] + const sessionState = getInitialSessionState(mockFileContext) + + await processStream({ + ...agentRuntimeImpl, + agentContext: {}, + agentState: sessionState.mainAgentState, + agentStepId: 'test-step-id', + agentTemplate: parent, + ancestorRunIds: [], + clientSessionId: 'test-session', + fileContext: mockFileContext, + fingerprintId: 'test-fingerprint', + fullResponse: '', + localAgentTemplates: { 'test-agent': parent, basher }, + messages: [], + prompt: 'test prompt', + repoId: undefined, + repoUrl: undefined, + runId: 'test-run-id', + signal: new AbortController().signal, + stream: mockStream(), + system: 'test system', + tools: {}, + userId: 'test-user', + userInputId: 'test-input-id', + onCostCalculated: async () => {}, + onResponseChunk: (chunk) => responseChunks.push(chunk), + }) + + const errorEvents = responseChunks.filter( + (chunk): chunk is Extract => + typeof chunk !== 'string' && chunk.type === 'error', + ) + const spawnError = errorEvents.find((event) => + event.message.startsWith('Some agents could not be spawned'), + ) + expect(spawnError).toBeDefined() + // The detailed agent-facing message is unchanged (FIX A keeps `message`). + expect(spawnError!.message).toContain('Missing required: command') + // The calm CLI summary rides along and the event is auto-recovering. + expect(spawnError!.userMessage).toContain( + 'could not be spawned due to invalid parameters', + ) + expect(spawnError!.userMessage).not.toContain('Raw validation issues') + expect(spawnError!.userMessage).not.toContain('Exact params contract') + expect(spawnError!.autoRecovering).toBe(true) + }) + it('repairs a single-agent mis-braced spawn payload and publishes it to the handler', async () => { const parent: AgentTemplate = { ...testAgentTemplate, diff --git a/packages/agent-runtime/src/tools/handlers/tool/spawn-agent-utils.ts b/packages/agent-runtime/src/tools/handlers/tool/spawn-agent-utils.ts index 9868bc8e5b..d757aaf240 100644 --- a/packages/agent-runtime/src/tools/handlers/tool/spawn-agent-utils.ts +++ b/packages/agent-runtime/src/tools/handlers/tool/spawn-agent-utils.ts @@ -1809,24 +1809,24 @@ export function validateAgentInput( params && typeof params === 'object' && !Array.isArray(params) ? (params as Record) : undefined - const rawSnapshotId = - typeof paramsRecord?.snapshot_id === 'string' - ? paramsRecord.snapshot_id.trim() - : '' - const isBareBundleHex = /^[a-f0-9]{64}$/i.test(rawSnapshotId) - const bareHexNote = isBareBundleHex - ? ` Received bare 64-hex bundle snapshotId "${rawSnapshotId.slice(0, 12)}…" (evidence-only, from get_change_review_bundle) — not gate attestation. Recompute the gate-owned v3 token via hashGateSnapshotDetails(pendingGateFiles) and pass that as params.snapshot_id.` - : '' + // Branch the recovery hint on whether the caller actually supplied a + // snapshot_id key, so exactly one directive is emitted per situation. + const snapshotIdSupplied = Object.hasOwn( + paramsRecord ?? {}, + 'snapshot_id', + ) const recoveryHint = normalizedAgentType === 'basher' && issuePaths.has('command') ? '\n\nRecovery: spawn Basher with { "agent_type": "basher", "params": { "command": "" } }. A command mentioned only in prompt prose is never executed.' : reviewerFamilyRequiredSnapshotIds.has(normalizedAgentType) && issuePaths.has('snapshot_id') - ? `\n\nRecovery: set params.snapshot_id to the gate-assigned opaque v3:… token from the parent gate (hashGateSnapshotDetails(pendingGateFiles) / specialistCreditFingerprint), for example { "agent_type": "${normalizedAgentType}", "params": { "snapshot_id": "v3:<64-hex>" } }. Bare hex from get_change_review_bundle.snapshotId is evidence-only and will fail attestation; never invent or reuse a stale fingerprint.${bareHexNote}` + ? snapshotIdSupplied + ? `\n\nRecovery: the supplied params.snapshot_id is invalid — the gate-assigned opaque v3:… token is minted only for runtime-owned programmatic spawns, and no caller-side call can obtain or derive one. Bare hex from get_change_review_bundle.snapshotId is evidence-only and will fail attestation; never invent or reuse a stale fingerprint, and never use a truncated 16-char display prefix from gate blocks or telemetry. A manual spawn cannot supply a valid token: omit params.snapshot_id entirely — put the scoped file list in params.files and the review question in the prompt — and end the turn to wait for the runtime-owned gate. Only security-reviewer accepts params.snapshot_fingerprint; reviewer-family agents never do.` + : `\n\nRecovery: manual spawns omit \`params.snapshot_id\` entirely — put the scoped file list in \`params.files\` and the review question in the prompt. Post-edit reviewer-family spawns are runtime-owned: end the turn and wait for the gate instead of spawning manually. Only security-reviewer accepts \`params.snapshot_fingerprint\`; reviewer-family agents never do.` : normalizedAgentType === 'security-reviewer' && (issuePaths.has('snapshot_fingerprint') || Object.hasOwn(paramsRecord ?? {}, 'snapshot_id')) - ? '\n\nRecovery: replace params.snapshot_id with params.snapshot_fingerprint, or add params.snapshot_fingerprint when it is missing. Retain params.changed_files and preserve both canonical field names exactly.' + ? '\n\nRecovery: security-reviewer is the documented exception to the omit-for-manual contract — its schema still requires params.changed_files and params.snapshot_fingerprint on manual spawns too. Replace params.snapshot_id with params.snapshot_fingerprint, or add params.snapshot_fingerprint with the stable fingerprint value to echo exactly; the schema imposes no v3: pattern on that key, so no gate-owned token is needed. Retain params.changed_files and preserve both canonical field names exactly.' : normalizedAgentType === 'dependency-manager' && (issuePaths.has('manager') || issuePaths.has('operation')) ? '\n\nRecovery: place both canonical keys in params, for example { "agent_type": "dependency-manager", "params": { "manager": "npm", "operation": "add" } }. manager must come from repository manifest/environment evidence. operation must be one of add, remove, sync, restore, or update. Do not infer dependency mutation authorization from a validation failure.' diff --git a/packages/agent-runtime/src/tools/tool-executor.ts b/packages/agent-runtime/src/tools/tool-executor.ts index f92798ede8..294cb0311f 100644 --- a/packages/agent-runtime/src/tools/tool-executor.ts +++ b/packages/agent-runtime/src/tools/tool-executor.ts @@ -1597,6 +1597,13 @@ const GIT_COMMITTER_WITHHELD_USER_MESSAGE = // paths, so the two stay in sync. const malformedToolCallUserMessage = (toolName: string): string => `The model sent a malformed \`${toolName}\` tool call and is correcting it automatically. No action is needed.` +// Concise, calm summary for spawn pre-validation failures (the partial +// spawn_agents failure path and the spawn_agent_inline pre-publication check). +// Like the followups ordering rejections, this is normal agent self-correction +// resolved from the detailed contract in `message`, so the CLI suppresses the +// visible banner (see `autoRecovering` in `common/src/types/print-mode.ts`). +const SPAWN_INVALID_PARAMS_USER_MESSAGE = + 'One or more requested sub-agents could not be spawned due to invalid parameters. The agent received the detailed contract and is retrying with corrected parameters. No action is needed.' function isTerminalFollowupCompanion(name: string): boolean { return ( @@ -2800,7 +2807,12 @@ export async function executeToolCall( ) } else { const errorMsg = `Some agents could not be spawned: ${errors.join('; ')}. Proceeding with valid agents only.` - onResponseChunk({ type: 'error', message: errorMsg }) + onResponseChunk({ + type: 'error', + message: errorMsg, + userMessage: SPAWN_INVALID_PARAMS_USER_MESSAGE, + autoRecovering: true, + }) effectiveInput = { ...effectiveInput, agents: validAgents } } } @@ -2828,7 +2840,12 @@ export async function executeToolCall( ) } catch (error) { const message = error instanceof Error ? error.message : String(error) - onResponseChunk({ type: 'error', message }) + onResponseChunk({ + type: 'error', + message, + userMessage: SPAWN_INVALID_PARAMS_USER_MESSAGE, + autoRecovering: true, + }) logger.debug( { toolName, error: message }, 'spawn_agent_inline input failed pre-publication validation', From 9d4a8f30b5373562ee4be0facdf0de820bf7a8af Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Tue, 15 Sep 2026 08:52:56 +0300 Subject: [PATCH 20/22] docs(agents): label manual reviewer echoes unattested + refresh knowledge formatStructuredReviewReceipt now renders an empty snapshotFingerprint echo as (manual/unattested) so a manual-omit-contract review is never mistaken for a gate-attested fingerprint, with a new pinning test asserting gate attestation fails closed on the empty echo (missing-fingerprint issue, no drift credit, base2 inline-mirror parity). Tail knowledge-refresh entries in cli/knowledge.md and common/knowledge.md document the optional snapshot_id/security-reviewer exception, the calm userMessage/autoRecovering spawn error UX, and the unattested echo, clearing the guard:memory-drift staleness check. --- agents/__tests__/gate-reviewer.test.ts | 34 ++++++++++++++++++++++++++ agents/context-pruner.ts | 11 +++++++-- cli/knowledge.md | 2 ++ common/knowledge.md | 2 ++ 4 files changed, 47 insertions(+), 2 deletions(-) diff --git a/agents/__tests__/gate-reviewer.test.ts b/agents/__tests__/gate-reviewer.test.ts index 927bf63029..2c6535c3bc 100644 --- a/agents/__tests__/gate-reviewer.test.ts +++ b/agents/__tests__/gate-reviewer.test.ts @@ -2478,6 +2478,40 @@ describe('gate-reviewer helpers', () => { ]) }) + // A manual spawn echoes the documented omit-for-manual contract as an EMPTY + // `snapshotFingerprint` string rather than `undefined`. That echo is not an + // attestation: resolveReviewerAttestation skips zero-length fingerprints, so + // the receipt must fail closed exactly like the undefined-fingerprint case, + // and the unattestable echo must never be credited as drift evidence. + test('fails closed when a coverage-complete manual review echoes an empty snapshotFingerprint', () => { + const toolResult = { + type: 'json', + value: [ + { + schemaVersion: 1, + verdict: 'LOOKS_GOOD', + snapshotFingerprint: '', + reviewedFiles: ['src/a.ts'], + }, + ], + } + const expected = 'v3:' + 'a'.repeat(64) + expect( + collectReviewerAttestationIssues(toolResult, expected, ['src/a.ts']), + ).toEqual([ + 'BLOCKING: reviewer did not report an attestable snapshot fingerprint', + ]) + // An unattestable echo is never recorded as drift. + expect(collectReviewerFingerprintDrift(toolResult, expected)).toBe('') + // base2's inline attestation copy is the gate's runtime authority. + const inlineHelpers = loadInlineGateReviewerHelpers() + expect( + inlineHelpers.collectReviewerAttestationIssues(toolResult, expected, [ + 'src/a.ts', + ]), + ).toEqual(collectReviewerAttestationIssues(toolResult, expected, ['src/a.ts'])) + }) + test('blocks a coverage gap with an attestable-but-wrong fingerprint on both issues', () => { expect( collectReviewerAttestationIssues( diff --git a/agents/context-pruner.ts b/agents/context-pruner.ts index 37c91521eb..28df3436e3 100644 --- a/agents/context-pruner.ts +++ b/agents/context-pruner.ts @@ -1954,9 +1954,16 @@ const definition: AgentDefinition = { const verdict = isStaleSnapshotReviewerOutput(value) ? 'STALE_SNAPSHOT' : String(record.verdict).trim().toUpperCase() + const rawFingerprint = record.snapshotFingerprint const fingerprint = - typeof record.snapshotFingerprint === 'string' - ? record.snapshotFingerprint + typeof rawFingerprint === 'string' + ? rawFingerprint.trim().length === 0 + ? // Manual spawns echo the documented omit-for-manual contract as an + // empty string; that echo carries no attestation, so label it + // instead of rendering a bare `snapshot=` a reader could mistake + // for a gate-attested fingerprint. + '(manual/unattested)' + : rawFingerprint : '(legacy/unattested)' const coverage = typeof record.coverage === 'string' ? record.coverage : 'n/a' diff --git a/cli/knowledge.md b/cli/knowledge.md index 3bb7631628..2bc7555307 100644 --- a/cli/knowledge.md +++ b/cli/knowledge.md @@ -932,3 +932,5 @@ Streaming markdown renders as plain text until the message or agent finishes. Th - _Knowledge refresh 2026-08-31 (UI polish): `cli/src/components/status-bar.tsx` is now three regions — status label left, chip cluster left-aligned in the growing middle (`flexGrow: 1` + `flexBasis: 0`), and every width-varying control (scroll-to-bottom, then the `■ Esc` stop hint) in a `flexShrink: 0` right region with no `minWidth: 0`, so a hover cannot reflow the label or the chips. `cli/src/components/scroll-to-bottom-button.tsx` exports `SCROLL_HINT_LABEL`/`SCROLL_GLYPH` plus `string-width`-derived `SCROLL_BUTTON_WIDTH` (10) and `SCROLL_BUTTON_COMPACT_WIDTH` (3) and renders at a fixed width in both hover states; its `isScrollButtonCompact(width)` predicate is the single source `StatusBar` also passes as `scrollButtonCompact`, so `statusBarChipBudget`'s duplicated `SCROLL_BUTTON_RESERVATION`/`SCROLL_BUTTON_COMPACT_RESERVATION` in `cli/src/utils/status-bar-chips.ts` always reserve the columns actually rendered (test-enforced agreement, since the util must not import a component module). `cli/src/components/renderers/completion-summary-box.tsx` renders a titled `Run summary` `HarnessBox` (`gap={0}`, `paddingBottom={0}`) of aligned `Label value` rows built from `ROW_LABELS` + derived `LABEL_COLUMN_WIDTH`, with no status emoji — meaning lives in the value words, not color. Reconciler-level coverage for the status bar lives in `cli/src/components/__tests__/status-bar.test.tsx`, which reuses the dev-only `renderTest`/`renderFrame` convention from `text-nesting.test.tsx` (`@opentui/react/test-utils` cannot be imported under `NODE_ENV=production`)._ - _Knowledge refresh 2026-09-05 (compaction progress + self-dismissing cards): `cli/src/utils/sdk-event-handlers.ts` consumes the additive `context_compaction_progress` event in `handleContextCompactionProgress`, clamping each reported percent to a whole 0..100 and writing only the MAXIMUM of what the card or notice already holds, so two producers for one pass (the agent loop's milestones and the inline spawn path's activity ticks) plus replayed or out-of-order events can never rewind the bar. Card updates stay root-scoped and paired by `runId`, while the status-bar notice tracks root and nested passes alike; a progress event never creates a notice or revives a settled one. `compactionResultIsDegraded` is the single site deciding `CompactionContentBlock.transient`: a healthy settled pass is stamped `progressPercent: 100` plus `transient: true`, while a mechanical or request-time trim, a pass that missed its budget, an escalated pass, and a low-yield streak all stay permanent warning cards, as do declined and interrupted passes. `cli/src/components/renderers/compaction-box.tsx` renders `cli/src/components/progress-bar.tsx` for pending and transient passes and hides a transient card after a short hold, but hiding is purely visual: `dropTransientCompactionBlocks` in `cli/src/utils/message-block-helpers.ts` is what actually removes it from state, composed into both the turn-end path in `handleFinish` and the abort path in `cli/src/hooks/helpers/send-message.ts`, so a self-dismissing card can never persist to the transcript. `cli/src/utils/status-bar-chips.ts` reports `⇲ compacting NN%` at md/lg when a live percent is finite and above zero and otherwise keeps the previous ellipsis label (xs/sm labels unchanged). `cli/src/types/chat.ts` carries the new optional `progressPercent` and `transient` block fields, the notice `progressPercent`, and the `boundedFileReads` category. `cli/src/components/terminal-command-display.tsx` now shows a timeout label only for a finite positive bound, because `timeout_seconds` defaults to no timeout and the removed 30s default would otherwise be implied. Coverage: `cli/src/utils/__tests__/sdk-event-handlers.test.ts`, `cli/src/utils/__tests__/status-bar-chips.test.ts`, and `cli/src/components/__tests__/sweep-boxes.test.tsx`._ + +- _Knowledge refresh 2026-09-12 (spawn attestation followups): reviewer-family specialist `spawn_agents` params now treat `snapshot_id` as OPTIONAL — runtime-owned programmatic spawns pass the gate-minted opaque `v3:<64-hex>` token, while manual/advisory prompt-authored spawns omit the key entirely (scoped files go in `params.files`, the question in the prompt); `security-reviewer` is the exception and its schema still requires `changed_files` + `snapshot_fingerprint` on manual spawns too (no v3 pattern on that key). Covered by `agents/specialists/create-specialist.ts`, `common/src/tools/params/tool/spawn-agents.ts`, and `packages/agent-runtime/src/tools/handlers/tool/spawn-agent-utils.ts` (branched recovery hints), with the routing matrix in `agents/guides/specialist-routing.md`. Spawn partial-failure and `spawn_agent_inline` pre-validation errors now carry a concise `userMessage` + `autoRecovering: true` (see `common/src/types/print-mode.ts`, consumed through `cli/src/utils/sdk-event-handlers.ts`'s log-only auto-recovering path), so the CLI shows a calm one-liner instead of the raw multi-KB validation wall while the full contract still reaches the agent's message history (`packages/agent-runtime/src/tools/tool-executor.ts`). The context-pruner's structured review receipt renders an empty `snapshotFingerprint` echo as `(manual/unattested)` (`agents/context-pruner.ts` `formatStructuredReviewReceipt`), and gate attestation fails closed on an empty echo with the missing-fingerprint issue and no drift credit (`agents/base2/gate-reviewer.ts`; pinned in `agents/__tests__/gate-reviewer.test.ts`)._ diff --git a/common/knowledge.md b/common/knowledge.md index 2945cb09a4..ea71d7ba63 100644 --- a/common/knowledge.md +++ b/common/knowledge.md @@ -77,6 +77,8 @@ This package contains code shared across the Openbuff monorepo, especially the l - _Knowledge refresh 2026-09-05 (temp-root scope widening + quoted-slash terminal fix): `common/src/util/project-path-containment.ts` widened the OS-temp exception from an openbuff-owned NAME gate to plain containment — any path strictly inside a temp root (`os.tmpdir()` plus `/tmp` on POSIX) now resolves with `scope: 'owned-temp'` and an absolute `relativePath` whatever its segment names, so `/notes.txt` behaves exactly like `/openbuff-job-1.log`. `OWNED_TEMP_SEGMENT_PATTERNS` is DOCUMENTATION ONLY now (it enumerates the namespaces openbuff itself creates); `isInsideOwnedTempNamespace` was renamed `isInsideTempRoot`; the strictly-inside rule, raw-`..` refusal, and the single-realpath TOCTOU discipline are unchanged. A NEW fail-closed `isMandatorySensitiveReadPath` refusal (lexical AND dereferenced path, mirroring `resolveExternalReadRealPath`) keeps `/.env`, `/credentials.json`, private keys, and path-aware carriers like `.aws/config` unreachable for READS and WRITES — with the name gate gone there is no longer an incidental pattern blocking them, so the resolver is the only guard. `isOwnedTempPathForFileSystem` is newly exported and `sdk/src/tools/path-utils.ts` deleted its private fs-aware duplicate in favor of it. `packages/agent-runtime/src/tools/tool-executor.ts` gained `OWNED_TEMP_WRITE_EXEMPT_TOOLS` (the file-changing tools) so writes into temp are no longer hard-blocked by the backstop, while the external-read allowlist stays strictly read-only (the write side never consults `isExternalReadPath`) and `code_search`/`glob`/`find_files_matching_content` temp cwds stay hard-blocked because their handlers do not contain. `ownedTempMutationRefusal` in `sdk/src/tools/filesystem-authority.ts` is unchanged in mechanism and is the ONLY defense against writing an executable-extension basename anywhere under temp, including `tmux-helper-.sh`, which containment no longer excludes; its `OWNED_TEMP_REFUSED_EXTENSIONS` set now also refuses interpreter-executed extensions (.js/.mjs/.cjs/.jsx/.ts/.tsx/.mts/.cts/.py/.pyw/.pl/.rb/.lua/.php/.r/.jl/.tcl) because a `write_file /tmp/x.js` followed by `node /tmp/x.js` would otherwise execute staged code under terminal profiles that permit `node ` (create/overwrite/move refused; delete stays allowed for cleanup); `run_terminal_command` still refuses an owned-temp cwd. A follow-up repair extends the same refusals to Win32 trailing dot/space aliases: `ownedTempMutationRefusal` now evaluates the job-artifact pattern, the tmux-capture segment pattern, and the executable-extension set against BOTH the resolved path and its Win32-normalized form (every segment's trailing dots/spaces stripped, via a module-local `win32NormalizeSegments` in `filesystem-authority.ts` mirroring `refusesWin32AliasedSensitivePath`), so a lexical `payload.sh ` — whose extname `.sh ` misses the set while the OS creates the real `payload.sh` — can no longer stage an executable, clobber live `openbuff-*.log`/`.json` job artifacts, or forge `tmux-captures-*` capture evidence on Windows; refusal codes, the owned-temp scope gate, the read exemption, and the delete cleanup carve-out are unchanged. Two more hardenings from the security review: the temp resolvers also refuse Win32-aliased sensitive paths (`/.env ` with a trailing space/dot normalizes to `.env` on win32, and the same per-segment normalization covers aliased INTERMEDIATE directories so `/.aws /config` cannot open the real `.aws/config` — `refusesWin32AliasedSensitivePath` in the same module normalizes EVERY segment, applied to lexical AND dereferenced paths in both sync and async resolvers, while non-sensitive trailing-dot names stay admitted), and `findOutsideAbsolutePath` re-refuses a QUOTED root-only operand (`rm -rf '/'`, `cp x '/'`) when the command invokes a filesystem-mutating executable (rm/mv/cp/chmod/chown/chgrp/dd/shred/truncate/ln/install, optionally behind sudo/doas) — the quoted-root skip itself exists only for `sed`/`awk` expression delimiters, and `ls /` unquoted, `cat '/etc/passwd'`, and `bash -c 'cat /etc/passwd'` stay refused. Separately, `sdk/src/tools/terminal-command-policy.ts` fixed a false positive where a bare `/` inside a quoted word — typically the delimiter tail of `sed 's/^/X /'` — was treated as an absolute path operand: `findOutsideAbsolutePath` now skips only a root-only token that sits inside a quoted region (one linear quote scan per call), while an unquoted `ls /`, a quoted `cat '/etc/passwd'`, and an embedded `bash -c 'cat /etc/passwd'` stay refused. Containment negative fixtures that used `os.tmpdir()` mkdtemp dirs as their "outside the project" target were re-anchored to a gitignored `.containment-test-scratch/` under each package directory so the refusals remain attributable._ +- _Knowledge refresh 2026-09-12 (spawn attestation followups): `common/src/tools/params/tool/spawn-agents.ts` made reviewer-family specialist `snapshot_id` OPTIONAL — runtime-owned programmatic spawns pass the gate-minted opaque `v3:<64-hex>` token while manual/advisory prompt-authored spawns omit the key entirely (scoped files go in `params.files`, the question in the prompt); `security-reviewer` remains the exception, still requiring `changed_files` + `snapshot_fingerprint` on manual spawns too (no v3 pattern on that key). Consumers: `agents/specialists/create-specialist.ts`, `packages/agent-runtime/src/tools/handlers/tool/spawn-agent-utils.ts` (branched recovery hints), and `agents/guides/specialist-routing.md`. The existing optional `printModeErrorSchema` `userMessage`/`autoRecovering` fields (see `common/src/types/print-mode.ts`) are now also set on spawn partial-failure and `spawn_agent_inline` pre-validation errors by `packages/agent-runtime/src/tools/tool-executor.ts`, so the CLI shows a calm one-liner instead of the raw multi-KB validation wall while the full `message` still reaches the agent's message history. Relatedly, `agents/context-pruner.ts` `formatStructuredReviewReceipt` renders an empty `snapshotFingerprint` echo as `(manual/unattested)`, and `agents/base2/gate-reviewer.ts` fails gate attestation closed on an empty echo with the missing-fingerprint issue and no drift credit (pinned in `agents/__tests__/gate-reviewer.test.ts`)._ + ## Scope Notes Openbuff is CLI/SDK-focused and local/BYOK. Do not add new dependencies from `common/` to hosted web, billing, credit, subscription, or BigQuery product surfaces. Provider-owned billing, quota, token usage, and OAuth flows may still be documented when they refer to the user's configured provider rather than an Openbuff-hosted product. From 895e2b7fb3335d71e7c7afa623fb7e399bc13cea Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Tue, 15 Sep 2026 08:56:12 +0300 Subject: [PATCH 21/22] test(specialists): pin optional snapshot_id omit-for-manual contract The stale required-key and spawnerPrompt assertions expected snapshot_id as a required param; update them to the new optional schema where manual spawns omit params.snapshot_id entirely and the v3 pattern applies only when supplied. --- agents/__tests__/specialists.test.ts | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/agents/__tests__/specialists.test.ts b/agents/__tests__/specialists.test.ts index 3c111bad3d..84b182c06b 100644 --- a/agents/__tests__/specialists.test.ts +++ b/agents/__tests__/specialists.test.ts @@ -96,7 +96,9 @@ describe('specialist agents', () => { const reviewerInput = dependencyReviewer.inputSchema as any const reviewerOutput = dependencyReviewer.outputSchema as any - expect(reviewerInput.params.required).toContain('snapshot_id') + // snapshot_id is optional under the omit-for-manual contract: manual + // spawns omit the key entirely; the v3 pattern applies when supplied. + expect(reviewerInput.params.required).not.toContain('snapshot_id') expect(reviewerOutput.required).toContain('verdict') expect(reviewerOutput.required).toContain('coverage') expect(reviewerOutput.properties.family.enum).toEqual(['reviewer']) @@ -124,10 +126,10 @@ describe('specialist agents', () => { 'never JSON.stringify', ) expect(compatibilityReviewer.spawnerPrompt).toContain( - 'Requires params.snapshot_id', + 'When supplied, params.snapshot_id must be the assigned gate snapshot fingerprint', ) expect(compatibilityReviewer.spawnerPrompt).toContain( - 'assigned gate snapshot fingerprint for this spawn', + 'manual spawns omit params.snapshot_id entirely', ) expect( (compatibilityReviewer.inputSchema as any).params.properties.snapshot_id From fd7e9788505c9bb33126a68b210a747b77a4029e Mon Sep 17 00:00:00 2001 From: AnzoBenjamin Date: Tue, 15 Sep 2026 09:27:35 +0300 Subject: [PATCH 22/22] test(quality-prompts): fix stale snapshot_id contract comment The comment above the specialistRoutingSection snapshot param contract test still described the old require-snapshot_id contract; it now states the omit-for-manual two-mode contract (runtime-owned spawns pass the gate-owned v3 token, manual spawns omit the key entirely; security-reviewer requires changed_files + snapshot_fingerprint). Comment-only change, no assertions modified. --- agents/__tests__/quality-prompt-snapshot.test.ts | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/agents/__tests__/quality-prompt-snapshot.test.ts b/agents/__tests__/quality-prompt-snapshot.test.ts index 8a37dd4a4a..a4f2fcf387 100644 --- a/agents/__tests__/quality-prompt-snapshot.test.ts +++ b/agents/__tests__/quality-prompt-snapshot.test.ts @@ -224,9 +224,11 @@ describe('shared craftsmanship prompt sections', () => { }) test('specialistRoutingSection names the exact snapshot param contract for reviewer-family specialists', () => { - // Reviewer-family specialists require params.snapshot_id as the gate-owned - // v3 token (not bare get_change_review_bundle.snapshotId), while - // security-reviewer requires changed_files + snapshot_fingerprint. + // Reviewer-family spawns treat params.snapshot_id as optional under the + // omit-for-manual contract: runtime-owned spawns pass the gate-owned v3 + // token (never bare get_change_review_bundle.snapshotId) and manual spawns + // omit the key entirely, while security-reviewer requires changed_files + + // snapshot_fingerprint. expect(specialistRoutingSection).toContain('snapshot_id') expect(specialistRoutingSection).toContain('gate-assigned opaque') expect(specialistRoutingSection).toContain('v3:')