diff --git a/cli/knowledge.md b/cli/knowledge.md index ca1c6ef006..1e9bfaf5ce 100644 --- a/cli/knowledge.md +++ b/cli/knowledge.md @@ -16,14 +16,14 @@ - The SQLite repository projects the latest `coverage.recorded` event per `(taskId, dimension)` into the `currentCoverage` retrieval category, filtered by exact workspace revision/snapshot freshness and optional `taskId` narrowing. - The `query_index` tool handler (`packages/agent-runtime/src/tools/handlers/tool/query-index.ts`) records its results into `agentState.discoveryCoverage` via `recordDiscoveryResult` with a bounded (4000-char) question string; recording is wrapped in try/catch so a coverage failure can never break the tool call. - When testing tool handlers that import a function to spy on, prefer `spyOn(namespaceImport, 'fn')` over `mock.module()` (repo convention in `docs/testing.md`/`CONTRIBUTING.md`); a relative `mock.module` specifier resolves from the test file's directory and silently misses the module the subject under test imports when the two live at different depths. -- Memory V2 authority is selected by `OPENBUFF_MEMORY_AUTHORITY` via `getMemoryAuthoritySelection` (`cli/src/utils/env.ts`); the default is `sqlite-v2-opt-in` (fail-closed — no V1 fallback), with `json-v1` as an explicit opt-out escape hatch and `shadow-v2` as an explicit shadow mode. The provider (`cli/src/services/memory-v2/provider.ts`) keys its shadow-vs-fail-closed behavior off `authority.effective`, so flipping the default flips which degradation path default runs take. +- Memory V2 authority is selected by `OPENBUFF_MEMORY_AUTHORITY` via `getMemoryAuthoritySelection` (`cli/src/utils/env.ts`); the default is `sqlite-v2-opt-in` (fail-closed — no V1 fallback), with `json-v1` as an explicit opt-out escape hatch and `shadow-v2` as an explicit shadow mode. Release N keeps `json-v1` and `shadow-v2` supported but deprecates both compatibility modes; `sqlite-v2-opt-in` is the default and replacement. Authority selection, degradation, and fallback behavior are unchanged in this release, no removal date/version is implied, and migration is not claimed complete. Follow the normative [Memory V1 removal readiness plan](../docs/memory-v1-removal-readiness.md) before any later removal decision. ## Slash Commands and Plan Mode - Durable planning is entered through `mode:plan`; the standalone `/plan` command is intentionally absent from `COMMAND_REGISTRY` and `SLASH_COMMANDS` so there is one plan-entry path. - Keep the durable-plan quartet registered: `/resume-plan` (`rp`), `/update-plan` (`up`), `/plan-status` (`ps`), and `/lessons` (`lesson`). These commands operate on `.agents/sessions//` artifacts and fall back to the plan-session picker when no target is provided. - `/plans` (`plan-ls`) lists the artifact-bearing sessions returned by `listPlanSessions()` and never prompts the agent, while `/plan-use` (`plan-active`, `use-plan`) writes the `ACTIVE_SESSION` pointer file under `.agents/` and only accepts a bare slug or `.agents/sessions/` — the resolved directory must be exactly one segment under `.agents/sessions/` because the pointer stores bare slugs, so nested paths, non-session paths, missing directories, and artifact-less directories are all rejected before any write. Both share `formatPlanSessionListRow` / `PLAN_SESSIONS_DIR_PREFIX` from `cli/src/commands/plan-artifacts.ts` so the rendered box and the text report cannot drift. -- `/memory` (alias `/mem`, `cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts`) inspects the persisted cross-session task memory for the current project (`.openbuff/memory/task-memory.json`). `/memory status` (the default) reports the record's revision and age, its goal and per-list counts, and how much of its evidence still verifies against disk, listing up to five stale paths; `/memory prune` drops evidence that no longer verifies. Both subcommands are move-aware via `collectWorkspaceMoves` + `WorkspaceJournalService` so a renamed file's evidence rebinds rather than being reported stale and deleted. +- `/memory` (alias `/mem`, `cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts`) inspects the persisted cross-session task memory for the current project (`.openbuff/memory/task-memory.json`). `/memory status` (the default) reports the record's revision and age, its goal and per-list counts, and how much of its evidence still verifies against disk, listing up to five stale paths; `/memory prune` drops evidence that no longer verifies. Both subcommands are move-aware via `collectWorkspaceMoves` + `WorkspaceJournalService` so a renamed file's evidence rebinds rather than being reported stale and deleted. `/memory audit-migration` is dedicated and read-only: before acquiring V2 it calls `inspectPersistedTaskMemoryV1`, blocks absent/invalid/unreadable states with bounded distinct reports, and passes the exact checksum-verified memory to the SDK audit without a second read. An `exact` audit reconstructs and compares every deterministic source-derived task/observation header and payload plus marker metadata; it is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`. The legacy loader still collapses non-valid inspection states to `undefined` for compatibility. `not-migrated` means removal is not ready and is not a product defect; all outcomes are rendered distinctly with bounded reason text. The audit's exactness check is order-insensitive (canonicalized before comparison) because the SQLite backend persists payloads via key-sorted `stableJson` and re-parses them on export; this is covered end-to-end by `cli/src/services/memory-v2/__tests__/memory-v1-migration-sqlite-roundtrip.test.ts`. - Slash-command descriptions should stay model-agnostic under BYOK/local mode. Use wording such as "configured reviewer" rather than naming hosted models. ## Import Guidelines @@ -919,7 +919,7 @@ Streaming markdown renders as plain text until the message or agent finishes. Th - _Knowledge refresh 2026-08-23: add `/memory` (alias `/mem`) slash command; staleness guard touch._ -- _Knowledge refresh 2026-09-12: Memory V2 implementation across `cli/src/services/memory-v2/` (SQLite kernel with WAL/append-only triggers/canonical projections/project binding, contained file I/O with O_NOFOLLOW/proc-fd anchoring, lease-based provider with opt-in fail-closed), `/memory` command expanded to 18 subcommands (authority/diagnose/query/inspect/consolidate/repair/revalidate/correct/forget/pin/export/import plus existing status/prune) with sanitized error boundaries and preview-by-default mutators, `cli/src/utils/codebuff-client.ts` gained `ManagedOpenbuffClient` with lease-based lifecycle and `memoryV2ClientConfigFromProvider` wiring, `cli/src/utils/env.ts` gained `getMemoryAuthoritySelection` for the `OPENBUFF_MEMORY_AUTHORITY` env var, `cli/src/types/chat.ts` gained `MemoryReportContentBlock` and `v2Lines` additive fields, and `cli/src/components/renderers/memory-box.tsx` gained the `report` state renderer with insert-command buttons._ +- _Knowledge refresh 2026-09-12: Memory V2 implementation across `cli/src/services/memory-v2/` (SQLite kernel with WAL/append-only triggers/canonical projections/project binding, contained file I/O with O_NOFOLLOW/proc-fd anchoring, lease-based provider with opt-in fail-closed), `/memory` expanded with authority/diagnose/query/inspect/consolidate/repair/revalidate/correct/forget/pin/export/import and the read-only SDK-backed `audit-migration` command plus existing status/prune, all with sanitized error boundaries and preview-by-default mutators; `cli/src/utils/codebuff-client.ts` gained `ManagedOpenbuffClient` with lease-based lifecycle and `memoryV2ClientConfigFromProvider` wiring, `cli/src/utils/env.ts` gained `getMemoryAuthoritySelection` for the `OPENBUFF_MEMORY_AUTHORITY` env var, `cli/src/types/chat.ts` gained `MemoryReportContentBlock` and `v2Lines` additive fields, and `cli/src/components/renderers/memory-box.tsx` gained the `report` state renderer with insert-command buttons._ - _Knowledge refresh 2026-08-31: live compaction status rendering (`context_compaction_status` consumption, run-correlated pending/settled pairing, replayed-pending-as-interrupted) in `cli/src/utils/sdk-event-handlers.ts` and `cli/src/components/renderers/compaction-box.tsx`._ diff --git a/cli/src/commands/__tests__/memory-command.test.ts b/cli/src/commands/__tests__/memory-command.test.ts index 06d7562133..7390d03736 100644 --- a/cli/src/commands/__tests__/memory-command.test.ts +++ b/cli/src/commands/__tests__/memory-command.test.ts @@ -89,6 +89,10 @@ function createDeps(options: { if (options.loadThrows) throw new Error('load exploded') return options.memory }, + inspectPersistedTaskMemoryV1: async () => + options.memory + ? { status: 'valid' as const, memory: options.memory } + : { status: 'absent' as const }, reconcileTaskMemoryEvidence: async (params: { workspaceMoves?: WorkspaceMoveRecord[] }) => { @@ -122,6 +126,22 @@ describe('/memory command', () => { expect(result).toContain('first successful run') }) + test('legacy typed dependency injectors may omit the V1 inspector', async () => { + const { deps } = createDeps({ memory: undefined }) + const { + inspectPersistedTaskMemoryV1: _newInspector, + ...legacyDependencies + } = deps + const compatibleDependencies: MemoryCommandDeps = legacyDependencies + + const result = await handleMemoryCommand( + 'audit-migration', + compatibleDependencies, + ) + + expect(result).toContain('Outcome: absent') + }) + test('status summarizes counts and lists stale evidence paths', async () => { const memory = makeMemory({ updatedAt: Date.now() - 2 * 60 * 60 * 1_000, @@ -333,10 +353,10 @@ describe('/memory command', () => { const { deps, calls } = createDeps({ memory: makeMemory() }) expect(await handleMemoryCommand('wat', deps)).toContain( - 'Usage: /memory [status|authority|diagnose|query ', + 'Usage: /memory [status|authority|diagnose|audit-migration|query ', ) expect(await handleMemoryCommand('PRUNE-ish', deps)).toContain( - 'Usage: /memory [status|authority|diagnose|query ', + 'Usage: /memory [status|authority|diagnose|audit-migration|query ', ) expect(calls.prune).toBe(0) }) @@ -344,12 +364,20 @@ describe('/memory command', () => { test('authority reports valid values and current selection without mutation', async () => { const { deps } = createDeps({ memory: makeMemory() }) deps.getMemoryV2 = async () => ({ - status: 'unavailable', requestedAuthority: 'sqlite-v2-opt-in', - effectiveAuthority: 'json-v1', degradation: 'fallback', retryable: true, + status: 'unavailable', + requestedAuthority: 'sqlite-v2-opt-in', + effectiveAuthority: 'json-v1', + degradation: 'fallback', + retryable: true, }) const result = await handleMemoryCommand('authority', deps) expect(result).toContain('json-v1, shadow-v2, sqlite-v2-opt-in') expect(result).toContain('Requested: sqlite-v2-opt-in; effective: json-v1') + expect(result).toContain('Release N') + expect(result).toContain( + 'json-v1 and shadow-v2 remain supported but are deprecated', + ) + expect(result).toContain('sqlite-v2-opt-in is the default and replacement') expect(result).toContain('reset/restart') }) @@ -390,13 +418,27 @@ describe('/memory blocks', () => { return { outcome: 'preview', plannedEvents: [] } } const observation = { - observationId: 'observation-1', taskId: 'task-1', kind: 'discovery', summary: 'old', detail: 'old', - confidence: 1, selectors: [], evidence: [], tags: [], observedAt: '2025-01-01T00:00:00.000Z', + observationId: 'observation-1', + taskId: 'task-1', + kind: 'discovery', + summary: 'old', + detail: 'old', + confidence: 1, + selectors: [], + evidence: [], + tags: [], + observedAt: '2025-01-01T00:00:00.000Z', } const observationEvent = MemoryEventEnvelopeSchema.parse({ - schemaVersion: 2, eventSchemaVersion: 1, eventType: 'observation.recorded', - eventId: 'observation-event-1', projectId: 'project-1', sessionId: 'session-1', sequence: 1, - occurredAt: '2025-01-01T00:00:00.000Z', payload: { payloadSchemaVersion: 1, observation }, + schemaVersion: 2, + eventSchemaVersion: 1, + eventType: 'observation.recorded', + eventId: 'observation-event-1', + projectId: 'project-1', + sessionId: 'session-1', + sequence: 1, + occurredAt: '2025-01-01T00:00:00.000Z', + payload: { payloadSchemaVersion: 1, observation }, }) const operator = { consolidate: invoke('consolidate'), @@ -404,11 +446,24 @@ describe('/memory blocks', () => { revalidate: invoke('revalidate'), correct: invoke('correct'), } - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', operator, - repository: { export: async () => ({ outcome: 'page', events: [observationEvent], nextAfterEventId: null }) }, - release: async () => {}, - }) as unknown as Awaited>> + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator, + repository: { + export: async () => ({ + outcome: 'page', + events: [observationEvent], + nextAfterEventId: null, + }), + }, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > const commands = [ 'consolidate', @@ -419,23 +474,36 @@ describe('/memory blocks', () => { ] for (const command of commands) { const preview = await handleMemoryCommandBlocks(command, deps) - const applied = await handleMemoryCommandBlocks(`${command} --apply`, deps) + const applied = await handleMemoryCommandBlocks( + `${command} --apply`, + deps, + ) expect(preview.state).toBe('report') expect(applied.state).toBe('report') const previewCall = calls.at(-2)?.request as { mode?: string } const applyCall = calls.at(-1)?.request as { mode?: string } expect(previewCall.mode).toBe('preview') expect(applyCall.mode).toBe('apply') - if (preview.state === 'report') expect(preview.insertCommands?.[0]?.command).toContain('--apply') + if (preview.state === 'report') + expect(preview.insertCommands?.[0]?.command).toContain('--apply') } }) test('failed rejected and busy mutators render error tone with sanitized retryability', async () => { const { deps } = createDeps({ memory: makeMemory() }) const outcomes = [ - { outcome: 'failed', error: { message: '/secret/store.sqlite locked', retryable: true } }, - { outcome: 'rejected', error: { message: 'Request was rejected.', retryable: false } }, - { outcome: 'busy', error: { message: 'Try again shortly.', retryable: true } }, + { + outcome: 'failed', + error: { message: '/secret/store.sqlite locked', retryable: true }, + }, + { + outcome: 'rejected', + error: { message: 'Request was rejected.', retryable: false }, + }, + { + outcome: 'busy', + error: { message: 'Try again shortly.', retryable: true }, + }, ] let index = 0 const observationEvent = MemoryEventEnvelopeSchema.parse({ @@ -463,18 +531,24 @@ describe('/memory blocks', () => { }, }, }) - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', - operator: { correct: async () => outcomes[index++]! }, - repository: { - export: async () => ({ - outcome: 'page', - events: [observationEvent], - nextAfterEventId: null, - }), - }, - release: async () => {}, - }) as unknown as Awaited>> + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: { correct: async () => outcomes[index++]! }, + repository: { + export: async () => ({ + outcome: 'page', + events: [observationEvent], + nextAfterEventId: null, + }), + }, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > for (const command of ['pin one', 'pin one', 'pin one']) { const block = await handleMemoryCommandBlocks(command, deps) @@ -486,42 +560,392 @@ describe('/memory blocks', () => { } }) - test('query and diagnose render bounded generic reports', async () => { + test('audit-migration blocks every non-valid inspection before provider acquisition', async () => { + const cases = [ + [{ status: 'absent' as const }, 'Outcome: absent', 'secondary'], + [ + { status: 'invalid' as const, reason: 'malformed-json' as const }, + 'malformed JSON', + 'warning', + ], + [ + { status: 'invalid' as const, reason: 'schema-invalid' as const }, + 'schema validation', + 'warning', + ], + [ + { status: 'invalid' as const, reason: 'checksum-mismatch' as const }, + 'checksum validation', + 'warning', + ], + [ + { status: 'unreadable' as const, reason: 'read-failed' as const }, + 'could not be read safely', + 'error', + ], + ] as const + for (const [inspection, expected, tone] of cases) { + const { deps } = createDeps({ memory: makeMemory() }) + let providers = 0 + let audits = 0 + deps.inspectPersistedTaskMemoryV1 = async () => inspection + deps.getMemoryV2 = async () => { + providers++ + throw new Error('/private/provider') + } + deps.auditTaskMemoryV1Migration = async () => { + audits++ + return { outcome: 'no-record' } + } + const block = await handleMemoryCommandBlocks('audit-migration', deps) + if (block.state !== 'report') throw new Error('expected report') + expect(block.tone).toBe(tone) + expect(block.lines.join('\n')).toContain(expected) + expect(block.lines.join('\n')).not.toMatch(/private|provider/i) + expect(providers).toBe(0) + expect(audits).toBe(0) + } + }) + + test('audit-migration sanitizes an inspector throw without opening V2', async () => { const { deps } = createDeps({ memory: makeMemory() }) + let providers = 0 + deps.inspectPersistedTaskMemoryV1 = async () => { + throw new Error('/secret/v1.json contents') + } + deps.getMemoryV2 = async () => { + providers++ + throw new Error('unused') + } + const block = await handleMemoryCommandBlocks('audit-migration', deps) + if (block.state !== 'report') throw new Error('expected report') + expect(block.tone).toBe('error') + expect(block.lines.join('\n')).not.toMatch(/secret|v1\.json|contents/i) + expect(providers).toBe(0) + }) + + test('audit-migration preserves valid unavailable authority behavior without audit or release', async () => { + const memory = makeMemory() + const { deps } = createDeps({ memory }) + let audits = 0 + let releases = 0 deps.getMemoryV2 = async () => ({ - status: 'available', projectId: 'project-1', operator: {}, - repository: { - query: async () => ({ outcome: 'result', result: { matchedTasks: [], verifiedKnowledge: [], reusableDiscovery: [], rereadRequired: [], historicalContext: [], degradation: { state: 'none' } } }), - health: async () => ({ status: 'healthy', authority: { kind: 'authoritative' }, backend: { backendId: 'bun-sqlite-memory-v2', capabilities: ['query'] }, issues: [] }), - kernelHealth: async () => ({ status: 'healthy', schemaVersion: 2, projectionCursor: 0 }), - getCapabilities: async () => ({ status: 'ok', capabilities: [{ name: 'query', available: true }] }), - export: async () => ({ outcome: 'page', events: [], nextAfterEventId: null }), + status: 'unavailable', + requestedAuthority: 'sqlite-v2-opt-in', + effectiveAuthority: 'sqlite-v2-opt-in', + degradation: + 'Memory V2 storage is unavailable; V1 remains disabled under opt-in authority.', + retryable: true, + }) + deps.auditTaskMemoryV1Migration = async () => { + audits++ + return { outcome: 'no-record' } + } + const block = await handleMemoryCommandBlocks('audit-migration', deps) + if (block.state !== 'report') throw new Error('expected report') + expect(block.lines.join('\n')).toContain( + 'Requested authority: sqlite-v2-opt-in; active authority: sqlite-v2-opt-in', + ) + expect(audits).toBe(0) + expect(releases).toBe(0) + }) + + test('audit-migration renders an exact lossless SDK outcome with success tone', async () => { + const memory = makeMemory() + const { deps, calls } = createDeps({ memory }) + const repository = { + append: async () => { + throw new Error('must not append') }, - release: async () => {}, - }) as unknown as Awaited>> - const query = await handleMemoryCommandBlocks('query deterministic lookup', deps) + } + let request: unknown + let releases = 0 + deps.auditTaskMemoryV1Migration = async (value) => { + request = value + return { + outcome: 'exact', + revision: 3, + checksum: 'full-source-checksum', + identity: 'v1:3:id', + markerEventId: 'marker', + repositoryLastEventId: 'tail', + importedTaskId: 'task', + importedObservationIds: ['observation-1', 'observation-2'], + omittedFields: 0, + warnings: [], + } as unknown as Awaited< + ReturnType + > + } + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'sqlite-v2-opt-in', + effectiveAuthority: 'sqlite-v2-opt-in', + projectId: 'project-1', + operator: {}, + repository, + release: async () => { + releases++ + }, + }) as unknown as Awaited< + ReturnType> + > + + const block = await handleMemoryCommandBlocks('audit-migration', deps) + + if (block.state !== 'report') throw new Error('expected report') + expect(block.tone).toBe('success') + expect(block.lines.join('\n')).toContain('Outcome: exact') + expect(block.lines.join('\n')).toContain('Source revision: 3') + expect(block.lines.join('\n')).toContain( + 'Source checksum: full-source-checksum', + ) + expect(block.lines.join('\n')).toContain( + 'Marker verification and full deterministic source-derived task/observation body equality: exact', + ) + expect(block.lines.join('\n')).toContain('Imported observations: 2') + expect(block.lines.join('\n')).toContain('Omitted fields: 0') + expect(block.lines.join('\n')).toContain('Truncated fields: 0') + expect(block.lines.join('\n')).toContain('Warnings: 0 (none)') + expect(block.lines.join('\n')).toContain('Lossless migration evidence: yes') + expect(block.lines.join('\n')).toContain('No writes were performed') + expect(request).toEqual({ memory, projectId: 'project-1', repository }) + expect(calls.prune).toBe(0) + expect(releases).toBe(1) + }) + + test('audit-migration warns when an exact outcome documents loss', async () => { + const { deps } = createDeps({ memory: makeMemory() }) + deps.auditTaskMemoryV1Migration = async () => + ({ + outcome: 'exact', + revision: 3, + checksum: 'checksum', + identity: 'v1:3:id', + markerEventId: 'marker', + repositoryLastEventId: 'tail', + importedTaskId: 'task', + importedObservationIds: [], + omittedFields: 2, + truncatedFields: 1, + warnings: ['text-truncated', 'goal-excluded'], + }) as unknown as Awaited< + ReturnType + > + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: {}, + repository: {}, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > + + const block = await handleMemoryCommandBlocks('audit-migration', deps) + + if (block.state !== 'report') throw new Error('expected report') + expect(block.tone).toBe('warning') + expect(block.lines.join('\n')).toContain('Omitted fields: 2') + expect(block.lines.join('\n')).toContain('Truncated fields: 1') + expect(block.lines.join('\n')).toContain( + 'Warnings: 2 (text-truncated, goal-excluded)', + ) + expect(block.lines.join('\n')).toContain('Lossless migration evidence: no') + }) + + test('audit-migration renders every non-exact outcome distinctly and sanitizes failures', async () => { + const { deps } = createDeps({ memory: makeMemory() }) + const outcomes = [ + { outcome: 'no-record' }, + { outcome: 'not-migrated', revision: 3, checksum: 'checksum' }, + { + outcome: 'incomplete', + reason: 'reservation-only', + revision: 3, + checksum: 'checksum', + }, + { + outcome: 'mismatch', + reason: 'imported-body-mismatch', + revision: 3, + checksum: 'checksum', + }, + { outcome: 'rejected', reason: 'repository-rejected' }, + { outcome: 'failed', reason: 'repository-failed' }, + ] as const + let index = 0 + let releases = 0 + deps.auditTaskMemoryV1Migration = async () => + outcomes[index++] as unknown as Awaited< + ReturnType + > + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: {}, + repository: {}, + release: async () => { + releases++ + }, + }) as unknown as Awaited< + ReturnType> + > + + const rendered = [] + for (let i = 0; i < outcomes.length; i++) { + const block = await handleMemoryCommandBlocks('audit-migration', deps) + if (block.state !== 'report') throw new Error('expected report') + rendered.push(block.lines.join('\n')) + } + expect(rendered[0]).toContain('Outcome: no-record') + expect(rendered[0]).toContain('No currently loaded V1') + expect(rendered[1]).toContain('Outcome: not-migrated') + expect(rendered[2]).toContain('Outcome: incomplete') + expect(rendered[3]).toContain('Outcome: mismatch') + expect(rendered[4]).toContain('Outcome: rejected') + expect(rendered[5]).toContain('Outcome: failed') + expect(rendered.join('\n')).not.toMatch(/\.sqlite|SELECT|\/secret/i) + expect(releases).toBe(outcomes.length) + }) + + test('audit-migration releases the provider lease once when the injected audit throws', async () => { + const { deps } = createDeps({ memory: makeMemory() }) + let releases = 0 + deps.auditTaskMemoryV1Migration = async () => { + throw new Error('/secret/store.sqlite SELECT token') + } + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: {}, + repository: {}, + release: async () => { + releases++ + }, + }) as unknown as Awaited< + ReturnType> + > + + const block = await handleMemoryCommandBlocks('audit-migration', deps) + + expect(releases).toBe(1) + if (block.state !== 'report') throw new Error('expected report') + expect(block.tone).toBe('error') + expect(block.lines.join('\n')).not.toMatch(/secret|sqlite|SELECT|token/i) + }) + + test('query and diagnose render bounded generic reports', async () => { + const { deps } = createDeps({ memory: makeMemory() }) + deps.getMemoryV2 = async () => + ({ + status: 'available', + projectId: 'project-1', + operator: {}, + repository: { + query: async () => ({ + outcome: 'result', + result: { + matchedTasks: [], + verifiedKnowledge: [], + reusableDiscovery: [], + rereadRequired: [], + historicalContext: [], + degradation: { state: 'none' }, + }, + }), + health: async () => ({ + status: 'healthy', + authority: { kind: 'authoritative' }, + backend: { + backendId: 'bun-sqlite-memory-v2', + capabilities: ['query'], + }, + issues: [], + }), + kernelHealth: async () => ({ + status: 'healthy', + schemaVersion: 2, + projectionCursor: 0, + }), + getCapabilities: async () => ({ + status: 'ok', + capabilities: [{ name: 'query', available: true }], + }), + export: async () => ({ + outcome: 'page', + events: [], + nextAfterEventId: null, + }), + }, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > + const query = await handleMemoryCommandBlocks( + 'query deterministic lookup', + deps, + ) const diagnose = await handleMemoryCommandBlocks('diagnose', deps) expect(query.state).toBe('report') expect(diagnose.state).toBe('report') if (query.state === 'report') expect(query.lines[0]).toContain('Tasks: 0') - if (diagnose.state === 'report') expect(diagnose.lines.join('\n')).toContain('Kernel: healthy') + if (diagnose.state === 'report') { + expect(diagnose.lines.join('\n')).toContain('Kernel: healthy') + expect(diagnose.lines.join('\n')).toContain('Release N') + expect(diagnose.lines.join('\n')).toContain( + 'json-v1 and shadow-v2 remain supported but are deprecated', + ) + } }) test('inspect and diagnose use project-scoped canonical export inventories', async () => { const { deps } = createDeps({ memory: makeMemory() }) let exports = 0 let lowLevelLists = 0 - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', operator: {}, - repository: { - export: async () => { exports++; return { outcome: 'page', events: [], nextAfterEventId: null } }, - listEvents: async () => { lowLevelLists++; return { status: 'ok', events: [] } }, - health: async () => ({ status: 'healthy', authority: { kind: 'authoritative' }, backend: { backendId: 'bun-sqlite-memory-v2', capabilities: [] }, issues: [] }), - kernelHealth: async () => ({ status: 'healthy', schemaVersion: 2, projectionCursor: 0 }), - getCapabilities: async () => ({ status: 'ok', capabilities: [] }), - }, - release: async () => {}, - }) as unknown as Awaited>> + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: {}, + repository: { + export: async () => { + exports++ + return { outcome: 'page', events: [], nextAfterEventId: null } + }, + listEvents: async () => { + lowLevelLists++ + return { status: 'ok', events: [] } + }, + health: async () => ({ + status: 'healthy', + authority: { kind: 'authoritative' }, + backend: { backendId: 'bun-sqlite-memory-v2', capabilities: [] }, + issues: [], + }), + kernelHealth: async () => ({ + status: 'healthy', + schemaVersion: 2, + projectionCursor: 0, + }), + getCapabilities: async () => ({ status: 'ok', capabilities: [] }), + }, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > await handleMemoryCommandBlocks('inspect', deps) await handleMemoryCommandBlocks('diagnose', deps) expect(exports).toBe(2) @@ -535,12 +959,28 @@ describe('/memory blocks', () => { symlinkSync(outside, join(root, '.openbuff', 'memory')) const { deps } = createDeps({ memory: makeMemory() }) deps.getRootDir = () => root - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', - operator: { exportManifest: async () => ({ outcome: 'exported', manifest: { checksum: 'sha256:1234567890abcdef', canonicalEventCount: 0, events: [], warnings: [] } }) }, - repository: {}, - release: async () => {}, - }) as unknown as Awaited>> + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: { + exportManifest: async () => ({ + outcome: 'exported', + manifest: { + checksum: 'sha256:1234567890abcdef', + canonicalEventCount: 0, + events: [], + warnings: [], + }, + }), + }, + repository: {}, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > const block = await handleMemoryCommandBlocks('export', deps) if (block.state !== 'report') throw new Error('expected report') expect(block.tone).toBe('error') @@ -560,13 +1000,36 @@ describe('/memory blocks', () => { const { deps } = createDeps({ memory: makeMemory() }) deps.getRootDir = () => root let imports = 0 - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', - operator: { importManifest: async () => { imports++; return { outcome: 'imported' } } }, - repository: { export: async () => ({ outcome: 'page', events: [{ eventId: 'existing' }], nextAfterEventId: null }) }, - release: async () => {}, - }) as unknown as Awaited>> - for (const path of ['/absolute.json', '../escape.json', 'imports/link.json', 'imports/oversized.json', 'imports/malformed.json']) { + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: { + importManifest: async () => { + imports++ + return { outcome: 'imported' } + }, + }, + repository: { + export: async () => ({ + outcome: 'page', + events: [{ eventId: 'existing' }], + nextAfterEventId: null, + }), + }, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > + for (const path of [ + '/absolute.json', + '../escape.json', + 'imports/link.json', + 'imports/oversized.json', + 'imports/malformed.json', + ]) { const block = await handleMemoryCommandBlocks(`import ${path}`, deps) if (block.state !== 'report') throw new Error('expected report') expect(block.tone).toBe('error') @@ -577,21 +1040,54 @@ describe('/memory blocks', () => { test('available command leases release exactly once on success and throw', async () => { const { deps } = createDeps({ memory: makeMemory() }) let releases = 0 - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', - operator: {}, - repository: { - query: async () => ({ outcome: 'result', result: { matchedTasks: [], verifiedKnowledge: [], reusableDiscovery: [], rereadRequired: [], historicalContext: [], degradation: { state: 'none' } } }), - }, - release: async () => { releases++ }, - }) as unknown as Awaited>> - expect((await handleMemoryCommandBlocks('query safe', deps)).state).toBe('report') + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: {}, + repository: { + query: async () => ({ + outcome: 'result', + result: { + matchedTasks: [], + verifiedKnowledge: [], + reusableDiscovery: [], + rereadRequired: [], + historicalContext: [], + degradation: { state: 'none' }, + }, + }), + }, + release: async () => { + releases++ + }, + }) as unknown as Awaited< + ReturnType> + > + expect((await handleMemoryCommandBlocks('query safe', deps)).state).toBe( + 'report', + ) expect(releases).toBe(1) - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', operator: {}, - repository: { query: async () => { throw new Error('/secret/store.sqlite SELECT token') } }, - release: async () => { releases++ }, - }) as unknown as Awaited>> + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: {}, + repository: { + query: async () => { + throw new Error('/secret/store.sqlite SELECT token') + }, + }, + release: async () => { + releases++ + }, + }) as unknown as Awaited< + ReturnType> + > const failed = await handleMemoryCommandBlocks('query safe', deps) expect(releases).toBe(2) if (failed.state !== 'report') throw new Error('expected report') @@ -601,21 +1097,37 @@ describe('/memory blocks', () => { test('pagination rejects repeated and empty cursors without leaking details', async () => { const { deps } = createDeps({ memory: makeMemory() }) let calls = 0 - deps.getMemoryV2 = async () => ({ - status: 'available', requestedAuthority: 'shadow-v2', effectiveAuthority: 'shadow-v2', projectId: 'project-1', operator: {}, - repository: { export: async () => { calls++; return { outcome: 'page', events: [], nextAfterEventId: 'same' } } }, - release: async () => {}, - }) as unknown as Awaited>> + deps.getMemoryV2 = async () => + ({ + status: 'available', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'shadow-v2', + projectId: 'project-1', + operator: {}, + repository: { + export: async () => { + calls++ + return { outcome: 'page', events: [], nextAfterEventId: 'same' } + }, + }, + release: async () => {}, + }) as unknown as Awaited< + ReturnType> + > const block = await handleMemoryCommandBlocks('inspect target', deps) if (block.state !== 'report') throw new Error('expected report') expect(block.tone).toBe('error') expect(calls).toBe(1) - expect(block.lines.join('\n')).toContain('operation could not be completed safely') + expect(block.lines.join('\n')).toContain( + 'operation could not be completed safely', + ) }) test('sanitized boundary hides arbitrary provider failures', async () => { const { deps } = createDeps({ memory: makeMemory() }) - deps.getMemoryV2 = async () => { throw new Error('/home/private/store.sqlite SELECT api_key=secret') } + deps.getMemoryV2 = async () => { + throw new Error('/home/private/store.sqlite SELECT api_key=secret') + } const block = await handleMemoryCommandBlocks('diagnose', deps) if (block.state !== 'report') throw new Error('expected report') const output = block.lines.join('\n') @@ -985,11 +1497,34 @@ describe('/memory blocks', () => { expect(block.type).toBe('memory') expect(block.state).toBe('report') - if (block.state === 'report') expect(block.lines.join('\n')).toContain('Usage: /memory [status|authority|diagnose|query ') + if (block.state === 'report') + expect(block.lines.join('\n')).toContain( + 'Usage: /memory [status|authority|diagnose|audit-migration|query ', + ) expect(calls.prune).toBe(0) expect(calls.moves).toBe(0) }) + test('V2 status lines include Release N authority deprecation guidance', async () => { + const memory = makeMemory() + const { deps } = createDeps({ memory, reconciled: memory }) + deps.getMemoryV2 = async () => ({ + status: 'unavailable', + requestedAuthority: 'shadow-v2', + effectiveAuthority: 'json-v1', + degradation: 'V1 JSON memory is authoritative; SQLite was not opened.', + retryable: false, + }) + + const output = await handleMemoryCommand('status', deps) + + expect(output).toContain('Release N') + expect(output).toContain( + 'json-v1 and shadow-v2 remain supported but are deprecated', + ) + expect(output).toContain('sqlite-v2-opt-in is the default and replacement') + }) + test('status forwards journal moves so evidence rebinds', async () => { const memory = makeMemory({ evidence: [makeEvidence({ id: 'ev-moved', path: 'old.ts' })], diff --git a/cli/src/commands/memory-command.ts b/cli/src/commands/memory-command.ts index 8fe693922d..9df4bee2a6 100644 --- a/cli/src/commands/memory-command.ts +++ b/cli/src/commands/memory-command.ts @@ -1,12 +1,14 @@ /** - * `/memory` — inspect and prune persisted cross-session task memory for the - * current project root. Read-only by default; `prune` drops stale evidence. + * `/memory` — inspect, audit, and prune persisted cross-session task memory for + * the current project root. Read-only by default; `prune` drops stale evidence. */ import { createHash } from 'node:crypto' import { + auditTaskMemoryV1Migration, collectWorkspaceMoves, getHarnessStateDir, + inspectPersistedTaskMemoryV1, loadPersistedTaskMemory, pruneStaleTaskMemoryEvidence, reconcileTaskMemoryEvidence, @@ -27,13 +29,20 @@ import { import { getProjectMemoryV2Provider } from '../services/memory-v2/provider' import { formatAge, pluralizeEntries } from '../utils/format-helpers' -import type { TaskMemoryPruneOutcome, WorkspaceMoveRecord } from '@openbuff/sdk' +import type { + TaskMemoryPruneOutcome, + TaskMemoryV1Inspection, + V1MigrationAuditOutcome, + WorkspaceMoveRecord, +} from '@openbuff/sdk' export type MemoryCommandDeps = { getRootDir: () => string loadPersistedTaskMemory: typeof loadPersistedTaskMemory + inspectPersistedTaskMemoryV1?: typeof inspectPersistedTaskMemoryV1 reconcileTaskMemoryEvidence: typeof reconcileTaskMemoryEvidence pruneStaleTaskMemoryEvidence: typeof pruneStaleTaskMemoryEvidence + auditTaskMemoryV1Migration: typeof auditTaskMemoryV1Migration /** * Journal-recorded file moves for this project. Both subcommands reconcile * evidence, and reconciliation without moves reports a renamed file's @@ -68,8 +77,10 @@ async function loadWorkspaceMoves( const defaultDeps: MemoryCommandDeps = { getRootDir: getProjectRoot, loadPersistedTaskMemory, + inspectPersistedTaskMemoryV1, reconcileTaskMemoryEvidence, pruneStaleTaskMemoryEvidence, + auditTaskMemoryV1Migration, getWorkspaceMoves: loadWorkspaceMoves, getMemoryV2: getProjectMemoryV2Provider, } @@ -196,20 +207,38 @@ function memoryBlockToString( } } -const MEMORY_USAGE = 'Usage: /memory [status|authority|diagnose|query |inspect [eventId]|consolidate [--apply] [--task ]|repair [--apply]|revalidate [--apply]|correct [--apply]|forget [--apply]|pin [--apply]|export [--format json|markdown] [--include-stale]|import [--apply]|prune]' +const MEMORY_USAGE = + 'Usage: /memory [status|authority|diagnose|audit-migration|query |inspect [eventId]|consolidate [--apply] [--task ]|repair [--apply]|revalidate [--apply]|correct [--apply]|forget [--apply]|pin [--apply]|export [--format json|markdown] [--include-stale]|import [--apply]|prune]' +const RELEASE_N_AUTHORITY_WARNING = + 'Release N: json-v1 and shadow-v2 remain supported but are deprecated; sqlite-v2-opt-in is the default and replacement.' const CLI_SESSION_ID = 'memory-cli' const EXPORT_MAX_BYTES = 8 * 1024 * 1024 -function report(title: string, lines: string[], tone: import('../types/chat').MemoryReportTone = 'secondary', insertCommands?: Array<{ label: string; command: string }>): import('../types/chat').MemoryContentBlock { - return { type: 'memory', state: 'report', title, tone, lines: lines.slice(0, 100), ...(insertCommands?.length ? { insertCommands } : {}) } +function report( + title: string, + lines: string[], + tone: import('../types/chat').MemoryReportTone = 'secondary', + insertCommands?: Array<{ label: string; command: string }>, +): import('../types/chat').MemoryContentBlock { + return { + type: 'memory', + state: 'report', + title, + tone, + lines: lines.slice(0, 100), + ...(insertCommands?.length ? { insertCommands } : {}), + } } -function commandError(message: string): import('../types/chat').MemoryContentBlock { +function commandError( + message: string, +): import('../types/chat').MemoryContentBlock { return report('Memory V2', [message.slice(0, 1_024)], 'error') } function safeOperationMessage(value: unknown): string { - if (typeof value !== 'string' || value.length === 0) return 'The operation could not be completed.' + if (typeof value !== 'string' || value.length === 0) + return 'The operation could not be completed.' const allowed = [ 'Canonical inventory is temporarily unavailable.', 'Canonical inventory failed validation.', @@ -220,28 +249,212 @@ function safeOperationMessage(value: unknown): string { 'V1 JSON memory is authoritative; SQLite was not opened.', 'Invalid memory authority; using V1 memory only.', ] - return allowed.includes(value) ? value : 'The operation could not be completed safely.' + return allowed.includes(value) + ? value + : 'The operation could not be completed safely.' } -function operationLines(outcome: { outcome: string; error?: { message?: unknown; retryable?: unknown } }, extra: string[] = []): string[] { +function operationLines( + outcome: { + outcome: string + error?: { message?: unknown; retryable?: unknown } + }, + extra: string[] = [], +): string[] { return [ `Outcome: ${outcome.outcome}.`, ...extra, - ...(outcome.error ? [ - `Error: ${safeOperationMessage(outcome.error.message)}`, - `Retryable: ${outcome.error.retryable === true ? 'yes' : 'no'}.`, - ] : []), + ...(outcome.error + ? [ + `Error: ${safeOperationMessage(outcome.error.message)}`, + `Retryable: ${outcome.error.retryable === true ? 'yes' : 'no'}.`, + ] + : []), ] } function isFailureOutcome(outcome: { outcome: string }): boolean { - return ['failed', 'rejected', 'busy', 'integrity-mismatch'].includes(outcome.outcome) + return ['failed', 'rejected', 'busy', 'integrity-mismatch'].includes( + outcome.outcome, + ) } function deterministicId(prefix: string, value: string): string { return `${prefix}:${createHash('sha256').update(value).digest('hex').slice(0, 24)}` } +const AUDIT_INCOMPLETE_REASONS: Record< + Extract< + V1MigrationAuditOutcome, + { outcome: 'incomplete' | 'mismatch' } + >['reason'], + string +> = { + 'reservation-only': 'Only the migration reservation was found.', + 'legacy-marker-unverifiable': + 'The legacy migration marker cannot verify this source record.', + 'missing-imported-task': 'The imported task body is missing.', + 'missing-imported-observations': + 'One or more imported observations are missing.', + 'imported-body-mismatch': + 'The imported body or provenance does not match the source marker.', + 'revision-conflict': + 'Conflicting migration evidence exists for this source revision.', + 'checksum-conflict': + 'Conflicting migration evidence exists for this source checksum.', +} + +const AUDIT_FAILURE_REASONS: Record< + Extract< + V1MigrationAuditOutcome, + { outcome: 'rejected' | 'failed' } + >['reason'], + string +> = { + 'checksum-mismatch': 'The V1 source record failed checksum validation.', + 'repository-rejected': 'The audit read was rejected safely.', + 'repository-failed': 'The audit read could not be completed safely.', + 'invalid-export': + 'The audit could not validate the exported migration evidence.', + 'wrong-project': + 'The exported migration evidence belongs to another project.', + 'pagination-invalid': + 'The exported migration evidence did not form a valid bounded sequence.', + 'page-limit-exceeded': + 'The migration evidence exceeded the bounded audit scan.', +} + +function renderMigrationAudit( + outcome: V1MigrationAuditOutcome, +): import('../types/chat').MemoryContentBlock { + if (outcome.outcome === 'no-record') { + return report( + 'Memory V1 migration audit', + [ + 'Outcome: no-record.', + 'No currently loaded V1 task-memory record is available to audit.', + 'No writes were performed.', + ], + 'secondary', + ) + } + if (outcome.outcome === 'exact') { + const truncatedFields = outcome.truncatedFields ?? 0 + const warningCount = outcome.warnings.length + const lossless = + outcome.omittedFields === 0 && truncatedFields === 0 && warningCount === 0 + return report( + 'Memory V1 migration audit', + [ + 'Outcome: exact.', + `Source revision: ${outcome.revision}.`, + `Source checksum: ${outcome.checksum}.`, + 'Marker verification and full deterministic source-derived task/observation body equality: exact.', + `Imported observations: ${outcome.importedObservationIds.length}.`, + `Omitted fields: ${outcome.omittedFields}.`, + `Truncated fields: ${truncatedFields}.`, + `Warnings: ${warningCount}${warningCount ? ` (${outcome.warnings.slice(0, 100).join(', ')})` : ' (none)'}.`, + `Lossless migration evidence: ${lossless ? 'yes' : 'no'}.`, + 'No writes were performed.', + ], + lossless ? 'success' : 'warning', + ) + } + if (outcome.outcome === 'not-migrated') { + return report( + 'Memory V1 migration audit', + [ + 'Outcome: not-migrated.', + `Source revision: ${outcome.revision}.`, + `Source checksum: ${outcome.checksum}.`, + 'No matching migration marker was found in the bounded V2 audit.', + 'No writes were performed.', + ], + 'warning', + ) + } + if (outcome.outcome === 'incomplete' || outcome.outcome === 'mismatch') { + return report( + 'Memory V1 migration audit', + [ + `Outcome: ${outcome.outcome}.`, + `Source revision: ${outcome.revision}.`, + `Source checksum: ${outcome.checksum}.`, + AUDIT_INCOMPLETE_REASONS[outcome.reason], + 'No writes were performed.', + ], + outcome.outcome === 'mismatch' ? 'error' : 'warning', + ) + } + if (outcome.outcome === 'rejected' || outcome.outcome === 'failed') { + return report( + 'Memory V1 migration audit', + [ + `Outcome: ${outcome.outcome}.`, + AUDIT_FAILURE_REASONS[outcome.reason], + ...(outcome.revision === undefined + ? [] + : [`Source revision: ${outcome.revision}.`]), + ...(outcome.checksum === undefined + ? [] + : [`Source checksum: ${outcome.checksum}.`]), + 'No writes were performed.', + ], + outcome.outcome === 'failed' ? 'error' : 'warning', + ) + } + return report( + 'Memory V1 migration audit', + ['The audit returned an unsupported outcome.', 'No writes were performed.'], + 'error', + ) +} + +const V1_INSPECTION_INVALID_REASONS: Record< + Extract['reason'], + string +> = { + 'malformed-json': 'The persisted V1 record is malformed JSON.', + 'schema-invalid': 'The persisted V1 record failed schema validation.', + 'checksum-mismatch': 'The persisted V1 record failed checksum validation.', +} + +function renderV1InspectionBlock( + inspection: Exclude, +): import('../types/chat').MemoryContentBlock { + if (inspection.status === 'absent') { + return report( + 'Memory V1 migration audit', + [ + 'Outcome: absent.', + 'Independent read-only inspection verified that no persisted V1 record exists.', + 'Memory V2 was not opened. No writes were performed.', + ], + 'secondary', + ) + } + if (inspection.status === 'invalid') { + return report( + 'Memory V1 migration audit', + [ + 'Outcome: invalid.', + V1_INSPECTION_INVALID_REASONS[inspection.reason], + 'Audit blocked; Memory V2 was not opened. No writes were performed.', + ], + 'warning', + ) + } + return report( + 'Memory V1 migration audit', + [ + 'Outcome: unreadable.', + 'The persisted V1 record could not be read safely.', + 'Audit blocked; Memory V2 was not opened. No writes were performed.', + ], + 'error', + ) +} + async function getV2(deps: MemoryCommandDeps) { return deps.getMemoryV2 ? deps.getMemoryV2(deps.getRootDir()) @@ -254,153 +467,416 @@ async function getV2(deps: MemoryCommandDeps) { } } -function parseArgs(rawArgs: string): { command: string; args: string[]; apply: boolean } { +function parseArgs(rawArgs: string): { + command: string + args: string[] + apply: boolean +} { const tokens = rawArgs.trim().split(/\s+/).filter(Boolean) const command = (tokens.shift() ?? 'status').toLowerCase() const apply = tokens.includes('--apply') return { command, args: tokens.filter((token) => token !== '--apply'), apply } } -async function runV2Command(rawArgs: string, deps: MemoryCommandDeps): Promise { +async function runV2Command( + rawArgs: string, + deps: MemoryCommandDeps, +): Promise { const parsed = parseArgs(rawArgs) - if (!['authority', 'diagnose', 'query', 'inspect', 'consolidate', 'repair', 'revalidate', 'correct', 'forget', 'pin', 'export', 'import'].includes(parsed.command)) return commandError(MEMORY_USAGE) + if ( + ![ + 'authority', + 'diagnose', + 'audit-migration', + 'query', + 'inspect', + 'consolidate', + 'repair', + 'revalidate', + 'correct', + 'forget', + 'pin', + 'export', + 'import', + ].includes(parsed.command) + ) + return commandError(MEMORY_USAGE) + let inspectedMemory: + | Extract + | undefined + if (parsed.command === 'audit-migration') { + const inspect = + deps.inspectPersistedTaskMemoryV1 ?? inspectPersistedTaskMemoryV1 + const inspected = await inspect({ rootDir: deps.getRootDir() }) + if (inspected.status !== 'valid') return renderV1InspectionBlock(inspected) + inspectedMemory = inspected + } const v2 = await getV2(deps) try { if (parsed.command === 'authority') { - return report('Memory authority', [ - 'Valid values: json-v1, shadow-v2, sqlite-v2-opt-in.', - `Requested: ${v2.requestedAuthority}; effective: ${v2.effectiveAuthority}.`, - 'Switch via OPENBUFF_MEMORY_AUTHORITY or the SDK authority option, then reset/restart the client.', - ]) - } - if (v2.status === 'unavailable') return report('Memory V2 unavailable', [ - `Requested authority: ${v2.requestedAuthority}; active authority: ${v2.effectiveAuthority}.`, - safeOperationMessage(v2.degradation), - ], 'warning') - const scope = { schemaVersion: 2 as const, projectId: v2.projectId, sessionId: MemorySessionIdSchema.parse(CLI_SESSION_ID) } - const now = new Date().toISOString() - - if (parsed.command === 'diagnose') { - const [health, kernel, capabilities, inventory] = await Promise.all([ - v2.repository.health({ schemaVersion: 2, projectId: v2.projectId }), - v2.repository.kernelHealth(), - v2.repository.getCapabilities(), - canonicalInventory(v2, 1_000), - ]) - const events = inventory.events - return report('Memory V2 diagnosis', [ - `Requested authority: ${v2.requestedAuthority}; active authority: ${v2.effectiveAuthority}.`, - 'V1 compatibility shadow: available when task-memory.json is hydrated and remains normally persisted.', - `Health: ${health.status}; repository authority: ${health.authority.kind}.`, - `Backend: ${health.backend.backendId}; capabilities: ${health.backend.capabilities.join(', ')}.`, - `Kernel: ${kernel.status}; schema: ${kernel.schemaVersion ?? 'unknown'}; projection cursor: ${kernel.projectionCursor ?? 'unknown'}.`, - `Integrity: ${health.issues.length ? health.issues.map(safeOperationMessage).join('; ') : 'ok'}.`, - `Events inspected: ${inventory.error ? 'unavailable' : events.length}.`, - !inventory.error - ? (() => { - const marker = [...events].reverse().find((event) => event.eventType === 'migration.v1.imported') - const payload = marker ? V1MigrationPayloadSchema.safeParse(marker.payload) : undefined - return payload?.success - ? `Last V1 import: revision ${payload.data.sourceRevision ?? 'unknown'}; checksum ${payload.data.sourceChecksum?.slice(0, 24) ?? 'unknown'}; identity ${payload.data.legacyRecordKey.slice(0, 64)}.` - : 'Last V1 import: unavailable.' - })() - : 'Last V1 import: unavailable.', - 'Last parity: unavailable in this command session.', - capabilities.status === 'ok' ? `Kernel capabilities: ${capabilities.capabilities.filter(({ available }) => available).map(({ name }) => name).join(', ')}.` : 'Kernel capabilities unavailable.', - health.status === 'healthy' ? 'Recovery guidance: none required.' : 'Recovery guidance: run /memory repair for a non-destructive projection preview.', - ], health.status === 'healthy' ? 'success' : 'warning') - } + return report('Memory authority', [ + 'Valid values: json-v1, shadow-v2, sqlite-v2-opt-in.', + RELEASE_N_AUTHORITY_WARNING, + `Requested: ${v2.requestedAuthority}; effective: ${v2.effectiveAuthority}.`, + 'Switch via OPENBUFF_MEMORY_AUTHORITY or the SDK authority option, then reset/restart the client.', + ]) + } + if (v2.status === 'unavailable') + return report( + 'Memory V2 unavailable', + [ + `Requested authority: ${v2.requestedAuthority}; active authority: ${v2.effectiveAuthority}.`, + RELEASE_N_AUTHORITY_WARNING, + safeOperationMessage(v2.degradation), + ], + 'warning', + ) + const scope = { + schemaVersion: 2 as const, + projectId: v2.projectId, + sessionId: MemorySessionIdSchema.parse(CLI_SESSION_ID), + } + const now = new Date().toISOString() - if (parsed.command === 'query') { - const text = parsed.args.join(' ').trim() - if (!text) return commandError(MEMORY_USAGE) - const outcome = await v2.repository.query({ ...scope, queryId: QueryIdSchema.parse(deterministicId('query', text)), query: text, selectors: [], artifactKinds: [], includeHistorical: false, maxResultsPerCategory: 10 }) - if (outcome.outcome !== 'result') return commandError(`Memory query failed: ${safeOperationMessage(outcome.error.message)} Retryable: ${outcome.error.retryable ? 'yes' : 'no'}.`) - const result = outcome.result - const lines = [ - `Tasks: ${result.matchedTasks.length}; verified: ${result.verifiedKnowledge.length}; discoveries: ${result.reusableDiscovery.length}; reread: ${result.rereadRequired.length}; historical: ${result.historicalContext.length}.`, - ...result.matchedTasks.map((item) => `Task ${item.taskId} [${item.score.toFixed(2)}]: ${item.title}.`), - ...result.verifiedKnowledge.map((item) => `Verified ${item.observation.observationId} [${item.score.toFixed(2)}]: ${item.observation.summary}. Reasons: ${item.reasons.map(({ code }) => code).join(', ')}.`), - ...result.reusableDiscovery.map((item) => `Discovery ${item.observation.observationId} [${item.score.toFixed(2)}]: ${item.observation.summary}.`), - ...result.rereadRequired.map((item) => `Reread ${item.observationId}: ${'path' in item.selector ? item.selector.path : item.selector.uri} (${item.reason}).`), - `Degradation: ${result.degradation.state === 'none' ? 'none' : result.degradation.reasons.map(({ detail }) => detail).join('; ')}.`, - ] - return report('Memory V2 query', lines, result.rereadRequired.length ? 'warning' : 'success') - } + if (parsed.command === 'audit-migration') { + return renderMigrationAudit( + await deps.auditTaskMemoryV1Migration({ + memory: inspectedMemory!.memory, + projectId: v2.projectId, + repository: v2.repository, + }), + ) + } - if (parsed.command === 'inspect') { - const inventory = await canonicalInventory(v2, parsed.args[0] ? 10_000 : 100) - if (inventory.error) return commandError(`Memory inspect failed: ${safeOperationMessage(inventory.error.message)}`) - const selected = parsed.args[0] ? inventory.events.filter(({ eventId }) => eventId === parsed.args[0]) : inventory.events - if (parsed.args[0] && selected.length === 0) return commandError('The requested canonical event was not found in the bounded inventory.') - return report('Memory V2 canonical events', selected.slice(0, 100).map((event) => `#${event.sequence} ${event.eventId} · ${event.eventType} · ${event.occurredAt}`)) - } + if (parsed.command === 'diagnose') { + const [health, kernel, capabilities, inventory] = await Promise.all([ + v2.repository.health({ schemaVersion: 2, projectId: v2.projectId }), + v2.repository.kernelHealth(), + v2.repository.getCapabilities(), + canonicalInventory(v2, 1_000), + ]) + const events = inventory.events + return report( + 'Memory V2 diagnosis', + [ + `Requested authority: ${v2.requestedAuthority}; active authority: ${v2.effectiveAuthority}.`, + RELEASE_N_AUTHORITY_WARNING, + 'V1 compatibility shadow: available when task-memory.json is hydrated and remains normally persisted.', + `Health: ${health.status}; repository authority: ${health.authority.kind}.`, + `Backend: ${health.backend.backendId}; capabilities: ${health.backend.capabilities.join(', ')}.`, + `Kernel: ${kernel.status}; schema: ${kernel.schemaVersion ?? 'unknown'}; projection cursor: ${kernel.projectionCursor ?? 'unknown'}.`, + `Integrity: ${health.issues.length ? health.issues.map(safeOperationMessage).join('; ') : 'ok'}.`, + `Events inspected: ${inventory.error ? 'unavailable' : events.length}.`, + !inventory.error + ? (() => { + const marker = [...events] + .reverse() + .find((event) => event.eventType === 'migration.v1.imported') + const payload = marker + ? V1MigrationPayloadSchema.safeParse(marker.payload) + : undefined + return payload?.success + ? `Last V1 import: revision ${payload.data.sourceRevision ?? 'unknown'}; checksum ${payload.data.sourceChecksum?.slice(0, 24) ?? 'unknown'}; identity ${payload.data.legacyRecordKey.slice(0, 64)}.` + : 'Last V1 import: unavailable.' + })() + : 'Last V1 import: unavailable.', + 'Last parity: unavailable in this command session.', + capabilities.status === 'ok' + ? `Kernel capabilities: ${capabilities.capabilities + .filter(({ available }) => available) + .map(({ name }) => name) + .join(', ')}.` + : 'Kernel capabilities unavailable.', + health.status === 'healthy' + ? 'Recovery guidance: none required.' + : 'Recovery guidance: run /memory repair for a non-destructive projection preview.', + ], + health.status === 'healthy' ? 'success' : 'warning', + ) + } - if (parsed.command === 'consolidate') { - const taskIndex = parsed.args.indexOf('--task') - const taskId = taskIndex >= 0 ? parsed.args[taskIndex + 1] : undefined - if (taskIndex >= 0 && !taskId) return commandError(MEMORY_USAGE) - const outcome = await v2.operator.consolidate({ ...scope, ...(taskId ? { taskId } : {}), policyVersion: 'cli-v1', mode: parsed.apply ? 'apply' : 'preview', occurredAt: now, maxGroups: 5 }) - const lines = operationLines(outcome, ['Canonical events are append-only; consolidation does not delete source events.']) - const command = `/memory consolidate${taskId ? ` --task ${taskId}` : ''} --apply` - return report('Memory V2 consolidation', lines, isFailureOutcome(outcome) ? 'error' : 'secondary', parsed.apply ? undefined : [{ label: 'Insert apply command', command }]) - } + if (parsed.command === 'query') { + const text = parsed.args.join(' ').trim() + if (!text) return commandError(MEMORY_USAGE) + const outcome = await v2.repository.query({ + ...scope, + queryId: QueryIdSchema.parse(deterministicId('query', text)), + query: text, + selectors: [], + artifactKinds: [], + includeHistorical: false, + maxResultsPerCategory: 10, + }) + if (outcome.outcome !== 'result') + return commandError( + `Memory query failed: ${safeOperationMessage(outcome.error.message)} Retryable: ${outcome.error.retryable ? 'yes' : 'no'}.`, + ) + const result = outcome.result + const lines = [ + `Tasks: ${result.matchedTasks.length}; verified: ${result.verifiedKnowledge.length}; discoveries: ${result.reusableDiscovery.length}; reread: ${result.rereadRequired.length}; historical: ${result.historicalContext.length}.`, + ...result.matchedTasks.map( + (item) => + `Task ${item.taskId} [${item.score.toFixed(2)}]: ${item.title}.`, + ), + ...result.verifiedKnowledge.map( + (item) => + `Verified ${item.observation.observationId} [${item.score.toFixed(2)}]: ${item.observation.summary}. Reasons: ${item.reasons.map(({ code }) => code).join(', ')}.`, + ), + ...result.reusableDiscovery.map( + (item) => + `Discovery ${item.observation.observationId} [${item.score.toFixed(2)}]: ${item.observation.summary}.`, + ), + ...result.rereadRequired.map( + (item) => + `Reread ${item.observationId}: ${'path' in item.selector ? item.selector.path : item.selector.uri} (${item.reason}).`, + ), + `Degradation: ${result.degradation.state === 'none' ? 'none' : result.degradation.reasons.map(({ detail }) => detail).join('; ')}.`, + ] + return report( + 'Memory V2 query', + lines, + result.rereadRequired.length ? 'warning' : 'success', + ) + } - if (parsed.command === 'repair') { - if (parsed.args.includes('--reset-corrupt')) return commandError('Destructive reset-corrupt is not supported in this release.') - const outcome = await v2.operator.repair({ ...scope, mode: parsed.apply ? 'apply' : 'preview', rebuildId: deterministicId('repair', String(v2.projectId)), projectionNames: ['tasks', 'sessions', 'artifacts', 'claims', 'evidence', 'discoveries'] }) - return report('Memory V2 projection repair', operationLines(outcome, ['Canonical events are never deleted or reset.']), isFailureOutcome(outcome) ? 'error' : 'secondary', parsed.apply ? undefined : [{ label: 'Insert apply command', command: '/memory repair --apply' }]) - } + if (parsed.command === 'inspect') { + const inventory = await canonicalInventory( + v2, + parsed.args[0] ? 10_000 : 100, + ) + if (inventory.error) + return commandError( + `Memory inspect failed: ${safeOperationMessage(inventory.error.message)}`, + ) + const selected = parsed.args[0] + ? inventory.events.filter(({ eventId }) => eventId === parsed.args[0]) + : inventory.events + if (parsed.args[0] && selected.length === 0) + return commandError( + 'The requested canonical event was not found in the bounded inventory.', + ) + return report( + 'Memory V2 canonical events', + selected + .slice(0, 100) + .map( + (event) => + `#${event.sequence} ${event.eventId} · ${event.eventType} · ${event.occurredAt}`, + ), + ) + } - if (parsed.command === 'revalidate') { - if (parsed.args.length !== 2) return commandError(MEMORY_USAGE) - const observationId = parsed.args[0]! - const requestedPath = parsed.args[1]! - const pathParts = requestedPath.split('/') - if (!requestedPath || requestedPath.startsWith('/') || requestedPath.includes('\\') || pathParts.some((part) => !part || part === '.' || part === '..')) { - return commandError('Revalidation requires a contained project-relative path.') + if (parsed.command === 'consolidate') { + const taskIndex = parsed.args.indexOf('--task') + const taskId = taskIndex >= 0 ? parsed.args[taskIndex + 1] : undefined + if (taskIndex >= 0 && !taskId) return commandError(MEMORY_USAGE) + const outcome = await v2.operator.consolidate({ + ...scope, + ...(taskId ? { taskId } : {}), + policyVersion: 'cli-v1', + mode: parsed.apply ? 'apply' : 'preview', + occurredAt: now, + maxGroups: 5, + }) + const lines = operationLines(outcome, [ + 'Canonical events are append-only; consolidation does not delete source events.', + ]) + const command = `/memory consolidate${taskId ? ` --task ${taskId}` : ''} --apply` + return report( + 'Memory V2 consolidation', + lines, + isFailureOutcome(outcome) ? 'error' : 'secondary', + parsed.apply ? undefined : [{ label: 'Insert apply command', command }], + ) } - const taskId = await lookupObservationTaskId(v2, [observationId]) - if (!taskId) return commandError('The observation task could not be resolved.') - const action = parsed.apply - ? { kind: 'verify' as const, observationId, selector: { kind: 'file' as const, path: requestedPath }, observedDigest: readContainedProjectFile(deps.getRootDir(), requestedPath, EXPORT_MAX_BYTES).digest } - : { kind: 'verify' as const, observationId, selector: { kind: 'file' as const, path: requestedPath } } - const outcome = await v2.operator.revalidate({ ...scope, taskId, mode: parsed.apply ? 'apply' : 'preview', actions: [action] }) - return report('Memory V2 revalidation', operationLines(outcome, [parsed.apply ? 'Live file verification completed.' : 'Selector validated; live file verification is pending apply.']), isFailureOutcome(outcome) ? 'error' : 'secondary', parsed.apply ? undefined : [{ label: 'Insert apply command', command: `/memory revalidate ${parsed.args[0]} ${parsed.args[1]} --apply` }]) - } - if (['correct', 'forget', 'pin'].includes(parsed.command)) { - const observationId = parsed.args[0] - if (!observationId) return commandError(MEMORY_USAGE) - let action: unknown - if (parsed.command === 'forget') { - action = { kind: 'forget', observationIds: parsed.args, reason: 'user-request', requestedBy: 'cli-user', evidenceDisposition: 'remove-references' } - } else if (parsed.command === 'pin') { - if (parsed.args.length !== 1) return commandError(MEMORY_USAGE) - action = { kind: 'pin', observationId, reason: 'Pinned by user from the CLI', pinnedBy: 'cli-user' } - } else { - const replacement = parsed.args.slice(1).join(' ').trim() - if (!replacement) return commandError(MEMORY_USAGE) - const inventory = await canonicalInventory(v2, 10_000) - if (inventory.error) return commandError(`Correction lookup failed: ${safeOperationMessage(inventory.error.message)}`) - const source = inventory.events.find((event) => event.eventType === 'observation.recorded' && event.payload.observation.observationId === observationId) - if (!source || source.eventType !== 'observation.recorded') return commandError('The source observation could not be found; correction was not created.') - const original = source.payload.observation - action = { kind: 'correct', observationId, reason: 'User correction from the CLI', correction: { ...original, observationId: deterministicId('observation', `${observationId}:${replacement}`), summary: replacement.slice(0, 1_024), detail: replacement.slice(0, 16_384), evidence: original.evidence.map(({ excerpt: _excerpt, ...evidence }) => evidence), observedAt: now } } + if (parsed.command === 'repair') { + if (parsed.args.includes('--reset-corrupt')) + return commandError( + 'Destructive reset-corrupt is not supported in this release.', + ) + const outcome = await v2.operator.repair({ + ...scope, + mode: parsed.apply ? 'apply' : 'preview', + rebuildId: deterministicId('repair', String(v2.projectId)), + projectionNames: [ + 'tasks', + 'sessions', + 'artifacts', + 'claims', + 'evidence', + 'discoveries', + ], + }) + return report( + 'Memory V2 projection repair', + operationLines(outcome, [ + 'Canonical events are never deleted or reset.', + ]), + isFailureOutcome(outcome) ? 'error' : 'secondary', + parsed.apply + ? undefined + : [ + { + label: 'Insert apply command', + command: '/memory repair --apply', + }, + ], + ) } - const targetObservationIds = parsed.command === 'forget' ? parsed.args : [observationId] - const taskId = parsed.command === 'correct' && typeof action === 'object' && action !== null && 'correction' in action - ? (action as { correction: { taskId: string } }).correction.taskId - : await lookupObservationTaskId(v2, targetObservationIds) - if (!taskId) return commandError('The observation task could not be resolved.') - const outcome = await v2.operator.correct({ ...scope, taskId, mode: parsed.apply ? 'apply' : 'preview', occurredAt: now, action }) - const applyCommand = `/memory ${parsed.command} ${parsed.args.join(' ')} --apply` - return report(`Memory V2 ${parsed.command}`, operationLines(outcome, ['Preview is the default; apply appends canonical lifecycle events.']), isFailureOutcome(outcome) ? 'error' : 'secondary', parsed.apply ? undefined : [{ label: 'Insert apply command', command: applyCommand }]) - } - if (parsed.command === 'export') return runExport(v2, parsed.args, deps.getRootDir()) - if (parsed.command === 'import') return runImport(v2, parsed.args, parsed.apply, deps.getRootDir()) - return commandError(MEMORY_USAGE) + if (parsed.command === 'revalidate') { + if (parsed.args.length !== 2) return commandError(MEMORY_USAGE) + const observationId = parsed.args[0]! + const requestedPath = parsed.args[1]! + const pathParts = requestedPath.split('/') + if ( + !requestedPath || + requestedPath.startsWith('/') || + requestedPath.includes('\\') || + pathParts.some((part) => !part || part === '.' || part === '..') + ) { + return commandError( + 'Revalidation requires a contained project-relative path.', + ) + } + const taskId = await lookupObservationTaskId(v2, [observationId]) + if (!taskId) + return commandError('The observation task could not be resolved.') + const action = parsed.apply + ? { + kind: 'verify' as const, + observationId, + selector: { kind: 'file' as const, path: requestedPath }, + observedDigest: readContainedProjectFile( + deps.getRootDir(), + requestedPath, + EXPORT_MAX_BYTES, + ).digest, + } + : { + kind: 'verify' as const, + observationId, + selector: { kind: 'file' as const, path: requestedPath }, + } + const outcome = await v2.operator.revalidate({ + ...scope, + taskId, + mode: parsed.apply ? 'apply' : 'preview', + actions: [action], + }) + return report( + 'Memory V2 revalidation', + operationLines(outcome, [ + parsed.apply + ? 'Live file verification completed.' + : 'Selector validated; live file verification is pending apply.', + ]), + isFailureOutcome(outcome) ? 'error' : 'secondary', + parsed.apply + ? undefined + : [ + { + label: 'Insert apply command', + command: `/memory revalidate ${parsed.args[0]} ${parsed.args[1]} --apply`, + }, + ], + ) + } + + if (['correct', 'forget', 'pin'].includes(parsed.command)) { + const observationId = parsed.args[0] + if (!observationId) return commandError(MEMORY_USAGE) + let action: unknown + if (parsed.command === 'forget') { + action = { + kind: 'forget', + observationIds: parsed.args, + reason: 'user-request', + requestedBy: 'cli-user', + evidenceDisposition: 'remove-references', + } + } else if (parsed.command === 'pin') { + if (parsed.args.length !== 1) return commandError(MEMORY_USAGE) + action = { + kind: 'pin', + observationId, + reason: 'Pinned by user from the CLI', + pinnedBy: 'cli-user', + } + } else { + const replacement = parsed.args.slice(1).join(' ').trim() + if (!replacement) return commandError(MEMORY_USAGE) + const inventory = await canonicalInventory(v2, 10_000) + if (inventory.error) + return commandError( + `Correction lookup failed: ${safeOperationMessage(inventory.error.message)}`, + ) + const source = inventory.events.find( + (event) => + event.eventType === 'observation.recorded' && + event.payload.observation.observationId === observationId, + ) + if (!source || source.eventType !== 'observation.recorded') + return commandError( + 'The source observation could not be found; correction was not created.', + ) + const original = source.payload.observation + action = { + kind: 'correct', + observationId, + reason: 'User correction from the CLI', + correction: { + ...original, + observationId: deterministicId( + 'observation', + `${observationId}:${replacement}`, + ), + summary: replacement.slice(0, 1_024), + detail: replacement.slice(0, 16_384), + evidence: original.evidence.map( + ({ excerpt: _excerpt, ...evidence }) => evidence, + ), + observedAt: now, + }, + } + } + const targetObservationIds = + parsed.command === 'forget' ? parsed.args : [observationId] + const taskId = + parsed.command === 'correct' && + typeof action === 'object' && + action !== null && + 'correction' in action + ? (action as { correction: { taskId: string } }).correction.taskId + : await lookupObservationTaskId(v2, targetObservationIds) + if (!taskId) + return commandError('The observation task could not be resolved.') + const outcome = await v2.operator.correct({ + ...scope, + taskId, + mode: parsed.apply ? 'apply' : 'preview', + occurredAt: now, + action, + }) + const applyCommand = `/memory ${parsed.command} ${parsed.args.join(' ')} --apply` + return report( + `Memory V2 ${parsed.command}`, + operationLines(outcome, [ + 'Preview is the default; apply appends canonical lifecycle events.', + ]), + isFailureOutcome(outcome) ? 'error' : 'secondary', + parsed.apply + ? undefined + : [{ label: 'Insert apply command', command: applyCommand }], + ) + } + + if (parsed.command === 'export') + return runExport(v2, parsed.args, deps.getRootDir()) + if (parsed.command === 'import') + return runImport(v2, parsed.args, parsed.apply, deps.getRootDir()) + return commandError(MEMORY_USAGE) } finally { if (v2.status === 'available') await v2.release?.() } @@ -415,13 +891,22 @@ async function lookupObservationTaskId( const wanted = new Set(observationIds) const taskIds = new Set() for (const event of inventory.events) { - if (event.eventType === 'observation.recorded' && wanted.has(event.payload.observation.observationId)) { + if ( + event.eventType === 'observation.recorded' && + wanted.has(event.payload.observation.observationId) + ) { taskIds.add(event.payload.observation.taskId) wanted.delete(event.payload.observation.observationId) - } else if (event.eventType === 'claim.consolidated' && wanted.has(event.payload.canonicalObservation.observationId)) { + } else if ( + event.eventType === 'claim.consolidated' && + wanted.has(event.payload.canonicalObservation.observationId) + ) { taskIds.add(event.payload.canonicalObservation.taskId) wanted.delete(event.payload.canonicalObservation.observationId) - } else if (event.eventType === 'claim.corrected' && wanted.has(event.payload.correction.observationId)) { + } else if ( + event.eventType === 'claim.corrected' && + wanted.has(event.payload.correction.observationId) + ) { taskIds.add(event.payload.correction.taskId) wanted.delete(event.payload.correction.observationId) } @@ -436,50 +921,103 @@ async function runExport( ): Promise { const formatIndex = args.indexOf('--format') const format = formatIndex >= 0 ? args[formatIndex + 1] : 'json' - if (format !== 'json' && format !== 'markdown') return commandError('Export format must be json or markdown.') - const outcome = await v2.operator.exportManifest({ schemaVersion: 2, projectId: v2.projectId, generatedAt: new Date().toISOString(), includeStale: args.includes('--include-stale'), rendering: format === 'markdown' ? 'json-and-markdown' : 'json' }) - if (outcome.outcome !== 'exported') return commandError(`Memory export failed: ${safeOperationMessage(outcome.error.message)} Retryable: ${outcome.error.retryable ? 'yes' : 'no'}.`) + if (format !== 'json' && format !== 'markdown') + return commandError('Export format must be json or markdown.') + const outcome = await v2.operator.exportManifest({ + schemaVersion: 2, + projectId: v2.projectId, + generatedAt: new Date().toISOString(), + includeStale: args.includes('--include-stale'), + rendering: format === 'markdown' ? 'json-and-markdown' : 'json', + }) + if (outcome.outcome !== 'exported') + return commandError( + `Memory export failed: ${safeOperationMessage(outcome.error.message)} Retryable: ${outcome.error.retryable ? 'yes' : 'no'}.`, + ) const stem = `memory-v2-${outcome.manifest.checksum.slice(7, 23)}` const jsonName = `${stem}.json` const json = `${JSON.stringify(outcome.manifest, null, 2)}\n` - if (Buffer.byteLength(json) > EXPORT_MAX_BYTES) return commandError('Memory export exceeds the bounded file size.') - const directory = createContainedProjectDirectory(root, '.openbuff/memory/exports') + if (Buffer.byteLength(json) > EXPORT_MAX_BYTES) + return commandError('Memory export exceeds the bounded file size.') + const directory = createContainedProjectDirectory( + root, + '.openbuff/memory/exports', + ) let jsonCreated = false try { try { directory.writeExclusive(jsonName, json) jsonCreated = true } catch (error) { - if (!(error instanceof ContainedFileIoError) || error.code !== 'exists') throw error - const existing = readContainedProjectFile(root, `.openbuff/memory/exports/${jsonName}`, EXPORT_MAX_BYTES).text + if (!(error instanceof ContainedFileIoError) || error.code !== 'exists') + throw error + const existing = readContainedProjectFile( + root, + `.openbuff/memory/exports/${jsonName}`, + EXPORT_MAX_BYTES, + ).text if (existing !== json) throw error } const paths = [`.openbuff/memory/exports/${jsonName}`] if (format === 'markdown') { - if (!outcome.markdown || Buffer.byteLength(outcome.markdown) > EXPORT_MAX_BYTES) throw new ContainedFileIoError('too-large') + if ( + !outcome.markdown || + Buffer.byteLength(outcome.markdown) > EXPORT_MAX_BYTES + ) + throw new ContainedFileIoError('too-large') const markdownName = `${stem}.md` try { directory.writeExclusive(markdownName, outcome.markdown) } catch (error) { - if (!(error instanceof ContainedFileIoError) || error.code !== 'exists') throw error - const existingMarkdown = readContainedProjectFile(root, `.openbuff/memory/exports/${markdownName}`, EXPORT_MAX_BYTES).text + if (!(error instanceof ContainedFileIoError) || error.code !== 'exists') + throw error + const existingMarkdown = readContainedProjectFile( + root, + `.openbuff/memory/exports/${markdownName}`, + EXPORT_MAX_BYTES, + ).text if (existingMarkdown !== outcome.markdown) throw error } paths.push(`.openbuff/memory/exports/${markdownName}`) } - return report('Memory V2 export', [`Created: ${paths.join(', ')}.`, `Checksum: ${outcome.manifest.checksum}.`, `Canonical events: ${outcome.manifest.canonicalEventCount}; exported: ${outcome.manifest.events.length}.`, `Warnings: ${outcome.manifest.warnings.length ? outcome.manifest.warnings.slice(0, 20).map(safeOperationMessage).join('; ') : 'none'}.`], 'success') + return report( + 'Memory V2 export', + [ + `Created: ${paths.join(', ')}.`, + `Checksum: ${outcome.manifest.checksum}.`, + `Canonical events: ${outcome.manifest.canonicalEventCount}; exported: ${outcome.manifest.events.length}.`, + `Warnings: ${outcome.manifest.warnings.length ? outcome.manifest.warnings.slice(0, 20).map(safeOperationMessage).join('; ') : 'none'}.`, + ], + 'success', + ) } catch { if (jsonCreated) directory.remove(jsonName) - return commandError('Memory export failed: local output could not be created safely.') + return commandError( + 'Memory export failed: local output could not be created safely.', + ) } finally { directory.close() } } function stableManifestJson(value: unknown): string { - if (value === null || typeof value === 'boolean' || typeof value === 'number' || typeof value === 'string') return JSON.stringify(value) - if (Array.isArray(value)) return `[${value.map(stableManifestJson).join(',')}]` - if (typeof value === 'object') return `{${Object.keys(value as Record).sort().map((key) => `${JSON.stringify(key)}:${stableManifestJson((value as Record)[key])}`).join(',')}}` + if ( + value === null || + typeof value === 'boolean' || + typeof value === 'number' || + typeof value === 'string' + ) + return JSON.stringify(value) + if (Array.isArray(value)) + return `[${value.map(stableManifestJson).join(',')}]` + if (typeof value === 'object') + return `{${Object.keys(value as Record) + .sort() + .map( + (key) => + `${JSON.stringify(key)}:${stableManifestJson((value as Record)[key])}`, + ) + .join(',')}}` return JSON.stringify(null) } @@ -490,22 +1028,72 @@ async function runImport( root: string, ): Promise { if (args.length !== 1) return commandError(MEMORY_USAGE) - if (!args[0]!.toLowerCase().endsWith('.json')) return commandError('Memory import rejected: Import requires a project-relative JSON path.') + if (!args[0]!.toLowerCase().endsWith('.json')) + return commandError( + 'Memory import rejected: Import requires a project-relative JSON path.', + ) try { - const manifest = MemoryExportManifestV2Schema.parse(JSON.parse(readContainedProjectFile(root, args[0]!, EXPORT_MAX_BYTES).text)) - if (manifest.projectId !== v2.projectId) return commandError('Memory import rejected: Manifest project does not match the current project.') + const manifest = MemoryExportManifestV2Schema.parse( + JSON.parse( + readContainedProjectFile(root, args[0]!, EXPORT_MAX_BYTES).text, + ), + ) + if (manifest.projectId !== v2.projectId) + return commandError( + 'Memory import rejected: Manifest project does not match the current project.', + ) const { checksum: _checksum, ...checksumManifest } = manifest const checksum = `sha256:${createHash('sha256').update(stableManifestJson(checksumManifest)).digest('hex')}` - if (checksum !== manifest.checksum) return commandError('Memory import rejected: Manifest checksum is invalid.') - if (!apply) return report('Memory V2 import preview', [`Validated ${args[0]}.`, `Events: ${manifest.events.length}; checksum: ${manifest.checksum}.`, 'No writes were performed.'], 'secondary', [{ label: 'Insert apply command', command: `/memory import ${args[0]} --apply` }]) - const outcome = await v2.operator.importManifest({ schemaVersion: 2, projectId: v2.projectId, manifest, rebuildId: deterministicId('import', manifest.checksum), projectionNames: ['tasks', 'sessions', 'artifacts', 'claims', 'evidence', 'discoveries'], pageSize: 100 }) - return report('Memory V2 import', operationLines(outcome, [`Source: ${args[0]}.`]), outcome.outcome === 'imported' || outcome.outcome === 'no-op' ? 'success' : 'error') + if (checksum !== manifest.checksum) + return commandError( + 'Memory import rejected: Manifest checksum is invalid.', + ) + if (!apply) + return report( + 'Memory V2 import preview', + [ + `Validated ${args[0]}.`, + `Events: ${manifest.events.length}; checksum: ${manifest.checksum}.`, + 'No writes were performed.', + ], + 'secondary', + [ + { + label: 'Insert apply command', + command: `/memory import ${args[0]} --apply`, + }, + ], + ) + const outcome = await v2.operator.importManifest({ + schemaVersion: 2, + projectId: v2.projectId, + manifest, + rebuildId: deterministicId('import', manifest.checksum), + projectionNames: [ + 'tasks', + 'sessions', + 'artifacts', + 'claims', + 'evidence', + 'discoveries', + ], + pageSize: 100, + }) + return report( + 'Memory V2 import', + operationLines(outcome, [`Source: ${args[0]}.`]), + outcome.outcome === 'imported' || outcome.outcome === 'no-op' + ? 'success' + : 'error', + ) } catch (error) { - const message = error instanceof ContainedFileIoError && error.code === 'invalid-path' - ? 'Import requires a project-relative JSON path.' - : error instanceof ContainedFileIoError && ['missing', 'not-regular', 'too-large'].includes(error.code) - ? 'Import file is missing, not regular, or too large.' - : 'The input could not be validated safely.' + const message = + error instanceof ContainedFileIoError && error.code === 'invalid-path' + ? 'Import requires a project-relative JSON path.' + : error instanceof ContainedFileIoError && + ['missing', 'not-regular', 'too-large'].includes(error.code) + ? 'Import file is missing, not regular, or too large.' + : 'The input could not be validated safely.' return commandError(`Memory import rejected: ${message}`) } } @@ -513,39 +1101,67 @@ async function runImport( async function canonicalInventory( v2: Extract>, { status: 'available' }>, maximum: number, -): Promise<{ events: import('@openbuff/sdk').MemoryEventEnvelope[]; error?: { message: string } }> { +): Promise<{ + events: import('@openbuff/sdk').MemoryEventEnvelope[] + error?: { message: string } +}> { const events: import('@openbuff/sdk').MemoryEventEnvelope[] = [] const cursors = new Set() let afterEventId: import('@openbuff/sdk').MemoryEventId | undefined const pageBound = Math.min(100, Math.ceil(maximum / 1_000) + 1) - for (let pageIndex = 0; pageIndex < pageBound && events.length < maximum; pageIndex++) { + for ( + let pageIndex = 0; + pageIndex < pageBound && events.length < maximum; + pageIndex++ + ) { const page = await v2.repository.export({ schemaVersion: 2, projectId: v2.projectId, ...(afterEventId ? { afterEventId } : {}), limit: Math.min(1_000, maximum - events.length), }) - if (page.outcome !== 'page') return { events: [], error: { message: 'Canonical inventory is temporarily unavailable.' } } + if (page.outcome !== 'page') + return { + events: [], + error: { message: 'Canonical inventory is temporarily unavailable.' }, + } const parsedEvents: import('@openbuff/sdk').MemoryEventEnvelope[] = [] for (const candidate of page.events) { const parsed = MemoryEventEnvelopeSchema.safeParse(candidate) - if (!parsed.success || parsed.data.projectId !== v2.projectId) return { events: [], error: { message: 'Canonical inventory failed validation.' } } + if (!parsed.success || parsed.data.projectId !== v2.projectId) + return { + events: [], + error: { message: 'Canonical inventory failed validation.' }, + } parsedEvents.push(parsed.data) } - if (page.nextAfterEventId && parsedEvents.length === 0) return { events: [], error: { message: 'Canonical inventory pagination did not advance.' } } + if (page.nextAfterEventId && parsedEvents.length === 0) + return { + events: [], + error: { message: 'Canonical inventory pagination did not advance.' }, + } if (page.nextAfterEventId) { if ( page.nextAfterEventId === afterEventId || cursors.has(page.nextAfterEventId) || page.nextAfterEventId !== parsedEvents.at(-1)?.eventId - ) return { events: [], error: { message: 'Canonical inventory pagination did not advance.' } } + ) + return { + events: [], + error: { message: 'Canonical inventory pagination did not advance.' }, + } cursors.add(page.nextAfterEventId) } events.push(...parsedEvents) if (!page.nextAfterEventId) return { events } afterEventId = page.nextAfterEventId } - return events.length >= maximum ? { events } : { events: [], error: { message: 'Canonical inventory exceeded its page bound.' } } + return events.length >= maximum + ? { events } + : { + events: [], + error: { message: 'Canonical inventory exceeded its page bound.' }, + } } export async function handleMemoryCommandBlocks( @@ -558,7 +1174,9 @@ export async function handleMemoryCommandBlocks( if (parsed.command === 'status') return await runStatusBlock(deps) return await runV2Command(rawArgs, deps) } catch { - return commandError('The memory operation could not be completed safely. Please retry.') + return commandError( + 'The memory operation could not be completed safely. Please retry.', + ) } } @@ -578,41 +1196,46 @@ async function runStatusBlock( if (deps.getMemoryV2) { const v2 = await getV2(deps) try { - if (v2.status === 'available') { - const [health, kernel, inventory] = await Promise.all([ - v2.repository.health({ schemaVersion: 2, projectId: v2.projectId }), - v2.repository.kernelHealth(), - canonicalInventory(v2, 1_000), - ]) - const migration = !inventory.error - ? [...inventory.events].reverse().find((event) => event.eventType === 'migration.v1.imported') - : undefined - const migrationPayload = migration - ? V1MigrationPayloadSchema.safeParse(migration.payload) - : undefined - v2Lines = [ - `Memory authority: requested ${v2.requestedAuthority}; active ${v2.effectiveAuthority}.`, - `V1 compatibility shadow: ${context ? 'available' : 'not yet available'}.`, - `Memory V2: ${health.status}; repository authority ${health.authority.kind}; backend ${health.backend.backendId}.`, - migrationPayload?.success - ? `Last V1 import: revision ${migrationPayload.data.sourceRevision ?? 'unknown'}; checksum ${migrationPayload.data.sourceChecksum?.slice(0, 24) ?? 'unknown'}; identity ${migrationPayload.data.legacyRecordKey.slice(0, 64)}.` - : 'Last V1 import: unavailable.', - 'Last parity: unavailable in this command session.', - `Schema ${kernel.schemaVersion ?? 'unknown'}; capabilities ${health.backend.capabilities.join(', ')}; events ${inventory.error ? 'unavailable' : inventory.events.length}; projection ${kernel.projectionCursor ?? 'unknown'}.`, - `Degradation: ${health.issues.length ? health.issues.map(safeOperationMessage).join('; ') : 'none'}.`, - ] - } else { - v2Lines = [ - `Memory authority: requested ${v2.requestedAuthority}; active ${v2.effectiveAuthority}.`, - `V1 compatibility shadow: ${context ? 'available' : 'not yet available'}.`, - `Memory V2: ${safeOperationMessage(v2.degradation)}`, - ] - } + if (v2.status === 'available') { + const [health, kernel, inventory] = await Promise.all([ + v2.repository.health({ schemaVersion: 2, projectId: v2.projectId }), + v2.repository.kernelHealth(), + canonicalInventory(v2, 1_000), + ]) + const migration = !inventory.error + ? [...inventory.events] + .reverse() + .find((event) => event.eventType === 'migration.v1.imported') + : undefined + const migrationPayload = migration + ? V1MigrationPayloadSchema.safeParse(migration.payload) + : undefined + v2Lines = [ + `Memory authority: requested ${v2.requestedAuthority}; active ${v2.effectiveAuthority}.`, + RELEASE_N_AUTHORITY_WARNING, + `V1 compatibility shadow: ${context ? 'available' : 'not yet available'}.`, + `Memory V2: ${health.status}; repository authority ${health.authority.kind}; backend ${health.backend.backendId}.`, + migrationPayload?.success + ? `Last V1 import: revision ${migrationPayload.data.sourceRevision ?? 'unknown'}; checksum ${migrationPayload.data.sourceChecksum?.slice(0, 24) ?? 'unknown'}; identity ${migrationPayload.data.legacyRecordKey.slice(0, 64)}.` + : 'Last V1 import: unavailable.', + 'Last parity: unavailable in this command session.', + `Schema ${kernel.schemaVersion ?? 'unknown'}; capabilities ${health.backend.capabilities.join(', ')}; events ${inventory.error ? 'unavailable' : inventory.events.length}; projection ${kernel.projectionCursor ?? 'unknown'}.`, + `Degradation: ${health.issues.length ? health.issues.map(safeOperationMessage).join('; ') : 'none'}.`, + ] + } else { + v2Lines = [ + `Memory authority: requested ${v2.requestedAuthority}; active ${v2.effectiveAuthority}.`, + RELEASE_N_AUTHORITY_WARNING, + `V1 compatibility shadow: ${context ? 'available' : 'not yet available'}.`, + `Memory V2: ${safeOperationMessage(v2.degradation)}`, + ] + } } finally { if (v2.status === 'available') await v2.release?.() } } - if (!context) return { type: 'memory', state: 'empty', ...(v2Lines ? { v2Lines } : {}) } + if (!context) + return { type: 'memory', state: 'empty', ...(v2Lines ? { v2Lines } : {}) } return { type: 'memory', state: 'status', diff --git a/cli/src/data/__tests__/slash-commands.test.ts b/cli/src/data/__tests__/slash-commands.test.ts index 47a7cc5062..cf8ae589ac 100644 --- a/cli/src/data/__tests__/slash-commands.test.ts +++ b/cli/src/data/__tests__/slash-commands.test.ts @@ -40,7 +40,7 @@ describe('slash-commands module', () => { expect(command).toBeDefined() expect(command!.label).toBe('memory') expect(command!.description).toBe( - 'Inspect, query, diagnose, export, or maintain project memory', + 'Inspect, query, audit migrations, export, or maintain project memory', ) expect(command!.aliases).toEqual(['mem']) // Stateful command: must not fire without an explicit leading slash. diff --git a/cli/src/data/slash-commands.ts b/cli/src/data/slash-commands.ts index 0de1de0973..02876d5390 100644 --- a/cli/src/data/slash-commands.ts +++ b/cli/src/data/slash-commands.ts @@ -59,7 +59,7 @@ const ALL_SLASH_COMMANDS: SlashCommand[] = [ { id: 'memory', label: 'memory', - description: 'Inspect, query, diagnose, export, or maintain project memory', + description: 'Inspect, query, audit migrations, export, or maintain project memory', aliases: ['mem'], }, { diff --git a/cli/src/services/memory-v2/__tests__/memory-v1-migration-sqlite-roundtrip.test.ts b/cli/src/services/memory-v2/__tests__/memory-v1-migration-sqlite-roundtrip.test.ts new file mode 100644 index 0000000000..8c4bac6857 --- /dev/null +++ b/cli/src/services/memory-v2/__tests__/memory-v1-migration-sqlite-roundtrip.test.ts @@ -0,0 +1,151 @@ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import path from 'node:path' + +import { afterEach, describe, expect, test } from 'bun:test' + +import { + MemorySessionIdSchema, + ProjectIdSchema, +} from '@codebuff/common/types/memory-v2' + +import { + auditTaskMemoryV1Migration, + importTaskMemoryV1, +} from '../../../../../sdk/src/services/memory-v2/v1-migration' +import { + loadPersistedTaskMemory, + saveMergedTaskMemory, +} from '../../../../../sdk/src/services/task-memory-store' + +import type { TaskMemoryV1 } from '@codebuff/common/types/task-memory' + +// Package placement: this test lives in the cli package, beside the existing +// real-backend test (bun-sqlite-memory-repository.test.ts in this directory), +// so the Bun SQLite backend is imported through a normal package-local +// relative path — a reviewer advisory flagged the former sdk-side location's +// cross-package relative import of this cli internal as fragile, since the sdk +// suite would break if the cli backend file moved. The backend needs nothing +// beyond the plain `bun test` runner: `bun:sqlite` is built into the Bun +// runtime and the sibling backend test runs with no special preload or flags. +// The migration (`importTaskMemoryV1`/`auditTaskMemoryV1Migration`) and V1 +// store (`saveMergedTaskMemory`/`loadPersistedTaskMemory`) functions are the +// exact modules re-exported from the `@openbuff/sdk` package root +// (sdk/src/index.ts); they are imported from SDK source here because the +// `@openbuff/sdk` specifier resolves to the built dist/, which plain +// `bun test` does not build first. +import { openBunSQLiteMemoryRepository } from '../bun-sqlite-memory-repository' + +const projectId = ProjectIdSchema.parse('project:sqlite-roundtrip') +const sessionId = MemorySessionIdSchema.parse('memory-cli') + +// Non-empty categories whose marker sourceItemCounts keys are NOT in +// alphabetical order (`decisions` < `path-evidence` < `requirements`). The +// Bun SQLite repository persists event payloads via key-sorted stableJson and +// export() re-parses them, so the re-read marker arrives with sorted record +// keys while the in-memory draft keeps insertion order. Without the +// canonicalizeForCompare fix in equalEventDraft, the audit below mismatches +// on exactly this key-order round-trip and reports imported-body-mismatch. +const runMemory: TaskMemoryV1 = { + schemaVersion: 1, + goal: 'Excluded from the import; retained only in the persisted V1 record.', + requirements: ['req-1'], + decisions: ['dec-1'], + filesInspected: [], + editsMade: [], + validationResults: [], + reviewReceipts: [], + blockers: [], + nextActions: [], + historicalSummary: '', + evidence: [ + { + id: 'ev-1', + kind: 'read', + summary: 'round-trip path evidence', + path: 'src/roundtrip.ts', + stale: false, + }, + ], + revision: 0, + updatedAt: 0, + checksum: 'seeded-by-save', +} + +describe('V1→V2 migration against the real Bun SQLite backend', () => { + let tempRoot: string | undefined + + afterEach(async () => { + if (tempRoot !== undefined) { + rmSync(tempRoot, { recursive: true, force: true }) + tempRoot = undefined + } + }) + + test('import + audit round-trip the marker sourceItemCounts through real SQLite storage', async () => { + tempRoot = mkdtempSync(path.join(tmpdir(), 'openbuff-memory-v2-roundtrip-')) + const rootDir = tempRoot + + // Seed a valid V1 record through the real store and load it back, so the + // persisted checksum/revision round-trip is exercised too. + const saved = await saveMergedTaskMemory({ rootDir, runMemory }) + expect(saved).toBeDefined() + const memory = await loadPersistedTaskMemory({ rootDir }) + expect(memory).toBeDefined() + if (!memory) throw new Error('expected persisted V1 task memory') + expect(memory.requirements).toEqual(['req-1']) + expect(memory.decisions).toEqual(['dec-1']) + expect(memory.evidence).toHaveLength(1) + + // Open the REAL backend: creates /.openbuff/memory/memory-v2.sqlite. + const opened = await openBunSQLiteMemoryRepository({ + repositoryRoot: rootDir, + }) + if (opened.status !== 'ok') { + throw new Error( + `expected the real backend to open, got ${opened.error.kind}: ${opened.error.message}`, + ) + } + const repository = opened.repository + try { + expect(repository.databasePath).toBe( + path.join(rootDir, '.openbuff', 'memory', 'memory-v2.sqlite'), + ) + + const imported = await importTaskMemoryV1({ + memory, + projectId, + sessionId, + repository, + }) + expect(imported.outcome).toBe('imported') + if (imported.outcome !== 'imported') return + expect(imported.sourceItemCounts).toEqual({ + blockers: 0, + decisions: 1, + 'edits-made': 0, + 'files-inspected': 0, + 'historical-summary': 0, + 'next-actions': 0, + 'path-evidence': 1, + requirements: 1, + 'review-receipts': 0, + 'validation-results': 0, + }) + + const audit = await auditTaskMemoryV1Migration({ + memory, + projectId, + repository, + }) + expect(audit.outcome).toBe('exact') + if (audit.outcome !== 'exact') return + // The key assertion: the audit re-derived its counts from the + // SQLite-reparsed marker envelope (sorted record keys) and still + // certifies an exact match against the in-memory import outcome. + expect(audit.sourceItemCounts).toEqual(imported.sourceItemCounts) + } finally { + await repository.close() + } + }) +}) diff --git a/common/knowledge.md b/common/knowledge.md index c04ea59fe6..2945cb09a4 100644 --- a/common/knowledge.md +++ b/common/knowledge.md @@ -26,11 +26,11 @@ This package contains code shared across the Openbuff monorepo, especially the l - **Provider config**: Shared validation and contracts for OpenAI-compatible, Anthropic-compatible, and other BYOK providers - **Workspace state**: `common/src/types/workspace-state.ts` tracks monotonically increasing local workspace revisions and bounded change records so SDK/runtime tools can tie mutation receipts, review bundles, and snapshot-aware validation to a stable `workspace.v1..` identifier. - **Task memory**: `common/src/types/task-memory.ts` defines bounded Zod schemas for durable requirements, evidence, blockers, validation, and review notes that survive long task execution without leaking unbounded context. `TASK_MEMORY_LIST_CAPS` is exported alongside the schema as the single source of truth for per-list array caps, so consumers (notably the SDK task-memory store that merges a finished run's memory into the persisted record) cannot hand-mirror limits and drift from what the schema enforces. -- **Published task-memory API (`@openbuff/sdk`)**: the SDK publishes `loadPersistedTaskMemory`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, and `codebuffFsToNodePromises`, together with the complete type closure their signatures name — `TaskMemoryV1`, `TaskMemoryDraftV1`, `TaskMemoryEvidenceV1` (re-exported from `common/src/types/task-memory.ts`) plus `TaskMemoryStoreFs`, `WorkspaceMoveRecord`, and `TaskMemoryPruneOutcome`. External consumers must be able to _name_ every parameter/return type of a published function, so adding a task-memory export means publishing its types from `sdk/src/index.ts` in the same change. -- **task-memory.json writers**: `.openbuff/memory/task-memory.json` has two writers — `saveMergedTaskMemory` (end of run) and `pruneStaleTaskMemoryEvidence` (`/memory prune`) — and its `revision` must stay monotonic and unique across both. Uniqueness is enforced by serializing each writer's load→revision→commit section on a shared lock: an in-process promise chain keyed by the resolved record path, plus an exclusive-create `.lock` file for other processes that is reclaimed after a staleness timeout. Without it a save landing between prune's revision check and its rename, or two saves loading the same record, published one revision twice with different payloads. Evidence reconciliation deliberately stays outside the locked section because it hashes every evidence file. The lock is advisory — adapters that ignore exclusive-create, or a lock not obtained within the bounded attempt budget, degrade to running unlocked — so both writers keep their own guards: saves re-read the on-disk record and merge against whichever of it and the caller's hydrated `priorMemory` is newer, emitting one past both, and prune reports `status: 'failed', reason: 'concurrent-write'` when the record moved off the revision it loaded. +- **Published task-memory API (`@openbuff/sdk`)**: the SDK publishes `loadPersistedTaskMemory`, `inspectPersistedTaskMemoryV1`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, `auditTaskMemoryV1Migration`, and `codebuffFsToNodePromises`, together with the complete type closure their signatures name — `TaskMemoryV1`, `TaskMemoryDraftV1`, `TaskMemoryEvidenceV1` (re-exported from `common/src/types/task-memory.ts`) plus `TaskMemoryStoreFs`, `TaskMemoryV1Inspection`, `WorkspaceMoveRecord`, `TaskMemoryPruneOutcome`, `V1MigrationAuditReader`, and `V1MigrationAuditOutcome`. The inspector is additive/read-only and distinguishes absent, valid, invalid, and unreadable records; the legacy loader delegates to it but still collapses all non-valid states to `undefined`. The audit reconstructs every deterministic source-derived task/observation event using the exact marker session and compares full normalized headers/payloads plus marker metadata without appending. `exact` is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`; `not-migrated` means removal is not ready and is not a defect. Keep the audit API/types as transitional read-only contracts through rollback, and follow the normative [Memory V1 removal readiness plan](../docs/memory-v1-removal-readiness.md) before any later removal decision. External consumers must be able to _name_ every parameter/return type of a published function, so adding a task-memory export means publishing its types from `sdk/src/index.ts` in the same change. +- **task-memory.json writers**: `.openbuff/memory/task-memory.json` has two writers — `saveMergedTaskMemory` (end of run) and `pruneStaleTaskMemoryEvidence` (`/memory prune`) — and its `revision` must stay monotonic and unique across both. Uniqueness is enforced by serializing each writer's load→revision→commit section on a shared lock: an in-process promise chain keyed by the resolved record path, plus an explicit atomic-create `.lock` file for other processes. A stale same-host lock is reclaimed only after its recorded owner PID is proven dead; live locks are never stolen. Without it a save landing between prune's revision check and its rename, or two saves loading the same record, published one revision twice with different payloads. Evidence reconciliation deliberately stays outside the locked section because it hashes every evidence file. The cross-process lock is mandatory and ownership-token protected: contention or adapters without the explicit atomic-create capability fail closed without writing, acquisition verifies the token it wrote, and cleanup unlinks only the caller's own token. Large evidence hashes are tagged `sha256-whole:`; unversioned legacy one-megabyte-prefix hashes are validated against both legacy and whole digests, then backfilled only after a match, while unknown/unverifiable formats are preserved so prune cannot delete them — while both writers keep their own guards: saves re-read the on-disk record and merge against whichever of it and the caller's hydrated `priorMemory` is newer, emitting one past both, and prune reports `status: 'failed', reason: 'concurrent-write'` when the record moved off the revision it loaded. - **Pruned evidence must not come back**: a revision guard prevents revision reuse, not content resurrection. `/memory prune` runs from inside a live session whose `runMemory.evidence` still holds the entries it dropped, so `saveMergedTaskMemory` also filters the run's contribution: when the on-disk record is the merge base, every evidence id the caller's hydrated snapshot carries but that record no longer does was deliberately removed by the other writer and is dropped from the run's evidence. Evidence the run recorded itself is untouched. - **Prune honors the move contract**: `pruneStaleTaskMemoryEvidence` takes the same optional `workspaceMoves` as `reconcileTaskMemoryEvidence`, because prune DELETES what reconciles stale. Callers that can see journal-recorded moves must pass them, or evidence bound to a renamed file reconciles stale and is permanently lost instead of rebinding to its destination as hydration documents. `collectWorkspaceMoves` is published from `@openbuff/sdk` for exactly that reason: non-run callers need the same journal-derived move set hydration uses. -- **`/memory` CLI command** (`cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts` as `memory`, alias `mem`): `/memory status` summarizes the persisted record and its fresh/stale evidence; `/memory prune` drops stale evidence. Both subcommands read journal moves via `collectWorkspaceMoves` + `WorkspaceJournalService` and pass them into reconciliation/prune, so a renamed file's evidence rebinds instead of being reported stale and deleted; a journal read that fails surfaces as a command failure rather than a blind prune. Prune reports the store's `TaskMemoryPruneOutcome` faithfully — `no-record` (nothing persisted yet), `pruned` (including `removed: 0`, "nothing to prune"), and `failed` with its reason (schema reject, lost revision race, unwritable target or rename-less filesystem adapter). A failed write is never phrased as an absent record or as "nothing to prune"; it names the cause and reports the stale entries still present. +- **`/memory` CLI command** (`cli/src/commands/memory-command.ts`, registered in `command-registry.ts` and listed in `data/slash-commands.ts` as `memory`, alias `mem`): `/memory status` summarizes the persisted record and its fresh/stale evidence; `/memory prune` drops stale evidence; `/memory audit-migration` invokes the SDK audit against the currently loaded V1 record and leased V2 repository without mutation. Both status/prune subcommands read journal moves via `collectWorkspaceMoves` + `WorkspaceJournalService` and pass them into reconciliation/prune, so a renamed file's evidence rebinds instead of being reported stale and deleted; a journal read that fails surfaces as a command failure rather than a blind prune. Prune reports the store's `TaskMemoryPruneOutcome` faithfully — `no-record` (nothing persisted yet), `pruned` (including `removed: 0`, "nothing to prune"), and `failed` with its reason (schema reject, lost revision race, unwritable target or rename-less filesystem adapter). A failed write is never phrased as an absent record or as "nothing to prune"; it names the cause and reports the stale entries still present. Release N keeps `json-v1` and `shadow-v2` supported but deprecates them in favor of the default/replacement `sqlite-v2-opt-in`; authority selection and compatibility/fallback behavior remain unchanged, with no removal date implied and migration not claimed complete. Any future removal requires a later decision after the canonical readiness gates pass, not merely a current `exact` audit. - **Stable hash (`common/src/util/stable-hash.ts`)**: canonical FNV-1a 32-bit string hash rendered as padded 8-hex. It is the one implementation shared by agent-runtime `commitTaskMemory` checksums, the SDK task-memory store's persisted-record checksum, and the `git_status` change-gate fingerprint; byte-format vectors (`'' -> 811c9dc5`) are pinned by tests because the same digest must round-trip across packages. - **Gate telemetry sink (`common/src/util/gate-telemetry.ts`)**: append-only JSONL sink for base2's `emitGateTelemetry` payloads under `/.openbuff/telemetry/base2-gate.jsonl`, so per-round gate evidence outlives a single turn's `console.info` and questions like "what share of repair rounds were nit-driven" become mechanically answerable. Best-effort by contract: it never throws (not even when `logger.warn` itself throws), reports failure through its boolean return, and keeps owner-only modes (`0o700` directory, `0o600` file) like the SDK task-memory store. Lines are size-bounded in three escalating steps — array-item caps, per-field byte caps, then an event-name-only marker — and `truncated` / `truncatedFields` / `droppedPayloadKeys` are written by the SINK, so a payload field of one of those names is dropped rather than merged and cannot forge the truncation signal. Rotation keeps at most one `.1` generation and deliberately accepts two documented races; `GATE_TELEMETRY_MAX_BYTES` is the authoritative note for both. `errorCode` in `common/src/util/error.ts` is the shared helper it keys `ENOENT` on, and `common/src/testing/gate-telemetry-fixtures.ts` owns the temp-root helpers plus the platform skip policy shared with the agent-runtime suite (a skip rule that drifts between the two silently stops covering one). - **On-demand guide fallback (`common/src/util/guides.ts`)**: base2's progressive prompt disclosure replaces each verbose advisory section with a compact pointer naming a workspace-relative `agents/guides/*.md` path, which `read_files` resolves against the EMBEDDER's project root — and no publish pipeline ships the guides, so in any workspace other than this repo the pointer read fails and the model loses the whole section. `FALLBACK_GUIDES` is the single copy of those paths (base2's `GUIDE_PATHS` is an alias of it and its `GuidePath` union is derived from it); the table lives here rather than beside base2's pointers because `packages/agent-runtime` must not import from `agents/`. `findMissingGuides` returns `[]` for a falsy or non-string root — "unknown root" must never mean "everything is missing", or every prompt formatted without a real workspace regrows by six full sections — and `formatGuideFallbackSection` returns `''` for a guide that is present, so the resolved in-repo prompt stays byte-identical. Recovery is PER POINTER and clause-keyed: a mode that deliberately omits a pointer (plan mode is read-only and emits neither the git-discipline pointer nor its body) never recovers that body, and `BROAD_AUDIT_FALLBACK_SECTIONS` keys the one clause-parameterized body by finalize clause so a guide-less plan-mode prompt cannot receive the contradictory implementation clause. diff --git a/common/src/types/session-state.ts b/common/src/types/session-state.ts index b76e427782..40b6e369b4 100644 --- a/common/src/types/session-state.ts +++ b/common/src/types/session-state.ts @@ -396,7 +396,11 @@ export type AgentState = { childRunId?: string receipt?: AgentReceipt }> - /** Typed operational memory compiled into each model request independently of chat summaries. */ + /** + * Typed operational memory compiled into each model request independently of chat summaries. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ taskMemory?: TaskMemoryV1 /** Opt-in runtime-neutral Memory V2 lifecycle state. */ memoryV2?: MemoryRuntimeStateV2 diff --git a/common/src/types/task-memory.ts b/common/src/types/task-memory.ts index 803684d32b..1613035899 100644 --- a/common/src/types/task-memory.ts +++ b/common/src/types/task-memory.ts @@ -7,6 +7,8 @@ const boundedText = (max: number) => z.string().max(max) * consumers (e.g. the SDK task-memory store) merge against the exact * limits this schema enforces instead of hand-mirroring them and * drifting silently. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export const TASK_MEMORY_LIST_CAPS = { requirements: 64, @@ -20,6 +22,9 @@ export const TASK_MEMORY_LIST_CAPS = { evidence: 256, } as const +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export const taskMemoryEvidenceV1Schema = z.object({ id: z.string().min(1).max(160), kind: z.enum([ @@ -43,6 +48,9 @@ export const taskMemoryEvidenceV1Schema = z.object({ stale: z.boolean().optional(), }) +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export const taskMemoryDraftV1Schema = z.object({ schemaVersion: z.literal(1), goal: boundedText(8_000).default(''), @@ -87,12 +95,24 @@ export const taskMemoryDraftV1Schema = z.object({ workspaceSnapshotId: boundedText(256).optional(), }) +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export const taskMemoryV1Schema = taskMemoryDraftV1Schema.extend({ revision: z.number().int().nonnegative(), updatedAt: z.number().int().nonnegative(), checksum: z.string().min(1).max(64), }) +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export type TaskMemoryEvidenceV1 = z.infer +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export type TaskMemoryDraftV1 = z.infer +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export type TaskMemoryV1 = z.infer diff --git a/docs/memory-v1-removal-readiness.md b/docs/memory-v1-removal-readiness.md new file mode 100644 index 0000000000..ad09f6f24a --- /dev/null +++ b/docs/memory-v1-removal-readiness.md @@ -0,0 +1,245 @@ +# Memory V1 Removal Readiness Plan + +## Non-claims and current status + +This is the canonical readiness plan for a later Memory V1 removal decision. It is not removal approval. + +- Release N retains V1 compatibility. `json-v1`, `shadow-v2`, V1 persistence, migration, fallback behavior, and the published V1 APIs remain supported as currently documented. +- No removal date or version is set. Removal, if any, requires a later release decision after every gate below passes. +- Migration is not claimed complete. Current evidence is insufficient to authorize removal. +- `/memory audit-migration` outcome `not-migrated` means removal is **not ready** for that project. It reports migration state and is not a product defect. +- Outcome `exact` means every deterministic source-derived task/observation body field and header, plus marker metadata, was reconstructed from the checksum-verified source and exact marker session and compared after schema normalization. +- Even `exact` is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`. Warnings are never waived. + +Normative terms such as **MUST**, **MUST NOT**, and **SHOULD** define acceptance criteria for a later removal decision. + +## Readiness definition + +Memory V1 is removal-ready only when all in-scope projects and supported deployment paths have mechanically reviewable evidence that removing V1 will not lose data, misstate authority, strand mixed-version clients, or prevent rollback. Before any V1 behavior or API is removed: + +1. Every source blocker below **MUST** be closed in implementation and tests. +2. Every in-scope project **MUST** have the per-project evidence bundle below. A project with `not-migrated`, lossy `exact`, an ambiguous `no-record`, incomplete evidence, or a changed V1 revision/checksum is a no-go. +3. The compatibility window and every staged rollout gate **MUST** complete without an unresolved rollback trigger. +4. Storage, restore, mixed-version, public-consumer, operator, telemetry/support, and ownership criteria **MUST** be signed off. +5. `auditTaskMemoryV1Migration`, `V1MigrationAuditReader`, `V1MigrationAuditOutcome`, and the related audit types **MUST** remain supported transitional read-only contracts until the rollback window closes, unless an equivalent supported diagnostic is available first. + +## Current removal-readiness blockers + +These blockers are grounded in the current source and make the present answer **not ready**: + +1. **Full source-derived body equality is implemented.** `auditMigrationMarkerBody` reconstructs deterministic task/observation drafts from the V1 source and marker session, compares full normalized headers and payloads, and fails closed on missing, duplicate, reordered, substituted, or tampered deterministic evidence. This Gate 0 blocker is complete. +2. **Four-state read-only inspection is implemented.** `inspectPersistedTaskMemoryV1` distinguishes `absent`, checksum-verified `valid`, bounded-reason `invalid`, and `unreadable`; only `ENOENT` is absence. The legacy loader continues collapsing non-valid states for compatibility. Verified inspector `absent` is independent absence evidence. This Gate 0 blocker is complete. +3. **The importer deliberately loses or excludes some data.** `importTaskMemoryV1` warns or omits data including a non-empty goal (`goal-excluded`) and legacy evidence (`legacy-evidence-unverified`), and can omit stale/unsafe/empty/capped data or truncate text. Any warning, omission, or truncation is lossy and cannot authorize removal. Common legitimate warning cases do not create an exception; warnings **MUST NOT** be waived. +4. **Interactive audit is bounded.** `scanV1MigrationAuditEvents` scans at most 10 export pages. `/memory audit-migration` therefore cannot be the sole release-evidence path for repositories beyond that bound. Release evidence requires a full or resumable, schema-validated audit that validates pagination and reaches the canonical end of every in-scope repository. +5. **Post-V1 authority failure semantics are undecided.** Before removal, behavior for legacy or invalid authority values and for V2-open/reset failure **MUST** be explicitly specified, tested, and communicated. It **MUST** fail closed and must not silently report successful V2 authority when canonical V2 storage did not open. The decision must cover `OPENBUFF_MEMORY_AUTHORITY`, SDK authority inputs, provider result types, CLI reporting, and persisted/session authority state. + +## Per-project evidence bundle + +For each project in the release scope, retain a reviewable bundle outside this canonical policy document. Do not paste local machine paths, local source revisions/checksums, or artifact hashes into this file. The bundle **MUST** contain: + +- project identity and evidence collection time, tool/release identity, and the declared release-scope owner; +- the non-mutating V1 inspection result (`valid` or independently proven `absent`; never inferred from `no-record`); +- for a valid record, the checksum-verified V1 backup and its recorded revision/checksum in the controlled evidence system; +- import outcome and a fresh full/resumable audit against that same V1 revision/checksum; +- an `exact` result from the future full-body-comparison audit, with `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`; +- a validated canonical V2 export, including validated manifest/event schemas, project identity, pagination continuity, and terminal cursor/end-of-export evidence; +- restore-drill evidence from a disposable copy, plus read/query verification after restore; +- mixed-version test results, authority/degradation diagnostics, and operator acknowledgement of any retry or remediation; +- evidence-review signoff and an explicit pass/fail disposition for every checklist item. + +Any stale, partial, unparsable, mismatched, `not-migrated`, `incomplete`, `mismatch`, `rejected`, or `failed` evidence blocks that project. + +## Compatibility window + +Release N is the deprecation release and keeps compatibility. During the compatibility window: + +- V1 read/write persistence, import, shadow/fallback behavior, authority values, CLI guidance, and public SDK/common contracts remain available. +- Deprecation notices direct operators to collect evidence; they do not claim migration completion or imply a removal date/version. +- A prior compatible artifact and configuration **MUST** remain available and tested for rollback. +- The window ends only through a later release decision after all gates and signoffs pass. It is not ended merely by low observed V1 use or by an `exact` result from the current bounded audit. + +## Staged rollout gates + +Each gate is blocking and advances only with recorded evidence and owner approval. No numeric threshold or fixed hold duration is defined here; release owners must judge the scoped evidence and unresolved risk rather than retrofit an arbitrary percentage. + +### Gate 0: blocker closure + +- [x] Full source-derived task/observation body comparison is implemented and tampered-body tests fail closed. +- [x] Non-mutating V1 inspection distinguishes `absent`, `valid`, `invalid`, and `unreadable`. +- [ ] A full/resumable validated audit supports stores beyond the interactive 10-page bound. +- [ ] Post-V1 authority and V2-open/reset failure semantics are approved and fail closed. +- [ ] Lossless evaluation uses the exact zero-omission/zero-truncation/zero-warning predicate. + +### Gate 1: evidence tooling and rehearsal + +- [ ] Evidence tooling is read-only except for an explicit import operation and produces the complete per-project bundle. +- [ ] Backup, canonical V2 export, disposable restore, re-import, and rollback procedures pass in supported storage environments. +- [ ] Unit, contract, integration, storage, consumer, and release validation below pass. + +### Gate 2: compatibility-window observation + +- [ ] All in-scope projects have fresh passing bundles. +- [ ] Telemetry and support review finds no unresolved V1 dependency, data-loss signal, authority misreporting, or rollback blocker. +- [ ] Mixed-version operation and rollback remain supported with both stores preserved. + +### Gate 3: reversible removal candidate + +- [ ] A candidate artifact can disable the proposed V1 surfaces without deleting either store or preventing re-enable/rollback. +- [ ] Upgrade, downgrade, mixed-version, invalid-authority, V2-open failure, and restored-data scenarios pass. +- [ ] Public API and CLI removal inventories have consumer migration evidence and release-note review. + +### Gate 4: later removal decision + +- [ ] Every go/no-go item is checked and required owners have signed off. +- [ ] No rollback trigger is open. +- [ ] The decision explicitly identifies the surfaces approved for removal; unlisted V1 behavior remains supported. +- [ ] The rollback window and criteria are declared for that release while transitional diagnostics remain supported. + +## Public SDK/common removal inventory + +A later decision **MUST** explicitly inventory and assess, rather than implicitly delete, at least: + +- SDK task-memory functions: `inspectPersistedTaskMemoryV1`, `loadPersistedTaskMemory`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, and `codebuffFsToNodePromises`; + +- SDK migration functions: `importTaskMemoryV1`, `getV1MigrationIdentity`, and the transitional read-only `auditTaskMemoryV1Migration`; +- SDK types: `TaskMemoryStoreFs`, `TaskMemoryV1Inspection`, `WorkspaceMoveRecord`, `TaskMemoryPruneOutcome`, `V1MigrationOutcome`, `V1MigrationWarningCode`, `V1MigrationSourceItemCounts`, `V1MigrationAuditReader`, and `V1MigrationAuditOutcome`; + +- common schemas/constants/types: `TASK_MEMORY_LIST_CAPS`, `taskMemoryEvidenceV1Schema`, `taskMemoryDraftV1Schema`, `taskMemoryV1Schema`, `TaskMemoryEvidenceV1`, `TaskMemoryDraftV1`, and `TaskMemoryV1`; +- common session/checkpoint contracts that carry V1 import, compatibility-shadow, parity, warning, or authority state, including `MemoryV1ImportWarningCode`, `MemoryV1ImportState`, and `MemoryAuthorityStateV2`; +- persisted `.openbuff/memory/task-memory.json` semantics and any runtime hydration, reconciliation, save, prune, shadow, fallback, or migration callers. + +For each item, record public/internal status, known consumers, replacement, compatibility impact, test coverage, documentation, and rollback behavior. Audit contracts follow the longer transitional lifetime stated above. + +## CLI/env/provider removal inventory + +A later decision **MUST** separately inventory: + +- `/memory status`, `/memory prune`, and `/memory audit-migration`, including all outcome and warning guidance; +- `OPENBUFF_MEMORY_AUTHORITY`, `getMemoryAuthoritySelection`, and authority values `json-v1`, `shadow-v2`, and `sqlite-v2-opt-in`; +- invalid/legacy authority handling, the default authority, and SDK-supplied authority options; +- `ProjectMemoryV2Provider`, `MemoryV2ProviderResult`, effective/requested authority reporting, and storage-unavailable/reset-during-open degradation; +- CLI status/diagnose/authority output, V1 compatibility-shadow reporting, V2 export/import, and operator recovery instructions; +- packaged configuration, environment documentation, release notes, scripts, and external automation that set or parse these values. + +Removal readiness requires tested replacements and fail-closed reporting for every inventoried path. + +## Storage, backup, and no-deletion rules + +Before a project can pass readiness: + +- Create and retain a checksum-verified V1 backup before import or authority changes. +- Produce and validate a canonical V2 export as described in the evidence bundle. +- Perform a restore drill on a disposable copy and validate restored reads/queries; never rehearse destructively on the only copy. +- Preserve both V1 and V2 stores, their backups, and the prior compatible artifact/configuration throughout rollout and the rollback window. +- **MUST NOT** automatically delete V1 data, automatically clean up either store, or migrate in place. Import/copy to V2 and verify independently. +- Destructive cleanup, if ever proposed after the rollback window, requires a separate explicit decision, fresh backups, operator-visible scope, and its own rollback analysis. It is not authorized by this plan. + +## Mixed-version and rollback criteria + +Older compatible clients may continue to revise V1 while newer clients use V2. Therefore: + +- After any V1 revision or checksum change, the prior migration evidence is stale: re-import and collect a fresh lossless full audit before proceeding. +- Mixed-version tests **MUST** cover old-writer/new-reader, new-writer/old-reader, concurrent/sequential use, upgrade, downgrade, and interrupted V2 open/import. +- Rollback **MUST** restore the prior compatible artifact and configuration without deleting or overwriting either store. Both stores remain intact for diagnosis and a later retry. +- A rollback is complete only when authority reporting matches actual storage use, the V1 backup still verifies, V1 operation is usable where expected, and no post-backup V1 revision is silently discarded. +- `auditTaskMemoryV1Migration` and its audit types remain available through the rollback window unless an equivalent supported read-only diagnostic exists. + +## Telemetry and support review + +Use privacy-appropriate, bounded signals already available to the product; do not add project contents, local paths, checksums, or raw repository errors to telemetry. Review by release scope and supported deployment path: + +- requested versus effective authority, fallback/degradation, V2 open/reset failures, and invalid/legacy authority selection; +- migration outcomes and warning/omission/truncation presence, without treating `not-migrated` as a defect; +- audit incompleteness/mismatch/failure and page-bound encounters; +- restore, rollback, data-loss, stale-evidence, and mixed-version support cases; +- external consumer reports concerning deprecated SDK/common/CLI contracts. + +No numeric threshold, support-volume limit, percentage, or hold duration is prescribed here. Owners must document the reviewed evidence, unresolved cases, and why the observed record is sufficient for the scoped decision. Any credible unresolved data loss, false authority claim, inability to restore, or required V1 dependency is a no-go. + +## Decision ownership and signoffs + +Roles may be filled according to the release process; this plan does not name individuals. The release record **MUST** include approval from: + +- the Memory implementation owner, for blocker closure and semantic correctness; +- the storage/reliability owner, for backup, canonical export, restore, mixed-version, and rollback evidence; +- SDK/common and CLI maintainers, for public contract and operator inventory; +- test/release engineering, for the validation matrix and artifact reproducibility; +- support/operations, for migration guidance, observed cases, and rollback readiness; +- the release decision owner, for final scope, accepted residual risk, rollback window, and go/no-go disposition. + +A signoff identifies its evidence bundle and scope; silence or a general release approval is not a Memory V1 removal signoff. + +## Rollback triggers + +Stop rollout and restore the prior compatible artifact/configuration when any of the following occurs: + +- V1 inspection is ambiguous, invalid, unreadable, or disagrees with the retained backup; +- audit is not fresh and lossless, any warning/omission/truncation appears, or a source-derived body tamper is not detected; +- a V1 revision/checksum changes without re-import and fresh audit; +- canonical V2 export, restore, pagination, integrity, or query validation fails; +- requested and effective authority diverge without an explicit fail-closed degradation, or V2-open failure is reported as successful V2 authority; +- a supported mixed-version, upgrade, downgrade, or consumer path requires a removed surface; +- either store, backup, prior artifact/configuration, or transitional diagnostic is unavailable during the rollback window; +- telemetry/support review identifies credible data loss, inaccessible memory, or an unresolved rollback blocker. + +## Validation matrix + +### Unit and contract + +- Import mapping and lossless predicate tests, including every warning/omission/truncation code. +- Full reconstruction/equality tests for every source-derived task and observation field; tamper each field and require fail-closed non-`exact` results. +- V1 inspector tests for `absent`, `valid`, `invalid`, `unreadable`, and checksum failure. +- Audit outcome, schema, project-binding, duplicate-ID, pagination, terminal-cursor, and resumability tests. +- Authority parsing/provider-result tests for current, legacy, invalid, unavailable, reset, and thrown-open paths; reported authority must equal actual authority. +- Public type/export contract tests, retaining transitional audit contracts. + +### Integration and storage + +- Checksum-verified V1 backup, import, full canonical V2 export, disposable restore, and restored query/read validation. +- Stores larger than the interactive 10-page audit bound, malformed pages, interrupted/resumed scans, and repository failure. +- Filesystem permission/unreadable/invalid/checksum-failed cases without mutation. +- Both stores preserved across upgrade, downgrade, rollback, failed import, and failed V2 open. +- V1 revision/checksum change followed by mandatory re-import and fresh lossless audit. + +### Consumer and release + +- Published SDK/common API inventory and representative external-consumer compile/contract validation. +- CLI commands, environment/config parsing, diagnostics, operator messages, package upgrade/downgrade, and prior-artifact rollback. +- Release artifact smoke tests for each supported deployment/storage path. +- Release notes state Release N compatibility, no set removal date/version, no migration-complete claim, and the later-decision requirement. +- Evidence bundles and signoffs are reviewable without embedding local paths or secret/raw project content in committed documentation. + +## Operator migration guidance + +1. Keep a known-compatible artifact and the current authority configuration available. +2. Inspect V1 non-mutatingly with `inspectPersistedTaskMemoryV1`. If the result is `invalid` or `unreadable`, stop and repair access/record handling; do not interpret it as absence. Only a verified `absent` result is independent absence evidence. +3. Create and verify a V1 backup. Do not migrate in place or delete the source. +4. Import to V2 using the supported path, then run the full/resumable audit. `/memory audit-migration` remains useful interactively but its 10-page bound cannot be sole evidence for larger stores. +5. Stop on `not-migrated`, `incomplete`, `mismatch`, `rejected`, or `failed`. `not-migrated` means readiness work remains; it is not itself a product defect. +6. For `exact`, require `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`; full deterministic source-derived body equality is already part of `exact`. Do not waive `goal-excluded`, `legacy-evidence-unverified`, or any other warning. +7. Validate and retain a canonical V2 export, then restore and test it on a disposable copy. +8. If V1 revision/checksum changes, re-import and repeat the fresh lossless audit and export validation. +9. Preserve both stores until the declared rollback window closes. On a trigger, restore the prior artifact/configuration; do not delete either store. + +## Explicit go/no-go checklist + +A later removal decision is **GO** only when every applicable item is checked: + +- [ ] This is a later release decision; Release N was compatibility-only, and the proposed scope/date is explicit in that later record. +- [ ] Migration is not assumed complete; every in-scope project has a fresh per-project evidence bundle. +- [ ] All remaining current blockers are closed with reviewed implementation and tests. +- [ ] Every valid V1 record has full-body `exact` evidence and the exact lossless predicate passes with no warnings waived. +- [ ] Every claimed absent V1 record is independently inspected as `absent`; no decision relies on `no-record` alone. +- [ ] No project has `not-migrated`, stale, incomplete, mismatched, rejected, failed, lossy, invalid, unreadable, or partial evidence. +- [ ] Repositories beyond 10 pages were audited through the full/resumable validated path. +- [ ] Checksum-verified V1 backups, validated canonical V2 exports, and disposable restore drills pass. +- [ ] Both stores and a prior compatible artifact/configuration will remain intact through rollback; no automatic deletion or migrate-in-place is enabled. +- [ ] Every V1 revision/checksum change was followed by re-import and a fresh lossless audit. +- [ ] Mixed-version, authority, V2-open failure, upgrade/downgrade, rollback, and consumer validation pass fail-closed. +- [ ] Public SDK/common and CLI/env/provider inventories have replacements, consumer evidence, documentation, and rollback coverage. +- [ ] Transitional `auditTaskMemoryV1Migration` contracts remain supported through rollback or an equivalent diagnostic is available. +- [ ] Telemetry/support review has no unresolved removal blocker and invents no threshold to hide one. +- [ ] Required role-based signoffs reference the scoped evidence. +- [ ] No rollback trigger is open. + +Any unchecked item is **NO-GO**. A no-go preserves current compatibility and sets no removal date/version. diff --git a/sdk/CHANGELOG.md b/sdk/CHANGELOG.md index 3bd7e5788e..0835f9580f 100644 --- a/sdk/CHANGELOG.md +++ b/sdk/CHANGELOG.md @@ -6,6 +6,10 @@ All notable changes to the @openbuff/sdk package will be documented in this file ### Added +- New additive, read-only `auditTaskMemoryV1Migration` API and `V1MigrationAuditOutcome` type inspect a checksum-verified V1 record against a V2 repository without appending or mutating. An `exact` result reconstructs every deterministic source-derived task/observation event from the source and the marker session, then compares full schema-normalized headers and payloads (excluding only envelope `sequence`) plus all marker metadata. It is lossless only when `omittedFields === 0`, `(truncatedFields ?? 0) === 0`, and `warnings.length === 0`. Other outcomes distinguish no record, not migrated, incomplete, mismatch, rejected, and failed states using bounded reason codes rather than repository error text. +- New additive, read-only `inspectPersistedTaskMemoryV1` API and `TaskMemoryV1Inspection` type distinguish `absent`, checksum-verified `valid`, `invalid` (malformed JSON, schema invalid, or checksum mismatch), and `unreadable` records without exposing paths, contents, validation issues, or raw errors. The legacy `loadPersistedTaskMemory` signature and behavior are unchanged and now delegate to the inspector, collapsing every non-valid result to `undefined` for compatibility. +- Release N deprecation guidance: `json-v1` and `shadow-v2` remain supported compatibility modes but are deprecated; `sqlite-v2-opt-in` is the default and replacement. This release does not remove V1 APIs or persistence, migration import, authority selection, shadow behavior, or fallback behavior, sets no removal date/version, and does not claim migration is complete. Any removal requires a later release decision after the normative gates in the [Memory V1 removal readiness plan](../docs/memory-v1-removal-readiness.md) pass; `not-migrated` means removal is not ready, not that the product is defective. + - New published task-memory API for durable cross-session task state: `loadPersistedTaskMemory`, `reconcileTaskMemoryEvidence`, `saveMergedTaskMemory`, `pruneStaleTaskMemoryEvidence`, and `codebuffFsToNodePromises`, backed by `.openbuff/memory/task-memory.json` under the project root. The complete type closure those signatures name is published alongside them so consumers can name every parameter and return type: `TaskMemoryV1`, `TaskMemoryDraftV1`, and `TaskMemoryEvidenceV1` (re-exported from `@codebuff/common/types/task-memory`), plus `TaskMemoryStoreFs`, `WorkspaceMoveRecord`, and `TaskMemoryPruneOutcome`. - `pruneStaleTaskMemoryEvidence` returns a discriminated `TaskMemoryPruneOutcome` rather than a nullable count pair, so "there is no record" (`status: 'no-record'`) is distinguishable from "the prune could not be committed" (`status: 'failed'`, with `reason: 'invalid-record' | 'concurrent-write' | 'write-failed'` and the `removed`/`remaining` counts the prune would have written). A successful prune is `status: 'pruned'`, including the no-op `removed: 0` case for a fully fresh record. - `task-memory.json` has two writers, so `revision` stays monotonic and unique across both. `saveMergedTaskMemory` re-reads the on-disk record and merges against whichever of it and the caller's `priorMemory` is newer, emitting one revision past both; pruning refuses to write when the record advanced while it was reconciling. A session that hydrated a pre-prune record therefore cannot resurrect pruned evidence under an already-published revision. diff --git a/sdk/src/__tests__/memory-v2-contract.test.ts b/sdk/src/__tests__/memory-v2-contract.test.ts index d491968426..8a5d8b7c55 100644 --- a/sdk/src/__tests__/memory-v2-contract.test.ts +++ b/sdk/src/__tests__/memory-v2-contract.test.ts @@ -16,12 +16,19 @@ import type { MemoryRetrievalRequest, MemoryVerifyOutcome, MemoryVerifyRequest, + TaskMemoryV1, + TaskMemoryV1Inspection, + V1MigrationAuditOutcome, + V1MigrationAuditReader, } from '../index' import { + auditTaskMemoryV1Migration, + inspectPersistedTaskMemoryV1, MemoryAppendRequestSchema, MemoryEventDraftSchema, MemoryEventEnvelopeSchema, MemoryRetrievalRequestSchema, + ProjectIdSchema, } from '../index' const timestamp = '2026-09-10T19:41:53.753Z' @@ -41,26 +48,38 @@ class FakeMemoryRepositoryV2 implements MemoryRepositoryV2 { }, } } - const projectEvents = this.events.filter((event) => event.projectId === request.projectId) + const projectEvents = this.events.filter( + (event) => event.projectId === request.projectId, + ) const currentLastEventId = projectEvents.at(-1)?.eventId - const expectedTail = request.expectedTail ?? ( - request.expectedLastEventId === undefined + const expectedTail = + request.expectedTail ?? + (request.expectedLastEventId === undefined ? { kind: 'any' as const } - : { kind: 'event' as const, eventId: request.expectedLastEventId } - ) - const tailMatches = expectedTail.kind === 'any' - || (expectedTail.kind === 'empty' && currentLastEventId === undefined) - || (expectedTail.kind === 'event' && expectedTail.eventId === currentLastEventId) + : { kind: 'event' as const, eventId: request.expectedLastEventId }) + const tailMatches = + expectedTail.kind === 'any' || + (expectedTail.kind === 'empty' && currentLastEventId === undefined) || + (expectedTail.kind === 'event' && + expectedTail.eventId === currentLastEventId) if (!tailMatches) { return { outcome: 'rejected', - error: { code: 'conflict', message: 'The memory store changed.', retryable: true }, + error: { + code: 'conflict', + message: 'The memory store changed.', + retryable: true, + }, } } const stagedDrafts = new Map(this.drafts) const pending: MemoryEventEnvelope[] = [] - const entries: Array<{ eventId: MemoryEventDraft['eventId']; sequence: number; duplicate: boolean }> = [] + const entries: Array<{ + eventId: MemoryEventDraft['eventId'] + sequence: number + duplicate: boolean + }> = [] for (const draft of request.events) { const existingDraft = stagedDrafts.get(draft.eventId) const existingEvent = [...this.events, ...pending].find( @@ -70,10 +89,18 @@ class FakeMemoryRepositoryV2 implements MemoryRepositoryV2 { if (JSON.stringify(existingDraft) !== JSON.stringify(draft)) { return { outcome: 'rejected', - error: { code: 'conflict', message: 'The event ID already exists.', retryable: false }, + error: { + code: 'conflict', + message: 'The event ID already exists.', + retryable: false, + }, } } - entries.push({ eventId: draft.eventId, sequence: existingEvent.sequence, duplicate: true }) + entries.push({ + eventId: draft.eventId, + sequence: existingEvent.sequence, + duplicate: true, + }) continue } @@ -83,7 +110,11 @@ class FakeMemoryRepositoryV2 implements MemoryRepositoryV2 { }) stagedDrafts.set(draft.eventId, draft) pending.push(event) - entries.push({ eventId: draft.eventId, sequence: event.sequence, duplicate: false }) + entries.push({ + eventId: draft.eventId, + sequence: event.sequence, + duplicate: false, + }) } this.events.push(...pending) @@ -117,12 +148,20 @@ class FakeMemoryRepositoryV2 implements MemoryRepositoryV2 { async verify(_request: MemoryVerifyRequest): Promise { return { outcome: 'failed', - error: { code: 'not-found', message: 'Observation not found', retryable: false }, + error: { + code: 'not-found', + message: 'Observation not found', + retryable: false, + }, } } async rebuild(request: MemoryRebuildRequest): Promise { - return { outcome: 'rebuilt', rebuildId: request.rebuildId, processedEvents: this.events.length } + return { + outcome: 'rebuilt', + rebuildId: request.rebuildId, + processedEvents: this.events.length, + } } async health(_request: MemoryHealthRequest): Promise { @@ -135,7 +174,14 @@ class FakeMemoryRepositoryV2 implements MemoryRepositoryV2 { backendId: 'fake', kind: 'in-memory', persistence: 'ephemeral', - capabilities: ['append', 'query', 'verify', 'rebuild', 'health', 'export'], + capabilities: [ + 'append', + 'query', + 'verify', + 'rebuild', + 'health', + 'export', + ], }, issues: [], } @@ -151,6 +197,49 @@ class FakeMemoryRepositoryV2 implements MemoryRepositoryV2 { } describe('MemoryRepositoryV2 public contract', () => { + test('exports the four-state V1 inspector with a nameable exhaustive union', async () => { + const result: TaskMemoryV1Inspection = await inspectPersistedTaskMemoryV1({ + rootDir: '/absent-contract-record', + fs: { + readFile: async () => { + throw Object.assign(new Error('missing'), { code: 'ENOENT' }) + }, + } as never, + }) + const describe = (inspection: TaskMemoryV1Inspection): string => { + switch (inspection.status) { + case 'absent': + return 'absent' + case 'valid': { + const memory: TaskMemoryV1 = inspection.memory + return String(memory.revision) + } + case 'invalid': + return inspection.reason + case 'unreadable': + return inspection.reason + } + } + expect(describe(result)).toBe('absent') + }) + + test('the V1 migration audit accepts an export-only reader', async () => { + let exportCalls = 0 + const reader: V1MigrationAuditReader = { + async export() { + exportCalls++ + return { outcome: 'page', events: [], nextAfterEventId: null } + }, + } + const outcome: V1MigrationAuditOutcome = await auditTaskMemoryV1Migration({ + projectId: ProjectIdSchema.parse('project:demo'), + repository: reader, + }) + + expect(outcome).toEqual({ outcome: 'no-record' }) + expect(exportCalls).toBe(0) + }) + test('supports a compile-time and runtime fake implementation', async () => { const event = MemoryEventDraftSchema.parse({ schemaVersion: 2, @@ -187,10 +276,12 @@ describe('MemoryRepositoryV2 public contract', () => { const repository: MemoryRepositoryV2 = new FakeMemoryRepositoryV2() const appendOutcome = await repository.append(appendRequest) - const staleEmpty = await repository.append(MemoryAppendRequestSchema.parse({ - ...appendRequest, - expectedTail: { kind: 'empty' }, - })) + const staleEmpty = await repository.append( + MemoryAppendRequestSchema.parse({ + ...appendRequest, + expectedTail: { kind: 'empty' }, + }), + ) const duplicateOutcome = await repository.append(appendRequest) const queryOutcome = await repository.query(queryRequest) const exportOutcome = await repository.export({ diff --git a/sdk/src/__tests__/task-memory-store.test.ts b/sdk/src/__tests__/task-memory-store.test.ts index 50093250cd..6b9aceb390 100644 --- a/sdk/src/__tests__/task-memory-store.test.ts +++ b/sdk/src/__tests__/task-memory-store.test.ts @@ -6,6 +6,8 @@ import { readFile, rm, stat, + symlink, + utimes, writeFile, } from 'node:fs/promises' import * as nodeFsPromises from 'node:fs/promises' @@ -20,6 +22,7 @@ import { stableHash } from '@codebuff/common/util/stable-hash' import { collectWorkspaceMoves, persistRunTaskMemory } from '../run' import { codebuffFsToNodePromises, + inspectPersistedTaskMemoryV1, loadPersistedTaskMemory, pruneStaleTaskMemoryEvidence, reconcileTaskMemoryEvidence, @@ -83,6 +86,77 @@ describe('task-memory-store', () => { await rm(rootDir, { recursive: true, force: true }) }) + test('inspector distinguishes absent, valid, invalid, and unreadable without writes', async () => { + expect(await inspectPersistedTaskMemoryV1({ rootDir })).toEqual({ + status: 'absent', + }) + + const saved = await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory(), + }) + if (!saved) throw new Error('expected saved memory') + expect(await inspectPersistedTaskMemoryV1({ rootDir })).toEqual({ + status: 'valid', + memory: saved, + }) + const memoryPath = path.join( + rootDir, + '.openbuff', + 'memory', + 'task-memory.json', + ) + await writeFile(memoryPath, '{private malformed contents') + expect(await inspectPersistedTaskMemoryV1({ rootDir })).toEqual({ + status: 'invalid', + reason: 'malformed-json', + }) + await writeFile(memoryPath, JSON.stringify({ schemaVersion: 1 })) + expect(await inspectPersistedTaskMemoryV1({ rootDir })).toEqual({ + status: 'invalid', + reason: 'schema-invalid', + }) + await writeFile(memoryPath, JSON.stringify({ ...saved, checksum: 'wrong' })) + expect(await inspectPersistedTaskMemoryV1({ rootDir })).toEqual({ + status: 'invalid', + reason: 'checksum-mismatch', + }) + + let writeCalls = 0 + const readOnlyFs = { + readFile: async () => { + throw Object.assign(new Error('/private/path denied'), { + code: 'EACCES', + }) + }, + mkdir: async () => { + writeCalls++ + }, + rename: async () => { + writeCalls++ + }, + stat: async () => { + throw new Error('unused') + }, + unlink: async () => { + writeCalls++ + }, + writeFile: async () => { + writeCalls++ + }, + } as unknown as import('../services/task-memory-store').TaskMemoryStoreFs + await expect( + inspectPersistedTaskMemoryV1({ rootDir, fs: readOnlyFs }), + ).resolves.toEqual({ + status: 'unreadable', + reason: 'read-failed', + }) + expect(writeCalls).toBe(0) + expect( + await loadPersistedTaskMemory({ rootDir, fs: readOnlyFs }), + ).toBeUndefined() + }) + test('AC1: save then load+reconcile verifies fresh evidence', async () => { await writeFile(path.join(rootDir, 'a.ts'), 'export const a = 1') const memory = makeMemory({ @@ -332,6 +406,139 @@ describe('task-memory-store', () => { } }) + test('ancestor symlink escapes go stale without reading outside the project', async () => { + if (process.platform === 'win32') return + const outsideDir = await mkdtemp( + path.join(tmpdir(), 'task-memory-symlink-outside-'), + ) + try { + const outsideContents = 'outside through ancestor link' + await writeFile(path.join(outsideDir, 'secret.ts'), outsideContents) + await symlink(outsideDir, path.join(rootDir, 'linked')) + + const readFileCalls: string[] = [] + const spyingFs = new Proxy(nodeFsPromises, { + get(target, prop) { + if (prop === 'readFile') { + return async ( + ...args: Parameters + ) => { + readFileCalls.push(String(args[0])) + return nodeFsPromises.readFile(...args) + } + } + return Reflect.get(target, prop) + }, + }) + const reconciled = await reconcileTaskMemoryEvidence({ + memory: makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-ancestor-link', + path: 'linked/secret.ts', + freshnessHash: sha256(outsideContents), + }), + ], + }), + rootDir, + fs: spyingFs, + }) + + expect(reconciled.evidence[0]!.stale).toBe(true) + expect(readFileCalls).toEqual([]) + } finally { + await rm(outsideDir, { recursive: true, force: true }) + } + }) + + test('ancestor replacement after realpath cannot redirect descriptor hashing outside', async () => { + if (process.platform === 'win32') return + const outsideDir = await mkdtemp( + path.join(tmpdir(), 'task-memory-race-outside-'), + ) + try { + await mkdir(path.join(rootDir, 'inside')) + await writeFile(path.join(rootDir, 'inside', 'secret.ts'), 'inside body') + await writeFile(path.join(outsideDir, 'secret.ts'), 'outside secret') + const linkedPath = path.join(rootDir, 'linked') + await symlink(path.join(rootDir, 'inside'), linkedPath) + const candidate = path.join(linkedPath, 'secret.ts') + let replaced = false + const racingFs = new Proxy(nodeFsPromises, { + get(target, prop) { + if (prop === 'realpath') { + return async (requested: string) => { + const resolved = await nodeFsPromises.realpath(requested) + if ( + !replaced && + path.resolve(requested) === path.resolve(candidate) + ) { + replaced = true + await rm(linkedPath) + await symlink(outsideDir, linkedPath) + } + return resolved + } + } + return Reflect.get(target, prop) + }, + }) + + const reconciled = await reconcileTaskMemoryEvidence({ + memory: makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-raced-link', + path: 'linked/secret.ts', + freshnessHash: sha256('outside secret'), + }), + ], + }), + rootDir, + fs: racingFs, + }) + + expect(replaced).toBe(true) + expect(reconciled.evidence[0]!.stale).toBe(true) + } finally { + await rm(outsideDir, { recursive: true, force: true }) + } + }) + + test('small evidence uses bounded descriptor reads rather than pathname readFile', async () => { + const contents = 'small stable body' + await writeFile(path.join(rootDir, 'small.ts'), contents) + let readFileCalls = 0 + const boundedFs = new Proxy(nodeFsPromises, { + get(target, prop) { + if (prop === 'readFile') { + return async () => { + readFileCalls++ + return Buffer.alloc(2_000_000) + } + } + return Reflect.get(target, prop) + }, + }) + + const reconciled = await reconcileTaskMemoryEvidence({ + memory: makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-small-bounded', + path: 'small.ts', + freshnessHash: sha256(contents), + }), + ], + }), + rootDir, + fs: boundedFs, + }) + + expect(reconciled.evidence[0]!.stale).toBe(false) + expect(readFileCalls).toBe(0) + }) + test('saveMergedTaskMemory never throws on schema-invalid run memory', async () => { const saved = await saveMergedTaskMemory({ rootDir, @@ -363,19 +570,27 @@ describe('task-memory-store', () => { test('concurrent saves serialize: unique tmp names, distinct monotonic revisions', async () => { const writtenPaths: string[] = [] - const spyingFs = new Proxy(nodeFsPromises, { - get(target, prop) { - if (prop === 'writeFile') { - return async ( - ...args: Parameters - ) => { - writtenPaths.push(String(args[0])) - return nodeFsPromises.writeFile(...args) + const spyingFs = { + ...new Proxy(nodeFsPromises, { + get(target, prop) { + if (prop === 'writeFile') { + return async ( + ...args: Parameters + ) => { + writtenPaths.push(String(args[0])) + return nodeFsPromises.writeFile(...args) + } } - } - return Reflect.get(target, prop) + return Reflect.get(target, prop) + }, + }), + createFileExclusive: async ( + file: Parameters[0], + data: Parameters[1], + ) => { + await writeFile(file, data, { flag: 'wx', mode: 0o600 }) }, - }) + } const [savedA, savedB] = await Promise.all([ saveMergedTaskMemory({ @@ -420,6 +635,98 @@ describe('task-memory-store', () => { expect([...reloaded!.decisions].sort()).toEqual(['Save A', 'Save B']) }) + test('an old live lock fails closed without stealing or committing', async () => { + const initial = await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ decisions: ['Initial'] }), + }) + const memoryPath = path.join( + rootDir, + '.openbuff', + 'memory', + 'task-memory.json', + ) + const lockPath = `${memoryPath}.lock` + const owner = `${JSON.stringify({ + token: 'external-owner', + pid: process.pid, + createdAt: Date.now() - 60_000, + })}\n` + await writeFile(lockPath, owner, { flag: 'wx', mode: 0o600 }) + const old = new Date(Date.now() - 60_000) + await utimes(lockPath, old, old) + + const blocked = await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ decisions: ['Must not commit'] }), + }) + + expect(blocked).toBeUndefined() + expect(await readFile(lockPath, 'utf8')).toBe(owner) + const reloaded = await loadPersistedTaskMemory({ rootDir }) + expect(reloaded?.revision).toBe(initial?.revision) + expect(reloaded?.decisions).toEqual(['Initial']) + await rm(lockPath) + }) + + test('an old orphaned lock fails closed instead of risking replacement-owner deletion', async () => { + const memoryPath = path.join( + rootDir, + '.openbuff', + 'memory', + 'task-memory.json', + ) + await mkdir(path.dirname(memoryPath), { recursive: true }) + const lockPath = `${memoryPath}.lock` + await writeFile( + lockPath, + `${JSON.stringify({ + token: 'crashed-owner', + pid: 2_147_483_647, + createdAt: Date.now() - 60_000, + })}\n`, + { flag: 'wx', mode: 0o600 }, + ) + const old = new Date(Date.now() - 60_000) + await utimes(lockPath, old, old) + + const saved = await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ decisions: ['Recovered'] }), + }) + + expect(saved).toBeUndefined() + expect(await readFile(lockPath, 'utf8')).toContain('crashed-owner') + }) + + test('adapter that ignores wx cannot enter persistence without explicit exclusive create', async () => { + let renameCalls = 0 + const unsafeAdapter = { + mkdir: async () => {}, + readFile: async () => { + throw Object.assign(new Error('missing'), { code: 'ENOENT' }) + }, + rename: async () => { + renameCalls++ + }, + stat: async () => { + throw Object.assign(new Error('missing'), { code: 'ENOENT' }) + }, + unlink: async () => {}, + // Deliberately accepts and ignores the wx option. + writeFile: async () => {}, + } as unknown as import('../services/task-memory-store').TaskMemoryStoreFs + + expect( + await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory(), + fs: unsafeAdapter, + }), + ).toBeUndefined() + expect(renameCalls).toBe(0) + }) + test('missing renameFile capability degrades to a skipped save with no tmp litter', async () => { // Adapter without the optional renameFile capability (only the members // the store consumes): mirrors virtual-fs hosts that cannot rename @@ -445,9 +752,11 @@ describe('task-memory-store', () => { expect(await readdir(memoryDir)).toEqual([]) }) - test('oversized evidence hashing streams leading bytes instead of buffering the whole file', async () => { + test('large evidence uses a streamed whole-content hash and detects tail changes', async () => { const prefix = 'b'.repeat(1_000_000) - await writeFile(path.join(rootDir, 'huge.ts'), `${prefix}tail-beyond-cap`) + const original = `${prefix}original-tail` + const filePath = path.join(rootDir, 'huge.ts') + await writeFile(filePath, original) const readFileCalls: string[] = [] const spyingFs = new Proxy(nodeFsPromises, { @@ -463,33 +772,37 @@ describe('task-memory-store', () => { return Reflect.get(target, prop) }, }) - const memory = makeMemory({ evidence: [ makeEvidence({ id: 'ev-huge', path: 'huge.ts', - freshnessHash: sha256(prefix), + freshnessHash: sha256(original), }), ], }) + const reconciled = await reconcileTaskMemoryEvidence({ memory, rootDir, fs: spyingFs, }) - // Digest still honors the leading-bytes contract AND the whole-file - // buffered read was never taken on the default node fs path. expect(reconciled.evidence[0]!.stale).toBe(false) expect(readFileCalls).toEqual([]) + + await writeFile(filePath, `${prefix}changed-tail`) + const changed = await reconcileTaskMemoryEvidence({ + memory, + rootDir, + fs: spyingFs, + }) + expect(changed.evidence[0]!.stale).toBe(true) }) - test('codebuffFsToNodePromises forwards host open so adapter-backed hashing streams', async () => { + test('codebuffFsToNodePromises forwards host open for whole-content streaming', async () => { const prefix = 'c'.repeat(1_000_000) - await writeFile( - path.join(rootDir, 'huge-adapter.ts'), - `${prefix}tail-beyond-cap`, - ) + const contents = `${prefix}tail-beyond-cap` + await writeFile(path.join(rootDir, 'huge-adapter.ts'), contents) // Host carrying `open` beyond its published type — what a spread of // fs.promises / createNodeFileSystem() looks like — plus a readFile spy. @@ -517,15 +830,15 @@ describe('task-memory-store', () => { makeEvidence({ id: 'ev-huge-adapter', path: 'huge-adapter.ts', - freshnessHash: sha256(prefix), + freshnessHash: sha256(contents), }), ], }), rootDir, fs: storeFs, }) - // Digest honors the leading-bytes contract AND the buffered whole-file - // read was never taken on the adapter path either. + // The canonical whole-content digest is produced without a buffered + // read on the adapter path. expect(reconciled.evidence[0]!.stale).toBe(false) expect(readFileCalls).toEqual([]) }) @@ -562,10 +875,155 @@ describe('task-memory-store', () => { rootDir, fs: codebuffFsToNodePromises(codebuffFs), }) - // Fail-closed: without a partial-read primitive the multi-GB body is - // never buffered; the unverified entry goes stale instead. + // Fail closed without declaring the evidence stale: inability to verify is + // not proof that the file changed, and prune must preserve it. expect(readFileCalls).toEqual([]) - expect(reconciled.evidence[0]!.stale).toBe(true) + expect(reconciled.evidence[0]!.stale).not.toBe(true) + }) + + test('legacy large-file prefix hashes validate and backfill to versioned whole hashes', async () => { + const prefix = 'p'.repeat(1_000_000) + const contents = `${prefix}legacy-tail` + await writeFile(path.join(rootDir, 'legacy-large.ts'), contents) + const legacyMemory = makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-legacy-large', + path: 'legacy-large.ts', + freshnessHash: sha256(prefix), + }), + ], + }) + + const reconciled = await reconcileTaskMemoryEvidence({ + memory: legacyMemory, + rootDir, + }) + expect(reconciled.evidence[0]).toMatchObject({ + stale: false, + freshnessHash: `sha256-whole:${sha256(contents)}`, + }) + + await saveMergedTaskMemory({ rootDir, runMemory: legacyMemory }) + expect(await pruneStaleTaskMemoryEvidence({ rootDir })).toEqual({ + status: 'pruned', + removed: 0, + remaining: 1, + }) + const persisted = await loadPersistedTaskMemory({ rootDir }) + expect(persisted?.evidence[0]?.freshnessHash).toBe( + `sha256-whole:${sha256(contents)}`, + ) + }) + + test('prune preserves stale large evidence when an unversioned hash does not match', async () => { + const currentContents = `${'c'.repeat(1_000_000)}current-tail` + await writeFile(path.join(rootDir, 'legacy-algorithm.ts'), currentContents) + await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-unversioned-mismatch', + path: 'legacy-algorithm.ts', + // Above the historical boundary, a bare digest cannot say whether + // this mismatch reflects changed content or a legacy prefix + // algorithm. Even an inherited stale verdict is therefore + // insufficient authority for destructive pruning. + freshnessHash: sha256(`${'l'.repeat(1_000_000)}legacy-tail`), + stale: true, + }), + ], + }), + }) + + const persistedBeforePrune = (await loadPersistedTaskMemory({ rootDir }))! + const reconciled = await reconcileTaskMemoryEvidence({ + memory: persistedBeforePrune, + rootDir, + }) + expect(reconciled.evidence[0]?.stale).toBe(true) + + // Reconciliation truthfully reports the mismatch, but prune must not use + // an ambiguous unversioned algorithm as sole authority to delete a present + // file's evidence. + expect(await pruneStaleTaskMemoryEvidence({ rootDir })).toEqual({ + status: 'pruned', + removed: 0, + remaining: 1, + }) + expect((await loadPersistedTaskMemory({ rootDir }))?.evidence[0]?.id).toBe( + 'ev-unversioned-mismatch', + ) + }) + + test('unknown and malformed hash formats cannot turn inherited stale flags into prune authority', async () => { + await writeFile(path.join(rootDir, 'unknown-format.ts'), 'contents') + await writeFile(path.join(rootDir, 'malformed-version.ts'), 'contents') + await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-unknown-format', + path: 'unknown-format.ts', + freshnessHash: 'future-digest-format:value', + stale: true, + }), + makeEvidence({ + id: 'ev-malformed-version', + path: 'malformed-version.ts', + freshnessHash: 'sha256-whole:not-a-valid-digest', + stale: true, + }), + { + id: 'ev-pathless-legacy', + kind: 'read', + summary: 'Legacy evidence without a path', + freshnessHash: 'legacy-unknown-format', + stale: true, + }, + ], + }), + }) + + expect(await pruneStaleTaskMemoryEvidence({ rootDir })).toEqual({ + status: 'pruned', + removed: 0, + remaining: 3, + }) + expect( + (await loadPersistedTaskMemory({ rootDir }))?.evidence.map( + (item) => item.id, + ), + ).toEqual([ + 'ev-unknown-format', + 'ev-malformed-version', + 'ev-pathless-legacy', + ]) + }) + + test('an explicit versioned mismatch authorizes pruning changed present evidence', async () => { + await writeFile(path.join(rootDir, 'changed-versioned.ts'), 'new contents') + await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-versioned-mismatch', + path: 'changed-versioned.ts', + freshnessHash: `sha256-whole:${sha256('old contents')}`, + }), + ], + }), + }) + + expect(await pruneStaleTaskMemoryEvidence({ rootDir })).toEqual({ + status: 'pruned', + removed: 1, + remaining: 0, + }) + expect((await loadPersistedTaskMemory({ rootDir }))?.evidence).toEqual([]) }) test('batched reconciliation preserves order and per-item verdicts past one chunk', async () => { @@ -580,10 +1038,10 @@ describe('task-memory-store', () => { makeEvidence({ id: `ev-${index}`, path: name, - // Odd indexes carry a mismatching hash and must flip stale even - // when they land in later batches. + // Odd indexes carry a valid but mismatching whole-content hash and + // must flip stale even when they land in later batches. ...(index % 2 === 1 - ? { freshnessHash: 'mismatched' } + ? { freshnessHash: '0'.repeat(64) } : { freshnessHash: sha256(contents) }), }), ) @@ -600,29 +1058,20 @@ describe('task-memory-store', () => { }) }) - test('AC7: oversized evidence files hash only their leading bytes', async () => { - const prefix = 'a'.repeat(1_000_000) + test('AC7: the streaming boundary retains the whole-content hash contract', async () => { + const contents = 'a'.repeat(1_000_001) const memory = makeMemory({ evidence: [ makeEvidence({ id: 'ev-big', path: 'big.ts', - freshnessHash: sha256(prefix), + freshnessHash: sha256(contents), }), ], }) - await writeFile(path.join(rootDir, 'big.ts'), prefix) + await writeFile(path.join(rootDir, 'big.ts'), contents) const reconciled = await reconcileTaskMemoryEvidence({ memory, rootDir }) expect(reconciled.evidence[0]!.stale).toBe(false) - - // Documented trade-off: mutations beyond the size cap do not flip - // staleness because only the leading bytes feed the digest. - await writeFile(path.join(rootDir, 'big.ts'), `${prefix}tail-changed`) - const afterBeyondCap = await reconcileTaskMemoryEvidence({ - memory, - rootDir, - }) - expect(afterBeyondCap.evidence[0]!.stale).toBe(false) }) }) @@ -878,6 +1327,15 @@ describe('run integration gates', () => { files.delete(oldPath) files.set(newPath, contents) }, + createFileExclusive: async ( + filePath: string, + data: string | Buffer, + ) => { + if (files.has(filePath)) { + throw Object.assign(new Error('exists'), { code: 'EEXIST' }) + } + files.set(filePath, String(data)) + }, stat: async () => ({ size: 0, mode: 0o600 }), unlink: async (filePath: string) => { files.delete(filePath) @@ -1151,17 +1609,57 @@ describe('pruneStaleTaskMemoryEvidence', () => { ), ).toEqual(['ev-moved']) - // Without them the same record reconciles stale and the entry is lost — - // which is exactly why callers must pass the moves they know about. + // The successful compatibility backfill persists the rebound path and + // versioned whole-content hash. A later prune no longer needs the journal + // move to preserve the same evidence. expect(await pruneStaleTaskMemoryEvidence({ rootDir })).toEqual({ status: 'pruned', - removed: 1, - remaining: 0, + removed: 0, + remaining: 1, + }) + expect( + (await loadPersistedTaskMemory({ rootDir }))?.evidence[0], + ).toMatchObject({ + id: 'ev-moved', + path: 'nested/new.ts', + freshnessHash: `sha256-whole:${sha256('moved body')}`, }) - expect((await loadPersistedTaskMemory({ rootDir }))?.evidence).toEqual([]) }) - test('returns zero removals for a fully fresh record without rewriting', async () => { + test('pruning treats evidence through an escaping ancestor symlink as stale', async () => { + if (process.platform === 'win32') return + const outsideDir = await mkdtemp( + path.join(tmpdir(), 'task-memory-prune-outside-'), + ) + try { + const outsideContents = 'outside prune target' + await writeFile(path.join(outsideDir, 'secret.ts'), outsideContents) + await symlink(outsideDir, path.join(rootDir, 'linked')) + await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ + evidence: [ + makeEvidence({ + id: 'ev-prune-link', + path: 'linked/secret.ts', + freshnessHash: sha256(outsideContents), + }), + ], + }), + }) + + expect(await pruneStaleTaskMemoryEvidence({ rootDir })).toEqual({ + status: 'pruned', + removed: 1, + remaining: 0, + }) + expect((await loadPersistedTaskMemory({ rootDir }))?.evidence).toEqual([]) + } finally { + await rm(outsideDir, { recursive: true, force: true }) + } + }) + + test('returns zero removals for a fully fresh versioned record without rewriting', async () => { await writeFile(path.join(rootDir, 'ok.ts'), 'ok') const saved = await saveMergedTaskMemory({ rootDir, @@ -1170,7 +1668,7 @@ describe('pruneStaleTaskMemoryEvidence', () => { makeEvidence({ id: 'ev-ok', path: 'ok.ts', - freshnessHash: sha256('ok'), + freshnessHash: `sha256-whole:${sha256('ok')}`, }), ], }), @@ -1199,6 +1697,14 @@ describe('pruneStaleTaskMemoryEvidence', () => { mkdir: nodeFsPromises.mkdir.bind(nodeFsPromises), readFile: nodeFsPromises.readFile.bind(nodeFsPromises), stat: nodeFsPromises.stat.bind(nodeFsPromises), + realpath: nodeFsPromises.realpath.bind(nodeFsPromises), + createFileExclusive: async ( + file: Parameters[0], + data: Parameters[1], + ) => { + await writeFile(file, data, { flag: 'wx', mode: 0o600 }) + }, + open: nodeFsPromises.open.bind(nodeFsPromises), unlink: nodeFsPromises.unlink.bind(nodeFsPromises), writeFile: nodeFsPromises.writeFile.bind(nodeFsPromises), } as unknown as CodebuffFileSystem @@ -1235,28 +1741,48 @@ describe('pruneStaleTaskMemoryEvidence', () => { }) const before = (await loadPersistedTaskMemory({ rootDir }))! - // Simulate a concurrent save landing while reconciliation hashes evidence: - // the stat call for the (missing) evidence file is the reconcile step, so - // advance the record from there. + // Simulate a concurrent save landing after prune reads its initial record. + // Intercept only task-memory.json (never the ownership-token lock file), + // return the original bytes to prune, and publish the newer revision before + // prune's guarded reload inside the lock. + const memoryPath = path.resolve( + rootDir, + '.openbuff', + 'memory', + 'task-memory.json', + ) let advanced = false - const racingFs: typeof nodeFsPromises = new Proxy(nodeFsPromises, { - get(target, prop) { - if (prop === 'stat') { - return async (...args: Parameters) => { - if (!advanced) { - advanced = true - await saveMergedTaskMemory({ - rootDir, - runMemory: makeMemory({ decisions: ['Concurrent save'] }), - priorMemory: before, - }) + const racingFs = { + ...new Proxy(nodeFsPromises, { + get(target, prop) { + if (prop === 'readFile') { + return async ( + ...args: Parameters + ) => { + const requestedPath = path.resolve(String(args[0])) + if (!advanced && requestedPath === memoryPath) { + const original = await nodeFsPromises.readFile(...args) + advanced = true + await saveMergedTaskMemory({ + rootDir, + runMemory: makeMemory({ decisions: ['Concurrent save'] }), + priorMemory: before, + }) + return original + } + return nodeFsPromises.readFile(...args) } - return nodeFsPromises.stat(...args) } - } - return Reflect.get(target, prop) + return Reflect.get(target, prop) + }, + }), + createFileExclusive: async ( + file: Parameters[0], + data: Parameters[1], + ) => { + await writeFile(file, data, { flag: 'wx', mode: 0o600 }) }, - }) + } as import('../services/task-memory-store').TaskMemoryStoreFs const result = await pruneStaleTaskMemoryEvidence({ rootDir, diff --git a/sdk/src/index.ts b/sdk/src/index.ts index 35bad25500..bbcddeb410 100644 --- a/sdk/src/index.ts +++ b/sdk/src/index.ts @@ -71,6 +71,7 @@ export type { AgentDefinition } from '@codebuff/common/templates/initial-agents- export type { ToolName } from '@codebuff/common/tools/constants' export { codebuffFsToNodePromises, + inspectPersistedTaskMemoryV1, loadPersistedTaskMemory, pruneStaleTaskMemoryEvidence, reconcileTaskMemoryEvidence, @@ -79,6 +80,7 @@ export { export type { TaskMemoryPruneOutcome, TaskMemoryStoreFs, + TaskMemoryV1Inspection, WorkspaceMoveRecord, } from './services/task-memory-store' // The persisted-record type closure the task-memory API above is typed with. @@ -107,11 +109,15 @@ export type { export type * from './services/memory-v2/types' export { MemoryV2Coordinator } from './services/memory-v2/coordinator' export { + auditTaskMemoryV1Migration, getV1MigrationIdentity, importTaskMemoryV1, } from './services/memory-v2/v1-migration' export type { + V1MigrationAuditOutcome, + V1MigrationAuditReader, V1MigrationOutcome, + V1MigrationSourceItemCounts, V1MigrationWarningCode, } from './services/memory-v2/v1-migration' export { MemoryV2OperatorService } from './services/memory-v2/operator-service' diff --git a/sdk/src/services/memory-v2/__tests__/coordinator.test.ts b/sdk/src/services/memory-v2/__tests__/coordinator.test.ts index 5d753ae3c2..0efddef95b 100644 --- a/sdk/src/services/memory-v2/__tests__/coordinator.test.ts +++ b/sdk/src/services/memory-v2/__tests__/coordinator.test.ts @@ -50,7 +50,9 @@ class RepositoryStub implements MemoryRepositoryV2 { queryDegradation: MemoryRetrievalResult['degradation'] = { state: 'none' } appendGate: ((requestIndex: number) => Promise) | undefined - async append(request: Parameters[0]): Promise { + async append( + request: Parameters[0], + ): Promise { const parsed = MemoryAppendRequestSchema.parse(request) this.inFlight++ this.maxInFlight = Math.max(this.maxInFlight, this.inFlight) @@ -89,13 +91,17 @@ class RepositoryStub implements MemoryRepositoryV2 { request: MemoryRetrievalRequest, ): Promise> { this.queryRequests.push(request) - if (this.queryFailure === 'throw') throw new Error('sensitive backend detail') + if (this.queryFailure === 'throw') + throw new Error('sensitive backend detail') if (this.queryFailure === 'invalid') return { malformed: 'result' } as never if (this.queryFailure === 'rejected' || this.queryFailure === 'failed') { return MemoryQueryOutcomeSchema.parse({ outcome: this.queryFailure, error: { - code: this.queryFailure === 'rejected' ? 'invalid-request' : 'unavailable', + code: + this.queryFailure === 'rejected' + ? 'invalid-request' + : 'unavailable', message: 'sensitive backend detail', retryable: this.queryFailure === 'failed', }, @@ -135,17 +141,19 @@ class RepositoryStub implements MemoryRepositoryV2 { if (scripted !== undefined) return scripted as MemoryExportOutcome type ExportPage = Extract const events = this.exportTailEventId - ? ([{ - schemaVersion: 2, - eventSchemaVersion: 1, - eventType: 'session.started', - eventId: this.exportTailEventId, - projectId, - sessionId: 'session:export', - occurredAt: generatedAt, - sequence: 1, - payload: { payloadSchemaVersion: 1, startedAt: generatedAt }, - }] as unknown as ExportPage['events']) + ? ([ + { + schemaVersion: 2, + eventSchemaVersion: 1, + eventType: 'session.started', + eventId: this.exportTailEventId, + projectId, + sessionId: 'session:export', + occurredAt: generatedAt, + sequence: 1, + payload: { payloadSchemaVersion: 1, startedAt: generatedAt }, + }, + ] as unknown as ExportPage['events']) : [] return { outcome: 'page' as const, events, nextAfterEventId: null } } @@ -190,7 +198,9 @@ const taskMemory = (requirements: string[] = ['sensitive requirement']) => { ...draft, revision, updatedAt, - checksum: stableHash(JSON.stringify({ revision, updatedAt, memory: draft })), + checksum: stableHash( + JSON.stringify({ revision, updatedAt, memory: draft }), + ), } } @@ -201,24 +211,25 @@ const migrationOutcome = (params: { importedObservationCount: number sourceItemCounts?: unknown truncatedFields?: unknown -}): V1MigrationOutcome => ({ - outcome: 'imported', - revision: 7, - checksum: 'checksum:parity', - identity: 'identity:parity', - importedObservationIds: Array.from( - { length: params.importedObservationCount }, - (_, index) => `observation:parity-${index}`, - ), - omittedFields: [], - warnings: [], - ...(params.sourceItemCounts !== undefined - ? { sourceItemCounts: params.sourceItemCounts } - : {}), - ...(params.truncatedFields !== undefined - ? { truncatedFields: params.truncatedFields } - : {}), -} as unknown as V1MigrationOutcome) +}): V1MigrationOutcome => + ({ + outcome: 'imported', + revision: 7, + checksum: 'checksum:parity', + identity: 'identity:parity', + importedObservationIds: Array.from( + { length: params.importedObservationCount }, + (_, index) => `observation:parity-${index}`, + ), + omittedFields: [], + warnings: [], + ...(params.sourceItemCounts !== undefined + ? { sourceItemCounts: params.sourceItemCounts } + : {}), + ...(params.truncatedFields !== undefined + ? { truncatedFields: params.truncatedFields } + : {}), + }) as unknown as V1MigrationOutcome const emptyRetrievalResult = (): MemoryRetrievalResult => { const outcome = MemoryQueryOutcomeSchema.parse({ @@ -243,7 +254,10 @@ const emptyRetrievalResult = (): MemoryRetrievalResult => { describe('Memory V2 event factory', () => { test('derives stable, source-isolated IDs from trusted identity fields', () => { - const sessionId = deriveMemorySessionId({ projectId, userInputId: 'input:1' }) + const sessionId = deriveMemorySessionId({ + projectId, + userInputId: 'input:1', + }) const input = { projectId, sessionId, @@ -263,11 +277,22 @@ describe('MemoryV2Coordinator lifecycle', () => { test('explicit authority wins over legacy mode and json-v1 performs no V2 operations', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator({ ...config(repository, 'inject'), authority: 'json-v1' }) - await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:json-v1', query: 'none' }) + const coordinator = new MemoryV2Coordinator({ + ...config(repository, 'inject'), + authority: 'json-v1', + }) + await coordinator.prepareTurn({ + agentState: state, + trustedUserInputId: 'input:json-v1', + query: 'none', + }) expect(repository.requests).toHaveLength(0) expect(repository.queryRequests).toHaveLength(0) - expect(state.memoryAuthority).toMatchObject({ requested: 'json-v1', active: 'json-v1', fallbackOccurred: false }) + expect(state.memoryAuthority).toMatchObject({ + requested: 'json-v1', + active: 'json-v1', + fallbackOccurred: false, + }) }) test('invalid runtime authority safely selects json-v1 and performs no V2 work', async () => { @@ -321,7 +346,11 @@ describe('MemoryV2Coordinator lifecycle', () => { test('isolates fresh tasks and reuses an interrupted persisted turn', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:1', @@ -329,7 +358,11 @@ describe('MemoryV2Coordinator lifecycle', () => { }) const first = structuredClone(state.memoryV2!) - const fresh = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const fresh = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await fresh.prepareTurn({ agentState: state, trustedUserInputId: 'input:2', @@ -341,7 +374,11 @@ describe('MemoryV2Coordinator lifecycle', () => { state.memoryV2 = first const beforeResume = allEvents(repository).length const queriesBeforeResume = repository.queryRequests.length - const resumed = new MemoryV2Coordinator(config(repository), undefined, () => '2027-01-01T00:00:00.000Z') + const resumed = new MemoryV2Coordinator( + config(repository), + undefined, + () => '2027-01-01T00:00:00.000Z', + ) await resumed.prepareTurn({ agentState: state, trustedUserInputId: 'different-runtime-input', @@ -367,7 +404,11 @@ describe('MemoryV2Coordinator lifecycle', () => { revision: 9, snapshotId: 'snapshot:9', } - const inject = new MemoryV2Coordinator(config(repository, 'inject'), undefined, () => generatedAt) + const inject = new MemoryV2Coordinator( + config(repository, 'inject'), + undefined, + () => generatedAt, + ) await inject.prepareTurn({ agentState: state, trustedUserInputId: 'input:inject', @@ -387,7 +428,11 @@ describe('MemoryV2Coordinator lifecycle', () => { }) expect(state.memoryAuthority?.active).toBe('sqlite-v2-opt-in') - const shadow = new MemoryV2Coordinator(config(repository, 'shadow'), undefined, () => generatedAt) + const shadow = new MemoryV2Coordinator( + config(repository, 'shadow'), + undefined, + () => generatedAt, + ) await shadow.prepareTurn({ agentState: state, trustedUserInputId: 'input:shadow', @@ -396,7 +441,11 @@ describe('MemoryV2Coordinator lifecycle', () => { expect(state.memoryV2Context).toBeUndefined() expect(state.memoryAuthority?.active).toBe('json-v1') - const successful = new MemoryV2Coordinator(config(repository, 'inject'), undefined, () => generatedAt) + const successful = new MemoryV2Coordinator( + config(repository, 'inject'), + undefined, + () => generatedAt, + ) await successful.prepareTurn({ agentState: state, trustedUserInputId: 'input:successful', @@ -404,7 +453,11 @@ describe('MemoryV2Coordinator lifecycle', () => { }) expect(state.memoryV2Context).toBeDefined() repository.queryFailure = 'failed' - const failed = new MemoryV2Coordinator(config(repository, 'inject'), undefined, () => generatedAt) + const failed = new MemoryV2Coordinator( + config(repository, 'inject'), + undefined, + () => generatedAt, + ) await failed.prepareTurn({ agentState: state, trustedUserInputId: 'input:failed', @@ -424,35 +477,48 @@ describe('MemoryV2Coordinator lifecycle', () => { ['failed', 'query-failed'], ['throw', 'query-threw'], ['invalid', 'query-invalid-result'], - ] as const)('bounds %s query failures and keeps opt-in fail-closed', async (failure, reason) => { - const repository = new RepositoryStub() - repository.queryFailure = failure - const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository, 'inject'), undefined, () => generatedAt) + ] as const)( + 'bounds %s query failures and keeps opt-in fail-closed', + async (failure, reason) => { + const repository = new RepositoryStub() + repository.queryFailure = failure + const state = getInitialAgentState() + const coordinator = new MemoryV2Coordinator( + config(repository, 'inject'), + undefined, + () => generatedAt, + ) - await expect( - coordinator.prepareTurn({ - agentState: state, - trustedUserInputId: `input:${failure}`, - query: 'sensitive query text', - }), - ).resolves.toBeUndefined() + await expect( + coordinator.prepareTurn({ + agentState: state, + trustedUserInputId: `input:${failure}`, + query: 'sensitive query text', + }), + ).resolves.toBeUndefined() - expect(state.memoryV2Context).toBeUndefined() - expect(state.memoryAuthority).toMatchObject({ - requested: 'sqlite-v2-opt-in', - active: 'sqlite-v2-opt-in', - fallbackOccurred: false, - reason, - }) - expect(JSON.stringify(state.memoryAuthority)).not.toContain('sensitive backend detail') - }) + expect(state.memoryV2Context).toBeUndefined() + expect(state.memoryAuthority).toMatchObject({ + requested: 'sqlite-v2-opt-in', + active: 'sqlite-v2-opt-in', + fallbackOccurred: false, + reason, + }) + expect(JSON.stringify(state.memoryAuthority)).not.toContain( + 'sensitive backend detail', + ) + }, + ) test('rejects a mismatched query project and keeps opt-in fail-closed', async () => { const repository = new RepositoryStub() repository.queryProjectId = ProjectIdSchema.parse('project:other') const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository, 'inject'), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository, 'inject'), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, @@ -473,14 +539,20 @@ describe('MemoryV2Coordinator lifecycle', () => { const repository = new RepositoryStub() repository.queryDegradation = { state: 'degraded', - reasons: [{ - code: 'resource-budget', - detail: 'The bounded scan stopped at its payload budget.', - retryable: true, - }], + reasons: [ + { + code: 'resource-budget', + detail: 'The bounded scan stopped at its payload budget.', + retryable: true, + }, + ], } const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository, 'inject'), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository, 'inject'), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, @@ -503,7 +575,11 @@ describe('MemoryV2Coordinator lifecycle', () => { const repository = new RepositoryStub() repository.queryFailure = failure const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository, 'shadow'), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository, 'shadow'), + undefined, + () => generatedAt, + ) await expect( coordinator.prepareTurn({ agentState: state, @@ -520,9 +596,21 @@ describe('MemoryV2Coordinator lifecycle', () => { ) test.each([ - { classification: 'match', sourceItemCounts: { requirements: 1, evidence: 1 }, imported: 2 }, - { classification: 'v1-ahead', sourceItemCounts: { requirements: 2, evidence: 1 }, imported: 2 }, - { classification: 'v2-ahead', sourceItemCounts: { requirements: 1 }, imported: 2 }, + { + classification: 'match', + sourceItemCounts: { requirements: 1, evidence: 1 }, + imported: 2, + }, + { + classification: 'v1-ahead', + sourceItemCounts: { requirements: 2, evidence: 1 }, + imported: 2, + }, + { + classification: 'v2-ahead', + sourceItemCounts: { requirements: 1 }, + imported: 2, + }, { classification: 'unavailable', sourceItemCounts: undefined, imported: 2 }, ] as const)( 'computes $classification parity from migration source item counts', @@ -549,26 +637,42 @@ describe('MemoryV2Coordinator lifecycle', () => { ) test('extracts summed source counts and one direct safe truncated-field count', () => { - expect(extractV1MigrationExtras(migrationOutcome({ - importedObservationCount: 0, - sourceItemCounts: { requirements: 2, evidence: [1, 2] }, - truncatedFields: 4, - }))).toMatchObject({ sourceItemCount: 5, truncatedFields: 4 }) - - expect(extractV1MigrationExtras(migrationOutcome({ - importedObservationCount: 0, - truncatedFields: [4], - })).truncatedFields).toBeUndefined() - expect(extractV1MigrationExtras(migrationOutcome({ - importedObservationCount: 0, - truncatedFields: Number.MAX_SAFE_INTEGER + 1, - })).truncatedFields).toBeUndefined() + expect( + extractV1MigrationExtras( + migrationOutcome({ + importedObservationCount: 0, + sourceItemCounts: { requirements: 2, evidence: [1, 2] }, + truncatedFields: 4, + }), + ), + ).toMatchObject({ sourceItemCount: 5, truncatedFields: 4 }) + + expect( + extractV1MigrationExtras( + migrationOutcome({ + importedObservationCount: 0, + truncatedFields: [4], + }), + ).truncatedFields, + ).toBeUndefined() + expect( + extractV1MigrationExtras( + migrationOutcome({ + importedObservationCount: 0, + truncatedFields: Number.MAX_SAFE_INTEGER + 1, + }), + ).truncatedFields, + ).toBeUndefined() }) test('rebases retryable append conflicts and retries the same deterministic batch at most twice', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:conflict-retry', @@ -604,17 +708,27 @@ describe('MemoryV2Coordinator lifecycle', () => { kind: 'event', eventId: repository.exportTailEventId, }) - expect(attempts.every((attempt) => attempt.expectedTail !== undefined)).toBe(true) - expect(attempts.every((attempt) => attempt.expectedLastEventId === undefined)).toBe(true) + expect( + attempts.every((attempt) => attempt.expectedTail !== undefined), + ).toBe(true) + expect( + attempts.every((attempt) => attempt.expectedLastEventId === undefined), + ).toBe(true) expect(attempts[1]!.events).toEqual(attempts[0]!.events) expect(attempts[2]!.events).toEqual(attempts[0]!.events) - expect(state.memoryV2!.lastEventId).toBe(attempts[2]!.events.at(-1)!.eventId) + expect(state.memoryV2!.lastEventId).toBe( + attempts[2]!.events.at(-1)!.eventId, + ) }) test('serializes concurrent captures in invocation order with exact cursor chaining through terminal append', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:gated', @@ -631,12 +745,20 @@ describe('MemoryV2Coordinator lifecycle', () => { } const first = coordinator.recordToolObservation({ - toolName: 'get_build_targets', callId: 'call:first', userInputId: 'input:gated', - input: { files: ['src/first.ts'] }, output: [{ type: 'json', value: { targets: ['first'] } }], native: true, + toolName: 'get_build_targets', + callId: 'call:first', + userInputId: 'input:gated', + input: { files: ['src/first.ts'] }, + output: [{ type: 'json', value: { targets: ['first'] } }], + native: true, }) const second = coordinator.recordToolObservation({ - toolName: 'get_affected_tests', callId: 'call:second', userInputId: 'input:gated', - input: { files: ['src/second.ts'] }, output: [{ type: 'json', value: { tests: ['second.test.ts'] } }], native: true, + toolName: 'get_affected_tests', + callId: 'call:second', + userInputId: 'input:gated', + input: { files: ['src/second.ts'] }, + output: [{ type: 'json', value: { tests: ['second.test.ts'] } }], + native: true, }) const terminal = coordinator.finishTurn({ agentState: state, @@ -678,7 +800,11 @@ describe('MemoryV2Coordinator lifecycle', () => { test('serializes concurrent captures, advances cursors, and finishes once', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:1', @@ -715,15 +841,29 @@ describe('MemoryV2Coordinator lifecycle', () => { expect(repository.maxInFlight).toBe(1) expect(allEvents(repository).length).toBe(count) expect(state.memoryV2!.turn.status).toBe('completed') - expect(state.memoryV2!.lastEventId).toBe(repository.requests.at(-1)!.events.at(-1)!.eventId) - expect(repository.requests.every((request) => request.expectedTail !== undefined)).toBe(true) - expect(repository.requests.every((request) => request.expectedLastEventId === undefined)).toBe(true) + expect(state.memoryV2!.lastEventId).toBe( + repository.requests.at(-1)!.events.at(-1)!.eventId, + ) + expect( + repository.requests.every( + (request) => request.expectedTail !== undefined, + ), + ).toBe(true) + expect( + repository.requests.every( + (request) => request.expectedLastEventId === undefined, + ), + ).toBe(true) }) test('parks a failed terminal append and later commits the unchanged decision', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:terminal-retry', @@ -785,28 +925,39 @@ describe('MemoryV2Coordinator lifecycle', () => { signal: AbortSignal.abort(), expected: 'cancelled' as const, }, - ])('records $name terminal lifecycle status', async ({ output, signal, expected }) => { - const repository = new RepositoryStub() - const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) - await coordinator.prepareTurn({ - agentState: state, - trustedUserInputId: `input:${expected}`, - query: expected, - }) - await coordinator.finishTurn({ agentState: state, output, signal }) + ])( + 'records $name terminal lifecycle status', + async ({ output, signal, expected }) => { + const repository = new RepositoryStub() + const state = getInitialAgentState() + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) + await coordinator.prepareTurn({ + agentState: state, + trustedUserInputId: `input:${expected}`, + query: expected, + }) + await coordinator.finishTurn({ agentState: state, output, signal }) - expect(state.memoryV2!.turn.status).toBe(expected) - expect(allEvents(repository).at(-1)).toMatchObject({ - eventType: 'session.ended', - payload: { status: expected }, - }) - }) + expect(state.memoryV2!.turn.status).toBe(expected) + expect(allEvents(repository).at(-1)).toMatchObject({ + eventType: 'session.ended', + payload: { status: expected }, + }) + }, + ) test('captures only recognized successful native structured metadata with bounded paths', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:1', @@ -829,9 +980,9 @@ describe('MemoryV2Coordinator lifecycle', () => { { kind: 'file', path: 'src/good.ts' }, ]) expect(observationEvent.payload.observation.evidence).toEqual([]) - expect(observationEvent.payload.observation.provenance?.metadata.outputDigest).toMatch( - /^sha256:[a-f0-9]{64}$/, - ) + expect( + observationEvent.payload.observation.provenance?.metadata.outputDigest, + ).toMatch(/^sha256:[a-f0-9]{64}$/) } for (const attempt of [ @@ -868,32 +1019,64 @@ describe('MemoryV2Coordinator lifecycle', () => { test('filters private memory, generated, dependency, binary, secret, log, and build paths', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) - await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:policy', query: 'capture' }) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) + await coordinator.prepareTurn({ + agentState: state, + trustedUserInputId: 'input:policy', + query: 'capture', + }) const before = allEvents(repository).length await coordinator.recordToolObservation({ - toolName: 'code_search', callId: 'call:policy', userInputId: 'input:policy', - input: { files: ['.openbuff/memory/export.json', 'node_modules/pkg/index.js', 'generated/client.ts', 'assets/logo.png', '.env', 'logs/agent.log', 'dist/index.js'] }, - output: [{ type: 'json', value: { matches: [] } }], native: true, + toolName: 'code_search', + callId: 'call:policy', + userInputId: 'input:policy', + input: { + files: [ + '.openbuff/memory/export.json', + 'node_modules/pkg/index.js', + 'generated/client.ts', + 'assets/logo.png', + '.env', + 'logs/agent.log', + 'dist/index.js', + ], + }, + output: [{ type: 'json', value: { matches: [] } }], + native: true, }) expect(allEvents(repository)).toHaveLength(before) await coordinator.recordToolObservation({ - toolName: 'code_search', callId: 'call:mixed-policy', userInputId: 'input:policy', + toolName: 'code_search', + callId: 'call:mixed-policy', + userInputId: 'input:policy', input: { files: ['src/good.ts', '.openbuff/backups/memory.json'] }, - output: [{ type: 'json', value: { matches: [] } }], native: true, + output: [{ type: 'json', value: { matches: [] } }], + native: true, }) const event = allEvents(repository).at(-1)! expect(event.eventType).toBe('observation.recorded') if (event.eventType === 'observation.recorded') { - expect(event.payload.observation.selectors).toEqual([{ kind: 'file', path: 'src/good.ts' }]) - expect(event.payload.observation.detail).toContain('1 project-relative path') + expect(event.payload.observation.selectors).toEqual([ + { kind: 'file', path: 'src/good.ts' }, + ]) + expect(event.payload.observation.detail).toContain( + '1 project-relative path', + ) } }) test('captures only confirmed mutation actions', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:1', @@ -921,7 +1104,8 @@ describe('MemoryV2Coordinator lifecycle', () => { path: 'src/new.ts', outcome: 'applied', beforeHash: null, - afterHash: 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', + afterHash: + 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', }, ], authorityTier: 'portable_path', @@ -942,7 +1126,8 @@ describe('MemoryV2Coordinator lifecycle', () => { path: 'src/new.ts', status: 'committed', beforeHash: null, - afterHash: 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', + afterHash: + 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', }, ], finalHashes: { @@ -964,7 +1149,8 @@ describe('MemoryV2Coordinator lifecycle', () => { action: 'create', path: 'src/new.ts', beforeHash: null, - afterHash: 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', + afterHash: + 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', }, ]) } @@ -984,23 +1170,44 @@ describe('MemoryV2Coordinator lifecycle', () => { version: 1, operationId: 'operation:private', outcome: 'applied', - actions: [{ - actionId: 'action:private', index: 0, action: 'create', - path: '.openbuff/memory/export.json', outcome: 'applied', beforeHash: null, - afterHash: 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', - }], + actions: [ + { + actionId: 'action:private', + index: 0, + action: 'create', + path: '.openbuff/memory/export.json', + outcome: 'applied', + beforeHash: null, + afterHash: + 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', + }, + ], authorityTier: 'portable_path', receiptId: 'receipt:private', authorityReceipt: { - kind: 'commit_receipt', version: 1, receiptId: 'receipt:private', - operationId: 'operation:private', callId: 'call:private-write', - authorityTier: 'portable_path', status: 'committed', - actions: [{ - actionId: 'action:private', index: 0, action: 'create', - path: '.openbuff/memory/export.json', status: 'committed', beforeHash: null, - afterHash: 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', - }], - finalHashes: { '.openbuff/memory/export.json': 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' }, + kind: 'commit_receipt', + version: 1, + receiptId: 'receipt:private', + operationId: 'operation:private', + callId: 'call:private-write', + authorityTier: 'portable_path', + status: 'committed', + actions: [ + { + actionId: 'action:private', + index: 0, + action: 'create', + path: '.openbuff/memory/export.json', + status: 'committed', + beforeHash: null, + afterHash: + 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', + }, + ], + finalHashes: { + '.openbuff/memory/export.json': + 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', + }, }, errors: [], freshCapabilities: [], @@ -1018,24 +1225,38 @@ describe('MemoryV2Coordinator lifecycle', () => { const logger = { warn: (fields: unknown) => warnings.push(fields) } const shadowState = getInitialAgentState() - const shadow = new MemoryV2Coordinator(config(repository, 'shadow'), logger, () => generatedAt) - await expect(shadow.prepareTurn({ - agentState: shadowState, - trustedUserInputId: 'input:shadow-failure', - query: 'continue despite failure', - })).resolves.toBeUndefined() + const shadow = new MemoryV2Coordinator( + config(repository, 'shadow'), + logger, + () => generatedAt, + ) + await expect( + shadow.prepareTurn({ + agentState: shadowState, + trustedUserInputId: 'input:shadow-failure', + query: 'continue despite failure', + }), + ).resolves.toBeUndefined() expect(shadowState.memoryAuthority).toMatchObject({ - requested: 'shadow-v2', active: 'json-v1', fallbackOccurred: false, + requested: 'shadow-v2', + active: 'json-v1', + fallbackOccurred: false, reason: 'lifecycle-append-failed', }) const optInState = getInitialAgentState() - const optIn = new MemoryV2Coordinator(config(repository, 'inject'), logger, () => generatedAt) - await expect(optIn.prepareTurn({ - agentState: optInState, - trustedUserInputId: 'input:opt-in-failure', - query: 'fall back', - })).resolves.toBeUndefined() + const optIn = new MemoryV2Coordinator( + config(repository, 'inject'), + logger, + () => generatedAt, + ) + await expect( + optIn.prepareTurn({ + agentState: optInState, + trustedUserInputId: 'input:opt-in-failure', + query: 'fall back', + }), + ).resolves.toBeUndefined() expect(optInState.memoryV2!.lastEventId).toBeUndefined() expect(optInState.memoryV2Context).toBeUndefined() expect(optInState.memoryAuthority).toMatchObject({ @@ -1056,7 +1277,11 @@ describe('MemoryV2Coordinator lifecycle', () => { }) const state = getInitialAgentState() state.taskMemory = taskMemory() - const coordinator = new MemoryV2Coordinator(config(repository, 'inject'), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository, 'inject'), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, @@ -1068,22 +1293,37 @@ describe('MemoryV2Coordinator lifecycle', () => { expect(state.memoryAuthority?.parity).toBeUndefined() expect(state.memoryAuthority).toMatchObject({ active: 'sqlite-v2-opt-in', - reason: 'query-invalid-result', + reason: 'migration-failed', }) }) test('replays persisted terminal data before a new lifecycle', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const first = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) - await first.prepareTurn({ agentState: state, trustedUserInputId: 'input:first', query: 'first' }) + const first = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) + await first.prepareTurn({ + agentState: state, + trustedUserInputId: 'input:first', + query: 'first', + }) repository.failNextAppend = true - await first.finishTurn({ agentState: state, output: { type: 'structuredOutput', value: {} } }) + await first.finishTurn({ + agentState: state, + output: { type: 'structuredOutput', value: {} }, + }) const failedBatch = structuredClone(repository.requests.at(-1)!.events) const checkpoint = structuredClone(state) const before = repository.requests.length - const second = new MemoryV2Coordinator(config(repository), undefined, () => '2027-01-01T00:00:00.000Z') + const second = new MemoryV2Coordinator( + config(repository), + undefined, + () => '2027-01-01T00:00:00.000Z', + ) await second.prepareTurn({ agentState: checkpoint, trustedUserInputId: 'input:second', @@ -1091,14 +1331,20 @@ describe('MemoryV2Coordinator lifecycle', () => { }) expect(repository.requests[before]!.events).toEqual(failedBatch) - expect(repository.requests[before + 1]!.events[0]!.eventType).toBe('session.started') + expect(repository.requests[before + 1]!.events[0]!.eventType).toBe( + 'session.started', + ) expect(checkpoint.memoryV2!.pendingTerminal).toBeUndefined() }) test('emits coverage.recorded events from evaluate_audit_coverage results', async () => { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:coverage', @@ -1110,13 +1356,28 @@ describe('MemoryV2Coordinator lifecycle', () => { callId: 'call:coverage', userInputId: 'input:coverage', input: {}, - output: [{ type: 'json', value: { status: 'complete', features: [{ feature: 'auth' }] } }], + output: [ + { + type: 'json', + value: { status: 'complete', features: [{ feature: 'auth' }] }, + }, + ], native: true, - workspaceState: { schemaVersion: 1 as const, revision: 5, snapshotId: 'snap:5', updatedAt: Date.now(), changes: [] }, + workspaceState: { + schemaVersion: 1 as const, + revision: 5, + snapshotId: 'snap:5', + updatedAt: Date.now(), + changes: [], + }, }) const newEvents = allEvents(repository).slice(before) - const observationEvents = newEvents.filter((e) => e.eventType === 'observation.recorded') - const coverageEvents = newEvents.filter((e) => e.eventType === 'coverage.recorded') + const observationEvents = newEvents.filter( + (e) => e.eventType === 'observation.recorded', + ) + const coverageEvents = newEvents.filter( + (e) => e.eventType === 'coverage.recorded', + ) expect(observationEvents).toHaveLength(1) expect(coverageEvents).toHaveLength(1) const coveragePayload = coverageEvents[0]!.payload @@ -1133,7 +1394,10 @@ describe('MemoryV2Coordinator lifecycle', () => { test('fails closed when conflict tail export cannot produce a valid page', async () => { const failures: Array = [ - { outcome: 'failed', error: { code: 'unavailable', message: 'no', retryable: true } }, + { + outcome: 'failed', + error: { code: 'unavailable', message: 'no', retryable: true }, + }, { malformed: true }, new Error('export threw'), { outcome: 'page', events: [], nextAfterEventId: 'event:next' }, @@ -1141,14 +1405,26 @@ describe('MemoryV2Coordinator lifecycle', () => { for (const scripted of failures) { const repository = new RepositoryStub() const state = getInitialAgentState() - const coordinator = new MemoryV2Coordinator(config(repository), undefined, () => generatedAt) - await coordinator.prepareTurn({ agentState: state, trustedUserInputId: 'input:tail-failure', query: 'tail' }) + const coordinator = new MemoryV2Coordinator( + config(repository), + undefined, + () => generatedAt, + ) + await coordinator.prepareTurn({ + agentState: state, + trustedUserInputId: 'input:tail-failure', + query: 'tail', + }) repository.appendConflictsRemaining = 1 repository.exportScript = [scripted] const before = repository.requests.length await coordinator.recordToolObservation({ - toolName: 'get_build_targets', callId: 'call:tail-failure', userInputId: 'input:tail-failure', - input: { files: ['src/a.ts'] }, output: [{ type: 'json', value: { targets: ['sdk'] } }], native: true, + toolName: 'get_build_targets', + callId: 'call:tail-failure', + userInputId: 'input:tail-failure', + input: { files: ['src/a.ts'] }, + output: [{ type: 'json', value: { targets: ['sdk'] } }], + native: true, }) expect(repository.requests.slice(before)).toHaveLength(1) expect(repository.requests.at(-1)!.expectedTail).toBeDefined() diff --git a/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts b/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts index 2b0bbece4e..eb1e884176 100644 --- a/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts +++ b/sdk/src/services/memory-v2/__tests__/v1-migration.test.ts @@ -2,8 +2,11 @@ import { describe, expect, test } from 'bun:test' import { MemoryAppendRequestSchema, + MemoryEventEnvelopeSchema, + MemoryEventIdSchema, MemorySessionIdSchema, ProjectIdSchema, + TaskIdSchema, type MemoryEventEnvelope, } from '@codebuff/common/types/memory-v2' import { @@ -12,7 +15,12 @@ import { } from '@codebuff/common/types/task-memory' import { stableHash } from '@codebuff/common/util/stable-hash' -import { getV1MigrationIdentity, importTaskMemoryV1 } from '../v1-migration' +import { + auditTaskMemoryV1Migration, + getV1MigrationIdentity, + importTaskMemoryV1, + type V1MigrationAuditReader, +} from '../v1-migration' import type { MemoryRepositoryV2 } from '../types' const projectId = ProjectIdSchema.parse('project:migration-test') @@ -32,9 +40,27 @@ const defaultDraft = { nextActions: ['next'], historicalSummary: 'history', evidence: [ - { id: 'fresh', kind: 'read' as const, summary: 'fresh', path: 'src/live.ts', stale: false }, - { id: 'stale', kind: 'read' as const, summary: 'stale', path: 'src/stale.ts', stale: true }, - { id: 'private', kind: 'read' as const, summary: 'private', path: '.env', stale: false }, + { + id: 'fresh', + kind: 'read' as const, + summary: 'fresh', + path: 'src/live.ts', + stale: false, + }, + { + id: 'stale', + kind: 'read' as const, + summary: 'stale', + path: 'src/stale.ts', + stale: true, + }, + { + id: 'private', + kind: 'read' as const, + summary: 'private', + path: '.env', + stale: false, + }, ], } @@ -45,10 +71,15 @@ function memory(overrides: Partial = {}): TaskMemoryV1 { checksum: checksumOverride, ...draftOverrides } = overrides - const draft = taskMemoryDraftV1Schema.parse({ ...defaultDraft, ...draftOverrides }) + const draft = taskMemoryDraftV1Schema.parse({ + ...defaultDraft, + ...draftOverrides, + }) const checksum = checksumOverride ?? - stableHash(JSON.stringify({ revision, updatedAt: sourceUpdatedAt, memory: draft })) + stableHash( + JSON.stringify({ revision, updatedAt: sourceUpdatedAt, memory: draft }), + ) return { ...draft, revision, updatedAt: sourceUpdatedAt, checksum } } @@ -56,8 +87,10 @@ class Repository implements MemoryRepositoryV2 { events = new Map() sequence = 0 appendCalls = 0 + exportCalls = 0 failAppendCall: number | undefined failAppendAsConflict = false + beforeFailedAppend?: (repository: Repository) => void appendRequests: Array> = [] async append(input: Parameters[0]) { @@ -65,35 +98,53 @@ class Repository implements MemoryRepositoryV2 { this.appendCalls++ this.appendRequests.push(request) if (this.appendCalls === this.failAppendCall) { + this.beforeFailedAppend?.(this) return this.failAppendAsConflict ? { outcome: 'rejected' as const, - error: { code: 'conflict' as const, message: 'cursor conflict', retryable: true }, + error: { + code: 'conflict' as const, + message: 'cursor conflict', + retryable: true, + }, } : { outcome: 'failed' as const, - error: { code: 'unavailable' as const, message: 'offline', retryable: true }, + error: { + code: 'unavailable' as const, + message: 'offline', + retryable: true, + }, } } const canonicalTail = [...this.events.values()].at(-1)?.eventId - const expectedTail = request.expectedTail ?? ( - request.expectedLastEventId === undefined + const expectedTail = + request.expectedTail ?? + (request.expectedLastEventId === undefined ? { kind: 'any' as const } - : { kind: 'event' as const, eventId: request.expectedLastEventId } - ) - const tailMatches = expectedTail.kind === 'any' - || (expectedTail.kind === 'empty' && canonicalTail === undefined) - || (expectedTail.kind === 'event' && expectedTail.eventId === canonicalTail) + : { kind: 'event' as const, eventId: request.expectedLastEventId }) + const tailMatches = + expectedTail.kind === 'any' || + (expectedTail.kind === 'empty' && canonicalTail === undefined) || + (expectedTail.kind === 'event' && expectedTail.eventId === canonicalTail) if (!tailMatches) { return { outcome: 'rejected' as const, - error: { code: 'conflict' as const, message: 'cursor conflict', retryable: true }, + error: { + code: 'conflict' as const, + message: 'cursor conflict', + retryable: true, + }, } } const staged = new Map(this.events) - const entries: Array<{ eventId: MemoryEventEnvelope['eventId']; sequence: number; duplicate: boolean }> = [] + const entries: Array<{ + eventId: MemoryEventEnvelope['eventId'] + sequence: number + duplicate: boolean + }> = [] let sequence = this.sequence for (const event of request.events) { const existing = staged.get(event.eventId) @@ -102,10 +153,18 @@ class Repository implements MemoryRepositoryV2 { if (JSON.stringify(existingDraft) !== JSON.stringify(event)) { return { outcome: 'rejected' as const, - error: { code: 'conflict' as const, message: 'event content conflict', retryable: false }, + error: { + code: 'conflict' as const, + message: 'event content conflict', + retryable: false, + }, } } - entries.push({ eventId: event.eventId, sequence: existing.sequence, duplicate: true }) + entries.push({ + eventId: event.eventId, + sequence: existing.sequence, + duplicate: true, + }) continue } const envelope = { ...event, sequence: ++sequence } as MemoryEventEnvelope @@ -122,6 +181,7 @@ class Repository implements MemoryRepositoryV2 { } async export(input: Parameters[0]) { + this.exportCalls++ const events = [...this.events.values()] const afterIndex = input.afterEventId ? events.findIndex((event) => event.eventId === input.afterEventId) @@ -152,6 +212,15 @@ class Repository implements MemoryRepositoryV2 { const run = (repository: Repository, value?: TaskMemoryV1) => importTaskMemoryV1({ memory: value, projectId, sessionId, repository }) +const audit = (repository: V1MigrationAuditReader, value?: TaskMemoryV1) => + auditTaskMemoryV1Migration({ memory: value, projectId, repository }) + +const cloneEvent = ( + event: MemoryEventEnvelope, + overrides: Partial, +): MemoryEventEnvelope => + MemoryEventEnvelopeSchema.parse({ ...event, ...overrides }) + const migrationReservations = (repository: Repository) => [...repository.events.values()].filter( (event) => event.eventType === 'migration.v1.reserved', @@ -197,7 +266,9 @@ describe('V1 memory migration', () => { ]), ) expect(JSON.stringify(events)).not.toContain(source.goal) - expect(events.some((event) => event.eventType === 'evidence.verified')).toBe(false) + expect( + events.some((event) => event.eventType === 'evidence.verified'), + ).toBe(false) expect(JSON.stringify(events)).not.toContain('src/stale.ts') expect(JSON.stringify(events)).not.toContain('.env') @@ -228,7 +299,10 @@ describe('V1 memory migration', () => { const marker = migrationMarkers(repository)[0] expect(marker?.eventType).toBe('migration.v1.imported') - if (marker?.eventType === 'migration.v1.imported' && outcome.outcome === 'imported') { + if ( + marker?.eventType === 'migration.v1.imported' && + outcome.outcome === 'imported' + ) { expect(marker.payload.sourceItemCounts).toEqual(outcome.sourceItemCounts) expect(marker.payload.truncatedFields).toBe(0) expect(marker.payload.omittedFields).toBe(outcome.omittedFields) @@ -258,18 +332,228 @@ describe('V1 memory migration', () => { const repeated = await run(repository, first) expect(repeated.outcome).toBe('no-op') if (repeated.outcome === 'no-op' && imported.outcome === 'imported') { - expect(repeated.importedObservationIds).toEqual(imported.importedObservationIds) + expect(repeated.importedObservationIds).toEqual( + imported.importedObservationIds, + ) + expect(repeated.sourceItemCounts).toEqual(imported.sourceItemCounts) + expect(repeated.truncatedFields).toBe(imported.truncatedFields) + expect(repeated.lastEventId).toBe( + [...repository.events.values()].at(-1)!.eventId, + ) + } + expect(repository.events.size).toBe(eventCount) + }) + + test('binds an ordinary exact-repeat no-op to the marker whose complete body was audited', async () => { + const repository = new Repository() + const source = memory() + const imported = await run(repository, source) + expect(imported.outcome).toBe('imported') + const canonicalMarker = migrationMarkers(repository)[0]! + if ( + canonicalMarker.eventType !== 'migration.v1.imported' || + imported.outcome !== 'imported' + ) + return + + const forgedMarker = MemoryEventEnvelopeSchema.parse({ + ...canonicalMarker, + eventId: 'event:later-forged-exact-source-marker', + payload: { + ...canonicalMarker.payload, + importedTaskId: 'task:forged', + importedObservationIds: [], + omittedFields: 99, + sourceItemCounts: { requirements: 99 }, + truncatedFields: 99, + warnings: [], + }, + sequence: repository.sequence + 1, + }) + repository.events.set(forgedMarker.eventId, forgedMarker) + repository.sequence = forgedMarker.sequence + const eventCount = repository.events.size + + const repeated = await run(repository, source) + + expect(repeated.outcome).toBe('no-op') + if (repeated.outcome === 'no-op') { + expect(repeated.importedTaskId).toBe(imported.importedTaskId) + expect(repeated.importedObservationIds).toEqual( + imported.importedObservationIds, + ) + expect(repeated.omittedFields).toBe(imported.omittedFields) expect(repeated.sourceItemCounts).toEqual(imported.sourceItemCounts) expect(repeated.truncatedFields).toBe(imported.truncatedFields) - expect(repeated.lastEventId).toBe([...repository.events.values()].at(-1)!.eventId) + expect(repeated.importedTaskId).not.toBe('task:forged') } expect(repository.events.size).toBe(eventCount) }) + test('repairs missing body evidence despite a matching persisted marker', async () => { + const repository = new Repository() + const source = memory() + expect((await run(repository, source)).outcome).toBe('imported') + const missing = [...repository.events.values()].find( + (event) => event.eventType === 'task.created', + )! + repository.events.delete(missing.eventId) + + expect(await run(repository, source)).toMatchObject({ outcome: 'imported' }) + expect(await audit(repository, source)).toMatchObject({ outcome: 'exact' }) + const eventCount = repository.events.size + expect(await run(repository, source)).toMatchObject({ outcome: 'no-op' }) + expect(repository.events.size).toBe(eventCount) + expect(migrationMarkers(repository)).toHaveLength(2) + }) + + test('repairs a tampered marker even when its deterministic body remains complete', async () => { + const repository = new Repository() + const source = memory() + expect((await run(repository, source)).outcome).toBe('imported') + const marker = migrationMarkers(repository)[0]! + if (marker.eventType !== 'migration.v1.imported') return + repository.events.set( + marker.eventId, + MemoryEventEnvelopeSchema.parse({ + ...marker, + payload: { ...marker.payload, omittedFields: 99 }, + }), + ) + + const repaired = await run(repository, source) + + expect(repaired).toMatchObject({ outcome: 'imported' }) + expect(await audit(repository, source)).toMatchObject({ outcome: 'exact' }) + expect(migrationMarkers(repository)).toHaveLength(2) + }) + + test('repairs incompatible deterministic body evidence with new event IDs', async () => { + const repository = new Repository() + const source = memory() + expect((await run(repository, source)).outcome).toBe('imported') + const original = [...repository.events.values()].find( + (event) => event.eventType === 'observation.recorded', + )! + if (original.eventType !== 'observation.recorded') return + repository.events.set( + original.eventId, + MemoryEventEnvelopeSchema.parse({ + ...original, + payload: { + ...original.payload, + observation: { + ...original.payload.observation, + summary: 'older incompatible representation', + }, + }, + }), + ) + + expect(await run(repository, source)).toMatchObject({ outcome: 'imported' }) + expect(await audit(repository, source)).toMatchObject({ outcome: 'exact' }) + expect(migrationMarkers(repository)).toHaveLength(2) + }) + + test('fails closed when a resumed deterministic event ID has corrupt content', async () => { + const repository = new Repository() + const source = memory() + repository.failAppendCall = 3 + expect(await run(repository, source)).toMatchObject({ outcome: 'failed' }) + const task = [...repository.events.values()].find( + (event) => event.eventType === 'task.created', + )! + repository.events.set( + task.eventId, + MemoryEventEnvelopeSchema.parse({ + ...task, + payload: { ...task.payload, title: 'non-migration data' }, + }), + ) + repository.failAppendCall = undefined + const eventCount = repository.events.size + + expect(await run(repository, source)).toMatchObject({ + outcome: 'failed', + reason: 'repository-failed', + }) + expect(repository.events.size).toBe(eventCount) + expect(migrationMarkers(repository)).toHaveLength(0) + }) + + test('does not accept an incomplete exact marker discovered during conflict recovery', async () => { + const repository = new Repository() + const source = memory() + const concurrent = new Repository() + expect((await run(concurrent, source)).outcome).toBe('imported') + const concurrentMarker = migrationMarkers(concurrent)[0]! + repository.failAppendCall = 1 + repository.failAppendAsConflict = true + repository.beforeFailedAppend = (target) => { + target.events.set(concurrentMarker.eventId, concurrentMarker) + target.sequence = concurrentMarker.sequence + } + + expect(await run(repository, source)).toMatchObject({ + outcome: 'failed', + reason: 'repository-failed', + }) + expect( + [...repository.events.values()].filter( + (event) => + event.eventType === 'task.created' || + event.eventType === 'observation.recorded', + ), + ).toHaveLength(0) + }) + + test('binds conflict recovery no-op fields to the validated complete marker', async () => { + const repository = new Repository() + const source = memory() + const concurrent = new Repository() + const imported = await run(concurrent, source) + expect(imported.outcome).toBe('imported') + const canonicalMarker = migrationMarkers(concurrent)[0]! + if (canonicalMarker.eventType !== 'migration.v1.imported') return + const forgedMarker = MemoryEventEnvelopeSchema.parse({ + ...canonicalMarker, + eventId: 'event:concurrent-forged-marker', + payload: { + ...canonicalMarker.payload, + importedTaskId: 'task:forged', + importedObservationIds: [], + }, + sequence: concurrent.sequence + 1, + }) + + repository.failAppendCall = 1 + repository.failAppendAsConflict = true + repository.beforeFailedAppend = (target) => { + for (const event of concurrent.events.values()) { + target.events.set(event.eventId, event) + } + target.events.set(forgedMarker.eventId, forgedMarker) + target.sequence = forgedMarker.sequence + } + + const recovered = await run(repository, source) + expect(recovered.outcome).toBe('no-op') + if (recovered.outcome === 'no-op' && imported.outcome === 'imported') { + expect(recovered.importedTaskId).toBe(imported.importedTaskId) + expect(recovered.importedObservationIds).toEqual( + imported.importedObservationIds, + ) + expect(recovered.importedTaskId).not.toBe('task:forged') + } + }) + test('records all bounded text truncation in the marker and outcome', async () => { const repository = new Repository() const source = memory({ - requirements: ['x'.repeat(1_100), ...Array.from({ length: 17 }, () => 'y'.repeat(1_000))], + requirements: [ + 'x'.repeat(1_100), + ...Array.from({ length: 17 }, () => 'y'.repeat(1_000)), + ], historicalSummary: 'h'.repeat(1_100), evidence: [ { @@ -301,7 +585,9 @@ describe('V1 memory migration', () => { } for (const event of repository.events.values()) { if (event.eventType === 'observation.recorded') { - expect(event.payload.observation.detail.length).toBeLessThanOrEqual(16_384) + expect(event.payload.observation.detail.length).toBeLessThanOrEqual( + 16_384, + ) } } }) @@ -320,7 +606,9 @@ describe('V1 memory migration', () => { expect(forgotten).toHaveLength(first.importedObservationIds.length) expect( forgotten.flatMap((event) => - event.eventType === 'claim.forgotten' ? event.payload.observationIds : [], + event.eventType === 'claim.forgotten' + ? event.payload.observationIds + : [], ), ).toEqual(first.importedObservationIds) for (const event of forgotten) { @@ -349,6 +637,108 @@ describe('V1 memory migration', () => { } }) + test('does not retire claims from a lower-revision marker with forged observation references', async () => { + const repository = new Repository() + expect((await run(repository, memory())).outcome).toBe('imported') + const marker = migrationMarkers(repository)[0]! + if (marker.eventType !== 'migration.v1.imported') return + repository.events.set( + marker.eventId, + MemoryEventEnvelopeSchema.parse({ + ...marker, + payload: { + ...marker.payload, + importedObservationIds: ['observation:user-owned'], + }, + }), + ) + + expect((await run(repository, memory({ revision: 4 }))).outcome).toBe( + 'imported', + ) + expect( + [...repository.events.values()].filter( + (event) => event.eventType === 'claim.forgotten', + ), + ).toEqual([]) + }) + + test('does not retire claims from a lower-revision marker without deterministic migration identity', async () => { + const repository = new Repository() + expect((await run(repository, memory())).outcome).toBe('imported') + const marker = migrationMarkers(repository)[0]! + if (marker.eventType !== 'migration.v1.imported') return + repository.events.set( + marker.eventId, + MemoryEventEnvelopeSchema.parse({ + ...marker, + payload: { + ...marker.payload, + legacyRecordKey: 'v1:forged-lower-revision-marker', + }, + }), + ) + + expect((await run(repository, memory({ revision: 4 }))).outcome).toBe( + 'imported', + ) + expect( + [...repository.events.values()].filter( + (event) => event.eventType === 'claim.forgotten', + ), + ).toEqual([]) + }) + + test('does not retire claims when a lower-revision marker has no migration reservation', async () => { + const repository = new Repository() + expect((await run(repository, memory())).outcome).toBe('imported') + const reservation = migrationReservations(repository)[0]! + repository.events.delete(reservation.eventId) + + expect((await run(repository, memory({ revision: 4 }))).outcome).toBe( + 'imported', + ) + expect( + [...repository.events.values()].filter( + (event) => event.eventType === 'claim.forgotten', + ), + ).toEqual([]) + }) + + test('does not retire claims when a referenced observation is not migration-owned', async () => { + const repository = new Repository() + expect((await run(repository, memory())).outcome).toBe('imported') + const observation = [...repository.events.values()].find( + (event) => event.eventType === 'observation.recorded', + )! + if (observation.eventType !== 'observation.recorded') return + repository.events.set( + observation.eventId, + MemoryEventEnvelopeSchema.parse({ + ...observation, + payload: { + ...observation.payload, + observation: { + ...observation.payload.observation, + provenance: { + ...observation.payload.observation.provenance, + recordedBy: 'user-authored', + }, + }, + }, + }), + ) + + expect((await run(repository, memory({ revision: 4 }))).outcome).toBe( + 'imported', + ) + expect( + [...repository.events.values()].filter( + (event) => event.eventType === 'claim.forgotten', + ), + ).toEqual([]) + }) + test('reports imported when retry commits a marker after the body already committed', async () => { const repository = new Repository() const source = memory() @@ -365,9 +755,13 @@ describe('V1 memory migration', () => { const recovered = await run(repository, source) expect(recovered).toMatchObject({ outcome: 'imported' }) if (recovered.outcome === 'imported') { - expect(recovered.lastEventId).toBe([...repository.events.values()].at(-1)!.eventId) + expect(recovered.lastEventId).toBe( + [...repository.events.values()].at(-1)!.eventId, + ) } - expect([...repository.events.keys()].slice(0, bodyIds.length)).toEqual(bodyIds) + expect([...repository.events.keys()].slice(0, bodyIds.length)).toEqual( + bodyIds, + ) expect(new Set(repository.events.keys()).size).toBe(repository.events.size) expect(migrationMarkers(repository)).toHaveLength(1) }) @@ -420,9 +814,13 @@ describe('V1 memory migration', () => { expect(recovered).toEqual(imported) expect([...recovering.events.keys()]).toEqual([...clean.events.keys()]) expect( - [...recovering.events.values()].map(({ sequence: _sequence, ...event }) => event), + [...recovering.events.values()].map( + ({ sequence: _sequence, ...event }) => event, + ), ).toEqual( - [...clean.events.values()].map(({ sequence: _sequence, ...event }) => event), + [...clean.events.values()].map( + ({ sequence: _sequence, ...event }) => event, + ), ) }) @@ -432,7 +830,11 @@ describe('V1 memory migration', () => { const first = memory() expect(await run(repository, first)).toMatchObject({ outcome: 'failed' }) expect(migrationReservations(repository)).toHaveLength(1) - expect([...repository.events.values()].filter((event) => event.eventType !== 'migration.v1.reserved')).toHaveLength(0) + expect( + [...repository.events.values()].filter( + (event) => event.eventType !== 'migration.v1.reserved', + ), + ).toHaveLength(0) repository.failAppendCall = undefined const changed = memory({ @@ -444,7 +846,11 @@ describe('V1 memory migration', () => { outcome: 'rejected', reason: 'checksum-mismatch', }) - expect([...repository.events.values()].filter((event) => event.eventType !== 'migration.v1.reserved')).toHaveLength(0) + expect( + [...repository.events.values()].filter( + (event) => event.eventType !== 'migration.v1.reserved', + ), + ).toHaveLength(0) expect(await run(repository, first)).toMatchObject({ outcome: 'imported' }) }) @@ -472,7 +878,8 @@ describe('V1 memory migration', () => { expect(outcome.outcome).toBe('imported') expect(repository.appendRequests).toHaveLength(4) - const [reservationPage, firstPage, secondPage, markerPage] = repository.appendRequests + const [reservationPage, firstPage, secondPage, markerPage] = + repository.appendRequests expect(reservationPage.events).toHaveLength(1) expect(reservationPage.events[0]!.eventType).toBe('migration.v1.reserved') expect(reservationPage.expectedTail).toEqual({ kind: 'empty' }) @@ -493,3 +900,618 @@ describe('V1 memory migration', () => { } }) }) + +describe('V1 memory migration audit', () => { + test('no record and checksum mismatch make no repository calls', async () => { + const repository = new Repository() + expect(await audit(repository)).toEqual({ outcome: 'no-record' }) + expect( + await audit(repository, memory({ checksum: 'invalid' })), + ).toMatchObject({ + outcome: 'rejected', + reason: 'checksum-mismatch', + }) + expect(repository.exportCalls).toBe(0) + expect(repository.appendCalls).toBe(0) + }) + + test('reports not-migrated from a complete empty scan', async () => { + const repository = new Repository() + const source = memory() + expect(await audit(repository, source)).toEqual({ + outcome: 'not-migrated', + revision: source.revision, + checksum: source.checksum, + }) + expect(repository.appendCalls).toBe(0) + }) + + test('certifies an imported body without writes and preserves lossy metadata', async () => { + const repository = new Repository() + const source = memory({ historicalSummary: 'h'.repeat(1_100) }) + const imported = await run(repository, source) + expect(imported.outcome).toBe('imported') + const appendCalls = repository.appendCalls + + const outcome = await audit(repository, source) + expect(outcome).toMatchObject({ + outcome: 'exact', + revision: source.revision, + checksum: source.checksum, + identity: getV1MigrationIdentity({ + projectId, + revision: source.revision, + checksum: source.checksum, + }), + repositoryLastEventId: [...repository.events.values()].at(-1)!.eventId, + warnings: expect.arrayContaining([ + 'goal-excluded', + 'legacy-evidence-unverified', + 'text-truncated', + ]), + truncatedFields: 1, + }) + if (outcome.outcome === 'exact' && imported.outcome === 'imported') { + expect(outcome.markerEventId).toBe( + [...migrationMarkers(repository)][0]!.eventId, + ) + expect(outcome.importedTaskId).toBe(imported.importedTaskId) + expect(outcome.importedObservationIds).toEqual( + imported.importedObservationIds, + ) + expect(outcome.omittedFields).toBe(imported.omittedFields) + expect(outcome.sourceItemCounts).toEqual(imported.sourceItemCounts) + } + expect(repository.appendCalls).toBe(appendCalls) + }) + + test('certifies exact after a JSON round-trip that sorts record-valued payload keys', async () => { + // Regression: the Bun SQLite repository persists payloads via stableJson, + // which recursively sorts object keys, and export() re-parses from that + // sorted JSON. The marker's sourceItemCounts (a z.record) therefore comes + // back in alphabetical order while the in-memory draft keeps insertion + // order. equalEventDraft must compare content, not key order, or the real + // provider always reports imported-body-mismatch. This subclass mimics + // that round-trip by re-parsing each exported event with sorted keys. + class SortedJsonRepository extends Repository { + override async export( + input: Parameters[0], + ) { + const page = await super.export(input) + if (page.outcome !== 'page') return page + return { + ...page, + events: page.events.map((event) => + MemoryEventEnvelopeSchema.parse( + JSON.parse( + JSON.stringify(event, (_key, value) => + value !== null && + typeof value === 'object' && + !Array.isArray(value) + ? Object.fromEntries( + Object.entries(value as Record).sort( + ([a], [b]) => (a < b ? -1 : a > b ? 1 : 0), + ), + ) + : value, + ), + ), + ), + ), + } + } + } + + const repository = new SortedJsonRepository() + const source = memory({ requirements: ['req-1'], decisions: ['dec-1'] }) + expect((await run(repository, source)).outcome).toBe('imported') + expect(await audit(repository, source)).toMatchObject({ + outcome: 'exact', + revision: source.revision, + checksum: source.checksum, + }) + }) + + test('reports a reservation-only partial import as incomplete', async () => { + const repository = new Repository() + const source = memory() + repository.failAppendCall = 2 + expect(await run(repository, source)).toMatchObject({ outcome: 'failed' }) + + expect(await audit(repository, source)).toMatchObject({ + outcome: 'incomplete', + reason: 'reservation-only', + repositoryLastEventId: migrationReservations(repository)[0]!.eventId, + }) + }) + + test('reports same-revision checksum ownership as a mismatch', async () => { + const repository = new Repository() + const first = memory() + expect((await run(repository, first)).outcome).toBe('imported') + const changed = memory({ + revision: first.revision, + updatedAt: first.updatedAt + 1, + decisions: [...first.decisions, 'changed'], + }) + + expect(await audit(repository, changed)).toMatchObject({ + outcome: 'mismatch', + reason: 'checksum-conflict', + checksum: changed.checksum, + }) + }) + + test('detects missing imported task and observation bodies', async () => { + const missingTaskRepository = new Repository() + const source = memory() + const taskImport = await run(missingTaskRepository, source) + expect(taskImport.outcome).toBe('imported') + const taskEvent = [...missingTaskRepository.events.values()].find( + (event) => event.eventType === 'task.created', + )! + missingTaskRepository.events.delete(taskEvent.eventId) + expect(await audit(missingTaskRepository, source)).toMatchObject({ + outcome: 'incomplete', + reason: 'missing-imported-task', + }) + + const repository = new Repository() + const imported = await run(repository, source) + expect(imported.outcome).toBe('imported') + if (imported.outcome !== 'imported') return + const removed = [...repository.events.values()].find( + (event) => + event.eventType === 'observation.recorded' && + event.payload.observation.observationId === + imported.importedObservationIds[0], + )! + repository.events.delete(removed.eventId) + + expect(await audit(repository, source)).toMatchObject({ + outcome: 'incomplete', + reason: 'missing-imported-observations', + }) + }) + + test('detects mismatched imported observation provenance', async () => { + const repository = new Repository() + const source = memory() + const imported = await run(repository, source) + expect(imported.outcome).toBe('imported') + if (imported.outcome !== 'imported') return + const original = [...repository.events.values()].find( + (event) => + event.eventType === 'observation.recorded' && + event.payload.observation.observationId === + imported.importedObservationIds[0], + )! + if (original.eventType !== 'observation.recorded') return + const mutated = MemoryEventEnvelopeSchema.parse({ + ...original, + payload: { + ...original.payload, + observation: { + ...original.payload.observation, + provenance: { + ...original.payload.observation.provenance, + metadata: { + ...original.payload.observation.provenance?.metadata, + checksum: 'different', + }, + }, + }, + }, + }) + repository.events.set(original.eventId, mutated) + + expect(await audit(repository, source)).toMatchObject({ + outcome: 'mismatch', + reason: 'imported-body-mismatch', + }) + }) + + test('detects full deterministic task, observation, header, and marker tampering', async () => { + const source = memory() + const assertTamper = async ( + select: (repository: Repository) => MemoryEventEnvelope, + mutate: (event: MemoryEventEnvelope) => unknown, + ) => { + const repository = new Repository() + expect((await run(repository, source)).outcome).toBe('imported') + const original = select(repository) + const replacement = MemoryEventEnvelopeSchema.parse(mutate(original)) + repository.events.set(original.eventId, replacement) + expect(await audit(repository, source)).toMatchObject({ + outcome: 'mismatch', + reason: 'imported-body-mismatch', + }) + } + const task = (repository: Repository) => + [...repository.events.values()].find( + (event) => event.eventType === 'task.created', + )! + const observation = (repository: Repository) => + [...repository.events.values()].find( + (event) => event.eventType === 'observation.recorded', + )! + const marker = (repository: Repository) => migrationMarkers(repository)[0]! + + for (const payload of [ + { title: 'tampered' }, + { objective: 'tampered' }, + { initialStatus: 'created' as const, taskId: 'task:tampered' }, + { payloadSchemaVersion: 1 as const, title: 'tampered' }, + ]) { + await assertTamper(task, (event) => ({ + ...event, + payload: { ...event.payload, ...payload }, + })) + } + for (const header of [ + { sessionId: 'session:tampered' }, + { occurredAt: '2027-01-01T00:00:00.000Z' }, + { projectId: 'project:tampered' }, + ]) { + await assertTamper(task, (event) => ({ ...event, ...header })) + } + + const observationMutations: Array< + (value: Record) => Record + > = [ + (value) => ({ ...value, observationId: 'observation:tampered' }), + (value) => ({ ...value, taskId: 'task:tampered' }), + (value) => ({ ...value, kind: 'fact' }), + (value) => ({ ...value, summary: 'tampered' }), + (value) => ({ ...value, detail: 'tampered' }), + (value) => ({ ...value, confidence: 0.5 }), + (value) => ({ + ...value, + evidence: [ + { + artifact: { + artifactId: 'artifact:tampered', + location: 'src/tampered.ts', + classification: { + kind: 'source', + generated: false, + sensitivity: 'internal', + labels: [], + }, + }, + selector: { kind: 'file', path: 'src/tampered.ts' }, + provenance: { + origin: 'migration', + recordedBy: 'tampered', + sourceEventIds: [], + sourceSessionId: sessionId, + metadata: {}, + }, + capturedAt: '2026-09-10T19:41:53.753Z', + }, + ], + }), + (value) => ({ + ...value, + selectors: [{ kind: 'file', path: 'tampered.ts' }], + }), + (value) => ({ + ...value, + provenance: { ...(value.provenance as object), origin: 'derived' }, + }), + (value) => ({ + ...value, + provenance: { ...(value.provenance as object), recordedBy: 'tampered' }, + }), + (value) => ({ + ...value, + provenance: { + ...(value.provenance as object), + sourceEventIds: ['event:tampered'], + }, + }), + (value) => ({ + ...value, + provenance: { + ...(value.provenance as object), + sourceSessionId: 'session:tampered', + }, + }), + (value) => ({ + ...value, + provenance: { + ...(value.provenance as object), + metadata: { + category: 'tampered', + revision: 3, + checksum: source.checksum, + }, + }, + }), + (value) => ({ ...value, tags: ['tampered'] }), + (value) => ({ ...value, observedAt: '2027-01-01T00:00:00.000Z' }), + ] + for (const mutate of observationMutations) { + await assertTamper(observation, (event) => { + if (event.eventType !== 'observation.recorded') return event + return { + ...event, + payload: { + ...event.payload, + observation: mutate( + event.payload.observation as unknown as Record, + ), + }, + } + }) + } + + for (const payload of [ + { importedTaskId: 'task:tampered' }, + { importedObservationIds: [] }, + { importedObservationIds: ['observation:tampered'] }, + { omittedFields: 99 }, + { warnings: [] }, + { sourceItemCounts: { requirements: 99 } }, + { truncatedFields: 99 }, + { legacyRecordKey: 'tampered' }, + ]) { + await assertTamper(marker, (event) => ({ + ...event, + payload: { ...event.payload, ...payload }, + })) + } + }) + + test('allows unrelated events but fails duplicate deterministic IDs', async () => { + const source = memory() + const repository = new Repository() + expect((await run(repository, source)).outcome).toBe('imported') + const template = [...repository.events.values()].find( + (event) => event.eventType === 'task.created', + )! + repository.events.set( + 'event:unrelated', + MemoryEventEnvelopeSchema.parse({ + ...template, + eventId: 'event:unrelated', + payload: { ...template.payload, taskId: 'task:unrelated' }, + sequence: repository.sequence + 1, + }), + ) + expect(await audit(repository, source)).toMatchObject({ outcome: 'exact' }) + + const events = [...repository.events.values()] + const reader: V1MigrationAuditReader = { + async export() { + return { + outcome: 'page', + events: [...events, template], + nextAfterEventId: null, + } + }, + } + expect(await audit(reader, source)).toMatchObject({ + outcome: 'mismatch', + reason: 'imported-body-mismatch', + }) + }) + + test('treats a legacy marker without source proof as unverifiable', async () => { + const repository = new Repository() + const source = memory() + expect((await run(repository, source)).outcome).toBe('imported') + const marker = migrationMarkers(repository)[0]! + if (marker.eventType !== 'migration.v1.imported') return + const { + sourceRevision: _sourceRevision, + sourceChecksum: _sourceChecksum, + ...legacyPayload + } = marker.payload + repository.events.set( + marker.eventId, + MemoryEventEnvelopeSchema.parse({ ...marker, payload: legacyPayload }), + ) + + expect(await audit(repository, source)).toMatchObject({ + outcome: 'incomplete', + reason: 'legacy-marker-unverifiable', + markerEventId: marker.eventId, + }) + }) + + test('maps repository failures and fails closed on malformed or wrong-project output', async () => { + const source = memory() + const repository = new Repository() + await run(repository, source) + const event = [...repository.events.values()][0]! + const otherProjectEvent = cloneEvent(event, { + projectId: ProjectIdSchema.parse('project:other'), + }) + const cases: Array<{ + reader: V1MigrationAuditReader + expected: { outcome: string; reason: string } + }> = [ + { + reader: { + async export() { + return { + outcome: 'rejected', + error: { + code: 'invalid-request', + message: 'no', + retryable: false, + }, + } + }, + }, + expected: { outcome: 'rejected', reason: 'repository-rejected' }, + }, + { + reader: { + async export() { + return { + outcome: 'failed', + error: { code: 'unavailable', message: 'no', retryable: true }, + } + }, + }, + expected: { outcome: 'failed', reason: 'repository-failed' }, + }, + { + reader: { + async export(): Promise { + throw new Error('private failure') + }, + }, + expected: { outcome: 'failed', reason: 'repository-failed' }, + }, + { + reader: { + async export() { + return { + outcome: 'page', + events: 'invalid', + nextAfterEventId: null, + } + }, + } as unknown as V1MigrationAuditReader, + expected: { outcome: 'failed', reason: 'invalid-export' }, + }, + { + reader: { + async export() { + return { + outcome: 'page', + events: [otherProjectEvent], + nextAfterEventId: null, + } + }, + }, + expected: { outcome: 'failed', reason: 'wrong-project' }, + }, + ] + + for (const { reader, expected } of cases) { + expect(await audit(reader, source)).toMatchObject(expected) + } + }) + + test('rejects empty, non-tail, and repeated pagination cursors', async () => { + const source = memory() + const repository = new Repository() + await run(repository, source) + const first = [...repository.events.values()][0]! + const second = cloneEvent(first, { + eventId: MemoryEventIdSchema.parse('event:audit-second'), + sequence: first.sequence + 1, + }) + const cursor = first.eventId + const readers: V1MigrationAuditReader[] = [ + { + async export() { + return { outcome: 'page', events: [], nextAfterEventId: cursor } + }, + }, + { + async export() { + return { + outcome: 'page', + events: [first], + nextAfterEventId: second.eventId, + } + }, + }, + { + async export(request) { + return request.afterEventId + ? { outcome: 'page', events: [second], nextAfterEventId: cursor } + : { outcome: 'page', events: [first], nextAfterEventId: cursor } + }, + }, + { + async export(request) { + return request.afterEventId + ? { outcome: 'page', events: [first], nextAfterEventId: null } + : { outcome: 'page', events: [first], nextAfterEventId: cursor } + }, + }, + ] + for (const reader of readers) { + expect(await audit(reader, source)).toMatchObject({ + outcome: 'failed', + reason: 'pagination-invalid', + }) + } + }) + + test('fails rather than certifying a scan with a tenth continuation page', async () => { + const source = memory() + const repository = new Repository() + await run(repository, source) + const template = [...repository.events.values()][0]! + let page = 0 + const reader: V1MigrationAuditReader = { + async export() { + const event = cloneEvent(template, { + eventId: MemoryEventIdSchema.parse(`event:audit-page-${page}`), + sequence: page + 1, + }) + page++ + return { + outcome: 'page', + events: [event], + nextAfterEventId: event.eventId, + } + }, + } + + expect(await audit(reader, source)).toMatchObject({ + outcome: 'failed', + reason: 'page-limit-exceeded', + }) + expect(page).toBe(10) + }) + + test('allows a full final page to be followed by an empty terminal page', async () => { + const source = memory() + const repository = new Repository() + expect((await run(repository, source)).outcome).toBe('imported') + const canonical = [...repository.events.values()] + const template = canonical.find( + (event) => event.eventType === 'task.created', + )! + const padding = Array.from( + { length: 1_000 - canonical.length }, + (_, index) => + cloneEvent(template, { + eventId: MemoryEventIdSchema.parse(`event:audit-padding-${index}`), + sequence: canonical.length + index + 1, + sessionId: MemorySessionIdSchema.parse( + `session:audit-padding-${index}`, + ), + payload: { + ...template.payload, + taskId: TaskIdSchema.parse(`task:audit-padding-${index}`), + }, + }), + ) + const events = [...canonical, ...padding] + let calls = 0 + const reader: V1MigrationAuditReader = { + async export(request) { + calls++ + return request.afterEventId + ? { outcome: 'page', events: [], nextAfterEventId: null } + : { + outcome: 'page', + events, + nextAfterEventId: events.at(-1)!.eventId, + } + }, + } + + expect(await audit(reader, source)).toMatchObject({ outcome: 'exact' }) + expect(calls).toBe(2) + }) +}) diff --git a/sdk/src/services/memory-v2/v1-migration.ts b/sdk/src/services/memory-v2/v1-migration.ts index 6cd8d187a9..405e36288f 100644 --- a/sdk/src/services/memory-v2/v1-migration.ts +++ b/sdk/src/services/memory-v2/v1-migration.ts @@ -1,10 +1,13 @@ import { MemoryAppendOutcomeSchema, MemoryAppendRequestSchema, + MemoryEventDraftSchema, MemoryEventIdSchema, + MemoryExportOutcomeSchema, ObservationIdSchema, TaskIdSchema, type MemoryEventDraft, + type MemoryEventEnvelope, type MemoryEventId, type MemorySessionId, type ObservationId, @@ -41,7 +44,7 @@ const MIGRATION_CATEGORIES = [ ] as const type V1MigrationCategory = (typeof MIGRATION_CATEGORIES)[number] -type V1MigrationSourceItemCounts = Record +export type V1MigrationSourceItemCounts = Record function newSourceItemCounts(): V1MigrationSourceItemCounts { return Object.fromEntries( @@ -60,6 +63,9 @@ function markerSourceItemCounts( return normalized } +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export type V1MigrationWarningCode = | 'goal-excluded' | 'observation-cap-reached' @@ -69,6 +75,9 @@ export type V1MigrationWarningCode = | 'empty-field-omitted' | 'text-truncated' +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export type V1MigrationOutcome = | { outcome: 'no-record' } | { @@ -102,9 +111,21 @@ const boundedText = ( return { value: trimmed.slice(0, limit), truncated: trimmed.length > limit } } +/** + * stableHash is FNV-1a 32-bit -> a fixed 8-hex-char token. The sanitizing + * regex strips nothing from that output, so the slice(0, 96) cap is a no-op + * safeguard. Collision tolerance is governed by the 32-bit hash space + * (birthday-bound collisions become non-negligible around ~2^16 distinct + * tokens), not by the 96-char cap, which never further shortens the token. + */ const hashToken = (value: string): string => - stableHash(value).replace(/[^A-Za-z0-9._:-]/g, '').slice(0, 96) + stableHash(value) + .replace(/[^A-Za-z0-9._:-]/g, '') + .slice(0, 96) +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export function getV1MigrationIdentity(params: { projectId: ProjectId revision: number @@ -124,11 +145,15 @@ function getV1MigrationBodyIdentity(params: { } function reservationEventId(identity: string): MemoryEventId { - return MemoryEventIdSchema.parse(`event:migration:reservation:${hashToken(identity)}`) + return MemoryEventIdSchema.parse( + `event:migration:reservation:${hashToken(identity)}`, + ) } function eventId(identity: string, index: number) { - return MemoryEventIdSchema.parse(`event:migration:${hashToken(`${identity}:${index}`)}`) + return MemoryEventIdSchema.parse( + `event:migration:${hashToken(`${identity}:${index}`)}`, + ) } type MigrationMarkerMetadata = { @@ -141,18 +166,25 @@ type MigrationMarkerMetadata = { truncatedFields?: number } +type ImportedMarkerEvent = Extract< + MemoryEventEnvelope, + { eventType: 'migration.v1.imported' } +> + type MigrationMarkerLookup = | { state: 'none' lastEventId?: MemoryEventId prior?: MigrationMarkerMetadata reserved: boolean + events: MemoryEventEnvelope[] } | { state: 'conflict' lastEventId?: MemoryEventId prior?: MigrationMarkerMetadata reserved: boolean + events: MemoryEventEnvelope[] } | { state: 'exact' @@ -160,9 +192,12 @@ type MigrationMarkerLookup = marker: MigrationMarkerMetadata prior?: MigrationMarkerMetadata reserved: boolean + events: MemoryEventEnvelope[] } -function isV1MigrationWarningCode(value: string): value is V1MigrationWarningCode { +function isV1MigrationWarningCode( + value: string, +): value is V1MigrationWarningCode { switch (value) { case 'goal-excluded': case 'observation-cap-reached': @@ -177,6 +212,388 @@ function isV1MigrationWarningCode(value: string): value is V1MigrationWarningCod } } +function markerMetadata(event: ImportedMarkerEvent): MigrationMarkerMetadata { + return { + eventId: event.eventId, + importedTaskId: event.payload.importedTaskId, + importedObservationIds: [...event.payload.importedObservationIds], + omittedFields: event.payload.omittedFields ?? 0, + warnings: event.payload.warnings.filter(isV1MigrationWarningCode), + ...(event.payload.sourceItemCounts + ? { + sourceItemCounts: markerSourceItemCounts( + event.payload.sourceItemCounts, + ), + } + : {}), + ...(event.payload.truncatedFields === undefined + ? {} + : { truncatedFields: event.payload.truncatedFields }), + } +} + +function isV1MigrationCategory(value: unknown): value is V1MigrationCategory { + return ( + typeof value === 'string' && + (MIGRATION_CATEGORIES as readonly string[]).includes(value) + ) +} + +type ImportedObservationEvent = Extract< + MemoryEventEnvelope, + { eventType: 'observation.recorded' } +> + +type CanonicalMarkerSourceItemCounts = NonNullable< + ImportedMarkerEvent['payload']['sourceItemCounts'] +> + +/** + * The marker payload must carry the V1 source fields every remaining + * ownership check depends on. + */ +function hasCanonicalMarkerSourceFields( + payload: ImportedMarkerEvent['payload'], +): payload is ImportedMarkerEvent['payload'] & { + sourceRevision: number + sourceChecksum: string + sourceItemCounts: CanonicalMarkerSourceItemCounts +} { + return ( + payload.sourceRevision !== undefined && + payload.sourceChecksum !== undefined && + payload.sourceItemCounts !== undefined + ) +} + +/** + * The marker payload must reproduce the deterministic identities, payload + * schema versions, and observation id sequence emitted by this migration. + */ +function hasCanonicalMarkerPayloadShape(params: { + marker: ImportedMarkerEvent + projectId: ProjectId + identity: string + expectedTaskId: TaskId + expectedObservationIds: ObservationId[] +}): boolean { + const { + marker, + projectId, + identity, + expectedTaskId, + expectedObservationIds, + } = params + return !( + marker.payload.legacyRecordKey !== identity || + marker.projectId !== projectId || + marker.eventId !== eventId(identity, MAX_OBSERVATIONS + 1) || + marker.payload.payloadSchemaVersion !== 1 || + marker.payload.sourceSchemaVersion !== 1 || + marker.payload.importedTaskId !== expectedTaskId || + expectedObservationIds.some( + (observationId, index) => + observationId !== marker.payload.importedObservationIds[index], + ) + ) +} + +/** + * Exactly one canonical reservation event must exist, matching the marker + * envelope and the V1 source fields. + */ +function hasValidMigrationReservation(params: { + marker: ImportedMarkerEvent + events: MemoryEventEnvelope[] + projectId: ProjectId + identity: string + sourceRevision: number + sourceChecksum: string +}): boolean { + const { + marker, + events, + projectId, + identity, + sourceRevision, + sourceChecksum, + } = params + const reservations = events.filter( + (event) => event.eventId === reservationEventId(identity), + ) + if (reservations.length !== 1) return false + const reservation = reservations[0]! + return !( + reservation.eventType !== 'migration.v1.reserved' || + reservation.projectId !== projectId || + reservation.sessionId !== marker.sessionId || + reservation.occurredAt !== marker.occurredAt || + reservation.payload.payloadSchemaVersion !== 1 || + reservation.payload.migrationId !== identity || + reservation.payload.sourceRevision !== sourceRevision || + reservation.payload.sourceChecksum !== sourceChecksum + ) +} + +/** + * Exactly one canonical task.created event must exist for the imported task, + * matching the marker envelope and the deterministic import text. + */ +function hasValidImportedTaskEvent(params: { + marker: ImportedMarkerEvent + events: MemoryEventEnvelope[] + projectId: ProjectId + bodyIdentity: string + expectedTaskId: TaskId + sourceRevision: number +}): boolean { + const { + marker, + events, + projectId, + bodyIdentity, + expectedTaskId, + sourceRevision, + } = params + const expectedTaskEventId = eventId(bodyIdentity, 0) + const taskEvents = events.filter( + (event) => event.eventId === expectedTaskEventId, + ) + if (taskEvents.length !== 1) return false + const task = taskEvents[0]! + return !( + task.eventType !== 'task.created' || + task.projectId !== projectId || + task.sessionId !== marker.sessionId || + task.occurredAt !== marker.occurredAt || + task.payload.payloadSchemaVersion !== 1 || + task.payload.taskId !== expectedTaskId || + task.payload.title !== + `Imported legacy task memory revision ${sourceRevision}` || + task.payload.objective !== + 'Preserve bounded legacy operational memory without importing its goal.' || + task.payload.initialStatus !== 'created' + ) +} + +/** + * A single imported observation event must match the deterministic body shape + * for its category, within the source item counts and canonical ordering. + */ +function hasValidImportedObservationEvent(params: { + marker: ImportedMarkerEvent + event: ImportedObservationEvent + projectId: ProjectId + expectedTaskId: TaskId + observationId: ObservationId + category: V1MigrationCategory + categoryIndex: number + occurrences: number + previousCategoryIndex: number + sourceRevision: number + sourceChecksum: string + sourceItemCounts: CanonicalMarkerSourceItemCounts +}): boolean { + const { + marker, + event, + projectId, + expectedTaskId, + observationId, + category, + categoryIndex, + occurrences, + previousCategoryIndex, + sourceRevision, + sourceChecksum, + sourceItemCounts, + } = params + const observation = event.payload.observation + const expectedSummary = + category === 'path-evidence' + ? 'Legacy path evidence requires reread' + : category === 'historical-summary' + ? 'Legacy historical summary' + : `Legacy ${category} items (${sourceItemCounts[category] ?? 0})` + if ( + categoryIndex < previousCategoryIndex || + (occurrences > 1 && category !== 'path-evidence') || + (sourceItemCounts[category] ?? 0) < occurrences || + event.projectId !== projectId || + event.sessionId !== marker.sessionId || + event.occurredAt !== marker.occurredAt || + event.payload.payloadSchemaVersion !== 1 || + observation.observationId !== observationId || + observation.taskId !== expectedTaskId || + observation.kind !== (category === 'blockers' ? 'warning' : 'discovery') || + observation.summary !== expectedSummary || + observation.detail.length > MAX_AGGREGATE_DETAIL || + observation.confidence !== 0.25 || + observation.evidence.length !== 0 || + observation.observedAt !== marker.occurredAt || + observation.provenance?.origin !== 'migration' || + observation.provenance.recordedBy !== 'sdk-memory-v1-import' || + observation.provenance.sourceEventIds.length !== 0 || + observation.provenance.sourceSessionId !== marker.sessionId || + observation.provenance.metadata.revision !== sourceRevision || + observation.provenance.metadata.checksum !== sourceChecksum || + JSON.stringify(observation.tags) !== + JSON.stringify(['legacy-v1', category, 'unverified']) + ) + return false + const selectors = observation.selectors ?? [] + if (category === 'path-evidence') { + if (selectors.length !== 1 || selectors[0]?.kind !== 'file') return false + const decision = classifyMemoryArtifactPath(selectors[0].path) + if (!decision.allowed || !decision.normalizedPath) return false + } else if (selectors.length !== 0) { + return false + } + return true +} + +/** + * Every expected observation event must exist exactly once, in canonical + * category order and within the per-category source item counts. + */ +function hasValidImportedObservationEvents(params: { + marker: ImportedMarkerEvent + events: MemoryEventEnvelope[] + projectId: ProjectId + bodyIdentity: string + expectedTaskId: TaskId + expectedObservationIds: ObservationId[] + sourceRevision: number + sourceChecksum: string + sourceItemCounts: CanonicalMarkerSourceItemCounts +}): boolean { + const { + marker, + events, + projectId, + bodyIdentity, + expectedTaskId, + expectedObservationIds, + sourceRevision, + sourceChecksum, + sourceItemCounts, + } = params + let previousCategoryIndex = -1 + const categoryOccurrences = new Map() + for (const [index, observationId] of expectedObservationIds.entries()) { + const expectedEventId = eventId(bodyIdentity, index + 1) + const matching = events.filter((event) => event.eventId === expectedEventId) + if (matching.length !== 1) return false + const event = matching[0]! + if (event.eventType !== 'observation.recorded') return false + const observation = event.payload.observation + const category = observation.provenance?.metadata.category + if (!isV1MigrationCategory(category)) return false + const categoryIndex = MIGRATION_CATEGORIES.indexOf(category) + const occurrences = (categoryOccurrences.get(category) ?? 0) + 1 + categoryOccurrences.set(category, occurrences) + if ( + !hasValidImportedObservationEvent({ + marker, + event, + projectId, + expectedTaskId, + observationId, + category, + categoryIndex, + occurrences, + previousCategoryIndex, + sourceRevision, + sourceChecksum, + sourceItemCounts, + }) + ) + return false + previousCategoryIndex = categoryIndex + } + return true +} + +/** + * Return cleanup authority only for a canonical marker whose reservation and + * complete referenced body have the deterministic shape emitted by this + * migration. Repair markers and migration-looking provenance deliberately do + * not authorize destructive retirement: without the old V1 source they cannot + * be distinguished from user-authored events strongly enough to delete data. + */ +function validatedMigrationOwnedMarkerMetadata(params: { + marker: ImportedMarkerEvent + events: MemoryEventEnvelope[] + projectId: ProjectId +}): MigrationMarkerMetadata | undefined { + const { marker, events, projectId } = params + if (!hasCanonicalMarkerSourceFields(marker.payload)) return undefined + const { sourceRevision, sourceChecksum, sourceItemCounts } = marker.payload + + const identity = getV1MigrationIdentity({ + projectId, + revision: sourceRevision, + checksum: sourceChecksum, + }) + const bodyIdentity = getV1MigrationBodyIdentity({ + projectId, + revision: sourceRevision, + checksum: sourceChecksum, + }) + const expectedTaskId = TaskIdFor(bodyIdentity) + const expectedObservationIds = marker.payload.importedObservationIds.map( + (_, index) => ObservationIdFor(bodyIdentity, index), + ) + if ( + !hasCanonicalMarkerPayloadShape({ + marker, + projectId, + identity, + expectedTaskId, + expectedObservationIds, + }) + ) + return undefined + if ( + !hasValidMigrationReservation({ + marker, + events, + projectId, + identity, + sourceRevision, + sourceChecksum, + }) + ) + return undefined + if ( + !hasValidImportedTaskEvent({ + marker, + events, + projectId, + bodyIdentity, + expectedTaskId, + sourceRevision, + }) + ) + return undefined + if ( + !hasValidImportedObservationEvents({ + marker, + events, + projectId, + bodyIdentity, + expectedTaskId, + expectedObservationIds, + sourceRevision, + sourceChecksum, + sourceItemCounts, + }) + ) + return undefined + + return markerMetadata(marker) +} + async function findMigrationMarker( repository: MemoryRepositoryV2, projectId: ProjectId, @@ -188,7 +605,11 @@ async function findMigrationMarker( let exact: MigrationMarkerMetadata | undefined let conflict = false let reserved = false - let prior: MigrationMarkerMetadata | undefined + const events: MemoryEventEnvelope[] = [] + const priorCandidates: ImportedMarkerEvent[] = [] + // Loop-invariant: projectId, revision, and checksum are function params, so + // compute the migration identity once instead of per reservation event. + const identity = getV1MigrationIdentity({ projectId, revision, checksum }) for (let page = 0; page < MAX_EXPORT_PAGES; page++) { const outcome = await repository.export({ @@ -197,52 +618,54 @@ async function findMigrationMarker( ...(afterEventId ? { afterEventId } : {}), limit: 1_000, }) - if (outcome.outcome !== 'page') throw new Error('Migration marker lookup failed') + if (outcome.outcome !== 'page') + throw new Error('Migration marker lookup failed') for (const event of outcome.events) { + events.push(event) lastEventId = event.eventId if (event.eventType === 'migration.v1.reserved') { if (event.payload.sourceRevision === revision) { if ( - event.payload.migrationId === getV1MigrationIdentity({ projectId, revision, checksum }) && + event.payload.migrationId === identity && event.payload.sourceChecksum === checksum - ) reserved = true + ) + reserved = true else conflict = true } continue } if (event.eventType !== 'migration.v1.imported') continue - const marker: MigrationMarkerMetadata = { - eventId: event.eventId, - importedTaskId: event.payload.importedTaskId, - importedObservationIds: [...event.payload.importedObservationIds], - omittedFields: event.payload.omittedFields ?? 0, - warnings: event.payload.warnings.filter(isV1MigrationWarningCode), - ...(event.payload.sourceItemCounts - ? { - sourceItemCounts: markerSourceItemCounts( - event.payload.sourceItemCounts, - ), - } - : {}), - ...(event.payload.truncatedFields === undefined - ? {} - : { truncatedFields: event.payload.truncatedFields }), - } const sourceRevision = event.payload.sourceRevision if (sourceRevision === revision) { - if (event.payload.sourceChecksum === checksum) exact = marker + if (event.payload.sourceChecksum === checksum) + exact = markerMetadata(event) else conflict = true } else if (sourceRevision !== undefined && sourceRevision < revision) { - prior = marker + priorCandidates.push(event) } } if (!outcome.nextAfterEventId) { - if (conflict) return { state: 'conflict', lastEventId, prior, reserved } - if (exact && lastEventId) return { state: 'exact', lastEventId, marker: exact, prior, reserved } - return { state: 'none', lastEventId, prior, reserved } + const prior = [...priorCandidates] + .reverse() + .map((marker) => + validatedMigrationOwnedMarkerMetadata({ marker, events, projectId }), + ) + .find((metadata) => metadata !== undefined) + if (conflict) + return { state: 'conflict', lastEventId, prior, reserved, events } + if (exact && lastEventId) + return { + state: 'exact', + lastEventId, + marker: exact, + prior, + reserved, + events, + } + return { state: 'none', lastEventId, prior, reserved, events } } afterEventId = outcome.nextAfterEventId lastEventId = outcome.nextAfterEventId @@ -268,58 +691,113 @@ function outcomeFromMarker(params: { omittedFields: marker.omittedFields, warnings: marker.warnings, lastEventId, - ...(marker.sourceItemCounts ? { sourceItemCounts: marker.sourceItemCounts } : {}), + ...(marker.sourceItemCounts + ? { sourceItemCounts: marker.sourceItemCounts } + : {}), ...(marker.truncatedFields === undefined ? {} : { truncatedFields: marker.truncatedFields }), } } -export async function importTaskMemoryV1(params: { - memory?: TaskMemoryV1 - projectId: ProjectId - sessionId: MemorySessionId - repository: MemoryRepositoryV2 -}): Promise { - const { memory, projectId, sessionId, repository } = params - if (!memory) return { outcome: 'no-record' } +type V1MigrationBuild = { + identity: string + representationIdentity: string + bodyIdentity: string + occurredAt: string + importedTaskId: TaskId + importedObservationIds: ObservationId[] + omittedFields: number + warnings: V1MigrationWarningCode[] + sourceItemCounts: V1MigrationSourceItemCounts + truncatedFields: number + bodyDrafts: MemoryEventDraft[] + markerDraft: MemoryEventDraft +} - const { revision, updatedAt, checksum, ...candidateDraft } = memory - const parsedDraft = taskMemoryDraftV1Schema.safeParse(candidateDraft) - if (!parsedDraft.success) { - return { outcome: 'rejected', reason: 'checksum-mismatch', revision, checksum } +function normalizedEventDraft(event: MemoryEventEnvelope): MemoryEventDraft { + const { sequence: _sequence, ...draft } = event + return MemoryEventDraftSchema.parse(draft) +} + +/** + * Canonicalize a value for order-insensitive comparison: plain-object keys are + * emitted in sorted order (matching the recursive key-sort the Bun SQLite + * repository applies via stableJson before persisting), arrays keep their + * order, and primitives pass through. Without this, a record-valued payload + * field (e.g. sourceItemCounts) survives a stableJson round-trip with + * alphabetically sorted keys while the in-memory draft keeps insertion order, + * so a naive JSON.stringify comparison would always report a mismatch on the + * real provider even though both sides carry identical content. + */ +function canonicalizeForCompare(value: unknown): unknown { + if (Array.isArray(value)) return value.map(canonicalizeForCompare) + if (value !== null && typeof value === 'object') { + const record = value as Record + const sorted: Record = {} + for (const key of Object.keys(record).sort()) { + sorted[key] = canonicalizeForCompare(record[key]) + } + return sorted } - const recomputed = stableHash( - JSON.stringify({ revision, updatedAt, memory: parsedDraft.data }), + return value +} + +function equalEventDraft( + expected: MemoryEventDraft, + actual: MemoryEventEnvelope, +): boolean { + return ( + JSON.stringify( + canonicalizeForCompare(MemoryEventDraftSchema.parse(expected)), + ) === JSON.stringify(canonicalizeForCompare(normalizedEventDraft(actual))) ) - if (recomputed !== checksum) { - return { outcome: 'rejected', reason: 'checksum-mismatch', revision, checksum } - } +} - const identity = getV1MigrationIdentity({ projectId, revision, checksum }) - const bodyIdentity = getV1MigrationBodyIdentity({ projectId, revision, checksum }) - let lookup: MigrationMarkerLookup - try { - lookup = await findMigrationMarker(repository, projectId, revision, checksum) - } catch { - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } - } - if (lookup.state === 'conflict') { - return { outcome: 'rejected', reason: 'checksum-mismatch', revision, checksum } - } - if (lookup.state === 'exact') { - return outcomeFromMarker({ - marker: lookup.marker, - revision, - checksum, - identity, - lastEventId: lookup.lastEventId, - }) - } +function validatedCompleteBuildMarker( + lookup: Extract, + build: V1MigrationBuild, +): MigrationMarkerMetadata | undefined { + const markerEvents = lookup.events.filter( + (event): event is ImportedMarkerEvent => + event.eventId === build.markerDraft.eventId && + event.eventType === 'migration.v1.imported', + ) + if ( + markerEvents.length !== 1 || + !equalEventDraft(build.markerDraft, markerEvents[0]!) + ) + return undefined - let expectedTail = lookup.lastEventId - ? { kind: 'event' as const, eventId: lookup.lastEventId } - : { kind: 'empty' as const } + const completeBody = build.bodyDrafts.every((draft) => { + const matching = lookup.events.filter( + (event) => event.eventId === draft.eventId, + ) + return matching.length === 1 && equalEventDraft(draft, matching[0]!) + }) + return completeBody ? markerMetadata(markerEvents[0]!) : undefined +} + +/** Pure source-to-events transform shared by import and audit. */ +function buildTaskMemoryV1Migration(params: { + memory: TaskMemoryV1 + projectId: ProjectId + sessionId: MemorySessionId + representationIdentity?: string +}): V1MigrationBuild { + const { memory, projectId, sessionId } = params + const { revision, updatedAt, checksum } = memory + const identity = getV1MigrationIdentity({ projectId, revision, checksum }) + const representationIdentity = params.representationIdentity ?? identity + const canonicalBodyIdentity = getV1MigrationBodyIdentity({ + projectId, + revision, + checksum, + }) + const bodyIdentity = + representationIdentity === identity + ? canonicalBodyIdentity + : `${canonicalBodyIdentity}:${hashToken(representationIdentity)}` const occurredAt = new Date(updatedAt).toISOString() const importedTaskId = TaskIdFor(bodyIdentity) const warnings = new Set() @@ -333,14 +811,15 @@ export async function importTaskMemoryV1(params: { omittedFields++ warnings.add('text-truncated') } - const observations: Array<{ category: V1MigrationCategory summary: string detail: string selector?: { kind: 'file'; path: string } }> = [] - const addObservation = (observation: (typeof observations)[number]): boolean => { + const addObservation = ( + observation: (typeof observations)[number], + ): boolean => { if (observations.length >= MAX_OBSERVATIONS) { omittedFields++ warnings.add('observation-cap-reached') @@ -419,8 +898,7 @@ export async function importTaskMemoryV1(params: { }) } - const drafts: MemoryEventDraft[] = [] - drafts.push( + const bodyDrafts: MemoryEventDraft[] = [ withMigrationEventId( createMemoryEventDraft({ projectId, @@ -432,20 +910,20 @@ export async function importTaskMemoryV1(params: { payloadSchemaVersion: 1, taskId: importedTaskId, title: `Imported legacy task memory revision ${revision}`, - objective: 'Preserve bounded legacy operational memory without importing its goal.', + objective: + 'Preserve bounded legacy operational memory without importing its goal.', initialStatus: 'created', }, }), bodyIdentity, 0, ), - ) - + ] const importedObservationIds: ObservationId[] = [] observations.forEach((item, index) => { const observationId = ObservationIdFor(bodyIdentity, index) importedObservationIds.push(observationId) - drafts.push( + bodyDrafts.push( withMigrationEventId( createMemoryEventDraft({ projectId, @@ -483,6 +961,587 @@ export async function importTaskMemoryV1(params: { ) }) + const warningList = [...warnings].slice(0, 100) + const markerDraft = withMigrationEventId( + createMemoryEventDraft({ + projectId, + sessionId, + userInputId: representationIdentity, + sourceIndex: MAX_OBSERVATIONS + 1, + occurredAt, + eventType: 'migration.v1.imported', + payload: { + payloadSchemaVersion: 1, + sourceSchemaVersion: 1, + legacyRecordKey: representationIdentity, + importedTaskId, + importedObservationIds, + sourceRevision: revision, + sourceChecksum: checksum, + omittedFields, + sourceItemCounts, + truncatedFields, + warnings: warningList, + }, + }), + representationIdentity, + MAX_OBSERVATIONS + 1, + ) + return { + identity, + representationIdentity, + bodyIdentity, + occurredAt, + importedTaskId, + importedObservationIds, + omittedFields, + warnings: warningList, + sourceItemCounts, + truncatedFields, + bodyDrafts, + markerDraft, + } +} + +export type V1MigrationAuditReader = Pick + +export type V1MigrationAuditOutcome = + | { outcome: 'no-record' } + | { + outcome: 'not-migrated' + revision: number + checksum: string + repositoryLastEventId?: MemoryEventId + } + | { + outcome: 'exact' + revision: number + checksum: string + identity: string + markerEventId: MemoryEventId + repositoryLastEventId: MemoryEventId + importedTaskId: TaskId + importedObservationIds: ObservationId[] + omittedFields: number + warnings: V1MigrationWarningCode[] + sourceItemCounts?: V1MigrationSourceItemCounts + truncatedFields?: number + } + | { + outcome: 'incomplete' | 'mismatch' + reason: + | 'reservation-only' + | 'legacy-marker-unverifiable' + | 'missing-imported-task' + | 'missing-imported-observations' + | 'imported-body-mismatch' + | 'revision-conflict' + | 'checksum-conflict' + revision: number + checksum: string + repositoryLastEventId?: MemoryEventId + markerEventId?: MemoryEventId + missingEventIds?: MemoryEventId[] + } + | { + outcome: 'rejected' | 'failed' + reason: + | 'checksum-mismatch' + | 'repository-rejected' + | 'repository-failed' + | 'invalid-export' + | 'wrong-project' + | 'pagination-invalid' + | 'page-limit-exceeded' + revision?: number + checksum?: string + } + +type V1MigrationAuditScan = + | { + outcome: 'complete' + events: MemoryEventEnvelope[] + duplicateEventIds: MemoryEventId[] + wrongProjectEventIds: MemoryEventId[] + repositoryLastEventId?: MemoryEventId + } + | Extract + +async function scanV1MigrationAuditEvents( + repository: V1MigrationAuditReader, + projectId: ProjectId, +): Promise { + const events: MemoryEventEnvelope[] = [] + const eventIds = new Set() + const duplicateEventIds = new Set() + const wrongProjectEventIds = new Set() + const cursors = new Set() + let afterEventId: MemoryEventId | undefined + let repositoryLastEventId: MemoryEventId | undefined + + for (let page = 0; page < MAX_EXPORT_PAGES; page++) { + let raw: unknown + try { + raw = await repository.export({ + schemaVersion: 2, + projectId, + ...(afterEventId ? { afterEventId } : {}), + limit: 1_000, + }) + } catch { + return { outcome: 'failed', reason: 'repository-failed' } + } + const parsed = MemoryExportOutcomeSchema.safeParse(raw) + if (!parsed.success) return { outcome: 'failed', reason: 'invalid-export' } + const exported = parsed.data + if (exported.outcome !== 'page') { + return { + outcome: exported.outcome, + reason: + exported.outcome === 'rejected' + ? 'repository-rejected' + : 'repository-failed', + } + } + + for (const event of exported.events) { + if (event.projectId !== projectId) wrongProjectEventIds.add(event.eventId) + if (eventIds.has(event.eventId)) duplicateEventIds.add(event.eventId) + else eventIds.add(event.eventId) + events.push(event) + repositoryLastEventId = event.eventId + } + + const nextAfterEventId = exported.nextAfterEventId ?? undefined + if (!nextAfterEventId) { + return { + outcome: 'complete', + events, + duplicateEventIds: [...duplicateEventIds], + wrongProjectEventIds: [...wrongProjectEventIds], + repositoryLastEventId, + } + } + const pageLastEventId = exported.events.at(-1)?.eventId + if ( + !pageLastEventId || + nextAfterEventId !== pageLastEventId || + nextAfterEventId === afterEventId || + cursors.has(nextAfterEventId) + ) { + return { outcome: 'failed', reason: 'pagination-invalid' } + } + cursors.add(nextAfterEventId) + afterEventId = nextAfterEventId + } + + return { outcome: 'failed', reason: 'page-limit-exceeded' } +} + +function auditMigrationMarkerBody(params: { + marker: ImportedMarkerEvent + events: MemoryEventEnvelope[] + memory: TaskMemoryV1 + projectId: ProjectId + repositoryLastEventId: MemoryEventId +}): V1MigrationAuditOutcome { + const { marker, events, memory, projectId, repositoryLastEventId } = params + const { revision, checksum } = memory + const build = buildTaskMemoryV1Migration({ + memory, + projectId, + sessionId: marker.sessionId, + representationIdentity: marker.payload.legacyRecordKey, + }) + const markerResult = ( + outcome: 'incomplete' | 'mismatch', + reason: + | 'missing-imported-task' + | 'missing-imported-observations' + | 'imported-body-mismatch', + missingEventIds?: MemoryEventId[], + ): V1MigrationAuditOutcome => ({ + outcome, + reason, + revision, + checksum, + repositoryLastEventId, + markerEventId: marker.eventId, + ...(missingEventIds?.length + ? { missingEventIds: missingEventIds.slice(0, MAX_OBSERVATIONS) } + : {}), + }) + if (!equalEventDraft(build.markerDraft, marker)) { + return markerResult('mismatch', 'imported-body-mismatch') + } + const byEventId = new Map() + for (const event of events) { + const matching = byEventId.get(event.eventId) ?? [] + matching.push(event) + byEventId.set(event.eventId, matching) + } + const expectedTask = build.bodyDrafts[0]! + const taskMatches = byEventId.get(expectedTask.eventId) ?? [] + if (taskMatches.length === 0) { + return markerResult('incomplete', 'missing-imported-task') + } + if ( + taskMatches.length !== 1 || + !equalEventDraft(expectedTask, taskMatches[0]!) + ) { + return markerResult('mismatch', 'imported-body-mismatch') + } + + const missingObservationIds: MemoryEventId[] = [] + for (const expected of build.bodyDrafts.slice(1)) { + const matching = byEventId.get(expected.eventId) ?? [] + if (matching.length === 0) { + missingObservationIds.push(expected.eventId) + continue + } + if (matching.length !== 1 || !equalEventDraft(expected, matching[0]!)) { + return markerResult('mismatch', 'imported-body-mismatch') + } + } + if (missingObservationIds.length) { + return markerResult( + 'incomplete', + 'missing-imported-observations', + missingObservationIds, + ) + } + + return { + outcome: 'exact', + revision, + checksum, + identity: build.identity, + markerEventId: marker.eventId, + repositoryLastEventId, + importedTaskId: build.importedTaskId, + importedObservationIds: build.importedObservationIds, + omittedFields: build.omittedFields, + warnings: build.warnings, + sourceItemCounts: build.sourceItemCounts, + truncatedFields: build.truncatedFields, + } +} + +export async function auditTaskMemoryV1Migration(params: { + memory?: TaskMemoryV1 + projectId: ProjectId + repository: V1MigrationAuditReader +}): Promise { + const { memory, projectId, repository } = params + if (!memory) return { outcome: 'no-record' } + + const { revision, updatedAt, checksum, ...candidateDraft } = memory + const parsedDraft = taskMemoryDraftV1Schema.safeParse(candidateDraft) + if (!parsedDraft.success) { + return { + outcome: 'rejected', + reason: 'checksum-mismatch', + revision, + checksum, + } + } + const recomputed = stableHash( + JSON.stringify({ revision, updatedAt, memory: parsedDraft.data }), + ) + if (recomputed !== checksum) { + return { + outcome: 'rejected', + reason: 'checksum-mismatch', + revision, + checksum, + } + } + + const scanned = await scanV1MigrationAuditEvents(repository, projectId) + if (scanned.outcome !== 'complete') { + return { ...scanned, revision, checksum } + } + const identity = getV1MigrationIdentity({ projectId, revision, checksum }) + const matchingReservations = scanned.events.filter( + (event) => + event.eventType === 'migration.v1.reserved' && + event.payload.sourceRevision === revision && + event.payload.sourceChecksum === checksum && + event.payload.migrationId === identity, + ) + const conflictingReservation = scanned.events.some( + (event) => + event.eventType === 'migration.v1.reserved' && + event.payload.sourceRevision === revision && + (event.payload.sourceChecksum !== checksum || + event.payload.migrationId !== identity), + ) + const exactMarkers = scanned.events.filter( + (event): event is ImportedMarkerEvent => + event.eventType === 'migration.v1.imported' && + event.payload.sourceRevision === revision && + event.payload.sourceChecksum === checksum, + ) + const conflictingMarker = scanned.events.some( + (event) => + event.eventType === 'migration.v1.imported' && + event.payload.sourceRevision === revision && + event.payload.sourceChecksum !== undefined && + event.payload.sourceChecksum !== checksum, + ) + const legacyMarkers = scanned.events.filter( + (event): event is ImportedMarkerEvent => + event.eventType === 'migration.v1.imported' && + event.payload.legacyRecordKey === identity && + (event.payload.sourceRevision === undefined || + event.payload.sourceChecksum === undefined), + ) + const unverifiableRevisionMarker = scanned.events.some( + (event) => + event.eventType === 'migration.v1.imported' && + event.payload.sourceRevision === revision && + event.payload.sourceChecksum === undefined, + ) + const auditEvidence = { + revision, + checksum, + ...(scanned.repositoryLastEventId + ? { repositoryLastEventId: scanned.repositoryLastEventId } + : {}), + } + + if (exactMarkers.length > 0 && scanned.wrongProjectEventIds.length > 0) { + const build = buildTaskMemoryV1Migration({ + memory, + projectId, + sessionId: exactMarkers[0]!.sessionId, + }) + const expectedIds = new Set([ + build.markerDraft.eventId, + ...build.bodyDrafts.map((draft) => draft.eventId), + ]) + if ( + scanned.wrongProjectEventIds.some((eventId) => expectedIds.has(eventId)) + ) { + return { + outcome: 'mismatch', + reason: 'imported-body-mismatch', + markerEventId: exactMarkers[0]!.eventId, + ...auditEvidence, + } + } + return { outcome: 'failed', reason: 'wrong-project', revision, checksum } + } + if (scanned.wrongProjectEventIds.length > 0) { + return { outcome: 'failed', reason: 'wrong-project', revision, checksum } + } + if (scanned.duplicateEventIds.length > 0) { + if (exactMarkers.length > 0) { + const build = buildTaskMemoryV1Migration({ + memory, + projectId, + sessionId: exactMarkers[0]!.sessionId, + }) + const expectedIds = new Set([ + build.markerDraft.eventId, + ...build.bodyDrafts.map((draft) => draft.eventId), + ]) + if ( + scanned.duplicateEventIds.some((eventId) => expectedIds.has(eventId)) + ) { + return { + outcome: 'mismatch', + reason: 'imported-body-mismatch', + markerEventId: exactMarkers[0]!.eventId, + ...auditEvidence, + } + } + } + return { + outcome: 'failed', + reason: 'pagination-invalid', + revision, + checksum, + } + } + if (conflictingReservation || conflictingMarker) { + return { + outcome: 'mismatch', + reason: 'checksum-conflict', + ...auditEvidence, + } + } + if ( + matchingReservations.length > 1 || + legacyMarkers.length > 1 || + (unverifiableRevisionMarker && exactMarkers.length > 0) + ) { + return { + outcome: 'mismatch', + reason: 'revision-conflict', + ...auditEvidence, + } + } + if (exactMarkers.length > 0) { + let latestFailure: V1MigrationAuditOutcome | undefined + for (const marker of [...exactMarkers].reverse()) { + const audited = auditMigrationMarkerBody({ + marker, + events: scanned.events, + memory, + projectId, + repositoryLastEventId: scanned.repositoryLastEventId!, + }) + if (audited.outcome === 'exact') return audited + latestFailure ??= audited + } + return latestFailure! + } + if (legacyMarkers.length === 1) { + return { + outcome: 'incomplete', + reason: 'legacy-marker-unverifiable', + markerEventId: legacyMarkers[0]!.eventId, + ...auditEvidence, + } + } + if (unverifiableRevisionMarker) { + return { + outcome: 'mismatch', + reason: 'revision-conflict', + ...auditEvidence, + } + } + if (matchingReservations.length === 1) { + return { + outcome: 'incomplete', + reason: 'reservation-only', + ...auditEvidence, + } + } + return { outcome: 'not-migrated', ...auditEvidence } +} + +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ +export async function importTaskMemoryV1(params: { + memory?: TaskMemoryV1 + projectId: ProjectId + sessionId: MemorySessionId + repository: MemoryRepositoryV2 +}): Promise { + const { memory, projectId, sessionId, repository } = params + if (!memory) return { outcome: 'no-record' } + + const { revision, updatedAt, checksum, ...candidateDraft } = memory + const parsedDraft = taskMemoryDraftV1Schema.safeParse(candidateDraft) + if (!parsedDraft.success) { + return { + outcome: 'rejected', + reason: 'checksum-mismatch', + revision, + checksum, + } + } + const recomputed = stableHash( + JSON.stringify({ revision, updatedAt, memory: parsedDraft.data }), + ) + if (recomputed !== checksum) { + return { + outcome: 'rejected', + reason: 'checksum-mismatch', + revision, + checksum, + } + } + + let build = buildTaskMemoryV1Migration({ memory, projectId, sessionId }) + const identity = build.identity + let lookup: MigrationMarkerLookup + try { + lookup = await findMigrationMarker( + repository, + projectId, + revision, + checksum, + ) + } catch { + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } + } + if (lookup.state === 'conflict') { + return { + outcome: 'rejected', + reason: 'checksum-mismatch', + revision, + checksum, + } + } + if (lookup.state === 'exact') { + const audited = await auditTaskMemoryV1Migration({ + memory, + projectId, + repository, + }) + if (audited.outcome === 'exact') { + return outcomeFromMarker({ + marker: { + eventId: audited.markerEventId, + importedTaskId: audited.importedTaskId, + importedObservationIds: audited.importedObservationIds, + omittedFields: audited.omittedFields, + warnings: audited.warnings, + ...(audited.sourceItemCounts + ? { sourceItemCounts: audited.sourceItemCounts } + : {}), + ...(audited.truncatedFields === undefined + ? {} + : { truncatedFields: audited.truncatedFields }), + }, + revision, + checksum, + identity, + lastEventId: lookup.lastEventId, + }) + } + if (audited.outcome === 'rejected' || audited.outcome === 'failed') { + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } + } + const representationIdentity = `v1-repair:${hashToken( + `${identity}:${lookup.marker.eventId}:representation-2`, + )}` + build = buildTaskMemoryV1Migration({ + memory, + projectId, + sessionId, + representationIdentity, + }) + } + + const { + bodyIdentity, + occurredAt, + importedTaskId, + importedObservationIds, + omittedFields, + sourceItemCounts, + truncatedFields, + } = build + let expectedTail = lookup.lastEventId + ? { kind: 'event' as const, eventId: lookup.lastEventId } + : { kind: 'empty' as const } + const drafts = [...build.bodyDrafts] + if (lookup.prior) { const currentObservationIds = new Set(importedObservationIds) const retirementIdentity = `v1-retire:${hashToken( @@ -532,38 +1591,88 @@ export async function importTaskMemoryV1(params: { eventId: reservationEventId(identity), } - const reloadOwnership = async (): Promise => { + const reloadOwnership = async (): Promise< + MigrationMarkerLookup | V1MigrationOutcome + > => { try { - const reloaded = await findMigrationMarker(repository, projectId, revision, checksum) + const reloaded = await findMigrationMarker( + repository, + projectId, + revision, + checksum, + ) if (reloaded.state === 'conflict') { - return { outcome: 'rejected', reason: 'checksum-mismatch', revision, checksum } + return { + outcome: 'rejected', + reason: 'checksum-mismatch', + revision, + checksum, + } } return reloaded } catch { - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } } if (!lookup.reserved) { try { const reserved = MemoryAppendOutcomeSchema.parse( - await repository.append(MemoryAppendRequestSchema.parse({ - schemaVersion: 2, - projectId, - expectedTail, - events: [reservation], - })), + await repository.append( + MemoryAppendRequestSchema.parse({ + schemaVersion: 2, + projectId, + expectedTail, + events: [reservation], + }), + ), ) if (reserved.outcome === 'appended') { expectedTail = { kind: 'event', eventId: reserved.lastEventId } - lookup = { ...lookup, reserved: true, lastEventId: reserved.lastEventId } - } else if (reserved.error.code === 'conflict' && reserved.error.retryable) { + lookup = { + ...lookup, + reserved: true, + lastEventId: reserved.lastEventId, + events: [ + ...lookup.events, + { ...reservation, sequence: reserved.entries[0]!.sequence }, + ], + } + } else if ( + reserved.error.code === 'conflict' && + reserved.error.retryable + ) { const reloaded = await reloadOwnership() if ('outcome' in reloaded) return reloaded if (reloaded.state === 'exact') { - return outcomeFromMarker({ marker: reloaded.marker, revision, checksum, identity, lastEventId: reloaded.lastEventId }) + const validatedMarker = validatedCompleteBuildMarker(reloaded, build) + if (!validatedMarker) + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } + return outcomeFromMarker({ + marker: validatedMarker, + revision, + checksum, + identity, + lastEventId: reloaded.lastEventId, + }) } - if (!reloaded.reserved) return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + if (!reloaded.reserved) + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } lookup = reloaded expectedTail = reloaded.lastEventId ? { kind: 'event', eventId: reloaded.lastEventId } @@ -571,29 +1680,62 @@ export async function importTaskMemoryV1(params: { } else { return { outcome: reserved.outcome === 'rejected' ? 'rejected' : 'failed', - reason: reserved.outcome === 'rejected' ? 'repository-rejected' : 'repository-failed', + reason: + reserved.outcome === 'rejected' + ? 'repository-rejected' + : 'repository-failed', revision, checksum, } } } catch { - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } } - const revalidateAfterConflict = async (): Promise => { + const revalidateAfterConflict = async (): Promise< + MigrationMarkerLookup | V1MigrationOutcome + > => { let revalidated: MigrationMarkerLookup try { - revalidated = await findMigrationMarker(repository, projectId, revision, checksum) + revalidated = await findMigrationMarker( + repository, + projectId, + revision, + checksum, + ) } catch { - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } if (revalidated.state === 'conflict') { - return { outcome: 'rejected', reason: 'checksum-mismatch', revision, checksum } + return { + outcome: 'rejected', + reason: 'checksum-mismatch', + revision, + checksum, + } } if (revalidated.state === 'exact') { + const validatedMarker = validatedCompleteBuildMarker(revalidated, build) + if (!validatedMarker) + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } return outcomeFromMarker({ - marker: revalidated.marker, + marker: validatedMarker, revision, checksum, identity, @@ -607,7 +1749,7 @@ export async function importTaskMemoryV1(params: { let recoveries = 0 while (true) { const remainingDrafts = drafts.slice(offset, offset + APPEND_PAGE_SIZE) - const existingIds = new Map() + const existingEvents = new Map() try { let afterEventId: MemoryEventId | undefined for (let page = 0; page < MAX_EXPORT_PAGES; page++) { @@ -617,15 +1759,40 @@ export async function importTaskMemoryV1(params: { ...(afterEventId ? { afterEventId } : {}), limit: 1_000, }) - if (exported.outcome !== 'page') throw new Error('Migration progress lookup failed') - for (const event of exported.events) existingIds.set(event.eventId, event.eventId) + if (exported.outcome !== 'page') + throw new Error('Migration progress lookup failed') + for (const event of exported.events) { + const matching = existingEvents.get(event.eventId) ?? [] + matching.push(event) + existingEvents.set(event.eventId, matching) + } if (!exported.nextAfterEventId) break afterEventId = exported.nextAfterEventId } } catch { - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } - const events = remainingDrafts.filter((draft) => !existingIds.has(draft.eventId)) + for (const draft of remainingDrafts) { + const matching = existingEvents.get(draft.eventId) ?? [] + if ( + matching.length > 0 && + (matching.length !== 1 || !equalEventDraft(draft, matching[0]!)) + ) + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } + } + const events = remainingDrafts.filter( + (draft) => !existingEvents.has(draft.eventId), + ) if (events.length === 0) break let outcome try { @@ -640,13 +1807,22 @@ export async function importTaskMemoryV1(params: { ), ) } catch { - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } if (outcome.outcome === 'appended') { expectedTail = { kind: 'event', eventId: outcome.lastEventId } break } - if (outcome.error.code === 'conflict' && outcome.error.retryable && recoveries < 2) { + if ( + outcome.error.code === 'conflict' && + outcome.error.retryable && + recoveries < 2 + ) { recoveries++ const recovered = await revalidateAfterConflict() if ('outcome' in recovered) return recovered @@ -657,39 +1833,17 @@ export async function importTaskMemoryV1(params: { } return { outcome: outcome.outcome === 'rejected' ? 'rejected' : 'failed', - reason: outcome.outcome === 'rejected' ? 'repository-rejected' : 'repository-failed', + reason: + outcome.outcome === 'rejected' + ? 'repository-rejected' + : 'repository-failed', revision, checksum, } } } - const warningList = [...warnings].slice(0, 100) - const marker = withMigrationEventId( - createMemoryEventDraft({ - projectId, - sessionId, - userInputId: identity, - sourceIndex: MAX_OBSERVATIONS + 1, - occurredAt, - eventType: 'migration.v1.imported', - payload: { - payloadSchemaVersion: 1, - sourceSchemaVersion: 1, - legacyRecordKey: identity, - importedTaskId, - importedObservationIds, - sourceRevision: revision, - sourceChecksum: checksum, - omittedFields, - sourceItemCounts, - truncatedFields, - warnings: warningList, - }, - }), - identity, - MAX_OBSERVATIONS + 1, - ) + const marker = build.markerDraft try { const markerOutcome = MemoryAppendOutcomeSchema.parse( await repository.append( @@ -702,10 +1856,18 @@ export async function importTaskMemoryV1(params: { ), ) if (markerOutcome.outcome !== 'appended') { - if (markerOutcome.error.code === 'conflict' && markerOutcome.error.retryable) { + if ( + markerOutcome.error.code === 'conflict' && + markerOutcome.error.retryable + ) { const recovered = await revalidateAfterConflict() if ('outcome' in recovered) return recovered - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } return { outcome: markerOutcome.outcome === 'rejected' ? 'rejected' : 'failed', @@ -718,6 +1880,18 @@ export async function importTaskMemoryV1(params: { } } const markerDuplicate = markerOutcome.entries[0]?.duplicate === true + const completed = await reloadOwnership() + if ( + 'outcome' in completed || + completed.state !== 'exact' || + !validatedCompleteBuildMarker(completed, build) + ) + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } return { outcome: markerDuplicate ? 'no-op' : 'imported', revision, @@ -726,13 +1900,18 @@ export async function importTaskMemoryV1(params: { importedTaskId, importedObservationIds, omittedFields, - warnings: warningList, - lastEventId: markerOutcome.lastEventId, + warnings: build.warnings, + lastEventId: completed.lastEventId, sourceItemCounts, truncatedFields, } } catch { - return { outcome: 'failed', reason: 'repository-failed', revision, checksum } + return { + outcome: 'failed', + reason: 'repository-failed', + revision, + checksum, + } } } @@ -741,7 +1920,9 @@ function TaskIdFor(identity: string): TaskId { } function ObservationIdFor(identity: string, index: number): ObservationId { - return ObservationIdSchema.parse(`observation:migration:${hashToken(`${identity}:${index}`)}`) + return ObservationIdSchema.parse( + `observation:migration:${hashToken(`${identity}:${index}`)}`, + ) } function withMigrationEventId( diff --git a/sdk/src/services/task-memory-store.ts b/sdk/src/services/task-memory-store.ts index 44adaadd6a..d5b7d782c8 100644 --- a/sdk/src/services/task-memory-store.ts +++ b/sdk/src/services/task-memory-store.ts @@ -1,5 +1,5 @@ import { createHash, randomUUID } from 'node:crypto' -import type { Stats } from 'node:fs' +import type { PathLike, Stats } from 'node:fs' import * as nodeFsPromises from 'node:fs/promises' import path from 'node:path' @@ -8,14 +8,21 @@ import { taskMemoryDraftV1Schema, taskMemoryV1Schema, } from '@codebuff/common/types/task-memory' +import { errorCode } from '@codebuff/common/util/error' import { stableHash } from '@codebuff/common/util/stable-hash' -import type { CodebuffFileSystem } from '@codebuff/common/types/filesystem' +import type { + CodebuffFileContent, + CodebuffFileSystem, +} from '@codebuff/common/types/filesystem' import type { TaskMemoryEvidenceV1, TaskMemoryV1, } from '@codebuff/common/types/task-memory' +/** + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. + */ export interface WorkspaceMoveRecord { from: string to: string @@ -29,6 +36,8 @@ type FsModule = typeof nodeFsPromises * them as the full `FsModule` would let callers invoke anything else (e.g. * `copyFile`) and hit a silent runtime `undefined` instead of a compile * error. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export interface TaskMemoryStoreFs { mkdir( @@ -37,10 +46,15 @@ export interface TaskMemoryStoreFs { ): Promise readFile(file: Parameters[0]): Promise /** - * Partial-read primitive; exposing it enables streamed hashing of - * oversized evidence files. Stores without it never buffer past - * MAX_EVIDENCE_HASH_BYTES: hashFile treats an oversized target as - * unverifiable (stale) rather than reading it whole. + * Explicit atomic-create capability used for cross-process locking. Merely + * accepting node's `wx` write option is not sufficient: virtual adapters + * have historically ignored it while reporting a successful write. + */ + createFileExclusive?(file: PathLike, data: CodebuffFileContent): Promise + /** + * Partial-read primitive used to hash large evidence files incrementally. + * Stores without it treat large targets as unverifiable rather than change + * the digest contract or buffer an unbounded body. */ open?: FsModule['open'] rename( @@ -48,9 +62,7 @@ export interface TaskMemoryStoreFs { newPath: Parameters[1], ): Promise stat(path: Parameters[0]): Promise - lstat?(path: Parameters[0]): Promise realpath?(path: Parameters[0]): Promise - readlink?(path: Parameters[0]): Promise unlink(path: Parameters[0]): Promise writeFile( file: Parameters[0], @@ -72,14 +84,14 @@ function truncateToCap(values: T[], cap: number): T[] { return values.length > cap ? values.slice(-cap) : values } -/** - * Upper bound for hashing a single evidence target during reconciliation - * (same spirit as MAX_DISCOVERED_PROJECT_READ_BYTES in run-state.ts). - * Freshness digests only need a stable prefix; capping bounds memory and - * cold-cache read cost per evidence item. Known trade-off: mutations beyond - * the cap do not flip staleness. - */ -const MAX_EVIDENCE_HASH_BYTES = 1_000_000 +/** Files above this size are hashed incrementally instead of buffered whole. */ +const EVIDENCE_STREAM_THRESHOLD_BYTES = 1_000_000 +const WHOLE_CONTENT_HASH_PREFIX = 'sha256-whole:' +const LEGACY_PREFIX_HASH_PREFIX = 'sha256-prefix-1m:' +const SHA256_HEX = /^[0-9a-f]{64}$/ + +/** Fixed working buffer for whole-content hashing of large evidence files. */ +const EVIDENCE_HASH_BUFFER_BYTES = 64 * 1024 /** Evidence reads processed per Promise.all batch during reconciliation. */ const EVIDENCE_HASH_CONCURRENCY = 16 @@ -91,57 +103,97 @@ const EVIDENCE_HASH_CONCURRENCY = 16 */ const IN_PROCESS_MEMORY_LOCKS = new Map>() -/** Bounded wait for another process's lock: 50 * 20ms ≈ 1s, then degrade. */ +/** Bounded wait for another process's lock: failure skips the write. */ const LOCK_ACQUIRE_ATTEMPTS = 50 const LOCK_RETRY_DELAY_MS = 20 +const DEFAULT_TASK_MEMORY_FS: TaskMemoryStoreFs = { + ...nodeFsPromises, + createFileExclusive: async (file, data) => { + await nodeFsPromises.writeFile(file, data, { flag: 'wx', mode: 0o600 }) + }, +} + +function taskMemoryFs(fs: TaskMemoryStoreFs | undefined): TaskMemoryStoreFs { + if (!fs) return DEFAULT_TASK_MEMORY_FS + // Direct node:fs/promises adapters have native exclusive-create semantics. + // Virtual adapters must instead opt in through createFileExclusive. + if (fs === nodeFsPromises && typeof fs.createFileExclusive !== 'function') { + return { + ...fs, + createFileExclusive: DEFAULT_TASK_MEMORY_FS.createFileExclusive, + } + } + return fs +} + +export type TaskMemoryV1Inspection = + | { status: 'absent' } + | { status: 'valid'; memory: TaskMemoryV1 } + | { + status: 'invalid' + reason: 'malformed-json' | 'schema-invalid' | 'checksum-mismatch' + } + | { status: 'unreadable'; reason: 'read-failed' } + /** - * A lock older than this is treated as abandoned by a crashed writer and - * reclaimed. Locked sections are short (one load plus one rename), so a - * lock this old cannot belong to a live writer. + * Inspect the persisted V1 record without mutating storage. The bounded result + * deliberately exposes no path, contents, validation issues, or raw read + * error. Only ENOENT proves absence; every other read failure is unreadable. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ -const LOCK_STALE_MS = 10_000 +export async function inspectPersistedTaskMemoryV1(params: { + rootDir: string + fs?: Pick +}): Promise { + const fs = params.fs ?? nodeFsPromises + let raw: Buffer + try { + raw = await fs.readFile(getMemoryFilePath(params.rootDir)) + } catch (error) { + return errorCode(error) === 'ENOENT' + ? { status: 'absent' } + : { status: 'unreadable', reason: 'read-failed' } + } + + let candidate: unknown + try { + candidate = JSON.parse(raw.toString('utf8')) + } catch { + return { status: 'invalid', reason: 'malformed-json' } + } + + const parsed = taskMemoryV1Schema.safeParse(candidate) + if (!parsed.success) return { status: 'invalid', reason: 'schema-invalid' } + const parsedDraft = taskMemoryDraftV1Schema.safeParse(parsed.data) + if (!parsedDraft.success) + return { status: 'invalid', reason: 'schema-invalid' } + const expectedChecksum = stableHash( + JSON.stringify({ + revision: parsed.data.revision, + updatedAt: parsed.data.updatedAt, + memory: parsedDraft.data, + }), + ) + if (expectedChecksum !== parsed.data.checksum) { + return { status: 'invalid', reason: 'checksum-mismatch' } + } + return { status: 'valid', memory: parsed.data } +} /** - * Load and schema-validate the persisted task memory for a project root, - * re-verifying the stored checksum against the loaded payload. Missing, - * corrupt, or checksum-mismatched data yields undefined; never throws. + * Load and schema-validate persisted task memory. Compatibility callers keep + * the historical behavior: every non-valid inspection collapses to undefined. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export async function loadPersistedTaskMemory(params: { rootDir: string fs?: TaskMemoryStoreFs }): Promise { - const fs = params.fs ?? nodeFsPromises - try { - const raw = await fs.readFile(getMemoryFilePath(params.rootDir)) - const parsed = taskMemoryV1Schema.safeParse( - JSON.parse(raw.toString('utf8')), - ) - if (!parsed.success) return undefined - // A corrupted-but-schema-valid file must not hydrate silently: recompute - // the checksum over the same draft-shaped payload saveMergedTaskMemory - // hashed (memory WITHOUT the revision/updatedAt/checksum envelope) and - // reject mismatches. - // - // Compatibility: records written before checksum enforcement are also - // rejected here (fail-closed). No pre-checksum format was ever deployed - // — this store shipped alongside enforcement — so tolerate-and-upgrade - // would only weaken the corruption guard above. Revisit only if a - // deployed legacy format ever materializes. - const parsedDraft = taskMemoryDraftV1Schema.safeParse(parsed.data) - if (!parsedDraft.success) return undefined - const expectedChecksum = stableHash( - JSON.stringify({ - revision: parsed.data.revision, - updatedAt: parsed.data.updatedAt, - memory: parsedDraft.data, - }), - ) - if (expectedChecksum !== parsed.data.checksum) return undefined - return parsed.data - } catch { - return undefined - } + const inspected = await inspectPersistedTaskMemoryV1(params) + return inspected.status === 'valid' ? inspected.memory : undefined } // Evidence paths and journal destinations are stored with forward slashes; @@ -181,54 +233,137 @@ function resolveMoveTarget( return visited.size > 1 ? current : relativePath } +type EvidenceFileHashes = + | { status: 'hashed'; whole: string; legacyPrefix?: string } + | { status: 'missing' } + | { status: 'outside' } + | { status: 'unverifiable' } + +/** + * Open, contain, and hash one stable file descriptor. Opening before canonical + * validation closes the check/use gap: even if an ancestor pathname is + * replaced while realpath runs, descriptor identity must still match the + * contained canonical target before any bytes are read. Every size uses the + * same fixed buffer, and reads stop at the descriptor's initial size so growth + * cannot turn a small-file stat into an unbounded allocation or read. + */ async function hashFile( fs: TaskMemoryStoreFs, + rootDir: string, absolutePath: string, -): Promise { +): Promise { + if (!isPathInsideRootLexical(rootDir, absolutePath)) { + return { status: 'outside' } + } + if (typeof fs.open !== 'function' || typeof fs.realpath !== 'function') { + return { status: 'unverifiable' } + } + + let handle: Awaited> | undefined try { - const stats = await fs.stat(absolutePath) - if (stats.size > MAX_EVIDENCE_HASH_BYTES) { - if (typeof fs.open !== 'function') { - // Oversized target with no partial-read primitive: refuse to buffer - // a multi-GB body just to feed the digest. Returning undefined marks - // the entry stale (unverifiable) instead of risking OOM at session - // start. - return undefined - } - // Read only the leading bytes so a multi-GB evidence file never gets - // buffered whole just to feed the digest. - const handle = await fs.open(absolutePath, 'r') - try { - const leading = Buffer.alloc(MAX_EVIDENCE_HASH_BYTES) - const { bytesRead } = await handle.read( - leading, - 0, - MAX_EVIDENCE_HASH_BYTES, - 0, + handle = await fs.open(absolutePath, 'r') + const realRoot = await fs.realpath(rootDir) + const realCandidate = await fs.realpath(absolutePath) + if (!isPathInsideRootLexical(realRoot, realCandidate)) { + return { status: 'outside' } + } + + const [openedStats, canonicalStats] = await Promise.all([ + handle.stat(), + fs.stat(realCandidate), + ]) + if ( + !openedStats.isFile() || + openedStats.dev !== canonicalStats.dev || + openedStats.ino !== canonicalStats.ino || + !Number.isSafeInteger(openedStats.size) || + openedStats.size < 0 + ) { + return { status: 'unverifiable' } + } + + const wholeDigest = createHash('sha256') + const legacyPrefixDigest = createHash('sha256') + const buffer = Buffer.allocUnsafe(EVIDENCE_HASH_BUFFER_BYTES) + let position = 0 + let legacyBytes = 0 + while (position < openedStats.size) { + const length = Math.min(buffer.length, openedStats.size - position) + const { bytesRead } = await handle.read(buffer, 0, length, position) + if (bytesRead === 0) return { status: 'unverifiable' } + const bytes = buffer.subarray(0, bytesRead) + wholeDigest.update(bytes) + if (legacyBytes < EVIDENCE_STREAM_THRESHOLD_BYTES) { + const prefixBytes = Math.min( + bytesRead, + EVIDENCE_STREAM_THRESHOLD_BYTES - legacyBytes, ) - return createHash('sha256') - .update(leading.subarray(0, bytesRead)) - .digest('hex') - } finally { - try { - await handle.close() - } catch { - // Ignore: the digest is already computed. - } + legacyPrefixDigest.update(bytes.subarray(0, prefixBytes)) + legacyBytes += prefixBytes } + position += bytesRead } - let contents = await fs.readFile(absolutePath) - if (contents.length > MAX_EVIDENCE_HASH_BYTES) { - // Growth raced past the cap between stat and read: still digest only - // the leading bytes. - contents = contents.subarray(0, MAX_EVIDENCE_HASH_BYTES) + + const finalStats = await handle.stat() + if ( + finalStats.dev !== openedStats.dev || + finalStats.ino !== openedStats.ino || + finalStats.size !== openedStats.size || + finalStats.mtimeMs !== openedStats.mtimeMs || + finalStats.ctimeMs !== openedStats.ctimeMs + ) { + return { status: 'unverifiable' } } - return createHash('sha256').update(contents).digest('hex') - } catch { - return undefined + return { + status: 'hashed', + whole: wholeDigest.digest('hex'), + legacyPrefix: legacyPrefixDigest.digest('hex'), + } + } catch (error) { + return handle === undefined && errorCode(error) === 'ENOENT' + ? { status: 'missing' } + : { status: 'unverifiable' } + } finally { + await handle?.close().catch(() => {}) } } +function reconcileFreshnessHash( + recorded: string | undefined, + hashes: Extract, +): { status: 'fresh' | 'stale' | 'unverifiable'; freshnessHash?: string } { + if (recorded === undefined) return { status: 'fresh' } + + let expected: string + let matches: boolean + if (recorded.startsWith(WHOLE_CONTENT_HASH_PREFIX)) { + expected = recorded.slice(WHOLE_CONTENT_HASH_PREFIX.length) + if (!SHA256_HEX.test(expected)) return { status: 'unverifiable' } + matches = expected === hashes.whole + } else if (recorded.startsWith(LEGACY_PREFIX_HASH_PREFIX)) { + expected = recorded.slice(LEGACY_PREFIX_HASH_PREFIX.length) + if (!SHA256_HEX.test(expected)) return { status: 'unverifiable' } + // For files at or below the historical prefix limit, the prefix digest is + // identical to the whole-content digest and hashFile need not produce both. + matches = expected === (hashes.legacyPrefix ?? hashes.whole) + } else if (SHA256_HEX.test(recorded)) { + // Unversioned records may be either the historical one-megabyte prefix + // digest or the newer whole-content digest. A match can be upgraded. A + // mismatch is still reported stale by reconciliation, but prune separately + // refuses to delete a present file on this ambiguous evidence alone. + matches = recorded === hashes.whole || recorded === hashes.legacyPrefix + } else { + return { status: 'unverifiable' } + } + + return matches + ? { + status: 'fresh', + freshnessHash: `${WHOLE_CONTENT_HASH_PREFIX}${hashes.whole}`, + } + : { status: 'stale' } +} + function isPathInsideRootLexical( rootDir: string, candidatePath: string, @@ -245,47 +380,12 @@ function isPathInsideRootLexical( ) } -/** - * True when candidatePath resolves inside rootDir. Evidence paths originate - * from persisted state and journal destinations, so anything resolving - * outside the project root is treated as untrusted and never read. - * - * Lexical check is followed by a symlink-escape guard when the filesystem - * exposes lstat/realpath: a symlink planted inside rootDir that points - * outside is treated as outside and never hashed. Exposure on adapters - * without those primitives remains limited to feeding outside bytes into the - * freshness digest (contents are never surfaced to callers). Degrades to - * lexical-only when those primitives are unavailable or the target is - * missing (hashFile will then mark stale). - */ -async function isPathInsideRoot( - rootDir: string, - candidatePath: string, - fs: TaskMemoryStoreFs, -): Promise { - if (!isPathInsideRootLexical(rootDir, candidatePath)) return false - if (typeof fs.lstat !== 'function' || typeof fs.realpath !== 'function') { - return true - } - try { - const lst = await fs.lstat(candidatePath) - const isSymlink = - typeof (lst as unknown as { isSymbolicLink?: () => boolean }) - .isSymbolicLink === 'function' - ? (lst as unknown as { isSymbolicLink: () => boolean }).isSymbolicLink() - : false - if (!isSymlink) return true - const real = await fs.realpath(candidatePath) - return isPathInsideRootLexical(rootDir, real) - } catch { - return true - } -} - /** * Re-evaluate each evidence item against current disk state. Missing or * changed files mark the entry stale; a matching workspace move rebinds the * path to its destination before evaluating. Entries are never deleted. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export async function reconcileTaskMemoryEvidence(params: { memory: TaskMemoryV1 @@ -293,7 +393,7 @@ export async function reconcileTaskMemoryEvidence(params: { fs?: TaskMemoryStoreFs workspaceMoves?: WorkspaceMoveRecord[] }): Promise { - const fs = params.fs ?? nodeFsPromises + const fs = taskMemoryFs(params.fs) const reconcileItem = async ( item: TaskMemoryEvidenceV1, ): Promise => { @@ -302,24 +402,43 @@ export async function reconcileTaskMemoryEvidence(params: { } const boundPath = resolveMoveTarget(item.path, params.workspaceMoves) const absolutePath = path.join(params.rootDir, boundPath) - if (!(await isPathInsideRoot(params.rootDir, absolutePath, fs))) { - // evidence.path comes from persisted state and journal destinations; - // refuse to read (or adopt) any path resolving outside the root, - // including symlink-escape when the filesystem exposes lstat/realpath. + const hashes = await hashFile(fs, params.rootDir, absolutePath) + if (hashes.status === 'outside') { + // A proven lexical or descriptor-validated canonical escape is stale and + // no bytes from the opened target are read. return { ...item, stale: true } } - const digest = await hashFile(fs, absolutePath) - const isFresh = - digest !== undefined && - (item.freshnessHash === undefined || digest === item.freshnessHash) - return isFresh - ? { ...item, path: boundPath, stale: false, verifiedAt: Date.now() } + if (hashes.status === 'missing') { + // ENOENT while opening is positive evidence that the file is gone. + return { ...item, path: boundPath, stale: true } + } + if (hashes.status === 'unverifiable') { + // Capability/read failures and unknown hash formats are not evidence + // that the file changed. Preserve the prior verdict so prune cannot + // destroy evidence merely because this adapter cannot validate it. + return { ...item, path: boundPath } + } + const freshness = reconcileFreshnessHash(item.freshnessHash, hashes) + if (freshness.status === 'unverifiable') { + // Unknown formats and read-capability failures cannot establish a new + // verdict. Preserve the prior state; prune treats them non-destructively. + return { ...item, path: boundPath } + } + return freshness.status === 'fresh' + ? { + ...item, + path: boundPath, + ...(freshness.freshnessHash + ? { freshnessHash: freshness.freshnessHash } + : {}), + stale: false, + verifiedAt: Date.now(), + } : { ...item, path: boundPath, stale: true } } - // Evidence lists are capped (LIST_CAPS.evidence), but each item may read - // up to MAX_EVIDENCE_HASH_BYTES, so hashing every entry concurrently - // spiked ~256MB of transient buffers on a cold cache. Chunks keep reads - // pipelined while bounding peak memory; output order follows input. + // Evidence lists are capped (LIST_CAPS.evidence), and large files use a + // fixed streaming buffer. Chunks bound the number of simultaneous file + // descriptors and buffered small-file reads while preserving output order. const evidence: TaskMemoryEvidenceV1[] = [] for ( let start = 0; @@ -393,6 +512,8 @@ function collectDroppedEvidenceIds( * longer carries but the caller's hydrated snapshot still does is treated as * deliberately dropped by the other writer and filtered out of the run's * still-hydrated `evidence` instead of being merged back in. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export async function saveMergedTaskMemory(params: { rootDir: string @@ -402,7 +523,7 @@ export async function saveMergedTaskMemory(params: { }): Promise { const { runMemory } = params if (!runMemory) return undefined - const fs = params.fs ?? nodeFsPromises + const fs = taskMemoryFs(params.fs) // Load, revision derivation and commit run as one serialized section: the // revision this save reads from disk must not also be publishable by the // other writer (see withMemoryFileLock). @@ -576,64 +697,44 @@ async function writeRecordAtomically( /** * Serialize the load→revision→commit section shared by both task-memory - * writers. - * - * `saveMergedTaskMemory` and `pruneStaleTaskMemoryEvidence` each read the - * record, derive the next `revision` from what they read, and commit by - * rename. With no mutual exclusion those windows interleave — a save landing - * between prune's revision check and its rename, or two saves loading the - * same record — and both writers publish the SAME revision with different - * payloads, which is exactly what the record's monotonic-and-unique revision - * contract forbids. - * - * Two layers, because neither alone covers the writers: - * - an in-process promise chain keyed by the resolved record path, which - * handles the common case (a CLI `/memory prune` and an SDK save in one - * process) without needing any filesystem support; - * - an exclusive-create (`wx`) lock file for other processes sharing the - * project, reclaimed after {@link LOCK_STALE_MS} so a crashed writer cannot - * block later ones forever. - * - * The cross-process layer is advisory: an adapter that ignores the `flag` - * option, or a lock that cannot be taken within the bounded attempt budget, - * degrades to running the section unlocked rather than dropping the write. - * Both callers keep their own on-disk re-read (save) and revision guard - * (prune), so the degraded path is exactly as safe as before this lock - * existed. + * writers. The in-process chain covers callers in this process, while an + * exclusive-create lock covers other processes. * - * Evidence reconciliation deliberately stays OUTSIDE the section: it hashes - * every evidence file, so holding the lock across it would stall the other - * writer, and a run finishing mid-prune (which reconciliation's own IO can - * trigger) would deadlock against it. + * Cross-process exclusion is mandatory: failure to acquire the lock returns + * `undefined` and the write is skipped. Acquisition requires an explicit + * atomic-create capability, rather than trusting an adapter's handling of + * `writeFile(..., { flag: 'wx' })`. An old same-host lock is reclaimed only + * after its recorded owner process is proven dead; age alone never steals a + * live writer's lock. Automatic stale-lock deletion is deliberately avoided: + * portable filesystem APIs cannot atomically compare ownership and unlink, so + * reclaiming could remove a replacement owner's lock. The unique token is + * checked before release. */ async function withMemoryFileLock( fs: TaskMemoryStoreFs, filePath: string, section: () => Promise, -): Promise { +): Promise { const key = path.resolve(filePath) const previous = IN_PROCESS_MEMORY_LOCKS.get(key) ?? Promise.resolve() - const run = async (): Promise => { - // The lock file sits beside the record, so its directory must exist first. - // A failing mkdir is not fatal here: the section's own write path creates - // the directory and reports the failure through its normal outcome. + const run = async (): Promise => { await fs.mkdir(path.dirname(filePath), { recursive: true }).catch(() => {}) const lockPath = `${filePath}.lock` - const locked = await acquireRecordLock(fs, lockPath) + const token = await acquireRecordLock(fs, lockPath) + if (!token) return undefined try { return await section() } finally { - if (locked) { - await fs.unlink(lockPath).catch(() => { - // Ignore: a reclaimed or already-removed lock is not this writer's - // problem, and the section has already committed. - }) + const currentToken = await fs + .readFile(lockPath) + .then((contents) => contents.toString('utf8')) + .catch(() => undefined) + if (currentToken === token) { + await fs.unlink(lockPath).catch(() => {}) } } } const result = previous.then(run, run) - // Keep the chain alive across failures: one rejected section must not - // poison every later writer in this process. const settled = result.then( () => {}, () => {}, @@ -647,47 +748,41 @@ async function withMemoryFileLock( return result } +type RecordLockPayload = { + token: string + pid: number + createdAt: number +} + /** - * Take the cross-process lock file, or report that this writer is proceeding - * without it. Returns true only when the lock was created here, so the caller - * never unlinks a lock it does not hold. + * Take the cross-process lock and return its ownership bytes. Contention is + * bounded, but exhaustion and adapters without explicit atomic create fail + * closed: callers never enter the mutation section without proven exclusion. */ async function acquireRecordLock( fs: TaskMemoryStoreFs, lockPath: string, -): Promise { - let missingLockObservations = 0 +): Promise { + if (typeof fs.createFileExclusive !== 'function') return undefined + const payload: RecordLockPayload = { + token: randomUUID(), + pid: process.pid, + createdAt: Date.now(), + } + const token = `${JSON.stringify(payload)}\n` for (let attempt = 0; attempt < LOCK_ACQUIRE_ATTEMPTS; attempt += 1) { try { - await fs.writeFile(lockPath, `${process.pid}\n`, { - flag: 'wx', - mode: 0o600, - }) - return true + await fs.createFileExclusive(lockPath, token) + const observed = await fs.readFile(lockPath) + if (observed.toString('utf8') !== token) return undefined + return token } catch { - const stats = await fs.stat(lockPath).catch(() => undefined) - if (!stats) { - // Nothing holds the lock, so the rejection was not contention: this - // adapter does not honor exclusive-create. Give up after a second - // observation (which absorbs a holder releasing between the write and - // this stat) instead of burning the whole budget on every write. - missingLockObservations += 1 - if (missingLockObservations >= 2) return false - continue - } - missingLockObservations = 0 - // Reclaim a lock abandoned by a crashed writer. A stat without a usable - // mtime keeps waiting rather than stealing a possibly-live lock. - if (Date.now() - stats.mtimeMs > LOCK_STALE_MS) { - await fs.unlink(lockPath).catch(() => { - // Ignore: another writer may have reclaimed it first. - }) - continue + if (attempt + 1 < LOCK_ACQUIRE_ATTEMPTS) { + await new Promise((resolve) => setTimeout(resolve, LOCK_RETRY_DELAY_MS)) } - await new Promise((resolve) => setTimeout(resolve, LOCK_RETRY_DELAY_MS)) } } - return false + return undefined } /** @@ -699,6 +794,8 @@ async function acquireRecordLock( * user as "nothing to prune". On `failed`, `removed`/`remaining` describe the * prune that WOULD have been written, so the record still holds `removed` * stale entries. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export type TaskMemoryPruneOutcome = | { status: 'pruned'; removed: number; remaining: number } @@ -710,6 +807,53 @@ export type TaskMemoryPruneOutcome = remaining: number } +/** + * Decide whether the current disk state supplies format-compatible authority + * to remove one reconciled item. A persisted or inherited `stale` flag is + * never sufficient by itself: older writers used unversioned hashes, and + * future writers may introduce formats this process cannot interpret. + * + * Missing and escaping targets are independently destructive-safe. For a + * present contained target, only a syntactically valid, explicitly versioned + * digest can prove a content mismatch. Bare and unknown digests are always + * preserved, as are pathless items and any item whose current bytes cannot be + * read. This deliberately repeats the bounded hash after reconciliation so a + * transient read failure cannot turn an inherited stale verdict into deletion + * authority. + */ +async function canPruneReconciledEvidence(params: { + item: TaskMemoryEvidenceV1 + recordedHash: string | undefined + rootDir: string + fs: TaskMemoryStoreFs +}): Promise { + const { item, recordedHash, rootDir, fs } = params + if (item.stale !== true || !item.path) return false + + const absolutePath = path.join(rootDir, item.path) + const hashes = await hashFile(fs, rootDir, absolutePath) + if (hashes.status === 'missing' || hashes.status === 'outside') return true + if (hashes.status !== 'hashed' || recordedHash === undefined) return false + + let expected: string + let algorithm: 'whole' | 'legacy-prefix' + if (recordedHash.startsWith(WHOLE_CONTENT_HASH_PREFIX)) { + expected = recordedHash.slice(WHOLE_CONTENT_HASH_PREFIX.length) + algorithm = 'whole' + } else if (recordedHash.startsWith(LEGACY_PREFIX_HASH_PREFIX)) { + expected = recordedHash.slice(LEGACY_PREFIX_HASH_PREFIX.length) + algorithm = 'legacy-prefix' + } else { + // Unversioned and unknown formats cannot authorize destructive pruning. + return false + } + if (!SHA256_HEX.test(expected)) return false + + const actual = + algorithm === 'whole' ? hashes.whole : (hashes.legacyPrefix ?? hashes.whole) + return expected !== actual +} + /** * Drop stale evidence from the persisted record and rewrite it atomically * (revision bumped past the loaded record, checksum recomputed over the same @@ -734,6 +878,8 @@ export type TaskMemoryPruneOutcome = * that can see workspace moves must pass them, or evidence bound to a renamed * file reconciles stale and is permanently deleted instead of rebinding to * its destination. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export async function pruneStaleTaskMemoryEvidence(params: { rootDir: string @@ -747,7 +893,7 @@ export async function pruneStaleTaskMemoryEvidence(params: { */ workspaceMoves?: WorkspaceMoveRecord[] }): Promise { - const fs = params.fs ?? nodeFsPromises + const fs = taskMemoryFs(params.fs) const persisted = await loadPersistedTaskMemory({ rootDir: params.rootDir, fs, @@ -762,10 +908,28 @@ export async function pruneStaleTaskMemoryEvidence(params: { fs, workspaceMoves: params.workspaceMoves, }) - const kept = memory.evidence.filter((item) => item.stale !== true) + const persistedById = new Map( + persisted.evidence.map((item) => [item.id, item] as const), + ) + const kept: TaskMemoryEvidenceV1[] = [] + for (const item of memory.evidence) { + const removalSafe = await canPruneReconciledEvidence({ + item, + recordedHash: persistedById.get(item.id)?.freshnessHash, + rootDir: params.rootDir, + fs, + }) + if (!removalSafe) kept.push(item) + } const removed = memory.evidence.length - kept.length const remaining = kept.length - if (removed === 0) return { status: 'pruned', removed: 0, remaining } + const needsHashBackfill = memory.evidence.some( + (item, index) => + item.freshnessHash !== persisted.evidence[index]?.freshnessHash, + ) + if (removed === 0 && !needsHashBackfill) { + return { status: 'pruned', removed: 0, remaining } + } const parsedDraft = taskMemoryDraftV1Schema.safeParse({ ...memory, @@ -777,7 +941,7 @@ export async function pruneStaleTaskMemoryEvidence(params: { // Serialized section: the revision check and the commit must not straddle // another writer's commit (reconciliation above is deliberately outside it). - return withMemoryFileLock( + const outcome = await withMemoryFileLock( fs, getMemoryFilePath(params.rootDir), async (): Promise => { @@ -827,6 +991,9 @@ export async function pruneStaleTaskMemoryEvidence(params: { return { status: 'pruned', removed, remaining } }, ) + return ( + outcome ?? { status: 'failed', reason: 'write-failed', removed, remaining } + ) } /** @@ -839,8 +1006,10 @@ export async function pruneStaleTaskMemoryEvidence(params: { * require the optional `renameFile` capability; without it, persistence * degrades to a skipped save (saveMergedTaskMemory returns undefined) * rather than a non-atomic write. A native `open` on the host (real node - * fs) is forwarded so oversized evidence hashing streams leading bytes - * instead of buffering the file. + * fs) is forwarded so large evidence files receive the same whole-content + * digest as buffered files without unbounded memory use. + * + * @deprecated Memory V1 compatibility surface; use Memory V2. Removal will occur only after the documented compatibility window and migration audit. */ export function codebuffFsToNodePromises( codebuffFs: CodebuffFileSystem, @@ -854,18 +1023,15 @@ export function codebuffFsToNodePromises( } // CodebuffFileSystem's published type omits `open`, but node-fs-backed // hosts (spreads of fs.promises, createNodeFileSystem()) still carry it at - // runtime. Detect and forward it so oversized evidence hashing streams - // leading bytes instead of taking the buffered fallback. + // runtime. Detect and forward it for bounded whole-content streaming. const maybeOpen = (codebuffFs as { open?: FsModule['open'] }).open const open: TaskMemoryStoreFs['open'] = typeof maybeOpen === 'function' ? (file, flags, mode) => maybeOpen(file, flags, mode) : undefined - const maybeLstat = (codebuffFs as { lstat?: FsModule['stat'] }).lstat - const lstat: TaskMemoryStoreFs['lstat'] = - typeof maybeLstat === 'function' - ? (p) => - (maybeLstat as unknown as (p: string) => Promise)(p as string) + const createFileExclusive: TaskMemoryStoreFs['createFileExclusive'] = + typeof codebuffFs.createFileExclusive === 'function' + ? (file, data) => codebuffFs.createFileExclusive!(file, data) : undefined const maybeRealpath = ( codebuffFs as { realpath?: (p: string) => Promise } @@ -874,13 +1040,6 @@ export function codebuffFsToNodePromises( typeof maybeRealpath === 'function' ? (p) => maybeRealpath(p as string) : undefined - const maybeReadlink = ( - codebuffFs as { readlink?: (p: string) => Promise } - ).readlink - const readlink: TaskMemoryStoreFs['readlink'] = - typeof maybeReadlink === 'function' - ? (p) => maybeReadlink(p as string) - : undefined return { // Discard recursive mkdir's first-created-path result; the store only // needs completion, and TaskMemoryStoreFs declares Promise. @@ -888,11 +1047,10 @@ export function codebuffFsToNodePromises( await codebuffFs.mkdir(path, options) }, readFile: (file) => codebuffFs.readFile(file) as Promise, + createFileExclusive, rename, open, - lstat, realpath, - readlink, stat: (path) => codebuffFs.stat(path), unlink: (path) => codebuffFs.unlink(path), writeFile: (file, data, options) =>