fix(FN-7952): complete PostgreSQL runtime cutover#2105
Conversation
Make PostgreSQL the only live persistence backend, close lifecycle and migration parity gaps, enforce project-scoped plugin schemas, and retain SQLite only as read-only legacy migration input. Add end-to-end regression coverage, operator documentation, and the major release changeset for the completed cutover. Fusion-Task-Id: FN-7952
|
Too many files changed for review. ( Bypass the limit by tagging |
|
Important Review skippedToo many files! This PR contains 275 files, which is 125 over the limit of 150. To get a review, narrow the scope: Upgrade to a paid plan to raise the limit. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (275)
You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Preserve PostgreSQL-only store construction while carrying forward project-store reuse and lock-free multi-tab planning semantics. Serialize planning persistence so asynchronous state writes cannot land out of order. Fusion-Task-Id: FN-7952
|
Superseded by the dependency-safe PostgreSQL cutover stack, with every PR below 100 changed files. Merge bottom-up: #2108 (core authority, 99 files) → #2109 (engine/dashboard runtime, 62) → #2110 (CLI/desktop/ops, 54) → #2111 (bundled plugins, 40) → #2112 (docs/release, 22). All replacement PRs are currently mergeable; local typechecks and the 478-test merge gate pass on each implementation layer. |
## Summary Fusion’s core runtime now treats PostgreSQL as the authoritative metadata store without leaving current CLI, dashboard, desktop, or engine composition roots uncompilable between stack layers. This is the 99-file foundation for the larger cutover: subsequent PRs migrate the remaining consumers, plugins, and operator surfaces. ## Design decisions - Runtime store construction fails closed when an asynchronous PostgreSQL layer is unavailable; SQLite remains readable only at explicit migration and identity-recovery boundaries. - Project ownership is enforced across active, archived, workflow, mission, analytics, and plugin-schema data. - The small set of cross-package files in this layer are compatibility-critical call sites required for a green intermediate commit, not the complete consumer migration. - Schema migration 0008 remains assigned to session-advisor state from current `main`; mission lineage idempotency advances to 0009 so neither invariant can be skipped. ## Validation - All affected package typechecks pass: Core, Engine, Dashboard, CLI, and Desktop. - `pnpm test:gate` passes: 478 tests across the engine gate, PostgreSQL core gate, and CLI workflow shape. - The PR changes exactly 99 files. ## Stack This is the base PR. Engine/dashboard, CLI/desktop/ops, plugins, and docs/release follow as stacked PRs, each below 100 changed files. Related: #2105 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * PostgreSQL is now the standard runtime backend, with embedded PostgreSQL enabled by default. * Added project-scoped storage for tasks, archives, chat sessions, missions, knowledge pages, and operational data. * Improved archived-task search, filtering, pagination, and restoration. * Added safer plugin schema initialization with validation and project isolation. * Added PostgreSQL-backed workflow, mission, validator, and dashboard capabilities. * **Bug Fixes** * Improved startup timeout cancellation and resource cleanup. * Prevented cross-project data access and phantom reservation cleanup errors. * Ensured archived tasks remain read-only and asynchronous writes complete reliably. * Retired SQLite opt-out settings with clear startup errors. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary Engine and dashboard traffic now stays on the authoritative PostgreSQL layer across execution, recovery, project discovery, planning sessions, analytics, and shutdown. The dashboard no longer presents a migration notice for a cutover that is already mandatory. ## Design decisions - Runtime composition requires an async data layer instead of constructing a hidden SQLite fallback. - Engine workflow, mission, claim, and self-healing reads await their PostgreSQL-backed store contracts. - Project-scoped dashboard stores retain and close their backend owner exactly once. - The dashboard test quarantine entry remains paired with its Vitest exclusion, preserving the repository’s deletion-ratchet policy. ## Validation - Core, Engine, Dashboard, CLI, and Desktop typechecks pass on the stacked branch. - `pnpm test:gate` passes all 478 gate tests. - This PR changes 62 files. ## Stack - Depends on #2108. - CLI/desktop/ops, plugins, and docs/release follow in later PRs. Related: #2105 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Project discovery now recognizes projects using the `.fusion/project.json` marker. * Knowledge indexing and search are more reliable across project-scoped storage. * **Bug Fixes** * Improved session, audit timeline, approval, monitoring, and analytics data consistency. * Prevented stale planning-session updates and project-store shutdown races. * Ensured chat usage and CLI session status are saved before continuing. * **UI Changes** * Removed the storage migration notice banner now that the PostgreSQL transition is complete. * **Reliability** * Improved shutdown handling, workflow execution, and worktree behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary CLI commands, daemon/dashboard startup, packaged desktop startup, and live-data maintenance scripts now share the mandatory PostgreSQL lifecycle. Operators no longer risk a command silently reading or writing a disconnected SQLite shadow when PostgreSQL setup fails. ## Design decisions - Every startup owner retains and awaits its PostgreSQL shutdown callback, including partial-startup failure paths. - CLI project context and lock-retry flows resolve through asynchronous project stores. - Maintenance scripts use the shared backend helper; explicit database migration/inspection remains the only CLI surface allowed to read legacy SQLite sources. ## Validation - CLI and Desktop typechecks pass on the stacked branch. - `pnpm test:gate` passes all 478 gate tests. - This PR changes 54 files. ## Stack - Depends on #2109, which depends on #2108. - Bundled plugins and docs/release follow in later PRs. Related: #2105 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * PostgreSQL is now the authoritative store for structured project and task metadata. * Projects can be recognized and initialized using `.fusion/project.json`, without creating a legacy SQLite database. * CLI commands now retry transient PostgreSQL contention errors. * **Bug Fixes** * Improved cleanup when commands complete, fail, or run in the background, preventing lingering resources. * Improved desktop, server, and session shutdown reliability. * **Documentation** * Updated storage and standalone binary guidance to reflect PostgreSQL and legacy SQLite compatibility. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary Operators and contributors now have one current contract for the completed PostgreSQL cutover: PostgreSQL is mandatory runtime authority, legacy SQLite files are read-only migration evidence, and rollback is restore-based rather than a fallback toggle. The major changeset communicates the same boundary to published CLI users. ## Operational guidance - Documents the authoritative storage inventory and the five intentional legacy SQLite readers. - Defines maintenance-window migration, backup, restore-test, ownership, and isolation checks. - Clarifies lifecycle ownership for CLI, desktop, dashboard, engine, and bundled plugins. - Supersedes the earlier readiness review without deleting its historical record. ## Validation - Changeset format check passes. - `pnpm test:gate` passes all 478 gate tests on the completed stack. - Every file from the original oversized PR is represented in the replacement stack; this PR changes 22 files. ## Stack - Depends on #2111 → #2110 → #2109 → #2108. - Merge bottom-up in that order. Related: #2105
## Summary Operators and contributors now have one current contract for the completed PostgreSQL cutover: PostgreSQL is mandatory runtime authority, legacy SQLite files are read-only migration evidence, and rollback is restore-based rather than a fallback toggle. The major changeset communicates the same boundary to published CLI users. ## Operational guidance - Documents the authoritative storage inventory and the five intentional legacy SQLite readers. - Defines maintenance-window migration, backup, restore-test, ownership, and isolation checks. - Clarifies lifecycle ownership for CLI, desktop, dashboard, engine, and bundled plugins. - Supersedes the earlier readiness review without deleting its historical record. ## Validation - Changeset format check passes. - `pnpm test:gate` passes all 478 gate tests on the completed stack. - Every file from the original oversized PR is represented in the replacement stack; this PR changes 22 files. ## Stack - Depends on #2111 → #2110 → #2109 → #2108. - Merge bottom-up in that order. Related: #2105
## Summary Operators and contributors now have one current contract for the completed PostgreSQL cutover: PostgreSQL is mandatory runtime authority, legacy SQLite files are read-only migration evidence, and rollback is restore-based rather than a fallback toggle. The major changeset communicates the same boundary to published CLI users. ## Operational guidance - Documents the authoritative storage inventory and the five intentional legacy SQLite readers. - Defines maintenance-window migration, backup, restore-test, ownership, and isolation checks. - Clarifies lifecycle ownership for CLI, desktop, dashboard, engine, and bundled plugins. - Supersedes the earlier readiness review without deleting its historical record. ## Validation - Changeset format check passes. - `pnpm test:gate` passes all 478 gate tests on the completed stack. - Every file from the original oversized PR is represented in the replacement stack; this PR changes 22 files. ## Stack - Depends on #2111 → #2110 → #2109 → #2108. - Merge bottom-up in that order. Related: #2105
## Summary Bundled plugins now persist shared runtime state in project-scoped PostgreSQL tables instead of maintaining independent SQLite authority. Reports, CLI Printing Press, Compound Engineering, Roadmap, Even Realities, and WhatsApp all follow the same ownership and startup contract as Fusion core. ## Design decisions - Plugin schema hooks run through the host’s PostgreSQL owner and enforce project isolation. - The SDK exposes the host contract needed by bundled plugins without importing engine internals. - Legacy Roadmap ownership fixtures use the supported empty-owner sentinel, preserving current composite primary/foreign keys while exercising backfill behavior. - The lockfile travels with the Even Realities PostgreSQL dependency so packaged installs remain reproducible. ## Validation - All six affected plugin builds pass. - Affected plugin suites pass: 773 tests across Printing Press, Compound Engineering, Even Realities, Reports, Roadmap, and WhatsApp. - `pnpm test:gate` passes all 478 gate tests. - This PR changes 40 files. ## Stack - Depends on #2110 → #2109 → #2108. - The documentation/release PR completes the stack. Related: #2105 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Breaking Changes** * PostgreSQL is now required for runtime storage; SQLite files are used only as one-time migration inputs. * The legacy `FUSION_NO_EMBEDDED_PG` fallback has been removed. * **New Features** * Added project-isolated PostgreSQL storage for plugins, reports, tasks, notifications, and other plugin data. * Added agent tools for reports and CLI service drafts. * Added PostgreSQL schema initialization support for plugin authors. * **Bug Fixes** * Improved migration and recovery of legacy plugin state. * Prevented cross-project data access and strengthened transactional schema updates. * **Documentation** * Updated storage, migration, deployment, plugin authoring, CLI, and dashboard guidance for PostgreSQL. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary Fusion’s core runtime now treats PostgreSQL as the authoritative metadata store without leaving current CLI, dashboard, desktop, or engine composition roots uncompilable between stack layers. This is the 99-file foundation for the larger cutover: subsequent PRs migrate the remaining consumers, plugins, and operator surfaces. ## Design decisions - Runtime store construction fails closed when an asynchronous PostgreSQL layer is unavailable; SQLite remains readable only at explicit migration and identity-recovery boundaries. - Project ownership is enforced across active, archived, workflow, mission, analytics, and plugin-schema data. - The small set of cross-package files in this layer are compatibility-critical call sites required for a green intermediate commit, not the complete consumer migration. - Schema migration 0008 remains assigned to session-advisor state from current `main`; mission lineage idempotency advances to 0009 so neither invariant can be skipped. ## Validation - All affected package typechecks pass: Core, Engine, Dashboard, CLI, and Desktop. - `pnpm test:gate` passes: 478 tests across the engine gate, PostgreSQL core gate, and CLI workflow shape. - The PR changes exactly 99 files. ## Stack This is the base PR. Engine/dashboard, CLI/desktop/ops, plugins, and docs/release follow as stacked PRs, each below 100 changed files. Related: #2105 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * PostgreSQL is now the standard runtime backend, with embedded PostgreSQL enabled by default. * Added project-scoped storage for tasks, archives, chat sessions, missions, knowledge pages, and operational data. * Improved archived-task search, filtering, pagination, and restoration. * Added safer plugin schema initialization with validation and project isolation. * Added PostgreSQL-backed workflow, mission, validator, and dashboard capabilities. * **Bug Fixes** * Improved startup timeout cancellation and resource cleanup. * Prevented cross-project data access and phantom reservation cleanup errors. * Ensured archived tasks remain read-only and asynchronous writes complete reliably. * Retired SQLite opt-out settings with clear startup errors. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary Engine and dashboard traffic now stays on the authoritative PostgreSQL layer across execution, recovery, project discovery, planning sessions, analytics, and shutdown. The dashboard no longer presents a migration notice for a cutover that is already mandatory. ## Design decisions - Runtime composition requires an async data layer instead of constructing a hidden SQLite fallback. - Engine workflow, mission, claim, and self-healing reads await their PostgreSQL-backed store contracts. - Project-scoped dashboard stores retain and close their backend owner exactly once. - The dashboard test quarantine entry remains paired with its Vitest exclusion, preserving the repository’s deletion-ratchet policy. ## Validation - Core, Engine, Dashboard, CLI, and Desktop typechecks pass on the stacked branch. - `pnpm test:gate` passes all 478 gate tests. - This PR changes 62 files. ## Stack - Depends on #2108. - CLI/desktop/ops, plugins, and docs/release follow in later PRs. Related: #2105 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Project discovery now recognizes projects using the `.fusion/project.json` marker. * Knowledge indexing and search are more reliable across project-scoped storage. * **Bug Fixes** * Improved session, audit timeline, approval, monitoring, and analytics data consistency. * Prevented stale planning-session updates and project-store shutdown races. * Ensured chat usage and CLI session status are saved before continuing. * **UI Changes** * Removed the storage migration notice banner now that the PostgreSQL transition is complete. * **Reliability** * Improved shutdown handling, workflow execution, and worktree behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary CLI commands, daemon/dashboard startup, packaged desktop startup, and live-data maintenance scripts now share the mandatory PostgreSQL lifecycle. Operators no longer risk a command silently reading or writing a disconnected SQLite shadow when PostgreSQL setup fails. ## Design decisions - Every startup owner retains and awaits its PostgreSQL shutdown callback, including partial-startup failure paths. - CLI project context and lock-retry flows resolve through asynchronous project stores. - Maintenance scripts use the shared backend helper; explicit database migration/inspection remains the only CLI surface allowed to read legacy SQLite sources. ## Validation - CLI and Desktop typechecks pass on the stacked branch. - `pnpm test:gate` passes all 478 gate tests. - This PR changes 54 files. ## Stack - Depends on #2109, which depends on #2108. - Bundled plugins and docs/release follow in later PRs. Related: #2105 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * PostgreSQL is now the authoritative store for structured project and task metadata. * Projects can be recognized and initialized using `.fusion/project.json`, without creating a legacy SQLite database. * CLI commands now retry transient PostgreSQL contention errors. * **Bug Fixes** * Improved cleanup when commands complete, fail, or run in the background, preventing lingering resources. * Improved desktop, server, and session shutdown reliability. * **Documentation** * Updated storage and standalone binary guidance to reflect PostgreSQL and legacy SQLite compatibility. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary Bundled plugins now persist shared runtime state in project-scoped PostgreSQL tables instead of maintaining independent SQLite authority. Reports, CLI Printing Press, Compound Engineering, Roadmap, Even Realities, and WhatsApp all follow the same ownership and startup contract as Fusion core. ## Design decisions - Plugin schema hooks run through the host’s PostgreSQL owner and enforce project isolation. - The SDK exposes the host contract needed by bundled plugins without importing engine internals. - Legacy Roadmap ownership fixtures use the supported empty-owner sentinel, preserving current composite primary/foreign keys while exercising backfill behavior. - The lockfile travels with the Even Realities PostgreSQL dependency so packaged installs remain reproducible. ## Validation - All six affected plugin builds pass. - Affected plugin suites pass: 773 tests across Printing Press, Compound Engineering, Even Realities, Reports, Roadmap, and WhatsApp. - `pnpm test:gate` passes all 478 gate tests. - This PR changes 40 files. ## Stack - Depends on #2110 → #2109 → #2108. - The documentation/release PR completes the stack. Related: #2105 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Breaking Changes** * PostgreSQL is now required for runtime storage; SQLite files are used only as one-time migration inputs. * The legacy `FUSION_NO_EMBEDDED_PG` fallback has been removed. * **New Features** * Added project-isolated PostgreSQL storage for plugins, reports, tasks, notifications, and other plugin data. * Added agent tools for reports and CLI service drafts. * Added PostgreSQL schema initialization support for plugin authors. * **Bug Fixes** * Improved migration and recovery of legacy plugin state. * Prevented cross-project data access and strengthened transactional schema updates. * **Documentation** * Updated storage, migration, deployment, plugin authoring, CLI, and dashboard guidance for PostgreSQL. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Summary
Fusion now uses PostgreSQL as the only live metadata backend. Before this cutover, several CLI, dashboard, runtime, archive, mission, session, and plugin paths could still fall back to SQLite, skip migrated state, or leave database ownership ambiguous; those paths now fail closed or use the shared PostgreSQL layer.
Legacy SQLite files remain supported only as read-only migration evidence. Normal runtime traffic cannot select them, and rollback is restore-only through a tested PostgreSQL backup rather than re-enabling SQLite.
Design decisions
DATABASE_URLis mandatory;FUSION_NO_EMBEDDED_PGis rejected.project_id, with forced RLS, policies, ownership triggers, and steady-state audits.onLoadside effects.The cutover also completes PostgreSQL parity for cold archives, workflows, missions, knowledge, CLI sessions, maintenance, phantom reservations, bundled plugin stores, and operational repair scripts. Hot archive reads and plugin runtime catalogs are bounded to avoid loading unrelated project history.
Migration and operations
fusion.db,archive.db, andfusion-central.dbas immutable import/recovery inputs, never write targets.docs/postgres-migration-review-2026-07-14.md.Validation
pnpm lintandpnpm buildpassed.pnpm test:gatepassed: 40 files and 478 tests.pnpm smoke:bootpassed with CLI help, an ephemeral-port health check, and clean shutdown.pnpm verify:fastpassed.One unrelated dashboard CPU-sampling assertion flaked during the broader review run and was quarantined under the repository's deletion-ratchet policy; its timeout and assertion were not weakened.
New concepts
Declarative privilege separation for plugin schemas
A plugin describes the PostgreSQL tables it needs, but never receives the privileged connection that creates them. Fusion validates the declaration first, applies it through a short-lived migration executor, then starts the plugin with its ordinary project-scoped runtime connection.
onLoadreceives only the normal scoped plugin context after schema setup succeeds.This avoids exposing a superuser-like handle to third-party runtime code while still allowing independently shipped plugins to own tables. It is not intended for arbitrary data migrations or cross-project administration; those remain operator-controlled migration work.