Skip to content

fix(export): keep preflight coverage aligned with the schema - #874

Merged
WaylandYang merged 2 commits into
deeplethe:devfrom
ansonnmm:fix/export-preflight-coverage-846
Sep 24, 2026
Merged

WaylandYang merged 2 commits into
deeplethe:devfrom
ansonnmm:fix/export-preflight-coverage-846

Conversation

@ansonnmm

Copy link
Copy Markdown
Contributor

Problem

#832 made schema coverage mechanically complete: the catalog-driven guard in migration_0070_runs_under_any_search_path.rs enumerates every column-level reference edge on the ledger surface from pg_catalog and proves each is protected by a composite FK or an owner-derived trigger. But after #833, export preflight remained a third, hand-maintained representation of that same reference surface (migration 0070 §0, the actual schema, and export::provenance_integrity). A future migration could therefore add a correctly schema-protected same-KB edge and silently omit it from fail-closed export validation — and a stale preflight branch could outlive the edge it names. This is the widened #846 follow-up identified during #833 review ("the edge list now exists in three places, by hand").

Decision record

This maintains ADR 0048's same-knowledge-base provenance invariant and ADR 0020's fail-closed export contract; it introduces no new decision and overturns none. ADR 0048 is updated in place (the regression now covers export preflight, and a factually wrong sentence about replica-mode loads suppressing only user triggers is corrected — FK constraint triggers are suppressed too, which is exactly why all 39 edges need the export backstop).

Fix

  • Extract the provenance_integrity scan into one checked-in source, crates/utopia-store/src/export_provenance_integrity.sql. Runtime executes it via include_str! — the guard parses the identical bytes, so there is no fourth representation to drift.
  • Each scan branch carries one machine-readable marker: -- @edge src_table.src_col -> tgt_table.tgt_col for structural same-KB edges, -- @filter label for merged-entity export-filter checks. Coverage is keyed by the 4-tuple structural identity, not by error label — entity_type_disjoint's a_id/b_id legs still share the class.disjoint label while counting as two edges.
  • Widen preflight from 23 to all 39 0070-protected structural edges (26 declarative composite-FK + 13 trigger-covered), in the same read-only REPEATABLE READ transaction, still before the first byte streams.
  • The catalog guard now fails when: a protected edge has no preflight branch (PREFLIGHT_MISSING), a branch names an edge the schema no longer protects (STALE_PREFLIGHT), a marker is malformed/duplicated/points the wrong way, or a branch lacks a marker. Merged-target checks stay a separate class and are unchanged.
  • New probe C: a synthetic schema-protected edge deliberately absent from preflight → schema protection passes, preflight coverage fails naming the edge. Plus a DB-free unit check that removing one preflight edge makes the validator fail.
  • The malformed-ledger fixture now plants a bad row for every one of the 39 structural edges (plus the 5 merged/filter checks) and expects the export to refuse with the right labels.

Regression evidence

  • SCHEMA_PROTECTED_EDGES=39 (declarative 26 + trigger 13), PREFLIGHT_COVERED_STRUCTURAL_EDGES=39, PREFLIGHT_MISSING_EDGES=0, STALE_PREFLIGHT_EDGES=0 — asserted by every_reference_edge_on_the_ledger_is_classified.
  • UTOPIA_TEST_REQUIRE_DB=1 cargo test --locked -p utopia-store: all 122 test binaries pass, including migration_0070_runs_under_any_search_path (8/8: catalog completeness, probes A/B/C, stale-edge unit), exported_references_never_cross_a_kb, cross_kb_provenance_fails_closed, export_surfaces, merged-target tests.
  • cargo test -p utopia-server export/MCP subset (30 tests incl. concurrent exports on a minimum pool, authenticated RDF export): pass. (Unrelated LLM-endpoint/WebDAV unit failures reproduce identically on base — local proxy environment issue, no LLM service.)
  • cargo fmt --all --check: clean. cargo clippy --locked --workspace --all-targets -- -D warnings: clean.
  • Measured cost, same machine/Postgres 16/corpus (~1000 documents / 3000 chunks / 1500 entities / 3000 facts / 3000 evidence / 500 derived + seeded rows in every table the widened scan touches; ~4.18 MB Turtle):
    • provenance_integrity alone, warm: base ≈13.2 ms → patched ≈17.3 ms (+~4 ms).
    • End-to-end /export?format=turtle wall time (5 alternating runs each, after warmup): base median 345.3 ms, patched median 332.3 ms — delta within run-to-run noise.
    • Output bytes on the clean corpus: identical SHA-256 for base and patched.

Compatibility / risk

  • Clean-ledger RDF output is byte-identical; no serializer or page-query change.
  • Malformed ledgers may now be refused for a same-KB invariant violation on an edge the serializer does not yet emit — the intended conservative reading of the invariant.
  • Preflight performs additional read-only checks inside the same long-lived snapshot; measured +~4 ms on the bench corpus. The snapshot still holds for the whole stream (already documented; vacuum-delay caveat unchanged).
  • No schema migration, no write-path change, no new public API.

Closes #846

@ansonnmm
ansonnmm force-pushed the fix/export-preflight-coverage-846 branch from 40ace16 to d518d4f Compare September 23, 2026 08:08

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the right fix for the "three hand-maintained copies of the edge list"
problem, and the shape is the one this repository prefers: one checked-in SQL that
the runtime executes via include_str! and the catalog guard parses byte-for-byte,
markers keyed by the structural 4-tuple rather than the error label, and probes for
both directions of drift (PREFLIGHT_MISSING, STALE_PREFLIGHT). The malformed-
ledger fixture planting a bad row for all 39 edges is what makes the 39 a number the
suite actually checks rather than a comment. I want to land it. Two things first.

1. It does not compile on current dev. #886 moved the store integration tests
into one binary under tests/store/, and git carries your edits across the rename,
but this path in the test is relative to where the file used to be:

error: couldn't read `crates/utopia-store/tests/store/../src/export_provenance_integrity.sql`
   --> crates/utopia-store/tests/store/migration_0070_runs_under_any_search_path.rs:597:23

That is include_str!("../src/export_provenance_integrity.sql") at line 597 of the
moved file; it needs to be ../../src/… now (the runtime include_str! in
export.rs is unaffected). Please rebase onto dev and re-run
UTOPIA_TEST_REQUIRE_DB=1 cargo test -p utopia-store; the branch's own CI ran before
#886 landed, which is why it is green while the merge is not.

2. The record's original sentences should stay. The revision note you added under
## Revisions is exactly how this repository corrects a record, and it is well
written. But the diff also rewrites the two bullets the note corrects (the
pg_restore --disable-triggers rationale and the "Uniform triggers everywhere"
alternative) in place. The convention in docs/decisions/README.md and
docs/design/README.md is that a record is written once and never rewritten; a
change of mind is a dated note beside the original, so a reader six months from now
can see both what was believed and what corrected it. Please restore those two
bullets verbatim and let the Revisions entry carry the correction; the Status line
update and the README rows are fine as they are.

Everything else — the SQL extraction, preflight_surface/check_preflight, probe C,
the DB-free unit check that removing an edge fails the validator — reads well and I
have no changes to ask for there.

The catalog-derived guard added for migration 0070 proved every
column-level reference edge on the ledger surface is protected in the
schema, but export preflight remained a third, hand-maintained
representation of the same reference surface: a future migration could
add a correctly protected same-KB edge and forget to scan it before
streaming, and a stale preflight branch could outlive its edge.

Factor the preflight scan into a single checked-in source,
export_provenance_integrity.sql, which the runtime executes via
include_str! and the guard parses for -- @edge / -- @filter markers.
Coverage is keyed by structural edge identity (table, column, target),
not by error label, so entity_type_disjoint's two legs still share one
label. Merged-entity checks stay a separate export-filter class.

Widen the scan to all 39 protected structural edges (26 declarative
composite-FK + 13 trigger-covered). The catalog guard now fails when a
protected edge has no preflight branch, when a branch names an edge
the schema no longer protects, or when markers are malformed or
duplicated; a third drift probe plants a protected edge absent from
preflight and expects the new failure bucket.

Clean ledgers emit byte-identical Turtle; malformed ledgers may now be
refused for a same-KB violation on an edge the serializer does not yet
emit, which is the intended fail-closed reading of ADR 0048. Measured
preflight cost is ~4 ms on a ~1000-document / ~3000-fact corpus; total
export time stays within run-to-run noise.

Signed-off-by: Anson <310461893+ansonnmm@users.noreply.github.com>

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
@WaylandYang
WaylandYang force-pushed the fix/export-preflight-coverage-846 branch from d518d4f to 46e083b Compare September 24, 2026 04:04
@WaylandYang

Copy link
Copy Markdown
Contributor

Applied both review points as a maintainer edit so this can land: rebased onto dev (the test now lives under tests/store/ after #886) and fixed the include_str! path to ../../src/export_provenance_integrity.sql; restored the two record bullets to their original wording and kept your Revisions entry as the correction. cargo test -p utopia-store on a fresh database and clippy pass on the result. Your authorship and sign-off are preserved; if you have local changes, git fetch && git reset --hard origin/fix/export-preflight-coverage-846.

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both review points applied in the rebuilt commit (include_str path fixed after #886's move; the two record bullets restored with your Revisions note carrying the correction); cargo test -p utopia-store 343/0 and clippy on a fresh database; CI green. Approving to clear my earlier request-changes.

@WaylandYang
WaylandYang merged commit 7c6387d into deeplethe:dev Sep 24, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The same-KB invariant does not notice a reference column added later

2 participants