fix(export): keep preflight coverage aligned with the schema - #874
Conversation
40ace16 to
d518d4f
Compare
WaylandYang
left a comment
There was a problem hiding this comment.
This is the right fix for the "three hand-maintained copies of the edge list"
problem, and the shape is the one this repository prefers: one checked-in SQL that
the runtime executes via include_str! and the catalog guard parses byte-for-byte,
markers keyed by the structural 4-tuple rather than the error label, and probes for
both directions of drift (PREFLIGHT_MISSING, STALE_PREFLIGHT). The malformed-
ledger fixture planting a bad row for all 39 edges is what makes the 39 a number the
suite actually checks rather than a comment. I want to land it. Two things first.
1. It does not compile on current dev. #886 moved the store integration tests
into one binary under tests/store/, and git carries your edits across the rename,
but this path in the test is relative to where the file used to be:
error: couldn't read `crates/utopia-store/tests/store/../src/export_provenance_integrity.sql`
--> crates/utopia-store/tests/store/migration_0070_runs_under_any_search_path.rs:597:23
That is include_str!("../src/export_provenance_integrity.sql") at line 597 of the
moved file; it needs to be ../../src/… now (the runtime include_str! in
export.rs is unaffected). Please rebase onto dev and re-run
UTOPIA_TEST_REQUIRE_DB=1 cargo test -p utopia-store; the branch's own CI ran before
#886 landed, which is why it is green while the merge is not.
2. The record's original sentences should stay. The revision note you added under
## Revisions is exactly how this repository corrects a record, and it is well
written. But the diff also rewrites the two bullets the note corrects (the
pg_restore --disable-triggers rationale and the "Uniform triggers everywhere"
alternative) in place. The convention in docs/decisions/README.md and
docs/design/README.md is that a record is written once and never rewritten; a
change of mind is a dated note beside the original, so a reader six months from now
can see both what was believed and what corrected it. Please restore those two
bullets verbatim and let the Revisions entry carry the correction; the Status line
update and the README rows are fine as they are.
Everything else — the SQL extraction, preflight_surface/check_preflight, probe C,
the DB-free unit check that removing an edge fails the validator — reads well and I
have no changes to ask for there.
The catalog-derived guard added for migration 0070 proved every column-level reference edge on the ledger surface is protected in the schema, but export preflight remained a third, hand-maintained representation of the same reference surface: a future migration could add a correctly protected same-KB edge and forget to scan it before streaming, and a stale preflight branch could outlive its edge. Factor the preflight scan into a single checked-in source, export_provenance_integrity.sql, which the runtime executes via include_str! and the guard parses for -- @edge / -- @filter markers. Coverage is keyed by structural edge identity (table, column, target), not by error label, so entity_type_disjoint's two legs still share one label. Merged-entity checks stay a separate export-filter class. Widen the scan to all 39 protected structural edges (26 declarative composite-FK + 13 trigger-covered). The catalog guard now fails when a protected edge has no preflight branch, when a branch names an edge the schema no longer protects, or when markers are malformed or duplicated; a third drift probe plants a protected edge absent from preflight and expects the new failure bucket. Clean ledgers emit byte-identical Turtle; malformed ledgers may now be refused for a same-KB violation on an edge the serializer does not yet emit, which is the intended fail-closed reading of ADR 0048. Measured preflight cost is ~4 ms on a ~1000-document / ~3000-fact corpus; total export time stays within run-to-run noise. Signed-off-by: Anson <310461893+ansonnmm@users.noreply.github.com> Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
d518d4f to
46e083b
Compare
|
Applied both review points as a maintainer edit so this can land: rebased onto |
WaylandYang
left a comment
There was a problem hiding this comment.
Both review points applied in the rebuilt commit (include_str path fixed after #886's move; the two record bullets restored with your Revisions note carrying the correction); cargo test -p utopia-store 343/0 and clippy on a fresh database; CI green. Approving to clear my earlier request-changes.
Problem
#832 made schema coverage mechanically complete: the catalog-driven guard in
migration_0070_runs_under_any_search_path.rsenumerates every column-level reference edge on the ledger surface frompg_catalogand proves each is protected by a composite FK or an owner-derived trigger. But after #833, export preflight remained a third, hand-maintained representation of that same reference surface (migration 0070 §0, the actual schema, andexport::provenance_integrity). A future migration could therefore add a correctly schema-protected same-KB edge and silently omit it from fail-closed export validation — and a stale preflight branch could outlive the edge it names. This is the widened #846 follow-up identified during #833 review ("the edge list now exists in three places, by hand").Decision record
This maintains ADR 0048's same-knowledge-base provenance invariant and ADR 0020's fail-closed export contract; it introduces no new decision and overturns none. ADR 0048 is updated in place (the regression now covers export preflight, and a factually wrong sentence about replica-mode loads suppressing only user triggers is corrected — FK constraint triggers are suppressed too, which is exactly why all 39 edges need the export backstop).
Fix
provenance_integrityscan into one checked-in source,crates/utopia-store/src/export_provenance_integrity.sql. Runtime executes it viainclude_str!— the guard parses the identical bytes, so there is no fourth representation to drift.-- @edge src_table.src_col -> tgt_table.tgt_colfor structural same-KB edges,-- @filter labelfor merged-entity export-filter checks. Coverage is keyed by the 4-tuple structural identity, not by error label —entity_type_disjoint'sa_id/b_idlegs still share theclass.disjointlabel while counting as two edges.REPEATABLE READtransaction, still before the first byte streams.PREFLIGHT_MISSING), a branch names an edge the schema no longer protects (STALE_PREFLIGHT), a marker is malformed/duplicated/points the wrong way, or a branch lacks a marker. Merged-target checks stay a separate class and are unchanged.Regression evidence
SCHEMA_PROTECTED_EDGES=39(declarative 26 + trigger 13),PREFLIGHT_COVERED_STRUCTURAL_EDGES=39,PREFLIGHT_MISSING_EDGES=0,STALE_PREFLIGHT_EDGES=0— asserted byevery_reference_edge_on_the_ledger_is_classified.UTOPIA_TEST_REQUIRE_DB=1 cargo test --locked -p utopia-store: all 122 test binaries pass, includingmigration_0070_runs_under_any_search_path(8/8: catalog completeness, probes A/B/C, stale-edge unit),exported_references_never_cross_a_kb,cross_kb_provenance_fails_closed,export_surfaces, merged-target tests.cargo test -p utopia-serverexport/MCP subset (30 tests incl. concurrent exports on a minimum pool, authenticated RDF export): pass. (Unrelated LLM-endpoint/WebDAV unit failures reproduce identically on base — local proxy environment issue, no LLM service.)cargo fmt --all --check: clean.cargo clippy --locked --workspace --all-targets -- -D warnings: clean.provenance_integrityalone, warm: base ≈13.2 ms → patched ≈17.3 ms (+~4 ms)./export?format=turtlewall time (5 alternating runs each, after warmup): base median 345.3 ms, patched median 332.3 ms — delta within run-to-run noise.Compatibility / risk
Closes #846