Implied-fact lineage is outside the same-KB provenance invariant

[ansonnmm]

Problem

Migration 0073 (#882) added implied_fact_sources, the durable provenance link for
rule-derived (facts.implied) facts:

CREATE TABLE implied_fact_sources (
    fact_id      UUID NOT NULL REFERENCES facts(id) ON DELETE CASCADE,
    rule_id      UUID NOT NULL REFERENCES implication_rules(id) ON DELETE CASCADE,
    statement_id UUID REFERENCES facts(id) ON DELETE CASCADE,
    entity_id    UUID REFERENCES entities(id) ON DELETE CASCADE,
    PRIMARY KEY (fact_id, rule_id),
    CHECK ((statement_id IS NOT NULL) <> (entity_id IS NOT NULL))
);

Every reference is a plain single-column foreign key. A column FK proves the target
row exists — it does not prove the target lives in the same knowledge base as the
implied fact that cites it. Three semantic lineage edges are exposed:

• rule_id → implication_rules.id (which rule implied the fact)
• statement_id → facts.id (which statement triggered it, phrase rules)
• entity_id → entities.id (which entity triggered it, kind_word rules)

fact_id → facts.id is the ownership edge: the row carries no kb_id, so its KB
authority is the owning fact's kb_id — the same shape as typed_fact_sources
(fact_id → facts), which is already in the protected family.

Why this is a generic Utopia invariant

implied_fact_sources is the same kind of durable semantic provenance as
typed_fact_sources: it records which stored rows an implied fact is derived from,
and the ledger consumes it internally — the materialize sweep deletes a source row
when its rule is unapproved or its trigger died, and invalidates a computed fact
whose sources have all gone. A lineage row that names a rule, statement, or entity
in another base is wrong on its face under a local knowledge-base interpretation,
independently of whether the exporter ever serializes it.

Current protection gap

ADR 0048 / migration 0070 (#832) protects 39 reference edges — 26 by composite
(kb_id, ref) foreign keys, 13 covered by owner-derived triggers — plus kb_id
immutability on the owned tables. #874 added the catalog-derived guard in
migration_0070_runs_under_any_search_path.rs and the per-edge export preflight in
export_provenance_integrity.sql.

implied_fact_sources is outside all of it:

• It is not in LEDGER_TABLES, so the catalog guard never enumerates its columns —
  the "reference column added later" failure mode The same-KB invariant does not notice a reference column added later #846 closed has recurred at table
  granularity. Structurally the gap is wider than a missing registry entry: the
  guard's scratch database is migrated only through 0070, so post-0070 tables are
  invisible to it — registering the table today would fail until the scratch chain
  is extended too.
• No composite FK or *_same_kb trigger covers rule_id, statement_id, or
  entity_id; implication_rules is also absent from the *_keep_their_kb
  immutability triggers (and has no UNIQUE (kb_id, id)), so
  UPDATE implication_rules SET kb_id can silently re-anchor every source row
  pointing at it.
• It has no branch in export_provenance_integrity.sql (preflight covers exactly
  the protected edges, so this follows automatically — but there is no protection
  to backstop).
• Store-layer write paths (imply_in_tx, implication_rules::propose,
  record_reading) construct all references inside the acting kb_id, but nothing
  downstream enforces it: a plain INSERT/UPDATE through any SQL path persists a
  cross-KB row today — no replica-mode session is even required, since there is no
  trigger to bypass.

The same migration left the rest of the rule surface outside the guard as well:
implication_rules.subject_type_id / object_type_id → entity_types and
conclude_property_id → relation_types, and phrase_readings.entity_id →
entities are likewise plain FKs on kb-owned rows. (For conclude_property_id and
a reading's entity_id, the composite keys 0070 put on facts turn a violation
into a materialize-time FK failure rather than silent corruption; the cross-KB rule
and reading rows themselves still persist, and foreign type references fail
silently as rules that can never match.)

Failure shape

A single INSERT is enough:

-- KB A holds implied fact F_A; KB B holds approved rule R_B.
INSERT INTO implied_fact_sources (fact_id, rule_id, statement_id)
VALUES ('<F_A>', '<R_B>', '<a statement in KB B>');

Every foreign key is satisfied; nothing objects. Consequences through existing code
paths, in both directions:

• The next materialize run for KB B joins implied_fact_sources to
  implication_rules on r.kb_id = B; when R_B is rejected or its trigger dies,
  the sweep deletes F_A's source row. F_A then has zero sources, so on KB A's
  next materialize it is found sourceless and invalidated — a rule decision in KB B
  silently retires a fact in KB A.
• While the phantom row exists it counts as a live source for F_A (the
  zero-source check only tests NOT EXISTS): if F_A's legitimate sources die
  first, the cross-KB row pins it alive — the opposite failure, an implied fact
  that can no longer be retired.
• The lineage is also orphaned from KB A's side: A's sweep can never reach the row
  (r.kb_id = B fails the join), and resolving "which rule implied F_A" through
  implication_rules scoped to A finds nothing — a durable provenance pointer
  naming a row the owning base cannot see.

Desired invariant

Every durable implied-fact provenance reference resolves to a semantically valid
target in the same knowledge base as the implied fact at the appropriate
transaction boundary — rule_id, statement_id, entity_id alike.

Design question

Should implied_fact_sources (and the rest of the 0073 rule surface) join the
ADR 0048 / migration-0070 invariant family?

• implied_fact_sources has no kb_id column; its authority is the owning fact —
  the owner-derived pattern already used for typed_fact_sources fits, if the
  trigger path is acceptable here.
• implication_rules and phrase_readings carry their own kb_id, so their
  references could take the direct-kb (composite FK) pattern instead — which would
  also need the supporting UNIQUE (kb_id, id) on implication_rules — plus
  *_keep_their_kb on implication_rules.

The membership question is already inconsistent elsewhere in the registry:
name_vectors (0080) carries full _same_kb composite keys and a keep_their_kb
trigger yet sits outside LEDGER_TABLES and the preflight, and
errata_runs / errata_actions (0074) carry kb-owned plain-FK semantic references
unregistered. Happy to split that wider audit out if you would rather keep this
issue to the 0073 provenance chain.

Suggested regression

In the spirit of cross_kb_provenance_fails_closed.rs (which seeds cross-KB rows
under SET LOCAL session_replication_role = 'replica'): a minimal test inserting
an implied_fact_sources row whose rule/statement/entity lives in a second base
must be rejected, while an all-same-KB row remains valid — and the catalog guard
should enumerate the table's edges so a future column cannot slip again.

Related

• fix(store): enforce same-KB provenance references #832 — same-KB provenance references (migration 0070)
• fix(export): keep preflight coverage aligned with the schema #874 — schema ↔ export-preflight completeness (catalog guard)
• ADR 0048

[WaylandYang]

Agreed on all of it, and thank you for tracing the guard's blind spot rather than just the missing row: a table added after 0070 is invisible to the catalog guard until its scratch chain is extended, so registering implied_fact_sources alone would fail.

The fix is the one you outline: implied_fact_sources, implication_rules and phrase_readings join the same-KB family with composite keys where the target has (kb_id, id) and owner-derived triggers where it does not, implication_rules gets UNIQUE (kb_id, id) and the keep-their-kb trigger, the tables enter LEDGER_TABLES, and the guard's scratch database migrates through the current chain so later tables cannot fall out again. The export preflight then follows on its own, as you say. Two more tables have appeared since (document_versions.doc_time is a column, not an edge, but typed_fact_sources and the rule history from #913 want the same check), so the sweep should be over every kb-owned table with a reference column, not the three named here.

You wrote #832; if you want this one it is yours, after the release candidate being cut now so it does not ride into rc7 untested. Say so here and I will hold the migration number for you.

[ansonnmm]

Happy to take this one. Please hold the migration number for me; I’ll wait until the current release candidate is cut before starting the implementation.

[WaylandYang]

Yours, thank you. Migration number 0091 is held for it (0090 is the last on dev; 0075 to 0080 are taken, and 0077 to 0079 belong to work in flight). The CLI's schema constant counts files, so it goes to 79 with it. rc7 is being cut from current dev; start whenever you like after that, and rebase onto dev before opening the PR since #913 and #916 touched the rule and governance tables.

[WaylandYang]

Landed as #938 (migration 0091): composite same-KB keys and owner-derived trigger on the post-0070 tables, keep_their_kb on the five that lacked it, the three unregistered edges in the ledger list, and the catalog guard's scratch database migrating to the chain end. Closing.