Extraction runs: proposition lineage

[jimador]

Slice 9 of the extraction-integrity train (#67), stacked on #99. A stored claim can now be traced to the run that produced it. PropositionStore.saveAllReturningCanonical and PersistablePropositions.persistReturningCanonical are additive default-bodied calls returning PropositionPersistenceResult — the canonical proposition for each input, the input-id to stored-id mapping, and a distinct-by-canonical-id view; the result type rejects length mismatches, one input landing on two ids, per-position tenant mismatches, and one canonical id answered under two contexts. PropositionRunLinkStore (in-memory and Drivine, behind a 14-case cross-backend contract suite) writes a MERGE-idempotent (:Proposition)-[:PRODUCED_BY_RUN]->(:ExtractionRun) edge, tenant-guarded on both endpoints, with all-or-nothing batch writes proven inside the TOCTOU window. The headline invariant on Neo4j: two identical-content runs leave one proposition, one source grounding, and two run links. Lineage is recorded immediately after the claims are saved, before any fallible pass, so a throwing projector or grounding service leaves saved claims attributed and still propagates.

Changed in this review round:

• Canonical persistence is the only path. persistAndProject used to take the repository's canonical propositions only when a run was present and the pre-save minted ones otherwise, which let an audit setting decide whether the graph was written correctly. It now wires projection, grounding and structural relationships against what save returned on every call; a run's only effect is the lineage write. A mutation restoring the old no-run wiring fails three tests, projection-targets-a-pre-save-id among them.
• Attribution fails loud. A run present with no way to record its lineage is an error by policy; duplicate PRODUCED_BY_RUN edges cannot occur (MERGE on the pair, proven by the linking-twice case).
• ExtractionRun.invocations normalize to plan order at construction (the Extraction runs: the Drivine store #99 follow-up); stored fingerprints are unaffected.
• The experimental extraction surfaces are marked as such in the CHANGELOG.

Breaking changes: none. Every existing public signature, descriptor and default-body behavior is byte-identical, pinned by a binary-compatibility test asserting the @JvmOverloads synthetic constructor descriptors (arities 9–16) survived. One behavioral change, sanctioned and on an unreleased experimental surface: the no-run extraction path now also persists, projects and grounds against canonical ids — the audit-metadata-never-changes-product-behavior rule, applied.

Opt-in and status: EXPERIMENTAL, marked @ApiStatus.Experimental in code. The Drivine stores and the run schema DDL sit behind embabel.dice.extraction.runs.enabled (default false) alongside the graph backend condition; a host that never sets it gets no run beans and no constraints or indexes written, and the CHANGELOG names exactly what turning it on writes. Run attribution happens when ExtractionRequest.currentRun is set and a PropositionRunLinkStore is bound; with neither, behavior is the canonical path alone. Honest limit, in shipped docs: lineage is best-effort for application-level failures only — a statement failing at the server inside a wrapping host transaction terminates that transaction beneath Spring, demonstrated by a fault-injection test; the structural fix (committing claims before recording lineage) belongs to the coordinator slice and the docs name it.

The lineage shape. The root ref is denormalized onto every run, so a whole-lineage lookup is one indexed read, with no chain walk:

flowchart LR
    R[run A, pass 0, root = self] --> B[run B, pass 1, parent = A]
    B --> C[run C, pass 2, parent = B]
    B -. "root = A" .-> R
    C -. "root = A" .-> R
    PA[proposition] -->|PRODUCED_BY_RUN| C
    PB[proposition] -->|PRODUCED_BY_RUN| C

Loading

Chain walks stay bounded and cycle-safe, because the denormalized root is a read optimization and never the integrity check.

Landing

• Build the union of this stack's tip and the metamodel stack's tip (feat(dice-storage-autoconfigure): opt-in metamodel wiring #88) before either lands; the two share six recurring conflict files with recorded resolutions; DiceStorageAutoConfiguration.kt conflicts too and is resolved by hand from the union worktree's committed adaptation, which also carries the runs gate on both schema beans.
• Whichever stack lands second restacks and applies one adaptation: ExtractionRunSchema joins the metamodel stack's DiceStorageSchema derived-ownership contract (object declaration, override on specs(), bookkeeping relationship types, catalog and test beans). DiceStorageSchemaRegistrationTest fails until it is applied — that is the guard working.
• No consumer migration is required to land: the four-argument undoSingleCollapse stays as a deprecated overload (Source-revision provenance: authoritative collector fold and undo #92). Downstream callers move to CollapseUndoCommand when convenient.

[igordayen]

Findings from CODEX:

Findings

1. STRICT lineage does not fail the async event path.
   IncrementalPropositionExtraction.kt:472-476 (https://github.com/embabel/dice/pull/101/files#diff-1) catches every exception from
   persistAndProject, including LineageNotRecordedException, and only logs it. But SourceAnalysisRequestEvent.kt:67-77
   (https://github.com/embabel/dice/pull/101/files#diff-2) now lets async publishers attach currentRun(). Example: an event carries
   currentRun, no PropositionRunLinkStore is bound, STRICT throws, then the listener swallows it. The caller/coordinator sees no
   failure signal, so “strict attribution fails the operation” is only true for direct calls, not for the async path added here.

2. The current ordering can leave saved claims unattributed when projection/grounding fails.
   IncrementalPropositionExtraction.kt:611-647 (https://github.com/embabel/dice/pull/101/files#diff-3) saves canonical propositions,
   then structural wiring, projection, grounding, and only then writes lineage. So if projection throws after the save, the
   proposition is stored but no PRODUCED_BY_RUN edge is attempted. Example: run R1 extracts proposition P1; save succeeds; projector
   throws; P1 remains in the store, but propositionsOf(R1) is empty. This contradicts the PR body and
   PersistablePropositions.kt:120-125 (https://github.com/embabel/dice/pull/101/files#diff-4), which say the split exists so saved
   claims are not stranded without the run that produced them. If lineage-last is intentional, those statements need to be corrected;
   if traceability is the invariant, lineage needs to happen immediately after canonical save.

3. InMemoryPropositionRunLinkStore retains stale links forever.
   InMemoryPropositionRunLinkStore.kt:57-58 (https://github.com/embabel/dice/pull/101/files#diff-5) keeps every run-to-proposition id
   in byRun, and lines 108-110 (https://github.com/embabel/dice/pull/101/files#diff-5) explicitly leave deleted proposition ids in
   the map. Reads filter stale ids, but memory still grows with every extraction/run/proposition ever linked. For a long-running
   in-memory host or tests that reuse the store, deleted or expired propositions disappear from API results but remain retained
   internally. Consider opportunistic pruning on reads, a clear/delete hook, or making this explicitly test/dev-only.

[igordayen]

@jimador - would you please review some findings from Codex, concern with stale memory, thank you

[jimador]

All three addressed.

1. Lineage now runs right after the canonical save, before structural wiring, projection and grounding, so a claim is attributed the moment it exists and a throwing projector cannot strand it. The KDoc and the design doc describe the state a STRICT failure leaves and no longer argue for lineage-last. d3cbee7.
2. The async path honours STRICT: a LineageNotRecordedException is logged at error and rethrown to the publisher, and the queue keeps draining so one failing event does not drop the next. f8eb4f7.
3. The in-memory link store prunes an id on read when the proposition no longer exists at all, and drops an empty run entry; a read from the wrong tenant still fails closed and touches nothing. af3ff70.

[igordayen]

3. The in-memory link store prunes an id on read when the proposition no longer exists at all, and drops an empty run entry; a read from the wrong tenant still fails closed and touches nothing. af3ff7

@jimador - would an eviction policy be useful for this use case? Thank you

[igordayen]

@jimador - thanks for addressing inquiries, looks good. thanks

[igordayen]

@jimador - looks like rebase is required, thx

[jimador]

Yes, and it is in ff8e343. Pruning on read only helps a run somebody reads, so InMemoryPropositionRunLinkStore now takes maxRuns (default 10,000, same as the reference run store) and past it drops the links of the runs it linked earliest once the run store says they have ended. A run still RUNNING keeps its links, and a breach with nothing to evict logs once. Design doc and changelog updated.

On the rebase: the branch is on current main, GitHub reports it clean. What it is showing is the stacked base, since #101 sits on #99, which sits on #98 and so on down to #91. It merges after those, and the stack tool moves each remaining branch onto main as the one below it lands.