[Feature]: Concept versioning engine (P0 core) — schema, atomic publish, safe transition, freshness by mvkonchits-db · Pull Request #703 · databrickslabs/ontos

mvkonchits-db · 2026-08-12T16:18:55Z

What this is

The P0 correctness core of the concept-versioning engine. Version the CONCEPT ((iri, version); the IRI is the stable UC join key). Backend-only; no UI in this PR.

DRAFT — not review-ready until the companion versioning UI also passes E2E on top. Opened for early visibility + CI.

What's included (P0-1/2/3/6/8)

P0-1 schema: concept_version table (iri, version, is_current, status, parent_version_id, replaces_iri), concept_version_id FK on rdf_triples, scheme_membership. Partial unique index UNIQUE(iri) WHERE is_current makes two-current-per-iri structurally impossible. Backfill stamps existing concepts v1/current by subject-IRI ownership.
P0-2 write-time current/history split: hot graph builds from current-only via rdf_triples_current view; reads carry NO version predicate (Lars's hard requirement). History is cold, keyed by (iri, version).
P0-3 atomic publish (POST /concepts/version/publish): single-transaction version swap, demote-then-insert so the partial index never sees two current rows; DB-first-then-graph-patch with a rebuild recovery contract. Snapshots the prior version's triples so the old definition text is preserved (answers "what was the previous definition"). Migration m4 widens rdf_triples uniqueness to allow per-version snapshots.
P0-6 safe transition: reference-count (entity_semantic_links + concept→concept refs) as the retire gate; deprecate (writes isReplacedBy/wasRevisionOf on a 2B split); retire gated on refs==0 → tombstone, never hard-delete. Prior versions get status superseded (distinct from deprecated/retired).
P0-8 graph freshness: graph/freshness + graph/reload; every write path dual-writes (DB-first-then-patch); UC-sync timing surfaced separately.

Explicitly out of scope

P0-4 diff engine (file re-upload as a versioning event) — deferred, all-or-nothing decision gated on a real power-user round-tripping ontology files.
Release manifests — P2, gated on a named version-pinning consumer.
Versioning UI — companion track, separate.

Verification

All migrations reversible; single alembic head (m4_rdf_triple_version_uq); chain resolves from base.
API-driven E2E on a deployed app + live Lakebase: 26/26 — publish→v2, history (prior=superseded), snapshot (v1 keeps old definition), read-isolation (no history leak), retire gate (real reference → 409 refused → unlink → succeeds), 2B split (isReplacedBy), tombstone-not-delete, freshness advances.
Not E2E-testable over REST: concurrent-reader swap atomicity (verified at the repository/transaction level).

This pull request and its description were written by Isaac.

Add concept_version table (versioned unit = (iri, version); is_current hot set with a partial unique index UNIQUE(iri) WHERE is_current), a concept_version_id FK column on rdf_triples (subject-IRI ownership), and the scheme_membership m:n table. Idempotent + reversible backfill: every existing concept -> v1 is_current, triples owned by subject IRI, scheme membership from skos:inScheme. Chains onto the single live head l1_entity_domain_associations. Co-authored-by: Isaac

Build the served hot graph from CURRENT-ONLY triples so reads carry no version predicate. Add rdf_triples_current view (m2) and list_current() repo method: keep unowned scheme/collection/metadata triples, drop only history owned by a non-current concept-version. Point the single graph-build call site (_load_triples_from_db_to_graph) at list_current; ~20 serve-time self._graph read sites unchanged. Co-authored-by: Isaac

Add ConceptVersionsRepository with get_current / get_by_iri_version / list_versions / max_version / demote_current / create_version. demote_current flips the old current to history and flushes BEFORE the new current is inserted, so the partial unique index never sees two is_current rows per iri. Co-authored-by: Isaac

DB-first atomic swap in one Postgres transaction: demote current concept_version to history (flush before insert so the partial unique index never sees two current rows), insert new current (version=max+1, parent=demoted), overwrite changed SKOS fields, reassign the subject's triples to the new version, commit. Then patch the served graph; on patch failure force rebuild_graph_from_enabled and re-raise (recovery contract, named in code). Stable IRI never changes. Add POST /semantic-models/concepts/version/publish (READ_WRITE, editable-scheme gated) with PublishVersionRequest/Response per signed-off API contract §4 (no graph_refreshed field; DB-committed-but-patch-failed returns 500, not success). Add reassign_subject_to_concept_version repo helper. Co-authored-by: Isaac

…ecated' A prior concept-version replaced by a newer one is superseded (concept stays active; only that version is historical), semantically distinct from deprecated (stop using the concept) / retired (tombstoned). Add ConceptStatus.SUPERSEDED; demote_current defaults to it. Per product decision on P0-3. Co-authored-by: Isaac

Add count_for_iri to the entity_semantic_links repo (the physical UC/asset reference count) and DCT/PROV namespaces to the manager for 2B split lineage links (dct:isReplacedBy / dct:replaces / prov:wasRevisionOf). Co-authored-by: Isaac

…(P0-6) Add reference_count (entity_semantic_links rows + concept->concept broader/ narrower/related/subClassOf refs), deprecate_concept (status=deprecated, stays resolvable; 2B split writes dct:isReplacedBy old->new, prov:wasRevisionOf and dct:replaces new->old), retire_concept (gated on reference_count==0, tombstone status=retired never hard-delete; ReferenceCountError -> 409). Add the three endpoints + Pydantic models per signed-off API contract §5, READ_WRITE + editable-scheme gated, responses carry human label. Co-authored-by: Isaac

Track _graph_last_refreshed on the manager singleton, advanced on every full rebuild AND every targeted write (create/update/publish/deprecate/retire all route through _invalidate_cache, which now bumps it after the post-commit graph patch). Add graph_freshness() (last_refreshed + concept_count + separate UC sync timing reusing the existing uc_tag_sync job-run history, null if absent) and reload_graph() (forces rebuild_graph_from_enabled). Add GET /semantic-models/graph/freshness (READ_ONLY) and POST /graph/reload (READ_WRITE) + GraphFreshnessResponse per signed-off API contract §6. Audit: all new write paths (P0-3 publish, P0-6 deprecate/retire) already DB-first-then-patch the graph and call _invalidate_cache; no gap found. Co-authored-by: Isaac

P0-1 backfill only versioned pre-existing concepts; freshly created ones had no concept_version row, so a later publish computed max_version=0 -> new_version=1 and history was empty. create_concept now inserts version=1 (is_current=true, status=draft) and assigns the new triples' concept_version_id by subject IRI, in the same commit as the concept create. Co-authored-by: Isaac

Build the signed-off contract §1/§2 GET reads that were never implemented: - GET /semantic-models/concepts/version?iri= -> ConceptVersionInfo (current version + newest-first history, required label, replaces_iri/replaced_by_iris lineage from the graph). - GET /semantic-models/concepts/version/detail?iri=&version= -> concept detail by (iri,version) key + version/is_current (cold fetch, no hot-graph touch). Both registered ABOVE the /concepts/{concept_iri:path} catch-all so it does not swallow /version. READ_ONLY. Logic layered into the manager + concept_versions repo (list_versions/get_by_iri_version). Co-authored-by: Isaac

The 6-col uq_rdf_triple (s,p,o,lang,datatype,context) BLOCKS a per-version snapshot row (same triple, different concept_version_id) — verified: 2nd insert raises UniqueViolation. Migration m4 widens it to 7 cols with UNIQUE NULLS NOT DISTINCT so snapshots coexist while NULL-owned/metadata triples stay deduped. add_triple/add_triples_bulk conflict target now includes concept_version_id; add_triple accepts concept_version_id so a snapshot row is born owned by v2. Model UniqueConstraint updated to match (postgresql_nulls_not_distinct). Co-authored-by: Isaac

Publish previously destroyed the prior version's field values: it mutated the concept's triples in place then reassigned ALL of them to v2, leaving v1 owning zero triples — the old definition was gone and P0-4 had nothing to diff. Rework publish to snapshot-per-version: demote v1 (its rows stay owned by the demoted id = frozen snapshot), create v2, COPY the concept's current triples into a new set owned by v2 (copy_triples_to_version), then apply changes to v2's set ONLY (remove_by_subject_predicate + add_triple now version-scoped via concept_version_id). v1's rows are never touched. Served graph shows current-only: _swap_concept_in_graph now re-adds via list_current_by_subject (not list_by_subject) so the frozen v1 rows don't leak. get_concept_version_detail reads the (iri,version)'s OWN triples (list_by_concept_version) so v1 returns its OLD definition text, v2 the new. Co-authored-by: Isaac

…ngle-current index; empirical uc-sync cadence - Add src/tests/integration/test_concept_versioning.py (8 tests): publish bumps version + stable IRI, snapshot preserves prior definition, prior = superseded, current-read no history leak, retire gate 409-then-tombstone, deprecate-with-successor records isReplacedBy, partial unique index rejects a second is_current row. Mirrors test_knowledge_routes.py fixtures. - concept_versions: declare the partial unique index with sqlite_where too, so metadata.create_all reproduces the single-current invariant on the SQLite test DB (was degrading to a full-unique index there). No Postgres change. - _uc_sync_timing: derive next-run estimate from observed run cadence instead of a hardcoded 4h assumption (interval is configurable); null when it can't be inferred from a single run. Co-authored-by: Isaac

…bel, malformed-import negative Fills the programmatic gaps the E2E test plan flagged: - served graph is current-only: list_current + a rebuilt graph exclude the superseded (history) triple while keeping unowned metadata (read-isolation at the build layer, independent of the concept-detail read path). - Simple/Advanced contract: version-info and publish responses always carry a human label, so the Simple view never resolves a raw IRI. - negative import: malformed RDF -> 400 and the collection's concept_count is unchanged (no partial apply). No migration (tests only); single head m4 unchanged. 5/5 pass, versioning suite regression 8/8 green. Co-authored-by: Isaac

mvkonchits-db · 2026-08-21T07:59:31Z

Superseded by #734 (feat/concepts-v2). Verified the content is folded in: the distinctive modules/tests for this PR are present on the concepts-v2 branch. Closing to consolidate the concepts-v2 work into a single PR.

mvkonchits-db added 12 commits August 12, 2026 18:08

mvkonchits-db mentioned this pull request Aug 13, 2026

[Draft] Concept versioning UI — save-new-version, history panel, successor badges mvkonchits-db/ontos#1

Draft

This was referenced Aug 13, 2026

[Draft] Concept reference-reconciliation worklist — split (1→N) + merge (N→1) #704

Draft

[Draft] Concept diff engine — file re-upload as a bulk versioning event (P0-4) #705

Closed

[Draft] Concept upload history + rollback (P2-1) #715

Closed

mvkonchits-db closed this Aug 21, 2026

Navigation Menu

[Feature]: Concept versioning engine (P0 core) — schema, atomic publish, safe transition, freshness - #703

[Feature]: Concept versioning engine (P0 core) — schema, atomic publish, safe transition, freshness#703
mvkonchits-db wants to merge 14 commits into
developmentfrom
feat/concept-versioning-clean

mvkonchits-db commented Aug 12, 2026

Uh oh!

mvkonchits-db commented Aug 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mvkonchits-db commented Aug 12, 2026

What this is

What's included (P0-1/2/3/6/8)

Explicitly out of scope

Verification

Uh oh!

mvkonchits-db commented Aug 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant