[Draft] Concept diff engine — file re-upload as a bulk versioning event (P0-4) - #705
Closed
mvkonchits-db wants to merge 5 commits into
Closed
mvkonchits-db wants to merge 5 commits into
mvkonchits-db wants to merge 5 commits into
Conversation
New module concept_diff.py: canonicalizes both the incoming graph and the
current stored triples with rdflib to_canonical_graph BEFORE grouping by
concept subject IRI, so blank-node-heavy re-uploads (owl:Restriction etc.)
do not report spurious changes. Returns {unchanged, modified, new, removed}
buckets, with incoming human-field values for modified/new.
Co-authored-by: Isaac
apply_upload_as_versioning_event composes the existing versioning primitives per diff bucket: MODIFIED -> publish_concept_version (v2), NEW -> import file-native triples + mint v1, REMOVED -> deprecate_concept (tombstone only, never retire/hard-delete), UNCHANGED -> no-op. Atomicity: inner primitive commits are redirected to flush for the duration of the apply so the whole changeset is ONE Postgres transaction; any exception restores commit and rolls back (no half-replace). The served graph is refreshed once after commit (DB-first recovery contract). Also adds _ensure_concept_version_v1 (lazily mints v1 for file-imported concepts that never got a concept_version row) and _import_new_concept_from_graph (inserts a new concept keeping its file-native IRI + blank-node closure). Co-authored-by: Isaac
replace_content now routes to apply_upload_as_versioning_event when the model already has stored triples (a re-upload), diffing instead of the blind remove_by_context + re-import. First-ever uploads (no prior context) stay on the plain import path since there is nothing to diff. Co-authored-by: Isaac
Restrict bucketing to subjects carrying a versionable rdf:type (skos:Concept, owl:Class, rdfs:Class, property types). ConceptScheme / owl:Ontology header subjects and blank nodes are excluded so they are never routed through the concept versioning primitives (which call get_concept and would fail on non-concept subjects). Co-authored-by: Isaac
test_concept_diff_engine.py covers all acceptance bullets: byte-identical bnode-heavy (owl:Restriction) re-upload mints ZERO versions (canonicalization proof), one changed definition -> exactly one v2, added concept -> v1, removed-but-referenced -> deprecated + resolvable + ref survives, removed-unreferenced -> deprecated tombstone (no hard delete), mid-apply failure -> whole changeset rolls back. publish_concept_version / deprecate_concept gain bypass_editable_gate (default False, all existing callers unchanged): the re-upload path is the sanctioned write channel for file-sourced collections auto-registered as non-editable. Exception path now also rebuilds the served graph so a partial in-memory patch is reconciled with the reverted DB. Co-authored-by: Isaac
Contributor
Author
|
Superseded by #734 (feat/concepts-v2). Verified the content is folded in: the distinctive modules/tests for this PR are present on the concepts-v2 branch. Closing to consolidate the concepts-v2 work into a single PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DRAFT — not for review yet. Stacked on the concept-versioning engine (#703); must merge AFTER it. Held from review-ready until the full stack passes E2E.
What this is (P0-4)
Turns re-uploading an ontology file from a wholesale wipe-and-replace into a smart, atomic versioning event. Backend only. Composes the existing versioning primitives — no new schema, single alembic head unchanged (
m4_rdf_triple_version_uq).How it works
replace_content()routes a re-upload (a model that already has stored triples) toapply_upload_as_versioning_event(); first-ever uploads stay on the plain import path (nothing to diff).compute_concept_diff()canonicalizes BOTH the incoming graph and the current stored graph with URDNA2015 (rdflib.compare.to_canonical_graph) BEFORE grouping by concept subject IRI, then buckets each concept:publish_concept_version(demote + snapshot + swap = v2).deprecate_conceptONLY — tombstone, never retire/hard-delete, regardless of reference count.Stored skolemized blank nodes are de-skolemized before canonicalization so URDNA2015 re-derives stable labels; object-fingerprints capture bnode closures so
owl:Restriction-style structures compare stably.All-or-nothing atomicity
The composed primitives each
commit()internally; the orchestrator neutralizes those (redirectscommit→flush) for the duration of the apply and issues ONE real commit at the end. Any exception restorescommit, rolls back (no half-replaced file), rebuilds the served graph, and re-raises.commitis restored on every path (except + finally).Tests —
test_concept_diff_engine.py, 6 passedowl:Restriction) re-upload → 0 versions (canonicalization proof).Regression: the versioning suite (8) still green.
Not included
This pull request and its description were written by Isaac.