feat(concepts): concepts-v2 lifecycle, review, versioning, enrich + fixes - #734
Open
mvkonchits-db wants to merge 127 commits into
Open
mvkonchits-db wants to merge 127 commits into
mvkonchits-db wants to merge 127 commits into
Conversation
Add a 'Group by' lens (none/scheme/source/domain) to the Concepts browse view, layered onto the existing glossary preferences store. The easy-bar chip selection keeps its OR/union filtering via hiddenSources; the lens only re-organizes terms, it is not a second filter. - store: canonical groupByDimension field + setGroupByDimension; legacy groupBySource/groupByDomain kept in sync both ways; persist v1 migration derives the lens from old booleans. - filter panel: single Group-by Select replaces the two grouping switches. - new Vitest coverage for the store (lens, sync, union, migration). Co-authored-by: Isaac
The linked-objects panel deleted a semantic link via a raw fetch(), with
no confirmation and an ambiguous X icon. Route it through the app's api
client (useApi().delete), add an AlertDialog confirm ("unlinks, does not
delete the entity"), and use the Trash2 icon. Refresh reuses the existing
bumpKnowledgeGraphRefresh + fetchLinks sequence. Backend endpoint
(DELETE /api/semantic-links/{id}) unchanged; already gated + audited.
Addresses the Claro single-link-delete usability gap (CB-9).
Co-authored-by: Isaac
…merge) Databricks Apps cloud build runs npm; development still ships yarn.lock, so the frontend build fails. main is already npm-only (#657) but that fix has not been back-merged to development (blocked on PR #656 conflicts). Cherry-pick main's matched npm build system so this branch can deploy to Apps for FEVM validation: - take main's src/frontend/package.json + package-lock.json (matched pair) - take main's npm-only src/scripts/build_static.sh - remove src/frontend/yarn.lock and .npmrc TEMPORARY: drop this commit once #656 lands the back-merge on development. Revert point tagged pre-npm-cherrypick. Co-authored-by: Isaac
…EMP) Fresh-DB startup crashed: 'domain_id is an invalid keyword argument for TeamDb'. The domain migration removed the domain_id column from TeamDb (teams now use the entity_domain_associations junction), but the default Admin Team seeding in ensure_default_team_and_project still passed domain_id=None to the TeamDb constructor. Pre-existing bug on development (identical on upstream/development; this branch never touched settings_manager.py). Only fires on a cold start / fresh database, so already-seeded apps never hit it. Filing separately. TEMP: carry until development is healthy via #656 back-merge + upstream fix. Co-authored-by: Isaac
Restructure the Concepts nav from a 7-item vertical sidebar into the v2 three-section IA: horizontal Define/Explore/Enrich tab strip with a section-scoped sub-nav; active section derived from the URL. - Define: Collections (author) + Generator + Import - Explore (default): concept browser + Search + Graph + Hierarchy as view toggles over the existing view components (internals untouched) - Enrich: Mapping (delivery lane TODO — no standalone view yet) Nav shell + routing only; the 7 view components are reused as-is. All existing /concepts/* URLs preserved (zero churn), browser/:iri deep link intact, legacy redirects unchanged. Co-authored-by: Isaac
Explore was 4 self-contained views behind a toggle. Fold List/Tree/Graph into ONE surface driven by a single fetch + shared filtered selection, per the v2 wireframes. Rendering engines untouched (ConceptsTab, KnowledgeGraph/cytoscape, GraphTab are fed by props only). - new hooks/use-explore-concepts.ts: single concepts-grouped fetch + the filteredConcepts memo (was duplicated in both views) - new views/explore.tsx: container with GlossaryFilterPanel + Group-by above a List|Tree|Graph view-mode switch (?view= param); graph-local hiddenRoots stays local to Graph mode - business-terms.tsx -> thin re-export of explore; ontology-home unrouted - SPARQL search + estate (instance) hierarchy demoted out of the Explore toggle to power-user links; all legacy routes still redirect (no 404s) - browser/:iri deep link preserved TODO(cb-v2): dedicated List-vs-Tree sub-toggle inside ConceptsTab (today both render via ConceptsTab's grouping; not forced, no engine rewrite). Co-authored-by: Isaac
Three UI fixes on the unified Explore surface: - remove the Search/SPARQL + Estate-hierarchy secondary links from the Explore nav (they are not concept-browse); routes preserved, no 404s - drop the duplicate 'Concepts' <h1> in the layout; the view owns the single header, reclaiming vertical space - add a real flat List view-mode to ConceptsTab (viewMode 'list'|'tree', default tree). List renders concepts alphabetically with no broader/ narrower nesting, reusing the existing row markup via a defaulted flat param so the tree render path is unchanged. Explore wires viewMode to the ?view switch; graph unchanged. Co-authored-by: Isaac
Parallel build of the three remaining v2 frames (Explore already unified), integrated behind /concepts/define and /concepts/enrich routes + nav. - Define (views/define.tsx): Author/Generate/Import path cards + in-progress runs list; import dialog extended to multi-file + scheme-strategy (merge vs one-per-file), conflicts deferred to Review Board. Real: per-file import loop. TODO(cb-v2): atomic multi-file + structural summary endpoints. - Enrich (views/enrich.tsx + components/enrich/*): Map coverage matrix with inline 'Review suggested matches' (reuses the real term-mapping suggester); Deliver lane (Tags live / descriptions planned / glossary coming) + mode cards (advanced). Coverage counts are placeholder pending a read-model. - term-detail: linked-assets aligned columns + nested chains + per-row Open; Compare control by the version; 'Changes from v1.0.0' diff panel MOCK (TODO(cb-v2): real version-diff, Track 3, pending granularity). Routes/nav wired; engines untouched; all URLs preserved. tsc clean; 20 tests. Co-authored-by: Isaac
… nested links Add three additive read-model capabilities behind the Concept Builder v2 UX: - POST /api/semantic-links/mapping-status: batch per-IRI asset/product/contract layer flags in a single query (list_for_iris), for the Explore mapping column. - GET /api/knowledge/coverage: per-concept-scheme enrichment coverage (concepts, coverage %, distinct product/contract/asset link counts) + totals. - parent_entity_id on EntitySemanticLink, derived at serialization time for contract->schema->property chains, so linked assets can render nested. Co-authored-by: Isaac
…le, merged relations Browser-validation feedback across Define/Explore/Enrich/term-detail: - Nav: drop legacy sub-menus; compact Explore header (no redundant title). - Shared Simple/Advanced mode switch (mode-switch.tsx) on Explore/Define/Enrich/ detail via one useSyncExternalStore store; real adv-only content on each screen (IRIs, SKOS/RDF footers, Under-the-hood legend) so the toggle has visible effect. - Explore list: real columnar table (checkbox, Kind, Scheme, Mapping, Status, edit pencil), mapping column consumes /mapping-status, bulk Set-status wired to lifecycle endpoints; tree gains an edit pencil. - term-detail: two-column panel grid; Relations + neighbourhood graph merged into one block with a List/Graph switch (same /neighbors data); Preview change-history. - Define: Author opens New-concept-scheme dialog in place; Generate opens a guided prompt-builder (guided-generate-dialog.tsx) that feeds the generator via ?guidelines=; generator breadcrumb Concepts > Define > Ontology Generator. - enrich.tsx consumes the real coverage read-model. Co-authored-by: Isaac
…st Enrich actions Round-3 browser-validation fixes: - Explore Status column was always blank: _compute_all_concepts (the list cache builder) never read ONTOS.status. Populate it so lifecycle status renders. - Linked-assets nav threw 'Cannot navigate' for data_contract_schema / data_contract_property rows (no case) — route them to the parent contract. - Explore Create dropdown relabelled to 'New concept' / 'New concept scheme' (was the confusing 'Collection' term). - Enrich Suggest matches + Tag sync were clickable no-ops; disable them with a 'coming soon' tooltip until the suggester run / uc_tag_sync trigger are wired. Co-authored-by: Isaac
The Create/Edit concept dialog had no mode switch, so the ontology layer was always shown. Add the shared ConceptModeSwitch to the header and gate the Type selector and property Domain/Range config behind Advanced view. Simple mode shows Name/Definition/Synonyms/Examples and defaults new concepts to skos:Concept; Advanced reveals the type + property configuration. Co-authored-by: Isaac
Expose what was hidden in Settings, per the Map/Deliver design:
- Suggest matches: a Configure dialog (concept source: customer ontologies or
an opted-in shipped taxonomy; target type; heuristic vs LLM engine) that runs
POST /api/term-mappings/runs, spawns a Review Board request via /runs/{id}/review,
and navigates there. Handles 'no customer ontology' (422 guidance) and
'no matches' (informational, not an error) honestly. Requires term-mapping write.
- Tag sync: a single Configure dialog (scope: all vs specific schemes; run mode:
run once now vs on-schedule, schedule read-only) that triggers the uc_tag_sync
job via POST /api/jobs/workflows/uc_tag_sync/start. Disabled with a clear reason
when the user lacks jobs admin or the job is not installed. Per-run scope is
captured but advisory (the start endpoint takes no scope param today).
- enrich.test wraps EnrichView in a Router (dialogs use useNavigate).
Co-authored-by: Isaac
Replace the lightweight custom Enrich dialogs with the real, full-featured surfaces, avoiding duplication and a conceptual conflict: - Suggest matches now opens the existing RunConfigDialog (full run config: customer ontologies + shipped contexts + target types + engine), then the shared GenerateReviewDialog (spawns the Review Board request and navigates). Deletes suggest-matches-dialog.tsx. - Tag sync: the uc_tag_sync job is a single shared job spanning many aspects (semantic assignment is just one config), so running/scoping it from Enrich misrepresents it. The Tags row now shows stats read-only with a 'Manage in Settings' action to /settings/jobs, where its real scope + schedule live. Deletes tag-sync-dialog.tsx; adds manageHref to DeliveryTarget. Co-authored-by: Isaac
…coverage note
- New GET /api/knowledge/tag-delivery-stats: eligible concept->asset links,
pending = links created since the last successful uc_tag_sync run, synced =
eligible - pending, plus last-run state/time and install status. Honest
'changes since last sync' signal from existing data (link created_at vs job
end_time); no per-link delivery log, edits not counted. Tags row now shows
these real numbers instead of placeholder {82/100}.
- Map lane: removed the Heuristic/LLM engine switch (engine selection lives in
the reused run-config dialog now).
- Coverage matrix: the 'sample data / endpoint does not exist' note now shows
only when the live coverage read-model is actually unavailable (isLive=false).
Co-authored-by: Isaac
…ralized mode switch - Concept page: add a Change-status control offering the valid lifecycle transitions from the current status (submit-review/approve/publish/certify/ deprecate/archive), wired to the by-iri action endpoints and permission-gated. This was entirely missing before. - Concept header declutter: name on its own row; type/status/version/source as a single badge row (version no longer orphaned under the icon); raw IRI on its own advanced-only row; synonyms and examples each on their own labelled row. - Move the Simple/Advanced switch UP into the Concepts section-tab row so it aligns with the menu, with a bottom boundary under the nav; removed the four duplicated per-view switches. Tightened per-view top spacing. Co-authored-by: Isaac
- Concept header: move the version badge to the leftmost position of the meta row (before type/status/scheme), per feedback. - Tag delivery: tag-delivery-stats now returns the actual pending items (not just a count); the 'N pending' figure in the Enrich Tags row is clickable and opens a dialog listing the concept-to-asset links pending since the last sync. - Coverage readout shows synced/pending whenever coverage data exists (not only when synced>0), so pending is visible before the first successful run. Co-authored-by: Isaac
Drop the provenance gate that limited the suggester to urn:semantic-model:* uploaded ontologies. A concept scheme is a concept scheme: authored/imported schemes (urn:glossary / urn:taxonomy / urn:ontology) and uploaded RDF sources (urn:semantic-model) are all selectable now. Internal indexes stay blocked and shipped taxonomies stay opt-in (excluded from defaults to avoid noise). is_customer_context now accepts any concept-scheme prefix minus internal/shipped; validate_contexts and the default-context resolver follow. The bulk run's default (all schemes) now includes authored glossary schemes, so Suggest matches from Enrich works against Concept Builder schemes without a separate RDF upload. Co-authored-by: Isaac
… scheme dialog - Explore filter panel now seeds sources from editable collections (flattened across the hierarchy), so schemes with 0 concepts — including nested child schemes — appear as chips instead of vanishing. - ?source= handler normalizes full IRIs to stripped names and guards against hiding every source when the target is absent (no more blank Explore). - Author/Edit dialog renamed Collection -> scheme; Type gains a hover info tooltip explaining the glossary/taxonomy/ontology maturity ladder. Co-authored-by: Isaac
… truncation, cap indicators The agent loop treated ANY non-tool text message as final Turtle, so a tool-call-emitted-as-text (a known Llama tool-calling instability) was parsed as an ontology, failed, and still reported success with 0 classes. - Reliability: non-usable final output (parse fails / <1 class+property / leaked tool call) triggers a corrective nudge + loop continue, up to 2 repair attempts, before giving up. - Truncation: inspect finish_reason; on 'length' continue-generate and stitch (bounded, MAX_CONTINUATIONS=6) so full ontologies complete up to the caps. - Honest success: success=True only after a parse yielding >=1 class/property; otherwise the run is recorded failed with the real reason. - Cap indicators: surface tables_resolved vs tables_used + a caps_note; the generator page shows an informational banner when a cap was hit. Does not raise the caps (large-estate chunk/merge remains a separate feature). - No change to owl_parser behavior (shared by other managers). Co-authored-by: Isaac
…ve in results header Co-authored-by: Isaac
Add concept_version table (versioned unit = (iri, version); is_current hot set with a partial unique index UNIQUE(iri) WHERE is_current), a concept_version_id FK column on rdf_triples (subject-IRI ownership), and the scheme_membership m:n table. Idempotent + reversible backfill: every existing concept -> v1 is_current, triples owned by subject IRI, scheme membership from skos:inScheme. Chains onto the single live head l1_entity_domain_associations. Co-authored-by: Isaac
…ng + Define capability clarity - Cap warning no longer false-alarms on schema/catalog selection (can't count tables client-side); neutral always-on caption + amber only when >50 individual tables selected. Real overflow still caught by post-run banner. - Generated concepts saved as Draft (import_rdf_to_collection default_status) so they flow through review, matching manual authoring. New test covers it. - Rename Collection -> concept scheme in generator save dialog. - Define page: capability-clarity block framing outputs (Concepts vs Classes & properties) + what each path produces. Co-authored-by: Isaac
Build the served hot graph from CURRENT-ONLY triples so reads carry no version predicate. Add rdf_triples_current view (m2) and list_current() repo method: keep unowned scheme/collection/metadata triples, drop only history owned by a non-current concept-version. Point the single graph-build call site (_load_triples_from_db_to_graph) at list_current; ~20 serve-time self._graph read sites unchanged. Co-authored-by: Isaac
…nce, show per-card Replace the bulky capability matrix with a slim one-line legend (Glossary/Taxonomy/ Ontology, each with a reused icon + one-line definition), then repeat the same three icons inside every path card as a 'Builds' row — lit for what the path produces, dimmed otherwise, hover for a plain explanation. Teaches the vocabulary once and reuses it per card. whitespace-nowrap guards against row wrap. Co-authored-by: Isaac
Add ConceptVersionsRepository with get_current / get_by_iri_version / list_versions / max_version / demote_current / create_version. demote_current flips the old current to history and flushes BEFORE the new current is inserted, so the partial unique index never sees two is_current rows per iri. Co-authored-by: Isaac
DB-first atomic swap in one Postgres transaction: demote current concept_version to history (flush before insert so the partial unique index never sees two current rows), insert new current (version=max+1, parent=demoted), overwrite changed SKOS fields, reassign the subject's triples to the new version, commit. Then patch the served graph; on patch failure force rebuild_graph_from_enabled and re-raise (recovery contract, named in code). Stable IRI never changes. Add POST /semantic-models/concepts/version/publish (READ_WRITE, editable-scheme gated) with PublishVersionRequest/Response per signed-off API contract §4 (no graph_refreshed field; DB-committed-but-patch-failed returns 500, not success). Add reassign_subject_to_concept_version repo helper. Co-authored-by: Isaac
…ecated' A prior concept-version replaced by a newer one is superseded (concept stays active; only that version is historical), semantically distinct from deprecated (stop using the concept) / retired (tombstoned). Add ConceptStatus.SUPERSEDED; demote_current defaults to it. Per product decision on P0-3. Co-authored-by: Isaac
Add count_for_iri to the entity_semantic_links repo (the physical UC/asset reference count) and DCT/PROV namespaces to the manager for 2B split lineage links (dct:isReplacedBy / dct:replaces / prov:wasRevisionOf). Co-authored-by: Isaac
…(P0-6) Add reference_count (entity_semantic_links rows + concept->concept broader/ narrower/related/subClassOf refs), deprecate_concept (status=deprecated, stays resolvable; 2B split writes dct:isReplacedBy old->new, prov:wasRevisionOf and dct:replaces new->old), retire_concept (gated on reference_count==0, tombstone status=retired never hard-delete; ReferenceCountError -> 409). Add the three endpoints + Pydantic models per signed-off API contract §5, READ_WRITE + editable-scheme gated, responses carry human label. Co-authored-by: Isaac
…inkable The dialog rendered "linked to <long id>" (the asset UUID / a urn). Surface the meaning instead: - CONCEPT (linkable -> /concepts/browser/<iri>) + its SCHEME (linkable -> /concepts/browser?source=<scheme>). Raw urn only in the tooltip. - physical ASSET it will be tagged on, by human name, linkable: asset -> /assets/<id>; uc_table/column -> /catalog-commander?table=<fqn>. Backend: TagPendingItem gains scheme/scheme_label/asset_name; get_tag_delivery_stats enriches each item best-effort (concept source_context + scheme labels from get_collections + batch-resolved AssetDb names), degrading to None on any error. Co-authored-by: Isaac
…t, submit governance signal Concepts v2 feedback batch: - Import conflict dialog lists the conflicting concepts (name + "Already in <scheme>"), not the parent scheme repeated per row. - Deleting a scheme now purges concept_version rows for file-native IRIs too (collect the context's actual subjects, not just prefix-matched), fixing version-history leak across repeated deletes. - sourceFile provenance overwrites on re-upload instead of sticking to the first filename. - Source-context groups show "from <file>" when derived from a single upload. - File uploads always land as Draft: the bulk concept_changeset approval gate is disabled (gate_or_apply_upload applies directly; merge-in new concepts land draft). - concept_changeset removed as a selectable workflow trigger entity (gate is off); EntityType union kept. - skos:ConceptScheme header node no longer enumerated as a browseable concept (removed phantom scheme-named concept). - Imported concepts get a v1 (draft) version row so the badge isn't blank. - Submit-for-review dialog shows whether an approval workflow will gate the transition (new read-only preview endpoint, fires nothing). - "Save new version" disabled until the concept is past draft/under_review. Changeset-gate acceptance tests updated to the direct-apply contract (2 parked/skipped with un-skip pointers). Co-authored-by: Isaac
Headless REST suite against a live Ontos app covering the failure patterns unit/regression miss: lifecycle/state (delete->recreate no version leak, upload-as-draft, v1 mint), cross-surface (conflict payload shape, no phantom ConceptScheme concept, sourceFile provenance), governance (submit-review preview + no side effect, changeset trigger hidden). Self-cleaning E2E-AUTO- schemes. 11/11 pass on ontos-cbv2b (deployment 01f19af4). Co-authored-by: Isaac
…upload, ConceptScheme count/detect Adversarial review of the prior batch surfaced: - C2 (real): a re-uploaded MODIFIED concept kept its stale ontos:sourceFile because publish_concept_version copies the demoted version's triples into v2 and _apply_publish_changes_to_db only rewrote a fixed allowlist. Thread source_file through concept_diff._extract_changes + _PUBLISH_LITERAL_FIELDS so the new version's provenance is rewritten. Added a regression test. - ConceptScheme count mismatch: class_count_query still UNIONed skos:ConceptScheme after it was dropped from enumeration, so the displayed concept count could exceed the listed rows by one. Removed the UNION. - ConceptScheme in _detect_concept_type: a direct get_concept on a scheme IRI still resolved as concept_type='concept'. Removed the residual mapping. - Doc accuracy: refreshed gate_or_apply_upload + confirm-route docstrings to the Scenario-D direct-apply contract; clarified the now-vestigial monkeypatch in the rewritten gate test (kept as a re-open canary). Co-authored-by: Isaac
…-upload Adds t_reupload_refreshes_sourcefile: import v1 -> re-upload modified concept from a new filename -> confirm -> assert source_file adopts the new name. 12/12 pass on ontos-cbv2b (deployment 01f19afa). Co-authored-by: Isaac
…submission
Concept review now supports the intended TWO-workflow model:
- APPROVAL wizard (for_request_status_change / ontology_concept) runs BEFORE
submit; on completion it replays the real /submit-review call.
- PROCESS gate (on_request_status_change) holds the concept after submit until
an approver decides (already wired).
Backend:
- enforce_wizard_permission is now entity-aware: for_request_status_change on
ontology_concept gates on semantic-models:READ_WRITE (not data-products), via
_WIZARD_ENTITY_PERMISSION_OVERRIDES. entity_type threaded through /for-trigger,
GET /{workflow_id}, and the approval-session routes. Products unchanged.
- get_workflow_by_trigger_type already filters on entity_type (confirmed).
- 5 focused permission tests (concept steward allowed w/o data-products; denied
w/o semantic-models; product default unchanged).
Frontend:
- SUPPORTED_TRIGGER_ENTITY_MAP.for_request_status_change += ontology_concept so
a concept approval wizard is authorable in the designer.
- concept-detail submit flow resolves the concept-scoped wizard via
/for-trigger?entity_type=ontology_concept; launches ApprovalWizardDialog when
configured, else the plain submit dialog (S3 banner intact). Submit fetch
refactored into a shared fn called from both paths.
Co-authored-by: Isaac
Adds t_approval_workflow_model: verifies (a) the APPROVAL wizard resolves for the concept entity via /for-trigger?entity_type=ontology_concept, and (b) the PROCESS gate holds a submitted concept in under_review (governed=True). 14/14 pass on ontos-cbv2b (deployment 01f19b04) with both workflow instances configured. Co-authored-by: Isaac
…view (B1.3) Approver notification popup improvements for concept review: - Backend: submit_concept_for_review now carries the concept's definition, concept_type and source_context into review_context, so the approval notification payload shows what's being approved (was label + diff only). - FE: the approval dialog surfaces Definition / Type / Scheme rows and makes the concept a linkable Resource (entity-detail-path resolves ontology_concept to /concepts/browser/<iri>). humanizeEntityType renders 'ontology_concept' as 'Concept'. - FE: dialog now caps height (max-h-85vh) and scrolls its body so a rich concept payload no longer overflows the dialog shape. - FE: on an approval decision for a concept, bump the knowledge-graph refresh nonce so an open concept-detail page refetches immediately (fixes the refresh-lag where a rejected concept still showed under_review until manual reload; backend already moved it to draft). E2E: assert the approval notification action_payload carries the definition. Co-authored-by: Isaac
…by file Two fixes for multi-file concept-scheme merges: - Additive merge (data-integrity bug): merging N files into one scheme sent files 2..N through the re-upload DIFF path, which deprecated earlier files' concepts (diff vs the whole scheme treated them as "removed"). New `additive` query param on POST .../import skips the diff/preview path so every file appends as Draft, nothing deprecated. FE gates it on a multi-file "One scheme" merge (schemeStrategy==='merge' && files>1); single-file re-upload keeps the diff/version path. New integration test covers both. - Group-by cleanup: "Scheme" and "Source" were redundant (both keyed on source_context). Repurposed "Source" to group by the originating FILE (source_file) while "Scheme" stays the concept scheme (source_context), so a merged multi-file scheme splits into its per-file origins. Concepts with no source_file fall under a "No source file" group. Factored a groupKeyOf() extractor; Scheme grouping is byte-identical when Source/File isn't selected. Co-authored-by: Isaac
… comment visibility
Three tracker-failure fixes (CV2-UI-08, CV2-UI-03) + a bug found via E2E:
- CV2-UI-08 deprecate reference gate: deprecate_concept now refuses (409) a
still-referenced concept unless successor IRIs are supplied (2B remap); the
re-upload/versioning engine (bypass_editable_gate=True) still tombstones. Route
maps ReferenceCountError->409. New test_deprecate_reference_gate.py.
- CV2-UI-03 Enrich Map apply: the "Review suggested matches" flow now runs on
REAL data. New GET /knowledge/coverage/{scheme}/pending-suggestions +
repo.list_pending_by_target_concepts. FE Accept&apply-all posts run decisions
({id,decision:'accept'}) then /apply, then refreshes coverage + tag stats.
submitting/done guards fix the endless-click; table + Deliver counter refresh
without a manual reload.
- Reviewer send-back comment visibility (found via B1/B2 E2E): get_concept /
get_concept_details now read ONTOS.reviewComment + reviewDecision into the
model (via the shared metadata extractor); the FE callout shows in DRAFT
(post send-back), not just under_review, so the owner sees why it was returned.
Verified: baseline 7-fail/413-pass -> with changes 7-fail/420-pass (same
pre-existing failures, +7 new passing tests, zero regressions). FE tsc + build clean.
Co-authored-by: Isaac
tasks/concept_lifecycle_e2e.py drives the review lifecycle over REST against a live app: B1 governed ping-pong (submit->reject-via-workflow->resubmit->approve), B2 ungoverned (clean SKIP — the concept-review gate is scope=ALL by design), and a CUJ full lifecycle (draft->approved->published->certified) + the deprecate reference gate. Self-cleaning E2E-AUTO- schemes. Co-authored-by: Isaac
…links IRIs containing '//' (e.g. https://ontos.example.org/billing#Invoice) had their %2F%2F collapsed by the Databricks Apps proxy on the path-form endpoint GET /api/semantic-links/iri/{iri:path}, which 301-redirected to a mangled path so the backend matched nothing and returned []. The link existed in the DB (Explore's POST-body mapping-status showed asset=true), but the concept detail LinkedObjectsPanel showed nothing and a hard refresh could 'lose' the concept. Add GET /api/semantic-links/by-iri?iri= (query form, proxy-safe, mirrors concepts/by-iri) and switch the 3 FE callers (linked-objects-panel, concepts-search, properties-search) to it. Keep the legacy path route for back-compat. Co-authored-by: Isaac
Move the two Concepts-v2 live-app E2E suites from tasks/ into src/backend/src/tests/e2e/ so they live with the rest of the test tree. They are manual/live harnesses (need a running app URL + bearer token), not headless CI tests, so: (1) filenames keep the *_e2e.py suffix (pytest collects test_*.py, so these are never auto-collected/imported), and (2) --base now defaults to ONTOS_BASE_URL with no hardcoded host, exiting 2 (skip) when app URL/token are absent instead of failing. Co-authored-by: Isaac
…g-e2e-cbv2b # Conflicts: # src/backend/src/controller/semantic_links_manager.py # src/backend/src/repositories/rdf_triples_repository.py # src/backend/src/repositories/semantic_links_repository.py # src/backend/src/routes/business_owners_routes.py # src/backend/src/routes/semantic_links_routes.py # src/frontend/package-lock.json # src/frontend/package.json # src/frontend/src/components/common/ownership-panel.tsx # src/frontend/src/components/semantic/linked-objects-panel.test.tsx
Two post-merge conflict-resolution fixes after merging upstream/development:
1. semantic_links_routes: the merge took upstream's version of this file, which
never had POST /semantic-links/mapping-status (our endpoint, consumed by
concepts-tab.tsx for the per-concept mapping badges). Re-add the route +
its MappingStatusBatch{Request,Response} imports. Upstream's entity-prefix
and by-iri routes are kept.
2. linked-objects-panel.test: the merge took upstream's simpler test (expects
direct DELETE on click), but our component gates removal behind an
AlertDialog confirm. Restore our test that clicks the trash affordance,
asserts no DELETE fires yet, then confirms and asserts the DELETE. 8/8 pass.
Co-authored-by: Isaac
… column
Map tab, two issues:
1. 'Deliver to' listed Snowflake/BigQuery/Power BI as selectable, but only the
host (Databricks/UC) has a delivery path and the others had no backing. Wire
the dropdown to ENABLED connections from Settings > Connectors: UC is always
offered (the host), plus one entry per enabled connector_type (deduped;
databricks/uc skipped to avoid a duplicate). With only the default Databricks
connection present, the list now correctly shows just Unity Catalog; adding a
Snowflake/BigQuery connection surfaces it automatically. Platform type widened
to string; PLATFORM_NOUN gets a title-cased fallback for unknown connectors.
2. The coverage matrix 'Last run' column was always empty. Term-mapping runs
store ontology_contexts as full graph IRIs (e.g. 'urn:glossary:finance'), but
the coverage rows key on the SHORT source-context bucket ('finance') from
_extract_source_context, so last_run_by_scheme.get(scheme) never matched.
Normalize the run map through the same extractor in get_coverage_metrics,
keeping the newest run per bucket. Verified against live data: a run with
ontology_contexts=['urn:glossary:finance'] now lands on the finance row.
Co-authored-by: Isaac
…-import) The merge combined our concept-versioning uniqueness (7-col uq_rdf_triple with concept_version_id + postgresql_nulls_not_distinct) with upstream's blank-node re-import idempotency test. NULLS NOT DISTINCT is a Postgres-only clause; on SQLite (the unit harness) NULLs are always DISTINCT, so ON CONFLICT DO NOTHING on the 7-col key never fired for unversioned rows and re-imports of blank-node ontologies duplicated (test_rdf_bnode_dedup: 'assert 12 == 0'). Fix, production-behaviour-preserving: - Add a PARTIAL unique index uq_rdf_triple_null_version on the 6-col natural key WHERE concept_version_id IS NULL (model Index with postgresql_where/sqlite_where + migration m5). Dedups unversioned triples identically on Postgres AND SQLite. - Point the unversioned insert paths (add_triples_bulk — always unversioned; and add_triple when concept_version_id is None) at that arbiter via index_where=concept_version_id IS NULL. Versioned snapshot rows keep the 7-col target untouched. On Postgres this is functionally redundant with uq_rdf_triple's NULLS NOT DISTINCT for NULL-version rows (both forbid the same duplicates), so no observable production change — it only makes the ON CONFLICT arbiter inferrable on both dialects. Unit + key integration suite: 1362 passed, 1 skipped, 0 failed. Co-authored-by: Isaac
mvkonchits-db
changed the title
mvkonchits-db
changed the title
hatchling 1.32.0 enforces 'readme path must be within the project directory' and rejects this package's readme = "../README.md" (the README lives at the repo root, one level above src/). The build requirement was unbounded (hatchling>=1.21.0), so a fresh CI env pulls 1.32.x and every hatch-built job (Backend Tests, Coverage, Alembic single-head) fails at metadata generation with 'AttributeError: module hatchling.build has no attribute prepare_metadata_for_build_editable' -> 'ValueError: Readme path must be within the project directory: ../README.md'. Pre-existing on development (its CI passed on 2026-08-14 only because an older hatchling was resolved then); surfaced here as a fresh CI run. Cap to <1.32 to restore the working behaviour. Proper follow-up: relocate the README into src/ or make the readme field project-local, then lift the cap. Co-authored-by: Isaac
The merged feature branch added substantial branchy UI/logic, diluting vitest branch coverage from upstream's 72.1% baseline to 68.48% and tripping the thresholds.branches:70 floor (all tests passed; it was a coverage-dilution gate). Add tests for high-ROI new/changed code (no production changes, threshold untouched): - enrich: delivery-modes, delivery-targets, coverage-matrix, plus a systematic branch-boost suite over status/permission combinations - common/paginated-list (pagination edge cases) + semantic/status-progress-bar - lib pure-function branches: uc-search-parser display names + default arm, ontology-utils resolveLabel/resolveComment fallbacks + English-first sort, entity-detail-path snake_case/concept arms + non-string guard, rdf-filename short-all-caps-with-digit vs acronym arms Result: branch coverage 70.04% (vitest --coverage exits 0), 1096 tests pass, 0 failures, tsc clean. Co-authored-by: Isaac
A structured design/decision doc for the Concepts-v2 work: a high-level architecture overview (Define/Explore/Enrich journey, versioning engine, review workflow, Enrich) followed by grouped ADRs (Context/Decision/Alternatives/ Consequences) covering the versioning engine, RDF storage + dialect-portable dedup, the two-workflow review model + governance, the import diff engine, Enrich, frontend/UX, and cross-cutting infra (proxy-safe IRI routes, hatchling cap, live E2E harness placement). Closes with the intentionally-deferred scope and its triggers. Indexed with a table of contents + anchor cross-links. Co-authored-by: Isaac
mvkonchits-db
marked this pull request as ready for review
August 19, 2026 14:24
mvkonchits-db
added
scope/ontology
…s-v2-merge # Conflicts: # src/frontend/src/components/workflows/workflow-approval-response-dialog.tsx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR delivers Concepts-v2: a restructure of the Concepts area from a flat glossary into a guided Define -> Explore -> Enrich journey, with full concept lifecycle/versioning, a governed review workflow, and a delivery lane that pushes governed metadata onto the platform. It also carries a set of Enrich/Map and semantic-links fixes and is merged with
developmentso it lands clean (0 behind at time of push).The Concepts-v2 journey (new UI)
The
/conceptsarea is now a sidebar-nested flow with three primary surfaces plus supporting views (Collections, Search/SPARQL console, Generator, Hierarchy):1. Define -
views/define.tsx(new)A landing page with three "path cards" for getting concepts into the catalog, each showing what rungs of the model it builds:
guided-generate-dialog.tsx), landing all generated items as Draft for review.An "In progress" feed surfaces recent generator runs (real data).
2. Explore -
views/explore.tsx(new)A unified browse surface over a single fetch with List | Tree | Graph view-modes (the old
/concepts/graphredirects in). Grouping is meaningful: "Scheme" groups bysource_context, "Source" groups by the originating file (source_file). From here you open a concept's detail page.3. Concept detail -
views/concept-detail.tsx(heavily expanded)The concept page gained a set of governance/versioning panels:
status-progress-bar.tsx) showing the full lifecycle and current position, with single-hop status transitions (no confusing multi-hop "(via ...)").version-history-panel.tsx) + Publish version dialog (publish-version-dialog.tsx) - version the concept as an integer, publish a snapshot per version.deprecate-concept-dialog.tsx) with a reference gate (can't deprecate/retire a referenced concept without a successor).turtle-serialization-panel.tsx) + an in-page SPARQL query scoped to the concept (Advanced view).4. Enrich -
views/enrich.tsx(new): Map + Deliver lanescoverage-matrix.tsx) showing concepts, coverage %, linked products/contracts/assets, pending suggestions, and last run; with an inline "Review suggested matches" dialog (review-suggestions-dialog.tsx) that embeds the shared term-mapping suggester and supports accept-and-apply-all.delivery-targets.tsx: Tags live viauc_tag_sync, Column descriptions / UC Glossary as roadmap) and delivery modes (delivery-modes.tsx: Direct / Indirect / Manual, mapped to Ontos' real Delivery Mode). The "Deliver to" platform picker is wired to enabled connections in Settings > Connectors.Concept review workflow (two-workflow model)
Submitting a concept for review runs an approval wizard (
for_request_status_change) before submit when configured, and an async process workflow (on_request_status_change) gates thedraft -> under_reviewtransition with a real approver ping-pong. The full lifecycle:draft -> under_review -> approved -> published -> certified, withchanges_requestedbouncing back todraft(carrying the reviewer comment) and reference-gated deprecation.Concept versioning engine (P0-1)
The versioning spine (migration
m1_concept_versioning, mirroring the Data Products versioning pattern) makes the versioned unit(iri, version)whereiriis the stable identity:concept_versiontable: exactly oneis_current=truerow periri, enforced by a partial unique indexUNIQUE(iri) WHERE is_currentso the two-current corruption is structurally impossible, not just transaction-disciplined.rdf_triples.concept_version_id(one new FK column): a triple's owning concept-version is determined by its subject IRI; blank-node closures follow the IRI subject they hang off. This is what publish snapshots per version - v1 keeps its old field values, v2 gets a copy with the new ones - so the diff engine always has two versions to compare.scheme_membership: unversioned many-to-many(concept_iri, scheme_iri)forskos:inScheme.is_current, every subject-owned triple gets stamped, memberships seeded from existingskos:inScheme.m4,m5) tune the triple-uniqueness constraint so per-version snapshots coexist while unversioned triples still dedup on both Postgres and SQLite.Deferred to later stages (intentional)
release_manifest,manifest_pin) - not built here, deferred to P2, gated on a named version-pinning consumer. A "release" is a manifest over concept versions; the engine ships the per-concept spine first.concept_changesetbulk-approval gate (P3) - a single aggregate approval over a bulk RDF upload is parked (uploads land Draft and follow per-concept approval instead); the plumbing is present but off by default.EntityStatusis deferred to a future milestone.Core user journeys (CUJs) covered end to end
Notable fixes in this batch
fix(semantic-links)):GET /api/semantic-links/by-iri?iri=- IRIs containing//had%2F%2Fcollapsed by the Apps proxy on the path form, so linked assets never rendered on a concept though the link existed. FE callers switched to the query form.ontology_contextsas full IRIs while coverage rows key on the short source-context bucket, so the lookup always missed. Normalized through the same extractor.fix(rdf)): the concept-versioning uniqueness constraint (PostgresNULLS NOT DISTINCT) is a no-op on SQLite, breaking blank-node re-import idempotency. Added a partial unique indexWHERE concept_version_id IS NULL(migrationm5); no observable production change.Related issues and PRDs
Closes:
source_context) vs Group-by-Source (originating file,source_file) as independent axes, a single loaded file can declare multiple schemes, and imported ontologies withoutskos:inScheme(FIBO, SNOMED) stay fully browsable without fabricating membership.Advances (delivers a major slice; does NOT fully close):
Request...-style single-hop status transitions, and reference-gated deprecation.ontology_concept/ontology_collection, real Collection ownership (team/domain foreign keys, not cosmetic scope), a Collection detail page, a "My drafts" personal-scope workspace, and ConceptStatus alignment with the unified EntityStatus model (see Deferred to later stages).Related / deferred (not addressed here):
Testing
38 new test files (25 backend, 13 frontend). Backend integration tests cover the governed logic directly (review ping-pong, bypass gate, deprecate reference gate, diff engine, additive merge, upload preview, changeset gate, coverage counts). Two API-driven live-app E2E harnesses in
src/backend/src/tests/e2e/drive the CUJs above end to end over REST (concept_v2_api_e2e.py~50 checks;concept_lifecycle_e2e.py~33 checks covering B1 governed ping-pong, B2 ungoverned, and the full CUJ) - they SKIP without an app URL/token, so they are not headless-CI gates. Frontend branch coverage kept above the repo's 70% floor (70.04%).Migrations
Single alembic head (
m5_rdf_triple_null_version_uq, linear chain). The new migration is a partial unique index, additive and functionally redundant with the existing PG constraint for NULL-version rows.CI note
Includes
ci(build): cap hatchling <1.32- hatchling 1.32.0 rejects this package'sreadme = "../README.md"path, breaking every hatch-built job. Pre-existing on development; capped here to unblock. Follow-up: relocate the README intosrc/and lift the cap.Deploy
Verified live on a demo app: the Define/Explore/Enrich journey, concept lifecycle + versioning, coverage + Last-run, and the Deliver-to connection wiring all confirmed working post-deploy.
This pull request and its description were written by Isaac.