nebw #2: contract/catalog drift detection, Asset Review, semver adoption - #736
Draft
larsgeorge-db wants to merge 2 commits into
Draft
larsgeorge-db wants to merge 2 commits into
larsgeorge-db wants to merge 2 commits into
Conversation
Detect when an external catalog's schema has drifted from the governing
data contract and adopt an approved change as a new version or in place.
Deterministic only (no LLM): reuses ContractChangeAnalyzer for the diff and
suggested semver bump, build_odcs_from_db for the contract side, and
clone_for_new_version for versioning.
New:
- DataContractsManager.build_candidate_odcs_from_schema_info(): turns a live
connector SchemaInfo (columns + PK/FK from the UC import work) into the
aliased ODCS shape the analyzer consumes, so the current catalog state can
be diffed against the contract.
- DataContractsManager.replace_contract_schema(): swap a contract's schema
objects in place from ODCS schema data (used by in-place adoption).
- ContractDriftManager: analyze_contract_drift(), find_linked_asset_fqn(),
and adopt_drift(). The in-place-vs-new-version choice is severity-driven:
breaking changes (major) are forced to a new version; a bump override may
not weaken the required severity.
- Routes: POST /data-contracts/{id}/check-drift and /adopt-drift, resolving
live schema via the connection's connector.
Scheduled drift-check job + auto-creating an Asset Review from detected drift
land in follow-up commits on this branch. See docs/plans/nebw-contract-drift.md.
Session: claude -r 01d95399-c840-4b99-b76c-735490295c66
Add ContractDriftManager.create_drift_review(): on detected drift, open an
Asset Review for a steward with the diff summary and suggested bump, deduped
against any already-open review for the same asset FQN. Exposed via
POST /data-contracts/{id}/create-drift-review (resolves live schema through the
connection's connector, runs analysis, creates the review when drift exists).
Per decision, drift->review is app-side/on-demand in this PR; wiring the
existing data_contract_validation cluster job to auto-create reviews is a
documented follow-up (it runs against Lakebase with no access to app managers).
Tests cover review creation, open-review dedup, and the no-reviews-manager
path; the drift test module purges committed rows (clone/replace commit
internally) so it does not pollute other suites.
Session: claude -r 01d95399-c840-4b99-b76c-735490295c66
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nebw feature #2: detect when an external catalog's schema has drifted from the data contract that governs it, surface the diff through an Asset Review, and adopt an approved change as either a new contract version (semver-bumped) or an in-place update.
Deterministic, no LLM (per scoping decision). Reuses the existing
ContractChangeAnalyzer(diff + suggested bump),build_odcs_from_db, andclone_for_new_version.Base is nebw-bugfixes (stacked on PR #735); it will retarget/merge cleanly once #735 lands on
development.Changes
DataContractsManager.build_candidate_odcs_from_schema_info()— turns a live connectorSchemaInfo(columns + PK/FK from the UC-import work in nebw batch: custom UUID, RDF export fix, asset-type relationship CRUD, UC FK/PK import #735) into the aliased ODCS shape the analyzer consumes, so the current catalog state can be diffed against the contract.DataContractsManager.replace_contract_schema()— swap a contract's schema objects in place from ODCS schema data.ContractDriftManager—analyze_contract_drift,find_linked_asset_fqn,adopt_drift,create_drift_review.create_drift_reviewopens an Asset Review with the diff summary + suggested bump, deduped against any already-open review for the same asset.POST /data-contracts/{id}/check-drift,/adopt-drift,/create-drift-review(resolve live schema via the connection's connector).Testing
11 unit tests: detection (none/minor/major), adoption (new-version / in-place / breaking-reject / override-guard / no-drift), review creation + dedup. No cross-suite pollution (the drift module purges rows that the internal
commit()in clone/replace would otherwise leak). Broader contract/version/review suites stay green (91 passed).Follow-ups (documented, not in this PR)
data_contract_validationcluster job (which already detects drift) to auto-create reviews. It runs against Lakebase with no access to app managers, so needs a job-side reimplementation or app callback.Note
Push used
SKIP_SECRET_SCAN=1for the same pre-existing historical secrets already onmain/development(see #735); none are new in this branch.