Skip to content

feat(metamodel): schema versioning core with per-type governance - #83

Merged
jimador merged 11 commits into
mainfrom
feat/metamodel-versioning
Sep 8, 2026
Merged

jimador merged 11 commits into
mainfrom
feat/metamodel-versioning

Conversation

@jimador

@jimador jimador commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

PR 1 of the metamodel train (refs #45). dice-metamodel is a new pure-JVM module holding the schema-versioning contracts: MetamodelVersion, a content-addressed stamp of a declared schema whose fingerprint covers entity types, labels, full property signatures and relationships, canonically ordered and length-prefixed, with a derived, unforgeable contentHash; GovernedTypeSelector for per-type opt-in — you version what you declare, and ungoverned exploratory types never perturb a governed stamp; DeclaredSchema/DeclaredSchemaSource; and the MetamodelVersionStore contract. Declared renames ride SchemaAliases (EXPERIMENTAL): a declaration states the names a type or property used to go by, so a later diff pairs a rename and stops reading it as remove+add. Aliases hash only when non-empty, so every pre-alias contentHash is byte-stable, pinned by golden-digest tests.

Changed in this review round:

  • Schema attribution is answered through run lineage. The DiceMetadataKeys.METAMODEL_VERSION metadata key is removed; the extraction run carries the declared schema's contentHash and per-proposition attribution resolves through PRODUCED_BY_RUN. docs/design/metamodel-versioning.md states the mechanism and that the coordinator may write a denormalised copy once it exists.
  • The doc now states context scope: versions carry no context dimension by design — one declared schema across all tenants, while drift reports and observation are per ContextId.
  • StampProvenance stayed out: nothing constructs a value outside deserialization, so the type returns with the first stamping caller that records it.
  • Naming and doc passes across the module.

Breaking changes: none on any released surface. The module is new; @JvmOverloads and separate overloads keep every previously shipped JVM descriptor byte-identical. The removed metadata key existed only inside this stack. The changed Kotlin data-class synthetics on PropertySignature are the stated boundary: Kotlin callers recompile.

Opt-in: declaring a DeclaredSchemaSource bean is the trigger for everything downstream; with no declared schema, nothing in this train activates.

The governance model at a glance. A type reaches the stamp only by being declared and selected:

flowchart LR
    A[DeclaredSchemaSource bean] --> C[DeclaredSchema]
    B[GovernedTypeSelector] --> C
    C --> D[MetamodelVersion]
    D --> E[entityTypeNames]
    D --> F[entityTypeLabels, full closure]
    D --> G[property signatures: name, kind, type, cardinality]
    D --> H[entityTypeAliases, former names]
    D --> I[relationships, attributed to declaring type]
    E --> J[contentHash: derived, unforgeable]
    F --> J
    G --> J
    H --> J
    I --> J
    K[ungoverned exploratory types] -. excluded .-> J
Loading

Stack: #84 (Drivine version store) and the later slices are stacked on this branch.

jimador added a commit that referenced this pull request Aug 31, 2026
Rewrite both specs in the repo's documentation voice. Delivery status
now reports the open PR train (#83-#88) per PR and corrects entries
that were marked planned after the implementing PRs opened.
Comment thread dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt Outdated
@embabel embabel deleted a comment from james-gsb Aug 31, 2026
@jimador
jimador force-pushed the feat/metamodel-versioning branch from fbbecf3 to db80548 Compare August 31, 2026 19:40
jimador added a commit that referenced this pull request Sep 2, 2026
Rewrite both specs in the repo's documentation voice. Delivery status
now reports the open PR train (#83-#88) per PR and corrects entries
that were marked planned after the implementing PRs opened.
@jimador
jimador requested a review from igordayen September 2, 2026 20:41
Comment thread docs/design/metamodel-versioning.md
Comment thread docs/design/metamodel-versioning.md
Comment thread CHANGELOG.md

@igordayen igordayen left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jimador - first batch of manual review comments, thank you

Comment thread dice-metamodel/pom.xml Outdated
@igordayen

Copy link
Copy Markdown

from claude - 1st batch:

──────────────────────────────────────┬──────────────────────────────────────────────────────────────┐
  │               Comment                │                            Valid?                            │
  ├──────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
  │ compareAliases readability rewrite   │ Yes — the indexed loop is more readable than index-based     │
  │                                      │ access                                                       │
  ├────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
  │ Avoid string concat → multiline         │ Yes — valid Kotlin style                                  │
  │ template                                │                                                           │
  ├─────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
  │ Fluent API from(dict).governedBy(...)   │ Debatable — adds intermediate builder objects for         │
  │                                         │ marginal readability gain                                 │
  ├─────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
  │ Test method names as English sentences  │ Yes — standard Kotlin/JUnit5 style                        │
  ├─────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
  │ is Version in name fully justified?     │ Reasonable question — MetamodelStamp or MetamodelSnapshot │
  │                                         │  might read better                                        │
  ├─────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
  │ contentHash is derived from instance?   │ Valid clarification question — answer is yes              │
  ├─────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
  │ too cryptic without comments (lines     │ Valid — hash computation code with no inline explanation  │
  │ 369/374)                                │                                                           │
  ├─────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
  │ Version not in                          │ Valid housekeeping question                               │
  │ embabel-parent-dependencies             │                                                           │
  └─────────────────────────────────────────┴───────────────────────────────────────────────────────────┘

@igordayen

Copy link
Copy Markdown

feedback from Claude:

Overall: Well-engineered. The design decisions are sound and thoroughly documented. A few issues worth
raising:


  1. equals / hasSameContentAs asymmetry — potential API trap

MetamodelVersion.equals includes schemaName. contentHash and hasSameContentAs exclude it by design. This
creates two notions of equality on the same object. The store's natural key is (schemaName, contentHash),
so equality-based data structures (e.g. HashSet) behave differently from a store lookup
by contentHash. Nothing is wrong, but a reader arriving from the store will be confused. equals KDoc
should explicitly say it includes schemaName while hasSameContentAs does not, with a cross-reference.


  1. Validation runs twice in from() — code duplication risk

requireNoTypeAliasReuse and requireNoAliasesOnDuplicateNames are called in both the from() factory and the
init block. The comment explains the attribution reason (fail at the call site). That's valid. But if the
validation logic ever changes, the two call sites can silently diverge. Consider a single validate(names,
aliases, properties) function called from both — so there's one body, two call sites.


  1. MetamodelVersionStore.findVersion default scans full history

The interface KDoc acknowledges this: "reads the whole history to answer a keyed question." But there's no
signal to implementors about the performance expectation. If history grows large (which it will in a
long-lived schema), the default is quietly O(n). Either mark it @deprecated to push implementors to
override, or add a stronger advisory in the KDoc noting this is unsuitable for production use without
override.


  1. SchemaAliases has manual equals/hashCode/toString — same pattern as DeclaredSchema

The same reason applies: java.util.Set.copyOf() in the constructor body means a data class wouldn't
include typeAliases / propertyAliases as components. Consistent with the rest, but worth noting it's a
deliberate pattern across the module, not an oversight. A module-level design note would prevent reviewers
raising this on every class.


  1. from() builds entityTypeProperties in a loop that's already a groupBy — two traversals

val entityTypeProperties = governedTypes
.groupBy { it.name }
.mapValues { (typeName, types) ->
types.flatMap { type -> type.properties.map { ... } }.toSet()
}

entityTypeLabels does a separate .groupBy { it.name }. Both traverse governedTypes and group by the same
key. Could be a single pass: groupBy once, then derive both labels and properties from the grouped map.
Minor, but this is a hot path if called frequently.


  1. toString() uses + concatenation across multiple lines

MetamodelVersion.toString() concatenates 7 fields with +. In Kotlin this should be a multiline string
template or buildString. Minor style issue but inconsistent with the otherwise idiomatic codebase.


  1. No concern about contentHash stability on JVM charset differences

The hash is computed as hashInput.toByteArray(Charsets.UTF_8) — explicitly UTF-8. Good. The KDoc says
"Stable across JVM restarts" which is true given deterministic ordering and UTF-8. The golden-digest test
pins this. No issue — just confirming this is correctly handled.


Summary: The core correctness invariants (length-prefixed hash encoding, immutable collections,
deduplication in init) are solid. Items 1 and 3 are the most actionable.

@igordayen

Copy link
Copy Markdown

i just realized that hashcode includes already schema name, therefore asked claude to validate:

why search by (name, hashcode), is not hashcode sufficient?

● Good catch. If contentHash already excludes schemaName and is a SHA-256 of the full structural content,
then contentHash alone is the natural key — it's globally unique by construction. The (schemaName,
contentHash) composite key in MetamodelVersionStore is redundant.

The only scenario where schemaName adds value as part of the key is if you want to scope history queries
per schema (latestVersion(schemaName), versionHistory(schemaName)). But that's a query filter, not a
uniqueness requirement — contentHash alone already guarantees uniqueness.

Worth raising: either drop schemaName from the store's natural key and keep it only as a query dimension,
or document explicitly why it's in the key. Right now the KDoc says (schemaName, contentHash) is the
natural key without justifying why contentHash alone isn't sufficient.

@igordayen igordayen left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jimador - looks good, thank you

@jimador

jimador commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

Went through all three. The table in the first batch is covered by the thread replies. On the second batch and the key question:

1. equals and hasSameContentAs. Real. Both KDocs now cross-reference: equals compares the schema name, hasSameContentAs and contentHash leave it out, so a Set<MetamodelVersion> keys the way the store does and hasSameContentAs asks whether two schemas have the same shape.

2. Validation in two places. Real. The two refusals now run through one requireDeclarable, called from from and from the constructor. One body, two call sites, and they cannot drift on which checks apply.

3. findVersion default. The Drivine store already overrides it with a keyed MATCH (#84). The KDoc now says the default is for the in-memory reference and test doubles, and that a durable backend should override it because a schema's history only grows. Not deprecating it: it is the correct contract and the reference implementation.

4. Manual equals and hashCode. Added a section to docs/design/metamodel-versioning.md, "Plain classes, not data classes", so the pattern reads as one module decision. The reason is that each class copies its arguments into an immutable collection in the body, which a data class cannot express.

5. Two traversals in from. Real. It groups by name once now and derives labels, properties and the governed name set from that one grouping.

6. toString concatenation. Keeping it, for the same reason as the thread replies: one-line messages stay one line, and embabel-agent builds its own the same way.

7. UTF-8. Agreed, and the golden-digest test is what pins it.

Is the schema name redundant in the key? No, and the store KDoc now says why. contentHash excludes the name on purpose, so two schemas with the same shape share a hash: a schema and its staging copy, or a schema forked under a new name. History is per schema, which is what latestVersion and versionHistory answer, so the same content has to be a separate record under each name. Keyed on the hash alone, one schema adopting a shape another had earlier would land on the other schema's record and pull that history into its own. The name is what keeps two schemas' histories apart.

New dice-metamodel module, first slice of the metamodel train:
MetamodelVersion content-hash stamping over governed types only
(GovernedTypeSelector — you version what you declare), property
signatures (name, kind, type, cardinality) in the fingerprint,
JVM-immutable value types, DeclaredSchema/DeclaredSchemaSource as the
opt-in seam, and the MetamodelVersionStore contract. Pure JVM — no
Spring, no database, no dice-core dependency.

Refs #45.
Rewrite the design doc, KDoc, and comments from this slice in the
repo's documentation voice. Comment and doc text only; no code change.
Declared renames enter as aliases on property signatures and entity types,
hashed only when non-empty so every existing stamp keeps its digest, guarded
against reuse collisions and duplicate-name ambiguity, and defensively copied
down to the alias set inside each signature. Stamps also carry optional
origin and last-stamped provenance, capped and never hashed. Prior art:
Iceberg field identity adapted to declaration-time aliases, Snowflake's
schema-evolution record adapted to two provenance pairs.
The key existed with nothing specifying who writes it or what the value
means. The KDoc now states the contract: the extraction persistence path
stamps the declared schema's content hash onto canonical proposition
metadata, a missing key marks pre-governance extraction, and the value
is opaque. A test proves the stamp round-trips through the in-memory
store and propositions are selectable by it. Production stamping lands
in a follow-up slice once the extraction-run stack merges.
The metadata key was a second mechanism for one fact: nothing wrote
dice.metamodel.version, while the extraction-run stack carries the
declared schema's content hash on the run record. The key and its
stamping contract go. A proposition's schema version is answered
through the run that produced it, the coordinator resolves the hash
from the host's DeclaredSchemaSource, and a denormalized per-proposition
copy stays a coordinator concern for a slice whose reads demand it. The
versioning doc also states the scope rule: versions are application-wide
while drift reports and observation are per context.
The new public types in dice-metamodel carry ApiStatus.Experimental, so
a consumer sees in the IDE what the changelog says in prose: the shape
may change before 1.0.
Manage org.jetbrains:annotations in dice-parent. Neither embabel BOM manages it, so each module pinned its own version and the two had already drifted apart. Simplify the alias comparator. Name the Java compatibility tests in sentences, which is what @DisplayName is for.
Cross-reference equals and hasSameContentAs, since one compares the schema name and the other does not. Run the two declaration refusals through one requireDeclarable so the constructor and from cannot drift. Group governed types by name once. Say why the store key carries the schema name: two schemas with the same shape share a hash, and history is per schema. Say when findVersion's default is enough and when to override it. A design note explains why three classes write their own equals.
@jimador
jimador force-pushed the feat/metamodel-versioning branch from 9d32ead to e423a0a Compare September 8, 2026 20:47
@jimador
jimador merged commit 9d083d8 into main Sep 8, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants