diff --git a/CHANGELOG.md b/CHANGELOG.md index cd5fa989..6524fd6f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -45,3 +45,34 @@ and the consumer PRs that deliver it). metadata key is removed; lineage answers per-proposition attribution. **Compatibility: breaking.** The key is no longer available; code holding it must migrate to extraction-run queries. +- Drivine/Neo4j-backed `MetamodelVersionStore` in `dice-storage` + (`DrivineMetamodelVersionStore`): stamps persist as `(:MetamodelVersion)` nodes, + MERGEd on the natural key `(schemaName, contentHash)`, so a re-stamp updates in + place. `latestVersion`, `versionHistory` and `findVersion` all resolve in Cypher. + History is ordered by a persisted per-schema sequence, taken off a + `(:MetamodelSchemaCounter)` node in the same statement that creates the version; + `savedAt` and `savedAtEpochMillis` are informational, and nothing sorts on them. + An idempotent re-save leaves the counter and the sequence alone. Concurrent saves + of one version leave one node. Hosts must declare three uniqueness constraints: + `MetamodelVersion(schemaName, contentHash)`, `MetamodelSchemaCounter(schemaName)`, + and `MetamodelVersion(schemaName, sequence)`. + Declared aliases persist at both levels: the version-level `entityTypeAliases` map + as its own node property, and a property signature's former names as a fifth + `aliases` field inside the stored signature. Both are written only when they hold + something, so an alias-free stamp writes exactly the properties this mapper wrote + before aliases existed, and a node from that older build reads back as a stamp + declaring none. Aliases feed `contentHash`, and the mapper recomputes the hash from + the persisted fields, so a stamp that failed to store them would be unreadable for + good — pinned by an integration test that writes a row in the old four-field shape + through raw Cypher and reads it back, one that round-trips a stamp carrying both + alias kinds, and one that removes the stored alias map and asserts the integrity + check rejects the row. + `savedAt` and `savedAtEpochMillis` keep their existing behavior: set on create, + untouched by a re-save. `dice-metamodel` gains `InMemoryMetamodelVersionStore`, the + reference implementation of the store contract, promoted from a private class in + that module's own tests. `AbstractMetamodelVersionStoreContractTest` runs one suite + against both stores. + **Compatibility: additive.** New classes, and a new `dice-storage` → `dice-metamodel` + module dependency; no existing API touched. Stored nodes stay readable: every + property that existed before keeps its name, meaning, and encoding, and the two + new alias fields are absent when nothing declares them. diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/InMemoryMetamodelVersionStore.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/InMemoryMetamodelVersionStore.kt new file mode 100644 index 00000000..c758d728 --- /dev/null +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/InMemoryMetamodelVersionStore.kt @@ -0,0 +1,53 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +/** + * Reference [MetamodelVersionStore] that keeps stamps in a list. + * + * It is the executable statement of what the contract means, so a durable backend can be held to + * the same suite of tests. It also lets a host stamp and compare schemas before it has a database, + * which is most of what the first tier of versioning is for. + * + * Nothing here survives the JVM, and two instances know nothing about each other. + */ +class InMemoryMetamodelVersionStore : MetamodelVersionStore { + + private val saved = mutableListOf() + + /** + * Upsert on `(schemaName, contentHash)`. A stamp that is already there keeps its place in the + * write order, so re-saving an old version doesn't make it the latest; the incoming stamp + * replaces it in place. + * + * Everything runs under the list's own lock, so two threads saving the same version can't + * interleave the search for an existing stamp with the write that lands the new one. + */ + override fun saveVersion(version: MetamodelVersion) { + synchronized(saved) { + val existingIndex = saved.indexOfFirst { + it.schemaName == version.schemaName && it.contentHash == version.contentHash + } + if (existingIndex < 0) saved += version else saved[existingIndex] = version + } + } + + override fun latestVersion(schemaName: String): MetamodelVersion? = + synchronized(saved) { saved.lastOrNull { it.schemaName == schemaName } } + + override fun versionHistory(schemaName: String): List = + synchronized(saved) { saved.filter { it.schemaName == schemaName }.reversed() } +} diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt index 62e06587..6e2a0190 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt @@ -49,6 +49,17 @@ interface MetamodelVersionStore { * Save a version stamp, keyed on `(schemaName, contentHash)`. Saving the same version twice * leaves one stored version. * + * Everything about a stored stamp is content the key already determines, so a re-save + * overwrites it with an identical value. + * + * Whatever an implementation records as the moment of the save keeps its existing value on a + * re-save, along with the stamp's place in the write order: a re-saved old stamp does not + * become the latest. + * + * An implementation may fail a save with its backend's own concurrency exception when two + * writers race to save the same schema at once. Since the write is idempotent, the caller can + * simply retry it. + * * @param version The version to save. */ fun saveVersion(version: MetamodelVersion) diff --git a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionStoreTest.kt b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionStoreTest.kt index e83a0f88..15cf86ce 100644 --- a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionStoreTest.kt +++ b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionStoreTest.kt @@ -24,28 +24,13 @@ import org.junit.jupiter.api.Test * Covers the one piece of behaviour the contract itself ships: the default [findVersion], which a * backend is free to override with a keyed lookup. A store implementation gets its own tests * wherever it lives. + * + * The upsert rules [MetamodelVersionStore.saveVersion] states are checked by + * `AbstractMetamodelVersionStoreContractTest`, which runs the same suite against + * [InMemoryMetamodelVersionStore] and the graph-backed store. */ class MetamodelVersionStoreTest { - /** Minimal store honouring the contract: upsert on (schemaName, contentHash), newest first. */ - private class InMemoryVersionStore : MetamodelVersionStore { - - private val saved = mutableListOf() - - override fun saveVersion(version: MetamodelVersion) { - // Idempotent: re-saving an existing version keeps its original position in write order. - if (saved.none { it.schemaName == version.schemaName && it.contentHash == version.contentHash }) { - saved.add(version) - } - } - - override fun latestVersion(schemaName: String): MetamodelVersion? = - versionHistory(schemaName).firstOrNull() - - override fun versionHistory(schemaName: String): List = - saved.filter { it.schemaName == schemaName }.reversed() - } - private fun version(schemaName: String, vararg typeNames: String): MetamodelVersion = MetamodelVersion.from( DataDictionary.fromDomainTypes(schemaName, typeNames.map { DynamicType(name = it) }), @@ -53,7 +38,7 @@ class MetamodelVersionStoreTest { @Test fun `findVersion returns the stamp with that hash`() { - val store = InMemoryVersionStore() + val store = InMemoryMetamodelVersionStore() val first = version("app", "Person") val second = version("app", "Person", "Company") store.saveVersion(first) @@ -67,7 +52,7 @@ class MetamodelVersionStoreTest { fun `findVersion is scoped to the schema name`() { // Two schemas can hold structurally identical versions, because the hash excludes the // name, so the lookup has to match on both halves of the key. - val store = InMemoryVersionStore() + val store = InMemoryMetamodelVersionStore() val mine = version("mine", "Person") store.saveVersion(mine) @@ -77,7 +62,7 @@ class MetamodelVersionStoreTest { @Test fun `findVersion returns null for an unknown hash`() { - val store = InMemoryVersionStore() + val store = InMemoryMetamodelVersionStore() store.saveVersion(version("app", "Person")) assertNull(store.findVersion("app", "not-a-hash")) @@ -85,7 +70,7 @@ class MetamodelVersionStoreTest { @Test fun `re-saving a version leaves one record, not two`() { - val store = InMemoryVersionStore() + val store = InMemoryMetamodelVersionStore() val v = version("app", "Person") store.saveVersion(v) store.saveVersion(v) @@ -96,7 +81,7 @@ class MetamodelVersionStoreTest { @Test fun `an empty store has no latest version and an empty history`() { - val store = InMemoryVersionStore() + val store = InMemoryMetamodelVersionStore() assertNull(store.latestVersion("app")) assertEquals(emptyList(), store.versionHistory("app")) diff --git a/dice-storage/pom.xml b/dice-storage/pom.xml index 46bc43e0..c461dc10 100644 --- a/dice-storage/pom.xml +++ b/dice-storage/pom.xml @@ -31,6 +31,12 @@ dice + + + com.embabel.dice + dice-metamodel + + com.embabel.agent diff --git a/dice-storage/src/main/kotlin/com/embabel/dice/storage/DrivineMetamodelVersionStore.kt b/dice-storage/src/main/kotlin/com/embabel/dice/storage/DrivineMetamodelVersionStore.kt new file mode 100644 index 00000000..02f7105e --- /dev/null +++ b/dice-storage/src/main/kotlin/com/embabel/dice/storage/DrivineMetamodelVersionStore.kt @@ -0,0 +1,212 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.storage + +import com.embabel.dice.metamodel.MetamodelVersion +import com.embabel.dice.metamodel.MetamodelVersionStore +import org.drivine.manager.PersistenceManager +import org.drivine.query.QuerySpecification +import org.slf4j.LoggerFactory +import org.springframework.transaction.annotation.Transactional +import java.time.Clock + +/** + * Drivine/Neo4j implementation of [MetamodelVersionStore]. Every schema stamp is a + * `(:MetamodelVersion)` node. + * + * The write MERGEs on the natural key `(schemaName, contentHash)`, so a retry or a re-stamp of an + * unchanged schema updates the node that's already there. That is race-free only under a uniqueness + * constraint on the same pair of properties: without one, concurrent MERGEs all miss, all take the + * CREATE branch, and history fills with copies of one version. + * + * Every statement is parameterized; nothing user-derived is interpolated into Cypher. Ordering and + * the keyed lookup both run in the database. + * + * Ordered reads sort on a per-schema counter. "Most recent" in the contract means logical write + * order, which a wall clock can't express: two saves can land in the same millisecond, and an NTP + * correction or a failover to a differently-skewed node can make the clock run backwards between + * them. Each schema owns a `(:MetamodelSchemaCounter)` node, and a version takes the next value off + * it when its node is first created. `savedAt` and `savedAtEpochMillis` are informational; nothing + * sorts on them. + * + * The counter is bumped in the same statement, and so the same transaction, as the MERGE that + * creates the version, so a version node always carries its place in the order. A re-save of an + * existing version leaves the counter and the sequence alone, which keeps the write idempotent and + * holds an old stamp at its original position. The in-memory reference implementation behaves the + * same way. + * + * Three uniqueness constraints are required, and the host declares them in a `SchemaCatalog` bean + * (the module's `TestApplication` shows the shape): + * - `UniquenessConstraintSpec("MetamodelVersion", listOf("schemaName", "contentHash"))` makes the + * version MERGE race-free, as above. + * - `UniquenessConstraintSpec("MetamodelSchemaCounter", "schemaName")` makes the counter MERGE + * race-free, so one schema can only have one counter handing out numbers. + * - `UniquenessConstraintSpec("MetamodelVersion", listOf("schemaName", "sequence"))` makes two + * versions sharing a position in the order unstorable, so a lost counter update fails with a + * constraint violation the caller can retry. + * + * **Retrying a failed save.** If the counter's read-modify-write ever does lose an update, the + * second writer fails with a uniqueness-constraint violation on `(schemaName, sequence)`. + * [saveVersion] just lets that exception propagate; it doesn't retry internally, because the + * failure has already ended the surrounding Neo4j transaction, and a retry needs a new + * transaction, which only the caller can open. That's safe to do: the write is an idempotent + * upsert, so retrying a failed save never produces a duplicate or a wrong result. The drift check + * re-stamps its schema on every pass anyway, so for that caller the next pass already is the + * retry. + * + * @param persistenceManager Drivine's handle on the `neo` datasource. + * @param clock supplies the instant a version is stamped as saved at. Injectable so a test can pin + * the instants of two saves. + */ +@Transactional +class DrivineMetamodelVersionStore( + private val persistenceManager: PersistenceManager, + private val clock: Clock = Clock.systemUTC(), +) : MetamodelVersionStore { + + private val logger = LoggerFactory.getLogger(DrivineMetamodelVersionStore::class.java) + + private companion object { + + /** + * Upsert the version node and, on first insert, give it the next number off its schema's + * counter. One statement, so one transaction: a version node always carries its place in + * the write order. + * + * `WITH n WHERE n.sequence IS NULL` separates the two halves. A re-save filters the row + * away, so the counter stays put and the existing sequence is kept; only the content is + * refreshed. + * + * `entityTypeAliases` binds null when the version declares no former names. Setting a + * property to null removes it, so an alias-free stamp leaves a node with no such property, + * which is what a writer from before aliases existed left. + * + * `SET c.lockedBy = $contentHash` writes a property nobody reads. It takes the exclusive + * lock on the counter before `SET c.sequence = coalesce(c.sequence, 0) + 1` reads it. That + * increment is a read-modify-write, whose textbook failure is two concurrent saves both + * reading 5, both writing 6, and two versions claiming one position; the lock write is the + * documented Neo4j idiom for avoiding it. This store's own tests at 12 and at 48 concurrent + * savers could not tell the locked and unlocked statements apart, so Neo4j appears to + * serialise the increment here anyway, and the line is cheap insurance. + * + * Correctness rests on the uniqueness constraint on `(schemaName, sequence)`, which makes a + * duplicate position impossible to store. If the increment ever did lose an update (a Neo4j + * version with different locking, a cluster, contention beyond what has been tried) the + * second writer fails with a constraint violation and the caller retries. + */ + private val SAVE_VERSION = """ + MERGE (n:MetamodelVersion {schemaName: ${'$'}schemaName, contentHash: ${'$'}contentHash}) + ON CREATE SET n.savedAt = ${'$'}savedAt, + n.savedAtEpochMillis = ${'$'}savedAtEpochMillis + SET n.entityTypeNames = ${'$'}entityTypeNames, + n.entityTypeLabels = ${'$'}entityTypeLabels, + n.entityTypeProperties = ${'$'}entityTypeProperties, + n.relationshipNames = ${'$'}relationshipNames, + n.entityTypeAliases = ${'$'}entityTypeAliases + WITH n + WHERE n.sequence IS NULL + MERGE (c:MetamodelSchemaCounter {schemaName: ${'$'}schemaName}) + SET c.lockedBy = ${'$'}contentHash + WITH n, c + SET c.sequence = coalesce(c.sequence, 0) + 1 + WITH n, c + SET n.sequence = c.sequence + """.trimIndent() + + /** + * Every stamp for one schema, newest first. + * + * The sort key is `coalesce(n.sequence, -1)`: Neo4j sorts null as the largest value, so a + * node that somehow has no sequence would sort to the front of a DESC order and be handed + * back as the newest. A node with no sequence never took a place in the write order, so it + * belongs last. + */ + private val VERSIONS_NEWEST_FIRST = """ + MATCH (n:MetamodelVersion {schemaName: ${'$'}schemaName}) + RETURN n + ORDER BY coalesce(n.sequence, -1) DESC + """.trimIndent() + } + + override fun saveVersion(version: MetamodelVersion) { + logger.debug( + "Saving metamodel version schemaName={} contentHash={}", + version.schemaName, + version.contentHash.take(8), + ) + persistenceManager.execute( + QuerySpecification.withStatement(SAVE_VERSION) + .bind(MetamodelVersionRowMapper.bindMap(version, clock.instant())), + ) + } + + @Transactional(readOnly = true) + override fun latestVersion(schemaName: String): MetamodelVersion? = + readVersions(VERSIONS_NEWEST_FIRST, mapOf("schemaName" to schemaName)).firstOrNull() + + @Transactional(readOnly = true) + override fun versionHistory(schemaName: String): List = + readVersions(VERSIONS_NEWEST_FIRST, mapOf("schemaName" to schemaName)) + + /** + * Overridden to resolve a recorded hash with a single keyed `MATCH`; the interface default + * reads the schema's whole history and filters it in memory. Both halves of the natural key are + * in the pattern, which is what the uniqueness constraint indexes. + */ + @Transactional(readOnly = true) + override fun findVersion(schemaName: String, contentHash: String): MetamodelVersion? = readVersions( + """ + MATCH (n:MetamodelVersion {schemaName: ${'$'}schemaName, contentHash: ${'$'}contentHash}) + RETURN n + LIMIT 1 + """.trimIndent(), + mapOf("schemaName" to schemaName, "contentHash" to contentHash), + ).firstOrNull() + + /** + * Run one of the version queries and turn its rows into stamps, dropping any row that won't + * deserialize. + * + * A single corrupt or tampered node shouldn't take down a whole history read, so the row is + * logged at warn and skipped. [MetamodelVersionRowMapper] throws on bad data so that this can + * happen; the warning names the missing property or the failed integrity check, which is what + * an operator needs to go find the node. A row that isn't even a `Map` is logged and skipped + * the same way, naming its runtime class, so a count mismatch against what was expected still + * shows up in the log. + * + * [latestVersion] deliberately keeps `LIMIT 1` out of the Cypher. If the newest node were the + * corrupt one, a database-side limit would read it, drop it, and answer "this schema has no + * versions", hiding the good history behind it and disagreeing with [versionHistory], whose + * first element is meant to be the same stamp. It sorts in the database and takes the first + * survivor here. + */ + private fun readVersions(statement: String, bindings: Map): List { + @Suppress("UNCHECKED_CAST") + val spec = QuerySpecification.withStatement(statement).bind(bindings) as QuerySpecification + return persistenceManager.query(spec).mapNotNull { row -> + if (row !is Map<*, *>) { + logger.warn( + "Skipping MetamodelVersion row: expected a Map, got {}", + row?.javaClass?.name ?: "null", + ) + return@mapNotNull null + } + runCatching { MetamodelVersionRowMapper.fromRow(row) } + .onFailure { logger.warn("Skipping unreadable MetamodelVersion row: {}", it.message) } + .getOrNull() + } + } +} diff --git a/dice-storage/src/main/kotlin/com/embabel/dice/storage/MetamodelRowMappers.kt b/dice-storage/src/main/kotlin/com/embabel/dice/storage/MetamodelRowMappers.kt new file mode 100644 index 00000000..001e6de9 --- /dev/null +++ b/dice-storage/src/main/kotlin/com/embabel/dice/storage/MetamodelRowMappers.kt @@ -0,0 +1,276 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.storage + +import com.embabel.agent.core.Cardinality +import com.embabel.dice.metamodel.MetamodelVersion +import com.embabel.dice.metamodel.PropertySignature +import com.fasterxml.jackson.core.type.TypeReference +import com.fasterxml.jackson.databind.ObjectMapper +import java.time.Instant + +private val objectMapper = ObjectMapper() + +/** + * Translate metamodel versions to and from the property maps the Neo4j graph store reads and + * writes. + * + * Neo4j properties are scalars and flat arrays, while a version's content is lists, a map of label + * sets, and a map of property signature sets, so all four structural fields are serialized to JSON + * strings. JSON also handles names containing pipes, tabs, newlines and quotes, which these names + * routinely do: they come out of LLM extraction. + * + * The save instant is informational, and is written twice. `savedAt` is the ISO-8601 string, which + * is what you want when you're looking at a node and wondering when it landed. `savedAtEpochMillis` + * is the same instant as a number, for filtering or grouping by time in an ad-hoc query; the string + * is no use for that, since `Instant.toString()` drops the fraction entirely at a whole second and + * `'Z'` sorts above `'.'`, making `"…T00:00:00Z"` compare greater than `"…T00:00:00.500Z"`. + * + * Neither field orders the history. The `sequence` property does that; see + * [DrivineMetamodelVersionStore] for why a clock can't express write order. Nothing here writes or + * reads `sequence`: Cypher assigns it off a per-schema counter, and it is storage bookkeeping, so + * it stays out of the strict round-trip below. + * + * Reads are strict. A property this mapper wrote must be present when it is read again; a node + * missing one is corrupt, so the accessor throws and the store's surrounding guard skips the row + * with a warning. An empty string is corrupt too: an empty collection is written as `[]` or + * `{}`, so `""` never comes from this mapper, and it fails the read like any other bad JSON. + * + * Two things are optional, and absent means "no former names were declared": the version-level + * `entityTypeAliases` property, and the `aliases` field inside a stored property signature. Writing + * them only when they hold something means an alias-free stamp stores exactly the properties this + * mapper stored before either existed, and a node written by that older build reads back here as a + * stamp declaring neither. Aliases feed the content hash, so a stamp carrying them and failing to + * store them would fail its own integrity check on the way back in and be unreadable for good. + */ +object MetamodelVersionRowMapper { + + /** + * Bind values for a write. The natural key is (schemaName, contentHash). + * + * [savedAt] is a parameter, so this stays a pure function of its arguments and a test can pin + * the instant a version was stored at. + * + * `entityTypeAliases` binds `null` when the version declares no former names. A Cypher `SET` of + * `null` leaves no property behind, which is the encoding the read side expects and the shape an + * older writer left. + */ + fun bindMap(version: MetamodelVersion, savedAt: Instant): Map = mapOf( + "schemaName" to version.schemaName, + "contentHash" to version.contentHash, + "entityTypeNames" to serializeList(version.entityTypeNames), + "entityTypeLabels" to serializeMapOfLabelSets(version.entityTypeLabels), + "entityTypeProperties" to serializeMapOfSignatureSets(version.entityTypeProperties), + "relationshipNames" to serializeList(version.relationshipNames), + "entityTypeAliases" to serializeAliasMap(version.entityTypeAliases), + "savedAt" to savedAt.toString(), + "savedAtEpochMillis" to savedAt.toEpochMilli(), + ) + + /** + * Rebuild a [MetamodelVersion] from a returned node's property map, and check its integrity + * on the way. + * + * A version's content hash is derived from its structural fields, so the reconstructed object + * computes its own hash and the `contentHash` property on the node acts as a checksum. + * Recomputing it and getting a different answer means the node was written by an older hash + * format, hand-edited, or corrupted, so this throws and the caller skips it. The stored hash is + * also half the natural key, so a mismatch also means a re-save of the same content lands on a + * different node. + * + * Aliases are part of that derivation, at both levels, so a node that dropped either alias + * field fails here rather than reading back as an alias-free stamp with the wrong hash. + */ + fun fromRow(row: Map<*, *>): MetamodelVersion { + val storedHash = row.str("contentHash") + val version = MetamodelVersion( + schemaName = row.str("schemaName"), + entityTypeNames = deserializeList(row.str("entityTypeNames")), + entityTypeLabels = deserializeMapOfLabelSets(row.str("entityTypeLabels")), + entityTypeProperties = deserializeMapOfSignatureSets(row.str("entityTypeProperties")), + relationshipNames = deserializeList(row.str("relationshipNames")), + entityTypeAliases = deserializeAliasMap(row.optionalStr("entityTypeAliases")), + ) + require(version.contentHash == storedHash) { + "MetamodelVersion '${version.schemaName}' fails its integrity check: stored contentHash " + + "$storedHash, but the persisted structural fields hash to ${version.contentHash}" + } + return version + } +} + +// Serialization helpers: JSON, for escape-safe round-trip encoding. + +private fun serializeList(items: List): String = + objectMapper.writeValueAsString(items) + +private fun deserializeList(serialized: String): List = + objectMapper.readValue( + serialized, + objectMapper.typeFactory.constructCollectionType(List::class.java, String::class.java) + ) + +/** + * Serialize the per-type label sets as `{"Person": ["Agent", "Entity"], ...}`. + * + * Sets have no order, so they're written sorted. Nothing reads the order back, but a deterministic + * encoding means re-saving the same version writes byte-identical JSON, which keeps an idempotent + * MERGE a no-op and makes a stored node diffable by hand. + */ +private fun serializeMapOfLabelSets(map: Map>): String = + objectMapper.writeValueAsString(map.toSortedMap().mapValues { (_, labels) -> labels.sorted() }) + +/** + * Serialize the former names each entity type goes by, in the same shape as the label sets, and + * write nothing at all when no type declares any. + * + * The empty case has to leave no property behind. This map feeds the content hash, and a stamp that + * declares no aliases hashes to the same digest it did before aliases existed, so its node must + * also look the way the older writer left it — otherwise the two spellings of one schema are two + * different-looking nodes on the same key. + */ +private fun serializeAliasMap(aliases: Map>): String? = + if (aliases.isEmpty()) null else serializeMapOfLabelSets(aliases) + +/** Inverse of [serializeAliasMap]; an absent property means no type declared a former name. */ +private fun deserializeAliasMap(serialized: String?): Map> = + if (serialized == null) emptyMap() else deserializeMapOfLabelSets(serialized) + +/** Inverse of [serializeMapOfLabelSets]. */ +private fun deserializeMapOfLabelSets(serialized: String): Map> { + val mapOfLists = objectMapper.readValue( + serialized, + object : TypeReference>>() {}, + ) + return mapOfLists.mapValues { (_, labels) -> labels.toSet() } +} + +/** + * Serialize the per-type property signatures as a JSON object of arrays of four-field objects: + * + * ```json + * {"Person": [{"name": "age", "kind": "VALUE", "type": "integer", "cardinality": "ONE"}]} + * ``` + * + * The fields are written out one by one. This shape on disk is a persisted format that feeds the + * version's own content hash on the way back in, so it has to stay put when someone renames a + * Kotlin property or when `jackson-module-kotlin` leaves the classpath, which is what handing the + * object to Jackson's bean serializer would risk. + * + * A property that declares former names gets a fifth field, `"aliases": ["oldName", ...]`, sorted + * for the same determinism as everywhere else. A property with none gets exactly the four fields + * above, so a signature that declares no aliases encodes the bytes this mapper wrote before aliases + * existed and a stored four-field signature still reads. + * + * Enums are stored by `name`. An ordinal would re-point at a different constant the moment someone + * inserts a value into [Cardinality] or [PropertySignature.Kind]. + */ +private fun serializeMapOfSignatureSets(map: Map>): String = + objectMapper.writeValueAsString( + map.toSortedMap().mapValues { (_, signatures) -> + signatures.sorted().map { signature -> + // A LinkedHashMap, so the keys land in the JSON in this order and the encoding is + // fully determined by the content. + linkedMapOf( + "name" to signature.name, + "kind" to signature.kind.name, + "type" to signature.type, + "cardinality" to signature.cardinality.name, + ).apply { + if (signature.aliases.isNotEmpty()) put("aliases", signature.aliases.sorted()) + } + } + } + ) + +/** + * Inverse of [serializeMapOfSignatureSets], and strict about it: a signature object missing a + * field, or naming an enum constant this build doesn't have, throws. Patching it up with a default + * would change the structural content, and the version's integrity check would then reject the + * whole row with a message about a hash mismatch that hides the real problem. + */ +private fun deserializeMapOfSignatureSets(serialized: String): Map> { + val mapOfLists = objectMapper.readValue( + serialized, + object : TypeReference>>() {}, + ) + return mapOfLists.mapValues { (typeName, encoded) -> + encoded.map { element -> + val fields = element as? Map<*, *> + require(fields != null) { + "entityTypeProperties for '$typeName' holds ${element?.javaClass?.simpleName ?: "null"} " + + "where a property signature object was expected" + } + PropertySignature( + name = fields.signatureField(typeName, "name"), + kind = enumConstant(fields.signatureField(typeName, "kind"), typeName, "kind"), + type = fields.signatureField(typeName, "type"), + cardinality = enumConstant(fields.signatureField(typeName, "cardinality"), typeName, "cardinality"), + aliases = fields.signatureAliases(typeName), + ) + }.toSet() + } +} + +/** Read one field of a stored property signature, blowing up by name if it isn't there. */ +private fun Map<*, *>.signatureField(typeName: String, field: String): String = + requireNotNull(this[field]) { + "a property signature for '$typeName' is missing its '$field' field" + }.toString() + +/** + * Read a stored signature's former names. An absent `aliases` field means none were declared, which + * is every signature written before aliases existed. Anything present but not a list of names + * throws: aliases are part of the signature and feed the content hash, so quietly dropping a + * malformed one would surface later as a hash mismatch instead. + */ +private fun Map<*, *>.signatureAliases(typeName: String): Set { + val encoded = this["aliases"] ?: return emptySet() + val names = encoded as? List<*> + require(names != null) { + "a property signature for '$typeName' has an 'aliases' field holding a " + + "${encoded.javaClass.simpleName} where a list of former names was expected" + } + return names.map { name -> + requireNotNull(name) { + "a property signature for '$typeName' has a null entry in its 'aliases' field" + }.toString() + }.toSet() +} + +/** Turn a stored enum constant name back into the constant, naming what failed if it's unknown. */ +private inline fun > enumConstant(stored: String, typeName: String, field: String): E = + requireNotNull(enumValues().firstOrNull { it.name == stored }) { + "a property signature for '$typeName' has '$field' = '$stored', which is not a known " + + "${E::class.simpleName}. The node was written by a different version of the schema model" + } + +/** + * Read a property that must be there, and blow up if it isn't. + * + * Returning `""` for an absent property would let a node missing `schemaName` come back as a + * real-looking version named `""`, indistinguishable from data, and the caller's "skip the + * unreadable row" guard would never fire for the most likely kind of corruption there is. Throwing + * is what gives that guard something to catch. + */ +private fun Map<*, *>.str(key: String): String = + requireNotNull(this[key]) { "required property '$key' is missing from the stored node" }.toString() + +/** + * Read a property that may legitimately not be there, where absent means the stamp declared nothing + * to put in it. Only the alias map is read this way; everything else goes through [str]. + */ +private fun Map<*, *>.optionalStr(key: String): String? = this[key]?.toString() diff --git a/dice-storage/src/test/kotlin/com/embabel/dice/storage/AbstractMetamodelVersionStoreContractTest.kt b/dice-storage/src/test/kotlin/com/embabel/dice/storage/AbstractMetamodelVersionStoreContractTest.kt new file mode 100644 index 00000000..f865dcc7 --- /dev/null +++ b/dice-storage/src/test/kotlin/com/embabel/dice/storage/AbstractMetamodelVersionStoreContractTest.kt @@ -0,0 +1,140 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.storage + +import com.embabel.dice.metamodel.MetamodelVersion +import com.embabel.dice.metamodel.MetamodelVersionStore +import org.junit.jupiter.api.Assertions.assertEquals +import org.junit.jupiter.api.Assertions.assertNull +import org.junit.jupiter.api.Test + +/** + * Cross-backend contract for [MetamodelVersionStore]: the upsert, history ordering, keyed lookup, + * and schema isolation. Each subclass supplies a store and inherits the whole suite, so a backend + * that disagrees with the in-memory reference fails at authoring time. + * + * The rules here matter because the drift check re-stamps its schema on every pass. A store that + * treated each of those re-stamps as a new record would fill the history with copies of one version, + * and one that moved a re-stamped version to the front would report the wrong stamp as the latest. + */ +abstract class AbstractMetamodelVersionStoreContractTest { + + /** A store holding nothing for the schema names below. */ + protected abstract fun store(): MetamodelVersionStore + + /** A stamp of one entity type. */ + private fun version( + schemaName: String, + typeName: String = "Person", + ): MetamodelVersion = MetamodelVersion( + schemaName = schemaName, + entityTypeNames = listOf(typeName), + entityTypeLabels = emptyMap(), + entityTypeProperties = emptyMap(), + relationshipNames = emptyList(), + entityTypeAliases = emptyMap(), + ) + + // ---- the upsert ---- + + @Test + fun `re-saving a version leaves one record`() { + val store = store() + val schemaName = "contract-idempotent" + val stamp = version(schemaName) + + store.saveVersion(stamp) + store.saveVersion(stamp) + + assertEquals(listOf(stamp), store.versionHistory(schemaName)) + assertEquals(stamp, store.latestVersion(schemaName)) + } + + @Test + fun `a re-save leaves the stamp where it was in the history`() { + // The write lands on an existing key, so it has to behave like any other re-save: content + // refreshed, position in the write order untouched. + val store = store() + val schemaName = "contract-order" + val first = version(schemaName, "First") + val second = version(schemaName, "Second") + store.saveVersion(first) + store.saveVersion(second) + + store.saveVersion(version(schemaName, "First")) + + assertEquals( + listOf("Second", "First"), + store.versionHistory(schemaName).map { it.entityTypeNames.single() }, + "a re-save must not make an old version the latest", + ) + assertEquals(first, store.findVersion(schemaName, first.contentHash)) + } + + // ---- keyed lookup ---- + + @Test + fun `findVersion returns null for a hash the schema has never stored`() { + val store = store() + val schemaName = "contract-find-miss" + val stamp = version(schemaName) + store.saveVersion(stamp) + + assertNull(store.findVersion(schemaName, "not-a-real-hash")) + assertNull(store.findVersion("contract-find-miss-other-schema", stamp.contentHash)) + } + + // ---- ordering ---- + + @Test + fun `versionHistory is newest first`() { + val store = store() + val schemaName = "contract-newest-first" + + store.saveVersion(version(schemaName, "First")) + store.saveVersion(version(schemaName, "Second")) + store.saveVersion(version(schemaName, "Third")) + + assertEquals( + listOf("Third", "Second", "First"), + store.versionHistory(schemaName).map { it.entityTypeNames.single() }, + ) + assertEquals("Third", store.latestVersion(schemaName)?.entityTypeNames?.single()) + } + + // ---- schema isolation ---- + + @Test + fun `one schema's writes are invisible to another`() { + val store = store() + val schemaA = "contract-isolation-a" + val schemaB = "contract-isolation-b" + + store.saveVersion(version(schemaA)) + + assertEquals(emptyList(), store.versionHistory(schemaB)) + assertNull(store.latestVersion(schemaB)) + + store.saveVersion(version(schemaB)) + + assertEquals(1, store.versionHistory(schemaA).size, "schema B's save must not touch schema A's history") + } + + @Test + fun `latestVersion is null for a schema with no versions`() { + assertNull(store().latestVersion("contract-never-saved")) + } +} diff --git a/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineCollectorTraceStoreIntegrationTest.kt b/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineCollectorTraceStoreIntegrationTest.kt index bdcc8fda..1c6ffa26 100644 --- a/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineCollectorTraceStoreIntegrationTest.kt +++ b/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineCollectorTraceStoreIntegrationTest.kt @@ -40,14 +40,25 @@ import org.junit.jupiter.api.Assertions.assertTrue import org.junit.jupiter.api.Test import org.springframework.beans.factory.annotation.Autowired import org.springframework.boot.test.context.SpringBootTest +import org.springframework.test.context.DynamicPropertyRegistry +import org.springframework.test.context.DynamicPropertySource /** - * Integration tests for [DrivineCollectorTraceStore] against a Neo4j testcontainer (provided by - * Drivine's test support). Each test starts from an empty graph via [cleanUp]. + * Integration tests for [DrivineCollectorTraceStore] against a Neo4j testcontainer. Each test + * starts from an empty graph via [cleanUp]. + * + * Uses the shared [Neo4jTestContainer]; see that class for why Drivine's built-in testcontainer + * is bypassed. */ @SpringBootTest(classes = [TestApplication::class]) class DrivineCollectorTraceStoreIntegrationTest { + companion object { + @JvmStatic + @DynamicPropertySource + fun neo4jProperties(registry: DynamicPropertyRegistry) = Neo4jTestContainer.registerProperties(registry) + } + @Autowired private lateinit var traceStore: DrivineCollectorTraceStore diff --git a/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineGraphQueryParityIntegrationTest.kt b/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineGraphQueryParityIntegrationTest.kt index 59a82aaf..423df71c 100644 --- a/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineGraphQueryParityIntegrationTest.kt +++ b/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineGraphQueryParityIntegrationTest.kt @@ -40,6 +40,8 @@ import org.junit.jupiter.api.Test import org.drivine.manager.PersistenceManager import org.springframework.beans.factory.annotation.Autowired import org.springframework.boot.test.context.SpringBootTest +import org.springframework.test.context.DynamicPropertyRegistry +import org.springframework.test.context.DynamicPropertySource import java.time.Instant /** @@ -52,10 +54,19 @@ import java.time.Instant * exists (which propositions land on a `via` or on a shortest path when parallel edges / ties are * present), we assert each returned edge is *valid* rather than object-identical — both engines are * free to pick a different but correct edge. + * + * Uses the shared [Neo4jTestContainer]; see that class for why Drivine's built-in testcontainer + * is bypassed. */ @SpringBootTest(classes = [TestApplication::class]) class DrivineGraphQueryParityIntegrationTest { + companion object { + @JvmStatic + @DynamicPropertySource + fun neo4jProperties(registry: DynamicPropertyRegistry) = Neo4jTestContainer.registerProperties(registry) + } + @Autowired private lateinit var repository: DrivinePropositionRepository diff --git a/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineLineageRecordStoreIntegrationTest.kt b/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineLineageRecordStoreIntegrationTest.kt index 9906a7eb..290a8701 100644 --- a/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineLineageRecordStoreIntegrationTest.kt +++ b/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineLineageRecordStoreIntegrationTest.kt @@ -31,15 +31,26 @@ import org.junit.jupiter.api.Assertions.assertTrue import org.junit.jupiter.api.Test import org.springframework.beans.factory.annotation.Autowired import org.springframework.boot.test.context.SpringBootTest +import org.springframework.test.context.DynamicPropertyRegistry +import org.springframework.test.context.DynamicPropertySource import java.time.Instant /** * Integration tests for the durable lineage stores against a Neo4j testcontainer (provided by * Drivine's test support). Each test starts from an empty graph via [cleanUp]. + * + * Uses the shared [Neo4jTestContainer]; see that class for why Drivine's built-in testcontainer + * is bypassed. */ @SpringBootTest(classes = [TestApplication::class]) class DrivineLineageRecordStoreIntegrationTest { + companion object { + @JvmStatic + @DynamicPropertySource + fun neo4jProperties(registry: DynamicPropertyRegistry) = Neo4jTestContainer.registerProperties(registry) + } + @Autowired private lateinit var projectionStore: DrivineProjectionRecordStore diff --git a/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineMetamodelVersionStoreContractIntegrationTest.kt b/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineMetamodelVersionStoreContractIntegrationTest.kt new file mode 100644 index 00000000..a693d4f3 --- /dev/null +++ b/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineMetamodelVersionStoreContractIntegrationTest.kt @@ -0,0 +1,59 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.storage + +import com.embabel.dice.metamodel.MetamodelVersionStore +import org.drivine.manager.PersistenceManager +import org.drivine.query.QuerySpecification +import org.junit.jupiter.api.AfterEach +import org.springframework.beans.factory.annotation.Autowired +import org.springframework.boot.test.context.SpringBootTest +import org.springframework.test.context.DynamicPropertyRegistry +import org.springframework.test.context.DynamicPropertySource + +/** + * Runs the [AbstractMetamodelVersionStoreContractTest] suite against the Neo4j-backed + * [DrivineMetamodelVersionStore] (testcontainer). This is the half that catches the graph backend + * disagreeing with the in-memory reference on the upsert rules, which is easy to do: they live in a + * Cypher `MERGE` there and in a list index here. + * + * Uses the shared [Neo4jTestContainer]; see that class for why Drivine's built-in testcontainer + * is bypassed. + */ +@SpringBootTest(classes = [TestApplication::class]) +class DrivineMetamodelVersionStoreContractIntegrationTest : AbstractMetamodelVersionStoreContractTest() { + + companion object { + @JvmStatic + @DynamicPropertySource + fun neo4jProperties(registry: DynamicPropertyRegistry) = Neo4jTestContainer.registerProperties(registry) + } + + @Autowired + private lateinit var graphStore: DrivineMetamodelVersionStore + + @Autowired + private lateinit var persistenceManager: PersistenceManager + + override fun store(): MetamodelVersionStore = graphStore + + @AfterEach + fun cleanUp() { + listOf("MetamodelVersion", "MetamodelSchemaCounter").forEach { label -> + persistenceManager.execute(QuerySpecification.withStatement("MATCH (n:$label) DETACH DELETE n")) + } + } +} diff --git a/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineMetamodelVersionStoreIntegrationTest.kt b/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineMetamodelVersionStoreIntegrationTest.kt new file mode 100644 index 00000000..f52d54e2 --- /dev/null +++ b/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivineMetamodelVersionStoreIntegrationTest.kt @@ -0,0 +1,854 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.storage + +import ch.qos.logback.classic.Level +import ch.qos.logback.classic.Logger +import ch.qos.logback.classic.spi.ILoggingEvent +import ch.qos.logback.core.read.ListAppender +import com.embabel.agent.core.Cardinality +import com.embabel.dice.metamodel.MetamodelVersion +import com.embabel.dice.metamodel.PropertySignature +import com.embabel.dice.metamodel.PropertySignature.Kind +import org.drivine.manager.PersistenceManager +import org.drivine.query.QuerySpecification +import org.junit.jupiter.api.AfterEach +import org.junit.jupiter.api.Assertions.assertEquals +import org.junit.jupiter.api.Assertions.assertNotEquals +import org.junit.jupiter.api.Assertions.assertNull +import org.junit.jupiter.api.Assertions.assertTrue +import org.junit.jupiter.api.Test +import org.slf4j.LoggerFactory +import org.springframework.beans.factory.annotation.Autowired +import org.springframework.boot.test.context.SpringBootTest +import org.springframework.test.context.DynamicPropertyRegistry +import org.springframework.test.context.DynamicPropertySource +import java.time.Instant +import java.util.concurrent.CountDownLatch +import java.util.concurrent.Executors +import java.util.concurrent.TimeUnit +import java.util.concurrent.atomic.AtomicInteger + +/** + * Integration tests for [DrivineMetamodelVersionStore] against a Neo4j testcontainer. Each test + * starts from an empty graph via [cleanUp]. + * + * Uses the shared [Neo4jTestContainer]; see that class for why Drivine's built-in testcontainer + * is bypassed. + * + * None of the `MetamodelVersion`s built here carries a hand-written content hash: the hash is + * derived from the structural fields. Tests that need two distinct versions of one schema give them + * genuinely different content. + */ +@SpringBootTest(classes = [TestApplication::class]) +class DrivineMetamodelVersionStoreIntegrationTest { + + companion object { + @JvmStatic + @DynamicPropertySource + fun neo4jProperties(registry: DynamicPropertyRegistry) = Neo4jTestContainer.registerProperties(registry) + } + + @Autowired + private lateinit var store: DrivineMetamodelVersionStore + + @Autowired + private lateinit var persistenceManager: PersistenceManager + + @Autowired + private lateinit var clock: PinnableClock + + @AfterEach + fun cleanUp() { + clock.unpin() + listOf("MetamodelVersion", "MetamodelSchemaCounter").forEach { label -> + persistenceManager.execute(QuerySpecification.withStatement("MATCH (n:$label) DETACH DELETE n")) + } + } + + // ---- CRUD ---- + + @Test + fun `a version persists and reads back every field`() { + val version = MetamodelVersion( + schemaName = "test-schema", + entityTypeNames = listOf("Person", "Company", "Location"), + entityTypeLabels = mapOf( + "Person" to setOf("Agent", "Entity"), + "Company" to setOf("Organization", "Entity"), + ), + entityTypeProperties = mapOf( + "Person" to setOf( + PropertySignature("name", Kind.VALUE, "string", Cardinality.ONE), + PropertySignature("age", Kind.VALUE, "integer", Cardinality.OPTIONAL), + ), + "Company" to setOf( + PropertySignature("name", Kind.VALUE, "string", Cardinality.ONE), + PropertySignature("employs", Kind.REFERENCE, "Person", Cardinality.SET), + ), + ), + relationshipNames = listOf("WORKS_FOR", "LOCATED_IN"), + ) + + store.saveVersion(version) + + val reloaded = store.latestVersion("test-schema") + assertEquals(version, reloaded) + assertEquals(version.contentHash, reloaded!!.contentHash) + } + + @Test + fun `version history returns empty for unknown schema`() { + assertEquals(emptyList(), store.versionHistory("unknown-schema")) + assertNull(store.latestVersion("unknown-schema")) + } + + @Test + fun `each schema sees only its own versions`() { + val schemaA = MetamodelVersion("schema-a", listOf("TypeA"), emptyMap(), emptyMap(), emptyList()) + val schemaB = MetamodelVersion("schema-b", listOf("TypeB"), emptyMap(), emptyMap(), emptyList()) + + store.saveVersion(schemaA) + store.saveVersion(schemaB) + + assertEquals(schemaA, store.latestVersion("schema-a")) + assertEquals(schemaB, store.latestVersion("schema-b")) + assertEquals(listOf(schemaA), store.versionHistory("schema-a")) + } + + // ---- Property signatures round-trip ---- + + @Test + fun `a property signature round-trips its kind, type and cardinality, not just its name`() { + // entityTypeProperties holds signatures, so turning a single `age` string into a list of + // integers registers as the schema change it is. If the store dropped kind/type/cardinality + // on the way to disk, the reloaded stamp would hash differently from the saved one, and the + // mapper's integrity check would reject its own write. + val everyShape = setOf( + PropertySignature("optionalString", Kind.VALUE, "string", Cardinality.OPTIONAL), + PropertySignature("oneInteger", Kind.VALUE, "integer", Cardinality.ONE), + PropertySignature("listOfDates", Kind.VALUE, "date", Cardinality.LIST), + PropertySignature("setOfCompanies", Kind.REFERENCE, "Company", Cardinality.SET), + PropertySignature("mystery", Kind.UNKNOWN, "", Cardinality.ONE), + ) + val version = MetamodelVersion( + schemaName = "signature-schema", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf("Person" to everyShape), + relationshipNames = emptyList(), + ) + + store.saveVersion(version) + + val reloaded = store.latestVersion("signature-schema")!! + assertEquals(everyShape, reloaded.entityTypeProperties["Person"]) + assertEquals(version.contentHash, reloaded.contentHash) + } + + @Test + fun `two versions differing only in one property's cardinality are two stored versions`() { + // Same type name, same property name: an encoding that stored only property names would + // miss this change. It has to survive as two nodes with two hashes. + val schemaName = "cardinality-change-schema" + fun withCardinality(cardinality: Cardinality) = MetamodelVersion( + schemaName = schemaName, + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf("Person" to setOf(PropertySignature("nickname", Kind.VALUE, "string", cardinality))), + relationshipNames = emptyList(), + ) + val one = withCardinality(Cardinality.ONE) + val many = withCardinality(Cardinality.LIST) + assertNotEquals(one.contentHash, many.contentHash, "precondition: the two stamps must differ") + + store.saveVersion(one) + store.saveVersion(many) + + assertEquals(setOf(one, many), store.versionHistory(schemaName).toSet()) + assertEquals(Cardinality.ONE, store.findVersion(schemaName, one.contentHash)!!.entityTypeProperties["Person"]!!.single().cardinality) + assertEquals(Cardinality.LIST, store.findVersion(schemaName, many.contentHash)!!.entityTypeProperties["Person"]!!.single().cardinality) + } + + // ---- findVersion ---- + + @Test + fun `findVersion resolves a recorded hash back to the stamp it named`() { + val schemaName = "find-schema" + val v1 = MetamodelVersion(schemaName, listOf("Type1"), emptyMap(), emptyMap(), emptyList()) + val v2 = MetamodelVersion(schemaName, listOf("Type2"), emptyMap(), emptyMap(), emptyList()) + store.saveVersion(v1) + store.saveVersion(v2) + + assertEquals(v1, store.findVersion(schemaName, v1.contentHash)) + assertEquals(v2, store.findVersion(schemaName, v2.contentHash)) + } + + @Test + fun `findVersion is null for an unknown hash, and keyed on the schema name too`() { + val v = MetamodelVersion("find-null-schema", listOf("Type1"), emptyMap(), emptyMap(), emptyList()) + store.saveVersion(v) + + assertNull(store.findVersion("find-null-schema", "no-such-hash")) + // The hash is real, but it belongs to another schema: the natural key is the pair. + assertNull(store.findVersion("some-other-schema", v.contentHash)) + } + + @Test + fun `findVersion applies the same integrity check as the history read`() { + val schemaName = "find-tampered-schema" + val version = MetamodelVersion(schemaName, listOf("Original"), emptyMap(), emptyMap(), emptyList()) + store.saveVersion(version) + tamperWithEntityTypeNames(schemaName, """["SwappedInBehindTheHash"]""") + + val (found, logged) = capturingStoreWarnings { store.findVersion(schemaName, version.contentHash) } + + assertNull(found, "a keyed lookup must not hand back a node that fails its own checksum") + assertTrue(logged.any { it.contains("fails its integrity check") }, "warnings were: $logged") + } + + // ---- Natural-key idempotency ---- + + @Test + fun `saving the same version twice leaves one node, not two`() { + val version = MetamodelVersion( + schemaName = "idempotent-schema", + entityTypeNames = listOf("TypeA"), + entityTypeLabels = mapOf("TypeA" to setOf("LabelA")), + entityTypeProperties = mapOf("TypeA" to setOf(PropertySignature("prop", Kind.VALUE, "string", Cardinality.ONE))), + relationshipNames = listOf("REL"), + ) + + store.saveVersion(version) + store.saveVersion(version) + + val history = store.versionHistory("idempotent-schema") + assertEquals(1, history.size) + assertEquals(version, history.single()) + } + + @Test + fun `re-saving an old version neither bumps the counter nor moves it in the history`() { + val schemaName = "history-order-schema" + val v1 = MetamodelVersion(schemaName, listOf("Type1"), emptyMap(), emptyMap(), emptyList()) + val v2 = MetamodelVersion(schemaName, listOf("Type2"), emptyMap(), emptyMap(), emptyList()) + + clock.pin(Instant.parse("2026-01-01T00:00:00Z")) + store.saveVersion(v1) + clock.pin(Instant.parse("2026-01-02T00:00:00Z")) + store.saveVersion(v2) + assertEquals(v2, store.latestVersion(schemaName)) + assertEquals(1L, storedSequence(schemaName, v1)) + assertEquals(2L, storedSequence(schemaName, v2)) + + // Re-stamp the old one much later. The idempotent path refreshes v1's content and leaves + // its sequence and the counter alone; bumping the counter would make the next new version + // skip a number. + clock.pin(Instant.parse("2026-06-01T00:00:00Z")) + store.saveVersion(v1) + + assertEquals(v2, store.latestVersion(schemaName), "re-saving v1 must not make it the latest") + assertEquals(listOf(v2, v1), store.versionHistory(schemaName)) + assertEquals(1L, storedSequence(schemaName, v1), "v1 must keep the position it has always had") + assertEquals(2L, counterValue(schemaName), "an idempotent re-save must not consume a sequence number") + } + + @Test + fun `many threads saving the identical version leave exactly one node`() { + // The write is a MERGE, race-free only under a uniqueness constraint on the key it merges + // on; see TestApplication.metamodelSchema. Without one, concurrent MERGEs all miss, all take + // the CREATE branch, and the history fills with duplicates of one version. + val threads = 12 + val version = MetamodelVersion( + schemaName = "concurrent-schema", + entityTypeNames = listOf("Contended"), + entityTypeLabels = mapOf("Contended" to setOf("LabelC")), + entityTypeProperties = mapOf("Contended" to setOf(PropertySignature("prop", Kind.VALUE, "string", Cardinality.ONE))), + relationshipNames = listOf("REL"), + ) + + val startTogether = CountDownLatch(1) + val succeeded = AtomicInteger() + val failures = mutableListOf() + val pool = Executors.newFixedThreadPool(threads) + try { + repeat(threads) { + pool.submit { + startTogether.await() + // A loser in a MERGE race can surface a constraint violation or a lock timeout, + // which a caller retries. The surviving node count is what's asserted below. + runCatching { store.saveVersion(version) } + .onSuccess { succeeded.incrementAndGet() } + .onFailure { t -> synchronized(failures) { failures += t } } + } + } + startTogether.countDown() + pool.shutdown() + assertTrue(pool.awaitTermination(60, TimeUnit.SECONDS), "concurrent saves did not finish in time") + } finally { + pool.shutdownNow() + } + + assertTrue(succeeded.get() > 0, "every concurrent save failed: ${failures.firstOrNull()}") + val history = store.versionHistory("concurrent-schema") + assertEquals( + 1, + history.size, + "$threads concurrent saves of one version must leave one node, not ${history.size} " + + "(${succeeded.get()} succeeded, ${failures.size} failed)", + ) + assertEquals(version, history.single()) + // The sequence is assigned on create only, in the same transaction as the MERGE, so the + // losing threads matched the existing node and took no number. + assertEquals(1L, storedSequence("concurrent-schema", version), "the one node must hold the first sequence") + assertEquals(1L, counterValue("concurrent-schema"), "only the creating save may consume a number") + } + + @Test + fun `concurrent saves of distinct versions each get their own place in the order`() { + // The lost-update test. Every thread creates a different version of one schema, so all of + // them hit the counter at the same moment. If the increment lost an update, two versions + // would claim one position, and the order between that pair would be arbitrary. + val schemaName = "concurrent-distinct-schema" + val threads = 12 + val versions = (1..threads).map { + MetamodelVersion(schemaName, listOf("Type$it"), emptyMap(), emptyMap(), emptyList()) + } + + val startTogether = CountDownLatch(1) + val failures = mutableListOf() + val pool = Executors.newFixedThreadPool(threads) + try { + versions.forEach { version -> + pool.submit { + startTogether.await() + runCatching { store.saveVersion(version) } + .onFailure { t -> synchronized(failures) { failures += t } } + } + } + startTogether.countDown() + pool.shutdown() + assertTrue(pool.awaitTermination(60, TimeUnit.SECONDS), "concurrent saves did not finish in time") + } finally { + pool.shutdownNow() + } + assertTrue(failures.isEmpty(), "distinct versions must not contend for the same node: ${failures.firstOrNull()}") + + val sequences = versions.map { storedSequence(schemaName, it) } + assertEquals( + (1L..threads.toLong()).toSet(), + sequences.toSet(), + "each version must hold its own sequence; got $sequences", + ) + // The history the store reports has to hold all of them, with no shared positions. + assertEquals(threads, store.versionHistory(schemaName).size) + assertEquals(threads.toLong(), counterValue(schemaName)) + } + + @Test + fun `two versions of one schema cannot be stored at the same position`() { + // The safety net under the sequence. The counter increment appears to serialise on its own + // (removing the lock from the save statement doesn't fail the test above, even at four times + // the contention), so its atomicity is an observation rather than a proof. The guarantee + // rests on this constraint: whatever the counter does, the database will not hold two + // versions of one schema at one place in the write order, so a lost update becomes a + // retryable failure. + val schemaName = "position-constraint-schema" + val first = MetamodelVersion(schemaName, listOf("First"), emptyMap(), emptyMap(), emptyList()) + store.saveVersion(first) + assertEquals(1L, storedSequence(schemaName, first)) + + val collision = runCatching { + persistenceManager.execute( + QuerySpecification.withStatement( + """ + CREATE (n:MetamodelVersion { + schemaName: ${'$'}schemaName, contentHash: 'a-different-hash', sequence: 1 + }) + """.trimIndent(), + ).bind(mapOf("schemaName" to schemaName)), + ) + } + + assertTrue( + collision.isFailure, + "the database must refuse a second version at position 1; a lost counter update has to be loud", + ) + assertEquals(listOf(first), store.versionHistory(schemaName), "and the history is untouched") + } + + // ---- Chronological ordering ---- + + @Test + fun `version history is newest-first`() { + val schemaName = "ordering-schema" + val v1 = MetamodelVersion(schemaName, listOf("Type1"), emptyMap(), emptyMap(), emptyList()) + val v2 = MetamodelVersion(schemaName, listOf("Type2"), emptyMap(), emptyMap(), emptyList()) + + store.saveVersion(v1) + store.saveVersion(v2) + + assertEquals(listOf(v2, v1), store.versionHistory(schemaName)) + assertEquals(v2, store.latestVersion(schemaName)) + } + + @Test + fun `two versions saved in the very same millisecond still order by write order`() { + // No sleep, and the clock is pinned to one instant for both saves, so every timestamp on + // both nodes is byte-identical. No clock, at any precision, can separate the two; the + // counter can. Back-to-back saves land in the same millisecond routinely. + val schemaName = "same-millisecond-schema" + val first = MetamodelVersion(schemaName, listOf("First"), emptyMap(), emptyMap(), emptyList()) + val second = MetamodelVersion(schemaName, listOf("Second"), emptyMap(), emptyMap(), emptyList()) + + clock.pin(Instant.parse("2026-01-01T00:00:00Z")) + store.saveVersion(first) + store.saveVersion(second) + + assertEquals( + listOf(second, first), + store.versionHistory(schemaName), + "identical timestamps must not make the order arbitrary", + ) + assertEquals(second, store.latestVersion(schemaName)) + assertEquals(listOf(1L, 2L), listOf(storedSequence(schemaName, first), storedSequence(schemaName, second))) + } + + @Test + fun `write order survives a clock that runs backwards`() { + // An NTP correction, or a failover to a node with a different skew, can move the wall clock + // backwards between two saves. Ordering on any timestamp then reports the older stamp as the + // newest. The sequence is monotonic whatever the clock does. + val schemaName = "clock-skew-schema" + val earlier = MetamodelVersion(schemaName, listOf("WrittenFirst"), emptyMap(), emptyMap(), emptyList()) + val later = MetamodelVersion(schemaName, listOf("WrittenSecond"), emptyMap(), emptyMap(), emptyList()) + + clock.pin(Instant.parse("2026-01-01T12:00:00Z")) + store.saveVersion(earlier) + clock.pin(Instant.parse("2026-01-01T11:00:00Z")) // an hour backwards + store.saveVersion(later) + + assertEquals(later, store.latestVersion(schemaName), "the last write must be the latest, whatever the clock says") + assertEquals(listOf(later, earlier), store.versionHistory(schemaName)) + } + + // ---- Corrupt rows are skipped ---- + + @Test + fun `a version node missing a required property is skipped and warned about, not read as a blank version`() { + // Written straight through Cypher, so the node looks the way a partially-failed write or a + // hand-edit would leave it: one required property absent. The absent property is + // `entityTypeNames`, because every read MATCHes on schemaName, and a node without that is + // filtered out by the query before the mapper sees it. + val schemaName = "corrupt-row-schema" + val good = MetamodelVersion(schemaName, listOf("Sound"), emptyMap(), emptyMap(), emptyList()) + store.saveVersion(good) + // Spelled out property by property so the defect is visible here: every property the mapper + // writes except `entityTypeNames`. + val brokenSavedAt = Instant.parse("2026-01-01T00:00:00Z") + persistenceManager.execute( + QuerySpecification.withStatement( + """ + CREATE (broken:MetamodelVersion { + schemaName: ${'$'}schemaName, + contentHash: ${'$'}contentHash, + entityTypeLabels: '{}', + entityTypeProperties: '{}', + relationshipNames: '[]', + savedAt: ${'$'}savedAt, + savedAtEpochMillis: ${'$'}savedAtEpochMillis + }) + """.trimIndent(), + ).bind( + mapOf( + "schemaName" to schemaName, + // Distinct from the good node's, so the (schemaName, contentHash) uniqueness + // constraint lets both nodes exist. + "contentHash" to "a-different-hash-so-the-natural-key-does-not-collide", + "savedAt" to brokenSavedAt.toString(), + "savedAtEpochMillis" to brokenSavedAt.toEpochMilli(), + ), + ), + ) + assertEquals(2, rawNodeCount(), "the corrupt node must really be in the graph") + + val (history, logged) = capturingStoreWarnings { store.versionHistory(schemaName) } + + assertEquals(listOf(good), history, "the readable version survives; the corrupt one is dropped") + assertTrue( + logged.any { it.contains("Skipping unreadable MetamodelVersion row") && it.contains("entityTypeNames") }, + "the skip must be warned about and name the missing property; warnings were: $logged", + ) + } + + @Test + fun `a stored property signature missing a field is skipped and warned about by name`() { + // The signature encoding is a persisted format of its own, so a node can be structurally + // fine and still hold a half-written signature (an older writer, a hand-edit). Guessing a + // default cardinality would change the content and surface later as a hash mismatch, so the + // mapper names the missing field. + val schemaName = "corrupt-signature-schema" + val version = MetamodelVersion( + schemaName = schemaName, + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf("Person" to setOf(PropertySignature("age", Kind.VALUE, "integer", Cardinality.ONE))), + relationshipNames = emptyList(), + ) + store.saveVersion(version) + persistenceManager.execute( + QuerySpecification.withStatement( + """ + MATCH (n:MetamodelVersion {schemaName: ${'$'}schemaName}) + SET n.entityTypeProperties = ${'$'}halfWritten + """.trimIndent(), + ).bind( + mapOf( + "schemaName" to schemaName, + "halfWritten" to """{"Person":[{"name":"age","kind":"VALUE","type":"integer"}]}""", + ), + ), + ) + + val (history, logged) = capturingStoreWarnings { store.versionHistory(schemaName) } + + assertEquals(emptyList(), history) + assertTrue( + logged.any { it.contains("missing its 'cardinality' field") }, + "the warning must name the missing signature field; warnings were: $logged", + ) + } + + @Test + fun `a version node whose stored hash disagrees with its stored fields is skipped and warned about`() { + // The content hash is derived from the structural fields, so the copy on the node is a + // checksum. Disagreement means the node was written by an older hash format or tampered + // with. + val schemaName = "tampered-hash-schema" + val version = MetamodelVersion(schemaName, listOf("Original"), emptyMap(), emptyMap(), emptyList()) + store.saveVersion(version) + tamperWithEntityTypeNames(schemaName, """["SwappedInBehindTheHash"]""") + + val (history, logged) = capturingStoreWarnings { store.versionHistory(schemaName) } + + assertEquals(emptyList(), history) + assertTrue(logged.any { it.contains("fails its integrity check") }, "warnings were: $logged") + assertNull(store.latestVersion(schemaName), "and it must not come back as the latest version either") + } + + @Test + fun `a corrupt newest node hides only itself, not the readable version behind it`() { + // latestVersion sorts in Cypher and takes the first readable row, keeping LIMIT 1 out of the + // query. With the limit in the query, a corrupt newest node would make the store answer "no + // versions at all" while versionHistory still returned the older one: two reads disagreeing + // about the same graph. + val schemaName = "corrupt-head-schema" + val readable = MetamodelVersion(schemaName, listOf("Readable"), emptyMap(), emptyMap(), emptyList()) + val doomed = MetamodelVersion(schemaName, listOf("Doomed"), emptyMap(), emptyMap(), emptyList()) + + clock.pin(Instant.parse("2026-01-01T00:00:00Z")) + store.saveVersion(readable) + clock.pin(Instant.parse("2026-01-02T00:00:00Z")) + store.saveVersion(doomed) + // Tamper with the newer node only, keyed on its own hash. + persistenceManager.execute( + QuerySpecification.withStatement( + """ + MATCH (n:MetamodelVersion {schemaName: ${'$'}schemaName, contentHash: ${'$'}contentHash}) + SET n.entityTypeNames = '["NotWhatTheHashSays"]' + """.trimIndent(), + ).bind(mapOf("schemaName" to schemaName, "contentHash" to doomed.contentHash)), + ) + + assertEquals(readable, store.latestVersion(schemaName)) + assertEquals(listOf(readable), store.versionHistory(schemaName)) + } + + // ---- Adversarial serialization: names carrying delimiter characters ---- + + @Test + fun `names containing delimiter characters survive the round-trip intact`() { + // These are the characters that would break a delimiter-joined encoding; this one is JSON. + // Entity type names, labels, property names, property types and relationship names all go + // through it, so all five carry one here. + listOf("|" to "pipe", "\t" to "tab", "\n" to "newline", "\"" to "quote", "\\" to "backslash") + .forEach { (delimiter, label) -> + val schemaName = "delimiter-$label" + val typeName = "Type${delimiter}WithIt" + val version = MetamodelVersion( + schemaName = schemaName, + entityTypeNames = listOf(typeName, "Normal"), + entityTypeLabels = mapOf(typeName to setOf("Label${delimiter}1", "Label2")), + entityTypeProperties = mapOf( + typeName to setOf( + PropertySignature("prop${delimiter}1", Kind.VALUE, "string${delimiter}ish", Cardinality.ONE), + ), + ), + relationshipNames = listOf("REL${delimiter}WITH${delimiter}IT"), + ) + + store.saveVersion(version) + + assertEquals(version, store.latestVersion(schemaName), "'$label' did not survive the round-trip") + } + } + + // ---- Aliases, and rows written before they existed ---- + + @Test + fun `a version carrying both kinds of alias round-trips with its integrity check passing`() { + val schemaName = "alias-round-trip-schema" + val version = MetamodelVersion( + schemaName = schemaName, + entityTypeNames = listOf("Organisation"), + entityTypeLabels = mapOf("Organisation" to setOf("Entity")), + entityTypeProperties = mapOf( + "Organisation" to setOf( + PropertySignature("legalName", Kind.VALUE, "string", Cardinality.ONE, setOf("companyName", "name")), + PropertySignature("staff", Kind.REFERENCE, "Person", Cardinality.SET), + ), + ), + relationshipNames = listOf("Organisation-[EMPLOYS]->Person"), + entityTypeAliases = mapOf("Organisation" to setOf("Company", "Firm")), + ) + + store.saveVersion(version) + + // The mapper recomputes the hash from the persisted fields and throws on a mismatch, so a + // reloaded stamp at all is already proof that both alias kinds reached the node. + val reloaded = store.latestVersion(schemaName)!! + assertEquals(version, reloaded) + assertEquals(version.contentHash, reloaded.contentHash) + assertEquals(mapOf("Organisation" to setOf("Company", "Firm")), reloaded.entityTypeAliases) + assertEquals( + setOf("companyName", "name"), + reloaded.entityTypeProperties["Organisation"]!!.single { it.name == "legalName" }.aliases, + ) + assertEquals( + emptySet(), + reloaded.entityTypeProperties["Organisation"]!!.single { it.name == "staff" }.aliases, + ) + } + + @Test + fun `a type-aliased version whose alias map is missing from the node cannot be read back`() { + // Why the map has to be stored: it feeds the content hash, so a writer that dropped it + // would produce nodes that fail their own checksum and are unreadable for good. Removing + // the property is exactly what that bug would leave behind. + val schemaName = "alias-map-dropped-schema" + val version = MetamodelVersion( + schemaName = schemaName, + entityTypeNames = listOf("Organisation"), + entityTypeLabels = emptyMap(), + entityTypeProperties = emptyMap(), + relationshipNames = emptyList(), + entityTypeAliases = mapOf("Organisation" to setOf("Company")), + ) + store.saveVersion(version) + assertEquals(version, store.latestVersion(schemaName), "precondition: it reads back while the map is stored") + + persistenceManager.execute( + QuerySpecification.withStatement( + """ + MATCH (n:MetamodelVersion {schemaName: ${'$'}schemaName}) + REMOVE n.entityTypeAliases + """.trimIndent(), + ).bind(mapOf("schemaName" to schemaName)), + ) + + val (history, logged) = capturingStoreWarnings { store.versionHistory(schemaName) } + + assertEquals(emptyList(), history) + assertTrue(logged.any { it.contains("fails its integrity check") }, "warnings were: $logged") + } + + @Test + fun `an alias-free stamp leaves no entityTypeAliases property on the node`() { + // The other half of the same rule. A stamp declaring no former names has to store the + // properties the writer stored before aliases existed, so the two spellings of one schema + // are one node rather than two shapes on one key. + val schemaName = "alias-free-schema" + val version = MetamodelVersion(schemaName, listOf("Person"), emptyMap(), emptyMap(), emptyList()) + + store.saveVersion(version) + + assertEquals("", storedProperty(schemaName, version, StoredProperty.ENTITY_TYPE_ALIASES)) + assertEquals(emptyMap>(), store.latestVersion(schemaName)!!.entityTypeAliases) + } + + @Test + fun `a node written in the old four-field shape reads back through the new mapper`() { + // A stamp saved before aliases existed: property signatures with exactly four fields, and + // no entityTypeAliases. Written straight through Cypher, so nothing in the current writer + // can quietly supply the missing properties. + val schemaName = "old-shape-schema" + val expected = MetamodelVersion( + schemaName = schemaName, + entityTypeNames = listOf("Person"), + entityTypeLabels = mapOf("Person" to setOf("Agent")), + entityTypeProperties = mapOf( + "Person" to setOf( + PropertySignature("age", Kind.VALUE, "integer", Cardinality.OPTIONAL), + PropertySignature("name", Kind.VALUE, "string", Cardinality.ONE), + ), + ), + relationshipNames = listOf("Person-[WORKS_FOR]->Company"), + ) + persistenceManager.execute( + QuerySpecification.withStatement( + """ + CREATE (n:MetamodelVersion { + schemaName: ${'$'}schemaName, + contentHash: ${'$'}contentHash, + entityTypeNames: '["Person"]', + entityTypeLabels: '{"Person":["Agent"]}', + entityTypeProperties: ${'$'}entityTypeProperties, + relationshipNames: '["Person-[WORKS_FOR]->Company"]', + savedAt: '2026-01-01T00:00:00Z', + savedAtEpochMillis: 1767225600000, + sequence: 1 + }) + """.trimIndent(), + ).bind( + mapOf( + "schemaName" to schemaName, + "contentHash" to expected.contentHash, + "entityTypeProperties" to + """{"Person":[{"name":"age","kind":"VALUE","type":"integer","cardinality":"OPTIONAL"},""" + + """{"name":"name","kind":"VALUE","type":"string","cardinality":"ONE"}]}""", + ), + ), + ) + + val reloaded = store.latestVersion(schemaName) + + assertEquals(expected, reloaded, "an old-shape node must still read, and pass its integrity check") + assertEquals(expected.contentHash, reloaded!!.contentHash) + assertEquals(emptyMap>(), reloaded.entityTypeAliases) + assertTrue( + reloaded.entityTypeProperties["Person"]!!.all { it.aliases.isEmpty() }, + "four-field signatures carry no former names", + ) + } + + @Test + fun `re-saving an old-shape node through the current writer leaves it in the old shape`() { + // The upgrade path: an application that boots against a graph written by an older build + // re-stamps its unchanged schema. The write lands on the existing node, and because the + // stamp declares no aliases, neither of the two new fields appears. + val schemaName = "old-shape-restamp-schema" + val version = MetamodelVersion( + schemaName = schemaName, + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf( + "Person" to setOf(PropertySignature("name", Kind.VALUE, "string", Cardinality.ONE)), + ), + relationshipNames = emptyList(), + ) + store.saveVersion(version) + + store.saveVersion(version) + + assertEquals(1, rawNodeCount()) + assertEquals("", storedProperty(schemaName, version, StoredProperty.ENTITY_TYPE_ALIASES)) + assertEquals( + """{"Person":[{"name":"name","kind":"VALUE","type":"string","cardinality":"ONE"}]}""", + storedProperty(schemaName, version, StoredProperty.ENTITY_TYPE_PROPERTIES), + "a signature with no former names keeps its four fields", + ) + } + + // ---- helpers ---- + + /** The stored properties a test can read back raw, each with the Cypher that returns it. */ + private enum class StoredProperty(val returnExpression: String) { + ENTITY_TYPE_ALIASES("coalesce(n.entityTypeAliases, '')"), + ENTITY_TYPE_PROPERTIES("n.entityTypeProperties"), + } + + /** + * Read one property straight off a version node, bypassing the mapper. `` stands for a + * property that isn't on the node, which is what a Cypher `SET` of null leaves behind and what + * these tests are checking for. + */ + private fun storedProperty(schemaName: String, version: MetamodelVersion, property: StoredProperty): String? = + persistenceManager.maybeGetOne( + QuerySpecification.withStatement( + """ + MATCH (n:MetamodelVersion {schemaName: ${'$'}schemaName, contentHash: ${'$'}contentHash}) + RETURN ${property.returnExpression} AS value + """.trimIndent(), + ).bind(mapOf("schemaName" to schemaName, "contentHash" to version.contentHash)) + .transform(String::class.java), + ) + + /** Rewrite a version node's serialized entity type names, leaving its stored hash untouched. */ + private fun tamperWithEntityTypeNames(schemaName: String, serializedNames: String) { + persistenceManager.execute( + QuerySpecification.withStatement( + """ + MATCH (n:MetamodelVersion {schemaName: ${'$'}schemaName}) + SET n.entityTypeNames = ${'$'}serializedNames + """.trimIndent(), + ).bind(mapOf("schemaName" to schemaName, "serializedNames" to serializedNames)), + ) + } + + /** + * The `sequence` a version node holds. Read straight out of the graph, since the sequence is + * storage bookkeeping and [MetamodelVersion] doesn't carry it. + */ + private fun storedSequence(schemaName: String, version: MetamodelVersion): Long? = + persistenceManager.maybeGetOne( + QuerySpecification.withStatement( + """ + MATCH (n:MetamodelVersion {schemaName: ${'$'}schemaName, contentHash: ${'$'}contentHash}) + RETURN n.sequence AS sequence + """.trimIndent(), + ).bind(mapOf("schemaName" to schemaName, "contentHash" to version.contentHash)) + .transform(Long::class.java), + ) + + /** How far the schema's counter has been advanced. */ + private fun counterValue(schemaName: String): Long? = persistenceManager.maybeGetOne( + QuerySpecification.withStatement( + "MATCH (c:MetamodelSchemaCounter {schemaName: ${'$'}schemaName}) RETURN c.sequence AS sequence", + ).bind(mapOf("schemaName" to schemaName)).transform(Long::class.java), + ) + + /** Count version nodes without going through the store's mapper. */ + private fun rawNodeCount(): Int = persistenceManager.maybeGetOne( + QuerySpecification.withStatement("MATCH (n:MetamodelVersion) RETURN count(n) AS c").transform(Long::class.java), + )?.toInt() ?: 0 + + /** + * Run [block] with a listener attached to the store's logger, and hand back its result along + * with every WARN message the store emitted. The tests assert on the warning text as well as the + * skip, because an operator needs the message to find the bad node. + */ + private fun capturingStoreWarnings(block: () -> T): Pair> { + val logger = LoggerFactory.getLogger(DrivineMetamodelVersionStore::class.java) as Logger + val appender = ListAppender().apply { start() } + logger.addAppender(appender) + return try { + block() to appender.list.filter { it.level == Level.WARN }.map { it.formattedMessage } + } finally { + logger.detachAppender(appender) + appender.stop() + } + } +} diff --git a/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivinePropositionStoreContractIntegrationTest.kt b/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivinePropositionStoreContractIntegrationTest.kt index ec2b17bf..eba396e5 100644 --- a/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivinePropositionStoreContractIntegrationTest.kt +++ b/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivinePropositionStoreContractIntegrationTest.kt @@ -21,15 +21,26 @@ import org.drivine.query.QuerySpecification import org.junit.jupiter.api.AfterEach import org.springframework.beans.factory.annotation.Autowired import org.springframework.boot.test.context.SpringBootTest +import org.springframework.test.context.DynamicPropertyRegistry +import org.springframework.test.context.DynamicPropertySource /** * Runs the [AbstractPropositionStoreContractTest] suite against the Neo4j-backed * [DrivinePropositionRepository] (testcontainer). This is the half that catches a graph backend * silently disagreeing with the in-memory contract — substitutability enforced, not assumed. + * + * Uses the shared [Neo4jTestContainer]; see that class for why Drivine's built-in testcontainer + * is bypassed. */ @SpringBootTest(classes = [TestApplication::class]) class DrivinePropositionStoreContractIntegrationTest : AbstractPropositionStoreContractTest() { + companion object { + @JvmStatic + @DynamicPropertySource + fun neo4jProperties(registry: DynamicPropertyRegistry) = Neo4jTestContainer.registerProperties(registry) + } + @Autowired private lateinit var repository: DrivinePropositionRepository diff --git a/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivinePropositionStoreIntegrationTest.kt b/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivinePropositionStoreIntegrationTest.kt index 4b89a448..07702dcd 100644 --- a/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivinePropositionStoreIntegrationTest.kt +++ b/dice-storage/src/test/kotlin/com/embabel/dice/storage/DrivinePropositionStoreIntegrationTest.kt @@ -43,18 +43,29 @@ import org.drivine.manager.PersistenceManager import org.drivine.query.QuerySpecification import org.springframework.beans.factory.annotation.Autowired import org.springframework.boot.test.context.SpringBootTest +import org.springframework.test.context.DynamicPropertyRegistry +import org.springframework.test.context.DynamicPropertySource import org.springframework.transaction.PlatformTransactionManager import java.time.Duration import java.time.Instant /** - * Integration tests for the graph storage stack against a Neo4j testcontainer (provided by Drivine's - * test support). Not `@Transactional`: dedup commits via its own [org.springframework.transaction.support.TransactionTemplate], - * so isolation is by explicit `clearAll()` per test rather than rollback. + * Integration tests for the graph storage stack against a Neo4j testcontainer. Not `@Transactional`: + * dedup commits via its own [org.springframework.transaction.support.TransactionTemplate], so + * isolation is by explicit `clearAll()` per test rather than rollback. + * + * Uses the shared [Neo4jTestContainer]; see that class for why Drivine's built-in testcontainer + * is bypassed. */ @SpringBootTest(classes = [TestApplication::class]) class DrivinePropositionStoreIntegrationTest { + companion object { + @JvmStatic + @DynamicPropertySource + fun neo4jProperties(registry: DynamicPropertyRegistry) = Neo4jTestContainer.registerProperties(registry) + } + @Autowired private lateinit var repository: DrivinePropositionRepository diff --git a/dice-storage/src/test/kotlin/com/embabel/dice/storage/InMemoryMetamodelVersionStoreContractTest.kt b/dice-storage/src/test/kotlin/com/embabel/dice/storage/InMemoryMetamodelVersionStoreContractTest.kt new file mode 100644 index 00000000..2afca2f1 --- /dev/null +++ b/dice-storage/src/test/kotlin/com/embabel/dice/storage/InMemoryMetamodelVersionStoreContractTest.kt @@ -0,0 +1,28 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.storage + +import com.embabel.dice.metamodel.InMemoryMetamodelVersionStore +import com.embabel.dice.metamodel.MetamodelVersionStore + +/** + * Runs the [AbstractMetamodelVersionStoreContractTest] suite against the in-memory reference store. + * No Docker, so it runs in the normal test phase — the always-on half of the cross-backend check + * the graph IT completes. + */ +class InMemoryMetamodelVersionStoreContractTest : AbstractMetamodelVersionStoreContractTest() { + override fun store(): MetamodelVersionStore = InMemoryMetamodelVersionStore() +} diff --git a/dice-storage/src/test/kotlin/com/embabel/dice/storage/MetamodelRowMapperTest.kt b/dice-storage/src/test/kotlin/com/embabel/dice/storage/MetamodelRowMapperTest.kt new file mode 100644 index 00000000..4a8efd80 --- /dev/null +++ b/dice-storage/src/test/kotlin/com/embabel/dice/storage/MetamodelRowMapperTest.kt @@ -0,0 +1,280 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.storage + +import com.embabel.agent.core.Cardinality +import com.embabel.dice.metamodel.MetamodelVersion +import com.embabel.dice.metamodel.PropertySignature +import com.embabel.dice.metamodel.PropertySignature.Kind +import org.junit.jupiter.api.Assertions.assertEquals +import org.junit.jupiter.api.Assertions.assertNull +import org.junit.jupiter.api.Assertions.assertTrue +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.assertThrows +import java.time.Instant + +/** + * Unit tests for [MetamodelVersionRowMapper]: no database, just the property map it produces and + * consumes. + * + * Most of these pin strict reads. A stored node missing a property the mapper wrote is corrupt, and + * the mapper has to throw: the store wraps every read in "skip the unreadable row and warn", and + * that guard needs unreadable rows to throw. `DrivineMetamodelVersionStoreIntegrationTest` shows + * the guard firing end to end; these cover the mapper's half of the contract, including the cases + * the store's own `MATCH` filters out before they can reach it. + */ +class MetamodelRowMapperTest { + + private val version = MetamodelVersion( + schemaName = "test-schema", + entityTypeNames = listOf("Person", "Company"), + entityTypeLabels = mapOf("Person" to setOf("Agent"), "Company" to setOf("Org")), + entityTypeProperties = mapOf( + "Person" to setOf( + PropertySignature("name", Kind.VALUE, "string", Cardinality.ONE), + PropertySignature("age", Kind.VALUE, "integer", Cardinality.OPTIONAL), + ), + "Company" to setOf(PropertySignature("employs", Kind.REFERENCE, "Person", Cardinality.SET)), + ), + relationshipNames = listOf("WORKS_FOR"), + ) + + /** The same schema with both kinds of alias declared on it. */ + private val aliased = MetamodelVersion( + schemaName = "aliased-schema", + entityTypeNames = listOf("Organisation"), + entityTypeLabels = mapOf("Organisation" to setOf("Entity")), + entityTypeProperties = mapOf( + "Organisation" to setOf( + PropertySignature("legalName", Kind.VALUE, "string", Cardinality.ONE, setOf("companyName", "name")), + PropertySignature("staff", Kind.REFERENCE, "Person", Cardinality.SET), + ), + ), + relationshipNames = listOf("Organisation-[EMPLOYS]->Person"), + entityTypeAliases = mapOf("Organisation" to setOf("Company", "Firm")), + ) + + private val savedAt = Instant.parse("2026-01-01T00:00:00.500Z") + + private fun row(savedAtInstant: Instant = savedAt): MutableMap = + MetamodelVersionRowMapper.bindMap(version, savedAtInstant).toMutableMap() + + @Test + fun `a version round-trips through its own property map`() { + assertEquals(version, MetamodelVersionRowMapper.fromRow(row())) + } + + @Test + fun `property signatures are written as explicit named fields, enums by name, in a fixed order`() { + // The encoding on disk feeds the content hash on the way back in, so it is a persisted + // format, and this is where its shape is pinned: + // - enum names, so inserting a constant into Cardinality can't re-point a stored ordinal; + // - map keys sorted (Company before Person, though Person was declared first); + // - signatures within a type sorted (age before name). + // The sorting is why re-saving an unchanged version writes byte-identical JSON. Left + // unsorted, the order would come from `java.util.Set.copyOf`, whose iteration order is + // randomised per JVM, so the same stamp would encode differently after every restart. + assertEquals( + """{"Company":[{"name":"employs","kind":"REFERENCE","type":"Person","cardinality":"SET"}],""" + + """"Person":[{"name":"age","kind":"VALUE","type":"integer","cardinality":"OPTIONAL"},""" + + """{"name":"name","kind":"VALUE","type":"string","cardinality":"ONE"}]}""", + row()["entityTypeProperties"], + ) + assertEquals("""["Company","Person"]""", row()["entityTypeNames"]) + assertEquals("""{"Company":["Org"],"Person":["Agent"]}""", row()["entityTypeLabels"]) + } + + @Test + fun `bindMap stamps the instant it is given, not the wall clock`() { + assertEquals(savedAt.toString(), row()["savedAt"]) + assertEquals(savedAt.toEpochMilli(), row()["savedAtEpochMillis"]) + } + + @Test + fun `every timestamp gets a sortable numeric twin, because the ISO string is not sortable`() { + // Half a second apart, and the older one has no fractional part. As strings the older sorts + // higher, because 'Z' outranks '.'; as numbers it sorts lower. Anything ordering on the + // string hands back the wrong row. + val older = row(Instant.parse("2026-01-01T00:00:00Z")) + val newer = row(Instant.parse("2026-01-01T00:00:00.500Z")) + + assertTrue(older["savedAt"].toString() > newer["savedAt"].toString(), "the string order is backwards") + assertTrue( + (older["savedAtEpochMillis"] as Long) < (newer["savedAtEpochMillis"] as Long), + "the numeric order is the true one", + ) + } + + @Test + fun `every property the mapper writes is required when reading it back`() { + // schemaName is in the list on purpose. The store's readers all MATCH on schemaName, so a + // node without one is filtered out upstream and never reaches the mapper; the mapper is + // where this contract lives, so it is pinned here. + listOf("schemaName", "contentHash", "entityTypeNames", "entityTypeLabels", "entityTypeProperties", "relationshipNames") + .forEach { property -> + val corrupt = row().apply { remove(property) } + val thrown = assertThrows("removing '$property' must fail the read") { + MetamodelVersionRowMapper.fromRow(corrupt) + } + assertTrue(thrown.message!!.contains(property), "the failure must name '$property': ${thrown.message}") + } + } + + @Test + fun `a property signature missing a field fails the read, naming the field and the type`() { + val corrupt = row().apply { + put("entityTypeProperties", """{"Person":[{"name":"name","kind":"VALUE","type":"string"}]}""") + } + + val thrown = assertThrows { MetamodelVersionRowMapper.fromRow(corrupt) } + assertTrue(thrown.message!!.contains("cardinality"), thrown.message) + assertTrue(thrown.message!!.contains("Person"), thrown.message) + } + + @Test + fun `a property signature naming an enum constant this build does not have fails the read`() { + // A node written by a build whose Cardinality had a constant ours doesn't. Substituting a + // default would change the content and then fail the integrity check with a message about + // hashes; failing here says what actually happened. + val corrupt = row().apply { + put( + "entityTypeProperties", + """{"Person":[{"name":"name","kind":"VALUE","type":"string","cardinality":"MANY_ISH"}]}""", + ) + } + + val thrown = assertThrows { MetamodelVersionRowMapper.fromRow(corrupt) } + assertTrue(thrown.message!!.contains("MANY_ISH"), thrown.message) + assertTrue(thrown.message!!.contains("Cardinality"), thrown.message) + } + + @Test + fun `a stored hash that disagrees with the stored fields fails the integrity check`() { + // contentHash is derived from the structural fields, so the copy on the node is a checksum. + // Rewriting the fields underneath it (an old hash format, a hand-edit) has to be caught. + val corrupt = row().apply { put("relationshipNames", """["SOMETHING_ELSE"]""") } + + val thrown = assertThrows { MetamodelVersionRowMapper.fromRow(corrupt) } + assertTrue(thrown.message!!.contains("integrity check"), "the failure must say what went wrong: ${thrown.message}") + } + + @Test + fun `an empty schema round-trips as empty, not as null`() { + val empty = MetamodelVersion("empty-schema", emptyList(), emptyMap(), emptyMap(), emptyList()) + + assertEquals(empty, MetamodelVersionRowMapper.fromRow(MetamodelVersionRowMapper.bindMap(empty, savedAt))) + } + + @Test + fun `a required JSON field holding the empty string fails the read`() { + // The writer never produces "" for one of these fields: an empty list or map is written as + // "[]" or "{}". A bare empty string only shows up through corruption, and it has to fail + // loudly, not read back as an empty collection that hides the problem. + val corrupt = row().apply { put("entityTypeNames", "") } + + assertThrows { MetamodelVersionRowMapper.fromRow(corrupt) } + } + + // ---- Aliases: written only when declared, absent read as none ---- + + @Test + fun `a version declaring no aliases binds neither alias field`() { + // The four-field signature encoding and the absent alias map are what a writer from before + // aliases existed produced. Keeping the empty case byte-identical is what lets old nodes + // and new ones share a natural key. + assertNull(row()["entityTypeAliases"], "an empty alias map must leave no property behind") + assertEquals( + """{"Company":[{"name":"employs","kind":"REFERENCE","type":"Person","cardinality":"SET"}],""" + + """"Person":[{"name":"age","kind":"VALUE","type":"integer","cardinality":"OPTIONAL"},""" + + """{"name":"name","kind":"VALUE","type":"string","cardinality":"ONE"}]}""", + row()["entityTypeProperties"], + ) + } + + @Test + fun `both kinds of alias are written when declared, sorted, and round-trip`() { + val bound = MetamodelVersionRowMapper.bindMap(aliased, savedAt) + + assertEquals("""{"Organisation":["Company","Firm"]}""", bound["entityTypeAliases"]) + assertEquals( + """{"Organisation":[{"name":"legalName","kind":"VALUE","type":"string","cardinality":"ONE",""" + + """"aliases":["companyName","name"]},""" + + """{"name":"staff","kind":"REFERENCE","type":"Person","cardinality":"SET"}]}""", + bound["entityTypeProperties"], + "a signature with no former names keeps exactly four fields", + ) + + val reloaded = MetamodelVersionRowMapper.fromRow(bound) + assertEquals(aliased, reloaded) + assertEquals(aliased.contentHash, reloaded.contentHash) + assertEquals(mapOf("Organisation" to setOf("Company", "Firm")), reloaded.entityTypeAliases) + assertEquals( + setOf("companyName", "name"), + reloaded.entityTypeProperties["Organisation"]!!.single { it.name == "legalName" }.aliases, + ) + } + + @Test + fun `dropping the stored alias map fails the integrity check`() { + // Aliases feed the content hash, so this is the failure mode a mapper that forgot to write + // the map would produce on every read: the stamp is unreadable, not silently alias-free. + val corrupt = MetamodelVersionRowMapper.bindMap(aliased, savedAt).toMutableMap() + .apply { remove("entityTypeAliases") } + + val thrown = assertThrows { MetamodelVersionRowMapper.fromRow(corrupt) } + assertTrue(thrown.message!!.contains("integrity check"), thrown.message) + } + + @Test + fun `dropping a signature's stored aliases fails the integrity check`() { + val corrupt = MetamodelVersionRowMapper.bindMap(aliased, savedAt).toMutableMap().apply { + put( + "entityTypeProperties", + """{"Organisation":[{"name":"legalName","kind":"VALUE","type":"string","cardinality":"ONE"},""" + + """{"name":"staff","kind":"REFERENCE","type":"Person","cardinality":"SET"}]}""", + ) + } + + val thrown = assertThrows { MetamodelVersionRowMapper.fromRow(corrupt) } + assertTrue(thrown.message!!.contains("integrity check"), thrown.message) + } + + @Test + fun `a stored aliases field that is not a list of names fails the read, naming the type`() { + val corrupt = row().apply { + put( + "entityTypeProperties", + """{"Person":[{"name":"name","kind":"VALUE","type":"string","cardinality":"ONE","aliases":"nickname"}]}""", + ) + } + + val thrown = assertThrows { MetamodelVersionRowMapper.fromRow(corrupt) } + assertTrue(thrown.message!!.contains("aliases"), thrown.message) + assertTrue(thrown.message!!.contains("Person"), thrown.message) + } + + @Test + fun `a row with no alias property at all reads back as a stamp declaring none`() { + // A node written before aliases existed. Removing the key is the same thing the graph does + // when a property was never set. + val old = row().apply { remove("entityTypeAliases") } + + val reloaded = MetamodelVersionRowMapper.fromRow(old) + + assertEquals(version, reloaded) + assertEquals(emptyMap>(), reloaded.entityTypeAliases) + } +} diff --git a/dice-storage/src/test/kotlin/com/embabel/dice/storage/Neo4jTestContainer.kt b/dice-storage/src/test/kotlin/com/embabel/dice/storage/Neo4jTestContainer.kt new file mode 100644 index 00000000..704c7356 --- /dev/null +++ b/dice-storage/src/test/kotlin/com/embabel/dice/storage/Neo4jTestContainer.kt @@ -0,0 +1,60 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.storage + +import org.springframework.test.context.DynamicPropertyRegistry +import org.testcontainers.containers.Neo4jContainer +import org.testcontainers.utility.DockerImageName + +/** + * One self-managed Neo4j testcontainer, shared by every IT in this module's test JVM, on a version + * we pick. `@EnableDrivineTestConfig` hardcodes `neo4j:5.26.1-community` with no override hook, and + * 0.0.58 is drivine4j's newest release. `5.26.1-community` has a confirmed upstream bug + * (https://github.com/neo4j/neo4j/issues/13597): a dynamic relationship-type parameter gets baked + * into the query-plan cache on first execution and silently reused for every later execution of the + * same query text with a different value. Confirmed fixed on `neo4j:2026.05-community`, which is + * what this starts. + * + * Each test class wires this container in with `test.neo4j.use-local=true`, the supported switch + * that tells `DrivineTestConfiguration` to take the `neo` datasource's host, port and password from + * the Spring `Environment` as they are. A `@DynamicPropertySource` method in each test class (see + * [registerProperties]) supplies those values from this container, started on first use and reused + * (via Kotlin `object`/`by lazy`) for the rest of the test JVM. + * + * `dice-storage-autoconfigure` keeps its own copy; each module's tests run in a forked JVM with its + * own classpath. + */ +object Neo4jTestContainer { + + const val PASSWORD = "test-password" + + val instance: Neo4jContainer<*> by lazy { + Neo4jContainer(DockerImageName.parse("neo4j:2026.05-community").asCompatibleSubstituteFor(DockerImageName.parse("neo4j"))) + .withAdminPassword(PASSWORD) + .withPlugins("apoc") + .withNeo4jConfig("dbms.security.procedures.unrestricted", "apoc.*") + .withNeo4jConfig("dbms.security.procedures.allowlist", "apoc.*") + .also { it.start() } + } + + /** Points Drivine's `neo` datasource at [instance]. */ + fun registerProperties(registry: DynamicPropertyRegistry) { + registry.add("test.neo4j.use-local") { "true" } + registry.add("database.datasources.neo.host") { instance.host } + registry.add("database.datasources.neo.port") { instance.getMappedPort(7687) } + registry.add("database.datasources.neo.password") { PASSWORD } + } +} diff --git a/dice-storage/src/test/kotlin/com/embabel/dice/storage/TestApplication.kt b/dice-storage/src/test/kotlin/com/embabel/dice/storage/TestApplication.kt index 37776c52..1abdcec8 100644 --- a/dice-storage/src/test/kotlin/com/embabel/dice/storage/TestApplication.kt +++ b/dice-storage/src/test/kotlin/com/embabel/dice/storage/TestApplication.kt @@ -31,6 +31,10 @@ import org.springframework.context.annotation.Bean import org.springframework.context.annotation.Configuration import org.springframework.context.annotation.EnableAspectJAutoProxy import org.springframework.transaction.PlatformTransactionManager +import java.time.Clock +import java.time.Instant +import java.time.ZoneId +import java.time.ZoneOffset import kotlin.random.Random /** @@ -49,6 +53,33 @@ class FakeEmbeddingService(override val dimensions: Int = 16) : EmbeddingService override fun embed(texts: List): List = texts.map(::embed) } +/** + * A clock a test can pin to an instant of its choosing. Left alone it reads the system clock, so + * anything that doesn't care about the exact save instant behaves as it would in production. Pin it + * and [DrivineMetamodelVersionStore.saveVersion] stamps the version at that instant, which is how a + * test places two saves at timestamps it chooses. + */ +class PinnableClock : Clock() { + + @Volatile + private var pinned: Instant? = null + + fun pin(instant: Instant) { + pinned = instant + } + + fun unpin() { + pinned = null + } + + override fun instant(): Instant = pinned ?: Instant.now() + + override fun getZone(): ZoneId = ZoneOffset.UTC + + /** There is only one of these, and its zone never matters: it only produces instants. */ + override fun withZone(zone: ZoneId): Clock = this +} + /** * Test wiring: Drivine's test support spins a Neo4j testcontainer and transaction management; * we add the graph stores and a fake embedding service. [SchemaCatalog] beans are ensured on @@ -126,4 +157,30 @@ open class TestApplication { repository: DrivinePropositionRepository, persistenceManager: PersistenceManager, ): GraphDecayManager = GraphDecayManager(repository, persistenceManager) + + /** + * Both MERGEs the version store performs need their key to be unique, because a MERGE is + * race-free only then. Without the first, concurrent saves of one version all miss the match, + * all create, and the history fills with duplicates. Without the second, a schema can end up + * with two counter nodes handing out the same sequence numbers. + * + * The third backs the sequence itself: it makes two versions of one schema sharing a position + * impossible to store, so a lost counter update fails with a constraint violation the caller + * can retry. `DrivineMetamodelVersionStoreIntegrationTest` pins all three. + */ + @Bean + open fun metamodelSchema(): SchemaCatalog = SchemaCatalog.of( + UniquenessConstraintSpec(label = "MetamodelVersion", properties = listOf("schemaName", "contentHash")), + UniquenessConstraintSpec(label = "MetamodelSchemaCounter", property = "schemaName"), + UniquenessConstraintSpec(label = "MetamodelVersion", properties = listOf("schemaName", "sequence")), + ) + + @Bean + open fun metamodelClock(): PinnableClock = PinnableClock() + + @Bean + open fun metamodelVersionStore( + persistenceManager: PersistenceManager, + clock: PinnableClock, + ): DrivineMetamodelVersionStore = DrivineMetamodelVersionStore(persistenceManager, clock) } diff --git a/docs/design/architecture.md b/docs/design/architecture.md index 6e2570cb..0bd50e7e 100644 --- a/docs/design/architecture.md +++ b/docs/design/architecture.md @@ -12,11 +12,11 @@ DICE is a multi-module Maven build. Each module's intent, and what it's allowed | Module | Intent | |---|---| | `dice` | The core: proposition model, pipeline, gates, projection interfaces, query facades, agent tools, REST controllers. In-memory implementations only — no database driver. | -| `dice-storage` | The durable Neo4j backend: `Drivine`-based repository, graph/Prolog/lineage projectors, schema and index bootstrap. Depends on `dice`. | +| `dice-storage` | The durable Neo4j backend: `Drivine`-based repository, graph/Prolog/lineage projectors, schema and index bootstrap, `MetamodelVersionStore` persistence. Depends on `dice` and `dice-metamodel`. | | `dice-storage-autoconfigure` | Spring Boot autoconfiguration that wires `dice-storage`'s beans (repository, projectors, trust scorer) into a host application. Depends on `dice-storage`. | | `dice-ingestion` | Content-hash dedup ledger and source adapters that sit in front of `PropositionPipeline`, so the same artifact is never extracted twice concurrently. Depends on `dice`. | | `dice-report` | Rationale and structured report generation over propositions and their lineage. Depends on `dice`. | -| `dice-metamodel` | Schema versioning: content-hash stamps over the governed part of a `DataDictionary`, the declared-schema seam, and the version store contract. Pure JVM. Depends on no other DICE module. | +| `dice-metamodel` | Schema versioning: content-hash stamps over the governed part of a `DataDictionary`, the declared-schema seam, and the version store contract. Pure JVM. Depends on no other DICE module. `dice-storage` implements its store contract. | | `dice-integration-tests` | End-to-end tests exercising the real Neo4j backend and full pipeline across module boundaries. Depends on `dice`, `dice-ingestion`, `dice-report` (and transitively `dice-storage`). Not shipped. | ```mermaid @@ -30,6 +30,7 @@ flowchart TB itest["dice-integration-tests"] storage --> dice + storage --> metamodel autoconf --> storage ingestion --> dice report --> dice @@ -39,11 +40,11 @@ flowchart TB ``` `dice` never depends on any other DICE module — it's the leaf of the graph, so every other module -can be added or removed without touching core logic. `dice-metamodel` is a second leaf with no -edges: it stamps a schema, and depends only on Embabel's agent core types. -`dice-storage-autoconfigure` is the only module that knows about Spring -Boot autoconfiguration; plain `dice-storage` stays framework-neutral so it can be wired by hand -outside Spring Boot. +can be added or removed without touching core logic. `dice-metamodel` stamps a schema, and depends +only on Embabel's agent core types. One DICE module depends on it: `dice-storage`, which implements +its `MetamodelVersionStore` against Neo4j. `dice-storage-autoconfigure` is the only module that +knows about Spring Boot autoconfiguration; plain `dice-storage` stays framework-neutral so it can +be wired by hand outside Spring Boot. ### Subsystem design docs diff --git a/docs/design/metamodel-versioning.md b/docs/design/metamodel-versioning.md index b5ff7b86..48f8c697 100644 --- a/docs/design/metamodel-versioning.md +++ b/docs/design/metamodel-versioning.md @@ -14,7 +14,8 @@ stamp against a live graph, comes later — see [the tiers ahead](#the-tiers-ahe The types live in `dice-metamodel`, a small pure-JVM module: `MetamodelVersion`, `GovernedTypeSelector`, `DeclaredSchema`/`DeclaredSchemaSource`, `SchemaAliases`, and the -`MetamodelVersionStore` contract. It depends on Embabel's agent core types and nothing else. +`MetamodelVersionStore` contract with its `InMemoryMetamodelVersionStore` reference +implementation. It depends on Embabel's agent core types and nothing else. `SchemaAliases` and the alias fields on `PropertySignature` and `MetamodelVersion` are experimental; their shape may change before 1.0. @@ -332,9 +333,57 @@ built for. `findVersion` resolves a recorded hash back to the schema shape it named. The default scans `versionHistory`, which is correct for any implementation but reads the whole history to answer a -keyed question; a backend that can push the lookup down to the database should override it. This -module ships no implementation. Storage is a separate concern, and a stamp is useful in memory -before anything durable exists. +keyed question; a backend that can push the lookup down to the database should override it. + +The only implementation this module ships is `InMemoryMetamodelVersionStore`, which keeps stamps in +a list. It exists so a host can stamp and compare schemas before it has a database, and so the +contract has an executable statement of what its rules mean; durable storage is a separate concern. + +The durable implementation lives in `dice-storage`. `DrivineMetamodelVersionStore` keeps each stamp +as a `(:MetamodelVersion)` node and MERGEs on `(schemaName, contentHash)`, so re-stamping an +unchanged schema updates the node already there. Three things govern how it behaves: + +- **It needs three uniqueness constraints**, declared in a `SchemaCatalog` bean. A MERGE is + race-free only when what it merges on is unique, so `MetamodelVersion(schemaName, contentHash)` + and `MetamodelSchemaCounter(schemaName)` are both required. Without the first, concurrent saves of + one version all miss the match, all create, and history fills with copies. The third, + `MetamodelVersion(schemaName, sequence)`, guards the ordering described below. +- **Ordered reads sort on a per-schema counter.** "Most recent" here means logical write order, + which no timestamp can express: two saves land in the same millisecond routinely, and an NTP + correction or a failover can move the clock backwards between them. Each schema owns a + `(:MetamodelSchemaCounter)` node, and a version takes the next number off it in the same statement + that creates the version node. `savedAt` and `savedAtEpochMillis` are informational; nothing sorts + on them. Because `(schemaName, sequence)` is unique, a lost counter update surfaces as a retryable + failure. +- **A re-save updates content only.** Sequence, counter, and `savedAt` keep their existing values, + so an old stamp stays at its original position in the history. `InMemoryMetamodelVersionStore`, + the reference implementation `dice-metamodel` ships, behaves the same way. + +The structural fields are stored as JSON strings, since Neo4j properties are scalars and flat +arrays. Property signatures get explicit named fields with enums by name +(`{"name": "age", "kind": "VALUE", "type": "integer", "cardinality": "ONE"}`); an ordinal would +re-point the day someone inserts a constant into `Cardinality`. The content hash is derived, so the +`contentHash` on a node is a checksum: the store recomputes it on read and skips a node that +disagrees with itself, logging a warning. + +### Aliases in storage + +Declared aliases land in two places on the node: the version-level `entityTypeAliases` map as its +own property, and a property's former names as a fifth `aliases` field inside its stored signature. +Both are written only when they hold something, so a stamp that declares no former names writes +exactly the properties the store wrote before aliases existed, and a node from that older build +reads back as a stamp declaring none. + +Getting that wrong is unrecoverable. Aliases feed `contentHash`, and the read side recomputes the +hash from the persisted fields, so a writer that dropped the alias map would produce nodes that +fail their own checksum on every read and can never be read back. Three tests pin it: one writes a +row in the old four-field shape through raw Cypher and reads it back, one round-trips a stamp +carrying both alias kinds, and one deletes the stored alias map and asserts the integrity check +rejects the row. + +`AbstractMetamodelVersionStoreContractTest` runs one suite against the graph store and the in-memory +reference, so the two can't drift apart on rules that live in Cypher on one side and Kotlin on the +other. ## Plain classes, not data classes