From c675f18223a690e170a975f867db1b4ee767dfb3 Mon Sep 17 00:00:00 2001 From: David Spencer <1526975+DecisionNerd@users.noreply.github.com> Date: Sun, 30 Aug 2026 15:18:37 -0600 Subject: [PATCH] docs(adr): record storage format exceptions --- docs-site/astro.config.mjs | 4 + docs-site/scripts/sync-content.mjs | 1 + docs/adr/0024-storage-format-exceptions.md | 109 ++++++++++++++++++ docs/adr/README.md | 1 + .../project-format-compatibility.md | 10 ++ docs/book/architecture/storage.md | 30 +++-- docs/engineering/adrs/README.md | 1 + 7 files changed, 149 insertions(+), 7 deletions(-) create mode 100644 docs/adr/0024-storage-format-exceptions.md diff --git a/docs-site/astro.config.mjs b/docs-site/astro.config.mjs index 7cd1df86e..1cee1e443 100644 --- a/docs-site/astro.config.mjs +++ b/docs-site/astro.config.mjs @@ -291,6 +291,10 @@ export default defineConfig({ label: '0023 Composable multi-ontology', slug: 'adr/0023-composable-multi-ontology', }, + { + label: '0024 Storage format exceptions', + slug: 'adr/0024-storage-format-exceptions', + }, ], }, ], diff --git a/docs-site/scripts/sync-content.mjs b/docs-site/scripts/sync-content.mjs index efa393ce3..0c034a21b 100755 --- a/docs-site/scripts/sync-content.mjs +++ b/docs-site/scripts/sync-content.mjs @@ -141,6 +141,7 @@ const PAGES = [ 'adr/0020-ntfs-write-through-namespace-durability.md', 'adr/0021-portable-project-v2.md', 'adr/0023-composable-multi-ontology.md', + 'adr/0024-storage-format-exceptions.md', 'releases/roadmap.md', 'legal/licensing.md', 'community/security.md', diff --git a/docs/adr/0024-storage-format-exceptions.md b/docs/adr/0024-storage-format-exceptions.md new file mode 100644 index 000000000..047615fca --- /dev/null +++ b/docs/adr/0024-storage-format-exceptions.md @@ -0,0 +1,109 @@ +# ADR 0024: Storage format exceptions for GFDR and compiled ontologies + +**Status:** Accepted +**Date:** 2026-08-30 +**Build target:** v0.5.x +**Related:** ADR 0019 (authoritative graph delta journal), issue #1022 + +## Context + +The default storage rule is intentionally simple: graph data uses +Arrow/Parquet, while ontology definitions and metadata use JSON or YAML. Two +shipped formats do not fit that shorthand: + +1. authoritative graph delta runs are custom binary `.gfdr` containers whose + mutation payloads are JSON; and +2. a compiled ontology runtime can be persisted as eight Parquet tables. + +Leaving these as undocumented implementation details makes the default rule +misleading. It also obscures which bytes are authority, where compatibility +versions live, and whether an incompatible representation must be migrated or +can be rebuilt. + +## Decision + +These are permanent, named exceptions to the default storage rule. “Permanent” +means that each is an intentional architecture category, not a temporary rule +violation. It does not freeze today's byte layout: every persisted exception +must have an explicit compatibility boundary, fail closed outside that +boundary, and carry its migration or rebuild policy. + +### GFDR is an authoritative, versioned graph-data exception + +An inventory-listed `graph/deltas/run_*.gfdr` file is authoritative graph state +when the selected generation represents its graph as canonical Parquet base +files plus an ordered delta chain. The `.gfdr` extension alone is not a format +marker. Version 1 is identified and framed as follows; all integers are little +endian: + +| Scope | Current marker | Authoritative content | +| --- | --- | --- | +| File | four bytes `GFDR`, then `u32` run-format version `1` | run sequence, run UUID, transaction UUID, record count, records, and SHA-256 framing checksums | +| Record | `u16` record version `1` | operation UUID and sequence, closed mutation kind, payload schema ID, payload length and bytes, and record checksum | +| Payload | `u16` schema ID `1` | `serde_json` encoding of the tagged Rust `GraphDeltaPayload`; typed property values are `serde_json` encodings of `IrLiteral` carried in the payload's value field | + +The JSON payload does not make GFDR a JSON sidecar. The checked binary framing, +ordered records, and schema-qualified payload together are the authoritative +mutation contract. Unknown run-format versions fail with +`GF_UNSUPPORTED_PROJECT_FORMAT`; malformed framing, lengths, order, or digests +fail as corruption. Replay never guesses from an extension, partial prefix, or +unversioned payload. + +GFDR's migration debt is explicit. The pre-release routing-free and +string-only payload prototype is not losslessly migratable and remains +rejected. A future framing or payload version must add a bounded reader and an +explicit generation migration/compaction path, or remain unsupported. It must +not silently reinterpret old bytes. This specializes, rather than replaces, +the authority and publication rules in [ADR 0019](0019-authoritative-graph-delta-journal.md). + +### Ontology Parquet is a derived runtime-persistence exception + +An adopted ontology's canonical `OntologyDoc` JSON inside the CURRENT-selected +workspace participant is durable ontology authority. YAML and JSON files are +authoring/import inputs; a file elsewhere in the project tree cannot override +the committed participant. Startup compiles the committed document into Arrow +runtime tables. `graphforge-ontology` may also persist the compiled runtime as +eight Parquet files (`ontology_meta`, `entity_types`, `relation_types`, +`property_types`, `type_constraints`, +`semantic_flags`, `cardinality_rules`, and `aliases`) to avoid repeating parse, +validation, and compilation work. + +Those Parquet files are derived runtime data, not a second ontology-definition +authority. They carry the ontology ID, ontology version, source checksum, and +writer version in Arrow schema metadata. When a caller supplies the expected +ontology checksum, a mismatch means the snapshot was compiled from a different +document and must be discarded and recompiled from the committed JSON. +Corrupt, missing, or incompatible compiled tables likewise do not authorize +changes to that document. + +The compiled snapshot's migration debt is rebuildable rather than semantic: +the current writer records `graphforge.writer_version = 0.5.0`, but the loader +does not yet enforce that marker as a closed snapshot-format version. Until an +enforced compiled-snapshot version gate exists, cross-release reuse must be +treated as unsupported unless the current reader accepts the tables and their +ontology checksum matches the committed document. Recompilation from that +authoritative JSON document is the upgrade path; in-place inference or repair +of unknown Parquet schemas is not. + +## Consequences + +- The concise storage rule must name both exceptions wherever it is presented + as an architecture constraint. +- GFDR receives the same review rigor as any authoritative graph format: + versioning, bounds, checksums, compatibility failures, and explicit migration + are correctness concerns. +- Compiled ontology Parquet may be deleted and rebuilt without changing + ontology meaning, provided the authoritative committed JSON document is + retained. +- A `.gfdr` run may not be deleted or rebuilt independently of its selected + generation; compaction must publish an equivalent new generation through the + normal authority protocol. + +## Rejected alternatives + +| Alternative | Reason | +| --- | --- | +| Describe GFDR as JSON metadata | Its binary framing and ordered payloads are authoritative graph mutations, not metadata or a sidecar. | +| Encode each small mutation as Parquet | Reintroduces the rewrite amplification ADR 0019 was designed to avoid. | +| Treat compiled ontology Parquet as source authority | Creates two competing ontology definitions and makes checksum mismatch ambiguous. | +| Promise transparent pre-v1 migration | The old GFDR payload is not losslessly recoverable, and the compiled snapshot lacks an enforced format-version gate. | diff --git a/docs/adr/README.md b/docs/adr/README.md index eb49a9415..947ccded2 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -29,6 +29,7 @@ are not retained in this tree. | 0021 | [Portable project v2 package layout and identity](0021-portable-project-v2.md) | `0021-portable-project-v2.md` | | 0022 | [Multi-ontology semantics in portable project v2](0022-portable-v2-multi-ontology-compatibility.md) | `0022-portable-v2-multi-ontology-compatibility.md` | | 0023 | [Composable ontology modules and semantic bridges](0023-composable-multi-ontology.md) | `0023-composable-multi-ontology.md` | +| 0024 | [Storage format exceptions for GFDR and compiled ontologies](0024-storage-format-exceptions.md) | `0024-storage-format-exceptions.md` | ## Numbering diff --git a/docs/book/architecture/project-format-compatibility.md b/docs/book/architecture/project-format-compatibility.md index 877af96bf..d422dd3bb 100644 --- a/docs/book/architecture/project-format-compatibility.md +++ b/docs/book/architecture/project-format-compatibility.md @@ -33,6 +33,16 @@ and plain-string property representation are not losslessly migratable and are rejected with `GF_UNSUPPORTED_PROJECT_FORMAT`; GraphForge never silently reinterprets them. +GFDR is therefore a permanent, versioned exception to the default Parquet +graph-data rule, not an unversioned sidecar. Its exact magic, run/record/schema +markers, authoritative JSON payload contract, and migration debt are recorded +in [ADR 0024](../../adr/0024-storage-format-exceptions.md). The same ADR records +compiled ontology Parquet as a different exception: a derived runtime snapshot +that never replaces the CURRENT-selected workspace's canonical ontology JSON +and may be discarded and recompiled when incompatible or stale. External YAML +or JSON files remain authoring/import inputs rather than project commit +authority. + Portable-v2 selection is dependency-closed for graph semantic bindings. When the selected participant set includes the graph semantic-bindings participant, the bundle must also contain the exact workspace ontology-composition diff --git a/docs/book/architecture/storage.md b/docs/book/architecture/storage.md index d23b944b2..ef9c1b22e 100644 --- a/docs/book/architecture/storage.md +++ b/docs/book/architecture/storage.md @@ -1,7 +1,7 @@ # Storage Architecture **Status:** v0.5.0 — Parquet project storage shipped -**Last Updated:** 2026-07-27 +**Last Updated:** 2026-08-30 --- @@ -26,7 +26,9 @@ Parquet is the sole storage provider for the Rust core. It: - Stores graph tables and opaque domain-owned participants as columnar Parquet files; `graphforge-storage` does not define provenance or knowledge semantics - Carries GraphForge metadata at the file level (ontology version, IR version, query ID) -- Persists the compiled ontology runtime tables for rapid startup +- Persists derived compiled ontology runtime tables for rapid startup; the + CURRENT-selected workspace's canonical ontology JSON remains durable + authority Parquet file-level metadata: @@ -312,7 +314,10 @@ or nonmaximal shapes, wrong routes, duplicate references, and corrupt objects. Authoritative small-write delta runs, when present, live under `graph/deltas/` inside the same generation and are inventory-verified -([ADR 0019](../../adr/0019-authoritative-graph-delta-journal.md)). Compaction +([ADR 0019](../../adr/0019-authoritative-graph-delta-journal.md)). GFDR's +binary framing and JSON payload schema are a permanent, versioned exception to +the default Parquet graph-data rule; see +[ADR 0024](../../adr/0024-storage-format-exceptions.md). Compaction and ordinary opens decode the compact base from these canonical Parquet files; there is no duplicate JSON graph-state authority. A delta-bearing open verifies the contiguous typed GFDR chain, materializes a contained private Parquet view, @@ -757,7 +762,7 @@ The ontology is a **runtime-loadable knowledge schema**, not Rust structs genera |---|---| | **YAML / JSON** | Human-authored ontology definitions (Serde-based load) | | **Arrow tables** | Compiled execution format — cheap joins during binding and planning | -| **Parquet** | Persisted for rapid startup or reproducible deployments | +| **Parquet** | Derived compiled-runtime snapshot for rapid startup or reproducible deployments; discard and recompile on ontology-checksum mismatch | ### Ontology authoring format (YAML) @@ -804,7 +809,9 @@ At load time this compiles into Arrow lookup tables keyed by integer type IDs. S | `cardinality_rules` | Endpoint multiplicity (min/max per relation type) | | `semantic_flags` | `transitive`, `symmetric`, `reflexive`, `functional`, `acyclic` | | `aliases` | Human-facing and deprecated names | -| `migrations` | Versioned ontology upgrade transforms | + +Versioned migration transforms remain part of the authoritative `OntologyDoc`; +they are not a ninth compiled runtime table or Parquet snapshot file. ### Ontology versioning @@ -829,14 +836,23 @@ A new ontology version does not require a new IR version, and vice versa. Persis ## Serialization Systems -**Never mix these two systems:** +**Default rule (with two named exceptions):** | System | Purpose | Format | |---|---|---| | **Arrow / Parquet** (`graphforge-storage`) | Graph topology/properties and generic persistence of domain-owned participants | Binary columnar (Arrow IPC / Parquet) | | **JSON / YAML** (`graphforge-ontology`) | Ontology definitions and metadata | Text (human-readable, validatable) | -Graph data → Arrow/Parquet. Ontology/metadata → JSON or YAML. Arrow schema metadata carries version and provenance annotations across language boundaries. +Graph data → Arrow/Parquet. Ontology definitions and metadata → JSON or YAML. +The permanent exceptions are (1) authoritative, versioned GFDR binary delta +runs with schema-qualified JSON mutation payloads, and (2) derived compiled +ontology runtime tables persisted as Parquet. The latter never supersede the +CURRENT-selected workspace's canonical ontology JSON and are +discarded/recompiled when its checksum differs. External YAML/JSON files are +authoring/import inputs, not alternate project commit pointers. +[ADR 0024](../../adr/0024-storage-format-exceptions.md) +defines both compatibility and migration boundaries. Arrow schema metadata +carries version and provenance annotations across language boundaries. --- diff --git a/docs/engineering/adrs/README.md b/docs/engineering/adrs/README.md index 1e10c9c42..49b139b9b 100644 --- a/docs/engineering/adrs/README.md +++ b/docs/engineering/adrs/README.md @@ -60,3 +60,4 @@ Keeper set after #2730 (mirrors [`../../adr/README.md`](../../adr/README.md)): | 0021 | Portable project v2 package layout and identity | Proposed | [`../../adr/0021-portable-project-v2.md`](../../adr/0021-portable-project-v2.md) | | 0022 | Multi-ontology semantics in portable project v2 | Accepted | [`../../adr/0022-portable-v2-multi-ontology-compatibility.md`](../../adr/0022-portable-v2-multi-ontology-compatibility.md) | | 0023 | Composable ontology modules and semantic bridges | Accepted | [`../../adr/0023-composable-multi-ontology.md`](../../adr/0023-composable-multi-ontology.md) | +| 0024 | Storage format exceptions for GFDR and compiled ontologies | Accepted | [`../../adr/0024-storage-format-exceptions.md`](../../adr/0024-storage-format-exceptions.md) |