diff --git a/.github/DISCUSSION_TEMPLATE/rfc.yml b/.github/DISCUSSION_TEMPLATE/rfc.yml index 2a635252..115fa8aa 100644 --- a/.github/DISCUSSION_TEMPLATE/rfc.yml +++ b/.github/DISCUSSION_TEMPLATE/rfc.yml @@ -6,8 +6,9 @@ body: Use this to **incubate an RFC** — socialize a design and reach rough consensus before writing the formal document. When it's ready, graduate it into a pull request that adds `docs/rfcs/NNNN-title.md` + from the public template with `Author track: Public contribution` (see [docs/rfcs/README.md](../blob/main/docs/rfcs/README.md)); a - maintainer merging that PR is acceptance. + maintainer merging that public-track PR is acceptance. For a plain feature request or open-ended idea, use the **Ideas** category instead. For bugs, open an [Issue](../../issues/new/choose). diff --git a/AGENTS.md b/AGENTS.md index 3ad1e9ef..b60cd05c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -72,6 +72,7 @@ Full diagram and concurrency model: [docs/dev/architecture.md](docs/dev/architec | **Architectural invariants & deny-list (read before any non-trivial proposal or review)** | **[docs/dev/invariants.md](docs/dev/invariants.md)** | | **Lance docs index — fetch upstream Lance docs by problem domain** | **[docs/dev/lance.md](docs/dev/lance.md)** | | **Test coverage map — what's covered, what helpers to reuse, before-every-task checklist** | **[docs/dev/testing.md](docs/dev/testing.md)** | +| The canon — linear internal narrative of the whole system (philosophy, read/write/crash walkthroughs, exclusions, risk register, roadmap) | [docs/dev/canon.md](docs/dev/canon.md) | | Architecture, L1/L2 framing, concurrency model | [docs/dev/architecture.md](docs/dev/architecture.md) | | Storage layout, `__manifest` schema, URI schemes, S3 env vars | [docs/user/concepts/storage.md](docs/user/concepts/storage.md) | | `.pg` schema language, types, constraints, annotations, migration planning | [docs/user/schema/index.md](docs/user/schema/index.md) | @@ -106,7 +107,7 @@ Full diagram and concurrency model: [docs/dev/architecture.md](docs/dev/architec | CI / release workflows | [docs/dev/ci.md](docs/dev/ci.md) | | Branch protection policy (declarative, applied via `scripts/apply-branch-protection.sh`) | [docs/dev/branch-protection.md](docs/dev/branch-protection.md) | | Constants & tunables cheat sheet | [docs/user/reference/constants.md](docs/user/reference/constants.md) | -| RFC process — public contribution track (substantial / irreversible changes) | [docs/rfcs/README.md](docs/rfcs/README.md) | +| RFC process — public contribution and maintainer design-series tracks | [docs/rfcs/README.md](docs/rfcs/README.md) | | Per-version release notes | [docs/releases/](docs/releases/) | --- diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1029b2fd..66bf4e4b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -9,7 +9,7 @@ rules and decision authority behind it live in [GOVERNANCE.md](GOVERNANCE.md). |---|---|---| | **Report a bug** or wrong behavior | **[Open an Issue](../../issues/new/choose)** | Concrete and reproducible. A maintainer triages it; once labelled **`accepted`** it's open for a PR. | | **Suggest a feature / share an idea / ask** | **[Start a Discussion](../../discussions)** | Ideas and questions live here, not in Issues. | -| **Propose a design / RFC** | **An RFC pull request** | Anyone can author one — see [docs/rfcs/README.md](docs/rfcs/README.md). A maintainer merging it is acceptance. | +| **Propose a design / RFC** | **An RFC pull request** | Anyone can copy the public template and declare `Author track: Public contribution` — see [docs/rfcs/README.md](docs/rfcs/README.md). A maintainer merging a public RFC is acceptance; maintainer design series use their explicit status lifecycle. | | **Fix something / implement a change** | **A pull request** | Must link an `accepted` issue or an accepted RFC — unless it's trivial (below). | | **Report a security vulnerability** | **[SECURITY.md](SECURITY.md)** | Do **not** open a public Issue. | diff --git a/GOVERNANCE.md b/GOVERNANCE.md index 2768e5b0..2b72e84d 100644 --- a/GOVERNANCE.md +++ b/GOVERNANCE.md @@ -65,10 +65,11 @@ first, or pick one already labelled `accepted` / `help wanted`. ### Discussions → RFCs → accepted Ideas and feature requests start in **Discussions**. Anyone — including external contributors — may then **author an RFC** by opening a pull request that adds -`docs/rfcs/NNNN-title.md` (see [docs/rfcs/README.md](docs/rfcs/README.md)). The -RFC is reviewed as code; **a maintainer merging it is the act of acceptance** -(it becomes the durable decision record). Implementation PRs then reference the -accepted RFC. +`docs/rfcs/NNNN-title.md` from the public template with +`Author track: Public contribution` (see +[docs/rfcs/README.md](docs/rfcs/README.md)). The RFC is reviewed as code; **a +maintainer merging it is the act of acceptance** (it becomes the durable +decision record). Implementation PRs then reference the accepted RFC. Authoring an RFC is open to everyone; **accepting one is a maintainer decision.** Maintainers may also decline an RFC, with rationale, by closing it. diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index 6da3c9e9..411c490c 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -1265,19 +1265,18 @@ impl Omnigraph { Ok(()) } - /// Legacy write-entry heal: converge any roll-forward-eligible recovery + /// Broad write-entry heal: converge any roll-forward-eligible recovery /// sidecars and leave rollback-eligible intents for the next ReadWrite open. /// - /// This preserves the pre-RFC-022 behavior for adapters that have not yet - /// enrolled in the closed recovery barrier (schema apply and maintenance - /// adapters in this slice). Mutation/load and branch merge use + /// Schema apply calls this broad barrier before acquiring its schema gate; + /// exact adapters then relist and revalidate relevant recovery and authority + /// under their own ordered effect gates. Mutation/load and branch merge use /// [`heal_pending_recovery_sidecars_for_write`](Self::heal_pending_recovery_sidecars_for_write) - /// instead and reject relevant unresolved intents before capturing a base. + /// to reject relevant unresolved intents before capturing a base. /// /// Steady-state cost here is one `list_dir` of `__recovery/` (typically - /// empty → immediate return). Enrolled mutation/load and branch-merge - /// attempts perform a second check under their effect gates to close the - /// post-prepare race. See + /// empty → immediate return). Exact adapters perform a second check under + /// their effect gates to close the post-prepare race. See /// `recovery::heal_pending_sidecars_roll_forward` for the /// concurrency contract (root-scoped ordered gate acquisition). pub(crate) async fn heal_pending_recovery_sidecars(&self) -> Result<()> { diff --git a/crates/omnigraph/tests/lance_surface_guards.rs b/crates/omnigraph/tests/lance_surface_guards.rs index 6871243a..c1c34a0c 100644 --- a/crates/omnigraph/tests/lance_surface_guards.rs +++ b/crates/omnigraph/tests/lance_surface_guards.rs @@ -30,12 +30,15 @@ use arrow_schema::{DataType, Field, Schema}; use lance::Dataset; use lance::dataset::builder::DatasetBuilder; use lance::dataset::optimize::{CompactionOptions, compact_files}; -use lance::dataset::transaction::Operation; +use lance::dataset::transaction::{Operation, Transaction}; use lance::dataset::write::delete::DeleteResult; +use lance::dataset::write::merge_insert::UncommittedMergeInsert; +use lance::dataset::write::merge_insert::inserted_rows::{FilterType, KeyExistenceFilter}; use lance::dataset::{ CommitBuilder, InsertBuilder, MergeInsertBuilder, WhenMatched, WhenNotMatched, WriteMode, WriteParams, }; +use lance::datatypes::LANCE_UNENFORCED_PRIMARY_KEY; use lance::index::DatasetIndexExt; use lance_file::version::LanceFileVersion; use lance_index::IndexType; @@ -69,6 +72,175 @@ async fn fresh_dataset(uri: &str) -> Dataset { Dataset::write(reader, uri, Some(params)).await.unwrap() } +/// RFC-023 substrate fixture: a V2_2 table whose internal `id` is Lance's +/// unenforced primary key. `note` is nullable so the matched-only partial-schema +/// guard can omit it without testing an unrelated nullability rejection. +async fn fresh_pk_dataset(uri: &str) -> Dataset { + let schema = Arc::new(Schema::new(vec![ + Field::new("id", DataType::Utf8, false).with_metadata( + [(LANCE_UNENFORCED_PRIMARY_KEY.to_string(), "true".to_string())] + .into_iter() + .collect(), + ), + Field::new("value", DataType::Int32, false), + Field::new("note", DataType::Utf8, true), + ])); + let batch = RecordBatch::try_new( + schema.clone(), + vec![ + Arc::new(StringArray::from(vec!["alice", "bob"])), + Arc::new(Int32Array::from(vec![1, 2])), + Arc::new(StringArray::from(vec![Some("a"), Some("b")])), + ], + ) + .unwrap(); + let reader = RecordBatchIterator::new(vec![Ok(batch)], schema); + let params = WriteParams { + mode: WriteMode::Create, + enable_stable_row_ids: true, + data_storage_version: Some(LanceFileVersion::V2_2), + ..Default::default() + }; + Dataset::write(reader, uri, Some(params)).await.unwrap() +} + +fn pk_full_row(dataset: &Dataset, id: &str, value: i32) -> RecordBatch { + let schema = Arc::new(Schema::from(dataset.schema())); + RecordBatch::try_new( + schema, + vec![ + Arc::new(StringArray::from(vec![id])), + Arc::new(Int32Array::from(vec![value])), + Arc::new(StringArray::from(vec![Some("guard")])), + ], + ) + .unwrap() +} + +async fn stage_pk_merge( + dataset: Arc, + batch: RecordBatch, + on: &str, + when_matched: WhenMatched, + when_not_matched: WhenNotMatched, + use_index: Option, +) -> UncommittedMergeInsert { + let schema = batch.schema(); + let reader = RecordBatchIterator::new(vec![Ok(batch)], schema); + let mut builder = MergeInsertBuilder::try_new(dataset, vec![on.to_string()]).unwrap(); + builder + .when_matched(when_matched) + .when_not_matched(when_not_matched) + .conflict_retries(0); + if let Some(use_index) = use_index { + builder.use_index(use_index); + } + builder + .try_build() + .unwrap() + .execute_uncommitted(reader) + .await + .unwrap() +} + +fn transaction_inserted_rows_filter(transaction: &Transaction) -> Option<&KeyExistenceFilter> { + match &transaction.operation { + Operation::Update { + inserted_rows_filter, + .. + } => inserted_rows_filter.as_ref(), + other => panic!("expected merge_insert to stage Operation::Update, got {other:?}"), + } +} + +fn staged_inserted_rows_filter(staged: &UncommittedMergeInsert) -> Option<&KeyExistenceFilter> { + let transaction_filter = transaction_inserted_rows_filter(&staged.transaction); + assert_eq!( + staged.inserted_rows_filter.as_ref(), + transaction_filter, + "the public uncommitted result and its transaction must expose the same key filter" + ); + transaction_filter +} + +fn assert_bloom_empty(filter: &KeyExistenceFilter, expected_empty: bool, case: &str) { + let FilterType::Bloom { bitmap, .. } = &filter.filter else { + panic!("{case}: beta.21 merge_insert should emit a Bloom key filter") + }; + assert_eq!( + bitmap.iter().all(|byte| *byte == 0), + expected_empty, + "{case}: unexpected Bloom-filter population state" + ); +} + +#[derive(Debug, Clone, Copy)] +enum ConflictMatrixTxn { + Filtered { id: &'static str, value: i32 }, + UnfilteredUpdate { id: &'static str, value: i32 }, + Append { id: &'static str, value: i32 }, +} + +async fn stage_conflict_matrix_txn(dataset: Arc, kind: ConflictMatrixTxn) -> Transaction { + let (id, value) = match kind { + ConflictMatrixTxn::Filtered { id, value } + | ConflictMatrixTxn::UnfilteredUpdate { id, value } + | ConflictMatrixTxn::Append { id, value } => (id, value), + }; + let batch = pk_full_row(dataset.as_ref(), id, value); + + let transaction = match kind { + ConflictMatrixTxn::Filtered { .. } => { + stage_pk_merge( + dataset, + batch, + "id", + WhenMatched::UpdateAll, + WhenNotMatched::InsertAll, + Some(false), + ) + .await + .transaction + } + ConflictMatrixTxn::UnfilteredUpdate { .. } => { + stage_pk_merge( + dataset, + batch, + "value", + WhenMatched::UpdateAll, + WhenNotMatched::InsertAll, + Some(false), + ) + .await + .transaction + } + ConflictMatrixTxn::Append { .. } => InsertBuilder::new(dataset) + .with_params(&WriteParams { + mode: WriteMode::Append, + ..Default::default() + }) + .execute_uncommitted(vec![batch]) + .await + .unwrap(), + }; + + match kind { + ConflictMatrixTxn::Filtered { .. } => assert!( + transaction_inserted_rows_filter(&transaction).is_some(), + "filtered matrix fixture must carry the PK filter" + ), + ConflictMatrixTxn::UnfilteredUpdate { .. } => assert!( + transaction_inserted_rows_filter(&transaction).is_none(), + "non-PK merge matrix fixture must be an unfiltered Update" + ), + ConflictMatrixTxn::Append { .. } => assert!( + matches!(&transaction.operation, Operation::Append { .. }), + "append matrix fixture must stage Operation::Append" + ), + } + transaction +} + // --- Guard 1: LanceError::TooMuchWriteContention variant exists ------------ // // `db/manifest/publisher.rs::map_lance_publish_error` pattern-matches on this @@ -1154,6 +1326,281 @@ async fn unenforced_primary_key_is_immutable_once_set() { ); } +// --- Guard 19b: beta.21 merge_insert PK-filter shape ----------------------- +// +// RFC-023 can rely on Lance's key-conflict fencing only when every keyed +// insert produces an `Operation::Update.inserted_rows_filter`. In beta.21 that +// is a route-dependent contract: the v2 plan emits a filter when the ordered +// ON field ids exactly match the unenforced PK, while the scalar-index v1 path +// and non-PK ON shapes emit `None`. A matched-only v2 update still emits +// `Some(empty Bloom)`, which is important because `Some` selects the strict +// conflict-resolver branch even though this attempt inserted no key. +#[tokio::test] +async fn unenforced_pk_filter_shape_is_route_dependent_on_beta21() { + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().join("rfc023_filter_v2.lance"); + let dataset = Arc::new(fresh_pk_dataset(uri.to_str().unwrap()).await); + let id_field_id = dataset.schema().field("id").unwrap().id; + + // Exact PK ON + forced non-index plan: a real insert produces a populated + // Bloom filter over exactly the PK field id. + let upsert = stage_pk_merge( + dataset.clone(), + pk_full_row(dataset.as_ref(), "fresh-upsert", 10), + "id", + WhenMatched::UpdateAll, + WhenNotMatched::InsertAll, + Some(false), + ) + .await; + assert_eq!(upsert.stats.num_inserted_rows, 1); + assert_eq!(upsert.stats.num_updated_rows, 0); + let filter = staged_inserted_rows_filter(&upsert) + .expect("exact PK v2 upsert must carry a key-existence filter"); + assert_eq!(filter.field_ids, vec![id_field_id]); + assert_bloom_empty(filter, false, "exact-PK upsert"); + + // Strict insert uses the same filter-bearing v2 route. The source + // key must be fresh: `WhenMatched::Fail` correctly errors during staging + // if the key already exists. + let strict_create = stage_pk_merge( + dataset.clone(), + pk_full_row(dataset.as_ref(), "fresh-strict", 11), + "id", + WhenMatched::Fail, + WhenNotMatched::InsertAll, + Some(false), + ) + .await; + let filter = staged_inserted_rows_filter(&strict_create) + .expect("strict create on the PK must retain the key-existence filter"); + assert_eq!(filter.field_ids, vec![id_field_id]); + assert_bloom_empty(filter, false, "strict PK create"); + + // A valid but non-PK ON column still uses v2 when indices are disabled, + // yet it must not claim PK conflict coverage. + let mismatched_on = stage_pk_merge( + dataset.clone(), + pk_full_row(dataset.as_ref(), "fresh-non-pk", 12), + "value", + WhenMatched::UpdateAll, + WhenNotMatched::InsertAll, + Some(false), + ) + .await; + assert!( + staged_inserted_rows_filter(&mismatched_on).is_none(), + "v2 merge on a non-PK field must not emit a PK filter" + ); + + // Matched-only partial-schema v2 update: no Insert action touches the + // filter builder, but beta.21 deliberately retains `Some(empty Bloom)`. + let partial_schema = Arc::new(Schema::new(vec![ + Field::new("id", DataType::Utf8, false), + Field::new("value", DataType::Int32, false), + ])); + let partial_batch = RecordBatch::try_new( + partial_schema.clone(), + vec![ + Arc::new(StringArray::from(vec!["alice"])), + Arc::new(Int32Array::from(vec![42])), + ], + ) + .unwrap(); + let matched_only = stage_pk_merge( + dataset, + partial_batch, + "id", + WhenMatched::UpdateAll, + WhenNotMatched::DoNothing, + Some(false), + ) + .await; + assert_eq!(matched_only.stats.num_updated_rows, 1); + assert_eq!(matched_only.stats.num_inserted_rows, 0); + let filter = staged_inserted_rows_filter(&matched_only) + .expect("matched-only PK v2 update must emit Some(empty filter)"); + assert_eq!(filter.field_ids, vec![id_field_id]); + assert_bloom_empty(filter, true, "matched-only partial-schema PK update"); + + // With a scalar index on every ON column and the default `use_index=true`, + // beta.21 selects v1. That route hardcodes `inserted_rows_filter=None`. + let indexed_uri = dir.path().join("rfc023_filter_indexed_v1.lance"); + let mut indexed = fresh_pk_dataset(indexed_uri.to_str().unwrap()).await; + indexed + .create_index_builder(&["id"], IndexType::BTree, &ScalarIndexParams::default()) + .replace(true) + .await + .unwrap(); + let indexed = Arc::new(indexed); + let indexed_route = stage_pk_merge( + indexed.clone(), + pk_full_row(indexed.as_ref(), "fresh-indexed", 13), + "id", + WhenMatched::UpdateAll, + WhenNotMatched::InsertAll, + None, // preserve MergeInsertBuilder's default use_index=true + ) + .await; + assert!( + staged_inserted_rows_filter(&indexed_route).is_none(), + "the beta.21 all-keys-indexed v1 route must remain visibly unfenced" + ); +} + +// --- Guard 19c: beta.21 key-filter conflict checks are directional --------- +// +// Lance evaluates compatibility from the transaction currently being rebased +// against the transaction already committed. beta.21 is deliberately strict +// for `filtered current / unfiltered committed`, but the reverse order is +// accepted. RFC-023 must not assume a symmetric conflict matrix until the +// upstream resolver actually supplies one. +#[tokio::test] +async fn unenforced_pk_conflict_matrix_is_directional_on_beta21() { + struct Case { + name: &'static str, + committed: ConflictMatrixTxn, + current: ConflictMatrixTxn, + current_succeeds: bool, + assert_disjoint_filters: bool, + } + + let cases = [ + Case { + name: "same-key filtered current / filtered committed", + committed: ConflictMatrixTxn::Filtered { + id: "same-fresh-key", + value: 10, + }, + current: ConflictMatrixTxn::Filtered { + id: "same-fresh-key", + value: 11, + }, + current_succeeds: false, + assert_disjoint_filters: false, + }, + Case { + name: "disjoint filtered current / filtered committed", + committed: ConflictMatrixTxn::Filtered { + id: "left-fresh-key", + value: 10, + }, + current: ConflictMatrixTxn::Filtered { + id: "right-fresh-key", + value: 11, + }, + current_succeeds: true, + assert_disjoint_filters: true, + }, + Case { + name: "filtered current / unfiltered Update committed", + committed: ConflictMatrixTxn::UnfilteredUpdate { + id: "same-mixed-update-key", + value: 10, + }, + current: ConflictMatrixTxn::Filtered { + id: "same-mixed-update-key", + value: 11, + }, + current_succeeds: false, + assert_disjoint_filters: false, + }, + Case { + name: "unfiltered Update current / filtered committed", + committed: ConflictMatrixTxn::Filtered { + id: "same-mixed-update-key", + value: 10, + }, + current: ConflictMatrixTxn::UnfilteredUpdate { + id: "same-mixed-update-key", + value: 11, + }, + current_succeeds: true, + assert_disjoint_filters: false, + }, + Case { + name: "filtered current / Append committed", + committed: ConflictMatrixTxn::Append { + id: "same-mixed-append-key", + value: 10, + }, + current: ConflictMatrixTxn::Filtered { + id: "same-mixed-append-key", + value: 11, + }, + current_succeeds: false, + assert_disjoint_filters: false, + }, + Case { + name: "Append current / filtered Update committed", + committed: ConflictMatrixTxn::Filtered { + id: "same-mixed-append-key", + value: 10, + }, + current: ConflictMatrixTxn::Append { + id: "same-mixed-append-key", + value: 11, + }, + current_succeeds: true, + assert_disjoint_filters: false, + }, + ]; + + let dir = tempfile::tempdir().unwrap(); + for (case_index, case) in cases.into_iter().enumerate() { + let uri = dir + .path() + .join(format!("rfc023_conflict_{case_index}.lance")); + let base = Arc::new(fresh_pk_dataset(uri.to_str().unwrap()).await); + + // Stage both operations from the exact same stale base. Staging the + // second after the first commit would avoid the rebase this guard owns. + let committed_tx = stage_conflict_matrix_txn(base.clone(), case.committed).await; + let current_tx = stage_conflict_matrix_txn(base.clone(), case.current).await; + + if case.assert_disjoint_filters { + let committed_filter = transaction_inserted_rows_filter(&committed_tx).unwrap(); + let current_filter = transaction_inserted_rows_filter(¤t_tx).unwrap(); + assert_eq!( + committed_filter.intersects(current_filter).unwrap(), + (false, false), + "{}: chosen Bloom-filter fixtures must be definitively disjoint", + case.name + ); + } + + CommitBuilder::new(base.clone()) + .with_max_retries(0) + .execute(committed_tx) + .await + .unwrap_or_else(|error| panic!("{}: first commit failed: {error}", case.name)); + + // This is the raw commit/rebase result. `MergeInsertJob::execute` + // wraps an exhausted semantic retry as TooMuchWriteContention; the + // substrate contract exposed here is RetryableCommitConflict. + let current_result = CommitBuilder::new(base) + .with_max_retries(0) + .execute(current_tx) + .await; + if case.current_succeeds { + assert!( + current_result.is_ok(), + "{}: beta.21 should accept this direction, got {current_result:?}", + case.name + ); + } else { + assert!( + matches!( + ¤t_result, + Err(lance::Error::RetryableCommitConflict { .. }) + ), + "{}: expected beta.21 RetryableCommitConflict, got {current_result:?}", + case.name + ); + } + } +} + // --- Guard 20: camelCase @index equality routes to the scalar index (#283) ---- // // The #283 read-pushdown fix builds the filter column with datafusion `ident()` diff --git a/docs/dev/architecture.md b/docs/dev/architecture.md index 99444bf4..c28bae7c 100644 --- a/docs/dev/architecture.md +++ b/docs/dev/architecture.md @@ -66,11 +66,11 @@ flowchart TB coord[coordinator
ManifestCoordinator · CommitGraph]:::l2 end - subgraph storage[storage trait — wraps Lance] - ts[table_store · storage.rs
direct lance::Dataset today]:::l2 + subgraph storage[storage boundary — wraps Lance] + ts[sealed TableStorage writes
read-only snapshot facade]:::l2 end - subgraph lance_layer[Lance 4.x — substrate] + subgraph lance_layer[Lance 9.x — substrate] lance[per-dataset versions, fragments
BTREE · Inverted FTS · IVF/HNSW vector
merge_insert · compact_files · cleanup_old_versions]:::l1 end @@ -86,7 +86,10 @@ flowchart TB lance_layer -- bytes --> object_store ``` -The storage seam is partly aspirational. `TableStorage` exists as the sealed staged-write trait, but capability/stat surfaces and full call-site migration are still roadmap. The diagram shows the intended boundary. +The write-side storage seam is enforced: supported data-table write effects route +through the sealed `TableStorage` staging surface, with Optimize as the documented +bounded maintenance exception. Read, capability, and statistics surfaces are not +yet one complete substrate trait; that planner-facing boundary remains roadmap. ## Component zoom-ins @@ -230,19 +233,17 @@ flowchart LR classDef future fill:#fff,stroke:#888,stroke-dasharray:5 5,color:#444 subgraph today[Today] - d1[table_store
opens lance::Dataset directly]:::now - d2[storage.rs
S3 / file URI plumbing]:::now + d1[sealed TableStorage
staged write primitives]:::now + d2[TableStore + snapshots
private Lance reads]:::now end - subgraph roadmap[Roadmap - storage capabilities] - t[trait Dataset
schema · stats · placement
capabilities · scan · write]:::future - impl1[LanceStorage]:::future - impl2[future test impl]:::future + subgraph roadmap[Roadmap - planner-facing capabilities] + t[read/capability surface
schema · stats · placement
pushdown support]:::future + impl1[Lance-backed implementation]:::future end today -.-> roadmap t --> impl1 - t --> impl2 ``` The staged-write trait exists today as `TableStorage`, implemented by @@ -263,7 +264,7 @@ flowchart LR classDef future fill:#fff,stroke:#888,stroke-dasharray:5 5,color:#444 subgraph today[Today] - ei[ensure_indices
omnigraph.rs:445]:::now + ei[ensure_indices]:::now manual[called manually
or from optimize]:::now end diff --git a/docs/dev/canon.md b/docs/dev/canon.md new file mode 100644 index 00000000..e6c27831 --- /dev/null +++ b/docs/dev/canon.md @@ -0,0 +1,936 @@ +# The OmniGraph Canon + +**Audience:** maintainers, contributors, and coding agents — internal +**Type:** narrative reference ("the book"), read top-to-bottom +**Status:** living document +**Surveyed:** OmniGraph 0.8.1 (`main`); Lance 9.0.0-beta.21 (git rev `1aec1465`); internal manifest schema v4 + +--- + +## What this document is (and is not) + +This is the linear narrative of OmniGraph: why it exists, what it is built on, +how a read, a write, a merge, and a crash actually unfold, what we deliberately +refuse to build, where the risks are, and where the design is going. Read it +start to finish to build the complete mental model; use it to onboard, to +re-anchor after time away, or to check a design instinct against the system's +reasoning. + +It is **not** the mechanical authority for any subsystem. Every section links to +the per-area doc that owns the details ([invariants.md](invariants.md), +[writes.md](writes.md), [execution.md](execution.md), …); when this document and +an area doc disagree, the area doc wins and this one gets fixed. It is also not +user documentation — that lives under [docs/user/](../user/index.md). + +Truth discipline: claims here are current shipped behavior unless explicitly +marked **(roadmap)**, **(draft RFC)**, or **(research-blocked)**. There is no +single "status disclaimer" that licenses aspirational prose elsewhere — each +claim carries its own status, per the maintenance contract's "don't lie" rule. + +--- + +## TL;DR + +**The problem.** Fleets of agents (and the humans supervising them) need shared +operational state that is simultaneously: a typed graph (entities and +relationships with schema), multi-modal retrievable (vector ANN + full-text + +graph traversal fused in one query), *reviewable* (isolated proposals, diffs, +merges — Git semantics over data, not files), time-travelable, and deployable on +plain object storage without a database server fleet. Existing systems give you +one or two of these. Gluing a graph database, a vector database, a search +engine, and an audit trail together means N consistency boundaries, N failure +modes, and no coherent notion of "the state of the graph at commit X." + +**The solution.** OmniGraph is a typed property-graph engine built as a +*coordination layer* over many [Lance](https://lance.org) datasets — one +columnar, versioned, branchable dataset per node/edge type — with one manifest +table (`__manifest`) making multi-dataset changes visible atomically. On top of +that substrate it adds: + +1. **Git-style branches and commits across the whole graph.** Every publish + appends to a commit DAG; branches fork lazily (copy-on-write via Lance); + merges are three-way at the row level with typed conflicts. Branches *are* + the multi-query transaction model — there is deliberately no cross-query + `BEGIN`/`COMMIT`. +2. **One unified write protocol (RFC-022, implemented).** Every graph-visible + write — mutation, bulk load, schema apply, branch merge, index build — + follows one state machine: prepare against a pinned authority token, arm a + durable recovery intent, apply exact physical effects, publish exactly one + manifest CAS. Crash anywhere and recovery converges all-or-nothing. +3. **Multi-modal querying in one runtime.** A `.gq` query can combine vector + `nearest`, BM25/FTS, Reciprocal Rank Fusion, property filters, and graph + traversal (`Expand`, anti-join) against one snapshot. +4. **A declared control plane.** A `.pg` schema language with migration + planning; a cluster directory (`cluster.yaml`) that declares graphs, + policies, and stored queries as code; Cedar policy enforced engine-wide on + every writer; an Axum HTTP server and a CLI over the same engine gate. + +Storage is any S3-compatible object store or local filesystem. The whole system +is a single process per server; correctness never depends on a coordinator +service. + +--- + +## Why OmniGraph — the gap it fills + +The design center is **context assembly and coordination for agent fleets**: +hundreds of writers proposing changes concurrently, every change attributable +and reviewable, retrieval spanning similarity + text + structure. Measured +against the systems you would otherwise compose: + +| You could use… | What's missing for this workload | +|---|---| +| A graph database (Neo4j-class, server-based) | No whole-graph branching/merge as a data-review workflow; storage not object-store-native; vector/FTS usually bolted on with separate consistency | +| An embedded analytics graph DB (Kùzu-class) | Same branching gap; single-process analytics focus rather than coordinated multi-writer state | +| A vector DB / LanceDB directly | Per-dataset versioning only — no *cross-dataset* atomic commit, no typed graph semantics, no traversal, no merge | +| Postgres + pgvector + AGE | Real transactions, but no git-style branch/merge of data, no object-store-native storage; multi-modal fusion is manual | +| Git/dolt-style versioned tables | Versioning without the retrieval runtime (ANN/FTS/traversal) or graph typing | + +(Positioning summary for internal orientation — verify specific competitor +claims before using them externally; they age.) + +The honest inverse — what those systems have that OmniGraph deliberately does +not — is catalogued in [What we deliberately exclude](#what-we-deliberately-exclude-and-why) +and [Known gaps](invariants.md#known-gaps). Chief among them: no cross-query +transactions (branches instead), single-writer-process support boundary for +destructive recovery (a distributed fence is roadmap), and a pinned pre-stable +Lance version. + +--- + +## Design philosophy + +These are the reasoning tools behind every section that follows. The normative +statements live in [invariants.md](invariants.md); this is the narrative form. + +### 1. Engineering is programming integrated over time + +The operative question for any change is *which option has the lower ongoing +liability* — not shorter now, not fastest to ship. Complexity must be earned by +demonstrated correctness, performance, or future-shape cost, never by +speculation. This cuts both ways: sometimes the lower-liability option is more +code (a centralized dispatcher instead of scattered hooks), sometimes less (no +migration framework until a concrete graph demands one — see the +[strand model](#schema-and-migration--the-strand-model)). Ask: *what does this +look like after five more changes like it?* + +### 2. Respect the substrate + +Lance owns columnar storage, per-dataset versioning, fragments, branches, +compaction, cleanup, and index primitives. DataFusion owns relational execution +where it fits. We do not build WALs, transaction managers, buffer pools, or +local clones of substrate behavior — and we *use* the substrate idiomatically: +long-lived handles, shared sessions, cheap freshness probes instead of +re-scans. Re-deriving per call what the substrate keeps warm is a substrate +violation even when no code is reimplemented. + +### 3. Logical contract over physical state + +Logical state is the contract; physical state — index coverage, fragment +layout, staged writes — is derived, rebuildable, possibly asynchronous. **A +physical operation must never fail a logical one.** Preconditions are checked +against logical state; physical reconciliation is idempotent and may lag. The +smell to watch for: a logical operation whose precondition is a physical fact +(an index's existence, a fragment count). Genuine logical conflicts still fail +loudly — the licence to lag covers convergence, not correctness. + +### 4. One source of truth, cheaply derived + +Lance and the manifest are the source of truth; everything else is a derived +view — held warm, refreshed by a cheap probe. Two failure modes are forbidden: +a *parallel copy* that can drift (divergence compounds), and *cold +re-derivation* on every call (cost grows with history instead of the working +set). The commit graph, the compiled catalog, the CSR topology index, and the +handle cache are all derived views engineered under this rule. + +### 5. Graph visibility is manifest-atomic + +Lance commits are per dataset. Graph-level atomicity is manufactured: one +`__manifest` update flips every touched sub-table version visible together. +No write path may make a subset of touched tables visible as a graph commit, +and any writer that can advance a Lance HEAD before manifest publish must carry +a durable recovery intent covering the gap. + +### 6. Branches are the transaction model + +Per-query writes are atomic at the manifest boundary. Anything larger — a batch +of related changes, an agent's proposal, a risky migration — is a branch: +isolated, diffable, mergeable, discardable. This replaces cross-query +`BEGIN`/`COMMIT` deliberately (deny-listed), because branches give the same +isolation with review, attribution, and history for free, and they compose with +the object-store substrate where long-lived interactive transactions do not. + +### 7. Strong consistency, loud failures + +Reads are snapshot-isolated; writes are durable before acknowledgement; branch +reads observe current committed state. Integrity violations (type errors, +missing endpoints, cardinality, uniqueness) fail *before* publish. OOM, +timeout, partial results, recovery, and conflicts are surfaced and typed, never +swallowed. Any eventual-consistency mode must be explicit, read-only, and +non-default (none ships today). + +### 8. Semantics are first-class structures + +Search modes, mutations, polymorphism, traversal, scores, and policy predicates +belong in typed AST/IR/planner structures — never smuggled through magic +strings, side tables, globals, or transport flags. Transport and auth stay at +the boundary: kernel crates know nothing of HTTP or bearer tokens. + +### 9. Correctness > simplicity > performance; reversibility shapes evidence + +Lexicographic: give up performance for simpler code, simplicity for correct +code, correctness never. And the evidence demanded of a change scales with its +reversibility: reversible changes wait for production evidence; irreversible +ones (substrate choice, on-disk format, consistency guarantees) earn an RFC, +because by the time production proves them wrong you've shipped years of +dependent code. + +### 10. Observable behavior is the contract (Hyrum's Law) + +Output ordering, error text, timestamp precision, default flags, latency +profiles — once shipped, someone depends on them. We don't expose what we won't +commit to, and we treat behavior changes found in substrate upgrades (e.g. a +BM25 stop-word change reordering ties) as contract events to pin in tests, not +incidental noise. + +--- + +## Architecture at a glance + +One process, layered: + +``` +CLI (omnigraph) HTTP Server (omnigraph-server: Axum + Cedar + admission) + │ │ + └─────────────┬──────────────┘ + ▼ + omnigraph-compiler Pest grammars (.pg / .gq), catalog, typecheck, + │ IR lowering, lint, migration planning — zero Lance dependency + ▼ + omnigraph (engine) exec (query/mutation/loader), MutationStaging, + │ GraphCoordinator/ManifestCoordinator, CommitGraph projection, + │ GraphIndex (CSR/CSC), merge, recovery, validate + ▼ + storage boundary sealed TableStorage (staged writes only) + + │ read-only snapshot facade + ▼ + Lance 9.x columnar Arrow, per-dataset versions/branches, + │ BTREE/FTS/vector indexes, merge_insert, compaction + ▼ + object store local FS · S3 · RustFS · MinIO · S3-compatible +``` + +Workspace crates: `omnigraph-compiler`, `omnigraph` (package name +`omnigraph-engine` — the directory and package names differ), +`omnigraph-policy`, `omnigraph-api-types` (shared wire DTOs), `omnigraph-cluster` +(control plane), `omnigraph-cli`, `omnigraph-server`. Full diagrams and code +paths: [architecture.md](architecture.md). + +Two structural boundaries deserve emphasis because everything else leans on +them: + +- **The compiler knows no Lance.** Schema and query semantics are decided in + typed structures before the engine binds them to storage. This is what makes + invariant 8 (semantics are first-class) enforceable rather than aspirational. +- **The write surface is closed by Rust visibility.** Raw storage, + handle-cache, and coordinator modules are crate-private; public snapshot + access is a read-only facade that does not expose Lance's raw scanner. A + defense-in-depth source guard (`tests/forbidden_apis.rs`) classifies every + public async engine method and exact-counts registered durable-call shapes, + so adding a writer is an explicit registry change, never an accidental call + site. + +--- + +## The substrate contract: Lance + +OmniGraph's relationship with Lance is a *contract*, managed like one: + +- **Pinned, audited versions.** The engine pins one Lance version + (currently 9.0.0-beta.21 via git rev, until 9.0.0 stable reaches crates.io). + Every bump gets a full alignment audit — all intervening upstream commits + reviewed, findings recorded in [lance.md](lance.md)'s dated audit stanzas. + History has justified the paranoia: audits have caught a default flip that + would have GC'd manifest-pinned versions (`auto_cleanup`), a row-id overlap + that corrupted filtered reads (lance#7444, temporarily fixed by a vendored + one-hunk pin), and behavioral changes in merge_insert, BTREE range bounds, + and BM25 scoring. +- **Surface guards as tripwires.** `tests/lance_surface_guards.rs` pins every + Lance API shape and behavior we depend on — compile-shape guards, runtime + behavior guards, and "this upstream bug is fixed" guards that turn red when + reality changes. A Lance bump runs this file first; a clean build is *not* a + clean alignment. +- **The L1/L2 split as a review tool.** Every capability is classified as L1 + (inherited from Lance: columnar storage, per-dataset versioning/branches, + index primitives, merge_insert, compaction) or L2 (added by OmniGraph: + typing, graph semantics, cross-dataset atomicity, graph-level branches and + lineage, the query runtime, policy, serving). The full matrix lives in + [AGENTS.md](../../AGENTS.md). When a proposal reimplements something in the + L1 column, it's deny-listed on sight. + +What Lance does **not** give us — and where OmniGraph's hardest engineering +lives — is anything *cross-dataset*: no multi-dataset atomic commit, no +conditional branch-ref create/delete, no caller-controlled transaction identity +for maintenance operations. The next three sections are the story of +manufacturing those guarantees on top. + +--- + +## Anatomy of a graph on disk + +A graph is one directory (or S3 prefix). Details: [storage.md](../user/concepts/storage.md). + +``` +graph-root/ + __manifest/ # the coordination table (a Lance dataset itself) + nodes/{fnv1a64-hex(TypeName)}/ # one Lance dataset per node type + edges/{fnv1a64-hex(EdgeName)}/ # one Lance dataset per edge type + _graph_commit_recoveries.lance/ # internal crash-recovery audit log + __recovery/{ulid}.json # transient recovery sidecars (empty at steady state) + _refs/branches/{name}.json # graph-level branch metadata +``` + +`__manifest` is the load-bearing object. Its rows describe, per branch, which +version of each sub-table is published (`table_version` rows, minus +tombstones), **and** — since internal schema v4 (RFC-013 Phase 7) — the graph +commit lineage itself: one immutable `graph_commit` row per commit (ULID id, +parents, merge parents, actor, timestamp) plus one mutable `graph_head:` +pointer per branch. Lineage rows are written *in the same merge-insert commit* +as the table-version rows, so a graph commit and its lineage land at one +manifest version atomically — there is no second write to fail between. The +in-memory `CommitGraph` is a pure projection of these rows (invariant 15 in +action; the former `_graph_commits.lance` tables are retired). + +Two mechanisms make concurrent publishes safe: + +- `__manifest.object_id` carries Lance's unenforced-primary-key annotation, so + the substrate's bloom-filter conflict resolver rejects two concurrent commits + landing the same row — **row-level CAS**. Without it, Lance's transparent + rebase would admit silent duplicates from racing publishers. +- Same-branch writers all touch the shared `graph_head:` row, so even + commits to *disjoint tables* contend there: one wins, the other retries and + re-parents inside the publisher's retry loop. This closes the + disjoint-table-fork race and yields a linear per-branch chain (pinned by the + N-writer convergence tests). + +The internal manifest schema is stamped +(`omnigraph:internal_schema_version`, currently v4) and **strict +single-version** — see [the strand model](#schema-and-migration--the-strand-model). + +--- + +## The life of a read + +Contract: **a query holds one snapshot for its lifetime** (invariant 3). It +never re-reads the branch head mid-query, so concurrent writes cannot leak in. + +The path (details: [execution.md](execution.md)): + +1. **Capture.** Resolve the target (branch or historical version) to a manifest + snapshot plus the compiled catalog. Capture happens under a short + process-local schema gate so snapshot and catalog are *coherent* — a + concurrently applying schema can't be observed half-published. Execution + then releases the gate and runs entirely on the captured pair. +2. **Compile.** Parse + typecheck the `.gq` against the catalog, lower to a + typed IR pipeline (`NodeScan`, `Filter`, `Expand`, `AntiJoin`, projections, + ordering). +3. **Topology, if needed.** If the pipeline traverses, build or fetch a CSR/CSC + `GraphIndex` **scoped to exactly the edge types the query touches** — + never the whole catalog. The cache key is each edge table's physical + identity `(table_key, version, branch, e_tag)`, so a lazy-fork branch whose + edge tables physically *are* main's reuses main's built index instead of + cold-scanning. +4. **Execute.** Scans push structured filters down to Lance (BTREE/FTS/vector + indexes accelerate what they cover; correctness never depends on coverage — + invariant 7). Multi-modal ops (`nearest`, `bm25`, `rrf`) run in the same + pipeline; RRF fans out sub-rankings and fuses by rank. + +**The cost model is a tested contract, not an aspiration.** The warm read path +was once O(commit-history) per query — fresh coordinator per read, full +manifest re-scans, no shared session. The query-latency work drove it to: one +cheap freshness probe, one schema read, zero dataset opens on a warm repeat. +This is *pinned* by IO-counted cost-budget tests (`warm_read_cost.rs`) that +count object-store operations at commit-history depth — because cost-scaling +bugs pass every correctness test and only bite in production. See +[testing.md](testing.md) "Cost-budget tests". + +--- + +## The life of a write + +This is the heart of the system. Read [writes.md](writes.md) for the mechanics +and [RFC-022](../rfcs/0022-unified-write-path.md) for the full protocol; this +is the story. + +### The problem being solved + +Lance has no multi-dataset atomic commit. A graph write touching three tables +makes three independent Lance commits plus one manifest publish — four durable +operations, any prefix of which can survive a crash. Meanwhile other writers +race on the same branch, schema state can change under a prepared plan, and a +first write to a branch may need to *create* the per-table Lance fork it is +writing to. The unified protocol makes all of this safe with one state machine: + +``` +recovery barrier # never write over an unresolved crash + → prepare pinned base + read set # capture (branch identity, exact graph head, + → stage effects (no HEAD moves) # schema identity, table pins); validate everything + → acquire ordered gates # schema → branch → sorted tables (process-local) + → revalidate the complete token # or restart / typed conflict — never rebase a stale plan + → arm durable recovery intent # __recovery/{ulid}.json, before any durable effect + → apply exact physical effects # commit_staged with pre-minted identities, zero retries + → publish ONE __manifest CAS # entire graph delta + lineage, exact precondition + → finalize (delete sidecar) +``` + +### Mutations and loads, concretely + +`mutate_as` and every `load` mode accumulate work in an in-memory +`MutationStaging` — inserts/updates as pending `RecordBatch`es, deletes as +predicates. **No Lance HEAD advances during statement execution.** Reads inside +the query union the committed snapshot with the pending batches (DataFusion +`MemTable`), so a multi-statement mutation gets read-your-writes: statement N+1 +sees statement N's inserts when validating referential integrity or +cardinality. + +The **D₂ rule** keeps this unambiguous: one mutation query is constructive +(insert/update) XOR destructive (delete), enforced at parse time. This is a +deliberate boundary, not scaffolding — allowing mixing would require an +in-query delete view, pending-batch pruning, and per-table two-commit ordering +in the hot path. Compose mixed work as two mutations, or a branch for one +atomic commit. + +At end-of-query, `stage_all` prepares exactly one staged Lance transaction per +touched table (append / merge-insert deduped by id / deletion-vector delete / +overwrite), still without moving HEAD. Then the gates are acquired, the full +authority token revalidated, the schema-v3 sidecar armed, tables committed with +their exact pre-minted transaction identities and **zero transparent conflict +retries**, and the pre-minted lineage published under the exact +native-branch/head + table-version precondition. + +Failure semantics are typed by *when* the failure happens: + +- **Before any effect** (validation failure, or authority changed): + the attempt is discarded whole. Insert-only mutations and Append/Merge loads + silently reprepare from fresh authority with a bounded retry; strict + Update/Delete/Overwrite (and branch merge) return `ReadSetChanged` (HTTP 409) + — a stale plan is never rebased onto a moved base. +- **After any effect**: any later error returns `RecoveryRequired` (HTTP 503) + and leaves the sidecar authoritative. This is deliberately *not* a retry + loop — the fixed outcome converges through recovery, preserving the + interrupted writer's exact commit identity and actor. + +A cancelled future leaves no graph-visible state: the accumulator evaporates, +and anything durable is sidecar-covered. + +### Validation is unified and Δ-scoped + +Value/enum constraints, uniqueness, edge referential integrity, and +cardinality route through **one** catalog-derived evaluator (`crate::validate`) +on all three write surfaces — mutation, load, and branch merge — so the +surfaces cannot drift. It checks the *delta* against a pinned committed view +(not the whole graph), uses a BTREE probe when the index is reconciled, and +stays correct by scanning while it is pending (invariant 7 again). + +### Writer-specific adapters + +"Unified" means one set of safety obligations, not one Lance primitive. Each +writer describes its physical effects to the shared coordinator: + +| Writer | Sidecar schema | Physical shape | +|---|---|---| +| Mutation / Load | v3 | one exact staged transaction per touched table | +| Branch merge | v4 | an *ordered chain* of exact transactions per table (append → upsert → delete), pointer-only deltas recorded too | +| Schema apply | v7 | exact `Overwrite` per rewritten table + strict read-version-zero `Create` per new type; plus the schema registration/tombstone delta (a metadata-only apply has an empty effect set but still arms — schema staging is durable state) | +| EnsureIndices | v8 | one pre-minted *mixed* CreateIndex transaction per table (every missing BTREE + FTS + full-table vector together) | +| Optimize | v2 (bounded) | compaction + index folds have **no** public caller-controlled Lance transaction identity, so Optimize keeps a looser, bounded envelope: one graph-wide sidecar pinning the complete productive set, one monotonic batch CAS for visibility. Exact provenance is trigger-gated on upstream API + distributed fencing | + +First-touch tables (a branch's first write to a table) follow +**sidecar-before-ref** ordering: the recovery intent that names the +`(table_path, target ref)` is durable *before* the Lance ref exists, so an +orphaned fork is always attributable and reclaimable, and reclaim/cleanup treat +any pending claim as a hard stop. + +### Gates, and what they are not + +Every handle for one canonical graph root shares a process-local +`WriteQueueManager`: schema gate → branch gate → sorted table gates, one +deadlock-free order used by writers, healers, and the recovery sweep alike. +These gates prevent same-process races and reduce publisher retries. **They are +not distributed locks.** Cross-process safety comes from the publisher's exact +CAS precondition (one winner; the loser gets a typed conflict) — and the honest +support boundary that follows from it is described in +[Concurrency and the support boundary](#concurrency-and-the-support-boundary). + +--- + +## The life of a crash + +Recovery is part of the commit protocol, not an afterthought (invariant 5). +Full mechanics: [writes.md](writes.md) "Open-time recovery sweep". + +The lifecycle every staged writer follows: **Phase A** write the sidecar +(before any independently durable effect) → **Phase B** apply per-table +effects, then atomically *confirm* the exact achieved versions/identities into +the sidecar → **Phase C** publish the manifest → **Phase D** delete the +sidecar. The Phase-B confirmation is the commit point of this recovery WAL: + +- Crash **before confirmation** → roll back: restore each table to its + manifest pin, then publish the restored state so `manifest == HEAD` converges + with *no residual drift* (this symmetry is what lets a failed-then-retried + operation succeed instead of failing one version higher each time). +- Crash **after confirmation** → roll forward, but only under the captured + authority token, and only through *exact* proof: the observed Lance + transaction UUIDs and versions must match the pre-minted plan. Roll-forward + publishes the interrupted writer's fixed lineage — original commit ID, + original actor — so history is what the writer intended, not a synthetic + recovery artifact. + +The exactness is the security property. Recovery never adopts a foreign commit +that happens to sit at the expected version; never reparents a prepared write +onto a newer branch winner; preserves a disjoint foreign winner while +compensating only owned effects; and **fails closed** when a foreign commit +buries an owned effect on the same table (restoring through someone else's +data is the one thing it will never do). Every completed action lands an +internal audit row in `_graph_commit_recoveries.lance` (no public CLI query for +it yet — a known gap). + +When recovery runs: + +- **Read-write open** runs the full all-or-nothing sweep. +- **Every write entry point and `refresh`** runs a roll-forward-only heal + in-process, so a long-lived server converges on its next write without a + restart. Rollback-requiring sidecars defer to the next read-write open + (a background reconciler for that residual is **(roadmap)**). +- **Read-only open** repairs nothing, but refuses to serve a torn + schema-apply state (fixed manifest outcome visible, schema identity not yet + live) rather than lying. + +Older sidecar formats (v5 SchemaApply, v6 EnsureIndices) remain readable +forever under their original, looser semantics — a v6 file is never +reinterpreted as a v8 ownership proof. + +Ahead-of-manifest drift *not* covered by any sidecar is never silently +adopted: writers refuse it and point at `omnigraph repair`, which classifies it +explicitly — verified maintenance drift (`ReserveFragments`/`Rewrite`) publishes +with `--confirm`; anything semantic requires `--force --confirm` after +deliberate review. + +--- + +## Branches, commits, and merge + +### Branch mechanics + +A graph branch is logically one row set in `__manifest` (`BranchContents` is +the single authority); physically, per-table Lance forks materialize **lazily** +on first write — a fresh branch costs O(1) regardless of graph size, and its +unwritten tables physically *are* the parent's (which is why the read path's +cache keys use physical identity, and why cross-branch index reuse is free). + +Branch create/delete are *control operations*, not graph commits: they emit no +lineage, and they deliberately have no recovery sidecar. Lance's native +create/delete are each physically two-phase with no conditional primitive, so +OmniGraph wraps them in authority-derived reconcilers: names are prevalidated, +live names are path-prefix-disjoint, an absent-ref clone-only tree is +reclaimed by the next same-name create, and delete removes authority before +best-effort tree cleanup (absent ref ⇒ success). This is invariant 3's shape +applied to metadata: when the logical target is fully derivable from one +existing authority, a reconciler beats a sidecar — and it degrades to a no-op +if Lance later closes the physical gap. + +### Commits and time travel + +Every publish appends one `graph_commit` row (ULID, parents, actor) and moves +`graph_head:` — atomically with the data, as described in +[Anatomy](#anatomy-of-a-graph-on-disk). Time travel (`snapshot_at_version`, +`entity_at`, historical queries, diffs via `diff_between`/`diff_commits`) reads +older manifest states; retention is governed by `cleanup` (see +[Maintenance](#maintenance-optimize-cleanup-repair)). Checkpoint-pinned +retention — named snapshots as authoritative retention roots — is +**(draft RFC-025)**. + +### Three-way merge + +`branch_merge` computes the merge base from captured commit IDs, classifies +row-by-row against immutable base/source/target snapshots (ordered cursor +merge, batched staging writer), and publishes one atomic manifest update. +Conflicts are typed (`DivergentInsert`, `DivergentUpdate`, `DeleteVsUpdate`, +`OrphanEdge`, plus constraint violations re-checked Δ-scoped at the merge +boundary) and surface as structured 409s. The merge-pair truth table test pins +all 81 `(left_op, right_op)` cells — adding an op forces a compile error until +the new row and column are dispositioned. + +The contract under concurrency: **"merge the captured source commit," never +"substitute whatever source is latest."** A source advance after capture is +fine; a target change is `ReadSetChanged`. Merge's recovery adapter (schema v4) +pre-mints each table's exact ordered transaction chain, so recovery proves a +contiguous prefix of *this merge's* commits rather than inferring ownership +from version arithmetic. + +Cost honesty: an append-only fast-forward routes new rows through cheap +appends (structurally pinned — the test asserts *which* staged primitive runs, +not a timing threshold), but a diverged merge still classifies full-width; +its `__manifest` open amplification still grows with history. O(delta) merge +via row-version lineage is **(research-blocked RFC-027)** — the deletion-delta +source doesn't exist yet — and fragment-adoption merge is **(draft RFC-0001)**. + +--- + +## Schema and migration — the strand model + +The `.pg` language declares node/edge types, interfaces, properties, +constraints (`@key`, `@unique`, `@card`, `@range`, `@check`, enums), and +physical intent (`@index`, `@embed`). The compiler produces a typed catalog; +a linter (`OG-XXX-NNN` codes) gates footguns. `schema plan` diffs the accepted +schema against the proposal and produces a migration plan; `schema apply` +executes it under the `__schema_apply_lock__` system branch, as a first-class +RFC-022 writer (v7 adapter — the fixed manifest outcome lands *before* schema +staging is promoted, and capture-time coherence means readers can never observe +the manifest-before-catalog window on a live handle). + +Applies are metadata-only wherever possible — adding an `@index` or widening an +enum bumps no table version. Destructive or narrowing changes are refused +(`OG-MF-106`) rather than lossy. + +**Storage versioning is strict single-version** (the strand model, +[versioning.md](versioning.md)): this binary reads exactly one internal +manifest schema (`MIN_SUPPORTED == CURRENT == 4`). An older graph is refused +with a self-service export/import rebuild recipe naming the right old release; +a newer graph is refused with "upgrade omnigraph". There is deliberately no +in-place migration dispatcher — that machinery is permanent liability (every +format change would carry a tested `vN→vN+1` walker plus legacy readers plus +crash-recovery paths, forever) for a pre-1.0 format. The stamp + +`refuse_if_stamp_unsupported` floor is exactly the seam that would re-introduce +it if a concrete graph ever demands it. Note the four version axes (release / +wire / storage / Lance) have deliberately different policies — conflating them +is how you ship a silent misread or carry migration code you don't need. + +Known gap worth internalizing: type IDs are still derived from `kind:name`, so +**rename-stable schema identity is not yet real** — don't build on renamed IDs +surviving across accepted schemas until draft +[RFC-028](../rfcs/0028-stable-schema-identity.md) is accepted and implemented. + +--- + +## The query engine + +The `.gq` language: named queries with parameters, `match` patterns over typed +nodes/edges (including interface polymorphism), `where` filters, `not { … }` +anti-joins, variable-length traversal, `order`/`limit`, aggregations, and the +search functions. Everything lowers to typed IR — there is no string-SQL +side-channel; filters push down to Lance as structured expressions. + +Multi-modal retrieval is the differentiating runtime capability: one query can +rank by `rrf(nearest($d.embedding, $q), bm25($d.body, $q_text))` — vector ANN +fused with BM25 by reciprocal rank — then expand graph structure from the +survivors, all against one snapshot. + +Traversal executes on the scoped CSR/CSC topology index by default, with a +BTREE-indexed `Expand` mode asserted semantically equivalent (property-based +tests generate adversarial graphs — cross-type ID collisions, cycles, +self-loops — so a future divergence between modes fails loudly). + +Current honesty (**roadmap** items, from [Known gaps](invariants.md#known-gaps)): +execution is lowering-ordered, not cost-based — planner capability/statistics +surfaces don't exist yet; multi-hop still uses `TypeIndex` and eager +materialization in places (stable row-IDs, SIP, factorization are target +patterns); rank/score don't yet propagate everywhere as ordinary columns; +Cedar predicates in the planner and a unified `Source` operator are design +directions, not code. + +--- + +## Derived state: indexes, embeddings, topology + +Invariant 7's family, all one shape — *declared intent, derived +materialization, correct reads at any coverage*: + +- **Physical indexes.** `@index`/`@key` declare intent; type dispatch picks the + kind (enum/orderable scalar → BTREE, free-text String → FTS, Vector → vector + ANN). Writes never build indexes inline — mutation/load/schema-apply publish + only their exact data effects. `ensure_indices` materializes every missing + artifact for a table in **one** staged mixed CreateIndex transaction (v8 + adapter); `optimize` separately folds new fragments into existing indexes. + Reads are correct under partial coverage (Lance unions indexed and scan + paths; vector search falls back to brute force). A background reconciler to + automate the explicit calls is **(roadmap)**. +- **Embeddings.** `@embed` records which text property seeds which vector and + with which model; the loader does *not* call an embedding API on the write + path (deny-listed — a network call in the commit path). Vectors arrive in + the load data or via the offline `omnigraph embed` pipeline; query-time + `nearest($v, "text")` auto-embeds the query string. Ingest-time embedding via + an `ensure_embeddings`-style reconciler — an embedding is derived state, same + class as an index — is **(draft RFC-015 / RFC-012 phase)**. +- **Topology.** The CSR/CSC graph index is built per query, scoped to + traversed edge types, cached by physical table identity, reused across + lazy-fork branches. Never persisted, never authoritative. + +--- + +## Maintenance: optimize, cleanup, repair + +- **`optimize`** compacts fragments and folds index coverage across all tables + with bounded parallelism and **one graph visibility envelope**: one bounded + v2 sidecar pins the complete productive set before any HEAD moves, and one + monotonic manifest CAS publishes everything together — two changed tables + become visible atomically, a no-work run leaves no trace. It also compacts + `__manifest` itself (physical-only, no graph commit), which is what keeps + write/read cost flat in history on a periodically-optimized graph. +- **`cleanup`** is version GC: explicit `--keep`/`--older-than` cutoffs derived + from Lance's actual version lists, floored by live lazy-branch inheritance + and recovery needs, refusing on pending sidecars or uncovered drift (GC must + never outrun the recovery barrier). Internal-table version GC is not yet + wired in **(deferred — needs the cleanup-resurrection watermark)**, so + `__manifest/_versions/` grows until explicit cleanup. +- **`repair`** is the human-in-the-loop path for uncovered drift, described in + [The life of a crash](#the-life-of-a-crash). + +None of these are background loops today — they are explicit operator/agent +calls (the cluster control plane and a future reconciler are the automation +story). + +--- + +## Serving: server, cluster, policy + +- **Engine-wide Cedar enforcement.** Every `_as` writer — mutation, load, + schema apply, branch create/delete/merge — calls + `Omnigraph::enforce(action, scope, actor)` inside the engine, so HTTP, CLI, + and embedded SDK callers hit the *same* gate. HTTP additionally resolves + bearer → actor and applies per-actor admission control before the engine. +- **Auth hygiene.** Bearer tokens are hashed (SHA-256) at startup — plaintext + never persists in process memory; comparison is constant-time; the actor ID + is server-resolved from the hash match and never client-settable. Kernel + crates have no HTTP/auth dependency (invariant 11). +- **Cluster-only boot (RFC-011).** The server always boots from a cluster + directory (`--cluster `) and serves N ≥ 1 graphs under + `/graphs/{id}/…`. The cluster directory (`cluster.yaml` + content-addressed + state ledger) declares graphs, schemas, stored queries, embedding providers, + and policies as code; `cluster apply` converges it (with digest-bound + approvals for destructive dispositions, sidecar-covered executors, and + crash-window failpoint coverage in `omnigraph-cluster`). Runtime add/remove + endpoints are deliberately absent: operators `cluster apply` and restart. +- **Two-surface operator config (RFC-007/008/011):** the team-owned cluster + directory plus per-operator `~/.omnigraph/config.yaml` (servers, credentials, + actor, profiles, aliases). The CLI's embedded and remote paths are held + equivalent by a parity-matrix test (every forked verb run against both arms; + divergences pinned in a ledger, never silently repaired). + +--- + +## Concurrency and the support boundary + +What is guaranteed, from strongest to most bounded: + +1. **Snapshot isolation per query** — always, any topology. +2. **Same-process concurrency** — fully arbitrated: shared root-scoped gates + order writers, healers, and recovery; readers are never gated; capture-time + coherence protects snapshot/catalog pairs. +3. **Cross-process, non-destructive** — safe by CAS: the publisher's exact + precondition admits one winner; losers get typed conflicts + (`ReadSetChanged`/`RecoveryRequired`), never silent rebase. Concurrent + multi-process writers on one graph are *functional* but documented as + one-winner-CAS territory, not a supported high-contention topology. +4. **Cross-process, destructive recovery** — **the documented boundary.** + Recovery's rollback (`Dataset::restore`) and first-touch reclaim are unsafe + beside a live writer in *another process*: Lance's restore silently orphans + concurrent commits (empirically pinned), Lance exposes no conditional + ref create/delete, and the gates are process-local. Exact v3/v4/v7/v8 + ownership prevents *false adoption* — recovery will never claim foreign + work — but it cannot *fence* a live foreign process. Closing this needs a + distributed fence (a lease on the schema-apply lock branch is the sketched + direction, **(roadmap)**; RFC-019's fencing direction now lives in + RFC-023/024). + +Multi-version binaries against one graph are refused by the storage stamp +(open- and publish-time checks); the residual read-only hole is a recorded +known gap, deliberately not defended because the topology itself is +unsupported. + +--- + +## How we test + +[testing.md](testing.md) is the map; these are the principles that make the +suite worth trusting: + +- **Extend before adding; test at the boundary being changed.** Planner + changes get planner-level tests; storage changes get storage/recovery tests; + end-to-end tests never substitute for missing lower-level assertions. +- **Cost budgets are tests.** IO-counted budgets at commit-history depth + (`warm_read_cost.rs`, `write_cost.rs`, `merge_cost.rs`) pin that hot-path + cost is bounded by working set, not history — because an O(commits) bug is + green in every correctness test. +- **Failure injection over hope.** The `failpoints` suites crash every writer + at every protocol boundary (post-stage, pre-effect, post-effect, + pre-publish, post-publish…) and assert exact recovery outcomes, including + the adversarial cells: foreign winners preserved not adopted, buried effects + failing closed, ABA on branch delete/recreate fenced. +- **Truth tables force disposition.** The 9×9 merge-pair matrix compiles-in + completeness: a new op variant fails the build until every combination is + dispositioned. +- **Guards pin the outside world.** Lance surface guards pin substrate + behavior; `forbidden_apis.rs` pins the closed write surface; + `failpoint_names_guard.rs` pins that no failpoint can silently never fire. + Red guards are *information* — several are deliberately built to turn red + when an upstream bug gets fixed, so the workaround gets removed. +- **Test-first bug fixes**, with the red commit landing immediately before the + green one so any reviewer can reproduce the failure from history. + +--- + +## What we deliberately exclude (and why) + +The normative deny-list is in [invariants.md](invariants.md); this is the +reader-facing rationale for the exclusions people actually ask about. None are +"not yet" — each is a decision with reasoning. (The burden on any exception is +on the proposer.) + +- **Cross-query `BEGIN`/`COMMIT` transactions.** Branches are strictly better + for this substrate and workload: same isolation, plus review, diff, + attribution, and history; no interactive lock lifetime held over object + storage. See [design principle 6](#6-branches-are-the-transaction-model). +- **A custom WAL / transaction manager / buffer pool.** Lance owns durability + primitives. Our recovery sidecars are *intents over Lance commits*, not a + parallel log of data. (Streaming ingest will consume Lance's MemWAL rather + than building one — **(draft RFC-026)**.) +- **Mixed constructive/destructive single mutations (D₂).** Keeps in-query + read-your-writes unambiguous and each table at one commit per query; the + alternative buys an in-query delete-view machine in the hot path. +- **Inline index/embedding builds on the write path.** Expensive derived state + converges from manifest state (reconciler shape); a network call or index + retrain in the commit path is deny-listed. +- **Placeholder nodes for orphan edges** and any silent integrity weakening. + Integrity failures are loud, pre-publish. +- **In-place storage migration.** The strand model — see + [Schema and migration](#schema-and-migration--the-strand-model). +- **Job queues for manifest-derivable state.** A reconciler that re-derives + from the source of truth is drift-proof; a queue is a second copy of intent. +- **Actor identity from clients.** The server resolves actor from the token + hash; a client-supplied actor would make the audit trail decorative. +- **Cloud-only correctness fixes / engine forks.** Correctness is always OSS; + Cloud extends by trait, never by fork. +- **Runtime graph add/remove endpoints.** The cluster directory is the source + of truth; converge it and restart — a mutable serving topology invites + drift between declared and actual state. + +--- + +## Current implementation status + +The always-current shipped-vs-roadmap ledger is +[invariants.md](invariants.md)'s **Current Truth Matrix** and **Known Gaps** — +this section is the orientation summary, not the authority. + +**Solid and shipped:** the unified write protocol with exact per-writer +recovery (RFC-022 implemented 2026-07-13, across PRs #343–#353); manifest-atomic +multi-table publish with lineage-in-CAS; branches/commits/time-travel/merge +with typed conflicts; the strand storage model (v4); multi-modal query runtime; +unified Δ-scoped validation; engine-wide Cedar; cluster control plane and +cluster-only serving; the warm read-path cost contract; sealed write surface; +the full failpoint/recovery test lattice. + +**Explicitly bounded:** Optimize's recovery envelope (bounded v2, exact +provenance trigger-gated on upstream API + distributed fencing); destructive +recovery's single-writer-process boundary; merge cost at divergence +(full-width classification). + +**Roadmap / draft:** stable schema identity and table incarnation **(RFC-028, +draft)**; substrate-native key-conflict fencing **(RFC-023, draft; +substrate probes landed, production routing unchanged)**; durable table heads / +heads format **(RFC-024, draft)**; checkpoint-pinned retention **(RFC-025, +draft)**; MemWAL streaming ingest **(RFC-026, draft)**; lineage-based merge +deltas **(RFC-027, research-blocked)**; background reconciler; planner +statistics/cost model; policy pushdown; ingest-time embeddings; per-query +resource budgets. + +--- + +## Risk register + +| Risk | Severity | Posture / mitigation | +|---|---|---| +| **R1: Destructive recovery beside a live foreign process.** Lance restore orphans concurrent commits; no conditional ref primitives; gates are process-local. | High | Documented single-writer-process support boundary; exact ownership prevents false adoption; distributed fence (lease on the schema-apply lock branch) is the sketched close **(roadmap)**. Do not promote multi-process write topologies before it exists. | +| **R2: Pre-stable Lance pin.** 9.0.0-beta.21 via git rev; betas have regressed mid-line before (blob reads broke in beta.13, fixed beta.15). Blocks crates.io publishing (v0.8.1 is binaries-only; v0.9.0 gated on 9.0.0 stable). | High | Full alignment audit per bump (all commits reviewed, findings in [lance.md](lance.md)); surface guards as first smoke check; `cargo test --workspace` as the alignment gate, never the build alone. | +| **R3: Keyed-write fencing rests on engine revalidation, not a substrate primitive.** Lance's key-filter behavior is route-dependent and directional (probed 2026-07-14). | Medium | Branch-head CAS serializes same-branch commits; Δ-scoped revalidation runs on reprepare; RFC-023 owns the substrate-native fence behind a fleet/format barrier; guards pin both conflict orders so an upstream symmetry change forces an audit. | +| **R4: Manifest fold cost grows with commit count.** Current-state resolution folds history. | Medium | `optimize` compacts internal tables (keeps periodically-optimized graphs flat — cost-gated every PR); durable head rows are the structural fix **(RFC-024)**; internal-table *cleanup* still deferred behind the resurrection watermark. | +| **R5: Rename-stable schema identity not yet real** (type IDs from `kind:name`). | Medium | Recorded known gap; RFC-028 owns graph-scoped IDs, incarnation, SchemaApply recovery, and strict rebuild. Do not build features assuming renamed IDs survive until it is accepted and implemented. | +| **R6: Merge cost at divergence** — full-width classification, history-growing manifest opens. | Medium | Fast-forward path structurally pinned; `merge_cost.rs` keeps the terms visible; O(delta) merge blocked on a real deletion-delta source **(RFC-027)**; fragment adoption **(RFC-0001, draft)**. | +| **R7: No streaming ingest** — per-branch write throughput is capped by the `graph_head` CAS rate; high-frequency small writes are wasteful. | Medium | Deliberate: the interactive path's guarantees come first. MemWAL-based ingest with durable per-row ack + graph-atomic folds is the design **(RFC-026)**. MemWAL is the strategic substrate, but beta.21's public initializer commits opaquely and shard provisioning is separate; activation waits for a public exact enrollment receipt plus reversible admission seal rather than using private APIs. | +| **R8: Some operations lack enforced memory/time budgets.** | Medium | Known gap; the merge memory blow-up class was closed structurally (fast-forward append routing); new long-running work must add explicit bounds rather than widen the gap. | +| **R9: Local-FS conditional-write emulation** (`write_text_if_match` check-then-act gap). | Low | All current callers sit behind the cluster lock protocol; S3 uses true conditional puts; close before admitting any lock-free caller. | +| **R10: Doc/spec drift as the system grows** — this document included. | Low | Maintenance contract (same-PR doc updates, `check-agents-md.sh` link CI, "don't lie" stale markers); this canon defers to area docs by construction. | + +--- + +## Open questions + +Live design questions, each owned by an RFC or a known gap — not a wishlist: + +1. **What fences a live foreign process?** Lease on `__schema_apply_lock__`, a + Lance conditional-ref primitive if upstream ships one, or an external lease + service? Owns the R1 close; prerequisite for exact Optimize provenance and + any multi-process write story. +2. **What makes deletion discovery sublinear?** RFC-027 is blocked precisely + here: a deleted row is absent from the target snapshot, so version columns + can't identify it, and `_row_last_updated_at_version` filtering is O(rows) + without a substrate index or change log. +3. **Which accepted capabilities co-release at the next rebuild boundary?** + RFC-028 provisionally owns the next format through stable identity; + RFC-023's fence, RFC-024's heads, RFC-025's retention, and RFC-026's stream + capability remain independently reviewable. Co-release can reduce operator + cutovers only after the combined initialization/recovery matrix passes; a + separate format release requires a separate export/init/load rebuild. +4. **What is the checkpoint/retention contract?** RFC-025: checkpoint rows as + logical authority, Lance tags as physical pins, and the asymmetric safe + ordering between them — plus how the Q8 resurrection watermark unlocks + internal-table cleanup. +5. **How does the background reconciler arrive without violating the recovery + model?** It must serialize with writers through the same gates and stay + roll-forward-only until the fence exists. +6. **When Lance 9.0.0 stabilizes**, what does the beta→stable alignment audit + surface, and does crates.io publishing (v0.9.0) unblock cleanly? + +--- + +## Roadmap — the RFC family + +The plan of record is the RFC-022…028 family (all under +[docs/rfcs/](../rfcs/README.md), maintainer design series, reviewed in the +[RFC-022–028 ledger](rfc-022-027-architecture-review.md)): + +| RFC | Owns | Status | +|---|---|---| +| [0022 — Unified graph-write protocol](../rfcs/0022-unified-write-path.md) | One correctness protocol for all graph-visible writes; per-writer effect adapters; synchronous recovery | **Implemented** (2026-07-13) | +| [0028 — Stable schema identity and table incarnation](../rfcs/0028-stable-schema-identity.md) | Graph-scoped rename-stable type/property IDs, table lifetimes, SchemaApply recovery, and the shared strict-rebuild prerequisite | Draft | +| [0023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | Substrate-native keyed-write fencing via Lance's unenforced-PK filter; fleet/format activation barrier | Draft | +| [0024 — Durable table heads](../rfcs/0024-durable-table-heads.md) | O(1) current-state resolution via materialized head rows; heads-format initialization and strict rebuild boundary | Draft | +| [0025 — Checkpoint-pinned retention](../rfcs/0025-checkpoint-retention.md) | Named checkpoints as authoritative retention roots, materialized as Lance tags | Draft | +| [0026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | Durability-first streaming writes: ack on WAL durability, asynchronous graph-atomic folds | Draft | +| [0027 — Lineage merge deltas](../rfcs/0027-lineage-merge-deltas.md) | O(delta) merge classification from row-version lineage | Research-blocked | + +Deliberately split, not one mega-format: identity, key fencing, head rows, +retention, ingest, and merge deltas are separate irreversible decisions with +different substrate gates and rollout barriers ("reversibility shapes evidence +demand"). +Release-wise: v0.8.1 ships as binaries only (the Lance git pin blocks +crates.io); v0.9.0 is gated on Lance 9.0.0 stable. + +--- + +## Maintaining this document + +- This canon is a **narrative over the area docs**, which stay authoritative. + When behavior changes, update the owning area doc in the same PR (per the + maintenance contract) and fix the corresponding narrative here; if you can't + rewrite a section fully, mark it `*(stale — needs update after )*`. +- Every claim is either current-shipped or carries an explicit status marker. + Keep it that way — one top-of-file disclaimer does not license aspirational + prose below it. +- Update the **Surveyed** line, the status ledger, the risk register, and the + RFC table when the underlying facts move (release, Lance bump, RFC status + change, gap opened/closed). +- Structural changes (new sections) should keep the reading order a *story*: + substrate → disk → read → write → crash → branch/merge → schema → query → + derived state → operations → boundaries → status → future. diff --git a/docs/dev/index.md b/docs/dev/index.md index 5311456f..d45beebc 100644 --- a/docs/dev/index.md +++ b/docs/dev/index.md @@ -7,6 +7,11 @@ invariants, implementation contracts, test ownership, and upstream Lance constraints. User-facing behavior should still be documented through [docs/user/index.md](../user/index.md) and the relevant public reference docs. +New to the codebase, or re-anchoring after time away? Read +[canon.md](canon.md) — the linear narrative of the whole system (why it +exists, how a read/write/merge/crash unfolds, deliberate exclusions, risks, +roadmap). The per-area docs below remain the mechanical authority. + ## Required For Every Non-Trivial Change | Need | Read | @@ -58,11 +63,14 @@ constraints. User-facing behavior should still be documented through |---|---| | How to contribute (external) | [CONTRIBUTING.md](../../CONTRIBUTING.md) | | Governance model, roles, decision authority | [GOVERNANCE.md](../../GOVERNANCE.md) | -| Public contribution RFC track | [rfcs/](../rfcs/) | +| RFC process — public contribution and maintainer design-series tracks | [rfcs/](../rfcs/) | -The `docs/rfcs/` track is the **public, externally-authorable** RFC process. The -maintainer/internal RFCs below (`rfc-00N-*.md`) are a separate, team-owned -track; don't conflate the two. +`docs/rfcs/` is the durable home for both formal tracks. Every formal RFC uses +the four-digit `NNNN-*` filename namespace; explicit `Author track` and `Status` +metadata select the public contribution or maintainer design-series lifecycle. +Public RFC merge means acceptance, while maintainer design series may remain +draft across merged revisions. Existing `docs/dev/rfc-00N-*` files are legacy +internal design and implementation records that retain their current lifecycle. ## Case Studies @@ -81,7 +89,7 @@ as durable disposition history after closure, so RFC backlinks stay valid. | Area | Read | |---|---| -| RFC-022–027 split architecture review — open correctness blockers, dependency corrections, and required acceptance evidence | [rfc-022-027-architecture-review.md](rfc-022-027-architecture-review.md) | +| RFC-022–028 split architecture review — RFC-022 implemented; sibling blockers, dependency corrections, and acceptance evidence remain tracked | [rfc-022-027-architecture-review.md](rfc-022-027-architecture-review.md) | ## Active Implementation Plans diff --git a/docs/dev/invariants.md b/docs/dev/invariants.md index 7adfdc08..11cbc082 100644 --- a/docs/dev/invariants.md +++ b/docs/dev/invariants.md @@ -21,23 +21,28 @@ The hard invariants below are instances of one rule. Keep it in view whenever a change touches the boundary between what the graph *means* and how it is physically stored. -> **Logical state is the contract. Physical state — index coverage, fragment -> layout, compaction versions, staged writes — is derived, rebuildable, and may -> be produced asynchronously. A physical operation must never fail a logical -> one. Preconditions are checked against logical state; physical reconciliation -> is idempotent and may lag or retry. Genuine logical conflicts still fail -> loudly: the licence to lag covers physical convergence, not correctness.** - -Invariants that instantiate it: **2** (manifest-atomic visibility) and **5** +> **Accepted schema and manifest state define the logical contract. Derived +> physical state — fragment layout, index coverage, compaction output, and +> caches — may lag, retry, or be rebuilt. Pending physical state — including an +> unpublished Lance HEAD, staged effects, or, once streaming is activated, +> acknowledged MemWAL data — is not graph-visible, but must remain durable and +> recoverable until publication or a contract-defined terminal disposition; +> compensation must preserve every acknowledgement guarantee. Neither category +> may silently change logical correctness. +> Genuine logical conflicts still fail loudly: the licence to lag covers +> physical convergence, not correctness.** + +Invariants that instantiate it: **2** (one graph-content publication door) and **5** (recovery is part of the commit protocol) — a partially-written physical layer -never changes what a graph commit means; **7** (indexes are derived state) — a -query is correct under partial index coverage, and expensive index work -converges from manifest state instead of gating the write path; **13** (failures -bounded and observable) — the licence to lag is not a licence to drop, so a -physical step that cannot make progress is surfaced, not swallowed. Deny-list -items that enforce it: synchronous inline vector/FTS index rebuilds on the -commit path; state that drifts from Lance or the manifest when it can be -derived; job queues for manifest-derivable state where a reconciler fits. +never changes what a graph commit means; **7** (physical acceleration and +layout are derived state) — a query is correct under partial index coverage, +and expensive index work converges from manifest state instead of gating the +write path; and **13** (failures are bounded, typed, and observable) — the +licence to lag is not a licence to drop, so a physical step that cannot make +progress is surfaced, not swallowed. Deny-list items that enforce it include +synchronous inline vector/FTS index rebuilds on the commit path, state that +drifts from Lance or the manifest when it can be derived, and job queues for +manifest-derivable state where a reconciler fits. The failure shape it rules out: a legitimate background operation on the physical layer (compaction, an index build, an interrupted staged write) is @@ -49,141 +54,122 @@ converge the physical state. ## Hard Invariants -1. **Respect the substrate.** Lance owns columnar storage, per-dataset - versioning, fragments, branches, compaction, cleanup, and index primitives. - DataFusion should own relational execution where it fits. Do not add custom - WALs, transaction managers, buffer pools, page formats, or local clones of - substrate behavior. Read [lance.md](lance.md) before guessing. Respecting the - substrate also means *using* it idiomatically, not only refraining from - rebuilding it: reuse long-lived handles instead of re-opening per call, - resolve latest state through the substrate's cheap primitive instead of - re-scanning, and share its caches/session. Re-deriving per call what the - substrate keeps warm is a substrate violation even when no code is - reimplemented. - -2. **Graph visibility is manifest-atomic.** Lance commits are per dataset. - OmniGraph's graph-level atomicity comes from publishing one manifest update - for the whole graph, guarded by expected table versions and sidecar recovery. - No write path may make a subset of touched node/edge tables visible as a - graph commit. - -3. **A query reads one snapshot.** Query execution captures a manifest snapshot - for its lifetime. Do not re-read branch head mid-query to discover newer - table versions. - -4. **Mutations publish at one boundary.** A `mutate_as` or `load` operation - accumulates writes — inserts/updates as pending batches, deletes as - predicates — stages each touched table at the end (deletes via - `stage_delete`, no inline HEAD advance), then publishes one manifest update. - Do not commit per statement. The parse-time D2 rule is a deliberate - boundary: one mutation query is constructive (insert/update) or destructive - (delete), not both — so read-your-writes within a query stays unambiguous - and each table commits at most one version. Compose mixed operations via - separate mutations, or a branch for single-commit atomicity. - Read [writes.md](writes.md) and [execution.md](execution.md). - -5. **Recovery is part of the commit protocol.** Writers that can advance Lance - HEAD before manifest publish must write `__recovery/{ulid}.json` sidecars. - Under their final schema → branch → table gates, they must first prove every - existing physical target still equals its manifest pin; a new sidecar must - never claim an older writer's effect or uncovered drift. First-touch refs use - sidecar-before-ref ordering. A non-noop SchemaApply writes a sidecar even - with zero table pins because schema-contract staging is durable state. Its - schema-v7 sidecar captures native main authority + accepted schema identity, - fixed actor-bearing lineage, exact overwrite/first-touch transaction - identities, and the complete registration/update/tombstone delta. `Armed` - rolls back; only an exact `EffectsConfirmed` outcome can roll forward, and - schema staging is promoted only after that fixed manifest outcome is visible. - Owned first-touch paths are reclaimed on rollback. An unregistered foreign - first-touch winner is preserved but never adopted; foreign movement on a - manifest-owned table or an owned effect buried by a same-table winner fails - closed. Query, export, graph-index, and blob-read capture joins the same - process-local schema gate, keeping snapshot + catalog coherent across live - publication. Read-only open does not repair, but refuses a fixed SchemaApply - manifest outcome until the matching schema identity is live. Schema-v5 files - remain readable under their original bridge semantics. EnsureIndices uses - the same exact boundary in schema v8: one pre-minted mixed CreateIndex - transaction per touched table, captured branch/schema authority, fixed - lineage and manifest delta, and exact first-touch ref ownership. Schema-v6 - EnsureIndices files remain readable under their original loose bridge - semantics and are never reinterpreted as v8 ownership proofs. - `Omnigraph::open` in read-write mode runs the all-or-nothing sweep; the - write entry points (`load_as`, `mutate_as`, `apply_schema_as`, - `branch_merge_as`) and `refresh` run roll-forward-only recovery in-process, - so a long-lived process converges on its next write rather than at restart. - Rust visibility closes the supported SDK surface: raw storage, handle-cache, - and graph/manifest coordinator modules are crate-private. Public snapshot - access is read-only and its scan facade does not expose Lance's raw scanner - or physical plan. The defense-in-depth registry in - `tests/forbidden_apis.rs` classifies public async inherent `Omnigraph` and - loader surfaces, every crate-visible async coordinator method, and exact - per-file occurrences of registered durable-call shapes, including recovery. - Do not add a new writer kind without registering sidecar coverage or an - explicit exception whose ordering proof shows why no independently durable - graph-visible effect can escape the coordinator. +1. **Respect the substrate.** Lance owns storage, per-dataset versions, + branches and transactions, fragments, indexes, compaction, cleanup + primitives, and MemWAL; DataFusion owns relational execution where it fits. + Adopt public substrate primitives once their contracts pass the required + evidence gates. Do not clone them or reach through private APIs. Use them + idiomatically: share sessions and caches, reuse long-lived handles, and + prefer a cheap substrate probe over reopening or rescanning. Read + [lance.md](lance.md) before guessing. + +2. **There is one graph-content publication door.** Every graph-content, + accepted-schema, and maintenance-pointer transition becomes authoritative at + one atomic `__manifest` publish. Those transitions use the shared publisher + and recovery protocol; writer-specific physical-effect adapters are allowed, + but alternate graph-visibility paths and per-table graph publication are not. + Native branch-ref create/delete is the control-plane exception: it uses the + documented authority-derived protocol, with Lance `BranchContents` as the + sole logical authority, and never exposes partial per-table state. Lance HEADs + carrying data or schema effects may move earlier only under durable recovery + ownership sufficient for the operation's documented support boundary. + +3. **Each operation uses one coherent accepted view.** A read uses one + immutable manifest snapshot for its lifetime. A graph-content or schema + publication attempt captures every schema, catalog, base-snapshot, and + authority fact it needs from one accepted view, then revalidates the complete + token before effects. Control and maintenance paths likewise capture and + revalidate their complete logical authority token at the documented + boundary. A retry starts a new attempt; it never combines a fresh head with + stale planning state. + +4. **A mutation publishes once.** `mutate`, `load`, and equivalent + multi-statement operations stage every participant and publish once; they do + not commit per statement. The parse-time D2 rule remains the explicit + constructive-versus-destructive boundary. Compose mixed operations through + separate mutations, or use a branch when one later merge commit must make + them visible together. See [writes.md](writes.md) and + [execution.md](execution.md). + +5. **Recovery is part of the commit protocol.** Any independently durable data + or schema effect that can later become graph-visible requires durable + recovery ownership sufficient for the operation's documented support + boundary. Authority-derived metadata residue may use a reconciler only when + it cannot be adopted as graph state and its target is fully derivable from + existing authority. Before proceeding, a writer resolves or refuses every + recovery intent relevant to its authority or effects; it never replans around + unresolved partial state. Recovery may roll forward or compensate only when + its identity and authority proof is sufficient for that boundary; ambiguity + fails closed. A new writer kind must join the shared protocol or carry an + explicit no-escape and ordering proof. See [writes.md](writes.md) for adapter + schemas, gate ordering, legacy readers, and the current process boundary. 6. **Strong consistency is the default.** Reads are snapshot-isolated, writes - are durable before acknowledgement, and branch reads observe the current - committed graph state. Any eventual-consistency mode must be explicit, - read-only, auditable, and non-default. - -7. **Indexes are derived state.** Reads must see the correct result for the - branch they read even when index coverage is partial. Expensive index work - should converge from manifest state instead of extending the critical write - path. BTREE, FTS, and full-table vector artifacts stage together as one - CreateIndex transaction per touched table; Optimize's index-coverage fold is - a separate bounded schema-v2 maintenance adapter. See [writes.md](writes.md) and - [indexes.md](../user/search/indexes.md). - -8. **Schema identity survives renames.** Accepted schema identity must remain - stable across type and property renames. Rename support belongs in migration - planning, not in "drop and recreate" behavior. See the known gap below. - -9. **Schema/data integrity failures are loud.** Type errors, required-field - misses, invalid edge endpoints, cardinality violations, and unsupported - mixed mutation modes fail before a graph commit is published. The system must - not invent placeholder nodes or silently weaken integrity. + are durable before acknowledgement, and normal reads observe + manifest-committed state. Any weaker mode must be explicit, read-only, + auditable, and non-default. A stream-admission acknowledgement is a distinct + durability contract, not acknowledgement of a graph-visible write; it follows + [RFC-026](../rfcs/0026-memwal-streaming-ingest.md) and never weakens + manifest-visible reads. + +7. **Physical acceleration and layout are derived state.** Indexes, + graph-topology structures, fragment layout, and similar physical structures + may lag or rebuild. Missing or partial coverage must not make a logically + valid operation incorrect. Expensive reconciliation converges from manifest + authority outside the critical write path. See + [indexes.md](../user/search/indexes.md) and [writes.md](writes.md). + +8. **Schema identity survives renames.** Accepted type and property identities + remain stable across supported renames. Drop/re-add creates a new logical + lifetime. Names, paths, Lance versions, and Lance field IDs are not + substitutes for graph-level identity. See the known gap below and + [RFC-028](../rfcs/0028-stable-schema-identity.md). + +9. **Schema and data integrity failures are loud.** Type, required-field, + uniqueness, endpoint, cardinality, schema, and mutation-mode violations fail + before publication. The engine never invents placeholders, weakens + constraints, or silently accepts ambiguous state. 10. **Query semantics are first-class IR concepts.** Search modes, mutations, - polymorphism, traversal, retrieval scores, imports, and policy predicates - belong in typed AST/IR/planner structures. Do not smuggle semantics through - strings, side tables, global state, or transport-specific flags. - -11. **Transport/auth stay at the boundary.** Kernel crates should not depend on - HTTP, OpenAPI, bearer-token parsing, or future transport protocols. The - server resolves bearer tokens to actors; clients cannot set actor identity - directly. - -12. **Bearer-token plaintext is not retained.** Server startup hashes bearer - tokens, authentication uses constant-time comparison, and request handling - carries only the resolved actor identity and hash-derived match state. - -13. **Operational failures are bounded and observable.** Timeout, memory, OOM, - partial result, recovery, and conflict paths must fail loudly or degrade in - a documented way. If a metric affects plan choice or operator behavior, it - must be exposed through the relevant trait or observability surface. - -14. **Tests match the boundary being changed.** Prefer extending the existing - test that owns the area. Planner changes need planner-level coverage, - storage changes need storage/recovery coverage, and end-to-end tests are not - a substitute for missing lower-level assertions. Read [testing.md](testing.md) - before adding tests. - -15. **One source of truth, cheaply derived.** Lance and the manifest are the - source of truth. Everything the engine needs at runtime is a derived view of - them: read or projected on demand, held warm, refreshed by a cheap probe. Two - failure modes are forbidden. A *parallel copy* the engine maintains can drift - from the source, and that divergence compounds over time. *Cold - re-derivation* rebuilds the view from the full source on every call, so its - cost grows with history. Invariants 1 and 7, and the deny-list "state that - drifts" and "manifest-derivable reconciler" items, are instances; so is - bounding a read's cost to its working set rather than the commit count. This - is the structural face of "engineering is programming integrated over time": - both failure modes are liabilities that compound as the system grows. At a - write/control boundary, schema identity, compiled catalog, and manifest - authority must come from one operation-local accepted view: validating a - fresh on-disk marker while planning or enumerating gates from a stale warm - catalog is not a coherent derivation. + traversal, polymorphism, retrieval scores, imports, and policy predicates + belong in typed AST, IR, and planner structures. Do not smuggle semantics + through strings, side tables, global state, or transport-specific flags. + +11. **Transport and authentication stay at the boundary.** Kernel crates remain + independent of HTTP, OpenAPI, bearer-token parsing, and deployment-specific + protocols. The trusted server boundary resolves actors; HTTP clients cannot + choose the server-resolved actor identity. + +12. **Bearer-token plaintext is not retained.** Server startup hashes tokens, + authentication uses constant-time comparison, and request execution carries + only resolved identity and hash-derived match state. + +13. **Failures are bounded, typed, and observable.** Conflict, recovery, + timeout, memory, OOM, partial-result, and backpressure paths must have + explicit outcomes and enforced bounds where process survival permits. + Nothing is dropped, auto-merged, or degraded silently. Permitted retries are + defined by the operation's contract and bounded; failures are never + swallowed. Metrics and capabilities must be exposed before planning or + operator behavior depends on them. + +14. **Evidence matches the boundary and reversibility.** Tests exercise the + layer whose contract changed; end-to-end coverage does not replace missing + lower-level assertions. Irreversible substrate, on-disk format, protocol, + or database-guarantee changes require an RFC plus the applicable + compatibility, refusal, recovery, crash, and rebuild evidence. Performance + or complexity claims require a checked-in instrument and representative + measurements; a claim without an instrument does not count. Prefer + extending the existing test owner described in [testing.md](testing.md). + +15. **One source of truth, cheaply derived.** The persisted accepted schema + contract, Lance, and `__manifest` are authoritative. Immutable, + version-pinned state may be cached. Mutable-tip projections may remain warm + only to locate or hint at authority; they are never the commit's source of + current truth. Every commit reads or arbitrates durable authority. A parallel + maintained copy that can drift and cold full-history reconstruction on every + call are both forbidden. Runtime cost should scale with the working set, not + accumulated history; every known exception remains an explicit, instrumented + gap. ## Current Truth Matrix @@ -192,9 +178,10 @@ converge the physical state. | Multi-table commit | Manifest CAS plus recovery sidecars; not a single Lance primitive | [writes.md](writes.md), [architecture.md](architecture.md) | | Constructive mutations | In-memory `MutationStaging`, one end-of-query table commit per touched table, then one manifest publish | [writes.md](writes.md), [execution.md](execution.md) | | Deletes | Staged like inserts/updates (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, MR-A) — no inline HEAD advance; mixed insert/update/delete in one query rejected by D2 as a deliberate boundary (constructive XOR destructive per query; compose via separate mutations or a branch) | [query-language.md](../user/queries/index.md), [writes.md](writes.md) | +| Streaming ingest | Not implemented yet. RFC-026 adopts Lance MemWAL as the strategic substrate; activation waits on its stated public API and evidence gates. Once active, append acknowledgement means MemWAL-durable, while graph visibility still occurs only through graph-atomic RFC-022 folds | [RFC-026](../rfcs/0026-memwal-streaming-ingest.md), [writes.md](writes.md) | | Branch create/delete | `__manifest` `BranchContents` is the single logical authority. Lance create is physically two-phase, so OmniGraph prevalidates names, enforces path-prefix-disjoint live graph names, reclaims an absent-ref clone-only tree, and uses a bounded completion classifier; delete removes authority before tree cleanup, so an absent ref is success and derived tree reclaim may converge later. Neither control emits graph lineage. Per-table forks are derived state, reclaimed best-effort with `cleanup` as backstop. Under schema/target/all-table gates, a target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Cleanup retention | Explicit cleanup derives exact `keep` cutoffs from Lance's available version list, caps each main-table GC cutoff at the oldest exact main version inherited by any live lazy graph branch, and refuses uncovered main HEAD drift. Lance protects native per-table branch refs itself. The graph-wide live-reference preflight fails closed before the first table GC, after which individual table failures remain fault-isolated | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | -| Optimize visibility | One operation-local accepted catalog and fresh main snapshot are planned under schema → main → all-table gates. Every productive table shares one bounded schema-v2 recovery sidecar; bounded-parallel physical effects become visible through at most one monotonic manifest/lineage CAS. Complete crash residuals roll forward together and partial residuals compensate before visibility. The adapter is supported within the single-writer-process recovery boundary. Exact provenance is deferred until Lance exposes a stable public caller-controlled maintenance transaction API and OmniGraph has distributed recovery fencing | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md), [RFC-022](../rfcs/rfc-022-unified-write-path.md) | +| Optimize visibility | One operation-local accepted catalog and fresh main snapshot are planned under schema → main → all-table gates. Every productive table shares one bounded schema-v2 recovery sidecar; bounded-parallel physical effects become visible through at most one monotonic manifest/lineage CAS. Complete crash residuals roll forward together and partial residuals compensate before visibility. The adapter is supported within the single-writer-process recovery boundary. Exact provenance is deferred until Lance exposes a stable public caller-controlled maintenance transaction API and OmniGraph has distributed recovery fencing | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md), [RFC-022](../rfcs/0022-unified-write-path.md) | | Schema validation | Type checks, required fields, defaults, edge endpoint checks, and edge cardinality are enforced on write paths | [schema-language.md](../user/schema/index.md), [execution.md](execution.md) | | Unique constraints | Value/enum, uniqueness, edge-RI, and cardinality route through ONE unified, catalog-derived evaluator (`crate::validate`) on ALL THREE write surfaces — branch-merge, mutation, and bulk load: Δ-scoped (checks the delta, not the whole graph) and structured-filter-backed (committed probes use a BTREE when reconciled and remain correct by scanning while it is pending), reusing the leaf checks (`loader::validate_value_constraints`/`validate_enum_constraints`/`composite_unique_key`) so the surfaces cannot drift. This closed the prior merge bug (merge validated `@range`/`@check` but not enum) AND the **cross-version uniqueness gap** on the mutation and load paths (a duplicate of a committed `@unique` value is now rejected; the merge path always enforced it). The committed view is the merge target snapshot (merge), the write's pinned `txn.base` (mutation), or the pinned pre-load base (load — `Overwrite` validates the batch as the whole new image, committed view empty); `@card` refreshes the manifest-visible graph-branch snapshot on the mutation path only (the #298 stale-handle fix), then follows each entry's actual inherited/owned Lance ref. `@key` is id-backed, so it is checked intra-delta only (a committed holder of a key value is always the same row — an upsert), skipping a wasted O(Δ) probe per keyed row; `@unique` (non-key) groups do the committed lookup. | [schema-language.md](../user/schema/index.md) | | Storage trait | `TableStorage` (via `db.storage()`) is sealed and staged-only. `stage_create_indices` builds a typed mixed BTREE/FTS/full-table-vector batch without moving HEAD, and `commit_staged{,_exact}` is the only publication boundary; `InlineCommitResidual` and `storage_inline_residual()` are removed. Capability/stat surfaces remain roadmap | [writes.md](writes.md), [architecture.md](architecture.md) | @@ -207,10 +194,11 @@ The branch-control reconcilers are authority-derived: same-name create reclaims an absent-ref clone-only manifest tree, and delete/cleanup reclaim orphaned per-table forks. They degrade to no-ops if Lance closes the corresponding physical gaps, so the design composes with that future rather than blocking it. -This is the same shape as invariant 7 (indexes are derived state); prefer it -over a recovery-sidecar-style approach for metadata work whose logical target -is fully derivable from one existing authority. A graph-visible table effect is -different and still requires durable ownership before it can advance HEAD. +This is the same shape as invariant 7 (physical acceleration and layout are +derived state); prefer it over a recovery-sidecar-style approach for metadata +work whose logical target is fully derivable from one existing authority. A +graph-visible table effect is different and still requires durable ownership +before it can advance HEAD. For legacy path-prefix overlaps, an ancestor first-touch tree is not proven unreachable while a live child remains. Full recovery may leave the sidecar intact and allow the graph to open for leaf-first child deletion **only when the @@ -228,7 +216,10 @@ them explicit. - **Rename-stable schema identity:** the invariant is that accepted IDs survive renames. The current compiler still derives type IDs from `kind:name`; this - must be fixed before relying on renamed IDs across accepted schemas. + must be fixed before relying on renamed IDs across accepted schemas. Draft + [RFC-028](../rfcs/0028-stable-schema-identity.md) now owns the identity, + incarnation, allocation, and strict-rebuild design; the gap remains open + until that RFC is accepted and implemented. - **Storage abstraction:** `TableStorage` is present, sealed, and canonical for staged writes. MR-854 made `db.storage()` staged-only; the exact EnsureIndices adapter then migrated beta.21's full-table vector shape into the typed mixed @@ -369,12 +360,20 @@ them explicit. and factorization are target patterns, not current fact. - **Retrieval ranks:** hybrid search works, but rank/score are not yet carried everywhere as ordinary columns through the plan. -- **Policy pushdown and `Source`:** Cedar enforcement is at the HTTP boundary - today, and imports are still loader-shaped. Planner predicates and a unified - `Source` operator are roadmap. +- **Policy pushdown and `Source`:** Cedar enforcement is engine-wide at writer + entry points, but policy-predicate pushdown is not implemented and imports are + still loader-shaped. Planner predicates and a unified `Source` operator are + roadmap. - **Resource bounds:** some operations still lack enforced per-query memory or time budgets. New long-running work should add explicit bounds rather than widening the gap. +- **Branch-merge manifest history amplification:** branch merge still performs + multiple cold cross-branch coordinator opens, so `__manifest` reads grow with + commit depth on an uncompacted graph. The checked-in + `merge_cost.rs::merge_manifest_cost_grows_with_history` instrument pins this + as a known non-flat term rather than disguising it as an acceptance result. + This remains an explicit invariant-15 gap; do not claim history-flat merge + cost until the same instrument proves it. - **Read-path re-derivation (largely closed by the query-latency work):** snapshot resolution used to re-open a fresh coordinator per read (a full `__manifest` re-scan plus the then-separate commit-graph-table scans, since @@ -447,11 +446,14 @@ case is exceptional. - Per-table graph publishing outside the manifest publisher. - Re-reading current branch head during a query instead of using the captured snapshot. -- New write paths that can advance Lance HEAD before manifest publish without a - recovery sidecar. +- New write or control paths with independently durable pre-manifest effects and + neither durable recovery ownership nor an explicit no-escape, + authority-derived reconciliation proof. - Cross-query `BEGIN`/`COMMIT` transactions in the OSS engine. Use branches and merges for multi-query workflows. -- Acknowledging writes before durable Lance and manifest persistence. +- Acknowledging a graph-visible write before its Lance and manifest persistence, + or, once streaming ingest exists, acknowledging an append before its MemWAL + durability barrier. - Silent fallback to eventual consistency, partial results, or dropped rows. - State that drifts from Lance or the manifest when it can be derived. - Job queues for manifest-derivable state where a reconciler is the right shape. @@ -461,6 +463,11 @@ case is exceptional. flags, or out-of-band metadata. - Cost-blind plan choice when statistics are available or required. - Hidden statistics for behavior that affects planning or operator choice. +- An irreversible substrate, on-disk format, protocol, or database-guarantee + change without an RFC and the applicable compatibility, refusal, recovery, + crash, and rebuild evidence. +- A performance or complexity claim without a checked-in instrument and + representative measurements at the claimed scale. - Hash-map iteration order in result ordering, plan choice, or migration output. - Cold re-derivation on the hot path: rebuilding from the full source what could be held warm and refreshed cheaply, so cost scales with history rather than the @@ -489,11 +496,18 @@ case is exceptional. Use this as yes/no/NA for any non-trivial design or PR: -- Does it respect Lance/DataFusion instead of rebuilding them? -- Does it preserve manifest-atomic graph visibility? -- Does every query keep one snapshot for its lifetime? +- Does it use the public Lance/DataFusion substrate instead of rebuilding or + reaching through it? +- Does every graph-content, schema, and maintenance-pointer transition use the + one `__manifest` publication door, and do native branch controls derive + visibility only from Lance `BranchContents` authority? +- Does each operation derive every snapshot/schema/catalog/authority fact from + one coherent accepted view? - Do mutations publish once at the commit boundary? -- Can every Lance-HEAD-before-manifest gap recover all-or-nothing? +- Does the writer resolve or refuse every relevant recovery intent? Does each + independently durable pre-manifest data/schema effect have recovery ownership, + or each authority-derived residue have an explicit no-escape reconciliation + proof? - Are schema and edge integrity checks strict by default? - Are query semantics represented in AST/IR/planner structures? - Are transport, auth, and policy boundaries preserved? @@ -501,9 +515,16 @@ Use this as yes/no/NA for any non-trivial design or PR: - Are result ordering and plan choices deterministic within a snapshot? - Are stats/capabilities exposed when behavior depends on them? - Are existing known gaps left no worse and documented if touched? -- Does the test live at the same boundary as the change? +- Does the evidence live at the same boundary as the change? +- If the change is irreversible, does it have an RFC and the applicable + compatibility, refusal, recovery, crash, and rebuild proof? +- If it makes a performance or complexity claim, is the instrument checked in + and is the claimed scale measured? - Is this operation's cost bounded with respect to history and scale, or does it re-derive warm state from cold storage per call? +- Is mutable-current runtime state only a non-authoritative hint, never the + commit's source of current truth, with the commit revalidating against or + arbitrating through durable authority? - Does the change avoid every deny-list pattern, or justify the exception? ## Maintenance Policy diff --git a/docs/dev/lance.md b/docs/dev/lance.md index ee3739f5..17a66d58 100644 --- a/docs/dev/lance.md +++ b/docs/dev/lance.md @@ -197,6 +197,21 @@ Behavior-affecting findings in this audit: maintenance-transaction API and OmniGraph has distributed recovery fencing; the latter is independently required before destructive recovery is safe against a live foreign process. +- **RFC-023 key-filter behavior remains route-dependent and directional.** A + 2026-07-14 follow-up probe on this same beta.21 pin shows that an explicitly + selected v2 merge plan (`use_index(false)`) over the exact unenforced PK emits + `Some(KeyExistenceFilter)`: a fresh insert and fresh-key + `WhenMatched::Fail` populate the Bloom filter, while a matched-only + partial-schema UpdateAll + DoNothing emits a semantically empty filter rather + than `None`. A mismatched ON set emits `None`; when all ON columns have scalar + indexes and `use_index` remains enabled, Lance selects legacy v1, which also + emits `None`. Conflict resolution is still directional: filtered-current + conflicts with committed unfiltered Update or Append, but current unfiltered + Update or Append can rebase after a committed filtered Update. The RFC-023 + guards pin both orders so a future symmetry fix forces an alignment audit. + This is substrate evidence only: OmniGraph has not changed production + routing, removed keyed Append, installed PK metadata graph-wide, or crossed a + fencing-compatible fleet/format barrier. - **Index construction gained correctness and bounded-resource fixes:** beta.17 prevents an FTS builder thread-pool deadlock and bounds tail-partition merge memory; beta.18 fixes a streaming IVF training hang; beta.19 caps nullable @@ -240,9 +255,11 @@ Behavior-affecting findings in this audit: improving the warm-access shape without changing branch identity or commit semantics. -The existing Lance surface guards plus the canonical workspace and failpoint -suites are the compatibility gate for this pin. Keep the beta.15 audit below as -historical provenance for the larger 7.0 → 9.0 migration. +The Lance surface guards, including the RFC-023 route and conflict-order probes, +plus the canonical workspace and failpoint suites are the compatibility gate +for this pin. Keep the beta.15 audit below as historical provenance for the +larger 7.0 → 9.0 migration. A substrate guard records current Lance truth; it +does not by itself satisfy an RFC activation gate. ### Prior alignment audit: 2026-07-05 (Lance 9.0.0-beta.15 upstream; omnigraph pinned at 9.0.0-beta.15 via git rev) diff --git a/docs/dev/rfc-011-cli-refactoring.md b/docs/dev/rfc-011-cli-refactoring.md index d26dd84d..bbaee976 100644 --- a/docs/dev/rfc-011-cli-refactoring.md +++ b/docs/dev/rfc-011-cli-refactoring.md @@ -514,7 +514,8 @@ deferred (see below); they do not block the model. policy-gated, audited, single-coordinator — with `direct` retained only as break-glass (`repair` when the server is down). Runs out-of-band (a worker + async job routes, the `POST …` / `GET …/{id}` shape of the bulk-data-plane RFC - (`docs/rfcs/0001-bulk-data-plane.md`, PR #219, not yet merged)), never inline in + proposed in [closed PR #219](https://github.com/ModernRelay/omnigraph/pull/219), + which never landed in `docs/rfcs/`), never inline in serving; `schema plan` is excluded (≈ `cluster plan` in cluster mode). The **mechanism** (job routes, worker, scheduling) is a follow-up RFC; until it lands the capability table above @@ -571,9 +572,10 @@ keyed credentials; RFC-008 demoted `omnigraph.yaml`; RFC-009 unified execution behind `GraphClient`; RFC-010 declared the planes. This RFC removes the last legacy addressing surface so the plane model becomes a clean function of the three real entities, and folds the planes into a single capability rule. It is adjacent -to the public-track bulk-data-plane RFC (`docs/rfcs/0001-bulk-data-plane.md`, -PR #219, not yet merged), which canonicalizes `load`/`export` verbs; this RFC -canonicalizes how every verb *addresses* a graph. +to the bulk-data-plane proposal in +[closed PR #219](https://github.com/ModernRelay/omnigraph/pull/219), which never +landed in the RFC directory and would have canonicalized `load`/`export` verbs; +this RFC canonicalizes how every verb *addresses* a graph. ## Appendix: target CLI taxonomy (end state) diff --git a/docs/dev/rfc-022-027-architecture-review.md b/docs/dev/rfc-022-027-architecture-review.md index e77d389a..d5218df3 100644 --- a/docs/dev/rfc-022-027-architecture-review.md +++ b/docs/dev/rfc-022-027-architecture-review.md @@ -1,6 +1,6 @@ -# Architecture review: RFC-022 through RFC-027 +# Architecture review: RFC-022 through RFC-028 -**Status:** Open review findings +**Status:** RFC-022 implemented; RFC-023–028 review and acceptance work remains open **Date:** 2026-07-11 **Audience:** RFC authors, engine/storage maintainers, and release reviewers **Reviewed against:** OmniGraph 0.8.1; Lance 9.0.0-beta.15 at @@ -8,9 +8,23 @@ branch/tag, MemWAL, row-lineage, read/write, compaction, and cleanup specifications; pinned Rust implementation where the public specification is not precise enough +**RFC-022 close-out revalidated against:** Lance 9.0.0-beta.21 at +`1aec14652dcbace23ac277fa8ced35000bea0c40`; see the +[2026-07-12 alignment audit](lance.md#last-alignment-audit-2026-07-12-lance-900-beta21-upstream-omnigraph-pinned-at-900-beta21-via-git-rev). +**RFC-023 substrate evidence revalidated against:** the same beta.21 revision; +filter-shape and conflict-order probes are recorded in RFC-023 §2 and do not +constitute fencing activation or acceptance. +**RFC-026 substrate contract revalidated against:** the same beta.21 revision; +the full MemWAL table and system-index specifications, including the +durability/fencing behavior carried forward from beta.17, are reflected in the +RFC's survey and acceptance guards. +**RFC-024/025/027 substrate claims revalidated against:** the same beta.21 +revision and the full matching table/index/branch/tag/cleanup, transaction, and +row-lineage specification sections; their survey headers now record beta.21. This is a review ledger, not a competing specification. The RFCs remain the -normative proposals. A finding is closed only when the affected RFC either: +normative design records: RFC-022 documents the implemented protocol and its +siblings remain proposals. A finding is closed only when the affected RFC either: 1. incorporates the required contract and tests; or 2. explicitly rejects the finding with evidence and records the resulting @@ -31,17 +45,19 @@ The central architecture remains the right one: - Lance is the physical storage/versioning truth. - `__manifest` is the graph-visible authority and one graph publish is the visibility point for a logical graph commit. -- RFC-022 should define one correctness protocol with explicit physical-effect - adapters, not pretend that Lance offers one universal staged primitive. -- Key fencing, durable heads, retention, MemWAL, and lineage acceleration are - separate irreversible decisions and should keep separate evidence and format - gates. +- RFC-022 defines one correctness protocol with explicit physical-effect + adapters rather than pretending that Lance offers one universal staged + primitive. +- Stable schema identity, key fencing, durable heads, retention, MemWAL, and + lineage acceleration are separate irreversible decisions and should keep + separate evidence and format gates. The remaining open blockers are boundary problems, not a reason to replace that -core. They concern actual cross-process fencing, capability activation, format -rollout, and public RFC lifecycle. The native branch crash classifier, -foreign-source merge semantics, and shipped lazy-branch cleanup exposure found -by this review are now dispositioned below. +core. They concern sibling capability activation, strict rebuild rollout, +cross-process claims, and acceptance evidence. RFC-022's structural +rollout and the RFC-track classification are now closed. The native branch crash +classifier, foreign-source merge semantics, and shipped lazy-branch cleanup +exposure found by this review are also dispositioned below. > 💬 **Second-pass verification (2026-07-11):** I independently re-verified this > ledger's two sharpest new substrate claims against the pinned checkout before @@ -75,30 +91,33 @@ The intended dependency shape is therefore: ```text RFC-022 unified write protocol | - +-- stable identity/incarnation capability - | +-- RFC-024 durable heads - | +-- RFC-025 checkpoint rows - | `-- RFC-026 stream/reject authority - | - +-- RFC-023 key fencing - | `-- RFC-026 keyed streaming mode - | - +-- RFC-025 retention - +-- RFC-026 MemWAL - `-- RFC-027 merge-delta research + `-- RFC-028 stable identity/incarnation capability + +-- RFC-023 key fencing + | `-- RFC-026 keyed streaming mode + +-- RFC-024 durable heads + +-- RFC-025 checkpoint rows + `-- RFC-026 stream/reject authority ``` -The public RFC process accepts whole RFCs, not independently accepted sections. -The identity capability must therefore be extracted into its own RFC, or -RFC-024 must be accepted in full before identity-dependent siblings. An -implementation may phase an accepted RFC internally, but it cannot treat a -sub-contract of an unaccepted RFC as a separately accepted decision. +RFC-023 through RFC-026 also consume RFC-022's write/recovery protocol +directly; RFC-025 additionally owns retention, RFC-026 owns MemWAL, and RFC-027 +depends on RFC-022's branch-merge pipeline without consuming RFC-028 yet. The +diagram highlights the identity dependency rather than duplicating every direct +protocol edge. + +The maintainer design-series process permits incremental draft merges, but it +does not make a sub-contract of an unaccepted RFC independently authoritative. +RFC-028 now owns the extracted identity capability. RFC-023 through RFC-026 +depend on that shared contract and cannot activate their identity-bearing +formats until RFC-028 is accepted and implemented. An implementation may phase +an accepted RFC internally, but it cannot silently promote one section of a +draft into a shared format contract. ## Blocking findings ### BLOCKER-01 — native graph-branch control is multi-phase -**Affected:** [RFC-022 §7](../rfcs/rfc-022-unified-write-path.md#7-native-graph-branch-ref-control-protocol) +**Affected:** [RFC-022 §7](../rfcs/0022-unified-write-path.md#7-native-graph-branch-ref-control-protocol) **Status:** Closed in specification and implementation on 2026-07-11. @@ -164,14 +183,16 @@ and between delete authority removal and directory cleanup. ### BLOCKER-02 — create-if-absent ownership is not a fencing lease -**Affected:** RFC-023 migration claim, RFC-024 migration claim, and -[RFC-025 §2.3](../rfcs/rfc-025-checkpoint-retention.md#23-cross-process-retention-claim) +**Affected:** [RFC-025 §2.3](../rfcs/0025-checkpoint-retention.md#23-cross-process-retention-claim) + +**Status:** Closed in specification on 2026-07-14; subprocess death proof and +offline-repair implementation remain RFC-025 acceptance gates. `PutMode::Create` correctly elects one initial owner. A random token in that -object does not by itself fence a paused owner, because Lance tag, branch, -migration, and cleanup effects do not condition their commits on that token. -RFC-025 proposes a check-then-delete release; RFC-023 does not yet specify a -release protocol at all. For the check-then-delete shape, the race is: +object does not by itself fence a paused owner, because Lance tag, branch, and +cleanup effects do not condition their commits on that token. The reviewed +RFC-025 proposed a check-then-delete release plus takeover. For that shape, the +race is: 1. old owner reads and verifies token A; 2. old owner pauses; @@ -206,14 +227,21 @@ that takeover is refused until external fencing/death proof is established. > uses explicit offline repair. The substrate-enforced lease is the right end > state but should be written as an upstream/adapter work item, not assumed. +**Disposition:** RFC-025 now calls the value an owner token, makes a crashed +claim non-takeoverable, and refuses stale-claim removal until an operator proves +the prior process/host cannot resume or runs offline repair after revoking that +ability. A fleet outage, elapsed time, and token recheck are explicitly +insufficient. Acceptance tests pause a live owner and require takeover refusal, +then exercise removal only after subprocess death/offline isolation. + ### BLOCKER-03 — key-conflict retry must first resolve partial effects -**Affected:** [RFC-022 §9](../rfcs/rfc-022-unified-write-path.md#9-concurrency-and-retry-semantics), -[RFC-023 §6](../rfcs/rfc-023-key-conflict-fencing.md#6-retry-and-validation-semantics), -and [RFC-026 §6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol) +**Affected:** [RFC-022 §9](../rfcs/0022-unified-write-path.md#9-concurrency-and-retry-semantics), +[RFC-023 §6](../rfcs/0023-key-conflict-fencing.md#6-retry-and-validation-semantics), +and [RFC-026 §6](../rfcs/0026-memwal-streaming-ingest.md#6-fold-protocol) -**Status:** The RFC-022 mutation/load portion is closed in the shipped adapter -as of 2026-07-11. RFC-023 and RFC-026 retain their own open dispositions. +**Status:** Closed in specification on 2026-07-14. RFC-022 is shipped; +RFC-023/RFC-026 failpoint evidence remains an acceptance gate for those drafts. A retryable concurrent inserted-row-filter conflict is detected when a table transaction attempts to commit. (`WhenMatched::Fail` may instead report a @@ -232,9 +260,9 @@ forbids. before a new semantic attempt can be prepared. If rollback is not safe, return a typed recovery-required failure rather than retrying around it. -For strict insert, finalize an empty intent or retain a partial-effect sidecar, -as applicable, and return the terminal `KeyConflict`; RFC-022 permits partial -recovery to finish at the next synchronous barrier. Do not start another +For strict insert, finalize a proven-empty intent and return the terminal +`KeyConflict`; if an earlier participant advanced or emptiness is not provable, +retain the sidecar and return `RecoveryRequired` instead. Do not start another attempt around that sidecar. For non-strict upsert, the barrier must resolve the sidecar before the bounded automatic whole-operation retry. Add a failpoint/race in which table N conflicts after table 1 has committed. @@ -251,9 +279,17 @@ in which table N conflicts after table 1 has committed. > for safety. RFC-023's fenced key-conflict contract and RFC-026's MemWAL fold > protocol still need their adapter-specific resolutions. +**RFC-023/RFC-026 disposition:** RFC-023 §6 now permits a bounded whole-operation +retry only before sidecar arm or after exact classification proves the armed +intent effect-free and finalizes it. Any landed or unclassifiable participant +retains the sidecar and returns `RecoveryRequired`; strict insert applies the +same recovery precedence and never changes mode. RFC-026 §6 applies that rule +to folds and forbids replanning `merged_generations` around unresolved state. +Both RFCs require the table-N-after-table-1 failpoint before acceptance. + ### BLOCKER-04 — live graph branches need physical GC protection -**Affected:** [RFC-025 §6](../rfcs/rfc-025-checkpoint-retention.md#6-cleanup-protocol-and-pruned-through-boundary) +**Affected:** [RFC-025 §6](../rfcs/0025-checkpoint-retention.md#6-cleanup-protocol-and-pruned-through-boundary) **Status:** Closed in the shipped baseline and RFC-025 on 2026-07-11. @@ -303,7 +339,10 @@ runs, and the lazy branch still opens and reads the exact pinned state. ### BLOCKER-05 — durable-head OCC must compare the full head token -**Affected:** [RFC-024 §5](../rfcs/rfc-024-durable-table-heads.md#5-atomic-publisher-contract) +**Affected:** [RFC-024 §5](../rfcs/0024-durable-table-heads.md#5-atomic-publisher-contract) + +**Status:** Closed in specification on 2026-07-14; backend-token proof and ABA +tests remain RFC-024 acceptance gates. The RFC says expected table versions are compared against table heads. Lance version numbers can repeat across recreated datasets/branches, so version-only @@ -314,24 +353,36 @@ identity, at least: ```text (state, stable_table_id, incarnation_id, table_path, - table_branch, physical_ref_incarnation, table_version, schema_hash) + table_branch, physical_ref_incarnation, table_version, schema_ir_hash) ``` The publisher may encode that token differently, but retry/revalidation must not reduce it to `table_version`. `physical_ref_incarnation` means an e_tag when the backend supplies one, or another proven token that changes when a dataset or native ref is deleted and recreated at the same path/branch/version. This is -distinct from the logical `incarnation_id`, which RFC-024 deliberately preserves +distinct from the logical `incarnation_id`, which RFC-028 deliberately preserves across an owner handoff. Add stale-writer races for both logical drop/recreate and physical ref recreation at the same numeric version. +**Disposition:** RFC-024 §3.2 now persists the separate physical-ref token and +refuses activation on a backend without a proven source. §5 compares the full +token on every retry and derives the desired token from the achieved effect; +§12 adds local and S3/RustFS same-version ABA tests. Its open gates retain the +backend proof rather than silently accepting a timestamp/version fallback. + ### BLOCKER-06 — MemWAL needs a capability/format activation barrier -**Affected:** [RFC-026 §3](../rfcs/rfc-026-memwal-streaming-ingest.md#3-enrollment-is-a-recoverable-inline-commit), -[§5](../rfcs/rfc-026-memwal-streaming-ingest.md#5-ack-path-validation-and-writer-lifecycle), -[§6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol), -[§8](../rfcs/rfc-026-memwal-streaming-ingest.md#8-epoch-fenced-quiescence-barrier), -and [§12](../rfcs/rfc-026-memwal-streaming-ingest.md#12-phasing) +**Affected:** [RFC-026 §3](../rfcs/0026-memwal-streaming-ingest.md#3-enrollment-is-a-recoverable-multi-effect-adapter), +[§5](../rfcs/0026-memwal-streaming-ingest.md#5-ack-path-validation-and-writer-lifecycle), +[§6](../rfcs/0026-memwal-streaming-ingest.md#6-fold-protocol), +[§8](../rfcs/0026-memwal-streaming-ingest.md#8-epoch-fenced-quiescence-barrier), +[§9](../rfcs/0026-memwal-streaming-ingest.md#9-fresh-read-cuts), +and [§11–13](../rfcs/0026-memwal-streaming-ingest.md#11-format-activation-and-rebuild) + +**Status:** Open. The format/refusal portion is specified, but beta.21 lacks the +public exact enrollment and reversible shard-admission surface needed by the +recovery/quiescence adapter. RFC-026 cannot be accepted or activated until that +API gate and its crash evidence close. Enrollment creates persistent MemWAL metadata and `stream_state` changes the correctness preconditions for schema, branch, maintenance, and data operations. @@ -339,15 +390,35 @@ An older binary that understands key fencing but not stream lifecycle can ignore `OPEN | DRAINING | SEALED`, mutate the base table, or perform a schema/branch operation without draining acknowledged rows. -**Required disposition:** define: +**Format disposition:** RFC-026 §11 makes streaming a graph-format capability present +at new-root initialization, never a feature stamp added by first enrollment. +It defines: - a graph capability/internal-format stamp written only after every enrolled table and lifecycle authority is valid; - old-binary/new-format and new-binary/partial-format refusal behavior; -- migration and rollback/roll-forward ordering; +- strict export/init/load rebuild and source/target failure isolation; - preservation rules for later heads/retention formats; - genuine cross-version tests, not a stamp-rewind simulation. +RFC-026 §12 carries those tests and §13 places them in Phase A before the public +stream path. RFC-026 §3/§8 also make a rematerialized table a separate physical +enrollment boundary: SchemaApply leaves the preserved logical identity +`SEALED`, and only an exact sidecar-covered rebind with a fresh enrollment and +shard namespace may publish `OPEN`. Recovery never infers adoption from the +logical pair, name, path, or a compatible-looking index. + +**Remaining required disposition:** in beta.21, +`InitializeMemWalBuilder::execute` internally commits the `CreateIndex` +transaction and returns only `Result<()>`; `mem_wal_writer` separately creates +or claims shard-manifest objects. That cannot satisfy a sidecar that pre-arms +one exact combined effect. RFC-026 §3 now rejects private-module/object-store +emulation and requires a public caller-controlled transaction plus idempotent +shard provision/classify/seal/reopen/reclaim APIs, or one recoverable enrollment +receipt covering both effects. §5/§8 require the physical admission seal that +closes the final lifecycle-check-to-put race. The same-path/ref ABA, multi- +effect crash, and paused-claimant tests remain red acceptance gates. + Replica scope must also match RFC-023's recovery support boundary. Multiple replicas may route acknowledgement traffic to one shard owner, but enrollment, fold, and sidecar recovery remain single-writer-process operations until @@ -356,24 +427,40 @@ failover for those operations merely because MemWAL has a shard epoch. ### BLOCKER-07 — stable identity ownership contradicts sibling dependencies -**Affected:** [RFC-024 §3.1](../rfcs/rfc-024-durable-table-heads.md#31-identity-and-object-key), -[RFC-025 §2.1](../rfcs/rfc-025-checkpoint-retention.md#21-logical-authority-manifest-rows), -[RFC-026 §7](../rfcs/rfc-026-memwal-streaming-ingest.md#7-fold-time-rejection-is-atomic), -and [RFC-026 §8](../rfcs/rfc-026-memwal-streaming-ingest.md#8-epoch-fenced-quiescence-barrier) - -RFC-024 now says it exclusively owns stable table identity and incarnation and -that identity-dependent sibling claims must wait for it. RFC-025 still calls -RFC-024 optional, and RFC-026 persists `stable-table-id` without declaring the -dependency. - -**Required disposition:** extract the identity/incarnation format and migration -as a shared prerequisite RFC, or accept RFC-024 in full before RFC-025 and -RFC-026. Durable-head implementation can remain a later phase only after its -owning RFC and format contract are accepted. - -The `_ingest_rejects` deterministic key must use stable table ID plus -incarnation rather than mutable `table_key`, or rename/recreate can change reject -identity and break replay idempotence. +**Affected:** [RFC-023 §4.1](../rfcs/0023-key-conflict-fencing.md#41-keyed-graph-tables), +[RFC-024 §3.1](../rfcs/0024-durable-table-heads.md#31-identity-and-object-key), +[RFC-025 §2.1](../rfcs/0025-checkpoint-retention.md#21-logical-authority-manifest-rows), +[RFC-026 §2](../rfcs/0026-memwal-streaming-ingest.md#2-stream-mode-and-key-semantics), +[RFC-026 §7](../rfcs/0026-memwal-streaming-ingest.md#7-fold-time-rejection-is-atomic), +and [RFC-026 §8](../rfcs/0026-memwal-streaming-ingest.md#8-epoch-fenced-quiescence-barrier), +with the shared contract in [RFC-028](../rfcs/0028-stable-schema-identity.md) + +**Status:** Closed in specification on 2026-07-14; RFC-028 acceptance and +implementation remain prerequisites for every identity-consuming sibling. + +The reviewed drafts had RFC-024 exclusively owning stable table identity and +incarnation while RFC-025 called heads optional and RFC-026 persisted a stable +table ID without declaring that dependency. + +**Disposition:** RFC-028 now exclusively owns graph-scoped type/property IDs, +table incarnation, allocation, SchemaApply recovery, and strict rebuild. RFC-023 +through RFC-026 declare it as the prerequisite; RFC-024 owns only durable-head +storage and lookup. RFC-023 uses the stable pair for keyed-table format +authority, and RFC-026 distinguishes that preserved logical pair from its +separately fenced physical MemWAL enrollment. + +RFC-026's `_ingest_rejects` deterministic key now uses stable table ID plus +incarnation rather than mutable `table_key`, while a rematerializing rename +mints a fresh shard namespace under an exact sidecar-covered rebind. Thus +logical history ownership survives rename, physical WAL artifacts are never +adopted by implication, and drop/re-add cannot claim a predecessor's rows. + +The same correction keeps logical identity separate from physical ABA proof. +RFC-025 now owns a heads-independent exact manifest/table version token, +backend refusal, recovery/reconciler checks, and local/S3 same-coordinate ABA +tests. RFC-026 owns a binding-stable physical-ref incarnation token and remains +blocked under BLOCKER-06 until its backend/API guards pass. Neither silently +imports RFC-024's head storage merely to obtain those proofs. > 💬 **Concur; supersedes a 2026-07-11 correction (2026-07-11):** the earlier > pass placed identity ownership *inside* RFC-024 §3.1, which created exactly @@ -385,46 +472,62 @@ identity and break replay idempotence. ### BLOCKER-08 — retention activation cannot precede migration selection -**Affected:** [RFC-025 §8](../rfcs/rfc-025-checkpoint-retention.md#8-migration-and-compatibility) -and [§11](../rfcs/rfc-025-checkpoint-retention.md#11-phasing) - -RFC-025 leaves in-place migration versus export/import undecided. Its phase -table nevertheless puts “format activation” in Phase A and “selected migration -path” in Phase D. Activation cannot be implemented or reviewed before the -upgrade contract is known. - -**Required disposition:** select the migration before Phase A, then specify -quiescence, partial-state refusal, main-stamp ordering, branch coverage, crash -recovery, and later capability preservation. If export/import is selected, -state the irreversible loss of branches, commit DAG, snapshots, and historical -checkpoints as part of the acceptance decision. - -### BLOCKER-09 — decide whether these are public or internal RFCs - -**Affected:** [public RFC process](../rfcs/README.md), RFC-022 through RFC-027 +**Affected:** [RFC-025 §8](../rfcs/0025-checkpoint-retention.md#8-format-activation-and-rebuild-compatibility) +and [§11](../rfcs/0025-checkpoint-retention.md#11-phasing) -The files currently live in the public RFC track but use internal-style -`rfc-022-*` names, noncanonical `draft`/`research-blocked` statuses, and empty -owner metadata. In the public process, merging an RFC is acceptance; a merged -public RFC cannot simultaneously retain undispositioned acceptance blockers. +**Status:** Closed in specification on 2026-07-14; rebuild implementation and +cross-version evidence remain RFC-025 acceptance gates. -**Required disposition:** choose the track before merge: +The reviewed draft left in-place migration versus export/import undecided while +putting format activation before migration selection. -- for the public track, rename to `NNNN-title.md`, use `Status: Proposed`, fill - the template's author/discussion/implementation metadata, and resolve every - blocker before the RFC PR merges; or -- move the proposals to `docs/dev/` and follow the maintainer-internal process, - leaving the public RFC directory for externally authorable accepted records. +**Disposition:** RFC-025 §8 selects strict export/init/load rebuild before Phase +A, defines old/new-format refusal and co-release ordering, and names the loss of +branches not separately rebuilt, commit DAG, snapshots, tombstones, recovery +history, time travel, and historical checkpoints. Its phasing now puts the +format/refusal/rebuild contract in Phase A. -RFC-027 may state “research-blocked” prominently as its technical state while -retaining the public lifecycle status `Proposed`. +### BLOCKER-09 — decide whether these are public or internal RFCs -> 💬 **Concur (2026-07-11):** same recommendation as the original family -> review — these are maintainer-internal proposals mid-revision; `docs/dev/` -> with the internal process is the low-friction disposition, keeping -> `docs/rfcs/` for its defined merge-equals-acceptance lifecycle. Whichever -> track is chosen, decide it before any of the set merges, since it defines -> what "dispositioned before acceptance" means for every other finding here. +**Affected:** [RFC process](../rfcs/README.md), RFC-022 through RFC-028 + +**Status:** Closed in process and documentation on 2026-07-13. + +At review time the files lived in a directory defined exclusively as the public +track, while using internal-style `rfc-022-*` names, `draft` and +`research-blocked` statuses, and empty owner metadata. That made merge mean both +“accepted” and “still under review.” + +**Disposition:** keep formal RFCs together in `docs/rfcs/` and use one +four-digit `NNNN-title.md` namespace. The filename identifies the RFC; explicit +metadata selects its lifecycle: + +- `Author track: Public contribution` uses the public template and status + vocabulary. Anyone may author it, and merge means maintainer acceptance. +- `Author track: Maintainer design series` permits `draft`, + `research-blocked`, `accepted`, `implemented`, and `superseded` as + machine-readable statuses. A merged revision is durable review state, not + automatic acceptance. +- Every RFC declares an explicit status. If YAML frontmatter and rendered + metadata are both present, they agree. + +RFC-022 through RFC-028 now identify the maintainer design-series track and an +owner. RFC-022 is `implemented` at its documented support boundary, RFC-023 +through RFC-026 and RFC-028 remain `draft`, and RFC-027 remains +`research-blocked`. +All formal RFC filenames in the central directory are normalized to four +digits. Legacy `docs/dev/rfc-00N-*` files remain in place with their existing +lifecycle; they are internal design and implementation records, not members of +the formal RFC filename namespace. + +> 📝 **Historical review record (2026-07-11):** the original disposition offered +> two choices under the then-current process: normalize the files into the +> public track and resolve every blocker before merge, or move the in-flight +> series to `docs/dev/`. The reviewer recommended the latter as the +> lower-friction choice. Maintainers instead selected a third option: keep one +> RFC directory, distinguish lifecycles through explicit `Author track` and +> `Status` metadata, and normalize every formal filename to four digits. The +> public merge-equals-acceptance rule was preserved rather than weakened. > ✅ **RFC-022 implementation close-out (2026-07-13):** every currently > supported graph-write surface is enrolled in an exact adapter, the bounded @@ -437,15 +540,17 @@ retaining the public lifecycle status `Proposed`. > surfaces, crate-visible low-level coordinator methods, and exact per-file > occurrences of registered durable-call shapes including recovery. Exact Optimize > provenance is intentionally deferred until both a stable public Lance -> maintenance-transaction API and distributed recovery fencing exist. This -> closes RFC-022's structural rollout work; it does **not** close BLOCKER-09 or -> change the RFC's public lifecycle status. +> maintenance-transaction API and distributed recovery fencing exist. Together +> with the lifecycle disposition above, this closes RFC-022 and marks it +> implemented at the documented single-writer-process support boundary. It does +> not claim distributed destructive recovery or make exact Optimize provenance +> a rollout gate. ## Protocol clarifications ### BLOCKER-10 — do not put foreign-branch facts in an atomic `ReadSet` -**Affected:** [RFC-022 §6.2](../rfcs/rfc-022-unified-write-path.md#62-branch-merge) +**Affected:** [RFC-022 §6.2](../rfcs/0022-unified-write-path.md#62-branch-merge) **Status:** Closed in specification and implementation on 2026-07-11. Branch merge captures an immutable source commit and revalidates only its incarnation; @@ -479,12 +584,16 @@ Keep target-branch values that must remain stable in the atomic `ReadSet`. ### BLOCKER-11 — RFC-022 and no-heads MemWAL need coarse OCC -**Affected:** [RFC-022 §10](../rfcs/rfc-022-unified-write-path.md#10-rollout-and-compatibility) -and [RFC-026 §6](../rfcs/rfc-026-memwal-streaming-ingest.md#6-fold-protocol) +**Affected:** [RFC-022 §10](../rfcs/0022-unified-write-path.md#10-rollout-and-compatibility) +and [RFC-026 §6](../rfcs/0026-memwal-streaming-ingest.md#6-fold-protocol) + +**Status:** RFC-022 is closed in the shipped adapter. RFC-026 is closed in +specification on 2026-07-14; its no-heads concurrency/failpoint evidence remains +an acceptance gate. -[RFC-022's rollout](../rfcs/rfc-022-unified-write-path.md#10-rollout-and-compatibility) -currently says mutation/load read-set arbitration is likely blocked on -RFC-024 because probed-but-untouched tables lack mutable head rows. +[RFC-022's rollout](../rfcs/0022-unified-write-path.md#10-rollout-and-compatibility) +said at review time that mutation/load read-set arbitration was likely blocked +on RFC-024 because probed-but-untouched tables lacked mutable head rows. For graph-content writes, `(branch incarnation, optional graph head)` is already a conservative branch-wide authority token. An established branch changes its @@ -523,6 +632,14 @@ performance. > on agent-fleet branches, and precisely the honest motivation for RFC-024's > later narrowing (rather than a correctness argument for it). +**Disposition:** shipped RFC-022 mutation/load and BranchMerge retain their +documented coarse branch/schema authority. RFC-026 §6 now captures +`(native branch incarnation, optional graph_head)` plus schema identity and the +complete stream/probe cut, and requires every publisher retry to return to full +fold revalidation on a change rather than reparenting. Its acceptance suite +races a concurrent commit on a probed-but-untouched table and requires that +revalidation without RFC-024. + > **Implementation disposition (2026-07-11):** mutation/load now capture the > native Lance `BranchIdentifier`, exact optional `graph_head`, and accepted > schema identity; revalidate under a branch-then-table gate; and pass the same @@ -541,8 +658,38 @@ performance. > advance after effects becomes `RecoveryRequired` and is compensated rather > than re-parented. +### BLOCKER-12 — in-place format conversion contradicted the strand model + +**Affected:** [RFC-023 §7–8](../rfcs/0023-key-conflict-fencing.md#7-format-boundary-and-rebuild-cutover), +[RFC-024 §9–11](../rfcs/0024-durable-table-heads.md#9-heads-format-boundary-and-compatibility), +[RFC-025 §8](../rfcs/0025-checkpoint-retention.md#8-format-activation-and-rebuild-compatibility), +[RFC-026 §11](../rfcs/0026-memwal-streaming-ingest.md#11-format-activation-and-rebuild), +and [RFC-028 §8](../rfcs/0028-stable-schema-identity.md#8-format-activation-and-upgrade) + +**Status:** Closed in specification on 2026-07-14; each format still requires +its own implementation and genuine old/new-binary evidence before acceptance. + +The reviewed RFC-023 and RFC-024 drafts designed all-branch, in-place conversion +protocols even though OmniGraph's strict-single-version strand intentionally has +no migration dispatcher or compatibility reader. That would create permanent +claims, ledgers, crash recovery, and branch traversal solely to preserve a +transition model the project had already rejected. + +**Disposition:** RFC-023 through RFC-026 and RFC-028 now use one rule. A format +capability is complete at new-graph initialization; old and new binaries refuse +the opposite format; an existing graph moves by quiescing, exporting current +logical state with the old binary, initializing a different root with the new +binary, and loading through RFC-022. No source graph is mutated in place. +Capabilities may co-release to reduce cutovers only after independent +acceptance and a combined initialization/recovery matrix. The rebuild's loss of +unselected branches and history is explicit rather than hidden behind the word +“migration.” + ### TIGHTENING-01 — symmetric Lance conflicts are not obviously an activation gate +**Status:** the deciding beta.21 substrate check is closed; the production +routing and activation dispositions remain open. + RFC-023 correctly identifies Lance's directional filtered/unfiltered behavior. However, its own mandatory fleet outage, recovery drain, historical-duplicate validation, stamp-last activation, and old-binary refusal are intended to make @@ -570,27 +717,40 @@ protected by symmetry in either case. > sound. That is a one-test question against the pinned revision and should be > pinned as a surface guard before the demotion is accepted. +> 🧪 **Beta.21 evidence (2026-07-14):** with `use_index(false)`, the matched-only +> partial-schema UpdateAll + DoNothing shape emits `Some` with a semantically +> empty Bloom filter, not `None`. That closes the narrow substrate question in +> favor of a possible demotion. It does **not** close activation: the +> scalar-indexed/default-v1 path still emits `None`, keyed Append remains +> reachable in today's engine, and beta.21 still permits unfiltered Update or +> Append to land second after a filtered Update. TIGHTENING-01 remains open +> until the engine proves every insertion-bearing keyed path is filtered, +> structurally forbids keyed Append, and proves any remaining unfiltered update +> cannot insert or alter `id`. Upstream symmetry is defense in depth only after +> that proof. + ## Specification and acceptance tightening These are smaller than the blockers above, but should be resolved before the -RFC set is merged: +affected RFC is accepted or implemented: 1. **Checkpoint-name normalization:** define allowed bytes, Unicode normalization, case sensitivity, maximum encoded length, reserved prefixes, and normalization-version compatibility. Test collisions and reuse across versions. -2. **RFC lifecycle values:** [`docs/rfcs/README.md`](../rfcs/README.md#status-values) - defines `Proposed`, `Accepted`, `Declined`, `Superseded by NNNN`, and - `Implemented`. Either use those values or amend the process before using - `draft` and `research-blocked` as machine-readable statuses. Research-blocked - can remain prominent in RFC-027's body while its lifecycle status is - `Proposed`. -3. **Provisional version naming:** sibling RFCs should say “RFC-024 heads - format” rather than treating `v5` as permanent while RFC-024 says the numeral - will change if another format lands first. -4. **Capability ordering:** RFC-025 must specify retention-first, heads-first, - and co-release preservation rules. RFC-026 is a conditional dependency when - streams are enrolled because cleanup must persistently quiesce them. +2. **RFC lifecycle values (closed 2026-07-13):** + [`docs/rfcs/README.md`](../rfcs/README.md) now uses one four-digit filename + namespace and distinguishes the public contribution lifecycle from the + maintainer design-series lifecycle through explicit `Author track` and + `Status` metadata. `draft` and `research-blocked` are valid only in the + latter; public merge remains acceptance. +3. **Provisional version naming (closed 2026-07-14):** RFC-024 now says “heads + format”; RFC-028 provisionally takes the next number while explicitly + allowing the numeral to move. No sibling treats `v5` as permanent. +4. **Capability ordering (closed in specification 2026-07-14):** RFC-025 and + RFC-026 define independent-release rebuilds and combined initialization / + recovery gates for co-release. Cleanup continues to require persistent + quiescence whenever streams are enrolled. 5. **Memory measurement:** `helpers::cost` measures I/O, not peak RSS. RFC-023's adopted-merge and RFC-027's lineage memory gates should use the subprocess `scenarios.rs` harness or an equivalent `wait4`/`ru_maxrss` instrument and @@ -609,26 +769,28 @@ RFC set is merged: The review does not require all RFCs to land together. A safe order is: -1. accept RFC-022 after branch-control recovery, coarse `graph_head` OCC, and - foreign-branch fact classification are explicit (all three are now - dispositioned; the public lifecycle decision in BLOCKER-09 still governs - what “accepted” means); -2. accept and land the stable identity RFC/capability; -3. accept RFC-023 once partial-effect retry and its format/fleet barrier are - complete; +1. RFC-022 is implemented at the documented single-writer-process boundary; + branch-control recovery, coarse `graph_head` OCC, foreign-branch fact + classification, writer-surface closure, and lifecycle are dispositioned; +2. accept and land RFC-028's stable identity capability; +3. accept RFC-023 once partial-effect retry and its format/fleet implementation + and evidence are complete; 4. evaluate RFC-024 independently on its physical lookup cost gate; 5. accept RFC-025 after physical protection of both checkpoints and live graph - branches plus a selected upgrade path; -6. accept RFC-026 after its capability barrier and writer-ownership scope are - explicit; + branches plus strict-rebuild implementation evidence; +6. accept RFC-026 only after Lance exposes the public exact-enrollment and + admission-seal surface, then its capability/refusal/rebuild, multi-effect + recovery, and writer-ownership evidence match the specified scope; 7. keep RFC-027 in research until deletion-delta discovery passes its stated correctness and flat-cost gates. This ordering preserves the split's main benefit: a blocked performance optimization or research result does not hold correctness work hostage. -> 💬 **Order update (2026-07-11):** BLOCKER-04's shipped-cleanup prerequisite is -> now complete and RFC-025 §6 incorporates its floor. Step 3's TIGHTENING-01 -> demotion still hinges on the partial-update filter check -> noted there; if that check lands `None`, upstream symmetry returns to the -> activation gate and RFC-023's timeline moves accordingly. +> 💬 **Order update (2026-07-14):** BLOCKER-04's shipped-cleanup prerequisite is +> complete and RFC-025 §6 incorporates its floor. TIGHTENING-01's beta.21 check +> landed `Some(empty)`, so an upstream symmetry change may be demoted after the +> engine routing proof. Step 3 remains blocked on that proof, RFC-023's +> partial-effect recovery implementation evidence, stable identity, the cost +> gates, and its format/fleet barrier; the substrate result alone does not move +> acceptance. diff --git a/docs/dev/testing.md b/docs/dev/testing.md index 8a850b72..17956d1c 100644 --- a/docs/dev/testing.md +++ b/docs/dev/testing.md @@ -6,7 +6,7 @@ This file is the always-on map of the test surface. **Consult it before every ta | Crate | Path | Style | |---|---|---| -| `omnigraph` (engine) | `crates/omnigraph/tests/` | Integration tests (36 files), fixture-driven, share `tests/helpers/mod.rs` | +| `omnigraph` (engine) | `crates/omnigraph/tests/` | Integration tests (one file per behavior area — see the table below), fixture-driven, share `tests/helpers/mod.rs` | | `omnigraph-cli` | `crates/omnigraph-cli/tests/` | Per-area suites (post-modularization): `cli_cluster.rs` (cluster command surface + operator-actor cascade), `cli_cluster_e2e.rs` (spawned-binary lifecycle compositions — lost-state re-import recovery, out-of-band drift, graph-root destruction, multi-graph mixed-disposition convergence), `cli_data.rs` (load/read/change/branch/commit/export/snapshot/policy/embed/maintenance + operator format cascade), `cli_schema_config.rs` (init/config, schema plan/apply), `cli_queries.rs`, `parity_matrix.rs` (RFC-009 Phase 1: the embedded-vs-remote referee — every forked verb run against both arms with matched Cedar policy and the same actor, scrubbed-JSON + exit-code equality; divergences are pinned in its `KNOWN_DIVERGENCES` ledger, never silently repaired), `system_local.rs` (full-cycle cluster lifecycle with a spawned `--cluster` server, applied-policy enforcement over HTTP, keyed-credential auth, operator aliases), `system_remote.rs`, `crossversion_upgrade.rs` (gated genuine-v3 upgrade round-trip — see "Cross-version upgrade" below); share `tests/support/mod.rs` (hermetic `OMNIGRAPH_HOME` by default) | | `omnigraph-cluster` | mostly in-source `#[cfg(test)] mod tests`; `tests/failpoints.rs` (feature-gated); `tests/s3_cluster.rs` (bucket-gated full lifecycle on object storage) | Cluster config parser, local JSON state diff, state CAS/lock handling/recovery, read-only validate/plan/status plus explicit refresh/import graph observations, config-only apply (content-addressed payload publish, disposition gating, composite-digest convergence, idempotent re-apply), catalog payload verification (status read-only, refresh drift + self-heal), failpoint crash-mid-apply / CAS-race coverage, Stage 4A graph creation (create executor, recovery sidecars + sweep rows, create crash windows), Stage 4B schema apply (migration previews in plan, schema executor, schema-apply sweep classification, schema crash windows), Stage 4C gated deletes (digest-bound approvals, delete executor + tombstones, delete sweep rows, delete crash windows), and 5A policy binding metadata (applies_to in the applied revision, binding-change diffing + convergence, pre-5A backfill), and the 5B serving-snapshot read API (converged read, refusal rows) | | `omnigraph-server` | `crates/omnigraph-server/tests/` | Per-area suites (post-modularization): `auth_policy.rs`, `data_routes.rs`, `schema_routes.rs`, `stored_queries.rs`, `multi_graph.rs` (cluster-mode boot — converged serving, policy binding wiring, boot refusals — + the concurrent branch-ops matrix), `boot_settings.rs` (mode inference, PolicySource), `s3.rs` (bucket-gated: single-graph serving + config-free `--cluster s3://` boot), `openapi.rs` (OpenAPI drift / regeneration); share `tests/support/mod.rs` | @@ -25,8 +25,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav | `writes.rs` | Direct-publish writes: cancellation, RFC-022 non-strict full-attempt reprepare from fresh branch authority, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); the lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) | | `src/table_store/staged_tests.rs` | Crate-internal TableStore staged-write primitives (`stage_append`, `stage_merge_insert`, `stage_create_indices`, `commit_staged{,_exact}`, `scan_with_staged`, `count_rows_with_staged`) — primitive-level unit tests kept behind the same visibility boundary as the raw storage surface. `stage_create_indices_batches_mixed_types_into_one_exact_commit` pins the beta.21 mixed BTREE + FTS + vector shape: staging does not move held or on-disk HEAD, the exact transaction identity lands, the table advances exactly once, and every index becomes visible together | | `forbidden_apis.rs` | Defense-in-depth syntax-tree/source guard over the whole engine. The primary boundary is Rust visibility: raw storage/coordinator/handle-cache modules are crate-private; public `Snapshot::open` returns `SnapshotTable`; and `SnapshotScanner` executes reads without exposing Lance's raw scanner or physical plan. The guard pins those visibility/return-type boundaries, classifies public async inherent `Omnigraph` methods plus loader conveniences, classifies every crate-visible async method on `GraphCoordinator` / `ManifestCoordinator`, and exact-counts registered method/UFCS durable-call shapes including recovery. It also counts selected raw `SnapshotHandle` / Dataset shapes, rejects known renamed-owner and macro-token forms, rejects `include!` and path-shaped allow-list lookalikes, skips structurally test-only modules/functions, and pins retired `InlineCommitResidual` / `storage_inline_residual` symbols absent. Scanner self-tests pin those concrete shapes. This is intentionally not a Rust macro-expander or general alias analysis; `// forbidden-api-allow: ` exempts reviewed inline-Lance lines only | -| `failpoint_names_guard.rs` | Source-walk guard (same defense-in-depth shape as `forbidden_apis.rs`): every failpoint call site (`maybe_fail`, `ScopedFailPoint::new`/`with_callback`, `Rendezvous::park_first`) must reference a `failpoints::names` const, never a bare string literal — a typo'd literal compiles and silently never fires | -| `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump. `_compile_uncommitted_full_table_vector_index_shape` pins beta.21 returning public `IndexMetadata` suitable for `Operation::CreateIndex`; `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction; Guard 9 pins clone-only branch reclaim semantics | +| `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump. `_compile_uncommitted_full_table_vector_index_shape` pins beta.21 returning public `IndexMetadata` suitable for `Operation::CreateIndex`; `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction; Guard 9 pins clone-only branch reclaim semantics. RFC-023's `unenforced_pk_filter_shape_is_route_dependent_on_beta21` explicitly forces v2 versus indexed routes and pins the `Some(populated)` / `Some(empty)` / `None` key-filter shapes; `unenforced_pk_conflict_matrix_is_directional_on_beta21` pins beta.21's filtered/unfiltered and filtered/Append commit-order matrix. Those are substrate upgrade tripwires, not evidence that RFC-023 production routing, migration, recovery, performance, or fleet/format activation is complete | | `warm_read_cost.rs` | Cost-budget tests for the warm read path (query-latency work), measured at the object-store boundary with Lance `IOTracker` (the LanceDB IO-counted pattern): a warm same-branch read does 0 manifest opens, 1 version probe, validates the schema once (Fix 1 / finding A / Fix 2 at commit-history depth); stale same-branch reads perform exactly 2 probes and refresh manifest-only; recreated non-main branches with the same Lance version refresh by incarnation; recreated branch-owned table handles are distinguished by table e_tag or refresh-time cache clearing; recreated traversal topology is protected by per-edge-table e_tag in the graph-index cache key or refresh-time cache clearing; a warm *repeat* read does 0 table opens via the held-handle cache and a write re-opens only the changed table at its new version/e_tag (Fix 3/6A). Also the CSR topology-build cost guards: `fresh_branch_traversal_reuses_main_graph_index` (A1 — a lazy-fork branch reuses main's cached CSR index, 0 rebuilds via `graph_build_count`) and `single_edge_query_builds_only_referenced_edge` (A2 — a one-edge query builds only that edge via `graph_edges_built`); both force CSR via the scoped `with_traversal_mode` seam, so they need no `#[serial]`. See "Cost-budget tests" below. | | `write_cost.rs` | Cost-budget tests for the WRITE path (RFC-013), the latency twin of `warm_read_cost.rs` on the **shared `helpers::cost` harness** (`measure`/`IoCounts`/`assert_flat`/`local_graph`). Runs on **local FS**; gates the **internal-table** term (`__manifest` scans flat in commit-history depth, lineage rows included — `internal_table_scans_are_flat_in_history`, now **green every-PR** since RFC-013 step 2 brought the internal tables into `optimize`; the test compacts at each depth before measuring), graph-visible maintenance arbitration (`ensure_indices_manifest_reads_are_flat_in_history` and `optimize_manifest_reads_are_flat_in_history`), plus green every-PR guards (single-insert `data_writes` bounded, a per-write read-op ceiling that fails the moment a round-trip is added, and a `measure_with_staged` fitness assert that a keyed insert routes through `stage_merge_insert` once with no `stage_append`/vector-index build). The **data-table opener** term is S3-only — see `write_cost_s3.rs` and the backend-split note in "Cost-budget tests" below | | `write_cost_s3.rs` | Bucket-gated (skips without `OMNIGRAPH_S3_TEST_BUCKET`) twin of `write_cost.rs` on the same `helpers::cost` harness: gates the **data-table opener** term (per-write latest-version resolution flat across commit depth on a real object store — per-version GETs are invisible on local FS). A cost gate, not a correctness test — run on demand, not in the every-merge `rustfs_integration` job (see the backend-split note in "Cost-budget tests" below) | @@ -53,7 +52,7 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav | `policy_engine_chassis.rs` | Engine-layer Cedar enforcement (MR-722): allow + deny through every `_as` writer via the SDK directly — no HTTP — proving embedded and CLI callers hit the same gate as the server, with action × scope shapes matching `authorize_request` | | `maintenance.rs` | `ensure_indices`, `optimize` (compaction), `repair` (explicit uncovered-drift publish), and `cleanup` (version GC): empty/idempotent/no-op edges, policy validation, head preservation. EnsureIndices refuses uncovered drift before arming v8 and keeps untrainable Vector work pending. Cleanup pins exact keep-count behavior, lazy-branch retention, graph-wide fail-closed ordering, and refusal of uncovered main HEAD drift before GC. Optimize's bounded schema-v2 adapter publishes multiple productive data tables through one graph commit, emits no lineage/sidecar at steady state, skips uncovered drift, refuses pending recovery, and compacts blob-v2 tables. Repair previews/heals verified maintenance drift and requires `--force` for semantic drift | | `failpoints.rs` | Failure-injection coverage (gated on `failpoints` feature). RFC-022 includes deterministic post-stage/pre-effect races for mutation/load uniqueness and strict disjoint-head changes, plus the cross-handle post-effect `RecoveryRequired` → read-write-open rollback cell. Branch merge adds the captured-source advance cell; post-confirm target-winner compensation; mixed physical + pointer-only delta recovery with fixed commit id/actor/parents; and both sidecar-before-first-ref and ambiguous-ref-create recovery. SchemaApply v7 is pinned by `schema_apply_phase_b_failure_recovered_on_next_open` (exact confirmed roll-forward with fixed commit id + initiating actor), `schema_apply_partial_table_effect_rolls_back_exactly` (Armed proper-prefix compensation), `schema_apply_recovery_reclaims_owned_add_type_target_and_retry_succeeds` (strict owned first-touch cleanup), `schema_apply_first_touch_foreign_winner_is_preserved_not_adopted` (foreign unregistered winner preservation), `schema_apply_post_effect_disjoint_winner_is_preserved` (winner-preserving compensation), `schema_apply_post_effect_same_table_winner_fails_closed` (buried-effect refusal), `schema_apply_recovers_partial_schema_promotion_after_commit_crash` (read-only refusal for both valid and corrupt intents in the torn manifest/schema window, followed by fixed-outcome completion of a partial source/IR/state promotion), and `schema_apply_live_query_waits_for_coherent_schema_publication` (same-handle publication wait plus pre-apply-handle query/export/whole-graph-index capture from the operation-local accepted catalog). Metadata-only before/after-staging and rollback-retry cells keep the empty-effect intent and v5-compatible outcome boundaries pinned. EnsureIndices v8 is pinned by `ensure_indices_stage_btree_failure_leaves_existing_tables_writable` (expensive mixed-index staging before final authority/gates), `ensure_indices_first_touch_crash_before_ref_recovers_cleanly` (sidecar-before-ref no-effect recovery), `ensure_indices_mixed_first_touch_rollback_does_not_delete_moved_ref` (owned-effect rollback and sibling first-touch cleanup), and the no-work/no-sidecar failpoint cell; the recovery module separately pins v8 existing + first-touch payload round-trip while retaining v6 parsing. Optimize's graph-wide v2 envelope is pinned by `optimize_phase_b_failure_recovered_on_next_open` (two-table roll-forward), `optimize_multi_table_partial_effect_rolls_back_under_one_v2_sidecar` (one shared sidecar, no partial visibility, compensation), `optimize_post_manifest_failure_finalizes_multi_table_v2_sidecar` (lost publish acknowledgement), and `optimize_excludes_pending_only_vector_table_from_v2_sidecar` (pending status cannot poison sibling recovery), plus its late-sidecar/main-gate/retry cells. Native controls are pinned by `native_branch_controls_reclassify_lost_acknowledgements` (matching create and absent-ref delete, with no version/lineage movement); `armed_first_touch_recovery_accepts_missing_target_ref` additionally forges and reclaims the clone-only/no-`BranchContents` table state. Legacy path overlap has both sides pinned: `armed_first_touch_recovery_defers_legacy_path_overlap_until_leaf_delete` permits open only for a proven no-effect intent, while `partial_first_touch_recovery_fails_closed_on_legacy_path_overlap` leaves one exact multi-table effect and verifies open fails closed until offline leaf cleanup lets rollback converge. Other control/recovery race cells include `first_touch_post_create_open_error_keeps_recovery_ownership`, `branch_delete_orphans_sidecar_armed_after_initial_barrier`, `branch_merge_fences_target_delete_recreate_aba`, `branch_merge_fences_concurrent_sync_on_same_handle`, `branch_merge_rejects_fresh_target_manifest_change_before_effects`, `branch_merge_rechecks_late_sidecar_after_table_gates`, `optimize_rechecks_late_schema_apply_sidecar_after_main_gate` (late zero-pin graph-global intent), `optimize_rechecks_late_disjoint_main_sidecar_after_main_gate` (table-disjoint intent sharing `graph_head:main`), `optimize_holds_main_gate_through_disjoint_table_effects` (post-relist branch-gate lifetime), `cleanup_rechecks_sidecars_under_gc_gates`, and `full_recovery_rereads_sidecar_body_after_discovery`. The suite also includes the five per-writer effect → manifest-CAS recovery tests, write-entry in-process heal contract, storage-fault matrix, S3 recovery twin, and convergence-idempotent roll-forward regression. | -| `failpoint_names_guard.rs` | Source-walk guard: every `maybe_fail(...)` call site (engine + cluster) must reference the compile-checked `failpoints::names` consts, never a bare string literal — a typo'd literal compiles but silently never fires; same defense-in-depth shape as `forbidden_apis.rs` | +| `failpoint_names_guard.rs` | Source-walk guard (same defense-in-depth shape as `forbidden_apis.rs`): every failpoint call site across engine + cluster (`maybe_fail`, `ScopedFailPoint::new`/`with_callback`, `Rendezvous::park_first`) must reference a compile-checked `failpoints::names` const, never a bare string literal — a typo'd literal compiles but silently never fires | | `recovery.rs` | Open-time recovery sweep — legacy sidecars (including schema-v5 SchemaApply and schema-v6 EnsureIndices bridges under their original loose semantics); schema-v3 Mutation/Load exact transaction identity; schema-v4 BranchMerge exact chains/ref-only effects + complete delta; schema-v7 SchemaApply exact overwrite/create ownership + schema promotion; schema-v8 EnsureIndices exact mixed-index transaction, fixed authority/lineage/delta, and first-touch ref identity (`ensure_indices_v8_round_trips_existing_and_first_touch_effects`); restartable compensation, fixed logical/rollback IDs, branch-token comparison, fresh under-gate reread/reparse, all-or-nothing roll-forward/rollback/refusal, recovery audit, and read-only schema-coherence guard | | `composite_flow.rs` | Compositional/narrative end-to-end stories — multi-step flows that compose mechanics covered by other test files. Catches integration regressions where individual operations all pass their unit tests but their composition breaks (sequential merges, post-merge main writes, time-travel through merge DAG, reopen consistency over multi-merge histories, post-optimize and post-cleanup strict writes). | diff --git a/docs/rfcs/0000-template.md b/docs/rfcs/0000-template.md index 48f4bda0..9fe5525f 100644 --- a/docs/rfcs/0000-template.md +++ b/docs/rfcs/0000-template.md @@ -1,8 +1,13 @@ # RFC NNNN: +> This template is for the public contribution track. See +> [README.md](README.md#maintainer-design-series-lifecycle) +> for the separately versioned maintainer design-series lifecycle. + | | | |---|---| | **Status** | Proposed | +| **Author track** | Public contribution | | **Author(s)** | <your name / handle> | | **Discussion** | <link to the originating Discussion, if any> | | **Implementation** | <issue/PR links, filled in as work lands> | diff --git a/docs/rfcs/0001-fragment-adopt-branch-merge.md b/docs/rfcs/0001-fragment-adopt-branch-merge.md index 1d69dfe5..6e69d647 100644 --- a/docs/rfcs/0001-fragment-adopt-branch-merge.md +++ b/docs/rfcs/0001-fragment-adopt-branch-merge.md @@ -1,14 +1,31 @@ -# RFC 0001: Branch merge by fragment adoption +--- +type: spec +title: "RFC-0001 — Branch merge by fragment adoption" +description: Adopt source-branch Lance fragments by reference, then re-home them before the source branch can be deleted. +status: draft +tags: [eng, rfc, branch, merge, lance, fragments] +timestamp: 2026-06-30 +owner: Ragnor Comerford +--- + +# RFC-0001: Branch merge by fragment adoption | | | |---|---| -| **Status** | Proposed | +| **Status** | Draft | | **Author(s)** | Ragnor Comerford | +| **Owner** | Ragnor Comerford | +| **Author track** | Maintainer design series | +| **Date** | 2026-06-30 | | **Discussion** | — | | **Implementation** | — | -> Status is maintained by maintainers: `Proposed` while the PR is open, -> `Accepted` on merge, `Declined` on close, `Superseded by NNNN` later. +> This working specification was merged with +> [PR #314](https://github.com/ModernRelay/omnigraph/pull/314) as part of a +> maintainer-internal change, not as public RFC acceptance. It was reclassified +> under the maintainer design-series lifecycle on 2026-07-13. Its explicit +> author track and `Draft` status mean that merge does not imply acceptance in +> this track. ## Summary diff --git a/docs/rfcs/rfc-0015-ingest-embeddings.md b/docs/rfcs/0015-ingest-embeddings.md similarity index 97% rename from docs/rfcs/rfc-0015-ingest-embeddings.md rename to docs/rfcs/0015-ingest-embeddings.md index fe139b11..ef243be1 100644 --- a/docs/rfcs/rfc-0015-ingest-embeddings.md +++ b/docs/rfcs/0015-ingest-embeddings.md @@ -1,10 +1,11 @@ # RFC-015: Ingest-time `@embed` via the embedding reconciler -**Status:** Proposed +**Status:** Draft **Date:** 2026-06-21 +**Owner:** OmniGraph maintainers **Tickets:** `iss-ingest-embed` (umbrella); relates to RFC-012 (embedding client), invariant 7 (indexes are derived state) -**Author track:** maintainer-internal RFC +**Author track:** Maintainer design series > Evidence tags: **[S]** verified in v0.7.1 source (`file:line`), **[U]** verified > against an external system, **[D]** documented behavior. Unmarked = design intent. @@ -35,13 +36,13 @@ The natural request — "embed automatically when I add data" — has one obviou implementation (embed inline during `mutate`/`load`) that is **wrong for this codebase on three independent counts**: -1. **It is on the deny-list.** [invariants.md](invariants.md) forbids +1. **It is on the deny-list.** [invariants.md](../dev/invariants.md) forbids "synchronous inline vector/FTS index rebuilds on the write path." An embedding is the same shape (expensive, network-bound, derivable). The deny-list does not distinguish "build the index" from "compute the value the index covers." -2. **It blows the write-path cost contract.** RFC-013 (and step 3b, - [rfc-013-step-3b-writetxn.md](rfc-013-step-3b-writetxn.md)) is making the +2. **It blows the write-path cost contract.** RFC-013 (and its + [Step 3b handoff](../dev/handoff-rfc-013-write-path.md)) is making the write path O(1) and failure-bounded. A per-row embedding call is O(network-latency) and fallible — a single-row `mutate` would block on a model round-trip (the latency that makes this unacceptable for interactive diff --git a/docs/rfcs/rfc-0018-ingest-wal.md b/docs/rfcs/0018-ingest-wal.md similarity index 99% rename from docs/rfcs/rfc-0018-ingest-wal.md rename to docs/rfcs/0018-ingest-wal.md index b7508816..0e289c71 100644 --- a/docs/rfcs/rfc-0018-ingest-wal.md +++ b/docs/rfcs/0018-ingest-wal.md @@ -5,13 +5,14 @@ description: Adds a durability-first streaming ingest path (ack on WAL durabilit status: superseded tags: [eng, rfc, wal, ingest, lance, omnigraph] timestamp: 2026-07-02 -owner: +owner: OmniGraph maintainers --- # RFC-018 — Streaming-ingest WAL on Lance MemWAL -**Status:** Superseded by [RFC-026](rfc-026-memwal-streaming-ingest.md) +**Status:** Superseded by [RFC-026](0026-memwal-streaming-ingest.md) **Date:** 2026-07-02 +**Author track:** Maintainer design series **Surveyed version:** 0.7.2 (branch `dst-extract-crate`); Lance pinned at 7.0.0 **Upstream surveyed:** Lance v8.0.0 (released; RC votes closed 2026-07-01), v9.0.0-beta.10; MemWAL spec (`lance.org/format/table/mem_wal/`, fetched in full 2026-07-02); discussions #7260, #7264, #7222, #7176 **Audience:** OmniGraph maintainers diff --git a/docs/rfcs/rfc-0019-heads-and-fences.md b/docs/rfcs/0019-heads-and-fences.md similarity index 99% rename from docs/rfcs/rfc-0019-heads-and-fences.md rename to docs/rfcs/0019-heads-and-fences.md index c68f5e1e..98b7a817 100644 --- a/docs/rfcs/rfc-0019-heads-and-fences.md +++ b/docs/rfcs/0019-heads-and-fences.md @@ -5,13 +5,14 @@ description: Replaces the warm-publish/pinned-open machinery of PR #318 with two status: superseded tags: [eng, rfc, write-path, manifest, lance, omnigraph] timestamp: 2026-07-04 -owner: +owner: OmniGraph maintainers --- # RFC-019 — Heads and Fences -**Status:** Superseded by [RFC-023](rfc-023-key-conflict-fencing.md) and [RFC-024](rfc-024-durable-table-heads.md) +**Status:** Superseded by [RFC-023](0023-key-conflict-fencing.md) and [RFC-024](0024-durable-table-heads.md) **Date:** 2026-07-04 +**Author track:** Maintainer design series **Surveyed:** omnigraph `main` @ 98530a0e (0.8.0); Lance pinned 7.0.0 (+ vendored lance-table carrying lance#7480); upstream Lance v8.0.0 (released 2026-07-01), v9.0.0-beta.15; PR #318 at `2aab48ba` (reviewed 2026-07-04, 8 verified findings) **Companion docs:** RFC-018 (streaming-ingest WAL), PR #318's plan doc (`unlimited-history-latency-plan.md`, whose §9 "U2" this RFC promotes from follow-up to prerequisite) **Audience:** OmniGraph maintainers diff --git a/docs/rfcs/rfc-022-unified-write-path.md b/docs/rfcs/0022-unified-write-path.md similarity index 95% rename from docs/rfcs/rfc-022-unified-write-path.md rename to docs/rfcs/0022-unified-write-path.md index ccad95d9..546e4b81 100644 --- a/docs/rfcs/rfc-022-unified-write-path.md +++ b/docs/rfcs/0022-unified-write-path.md @@ -2,26 +2,44 @@ type: spec title: "RFC-022 — Unified graph-write protocol" description: One correctness protocol for graph-visible writes, with synchronous recovery, complete read-set arbitration, writer-specific physical-effect adapters, and explicit control-plane exceptions. -status: draft +status: implemented tags: [eng, rfc, write-path, manifest, recovery, concurrency, lance, omnigraph] -timestamp: 2026-07-12 -owner: +timestamp: 2026-07-13 +owner: OmniGraph maintainers --- # RFC-022: Unified graph-write protocol -**Status:** Draft / for team review +**Status:** Implemented on 2026-07-13 **Date:** 2026-07-12 +**Author track:** Maintainer design series **Surveyed:** OmniGraph 0.8.1 (`main`); Lance 9.0.0-beta.21, git rev `1aec1465` **Audience:** engine and storage maintainers -**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md). -Findings marked **BLOCKER** must be dispositioned before acceptance. +**Architecture review:** [RFC-022–028 review ledger](../dev/rfc-022-027-architecture-review.md). +RFC-022's structural rollout and lifecycle findings are closed; the ledger +remains active for RFC-023–027. + +**Implementation evidence:** [#343](https://github.com/ModernRelay/omnigraph/pull/343), +[#344](https://github.com/ModernRelay/omnigraph/pull/344), +[#345](https://github.com/ModernRelay/omnigraph/pull/345), +[#346](https://github.com/ModernRelay/omnigraph/pull/346), +[#347](https://github.com/ModernRelay/omnigraph/pull/347), +[#348](https://github.com/ModernRelay/omnigraph/pull/348), +[#349](https://github.com/ModernRelay/omnigraph/pull/349), +[#350](https://github.com/ModernRelay/omnigraph/pull/350), +[#351](https://github.com/ModernRelay/omnigraph/pull/351), and +[#353](https://github.com/ModernRelay/omnigraph/pull/353). + +Implementation is complete for the documented single-writer-process support +boundary. This status does not claim distributed destructive recovery. Exact +Optimize provenance remains trigger-gated by §6.4, and MemWAL fold enrollment +remains owned by RFC-026. --- ## 0. Summary -OmniGraph will have one **correctness protocol** for every operation that changes +OmniGraph has one **correctness protocol** for every operation that changes manifest-resolved graph state. The protocol does not require every writer to use the same Lance primitive. A mutation can commit one staged transaction per table, a branch merge can make several commits to a table, and optimize can use Lance @@ -56,11 +74,11 @@ This RFC deliberately does not combine key fencing, durable table heads, checkpoint retention, MemWAL ingest, or lineage-based merge-delta discovery into one format and rollout. They are focused follow-ups: -- [RFC-023 — Key-conflict fencing](rfc-023-key-conflict-fencing.md) -- [RFC-024 — Durable table heads](rfc-024-durable-table-heads.md) -- [RFC-025 — Checkpoint retention](rfc-025-checkpoint-retention.md) -- [RFC-026 — MemWAL streaming ingest](rfc-026-memwal-streaming-ingest.md) -- [RFC-027 — Lineage merge deltas](rfc-027-lineage-merge-deltas.md) +- [RFC-023 — Key-conflict fencing](0023-key-conflict-fencing.md) +- [RFC-024 — Durable table heads](0024-durable-table-heads.md) +- [RFC-025 — Checkpoint retention](0025-checkpoint-retention.md) +- [RFC-026 — MemWAL streaming ingest](0026-memwal-streaming-ingest.md) +- [RFC-027 — Lineage merge deltas](0027-lineage-merge-deltas.md) ## 1. Scope and authority @@ -746,7 +764,7 @@ Retry rules are phase-specific: ## 10. Rollout and compatibility This RFC authorizes a protocol refactor, not a manifest v5 format moment. RFC-023 -through RFC-027 own their respective format and public-surface changes. +through RFC-028 own their respective format and public-surface changes. It also does not require a mutable-tip `GraphState` singleton. Three measured, local latency fixes can land independently of the adapter conversion: @@ -764,11 +782,11 @@ These are narrow access-shape fixes, not a second commit-input authority. They must preserve snapshot pinning and still cross the recovery/read-set barriers defined above. -Implementation proceeds in this order: +The implementation landed in this order: -1. Introduce `PreparedWrite`, `ReadSet`, effect-adapter, and recovery-plan concepts - while preserving existing behavior. -2. Ship conservative branch-wide arbitration first. Mutation/load captures +1. `PreparedWrite`, `ReadSet`, effect-adapter, and recovery-plan concepts were + introduced while preserving existing behavior. +2. Conservative branch-wide arbitration shipped first. Mutation/load captures `(Lance BranchIdentifier, exact optional graph_head, accepted schema identity)`; every publisher retry compares that token instead of reparenting. Because every supported graph-content and schema apply advances `graph_head:<branch>` before @@ -825,11 +843,11 @@ Implementation proceeds in this order: > branch pin, computes `keep` from Lance's actual version list, refuses > uncovered main HEAD drift, and completes its live-root preflight before the > first table GC. -3. Convert every current graph-visible effect writer. This is structurally complete +3. Every current graph-visible effect writer was converted. This is structurally complete as of 2026-07-13: Mutation/Load, BranchMerge, SchemaApply, and EnsureIndices use exact adapters; Optimize uses the bounded schema-v2 adapter described in §6.4. The deferred exact Optimize upgrade is not an RFC-022 rollout gate. -4. Keep the supported writer set closed. This is complete as of 2026-07-13: +4. The supported writer set was closed. This is complete as of 2026-07-13: raw storage/coordinator/handle-cache modules are crate-private and public snapshots expose a read-only table/scan facade without Lance's raw scanner or physical plan. `crates/omnigraph/tests/forbidden_apis.rs` adds a @@ -841,10 +859,10 @@ Implementation proceeds in this order: the concrete method/UFCS/raw-Dataset and selected rename/macro shapes pinned by its self-tests; Rust visibility, not heuristic macro expansion, is the structural closure. -5. Remove superseded orchestration after its crash and concurrency cells pass through - the adapter. The old bypass surfaces are removed; writer-specific physical-effect +5. Superseded orchestration was removed after its crash and concurrency cells passed + through the adapter. The old bypass surfaces are removed; writer-specific physical-effect implementations remain intentionally behind their registered adapters. -6. Treat further work as post-close-out latency or capability work: preserve the +6. Further work is post-close-out latency or capability work: preserve the synchronous barrier and recovery classifications, and require the two §6.4 triggers before replacing Optimize's bounded adapter. @@ -1029,7 +1047,7 @@ substrate. The correctness contract is intentionally difficult to reverse: after writers rely on complete read-set arbitration and recovery-before-write, weakening either would reintroduce silent integrity or recovery races. Focused irreversible changes are -reviewed in RFC-023 through RFC-027. +reviewed in RFC-023 through RFC-028. ## 15. Follow-up RFC boundaries @@ -1043,6 +1061,8 @@ reviewed in RFC-023 through RFC-027. behavior, stream quiescence, fresh reads, public surfaces, and upgrade fencing. - **RFC-027** owns candidate discovery from row lineage, deletion discovery, fallback semantics, and evidence for any O(delta) merge claim. +- **RFC-028** owns graph-scoped schema identity, rename/drop lifecycle, + allocation, SchemaApply integration, and the shared identity-format gate. Those RFCs call this protocol when they produce a graph-visible write. None may weaken the recovery barrier, omit read dependencies from `ReadSet`, or create a second graph diff --git a/docs/rfcs/0023-key-conflict-fencing.md b/docs/rfcs/0023-key-conflict-fencing.md new file mode 100644 index 00000000..f48bb006 --- /dev/null +++ b/docs/rfcs/0023-key-conflict-fencing.md @@ -0,0 +1,526 @@ +--- +type: spec +title: "RFC-023 — Substrate-native key-conflict fencing" +description: Make concurrent keyed writes fail or retry through Lance's transaction conflict filters, forbid keyed Append, and activate the contract only in a fencing-compatible internal format reached through rebuild cutover. +status: draft +tags: [eng, rfc, write-path, concurrency, lance, primary-key] +timestamp: 2026-07-10 +owner: OmniGraph maintainers +--- + +# RFC-023: Substrate-native key-conflict fencing + +**Status:** Draft / for team review +**Date:** 2026-07-10 +**Author track:** Maintainer design series +**Surveyed:** omnigraph 0.8.1 (`main`); Lance 9.0.0-beta.21 at git rev +`1aec14652dcbace23ac277fa8ced35000bea0c40`; full Lance transaction, +table-schema, read/write, branching, and MemWAL specifications; pinned Rust +conflict-resolver and merge-insert sources plus runtime surface probes +**Evidence status:** the beta.21 filter shape and directional conflict matrix +are pinned substrate facts. OmniGraph has not activated keyed fencing, changed +production routing, or crossed a fencing-compatible format barrier. +**Relationship to RFC-022:** this RFC is the fencing decision split from the +earlier monolithic RFC-022 draft. [RFC-022](0022-unified-write-path.md) +defines the shared write/recovery protocol; this RFC owns the substrate, +compatibility stamp, and rollout requirements for key conflicts. Stable table +identity and incarnation come from +[RFC-028](0028-stable-schema-identity.md), which is a prerequisite. This RFC +may share a release with [RFC-024](0024-durable-table-heads.md), but neither +RFC depends on the other's storage decision. +**Audience:** engine, storage, migration, and release maintainers +**Open architecture review:** [RFC-022–028 review ledger](../dev/rfc-022-027-architecture-review.md). +Findings marked **BLOCKER** must be dispositioned before acceptance. + +--- + +## 0. Decision summary + +OmniGraph will use Lance's unenforced-primary-key merge-insert filter as the +key-conflict primitive. It will not emulate the filter in the engine and will +not add a lock table or custom transaction manager. + +The guarantee is deliberately narrow and strong: + +> For a keyed table, two concurrent operations that may insert the same `id` +> MUST NOT both commit silently. They either serialize, retry from a new graph +> base, or fail loudly. + +That guarantee cannot be rolled out by annotating a few new tables in an +otherwise v4 graph. Lance's current mixed filtered/unfiltered conflict handling +is directional, and a bare `Append` can commit after a filtered update. The +contract therefore exists only in a fencing-compatible internal format whose +graphs are created with the PK metadata and fenced routing before the first row +is accepted. Existing graphs reach it through the project's strict +export/init/load rebuild cutover, not an in-place all-branch conversion. + +Normative decisions: + +1. Node and edge table PK = `id`. +2. Bare Lance `Append` is forbidden for keyed tables. +3. Every keyed insert/upsert path produces the same Lance key filter. +4. Mixed-version serving is forbidden; old and new binaries mutually refuse + incompatible graph formats. +5. PK metadata is permanent and preserved by every later schema/data rewrite. +6. Existing graphs are exported with the old binary and rebuilt at a new graph + root whose initial schema is already fencing-compatible. +7. Fencing does not replace read-set OCC for `@unique`, RI, or cardinality. + +## 1. Problem + +OmniGraph validates duplicate `@key` values inside one delta, but today two +processes can both read a base where `id = K` is absent, stage disjoint Lance +fragments containing `K`, and let Lance rebase both commits. The graph manifest +CAS orders graph publication; it does not tell Lance that the two data-table +transactions inserted the same logical key. + +The result is worse than an ordinary write conflict: both callers can receive +success while a keyed table contains duplicate IDs. Every subsequent upsert, +edge lookup, uniqueness check, and merge then operates on a broken identity +relation. + +The desired primitive already exists in Lance. The missing work is to use it on +every keyed path and to define a rollout that never admits an unfenced writer. + +## 2. Substrate facts and the directional asymmetry + +This section is load-bearing. Tests pin every statement before implementation. + +### 2.1 Filter attachment + +On the pinned Lance revision, the explicitly selected v2 merge plan +(`use_index(false)`) compares the ordered ON field IDs with the schema's +non-empty unenforced-PK field IDs. For the insert-capable and matched-only +update shapes this RFC consumes, an exact match attaches +`Some(KeyExistenceFilter)` to the resulting `Operation::Update`: + +- a fresh insert produces a populated Bloom filter; +- `WhenMatched::Fail` on a fresh key preserves that populated filter; +- a matched-only partial-schema `UpdateAll` with + `WhenNotMatched::DoNothing` produces `Some` with a semantically empty Bloom + filter, not `None`, because no row took the Insert action; and +- a mismatched ON field set produces `None`. + +The filter contains keys only for rows classified as inserts. Updates of +existing rows continue to use Lance's affected-row / fragment conflict +machinery, and delete-only operations do not need an insertion filter. + +When every ON column has a scalar index and `use_index` remains enabled, Lance +selects the legacy v1/indexed merge path, which hardcodes the filter to `None`. +Therefore a BTREE on `id` can route an otherwise correct merge onto an unfenced +path unless the caller disables that path or Lance wires the filter into it. + +### 2.2 Conflict compatibility is directional + +Lance transaction compatibility is evaluated from the transaction currently +attempting to commit against transactions that committed after its read +version. It is not implicitly symmetric. + +At `1aec1465`, with each probe arranged so no independent fragment conflict +decides the result, `check_update_txn` has this matrix. “Current” means the +second transaction attempting to commit; “committed” means the first +transaction already visible after the current transaction's read version. + +| Current / second transaction | Committed / first transaction | beta.21 result | +|---|---|---| +| filtered Update | filtered Update, overlapping fresh key | retryable conflict | +| filtered Update | filtered Update, disjoint fresh key | succeeds | +| filtered Update | unfiltered Update | retryable conflict | +| unfiltered Update | filtered Update | succeeds | +| filtered Update | bare Append | retryable conflict | +| bare Append | filtered Update | succeeds | + +The matched-only v2 result above resolves the narrow empty-filter question: it +stays in the filtered class even though it inserted no row. It does not make +beta.21 symmetric. The indexed v1 route remains unfiltered, and a bare keyed +Append can still land second. + +Consequently, "filtered vs unfiltered conflicts" is not a sufficient rollout +argument. Commit order matters. A filtered writer can win first and a stale +unfiltered writer or bare keyed append can still land second. + +### 2.3 What a PK annotation does not do + +The key is explicitly *unenforced*. Merely setting the metadata: + +- does not validate historical uniqueness; +- does not make bare appends unique; +- does not protect a merge whose ON set differs from the PK; +- does not repair an existing duplicate; +- does not replace OmniGraph's semantic validators. + +## 3. Scope and non-goals + +In scope: + +- node and edge data tables; +- mutation, load, branch merge, recovery replay, and WAL fold paths; +- new-format table creation and export/import cutover activation; +- schema/overwrite preservation of PK metadata; +- typed retry behavior and coverage gates. + +Out of scope: + +- using a composite edge PK (`src`, `dst`); +- enforcing arbitrary `@unique` groups through the Lance PK; +- keyless streaming-table deduplication; +- a custom OmniGraph WAL, lock table, or transaction manager; +- declaring general multi-process writes supported before foreign-process + recovery-sidecar ownership is solved. + +## 4. Table classes and PK contract + +### 4.1 Keyed graph tables + +Every normal node and edge table has a non-null `id` field. Its Lance schema +MUST mark exactly that field as the unenforced PK. For edges, `src` and `dst` +remain ordinary fields governed by referential-integrity and cardinality +validation. An edge endpoint move is an update of the row identified by `id`. + +The PK field is addressed by the pinned Lance schema's stable field ID, not +column position or mutable name. The surrounding logical table contract is +bound to [RFC-028](0028-stable-schema-identity.md)'s stable table ID and +incarnation. Fencing cannot activate before that capability lands. + +### 4.2 Keyless append-only tables + +A table explicitly declared append-only may omit a PK. Such a table may use +Lance `Append`, including MemWAL append-only operation. It receives no +same-logical-key guarantee because it has no logical key. + +Current node and edge types are not in this class: both have graph identity on +`id`. The class is reserved for an explicitly designed internal or future +non-graph table surface. + +The distinction is catalog-derived and first-class. Callers do not choose it +with an ad-hoc flag. + +## 5. Normative write routing + +| Logical operation | Keyed table | Keyless append-only table | +|---|---|---| +| Strict insert / load append | merge-insert ON exactly `id`, `WhenMatched::Fail`, filtered path | `Append` allowed | +| Upsert / load merge | merge-insert ON exactly `id`, `WhenMatched::UpdateAll`, filtered path | workload-specific | +| Fast-forward branch merge of new rows | filtered merge-insert with `WhenMatched::Fail`, even when every row was classified new | `Append` allowed | +| WAL upsert fold | filtered merge-insert with `merged_generations` | append transaction allowed | +| Update existing row | merge-insert/update with affected-row conflict metadata | workload-specific | +| Delete | staged Lance delete; PK filter is not the delete-conflict primitive | staged Lance delete | +| Overwrite | staged overwrite whose output schema preserves the exact PK | staged overwrite | + +### 5.1 No keyed `Append` + +The prohibition includes internal optimization paths. A caller may not infer +"all rows are new" and switch a keyed table to `stage_append`: that inference +was made against a snapshot and is exactly what a concurrent same-key writer can +invalidate. + +Routing through merge-insert does not collapse strict and upsert semantics. +Strict `load --mode append` and other insert-only surfaces use +`WhenMatched::Fail`; a row already present at the pinned base or discovered at +execution remains an error. Only declared upsert surfaces use `UpdateAll`. + +The storage trait and `forbidden_apis` guard MUST make a keyed append difficult +to express accidentally. The fast-forward merge optimization is retained only +for keyless append-only tables unless Lance ships a key-filtered append +transaction. + +This prohibition knowingly retires a measured fix. The fast-forward append +path exists because a whole-delta merge-insert join exhausted the query memory +pool on embedding-bearing tables (#277); routing adopted rows back through a +filtered merge-insert re-exposes that workload to join memory behavior. The +regression class is therefore a named ship gate: the fenced bulk adopt-merge +must pass the §11.4 memory/cost gate on embedding-bearing tables — via bounded +batched fenced merges inside one staged transaction, a pool-bounded execution +mode, or an upstream key-filtered append — before the keyed fast-forward path +is removed. Correctness wins the ordering, but the memory bound is not +optional. + +### 5.2 Routing choice + +There are two acceptable implementations: + +1. use the v2 merge path (`use_index(false)`) and pass its scale gate; or +2. consume a pinned Lance revision whose indexed path emits the identical + filter and passes the same surface guards. + +If the v2 hash-join cost scales unacceptably at the Phase-B workload, fencing +waits for option 2. Correctness is not traded for the old indexed-path speed. + +### 5.3 Symmetric mixed-transaction behavior + +Beta.21 does not conservatively reject both orders of filtered Update vs +unfiltered Update or filtered Update vs bare Append. Under today's production +routing that directional behavior remains an activation blocker. + +Activation first requires an engine-boundary proof that every +insertion-bearing keyed path uses the exact-PK filtered primitive and keyed +`Append` is structurally unreachable. For any unfiltered Update that remains, +the RFC must then either consume an upstream Lance revision with symmetric +conflict-resolver behavior, or prove that the operation can neither insert a +row nor alter `id`. Affected-row conflict metadata must continue to cover two +updates to the same existing row. + +The beta.21 matched-only `Some(empty)` result makes the latter proof possible +for that one route; it does not complete the engine-wide proof. A +workspace-only Lance fork is not an accepted permanent design. The fleet +barrier remains necessary under either disposition because two old, unfiltered +writers still have no filters to compare. + +## 6. Retry and validation semantics + +A Lance retryable key conflict does **not** by itself authorize a restart. The +writer first classifies the current RFC-022 attempt: + +1. A conflict detected before the recovery sidecar is armed has no physical + effect. The writer discards the staged attempt and may perform a bounded + semantic retry from fresh authority. +2. If the sidecar is armed but exact classification proves that no participant + advanced, the writer finalizes that empty intent before retrying. +3. If any participant advanced, or emptiness cannot be proved, the writer keeps + the sidecar and returns `RecoveryRequired`. The synchronous recovery barrier + must resolve that exact attempt before any later semantic attempt is + prepared; the writer never replans around partial physical state. + +An eligible semantic retry always restarts the entire logical attempt: + +1. gather a new graph snapshot and schema identity; +2. rerun all delta and committed-state validation; +3. restage from that base; +4. commit and publish through the normal recovery-covered pipeline. + +It is incorrect to retry only `commit_staged`: an insert may have become an +update, defaults or checks may now differ, and cross-table validation may have +changed. + +Upsert surfaces may perform that bounded whole-operation retry only after the +prior attempt is proven effect-free and finalized. Strict insert surfaces, +including `load --mode append`, never retry and never change meaning under +contention. An already-present match from `WhenMatched::Fail`, or a concurrent +same-key conflict from an effect-free finalized attempt, normalizes to typed +`KeyConflict` / HTTP 409 for the whole strict operation. A concurrent conflict +after another participant advanced instead returns `RecoveryRequired` and +retains the sidecar; recovery takes precedence over reporting a terminal key +result. Strict callers decide whether to resubmit after that barrier. Strict +inserts never switch to `UpdateAll`; other strict read-modify-write surfaces +retain their typed write conflict. Retry exhaustion on a non-strict upsert +remains a retryable 409. + +Fencing covers the PK insertion race only. `@unique` values on different IDs, +edge RI, and cardinality depend on a read set. Their correctness requires the +read-set-in-CAS design or equivalent revalidation before HEAD movement; this RFC +does not claim that fences close those races. + +## 7. Format boundary and rebuild cutover + +### 7.1 No partial activation in the old format + +OmniGraph MUST NOT annotate a new data table and advertise fencing while the +graph remains generally writable by older binaries. An older process can select +the legacy merge path or keyed append and bypass the guarantee. + +The graph-wide internal-schema stamp is read from the reserved main manifest +before any named-branch open; selecting a named branch cannot bypass the +compatibility check. A fencing-capable binary keeps +`MIN_SUPPORTED == CURRENT`, refuses an older graph with the documented rebuild +instructions, and refuses a newer graph with “upgrade omnigraph.” An older +binary refuses the fencing-capable stamp. + +The exact format number is assigned when this RFC is accepted. If RFC-028, +RFC-024, or another independently accepted capability is ready for the same +release, they may share one new-graph format after a combined initialization and +recovery review. If they release separately, each internal-format change +requires its own rebuild under the strand policy; no draft capability is +silently smuggled into another RFC's stamp. + +Quiescence is required only to obtain a consistent export and perform operator +cutover. The new binary never opens the old graph in a special write mode, +claims an old-format migration, annotates a branch in place, or writes a +completion stamp back to the source. + +### 7.2 New graphs + +A graph created directly at the fencing-compatible format creates every keyed +table with the PK metadata already present and enables only the write routing +in §5. There is no post-create annotation window. + +## 8. Strict-strand activation + +Existing graphs activate fencing only through the documented rebuild path: + +1. quiesce the selected old graph branch and export it with a binary that reads + the old format; +2. show the accepted schema with that old binary; +3. initialize a different graph root with the fencing-capable binary; every + keyed table is created with the exact PK metadata before any data load; +4. import through ordinary `load`, using RFC-022 staging/recovery and the + production fenced route; +5. fail before serving if the export contains duplicate `id` values or the + physical PK contract differs from the accepted RFC-028 identity/catalog; +6. verify the target, then cut clients over. + +The source is never mutated, so a failed target init or import is discarded or +repaired and retried. There is no lazy-branch annotation problem because no old +physical version is adopted into the new root. + +Rebuild preserves the selected branch's current rows, vectors, blobs, and +schema shape. It does not preserve the old graph's identities, branches, commit +DAG, snapshots, tombstones, or time-travel history. A branch whose state must +survive is exported and rebuilt separately today, producing a separate graph +root. This loss is shown in plan/runbook output rather than hidden behind the +word “migration.” + +## 9. Preservation after activation + +Once set, the following are storage invariants: + +- a table overwrite within one physical dataset carries the same PK field ID + and position; +- schema apply cannot remove, replace, reorder semantically, or make nullable a + PK field; +- a property rename within one dataset preserves the PK field ID and metadata; +- a supported type rename may rematerialize a new dataset with different Lance + field IDs, but the target is created with `id` as its exact PK before data is + accepted and retains RFC-028's logical table ID/incarnation; +- branch fork/clone preserves it; +- import/rebuild creates it before accepting data; +- recovery restore may select an older data image only if that image is already + fencing/PK-compatible; +- a table recreation uses RFC-028's new stable table ID and incarnation and + installs the catalog-derived PK contract at creation; +- `__manifest`'s existing legacy PK key form is preserved exactly as stored; + fresh initialization writes that selected form directly rather than + “normalizing” an existing table. Lance forbids changing a set PK, and the + native-namespace decoupling documented in the Lance alignment audit depends + on that legacy form remaining in place. + +Every open-for-write path verifies the physical schema matches the catalog PK +contract. The check is against the pinned physical schema and is not a +maintained parallel registry. + +## 10. Recovery and multi-process scope + +All data writes retain the existing Phase A-D sidecar protocol. The key filter +does not close the table-HEAD-before-graph-manifest window. + +The fenced data-table transaction is cross-process safe in its failure-free +commit path. OmniGraph's current recovery sweep, however, serializes with live +writers only in-process; a foreign recovery process can still inspect a live +sidecar, and destructive `Restore` cannot be made convergence-idempotent. + +Therefore this RFC MUST use one of two honest dispositions: + +1. retain the documented single-writer-process support boundary and describe + fences as closing the silent key-race primitive only; or +2. land a cross-process sidecar claim/lease before advertising general + multi-process writes. + +Fences alone are not evidence for disposition 2. + +## 11. Tests and acceptance gates + +### 11.1 Lance surface guards + +The beta.21 guards pin current substrate truth, not activation readiness: + +- exact PK ON + v2 produces a populated filter for a fresh insert and a + fresh-key `WhenMatched::Fail`; +- matched-only partial-schema UpdateAll + DoNothing produces + `Some(empty)`, while a mismatched ON set produces `None`; +- a scalar-indexed/default-v1 route produces `None` until Lance changes that + path; +- filtered/filtered overlapping keys retry and disjoint keys may rebase; +- filtered current vs unfiltered committed retries, while the reverse order + succeeds; +- filtered current vs committed Append retries, while current Append vs a + committed filtered Update succeeds; and +- PK metadata cannot be changed or removed once installed. + +The directional guards deliberately turn red if a future Lance revision gains +symmetry, forcing a new alignment audit and RFC disposition. They do not prove +that OmniGraph production routing has eliminated unfiltered insertion-bearing +operations, installed the PK on every reachable table image, or crossed the +fleet/format barrier. + +### 11.2 Engine concurrency tests + +- the same-key DST cell becomes a hard assertion with N concurrent writers; +- different keys remain concurrently writable; +- every keyed load/mutation/merge/fold path is observed to use the filtered + primitive; +- strict append of an existing `id` still fails and never updates the row; +- strict pre-existing-match and concurrent-insert cases normalize to the same + external `KeyConflict` when the attempt is effect-free, while preserving + `WhenMatched::Fail`; +- a conflict on table N after table 1 committed retains the exact sidecar, + returns `RecoveryRequired`, and starts no semantic retry until the recovery + barrier resolves that attempt; +- a source-walk guard rejects keyed `stage_append`, including the former + fast-forward path; +- an effect-free retry finalizes its prior intent and reruns validation rather + than committing the stale staged batch. + +### 11.3 Format and rebuild tests + +- a genuine old-format graph is refused before schema/recovery work by the new + binary, and the old binary refuses a fencing-format graph; +- old-binary export followed by new-binary init/load round-trips rows, vectors, + and blobs into a graph whose keyed tables were PK-annotated from creation; +- duplicate IDs abort target import before serving and leave the source intact; +- a separately rebuilt branch has the documented independent-root identity and + no inherited commit history; +- a co-release, when selected, creates every accepted capability in the first + valid target state and never exposes a partial-format serving window; +- overwrite, schema apply, branch fork, restore, and later imports preserve PK + metadata inside an already compatible graph. + +### 11.4 Cost gate + +Measure a small upsert into 10K, 100K, and 1M-row indexed tables using the +shared cost harness. If `use_index(false)` makes work scale with table size +beyond the accepted budget, the indexed-path upstream work is a ship blocker. + +Additionally measure the bulk adopt-merge shape that motivated the keyed +fast-forward path (#277): a many-row, all-new-rows fenced merge into an +embedding-bearing table, asserting peak memory bounded by batch size rather +than table or delta width. If the fenced path cannot meet that bound, the +keyed fast-forward removal waits for the mitigation named in §5.1. + +> 💬 **Instrument required (tightening 5 in the +> [review ledger](../dev/rfc-022-027-architecture-review.md)):** +> `helpers::cost` measures I/O, not peak RSS, so this memory bound is +> unenforceable as written. Use the subprocess `scenarios.rs` harness or an +> equivalent `wait4`/`ru_maxrss` instrument, and name dataset sizes, baseline, +> cap, and pass threshold. + +## 12. Decisions and open gates + +### Decided + +- `id`, not `src`+`dst`, is the edge PK. +- No keyed append, including optimization-only append. +- No mixed-fleet or new-table-only v4 rollout. +- Existing graphs activate fencing only through strict-strand export/init/load + into a new graph root; there is no in-place PK migration. +- A retryable upsert conflict retries the whole logical operation only after + the prior attempt is proven effect-free and finalized. Strict insert maps + effect-free existing and concurrent matches to `KeyConflict` without + changing mode; a partial or unclassifiable attempt returns + `RecoveryRequired` first. +- Read-set validation remains a separate required concurrency design. + +### Open ship gates + +1. Close post-activation routing under beta.21's directional resolver: every + insertion-bearing keyed path carries the exact-PK filter, keyed `Append` is + structurally unreachable, and each remaining unfiltered Update either is + proven unable to insert or alter `id` with adequate affected-row conflict + coverage, or is protected by a symmetric resolver on the pinned Lance + revision. +2. v2-path scale result versus indexed-path filter availability. +3. Operator repair procedure for pre-existing duplicate IDs. +4. Accepted and implemented [RFC-028](0028-stable-schema-identity.md) identity. +5. Cross-process recovery ownership before any broadened topology claim. +6. Final format-number/co-release sequencing and genuine cross-version rebuild + evidence. +7. The fenced bulk adopt-merge memory/cost gate on embedding-bearing tables + (the #277 regression class) — see §5.1 and §11.4. diff --git a/docs/rfcs/rfc-024-durable-table-heads.md b/docs/rfcs/0024-durable-table-heads.md similarity index 50% rename from docs/rfcs/rfc-024-durable-table-heads.md rename to docs/rfcs/0024-durable-table-heads.md index 13625975..afc7f51a 100644 --- a/docs/rfcs/rfc-024-durable-table-heads.md +++ b/docs/rfcs/0024-durable-table-heads.md @@ -1,36 +1,43 @@ --- type: spec -title: "RFC-024 — Durable table heads and the v5 manifest" -description: Materialize one live-or-tombstoned current-state row per table inside each manifest branch, prove bounded physical lookup, and migrate every branch atomically before stamping v5. +title: "RFC-024 — Durable table heads" +description: Materialize one live-or-tombstoned current-state row per stable table identity inside each manifest branch and prove bounded physical lookup without weakening derived-index correctness. status: draft tags: [eng, rfc, manifest, write-path, versioning, migration, lance] timestamp: 2026-07-10 -owner: +owner: OmniGraph maintainers --- -# RFC-024: Durable table heads and the v5 manifest +# RFC-024: Durable table heads **Status:** Draft / for team review **Date:** 2026-07-10 -**Surveyed:** omnigraph 0.8.1 (`main`); Lance 9.0.0-beta.15 at git rev -`f24e42c1`; full Lance table layout, transaction, branching, indexing, -compaction, cleanup, and object-store specifications +**Author track:** Maintainer design series +**Surveyed:** omnigraph 0.8.1 (`main`); Lance 9.0.0-beta.21 at git rev +`1aec14652dcbace23ac277fa8ced35000bea0c40`; full Lance table layout, +transaction, branching, indexing, compaction, cleanup, and object-store +specifications **Relationship to RFC-022:** this RFC is the durable-heads decision split from -the earlier monolithic RFC-022 draft. [RFC-022](rfc-022-unified-write-path.md) -defines the shared publisher/recovery protocol; this RFC owns the v5 format and -migration boundary. It deliberately excludes checkpoint retention, which -[RFC-025](rfc-025-checkpoint-retention.md) reviews separately. Key fencing in -[RFC-023](rfc-023-key-conflict-fencing.md) is also independently reviewable; +the earlier monolithic RFC-022 draft. [RFC-022](0022-unified-write-path.md) +defines the shared publisher/recovery protocol; this RFC owns the heads-format +rows, lookup, and publication boundary. Stable table identity and incarnation come from +[RFC-028](0028-stable-schema-identity.md), which is a prerequisite; this RFC +owns only head storage, lookup, and publication. It deliberately excludes +checkpoint retention, which +[RFC-025](0025-checkpoint-retention.md) reviews separately. Key fencing in +[RFC-023](0023-key-conflict-fencing.md) is also independently reviewable; the two may share a release but do not block one another's evidence gates. **Audience:** engine, manifest, migration, branch, and release maintainers -**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md). +**Open architecture review:** [RFC-022–028 review ledger](../dev/rfc-022-027-architecture-review.md). Findings marked **BLOCKER** must be dispositioned before acceptance. --- -Throughout this draft, **v5** means the next internal schema after today's v4. -If another independently accepted format change lands first, the release number -is reassigned; none of the head-row semantics depend on the numeral. +Throughout this draft, **heads format** means the first internal schema that +contains the table-head contract. RFC-028 provisionally takes the next internal +schema after today's v4, so this RFC deliberately does not reserve the numeral +v5. The exact number is assigned when the independently accepted capabilities +for a release are known. ## 0. Decision summary @@ -39,7 +46,7 @@ ask "what is current?" Current-state resolution folds history, so its physical cost grows with commit count even though the answer contains only one value per table. -v5 adds one mutable, durable `table_head` row per stable table identity inside +The heads format adds one mutable, durable `table_head` row per stable table identity inside each native `__manifest` branch. The publisher updates the head in the same Lance merge-insert transaction as immutable table-version rows, tombstone events, graph lineage, and `graph_head`. The journal remains the history source; @@ -47,7 +54,7 @@ heads become the current-state source. The format does **not** ship merely because the logical result has O(tables) rows. A filtered scan over a history-sized Lance table is still O(history) -physical work. v5 is gated on a Lance-native indexed lookup whose measured I/O +physical work. The heads format is gated on a Lance-native indexed lookup whose measured I/O is flat at history depth and whose uncovered tail is bounded and observable. Normative decisions: @@ -57,10 +64,11 @@ Normative decisions: 2. Head state is explicitly `live` or `tombstoned` and carries an incarnation. 3. Every publish and recovery outcome updates journal and head atomically. 4. Current-state reads use heads; history reads use the journal. -5. Missing or duplicate heads in v5 are corruption, not a reason to silently +5. Missing or duplicate heads in the heads format are corruption, not a reason to silently return to the history fold. -6. Migration covers every live manifest branch, then stamps main v5 last. -7. If RFC-023 is co-released, the combined migration also verifies PK fencing; +6. A new graph's first valid state contains complete identity-bearing journal + rows and heads; an existing graph reaches the format only by export/init/load. +7. If RFC-023 is co-released, target initialization also verifies PK fencing; durable heads do not depend on that decision. 8. Checkpoint/retention markers are deferred to a separate RFC. @@ -86,11 +94,11 @@ as the journal removes both liabilities: In scope: -- v5 table-head schema and state transitions; +- heads-format table-head schema and state transitions; - atomic publisher, recovery, and current-read semantics; - bounded physical access proof; -- all-branch predecessor→heads migration and compatibility refusal; -- optional co-release integration with RFC-023's all-branch PK activation. +- strict-format refusal and export/init/load rebuild activation; +- optional co-release integration with RFC-023's new-graph PK activation. Out of scope: @@ -106,7 +114,7 @@ itself pin a version in another Lance dataset. Lance cleanup recognizes its own versions, tags, and branch references, not foreign references. A later RFC must define substrate-native per-table pins and their crash-safe lifecycle. -## 3. v5 table-head schema +## 3. Table-head schema Each native manifest branch contains exactly one current head row for each stable table identity known to that branch. @@ -118,41 +126,43 @@ object_id = "table_head:<stable_table_id>" object_type = "table_head" ``` -`stable_table_id` survives a type rename. It is not the display name and is not -derived anew from `kind:name`. Closing the current rename-stable identity gap is -a v5 ship gate, and **this RFC owns it**: the stable-ID encoding, the compiler -contract that mints and preserves it, and the incarnation baseline are defined -here. RFC-023 §4.1 (rename-safe PK claims) and RFC-025 §2.1 (checkpoint table -rows) consume this identity and must not ship identity-dependent claims before -it lands. No other RFC in the family defines a competing identity scheme. - -> 💬 **Superseded by review (BLOCKER-07 in the -> [review ledger](../dev/rfc-022-027-architecture-review.md)):** ownership -> *inside* this RFC contradicts siblings that consume the identity while -> calling this RFC optional (RFC-025) or without declaring the dependency -> (RFC-026). Disposition: extract identity/incarnation into its own -> prerequisite RFC, or accept this RFC in full before the identity-dependent -> siblings. Replace this ownership note accordingly. +`stable_table_id` and `incarnation_id` are the graph-scoped nonzero `u64` +identities defined and minted by +[RFC-028](0028-stable-schema-identity.md). `stable_table_id` is the node or edge +type ID; it survives a supported rename and is never derived from the display +name, `kind:name`, path, or Lance field ID. This RFC consumes those values and +does not define a competing allocator or identity registry. The head row is mutable under `WhenMatched::UpdateAll`, just like -`graph_head:<branch>`. There is one object ID per stable identity; recreation -does not leave multiple candidate heads. +`graph_head:<branch>`. There is one object ID per stable identity. Under +RFC-028's current drop/re-add rule, the old identity remains tombstoned and the +new declaration receives a distinct head object; there is still never more than +one candidate head for one stable identity. ### 3.2 Payload -v5 reuses the manifest's typed columns where they already fit and stores the +The heads format reuses the manifest's typed columns where they already fit and stores the remaining versioned payload in a typed JSON structure: ```text TableHeadMetadata { state: "live" | "tombstoned", - stable_table_id: String, - incarnation_id: ULID, - schema_hash: String, + stable_table_id: u64, + incarnation_id: u64, + physical_ref_incarnation: String, + schema_ir_hash: String, head_graph_commit_id: Option<ULID>, } ``` +`physical_ref_incarnation` is an opaque, backend-derived token that changes +when the Lance dataset or native ref at the same path/branch/version is deleted +and recreated. Use the manifest/ref e_tag when the backend provides one; +otherwise the implementation must prove an equivalent Lance-native token. A +backend with no proven token cannot activate the heads format. Logical +`incarnation_id` cannot substitute for this field because a physical owner or +ref may be replaced while logical table identity is preserved. + The row columns have these meanings: | Column | `live` | `tombstoned` | @@ -170,33 +180,35 @@ other row. ### 3.3 Incarnations -`incarnation_id` distinguishes drop/recreate ABA: +`incarnation_id` distinguishes table-lifetime ABA: - rename preserves stable ID and incarnation; - ordinary writes preserve both; - physical owner handoff preserves both; -- dropping the type transitions the one head to `tombstoned`; -- recreating a logically new table mints a new incarnation and updates the same - stable-identity head to `live` only through an explicit schema operation. +- dropping the type transitions its head to `tombstoned`; +- an ordinary later same-name add mints a new stable ID and incarnation, creates + a new live head, and leaves the old head tombstoned; +- no name comparison revives an old head. -If recreation is assigned a new stable table identity by schema semantics, it -gets a new head object and the old identity remains tombstoned. The schema -planner, not a name comparison, chooses this outcome. +Any future explicit rematerialization that preserves stable type identity but +changes incarnation must first be specified by RFC-028 or a successor. The head +transition then follows that accepted schema outcome; RFC-024 never invents it. ### 3.4 Journal identity -The mutable head is not the only place that records identity. Every v5 +The mutable head is not the only place that records identity. Every heads-format table-version, registration, rename, and tombstone journal event carries -`stable_table_id` and `incarnation_id`; otherwise drop/recreate followed by a -new physical dataset whose Lance versions restart cannot be replayed -unambiguously. - -Migration writes one immutable v5 incarnation-baseline row per known identity, -bound to the source-tip digest. New registrations/recreations write an immutable -incarnation transition. Head repair starts from that baseline and replays only -identity-bearing v5 events. It does not infer identity from mutable names or -compare unrelated Lance version numbers. If pre-v5 history cannot be mapped to -one baseline identity deterministically, migration refuses before the stamp. +`stable_table_id`, `incarnation_id`, and the exact physical location/ref token +for its table version; otherwise drop/recreate followed by a new physical +dataset whose Lance versions restart cannot be replayed unambiguously or repair +the full head token. + +A fresh graph's first valid manifest writes identity-bearing registration events +and matching heads for its initial tables. Later registrations write immutable +identity-bearing events in the same publish as their new heads. Offline head +repair replays only these identity-capable events from graph initialization; it +does not infer identity from mutable names, compare unrelated Lance version +numbers, or translate predecessor history. ## 4. State-transition rules @@ -207,7 +219,7 @@ one baseline identity deterministically, migration refuses before the stamp. | Owner-branch handoff | live owner A → live owner B, even if version is equal | | Rename | live key/name A → live key/name B; identity/incarnation unchanged | | Drop table | live → tombstoned in the same graph publish as the tombstone journal row | -| Recreate | tombstoned → live with the schema-planned identity/incarnation outcome | +| Same-name add after drop | old identity stays tombstoned; new identity → live | | Recovery roll-forward | apply the failed writer's intended live/tombstone transition | | Recovery rollback | publish a head matching the restored physical version and logical pre-write state | @@ -216,7 +228,7 @@ its corresponding head mutation in the same publisher source batch. ## 5. Atomic publisher contract -The v5 publisher constructs one merge-insert source containing: +The heads-format publisher constructs one merge-insert source containing: - immutable, identity-bearing table-version rows; - immutable, identity-bearing table-tombstone/transition rows; @@ -231,20 +243,33 @@ resolves the graph parent and re-reads commit authority inside every CAS retry. Expected table versions are compared against table heads, not reconstructed by folding the journal. +The comparison is not version-only. Every writer captures and revalidates the +complete expected token: + +```text +(state, stable_table_id, incarnation_id, location, table_branch, + physical_ref_incarnation, table_version, schema_ir_hash) +``` + +Any difference returns control to full RFC-022 revalidation before effects; a +publisher retry may not reparent a prepared table effect across the mismatch. +The desired head token is derived from the exact achieved dataset/ref after the +effect, never copied forward from a stale expected head. + Two graph-content writers touching disjoint data tables still contend on `graph_head:<branch>`, form one linear graph history, and re-parent on retry. Writers touching the same table also contend on its one `table_head` object. Metadata-only CASes contend on the stable authority rows named by their complete `ReadSet`; they do not update `graph_head` merely to create contention. -The immutable journal remains necessary for snapshots, diffs, audit, migration +The immutable journal remains necessary for snapshots, diffs, audit, rebuild verification, and head repair. Head rows do not replace or truncate it. ## 6. Read contract ### 6.1 Current state -A v5 current-state read: +A heads-format current-state read: 1. derives the expected live stable table IDs from the pinned catalog and fixed system-table registry; @@ -252,29 +277,30 @@ A v5 current-state read: 3. requires exactly one valid row per expected identity; 4. includes only rows whose authoritative state is `live`; 5. validates schema identity from the head payload; -6. returns one immutable `Snapshot` used for the operation's lifetime. +6. opens the exact pinned physical table/ref and validates + `physical_ref_incarnation` before exposing it; +7. returns one immutable `Snapshot` used for the operation's lifetime. Missing, duplicate, unknown-state, or schema-mismatched **live** heads fail loudly. The hot path does not enumerate every identity ever dropped merely to -prove all tombstone heads exist; the branch completion digest plus explicit -`heads verify`/repair owns bounded tombstone-set validation. A missing or +prove all tombstone heads exist; explicit `heads verify`/repair owns tombstone +validation by replaying identity-bearing history offline. A missing or duplicate tombstone is still corruption, but normal reads do not regain an O(history) scan to discover it. ### 6.2 History `snapshot_at_version`, commit resolution, change feeds, and audit continue to -read immutable journal/lineage state at the requested manifest version. A v5 +read immutable journal/lineage state at the requested manifest version. A heads-format manifest version contains the heads as they stood at that version, but the journal remains the normative explanation of how the state arose. ### 6.3 Diagnostic repair -An explicit offline repair loads the branch's v5 incarnation baseline, replays -identity-bearing journal transitions from that baseline, compares the result to -its heads/marker digest, and publishes corrected heads with an audited system -actor. Repair is not part of the read hot path and never silently runs from a -query. +An explicit offline repair replays identity-bearing journal transitions from +the graph's first heads-format manifest version, compares the result to current +heads, and publishes corrected heads with an audited system actor. Repair is not +part of the read hot path and never silently runs from a query. ## 7. Bounded physical lookup is a ship gate @@ -284,7 +310,7 @@ compaction reduces files but not semantic row count. ### 7.1 Required property -At fixed catalog width, a reconciled v5 current-state lookup MUST have zero +At fixed catalog width, a reconciled heads-format current-state lookup MUST have zero positive slope in: - manifest object-store reads; @@ -321,9 +347,10 @@ A separate heads dataset is rejected. Lance commits are per dataset, so it would reintroduce a journal→heads crash gap and require another sidecar protocol for the very pointer whose purpose is to remove drift. -If no in-manifest Lance-native access shape passes the gate, v5 does not ship -with head rows. The fallback is to retain v4 plus the local session/view-passing -improvements, not to waive the cost claim. +If no in-manifest Lance-native access shape passes the gate, the heads format +does not ship. The fallback is to retain the then-current identity-capable +format plus the local session/view-passing improvements, not to waive the cost +claim. ## 8. Recovery protocol @@ -334,142 +361,109 @@ Data-table writers still use the existing four phases: 3. publish `__manifest`; 4. delete sidecar. -The sidecar's logical intent in v5 includes the expected and desired table-head +The sidecar's logical intent in the heads format includes the expected and desired table-head payload. Recovery behavior is therefore complete: - roll-forward publishes the data version, journal row, table head, lineage, and graph head together; - rollback restores the physical version, then publishes journal/audit state and a table head matching the restored logical state; -- a stale sidecar whose goal is already represented by the exact head - incarnation/version converges idempotently; +- a stale sidecar whose goal is already represented by the complete exact head + token converges idempotently; - a partially matching head is not treated as success. Recovery remains a synchronous barrier before any later writer advances a touched table. Index reconciliation may be asynchronous; unresolved commit protocol state may not. -## 9. v5 boundary and compatibility +## 9. Heads-format boundary and compatibility -v5 comprises: +The heads capability comprises: 1. table-head rows and their publish/read semantics; -2. branch-local v5 completion markers; and -3. the graph-level internal-schema stamp written after verification. +2. identity-bearing table journal events from the graph's first valid state; +3. publisher and recovery rules that update those rows atomically; and +4. the graph-level internal-schema stamp that declares the capability. -It does **not** comprise checkpoint/retention markers. +It does **not** comprise identity minting (RFC-028), PK fencing (RFC-023), or +checkpoint/retention markers (RFC-025). -If RFC-023 is independently accepted and ready for the same release, v5 may -also carry its PK annotation and fencing-compatible marker after a combined -migration review. That is release coordination, not a prerequisite of heads. +Serving remains strict-single-version: -Format sequencing is explicit: - -| Order | Result | -|---|---| -| Heads first | proposed v5 contains heads; later fencing migration preserves and atomically updates heads for every PK-version repoint | -| Fencing first | fencing takes the next format number; this heads migration takes the following number, accepts that exact predecessor, and preserves its PK/stamp invariant | -| Co-release | one format maps to both independently accepted capabilities and runs the combined failure matrix | - -Migration code dispatches from an exact supported predecessor feature set; it -never assumes that “pre-heads” means pristine v4 or drops a capability it does -not own. - -After upgrade, serving is strict-single-version: - -- v5 binaries refuse unstamped or partially migrated graphs in normal mode; -- older binaries refuse v5 with the existing upgrade message; +- a heads-capable binary refuses an older graph with the documented + export/import rebuild instructions; +- an older binary refuses the heads-format stamp; - every open reads the graph-wide main stamp before selecting a named branch; -- only the dedicated offline migration command may open the exact supported - predecessor for conversion; -- there is no mixed v4/v5 serving period. - -## 10. All-branch predecessor→heads migration - -### 10.1 Preconditions - -The operator stops every graph writer and acquires an atomic -create-if-absent migration claim (with exact owner/fencing token; not a Lance -branch sentinel). The barrier covers server, CLI, embedded, maintenance, -branch, and schema writes. Because predecessor binaries do not understand the -new in-graph migration marker, the offline fleet barrier remains mandatory -until the final stamp. - -The claim uses `PutMode::Create`, records the migration operation and owner -token, and permits no time-only takeover. Recovery classifies the durable -ledger/sidecars under the fleet outage before replacing a stale token. - -After acquiring the claim, migration completes RFC-022's recovery barrier -before pinning any branch source tip. - -If fencing is co-released, the migration first executes RFC-023's all-branch -table-PK plan. Otherwise head migration neither adds nor changes PK metadata. - -### 10.2 Per-branch conversion - -For each live native `__manifest` branch: - -1. capture branch name, ref incarnation, manifest version, and e_tag/timestamp; -2. run the predecessor journal+tombstone fold once at that pinned tip; -3. construct exactly one live or tombstoned head plus one immutable incarnation - baseline per stable table identity; -4. validate table schema hashes and, only in a combined release, RFC-023 PK state; -5. publish all heads/baselines, an audited migration record - (`actor = omnigraph:migration/v5`), and a branch-local completion marker in - one manifest CAS that revalidates the captured ref incarnation/source tip; - this physical representation migration does not create a graph-content - commit or move `graph_head`; -6. store in the marker the source tip, head count, identity-baseline digest, - head-set digest, and heads-format version; the marker is content-scoped and - deliberately does not embed the source branch incarnation; source tip is - provenance, while digest/format determine inherited-marker validity; -7. verify by reopening the produced branch version through the v5 head reader. - -The completion marker has a deterministic object ID within each manifest -branch. A retry that finds a matching source/digest is complete; a mismatching -marker is a loud migration conflict. - -### 10.3 Finalization - -After all branches report completion, migration re-enumerates native manifest -refs and verifies the same incarnation set. Branch create/delete is blocked, so -any difference indicates out-of-band modification and aborts finalization. - -Only then does it stamp main as internal schema v5. The stamp is the fleet -commit point: before it, no serving process may start; after it, only v5 code may -open the graph. - -New branches created after v5 fork a source manifest that already contains -complete heads and the content-scoped v5 marker. The inherited marker remains -valid for the identical snapshot; branch open separately binds that content to -the new native ref incarnation and validates the graph-wide stamp. No -post-create marker rewrite is required. - -## 11. Migration recovery - -The migration keeps a durable ledger outside the ordinary read path with one -record per branch and, in a combined release, per RFC-023 table conversion. It -records expected source incarnations, achieved physical versions, produced -manifest versions, and digests. - -Recovery is idempotent and roll-forward-only: - -- completed, digest-matching branch conversions are skipped; -- a table HEAD advance not yet represented in its graph branch is recovered by - the normal sidecar and then branch conversion resumes; -- an uncommitted head-row source leaves no visible state and is rebuilt; -- a committed branch marker is authoritative for that branch conversion; -- the main v5 stamp is never written while any ledger unit is incomplete; -- co-released PK metadata is never cleared to simulate rollback. - -If migration crashes, the graph remains offline. Restarting an old serving -binary is not a recovery procedure; the operator resumes the v5 migration. +- missing or partial head state under a heads-format stamp is corruption; +- there is no mixed-format serving period or compatibility reader. + +If RFC-023 or another capability is independently accepted for the same +release, one new format may initialize all of them after the combined +initialization/recovery matrix passes. If capabilities release separately, +each format bump requires its own rebuild under the strand policy. Co-release +is operator-cost coordination, not a dependency between heads and fencing. + +## 10. Fresh graph initialization and rebuild cutover + +### 10.1 First valid target state + +Initialization constructs the identity-capable schema and complete heads before +the graph becomes writable: + +1. RFC-028 mints the target graph identity, stable IDs, and incarnations; +2. every initial node/edge table is created with its accepted logical identity + and exact physical schema; +3. the first valid manifest publish writes one identity-bearing registration + event and matching live head per table, plus ordinary graph authority; +4. initialization verifies exact head cardinality, payload/schema identity, and + any independently accepted co-released capability; +5. only a complete target carrying the new internal-schema stamp may open for + normal writes. + +There is no branch-local completion marker or predecessor-derived incarnation +baseline. A new branch forks a source manifest that already contains complete +heads and identity-bearing history; its native ref incarnation is validated +separately as physical authority. + +### 10.2 Existing graph rebuild + +An existing graph reaches the heads format only through the standard strand: + +1. quiesce and export the selected source branch with a binary that reads the + old format; +2. initialize a different graph root with the heads-capable binary; +3. import through ordinary RFC-022 `load` staging/recovery; +4. verify head/current-state equivalence to the imported rows and any + co-released capability, then cut clients over. + +The source graph is never annotated or stamped by the new binary. The target +starts a new graph identity and contains no predecessor journal to fold. It +preserves current rows, vectors, blobs, and schema shape, but not branches, +commit DAG, snapshots, tombstones, checkpoints, or time-travel history. A source +branch that matters is exported into a separate target graph today. + +## 11. Rebuild failure and ordinary recovery + +A failed target initialization or import does not mutate the source. Before the +target receives a complete format stamp it is not a serveable graph; discard or +repair it and retry. After initialization, import failures use the ordinary +RFC-022 sidecar protocol: + +- unresolved table effects block later writes and recover all-or-nothing; +- head, journal, graph lineage, and graph head publish together; +- a stale sidecar converges only when the exact logical identity/incarnation + and physical outcome match; +- co-released PK metadata or other immutable capability state is never cleared + to simulate rollback to an older format. + +No migration claimant, per-branch conversion ledger, old-format writer mode, or +`omnigraph:migration/*` actor is introduced. ## 12. Tests and acceptance gates ### 12.1 Head semantics -- current state from heads is byte-equivalent to the predecessor fold across realistic +- current state from heads is byte-equivalent to the existing journal fold across realistic histories; - live→tombstoned never resurrects an older live version; - drop/recreate distinguishes incarnations; @@ -477,9 +471,15 @@ binary is not a recovery procedure; the operator resumes the v5 migration. dataset restarts Lance version numbering; - rename preserves identity/incarnation and changes the public key only; - owner-branch handoff at an equal table version updates the head; +- delete/recreate of a dataset or native ref at the same path, branch, and + numeric version changes `physical_ref_incarnation` and rejects a stale + writer, on local FS and S3/RustFS; +- a current read refuses that same replacement until an authoritative publish + selects its exact token; it never opens replacement data under the old head; - missing, duplicate, malformed, and schema-mismatched heads fail loudly. -- `heads verify` detects a missing/duplicate tombstone against the baseline and - completion digest without adding that enumeration to every current read. +- `heads verify` detects a missing/duplicate tombstone by replaying the + identity-bearing journal without adding that enumeration to every current + read. ### 12.2 Publisher and recovery @@ -489,21 +489,26 @@ binary is not a recovery procedure; the operator resumes the v5 migration. matching physical version, journal, table head, and graph head; - rollback and roll-forward assertions include head payloads, not only table versions; +- publisher retry compares the complete expected token and never reparents a + prepared effect across a physical-ref incarnation change; - a stale sidecar converges exactly once with one audit record. -### 12.3 All-branch migration - -- main, owned named branches, and lazy-inherited branches all convert; -- tombstoned types are represented on every relevant branch; -- crashes before/after every per-branch CAS resume from the ledger; -- branch ref deletion/recreation is caught by incarnation verification; -- final stamp is impossible while any table or branch marker is absent; -- each branch's graph head and content lineage are unchanged by head migration; -- old/new binary refusal matrix covers every supported predecessor capability - set, partial heads migration, and complete heads format; -- a post-v5 branch inherits and validates complete heads. -- the inherited completion marker is content-scoped while ref-incarnation - validation is performed separately. +### 12.3 Format and rebuild + +- a genuine old-format graph is refused before head/schema/recovery parsing by + the new binary, and the old binary refuses the heads format; +- old-binary export followed by new-binary init/load produces current state + byte-equivalent to the selected source branch; +- the target's first valid manifest contains exactly one live head and one + identity-bearing registration event per initial table; +- crashes during target init never make a partial target serveable, and import + crashes converge through ordinary RFC-022 recovery; +- a separately rebuilt branch has the documented independent graph identity + and no inherited commit history; +- a post-activation branch inherits complete heads while native ref-incarnation + validation remains separate; +- co-release tests prove every accepted capability exists in the first valid + target state. ### 12.4 Cost gates @@ -527,12 +532,12 @@ gate even when I/O is flat. ### 12.5 Format guards -- exact v5 head metadata schema and object IDs; +- exact heads-format metadata schema and object IDs; - one head row per stable identity; -- incarnation-baseline and identity-bearing journal event schemas; -- content-scoped branch completion marker schema/digest; +- RFC-028 stable-ID/incarnation types plus `physical_ref_incarnation` in head + and identity-bearing journal event schemas; - RFC-023 PK metadata on node and edge tables when the release combines them; -- v5 publisher source always pairs a journal/tombstone event with a head row. +- heads-format publisher source always pairs a journal/tombstone event with a head row. ## 13. Decisions and open gates @@ -542,17 +547,21 @@ gate even when I/O is flat. - Current reads use heads; historical reads keep the journal. - Heads represent `live | tombstoned` plus incarnation explicitly. - A separate heads dataset and a mutable in-process tip authority are rejected. -- Migration is offline, all-branch, resumable, and stamps v5 last. -- RFC-023 PK activation is verified by v5 only when deliberately co-released. +- Existing graphs use strict-strand export/init/load; there is no in-place + all-branch heads migration. +- RFC-023 PK activation is verified at target initialization only when + deliberately co-released. - Checkpoint retention is deferred. ### Open ship gates -1. Rename-stable table/type identity and final stable-ID encoding — owned by - this RFC; consumed by RFC-023 and RFC-025 (§3.1). +1. Accepted and implemented + [RFC-028](0028-stable-schema-identity.md) identity/incarnation contract. 2. The in-manifest indexed lookup implementation and bounded uncovered-tail proof. 3. Passing local and object-store depth-slope cost gates. -4. Atomic cross-process migration-claim implementation and operator runbook. -5. Final v5 metadata JSON/typed-column compatibility review. -6. Full all-branch migration/failpoint matrix. +4. Final heads metadata JSON/typed-column compatibility review. +5. Genuine old/new binary refusal plus export/init/load rebuild evidence. +6. Combined initialization/recovery matrix for any co-released capability. +7. A proven `physical_ref_incarnation` source on every supported backend; no + timestamp/version-only fallback is accepted without an ABA test. diff --git a/docs/rfcs/rfc-025-checkpoint-retention.md b/docs/rfcs/0025-checkpoint-retention.md similarity index 66% rename from docs/rfcs/rfc-025-checkpoint-retention.md rename to docs/rfcs/0025-checkpoint-retention.md index 39571efc..18f09875 100644 --- a/docs/rfcs/rfc-025-checkpoint-retention.md +++ b/docs/rfcs/0025-checkpoint-retention.md @@ -5,19 +5,24 @@ description: Makes named graph checkpoints authoritative retention roots, materi status: draft tags: [eng, rfc, retention, checkpoint, cleanup, manifest, lance, omnigraph] timestamp: 2026-07-10 -owner: +owner: OmniGraph maintainers --- # RFC-025 — Checkpoint-pinned retention **Status:** Draft / for team review **Date:** 2026-07-10 -**Depends on:** [RFC-022](rfc-022-unified-write-path.md)'s publisher and -recovery-sidecar protocol. It composes with -[RFC-024](rfc-024-durable-table-heads.md), but is not part of v5 by implication. -**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.15 (`f24e42c1`) +**Author track:** Maintainer design series +**Depends on:** [RFC-022](0022-unified-write-path.md)'s publisher and +recovery-sidecar protocol and +[RFC-028](0028-stable-schema-identity.md)'s stable table identity/incarnation. +It composes with [RFC-024](0024-durable-table-heads.md), but durable heads are +not required for checkpoint authority. +**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.21 at git rev +`1aec14652dcbace23ac277fa8ced35000bea0c40`; full Lance branch/tag, +transaction, cleanup, and object-store specifications **Audience:** engine, storage, CLI, and operations maintainers -**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md). +**Open architecture review:** [RFC-022–028 review ledger](../dev/rfc-022-027-architecture-review.md). Findings marked **BLOCKER** must be dispositioned before acceptance. --- @@ -51,7 +56,7 @@ This RFC specifies: 2. deterministic Lance tags for the pinned manifest and every pinned table version; 3. create, delete, and reconciliation protocols; 4. how `cleanup` computes and protects its root set; -5. CLI, policy, audit, observability, migration, and acceptance contracts. +5. CLI, policy, audit, observability, rebuild, and acceptance contracts. It does not change snapshot isolation, invent a second transaction log, or replace Lance cleanup. It does not make every historical graph commit a @@ -80,10 +85,23 @@ writes: immutable checkpoint ID; - `checkpoint:<checkpoint_id>` — header containing `name`, source logical branch, `graph_commit_id`, source physical manifest branch and version, - manifest schema stamp, `created_at`, and `actor`; -- `checkpoint_table:<checkpoint_id>:<stable-table-id>:<incarnation-hash>` — one + source-manifest `physical_ref_incarnation`, manifest schema stamp, + `created_at`, and `actor`; +- `checkpoint_table:<checkpoint_id>:<stable-table-id>:<incarnation-id>` — one row per pinned table containing stable table identity, table key, - `table_path`, `table_branch`, `table_version`, and physical incarnation. + `table_path`, `table_branch`, `table_version`, and a separately named + `physical_ref_incarnation` token (e_tag or the proven backend fallback). + +Because RFC-024 heads are optional, RFC-025 owns this token contract for both +the pinned manifest and every pinned table. The token identifies the immutable +physical version object and its dataset/native-ref lineage for the exact +`(path, branch, version)`; it is independent of RFC-028's logical incarnation. +Use a strong object e_tag plus the Lance-native branch/dataset identifier when +the backend exposes them, or another backend-specific token proven to change +when a dataset/ref/version is deleted and recreated at the same coordinates. +Timestamp, path, branch name, and numeric Lance version alone are not proofs. +A backend without a token that passes local and S3/RustFS ABA guards cannot +activate the retention format. The name reservation, header, and every table row land in one RFC-022 publisher CAS on main. First creation is insert-only; reuse requires the exact tombstoned @@ -92,17 +110,18 @@ normalized name therefore conflict even when they captured different source branches. A missing or duplicate table row is an invalid checkpoint, not a partial checkpoint. -Stable table identity/incarnation is a ship gate even when RFC-024 has not -landed; checkpoint rows cannot key retention to mutable names. A deployment may -reuse RFC-024's identity baseline, but durable heads themselves are not a -dependency. +[RFC-028](0028-stable-schema-identity.md)'s stable table ID and incarnation are +a ship gate; checkpoint rows cannot key retention to mutable names. RFC-024 may +provide the same manifest's bounded head/index access machinery when it lands, +but it does not own or supply checkpoint identity. Deletion changes the name reservation from `live` to `tombstoned` and writes manifest tombstones for the checkpoint header and all table rows in one CAS. The reservation and every tombstone carry the same fresh `delete_operation_id`; that CAS is the durable delete/recovery marker. The retention claim may repeat the operation ID while the writer is live, but it is -only a fence, never recovery authority. Checkpoint IDs are never reused. +only a live-owner serialization claim, never a fencing token or recovery +authority. Checkpoint IDs are never reused. Checkpoint create/delete are manifest metadata transactions. They do not create a graph-content commit or move `graph_head`: taking a checkpoint must not change @@ -162,22 +181,26 @@ explicitly rejected because its branch-metadata step is an existence check followed by an unconditional put. The claim payload records operation ID, action, actor, creation time, and a -random fencing/owner token. Every protected phase re-reads and verifies that -token. Normal release verifies the exact token before deleting the claim; no -other owner can replace it while create-if-absent observes the object. There is -no time-based lease stealing. Crash takeover requires an explicit fleet write -outage, classification of the sidecar/manifest operation marker, removal of the -stale claim, and acquisition with a new token. A resumed stale process must -fail its next token check. +random owner token. It is an ownership check, **not** a fencing token. Every +protected phase re-reads and verifies it, and normal release verifies the exact +token before deleting the claim. There is no time-based lease stealing and no +supported takeover while the prior process or host could resume. + +A crash leaves a non-takeoverable claim and safely over-retains. Operator +recovery first establishes external process/host death or runs an explicit +offline repair in an environment where the old credentials/process cannot +resume; only then may it classify the sidecar/manifest operation marker, remove +the stale claim, and acquire a new token. Merely declaring a fleet write outage, +waiting, or observing an old timestamp is not death proof. Without that proof, +recovery refuses and leaves the claim in place. A process-local mutex is only a contention optimization. The retention claim is held across tag creation or physical cleanup, preventing this race: cleanup computes roots, another process captures an untagged old version, then cleanup -deletes it before the tag lands. A crash leaves the claim and therefore -over-retains. Takeover first classifies the prior operation from its recovery -sidecar and/or exact manifest operation marker, then may release the claim -only after that operation is completed or safely aborted. It never steals on -elapsed wall time alone. +deletes it before the tag lands. After external death proof, offline repair +classifies the prior operation from its recovery sidecar and/or exact manifest +operation marker and may release the claim only after that operation is +completed or safely aborted. The claim is not presented as a distributed reader/writer lock for arbitrary graph writes. Initial destructive cleanup is an **offline** operation: operators @@ -195,9 +218,9 @@ Checkpoint creation is a writer on the RFC-022 pipeline: 2. Authorize, capture one immutable source-branch manifest snapshot and graph commit, and prepare the complete header/table/tag set plus a `ReadSet` that includes source branch incarnation, source manifest version, format/schema - identity, applicable GC boundaries, and the main-registry name reservation; - a source at or behind a pruned-through boundary is refused even if its files - happen to remain. + identity, the source-manifest and every table physical-ref token, applicable + GC boundaries, and the main-registry name reservation; a source at or behind + a pruned-through boundary is refused even if its files happen to remain. 3. Acquire the retention claim, checkpoint/cleanup process-local serialization, source branch-control gate, and adapter-declared metadata gates in RFC-022's canonical order. Ordinary data queues are not held merely because immutable @@ -205,9 +228,12 @@ Checkpoint creation is a writer on the RFC-022 pipeline: 4. Freshly revalidate the complete `ReadSet`. A changed branch incarnation, missing tag target, conflicting name, or format change releases the gates and restarts before any tag exists. -5. Write a generic recovery sidecar containing the intended rows and tags. -6. Create the manifest tag and every table tag idempotently. Existing correct - tags are no-ops; conflicting tags fail. +5. Write a generic recovery sidecar containing the intended rows, tags, and + exact physical-ref tokens. +6. Reopen and revalidate every exact version/token, then create the manifest + tag and every table tag idempotently. Existing tags are no-ops only when + their target and physical-ref token match; conflicting or recreated targets + fail closed. 7. Verify every tag, then release source branch control. A source-head advance is harmless because the tags already pin the exact captured version; source-branch deletion takes the same retention claim, classifies overlapping @@ -218,8 +244,10 @@ Checkpoint creation is a writer on the RFC-022 pipeline: 10. Release the retention claim after the outcome is durably classifiable. A crash before step 8 leaves tags but no checkpoint. Recovery may complete the -publish when every precondition still holds or remove the orphan tags. A crash -after step 8 leaves a valid checkpoint even if sidecar cleanup did not finish. +publish only when every exact version/token precondition still holds, or remove +only tags proven to belong to that sidecar. A same-path/ref/version replacement +is never adopted; ambiguity fails closed and safely over-retains. A crash after +step 8 leaves a valid checkpoint even if sidecar cleanup did not finish. ## 4. Checkpoint delete protocol @@ -252,8 +280,10 @@ The reconciler: 1. reads the live checkpoint rows once; 2. classifies checkpoint sidecars before touching apparently orphaned tags; -3. creates or verifies every required tag; -4. deletes internal tags proven to have no live or in-flight authority; +3. reopens every pinned manifest/table version and validates its recorded + physical-ref token before creating or verifying the required tag; +4. deletes internal tags only when their exact target/token is proven to have + no live or in-flight authority; 5. emits a typed result for repaired pins, reclaimed pins, and failures. `cleanup` is fail-closed: any missing, conflicting, unreadable, or ambiguous @@ -267,6 +297,22 @@ tag belonging to a foreign process's live create sidecar. ## 6. Cleanup protocol and pruned-through boundary +The retention format stores exactly one current +`gc_boundary:<dataset-lineage-hash>` row for every registered cleanup-eligible +manifest or data-table lineage. Target initialization creates those rows with +`cutoff = 0`, `cleanup_operation_id = null`, and `advanced_at = null`. A later +AddType, physical rematerialization, or new cleanup-eligible lineage creates its +own zero row in the same SchemaApply/registration publish that makes the +lineage live; it never inherits another physical lineage's cutoff. A removed or +rematerialized old lineage retains its row while any branch, checkpoint, +recovery intent, or physical cleanup can still address it. The row is +tombstoned only after that lineage is provably unreachable and reclaimed under +the retention claim. Missing, duplicate, or mismatched rows are corruption and +fail cleanup/recovery closed—absence is not an implicit zero. The lineage hash +excludes mutable HEAD version but includes the dataset/ref lineage component of +§2.1's physical-ref token, so ordinary commits retain the row while +delete/recreate cannot revive it. + The cleanup root set is: - every live graph branch head; @@ -287,11 +333,13 @@ must be repaired before retention work. Read-only preview may run without a claim, but it is explicitly provisional. Confirmed execution uses this order: -1. establish the operator write outage, persistently seal/fold streams through - RFC-026 **before** taking the retention claim, and complete the RFC-022 - recovery barrier; -2. acquire the retention claim, then revalidate the fleet outage, every stream's - `SEALED` cut, and absence of recovery sidecars; +1. establish the operator write outage; when the graph format includes RFC-026 + and any target table is stream-enrolled, persistently seal/fold those streams + **before** taking the retention claim; then complete the RFC-022 recovery + barrier; +2. acquire the retention claim, then revalidate the fleet outage, absence of + recovery sidecars, and—only when streaming is present—the exact `SEALED` + cuts; a retention-only graph has no stream gate or RFC-026 dependency; 3. run pin reconciliation and recompute the exact root set and per-dataset cutoffs under the claim, including the oldest live lazy-branch inherited-main floor above; @@ -321,9 +369,10 @@ never starts with an armed recovery that may still need an older version. Per-table cleanup remains fault-isolated, but partial operational success is reported explicitly. A successful table does not hide another table's error. The claim metadata plus `gc_boundary` operation ID is the cleanup recovery -marker: takeover resumes or reports the remaining table units under the same -root digest before releasing the claim; it does not silently recompute a -broader destructive plan after a partial run. +marker: after externally proven owner death, authorized offline recovery +resumes or reports the remaining table units under the same root digest before +releasing the claim; it does not silently recompute a broader destructive plan +after a partial run. ### 6.1 Operational cadence — the cost of never running this @@ -366,27 +415,45 @@ hit the same engine gate as the CLI. HTTP management endpoints are out of scope until server-side maintenance has a general design; no transport-only bypass is introduced here. -## 8. Migration and compatibility +## 8. Format activation and rebuild compatibility This RFC owns its own internal-format activation. RFC-022 authorizes no format bump, and RFC-024 explicitly excludes checkpoint rows from its heads format. -The retention format may receive the next schema version after heads, or the -two may share one release only if both RFCs are independently accepted and -RFC-024 is amended before implementation. A fresh activated graph starts with -no named checkpoints and a zero pruned-through boundary. No older commit is -silently promoted into a checkpoint. - -The upgrade mechanism must be chosen before implementation: - -- an in-place upgrade preserves branches and history and must define quiescence, - crash recovery, stamp ordering, and rollback; -- export/import creates a fresh graph at the new format but loses the old commit - DAG, branches, snapshots, and time-travel history. Those losses must be shown - in the plan and confirmation output; there is nothing left to backfill as a - checkpoint. - -Mixed-version writers are unsupported. An old binary must not write after the -retention format stamp or tag protocol becomes authoritative. +Retention may therefore be the next independently accepted format capability +after RFC-028 even when durable heads are absent. A heads-free target writes +RFC-028 identity plus retention rows in its first valid state and resolves the +bounded checkpoint registry through retention's own structured lookup; it does +not rely on table-head rows. If heads already exist in the source format, a +later retention target preserves that accepted capability and initializes the +combined format. Heads and retention may also co-release only after both RFCs +are independently accepted and the combined initialization, lookup, refusal, +recovery, and rebuild matrix passes. A fresh activated graph starts with no +named checkpoints and the explicit per-lineage zero boundary rows defined in +§6. No older commit is silently promoted into a checkpoint. + +Activation follows the existing strict-single-version strand. A retention- +capable binary refuses an older graph before checkpoint/tag logic runs; an old +binary refuses the retention-format stamp. Existing graphs are exported with a +binary that reads the source format, then loaded into a separately initialized +target graph whose first valid state already carries the retention capability. +There is no in-place checkpoint backfill, predecessor dispatcher, or partial +activation. + +The rebuilt target starts with zero named checkpoints and explicit zero +pruned-through rows for every initial cleanup-eligible lineage. It preserves +the selected branch's current rows, vectors, blobs, and schema shape, but loses +the old commit DAG, branches, snapshots, tombstones, recovery intents, recovery +audit/history, time-travel history, and historical checkpoints. The source +recovery barrier must resolve every active intent before export; completed +recovery history is not transferred. An important checkpoint or branch snapshot +must be exported into a separate target graph before cutover. Those losses +appear in plan/confirmation output. A failed target rebuild does not mutate the +source. + +Mixed-version writers are unsupported. If retention co-releases with heads or +another independently accepted capability, the first valid target state +contains all of them and the combined initialization/recovery matrix must pass. +Separate format releases imply separate rebuilds. ## 9. Observability and bounds @@ -422,6 +489,10 @@ share the main-manifest CAS. to bypass the pin. - The source `__manifest` version survives cleanup; data-table tags without the manifest tag fail the checkpoint-validity test. +- Local and S3/RustFS delete/recreate races reuse the same manifest/table + path, branch name, and numeric version with a different physical-ref token; + checkpoint create, recovery, and reconciliation all refuse the replacement. + A backend without a proven token refuses retention-format activation. - Concurrent creation of the same normalized name yields exactly one authority record; the losing attempt leaves only safely reclaimable tags. - After deletion, concurrent attempts to reuse the tombstoned name generation @@ -431,16 +502,26 @@ share the main-manifest CAS. converges to valid pins or safe over-retention. - Missing live tags are repaired before cleanup; conflicting tags block cleanup. - The retention claim blocks checkpoint create/delete/reconcile and branch delete - during cleanup on local FS and S3; destructive cleanup refuses active streams - or recovery sidecars. -- Two-process races prove the atomic retention claim, not a process-local gate, - closes checkpoint-vs-branch-delete windows; crash takeover never relies on - wall-clock expiry alone. + during cleanup on local FS and S3; destructive cleanup refuses recovery + sidecars and, when streaming is an active capability, non-`SEALED` streams. +- Initialization, AddType, and rematerialization tests create the exact + per-lineage boundary row; removal retains it through the final reachable + checkpoint/recovery/physical cleanup and tombstones it only afterward. + Missing/duplicate rows fail closed and a new physical lineage never inherits + an old nonzero cutoff. +- Two-process races prove that the atomic retention claim, not a process-local + gate, closes checkpoint-vs-branch-delete windows; a paused live owner causes + takeover to be refused even after its final token check. - Local and S3 claim tests prove `PutMode::Create` admits exactly one owner and - that mismatched owner tokens cannot release another operation's claim. + that mismatched owner tokens cannot perform a normal release. A subprocess + death/offline-repair test is required before stale-claim removal. - A documented fleet write-outage test runs writers before and after cleanup; online writer-vs-cleanup concurrency is explicitly not advertised. -- Genuine cross-version tests cover the selected upgrade path. +- Genuine cross-version tests prove old/new mutual refusal and old-binary + export followed by new-binary init/load, including the documented loss of + checkpoints, branches, commit DAG, snapshots, tombstones, recovery + intents/audit/history, and time travel; export refuses an active recovery + intent. - Cost tests use `helpers::cost` at realistic commit depth and prove that a steady checkpoint list/lookup and cleanup plan do not grow with commit count once physical layout is held constant, including a bounded uncovered tail. @@ -449,7 +530,7 @@ share the main-manifest CAS. | Phase | Content | Gate | |---|---|---| -| A | row schemas, engine DTOs, format activation, deterministic tag encoding | schema and tag surface guards | +| A | row schemas, engine DTOs, strict-format/rebuild activation, deterministic tag encoding | schema, refusal, and tag surface guards | | B | create sidecar, delete-operation fields in the authority CAS, and reconciler | crash matrix; sparse-pin correctness | | C | offline cleanup integration and GC boundary enforcement | quiescence/refusal tests; cost budgets | -| D | CLI, policy, audit, docs, and selected migration path | CLI outputs; genuine upgrade test | +| D | CLI, policy, audit, docs, and rebuild runbook | CLI outputs; genuine upgrade test | diff --git a/docs/rfcs/0026-memwal-streaming-ingest.md b/docs/rfcs/0026-memwal-streaming-ingest.md new file mode 100644 index 00000000..f74db9df --- /dev/null +++ b/docs/rfcs/0026-memwal-streaming-ingest.md @@ -0,0 +1,690 @@ +--- +type: spec +title: "RFC-026 — MemWAL streaming ingest" +description: Adopts Lance MemWAL as OmniGraph's strategic streaming-write architecture, with durable per-row acknowledgement, graph-atomic folds, epoch-fenced quiescence, and explicit fresh-read cuts on the RFC-022 unified write path. +status: draft +tags: [eng, rfc, streaming, ingest, wal, memwal, lance, omnigraph] +timestamp: 2026-07-10 +owner: OmniGraph maintainers +--- + +# RFC-026 — MemWAL streaming ingest + +**Status:** Draft / for team review +**Date:** 2026-07-10 +**Author track:** Maintainer design series +**Depends on:** [RFC-022](0022-unified-write-path.md)'s unified write and +generic recovery-sidecar protocol, plus +[RFC-028](0028-stable-schema-identity.md)'s stable table identity and +incarnation contract, plus +[RFC-023](0023-key-conflict-fencing.md) for the initial keyed graph-stream +mode. Durable heads from +[RFC-024](0024-durable-table-heads.md) are compatible but not required. +**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.21 at git rev +`1aec14652dcbace23ac277fa8ced35000bea0c40`; complete MemWAL table and +system-index specifications, including the durability and writer-fencing +changes carried forward from beta.17 +**Audience:** engine, server, CLI, policy, and operations maintainers +**Open architecture review:** [RFC-022–028 review ledger](../dev/rfc-022-027-architecture-review.md). +Findings marked **BLOCKER** must be dispositioned before acceptance. + +--- + +## 0. Decision and risk posture + +OmniGraph adopts Lance MemWAL as its strategic streaming-write architecture. +MemWAL is a major Lance architectural bet: a sharded LSM write path with durable +WAL entries, flushed Lance generations, merge progress committed with base-table +data, maintained indexes, and epoch-fenced writers. OmniGraph consumes that +architecture rather than building a WAL, shard protocol, or LSM reader. + +This RFC does **not** characterize the architecture as experimental. The risk is +narrower: Rust API names, some format details, and operational helpers are still +maturing across Lance releases. We manage that API/format-maturity risk with a +small adapter, compile/runtime surface guards, a quiescence requirement before +Lance upgrades, and a fresh full-spec alignment audit on every bump. It is not a +reason to fork or reimplement MemWAL. + +One beta.21 API gap is a ship blocker, not evidence against the architecture: +`InitializeMemWalBuilder::execute` internally commits the MemWAL `CreateIndex` +transaction and returns only `Result<()>`, while initial shard-manifest creation +is a separate object-store effect reached through `mem_wal_writer`. The public +surface therefore cannot yet provide the exact pre-arm effect identities and +reversible admission seal required by §3/§8. OmniGraph does not reach through +private Lance modules or hand-roll those objects. Streaming activation waits +for the public recoverable-enrollment/seal gate defined below. + +The contract is: + +- stream acknowledgement means the row's WAL entry is durable; +- acknowledgement does not mean graph visibility; +- default queries see only the manifest-committed graph; +- a fold is an ordinary RFC-022 graph writer and is the sole visibility point; +- fresh reads are explicit and never claim cross-table atomicity. + +## 1. Scope and non-goals + +This RFC specifies enrollment, the public stream API, acknowledgement semantics, +folding, fold-time integrity, dead-letter atomicity, branch/schema quiescence, +fresh-read cuts, resource bounds, observability, testing, and upgrade posture. + +It does not replace `load` or `mutate`, provide cross-query transactions, store +manifest mutations in MemWAL, create a second metadata authority, or weaken +default snapshot isolation. Stream-mode +deletes remain out of the first delivery and require the Lance tombstone surface +plus a separate acceptance pass. + +## 2. Stream mode and key semantics + +Initial delivery exposes +`@stream(mode="upsert", on_reject="strict")` on a node or edge type; +`on_reject` accepts `strict` or `dead_letter`. +It requires the table's immutable unenforced primary key to equal OmniGraph's +merge key: `id` for nodes and edges. All occurrences of one key map to one shard +and MemWAL applies last-write-wins ordering. + +Every stream contract is bound to RFC-028's +`(stable_table_id, incarnation_id)` pair. A rename preserves that pair, so the +logical stream contract and ownership of reject history remain continuous. +That continuity does **not** authorize adoption of physical WAL artifacts. A +same-dataset rename preserves the current physical enrollment; a rename or +other SchemaApply that rematerializes the table binds the preserved logical +pair to a fresh physical enrollment and fresh shard namespace through §3 and +§8. Dropping and later re-adding the same name mints a new pair; the new table +cannot adopt the old table's WAL, lifecycle row, reject rows, epochs, or merge +progress. + +Public append mode is deliberately out of scope. Nodes and edges always have +logical identity; allowing a retry to append the same `id` twice would violate +that contract. A future explicitly keyless, non-graph append-only table class +may consume MemWAL append semantics under its own schema/API decision. + +Stream ordering intentionally differs from the interactive fence: + +- concurrent interactive same-key writes serialize or fail/retry loudly; +- same-key stream entries resolve by MemWAL generation/position order; +- duplicate keys inside one bulk-load input retain the existing load error. + +The schema and user docs state all three together. + +## 3. Enrollment is a recoverable multi-effect adapter + +Schema apply records `@stream` intent only. First stream use enrolls the physical +table by creating the singleton `__lance_mem_wal` system index and its sharding +configuration. + +RFC-024 heads are optional, so this RFC carries its own exact physical binding +in every lifecycle row: + +```text +StreamPhysicalBinding { + table_location, + table_branch, + physical_ref_incarnation, + enrollment_id, // pre-minted UUID, never reused + shard_ids, // sorted UUID namespace, never reused + stream_config_hash, +} +``` + +The binding is valid only when the current manifest table entry resolves to the +exact location/ref incarnation and that physical table contains the expected +MemWAL system index plus shard manifests for the recorded namespace. +`physical_ref_incarnation` is an opaque Lance/backend proof that remains stable +under ordinary HEAD advances but changes when the dataset or native ref is +deleted and recreated at the same path/name. For a non-main ref it includes the +native `BranchIdentifier`; for main it requires an equivalent dataset-root +incarnation proof. Path, branch name, numeric version, timestamp, and the +RFC-028 logical incarnation are not substitutes. A backend without a token +that passes local and S3/RustFS same-path/ref ABA guards cannot activate +streaming. + +The MemWAL index UUID is not used as enrollment identity because Lance may +replace that metadata entry while advancing merged-generation state. A mutable +table name, the RFC-028 logical pair alone, or a compatible-looking index is +never enough. RFC-024 may project related table/ref facts into a durable head, +but RFC-026 persists and validates this binding without depending on heads. + +Enrollment advances Lance HEAD and provisions shard-manifest objects. Before +implementation, Lance must expose one of these public, surface-guarded shapes: + +1. a caller-controlled staged/uncommitted MemWAL initialization transaction + with an exact transaction identity, plus idempotent public APIs to provision, + classify, seal/reopen, and reclaim a pre-minted shard manifest; or +2. one recoverable enrollment primitive that returns an exact receipt covering + both the index transaction and initial shard objects, with documented + classify/roll-forward/rollback semantics. + +The beta.21 `execute() -> Result<()>` initializer plus separately claimed shard +writer satisfies neither shape. Private-module access, compatible-looking +index inference, and direct object-store emulation are rejected. Until this +gate lands, `@stream` may be parsed/planned in draft work but the streaming +format and first enrollment do not activate. + +Once that public gate exists, enrollment uses one RFC-022 multi-effect sidecar, +not an ad-hoc state machine: + +1. run and await RFC-022's synchronous recovery barrier; +2. authorize, pin the manifest/schema/table state, and prepare a complete + `ReadSet` containing schema identity, stable table ID and incarnation, exact + physical-ref incarnation and pre-enrollment HEAD, PK metadata, stream + intent/configuration, and lifecycle-row absence for a first enrollment or + the exact `SEALED` prior binding for a physical rebind; +3. acquire any global claims and then the `(table, branch)` write queue in + RFC-022 order, then freshly revalidate the complete `ReadSet`; a mismatch + restarts before any physical effect; +4. verify RFC-023's already-installed PK; enrollment never performs a first-use + PK migration; validate the sharding configuration; +5. pre-mint a never-reused enrollment UUID and shard UUID namespace, then arm a + sidecar with writer kind `stream_enrollment` or `stream_rebind`, the exact + target/ref token, the index transaction/receipt, the deterministic initial + shard object set in physically non-admitting `SEALED` state, and the complete + intended `StreamPhysicalBinding`; +6. commit the exact index effect, then provision the exact `SEALED` shard + objects; neither effect admits appends; +7. classify both effects and mark the sidecar `EffectsConfirmed` only when the + complete intended set exists with exact ownership; +8. publish the achieved table version, exact physical binding, per-shard epoch + floors, and `stream_state = OPEN` in one manifest CAS, including the + table-head row when RFC-024 is active; +9. activate the bound shard claims idempotently, then delete the sidecar + best-effort. Ack paths run the recovery barrier and refuse a still-covered + enrollment, so `OPEN` plus an unresolved sidecar cannot acknowledge. + +Recovery rolls forward only after classifying both exact effects. Rollback +reclaims the exact empty owned shard objects first, then restores/removes the +owned index transaction; a foreign object, a nonempty shard, or an effect buried +by another transaction fails closed. It never infers ownership from the +preserved logical pair, a table name/path, or a compatible-looking index. +Repeating enrollment with the same exact binding is a no-op. A different PK, +physical binding, sharding spec, maintained-index set, or writer-default +configuration is a typed conflict. + +No row is acknowledged until enrollment is manifest-committed, every sidecar +effect is resolved, and the exact bound shard is physically admitting writes. + +Initial delivery supports one unsharded shard per `(table, main)`. Non-main +branches remain refused until the Lance branch-scoping question is proven by a +surface guard and end-to-end test. Later `bucket(id, N)` sharding must preserve +the one-key-to-one-shard rule. + +## 4. New public API + +The shipped `POST /graphs/{id}/ingest` path remains the deprecated, compatible +alias of `/load`. Streaming receives a new, non-conflicting surface: + +```text +POST /graphs/{graph_id}/streams/{type_name}/ingest?branch=main +GET /graphs/{graph_id}/streams +GET /graphs/{graph_id}/streams/{type_name} +POST /graphs/{graph_id}/streams/{type_name}/fold +POST /graphs/{graph_id}/streams/{type_name}/quiesce +POST /graphs/{graph_id}/streams/{type_name}/resume +``` + +The ingest request and response use `Content-Type: application/x-ndjson` and +`Accept: application/x-ndjson`. + +Each input line is one row payload. Each output line corresponds to the same +input ordinal: + +```json +{"ordinal":17,"status":"durable","shard_id":"...","writer_epoch":8,"wal_position":42} +``` + +Synchronous validation failures return a per-row error before a WAL append. +Previously acknowledged rows in the same request remain durable; the response +is a stream, not an all-request transaction. Ordering, cancellation, and retry +rules are explicit: + +- acknowledgements are emitted in input order for one HTTP stream; +- disconnecting does not cancel entries whose durability waiter resolved; +- a missing response is ambiguous; retrying the same `id` and payload may add + another WAL entry but produces the same last-write-wins graph state; +- server shutdown stops admission, drains durability waiters up to a bound, and + reports any unacknowledged tail as unknown to the client. + +CLI commands mirror the new namespace rather than overloading deprecated +`omnigraph ingest`: + +```text +omnigraph stream ingest <type> --data <ndjson> ... +omnigraph stream status [<type>] ... +omnigraph stream fold [<type>] ... +omnigraph stream quiesce [<type>] ... +omnigraph stream resume [<type>] ... +``` + +Every endpoint has a dedicated OpenAPI operation and handler tests. Ingest +passes the engine `stream_ingest` Cedar action and per-actor admission +accounting before acquiring a shard writer; fold/quiesce/resume use a separate +`stream_manage` action. Status is authorized like other graph operational +metadata. The same engine gates apply to embedded and remote CLI use. + +## 5. Ack-path validation and writer lifecycle + +Before append, OmniGraph applies checks that need no base-table read: Arrow +shape/type, required/default fields, enum/range/check constraints, reserved +columns, and stream mode. RI, cardinality, cross-version uniqueness, and +external embedding computation remain fold-time work. + +One warm `ShardWriter` is held per active shard behind a bounded registry. The +registry has idle eviction and hard limits for resident writers, MemTable bytes, +unflushed WAL bytes, pending generations, and per-actor inflight bytes. Exceeding +a bound backpressures with a typed retryable response; it never drops a row. + +Admission is keyed by the exact +`(stable_table_id, incarnation_id, enrollment_id, shard_id)` binding. Before +claiming a writer, the append path captures `OPEN`, the complete physical +binding, and that shard's epoch floor. After `mem_wal_writer` claims an epoch, +it re-reads the lifecycle row and physical shard status immediately before the +first `put`: the binding must still be identical, state must still be `OPEN`, +the shard must be admitting, and the claimed epoch must exceed the recorded +floor for that same shard. A mismatch closes the writer and returns a typed +retry without appending. + +That check is paired with §8's substrate admission seal. After publishing +`DRAINING`, the drainer atomically prevents later claims and advances the epoch +of each existing shard. A writer that claimed before the seal either put first +and is included in the in-flight durability drain, or is fenced before its put; +a claimant after the seal is refused by the substrate. A check-then-put against +manifest state alone is insufficient. The public claim-seal/reopen primitive is +therefore part of §3's ship gate, not an optional optimization. + +Initial topology has one active ingest owner for each `(graph, table, main)` +shard. MemWAL's epoch fence makes restart/failover safe; it is not a load +balancer. A deployment with multiple server replicas must route a shard to its +current owner (or return a typed retry/redirect) instead of letting replicas +reclaim the epoch per request. General multi-owner routing waits for the +multi-shard phase and its ownership protocol. + +## 6. Fold protocol + +The fold consumes flushed generations in ascending order. Embeddings and +base-dependent validation run outside the table queue. With or without +RFC-024, the `ReadSet` carries schema identity, the complete stream +binding/configuration/generation cut, every probed table, and the conservative +branch authority token `(native branch incarnation, optional graph_head)`. +Absence of `graph_head` on a fresh branch is part of that token. Any publisher +retry compares the captured token and returns to full fold revalidation on a +change; it never reparents a validation-sensitive fold around a concurrent +commit. RFC-024 may later narrow false contention with table heads but is not a +correctness dependency. The commit phase then: + +1. stages accepted rows with Lance merge-insert and includes + `merged_generations` in that transaction; +2. stages any rejection/audit rows required by §7; +3. acquires every affected queue in canonical sorted order and revalidates the + complete `ReadSet`; any mismatch discards and replans the whole fold; +4. writes one generic RFC-022 recovery sidecar before the first + `commit_staged` call; +5. commits every staged Lance transaction; +6. publishes all data/internal table versions and lineage in one `__manifest` + CAS, including table heads when RFC-024 is active; +7. deletes the sidecar after successful publication. + +The sidecar is mandatory even though merge-insert is staged. After +`commit_staged`, Lance HEAD and `merged_generations` have moved while the graph +manifest has not. A failure in that window is the ordinary multi-table recovery +gap, not invisible staged state. + +A commit-time key conflict follows RFC-023's partial-effect rule. If exact +classification proves that no fold participant advanced, the fold finalizes +the empty sidecar and may perform one bounded full replan from fresh authority. +If any participant advanced, or emptiness cannot be proved, it keeps the +sidecar and returns `RecoveryRequired`. No later fold is prepared until the +synchronous recovery barrier resolves that exact attempt; `merged_generations` +is never replanned around an unresolved partial fold. + +MemWAL generation GC starts only after the exact fold is graph-visible, its +sidecar is resolved, index catchup permits reclamation, and no `FreshReadCut` +retention guard references the generation. Data-HEAD merge progress alone is +never permission to delete the only fresh-tier copy. + +Concurrent folders reload `merged_generations`: a generation already committed +is skipped; otherwise the fold is replanned from current state. A forced fold +stops new flush creation for its cut, waits for in-flight durability waiters, +and never aborts an acknowledged row. + +Fold lineage uses `omnigraph:ingest` as the mechanism actor and persists the +authenticated contributor actor for every folded WAL range in the same internal +audit participant. Commit and status output can therefore answer both who ran +the fold and who supplied the data after WAL GC. + +## 7. Fold-time rejection is atomic + +`strict` is the default. A permanent RI, cardinality, uniqueness, or embedding +failure stops the fold at the offending generation, marks the shard blocked, +and backpressures new ingestion once configured lag bounds are reached. + +`dead_letter` is explicit. `_ingest_rejects` is then a versioned internal Lance +table and a participant in the same fold pipeline, not best-effort state outside +the commit protocol. Every reject has deterministic identity: + +```text +(stable_table_id, incarnation_id, shard_id, generation, wal_position) +``` + +The key never uses mutable `table_key`, type name, or path. A rename therefore +preserves ownership of reject history, while a same-name replacement cannot +claim the predecessor's rows. A same-dataset rename retains the shard namespace +and replay positions. A rematerializing rename uses §3's fresh never-reused +shard UUID namespace, so its generation/WAL positions cannot collide with the +old physical enrollment even though the logical table pair is preserved. + +The fold stages reject rows, accepted rows, and merge progress before committing +any of them. The generic sidecar covers every participant; the single manifest +publish records both the base-table and reject-table versions. Replay is +idempotent by reject identity. There is no ordering in which progress can become +visible while the corresponding rejection is lost. + +`stream status` reports blocked generations and typed reject details. Reject +retention is explicit and cannot be shorter than the WAL/fold audit retention +needed to explain a durable acknowledgement. + +## 8. Epoch-fenced quiescence barrier + +Branch operations, schema changes, stream teardown, and Lance upgrades require a +real barrier, not an empty check. + +Each enrolled table has a durable +`stream_state:<stable-table-id>:<incarnation-id>` row in its manifest branch with +`OPEN | DRAINING | SEALED`, configuration hash, an +`epoch_floor_by_shard: Map<shard_id, u64>`, and the §3 physical binding. Epochs +are comparable only within the same enrollment and shard ID; a fresh shard in a +new binding may start at 1 and is fenced from its predecessor by enrollment and +shard identity, not by a larger number. The row is the logical lifecycle +authority and is updated by an RFC-022 CAS; MemWAL shard epochs and admission +status are the physical writer fence. Neither an in-memory registry nor an +empty-generation observation can substitute for both. +Lifecycle-only transitions are audited manifest metadata transactions; they do +not create graph-content commits or move `graph_head`. + +The shared drain sequence is: + +1. publish stream intent `OPEN -> DRAINING` for the exact binding with a target + epoch floor for every current shard; +2. through the public substrate primitive required by §3, seal admission for + each shard and advance its epoch to at least that target, fencing stale + writers and refusing later claims across processes; +3. reject or backpressure new appends; §5's post-claim check rejects a claimant + that entered between steps 1 and 2, while the physical seal closes the final + check-to-put race; +4. wait for in-flight durability waiters, flush active MemTables, and fold every + generation to empty; +5. verify shard manifests and base `merged_generations` agree on emptiness; +6. publish `DRAINING -> SEALED` with the verified generation cut and exact + achieved per-shard epoch map; +7. for an operation-scoped drain, perform the guarded operation; persistent + public quiesce stops after step 6. + +`OPEN -> DRAINING` and `DRAINING -> SEALED` are RFC-022 authority-first +metadata writes; each fold is a separate normal RFC-022 graph write. +`DRAINING` fully encodes every target per-shard floor, so interrupted physical +seals are idempotently resumed from that row. A `SEALED -> OPEN` transition is +different: an exact `stream_resume` sidecar covers the physical reopen and is +resolved before any ack path proceeds, so no naked admitting effect precedes +the CAS. The drain itself is not one giant sidecar spanning multiple commits. + +There are two dispositions after the drain reaches `SEALED`: + +- **operation-scoped drain** — branch/schema maintenance automatically publishes + `SEALED -> OPEN` only after the guarded operation succeeds, the stream + contract remains compatible, and the §3 physical binding still names the + exact table/ref. Under an exact resume sidecar, reopening the same binding + advances each same-shard epoch above its recorded floor before the CAS; a + pre-CAS failure reseals those exact shards. If the operation rematerialized + the table or changed/recreated its native ref incarnation, it must complete + `stream_rebind` instead of applying this transition directly; +- **persistent quiesce** — the public `quiesce` command leaves the stream + `SEALED`. It never auto-reopens. `stream resume` explicitly revalidates schema, + PK, configuration, MemWAL format, physical binding, and every same-shard + epoch, then publishes `OPEN`. Stream teardown deletes intent only from + `SEALED`. + +The barrier never holds the table write queue while waiting for a fold that +needs that queue. State transition and epoch fencing happen first; fold commit +then acquires the normal queue. Crash recovery resumes from the durable state +and per-shard epoch map. + +Schema apply must drain every affected enrolled type before changing fields, +constraints, PK, embeddings, or `@stream` and resumes only when compatible. A +rematerializing type rename preserves RFC-028's logical pair but not the old +physical enrollment: + +1. drain, fold, fence, and publish the old binding as `SEALED`; +2. let SchemaApply rematerialize and publish the target table while leaving the + lifecycle row `SEALED`; +3. run the recovery-covered §3 `stream_rebind` against the exact target, with a + fresh never-reused enrollment UUID and shard UUID namespace; and +4. publish `OPEN` only in the manifest CAS that confirms the new binding and + its initial per-shard epochs. Those epoch values are not compared with the + old shard namespace. + +The old shard namespace and artifacts remain retained until SchemaApply and +rebind sidecars are resolved and every fresh-read/recovery guard releases them. +A preserved physical enrollment never resets its generation or WAL-position +counters. A rebind never reuses an old shard UUID; beta.21 surface guards pin +that OmniGraph supplies a fresh UUID v4 to `mem_wal_writer` and that the new +namespace is disjoint from every prior binding for the logical table lifetime. + +A Lance version upgrade requires persistent `stream quiesce --all`, but empty +generations alone are insufficient: the MemWAL system index, shard manifests, +epoch records, and generation directories may still use the old format. Before +the bump, the implementation must prove one of: (a) upstream guarantees and +cross-version tests cover every retained MemWAL artifact, (b) a public Lance +metadata migration converts them, or (c) OmniGraph tears down the enrolled +MemWAL metadata under recovery and re-enrolls after the bump. Without one of +those gates the upgrade refuses; `resume` never opens unverified old metadata. +This paragraph governs a Lance-only bump that preserves OmniGraph's current +internal format. An OmniGraph format rebuild follows §11 instead and never +opens retained source-graph MemWAL artifacts in the target graph. + +## 9. Fresh-read cuts + +Freshness is a first-class engine/IR enum: + +```text +Committed +Fresh +``` + +At query planning, `Fresh` captures one `FreshReadCut` containing: + +- the ordinary manifest snapshot; +- each selected table's exact lifecycle state and `StreamPhysicalBinding`; +- each selected shard-manifest version and writer epoch; +- included flushed-generation paths and maximum generation; +- the active same-process MemTable row-position watermark, when available; +- the base table's `merged_generations` and index-catchup state read from the + exact table version selected by the manifest snapshot, never from live HEAD. + +Capture uses a retrying handshake: + +1. read each selected lifecycle row and require `OPEN`; capture its exact + physical binding, then read only that binding's shard manifests/epochs, + acquire Lance generation retention guards for the flushed files in the + tentative cut, and under one same-process writer snapshot capture/pin any + active-MemTable watermark; +2. pin the graph manifest snapshot and require it to select the same lifecycle + binding and physical base table/ref captured in step 1; read + `merged_generations` from each exact base-table version it selects. A + `DRAINING`, `SEALED`, or different enrollment restarts the whole capture; +3. re-read the lifecycle rows, complete physical bindings, shard manifest + versions, and per-shard epochs. Any enrollment, ref-incarnation, + configuration, state, shard-set, manifest-version, or epoch change restarts + the whole capture; +4. if a generation from step 1 disappeared, accept that only when the pinned + base's `merged_generations` proves it is included; otherwise release guards, + discard the whole graph snapshot, and retry from step 1; +5. exclude generations that appeared after step 1 and hold the generation and + MemTable read guards captured in step 1 until query completion. + +If Lance exposes no guard that prevents generation GC for the query lifetime, +cross-process `Fresh` does not ship. A missing generation is never interpreted +as “probably folded” against an older pinned base. + +Execution never refreshes that cut mid-query. It excludes every flushed +generation `<= merged_generations[shard]`; otherwise old WAL data could outrank +or duplicate its newer base-table image. + +Fresh reads have no cross-table atomicity. Same-process active MemTables provide +read-your-writes; other processes can promise only the latest flushed state +captured by their shard-manifest reads. The HTTP request and query docs state +those limits wherever the tier is exposed. + +## 10. Observability and resource contracts + +Per shard expose durable WAL position, replay position, active epoch, current +generation, flushed and merged generation, index catchup, pending rows/bytes, +oldest acknowledged age, last fold error, blocked reject, and quiescence state. + +Metrics cover ack latency, durability-wait batching, fenced writers, replayed +entries, fold rows/bytes/generations, fold retries, lag, reject counts, and +sidecar recovery. Defaults for every byte/count/time bound are documented and +configuration changes are observable behavior. + +`stream status` resolves the exact lifecycle rows and MemWAL metadata through a +structured, bounded access path; it may reuse RFC-024's scalar-index machinery +but cannot claim history-flat cost while scanning manifest history. + +## 11. Format activation and rebuild + +Streaming is a graph-format capability, not a feature activated by the first +enrollment. A newly initialized stream-capable graph writes the complete +internal-format stamp, RFC-028 identity authority, and every other co-released +capability before it accepts data. First-use enrollment is permitted only on a +graph whose format already declares streaming; enrollment adds one table's +physical MemWAL index and exact lifecycle row but does not change the graph +format. + +The capability stamp is not assigned merely because beta.21 MemWAL APIs are +present. Initialization can emit a stream-capable first state only after §3's +public exact-enrollment and admission-seal gate passes; otherwise both new-root +activation and first enrollment remain disabled. + +Old binaries refuse a stream-capable graph before reading or writing any table. +A stream-capable binary refuses an older graph before running recovery. On a +compatible stamp it resolves any exact RFC-022 enrollment sidecar first, then +refuses an **uncovered** partial-format state in which the stamp, accepted +schema identity, enrolled MemWAL metadata, and lifecycle authorities disagree. +A covered enrollment crash is recoverable intent, not format corruption. The +engine never repairs an uncovered mismatch by inferring ownership from a table +name, path, or compatible-looking system index. + +The same rule covers schema rematerialization inside an already stream-capable +graph. A preserved `(stable_table_id, incarnation_id)` with a new physical +table is a `SEALED` stream awaiting the exact §3 rebind, not an `OPEN` stream +and not permission to attach old shards. Recovery classifies the old and new +bindings independently and retains the old artifacts until their sidecars and +read guards permit reclamation. + +There is no in-place activation or rollback to an old format. An existing graph +moves to a stream-capable format only through the strict export/init/load +strand: + +1. persistently quiesce every enrolled stream and fold every acknowledged row + into the manifest-visible base tables; +2. verify `SEALED`, empty-generation, merged-generation, sidecar, and uncovered + drift invariants on the source graph; +3. use the old-format binary to export each selected branch's current logical + state; +4. use the new binary to initialize a different graph root and load the export + through RFC-022; +5. validate the rebuilt graph before cutting clients over. + +Ordinary export contains only manifest-visible graph state. WAL-only rows, +MemWAL indexes, shard manifests, epochs, lifecycle rows, reject history, +fresh-read guards, and fold checkpoints are not transferred. Therefore step 1 +is mandatory: an acknowledged but unfolded row would otherwise be lost. The +new graph starts with no physical stream enrollment; first use enrolls each +declared stream through §3 after cutover. The rebuild also loses branches not +separately exported, commit DAG, snapshots, tombstones, recovery history, and +time travel, as specified by RFC-028's common format strand. + +The retained source graph and its MemWAL artifacts remain owned by the old +binary and are never opened by the new target as migration input. A failed +target init/load cannot mutate the source. Independently released later format +capabilities require another rebuild; co-release with RFC-023, RFC-024, RFC-025, +or RFC-028 is allowed only after every participating RFC is independently +accepted and their combined init, refusal, recovery, and rebuild matrix passes. + +Enrollment, fold, and recovery retain RFC-022's single-writer-process support +boundary. MemWAL's shard epoch permits an explicitly routed acknowledgement +owner; it does not turn OmniGraph sidecar recovery into distributed fencing or +authorize general replica failover. + +## 12. Acceptance gates + +- Surface guards pin claim, append, durability waiter, flush, per-shard epoch + fencing, staged merge with `merged_generations`, and index catchup. They also + pin beta.21's blocking fact that initialization commits internally and shard + creation is separate. Acceptance requires the public exact enrollment + receipt/staging plus admission seal/reopen surface from §3; until then this + gate remains red. +- A WAL append failure emits no durable acknowledgement. Every acknowledged row + survives crash, replay, fold, and recovery. +- Failpoints cover enrollment's index/shard multi-effect gap and every fold participant + around sidecar, `commit_staged`, reject persistence, and manifest publish. +- A key conflict on fold participant N after participant 1 committed retains + the exact sidecar, returns `RecoveryRequired`, and starts no replan until the + recovery barrier resolves it; the no-effect case finalizes the empty intent + before its bounded full replan. +- Enrollment restarts without an inline effect when schema, PK, table head, or + stream configuration changes between prepare and gated revalidation. +- Local and S3/RustFS same-path/ref delete-recreate races change the physical-ref + incarnation and make enrollment, ack, fold, recovery, and fresh capture refuse + the replacement. A backend without a proven token refuses activation. +- A rematerializing type rename preserves the RFC-028 logical pair, leaves the + target `SEALED`, and cannot acknowledge until an exact sidecar-covered rebind + publishes a fresh enrollment/shard namespace. Crash tests cover every old- + drain, SchemaApply, rebind-effect, and rebind-publish boundary; recovery never + adopts old shards by name/path or reuses their UUIDs. +- Two folders converge exactly once; a fenced stale writer can never produce a + false durable acknowledgement after WAL GC. +- Without RFC-024, a concurrent graph commit that changes a probed-but-untouched + table after fold validation changes the captured branch token, forces a full + fold revalidation, and cannot be accepted by publisher reparenting. +- Two server replicas do not epoch-ping-pong one shard; owner failover fences + the old process before the new owner acknowledges. +- Quiescence tests race appends with branch and schema operations across two + coordinators and prove no post-drain tail appears; failpoints cover every + lifecycle-row, per-shard epoch-fence, admission-seal, fold, and `SEALED` + boundary. A claimant paused before its final lifecycle check and one paused + after that check are both either included before the drain cut or fenced + before `put`. +- Persistent quiesce never auto-reopens; explicit same-binding resume advances + each same-shard epoch, while a rebind starts an unrelated epoch namespace. + Upgrade tests cover every retained MemWAL artifact through declared + compatibility, migration, or teardown/re-enrollment. +- Genuine cross-version tests prove old-binary/new-format and + new-binary/old-format refusal. Rebuild tests quiesce and fold acknowledged + WAL rows before export, prove those rows survive in the target, and prove an + unfolded acknowledged row makes export/cutover fail closed. +- Partial-format tests reject a capability stamp without matching identity and + stream authorities, and reject enrolled MemWAL metadata without the declared + capability, while an exact sidecar-covered enrollment gap recovers before + consistency validation. No test simulates compatibility by merely rewinding + a stamp. +- Fresh reads race capture with fold/GC and a completed physical rebind. They + retry on a lifecycle/binding change or unexplained disappearing generation, + hold generation/MemTable guards through execution, exclude merged + generations, and document cross-table inconsistency explicitly; a cut can + never pair an old shard namespace with the new base manifest. +- Server/OpenAPI tests preserve the old `/ingest` alias and cover the new route; + CLI parity covers embedded and remote stream commands. +- Ack-path object-store operations are O(1) and flat in graph history and WAL + depth. Fold cost is bounded by generations/rows folded, not graph history. +- S3 correctness runs against RustFS; API/format guards are rerun before every + Lance bump. + +## 13. Phasing + +| Phase | Content | Gate | +|---|---|---| +| A | graph-format capability/refusal gate, strict rebuild path, MemWAL adapter, surface guards, enrollment sidecar | public exact-enrollment plus admission-seal API gate; genuine cross-version/refusal and rebuild suite; multi-effect crash matrix | +| B | new ingest route/CLI, durable ack, strict fold | ack durability; API compatibility; cost budget | +| C | atomic dead letter, audit provenance, status/bounds | reject crash matrix; backpressure tests | +| D | epoch-fenced drain, persistent quiesce/resume, schema/branch/upgrade integration, rematerialization rebind | two-coordinator race, old/new physical-binding crash matrix, and format-transition suite | +| E | fresh cuts and maintained-index reads; cross-process `Fresh` ships only if the substrate generation-retention guard exists (§9), otherwise same-process only | cut consistency; merged-generation exclusion | +| F | multi-shard upsert and stream deletes | one-key-one-shard proof; Lance re-audit | diff --git a/docs/rfcs/rfc-027-lineage-merge-deltas.md b/docs/rfcs/0027-lineage-merge-deltas.md similarity index 92% rename from docs/rfcs/rfc-027-lineage-merge-deltas.md rename to docs/rfcs/0027-lineage-merge-deltas.md index a6dbfab0..f3f6359f 100644 --- a/docs/rfcs/rfc-027-lineage-merge-deltas.md +++ b/docs/rfcs/0027-lineage-merge-deltas.md @@ -5,18 +5,21 @@ description: Research specification for replacing full-width branch-merge classi status: research-blocked tags: [eng, rfc, merge, lineage, change-feed, performance, lance, omnigraph] timestamp: 2026-07-10 -owner: +owner: OmniGraph maintainers --- # RFC-027 — Lineage-based merge deltas -**Status:** Research / blocked on deletion-delta discovery +**Status:** Research blocked — deletion-delta discovery **Date:** 2026-07-10 -**Depends on:** [RFC-022](rfc-022-unified-write-path.md)'s unified branch-merge +**Author track:** Maintainer design series +**Depends on:** [RFC-022](0022-unified-write-path.md)'s unified branch-merge pipeline and capture-once write view -**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.15 (`f24e42c1`) +**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.21 at git rev +`1aec14652dcbace23ac277fa8ced35000bea0c40`; full Lance row-lineage, +transaction, branching, and read/write specifications **Audience:** merge, storage, and performance maintainers -**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md). +**Open architecture review:** [RFC-022–028 review ledger](../dev/rfc-022-027-architecture-review.md). Findings marked **BLOCKER** must be dispositioned before acceptance. --- @@ -126,8 +129,10 @@ not a dependency; the production surface must be public and pinned by Write immutable per-commit deleted-ID rows in the same manifest CAS as the graph commit. This is first-class commit metadata, not a side channel. It adds storage -and format liability and therefore needs a separate format decision before use; -it is not silently folded into v5. +and format liability and therefore needs its own accepted format-capability +decision, or an explicit co-release decision with independently accepted +siblings, before use. It is not implicitly part of RFC-028's identity format or +any other sibling's provisional format number. Until one option passes the cost and correctness gates, delete-bearing histories use the existing classifier. There is no "usually O(delta)" claim that excludes @@ -251,4 +256,6 @@ required to catch an apparently correct path silently regressing to O(rows). | R5 | Make lineage default and consider cursor retirement | all fallbacks dispositioned; no physical gap can break merge | The RFC remains **research / blocked** through R2. Choosing an OmniGraph commit -delta format in §4.3 requires its own format amendment before R3. +delta format in §4.3 requires its own accepted format-capability decision before +R3, including the strict rebuild and cross-version refusal contract or an +explicitly tested co-release with another accepted capability. diff --git a/docs/rfcs/0028-stable-schema-identity.md b/docs/rfcs/0028-stable-schema-identity.md new file mode 100644 index 00000000..b5b62106 --- /dev/null +++ b/docs/rfcs/0028-stable-schema-identity.md @@ -0,0 +1,610 @@ +--- +type: spec +title: "RFC-028 — Stable schema identity and table incarnation" +description: Defines accepted rename-stable type, property, and table identities plus drop/recreate incarnation semantics as the shared prerequisite for RFC-023 through RFC-026. +status: draft +tags: [eng, rfc, schema, identity, incarnation, migration, versioning, lance, omnigraph] +timestamp: 2026-07-14 +owner: OmniGraph maintainers +--- + +# RFC-028: Stable schema identity and table incarnation + +**Status:** Draft / for team review +**Date:** 2026-07-14 +**Author track:** Maintainer design series +**Depends on:** [RFC-022](0022-unified-write-path.md)'s accepted-schema capture, +SchemaApply recovery, and strict publication boundary +**Surveyed:** OmniGraph 0.8.1 (`main`); Lance 9.0.0-beta.21 at git rev +`1aec14652dcbace23ac277fa8ced35000bea0c40`; full Lance table-schema, +schema-evolution, transaction, and versioning specifications +**Audience:** compiler, schema, engine, migration, and release maintainers +**Open architecture review:** [RFC-022–028 review ledger](../dev/rfc-022-027-architecture-review.md). +Findings marked **BLOCKER** must be dispositioned before acceptance. + +--- + +## 0. Decision summary + +OmniGraph will make the persisted accepted SchemaIR the sole authority for +schema identity. A `.pg` file describes canonical schema shape and migration +hints; it does not contain, derive, or let a caller choose authoritative IDs. + +SchemaIR v2 adds one graph identity domain, one monotonic no-reuse allocator, +rename-stable type and property IDs, and one table-incarnation ID for every +node and edge type. Within one graph identity domain: + +- IDs are nonzero `u64` newtypes allocated from `next_identity_id`; +- `stable_table_id` is the node or edge `type_id`; +- a rename preserves type ID, property ID, and table incarnation; +- an ordinary write, branch fork, or physical owner handoff preserves them; +- a drop followed by an add, even under the same name, is a new declaration + and mints a new stable ID and incarnation; +- a property removed and later re-added mints a new property ID; +- no ID is inferred from a name, path, position, Lance version, Lance field ID, + or native branch identifier. + +The accepted IR hash remains the write/recovery authority. Source drift is +checked against a separate identity-free shape projection, so opening a graph +does not regenerate IDs. Catalogs are derived directly from the accepted IR +and retain its IDs instead of round-tripping through a name-only AST. + +This is an internal-format change. It follows the existing strand model: +older graphs are refused and rebuilt with old-binary `export` followed by +new-binary `init` and `load`. There is no in-place identity backfill or +all-branch conversion framework. + +## 1. Problem + +The current SchemaIR calls its IDs stable but computes them from mutable names: + +```text +type_id = fnv1a(kind + ":" + name) +prop_id = fnv1a(owner + ":" + property_name) +``` + +A type or property rename therefore changes identity. Rebuilding `Catalog` +from SchemaIR currently discards those IDs, and open-time source validation +recompiles `.pg` into a fresh name-derived IR before comparing the exact hash. +Preserving an old ID in only SchemaApply would consequently make the next open +reject the graph. + +That gap is already load-bearing for four independent drafts: + +- [RFC-023](0023-key-conflict-fencing.md) must preserve the key contract across + table and field renames; +- [RFC-024](0024-durable-table-heads.md) needs a head key that does not change + when the public table name changes and a token that detects table ABA; +- [RFC-025](0025-checkpoint-retention.md) must bind retained table versions to + the intended logical table lifetime; +- [RFC-026](0026-memwal-streaming-ingest.md) must keep logical stream lifecycle + and reject-history ownership stable across rename while separately rebinding + any rematerialized physical WAL and refusing drop/recreate adoption. + +Letting each consumer invent identity would create four formats that can drift. +Making RFC-024 own the shared capability is also wrong: durable heads remain an +independently gated optimization and cannot be a hidden prerequisite for every +correctness consumer. This RFC extracts the one common contract. + +## 2. Scope and non-goals + +In scope: + +- accepted identity for interfaces, node types, edge types, and properties; +- node/edge table incarnation; +- identity-aware schema planning, source validation, catalog derivation, and + SchemaApply recovery; +- graph-root and branch semantics; +- the boundary between logical identity and Lance physical field/ref identity; +- internal-format activation and upgrade behavior; +- the contract consumed by RFC-023 through RFC-026. + +Out of scope: + +- durable table-head rows or a manifest-head lookup design; +- primary-key installation or keyed-write routing; +- checkpoint rows, Lance tag reconciliation, or cleanup policy; +- MemWAL enrollment, acknowledgement, folding, or fresh reads; +- user-authored identity annotations in `.pg`; +- implicit rename detection, resurrection, or cross-owner property moves; +- adding rename syntax for declaration kinds the current planner cannot rename, + including interfaces; their IDs are persisted now, and any future supported + rename must preserve them; +- identity continuity across export/import into a new graph root; +- a public identity-inspection or identity-import API. + +## 3. Identity model + +### 3.1 Graph identity domain + +Every graph initialized at the identity-capable format mints one opaque +`schema_identity_domain` and persists it authoritatively in SchemaIR. Schema +state repeats the value only as a cheap integrity/probe field; open requires it +to equal the domain in the hash-validated IR and treats any mismatch as format +corruption. Consumer tokens always take the domain from that validated IR. It +is a ULID used only as a namespace token; its timestamp and lexical ordering +have no semantic meaning. + +All numeric identities below are scoped to that domain. Internal rows in one +graph may store only the numeric ID because the domain is implicit. Any future +token that crosses graph roots MUST carry the domain as well as the numeric ID. +Two independently initialized graphs may allocate the same numeric IDs without +claiming continuity. + +A native branch and every lazy table fork share the graph's identity domain. +A byte-for-byte backup restored as the sole continuation of that graph +preserves the domain. Operating the same copied root as a second, independently +writable graph is unsupported: both copies could allocate the same next numeric +ID to different declarations. There is no public clone/rekey operation in this +RFC. Creating an independent graph uses export/init/load and mints a new domain. + +### 3.2 ID types and allocation + +SchemaIR v2 uses distinct newtypes over nonzero `u64` values: + +```text +StableTypeId(u64) +StablePropertyId(u64) +TableIncarnationId(u64) +``` + +Interfaces, nodes, and edges receive `StableTypeId`. Every interface property +and every node/edge effective property receives `StablePropertyId`. Nodes and +edges additionally receive `TableIncarnationId`; interfaces have no table and +therefore no incarnation. +For a node or edge: + +```text +stable_table_id == stable_type_id +``` + +A property ID has exactly one stable owner type. Interface expansion does not +reuse an interface property's ID as a concrete table-column ID: + +- each interface property has its interface-owned ID; +- each node's effective property has one node-owned ID, whether it was written + directly or injected by `implements`; +- that node property records the sorted set of interface property IDs whose + contracts it satisfies; two compatible interfaces may therefore map to one + node property without choosing one interface as the owner; +- an edge property is owned only by its edge type. + +Interface queries resolve through those ID links to the node-owned effective +property and then to the pinned table's Lance field ID. Names are retained for +diagnostics, not used to reconstruct the relationship. + +The engine-owned `id`, `src`, and `dst` columns do not consume user-property +IDs. They are addressed by `(stable_table_id, incarnation_id, +SystemFieldRole::{Id, Src, Dst})`, where the role is a sealed internal enum and +only roles valid for that table kind may appear. These fields cannot be renamed +or supplied by a schema declaration. + +One `next_identity_id` high-water mark lives in the accepted SchemaIR. A new +graph initializes it to `1`; zero is invalid. Allocation consumes the current +value and increments the mark; overflow is a typed hard error. Type, property, +and incarnation allocations share the sequence, so one accepted graph never +reuses a value even across ID kinds. + +Allocation is canonical and independent of source order: + +1. new types, ordered by `(kind, canonical name)`, with kind order + `interface < node < edge`; +2. new properties, ordered by `(owner stable type ID, canonical name)`; +3. new table incarnations, ordered by stable type ID. + +“Canonical name” here is the parser-accepted identifier's exact ASCII byte +sequence. Type and property names remain case-sensitive; there is no Unicode +normalization, locale folding, or case folding in allocation order. + +Read-only planning may calculate the resulting values from the captured +high-water mark, but they become fixed attempt data only when SchemaApply arms +the complete desired IR; they become accepted authority only when that exact IR +is promoted. An attempt that fails before arming publishes nothing. An armed +attempt fixes the values in its staged IR and recovery intent; it may not remint +them on retry. A rolled-back attempt never made those IDs part of accepted graph +state, so a later fresh preparation may reuse its unaccepted numbers without +violating the no-reuse contract. + +### 3.3 Incarnation is not identity twice + +The type ID answers “which accepted schema declaration is this?” The table +incarnation answers “which logical table lifetime implements it?” +Consumers compare both even though initial creation and today's drop/re-add +semantics mint both together. Keeping the dimensions explicit prevents a +future physical replacement that preserves logical type identity from requiring +another format change. + +The incarnation is not any of: + +- a Lance dataset version; +- a Lance field ID; +- a native branch `BranchIdentifier`, manifest e-tag, or timestamp; +- a table path or public `table_key`; +- a graph commit ID. + +Those physical tokens remain necessary for exact effect ownership and ref ABA. +They are compared alongside, not substituted for, logical incarnation. + +In particular, today's supported type rename may materialize a new physical +dataset/path while preserving the logical type. That changes the exact physical +effect/ref identity but not the RFC-028 type ID or table incarnation. Only an +accepted schema event that explicitly starts a new logical table lifetime may +change the incarnation. + +## 4. One authority, two projections + +### 4.1 Identity-free schema shape + +Parsing and type-checking `.pg` produces a canonical `SchemaShape`. It contains +the public names, types, constraints, annotations, endpoints, and interface +relationships required to describe behavior, but no authoritative type, +property, or incarnation IDs. Source comments and declaration order are already +non-semantic. The migration-only `@rename_from` hint is excluded from the +accepted shape hash after it has been resolved. + +`SchemaShape` preserves whether a node property was declared directly and which +interface property contracts contribute to its effective shape. The current +name-only AST expansion may be derived from that information, but it cannot be +the sole identity input because destructive injection loses multi-interface +provenance. + +The shape compiler is deterministic and pure. It does not consult a clock, +random generator, table path, or persisted allocator. + +### 4.2 Accepted SchemaIR + +Identity resolution combines one captured accepted SchemaIR with a desired +shape: + +- a declaration with the same accepted name inherits its IDs; exact-name + matching wins before rename-hint evaluation, and any carried hint on that + declaration is already consumed and inert; +- an unambiguous, kind-compatible `@rename_from` inherits the named accepted + declaration's IDs only when the desired name has no exact accepted match; +- an unchanged property or an explicit property rename inherits its property + ID only within the same stable owner type; +- a new declaration allocates an ID, and a new node/edge also allocates an + incarnation; +- removed declarations disappear from the live IR; their IDs remain below the + allocator high-water mark and can never be allocated again; +- a same-name add in a later accepted schema is still new because no live + declaration exists to inherit from. + +Rename matching is explicit. Name similarity, identical field sets, physical +path reuse, and a tombstoned manifest row are never evidence of identity. Once +an exact-name match succeeds, a persisted stale `@rename_from` is ignored; it +cannot redirect or invalidate that already-accepted declaration. Otherwise, +during evolution of a non-empty accepted schema, `@rename_from` cannot target a +missing or dropped declaration, cross type kinds, move a property between +owners, or map two desired declarations to one accepted ID. Those cases fail +before effects. +During initialization against an empty accepted base, there is no predecessor +to resolve: any carried `@rename_from` hint is inert, emits a diagnostic, and is +stripped from accepted semantics. This lets an old binary's schema output seed a +strict rebuild even when its source retained an already-consumed hint; the hint +never imports identity from the old graph. + +Identity resolution consumes `@rename_from`; the accepted IR does not retain it +in the declaration's semantic annotation set. The persisted source may keep the +hint for human context, but both projections ignore it after resolution, so +removing a stale hint later is not a schema change. + +The identity-bearing IR stores internal references by stable ID: edge endpoint +types, implemented interfaces, property-backed constraints, index declarations, +embedding sources, interface-property satisfaction links, and other schema +relationships resolve to IDs. Current names remain beside them for diagnostics +and source rendering, and validation proves that each name/ID pair refers to +the same declaration. + +### 4.3 Hashes and schema token + +Schema state records two hashes: + +```text +schema_shape_hash = hash(canonical identity-free semantic projection) +schema_ir_hash = hash(complete accepted SchemaIR, including domain, + IDs, incarnations, and next_identity_id) +``` + +Open validates `_schema.ir.json` against `schema_ir_hash`, compiles `_schema.pg` +only to `SchemaShape`, and compares its projection to `schema_shape_hash`. It +never rebuilds accepted IDs from source. Mutation, load, schema apply, branch +merge, maintenance, and recovery capture `schema_ir_hash` plus the catalog +derived from that exact IR as one operation-local schema token. + +Changing only a name through an accepted rename changes both hashes while +preserving the relevant IDs. Removing a stale `@rename_from` hint after the +accepted rename changes neither semantic hash. + +### 4.4 Catalog derivation + +`Catalog` is built directly from the accepted SchemaIR and exposes stable IDs +and table incarnation beside current names. It MUST NOT reconstruct a name-only +AST and call a builder that mints or discards identity. + +The warm catalog remains derived state. The hash-validated persisted IR is the +identity authority; schema state is its validated format/hash/domain envelope, +not a second source of IDs. The handle refreshes the catalog under the existing +schema gate and cheap schema-identity probe. This does not introduce a second +maintained identity registry. + +## 5. Lifecycle truth table + +| Event | Stable type/table ID | Property ID | Table incarnation | +|---|---|---|---| +| Initial interface | mint | mint interface properties | none | +| Initial node/edge | mint | mint effective properties | mint | +| Add interface/node/edge | mint | mint interface/effective properties | none / mint | +| Add property | preserve owner | mint | preserve owner incarnation | +| Reorder or metadata-only change | preserve | preserve | preserve | +| Supported explicit type rename | preserve | preserve | preserve | +| Explicit property rename in same owner | preserve | preserve | preserve | +| Node/edge owner-branch handoff or lazy fork | preserve | preserve | preserve | +| Ordinary data/index/maintenance write | preserve | preserve | preserve | +| Drop property, then later same-name add | preserve owner | mint new | preserve owner incarnation | +| Drop type, then later same-name add | mint new | mint new | mint new | +| Node ↔ edge/interface kind change | mint new; no cross-kind rename | mint new | according to new kind | +| Remove under owner A, add equivalent property under owner B | preserve both live owner IDs | mint under owner B; no move continuity | preserve each live owner incarnation | +| Export/import into new graph root | new identity domain; allocate anew | allocate anew | allocate anew | + +Dropping a type may leave identity-bearing tombstone/history rows in formats +that support them, but those rows are not an implicit resurrection registry. +The old stable ID and incarnation remain historical facts and never authorize a +new live declaration. A future explicit resurrection feature would require its +own schema surface and RFC. + +## 6. SchemaApply and recovery + +Identity resolution is part of SchemaApply preparation, before any Lance HEAD +or graph-visible state can move: + +1. capture the accepted SchemaIR, both schema hashes, and graph authority under + the schema gate; +2. compile desired source to `SchemaShape`; +3. resolve unchanged declarations and explicit renames against the captured IR; +4. allocate every new ID/incarnation canonically from the captured high-water + mark; +5. build the desired catalog directly from that complete desired IR; +6. validate identity uniqueness and every name/ID relationship; +7. stage desired source, desired IR, schema state, and the exact identity-bearing + migration plan; +8. arm the RFC-022 SchemaApply sidecar with the accepted and desired schema + tokens before the first table effect. + +After arming, the desired domain, IDs, incarnations, hashes, and allocator mark +are immutable attempt data. A pre-effect authority conflict discards the whole +attempt and reprepares. A post-effect failure returns `RecoveryRequired`; +recovery uses the staged IR and exact sidecar rather than recompiling source or +reminting identity. + +Roll-forward publishes the fixed manifest outcome before promoting the staged +schema contract, preserving RFC-022's existing ordering. Rollback restores or +reclaims only effects owned by the attempt and leaves the previously accepted +IR authoritative. A first-touch table path is never adopted because its name +matches a desired type; ownership still requires the exact sidecar and physical +transaction/ref identity. + +Schema evolution remains graph-global and currently requires no public +non-main branches. Branch creation after an accepted schema captures the same +identity domain, IDs, and incarnations. This RFC does not weaken that support +boundary or introduce branch-local schemas. + +## 7. Lance physical identity mapping + +Lance field IDs are immutable within one Lance table schema and survive column +rename and reorder. OmniGraph uses that property for physical schema evolution, +but a Lance field ID is scoped to one exact physical dataset/schema lineage, +not to RFC-028's logical table incarnation, and is not a graph-level type or +property ID. + +For each pinned table image, OmniGraph derives and validates the mapping from +an accepted user-property ID or sealed system-field role to the Lance field ID +in that physical schema. The mapping comes from the accepted catalog plus the +pinned Lance schema; it is not an independently maintained registry. An +in-place property rename captures the old field ID and proves the resulting +schema kept it. A newly materialized table, including the target of today's +supported type rename, may assign different Lance field IDs while its logical +contract is governed by the accepted type/property IDs and incarnation. +For an interface projection, the catalog first resolves the interface-owned +property ID through the node property's satisfaction link; only the resulting +node-owned property ID maps to that table's Lance field. + +RFC-023 therefore addresses primary-key metadata by the pinned table's Lance +field ID while binding the surrounding table contract to +`(stable_table_id, incarnation_id)`. RFC-024 through RFC-026 similarly carry +logical identity and retain their own exact physical tokens. + +## 8. Format activation and upgrade + +SchemaIR v2 and the identity domain/allocator are an internal storage-format +capability. The implementation takes the next internal manifest schema number +(provisionally v5 while current is v4), keeps `MIN_SUPPORTED == CURRENT`, and +adds the identity version to the release/stamp history. The exact numeral may +move if another accepted format change lands first; the capability semantics do +not. + +There is no v1→v2 IR backfill and no new-binary in-place conversion of an old +graph. Activation follows the documented strand: + +1. quiesce the old graph long enough to take a consistent export; +2. use a binary that supports the old format to export the selected branch and + show its schema; +3. use the new binary to initialize a different graph root; init mints the new + identity domain and writes a complete identity-capable schema contract before + accepting data; +4. load the export through the ordinary RFC-022 recovered write path; +5. verify the rebuilt graph, then cut clients over. + +The rebuilt root does not preserve old identities, branches, commit DAG, +snapshots, tombstones, recovery intents, checkpoints, stream lifecycle, or +time-travel history. A branch whose current state must survive is exported and +rebuilt separately today. Numeric IDs may coincidentally match because +allocation is canonical; the new identity domain makes them distinct. + +Old binaries refuse the new manifest stamp. New binaries refuse the old stamp +before parsing an incompatible SchemaIR or running recovery. A failed target +init/import never mutates the source graph; discard or repair the target and +retry. + +An independently released later capability requires another rebuild under the +same policy. RFC-023, RFC-024, RFC-025, or RFC-026 may co-release with this +format to reduce operator cutovers only after each RFC is independently +accepted and the combined initialization/recovery matrix passes. Co-release is +release coordination, not permission for a draft sibling to define identity. + +## 9. Consumer contract + +- **RFC-023:** every keyed table is identified by stable table ID plus + incarnation; the pinned Lance field ID addresses PK metadata. Rename preserves + the pair; drop/re-add cannot inherit the old fencing authority. +- **RFC-024:** `table_head` keys use stable table ID and head payloads carry + incarnation. It owns head storage and lookup, not ID minting. +- **RFC-025:** checkpoint table entries carry stable table ID and incarnation in + addition to exact physical location/ref/version pins. It owns retention + authority and Lance tags, not identity. +- **RFC-026:** stream lifecycle rows and deterministic reject keys carry stable + table ID plus incarnation. A same-dataset rename preserves the physical + enrollment; a rematerializing rename preserves logical history ownership but + must fence and rebind through a fresh shard namespace. Drop/re-add cannot + adopt old WAL/reject state. + +No consumer may fall back to mutable `table_key`, path, or name when an identity +is absent. Missing, duplicate, zero, out-of-domain, or mismatched identity is a +typed format/authority error. + +## 10. Invariants and deny-list check + +- **Invariant 8 moves from gap to implemented contract only when this RFC's + implementation lands.** The draft itself does not close the current gap. +- **Invariant 5:** identity minting joins the existing exact SchemaApply sidecar; + there is no second recovery protocol. +- **Invariant 15:** accepted SchemaIR is the sole authority and `Catalog` is a + cheaply refreshed derived view. There is no shadow identity registry and no + cold history replay. +- **Logical over physical:** Lance field/ref/version identity remains physical + evidence; it cannot redefine logical schema identity. +- **Strict strand:** no migration dispatcher or old-format compatibility reader + is introduced. + +The design adds durable allocator state because identity cannot be derived from +mutable names. That state is part of the accepted schema authority, not a job +queue or parallel copy. No hard invariant is weakened. + +## 11. Tests and acceptance gates + +### 11.1 Compiler and IR + +- source order, comments, and repeated planning produce the same shape and + allocation plan; +- new graphs start at allocator value `1`; kind ordering and exact ASCII-byte + name ordering are pinned by golden SchemaIR fixtures; +- type and same-owner property renames preserve IDs; +- interface expansion gives each node-effective property one owner ID and + preserves sorted interface-property satisfaction links, including two + compatible interfaces contributing the same property name; +- removing one of several satisfied interface contracts preserves the + still-live node property ID; removing the effective property and later + reintroducing it mints a new node-owned ID; +- edge endpoints, interfaces, constraints, indexes, and embedding references + resolve through stable IDs after rename; +- `id`/`src`/`dst` use only the valid sealed system-field role and consume no + allocator values; +- drop/re-add mints a new type/property ID and new table incarnation; +- implicit rename, cross-kind rename, cross-owner property move, duplicate ID, + zero ID, allocator regression, and overflow fail loudly; +- `build_catalog_from_ir` preserves every ID/incarnation and never mints one; +- SchemaIR v1 is refused by the identity-capable binary. + +### 11.2 Engine and recovery + +- init writes one domain, a canonical allocator mark, matching shape/IR hashes, + and a catalog carrying the same identities; +- open rejects a schema-state domain that differs from the hash-validated IR, + and every consumer token takes its domain from the IR; +- reopen validates source shape without regenerating IDs; +- type and property rename tests inspect the promoted IR and prove IDs and + incarnation are unchanged while names and hashes change; +- SchemaApply failpoints before arming, after each table effect, after manifest + publish, and during schema promotion prove fixed IDs roll forward or old IDs + remain authoritative on rollback; +- first-touch and lazy-fork tests prove table path/ref identity cannot substitute + for logical identity; +- long-lived handles refresh schema token and catalog together; +- branch create/fork preserves the graph identity domain and table pair. + +### 11.3 Format and rebuild + +- genuine old-format graphs are refused before recovery/schema parsing by the + new binary and new-format graphs are refused by the old binary; +- old-binary export followed by new-binary init/load round-trips current rows, + vectors, and blobs into a new identity domain; +- rebuild initialization accepts and strips a stale, previously consumed + `@rename_from` hint from old-binary schema output, while a missing rename + target during non-empty schema evolution still fails; +- a subsequent non-empty SchemaApply with the same accepted name and retained + stale hint treats the hint as inert, preserves IDs, and removing it later is + a semantic no-op; +- branch/history loss and per-branch separate rebuild behavior match the + upgrade guide; +- co-released capability tests, if any, prove the first valid target manifest + already contains every accepted capability—there is no partially activated + serving window. + +## 12. Drawbacks and alternatives + +### Keep name-derived hashes + +Rejected. A wider hash reduces collisions but remains identity derived from a +mutable name; rename still becomes drop/recreate. + +### Use Lance field IDs as OmniGraph IDs + +Rejected. Lance field IDs solve rename/reorder inside one physical table. They +do not identify graph types, interfaces, tables across physical incarnations, +or the same logical property materialized in multiple tables. + +### Mint a random UUID/ULID for every declaration + +Rejected for the first delivery. It is collision-resistant but makes repeated +planning and diagnostics nondeterministic and requires retry/collision policy +for no benefit inside one serialized graph schema. One graph-domain ULID plus a +monotonic allocator is smaller and exact. + +### Preserve stable ID across drop/re-add + +Rejected for now. It requires a durable tombstone/resurrection registry and a +public way to distinguish intentional resurrection from an unrelated same-name +declaration. Minting both values anew is unambiguous; the incarnation dimension +keeps a future explicit rematerialization feature possible without changing +consumer tokens. + +### In-place identity migration + +Rejected. Current IDs are not rename-stable authority, and retaining a reader, +converter, all-branch claim, crash protocol, and old-format recovery path would +reverse the project's strict-single-version liability decision. Export/import +already gives an atomic source-preserving cutover. + +## 13. Reversibility and open gates + +The document is reversible while draft. Once written, the identity domain, +numeric encoding, allocation order, shape/IR hash split, and drop/re-add rule +become on-disk and recovery contracts. Changing them requires another internal +format and rebuild. + +Before acceptance: + +1. review the SchemaIR v2 wire shape and canonical allocation order against the + implementation plan; +2. confirm every name-bearing internal reference that must become ID-bearing; +3. confirm format refusal runs before schema/recovery parsing on every open + mode; +4. enumerate the exact RFC-022 SchemaApply sidecar fields that fix desired + identity and allocator state; +5. name the implementation tests that extend existing compiler, + `lifecycle.rs`, `schema_apply.rs`, recovery, and cross-version owners. + +Extracting this contract closes BLOCKER-07's specification-ownership +contradiction; accepting RFC-028 makes the shared decision authoritative. The +implementation and format tests must land before RFC-023 through RFC-026 may +claim the capability is active. diff --git a/docs/rfcs/README.md b/docs/rfcs/README.md index 99cdd769..3fc2b7ee 100644 --- a/docs/rfcs/README.md +++ b/docs/rfcs/README.md @@ -2,19 +2,25 @@ Substantial changes to OmniGraph — new user-facing surface, format or protocol changes, anything irreversible or cross-cutting — go through a lightweight RFC -so the design is agreed *as reviewable code* before implementation starts. This -is the public RFC track, open to **anyone, including external contributors**. +so the design is agreed *as reviewable code* before implementation starts. +`docs/rfcs/` is the durable home for both externally authored proposals and +maintainer design series. Every formal RFC uses the same four-digit filename +namespace; explicit metadata selects the applicable lifecycle. This complements the always-on review bar in [../dev/invariants.md](../dev/invariants.md): the invariants say *what every change must respect*; an RFC says *why this particular change is worth making and how*. -> **Two tracks, don't conflate them.** This `docs/rfcs/` directory is the -> **public contribution** track (anyone authors; maintainers accept). The -> maintainer-internal RFCs under `docs/dev/rfc-00N-*.md` are a separate, -> team-owned track for in-flight internal work. If you're an outside -> contributor, you're in the right place here. +> **Two tracks, don't conflate their lifecycle.** Every RFC must declare an +> `Author track` of either `Public contribution` or +> `Maintainer design series`, plus an explicit `Status`. Those fields—not the +> filename—select the lifecycle. Anyone may author a public contribution RFC, +> and merge means maintainer acceptance. Maintainer design series may be merged +> and revised while still draft so design, review findings, and implementation +> evidence remain visible together. Existing `docs/dev/rfc-00N-*.md` files are +> legacy internal design and implementation records that retain their existing +> lifecycle; new formal RFCs live here. ## When you need one @@ -28,13 +34,14 @@ how*. If you're unsure, start a [Discussion](../../../discussions); a maintainer will tell you whether it needs an RFC. -## Lifecycle +## Public contribution lifecycle ``` Discussion (incubate, get rough consensus) │ graduate ▼ -RFC pull request → adds docs/rfcs/NNNN-title.md (Status: Proposed) +RFC pull request → adds docs/rfcs/NNNN-title.md + (Author track: Public contribution; Status: Proposed) │ maintainer review ──▶ changes requested / declined (PR closed, with rationale) │ @@ -53,14 +60,40 @@ Implementation PR(s) reference the accepted RFC ## Numbering & naming -- File: `docs/rfcs/NNNN-kebab-title.md`, where `NNNN` is the next free - zero-padded integer (`0001`, `0002`, …). `0000-template.md` is reserved. +- Every formal RFC file is `docs/rfcs/NNNN-kebab-title.md`, where `NNNN` is + the next free zero-padded integer. `0000-template.md` is reserved. - Pick the number when you open the PR; if it collides with another in-flight RFC, the second to merge bumps theirs. +- Numbers are never reused. A maintainer-series or superseded record still + reserves its number. +- Every RFC declares exactly one `Author track` and an explicit `Status`. If a + file has both YAML frontmatter and rendered metadata, the values must agree. -## Status values +### Public status values `Proposed` (open PR) · `Accepted` (merged) · `Declined` (closed) · `Superseded by NNNN` · `Implemented` (set once the work lands, optional). -Copy [0000-template.md](0000-template.md) to start. +Copy [0000-template.md](0000-template.md) to start a public contribution RFC. + +## Maintainer design-series lifecycle + +Maintainers use this lane for a related architecture series that benefits from +durable in-repository review before every decision or implementation dependency +is ready. Merge publishes the current reviewed design state; it does **not** by +itself mean acceptance. Each file names `Maintainer design series` as its author +track, identifies an owner, links its review ledger when one exists, and records +support boundaries alongside implementation evidence. This metadata is required +for new or materially revised series; older records are not silently reclassified. + +When YAML frontmatter is present, its machine-readable status is `draft`, +`research-blocked`, `accepted`, `implemented`, or `superseded`; headings render +these as `Draft`, `Research blocked`, `Accepted`, `Implemented`, and +`Superseded by RFC-<number>`. +Moving a file to `accepted` or `implemented` requires dispositioning the blockers +owned by that RFC. Open findings in a sibling RFC do not prevent an independently +reviewable member of the series from advancing. + +The explicit author track and status are a process boundary, not an authority +distinction: both tracks are reviewed by maintainers, and neither may bypass +the invariants, compatibility gates, or evidence required by the change. diff --git a/docs/rfcs/rfc-023-key-conflict-fencing.md b/docs/rfcs/rfc-023-key-conflict-fencing.md deleted file mode 100644 index c0c3e2ae..00000000 --- a/docs/rfcs/rfc-023-key-conflict-fencing.md +++ /dev/null @@ -1,480 +0,0 @@ ---- -type: spec -title: "RFC-023 — Substrate-native key-conflict fencing" -description: Make concurrent keyed writes fail or retry through Lance's transaction conflict filters, forbid keyed Append, and activate the contract only behind an all-branch fleet/format barrier. -status: draft -tags: [eng, rfc, write-path, concurrency, lance, primary-key] -timestamp: 2026-07-10 -owner: ---- - -# RFC-023: Substrate-native key-conflict fencing - -**Status:** Draft / for team review -**Date:** 2026-07-10 -**Surveyed:** omnigraph 0.8.1 (`main`); Lance 9.0.0-beta.15 at git rev -`f24e42c1`; full Lance transaction, table-schema, read/write, branching, and -MemWAL specifications; pinned Rust conflict-resolver and merge-insert sources -**Relationship to RFC-022:** this RFC is the fencing decision split from the -earlier monolithic RFC-022 draft. [RFC-022](rfc-022-unified-write-path.md) -defines the shared write/recovery protocol; this RFC owns the substrate, -compatibility stamp, and rollout requirements for key conflicts. It may share a -release with [RFC-024](rfc-024-durable-table-heads.md), but neither RFC depends -on the other's storage decision. -**Audience:** engine, storage, migration, and release maintainers -**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md). -Findings marked **BLOCKER** must be dispositioned before acceptance. - ---- - -## 0. Decision summary - -OmniGraph will use Lance's unenforced-primary-key merge-insert filter as the -key-conflict primitive. It will not emulate the filter in the engine and will -not add a lock table or custom transaction manager. - -The guarantee is deliberately narrow and strong: - -> For a keyed table, two concurrent operations that may insert the same `id` -> MUST NOT both commit silently. They either serialize, retry from a new graph -> base, or fail loudly. - -That guarantee cannot be rolled out by annotating a few new tables in an -otherwise v4 graph. Lance's current mixed filtered/unfiltered conflict handling -is directional, and a bare `Append` can commit after a filtered update. The -contract therefore activates only after all supported writers and every table -state reachable from every graph branch have crossed a fencing-compatible -format barrier. - -Normative decisions: - -1. Node and edge table PK = `id`. -2. Bare Lance `Append` is forbidden for keyed tables. -3. Every keyed insert/upsert path produces the same Lance key filter. -4. Mixed-version serving is forbidden during activation; the fencing-compatible - format stamp is written last and older binaries refuse it. -5. PK metadata is permanent and preserved by every later schema/data rewrite. -6. Existing-table migration covers every graph branch, including lazy-inherited - table states, and is recoverable by rolling forward. -7. Fencing does not replace read-set OCC for `@unique`, RI, or cardinality. - -## 1. Problem - -OmniGraph validates duplicate `@key` values inside one delta, but today two -processes can both read a base where `id = K` is absent, stage disjoint Lance -fragments containing `K`, and let Lance rebase both commits. The graph manifest -CAS orders graph publication; it does not tell Lance that the two data-table -transactions inserted the same logical key. - -The result is worse than an ordinary write conflict: both callers can receive -success while a keyed table contains duplicate IDs. Every subsequent upsert, -edge lookup, uniqueness check, and merge then operates on a broken identity -relation. - -The desired primitive already exists in Lance. The missing work is to use it on -every keyed path and to define a rollout that never admits an unfenced writer. - -## 2. Substrate facts and the directional asymmetry - -This section is load-bearing. Tests pin every statement before implementation. - -### 2.1 Filter attachment - -On the pinned Lance revision, a v2 merge-insert whose ON field IDs exactly equal -the schema's unenforced PK field IDs attaches an `inserted_rows_filter` to its -`Operation::Update`. The filter contains keys for rows classified as inserts; -updates of existing rows continue to use Lance's affected-row / fragment -conflict machinery. - -The legacy indexed merge path does not attach this filter. Therefore a BTREE on -`id` can route an otherwise correct merge onto an unfenced path unless the -caller disables that path or Lance wires the filter into it. - -### 2.2 Conflict compatibility is directional - -Lance transaction compatibility is evaluated from the transaction currently -attempting to commit against transactions that committed after its read -version. It is not implicitly symmetric. - -At `f24e42c1`, `check_update_txn` behaves as follows: - -- current `Some(filter)` vs committed `Some(filter)` — compare field IDs and - filter intersection; overlap or incompatible filter configuration retries; -- current `Some(filter)` vs committed `None` — retry conservatively; -- current `None` vs committed `Some(filter)` — no corresponding conservative - arm; it may rebase; -- current bare `Append` vs committed filtered `Update` — `Append` treats the - `Update` as compatible. - -Consequently, "filtered vs unfiltered conflicts" is not a sufficient rollout -argument. Commit order matters. A filtered writer can win first and a stale -unfiltered writer or bare keyed append can still land second. - -### 2.3 What a PK annotation does not do - -The key is explicitly *unenforced*. Merely setting the metadata: - -- does not validate historical uniqueness; -- does not make bare appends unique; -- does not protect a merge whose ON set differs from the PK; -- does not repair an existing duplicate; -- does not replace OmniGraph's semantic validators. - -## 3. Scope and non-goals - -In scope: - -- node and edge data tables; -- mutation, load, branch merge, recovery replay, and WAL fold paths; -- new-table creation and all-branch existing-table activation; -- schema/overwrite preservation of PK metadata; -- typed retry behavior and coverage gates. - -Out of scope: - -- using a composite edge PK (`src`, `dst`); -- enforcing arbitrary `@unique` groups through the Lance PK; -- keyless streaming-table deduplication; -- a custom OmniGraph WAL, lock table, or transaction manager; -- declaring general multi-process writes supported before foreign-process - recovery-sidecar ownership is solved. - -## 4. Table classes and PK contract - -### 4.1 Keyed graph tables - -Every normal node and edge table has a non-null `id` field. Its Lance schema -MUST mark exactly that field as the unenforced PK. For edges, `src` and `dst` -remain ordinary fields governed by referential-integrity and cardinality -validation. An edge endpoint move is an update of the row identified by `id`. - -The PK field is addressed by stable field ID, not column position or mutable -name. Until rename-stable OmniGraph type/property identity is closed, the -fencing migration cannot claim rename safety. - -### 4.2 Keyless append-only tables - -A table explicitly declared append-only may omit a PK. Such a table may use -Lance `Append`, including MemWAL append-only operation. It receives no -same-logical-key guarantee because it has no logical key. - -Current node and edge types are not in this class: both have graph identity on -`id`. The class is reserved for an explicitly designed internal or future -non-graph table surface. - -The distinction is catalog-derived and first-class. Callers do not choose it -with an ad-hoc flag. - -## 5. Normative write routing - -| Logical operation | Keyed table | Keyless append-only table | -|---|---|---| -| Strict insert / load append | merge-insert ON exactly `id`, `WhenMatched::Fail`, filtered path | `Append` allowed | -| Upsert / load merge | merge-insert ON exactly `id`, `WhenMatched::UpdateAll`, filtered path | workload-specific | -| Fast-forward branch merge of new rows | filtered merge-insert with `WhenMatched::Fail`, even when every row was classified new | `Append` allowed | -| WAL upsert fold | filtered merge-insert with `merged_generations` | append transaction allowed | -| Update existing row | merge-insert/update with affected-row conflict metadata | workload-specific | -| Delete | staged Lance delete; PK filter is not the delete-conflict primitive | staged Lance delete | -| Overwrite | staged overwrite whose output schema preserves the exact PK | staged overwrite | - -### 5.1 No keyed `Append` - -The prohibition includes internal optimization paths. A caller may not infer -"all rows are new" and switch a keyed table to `stage_append`: that inference -was made against a snapshot and is exactly what a concurrent same-key writer can -invalidate. - -Routing through merge-insert does not collapse strict and upsert semantics. -Strict `load --mode append` and other insert-only surfaces use -`WhenMatched::Fail`; a row already present at the pinned base or discovered at -execution remains an error. Only declared upsert surfaces use `UpdateAll`. - -The storage trait and `forbidden_apis` guard MUST make a keyed append difficult -to express accidentally. The fast-forward merge optimization is retained only -for keyless append-only tables unless Lance ships a key-filtered append -transaction. - -This prohibition knowingly retires a measured fix. The fast-forward append -path exists because a whole-delta merge-insert join exhausted the query memory -pool on embedding-bearing tables (#277); routing adopted rows back through a -filtered merge-insert re-exposes that workload to join memory behavior. The -regression class is therefore a named ship gate: the fenced bulk adopt-merge -must pass the §11.4 memory/cost gate on embedding-bearing tables — via bounded -batched fenced merges inside one staged transaction, a pool-bounded execution -mode, or an upstream key-filtered append — before the keyed fast-forward path -is removed. Correctness wins the ordering, but the memory bound is not -optional. - -### 5.2 Routing choice - -There are two acceptable implementations: - -1. use the v2 merge path (`use_index(false)`) and pass its scale gate; or -2. consume a pinned Lance revision whose indexed path emits the identical - filter and passes the same surface guards. - -If the v2 hash-join cost scales unacceptably at the Phase-B workload, fencing -waits for option 2. Correctness is not traded for the old indexed-path speed. - -### 5.3 Symmetric mixed-transaction behavior - -Before activation, the pinned Lance revision MUST conservatively reject both -orders of: - -- filtered Update vs unfiltered Update; -- filtered Update vs bare Append. - -The preferred fix is upstream conflict-resolver symmetry. A workspace-only -fork is not an accepted permanent design. The fleet barrier remains necessary -even after that fix because two old, unfiltered writers still have no filters -to compare. - -## 6. Retry and validation semantics - -A Lance retryable key conflict restarts the entire logical attempt: - -1. gather a new graph snapshot and schema identity; -2. rerun all delta and committed-state validation; -3. restage from that base; -4. commit and publish through the normal recovery-covered pipeline. - -It is incorrect to retry only `commit_staged`: an insert may have become an -update, defaults or checks may now differ, and cross-table validation may have -changed. - -Upsert surfaces may perform the bounded semantic retry. Strict insert surfaces, -including `load --mode append`, do not change meaning under contention: both an -already-present match from `WhenMatched::Fail` and a concurrent same-key commit -normalize to typed `KeyConflict` / HTTP 409 for the whole strict operation. They -do not switch to `UpdateAll`; callers decide whether to resubmit. Other strict -read-modify-write surfaces retain their typed write conflict. Retry exhaustion -on a non-strict upsert remains a retryable 409. - -Fencing covers the PK insertion race only. `@unique` values on different IDs, -edge RI, and cardinality depend on a read set. Their correctness requires the -read-set-in-CAS design or equivalent revalidation before HEAD movement; this RFC -does not claim that fences close those races. - -## 7. Version and fleet barrier - -### 7.1 No partial activation in the old format - -OmniGraph MUST NOT annotate a new data table and advertise fencing while the -graph remains generally writable by older binaries. An older process can select -the legacy merge path or keyed append and bypass the guarantee. - -This RFC owns its activation boundary: - -- operators quiesce every server, CLI writer, and embedded writer for the - graph; -- one migration claimant holds an atomic create-if-absent claim with a random - owner/fencing token; a native Lance branch sentinel is not accepted as CAS; -- only the dedicated migration binary may open the old graph for writes; -- the fencing-compatible stamp is written after every branch/table verification; -- normal serving begins only after the stamp; older binaries then refuse. - -The migration claim uses the storage adapter's `PutMode::Create` contract, -records operation/owner token, and has no time-only takeover. Recovery under the -fleet outage must classify the migration ledger/sidecars before replacing a -stale token. - -The stamp is graph-wide and read from the reserved main manifest before any -named-branch open; selecting a named branch cannot bypass the compatibility -check. - -An in-process mutex is not a fleet barrier. A marker unknown to v4 binaries is -also not a fleet barrier. The operator procedure and cluster control plane must -keep old writers stopped until finalization. - -The exact format number is assigned when this RFC is accepted. If RFC-024 is -also accepted and ready, the two migrations may deliberately share its v5 -release after a combined failure-matrix review. If durable heads fail their -cost gate, fencing still proceeds with its own next compatible stamp; if -fencing is blocked upstream, durable heads need not wait. - -If durable heads are already active when fencing migrates, every PK metadata -version repoint also emits the identity-bearing journal event and matching -`table_head` transition in the same manifest CAS. If fencing lands first, its -format/stamp becomes an explicit predecessor that a later heads migration must -preserve. Acceptance covers heads-first, fencing-first, and co-release orders. - -### 7.2 New graphs - -A graph created directly at the fencing-compatible format creates every keyed -table with the PK metadata already present and enables only the write routing -in §5. There is no post-create annotation window. - -## 8. All-branch PK migration - -Migration operates on graph states, not merely table roots. The unit is every -reachable tuple: - -``` -(graph_branch, table_key, table_path, pinned_table_branch, pinned_table_version) -``` - -This matters for lazy branches: a graph branch may still point at an old main -table version whose schema predates the PK, even after main HEAD is annotated. - -Under the fleet barrier, the migration: - -1. enumerates and incarnation-pins every live graph branch; -2. folds each branch manifest and enumerates its live keyed tables; -3. validates that every pinned table image has non-null, unique `id` values; -4. acquires branch/table gates in RFC-022 order and freshly revalidates the - pinned tuple, schema identity, and migration claim; -5. writes a per-unit RFC-022 sidecar declaring expected branch/table state, the - optional native fork effect, PK metadata effect, and intended manifest delta - before either effect can persist; -6. for an owned table branch, commits a set-if-absent PK metadata update; -7. for a lazy-inherited state, forks an owned table branch from the *exact - pinned version*, applies the PK metadata there, and leaves row contents - unchanged; -8. records exact achieved fork identity and table version in the sidecar; -9. publishes that graph branch's manifest to the annotated physical version - with an audited migration marker but no graph-content commit or graph-head - movement, including a table-head transition when heads are active; -10. records a branch/table completion digest; -11. re-enumerates branches and verifies every live branch before writing the - fencing-compatible stamp. - -PK installation advances a Lance table version before the graph manifest can -publish it, and a lazy fork creates native ref state first. The sidecar covers -both gaps and lets recovery reclaim or adopt the exact fork rather than infer -from a branch name. Because Lance forbids clearing/changing a set PK, -migration is roll-forward-only: - -- an already-correct PK is an idempotent success and is not rewritten; -- an absent PK resumes installation; -- a different PK is a loud, operator-visible refusal; -- recovery never attempts to "undo" PK metadata. - -Branch create/delete, schema apply, and normal data writes remain blocked for -the whole enumeration/install/verify interval. The migration ledger makes a -crash resumable without treating partial annotation as a served graph. - -## 9. Preservation after activation - -Once set, the following are storage invariants: - -- a table overwrite carries the same PK field IDs and positions; -- schema apply cannot remove, replace, reorder semantically, or make nullable a - PK field; -- rename preserves the PK field ID and metadata; -- branch fork/clone preserves it; -- import/rebuild creates it before accepting data; -- recovery restore may select an older data image only if that image is already - fencing/PK-compatible; -- a table recreation uses a new table incarnation but installs the same - catalog-derived PK contract at creation; -- `__manifest`'s existing legacy PK key form is preserved exactly as stored; - the migration never rewrites or "normalizes" it. Lance forbids changing a - set PK, and the native-namespace decoupling documented in the Lance - alignment audit depends on that legacy form remaining in place. - -Every open-for-write path verifies the physical schema matches the catalog PK -contract. The check is against the pinned physical schema and is not a -maintained parallel registry. - -## 10. Recovery and multi-process scope - -All data writes retain the existing Phase A-D sidecar protocol. The key filter -does not close the table-HEAD-before-graph-manifest window. - -The fenced data-table transaction is cross-process safe in its failure-free -commit path. OmniGraph's current recovery sweep, however, serializes with live -writers only in-process; a foreign recovery process can still inspect a live -sidecar, and destructive `Restore` cannot be made convergence-idempotent. - -Therefore this RFC MUST use one of two honest dispositions: - -1. retain the documented single-writer-process support boundary and describe - fences as closing the silent key-race primitive only; or -2. land a cross-process sidecar claim/lease before advertising general - multi-process writes. - -Fences alone are not evidence for disposition 2. - -## 11. Tests and acceptance gates - -### 11.1 Lance surface guards - -- exact PK ON set + v2 path produces a non-empty filter for inserts; -- `WhenMatched::Fail` preserves that filter and reports an existing match - without writing; -- mismatched ON set produces no filter; -- legacy/indexed path behavior is pinned until replaced; -- filtered/filtered overlapping keys retry; -- filtered/filtered disjoint keys may rebase; -- filtered/unfiltered retries in **both** commit orders; -- filtered Update/bare Append retries in **both** commit orders; -- PK metadata cannot be changed or removed once installed. - -### 11.2 Engine concurrency tests - -- the same-key DST cell becomes a hard assertion with N concurrent writers; -- different keys remain concurrently writable; -- every keyed load/mutation/merge/fold path is observed to use the filtered - primitive; -- strict append of an existing `id` still fails and never updates the row; -- strict pre-existing-match and concurrent-insert cases normalize to the same - external `KeyConflict` while preserving `WhenMatched::Fail`; -- a source-walk guard rejects keyed `stage_append`, including the former - fast-forward path; -- a retry reruns validation rather than committing the stale staged batch. - -### 11.3 Migration and recovery tests - -- main plus owned and lazy-inherited graph branches all emerge PK-annotated; -- duplicate historical IDs abort before the fencing-compatible stamp; -- crash after each table annotation and before each manifest repoint resumes - without data change; -- crash before/after lazy fork and PK metadata commit recovers the exact - sidecar-recorded ref/version; -- branch enumeration is incarnation-safe; -- old binary/new graph and new binary/partially migrated graph refuse loudly; -- heads-first, fencing-first, and co-release upgrades preserve every active - format capability and produce identical logical rows; -- overwrite, schema apply, branch fork, restore, and import preserve PK metadata. - -### 11.4 Cost gate - -Measure a small upsert into 10K, 100K, and 1M-row indexed tables using the -shared cost harness. If `use_index(false)` makes work scale with table size -beyond the accepted budget, the indexed-path upstream work is a ship blocker. - -Additionally measure the bulk adopt-merge shape that motivated the keyed -fast-forward path (#277): a many-row, all-new-rows fenced merge into an -embedding-bearing table, asserting peak memory bounded by batch size rather -than table or delta width. If the fenced path cannot meet that bound, the -keyed fast-forward removal waits for the mitigation named in §5.1. - -> 💬 **Instrument required (tightening 5 in the -> [review ledger](../dev/rfc-022-027-architecture-review.md)):** -> `helpers::cost` measures I/O, not peak RSS, so this memory bound is -> unenforceable as written. Use the subprocess `scenarios.rs` harness or an -> equivalent `wait4`/`ru_maxrss` instrument, and name dataset sizes, baseline, -> cap, and pass threshold. - -## 12. Decisions and open gates - -### Decided - -- `id`, not `src`+`dst`, is the edge PK. -- No keyed append, including optimization-only append. -- No mixed-fleet or new-table-only v4 rollout. -- PK migration is all-branch, offline, idempotent, and roll-forward-only. -- A retryable upsert conflict retries the logical operation; strict insert maps - both existing and concurrent matches to `KeyConflict` without changing mode. -- Read-set validation remains a separate required concurrency design. - -### Open ship gates - -1. Upstream symmetric filtered/unfiltered and filtered/Append conflict behavior. -2. v2-path scale result versus indexed-path filter availability. -3. Operator repair procedure for pre-existing duplicate IDs. -4. Rename-stable field/type identity. -5. Cross-process recovery ownership before any broadened topology claim. -6. Final format-number/release sequencing and the all-branch fleet stamp. -7. The fenced bulk adopt-merge memory/cost gate on embedding-bearing tables - (the #277 regression class) — see §5.1 and §11.4. diff --git a/docs/rfcs/rfc-026-memwal-streaming-ingest.md b/docs/rfcs/rfc-026-memwal-streaming-ingest.md deleted file mode 100644 index 24761fc8..00000000 --- a/docs/rfcs/rfc-026-memwal-streaming-ingest.md +++ /dev/null @@ -1,415 +0,0 @@ ---- -type: spec -title: "RFC-026 — MemWAL streaming ingest" -description: Adopts Lance MemWAL as OmniGraph's strategic streaming-write architecture, with durable per-row acknowledgement, graph-atomic folds, epoch-fenced quiescence, and explicit fresh-read cuts on the RFC-022 unified write path. -status: draft -tags: [eng, rfc, streaming, ingest, wal, memwal, lance, omnigraph] -timestamp: 2026-07-10 -owner: ---- - -# RFC-026 — MemWAL streaming ingest - -**Status:** Draft / for team review -**Date:** 2026-07-10 -**Depends on:** [RFC-022](rfc-022-unified-write-path.md)'s unified write and -generic recovery-sidecar protocol, plus -[RFC-023](rfc-023-key-conflict-fencing.md) for the initial keyed graph-stream -mode. Durable heads from -[RFC-024](rfc-024-durable-table-heads.md) are compatible but not required. -**Surveyed:** omnigraph 0.8.1; Lance 9.0.0-beta.15 (`f24e42c1`); complete MemWAL format specification -**Audience:** engine, server, CLI, policy, and operations maintainers -**Open architecture review:** [RFC-022–027 review ledger](../dev/rfc-022-027-architecture-review.md). -Findings marked **BLOCKER** must be dispositioned before acceptance. - ---- - -## 0. Decision and risk posture - -OmniGraph adopts Lance MemWAL as its strategic streaming-write architecture. -MemWAL is a major Lance architectural bet: a sharded LSM write path with durable -WAL entries, flushed Lance generations, merge progress committed with base-table -data, maintained indexes, and epoch-fenced writers. OmniGraph consumes that -architecture rather than building a WAL, shard protocol, or LSM reader. - -This RFC does **not** characterize the architecture as experimental. The risk is -narrower: Rust API names, some format details, and operational helpers are still -maturing across Lance releases. We manage that API/format-maturity risk with a -small adapter, compile/runtime surface guards, a quiescence requirement before -Lance upgrades, and a fresh full-spec alignment audit on every bump. It is not a -reason to fork or reimplement MemWAL. - -The contract is: - -- stream acknowledgement means the row's WAL entry is durable; -- acknowledgement does not mean graph visibility; -- default queries see only the manifest-committed graph; -- a fold is an ordinary RFC-022 graph writer and is the sole visibility point; -- fresh reads are explicit and never claim cross-table atomicity. - -## 1. Scope and non-goals - -This RFC specifies enrollment, the public stream API, acknowledgement semantics, -folding, fold-time integrity, dead-letter atomicity, branch/schema quiescence, -fresh-read cuts, resource bounds, observability, testing, and upgrade posture. - -It does not replace `load` or `mutate`, provide cross-query transactions, store -manifest mutations in MemWAL, create a second metadata authority, or weaken -default snapshot isolation. Stream-mode -deletes remain out of the first delivery and require the Lance tombstone surface -plus a separate acceptance pass. - -## 2. Stream mode and key semantics - -Initial delivery exposes -`@stream(mode="upsert", on_reject="strict")` on a node or edge type; -`on_reject` accepts `strict` or `dead_letter`. -It requires the table's immutable unenforced primary key to equal OmniGraph's -merge key: `id` for nodes and edges. All occurrences of one key map to one shard -and MemWAL applies last-write-wins ordering. - -Public append mode is deliberately out of scope. Nodes and edges always have -logical identity; allowing a retry to append the same `id` twice would violate -that contract. A future explicitly keyless, non-graph append-only table class -may consume MemWAL append semantics under its own schema/API decision. - -Stream ordering intentionally differs from the interactive fence: - -- concurrent interactive same-key writes serialize or fail/retry loudly; -- same-key stream entries resolve by MemWAL generation/position order; -- duplicate keys inside one bulk-load input retain the existing load error. - -The schema and user docs state all three together. - -## 3. Enrollment is a recoverable inline commit - -Schema apply records `@stream` intent only. First stream use enrolls the physical -table by creating the singleton `__lance_mem_wal` system index and its sharding -configuration. - -Enrollment advances Lance HEAD inline. It therefore uses the RFC-022 generic -recovery protocol, not an ad-hoc state machine: - -1. run and await RFC-022's synchronous recovery barrier; -2. authorize, pin the manifest/schema/table state, and prepare a complete - `ReadSet` containing schema identity, table entry/head, PK metadata, stream - intent/configuration, and lifecycle-row absence; -3. acquire any global claims and then the `(table, branch)` write queue in - RFC-022 order, then freshly revalidate the complete `ReadSet`; a mismatch restarts - before any inline effect; -4. verify RFC-023's already-installed PK; enrollment never performs a first-use - PK migration; validate the sharding configuration; -5. write a generic sidecar with writer kind `stream_enrollment` and pre-commit - table pin; -6. create the MemWAL index, advancing Lance HEAD; -7. publish the new table version plus `stream_state = OPEN` in one manifest CAS, - including the table-head row when RFC-024 is active; -8. delete the sidecar best-effort after publication. - -Recovery rolls the enrollment forward or back under the same classification and -audit machinery as other inline residuals. Repeating enrollment with identical -metadata is a no-op. A different PK, sharding spec, maintained-index set, or -writer-default configuration is a typed conflict. - -No row is acknowledged until enrollment is manifest-committed. - -Initial delivery supports one unsharded shard per `(table, main)`. Non-main -branches remain refused until the Lance branch-scoping question is proven by a -surface guard and end-to-end test. Later `bucket(id, N)` sharding must preserve -the one-key-to-one-shard rule. - -## 4. New public API - -The shipped `POST /graphs/{id}/ingest` path remains the deprecated, compatible -alias of `/load`. Streaming receives a new, non-conflicting surface: - -```text -POST /graphs/{graph_id}/streams/{type_name}/ingest?branch=main -GET /graphs/{graph_id}/streams -GET /graphs/{graph_id}/streams/{type_name} -POST /graphs/{graph_id}/streams/{type_name}/fold -POST /graphs/{graph_id}/streams/{type_name}/quiesce -POST /graphs/{graph_id}/streams/{type_name}/resume -``` - -The ingest request and response use `Content-Type: application/x-ndjson` and -`Accept: application/x-ndjson`. - -Each input line is one row payload. Each output line corresponds to the same -input ordinal: - -```json -{"ordinal":17,"status":"durable","shard_id":"...","writer_epoch":8,"wal_position":42} -``` - -Synchronous validation failures return a per-row error before a WAL append. -Previously acknowledged rows in the same request remain durable; the response -is a stream, not an all-request transaction. Ordering, cancellation, and retry -rules are explicit: - -- acknowledgements are emitted in input order for one HTTP stream; -- disconnecting does not cancel entries whose durability waiter resolved; -- a missing response is ambiguous; retrying the same `id` and payload may add - another WAL entry but produces the same last-write-wins graph state; -- server shutdown stops admission, drains durability waiters up to a bound, and - reports any unacknowledged tail as unknown to the client. - -CLI commands mirror the new namespace rather than overloading deprecated -`omnigraph ingest`: - -```text -omnigraph stream ingest <type> --data <ndjson> ... -omnigraph stream status [<type>] ... -omnigraph stream fold [<type>] ... -omnigraph stream quiesce [<type>] ... -omnigraph stream resume [<type>] ... -``` - -Every endpoint has a dedicated OpenAPI operation and handler tests. Ingest -passes the engine `stream_ingest` Cedar action and per-actor admission -accounting before acquiring a shard writer; fold/quiesce/resume use a separate -`stream_manage` action. Status is authorized like other graph operational -metadata. The same engine gates apply to embedded and remote CLI use. - -## 5. Ack-path validation and writer lifecycle - -Before append, OmniGraph applies checks that need no base-table read: Arrow -shape/type, required/default fields, enum/range/check constraints, reserved -columns, and stream mode. RI, cardinality, cross-version uniqueness, and -external embedding computation remain fold-time work. - -One warm `ShardWriter` is held per active shard behind a bounded registry. The -registry has idle eviction and hard limits for resident writers, MemTable bytes, -unflushed WAL bytes, pending generations, and per-actor inflight bytes. Exceeding -a bound backpressures with a typed retryable response; it never drops a row. - -Initial topology has one active ingest owner for each `(graph, table, main)` -shard. MemWAL's epoch fence makes restart/failover safe; it is not a load -balancer. A deployment with multiple server replicas must route a shard to its -current owner (or return a typed retry/redirect) instead of letting replicas -reclaim the epoch per request. General multi-owner routing waits for the -multi-shard phase and its ownership protocol. - -## 6. Fold protocol - -The fold consumes flushed generations in ascending order. Embeddings and -base-dependent validation run outside the table queue and register every probed -table plus stream configuration/generation in RFC-022's `ReadSet`. The commit -phase then: - -1. stages accepted rows with Lance merge-insert and includes - `merged_generations` in that transaction; -2. stages any rejection/audit rows required by §7; -3. acquires every affected queue in canonical sorted order and revalidates the - complete `ReadSet`; any mismatch discards and replans the whole fold; -4. writes one generic RFC-022 recovery sidecar before the first - `commit_staged` call; -5. commits every staged Lance transaction; -6. publishes all data/internal table versions and lineage in one `__manifest` - CAS, including table heads when RFC-024 is active; -7. deletes the sidecar after successful publication. - -The sidecar is mandatory even though merge-insert is staged. After -`commit_staged`, Lance HEAD and `merged_generations` have moved while the graph -manifest has not. A failure in that window is the ordinary multi-table recovery -gap, not invisible staged state. - -MemWAL generation GC starts only after the exact fold is graph-visible, its -sidecar is resolved, index catchup permits reclamation, and no `FreshReadCut` -retention guard references the generation. Data-HEAD merge progress alone is -never permission to delete the only fresh-tier copy. - -Concurrent folders reload `merged_generations`: a generation already committed -is skipped; otherwise the fold is replanned from current state. A forced fold -stops new flush creation for its cut, waits for in-flight durability waiters, -and never aborts an acknowledged row. - -Fold lineage uses `omnigraph:ingest` as the mechanism actor and persists the -authenticated contributor actor for every folded WAL range in the same internal -audit participant. Commit and status output can therefore answer both who ran -the fold and who supplied the data after WAL GC. - -## 7. Fold-time rejection is atomic - -`strict` is the default. A permanent RI, cardinality, uniqueness, or embedding -failure stops the fold at the offending generation, marks the shard blocked, -and backpressures new ingestion once configured lag bounds are reached. - -`dead_letter` is explicit. `_ingest_rejects` is then a versioned internal Lance -table and a participant in the same fold pipeline, not best-effort state outside -the commit protocol. Every reject has deterministic identity: - -```text -(table_key, shard_id, generation, wal_position) -``` - -The fold stages reject rows, accepted rows, and merge progress before committing -any of them. The generic sidecar covers every participant; the single manifest -publish records both the base-table and reject-table versions. Replay is -idempotent by reject identity. There is no ordering in which progress can become -visible while the corresponding rejection is lost. - -`stream status` reports blocked generations and typed reject details. Reject -retention is explicit and cannot be shorter than the WAL/fold audit retention -needed to explain a durable acknowledgement. - -## 8. Epoch-fenced quiescence barrier - -Branch operations, schema changes, stream teardown, and Lance upgrades require a -real barrier, not an empty check. - -Each enrolled table has a durable -`stream_state:<stable-table-id>:<incarnation>` row in its manifest branch with -`OPEN | DRAINING | SEALED`, configuration hash, and epoch floor. The row is the -logical lifecycle authority and is updated by an RFC-022 CAS; MemWAL shard -epochs are the physical writer fence. Neither an in-memory registry nor an -empty-generation observation can substitute for both. -Lifecycle-only transitions are audited manifest metadata transactions; they do -not create graph-content commits or move `graph_head`. - -The shared drain sequence is: - -1. publish stream intent `OPEN -> DRAINING` for the target table/branch; -2. increment and persist each shard writer epoch and seal claims, fencing stale - writers across processes; -3. reject or backpressure new appends; -4. wait for in-flight durability waiters, flush active MemTables, and fold every - generation to empty; -5. verify shard manifests and base `merged_generations` agree on emptiness; -6. publish `DRAINING -> SEALED` with the verified generation/epoch cut; -7. for an operation-scoped drain, perform the guarded operation; persistent - public quiesce stops after step 6. - -Each lifecycle CAS is an RFC-022 authority-first metadata write; each fold is a -separate normal RFC-022 graph write. `DRAINING` fully encodes the target epoch -floor, so an interrupted epoch mutation is idempotently resumed from that row. -The sequence is not one giant sidecar spanning multiple commits. - -There are two dispositions after the drain reaches `SEALED`: - -- **operation-scoped drain** — branch/schema maintenance automatically publishes - `SEALED -> OPEN` with a newer epoch only after the guarded operation succeeds - and the stream contract remains compatible; -- **persistent quiesce** — the public `quiesce` command leaves the stream - `SEALED`. It never auto-reopens. `stream resume` explicitly revalidates schema, - PK, configuration, MemWAL format, and epoch, then publishes a newer `OPEN` - state. Stream teardown deletes intent only from `SEALED`. - -The barrier never holds the table write queue while waiting for a fold that -needs that queue. State transition and epoch fencing happen first; fold commit -then acquires the normal queue. Crash recovery resumes from the durable state -and epoch. - -Schema apply must drain every affected enrolled type before changing fields, -constraints, PK, embeddings, or `@stream` and resumes only when compatible. - -A Lance version upgrade requires persistent `stream quiesce --all`, but empty -generations alone are insufficient: the MemWAL system index, shard manifests, -epoch records, and generation directories may still use the old format. Before -the bump, the implementation must prove one of: (a) upstream guarantees and -cross-version tests cover every retained MemWAL artifact, (b) a public Lance -metadata migration converts them, or (c) OmniGraph tears down the enrolled -MemWAL metadata under recovery and re-enrolls after the bump. Without one of -those gates the upgrade refuses; `resume` never opens unverified old metadata. - -## 9. Fresh-read cuts - -Freshness is a first-class engine/IR enum: - -```text -Committed -Fresh -``` - -At query planning, `Fresh` captures one `FreshReadCut` containing: - -- the ordinary manifest snapshot; -- each selected shard-manifest version and writer epoch; -- included flushed-generation paths and maximum generation; -- the active same-process MemTable row-position watermark, when available; -- the base table's `merged_generations` and index-catchup state read from the - exact table version selected by the manifest snapshot, never from live HEAD. - -Capture uses a retrying handshake: - -1. read the selected shard manifests/epochs, acquire Lance generation retention - guards for the flushed files in the tentative cut, and under one - same-process writer snapshot capture/pin any active-MemTable watermark; -2. pin the graph manifest snapshot and read `merged_generations` from each exact - base-table version it selects; -3. re-read the shard manifest versions/epochs; any epoch/configuration change - restarts the whole capture; -4. if a generation from step 1 disappeared, accept that only when the pinned - base's `merged_generations` proves it is included; otherwise release guards, - discard the whole graph snapshot, and retry from step 1; -5. exclude generations that appeared after step 1 and hold the generation and - MemTable read guards captured in step 1 until query completion. - -If Lance exposes no guard that prevents generation GC for the query lifetime, -cross-process `Fresh` does not ship. A missing generation is never interpreted -as “probably folded” against an older pinned base. - -Execution never refreshes that cut mid-query. It excludes every flushed -generation `<= merged_generations[shard]`; otherwise old WAL data could outrank -or duplicate its newer base-table image. - -Fresh reads have no cross-table atomicity. Same-process active MemTables provide -read-your-writes; other processes can promise only the latest flushed state -captured by their shard-manifest reads. The HTTP request and query docs state -those limits wherever the tier is exposed. - -## 10. Observability and resource contracts - -Per shard expose durable WAL position, replay position, active epoch, current -generation, flushed and merged generation, index catchup, pending rows/bytes, -oldest acknowledged age, last fold error, blocked reject, and quiescence state. - -Metrics cover ack latency, durability-wait batching, fenced writers, replayed -entries, fold rows/bytes/generations, fold retries, lag, reject counts, and -sidecar recovery. Defaults for every byte/count/time bound are documented and -configuration changes are observable behavior. - -`stream status` resolves the exact lifecycle rows and MemWAL metadata through a -structured, bounded access path; it may reuse RFC-024's scalar-index machinery -but cannot claim history-flat cost while scanning manifest history. - -## 11. Acceptance gates - -- Surface guards pin claim, append, durability waiter, flush, epoch fencing, - staged merge with `merged_generations`, index catchup, and seal/reopen APIs. -- A WAL append failure emits no durable acknowledgement. Every acknowledged row - survives crash, replay, fold, and recovery. -- Failpoints cover enrollment's inline-commit gap and every fold participant - around sidecar, `commit_staged`, reject persistence, and manifest publish. -- Enrollment restarts without an inline effect when schema, PK, table head, or - stream configuration changes between prepare and gated revalidation. -- Two folders converge exactly once; a fenced stale writer can never produce a - false durable acknowledgement after WAL GC. -- Two server replicas do not epoch-ping-pong one shard; owner failover fences - the old process before the new owner acknowledges. -- Quiescence tests race appends with branch and schema operations across two - coordinators and prove no post-drain tail appears; failpoints cover every - lifecycle-row, epoch-fence, fold, and `SEALED` boundary. -- Persistent quiesce never auto-reopens; explicit resume validates a newer - epoch. Upgrade tests cover every retained MemWAL artifact through declared - compatibility, migration, or teardown/re-enrollment. -- Fresh reads race capture with fold/GC, retry on an unexplained disappearing - generation, hold generation/MemTable guards through execution, exclude merged - generations, and document cross-table inconsistency explicitly. -- Server/OpenAPI tests preserve the old `/ingest` alias and cover the new route; - CLI parity covers embedded and remote stream commands. -- Ack-path object-store operations are O(1) and flat in graph history and WAL - depth. Fold cost is bounded by generations/rows folded, not graph history. -- S3 correctness runs against RustFS; API/format guards are rerun before every - Lance bump. - -## 12. Phasing - -| Phase | Content | Gate | -|---|---|---| -| A | MemWAL adapter, surface guards, enrollment sidecar | inline-commit crash matrix | -| B | new ingest route/CLI, durable ack, strict fold | ack durability; API compatibility; cost budget | -| C | atomic dead letter, audit provenance, status/bounds | reject crash matrix; backpressure tests | -| D | epoch-fenced drain, persistent quiesce/resume, schema/branch/upgrade integration | two-coordinator race and format-transition suite | -| E | fresh cuts and maintained-index reads; cross-process `Fresh` ships only if the substrate generation-retention guard exists (§9), otherwise same-process only | cut consistency; merged-generation exclusion | -| F | multi-shard upsert and stream deletes | one-key-one-shard proof; Lance re-audit |