Evaluate RFC-024 durable-head lookup gate

[aaltshuler]

Summary

• add an isolated, production-shaped Gate A cost matrix for exact durable-head lookup across compacted/uncompacted history, cold/warm access, absent/reconciled/partial BTREE coverage, and local/RustFS backends
• prove a public physical-ref ABA token from BranchIdentifier, current Transaction.uuid, and ManifestLocation.e_tag on local and RustFS, including same-version main and named-ref replacement
• record the measured no-go: exact indexed scan work is bounded, but the complete latest-manifest/object-byte path is not history-flat
• mark RFC-024 research-blocked, keep production on internal schema v6, and pin the exact RustFS token guard in correctness CI

Gate A result

Reconciled RustFS term, 20 to 80 publishes	Result
Uncompacted cold object reads	34 to 94
Uncompacted cold object bytes	about 61.9 KB to 121.6 KB
Compacted cold object bytes	about 30.5 KB to 61.6 KB
Compacted cold plan bytes	about 39.1 KB to 61.9 KB
Compacted warm object bytes	about 22.9 KB to 45.4 KB

The beta.21 rows_scanned proxy, ranges, result fragments, and BTREE pages remain bounded at fixed catalog width. The absent-index control grows, one/eight-fragment uncovered tails remain correct and observable, and optimize_indices restores full coverage. Those passing terms do not cancel the failing physical cost terms.

Correctness boundaries

• no production manifest schema, format number, publisher, or write-path code is added
• the v6 journal-fold current-state path remains authoritative
• the physical token is an ABA fence for the supported OmniGraph writer topology, not authentication
• substrate token proof is closed; publisher-level stale-writer rejection remains unimplemented because the heads publisher does not ship
• a successor RFC-024 access shape must pass the original gate rather than narrow it

Validation

• cargo test --workspace --locked
• focused durable_head_lookup_cost, lance_surface_guards, write_cost, merge_cost, and warm_read_cost suites
• configured RustFS exact physical-token guard: 1 passed
• configured RustFS Gate A cost matrix: 1 passed
• scripts/check-agents-md.sh
• git diff --check

Greptile Summary

This PR evaluates the RFC-024 durable-head lookup design and records the result. The main changes are:

• Adds local and S3 cost matrices across history, index, compaction, and cache states.
• Adds physical-reference ABA checks for main and named references.
• Adds cold-open object-store I/O tracking.
• Records the measured no-go and keeps the production manifest on schema v6.
• Runs the RustFS physical-reference guard in CI.

Confidence Score: 5/5

This looks safe to merge.

• The S3 matrix now cleans its unique prefix before propagating a matrix panic.
• The focused test confirms cleanup runs on the panic path.
• The RustFS job supplies the required S3 configuration and rejects a zero-test match.
• No blocking issues remain in the updated code.

Important Files Changed

Filename	Overview
.github/workflows/ci.yml	Runs the exact RustFS physical-reference guard and checks that one test executed.
crates/omnigraph/tests/durable_head_lookup_cost.rs	Adds the Gate A cost matrix and ensures S3 fixtures are cleaned before a matrix panic is propagated.
crates/omnigraph/tests/helpers/cost.rs	Adds tracked dataset opening so cold measurements include manifest-resolution I/O.
crates/omnigraph/tests/lance_surface_guards.rs	Adds physical-reference ABA coverage for local and RustFS storage.
docs/rfcs/0024-durable-table-heads.md	Records the Gate A result and marks the proposal as research-blocked.

_{Reviews (2): Last reviewed commit: "Clean up RFC-024 S3 fixtures after failu..." | Re-trigger Greptile}

Context used:

• Context used - AGENTS.md (source)
• Context used - CLAUDE.md (source)