feat: native single-hop Expand via CSR adjacency index (Phase 2, #159)

[jja725]

Summary

Phase 2 of #159: execute single-hop Cypher Expand natively using the Phase 1 CsrIndex instead of relationship-scan + two SQL joins. Implemented as custom DataFusion operators, with automatic fallback to the existing join path for anything not yet supported natively.

This is a DuckPGQ-style relational engine + CSR acceleration integration:

• CsrExpandExec — topology only: for each source row, look up neighbors in the CSR and emit one row per neighbor with the neighbor's row id appended.
• LanceTakeExec — materializes target node properties from those row ids via a RowMaterializer (in-memory arrow::compute::take now; a Lance-dataset take lands in Phase 4).
• CsrExpandNode / LanceTakeNode — logical extension nodes; CsrExtensionPlanner + CsrQueryPlanner build the CSR and materializer at physical-planning time.
• LanceNativePlanner overrides only Expand lowering and delegates everything else to DataFusionPlanner, so ExecutionStrategy::LanceNative is always correct — it uses CSR when it can and joins otherwise.

Design decisions

• Dense-ROWID model: the CSR vertex id is the node's row id (csr.neighbors(src_rowid) -> dst_rowids), mirroring how every Lance index works (key → row ids → take() to materialize). Generalizes to Lance stable row ids in Phase 4.
• Materialize all target columns via take() (schema parity with the join path) rather than analyzing which are referenced.

Full design and task breakdown: docs/superpowers/specs/2026-06-22-csr-native-expand-operator-design.md and docs/superpowers/plans/2026-06-22-csr-native-expand-operator.md.

Native vs. fallback

Served natively: exactly one single-hop Expand, single relationship type, Outgoing/Incoming, no inline relationship/target property filters, no bound relationship variable. Wrapping Project/Filter/Sort/Limit/Offset/Distinct run as normal DataFusion operators on the native stream.

Falls back to the DataFusion join path: variable-length / multi-hop, multiple relationship types, undirected, inline {k:v} filters, bound relationship variable, Join, Unwind.

Out of scope (later phases)

• Multi-hop / VariableLengthExpand, BFS/DFS/shortest-path operators (Phase 3)
• Persisting CSR as Lance datasets, incremental updates, stable row ids, LanceDatasetMaterializer, namespace native path (Phase 4)
• Hybrid CSR + vector search (Phase 5)

Test plan

• Unit tests for expand_batch, take_batch, InMemoryMaterializer, the CSR builder column generalization, and the planner native/fallback decision.
• End-to-end parity tests (tests/test_lance_native_expand.rs) asserting LanceNative returns identical results to DataFusion for: single-hop RETURN a.name, b.name, with WHERE b.age > 30, incoming direction, and a variable-length query (fallback).
• Full crate suite green: cargo test -p lance-graph (15 binaries, 0 failures); cargo clippy -p lance-graph --all-targets clean.

Known follow-ups (not blocking)

• Resolve the source-id column index at execution time (by name) rather than at plan time — robust against future multi-partition / column-pruning passes (safe under DataFusion 50 today).

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 58.00604% with 417 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...rates/lance-graph/src/lance_native_planner/take.rs	53.08%	114 Missing ⚠️
...lance-graph/src/lance_native_planner/csr_expand.rs	49.09%	112 Missing ⚠️
crates/lance-graph/src/lance_native_planner/mod.rs	72.52%	100 Missing ⚠️
crates/lance-graph/src/query.rs	8.33%	55 Missing ⚠️
...raph/src/lance_native_planner/extension_planner.rs	0.00%	32 Missing ⚠️
crates/lance-graph/src/csr_index.rs	94.36%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

[beinan]

The DataFusion extension-node direction is reasonable, but the current implementation is not correct for the existing GraphConfig contract.

I reproduced three blocking failures:

• Valid Int64 endpoint IDs are selected for native execution and fail instead of falling back.
• With node IDs [1, 0] and edge 1 -> 0, native returns b.id = 0 with the properties from physical row 0, silently mixing identity and data.
• A bound relationship variable is selected for native execution and fails with a missing r__since column instead of falling back.

The PR-specific tests pass because they only use UInt64 IDs that exactly match 0-based batch offsets. The full workspace suite also passes, but it does not cover the public ID semantics above. Please address the ID/address model and the native capability checks before merging.

One additional correctness case to cover in this builder is nullable edge endpoints: reading a null slot with value(i) can create a spurious edge, whereas the DataFusion inner-join path does not match null endpoints.

For the longer-term native design, logical node IDs, compact CSR ordinals, and Lance row IDs/addresses should be separate concepts. Adjacency entries will also need an edge row ID to support relationship properties and parallel-edge semantics. The materialization/adjacency provider interfaces should be asynchronous and version-aware before the persisted Lance implementation, since Lance dataset take is asynchronous and graph queries span independently versioned datasets.

[beinan]

The value used here comes from the relationship endpoint column; it is a logical node ID, not necessarily a batch row offset. GraphConfig::NodeMapping::id_field does not require IDs to be ordered, dense, or equal to 0..num_rows. Passing the value directly to Arrow take() can therefore return silently mismatched properties.

I reproduced this with node IDs [1, 0] and edge 1 -> 0: native returned b.id = 0 but b.name from physical row 0 ("one") instead of the row whose ID is 0 ("zero"). Please introduce an explicit logical-ID-to-row-address mapping for the in-memory path, or reject/fall back unless the invariant is validated.

[beinan]

This hard UInt64 requirement is not part of the native capability check. Existing repository fixtures use Int64 node and relationship IDs, so a valid single-hop query is selected for native execution and then fails during physical planning with src_id column must be UInt64, instead of using the DataFusion fallback.

Please either normalize all supported integer endpoint columns here, or inspect the relationship schema before selecting native execution and fall back when the ID representation is unsupported. Sparse/string IDs still require an explicit ID-to-ordinal mapping rather than a cast.

[beinan]

walk_supported does not inspect relationship_variable, while build_expand_native does not materialize any relationship-table columns. As a result, MATCH (a)-[r:KNOWS]->(b) RETURN r.since is selected for native execution and fails with No field named r__since. The PR description says bound relationship variables should fall back. Please reject relationship_variable.is_some() here and add an end-to-end parity test.

[beinan]

This operator is partition-preserving, but it advertises exactly one output partition. execute(partition) then forwards only that partition number to the child. With a multi-partition child, DataFusion can request only output partition 0 and the remaining child partitions are never consumed.

Please inherit the child output partitioning (the same issue exists in LanceTakeExec), or declare a single-partition input requirement and ensure a coalesce is inserted. Add a multi-partition MemTable execution test.