perf(encoding): auto-select sparse structural pages

[Xuanwo]

Part of #7750

Summary

• In Lance 2.3+, automatically choose sparse structural pages only when the structural encoding is implicit, the dense mini-block rep/def plan requires top-level page splitting or has a single row over budget, and the sparse writer supports the value path.
• Keep within-budget pages on the dense path without constructing a sparse value block or sparse plan.
• Preserve every explicit structural mode, Lance 2.2 behavior, and dense fallback/error behavior for dictionary, variable packed-struct, and other sparse-ineligible value paths.
• Keep the policy decision separate from sparse serialization; this adds no wire-format fields.

This is the policy/performance layer that turns the already-readable and explicitly-writable sparse format on automatically only for over-budget eligible pages.

Dependencies

• PR4 #7755 is the direct stack base and provides the explicit sparse writer.
• PR1 #7751 is the naming dependency. It is already merged, but this stacked PR intentionally uses the names present in PR4 and does not duplicate its pure rename; the stack can absorb that naming change through its normal base update.

Verification

• cargo fmt --all
• cargo check --workspace --tests --benches
• cargo test -p lance-encoding encodings::logical::primitive::layout::tests (3 passed)
• cargo test -p lance-encoding encodings::logical::primitive::sparse::writer::tests (17 passed)
• cargo test -p lance-encoding --lib (464 passed, 5 ignored)
• cargo test -p lance-file (92 unit tests and doctests passed)
• cargo clippy --all --tests --benches -- -D warnings

The focused matrix covers within-budget lazy selection, split-required and single-row-over-budget auto selection, sparse eligibility, dictionary, variable packed-struct, wide fixed-size-binary, and wide fixed-size-list dense preservation, all explicit modes, Lance 2.2 compatibility, page counts, range/take reads, and round trips.

100M-row S3 benchmark

Environment: r7i.8xlarge in us-east-2b, Lance V2.3, 100 files of 1,000,000 rows, 65,536-row write batches, and 1,024-row takes. The final committed harness is at 66c936391aa46bfcd14f7ac1c820878da68fcc0b on xuanwo/sparse-stack-5-auto-sparse-bench-final, based on PR head ed0ce68d7f4cceafe6a249c912f7f1da8c4fecd9.

Cases:

• hnsw: List<UInt32>; the first 1/7 of rows (14,285,714 of 100M) contain 32 consecutive values, followed by an empty tail.
• uniform: List<UInt32>; one singleton non-empty row every 10,000 rows.
• deep: Struct<events: List<Struct<id: Int32, tags: List<Int32>, pair: FixedSizeList<Int32; 2>>>>; non-empty every 4,096 rows with nullable top struct, event list, event struct, id, tags, pair, and pair items.

Command:

EXPECTED_SHA=66c936391aa46bfcd14f7ac1c820878da68fcc0b \
STAGE=all \
RUN_ID=20260713T144500Z \
S3_PREFIX=xuanwo-sparse-pr5-20260713T133525Z \
RESULT_DIR=/home/ec2-user/sparse-pr5-results/20260713T144500Z \
benchmarks/sparse-structural-final/run.sh

cold is the first operation on a newly opened Dataset; warm is the immediate repeat on the same Dataset. Times are milliseconds. Bytes are total dataset object bytes. Every dataset has 102 objects, including 100 Lance data files.

Case	Mode	Bytes	Pages	Cold full scan	Cold random	Warm full scan	Warm random	Warm empty	Warm non-empty
hnsw	sparse	1,600,850,504	317	1,627.830	466.940	1,472.636	272.931	0.484	29.238
hnsw	mini-block	1,663,361,256	328	1,720.362	484.619	1,655.289	273.812	0.466	20.664
hnsw	full-zip	2,342,961,549	328	3,239.070	478.839	3,104.055	273.603	0.659	25.735
uniform	sparse	117,091	100	99.990	77.346	76.518	3.408	0.548	28.299
uniform	mini-block	689,601	1,600	111.844	137.503	606.852	123.301	93.404	94.822
uniform	full-zip	496,906,995	1,600	1,589.645	1,205.110	650.542	359.496	72.292	58.997
deep	sparse	1,144,900	300	159.361	77.122	135.065	9.309	0.884	25.580
deep	mini-block	376,199,150	10,100	6,735.485	4,673.904	2,560.397	582.959	174.014	189.929
deep	full-zip	994,111,217	10,100	failed	failed	failed	failed	failed	failed

deep/full-zip wrote all 100M rows and produced the reported size/page/object counts, then its first full scan panicked at primitive.rs:2588 by unwrapping a missing fixed full-zip decode task. Sparse and mini-block completed all reads. A separate 200K-row check at the exact PR4 base SHA da06b79a7f5eda4f33e2d80990c46766095ba383 independently confirmed that this nested-null deep/full-zip read was already broken before this PR, returning Structural validity has 6 entries for an array with 7 values. The precise failure site varies with this decoder path, but explicit full-zip never enters PR5's automatic sparse candidate closure, so this is not an auto-sparse regression. The policy deliberately does not disguise that behavior as sparse support.

Size-only comparisons

Parquet uses ZSTD level 3. Vortex uses 0.78.0 defaults.

Case	Format	Bytes
hnsw	Parquet ZSTD3	1,242,177,623
hnsw	Vortex default	1,120,652
uniform	Parquet ZSTD3	543,262
uniform	Vortex default	858,884
deep	Parquet ZSTD3	2,269,989
deep	Vortex default	29,698,760

Artifacts

• S3 CSV: s3://lance-bench-054483968661-us-east-2/xuanwo-sparse-pr5-20260713T133525Z/artifacts/lance-100m-20260713T144500Z.csv
• S3 failure/backtrace: s3://lance-bench-054483968661-us-east-2/xuanwo-sparse-pr5-20260713T133525Z/artifacts/lance-100m-20260713T144500Z.err
• S3 failure index: s3://lance-bench-054483968661-us-east-2/xuanwo-sparse-pr5-20260713T133525Z/artifacts/lance-100m-20260713T144500Z-failures.csv
• S3 format CSV: s3://lance-bench-054483968661-us-east-2/xuanwo-sparse-pr5-20260713T133525Z/artifacts/formats-100m-20260713T144500Z.csv
• S3 format files: s3://lance-bench-054483968661-us-east-2/xuanwo-sparse-pr5-20260713T133525Z/formats/100m-20260713T144500Z/
• PR4 baseline CSV: s3://lance-bench-054483968661-us-east-2/xuanwo-sparse-pr5-20260713T133525Z/artifacts/pr4-baseline-deep-fullzip-200k.csv
• PR4 baseline backtrace: s3://lance-bench-054483968661-us-east-2/xuanwo-sparse-pr5-20260713T133525Z/artifacts/pr4-baseline-deep-fullzip-200k.err
• EC2 result directory: /home/ec2-user/sparse-pr5-results/20260713T144500Z/

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 69ce724c-4b14-480c-9e4c-7a6b3ebb39bc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch xuanwo/sparse-stack-5-auto-sparse

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.