feat(encoding): write sparse structural pages explicitly

[Xuanwo]

Part of #7750

Depends on #7754.

Summary

• Add opt-in Lance 2.3 sparse structural page emission for fields with lance-encoding:structural-encoding=sparse, and reject the request for older file versions.
• Normalize Arrow structural layers once and share that semantic plan between dense rep/def serialization and sparse position/count serialization.
• Emit the PR3 sparse wire forms, choose validity polarity by semantic encoded cost, and mini-block compress leaf values.
• Keep structural-only, all-null, and empty-struct pages on the existing ConstantLayout boundary; explicit non-null constants emit sparse pages.
• Deliberately leave automatic sparse selection for PR5. Default Lance 2.3 selection and explicit miniblock / fullzip behavior are unchanged.

Coverage

Round trips cover nullable primitive and struct values, list, large-list, map, fixed-size-list, nested combinations, null versus empty lists, both validity polarities, every semantic position/count form, structural-only and all-null pages, page boundaries, scans, ranges, and discontiguous takes. Metadata assertions verify the selected page and set representations. Lance 2.2 default output is compared with explicit mini-block output, and explicit sparse requests are rejected before encoding.

Validation

• cargo fmt --all -- --check
• cargo test -p lance-encoding sparse:: --lib (25 passed)
• cargo test -p lance-encoding (455 passed, 5 ignored)
• cargo test -p lance-file (92 passed)
• cargo clippy --all --tests --benches -- -D warnings
• uv run mkdocs build from docs/ (passed with existing repository link warnings)

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: afdee0c0-643e-40d3-bc7c-49ccbd43c160

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch xuanwo/sparse-stack-4-sparse-writer

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.