[zehiko]

Related

• Stacked on perf(scan): backport #10042 and #9614 (per-fragment scan overhead) #93; review the last commit only.
• Not upstream. The closest upstream work, perf: cache per-fragment reader templates lance-format/lance#6644, takes a different approach and is stalled.

Problem

A filtered scan opens every fragment it can't rule out with an index, and each open repeats the same setup.
For every data file, it projects the full table schema down to the file's fields (data_file.schema), intersects that with the read projection, builds a field-id → column map, and derives a ReaderProjection.
read_fragment also rebuilds the output and read schemas for every fragment.

None of this depends on the fragment.
It depends only on the table schema, the read projection and the file's layout (field ids, column indices, format version), and fragments written the same way share a layout.
On tables with many small unindexed fragments, this per-fragment setup costs far more than reading the data: about 0.35 ms of CPU per fragment even after #93.

Solution

Compute that state once per layout per scan, and reuse it for every fragment with the same layout.

• plan_to_scoped_fragments builds the read schemas once per distinct read projection and shares them across fragments.
• A FragmentOpenCache, carried in FragReadConfig.open_cache, memoizes the per-file state for each data-file layout.
  • It lives for one scan and holds at most 16 layouts; any other layout is computed as before.
  • Only scan opens set it, the same way upstream's BaseSchedulers carries a per-scan cache in FragReadConfig.
• Scan opens pass that reader projection to the full-metadata open instead of building a whole-file projection they never use; other opens are unchanged.

Wins

CPU per filtered scan on top of #93 (4-column projection, release build, 6 cores):

fragments	with #93	+ this PR
8	5.35 ms	4.50 ms (−16%)
32	16.7 ms	12.9 ms (−23%)
128	61.9 ms	46.1 ms (−26%)

With #93 that's −35% to −43%.
On a dataplatform stack, incremental registration used 19% less writer CPU per segment.

Safety

• An entry is reused only for the same dataset Arc, the same projection Arc, and a data file with equal field ids, column indices and format version. Those are the only inputs to the cached values. The cache holds both Arcs, so a different object can never be mistaken for them.
• The cache is never shared across scans and holds no readers or schedulers.
• Concurrent misses on one layout keep a single entry.
• Independently reviewed twice for correctness and safety.

Testing

• lance --lib 2699 passed; lance-file 141 passed; lance-datafusion 85 passed. Clippy (-D warnings) and rustfmt are clean.
• New tests:
  • cached state equals uncached state, and concurrent inserts keep one entry;
  • overflow past 16 layouts;
  • other projections and datasets never hit the cache;
  • parallel fragment opens;
  • schema evolution: fragments with two data files, a dropped column's field ids in old files, and appended single-file fragments.

Compatibility

• No format change.
• New public field FragReadConfig::open_cache, defaulting to None. Callers that build FragReadConfig with struct-literal syntax need to add it.

Agent

🤖 This PR was opened by a coding agent (Claude Opus 5.5).

[Copilot]

🟢 Approval recommended

No blocking issues were identified in cache isolation, concurrent insertion, schema reuse, or projection handling.

No open issues 🎉

What changed in this PR

Reduces repeated setup when filtered scans open many fragments, building on the shared filter planning in #93. The optimization reuses scan-local schemas and reader-open state without changing the file format.

Changes:

• Shares derived read schemas across fragments with the same read projection.
• Caches reader-open state for up to 16 layouts, guarded by dataset and projection identity.
• Avoids unused whole-file projection construction and adds cache, concurrency, and schema-evolution tests.

File	Description
rust/​lance/​src/​io/​exec/​filtered_read.rs	Shares schemas and caches across scan fragments; adds integration coverage.
rust/​lance/​src/​dataset/​fragment.rs	Adds the bounded layout cache and optional FragReadConfig::open_cache field.
rust/​lance-file/​src/​reader.rs	Constructs the whole-file projection only when needed.

_{Review effort: Balanced}