Repository navigation
Conversation
On tables with many columns and many data files per fragment, scans with narrow projections spent most of their CPU cloning schemas rather than reading data. * Computing a data file's per-file schema built the file's full schema (a deep clone of every field it stores, with a linear id lookup per field) only to intersect it with the projection. It is now built from the top-level fields the projection can match, which gives the same result. This is the miss path of the per-scan open cache, and the whole path for opens without one. The full schema the metadata index needs is built only on a cache miss, so FileOpenState no longer holds it. * FileReadCore built a whole-file base projection eagerly whenever none was passed. It is now built on first use. * Scanner::create_filter_plan rebuilt the dataset's filterable schema and converted it to Arrow on every scan. Both forms are now cached in the dataset metadata cache per manifest location (store, version and e-tag) and per scanner options that add columns, and an entry is only served to a dataset holding the manifest it was built from. Manifests without an e-tag are not cached. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related
Problem
On tables with many columns and many data files per fragment, filtered scans with narrow projections spend most of their CPU cloning schemas rather than reading data.
Profiled on a Rerun Hub querier at 1 CU (wide chunks table, about 300 columns, 1,051 data files over 105 fragments), schema projection was about 45% of querier CPU and column decode about 0.3%.
The biggest call sites were the per-data-file schema in
open_reader_impl(6.6–8.3%),Scanner::filterable_schema(6.6–7.6%) andReaderProjection::from_whole_schema(3.8–3.9%).#94 reuses the per-file state across the fragments of one scan, but each scan still builds it the old way once per layout, and the other two costs are untouched.
Solution
data_file_projectionbuilds a data file's schema only from the dataset's top-level fields whose names resolve in the projection, then intersects as before.The intersection keeps a top-level field only when its name resolves in the projection, and projecting by ids treats top-level fields independently, so the result is identical; a test checks it against the old computation across flat, List, Struct, quoted-name, dropped-id and empty cases.
It replaces the old computation in perf(scan): memoize per-scan reader-open state across fragments #94's cache-miss path and for opens without an open cache.
FileOpenStateno longer holds the full data-file schema; the metadata index builds it only on a metadata-cache miss.FileReadCorebuilds its whole-file base projection on first use instead of eagerly.create_filter_plancaches the Lance and Arrow filterable schemas in the dataset metadata cache, keyed by store, manifest version, manifest e-tag and the scanner options that add columns.An entry is only served to a dataset holding the exact manifest it was built from (a
Weak<Manifest>identity check), so a manifest swapped into aDatasetwithout changing its location is rebuilt; manifests without an e-tag are not cached.Results
On
disposable-query-perf(1 CU, abc-130k), the release-10.0.0 version of this change plus a separate redap fix (a per-request planning-hint scan) raised querier throughput by 74% on a random-window read workload and 70% on a training dataloader, with the querier still CPU-bound.The two were measured together; this PR alone has not been measured yet.
Testing
lancelib suite (2,694 passed on the release-10.0.0 version), and on this branch:dataset::fragment(107, including perf(scan): memoize per-scan reader-open state across fragments #94's open-cache tests),dataset::scanner(190),io::exec::filtered_read(63),dataset::schema_evolution(48),lance-file(136); clippy-D warningsand fmt clean.test_data_file_projection_matches_full_intersection,test_filterable_schema_follows_dataset_version(version separation, cache hit, manifest swap; the swap case fails with the identity check disabled).Agent
🤖 This PR was opened by a coding agent (Claude Opus 5.5).
🤖 Generated with Claude Code