Repository navigation
feat(io): a segment-keyed IO protocol for planner-driven scans - #10568
joseph-isaacs wants to merge 1 commit into
Conversation
Merging this PR will degrade performance by 9.95%4 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Polar Signals Profiling ResultsLatest Run
Previous Runs (1)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.037x ➖, 0↑ 2↓)
datafusion / vortex-compact / ns (1.003x ➖, 1↑ 2↓)
datafusion / parquet / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.947x ➖, 4↑ 2↓)
duckdb / vortex-compact / ns (0.994x ➖, 2↑ 1↓)
duckdb / parquet / ns (0.989x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (1.014x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: String Encoding 📖Commits: PR vortex / vortex-file-compressed / ms (1.003x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / % (1.000x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.984x ➖, 1↑ 1↓)
duckdb / vortex-compact / ns (0.997x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.967x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.020x ➖, 0↑ 0↓)
File Size Changes (200 files changed, +0.0% overall, 113↑ 87↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 1↑ 2↓)
duckdb / vortex-compact / ns (0.989x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.991x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: JOB on NVME 📖Commits: PR No baseline is available for this benchmark yet; PR measurements are shown without comparison. duckdb / vortex-file-compressed / ns (no group data, 0↑ 0↓)
duckdb / parquet / ns (no group data, 0↑ 0↓)
duckdb / duckdb / ns (no group data, 0↑ 0↓)
No baseline file sizes found for base commit. |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.268x ➖, 1↑ 5↓)
datafusion / vortex-compact / ns (1.619x ❌, 0↑ 7↓)
datafusion / parquet / ns (1.250x ➖, 0↑ 4↓)
duckdb / vortex-file-compressed / ns (1.352x ❌, 0↑ 6↓)
duckdb / vortex-compact / ns (1.289x ➖, 0↑ 4↓)
duckdb / parquet / ns (1.145x ➖, 0↑ 2↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.999x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 3↑ 3↓)
duckdb / vortex-compact / ns (0.996x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.005x ➖, 2↑ 4↓)
duckdb / duckdb / ns (1.000x ➖, 3↑ 2↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.988x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.011x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.987x ➖, 1↑ 2↓)
duckdb / vortex-compact / ns (0.991x ➖, 4↑ 1↓)
duckdb / parquet / ns (0.993x ➖, 1↑ 1↓)
duckdb / duckdb / ns (0.996x ➖, 2↑ 1↓)
No file size changes detected. |
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.007x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (0.991x ➖, 1↑ 0↓)
vortex / parquet / ns (1.012x ➖, 0↑ 1↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.980x ➖, 3↑ 0↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.985x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.052x ➖, 0↑ 2↓)
datafusion / vortex-compact / ns (0.933x ➖, 2↑ 1↓)
datafusion / parquet / ns (1.100x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (0.951x ➖, 3↑ 1↓)
duckdb / vortex-compact / ns (0.922x ➖, 2↑ 1↓)
duckdb / parquet / ns (0.985x ➖, 0↑ 2↓)
|
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.015x ➖, 1↑ 2↓)
datafusion / vortex-compact / ns (1.189x ➖, 1↑ 8↓)
datafusion / parquet / ns (1.102x ➖, 0↑ 3↓)
duckdb / vortex-file-compressed / ns (0.797x ➖, 4↑ 0↓)
duckdb / vortex-compact / ns (0.886x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
Benchmarks: Random Access (S3) 📖Commits: PR How to read Verdict and Engines
random-access / vortex-file-compressed / ns (0.935x ➖, 1↑ 0↓)
random-access / parquet / ns (0.979x ➖, 0↑ 0↓)
random-access / lance / ns (1.091x ➖, 0↑ 2↓)
|
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
vortex / arrow-ipc / ns (0.978x ➖, 3↑ 3↓)
random-access / vortex-file-compressed / ns (1.003x ➖, 1↑ 1↓)
random-access / parquet / ns (1.004x ➖, 0↑ 0↓)
random-access / lance / ns (1.002x ➖, 0↑ 0↓)
|
Adds `vortex_io::segments`, keyed by segment source and source request, next
to the byte-range `vortex_io::request` protocol that the planner-driven scan
still uses.
- A request names `IoKey { source: SegmentSourceId, request: SourceRequest }`,
where `SourceRequest ::= Size | Segment(id)`, with an `IoIntent`: `Fetch`,
`Prefetch` or `Announce`. One service serves many segment sources, and reads
resolve to a `SourceResult` (`Size(u64)` or `Bytes(BufferHandle)`).
- `SegmentIoSession::submit` registers a whole batch before any read starts and
returns one `IoHandle` per request. The session holds fetches and prefetches
until they finish; an announcement is withdrawn by dropping its handles.
- `poll_ready` reports every fetch's owner once it finishes and drives the
session's prefetches, through the shared `FetchTracker` helper.
`SegmentIoService::session` opens one session per scope, and
`RegisteredSource` is a source's registration with a service.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9
Signed-off-by: Claude <noreply@anthropic.com>
24ef18f to
8e2defb
Compare
|
CodSpeed Performance Analysis reports a −9.95% regression on
No fix is needed, and I can't re-run CodSpeed from here. Generated by Claude Code |
Summary
Adds
vortex_io::segments, an IO protocol for planner-driven scans that is keyed by segment source and source request. It sits next to the byte-rangevortex_io::requestprotocol, which the V2 executor ondevelopstill uses. This PR is additive: it changes nothing else, so it compiles ondevelopby itself.#10527 is stacked on this PR. It switches the planning traits and the V2 executor to this protocol, adds
FileSegmentIoto serve it, and removesrequest.Changes
IoKey { source: SegmentSourceId, request: SourceRequest }, whereSourceRequest ::= Size | Segment(id). It also carries anIoIntent:Fetch,PrefetchorAnnounce. One service can serve many segment sources, and a source can answer more than segment reads.SourceResult: eitherSize(u64)orBytes(BufferHandle). The checked accessors aresize()andinto_bytes().SegmentIoSession::submitregisters a whole batch before any read starts, and returns oneIoHandleper request.SegmentIoSession::poll_readyreports each fetch's owner once the fetch finishes, including fetches that finished at submit, and drives the session's prefetches.FetchTrackerimplements this for sessions.SegmentIoService::session()opens one session per scope.RegisteredSourceis a source's registration with a service.Tests cover handles, results checked against what was asked, and
FetchTracker.Checks run:
cargo clippy -p vortex-io --all-targets --all-features -- -D warningscargo +nightly-2026-09-10 fmt -p vortex-ioRUSTDOCFLAGS="-D warnings --cfg docsrs" cargo doc --no-deps -p vortex-io --all-featurescargo test -p vortex-io --lib segments(4 tests)typosAPI Changes
New:
vortex_io::segments.🤖 Generated with Claude Code
https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9