bench: V2 scan decompression on the segment IO protocol by joseph-isaacs · Pull Request #10569 · vortex-data/vortex

joseph-isaacs · 2026-10-16T16:47:44Z

Summary

Benchmark only, do not merge. This PR runs the compression benchmark on the stack of #10568 (the segment IO protocol) and #10527 (the V2 executor on that protocol, served by FileSegmentIo). Its branch is the same tree as #10527, and it targets develop so the label-triggered benchmark runs.

The comparison is V2 decompression on the new segment IO against develop's V2 on the byte-range request protocol, so the difference measured is the IO layer.

See #10527 for the changes and the checks run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9

codspeed · 2026-10-16T17:53:44Z

Merging this PR will degrade performance by 25.8%

4 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ Unable to measure the execution of 1 benchmark

This benchmark measured no execution time, so its result is not comparable and counts as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 23 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 40 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

⚡ 2 improved benchmarks
❌ 7 regressed benchmarks
✅ 2399 untouched benchmarks
⏩ 12 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Walltime	`filter_neon[8]`	893 ns	1,900 ns	-53%
❌	Walltime	`threshold_neon[8]`	1.4 µs	2.5 µs	-45.09%
❌	Walltime	`filter_neon[64]`	1.7 µs	2.8 µs	-37.76%
❌	Walltime	`threshold_neon[80]`	3.5 µs	4.9 µs	-28.3%
❌	Walltime	`threshold_neon[64]`	2.5 µs	3.5 µs	-27.86%
❌	CPU Simulation	`chunked_dict_primitive_canonical_into[f64, (1000, 1000, 10)]`	151.9 µs	205.5 µs	-26.08%
❌	Walltime	`filter_neon[80]`	2.6 µs	3.4 µs	-24.99%
⚡	CPU Simulation	`test_repeated_scan_scalar_at[int]`	318.8 µs	238.7 µs	+33.54%
⚡	CPU Simulation	`prepare_and_compile[1000]`	2.4 ms	2.2 ms	+10.78%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing ji/v2-scan-segment-io (a5654f2) with develop (91b38f4)}

12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

joseph-isaacs · 2026-10-16T17:54:07Z

CodSpeed Performance Analysis flags one regression: vortex-python test_scan_filter[int], 3.1 ms → 3.4 ms (−10.94%). I don't think this PR causes it:

The path: that benchmark runs a filtered scan, which goes through the V1 executor (the V2 builder doesn't take filters). This PR changes only V2 scan code, its IO protocol and FileSegmentIo.
History: the same family of Python scan benchmarks reported −11% to −13% on feat: switch vortex compress to use v2 scan api #10540.
Measurement: CodSpeed itself flags these runs for heavy system-call time and different runtime environments.

No fix is needed, and CodSpeed can't be re-run from here. The result this PR exists for is the compression benchmark (action/bench-compress).

Generated by Claude Code

github-actions · 2026-10-16T17:57:30Z

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	`a5654f2`	compress-bench	1	Explore Profiling Data

Previous Runs (1)

Status	Commit	Job	Attempt	Link
🟢 Done	`32b5cd1`	compress-bench	1	Explore Profiling Data

Powered by Polar Signals Cloud

github-actions · 2026-10-16T18:04:43Z

Benchmarks: Compression 📖

Commits: PR a5654f20 vs base 91b38f41

vortex / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
compress time/Arade	739949833 / 756637822 / -2.2%
compress time/Bimbo	3216593276 / 3274827395 / -1.8%
compress time/CMSprovider	2351526900 / 2338035782 / +0.6%
compress time/Euro2016	250433731 / 250180716 / +0.1%
compress time/Food	302916754 / 309497435 / -2.1%
compress time/HashTags	493856516 / 495021006 / -0.2%
compress time/TPC-H l_comment canonical	757298742 / 756266299 / +0.1%
compress time/TPC-H l_comment chunked	758651612 / 763797894 / -0.7%
compress time/taxi	355590909 / 355999278 / -0.1%
compress time/wide table cols=100 chunks=1 rows=1000	4059545 / 4215369 / -3.7%
compress time/wide table cols=100 chunks=50 rows=1000	4054711 / 4185418 / -3.1%
compress time/wide table cols=1000 chunks=1 rows=1000	40867895 / 40598383 / +0.7%
compress time/wide table cols=1000 chunks=50 rows=1000	40737586 / 41421125 / -1.7%
compress time/wide table cols=10000 chunks=1 rows=1000	461526079 / 468354021 / -1.5%
compress time/wide table cols=10000 chunks=50 rows=1000	460781232 / 471332697 / -2.2%
decompress time/Arade	12379355 / 12214218 / +1.4%
decompress time/Bimbo	56533281 / 55042771 / +2.7%
decompress time/CMSprovider	36312631 / 36341694 / -0.1%
decompress time/Euro2016	11713659 / 11840857 / -1.1%
decompress time/Food	3325610 / 3213922 / +3.5%
decompress time/HashTags	34557191 / 34866383 / -0.9%
decompress time/TPC-H l_comment canonical	13313196 / 13422936 / -0.8%
decompress time/TPC-H l_comment chunked	13607068 / 13420327 / +1.4%
decompress time/taxi	8799763 / 9040092 / -2.7%
decompress time/wide table cols=100 chunks=1 rows=1000	1213909 / 1166301 / +4.1%
decompress time/wide table cols=100 chunks=50 rows=1000	1213061 / 1165540 / +4.1%
decompress time/wide table cols=1000 chunks=1 rows=1000	11962586 / 11300775 / +5.9%
decompress time/wide table cols=1000 chunks=50 rows=1000	11977200 / 11312585 / +5.9%
decompress time/wide table cols=10000 chunks=1 rows=1000	130660478 / 124212055 / +5.2%
decompress time/wide table cols=10000 chunks=50 rows=1000	131105249 / 125864983 / +4.2%

vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
vortex-file-compressed size/Arade	142213756 / 142213756 / +0.0%
vortex-file-compressed size/Bimbo	458581212 / 458581212 / +0.0%
vortex-file-compressed size/CMSprovider	420794836 / 420794836 / +0.0%
vortex-file-compressed size/Euro2016	164130260 / 164130260 / +0.0%
vortex-file-compressed size/Food	38950752 / 38950752 / +0.0%
vortex-file-compressed size/HashTags	196170516 / 196170516 / +0.0%
vortex-file-compressed size/TPC-H l_comment canonical	178688232 / 178688232 / +0.0%
vortex-file-compressed size/TPC-H l_comment chunked	178688232 / 178688232 / +0.0%
vortex-file-compressed size/taxi	49998092 / 49998092 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000	934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000	934080 / 934080 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000	9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000	9325680 / 9325680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000	93277680 / 93277680 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000	93277680 / 93277680 / +0.0%

vortex / vortex-file-compressed / ratio (1.006x ➖, 0↑ 0↓)

name	ratio (PR / base / %diff)
vortex:parquet-zstd ratio compress time/Arade	1.01549263 / 1.03122601 / -1.5%
vortex:parquet-zstd ratio compress time/Bimbo	1.65520742 / 1.64931542 / +0.4%
vortex:parquet-zstd ratio compress time/CMSprovider	2.23559775 / 2.20753639 / +1.3%
vortex:parquet-zstd ratio compress time/Euro2016	0.39108522 / 0.389432197 / +0.4%
vortex:parquet-zstd ratio compress time/Food	0.959352701 / 0.971556831 / -1.3%
vortex:parquet-zstd ratio compress time/HashTags	1.44633524 / 1.45564236 / -0.6%
vortex:parquet-zstd ratio compress time/TPC-H l_comment canonical	0.72267192 / 0.717267119 / +0.8%
vortex:parquet-zstd ratio compress time/TPC-H l_comment chunked	0.72382256 / 0.72438371 / -0.1%
vortex:parquet-zstd ratio compress time/taxi	1.85782201 / 1.84153258 / +0.9%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=1 rows=1000	1.09630173 / 1.125002 / -2.6%
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=50 rows=1000	1.18113665 / 1.2509233 / -5.6%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=1 rows=1000	1.11334389 / 1.13558216 / -2.0%
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=50 rows=1000	1.14337414 / 1.16035717 / -1.5%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=1 rows=1000	1.22145112 / 1.26636542 / -3.5%
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=50 rows=1000	1.23518947 / 1.2465682 / -0.9%
vortex:parquet-zstd ratio decompress time/Arade	0.25261044 / 0.249395583 / +1.3%
vortex:parquet-zstd ratio decompress time/Bimbo	0.633654965 / 0.618233363 / +2.5%
vortex:parquet-zstd ratio decompress time/CMSprovider	0.354417638 / 0.352918696 / +0.4%
vortex:parquet-zstd ratio decompress time/Euro2016	0.0781926782 / 0.079045465 / -1.1%
vortex:parquet-zstd ratio decompress time/Food	0.133359335 / 0.128548632 / +3.7%
vortex:parquet-zstd ratio decompress time/HashTags	0.0827434807 / 0.0832342668 / -0.6%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment canonical	0.161851384 / 0.163227678 / -0.8%
vortex:parquet-zstd ratio decompress time/TPC-H l_comment chunked	0.165856958 / 0.163869914 / +1.2%
vortex:parquet-zstd ratio decompress time/taxi	0.150677594 / 0.154107271 / -2.2%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=1 rows=1000	0.596739806 / 0.564470205 / +5.7%
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=50 rows=1000	0.601913224 / 0.573395274 / +5.0%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=1 rows=1000	0.509318579 / 0.479718164 / +6.2%
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=50 rows=1000	0.514862124 / 0.477428148 / +7.8%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=1 rows=1000	0.521704168 / 0.485947139 / +7.4%
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=50 rows=1000	0.530505897 / 0.49936346 / +6.2%
vortex:parquet-zstd size/Arade	0.551185597 / 0.551185597 / -0.0%
vortex:parquet-zstd size/Bimbo	1.19261532 / 1.19261532 / +0.0%
vortex:parquet-zstd size/CMSprovider	1.11649863 / 1.11649863 / -0.0%
vortex:parquet-zstd size/Euro2016	1.33461969 / 1.33461969 / +0.0%
vortex:parquet-zstd size/Food	1.09107276 / 1.09107276 / -0.0%
vortex:parquet-zstd size/HashTags	1.46946859 / 1.46946859 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment canonical	1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/TPC-H l_comment chunked	1.12837977 / 1.12837977 / -0.0%
vortex:parquet-zstd size/taxi	0.904392267 / 0.904392267 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=1 rows=1000	1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=100 chunks=50 rows=1000	1.0017975 / 1.0017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=1 rows=1000	1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=1000 chunks=50 rows=1000	1.00017975 / 1.00017975 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=1 rows=1000	1.00040408 / 1.00040408 / -0.0%
vortex:parquet-zstd size/wide table cols=10000 chunks=50 rows=1000	1.00040408 / 1.00040408 / -0.0%

vortex / parquet / ns (0.996x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
parquet_rs-zstd compress time/Arade	728660955 / 733726472 / -0.7%
parquet_rs-zstd compress time/Bimbo	1943317342 / 1985567685 / -2.1%
parquet_rs-zstd compress time/CMSprovider	1051855995 / 1059115399 / -0.7%
parquet_rs-zstd compress time/Euro2016	640355908 / 642424324 / -0.3%
parquet_rs-zstd compress time/Food	315751187 / 318558241 / -0.9%
parquet_rs-zstd compress time/HashTags	341453697 / 340070486 / +0.4%
parquet_rs-zstd compress time/TPC-H l_comment canonical	1047914996 / 1054371905 / -0.6%
parquet_rs-zstd compress time/TPC-H l_comment chunked	1048118219 / 1054410644 / -0.6%
parquet_rs-zstd compress time/taxi	191402033 / 193316850 / -1.0%
parquet_rs-zstd compress time/wide table cols=100 chunks=1 rows=1000	3702945 / 3746988 / -1.2%
parquet_rs-zstd compress time/wide table cols=100 chunks=50 rows=1000	3432889 / 3345863 / +2.6%
parquet_rs-zstd compress time/wide table cols=1000 chunks=1 rows=1000	36707342 / 35751163 / +2.7%
parquet_rs-zstd compress time/wide table cols=1000 chunks=50 rows=1000	35629270 / 35696875 / -0.2%
parquet_rs-zstd compress time/wide table cols=10000 chunks=1 rows=1000	377850632 / 369841131 / +2.2%
parquet_rs-zstd compress time/wide table cols=10000 chunks=50 rows=1000	373044982 / 378104219 / -1.3%
parquet_rs-zstd decompress time/Arade	49005714 / 48975278 / +0.1%
parquet_rs-zstd decompress time/Bimbo	89217767 / 89032353 / +0.2%
parquet_rs-zstd decompress time/CMSprovider	102457178 / 102974692 / -0.5%
parquet_rs-zstd decompress time/Euro2016	149805062 / 149798056 / +0.0%
parquet_rs-zstd decompress time/Food	24937212 / 25001604 / -0.3%
parquet_rs-zstd decompress time/HashTags	417642462 / 418894577 / -0.3%
parquet_rs-zstd decompress time/TPC-H l_comment canonical	82255682 / 82234436 / +0.0%
parquet_rs-zstd decompress time/TPC-H l_comment chunked	82040984 / 81896223 / +0.2%
parquet_rs-zstd decompress time/taxi	58401271 / 58661035 / -0.4%
parquet_rs-zstd decompress time/wide table cols=100 chunks=1 rows=1000	2034235 / 2066187 / -1.5%
parquet_rs-zstd decompress time/wide table cols=100 chunks=50 rows=1000	2015342 / 2032699 / -0.9%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=1 rows=1000	23487433 / 23557113 / -0.3%
parquet_rs-zstd decompress time/wide table cols=1000 chunks=50 rows=1000	23262927 / 23694843 / -1.8%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=1 rows=1000	250449366 / 255608162 / -2.0%
parquet_rs-zstd decompress time/wide table cols=10000 chunks=50 rows=1000	247132501 / 252050847 / -2.0%

vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
parquet size/Arade	258014282 / 258014282 / +0.0%
parquet size/Bimbo	384517292 / 384517292 / +0.0%
parquet size/CMSprovider	376887911 / 376887911 / +0.0%
parquet size/Euro2016	122979049 / 122979049 / +0.0%
parquet size/Food	35699500 / 35699500 / +0.0%
parquet size/HashTags	133497590 / 133497590 / +0.0%
parquet size/TPC-H l_comment canonical	158358238 / 158358238 / +0.0%
parquet size/TPC-H l_comment chunked	158358238 / 158358238 / +0.0%
parquet size/taxi	55283635 / 55283635 / +0.0%
parquet size/wide table cols=100 chunks=1 rows=1000	932404 / 932404 / +0.0%
parquet size/wide table cols=100 chunks=50 rows=1000	932404 / 932404 / +0.0%
parquet size/wide table cols=1000 chunks=1 rows=1000	9324004 / 9324004 / +0.0%
parquet size/wide table cols=1000 chunks=50 rows=1000	9324004 / 9324004 / +0.0%
parquet size/wide table cols=10000 chunks=1 rows=1000	93240004 / 93240004 / +0.0%
parquet size/wide table cols=10000 chunks=50 rows=1000	93240004 / 93240004 / +0.0%

vortex / arrow-ipc / ns (0.993x ➖, 0↑ 0↓)

name	ns (PR / base / %diff)
arrow-ipc compress time/Arade	98690444 / 100721893 / -2.0%
arrow-ipc compress time/Bimbo	614961504 / 629401904 / -2.3%
arrow-ipc compress time/CMSprovider	382460919 / 390467991 / -2.1%
arrow-ipc compress time/Euro2016	37970836 / 38622446 / -1.7%
arrow-ipc compress time/Food	33704728 / 33934752 / -0.7%
arrow-ipc compress time/HashTags	78389785 / 79281560 / -1.1%
arrow-ipc compress time/TPC-H l_comment canonical	129850786 / 131057139 / -0.9%
arrow-ipc compress time/TPC-H l_comment chunked	129979691 / 129646509 / +0.3%
arrow-ipc compress time/taxi	74599830 / 75548622 / -1.3%
arrow-ipc compress time/wide table cols=100 chunks=1 rows=1000	198645 / 204829 / -3.0%
arrow-ipc compress time/wide table cols=100 chunks=50 rows=1000	199776 / 202884 / -1.5%
arrow-ipc compress time/wide table cols=1000 chunks=1 rows=1000	2252143 / 2280848 / -1.3%
arrow-ipc compress time/wide table cols=1000 chunks=50 rows=1000	2229072 / 2242352 / -0.6%
arrow-ipc compress time/wide table cols=10000 chunks=1 rows=1000	25293655 / 25630240 / -1.3%
arrow-ipc compress time/wide table cols=10000 chunks=50 rows=1000	25504317 / 25450104 / +0.2%
arrow-ipc decompress time/Arade	218205907 / 212707220 / +2.6%
arrow-ipc decompress time/Bimbo	407764029 / 423042577 / -3.6%
arrow-ipc decompress time/CMSprovider	676835501 / 698834348 / -3.1%
arrow-ipc decompress time/Euro2016	83752082 / 84200642 / -0.5%
arrow-ipc decompress time/Food	67648344 / 66291689 / +2.0%
arrow-ipc decompress time/HashTags	160844140 / 161855138 / -0.6%
arrow-ipc decompress time/TPC-H l_comment canonical	191679569 / 190701095 / +0.5%
arrow-ipc decompress time/TPC-H l_comment chunked	191652255 / 190680900 / +0.5%
arrow-ipc decompress time/taxi	51433183 / 51378109 / +0.1%
arrow-ipc decompress time/wide table cols=100 chunks=1 rows=1000	315121 / 319709 / -1.4%
arrow-ipc decompress time/wide table cols=100 chunks=50 rows=1000	316454 / 325302 / -2.7%
arrow-ipc decompress time/wide table cols=1000 chunks=1 rows=1000	3729506 / 3610497 / +3.3%
arrow-ipc decompress time/wide table cols=1000 chunks=50 rows=1000	3567778 / 3629275 / -1.7%
arrow-ipc decompress time/wide table cols=10000 chunks=1 rows=1000	35607068 / 35393034 / +0.6%
arrow-ipc decompress time/wide table cols=10000 chunks=50 rows=1000	36650049 / 36157671 / +1.4%

vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)

name	bytes (PR / base / %diff)
arrow-ipc size/Arade	691765162 / 691765162 / +0.0%
arrow-ipc size/Bimbo	3279590114 / 3279590114 / +0.0%
arrow-ipc size/CMSprovider	2422259682 / 2422259682 / +0.0%
arrow-ipc size/Euro2016	360582666 / 360582666 / +0.0%
arrow-ipc size/Food	255782338 / 255782338 / +0.0%
arrow-ipc size/HashTags	704785810 / 704785810 / +0.0%
arrow-ipc size/TPC-H l_comment canonical	1033075618 / 1033075618 / +0.0%
arrow-ipc size/TPC-H l_comment chunked	1033075618 / 1033075618 / +0.0%
arrow-ipc size/taxi	482329690 / 482329690 / +0.0%
arrow-ipc size/wide table cols=100 chunks=1 rows=1000	1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=100 chunks=50 rows=1000	1257290 / 1257290 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=1 rows=1000	12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=50 rows=1000	12568506 / 12568506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=1 rows=1000	125752506 / 125752506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=50 rows=1000	125752506 / 125752506 / +0.0%

Adds `vortex_io::segments`, keyed by segment source and source request, next to the byte-range `vortex_io::request` protocol that the planner-driven scan still uses. - A request names `IoKey { source: SegmentSourceId, request: SourceRequest }`, where `SourceRequest ::= Size | Segment(id)`, with an `IoIntent`: `Fetch`, `Prefetch` or `Announce`. One service serves many segment sources, and reads resolve to a `SourceResult` (`Size(u64)` or `Bytes(BufferHandle)`). - `SegmentIoSession::submit` registers a whole batch before any read starts and returns one `IoHandle` per request. The session holds fetches and prefetches until they finish; an announcement is withdrawn by dropping its handles. - `poll_ready` reports every fetch's owner once it finishes and drives the session's prefetches, through the shared `FetchTracker` helper. `SegmentIoService::session` opens one session per scope, and `RegisteredSource` is a source's registration with a service. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

Switches the planner-driven scan from the byte-range `vortex_io::request` protocol to `vortex_io::segments`, and serves it with a session-wide `FileSegmentIo` that reads through files' v1 segment sources. - vortex-scan: `Planner` and `Morsel` drop the `IoConsumer` supertrait, publish `NeedsIO(Vec<IoRequest>)`, and receive `deliver(key, result)` once per successful fetch. `Run`/`Driver` submit each root's requests to its `SegmentIoSession` and hold the handles; each item's fetches finish through its own `FuturesUnordered`, so a `poll_ready` report yields exactly the finished fetches. `Run::poll_io` replaces `Run::complete`. The scripted harness is a recording `SegmentIoSession`. - vortex-layout: `ScanAdaptor` fetches each segment once by `IoKey::segment` while any scan read of it is outstanding, and hands the result to every waiting read. The filter planner announces each segment once. `ScanFile` carries the file's `RegisteredSource`, and each split reads through a new session of its service. `SegmentScanIo` and the cached byte-range index it needed are removed. - vortex-file: `FileSegmentIo`, one per `VortexSession`. Size requests are answered at submit; segment requests call `SegmentSource::request`, so they share the file's segment cache, request sharing and coalescing read driver. `VortexFile::scan_file` registers the file's segment source, and a caching file keeps one registration in its scan cache, renewed with its segment source. - vortex-datafusion: the V2 opener shares a file's segment source across partitions through `VortexFile::with_segment_source`. - vortex-io: `request` is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9 Signed-off-by: Claude <noreply@anthropic.com>

joseph-isaacs added changelog/skip Do not list PR in the changelog action/bench-compress Run only the compression benchmark on this PR and removed action/bench-compress Run only the compression benchmark on this PR labels Oct 16, 2026 — with Claude

github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Oct 16, 2026

claude added 2 commits October 23, 2026 23:34

joseph-isaacs force-pushed the ji/v2-scan-segment-io branch from d3f81d8 to a5654f2 Compare October 23, 2026 23:56

joseph-isaacs added the action/bench-compress Run only the compression benchmark on this PR label Oct 23, 2026 — with Claude

github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Oct 23, 2026

joseph-isaacs mentioned this pull request Oct 24, 2026

feat(scan): run the V2 executor on the segment IO protocol #10527

Draft

Navigation Menu

Repository navigation

bench: V2 scan decompression on the segment IO protocol - #10569

bench: V2 scan decompression on the segment IO protocol#10569
joseph-isaacs wants to merge 2 commits into
developfrom
ji/v2-scan-segment-io

joseph-isaacs commented Oct 16, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Oct 16, 2026 •

edited

Loading

Uh oh!

joseph-isaacs commented Oct 16, 2026

Uh oh!

github-actions Bot commented Oct 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 16, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

joseph-isaacs commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

codspeed Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 25.8%

Performance Changes

Footnotes

Uh oh!

joseph-isaacs commented Oct 16, 2026

Uh oh!

github-actions Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Polar Signals Profiling Results

Latest Run

Uh oh!

github-actions Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks: Compression 📖

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joseph-isaacs commented Oct 16, 2026 •

edited

Loading

codspeed Bot commented Oct 16, 2026 •

edited

Loading

github-actions Bot commented Oct 16, 2026 •

edited

Loading

github-actions Bot commented Oct 16, 2026 •

edited

Loading