[xudong963]

• Closes Blog / paper post about parquet push decoder arrow-rs#8035

Publish an accessible blog version of the CIDR 2027 paper Implementing Analytic File Formats Sans I/O. The post explains why separating format decoding from storage I/O gives applications control over scheduling, caching, buffering, and storage-specific optimizations, and connects the design to its production implementation in arrow-rs.

[github-actions]

Preview URL: https://apache.github.io/arrow-site

If the preview URL doesn't work, you may have forgotten to configure your fork repository for preview.
See https://github.com/apache/arrow-site/blob/main/README.md#forks for instructions on how to configure.

[alamb]

As an update here, we submitted a paper based on these ideas to CIDR and it was rejected. Thus I am thinking we should proceed with this writeup (perhaps also addressing some of the comments from the CIDR reviewers).

What do you think @xudong963 ?

[xudong963]

As an update here, we submitted a paper based on these ideas to CIDR and it was rejected. Thus I am thinking we should proceed with this writeup (perhaps also addressing some of the comments from the CIDR reviewers).

What do you think @xudong963 ?

Sounds good, I'll work on this

[xudong963]

I made some changes based on the reviews:

1. Explain Sans I/O earlier. Moved the API contract, decoding loop, and architecture diagram toward the beginning so readers understand the design before the detailed discussion.

2. Clarify the distinction from callbacks and async readers. Acknowledge that caching, retries, read racing, and shared scheduling can already work through callbacks. Explain the additional value of exposing byte dependencies, decoder resumption points, and independently schedulable decoding work.

3. Add concrete motivation. Include Andrew’s description of whole-file buffering at InfluxDB and a production report about memory pressure from wide columns. These illustrate integration challenges without claiming that Sans I/O automatically solves them.

4. Show a concrete scheduling example. Explain how try_next_reader lets an application overlap batch decoding with subsequent reads, including the need to bound concurrency and preserve output ordering.

5. Clarify prefetching limitations. Distinguish immediate NeedsData requirements from future read planning. Explain why selective, data-dependent prefetching may require additional format-aware APIs.

6. Improve positioning and scope. Replace the lengthy reader survey with a focused discussion of existing designs, including networking Sans I/O, streaming parsers, parquet2, Polars, and cuDF. Describe cross-format integration as an architectural direction rather than an existing universal interface.

7. Make the policy examples more useful. Add guidance on when exact-range reads, coalescing, read-ahead, and whole-file caching are appropriate.

[alamb]

Thank you for this post @xudong963

After some time I have had some time to reflect and some observations:

1. I feel like this blog is too long as. Perhaps we should break it up into a few separate posts.
2. The core critique of the reviewes also feels unaddressed -- which is how to show (rather than tell about) the benefits of this design. I think the demo you made is good but we either need to quanitfy the benefits (with numbers that are plausbily connected to a real world usecase)

I feel like we will be in a better position to show the beneifts after we complete the work that @adriangb @HippoBaro and others are working on in:

• apache/arrow-rs#11234 (comment)

How about a high level story like:

1. We made DataFusion (or some other system) 3x faster when reading from Object Storage while also allowing keeping performance for local disk
2. We did this in a disaggregated system (arrow-rs is not part of DataFusion) where previously it requires careful coupling of the engine and the reader (e.g. Lance or DuckDB)
3. The key techniques were the API design between decoder and perfetching
4. Show the key difference (interleaved I/O vs sequential I/O and stalls)

Then we could have two follow on posts with details:

1. The PushDecoder design and contract
2. The ScanPlan API

And each refer back to the overall result

I am not sure this is the best overall story, but I do feel like this post would be much stronger with a more impactful "what did we accomplish"

What do you think?

[alamb]

Alternatively, we could maybe just focus on this post on explaining the API and leave the benefits somewhat vague.

I think we could trim the current content down (for example, we could make the intro more concise, remove the "Where the boundary helps in practice" section (it seems somewhat vague at the moment)

And then we could say "we plan to write more in the future about how we used this API to improve performance..."

Sorry for this back and forth @xudong963

[alamb]

I think we could probably remove this section as I don't think the integration details would be interesting to a broader audience

[xudong963]

Makes sense, removed the integration details.

[alamb]

This would be stronger if we had some specific exmaple of actually using this in practice to improve things or a specific demo with faster performance

Maybe we could make some version of a reader from object store that used prefetching 🤔

[xudong963]

Agree that a performance story needs a concrete demonstration. For this API-focused post, I added a small worked example showing exactly how two logical ranges become one physical read and two shared buffer slices.

For the follow-up, I propose building on DataFusion #24086: compare demand-only reads with budgeted read-ahead on the same Parquet scan, then show the I/O/decode timeline, time to first batch, total scan time, requests, bytes fetched, and peak input buffering. We should include real object storage and a local-disk baseline, and distinguish batch decoding from page release so the results can be attributed to the right changes. That experiment has not been run for this revision; the post now explicitly leaves performance evaluation to the follow-up

[alamb]

This driver is cool -- but now looking back on it, I feel like it mostly "tells" rather than "shows" -- it asserts properties of the various driver policies but doesn't explain them in enough detail or offer convincing evidence that the properties are achieved

[xudong963]

I replaced the four-policy table with a worked byte-range example. Requests for 100..200 and 220..300 can be served by one read of 100..300, followed by slices 0..100 and 120..200. This makes the mechanism and cost explicit: one fewer storage request, 20 extra bytes fetched, and identical logical input to the decoder. It does not assume that the tradeoff is always faster.

I also tied the overlap explanation to try_next_reader: once an independent reader owns ready-to-decode input, a CPU worker can consume it while the host advances the push decoder and fetches subsequent data. The post states the host’s responsibilities for bounding work and preserving order, and acknowledges what callbacks can already do.

[alamb]

If we want to focus the blog on architectural guidance / convincing other implementations, I think we should put that in the intro. I am not sure if that is the main argument we shuld make in this post or not 🤔

[xudong963]

I removed “A shared boundary across formats.” The post now centers on the Parquet push API and the control it exposes to an application. The original design-guidance paragraph remains in the tradeoffs section, but a common architecture across Parquet, Lance, and Vortex is no longer a separate argument or part of the conclusion.

[xudong963]

Alternatively, we could maybe just focus on this post on explaining the API and leave the benefits somewhat vague.

I think we could trim the current content down (for example, we could make the intro more concise, remove the "Where the boundary helps in practice" section (it seems somewhat vague at the moment)

And then we could say "we plan to write more in the future about how we used this API to improve performance..."

Sorry for this back and forth @xudong963

Sounds good, let's do this first. I'll follow up on the current work, then we can publish a series blogs later

[alamb]

Thanks @xudong963 -- I plan to go through this carefully tomorrow