diff --git a/_posts/2026-10-12-analytic-file-formats-sans-io.md b/_posts/2026-10-12-analytic-file-formats-sans-io.md new file mode 100644 index 000000000000..998eb57a059b --- /dev/null +++ b/_posts/2026-10-12-analytic-file-formats-sans-io.md @@ -0,0 +1,284 @@ +--- +layout: post +title: "Implementing Analytic File Formats Sans I/O" +description: "How the Arrow Rust Parquet push decoder exposes byte requirements so applications can control storage I/O" +date: "2026-10-12 00:00:00" +author: "Andrew Lamb, Xudong Wang, Neil Conway, and Daniël Heres" +categories: [parquet, arrow] +--- + + + + +A **Sans I/O** decoder interprets bytes but never fetches them. When it cannot +continue, it returns the byte ranges it needs; the application supplies buffers +and decides when to resume decoding. This exposes a boundary where an +application can use its own cache, storage scheduler, and memory policy. + +Sans I/O is an established pattern in [network protocol libraries][sans-io]. +This post introduces the [Arrow Rust Parquet reader's][arrow-rs-parquet] push +API, shows how an application drives it, and explains what control that API +provides. We plan to cover prefetching and its measured performance in a +follow-up post. + +## The Sans I/O contract + +The decoder performs no I/O, including through callbacks. Each time the +application asks it to advance, it does one of three things: + +1. Produces decoded data. +2. Reports that decoding is finished. +3. Returns the byte ranges needed to make progress. + +[`ParquetPushDecoder`] exposes this interaction through [`try_decode`] and +[`push_ranges`]. In simplified Rust, with application-provided `fetch` and +`consume` helpers: + +
+{% highlight rust %} +loop { + match decoder.try_decode()? { + DecodeResult::NeedsData(ranges) => { + // One buffer per logical range, in order. + let buffers = io_policy.fetch(&ranges)?; + decoder.push_ranges(ranges, buffers)?; + } + DecodeResult::Data(batch) => consume(batch)?, + DecodeResult::Finished => break, + } +} +{% endhighlight %} +
+ +These APIs are available in `arrow-rs` `57.1.0`, the version linked here. +The loop starts after file metadata is available; footer and metadata decoding +use the same request-and-supply pattern through [`ParquetMetaDataPushDecoder`]. +Errors propagate through the API as usual. + +
+ An integrated reader invokes a storage callback from its decoding loop; a Sans I/O decoder returns byte requirements for the application to schedule and supply +
Figure 1: The decoder determines its byte dependencies; the application schedules the I/O that satisfies them.
+
+ +"Push" describes the application supplying bytes to the decoder. It does not +prescribe the execution model of the rest of the query engine. + +## Why Parquet needs a conversation + +Streaming CSV and JSON parsers can accept chunks and ask for more input, usually +at the next position in the stream. [Apache Parquet] reads are non-sequential +and depend on earlier data. + +
+ Parquet decoding alternates between I/O and CPU stages for the file footer, metadata, optional pruning structures, and data pages +
Figure 2: A simplified Parquet decoding state machine. Pink states perform I/O, yellow states perform CPU work, and dashed states are optional.
+
+ +The trailing footer locates metadata, which locates row groups and column +chunks. Optional Bloom filters and page indexes can help eliminate row groups +or pages. Evaluating a predicate on one column can further reduce the pages +needed from another. The decoder retains this format-dependent state machine: +the application supplies the projection, row selections, and predicates, then +satisfies the byte requests that result. + +`NeedsData` describes what is missing **now**, not every range that might be +needed later. In particular, a predicate that has not yet run may eliminate +future reads. Selective prefetching therefore needs additional planning +information; the basic push loop does not provide a complete future read plan. + +## From logical ranges to physical reads + +Suppose `NeedsData` asks for two parts of a file: `100..200` (100 bytes) +and `220..300` (80 bytes). The end offset is excluded in this notation. +The 20-byte gap at `200..220` is not needed by the decoder. + +The application could issue two reads, fetching exactly 180 bytes. Alternatively, +it could **combine them into one read of `100..300`**, fetching 200 bytes. +This saves one storage request at the cost of reading the 20-byte gap. + +After the combined read, the returned buffer starts at index **0**, which +corresponds to file offset **100**. To locate a file byte in that buffer, +subtract 100 from its file offset: + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
File offsets and buffer indices for a combined read of bytes 100 through 299.
Part of the readFile offsetsBuffer indices
First requested range100..2000..100
Unneeded gap200..220100..120
Second requested range220..300120..200
+
+ +The application then supplies just the two requested parts to the decoder. +Here, `read_range` is an application helper returning a reference-counted +`Bytes` buffer: + +
+{% highlight rust %} +// 1. One storage read returns 200 bytes, including the gap. +let buffer = read_range(100..300)?; +// +// 2. Slice with BUFFER indices, which start at zero. +// File offset 100 -> index 0; offset 220 -> index 120. +let first = buffer.slice(0..100); // File bytes 100..200 +let second = buffer.slice(120..200); // File bytes 220..300 +// +// 3. Match FILE ranges to slices, in the same order. +// Omit the gap at buffer indices 100..120. +decoder.push_ranges( + vec![100..200, 220..300], // Original file offsets + vec![first, second], // Matching slices +)?; +{% endhighlight %} +
+ +The decoder receives the same two byte ranges as it would with separate reads. +Both slices share the fetched allocation without copying their contents; +decoding and decompression can still allocate output buffers. Whether saving +a request outweighs reading extra bytes depends on the storage system. This +example shows how the application controls that tradeoff, not a measured +speedup. + +A cache fits at the same boundary. If `100..300` is already resident, the host +can supply those slices without another read. If it fetches beyond the current +request, it can retain the extra bytes in its own cache and use them to satisfy +later requests. That also makes the host responsible for retaining and evicting +those bytes. Our [example driver][prototype driver] implements several such +policies around the same decoder. + +## Scheduling decoding and I/O + +A storage callback can already implement caching, retries, and request +coalescing. An async callback can submit requests to a shared scheduler. These +capabilities do not inherently require Sans I/O, as the [original design +discussion][async-callback] also considered. + +The push API makes advancing the decoder a separate operation from satisfying +its requests. A host can keep missing ranges as ordinary data and resume each +decoder when its scheduler is ready. Additional control requires additional API +support. For example, [`try_next_reader`] returns a +`ParquetRecordBatchReader` once the input needed for that reader is available. +The host can then: + +1. Send that reader to a CPU worker to produce batches from buffered input. +2. Advance the push decoder to discover the next byte requirements. +3. Fetch those bytes while the worker decodes the preceding data. + +The independent reader is what makes this handoff possible. The host must still +bound active readers and pending reads, and preserve output order where +required. Merely using an async fetch callback does not define that handoff. +Likewise, reducing the output batch size does not by itself reduce the input +that must be buffered before a reader is ready. + +This separation builds on existing designs: [`parquet2`][parquet2] separates +page reading from decompression and deserialization, while cuDF's +[experimental hybrid scan reader][cudf-hybrid] exposes staged byte-range and +buffer operations. The push decoder packages Parquet's dependent read steps +into a state machine that the host can drive through one request-and-supply +interface. + +## Tradeoffs and design guidance + +Sans I/O moves complexity to an explicit boundary; it does not make that +complexity disappear. + +- **The decoder is harder to implement.** State that was implicit in a call + stack or async future must become an explicit, testable state machine. +- **The host must drive the feedback loop.** Applications with no special I/O + requirements should not have to write that loop themselves, so libraries + should ship synchronous and asynchronous adapters. +- **Performance is not automatic.** Sans I/O enables a system-specific policy; + the application still has to choose and implement a good one. +- **The boundary must preserve zero-copy operation.** Callers should be able to + provide reference-counted or otherwise borrowed resident buffers, including + data fetched speculatively. + +For new analytic formats, we recommend making the Sans I/O state machine the +decoder core, then layering blocking, async, memory-mapped, and prefetching +readers on top. Format-specific choices such as pruning remain inside that +core. Storage-specific choices such as request size, concurrency, caching, and +buffer ownership remain outside. + +## What comes next + +The API gives applications explicit byte requirements and control over when to +resume decoding. Effective prefetching also needs information about future +reads and a policy for spending bandwidth and memory. + +Recent work addresses those pieces separately: [`scan_plan()`][scan-plan] has +merged into `arrow-rs` main to expose candidate future ranges, while +[batch-granular decoding][batch-decoding] and a [DataFusion read-ahead +prototype][datafusion-read-ahead] are under review. These additions are separate +from the `57.1.0` examples above. We plan to use a follow-up post to explain the +prefetching policy and evaluate it on object storage alongside a local-storage +baseline. + +This post is based on *Implementing Analytic File Formats Sans I/O*, by Andrew +Lamb, Xudong Wang, Neil Conway, and Daniël Heres. The two diagrams are reproduced +from that manuscript under [CC BY 4.0]. We thank InfluxData, Massive, and +Coralogix for supporting this work. + +[Apache Parquet]: https://parquet.apache.org/ +[arrow-rs-parquet]: https://github.com/apache/arrow-rs/tree/57.1.0/parquet +[sans-io]: https://sans-io.readthedocs.io/ +[`ParquetPushDecoder`]: https://docs.rs/parquet/57.1.0/parquet/arrow/push_decoder/struct.ParquetPushDecoder.html +[`try_decode`]: https://docs.rs/parquet/57.1.0/parquet/arrow/push_decoder/struct.ParquetPushDecoder.html#method.try_decode +[`try_next_reader`]: https://docs.rs/parquet/57.1.0/parquet/arrow/push_decoder/struct.ParquetPushDecoder.html#method.try_next_reader +[`push_ranges`]: https://docs.rs/parquet/57.1.0/parquet/arrow/push_decoder/struct.ParquetPushDecoder.html#method.push_ranges +[`ParquetMetaDataPushDecoder`]: https://docs.rs/parquet/57.1.0/parquet/file/metadata/struct.ParquetMetaDataPushDecoder.html +[async-callback]: https://github.com/apache/arrow-rs/issues/7983#issuecomment-3165065017 +[prototype driver]: https://github.com/xudong963/push_decoder_policies +[parquet2]: https://docs.rs/crate/parquet2/0.17.2 +[cudf-hybrid]: https://github.com/NVIDIA/cudf/blob/4f4de11b87d5f639d51f1ace71dbb3c19eaa15dc/cpp/include/cudf/io/experimental/hybrid_scan.hpp +[scan-plan]: https://github.com/apache/arrow-rs/pull/10555 +[batch-decoding]: https://github.com/apache/arrow-rs/pull/11238 +[datafusion-read-ahead]: https://github.com/apache/datafusion/pull/24086 +[CC BY 4.0]: https://creativecommons.org/licenses/by/4.0/ diff --git a/img/sans-io-analytic-formats/architecture.png b/img/sans-io-analytic-formats/architecture.png new file mode 100644 index 000000000000..36f1590bda2b Binary files /dev/null and b/img/sans-io-analytic-formats/architecture.png differ diff --git a/img/sans-io-analytic-formats/parquet-state-machine.png b/img/sans-io-analytic-formats/parquet-state-machine.png new file mode 100644 index 000000000000..58482119473b Binary files /dev/null and b/img/sans-io-analytic-formats/parquet-state-machine.png differ