[moshap-firebolt]

Background

FB-1380: PackDB's arrow_reader fuzzer found a 1,145-byte Arrow IPC stream that makes the reader allocate 973 MB. Each compressed body buffer starts with its declared uncompressed size, and DecompressBuffer allocates that many bytes before decompressing, checking the size only afterwards (Failed to fully decompress buffer). apache#84 (FIR-47668) bounded where a raw buffer may point in the file; the decompressed size was still unbounded. Upstream apache/arrow main has the same code.

Summary

DecompressBuffer rejects the declared uncompressed size before allocating when it is:

• negative (other than -1, which marks an uncompressed buffer), or
• larger than the codec can decode from the compressed bytes: 255× for LZ4_FRAME (an LZ4 match spends at least one byte per 255 bytes of length) and 32768× for ZSTD (an RLE block expands one byte into at most a 128 KiB block behind a 3-byte header). These are the only IPC body codecs.

Real data, all-zero buffers included, stays within these ratios, so no valid stream is rejected.

Test Plan

• TestWriteRecordBatch.ReadRejectsImpossibleUncompressedSize (arrow-ipc-read-write-test): for LZ4_FRAME and ZSTD, a compressed batch round-trips unchanged; with its values buffer's declared size set one byte past the codec's ratio the read fails with Invalid, and a ProxyMemoryPool shows the size was never allocated; a negative size fails with Invalid. Without the fix the test fails (the reader allocates the declared size, then reports Failed to fully decompress buffer).
• arrow-ipc-read-write-test passes in full (411 tests, with ARROW_TEST_DATA pointing at testing/data).
• The packdb submodule bump will carry the fuzzer's input as an arrow_reader regression seed.

🤖 Generated with Claude Code

---

Note

High Risk
Changes IPC read-path memory allocation for compressed bodies; incorrect ratio bounds could reject valid streams or still allow abuse, but ratios are codec-specific and covered by new tests.

Overview
Hardens Arrow IPC decompression against malicious declared sizes (FB-1380): a fuzzer could trigger ~1 GB allocations from a tiny stream because DecompressBuffer trusted the little-endian prefix and allocated before decompressing.

DecompressBuffer in reader.cc now rejects invalid prefixes before allocation: any negative value other than -1 (uncompressed sentinel), and any declared size above what the body codec can plausibly expand—255× compressed length for LZ4_FRAME and 32768× for ZSTD—via new MaxDecompressionRatio. Other codecs keep an unbounded ratio.

Adds ReadRejectsImpossibleUncompressedSize in read_write_test.cc: round-trip for honest LZ4/ZSTD batches; patched impossible or negative declared sizes fail with Invalid and a ProxyMemoryPool proves the oversized buffer was never allocated.

^{Reviewed by Cursor Bugbot for commit 5ae2a08. Bugbot is set up for automated code reviews on this repo. Configure here.}

[moshap-firebolt]

Closing in favor of #51, which applies the same codec-ratio bound (255x LZ4_FRAME, 32768x ZSTD) and negative-size check, and additionally stream-decodes declared sizes over 64 MiB. The unit test here (TestWriteRecordBatch.ReadRejectsImpossibleUncompressedSize, with a ProxyMemoryPool asserting the declared size is never allocated) could be carried over to #51.