[Parquet] ALP encoder/decoder support by sdf-jkl · Pull Request #9372 · apache/arrow-rs

sdf-jkl · 2026-02-07T00:01:58Z

Which issue does this PR close?

Part of Implement ALP encoding #10541
Closes [Parquet] Prototype ALP encoding #8748 .
Related spec PR: GH-533: Add ALP (Adaptive Lossless floating-Point) encoding specification parquet-format#557

Rationale for this change

check issue

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

alamb · 2026-02-08T13:32:29Z

amaaazing

sdf-jkl · 2026-02-20T23:16:14Z

@alamb I worked with codex on replicating the c++ implementation reviewing commit by commit.

I'll do one final read myself, but this should be ready for initial review.

alamb · 2026-02-26T20:29:50Z

Thank you -- I will put this on my short list to reivew I was out last week

devanbenz

A few comments

sdf-jkl · 2026-03-10T20:41:27Z

@devanbenz Sorry for taking so long to address your comments. I'll make sure to work on it this week!

alamb · 2026-03-11T18:03:12Z

The final spec is

GH-533: Add ALP (Adaptive Lossless floating-Point) encoding specification parquet-format#557

devanbenz

Comment about header regarding publication of spec.

sdf-jkl · 2026-03-12T20:37:12Z

Thanks @devanbenz, I've finally addressed your reviews!

devanbenz · 2026-03-12T22:54:40Z

Thanks @devanbenz, I finally addressed your reviews!

Please cargo fmt the code 🫡

sdf-jkl · 2026-03-12T23:12:46Z

~~I swear I did 😿~~
Had to run

cargo fmt -p parquet --  --config skip_children=true `find ./parquet -name "*.rs" \! -name format.rs`

5e5c8d2

alamb · 2026-08-11T18:35:34Z

FWIW I plan to merge this once the upstream parquet-testing PR is merged. I will prod the upstream PR tomorrow at the parquet sync

sdf-jkl · 2026-08-21T18:22:10Z

added #10785 to update the parquet-testing dep

sdf-jkl · 2026-08-21T18:34:54Z

Added another one with just the pinned rev change - #10786

# Conflicts: # parquet-testing # parquet/src/encodings/encoding/mod.rs

sdf-jkl · 2026-08-25T16:33:47Z

@alamb

alamb · 2026-08-25T19:59:01Z

@Jefffrey and @etseidl -- do you think it is ok to merge the Rust ALP implementation for inclusion in arrow/parquet 60?

I think we could be one of the first major open source implementations to ship ALP if we did so

(thank you @sdf-jkl for all your work with this)

alamb · 2026-08-25T19:59:20Z

BTW if you want to see a sneak preview of a blog post about what ALP is

Add ALP support blog parquet-site#195

sdf-jkl · 2026-08-25T20:03:25Z

@alamb @devanbenz thanks for your help 🙏

etseidl · 2026-08-25T20:03:45Z

@Jefffrey and @etseidl -- do you think it is ok to merge the Rust ALP implementation for inclusion in arrow/parquet 60?

Yes, I don't see a problem with that. It's in "preview" in the spec, and it requires opt-in from users.

I think we could be one of the first major open source implementations to ship ALP if we did so

(thank you @sdf-jkl for all your work with this)

Yes thanks to all for getting this done so soon!

Jefffrey · 2026-08-26T06:28:23Z

sounds good; i was hoping to take a proper look at this since ive just skimmed it, but kept getting sidetracked by other PRs 😅

alamb · 2026-08-26T12:44:21Z

sounds good; i was hoping to take a proper look at this since ive just skimmed it, but kept getting sidetracked by other PRs 😅

I think we still have some more time until the 60 release -- so we can wait a while to merge it in

# Which issue does this PR close? - Follow-up to #10786, which updated the `parquet-testing` revision. - No tracking issue yet; one can be added if needed. # Rationale for this change The updated `parquet-testing` revision included: - [apache/parquet-testing#113](apache/parquet-testing#113): malformed and edge-case Variant files. - [apache/parquet-testing#117](apache/parquet-testing#117): mark four shredded Variant cases as invalid because they omit required `value` columns. - [apache/parquet-testing#118](apache/parquet-testing#118): JSON and BSON logical-type files. - [apache/parquet-testing#119](apache/parquet-testing#119): an extended ALP fixture. #10786 made these fixtures available to Arrow Rust, but did not exercise the JSON, BSON, or malformed Variant files. The same revision also added an extended ALP fixture. That fixture is covered by the ALP encoder/decoder work in #9372, where the required decoding support exists. # What changes are included in this PR? - Read the JSON logical-type fixture and verify all decoded string values. - Read the BSON logical-type fixture and verify all decoded binary values. - Validate all 14 files under `bad_data/variants`, including the one valid duplicate-offset case and 13 malformed cases. The four shredded Variant cases renamed as invalid by the same `parquet-testing` update are already exercised by the existing Variant integration harness. # Are these changes tested? Yes. The Parquet and Variant integration tests pass locally, along with formatting and clippy checks. The PR's CI checks are also green. # Are there any user-facing changes? No. This PR only adds integration-test coverage for existing `parquet-testing` fixtures. # AI usage This PR was prepared with OpenAI Codex and reviewed by a human. The integration tests, formatting, and clippy checks described above were run against the final branch, and the GitHub CI checks passed. --------- Co-authored-by: cetra3 <cetra3@hotmail.com>

alamb · 2026-09-02T10:56:21Z

I plan to merge this PR tomorrow unless anyone would like more time to review beforehand

alamb · 2026-09-03T18:54:28Z

Ignored one new test for MIRI in ea6d049

Followed the same pattern as existing tests that try to use zstd

arrow-rs/parquet/src/compression.rs

Line 959 in ec21627

#[cfg_attr(miri, ignore)] // Zstd calls native C functions unsupported by Miri

sdf-jkl added 2 commits February 6, 2026 18:48

Add alp to the thrift definitions

54bc343

Add alp to the encoding enum

622841b

github-actions Bot added the parquet Changes to the parquet crate label Feb 7, 2026

sdf-jkl and others added 10 commits February 12, 2026 16:43

Add alp header and page layout structs

bf754b2

Add Interleaved Vectors support

8337fcd

Merge branch 'main' into alp

5349a9e

Add bit unpacking, reverse FOR and exceptions patching

148b1d5

Decode page works

5f0222e

Add more checks and tests

027172f

Make decode page lazy + some tests

8ee1b7e

Find vectors using offsets at decode time, not parse time

18f885e

get decode fast path

a9fa0c3

Simplify decode_page_values

7754ebc

devanbenz suggested changes Feb 27, 2026

View reviewed changes

Comment thread parquet/src/encodings/decoding/alp.rs Outdated

Comment thread parquet/src/encodings/decoding/alp.rs Outdated

devanbenz reviewed Feb 27, 2026

View reviewed changes

Comment thread parquet/src/encodings/decoding/alp.rs Outdated

devanbenz previously requested changes Mar 11, 2026

View reviewed changes

Comment thread parquet/src/encodings/decoding/alp.rs Outdated

sdf-jkl added 6 commits March 12, 2026 14:09

remove version from page header

6b7e923

change log_vetctor_size range consts

a13f244

redo the integration test using a test file from the c++ pr

6e86d6f

cache parsed vectors

ac56cc2

use Bytes for packed_values

07ca6f0

clippy too many args

def5c09

This was referenced Aug 11, 2026

[Parquet] Support configurable ALP vector lengths #10645

Open

[Parquet] Validate the ALP sampling heuristic #10646

Open

This was referenced Aug 12, 2026

[Parquet] Add ALP encoder and decoder benchmarks sdf-jkl/arrow-rs#6

Closed

[Parquet] Alp criterion bench #10662

Draft

alamb mentioned this pull request Aug 14, 2026

Test: read/write ALP encoded parquet files (apache/arrow-rs#9372) apache/datafusion#24373

Draft

Merge branch 'main' into alp

f9794b4

mmgeorge mentioned this pull request Aug 19, 2026

Add PBF encoding extension proposal opengeospatial/geoparquet#298

Draft

sdf-jkl added 2 commits August 25, 2026 12:03

Merge branch 'main' of https://github.com/apache/arrow-rs into alp

2d8002b

# Conflicts: # parquet-testing # parquet/src/encodings/encoding/mod.rs

clippy

baea4e9

sdf-jkl mentioned this pull request Aug 25, 2026

Add ALP support blog apache/parquet-site#195

Open

Jefffrey added enhancement Any new improvement worthy of a entry in the changelog api-change Changes to the arrow API labels Aug 31, 2026

etseidl added the next-major-release the PR has API changes and it waiting on the next major version label Sep 1, 2026

sdf-jkl mentioned this pull request Sep 2, 2026

Add new parquet tests based on updated variant/json files #10785

Merged

alamb added 3 commits September 3, 2026 08:21

Merge remote-tracking branch 'apache/main' into alp

984968d

Merge branch 'main' into alp

e27676e

Ignore test_alp_extended under Miri (zstd calls native C functions)

ea6d049

Uh oh!

Conversation

sdf-jkl commented Feb 7, 2026 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Feb 8, 2026

Uh oh!

sdf-jkl commented Feb 20, 2026

Uh oh!

alamb commented Feb 26, 2026

Uh oh!

devanbenz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sdf-jkl commented Mar 10, 2026

Uh oh!

alamb commented Mar 11, 2026

Uh oh!

devanbenz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sdf-jkl commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devanbenz commented Mar 12, 2026

Uh oh!

sdf-jkl commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb commented Aug 11, 2026

Uh oh!

sdf-jkl commented Aug 21, 2026

Uh oh!

sdf-jkl commented Aug 21, 2026

Uh oh!

sdf-jkl commented Aug 25, 2026

Uh oh!

alamb commented Aug 25, 2026

Uh oh!

alamb commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sdf-jkl commented Aug 25, 2026

Uh oh!

etseidl commented Aug 25, 2026

Uh oh!

Jefffrey commented Aug 26, 2026

Uh oh!

alamb commented Aug 26, 2026

Uh oh!

alamb commented Sep 2, 2026

Uh oh!

alamb commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sdf-jkl commented Feb 7, 2026 •

edited by alamb

Loading

sdf-jkl commented Mar 12, 2026 •

edited

Loading

sdf-jkl commented Mar 12, 2026 •

edited

Loading

alamb commented Aug 25, 2026 •

edited

Loading