[rapour]

Summary

• Tracking Issue: Tracking Issue: Elias-Fano Encoding #10605

This is the first step towards Elias-Fano: the Unary encoding that will hold EF's upper bits. It lives in a new encodings/elias-fano crate, which will also hold the EliasFano parent, as discussed on #10377.

It is deliberately naive. There is no index, no slice kernel and no compute kernel, so the PR is mostly boilerplate, and the benchmarks set the baselines that later work is measured against. It is in no file edition and vortex-file does not register it, so the default writers reject it.

Next up is slicing for Unary, then the naive EliasFano parent with upper and lower children.

Changes

Layout. One BitBuffer. Each value writes its gap from the previous value as unset bits, then one set bit. The first value is stored as base, so the set bit for index i sits at (v[i] - base) + i, and the array takes len + (max - base) bits. For [3, 5, 5, 9], base is 3 and the bits are 1 001 1 00001.

This is the upper-bits array of Vigna's quasi-succinct indices (section 4), counted from base rather than from 0. Unlike rise-rs and vers, there is no sentinel bit before the first value and no guard bit after the last. That way, slicing can trim the bits as they are, and a search by value will need a branch at each end instead.

Read per bucket, these are the same bits as the bucket form of EF's upper bitmap, where each bucket's count is written in negated unary, minus the last bucket's closing zero.

The metadata is two u64s, base and max. The dtype must be a non-nullable unsigned integer. With FoR above EF, as the issue proposes, the upper parts are never negative, so the parent can always give this child an unsigned ptype.

Logical values. A Unary array holds one value per element, not one count per bucket. As EF's upper child it then has the parent's length, like the children of FoR and BitPacked, and an unfused EF decode is just (upper << width) | lower.

Validation. try_new, deserialize and with_buffers check the dtype, that base <= max and max fits the ptype, the bit length, a zero bit offset, that the first and last bits are set, and that exactly len bits are set. The last check is one popcount. Any array that passes encodes some non-decreasing sequence from base to max, so the readers need no per-value checks, and a point read always agrees with a full decode.

Encoding. unary_encode returns None for nullable, signed or non-integer input, for unsorted input, and when the bits would take more than the plain array's len × bit width.

Reads.

• execute walks the set bits once.
• scalar_at finds its set bit with BitBuffer::select, which scans from the first bit.
• Slicing uses the default, which decodes and then slices. A lazy slice can't be serialised until it is executed, so Unary slicing should land before the EF parent.
• Construction sets IsSorted as exact.

Wiring. The crate is a workspace member and dependency, and runs in CodSpeed shard 8.

[codspeed]

Merging this PR will degrade performance by 10.95%

3 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 17 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 40 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

❌ 1 regressed benchmark
✅ 2407 untouched benchmarks
🆕 20 new benchmarks
⏩ 12 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	CPU Simulation	test_scan_filter[int]	3.1 ms	3.4 ms	-10.95%
🆕	CPU Simulation	decode[Dense]	N/A	1.4 ms	N/A
🆕	CPU Simulation	decode[Gap]	N/A	2.1 ms	N/A
🆕	CPU Simulation	decode[Runs]	N/A	91.1 µs	N/A
🆕	CPU Simulation	decode[Spread]	N/A	2.8 ms	N/A
🆕	CPU Simulation	encode[Dense]	N/A	1.4 ms	N/A
🆕	CPU Simulation	encode[Gap]	N/A	2 ms	N/A
🆕	CPU Simulation	encode[Runs]	N/A	77 µs	N/A
🆕	CPU Simulation	encode[Spread]	N/A	2.7 ms	N/A
🆕	CPU Simulation	scalar_at_random[Dense]	N/A	477.9 µs	N/A
🆕	CPU Simulation	scalar_at_random[Gap]	N/A	562 µs	N/A
🆕	CPU Simulation	scalar_at_random[Runs]	N/A	344 µs	N/A
🆕	CPU Simulation	scalar_at_random[Spread]	N/A	643.6 µs	N/A
🆕	CPU Simulation	scalar_at_sequential[Dense]	N/A	470.2 µs	N/A
🆕	CPU Simulation	scalar_at_sequential[Gap]	N/A	545.7 µs	N/A
🆕	CPU Simulation	scalar_at_sequential[Runs]	N/A	359.3 µs	N/A
🆕	CPU Simulation	scalar_at_sequential[Spread]	N/A	629.8 µs	N/A
🆕	CPU Simulation	slice_decode[Dense]	N/A	1.5 ms	N/A
🆕	CPU Simulation	slice_decode[Gap]	N/A	2.1 ms	N/A
🆕	CPU Simulation	slice_decode[Runs]	N/A	108.6 µs	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing rapour:feat/elias-fano-unary-delta (e96311a) with develop (d270200)}

Footnotes

1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[connortsui20]

Ok so I just realized that there are multiple ways that we can imagine the canonical / semantic values of the upper bits.

For example, these two arrays can be thought of as the same:

"delta" encode upper values:   [0, 0, 2, 2, 2]
                                0, 0,  2,  0, 0
->                              1, 1, 001, 1, 1, (0)


                    bucket      00   01   10
bucket counts:                 [2,   0,    3]
->                             110  0    1110 


                                11001110

Notice how these are the same thing!

So I think what we need to decide is what we actually want this logical array to represent, should it be the "delta" encoded upper values (which are not exactly delta but close), or the "bucket counts", by which I mean how many upper bit values fall into each bucket.

I was thinking about the buckets, but I can see how both of these are valid representations of the data. However, in my mind, a general Unary encoding should probably be represented as n 1s followed by a closing 0.

I don't think would change basically anything about the Elias-Fano encoding, but it does mean that this implementation of Unary would have to change. For example, we should not require the inputs to be sorted, as a start.

[rapour]

Thanks, that makes sense. Let's go with bucket counts: Unary becomes a general unsigned encoding where each value v is written as v ones and a closing zero, and the input doesn't need to be sorted. EF's upper child then holds one count per bucket instead of one value per element.