Skip to content

feat(buffer): add NullBufferBuilder::append_word - #11867

Merged
Jefffrey merged 4 commits into
apache:mainfrom
Rich-T-kid:rich-T-kid/null-buffer-append-word
Oct 24, 2026
Merged

feat(buffer): add NullBufferBuilder::append_word#11867
Jefffrey merged 4 commits into
apache:mainfrom
Rich-T-kid:rich-T-kid/null-buffer-append-word

Conversation

@Rich-T-kid

@Rich-T-kid Rich-T-kid commented Oct 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

N/A. Follow-up to #11732.

Rationale for this change

Callers with packed validity bits currently need to append them one bit at a time or first build a NullBuffer.

What changes are included in this PR?

This adds NullBufferBuilder::append_word, matching BooleanBufferBuilder::append_word while preserving lazy allocation for all-valid words. It also adds focused unit tests and benchmarks for different validity patterns.

Are these changes tested?

The tests cover full and partial all-valid words without materialization, plus a mixed partial word at an unaligned offset and an all-valid append after materialization. The benchmark covers all-valid, very sparse, sparse, densely packed, and alternating validity words.

Are there any significant user-facing changes?

Adds NullBufferBuilder::append_word.

@Rich-T-kid
feat(buffer): add NullBufferBuilder::append_word
ea34379 
Append packed validity words while preserving lazy allocation for all-valid data. Include focused correctness tests and per-bit comparison benchmarks.
@Rich-T-kid
Rich-T-kid marked this pull request as draft October 23, 2026 17:39
@github-actions github-actions Bot added the arrow-core Core crates related to representing arrow data label Oct 23, 2026
@Rich-T-kid

Rich-T-kid commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

run benchmark null_buffer_builder

@adriangbot

adriangbot commented Oct 23, 2026

Copy link
Copy Markdown

🤖 Benchmark starting (GKE) | trigger

Target: null_buffer_builder

Sharding factor: 1 (1 workers).

Comparing rich-T-kid/null-buffer-append-word (ea34379) to df4dc3e (merge-base) diff

Run configuration 
run benchmark null_buffer_builder
shards: 1

Results will be posted when all workers finish.

File an issue against this benchmark runner

@adriangbot

adriangbot commented Oct 23, 2026

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Target: null_buffer_builder

Comparing rich-T-kid/null-buffer-append-word (ea34379) to df4dc3e (merge-base) diff

Run configuration 
run benchmark null_buffer_builder
shards: 1

Baseline build unavailable for shard(s) 1; changed-only measurements for those workers.

Details 
group                                         changed
-----                                         -------
NullBufferBuilder append/bit/all_valid        1.00     65.8±0.03µs 474.7 MElem/sec
NullBufferBuilder append/bit/alternating      1.00     81.9±8.02µs 381.6 MElem/sec
NullBufferBuilder append/bit/sparse_nulls     1.00     71.2±2.11µs 438.9 MElem/sec
NullBufferBuilder append/word/all_valid       1.00   1007.7±0.26ns 30.3 GElem/sec
NullBufferBuilder append/word/alternating     1.00      5.2±0.13µs  5.9 GElem/sec
NullBufferBuilder append/word/sparse_nulls    1.00      2.0±0.10µs 15.2 GElem/sec
Per-runner information
null_buffer_builder — shard 1/1

Node: gk3-benchmark-cluster-nap-ra81rov6-4c2cf172-qb9h

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6371425566-3800-kjn2k 6.12.105+ #1 SMP Sat Sep  5 07:54:11 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench null_buffer_builder
CPU Details (lscpu) 
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Resource Usage

branch

Metric Value
Wall time 65.0s
Peak memory 9.8 MiB
Avg memory 7.9 MiB
CPU user 59.9s
CPU sys 0.0s
Peak spill 0 B

File an issue against this benchmark runner

@Rich-T-kid Rich-T-kid mentioned this pull request Oct 23, 2026
Open
Rich-T-kid
Rich-T-kid commented Oct 23, 2026
View reviewed changes
Comment thread arrow-buffer/benches/null_buffer_builder.rs Outdated
@Rich-T-kid Rich-T-kid changed the title feat(buffer): add NullBufferBuilder::append_word feat(buffer): add NullBufferBuilder::append_word Oct 23, 2026
@Rich-T-kid
Rich-T-kid marked this pull request as ready for review October 23, 2026 18:17
@Rich-T-kid

Rich-T-kid commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

run benchmark null_buffer_builder

@adriangbot

adriangbot commented Oct 23, 2026

Copy link
Copy Markdown

🤖 Benchmark starting (GKE) | trigger

Target: null_buffer_builder

Sharding factor: 1 (1 workers).

Comparing rich-T-kid/null-buffer-append-word (2ae1e95) to df4dc3e (merge-base) diff

Run configuration 
run benchmark null_buffer_builder
shards: 1

Results will be posted when all workers finish.

File an issue against this benchmark runner

@Rich-T-kid
Rich-T-kid requested review from Jefffrey and a balanced review from Copilot October 23, 2026 18:21
Copilot AI reviewed Oct 23, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@adriangbot

adriangbot commented Oct 23, 2026

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Target: null_buffer_builder

Comparing rich-T-kid/null-buffer-append-word (2ae1e95) to df4dc3e (merge-base) diff

Run configuration 
run benchmark null_buffer_builder
shards: 1

Baseline build unavailable for shard(s) 1; changed-only measurements for those workers.

Details 
group                                              changed
-----                                              -------
NullBufferBuilder append/word/all_valid            1.00   1009.9±1.62ns 30.2 GElem/sec
NullBufferBuilder append/word/alternating          1.00      5.2±0.13µs  5.8 GElem/sec
NullBufferBuilder append/word/dense_nulls          1.00      5.2±0.13µs  5.8 GElem/sec
NullBufferBuilder append/word/sparse_nulls         1.00      2.0±0.10µs 15.1 GElem/sec
NullBufferBuilder append/word/very_sparse_nulls    1.00  1777.0±81.25ns 17.2 GElem/sec
Per-runner information
null_buffer_builder — shard 1/1

Node: gk3-benchmark-cluster-nap-1v24fx0a-1e5965c3-6r4x

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

uname:

Linux bench-c6371804258-3801-hncbw 6.12.105+ #1 SMP Sat Sep  5 07:54:11 UTC 2026 aarch64 GNU/Linux

BENCH_COMMAND:

cargo bench --features=arrow,async,test_common,experimental,object_store --bench null_buffer_builder
CPU Details (lscpu) 
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Resource Usage

branch

Metric Value
Wall time 55.0s
Peak memory 9.4 MiB
Avg memory 7.3 MiB
CPU user 48.9s
CPU sys 0.0s
Peak spill 0 B

File an issue against this benchmark runner

@Rich-T-kid
fix(buffer): validate append_word count
338b370 
Reject counts above 64 in optimized builds as well as debug builds. Document the public contract and cover the invalid input with a regression test.
@Jefffrey
Jefffrey added this pull request to the merge queue Oct 24, 2026
Merged via the queue into apache:main with commit cb1ed6e Oct 24, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@Jefffrey Jefffrey Jefffrey approved these changes

Assignees

No one assigned

Labels

arrow-core Core crates related to representing arrow data performance

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Rich-T-kid @adriangbot @Jefffrey