Use SWAR + VarHandle for 7-bit binary Smile encode/decode

[pjfanning]

Smile packs binary values 7 payload bytes into 8 encoded bytes of 7 significant bits each (ENCODE_BINARY_AS_7BIT, enabled by default, so this is the path for every Smile binary value). Both directions were done a byte at a time and were not part of #757:

• decoder: 8 separate loads, ~10 ALU ops, 7 separate stores per chunk
• encoder: 7 loads, ~14 ops, 8 stores

Both collapse to a single 8-byte load, ~9 ALU ops and a single 8-byte store:

// decode: 8 seven-bit bytes -> 56 bits
long v = SmileVarHandleUtil.getLongBE(in, inPtr);
v = ((v & 0x7F007F007F007F00L) >>> 1) | (v & 0x007F007F007F007FL);
v = ((v & 0x3FFF00003FFF0000L) >>> 2) | (v & 0x00003FFF00003FFFL);
v = ((v & 0x0FFFFFFF00000000L) >>> 4) | (v &         0x0FFFFFFFL);
SmileVarHandleUtil.setLongBE(out, outPtr, v << 8);

with the exact inverse for encode.

Structure

New package-private Smile7BitBinaryCodec holds both directions and probes VarHandle availability using the same pattern as SmileParserBase._decodeQuad() (the byte-shifting fallback lives in a class that never references VarHandle, so the fallback path cannot fail to link). SmileVarHandleUtil gains getLongBE / setLongBE.

Merged 3.x after #779, which removed SmileByteShiftUtil in favour of ByteArrayUtil in jackson-core. This branch originally mirrored getLongBE / setLongBE into SmileByteShiftUtil; those copies were never called (see below) and are dropped, ByteArrayUtil now being the byte-shifting class the rule above refers to.

Where VarHandle is unusable the codec reports failure and each caller runs its existing loop, unchanged. This is deliberate, and stronger than "nothing to gain": routing the fallback through ByteArrayUtil would be an outright slowdown, because composing the 8-byte load and store out of shifts costs more per chunk than the per-byte loop it would be replacing.

Measured on JDK 17.0.19 (Temurin), inner loops in isolation, 150k chunks/pass, median of 31 passes x 3 alternating rounds (reproducible to ~1% across runs):

direction	perByte (today)	SWAR + ByteArrayUtil	SWAR + VarHandle
encode	661 us	838 us (0.79x)	336 us (1.97x)
decode	601 us	837 us (0.72x)	372 us (1.61x)

So the per-byte loop stays the right fallback. That same fallback covers chunks that would over-read or over-write by the one byte of overhang the 8-vs-7 asymmetry needs (the decode store writes 8 bytes for 7 wanted; the encode load reads 8 for 7 wanted).

Applied at all five sites: three decode (_readBinaryEncoded, _finishBinary7Bit, _finishBinary7BitLong) and two encode (the byte[] and InputStream variants of _write7BitBinaryWithLength).

Benchmarks

JDK 17.0.19 (Temurin). Inner loops in isolation, both forms lifted verbatim from the call sites:

encode old      922.0 us/pass      0.12 GB/s payload
encode new      309.7 us/pass      0.37 GB/s payload
decode old      794.6 us/pass      0.14 GB/s payload
decode new      271.3 us/pass      0.42 GB/s payload

encode speedup: 2.98x    decode speedup: 2.93x

End-to-end through the streaming API (writeBinary / getBinaryValue), median of 3 alternating passes against a 3.x build:

payload	write GB/s	read GB/s
1KB	0.66 → 0.95 (1.44x)	0.85 → 0.99 (1.17x)
64KB	0.68 → 0.92 (1.35x)	0.88 → 1.00 (1.16x)
1MB	0.51 → 0.98 (1.93x)	0.71 → 0.86 (1.21x)

Read gains are smaller because getBinaryValue() on large values also pays for ByteArrayBuilder segment copies, which the codec does not touch.

Tests

Binary7BitRoundtripTest covers every length 0–63 plus sizes past the internal buffers and the 250k "long binary" threshold, across all four write/read path combinations (byte[] vs InputStream write, in-memory vs streaming read), so both the SWAR path and the overhang fallback are exercised at every chunk boundary. Also asserts the encoded bytes themselves are unchanged against an independent reference.

Verified the tests actually catch regressions in the new code: perturbing a shift in either direction fails 4 of the 5 tests.

Full smile module test suite passes.

Note on malformed input

The current decoder reads buffer[i] without & 0xFF, so an encoded byte with the high bit set (illegal in Smile) sign-extends and corrupts the result. The SWAR form masks it to 0 instead. Both are garbage for garbage input, but the bytes differ — flagging in case that matters for any fuzz corpus.

🤖 Generated with Claude Code

[github-actions]

🧪 Code Coverage Report

Coverage Type	Coverage	Change
📝 Instructions		📉 -0.11%
🔀 Branches		📈 +0.14%