Repository navigation
perf(sdk): native CRC-32 on Node, slice-by-16 elsewhere (DSPX-5232) - #1096
Draft
dmihalcik-virtru wants to merge 1 commit into
Draft
dmihalcik-virtru wants to merge 1 commit into
dmihalcik-virtru wants to merge 1 commit into
Conversation
Every encrypted output byte goes through CRC-32 for the ZIP data
descriptors, and the byte-at-a-time JS loop (about 500 MiB/s) capped
encrypt near 400 MiB/s.
Use zlib's native crc32 where the runtime has it (Node >= 20.15 / 22.2,
Bun). It is reached through process.getBuiltinModule('node:zlib') rather
than an import, so browser bundles never see node:zlib. Everywhere else,
a slice-by-16 table loop runs at about 2.2 GB/s in V8 (4x the old loop).
With the segment-buffer fix, encrypt goes from about 420 MiB/s to
2.3-2.8 GB/s in Node (1 GiB in 64 KiB chunks, 2-256 MiB segments).
npm run bench:encrypt now reports which CRC-32 backend it measured.
Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>
dmihalcik-virtru
force-pushed
the
DSPX-5232-segment-buffer
branch
from
October 19, 2026 17:27
7a9e2a4 to
f961675
Compare
dmihalcik-virtru
force-pushed
the
DSPX-5232-crc
branch
from
October 19, 2026 17:27
434da28 to
46ae9f6
Compare
X-Test Failure Reportcoverage-web-app |
|
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What / why
DSPX-5232, part 2 of 2. Stacked on #1095; review only the top commit until that merges.
Every encrypted output byte goes through CRC-32 for the ZIP data descriptors. The byte-at-a-time JS loop in
buffer-crc32.tsran at about 500 MiB/s and capped encrypt near 400 MiB/s (DSPX-5124 analysis item #11). ZIP requires the CRC on every entry, so it can't be skipped. It has to be fast instead.crc32, reached throughprocess.getBuiltinModule('node:zlib')rather than an import. Browser bundles therefore never seenode:zlib; karma's webpack bundle and web-test-runner both build and pass.unsigned(buf, previous)keeps its signature. The module addsportableCrc32andcrc32Backend: 'zlib' | 'portable'.npm run bench:encryptnow reports which backend it measured.Results
Node 24.21.0, Apple M5 Pro, 1 GiB in 64 KiB chunks (
npm run bench:encrypt):CRC-32 alone:
This meets the ticket's acceptance criterion of encrypt ≥ 1 GB/s in Node.
xtest paired A/B (
js@mainvs. this stack, local platform): 7 of 7 cells pass with no regressions.js-encrypt-32MiBwall ratio is 0.923 [0.903, 0.978]. Startup (about 1.1 s per CLI call) hides most of the gain at those sizes; larger xtest payloads are being reworked in opentdf/tests#622.How to test
The tests check both implementations against the standard check values. They also check continuation across every split point, and compare
portableCrc32against a bit-at-a-time reference CRC on every length and alignment around a 16-byte block. Under Node they assert that the zlib backend is the one in use. Separately,unzip -tfound no CRC errors in a TDF written by the js CLI with this change, and main's CLI decrypts it.Risk
Touches ZIP integrity metadata, though not cryptographic integrity: readers don't check the CRC today. A wrong CRC would make TDFs fail strict ZIP tools, which is what the reference-equivalence tests guard against.
lib/package.jsonstill lists the unusedbuffer-crc32npm dependency; removing it is left for a follow-up to avoid lockfile churn here.