Repository navigation
feat(sdk): pure scale-limit sizing helpers and a segment-integrity ADR (DSPX-4650) - #1020
Draft
dmihalcik-virtru wants to merge 1 commit into
Draft
dmihalcik-virtru wants to merge 1 commit into
dmihalcik-virtru wants to merge 1 commit into
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
dmihalcik-virtru
added this pull request to stack #1033
September 10, 2026 20:52
dmihalcik-virtru
force-pushed
the
DSPX-4650-scale-limits
branch
from
September 22, 2026 14:23
1d6a578 to
849d7fe
Compare
dmihalcik-virtru
removed this pull request from stack #1033
September 22, 2026 14:24
dmihalcik-virtru
force-pushed
the
DSPX-4650-scale-limits
branch
from
September 22, 2026 15:53
849d7fe to
8cbaf25
Compare
dmihalcik-virtru
force-pushed
the
DSPX-4651-manifest-fail-fast
branch
from
September 23, 2026 18:15
b671a6b to
5ebc5fd
Compare
dmihalcik-virtru
force-pushed
the
DSPX-4650-scale-limits
branch
2 times, most recently
from
September 23, 2026 18:15
8cbaf25 to
98c06fe
Compare
X-Test Failure Reportcoverage-web-app-browser |
dmihalcik-virtru
force-pushed
the
DSPX-4651-manifest-fail-fast
branch
from
September 24, 2026 14:34
5ebc5fd to
71aa201
Compare
dmihalcik-virtru
force-pushed
the
DSPX-4650-scale-limits
branch
from
September 24, 2026 14:34
98c06fe to
474c963
Compare
X-Test Failure Reportopentdf-ctl |
dmihalcik-virtru
force-pushed
the
DSPX-4651-manifest-fail-fast
branch
from
September 25, 2026 19:15
71aa201 to
683c746
Compare
dmihalcik-virtru
force-pushed
the
DSPX-4650-scale-limits
branch
from
September 25, 2026 19:15
474c963 to
29f6f17
Compare
X-Test Failure Reportcoverage-lib |
dmihalcik-virtru
force-pushed
the
DSPX-4650-scale-limits
branch
from
September 28, 2026 15:04
29f6f17 to
b5c3507
Compare
dmihalcik-virtru
force-pushed
the
DSPX-4651-manifest-fail-fast
branch
from
September 28, 2026 15:04
683c746 to
0cab1f8
Compare
dmihalcik-virtru
marked this pull request as draft
September 28, 2026 15:08
X-Test Failure Reportcoverage-web-app |
Item 0 of DSPX-4648: fix the per-segment integrity algorithm before any of the 50 TiB limits are derived from it, and extract the size arithmetic those limits need into pure functions. Decision (spec/DSPX-4650-segment-integrity-algorithm.md): per-segment integrity stays GMAC -- free, since getSignature just returns the AES-GCM auth tag the cipher already computed -- while the root signature stays HS256. No default changes. Every limit is nonetheless derived for HS256, the more expensive case at 56 vs 36 bytes per manifest segment entry, so moving the default later loosens nothing and needs no re-derivation. Adds lib/tdf3/src/utils/scale-limits.ts: base64Length, perSegmentEntryBytes, finalSegmentEntryBytes, segmentCountFor, estimateSegmentsArrayBytes, estimateManifestBytes and minSegmentSizeFor. All pure, all functions of sizes only, so 50 TiB behaviour is reachable in a unit test rather than only by encrypting 50 TiB. Nothing calls them yet; this is behaviour-neutral. The constants are pinned against the real serializer rather than a hand-built entry: tests/mocha/helpers/write-tdf.ts drives writeStream with no KAS and no network, and the specs measure what a 5th segment actually adds to an emitted manifest for both GMAC and HS256. Perturbing any entry-cost constant by one byte fails 13 of the 19 tests. Also moves the plan document into spec/ as DSPX-4648-web-sdk-large-files.md. DSPX-4650
dmihalcik-virtru
force-pushed
the
DSPX-4651-manifest-fail-fast
branch
from
October 15, 2026 13:11
0cab1f8 to
27bbf4a
Compare
dmihalcik-virtru
force-pushed
the
DSPX-4650-scale-limits
branch
from
October 15, 2026 13:11
b5c3507 to
5a7214a
Compare
X-Test Failure Reportcoverage-lib |
|
This was referenced Oct 15, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Item 0 of the 50 TiB workstream. Two things, both prerequisites for everything after them, neither of which changes any behaviour:
spec/DSPX-4650-segment-integrity-algorithm.mdrecords that per-segment integrity stays GMAC and the root signature stays HS256 — the current defaults — and why.lib/tdf3/src/utils/scale-limits.tsaddsbase64Length,perSegmentEntryBytes,finalSegmentEntryBytes,segmentCountFor,estimateSegmentsArrayBytes,estimateManifestBytesandminSegmentSizeFor. All pure, all functions of sizes only. Nothing calls them yet.Also moves the plan document into
spec/asDSPX-4648-web-sdk-large-files.md, matching the existingspec/DSPX-*.mdconvention.Why the ADR blocks the rest
The manifest carries one
segments[]entry per segment, and the digest in that entry is the only part of the manifest that scales with payload size:{"hash":"…"},)At 50 TiB with 16 MiB segments that is 118 MB of manifest versus 184 MB — a 1.56× factor on the single quantity the whole plan is trying to bound. Picking a segment size, a manifest cap, or a maximum encryptable payload before fixing this constant means deriving all three twice.
The decision: GMAC is free —
getSignature()(lib/tdf3/src/tdf.ts:369-372) returnscontent.slice(-16), the AES-GCM auth tag the cipher already computed. HS256 is a separate HMAC-SHA256 pass over every segment's ciphertext: at 50 TiB, 50 TiB of extra hashing on encrypt and another 50 TiB on decrypt-verify, competing with the encrypt for the same WebCrypto queue. It buys an integrity value nominally independent of the cipher, butgetSignaturekeys the HMAC from the same unwrapped key, so that independence is weaker in practice than in principle. Not worth it per-segment; kept for the root signature, where it is computed once over the concatenated segment hashes rather than over the payload.But every limit is still derived for HS256.
perSegmentEntryBytes(alg)takes the algorithm as a parameter, so the limits are a function of it rather than of the shipped default. The landing point in item 1 (16 MiB segments, 256 MiB cap) is chosen to fit 50 TiB at 56 B/entry; running at 36 B/entry just leaves headroom. If the default ever moves, nothing needs re-deriving and no limit silently loosens.Why extract the arithmetic
Whether a manifest will fit, how big a payload may be, what segment size to pick and how much to prefetch are all pure functions of sizes — none of them need a byte of payload. Separating them means the behaviour at 50 TiB is reachable in a unit test instead of only by encrypting 50 TiB. That is the whole testing strategy for DSPX-4648, and it only works if the estimate provably matches what the serializer really emits.
How the constants are pinned
Not by asserting on a hand-built entry — that would just restate the bug.
lib/tests/mocha/helpers/write-tdf.tsdriveswriteStreamdirectly with mock keys, no KAS and no network, so a real container can be written with a 64-byte segment size. The specs then measure deltas against real emitted manifests, for both algorithms:segmentsarray minus a 4-segment one equalsperSegmentEntryBytes(alg)exactly;segmentSize/encryptedSegmentSize) costsfinalSegmentEntryBytes(...)exactly;estimateSegmentsArrayBytesnever under-estimates, swept over[0, 1, 63, 64, 65, 127, 128, 200, 640]bytes.If the serializer ever changes shape — a renamed field, a new per-segment key — those fail rather than the estimate silently drifting. Verified red-first: perturbing
SEGMENT_ENTRY_OVERHEAD_BYTESby one byte fails 13 of the 19 tests.The estimate is deliberately an over-estimate: every entry is counted with its separating comma even though the last has none, which pays for the closing bracket with a byte to spare. Under-estimating would let a doomed encrypt start.
The numbers
A table-driven block asserts the plan's sizing directly, at 50 TiB, in milliseconds:
minSegmentSizeFor(50 TiB) === 12,800— the floor imposed byMAX_GCM_INVOCATIONS_PER_KEYfrom #1018. Every segment size under consideration is orders of magnitude above it, so the IV ceiling is not the binding constraint, but it is the floor any future default has to clear.How to test
19 passing, ~30 ms. Full suite: 410 passing, 0 failing.
Risk
None to runtime behaviour — no existing code path calls any of this. The risk is that the estimator is wrong, which is what the serializer-agreement tests are for.