fix(sdk): make the manifest size limit opt-in (DSPX-4502) by dmihalcik-virtru · Pull Request #4214 · opentdf/platform

dmihalcik-virtru · 2026-10-13T13:14:47Z

Summary

Removes the 10 MB default manifest cap from the SDK reader (defaultMaxManifestSize) and from zipstream (manifestMaxSize).
DSPX-4502 targets payloads up to 50 TiB, whose manifests grow past 1 GiB (policy size plus per-segment entries). No fixed default can serve that, and it silently broke loading of large TDFs.
Forged ZIP sizes do not need a cap: NewReader already checks every declared size and offset against the archive bounds (DSPX-4590, fix(sdk): bound zip64 reader offsets against the archive (DSPX-4590) #4043).
WithMaxManifestSize(n) stays as an opt-in memory bound; n must be > 0. WithTDFManifestMaxSize treats a non-positive size as no cap.

Behaviour change

LoadTDF and IsValidTdf now accept manifests over 10 MB by default. Callers that relied on the old default to bound memory on untrusted input should pass WithMaxManifestSize. Memory scaling of very large manifests is tracked separately in DSPX-4905.

Testing

Test_LargeManifest_WithMaxManifest: an 11 MB manifest loads with default options; an explicit limit still rejects larger ones; WithMaxManifestSize(0) is an error.
TestIsValidTdf_NoDefaultManifestSizeCap and new zipstream TDFReader tests cover the default and opt-in paths.
go test ./... in sdk and sdk/internal/zipstream; golangci-lint run ./... in sdk shows no findings in touched files. Root make lint and make test not yet run.

Split out of the DSPX-5102 hardening work; independent of #4212.

Summary by CodeRabbit

New Features
- TDF manifests larger than 10 MiB can now be read and validated with default settings, provided they fit within the archive’s bounds.
- Manifest size limits are now opt-in: configure a positive limit to cap the size. Zero or negative values do not impose an application-level limit.
Bug Fixes
- Manifests within a configured positive limit remain accepted, while those exceeding it are rejected.

Remove the 10 MB default manifest cap from the SDK reader (defaultMaxManifestSize) and zipstream (manifestMaxSize). A 50 TiB payload, the DSPX-4502 target, carries a manifest of more than 1 GiB, so no fixed default can serve it. A declared manifest size is already bounded by the archive itself in NewReader, so forged ZIP sizes are rejected without a cap. WithMaxManifestSize remains as an opt-in memory bound and must be > 0. Internally a non-positive zipstream limit means unlimited. Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

coderabbitai · 2026-10-13T13:15:00Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI
Review profile: ASSERTIVE
Plan: Advanced
Run ID: 60b2e3be-9c10-4c55-841b-1810e68248d0

📥 Commits

Reviewing files that changed from the base of the PR and between 4d9e555 and 8c65d7d.

📒 Files selected for processing (5)

sdk/internal/zipstream/tdf3_reader.go
sdk/internal/zipstream/tdf3_reader_test.go
sdk/manifest_size_test.go
sdk/tdf_config.go
sdk/tdf_test.go

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

The SDK no longer applies a default 10 MiB manifest-size limit. Nonpositive limits are uncapped, while positive limits remain enforced. Tests cover oversized manifests, configured boundaries, and malformed ZIP64 size declarations.

Changes: Manifest Size Limit

Layer / File(s)	Summary
Configure the manifest size limit `sdk/tdf_config.go`, `sdk/internal/zipstream/tdf3_reader.go`	The default 10 MiB limit is removed. The option documentation describes uncapped defaults and positive limits.
Apply limits and validate manifest sizes `sdk/internal/zipstream/tdf3_reader.go`, `sdk/internal/zipstream/tdf3_reader_test.go`, `sdk/manifest_size_test.go`, `sdk/tdf_test.go`	The reader uses an uncapped effective limit when the configured value is nonpositive. Tests cover uncapped reads, positive limit boundaries, oversized manifests, and malformed ZIP64 sizes.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Suggested reviewers: elizabethhealy

Merge Risk: ⚪ Minimal · up to 8c65d

No actionable merge-blocking issue was identified. The change is ready for normal checks.

Pre-merge checks · 5

Passed checks (5 passed)

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: making the SDK manifest size limit opt-in.
Docstring Coverage		Docstring coverage is 88.89% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 9 functions across 5 files.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{A rabbit reads a manifest wide,
With bytes that stretch from side to side.
A limit waits when set with care,
While uncapped pages fill the air.
The ZIP stays checked from end to end.}

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-10-13T13:20:44Z

Benchmark results, click to expand

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	422.342375ms
Throughput	236.77 requests/second

TDF3 Benchmark Results

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	45.803047604s
Average Latency	457.050135ms
Throughput	109.16 requests/second

github-actions · 2026-10-13T13:22:02Z

⚠️ Govulncheck found vulnerabilities ⚠️

The following modules have known vulnerabilities:

otdfctl
sdk
service
lib/fixtures
tests-bdd

See the workflow run for details.

github-actions · 2026-10-13T13:30:09Z

github-actions Bot added the comp:sdk A software development kit, including library, for client applications and inter-service communicati label Oct 13, 2026

github-actions Bot added the size/s label Oct 13, 2026

dmihalcik-virtru marked this pull request as ready for review October 13, 2026 13:43

dmihalcik-virtru requested review from a team as code owners October 13, 2026 13:43

sujankota approved these changes Oct 14, 2026

View reviewed changes

dmihalcik-virtru added this pull request to the merge queue Oct 14, 2026

Merged via the queue into main with commit 08ecb66 Oct 14, 2026
50 of 51 checks passed

dmihalcik-virtru deleted the DSPX-4502-manifest-cap-opt-in branch October 14, 2026 14:40

opentdf-automation Bot mentioned this pull request Oct 14, 2026

chore(main): release sdk 0.35.0 #4123

Merged

This was referenced Oct 14, 2026

chore(sdk): pin the #4202 ZIP64 MaxUint64 stored size in zipstream fuzz test (DSPX-5102) #4217

Merged

fix(sdk): reject malformed ZIP64 lengths and null assertion statements; refuse isEncrypted=false (DSPX-5102) #4207

Closed

Navigation Menu

Repository navigation

fix(sdk): make the manifest size limit opt-in (DSPX-4502) - #4214

fix(sdk): make the manifest size limit opt-in (DSPX-4502)#4214
dmihalcik-virtru merged 1 commit into
mainfrom
DSPX-4502-manifest-cap-opt-in

dmihalcik-virtru commented Oct 13, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 13, 2026

Bulk Benchmark Results

TDF3 Benchmark Results

Uh oh!

github-actions Bot commented Oct 13, 2026

Uh oh!

github-actions Bot commented Oct 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dmihalcik-virtru commented Oct 13, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Behaviour change

Testing

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 13, 2026

Bulk Benchmark Results

TDF3 Benchmark Results

Uh oh!

github-actions Bot commented Oct 13, 2026

⚠️ Govulncheck found vulnerabilities ⚠️

Uh oh!

github-actions Bot commented Oct 13, 2026

X-Test Results

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dmihalcik-virtru commented Oct 13, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 13, 2026 •

edited

Loading