Skip to content

Bound chunked decoding: stream chunk data, cap the trailer section - #83

Merged
MDA2AV merged 4 commits into
mainfrom
fix/63-chunked-limits
Nov 3, 2026
Merged

MDA2AV merged 4 commits into
mainfrom
fix/63-chunked-limits

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Fixes #63

Why

ParserLimits caps the header block, but nothing caps a chunked body's framing:

  • (a) ChunkedBodyStream returns NeedMoreData until a whole chunk is in the buffer. A peer that declares FFFFFFFF and keeps sending makes the caller's buffer grow without limit.
  • (b) The trailer section has no cap. Endless X: y lines after the last chunk bypass MaxTotalHeaderBytes. Because NeedMoreData rolls back, the decoder also asks for the whole section to be re-presented until it ends, so the trailers get buffered too.

What changes

Chunk data is handed out as it arrives (span and ReadOnlySequence overloads):

  • A Chunk result now carries a whole chunk or the part of one that has arrived. It is never empty.
  • Data that is complete before its CRLF is handed out too; the CRLF is checked on the next call, and a bad one still throws Missing CRLF after chunk data..
  • NeedMoreData keeps its contract: nothing is consumed, the state is rolled back, and the caller re-presents the same bytes. It now only happens when not a single data byte is available, for example a size line with no data after it yet.
  • Buffering is bounded by the caller's read window, not by the declared size. The C decoder already works this way.

The trailer section is capped:

  • New ParserLimits.MaxTotalTrailerBytes, default 16 KiB. It counts every trailer field line after the last chunk, including its CRLF.
  • Exceeding it throws HttpParseException("Chunked trailer section exceeds limit.", 431) from both stream overloads and from ChunkedBodyValidator.
  • The bytes seen so far are checked even while the section is incomplete, so an endless section is refused once it passes the cap instead of being re-presented forever.
  • A CR at the very end of the input is not counted yet, since it may be the first half of the final CRLF. Without that, a section exactly at the cap whose last CRLF arrived split across reads would be refused by mistake.
  • Limits are passed in two new ways: a ChunkedBodyStream(in ParserLimits) constructor and a ChunkedBodyValidator.TryValidate(body, in ParserLimits, out consumed) overload. new ChunkedBodyStream(), default(ChunkedBodyStream) and the existing TryValidate overload apply ParserLimits.Default, so existing callers get the cap without code changes.

No cost on the common path:

  • The two new fields sit in the padding after the byte-sized phase, so the decoder state stays 32 bytes, as on main. It is copied on every call for the NeedMoreData rollback.
  • The ReadOnlySequence overload marks sliced chunk data with a flag instead of testing ReadOnlySequence.IsEmpty on every chunk.
  • In pinned, alternating BenchmarkDotNet rounds against main, multi-segment chunked decoding was about 7% slower without these two changes and is within noise with them.

Docs: the limits tables in SECURITY.md and on the site, the chunked section of the site (its NeedMoreData comment said "partial chunk"), and one line in the README.

Decisions worth reviewing

  • Chunk no longer means "one whole chunk". Chunk boundaries carry no meaning in HTTP, and callers that append each Chunk's bytes and advance by bytesConsumed need no change. GenHTTP's ChunkedFraming is one: it copies data and advances to consumed on the sequence overload, so partial chunks pass through it unchanged. Code that assumed one Chunk per wire chunk would see more results. I found no such code. One GenHTTP test did pin whole-chunk buffering: ChunkedFramingTests.TestTruncatedBodyFails expected the first read of 5\r\nHel to fail. It now gets the three bytes first and the 400 on the next read. chore(engines): Update Glyph11 to 0.4.0 and drop Glyph11.Pico Kaliumhexacyanoferrat/GenHTTP#973 adapts it.
  • No MaxChunkSize, although the issue mentions one. Once data is streamed, a per-chunk cap only rejects legitimate large chunks. Buffering is already bounded, and the total body size is the application's limit.
  • An explicit MaxTotalTrailerBytes = 0 means "no trailers": 0\r\n\r\n is accepted and any trailer line is refused. This matches how the other ParserLimits fields treat 0. Only a decoder built without limits falls back to the default.

Overlap with sibling PRs

Tests

New tests/Tests/ChunkedBodyStream.Limits.cs (44 tests):

  • Byte-at-a-time and random-split reassembly for both overloads over several bodies (extensions, trailers, a 1000-byte chunk, 20 random-size chunks). Each Chunk must be non-empty, NeedMoreData must consume nothing, and the decoded body must match.
  • A 1 MiB chunk read 4 KiB at a time never has more than one read window buffered.
  • FFFFFFFF with a few data bytes yields a Chunk, not NeedMoreData. A size line alone still yields NeedMoreData with nothing consumed.
  • Data that is complete before its CRLF (none of the CRLF yet, or only the CR) is handed out, and the CRLF is validated next call.
  • Trailer section at 0, 100, cap−1 and cap completes. Cap+1 and 4×cap are refused with 431 for the span overload, the sequence overload (1, 3 and 17 segments), the validator and default(ChunkedBodyStream). Custom cap (64), a zero cap, an incomplete section under the cap, an at-cap section whose final CRLF is split, an endless single line, and endless short lines are all covered. The endless lines are fed through a 512-byte read loop and refused within one read of crossing the cap.
  • The decoder state is pinned at 32 bytes, so a field that grows the per-call rollback copy fails a test.

Three existing tests pinned whole-chunk buffering and were updated:

  • Incremental_ChunkDataAcrossCalls now expects He then llo.
  • Incremental_MultiChunkByteByByte asserts the reassembled body rather than two chunks.
  • Sequence_NeedMoreData_RollsBack_ThenCompletes uses a prefix that ends before any data, so it still covers the rollback.

Full suite: 447 passing (403 before).

SonarCloud

S3776 (cognitive complexity) is reported on the new TryValidate(body, in ParserLimits, out int) overload. It is the validator's existing issue: the implementation now sits under the new signature, and the trailer cap adds to it (109 → 118).

ParserLimits caps every part of the header block, but nothing bounded a
chunked body's framing:

  - ChunkedBodyStream returned NeedMoreData until a whole chunk was
    buffered, so a peer that declares FFFFFFFF and keeps sending makes the
    caller's buffer grow without limit.
  - The trailer section had no cap at all. Endless "X: y" lines after the
    last chunk bypass MaxTotalHeaderBytes, and because NeedMoreData rolls
    back, the decoder asks for the whole section to be re-presented until
    it ends - so it is buffered too.

Chunk data is now handed out as it arrives, in both the span and the
ReadOnlySequence overloads: a Chunk result carries a whole chunk or the
part of one that has arrived, and is never empty. Data that is complete
before its CRLF is handed out as well; the CRLF is checked on the next
call. NeedMoreData keeps its contract - nothing consumed, state rolled
back - and now only occurs when no data byte is available. Buffering is
bounded by the caller's read window instead of the declared size, which
is how the C decoder already behaves. Callers that append each Chunk's
bytes and advance by bytesConsumed (GenHTTP's ChunkedFraming does) need
no change.

The trailer section is capped by the new ParserLimits.MaxTotalTrailerBytes
(default 16 KiB): every trailer field line after the last chunk, including
its CRLF. Exceeding it throws HttpParseException with status 431 from the
span overload, the sequence overload and ChunkedBodyValidator. The bytes
seen so far are checked even while the section is incomplete, so an
endless section is refused once it passes the cap rather than buffered; a
CR at the very end of the input is not counted yet, since it may be the
first half of the final CRLF. ChunkedBodyStream takes the limits through a
new constructor and ChunkedBodyValidator through a new TryValidate
overload; new ChunkedBodyStream() and the existing overload apply
ParserLimits.Default.

No MaxChunkSize: with data streamed, a per-chunk cap would only reject
legitimate large chunks - the total body size is the application's limit.

Fixes #63.
The Bodies source returned untyped object[] rows (xUnit1042). Same
cases.
The two limit fields grew ChunkedBodyStream from 32 to 40 bytes, and
the whole state is copied on every call (the NeedMoreData rollback
snapshot). They now sit in the padding after the byte-sized phase, so
the state is 32 bytes again, as on main.

The ReadOnlySequence overload also tested ReadOnlySequence.IsEmpty on
every completed chunk; a local flag set where the data is sliced does
the same job without computing a length.

Together these were most of a ~7% slowdown in multi-segment chunked
decoding against main. Behaviour is unchanged.
Every TryReadChunk call copies the whole state for the NeedMoreData
rollback, so its size is on the hot path. A test now fails if a field
grows it past 32 bytes again.
@MDA2AV
MDA2AV merged commit 1f14be1 into main Nov 3, 2026
1 of 2 checks passed
Kaliumhexacyanoferrat pushed a commit to Kaliumhexacyanoferrat/GenHTTP that referenced this pull request Nov 4, 2026
* Update Glyph11 to 0.4.0 and parse with TryParseRequestHeader

Glyph11 0.4.0 marks TryExtractFullHeaderValidated and TryExtractFullHeaderROM
[Obsolete]: they report one less than the length of the header block, which
is why every call site here sliced at bytesRead + 1. TryParseRequestHeader
reports the length itself, so the body starts at consumed.

With TreatWarningsAsErrors the update does not build without the move, and the
rename and the dropped + 1 have to land together - renaming alone would skip
the first byte of every body or pipelined request. Both engines, the body
lifecycle tests and the benchmark context move. Glyph11.Pico stays at 0.0.1:
PicoParser.TryParse keeps its convention, so the Pico path keeps its + 1.

Closes #972.

* Expect 400 for a malformed HTTP version, 505 for an unsupported one

Glyph11 0.4.0 (dotnet-web-stack/Glyph11#79) splits what it used to report
as one 505: a version that matches HTTP-version ("HTTP/" DIGIT "." DIGIT)
but is not 1.0/1.1 stays a 505, "Unsupported HTTP version.", while bytes
that are not an HTTP-version at all make the request line invalid, a 400
(RFC 9112 2.3 and 3).

"GET / HTTP/1" has no minor version, so it is malformed rather than
unsupported, which is also what http-probe's COMP-VERSION-MISSING-MINOR
expects. HTTP/2.0 keeps its 505, under the new message.

* Expect a truncated chunked body to hand out its data before failing

Glyph11 0.4.0 (dotnet-web-stack/Glyph11#83) hands chunk data out as it
arrives instead of waiting for the whole chunk, so a peer that declares a
huge chunk can no longer make the connection buffer it. For "5\r\nHel" the
first read now returns the three bytes that did arrive, and the truncation
surfaces as the 400 ProviderException on the next read - the point where
the body actually ends early.

* Drop the Glyph11.Pico dependency

Glyph11 discontinues Glyph11.Pico and Glyph11.Native
(dotnet-web-stack/Glyph11#84): they get no further releases. The Ioxide
engine's GENHTTP_IOXIDE_PARSER=pico switch goes with the package, so
requests are always parsed by UltraHardenedParser, and the body
lifecycle tests that need unvalidated framing headers get them from
Glyph11's FlexibleParser instead of picohttpparser.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chunked decoding has no resource ceiling: unbounded chunk buffering and unlimited trailers

1 participant