Repository navigation
Bound chunked decoding: stream chunk data, cap the trailer section - #83
Merged
Merged
Conversation
ParserLimits caps every part of the header block, but nothing bounded a
chunked body's framing:
- ChunkedBodyStream returned NeedMoreData until a whole chunk was
buffered, so a peer that declares FFFFFFFF and keeps sending makes the
caller's buffer grow without limit.
- The trailer section had no cap at all. Endless "X: y" lines after the
last chunk bypass MaxTotalHeaderBytes, and because NeedMoreData rolls
back, the decoder asks for the whole section to be re-presented until
it ends - so it is buffered too.
Chunk data is now handed out as it arrives, in both the span and the
ReadOnlySequence overloads: a Chunk result carries a whole chunk or the
part of one that has arrived, and is never empty. Data that is complete
before its CRLF is handed out as well; the CRLF is checked on the next
call. NeedMoreData keeps its contract - nothing consumed, state rolled
back - and now only occurs when no data byte is available. Buffering is
bounded by the caller's read window instead of the declared size, which
is how the C decoder already behaves. Callers that append each Chunk's
bytes and advance by bytesConsumed (GenHTTP's ChunkedFraming does) need
no change.
The trailer section is capped by the new ParserLimits.MaxTotalTrailerBytes
(default 16 KiB): every trailer field line after the last chunk, including
its CRLF. Exceeding it throws HttpParseException with status 431 from the
span overload, the sequence overload and ChunkedBodyValidator. The bytes
seen so far are checked even while the section is incomplete, so an
endless section is refused once it passes the cap rather than buffered; a
CR at the very end of the input is not counted yet, since it may be the
first half of the final CRLF. ChunkedBodyStream takes the limits through a
new constructor and ChunkedBodyValidator through a new TryValidate
overload; new ChunkedBodyStream() and the existing overload apply
ParserLimits.Default.
No MaxChunkSize: with data streamed, a per-chunk cap would only reject
legitimate large chunks - the total body size is the application's limit.
Fixes #63.
The Bodies source returned untyped object[] rows (xUnit1042). Same cases.
The two limit fields grew ChunkedBodyStream from 32 to 40 bytes, and the whole state is copied on every call (the NeedMoreData rollback snapshot). They now sit in the padding after the byte-sized phase, so the state is 32 bytes again, as on main. The ReadOnlySequence overload also tested ReadOnlySequence.IsEmpty on every completed chunk; a local flag set where the data is sliced does the same job without computing a length. Together these were most of a ~7% slowdown in multi-segment chunked decoding against main. Behaviour is unchanged.
Every TryReadChunk call copies the whole state for the NeedMoreData rollback, so its size is on the hot path. A test now fails if a field grows it past 32 bytes again.
Kaliumhexacyanoferrat
pushed a commit
to Kaliumhexacyanoferrat/GenHTTP
that referenced
this pull request
Nov 4, 2026
* Update Glyph11 to 0.4.0 and parse with TryParseRequestHeader Glyph11 0.4.0 marks TryExtractFullHeaderValidated and TryExtractFullHeaderROM [Obsolete]: they report one less than the length of the header block, which is why every call site here sliced at bytesRead + 1. TryParseRequestHeader reports the length itself, so the body starts at consumed. With TreatWarningsAsErrors the update does not build without the move, and the rename and the dropped + 1 have to land together - renaming alone would skip the first byte of every body or pipelined request. Both engines, the body lifecycle tests and the benchmark context move. Glyph11.Pico stays at 0.0.1: PicoParser.TryParse keeps its convention, so the Pico path keeps its + 1. Closes #972. * Expect 400 for a malformed HTTP version, 505 for an unsupported one Glyph11 0.4.0 (dotnet-web-stack/Glyph11#79) splits what it used to report as one 505: a version that matches HTTP-version ("HTTP/" DIGIT "." DIGIT) but is not 1.0/1.1 stays a 505, "Unsupported HTTP version.", while bytes that are not an HTTP-version at all make the request line invalid, a 400 (RFC 9112 2.3 and 3). "GET / HTTP/1" has no minor version, so it is malformed rather than unsupported, which is also what http-probe's COMP-VERSION-MISSING-MINOR expects. HTTP/2.0 keeps its 505, under the new message. * Expect a truncated chunked body to hand out its data before failing Glyph11 0.4.0 (dotnet-web-stack/Glyph11#83) hands chunk data out as it arrives instead of waiting for the whole chunk, so a peer that declares a huge chunk can no longer make the connection buffer it. For "5\r\nHel" the first read now returns the three bytes that did arrive, and the truncation surfaces as the 400 ProviderException on the next read - the point where the body actually ends early. * Drop the Glyph11.Pico dependency Glyph11 discontinues Glyph11.Pico and Glyph11.Native (dotnet-web-stack/Glyph11#84): they get no further releases. The Ioxide engine's GENHTTP_IOXIDE_PARSER=pico switch goes with the package, so requests are always parsed by UltraHardenedParser, and the body lifecycle tests that need unvalidated framing headers get them from Glyph11's FlexibleParser instead of picohttpparser.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #63
Why
ParserLimitscaps the header block, but nothing caps a chunked body's framing:ChunkedBodyStreamreturnsNeedMoreDatauntil a whole chunk is in the buffer. A peer that declaresFFFFFFFFand keeps sending makes the caller's buffer grow without limit.X: ylines after the last chunk bypassMaxTotalHeaderBytes. BecauseNeedMoreDatarolls back, the decoder also asks for the whole section to be re-presented until it ends, so the trailers get buffered too.What changes
Chunk data is handed out as it arrives (span and
ReadOnlySequenceoverloads):Chunkresult now carries a whole chunk or the part of one that has arrived. It is never empty.Missing CRLF after chunk data..NeedMoreDatakeeps its contract: nothing is consumed, the state is rolled back, and the caller re-presents the same bytes. It now only happens when not a single data byte is available, for example a size line with no data after it yet.The trailer section is capped:
ParserLimits.MaxTotalTrailerBytes, default 16 KiB. It counts every trailer field line after the last chunk, including its CRLF.HttpParseException("Chunked trailer section exceeds limit.", 431)from both stream overloads and fromChunkedBodyValidator.ChunkedBodyStream(in ParserLimits)constructor and aChunkedBodyValidator.TryValidate(body, in ParserLimits, out consumed)overload.new ChunkedBodyStream(),default(ChunkedBodyStream)and the existingTryValidateoverload applyParserLimits.Default, so existing callers get the cap without code changes.No cost on the common path:
main. It is copied on every call for theNeedMoreDatarollback.ReadOnlySequenceoverload marks sliced chunk data with a flag instead of testingReadOnlySequence.IsEmptyon every chunk.main, multi-segment chunked decoding was about 7% slower without these two changes and is within noise with them.Docs: the limits tables in
SECURITY.mdand on the site, the chunked section of the site (itsNeedMoreDatacomment said "partial chunk"), and one line in the README.Decisions worth reviewing
Chunkno longer means "one whole chunk". Chunk boundaries carry no meaning in HTTP, and callers that append eachChunk's bytes and advance bybytesConsumedneed no change. GenHTTP'sChunkedFramingis one: it copiesdataand advances toconsumedon the sequence overload, so partial chunks pass through it unchanged. Code that assumed oneChunkper wire chunk would see more results. I found no such code. One GenHTTP test did pin whole-chunk buffering:ChunkedFramingTests.TestTruncatedBodyFailsexpected the first read of5\r\nHelto fail. It now gets the three bytes first and the 400 on the next read. chore(engines): Update Glyph11 to 0.4.0 and drop Glyph11.Pico Kaliumhexacyanoferrat/GenHTTP#973 adapts it.MaxChunkSize, although the issue mentions one. Once data is streamed, a per-chunk cap only rejects legitimate large chunks. Buffering is already bounded, and the total body size is the application's limit.MaxTotalTrailerBytes = 0means "no trailers":0\r\n\r\nis accepted and any trailer line is refused. This matches how the otherParserLimitsfields treat 0. Only a decoder built without limits falls back to the default.Overlap with sibling PRs
ChunkSizephase rather thanChunkData/Trailers. A trial merge of both branches is clean and passes the combined suite (474 tests).ParserLimits(AllowBareLf), soParserLimits.cswill need a trivial merge.Tests
New
tests/Tests/ChunkedBodyStream.Limits.cs(44 tests):Chunkmust be non-empty,NeedMoreDatamust consume nothing, and the decoded body must match.FFFFFFFFwith a few data bytes yields aChunk, notNeedMoreData. A size line alone still yieldsNeedMoreDatawith nothing consumed.default(ChunkedBodyStream). Custom cap (64), a zero cap, an incomplete section under the cap, an at-cap section whose final CRLF is split, an endless single line, and endless short lines are all covered. The endless lines are fed through a 512-byte read loop and refused within one read of crossing the cap.Three existing tests pinned whole-chunk buffering and were updated:
Incremental_ChunkDataAcrossCallsnow expectsHethenllo.Incremental_MultiChunkByteByByteasserts the reassembled body rather than two chunks.Sequence_NeedMoreData_RollsBack_ThenCompletesuses a prefix that ends before any data, so it still covers the rollback.Full suite: 447 passing (403 before).
SonarCloud
S3776 (cognitive complexity) is reported on the new
TryValidate(body, in ParserLimits, out int)overload. It is the validator's existing issue: the implementation now sits under the new signature, and the trailer cap adds to it (109 → 118).