Repository navigation
[Question] ResumableParser: recommended way to detect a truncated stream at EOF? #1047
Description
In Fluentd we are migrating our JSON stream parsing from yajl-ruby to JSON::ResumableParser (fluent/fluentd#5383). One behavior we want to preserve: when an input stream ends in the middle of a JSON document, the consumer should be able to notice it (Yajl raised a ParseError in that situation; we now emit a warning instead of silently dropping the incomplete tail).
Problem
There doesn't seem to be a single API that answers "did the stream end mid-document?". Our first attempt used rest.empty?, but rest only reflects unconsumed tokenizer bytes, so a stream cut on a complete token boundary (right after a : or ,) leaves rest empty and eos? returning true, even though a document is clearly in progress:
parser = JSON::ResumableParser.new
parser << '{"a":1,'
parser.value while parser.parse
parser.rest # => ""
parser.eos? # => true (but the document is incomplete and gets silently dropped)
parser.partial_value # => {"a"=>1}Reproduction
require 'bundler/inline'
gemfile do
source 'https://rubygems.org'
gem 'json', '= 2.20.0'
end
CASES = [
['{"a":1}', false], # [input, truncated?]
['{"a":1}{"b":2}', false],
['{"a":1} ', false],
['{"a":1}{"b":2', true], # cut inside a number token
['{"a":1}{"b":', true], # cut right after a colon (token boundary)
['{"a":1}{', true], # cut right after object open
['{"a":1,', true], # cut right after a comma (token boundary)
]
CASES.each do |input, truncated|
parser = JSON::ResumableParser.new
parser << input
parser.value while parser.parse
detected = !parser.rest.empty? || !parser.partial_value.nil?
puts "%-18s rest=%-6s eos?=%-5s partial_value=%-12s detected=%s%s" % [
input.inspect, parser.rest.inspect, parser.eos?,
parser.partial_value.inspect, detected,
detected == truncated ? "" : " <-- MISMATCH"
]
endOutput on json 2.20.0 (on Ruby 4.0.5):
"{\"a\":1}" rest="" eos?=true partial_value=nil detected=false
"{\"a\":1}{\"b\":2}" rest="" eos?=true partial_value=nil detected=false
"{\"a\":1} " rest="" eos?=true partial_value=nil detected=false
"{\"a\":1}{\"b\":2" rest="2" eos?=false partial_value={"b"=>nil} detected=true
"{\"a\":1}{\"b\":" rest="" eos?=true partial_value={"b"=>nil} detected=true
"{\"a\":1}{" rest="{" eos?=false partial_value=nil detected=true
"{\"a\":1," rest="" eos?=true partial_value={"a"=>1} detected=true
What we ended up with
rest and partial_value turn out to be complementary (rest catches the mid-token and just-after-{ cuts, partial_value catches the token-boundary cuts), so we currently use:
truncated = !parser.rest.empty? || !parser.partial_value.nil?Questions
- Is this combination the intended way to detect truncation, and is it something we can rely on across versions? Both behaviors were established empirically, and building a
partial_valueobject just for a nil check feels like a workaround. - Is
eos?returningtruewhile a document is structurally in progress ('{"a":1,') the intended semantics? Ifeos?is meant to be tokenizer-level only, a doc note would help; if it's meant to answer "is the parser at a clean end of stream?", this looks like it should returnfalse. - Would you be open to a dedicated predicate (e.g.
incomplete?/in_document?) for this?
Activity
byroot commentedon Jul 3, 2026
Not at all.
Yes,
eos?is meant to tell you whether there are bytes left in the buffer, nothing more.Absolutely. Yes.
byroot commentedon Jul 3, 2026
So, just trying to think of the API with the most usability. Assuming a stream parsing fluentd style:
Watson1978 commentedon Jul 3, 2026
I like this shape. Inverting the polarity to
complete?also resolves the mid-drain ambiguity discussed in #1048 more elegantly than a rename on the negative side would: when a complete document is still sitting unconsumed in the buffer,complete? == falsereads naturally as "not done processing yet", whereasincomplete? == truemisleadingly suggested truncation.So the semantics would be:
complete?returns true iff all fed bytes have been consumed and no document is in progress — i.e., the stream so far ends cleanly on a document boundary (vacuously true for an empty stream). Your example above works as-is, and mid-drain it correctly answers false:I'll update #1048 to rename
incomplete?→complete?with inverted semantics, pin both your examples as tests, and use your loop as the canonical usage example in the rdoc. One question: for theloop/EOFErrorpattern in your sketch, would you want the rdoc to show logging/raising on!parser.complete?as the recommended EOF handling, or leave that to the caller?