Add support for parsing chunked data

This is similar to #854 but a different feature request. Both of #854 and this are for migrating to the json gem from yajl-ruby (in Fluentd. #854's description doesn't mention it but it's mentioned in a comment #854 (comment) ) but this proposes different API. We can't migrate from yajl-ruby with API proposed in #854.

#854 proposed parser.parse(io) like API but what we need is io.each_chunk {|chunk| parser.feed(chunk)} like API. An application wants to read data not the json gem read data. parser.parse(io) like API reads data by the json gem. Fluentd can't migrate to yajl-ruby with the API.

Fluentd that is a event based log collector reads data from multiple sockets efficiently by IO multiplexing. We may not be able to read a whole JSON (e.g. ["a", 2, true]) at a time with the IO multiplexing. We may get a whole JSON as multiple chunks (e.g. ["a", , 2, and true]). yajl-ruby can parse chunks without concatenating all of them:

yajl-ruby:

require "yajl"
parser = Yajl::Parser.new
parser.on_parse_complete = lambda {|x| pp x}
parser << "[\"a\""
parser << ", 2, "
parser << "true]" # lambda {|x| pp x} is called with ["a", 2, true]

The json gem:

require "json"
JSON.parse(["[\"a\"", ", 2, ", "true]"].join)

If an application can extract a whole JSON from an input without parsing input (*), we don't need yajl-ruby style API. But an input may have multiple JSONs (like Fluentd's use case), we need yajl-ruby style API. Because we need to parse JSON to split data in an input to JSONs for this case.

(*) For example:

Only one JSON exists in an input
Each JSON is delimited by new line (JSON lines)
...

API

JSON::Parser.new requires a source. So it may be better that we define a new class such as JSON::StreamParser instead of reusing existing JSON::Parser.

JSON::StreamParser accepts a chunk not a whole JSON:

parser = JSON::StreamParser.new
parser.parse_chunk("[\"a\"")
parser.parse_chunk(", 2, ")
parser.parse_chunk("true]")

Other candidate for parse_chunk: consume

We need to define callback style API to pass parsed data to application. Because one parse_chunk call may parse 0 or more JSONs. For example, if chunk is [1][2], [1] and [2] are parsed at once.

JSON::Parser accepts on_load callback. We may resuse the name for JSON::StreamParser but on_load is for JSON value not JSON. It may be better that we use different name. For example:

parser = JSON::StreamParser.new
parser.on_load = lambda {|x| pp x}
parser.parse_chunk("[\"a\"")
parser.parse_chunk(", 2, ")
parser.parse_chunk("true]") # lambda {x| pp x} is called with ["a", 2, true]
parser.parse_chunk("[1][2]") # lambda {x| pp x} is called twice with [1] and [2]

Thanks, I better understand the ask now (the streaming term is used for different things in JSON parsing context).

I'm open to such feature, as I think it can be useful, however I suspect it would require a fairly large refactoring of the parser.

Right now the parser use recursive C function calls, so it can't really be interrupted or resumed.

So the parser would first need to be changed quite a bit to look more like msgpack's parser: https://github.com/msgpack/msgpack-ruby/blob/677e6b888f1512550d9b3cdb4b5aca28f9fdb859/ext/msgpack/unpacker.c, which is something I considered in the past, but didn't act on.

The best approach is likely to keep a dedicated stack to represent the parser state. (e.g. .stack = { ARRAY, ARRAY, OBJECT, OBJECT_KEY... }).

And that's assuming such change wouldn't degrade performance too much, if it does, then we'd need to consider whether we have to maintain two parsers so that the classic usage keep performing well.

As for the API, I think I'd prefer something that doesn't use callback, like MessagePack:

parser = JSON::StreamParser.new
parser.feed("[\"a\"")
parser.read # JSON::EOFError

parser.feed(", 2, ")
parser.read # JSON::EOFError

parser.feed("true]")
parser.read # => ["a", 2, true]

parser.feed("[1][2]")
parser.read # => [1]
parser.read # => [2]

But perhaps exceptions aren't ideal. The problem is we can't just return nil given that null is a valid document in some contexts. So perhaps something like:

parser = JSON::StreamParser.new
parser.feed("[\"a\"")
parser.read? # => false

parser.feed(", 2, ")
parser.read? # => false

parser.feed("true]")
parser.read? # => true
parser.read # => ["a", 2, true]

parser.feed("[1][2]")
parser.read? # => true
parser.read # => [1]

parser.read? # => true
parser.read # => [2]

parser.read? # => false

Also this is just for the C parser, which I think can reasonably be converted to this new architecture. The JRuby parser still uses ragel and would be significantly more work. But I'd be OK with only having this feature in the C extension until someone offer to make significant changes to the Jave version.

This sounds likely to degrade performance significantly to me and result in a much more complicated parser.
Maybe one approach could be to basically:

buf = +""
buf << "[\"a\""
json = JSON.parse(buf) rescue nil # => nil
buf << ", 2, "
json = JSON.parse(buf) rescue nil # => nil
buf << "true]"
json = JSON.parse(buf) rescue nil # => ["a", 2, true]

That would also require the input to be given in chunks that terminate, would that be enough?
If not maybe JSON could have a mode where it accepts extra trailing bytes, and give a way to know how many bytes it parsed so we can restart parsing after that.
That would be needed for the case of buf = "[1][2]".

Another thought is if this is strict JSON with only native JSON types a Regexp (using Subexpression Calls: \g) might be enough to:

Find if an input is a complete JSON expression
And if so find the end of it

This sounds likely to degrade performance significantly to me and result in a much more complicated parser.

Not necessarily. It's an idea I toyed with for a while, but ultimately it just replace direct calls with a stack based dispatch, not that different from a bytecode interpreter.

Whether it can be as fast is hard to be certain until it's implemented, and it's a significant amount of work so...

My worry would be more about the mere concept of a significant rewrite, as it's potential for new bugs and such.

As for your suggestion, it may be fine if you only occasionally have partial documents, but from my limited understading of fluentd, it's a very common case, so having to restart parsing from 0 every time would be prohibitive I fear.

I originally proposed the streaming dumping and I agree this would be good to add.

That would also require the input to be given in chunks that terminate, would that be enough?

It's suboptimal. It parses one JSON multiple times. If all use cases don't require performance, it's acceptable. But Fluentd is a log collector and log collector needs performance because it needs to process many logs.

Not necessarily. It's an idea I toyed with for a while, but ultimately it just replace direct calls with a stack based dispatch, not that different from a bytecode interpreter.

Using that analogy the current version has no dispatch, like JITed Ruby code, and this would have dispatch overhead, like a bytecode interpreter. So that seems pretty clear there will be some overhead. Also it means the CPU can't predict the next function to be called, it will be just a function pointer I guess and that's well known to be slower.

Maybe it's worth asking the reverse question, what are the problems with yajl? It seems purpose-built for this, so seems a good fit, no? Most usages of JSON don't need this, but supporting it would slow parsing down and add significant complexity.

yajl-ruby may not be maintained actively. See also: https://github.com/brianmario/yajl-ruby

Yeah, yajl has maintenance concerns.

So that seems pretty clear there will be some overhead.

I suspect it would be negligible with the proper optimizations (direct threading & al), especially since from experience the dispatch really isn't where the most time is spent.

But regardless, if it's indeed a perf regression, it's always possible to have two parsers that would share a lot of code.

Most usages of JSON don't need this,

Indeed, which is my biggest hesitation to be honest. So far my stance has been that I wanted to support the 95% use case, and not grow the API too much, leaving more specialized use cases to alternative gems.

Something I wonder, and sorry if it's obvious because I know very little about fluentd, is why documents can't be separated by a newline or some other marker? Are the JSON blobs purely black boxes and you can't assume they're on a single line? Not trying to say you are holding it wrong, mostly just curious.

is why documents can't be separated by a newline or some other marker?

Because Fluentd's Forward Protocol Specification v1 https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1 didn't define it... There are tools that implement this specification. So Fluentd can't assume that JSONs are separated by something to keep backward compatibility.

I have a JSON parser that doesn't use recursion here: https://gist.github.com/kddnewton/3e90d0dceda633641933e98bd4ee8694. The original commit was what I pushed up for the JSON rewrite, I don't mind porting the second commit here so we can compare perf.

No concerns with chunked parser from the JRuby side. Dispatches from Java back into Ruby will soon be inlinable and should not impact performance.

A rewrite for JRuby may be coming soon.

This starts the work: #989 in case you're interested. Feel free to close otherwise.

@byroot

I will be submitting a PR for a new JRuby JSON parser this week. At worst, a new parser will need to be added for streaming but it will use 95+% of the same logic and just need to do a little extra tracking and it might not because the new parser uses a stack like @byroot mentions above with state like:

    enum ParserState {
        ELEMENT,        // Start of document (also to never have empty stack)
        OBJECT_KEY,     // "key"
        OBJECT_COLON,   // ':'
        OBJECT_VALUE,   // (string, number, object, array, true, false, null)
        OBJECT_COMMA,   // ','
        ARRAY_VALUE,    // (string, number, object, array, true, false, null)
        ARRAY_COMMA,    // ','
    }

in a stack:

    protected static class StateElement {
        ParserState state;
        IRubyObject key;
        Object value;
    }

So at a fail it knows if state is at OBJECT_COMMA, then it is expecting a ','. So the next entry into parse might just continue where it left off. Anyways, just pointing out I do not think this will be a big problem with the new parser.

@kddnewton

Alright, big thanks to @kddnewton for #989, after a few tweaks (mostly adding gotos) it's on par with the previous recursive implementation (sometimes show up to 3-4% slower, but it's within the variance, and honestly I'm OK to take a small hit to no longer be limited by the C stack).

That however is just the first step. The parser is now iterative, but there's still some substantial work to do to make it resumable.

Implement a new class (e.g. JSON::StreamParser or something, not yet sold on the name)
With bufferring.
Then the parsing loop need to cleanly return on EOF instead of raising.
- For example, parsing fal<EOS> right now raises unexpected token 'fal'
- For a stream parser, we'd need to cleanly return, and signaling that it may be valid JSON once we get more bytes.
- Whereas far<EOS> should directly raise a syntax error.
There's also the question of what to do when a syntax error is reached?
- I don't think we can possibly skip to the next valid document.
- I'll have to check what YAJL does.
- Edit: It does nothing in particular, once a syntax error is reached, there's no way to recover it seems.

All those edge cases are unit tests, except for the parser recovery. Does YAJL has a way to recover?

Thanks.

No, yajl-ruby does not have a built-in way to recover the same parser instance.
Once it encounters an invalid JSON chunk, its internal state machine halts. To recover from a Yajl::ParseError in our old implementation, we had to discard the dead parser and instantiate a completely new Yajl::Parser object to continue parsing the rest of the stream.

But, if we can use JSON::ResumableParser#clear, it allows us to gracefully drop the malformed buffer and reset the state without needing to reallocate the parser instance.
I think it is useful.

o recover from a Yajl::ParseError in our old implementation, we had to discard the dead parser and instantiate a completely new Yajl::Parser object to continue parsing the rest of the stream.

I see.

.clear will be a bit cheaper, and will keep the various allocated capacities, but the difference should be minor.

Performance Comparison: `yajl-ruby` vs `JSON::ResumableParser`

I conducted a performance test to evaluate the efficiency of JSON::ResumableParser compared to the current yajl-ruby implementation.

Test Scenario

A client sends a continuous stream of ~1MB JSON messages via TCP at fixed 100ms intervals (approx. 10 MB/s).
Monitored CPU and Memory (RSS) usage of the receiver during processing.

Results (Handling ~10 MB/s constant load)

yajl-ruby
- CPU Usage: ~70% (Average)
- Memory (Max RSS): ~126 MB
JSON::ResumableParser
- CPU Usage: ~5% (Average)
- Memory (Max RSS): ~127 MB

Ref. fluent/fluentd#5383 (comment)

Wow, 14 times less CPU usage seem too good to be true.

# frozen_string_literal: true

require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'benchmark-ips'
  gem 'yajl-ruby', require: 'yajl'
  gem 'json', git: 'https://github.com/ruby/json.git'
end

require 'securerandom'

def generate_payload
  msg = [
    "test.memory",
    Time.now.to_i,
    {
      "id" => SecureRandom.uuid,
      "message" => "This is a memory test for JSON::ResumableParser",
      "dummy_data" => ('a'..'z').to_a.sample * 1024 * 1024
    }
  ]
  JSON.generate(msg)
end

payload = generate_payload

Benchmark.ips do |x|
  x.time = 10

  yajl_parser = Yajl::Parser.new
  yajl_parser.on_parse_complete = ->(record){ record }

  x.report("yajl-ruby") do
    yajl_parser << payload
  end

  json_parser = JSON::ResumableParser.new({})
  x.report("JSON::ResumableParser") do
    json_parser << payload
    while json_parser.parse
      json_parser.value
    end
  end

  x.compare!
end

$ ruby benchmark.rb
ruby 4.0.5 (2026-05-20 revision 64336ffd0e) +PRISM [x86_64-linux]
Warming up --------------------------------------
            yajl-ruby   178.000 i/100ms
JSON::ResumableParser     1.154k i/100ms
Calculating -------------------------------------
            yajl-ruby      1.756k (± 2.7%) i/s  (569.58 μs/i) -     17.622k in  10.037184s
JSON::ResumableParser     11.318k (± 4.4%) i/s   (88.36 μs/i) -    114.246k in  10.094253s

Comparison:
JSON::ResumableParser:    11317.9 i/s
            yajl-ruby:     1755.7 i/s - 6.45x  slower

Yes, not saying I don't believe you, I'm just always a bit careful when I get huge difference like this, as I got bit by it a few time in the past.

On my mac/M3 the difference is 17x:

ruby 4.0.5 (2026-05-20 revision 64336ffd0e) +YJIT +PRISM [arm64-darwin25]
Warming up --------------------------------------
            yajl-ruby    91.000 i/100ms
JSON::ResumableParser     1.729k i/100ms
Calculating -------------------------------------
            yajl-ruby    991.559 (± 1.2%) i/s    (1.01 ms/i) -      9.919k in  10.003442s
JSON::ResumableParser     17.324k (± 1.5%) i/s   (57.72 μs/i) -    174.629k in  10.080112s

Comparison:
JSON::ResumableParser:    17324.1 i/s
            yajl-ruby:      991.6 i/s - 17.47x  slower

I'm still a bit impressed, because Yajl.parse (not the chunked version) while old is still quite competitive with JSON.parse, and since chunked parsing was the whole point of Yajl, I would have expected it to be relatively well optimized.

Yeah, the difference is definitely more pronounced on Apple Silicon. I tested it across two different architectures (Intel Core Ultra 9 on Linux vs. Apple M1 Max on macOS) to see why, and the results are quite interesting.

Looking at the absolute numbers, it seems the massive 15-17x gap on Mac isn't because ResumableParser gets a special boost on ARM, but rather because yajl suffers a significant performance penalty on it.

Linux

OS: Linux 7.0.12
CPU: Intel(R) Core(TM) Ultra 9 275HX
Compiler: gcc 16.1.1

ruby 4.0.5 (2026-05-20 revision 64336ffd0e) +PRISM [x86_64-linux]
Warming up --------------------------------------
            yajl-ruby   183.000 i/100ms
JSON::ResumableParser     1.364k i/100ms
Calculating -------------------------------------
            yajl-ruby      1.834k (± 2.7%) i/s  (545.29 μs/i) -     18.483k in  10.078627s
JSON::ResumableParser     13.554k (± 4.2%) i/s   (73.78 μs/i) -    136.400k in  10.063166s

Comparison:
JSON::ResumableParser:    13554.4 i/s
            yajl-ruby:     1833.9 i/s - 7.39x  slower

macOS

OS: macOS 27.0 beta
CPU: Apple M1 Max
Compiler: Apple clang version 21.0.0 (clang-2100.3.20.102)

ruby 4.0.5 (2026-05-20 revision 64336ffd0e) +PRISM [arm64-darwin27]
Warming up --------------------------------------
            yajl-ruby    81.000 i/100ms
JSON::ResumableParser     1.332k i/100ms
Calculating -------------------------------------
            yajl-ruby    847.237 (± 1.5%) i/s    (1.18 ms/i) -      8.505k in  10.038517s
JSON::ResumableParser     13.273k (± 1.4%) i/s   (75.34 μs/i) -    133.200k in  10.035483s

Comparison:
JSON::ResumableParser:    13272.9 i/s
            yajl-ruby:      847.2 i/s - 15.67x  slower

@kou

👋 @kou and @Watson1978 how the testing going? Let me know when you're done and I'll happily cut a new release.

@byroot

Hi @byroot! 👋

A huge thanks for your works!

We ran heavy load testing with Fluentd's in_forward plugin and it works perfectly now!
No more crashes, and the memory profile looks great.
We also did a thorough audit on GC safety and exception handling during streaming, and the C-extension looks incredibly solid.

I think you can release it anytime.

Thanks!

Great to hear. json 2.20.0 is released.

Thanks!

Sorry for my late work.

I have only one minor feature request: #1041
Others work well. Great!

Just in case, json 2.21.0 is out with the numerous fixes and improvements to ResumableParser you submitted.

Thanks

Navigation Menu

Repository navigation

Add support for parsing chunked data #983

API

16 remaining items

Performance Comparison: `yajl-ruby` vs `JSON::ResumableParser`

Linux

macOS

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add support for parsing chunked data #983

Description

API

Activity

byroot commented on May 28, 2026

eregon commented on May 28, 2026

eregon commented on May 28, 2026

byroot commented on May 28, 2026

headius commented on May 29, 2026

kou commented on May 31, 2026

eregon commented on May 31, 2026

kou commented on May 31, 2026

byroot commented on May 31, 2026

kou commented on Jun 1, 2026

kddnewton commented on Jun 1, 2026

headius commented on Jun 1, 2026

kddnewton commented on Jun 1, 2026

enebo commented on Jun 2, 2026

byroot commented on Jun 4, 2026

16 remaining items

byroot commented on Jun 16, 2026

Watson1978 commented on Jun 16, 2026

byroot commented on Jun 16, 2026

Watson1978 commented on Jun 18, 2026

Performance Comparison: yajl-ruby vs JSON::ResumableParser

byroot commented on Jun 18, 2026

Watson1978 commented on Jun 18, 2026

byroot commented on Jun 18, 2026

Watson1978 commented on Jun 18, 2026

Linux

macOS

byroot commented on Jun 23, 2026

Watson1978 commented on Jun 23, 2026

byroot commented on Jun 23, 2026

Watson1978 commented on Jun 23, 2026

kou commented on Jun 25, 2026

byroot commented on Jul 13, 2026

Watson1978 commented on Jul 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Performance Comparison: `yajl-ruby` vs `JSON::ResumableParser`