Add a built-in :jsonl (JSON Lines) format

CHANGELOG.md

@@ -9,6 +9,7 @@
 * [#2960](https://github.com/ruby-grape/grape/pull/2960): Lint every response of an API once under `lint!`, including the router's 404 and mounted Rack apps, instead of inside each endpoint's stack - [@ericproulx](https://github.com/ericproulx).
 * [#2966](https://github.com/ruby-grape/grape/pull/2966): Require Ruby 3.3.1, the first 3.3 release that parses anonymous argument forwarding from inside a block - [@ericproulx](https://github.com/ericproulx).
 * [#3008](https://github.com/ruby-grape/grape/pull/3008): Add the grape-on-rack app as a live-server memory profiling harness - [@ericproulx](https://github.com/ericproulx).
+* [#3029](https://github.com/ruby-grape/grape/pull/3029): Add a built-in `:jsonl` (JSON Lines) format that streams an Enumerator body - [@ericproulx](https://github.com/ericproulx).
 * Your contribution here.
 
 #### Fixes

README.md

@@ -3350,6 +3350,7 @@ Built-in formatters are the following.
 * `:txt`: use object's `to_txt` when available, otherwise `to_s`
 * `:serializable_hash`: use object's `serializable_hash` when available, otherwise fallback to `:json`
 * `:binary`: data will be returned "as is"
+* `:jsonl`: one line per element of an Array or Enumerator, otherwise one line, each rendered as `:json` renders it. Not a default content type, see [JSON Lines](#json-lines)
 
 #### Serving pre-rendered JSON
 
@@ -3390,6 +3391,50 @@ If a body is present in a request to an API, with a Content-Type header value th
 
 Response statuses that indicate no content as defined by [Rack](https://github.com/rack) [here](https://github.com/rack/rack/blob/master/lib/rack/utils.rb#L567) will bypass serialization and the body entity - though there should be none - will not be modified.
 
+#### JSON Lines
+
+`:jsonl` serves [JSON Lines](https://jsonlines.org): one JSON value per line, each line ended by a newline. It is not among the default content types, so declare it, as `application/jsonl`, or `application/x-ndjson` if your clients expect that one.
+
+```ruby
+class API < Grape::API
+  content_type :jsonl, 'application/jsonl'
+  format :jsonl
+
+  get :users do
+    User.all # {"id":1,"name":"Ada"}\n{"id":2,"name":"Grace"}\n
+  end
+end
+```
+
+An `Array` renders one line per element and any other object one line, each as the `:json` formatter renders it, so a `Grape::PrecompiledJson` element is served as its own line. A `String` is served as it is, taken to be JSON Lines already.
+
+An `Enumerator` is streamed: each element is rendered only as the server reads the body, so a large export is never held in memory whole.
+
+```ruby
+get :export do
+  User.find_each.lazy.map { |user| user.slice(:id, :name) }
+end
+```
+
+The status and headers of a streamed response are sent before its first line is rendered, so an exception raised while it streams is past `rescue_from`: the response just ends early.
+
+A request body of `application/jsonl` is parsed into an Array of its lines' values, skipping blank lines. Like a JSON body that is an Array, it is available in `env['api.request.body']` and is not merged into `params`. To validate the lines as params, wrap them in a Hash with a parser of your own.
+
+```ruby
+parser :jsonl, ->(body, env) { { items: Grape::Parser::Jsonl.call(body, env) } }
+
+params do
+  requires :items, type: Array do
+    requires :name, type: String
+  end
+end
+post :import do
+  declared(params)[:items]
+end
+```
+
+An error renders as a single JSON line.
+
 ### JSONP
 
 Grape supports JSONP via [Rack::JSONP](https://github.com/rack/rack-contrib), part of the [rack-contrib](https://github.com/rack/rack-contrib) gem. Add `rack-contrib` to your `Gemfile`.

UPGRADING.md

@@ -3,6 +3,16 @@ Upgrading Grape
 
 ### Upgrading to >= 4.1.0
 
+#### A `:jsonl` format you declared now gets Grape's JSON Lines formatter, parser and error formatter
+
+Grape now renders, parses and reports errors in JSON Lines under `:jsonl` ([#3029](https://github.com/ruby-grape/grape/pull/3029)). An API that declared `content_type :jsonl, ...` itself, with nothing of its own registered for it, gets Grape's:
+
+* A response body that is not a String renders as JSON Lines. A String is still served as it is.
+* A request body is parsed into an Array in `env['api.request.body']`, where it used to be the raw String. Keep the String with `parser :jsonl, nil`.
+* An error renders as a JSON line instead of through `default_error_formatter`.
+
+A formatter, parser or error formatter you register for `:jsonl` with `formatter`, `parser` or `error_formatter` still takes precedence.
+
 #### `logger` applies to the API it is set on
 
 `logger` kept the logger in settings shared by the whole process, so setting it on one API replaced it on every other: two APIs that each set a logger both logged to whichever was set last.

lib/grape/error_formatter/jsonl.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+module Grape
+  module ErrorFormatter
+    # An error renders as the JSON error formatter renders it, ended by a
+    # newline, so the body is a JSON Lines document of one line.
+    class Jsonl < Json
+      class << self
+        def format_structured_message(structured_message)
+          "#{super}\n"
+        end
+      end
+    end
+  end
+end

lib/grape/formatter/jsonl.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module Grape
+  module Formatter
+    # Renders JSON Lines (https://jsonlines.org): one JSON value per line, each
+    # line ended by a newline. An Array renders one line per element, and any
+    # other object one line. A String is served as it is, taken to be JSON
+    # Lines already.
+    #
+    # An Enumerator renders as it is read rather than up front, so the server
+    # streams each line as the enumerator yields its element:
+    #
+    #   get :export do
+    #     User.find_each # without a block, an Enumerator
+    #   end
+    class Jsonl < Base
+      class << self
+        def call(object, env)
+          return object if object.is_a?(String)
+          return Grape::ServeStream::StreamResponse.new(object.lazy.map { |element| line(element, env) }) if object.is_a?(Enumerator)
+          return object.to_ary.map { |element| line(element, env) }.join if object.respond_to?(:to_ary)
+
+          line(object, env)
+        end
+
+        private
+
+        def line(object, env)
+          "#{Formatter::Json.call(object, env)}\n"
+        end
+      end
+    end
+  end
+end

lib/grape/middleware/formatter.rb

@@ -126,11 +126,11 @@ def build_formatted_response(env, status, headers, bodies, negotiated)
         ensure_content_type!(env, headers)
         return stream_response(status, headers, bodies) if bodies.is_a?(Grape::ServeStream::StreamResponse)
 
-        # Picked by env['api.format']; the Content-Type header only counts when that is unset
-        formatter = fetch_formatter(env, headers)
-        bodymap = instrument_format_response(env, formatter) do
-          bodies.map { |body| formatter.call(body, env) }
-        end
+        bodymap = format_bodies(env, headers, bodies)
+        # A formatter that renders a body only as Rack reads it (see
+        # Grape::Formatter::Jsonl) hands back a stream, served as +stream+'s is.
+        return stream_response(status, headers, bodymap.first) if bodymap.first.is_a?(Grape::ServeStream::StreamResponse)
+
         # A bare Rack tuple rather than a Rack::Response: +headers+ is already
         # a Grape::Util::Header (a Rack::Headers on Rack 3), so wrapping only
         # re-normalizes the same keys into a second Headers hash that
@@ -145,6 +145,14 @@ def build_formatted_response(env, status, headers, bodies, negotiated)
         throw :error, Grape::Exceptions::ErrorResponse.new(status: 500, message: e.message, backtrace: e.backtrace, original_exception: e)
       end
 
+      def format_bodies(env, headers, bodies)
+        # Picked by env['api.format']; the Content-Type header only counts when that is unset
+        formatter = fetch_formatter(env, headers)
+        instrument_format_response(env, formatter) do
+          bodies.map { |body| formatter.call(body, env) }
+        end
+      end
+
       def stream_response(status, headers, bodies)
         Grape::ServeStream::SendfileResponse.new([], status, headers) do |resp|
           resp.body = bodies.stream

lib/grape/parser/jsonl.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module Grape
+  module Parser
+    # Parses a JSON Lines body into an Array of its values. Like a JSON body
+    # that is an Array, it is left in +env['api.request.body']+ rather than
+    # merged into +params+, which only takes a Hash.
+    #
+    # Blank lines are skipped, so a body may end with a newline or not, and
+    # its lines may end in CRLF.
+    class Jsonl < Base
+      def self.call(object, _env)
+        object.each_line.reject { |line| line.strip.empty? }.map { |line| ::Grape::Json.parse(line) }
+      rescue ::Grape::Json::ParseError
+        raise Grape::Exceptions::InvalidMessageBody.new('application/jsonl')
+      end
+    end
+  end
+end

spec/grape/error_formatter/jsonl_spec.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+describe Grape::ErrorFormatter::Jsonl do
+  let(:app) do
+    Class.new(Grape::API) do
+      content_type :jsonl, 'application/jsonl'
+      format :jsonl
+
+      get(:message) { error!('nope', 422) }
+      get(:hash) { error!({ code: 'nope', detail: 'line' }, 422) }
+    end
+  end
+
+  it 'renders a message as one JSON line' do
+    get '/message'
+    expect(last_response.status).to eq(422)
+    expect(last_response.content_type).to eq('application/jsonl')
+    expect(last_response.body).to eq(%({"error":"nope"}\n))
+  end
+
+  it 'renders a Hash as one JSON line' do
+    get '/hash'
+    expect(last_response.body).to eq(%({"code":"nope","detail":"line"}\n))
+  end
+end

spec/grape/formatter/jsonl_spec.rb

@@ -0,0 +1,104 @@
+# frozen_string_literal: true
+
+describe Grape::Formatter::Jsonl do
+  let(:app) do
+    Class.new(Grape::API) do
+      content_type :jsonl, 'application/jsonl'
+      format :jsonl
+
+      get(:hash) { { id: 1 } }
+      get(:array) { [{ id: 1 }, 'two', nil] }
+      get(:empty) { [] }
+      get(:string) { %({"id":1}\n{"id":2}\n) }
+      get(:precompiled) { [Grape::PrecompiledJson.new('{"id":1}')] }
+      get(:enumerator) { [{ id: 1 }, { id: 2 }].each }
+    end
+  end
+
+  it 'renders an object as one line' do
+    get '/hash'
+    expect(last_response.content_type).to eq('application/jsonl')
+    expect(last_response.body).to eq(%({"id":1}\n))
+  end
+
+  it 'renders each element of an Array as a line of its own' do
+    get '/array'
+    expect(last_response.body).to eq(%({"id":1}\n"two"\nnull\n))
+  end
+
+  it 'renders an empty Array as an empty body' do
+    get '/empty'
+    expect(last_response.status).to eq(200)
+    expect(last_response.body).to eq('')
+  end
+
+  it 'serves a String as it is' do
+    get '/string'
+    expect(last_response.body).to eq(%({"id":1}\n{"id":2}\n))
+  end
+
+  it 'serves a Grape::PrecompiledJson element as its line' do
+    get '/precompiled'
+    expect(last_response.body).to eq(%({"id":1}\n))
+  end
+
+  it 'renders each element of an Enumerator as a line of its own' do
+    get '/enumerator'
+    expect(last_response.content_type).to eq('application/jsonl')
+    expect(last_response.body).to eq(%({"id":1}\n{"id":2}\n))
+  end
+
+  it 'streams an Enumerator, rendering each element only as Rack reads it' do
+    produced = []
+    api = Class.new(Grape::API) do
+      content_type :jsonl, 'application/jsonl'
+      format :jsonl
+      get do
+        Enumerator.new do |lines|
+          2.times do |i|
+            produced << i
+            lines << { id: i }
+          end
+        end
+      end
+    end
+
+    _status, _headers, body = api.call(Rack::MockRequest.env_for('/'))
+    expect(produced).to be_empty
+
+    chunks = body.enum_for(:each).to_a
+    body.close if body.respond_to?(:close)
+    expect(chunks).to eq([%({"id":0}\n), %({"id":1}\n)])
+    expect(produced).to eq([0, 1])
+  end
+
+  context 'with a content type registered for :jsonl' do
+    let(:app) do
+      Class.new(Grape::API) do
+        content_type :jsonl, 'application/x-ndjson'
+        format :jsonl
+
+        get { [{ id: 1 }, { id: 2 }] }
+      end
+    end
+
+    it 'renders under that content type' do
+      get '/'
+      expect(last_response.content_type).to eq('application/x-ndjson')
+      expect(last_response.body).to eq(%({"id":1}\n{"id":2}\n))
+    end
+  end
+
+  context 'when the API declares no content type' do
+    let(:app) do
+      Class.new(Grape::API) do
+        get { { id: 1 } }
+      end
+    end
+
+    it 'is not negotiated' do
+      get '/', {}, 'HTTP_ACCEPT' => 'application/jsonl'
+      expect(last_response.content_type).to eq('text/plain')
+    end
+  end
+end

spec/grape/parser/jsonl_spec.rb

@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+describe Grape::Parser::Jsonl do
+  let(:app) do
+    Class.new(Grape::API) do
+      content_type :jsonl, 'application/jsonl'
+      format :jsonl
+
+      post do
+        { body: env[Grape::Env::API_REQUEST_BODY], params: params.to_h }
+      end
+    end
+  end
+
+  let(:content_type) { 'application/jsonl' }
+
+  def post_lines(body)
+    post '/', body, 'CONTENT_TYPE' => content_type
+  end
+
+  it 'answers each line as an element of the request body, leaving params empty' do
+    post_lines %({"a":1}\n{"b":2}\n)
+    expect(last_response.status).to eq(201)
+    expect(JSON.parse(last_response.body)).to eq('body' => [{ 'a' => 1 }, { 'b' => 2 }], 'params' => {})
+  end
+
+  it 'reads a last line that has no newline' do
+    post_lines %({"a":1}\n{"b":2})
+    expect(JSON.parse(last_response.body)['body']).to eq([{ 'a' => 1 }, { 'b' => 2 }])
+  end
+
+  it 'skips blank lines and reads CRLF line ends' do
+    post_lines %({"a":1}\r\n\r\n  \n{"b":2}\r\n)
+    expect(JSON.parse(last_response.body)['body']).to eq([{ 'a' => 1 }, { 'b' => 2 }])
+  end
+
+  it 'keeps a line holding null or false' do
+    post_lines %(null\nfalse\n)
+    expect(JSON.parse(last_response.body)['body']).to eq([nil, false])
+  end
+
+  it 'answers 400 when a line is not JSON' do
+    post_lines %({"a":1}\nnope\n)
+    expect(last_response.status).to eq(400)
+    expect(last_response.body).to include('message body does not match declared format')
+  end
+
+  context 'with a content type registered for :jsonl' do
+    let(:app) do
+      Class.new(Grape::API) do
+        content_type :jsonl, 'application/x-ndjson'
+        format :jsonl
+
+        post { env[Grape::Env::API_REQUEST_BODY] }
+      end
+    end
+
+    let(:content_type) { 'application/x-ndjson' }
+
+    it 'parses a body of that content type' do
+      post_lines %({"a":1}\n{"b":2}\n)
+      expect(last_response.body).to eq(%({"a":1}\n{"b":2}\n))
+    end
+  end
+end