diff --git a/CHANGELOG.md b/CHANGELOG.md index 291361cd9..fab22b4dc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ * [#2960](https://github.com/ruby-grape/grape/pull/2960): Lint every response of an API once under `lint!`, including the router's 404 and mounted Rack apps, instead of inside each endpoint's stack - [@ericproulx](https://github.com/ericproulx). * [#2966](https://github.com/ruby-grape/grape/pull/2966): Require Ruby 3.3.1, the first 3.3 release that parses anonymous argument forwarding from inside a block - [@ericproulx](https://github.com/ericproulx). * [#3008](https://github.com/ruby-grape/grape/pull/3008): Add the grape-on-rack app as a live-server memory profiling harness - [@ericproulx](https://github.com/ericproulx). +* [#3029](https://github.com/ruby-grape/grape/pull/3029): Add a built-in `:jsonl` (JSON Lines) format that streams an Enumerator body - [@ericproulx](https://github.com/ericproulx). * Your contribution here. #### Fixes diff --git a/README.md b/README.md index 990c93dc9..c11c3b260 100644 --- a/README.md +++ b/README.md @@ -3350,6 +3350,7 @@ Built-in formatters are the following. * `:txt`: use object's `to_txt` when available, otherwise `to_s` * `:serializable_hash`: use object's `serializable_hash` when available, otherwise fallback to `:json` * `:binary`: data will be returned "as is" +* `:jsonl`: one line per element of an Array or Enumerator, otherwise one line, each rendered as `:json` renders it. Not a default content type, see [JSON Lines](#json-lines) #### Serving pre-rendered JSON @@ -3390,6 +3391,50 @@ If a body is present in a request to an API, with a Content-Type header value th Response statuses that indicate no content as defined by [Rack](https://github.com/rack) [here](https://github.com/rack/rack/blob/master/lib/rack/utils.rb#L567) will bypass serialization and the body entity - though there should be none - will not be modified. +#### JSON Lines + +`:jsonl` serves [JSON Lines](https://jsonlines.org): one JSON value per line, each line ended by a newline. It is not among the default content types, so declare it, as `application/jsonl`, or `application/x-ndjson` if your clients expect that one. + +```ruby +class API < Grape::API + content_type :jsonl, 'application/jsonl' + format :jsonl + + get :users do + User.all # {"id":1,"name":"Ada"}\n{"id":2,"name":"Grace"}\n + end +end +``` + +An `Array` renders one line per element and any other object one line, each as the `:json` formatter renders it, so a `Grape::PrecompiledJson` element is served as its own line. A `String` is served as it is, taken to be JSON Lines already. + +An `Enumerator` is streamed: each element is rendered only as the server reads the body, so a large export is never held in memory whole. + +```ruby +get :export do + User.find_each.lazy.map { |user| user.slice(:id, :name) } +end +``` + +The status and headers of a streamed response are sent before its first line is rendered, so an exception raised while it streams is past `rescue_from`: the response just ends early. + +A request body of `application/jsonl` is parsed into an Array of its lines' values, skipping blank lines. Like a JSON body that is an Array, it is available in `env['api.request.body']` and is not merged into `params`. To validate the lines as params, wrap them in a Hash with a parser of your own. + +```ruby +parser :jsonl, ->(body, env) { { items: Grape::Parser::Jsonl.call(body, env) } } + +params do + requires :items, type: Array do + requires :name, type: String + end +end +post :import do + declared(params)[:items] +end +``` + +An error renders as a single JSON line. + ### JSONP Grape supports JSONP via [Rack::JSONP](https://github.com/rack/rack-contrib), part of the [rack-contrib](https://github.com/rack/rack-contrib) gem. Add `rack-contrib` to your `Gemfile`. diff --git a/UPGRADING.md b/UPGRADING.md index e1107367b..210034613 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -3,6 +3,16 @@ Upgrading Grape ### Upgrading to >= 4.1.0 +#### A `:jsonl` format you declared now gets Grape's JSON Lines formatter, parser and error formatter + +Grape now renders, parses and reports errors in JSON Lines under `:jsonl` ([#3029](https://github.com/ruby-grape/grape/pull/3029)). An API that declared `content_type :jsonl, ...` itself, with nothing of its own registered for it, gets Grape's: + +* A response body that is not a String renders as JSON Lines. A String is still served as it is. +* A request body is parsed into an Array in `env['api.request.body']`, where it used to be the raw String. Keep the String with `parser :jsonl, nil`. +* An error renders as a JSON line instead of through `default_error_formatter`. + +A formatter, parser or error formatter you register for `:jsonl` with `formatter`, `parser` or `error_formatter` still takes precedence. + #### `logger` applies to the API it is set on `logger` kept the logger in settings shared by the whole process, so setting it on one API replaced it on every other: two APIs that each set a logger both logged to whichever was set last. diff --git a/lib/grape/error_formatter/jsonl.rb b/lib/grape/error_formatter/jsonl.rb new file mode 100644 index 000000000..b355de5a0 --- /dev/null +++ b/lib/grape/error_formatter/jsonl.rb @@ -0,0 +1,15 @@ +# frozen_string_literal: true + +module Grape + module ErrorFormatter + # An error renders as the JSON error formatter renders it, ended by a + # newline, so the body is a JSON Lines document of one line. + class Jsonl < Json + class << self + def format_structured_message(structured_message) + "#{super}\n" + end + end + end + end +end diff --git a/lib/grape/formatter/jsonl.rb b/lib/grape/formatter/jsonl.rb new file mode 100644 index 000000000..cd736af56 --- /dev/null +++ b/lib/grape/formatter/jsonl.rb @@ -0,0 +1,34 @@ +# frozen_string_literal: true + +module Grape + module Formatter + # Renders JSON Lines (https://jsonlines.org): one JSON value per line, each + # line ended by a newline. An Array renders one line per element, and any + # other object one line. A String is served as it is, taken to be JSON + # Lines already. + # + # An Enumerator renders as it is read rather than up front, so the server + # streams each line as the enumerator yields its element: + # + # get :export do + # User.find_each # without a block, an Enumerator + # end + class Jsonl < Base + class << self + def call(object, env) + return object if object.is_a?(String) + return Grape::ServeStream::StreamResponse.new(object.lazy.map { |element| line(element, env) }) if object.is_a?(Enumerator) + return object.to_ary.map { |element| line(element, env) }.join if object.respond_to?(:to_ary) + + line(object, env) + end + + private + + def line(object, env) + "#{Formatter::Json.call(object, env)}\n" + end + end + end + end +end diff --git a/lib/grape/middleware/formatter.rb b/lib/grape/middleware/formatter.rb index fc386ce12..1d0a4832b 100644 --- a/lib/grape/middleware/formatter.rb +++ b/lib/grape/middleware/formatter.rb @@ -126,11 +126,11 @@ def build_formatted_response(env, status, headers, bodies, negotiated) ensure_content_type!(env, headers) return stream_response(status, headers, bodies) if bodies.is_a?(Grape::ServeStream::StreamResponse) - # Picked by env['api.format']; the Content-Type header only counts when that is unset - formatter = fetch_formatter(env, headers) - bodymap = instrument_format_response(env, formatter) do - bodies.map { |body| formatter.call(body, env) } - end + bodymap = format_bodies(env, headers, bodies) + # A formatter that renders a body only as Rack reads it (see + # Grape::Formatter::Jsonl) hands back a stream, served as +stream+'s is. + return stream_response(status, headers, bodymap.first) if bodymap.first.is_a?(Grape::ServeStream::StreamResponse) + # A bare Rack tuple rather than a Rack::Response: +headers+ is already # a Grape::Util::Header (a Rack::Headers on Rack 3), so wrapping only # re-normalizes the same keys into a second Headers hash that @@ -145,6 +145,14 @@ def build_formatted_response(env, status, headers, bodies, negotiated) throw :error, Grape::Exceptions::ErrorResponse.new(status: 500, message: e.message, backtrace: e.backtrace, original_exception: e) end + def format_bodies(env, headers, bodies) + # Picked by env['api.format']; the Content-Type header only counts when that is unset + formatter = fetch_formatter(env, headers) + instrument_format_response(env, formatter) do + bodies.map { |body| formatter.call(body, env) } + end + end + def stream_response(status, headers, bodies) Grape::ServeStream::SendfileResponse.new([], status, headers) do |resp| resp.body = bodies.stream diff --git a/lib/grape/parser/jsonl.rb b/lib/grape/parser/jsonl.rb new file mode 100644 index 000000000..636ed6cbe --- /dev/null +++ b/lib/grape/parser/jsonl.rb @@ -0,0 +1,19 @@ +# frozen_string_literal: true + +module Grape + module Parser + # Parses a JSON Lines body into an Array of its values. Like a JSON body + # that is an Array, it is left in +env['api.request.body']+ rather than + # merged into +params+, which only takes a Hash. + # + # Blank lines are skipped, so a body may end with a newline or not, and + # its lines may end in CRLF. + class Jsonl < Base + def self.call(object, _env) + object.each_line.reject { |line| line.strip.empty? }.map { |line| ::Grape::Json.parse(line) } + rescue ::Grape::Json::ParseError + raise Grape::Exceptions::InvalidMessageBody.new('application/jsonl') + end + end + end +end diff --git a/spec/grape/error_formatter/jsonl_spec.rb b/spec/grape/error_formatter/jsonl_spec.rb new file mode 100644 index 000000000..4ce03532e --- /dev/null +++ b/spec/grape/error_formatter/jsonl_spec.rb @@ -0,0 +1,25 @@ +# frozen_string_literal: true + +describe Grape::ErrorFormatter::Jsonl do + let(:app) do + Class.new(Grape::API) do + content_type :jsonl, 'application/jsonl' + format :jsonl + + get(:message) { error!('nope', 422) } + get(:hash) { error!({ code: 'nope', detail: 'line' }, 422) } + end + end + + it 'renders a message as one JSON line' do + get '/message' + expect(last_response.status).to eq(422) + expect(last_response.content_type).to eq('application/jsonl') + expect(last_response.body).to eq(%({"error":"nope"}\n)) + end + + it 'renders a Hash as one JSON line' do + get '/hash' + expect(last_response.body).to eq(%({"code":"nope","detail":"line"}\n)) + end +end diff --git a/spec/grape/formatter/jsonl_spec.rb b/spec/grape/formatter/jsonl_spec.rb new file mode 100644 index 000000000..e7792ac41 --- /dev/null +++ b/spec/grape/formatter/jsonl_spec.rb @@ -0,0 +1,104 @@ +# frozen_string_literal: true + +describe Grape::Formatter::Jsonl do + let(:app) do + Class.new(Grape::API) do + content_type :jsonl, 'application/jsonl' + format :jsonl + + get(:hash) { { id: 1 } } + get(:array) { [{ id: 1 }, 'two', nil] } + get(:empty) { [] } + get(:string) { %({"id":1}\n{"id":2}\n) } + get(:precompiled) { [Grape::PrecompiledJson.new('{"id":1}')] } + get(:enumerator) { [{ id: 1 }, { id: 2 }].each } + end + end + + it 'renders an object as one line' do + get '/hash' + expect(last_response.content_type).to eq('application/jsonl') + expect(last_response.body).to eq(%({"id":1}\n)) + end + + it 'renders each element of an Array as a line of its own' do + get '/array' + expect(last_response.body).to eq(%({"id":1}\n"two"\nnull\n)) + end + + it 'renders an empty Array as an empty body' do + get '/empty' + expect(last_response.status).to eq(200) + expect(last_response.body).to eq('') + end + + it 'serves a String as it is' do + get '/string' + expect(last_response.body).to eq(%({"id":1}\n{"id":2}\n)) + end + + it 'serves a Grape::PrecompiledJson element as its line' do + get '/precompiled' + expect(last_response.body).to eq(%({"id":1}\n)) + end + + it 'renders each element of an Enumerator as a line of its own' do + get '/enumerator' + expect(last_response.content_type).to eq('application/jsonl') + expect(last_response.body).to eq(%({"id":1}\n{"id":2}\n)) + end + + it 'streams an Enumerator, rendering each element only as Rack reads it' do + produced = [] + api = Class.new(Grape::API) do + content_type :jsonl, 'application/jsonl' + format :jsonl + get do + Enumerator.new do |lines| + 2.times do |i| + produced << i + lines << { id: i } + end + end + end + end + + _status, _headers, body = api.call(Rack::MockRequest.env_for('/')) + expect(produced).to be_empty + + chunks = body.enum_for(:each).to_a + body.close if body.respond_to?(:close) + expect(chunks).to eq([%({"id":0}\n), %({"id":1}\n)]) + expect(produced).to eq([0, 1]) + end + + context 'with a content type registered for :jsonl' do + let(:app) do + Class.new(Grape::API) do + content_type :jsonl, 'application/x-ndjson' + format :jsonl + + get { [{ id: 1 }, { id: 2 }] } + end + end + + it 'renders under that content type' do + get '/' + expect(last_response.content_type).to eq('application/x-ndjson') + expect(last_response.body).to eq(%({"id":1}\n{"id":2}\n)) + end + end + + context 'when the API declares no content type' do + let(:app) do + Class.new(Grape::API) do + get { { id: 1 } } + end + end + + it 'is not negotiated' do + get '/', {}, 'HTTP_ACCEPT' => 'application/jsonl' + expect(last_response.content_type).to eq('text/plain') + end + end +end diff --git a/spec/grape/parser/jsonl_spec.rb b/spec/grape/parser/jsonl_spec.rb new file mode 100644 index 000000000..ab0b6ab57 --- /dev/null +++ b/spec/grape/parser/jsonl_spec.rb @@ -0,0 +1,65 @@ +# frozen_string_literal: true + +describe Grape::Parser::Jsonl do + let(:app) do + Class.new(Grape::API) do + content_type :jsonl, 'application/jsonl' + format :jsonl + + post do + { body: env[Grape::Env::API_REQUEST_BODY], params: params.to_h } + end + end + end + + let(:content_type) { 'application/jsonl' } + + def post_lines(body) + post '/', body, 'CONTENT_TYPE' => content_type + end + + it 'answers each line as an element of the request body, leaving params empty' do + post_lines %({"a":1}\n{"b":2}\n) + expect(last_response.status).to eq(201) + expect(JSON.parse(last_response.body)).to eq('body' => [{ 'a' => 1 }, { 'b' => 2 }], 'params' => {}) + end + + it 'reads a last line that has no newline' do + post_lines %({"a":1}\n{"b":2}) + expect(JSON.parse(last_response.body)['body']).to eq([{ 'a' => 1 }, { 'b' => 2 }]) + end + + it 'skips blank lines and reads CRLF line ends' do + post_lines %({"a":1}\r\n\r\n \n{"b":2}\r\n) + expect(JSON.parse(last_response.body)['body']).to eq([{ 'a' => 1 }, { 'b' => 2 }]) + end + + it 'keeps a line holding null or false' do + post_lines %(null\nfalse\n) + expect(JSON.parse(last_response.body)['body']).to eq([nil, false]) + end + + it 'answers 400 when a line is not JSON' do + post_lines %({"a":1}\nnope\n) + expect(last_response.status).to eq(400) + expect(last_response.body).to include('message body does not match declared format') + end + + context 'with a content type registered for :jsonl' do + let(:app) do + Class.new(Grape::API) do + content_type :jsonl, 'application/x-ndjson' + format :jsonl + + post { env[Grape::Env::API_REQUEST_BODY] } + end + end + + let(:content_type) { 'application/x-ndjson' } + + it 'parses a body of that content type' do + post_lines %({"a":1}\n{"b":2}\n) + expect(last_response.body).to eq(%({"a":1}\n{"b":2}\n)) + end + end +end