GH-51395: [C++][Docs] Replace RapidJSON with simdjson in row converter example by Reranko05 · Pull Request #51396 · apache/arrow

Reranko05 · 2026-09-19T19:44:20Z

Rationale for this change

This PR continues the simdjson migration by replacing RapidJSON usage in the C++ row converter example with simdjson and Arrow's existing JsonWriter.

Changes

Replace RapidJSON DOM parsing with simdjson's DOM API.
Replace RapidJSON JSON generation with Arrow's JsonWriter.
Preserve the existing row-to-Arrow and Arrow-to-row conversion flow.
Update the example to link against arrow::simdjson.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

Human
AI

Reviewed before submission by:

Human
AI
Not reviewed

GitHub Issue: [C++][Docs] Replace RapidJSON with simdjson in row converter example #51395

Reranko05 · 2026-09-20T18:48:59Z

Will mark this PR Ready to Review after #51038 is merged.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

pitrou · 2026-10-08T15:36:48Z

+//  * arrow::TableBatchReader and Arrow iterators for converting Arrow tables back
+//    into row-based JSON data
+
+namespace arrow {


Example code shouldn't be in the arrow namespace because it showcases how third-party code using Arrow C++ could look like.

Ah, okay. I got it.

pitrou · 2026-10-08T15:37:01Z

+//  * arrow::internal::JsonWriter for writing JSON values
+//  * arrow::internal::ParseJsonObject and related helpers for parsing JSON


Examples should not use any internal Arrow APIs.

Okay, understood.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Reranko05 · 2026-10-11T09:52:15Z

I have not got the time to review this myself after changing it into plain simdjson. I will mark it as Ready for review after I review this.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Reranko05 · 2026-10-16T11:20:48Z

@pitrou @kou Could you review this when you have time? (CI failures seem unrelated)

kou · 2026-10-16T13:58:11Z

+  // Parse each row and append its values to the corresponding Arrow builders.
+  for (const auto& json : rows) {
+    ARROW_ASSIGN_OR_RAISE(auto element,
+                          Unwrap(parser.parse(json), "Failed to parse JSON row"));
+    ARROW_ASSIGN_OR_RAISE(auto object,
+                          Unwrap(element.get_object(), "Expected a JSON object"));
+
+    for (int i = 0; i < schema->num_fields(); ++i) {
+      const auto& field = schema->field(i);
+      arrow::ArrayBuilder* builder = batch_builder->GetField(i);
+
+      ARROW_ASSIGN_OR_RAISE(auto value, GetField(object, field->name()));
+
+      if (!value.has_value()) {
+        ARROW_RETURN_NOT_OK(builder->AppendNull());
+      } else {
+        ARROW_RETURN_NOT_OK(AppendJsonValue(*value, field->type(), builder));
+      }
+    }
+  }


Can we keep column based iteration (not row based iteration) like the RapidJSON version so that we demonstrate how to use a visitor to convert row based data to Arrow data?

I changed the impl to do column based iteration.

kou · 2026-10-16T14:03:30Z

+  for (int64_t i = 0; i < batch->num_rows(); ++i) {
+    ARROW_ASSIGN_OR_RAISE(auto row, ConvertRowToJson(*batch, i));
+    rows.push_back(std::make_shared<std::string>(std::move(row)));
+  }


Can we keep column based iteration (not row based iteration) like the RapidJSON version so that we demonstrate how to use a visitor to convert Arrow data to row based data?

As said above.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Reranko05 · 2026-10-16T18:01:34Z

I have not updated the docs yet, once the impl is approved I will update the docs accordingly.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Reranko05 · 2026-10-19T12:49:02Z

@kou @pitrou Could you review this when you have time? Thanks :)

Reranko05 · 2026-10-22T04:24:29Z

[1342/2536] Building CXX object _deps/grpc-build/CMakeFiles/grpc.dir/src/core/credentials/transport/tls/grpc_tls_crl_provider.cc.o
FAILED: [code=1] _deps/grpc-build/CMakeFiles/grpc.dir/src/core/credentials/transport/tls/grpc_tls_crl_provider.cc.o 
/opt/homebrew/bin/ccache /usr/bin/c++ -DARROW_HAVE_NEON -I/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src -I/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/third_party/address_sorting/include -I/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/src/core/ext/upb-gen -I/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/src/core/ext/upbdefs-gen -I/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/third_party/upb -I/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/third_party/utf8_range -I/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/third_party/xxhash -I/Users/runner/work/arrow/arrow/build/cpp/_deps/cares-build -I/Users/runner/work/arrow/arrow/build/cpp/_deps/cares-src -I/Users/runner/work/arrow/arrow/build/cpp/_deps/cares-src/include -isystem /Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/include -isystem /Users/runner/work/arrow/arrow/build/cpp/_deps/re2-src -isystem /Users/runner/work/arrow/arrow/build/cpp/zlib_ep/src/zlib_ep-install/include -isystem /Users/runner/work/arrow/arrow/build/cpp/_deps/absl-src -isystem /opt/homebrew/opt/openssl@4/include -fno-aligned-new  -I/opt/homebrew/Cellar/libiodbc/3.52.16/include -Qunused-arguments -fcolor-diagnostics  -Wall -Wno-unknown-warning-option -Wno-pass-failed -march=armv8-a  -Wno-attributes -Wno-format-security -Wno-unknown-warning-option  -g -O0 -ggdb -g1 -Wno-error -std=c++20 -arch arm64 -fPIC -Wno-error=deprecated-declarations -MD -MT _deps/grpc-build/CMakeFiles/grpc.dir/src/core/credentials/transport/tls/grpc_tls_crl_provider.cc.o -MF _deps/grpc-build/CMakeFiles/grpc.dir/src/core/credentials/transport/tls/grpc_tls_crl_provider.cc.o.d -o _deps/grpc-build/CMakeFiles/grpc.dir/src/core/credentials/transport/tls/grpc_tls_crl_provider.cc.o -c /Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/src/core/credentials/transport/tls/grpc_tls_crl_provider.cc
/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/src/core/credentials/transport/tls/grpc_tls_crl_provider.cc:59:14: error: cannot initialize a variable of type 'X509_NAME *' (aka 'X509_name_st *') with an rvalue of type 'const X509_NAME *' (aka 'const X509_name_st *')
   59 |   X509_NAME* issuer = X509_CRL_get_issuer(crl);
      |              ^        ~~~~~~~~~~~~~~~~~~~~~~~~

@kou @pitrou @taepper I am a bit confused about whether this failure is related to my PR, since another PR passes the same CI job even after I rebased mine. Could someone clarify?

Reranko05 · 2026-10-22T04:35:58Z

BTW, I have not yet updated the .rst doc file, I will do that once the simdjson_row_converter.cc changes are approved.

taepper · 2026-10-22T09:31:58Z

Hi, yes, I can take a look at this

Can you quickly confirm my understanding re/ the commit history because I am a little bit confused (and the history has since been rebased & force-pushed)

The initial commit Replace example was the first draft and the subsequent commits (which change the structure considerably), were the iterations based on review feedback?

@taepper I am a bit confused about whether this failure is related to my PR, since another #54839 passes the same CI job even after I rebased mine. Could someone clarify?

Yes, I would also think that your change to CMakeLists.txt is unrelated

pitrou · 2026-10-22T09:35:31Z

@kou @pitrou @taepper I am a bit confused about whether this failure is related to my PR

It's not, we see it on other PRs as well.

Reranko05 · 2026-10-22T14:51:28Z

Oh okay, thanks for confirming :)
This is ready for review.
cc @kou @pitrou @taepper

Reranko05 · 2026-10-22T15:04:05Z

were the iterations based on review feedback?

@taepper I initially used internal Arrow APIs to write the example code, then I got to know that I should write it in plain simdjson and then it was that my impl was row based but we needed column based iteration, so yeah they were considerable feedback changes 😅

github-actions Bot added Component: C++ awaiting review Awaiting review labels Sep 19, 2026

Reranko05 added the CI: Extra: C++ Run extra C++ CI label Sep 19, 2026

Reranko05 changed the title ~~GH-51395: [C++] Replace RapidJSON with Simdjson in row converter example~~ GH-51395: [C++] Replace RapidJSON with simdjson in row converter example Sep 19, 2026

Reranko05 force-pushed the example-doc branch from 3842058 to 9813fc2 Compare September 19, 2026 20:23

github-actions Bot added the Component: Documentation label Sep 19, 2026

Reranko05 changed the title ~~GH-51395: [C++] Replace RapidJSON with simdjson in row converter example~~ GH-51395: [C++][Docs] Replace RapidJSON with simdjson in row converter example Sep 19, 2026

Reranko05 force-pushed the example-doc branch from ebf072f to 4934519 Compare October 8, 2026 08:29

Reranko05 marked this pull request as ready for review October 8, 2026 09:37

Reranko05 requested a review from pitrou as a code owner October 8, 2026 09:37

Copilot AI balanced review requested due to automatic review settings October 8, 2026 09:37

Copilot AI reviewed Oct 8, 2026

Reranko05 requested a review from kou October 8, 2026 09:37

pitrou reviewed Oct 8, 2026

View reviewed changes

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Oct 8, 2026

Copilot AI balanced review requested due to automatic review settings October 9, 2026 09:20

Copilot AI reviewed Oct 9, 2026

Copilot AI balanced review requested due to automatic review settings October 9, 2026 09:40

Reranko05 force-pushed the example-doc branch from 84ed127 to 7b885c3 Compare October 9, 2026 09:40

Copilot AI reviewed Oct 9, 2026

Reranko05 marked this pull request as draft October 11, 2026 09:52

Reranko05 marked this pull request as ready for review October 16, 2026 11:19

Copilot AI balanced review requested due to automatic review settings October 16, 2026 11:19

Copilot AI reviewed Oct 16, 2026

kou reviewed Oct 16, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Oct 16, 2026

Copilot AI balanced review requested due to automatic review settings October 16, 2026 18:00

Copilot AI reviewed Oct 16, 2026

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 16, 2026

Copilot AI balanced review requested due to automatic review settings October 17, 2026 07:36

Reranko05 force-pushed the example-doc branch from 75fc21a to 6c304b8 Compare October 17, 2026 07:36

Copilot AI reviewed Oct 17, 2026

Reranko05 requested a review from kou October 17, 2026 13:51

Reranko05 force-pushed the example-doc branch from 6c304b8 to 9519b33 Compare October 22, 2026 03:33

Reranko05 added 7 commits October 22, 2026 09:40

Replace example

f8212f5

Update rst doc

646b0ef

Minor fixes

025a091

Self Review

3bc2678

Use plain simdjson

6470cc1

Add get_uint64()

5dac63e

Change to column based iteration

3673c4d

Reranko05 force-pushed the example-doc branch from 9519b33 to 3673c4d Compare October 22, 2026 04:11

		// * arrow::internal::JsonWriter for writing JSON values
		// * arrow::internal::ParseJsonObject and related helpers for parsing JSON

Conversation

Reranko05 commented Sep 19, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Changes

Was AI used for this PR?

Uh oh!

Reranko05 commented Sep 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Oct 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Oct 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Oct 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Oct 19, 2026

Uh oh!

Reranko05 commented Oct 22, 2026

Uh oh!

Reranko05 commented Oct 22, 2026

Uh oh!

taepper commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pitrou commented Oct 22, 2026

Uh oh!

Reranko05 commented Oct 22, 2026

Uh oh!

Reranko05 commented Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Reranko05 commented Sep 19, 2026 •

edited by github-actions Bot

Loading

taepper commented Oct 22, 2026 •

edited

Loading