sve2 HISTSEG support for fast ondemand parsing on ARM by emcastillo · Pull Request #108 · bytedance/sonic-cpp

emcastillo · 2025-05-11T05:50:23Z

This PR uses the SVE2 HISTSEG instruction to improve performance of ondemand parsing in ARM cpus

https://developer.arm.com/documentation/ddi0602/latest/SVE-Instructions/HISTSEG--Count-matching-elements-in-vector-segments-

Creating the bitmasks for each structural character is very expensive in ARM due to the lack of pmovmskb similar instructions. Histseg gives us the number of characters of interest in a single instruction with low latency and high throughput in neoverse v2 cpus.

This allows us to skip mask creation of characters that are not present and even to implement some fast path codes when we have only quotes for example. It is worth noting that x86 doesn't have an equivalent instruction and the avx2 VP2INTERSECT instruction doesn't support int8 datatypes.

Performance:

Master branch

twitter/SonicOnDemand_Normal            22625 ns        22624 ns        30718 bytes_per_second=10.8832Gi/s Normal
citm_catalog/SonicOnDemand_Fronter      12861 ns        12862 ns        54399 bytes_per_second=125.067Gi/s Fronter
twitter/SonicOnDemand_NotFound          22423 ns        22422 ns        31349 bytes_per_second=10.9814Gi/s NotFound

This PR

twitter/SonicOnDemand_Normal             6464 ns         6464 ns       108097 bytes_per_second=38.0904Gi/s Normal
citm_catalog/SonicOnDemand_Fronter       6978 ns         6978 ns       100373 bytes_per_second=230.538Gi/s Fronter
twitter/SonicOnDemand_NotFound           6325 ns         6325 ns       110703 bytes_per_second=38.9312Gi/s NotFound

This is faster than even the most recent x86 cpus which shows result around 24Gi/s for twitter.

Thanks to @supermartian !

This PR is contributed by nvidia.

codecov-commenter · 2025-05-12T06:53:14Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.67%. Comparing base (ee76cf2) to head (e00f215).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #108   +/-   ##
=======================================
  Coverage   69.67%   69.67%           
=======================================
  Files          27       27           
  Lines        3706     3706           
  Branches     1189     1189           
=======================================
  Hits         2582     2582           
  Misses        211      211           
  Partials      913      913

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

xiegx94 · 2025-06-11T04:59:19Z

#110 The CI bugs have been fixed，please rebase master

emcastillo · 2025-07-30T15:33:00Z

@xiegx94
Rebased! sorry for the long wait :(

xiegx94 · 2025-08-12T12:19:31Z

 #include "../common/arm_common/skip.inc.h"

+// Requires clang vx or GCC>=14
+#if (defined(__clang__) && (__clang_major__ >= 14)) || \


__has_include would be better

xiegx94 · 2025-08-12T12:20:02Z

+#if (defined(__clang__) && (__clang_major__ >= 14)) || \
+    (defined(__GNUC__) && !defined(__clang__) && (__GNUC__ >= 14))
+
+#define USE_SVE_HIST 1


check USE_SVE_HIST is not defined.

emcastillo · 2025-08-17T13:40:45Z

@xiegx94
Thanks for the comments! I think I've addressed them :)

emcastillo · 2025-09-26T05:03:22Z

Hi @xiegx94 :) any chance you can look at it soon?
Thanks!

emcastillo · 2025-11-28T04:00:57Z

@xiegx94 @liuq19 any chance of getting this merged? Thanks!

emcastillo · 2026-03-06T03:40:16Z

PTAL :)

emcastillo · 2026-09-02T06:35:56Z

Any chance of getting this merged? @yangzhg
@liuq19 @xiegx94

yangzhg · 2026-09-02T07:56:13Z

+  uint64_t escaped = 0;
+  uint64_t bs_bits = 0;
+  if (backslash_count) {
+    uint64_t bs_bits = v.eq('\\');


same variable bs_bits with line 54

yangzhg · 2026-09-02T07:57:10Z

+}
+
+sonic_force_inline uint32_t count_chars(const uint8x16_t &data,
+                                        svuint8_t &tokens, uint8_t left,


uint8_t left, uint8_t right unused ？

yangzhg · 2026-09-02T07:59:06Z


+#ifdef __has_include
+#if __has_include(<arm_neon_sve_bridge.h>)
+#include <arm_neon_sve_bridge.h>


why include sys header inside sonic_json::internal::sve2_128 namespace

yangzhg · 2026-09-02T07:59:32Z

+#ifdef __has_include
+#if __has_include(<arm_neon_sve_bridge.h>)
+#include <arm_neon_sve_bridge.h>
+#ifndef USE_SVE_HIST


Suggested change

#ifndef USE_SVE_HIST

#ifndef SONIC_USE_SVE_HIST

yangzhg · 2026-09-02T08:00:22Z

+#if __has_include(<arm_neon_sve_bridge.h>)
+#include <arm_neon_sve_bridge.h>
+#ifndef USE_SVE_HIST
+#define USE_SVE_HIST 1


missing #undefine

yangzhg · 2026-09-02T08:04:12Z

+      // only backslahes and no quotes, the whole 64 bytes are inside a string
+      // so we dont care about left & right
+      // just check if the last character is a backslash
+      prev_escaped = (p[63] == '\\');


This fast path tracks only whether the last byte of a 64-byte block is a backslash, but the escape state depends on the parity of consecutive trailing backslashes.

For example, consider a block ending with two backslashes, followed by a closing quote in the next block:

["<125 times 'a'>\\"]

The two backslashes represent one literal backslash, so the following quote is not escaped and should close the string. However, the current code sets prev_escaped = true, treats that quote as escaped, and may reject the valid JSON as an unclosed container.

emcastillo · 2026-09-30T03:11:05Z

@yangzhg all the issues should been solved :) thanks a lot for the awesome review

yangzhg

LGTM

emcastillo · 2026-10-10T04:28:43Z

@yangzhg thanks for landing this!!!

emcastillo changed the title ~~sve2 histcnt support for fast ondemand parsing~~ sve2 HISTCNT support for fast ondemand parsing on ARM May 11, 2025

emcastillo changed the title ~~sve2 HISTCNT support for fast ondemand parsing on ARM~~ sve2 HISTSEG support for fast ondemand parsing on ARM May 11, 2025

xiegx94 self-assigned this Jun 11, 2025

emcastillo force-pushed the sve-hist branch from 6603f46 to 36b675b Compare July 30, 2025 15:32

xiegx94 reviewed Aug 12, 2025

View reviewed changes

emcastillo force-pushed the sve-hist branch 2 times, most recently from d9a222b to 15ed14a Compare August 17, 2025 13:50

yangzhg reviewed Sep 2, 2026

View reviewed changes

emcastillo added 6 commits September 30, 2026 11:57

sve2 histcnt support for ondemand parsing

ec62d1e

Fixes

ca6f712

clang-format

33cf346

Review comments

1fd17e0

Clang format

754db78

Fix SVE HISTSEG review comments

e00f215

emcastillo force-pushed the sve-hist branch from dcf7c82 to e00f215 Compare September 30, 2026 03:04

yangzhg approved these changes Oct 10, 2026

View reviewed changes

yangzhg merged commit c8db640 into bytedance:master Oct 10, 2026
25 checks passed

Conversation

emcastillo commented May 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

xiegx94 commented Jun 11, 2025

Uh oh!

emcastillo commented Jul 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emcastillo commented Aug 17, 2025

Uh oh!

emcastillo commented Sep 26, 2025

Uh oh!

emcastillo commented Nov 28, 2025

Uh oh!

emcastillo commented Mar 6, 2026

Uh oh!

emcastillo commented Sep 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emcastillo commented Sep 30, 2026

Uh oh!

yangzhg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

emcastillo commented Oct 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

emcastillo commented May 11, 2025 •

edited

Loading

codecov-commenter commented May 12, 2025 •

edited

Loading