Skip to content

feat: add static Go 1.27 simd/archsimd backend (fixes #17) - #18

Open
hazyhaar wants to merge 1 commit into
cloudwego:mainfrom
hazyhaar:feat/go127-simd
Open

hazyhaar wants to merge 1 commit into
cloudwego:mainfrom
hazyhaar:feat/go127-simd

Conversation

@hazyhaar

Copy link
Copy Markdown

Description

Fixes #17.

This pull request introduces a native, static SIMD backend for Go 1.27+ (GOEXPERIMENT=simd via simd/archsimd), resolving the fatal initialization crash and enabling base64x in hardened security environments enforcing W^X policies.

Problem Addressed

  1. Go 1.27 Crash: In Go 1.27, updated internal runtime symbol tables cause bytedance/sonic/loader (WrapGoC) to abort the process immediately at package initialization (fatal error: invalid function symbol table in runtime.moduledataverify1).
  2. W^X / MemoryDenyWriteExecute Incompatibility: Hardened runtime environments (SELinux execmem disabled, systemd MemoryDenyWriteExecute=yes, PaX MPROTECT, iOS) explicitly forbid transitioning memory from writable to executable via mprotect(PROT_READ|PROT_EXEC), causing an unrecoverable panic.

Key Changes

  • Go 1.27+ Static Backend (base64x_simd_amd64.go): Conditioned on //go:build goexperiment.simd && amd64 && !noasm && !appengine. Uses pure ahead-of-time Go code with native simd/archsimd intrinsics (Wojciech Muła / Daniel Lemire AVX2 algorithm).
  • Pure AVX2 Kernel (internal/simd): Ahead-of-time vector kernel for standard and URL base64 alphabets (with and without padding).
  • Zero-Allocation Safety: Handled safe capacity guards for 32-byte chunk writes to prevent memory corruption when destination slices have tight capacities.
  • Strict Backward Compatibility:
    • Build tag !goexperiment.simd added to base64x_native.go and base64x_subr_native_amd64.go.
    • Go $\le$ 1.26 continues to run the existing bytecode loader unmodified.
    • Linkname symbols (_subr__b64decode, _subr__b64encode) remain preserved.

Verification & Performance

  • Bit-Exact Parity: 100% bit-exact against GCC -O2 -mavx2 (ASAN/UBSAN) on thousands of randomized vectors and differential testing against encoding/base64.
  • Passes all tests: Full test suite, CRLF, JSON unescaping, error detection, and fuzzing passes with -race:
    GOEXPERIMENT=simd go test -v -race .
    # PASS: TestUnquoteJSONBase64
    # PASS: TestUnquoteJSONBase64RejectsInvalidEscapes
    # PASS: TestFallbackDecodeMissingPaddingReturnsStdlibError
    # PASS: TestSubrCompatibilitySymbols
    # PASS: TestEncoder
    # PASS: TestDecoder
    # PASS: TestDecoderCRLF
    # PASS: TestDecoderJSON
    # PASS: TestDecoderError
    # PASS: FuzzMain
  • Throughput (i9-14900K, 0 B/op, 0 allocs):
    • Encoding: 5.5 to 6.5 GB/s (single-core) / ~19.6 GB/s (parallel 4kB benchmark)
    • Decoding: 3.2 to 3.8 GB/s (single-core) / ~2.4 GB/s (parallel benchmark)

@CLAassistant

CLAassistant commented Sep 26, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Fatal crash at init under Go 1.27 (runtime.moduledataverify1) and panic under W^X (MemoryDenyWriteExecute)

2 participants