Skip to content

Port runtime improvements with Android and NGE TeaVM coverage - #2

Merged
riccardobl merged 14 commits into
NostrGameEngine:mainfrom
toaster0123:integrate/verified-upstream-and-soxdev
Oct 27, 2026
Merged

riccardobl merged 14 commits into
NostrGameEngine:mainfrom
toaster0123:integrate/verified-upstream-and-soxdev

Conversation

@toaster0123

@toaster0123 toaster0123 commented Oct 27, 2026 •

Copy link
Copy Markdown

Conditional/table-only interpreter loops can ignore thread interruption, valid active/passive element references can be rejected, and JVM compilation can fail on equivalent reference bridges or unreachable/nested exception handlers. Fix these cases and reduce interpreter allocation and JVM dispatch overhead while retaining NGE's paged memory, atomic/parser implementations and org.ngengine.chicory coordinates. Add explicit allocated-page validation so scalar out-of-bounds accesses keep raising Wasm traps after compilation by TeaVM, where implicit JVM array/null checks may be omitted.

All integration commits are authored by Mr Robot toaster@rblb.it. Upstream production fixes are selected from 43d9100b; JIT-aware direct dispatch is adapted from ea8351c4 with bounded configuration values, generated-method size checks, documentation and a reproducible JMH workload.

soxdev branch Decision
main Identical to the NGE base.
fix/upstream-runtime-771 Adopt 7679b1d1: independently reproduced interruption and element-declaration fixes.
fix/operands-allocation-807 Adopt f1154c78: consume existing instructions in built-in handlers while retaining protected Operands extension hooks.
experiment/control-frame-reuse-817 Adopt and qualify a1f0240e: immutable reuse bounded to 64 inactive control frames per activation.
fix/tdb-lua-guard-503 Defer the application-specific guarded compiler SPI and its dependent fixes/tests.
archive/source-20261026/work/guarded-stage2-606 Defer the explicitly unqualified splitting checkpoint.
work/shared-local-exits-20261025 Defer the guarded splitting/custody provider and its follow-ups pending separate provider qualification and application-level measurements.

Upstream tail-call trampoline 1303ad00 is excluded: a cross-instance indirect tail call returns 42 with the original compiler but throws ChicoryException with that candidate. Regression coverage preserves calls to both compiled and interpreted foreign instances. Unrelated upstream documentation, dependency, CI and fuzzer rewrites are omitted.

Validation:

  • mvn -B -ntp -pl runtime-tests,compiler-tests,machine-tests,simd,jmh -am verify -Dmaven.javadoc.skip=true: all 21 reactor modules succeeded, including formatting, Checkstyle, Error Prone and dependency checks.
  • Compiler specification suite: 34,003 cases per run, 88 existing skips, zero failures/errors in both classpath and module-path executions.
  • Interpreter/SIMD specification suite: 62,144 cases per run, 83 existing skips, zero failures/errors in both executions. Machine tests: 87 passing cases per execution.
  • Java 11: 25 targeted runtime, parser and compiler regressions passed on Temurin 11.0.28.
  • NGE TeaVM org.ngengine:0.16.0-SNAPSHOT: compiled the actual parser/interpreter to JavaScript and ran it in headless Chrome; 10,000 loop iterations, 10,000 interpreter tail calls, active ref.func, memory reads/writes and bounds traps passed.
  • JDK 25 allocation probe, 10,000-iteration interpreter workload: median 640,344 → 320,384 allocated bytes/call with default escape analysis (~50% reduction).
  • JMH 1.37 / JDK 25, 2,000-function direct dispatch at indices 0/999/1999: 6.25–6.40 ns/op with default sizing versus 43.60–46.16 ns/op with legacy chunk dimensions (~6.8–7.4x in this microbenchmark; one fork, three 1-second warmups, five 1-second measurements).

Android and TeaVM compatibility:

  • Pin WebAssembly/wasi-testsuite to 6345da2237c562d9a94e281332c653b5528fdd52, the last prebuilt revision with the specification format supported by the generator. Keep both CI checkouts and the downloader aligned; archives now accept commit refs. The moving branch introduced root and broke Maven preparation before Android tests started.
  • Replace ByteArrayOutputStream.writeBytes in the new runtime regression fixtures with the compatible write(byte[], offset, length) API. Android API 28 reported 13 NoSuchMethodError failures with the Java 11 convenience method.
  • Add a required-failure TeaVM workflow matrix for javascript and wasm-gc, using published org.ngengine:teavm-* 0.16.0-SNAPSHOT artifacts from the NGE fork with snapshot refresh enabled.
  • Each target executes both externally supplied Wasm bytes through the interpreter and real Chicory build-time generated classes. The generated .meta is bundled through the NGE resource policy SPI; JVM compiler/ASM remain build-time dependencies.
  • Both memory implementations are exercised for loops, tail calls, function references, growth, cross-page access, scalar reads/writes and bounds traps. Precompiled tail-call depth is 128 because the current compiler uses the host stack; interpreted depth is 10,000. Node enables --experimental-wasm-exnref for the exception-handling instructions emitted by this TeaVM version.

Follow-up local validation:

  • WASI generation and tests: 74 tests passed, including a fresh download of the pinned archive.
  • Runtime unit tests: 87 passed, including the new scalar-bounds/cross-page tests and Android-compatible fixtures. A separate Java 17 launcher passed 25 targeted regressions.
  • NGE TeaVM JavaScript and Wasm GC Maven verification both passed, including generated application execution in Node.js. All four target/execution-mode combinations also passed in headless Chrome 155.0.8059.85 with default browser flags.
  • Reproduced the precompiled bounds failure on both targets before the page check: JavaScript TypeError and Wasm GC host array trap. Both now produce the expected Wasm runtime exception.
  • The interpreter/SIMD specification suite was re-run against the memory fix: 62,144 cases found, 62,061 passed, 83 existing skips, zero test/container failures.
  • Workflow syntax validated with actionlint 1.7.12.
  • Remote branch verified at 00102c7a26dc7a0c9d808f4b4c16f0e3556c7399; local/remote divergence 0 0.

CI audit follow-up:

  • The previous overall CI result concealed two failing jobs through job-level continue-on-error: Linux ARM Java 11 and 17 failed SpecV1CommentsTest while validating a quoted WAT module in embedded wasm-tools. The remaining 18 CI jobs passed. The failure was reproduced on both jobs in run 40510508884.
  • Make every matrix/test-results failure blocking. Preserve the original WAT conversion cause and upload Surefire reports, the generated comments fixture, generated/runtime JARs, and ARM CPU/JVM diagnostics on failure.
  • Throw scalar allocated-page traps directly instead of calling the generic range checker after the page has already been found out of range. The nested guard regresses execution on older ARM JVMs with SVE and SIMD memory copies enabled. Retain the explicit page check required for TeaVM's scalar bounds traps.
  • Cover LF, CR and CRLF comments together in one quoted module.
  • Reproduced the exact four CI errors locally with the failed runner's runtime/wasm-tools JARs, Temurin 17.0.21+9 and emulated 128-bit SVE with UseSIMDForMemoryOps enabled. Disabling either feature passes. The original page guard fails; changing only that guard passes all 36,158 specification cases (36,075 passing, 83 existing skips), twice, with both features enabled. No JVM optimization is disabled in CI.
  • Local validation passed: 87 runtime unit tests per classpath/module-path execution and both NGE TeaVM JavaScript/Wasm GC Maven verifications, including runtime-loaded and precompiled module execution in Node.js and bounds-trap checks.
    Remote validation on 00102c7a:
  • Linux ARM Java 11 and 17 now pass the full build, including 36,158 runtime specification cases per classpath/module-path execution with 83 existing skips and zero failures/errors: Java 11, Java 17.
  • Android API 28/35, NGE TeaVM JavaScript/Wasm GC, and performance workflows passed. Both Android emulators ran 86 tests; both TeaVM jobs executed runtime-loaded and build-time compiled modules.
  • The main CI run is not entirely green: 15 jobs passed; Windows ARM Java 21 has successful test/cleanup steps but is still awaiting job completion. macOS Java 17/25 were cancelled before checkout because GitHub could not allocate hosted ARM64 runners; macOS Java 11/21 remain queued. Runner annotations explicitly report macOS ARM64 capacity constraints. The Toaster account has Triage permissions and GitHub denied the job rerun with HTTP 403. A repository maintainer must rerun the affected macOS jobs once capacity is available.

This compatibility workload does not claim complete WebAssembly specification coverage inside TeaVM.

Port the Validator fix from upstream 43d9100 and soxdev 7679b1d1. Active and passive elements declare function references just as declarative elements do. Regression tests reproduced both rejected valid modules before the fix.
Port the interpreter fix from upstream 43d9100 and soxdev 7679b1d1. Real BR_IF-only and BR_TABLE-only loops now stop on thread interruption. Both regressions failed on the original implementation with bounded worker cleanup.
Port soxdev f1154c78. Read operands directly from the existing instruction while retaining the protected CALL, evalDefault and readMemPtr Operands hooks. Cover listener order, retained extension operands, memory failures and numeric/GC handlers.
Port and qualify soxdev a1f0240e. Reuse at most 64 inactive control frames only when every immutable field matches. Preserve detached frames, deep stack behavior, tail calls and exception unwinding. The 10000-iteration interpreter workload reduces median allocated bytes per call from 640344 to 320384 on JDK 25 with default escape analysis.
Select the production compiler fixes from upstream 43d9100. Deduplicate equivalent JVM multi-value bridges, register only reachable exception ranges with inner handlers first, and handle already-exited branch scopes. The original compiler reproduced duplicate-method ClassFormatError and ASM frame-analysis NullPointerException on the added binary fixtures. Compare execution with the interpreter and preserve cross-instance indirect tail calls to both compiled and interpreted targets.
Port upstream ea8351c while bounding chunk sizes for zero, negative and extreme chicory.hugeMethodLimit values. Keep indirect-call chunking unchanged. Check generated method sizes and dispatch across chunk boundaries, and include the upstream JMH workload. On JDK 25, the 2000-function microbenchmark measured 6.25-6.40 ns/op with the default budget versus 43.60-46.16 ns/op with the previous chunk dimensions.
@toaster0123 toaster0123 changed the title Port verified runtime and compiler improvements Port runtime improvements with Android and NGE TeaVM coverage Oct 27, 2026
@riccardobl
riccardobl merged commit c609354 into NostrGameEngine:main Oct 27, 2026
23 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants