perf(server): jemalloc global allocator + fat LTO release profile - #939
Conversation
Two compile-time changes, measured together on the c=128 saturation grid (same-rig before/after in the PR): - tikv-jemallocator 0.6 as the global allocator, gated to the targets we ship and bench (Linux glibc). Flamegraphs put glibc malloc/free at 15.5% of request CPU under saturation; jemalloc runs the same load at ~6%. - release profile thin -> fat LTO. codegen-units was already 1 and opt-level stays at the release default of 3. Other targets (macOS dev builds, musl) keep the system allocator. strip stays at "debuginfo" so shipped binaries remain profileable in the field (#847), and panic stays unwind for per-request isolation.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (5)
📝 WalkthroughWalkthroughRelease builds now use fat LTO. Linux GNU server builds use jemalloc as the global allocator. Other targets retain the system allocator. Docker image checks verify that jemalloc symbols are present. ChangesRelease optimization
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related issues
Suggested reviewers: 🚥 Pre-merge checks | ✅ 6✅ Passed checks (6 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds Linux GNU performance optimizations to the shipped AISIX binary.
Changes:
- Uses jemalloc globally on Linux GNU targets.
- Enables fat LTO for release builds.
- Updates dependency locking for jemalloc.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
Cargo.toml |
Enables fat LTO in release builds. |
Cargo.lock |
Locks jemalloc dependencies. |
crates/aisix-server/Cargo.toml |
Adds target-gated jemalloc dependency. |
crates/aisix-server/src/main.rs |
Configures jemalloc as the global allocator. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Audit follow-ups for the allocator PR: - The #847 symbol-table step now also requires prefixed _rjem_ symbols, so a future drift of the cfg gates in aisix-server cannot silently fall back to glibc malloc (the regression would cost the throughput this change bought with no other signal). - Dockerfile: note the JEMALLOC_SYS_WITH_LG_PAGE=16 requirement next to the release build RUN, where a future linux/arm64 buildx target would otherwise bake QEMU's 4K page size into the image.
|
Independent cold-start audit ran against this PR (correctness / reliability / security / leakage / breaking / coverage). Verdict: no HIGH; 2 MEDIUM, 3 LOW. Disposition:
Audit also verified: cfg-gate consistency across targets, lockfile checksums against crates.io (no advisories, nothing yanked), no fork() users in the server (atfork safety moot), release matrix claims (no musl/static artifacts, codegen-units already 1, strip/#847 contract intact), and that CI exercises the gated cfg at runtime (unit + e2e run the bin on x86_64-linux-gnu). |
|
Closing the audit's remaining LOW: the fat-LTO docker build measured 10m38s on this PR's run (thin-LTO history was 7–9 min) — well inside |
|
@coderabbitai full review |
✅ Action performedFull review finished. |
What
Two compile-time changes to the shipped binary, no code-path or default-behavior changes:
tikv-jemallocator0.6, the maintained binding, default features) for the aisix binary, gated to the targets we ship and bench:cfg(all(target_os = "linux", target_env = "gnu"))— the Docker image (bookworm, glibc) and both supported production arches (x86_64, aarch64). Every other target (macOS dev builds, musl, msvc) keeps the system allocator.lto = "thin"→"fat".codegen-unitswas already 1;opt-levelstays at the release default of 3;stripstays"debuginfo"so shipped binaries remain profileable in the field (Establish the on-CPU profiling workflow: adopt cargo-flamegraph and add a dedicated profiling build profile #847);panicstaysunwindfor per-request isolation.Why
The performance program's earlier flamegraphs at the c=128 saturation point put glibc
malloc/freeat 15.5 % of request CPU (prior rig, where glibc internals symbolized fully); the composite spike leg for AISIX-Cloud#1259 item 4 then measured the allocator swap alone at −11.2 µs/req (+9.2 % rps) on an anchored same-scene basis — the largest single verified item in the program, at one dependency + one attribute of code.This is the standard posture across mainstream Rust network infrastructure: maintained jemalloc bindings as the global allocator (gated off non-shipped targets) are shipped by mainstream Rust data stores, proxies, and AI gateways alike, because glibc malloc's arena locking and cache behavior under many-threads/many-small-allocations workloads is a known tax that jemalloc's per-thread caching removes. Fat LTO +
codegen-units = 1is likewise the common release-profile shape for latency-sensitive Rust servers; we deliberately do not adopt the fullstrip = truesome projects pair with it, because field profileability of the exact shipped binary is a supported workflow here (#847).Measured result (same rig, anchored before/after)
All legs c=128 saturation + c=768 @ 10 ms TTFT, 4 valid windows each, zero failed requests and zero invalid windows in the whole scene, gateway pinned to 4 cores at 399.3–399.4 % CPU, front/back anchor drift +0.07 % (far inside the ±5 % noise band):
Shape sanity at c=768 @ 10 ms TTFT (deep queue): 27,548 → 31,396 rps (+14.0 %), 144.8 → 127.0 µs/req, anchor drift on the point +0.01 % — the win holds under queue depth, no shape anomaly.
Memory (an allocator swap must answer this): idle RSS 117.2 → 112.7 MB (−4.5 MB); peak RSS (VmHWM, includes the 768-connection point) 203.1 → 219.2 MB (+16.1 MB / +7.9 %). The peak growth is jemalloc’s documented decay-based purging — dirty pages are returned on a decay schedule instead of immediately — and is the deliberate trade for the allocation-path speed.
Flamegraph (same scene, self-time accounting over allocator symbols): glibc allocator frames 4.10 % / 3.78 % on the two base legs → 0.13 % on this branch (the residue is C-dependency allocations, which deliberately stay on glibc); jemalloc frames appear at 2.70 % self. Accounting caveat, stated for honesty: on this rig’s kernel the DWARF unwind through glibc breaks into
[unknown]towers, so SVG-visible glibc shares undercount the true allocator cost — the quantitative claim rests on the anchored CPU counters above, the flamegraph is corroborating shape evidence (glibc allocator symbols vanish, jemalloc symbols appear at the mature share the composite-spike leg also showed).The composite-spike leg (measured on the #925-head base) predicted −11.2 µs for jemalloc alone, fat LTO worth ~2 µs more. The landing measures −16.7 µs on a base nine feature commits newer — the same direction with the allocator win growing as the request path gains per-request work, which is exactly how an allocator-side win should scale.
Compile-time cost (the trade this buys)
The release binary shrinks from 57.3 MiB to 52.5 MiB (−8.3 %): whole-graph LTO prunes more than the embedded jemalloc adds. Dev builds are untouched (profile.release only). The cost lands on release builds: CI docker builds and bench-rig builds take the ~2× cold hit; the touch-rebuild ~3.5× is felt only when iterating on release binaries locally.
Compatibility checklist
JEMALLOC_SYS_WITH_LG_PAGE; verified in the crate's build.rs — no per-arch override on linux). Our release matrix is unaffected: the published Docker image is x86_64 (4 K everywhere), and aarch64 deployments build from source on the deploy host, so configure autodetects the right value. The one hazard is cross-building an aarch64 binary on a 4 K-page host and running it on a 64 K-page kernel (RHEL-family aarch64): jemalloc aborts at startup with an explicit page-size error. Documented in the code comment; escape hatch isJEMALLOC_SYS_WITH_LG_PAGE=16at build time, which yields a binary that runs on both page sizes.linux-gnu, so a hypothetical musl build simply has no jemalloc in its graph and links exactly as before with the system allocator.background_threads_runtime_support;background_thread:trueis only injected by the non-defaultbackground_threadsfeature — verified in the crate build.rs). No extra threads appear in the process; this matches the spike leg byte-for-byte._RJEM_MALLOC_CONFvs glibc tunables, aarch64 source-build note). CI now positively asserts the jemalloc symbols are present in the shipped image (the Establish the on-CPU profiling workflow: adopt cargo-flamegraph and add a dedicated profiling build profile #847 verify step).unprefixed_malloc_on_supported_platformsfeature, jemalloc serves Rust's#[global_allocator]only; C-library allocations inside dependencies keep glibc. Identical to the measured spike leg.Tests
Full workspace suite: 3,017 tests pass, clippy/fmt clean. One environmental failure documented for honesty:
aisix-mcp::bridge::tests::connect_timeout_bounds_an_unreachable_upstreamfails on the author's dev box only because a local transparent-proxy TUN answers SYNs for the reserved TEST-NET-3 black-hole address in 4 ms, defeating the test's environmental assumption; it passes in CI and is untouched by this diff (the crate is byte-identical; the diff touches only the server bin crate and the release profile).Summary by CodeRabbit
Performance
Compatibility
Reliability