perf: three fixed per-call costs — concat-chain stack scratch, the Any/Any === tail, and the opaque write barrier

changelog.d/7885-fixed-per-call-costs.md

@@ -0,0 +1,51 @@
+### Three fixed per-call costs on the string-builder, generic-equality and write-barrier paths
+
+**1. `js_string_concat_chain` initialised ~2 KB of stack on every call, whatever the
+chain length.** The helper sized its scratch at the codegen cap of 32 parts, so a
+two-part chain paid the same `memset(_, _, 0x400)` for `num_bufs` plus 32 `str xzr` for
+the handle array as a 32-part one. Confirmed in the shipped release disassembly, not
+inferred: `sub sp, sp, #0x7e0` / `bl _memset` with `w1 = #0x400`. Real chains are 2-4
+parts — `seen = seen + "[" + names[i] + "]"` in an environment lookup is four, and the
+codegen fold (which does fire; that was checked) turns it into exactly one call per
+append. The body is now monomorphised on the scratch size and dispatched `n<=4` /
+`n<=8` / `<=32`, with `num_bufs` left `MaybeUninit` so only slots a numeric arm formats
+into are written.
+
+**2. A strict `===` whose operands are both statically unconstrained emitted one
+`js_eq` call and nothing else.** The shape that pays for this is a linear scan over a
+generic container's key array (`this.keys[i] === k` in a `Registry<K, V>`), which is
+dominated by *misses* — and a miss reaches `js_jsvalue_equals`'s pointer arm, which runs
+`resolve_forwarding` twice. An inline prefix now settles four cases without a call:
+identical bits (excluding a plain IEEE NaN), two SSO strings with different bits, two
+INT32s with different bits, and two `POINTER_TAG` values with distinct in-band addresses
+whose `GcHeader`s do not carry `GC_FLAG_FORWARDED`. The last is an exact restatement of
+`resolve_forwarding`'s "neither is forwarded, so fall through to 0", behind the same
+magnitude guard the runtime applies before any header dereference.
+
+**3. The opaque write-barrier wrapper had no value test.** `write_barrier_slot_inner`'s
+first action is `barrier_child_prologue(child)?`, so `js_write_barrier` does nothing at
+all for a non-pointer child — yet an array element store on a `number[]` emitted a bare,
+unconditional call next to every element write. #7511 put exactly this gate on the
+class-field slot store; `emit_write_barrier` never got it. It now goes behind
+`emit_may_carry_heap_pointer_check`, which is a deliberate superset of the runtime
+predicate (`gc::tests::inline_pointer_bearing_contract` enumerates the whole 16-bit tag
+space against it), so it can only skip calls that would have returned immediately.
+
+Two probes were added with the work: `gc-handoff/bench/strbuild.ts` (10M four-part
+concat chains) and `gc-handoff/bench/eqscan.ts` (2.4M generic-container key scans).
+
+Measured on the quiet M1 mini, best-of-5, interleaved, exit-checked, `VERDICT: CLEAN`
+(load 1.79 → 2.04, zero foreign processes at both ends). Absolute seconds:
+
+| bench | before | after | |
+|---|--:|--:|--:|
+| `strbuild` (concat probe) | 0.7448 | 0.5477 | −26.5% |
+| `eqscan` (key-scan probe) | 0.2181 | 0.1720 | −21.1% |
+| `iso_miss` | 1.2319 | 1.1283 | −8.4% |
+| `pipeline_big` | 2.5334 | 2.3291 | −8.1% |
+| `pipeline` | 0.2646 | 0.2444 | −7.6% |
+| the other 18 corpus programs | — | — | 0.978 – 1.005 |
+
+The 15 programs whose emitted IR contains no `js_string_concat_chain` call site and whose
+binaries are byte-identical across the two codegen arms set the run's noise floor at
+±0.5–2%; every mover is outside it.

crates/perry-codegen/src/expr/compare.rs

@@ -256,6 +256,181 @@ fn lower_string_literal_strict_eq(
     ctx.block().phi(I1, &incoming)
 }
 
+/// Magnitude comparands for the inline heap-address test in
+/// [`lower_strict_eq_inline_any`]. These mirror
+/// `perry-runtime::value::addr_class::{HANDLE_BAND_MAX, is_valid_obj_ptr}`:
+/// a `POINTER_TAG` payload below `HANDLE_BAND_MAX` is a registry id
+/// (net.Socket, fetch, zlib, revocable Proxy, UI widget), NOT an address, and
+/// dereferencing one reads unmapped low memory. Anything outside the window
+/// takes the runtime call instead of a header load.
+const HANDLE_BAND_MAX_I64: &str = "1048576";
+const HEAP_ADDR_CEILING_I64: &str = "140737488355328";
+
+/// Inline prefix for the generic `===`/`!==` tail — the arm where BOTH
+/// operands are statically unconstrained, which emitted one
+/// `js_eq` → `js_jsvalue_equals` call per comparison and nothing else.
+///
+/// The motivating shape is a linear scan over a generic container's key
+/// array: `this.keys[i] === k` in `gc-handoff/apps/pipeline.ts`'s
+/// `Registry<K, V>`, measured at 8.4% of that program. A scan is dominated by
+/// **misses**, so a fast path that settles only the hit is worth nothing —
+/// each case below settles one direction of the real traffic.
+///
+/// Four cases leave without a call. Each is an exact restatement of what
+/// `js_jsvalue_equals` computes for that input, not an approximation:
+///
+/// * **identical bits** ⇒ equal, *unless* the value is a plain (untagged)
+///   IEEE NaN. Perry's tags occupy top16 `0x7FF9..=0x7FFF`, so a tagged
+///   immediate (`undefined`, a pointer, an SSO string) stays equal to itself
+///   even though it *is* a NaN double, while canonical `0x7FF8…` NaN and the
+///   negative `0xFFF8…` NaN libm returns fall outside the band and take the
+///   call, which answers `false` (`NaN !== NaN`).
+/// * **both SSO strings, different bits** ⇒ different content. The SSO
+///   encoding is canonical — same bytes and same length give the same bit
+///   pattern — which is the argument `lower_string_strict_eq_inline` and the
+///   runtime's own both-short-string arm already rely on.
+/// * **both INT32, different bits** ⇒ different integers, same argument.
+/// * **both `POINTER_TAG`, different payloads, and neither header carries
+///   `GC_FLAG_FORWARDED`** ⇒ distinct objects. The runtime's pointer arm is
+///   `resolve_forwarding(a) == resolve_forwarding(b)`, and
+///   `resolve_forwarding` returns its argument unchanged when the forwarding
+///   bit is clear — so two *unforwarded* distinct addresses are exactly its
+///   `0` case. Anything forwarded (a post-`js_array_grow` alias, a stale
+///   pre-evacuation pointer) takes the call and gets the full walk. The
+///   header read is the same one `expr/array_push.rs` emits — `gc_flags` at
+///   `ptr - 7`, mask `GC_FLAG_FORWARDED` (0x80) — behind the same magnitude
+///   guard the runtime applies before any `GcHeader` dereference.
+///
+/// Everything else — a raw-bits module-level object slot (top16 zero), a heap
+/// string, a bigint, a mixed pair, a boxed wrapper — falls through to
+/// `js_eq`, which is what this site emitted unconditionally before.
+///
+/// Returns an i64 holding `TAG_TRUE`/`TAG_FALSE` (or `js_eq`'s own tagged
+/// boolean), i.e. the same value the bare call produced.
+fn lower_strict_eq_inline_any(ctx: &mut FnCtx<'_>, l: &str, r: &str) -> String {
+    let l_bits = ctx.block().bitcast_double_to_i64(l);
+    let r_bits = ctx.block().bitcast_double_to_i64(r);
+
+    let same_idx = ctx.new_block("anyeq.same");
+    let diff_idx = ctx.new_block("anyeq.diff");
+    let canon_idx = ctx.new_block("anyeq.canon");
+    let band_idx = ctx.new_block("anyeq.band");
+    let fwd_idx = ctx.new_block("anyeq.fwd");
+    let slow_idx = ctx.new_block("anyeq.slow");
+    let true_idx = ctx.new_block("anyeq.true");
+    let false_idx = ctx.new_block("anyeq.false");
+    let merge_idx = ctx.new_block("anyeq.merge");
+    let same_l = ctx.block_label(same_idx);
+    let diff_l = ctx.block_label(diff_idx);
+    let canon_l = ctx.block_label(canon_idx);
+    let band_l = ctx.block_label(band_idx);
+    let fwd_l = ctx.block_label(fwd_idx);
+    let slow_l = ctx.block_label(slow_idx);
+    let true_l = ctx.block_label(true_idx);
+    let false_l = ctx.block_label(false_idx);
+    let merge_l = ctx.block_label(merge_idx);
+
+    let same = ctx.block().icmp_eq(I64, &l_bits, &r_bits);
+    ctx.block().cond_br(&same, &same_l, &diff_l);
+
+    // Identical bits. Equal for every Perry tag and every non-NaN double.
+    ctx.current_block = same_idx;
+    let stag = ctx.block().lshr(I64, &l_bits, "48");
+    let tag_lo = ctx
+        .block()
+        .icmp_uge(I64, &stag, crate::nanbox::SHORT_STRING_TAG_TOP16_I64);
+    let tag_hi = ctx
+        .block()
+        .icmp_ule(I64, &stag, crate::nanbox::STRING_TAG_TOP16_I64);
+    let tagged = ctx.block().and(I1, &tag_lo, &tag_hi);
+    let is_nan = ctx.block().fcmp("uno", l, l);
+    let not_nan = ctx.block().xor(I1, &is_nan, "true");
+    let same_ok = ctx.block().or(I1, &tagged, &not_nan);
+    ctx.block().cond_br(&same_ok, &true_l, &slow_l);
+
+    // Different bits: only a same-tag pair whose encoding is canonical, or a
+    // pair of unforwarded heap pointers, is decidable here.
+    ctx.current_block = diff_idx;
+    let l_tag = ctx.block().lshr(I64, &l_bits, "48");
+    let r_tag = ctx.block().lshr(I64, &r_bits, "48");
+    let l_ptr = ctx
+        .block()
+        .icmp_eq(I64, &l_tag, crate::nanbox::POINTER_TAG_TOP16_I64);
+    let r_ptr = ctx
+        .block()
+        .icmp_eq(I64, &r_tag, crate::nanbox::POINTER_TAG_TOP16_I64);
+    let both_ptr = ctx.block().and(I1, &l_ptr, &r_ptr);
+    ctx.block().cond_br(&both_ptr, &band_l, &canon_l);
+
+    ctx.current_block = canon_idx;
+    let l_sso = ctx
+        .block()
+        .icmp_eq(I64, &l_tag, crate::nanbox::SHORT_STRING_TAG_TOP16_I64);
+    let r_sso = ctx
+        .block()
+        .icmp_eq(I64, &r_tag, crate::nanbox::SHORT_STRING_TAG_TOP16_I64);
+    let both_sso = ctx.block().and(I1, &l_sso, &r_sso);
+    let l_i32 = ctx
+        .block()
+        .icmp_eq(I64, &l_tag, crate::nanbox::INT32_TAG_TOP16_I64);
+    let r_i32 = ctx
+        .block()
+        .icmp_eq(I64, &r_tag, crate::nanbox::INT32_TAG_TOP16_I64);
+    let both_i32 = ctx.block().and(I1, &l_i32, &r_i32);
+    let canonical = ctx.block().or(I1, &both_sso, &both_i32);
+    ctx.block().cond_br(&canonical, &false_l, &slow_l);
+
+    // Both POINTER_TAG. Classify by magnitude before touching a header.
+    ctx.current_block = band_idx;
+    let l_addr = ctx.block().and(I64, &l_bits, POINTER_MASK_I64);
+    let r_addr = ctx.block().and(I64, &r_bits, POINTER_MASK_I64);
+    let l_above = ctx.block().icmp_uge(I64, &l_addr, HANDLE_BAND_MAX_I64);
+    let l_below = ctx.block().icmp_ult(I64, &l_addr, HEAP_ADDR_CEILING_I64);
+    let r_above = ctx.block().icmp_uge(I64, &r_addr, HANDLE_BAND_MAX_I64);
+    let r_below = ctx.block().icmp_ult(I64, &r_addr, HEAP_ADDR_CEILING_I64);
+    let l_heap = ctx.block().and(I1, &l_above, &l_below);
+    let r_heap = ctx.block().and(I1, &r_above, &r_below);
+    let both_heap = ctx.block().and(I1, &l_heap, &r_heap);
+    ctx.block().cond_br(&both_heap, &fwd_l, &slow_l);
+
+    ctx.current_block = fwd_idx;
+    let l_flags_addr = ctx.block().sub(I64, &l_addr, "7");
+    let l_flags_ptr = ctx.block().inttoptr(I64, &l_flags_addr);
+    let l_flags = ctx.block().load(I8, &l_flags_ptr);
+    let r_flags_addr = ctx.block().sub(I64, &r_addr, "7");
+    let r_flags_ptr = ctx.block().inttoptr(I64, &r_flags_addr);
+    let r_flags = ctx.block().load(I8, &r_flags_ptr);
+    let either = ctx.block().or(I8, &l_flags, &r_flags);
+    // GC_FLAG_FORWARDED = 0x80; LLVM i8 literals are signed.
+    let fwd_bits = ctx.block().and(I8, &either, "-128");
+    let no_fwd = ctx.block().icmp_eq(I8, &fwd_bits, "0");
+    ctx.block().cond_br(&no_fwd, &false_l, &slow_l);
+
+    ctx.current_block = slow_idx;
+    let slow_res = ctx
+        .block()
+        .call(I64, "js_eq", &[(I64, &l_bits), (I64, &r_bits)]);
+    let slow_pred = ctx.block().label.clone();
+    ctx.block().br(&merge_l);
+
+    ctx.current_block = true_idx;
+    let true_pred = ctx.block().label.clone();
+    ctx.block().br(&merge_l);
+    ctx.current_block = false_idx;
+    let false_pred = ctx.block().label.clone();
+    ctx.block().br(&merge_l);
+
+    ctx.current_block = merge_idx;
+    ctx.block().phi(
+        I64,
+        &[
+            (crate::nanbox::TAG_TRUE_I64, &true_pred),
+            (crate::nanbox::TAG_FALSE_I64, &false_pred),
+            (&slow_res, &slow_pred),
+        ],
+    )
+}
+
 /// Inline prefix for the `===`/`!==` string arms that have **no** literal
 /// operand — `names[i] === name` in an environment lookup, say.
 ///
@@ -655,17 +830,22 @@ pub(crate) fn lower(ctx: &mut FnCtx<'_>, expr: &Expr) -> Result<String> {
             if either_non_numeric && only_eq && unknown_l && unknown_r {
                 let l = lower_expr(ctx, left)?;
                 let r = lower_expr(ctx, right)?;
-                let blk = ctx.block();
                 // Use js_loose_eq for == / != (handles null==undefined,
-                // cross-type coercion). Use js_eq for === / !==.
-                let eq_fn = if matches!(op, CompareOp::LooseEq | CompareOp::LooseNe) {
-                    "js_loose_eq"
+                // cross-type coercion). STRICT `===`/`!==` gets the inline
+                // prefix instead: the operands that reach here are
+                // unconstrained, and a generic-container key scan
+                // (`this.keys[i] === k`) spends its whole cost on this one
+                // call. Loose `==`'s cross-type coercions are not
+                // bit-decidable, so it keeps the bare call.
+                let result_bits = if matches!(op, CompareOp::LooseEq | CompareOp::LooseNe) {
+                    let blk = ctx.block();
+                    let l_bits = blk.bitcast_double_to_i64(&l);
+                    let r_bits = blk.bitcast_double_to_i64(&r);
+                    blk.call(I64, "js_loose_eq", &[(I64, &l_bits), (I64, &r_bits)])
                 } else {
-                    "js_eq"
+                    lower_strict_eq_inline_any(ctx, &l, &r)
                 };
-                let l_bits = blk.bitcast_double_to_i64(&l);
-                let r_bits = blk.bitcast_double_to_i64(&r);
-                let result_bits = blk.call(I64, eq_fn, &[(I64, &l_bits), (I64, &r_bits)]);
+                let blk = ctx.block();
                 let result = blk.bitcast_i64_to_double(&result_bits);
                 if matches!(op, CompareOp::Ne | CompareOp::LooseNe) {
                     let cmp = blk.icmp_eq(I64, &result_bits, crate::nanbox::TAG_TRUE_I64);

crates/perry-codegen/src/expr/write_barrier.rs

@@ -41,8 +41,33 @@ pub(crate) fn emit_write_barrier(ctx: &mut FnCtx<'_>, parent_bits: &str, child_b
         false,
         Vec::new(),
     );
+    // Same gate #7511 put on the class-field slot store, applied to the
+    // opaque-store wrapper. `write_barrier_slot_inner`'s FIRST action is
+    // `barrier_child_prologue(child)?` — a non-pointer child returns before it
+    // touches the incremental-mark latch, the parent decode or the remembered
+    // set — so for every numeric store this call does nothing but cost a call.
+    //
+    // An array element store pays it unconditionally today: `this.vals[i] = v`
+    // in `gc-handoff/apps/pipeline.ts`'s `Registry` emits
+    // `js_typed_feedback_array_set_f64_extend` immediately followed by a bare
+    // `js_write_barrier`, on a `number[]`.
+    //
+    // `emit_may_carry_heap_pointer_check` is a deliberate SUPERSET of the
+    // runtime predicate (its doc records why the direction is load-bearing,
+    // and `gc::tests::inline_pointer_bearing_contract` enumerates the whole
+    // 16-bit tag space against it), so this can only skip calls that would
+    // have returned immediately.
+    let maybe_idx = ctx.new_block("wb.maybe");
+    let done_idx = ctx.new_block("wb.done");
+    let maybe_l = ctx.block_label(maybe_idx);
+    let done_l = ctx.block_label(done_idx);
+    let may_carry = emit_may_carry_heap_pointer_check(ctx.block(), child_bits);
+    ctx.block().cond_br(&may_carry, &maybe_l, &done_l);
+    ctx.current_block = maybe_idx;
     ctx.block()
         .call_void("js_write_barrier", &[(I64, parent_bits), (I64, child_bits)]);
+    ctx.block().br(&done_l);
+    ctx.current_block = done_idx;
 }
 
 pub(crate) fn emit_write_barrier_slot_on_block(

crates/perry-codegen/src/nanbox.rs

@@ -64,6 +64,11 @@ pub const SHORT_STRING_TAG_TOP16_I64: &str = "32761";
 /// against the u64 tags in `tag_strings_match_u64_values`.
 pub const POINTER_TAG_TOP16_I64: &str = "32765";
 pub const BIGINT_TAG_TOP16_I64: &str = "32762";
+/// `INT32_TAG >> 48`. Used by the inline `===` prefix, which settles two
+/// INT32-tagged operands by bit inequality — the encoding
+/// `INT32_TAG | (v as u32 as u64)` is canonical, so different bits are
+/// different integers. Asserted in `tag_strings_match_u64_values`.
+pub const INT32_TAG_TOP16_I64: &str = "32766";
 
 /// Format a `u64` as a signed LLVM i64 literal (LLVM IR integer literals are signed).
 pub fn i64_literal(v: u64) -> String {
@@ -126,6 +131,7 @@ mod tests {
         );
         assert_eq!(i64_literal(POINTER_TAG >> 48), POINTER_TAG_TOP16_I64);
         assert_eq!(i64_literal(BIGINT_TAG >> 48), BIGINT_TAG_TOP16_I64);
+        assert_eq!(i64_literal(INT32_TAG >> 48), INT32_TAG_TOP16_I64);
     }
 
     /// #7511 — the inline pointer-bearing test emitted at class-field stores