diff --git a/CLAUDE.md b/CLAUDE.md index 279c0bfabd..e69de29bb2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,254 +0,0 @@ -# CLAUDE.md - -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. - -**NOTE**: Keep this file concise. Detailed changelogs live as `changelog.d/` fragments (folded into GitHub Release notes at each tag); `CHANGELOG.md` is a frozen archive (≤ v0.5.1264). - -## Project Overview - -Perry is a native TypeScript compiler written in Rust that compiles TypeScript source code directly to native executables. It uses SWC for TypeScript parsing and LLVM for code generation. - -**Current Version:** 0.5.1287 - - -## TypeScript Parity Status - -Tracked via the gap test suite (`test-files/test_gap_*.ts`). Compared byte-for-byte against `node --experimental-strip-types`. Run via `./scripts/run_gap_tests.sh` (a thin wrapper over `run_parity_tests.sh --filter test_gap_` that builds the compiler itself and gates on no new untriaged failures). - -**The oracle is Node `26.5.0`, pinned in `.node-version` at the repo root** — the single source of truth every CI workflow reads via `setup-node`'s `node-version-file`. **Run the gap suite against that exact version locally**, or your results won't match CI. The version is a *correctness input*, not an incidental toolchain detail: when node can't run a test (a feature newer than the pinned node), node exits non-zero, the harness classifies it `node_fail`, and the test is **silently dropped from the gate** rather than going red. CI sat on Node 22 while the suite grew Node 24/26 features, which hid 14 tests — all of Temporal, plus DisposableStack, Float16Array, and `Uint8Array` base64/hex (#6364). Node patch releases also change observable output (error-message text, `v8` heap fields), which is why the pin is exact. Raising it is a deliberate act: measure the failure delta under both oracles first, then triage what it exposes. - -Two workflows are deliberately exempt and say so inline: `node-core-subset.yml` derives its Node from `test-compat/node-core/pinned-version.txt` (it runs Node's *own* test corpus, which must match its own Node line), and the two release workflows use Node purely as an npm *publishing* toolchain. - -**Last full sweep:** run `./run_parity_tests.sh` for the current snapshot. The umbrella tracker is #793 (Node.js + TypeScript compatibility roadmap); the previously-cited #447–#452 batch closed on 2026-05-04. Currently-open trackers worth knowing about: - -- **Effect framework end-to-end (#321)** — `#684` (Schema.ts ~310th-init `(number).slice` regression) and `#809` (object-literal computed-keys + cross-module spread) are the live HashRing/Schema blockers. -- **Async context** — `AsyncLocalStorage` (real tracking across `await`/microtasks/timers, `#788`) and `async_hooks.createHook` (real lifecycle + asyncId, `#789`) both landed (closed 2026-05-16); these are no longer stubs. -- **Compile-as-package** — `#348` (ink TUI end-to-end), `#488/#489` (Drizzle + MySQL), `#678` (linker emits native callsites for V8-fallback modules). -- **Test/CI mechanics** — `#794` (per-category parity thresholds), `#796` (gap-suite output truncation + O(n²) `normalize_output`), `#812` (42-module behavioral matrix), `#806/#807/#808` (test harnesses for mixins / async context / ≥300-init scale). -- **Skip-list audit** — `#797` covers `test-parity/known_failures.json` provenance (issue # + date per entry). - -### Node builtin compatibility matrix (`scripts/node_compat_matrix.mjs`) - -Breadth sweep over EVERY `require("module").builtinModules` entry, both import forms (`M` and `node:M`), against a **pinned, SRI-verified Node** (the "latest stable" oracle, pinned in `external-tools.json` `tools.node.version` — currently **26.5.1**, independent of the `.node-version` gap-suite oracle). It compares Perry's export-SHAPE fingerprint (sorted `name:typeof` over the module namespace + the default export's typeof) to the oracle's. This is the systematic version of the #812 "42-module behavioral matrix" — shape, not deep behavior (behavioral cases stay in the node-suite). - -```bash -# FAST LOOP — reach for this first when iterating on ONE builtin: -node scripts/node_compat_matrix.mjs --module fs # one module, both forms -node scripts/node_compat_matrix.mjs --module fs,path,crypto # a few -node scripts/node_compat_matrix.mjs --module fs --method readFileSync,promises # only these exports -node scripts/node_compat_matrix.mjs --only fs.readFileSync,path.join # combined mod.export form -# (the pinned Node download is skipped once cached under .cache/node-pin/) - -# FULL SWEEP + GATE: -node scripts/node_compat_matrix.mjs # whole matrix + summary table -node scripts/node_compat_matrix.mjs --check # CI gate: exit 1 on regressions vs the baseline -node scripts/node_compat_matrix.mjs --update-baseline # rewrite test-parity/node-compat-matrix.baseline.json -``` - -A `--module` selector scopes `--check`/`--update-baseline` to just that slice (a single-module refresh never rewrites the whole baseline). A `--method`/`--only` subset is a print-only fast diagnostic (it narrows the fingerprint, so it is refused for `--check`/`--update-baseline`). **Bump the pinned Node** by editing `tools.node.version` in `external-tools.json` (add per-platform sha512 SRI), then `--update-baseline` and review the diff. Needs the release binary (`cargo build --release -p perry`). Full page: `docs/src/testing/node-compat-matrix.md`. - -**Known categorical gaps**: `console.dir`/`console.group*` formatting, lone surrogate handling (WTF-8). (Lookbehind regex is NOT a gap anymore: `perry-runtime/src/regex.rs` falls back from the `regex` crate to `fancy-regex` for lookbehind/backreferences, with capture-group translation and replacement expansion.) - -## Workflow Requirements - -**Default flow is PR-based.** `main` is protected: pushes require a pull request, CI must pass (`lint`, `cargo-test`, `api-docs-drift`, `security-audit`), and only squash or rebase merges are allowed (no merge commits, linear history enforced). `parity` and `compile-smoke` are gated to tag pushes only (v0.5.1018) — they no longer run on PRs but still gate the release-packages.yml publish step. Admins can bypass for hotfixes/version bumps, but the standard path is: - -1. Branch from `main`, push, open a PR. -2. Wait for required checks to go green. -3. Squash- or rebase-merge. The PR branch auto-deletes on merge. - -**For every change that lands on `main`** (whether via PR or admin bypass): - -1. **Bump version**: Increment patch in `[workspace.package].version` in `Cargo.toml` and the `**Current Version:**` line above. That is the ONLY metadata edit CLAUDE.md needs. -2. **Add a changeset**: create `changelog.d/-.md` with the entry body (no version header — see `changelog.d/README.md`). Long-form root-cause writeups, file paths, validation notes all belong in the fragment, NOT in CLAUDE.md. **Never append to `CHANGELOG.md`** — it is frozen at v0.5.1264. Fragments are folded into the GitHub Release notes at tag time (`scripts/cut_release_notes.sh`) and deleted. -3. **Commit changes**: Include code, `Cargo.toml`/`Cargo.lock`, `CLAUDE.md` (version bump only), and the `changelog.d/` fragment together. - -**Do not write changelog entries into CLAUDE.md.** This file is for orientation (architecture, common pitfalls, build commands). Per-change history lives in `changelog.d/` → GitHub Releases so CLAUDE.md stays small and stable across context loads. - -### External contributor PRs - -PRs from outside contributors should **not** touch `[workspace.package] version` in `Cargo.toml` or the `**Current Version:**` line in `CLAUDE.md`. The maintainer bumps the version at merge time — usually by rebasing the PR branch and amending. This avoids the patch-version collisions that happen when Perry's `main` ships several commits while a PR is in review (each on-main commit bumps the version; a PR that bumped to the same patch on day 1 is already behind by merge day). Contributors **do** write their own `changelog.d/-.md` fragment — the filename is PR-keyed, so in-flight PRs never collide. - -## Build Commands - -### Which profile to use - -- **Local dev / testing (default choice)**: `cargo check -p perry` for fastest feedback, then `cargo build --profile perry-dev -p perry` (opt-level=1, codegen-units=16, incremental, no LTO — minutes instead of ~30). Use this for iterating on the compiler, running gap/parity tests, and reproducing bugs. Only fall back to `--release` if a bug is optimization-sensitive. -- **Shipping / official artifacts**: `--profile dist` (mirrors `release`: thin LTO, codegen-units=1, opt-level=3, strip). Slow by design — LLVM codegen runs single-threaded per crate at codegen-units=1, and the giant crates (perry-runtime ~340k lines, perry-codegen, perry-hir) serialize the build regardless of core count. Don't use it for iteration. -- **Local release-ish build when you need release perf**: override the compile-time killer, keep the optimization: `CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16 cargo build --release` (2–4× faster, ~1–3% runtime cost). - -```bash -cargo build --release # Build all crates -cargo build --profile perry-dev -p perry # Fast local dev build (#5422; perry-dev profile) -cargo build --release -p perry-runtime -p perry-stdlib # Rebuild runtime (MUST rebuild stdlib too!) -cargo build --release -p perry-runtime-static -p perry-stdlib-static # Emit libperry_{runtime,stdlib}.a (#5422: runtime/stdlib are now rlib-only; the .a comes from these wrapper crates) -cargo test --release --workspace \ - --exclude perry-ui-ios --exclude perry-ui-tvos --exclude perry-ui-watchos \ - --exclude perry-ui-visionos --exclude perry-ui-android --exclude perry-ui-windows \ - --exclude perry-ui-gtk4 # Run tests (exclude cross-host UI crates on macOS) -cargo run --release -- file.ts -o output && ./output # Compile and run TypeScript -cargo run --release -- file.ts --print-hir # Debug: print HIR -cargo run --release -- file.ts --trace hir --focus fnName # Debug: focused HIR for one fn (use to localize a miscompile) -cargo run --release -- file.ts --trace llvm # Debug: dump per-module LLVM IR to .perry-trace/llvm/ -``` - -When debugging a "compiled to the wrong thing" bug, reach for `--trace hir --focus ` to dump just the offending function's lowered HIR (functions/methods/classes matching the substring; import/init noise suppressed) instead of scrolling a full `--print-hir`. `--trace llvm` writes per-module `.ll` (it forces a no-cache rebuild so codegen actually runs). See `docs/src/cli/flags.md`. - -## Architecture - -``` -TypeScript (.ts) → Parse (SWC) → AST → Lower → HIR → Transform → Codegen (LLVM) → .o → Link (cc) → Executable -``` - -| Crate | Purpose | -|-------|---------| -| **perry** | CLI driver (parallel module codegen via rayon) | -| **perry-parser** | SWC wrapper for TypeScript parsing | -| **perry-hir** | HIR types and data structures, plus AST→HIR lowering | -| **perry-transform** | IR passes (closure conversion, async lowering, inlining) | -| **perry-codegen** | LLVM-based native code generation | -| **perry-runtime** | Runtime: value.rs, object.rs, array.rs, string.rs, gc.rs, arena.rs, thread.rs | -| **perry-stdlib** | Node.js API support (mysql2, redis, fetch, fastify, ws, etc.) | -| **perry-ui** / **perry-ui-macos** / **perry-ui-ios** / **perry-ui-tvos** | Native UI (AppKit/UIKit) | - -## NaN-Boxing - -Perry uses NaN-boxing to represent JavaScript values in 64 bits (`perry-runtime/src/value.rs`): - -``` -TAG_UNDEFINED = 0x7FFC_0000_0000_0001 BIGINT_TAG = 0x7FFA (lower 48 = ptr) -TAG_NULL = 0x7FFC_0000_0000_0002 POINTER_TAG = 0x7FFD (lower 48 = ptr) -TAG_FALSE = 0x7FFC_0000_0000_0003 INT32_TAG = 0x7FFE (lower 32 = int) -TAG_TRUE = 0x7FFC_0000_0000_0004 STRING_TAG = 0x7FFF (lower 48 = ptr) -``` - -Key functions: `js_nanbox_string/pointer/bigint`, `js_nanbox_get_pointer`, `js_get_string_pointer_unified`, `js_jsvalue_to_string`, `js_is_truthy` - -**Module-level variables**: uniform NaN-boxed doubles in `@perry_global___` LLVM globals, all registered as GC roots before module init (marked AND rewritten on evacuation). The old F64-strings/raw-I64-arrays split and `module_var_data_ids` no longer exist (a stale comment survives in `perry-transform/src/inline/mod.rs`). - -## Garbage Collection - -Generational mark-sweep GC in `crates/perry-runtime/src/gc.rs` (default since v0.5.237 / Phase D). Two regions in the per-thread arena: nursery (`ARENA`, fills with new allocations, swept on minor GC) and old-gen (`OLD_ARENA`, holds tenured/evacuated objects). Precise shadow-stack roots + ~55 registered side-table scanners (`gc/mod.rs:298+`); a conservative stack scan exists but production mode resolves to SkipDisabled, so liveness rests on codegen shadow-stack spilling plus `RuntimeHandleScope` in runtime helpers. Write barriers populate a remembered set so minor GC can avoid retracing the old-gen. Two-bit aging (`HAS_SURVIVED` / `TENURED`) promotes nursery survivors after 2 minor cycles; the C4b evacuation policy moves non-pinned tenured objects into old-gen with full reference rewriting only when generated write barriers are active and nursery/RSS pressure plus measured movable candidates justify the work. Idle nursery blocks observed empty for 2 GC cycles are `dealloc`'d back to the OS (C4b-δ, v0.5.235), and the next-trigger calc is hard-capped at the initial threshold (64 MB) so >90%-freed step-doubling can't blow up peak occupancy (C4b-δ-tune, v0.5.236). Triggers on arena block allocation (1 MB blocks since v0.5.196), malloc count threshold, or explicit `gc()` call. 8-byte GcHeader per allocation. - -**Escape hatches**: `PERRY_GEN_GC=0`/`off`/`false` reverts to full mark-sweep (bisection only). `PERRY_GEN_GC_EVACUATE=0`/`off`/`false` disables policy evacuation; `=1`/`on`/`true` is accepted as auto-policy allowed, not unconditional evacuation. `PERRY_GC_FORCE_EVACUATE=1` stress-copies every marked non-pinned nursery object only when generated write barriers are active and policy evacuation is allowed. `PERRY_GC_VERIFY_EVACUATION=1` panics if any mutable live slot still points at a forwarded nursery object after an evacuation/rewrite cycle. `PERRY_WRITE_BARRIERS=0`/`off`/`false` disables codegen-emitted write barriers at compile time and runtime exact helper barriers at runtime for benchmark/debug bisection; unset, `=1`/`on`/`true` keep barriers enabled. `PERRY_GC_DIAG=1` prints per-cycle diagnostics, including evacuation-policy decisions for considered cycles and `barriers_inactive` skips. - -### Rooting-bug instruments (#7154 family) — what each knob ACTUALLY gates - -A "GC value live but not rooted across a collection point" bug is invisible at collection time: there is nothing for the collector to find. It surfaces one or more cycles later, in a different function, as `TypeError: value is not a function`. These three knobs exist to collapse that latency. **All default-off; every boolean knob's OFF state is asserted in `gc/tests/fromspace_protect.rs`** (`…_DEPTH` is a magnitude, not a mode, so its floor and default are asserted instead). The instruments are **sabotage-tested**, not merely exercised: `quarantine_catches_a_planted_stale_from_space_deref` plants a #7184/#7192-shaped stale from-space pointer and asserts the instrument distinguishes it from the live object that would otherwise be recycled into those bytes — so a green protected run means the detector works, not that nothing was tried. - -| knob | gates EXACTLY | does NOT | -|---|---|---| -| `PERRY_GC_PROTECT_FROMSPACE=1` (or `poison`) | the from-space reset performed by the **copying minor** (`arena::copying_reset_from_spaces_and_flip`). Retired Eden + active-survivor blocks are detached into a bounded quarantine, poison-filled (`0xDEADBEEFBAADF0DE`, `obj_type = 0xDE`) and, at `=1`, `mprotect(PROT_NONE)`d. A stale deref then SIGSEGVs at the faulting instruction; the installed reporter names the address, the retiring minor, and the last-known object's `obj_type`/size, then restores `SIG_DFL` and re-faults so a core/debugger still sees the real site. `poison` skips `mprotect`. | change the non-moving minor's `arena_reset_empty_blocks`, the full mark-sweep's reclaim, old-gen defrag, or the malloc sweep. **A run with zero copying minors protects nothing** — check that `PERRY_GC_DIAG=1` prints a `[gc-fromspace-protect] retired_set=#N` line. | -| `PERRY_GC_PROTECT_FROMSPACE_DEPTH=N` (default 4) | how many retired page-sets stay quarantined. Evicted sets are restored to RW and **recycled back into Eden**, never `dealloc`'d, so footprint is bounded at `N × from-space bytes`. `0` is clamped to 1 — a depth of 0 would read as ON and protect nothing. **Raise this when a suspected bug does not fault**: a value can cross hundreds of collections between its last valid observation and its stale use (one per back-edge poll under zeal). #7154's `new C(…)` reproducer needs `800` — its constructor crosses 600 polls, so the default 4 misses it silently. | — | -| `PERRY_GC_ZEAL=1` | forces an evacuating minor at every **GC safepoint**: `js_gc_loop_safepoint` (loop back-edge) and the outermost microtask-pump safepoint. It bypasses exactly two things — the `GC_SAFEPOINT_PENDING` requirement in `js_gc_loop_safepoint`, and the `gc_budgeted_due_trigger()` "is anything due?" test in `gc_safepoint_moving_minor`. Also makes `gc_force_evacuate_enabled()` true, so survivors actually MOVE. | bypass `gc_safepoint_moving_minor`'s **entry guards**: a safepoint reached mid-allocation (`GC_FLAG_IN_ALLOC`), suppressed (`GC_FLAG_SUPPRESSED`), inside an unsafe FFI zone, under a non-zero `GC_ROOT_LOCK_DEPTH`, or during a budgeted cycle still returns without collecting. Nor does it override an explicit `PERRY_GEN_GC_EVACUATE=0` — that wins, and with it set zeal moves nothing and surfaces nothing. Nor does it emit loop polls — those need the **compile-time** `PERRY_GC_MOVING_LOOP_POLLS=1` (default off since #7161). Zeal on a binary compiled without polls only fires at event-loop boundaries; a compute-only loop never collects. Check `crate::gc::zeal_forced_collections()` is nonzero. There is deliberately **no level 2**: the alloc-point arm forces a conservative stack scan, which makes the copying minor ineligible, so an "every allocation" zeal would run non-moving minors and move nothing. | -| `PERRY_GC_FROMSPACE_SCAN_ABORT=1` | now **implies** `PERRY_GC_FROMSPACE_SCAN=1`. It used to be inert alone (the scan never ran, so nothing aborted, and the run reported success). | — | - -`PERRY_GC_ZEAL=1 PERRY_GC_PROTECT_FROMSPACE=1` together is the pairing that turns a #7154 bug into an immediate precise fault. Compile *and* run with `PERRY_GC_MOVING_LOOP_POLLS=1` for in-loop coverage. - -### GC knob kill-policy (binding) - -**Every GC env knob either has a required CI arm exercising its OFF state, or it is deleted after one release of soak.** At most one diagnostic-only knob may exist at a time, and it must be labelled untested. - -This is not tidiness. An unexercised mode is a configuration nobody has verified, and this project has repeatedly paid for that: - -- `PERRY_GC_FORCE_EVACUATE` was **inert** for every `gc()`-driven test — it is read only on the minor path, while `gc()` runs a full mark-sweep with a forced conservative scan (#6942/#6946). Months of "passes under evacuation" meant nothing. -- The matrix's `--pressure` knob **disabled the very path it was measuring** — the defer hard cap and the arena-trigger ceiling shared a formula and collapsed together, so the `default` arm ran zero copying minors on all 22 rows (#7024). -- `gc_incremental_enabled`'s doc said "EXPERIMENTAL — default OFF" eight lines above a body comment saying "DEFAULT ON" (#6987). A merge decision was made on the wrong one. - -**A mode that still exists is a decision that hasn't been made.** When a knob's off-state stops being exercised, delete the off-state and the branch behind it — the losing mode should stop compiling, not linger as an untested configuration that a future bisect will trust. - -## Threading (`perry/thread`) - -Single-threaded by default. `perry/thread` provides: -- **`parallelMap(array, fn)`** / **`parallelFilter(array, fn)`** — data-parallel across all cores -- **`spawn(fn)`** — background OS thread, returns Promise - -Values cross threads via `SerializedValue` deep-copy. Each thread has independent arena + GC. Results from `spawn` flow back via `PENDING_THREAD_RESULTS` queue, drained during `js_promise_run_microtasks()`. - -## Native UI (`perry/ui`) - -Declarative TypeScript compiles to AppKit/UIKit calls. Handle-based widget system (1-based i64 handles, NaN-boxed with POINTER_TAG). `--target ios-simulator`/`--target ios`/`--target tvos-simulator`/`--target tvos` for cross-compilation. - -**To add a new widget** — change 4 places: -1. Runtime: `crates/perry-ui-macos/src/widgets/` — create widget, `register_widget(view)` -2. FFI: `crates/perry-ui-macos/src/lib.rs` — `#[no_mangle] pub extern "C" fn perry_ui__create` -3. Codegen: `crates/perry-codegen/src/codegen.rs` — declare extern + NativeMethodCall dispatch -4. HIR: `crates/perry-hir/src/lower.rs` — only if widget has instance methods - -## Compiling npm Packages Natively (`perry.compilePackages`) - -Configured in `package.json`: -```json -{ "perry": { "compilePackages": ["@noble/curves", "@noble/hashes"] } } -``` -First-resolved directory cached in `compile_package_dirs`; subsequent imports redirect to the same copy (dedup). - -## Known Limitations - -- **No runtime type *validation***: declared TS types aren't enforced at runtime (a `string` param accepts a number, no throw). Annotations are mostly erased — the exception is `emitDecoratorMetadata`, which retains `design:type`/`design:paramtypes` from annotations on decorated members (see `docs/src/language/decorators.md`). Runtime type *discrimination* does exist: `typeof` via NaN-boxing tags, `instanceof` via class ID chain. -- **`SharedArrayBuffer` + `Atomics` cross-thread** (#4794 single-realm; #4913 Stage 2 cross-agent): the `Atomics` ops (`add`/`and`/`or`/`sub`/`xor`/`load`/`store`/`exchange`/`compareExchange`/`isLockFree`) match the spec on one thread. A `SharedArrayBuffer` captured into a `spawn`/`parallelMap` closure now **aliases the same physical bytes** across `perry/thread` agents (its backing is a process-global, never-freed allocation — `crate::shared_sab` — passed by reference, not deep-copied), and `Atomics.wait`/`notify`/`waitAsync` are **real**: `wait` parks the OS thread on a futex table keyed by the absolute slot address (`crate::atomics_futex`), `notify` wakes parked agents and returns the count, and `waitAsync` resolves its promise on a background thread when notified or on timeout. Caveat: only the `SharedArrayBuffer` itself shares — a typed-array *view* captured directly still deep-copies (build the view per-agent from the shared SAB). The agent-coordinated test262 cases (`$262.agent`) remain out of scope. - -## Common Pitfalls & Patterns - -### NaN-Boxing Mistakes -- **Double NaN-boxing**: If value is already F64, don't NaN-box again. Check `builder.func.dfg.value_type(val)`. -- **Wrong tag**: Strings=STRING_TAG, objects=POINTER_TAG, BigInt=BIGINT_TAG. -- **`as f64` vs `from_bits`**: `u64 as f64` is numeric conversion (WRONG). Use `f64::from_bits(u64)` to preserve bits. - -### LLVM Type Mismatches -- Loop counter optimization produces i32 — always convert before passing to f64/i64 functions -- Constructor parameters always f64 (NaN-boxed) at signature level - -### Async / Threading -- Thread-local arenas: JSValues from tokio workers invalid on main thread -- Use `spawn_for_promise_deferred()` — return raw Rust data, convert to JSValue on main thread -- Async closures: Promise pointer (I64) must be NaN-boxed with POINTER_TAG before returning as F64 - -### Cross-Module Issues -- ExternFuncRef values are NaN-boxed — use `js_nanbox_get_pointer` to extract -- Module init order: topological sort by import dependencies -- Optional params need `imported_func_param_counts` propagation through re-exports - -### Closure Captures -- `collect_local_refs_expr()` must handle all expression types — catch-all silently skips refs -- Captured string/pointer values must be NaN-boxed before storing, not raw bitcast -- Loop counter i32 values: `fcvt_from_sint` to f64 before capture storage - -### Handle-Based Dispatch -- TWO systems: `HANDLE_METHOD_DISPATCH` (methods) and `HANDLE_PROPERTY_DISPATCH` (properties) -- Both must be registered. Small pointer detection: value < 0x100000 = handle. - -### objc2 v0.6 API -- `define_class!` with `#[unsafe(super(NSObject))]`, `msg_send!` returns `Retained` directly -- All AppKit constructors require `MainThreadMarker` - -### Verifying a runtime change (read before you trust an A/B) -Build outputs are invisible to `git status`, so a clean tree tells you nothing about what you are actually linking. Three ways this bites: - -- **Wrong build command → stale archive.** `perry-runtime`/`perry-stdlib` are `crate-type = ["rlib"]`; `libperry_{runtime,stdlib}.a` come from the `perry-runtime-static`/`perry-stdlib-static` wrappers (see Build Commands). `cargo build -p perry-runtime -p perry-stdlib` does **not** emit them, so `perry compile` links a stale `.a`: your fix looks like a no-op **and both arms of an A/B behave identically** (a vacuous "zero regressions"). Build `-p perry -p perry-runtime-static -p perry-stdlib-static`, pin `PERRY_RUNTIME_DIR`, and confirm the `.a` mtime moved *after* your edit. Keep the package set identical across builds/bisect hops — dropping `-p perry` changes cargo feature unification. (`run_parity_tests.sh` builds the wrappers itself, so gap runs are safe; hand-rolled `.ts` probes are not.) -- **A prebuilt binary in another worktree is not evidence about the commit it's checked out to** — it may be built from that worktree's WIP tree. Never use one as a bisect `good` endpoint or a perf baseline; build your own reference and *verify the good endpoint is actually good* first. -- **Check the harness's exit code, not a wrapper shell's.** A job piping the gap harness through `grep` can report exit 0 while the harness itself failed. - -### CI gates that surprise people -- **2000-line-per-file cap** (`scripts/check_file_size.sh`) — run it before pushing; adding a long doc comment can trip it. -- **addr-class ratchet** (`scripts/addr_class_inventory.py`) — a file gaining a bare-address site fails `lint`. -- **`conformance-smoke` shards are flaky.** Before believing a red shard, re-run it and A/B the named tests against a pristine `main` build; several are already in `test-parity/known_failures.json`. -- **Integration suites under `crates/*/tests/*.rs` do not run per-PR** (nightly/tag only) — a regression there can land green and sit red for days. Prefer putting acceptance coverage in `cargo-test`-visible unit tests (#5960). - -### ★ Four ways a gate can be unable to fail - -All four look fine on the Actions page. None can turn a merge red. When adding or reviewing a gate, check all four — each has bitten this repo, three of them within one week: - -1. **`continue-on-error: true`** — `gc-stress` carried it for months while being the only job covering GC correctness. -2. **Not in branch protection's required contexts** — `gc-stress` again. This is why #6925's `PERRY_PTR_SHAPE_LOCALS=0` regression landed visibly red and survived three merges. A job that reports failure without blocking is documentation, not a gate. -3. **`concurrency` with unconditional `cancel-in-progress`** — on a branch with a slow runner queue, every new merge cancels the previous run before it reaches a runner. `gc-ratchet` had three consecutive `main` runs cancelled, zero executed. Scope cancellation to `pull_request` and let `main` runs queue. -4. **The gate runs but its subject never did** — the most dangerous, because the job is genuinely green. `PERRY_GC_FORCE_EVACUATE` was inert for every `gc()`-driven test (#6942/#6946); the matrix's `--pressure` knob disabled the very path it was measuring (#7024); its `moved=` counter summed two different collectors, so a cell could pass having run zero copying minors (#7025). **A gate must assert its subject was live**, not merely that nothing threw — e.g. `copied_objects > 0` before a green verdict. - -Corollary: a *new* gate has never been green, so promoting it to required immediately blocks every open PR. Run it once, then promote. Leaving that second step undone is how (2) happens. - -### Known-weak areas (symptom is often not the bug) -- **Async-to-generator transform, body locals.** It boxes every body local into a shared mutable cell typed `Any`. Two consequences seen in the wild: per-iteration `let`/`const` bindings collapse for closures created in a loop, and computed numeric-key calls (`arr[i](x)`) lose their type proof and silently resolve by *method name*, evaporating the call. -- **Native base-class subclassing.** A native base's surface is installed at `super()` time and its parent edge lives in the class registry; keying any of that on a literal `extends` name loses it for fieldless classes, indirect subclasses, and class expressions. -- **Two prototype-resolution paths.** `CLASS_PROTOTYPE_OBJECTS` (synthetic: `Object.create`, plain-function ctors) vs `CLASS_DECL_PROTOTYPE_OBJECTS` (declared classes). `in`/`for…in` and `getPrototypeOf` have disagreed about the same chain. -- **Root-store dominance in codegen.** *A GC-managed value's root store must **dominate** every subsequent site that can collect.* Three ways it has broken, all shipped: the store's slot index fell outside the pushed shadow frame so `js_shadow_slot_bind` bounds-checked it into a silent no-op (#7184); the store was emitted in-frame but **after** a call that allocates (#7192); and the value lives in a plain `alloca_entry` that is neither a shadow slot nor a temp root, so the collector never rewrites it (`lower_call/new.rs`'s inline-ctor `this_slot`, closed by #7207; `--unrooted-allocas` is the detector for that shape, and its remaining hits are #7210's). All three present identically — a *rooted* slot holding a dangling pointer, surfacing cycles later as `TypeError: value is not a function` — and **none is visible to any runtime GC probe**, because at the moment of the collection there is nothing for the collector to find. That is why #7154's from-space scan only ever saw offenders whose targets had already died. The instrument is static: `scripts/gc_root_dominance_check.py` over `--trace llvm` output (`--self-test` proves it can still fail). Only bites under `PERRY_GC_MOVING_LOOP_POLLS=1`, off by default since #7161 — so a green default run says nothing about this class. **Full writeup, every known shape and how to check your work: `docs/src/internals/gc-rooting-invariant.md`.** The CI gate is `gc-root-dominance.yml` over `scripts/gc_root_dominance_corpus.sh`; known-remaining hits are named one-per-entry in `scripts/gc_root_dominance_allowlist.json` (an entry that matches nothing FAILS, so a fix must delete its entry), and that list is currently **empty** — every new hit is a red build. -- **A runtime-side cache of a raw heap pointer is a GC root, and the static checker cannot see it.** `scripts/gc_root_dominance_check.py` reads emitted LLVM IR, so a thread-local or side table holding a `*mut` into the heap is structurally invisible to it — the runtime instruments above are the only detector, and they go at the workload *before* you grind the static checker's tail. Two tells. An unrooted *register* goes bad only when a collection lands in its window, so it is intermittent; an unrooted *cache* goes bad at collection #0 and stays bad, so **a perfectly reproducible GC bug means a table, not a register**. And the registry is `gc_register_mutable_root_scanner` in `gc/mod.rs` (~55 entries): when you add a cache of a heap pointer, add it there in the same commit. Worked examples: `changelog.d/7219-registry-gc-unrooted-caches.md`, `changelog.d/7239-gc-unrooted-runtime-caches.md`. diff --git a/Cargo.lock b/Cargo.lock index dfde7b446a..657d527fcb 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5547,7 +5547,7 @@ checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" [[package]] name = "perry" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", "base64", @@ -5607,14 +5607,14 @@ dependencies = [ [[package]] name = "perry-api-manifest" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "serde", ] [[package]] name = "perry-audio-miniaudio" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "cc", "libc", @@ -5622,7 +5622,7 @@ dependencies = [ [[package]] name = "perry-codegen" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", "inkwell", @@ -5638,7 +5638,7 @@ dependencies = [ [[package]] name = "perry-codegen-arkts" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", "perry-hir", @@ -5646,7 +5646,7 @@ dependencies = [ [[package]] name = "perry-codegen-glance" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", "perry-hir", @@ -5654,7 +5654,7 @@ dependencies = [ [[package]] name = "perry-codegen-js" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", "perry-dispatch", @@ -5663,7 +5663,7 @@ dependencies = [ [[package]] name = "perry-codegen-swiftui" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", "perry-hir", @@ -5671,7 +5671,7 @@ dependencies = [ [[package]] name = "perry-codegen-wasm" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", "base64", @@ -5683,7 +5683,7 @@ dependencies = [ [[package]] name = "perry-codegen-wear-tiles" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", "perry-hir", @@ -5691,7 +5691,7 @@ dependencies = [ [[package]] name = "perry-container-compose" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", "async-trait", @@ -5720,14 +5720,14 @@ dependencies = [ [[package]] name = "perry-container-e2e" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", ] [[package]] name = "perry-diagnostics" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "serde", "serde_json", @@ -5735,7 +5735,7 @@ dependencies = [ [[package]] name = "perry-dispatch" -version = "0.5.1287" +version = "0.5.1288" [[package]] name = "perry-doc-fixture-my-bindings" @@ -5746,7 +5746,7 @@ dependencies = [ [[package]] name = "perry-doc-tests" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", "clap", @@ -5761,7 +5761,7 @@ dependencies = [ [[package]] name = "perry-ext-ads" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "block2", "objc2", @@ -5771,7 +5771,7 @@ dependencies = [ [[package]] name = "perry-ext-argon2" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "argon2", "perry-ffi", @@ -5779,7 +5779,7 @@ dependencies = [ [[package]] name = "perry-ext-axios" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ffi", "reqwest", @@ -5788,7 +5788,7 @@ dependencies = [ [[package]] name = "perry-ext-bcrypt" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "bcrypt", "perry-ffi", @@ -5796,7 +5796,7 @@ dependencies = [ [[package]] name = "perry-ext-better-sqlite3" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ffi", "rusqlite", @@ -5804,7 +5804,7 @@ dependencies = [ [[package]] name = "perry-ext-cheerio" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ffi", "scraper", @@ -5812,7 +5812,7 @@ dependencies = [ [[package]] name = "perry-ext-commander" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ffi", "perry-runtime", @@ -5820,7 +5820,7 @@ dependencies = [ [[package]] name = "perry-ext-cron" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "chrono", "cron", @@ -5830,7 +5830,7 @@ dependencies = [ [[package]] name = "perry-ext-dayjs" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "chrono", "perry-ffi", @@ -5838,7 +5838,7 @@ dependencies = [ [[package]] name = "perry-ext-decimal" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ffi", "rust_decimal", @@ -5846,7 +5846,7 @@ dependencies = [ [[package]] name = "perry-ext-dotenv" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ffi", "serde_json", @@ -5854,7 +5854,7 @@ dependencies = [ [[package]] name = "perry-ext-ethers" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ffi", "rand 0.10.1", @@ -5862,7 +5862,7 @@ dependencies = [ [[package]] name = "perry-ext-events" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ffi", "perry-runtime", @@ -5870,14 +5870,14 @@ dependencies = [ [[package]] name = "perry-ext-exponential-backoff" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ffi", ] [[package]] name = "perry-ext-fastify" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "bytes", "http-body-util", @@ -5895,7 +5895,7 @@ dependencies = [ [[package]] name = "perry-ext-fetch" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "bytes", "lazy_static", @@ -5908,7 +5908,7 @@ dependencies = [ [[package]] name = "perry-ext-http" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "bytes", "h2", @@ -5932,7 +5932,7 @@ dependencies = [ [[package]] name = "perry-ext-ioredis" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "lazy_static", "perry-ffi", @@ -5942,7 +5942,7 @@ dependencies = [ [[package]] name = "perry-ext-jsonwebtoken" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "base64", "jsonwebtoken", @@ -5953,7 +5953,7 @@ dependencies = [ [[package]] name = "perry-ext-lru-cache" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "lru", "perry-ffi", @@ -5962,7 +5962,7 @@ dependencies = [ [[package]] name = "perry-ext-moment" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "chrono", "perry-ffi", @@ -5970,7 +5970,7 @@ dependencies = [ [[package]] name = "perry-ext-mongodb" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "bson", "futures-util", @@ -5982,7 +5982,7 @@ dependencies = [ [[package]] name = "perry-ext-mysql2" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "chrono", "perry-ffi", @@ -5992,7 +5992,7 @@ dependencies = [ [[package]] name = "perry-ext-nanoid" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "nanoid", "perry-ffi", @@ -6001,7 +6001,7 @@ dependencies = [ [[package]] name = "perry-ext-net" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "bytes", "perry-ffi", @@ -6014,7 +6014,7 @@ dependencies = [ [[package]] name = "perry-ext-node-forge" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "const-oid 0.9.6", "der 0.7.10", @@ -6033,7 +6033,7 @@ dependencies = [ [[package]] name = "perry-ext-nodemailer" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "lettre", "perry-ffi", @@ -6043,7 +6043,7 @@ dependencies = [ [[package]] name = "perry-ext-pdf" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ffi", "printpdf", @@ -6051,7 +6051,7 @@ dependencies = [ [[package]] name = "perry-ext-pg" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ffi", "sqlx", @@ -6060,7 +6060,7 @@ dependencies = [ [[package]] name = "perry-ext-ratelimit" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "governor", "perry-ffi", @@ -6068,7 +6068,7 @@ dependencies = [ [[package]] name = "perry-ext-sharp" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "fast_image_resize", "image", @@ -6078,14 +6078,14 @@ dependencies = [ [[package]] name = "perry-ext-slugify" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ffi", ] [[package]] name = "perry-ext-streams" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "lazy_static", "perry-ffi", @@ -6094,7 +6094,7 @@ dependencies = [ [[package]] name = "perry-ext-undici" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ffi", "perry-runtime", @@ -6103,7 +6103,7 @@ dependencies = [ [[package]] name = "perry-ext-uuid" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ffi", "uuid", @@ -6111,7 +6111,7 @@ dependencies = [ [[package]] name = "perry-ext-validator" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ffi", "regex", @@ -6121,7 +6121,7 @@ dependencies = [ [[package]] name = "perry-ext-ws" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "futures-util", "lazy_static", @@ -6134,7 +6134,7 @@ dependencies = [ [[package]] name = "perry-ext-zlib" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "brotli", "flate2", @@ -6144,7 +6144,7 @@ dependencies = [ [[package]] name = "perry-ffi" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "dashmap", "once_cell", @@ -6153,7 +6153,7 @@ dependencies = [ [[package]] name = "perry-hir" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", "perry-api-manifest", @@ -6171,7 +6171,7 @@ dependencies = [ [[package]] name = "perry-parser" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", "perry-diagnostics", @@ -6183,7 +6183,7 @@ dependencies = [ [[package]] name = "perry-runtime" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", "base64", @@ -6225,14 +6225,14 @@ dependencies = [ [[package]] name = "perry-runtime-static" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-runtime", ] [[package]] name = "perry-stdlib" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "aes 0.8.4", "aes 0.9.1", @@ -6327,14 +6327,14 @@ dependencies = [ [[package]] name = "perry-stdlib-static" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-stdlib", ] [[package]] name = "perry-transform" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", "perry-hir", @@ -6343,14 +6343,14 @@ dependencies = [ [[package]] name = "perry-ui" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ui-model", ] [[package]] name = "perry-ui-android" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "base64", "itoa", @@ -6367,7 +6367,7 @@ dependencies = [ [[package]] name = "perry-ui-geisterhand" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "rand 0.10.1", "serde", @@ -6377,7 +6377,7 @@ dependencies = [ [[package]] name = "perry-ui-gtk4" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "base64", "cairo-rs 0.22.0", @@ -6400,7 +6400,7 @@ dependencies = [ [[package]] name = "perry-ui-ios" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "base64", "block2", @@ -6416,7 +6416,7 @@ dependencies = [ [[package]] name = "perry-ui-macos" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "base64", "block2", @@ -6431,7 +6431,7 @@ dependencies = [ [[package]] name = "perry-ui-model" -version = "0.5.1287" +version = "0.5.1288" [[package]] name = "perry-ui-test" @@ -6442,11 +6442,11 @@ dependencies = [ [[package]] name = "perry-ui-testkit" -version = "0.5.1287" +version = "0.5.1288" [[package]] name = "perry-ui-tvos" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "base64", "block2", @@ -6462,7 +6462,7 @@ dependencies = [ [[package]] name = "perry-ui-visionos" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "base64", "block2", @@ -6478,7 +6478,7 @@ dependencies = [ [[package]] name = "perry-ui-watchos" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "block2", "libc", @@ -6491,7 +6491,7 @@ dependencies = [ [[package]] name = "perry-ui-windows" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "base64", "libc", @@ -6508,14 +6508,14 @@ dependencies = [ [[package]] name = "perry-ui-windows-winui" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "perry-ui-windows", ] [[package]] name = "perry-updater" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "anyhow", "base64", @@ -6531,7 +6531,7 @@ dependencies = [ [[package]] name = "perry-wasm-host" -version = "0.5.1287" +version = "0.5.1288" dependencies = [ "wasmi", ] diff --git a/Cargo.toml b/Cargo.toml index 7b69ac4aaa..8111ab8e42 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -315,7 +315,7 @@ codegen-units = 16 codegen-units = 16 [workspace.package] -version = "0.5.1287" +version = "0.5.1288" edition = "2021" license = "MIT" repository = "https://github.com/PerryTS/perry" diff --git a/changelog.d/7499-json-reparse-materialize.md b/changelog.d/7499-json-reparse-materialize.md new file mode 100644 index 0000000000..150f251be3 --- /dev/null +++ b/changelog.d/7499-json-reparse-materialize.md @@ -0,0 +1,69 @@ +### JSON: a lazy tape array now batch-materializes by re-parsing its blob (#7478) + +`force_materialize_lazy` built the `ArrayHeader` tree by walking the tape one +element at a time. #7478's decomposition measured that walk at ~2.3× the direct +parser's batch tree build — 56 ms/iter against 24 ms/iter on the 10k-record +fixture — and named the fix: a lazy array only ever stands for a *top-level* +array (`try_parse_via_tape` builds one only for `tape_entries[0].kind == +KIND_ARR_START`, always with `root_idx = 0`), so its retained `blob_str` is +exactly that array's source text and one `DirectParser::parse_value()` over it +rebuilds the identical tree in a single linear pass. Identical now includes the +numbers: #7483 put the DirectParser's decimal fast path on one correctly-rounded +division, so it agrees bit-for-bit with `str::parse::` (what +`materialize_number` uses) and with node — the #7477 divergence is what blocked +this the first time it was tried. + +The first attempt also SIGSEGV'd intermittently, and the reason is structural: +`DirectParser` is sound only inside a no-move window, three times over. It holds +`input: &'a [u8]` derived from the blob's `StringHeader` payload for the whole +parse with no way to re-derive it; it carries an **unrooted raw-pointer shape +cache** (`hot_shape_keys`, `hot_shape_array`) across every allocation — the +runtime-side-cache-of-a-heap-pointer shape the static root checker cannot see; +and `array_push_parse_fast` fills fresh arrays through +`note_array_slot_layout_only`, which deliberately skips the generational barrier +for young arrays *on the strength of that same suppression*. `js_json_parse` buys +all three with `gc_suppress()`. The reparse now opens a nesting-safe +`GcSuppressScope` — nesting-safe because `force_materialize_lazy` is reachable +from inside `try_stringify_lazy_array`, where the flat `gc_unsuppress()` would +end an outer window early. Around it, every header re-read is a +`RuntimeHandle::across_{mut,const}` paired with the call that can collect (no new +bare `get_raw_*_ptr`; `json_tape.rs` stays at its ceiling of 22), the parsed tree +is handed to `PARSE_ROOTS` before the window closes and promoted to a handle-scope +root before those are restored, and the refreshed header is returned on every +exit including the declining ones. + +Because the reparse rebuilds every element from source, the sparse per-element +cache is patched back over the fresh slots: a cached slot holds the JSValue user +code already has a reference to and may have **mutated** through it, while the +blob still says the old value. The patch loop preserves the value and the +identity (`parsed[i] === parsed[i]`), runs inside its own suppression window, and +stores through `store_array_slot` — a raw slot write would leave a reparsed +`RawF64`-layout array flagged pointer-free with a live pointer inside it. + +The reparse fires only below the measured crossover (`cached_count * 2 < +cached_length`). Past it the element-wise merge is the cheaper producer — it +copies the cached JSValues and materializes only the remainder — so a reparse +would rebuild subtrees it is about to discard. `bench_field_access` is +deliberately unchanged by this: it touches every element before stringifying, so +its bitmap is full by materialization time and its 2981 ms was always in the 10k +`lazy_get` calls, not the materializer. What benefits today is any full-array +operation on a lazy array whose elements have not been individually walked — +`.map`, `.filter`, spread, `for…of`, `sort`, `Object.keys` on a fresh +`JSON.parse` result — and the random-access `cumulative_walk_steps` trigger, +which fires with a cache count of ~4 on a 10k array. + +Validation: `cargo test -p perry-runtime --lib` 1718 passed / 0 failed. A new +sabotage test plants a mutated cached element, materializes through the reparse, +and asserts both the mutation and the object identity survive; the existing +`ForceLazyArrayRooted` copied-minor sabotage test keeps its safepoint and gains +an assertion that it now runs against the reparse producer; two more tests pin +the crossover *decision* and the non-blob-root decline, all witnessed by a +thread-local reparse counter so a test asserts which producer ran rather than +only that nothing threw. Deleting the patch loop turns three tests red. An +11-scenario probe (600 records × 13 fields, the 10k `i * 3.14159` float array +from the #7477 class, edge values, whitespace blobs, identity, sort, repeated +materialization) is byte-identical to node 26.5.1 under `PERRY_JSON_TAPE=1`, +`=0` and auto, and stays byte-identical under `PERRY_GC_ZEAL=1 +PERRY_GC_PROTECT_FROMSPACE=1` on a `PERRY_GC_MOVING_LOOP_POLLS=1` build with the +from-space quarantine confirmed live (`retired_set=#0…#5`, a copying minor moving +144,174 objects, 120 reparses inside the protected run). diff --git a/crates/perry-runtime/src/gc/tests/runtime_roots/callback_scanners.rs b/crates/perry-runtime/src/gc/tests/runtime_roots/callback_scanners.rs index 462d5cab18..afbb286e7c 100644 --- a/crates/perry-runtime/src/gc/tests/runtime_roots/callback_scanners.rs +++ b/crates/perry-runtime/src/gc/tests/runtime_roots/callback_scanners.rs @@ -319,7 +319,16 @@ fn test_json_tape_force_materialize_sparse_cache_handles_survive_copied_minor_gc let hook = JsonTapeSafepointHookGuard::new(crate::json_tape::JsonTapeSafepoint::ForceLazyArrayRooted); + let before_reparses = crate::json_tape::reparse_materializations(); let arr = unsafe { crate::json_tape::force_materialize_lazy(hdr_handle.get_raw_mut_ptr()) }; + // #7478: 1-of-4 cached is below the crossover, so the array under this + // instrument is the RE-PARSED one. Pin that — if the crossover ever + // inverts, the sabotage below silently changes subject. + assert_eq!( + crate::json_tape::reparse_materializations(), + before_reparses + 1, + "force materialization here must be the #7478 reparse path" + ); let original_arr = hook.fired_ptr(); let hdr_after = hdr_handle.get_raw_mut_ptr::(); assert_ne!( @@ -347,6 +356,64 @@ fn test_json_tape_force_materialize_sparse_cache_handles_survive_copied_minor_gc } } +/// #7478 cache-merge semantics under the reparse producer. +/// +/// The reparse rebuilds EVERY element from the blob's source text, so the +/// only thing keeping a handed-out element alive is the patch loop that +/// stores the sparse cache back over the fresh slots. That element may have +/// been MUTATED through the reference user code holds (`parsed[2].id = 99`), +/// and the blob still says `0` — a reparse that forgets the patch silently +/// reverts the mutation and hands back a different object for the same index. +/// Both are asserted here: the value (`99`) and the identity (same JSValue). +#[test] +fn test_json_tape_reparse_materialize_preserves_a_mutated_cache_entry() { + let _guard = CopyingNurseryTestGuard::new(0); + let _trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + register_runtime_handle_root_scanner_for_tests(); + + let input = br#"[{"id":0},{"id":1},{"id":2},{"id":3}]"#; + let hdr = unsafe { test_alloc_lazy_json_array(input) }; + let scope = RuntimeHandleScope::new(); + let hdr_handle = scope.root_raw_mut_ptr(hdr); + + // Hand element 2 out (caching it), then mutate it — `parsed[2].id = 99`. + // Every step allocates, so the header comes back out of `across_mut` + // rather than being carried across the call. + // The refreshed header from this first pairing is discarded on purpose: + // the mutation below allocates too, so the pairing after it is the one + // that supplies the pointer `force_materialize_lazy` is called with. + let (cached, _) = hdr_handle.across_mut::(|| unsafe { + crate::json_tape::lazy_get(hdr, 2) + }); + let cached_handle = scope.root_nanbox_u64(cached.bits()); + let (_, hdr) = hdr_handle.across_mut::(|| { + let key = crate::string::js_string_from_bytes(b"id".as_ptr(), 2); + // Re-derive the receiver AFTER the key allocation, not before it. + let obj = (cached_handle.get_nanbox_u64() & POINTER_MASK) as *mut crate::ObjectHeader; + crate::object::js_object_set_field_by_name(obj, key, 99.0); + }); + + let before_reparses = crate::json_tape::reparse_materializations(); + let arr = unsafe { crate::json_tape::force_materialize_lazy(hdr) }; + assert_eq!( + crate::json_tape::reparse_materializations(), + before_reparses + 1, + "1-of-4 cached must batch-materialize via the reparse" + ); + assert_eq!( + crate::array::js_array_get(arr, 2).bits(), + cached_handle.get_nanbox_u64(), + "the reparse must patch the handed-out element back, not a fresh copy" + ); + + let arr_handle = scope.root_nanbox_u64(ptr_bits(arr as usize)); + let output = + unsafe { crate::json::js_json_stringify(f64::from_bits(arr_handle.get_nanbox_u64()), 0) }; + unsafe { + assert_string_bytes(output, br#"[{"id":0},{"id":1},{"id":99},{"id":3}]"#); + } +} + #[test] fn test_promise_then_and_finally_handles_survive_setup_gc() { let _guard = CopyingNurseryTestGuard::new(0); diff --git a/crates/perry-runtime/src/json_tape.rs b/crates/perry-runtime/src/json_tape.rs index 89e39f23d2..50c2e3d8b2 100644 --- a/crates/perry-runtime/src/json_tape.rs +++ b/crates/perry-runtime/src/json_tape.rs @@ -1033,7 +1033,13 @@ mod tests { }) .expect("valid JSON should build a tape"); + let before = reparse_materializations(); let arr = unsafe { force_materialize_lazy(lazy) }; + assert_eq!( + reparse_materializations(), + before + 1, + "an uncached lazy array must batch-materialize via the #7478 reparse" + ); assert_eq!(crate::array::js_array_is_numeric_f64_layout(arr), 1); assert_eq!(crate::array::js_array_numeric_get_f64_unboxed(arr, 0), 1.0); @@ -1061,14 +1067,101 @@ mod tests { *(*lazy).materialized_bitmap |= 1u64 << 1; } + let before = reparse_materializations(); let arr = unsafe { force_materialize_lazy(lazy) }; + assert_eq!( + reparse_materializations(), + before + 1, + "1-of-3 cached is below the crossover, so this must reparse" + ); + // The reparse produces a RawF64-layout array for `[1,2,3]`; patching + // a STRING into slot 1 has to downgrade it, or the tracer would skip + // a live pointer in an array flagged pointer-free. assert_eq!(crate::array::js_array_is_numeric_f64_layout(arr), 0); assert_eq!( crate::gc::test_layout_pointer_slot_count(arr as usize, 3), Some(1) ); } + + /// #7478 crossover. Once MOST elements are already in the sparse cache + /// the element-wise merge is the cheap producer — it copies the cached + /// JSValues and materializes only the remainder — so a reparse would + /// rebuild subtrees it is about to throw away. This pins the decision, + /// not just the values: without the counter assertion the test passes + /// either way and the crossover could silently invert. + #[test] + fn force_materialize_majority_cached_uses_the_merge_walk_not_a_reparse() { + let input = br#"[10,20,30,40]"#; + let text = crate::string::js_string_from_bytes(input.as_ptr(), input.len() as u32); + let lazy = with_built_tape(input, |tape| unsafe { + alloc_lazy_array(tape, 0, count_array_length(tape, 0), text) + }) + .expect("valid JSON should build a tape"); + + unsafe { + *(*lazy).materialized_elements.add(0) = JSValue::number(1.5); + *(*lazy).materialized_elements.add(1) = JSValue::number(2.5); + *(*lazy).materialized_bitmap |= 0b11; + } + + let before = reparse_materializations(); + let arr = unsafe { force_materialize_lazy(lazy) }; + assert_eq!( + reparse_materializations(), + before, + "2-of-4 cached is at the crossover — the walk, not a reparse" + ); + + assert_eq!( + crate::array::js_array_get(arr, 0).bits(), + JSValue::number(1.5).bits() + ); + assert_eq!( + crate::array::js_array_get(arr, 1).bits(), + JSValue::number(2.5).bits() + ); + assert_eq!( + crate::array::js_array_get(arr, 2).bits(), + JSValue::number(30.0).bits() + ); + assert_eq!( + crate::array::js_array_get(arr, 3).bits(), + JSValue::number(40.0).bits() + ); + } + + /// A lazy header whose tape root is not the blob's first value cannot + /// have its blob re-parsed (the blob is not that array's source), so the + /// reparse must decline and the tape walk must still produce the array. + #[test] + fn force_materialize_declines_reparse_when_the_tape_root_is_not_the_blob_root() { + let input = br#"[[7,8],[9]]"#; + let text = crate::string::js_string_from_bytes(input.as_ptr(), input.len() as u32); + // root_idx 1 = the inner `[7,8]`, whose source is NOT the whole blob. + let lazy = with_built_tape(input, |tape| unsafe { + alloc_lazy_array(tape, 1, count_array_length(tape, 1), text) + }) + .expect("valid JSON should build a tape"); + + let before = reparse_materializations(); + let arr = unsafe { force_materialize_lazy(lazy) }; + assert_eq!( + reparse_materializations(), + before, + "a non-root tape index must decline the reparse" + ); + assert_eq!(unsafe { (*arr).length }, 2); + assert_eq!( + crate::array::js_array_get(arr, 0).bits(), + JSValue::number(7.0).bits() + ); + assert_eq!( + crate::array::js_array_get(arr, 1).bits(), + JSValue::number(8.0).bits() + ); + } } impl PartialEq for TapeEntry { @@ -1455,6 +1548,150 @@ pub unsafe fn lazy_get(hdr: *mut LazyArrayHeader, i: u32) -> JSValue { JSValue::from_bits(value_handle.get_nanbox_u64()) } +thread_local! { + /// #7478 witness: how many lazy arrays this thread batch-materialized + /// by RE-PARSING the retained blob rather than walking the tape. A + /// test that only asserts "the values came out right" cannot tell the + /// two producers apart — this is what lets it assert its subject ran. + static REPARSE_MATERIALIZATIONS: Cell = const { Cell::new(0) }; +} + +#[cfg_attr(not(test), allow(dead_code))] +pub(crate) fn reparse_materializations() -> u64 { + REPARSE_MATERIALIZATIONS.with(|c| c.get()) +} + +/// #7478: batch-materialize a lazy array by RE-PARSING its retained blob +/// with the `DirectParser`, instead of walking the tape element by +/// element. +/// +/// A lazy array only ever stands for a TOP-LEVEL array — `try_parse_via_tape` +/// builds one only when `tape_entries[0].kind == KIND_ARR_START`, and always +/// with `root_idx = 0` — so `blob_str` is exactly this array's source text +/// and a fresh `DirectParser::parse_value()` over it reproduces the identical +/// tree. Identical includes the numbers: #7483 put the DirectParser's decimal +/// fast path on one correctly-rounded division, so it now agrees bit-for-bit +/// with `str::parse::` (what `materialize_number` uses) and with node. +/// That divergence (#7477) is what blocked this change the first time. +/// +/// The batch parser builds the same tree ~2.3× faster than the per-element +/// materializer (#7478's decomposition: 24 ms/iter vs 56 ms/iter on the 10k- +/// record fixture), because it makes one linear pass instead of re-entering +/// the walk, the sparse cache and a fresh handle scope per element. +/// +/// GC contract — this is the part the first attempt got wrong, and it +/// SIGSEGV'd intermittently for it. `DirectParser` is only sound inside a +/// no-move window, in three separate ways: +/// * it holds `input: &'a [u8]` derived from the blob's `StringHeader` +/// payload for the whole parse and cannot re-derive it; +/// * it carries an UNROOTED one-entry shape cache (`hot_shape_keys`, +/// `hot_shape_array`) — raw heap pointers the collector cannot see, the +/// "runtime-side cache of a raw heap pointer" shape; +/// * `array_push_parse_fast` fills fresh arrays through +/// `note_array_slot_layout_only`, which deliberately skips the +/// generational barrier on the strength of that same suppression. +/// `js_json_parse` buys all three with `gc_suppress()`; so must this. The +/// window here is a nesting-safe `GcSuppressScope`, because +/// `force_materialize_lazy` is reachable from inside stringify and the flat +/// `gc_unsuppress()` would end an outer window early. +/// +/// Returns the refreshed header alongside the result: this function +/// allocates, so the caller's `hdr` is stale on EVERY exit, including the +/// declining ones. +unsafe fn reparse_materialize( + scope: &crate::gc::RuntimeHandleScope, + hdr_handle: &crate::gc::RuntimeHandle<'_>, + hdr: *mut LazyArrayHeader, + cached_length: u32, +) -> (Option<*mut crate::array::ArrayHeader>, *mut LazyArrayHeader) { + // The blob is this array's own source only when the tape root is the + // blob's first value. Every production lazy array is built that way; + // anything else declines rather than guessing. + if (*hdr).root_idx != 0 { + return (None, hdr); + } + let blob = (*hdr).blob_str; + if blob.is_null() { + return (None, hdr); + } + let blob_len = (*blob).byte_len as usize; + if blob_len == 0 { + return (None, hdr); + } + + // Nothing between this read of `blob` and the suppression window can + // collect, so the slice derived inside it names the live payload. + let saved_roots = crate::json::parse_root_save_len(); + let (parsed_bits, hdr) = hdr_handle.across_mut::(|| { + let _suppress = crate::gc::GcSuppressScope::new(); + let data = (blob as *const u8).add(std::mem::size_of::()); + let bytes = std::slice::from_raw_parts(data, blob_len); + let mut parser = crate::json::DirectParser::new(bytes); + let parsed = parser.parse_value(); + // Hand the tree to PARSE_ROOTS before the window closes — the + // handle-scope root below is pushed after it has already closed. + crate::json::parse_root_push(parsed); + parsed.bits() + }); + let arr_handle = scope.root_nanbox_u64(parsed_bits); + crate::json::parse_root_restore(saved_roots); + + // `clean_arr_ptr_mut` re-checks `obj_type == GC_TYPE_ARRAY`, so a + // non-array or failed parse declines here instead of publishing a + // bogus `materialized`. The length check is the same guard for a blob + // whose tape and text somehow disagree. + let arr_ptr = crate::array::clean_arr_ptr_mut(array_from_nanbox_handle(&arr_handle)); + if arr_ptr.is_null() || (*arr_ptr).length != cached_length { + return (None, hdr); + } + + let arr_addr = arr_ptr as usize; + let (_, hdr) = hdr_handle.across_mut::(|| { + json_tape_safepoint(JsonTapeSafepoint::ForceLazyArrayRooted, arr_addr) + }); + let arr_ptr = array_from_nanbox_handle(&arr_handle); + + // Patch the sparse cache back over the fresh slots. A cached slot holds + // the JSValue user code already has a reference to and may have MUTATED + // through it, so the cache — not the reparsed subtree — is authoritative + // there, and its identity has to survive (`parsed[i] === parsed[i]`). + // + // `store_array_slot` is the store that knows how to downgrade a + // RawF64-layout array when a pointer lands in it; a raw + // `*elements.add(i) = bits` would leave the array flagged pointer-free + // with a pointer inside it, which the tracer would never scan. + // + // The loop only stores, and runs inside a nesting-safe suppression + // window, so `hdr` / `bitmap` / `cache` / `arr_ptr` stay valid for its + // duration without a per-element re-read. + { + let _suppress = crate::gc::GcSuppressScope::new(); + let bitmap = (*hdr).materialized_bitmap; + let cache = (*hdr).materialized_elements; + if !bitmap.is_null() && !cache.is_null() { + for w in 0..(cached_length as usize).div_ceil(64) { + let mut word = *bitmap.add(w); + while word != 0 { + let i = w * 64 + word.trailing_zeros() as usize; + word &= word - 1; + if i >= cached_length as usize { + break; + } + crate::array::store_array_slot(arr_ptr, i, (*cache.add(i)).bits()); + } + } + } + (*hdr).materialized = arr_ptr; + note_lazy_raw_slot( + hdr, + &(*hdr).materialized as *const _ as usize, + arr_ptr as usize, + ); + } + REPARSE_MATERIALIZATIONS.with(|c| c.set(c.get().wrapping_add(1))); + (Some(arr_ptr), hdr) +} + /// Force-materialize a lazy array into an `ArrayHeader`-backed tree. /// Idempotent: subsequent calls return the cached `materialized` /// pointer. Callers of array accessors that don't have a lazy path @@ -1466,7 +1703,7 @@ pub unsafe fn force_materialize_lazy(hdr: *mut LazyArrayHeader) -> *mut crate::a let scope = crate::gc::RuntimeHandleScope::new(); let hdr_handle = scope.root_raw_mut_ptr(hdr); json_tape_safepoint(JsonTapeSafepoint::ForceLazyHeaderRooted, hdr as usize); - let hdr = hdr_handle.get_raw_mut_ptr::(); + let mut hdr = hdr_handle.get_raw_mut_ptr::(); if hdr.is_null() { return std::ptr::null_mut(); } @@ -1476,19 +1713,35 @@ pub unsafe fn force_materialize_lazy(hdr: *mut LazyArrayHeader) -> *mut crate::a let cached_length = (*hdr).cached_length; let bitmap = (*hdr).materialized_bitmap; let cache = (*hdr).materialized_elements; - let has_cache_hits = if !bitmap.is_null() && !cache.is_null() && cached_length > 0 { + let cached_count = if !bitmap.is_null() && !cache.is_null() && cached_length > 0 { let words = (cached_length as usize).div_ceil(64); - let mut any = false; + let mut count: u64 = 0; for w in 0..words { - if *bitmap.add(w) != 0 { - any = true; - break; - } + count += (*bitmap.add(w)).count_ones() as u64; } - any + count } else { - false + 0 }; + let has_cache_hits = cached_count > 0; + + // #7478: when most of the array still has to be built, re-parse the + // retained blob with the batch DirectParser instead of walking the + // tape element-by-element — same tree, ~2.3× the rate. When MOST + // elements are already cached the walk is the cheap producer (it + // copies cached JSValues and materializes only the remainder), and a + // reparse would rebuild subtrees it is about to throw away. The + // measured crossover is at ~43% uncached; `cached_count * 2 < + // cached_length` sits on the conservative side of it. + if cached_count * 2 < cached_length as u64 { + let (reparsed, refreshed) = reparse_materialize(&scope, &hdr_handle, hdr, cached_length); + if let Some(arr) = reparsed { + return arr; + } + // Declining still allocated, so take the refreshed header into the + // tape walk below. + hdr = refreshed; + } // Fast path: no cache hits — the tape is authoritative for // every element, walk it top-to-bottom.