[POC] Give source overlays their package's type environment and class-field semantics (T13)

benchmarks/regions/README.md

@@ -7,6 +7,7 @@ Programs that build generations of `ProgramLifetime` objects and free each gener
 | `generations.ts` | Per-file syntax trees and binder symbols in one region per file, checker state (types, signatures, node and symbol links in paged tables, literal and union caches) in one region per generation. `edit` re-parses one file per generation, `all` re-parses every file. |
 | `compiler.ts` | A tokenizer, recursive-descent parser, binder and checker over generated sources, rebuilt from scratch in one region per build, like a watch mode that does not reuse syntax trees. |
 | `emit.ts` | A large live graph of ordinary objects with parent links, standing in for a type checker, and a per-file emit that builds its output in a region released right after. |
+| `requests.ts` | Batches of concurrent async request handlers, each in its own region across its awaits: tokenize a JSON body, parse it into a syntax tree that points at its tokens, validate it into issues and an index, render a response, release the region. |
 
 ```console
 $ node benchmarks/regions/run.mjs --baseline=../scriptc-main --runs=7

benchmarks/regions/emit.ts

@@ -54,7 +54,7 @@ let level: Type[] = [new Type(id++, undefined)];
 while (id < typeCount) {
   const next: Type[] = [];
   for (const t of level) for (let k = 0; k < 4 && id < typeCount; k++) next.push(new Type(id++, t));
-  roots.push(...next);
+  for (const t of next) roots.push(t);
   level = next;
 }
 

benchmarks/regions/requests.ts

@@ -0,0 +1,221 @@
+// An async request handler: each request tokenizes a JSON body, awaits (as
+// if reading more of it), parses the tokens into a syntax tree whose nodes
+// point back at their tokens, awaits again (as if loading reference data),
+// validates the tree into issues held in an index, and renders a response.
+// Requests run in batches of concurrent handlers, so the handlers of one
+// batch interleave at every await.
+//
+//   requests <requests> <concurrency> <items-per-body> [plain]
+//
+// Each request runs in its own LifetimeRegion, released when its response
+// is rendered; `plain` never constructs a region.
+import { readFileSync } from "node:fs";
+import type { ProgramLifetime } from "@scriptc/types";
+import { LifetimeRegion } from "@scriptc/experimental";
+
+const T_PUNCT = 0;
+const T_STR = 1;
+const T_NUM = 2;
+const T_WORD = 3;
+
+class Token implements ProgramLifetime {
+  readonly kind: number;
+  readonly text: string;
+  readonly pos: number;
+  constructor(kind: number, text: string, pos: number) {
+    this.kind = kind;
+    this.text = text;
+    this.pos = pos;
+  }
+}
+
+const N_OBJECT = 0;
+const N_ARRAY = 1;
+const N_STRING = 2;
+const N_NUMBER = 3;
+const N_LITERAL = 4;
+
+class JsonNode implements ProgramLifetime {
+  readonly kind: number;
+  readonly token: Token;
+  parent: JsonNode | undefined = undefined;
+  readonly keys: string[] = [];
+  readonly children: JsonNode[] = [];
+  constructor(kind: number, token: Token) {
+    this.kind = kind;
+    this.token = token;
+  }
+}
+
+class Issue implements ProgramLifetime {
+  readonly path: string;
+  readonly node: JsonNode;
+  readonly message: string;
+  constructor(path: string, node: JsonNode, message: string) {
+    this.path = path;
+    this.node = node;
+    this.message = message;
+  }
+}
+
+// ---- request bodies (deterministic) ----
+
+function body(id: number, items: number): string {
+  let seed = 7 + id * 7919;
+  const rand = (n: number): number => {
+    seed = (seed * 1103515245 + 12345) % 2147483648;
+    return seed % n;
+  };
+  const parts: string[] = [];
+  for (let i = 0; i < items; i++) {
+    const qty = rand(12) - 1;
+    const sku = rand(9) === 0 ? "" : `sku-${rand(5000)}`;
+    const tags = Array.from({ length: rand(4) }, (_, k) => `"t${k}${rand(10)}"`).join(",");
+    parts.push(`{"sku":"${sku}","qty":${qty},"price":${rand(10000) / 100},"gift":${rand(2) === 0},"tags":[${tags}]}`);
+  }
+  return `{"order":${id},"customer":"c${id % 97}","items":[${parts.join(",")}]}`;
+}
+
+// ---- handler ----
+
+function tokenize(text: string): Token[] {
+  const tokens: Token[] = [];
+  let i = 0;
+  while (i < text.length) {
+    const c = text.charCodeAt(i);
+    const start = i;
+    if (c === 34) {
+      i++;
+      while (text.charCodeAt(i) !== 34) i++;
+      i++;
+      tokens.push(new Token(T_STR, text.slice(start + 1, i - 1), start));
+    } else if (c === 45 || (c >= 48 && c <= 57)) {
+      i++;
+      while (i < text.length && (text.charCodeAt(i) === 46 || (text.charCodeAt(i) >= 48 && text.charCodeAt(i) <= 57))) i++;
+      tokens.push(new Token(T_NUM, text.slice(start, i), start));
+    } else if (c >= 97 && c <= 122) {
+      while (i < text.length && text.charCodeAt(i) >= 97 && text.charCodeAt(i) <= 122) i++;
+      tokens.push(new Token(T_WORD, text.slice(start, i), start));
+    } else {
+      i++;
+      tokens.push(new Token(T_PUNCT, text[start]!, start));
+    }
+  }
+  return tokens;
+}
+
+class Parser {
+  private at = 0;
+  private readonly tokens: Token[];
+  constructor(tokens: Token[]) {
+    this.tokens = tokens;
+  }
+  value(): JsonNode {
+    const token = this.tokens[this.at++]!;
+    if (token.kind === T_STR) return new JsonNode(N_STRING, token);
+    if (token.kind === T_NUM) return new JsonNode(N_NUMBER, token);
+    if (token.kind === T_WORD) return new JsonNode(N_LITERAL, token);
+    const node = new JsonNode(token.text === "{" ? N_OBJECT : N_ARRAY, token);
+    const close = token.text === "{" ? "}" : "]";
+    while (this.tokens[this.at]!.text !== close) {
+      if (node.kind === N_OBJECT) {
+        node.keys.push(this.tokens[this.at]!.text);
+        this.at += 2; // the key and the colon
+      }
+      const child = this.value();
+      child.parent = node;
+      node.children.push(child);
+      if (this.tokens[this.at]!.text === ",") this.at++;
+    }
+    this.at++;
+    return node;
+  }
+}
+
+function member(node: JsonNode, key: string): JsonNode | undefined {
+  const i = node.keys.indexOf(key);
+  return i < 0 ? undefined : node.children[i];
+}
+
+function validate(root: JsonNode, issues: Issue[], index: Map<string, JsonNode>): number {
+  let total = 0;
+  const items = member(root, "items")!;
+  for (let i = 0; i < items.children.length; i++) {
+    const item = items.children[i]!;
+    const path = `items[${i}]`;
+    const sku = member(item, "sku")!;
+    if (sku.token.text === "") issues.push(new Issue(`${path}.sku`, sku, "empty sku"));
+    else if (index.has(sku.token.text)) issues.push(new Issue(`${path}.sku`, sku, `duplicate of ${index.get(sku.token.text)!.token.pos}`));
+    else index.set(sku.token.text, item);
+    const qty = Number(member(item, "qty")!.token.text);
+    if (qty <= 0) issues.push(new Issue(`${path}.qty`, member(item, "qty")!, "quantity must be positive"));
+    else total += qty * Number(member(item, "price")!.token.text);
+    const tags = member(item, "tags")!;
+    for (const tag of tags.children) if (tag.token.text.endsWith("0")) issues.push(new Issue(`${path}.tags`, tag, "reserved tag"));
+  }
+  return total;
+}
+
+const hop = (): Promise<void> => new Promise((resolve) => setImmediate(resolve));
+
+async function handle(text: string): Promise<string> {
+  const tokens = tokenize(text);
+  await hop();
+  const root = new Parser(tokens).value();
+  await hop();
+  const issues: Issue[] = [];
+  const index = new Map<string, JsonNode>();
+  const total = validate(root, issues, index);
+  await hop();
+  const first = issues[0];
+  return `${member(root, "order")!.token.text} ${tokens.length} ${index.size} ${issues.length} ${total.toFixed(2)}` +
+    (first === undefined ? "" : ` ${first.path}@${first.node.token.pos}`);
+}
+
+// ---- driver ----
+
+function rssMb(key: string): number {
+  const m = new RegExp(`${key}:\\s*(\\d+) kB`).exec(readFileSync("/proc/self/status", "utf8"));
+  return m === null ? 0 : Number(m[1]) / 1024;
+}
+
+const args = process.argv.slice(2);
+const requests = Number(args[0] ?? "4000");
+const concurrency = Number(args[1] ?? "16");
+const items = Number(args[2] ?? "100");
+const useRegions = args[3] !== "plain";
+
+async function serve(id: number): Promise<string> {
+  const text = body(id, items);
+  if (!useRegions) return await handle(text);
+  const region = new LifetimeRegion();
+  const response = await region.run(() => handle(text));
+  region.release();
+  return response;
+}
+
+async function main(): Promise<void> {
+  let hash = 0;
+  let lines = 0;
+  let sample = "";
+  let rssFirst = 0;
+  const t0 = performance.now();
+  for (let next = 0; next < requests; next += concurrency) {
+    const batch: Promise<string>[] = [];
+    for (let k = next; k < next + concurrency && k < requests; k++) batch.push(serve(k));
+    for (const response of await Promise.all(batch)) {
+      for (let i = 0; i < response.length; i++) hash = (hash * 31 + response.charCodeAt(i)) % 1000000007;
+      lines++;
+      if (lines === 1) sample = response;
+    }
+    if (next === 0) rssFirst = rssMb("VmRSS");
+  }
+  const ms = performance.now() - t0;
+  console.log(`${lines} responses, hash ${hash}, first ${sample}`);
+  console.log(
+    `total ${ms.toFixed(1)} ms per-request ${((ms * 1000) / requests).toFixed(1)} us` +
+      ` rss-first ${rssFirst.toFixed(1)} MB rss-last ${rssMb("VmRSS").toFixed(1)} MB peak ${rssMb("VmHWM").toFixed(1)} MB`,
+  );
+}
+
+main();

benchmarks/regions/run.mjs

@@ -39,6 +39,8 @@ const CONFIGS = [
   { name: "compiler-one-shot", program: "compiler.ts", args: ["400", "60", "1"] },
   // Per-file emit regions next to a large live heap of ordinary objects.
   { name: "emit-files", program: "emit.ts", args: ["400000", "200", "2000"] },
+  // Concurrent async request handlers, one region per request across its awaits.
+  { name: "requests-async", program: "requests.ts", args: ["4000", "16", "100"] },
 ];
 
 const work = join(tmpdir(), `scriptc-region-bench-${process.pid}`);
@@ -93,7 +95,7 @@ for (const config of CONFIGS) {
       const lines = r.stdout.trim().split("\n");
       outputs.set(name, lines[0].replace(/ gen0 .*/, ""));
       const metrics = {};
-      for (const [, key, value] of lines.at(-1).matchAll(/([a-z0-9-]+) ([0-9.]+) (?:ms|MB)/g))
+      for (const [, key, value] of lines.at(-1).matchAll(/([a-z0-9-]+) ([0-9.]+) (?:ms|us|MB)/g))
         metrics[key] = Number(value);
       samples[name].push(metrics);
     }

benchmarks/runtime/README.md

@@ -19,6 +19,7 @@ Application-shaped workloads for measuring the speed of compiled executables. Ea
 | `intern-table`, `intern-table-map`, `intern-table-1`, `intern-table-map-1` | cpu | worker threads interning the same type-like objects under string keys and records under number keys, then hits and misses: one shared `InternTable` per kind of key, or a module-level `Map` per thread (the `-map` entries), with 8 threads or 1 (the `-1` entries); under Node.js every thread has its own tables |
 | `atomics-contended` | cpu | four worker threads updating `SharedArrayBuffer` counters with `Atomics`: one shared counter, one counter per worker, a `compareExchange` loop, and `Atomics.wait`/`notify` handoffs |
 | `value-objects` | cpu | small immutable records and class instances: vector helpers, method chains, particle updates |
+| `value-records` | cpu | records that hold strings, spread state updates and destructured tuples: token helpers, an immutable reducer, tuple returns |
 | `region-builds` | cpu | a watch-mode build loop over `LifetimeRegion`: each build binds a program in a new region and releases the previous one, which a cache keeps referenced for two builds |
 | `optional-fields` | cpu | records and class instances with `number \| undefined` and other scalar-union fields |
 | `record-boundaries` | cpu | records built from unchecked array reads passed to and returned from annotated functions |

benchmarks/runtime/workloads.json

@@ -141,6 +141,13 @@
         "2"
       ]
     },
+    {
+      "name": "value-records",
+      "entry": "workloads/value-records.ts",
+      "args": [
+        "1"
+      ]
+    },
     {
       "name": "region-builds",
       "entry": "workloads/region-builds.ts",

benchmarks/runtime/workloads/value-records.ts

@@ -0,0 +1,93 @@
+// Records that hold references: a tokenizer that builds token records with
+// string fields and passes them to helpers, an immutable reducer that
+// updates its state with object spread, and tuple-returning helpers whose
+// results are destructured at once. None of these temporaries outlives
+// the statement or loop iteration that builds it.
+interface Token {
+  kind: string;
+  text: string;
+  pos: number;
+}
+const KEYWORDS = ["let", "const", "if", "else", "for", "while", "return", "function"];
+function classify(text: string): string {
+  return KEYWORDS.includes(text) ? "keyword" : "ident";
+}
+function token(kind: string, text: string, pos: number): Token {
+  return { kind, text, pos };
+}
+function weight(t: Token): number {
+  return t.kind === "keyword" ? t.text.length * 3 : t.text.length + (t.pos & 7);
+}
+function tokenize(source: string): number {
+  let sum = 0;
+  let start = -1;
+  for (let i = 0; i <= source.length; i++) {
+    const c = i < source.length ? source.charCodeAt(i) : 32;
+    const word = (c >= 97 && c <= 122) || (c >= 48 && c <= 57) || c === 95;
+    if (word) {
+      if (start < 0) start = i;
+      continue;
+    }
+    if (start >= 0) {
+      const text = source.slice(start, i);
+      const t = token(classify(text), text, start);
+      sum += weight(t);
+      start = -1;
+    }
+    if (c !== 32 && c !== 10) {
+      const punct = { kind: "punct", text: source[i]!, pos: i };
+      sum += weight(punct) + (punct.text === ";" ? 1 : 0);
+    }
+  }
+  return sum;
+}
+
+interface State {
+  mode: string;
+  depth: number;
+  count: number;
+  last: string;
+}
+function step(s: State, text: string): State {
+  if (text === "{") return { ...s, depth: s.depth + 1, last: text };
+  if (text === "}") return { ...s, depth: s.depth - 1, last: text };
+  return s.mode === "code" ? { ...s, count: s.count + 1 } : { ...s, mode: "code", last: text };
+}
+function reduce(texts: string[], rounds: number): number {
+  let total = 0;
+  for (let r = 0; r < rounds; r++) {
+    let state: State = { mode: "start", depth: 0, count: 0, last: "" };
+    for (const text of texts) {
+      state = step(state, text);
+      if (state.depth > 3) state = { ...state, depth: 3 };
+    }
+    total += state.count + state.depth + state.last.length + state.mode.length;
+  }
+  return total;
+}
+
+function divmod(a: number, b: number): [number, number] {
+  return [Math.floor(a / b), a % b];
+}
+function split(text: string, at: number): [string, string] {
+  return [text.slice(0, at), text.slice(at)];
+}
+function tuples(n: number, texts: string[]): number {
+  let sum = 0;
+  for (let i = 0; i < n; i++) {
+    const [q, r] = divmod(i * 31, 17);
+    const [head, tail] = split(texts[i % texts.length]!, r % 3);
+    sum += q + r + head.length * 2 + tail.length;
+  }
+  return sum;
+}
+
+const scaleArg = Number(process.argv[2] ?? "1");
+const line = "function step ( s , text ) { if ( text ) { return s ; } let x = s ; while ( x ) { x = x ; } }\n";
+const source = line.repeat(Math.floor(400 * scaleArg));
+const texts = source.split(" ").filter((t) => t.length > 0);
+let lexed = 0;
+for (let i = 0; i < 240; i++) lexed += tokenize(source);
+const reduced = reduce(texts, Math.floor(600 * scaleArg));
+const pairs = tuples(Math.floor(2_000_000 * scaleArg), texts);
+console.log("tokens", lexed, "reduce", reduced, "tuples", pairs);

docs/content/docs/contracts.mdx

@@ -331,7 +331,7 @@ $ scriptc build graph.ts
 
 ## ProgramLifetime
 
-`ProgramLifetime` marks a class whose instances a program keeps until it exits, such as the nodes of a syntax tree, the symbols of a symbol table or interned types. scriptc then never reference counts, collects or frees instances of the class and of its subclasses: they are never freed, every retain and release on them compiles to nothing, and the cycle collector neither walks them nor needs to trace the objects that point to them. Node.js behavior is unchanged; only memory use differs, because an instance the program drops is never reclaimed.
+`ProgramLifetime` marks a class whose instances a program keeps until it exits, such as the nodes of a syntax tree, the symbols of a symbol table or interned types. scriptc then never reference counts, collects or frees instances of the class and of its subclasses: they are never freed, every retain and release on them compiles to nothing, and the cycle collector neither walks them nor needs to trace the objects that point to them. Node.js behavior is unchanged; only memory use differs, because an instance the program drops is never reclaimed. On Linux, the instances a worker thread creates are returned with the rest of its memory after it exits, unless the worker shared objects with other threads (see [cost](/docs/threads#cost)).
 
 ```ts:tree.ts
 import type { ProgramLifetime } from "@scriptc/types";

docs/content/docs/dependencies.mdx

@@ -69,6 +69,10 @@ $ scriptc build app.ts --npm-static=three -o app >/dev/null
 
 Naming a package skips the declaration requirement above, but the package still has to compile. `--npm-static=auto` uses the default automatic selection. The option accepts a comma-separated list and can be repeated.
 
+## CommonJS packages
+
+CommonJS packages compile with Node.js's module semantics. Each `require()` evaluates its module at the call, once, and later calls return the cached exports. A package can assign `module.exports` and run code before it requires its dependencies. If that code reads a `const` or `let` require binding before its `require` runs, the read throws the same `ReferenceError` as Node.js. A `var` binding that earlier code can reach is reported at build time. A `require()` of a JSON file, such as a package's own `package.json`, returns one object parsed at build time, shared by every module that requires it. A require cycle compiles when each of its modules assigns its exports before requiring anything, as semver's classes do, and uses the other modules only inside functions that its top-level code can't call. Other require cycles are reported at build time. The [limitations](/docs/limitations) page lists the supported `require` forms.
+
 ## Unsupported code inside a package
 
 A JavaScript package can contain operations scriptc can't compile, often in functions your program never calls. These don't fail the build: each one compiles to code that throws a runtime error if it ever runs. Coverage reports list them as deferred sites, so review the list and test the parts of the package your program uses.