Skip to content

Compile polyfill packages that are Node builtins as the builtins - #1458

Merged
ctate merged 55 commits into
mainfrom
opt/npm-polyfill-shims
Oct 20, 2026
Merged

ctate merged 55 commits into
mainfrom
opt/npm-polyfill-shims

Conversation

@cramforce

@cramforce cramforce commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #1434. This branch is based on opt/npm-cjs-exports and merges #1425 (opt/native-undici) for its package-shim registry. Review those first.

Summary

Polyfills that are builtins. Several small npm packages exist only to back-fill Node APIs for old Node versions and browsers. On Node 24, their modules evaluate to the builtin itself. scriptc now compiles a require() of such a package as that builtin expression, at the require's position. The package's own files never join the program, so their feature-detection code (conditional module.exports, {}.toString, typeof window) no longer has to compile.

A require resolves to its builtin in three places:

  • an opted-in package's files, using the version installed where Node's resolution from that file finds it;
  • the program's own CommonJS files;
  • an esbuild __commonJS chunk whose module key names the package and its version (pnpm's node_modules/.pnpm/<name>@<version>/node_modules/<name>/<file>). The chunk's call sites become the builtin and the chunk disappears.

The table lives in builtin-package-aliases.ts. package-shims.ts consults it through the installed version, like the undici entries. Each entry is limited to the major versions whose Node entry file I read:

PackageVersionsCompiles asWhy it is identical on Node 24
isarray0.x, 1.x, 2.xArray.isArraymodule.exports = Array.isArray || function …. Node always has Array.isArray, so the export is that function.
safe-buffer5.xrequire("buffer")module.exports = buffer whenever Buffer.from, alloc, allocUnsafe and allocUnsafeSlow exist, which they do since Node 6. The rest of the file only builds the unexported SafeBuffer fallback.
process-nextick-args2.xprocessmodule.exports = process unless process.version is v0.x or a pre-1.8 v1.x. Consumers read .nextTick, which is process.nextTick.
inherits2.x (main file inherits.js)require("util").inherits2.0.1 exports require('util').inherits directly. 2.0.3 and later do the same whenever it is a function, which it always is on Node. The browser file is not aliased.
util-deprecate1.x (main file node.js)require("util").deprecateThe Node entry is exactly module.exports = require('util').deprecate. The browser file is not aliased.

readable-stream (maintainer-approved divergence). 2.x, 3.x and 4.x compile as node:stream. For 2.x and 4.x that is the package's own READABLE_STREAM=disable behavior:

  • In 2.x, readable.js exports stream.Readable carrying Readable, Writable, Duplex, Transform, PassThrough and Stream. writable.js exports stream.Writable. duplex.js, transform.js and passthrough.js read the entry.
  • In 4.x, lib/ours/index.js exports a plain object of named stream exports.
  • The lib/_stream_*.js files map to the matching builtin classes in every aliased major. Under Node they load the package's own implementations in every environment, so this part goes beyond the disable mode.
  • How the entry's export is used picks the replacement. If every use reads one of the carried class names (stream.Transform, { Transform } = require(…)), the replacement is the builtin module. If every use is a value use (inherits(X, Readable), Readable.from), it is stream.Readable. A mixture of the two, or any 4.x value use, keeps the package's own module.
  • 3.x (a copy of Node 10's streams): readable.js exports stream.Readable carrying the stream classes, Stream, finished and pipeline; it has no root subpath files. Its own disable branch throws on Node 24: it copies every stream export onto stream.Readable with Object.assign, and Readable inherits the getter-only promises accessor. So 3.x maps the members its default branch exports onto the builtin's, and the docs say this is not a mode the package itself offers on Node 24.
  • The divergence is documented on the limitations page, in the dependencies page's built-in packages section, and in internal/compatibility/README.md.

Global detection. A condition built from typeof tests whose answers are static now folds:

  • an identifier nothing declares answers "undefined", Node's answer for window, self and document;
  • global, globalThis and process answer "object";
  • this covers esbuild's minified typeof x < "u" as well as !==, &&, || and ! chains.

So the untaken arm of typeof window < "u" ? window : … never lowers. A detection chain used as a receiver, as in readable-stream's (typeof global < "u" ? global : typeof window < "u" ? window : …).Uint8Array || function () {}, resolves to the global object it selects. JavaScript files no longer report a name that the same file feature-tests with typeof as undeclared; a reached use still meets its located refusal. TypeScript files keep the error and its hint.

Fixes found along the way:

  • typeof global.setTimeout and typeof globalThis.queueMicrotask answered "string", the identity token's type. They now answer "function".
  • var p = process (the shape process-nextick-args consumers take once it compiles as process) silently lost process's members: p.nextTick was not a function. A var alias of a surfaced global other than the global object now aliases like the const spelling when nothing reassigns it and no earlier code can read it, the rule var Buffer = buffer.Buffer already followed.
  • var Transform = require("stream").Transform; new Transform(…) was refused, while the const and destructured spellings compiled. The declaration was already alias plumbing (builtinMemberRequireDecl); stream and EventEmitter construction now follows it too.

Evidence

  • npm-static cases. All three check stdout and exit status against Node, and that util-deprecate's warning reaches stderr on both sides. readable-stream runs with READABLE_STREAM=disable on the Node side.
    • 2585-builtin-polyfill-aliases: the published releases of the five polyfills, installed under a consumer package.
    • 2586-readable-stream-builtin: readable-stream 2.3.8 (with its dependencies) through the entry's members, its value, the transform, passthrough, writable and duplex subpaths, and lib/_stream_passthrough.js. Also the 4.7.0 entry.
    • 2587-bundled-polyfill-aliases: real esbuild 0.27 output of a pnpm-installed source using all six packages.
  • Corpus programs:
    • readable-stream-disable/ sets READABLE_STREAM=disable before its first require, so Node takes the same branch. Its header notes that default-environment output differs: rs.Readable === stream.Readable is false there.
    • readable-stream-3/ runs the vendored readable-stream 3.6.2 under Node, without the disable variable, and keeps to behavior its streams share with the builtin: piping through transforms, the pipeline and finished callbacks, and error propagation. Before the alias it was refused (SC2013, no declarations).
    • global-detection-typeof.cjs: the detection idioms, the folded chains, the selected global's members, and typeof through the global object.
  • Unit tests:
    • the alias table per major;
    • pnpm key parsing;
    • bundled chunk aliasing, run against the original bundle under Node;
    • the use-dependent readable-stream replacement, including the mixed case that keeps the chunk.
  • The npm-static suite (120 tests), the affected npm cases, the corpus selection, pnpm lint, pnpm format:check and test:ts7 --baselines-only all pass.
  • Zero-cost gate against this branch's merge base: PASS. All 2502 programs (bench, self, corpus) are byte-identical. Bench geomean is −1.6% and self +0.6%, both within tolerance.
  • Zero-cost gate for the readable-stream 3.x commit (together with the stacked unloaded change), against 8a40fb5: PASS. All 2502 programs (bench, self, corpus) are byte-identical; bench geomean is −0.18%.
  • The self-hosted compiler builds its IR cleanly, and self-hosting-native-driver.test.ts passes.
  • pnpm format:check also flags lowerer.ts and type-mapper.ts. Those come from Compile undici from a built-in implementation #1425's tree; this branch does not touch them.

🤖 Generated with Claude Code

cramforce and others added 15 commits October 19, 2026 17:31
A statement-level require() of a package nothing installed threw a plain
Error; the expression form already carried Node's MODULE_NOT_FOUND code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@types/node 20 and older declare NodeJS.Timeout and NodeJS.Immediate as
classes; 22+ (and scriptc's own declarations) declare interfaces. Accept
both as the native timer handle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…encies unloaded

package-shims.ts registers implementation modules shipped with the
compiler per package and major version (chosen by the importer's
installed version) and reads the project's never-loaded packages from
"scriptc": { "packages": { "<name>": "unloaded" } }. Module edges go
through resolveProjectImport; the npm resolution, the npm-edge walk and
the --npm-static set skip such packages; the tsgo host reads a shimmed
package's index.d.ts as a re-export of its implementation; an unloaded
package's require() throws Node's not-installed MODULE_NOT_FOUND.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A TypeScript implementation over node:net, node:tls and node:events:
Dispatcher, Client (HTTP/1.1 with keep-alive), Pool, Agent,
buildConnector, request(), errors, the global dispatcher, fetch, Headers
and the diagnostics_channel publications, with per-version behavior.
Differential npm cases run the vendored real undici under Node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	packages/compiler/src/frontend/lowering/builtins/timers.ts
#	packages/compiler/src/frontend/type-mapper.ts
…polyfill-shims

# Conflicts:
#	packages/compiler/src/frontend/lowering/lower-modules.ts
#	tests/harness/npm.test.ts
isarray, safe-buffer, process-nextick-args, inherits and util-deprecate
evaluate to a Node builtin on Node 24 at their checked major versions, so
a require of one compiles as that builtin expression at the require's
position: from an opted-in package's files (the installed version where
Node's resolution finds it), from the program's own CommonJS files, and
through an esbuild __commonJS chunk whose pnpm path key names the package
and its version. The package's files never join the program.

readable-stream 2.x and 4.x compile as node:stream, the package's own
READABLE_STREAM=disable behavior (a documented divergence). A use of the
entry's export picks the builtin module (member reads of the carried
classes) or stream.Readable (value uses); a mixture keeps the package.
3.x stays as is: its disable branch throws on Node 24.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A condition built from typeof tests whose answers are static (an
undeclared global is "undefined", the global object and process are
"object") folds, so the untaken arm of `typeof window < "u" ? window : …`
never lowers, and a detection chain used as a receiver resolves to the
global object it selects. JavaScript files no longer report a name the
file feature-tests with typeof as undeclared. `typeof global.setTimeout`
answers "function" instead of the identity token's "string".

`var pna = process` (process-nextick-args once it compiles as process)
and `var Transform = require("stream").Transform` alias like their const
spellings when nothing reassigns them; before, the first silently lost
process's members.

Also reads symbol declarations directly in the CommonJS accessor
snapshot path, after the checker helper's removal on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pace imports

A class a bundled ES module's __esm initializer stores into a module
binding (`Multi = class extends import_stream.Readable { … }`) becomes a
class declaration at the same text position: the initializer's body splits
around it into run-once pieces, so a class over a runtime-provided base no
longer has to exist as a stored class value. Only definitions whose
evaluation runs no code (no static fields, static blocks or computed names;
heritage absent, a builtin module member, or a hoisted class), declared
once, assigned only there and read only inside functions after the
assignment move.

npm selection now also scans the program modules a literal import()
reaches, with their static graphs: a workspace package loaded on demand
had its npm dependencies refused as "not part of the compiled module
graph".

`new import_x.C()` respelled `new require("x").C()`, which constructs the
require call; the interop binding now becomes `(require("x"))` there.

util.inherits onto a runtime-provided class (stream.Readable, the from2
and multistream shape) fences where it is written instead of failing in
Node's argument validation at run time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
scriptc Ready Ready Preview, v0 Oct 20, 2026 11:22pm UTC

…sting

The self-hosted compiler needs the same record type on both sides of the
call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel-security-reviewer

Copy link
Copy Markdown

Security review details

@vercel vercel Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional findings

  1. packages/compiler/src/frontend/npm-static-bundled-chunks.ts:562-564 — Builtin alias replacements can bind to a local process or Array instead of the builtin, changing the required package's value or making initialization fail.

    Fix on Vercel

  2. packages/compiler/src/frontend/package-shims.ts:180-185 — An installed polyfill marked unloaded in scriptc.packages is still rewritten to its builtin, so its require succeeds instead of throwing MODULE_NOT_FOUND.

    Fix on Vercel

  3. packages/compiler/shims/undici/core/request.ts:230-233 — A Host header containing CR/LF bypasses value validation and is written verbatim to the socket, allowing header or request injection in compiled undici clients.

    Fix on Vercel

cramforce and others added 2 commits October 19, 2026 21:44
Renumbers this branch's npm cases 2569-2571 to 2585-2587: #1411 now uses
2571.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Renumbers this branch's npm case 2572 to 2588, after the renumbered
polyfill cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cramforce and others added 6 commits October 19, 2026 22:44
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	tests/harness/npm-cases.ts
fs-extra builds its exports at load time (computed keys over graceful-fs,
universalify wrappers), which the static CommonJS model cannot follow, so
programs reading its members through the default import failed. The
package-shim registry now maps fs-extra 8.x-11.x to a TypeScript
implementation over node:fs: fs's members re-exported, the wrapped methods
following universalify's callback-or-promise rule over fs.promises, and
the extras (copy, move, remove, emptyDir, mkdirs, ensureFile, pathExists,
outputFile, the JSON helpers and their aliases) with fs-extra's messages.
Its declarations stand in for @types/fs-extra, including the default
import, and a namespace import resolves to the shim. What it does not
implement is not declared, so using it is a type error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ctate added 2 commits October 20, 2026 08:29
TypeScript's ES module emit for an exported enum is `export var Kind;`
followed by the IIFE. The rewrite replaced the declaration with a plain
`var Kind = { ... };`, so importers of the module lost the export. The
replacement now keeps the modifiers before `var`.
- An if statement whose condition is a global-detection test
  (static-typeof.ts) lowers only its taken arm, unless the untaken arm
  declares a var or function binding. Code Node never runs no longer
  counts as a deferred site, so global-detection-typeof.cjs is fully
  static.
- The resolver parity sweep skips specifiers the package-shim registry
  serves: they resolve to the built-in module by design.
- undici-shim-types.test.ts joins the TypeScript 5 import allowlist.
- node-types-timers uses a 200 ms timer, so the immediate runs first on
  a loaded machine as well.
Base automatically changed from opt/npm-esm-wrappers to main October 20, 2026 14:06
cramforce and others added 2 commits October 20, 2026 07:22
readable-stream 3.x is a copy of Node 10's streams. Like 2.x and 4.x it
now compiles as node:stream: readable.js as stream.Readable carrying the
stream classes, Stream, finished and pipeline, and lib/_stream_*.js as the
matching builtin classes. Unlike them, this is not the package's own
READABLE_STREAM=disable mode, which throws on Node 24 (it Object.assigns
every stream export onto stream.Readable, which inherits the getter-only
`promises` accessor). The limitations and dependencies pages and the
compatibility notes say so.

The corpus program readable-stream-3 runs the vendored 3.6.2 under Node and
keeps to behavior its streams share with the builtin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ctate added 2 commits October 20, 2026 10:36
…ource

copy and copySync now follow fs-extra's onLink: only a destination that
is itself a symlink is replaced; over a regular file or directory the
symlink call fails with EEXIST and the destination is kept, whatever
overwrite says. The fs-extra corpus program covers both forms.

@vercel vercel Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional findings

  1. packages/compiler/shims/fs-extra/core/api.ts:1083-1085 — Recursive copySync can delete a source file when a nested destination symlink points back into the source tree because child paths bypass the same-inode check.

    Fix on Vercel

…e hint

Main's supported-module hint now lists dns/promises, stream/web and v8,
so the SC0001 hints recorded for readable-stream-3 and
readable-stream-disable no longer matched.
@ctate
ctate merged commit f3da708 into main Oct 20, 2026
9 checks passed
@ctate
ctate deleted the opt/npm-polyfill-shims branch October 20, 2026 23:28

This branch was successfully deployed

1 active deployment
Preview — 74633db9 Deployed Oct 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants