Skip to content

Compile undici from a built-in implementation - #1425

Open
cramforce wants to merge 24 commits into
mainfrom
opt/native-undici
Open

cramforce wants to merge 24 commits into
mainfrom
opt/native-undici

Conversation

@cramforce

@cramforce cramforce commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

scriptc now implements undici itself and compiles that implementation in place of the package's published JavaScript. It does this through a general mechanism that maps an npm package name to a TypeScript module shipped with the compiler. The same registry also lets a project mark optional dependencies as never loaded.

Package shims (packages/compiler/src/frontend/package-shims.ts)

  • Implementation entries. A registry entry maps a package name and a supported major version to a module under packages/compiler/shims/. undici 5.x maps to shims/undici/v5.ts and 7.x to shims/undici/v7.ts.
    • The importing package's installed package.json picks the entry. A program with undici 5.12 in one workspace package and 7.30 in another compiles each against its own version.
    • When the package isn't installed, or its major version has no entry, nothing is shimmed and it resolves exactly as before.
  • Where the redirect happens.
    • Module edges go through resolveProjectImport (preflight, module order, the CommonJS link). It runs after provenance and tsconfig paths, so a project mapping still wins.
    • resolveNpmImport7, the pipeline's npm-edge walk and the --npm-static package set all skip shimmed packages.
  • How the checker sees the shim. This uses the tsgo host's file read, not tsconfig paths, because paths maps every importer to one file. The installed package's index.d.ts reads as export * from "<shim>", so types resolve per importer, the way the real declarations do.
    • Type errors inside shim files don't count as the project's own errors. They are compiler-owned code, much like library declarations under skipLibCheck.
  • Unloaded entries. "scriptc": { "packages": { "encoding": "unloaded" } } in the entry's package.json marks a package as never loaded.
    • require("encoding") then throws Node's not-installed MODULE_NOT_FOUND, so try { require("encoding") } catch {} takes its absent branch, and the package never joins the program. This applies decision D6 to node-fetch's encoding.
    • A static import of such a package reports Node's ERR_MODULE_NOT_FOUND.
  • Configured stand-ins (for example a no-op dd-trace tracer) are documented in the module header but not implemented. They are an implementation entry whose module the project supplies. A tsconfig paths mapping already does this today.
  • Shipping. The shims ship in the compiler package (files) and in native distributions (declarations/shims).

The undici shim (packages/compiler/shims/undici/)

The shim is plain TypeScript over node:net, node:tls and node:events. core/* is shared, and the v5.ts/v7.ts entries subclass Client, Pool and Agent with a version profile, so instanceof and default factories match each version.

  • Clients and dispatch.
    • Dispatcher is an EventEmitter with the dispatch(opts, handler) contract. It accepts both the 5.x handler callbacks and 7.x's controller API (onRequestStart and the rest).
    • Client is HTTP/1.1 over one connection: keep-alive reuse with Keep-Alive hints, headersTimeout/bodyTimeout/idle timeouts, a request queue, maxRequestsPerClient, and back-pressure. 7.x uses undici's 499 ms fast timers for headers and body timeouts.
    • Pool creates clients up to connections and exposes stats. 7.x drops clients whose connection failed.
    • Agent keeps one dispatcher per origin, with factory(origin, opts) and connect options. 7.x closes an origin's dispatcher once its connections are gone.
    • 5.x applies interceptors.Client, .Pool and .Agent. 7.x ignores them, as undici 7 does.
  • Connector. buildConnector dials over net/tls, honors connect.lookup for TCP, and applies the connect timeout.
  • request(). Returns { statusCode, statusText (7.x), headers, trailers, opaque, context, body }. Abort behaves per version: 5.x rejects with RequestAbortedError, 7.x with signal.reason. 5.x supports throwOnError; 7.x rejects it as undici 7 does.
  • Body. The response body follows Readable's contract for the members it declares: on/once/pause/resume/destroy, plus text/json/arrayBuffer/bytes/blob/dump. It is built on EventEmitter because scriptc does not compile node:stream subclasses under @types/node.
  • Errors. The errors module has undici's classes, names, messages and UND_ERR_* codes, with each version's export set.
  • Global dispatcher. setGlobalDispatcher/getGlobalDispatcher are provided.
  • fetch. Without a dispatcher it is the native fetch. Through a dispatcher it follows undici's fetch: default headers, redirect follow/manual/error, bad-port blocking, fetch failed errors with a cause, gzip/deflate decoding, and abort. Headers re-exports the native class.
  • diagnostics_channel. The shim publishes undici:request:* and undici:client:* with a stable boxed request per request, so WeakMap correlation works.

The rest of undici isn't declared, so using it is a compile-time type error. Known differences are listed in the docs:

  • The body can't be piped or used with for await.
  • Diagnostics messages omit the socket and connector.
  • fetch through a dispatcher returns an empty url and redirected: false, offers only gzip, deflate, and sends only string and byte bodies.
  • Pipelining above 1, CONNECT, stream bodies and socketPath are not supported.

Small compiler fixes found on the way

  • String(x) and template literals over string | URL unions now compile. The URL arm converts through href.
  • A statement-level require() of a missing package now throws with code: "MODULE_NOT_FOUND", as the expression form already did.
  • @types/node 20's class-declared NodeJS.Timeout/Immediate now map to the timer handle, as the interface form in @types/node 22+ already did.

Evidence

  • Differential npm cases, with a real vendored undici as the Node oracle and the shim as the native side, against a local node:http server: tests/fixtures/npm/cases/undici5 (5.12.0) and undici7 (7.30.0). Byte-identical stdout and exit, plain and sanitized, and stable over repeated runs. They cover:
    • request() with text, json, arrayBuffer, multi-value headers, chunked bodies and the default PUT method.
    • A transport-style Agent subclass with a per-origin Pool factory, a wrapped buildConnector, a client interceptor and pool stats.
    • Keep-alive reuse (connections opened), a Pool connection limit with queued and size stats, Client, and connect.lookup.
    • A subclass overriding dispatch.
    • The diagnostics-channel lifecycle.
    • Errors: connection refused, headers timeout, abort, throwOnError (5.x) and its 7.x refusal, protocol, and a destroyed body.
    • 7.x only: a controller-API handler, and fetch through an agent (POST, manual and followed redirects, refused, abort) and without one.
  • unloaded-optional-dependency: a node-fetch-style optional encoding require. tests/harness/package-shims.test.ts compiles it with encoding installed and asserts the output Node prints when it is absent. The same test covers an undici that isn't installed and an unsupported major version.
  • packages/compiler/test/undici-shim-types.test.ts:
    • A program written against undici's API type-checks with no diagnostics both against the real 5.12/7.30 declarations and against the shim, under @types/node.
    • Shape checks: shim exports, error classes and option keys are a subset of the real declarations, and response, stats and message records are assignable. The tolerated extras are listed with reasons.
  • Corpus 6250-string-of-string-or-url. Corpus 2195-require-missing-package now also prints the error code.
  • Zero-cost gate on the final candidate (bench, self, all corpus, two shards): 3188 programs identical, plus one intended difference. The one intended difference is 2195-require-missing-package, from the MODULE_NOT_FOUND code fix (+144 B). Self-compile time +0.4%, within tolerance.
  • @types/node 20 timers: tests/harness/node-types-timers.test.ts.
  • Other gates. pnpm lint, pnpm format:check, pnpm test:ts7 --baselines-only (4 new order baselines recorded, none changed), the docs gate, the self-host IR build (exit 0, no diagnostics), and the npm, npm-static and frontend unit suites. node24-url-authority fails identically on main in the sandbox.
  • Real tree (internal). With Compile esbuild's bundled CommonJS chunks and CommonJS cycles statically #1411, Compile class hierarchies across the CommonJS modules of npm packages #1412 and Serve Temporal's absence, v8, stream/web, dns/promises, EvalError and graceful-fs #1318 merged in, undici no longer compiles from its JavaScript: the shim modules join instead, and the transport-style code compiles against them. The one remaining shim site is fetch through a dispatcher: Headers.forEach has no lowering under @types/node.

🤖 Generated with Claude Code

cramforce and others added 6 commits October 19, 2026 17:31
A statement-level require() of a package nothing installed threw a plain
Error; the expression form already carried Node's MODULE_NOT_FOUND code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@types/node 20 and older declare NodeJS.Timeout and NodeJS.Immediate as
classes; 22+ (and scriptc's own declarations) declare interfaces. Accept
both as the native timer handle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…encies unloaded

package-shims.ts registers implementation modules shipped with the
compiler per package and major version (chosen by the importer's
installed version) and reads the project's never-loaded packages from
"scriptc": { "packages": { "<name>": "unloaded" } }. Module edges go
through resolveProjectImport; the npm resolution, the npm-edge walk and
the --npm-static set skip such packages; the tsgo host reads a shimmed
package's index.d.ts as a re-export of its implementation; an unloaded
package's require() throws Node's not-installed MODULE_NOT_FOUND.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A TypeScript implementation over node:net, node:tls and node:events:
Dispatcher, Client (HTTP/1.1 with keep-alive), Pool, Agent,
buildConnector, request(), errors, the global dispatcher, fetch, Headers
and the diagnostics_channel publications, with per-version behavior.
Differential npm cases run the vendored real undici under Node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
scriptc Ready Ready Preview, v0 Oct 20, 2026 5:32pm UTC

@vercel-security-reviewer

Copy link
Copy Markdown

Security review details

@vercel vercel Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional findings

  1. packages/compiler/shims/undici/core/client.ts:852-857 — When a response stays paused longer than bodyTimeout, resuming it leaves the timeout disabled, so a stalled server can hold the request and connection indefinitely.

    Fix on Vercel

  2. packages/compiler/shims/undici/core/request.ts:135-143 — Diagnostics subscribers calling the declared request.addHeader() method receive a TypeError because the published request view omits that method, breaking tracing-header injection.

    Fix on Vercel

  3. packages/compiler/shims/undici/core/request.ts:230-234 — A Host header containing CRLF bypasses value validation and is written verbatim to the socket, allowing header or request injection in programs using undici 7.x.

    Fix on Vercel

# Conflicts:
#	packages/compiler/src/frontend/lowering/builtins/timers.ts
#	packages/compiler/src/frontend/type-mapper.ts
cramforce and others added 2 commits October 19, 2026 18:53
…s as properties

@types/node's fetch declarations (undici-types) spell the Headers methods
and the body mixin as function-typed properties (readonly append: (...) =>
void), where scriptc's own declarations use methods. The receiver call
lowering treated such a standard-library property as a stored callback:
Headers.forEach was refused at build time and append/get/set/... compiled
to a runtime 'is not a function'. Standard-library members now take the
shared receiver dispatch either way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The net unit drained its poller every time the loop came back to the I/O
station, including the restarts that drain microtasks. I/O that became
ready during the iteration's own callbacks (a loopback connect started by
one of them) was delivered before the check phase, so setImmediate
callbacks queued earlier ran after it; Node delivers it in the next
iteration's poll phase. The poller now drains once per iteration (a full
batch keeps draining), reopening after the check phase. Back-to-back
undici requests now reuse connections like Node, so the fixtures drop
their setImmediate turn between requests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cramforce and others added 2 commits October 19, 2026 19:55
fetch's input under @types/node is string | URL | Request. A union with a
checked-dynamic arm collapses to the checked-dynamic representation only
when every sibling survives the collapse; URL did not qualify, so the type
(and anything typed as typeof globalThis.fetch, such as a custom fetch cast
through unknown) did not compile. URL's handle boxes by reference, keeps its
accessors and setters, answers instanceof URL and checks back out to the
typed value, so URL arms now join the collapse. A fetch() input typed
string | URL also converts through the checked-dynamic input path instead
of being narrowed to a string, which threw on the URL arm.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The undici5 and undici7 npm cases now also run peers.ts against Node with
the vendored undici: header values with CR or LF, chunk-size lines that
are valid, malformed, overflowing or 1 MiB long, and fetch bodies that
are endless (read once, then cancelled), gzipped, truncated, empty,
corrupt, cut off, aborted after the headers, redirected while endless,
or carry six content codings. The undici5 case leaves out the Host
values, which undici 5.12 writes into the request head, and the two
fetch cases where undici 5.12 differs (six codings, and an endless
redirect, which fails with a TypeError in 5.12).

package-shims.test.ts compiles two programs in copies of the cases: one
reads 64 MiB chunk-size lines and 64 MiB plain and gzipped fetch bodies
from a separate Node peer and checks that RSS stays under 48 MiB (plain
lane), the other checks that the 5.x profile rejects a Host value with
CR or LF with undici 5.29.0's error.
The undici cases compile the built-in undici implementation's module
graph; a sanitized build of it on a CI runner outlasted the default
120 s per case.
A full batch of 64 events polled again without returning to the event
loop. When synchronous socket callbacks kept 64 events ready on every
drain, timers and setImmediate callbacks never ran. The poll phase now
drains at most 48 full batches, the count libuv's uv__io_poll uses, and
the budget carries across the reopen that lets microtasks run, so the
remaining events wait for the next loop iteration.
cramforce and others added 2 commits October 20, 2026 02:50
Replaces the single-parent "Merge main" squash (6ab9a54) with a two-parent merge of the same tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The implicit constructor forwarded its arguments to EventEmitter's options-taking constructor, which compiles to a deferred runtime fence. Dispatcher takes no options, so it calls super() directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A redirect fetch followed or rejected was marked to drain: its body was
read and dropped until the server ended it, and the abort listener was
already removed. A body that never ends kept the connection busy after
cancellation and stalled a pool of one connection. Following or
rejecting a redirect now aborts its request, so the connection closes.
undici 7 leaves that connection open with its body paused, which stalls
the same pool; the follow-up request here gets a new connection.
ctate added 2 commits October 20, 2026 10:50
A redirect fetch followed or rejected was marked to drain: its body was
read and dropped until the server ended it, and the abort listener was
already removed. A body that never ends kept the connection busy after
cancellation and stalled a pool of one connection. Following or
rejecting a redirect now aborts its request, so the connection closes.
undici 7 leaves that connection open with its body paused, which stalls
the same pool; the follow-up request here gets a new connection.

This branch was successfully deployed

1 active deployment
Preview — 834a2bfc Deployed Oct 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants