Skip to content

Let NodeJS stream interface slots hold JavaScript stream objects - #1524

Merged
ctate merged 4 commits into
mainfrom
opt/node-stream-dyn
Oct 20, 2026
Merged

ctate merged 4 commits into
mainfrom
opt/node-stream-dyn

Conversation

@cramforce

Copy link
Copy Markdown
Contributor

Problem

Under the general stream representation (#1518), NodeJS.ReadableStream and NodeJS.WritableStream compile as the runtime Readable and Writable classes. JavaScript code also passes its own stream objects through TypeScript functions typed with these interfaces: constructor-function streams, plain objects implementing on and pipe, and the streams of bundled npm code. Such a value entering the slot took the checked unwrap, so it threw a TypeError (or the not-supported error at pipe and pipeline stages) instead of passing through.

Decision

The choice stays per program, as a further step of the #1518 decision. A program whose reachable code moves an untyped value into a runtime stream class slot (the checked unwrap), or passes an untyped pipe() destination or pipeline stage, is lowered once more with checked-dynamic stream slots:

  • NodeJS.ReadableStream and NodeJS.WritableStream, and intersections over them, map to the checked-dynamic representation. A native stream entering the slot is boxed as its typed capsule, so it unboxes to the same stream and keeps its identity in JavaScript. Any other value is the JavaScript object itself, and returning it to JavaScript gives back that object.
  • A member call on such a value dispatches per value at run time. stream.dynNative tests whether the value is a boxed native stream with the needed side. A native stream takes the ordinary class-method lowering on the unboxed stream, the same lowering a typed Readable or Writable gets. Any other value calls its own member through dynInvoke, with typed callbacks boxed behind checked thunks.
  • readable.pipe(destination) into an untyped destination pipes natively when the destination is a native stream. Otherwise stream.pipeDyn follows Node's Readable.prototype.pipe: it registers the same listeners in the same order (end and data on the source; unpipe, a prepended error, close and finish on the destination), emits pipe, writes each chunk, pauses on false until drain, ends the destination, and unpipes and cleans up on finish, close or error.
  • child.stdout, child.stderr and child.stdin reads keep producing the bridged runtime streams, and process.stdout and process.stderr box as their fd-backed Writable where they flow into such a slot.
  • for await over a checked-dynamic NodeJS.ReadableStream iterates a native stream and throws the not-supported error for other values. A pipeline stage that is not a native stream keeps the not-supported error, and its message now names JavaScript stream objects.

stream.Readable and stream.Writable (under @types/node), fs.ReadStream, the zlib classes and the other class spellings keep the runtime classes; only the two global interfaces become checked-dynamic.

Changes

  • dyn-streams.ts (new): the interface test, per-value member dispatch, the native/other branch helper, and for await over the slot.
  • type-mapper.ts: the dynNodeStreams mapping, including intersections and the ChildProcess stdio members.
  • lowerer.ts, general-streams.ts: the trigger (noteUntypedStreamFlow) and the extra lowering pass; receiverTypeOverrides for the native arm. lower-calls.ts calls the dispatch first among member-call lowerings, and lowerObjectMethodCall honors the receiver override.
  • lower-stream.ts: pipe() into an untyped destination branches between the native pipe and stream.pipeDyn.
  • stream.dynNative and stream.pipeDyn libCalls (signatures, validator, LLVM emission, effects).
  • scr_stream_dyn.c (new runtime unit) behind a new streamDyn runtime feature, so programs that do not use these libCalls link exactly the objects they linked before.
  • Docs limitations: the stream page describes the new boundary.

Zero cost

Programs without such a flow take the same lowering passes as before, and the new runtime code sits in its own unit selected only by the new libCalls. Gating that unit on the existing stream and dyn-invoke features instead added 40 bytes to 34 existing executables, so it has its own feature.

node scripts/zero-cost-gate.mjs --baseline=<opt/node-stream-interfaces 1f58ff625> --candidate=<this branch> --corpus=all: 3316 programs identical in IR, LLVM and release executables, 1 intended difference, 1 program fails to build on both trees for an unrelated environment reason (node24-file-url-options.ts). Compile time is within tolerance (benchmarks geomean -0.62%, self-hosted compiler -0.42%).

The intended difference is 6467-node-stream-untyped-sources: it returns an untyped JavaScript value as NodeJS.ReadableStream and casts unknown to it, so it now takes checked-dynamic stream slots (and links the dyn-invoke unit). Its output is unchanged.

Tests

  • tests/corpus/6480-node-stream-dyn-readable/: a plain object implementing on/pipe, a constructor-function stream and a native Readable pass through a function typed NodeJS.ReadableStream; TypeScript consumes them with on("data") and pipe() into a native Writable; JavaScript consumes the returned values and sees the same objects; a native Readable typed NodeJS.ReadableStream pipes into JavaScript sinks, one of which records the listeners Node's pipe protocol registers and removes.
  • tests/corpus/6481-node-stream-dyn-writable/: NodeJS.WritableStream slots holding a native Writable, a plain JavaScript sink and process.stdout; a copy helper over both interfaces pipes native and JavaScript sources into native and JavaScript destinations.
  • tests/corpus/6482-node-stream-dyn-helpers/: an async helper returning a file stream or a JavaScript part, pipe(out, { end: false }) and unpipe, pause/resume/setEncoding/readable, an optional destroy?.() through an intersection type, and for await and pipeline over native values in the slot.

All pass under Node and native in the plain and sanitized lanes.

With #1512 merged locally (not part of this branch), constructor functions built with util.inherits over EventEmitter and the legacy Stream also pass through these slots: on("data"), pipe() into native and legacy destinations, and handing them back to JavaScript all match Node. Those cases are not in the corpus here because they need #1512.

Not covered

  • pipeline() with a stage that is a JavaScript stream object throws the not-supported error. Node's pipeline over such objects depends on its end-of-stream detection for legacy streams, which this change does not model.
  • for await over a JavaScript stream object throws the not-supported error.
  • Native streams expose no emit to JavaScript, so a JavaScript stream whose own pipe() emits on its destination (the legacy Stream pipe from Let constructor functions inherit from EventEmitter and the stream classes #1512) cannot pipe into a native stream. Adding emit to the native stream view would change the runtime unit existing programs link.
  • A property read on a JavaScript stream object through the slot converts to the interface's declared type, so a missing readable throws instead of reading undefined.
  • stream.pipeDyn pauses the source when its JavaScript destination unpipes, as Node does for a source's last pipe, even if the source also pipes into native destinations.

🤖 Generated with Claude Code

cramforce and others added 2 commits October 20, 2026 06:49
Under the general stream representation, NodeJS.ReadableStream and
NodeJS.WritableStream compile as the runtime Readable and Writable
classes, so a JavaScript stream object (a constructor-function stream, a
plain object with on/pipe, a stream from bundled npm code) passed through
a TypeScript function typed with them took the checked unwrap and threw.

A program that moves an untyped value into a runtime stream class slot,
or passes an untyped pipe() destination or pipeline stage, now lowers
once more with checked-dynamic stream slots: the two interfaces (and
intersections over them) map to dyn. A native stream rides its typed
capsule and keeps its identity; any other value is the JavaScript object
itself. Member calls dispatch per value: stream.dynNative tests for a
boxed native stream, which takes the ordinary class-method lowering;
other values call their own member through dynInvoke. pipe() into a
JavaScript destination follows Node's Readable.prototype.pipe listener
protocol (stream.pipeDyn). for await and pipeline stages keep the
not-supported error for values that are not native streams.

The runtime code lives in scr_stream_dyn.c behind a new streamDyn
feature, so programs without these libCalls link the same objects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6480 passes a plain object, a constructor-function stream and a native
Readable through a NodeJS.ReadableStream function and consumes them in
TypeScript (on("data"), pipe()) and JavaScript, including identity and a
native pipe into JavaScript sinks. 6481 covers NodeJS.WritableStream
slots and a copy helper across native and JavaScript sources and
destinations. 6482 covers the helper shapes real code writes: an async
producer, pipe with { end: false } and unpipe, pause/resume/setEncoding,
an optional destroy through an intersection type, for await and pipeline
over native values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
scriptc Ready Ready Preview, v0 Oct 20, 2026 5:29pm UTC

Comment thread packages/compiler/src/frontend/lowering/general-streams.ts Outdated
Comment thread packages/compiler/src/frontend/type-mapper.ts
Comment thread packages/runtime/src/scr_stream_dyn.c
Base automatically changed from opt/node-stream-interfaces to main October 20, 2026 16:24
Comment thread packages/runtime/src/scr_stream_dyn.c Outdated
With checked-dynamic stream slots, readable.unpipe() also unpipes every
JavaScript destination stream.pipeDyn registered from the source, oldest
pipe first. Each destination hears 'unpipe' and the source pauses, as in
Node's unpipe(), which empties every pipe. Before, those pipes stayed
attached and later chunks still reached them.
@ctate
ctate merged commit 73a5956 into main Oct 20, 2026
18 of 22 checks passed
@ctate
ctate deleted the opt/node-stream-dyn branch October 20, 2026 17:32
euphemism pushed a commit to euphemism/scriptc that referenced this pull request Oct 20, 2026
…e headers

The backend can emit scr_wasm_memory_construct, scr_wasm_table_construct,
scr_wasm_global_construct, scr_wasm_construct_without_new,
scr_wasm_module_imports, scr_wasm_module_exports,
scr_wasm_module_custom_sections and scr_wasm_value_of (vercel-labs#1410), and
scr_stream_dyn_native, scr_stream_pipe_dyn and scr_stream_unpipe_dyn
(vercel-labs#1524). Their prototypes were in scr_wasm.c or nowhere, so the runtime
ABI guard failed. They are now in scr_wasm.h and scr_stream.h, and
scr_wasm.c no longer repeats the header's declarations.

The speculative-hoisting test relied on `v instanceof Num` in a
JavaScript function failing, which constructor-function instanceof now
compiles (vercel-labs#1512). Its hoisted function reads the prototype through
Reflect.getPrototypeOf instead, which still fences.

This branch was successfully deployed

1 active deployment
Preview — 28f16464 Deployed Oct 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants