Repository navigation
Give constructor functions native stream state over the stream classes - #1533
Merged
Merged
Conversation
Bundled npm code (from2, bl, tar-stream, yazl, multistream) builds its streams as ES5 constructor functions: util.inherits(Ctor, Readable) and Readable.call(this, opts), with _read/_write/_transform on the prototype. Readable.call(this) threw SC2020. A new runtime unit, scr_dyn_builtin_stream.c (feature dynBuiltinStream), gives the untyped stream constructors their stream half. Base.call(this, opts) creates a runtime stream over the receiver and keeps it in a hidden slot; the runtime calls the receiver's underscore methods through its property lookup; the stream prototypes carry Node's methods and accessors over the hidden stream; and EventEmitter's methods register listeners with the runtime stream's emitter, so runtime events and pipes reach JavaScript listeners with the receiver as this. _readableState/_writableState are live views. Stream class values now come from dyn.builtinStreamClass, so only programs that use them link the stream half. Typed code sees these objects as their runtime streams: checked casts to the stream classes take them, readable.pipe() into an untyped destination pipes into them, and runtime streams crossing into JavaScript answer the constructor object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pipe() into an untyped destination now comes from main's pipeIntoDyn (stream.unboxDyn); programs with constructor-function streams unbox through scr_dyn_builtin_stream.c, which also takes those streams, so the separate dyn.builtinStreamPipe entry point is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…orm corpus program Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Programs whose only stream value is the legacy Stream do not need the stream half. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
A constructor-function stream now reads `_final` and `_flush` when its writable side finishes, and `_transform` for each chunk, as Node does. A hook assigned after `Writable.call(this)` or `Transform.call(this)` runs, and an override of a PassThrough's `_transform` replaces the pass-through. Without `_final` or `_flush` the finish continues as Node's does. A runtime stream passed to a constructor-function stream's listeners, such as the source of a 'pipe' event, is the view whose pipe and unpipe accept constructor-function streams, so `src.unpipe(this)` in a 'pipe' listener unpipes instead of throwing.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #1512, which added the untyped EventEmitter and stream constructors.
Summary
Bundled npm code builds its streams as ES5 constructor functions:
from2, bl, tar-stream, yazl and multistream all do this (
Readable.call,Writable.call,Duplex.call,Transform.call,PassThrough.call). #1512 made the stream classes untyped constructor values with Node's prototype chain, but calling one with a constructor function's receiver threwSC2020.Native stream state on the receiver. A new runtime unit,
scr_dyn_builtin_stream.c, gives those constructors their stream half.Readable.call(this, opts)(and the other four) creates a runtime stream fromscr_stream.cwith the same option walknew Readable(untypedOptions)uses, and keeps it in a hidden slot of the receiver: an entry of the object's symbol table under a storage name no symbol produces, outside the key list, so reflection does not see it. As in Node, the constructor copies the option callbacks to_read,_write,_final,_destroy,_transformand_flush, setsallowHalfOpenon the duplex kinds, and adds_readableState/_writableState(read-only live views of the runtime state:highWaterMark,length,ended,endEmitted,flowing,destroyed,finished,needDrainand the other fields the runtime keeps). Listeners added before the constructor ran (bl adds itspipelistener first) are kept.Hooks. The runtime stream calls the receiver's
_read,_write,_transform,_flush,_finaland_destroythrough its property lookup with the receiver asthis, and with Node's argument shapes and error handling: a throwing_read,_final,_flushor_destroyreaches the stream the way Node's try/catch hands it on; a throwing_writeor_transformpropagates out ofwrite(). Completion callbacks keep the JavaScript error object, so'error'listeners receive the same object with itscode.Prototype methods. The stream prototypes carry Node's methods and accessors over the hidden stream:
push,unshift,read,setEncoding,pause,resume,isPaused,pipe,unpipe,destroy(with its callback),write,end,cork,uncork, the default_read/_write/_transform/_destroy, PassThrough's_transform, andreadable,readableFlowing,readableEnded,writableNeedDrain,destroyed(with Node's setter),closed,erroredand the other getters. EventEmitter's methods (from #1512) hand a receiver with stream state to this unit, which registers listeners with the runtime stream's own emitter, so the runtime's events and pipes reach JavaScript listeners with the receiver asthis; events with other names carry every argument (tar-stream'semit('entry', header, stream, next)).Typed code and runtime streams. In programs that link the unit, a checked cast of such an object to
Readable,Writable,Duplex,TransformorPassThroughtakes its runtime stream, a runtime stream that crosses into JavaScript shows as its constructor object, and the untyped views of other runtime streams pipe into constructor-function streams.stream.unboxDyn(#1518's untyped pipe destinations and pipeline stages) resolves toscr_dyn_builtin_stream_unboxin those programs, which takes constructor-function streams and otherwise behaves asscr_stream_unbox_dyn, sofs.createReadStream(file).pipe(ctorStream)and pipelines over them work. The backend picks these entry points only when the program links the unit; other programs keep their code.Readable, Writable, Duplex, Transform and PassThrough values now lower to
dyn.builtinStreamClass(EventEmitter and the legacyStreamkeepdyn.builtinClass), which links the newdynBuiltinStreamfeature. Programs that only use EventEmitter orStreamas a base do not link the stream half.scr_stream.cgains one function, thedestroyedsetter, which no existing program references.Not supported
These throw the not-supported Error where they run, and are listed on the limitations page:
objectMode,decodeStrings: false,signal,construct,writev, a non-UTF-8defaultEncoding);_writevand_constructmethods, symbol event names,listeners()/rawListeners(),wrap(), and strings written in encodings other than UTF-8;write()orend()(bl's callback form): the runtime pipe writes the destination's stream directly, where Node calls the overriding methods.Two existing divergences show up with these streams and are unchanged: a plain call made inside a listener or method keeps the caller's
this(soCtor()withoutnewinside a listener of anotherCtorinstance runs the constructor on that instance), and the runtime pipe does not emit'unpipe'on the destination when the source ends.Interplay with #1524
#1524 (
NodeJS.ReadableStream/WritableStreamas checked-dynamic slots) dispatches member calls on non-native values throughdynInvokeand pipes into them withstream.pipeDyn. Constructor-function streams are JavaScript objects there and work through their prototype methods. I verified this on a merge of an earlier state of this branch into an integration tree that contains #1518 and #1524: the new corpus programs, #1524's6480-6482, and6453/6454/5977pass. The conflicts are mechanical: both changes add a runtime feature to the same tables.Evidence
6560-constructor-function-readable.cjs: the from2 patterns (call-without-newguard,destroyedset before the base constructor,_readableState.highWaterMark, callback-driven_readwith asynchronous pushes, its owndestroy), paused mode with'readable'andread(),setEncoding,read(n)andunshift, an error from the source.6561-constructor-function-writable.cjs: a Writable with_write,_final, backpressure,'drain', write and end callbacks and the writable getters; a bl-style Duplex with pre-constructorpipe/unpipelisteners, anend()override callingDuplex.prototype.end, and reading back; piping between constructor-function streams with backpressure and{ end: false }.6562-constructor-function-transform.cjs: yazl's ByteCounter, a Transform with_flush, tar-stream's PassThrough entry stream with its owndestroy, a PassThrough without_transform, pipe chains intozlib.createGzip()andfs.createWriteStream, andfs.createReadStream(file)piped into constructor-function streams.6563-constructor-function-stream-errors.cjs: write errors with identity andcode,errored, destroy with an error and a callback, an asynchronous_destroy,ERR_METHOD_NOT_IMPLEMENTED, a throwing_read, write after end, custom events with three arguments,once/prependListener/removeListener/listenerCount, the unhandled'error'rethrow, andsetMaxListeners.Node and native agree in both lanes, including the sanitized lane's RC audit and LeakSanitizer.
Lint, format,
test:ts7 --baselines-only(baselines recorded for the four programs), the diagnostics suite, the npm stream and event cases, the runtime-pack and feature unit tests, and the focused stream, emitter, inheritance, zlib, class and util corpus (about 460 programs, plain lane; about 200 sanitized) pass apart from failures that also occur on the baseline in this environment. The self-hosting IR build passed before merging main; on current main it fails on both sides (wasm/binary.ts, SC2002), which this change does not touch.The real from2 package does not compile yet for reasons outside this change (
require("readable-stream")in an expression without the alias from Compile polyfill packages that are Node builtins as the builtins #1458, andnewon a function value), so there is no npm case.Zero cost
node scripts/zero-cost-gate.mjs --baseline=<main f4f2baf1a> --candidate=<this branch> --corpus=all: 3387 programs identical in IR, LLVM and release executables (benchmarks and corpus), 1 that fails on both sides (the self-hosted compiler, above), and 4 differences, all programs that use the untyped EventEmitter or stream constructor values from #1512. Bench geomean +1.12% with a 95% interval of [-0.77%, +1.84%], within tolerance (no bench program's code changed).5977-builtin-class-aliases.cjs(storesPassThroughin an array) and6454-constructor-function-stream-bases.cjs(util.inheritsonto the five stream classes) now read the stream classes throughdyn.builtinStreamClassand link the stream half and the stream unit: +11.0% and +24.0% binary size. Their output is unchanged.6453-constructor-function-emitter.cjsand5818-constructor-function-emitter-defaults.cjslink only the EventEmitter unit, whose methods now test for stream state: +40 bytes each.After the first full run, the legacy
Streamconstructor moved back todyn.builtinClass(it needs no stream half); a corpus rerun of the affected programs and the stream programs (97) confirmed the four differences above and nothing else.🤖 Generated with Claude Code