Repository navigation
feat(expression): lane VM and typed-mask engine for batch and columna evaluation - #553
Merged
Merged
Conversation
…r evaluation Adds a register-based lane VM (core/expression/src/lane) and a typed-mask engine (core/expression/src/typed) that evaluate ZEN expressions over batches and columns, matching the stack VM's results. zen-types gains shape-backed objects and dot-path helpers; existing public APIs only grow. The lane VM has no expression size limit: deep and wide expressions are compiled and run with the same stack growth the parser uses. ZEN_EXPRESSION_CONFIG.lane (off by default) switches Isolate::run_standard and run_unary to the lane VM at runtime. Lane internals and the typed module are doc(hidden); the documented entry points are LaneProgram, LaneRunner, Session, Columns and Output. Includes an executable specification of the expression language (core/expression/tests/spec, 28,852 cases) with differential tests that run every lane path against the stack VM. zen-expression and zen-types now declare rust-version 1.88.
Shapes are freed when nothing uses them, so the 4,096-children cap per shape no longer bounds memory. Shape::with, Shape::of and Shape::without now always return a shape; objects only switch to a hash map past SPILL_AT keys, as on master.
CI runs tests in the debug profile, where frames are larger and tail recursion is not optimised. Recursive lane, typed and parser helpers now grow the stack like the rest (Const::number, the kernel builder, typed lowering and lookup, lane compile helpers, Node::member_key). The parser's closure depth counter is u32 instead of u8, so 256 nested closures no longer wrap. median([]) and empty templates no longer overflow in debug builds; release results are unchanged. The deep/wide test runs on a 256 KiB stack so unprotected recursion fails quickly in any profile. The conformance helper compares with as_str(), which is unambiguous on Windows.
…indows In x86-64 debug builds Executor::cold has a ~93 KB frame, so nested closures (execute -> cold -> body) could use more stack between two checks than the stack-growth margin and overflow. execute and cold now check the stack themselves; both run per instruction batch, not per row. The expression test data is checked out with LF on every platform. Windows runners converted it to CRLF, which broke multi-line spec cases.
ivanmiletic
approved these changes
Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a register-based lane VM (core/expression/src/lane) and a typed-mask engine (core/expression/src/typed) that evaluate ZEN expressions over batches and columns, matching the stack VM's results. zen-types gains shape-backed objects and dot-path helpers; existing public APIs only grow.
The lane VM has no expression size limit: deep and wide expressions are compiled and run with the same stack growth the parser uses.
ZEN_EXPRESSION_CONFIG.lane (off by default) switches Isolate::run_standard and run_unary to the lane VM at runtime. Lane internals and the typed module are doc(hidden); the documented entry points are LaneProgram, LaneRunner, Session, Columns and Output.
Includes an executable specification of the expression language (core/expression/tests/spec, 28,852 cases) with differential tests that run every lane path against the stack VM.
zen-expression and zen-types now declare rust-version 1.88.