[mhk197]

Summary

An alternative design to #10502 and #10503, for comparison. It has the same features (variables, lambdas and list_transform), but a higher-order function is its own kind of node in expressions and arrays, holding its lambdas next to its children, instead of a scalar function that keeps its lambda in its options.

It's a draft to decide between the two designs, not meant to merge as is.

Changes

• Variables and lambdas, as in Add lambdas and variables for higher-order scalar functions #10502:

  • var("x") builds Expression::Variable, which displays as @x.
  • Lambda { params, body } is the written form. Lambda::bind rewrites each @x to $.x and binds the body to a struct of the parameters, giving a BoundLambda.
  • BoundLambda::evaluate runs it over parameter arrays.

• HigherOrderFnVTable, in a new higher_order_fn module:

  • a function that takes values and lambdas;
  • it binds its own lambdas against its value children's dtypes (bind_lambdas), in whatever order it needs;
  • it decides when they run (execute receives the bound lambdas).

  HigherOrderFnRef is its type-erased handle.

• Expression::HigherOrder { function, children, lambdas } and BoundExpression::HigherOrder { …, lambdas, bound }. The bound node keeps its written lambdas, so it displays @x and rebinds them when a rewrite changes its children's dtypes.

• HigherOrderArray runs a bound node. It has slice and filter pushdown, so a lambda never runs on rows that aren't selected.

• list_transform(list, lambda) now lives in higher_order_fn::fns:

  • it accepts List input only, takes one parameter, and is strict;
  • it runs over an exact ListArray: null lists are emptied, then list_from_list_view makes the elements back to back;
  • the result keeps the input's offsets and validity.

• The new variants are handled in binding, return_dtype, apply, apply_bound, display, optimize, traversal, validity (strict functions only) and serialization (an error for now). ScalarFnVTable is unchanged.

Compared with #10502 and #10503

	#10502 + #10503	This PR
Where the lambda lives	in the scalar function's options	in the node, next to its children
Non-test lines added against develop	656	1,679
list_transform itself	181 lines	160 lines
Lambdas visible to generic code	no	yes
A bound lambda displays as	(x) -> ($.x + 1i32)	(x) -> (@x + 1i32)
An element dtype changes in a rewrite	error	rebinds
Existing optimizations	all apply	only the ones re-taught

Six places assumed that a node which isn't a scalar function is the root or has no children, and compiled anyway:

• Traversal skipped the inside of higher-order nodes, so scanning a file with list_transform failed.
• The scan's field annotator marked the node as reading every field.
• apply_bound returned its input unchanged.
• Serialization wrote the node as $.
• apply panicked.
• Without slice and filter rules, a lambda that can fail errored on rows a filter had removed.

All six are fixed here.

Not done

• Dictionary pushdown for HigherOrderArray; an_infallible_transform_pushes_into_dictionary_values is ignored. The scan's own dictionary rule in vortex-layout doesn't know about the new node either.
• Reduce and simplify rules for higher-order nodes.
• Validity for functions that aren't strict.
• Serialization and a registry for the new function kind.

Testing

• cargo nextest run -p vortex-array -p vortex-layout -p vortex-file
• cargo clippy -p vortex-array -p vortex-layout --all-targets --all-features -- -D warnings
• cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-layout -p vortex-file --check
• cargo test --doc -p vortex-array -- lambda exprs::var list_transform higher_order
• RUSTDOCFLAGS="-D warnings --cfg docsrs" cargo doc --no-deps --document-private-items -p vortex-array -p vortex-layout --features vortex-array/_test-harness
• Also run locally but not included: the probes from the Add lambdas and variables for higher-order scalar functions #10502/Add list_transform #10503 review, about 1,500 lines covering file scans and 6,000 randomized list cases. They all pass after the fixes above.

[joseph-isaacs]

higher_order is a scalar?

[codspeed]

Merging this PR will regress 3 benchmarks

4 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ Unable to measure the execution of 1 benchmark

This benchmark measured no execution time, so its result is not comparable and counts as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 21 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 40 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

⚡ 6 improved benchmarks
❌ 3 regressed benchmarks
✅ 2399 untouched benchmarks
⏩ 12 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	CPU Simulation	density_sweep_dense_runs[0.001]	44.5 µs	61.8 µs	-27.9%
❌	Walltime	bitpack_blocked_compress_avx512	4.9 µs	5.6 µs	-13.02%
❌	Walltime	threshold_avx2[80]	2.6 µs	2.9 µs	-10.48%
⚡	Walltime	threshold_neon[8]	2.4 µs	1.4 µs	+71.86%
⚡	Walltime	threshold_neon[64]	4.3 µs	2.6 µs	+64.93%
⚡	Walltime	filter_neon[8]	1.7 µs	1.3 µs	+34.57%
⚡	Walltime	filter_neon[64]	2.7 µs	2.1 µs	+30.13%
⚡	Walltime	threshold_neon[80]	4.5 µs	4 µs	+12.09%
⚡	Walltime	filter_neon[80]	3.2 µs	2.9 µs	+11.51%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing mk/hof-parallel (58db9d3) with develop (dbaf5f8)}

Footnotes

1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[joseph-isaacs]

does this live in the bind method?

[mhk197]

can get rid of this?