feat(join): consume numeric expression MAPs in bounded join pipelines - #2086
Merged
Merged
Conversation
The bounded JOIN -> FILTER* -> MAP pipeline (#1475) accepted only projection MAPs. A MAP that computes anything, even `y + 1`, was reported as map-not-row-local, so the join was materialized in full before the MAP ran, or the step failed with ENOTSUP under WIRELOG_JOIN_BATCH_STRICT=1. The preflight now also accepts MAP expressions built only from numeric variables, numeric and boolean literals, integer and float arithmetic, and comparisons. Such an expression is a pure function of its row and reads no intern, extension or session state. Each batch is mapped by the ordinary col_op_map, so evaluation keeps that operator's semantics. String, digest, UUID, aggregate and extension expressions stay excluded, and FILTERs still admit no arithmetic. The sink used to report every FILTER or MAP failure as ENOMEM. It now records the operator's errno and returns it: ERANGE for a failed expression, as the one-shot MAP returns, and ENOSPC for a denied MAP admission. With a join output limit set, the order can differ: the pipeline counts mapped rows after each batch, so an early failing row yields ERANGE where the one-shot join, which counts join rows before the MAP runs, may stop first with EOVERFLOW. Either way the output relation is discarded and the left input is restored to the stack. Tests in test_join_pipeline.c: - The pipeline matches the one-shot oracle on a 256-row join, and the MAP never sees more than one 7-row batch. - A division by zero in a late batch returns ERANGE, as the oracle does. - A MAP failure injected on the fifth batch, the output limit, and a governor budget sweep that denies admission after committed batches each publish nothing and restore the left input. The sweep also checks that every governor reservation is released. - Stale input falls back to the one-shot path, or fails with ENOTSUP in strict mode. - An excluded digest MAP records a pipeline fallback, or fails with ENOTSUP in strict mode before it is evaluated. docs/MEMORY.md replaces the stale note that pipeline consumption "remains #1475" with the current eligibility and error contract. Fixes #1777
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1777.
The bounded
JOIN -> FILTER* -> MAPeval-stack pipeline (#1475) accepted only projection MAPs. This change makes it also consume MAPs whose expressions are numeric: integer and float arithmetic and comparisons over numeric variables and literals. These expressions are pure functions of the row, with no intern, extension or session access. Each batch still goes through the ordinarycol_op_map, so evaluation semantics are unchanged. String, digest, UUID, aggregate and extension expressions remain excluded. In those cases the step falls back withpipeline-not-consumable, or returnsENOTSUPin strict mode.FILTER and MAP failures inside the pipeline used to return
ENOMEMfor everything. They now return the operator's own errno:ERANGEfor a failed expression, matching the one-shot path, andENOSPCfor a denied admission. On failure nothing is published and the left input is restored.Tests (
tests/test_join_pipeline.c)Oracle match: the pipeline output equals the one-shot result on a 256-row join, and MAP only ever sees batches of at most 7 rows.
Late
ERANGE: a division by zero in a late batch returnsERANGE, the same as the oracle.Rollback:
None of these publish anything, all restore the left input, and the sweep checks that every reservation is released.
Stale input: the pipeline falls back to the one-shot path, or returns
ENOTSUPin strict mode.Excluded MAP: an excluded digest MAP records a fallback, or in strict mode returns
ENOTSUPwithout being evaluated.Preflight cases: admitted and excluded opcodes, stack-underflow guards (one case per guard), and FILTER still rejecting arithmetic.
Validation
meson test -C build: 432 OK, 0 failed, 14 skipped (perf and environment gates).test_join_pipelineis clean under ASAN/UBSAN and under TSan (-Dthreads=posix).Known limits (unchanged by this PR)
ERANGEwhere the one-shot path would first returnEOVERFLOW. The pipeline counts mapped rows; the one-shot path counts join rows.ENOMEM. This predates the PR, and in this shape MAP admission is always denied first.Peer review: an independent reviewer did three rounds. The first round blocked a test message that could not fail. The second blocked a check message that named the wrong source of a denial. The final tree had no blocking findings.