Skip to content

feat(join): consume numeric expression MAPs in bounded join pipelines - #2086

Merged
justinjoy merged 1 commit into
mainfrom
claude/1777-expr-map
Oct 5, 2026
Merged

justinjoy merged 1 commit into
mainfrom
claude/1777-expr-map

Conversation

@justinjoy

Copy link
Copy Markdown
Collaborator

Fixes #1777.

The bounded JOIN -> FILTER* -> MAP eval-stack pipeline (#1475) accepted only projection MAPs. This change makes it also consume MAPs whose expressions are numeric: integer and float arithmetic and comparisons over numeric variables and literals. These expressions are pure functions of the row, with no intern, extension or session access. Each batch still goes through the ordinary col_op_map, so evaluation semantics are unchanged. String, digest, UUID, aggregate and extension expressions remain excluded. In those cases the step falls back with pipeline-not-consumable, or returns ENOTSUP in strict mode.

FILTER and MAP failures inside the pipeline used to return ENOMEM for everything. They now return the operator's own errno: ERANGE for a failed expression, matching the one-shot path, and ENOSPC for a denied admission. On failure nothing is published and the left input is restored.

Tests (tests/test_join_pipeline.c)

  • Oracle match: the pipeline output equals the one-shot result on a 256-row join, and MAP only ever sees batches of at most 7 rows.

  • Late ERANGE: a division by zero in a late batch returns ERANGE, the same as the oracle.

  • Rollback:

    • an allocation failure injected on the fifth MAP batch;
    • the join output limit;
    • a governor budget sweep that denies admission after earlier batches were committed.

    None of these publish anything, all restore the left input, and the sweep checks that every reservation is released.

  • Stale input: the pipeline falls back to the one-shot path, or returns ENOTSUP in strict mode.

  • Excluded MAP: an excluded digest MAP records a fallback, or in strict mode returns ENOTSUP without being evaluated.

  • Preflight cases: admitted and excluded opcodes, stack-underflow guards (one case per guard), and FILTER still rejecting arithmetic.

Validation

  • meson test -C build: 432 OK, 0 failed, 14 skipped (perf and environment gates).
  • test_join_pipeline is clean under ASAN/UBSAN and under TSan (-Dthreads=posix).
  • Mutation checks: removing errno propagation, the unary or binary depth guards, the MAP classifier, or the arithmetic gate each fails a test. Admitting digests or arithmetic in FILTER also fails a test.
  • uncrustify and clang-tidy are clean on the touched files.

Known limits (unchanged by this PR)

  • With a join output limit set, an early failing row can return ERANGE where the one-shot path would first return EOVERFLOW. The pipeline counts mapped rows; the one-shot path counts join rows.
  • A denied growth of the pipeline's output relation still maps to ENOMEM. This predates the PR, and in this shape MAP admission is always denied first.

Peer review: an independent reviewer did three rounds. The first round blocked a test message that could not fail. The second blocked a check message that named the wrong source of a denial. The final tree had no blocking findings.

The bounded JOIN -> FILTER* -> MAP pipeline (#1475) accepted only
projection MAPs. A MAP that computes anything, even `y + 1`, was
reported as map-not-row-local, so the join was materialized in full
before the MAP ran, or the step failed with ENOTSUP under
WIRELOG_JOIN_BATCH_STRICT=1.

The preflight now also accepts MAP expressions built only from numeric
variables, numeric and boolean literals, integer and float arithmetic,
and comparisons. Such an expression is a pure function of its row and
reads no intern, extension or session state. Each batch is mapped by
the ordinary col_op_map, so evaluation keeps that operator's semantics.
String, digest, UUID, aggregate and extension expressions stay excluded,
and FILTERs still admit no arithmetic.

The sink used to report every FILTER or MAP failure as ENOMEM. It now
records the operator's errno and returns it: ERANGE for a failed
expression, as the one-shot MAP returns, and ENOSPC for a denied MAP
admission. With a join output limit set, the order can differ: the
pipeline counts mapped rows after each batch, so an early failing row
yields ERANGE where the one-shot join, which counts join rows before the
MAP runs, may stop first with EOVERFLOW. Either way the output relation
is discarded and the left input is restored to the stack.

Tests in test_join_pipeline.c:
- The pipeline matches the one-shot oracle on a 256-row join, and the
  MAP never sees more than one 7-row batch.
- A division by zero in a late batch returns ERANGE, as the oracle
  does.
- A MAP failure injected on the fifth batch, the output limit, and a
  governor budget sweep that denies admission after committed batches
  each publish nothing and restore the left input. The sweep also
  checks that every governor reservation is released.
- Stale input falls back to the one-shot path, or fails with ENOTSUP
  in strict mode.
- An excluded digest MAP records a pipeline fallback, or fails with
  ENOTSUP in strict mode before it is evaluated.

docs/MEMORY.md replaces the stale note that pipeline consumption
"remains #1475" with the current eligibility and error contract.

Fixes #1777
@justinjoy
justinjoy merged commit b18b034 into main Oct 5, 2026
29 checks passed
@justinjoy
justinjoy deleted the claude/1777-expr-map branch October 5, 2026 09:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consume expression MAPs in bounded join pipelines

1 participant