Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,17 @@ All notable changes to wirelog are documented in this file.

### Changed

- **Bounded join pipelines consume expression MAPs** (#1777): with
`WIRELOG_JOIN_BATCH_BYTES` set, a `JOIN -> FILTER* -> MAP` pipeline
whose MAP computes integer or float arithmetic or comparisons is now
evaluated batch by batch instead of materializing the whole join first;
previously only projection MAPs qualified. String, digest, UUID,
aggregate and extension expressions still fall back to the one-shot
join (`ENOTSUP` under `WIRELOG_JOIN_BATCH_STRICT=1`). A FILTER or MAP
failure inside the pipeline now returns that operator's errno -- for
MAP, `ERANGE` for an expression error and `ENOSPC` for a denied
admission -- where every such failure previously became `ENOMEM`.

- **Bounded join scratch is sized lazily** (#1481):
`col_join_batch_producer_create` reserved `WIRELOG_JOIN_BATCH_BYTES /
row_bytes` rows of governed scratch at create, whatever the join's
Expand Down
18 changes: 16 additions & 2 deletions docs/MEMORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -1219,8 +1219,22 @@ join; the session records the fallback count and last reason and logs one
`JOIN` warning per reason. `WIRELOG_JOIN_BATCH_STRICT=1` (ignored unless
the bytes knob is set) turns a fallback into an `ENOTSUP` failure, which is
the explicit bounded-mode result for unsupported shapes. Differential
keyed joins have the separate producer described below; pipeline consumption
of batches on the eval stack (JOIN -> FILTER* -> MAP) remains #1475.
keyed joins have the separate producer described below.

A non-differential eligible join followed by `FILTER* -> MAP` is consumed
batch by batch on the eval stack (#1475), so the full join intermediate is
not materialized while the pipeline runs: each batch is filtered and mapped
into one output relation, which is published only after the last batch. The
FILTERs may use only numeric variables, numeric and boolean literals, and
comparisons. The MAP may project columns or evaluate numeric expressions --
integer and float arithmetic and comparisons (#1777). A MAP with a string,
digest, UUID, aggregate or extension expression is not consumed; those have
no batch contract yet. A JOIN whose downstream operators are not
consumable, or whose input goes stale mid-stream, falls back to the one-shot
join with reason `pipeline-not-consumable` (`ENOTSUP` under
`WIRELOG_JOIN_BATCH_STRICT=1`). A FILTER or MAP failure inside the pipeline
returns that operator's errno -- for MAP, `ERANGE` for a failed expression
and `ENOSPC` for a denied admission -- and publishes no rows.

Ownership and admission:

Expand Down
1 change: 1 addition & 0 deletions tests/meson.build
Original file line number Diff line number Diff line change
Expand Up @@ -2069,6 +2069,7 @@ test_join_pipeline_exe = executable(
'test_join_pipeline',
files('test_join_pipeline.c'),
include_directories: [wirelog_inc, wirelog_src_inc],
c_args: ['-DWL_SESSION_TEST_HOOKS=1'],
dependencies: [nanoarrow_dep, threads_dep, xxhash_dep, mbedtls_dep, math_dep],
link_with: [testlib_prod],
)
Expand Down
Loading
Loading