Repository navigation
Conversation
Collect phase DAG nodes once and clear each cached column before a new block or device batch. Rebuild missing schema UPDATE common expressions with the filter mask while preserving existing selection evaluation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A CASE can leave shared transformer columns cached after one TsBlock or device batch is consumed. Reusing the DAG for the next input can then return stale values or use a column with the wrong length. This change resets the collected nodes before each input and rebuilds missing schema UPDATE common expressions only for devices that passed the filter. Re-evaluating a DIFF within the same block starts from that block's saved incoming state, preserving the previous block's contribution.
This is a narrow extraction from #18826. It keeps the existing selection evaluation and evaluation-time cache clearing mechanism.
Changes
The broader selection-cache redesign from #18826 remains deferred.
Validation
Performance
The new-input path clears a precomputed flat array in O(V) per evaluation phase, without a per-block graph walk or visited-set allocation. Schema UPDATE allocates a boolean selection mask and scans its rows only when common expressions exist. The original CASE/common recomputation cost remains.
The following exploratory JMH measurements were taken for the initial boundary-reset extraction, before the DIFF state checkpoint was added. These benchmarks contain no DIFF expressions and were not rerun for this commit. Comparison against master 673434b (JDK 17, 512 MiB/G1, 3 x 1 s warmup, 4 x 1 s measurement, two runs per version in baseline/new/new/baseline order):
All ten benchmark shapes (five scenarios x 1/1000 rows) validated every result across 16 blocks before measurement. Timings were noisy on the local machine, especially the wide filter scenario; these numbers are not evidence of a general speedup or absence of regressions.
The CASE/common allocation increase was stable: +192 bytes per one-row block and +9176 bytes per 1000-row block. A deterministic counting probe explains the extra work: shared x + 1 was computed 17 times across 16 blocks on master and 32 times here. Master partly benefits from a leftover cache whose predicate happens to remain true for these benchmark inputs; the reset removes that stale-cache reuse, while CASE still clears the result before common materialization. Expensive expressions can therefore amplify this cost.