[Bug] CASE cache invalidation evaluates shared DIFF twice and returns incorrect values #18863

Status

Implemented in PR #18826, in the cache ownership and DIFF fix. This issue duplicates work already tracked in that PR and will not be tracked separately. It is being closed as redundant. The PR is still open, so this does not imply that the fix has already landed on master.

Selected columns are now returned directly rather than entering the shared full-column cache. CASE no longer invalidates its descendants during evaluation. DIFF retains one full result per input block, independently of the ordinary cache's remaining references, and releases that result at the next input while preserving cross-block value/NULL state.

testStatefulDiffIsNotReevaluatedAfterCaseFilter is enabled, with no @Ignore, and passes. The same test fails against the pre-fix classes with expected:<4.0> but was:<-1.0>. The SQL regression returns 2, 3, 4; it also covers the always-true ELSE variant preserving the initial NULL and two CASE consumers plus a full DIFF output. Additional unit coverage checks both ignoreNull settings, all-false masks, cache exhaustion, consecutive blocks, and consumer order.

Original reproduction (before the fix)

Description

CASE can clear a shared stateful DIFF result during evaluation. Materializing the same expression for projection then evaluates the same input block again, advancing DIFF state twice and returning incorrect values. Earlier revisions of PR #18826 fixed stale caches across blocks and sibling-column problems but did not yet fix this within-block state transition.

Reproduction

CREATE TABLE dt2 (dev STRING TAG, x DOUBLE FIELD);
INSERT INTO dt2(time, dev, x) VALUES (1, 'd0', 1.0), (2, 'd0', 3.0), (3, 'd0', 6.0);
FLUSH;
INSERT INTO dt2(time, dev, x) VALUES (4, 'd0', 10.0), (5, 'd0', 11.0);
FLUSH;
SELECT time, x, diff(x) AS d FROM dt2
WHERE CASE WHEN diff(x) > 1.0 THEN 1.0 ELSE 0.0 END > 0.0;

The SQL reproduction in the review used a 1C1D instance with max_tsblock_line_number=1. The operator-level reproduction below controls block boundaries directly.

Expected d: 2.0, 3.0, 4.0 at times 2, 3, 4. The review reported a column-size error on merge base fbd5fd22a6, while the cross-block fix changes that case to a silent wrong value of -1.0 at time 4. See the SQL review reproduction.

A deterministic actual-operator reproduction has been confirmed on d7239384fd: feed [1, 3, 6] and [10, 11] as two blocks, share one DiffFunctionColumnTransformer between the CASE filter's first WHEN and the common-expression projection, and assert [2, 3] then [4]. The second assertion fails with expected:<4.0> but was:<-1.0>.

Mechanism

DiffFunctionColumnTransformer.evaluateWithSelection() evaluates every row because DIFF needs row continuity. The first evaluation of [10, 11] advances lastValue to 11. CASE cleanup invalidates that column. Re-evaluation begins with 10 - 11, yielding -1 instead of 4. A filter mask cannot roll back internal state, and resetting column caches at block boundaries cannot repair repeated evaluation within a block.

Selection-mode results currently enter the same ref-counted ColumnCache as full results, forcing evaluation-time invalidation and recomputation. Follow-up design work should separate selection-local results (for example through direct returned columns or a selection-local map) from reusable full columns, and explicitly ensure stateful expressions advance once per input block. Preserve CASE short-circuit guards and multi-consumer sharing while doing so.

Acceptance criteria

• The two-block reproduction returns [2, 3] then [4].
• An always-passing ELSE variant preserves the initial NULL and does not rewind DIFF state.
• Tests cover shared consumers, consecutive blocks, and selection-mode evaluation.
• A wrong numerical result is never treated as the expected behavior of a passing regression test.

Regression test

The original ignored reproducer has been enabled by the fix described above. It asserts the correct result and must remain enabled. See the current PR implementation and validation for execution details.