Skip to content

Fix nullable AND results in tree-model CASE - #18875

Open
ColinLeeo wants to merge 2 commits into
apache:masterfrom
ColinLeeo:fix/tree-case-nullable-and
Open

ColinLeeo wants to merge 2 commits into
apache:masterfrom
ColinLeeo:fix/tree-case-nullable-and

Conversation

@ColinLeeo

Copy link
Copy Markdown
Collaborator

Tree-model CASE can take the wrong branch when a later WHEN contains nullable AND. For example, when s1 is NULL and s2 is 1, CASE WHEN s1 > 100 THEN 1 WHEN s1 > 0 AND s2 > 0 THEN 2 ELSE 3 END returns 2 instead of 3.

LogicAndColumnTransformer.evaluateWithSelection() builds the correct three-valued AND result but caches the right operand. Cache the built result instead. This is a one-line production change and requires no cache-lifecycle or selection API changes.

Regression coverage includes all nine NULL/false/true operand combinations with full and partial selection masks, plus the CASE SQL example with NULL, true, false, and earlier-branch rows.

Closes #18864.

Validation:

  • Maven reactor verify from master 8effe5d6ce, selecting only LogicBinaryColumnTransformerTest (5 tests) and IoTDBCaseWhenThenIT#testNullableAndInLaterCaseBranch (1 test, rebuilt 1C1D nodes): passed.
  • Class-isolated comparison using the same unit-test runtime and master implementation: both newly added unit tests fail before the production change; all 5 pass after it.
  • Spotless, Checkstyle, and license checks: passed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No unresolved review issues remain, and validation passed.

0 open findings

What changed in this PR

Fixes nullable three-valued AND evaluation in tree-model CASE expressions by caching the computed result.

Changes:

  • Corrects LogicAndColumnTransformer caching.
  • Adds nullable AND unit coverage.
  • Adds a CASE integration regression test.
File Description
iotdb-core/​datanode/​src/​test/​java/​org/​apache/​iotdb/​db/​queryengine/​transformation/​dag/​column/​binary/​LogicBinaryColumnTransformerTest.java Tests nullable AND combinations and selection masks.
iotdb-core/​calc-commons/​src/​main/​java/​org/​apache/​iotdb/​calc/​transformation/​dag/​column/​binary/​LogicAndColumnTransformer.java Caches the computed AND result.
integration-test/​src/​test/​java/​org/​apache/​iotdb/​db/​it/​query/​IoTDBCaseWhenThenIT.java Adds SQL-level regression coverage.

🧠 Review effort: Lite


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix looks right; it now matches LogicOrColumnTransformer.evaluateWithSelection. The comments below are about test coverage.

if (!selection[i] || expected[i] == null) {
Assert.assertTrue(result.isNull(i));
} else {
Assert.assertEquals(expected[i], result.getBoolean(i));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This branch only checks getBoolean(i), and a NULL position in BooleanColumn reads as false, so the test can't tell FALSE from NULL. That leaves the "left is FALSE, so the result is FALSE" half of the fix unprotected:

  • Changing line 73 of LogicAndColumnTransformer to builder.appendNull() makes rows 3-5 (left = false) return NULL, and both new tests still pass.
  • With the pre-fix code, the test only catches row 2 (NULL AND TRUE, line 166). Rows 3-5 also came back NULL there, which the assertion on this line accepts.

The loop also reads selection after evaluateWithSelection has run, so it wouldn't notice the method mutating it: dropping the selection.clone() on line 36 likewise turns rows 3-5 into NULL, and the tests still pass.

Suggestion: keep a copy of the selection and assert non-null explicitly.

boolean[] selected = selection.clone();
transformer.evaluateWithSelection(selection);
Column result = transformer.getColumn();
Assert.assertEquals(expected.length, result.getPositionCount());
for (int i = 0; i < expected.length; i++) {
  if (!selected[i] || expected[i] == null) {
    Assert.assertTrue(result.isNull(i));
  } else {
    Assert.assertFalse(result.isNull(i));
    Assert.assertEquals(expected[i], result.getBoolean(i));
  }
}

With these assertions, the pre-fix code and both changes above fail, and the current code passes.

Minor: the partial mask i % 2 == 0 leaves rows 1 and 7 unselected, and they are the only rows that reach the "right is false" branch (lines 58-60), so that branch never runs with a partial selection. A mask that keeps row 1 or 7 selected would cover it.

"INSERT INTO root.sg.case_null(time, s1, s2) VALUES (2, 1, 1), (3, 0, 1), (4, 101, 1)"
});
resultSetEqualTest(
"SELECT CASE WHEN s1 > 100 THEN 1 WHEN s1 > 0 AND s2 > 0 THEN 2 ELSE 3 END AS result "

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the AND only appears as a WHEN condition, NULL and FALSE both mean "no match", so this query only covers the NULL AND TRUE half of the fix (time 1: 2.0 before the fix, 3.0 after). Time 3 (0 > 0 AND 1 > 0) is 3.0 whether the AND returns FALSE or NULL: with line 73 of LogicAndColumnTransformer changed to builder.appendNull(), this query still returns 3.0, 2.0, 3.0, 1.0.

A query that exposes the AND result itself would cover the FALSE half too. On the same data:

query before fix after fix
CASE WHEN s1 > 100 THEN 1 WHEN NOT (s1 > 0 AND s2 > 0) THEN 2 ELSE 3 END 3.0, 3.0, 3.0, 1.0 3.0, 3.0, 2.0, 1.0
CASE WHEN s1 > 100 THEN false WHEN s2 > 0 THEN (s1 > 0 AND s2 > 0) END true, true, null, false null, true, false, false

Both also fail with the appendNull() change (time 3 becomes 3.0 / null). I got these values by running the expressions through ColumnTransformerVisitor + FilterAndProjectOperator on this PR's code, with and without the fix.

right.addReferenceCount();
left.initFromTsBlock(block);
right.initFromTsBlock(block);
LogicAndColumnTransformer transformer = new LogicAndColumnTransformer(returnType, left, right);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bug came from LogicAndColumnTransformer and LogicOrColumnTransformer drifting apart (OR already ends evaluateWithSelection with builder.build()), but this helper hard-codes AND. So OR's evaluateWithSelection still has no NULL/selection coverage: testLogicOr only goes through evaluate(), and this is the only unit test that uses LogicOrColumnTransformer.

Could the helper take the constructor and the expected table, so it runs for both? Something like:

assertSelected(
    (l, r) -> new LogicAndColumnTransformer(returnType, l, r),
    new Boolean[] {null, false, null, false, false, false, null, false, true});
assertSelected(
    (l, r) -> new LogicOrColumnTransformer(returnType, l, r),
    new Boolean[] {null, null, true, null, false, true, true, true, true});

The OR table passes on the current code (full and partial selection), and fails if OR picks up the same initializeColumnCache(rightColumn) bug.

@ColinLeeo

Copy link
Copy Markdown
Collaborator Author

Thanks @JackieTien97! Addressed all three comments in 3316fa6: stronger NULL/selection assertions, AND/OR coverage, and both SQL regressions. All 7 unit tests and the targeted integration test pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Tree-model CASE selects a branch when nullable AND should be NULL

4 participants