Repository navigation
Conversation
Decline FLOAT and DOUBLE range comparisons and zero or NaN literals. Spark treats -0.0 as equal to 0.0 and sorts every NaN above all numbers, while iceberg-rust compares by IEEE 754 total order, and its row-group pruning trusts statistics that can omit NaN (apache/iceberg-rust#3494). Decline equality under NOT on nullable columns. Iceberg passes `<=>` as `=`, so the native reader dropped NULL rows that Spark keeps for `NOT (id <=> 1)`. On columns whose type changed in the schema history, push only equality outside NOT and null checks, since iceberg-rust compares older, narrower files incorrectly (apache/iceberg-rust#3122). An incomplete history treats every BIGINT, DOUBLE and DECIMAL column as promoted. Closes apache#7136. Closes apache#7137. Part of apache#7138.
github-actions
Bot
added
bug
Spark rewrites `b = false` on a nullable boolean as `b IS NOT NULL AND NOT b`, which Iceberg passes as `not_null(b) AND not(b = true)`. Declining equality under NOT on a nullable column left that residual unpushed, so CometFuzzIcebergSuite found no residual for its boolean column. A NULL value fails a NOT_NULL conjunct, and with it the whole AND, in Spark and in the native reader alike. Treat a column checked that way as non-nullable within the AND, so the null-safe concern no longer applies.
…cans Extract referencedColumn so the residual converter and notNullColumns share the bare-reference check. Share the residual predicate loop of the new native tests, and assert that `id != 1` and `b = false` are still pushed while `NOT (id <=> 1)` and `b IS NOT TRUE` are not. CometFuzzIcebergSuite samples its equality literal from the data. Skip zero as well as NaN, since floating-point comparisons with zero are no longer pushed. Note in the Iceberg guide that `!=` is not affected by the negated-equality rule.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #7136.
Closes #7137.
Part of #7138.
Rationale for this change
The native Iceberg scan pushes each task's residual into iceberg-rust, which uses it to prune row groups and pages and to filter rows during decode. The filter above the scan cannot restore rows dropped there. On
mainandbranch-1.1, three kinds of residual drop rows that Spark returns:FLOAT/DOUBLEpredicates: Spark treats-0.0as equal to0.0and sorts every NaN above all numbers. iceberg-rust compares by IEEE 754 total order and prunes on statistics that can omit NaN (Row-group pruning skips<and<=matches when the min statistic is NaN iceberg-rust#3494).WHERE f = 0.0drops-0.0rows.NOT (id <=> 1): Iceberg passes<=>with a non-null literal asequal, and the native reader drops NULL rows that Spark keeps.INTtoBIGINT: older files keep the narrower type, which iceberg-rust compares incorrectly (bug(arrow): filters lose rows after numeric schema promotion iceberg-rust#3122).These were found while reviewing #6847 and split out so the fixes can be backported.
What changes are included in this PR?
The residual serde leaves these predicates to the filter above the scan:
FLOAT/DOUBLEcolumns: range comparisons, comparisons with0.0,-0.0or NaN, andFLOATliterals that do not narrow exactly. Equality,!=andINwith other literals are still pushed.NOTon a nullable column, unless an enclosingANDchecks that column withIS NOT NULL. Spark infers that check forb = false, which it rewrites toNOT b, so that filter is still pushed.!=reaches Iceberg as a separate comparison and is not affected.=outsideNOT. If snapshot expiration removed older schemas, everyBIGINT,DOUBLEandDECIMALcolumn counts as promoted.This removes some pushdown that
mainhas today. Files imported byadd_files,migrateorsnapshot, and promotions made before Iceberg 0.12, leave no schema history, so they remain a known issue under #7138. The Iceberg guide lists the excluded predicates and that known issue.How are these changes tested?
Serde tests cover each excluded form, including under
AND,ORand nestedNOT, and the schema-history detection. Native tests compare with Spark for signed zeros and NaN, negated<=>, and promoted columns before and after snapshot expiration. They also assert which negated equalities are still pushed. A Rust test covers the bind-failure fallback, which the float tests no longer reach. The comparison and logical pushdown fixtures move fromDOUBLEtoBIGINTso their predicates are still pushed.Local, Spark 4.1 / JDK 21:
CometIcebergNativeScanSuite,CometIcebergNativeSuiteandCometFuzzIcebergSuitepass. Iceberg's Spark SQL tests (Iceberg 1.8 to 1.11) and the Comet suites on the other Spark profiles have not run yet. They need therun-iceberg-testsandrun-all-spark-profileslabels.