fix: avoid i8 overflow when computing the wider decimal precision - #24851
Open
dylanpulver wants to merge 1 commit into
Open
fix: avoid i8 overflow when computing the wider decimal precision#24851dylanpulver wants to merge 1 commit into
dylanpulver wants to merge 1 commit into
Conversation
`get_wider_decimal_type` and `get_wider_decimal_type_cross_variant`
computed the required precision `max(s1, s2) + max(p1 - s1, p2 - s2)`
using `i8` arithmetic, the type of a decimal scale.
`Decimal256` allows a precision and a scale of up to 76, so `p - s` can
reach 152 and the sum can reach 228. Neither fits in an `i8`, so comparing
two such types panicked with "attempt to add with overflow" (or "attempt
to subtract with overflow") in debug builds, and silently wrapped in
release builds.
`DECIMAL(76, 76)` is accepted by the SQL planner (see `make_decimal_type`),
so this was reachable from plain SQL, e.g.
SELECT CAST(1 AS DECIMAL(76,0)) < CAST(2 AS DECIMAL(76,52))
Compute the required precision in `i32` and saturate into `u8` instead.
The expression is non-negative for every input (`range + s >= max(p1, p2)`),
so the result is unchanged for every case that did not already overflow.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dylanpulver
force-pushed
the
fix/decimal256-coercion-precision-overflow
branch
from
September 1, 2026 15:24
9ab30e9 to
9a74736
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
SELECT CAST(1 AS DECIMAL(76,0)) < CAST(2 AS DECIMAL(76,52))panics during planning. The required precisionmax(s1, s2) + max(p1 - s1, p2 - s2)is computed ini8, the type of a decimal scale, butDecimal256allows a precision and a scale of 76, sop - sreaches 152 and the sum reaches 228. Debug builds panic; release builds wrap and return a wrong precision.What changes are included in this PR?
The expression moves into
required_decimal_precision(p1, s1, p2, s2), which computes ini32and clamps intou8. The five call sites inget_wider_decimal_typeandget_wider_decimal_type_cross_variantuse it.The change is inert on inputs that did not overflow. With
s = max(s1, s2) = s1without loss of generality,range >= p1 - s1, sorange + s >= p1 >= 0: the sum is never negative, the lower clamp never fires, and for values in0..=127the widened computation equals the old one bit for bit.What is the testing strategy for this PR?
test_decimal256_comparison_coercion_precision_overflowinbinary/tests/comparison.rs, three cases: sum 128, sum 152, and a negative scale wherep1 - s1is 152 before the sum is computed.cargo test -p datafusion-expr-common --lib: 207 passed on main, 208 with this branch. Reverting the source with the test kept panics atbinary.rs:1217:41withattempt to add with overflow. The naive fix —range.saturating_add(s)at all five sites — still fails, now atbinary.rs:1216:25withattempt to subtract with overflow, which is why the negative-scale case is in the test.End to end, same probe both ways: on main
create_logical_planfor the query above panics atbinary.rs:1217:41; with this branch it returnsProjection: CAST(Int64(1) AS Decimal256(76, 0)) < CAST(Int64(2) AS Decimal256(76, 52)) AS r.cargo fmt --all --check,cargo clippy -p datafusion-expr-common --all-targets --all-features -- -D warnings, andcargo testondatafusion-expr-common,datafusion-expr,datafusion-sql,datafusion-optimizerall pass. rustc 1.97.0, matchingrust-toolchain.toml.No
.sltcase, deliberately — see below.Are there any user-facing changes?
Two things a reviewer should weigh, both stated because I am not certain they are what you want:
This does not make the query execute. It moves the panic to
arrow-cast/src/cast/decimal.rs:190,(input_precision as i8) + delta_scale <= (output_precision as i8), which is the same bug upstream. I measured it: after this fix, planning succeeds andcollect()panics there. The two overflows are the same quantity, so no SQL can trigger DataFusion's without also triggering arrow's. That is why there is no.sltcase — one would fail CI. The arrow side needs its own fix.One release-mode behaviour change. In
get_wider_decimal_type_cross_variantwith a negative scale, the wrappedi8previously produced a smallrequired_precisionthat passed the variant checks and yielded a lossy type; it now yields 152, exceeds every maximum, and returnsNone. Negative scales cannot be written in DDL, but they can arrive from directly constructedDataTypes such as Parquet or IPC schemas. I believeNoneis correct — there is no common type that holds both — but it is a change, not a no-op, and it is the one part of this I would want a second opinion on.Not run: the workspace-wide clippy and the extended-features test run from
AGENTS.md,./dev/rust_lint.shas a whole, a release build, and benchmarks. The sqllogictest suite reports the same 39 failures with and without this change, all from an uninitialisedparquet-testingsubmodule in my checkout.Per the AI-assisted contributions policy: this patch, its test and the measurements above were produced with AI assistance (Claude Opus 5). The unknowns are the two items in the section above.