Skip to content

Handle sparse matrices of 2^63 or more entries in reductions, sprand and linear indexing - #1128

Merged
ViralBShah merged 4 commits into
mainfrom
vs/widelength-overflow
Oct 26, 2026
Merged

ViralBShah merged 4 commits into
mainfrom
vs/widelength-overflow

Conversation

@ViralBShah

@ViralBShah ViralBShah commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

widelength, the length that the sparse kernels compare with nnz, multiplied the dimensions as Int64. For a matrix of 2^63 entries or more it wrapped, so the reductions and sprand gave wrong results or threw, and linear indexing failed with a misleading BoundsError.

Before

julia> B = spzeros(2^62, 4);   # 2^64 entries, none stored

julia> any(iszero, B), all(isone, B), count(iszero, B; init = big(0))
(false, true, 0)

julia> sum(x -> 1.0, B)
ERROR: ArgumentError: reducing over an empty collection is not allowed; consider supplying `init` to the reducer

julia> findmax(B)
ERROR: ArgumentError: array slices must be non-empty

julia> nnz(sprand(Xoshiro(1), 2^62, 4, 1e-17))
0

julia> B[[1]] = [1.0]
ERROR: BoundsError: attempt to access 4611686018427387904×4 SparseMatrixCSC{Float64, Int64} with 0 stored entries at index [1]

After

julia> any(iszero, B), all(isone, B), count(iszero, B; init = big(0))
(true, false, 18446744073709551616)

julia> sum(x -> 1.0, B)
1.8446744073709552e19

julia> findmax(B)
(0.0, CartesianIndex(1, 1))

julia> nnz(sprand(Xoshiro(1), 2^62, 4, 1e-17))
169

julia> B[[1]] = [1.0]
ERROR: OverflowError: a 4611686018427387904×4 sparse matrix has more entries than an `Int` can index linearly; index it by row and column

widelength is now prod(widen, size(x)), an Int128 on a 64-bit build and Int64, as before, on a 32-bit one. count, the two-array mapreduce and _inorder use it in place of length or prod(size(A)), and the counters of unstored runs in the ordered reductions are widened the same way. An Int128 count would widen z * n for Complex{Int} and reach a different ^ method for an eltype that defines ^ for an Int power only, so _addzeros and _powzeros split it into Int factors and powers. For a matrix that fits an Int they make the same calls as before, and prod stays type-stable. When m*n overflows, sprand draws the rows column by column, which keeps the distribution; every size that fits draws the same entries as before.

Linear indexing into such a matrix is not supported. getindex and setindex! with a linear index, range or array, including the fixed-pattern setindex!, throw an OverflowError before writing anything; indexing by row and column works. This touches dispatch: scalar A[i] = v gets a method on AbstractSparseMatrixCSC, as Base's fallback checks the index against the wrapping length.

On a 10⁴×10⁴ matrix of density 10⁻³, main against this branch:

main branch
sum / prod / count (Float64) 10.1 / 13.7 / 11.6 μs 10.1 / 13.7 / 11.6 μs
ordered mapreduce (Float64 / Int) 101.6 / 80.5 μs 101.0 / 77.1 μs
mapreduce(*, -, A, A) (Float64 / Int) 420.8 / 135.2 μs 423.5 / 133.1 μs
findmax (Float64) 221.4 μs 221.5 μs
A[I] = x, A[I] (10⁴ indices) 448 / 152 μs 450 / 152 μs
A[i] = v, A[i] 8.2 / 6.7 ns 9.0 / 7.0 ns
sprand 580 μs 580 μs

The result types and values of 18 reductions over 10 eltypes, and the entries sprand draws in each of its forms, match main.

Left out: Base's own length(A) still wraps, and so do the checks that use it or m*n, such as isempty, Base.checkbounds and the dims reductions' nnz(A) == m*n.

🤖 Generated with Claude Code

@ViralBShah ViralBShah added backport 1.13 Change should be backported to release-1.13 backport 1.14 labels Oct 26, 2026
@codecov

codecov Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.36364% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.70%. Comparing base (4c87861) to head (0b50513).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/constructors.jl 88.23% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1128   +/-   ##
=======================================
  Coverage   94.70%   94.70%           
=======================================
  Files          16       16           
  Lines       10385    10423   +38     
=======================================
+ Hits         9835     9871   +36     
- Misses        550      552    +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ViralBShah and others added 4 commits October 26, 2026 18:39
…ries

`widelength` multiplied the dimensions as `Int64`, so for an array of 2^63
entries or more it wrapped, and the reductions, linear `setindex!` and
`sprand` that rely on it returned wrong results or threw.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d fix the remaining widelength checks

Linear getindex and setindex! on a sparse matrix of 2^63 or more entries
throw OverflowError up front instead of converting indices with divrem.
The two-array mapreduce and _inorder use widelength, prod splits an
Int128 count into Int powers so it stays type-stable, and the trim app
covers linear indexing and the two-array and ordered mapreduce.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ViralBShah
ViralBShah force-pushed the vs/widelength-overflow branch from 22fdbb3 to 0b50513 Compare October 26, 2026 22:41
@ViralBShah ViralBShah changed the title Compute widelength of a sparse array without overflow past 2^63 entries Handle sparse matrices of 2^63 or more entries in reductions, sprand and linear indexing Oct 26, 2026
@ViralBShah
ViralBShah enabled auto-merge (squash) October 26, 2026 22:43
@ViralBShah
ViralBShah merged commit 2bef1fd into main Oct 26, 2026
10 checks passed
@ViralBShah
ViralBShah deleted the vs/widelength-overflow branch October 26, 2026 22:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 1.13 Change should be backported to release-1.13 backport 1.14

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant