Cuda 2 mpi sync - #86
Draft
max-models wants to merge 3 commits into
Draft
max-models wants to merge 3 commits into
max-models wants to merge 3 commits into
Conversation
Make feectools work when cunumpy's backend is CuPy, without any device kernels yet: kernels that stay on the host still copy their arrays. - Wrap all Pyccel kernels (stencil, B-splines, field evaluation, DOF kernels) in cunumpy.PyccelKernel, so they accept CuPy arrays. - Keep host-only metadata on NumPy: MPI/index bookkeeping in ddm (cart, partition, petsc) and fem.partitioning, Kronecker solver sizes, and index arithmetic with Python ints (compute_diag_len, math.prod). - Stage data for host-only libraries by array, not by global backend: LAPACK/SuperLU direct solvers, SciPy FFT, SciPy sparse products. - Fix calls that ran host kernels on device arrays: the second stencil2coo call in StencilMatrix.tosparse and the conjugate transpose. - Vectorize the construction of the 1D collocation matrices in the global projectors (element-wise indexing was a device round trip per entry: 334 s of a 348 s Derham setup on the GPU). - GMRES: take real scalars from CuPy views before modifying them. - Fix StencilMatrix._update_ghost_regions_serial: the ghost region is pads * shifts wide (wrong whenever shifts > 1, on both backends). - Tests: work with CuPy arrays; skip PETSc tests without petsc4py. Serial tests pass on both backends (core, ddm, fem, linalg). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Allow MPI on the CuPy backend and make it correct with device buffers (requires a CUDA-aware MPI library). - feectools.ddm.mpi no longer disables MPI when ARRAY_BACKEND=cupy; the segfaults it guarded against come from MPI libraries that are not CUDA-aware. - Call cunumpy.synchronize_for_mpi before every MPI call on device buffers: CuPy kernels run asynchronously and MPI does not know about CUDA streams, so a buffer still being written would be sent silently wrong. Covers the blocking, non-blocking and interface data exchangers, the Allreduce in StencilVectorSpace.inner and the Alltoallv calls of the parallel Kronecker solver. Requires cunumpy >= 0.3.0. - Fix CuPy incompatibilities reached only by the MPI tests: xp.dot/vdot on .flat iterators in StencilInterfaceMatrix._dot and the pure-Python inner product, and test_cart_1d assigning Python lists to CuPy arrays. - Add test_mpi_device.py: distributed results against global references, and that the exchangers synchronize before MPI. With 2 MPI ranks and a CUDA-aware Open MPI, all MPI tests in ddm and linalg pass on both backends; serial tests are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 1, 2026
Merged
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MPI is allowed on the CuPy backend and gives correct results with device buffers. This requires a CUDA-aware MPI library.
Changes
feectools.ddm.mpino longer turns MPI off whenARRAY_BACKEND=cupy. The segfaults that check was guarding against come from MPI libraries that are not CUDA-aware.cunumpy.synchronize_for_mpiis called before every MPI call that uses device buffers. CuPy launches kernels asynchronously and MPI doesn't know about CUDA streams, so a buffer that is still being written would be sent with wrong data and no error. The call is added in:AllreduceinStencilVectorSpace.innerAlltoallvcalls of the parallel Kronecker solvercunumpy>=0.3.0, which providessynchronize_for_mpi.xp.dot/xp.vdoton.flatiterators inStencilInterfaceMatrix._dotand in the pure-Python inner producttest_cart_1dassigning Python lists to CuPy arrayslinalg/tests/test_mpi_device.py: compares distributed results against global references, and checks that the exchangers synchronize before calling MPI.Testing
With 2 MPI ranks and a CUDA-aware Open MPI, all MPI tests in
ddmandlinalgpass on both backends. Serial tests are unchanged.Stack
This is step 2 of 4 for CUDA support. The PRs are stacked, and each one targets
devel-tiny, so the diff here also contains the earlier steps. Review only commitccb39f8in this PR, and merge them in order:🤖 Generated with Claude Code