Repository navigation
MassMatrixPreconditioner on the CuPy backend - #717
Draft
max-models wants to merge 1 commit into
Draft
max-models wants to merge 1 commit into
max-models wants to merge 1 commit into
Conversation
The mass-matrix preconditioners could not be created on the CuPy backend: the assembled 1d mass matrices come back from StencilMatrix.toarray() as host (NumPy) arrays, and is_circulant/FFTSolver asserted xp.ndarray (cupy.ndarray on CuPy). The stencil-matrix fill also mixed xp.nonzero with host arrays. The 1d matrices and solvers are now host setup data on every backend (what KroneckerLinearSolver expects, and what feectools#96 builds its device inverses from); only the process-local 1d stencil matrices of the KroneckerStencilMatrix are copied to the device once. The duplicated 1d setup of both preconditioners moves into _solver_and_local_matrix_1d. FFTSolver accepts host or device matrices and solves device right-hand sides on the host. The NumPy path computes the same as before. Adds a fake-CuPy test (subprocess, CUDA launches emulated on the CPU) comparing both preconditioners for M0/M1/M2 (periodic and clamped) with NumPy, and the same check on a GPU. Part of #689 and #650. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Solves the following issue(s):
Part of #689 (
LinearVlasovAmpereOneSpecieson the GPU), tracked in #650.EfieldWeightsCouplingusesMassMatrixPreconditionerby default, and the preconditioner could not even be created on the CuPy backend.The failure (reproduced with cunumpy's fake CuPy, CUDA launches emulated on the CPU):
StencilMatrix.toarray()returns a host (NumPy) array on every backend, because feectools builds it through a scipy COO matrix.is_circulantandFFTSolver.__init__assertedxp.ndarray, which iscupy.ndarrayon CuPy. The next steps would have failed as well:xp.nonzero(M_arr)on a host array, and per-entry writes of host scalars into the device_dataof the local stencil matrix.Core changes:
FFTSolverfor circulant matrices,SparseSolverotherwise).KroneckerLinearSolverexpects host 1d solvers. feectools#96 builds its dense device inverses from these host solvers (M = solver.solve(I)on host arrays), so this matches the device Kronecker solve as well.KroneckerStencilMatrix. Each one is filled in a host array and copied toM_local._datain one step._solver_and_local_matrix_1d. The 1d setup (dense matrix, solver, local stencil matrix, consistency check) was duplicated inMassMatrixPreconditionerandMassMatrixDiagonalPreconditioner. Both now call this helper, which uses NumPy explicitly. Both classes had the same problem, and both are fixed.is_circulantaccepts host or device matrices and checks on the host. It returns a Pythonbool.FFTSolverkeeps its circulant column on the host and accepts a host or device matrix.solvewith device right-hand sides (orout) solves on the host and copies the result back. This is whatBandedSolver/SparseSolverdo in the current feectools, and it stays until the device Kronecker solve of feectools#96 is in. With Only run the unit tests which touch changed code in the ci #96,FFTSolveris only called on host arrays (to build the dense inverse), so it needs no change there.except xp.linalg.LinAlgErroris nownp.linalg.LinAlgError, which is what scipy raises.M_localis no longer copied a second time (M_local.copy()), because it is a fresh object.MassMatrixDiagonalPreconditioner,FFTSolver) are covered above. The module has no other preconditioners.Tests: new
feec/tests/test_preconditioner_cupy.py.check_preconditioners_on_cupybuildsMassMatrixPreconditionerandMassMatrixDiagonalPreconditionerfor M0, M1 and M2 (Colella, 6x5x4 elements, degrees (2, 3, 1)) on both backends. It covers periodic bcs and clamped bcs (Dirichlet in eta1/eta3, so the boundary-operator path and bothFFTSolverandSparseSolverare used).dotanddot(out=...), checks that the results live on the device, and compares with NumPy atrtol=1e-12.CudaKernellaunch (feectools'stencil_transpose_1d, struphy'skernel_evaluate) is emulated on the CPU through a small private_emulated_launches(). It is a minimal version ofemulated_launches()from CUDA version of the linear_vlasov_ampere accumulation #705 and can be replaced by it once CUDA version of the linear_vlasov_ampere accumulation #705 is merged.Model-specific changes:
None. Still missing for #689: the device Kronecker solve (feectools#96 / #703, so that applying the preconditioner makes no host copies), CG inner products returning host scalars, and an H100 run.
Documentation changes:
CUDA_STRATEGY.md: a short note in the feectools section on where the preconditioner data live and how they relate to feectools#96.Testing (macOS, no GPU; pyccel kernels compiled with GNU/Fortran; GPU tests not run):
develpreconditioner.py)test_preconditioner_cupy.py(fake CuPy)AssertionErrorinis_circulant), 2 skipped (GPU)test_preconditioner_transpose.pytest_mass_matrices.py::test_mass_preconditioner,::test_mass_preconditioner_array_weights_mpi,test_poisson.py::test_poisson_1d,test_curl_curl.py::test_convergence_1d,test_gyrokinetic_poisson.py::test_poisson_M1perp_1d,test_saddlepoint_massmatrices.py::test_saddlepointsolver_uzawa_smallNot run locally:
test_mass_preconditioner_polar(more than 30 min per case here, with the venv's uncompiled feectools kernels) and the MPI runs. CI covers them.🤖 Generated with Claude Code