Repository navigation
CUDA version of the vlasov_maxwell accumulation - #713
Draft
max-models wants to merge 9 commits into
Draft
max-models wants to merge 9 commits into
max-models wants to merge 9 commits into
Conversation
- linear_vlasov_ampere_cuda.cu: same arguments as pyccel, one thread per marker row, matrix blocks as Array6D<double> views (cunumpy 0.6.1) - device fill_mat, fill_mat_vec (filler_kernels.cuh), m_v_fill_b_v1_symm (particle_to_mat_kernels.cuh) and outer (linalg_kernels.cuh), with atomic adds - parity cases (Cuboid, Colella, HollowTorus, 3d spline) and a device-helper test of m_v_fill_b_v1_symm, on a GPU and by CPU emulation - test_accum_matrix_cupy.py: the matrix path of Accumulator on CuPy vs NumPy (device data, ghost regions, transposed blocks, Schur operator and solve), without a GPU on the fake CuPy with emulated kernel launches (emulated_launches() in pic/tests/cuda_emulation.py) - CUDA_STRATEGY.md: porting order and implementation notes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the CuPy backend, Particles._sendrecv_markers hands the marker rows to Isend/Irecv as device buffers through cunumpy.mpi.mpi_buffer (directly after synchronize_for_mpi with CUDA-aware MPI, staged through host memory otherwise). Rows from each rank arrive in a contiguous slice of one device receive buffer, and one device scatter fills the holes. The per-rank counts are small NumPy arrays on every backend (they are host integers already), and a uniform alpha is a Python scalar, so the exchange makes no host/device copy and no implicit host read of a device array. The NumPy path is unchanged. Adds pic/tests/test_sorting_device.py (fake CuPy under mpirun, and a GPU variant) and a CI step running it on the fake CuPy. Solves #698, part of #650. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the accumulation kernel of VlasovAmpereCoupling (VlasovAmpereOneSpecies, VlasovMaxwellOneSpecies) to CUDA, 1:1 with pyccel: Array6D<double> matrix blocks, atomic adds, holes skipped and boundary particles accumulated as in pyccel. The device helpers fill_mat, fill_mat_vec and m_v_fill_b_v1_symm, v1_symm_accumulation_data() and the m_v_fill_b_v1_symm wrapper tests are byte-identical to the linear_vlasov_ampere PR (#705). Parity cases in Cuboid, Colella, HollowTorus, ShafranovDshapedCylinder and a 3d spline mapping; the CPU emulation agrees with pyccel and fails for a transposed DF. Part of #688. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 7, 2026
The hole in marker_arguments() was only marked in column 8 (first_init_idx), which the pushers test. Accumulation kernels test column 0, so the linear_vlasov_ampere parity case had no real hole. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stack the PR on #711. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stack the PR on #712. Conflict in CUDA_STRATEGY.md: kept both sections (marker exchange notes, then linear_vlasov_ampere notes). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stack the PR on #705. Conflicts: CUDA_STRATEGY.md (kept both PRs' notes; vlasov_maxwell row stays done), cuda_parity_cases.py and test_cuda_parity.py (kept the linear_vlasov_ampere and vlasov_maxwell cases and imports). Semantic fix for #711 below: vlasov_maxwell has a CUDA version now, so test_compile_cuda_kernels expects VlasovAmpereOneSpecies to have no missing CUDA kernels and checks fail-fast with push_bxu_Hdiv instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
max-models
added a commit
that referenced
this pull request
Oct 8, 2026
Stack the PR on #723. Conflicts: eval_spline_mpi_{markers,matrix,sparse_meshgrid}_cuda.cu (SplineArgs of #721 with the qualified evaluation_kernels_3d::eval_spline_mpi), linalg_kernels.cuh, filler_kernels.cuh, particle_to_mat_kernels.cuh (helpers of #705/#713 moved into their module namespaces). Semantic fixes: the struphy_cuda::<pyccel module> rule applied to the CUDA code of the PRs below: linear_vlasov_ampere (#705), vlasov_maxwell (#713), bstar_parallel_3form (#718) and eval_spline_mpi_tensor_product_fixed (#724) use 'using namespace struphy_cuda;' and qualified helper calls; m_v_fill_b_v1_symm calls filler_kernels:: and evaluation_kernels_3d::; the fill_v1_symm test wrapper calls particle_to_mat_kernels::m_v_fill_b_v1_symm. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 8, 2026
Draft
max-models
added this pull request to stack #728
October 8, 2026 05:45
max-models
removed this pull request from stack #728
October 8, 2026 06:00
max-models
added this pull request to stack #729
October 8, 2026 06:00
max-models
marked this pull request as draft
October 8, 2026 09:14
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack: part 6 of 14, based on #705 (merge that first), next: #714. Full order: #709 → #708 → #711 → #712 → #705 → #713 → #714 → #718 → #720 → #721 → #724 → #725 → #723 → #722.
Stack fix:
simulation/tests/test_compile_cuda_kernels.py(from #711) now expectsVlasovAmpereOneSpeciesto have no kernel without a CUDA version (vlasov_maxwellhas one now). The fail-fast check usespush_bxu_Hdivinstead. TheCUDA_STRATEGY.mdexample was updated to match. Thelinear_vlasov_ampere(#705) andvlasov_maxwellparity cases and imports are both kept.Solves the following issue(s):
Part of #688 (blocked matrix accumulations), tracked in #650. Ports
vlasov_maxwellto CUDA. It is the accumulation ofVlasovAmpereCouplinginVlasovAmpereOneSpeciesandVlasovMaxwellOneSpecies(#3 in the porting order ofCUDA_STRATEGY.md). It uses theArray6D<double>views of cunumpy 0.6.1.This PR is based on
develand does not depend on #705 (linear_vlasov_ampere). The two PRs need the same device helpers. Here they are copied byte-identically from #705 (origin/cuda-linear-vlasov-ampereat 1ad4cd6), sofiller_kernels.cuh,particle_to_mat_kernels.cuh,kernel_test_args.py,test_device_helpers.pyandtest_cuda_emulation.pymerge without conflicts. Whichever PR merges second gets small conflicts in three files:cuda_parity_cases.py,test_cuda_parity.pyandCUDA_STRATEGY.md. In all three, both sides add lines in the same place, so keep both (checked withgit merge-tree).Core changes:
CUDA kernel
pic/accumulation/kernels/vlasov_maxwell/vlasov_maxwell_cuda.cu:df,matrix_inv,transpose,matrix_matrixandmatrix_vector. Thenm_v_fill_b_v1_symmaddsA_p = w_p DF^{-1} DF^{-T}andB_p = w_p DF^{-1} v_p.mat11 … mat33areArray6D<double>(any strides) and the vectors areArray3D<double>.markers[ip, -1] == -2) are accumulated, unlike inlinear_vlasov_ampere.Device helpers, 1:1 with pyccel, byte-identical to CUDA version of the linear_vlasov_ampere accumulation #705:
fill_matandfill_mat_vecinfiller_kernels.cuhm_v_fill_b_v1_symminparticle_to_mat_kernels.cuhEvery addition into matrix or vector data is a
cunumpy_atomic_add.outerfrom CUDA version of the linear_vlasov_ampere accumulation #705 is not needed here and is not copied.Tests:
PARITY_CASES["vlasov_maxwell"]covers 129 markers in five mappings: Cuboid, Colella, HollowTorus, ShafranovDshapedCylinder and a 3d spline mapping (geometry_domain0, 2, 5, 9, 12).markers[0, 0] = -1) and row 1 is a boundary particle.v1_symm_accumulation_data()from CUDA version of the linear_vlasov_ampere accumulation #705).G^{-1}.rtol=1e-12,atol=1e-8, for the atomic summation order on the GPU.atolis scaled up fromlinear_vlasov_ampere's 1e-10 because the entries are about 60 times larger.DF, for skipped boundary particles and for holes that are not skipped.vlasov_maxwellis added totest_vlasov_kernel_coverage.test_v1_symm_filler(GPU) andtest_emulated_v1_symm_fillercomparem_v_fill_b_v1_symmwith pyccel.Not in this PR: CUDA version of the linear_vlasov_ampere accumulation #705's
test_accum_matrix_cupy.pyandemulated_launches(). They test the matrix path ofAccumulatoron CuPy and are not copied, to keep this PR small. I ran an ad-hoc copy of that test withvlasov_maxwellin place oflinear_vlasov_ampere, not committed. It runs on the fake CuPy with emulated launches and matches NumPy: the nine kernel arrays, the dense blocks, the vector,BC.dot(x)and three Schur CG iterations, with no host copies from the accumulation toBC.dot(x). After both PRs are merged, that test could get avlasov_maxwellcase.Model-specific changes:
None. Status of
VlasovAmpereOneSpeciesandVlasovMaxwellOneSpecieson the GPU:PushEtapush_eta_stage,reflectPushVxB(analytic and implicit)push_vxb_analytic,push_vxb_implicitVlasovAmpereCouplingvlasov_maxwellpush_v_with_efieldPoissonSolvecharge_density_0formupdate_weights,Maxwellianonxp,jacobian_detthrough the geometry kernels)MaxwellWeakAmpere(VlasovMaxwell only)curl,M1,M2, Schur solve)KineticEnergyPIC(xp.sum)BilinearEnergyFEECStencilVector.inner, so a host scalargauss_errordiagnostictoarray(), a host copyStill missing for an end-to-end GPU run:
MassMatrixPreconditioner, the default ofVlasovAmpereCoupling,MaxwellWeakAmpereandPoissonSolve, cannot be created on CuPy (is_circulantasserts a host array). The device Kronecker solve is a separate PR.StencilVector.innerreturns a host scalar, so each CG iteration and the FEEC energy scalars copy a scalar from the device.Documentation changes:
CUDA_STRATEGY.md:vlasov_maxwellis marked ✓.vlasov_maxwellimplementation notes.The checklist and the "6D views are a gate" paragraph are left to #705, to keep the conflict small.
Testing (macOS, no GPU; pyccel kernels compiled with GNU/Fortran; GPU tests not run):
devel)test_kernel_backends.py test_cuda_parity.py test_device_helpers.py test_cuda_emulation.pympirun -n 2:test_verif_VlasovMaxwellOneSpecies.py test_verif_VlasovAmpereOneSpecies.py test_accum_vec_H1.pytest_emulated_parity[vlasov_maxwell]andtest_emulated_v1_symm_filler.vlasov_maxwellparity cases andtest_v1_symm_filler.🤖 Generated with Claude Code