Skip to content

CUDA version of the vlasov_maxwell accumulation - #713

Draft
max-models wants to merge 9 commits into
cuda-linear-vlasov-amperefrom
cuda-vlasov-maxwell
Draft

max-models wants to merge 9 commits into
cuda-linear-vlasov-amperefrom
cuda-vlasov-maxwell

Conversation

@max-models

@max-models max-models commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Stack: part 6 of 14, based on #705 (merge that first), next: #714. Full order: #709 → #708 → #711 → #712 → #705 → #713 → #714 → #718 → #720 → #721 → #724 → #725 → #723 → #722.

Stack fix: simulation/tests/test_compile_cuda_kernels.py (from #711) now expects VlasovAmpereOneSpecies to have no kernel without a CUDA version (vlasov_maxwell has one now). The fail-fast check uses push_bxu_Hdiv instead. The CUDA_STRATEGY.md example was updated to match. The linear_vlasov_ampere (#705) and vlasov_maxwell parity cases and imports are both kept.


Solves the following issue(s):

Part of #688 (blocked matrix accumulations), tracked in #650. Ports vlasov_maxwell to CUDA. It is the accumulation of VlasovAmpereCoupling in VlasovAmpereOneSpecies and VlasovMaxwellOneSpecies (#3 in the porting order of CUDA_STRATEGY.md). It uses the Array6D<double> views of cunumpy 0.6.1.

This PR is based on devel and does not depend on #705 (linear_vlasov_ampere). The two PRs need the same device helpers. Here they are copied byte-identically from #705 (origin/cuda-linear-vlasov-ampere at 1ad4cd6), so filler_kernels.cuh, particle_to_mat_kernels.cuh, kernel_test_args.py, test_device_helpers.py and test_cuda_emulation.py merge without conflicts. Whichever PR merges second gets small conflicts in three files: cuda_parity_cases.py, test_cuda_parity.py and CUDA_STRATEGY.md. In all three, both sides add lines in the same place, so keep both (checked with git merge-tree).

Core changes:

  • CUDA kernel pic/accumulation/kernels/vlasov_maxwell/vlasov_maxwell_cuda.cu:

    • Same arguments in the same order as the pyccel kernel, one thread per marker row.
    • Per marker it calls df, matrix_inv, transpose, matrix_matrix and matrix_vector. Then m_v_fill_b_v1_symm adds A_p = w_p DF^{-1} DF^{-T} and B_p = w_p DF^{-1} v_p.
    • The six blocks mat11 … mat33 are Array6D<double> (any strides) and the vectors are Array3D<double>.
    • As in pyccel, only holes are skipped. Boundary particles (markers[ip, -1] == -2) are accumulated, unlike in linear_vlasov_ampere.
  • Device helpers, 1:1 with pyccel, byte-identical to CUDA version of the linear_vlasov_ampere accumulation #705:

    • fill_mat and fill_mat_vec in filler_kernels.cuh
    • m_v_fill_b_v1_symm in particle_to_mat_kernels.cuh

    Every addition into matrix or vector data is a cunumpy_atomic_add. outer from CUDA version of the linear_vlasov_ampere accumulation #705 is not needed here and is not copied.

  • Tests:

    • PARITY_CASES["vlasov_maxwell"] covers 129 markers in five mappings: Cuboid, Colella, HollowTorus, ShafranovDshapedCylinder and a 3d spline mapping (geometry_domain 0, 2, 5, 9, 12).
    • The CPU emulation agrees with pyccel to 7e-12 or better. Entries reach 6e5 in the spline mapping, because of G^{-1}.
    • Tolerances are rtol=1e-12, atol=1e-8, for the atomic summation order on the GPU. atol is scaled up from linear_vlasov_ampere's 1e-10 because the entries are about 60 times larger.
    • Mutation checks: the emulated parity test fails for a transposed DF, for skipped boundary particles and for holes that are not skipped.
    • vlasov_maxwell is added to test_vlasov_kernel_coverage.
    • From CUDA version of the linear_vlasov_ampere accumulation #705 (identical): test_v1_symm_filler (GPU) and test_emulated_v1_symm_filler compare m_v_fill_b_v1_symm with pyccel.
  • Not in this PR: CUDA version of the linear_vlasov_ampere accumulation #705's test_accum_matrix_cupy.py and emulated_launches(). They test the matrix path of Accumulator on CuPy and are not copied, to keep this PR small. I ran an ad-hoc copy of that test with vlasov_maxwell in place of linear_vlasov_ampere, not committed. It runs on the fake CuPy with emulated launches and matches NumPy: the nine kernel arrays, the dense blocks, the vector, BC.dot(x) and three Schur CG iterations, with no host copies from the accumulation to BC.dot(x). After both PRs are merged, that test could get a vlasov_maxwell case.

Model-specific changes:

None. Status of VlasovAmpereOneSpecies and VlasovMaxwellOneSpecies on the GPU:

Part Kernels / operations GPU status
PushEta push_eta_stage, reflect ported
PushVxB (analytic and implicit) push_vxb_analytic, push_vxb_implicit ported
VlasovAmpereCoupling vlasov_maxwell this PR
push_v_with_efield ported
Schur solve needs the FEEC parts below
Initial PoissonSolve charge_density_0form ported
weights through the control variate (update_weights, Maxwellian on xp, jacobian_det through the geometry kernels) ported
stiffness solve needs the FEEC parts below
MaxwellWeakAmpere (VlasovMaxwell only) FEEC only (curl, M1, M2, Schur solve) needs the FEEC parts below
Scalars KineticEnergyPIC (xp.sum) runs on the device
BilinearEnergyFEEC goes through StencilVector.inner, so a host scalar
gauss_error diagnostic uses toarray(), a host copy

Still missing for an end-to-end GPU run:

  • Preconditioner: MassMatrixPreconditioner, the default of VlasovAmpereCoupling, MaxwellWeakAmpere and PoissonSolve, cannot be created on CuPy (is_circulant asserts a host array). The device Kronecker solve is a separate PR.
  • CG inner products: feectools' StencilVector.inner returns a host scalar, so each CG iteration and the FEEC energy scalars copy a scalar from the device.
  • FEEC operators: the derham operators and the mass matrices on the device need the feectools CUDA stack (stencil 6D views, device Kronecker solve, PR 17).
  • Marker sorting: multi-rank marker sorting still goes through the host.
  • H100 run: the first run on an H100 (First GPU run of the CUDA tests (H100) #687), then the models end to end (Run VlasovAmpereOneSpecies end to end on the GPU #689).

Documentation changes:

CUDA_STRATEGY.md:

The checklist and the "6D views are a gate" paragraph are left to #705, to keep the conflict small.

Testing (macOS, no GPU; pyccel kernels compiled with GNU/Fortran; GPU tests not run):

before (devel) after
test_kernel_backends.py test_cuda_parity.py test_device_helpers.py test_cuda_emulation.py 83 passed, 317 skipped 85 passed, 323 skipped
mpirun -n 2: test_verif_VlasovMaxwellOneSpecies.py test_verif_VlasovAmpereOneSpecies.py test_accum_vec_H1.py – 7 passed
  • The 2 new passes are test_emulated_parity[vlasov_maxwell] and test_emulated_v1_symm_filler.
  • The 6 new skips are GPU tests: 5 vlasov_maxwell parity cases and test_v1_symm_filler.
  • The MPI tests only use the pyccel kernels, which this PR does not change. I ran them only on this branch.

🤖 Generated with Claude Code

max-models and others added 3 commits October 7, 2026 19:12
- linear_vlasov_ampere_cuda.cu: same arguments as pyccel, one thread per marker
  row, matrix blocks as Array6D<double> views (cunumpy 0.6.1)
- device fill_mat, fill_mat_vec (filler_kernels.cuh), m_v_fill_b_v1_symm
  (particle_to_mat_kernels.cuh) and outer (linalg_kernels.cuh), with atomic adds
- parity cases (Cuboid, Colella, HollowTorus, 3d spline) and a device-helper
  test of m_v_fill_b_v1_symm, on a GPU and by CPU emulation
- test_accum_matrix_cupy.py: the matrix path of Accumulator on CuPy vs NumPy
  (device data, ghost regions, transposed blocks, Schur operator and solve),
  without a GPU on the fake CuPy with emulated kernel launches
  (emulated_launches() in pic/tests/cuda_emulation.py)
- CUDA_STRATEGY.md: porting order and implementation notes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the CuPy backend, Particles._sendrecv_markers hands the marker rows to
Isend/Irecv as device buffers through cunumpy.mpi.mpi_buffer (directly after
synchronize_for_mpi with CUDA-aware MPI, staged through host memory
otherwise). Rows from each rank arrive in a contiguous slice of one device
receive buffer, and one device scatter fills the holes. The per-rank counts
are small NumPy arrays on every backend (they are host integers already), and
a uniform alpha is a Python scalar, so the exchange makes no host/device copy
and no implicit host read of a device array. The NumPy path is unchanged.

Adds pic/tests/test_sorting_device.py (fake CuPy under mpirun, and a GPU
variant) and a CI step running it on the fake CuPy.

Solves #698, part of #650.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the accumulation kernel of VlasovAmpereCoupling (VlasovAmpereOneSpecies,
VlasovMaxwellOneSpecies) to CUDA, 1:1 with pyccel: Array6D<double> matrix
blocks, atomic adds, holes skipped and boundary particles accumulated as in
pyccel. The device helpers fill_mat, fill_mat_vec and m_v_fill_b_v1_symm,
v1_symm_accumulation_data() and the m_v_fill_b_v1_symm wrapper tests are
byte-identical to the linear_vlasov_ampere PR (#705).

Parity cases in Cuboid, Colella, HollowTorus, ShafranovDshapedCylinder and
a 3d spline mapping; the CPU emulation agrees with pyccel and fails for a
transposed DF.

Part of #688.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
max-models and others added 4 commits October 7, 2026 23:35
The hole in marker_arguments() was only marked in column 8 (first_init_idx),
which the pushers test. Accumulation kernels test column 0, so the
linear_vlasov_ampere parity case had no real hole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stack the PR on #711.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stack the PR on #712.

Conflict in CUDA_STRATEGY.md: kept both sections (marker exchange notes, then linear_vlasov_ampere notes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stack the PR on #705.

Conflicts: CUDA_STRATEGY.md (kept both PRs' notes; vlasov_maxwell row stays done), cuda_parity_cases.py and
test_cuda_parity.py (kept the linear_vlasov_ampere and vlasov_maxwell cases and imports).
Semantic fix for #711 below: vlasov_maxwell has a CUDA version now, so test_compile_cuda_kernels expects
VlasovAmpereOneSpecies to have no missing CUDA kernels and checks fail-fast with push_bxu_Hdiv instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@max-models
max-models changed the base branch from devel to cuda-linear-vlasov-ampere October 7, 2026 22:04
max-models added a commit that referenced this pull request Oct 7, 2026
Stack the PR on #713.

Conflict in CUDA_STRATEGY.md (porting-order gates): kept #714's removal of the MHD equilibria gate and
#705's note on 6D views; marker sorting is on the device since #712.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
max-models added a commit that referenced this pull request Oct 8, 2026
Stack the PR on #723.

Conflicts: eval_spline_mpi_{markers,matrix,sparse_meshgrid}_cuda.cu (SplineArgs of #721 with the qualified
evaluation_kernels_3d::eval_spline_mpi), linalg_kernels.cuh, filler_kernels.cuh, particle_to_mat_kernels.cuh
(helpers of #705/#713 moved into their module namespaces).
Semantic fixes: the struphy_cuda::<pyccel module> rule applied to the CUDA code of the PRs below:
linear_vlasov_ampere (#705), vlasov_maxwell (#713), bstar_parallel_3form (#718) and
eval_spline_mpi_tensor_product_fixed (#724) use 'using namespace struphy_cuda;' and qualified helper calls;
m_v_fill_b_v1_symm calls filler_kernels:: and evaluation_kernels_3d::; the fill_v1_symm test wrapper
calls particle_to_mat_kernels::m_v_fill_b_v1_symm.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@max-models
max-models added this pull request to stack #728 October 8, 2026 05:45
@max-models
max-models removed this pull request from stack #728 October 8, 2026 06:00
@max-models
max-models added this pull request to stack #729 October 8, 2026 06:00
@max-models max-models self-assigned this Oct 8, 2026
@max-models
max-models marked this pull request as draft October 8, 2026 09:14

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant