Skip to content

tools.igrog compiled gridders; packed grid-normal kernel; empty_cuda_memcache - #171

Merged
mcencini merged 8 commits into
mainfrom
claude/project-thread-2eaqzo
Nov 4, 2026
Merged

mcencini merged 8 commits into
mainfrom
claude/project-thread-2eaqzo

Conversation

@mcencini

@mcencini mcencini commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

tools.igrog now grids with compiled loops instead of batched torch:

  • Host (src/csrc/ops/grog.c, bartorch_grog_grid): ported from fast-encoding's grog.cpp, in single precision, with OpenMP across targets. For each target it interpolates Q (cubic Lagrange in the table), factors R_ss + mu I by Cholesky, applies W to every set and returns the noise gain.
  • Card (src/csrc/ops/grog.cu, bartorch_grog_grid_cuda): one target per warp. The packed lower triangle of the system, one row of W and the offsets sit in shared memory (1.6 KB per warp at 18). Lanes split the coil-pair interpolation (coalesced table rows), the Cholesky rows, the solves and the per-set reductions. The card holds only the samples and one batch of targets.
  • igrog(..., device=None) grids on the samples' device. double=True keeps the torch double-precision path for checks, and the torch path is also the fallback on a build without the card gridder.
  • also= moves extra sample sets (e.g. a noise draw) with the same kernels.
  • The torch kernels interpolate only the pairs i < j, using Q(-d) = Q(d)^H.

Two more changes for the gridded-positions normal (CartesianSense(positions=)):

  • Packed kernel. The per-place R x R kernel is Hermitian, so only r <= c is stored (packed upper triangle). When every entry is real (a real subspace basis) it is stored as floats. At rank 4 that is 10 floats per place instead of 16 complex. The host normal and kern_contract_grid read the packed layout.
  • bartorch.empty_cuda_memcache() returns BART's per-stream cache of device blocks, so a caller can free VRAM between stages without turning the cache off.

Measured (Deli-CS case, 256³, cycled R8, 6 coils, 3 sources, A40)

path igrog
torch on card, fp64 (0.0.21) 37 s
torch on card, fp32 23 s
card, thread per target (75d5917) 5.3 s (kernel 3.5 s)
card, warp per target (885e1f1) 2.7 s (kernel 0.95 s, peak 1.1 GB)
host loop, 96 cores 3.0 s

Card and host agree to 2e-6 on the values. The noise draw differs by 2.5e-4, which is single-precision rounding: host and card are each 2.3e-4 from double.

Full recon (mri2qmap, at e7cf777): the grid kernel goes from 0.35 to 0.11 GiB (2.92 M places, real basis). Emptying the memcache after each CG solve frees about 2.5 GB during the other stages, and CG is not slower (10.1 s vs 11.8 s). The process peak drops from 6.5 to 6.0 GB, or to 5.55 GB with coil kernels on the caller's side. The maps match the previous run within run-to-run noise.

Tests

  • test_single_precision_matches_double, test_also_moves_extra_samples_by_the_same_kernels, test_the_card_gridder_matches_the_host_loop (skips without a card).
  • test_points_normal_with_a_real_basis_keeps_a_real_kernel_and_is_the_two_applications, test_emptying_barts_device_cache_is_safe_with_or_without_a_device.
  • On an A40 CUDA build of e7cf777: test_linop_mri, test_igrog, test_cuda, test_abi give 237 passed, with 1 skip (the no-card refusal test).
  • Note for local CUDA builds made with clang: set CUDAHOSTCXX=clang++, otherwise cuFINUFFT's templates mangle differently.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MFB6QLjhTxFMFDEzathbEm

… double only on the host)

Consumer cards run complex128 at 1/64 rate: on an RTX 4060 the 256^3 Deli-CS case
spent 86 s gridding. Single and double agree to 2e-6 relative on the test object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFB6QLjhTxFMFDEzathbEm
@mcencini mcencini self-assigned this Nov 4, 2026
…e kernels

Saves solving every kernel a second time to propagate noise through the gridding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFB6QLjhTxFMFDEzathbEm
@mcencini mcencini changed the title tools.igrog: single-precision kernels on a card tools.igrog: single-precision kernels on a card, and also= for extra samples Nov 4, 2026
@codecov

codecov Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 61.24031% with 50 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/bartorch/tools/_igrog.py 57.75% 49 Missing ⚠️
src/bartorch/_cuda.py 50.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

…rch_grog_grid)

Ported from fast-encoding's grog.cpp to C, in single precision: per target,
Q at the source-pair and source-target displacements by cubic Lagrange in the
table, R_ss + mu I by Cholesky, the kernel applied to every set (kspace and
also=) and the noise gain. Targets are spread over OpenMP threads.

igrog now grids on the host with it by default (device=None); device= a card
keeps the batched torch solve, and double=True the torch double solve. The
torch kernels interpolate only the pairs i < j (Q(-d) = Q(d)^H) and gather
table rows with index_select.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFB6QLjhTxFMFDEzathbEm
Each thread interpolates Q into the packed lower triangle of R_ss + mu I,
factors it, and solves W a row at a time with that row of R_ts, then applies
it to every set: no workspace, the card holds the samples and one batch of
targets. igrog grids on the samples' device by default; torch remains for
double=True and builds without the card gridder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFB6QLjhTxFMFDEzathbEm

mcencini commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

CodeFactor reports 2 minor issues on 75d5917. Its page renders them client-side only and the check run carries no text, so I can't see which lines they are. They are most likely complexity notes on the new gridders (grog.cu's per-target kernel, tools/_igrog.py). I split the C loop into helpers in 1114d7d already. I'll restructure further once the build and test jobs (including cuda-build, which compiles grog.cu) are in, if the findings turn out to be actionable.


Generated by Claude Code

…ounding (2e-4)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFB6QLjhTxFMFDEzathbEm

mcencini commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

codecov/patch reports 60 % of the diff hit (target 93 %) on 037c83b. The lines it misses are the card path: grog.cu, _card_grid in tools/_igrog.py, and the torch fallback a card takes when the build has no gridder. The CI test runners have no GPU, so nothing there can reach them; cuda-build only compiles. test_the_card_gridder_matches_the_host_loop exercises that path. It ran on an A40 build of this branch and failed only on the noise draw: 2.5e-4 against the 1e-4 tolerance, single-precision rounding, with the host and card equally far from double. 037c83b loosens that tolerance to 1e-3; the test has not been re-run since. The host loop (grog.c) and the torch double path run in CI. I'm leaving coverage as it is rather than adding no cover pragmas.


Generated by Claude Code

The per-thread kernel kept 1.8 KB of system per thread in local memory,
which streamed through L2 (0.7 s per million targets on an A40). Now a warp
shares a target: a lane per coil pair interpolates Q (coalesced table rows),
a lane per row factors and solves, and the sets are applied by a warp
reduction; the packed triangle, one row of W and the offsets sit in shared
memory, 1.6 KB per warp at 18.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFB6QLjhTxFMFDEzathbEm
@mcencini mcencini changed the title tools.igrog: single-precision kernels on a card, and also= for extra samples tools.igrog: compiled gridders (host C loop, CUDA warp-per-target), single precision, also= Nov 4, 2026
…pty_cuda_memcache

The positions normal kept a full R x R complex kernel per kept place.
It is Hermitian, so only r <= c is stored (packed upper triangle), and
when every entry is real (real subspace basis) it is stored as floats:
R(R+1)/2 instead of R^2 complex entries, a further 2x for a real basis.
The host normal and kern_contract_grid read the packed layout and
conjugate for r > c.

bartorch.empty_cuda_memcache() frees BART's per-stream cache of device
blocks, so a caller can hand VRAM back between stages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFB6QLjhTxFMFDEzathbEm
@mcencini mcencini changed the title tools.igrog: compiled gridders (host C loop, CUDA warp-per-target), single precision, also= tools.igrog compiled gridders; packed grid-normal kernel; empty_cuda_memcache Nov 4, 2026
@mcencini
mcencini merged commit 52f3fd4 into main Nov 4, 2026
34 of 36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants