Repository navigation
Conversation
GEMV N keeps up to 64 vectors of y in ZA and adds four columns at a time with the indexed multi-vector FMLA, column after column as the reference BLAS. GEMV T accumulates 16 columns per block in ZA (one ZA vector per column, strided multi-vector loads) and sums them through a tile transpose. Both enter streaming mode once per call. Small problems, T column tails and CPUs without SME2 use NEON code (sgemv_n_neon.c for fp32 N).
The SME2 GEMV kernels share the SME unit of a P-cluster, so more threads than units are slower. Use num_cpu_avail(3), which sme_level3_threads() caps at the number of units, above 8 MB of matrix, and one thread below.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the GEMV part of #6149.
Before: on VORTEXM4, SGEMV/DGEMV used the NEON kernels of NEOVERSEN1 (
sgemv_n_neon.c,gemv_n.S,gemv_t.S): one thread, 1024 x 1024, sgemv N 44 and T 36 GFLOPS, dgemv N 16 and T 18. Accelerate on the same M4 Pro: 435 / 201 / 212 / 128. With the default threads, GEMV used up to 12 threads.After: new SME2 kernels accumulate in ZA and reach 394 / 209 / 224 / 106 GFLOPS there. With the default threads, matrices above 8 MB use one thread per SME unit, and are faster than Accelerate (e.g. sgemv N 4096 x 4096: 108 against 96 GFLOPS).
How:
kernel/arm64/gemv_n_sme2.c: y in ZA (up to 64 vectors per block), x four (fp32) or two (fp64) values at a time, scaled by alpha, with indexed multi-vector FMLA column by column, so the order of operations is that of the reference GEMV. One streaming-mode entry per call.kernel/arm64/gemv_t_sme2.c: blocks of 16 columns, one ZA accumulator per column, fed by strided four-vector loads; at the end of a block the 16 column sums are reduced through a vertical tile read. The inner loop is a short inline-asm block (with intrinsics, clang 21 adds register moves that cost about 2x at m <= 256); it relies on the 512-bit streaming vector length thats2_usable()checks.kernel/arm64/sme2_gemv.h: shared helpers. Small problems (where the 28 ns streaming-mode entry dominates) and CPUs without SME2 use NEON: for fp32 N the existingsgemv_n_neon.c, otherwise new NEON loops (develop's assembly kernels cannot be called from a C kernel without build changes). Strided x and y go throughbuffer, as in other kernels.KERNEL.VORTEXM4selects the four kernels inside its non-GCC block.interface/gemv.c(shared):get_gemv_optimal_nthreads_vortexm4(), in the style of the NEOVERSEV1/V2 helpers, for clang builds (and VORTEXM4 in DYNAMIC_ARCH viagotoblas_corename()): one thread up to 8 MB of matrix, above itnum_cpu_avail(3), whichsme_level3_threads()caps at the number of SME units. With the SME kernels, 12 threads were 2-4x slower than 2. Other targets are unchanged.Results
M4 Pro, column-major, inc 1, GFLOPS. Develop -> this PR (Accelerate). The machine was shared with other builds during the runs; the median spread between runs was 2-7%.
One thread (Accelerate with
VECLIB_MAXIMUM_THREADS=1):Default threads:
Known weaker case: sgemv N 64 x 65536 gives 124 GFLOPS on one thread but 78 with the two threads that the 8 MB rule selects, because
gemv_thread_nsplits the 64 rows; the hook only sees m * n.Tests
make: BLAS/CBLAS tests, utest 144/144 and 1564/1564.make lapack-test: 0 numerical errors. (With an FMA-based fp32 N fallback, SBD test 9 at M = 40, N = 30 reached 53.8 against the threshold 50; the fallback is now develop'ssgemv_n_neon.cand the test passes.)cblas_[sd]gemvagainst a long-double reference: both transposes and orders, alpha in {0, +-1, 0.7, random}, beta in {0, 1, -0.3, random} (beta = 0 with NaN in y), incx/incy in {1, 2, -1, -3}, m and n 0-300 and up to 4100, untouched elements between strides checked: 22000 runs, 0 failures, one and default threads.-DTARGET=VORTEXM4 -DNOFORTRAN=1 -DBUILD_TESTING=ON, ctest 14/14),DYNAMIC_ARCH=1 NO_FORTRAN=1(randomized check, 0 failures) and GCC 16 (TARGET=VORTEXM4, NEON kernels as before; BLAS tests and utest pass).AI assistance
AI coding tools (Claude Code) helped to prepare this PR. The changes were reviewed, tested and benchmarked as described above.