Repository navigation
Conversation
The SME unit moves data 2-4x faster than NEON or the generic copies. In sme2_gemm_copy.h, s2c_rows() uses predicated streaming vectors and s2c_cols() transposes through ZA, for pieces of 512 elements or more; smaller pieces and CPUs without SME2 stay on NEON. The new B-side copies (S/DGEMMONCOPY, S/DGEMMOTCOPY) use the same helpers with the layout of the generic copies (UNROLL_N, then 8/4/2/1). Part of OpenMathLib#6146
…ement tests The diagonal block goes through the panel copy; then each row gets its zero side and its (unit or inverted) diagonal element. Part of OpenMathLib#6146
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the copy part of #6146.
Before: on VORTEXM4 the B-side GEMM copies were the generic C copies, and the A-side copies of the SME2 kernels used NEON. At n <= 512 these copies took most of the level-3 run time (about 70-80% of SSYRK and STRMM at n = 256, one thread). NEON versions of the B-side copies were tried first: they gain little, because the copies are limited by the bandwidth that one core gets from L2.
After: the copies run in streaming mode. The SME unit moves the same data 2-4x faster (fp32, one thread, data in L2: oncopy 50 -> 165 GB/s, otcopy 75 -> 220 GB/s at m = n = 256).
How:
sme2_gemm_copy.h:s2c_rows()(contiguous rows) uses predicated streaming vector loads and stores;s2c_cols()(contiguous columns) loads columns into horizontal ZA slices and stores rows from vertical slices. Pieces smaller than 512 elements, and CPUs without SME2 (s2_usable()), keep the NEON code.gemm_[nt]copy_b_sme2.cfor S/DGEMMONCOPY and S/DGEMMOTCOPY: the same helpers, with the layout ofgeneric/gemm_[nt]copy_16.c(UNROLL_N columns, then 8/4/2/1), in one streaming-mode entry per call.Only VORTEXM4 files change. GCC builds keep the generic copies (the KERNEL.VORTEXM4 section for non-GCC compilers).
Not in this PR: the OUTER TRMM/TRSM/SYMM copies (right-side B panels) are fixed to the generic files in
kernel/Makefile.L3; changing them needs a shared-file change.Results
M4 Pro, one thread, GFLOPS, develop 791b02a -> this PR:
SGEMM/DGEMM do not change. Default threads: also faster, no case slower (SSYMM n = 1024 2540 -> 2955, STRMM 1753 -> 1967, at n = 4096 5-11%).
Tests
make: BLAS/CBLAS tests, utest 144/144 and 1564/1564.make lapack-test: 0 numerical errors.-DTARGET=VORTEXM4 -DNOFORTRAN=1 -DBUILD_TESTING=ON, ctest 14/14) andDYNAMIC_ARCH=1 NO_FORTRAN=1builds.AI assistance
AI coding tools (Claude Code) helped to prepare this PR. The changes were reviewed, tested and benchmarked as described above.