Repository navigation
Add radix 16 support - #28
Merged
Merged
Conversation
Stages of a single line with stride s below the complex count C of the widest vector (s = 1, 2, 4 for Float32 on AVX-512) ran on narrower vectors. stage_small! vectorises over f = q + s·p at full width: contiguous loads, per-lane twiddles (tw1 for s = 1, a new table twg otherwise) and an in-register interleave of the R outputs into contiguous stores, its permutation checked by the generator. Both chain forms use it. The kernels (radices 2, 4, 8, 16 × sign × stride, the widest vector type) take the stride as a constant and the length as run-time data, so they are a fixed set: plain methods in kernel_methods.jl, reached from RChain through cstage_small! and the new stage call OP_SMALL of the single-line schedule. Intel Core i7-1185G7, one line, Float32, RChain, radices of 16s and 8s (µs, without -> with; the same code against itself differs by up to 3 %): 128 0.093 -> 0.069 256 0.176 -> 0.131 1024 0.81 -> 0.61 2048 1.60 -> 1.35 4096 3.75 -> 3.29 8192 7.85 -> 6.69 16384 18.0 -> 15.4 Float64 has no such stage on this machine. Not done: m and s as constants of a single line. It needs one kernel per length (compiled Chain only), and in Chain it measured 0.90-1.04 of the time with run-time m and s (lengths 16-16384, both element types), inside the spread. Recorded in docs/src/explanation/design-decisions.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Powers of two from 128 to 16384 take as many 16s as possible and 8s for the
rest (512 stays 8 8 8). Same rule for Float32 and Float64.
Intel Core i7-1185G7, one line, RChain (µs, 4s and 8s -> 16s and 8s; the same
code against itself differs by up to 3 %):
F64 128 0.145 -> 0.128 256 0.302 -> 0.256 1024 1.51 -> 1.34
2048 3.99 -> 3.53 4096 7.64 -> 7.13 8192 18.0 -> 16.3
16384 36.9 -> 36.2
F32 128 0.074 -> 0.069 256 0.131 -> 0.131 1024 0.632 -> 0.611
2048 1.27 -> 1.35 4096 3.25 -> 3.29 8192 7.26 -> 6.69
16384 16.0 -> 15.4
Float64 is 2-15 % faster, Float32 between 6 % slower (2048) and 8 % faster.
512 as 16 4 8 was not measured on the run-time chains; it stays 8 8 8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
kernel() compiles its Bluestein branch, whose filter transform calls run! with an uninferred Chain. Inference enumerated the few run! methods and compiled run!(::RChain), whose dispatch over every radix, sign and vector type instantiates all codelets. max_methods 1 on run! leaves such calls to run-time dispatch; concretely typed calls (the run-time plans) are unaffected. First fft of a fresh session, n = 1024: F64 19.6 -> 13.9 s (607 -> 19 kernel specializations), F32 afterwards 22.7 -> 5.0 s. The test suite takes as long (it compiles the run-time table in its own testsets). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
stage_small! runs the first stage of a single line (stride 1, power-of-two radix) on the widest vector where the grouped first stage is narrower (Float32 on AVX-512). Its twiddles are the grouped first-stage table tw1, so the chains carry no per-lane tables, the kernels have no stride parameter and the RChain dispatch has no stride switch. Stages with a stride of 2 or 4 below the widest vector run on the vector that fits their stride. Only Float32 lengths whose radices start with 4, 4 have such a stage (stride 4; 625 lengths up to 70000, 65536 the only power of two), none with stride 2, none for Float64. The lengths with radices of 16s and 8s (128-16384) never had one. AMD EPYC 9354, one line, Float32, RChain, time without the full-width stride-4 stage over time with it, medians of 10 interleaved rounds (65536 and 262144: 5 rounds); the same code against itself differs by 5-20 % between rounds: 192 1.08 320 1.06 768 1.03 960 1.03 3072 0.96 4800 0.99 20480 0.97 65536 0.96 262144 0.97 Only the 6-8 % at 192 and 320 repeated in a second series. Not measured on other machines. Recorded in docs/src/explanation/design-decisions.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…kernels The stages vectorised over q (stage!, stage_range!, stage_last!) need few registers besides the data in their inner loop: - ccmul_at multiplies by the twiddle at a pointer: re(w) and im(w) are scalar broadcasts, a·w = a·re(w) -/+ swap(a)·im(w) with the sign alternating over the lanes. On x86 both are memory operands. Every complex multiply outside the FCMA path has this form (ccmul_parts), so all kernels round alike and threaded and single-threaded transforms stay bit-identical. - The R input and R output streams are addressed as base + odd·stride·2^e with odd in (1, 3, 5, 7) and one base per 8 streams. The base pointer of the loop passes through llvm.launder.invariant.group and the scaling is a typed GEP; without both, loop strength reduction rewrites the addresses into one hoisted pointer per stream. A radix-16 stage has 15 twiddles and 32 streams; with twiddles in registers and one pointer per stream its inner loop reloaded vectors and pointers from the stack. Instructions per butterfly in the inner loop, AMD EPYC 9354 (AVX-512), Float64 (Float32 within 3), before -> after: radix 16 middle stage 284 -> 198 last stage 182 -> 151 radix 8 middle stage 100 -> 79 last stage 71 -> 58 Time, twiddle multiply alone (an earlier form with the x86 vfmaddsub intrinsics), one line, plan_fft, medians of 8 interleaved rounds, after/before: Float32 512-16384 0.88-0.93 (8192: 0.94 on the minima), Float64 1024, 4096, 8192, 16384 0.94-0.96, 32768 and above 0.98-1.01; the same code against itself differs by 1-5 % for most lengths. The time of both changes together is not verified. Not verified on aarch64. The test suite was not run on this commit; spot checks against a direct DFT and the threaded rfft equality pass. Recorded in docs/src/explanation/design-decisions.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The two entries state the decision, the measured evidence and what was not done, without the detail that the code and the commit log carry. The entry on twiddles and stream addresses now reports both changes together (AMD EPYC 9354, one line, plan_fft, medians of 8 interleaved rounds against the kernels before): lengths 512-16384 take 0.79-0.94 of the time for Float32 and 0.82-0.92 for Float64, lengths from 32768 on are unchanged; the same code against itself differs by 1-5 %. The quick test selection passes with these kernels. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A radix-16 stage reads 16 streams n/16 elements apart. Once that spacing reaches the L1 set stride (4 KB) the streams share one cache set, and the sequences of 8s are faster. Single lines on AVX-512 therefore take radix 16 up to 4096 for Float32 and 2048 for Float64; 8192 and 16384 (and Float64 4096) use 8s with a 4 or two 4s again. AMD EPYC 9354, one line, all buffers at fixed page offsets (staggered by 1 KB), µs, sequence of 8s -> sequence of 16s: Float32 4096 3.56 -> 3.13 8192 7.48 -> 7.93 16384 15.95 -> 16.9 Float64 4096 6.59 -> 8.79 8192 14.81 -> 15.32 16384 31.76 -> 35.3 The ranking is the same with all buffers at one page offset. plan_fft against main (103b9b6), medians of 8 interleaved rounds, time of this branch over main: Float64 4096 0.86 (1.23 before this rule), 8192 0.94, 16384 0.95; Float32 8192 0.85, 16384 0.87. The same code against itself differs by 1-5 % for most lengths, up to 25 % for a few. The quick test selection passes. Not measured on Intel with this rule. Recorded in docs/src/explanation/design-decisions.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes AVX512 systems a good bit faster. (the radix16 is not most of it, most of the code here is to allow full 512 bit transforms for Float32 which needs stride-8)
AMD EPYC 9354 (Zen 4, AVX-512), one thread, 1-D complex
plan_fft+mul!out of place,main(103b9b6) vs PR (b60c8b9).