Optimize interleaved loads on x86

[Shnatsel]

...by removing the AVX2 specialization.

Yes, you read that right. The AVX2 specialization improves throughput somewhat but significantly degrades latency. And for loads you care about latency far more than throughput because processing can't start until the data is loaded.

Here's a table of latencies according to llvm-mca:

64-bit formulation	Broadwell	Skylake	znver3
Current YMM special case	20	21	26
Ordinary four-XMM unpack (this PR)	16	16	16
Constant swizzle_dyn	22	23	28

Throughputs of various AVX2 shuffle formulations also vary between Intel and AMD, but AVX2 latencies are universally worse.

I've also tried using AVX2 gather ops here, but they're slower on Intel and extremely slow on AMD, slower than individual scalar loads.

Man, to think that Intel made AVX-512 with proper shuffles all the way back in 2015 and then didn't start putting it into consumer CPUs until literally this year.

Closes #305

[LaurenzV]

Have you tried the pack and unpack benchmarks in vello?

[Shnatsel]

Vello is unaffected by this because it only calls interleaved loads/stores on 8-bit and 32-bit types. This affects 64-bit types only, since they're the only ones with an AVX2-specialized codepath.

In retrospect I should have mentioned that in the PR description.

[LaurenzV]

My bad, should have seen that it's only 64-bit, I hadn't checked the diff yet. Gonna trust your judgement here. 😄 Though it would still be interesting to see whether this holds up in an actual benchmark.

[Shnatsel]

Okay, once you plug this into an actual benchmark and disassemble, something really interesting happens. LLVM completely rewrites the AVX2 formulation. It is nothing like the AVX2 intrinsics as actually written. There are no two 256-bit loads and no permutes. Instead there are two 128-bit loads for the first two vectors and then broadcast loads plus a shuffle for the other two. Fascinating.

In isolation this branch should still win according llvm-mca (equal latency but higher throughput), but I'm actually seeing a 5% performance regression on a plausible color-processing loop (with the caveat that I'm measuring on zen4 in avx2 mode, while zen4 would actually use AVX-512 path).

The inlined code with per-vector processing comes out on top because LLVM avoids the final split in its heavily rewritten AVX2 formulation, by avoiding materializing blue and alpha channels into XMM registers and instead keeps them in a single YMM register. So the AVX2 benefit only appears once everything is inlined, but in isolation this branch is cheaper.

I'll sleep on it, see what I can do about this.

Hot loop used for the measurement:

    for (input, output) in input_chunks.iter().zip(output_chunks) {
        let [r, g, b, a] = simd.load_four_interleaved_f64x2(input);

        // A modest linear color transform. Keeping the work small makes load
        // latency relevant while still representing real processing.
        let corrected_r = r * 1.01 + g * 0.02;
        let corrected_g = g * 0.99 + b * 0.01;
        let corrected_b = b * 1.02 - r * 0.01;
        let corrected_a = a * 0.995 + 0.005;

        simd.store_four_interleaved_f64x2(
            [corrected_r, corrected_g, corrected_b, corrected_a],
            output,
        );
    }