docs(serving): measured verdict — gather is fit-not-speed; CUDA decode -24% - #2150
Merged
Conversation
…DA decode is 24% SLOWER Correctness closed both backends; speed is split and the CUDA half is negative. 5090 V4-Flash: copy 3.10 tok/s @1875 MiB vs gather 2.36 tok/s @51 MiB — 97% fewer bytes, 24% slower. M5 Metal: copy 12.23 vs gather 49.09 — 4.0x faster. Same mechanism, opposite outcomes, one explanation: the copy path consolidates scattered slots into a contiguous staging tensor ONCE and the matmul then reads it densely many times. Gather removes the transfer but scatters every read across a multi-GB pool, killing L2 and row-buffer locality in the hottest kernel. On the 5090 the copy was ~1-2 ms of a ~320 ms token — never the bottleneck. On the M5 it was 384 small per-expert copies at ~6 GB/s — overhead -bound, which is what removing them actually won. Rule recorded: gather wins where the copy is OVERHEAD-bound, loses where it is BANDWIDTH-efficient. Opt-in stays opt-in; measure both arms per backend. Also records what the mechanism is genuinely for (working sets over VRAM, NVMe tiers, and grid share where 'copy' means a network hop) and the synthesis that could recover both: co-occurrence-driven slot co-location (#228/#229) makes gathered reads dense again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Honest A/B both backends, the locality explanation, the overhead-vs-bandwidth rule, and the co-location synthesis.
🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo