cuda: implement real per-(layer,expert) LRU for --ssd-streaming-cache-experts - #647
cuda: implement real per-(layer,expert) LRU for --ssd-streaming-cache-experts#647nexus-cw wants to merge 4 commits into
Conversation
…eaming expert fetch path CUDA SSD streaming selected-expert loading (cuda_stream_selected_cache_begin_load in ds4_cuda.cu) has never had any per-generation observability: no fetch/hit/miss/byte counters, unlike ds4_metal.m which already tracks real per-(layer,expert) LRU hit/miss stats via --expert-profile. Add four host-side counters (fetch_calls, expert_fetches, cache_hits/misses, bytes_from_file/cache), incremented at the fetch decision point, and a new ds4_gpu_print_cuda_stream_stats() that prints them when DS4_CUDA_STREAM_STATS=1 is set in the environment, called from ds4_cli.c after a sampled generation completes. A no-op stub is added to ds4_metal.m for build symmetry (Metal already has its own richer instrumentation). At this point in the series there is still no persistent cache to hit, so hits/bytes_from_cache stay at zero -- the next commit makes them real.
…streaming-cache-experts Fixes antirez#639. --ssd-streaming-cache-experts has never had any effect on CUDA: cuda_stream_selected_cache_begin_load() unconditionally invalidates and re-fetches every selected expert's gate/up/down bytes from the mapped model file on every call, and ds4_gpu_set_streaming_expert_cache_budget()/_expert_bytes() were no-op stubs while ds4_gpu_stream_expert_cache_configured_count() hard-returned 0 -- unlike ds4_metal.m's g_stream_expert_cache, a real per-(layer,expert) LRU with genuine hit/miss tracking. This ports that design to CUDA: a persistent, device-resident (layer,expert)-keyed table (cuda_stream_expert_cache_entry, a fixed 80x384 array) consulted by cuda_stream_expert_cache_peek() before cuda_stream_selected_cache_begin_load() falls back to its existing mapped-file fetch. A hit is served with a device-to-device cudaMemcpy into the existing per-call packed staging buffer that the downstream decode kernels already read from, so the kernel-facing interface is unchanged; a miss still pays the existing fetch and then additionally installs those bytes into the persistent cache (cuda_stream_expert_cache_install()) so the entry survives past this call. Eviction is global least-recently-used across the whole table (cuda_stream_expert_cache_prune_global()), enforced against the budget now stored by the previously-stubbed ds4_gpu_set_streaming_expert_cache_budget(). Budget semantics match the existing CLI contract: N in --ssd-streaming-cache-experts N (or the NGB form, already converted to a plain expert count upstream of the GPU backend by ds4.c, identically for Metal and CUDA) is the entry-count budget, clamped to the table's fixed bound. Unlike Metal's single-size-class slab allocator (which excludes the Q3_K/Q5_K layers off its uniform size class from caching), CUDA's per-entry cudaMalloc has no such uniform-size requirement, so all routed layers -- including those excluded on Metal -- participate in one cache here; ds4_gpu_set_streaming_expert_cache_expert_bytes() is kept for CLI/log symmetry but does not gate CUDA caching. The DS4_CUDA_STREAM_STATS counters added in the previous commit now measure a real cache: hits/bytes_from_cache are populated on the peek() hit path instead of staying structurally zero.
…ee thrash at small budgets cuda_stream_expert_cache_install()/_clear_entry() called cudaMalloc/ cudaFree directly on every miss/eviction. That's fine once the cache reaches a steady hit rate, but at a too-small budget (relative to the per-token expert working set) the cache thrashes: install, evict, install, evict, on every call, with no offsetting hit-rate benefit -- regressing decode throughput below the no-cache baseline. Add a small pool of free device buffers keyed by exact byte size (cuda_stream_expert_pool_class_for/_alloc/_free/_release_all). Gate/up share one size per layer's size class and down has its own, so there are only a handful of distinct sizes in play; a short linear-scan array of size classes is simpler than a hash table at this scale and just as fast. On a miss/eviction, a freed buffer goes back onto its class's free list instead of being cudaFree'd; the next install needing that size pops it back off instead of calling cudaMalloc. Real cudaMalloc only happens the first time a class needs more buffers than have ever been freed back to it, i.e. it amortizes to the cache's steady-state entry count rather than firing once per eviction. The pool only returns memory to the driver via cuda_stream_expert_pool_release_all(), called alongside every existing whole-cache-reset site (model swap, streaming-mode toggle, budget change) that already calls cuda_stream_expert_cache_clear_all() -- those are rare events, never per-token, so real teardown is unchanged; only the hot miss/eviction path benefits from pooling.
|
Noting overlap with #605 (iCreil), which attacks the same missing-CUDA-expert-cache problem and was filed first — apologies for not spotting it before opening this. The approaches look complementary rather than competing: #605 adds a resident cache plus upload pipelining and pageable-copy optimizations (impressive 29-32 tok/s on an RTX PRO 6000), while this PR ports the Metal per-(layer,expert) LRU semantics — budget accounting per the CLI contract, frequency+recency eviction, pooled buffers, and env-gated hit/miss counters. Happy to rebase this on top of #605 if that lands first, or to fold the LRU/budget/counters pieces into #605's structure if iCreil prefers — whichever shape is most useful. Measurements in both PRs agree on the headline: the expert cache is worth 3-14x on streamed decode. |
…ry count The LRU budget was enforced only as an entry count derived from the dominant slab size class, while CUDA caches every routed layer at its actual per-expert size -- off-class layers cache entries larger than the slab figure, so N entries * slab_bytes under-states real device consumption. The buffer pool's free lists additionally park real cudaMalloc'd memory that no accounting saw. In production hardening on a GB10 node this let a nominal 100GB cache budget reach ~121GB of actual device memory. Now valid entries are charged at their actual gate+up+down byte sizes, pool free-list bytes are counted as parked, and installs make room before allocating -- trimming parked pool buffers first (data-lossless), then evicting LRU -- so counted + parked never exceeds budget_experts * expert_bytes, the exact dynamic-cache byte figure ds4.c plans and logs at startup. The entry-count cap remains as a secondary bound. Parked bytes reusable by the incoming request are treated as slack rather than new demand, so at the cap the evict -> park -> pop cycle reuses buffers with no cudaMalloc/cudaFree traffic (a first cut that double-counted them measured a ~30 percent decode throughput regression from exactly the alloc/free churn the pool exists to prevent). DS4_CUDA_STREAM_STATS=1 now also self-reports the cache's true memory footprint (budget, counted, pool-parked, device total, persistent pinned-host staging, entry count) every 4096 fetch calls, and the end-of-generation stats call in ds4_cli.c is compiled only into CUDA builds so make cpu links again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Pushed one more commit (ab98473) with a fix for an accounting bug we found while hardening this cache in production. The bug: the LRU budget was enforced only as an entry count derived from the dominant slab size class, while CUDA caches every routed layer at its actual per-expert size — layers off the slab class cache entries larger than The fix:
One subtlety: parked bytes reusable by the incoming request are treated as slack rather than new demand. A first cut that double-counted them forced a trim-then-malloc cycle on every miss at cap and cost ~30% decode throughput — exactly the alloc/free churn the pool exists to prevent.
The fix is production-verified on the same GB10 node at a 70GB budget: actual consumption stays at the nominal figure with no measurable throughput cost versus the pre-fix pooled path. 🤖 Generated with Claude Code |
2 participants
Fixes #639.
Problem
--ssd-streaming-cache-expertshas never had any effect on the CUDA backend:cuda_stream_selected_cache_begin_load()unconditionally invalidates and re-fetches every selected expert's gate/up/down bytes from the mapped model file on every call, andds4_gpu_set_streaming_expert_cache_budget()/_expert_bytes()were no-op stubs whileds4_gpu_stream_expert_cache_configured_count()hard-returned 0 -- at any budget, on any model. Metal's GLM streaming implementation, by contrast, has always had a real per-(layer,expert) LRU (g_stream_expert_cache) with genuine hit/miss tracking.Design
Metal-LRU parity. Ports Metal's design to CUDA: a persistent, device-resident
(layer,expert)-keyed table (cuda_stream_expert_cache_entry, a fixed 80x384 array) is consulted bycuda_stream_expert_cache_peek()beforecuda_stream_selected_cache_begin_load()falls back to its existing mapped-file fetch. A hit is served with a device-to-devicecudaMemcpyinto the existing per-call packed staging buffer that the downstream decode kernels already read from, so the kernel-facing interface is unchanged; a miss still pays the existing fetch and then additionally installs those bytes into the persistent cache (cuda_stream_expert_cache_install()) so the entry survives past this call. Eviction is global least-recently-used across the whole table.Budget semantics per CLI contract.
Nin--ssd-streaming-cache-experts N(or the NGB form, already converted to a plain expert count upstream of the GPU backend byds4.c, identically for Metal and CUDA) is the entry-count budget, clamped to the table's fixed bound -- no CUDA-specific CLI or budget-math changes were needed.Device-resident entries. Unlike Metal's single-size-class slab allocator (which excludes mixed-precision "boosted" layers off its uniform size class from caching), CUDA's per-entry
cudaMallochas no such uniform-size requirement, so all routed layers -- including any boosted ones -- participate in one cache here.Pooled allocator.
cuda_stream_expert_cache_install()/_clear_entry()originally calledcudaMalloc/cudaFreedirectly on every miss/eviction, which thrashes badly at a too-small budget (install, evict, install, evict, every call, with no offsetting hit-rate benefit) -- a real regression below the no-cache baseline. A size-keyed device-buffer pool (cuda_stream_expert_pool_class_for/_alloc/_free/_release_all) turns install/evict into pool pop/push, with realcudaMalloconly on genuine growth; every existing whole-cache-reset call site (model swap, streaming-mode toggle, budget change) still releases the pool back to the driver, so real teardown is unchanged.Env-gated counters. Adds
DS4_CUDA_STREAM_STATS=1: four host-side counters (fetch_calls, expert_fetches, cache hits/misses, bytes_from_file/cache) at the fetch decision point, printed via a newds4_gpu_print_cuda_stream_stats()called from the CLI's single-shot generation path (ds4_cli.c). A no-op stub is added tods4_metal.mfor build symmetry -- Metal already has its own richer instrumentation via--expert-profile. This closes the observability gap that made the original no-op possible to ship unnoticed, and gives reviewers a way to directly observe hit rate on their own hardware.Commits
cuda: add DS4_CUDA_STREAM_STATS=1 diagnostic counters for the SSD streaming expert fetch path-- observability scaffolding, structurally-zero hits/bytes_from_cache at this point (no cache exists yet).cuda: implement a real per-(layer,expert) LRU expert cache for --ssd-streaming-cache-experts-- the fix itself.cuda: pool the expert LRU's device buffers to avoid cudaMalloc/cudaFree thrash at small budgets-- correctness-adjacent quality fix so a too-small budget doesn't regress below the no-cache baseline.Measurements (from the linked issue, IQ2_XXS DeepSeek V4 Flash artifact, single GB10-class GPU)
Short-bench decode throughput at a 100 GB cache budget: 1.03 -> 2.96 t/s (measured hit rate ~81%). A long multi-prompt session converges to a ~96-98% hit rate as the working set of routed experts stabilizes. In this PR's own testing (20 GB budget, single process, an 8-question multi-topic prompt over ~220 generated tokens): hit_rate=0.515, decode 1.62 -> 4.41 t/s, cache saturating at its configured 2522-entry budget.
Test evidence
DS4_CUDA_STREAM_STATS=1on the multi-topic session reportshits=8256 misses=7788 hit_rate=0.515 ... budget=2522 entries=2522-- a real, populated cache, not the previous structural zero.--ssd-streamingomitted entirely, on branch versus upstream/main.make clean && make cuda-spark-- clean, no warnings../ds4_test: this branch shows the same pre-existing-flaky section set as a cleanupstream/mainrun in this environment (tool-call-quality,logprob-vectors,metal-kernels, andmetal-tensor-equivalence/think-tool-recovery, which flip between runs on both trees) -- no new failing sections introduced by this change.🤖 Generated with Claude Code