cuda: raise CUDA graph cache cap from 64 to 512 for tensor parallel

[angliubc]

Summary

-sm tensor (meta backend) never captures CUDA graphs in decode when a model produces more than 64 subgraphs per device, because max_cuda_graphs = 64 makes the LRU eviction run on every token, permanently resetting graph warmup.

• The meta backend dispatches one subgraph per AllReduce split point to every simple backend. For qwen4exp (48 layers, 2 allreduces/layer) that is ~128 subgraphs per device, each keyed separately in cuda_graphs.
• With the cap at 64, every token's inserts evict the previous token's entries → graph->uid never matches → warmup never completes → zero cudaGraphLaunch, fully eager decode (~14.6k cudaLaunchKernel per token per device ≈ 70 ms of CPU launch overhead).
• Raise the cap to 512. The existing 10 s idle sweep still bounds the map in steady state.

Measurements

8× V100-SXM2-32GB (NVLink), Qwen3.8-Flash-Next UD-Q4_K_XL, llama-bench -sm tensor -ngl 999 -p 0 -n 64 -r 3:

config	before	after
TP4 tg64	6.77 t/s	44.41 ± 6.26 t/s
TP4 tg512	—	47.93 t/s (no leak over long runs)
TP8 tg64	5.50 t/s	30.37 ± 5.86 t/s

nsys (TP4, one tg64 run): cudaGraphLaunch 0 → 48,888, cudaLaunchKernel 2.96M → 118k, 1552 graph captures. Layer-split mode needs only one entry per device and is unaffected. Output correctness verified (greedy, temp 0: arithmetic, counting, generation all correct).

The capture storm also disappears: with 128 keys cycling through a 64-entry map, each token re-instantiated 64+ graphs; after the fix, captures happen once and cudaGraphExecUpdate replays them.

Notes

• Memory cost per entry is one node_properties per graph node (ggml_tensor + src ptr/ne/nb), so 512 entries × ~1000-node subgraphs is tens of MB — negligible next to the captured graphs themselves.
• I picked 512 rather than unbounded to keep a hard ceiling; happy to tune (256/1024) or switch to env override if preferred.

🤖 Generated with Claude Code