Skip to content

cuda: raise CUDA graph cache cap from 64 to 512 for tensor parallel #232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
angliubc wants to merge 1 commit into
base: mtp/qwen4exp-nextn
Choose a base branch
from
Open

cuda: raise CUDA graph cache cap from 64 to 512 for tensor parallel #232

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion ggml/src/ggml-cuda/common.cuh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1467,7 +1467,13 @@ struct ggml_backend_cuda_context {
#ifdef USE_CUDA_GRAPH
std::unordered_map<uint64_t, std::unique_ptr<ggml_cuda_graph>> cuda_graphs;

static const size_t max_cuda_graphs = 64;
// NOTE: tensor-parallel (-sm tensor, meta backend) models dispatch one subgraph per AllReduce
// split point to every simple backend (~2 per layer for qwen4exp -> >100 keys per device).
// The old cap of 64 made the LRU evict entries every token, permanently resetting CUDA graph
// warmup and forcing fully eager decode (V100 x8, qwen4exp Q4_K_XL: TP4 6.8 -> 44 t/s after
// this change; nsys: cudaGraphLaunch 0 -> 48888 over a tg64 run). 512 covers large TP graphs;
// the 10s idle sweep below still bounds memory for steady state.
static const size_t max_cuda_graphs = 512;

int64_t last_graph_eviction_sweep = 0;

Expand Down