Expert Caching that can boost performance, heatmap, mmap pinning based on heatmap, real time cpu to device transfer and device to cpu transfer. Fully conditional on flags and self contained to not affect execution path outside of them - #26824
miltos22 wants to merge 71 commits into
Conversation
hooks into build_moe_ffn and process_ubatch, new file llama-expert-heatmap does the actual tracking, minimal diff on upstream files. --expert-heat-decay and --expert-heat-log-period for config.
heatmap only allocated when log period or hot S is set, hot store only when hot S is set, defaults off
decay_all once per batch, not once per layer; log on crossed boundaries instead of exact multiples Assisted-by: Gemini 3.1 Pro
- add MUL_MAT_ID_COLD op that skips hot experts on the CPU cold path
- port cold kernel from original llama-wackMall
- add llama-expert-tier register/build hook (hot LUT remap + cold op + scale + add)
- slim layer_lut to {hot_lut, cold_mask}, drop cold_lut/hot_mask
- add static-plant and hit-rate diagnostics
- add ggml-cpu-mul-mat-id-cold.c with the cold-expert kernel - add shared header for mmid helpers (mmid_row_mapping, incr_ptr_aligned, one_chunk) - drop static on the two helpers now shared with stock mul_mat_id
Expert hot store is only correct on CUDA; on Vulkan it produces corrupt output. Restrict the tiered hot store to devices named CUDA*, and add LLAMA_EXPERT_HOT_FORCE=1 to override for testing.
--expert-hot-s N>0 auto-pushes the MoE-to-CPU tensor override (with a warning) so hot store slot copies always read host pointers.
fit.cpp now reports how many MoE expert bytes the fitted placement leaves on GPU; common.cpp maps that to S slots and forces all experts to CPU so the hot store copy reads host pointers. S=0 disables the tier.
Swap a resident only when a cold expert scores >= hyst x the incumbent (default 1.3). Add --expert-hyst/--expert-dwell, raise decay default to 0.999. Dwell counts real tokens and initial fill is eligible so the first sync can correct drift.
sync fork with ggml-org upstream (15 commits): vulkan shaders, new model templates, gguf reader hardening, sycl/vulkan fixes. no conflicts with the expert hot store feature.
--expert-heat-decay help said 0.99 but the default is 0.999; --expert-hot-s help now documents -1 = autofit from free VRAM.
|
Built this since I think my usecase is the perfect target audience - running Ornith 1.0 35B A3B Q6_K_XL on a 5060Ti 16GB. PP is very low, around of third of usual. TG seems to be faster, but I haven't done an exact A/B test yet. I estimate TG to be about 33% faster on my regular workloads. I'll be happy to help out posting stats somewhere else if this PR isn't the best place to minimize noise. |
|
This type of pr is not acceptable as i violates multiple rules, for example the one change per pr rule, the no changes in multiple back ends rule, vibecodeing etc. |
|
@miltos22 See? That's why I told you to create an Issue explaining the architecture first and then link to smaller PRs sequentially. Anyway, don't lose hope. This feature will take a long time to fully implement. A ton of feedback will be required. Just make it modular. Show the roadmap. Refine and test that first module thoroughly. You got this. |
Ok I get it. However I I am not sure how much time I can put into this as it has taken over my life for 3 weeks More crucially the only parts that were "vibe coded" were the fixing compiler errors parts so I'm also very deterred by that. I did use AI to discuss about code or change the order of thing or do them more efficiently for the rest but that was mostly it. It really feels like the assumption is made due to the size of the code to me. I think anyone looking at the old pr and my constant updates can likely tell I put in a lot of work directly. And I would legitimately want to see someone attempt to vibe code something like this as with the models I have access to i feel like it's impossible and even if someone used fable it may or may not have worked but it would probably cost more than an actual computer. Perhaps it was because some initial commits felt too fast? Which was because I already had code ready and was just more or less copying it to make the commit history cleaner |
Would you like to discuss with me how I could split up the work so it would have a chance? |
|
Not sure if it helps, but the PR really does help boost tg/s for me, it increases performance by about 70% for me on an RTX 4070 Laptop (8GB VRAM) + 16GB DDR5 system (i7 13700HX (no AVX512)) BUT the prompt processing got divided by 3 essentially (almost 4 actually :/), so there is some sort of a drawback, perhaps I will try cranking up slots and/or batch size.. |
That's the expected outcome of this. You are trading PP for TG speed. |
|
Didn't think it would be this bad though. And I dont see why it should be this bad in the first place... |
sounds like a solid plan |
I think that was due to a workaround I had introduced earlier before I managed to make parallelization work with it and I may have forgotten to undo. I'll have a look later. It should theoretically have the capability to do it on the GPU so it should be a relatively small scale fix However Making it work with router mode would probably be a bit more difficult as I made no consideration so far for it. So if that turns out to be easy too, it would be pure luck |
I would like to do this but I am not sure how I could ever cut it down. 90% of the changes depend on the others to offer any real benefit. |
Personally I would create the issue that shows the complete implementation and it's benefits. After that you start breaking it down into parts so that they can be merged one by one. Each part does not have to explicitly provide a direct benefit other than to create the structure for the whole implementation. As far as I understand the whole purpose to doing small PRs is to be able to merge changes without creating new bugs and unintended behavior. Simply break it down. Heck, you can even use whatever LLM you want to create a plan as to which parts should be separated into their own PRs and then spend the time reviewing that instead if it helps you. There's not real point into giving up now. If you need help getting this sorted you know where to find me. |
|
I want to knowledge something I did wrong. Yesterday I pushed and realized I could not sync automatically due to a conflict that arose from having applied manually the latest sync on my local PC. It was 4 AM so i just told AI to get it sorted without a second look. What it ended up doing was tracking the entire folder including 2 files, counter.md and manuallog.md These 2 files were an issue. First of all, they were days behind the actual development and did not represent the final state of the code as they were only included in my sandbox repos and though my convoluted copy paste replace application method to the real repo, they rode along. Second, despite the content of the files being just general architecture notes or notes from ai to ai keeping track of past bugs and times to argue about changing my implementation into something that was instantly proven to be worse, I get that including them accidentally was sloppy and gives off a terrible taste as to the overall quality of the work. So I would like to apologize for that. I made no attempts to hide that this was largely accelerated with AI and likely it would have taken me months rather than weeks without the usage of AI both debugging and testing as disclosed. I removed those documents now and I am sorry. Resolving the sync conflict was something I should have checked manually rather than just ask AI to sort out. My initial resistance to split it into multiple parts is simple. I don't think this is something that should be fully automated. And manually doing so can take days while I already felt burnt out from 16 hours a day of coding. However I decided to commit to doing it the proper way as I believe this is something that could be very beneficial. |
Since it was closed, we can talk about stats and potential changes before I commit to a final roadmap and start splitting changes up in smaller parts |
|
Will have a more detailed look but I feel just from a quick first impression like it fell on many traps I went out of my way to avoid. For instance worse performance on static experts than stock, (mine is within 10% even with heatmap disabled), cuda only, etc. However it most definitely predates mine. And I was unaware of it. So I may need to give a warning that I acknowledge it's existence. However it's clear that I am far less professional in both my presentation and the way I respond which I will also take note of |
I linked to that in the first comment of the original PR for this :) See #26563 (comment) |
Oh right. I will admit I confused myself thinking this was a different one you were pointing out than that comment. I know this may seem intentional but what actually happened is I simply responded first thing when I woke up so I'm a bit out of it. I even went back and checked if the number was the same and somehow I managed to missread it. I won't make a list on how our implementations different but they are only vaguely similar |
|
[AGENT] @miltos22 Thanks for this work. I tested PR head The build succeeds, but multi-GPU loading fails:
All four GPUs were visible and empty; there was no Xid or host OOM. This looks like the multi-device store is being sized globally but allocated through CUDA0. I have this rig available and would be happy to test a fix, including P2P transfers and prefill/decode performance. |
Points at expert_streaming.py and lists the 3 real commits this branch carries on top of upstream (PR ggml-org#26824 head, gpt-oss compat, GLM-4.7-Flash compat) -- so anyone opening this fork's README sees what changed and why before reading upstream's own content.
|
Added to performance comparison here . |
|
This can be the next big thing! Don't give up |
|
I came across your PR because I was doing research on doing the same thing! I have a mixed 5 GPU rig. I'm going to dig deeper into this. Keep up the good work, It's really hard getting a PR accepted here - I've tried. |
|
@miltos22 are you continuing your work in this area? |
Its complicated. Im trying to experiment on and off and have a few people who have proposed to collaborate but i really need a mental health break at the moment. So, likely, soon-ish |
|
All the best! I'll be keeping an eye out. I'm also digging into #27861 now. |
I know a guy who coded an entire backend with expert caching from scratch with claude. Yes, it's totally doable, just pricey. I suggest you to chime in into expert cache thread in discussions |
15 participants
This is the successor to #26563
People seem to want full performance, and other options I had not initially included. If a reviewer wants me to remove features, or apply some more "hacky"fixes and make a smaller diff version or a cleaner commit history, I am very willing to do both
Additions:
Quick start:
Use -ehs N to set number of expert slots. Never use in conjunction with -ncmoe as that takes away vram for static experts. In most cases its recommended to just set fit target. Otherwise you'll have to figure out the best value to set.
copy vs move mode.
Copy mode stores all experts on system RAM and only copies them over to vram. Takes up more RAM, but is faster to heat up
Move mode only keeps 1 copy of an expert at a time, moving them between your cpu and gpu. When run with -no-mmap (you cant copy experts back to mmap), it saves ram but slower to heat up
-expert-sidecar saves your heatmap next to your model file on exit or loads a previously saved heatmap if present. Helps a lot with initial speed.
-expert-pin N. Pins the N% of hottest experts on your mmap to your ram using willneed. early testing shows the optimal value is between 30 and 60 depending on model size. You can do 100 if you want to use heatmap but the model fits so you just want to have it set all memory as willneed. However pinning on its own has shown minimal benefit especially on well optimized linux distros.
Who is this for?:
Everyone can test it, but most benefit will be visible to people with 1 gpu, who have a model they can fit at least 40% of the model in their vram. Configurations outside of that can also provide a benefit but it can be hit or miss.
Known limitations:
Multi gpu performance. I have designed a swap system that in theory should not have overhead. However, I lack proper testing apparatus so multi gpu performance may not be optimal
Back-end support: Others have reported past versions working on other back-ends too. I hope thats the case. However, i am not rich. I only have access to 2 GPUs one of which is Vulcan only and old. So i can only test on Vulcan and CUDA.
Due to differences in floating point calculations depending on where each expert is calculated, even at temp 0, when different experts are cached, the output may slightly vary. I have run extended tests and confirmed this is not silent corruption and happens on the original llama too if you change "--n-cpu-moe"
High end hardware. I do not have access to any. So its impossible to optimise. Some have reported good results, some bad. But I would need to work closer with people to understand
Under 256 token generation sessions: When llama.cpp itself (client, server, etc) has processed less than 256 tokens it may be slower than stock. If you do not close and re open the client it will quickly start outpacing it. Benchmarks that generate 128 tokens and restart the server will produce uncharacteristically bad results that are not representative of real world performance.
While -np2 works, the slowdown will be bigger than on stock. So its probably better for -np 1, about on par with -np 2, and worse after that.
Warnings:
Auto-fit. Relies on stock autofit. If you have manually set -cmoe, -ncmoe, -ngl, etc etc, it will fail and you will have to manually submit a -ehs value.
If your cpu or PCI express buss is your bottleneck, it may worsen performance
The flags equivalent of -no-mmap uses a bit more ram than no mmap on the same configuration on stock (without the flags, or the pull request) due to the need for a buffer to move experts around without overwriting each other. I designed it so it only has a buffer on the CPU side so vram usage would not increase. The increase should be minor in most cases but worth knowing about
Full model fit: If the full model fits in your vram, this will purely cause slowdowns as it adds more cpu synchronization overhead. However if only 80% of the model fits, this may be the single biggest upgrade you could find.
Performance may be worse on vulkan or other back ends.
Things I have talked about but were excluded for this commit but can plug into existing system:
Semantic permanent heat maps + prefetch. Could be added in a future commit but is too big of a feature to include along with the rest here
Would benefit the most: People who want to use models drastically larger than their combined Ram.
Next expert predictive prefetch. It can actually cause a slowdown fow smaller models but for huge ones, it can be very beneficial.
For people who really want this: I hear you, I will be researching for the best way to do this with the least rewriting of original llama code over the next month
Full technical explanation:
Coming soon, I will go to sleep now. Excuse me for any possible multi hour delays in my responses.
Benchmark methodology:
Prompt: Give me a guide on how to set up a Linux computer averaged over 3 runs of each (the deviation was within 5% so i wont be putting them individually to increase readability)
The sidecar test was run having pre warmed a sidecar with the phrase "Hello" over 128 tokens. Sidecars that were generated over longer periods will perform significantly better especially when there is topic overlap.
The only launch argument for both was -fitt 64 + -ehs -1 for mine.
Hardware: 8gb laptop 3070 + 32 GB of ram + laptop ssd. System conditions identical with all other processes ended.
I initially did multiple runs on stock, but realized it made no difference (within 1-2% regardless of token length), so i stuck with 1.
##Benchmark Results (short)
Benchmark results (detailed)
These are not performance numbers under ideal conditions. What I did was run stock first, and others later, with clearing my rams cache fully in between. The late runs, which were always the tiered runs are thermally throttled. Under ideal conditions performance is 20% higher when not ram limited.
Note: It appears to be building normally on everything. But if someone encounters a platform specific error, I'll make sure to have it fixed by tomorrow. In the meantime you can get one of the pre made builds: https://github.com/miltos22/llama.cpp-wackMall-merge-request/actions/runs/31344712283#artifacts
Edit after PR closure: But this was NOT vibe coded and I am confident the results reflect that + There is an established timeline of me working on this over weeks. At most you could call it debugged/accelerated by AI
** I find the fact Deepseek V4 flash was the ai that helped me during this weirdly poetic given how a previous version of this allowed a user to get a ~50% speed increase on running deepseek v4 flash locally on his single 24 GB gpu on the old push request)