Benchmarks against our systems

[Haroldyolo]

Hey,

What is the best way to benchmark the models against our systems and see / report the outcomes?

[VecSzn]

The quickest way is the built-in benchmark. After setup, double-click BENCH.bat (Linux: ./bench.sh). It starts the model with your own strata-<model>.json, sends a fixed suite through the API (short chats, a 4K and a 16K prompt, and an agent session that reuses the conversation cache), and writes a report to bench/results/<date>-community-<gpu>-<model>/: a README.md with your GPU, CPU, RAM, driver and settings filled in, and a results.json with every request. Home folders, user and host names and IP addresses are taken out. It takes 3 to 10 minutes once the model has loaded; close games and browsers first, they change the numbers.

A few options: --config strata-<model>.json picks another installed model, --long adds a 32K prompt, and --clients 3 sends three requests at once (useful with "parallel" in the config).

To share it, post the README.md here under Show and tell, or open a PR that adds a row to docs/COMMUNITY_BENCHMARKS.md, which also explains each test.

[Haroldyolo]

Thanks, so It picked up the model I used last and reported fine (I will post the findings soon) however when running H:\LMs\PolyStrata> .\BENCH --config run-qwen36-ud-q3_k_m.json it says No PolyStrata model is set up in this folder: run START-HERE.bat (Windows) or ./setup.sh (Linux) first, or give --url of a running server (even though the model is setup in this folder) - any thoughts on how to run the rest of the models?

[VecSzn]

--config takes the model's run config, which is named strata-<model>.json; run-<model>.bat is the start script setup writes next to it. So for that model:

.\BENCH.bat --config strata-qwen36-ud-q3_k_m.json

dir strata-*.json in the PolyStrata folder lists every installed model's config, and each one runs the same way, e.g. .\BENCH.bat --config strata-gemma4-ud-q4_k_m.json. The error message is misleading when the file named isn't there; I'll change it to say so and list the configs it found.

Looking forward to the numbers.

[Dampfinchen]

Using this thread as a general way to share benchmarks now.

Gemma 4 26B A4B QAT It on RTX 2060, 32 GB RAM, i7 9750H. 555 Experts on GPU. VRAM Reserve Budget at 120 MB, max context 64K.

At this level of context (18K) it has almost double the prefill (900 token/s vs 470 token/s) and decode (30 token/s vs 16 token/s) compared to llama.cpp.

The only drawback compared to it is that as the context grows, it slows down quite a lot until I reach llama.cpp decode speeds at around 85K context again. Prompt processing takes a few short pauses every time after it has processed a few 1024tk/2048tk chunks. Llama.cpp is steady.

This engine has a lot of potential!

[Dampfinchen]

Thanks I added it to docs/COMMUNITY_BENCHMARKS.md it does slow down because the attention cache grows with the chat and takes up VRAM from the experts but --kv int8 halves it for Gemma 4 so worth a try for long context

I see, so as the context grows, it reduces the number of experts on the GPU dynamically? Did I understand that correctly? That does make sense then. Int8 I'd rather not use for Gemma 4 since it is very sensitive to KV cache quanting even at q8. (though it is less of a problem for the recent QAT variants, which I have been using and should IMO be added to the launcher options as they are smaller and higher quality than q4_k_m, I had to rename mine for it to work.)

Great job however. And I assume the pauses for the prompt processing also have something to do with the lower amount of experts on the GPU when using bigger context sizes.

[VecSzn]

yes, when not every expert fits on the card the attention cache grows as the chat grows and takes from the expert cache, so you have less experts on the GPU as the chat grows. those pauses are probably the same thing, like every 4096 tokens it maps more VRAM for the cache and experts get moved out. and with 1024 or 2048 token chunks that's every 2 to 4 chunks. ok, no int8 for Gemma 4 then, planning on adding the QAT file to the setup so you don't have to rename it.

[Dampfinchen]

yes, when not every expert fits on the card the attention cache grows as the chat grows and takes from the expert cache, so you have less experts on the GPU as the chat grows. those pauses are probably the same thing, like every 4096 tokens it maps more VRAM for the cache and experts get moved out. and with 1024 or 2048 token chunks that's every 2 to 4 chunks. ok, no int8 for Gemma 4 then, planning on adding the QAT file to the setup so you don't have to rename it.

Is there a command that allows me to have the attention cache fixed for the context size I want? That would also get rid of the pauses since nothing has to be moved. I believe regular Strata and Llama.cpp handle it that way since increasing context size also decreases the amount of experts. Having this as an option would be nice, if it's not there already.

That's great, I am using the unsloth one which is more popular but some might also use the Google qat.

By the way, is there a way to turn off mmap to make use of pinned memory?

[VecSzn]

Yes, just add "--kv-whole" to the "args" list in strata-.json, and the attention cache will get VRAM for the whole context at load time, so nothing will be moved during the chat, but fewer experts will fit on the card from the start. The unsloth QAT is what I was planning to add. There's no mmap switch on strata-poly yet, it just reads the experts from the mapped file, so I'm planning to keep the experts in pinned RAM.

[VecSzn]

the unsloth QAT file is now in setup you can just run setup after UPDATE.bat and use --model QAT-UD-Q4_K_XL, no renaming needed