Repository navigation
[Engine] Layer-major prefill on a CUDA layer split: 32K prompts 2.0x faster, 0.47x drive reads (Qwen3.8-Flash-Next, v0.2.4.2) #3976
Description
Summary
Layer-major prefill was proposed in #1670 by @vizdatom for one GPU, and that issue was closed. 0.2.4 (and 0.2.4.2) has a layer-major prompt path only in the DeepSeek engine (src/dsv41), also for one device. The Qwen path (src/prefill/prefill.cpp) has no layer-major mode yet.
I extended @vizdatom's layer-major prefill to the CUDA layer split:
- Each stage runs its own layers layer-major over the same super-chunk.
- It then hands the super-chunk's rows to the next stage.
- The next stage starts on super-chunk k while the first stage reads super-chunk k+1, with one hand-off buffer.
On a 2-GPU split (RTX 3060 + RTX 5060 Ti) with a streamed UD-Q4_K_XL, compared with official v0.2.4.2:
| prompt | official 0.2.4.2 TTFT | layer-major split TTFT | drive reads |
|---|---|---|---|
| 8K | 28.9 s | 18.5 s (1.6x) | 25 → 19 GB |
| 20K | 54.2 s | 23.5 s (2.3x) | 61 → 18 GB |
| 32K | 65.0 s | 33.1 s (1.96x) | 75 → 35 GB |
(warm runs, i.e. the second run of each size; same prompts in both arms)
Quality is unchanged against the official v0.2.4.2 build on the same split (and the same as on 0.2.2):
- T4 needle: 9/9 in both arms, at 4K / 16K / 32K × depth 10 / 50 / 90 %.
- HumanEval 0–39: 39/38 base/plus in both.
- 12 Japanese questions: 11/12 in both, the same item fails.
- Tool calls: 8/8 in both.
Decode speed is the same within run-to-run noise: on 0.2.4.2, median 24.0 tok/s official and 24.1 tok/s with this change (on 0.2.2: 27.7–29.7 and 27.8–28.2). Decode does not use this path.
Everything is opt-in. With STRATA_PREFILL_LAYER_MAJOR unset, the engine runs the official path.
Credit
The layer-major prefill itself is @vizdatom's work from #1670, branch layer-major-0.1.43.1: commits 2e69b6db, f0173382, 149b8995 and the stager / warm commits.
I ported it to 0.2.2 and then rebased it onto 0.2.4 and 0.2.4.2. The port includes:
- an FP16 expert path for layers that MMQ does not take (all 48 layers of UD-Q4_K_XL);
- the
STRATA_HC_Q8dequant; - guards for APR, NVFP4 and WMMA layers.
The layer-split part below is new.
What the change does
- Layer-major on a split (
run_layer_major,lm_prepare_chain):- Every stage gets the same super-chunk size: the smallest that all stages fit.
- The first stage reads super-chunk k for its layers and writes the rows into the next stage's pinned
R_host. - The next stage runs on a thread (
next_run_). The first stage meanwhile reads super-chunk k+1, so there is one hand-off buffer. - The PLE layer must be in the first stage, which holds layers 0 and 1.
- Even super-chunks (
STRATA_PREFILL_LM_EVEN, default on when layer-major is on):- A prompt longer than the common super-chunk is split into equal parts. For example, 32K at a 28K fit becomes 2 × 16K, not 28K + 4K.
- This keeps each stage's residual rows in VRAM and keeps the stages overlapping.
- Without it, at
--vram-reserve-later-mib 1700, 32K went 28672 + 3271: the stages ran almost one after the other and TTFT rose from 30 s to 36–40 s. - The idea is similar to
--prefill-pipe(Prompt chunks sized for a layer split's pipeline (--prefill-pipe, opt-in; b/a from--calibrate) #1441), which evens the chunk-major path's chunks. Here--prefill-pipe 1gave no gain on the official path: its 32K chunks are already 4 × 8192.
- A stale hand-off no longer disables layer-major:
run_implleaves a later stage'shand_in_set after a chunk-major prompt.- Before this fix, one short prompt (under one sub-chunk) turned layer-major off for every later prompt of the session, with the message "a layer split's chunk hand-off is in use".
lm_prepare_chainnow clears it when no hand-off is in flight.
Why it helps
- With
--prefill autothe official path reads a 32K prompt in 4 chunks of 8192. Each chunk streams every layer's routed experts again. The plan line itself says "about 105.8 GB read from the drive per 32768-token prompt (4 chunks)". - Layer-major reads each layer's experts once per super-chunk. On a split, the stages' expert reads overlap.
- [gfx1200, 2x RX 9060 XT layer split] prompt-read stalls (watchdog #29 aborts) + run-to-run prefill degradation: fresh prompts slow 3-4x mid-run, file tier re-reads 24-35 GB per request #1407 reported the same re-read pattern on a layer split.
Setup
- i7-7800X, 62 GB RAM, Fedora 44.
- GPUs: RTX 3060 12 GB (PCIe Gen3 x16) runs layers 0–23; RTX 5060 Ti 16 GB (Gen3 x8) runs layers 24–47.
- Model drive: Solidigm P41 Plus, Gen3 x4, about 2 GB/s O_DIRECT.
- Qwen3.8-Flash-Next UD-Q4_K_XL (native).
- Common flags:
--resident-budget-gib 38,--expert-cache auto,"layer_split": "24",--trim-stage-weights,--kv int8,--max-context 40960,--spec 4with MTP,STRATA_HC_Q8=1, tail skip on (default). - Official arm:
--prefill auto,--vram-reserve-later-mib 1400. - Layer-major arm:
STRATA_PREFILL_LAYER_MAJOR=1,STRATA_PREFILL_LM_TOKENS=32768,--prefill 4096;--vram-reserve-mib 1700 --vram-reserve-later-mib 1700for the tables, or1700 / 1400(same speed at 32K).
- Method:
- Before each arm, the model's pages are dropped from the page cache and the server is started.
- Prompts are fresh word-list prompts with a per-arm nonce, the same in both arms; each size is run twice.
- Drive bytes come from
/proc/diskstats. - The results were reproduced four times with different prompt sets: on 0.2.2 twice, then on 0.2.4 and on 0.2.4.2.
| run | 32K official | 32K layer-major split |
|---|---|---|
| 0.2.2, set 1 | 57–62 s, 82–85 GB | 30–32 s, 34 GB |
| 0.2.2, set 2 | 59–61 s, 84–90 GB | 29.6 s, 35 GB |
| 0.2.4 | 61.6 s, 86 GB | 30.0 s, 36 GB |
| 0.2.4.2 | 65.0 s, 75 GB | 33.1 s, 35 GB |
Limits and open points
- Tested only on CUDA, on one 2-stage split, with one model and quant.
- Not tested: HIP, SYCL, more than 2 stages, the peer-GPU tier (it is declined), or prompts over 40K.
- The 3060 stage (about 19.5 s of a 32K prompt) is now the limit.
- A second hand-off buffer, or moving the layer boundary, may help further.
--layer-split autocosts layers by decode time, not prefill time.
--prefill 4096is used for the sub-chunk. Prompts under one sub-chunk stay on the chunk-major path, at the official speed.
The branch is a series of 12 commits on top of v0.2.4.2: Norhiro/Strata@dfa3f40...layer-major-split-0.2.4.2 (the first 4 are @vizdatom's #1670 commits, cherry-picked with authorship kept). I can open it as a PR if you are interested.
Activity
oscarlius commentedon Oct 25, 2026
We have a related workload on Qwen3.8-Flash-Next IQ3_S, four RTX 4090s, four stages and 16 slots. Our recorded 128K run had a median cold prompt read of 20.453 s; a 16-request wave (15 cold, one warmed) had TTFT p50 of 164.776 s. The first tokens arrived roughly one prompt-read interval apart. Stage-copy improvements in #4002 help cache hits, but the cold-admission queue remains.
Before extending the work by vizdatom and Norhiro here: is an opt-in Qwen split layer-major prefill path within the direction you want to maintain?
The proposed boundary is one prompt at a time, with the existing chunk-major path unchanged by default. It would preserve MTP/on_chunk and BYIELD state, fall back for incompatible checkpoint boundaries or concurrent-reader mode, and cap temporary device/pinned memory. This would reduce individual read cost, not introduce multi-sequence admission.
We have only a partial local port, with no new engine build or hardware validation and no performance claim. If this scope is useful, we can narrow the patch around your preferred interface and supply bounded same-hardware correctness/resource/performance comparisons. If the preference is to retain chunk-major Qwen prefill, we will keep this experiment out of a new PR and focus on the existing cache-copy work.
oscarlius commentedon Oct 25, 2026
We opened a separate diagnostic report in #4069 to keep the observations and validation limits distinct from this layer-major proposal.
On the ordinary chunk-major path, two fixed-config 130,658-token, prefill=2048 A/A requests matched at three early state checkpoints (first-token logits NRMSE=0, KL=0) and returned identical output, but had different per-verify timings and a later recorded suffix/MTP path difference. We stopped before the 8192 or MTP-off arms. The report includes the matched-window data, source/binary identity, instrumentation caveats and measurement gaps.
This was an instrumented older baseline, not current main/v0.2.4.2 and not your layer-major branch. It neither validates nor disproves the speedup here, and does not establish a shared root cause. Linking only as validation-method context; no fix is claimed.
prefill stager: a job waits for the previous job on its buffer; skip() (
prefill: the per-layer walk warms this layer's and the next layer's f…
prefill: kernels for a layer-major expert pass (moe_shared_init, moe_…
prefill: layer-major prefill (STRATA_PREFILL_LAYER_MAJOR=1) (#3976, #…
layer-major prefill: STRATA_HC_Q8 dequant, guards for APR/NVFP4/WMMA …
layer-major prefill: the FP16 expert path for layers MMQ does not tak…
layer-major prefill: the FP16 path needs no MMQ identity table or tok…
layer-major prefill: no MMQ row table without an MMQ layer; QSA shape…
layer-major prefill on a CUDA layer split: each stage runs its layers…
layer-major prefill on a split: the PLE layer is fine in a first stag…
layer-major prefill: even super-chunks (STRATA_PREFILL_LM_EVEN, defau…
layer-major prefill on a split: a stale hand_in_ left by a chunk-majo…
layer-major prefill: stays off on AMD builds (#3976)
14 remaining items
coryjett commentedon Oct 28, 2026
@udqe thanks, the floor arithmetic was right. With
--prefill 8192andSTRATA_PREFILL_LM_MARGIN_MIB=96the walk engages on our pair:At the default margin (512) the same prompts still decline at 1,034-1,122 MiB free, as you computed.
Engaged, it halves prompt reading here. Same server, same hour, 2 runs each (2x RTX 3090 on PCIe 3.0 x16, one per socket, 0.2.7 / engine 0.1.52, IQ3_S, split 25 with
--trim-stage-weights, ~78% of the experts resident,--prefill 8192,STRATA_SPLIT_RING=384,STRATA_SPLIT_MTP_BATCH=1):So on a PCIe 3.0 pair the walk costs rather than pays: the later stage keeps its rows in host memory and moves them over the link per layer, which is twice as expensive here as on the PCIe 4.0 pairs it was measured on. Your 2x and the 5x on the A4000s are not in doubt; the rule just needs the link in it before it engages by default. Raw files (engine logs with the walk lines, per-run outputs) are in our benchmark log (section 30); I can attach the two engine logs here if useful.
udqe commentedon Oct 29, 2026
@coryjett your run confirmed the arithmetic, and taking the hint we went one step further to make the walk engage on our box for the first time - then ran a single-variable link test. Two corrections and one table.
Correction first (our previous comment, now edited): our 2x 4090 had not one engage line before this morning - CUDA0 passed at margin 96 but the last stage always declined (the
layers 22..48 ... (734 MiB free)line), so every prompt fell back to chunk-major and our 3,239.7 tok/s "engaged" figure was chunk-major. The evaluator is per-stage (as above); the fallback is whole-prompt.What engaged it:
--vram-reserve-later-mib 972 -> 1500(stage-1 free 734 -> ~1260, room ~1160 >= the ~1.1 GiB floor). First engage lines in our log:The lever is the later card's reserve - the same last-card gap @cobra91's #4549 (keep the draft head's room out of the last stage's cache) fixes from the inside, and one more data point for per-card reserves (#2522).
Single-variable link test - same machine (2x 4090, single-socket TRX40), same config, same salted prompts, only the BIOS slot setting changed; cold 32K/66K/150K x2 each, median
prompt ... read inrate; margin 96 in all four arms, walk arms at later-reserve 1500 and chunk arms at 972 (which makes the comparison conservative against the walk):Decode held 120-150 tok/s in all four arms - the link cost is prefill-only, as you saw.
What this adds to your rule proposal: on a single-socket pair the walk degrades with the link but barely flips sign (-1.6% at 150K on Gen3); on your dual-Xeon pair (one card per socket) it collapses (-49%) - so beyond "the link must be in the rule" the route the R_host rows take matters, and a measured per-card gate is the right shape. The engine already probes each card's link at startup (26.4 vs 13.2 - it distinguishes the generations), though we saw the probe read ~4x low (6.7) twice when it caught the link at its idle Gen1 negotiation, so a gate would want a re-probe or a floor, not a single sample. (Edit: the Gen1 label was wrong and the trigger is now pinned - the misread fires on the FIRST start shortly after resume-from-sleep, always the display-attached card, always exactly half the true value: #4610 #issuecomment-6532493954. The floor/re-probe advice stands, strengthened.) A cheaper variant: run one super-chunk both ways at startup and pick the winner - the check is one prompt.
(AI-assisted again on our side - the model served by this repo's engine 0.1.52 ran the measurements and the log audit; all figures local.)
coryjett commentedon Oct 29, 2026
@udqe the route was a good guess, and it's part of the answer. We ran it on our pair this morning (2x RTX 3090 on PCIe 3.0 x16, card 0 on node 0, card 1 on node 1, dual E5-2620 v3, 0.2.7 / engine 0.1.52, IQ3_S, split 25 with trim,
--prefill 8192, layer-major engaged at margin 96).Pinned copy bandwidth per card and node (256 MiB, cudaMemcpyAsync x10):
We run under
numactl --interleave=all, and the arena andR_hostare plain pinned allocations, so they follow it (the engine says as much: "memory is not placed per node (known limitation)"). Switching the whole process to one node, 2 runs each, per-stage times from the walk lines:R_hosteach layer.R_hoston its card's node.So for the gate: a per-card measured probe would catch the cross-socket cost as well as the link generation, provided the probe's buffer sits where the stage's memory will sit. Raw logs and the copy test are in our benchmark log (section 32).
cobra91 commentedon Oct 29, 2026
A data point from a single-socket pair (different from the dual-Xeon above): RTX 5080 + RTX 4060 Ti, both 16 GB, the Ti in a chipset slot, Ryzen 7 9800X3D, 32 GB RAM, Windows 11, engine 0.1.53 built from main at 46c152a, IQ2_XS, 64K context, q4_0 KV,
--prefill 8192.On this rig the layer-major walk also falls back, but the blocker is free VRAM on the later card, not memory placement. Three configurations, same 49,361-token prompt, engine-log rates:
--vram-reserve-later-mib 1500The trim frees 2.86 GiB of dense weights and the expert caches grow from ~6,800 to ~8,200 + 7,500 slots, which is where the prefill gain comes from (fewer expert blobs read from the SSD). But the caches absorb the freed VRAM: the last card ends at 420 MiB free either way, and the super-chunk's R rows need ~1.1 GiB there, so the walk still declines (
layer-major off for this prompt: no VRAM for a super-chunk). Raising the later reserve to 1500 did not free enough (437 MiB - other allocations eat the reserved space) and shrank the cache, which is why prefill went down.So on a single-socket pair the engagement gate could be a VRAM budget in the cache sizing (leave the super-chunk's R rows out of the last stage's cache, the way #4549 does for the draft head) rather than a link or placement probe. We have not measured layer-major engaged on this rig, so we cannot say what it would give here.
udqe commentedon Oct 29, 2026
@coryjett @cobra91 a data point from the single-socket arm of this picture: 2x RTX 4090, TR3970X single socket, 128 GiB, UD-Q4_K_XL, engine 0.1.52, full matrix in #4573. Your decomposition matches ours from the other side:
The one honest outlier left on the board: your best walk arm is still ~2x behind chunk even on the local placement. One suspect we can now rule out is generic memory-side saturation: we starved our rig's memory side (4 pinned-copy workers, about 32 GB/s sustained against the engine, engine otherwise untouched) and chunk@150K fell 24.0% while walk fell only 17.7% - walk still beat chunk under contention, 3,099 vs 2,564 tok/s. So the residual is not "walk is DRAM-hungry". What is left points at the attention-row feed rate itself: our Gen4 walk hides the per-layer rows inside 3.0 s of "waiting for the attention halves" per 24,576-token super-chunk at 26 GB/s pinned copy, while your card feeds at 10.1 local and 4.4 remote - the same wait term turned into wall time by the card's own copy rate rather than by DRAM. That also predicts your residual would shrink with Gen-class copies on the same dual-socket board, if you ever get a second rig to check it on.
fioruccione commentedon Oct 29, 2026
@udqe thanks for the source reading, that settles the 311-353 MiB line as CUDA0.
One data point for the gate, with no new runs: on this pair the cards are PCIe 4.0 but x8 (the RTX 4060 Ti is x8 by design), and the engine measures them like your Gen3 x16 slot. Every one of 28 starts on v0.2.7.1 logs
layer split auto: CUDA0 link 13.3 GB/sandCUDA1 link 13.3 GB/s, and the PCIe probe gives 13.3 GB/s best of four (11.6-13.3). None of the 28 read low, so I didn't see the idle-link reading you got.So a gate based on the PCIe generation would put these cards on the Gen4 side, while by your Gen3 x16 numbers the walk would lose or break even here. A gate on the measured link, which the engine already has at startup, would treat them correctly. For that reason I'm not going to run the extra CUDA0 reserve arm: on this pair it would take VRAM from the experts for a walk that, by your table, wouldn't pay.
I did this with an AI coding assistant: it read the logs and the thread and drafted this text. I checked the numbers.
agurrrrr commentedon Oct 29, 2026
Note
AI-assisted — an agent drafted this text from our own engine logs and from this thread. Every number below was measured on this machine and each one is quoted from an engine log line in the Evidence section. Nothing here is copied or modelled.
V100×2 field data: the walk engages on every long prompt and still loses
@Niko1221 asked 0.2.7 testers to try their own cards. Here is our pair. This is a third shape on the board, different from @coryjett's dual Xeon (NUMA placement) and from @cobra91's last-card VRAM block: on this box the walk engages for every long prompt at stock settings, and it is slower than chunk-major. No gate tuning was needed to make it run. It runs and costs time.
Problem
On 2× Tesla V100-SXM2-16GB (
sm_70) in a CUDA layer split, the 0.2.7 default (layer-major prefill on) reads a cold prompt 26 % slower at ~9K tokens and 10 % slower at ~37K tokens than the same engine binary withSTRATA_PREFILL_LAYER_MAJOR=0. One variable changed, same day.Environment
46c152a1) + 5 local commits;strata --version→strata 0.1.53git diff --name-only v0.2.7.1..HEADtouches nothing undersrc/prefill,include/strata/prefillorsrc/kernels(onlyserve/server.py, slot logic insrc/program/generate.cpp, aSTRATA_HIP_GFX906guard insrc/core/dflash2.cpp, apps and docs)sm_70), driver 580.178.04, built against CUDA 12.8, both PCIe Gen3 x16 (nvidia-smi)--layer-split 23(CUDA0 layers 0–22, CUDA1 23–47) ·--expert-cache auto·--prefill 2048·--spec 4 --spec-min-p 0.5·--kv int8 --kv-resident 32768·--max-context 262144·--vram-reserve-mib 1200·--pool-workers 27·--pcie-frac 0.1·--shared-expert-arena /dev/shm/strata-iq3_s.arena·STRATA_SM70_TABLE=1·STRATA_STAGE_TRIM=1layer split auto: K=23 … the caches hold 9351 of 24576 profiled pairs (~97.1 % of the routed mass). At the measured explicit split: CUDA0expert cache 5327 slots, 9.57 GiB, CUDA1expert cache 4448 slots (8.89 GiB), and the decode hit rate in production sits at 87–93 %.prefill.cpptreatsSTRATA_HIP_GFX906as HIP)Reproduce
STRATA_PREFILL_LAYER_MAJORunset. Startup printslayer-major prefill is on for this layer split.max_tokens 16,temperature 0). One pass sends one ~9K-token and one ~37K-token system prompt.STRATA_PREFILL_LAYER_MAJOR=0in the engine environment and send the same two sizes again.strata serve: prompt … read in … (… tok/s)line for each request.We dropped any pass that overlapped live router traffic. The image-prompt decline is real and separate:
layer-major off for this prompt: an image prompt.Actual (engine log lines, both arms)
on= the 0.2.7 default,off=STRATA_PREFILL_LAYER_MAJOR=0:Raw lines for the ~9K pair:
Raw lines for the ~37K pair:
The two arms differ only in that one environment key. The token counts are not identical across arms on purpose — each prompt carries a fresh salt — and the rate is the engine's own per-token read rate.
Cross-check against live traffic. On the previous engine (0.1.47, chunk-major only), cold reads from token 0 in real traffic had medians of 836 tok/s at 8–16K (127 requests) and 923 tok/s at 32–64K (238 requests). The
offarm lands on those same numbers, so theoffarm is not a lucky run and the box has not drifted.Why the walk engages here, and where it loses time
Engagement is not the problem on this rig. With
--prefill 2048the super-chunks fit at the default margin on both stages, with 829–882 MiB free:Pairing those stage lines with their prompt line gives the PCIe bill directly:
on)12168.0 + 18313.7 MB)on)54,838 MBover 3 super-chunks + CUDA187,883 MBover 5)The walk's own timing puts almost all of that wall time in
waiting for the attention halves. On the chunk path the same box moves almost no expert bytes at all: the caches hold ~97 % of the routed mass and the KV stream hits VRAM at 99 %+ (KV streaming: 99.83 % of 753540 block reads hit VRAM, 5.3 MiB read from RAM).Our reading, flagged as unconfirmed static analysis: this box has little expert traffic left for the walk to save. Chunk-major reads cached experts and barely touches PCIe; layer-major sends each layer's expert blobs once per super-chunk instead, so with ~97 % residency the walk can only add a PCIe cost. Upstream's win was measured with experts mostly streamed from RAM (2× RTX A4000, 16 GB each), where the re-read saving is large. That fits @udqe's link-shaped picture from the other side — our cost looks like those 30–139 GB divided by a Gen3 x16 link. We did not profile it, so the cause is a hypothesis, not an observed fault in the walk.
Two points for the gate
@fioruccione, @cobra91 and @coryjett are right that a measured per-card gate is the right shape. Our data adds two things:
the caches hold 9351 of 24576 profiled pairs (~97.1 % of the routed mass).--layer-split 23, and none of them prints aCUDA0 link … GB/sline — those lines appear only onlayer split autostarts. So a gate built on the startup link probe would have no sample to read on rigs like ours that pin the split by hand.Validation
strata --version→ 0.1.53;git diff --name-only v0.2.7.1..HEADshows no change undersrc/prefill,include/strata/prefillorsrc/kernels; the live-traffic cross-check on engine 0.1.47 (365 requests).--batch;STRATA_PREFILL_LM_MARGIN_MIBand--vram-reserve-later-mibtuning; prompts above ~37K in both arms (this box reads 130K cold at 905.8 tok/s on the chunk path, but we did not run that size in both arms).What we do now
We run
STRATA_PREFILL_LAYER_MAJOR=0in the V100 instance's environment and keep the chunk path. We are not asking for a default change; we only want to change a default where a measurement supports it, so we are reporting our numbers instead. If it helps, we can run a longer A/B — decode included, and 66K/130K in both arms — on request in this thread.udqe commentedon Oct 29, 2026
Two knobs from the closed benchmark reports transferred cleanly to our pair; one did not, and the failure mode is a design input for the gate @cobra91 is proposing. Same rig as our matrix above (2x RTX 4090, TR3970X, 128 GiB, UD-Q4_K_XL, engine 0.1.52), same discipline: salted fixed prompts, engine-log rates, adopt-rule stated up front (adopt if any axis >= +3% median, no axis <= -3%).
STRATA_ARENA_PIN_GIB24 -> 40 (whole 33 GiB arena pins)STRATA_STAGE_TRIM=1withlayer_splitfixed at 22The trim result is the gate argument made concrete. Slots went 4,275 -> 5,069 and 4,889 -> 5,739 (+17-19%) - exactly the cache-budget mechanism cobra91 describes, on a box with twice the VRAM of his. But the win only exists with a fixed K, and the reason it only exists is a trap we hit first-hand: with
layer_split: auto, the first card is never trimmed - and it is worse than that, auto MOVED our split on its own today (22 -> 23 -> 22 across three same-config starts, we caught it because it silently confounded a different A/B). A gate or a bench that reads "auto" has no stable subject to measure; every recommendation written against auto should carry the K it assumed, and the standard-report proposal in #4610 is the right place to pin that fact down. agurrrrr's second point is the same blind spot from the gate side: an explicit--layer-splitprints no startup link line at all (their V100 box confirms ours), so a gate keyed on that probe is simply blind on rigs whose owner pinned K.pipeline-windows is box-dependent, and the engine states its price. The engine logs its own tax on our box:
--pipeline-windows 2: 265 MiB of CUDA0 kept out of the expert cache. Our first measurement of it was confounded by the auto-drift above; the clean single-variable arm (K=22 pinned, placement line identical) says pw2 BEATS pw0 here: decode median 123.6 vs 114.8 (+7.1%), prefill equal. #4603 measured the opposite sign (-14%, same layout: draft head on the last card). So a rule like "pw2 is off when the draft shares the last card" would be wrong for exactly the two boxes that published data - the honest version is that it is a per-box trade between the cache tax and the window overlap, which puts it in the same probe-or-measure bucket as the link discussion, not the generation-rule bucket.agurrrrr's report adds the second axis a gate has to read. Their walk loses on Gen3 (-26% / -10%) with the experts 97.1% resident - the chunk path streams almost nothing there, so the walk can only add cost. Our pair is the same-axis control with only the link moved: same machine, same caches, Gen3 -1.7% against Gen4 +16.1%, and our walk arms stream ~30 GB per 32K-token super-chunk (real mass worth saving). Read the two boxes together and the sign is a function of link rate AND streamed-mass share, not either alone: the planner's
caches hold ... % of the routed massline is the right residency input, and the walk's own per-super-chunk streamed MB is the number a gate would need for the other half.Caveats: single-session arms (day drift), decode probe's first iteration after a long prefill runs ~10 tok/s low (medians absorb it), and pin 40 on our box equals "pin everything" because the arena is 33 GiB - #4603's whole-arena arm regressed on their pack, so read our +3.4% as pack-specific, not a general "pin more" rule.
(AI-assisted as before: the model served by this repo's engine ran the arms and wrote this; every number is an engine-log line from the machine described.)
fioruccione commentedon Oct 29, 2026
A third data point for the residency axis, from my 2x RTX 4060 Ti 16 GB (PCIe 4.0 x8, engine measures 13.3 GB/s per card, 1 NUMA node), v0.2.7.1 (
46c152a1, engine 0.1.53), UD-IQ4_XS,--layer-split auto. I only read my engine log, I ran nothing new for this.K=23 ... the caches hold 7877 of 24576 profiled pairs (~96.6% of the routed mass)(7835-7899 across the K=23 starts, 96.5% on the two K=24 ones). The link is also in the same range as their Gen3 x16.layer-major prefill is on for this layer split, but in the log since 28/10 (31 starts, production config plus a few single-flag tests) there are 0layer-major (layers ...)lines and 57layer-major off for this prompt: no VRAM for a super-chunk, with 238-434 MiB free, plus the image-prompt declines. Cold prompts went up to 192K tokens. So on my box the VRAM check already keeps it on the chunk path, which by agurrrrr's numbers is the faster side for this residency. I can't say what the walk would do here, because it never started.--layer-split 23have noCUDA0/CUDA1 link ... GB/sline; all 27 auto starts have both.--vram-reserve-later-mib 1100, so a different config, not drift. That doesn't contradict @udqe, it just says the drift is not on every box, and it's one more reason to write the K that was used in the report ([Benchmark] record the host side in the standard report: measured PCIe per card, NUMA, memlock, GPU temperature and power #4610).My opinion, not measured: on a link like ours, with most of the routed mass already in VRAM, the chunk path we have now is the right one, and anything that costs expert cache slots to save PCIe traffic doesn't pay here, because there is little traffic left to save.
I have not tried
STRATA_STAGE_TRIM=1with a fixed K yet; this is my production instance, and I'd run that A/B only as a separate test.I did this with an AI coding assistant: it searched my engine log and drafted this text. The numbers are from my log and I checked them.
cobra91 commentedon Oct 29, 2026
A small follow-up for anyone who wants to reproduce the trim numbers from our earlier comment: the mechanism is PR #4522, standalone against current main. It extends the per-card weight trimming that main already has for explicit splits to
--layer-split auto(and adds CUDA0's trimmed reload, which main never does). The prefill gain comes from the freed dense-weight VRAM going to the expert caches — on our rig the caches grow from ~6,800 to ~8,200 + ~7,500 slots, which moves most expert reads from the SSD into VRAM. No code from the fork's other changes is needed for this piece.cedrik-fuoco commentedon Oct 30, 2026
I think I found a correctness bug in this path, and it only appears when
STRATA_HC_REQ8is enabled. The quality evidence in this thread was collected without that switch, so nothing below contradicts it, but it does mark where the walk stops being reliable.I have been testing the walk on a single RTX 3070 Ti with both the Q2_0 pack and IQ3_XXS, using a simple needle probe: a six-character code placed a short way into the prompt, the question at the end, greedy decoding, and a check for whether the code appears in the answer. With
STRATA_PREFILL_LAYER_MAJOR=1andSTRATA_HC_REQ8=1it never appears, at 1,567, 4,045, 12,037 or 100,003 tokens, and the failure is not subtle, since the model replies in Chinese or asks about "the image you provided" as though it had never been given the prompt. WithHC_REQ8turned off the same prompts answer exactly.I could rule out everything except that switch. Both packs behave the same way, and the failure reproduces at the walk's minimum size of one super-chunk, so nothing about prompt length is involved. Moving the needle to the start, middle or end of the prompt changes nothing either. It is not the expert path as well: with
STRATA_PREFILL_MMQ=0, which puts every layer on the FP16 path, it fails identically.The cause turned out to be small. The three hyper-connection
bf16_projcalls inrun_layer_majordo not pass the optionalstreamargument, and underHC_REQ8those projections are stored as int8, whichbf16_projexpands into a static scratch buffer on whichever stream it is handed. Without the argument that is the default stream, while the GEMM that reads the buffer runs on the prompt path's own stream, so the GEMM reads the matrix before it has been expanded.run_implpassesm.csat the equivalent call sites and the layer-major path does not.Passing the stream at those three call sites is the entire fix, and I have opened #4715 with it. The PR also adds a small test for the expansion primitive, since
bf16_projis file-static and no test can reach it.Hardware and settings:
-DCMAKE_CUDA_ARCHITECTURES=86, CUDA 12.6, MSVC 18.4--nativeQ2_0 GGUF shards,--ple-gguf,--expert-profile data/expert-profile.bin--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp --max-context 131072 --kv int8 --kv-resident 16384 --vram-reserve-mib 1400STRATA_HC_REQ8=1,STRATA_PREFILL_LAYER_MAJOR=1for the arm under test and=0for every control, plus the config'sSTRATA_ADAPT_NOWAIT=1,STRATA_ADAPT_LAG=2,STRATA_CPU_IQ_ALL_EXACT1=1,STRATA_SIDE_MIXER=2,STRATA_ATTN_DAILY=1,STRATA_GR_UP75=1,STRATA_GU_SWIGLU=1,STRATA_QSA_ROWS=1,STRATA_B6_MMVQ_ROWS=1[Engine] layer-major prefill: expand the hyper-connection projections…
udqe commentedon Oct 30, 2026
cedrik-fuoco's stream finding (#4715) - one independent tree confirmation: the three call sites are still missing the stream argument in the 0.2.8 tree, so every 0.2.8 build ships with the race.
We read our installed 0.2.8 source (Terminal CLI self-report
strata 0.2.8 (build 0.1.54 ...), the tree the update wrote in today):bf16_proj's signature takesvoid* stream = nullptras the 11th argument (prefill.cpp:2702-2704), and the int8 expansion comment says "the GEMM takes bf16, so the int8 codes are expanded into one reused scratch on this stream" (2708) - a static scratch underSTRATA_HC_REQ8, exactly as he describes.run_implpasses it: the down/up/inject calls end withm.cs(3744, 3773-3774).run_layer_major(starts 6150) does not: the same three calls - 6781 (hc_attn_down), 6783 (hc_*_up), 6784 (hc_ffn_inject) - stop atm.xn16_lo/m.lo16_lowith no stream, so the expansion lands on the default stream while the path's kernels run oncs.Three sites, the same ones he fixed in #4715. This is static reading of the source only - unconfirmed static analysis in the repo's words: we did not build the engine and did not reproduce the needle failure (our box runs
STRATA_HC_REQ8off and the chunk path, so we are outside the affected combination; our env'sSTRATA_HC_Q8is the different, upstream switch).We also checked main (870aedd, the only commit after the 0.2.8 release): the three sites are unchanged there too, and #4715 has no review yet - every 0.2.8 user who turns on
HC_REQ8+ walk is exposed until it lands.Two things the thread could take from this cheaply:
STRATA_HC_REQ8=1+STRATA_PREFILL_LAYER_MAJOR=1- one line worth having where walk users read.We are reporting from the Windows/CUDA 0.2.8 tree as installed; no claim about whether the fix in #4715 is complete beyond the three sites he named - the tree still matching his diagnosis is the whole of our claim. (His PR's patch anchors at the same lines we quote, so the fix and our reading are of identical code.)
(AI-assisted: the model served by this repo's engine read the tree and drafted this; the line numbers are ours from the installed 0.2.8 source.)
qsa select: a set STRATA_SELECT_TWO_PASS says once where the two-pass…
Norhiro commentedon Oct 31, 2026
Field data (requested for 0.2.7, measured on 0.2.8). One unequal card pair, RTX 3060 12 GB + RTX 5060 Ti 16 GB, on two hosts. Qwen UD-Q4_K_XL, source build of v0.2.8 (870aedd), explicit split K=24 with
--trim-stage-weights, order 3060 → 5060 Ti.Host A: i7-7800X, 62 GB RAM, one NUMA node, AVX-512 build. 2 runs per size.
Host B: 2 x E5-2630 v2, 251 GB RAM, AVX only (
STRATA_ISA_FLOOR=avx), both cards PCIe Gen3 x8 on node 0 (engine probe 6.0-6.7 GB/s), process bound to node 0, 3 runs per size.Time to first token, second run. Off =
STRATA_PREFILL_LAYER_MAJOR=0,--vram-reserve-later-mib 1400. Tuned =--vram-reserve-mib 1700 --vram-reserve-later-mib 1700 --prefill 4096 STRATA_PREFILL_LM_MARGIN_MIB=96.The walk ran in both tuned arms. On host B it halved the 20K and 32K reads on a Gen3 x8 link, unlike the Gen3 pairs above; I did not test why, and the 3060 holds far fewer experts in VRAM than those cards.
Limits: on host B the tuned arm ran after the off arm, and its near-zero reads include the page cache. T5 ran once per arm, off first, so -4% may be noise. On host A the extra reserve cost 19% decode; expert slots fell 2335 → 2001 on the 3060 and 2952 → 2851 on the 5060 Ti, the suspected cause, not isolated.
Quality, off against tuned, same on both hosts: needle 9/9 both, HumanEval 0-39 39/38 both, Japanese 11/12 both, tool calls 8/8 both. All with
STRATA_HC_Q8=1;STRATA_HC_REQ8was not set, so #4715 does not apply.Host A only: with the defaults, and with
--layer-split autoin both card orders (K=18 and K=34), layer-major declined and 32K took 50-65 s; no OOM. The startup line says layer-major is on in every default arm. Host B did not run these arms.