diff --git a/modules/dasLLAMA/ARCHITECTURE.md b/modules/dasLLAMA/ARCHITECTURE.md index 864b8157d0..951e6a43b2 100644 --- a/modules/dasLLAMA/ARCHITECTURE.md +++ b/modules/dasLLAMA/ARCHITECTURE.md @@ -39,9 +39,9 @@ re-transcoding `$LCPP/src/unicode-data.cpp`). ## 2. Mechanisms -- `ARCHITECTURE_IMAGE.md` - sec.2.1-2.1m: the prepared-image rail, the baked dev-W f16 plane, +- `ARCHITECTURE_IMAGE.md` - sec.2.1-2.1n: the prepared-image rail, the baked dev-W f16 plane, the baked tower twin-W plane, the layout stamp `REVIEW.das` keeps over the byte-placing code, - and the meta-field tripwire every struct serializer opens with. + the meta-field tripwire every struct serializer opens with, and the planar-vs-blob flavor rebake. - `ARCHITECTURE_GPU.md` - sec.2.2b, 2.2w-2.2x: the tensor-GEMM and fused-attention shapes that measured out, the tower attention routes, and the tower driver's encode chains. - `ARCHITECTURE_GPU_QUANT_PLANES.md` - sec.2.2y-2.2z: the Metal kq split scale plane, and the @@ -53,10 +53,10 @@ re-transcoding `$LCPP/src/unicode-data.cpp`). driver's prefill chain and byte stores - the prefill window chain, the Q8 requant byte store, the decode GEMV family's grid codebook buffer, the tile probe's shared descriptor set layout, and the recurrent block of the prefill window. -- `ARCHITECTURE_GPU_VULKAN_GEMM.md` - sec.2.2k-2.2m, 2.2q, 2.2ae: the cooperative-matrix tiles - the Vulkan tier's GEMMs run on - the cm2 decode lanes, the tile pick and the coopmat mode - ladder, the class-pipeline build seat, the MoE expert chain on those tiles, and the KHR arm's - hand-staged kq tile. +- `ARCHITECTURE_GPU_VULKAN_GEMM.md` - sec.2.2k-2.2m, 2.2q, 2.2ae, 2.2ah: the cooperative-matrix + tiles the Vulkan tier's GEMMs run on - the cm2 decode lanes, the tile pick and the coopmat mode + ladder, the class-pipeline build seat, the MoE expert chain on those tiles, the KHR arm's + hand-staged kq tile - and the decode GEMV family's lane split by row length. - `ARCHITECTURE_GPU_VULKAN_RESIDENCY.md` - sec.2.2n-2.2o: what a model has to fit on the card before the driver runs - the residency plan, and the GPU-slot marks swap that lets one slot serve many models. @@ -65,6 +65,9 @@ re-transcoding `$LCPP/src/unicode-data.cpp`). whole-token decode span, the deltanet decode step's per-session resident state, and the whole-model driver's hybrid token command and prefill (recurrent layers, gated attention, partial rotary, the prefill-to-decode state handoff). +- `ARCHITECTURE_GPU_VULKAN_MOE.md` - sec.2.2af, 2.2ag: the resident driver's routed block in its + two eras - the MoE block of the prefill window, and the whole-model driver's MoE token + command. - `ARCHITECTURE_GPU_MTP.md` - sec.2.28-2.39: the Metal speculative round over the batch driver's same-slab verify, the box knob that sets the depth a round drafts, and the kernel argument-alignment contract enforced at every dispatch. diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU.md b/modules/dasLLAMA/ARCHITECTURE_GPU.md index 4cae0f5f98..29d838e6ca 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU.md @@ -81,16 +81,16 @@ that a question answered for one backend has an obvious address in the other. Th family numbers, so no shared bind path may span the two layouts; and the in-engine moe mul_mm A/B race harnesses (`dasllama_metal_prefill.das`) encode through `kn_moe_mm_family_tail` rather than a per-class `enc_*` builder. -- **`dasllama_gpu_tier.das`** - the device-cooperation SPI: hook types, install/unset slots, - route/mark/want/status state, engine-facing forwarders. Vulkan implements it (per-op offload plus - resident plumbing, and the decode-era seats it alone fills: the cm2 expert chain - `set_moe_gpu_ffn_xf_hooks` / `_async_hooks`, the decode attention block - `set_moe_gpu_attn_dec_hooks`, the decode FFN tail `set_moe_gpu_ffn_tail_hooks`, the deltanet - decode step's state seams `set_moe_gpu_dn_state_hooks` (flush, invalidate, release), the - whole-token span `set_moe_gpu_span_dec_hook` - the span rides common's decode override - registry as `vulkan_moe_span`, selected by the MoE placement and declining per token - the - resident driver's q/k/v projection-bias seat `install_moe_gpu_resident_bias`, the OS - video-memory seat `install_moe_gpu_os_memory` the residency plan sizes against, and the +- **`dasllama_gpu_tier.das`** - the device-cooperation SPI: hook types, install/unset slots, route/mark/want/status + state, engine-facing forwarders. Vulkan implements it (per-op offload plus resident plumbing, and the decode-era + seats it alone fills: the cm2 expert chain `set_moe_gpu_ffn_xf_hooks` / `_async_hooks`, the decode attention block + `set_moe_gpu_attn_dec_hooks`, the decode FFN tail `set_moe_gpu_ffn_tail_hooks`, the deltanet decode step's state + seams `set_moe_gpu_dn_state_hooks` (flush, invalidate, release), the whole-token span `set_moe_gpu_span_dec_hook` - + the span rides common's decode override registry as `vulkan_moe_span`, selected by the MoE placement and declining + per token - the resident driver's q/k/v projection-bias seat `install_moe_gpu_resident_bias`, its MoE seats + `install_moe_gpu_resident_moe` (the tile admission per expert triple, the routing geometry with the router plane, an + MoE layer, and the routed block on a layer another seat built) behind the route lever `set_gpu_resident_route` / + `gpu_want_resident`, the OS video-memory seat `install_moe_gpu_os_memory` the residency plan sizes against, and the weight-bytes seat `install_rdec_note_weight_bytes` the decode warm-up guard reads). The installs are one-way: a test that arms the tier installs the seats and never restores them, because no uninstall exists and none is needed - a seat serves whatever model loads next; Metal @@ -134,8 +134,8 @@ in prefill) and the tuner calls those public entries. `MetalPrefillDecline`); decline COUNTING lives in `_common` beside `require_or_panic`, for both paths. -Sections 2.28-2.30, the Metal speculative round and the dispatch alignment contract, are -`ARCHITECTURE_GPU_MTP.md`. +Sections 2.28-2.39 - the Metal speculative round, the depth a round drafts, the kernel argument-alignment +contract, and the verify, drafter and batch-driver mechanics after them - are `ARCHITECTURE_GPU_MTP.md`. **The allowed asymmetries between the backends - this list is closed; a new one lands with its entry here:** @@ -260,10 +260,10 @@ consecutive staging runs, relaxed_precision always - are `REVIEW_GPU.md` rules a `modules/dasMetal/REVIEW.das` descriptor gate; this section keeps only the refuted shapes and why they lose. -Sections 2.2j, 2.2p, 2.2ab, 2.2ac and 2.2ad, the Vulkan resident driver's prefill chain and -byte stores, are `ARCHITECTURE_GPU_VULKAN.md`; its 2.2k-2.2m, 2.2q and 2.2ae - the cooperative-matrix -GEMM tiles - are `ARCHITECTURE_GPU_VULKAN_GEMM.md`; its 2.2n-2.2o - the residency plan and the -marks swap - are `ARCHITECTURE_GPU_VULKAN_RESIDENCY.md`. +The Vulkan resident driver's sections live in its companions, each head saying what it holds: 2.2j, +2.2p, 2.2ab, 2.2ac and 2.2ad in `ARCHITECTURE_GPU_VULKAN.md`; 2.2k-2.2m, 2.2q, 2.2ae and 2.2ah in +`ARCHITECTURE_GPU_VULKAN_GEMM.md`; 2.2n-2.2o in `ARCHITECTURE_GPU_VULKAN_RESIDENCY.md`; 2.2r-2.2v in +`ARCHITECTURE_GPU_VULKAN_DECODE.md`; 2.2af and 2.2ag in `ARCHITECTURE_GPU_VULKAN_MOE.md`. ### 2.2w The tower attention routes {#tower-attn-routes} diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_MTP.md b/modules/dasLLAMA/ARCHITECTURE_GPU_MTP.md index 025fc086b3..e432cf3bee 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_MTP.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_MTP.md @@ -29,7 +29,13 @@ round commits rows 0..a: row a's logits become `s.logits` (the token sampled nex post-final-norm hidden becomes `s.mtp_h` (the next round's carry, and the assistant drafter's h input), `s.mtp_h_pos1` and the mirror watermark move to `pos + a + 1`, and `n_past` advances by `a + 1`. The rows above the new watermark are the rejected drafts' - garbage the next round -rewrites - and only the watermark keeps them from being read. +rewrites - and only the watermark keeps them from being read. The CPU depth-1 step +(`mtp_spec_eval`) takes a rejected draft on a model with no recurrent layer without a re-forward: +the verify's row 0 already holds the committed token's logits and post-norm hidden, so they stand +and the draft head is re-seeded from them; a recurrent model restores the pre-verify state and +re-forwards the committed token. `set_mtp_force_reject_every(n)` rejects every n-th draft of that +step's greedy walk whatever the verify said - the seam a test reaches the reject arm through on a +fixture that accepts every draft. **A sampled stream's walk draws instead of comparing argmaxes, and the same walk serves every round - the two Metal rounds and the CPU depth-1 step.** The caller points `s.spec_params` at its diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN.md b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN.md index 13b9fb08e5..5541c185c0 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN.md @@ -4,17 +4,34 @@ Companion to `ARCHITECTURE_GPU.md`; section numbers are `ARCHITECTURE.md`'s. Thi carries sections 2.2j, 2.2p, 2.2ab, 2.2ac and 2.2ad - the Vulkan resident driver's prefill chain, its byte stores, and the tile probe's set layout: the prefill window chain, the Q8 requant byte store, the decode GEMV family's grid codebook buffer, the tile probe's shared -descriptor set layout, and the recurrent block of the prefill window. The cooperative-matrix +descriptor set layout, and the recurrent block of the prefill window. The MoE block of that +window and the token command's routed twin are `ARCHITECTURE_GPU_VULKAN_MOE.md`'s sections +2.2af and 2.2ag. The cooperative-matrix tiles the chain's GEMMs run on - the cm2 decode spelling, the tile pick and the coopmat mode ladder, the class-pipeline build seat, the MoE expert chain on those tiles, and the KHR arm's hand-staged kq tile - are `ARCHITECTURE_GPU_VULKAN_GEMM.md`'s sections 2.2k-2.2m, 2.2q and -2.2ae. What a model has to fit on +2.2ae, and the decode GEMV family's lane split by row length its section 2.2ah. What a model has to fit on the card before any of this runs - the residency plan, and the marks swap that lets one GPU slot serve many models - is `ARCHITECTURE_GPU_VULKAN_RESIDENCY.md`'s sections 2.2n-2.2o. The decode-era mechanisms of the per-op tier are `ARCHITECTURE_GPU_VULKAN_DECODE.md`'s sections 2.2r-2.2v. The GPU backend role table these sections build on stays in `ARCHITECTURE_GPU.md` sec.1.5. +The module gate's four Vulkan checks (`REVIEW.das`) read these files. `check_khr_stage16_abstract` +reads `class template KqCm2BatchT` in `dasllama_vulkan_classes.das` and licenses no names: its +`khr_stage16` is declared abstract. `check_ar_max_dim_triple` reads `AR_MAX_DIM` in +`dasllama_vulkan_common.das`, the `row` slab of `ArBase` in `dasllama_vulkan_classes.das` and the +`c.dim` cap of `attn_dec_shape_ok` in `dasllama_blocks.das`, and licenses no names: the three +numbers agree. `check_cm2_khr_set` walks every `class template Cm2T : KqCm2BatchT` in +`dasllama_vulkan_classes.das` and requires `KhrBatch`, its `kq_batch__khr_cls` stamp and +an arm in each of `khr_cls_ensure`, `khr_cls_set` and `khr_cls_enc` in +`dasllama_vulkan_prefill.das`; its licensed set is `Q8Cm2T` alone - q8 is no `kq_sb` format, its +cm2 tiles carry no KHR arm, and the KHR mode serves q8 through its own tile. +`check_no_hand_pipelines` walks `dasllama/`, `harness/` and `tests/` for a +`vkCreateComputePipelines(` call and licenses no names inside them; the two llama.cpp shader ports +under `performance/` (`coopmat_mulmm_reference.das`, `coopmat_mulmm_port.das`) sit outside the +walk as reference measurements of another engine's kernels. + ### 2.2j The Vulkan resident prefill window chain {#vk-prefill-window-chain} **A prompt longer than `PF_WINDOW` rows runs as SEQUENTIAL windows over the same activation @@ -30,8 +47,9 @@ starting 32 rows below the window's end (`fill_arena_batch_sched`'s `row0`, `Act `ArArgs.row0`). Thirty-two, not one, because the s tile - the cm2 tile with 32-row columns (`ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2l) - loads a whole 32-row column unclamped on its fast path, and the resident prefill's activation planes (`pf_xf`, `pf_hf`) carry no read slack -past the window - unlike the MoE chain's gathered image and hidden plane, which sec.2.2l sizes -with 32 rows of slack past their last region. Rows below the slice keep stale gate, up, +past the window - unlike the MoE chain's gathered image and hidden plane, which +`ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2l sizes with 128 rows of slack past their last region +(`TILE_READ_SLACK`). Rows below the slice keep stale gate, up, hidden and residual values that nothing reads. The sliced GEMMs do not split k: the split-k reduce sums partial planes from row 0, so a region starting below the window's end would reduce the wrong rows. The slice takes the f16-fed cm2 route only (`gu6 && dn6`); the other feeds run diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_DECODE.md b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_DECODE.md index 91a6913ce1..70a1a0471a 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_DECODE.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_DECODE.md @@ -1,14 +1,16 @@ # dasLLAMA Architecture - the Vulkan per-op tier's decode era Companion to `ARCHITECTURE_GPU_VULKAN.md`; section numbers are `ARCHITECTURE.md`'s. This -document carries sections 2.2r-2.2v: the decode attention block over per-layer K/V mirrors, -the streamed expert layer's GPU/CPU split, the whole-token decode span, the deltanet decode -step's per-session resident state, and the whole-model driver's hybrid token command. The -prefill window chain and byte stores these build on are `ARCHITECTURE_GPU_VULKAN.md` sections -2.2j, 2.2p, 2.2ab, 2.2ac and 2.2ad; the cm2 tiles, the MoE expert chain on them and the KHR -arm's kq tile are `ARCHITECTURE_GPU_VULKAN_GEMM.md` sections 2.2k-2.2m, 2.2q and 2.2ae; the -residency plan and the marks -swap under them are `ARCHITECTURE_GPU_VULKAN_RESIDENCY.md` sections 2.2n-2.2o. +document carries sections 2.2r-2.2v: the decode attention block over per-layer K/V mirrors, the +streamed expert layer's GPU/CPU split, the whole-token decode span, the deltanet decode step's +per-session resident state, and the whole-model driver's hybrid token command. The prefill +window chain and byte stores these build on are `ARCHITECTURE_GPU_VULKAN.md` sections 2.2j, +2.2p, 2.2ab, 2.2ac and 2.2ad; the routed block an MoE layer takes in either era - the prefill +window's and the token command's - is `ARCHITECTURE_GPU_VULKAN_MOE.md` sections 2.2af and +2.2ag; the cm2 tiles, the MoE expert chain on them, the KHR arm's kq tile and the decode GEMV +family's lane split are `ARCHITECTURE_GPU_VULKAN_GEMM.md` sections 2.2k-2.2m, 2.2q, 2.2ae and +2.2ah; the residency plan and +the marks swap under them are `ARCHITECTURE_GPU_VULKAN_RESIDENCY.md` sections 2.2n-2.2o. ### 2.2r The per-op tier's decode attention block {#decode-attention-block} @@ -39,9 +41,10 @@ VRAM is layers x rows x kvd x 4 bytes). The loader reports that need (`set_moe_gpu_dat_need`) and the tier carves it from the weight budget BEFORE placement, the way the stream slots are carved: the mirrors are allocated after placement, and un-carved they came out of the desktop reserve and paged the resident expert stacks (the FFN chain's submit -went from 147 us to 741 us). The prefill chain fills it from its own f16 K/V -shadows at the end of each layer's window (`ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2q's fa arm converts the attended prefix; -the fill is a device copy of it), and the block appends one row per served token. +went from 147 us to 741 us). The prefill chain fills it from its own f16 K/V shadows at the end +of each layer's window (`ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2q's fa arm converts the +attended prefix; the fill is a device copy of it), and the block appends one row per served +token. **Ownership is a generation plus a per-layer count.** Every prefill claims ONCE, at the end of `forward_prefill_body` - the token entry and the embedding entries alike - and the tier mints a @@ -85,8 +88,9 @@ three constants (`SPLIT_BUS_BPS`, `SPLIT_GPU_ROW_S`, `SPLIT_CPU_ROW_S` in `dasll sit at 13.5 GB/s, 1.2 us and 10 us - the reference box's measurements (zen2 16 lanes, 5060 Ti on PCIe gen4 x8; `benchmarks/lcpp_bench.das -t 16` pp512 under `DASLLAMA_GPU_PROF=1`: the bus moves a streamed expert's bytes at ~11 GB/s, the cm2 chain costs ~1.6 us per bucket row, the CPU kq -groupn chain ~10 us per bucket row), tuned to where the split balanced best. A head that would cover every expert falls back to -the plain rail - streaming the whole layer beats splitting it against an idle CPU. +groupn chain ~10 us per bucket row), tuned to where the split balanced best. A head that would +cover every expert falls back to the plain rail - streaming the whole layer beats splitting it +against an idle CPU. **The GPU head streams only its experts' regions** (`copy_stream_group_partial` copies the claimed experts' plane ranges into the stream slot at their expert offsets), records the cm2 @@ -102,13 +106,14 @@ and the whole-group rail's overlap - compute on one while the transfer queue fil **Every resident expert layer's decode runs as ONE recorded chain and ONE submit per token** (`DASLLAMA_GPU_DEC_SPAN`, on; the `vulkan_moe_span` decode override, selected when the MoE -placement leaves expert layers or the classifier on the device, deselected at the model drop). The per-layer form costs a submit per attention and one per FFN, and each submit -carries ~75 us of queue and fence latency the GPU spends idle (`DASLLAMA_GPU_PROF=1`'s -`vk_dec prof` ledger over 96 tokens) - on a 48-layer model with 35 -resident layers that is the largest single term of the token. The span keeps the host out of the -resident suffix [l0, n_layers): the host feeds layer l0's attention row in its plane's acts -form, the device runs per layer the attention chain (sec.2.2r above), the residual add fused with the -FFN rms (`cls_ar`, `add_on`), the gate/up feed requants, the router GEMV over an f32 plane +placement leaves expert layers or the classifier on the device, deselected at the model drop). +The per-layer form costs a submit per attention and one per FFN, and each submit carries ~75 us +of queue and fence latency the GPU spends idle (`DASLLAMA_GPU_PROF=1`'s `vk_dec prof` ledger +over 96 tokens) - on a 48-layer model with 35 resident layers that is the largest single term +of the token. The span keeps the host out of the resident suffix [l0, n_layers): the host feeds +layer l0's attention row in its plane's acts form, the device runs per layer the attention +chain (sec.2.2r above), the residual add fused with the FFN rms (`cls_ar`, `add_on`), the +gate/up feed requants, the router GEMV over an f32 plane (`router_gemv_cls`: one workgroup per expert row, f32 in and f32 out - the host router's own arithmetic rather than a quant chain, so the device's picks track the CPU's up to summation order) and @@ -131,8 +136,19 @@ row serves every slot) and j for the down stack (the act+requant output has one slot - the row stride is the down feed's blocks per row). The select mirrors the host's `moe_select_core` softmax arm: max, exp floored at -80, sum, k picks of the largest with the lowest index on ties and the winner knocked out, the picks renormalized (the sum clamped at -f16-min) and scaled. Only that arm rides - sigmoid gates, router biases, shared experts, -post-norms and biased experts decline to the per-layer path (`span_model_ok`). +f16-min) and scaled. Only that arm rides - sigmoid gates, router biases, gemma4's dense +shared expert with its sandwich norms, post-norms and biased experts decline to the per-layer +path (`span_model_ok`). + +**A qwen2moe-class shared expert rides inside the span beside the routed slots.** Its q8 triple +is resident under the shexp mark (the span's serve gate asks per layer), its feed is the same +normed row requantized into its own stacks' images, its metas are one fixed region the host +writes once (the top-k never touches them), its gate logit is one more `RouterGemv` row over the +layer's gate vector landing past the router's logits, and one combine (`DecCombineSh`) adds +the routed slots and `sigmoid(gate)` times the shared down row onto the residual; an ungated +shared expert (glm4moe) adds at weight one. Before this the span declined every shared-expert +model to the per-layer path, and there the resident layer's shared expert ran on the CPU +(`sh_gpu` in `dasllama_moe.das` serves the shexp rail only beside CPU experts). **The span declines whole, per token, never per layer.** `span_first_layer` walks the layers downward through the block's serve gate (so a hydrate a layer needs runs first) and stops at @@ -141,8 +157,9 @@ resident layer under the suffix - a resident layer there would arm its FFN tail suffix's first feed (the layer under is a streamed one, or there is none). A model whose gate is not the softmax-with-renorm the top-k kernel mirrors, whose expert count or routed slots exceed the kernel's one-workgroup reach, whose router planes exceed one SSBO range, or whose -arch binds a non-standard attention or FFN block declines whole (`span_model_ok`). The gemv sets carry no hazard bits on their feed and meta bindings, so the -span's recorder declares those edges by hand (`vhz_dep` on the span's own region bits) after +arch binds a non-standard attention or FFN block declines whole (`span_model_ok`). The gemv +sets carry no hazard bits on their feed and meta bindings, so the span's recorder declares +those edges by hand (`vhz_dep` on the span's own region bits) after each requant and after the top-k. The loader carves the router planes with the mirrors (`set_moe_gpu_dat_need`). diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_GEMM.md b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_GEMM.md index dc7d9cfc3b..130e81b525 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_GEMM.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_GEMM.md @@ -1,10 +1,11 @@ # dasLLAMA Architecture - the Vulkan tier's GEMM tile family Companion to `ARCHITECTURE_GPU_VULKAN.md`; section numbers are `ARCHITECTURE.md`'s. This -document carries sections 2.2k-2.2m, 2.2q and 2.2ae, the cooperative-matrix tiles the Vulkan -tier's GEMMs run on: how a cm2 tile decodes its quant bytes, how a tile and the served GEMM -mode are picked, the class-pipeline build seat both shader instruments hang on, the MoE expert -chain on those tiles, and the KHR arm's hand-staged tile. `ARCHITECTURE_GPU_VULKAN.md` carries +document carries sections 2.2k-2.2m, 2.2q, 2.2ae and 2.2ah, the cooperative-matrix tiles the +Vulkan tier's GEMMs run on and the decode GEMV family's lane split: how a cm2 tile decodes its +quant bytes, how a tile and the served GEMM mode are picked, the class-pipeline build seat both +shader instruments hang on, the MoE expert chain on those tiles, the KHR arm's hand-staged +tile, and how a GEMV subgroup splits across short rows. `ARCHITECTURE_GPU_VULKAN.md` carries the prefill window chain that dispatches them (sec.2.2j) and its recurrent block (sec.2.2ad), the Q8 requant byte store (sec.2.2p), the decode GEMV family's grid codebook buffer (sec.2.2ab), and the tile probe's shared descriptor set layout (sec.2.2ac). What a model has to @@ -16,12 +17,21 @@ sections build on stays in `ARCHITECTURE_GPU.md` sec.1.5. ### 2.2k The cm2 decode callbacks read their quant bytes as 16-bit lanes {#cm2-decode-16bit-lanes} A cm2 tile's decode callback runs inside the driver's block load, and the vendor driver's shader -compiler pattern-matches only one spelling into that path: a 16-bit load (`int16[N]` block -members) followed by `unpack8(w)[i & 1u]` - a byte2 lane select - with sub-fields pulled out by -shift and mask. A 32-bit word with a variable shift runs slower; an `unpack8` of a 32-bit word -indexed by a runtime value (a byte4 dynamic select) drops the whole kernel off the block-load -path, to about a third of the rate. Every cm2 decode - q8 and every kq superblock format - is -spelled the 16-bit way, which is why the block structs are `int16` arrays over the same bytes. +compiler pattern-matches only one load width into that path: a 16-bit load (`int16[N]` block +members), with sub-fields pulled out by shift and mask. A 32-bit word with a variable shift runs +slower; an `unpack8` of a 32-bit word indexed by a runtime value (a byte4 dynamic select) drops +the whole kernel off the block-load path, to about a third of the rate. Every cm2 decode - q8 +and every kq superblock format - is spelled the 16-bit way, which is why the block structs are +`int16` arrays over the same bytes. A byte the decode needs at a runtime position comes out of +its lane by a shift - `(uint(int(blk.qs[i >> 1u])) & 0xFFFFu) >> ((i & 1u) * 8u)` - not by an +`unpack8(w)[i & 1u]` byte2 lane select: the select reads the same lane, but a decode built on +selects runs 1.1x to 1.5x slower than the shift form on the expert-schedule shape +(`harness/vk_gemm_probe.das -- moe:`, RTX 5060 Ti, per gate/up plane, the shift form: +iq2xxs 724 us, iq3xxs 569, iq3s 766, iq2s 736 against llama.cpp's cm2 `mul_mat_id` tile at +754 / 788 / 870 / 797; the select form read 1.28x, 1.24x, 1.49x and 1.08x of those times on the +same shape). A sign index that straddles two bytes (the +IQ2_XXS and IQ3_XXS aux32 words) is assembled from its two lanes and shifted, never built from +two selected bytes. Every table a decode reads at a runtime index is staged into a `@workgroup` array ahead of the tile loop, never selected out of a register vector per element: the iq4 formats' 16-entry codebook (`kvalues_iq4nl`, shared by IQ4_XS and IQ4_NL) as f16, each grid format's codebook as @@ -53,17 +63,25 @@ times that SM count is the slots it allocates. The pick takes the tile whose wor the larger share of its allocated slots, the two ratios compared by cross-multiplying. The m tile wins only on a strict win; a tie goes to l, whose bigger tile carries twice the arithmetic intensity. Three rules sit ahead of the comparison: a region of 64 rows or fewer takes the s -tile (32-row columns - the MoE expert-bucket shape, where a 512-token window routes ~32 rows to -each of 128 experts and an m column would pad three quarters of every tile and take the edge -path on all of them), a window of 128 rows or fewer takes m (the l column would run half -empty), and a device that reports no SM count takes l and never splits k. Beyond `(d, cnt, -sm_count)` the pick reads only two values fixed at init - the served mode and -`DASLLAMA_CM2_TILE` - so the class the pipeline binds and the tile rule the meta fill writes +tile (32-row columns - the per-op tier's MoE expert-bucket shape, where a 512-token window +routes ~32 rows to each of 128 experts on average), a window of 128 rows or fewer takes m (the +l column would run half empty), and a device that reports no SM count takes l and never splits +k. Beyond `(d, cnt, sm_count)` the pick reads only two values fixed at init - the served mode +and `DASLLAMA_CM2_TILE` - so the class the pipeline binds and the tile rule the meta fill writes can never disagree; `cnt` is the AVERAGE rows per active region of the dispatch, so one tile -serves every region of a MoE schedule. A region below the s tile's row count goes to the decode -GEMV family, not to a tile. The s tile's fast path loads a partial 32-row column UNCLAMPED and -clamps only the store, so every f16 plane the chain feeds it - the gathered activation image and -the hidden plane - is sized with 32 rows of slack past its last region (`ffn_cm2_chunk_rows`). +serves every region of a per-op MoE schedule. The resident MoE block makes no pick: its device +schedule cuts every bucket into s and m pieces by size and dispatches both classes per plane +(`ARCHITECTURE_GPU_VULKAN_MOE.md` sec.2.2af), which is what a real window's skew needs - one tile +per bucket costs the same whatever its fill, and a 467-row bucket is 15 s tiles or 4 m +columns. A region below the s tile's row count goes to the decode GEMV family, not to a tile. +The s and m tiles' fast path loads a partial column UNCLAMPED (the layout's row dimension +rounded up to the column) and clamps only the store, so every f16 plane the chain feeds them - +the gathered activation image and the hidden plane - is sized with 128 rows of slack past its +last region (`TILE_READ_SLACK`, `ffn_cm2_chunk_rows`); the l tile takes the edge path on a +partial column, since only a window's last column is ever partial there. The dense chain's +planes carry no slack: they hold the whole window's rows whatever the last window's length, so +a partial m column's unclamped load stays inside them. The store-layout constant the m and s +tiles read (`STILE`) is inert on the KHR classes, whose tile never reads it. **The split-k pick counts the dispatch group, not the GEMM.** With long K (2048 and up), a grid that fills at most half the SMs splits its reduction across f32 partial planes that @@ -102,13 +120,18 @@ choice shapes an image byte, so the bake identity ignores it: a serve-only knob configuration field. `decvec_on` is the run's arm, announced on the `device ready` line. **The tile's fast path is what makes the loads unclamped.** It runs when the weight tile is -whole (`m0 + 128 <= d`), the token column is whole or stamped s, and K is a whole number of BK -steps; the layouts are then created clamp-Undefined and the B and output strides are masked to -a multiple of 8 f16 (`stride &= ~7`). The mask changes nothing while `n` and `d` are -32-multiples, which every served shape is; it exists to make the alignment PROVABLE to the -driver's address analysis, which is what keeps the loads on the wide path. The s column gates -only the weight tile: its partial token column loads unclamped and its store clamps. Everything -else takes the edge path with clamped layouts. +whole (`m0 + 128 <= d`), the token column is whole or the stamp carries the partial-column path +(`STILE`: the s and m columns), and K is a whole number of BK steps; the layouts are then +created clamp-Undefined and the B and output strides are masked to a multiple of 8 f16 +(`stride &= ~7`). The mask changes nothing while `n` and `d` are 32-multiples, which every +served shape is; it exists to make the alignment PROVABLE to the driver's address analysis, +which is what keeps the loads on the wide path. A partial-column stamp gates only the weight +tile: its partial token column loads unclamped and that column's store clamps +(`tensorLayout2DPad`), while a whole column stores unclamped on every stamp (the clamp on a +whole column measured free on the k4 m tile, 48.0 against 48.1 TFLOP/s at the gate shape +(`harness/vk_gemm_probe.das -- cm2:k4`, RTX 5060 Ti), so +the branch is there for the layout's meaning, not its cost). Everything else takes the edge +path with clamped layouts. **The no-split arm keeps literal loop bounds and a literal store base.** Where `ksplit` is zero the k loop runs the literal `0 .. n` with the store at the row base rather than the general @@ -158,6 +181,13 @@ the layer's FFN is the routing alone. The f16 form is the combined (`npos > 0`) combine is what makes the device-side gather pay, since neither the gathered image nor the bucket rows ever cross PCIe. Streamed groups take the same arm after the slot bind. +**The shared expert of a qwen2moe-class layer takes the same arm as ONE region over every +position of the window** - its q8 triple is resident under the shexp mark, the slot map is the +identity and the combine runs at unit weight, so the host reduce scales its rows by the per-row +sigmoid gate in the CPU form's order. The CPU form of the same triple costs 582 ms of a 945 ms +window on Qwen1.5-MoE-A2.7B (`benchmarks/lcpp_bench.das -p 512 --prof`, the Q4_K_M mint, RTX +5060 Ti), the one term the arm exists to move. + **The per-op attention chain runs the same cm2 flash-attention tile the resident chain runs** (`fa_cm2_h64` / `h128`, `ARCHITECTURE_GPU_VULKAN.md` sec.2.2j) when the device reports the cm2 flash-attention features (`has_coopmat2_fa` - cooperative-matrix reductions, conversions and @@ -223,3 +253,30 @@ tile pick answers 128 and split-k never engages, and `cm2_cls_ensure/set/enc` ro only on a 32-lane subgroup (`khr_kq_tile_on`): the body indexes eight subgroups over the tile, so a wave64 device (four subgroups per 256-thread workgroup) keeps its kq planes on the sdot4 batch tile. + +### 2.2ah The decode GEMV family splits a subgroup across rows by the row length {#kq-gemv-lanes} + +**A subgroup of the kq GEMV family takes one, two or four output rows, each row's lanes a cluster +of the fold.** A lane loads one 32-block per step (`gemv_shell`), so a row of nb blocks over 32 +lanes keeps nb / 32 loads in flight per lane: two at K 2048, under one at a MoE's expert rows +(K 512 to 1408, 16 to 44 blocks), where most of the subgroup idled and the DRAM rate fell to a +third of the k4 band. `gemv_lanes_per_row` picks the lanes per row from the row's blocks - 8 to +24 blocks; 8 for the grid formats and 16 for the k-lattice to 48; 16 to 96; past that the whole +subgroup for the k-lattice and 16 for the grid formats - and `gemv_enc` sizes the grid to match +(a workgroup's subgroups each take subgroup_size / lanes rows). The grid formats are the +codebook and grid decodes (iq2s, iq2xs, iq2xxs, iq3xxs, iq4xs, iq4nl, `gemv_grid_fmt`); iq3s +takes the k-lattice split (K 1408: 354 / 395 / 386 GB/s at 32 / 16 / 8 lanes). The push block +carries the lanes (0 = the whole subgroup, the q8 GEMV's one form); the fold is an xor-shuffle +butterfly over the row's lanes at 8 or 16 - the subgroup shuffle every GEMV already requires, +where a clustered add would ask for the clustered subgroup feature the tier never checks - and +the whole-subgroup add otherwise, and every lane reduces, a dead row at zero, so the butterflies +stay whole. The lane-to-block map sets the row's summation order, so a resident-vs-CPU bar reads +a different noise sample than the one-row form did, inside the same class. Measured on the RTX +5060 Ti (`harness/vk_gemv_probe.das `, DRAM-bound planes, GB/s at 32 / 16 / 8 lanes): K 512 iq2s +100 / 194 / 297, iq2xxs 142 / 233 / 351, k4 394 / 402 / 413, k6 386 / 414 / 407; K 768 iq2s +148 / 221 / 337, iq2xxs 213 / 269 / 376, k4 403 / 404 / 414, k6 407 / 392 / 387; K 1408 iq2s +198 / 299 / 368, k6 386 / 399 / 373, k4 394 / 394 / 382; K 2048 iq2s 311 / 384 / 388, k6 417 / +416 / 369; K 4096 iq2s 376 / 408 / 344, k6 403 / 387 / 326, k4 400 / 404 / 406; K 5632 k4 411 / +399 / 386, k6 396 / 384 / 287, iq2s 377 / 393 / 296. llama.cpp's mat-vec splits K over 16 threads +and blocks two to four rows per thread (`rm_kq`, `NUM_ROWS` in its `mul_mat_vec_*.comp`): the +same bytes in flight by the other axis. diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_MOE.md b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_MOE.md new file mode 100644 index 0000000000..9b872c7b5d --- /dev/null +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_MOE.md @@ -0,0 +1,156 @@ +# dasLLAMA Architecture - the Vulkan resident driver's MoE block + +Companion to `ARCHITECTURE_GPU_VULKAN.md`; section numbers are `ARCHITECTURE.md`'s. This +document carries sections 2.2af and 2.2ag - the resident driver's routed block in its two eras: +the MoE block of the prefill window, and the whole-model driver's MoE token command. The window +chain the prefill block runs inside (sec.2.2j), the recurrent block beside it (sec.2.2ad), the Q8 +requant byte store (sec.2.2p), the decode GEMV family's grid codebook buffer (sec.2.2ab) and the +tile probe's shared descriptor set layout (sec.2.2ac) are `ARCHITECTURE_GPU_VULKAN.md`'s. The +token command's attention and recurrent heads this block's tail follows, and the per-op tier's +decode era - the decode span whose kernels the routed block runs - are +`ARCHITECTURE_GPU_VULKAN_DECODE.md`'s sections 2.2r-2.2v. The cooperative-matrix tiles the +expert GEMMs run on and the per-op tier's MoE expert chain are +`ARCHITECTURE_GPU_VULKAN_GEMM.md`'s sections 2.2k-2.2m, 2.2q and 2.2ae, and the lane split of +the decode GEMV family the token command's expert GEMVs take is its section 2.2ah. What a model has to fit +on the card before any of this runs - the residency plan that sizes the expert planes, and the +marks swap - is `ARCHITECTURE_GPU_VULKAN_RESIDENCY.md`'s sections 2.2n-2.2o. The GPU backend +role table these sections build on stays in `ARCHITECTURE_GPU.md` sec.1.5. + +### 2.2af The MoE block of the prefill window {#vk-prefill-moe-block} + +**An MoE layer's window block replaces the dense FFN tail with a routed block on the device; +the attention head and the residual steps are shared.** The whole-model driver admits a MoE +whose expert stacks fit the arena beside its attention quads (`ARCHITECTURE_GPU_VULKAN_RESIDENCY.md` +sec.2.2n), and the window then never leaves the device between layers: the CPU's routing, +bucketing and combine of the per-op tier (`ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2q) become +five device stages over the window's FFN-normed rows. Every window millisecond and per-layer +microsecond below is the `DASLLAMA_GPU_PROF=1` window profile (`vk_rdpf`) on the RTX 5060 Ti, +except where a probe arm is named. + +- **The router GEMM** (`RouterGemm`) is the span's router GEMV batched: a 64 x 32 tile of + positions by router rows per workgroup, each invocation a 4 x 2 block whose two rows sit 16 + apart, K in 64-wide steps through a float4 stage in shared memory at a row stride of 17 + float4, the next step's rows fetched into registers while the current step computes, and each + output's four products per float4 added in k order (the scalar loop's sums to the bit). The + stride and the row split are the bank rule: sixteen lanes reading sixteen rows land on all + eight 16-byte bank groups, so a warp's two weight loads take two wavefronts each and its four + activation loads (two distinct rows) one. The scalar stage this replaced - rows 2te and 2te+1 + at a stride of 68 floats - put four of every sixteen lanes on one bank, and the tile ran at + about 17 FMAs per cycle per SM: 4.3 ms per 30B window against the float4 stage's 2.4 (a 16 x 16 tile of one + output each, stepping K by 32, was barrier-bound at 243 us per layer for 268 MFLOP; a 32 x + 32 tile of 2 x 2 blocks staging each step before computing it read 142). The 64-wide K step + is why the MoE seats ask for a 64-multiple row width. The router plane holds every MoE layer's f32 rows, + and a gated shared expert's gate vector rides as one more row past the experts, so one + dispatch writes the logits row `[ne | gate]` per position. +- **The per-row select** (`TopKRows`, one workgroup per position) is the decode top-k's core + over each row: the softmax, k picks largest-first with ties to the lower index, the + renormalized or scaled weights - the host `moe_select_core`'s arithmetic - written + position-major as the picked expert and its weight per slot. +- **The bucket schedule** (`MoeSched`, one workgroup) writes what the host fill writes for the + per-op chain: the experts' slot counts are an atomic tally over the window (the 256 threads + stride the picks, one workgroup atomic per slot), exclusive scans place each bucket's rows, + its region indices (experts with rows, in expert order) and its tile workgroups, and the + three expert planes' schedules land as 4-word records plus per-workgroup maps. A + bucket is cut into at most two pieces by the tile ladder: a bucket within the s column (32 + rows) is one s piece; a bigger bucket takes whole m columns (128 rows) with the last one + partial, unless the remainder past the whole columns fits the s column, which then takes it + (`sched_ladder_m_rows`). The s pieces' records sit at `[0, ne)`, the m pieces' at `[ne, 2 ne)`, + and each dispatch's map at its own offset past the records (`PF_MOE_MAP_OFF`, 2048 words for up + to 256 experts twice), the two tile counts scanned as one packed word (its halves stay under + 65536 for any admitted shape: 512 tokens by 64 slots over 256 experts). Every dispatch is sized + for its worst case - one s tile per expert; every expert's whole columns plus a partial one - + and the map's tail past the real workgroup count carries the sentinel (`SCHED_NONE`): a tile + workgroup that reads it sees a zero-row region and returns before its first barrier. A real + window's router is skewed - on the Qwen3-30B-A3B at 512 tokens, 69 of 128 experts route, nine + hold over 128 rows (the largest 467) - so the ladder runs about 90 column tiles where the s + column alone ran 175 (`DASLLAMA_GPU_PROF=1` prints the last MoE layer's buckets and both + counts): the expert planes at 563 / 652 us against 815 / 896 on the s column alone + (`harness/vk_gemm_probe.das -- moesk:iq2xxs`, that window's profile). The slot-to-bucket-row + map hands every slot the next row of its expert's bucket through an atomic cursor, so the + rows within a bucket land in an order the schedule does not fix - which nothing downstream + reads: the tiles compute rows apart and the combine reads each slot's row through the map, so + the sums are the CPU walk's to the bit (the kernel cell checks the map as a permutation of + each bucket's rows). The per-expert slot walks this replaced - thread e scanning every slot + twice, staged through workgroup memory in 256-slot chunks - cost 163 us per layer on the 30B. +- **The gather, the expert tiles and the act** are the per-op chain's: the f16 gather scatters + each position's row into its bucket rows, gate and up run the cm2 tiles over the gathered + image - each plane's m dispatch then its s dispatch, the two writing disjoint rows of one + plane under separate hazard bits (`VHZ_GATE_M`, `VHZ_UP_M`, `VHZ_MDN_M`) so they co-run and the + reader's barrier covers both - the act writes the f16 hidden rows, and down runs the tiles + again. The window planes carry 128 rows of read slack past the last bucket row (the s and m + tiles load a partial column unclamped, `ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2l) and hold a + whole window of `PF_WINDOW x k` bucket rows, so the block never chunks. +- **The combine rides the residual step.** The add+rms that follows the block (`ClsArComb`, + its f16 twin `ClsArCombF16B` where the next layer's head takes the f16 feed) adds the shared + expert's down rows at the sigmoid of the gate logit - the dense tail ran the shared triple + first, as the layer's dense triple at the shared width - and the k weighted expert rows + through the slot map, straight into the residual, then norms the row for the next layer. A + layer without a shared expert takes the same step with the add partner off. The step sums + the FFN row first - the gated shared row, then the slots in order - and adds it to the + residual, the order the separate combine dispatch and the plain add it replaced took, so the + resident-vs-CPU bars of the MoE files keep their calibration: the fold's natural order - the + residual first - moves the rounding enough to flip a router near-tie downstream, and one step + of the 35B two-window cell reads 1.50 logits off the CPU chain against a 1.39 bar where the + chain's order reads 0.39. The slot loop loads eight rows together, then four, then one at a + time - the token command's one-row form is latency, and the groups are its shape (sec.2.2ag); + a slot-major pass through the row stash instead read 2.4 ms more on the 30B window, the + shared-memory read-modify-write per slot costing what the register sum does not. The token + command's tail folds the same way; the residual step read those rows anyway, and the fold took + one dispatch per layer out of both chains. There is no Q8 requant leaf: a third form would + bind a ninth buffer, past the eight the hazard rail carries, so where the next layer's head + takes the Q8 feed the f32 leaf writes the normed row and a separate requant follows it. + +The router reads the f32 normed rows, so an MoE layer takes the split add+rms arm at the FFN +site (the fused twins never store `xb`), and the last-layer FFN slice of +`ARCHITECTURE_GPU_VULKAN.md` sec.2.2j does not apply to the routed block. The arm's requant of +the normed rows feeds the dense triple alone - the gather takes the f32 rows - so a layer with +no shared expert skips it. The tile family is the f16-fed cm2 tiles, so the plan admits a MoE +only in cm2 mode on a coopmat2 device with every expert format the f16 feed admits +(`rdec_moe_ok`). + +### 2.2ag The whole-model driver's MoE token command {#resident-moe-token} + +**An MoE layer rides the same recorded token command as a dense layer; its FFN tail is a routed +block over arena expert planes.** After the layer's attention head and the FFN norm, a layer +with a shared expert runs the dense tail over the shared triple (gate, up, the act, down into +`ffnout`), then the routed block: the router GEMV (`RouterGemv`) over the f32 normed row reads +the driver's router plane - every MoE layer's rows and, when the shared expert is gated, its gate +row last - into one logits row; the top-k (`TopK`, the span's kernel over the decode's core) +writes the k routing weights and the three expert GEMVs' slot regions, each a `(block, feed +block)` pair whose block is the expert plane's slab-local base plus the pick's stride; gate and +up run the class GEMV over k regions, the act writes k hidden rows, down runs k regions into the +routed rows; and the residual step that follows folds the combine in (`ClsArComb`, the prefill's +sec.2.2af kernel at one row): the shared expert's row in `ffnout` at the sigmoid of its gate +logit, the k weighted routed rows through the top-k's slot map, then the next layer's norm - a +layer without a shared expert takes the same step with the add partner off. A one-row dispatch is +latency: the step loads eight slots' rows together, then four, then one at a time (the sums +still in slot order), so an element waits on one load round per group rather than per slot - on +the `DASLLAMA_GPU_PROF=1` token profile (`vk_rdec moe avg/token`, RTX 5060 Ti) the Qwen1.5-MoE +twin's 24 layers at four slots read about 290 us per token in the step where a +plain slot loop read 360 and the add plus the separate combine 199 and 233; on the 30B's 48 +layers at eight slots 440 where a four-slot group alone read 490 (the compiler's own unroll of +the plain loop served eight slots but left four to a scalar tail). The slot regions are device +buffers the top-k fills each token; the dense triple's host-filled regions stay what they are. + +**A recurrent MoE layer takes the routed block after its deltanet head** (the hybrid MoE, +`ARCHITECTURE_GPU_VULKAN_DECODE.md` sec.2.2v's head with this section's tail): the deltanet +registration builds the layer, its shared expert rides as the dense triple, and the routed block +registers on the built layer through its own seat (`vk_rdec_set_moe_experts`, the seat the +attention form calls after its quad); the window chain's recurrent block precedes the same +routed block (sec.2.2af, the tail every head shares). + +**The routed block is the decode span's FFN half transplanted, not a second copy of the span.** +The kernels are the span's (`ARCHITECTURE_GPU_VULKAN_DECODE.md` sec.2.2t); what differs is the +home: the arena's expert planes and the resident driver's activation row instead of the per-op +tier's stacks and the span's own row. A model the plan admits therefore takes neither the span +nor the per-op rails - the resident prefill (sec.2.2af) fills the one mirror the token command +reads. + +The experts' feed is the layer's quantized row when their form is the dense triple's, else a +second requant of the normed row in their own form (a K-quant expert stack beside a shared +expert the loader could only transcode to q8; a shared expert kept in the file's K-quant form +shares the experts' feed). The fused add+rms+requant twins are off on a MoE: the router reads the +normed row those twins never store. The MoE seats install separately +(`install_moe_gpu_resident_moe`), so a tier without them declines a MoE by name, and the plan +declines a router the top-k kernels do not serve - a non-softmax gate, a router or selection +bias, biased or mx4 expert stacks, more than 256 experts or 64 routed slots - by name too. diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_RESIDENCY.md b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_RESIDENCY.md index c9d125e471..beb440ec92 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_RESIDENCY.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_RESIDENCY.md @@ -85,6 +85,24 @@ stays unfilled. It never shrinks any of the three, and it reports zero bytes whe fit - so the same model plans the plane in at a short context and out at a long one. The raw f32 embed table is the one optional plane today. +**A MoE is planned like a dense model with bigger FFN planes.** Its weight planes are the +attention quads, every MoE layer's expert triple (`[ne x nfe x dim]` twice and `[ne x dim x +nfe]`, sliced per expert as the per-op walk gathers them), its shared expert's triple where it +has one - in the file's K-quant format where the loader kept those planes beside the q8 +transcode the CPU chain reads (`wshk*_offs`; it keeps them on a load with a GPU tier armed, so +a CPU-only load carries the transcode alone), the transcode otherwise - the classifier, and the +router plane - every MoE layer's f32 rows with a gated shared expert's gate row beside them. The scratch adds the window's routed planes: the gathered f16 +rows, the gate and up rows, the f16 hidden rows and the routed down rows over `PF_WINDOW x k` +bucket rows plus `TILE_READ_SLACK` rows of tile slack (128 - the m column's unclamped partial +load, `dasllama/dasllama_gpu_tier.das`), and the routing smalls. The dense planes size at the shared +expert's width, or the expert width where no layer has one. The plan is all-or-nothing as ever: +a MoE whose stacks do not fit takes the per-op rails, which stream what the card cannot hold. +The plan is sized BEFORE the per-op reserves (the streamed slot, the decode mirrors), and a +fitting plan forgoes them - they would only shrink its room - so a decline past the plan (a +placement, a class rail) leaves the per-op rails without a streamed slot, said out loud. The +tile family the routed block rides is the f16-fed cm2 tiles (`ARCHITECTURE_GPU_VULKAN_MOE.md` +sec.2.2af), and `DASLLAMA_GPU_RESIDENT=0` keeps the per-op rails for any model, the A/B lever. + ### 2.2o One GPU slot, many models: the marks swap {#gpu-slot-marks} A multi-model host runs one device tier under several loaded models, and the tier's per-model diff --git a/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT.md b/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT.md index 7ff00b5ec0..9c2279158f 100644 --- a/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT.md +++ b/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT.md @@ -80,7 +80,17 @@ reads bit-exact where the lever leaves the arithmetic alone. The shipped class i against the k4 CPU oracle on six corners of the output, at the kernel cell's bar: `|gpu - cpu| <= 2e-2 |cpu| + 4e-3 max|y|`. The probe's exit code is non-zero on a compared arm over its bound, a CPU-oracle miss, an unknown `khrprof` arm, or a run that produced no result -row. +row. The `moe:` and `moesk:` arms are the first axis over the expert schedule: the +format's cm2 s and m tiles over the Qwen3-30B-A3B window's routed buckets (512 tokens, 8 routed +of 128 experts, 4096 bucket rows) at the gate/up shape (d 768, K 2048) and the down shape (d +2048, K 768), dispatched to the bound the device schedule sizes with its sentinel tail, both +decode arms interleaved as `cm2:` runs them. `moe:` gives every expert 32 rows - the +reference harness's uniform profile - and `moesk:` takes the 30B window's own skew (69 experts +route, nine holding the large buckets, eight at 96 rows, four at 48, the rest at 18). Each +profile runs two schedule forms: the whole buckets at the s column, and the ladder, where a +bucket past 32 rows takes whole m columns with the last one partial and a remainder of at most +32 rows past them takes the s column. The reference row is `test-backend-ops perf MUL_MAT_ID` +at `n_mats=128,n_used=8,m=768,n=512,k=2048`. **A measured number proves its kernel provenance through `tune_gate()` (`performance/profile_common.das`), one arm per world it can run in.** Four worlds, because diff --git a/modules/dasLLAMA/ENVIRONMENT.md b/modules/dasLLAMA/ENVIRONMENT.md index 65887dbc30..f953b008c9 100644 --- a/modules/dasLLAMA/ENVIRONMENT.md +++ b/modules/dasLLAMA/ENVIRONMENT.md @@ -37,6 +37,7 @@ Read by the inference engine itself, so these affect any program that loads a mo | `DASLLAMA_GPU_MOE_STREAM` | number | -1 (auto) | How many MoE layers to stream rather than hold resident; -1 is auto. | | `DASLLAMA_GPU_VRAM_MB` | number | probed | Override the detected VRAM budget in MiB that sizes the resident expert stacks. | | `DASLLAMA_GPU_MIN_CTX` | number | built-in floor | Lower the context floor for arming the resident decode driver, for a short-context session on a small card. | +| `DASLLAMA_GPU_RESIDENT` | flag | on | The whole-model resident driver for a model that fits the card, MoE included; 0 keeps the per-op rails (the A/B lever). | | `DASLLAMA_GPU_DN` | flag | follows DASLLAMA_GPU | DeltaNet rail on the GPU. | | `DASLLAMA_GPU_DND` | flag | follows DASLLAMA_GPU | DeltaNet decode rail on the GPU. | | `DASLLAMA_GPU_ATTN` | flag | follows DASLLAMA_GPU | Attention rail on the GPU. | diff --git a/modules/dasLLAMA/HOW_TO_ADD_A_FORMAT.md b/modules/dasLLAMA/HOW_TO_ADD_A_FORMAT.md index dcfc7d9ab4..84ef4123c9 100644 --- a/modules/dasLLAMA/HOW_TO_ADD_A_FORMAT.md +++ b/modules/dasLLAMA/HOW_TO_ADD_A_FORMAT.md @@ -180,7 +180,14 @@ decoded scale row needs no upload work - only the id bridge and the kernels. IQ4 `xscl * ws * idot` (q40's without the `- 8 * bsum`). 4. Ladders: `kq_batch_cls_ensure` / `kq_batch_cls_enc_for` / `gemv_cls_ensure` / `gemv_cls_enc` gain an arm; `gemv_cls_set`'s four-way `||` became `kq_sb(fmt)`. -5. Tests: `tests/_vkd_oracles.das` `kq_cls_ref` arm (the class-on-CPU oracle), the two family +5. The decode GEMV's lanes per row (`dasllama/dasllama_vulkan_classes.das`): join + `gemv_grid_fmt` where the format's decode gathers from a codebook or a grid, and take the + lanes from `gemv_lanes_per_row`, which every decode site reaches through `gemv_enc`. Which + side of that rule the format sits on - grid or k-lattice - is decided by the + `harness/vk_gemv_probe.das ` sweep, its three lane splits per format (0 = the whole + subgroup, 16, 8) over the row lengths the families serve; the rule itself is + `ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2ah (`kq-gemv-lanes`). +6. Tests: `tests/_vkd_oracles.das` `kq_cls_ref` arm (the class-on-CPU oracle), the two family cells in `tests/test_vulkan_kernels.das` go to five formats, and - because the codebook pack is new bit-math that a class-vs-device compare cannot see (both sides run the same `iq4_word`) - `iq4xs_gemv_float_oracle`, a float dequant straight off the plane bytes that diff --git a/modules/dasLLAMA/PERF_LEDGER.md b/modules/dasLLAMA/PERF_LEDGER.md index db3ceb219f..01c6973e34 100644 --- a/modules/dasLLAMA/PERF_LEDGER.md +++ b/modules/dasLLAMA/PERF_LEDGER.md @@ -32,6 +32,25 @@ what it costs today and what the fix would change. vehicles' vulkan pp512 rows re-minted by `gen_bench_records`, direction-grade against the rows they replace; done when they sit in `performance/records/zen2.json`. +- **LANDED (2026-09-09) - the shared expert's K-quant planes are minted beside the q8 + transcode, and the whole-model resident driver reads those.** A MoE's shared expert reached + the device only as the loader's q8 transcode of the file's Q4_K / Q6_K planes - twice the + bytes its decode GEMVs read. The loader now keeps the gate/up/down planes in the file's own + K-quant format beside that transcode (`wsh*_fmt`, `wshk*_offs`; `IMAGE_VERSION` 36, so every + image re-mints once) and the whole-model resident driver places the K-quant copies. On the + Qwen1.5-MoE-A2.7B-Chat Q4_K_M twin the device image reads 10107 -> 9756 MB and the token + 7.06 -> 6.12 ms (the `vk_rdec gpu avg/token` line of `benchmarks/lcpp_bench.das --prof + --jobque-profiling` under `DASLLAMA_GPU_PROF=1`, RTX 5060 Ti) [direction-grade - two commits]. + The pair to read is footprint against wall clock: the device image and the token both fall, + while the loaded model on a GPU box carries BOTH copies - the loader mints the K-quant planes + only on a load with a GPU tier armed, so a CPU-only box's planar image is what it was, and the + tier-armed load of the twin serves 11781 MB of weights against the CPU-only load's 11291 (the + 490 MB: three planes x layers x dim x n_ff_shexp weights in the file's format, k4 +381 MB and + k6 +109). That growth stands until the per-op and CPU rails read the + K-quant planes too and the q8 transcode goes, + which `followup_vulkan 43` owns in its still-open list (the CPU chain's shared expert on the + same K-quant planes; today the resident-vs-CPU bar carries the two forms' rounding). + - **OPEN (narrowed) - the gemma3v encode residual after the tower flash: ~0.92x vs the pair.** The slab road closed in three landings: the 96 head pad (guarded AV columns, 668 -> 486 -> 452), then the LIFTED dk72 flash (MetalTowerFlash + the per-head-contiguous diff --git a/modules/dasLLAMA/REVIEW.das b/modules/dasLLAMA/REVIEW.das index c1b744e4bb..5ab009a70e 100644 --- a/modules/dasLLAMA/REVIEW.das +++ b/modules/dasLLAMA/REVIEW.das @@ -544,8 +544,8 @@ let private IMAGE_FILE = "modules/dasLLAMA/dasllama/dasllama_image.das" // every *_prepare mint, the layout helpers and the layout constants' declaration lines, hashed // in file order. A closure change with IMAGE_VERSION unmoved is red; the finding prints the // value to re-stamp with. -let private IMAGE_LAYOUT_STAMP_VERSION = 35 -let private IMAGE_LAYOUT_STAMP_HASH = 0x4e552ab5bff69e17ul +let private IMAGE_LAYOUT_STAMP_VERSION = 36 +let private IMAGE_LAYOUT_STAMP_HASH = 0x7665c61f19ad379ful // The helpers that decide WHERE bytes land: the page pad, the plane and total sizing, the // writer's append / zero-fill / header patch, and the header's scalar stores. Changing one @@ -737,6 +737,172 @@ def private check_image_layout_stamp { } } +// ===== the Vulkan checklist's four mechanical checks (REVIEW_GPU_VULKAN.md) ===== + +let private VK_CLASSES = "modules/dasLLAMA/dasllama/dasllama_vulkan_classes.das" +let private VK_PREFILL = "modules/dasLLAMA/dasllama/dasllama_vulkan_prefill.das" +let private VK_COMMON = "modules/dasLLAMA/dasllama/dasllama_vulkan_common.das" +let private BLOCKS_FILE = "modules/dasLLAMA/dasllama/dasllama_blocks.das" + +// a das integer literal's value at the head of `s` (digit separators and the l/u suffix dropped), -1 when none +def private int_literal(s : string) : int { + let tok = replace(ident_prefix(strip(s)), "_", "") + var n = length(tok) + peek_data(tok) $(d) { + while (n > 0 && (int(d[n - 1]) == 'l' || int(d[n - 1]) == 'u')) { + n-- + } + } + return try_to_int(slice(tok, 0, n)) ?? -1 +} + +// the lines of the class whose header starts with `header`, in order (the body ends at the first +// line starting with `}`, the file's class convention); empty when the class is not in the file +def private class_lines(path, header : string) : array { + var out : array + var in_cls = false + for (line in split(strip_line_comments(fread(path)), "\n")) { + if (line |> starts_with(header)) { + in_cls = true + continue + } + if (in_cls && line |> starts_with("}")) { + break + } + if (in_cls) { + out |> push(line) + } + } + return <- out +} + +// The cm2 template's KHR stage stays abstract: a default body lets a format template that forgot +// its override compile, and that format decodes garbage on every KHR card while the cm2 arms stay green. +def private check_khr_stage16_abstract { + var found = false + for (line in class_lines(VK_CLASSES, "class template KqCm2BatchT")) { + let s = strip(line) + continue if (!(s |> starts_with("def ")) || find(s, "khr_stage16(") < 0) + found = true + if (!(s |> starts_with("def abstract khr_stage16"))) { + gate_finding(VK_CLASSES, "khr_stage16 on KqCm2BatchT has a body - it stays abstract, so a format template that forgets its override fails to compile instead of decoding garbage on every KHR card") + } + } + if (!found) { + gate_finding(VK_CLASSES, "no `def khr_stage16` inside `class template KqCm2BatchT` - the KHR stage's abstract rule has nothing to read; keep the spelling or move this check with it") + } +} + +// the value the first line of `lines` carrying `marker` declares right after it, -1 when no line does +def private literal_after(lines : array; marker : string) : int { + for (line in lines) { + let at = find(line, marker) + if (at >= 0) { + return int_literal(slice(line, at + length(marker))) + } + } + return -1 +} + +// The add+rms row slab triple agrees: the constant, ArBase's @workgroup row slab, and the dense-width +// cap of the Vulkan servability gate. A cap raised without the slab writes past the slab's end on the device. +def private check_ar_max_dim_triple { + var inscope common_lines <- split(strip_line_comments(fread(VK_COMMON)), "\n") + let constant = literal_after(common_lines, "let AR_MAX_DIM = ") + var inscope ar_lines <- class_lines(VK_CLASSES, "class ArBase") + let slab = literal_after(ar_lines, "@workgroup row : float[") + var gate = -1 + var inscope fns <- function_bodies(BLOCKS_FILE) $(header) { + return header |> starts_with("def private attn_dec_shape_ok") || header |> starts_with("def attn_dec_shape_ok") + } + for (f in fns) { + var inscope body_lines <- split(f.body, "\n") + gate = literal_after(body_lines, "c.dim > ") + } + if (constant < 0) { + gate_finding(VK_COMMON, "no `let AR_MAX_DIM = N` line - the add+rms slab triple has no constant to compare against") + } + if (slab < 0) { + gate_finding(VK_CLASSES, "no `@workgroup row : float[N]` inside `class ArBase` - the add+rms slab triple has no slab to compare") + } + if (gate < 0) { + gate_finding(BLOCKS_FILE, "no `c.dim > N` inside attn_dec_shape_ok - the add+rms slab triple has no servability cap to compare") + } + return if (constant < 0 || slab < 0 || gate < 0) + if (constant != slab || constant != gate) { + gate_finding(VK_CLASSES, "the add+rms row slab triple disagrees: AR_MAX_DIM {constant} (dasllama_vulkan_common.das), ArBase.row[{slab}], attn_dec_shape_ok's c.dim cap {gate} (dasllama_blocks.das) - a cap past the slab writes past its end on the device; change the three together") + } +} + +// q8 is no superblock format: its cm2 tiles are the fmt-0 cells with no KHR arm, and the KHR +// mode serves q8 through its own tile - the one template the KHR set does not reach. +var private CM2_KHR_EXEMPT <- {"Q8Cm2T"} + +// the `Cm2T` format templates on the cm2 base, by name +def private cm2_format_templates : array { + var out : array + for (line in split(strip_line_comments(fread(VK_CLASSES)), "\n")) { + continue if (!(line |> starts_with("class template "))) + let name = ident_prefix(slice(line, length("class template "))) + if (name |> ends_with("Cm2T") && find(line, ": KqCm2BatchT") >= 0 && !key_exists(CM2_KHR_EXEMPT, name)) { + out |> push(name) + } + } + return <- out +} + +// Every superblock format on the cm2 template ships its KHR instantiation and its arm in each of the +// prefill's three KHR ladders: the f16 feed admits every such format, so a format missing one panics +// inside the ladder on a card whose cooperative-matrix mode is KHR - a card class this box is not. +def private check_cm2_khr_set { + var inscope templates <- cm2_format_templates() + if (empty(templates)) { + gate_finding(VK_CLASSES, "no `class template Cm2T : KqCm2BatchT` found - the KHR set rule has no templates to walk; keep the spelling or move this check with it") + return + } + let classes = strip_line_comments(fread(VK_CLASSES)) + var inscope ladders <- function_bodies(VK_PREFILL) $(header) { + return header |> starts_with("def khr_cls_ensure") || header |> starts_with("def khr_cls_set") || header |> starts_with("def khr_cls_enc") + } + for (tpl in templates) { + let pfx = slice(tpl, 0, length(tpl) - length("Cm2T")) + let fmt = to_lower(pfx) + if (find(classes, "class {pfx}KhrBatch : {tpl}") < 0) { + gate_finding(VK_CLASSES, "{tpl} has no `class {pfx}KhrBatch : {tpl}` - a superblock format on the cm2 template ships its KHR instantiation in the same change, or the KHR ladders panic on a KHR-mode card") + } + if (find(classes, "name = \"kq_batch_{fmt}_khr_cls\"") < 0) { + gate_finding(VK_CLASSES, "{tpl} has no `[vk_dispatch(name = \"kq_batch_{fmt}_khr_cls\"` stamp - its KHR class has no dispatch") + } + for (f in ladders) { + let ladder = ident_prefix(slice(f.body, length("def "))) + let arm = "{slice(ladder, length("khr_cls_"))}_kq_batch_{fmt}_khr_cls(" + if (find(f.body, arm) < 0) { + gate_finding(VK_PREFILL, f.line, "{ladder} has no arm for {tpl} ({arm}) - the f16 feed admits the format and the ladder panics on a KHR-mode card") + } + } + } +} + +// A Vulkan pipeline is built only by the `[vk_dispatch]`-generated `ensure_*`: a hand build bypasses the +// hazard rail and the dispatch census, so its races and its coverage are invisible. The two llama.cpp +// shader ports under performance/ are reference measurements of another engine's kernels, outside this walk. +def private check_no_hand_pipelines { + var files : array + for (folder in ["modules/dasLLAMA/dasllama", "modules/dasLLAMA/harness", "modules/dasLLAMA/tests"]) { + collect_das_files(folder, files) + } + for (p in files) { + var line_no = 0 + for (line in split(strip_line_comments(fread(p)), "\n")) { + line_no++ + if (find(line, "vkCreateComputePipelines(") >= 0) { + gate_finding(p, line_no, "hand-built Vulkan pipeline - a pipeline is created only by a [vk_dispatch]-generated ensure_*, which puts it on the hazard rail and in the dispatch census") + } + } + } + delete files +} + [export] def main() : int { if (!fexist("modules/dasLLAMA/REVIEW.das") || !fexist(FACADE)) { @@ -750,6 +916,10 @@ def main() : int { check_gpu_role_partition() check_fastmath_default() check_image_backing_release() + check_khr_stage16_abstract() + check_ar_max_dim_triple() + check_cm2_khr_set() + check_no_hand_pipelines() var inscope tut_texts : array read_das_stripped(TUTORIAL_DIR, tut_texts) var inscope rst_texts : array diff --git a/modules/dasLLAMA/REVIEW.md b/modules/dasLLAMA/REVIEW.md index a8898934b4..1bbd82f5ff 100644 --- a/modules/dasLLAMA/REVIEW.md +++ b/modules/dasLLAMA/REVIEW.md @@ -1,8 +1,8 @@ # dasLLAMA Code Review Checklist **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture -docs: `ARCHITECTURE.md`, `ARCHITECTURE_ENGINE.md`, `ARCHITECTURE_MEASUREMENT.md` (the other -companions belong to the routed checklists). Planned work: `followup_general.md`, +docs: `ARCHITECTURE.md`, `ARCHITECTURE_ENGINE.md`, `ARCHITECTURE_RUNTIME.md`, `ARCHITECTURE_MEASUREMENT.md` +(the other companions belong to the routed checklists). Planned work: `followup_general.md`, `followup_vulkan.md`, `followup_metal.md` (the Metal tier, and CPU work measured on macOS), `PERF_LEDGER.md` (performance goes to the perf ledger, everything else to the followup ledgers). @@ -291,10 +291,10 @@ root) - is a `def` returning it, never a module global with a declaration initia or `var`).** A team lane never runs global initializers, so the global reads zero there while every single-threaded run reads the right value. -**A `resize` in `dasllama/` of a buffer whose element count scales with a model dimension is -preceded by a `reserve` of the same count - a `dasllama/dasllama_common.das` sizing helper that -reserves before it grows (`reserve_resize`, `grow_resize`, `ensure_length`, `overwrite_resize`, -`zeroed_resize`), the builtin `scratch_resize` on a `@scratch` carrier, or the pair spelled -out - whatever the size looks like at today's shapes.** A model dimension makes -the count unbounded, and a bare grow past the heap's unreserved-size cap (64 MB) panics the -load on the first big model rather than at the call site. +**A `resize` in `dasllama/` of a buffer whose element count scales with a model dimension (a +count the model file sets: layers, dim, experts, vocab, positions) is preceded by a `reserve` of +the same count - a `dasllama/dasllama_common.das` sizing helper that reserves before it grows +(`reserve_resize`, `grow_resize`, `ensure_length`, `overwrite_resize`, `zeroed_resize`), the +builtin `scratch_resize` on a `@scratch` carrier, or the pair spelled out - whatever the size +looks like at today's shapes.** Such a count is unbounded, and a bare grow past the heap's +unreserved-size cap (64 MB) panics the load on the first big model rather than at the call site. diff --git a/modules/dasLLAMA/REVIEW_GPU.md b/modules/dasLLAMA/REVIEW_GPU.md index b909a9ec6c..c05b3d11ef 100644 --- a/modules/dasLLAMA/REVIEW_GPU.md +++ b/modules/dasLLAMA/REVIEW_GPU.md @@ -153,11 +153,11 @@ is read, and its unread arm binds a placeholder the kernel never touches. differ on the compile-time choice the template carried, and names that choice in the surviving template's comment.** -**A `[metal_dispatch]` / `[vk_dispatch]` binding whose memory is load-once - a model plane, or -an `upload_region` upload never written after arming - is a defect unless a field at that -binding carries `@role = "weight"`, even when the kernel compiles and passes parity.** A field -the kernel reads under a run-time flag takes the role of its read arm; the placeholder its -unread arm binds is never read, so its lifetime does not decide the role. +**A `[metal_dispatch]` / `[vk_dispatch]` binding whose memory is never written after arming at +every site that binds it - a model plane, an `upload_region` upload - is a defect unless a field +at that binding carries `@role = "weight"`, even when the kernel compiles and passes parity.** +A field the kernel reads under a run-time flag takes the role of its read arm; the placeholder +its unread arm binds is never read, so its lifetime does not decide the role. **`@role = "weight"` on per-encode data the kernel reads - a pooled buffer the host refills each encode - is a defect; a per-encode field either omits `@role` or names the access its body @@ -169,11 +169,10 @@ instance of a template carrying one - either adds a census row to `CENSUS_NEVER_DISPATCHED` with the reason no row can reach it.** **A diff that adds a Vulkan kernel class under `dasllama/` - a `[vk_dispatch]` declaration, or a -new instance of a template carrying one - adds a row to the Vulkan serving census in -`tests/test_kernel_coverage.das` that dispatches it: a census model that reaches the class, or -an arm that forces the device mode the class is gated on; a class no census model reaches gets -a census model that does.** A Vulkan class never joins `CENSUS_NEVER_DISPATCHED`, which takes -Metal classes only. +new instance of a template carrying one - shows a Vulkan serving-census row in +`tests/test_kernel_coverage.das` that dispatches it, adding the row or the census model when +none does.** A Vulkan class never joins `CENSUS_NEVER_DISPATCHED`, which takes Metal classes +only. **Weakening the `[metal_dispatch]` / `[vk_dispatch]` lens's refusal to compile an `@ssbo` field with no `@binding`, or an `@ssbo` field the kernel body never accesses that declares no `@role`, @@ -201,8 +200,9 @@ host-side never reaches the device, so it does not count. **Never bind a scalar that the other bound scalars already determine - derive it in the builder instead.** Binding it separately adds a second place to get it wrong. -**Never key a cache on a host address alone - carry the span and the form, the element type and -layout the upload produces, in the key too.** A hit must cover the request. +**A cache key covers every input the cached result depends on: a host address, an offset, or a +handle alone is not a key - carry the span and the form, the element type and layout the upload +produces, in the key too.** A hit must cover the request. **A diff that lands a kernel class, driver arm, or backend capability in a `dasllama/` file whose `ARCHITECTURE_GPU.md` sec.1.5 role row does not sanction it extends that row's ledger in @@ -226,7 +226,8 @@ backend-only capability - a hook in sec.1.5's per-driver registered-hook or borr lists included - lands its own entry in `ARCHITECTURE_GPU.md` sec.1.5's closed asymmetry list in the same change, even when that list already carries an asymmetry of the same class.** One backend serving the same path faster or slower is not such a change; a seat of the -`dasllama_gpu_tier` cooperation SPI is sec.1.5's tier role row's, not this list's. +`dasllama_gpu_tier` cooperation SPI lands its entry in sec.1.5's tier role row instead, in the +same change. **A change to code that a served GPU decode or prefill path executes ships GPU-vs-CPU parity on one q8 and one kq (K-quant) model the changed path serves.** That code is anything a diff --git a/modules/dasLLAMA/REVIEW_GPU_RACE.md b/modules/dasLLAMA/REVIEW_GPU_RACE.md index 681f3ca95e..40580a6f0c 100644 --- a/modules/dasLLAMA/REVIEW_GPU_RACE.md +++ b/modules/dasLLAMA/REVIEW_GPU_RACE.md @@ -9,10 +9,10 @@ together with it.** **A hand-binding arm that binds a field at a position the class does not declare for that field is a defect.** A hand-binding arm is a race or knockout timing arm - a race times two -implementations of one computation on one queue and compares their outputs, a knockout skips a -stage to measure that stage's cost - that mirrors a class's binding order by hand, with a -literal bind number or a positional buffer and size array, instead of naming the class's fields. -A mis-bound arm dispatches, reads the wrong buffer, and its timing selects the wrong kernel +implementations of one computation on one queue, a knockout skips a stage to measure that +stage's cost - that restates a bind order no generated setter checks (Metal `kn_buffer(enc, n)` +calls, a probe class redeclaring `@binding` slots), instead of naming the class's fields. A +mis-bound arm dispatches, reads the wrong buffer, and its timing selects the wrong kernel silently. **A hand-binding arm outside `dasllama/`, or one whose pipeline source or threadgroup-memory @@ -47,10 +47,10 @@ count; the tile's own width where it is dispatched at one fixed width; each powe spans on a power-of-two batch grid.** A ranking timed at one width alone is applied at widths it was never ranked at. -**A timing arm for a prefill tile over a variable window, whose ranking a checked-in document, -box profile or sidecar records as decided, times its kernel at one window whose token count is -a whole multiple of the tile's row count and at one where it is not.** The short last tile is -what makes the tile take its partial-tile store path. +**A timing arm for a prefill tile over a variable region, whose ranking a checked-in document, +box profile or sidecar records as decided, times its kernel at one region whose row count is a +whole multiple of the tile's row count and at one where it is not.** A region that is not a +whole multiple is what makes the tile take its partial-tile store path. **An `ARCHITECTURE_GPU.md` sec.2.2b entry for a kernel ranked on a power-of-two batch grid names that grid.** diff --git a/modules/dasLLAMA/REVIEW_GPU_VULKAN.md b/modules/dasLLAMA/REVIEW_GPU_VULKAN.md index f10dd56eb8..4350bd5940 100644 --- a/modules/dasLLAMA/REVIEW_GPU_VULKAN.md +++ b/modules/dasLLAMA/REVIEW_GPU_VULKAN.md @@ -1,18 +1,20 @@ # dasLLAMA Vulkan Tier Code Review Checklist **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture -docs: `ARCHITECTURE_GPU_VULKAN.md`, `ARCHITECTURE_GPU_VULKAN_GEMM.md`. Planned work: +docs: `ARCHITECTURE_GPU_VULKAN.md` and the companions it routes to. Planned work: `followup_vulkan.md`. **Routed from `REVIEW_GPU.md`: a diff that checklist routes here applies this list together with `REVIEW_GPU.md`'s and `REVIEW.md`'s.** -**A hand-written Vulkan pipeline build is a defect - a Vulkan pipeline is created only by a -`[vk_dispatch]`-generated `ensure_*`.** +**A hand-written Vulkan pipeline build is a defect, and so is weakening the `REVIEW.das` check +that reports a `vkCreateComputePipelines(` call in `dasllama/`, `harness/` and `tests/` - a +Vulkan pipeline is created only by a `[vk_dispatch]`-generated `ensure_*`.** -**A diff that adds a Vulkan dispatch family adds its model-owned device buffers, descriptor-set -caches and `*_ready` latch to `vk_drop_model_state`'s sweep, in the same change.** Pipelines -are device-lifetime state that survives the drop and rebuilds lazily. +**A diff that adds a Vulkan dispatch family adds every piece of state the family keeps per +model - device buffers, descriptor-set caches, `*_ready` latches, profiler accumulators - to +`vk_drop_model_state`'s sweep, in the same change.** Pipelines are device-lifetime state that +survives the drop and rebuilds lazily. **Never size a buffer bound as one SSBO (shader storage buffer) range above `vk_max_storage_range()` - check the size at the site that computes it, not at the site that @@ -22,11 +24,12 @@ binds it.** The bind site cannot shrink a buffer that was sized wrong. dispatches in state that `vk_drop_model_state` does not clear** - hold it in `dasllama/dasllama_vulkan_common.das` module state that `vk_drop_model_state` clears. -**Never read a weight plane's quant bytes in a kernel body by indexing `unpack8` of a 32-bit -word with a runtime value - read them as 16-bit lanes instead: load the lane `w` from an -`int16[N]` block member, select the byte with `unpack8(w)[i & 1u]`, and pull sub-fields out by -shift and mask.** The vendor driver's shader compiler pattern-matches only the 16-bit spelling -into its block-load path, and a runtime byte select loses that path for the whole kernel. +**Never take a quant byte out of an `unpack8` select in a cm2 decode body or its four-wide twin +(`decode_v4`) - load the 16-bit lane from the `int16[N]` block member and shift the byte out, +and assemble a field that straddles two lanes from those lanes, never from selected bytes.** +Indexing `unpack8` of a 32-bit word with a runtime value drops the whole kernel off the +driver's block-load path, about 3x slower, and an `unpack8` lane select costs another 1.1x to +1.5x over the shift form. **A diff that changes when `vk_rdec_prefill_ids` - the resident prefill that takes token ids rather than embeddings - accepts a call, or when the override that routes to it @@ -36,10 +39,11 @@ change** - the override and the gate live in `dasllama/dasllama_gpu_resident.das embed when that gate returns true, so a gate true where the prefill path declines hands the next consumer an unfilled residual stream. -**A Vulkan-tier serving gate that decides at load - a predicate or per-layer loop whose false -branch or `continue` routes work to the CPU path - that does not log at load how many layers or -planes it left on the CPU and the reason it left them is a defect.** A silent decline is a -fallback a user finds only by profiling. +**A Vulkan-tier serving gate that decides at load - a predicate or per-layer loop whose true +branch dispatches through a `moe_gpu_*` or `vk_*` entry, wherever the diff puts it, and whose +false branch or `continue` routes work to the CPU path - that does not log at load how many +layers or planes it left on the CPU and the reason it left them is a defect.** A silent decline +is a fallback a user finds only by profiling. **A Vulkan-tier serving gate that decides per call - a predicate or loop whose false branch or `continue` routes work to the CPU path - that does not log the concrete reason it declined, @@ -65,25 +69,29 @@ cm2 tile is the NV_cooperative_matrix2 GEMM class stamped per weight format and width (the class's `BN`) in `dasllama/dasllama_vulkan_classes.das`. **A diff that changes what a kq superblock format's KHR tile emits - its `khr_stage16` -override, its instance set, or the shared `khr_tile` or `run` of `KqCm2BatchT` - puts that -format's kernel cell - that format's test block in `tests/test_vulkan_kernels.das` - run on its -KHR arm in the PR body, with either the `khrx` probe rows (`harness/vk_gemm_probe.das`) or a -`tests/test_gpu_resident_hybrid.das` run on a model in that format.** A KHR tile is the -`KhrBatch` class stamped per weight format in `dasllama/dasllama_vulkan_classes.das`. +override, a constant or typedef the KHR arm's emitted code reads, or the shared `khr_tile` or +`run` of `KqCm2BatchT` - puts that format's kernel cell - that format's test block in +`tests/test_vulkan_kernels.das` - run on its KHR arm in the PR body, with either the `khrx` +probe rows (`harness/vk_gemm_probe.das`) or a `tests/test_gpu_resident_hybrid.das` run on a +model in that format, or the claim that the format's KHR-stamped kernels are byte-identical to +master's.** A KHR tile is the `KhrBatch` class stamped per weight format in +`dasllama/dasllama_vulkan_classes.das`. **A `kq_sb` format (`dasllama/dasllama_kqformat.das`) that joins the cm2 template - a `Cm2T` format template in `dasllama/dasllama_vulkan_classes.das` - ships its KHR instantiation (`KhrBatch`, the `kq_batch__khr_cls` dispatch) and its arm in each of `khr_cls_ensure`, `khr_cls_set` and `khr_cls_enc` (`dasllama/dasllama_vulkan_prefill.das`), in -the same change.** `pf_f16_feed` admits every `kq_sb` format, so a format with no KHR -class panics in `khr_cls_ensure`, `khr_cls_set` or `khr_cls_enc` on a card whose -cooperative-matrix mode is KHR. +the same change; weakening the `REVIEW.das` check that requires that set for every `kq_sb` +format's `Cm2T` template is a defect.** `pf_f16_feed` admits every `kq_sb` format, so a +format with no KHR class panics in `khr_cls_ensure`, `khr_cls_set` or `khr_cls_enc` on a card +whose cooperative-matrix mode is KHR. **A `kq_sb` format that ships a KHR instantiation runs its KHR arm in that format's kernel cell, in the same change.** -**`khr_stage16` stays abstract on `KqCm2BatchT` (`dasllama/dasllama_vulkan_classes.das`) - a -diff that gives it a default body is a defect.** +**Weakening the `REVIEW.das` check that requires `def abstract khr_stage16` on `class template +KqCm2BatchT` (`dasllama/dasllama_vulkan_classes.das`) is a defect.** A default body lets a format +template that forgot its override compile and decode garbage on every KHR card. **A kernel body that calls a `[spirv_decode]` method directly passes the plane element itself (`decode(wq[i], ...)`), never a local copy of it (`let blk = wq[i]` then `decode(blk, ...)`).** @@ -101,17 +109,18 @@ on the scalar callback.** With `DECV4 = true` the class never reads `DECVEC`, so **A diff that changes how many GPU timestamps the resident decode's token command records - the `pfq_ts` calls in `dasllama/dasllama_vulkan_decode.das` - updates the stamp count `rdq_sample` -expects and, for every layer kind whose count moved, that file's role-name table -(`rdq_role_names`, `RDQ_DN_NAMES`) and its accumulator in `dasllama/dasllama_vulkan_common.das` -(`g_rdq_role`, `g_rdq_dn`), in the same change.** `rdq_sample` indexes a fixed count per layer, -so one extra or missing timestamp reports every later stamp under the wrong role name. +expects and, for every layer kind whose count moved, that file's role-name tables +(`rdq_role_names`, `RDQ_DN_NAMES`, `rd_moe_tail_names`) and the matching accumulators in +`dasllama/dasllama_vulkan_common.das` (`g_rdq_role`, `g_rdq_dn`, `g_rdq_moe`), in the same +change.** `rdq_sample` indexes a fixed count per layer, so one extra or missing timestamp +reports every later stamp under the wrong role name. **A decode GEMV class - a `KqGemvBase` leaf in `dasllama/dasllama_vulkan_classes.das` - that stages a codebook into `@workgroup` memory reads it from the family's grid buffer (`gridb`, filled by `kq_grid_dev` at the format's `KQ_GRID_` offset), never from a `*_grid_word` -accessor.** The accessor is a constant composite the driver reads lane-serially -per index, and a two-row workgroup pays that read on every row pair it walks -(`ARCHITECTURE_GPU_VULKAN.md` sec.2.2ab). +accessor.** The accessor is a constant composite the driver reads lane-serially per index, +while the grid buffer is staged once per workgroup and read by every row that workgroup serves, +whatever their number (`ARCHITECTURE_GPU_VULKAN.md` sec.2.2ab). **A diff that changes how many GPU timestamps the resident prefill's window command records - a `pfq_ts` call in `pf_run` or in any function `pf_run` reaches, all in @@ -119,8 +128,8 @@ per index, and a two-row workgroup pays that read on every row pair it walks `pf_prof_report` in the same change.** Both index a fixed count per layer, so one extra or missing timestamp reports every later stamp under the wrong role name. -**A diff that changes `AR_MAX_DIM` (`dasllama/dasllama_vulkan_common.das`) changes the `row` -`@workgroup` slab of `ArBase` (`dasllama/dasllama_vulkan_classes.das`) and the `c.dim` cap of -the Vulkan servability gate (`attn_dec_shape_ok`, `dasllama/dasllama_blocks.das`) to the same -number, in the same change.** The add+rms kernels stage a whole row in that slab, so a slab -shorter than the cap writes past its end. +**Weakening the `REVIEW.das` check that compares `AR_MAX_DIM` +(`dasllama/dasllama_vulkan_common.das`), the `row` `@workgroup` slab of `ArBase` +(`dasllama/dasllama_vulkan_classes.das`) and the `c.dim` cap of `attn_dec_shape_ok` +(`dasllama/dasllama_blocks.das`) is a defect.** The add+rms kernels stage a whole row in that +slab, so a cap past the slab writes past its end. diff --git a/modules/dasLLAMA/REVIEW_PLACEMENT.md b/modules/dasLLAMA/REVIEW_PLACEMENT.md index 976bbc2ef1..af0417ff4b 100644 --- a/modules/dasLLAMA/REVIEW_PLACEMENT.md +++ b/modules/dasLLAMA/REVIEW_PLACEMENT.md @@ -13,7 +13,8 @@ checklist's own. **A function whose KIND the file's sec.1 charter line seats in another file lands in that file, or the charter line changes in the same diff.** -**A tensor format conversion lands in `dasllama/dasllama_convert.das`.** +**A HOST-side tensor format conversion lands in `dasllama/dasllama_convert.das`; a kernel-side +decode helper rides its backend's kernel home.** **A disk-order -> compute-order transform lands by its consumer: a transform into the layout a CPU row core reads in `dasllama/dasllama_repack.das`, a transform into the layout a GPU plane @@ -38,10 +39,11 @@ a template declared elsewhere is not a kernel body: it compiles its own PSO wher **A quirk of one family - one model architecture's file, or one backend driver's - lands in that file, never sideways into a sibling.** -**A piece two files need lands in their nearest shared file (its own file when none exists) - -never a second copy.** A predicate, a constant, or a helper spelled once in each of two files -drifts on the first edit to one copy; an enum-and-int twin of one predicate inside one file is -the tier's idiom, not a copy. A piece two folders outside each other both need lands in the folder that +**A piece two files both execute lands in their nearest shared file (its own file when none +exists) - never a second copy.** A predicate, a constant, or a helper spelled once in each of +two files drifts on the first edit to one copy; an enum-and-int twin of one predicate inside one +file is the tier's idiom, not a copy, and a test's CPU oracle that restates the arithmetic is a +witness, not a copy. A piece two folders outside each other both need lands in the folder that owns the concern; one landing under `dasllama/` that code outside `modules/dasLLAMA/` drives lands as a public entry module - one `dasllama/dasllama_lint.das` licenses a consumer to require directly. @@ -49,7 +51,7 @@ require directly. **A family gaining an arm for a media kind adds that kind's span markers to that family's chat template, never to a second renderer.** Span markers are the template text that opens and closes the media rows. A family whose template or vocab lacks them has no arm for that media -kind - `create_chat_` panics at create, not at render. +kind. **No signature in `dasllama/dasllama_tower.das` takes a type that `dasllama/dasllama_audio.das`, `dasllama/dasllama_vision.das`, or a family file declares.** @@ -84,9 +86,10 @@ it does, `dasllama/dasllama_common.das` panics on the unset hook with a message module to require. A program root (test, harness, benchmark, tool) requires the registration module it needs directly. -**A `dasllama/` module whose `[init]` registers a hook the engine dispatches through is -required from `dasllama/dasllama_transformer.das` in the same change that adds it** - a -registration no umbrella reaches never fires for a consumer of the `dasllama.das` facade. +**A `dasllama/` module whose `[init]` registers a hook the engine dispatches through gets its +side-effect require in the same change that adds it - in `dasllama/dasllama_transformer.das`, +or in `dasllama/dasllama_common.das` where the rule above seats it there** - a registration +neither file reaches never fires for a consumer of the `dasllama.das` facade. **An architecture file (`dasllama/dasllama_arch_*.das`) that changes a forward loop, or tests a family name on a shared path, is a defect - it carries declarative registration only.** diff --git a/modules/dasLLAMA/dasllama/dasllama_blocks.das b/modules/dasLLAMA/dasllama/dasllama_blocks.das index 93f97aa73d..48c79b6b90 100644 --- a/modules/dasLLAMA/dasllama/dasllama_blocks.das +++ b/modules/dasLLAMA/dasllama/dasllama_blocks.das @@ -663,6 +663,8 @@ def private attn_dec_store_kv(t : Model; var s : Session; l, pos : int64) { var private g_span_args : MoeSpanDec // reused per token: the per-layer arrays keep their capacity var private g_span_k : array var private g_span_v : array +var private g_span_tokens = 0l //! tokens the span served - the tests' engage witness (read on the main context only) +def public span_tokens() : int64 => g_span_tokens [arch(at="../ARCHITECTURE_GPU_VULKAN_DECODE.md#whole-token-decode-span")] def private span_first_layer(t : Model; var s : Session; pos : int64) : int64 { @@ -671,7 +673,7 @@ def private span_first_layer(t : Model; var s : Session; pos : int64) : int64 { while (l0 > 0l) { let l = l0 - 1l if (l < c.n_layer_dense_lead || !moe_gpu_layer_on_gpu(l) || kq_sb(fmt_at(t.we1_fmt, l)) != kq_sb(fmt_at(t.we3_fmt, l)) - || !attn_dec_gpu_serves(t, s, l, pos)) { + || (c.n_ff_shexp > 0l && !moe_gpu_shexp_on_gpu(l)) || !attn_dec_gpu_serves(t, s, l, pos)) { break } l0 = l @@ -685,7 +687,7 @@ def private span_model_ok(t : Model) : bool { if (!g_env_gpu.gpu_dec_span || !moe_gpu_span_ok() || t.quant != QuantMode.q8 || c.n_expert == 0l || c.n_expert_used > 64l || c.n_expert > 256l || c.moe_gate != MoeGate.softmax || c.moe_router_bias || c.moe_exp_probs || c.n_layers * c.n_expert * c.dim * 4l > moe_gpu_binding_cap() || !t.blocks.std_chain - || c.n_ff_shexp > 0l || c.pre_post_norm || c.layer_out_scale || c.moe_exps_bias || t.experts_mx4) { + || (c.n_ff_shexp > 0l && c.moe_dense_shexp) || c.pre_post_norm || c.layer_out_scale || c.moe_exps_bias || t.experts_mx4) { return false } return c.ffn_act != FfnAct.swiglu_oai && !has_ple(c) && c.n_layer_nextn == 0l @@ -709,6 +711,12 @@ def private span_fill_layers(t : Model; var a : MoeSpanDec; l0, nl : int64) { a.f1 |> resize(nl) a.f3 |> resize(nl) a.f2 |> resize(nl) + a.wsh1 |> reserve(nl) + a.wsh1 |> resize(nl) + a.wsh3 |> reserve(nl) + a.wsh3 |> resize(nl) + a.wsh2 |> reserve(nl) + a.wsh2 |> resize(nl) a.rmsq |> resize(nl) a.rmsk |> resize(nl) a.bq |> reserve(nl) @@ -734,6 +742,10 @@ def private span_fill_layers(t : Model; var a : MoeSpanDec; l0, nl : int64) { a.f1[li] = int(fmt_at(t.we1_fmt, l)) a.f3[li] = int(fmt_at(t.we3_fmt, l)) a.f2[li] = int(fmt_at(t.we2_fmt, l)) + let she = dim * c.n_ff_shexp + a.wsh1[li] = c.n_ff_shexp > 0l ? t.wsh1_off + l * she : -1l + a.wsh3[li] = c.n_ff_shexp > 0l ? t.wsh3_off + l * she : -1l + a.wsh2[li] = c.n_ff_shexp > 0l ? t.wsh2_off + l * she : -1l a.rmsq[li] = c.qk_norm ? addr(t.fblob[t.rms_q_offs[l]]) : null a.rmsk[li] = c.qk_norm ? addr(t.fblob[t.rms_k_offs[l]]) : null a.bq[li] = c.attn_qkv_bias ? addr(t.bq[l * a.qd]) : null @@ -741,6 +753,7 @@ def private span_fill_layers(t : Model; var a : MoeSpanDec; l0, nl : int64) { a.bv[li] = c.attn_qkv_bias ? addr(t.bv[l * a.kvd]) : null } a.router = addr(t.fblob[t.router_off + l0 * c.n_expert * dim]) + a.shgate = c.n_ff_shexp > 0l && c.moe_shexp_gated ? addr(t.fblob[t.shexp_gate_off + l0 * dim]) : null a.att_norm = addr(t.fblob[t.rms_att_off + l0 * dim]) a.ffn_norm = addr(t.fblob[t.rms_ffn_off + l0 * dim]) a.blob = addr(t.fblob[0]) @@ -787,6 +800,8 @@ def private span_run(t : Model; var s : Session; pos, l0 : int64) { a.qk_norm = c.qk_norm a.norm_topk = c.norm_topk_prob a.is_gelu = c.ffn_act == FfnAct.gelu + a.nsh = c.n_ff_shexp + a.sh_gated = c.moe_shexp_gated if (!span_args_current(a, t, l0, nl)) { span_fill_layers(t, a, l0, nl) } @@ -841,6 +856,7 @@ def private moe_span_decode(t : Model; var s : Session; _token, pos : int64) : b prof_add("blk_ffn", ts_blk_ffn) } span_run(t, s, pos, l0) + g_span_tokens++ return true } @@ -1461,6 +1477,17 @@ def set_moe_gpu_prefill_route(on : bool) { def gpu_prefill_route_on() : bool => g_gpu_prefill_route +var private g_shexp_gpu_layers = 0l //! layers whose shared expert the device chain served in a prefill - the arm's engage witness (read on the main context only) +def public shexp_gpu_prefill_layers() : int64 => g_shexp_gpu_layers + +//! why this prefill's shared expert stays on the CPU rail: the model's shape, the tier's marks, or the call's size and route +def private shexp_gpu_decline_reason(c : Config; l, npos : int64) : string { + if (c.moe_dense_shexp) return "the model's shared expert is a dense FFN, not the device chain's call shape" + if (!moe_gpu_shexp_on_gpu(l)) return "the shared expert's planes are not resident on the tier" + if (npos < 32l) return "a prefill under 32 positions" + return "the prefill route lever is off" +} + [arch(at="../ARCHITECTURE_GPU_VULKAN_DECODE.md#streamed-layer-split"), arch(at="../ARCHITECTURE_RUNTIME.md#moe-region-split")] def private ffn_moe_prefill_grouped(t : Model; var s : Session; l : int64; npos : int64) { // nolint:STYLE037,STYLE038 — the grouped MoE prefill is one bucketed pipeline over experts; each stage feeds the next through shared scratch let c = t.config @@ -1590,6 +1617,46 @@ def private ffn_moe_prefill_grouped(t : Model; var s : Session; l : int64; npos requant_rows_q8(s.xb_b, dim, npos, s.xqb, s.xsb) } + // 2b. the shared expert on the device + let sh_gpu = (nsh > 0l && !c.moe_dense_shexp && moe_gpu_shexp_on_gpu(l) && npos >= 32l && gpu_prefill_route_on()) + if (nsh > 0l && !sh_gpu) { + moe_gpu_shexp_declined(shexp_gpu_decline_reason(c, l, npos)) + } + if (sh_gpu) { + trace_tag(TRACE_TAG_SHEXP) + let ts_shg = prof_ticks() + let she = dim * nsh + let q8 = int(KqFmt.q8) + s.moe_shout |> zeroed_resize(npos * dim) + s.moe_offs1 |> clear() + s.moe_offs1 |> push(t.wsh1_off + l * she) // nolint:STYLE012 — one region into reused scratch + s.moe_offs1 |> push(0l) + s.moe_offs1 |> push(npos) + s.moe_offs3 |> clear() + s.moe_offs3 |> push(t.wsh3_off + l * she) // nolint:STYLE012 — one region into reused scratch + s.moe_offs3 |> push(0l) + s.moe_offs3 |> push(npos) + s.moe_offs2 |> clear() + s.moe_offs2 |> push(t.wsh2_off + l * she) // nolint:STYLE012 — one region into reused scratch + s.moe_offs2 |> push(0l) + s.moe_offs2 |> push(npos) + if (gpu_combine_on() && moe_gpu_ffn_xf_ok(npos, dim, q8, q8, q8)) { + s.moe_sh_inv |> resize(npos) + s.moe_sh_w |> resize(npos) + for (p in range64(npos)) { + s.moe_sh_inv[p] = uint(p) + s.moe_sh_w[p] = 1.0 + } + matmul_moe_gpu_ffn_combined_xf(s.moe_shout, s.moe_offs1, s.moe_offs3, s.moe_offs2, 1l, + s.xb_b, s.moe_sh_w, s.moe_sh_inv, dim, nsh, npos, npos, q8, q8, q8, c.ffn_act == FfnAct.gelu) + } else { + matmul_moe_gpu_ffn(s.moe_shout, s.moe_offs1, s.moe_offs3, s.moe_offs2, 1l, + s.xqb, s.xsb, dim, nsh, npos, q8, q8, q8, c.ffn_act == FfnAct.gelu) + } + prof_add("shexp_gpu", ts_shg) + g_shexp_gpu_layers++ + } + // 3F. fused: gate/up/down each run as ONE region-list dispatch (3 fork/joins/layer, not 3/expert) if (fused) { trace_tag(TRACE_TAG_MOE) @@ -1896,7 +1963,7 @@ def private ffn_moe_prefill_grouped(t : Model; var s : Session; l : int64; npos } // 4. shared expert for ALL positions — a plain dense batched FFN off the whole-batch image - if (nsh > 0l) { + if (nsh > 0l && !sh_gpu) { trace_tag(TRACE_TAG_SHEXP) let she = dim * nsh mm_b_q8_pre(s.moe_ghb, t, t.wsh1_off + l * she, s.xqb, s.xsb, dim, nsh, npos) @@ -1916,7 +1983,7 @@ def private ffn_moe_prefill_grouped(t : Model; var s : Session; l : int64; npos let ep = addr(s.moe_eout[0]) let wp = split_gpu_regions > 0l ? addr(s.moe_w_cpu[0]) : addr(s.moe_w_b[0]) let split_gpu_rows = split_gpu_regions > 0l ? addr(s.moe_gpu_out[0]) : null - let sp = nsh > 0l ? addr(s.moe_gout[0]) : null + let sp = nsh > 0l ? (sh_gpu ? addr(s.moe_shout[0]) : addr(s.moe_gout[0])) : null let gp = nsh > 0l && c.moe_shexp_gated ? addr(s.moe_gl_b[0]) : null maybe_parallel_for(nk * dim >= g_act_par_threshold, 0, int(npos), get_dispatch_lanes()) $(rb, re) { unsafe { diff --git a/modules/dasLLAMA/dasllama/dasllama_common.das b/modules/dasLLAMA/dasllama/dasllama_common.das index 7498e8f6cf..8466e2f2d2 100644 --- a/modules/dasLLAMA/dasllama/dasllama_common.das +++ b/modules/dasLLAMA/dasllama/dasllama_common.das @@ -1048,6 +1048,12 @@ struct Model { wsh1_off : int64 // ffn_gate_shexp: layers x (dim x n_ff_shexp) wsh2_off : int64 // ffn_down_shexp: layers x (n_ff_shexp x dim) wsh3_off : int64 // ffn_up_shexp: layers x (dim x n_ff_shexp) + wsh1_fmt : array //! the shared expert's gate plane format per layer, beside its q8 transcode (q8 where the file's is not a K-quant the dense rail serves; an empty table = q8 alone) + wsh2_fmt : array //! ... its down plane's + wsh3_fmt : array //! ... its up plane's + wshk1_offs : array //! the shared expert's K-quant gate plane offset per layer (-1 = none) + wshk2_offs : array //! ... its down plane's + wshk3_offs : array //! ... its up plane's wcls_off : int64 // Tied-classifier Q8: the classifier lives in qblob at wcls_off (repacked); set only by // load_gguf for shared_weights + QuantMode.q8 + Q8_0 token_embd on disk. @@ -2152,6 +2158,9 @@ struct Session { @scratch moe_w_cpu : array // ... and with the GPU slots zeroed (the CPU reduce's) @scratch moe_exp_gpu : array // per expert: 1 = this layer's GPU share (n_expert) @scratch @exact_size moe_gpu_out : array // the GPU half's combined rows (npos x dim), joined after the CPU half + @scratch @exact_size moe_shout : array //! the shared expert's rows off the device chain (npos x dim) + @scratch moe_sh_inv : array //! the device shared expert's identity slot map (npos; the f16-fed arm) + @scratch moe_sh_w : array //! the device shared expert's unit combine weights (npos; the f16-fed arm) ffn_offs : array // dense/shexp gate+up fuse: 2 region (weight, activation) offset pairs ffn_gu : array // dense/shexp gate+up fuse output — gate half then up half (2 x max_ffn_width) mx4_wq : array // grouped-prefill expansion of ONE native-MXFP4 expert to exact Q8 (dim x n_ff_exp) @@ -4556,6 +4565,16 @@ def set_mtp_spec(on : bool) { //! Current state of the [[set_mtp_spec]] gate. def get_mtp_spec() : bool => g_mtp_spec +var private g_mtp_force_reject_every = 0l + +//! The reject arm's test seam: every n-th draft the depth-1 step's greedy walk verifies is rejected +//! whatever the verify said (0 = off), so a fixture that accepts every draft still walks the arm. +def set_mtp_force_reject_every(n : int64) { + g_mtp_force_reject_every = n +} + +def private mtp_reject_forced(s : Session) : bool => g_mtp_force_reject_every > 0l && s.mtp_drafted % g_mtp_force_reject_every == 0l + var private g_mtp_batch_cls = true //! the verify's row cap: depth k runs k+1 rows, so the deepest round drafts MTP_MAX_ROWS-1 @@ -4902,7 +4921,7 @@ def mtp_spec_eval(t : Model; var s : Session; tok : int64; var accepted : int64& copy_floats(s.logits, 0l, s.mtp_logits_b, c.vocab_size, c.vocab_size) hit = mtp_walk_sampled(s, 1l, c.vocab_size) == 1l } else { - hit = parallel_argmax(s.mtp_logits, c.vocab_size) == d + hit = parallel_argmax(s.mtp_logits, c.vocab_size) == d && !mtp_reject_forced(s) } prof_add("mtp.walk", ts_walk) if (hit) { diff --git a/modules/dasLLAMA/dasllama/dasllama_env.das b/modules/dasLLAMA/dasllama/dasllama_env.das index b853b048ed..3acada1f87 100644 --- a/modules/dasLLAMA/dasllama/dasllama_env.das +++ b/modules/dasLLAMA/dasllama/dasllama_env.das @@ -102,6 +102,9 @@ struct public GpuTierEnv { @clarg_doc = "Lower the context floor for arming the resident decode driver, for a short-context session on a small card." gpu_min_ctx : int64 = 0l + @clarg_doc = "The whole-model resident driver for a model that fits the card, MoE included; 0 keeps the per-op rails (the A/B lever)." + gpu_resident : bool = true + @clarg_default_doc = "follows DASLLAMA_GPU" @clarg_doc = "DeltaNet rail on the GPU." gpu_dn : Option diff --git a/modules/dasLLAMA/dasllama/dasllama_gpu_resident.das b/modules/dasLLAMA/dasllama/dasllama_gpu_resident.das index eb969e1997..00af5d8c0a 100644 --- a/modules/dasLLAMA/dasllama/dasllama_gpu_resident.das +++ b/modules/dasLLAMA/dasllama/dasllama_gpu_resident.das @@ -432,16 +432,87 @@ let private RDEC_VRAM_HEADROOM_PCT = 27l let private RDEC_OS_RESERVE = 1_073_741_824l let private RDEC_MISC_BYTES = 67_108_864l // decode smalls + batch act planes + metas +//! Layer `l` routes through expert stacks: past the dense lead, with its expert planes present. +def layer_is_moe(t : Model; l : int64) : bool => (t.config.n_expert > 0l && l >= t.config.n_layer_dense_lead + && l < long_length(t.we1_offs) && t.we1_offs[l] >= 0l && t.we3_offs[l] >= 0l && t.we2_offs[l] >= 0l) + +//! The width of layer `l`'s dense FFN triple on the resident driver: the shared expert's on an +//! MoE layer (0 = the layer has none), the layer's FFN width otherwise. +def resident_dense_width(t : Model; l : int64) : int64 => layer_is_moe(t, l) ? t.config.n_ff_shexp : layer_hidden(t, l) + +//! The dense planes' width the driver sizes at: the dense triple's, or the expert width where no +//! layer carries a dense triple (nothing dispatches the dense planes then; they only exist). +def private resident_hidden(t : Model) : int64 { + let w = resident_dense_width(t, 0l) + return w > 0l ? w : t.config.n_ff_exp +} + +//! an MoE layer's device bytes: the expert triple and the shared expert's q8 triple when it has one +def private resident_moe_layer_bytes(t : Model; l : int64) : int64 { + let c = t.config + let ne = c.n_expert + let nfe = c.n_ff_exp + var bytes = moe_gpu_plane_bytes(c.dim, ne * nfe, int(fmt_at(t.we1_fmt, l))) + bytes += moe_gpu_plane_bytes(c.dim, ne * nfe, int(fmt_at(t.we3_fmt, l))) + bytes += moe_gpu_plane_bytes(nfe, ne * c.dim, int(fmt_at(t.we2_fmt, l))) + if (c.n_ff_shexp > 0l) { + bytes += moe_gpu_plane_bytes(c.dim, c.n_ff_shexp, int(resident_shexp_plane(t, l, 1).f)) + bytes += moe_gpu_plane_bytes(c.dim, c.n_ff_shexp, int(resident_shexp_plane(t, l, 3).f)) + bytes += moe_gpu_plane_bytes(c.n_ff_shexp, c.dim, int(resident_shexp_plane(t, l, 2).f)) + } + return bytes +} + +//! the shared expert's plane of `kind` (1 gate, 3 up, 2 down) on layer l: the K-quant plane where the +//! loader kept one, else the q8 transcode the CPU chain reads +def private resident_shexp_plane(t : Model; l : int64; kind : int) : tuple { + let she = t.config.dim * t.config.n_ff_shexp + if (kind == 1) { + if (l < long_length(t.wshk1_offs) && t.wshk1_offs[l] >= 0l) return (f = fmt_at(t.wsh1_fmt, l), off = t.wshk1_offs[l]) + return (f = KqFmt.q8, off = t.wsh1_off + l * she) + } + if (kind == 3) { + if (l < long_length(t.wshk3_offs) && t.wshk3_offs[l] >= 0l) return (f = fmt_at(t.wsh3_fmt, l), off = t.wshk3_offs[l]) + return (f = KqFmt.q8, off = t.wsh3_off + l * she) + } + if (l < long_length(t.wshk2_offs) && t.wshk2_offs[l] >= 0l) return (f = fmt_at(t.wsh2_fmt, l), off = t.wshk2_offs[l]) + return (f = KqFmt.q8, off = t.wsh2_off + l * she) +} + +//! the router plane: every MoE layer's f32 rows, the shared expert's gate row beside them when it is gated +def private resident_router_bytes(c : Config; n_moe : int64) : int64 => ( + n_moe * (c.n_expert + (c.n_ff_shexp > 0l && c.moe_shexp_gated ? 1l : 0l)) * c.dim * 4l) + +//! a recurrent layer's device bytes: the deltanet triple and the beta/alpha rows (f16 on the f32 arm, q8 planes otherwise); 0 on an attention layer +def private resident_dn_layer_bytes(t : Model; l : int64) : int64 { + let c = t.config + if (!(layer_is_recurrent(c, l) && l < long_length(t.dnqkv_offs) && t.dnqkv_offs[l] >= 0l)) { + return 0l + } + var bytes = moe_gpu_plane_bytes(c.dim, dn_conv_dim(c), int(fmt_at(t.dnqkv_fmt, l))) + bytes += moe_gpu_plane_bytes(c.dim, c.ssm_d_inner, int(fmt_at(t.dngate_fmt, l))) + bytes += moe_gpu_plane_bytes(c.ssm_d_inner, c.dim, int(fmt_at(t.dnout_fmt, l))) + bytes += t.dn_ba_f32 ? 2l * c.ssm_dt_rank * c.dim * 4l : 2l * moe_gpu_plane_bytes(c.dim, c.ssm_dt_rank, int(KqFmt.q8)) + return bytes +} + // the driver's device allocations past weights + KV — the prefill window buffers dominate. // An estimate, deliberately a touch high; the flat term carries the decode/batch smalls. [arch(at="../ARCHITECTURE_GPU_VULKAN_RESIDENCY.md#resident-plan")] def private rdec_scratch_bytes(t : Model) : int64 { let c = t.config let qd = layer_qd(c, 0l) - let hid = layer_hidden(t, 0l) + let hid = resident_hidden(t) let wide = max(max(c.dim, qd), hid) + var moe = 0l + if (c.n_expert > 0l) { + let nk = RDEC_PF_ROWS * c.n_expert_used + let nfe = c.n_ff_exp + moe = ((nk + TILE_READ_SLACK) * (c.dim + nfe) * 2l + nk * (2l * nfe + c.dim) * 4l + + RDEC_PF_ROWS * (c.n_expert + 1l) * 4l + nk * 12l) + } return RDEC_PF_ROWS * (16l * c.dim + 2l * wide + 10l * qd + 12l * layer_kv_dim(c, 0l) - + 8l * hid + 4l * layer_head_size(c, 0l)) + c.vocab_size * 4l + RDEC_MISC_BYTES + + 8l * hid + 4l * layer_head_size(c, 0l)) + c.vocab_size * 4l + RDEC_MISC_BYTES + moe } //! The room a plan may fill: the tier's cap less its headroom, capped by the OS's room when the OS @@ -486,8 +557,8 @@ def resident_plan(t : Model; seq_cap : int64; kdt, vdt : KVDtype) : ResidentPlan p.reason = "no GPU tier armed" return p } - if (c.n_expert > 0l) { - p.reason = "MoE - the per-op offload tier serves routed experts, not the resident driver" + if (c.n_expert > 0l && !rdec_moe_installed()) { + p.reason = "MoE - the tier installed no MoE seats for the whole-model driver; the per-op offload tier serves routed experts" return p } var krow = 0l @@ -499,11 +570,15 @@ def resident_plan(t : Model; seq_cap : int64; kdt, vdt : KVDtype) : ResidentPlan p.kv_bytes = seq_cap * (krow + vrow) p.scratch_bytes = rdec_scratch_bytes(t) fill_plan_room(p) - // every 2D plane a dense forward reads: the attention quad, the FFN triple, the classifier + var n_moe = 0l for (l in range64(c.n_layers)) { let qd = layer_qd(c, l) let kvd = layer_kv_dim(c, l) let hid = layer_hidden(t, l) + if (layer_is_moe(t, l)) { + p.weight_bytes += resident_moe_layer_bytes(t, l) + n_moe++ + } if (t.wq_offs[l] >= 0l) { p.weight_bytes += moe_gpu_plane_bytes(c.dim, qd * (c.q_gated ? 2l : 1l), int(fmt_at(t.wq_fmt, l))) } @@ -516,21 +591,16 @@ def resident_plan(t : Model; seq_cap : int64; kdt, vdt : KVDtype) : ResidentPlan if (t.wo_offs[l] >= 0l) { p.weight_bytes += moe_gpu_plane_bytes(qd, c.dim, int(fmt_at(t.wo_fmt, l))) } - if (t.w1_offs[l] >= 0l) { + if (l < long_length(t.w1_offs) && t.w1_offs[l] >= 0l) { //! a pure MoE carries no dense FFN offsets at all p.weight_bytes += moe_gpu_plane_bytes(c.dim, hid, int(fmt_at(t.w1_fmt, l))) p.weight_bytes += moe_gpu_plane_bytes(c.dim, hid, int(fmt_at(t.w3_fmt, l))) p.weight_bytes += moe_gpu_plane_bytes(hid, c.dim, int(fmt_at(t.w2_fmt, l))) } - if (layer_is_recurrent(c, l) && l < long_length(t.dnqkv_offs) && t.dnqkv_offs[l] >= 0l) { - let q8 = int(KqFmt.q8) - p.weight_bytes += moe_gpu_plane_bytes(c.dim, dn_conv_dim(c), int(fmt_at(t.dnqkv_fmt, l))) - p.weight_bytes += moe_gpu_plane_bytes(c.dim, c.ssm_d_inner, int(fmt_at(t.dngate_fmt, l))) - p.weight_bytes += moe_gpu_plane_bytes(c.ssm_d_inner, c.dim, int(fmt_at(t.dnout_fmt, l))) - p.weight_bytes += t.dn_ba_f32 ? 2l * c.ssm_dt_rank * c.dim * 4l : 2l * moe_gpu_plane_bytes(c.dim, c.ssm_dt_rank, q8) - } + p.weight_bytes += resident_dn_layer_bytes(t, l) } let cls_fmt = c.shared_weights ? t.emb_fmt : t.wcls_fmt p.weight_bytes += moe_gpu_plane_bytes(c.dim, c.vocab_size, int(cls_fmt)) + p.weight_bytes += resident_router_bytes(c, n_moe) let total = p.weight_bytes + p.kv_bytes + p.scratch_bytes let usable = p.usable_bytes let emb_want = g_env_vulkan.vk_gpu_embed && rdec_prefill_ids_installed() && !t.cls_q8 && !cls_kq(t) && !t.planes_trimmed @@ -592,14 +662,15 @@ def private rdec_min_ctx : int64 { // the arena block index. Same gather the per-op tier uses (moe_gpu_gather_stack / _kq), then place // instead of upload_stack. def private resident_place(t : Model; f : KqFmt; woff : int64; n, rows : int64; - mr, kgroup, wbias : int64) : int64 { + mr, kgroup, wbias : int64; slice_rows : int64 = -1l) : int64 { var wq : array var ws : array + let srows = slice_rows > 0l ? slice_rows : rows if (f == KqFmt.q8) { - moe_gpu_gather_stack(t, woff, n, rows, rows, mr, kgroup, wbias, wq, ws) + moe_gpu_gather_stack(t, woff, n, rows, srows, mr, kgroup, wbias, wq, ws) } else { let kmr = kq_active_mr(t, f) - moe_gpu_gather_stack_kq(t, f, woff, n, rows, rows, t.kq_repacked, kmr, wq, ws) + moe_gpu_gather_stack_kq(t, f, woff, n, rows, srows, t.kq_repacked, kmr, wq, ws) } let blk = rdec_place(wq, ws, n, rows, int(f)) if (blk >= 0l) { @@ -660,13 +731,62 @@ def resident_unserved_features(c : Config) : string { name_list_add(names, c.pre_post_norm, "pre/post norms") name_list_add(names, c.q_gated && !c.qk_norm, "gated attention without qk-norm") name_list_add(names, c.rope_dim > 0l && c.rope_dim != c.head_size && !c.qk_norm, "partial rotary ({c.rope_dim} of {c.head_size}) without qk-norm") + if (c.n_expert > 0l) { + name_list_add(names, c.moe_gate != MoeGate.softmax, "a non-softmax router") + name_list_add(names, c.moe_router_bias, "a router bias") + name_list_add(names, c.moe_exp_probs, "an expert selection bias") + name_list_add(names, c.moe_exps_bias, "biased expert stacks") + name_list_add(names, c.ffn_act == FfnAct.swiglu_oai, "the clamped swiglu expert activation") + name_list_add(names, c.n_expert > 256l, "{c.n_expert} experts (the select kernels reach 256)") + name_list_add(names, c.n_expert_used > 64l, "{c.n_expert_used} routed experts (the slot metas hold 64)") + name_list_add(names, c.layer_out_scale, "a layer output scale") + } return names } +//! The decline reason for a MoE layer's shared expert triple - its formats on the dense rail, one +//! feed form for gate and up, the down plane's block multiple - or "" when the driver serves it. +def private resident_shexp_decline(t : Model; l : int64) : string { + let c = t.config + if (c.n_ff_shexp <= 0l) return "" + let s1 = resident_shexp_plane(t, l, 1).f + let s3 = resident_shexp_plane(t, l, 3).f + let s2 = resident_shexp_plane(t, l, 2).f + if (!dense_plane_ok(t, s1) || !dense_plane_ok(t, s3) || !dense_plane_ok(t, s2)) { + return "layer {l}'s shared expert carries a format ({int(s1)}/{int(s3)}/{int(s2)}) the dense rail does not serve" + } + if (rdec_quant_class(s1) != rdec_quant_class(s3)) { + return "layer {l}'s shared expert gate and up planes mix activation-quant forms ({int(s1)}/{int(s3)}) - one feed serves both" + } + if (!rdec_rows_ok(c.n_ff_shexp, s2)) { + return "layer {l}'s shared width {c.n_ff_shexp} is not a block multiple of its down plane's format {int(s2)}" + } + return "" +} + //! Why layer `l` keeps the whole-model driver off the model - "" when the layer fits the //! recorder's one fixed geometry (hidden `hid0`, head size `hs`, kv width `kvd`, own K/V). def resident_layer_decline(t : Model; l, hid0, hs, kvd : int64) : string { let c = t.config + if (layer_is_moe(t, l)) { + let fe1 = fmt_at(t.we1_fmt, l) + let fe3 = fmt_at(t.we3_fmt, l) + let fe2 = fmt_at(t.we2_fmt, l) + if (!rdec_moe_ok(int(fe1), int(fe3), int(fe2))) { + return "layer {l}'s expert formats {int(fe1)}/{int(fe3)}/{int(fe2)} are outside the resident MoE block's tile family (the f16-fed cm2 tiles)" + } + if (kq_sb(fe1) != kq_sb(fe3)) { + return "layer {l}'s expert gate and up stacks mix activation-quant forms ({int(fe1)}/{int(fe3)}) - one feed serves both" + } + if (!rdec_rows_ok(c.n_ff_exp, fe2)) { + return "layer {l}'s expert width {c.n_ff_exp} is not a block multiple of its down plane's format {int(fe2)}" + } + let shexp_why = resident_shexp_decline(t, l) + if (!empty(shexp_why)) return shexp_why + } elif (c.n_expert > 0l && l >= c.n_layer_dense_lead) { + return "layer {l} lacks an expert plane" + } + if (resident_dense_width(t, l) != hid0) return "layer {l}'s dense FFN width {resident_dense_width(t, l)} differs from layer 0's {hid0}" if (layer_is_recurrent(c, l)) { if (l >= long_length(t.dnqkv_offs) || t.dnqkv_offs[l] < 0l || t.dngate_offs[l] < 0l || t.dnout_offs[l] < 0l) { return "layer {l} lacks a deltanet plane" @@ -685,13 +805,11 @@ def resident_layer_decline(t : Model; l, hid0, hs, kvd : int64) : string { } if (!t.dn_ba_f32 && (t.dnbeta_offs[l] < 0l || t.dnalpha_offs[l] < 0l)) return "layer {l} lacks its beta/alpha planes" if (!rdec_dn_installed()) return "layer {l} is recurrent and the tier installed no deltanet seats for the whole-model driver" - if (layer_hidden(t, l) != hid0) return "layer {l} FFN width {layer_hidden(t, l)} differs from layer 0's {hid0}" return "" } if (layer_is_sliding(c, l)) return "layer {l} attends over a sliding window" if (t.kv_src[l] != l) return "layer {l} shares layer {t.kv_src[l]}'s K/V" if (t.wq_offs[l] < 0l || t.wk_offs[l] < 0l || t.wv_offs[l] < 0l) return "layer {l} lacks a q/k/v plane" - if (layer_hidden(t, l) != hid0) return "layer {l} FFN width {layer_hidden(t, l)} differs from layer 0's {hid0}" if (layer_head_size(c, l) != hs) return "layer {l} head size {layer_head_size(c, l)} differs from layer 0's {hs}" if (layer_kv_dim(c, l) != kvd) return "layer {l} kv width {layer_kv_dim(c, l)} differs from layer 0's {kvd}" return "" @@ -780,7 +898,8 @@ def resident_upload(t : Model; seq_cap : int64; rkdt : KVDtype; plan : ResidentP vulkan_bake_role(VkBakeRole.arena) if (!moe_gpu_resident_installed()) return rdec_decline("the tier installed no resident-driver seams") if (t.quant != QuantMode.q8) return rdec_decline("weights are not in the q8 serving mode ({t.quant})") - if (t.config.n_expert > 0l) return rdec_decline("routed experts ({t.config.n_expert}) - the per-op tier's expert rails serve MoE") + if (t.config.n_expert > 0l && !rdec_moe_installed()) return rdec_decline("routed experts ({t.config.n_expert}) - the tier installed no MoE seats; the per-op tier's expert rails serve") + if (t.experts_mx4) return rdec_decline("mx4 expert planes") // the driver is all-or-nothing across prefill AND decode (the KV lives in the device mirror // only), so a pin away from it declines BEFORE any upload — the per-op rails take over let pin = g_env_engine.pin_prefill @@ -802,7 +921,7 @@ def resident_upload(t : Model; seq_cap : int64; rkdt : KVDtype; plan : ResidentP if (n_attn == 0l) return rdec_decline("no attention layer (the whole-model driver serves attention or hybrid stacks)") let n_recurrent = c.n_layers - n_attn let dim = c.dim - let hid0 = layer_hidden(t, 0l) + let hid0 = resident_dense_width(t, 0l) let hs = layer_head_size(c, l_attn0) let qd = layer_qd(c, l_attn0) let kvd = layer_kv_dim(c, l_attn0) @@ -827,9 +946,21 @@ def resident_upload(t : Model; seq_cap : int64; rkdt : KVDtype; plan : ResidentP && !t.emb_q8 && !cls_kq(t) && !t.planes_trimmed // an untied q8 table has no f32 copy to upload: the CPU embed loop serves it && plan.emb_f32_bytes > 0l) // the plan counted it at this context var planes : array> - planes |> reserve(c.n_layers * 8l + 1l) + planes |> reserve(c.n_layers * 11l + 1l) + var n_moe = 0l for (l in range64(c.n_layers)) { let lf = resident_layer_fmts(t, l) + if (layer_is_moe(t, l)) { + planes |> push((n = dim, rows = c.n_expert * c.n_ff_exp, f = fmt_at(t.we1_fmt, l))) + planes |> push((n = dim, rows = c.n_expert * c.n_ff_exp, f = fmt_at(t.we3_fmt, l))) + planes |> push((n = c.n_ff_exp, rows = c.n_expert * dim, f = fmt_at(t.we2_fmt, l))) + if (c.n_ff_shexp > 0l) { + planes |> push((n = dim, rows = c.n_ff_shexp, f = resident_shexp_plane(t, l, 1).f)) + planes |> push((n = dim, rows = c.n_ff_shexp, f = resident_shexp_plane(t, l, 3).f)) + planes |> push((n = c.n_ff_shexp, rows = dim, f = resident_shexp_plane(t, l, 2).f)) + } + n_moe++ + } if (layer_is_recurrent(c, l)) { planes |> push((n = dim, rows = dng.cd, f = fmt_at(t.dnqkv_fmt, l))) planes |> push((n = dim, rows = dng.di, f = fmt_at(t.dngate_fmt, l))) @@ -844,9 +975,11 @@ def resident_upload(t : Model; seq_cap : int64; rkdt : KVDtype; plan : ResidentP planes |> push((n = dim, rows = kvd, f = lf.fv)) planes |> push((n = qd, rows = dim, f = lf.fo)) } - planes |> push((n = dim, rows = hid0, f = lf.f1)) - planes |> push((n = dim, rows = hid0, f = lf.f3)) - planes |> push((n = hid0, rows = dim, f = lf.f2)) + if (!layer_is_moe(t, l)) { + planes |> push((n = dim, rows = hid0, f = lf.f1)) + planes |> push((n = dim, rows = hid0, f = lf.f3)) + planes |> push((n = hid0, rows = dim, f = lf.f2)) + } } planes |> push((n = dim, rows = c.vocab_size, f = cls_fmt)) for (pl in planes) { @@ -875,12 +1008,16 @@ def resident_upload(t : Model; seq_cap : int64; rkdt : KVDtype; plan : ResidentP let kg = repacked ? active_q8_kgroup() : 4l let wb = repacked ? active_q8_wbias() : 0l let rot = c.rope_dim > 0l ? c.rope_dim : hs - if (!rdec_prepare(c.n_layers, n_attn, dim, qd, kvd, hs, c.n_heads, hid0, c.vocab_size, seq_cap, neox, + if (!rdec_prepare(c.n_layers, n_attn, dim, qd, kvd, hs, c.n_heads, resident_hidden(t), c.vocab_size, seq_cap, neox, c.norm_eps, c.attn_scale > 0.0 ? c.attn_scale : 1.0 / sqrt(float(hs)), c.qk_norm, rkdt == KVDtype.f16, c.q_gated, rot, dng)) { return rdec_decline("the device prepare declined (a kernel class or the mirror range - the tier's line above)") } rdec_note_weight_bytes(plan.weight_bytes) + let sh_gated = c.n_ff_shexp > 0l && c.moe_shexp_gated + if (n_moe > 0l && !resident_prepare_moe(t, n_moe, sh_gated)) { + return rdec_decline("the MoE seats declined (the tier's line above)") + } if (n_recurrent > 0l && dng.ba_f32) { resident_upload_dn_ba(t, n_recurrent) } @@ -891,15 +1028,40 @@ def resident_upload(t : Model; seq_cap : int64; rkdt : KVDtype; plan : ResidentP // gate/up another), so a group must agree on its quant form — mixed groups decline var mir_idx = 0l var rec_idx = 0l + var router_slot = 0l for (l in range64(c.n_layers)) { - let lf = resident_layer_fmts(t, l) - if (rdec_quant_class(lf.f1) != rdec_quant_class(lf.f3)) { - return rdec_decline("layer {l} mixes activation-quant forms (ffn {int(lf.f1)}/{int(lf.f3)})") + var lf = resident_layer_fmts(t, l) + let moe = layer_is_moe(t, l) + var b1 = -1l + var b3 = -1l + var b2 = -1l + if (moe) { + //! a layer with no shared expert keeps -1 dense blocks and takes the experts' feed form + if (c.n_ff_shexp > 0l) { + let p1 = resident_shexp_plane(t, l, 1) + let p3 = resident_shexp_plane(t, l, 3) + let p2 = resident_shexp_plane(t, l, 2) + lf.f1 = p1.f + lf.f3 = p3.f + lf.f2 = p2.f + b1 = resident_place(t, p1.f, p1.off, dim, c.n_ff_shexp, mr, kg, wb) + b3 = resident_place(t, p3.f, p3.off, dim, c.n_ff_shexp, mr, kg, wb) + b2 = resident_place(t, p2.f, p2.off, c.n_ff_shexp, dim, mr, kg, wb) + if (b1 < 0l || b3 < 0l || b2 < 0l) return rdec_decline("arena placement failed at layer {l} (the shared expert)") + } else { + lf.f1 = fmt_at(t.we1_fmt, l) + lf.f3 = fmt_at(t.we3_fmt, l) + lf.f2 = fmt_at(t.we2_fmt, l) + } + } else { + if (rdec_quant_class(lf.f1) != rdec_quant_class(lf.f3)) { + return rdec_decline("layer {l} mixes activation-quant forms (ffn {int(lf.f1)}/{int(lf.f3)})") + } + b1 = resident_place(t, lf.f1, t.w1_offs[l], dim, hid0, mr, kg, wb) + b3 = resident_place(t, lf.f3, t.w3_offs[l], dim, hid0, mr, kg, wb) + b2 = resident_place(t, lf.f2, t.w2_offs[l], hid0, dim, mr, kg, wb) + if (b1 < 0l || b3 < 0l || b2 < 0l) return rdec_decline("arena placement failed at layer {l}") } - let b1 = resident_place(t, lf.f1, t.w1_offs[l], dim, hid0, mr, kg, wb) - let b3 = resident_place(t, lf.f3, t.w3_offs[l], dim, hid0, mr, kg, wb) - let b2 = resident_place(t, lf.f2, t.w2_offs[l], hid0, dim, mr, kg, wb) - if (b1 < 0l || b3 < 0l || b2 < 0l) return rdec_decline("arena placement failed at layer {l}") if (layer_is_recurrent(c, l)) { let fqkv = fmt_at(t.dnqkv_fmt, l) let fz = fmt_at(t.dngate_fmt, l) @@ -921,6 +1083,12 @@ def resident_upload(t : Model; seq_cap : int64; rkdt : KVDtype; plan : ResidentP addr(t.fblob[t.dn_dt_off + l * dng.nvh])) } rec_idx++ + if (moe) { + let eb = resident_place_experts(t, l, mr, kg, wb) + if (eb.be1 < 0l || eb.be3 < 0l || eb.be2 < 0l) return rdec_decline("arena placement failed at recurrent layer {l} (the expert stacks)") + rdec_set_moe_experts(l, router_slot, eb.be1, eb.be3, eb.be2, int(fmt_at(t.we1_fmt, l)), int(fmt_at(t.we3_fmt, l)), int(fmt_at(t.we2_fmt, l))) + router_slot++ + } continue } let cq = rdec_quant_class(lf.fq) @@ -933,8 +1101,17 @@ def resident_upload(t : Model; seq_cap : int64; rkdt : KVDtype; plan : ResidentP let bo = resident_place(t, lf.fo, t.wo_offs[l], qd, dim, mr, kg, wb) // a placement can fail (arena exhausted); fail closed to the CPU loop rather than wire a -1 block if (bq < 0l || bk < 0l || bv < 0l || bo < 0l) return rdec_decline("arena placement failed at layer {l}") - rdec_set_layer(l, mir_idx, bq, bk, bv, bo, b1, b3, b2, - int(lf.fq), int(lf.fk), int(lf.fv), int(lf.fo), int(lf.f1), int(lf.f3), int(lf.f2)) + if (moe) { + let eb = resident_place_experts(t, l, mr, kg, wb) + if (eb.be1 < 0l || eb.be3 < 0l || eb.be2 < 0l) return rdec_decline("arena placement failed at layer {l} (the expert stacks)") + rdec_set_moe_layer(l, mir_idx, router_slot, bq, bk, bv, bo, b1, b3, b2, eb.be1, eb.be3, eb.be2, + int(lf.fq), int(lf.fk), int(lf.fv), int(lf.fo), int(lf.f1), int(lf.f3), int(lf.f2), + int(fmt_at(t.we1_fmt, l)), int(fmt_at(t.we3_fmt, l)), int(fmt_at(t.we2_fmt, l))) + router_slot++ + } else { + rdec_set_layer(l, mir_idx, bq, bk, bv, bo, b1, b3, b2, + int(lf.fq), int(lf.fk), int(lf.fv), int(lf.fo), int(lf.f1), int(lf.f3), int(lf.f2)) + } mir_idx++ } let bcls = resident_place(t, cls_fmt, t.wcls_off, dim, c.vocab_size, mr, kg, wb) @@ -995,6 +1172,43 @@ def resident_upload(t : Model; seq_cap : int64; rkdt : KVDtype; plan : ResidentP return true } +[arch(at="../ARCHITECTURE_GPU_VULKAN_RESIDENCY.md#resident-plan")] +def private resident_prepare_moe(t : Model; n_moe : int64; sh_gated : bool) : bool { + let c = t.config + let dim = c.dim + let ne = c.n_expert + let ner = ne + (sh_gated ? 1l : 0l) + var router : array + router |> reserve(n_moe * ner * dim) + router |> resize(n_moe * ner * dim) + var li = 0l + for (l in range64(c.n_layers)) { + continue if (!layer_is_moe(t, l)) + unsafe { + memcpy(addr(router[li * ner * dim]), addr(t.fblob[t.router_off + l * ne * dim]), ne * dim * 4l) + if (sh_gated) { + memcpy(addr(router[li * ner * dim + ne * dim]), addr(t.fblob[t.shexp_gate_off + l * dim]), dim * 4l) + } + } + li++ + } + let ok = rdec_prepare_moe(ne, c.n_expert_used, c.n_ff_exp, c.n_ff_shexp, n_moe, sh_gated, c.norm_topk_prob, + c.ffn_act == FfnAct.gelu, c.expert_weights_scale, router) + delete router + return ok +} + +//! the expert triple of an MoE layer into the arena, sliced per expert as the per-op walk gathers it +def private resident_place_experts(t : Model; l : int64; mr, kg, wb : int64) : tuple { + let c = t.config + let ne = c.n_expert + let nfe = c.n_ff_exp + let be1 = resident_place(t, fmt_at(t.we1_fmt, l), t.we1_offs[l], c.dim, ne * nfe, mr, kg, wb, nfe) + let be3 = be1 < 0l ? -1l : resident_place(t, fmt_at(t.we3_fmt, l), t.we3_offs[l], c.dim, ne * nfe, mr, kg, wb, nfe) + let be2 = be3 < 0l ? -1l : resident_place(t, fmt_at(t.we2_fmt, l), t.we2_offs[l], nfe, ne * c.dim, mr, kg, wb, c.dim) + return (be1 = be1, be3 = be3, be2 = be2) +} + // One position's packed rope row for the resident device kernels (see build_rope_row_packed — // the one source for every resident-override rope row). [unused_argument(half)] @@ -1544,10 +1758,36 @@ def moe_gpu_upload_resident(t : Model) { // nolint:STYLE037,STYLE038 — seven && !gpu_want_shexp() && !gpu_want_qkv() && !gpu_want_cls())) { return // tier absent or nothing requested } + // models that fully fit, MoE included: the whole-stack resident driver is tried FIRST - it + // subsumes every rail below; a decline falls through + let resident_try = (moe_gpu_resident_installed() && gpu_want_auto() && gpu_want_resident() + && (t.config.n_expert == 0l || rdec_moe_installed())) + // the mirror codec: f16 rows by default (native fa feed, ~2x plannable ctx); + // DASLLAMA_VK_KV32=1 arms f32 mirrors instead (A/B; then only f32 sessions serve) + let rkdt = g_env_vulkan.vk_kv32 != 0l ? KVDtype.f32 : KVDtype.f16 + var seq_cap = t.config.seq_len + var plan : ResidentPlan + if (resident_try) { + let bctx = resident_binding_ctx(t.config, rkdt, moe_gpu_binding_cap()) + if (bctx > 0l) { + seq_cap = min(seq_cap, bctx) + } + plan = resident_plan(t, seq_cap, rkdt, rkdt) + if (!plan.fits && plan.weight_bytes > 0l && plan.kv_bytes > 0l) { + // weights are fixed, the KV mirror is not: retry at the context that fits (the plan's + // own remedy), as long as it stays a usable window. The driver guards decode past the cap. + let fit_ctx = (max(plan.usable_bytes - plan.weight_bytes - plan.scratch_bytes, 0l)) / (plan.kv_bytes / seq_cap) + if (fit_ctx >= rdec_min_ctx()) { + seq_cap = fit_ctx + plan = resident_plan(t, seq_cap, rkdt, rkdt) + } + } + } + let resident_fits = resident_try && plan.fits // size the tier's streamed-slot reserve from THIS model's largest expert layer — a fixed // reserve wastes VRAM or asserts on big-expert geometries (GLM Air, Qwen1.5-MoE); must // precede the first upload (the tier carves the reserve at its lazy init) - if (t.config.n_expert > 0l) { + if (t.config.n_expert > 0l && !resident_fits) { var sgu = 0l var sdn = 0l for (sl in range64(t.config.n_layers + t.config.n_layer_nextn)) { @@ -1563,39 +1803,25 @@ def moe_gpu_upload_resident(t : Model) { // nolint:STYLE037,STYLE038 — seven } set_moe_gpu_stream_need(sgu > 0l ? 2l * sgu + sdn : 0l) } - if (at_want) { // the decode block's mirrors for any model whose quads ride the tier, the span's router planes for a MoE one + if (at_want && !resident_fits) { // the decode block's mirrors for any model whose quads ride the tier, the span's router planes for a MoE one set_moe_gpu_dat_need(t.config.n_layers * DAT_MIRROR_ROWS * layer_kv_dim(t.config, 0l) * 4l + (t.config.n_expert > 0l ? t.config.n_layers * t.config.n_expert * t.config.dim * 4l : 0l)) } - // dense models that fully fit: try the whole-stack resident driver FIRST — it subsumes every - // rail below (all planes + classifier resident, one-submit device prefill + mirror decode). - // Any decline (shape, fit, placement) falls through to the per-op rails unchanged. - if (moe_gpu_resident_installed() && gpu_want_auto() && t.config.n_expert == 0l) { - var seq_cap = t.config.seq_len - // the mirror codec: f16 rows by default (native fa feed, ~2x plannable ctx); - // DASLLAMA_VK_KV32=1 arms f32 mirrors instead (A/B; then only f32 sessions serve) - let rkdt = g_env_vulkan.vk_kv32 != 0l ? KVDtype.f32 : KVDtype.f16 - let bctx = resident_binding_ctx(t.config, rkdt, moe_gpu_binding_cap()) - if (bctx > 0l) { - seq_cap = min(seq_cap, bctx) - } - var plan = resident_plan(t, seq_cap, rkdt, rkdt) - if (!plan.fits && plan.weight_bytes > 0l && plan.kv_bytes > 0l) { - // weights are fixed, the KV mirror is not: retry at the context that fits (the plan's - // own remedy), as long as it stays a usable window. The driver guards decode past the cap. - let fit_ctx = (max(plan.usable_bytes - plan.weight_bytes - plan.scratch_bytes, 0l)) / (plan.kv_bytes / seq_cap) - if (fit_ctx >= rdec_min_ctx()) { - seq_cap = fit_ctx - plan = resident_plan(t, seq_cap, rkdt, rkdt) - } - } + if (resident_try) { if (plan.fits) { if (resident_upload(t, seq_cap, rkdt, plan)) { return } + if (t.config.n_expert > 0l) { + to_log(LOG_WARNING, "dasLLAMA: the per-op rails serve this MoE without the streamed slot or the decode mirrors the fitting plan forwent - layers the budget stops keep the CPU\n") + } } else { rdec_decline_say(empty(plan.reason) ? "the residency plan does not fit" : plan.reason) } + } elif (moe_gpu_resident_installed() && !gpu_want_resident()) { + to_log(LOG_INFO, "dasLLAMA: resident driver not attempted - the per-op rails were asked for (DASLLAMA_GPU_RESIDENT=0, or set_gpu_resident_route(false) in process)\n") + } elif (moe_gpu_resident_installed() && t.config.n_expert > 0l && !rdec_moe_installed()) { + to_log(LOG_INFO, "dasLLAMA: resident driver not attempted - the tier installed no MoE seats for its {t.config.n_expert} routed experts; the per-op tier's expert rails serve\n") } elif (moe_gpu_resident_installed() && t.config.n_expert == 0l) { to_log(LOG_INFO, "dasLLAMA: resident driver not attempted - DASLLAMA_GPU is not set (the individual DASLLAMA_GPU_* rails alone are armed)\n") } @@ -1728,8 +1954,7 @@ def moe_gpu_upload_resident(t : Model) { // nolint:STYLE037,STYLE038 — seven delete uploads break } - // deliberately NOT set_moe_gpu_dense_off: that would also route the PREFILL shexp GEMMs - // onto the dense rail (a separately measured wash) and stop this being a decode-only lever + //! the shexp mark, not set_moe_gpu_dense_off: the shared triple rides the expert chain, not the dense rail set_moe_gpu_shexp_layer(l) sh_marked++ delete uploads diff --git a/modules/dasLLAMA/dasllama/dasllama_gpu_tier.das b/modules/dasLLAMA/dasllama/dasllama_gpu_tier.das index 1b8c0d0226..c93e216c70 100644 --- a/modules/dasLLAMA/dasllama/dasllama_gpu_tier.das +++ b/modules/dasLLAMA/dasllama/dasllama_gpu_tier.das @@ -219,6 +219,58 @@ def public install_moe_gpu_resident_bias(upload_bias : RdecUploadBiasFn) { def public rdec_bias_installed() : bool => g_rdec_bias_installed def public rdec_upload_bias(rows : array) { invoke(g_rdec_upload_bias, rows) } +// ===== the resident driver's MoE seats ===== +//! Does the resident MoE block admit an expert triple of these formats - the tile family it +//! rides is the device's (the f16-fed cm2 tiles today). +typedef RdecMoeOkFn = function<(f1 : int; f3 : int; f2 : int) : bool> +//! The MoE geometry and the router plane (every MoE layer's [ne x dim] rows, then its gate row when the shared expert is gated) after the prepare; false = the class rails declined. +typedef RdecPrepareMoeFn = function<(ne : int64; k : int64; nfe : int64; nsh : int64; n_moe : int64; sh_gated : bool; norm_topk : bool; is_gelu : bool; wscale : float; router : array) : bool> +//! Register an MoE attention layer: the attention quad, the dense triple (the shared expert's, or -1 blocks for none) and the expert triple. +typedef RdecSetMoeLayerFn = function<(l : int64; mir_idx : int64; router_slot : int64; bq : int64; bk : int64; bv : int64; bo : int64; b1 : int64; b3 : int64; b2 : int64; be1 : int64; be3 : int64; be2 : int64; fq : int; fk : int; fv : int; fo : int; f1 : int; f3 : int; f2 : int; fe1 : int; fe3 : int; fe2 : int) : void> +//! Register the routed block on a layer already registered by its own seat (a recurrent MoE layer after its deltanet registration): the expert triple and the layer's router-plane slot. +typedef RdecSetMoeExpertsFn = function<(l : int64; router_slot : int64; be1 : int64; be3 : int64; be2 : int64; fe1 : int; fe3 : int; fe2 : int) : void> + +[unused_argument(f1, f3, f2)] +def private rdec_unset_moe_ok(f1, f3, f2 : int) : bool => false +[unused_argument(ne, k, nfe, nsh, n_moe, sh_gated, norm_topk, is_gelu, wscale, router)] +def private rdec_unset_prepare_moe(ne, k, nfe, nsh, n_moe : int64; sh_gated, norm_topk, is_gelu : bool; wscale : float; router : array) : bool => false +[unused_argument(l, mir_idx, router_slot, bq, bk, bv, bo, b1, b3, b2, be1, be3, be2, fq, fk, fv, fo, f1, f3, f2, fe1, fe3, fe2)] +def private rdec_unset_set_moe_layer(l, mir_idx, router_slot, bq, bk, bv, bo, b1, b3, b2, be1, be3, be2 : int64; fq, fk, fv, fo, f1, f3, f2, fe1, fe3, fe2 : int) {} +[unused_argument(l, router_slot, be1, be3, be2, fe1, fe3, fe2)] +def private rdec_unset_set_moe_experts(l, router_slot, be1, be3, be2 : int64; fe1, fe3, fe2 : int) {} + +var g_rdec_moe_ok = @@rdec_unset_moe_ok +var g_rdec_prepare_moe = @@rdec_unset_prepare_moe +var g_rdec_set_moe_layer = @@rdec_unset_set_moe_layer +var g_rdec_set_moe_experts = @@rdec_unset_set_moe_experts +var g_rdec_moe_installed = false + +//! Install the resident driver's MoE seats - separately, so a tier without them declines a MoE by name. +def public install_moe_gpu_resident_moe(moe_ok : RdecMoeOkFn; prepare_moe : RdecPrepareMoeFn; set_moe_layer : RdecSetMoeLayerFn; set_moe_experts : RdecSetMoeExpertsFn) { + g_rdec_moe_ok = moe_ok + g_rdec_prepare_moe = prepare_moe + g_rdec_set_moe_layer = set_moe_layer + g_rdec_set_moe_experts = set_moe_experts + g_rdec_moe_installed = true +} + +def public rdec_moe_installed() : bool => g_rdec_moe_installed +def public rdec_moe_ok(f1, f3, f2 : int) : bool => invoke(g_rdec_moe_ok, f1, f3, f2) +def public rdec_prepare_moe(ne, k, nfe, nsh, n_moe : int64; sh_gated, norm_topk, is_gelu : bool; wscale : float; router : array) : bool => invoke(g_rdec_prepare_moe, ne, k, nfe, nsh, n_moe, sh_gated, norm_topk, is_gelu, wscale, router) +def public rdec_set_moe_layer(l, mir_idx, router_slot, bq, bk, bv, bo, b1, b3, b2, be1, be3, be2 : int64; fq, fk, fv, fo, f1, f3, f2, fe1, fe3, fe2 : int) { invoke(g_rdec_set_moe_layer, l, mir_idx, router_slot, bq, bk, bv, bo, b1, b3, b2, be1, be3, be2, fq, fk, fv, fo, f1, f3, f2, fe1, fe3, fe2) } +def public rdec_set_moe_experts(l, router_slot, be1, be3, be2 : int64; fe1, fe3, fe2 : int) { invoke(g_rdec_set_moe_experts, l, router_slot, be1, be3, be2, fe1, fe3, fe2) } + +var private g_gpu_resident_route = true + +//! The in-process twin of DASLLAMA_GPU_RESIDENT: a test pins the whole-model driver off before a +//! load to hold the per-op rails' claims on a model the driver would take. +def public set_gpu_resident_route(on : bool) { + g_gpu_resident_route = on +} + +//! The whole-model driver is wanted on this load: the knob and the in-process lever both on. +def public gpu_want_resident : bool => g_env_gpu.gpu_resident && g_gpu_resident_route + //! Install the resident driver's hybrid (deltanet) seats - separately, so a tier without them //! declines recurrent layers by name instead of serving a model it cannot step. def public install_moe_gpu_resident_dn(set_dn_layer : RdecSetDnLayerFn; upload_dn_ba : RdecUploadDnBaFn; dn_own : RdecDnOwnFn; dn_prefill_own : RdecDnPrefillOwnFn) { @@ -265,6 +317,10 @@ typedef RdecUploadEmbF32Fn = function<(fblob : array; off : int64; vocab //! The raw f32 embd table rides the residency plan's headroom only up to this size; past it the CPU embed serves. let public RDEC_EMB_F32_CAP = 512l * 1024l * 1024l +//! Rows of read slack past a cm2 region's last row: the m column loads a partial column unclamped, so +//! every plane a tile reads (the residency plan's scratch, the prefill's gathered rows, a probe's) carries them. +let public TILE_READ_SLACK = 128l + //! The device bytes the f32 embed-gather arm would upload for a (vocab x dim) table - 0 past its cap. [arch(at="../ARCHITECTURE_GPU_VULKAN_RESIDENCY.md#resident-plan")] def public rdec_emb_f32_bytes(vocab, dim : int64) : int64 { @@ -643,6 +699,7 @@ def public vk_rdec_rerecord_on : bool => g_vk_rdec_rerecord var g_moe_gpu_shexp_layers : table // layers whose shexp gate/up/down triple is resident var g_moe_gpu_shexp_route = true +var private g_moe_gpu_shexp_said : table // the prefill's shared-expert decline reasons said for this model //! Loader contract: mark one layer's shared-expert triple resident on the tier. Reset per load. def public set_moe_gpu_shexp_layer(l : int64) { @@ -651,12 +708,23 @@ def public set_moe_gpu_shexp_layer(l : int64) { def public reset_moe_gpu_shexp_layers { g_moe_gpu_shexp_layers |> clear() + g_moe_gpu_shexp_said |> clear() } def public moe_gpu_shexp_on_gpu(l : int64) : bool { return g_moe_gpu_installed && g_moe_gpu_shexp_route && key_exists(g_moe_gpu_shexp_layers, l) } +//! The prefill's shared expert stays on the CPU for `why`: said once per reason per model while the tier is armed. +[cold_path] +def public moe_gpu_shexp_declined(why : string) { + if (!g_moe_gpu_installed || key_exists(g_moe_gpu_shexp_said, why)) { + return + } + g_moe_gpu_shexp_said |> insert(why) + to_log(LOG_INFO, "dasLLAMA: the prefill's shared expert takes the CPU rail - {why} (said once per reason)\n") +} + //! Runtime routing gate over the shared-expert marks (the --shexp-ab in-process lever). def public set_moe_gpu_shexp_route(on : bool) { g_moe_gpu_shexp_route = on @@ -1273,6 +1341,12 @@ struct MoeSpanDec { f1 : array f3 : array f2 : array + nsh : int64 //! the shared expert's width (0 = the model has none) + sh_gated : bool //! the shared expert's row scales by sigmoid(gate . xb) + wsh1 : array //! the shared expert's q8 gate plane offset per layer + wsh3 : array //! ... its up plane + wsh2 : array //! ... its down plane + shgate : float const? //! [nl x dim] f32 gate rows, layer-major (null when ungated or absent) rmsq : array // the qk-norm rows (null when off) rmsk : array bq : array // the q/k/v projection bias rows (null when the model has none) @@ -1691,6 +1765,18 @@ def private restore_gpu_tier_fn_defaults { // nolint:STYLE037,STYLE038 — flat if (g_rdec_upload_bias == null) { g_rdec_upload_bias = @@rdec_unset_upload_bias } + if (g_rdec_moe_ok == null) { + g_rdec_moe_ok = @@rdec_unset_moe_ok + } + if (g_rdec_prepare_moe == null) { + g_rdec_prepare_moe = @@rdec_unset_prepare_moe + } + if (g_rdec_set_moe_layer == null) { + g_rdec_set_moe_layer = @@rdec_unset_set_moe_layer + } + if (g_rdec_set_moe_experts == null) { + g_rdec_set_moe_experts = @@rdec_unset_set_moe_experts + } if (g_moe_gpu_binding_cap == null) { g_moe_gpu_binding_cap = @@moe_gpu_no_binding_cap } diff --git a/modules/dasLLAMA/dasllama/dasllama_image.das b/modules/dasLLAMA/dasllama/dasllama_image.das index 571dd2a33c..08a5a4f0f9 100644 --- a/modules/dasLLAMA/dasllama/dasllama_image.das +++ b/modules/dasLLAMA/dasllama/dasllama_image.das @@ -38,7 +38,7 @@ require dasllama/dasllama_load // WhisperModel.enc) contribute their planes under dotted names ("enc.fblob"); string-array // fields ride the meta blob via serialize_strings — raw string pointers can't be planes. -let IMAGE_VERSION = 35 // the vulkan flavor's iq4xs/iq3s/iq3xxs/iq2xxs scale planes are two-word rows; a BPE vocab without add_bos takes upstream's per-pre default +let IMAGE_VERSION = 36 // the shared expert's K-quant planes ride beside its q8 transcode on a tier-armed load (wsh*_fmt, wshk*_offs) //! The metal (blob-only) flavor's identity tag: q8 planes ride the 34B block_q8_0 blob and the //! kq scale planes their GPU forms (convert_model_to_metal_blob) — flavors are per-config and diff --git a/modules/dasLLAMA/dasllama/dasllama_layout.das b/modules/dasLLAMA/dasllama/dasllama_layout.das index 56f6eed6a9..aecaa25dc0 100644 --- a/modules/dasLLAMA/dasllama/dasllama_layout.das +++ b/modules/dasLLAMA/dasllama/dasllama_layout.das @@ -709,6 +709,17 @@ def private collect_kq_regions(t : Model; var regs : array) { push_repack_kq(regs, fmt_at(t.w2_fmt, l), t.w2_offs[l], c.hidden_dim, dim) push_repack_kq(regs, fmt_at(t.w3_fmt, l), t.w3_offs[l], dim, c.hidden_dim) } + if (c.n_ff_shexp > 0l && l < long_length(t.wshk1_offs)) { + if (t.wshk1_offs[l] >= 0l) { + push_repack_kq(regs, fmt_at(t.wsh1_fmt, l), t.wshk1_offs[l], dim, c.n_ff_shexp) + } + if (t.wshk2_offs[l] >= 0l) { + push_repack_kq(regs, fmt_at(t.wsh2_fmt, l), t.wshk2_offs[l], c.n_ff_shexp, dim) + } + if (t.wshk3_offs[l] >= 0l) { + push_repack_kq(regs, fmt_at(t.wsh3_fmt, l), t.wshk3_offs[l], dim, c.n_ff_shexp) + } + } } else { // dense arch layer OR a glm4moe dense-lead layer; pure-MoE arches never reach here let h = layer_hidden(t, l) diff --git a/modules/dasLLAMA/dasllama/dasllama_load.das b/modules/dasLLAMA/dasllama/dasllama_load.das index e048cd0c25..0cdf8f6188 100644 --- a/modules/dasLLAMA/dasllama/dasllama_load.das +++ b/modules/dasLLAMA/dasllama/dasllama_load.das @@ -394,6 +394,13 @@ def private layout_offsets(var t : Model) : LayoutSizes { // nolint:STYLE037,S cur.wo += layers * she t.wsh3_off = cur.wo cur.wo += layers * she + //! the K-quant plane offsets beside the transcode's, where the tags kept the file's format + t.wshk1_offs |> clear(); t.wshk1_offs |> reserve(layers) + for (l in range64(layers)) { t.wshk1_offs |> push(fmt_at(t.wsh1_fmt, l) == KqFmt.q8 ? -1l : kq_take(cur, fmt_at(t.wsh1_fmt, l), she)) } + t.wshk2_offs |> clear(); t.wshk2_offs |> reserve(layers) + for (l in range64(layers)) { t.wshk2_offs |> push(fmt_at(t.wsh2_fmt, l) == KqFmt.q8 ? -1l : kq_take(cur, fmt_at(t.wsh2_fmt, l), she)) } + t.wshk3_offs |> clear(); t.wshk3_offs |> reserve(layers) + for (l in range64(layers)) { t.wshk3_offs |> push(fmt_at(t.wsh3_fmt, l) == KqFmt.q8 ? -1l : kq_take(cur, fmt_at(t.wsh3_fmt, l), she)) } } if (c.moe_dense_shexp) { // gemma4: the DENSE parallel shared expert (plain ffn gate/up/down) fill_dense_ffn_offsets(t, cur, layers, dim, hidden) @@ -1549,6 +1556,22 @@ def private detect_kq_formats(var t : Model; m : GGUFMeta) { // nolint:STYLE03 any ||= t.we1_fmt[l] != KqFmt.q8 || t.we2_fmt[l] != KqFmt.q8 || t.we3_fmt[l] != KqFmt.q8 } } + //! the shared expert's tags: its K-quant planes ride beside the q8 transcode on a load with a GPU tier armed (the resident driver reads them; a CPU-only load keeps the transcode alone) + if (moe_gpu_tier_installed() && t.config.n_expert > 0l && t.config.n_ff_shexp > 0l && (t.config.dim * t.config.n_ff_shexp) % 256l == 0l) { + t.wsh1_fmt |> reserve(layers) + t.wsh1_fmt |> resize(layers) + t.wsh2_fmt |> reserve(layers) + t.wsh2_fmt |> resize(layers) + t.wsh3_fmt |> reserve(layers) + t.wsh3_fmt |> resize(layers) + for (l in range64(layers)) { + let has = l >= t.config.n_layer_dense_lead && l < t.config.n_layers + t.wsh1_fmt[l] = has ? kq_tag(m, "blk.{l}.ffn_gate_shexp.weight", t.config.dim) : KqFmt.q8 + t.wsh3_fmt[l] = has ? kq_tag(m, "blk.{l}.ffn_up_shexp.weight", t.config.dim) : KqFmt.q8 + t.wsh2_fmt[l] = has ? kq_tag(m, "blk.{l}.ffn_down_shexp.weight", t.config.n_ff_shexp) : KqFmt.q8 + any ||= t.wsh1_fmt[l] != KqFmt.q8 || t.wsh2_fmt[l] != KqFmt.q8 || t.wsh3_fmt[l] != KqFmt.q8 + } + } if (t.config.shared_weights) { t.emb_fmt = kq_tag(m, "token_embd.weight", t.config.dim) any ||= t.emb_fmt != KqFmt.q8 @@ -2650,6 +2673,17 @@ def private load_gguf_parsed(var t : Model; m : GGUFMeta; bytes : array | load_big(m, bytes, "blk.{l}.ffn_gate_shexp.weight", t, t.wsh1_off + l * she, she, scratch) load_big(m, bytes, "blk.{l}.ffn_down_shexp.weight", t, t.wsh2_off + l * she, she, scratch) load_big(m, bytes, "blk.{l}.ffn_up_shexp.weight", t, t.wsh3_off + l * she, she, scratch) + if (l < long_length(t.wshk1_offs)) { //! the same planes again in the file's K-quant form, beside the transcode above + if (t.wshk1_offs[l] >= 0l) { + load_big(m, bytes, "blk.{l}.ffn_gate_shexp.weight", t, t.wshk1_offs[l], she, scratch, 0l, fmt_at(t.wsh1_fmt, l)) + } + if (t.wshk2_offs[l] >= 0l) { + load_big(m, bytes, "blk.{l}.ffn_down_shexp.weight", t, t.wshk2_offs[l], she, scratch, 0l, fmt_at(t.wsh2_fmt, l)) + } + if (t.wshk3_offs[l] >= 0l) { + load_big(m, bytes, "blk.{l}.ffn_up_shexp.weight", t, t.wshk3_offs[l], she, scratch, 0l, fmt_at(t.wsh3_fmt, l)) + } + } } if (t.config.moe_dense_shexp) { // gemma4: the DENSE parallel shared expert (plain ffn gate/up/down) load_big(m, bytes, "blk.{l}.ffn_gate.weight", t, t.w1_offs[l], dim * hidden, scratch, 0l, fmt_at(t.w1_fmt, l)) diff --git a/modules/dasLLAMA/dasllama/dasllama_math_vulkan.das b/modules/dasLLAMA/dasllama/dasllama_math_vulkan.das index a1b1f3e7a8..41d720b613 100644 --- a/modules/dasLLAMA/dasllama/dasllama_math_vulkan.das +++ b/modules/dasLLAMA/dasllama/dasllama_math_vulkan.das @@ -361,6 +361,7 @@ def dasllama_math_vulkan_register() { @@vk_rdec_upload_norms, @@vk_rdec_set_layer, @@vk_rdec_set_cls, @@vk_rdec_token, @@vk_rdec_prefill, @@vk_rdec_sync_kv, @@vk_rdec_read_kv, @@vk_rdec_read_kv_bulk) install_moe_gpu_resident_dn(@@vk_rdec_set_dn_layer, @@vk_rdec_upload_dn_ba, @@vk_rdec_dn_own, @@vk_rdec_prefill_dn_own) + install_moe_gpu_resident_moe(@@vk_rdec_moe_ok, @@vk_rdec_prepare_moe, @@vk_rdec_set_moe_layer, @@vk_rdec_set_moe_experts) set_moe_gpu_dn_state_hooks(@@vk_dn_step_flush, @@vk_dn_step_invalidate, @@vk_dn_step_release) set_moe_gpu_heat_hooks(@@vk_moe_heat_query, @@vk_moe_heat_advise, @@vk_moe_ffn_begin, @@vk_moe_ffn_join) set_moe_gpu_qkv_hook(@@vk_moe_qkv) diff --git a/modules/dasLLAMA/dasllama/dasllama_vulkan_classes.das b/modules/dasLLAMA/dasllama/dasllama_vulkan_classes.das index 2cedc57b42..9153f3dbc3 100644 --- a/modules/dasLLAMA/dasllama/dasllama_vulkan_classes.das +++ b/modules/dasLLAMA/dasllama/dasllama_vulkan_classes.das @@ -39,6 +39,9 @@ struct ArArgs { eps : float ascale : float row0 : uint + slots : uint //! the combine twins: routed slots per position + lstride : uint //! the combine twins: the logits row stride (the gate logit, when `gated`, is the row's last) + gated : uint //! the combine twins: the shared expert's gate scales its row } // the shared Q8_0 quantize pieces (pure — the amax fold is subgroup-wide) @@ -189,6 +192,105 @@ class ClsArAddRmsRqB : ArRqBase { } } +class ArCombBase : ArBase { + @ssbo @binding = 4 slot_idx : array //! position -> its k slot rows + @ssbo @binding = 5 slot_w : array //! ... and their routing weights + @ssbo @binding = 6 down_rows : array //! the routed down-GEMM output rows + @ssbo @binding = 7 logits : array //! [npos x lstride]; the gate logit is the row's last + + def override accum_row(base : uint) : float { + let p = base / pa.dim + var g = 1.0 + if (pa.gated != 0u) { + g = 1.0 / (1.0 + exp(-logits[p * pa.lstride + pa.lstride - 1u])) + } + let pk = p * pa.slots + var ss = 0.0 + var k = gl_LocalInvocationID.x + while (k < pa.dim) { + var f = 0.0 + if (pa.add_on != 0u) { + f = g * aa[base + k] + } + var j = 0u + while (j + 8u <= pa.slots) { + let d0 = down_rows[slot_idx[pk + j] * pa.dim + k] + let d1 = down_rows[slot_idx[pk + j + 1u] * pa.dim + k] + let d2 = down_rows[slot_idx[pk + j + 2u] * pa.dim + k] + let d3 = down_rows[slot_idx[pk + j + 3u] * pa.dim + k] + let d4 = down_rows[slot_idx[pk + j + 4u] * pa.dim + k] + let d5 = down_rows[slot_idx[pk + j + 5u] * pa.dim + k] + let d6 = down_rows[slot_idx[pk + j + 6u] * pa.dim + k] + let d7 = down_rows[slot_idx[pk + j + 7u] * pa.dim + k] + f += slot_w[pk + j] * d0 + f += slot_w[pk + j + 1u] * d1 + f += slot_w[pk + j + 2u] * d2 + f += slot_w[pk + j + 3u] * d3 + f += slot_w[pk + j + 4u] * d4 + f += slot_w[pk + j + 5u] * d5 + f += slot_w[pk + j + 6u] * d6 + f += slot_w[pk + j + 7u] * d7 + j += 8u + } + if (j + 4u <= pa.slots) { + let d0 = down_rows[slot_idx[pk + j] * pa.dim + k] + let d1 = down_rows[slot_idx[pk + j + 1u] * pa.dim + k] + let d2 = down_rows[slot_idx[pk + j + 2u] * pa.dim + k] + let d3 = down_rows[slot_idx[pk + j + 3u] * pa.dim + k] + f += slot_w[pk + j] * d0 + f += slot_w[pk + j + 1u] * d1 + f += slot_w[pk + j + 2u] * d2 + f += slot_w[pk + j + 3u] * d3 + j += 4u + } + while (j < pa.slots) { + f += slot_w[pk + j] * down_rows[slot_idx[pk + j] * pa.dim + k] + j++ + } + let vv = (xs[base + k] + f) * pa.ascale + xs[base + k] = vv + row[k] = vv + ss += vv * vv + k += 256u + } + return ss + } +} + +[vk_dispatch(name = "cls_ar_comb", grid = "nrows", params = "nrows : int64")] +class ClsArComb : ArCombBase { + @ssbo @binding = 3 yo : array //! the normed output rows + + [spirv_kernel(local_size_x = 256, name = "cls_ar_comb_spv"), arch(at="../ARCHITECTURE_GPU_VULKAN_MOE.md#vk-prefill-moe-block")] + def run { + let base = (gl_WorkGroupID.x + pa.row0) * pa.dim + let ss = accum_row(base) + let inv = rms_inv(ss) + var k = gl_LocalInvocationID.x + while (k < pa.dim) { + yo[base + k] = wn[pa.woff + k] * (row[k] * inv) + k += 256u + } + } +} + +[vk_dispatch(name = "cls_ar_comb_f16_b", grid = "nrows", params = "nrows : int64")] +class ClsArCombF16B : ArCombBase { + @ssbo @binding = 3 outh : array //! the normed rows, f16 + + [spirv_kernel(local_size_x = 256, name = "cls_ar_comb_f16_b_spv"), arch(at="../ARCHITECTURE_GPU_VULKAN_MOE.md#vk-prefill-moe-block")] + def run { + let rbase = gl_WorkGroupID.x * pa.dim + let ss = accum_row(rbase) + let inv = rms_inv(ss) + var k = gl_LocalInvocationID.x + while (k < pa.dim) { + outh[rbase + k] = float16(wn[pa.woff + k] * (row[k] * inv)) + k += 256u + } + } +} + [vk_dispatch(name = "cls_ar_f16_b", grid = "nrows", params = "nrows : int64")] class ClsArAddRmsF16B : ArBase { @ssbo @binding = 3 outh : array // the normed rows, f16 @@ -545,6 +647,39 @@ class DecCombine { } } +struct DecCombineShArgs { + n : uint //! row width + k : uint //! routed slots + gidx : uint //! the shared expert's gate logit index in `gate` + gated : uint +} + +[vk_dispatch(name = "dec_combine_sh_cls", grid = "wgs", params = "wgs : int64")] +class DecCombineSh { + @ssbo @binding = 0 xin : array //! the residual row before the FFN + @ssbo @binding = 1 w : array //! the k routing weights + @ssbo @binding = 2 down_rows : array //! the routed down GEMV's k rows + @ssbo @binding = 3 sh_row : array //! the shared expert's down GEMV row + @ssbo @binding = 4 gate : array //! the logits row carrying the gate at pa.gidx + @ssbo @binding = 5 xout : array //! the residual row after the FFN + @push_constant pa : DecCombineShArgs + + [spirv_kernel(local_size_x = 256, name = "dec_combine_sh_cls_spv"), arch(at="../ARCHITECTURE_GPU_VULKAN_DECODE.md#whole-token-decode-span")] + def run { + let i = gl_GlobalInvocationID.x + if (i < pa.n) { + var acc = 0.0 + var j = 0u + while (j < pa.k) { + acc += w[j] * down_rows[j * pa.n + i] + j++ + } + let g = pa.gated != 0u ? 1.0 / (1.0 + exp(-gate[pa.gidx])) : 1.0 + xout[i] = xin[i] + acc + g * sh_row[i] + } + } +} + struct RouterArgs { ne : uint // expert rows dim : uint // row width @@ -686,14 +821,10 @@ struct TopkArgs { xnb2 : uint // the down feed's blocks per hidden row: slot j reads row j (gate/up read the one feed row) } -[vk_dispatch(name = "topk_cls", grid = "one", params = "one : int64")] -class TopK { - @ssbo @binding = 0 logits : array - @ssbo @binding = 1 meta1 : array // the gate stack's (wblock, xblock) per slot row - @ssbo @binding = 2 meta3 : array - @ssbo @binding = 3 meta2 : array - @ssbo @binding = 4 w : array // the k routing weights - @push_constant pa : TopkArgs +//! the select over one logit row, shared by the decode's single-row and the window's per-row +//! classes: thread tid holds logit lg, live = tid < ne, the picks land in pick / pickw for the +//! deriving class to write +class TopKCore { @workgroup part : float[64] @workgroup parti : uint[64] @workgroup pick : uint[64] @@ -701,12 +832,8 @@ class TopK { @workgroup wg_max : float[1] @workgroup wg_sum : float[1] - [spirv_kernel(local_size_x = 256, name = "topk_cls_spv"), arch(at="../ARCHITECTURE_GPU_VULKAN_DECODE.md#whole-token-decode-span")] - def run { + def select_row(lg : float; live : bool; ne, k : uint) { let tid = gl_LocalInvocationID.x - let live = tid < pa.ne - let lv = logits[min(tid, pa.ne - 1u)] - let lg = live ? lv : -3.0e38 let m0 = subgroupMax(lg) if (gl_SubgroupInvocationID == 0u) { part[gl_SubgroupID] = m0 @@ -742,7 +869,7 @@ class TopK { let wsum = wg_sum[0u] var p = live ? e / wsum : -3.0e38 var j = 0u - while (j < pa.k) { + while (j < k) { let vm = subgroupMax(p) let im = subgroupMin(p == vm ? tid : 0xFFFFFFFFu) if (gl_SubgroupInvocationID == 0u) { @@ -772,29 +899,52 @@ class TopK { } j++ } + } + + //! pick jj's routing weight: renormalized to a unit sum when norm is set, then scaled + def pick_weight(jj, k, norm : uint; wscale : float) : float { + var ws = 0.0 + var i = 0u + while (i < k) { + ws += pickw[i] + i++ + } + ws = max(ws, F16_MIN_NORMAL) + var wj = pickw[jj] + if (norm != 0u) { + wj /= ws + } + if (wscale != 0.0 && wscale != 1.0) { + wj *= wscale + } + return wj + } +} + +[vk_dispatch(name = "topk_cls", grid = "one", params = "one : int64")] +class TopK : TopKCore { + @ssbo @binding = 0 logits : array + @ssbo @binding = 1 meta1 : array //! the gate stack's (wblock, xblock) per slot row + @ssbo @binding = 2 meta3 : array + @ssbo @binding = 3 meta2 : array + @ssbo @binding = 4 w : array //! the k routing weights + @push_constant pa : TopkArgs + + [spirv_kernel(local_size_x = 256, name = "topk_cls_spv"), arch(at="../ARCHITECTURE_GPU_VULKAN_DECODE.md#whole-token-decode-span")] + def run { + let tid = gl_LocalInvocationID.x + let live = tid < pa.ne + let lv = logits[min(tid, pa.ne - 1u)] + select_row(live ? lv : -3.0e38, live, pa.ne, pa.k) if (tid == 0u) { write_picks() } } def write_picks { - var ws = 0.0 var jj = 0u while (jj < pa.k) { - ws += pickw[jj] - jj++ - } - ws = max(ws, F16_MIN_NORMAL) - jj = 0u - while (jj < pa.k) { - var wj = pickw[jj] - if (pa.norm != 0u) { - wj /= ws - } - if (pa.wscale != 0.0 && pa.wscale != 1.0) { - wj *= pa.wscale - } - w[jj] = wj + w[jj] = pick_weight(jj, pa.k, pa.norm, pa.wscale) let ex = min(pick[jj], pa.ne - 1u) // a NaN row leaves the sentinel: never a wild weight block meta1[jj * 2u] = pa.base1 + ex * pa.stride1 meta1[jj * 2u + 1u] = 0u @@ -807,6 +957,344 @@ class TopK { } } +struct TopkRowsArgs { + npos : uint + ne : uint + k : uint + norm : uint //! renormalize the selected weights to sum 1 + lstride : uint //! the logits row stride (elements) + wscale : float //! 0 or 1 = none +} + +[vk_dispatch(name = "topk_rows_cls", grid = "npos", params = "npos : int64")] +class TopKRows : TopKCore { + @ssbo @binding = 0 logits : array //! [npos x lstride] + @ssbo @binding = 1 idx : array //! [npos x k] the picked expert per slot + @ssbo @binding = 2 w : array //! ... and its routing weight + @push_constant pa : TopkRowsArgs + + [spirv_kernel(local_size_x = 256, name = "topk_rows_cls_spv"), arch(at="../ARCHITECTURE_GPU_VULKAN_MOE.md#vk-prefill-moe-block")] + def run { + let tid = gl_LocalInvocationID.x + let p = gl_WorkGroupID.x + let live = tid < pa.ne + let lv = logits[p * pa.lstride + min(tid, pa.ne - 1u)] + select_row(live ? lv : -3.0e38, live, pa.ne, pa.k) + if (tid == 0u) { + var jj = 0u + while (jj < pa.k) { + w[p * pa.k + jj] = pick_weight(jj, pa.k, pa.norm, pa.wscale) + idx[p * pa.k + jj] = min(pick[jj], pa.ne - 1u) + jj++ + } + } + } +} + +struct RouterGemmArgs { + npos : uint //! rows + ne : uint //! router rows (the experts, plus the shared expert's gate row when it rides along) + dim : uint //! row width + woff : uint //! this layer's row base in w (elements) + ostride : uint //! the logits row stride (elements) +} + +let ROUTER_TP = 64u //! the router tile's positions per workgroup +let ROUTER_TE = 32u //! ... and router rows +let ROUTER_TH = ROUTER_TE / 2u //! an invocation's two rows sit a half-tile apart: e and e + ROUTER_TH +let ROUTER_K4 = 16u //! a 64-wide K step in float4 +let ROUTER_LD = ROUTER_K4 + 1u //! the stage's row stride in float4: the step plus one of pad, so the sixteen rows a warp reads (and a fetch writes) spread the eight bank groups + +[vk_dispatch(name = "router_gemm_cls", grid = "wgs", params = "wgs : int64")] +class RouterGemm { + @ssbo @binding = 0 @role = "weight" w4 : array //! the router rows, row-major [ne x dim] per layer, as the stage's float4 + @ssbo @binding = 1 x4 : array //! the FFN-normed rows, as float4 + @ssbo @binding = 2 logits : array //! [npos x ostride] + @push_constant pa : RouterGemmArgs + @workgroup xs : float4[int(ROUTER_TP * ROUTER_LD)] + @workgroup ws : float4[int(ROUTER_TE * ROUTER_LD)] + + //! the stage's float4 of slot `s` (row s / ROUTER_K4, k4 s % ROUTER_K4) of x rows from `p0` at step `kb`; zero past the window + def fetch_x(p0, kb, s : uint) : float4 { + let row = s / ROUTER_K4 + if (p0 + row >= pa.npos) return float4(0.0) + return x4[((p0 + row) * pa.dim + kb) / 4u + (s % ROUTER_K4)] + } + + //! ... and of the router rows from `eg0`; zero past the experts + def fetch_w(eg0, kb, s : uint) : float4 { + let row = s / ROUTER_K4 + if (eg0 + row >= pa.ne) return float4(0.0) + return w4[(pa.woff + (eg0 + row) * pa.dim + kb) / 4u + (s % ROUTER_K4)] + } + + //! a fetched float4 into its stage row at k4 `s % ROUTER_K4` + def stage_x(s : uint; v : float4) { + xs[(s / ROUTER_K4) * ROUTER_LD + (s % ROUTER_K4)] = v + } + + def stage_w(s : uint; v : float4) { + ws[(s / ROUTER_K4) * ROUTER_LD + (s % ROUTER_K4)] = v + } + + //! `a` plus the four products in k order - the scalar loop's sums to the bit + def acc4(a : float; x, w : float4) : float { + var s = a + s += x.x * w.x + s += x.y * w.y + s += x.z * w.z + s += x.w * w.w + return s + } + + [spirv_kernel(local_size_x = 256, name = "router_gemm_cls_spv"), arch(at="../ARCHITECTURE_GPU_VULKAN_MOE.md#vk-prefill-moe-block")] + def run { + let tid = gl_LocalInvocationID.x + let egroups = (pa.ne + ROUTER_TE - 1u) / ROUTER_TE + let p0 = (gl_WorkGroupID.x / egroups) * ROUTER_TP + let eg0 = (gl_WorkGroupID.x % egroups) * ROUTER_TE + let tp = tid / ROUTER_TH //! the invocation's four positions 4tp..4tp+3 + let te = tid % ROUTER_TH //! ... and rows te, te + ROUTER_TH + var a00 = 0.0 + var a01 = 0.0 + var a10 = 0.0 + var a11 = 0.0 + var a20 = 0.0 + var a21 = 0.0 + var a30 = 0.0 + var a31 = 0.0 + //! the stage: 64 x rows x 16 float4 = four per invocation, 32 w rows x 16 = two + var xr0 = fetch_x(p0, 0u, tid) + var xr1 = fetch_x(p0, 0u, tid + 256u) + var xr2 = fetch_x(p0, 0u, tid + 512u) + var xr3 = fetch_x(p0, 0u, tid + 768u) + var wr0 = fetch_w(eg0, 0u, tid) + var wr1 = fetch_w(eg0, 0u, tid + 256u) + var kb = 0u + while (kb < pa.dim) { + stage_x(tid, xr0) + stage_x(tid + 256u, xr1) + stage_x(tid + 512u, xr2) + stage_x(tid + 768u, xr3) + stage_w(tid, wr0) + stage_w(tid + 256u, wr1) + barrier() + let nb = kb + 64u + if (nb < pa.dim) { + xr0 = fetch_x(p0, nb, tid) + xr1 = fetch_x(p0, nb, tid + 256u) + xr2 = fetch_x(p0, nb, tid + 512u) + xr3 = fetch_x(p0, nb, tid + 768u) + wr0 = fetch_w(eg0, nb, tid) + wr1 = fetch_w(eg0, nb, tid + 256u) + } + let xb = (4u * tp) * ROUTER_LD + let wb = te * ROUTER_LD + let wb1 = wb + ROUTER_TH * ROUTER_LD + var kk = 0u + while (kk < ROUTER_K4) { + let x0 = xs[xb + kk] + let x1 = xs[xb + ROUTER_LD + kk] + let x2 = xs[xb + 2u * ROUTER_LD + kk] + let x3 = xs[xb + 3u * ROUTER_LD + kk] + let w0 = ws[wb + kk] + let w1 = ws[wb1 + kk] + a00 = acc4(a00, x0, w0) + a01 = acc4(a01, x0, w1) + a10 = acc4(a10, x1, w0) + a11 = acc4(a11, x1, w1) + a20 = acc4(a20, x2, w0) + a21 = acc4(a21, x2, w1) + a30 = acc4(a30, x3, w0) + a31 = acc4(a31, x3, w1) + kk++ + } + barrier() + kb = nb + } + let pa0 = p0 + 4u * tp + let ea0 = eg0 + te + store_pair(pa0, ea0, a00, a01) + store_pair(pa0 + 1u, ea0, a10, a11) + store_pair(pa0 + 2u, ea0, a20, a21) + store_pair(pa0 + 3u, ea0, a30, a31) + } + + //! position p's two logits at rows e, e + ROUTER_TH, each within the window and the experts + def store_pair(p, e : uint; v0, v1 : float) { + if (p >= pa.npos) return + if (e < pa.ne) { + logits[p * pa.ostride + e] = v0 + } + if (e + ROUTER_TH < pa.ne) { + logits[p * pa.ostride + e + ROUTER_TH] = v1 + } + } +} + +struct SchedArgs { + nk : uint //! slot rows (npos x k) + ne : uint //! experts (at most 256: one thread per expert) + wt_gu : uint //! the gate/up GEMMs' weight tiles (ceil(nfe / 128)) + wt_dn : uint //! the down GEMM's (ceil(dim / 128)) + map_off : uint //! the s dispatch's per-wg map word offset in every sched plane; the records precede it + map_off_mgu : uint //! the m dispatch's map offset in the gate/up planes + map_off_mdn : uint //! ... and in the down plane + bound_gu : uint //! the s dispatch's workgroups per gate/up plane; the map's tail past the schedule reads the sentinel + bound_dn : uint + bound_mgu : uint //! the m dispatch's + bound_mdn : uint + base1 : uint //! expert 0's block base in the gate plane (slab-local, in the plane's units) + stride1 : uint //! blocks per expert + base3 : uint + stride3 : uint + base2 : uint + stride2 : uint +} + +let SCHED_S_ROWS = 32u //! the s column: the ladder's small-bucket and remainder tile +let SCHED_M_ROWS = 128u //! the m column: whole columns over a big bucket, the last one partial + +//! the ladder's m rows of a bucket of `c` rows: none up to the s column, else whole m columns plus a +//! partial one when the remainder passes the s column (a remainder within it takes the s column) +def sched_ladder_m_rows(c : uint) : uint { + if (c <= SCHED_S_ROWS) return 0u + return (c % SCHED_M_ROWS) > SCHED_S_ROWS ? c : (c / SCHED_M_ROWS) * SCHED_M_ROWS +} + +[vk_dispatch(name = "moe_sched_cls", grid = "one", params = "one : int64")] +class MoeSched { + @ssbo @binding = 0 idx : array //! [nk] the picked expert per slot row + @ssbo @binding = 1 inv : array //! [nk] slot row -> bucket row + @ssbo @binding = 2 sched1 : array //! the gate plane's records + maps + @ssbo @binding = 3 sched3 : array + @ssbo @binding = 4 sched2 : array + @push_constant pa : SchedArgs + @workgroup scan : uint[256] + @workgroup cnt : uint[256] //! the experts' slot counts, then their bucket cursors + + //! one piece's record and per-wg map on the three planes: expert `tid`'s `rows` from `row0`, + //! record `ri`, its `rt` column tiles from workgroup `wg_gu` / `wg_dn` of the dispatch whose + //! maps sit at `map_gu` / `map_dn` + def write_piece(ri, row0, rows, rt, wg_gu, wg_dn, map_gu, map_dn : uint) { + let tid = gl_LocalInvocationID.x + let rb = ri * 4u + sched1[rb] = pa.base1 + tid * pa.stride1 + sched1[rb + 1u] = row0 + sched1[rb + 2u] = rows + sched1[rb + 3u] = wg_gu + sched3[rb] = pa.base3 + tid * pa.stride3 + sched3[rb + 1u] = row0 + sched3[rb + 2u] = rows + sched3[rb + 3u] = wg_gu + sched2[rb] = pa.base2 + tid * pa.stride2 + sched2[rb + 1u] = row0 + sched2[rb + 2u] = rows + sched2[rb + 3u] = wg_dn + var g = 0u + while (g < rt * pa.wt_gu) { + sched1[map_gu + wg_gu + g] = ri + sched3[map_gu + wg_gu + g] = ri + g++ + } + g = 0u + while (g < rt * pa.wt_dn) { + sched2[map_dn + wg_dn + g] = ri + g++ + } + } + + //! the sentinel tails of the four maps, from the live workgroup counts (the s count in the low + //! half of each total, the m count in the high) to the dispatch bounds + def write_tails(total_gu, total_dn : uint) { + let tid = gl_LocalInvocationID.x + var t = (total_gu & 0xFFFFu) + tid + while (t < pa.bound_gu) { + sched1[pa.map_off + t] = SCHED_NONE + sched3[pa.map_off + t] = SCHED_NONE + t += 256u + } + t = (total_dn & 0xFFFFu) + tid + while (t < pa.bound_dn) { + sched2[pa.map_off + t] = SCHED_NONE + t += 256u + } + t = (total_gu >> 16u) + tid + while (t < pa.bound_mgu) { + sched1[pa.map_off_mgu + t] = SCHED_NONE + sched3[pa.map_off_mgu + t] = SCHED_NONE + t += 256u + } + t = (total_dn >> 16u) + tid + while (t < pa.bound_mdn) { + sched2[pa.map_off_mdn + t] = SCHED_NONE + t += 256u + } + } + + //! the exclusive prefix over the 256 threads' v; scan[255] holds the total after the call + def excl_scan(v : uint) : uint { + let tid = gl_LocalInvocationID.x + barrier() //! the previous scan's readers are done with scan + scan[tid] = v + barrier() + var off = 1u + while (off < 256u) { + var add = 0u + if (tid >= off) { + add = scan[tid - off] + } + barrier() + scan[tid] += add + barrier() + off = off * 2u + } + return scan[tid] - v + } + + [spirv_kernel(local_size_x = 256, name = "moe_sched_cls_spv"), arch(at="../ARCHITECTURE_GPU_VULKAN_MOE.md#vk-prefill-moe-block")] + def run { + let tid = gl_LocalInvocationID.x + let mine = tid < pa.ne + cnt[tid] = 0u + barrier() + var i = tid + while (i < pa.nk) { //! the top-k clamps every pick under ne, so the tally stays inside cnt + let _c = atomicAdd(cnt[idx[i]], 1u) + i += 256u + } + barrier() + let c = mine ? cnt[tid] : 0u + let st = excl_scan(c) + let mrows = mine ? sched_ladder_m_rows(c) : 0u + let srows = mine ? c - mrows : 0u + let ri_s = excl_scan(srows > 0u ? 1u : 0u) + let ri_m = pa.ne + excl_scan(mrows > 0u ? 1u : 0u) + let rt_s = (srows + SCHED_S_ROWS - 1u) / SCHED_S_ROWS + let rt_m = (mrows + SCHED_M_ROWS - 1u) / SCHED_M_ROWS + //! the packed word each scan carries: the s tile count in the low half, the m count in the high + let g_gu = excl_scan(rt_s * pa.wt_gu | ((rt_m * pa.wt_gu) << 16u)) + let total_gu = scan[255u] + let g_dn = excl_scan(rt_s * pa.wt_dn | ((rt_m * pa.wt_dn) << 16u)) + let total_dn = scan[255u] + if (srows > 0u) { + write_piece(ri_s, st + mrows, srows, rt_s, g_gu & 0xFFFFu, g_dn & 0xFFFFu, pa.map_off, pa.map_off) + } + if (mrows > 0u) { + write_piece(ri_m, st, mrows, rt_m, g_gu >> 16u, g_dn >> 16u, pa.map_off_mgu, pa.map_off_mdn) + } + write_tails(total_gu, total_dn) + barrier() //! every thread has read its count + cnt[tid] = st + barrier() + i = tid + while (i < pa.nk) { + inv[i] = atomicAdd(cnt[idx[i]], 1u) + i += 256u + } + } +} + struct GatherArgs { n : uint // row width k : uint // slots per position (grid entry r sits at position r / k) @@ -2458,12 +2946,17 @@ struct BatchArgs { // the MoE region rail every batch tile shares: sched = 4-word region records [wblk0, row0, cnt, // wg0] at 0, then the per-wg map at pa.map_off; region_rec = this workgroup's record - x = weight // base, y = activation row base, z = row count, w = the wg's tile index rebased to the region +let SCHED_NONE = 0xFFFFFFFFu //! the per-wg map's sentinel: a tile workgroup past a device-written schedule's end reads a zero-row region and returns + class MoeCmBase { @ssbo @binding = 2 sched : array // region records + per-wg map (bulk schedule data) @push_constant pa : BatchArgs def region_rec : uint4 { let rid = sched[pa.map_off + gl_WorkGroupID.x] + if (rid == SCHED_NONE) { + return uint4(0u, 0u, 0u, 0u) + } let rb = rid * 4u return uint4(sched[rb], sched[rb + 1u], sched[rb + 2u], gl_WorkGroupID.x - sched[rb + 3u]) } @@ -4713,6 +5206,9 @@ class template KqCm2BatchT : MoeCmBase { let wblk0 = reg.x let row0 = reg.y let cnt = reg.z + if (cnt == 0u) { + return + } let ttiles = (cnt + BN - 1u) / BN // split-k: wgs stack nsplit planes of (wtiles x ttiles); plane ks covers k [k0, k1) and // stores f32 partials at ybase (the reduce sums the planes into y). SINGLE-REGION (dense) @@ -4791,12 +5287,17 @@ class template KqCm2BatchT : MoeCmBase { tensorLayoutSetStride(fla, pa.n / BLKW, 1u) var flb : tensorLayout2D tensorLayoutCreate(flb) - tensorLayoutSetDimension(flb, row0 + (STILE ? ((cnt + 31u) & ~31u) : cnt), pa.n) + tensorLayoutSetDimension(flb, row0 + (STILE ? ((cnt + BN - 1u) & ~(BN - 1u)) : cnt), pa.n) tensorLayoutSetStride(flb, pa.n & ~7u, 1u) - var flo : FLO - tensorLayoutCreate(flo) - tensorLayoutSetDimension(flo, row0 + cnt, pa.d) - tensorLayoutSetStride(flo, pa.d & ~7u, 1u) + let whole = xt * BN + BN <= cnt + var flw : tensorLayout2D + tensorLayoutCreate(flw) + tensorLayoutSetDimension(flw, row0 + cnt, pa.d) + tensorLayoutSetStride(flw, pa.d & ~7u, 1u) + var flp : tensorLayout2DPad + tensorLayoutCreate(flp) + tensorLayoutSetDimension(flp, row0 + cnt, pa.d) + tensorLayoutSetStride(flp, pa.d & ~7u, 1u) if (pa.ksplit == 0u) { var k = 0u for (_i in range(int(pa.n / 512u))) { @@ -4824,7 +5325,11 @@ class template KqCm2BatchT : MoeCmBase { coopmatClamp(acc, -65504.0, 65504.0) var accw : ACCW coopmatConvert(accw, acc) - coopmatStoreTensor(accw, y, 0u, flo, t0, BN, m0, 128u, tv) + if (whole) { + coopmatStoreTensor(accw, y, 0u, flw, t0, BN, m0, 128u, tv) + } else { + coopmatStoreTensor(accw, y, 0u, flp, t0, BN, m0, 128u, tv) + } return } var k = k0 @@ -4853,7 +5358,11 @@ class template KqCm2BatchT : MoeCmBase { coopmatClamp(acc, -65504.0, 65504.0) var accw : ACCW coopmatConvert(accw, acc) - coopmatStoreTensor(accw, y, ybase, flo, t0, BN, m0, 128u, tv) + if (whole) { + coopmatStoreTensor(accw, y, ybase, flw, t0, BN, m0, 128u, tv) + } else { + coopmatStoreTensor(accw, y, ybase, flp, t0, BN, m0, 128u, tv) + } return } // EDGE PATH — clamp-Constant layouts everywhere (store discard load-bearing) @@ -5498,16 +6007,16 @@ class template Iq3sCm2T : KqCm2BatchT { let r = e & 31u let wj = r >> 2u let qi = g * 8u + wj - let qb = uint(int(unpack8(blk.qs[int(qi >> 1u)])[int(qi & 1u)])) & 0xFFu - let qh = uint(int(unpack8(blk.qh[int(g >> 1u)])[int(g & 1u)])) & 0xFFu + let qb = ((uint(int(blk.qs[int(qi >> 1u)])) & 0xFFFFu) >> ((qi & 1u) * 8u)) & 0xFFu //! the grid-index byte, shifted out of its lane + let qhw = (uint(int(blk.qh[int(g >> 1u)])) & 0xFFFFu) >> ((g & 1u) * 8u) //! the ninth-bit byte g, shifted out of its lane let si = g * 4u + (r >> 3u) - let sgb = uint(int(unpack8(blk.sg[int(si >> 1u)])[int(si & 1u)])) & 0xFFu - let gw = iq3s_gridc[qb | ((qh << (8u - wj)) & 256u)] + let sgw = (uint(int(blk.sg[int(si >> 1u)])) & 0xFFFFu) >> ((si & 1u) * 8u) //! the sign byte, shifted out of its lane + let gw = iq3s_gridc[qb | ((qhw << (8u - wj)) & 256u)] let gb = float((gw >> ((r & 3u) * 8u)) & 0xFFu) let srow = (wg_blk0 + bc.x * (pa.n >> 8u) + bc.y) * 2u let d = unpackHalf2x16(ws[srow]).x let sc = grid_sc(ws[srow + 1u], g) - return float16(d * float(sc) * (((sgb >> (((r >> 2u) & 1u) * 4u + (r & 3u))) & 1u) != 0u ? -gb : gb)) + return float16(d * float(sc) * (((sgw >> (((r >> 2u) & 1u) * 4u + (r & 3u))) & 1u) != 0u ? -gb : gb)) } [spirv_decode, arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-decode-16bit-lanes")] @@ -5517,16 +6026,16 @@ class template Iq3sCm2T : KqCm2BatchT { let r = e & 31u let wj = r >> 2u let qi = g * 8u + wj - let qb = uint(int(unpack8(blk.qs[int(qi >> 1u)])[int(qi & 1u)])) & 0xFFu - let qh = uint(int(unpack8(blk.qh[int(g >> 1u)])[int(g & 1u)])) & 0xFFu + let qb = ((uint(int(blk.qs[int(qi >> 1u)])) & 0xFFFFu) >> ((qi & 1u) * 8u)) & 0xFFu + let qhw = (uint(int(blk.qh[int(g >> 1u)])) & 0xFFFFu) >> ((g & 1u) * 8u) let si = g * 4u + (r >> 3u) - let sgb = uint(int(unpack8(blk.sg[int(si >> 1u)])[int(si & 1u)])) & 0xFFu - let gw = iq3s_gridc[qb | ((qh << (8u - wj)) & 256u)] + let sgw = (uint(int(blk.sg[int(si >> 1u)])) & 0xFFFFu) >> ((si & 1u) * 8u) + let gw = iq3s_gridc[qb | ((qhw << (8u - wj)) & 256u)] let srow = (wg_blk0 + bc.x * (pa.n >> 8u) + bc.y) * 2u let d = unpackHalf2x16(ws[srow]).x let sc = grid_sc(ws[srow + 1u], g) let ds = d * float(sc) - let sgn = sgb >> (((r >> 2u) & 1u) * 4u) + let sgn = sgw >> (((r >> 2u) & 1u) * 4u) let g0 = float(gw & 0xFFu) let g1 = float((gw >> 8u) & 0xFFu) let g2 = float((gw >> 16u) & 0xFFu) @@ -5577,13 +6086,11 @@ class template Iq3xxsCm2T : KqCm2BatchT { let r = e & 31u let wj = r >> 2u let qi = g * 8u + wj - let qb = uint(int(unpack8(blk.qs[int(qi >> 1u)])[int(qi & 1u)])) & 0xFFu + let qb = ((uint(int(blk.qs[int(qi >> 1u)])) & 0xFFFFu) >> ((qi & 1u) * 8u)) & 0xFFu let l = r >> 3u - let bl = 7u * l - let a0i = g * 4u + (bl >> 3u) - let b0 = uint(int(unpack8(blk.ax[int(a0i >> 1u)])[int(a0i & 1u)])) & 0xFFu - let b1 = uint(int(unpack8(blk.ax[int((a0i + 1u) >> 1u)])[int((a0i + 1u) & 1u)])) & 0xFFu - let sidx = ((b0 | (b1 << 8u)) >> (bl & 7u)) & 127u + let ai = int(g * 2u) //! sub-block g's aux32 as its two 16-bit lanes: four 7-bit ksigns indices + let aux = (uint(int(blk.ax[ai])) & 0xFFFFu) | ((uint(int(blk.ax[ai + 1])) & 0xFFFFu) << 16u) + let sidx = (aux >> (7u * l)) & 127u var tt = sidx ^ (sidx >> 4u) tt = tt ^ (tt >> 2u) tt = tt ^ (tt >> 1u) @@ -5603,13 +6110,11 @@ class template Iq3xxsCm2T : KqCm2BatchT { let r = e & 31u let wj = r >> 2u let qi = g * 8u + wj - let qb = uint(int(unpack8(blk.qs[int(qi >> 1u)])[int(qi & 1u)])) & 0xFFu + let qb = ((uint(int(blk.qs[int(qi >> 1u)])) & 0xFFFFu) >> ((qi & 1u) * 8u)) & 0xFFu let l = r >> 3u - let bl = 7u * l - let a0i = g * 4u + (bl >> 3u) - let b0 = uint(int(unpack8(blk.ax[int(a0i >> 1u)])[int(a0i & 1u)])) & 0xFFu - let b1 = uint(int(unpack8(blk.ax[int((a0i + 1u) >> 1u)])[int((a0i + 1u) & 1u)])) & 0xFFu - let sidx = ((b0 | (b1 << 8u)) >> (bl & 7u)) & 127u + let ai = int(g * 2u) + let aux = (uint(int(blk.ax[ai])) & 0xFFFFu) | ((uint(int(blk.ax[ai + 1])) & 0xFFFFu) << 16u) + let sidx = (aux >> (7u * l)) & 127u var tt = sidx ^ (sidx >> 4u) tt = tt ^ (tt >> 2u) tt = tt ^ (tt >> 1u) @@ -5723,16 +6228,16 @@ class template Iq2sCm2T : KqCm2BatchT { let l = r >> 3u let j = r & 7u let qi = g * 4u + l - let qb = uint(int(unpack8(blk.qs[int(qi >> 1u)])[int(qi & 1u)])) & 0xFFu - let qh = uint(int(unpack8(blk.qh[int(g >> 1u)])[int(g & 1u)])) & 0xFFu - let sgb = uint(int(unpack8(blk.sg[int(qi >> 1u)])[int(qi & 1u)])) & 0xFFu - let gw = iq2s_gridc[(qb | ((qh << (8u - 2u * l)) & 0x300u)) * 2u + (j >> 2u)] + let qb = ((uint(int(blk.qs[int(qi >> 1u)])) & 0xFFFFu) >> ((qi & 1u) * 8u)) & 0xFFu + let qhw = (uint(int(blk.qh[int(g >> 1u)])) & 0xFFFFu) >> ((g & 1u) * 8u) + let sgw = (uint(int(blk.sg[int(qi >> 1u)])) & 0xFFFFu) >> ((qi & 1u) * 8u) + let gw = iq2s_gridc[(qb | ((qhw << (8u - 2u * l)) & 0x300u)) * 2u + (j >> 2u)] let gb = float((gw >> ((j & 3u) * 8u)) & 0xFFu) let srow = (wg_blk0 + bc.x * (pa.n >> 8u) + bc.y) * 5u let d = unpackHalf2x16(ws[srow]).x let si = g * 2u + (r >> 4u) let sc = (ws[srow + 1u + (si >> 2u)] >> ((si & 3u) * 8u)) & 0xFFu - return float16(d * float(sc) * (((sgb >> j) & 1u) != 0u ? -gb : gb)) + return float16(d * float(sc) * (((sgw >> j) & 1u) != 0u ? -gb : gb)) } [spirv_decode, arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-decode-16bit-lanes")] @@ -5743,16 +6248,16 @@ class template Iq2sCm2T : KqCm2BatchT { let l = r >> 3u let j = r & 7u let qi = g * 4u + l - let qb = uint(int(unpack8(blk.qs[int(qi >> 1u)])[int(qi & 1u)])) & 0xFFu - let qh = uint(int(unpack8(blk.qh[int(g >> 1u)])[int(g & 1u)])) & 0xFFu - let sgb = uint(int(unpack8(blk.sg[int(qi >> 1u)])[int(qi & 1u)])) & 0xFFu - let gw = iq2s_gridc[(qb | ((qh << (8u - 2u * l)) & 0x300u)) * 2u + (j >> 2u)] + let qb = ((uint(int(blk.qs[int(qi >> 1u)])) & 0xFFFFu) >> ((qi & 1u) * 8u)) & 0xFFu + let qhw = (uint(int(blk.qh[int(g >> 1u)])) & 0xFFFFu) >> ((g & 1u) * 8u) + let sgw = (uint(int(blk.sg[int(qi >> 1u)])) & 0xFFFFu) >> ((qi & 1u) * 8u) + let gw = iq2s_gridc[(qb | ((qhw << (8u - 2u * l)) & 0x300u)) * 2u + (j >> 2u)] let srow = (wg_blk0 + bc.x * (pa.n >> 8u) + bc.y) * 5u let d = unpackHalf2x16(ws[srow]).x let si = g * 2u + (r >> 4u) let sc = (ws[srow + 1u + (si >> 2u)] >> ((si & 3u) * 8u)) & 0xFFu let ds = d * float(sc) - let sgn = sgb >> j + let sgn = sgw >> j let g0 = float(gw & 0xFFu) let g1 = float((gw >> 8u) & 0xFFu) let g2 = float((gw >> 16u) & 0xFFu) @@ -5882,12 +6387,10 @@ class template Iq2xxsCm2T : KqCm2BatchT { let l = r >> 3u let j = r & 7u let qi = g * 8u + l - let qb = uint(int(unpack8(blk.qs[int(qi >> 1u)])[int(qi & 1u)])) & 0xFFu - let bl = 7u * l - let a0i = g * 8u + 4u + (bl >> 3u) - let b0 = uint(int(unpack8(blk.qs[int(a0i >> 1u)])[int(a0i & 1u)])) & 0xFFu - let b1 = uint(int(unpack8(blk.qs[int((a0i + 1u) >> 1u)])[int((a0i + 1u) & 1u)])) & 0xFFu - let sidx = ((b0 | (b1 << 8u)) >> (bl & 7u)) & 127u + let qb = ((uint(int(blk.qs[int(qi >> 1u)])) & 0xFFFFu) >> ((qi & 1u) * 8u)) & 0xFFu + let ai = int(g * 4u + 2u) //! sub-block g's aux32 as its two 16-bit lanes: four 7-bit ksigns indices + let aux = (uint(int(blk.qs[ai])) & 0xFFFFu) | ((uint(int(blk.qs[ai + 1])) & 0xFFFFu) << 16u) + let sidx = (aux >> (7u * l)) & 127u var tt = sidx ^ (sidx >> 4u) tt = tt ^ (tt >> 2u) tt = tt ^ (tt >> 1u) @@ -5908,12 +6411,10 @@ class template Iq2xxsCm2T : KqCm2BatchT { let l = r >> 3u let j = r & 7u let qi = g * 8u + l - let qb = uint(int(unpack8(blk.qs[int(qi >> 1u)])[int(qi & 1u)])) & 0xFFu - let bl = 7u * l - let a0i = g * 8u + 4u + (bl >> 3u) - let b0 = uint(int(unpack8(blk.qs[int(a0i >> 1u)])[int(a0i & 1u)])) & 0xFFu - let b1 = uint(int(unpack8(blk.qs[int((a0i + 1u) >> 1u)])[int((a0i + 1u) & 1u)])) & 0xFFu - let sidx = ((b0 | (b1 << 8u)) >> (bl & 7u)) & 127u + let qb = ((uint(int(blk.qs[int(qi >> 1u)])) & 0xFFFFu) >> ((qi & 1u) * 8u)) & 0xFFu + let ai = int(g * 4u + 2u) + let aux = (uint(int(blk.qs[ai])) & 0xFFFFu) | ((uint(int(blk.qs[ai + 1])) & 0xFFFFu) << 16u) + let sidx = (aux >> (7u * l)) & 127u var tt = sidx ^ (sidx >> 4u) tt = tt ^ (tt >> 2u) tt = tt ^ (tt >> 1u) @@ -5939,16 +6440,15 @@ class Q8Cm2LBatch : Q8Cm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "q8_batch_cm2m_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Q8Cm2MBatch : Q8Cm2T { override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "q8_batch_cm2s_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -5958,7 +6458,6 @@ class Q8Cm2SBatch : Q8Cm2T { typedef BT = coopmatWgB_f16_64x32 typedef ACC = coopmatWgAcc_f16_128x32 typedef ACCW = coopmatWgAcc_f32_128x32 - typedef FLO = tensorLayout2DPad } [vk_dispatch(name = "kq_batch_k4_cm2l_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -5966,16 +6465,15 @@ class K4Cm2LBatch : K4Cm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_k4_cm2m_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class K4Cm2MBatch : K4Cm2T { override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_k4_cm2s_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -5985,7 +6483,6 @@ class K4Cm2SBatch : K4Cm2T { typedef BT = coopmatWgB_f16_64x32 typedef ACC = coopmatWgAcc_f16_128x32 typedef ACCW = coopmatWgAcc_f32_128x32 - typedef FLO = tensorLayout2DPad } [vk_dispatch(name = "kq_batch_k6_cm2l_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -5993,16 +6490,15 @@ class K6Cm2LBatch : K6Cm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_k6_cm2m_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class K6Cm2MBatch : K6Cm2T { override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_k6_cm2s_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6012,7 +6508,6 @@ class K6Cm2SBatch : K6Cm2T { typedef BT = coopmatWgB_f16_64x32 typedef ACC = coopmatWgAcc_f16_128x32 typedef ACCW = coopmatWgAcc_f32_128x32 - typedef FLO = tensorLayout2DPad } [vk_dispatch(name = "kq_batch_k5_cm2l_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6020,16 +6515,15 @@ class K5Cm2LBatch : K5Cm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_k5_cm2m_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class K5Cm2MBatch : K5Cm2T { override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_k5_cm2s_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6039,7 +6533,6 @@ class K5Cm2SBatch : K5Cm2T { typedef BT = coopmatWgB_f16_64x32 typedef ACC = coopmatWgAcc_f16_128x32 typedef ACCW = coopmatWgAcc_f32_128x32 - typedef FLO = tensorLayout2DPad } [vk_dispatch(name = "kq_batch_q40_cm2l_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6047,16 +6540,15 @@ class Q40Cm2LBatch : Q40Cm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_q40_cm2m_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Q40Cm2MBatch : Q40Cm2T { override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_q40_cm2s_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6066,7 +6558,6 @@ class Q40Cm2SBatch : Q40Cm2T { typedef BT = coopmatWgB_f16_64x32 typedef ACC = coopmatWgAcc_f16_128x32 typedef ACCW = coopmatWgAcc_f32_128x32 - typedef FLO = tensorLayout2DPad } [vk_dispatch(name = "kq_batch_iq4xs_cm2l_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6074,16 +6565,15 @@ class Iq4xsCm2LBatch : Iq4xsCm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq4xs_cm2m_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Iq4xsCm2MBatch : Iq4xsCm2T { override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq4xs_cm2s_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6093,7 +6583,6 @@ class Iq4xsCm2SBatch : Iq4xsCm2T { typedef BT = coopmatWgB_f16_64x32 typedef ACC = coopmatWgAcc_f16_128x32 typedef ACCW = coopmatWgAcc_f32_128x32 - typedef FLO = tensorLayout2DPad } [vk_dispatch(name = "kq_batch_k3_cm2l_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6101,16 +6590,15 @@ class K3Cm2LBatch : K3Cm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_k3_cm2m_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class K3Cm2MBatch : K3Cm2T { override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_k3_cm2s_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6120,7 +6608,6 @@ class K3Cm2SBatch : K3Cm2T { typedef BT = coopmatWgB_f16_64x32 typedef ACC = coopmatWgAcc_f16_128x32 typedef ACCW = coopmatWgAcc_f32_128x32 - typedef FLO = tensorLayout2DPad } [vk_dispatch(name = "kq_batch_iq3s_cm2l_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6128,16 +6615,15 @@ class Iq3sCm2LBatch : Iq3sCm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq3s_cm2m_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Iq3sCm2MBatch : Iq3sCm2T { override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq3s_cm2s_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6147,7 +6633,6 @@ class Iq3sCm2SBatch : Iq3sCm2T { typedef BT = coopmatWgB_f16_64x32 typedef ACC = coopmatWgAcc_f16_128x32 typedef ACCW = coopmatWgAcc_f32_128x32 - typedef FLO = tensorLayout2DPad } [vk_dispatch(name = "kq_batch_iq3xxs_cm2l_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6155,16 +6640,15 @@ class Iq3xxsCm2LBatch : Iq3xxsCm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq3xxs_cm2m_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Iq3xxsCm2MBatch : Iq3xxsCm2T { override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq3xxs_cm2s_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6174,7 +6658,6 @@ class Iq3xxsCm2SBatch : Iq3xxsCm2T { typedef BT = coopmatWgB_f16_64x32 typedef ACC = coopmatWgAcc_f16_128x32 typedef ACCW = coopmatWgAcc_f32_128x32 - typedef FLO = tensorLayout2DPad } [vk_dispatch(name = "kq_batch_iq4nl_cm2l_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6182,16 +6665,15 @@ class Iq4nlCm2LBatch : Iq4nlCm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq4nl_cm2m_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Iq4nlCm2MBatch : Iq4nlCm2T { override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq4nl_cm2s_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6201,7 +6683,6 @@ class Iq4nlCm2SBatch : Iq4nlCm2T { typedef BT = coopmatWgB_f16_64x32 typedef ACC = coopmatWgAcc_f16_128x32 typedef ACCW = coopmatWgAcc_f32_128x32 - typedef FLO = tensorLayout2DPad } [vk_dispatch(name = "kq_batch_k2_cm2l_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6209,16 +6690,15 @@ class K2Cm2LBatch : K2Cm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_k2_cm2m_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class K2Cm2MBatch : K2Cm2T { override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_k2_cm2s_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6228,7 +6708,6 @@ class K2Cm2SBatch : K2Cm2T { typedef BT = coopmatWgB_f16_64x32 typedef ACC = coopmatWgAcc_f16_128x32 typedef ACCW = coopmatWgAcc_f32_128x32 - typedef FLO = tensorLayout2DPad } [vk_dispatch(name = "kq_batch_iq2s_cm2l_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6236,16 +6715,15 @@ class Iq2sCm2LBatch : Iq2sCm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq2s_cm2m_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Iq2sCm2MBatch : Iq2sCm2T { override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq2s_cm2s_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6255,7 +6733,6 @@ class Iq2sCm2SBatch : Iq2sCm2T { typedef BT = coopmatWgB_f16_64x32 typedef ACC = coopmatWgAcc_f16_128x32 typedef ACCW = coopmatWgAcc_f32_128x32 - typedef FLO = tensorLayout2DPad } [vk_dispatch(name = "kq_batch_iq2xs_cm2l_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6263,16 +6740,15 @@ class Iq2xsCm2LBatch : Iq2xsCm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq2xs_cm2m_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Iq2xsCm2MBatch : Iq2xsCm2T { override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq2xs_cm2s_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6282,7 +6758,6 @@ class Iq2xsCm2SBatch : Iq2xsCm2T { typedef BT = coopmatWgB_f16_64x32 typedef ACC = coopmatWgAcc_f16_128x32 typedef ACCW = coopmatWgAcc_f32_128x32 - typedef FLO = tensorLayout2DPad } [vk_dispatch(name = "kq_batch_iq2xxs_cm2l_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6290,16 +6765,15 @@ class Iq2xxsCm2LBatch : Iq2xxsCm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq2xxs_cm2m_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Iq2xxsCm2MBatch : Iq2xxsCm2T { override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq2xxs_cm2s_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -6309,7 +6783,6 @@ class Iq2xxsCm2SBatch : Iq2xxsCm2T { typedef BT = coopmatWgB_f16_64x32 typedef ACC = coopmatWgAcc_f16_128x32 typedef ACCW = coopmatWgAcc_f32_128x32 - typedef FLO = tensorLayout2DPad } // the KHR arm of every kq format (KHR, BN 128); the cm2 typedefs belong to the template's uncalled @@ -6318,130 +6791,130 @@ class Iq2xxsCm2SBatch : Iq2xxsCm2T { class K4KhrBatch : K4Cm2T { override KHR = true override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_k5_khr_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class K5KhrBatch : K5Cm2T { override KHR = true override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_k6_khr_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class K6KhrBatch : K6Cm2T { override KHR = true override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_q40_khr_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Q40KhrBatch : Q40Cm2T { override KHR = true override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq4xs_khr_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Iq4xsKhrBatch : Iq4xsCm2T { override KHR = true override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_k3_khr_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class K3KhrBatch : K3Cm2T { override KHR = true override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq3s_khr_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Iq3sKhrBatch : Iq3sCm2T { override KHR = true override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq3xxs_khr_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Iq3xxsKhrBatch : Iq3xxsCm2T { override KHR = true override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq4nl_khr_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Iq4nlKhrBatch : Iq4nlCm2T { override KHR = true override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_k2_khr_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class K2KhrBatch : K2Cm2T { override KHR = true override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq2s_khr_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Iq2sKhrBatch : Iq2sCm2T { override KHR = true override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq2xs_khr_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Iq2xsKhrBatch : Iq2xsCm2T { override KHR = true override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "kq_batch_iq2xxs_khr_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] class Iq2xxsKhrBatch : Iq2xxsCm2T { override KHR = true override BN = 128u + override STILE = true typedef BT = coopmatWgB_f16_64x128 typedef ACC = coopmatWgAcc_f16_128x128 typedef ACCW = coopmatWgAcc_f32_128x128 - typedef FLO = tensorLayout2D } struct SkRedArgs { @@ -6832,6 +7305,7 @@ struct GemvArgs { d : uint // output rows per region nreg : uint // region count ybase : uint // output base row + lanes_per_row : uint //! 8, 16, or 0 = the whole subgroup (the q8 GEMV ignores it) } // ===== the Q8_0 decode GEMV pair ===== @@ -6886,10 +7360,10 @@ class Q8Gemv { } // ===== the K-quant decode GEMV family: one base shell, per-format block contributions ===== -// ONE LANE PER 32-BLOCK via uint4 loads (512B/warp in flight, one scale decode per block). -// The four formats share the region walk, the lane->block map, and the subgroup fold — only -// the block's dot + scale fold differ, so each format is an override of blk_contrib and the -// devirtualizer inlines the family back to four standalone kernels at emission. +// ONE LANE PER 32-BLOCK via uint4 loads (one scale decode per block). +// The formats share the region walk, the lane->block map, and the subgroup fold — only the +// block's dot + scale fold differ, so each format is an override of blk_contrib and the +// devirtualizer inlines the family back to standalone kernels at emission. let private KQC_LOW = 0x0F0F0F0Fu // nibble mask let private KQC_ONES = 0x01010101u // sdot4 against this sums 4 signed bytes @@ -6972,28 +7446,45 @@ class KqGemvBase { } def gemv_shell { - let rg = gl_WorkGroupID.x * gl_NumSubgroups + gl_SubgroupID - if (rg < pa.nreg * pa.d) { + let lanes_per_row = pa.lanes_per_row == 0u ? gl_SubgroupSize : pa.lanes_per_row + let lane = gl_SubgroupInvocationID + let row_lane = lane % lanes_per_row + let rg = (gl_WorkGroupID.x * gl_NumSubgroups + gl_SubgroupID) * (gl_SubgroupSize / lanes_per_row) + lane / lanes_per_row + let live = rg < pa.nreg * pa.d + var acc = 0.0 + if (live) { let r = rg / pa.d let row = rg % pa.d let nsb = pa.n / 256u let wsb0 = regions[r * 2u] + row * nsb let xsb0 = regions[r * 2u + 1u] - let lane = gl_SubgroupInvocationID let nb = nsb * 8u - var acc = 0.0 var b0 = 0u while (b0 < nb) { - let bb = b0 + lane + let bb = b0 + row_lane if (bb < nb) { acc += blk_contrib(wsb0 + bb / 8u, bb % 8u, xsb0 + bb / 8u) } - b0 += gl_SubgroupSize - } - let total = subgroupAdd(acc) - if (subgroupElect()) { - y[pa.ybase + rg] = total - } + b0 += lanes_per_row + } + } + //! the row's fold is an xor-shuffle butterfly over its lanes (the shuffle every GEMV already requires; + //! a clustered add would want the clustered subgroup feature the tier never checks) + var total = acc + if (lanes_per_row == 8u) { + total += subgroupShuffleXor(total, 4u) + total += subgroupShuffleXor(total, 2u) + total += subgroupShuffleXor(total, 1u) + } elif (lanes_per_row == 16u) { + total += subgroupShuffleXor(total, 8u) + total += subgroupShuffleXor(total, 4u) + total += subgroupShuffleXor(total, 2u) + total += subgroupShuffleXor(total, 1u) + } else { + total = subgroupAdd(acc) + } + if (live && row_lane == 0u) { + y[pa.ybase + rg] = total } } } @@ -7238,6 +7729,36 @@ def gemv_cls_enc(fmt : int; raw : VkCommandBuffer; var h : VkHaz; var set_ : VkD } } +//! the GEMV family's dispatch grid over `rows` output rows at `lanes_per_row` (0 = a whole subgroup per +//! row, the q8 GEMV's only form): a workgroup's subgroups each take subgroup_size / lanes_per_row rows +def gemv_wgs(rows : int64; lanes_per_row : uint) : int64 { + let l = lanes_per_row == 0u ? g_gpu.subgroup_size : int64(lanes_per_row) + let rpw = g_gpu.rows_per_wg * (g_gpu.subgroup_size / l) + return (rows + rpw - 1l) / rpw +} + +//! the codebook and grid formats: a block's decode gathers, so a lane wants more blocks in flight than the k-lattice's +def gemv_grid_fmt(fmt : int) : bool { + return fmt == int(KqFmt.iq2s) || fmt == int(KqFmt.iq2xs) || fmt == int(KqFmt.iq2xxs) || fmt == int(KqFmt.iq3xxs) || fmt == int(KqFmt.iq4xs) || fmt == int(KqFmt.iq4nl) +} + +[arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#kq-gemv-lanes")] +def gemv_lanes_per_row(fmt : int; n : int64) : uint { + if (fmt == int(KqFmt.q8)) return 0u + let nb = n / 32l + let grid = gemv_grid_fmt(fmt) + if (nb <= 24l) return 8u + if (nb <= 48l) return grid ? 8u : 16u + if (nb <= 96l) return 16u + return grid ? 16u : 0u +} + +//! the family's dispatch over `rows` output rows: the lanes per row by the row length, the grid to match +def gemv_enc(fmt : int; raw : VkCommandBuffer; var h : VkHaz; var set_ : VkDescriptorSet; var pc : GemvArgs; rows : int64) { + pc.lanes_per_row = gemv_lanes_per_row(fmt, int64(pc.n)) + gemv_cls_enc(fmt, raw, h, set_, pc, gemv_wgs(rows, pc.lanes_per_row)) +} + // Q4_K: w = d*sc*q - dmin*mn*1, q in [0,15] — per 32-block axs*(d*sc*idot - dmin*mn*bsum). // Every kq format shares family=kq_gemv_cls: ONE set layout + ONE set_kq_gemv_cls serves every // pipeline (identical inherited members), each leaf keeping its own ensure_/enc_ names. diff --git a/modules/dasLLAMA/dasllama/dasllama_vulkan_common.das b/modules/dasLLAMA/dasllama/dasllama_vulkan_common.das index 0e852a6cd1..8f2a2f2b9d 100644 --- a/modules/dasLLAMA/dasllama/dasllama_vulkan_common.das +++ b/modules/dasLLAMA/dasllama/dasllama_vulkan_common.das @@ -443,6 +443,13 @@ struct SpanLayer { s_comb : VkDescriptorSet // x += sum w_j * down_j s_ar_att : VkDescriptorSet // xb = rms(x) * the NEXT layer's att_norm (absent on the last) s_rq_att : VkDescriptorSet // xb -> the next layer's q-stack feed image + sh1 : int = -1 //! the shared expert's q8 gate stack (-1 = the model has none) + sh3 : int = -1 //! ... its up stack + sh2 : int = -1 //! ... its down stack + s_rq_sh1 : VkDescriptorSet //! xb -> the shared gate stack's feed image + s_rq_sh3 : VkDescriptorSet //! xb -> the shared up stack's feed image + s_shgate : VkDescriptorSet //! the gate row . xb -> the logit slot past the router's + s_comb_sh : VkDescriptorSet //! x += sum w_j * down_j + sigmoid(gate) * the shared down row } // one recurrent layer's pre-recorded deltanet decode-step chain + its device-resident state. @@ -580,12 +587,14 @@ struct GpuState { span_w_dev : uint64 // the combine weights the top-k writes span_logits_dev : uint64 // the router logits span_router_dev : uint64 // the span's router planes, f32, layer-major + span_shgate_dev : uint64 //! the shared expert's gate rows per span layer, f32 (gated models) span_norms_dev : uint64 // [att | ffn] norm rows per span layer span_xin : HostBuf // the residual row in (host-staged per token, copied at the chain's head) span_out : HostBuf // per-layer raw k/v rows, then the residual row (readback staging) span_cmd : VkCommandBuffer span_key : int64 = -1l // the recorded span's first q plane offset (-1 = none) span_nl : int64 + span_form : int64 //! ... and the routing shape the record depends on (experts, slots, the shared expert's width, its gate) span_layers : array ffn_pend_bytes : int64 qkv_cmds : table // q plane offset -> the layer's decode q/k/v group @@ -1735,6 +1744,7 @@ def vk_drop_model_state { for (i in range(13)) { g_rdq_dn[i] = 0.0lf } + g_rdq_moe |> clear() // host-visible buffers: host_mem tracks every make_host_buf result minus the exempt pair for (buf, mem in keys(g_gpu.host_mem), values(g_gpu.host_mem)) { unsafe { @@ -1763,6 +1773,7 @@ def vk_drop_model_state { g_gpu.tail_xout = null g_gpu.span_key = -1l g_gpu.span_nl = 0l + g_gpu.span_form = 0l g_gpu.dnd_steps |> clear() delete g_gpu.heat_pools g_gpu.stream_max_gu_wq = 0l @@ -2142,7 +2153,26 @@ struct RLayer { pf_dn_ba_set : VkDescriptorSet // f32 arm: dn_ba_dev + pf_xb -> this layer's smalls (batched router GEMV) pf_dn_cvt_set : VkDescriptorSet // the cm2 route: pf_xb -> pf_xf, the qkv / z GEMMs' f16 feed m_pf_dnqkv, m_pf_dnz, m_pf_dnout, m_pf_dnbeta, m_pf_dnalpha : HostBuf // the prefill GEMMs' region metas -} + //! the MoE form: the routed expert triple beside the dense triple (the shared expert's, or none: b1 < 0) + moe : bool + router_slot : int64 = -1l //! this layer's slot in the router plane (the MoE layers in order) + be1, be3, be2 : int64 //! arena blocks of the expert stacks [ne x nfe x dim] / [ne x nfe x dim] / [ne x dim x nfe] + fe1, fe3, fe2 : int + xe_own : bool //! the experts' feed is its own image: their quant form differs from the dense triple's + s_quant_xe : VkDescriptorSet //! the second requant of xb, the experts' form + s_router, s_topk : VkDescriptorSet + s_egate, s_eup : VkDescriptorSet + s_eact : VkDescriptorSet + s_edown : VkDescriptorSet + pf_router_set, pf_topk_set, pf_sched_set, pf_gather_set : VkDescriptorSet //! the window chain's MoE block + pf_e1_set, pf_e3_set, pf_e2_set : VkDescriptorSet //! the expert tiles over the device-written schedules: the s column's + pf_e1m_set, pf_e3m_set, pf_e2m_set : VkDescriptorSet //! ... and the m column's, over the same planes + pf_eact_set : VkDescriptorSet + pf_arcomb_set, pf_arcomb_f16_set : VkDescriptorSet //! the residual step with the combine folded in: the f32 and f16 normed-row forms +} + +//! the layer's dense FFN triple is placed: its own FFN, or an MoE layer's shared expert (an MoE layer without one has none) +def has_dense_triple(L : RLayer) : bool => L.b1 >= 0l struct RDec { ready : bool @@ -2216,6 +2246,7 @@ struct RDec { // is f16 — the fa tiles bind the mirrors directly, no shadow planes) pf_facm2 : bool pf_logits_host : HostBuf + pf_midx_host : HostBuf //! the last MoE layer's picks, read back for the profile's bucket report pf_meta : array // one host meta per (layer, role) + prologue/final/cls pf_sets : array @scratch pf_cls_gemm : table // GEMM-role class sets, keyed idx*16 + variant (lazy insert, steady-state hit) @@ -2236,6 +2267,28 @@ struct RDec { pf_dnrq_set : VkDescriptorSet // the o requant (dn_ws_dev DN_O_OFF rows -> hq/hs), shared by every recurrent layer pf_dnof : uint64 // the o rows' f16 feed for the cm2 out tile [np x di] (the cm2 route; 0 when the device has no cm2) pf_dnof_set : VkDescriptorSet // dn_ws_dev o rows -> pf_dnof, shared by every recurrent layer + //! the MoE form: the router plane and the routed block's device state (rdec_prepare_moe) + moe : bool + ne, k, nfe, nsh, ner : int64 //! experts, routed slots, the expert width, the shared expert's width (0 = none), the router rows per layer (ne, + the gate row) + n_moe : int64 //! MoE layers (the router plane's slots) + sh_gated, norm_topk, is_gelu : bool + wscale : float + router_dev : uint64 //! [n_moe x ner x dim] f32 router rows, the gate row last + router_bytes : int64 + moe_ident_dev : uint64 //! 0..MAX_ROUTED_SLOTS-1: the decode combine's identity slot map + moe_dlog_dev : uint64 //! the decode's logits row (ner floats) + moe_dw_dev : uint64 //! the decode's k routing weights + moe_meta1_dev, moe_meta3_dev, moe_meta2_dev : uint64 //! the decode GEMVs' slot regions, written by the top-k + moe_xq_dev, moe_xs_dev : uint64 //! the experts' own feed image when their form differs from the dense triple's + egate_dev, eup_dev, edown_dev : uint64 //! the decode's k expert rows + pf_nk : int64 //! the window's slot rows (PF_WINDOW x k) + pf_mlog : uint64 //! [np x ner] logits + pf_midx, pf_mw, pf_minv : uint64 //! [nk] picks, weights, slot -> bucket row + pf_msch1, pf_msch3, pf_msch2 : uint64 //! the three expert planes' device-written schedules + pf_mxg : uint64 //! [nk + slack x dim] the gathered f16 rows + pf_my1, pf_my2 : uint64 //! [nk x nfe] the gate / up rows + pf_mhf : uint64 //! [nk + slack x nfe] the hidden rows, f16 + pf_mdn : uint64 //! [nk x dim] the routed down rows } let PF_WINDOW = 512l // prefill activation-buffer rows; longer prompts run as sequential windows @@ -2296,6 +2349,7 @@ var g_rdq_pro = 0.0lf var g_rdq_tail = 0.0lf var g_rdq_role : double[16] var g_rdq_dn : double[13] // a hybrid's recurrent-head roles (rdq_sample's second table) +var g_rdq_moe : table //! an MoE layer's tail stamps by name (rdq_sample's third table) var g_rdq_wall_us = 0.0lf // the host side of the same tokens: memcpys + submit + fence + logits copy let PFQ_CAP = 2048u @@ -2376,6 +2430,7 @@ let VHZ_GATE = 0x800u // ffn gate plane let VHZ_UP = 0x1000u // ffn up plane let VHZ_HQ = 0x2000u // requantized hidden hq+hs let VHZ_FFO = 0x4000u // ffn down out +let VHZ_MRT = 0x8000u //! the MoE block's routing planes: logits, picks, weights, the slot map, the schedules let VHZ_LOG = 0x10000u // logits let VHZ_COS = 0x20000u // rope cos/sin rows @@ -2432,6 +2487,11 @@ let VHZ_DNWS = 0x800000u // the deltanet window chain's scan workspace (dn_ws_d let VHZ_DNBG = 0x1000000u // the prefill's beta/alpha rows scratch (pf_dnbg) let VHZ_DNC = 0x2000000u // the prefill's deltanet conv output rows (pf_dnconv) let VHZ_DNZ = 0x4000000u // the prefill's deltanet z rows (pf_dnz) +let VHZ_MXG = 0x8000000u //! the MoE block's gathered f16 rows (pf_mxg) +let VHZ_MDN = 0x10000000u //! the MoE block's routed down rows (pf_mdn / edown_dev) +let VHZ_GATE_M = 0x20000000u //! the m-column dispatch's rows of the VHZ_GATE plane +let VHZ_UP_M = 0x40000000u //! ... of the VHZ_UP plane +let VHZ_MDN_M = 0x80000000u //! ... of the VHZ_MDN plane struct VkHaz { w : uint // bits with a pending (unbarriered) write diff --git a/modules/dasLLAMA/dasllama/dasllama_vulkan_decode.das b/modules/dasLLAMA/dasllama/dasllama_vulkan_decode.das index a14fd38973..2562f0fb3a 100644 --- a/modules/dasLLAMA/dasllama/dasllama_vulkan_decode.das +++ b/modules/dasLLAMA/dasllama/dasllama_vulkan_decode.das @@ -220,6 +220,17 @@ def vk_rdec_set_layer(l, mir_idx : int64; bq, bk, bv, bo, b1, b3, b2 : int64; fq rd_layer_slot(l) unsafe { var L & = g_rd.layers[l] + rd_attn_sets(L, mir_idx, bq, bk, bv, bo, fq, fk, fv, fo, f1, f3, f2) + rd_ffn_sets(L, b1, b3, b2, f1, f3, f2) + L.made = true + } + g_rd.tok_recorded = false //! the recorded chain references the replaced sets +} + +//! the attention half of a layer's registration: the quad's blocks, formats and sets (f1 / f3 / f2 +//! name the layer's FFN feed forms for the class-rail check) +def private rd_attn_sets(var L : RLayer; mir_idx : int64; bq, bk, bv, bo : int64; fq, fk, fv, fo, f1, f3, f2 : int) { + unsafe { L.recurrent = false L.mir_idx = mir_idx L.bq = bq; L.bk = bk; L.bv = bv; L.bo = bo @@ -231,9 +242,7 @@ def vk_rdec_set_layer(l, mir_idx : int64; bq, bk, bv, bo, b1, b3, b2 : int64; fq let pk = arena_planes(fk, bk) let pv = arena_planes(fv, bv) let po = arena_planes(fo, bo) - // every role rides the class rail: params in push, per-token words in the shared TokMeta; - // kq consumers take the Q8_K quant/act forms (per-256 scales) - let anykq = kq_sb(fq) || kq_sb(fo) || kq_sb(f1) || kq_sb(f2) + let anykq = kq_sb(fq) || kq_sb(fo) || kq_sb(f1) || kq_sb(f2) //! kq consumers take the Q8_K quant/act forms let kv16 = g_rd.kv16 verify(ensure_cls_dn_rq() && ensure_cls_ar() && ensure_cls_ar_rq() && ensure_q8_actrq_cls() && (kv16 ? ensure_rope_kv_f16_cls() : ensure_rope_kv_cls()) @@ -257,9 +266,7 @@ def vk_rdec_set_layer(l, mir_idx : int64; bq, bk, bv, bo, b1, b3, b2 : int64; fq L.s_v = gemv_cls_set(fv, fixed_array(pv.wq, pv.ws, L.m_v.buf, g_rd.xq_dev, g_rd.xs_dev, g_rd.kv_dev), fixed_array(pv.wqb, pv.wsb, META_BYTES, dim, (dim / 32l) * 4l, (qd + 2l * kvd) * 4l), fixed_array(0u, 0u, 0u, VHZ_XQ, VHZ_XQ, VHZ_KVV)) - // same-format k/v in the SAME slab share arena planes — one two-region dispatch fills both - // kv halves (a slab boundary between them falls back to the split sets) - L.kv_merged = fk == fv && arena_slab_idx(bk) == arena_slab_idx(bv) + L.kv_merged = fk == fv && arena_slab_idx(bk) == arena_slab_idx(bv) //! same-format k/v in one slab: one two-region dispatch fills both halves if (L.kv_merged) { L.s_kv = gemv_cls_set(fk, fixed_array(pk.wq, pk.ws, L.m_k.buf, g_rd.xq_dev, g_rd.xs_dev, g_rd.kv_dev), fixed_array(pk.wqb, pk.wsb, META_BYTES, dim, (dim / 32l) * 4l, (qd + 2l * kvd) * 4l), @@ -297,10 +304,169 @@ def vk_rdec_set_layer(l, mir_idx : int64; bq, bk, bv, bo, b1, b3, b2 : int64; fq L.s_wo = gemv_cls_set(fo, fixed_array(po.wq, po.ws, L.m_wo.buf, g_rd.aq_dev, g_rd.as_dev, g_rd.xb2_dev), fixed_array(po.wqb, po.wsb, META_BYTES, qd, (qd / 32l) * 4l, dim * 4l), fixed_array(0u, 0u, 0u, VHZ_AQ, VHZ_AQ, VHZ_XB2)) - rd_ffn_sets(L, b1, b3, b2, f1, f3, f2) + } +} + +// ===== the MoE form: the router plane, the routed block of the token command ===== + +let private MOE_MAX_EXPERTS = 256l //! the select kernels' one-workgroup reach: one thread per expert + +[arch(at="../ARCHITECTURE_GPU_VULKAN_MOE.md#resident-moe-token")] +def vk_rdec_prepare_moe(ne, k, nfe, nsh, n_moe : int64; sh_gated, norm_topk, is_gelu : bool; wscale : float; router : array) : bool { + let ner = ne + (nsh > 0l && sh_gated ? 1l : 0l) + let reach = ne <= MOE_MAX_EXPERTS && k <= MAX_ROUTED_SLOTS && nfe % 32l == 0l + if (g_gpu != null && g_gpu.dry) { + return reach //! the reach IS the plan decision; everything below is device state + } + assert(g_rd != null, "vk_rdec_prepare_moe before prepare") + let dim = g_rd.dim + if (!reach || k * max(nfe, dim) * 4l > Y_BYTES || dim % 64l != 0l) { //! the router tile steps K by 64 + to_log(LOG_WARNING, "dasLLAMA vulkan resident: the MoE geometry ({ne} experts, {k} routed, expert width {nfe}, dim {dim}) is outside the routing kernels' reach\n") + return false + } + if (n_moe * ner * dim * 4l > vk_max_storage_range()) { //! the router plane binds as one range + to_log(LOG_WARNING, "dasLLAMA vulkan resident: the router plane ({n_moe} layers x {ner} rows x {dim}) passes the device's storage range {vk_max_storage_range()}\n") + return false + } + var r & = unsafe(*g_rd) + r.moe = true + r.ne = ne + r.k = k + r.nfe = nfe + r.nsh = nsh + r.n_moe = n_moe + r.ner = ner + r.sh_gated = sh_gated + r.norm_topk = norm_topk + r.is_gelu = is_gelu + r.wscale = wscale + if (!(ensure_router_gemv_cls() && ensure_topk_cls() && ensure_cls_ar_comb() + && ensure_q8_actrq_cls() && ensure_q8k_actrq_cls())) { + to_log(LOG_WARNING, "dasLLAMA vulkan resident: the routing class rails declined on this device\n") + return false + } + r.router_bytes = long_length(router) * 4l + assert(r.router_bytes == n_moe * r.ner * dim * 4l, "vk_rdec_prepare_moe: the router plane is not [n_moe x ner x dim]") + r.router_dev = make_device_buf(r.router_bytes) + upload_region_at(r.router_dev, 0l, unsafe(addr(router[0])), r.router_bytes) + r.moe_ident_dev = make_device_buf(MAX_ROUTED_SLOTS * 4l) + unsafe { + var mp = reinterpret(g_gpu.staging.mapped) + for (i in range64(MAX_ROUTED_SLOTS)) { + mp[i] = uint(i) + } + staged_upload(r.moe_ident_dev, 0l, MAX_ROUTED_SLOTS * 4l) + } + r.moe_dlog_dev = make_device_buf(r.ner * 4l) + r.moe_dw_dev = make_device_buf(MAX_ROUTED_SLOTS * 4l) + r.moe_meta1_dev = make_device_buf(META_BYTES) + r.moe_meta3_dev = make_device_buf(META_BYTES) + r.moe_meta2_dev = make_device_buf(META_BYTES) + r.moe_xq_dev = make_device_buf(dim) + r.moe_xs_dev = make_device_buf((dim / 32l) * 4l) + r.egate_dev = make_device_buf(k * nfe * 4l) + r.eup_dev = make_device_buf(k * nfe * 4l) + r.edown_dev = make_device_buf(k * dim * 4l) + return true +} + +[arch(at="../ARCHITECTURE_GPU_VULKAN_MOE.md#resident-moe-token")] +def private rd_moe_sets(var L : RLayer) { + let dim = g_rd.dim + let nfe = g_rd.nfe + let k = g_rd.k + let pe1 = arena_planes(L.fe1, L.be1) + let pe3 = arena_planes(L.fe3, L.be3) + let pe2 = arena_planes(L.fe2, L.be2) + verify(gemv_cls_ensure(L.fe1) && gemv_cls_ensure(L.fe3) && gemv_cls_ensure(L.fe2) + && (kq_sb(L.fe1) ? ensure_cls_q8k_rq() : ensure_cls_dn_rq()) + && (kq_sb(L.fe2) ? ensure_q8k_actrq_cls() : ensure_q8_actrq_cls()), + "vk decode: the expert class rails must engage on a live device") + L.xe_own = has_dense_triple(L) && kq_sb(L.fe1) != kq_sb(L.f1) + let xqb = L.xe_own ? g_rd.moe_xq_dev : g_rd.xq_dev + let xsb = L.xe_own ? g_rd.moe_xs_dev : g_rd.xs_dev + if (L.xe_own) { + L.s_quant_xe = set_rq_cls(fixed_array(g_rd.xb_dev, g_rd.moe_xq_dev, g_rd.moe_xs_dev), + fixed_array(dim * 4l, dim, (dim / 32l) * 4l), fixed_array(VHZ_XB, VHZ_XQ, VHZ_XQ)) + } + L.s_router = set_router_gemv_cls(fixed_array(g_rd.router_dev, g_rd.xb_dev, g_rd.moe_dlog_dev), + fixed_array(g_rd.router_bytes, dim * 4l, g_rd.ner * 4l), fixed_array(0u, VHZ_XB, VHZ_MRT)) + L.s_topk = set_topk_cls(fixed_array(g_rd.moe_dlog_dev, g_rd.moe_meta1_dev, g_rd.moe_meta3_dev, g_rd.moe_meta2_dev, g_rd.moe_dw_dev), + fixed_array(g_rd.ner * 4l, META_BYTES, META_BYTES, META_BYTES, MAX_ROUTED_SLOTS * 4l), + fixed_array(VHZ_MRT, VHZ_MRT, VHZ_MRT, VHZ_MRT, VHZ_MRT)) + L.s_egate = gemv_cls_set(L.fe1, fixed_array(pe1.wq, pe1.ws, g_rd.moe_meta1_dev, xqb, xsb, g_rd.egate_dev), + fixed_array(pe1.wqb, pe1.wsb, META_BYTES, dim, (dim / 32l) * 4l, k * nfe * 4l), + fixed_array(0u, 0u, VHZ_MRT, VHZ_XQ, VHZ_XQ, VHZ_GATE)) + L.s_eup = gemv_cls_set(L.fe3, fixed_array(pe3.wq, pe3.ws, g_rd.moe_meta3_dev, xqb, xsb, g_rd.eup_dev), + fixed_array(pe3.wqb, pe3.wsb, META_BYTES, dim, (dim / 32l) * 4l, k * nfe * 4l), + fixed_array(0u, 0u, VHZ_MRT, VHZ_XQ, VHZ_XQ, VHZ_UP)) + L.s_eact = set_actrq_cls(fixed_array(g_rd.egate_dev, g_rd.eup_dev, g_gpu.hq_dev, g_gpu.hs_dev), + fixed_array(k * nfe * 4l, k * nfe * 4l, BATCH_HQ_BYTES, BATCH_HS_BYTES), + fixed_array(VHZ_GATE, VHZ_UP, VHZ_HQ, VHZ_HQ)) + L.s_edown = gemv_cls_set(L.fe2, fixed_array(pe2.wq, pe2.ws, g_rd.moe_meta2_dev, g_gpu.hq_dev, g_gpu.hs_dev, g_rd.edown_dev), + fixed_array(pe2.wqb, pe2.wsb, META_BYTES, BATCH_HQ_BYTES, BATCH_HS_BYTES, k * dim * 4l), + fixed_array(0u, 0u, VHZ_MRT, VHZ_HQ, VHZ_HQ, VHZ_MDN)) + verify(ensure_cls_ar_comb(), "vk decode: the combine-folded residual class rail must engage on a live device") + L.s_addr_next = set_cls_ar_comb(fixed_array(g_rd.x_dev, g_rd.ffnout_dev, g_rd.norms_dev, g_rd.xb_dev, + g_rd.moe_ident_dev, g_rd.moe_dw_dev, g_rd.edown_dev, g_rd.moe_dlog_dev), + fixed_array(dim * 4l, dim * 4l, g_rd.norms_bytes, dim * 4l, MAX_ROUTED_SLOTS * 4l, MAX_ROUTED_SLOTS * 4l, k * dim * 4l, g_rd.ner * 4l), + fixed_array(VHZ_X, VHZ_FFO, 0u, VHZ_XB, 0u, VHZ_MRT, VHZ_MDN, VHZ_MRT)) +} + +//! a layer without a dense triple (an MoE layer with no shared expert): the tail sets the token +//! command encodes around the routed block - the FFN norm, its requant and the residual step +def private rd_tail_sets_bare(var L : RLayer; f1, f3, f2 : int) { + let dim = g_rd.dim + L.b1 = -1l; L.b3 = -1l; L.b2 = -1l + L.f1 = f1; L.f3 = f3; L.f2 = f2 + L.s_addr_ffn = set_cls_ar(fixed_array(g_rd.x_dev, g_rd.xb2_dev, g_rd.norms_dev, g_rd.xb_dev), + fixed_array(dim * 4l, dim * 4l, g_rd.norms_bytes, dim * 4l), + fixed_array(VHZ_X, VHZ_XB2, 0u, VHZ_XB)) + L.s_quant_xb2 = set_rq_cls(fixed_array(g_rd.xb_dev, g_rd.xq_dev, g_rd.xs_dev), + fixed_array(dim * 4l, dim, (dim / 32l) * 4l), + fixed_array(VHZ_XB, VHZ_XQ, VHZ_XQ)) + L.s_addr_next = set_cls_ar(fixed_array(g_rd.x_dev, g_rd.ffnout_dev, g_rd.norms_dev, g_rd.xb_dev), + fixed_array(dim * 4l, dim * 4l, g_rd.norms_bytes, dim * 4l), + fixed_array(VHZ_X, VHZ_FFO, 0u, VHZ_XB)) +} + +[arch(at="../ARCHITECTURE_GPU_VULKAN_MOE.md#resident-moe-token")] +def vk_rdec_set_moe_experts(l, router_slot, be1, be3, be2 : int64; fe1, fe3, fe2 : int) { + if (g_gpu != null && g_gpu.dry) { + return + } + assert(g_rd != null && g_rd.moe, "vk_rdec_set_moe_experts before prepare_moe") + unsafe { + var L & = g_rd.layers[l] + assert(L.made, "vk_rdec_set_moe_experts before the layer's own registration") + L.moe = true + L.router_slot = router_slot + L.be1 = be1; L.be3 = be3; L.be2 = be2 + L.fe1 = fe1; L.fe3 = fe3; L.fe2 = fe2 + rd_moe_sets(L) + } + g_rd.tok_recorded = false +} + +//! the dense triple is the shared expert's; b1 < 0 = the layer has none. +[arch(at="../ARCHITECTURE_GPU_VULKAN_MOE.md#resident-moe-token")] +def vk_rdec_set_moe_layer(l, mir_idx, router_slot, bq, bk, bv, bo, b1, b3, b2, be1, be3, be2 : int64; fq, fk, fv, fo, f1, f3, f2, fe1, fe3, fe2 : int) { + if (g_gpu != null && g_gpu.dry) { + return + } + assert(g_rd != null && g_rd.moe, "vk_rdec_set_moe_layer before prepare_moe") + rd_layer_slot(l) + unsafe { + var L & = g_rd.layers[l] + rd_attn_sets(L, mir_idx, bq, bk, bv, bo, fq, fk, fv, fo, f1, f3, f2) + if (b1 >= 0l) { + rd_ffn_sets(L, b1, b3, b2, f1, f3, f2) + } else { + rd_tail_sets_bare(L, f1, f3, f2) + } L.made = true } - g_rd.tok_recorded = false // the recorded chain references the replaced sets + vk_rdec_set_moe_experts(l, router_slot, be1, be3, be2, fe1, fe3, fe2) } //! one recurrent layer's constants (conv taps, out-norm weights, a, dt) into its smalls @@ -424,7 +590,11 @@ def vk_rdec_set_dn_layer(l, bqkv, bz, bout, bbeta, balpha, b1, b3, b2, ba_off : L.s_dnout = gemv_cls_set(fout, fixed_array(pout.wq, pout.ws, L.m_dnout.buf, g_rd.dn_oq_dev, g_rd.dn_os_dev, g_rd.xb2_dev), fixed_array(pout.wqb, pout.wsb, META_BYTES, di, (di / 32l) * 4l, dim * 4l), fixed_array(0u, 0u, 0u, VHZ_DNO, VHZ_DNO, VHZ_XB2)) - rd_ffn_sets(L, b1, b3, b2, f1, f3, f2) + if (b1 >= 0l) { + rd_ffn_sets(L, b1, b3, b2, f1, f3, f2) + } else { + rd_tail_sets_bare(L, f1, f3, f2) //! a recurrent MoE layer without a shared expert + } L.made = true } g_rd.tok_recorded = false @@ -549,9 +719,11 @@ def private rd_record_token { } fill_gemv_regions(L.m_wo, L.bo) } - fill_gemv_regions(L.m_gate, L.b1) - fill_gemv_regions(L.m_up, L.b3) - fill_gemv_regions(L.m_down, L.b2) + if (has_dense_triple(L)) { + fill_gemv_regions(L.m_gate, L.b1) + fill_gemv_regions(L.m_up, L.b3) + fill_gemv_regions(L.m_down, L.b2) + } } rd_encode_token() } @@ -560,7 +732,7 @@ def private rd_record_token { // gate/up groups, plus the classifier) reads a Q8_0-form image, i.e. the fuse may serve them all. def private rd_x_quants_b32 : bool { for (L in g_rd.layers) { - if (kq_sb(L.fq) || kq_sb(L.f1)) { + if (kq_sb(L.fq) || kq_sb(L.f1) || (L.moe && kq_sb(L.fe1))) { return false } } @@ -568,7 +740,7 @@ def private rd_x_quants_b32 : bool { } [arch(at="../ARCHITECTURE_GPU_VULKAN_DECODE.md#hybrid-token-command")] -def private rd_encode_attn_head(raw : VkCommandBuffer; var h : VkHaz; var L : RLayer; l : int64; fuse_qkr : bool; rpw : int64; +def private rd_encode_attn_head(raw : VkCommandBuffer; var h : VkHaz; var L : RLayer; l : int64; fuse_qkr : bool; var pc_rq_qd : RqArgs; var pc_rq_qd_k : RqArgs; rqwg_qd, rqwg_qd_k, nlfin : int64) { let dim = g_rd.dim let qd = g_rd.qd @@ -576,18 +748,18 @@ def private rd_encode_attn_head(raw : VkCommandBuffer; var h : VkHaz; var L : RL let qrows = g_rd.q_gated ? 2l * qd : qd let qstride = uint(g_rd.q_gated ? 2l * g_rd.head_size : g_rd.head_size) var pcq = GemvArgs(n = uint(dim), d = uint(qrows), nreg = 1u, ybase = 0u) - gemv_cls_enc(L.fq, raw, h, L.s_q, pcq, (qrows + rpw - 1l) / rpw) + gemv_enc(L.fq, raw, h, L.s_q, pcq, qrows) pfq_ts(raw) if (L.kv_merged) { var pckv = GemvArgs(n = uint(dim), d = uint(kvd), nreg = 2u, ybase = uint(qd)) - gemv_cls_enc(L.fk, raw, h, L.s_kv, pckv, (2l * kvd + rpw - 1l) / rpw) + gemv_enc(L.fk, raw, h, L.s_kv, pckv, 2l * kvd) pfq_ts(raw) } else { var pck = GemvArgs(n = uint(dim), d = uint(kvd), nreg = 1u, ybase = uint(qd)) - gemv_cls_enc(L.fk, raw, h, L.s_k, pck, (kvd + rpw - 1l) / rpw) + gemv_enc(L.fk, raw, h, L.s_k, pck, kvd) pfq_ts(raw) var pcv = GemvArgs(n = uint(dim), d = uint(kvd), nreg = 1u, ybase = uint(qd + kvd)) - gemv_cls_enc(L.fv, raw, h, L.s_v, pcv, (kvd + rpw - 1l) / rpw) + gemv_enc(L.fv, raw, h, L.s_v, pcv, kvd) pfq_ts(raw) } let hs = g_rd.head_size @@ -637,21 +809,21 @@ def private rd_encode_attn_head(raw : VkCommandBuffer; var h : VkHaz; var L : RL } pfq_ts(raw) var pcw = GemvArgs(n = uint(qd), d = uint(dim), nreg = 1u, ybase = 0u) - gemv_cls_enc(L.fo, raw, h, L.s_wo, pcw, (dim + rpw - 1l) / rpw) + gemv_enc(L.fo, raw, h, L.s_wo, pcw, dim) pfq_ts(raw) } [arch(at="../ARCHITECTURE_GPU_VULKAN_DECODE.md#hybrid-token-command")] -def private rd_encode_dn_head(raw : VkCommandBuffer; var h : VkHaz; var L : RLayer; rpw : int64) { +def private rd_encode_dn_head(raw : VkCommandBuffer; var h : VkHaz; var L : RLayer) { let dim = g_rd.dim let cd = g_rd.dn_cd let di = g_rd.dn_di let nvh = g_rd.dn_nvh var pcqkv = GemvArgs(n = uint(dim), d = uint(cd), nreg = 1u, ybase = 0u) - gemv_cls_enc(L.fqkv, raw, h, L.s_dnqkv, pcqkv, (cd + rpw - 1l) / rpw) + gemv_enc(L.fqkv, raw, h, L.s_dnqkv, pcqkv, cd) pfq_ts(raw) var pcz = GemvArgs(n = uint(dim), d = uint(di), nreg = 1u, ybase = uint(cd)) - gemv_cls_enc(L.fz, raw, h, L.s_dnz, pcz, (di + rpw - 1l) / rpw) + gemv_enc(L.fz, raw, h, L.s_dnz, pcz, di) pfq_ts(raw) if (g_rd.dn_ba_f32) { var pcb = RouterArgs(ne = uint(nvh), dim = uint(dim), woff = uint(L.ba_off), obase = uint(DN_SM_BETA)) @@ -664,9 +836,9 @@ def private rd_encode_dn_head(raw : VkCommandBuffer; var h : VkHaz; var L : RLay enc_cls_dn_rq(raw, h, L.s_quant_xb, pcr, (dim / 4l + 255l) / 256l) } var pcb = GemvArgs(n = uint(dim), d = uint(nvh), nreg = 1u, ybase = uint(DN_SM_BETA)) - gemv_cls_enc(int(KqFmt.q8), raw, h, L.s_dnbeta, pcb, (nvh + rpw - 1l) / rpw) + gemv_enc(int(KqFmt.q8), raw, h, L.s_dnbeta, pcb, nvh) var pcg = GemvArgs(n = uint(dim), d = uint(nvh), nreg = 1u, ybase = uint(DN_SM_G)) - gemv_cls_enc(int(KqFmt.q8), raw, h, L.s_dnalpha, pcg, (nvh + rpw - 1l) / rpw) + gemv_enc(int(KqFmt.q8), raw, h, L.s_dnalpha, pcg, nvh) } pfq_ts(raw) var pcd = DnStepArgs(cd = uint(cd), kd = uint(g_rd.dn_kd), nkh = uint(g_rd.dn_nkh), ds = uint(g_rd.dn_ds), @@ -681,16 +853,104 @@ def private rd_encode_dn_head(raw : VkCommandBuffer; var h : VkHaz; var L : RLay enc_cls_q8k_rq(raw, h, L.s_quant_dno, pcr, (di / 8l + 255l) / 256l) } var pco = GemvArgs(n = uint(di), d = uint(dim), nreg = 1u, ybase = 0u) - gemv_cls_enc(L.fout, raw, h, L.s_dnout, pco, (dim + rpw - 1l) / rpw) + gemv_enc(L.fout, raw, h, L.s_dnout, pco, dim) pfq_ts(raw) } -//! The token command's fuse gates, one source for the recorder and the profiler's stamp count: `fuse` = the add+rms+requant twins (b32 quants, no hybrid), `fuse_qkr` = the fused qk-norm+rope pair (also the gated and partial-rope arms, which the split pair lacks) +//! The token command's fuse gates, one source for the recorder and the profiler's stamp count: `fuse` = the +//! add+rms+requant twins (b32 quants, no hybrid, no MoE - the router reads the normed row the twins never +//! store), `fuse_qkr` = the fused qk-norm+rope pair (also the gated and partial-rope arms the split pair lacks) def private rd_fuse_gates : tuple { - let fuse = vk_fuse_on() && rd_x_quants_b32() && !g_rd.hybrid + let fuse = vk_fuse_on() && rd_x_quants_b32() && !g_rd.hybrid && !g_rd.moe return (fuse = fuse, fuse_qkr = g_rd.qk_norm && (fuse || g_rd.q_gated || g_rd.rot != g_rd.head_size)) } +def private rd_encode_ffn(raw : VkCommandBuffer; var h : VkHaz; var L : RLayer; hid : int64; var pc_act : ActArgs; var pc_act_k : ActArgs) { + let dim = g_rd.dim + var pcg = GemvArgs(n = uint(dim), d = uint(hid), nreg = 1u, ybase = 0u) + gemv_enc(L.f1, raw, h, L.s_gate, pcg, hid) + pfq_ts(raw) + gemv_enc(L.f3, raw, h, L.s_up, pcg, hid) + pfq_ts(raw) + if (kq_sb(L.f2)) { + enc_q8k_actrq_cls(raw, h, L.s_actrq, pc_act_k, (hid / 8l + 255l) / 256l) + } else { + enc_q8_actrq_cls(raw, h, L.s_actrq, pc_act, (hid / 4l + 255l) / 256l) + } + pfq_ts(raw) + var pcd = GemvArgs(n = uint(hid), d = uint(dim), nreg = 1u, ybase = 0u) + gemv_enc(L.f2, raw, h, L.s_down, pcd, dim) + pfq_ts(raw) +} + +[arch(at="../ARCHITECTURE_GPU_VULKAN_MOE.md#resident-moe-token")] +def private rd_encode_moe(raw : VkCommandBuffer; var h : VkHaz; var L : RLayer; + var pc_rq_dim : RqArgs; var pc_rq_dim_k : RqArgs; rqwg_dim, rqwg_dim_k : int64) { + let dim = g_rd.dim + let nfe = g_rd.nfe + let k = g_rd.k + if (L.xe_own) { + if (kq_sb(L.fe1)) { + enc_cls_q8k_rq(raw, h, L.s_quant_xe, pc_rq_dim_k, rqwg_dim_k) + } else { + enc_cls_dn_rq(raw, h, L.s_quant_xe, pc_rq_dim, rqwg_dim) + } + pfq_ts(raw) + } + var pcr = RouterArgs(ne = uint(g_rd.ner), dim = uint(dim), woff = uint(L.router_slot * g_rd.ner * dim), obase = 0u) + enc_router_gemv_cls(raw, h, L.s_router, pcr, g_rd.ner) + pfq_ts(raw) + var pct = TopkArgs(ne = uint(g_rd.ne), k = uint(k), norm = g_rd.norm_topk ? 1u : 0u, wscale = g_rd.wscale, + base1 = uint(arena_local_blk(L.be1)), stride1 = uint(nfe * dim / fmt_unit(L.fe1)), + base3 = uint(arena_local_blk(L.be3)), stride3 = uint(nfe * dim / fmt_unit(L.fe3)), + base2 = uint(arena_local_blk(L.be2)), stride2 = uint(nfe * dim / fmt_unit(L.fe2)), + xnb2 = uint(nfe / fmt_unit(L.fe2))) + enc_topk_cls(raw, h, L.s_topk, pct, 1l) + pfq_ts(raw) + var pcg = GemvArgs(n = uint(dim), d = uint(nfe), nreg = uint(k), ybase = 0u) + gemv_enc(L.fe1, raw, h, L.s_egate, pcg, k * nfe) + pfq_ts(raw) + gemv_enc(L.fe3, raw, h, L.s_eup, pcg, k * nfe) + pfq_ts(raw) + let dkq = kq_sb(L.fe2) + var pca = ActArgs(nelem = uint(k * nfe), gelu = g_rd.is_gelu ? 1u : 0u, nblk = uint(k * nfe / (dkq ? 256l : 32l))) + let awgs = (k * nfe / (dkq ? 8l : 4l) + 255l) / 256l + if (dkq) { + enc_q8k_actrq_cls(raw, h, L.s_eact, pca, awgs) + } else { + enc_q8_actrq_cls(raw, h, L.s_eact, pca, awgs) + } + pfq_ts(raw) + var pcd = GemvArgs(n = uint(nfe), d = uint(dim), nreg = uint(k), ybase = 0u) + gemv_enc(L.fe2, raw, h, L.s_edown, pcd, k * dim) + pfq_ts(raw) +} + +//! The residual step's push block over a MoE layer's combine: the shared row at its gate when the layer has +//! one, the k routed rows through the slot map, then the norm at `woff`. +def rd_ar_comb_args(L : RLayer; woff : int64) : ArArgs { + return ArArgs(dim = uint(g_rd.dim), add_on = has_dense_triple(L) ? 1u : 0u, woff = uint(woff), eps = g_rd.eps, ascale = 1.0, + slots = uint(g_rd.k), lstride = uint(g_rd.ner), gated = g_rd.sh_gated ? 1u : 0u) +} + +//! the stamps of an MoE layer's tail after the attention head, in the order rd_encode_token records them +def private rd_moe_tail_names(L : RLayer) : array { + var names <- ["ar1", "rq_f"] + if (has_dense_triple(L)) { + names |> push("sh_gate") + names |> push("sh_up") + names |> push("sh_act") + names |> push("sh_down") + } + if (L.xe_own) { + names |> push("rq_e") + } + var routed <- ["router", "topk", "e_gate", "e_up", "e_act", "e_down", "ar2"] //! ar2 carries the combine + names |> push_from(routed) + delete routed + return <- names +} + // The encode half alone: reset + re-encode the token cmd over the already-filled metas/sets. // The --rerecord-ab probe calls this per token to price the class twins' push-borne pos/cnt. [cold_path] // recording leg (also the rerecord A/B probe) — not the steady-state token path @@ -713,7 +973,6 @@ def private rd_encode_token { vhz_dep(raw, h, 0u, VHZ_COS, true) cmd_copy_whole(raw, g_rd.cos_host.buf, g_rd.cos_dev, g_rd.head_size * 4l) let (fuse, fuse_qkr) = rd_fuse_gates() - let rpw = g_gpu.rows_per_wg var pc_rq_dim = RqArgs(inbase = 0u, nblk = uint(dim / 32l)) var pc_rq_qd = RqArgs(inbase = 0u, nblk = uint(qd / 32l)) var pc_rq_dim_k = RqArgs(inbase = 0u, nblk = uint(dim / 256l)) @@ -742,9 +1001,9 @@ def private rd_encode_token { pfq_ts(raw) } if (L.recurrent) { - rd_encode_dn_head(raw, h, L, rpw) + rd_encode_dn_head(raw, h, L) } else { - rd_encode_attn_head(raw, h, L, l, fuse_qkr, rpw, pc_rq_qd, pc_rq_qd_k, rqwg_qd, rqwg_qd_k, nlfin) + rd_encode_attn_head(raw, h, L, l, fuse_qkr, pc_rq_qd, pc_rq_qd_k, rqwg_qd, rqwg_qd_k, nlfin) } var pc8 = ArArgs(dim = uint(dim), add_on = 1u, woff = uint((l * 2l + 1l) * dim), eps = g_rd.eps, ascale = 1.0) if (fuse) { @@ -760,23 +1019,18 @@ def private rd_encode_token { } pfq_ts(raw) } - var pcg = GemvArgs(n = uint(dim), d = uint(hid), nreg = 1u, ybase = 0u) - gemv_cls_enc(L.f1, raw, h, L.s_gate, pcg, (hid + rpw - 1l) / rpw) - pfq_ts(raw) - gemv_cls_enc(L.f3, raw, h, L.s_up, pcg, (hid + rpw - 1l) / rpw) - pfq_ts(raw) - if (kq_sb(L.f2)) { - enc_q8k_actrq_cls(raw, h, L.s_actrq, pc_act_k, (hid / 8l + 255l) / 256l) - } else { - enc_q8_actrq_cls(raw, h, L.s_actrq, pc_act, (hid / 4l + 255l) / 256l) + if (has_dense_triple(L)) { + rd_encode_ffn(raw, h, L, hid, pc_act, pc_act_k) } - pfq_ts(raw) - var pcd = GemvArgs(n = uint(hid), d = uint(dim), nreg = 1u, ybase = 0u) - gemv_cls_enc(L.f2, raw, h, L.s_down, pcd, (dim + rpw - 1l) / rpw) - pfq_ts(raw) - let nxt = l + 1l < g_rd.n_layers ? (l + 1l) * 2l * dim : nlfin // nolint:LINT021 — rms_att[l+1]|final, composed from int64 layout offsets + if (L.moe) { + rd_encode_moe(raw, h, L, pc_rq_dim, pc_rq_dim_k, rqwg_dim, rqwg_dim_k) + } + let nxt = l + 1l < g_rd.n_layers ? (l + 1l) * 2l * dim : nlfin //! rms_att[l+1] | final var pc14 = ArArgs(dim = uint(dim), add_on = 1u, woff = uint(nxt), eps = g_rd.eps, ascale = 1.0) - if (fuse) { + if (L.moe) { //! a MoE never takes the fused arm: the router reads xb + var pc14c = rd_ar_comb_args(L, nxt) + enc_cls_ar_comb(raw, h, L.s_addr_next, pc14c, 1l) + } elif (fuse) { enc_cls_ar_rq(raw, h, L.s_addr_next_rq, pc14, 1l) } else { enc_cls_ar(raw, h, L.s_addr_next, pc14, 1l) @@ -792,7 +1046,7 @@ def private rd_encode_token { pfq_ts(raw) } var pcc = GemvArgs(n = uint(dim), d = uint(g_rd.vocab), nreg = 1u, ybase = 0u) - gemv_cls_enc(g_rd.cls_fmt_marker, raw, h, g_rd.cls_set, pcc, (g_rd.vocab + rpw - 1l) / rpw) + gemv_enc(g_rd.cls_fmt_marker, raw, h, g_rd.cls_set, pcc, g_rd.vocab) pfq_ts(raw) vhz_dep(raw, h, VHZ_LOG, 0u, true) // compute -> transfer for the logits DMA cmd_copy_whole(raw, g_rd.logits_dev, g_rd.logits_host.buf, g_rd.vocab * 4l) @@ -815,6 +1069,7 @@ def private rdq_role_names(fuse, fuse_qkr, qk_norm : bool) : array { // a recurrent layer's stamps on the token command (rd_encode_dn_head + the shared tail), split rail only let private RDQ_DN_NAMES <- ["rq_x", "qkv", "z", "ba", "step", "out", "ar1", "rq_f", "gate", "up", "actrq", "down", "ar2"] let private RDQ_DN_ROLES = length(RDQ_DN_NAMES) +let private RDQ_DN_HEAD = 6 //! the deltanet head's stamps before the tail (rq_x .. out); a recurrent MoE layer's tail lands by name [cold_path] // DASLLAMA_GPU_PROF reporting leg def private rdq_sample { @@ -822,8 +1077,17 @@ def private rdq_sample { var want = fuse ? 3 : 4 // anchor + prologue + [fin_rq — split rail only] + cls // per ATTENTION layer counting the split v stamp: fused 13, split rail with the fused qk-norm+rope 15, plain 16 / 15 let att_rpl = fuse ? 13 : (fuse_qkr ? 15 : (g_rd.qk_norm ? 16 : 15)) + let dense_tail = fuse ? 6 : 7 //! ar1, rq_f, gate, up, actrq, down, ar2 (the fused rail folds rq_f into ar1) for (L in g_rd.layers) { - want += L.recurrent ? RDQ_DN_ROLES : att_rpl - (L.kv_merged ? 1 : 0) + if (L.moe) { + var names <- rd_moe_tail_names(L) + want += (L.recurrent ? RDQ_DN_HEAD : att_rpl - (L.kv_merged ? 1 : 0) - dense_tail) + length(names) + delete names + } elif (L.recurrent) { + want += RDQ_DN_ROLES + } else { + want += att_rpl - (L.kv_merged ? 1 : 0) + } } if (g_rdq_n != uint(want)) { // an unexpected stamp count: the whole command's GPU span reads off the first and last stamps @@ -854,24 +1118,35 @@ def private rdq_sample { let kv_at = fuse ? 1 : 2 var qi = 2u var ndn = 0l + var nmoe = 0l for (L in g_rd.layers) { if (L.recurrent) { - for (s in range(RDQ_DN_ROLES)) { + for (s in range(L.moe ? RDQ_DN_HEAD : RDQ_DN_ROLES)) { g_rdq_dn[s] += pfq_us(qi) qi++ } ndn++ - continue + } else { + var s = 0 + let head = L.moe ? nroles - dense_tail : nroles //! an MoE layer's tail stamps land by name below + while (s < head) { + g_rdq_role[s] += pfq_us(qi) + qi++ + if (s == kv_at && !L.kv_merged) { + g_rdq_role[s] += pfq_us(qi) // the split v stamp folds into the kv slot + qi++ + } + s++ + } } - var s = 0 - while (s < nroles) { - g_rdq_role[s] += pfq_us(qi) - qi++ - if (s == kv_at && !L.kv_merged) { - g_rdq_role[s] += pfq_us(qi) // the split v stamp folds into the kv slot + if (L.moe) { + var names <- rd_moe_tail_names(L) + for (nm in names) { + g_rdq_moe[nm] = (g_rdq_moe?[nm] ?? 0.0lf) + pfq_us(qi) qi++ } - s++ + delete names + nmoe++ } } // split rail: fin_rq + cls; fused: cls alone (the last addr_next already did the final norm) @@ -891,7 +1166,15 @@ def private rdq_sample { } delete names let natt = long_length(g_rd.layers) - ndn - to_log(LOG_INFO, "vk_rdec gpu avg/token over {g_rdq_count}: prologue {int(g_rdq_pro / n)} {rs}tail {int(g_rdq_tail / n)} total {int(g_rdq_total / n)} (attention layers {natt}, recurrent {ndn})\n") + to_log(LOG_INFO, "vk_rdec gpu avg/token over {g_rdq_count}: prologue {int(g_rdq_pro / n)} {rs}tail {int(g_rdq_tail / n)} total {int(g_rdq_total / n)} (attention layers {natt}, recurrent {ndn}, MoE {nmoe})\n") + if (nmoe > 0l) { + let ms = build_string() $(w) { + for (nm, us in keys(g_rdq_moe), values(g_rdq_moe)) { + w |> write("{nm} {int(us / n)} ") + } + } + to_log(LOG_INFO, "vk_rdec moe avg/token (the MoE layers' tails, summed over the layers): {ms}\n") + } if (ndn > 0l) { let ds = build_string() $(w) { for (s in range(RDQ_DN_ROLES)) { @@ -1126,11 +1409,10 @@ def private dnd_step_make(s_qkv, s_z, s_o : int; woq, woz, woo : int64; vhz_dep(raw, h, 0u, VHG_SM, true) // one declaration for the smalls copy group cmd_copy_range(raw, g_gpu.dnd_bg.buf, 0l, st.smalls_dev, DN_SM_BETA * 4l, nvh * 4l) cmd_copy_range(raw, g_gpu.dnd_bg.buf, nvh * 4l, st.smalls_dev, DN_SM_G * 4l, nvh * 4l) - let rpw = g_gpu.rows_per_wg var pcqkv = GemvArgs(n = uint(dim), d = uint(cd), nreg = 1u, ybase = 0u) - gemv_cls_enc(fqkv, raw, h, qset, pcqkv, (cd + rpw - 1l) / rpw) + gemv_enc(fqkv, raw, h, qset, pcqkv, cd) var pcz = GemvArgs(n = uint(dim), d = uint(di), nreg = 1u, ybase = uint(cd)) - gemv_cls_enc(fz, raw, h, st.z_set, pcz, (di + rpw - 1l) / rpw) + gemv_enc(fz, raw, h, st.z_set, pcz, di) var pcd = DnStepArgs(cd = uint(cd), kd = uint(kd), nkh = uint(nkh), ds = uint(ds), dconv = uint(taps + 1l), beta_off = uint(DN_SM_BETA), g_off = uint(DN_SM_G), taps_off = uint(DN_SM_TAPS), wnorm_off = uint(DN_SM_WNORM), a_off = uint(DN_SM_A), @@ -1138,7 +1420,7 @@ def private dnd_step_make(s_qkv, s_z, s_o : int; woq, woz, woo : int64; eps = eps, qscale = 1.0 / sqrt(float(ds))) enc_dn_step_cls(raw, h, st.fused_set, pcd, nvh) var pco = GemvArgs(n = uint(di), d = uint(dim), nreg = 1u, ybase = 0u) - gemv_cls_enc(fo, raw, h, st.out_set, pco, (dim + rpw - 1l) / rpw) + gemv_enc(fo, raw, h, st.out_set, pco, dim) vhz_dep(raw, h, VHG_YO, 0u, true) cmd_copy_whole(raw, g_gpu.stacks[s_o].y_dev, g_gpu.stacks[s_o].y.buf, dim * 4l) if (vk_prof()) { @@ -1375,11 +1657,9 @@ def private ffn_chain_enc(raw : VkCommandBuffer; var h : VkHaz; var fc : FfnCmd; var uset = stack_gemv_cls_set(s3) hz_set_bits(gset, 0u, 0u, 0u, 0u, 0u, VHG_Y1) hz_set_bits(uset, 0u, 0u, 0u, 0u, 0u, VHG_Y2) - let rpw = g_gpu.rows_per_wg - let ggu = (nrows * nfe + rpw - 1l) / rpw var pcg = GemvArgs(n = uint(n), d = uint(nfe), nreg = uint(nrows), ybase = 0u) - gemv_cls_enc(g_gpu.stacks[s1].fmt, raw, h, gset, pcg, ggu) - gemv_cls_enc(g_gpu.stacks[s3].fmt, raw, h, uset, pcg, ggu) + gemv_enc(g_gpu.stacks[s1].fmt, raw, h, gset, pcg, nrows * nfe) + gemv_enc(g_gpu.stacks[s3].fmt, raw, h, uset, pcg, nrows * nfe) let down_is_kq = !vk_fmt_b32(g_gpu.stacks[s2].fmt) var pca = ActArgs(nelem = uint(nrows * nfe), gelu = is_gelu ? 1u : 0u, nblk = uint(nrows * nfe / (down_is_kq ? 256l : 32l))) @@ -1390,7 +1670,7 @@ def private ffn_chain_enc(raw : VkCommandBuffer; var h : VkHaz; var fc : FfnCmd; enc_q8_actrq_cls(raw, h, fc.actrq_set, pca, awgs) } var pcd = GemvArgs(n = uint(nfe), d = uint(n), nreg = uint(nrows), ybase = 0u) - gemv_cls_enc(g_gpu.stacks[s2].fmt, raw, h, fc.down_set, pcd, (nrows * n + rpw - 1l) / rpw) + gemv_enc(g_gpu.stacks[s2].fmt, raw, h, fc.down_set, pcd, nrows * n) } def private ffn_cmd_for(s1, s3, s2 : int; nrows, n, nfe : int64; is_gelu : bool) : VkCommandBuffer { @@ -1600,13 +1880,42 @@ def private span_layer_sets(a : MoeSpanDec; li : int64) : SpanLayer { SL.s_rq_g3 = set_rq_cls(fixed_array(g_gpu.span_xb_dev, g_gpu.stacks[SL.s3].xq.buf, g_gpu.stacks[SL.s3].xs.buf), fixed_array(nb, XQ_BYTES, XS_BYTES), fixed_array(VHS_XB, VHS_GACTS, VHS_GACTS)) SL.s_router = set_router_gemv_cls(fixed_array(g_gpu.span_router_dev, g_gpu.span_xb_dev, g_gpu.span_logits_dev), - fixed_array(nl * a.ne * nb, nb, SPAN_MAX_EXPERTS * 4l), fixed_array(0u, VHS_XB, VHS_LOG)) + fixed_array(nl * a.ne * nb, nb, MOE_MAX_EXPERTS * 4l), fixed_array(0u, VHS_XB, VHS_LOG)) SL.s_topk = set_topk_cls( fixed_array(g_gpu.span_logits_dev, g_gpu.stacks[SL.s1].meta.buf, g_gpu.stacks[SL.s3].meta.buf, g_gpu.stacks[SL.s2].meta.buf, g_gpu.span_w_dev), - fixed_array(SPAN_MAX_EXPERTS * 4l, META_BYTES, META_BYTES, META_BYTES, MAX_ROUTED_SLOTS * 4l), + fixed_array(MOE_MAX_EXPERTS * 4l, META_BYTES, META_BYTES, META_BYTES, MAX_ROUTED_SLOTS * 4l), fixed_array(VHS_LOG, VHS_META, VHS_META, VHS_META, VHS_META)) SL.s_comb = set_dec_combine_cls(fixed_array(g_gpu.span_x_dev, g_gpu.span_w_dev, g_gpu.stacks[SL.s2].y_dev, g_gpu.span_x_dev), fixed_array(nb, MAX_ROUTED_SLOTS * 4l, Y_BYTES, nb), fixed_array(VHS_X, VHS_META, VHG_YO, VHS_X)) + if (a.nsh > 0l) { + let q8 = int(KqFmt.q8) + SL.sh1 = find_stack(a.wsh1[li], q8, "span shared gate") + SL.sh3 = find_stack(a.wsh3[li], q8, "span shared up") + SL.sh2 = find_stack(a.wsh2[li], q8, "span shared down") + ensure_gemv_scratch(SL.sh1) + ensure_gemv_scratch(SL.sh3) + ensure_gemv_scratch(SL.sh2) + var reg1 = fixed_array(a.wsh1[li], 0l, 1l) + var reg3 = fixed_array(a.wsh3[li], 0l, 1l) + var reg2 = fixed_array(a.wsh2[li], 0l, 1l) + unsafe { + fill_stack_sched_rows(SL.sh1, addr(reg1[0]), 1l, 1l, dim / 32l) + fill_stack_sched_rows(SL.sh3, addr(reg3[0]), 1l, 1l, dim / 32l) + fill_stack_sched_rows(SL.sh2, addr(reg2[0]), 1l, 1l, a.nsh / 32l) + } + SL.s_rq_sh1 = set_rq_cls(fixed_array(g_gpu.span_xb_dev, g_gpu.stacks[SL.sh1].xq.buf, g_gpu.stacks[SL.sh1].xs.buf), + fixed_array(nb, XQ_BYTES, XS_BYTES), fixed_array(VHS_XB, VHS_GACTS, VHS_GACTS)) + SL.s_rq_sh3 = set_rq_cls(fixed_array(g_gpu.span_xb_dev, g_gpu.stacks[SL.sh3].xq.buf, g_gpu.stacks[SL.sh3].xs.buf), + fixed_array(nb, XQ_BYTES, XS_BYTES), fixed_array(VHS_XB, VHS_GACTS, VHS_GACTS)) + if (a.sh_gated) { + SL.s_shgate = set_router_gemv_cls(fixed_array(g_gpu.span_shgate_dev, g_gpu.span_xb_dev, g_gpu.span_logits_dev), + fixed_array(nl * nb, nb, SPAN_LOGITS_BYTES), fixed_array(0u, VHS_XB, VHS_LOG)) + } + SL.s_comb_sh = set_dec_combine_sh_cls( + fixed_array(g_gpu.span_x_dev, g_gpu.span_w_dev, g_gpu.stacks[SL.s2].y_dev, g_gpu.stacks[SL.sh2].y_dev, g_gpu.span_logits_dev, g_gpu.span_x_dev), + fixed_array(nb, MAX_ROUTED_SLOTS * 4l, Y_BYTES, Y_BYTES, SPAN_LOGITS_BYTES, nb), + fixed_array(VHS_X, VHS_META, VHG_YO, VHG_YO, VHS_LOG, VHS_X)) + } if (li + 1l < nl) { let sqn = find_stack(a.woq[li + 1l], a.fq[li + 1l], "span next q") ensure_gemv_scratch(sqn) @@ -1618,7 +1927,7 @@ def private span_layer_sets(a : MoeSpanDec; li : int64) : SpanLayer { return SL } -let private SPAN_MAX_EXPERTS = 256l // the top-k kernel's one-workgroup reach +let private SPAN_LOGITS_BYTES = (MOE_MAX_EXPERTS + 1l) * 4l //! the router logits, then the shared expert's gate logit def private rq_enc(raw : VkCommandBuffer; var h : VkHaz; var rq_set : VkDescriptorSet; fmt : int; dim : int64) { if (kq_sb(fmt)) { @@ -1673,8 +1982,22 @@ def private span_record(a : MoeSpanDec) { vhz_dep(raw, h, VHS_META, 0u) var fc = g_gpu.ffn_cmds[ffn_cmd_key(SL.s1, SL.s3, SL.s2)] ffn_chain_enc(raw, h, fc, SL.s1, SL.s3, SL.s2, a.k, dim, a.nfe, a.is_gelu) - var pcc = DecCombineArgs(n = uint(dim), k = uint(a.k)) - enc_dec_combine_cls(raw, h, SL.s_comb, pcc, (dim + 255l) / 256l) + if (a.nsh > 0l) { + rq_enc(raw, h, SL.s_rq_sh1, int(KqFmt.q8), dim) + rq_enc(raw, h, SL.s_rq_sh3, int(KqFmt.q8), dim) + vhz_dep(raw, h, VHS_GACTS, 0u) + if (a.sh_gated) { + var pcsg = RouterArgs(ne = 1u, dim = uint(dim), woff = uint(li * dim), obase = uint(MOE_MAX_EXPERTS)) + enc_router_gemv_cls(raw, h, SL.s_shgate, pcsg, 1l) + } + var fsh = g_gpu.ffn_cmds[ffn_cmd_key(SL.sh1, SL.sh3, SL.sh2)] + ffn_chain_enc(raw, h, fsh, SL.sh1, SL.sh3, SL.sh2, 1l, dim, a.nsh, a.is_gelu) + var pcs = DecCombineShArgs(n = uint(dim), k = uint(a.k), gidx = uint(MOE_MAX_EXPERTS), gated = a.sh_gated ? 1u : 0u) + enc_dec_combine_sh_cls(raw, h, SL.s_comb_sh, pcs, (dim + 255l) / 256l) + } else { + var pcc = DecCombineArgs(n = uint(dim), k = uint(a.k)) + enc_dec_combine_cls(raw, h, SL.s_comb, pcc, (dim + 255l) / 256l) + } if (li + 1l < nl) { var pca2 = ArArgs(dim = uint(dim), add_on = 0u, woff = uint((li + 1l) * 2l * dim), eps = a.eps, ascale = 1.0) enc_cls_ar(raw, h, SL.s_ar_att, pca2, 1l) @@ -1690,24 +2013,44 @@ def private span_record(a : MoeSpanDec) { vk_check(vkEndCommandBuffer(raw), null) } +//! Forget the recorded span so the next token records anew: the span records once per model and +//! asserts on a changed suffix, and a second model in one process - the tests' shape - has no +//! model drop between the two. The old record's objects stay model-owned until the drop. +def vk_span_reset { + g_gpu.span_layers |> clear() + g_gpu.span_key = -1l + g_gpu.span_nl = 0l + g_gpu.span_form = 0l +} + +//! the routing shape a span record depends on, as one word: experts, slots, the shared expert's width and its gate +def private span_form_word(a : MoeSpanDec) : int64 => ((a.ne * 1024l + a.k) * 65536l + a.nsh) * 2l + (a.sh_gated ? 1l : 0l) + [cold_path] // lazy init, once per span def private ensure_span_state(a : MoeSpanDec) { let nl = long_length(a.woq) if (g_gpu.span_key >= 0l) { - assert(g_gpu.span_key == a.woq[0] && g_gpu.span_nl == nl, "dasLLAMA vulkan tier: the decode span changed under a live model") + assert(g_gpu.span_key == a.woq[0] && g_gpu.span_nl == nl && g_gpu.span_form == span_form_word(a), + "dasLLAMA vulkan tier: the decode span changed under a live model") return } - assert(a.ne <= SPAN_MAX_EXPERTS && a.k <= MAX_ROUTED_SLOTS, "dasLLAMA vulkan tier: decode span routing exceeds the top-k kernel's reach") + assert(a.ne <= MOE_MAX_EXPERTS && a.k <= MAX_ROUTED_SLOTS, "dasLLAMA vulkan tier: decode span routing exceeds the top-k kernel's reach") verify(ensure_router_gemv_cls() && ensure_topk_cls() && ensure_dec_combine_cls() && ensure_cls_ar() - && ensure_cls_q8k_rq() && ensure_cls_dn_rq(), + && ensure_cls_q8k_rq() && ensure_cls_dn_rq() && (a.nsh == 0l || ensure_dec_combine_sh_cls()), "vk decode span: the class rails must engage on a live device") let dim = a.dim g_gpu.span_x_dev = make_device_buf(dim * 4l) g_gpu.span_xb_dev = make_device_buf(dim * 4l) g_gpu.span_w_dev = make_device_buf(MAX_ROUTED_SLOTS * 4l) - g_gpu.span_logits_dev = make_device_buf(SPAN_MAX_EXPERTS * 4l) + g_gpu.span_logits_dev = make_device_buf(SPAN_LOGITS_BYTES) g_gpu.span_router_dev = make_device_buf(nl * a.ne * dim * 4l) g_gpu.span_norms_dev = make_device_buf(nl * 2l * dim * 4l) + if (a.nsh > 0l && a.sh_gated) { + g_gpu.span_shgate_dev = make_device_buf(nl * dim * 4l) + unsafe { + upload_region_at(g_gpu.span_shgate_dev, 0l, reinterpret(a.shgate), nl * dim * 4l) + } + } unsafe { upload_region_at(g_gpu.span_router_dev, 0l, reinterpret(a.router), nl * a.ne * dim * 4l) for (li in range64(nl)) { @@ -1725,7 +2068,9 @@ def private ensure_span_state(a : MoeSpanDec) { span_record(a) g_gpu.span_key = a.woq[0] g_gpu.span_nl = nl - to_log(LOG_INFO, "dasLLAMA vulkan tier: decode span engaged ({nl} resident layers as one submit per token; DASLLAMA_GPU_DEC_SPAN=0 runs the per-layer submits)\n") + g_gpu.span_form = span_form_word(a) + let sh_note = a.nsh > 0l ? " with the shared expert" : "" + to_log(LOG_INFO, "dasLLAMA vulkan tier: decode span engaged ({nl} resident layers{sh_note} as one submit per token; DASLLAMA_GPU_DEC_SPAN=0 runs the per-layer submits)\n") } //! MoeGpuSpanDecFn: one token through the resident layers - one submit; the host feeds the @@ -1819,13 +2164,12 @@ def private qkv_cmd_make(sq, sk, sv : int; woq, wok, wov : int64; n, dq, dk, dv var qset = stack_gemv_cls_set(sq) hz_set_bits(qset, 0u, 0u, 0u, 0u, 0u, VHG_Y1) var h : VkHaz - let rpw = g_gpu.rows_per_wg var pcq = GemvArgs(n = uint(n), d = uint(dq), nreg = 1u, ybase = 0u) - gemv_cls_enc(fq, raw, h, qset, pcq, (dq + rpw - 1l) / rpw) + gemv_enc(fq, raw, h, qset, pcq, dq) var pck = GemvArgs(n = uint(n), d = uint(dk), nreg = 1u, ybase = uint(dq)) - gemv_cls_enc(fk, raw, h, qc.k_set, pck, (dk + rpw - 1l) / rpw) + gemv_enc(fk, raw, h, qc.k_set, pck, dk) var pcv = GemvArgs(n = uint(n), d = uint(dv), nreg = 1u, ybase = uint(dq + dk)) - gemv_cls_enc(fv, raw, h, qc.v_set, pcv, (dv + rpw - 1l) / rpw) + gemv_enc(fv, raw, h, qc.v_set, pcv, dv) vhz_dep(raw, h, VHG_Y1 | VHG_Y2 | VHG_Y3, 0u, true) cmd_copy_whole(raw, g_gpu.stacks[sq].y_dev, g_gpu.stacks[sq].y.buf, qc.ybytes) vk_check(vkEndCommandBuffer(raw), null) @@ -2047,9 +2391,8 @@ def private ensure_cls(si : int; woff, n, d : int64) { let begin = VkCommandBufferBeginInfo() vk_check(vkBeginCommandBuffer(raw, begin), null) var h : VkHaz - let rpw = g_gpu.rows_per_wg var pc = GemvArgs(n = uint(n), d = uint(d), nreg = 1u, ybase = 0u) - gemv_cls_enc(fmt, raw, h, g_gpu.stacks[si].cls_set, pc, (d + rpw - 1l) / rpw) + gemv_enc(fmt, raw, h, g_gpu.stacks[si].cls_set, pc, d) vhz_dep(raw, h, VHG_YO, 0u, true) cmd_copy_whole(raw, g_gpu.stacks[si].y_dev, g_gpu.stacks[si].y.buf, d * 4l) vk_check(vkEndCommandBuffer(raw), null) @@ -2270,13 +2613,12 @@ def vk_moe_attn_dec_hydrate(woq : int64; kp : uint8 const?; vp : uint8 const?; n [arch(at="../ARCHITECTURE_GPU_VULKAN_DECODE.md#decode-attention-block")] def private at_chain_enc(raw : VkCommandBuffer; var h : VkHaz; var L : DatLayer; fq, fk, fv, fo : int; dim, qd, kvd, hs, n_heads, kv_mul : int64; scale, eps : float; neox, qk_norm : bool) { - let rpw = g_gpu.rows_per_wg var pcq = GemvArgs(n = uint(dim), d = uint(qd), nreg = 1u, ybase = 0u) - gemv_cls_enc(fq, raw, h, L.s_q, pcq, (qd + rpw - 1l) / rpw) + gemv_enc(fq, raw, h, L.s_q, pcq, qd) var pck = GemvArgs(n = uint(dim), d = uint(kvd), nreg = 1u, ybase = uint(qd)) - gemv_cls_enc(fk, raw, h, L.s_k, pck, (kvd + rpw - 1l) / rpw) + gemv_enc(fk, raw, h, L.s_k, pck, kvd) var pcv = GemvArgs(n = uint(dim), d = uint(kvd), nreg = 1u, ybase = uint(qd + kvd)) - gemv_cls_enc(fv, raw, h, L.s_v, pcv, (kvd + rpw - 1l) / rpw) + gemv_enc(fv, raw, h, L.s_v, pcv, kvd) let hb = L.has_bias ? 1u : 0u // the layer's own row is bound, so boff stays 0 if (qk_norm) { var pcf = QknRopeArgs(qd = uint(qd), kvd = uint(kvd), hs = uint(hs), half = uint(hs / 2l), @@ -2291,7 +2633,7 @@ def private at_chain_enc(raw : VkCommandBuffer; var h : VkHaz; var L : DatLayer; enc_da_attn_f16_cls(raw, h, L.s_attn, pca, n_heads) rq_enc(raw, h, L.s_rq, fo, qd) var pcw = GemvArgs(n = uint(qd), d = uint(dim), nreg = 1u, ybase = 0u) - gemv_cls_enc(fo, raw, h, L.s_wo, pcw, (dim + rpw - 1l) / rpw) + gemv_enc(fo, raw, h, L.s_wo, pcw, dim) } //! MoeGpuAttnDecFn: one layer's decode attention on the device. The chain diff --git a/modules/dasLLAMA/dasllama/dasllama_vulkan_prefill.das b/modules/dasLLAMA/dasllama/dasllama_vulkan_prefill.das index 3c990a3deb..5e5a4f0080 100644 --- a/modules/dasLLAMA/dasllama/dasllama_vulkan_prefill.das +++ b/modules/dasLLAMA/dasllama/dasllama_vulkan_prefill.das @@ -641,12 +641,194 @@ def private rd_pf_recurrent(raw : VkCommandBuffer; var h : VkHaz; l : int64; b : } // the prefill profile's stamps per layer: the attention head's roles (+ qkn), or the recurrent head's -// (rq_x qkv z [beta alpha | ba] conv scan onorm tail rq_o out), then the shared FFN tail's seven -def private pf_roles_per_layer(recurrent : bool) : int64 { - if (recurrent) { - return 7l + (g_rd.dn_ba_f32 ? 10l : 11l) +// (rq_x qkv z [beta alpha | ba] conv scan onorm tail rq_o out), then the shared FFN tail's seven; an MoE +// layer adds the routed block's eight and drops the dense block's four when it has no shared expert +def private pf_roles_per_layer(L : RLayer) : int64 { + var n = L.recurrent ? 7l + (g_rd.dn_ba_f32 ? 10l : 11l) : (g_rd.qk_norm ? 16l : 15l) + if (L.moe) { + n += PF_MOE_ROLES - (has_dense_triple(L) ? 0l : 4l) + } + return n +} + +//! the routed block's stamps of an MoE layer's tail past ar1 / rq_f at `m0`: the shared expert's four when the layer has one, then the block's eight and ar2 +def private pf_moe_tail_text(acc : double[26]; m0 : int; shexp : bool) : string { + let s = shexp ? "sh_gate {int(acc[m0])} sh_up {int(acc[m0 + 1])} sh_act {int(acc[m0 + 2])} sh_down {int(acc[m0 + 3])} " : "" + let r = m0 + (shexp ? 4 : 0) + return "{s}router {int(acc[r])} topk {int(acc[r + 1])} sched {int(acc[r + 2])} gather {int(acc[r + 3])} e_gate {int(acc[r + 4])} e_up {int(acc[r + 5])} e_act {int(acc[r + 6])} e_down {int(acc[r + 7])} ar2+comb {int(acc[r + 8])}" +} + +let private PF_MOE_ROLES = 8l //! router topk sched gather e_gate e_up e_act e_down (the combine rides the layer's ar2 stamp) +let private PF_MOE_MAP_OFF = 2_048l //! the schedules' first per-wg map word offset: 4-word records for up to 256 experts, the s pieces' then the m pieces' + +[arch(at="../ARCHITECTURE_GPU_VULKAN_MOE.md#vk-prefill-moe-block")] +def vk_rdec_moe_ok(f1, f3, f2 : int) : bool { + if (!vk_moe_init() || !g_gpu.has_coopmat2 || g_gpu.coopmat_mode != COOPMAT_CM2) { + return false + } + return pf_f16_feed(f1) && pf_f16_feed(f3) && pf_f16_feed(f2) +} + +//! the expert tiles' dispatch bounds for a window of `nk` slot rows: the s dispatch takes at most +//! one tile per expert (a bucket within the s column, or the remainder past whole m columns), the +//! m dispatch every expert's whole columns plus a partial one +def private pf_moe_bounds(nk : int64) : tuple { + let wt_gu = (g_rd.nfe + 127l) / 128l + let wt_dn = (g_rd.dim + 127l) / 128l + let rt_m = nk / int64(SCHED_M_ROWS) + g_rd.ne + return (gu = g_rd.ne * wt_gu, dn = g_rd.ne * wt_dn, mgu = rt_m * wt_gu, mdn = rt_m * wt_dn) +} + +//! the window chain's MoE block buffers and class rails, once per residency +[arch(at="../ARCHITECTURE_GPU_VULKAN_MOE.md#vk-prefill-moe-block")] +def private pf_moe_setup { + let np = g_rd.pf_np + let dim = g_rd.dim + let nfe = g_rd.nfe + let nk = np * g_rd.k + g_rd.pf_nk = nk + verify(ensure_router_gemm_cls() && ensure_topk_rows_cls() && ensure_moe_sched_cls() && ensure_moe_gather_f16_cls() + && ensure_actf16_cls() && ensure_cls_ar_comb() && ensure_cls_ar_comb_f16_b(), + "vk prefill: the MoE block's class rails must engage on a live device") + let bounds = pf_moe_bounds(nk) + assert((PF_MOE_MAP_OFF + max(bounds.gu + bounds.mgu, bounds.dn + bounds.mdn)) * 4l <= BATCH_META_BYTES, "dasLLAMA vulkan tier: the MoE block's schedule exceeds its plane") + g_rd.pf_mlog = make_device_buf(np * g_rd.ner * 4l) + g_rd.pf_midx = make_device_buf(nk * 4l) + g_rd.pf_mw = make_device_buf(nk * 4l) + g_rd.pf_minv = make_device_buf(nk * 4l) + g_rd.pf_msch1 = make_device_buf(BATCH_META_BYTES) + g_rd.pf_msch3 = make_device_buf(BATCH_META_BYTES) + g_rd.pf_msch2 = make_device_buf(BATCH_META_BYTES) + g_rd.pf_mxg = make_device_buf((nk + TILE_READ_SLACK) * dim * 2l) + g_rd.pf_my1 = make_device_buf(nk * nfe * 4l) + g_rd.pf_my2 = make_device_buf(nk * nfe * 4l) + g_rd.pf_mhf = make_device_buf((nk + TILE_READ_SLACK) * nfe * 2l) + g_rd.pf_mdn = make_device_buf(nk * dim * 4l) + g_rd.pf_midx_host = make_host_buf(nk * 4l, false, [cached = true]) +} + +[arch(at="../ARCHITECTURE_GPU_VULKAN_MOE.md#vk-prefill-moe-block")] +def private pf_moe_layer_sets(l : int64) { + var L & = unsafe(g_rd.layers[l]) + let np = g_rd.pf_np + let dim = g_rd.dim + let nfe = g_rd.nfe + let nk = g_rd.pf_nk + let ner = g_rd.ner + let tcs = int64(SCHED_S_ROWS) + let tcm = int64(SCHED_M_ROWS) + let xg_bytes = (nk + TILE_READ_SLACK) * dim * 2l + let y_bytes = nk * nfe * 4l + let hf_bytes = (nk + TILE_READ_SLACK) * nfe * 2l + let dn_bytes = nk * dim * 4l + let pe1 = arena_planes(L.fe1, L.be1) + let pe3 = arena_planes(L.fe3, L.be3) + let pe2 = arena_planes(L.fe2, L.be2) + verify(cm2_cls_ensure(L.fe1, tcs) && cm2_cls_ensure(L.fe3, tcs) && cm2_cls_ensure(L.fe2, tcs) + && cm2_cls_ensure(L.fe1, tcm) && cm2_cls_ensure(L.fe3, tcm) && cm2_cls_ensure(L.fe2, tcm), "vk prefill: the expert tile class rails must engage") + L.pf_router_set = set_router_gemm_cls(fixed_array(g_rd.router_dev, g_rd.pf_xb, g_rd.pf_mlog), + fixed_array(g_rd.router_bytes, np * dim * 4l, np * ner * 4l), fixed_array(0u, VHZ_XB, VHZ_MRT)) + L.pf_topk_set = set_topk_rows_cls(fixed_array(g_rd.pf_mlog, g_rd.pf_midx, g_rd.pf_mw), + fixed_array(np * ner * 4l, nk * 4l, nk * 4l), fixed_array(VHZ_MRT, VHZ_MRT, VHZ_MRT)) + L.pf_sched_set = set_moe_sched_cls(fixed_array(g_rd.pf_midx, g_rd.pf_minv, g_rd.pf_msch1, g_rd.pf_msch3, g_rd.pf_msch2), + fixed_array(nk * 4l, nk * 4l, BATCH_META_BYTES, BATCH_META_BYTES, BATCH_META_BYTES), + fixed_array(VHZ_MRT, VHZ_MRT, VHZ_MRT, VHZ_MRT, VHZ_MRT)) + L.pf_gather_set = set_moe_gather_f16_cls(fixed_array(g_rd.pf_xb, g_rd.pf_minv, g_rd.pf_mxg), + fixed_array(np * dim * 4l, nk * 4l, xg_bytes), fixed_array(VHZ_XB, VHZ_MRT, VHZ_MXG)) + L.pf_e1_set = cm2_cls_set(L.fe1, tcs, fixed_array(pe1.wq, pe1.ws, g_rd.pf_msch1, g_rd.pf_mxg, g_rd.pf_my1), + fixed_array(pe1.wqb, pe1.wsb, BATCH_META_BYTES, xg_bytes, y_bytes), fixed_array(0u, 0u, VHZ_MRT, VHZ_MXG, VHZ_GATE)) + L.pf_e1m_set = cm2_cls_set(L.fe1, tcm, fixed_array(pe1.wq, pe1.ws, g_rd.pf_msch1, g_rd.pf_mxg, g_rd.pf_my1), + fixed_array(pe1.wqb, pe1.wsb, BATCH_META_BYTES, xg_bytes, y_bytes), fixed_array(0u, 0u, VHZ_MRT, VHZ_MXG, VHZ_GATE_M)) + L.pf_e3_set = cm2_cls_set(L.fe3, tcs, fixed_array(pe3.wq, pe3.ws, g_rd.pf_msch3, g_rd.pf_mxg, g_rd.pf_my2), + fixed_array(pe3.wqb, pe3.wsb, BATCH_META_BYTES, xg_bytes, y_bytes), fixed_array(0u, 0u, VHZ_MRT, VHZ_MXG, VHZ_UP)) + L.pf_e3m_set = cm2_cls_set(L.fe3, tcm, fixed_array(pe3.wq, pe3.ws, g_rd.pf_msch3, g_rd.pf_mxg, g_rd.pf_my2), + fixed_array(pe3.wqb, pe3.wsb, BATCH_META_BYTES, xg_bytes, y_bytes), fixed_array(0u, 0u, VHZ_MRT, VHZ_MXG, VHZ_UP_M)) + L.pf_eact_set = set_actf16_cls(fixed_array(g_rd.pf_my1, g_rd.pf_my2, g_rd.pf_mhf), + fixed_array(y_bytes, y_bytes, hf_bytes), fixed_array(VHZ_GATE | VHZ_GATE_M, VHZ_UP | VHZ_UP_M, VHZ_HQ)) + L.pf_e2_set = cm2_cls_set(L.fe2, tcs, fixed_array(pe2.wq, pe2.ws, g_rd.pf_msch2, g_rd.pf_mhf, g_rd.pf_mdn), + fixed_array(pe2.wqb, pe2.wsb, BATCH_META_BYTES, hf_bytes, dn_bytes), fixed_array(0u, 0u, VHZ_MRT, VHZ_HQ, VHZ_MDN)) + L.pf_e2m_set = cm2_cls_set(L.fe2, tcm, fixed_array(pe2.wq, pe2.ws, g_rd.pf_msch2, g_rd.pf_mhf, g_rd.pf_mdn), + fixed_array(pe2.wqb, pe2.wsb, BATCH_META_BYTES, hf_bytes, dn_bytes), fixed_array(0u, 0u, VHZ_MRT, VHZ_HQ, VHZ_MDN_M)) + L.pf_arcomb_set = set_cls_ar_comb(fixed_array(g_rd.pf_x, g_rd.pf_ffnout, g_rd.norms_dev, g_rd.pf_xb, g_rd.pf_minv, g_rd.pf_mw, g_rd.pf_mdn, g_rd.pf_mlog), + fixed_array(np * dim * 4l, np * dim * 4l, g_rd.norms_bytes, np * dim * 4l, nk * 4l, nk * 4l, dn_bytes, np * ner * 4l), + fixed_array(VHZ_X, VHZ_FFO, 0u, VHZ_XB, VHZ_MRT, VHZ_MRT, VHZ_MDN | VHZ_MDN_M, VHZ_MRT)) + L.pf_arcomb_f16_set = set_cls_ar_comb_f16_b(fixed_array(g_rd.pf_x, g_rd.pf_ffnout, g_rd.norms_dev, g_rd.pf_xf, g_rd.pf_minv, g_rd.pf_mw, g_rd.pf_mdn, g_rd.pf_mlog), + fixed_array(np * dim * 4l, np * dim * 4l, g_rd.norms_bytes, np * dim * 2l, nk * 4l, nk * 4l, dn_bytes, np * ner * 4l), + fixed_array(VHZ_X, VHZ_FFO, 0u, VHZ_XQ, VHZ_MRT, VHZ_MRT, VHZ_MDN | VHZ_MDN_M, VHZ_MRT)) +} + +[arch(at="../ARCHITECTURE_GPU_VULKAN_MOE.md#vk-prefill-moe-block")] +def private pf_moe_block(raw : VkCommandBuffer; var h : VkHaz; l, wlen : int64) { + var L & = unsafe(g_rd.layers[l]) + let dim = g_rd.dim // nolint:LINT021 — composed into the plane offsets and the push words as int64 + let ne = g_rd.ne // nolint:LINT021 — the push words take the int64 geometry narrowed + let k = g_rd.k + let nfe = g_rd.nfe + let ner = g_rd.ner + let nk = wlen * k + var pcr = RouterGemmArgs(npos = uint(wlen), ne = uint(ner), dim = uint(dim), woff = uint(L.router_slot * ner * dim), ostride = uint(ner)) + enc_router_gemm_cls(raw, h, L.pf_router_set, pcr, ((wlen + int64(ROUTER_TP) - 1l) / int64(ROUTER_TP)) * ((ner + int64(ROUTER_TE) - 1l) / int64(ROUTER_TE))) + pfq_ts(raw) + var pct = TopkRowsArgs(npos = uint(wlen), ne = uint(ne), k = uint(k), norm = g_rd.norm_topk ? 1u : 0u, lstride = uint(ner), wscale = g_rd.wscale) + enc_topk_rows_cls(raw, h, L.pf_topk_set, pct, wlen) + pfq_ts(raw) + let bounds = pf_moe_bounds(nk) + let tcs = int64(SCHED_S_ROWS) + let tcm = int64(SCHED_M_ROWS) + let map_m_gu = uint(PF_MOE_MAP_OFF + bounds.gu) //! the m dispatch's maps follow the s dispatch's + let map_m_dn = uint(PF_MOE_MAP_OFF + bounds.dn) + var pcs = SchedArgs(nk = uint(nk), ne = uint(ne), wt_gu = uint((nfe + 127l) / 128l), wt_dn = uint((dim + 127l) / 128l), + map_off = uint(PF_MOE_MAP_OFF), map_off_mgu = map_m_gu, map_off_mdn = map_m_dn, + bound_gu = uint(bounds.gu), bound_dn = uint(bounds.dn), bound_mgu = uint(bounds.mgu), bound_mdn = uint(bounds.mdn), + base1 = uint(arena_local_blk(L.be1)), stride1 = uint(nfe * dim / fmt_unit(L.fe1)), + base3 = uint(arena_local_blk(L.be3)), stride3 = uint(nfe * dim / fmt_unit(L.fe3)), + base2 = uint(arena_local_blk(L.be2)), stride2 = uint(nfe * dim / fmt_unit(L.fe2))) + enc_moe_sched_cls(raw, h, L.pf_sched_set, pcs, 1l) + pfq_ts(raw) + var pcg = GatherArgs(n = uint(dim), k = uint(k), r0 = 0u, r1 = uint(nk)) + enc_moe_gather_f16_cls(raw, h, L.pf_gather_set, pcg, nk) + pfq_ts(raw) + var pc1 = BatchArgs(n = uint(dim), d = uint(nfe), map_off = uint(PF_MOE_MAP_OFF), ksplit = 0u) + var pc1m = BatchArgs(n = uint(dim), d = uint(nfe), map_off = map_m_gu, ksplit = 0u) + cm2_cls_enc(L.fe1, tcm, raw, h, L.pf_e1m_set, pc1m, bounds.mgu) + cm2_cls_enc(L.fe1, tcs, raw, h, L.pf_e1_set, pc1, bounds.gu) + pfq_ts(raw) + cm2_cls_enc(L.fe3, tcm, raw, h, L.pf_e3m_set, pc1m, bounds.mgu) + cm2_cls_enc(L.fe3, tcs, raw, h, L.pf_e3_set, pc1, bounds.gu) + pfq_ts(raw) + var pca = ActArgs(nelem = uint(nk * nfe), gelu = g_rd.is_gelu ? 1u : 0u, nblk = uint(nk * nfe / 32l), elem0 = 0u) + enc_actf16_cls(raw, h, L.pf_eact_set, pca, (nk * nfe / 4l + 255l) / 256l) + pfq_ts(raw) + var pc2 = BatchArgs(n = uint(nfe), d = uint(dim), map_off = uint(PF_MOE_MAP_OFF), ksplit = 0u) + var pc2m = BatchArgs(n = uint(nfe), d = uint(dim), map_off = map_m_dn, ksplit = 0u) + cm2_cls_enc(L.fe2, tcm, raw, h, L.pf_e2m_set, pc2m, bounds.mdn) + cm2_cls_enc(L.fe2, tcs, raw, h, L.pf_e2_set, pc2, bounds.dn) + pfq_ts(raw) +} + +//! the next layer's fused Q8 head skips its own requant, so this step encodes it. +[arch(at="../ARCHITECTURE_GPU_VULKAN_MOE.md#vk-prefill-moe-block")] +def private pf_moe_addr_next(raw : VkCommandBuffer; var h : VkHaz; l, wlen, nxt : int64) { + var L & = unsafe(g_rd.layers[l]) + let dim = g_rd.dim + var pcc = ArArgs(dim = uint(dim), add_on = has_dense_triple(L) ? 1u : 0u, woff = uint(nxt), eps = g_rd.eps, ascale = 1.0, + slots = uint(g_rd.k), lstride = uint(g_rd.ner), gated = g_rd.sh_gated ? 1u : 0u) + let next_fused = l + 1l < g_rd.n_layers && pf_qkv_feed_fused(l + 1l) + if (next_fused && pf_qkv6(l + 1l)) { + enc_cls_ar_comb_f16_b(raw, h, L.pf_arcomb_f16_set, pcc, wlen) + return + } + enc_cls_ar_comb(raw, h, L.pf_arcomb_set, pcc, wlen) + if (next_fused) { //! the Q8 feed's requant, the next layer's own quant_xb set + let nb = int((l + 1l) * PF_ROLES) + if (kq_sb(g_rd.layers[l + 1l].fq)) { + var pcq = RqArgs(inbase = 0u, nblk = uint(wlen * dim / 256l)) + enc_cls_q8k_rq(raw, h, g_rd.pf_sets[nb], pcq, (wlen * dim / 8l + 255l) / 256l) + } else { + var pcq = RqArgs(inbase = 0u, nblk = uint(wlen * dim / 32l)) + enc_cls_dn_rq(raw, h, g_rd.pf_sets[nb], pcq, (wlen * dim / 4l + 255l) / 256l) + } } - return g_rd.qk_norm ? 16l : 15l } [cold_path, arch(at="../ARCHITECTURE_GPU_VULKAN.md#vk-prefill-window-chain")] // once per residency — buffers, sets, fa arming @@ -771,6 +953,9 @@ def private pf_setup { if (nkvm > 0l) { to_log(LOG_INFO, "dasLLAMA vulkan tier: merged k/v prefill GEMM armed ({nkvm} of {g_rd.n_layers} layers; DASLLAMA_VK_KV_MERGE=0 pins the split pair)\n") } + if (g_rd.moe) { + pf_moe_setup() + } if (g_rd.hybrid) { ensure_dn_state() // dn_ws_dev (the scan workspace) + the staging buffers the reset/handoff ride verify(ensure_dn_conv_cls() && ensure_dn_scan_cls() && ensure_dn_scan_p3_cls() @@ -797,6 +982,9 @@ def private pf_setup { if (L.recurrent) { pf_dn_layer_sets(l, dim, g_rd.dn_cd, g_rd.dn_di) } + if (L.moe) { + pf_moe_layer_sets(l) + } let qkv6 = !L.recurrent && pf_qkv6(l) let wo6 = pf_f16_feed(L.fo) let gu6 = pf_gu6(l) @@ -980,6 +1168,7 @@ def private pf_run(x_batch : array; ids : array; use_ids : bool; e let pairs = int(qd / 2l + kvd / 2l) unsafe { var w0 = 0l + var last_wlen = 0l let taps = max(g_rd.dn_dconv - 1l, 0l) while (w0 < npos) { let tp0 = ref_time_ticks() @@ -1206,7 +1395,8 @@ def private pf_run(x_batch : array; ids : array; use_ids : bool; e po.wqb, po.wsb, aqn, asn, np * dim * 4l, VHZ_AQ, VHZ_XB2, qd, dim, g_rd.layers[l].bo, wlen, fo6) } var pc8 = ArArgs(dim = uint(dim), add_on = 1u, woff = uint((l * 2l + 1l) * dim), eps = g_rd.eps, ascale = 1.0) - if (fuse_arq && !guk) { + let moe = g_rd.layers[l].moe //! the routed block reads the normed rows the fused twin never stores + if (fuse_arq && !guk && !moe) { if (gu6) { enc_cls_ar_f16_b(raw, h, g_rd.pf_arq_sets[int(l * 2l)], pc8, wlen) } else { @@ -1216,53 +1406,61 @@ def private pf_run(x_batch : array; ids : array; use_ids : bool; e } else { enc_cls_ar(raw, h, g_rd.pf_sets[b + 8], pc8, wlen) pfq_ts(raw) - if (gu6) { + let feed_read = !moe || has_dense_triple(g_rd.layers[l]) //! the routed block takes the f32 rows; only a dense triple reads the feed + if (feed_read && gu6) { var pc9 = ActArgs(nelem = uint(wlen * dim), gelu = 0u, nblk = 0u) enc_f16cvt_cls(raw, h, g_rd.pf_sets[b + 9], pc9, (wlen * dim / 4l + 255l) / 256l) - } elif (guk) { + } elif (feed_read && guk) { var pc9 = RqArgs(inbase = 0u, nblk = uint(wlen * dim / 256l)) enc_cls_q8k_rq(raw, h, g_rd.pf_sets[b + 9], pc9, (wlen * dim / 8l + 255l) / 256l) - } else { + } elif (feed_read) { var pc9 = RqArgs(inbase = 0u, nblk = uint(wlen * dim / 32l)) enc_cls_dn_rq(raw, h, g_rd.pf_sets[b + 9], pc9, (wlen * dim / 4l + 255l) / 256l) } } pfq_ts(raw) - let p1 = arena_planes(g_rd.layers[l].f1, g_rd.layers[l].b1) - let p3 = arena_planes(g_rd.layers[l].f3, g_rd.layers[l].b3) - let gqb = gu6 ? g_rd.pf_xf : g_rd.pf_xq - let gsb = gu6 ? g_rd.dummy : g_rd.pf_xs - let gqn = gu6 ? np * dim * 2l : np * dim - let gsn = gu6 ? 256l : np * (dim / 32l) * 4l - let ffn_row0 = pf_ffn_slice_row0(l, g_rd.n_layers, wlen, gu6, dn6) + let ffn_row0 = pf_ffn_slice_row0(l, g_rd.n_layers, wlen, gu6 && !moe, dn6) let ffn_rows = wlen - ffn_row0 - let tgu = cm2_tiles(hid, ffn_rows) - pf_gemm_enc(raw, h, b + 10, g_rd.layers[l].f1, p1.wq, p1.ws, gqb, gsb, g_rd.pf_gate, - p1.wqb, p1.wsb, gqn, gsn, np * hid * 4l, VHZ_XQ, VHZ_GATE, dim, hid, g_rd.layers[l].b1, ffn_rows, gu6, tgu, ffn_row0) - pf_gemm_enc(raw, h, b + 11, g_rd.layers[l].f3, p3.wq, p3.ws, gqb, gsb, g_rd.pf_up, - p3.wqb, p3.wsb, gqn, gsn, np * hid * 4l, VHZ_XQ, VHZ_UP, dim, hid, g_rd.layers[l].b3, ffn_rows, gu6, tgu, ffn_row0) - if (dn6) { - var pc12 = ActArgs(nelem = uint(ffn_rows * hid), gelu = 0u, nblk = uint(ffn_rows * hid / 32l), elem0 = uint(ffn_row0 * hid)) - enc_actf16_cls(raw, h, g_rd.pf_sets[b + 12], pc12, (ffn_rows * hid / 4l + 255l) / 256l) - } elif (dnk) { - var pc12 = ActArgs(nelem = uint(wlen * hid), gelu = 0u, nblk = uint(wlen * hid / 256l)) - enc_q8k_actrq_cls(raw, h, g_rd.pf_sets[b + 12], pc12, (wlen * hid / 8l + 255l) / 256l) - } else { - var pc12 = ActArgs(nelem = uint(wlen * hid), gelu = 0u, nblk = uint(wlen * hid / 32l)) - enc_q8_actrq_cls(raw, h, g_rd.pf_sets[b + 12], pc12, (wlen * hid / 4l + 255l) / 256l) + if (has_dense_triple(g_rd.layers[l])) { + let p1 = arena_planes(g_rd.layers[l].f1, g_rd.layers[l].b1) + let p3 = arena_planes(g_rd.layers[l].f3, g_rd.layers[l].b3) + let gqb = gu6 ? g_rd.pf_xf : g_rd.pf_xq + let gsb = gu6 ? g_rd.dummy : g_rd.pf_xs + let gqn = gu6 ? np * dim * 2l : np * dim + let gsn = gu6 ? 256l : np * (dim / 32l) * 4l + let tgu = cm2_tiles(hid, ffn_rows) + pf_gemm_enc(raw, h, b + 10, g_rd.layers[l].f1, p1.wq, p1.ws, gqb, gsb, g_rd.pf_gate, + p1.wqb, p1.wsb, gqn, gsn, np * hid * 4l, VHZ_XQ, VHZ_GATE, dim, hid, g_rd.layers[l].b1, ffn_rows, gu6, tgu, ffn_row0) + pf_gemm_enc(raw, h, b + 11, g_rd.layers[l].f3, p3.wq, p3.ws, gqb, gsb, g_rd.pf_up, + p3.wqb, p3.wsb, gqn, gsn, np * hid * 4l, VHZ_XQ, VHZ_UP, dim, hid, g_rd.layers[l].b3, ffn_rows, gu6, tgu, ffn_row0) + if (dn6) { + var pc12 = ActArgs(nelem = uint(ffn_rows * hid), gelu = 0u, nblk = uint(ffn_rows * hid / 32l), elem0 = uint(ffn_row0 * hid)) + enc_actf16_cls(raw, h, g_rd.pf_sets[b + 12], pc12, (ffn_rows * hid / 4l + 255l) / 256l) + } elif (dnk) { + var pc12 = ActArgs(nelem = uint(wlen * hid), gelu = 0u, nblk = uint(wlen * hid / 256l)) + enc_q8k_actrq_cls(raw, h, g_rd.pf_sets[b + 12], pc12, (wlen * hid / 8l + 255l) / 256l) + } else { + var pc12 = ActArgs(nelem = uint(wlen * hid), gelu = 0u, nblk = uint(wlen * hid / 32l)) + enc_q8_actrq_cls(raw, h, g_rd.pf_sets[b + 12], pc12, (wlen * hid / 4l + 255l) / 256l) + } + pfq_ts(raw) + let p2 = arena_planes(g_rd.layers[l].f2, g_rd.layers[l].b2) + let hqb = dn6 ? g_rd.pf_hf : g_gpu.hq_dev + let hsb = dn6 ? g_rd.dummy : g_gpu.hs_dev + let hqn = dn6 ? np * hid * 2l : np * hid + let hsn = dn6 ? 256l : np * (hid / 32l) * 4l + pf_gemm_enc(raw, h, b + 13, g_rd.layers[l].f2, p2.wq, p2.ws, hqb, hsb, g_rd.pf_ffnout, + p2.wqb, p2.wsb, hqn, hsn, np * dim * 4l, VHZ_HQ, VHZ_FFO, hid, dim, g_rd.layers[l].b2, ffn_rows, dn6, 0l, ffn_row0) } - pfq_ts(raw) - let p2 = arena_planes(g_rd.layers[l].f2, g_rd.layers[l].b2) - let hqb = dn6 ? g_rd.pf_hf : g_gpu.hq_dev - let hsb = dn6 ? g_rd.dummy : g_gpu.hs_dev - let hqn = dn6 ? np * hid * 2l : np * hid - let hsn = dn6 ? 256l : np * (hid / 32l) * 4l - pf_gemm_enc(raw, h, b + 13, g_rd.layers[l].f2, p2.wq, p2.ws, hqb, hsb, g_rd.pf_ffnout, - p2.wqb, p2.wsb, hqn, hsn, np * dim * 4l, VHZ_HQ, VHZ_FFO, hid, dim, g_rd.layers[l].b2, ffn_rows, dn6, 0l, ffn_row0) - let nxt = l + 1l < g_rd.n_layers ? (l + 1l) * 2l * dim : nlfin // nolint:LINT021 — composed from int64 layout offsets + if (moe) { + pf_moe_block(raw, h, l, wlen) + } + let nxt = l + 1l < g_rd.n_layers ? (l + 1l) * 2l * dim : nlfin var pc14 = ArArgs(dim = uint(dim), add_on = 1u, woff = uint(nxt), eps = g_rd.eps, ascale = 1.0, row0 = uint(ffn_row0)) let next_qkv6 = l + 1l < g_rd.n_layers && pf_qkv6(l + 1l) - if (l + 1l < g_rd.n_layers && pf_qkv_feed_fused(l + 1l)) { + if (moe) { + pf_moe_addr_next(raw, h, l, wlen, nxt) + } elif (l + 1l < g_rd.n_layers && pf_qkv_feed_fused(l + 1l)) { if (next_qkv6) { enc_cls_ar_f16_b(raw, h, g_rd.pf_arq_sets[int(l * 2l + 1l)], pc14, wlen) } else { @@ -1283,14 +1481,17 @@ def private pf_run(x_batch : array; ids : array; use_ids : bool; e } pfq_ts(raw) var pcc = GemvArgs(n = uint(dim), d = uint(g_rd.vocab), nreg = 1u, ybase = 0u) - gemv_cls_enc(g_rd.cls_fmt_marker, raw, h, g_rd.cls_set, pcc, - (g_rd.vocab + g_gpu.rows_per_wg - 1l) / g_gpu.rows_per_wg) + gemv_enc(g_rd.cls_fmt_marker, raw, h, g_rd.cls_set, pcc, g_rd.vocab) pfq_ts(raw) vhz_dep(raw, h, VHZ_LOG, 0u, true) cmd_copy_whole(raw, g_rd.logits_dev, g_rd.pf_logits_host.buf, g_rd.vocab * 4l) } if (vk_prof()) { to_log(LOG_INFO, "vk_rdpf hz: {h.disp} nodes, {h.barriers} barriers\n") // nolint:PERF026,PERF028 — vk_prof()-gated report leg + if (g_rd.moe) { + vhz_dep(raw, h, VHZ_MRT, 0u, true) + cmd_copy_whole(raw, g_rd.pf_midx, g_rd.pf_midx_host.buf, wlen * g_rd.k * 4l) + } } pfq_end(raw) vk_check(vkEndCommandBuffer(raw), null) @@ -1300,20 +1501,74 @@ def private pf_run(x_batch : array; ids : array; use_ids : bool; e to_log(LOG_INFO, "vk_rdpf w0 {w0} rows {wlen} prep {t_prep} record {t_record} submit {get_time_usec(tp0) - t_prep - t_record}\n") // nolint:PERF026,PERF028 — vk_prof()-gated report leg } w0 += wlen + last_wlen = wlen } memcpy(addr(logits[0]), g_rd.pf_logits_host.mapped, g_rd.vocab * 4l) // per-role GPU aggregation, LAST window, only when every stamp landed; overlapped levels (q|k|v, gate|up) bill the level on the last member's delta if (vk_prof()) { pf_prof_report(npos) + if (g_rd.moe) { + pf_moe_bucket_report(last_wlen) + } } } } +//! the profile's bucket report: the last MoE layer's picks of the last window as per-expert row +//! counts - the shape the expert tiles run over - with the 32-row tile count the s column takes +//! and the count a 32 / 64 / 128 column ladder would take +[cold_path] +def private pf_moe_bucket_report(wlen : int64) { + let nk = wlen * g_rd.k + var cnt : array + cnt |> reserve(g_rd.ne) + cnt |> resize(g_rd.ne) + unsafe { + let picks = reinterpret(g_rd.pf_midx_host.mapped) + for (i in range64(nk)) { + let e = int64(picks[i]) + if (e < g_rd.ne) { + cnt[e]++ + } + } + } + var used = 0l + var largest = 0l + var b32 = 0l + var b64 = 0l + var b128 = 0l + var b_over = 0l + var tiles_s = 0l + var tiles_ladder = 0l + for (c in cnt) { + continue if (c == 0l) + used++ + largest = max(largest, c) + tiles_s += (c + 31l) / 32l + if (c <= 32l) { + b32++ + tiles_ladder++ + } elif (c <= 64l) { + b64++ + tiles_ladder++ + } else { + if (c <= 128l) { + b128++ + } else { + b_over++ + } + tiles_ladder += (c + 127l) / 128l + } + } + to_log(LOG_INFO, "vk_rdpf moe buckets (the last MoE layer, {nk} rows over {g_rd.ne} experts): {used} non-empty, the largest {largest}; sizes <=32: {b32} 33-64: {b64} 65-128: {b128} >128: {b_over}; 32-row tiles {tiles_s}, a 32/64/128 ladder {tiles_ladder}\n") + delete cnt +} + [cold_path] // DASLLAMA_GPU_PROF=1 report leg def private pf_prof_report(npos : int64) { var want = 5u for (L in g_rd.layers) { - want += uint(pf_roles_per_layer(L.recurrent)) + want += uint(pf_roles_per_layer(L)) } if (g_pfq_n != want) { to_log(LOG_INFO, "vk_rdpf gpu: roles unaggregated ({g_pfq_n} stamps, the table expects {want})\n") @@ -1321,21 +1576,33 @@ def private pf_prof_report(npos : int64) { } var att : double[16] var dn : double[18] + var moe : double[26] + var dnm : double[26] //! a recurrent MoE layer: the deltanet head's stamps, then the routed tail's var natt = 0l var ndn = 0l + var nmoe = 0l + var ndnm = 0l var qi = 3u for (L in g_rd.layers) { - let n = pf_roles_per_layer(L.recurrent) + let n = pf_roles_per_layer(L) for (s in range64(n)) { - if (L.recurrent) { + if (L.recurrent && L.moe) { + dnm[s] += pfq_us(qi + uint(s)) + } elif (L.recurrent) { dn[s] += pfq_us(qi + uint(s)) + } elif (L.moe) { + moe[s] += pfq_us(qi + uint(s)) } else { att[s] += pfq_us(qi + uint(s)) } } qi += uint(n) - if (L.recurrent) { + if (L.recurrent && L.moe) { + ndnm++ + } elif (L.recurrent) { ndn++ + } elif (L.moe) { + nmoe++ } else { natt++ } @@ -1349,8 +1616,32 @@ def private pf_prof_report(npos : int64) { for (v in dn) { tdn += v } - tot += tatt + tdn - to_log(LOG_INFO, "vk_rdpf gpu ({npos} tokens, last window) prologue {int(pfq_us(2u))} fin_rq {int(pfq_us(qi))} cls {int(pfq_us(qi + 1u))} attention layers {natt}: {int(tatt)} recurrent layers {ndn}: {int(tdn)} total {int(tot)} us\n") + var tmoe = 0.0lf + for (v in moe) { + tmoe += v + } + var tdnm = 0.0lf + for (v in dnm) { + tdnm += v + } + tot += tatt + tdn + tmoe + tdnm + to_log(LOG_INFO, "vk_rdpf gpu ({npos} tokens, last window) prologue {int(pfq_us(2u))} fin_rq {int(pfq_us(qi))} cls {int(pfq_us(qi + 1u))} attention layers {natt}: {int(tatt)} recurrent layers {ndn}: {int(tdn)} MoE layers {nmoe}: {int(tmoe)} recurrent MoE layers {ndnm}: {int(tdnm)} total {int(tot)} us\n") + var has_shexp = false + for (L in g_rd.layers) { + has_shexp ||= L.moe && has_dense_triple(L) + } + if (nmoe > 0l) { + //! an MoE layer's stamps: the attention roles (the qkn one shifts them by one), ar1, rq_f, then the tail + let q = g_rd.qk_norm ? 1 : 0 + let qkn = q > 0 ? "qkn {int(moe[4])} " : "" + to_log(LOG_INFO, "vk_rdpf moe rq_x {int(moe[0])} q {int(moe[1])} k {int(moe[2])} v {int(moe[3])} {qkn}rope_kv {int(moe[4 + q])} attn {int(moe[5 + q])} rq_a {int(moe[6 + q])} wo {int(moe[7 + q])} ar1 {int(moe[8 + q])} rq_f {int(moe[9 + q])} {pf_moe_tail_text(moe, 10 + q, has_shexp)}\n") + } + if (ndnm > 0l) { + //! a recurrent MoE layer's stamps: the deltanet head's (the f32 beta/alpha arm folds two into one), ar1, rq_f, then the tail + let b = g_rd.dn_ba_f32 ? 0 : 1 + let ba = g_rd.dn_ba_f32 ? "ba {int(dnm[3])}" : "beta {int(dnm[3])} alpha {int(dnm[4])}" + to_log(LOG_INFO, "vk_rdpf dnmoe rq_x {int(dnm[0])} qkv {int(dnm[1])} z {int(dnm[2])} {ba} conv {int(dnm[4 + b])} scan {int(dnm[5 + b])} onorm {int(dnm[6 + b])} tail {int(dnm[7 + b])} rq_o {int(dnm[8 + b])} out {int(dnm[9 + b])} ar1 {int(dnm[10 + b])} rq_f {int(dnm[11 + b])} {pf_moe_tail_text(dnm, 12 + b, has_shexp)}\n") + } if (natt > 0l) { // the qkn dispatch stamps between v and rope_kv, shifting every later role by one if (g_rd.qk_norm) { @@ -1439,10 +1730,9 @@ def vk_arena_ffn(var y : array; blk1, blk3, blk2 : int64; n, nfe : int64; let begin = VkCommandBufferBeginInfo() vk_check(vkBeginCommandBuffer(raw, begin), null) var h : VkHaz - let gwg = (nfe + g_gpu.rows_per_wg - 1l) / g_gpu.rows_per_wg var pcg = GemvArgs(n = uint(n), d = uint(nfe), nreg = 1u, ybase = 0u) - gemv_cls_enc(f1, raw, h, gate_set, pcg, gwg) - gemv_cls_enc(f3, raw, h, up_set, pcg, gwg) + gemv_enc(f1, raw, h, gate_set, pcg, nfe) + gemv_enc(f3, raw, h, up_set, pcg, nfe) // the requant form follows the DOWN plane — its GEMV consumes hq/hs var pca = ActArgs(nelem = uint(nfe), gelu = is_gelu ? 1u : 0u, nblk = uint(nfe / (dkq ? 256l : 32l))) let awgs = (nfe / (dkq ? 8l : 4l) + 255l) / 256l @@ -1452,7 +1742,7 @@ def vk_arena_ffn(var y : array; blk1, blk3, blk2 : int64; n, nfe : int64; enc_q8_actrq_cls(raw, h, g_gpu.af_actrq_set, pca, awgs) } var pcd = GemvArgs(n = uint(nfe), d = uint(n), nreg = 1u, ybase = 0u) - gemv_cls_enc(f2, raw, h, down_set, pcd, (n + g_gpu.rows_per_wg - 1l) / g_gpu.rows_per_wg) + gemv_enc(f2, raw, h, down_set, pcd, n) vhz_dep(raw, h, VHG_YO, 0u, true) cmd_copy_whole(raw, g_gpu.af_yo_dev, g_gpu.af_yo_host.buf, n * 4l) vk_check(vkEndCommandBuffer(raw), null) @@ -1552,7 +1842,7 @@ def vk_arena_gemv(var y : array; blk : int64; n, d : int64; fmt : int; xq vk_check(vkBeginCommandBuffer(raw, begin), null) var h : VkHaz var pc = GemvArgs(n = uint(n), d = uint(d), nreg = 1u, ybase = 0u) - gemv_cls_enc(fmt, raw, h, a.cls_sets[sl], pc, (d + g_gpu.rows_per_wg - 1l) / g_gpu.rows_per_wg) + gemv_enc(fmt, raw, h, a.cls_sets[sl], pc, d) vhz_dep(raw, h, VHG_YO, 0u, true) cmd_copy_whole(raw, a.y_dev, a.y.buf, d * 4l) vk_check(vkEndCommandBuffer(raw), null) @@ -2668,8 +2958,6 @@ def private ensure_xf_state { to_log(LOG_INFO, "dasLLAMA vulkan tier: cm2 expert chain engaged\n") } -let private TILE_READ_SLACK = 32l // rows of read slack past a region's last row - //! chunk row cap of the cm2 chain: the gathered f16 image (batch_xq window), the f16 hidden plane //! (hq_dev) and the f32 output planes must all fit; 128-aligned like the tiles' columns [arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-tile-pick-and-default")] diff --git a/modules/dasLLAMA/followup_vulkan.md b/modules/dasLLAMA/followup_vulkan.md index afc7569b6c..9ef2ba4d28 100644 --- a/modules/dasLLAMA/followup_vulkan.md +++ b/modules/dasLLAMA/followup_vulkan.md @@ -300,7 +300,8 @@ Ordered roughly by user-visible value; re-rank against zen2 measurements before the killer was ONE byte4 DYNAMIC select in the sub-scale extract (unpack8(word)[i&3]), the same death shape the Q8 chase found; byte2 [i&1] selects are fine. Respelled as shift + arithmetic-shift sign extension: 12.8 -> 32.9 TF/s. RULE for every future - decode: NEVER index unpack8 of a 32-bit word dynamically - shift+mask, or byte2 [i&1]. + decode: NEVER index unpack8 of a 32-bit word dynamically - shift+mask, or byte2 [i&1] + (and the byte2 select costs against the lane shift too: item 36's 2026-09-09 status). k6 UNPINNED: Qwen3-4B Q4_K_M pp 1626 (mode 3) -> 2669 (k4) -> 3188 (k4+k6) = +96%. NEXT: k5/q40 stamps (mechanical now the trap is named), then (d) driver-blocked. (ngfx GPU Trace, our gate loop vs their GEMM loop; counters now read UNELEVATED): @@ -664,6 +665,19 @@ module) is independent and can land any time - it is pure structure. twins do one grid lookup per four elements and beat the scalar arm by 30-65% at the tile (iq3s 43-48 vs 29-30 TF/s at the gate shape, iq2s 49-50 vs 30-32, iq2xxs 44-46 vs 34-35); the `DECVEC` opt-outs are gone. + 2026-09-09 (the MoE arc): the byte selects of those decodes respelled as lane shifts. A + `unpack8(lane)[i & 1]` byte2 select reads the same 16-bit lane as `(lane >> ((i & 1) * 8)) + & 0xFF` but costs against it, and the IQ2_XXS / IQ3_XXS sign index was built from two + selected bytes where the aux32 word assembled from its two lanes and shifted serves. + `harness/vk_gemm_probe.das -- moe:` (the 30B expert schedule: 128 buckets of 32 + rows, gate/up d 768 K 2048, per plane, two alternating output planes under fresh hazards, + RTX 5060 Ti): the shift form iq2xxs 724 us, iq3xxs 569, iq3s 766, iq2s 736, against + llama.cpp's cm2 `mul_mat_id` tile at 754 / 788 / 870 / 797 on the same uniform buckets (its + `test-backend-ops perf`), the select form 1.28x / 1.24x / 1.49x / 1.08x of those times; + iq2xs 651, iq4xs 632, k4 746 and q8 885 already sat under its 744 / 959 / 1009 / 998. The rule in + `ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2k. The selects left: k5's qh byte in its twin, + the k3 and K-quant scalar decodes (the edge path and the scalar-callback arm) - the same + lever, unmeasured. 37. **Device embed gather over a kq tied plane.** `vulkan_embed_gpu_gate` admits a model only when `rdec_set_emb` placed a q8 tied plane or the f32 table fit under `RDEC_EMB_F32_CAP`; @@ -896,4 +910,62 @@ module) is independent and can land any time - it is pure structure. construction, and the per-op tier reads 0.14x prefill / 0.75x decode against the reference engine on the one such row measured (the arc board's row 20, both engines fully resident). Done = every family and tower row on the board with a Vulkan column at parity, and the - serving census (`coverage-vk`) with no carrier the tier declines. + serving census (`coverage-vk`) with no carrier the tier declines. The fully-resident MoE + chain landed 2026-09-09: the whole-model driver admits a MoE whose expert stacks fit the arena + (`ARCHITECTURE_GPU_VULKAN_RESIDENCY.md` sec.2.2n), the window chain's routed block + (`ARCHITECTURE_GPU_VULKAN_MOE.md` sec.2.2af) and the token command's (same document, + sec.2.2ag); the per-op tier keeps the files that stream. The 30B and 35B rows ride the driver + whole (the plan forgoes the per-op reserves for a fitting file), the hybrid MoE registers its + routed block after the deltanet head, and the expert tiles took two levers: the grid decodes' + lane shifts (item 36's 2026-09-09 status) and the schedule's tile ladder (a bucket past the s + column takes m columns, the last partial: `ARCHITECTURE_GPU_VULKAN_MOE.md` sec.2.2af; the real + window's skew put 4096 rows in 175 s tiles where the ladder runs 85). Every pp512 / tg128 + rate under this item is a `benchmarks/lcpp_bench.das` reading on the RTX 5060 Ti box + (`bin/Release/daslang.exe -jit benchmarks/lcpp_bench.das -- -m -o md + --for-debug-purposes -r 3 -p 512 -n 128 -t 16` under `DASLLAMA_GPU=1 DASLLAMA_IMAGE=0 + DASLLAMA_ALLOW_UNTUNED=1 DASLLAMA_GPU_MIN_CTX=2048 DAS_JOBQUE_THREADS=16`), debug-jit, and + every reference rate beside one is `llama-bench -ngl 99 -fa 1 -t 16 -r 3` at b10660 + (build-vulkan-357) on that box; every window and token millisecond under this item is the + same bench's `DASLLAMA_GPU_PROF=1 ... -n 32 --prof --jobque-profiling` profile - its + `vk_rdpf` window lines and its `vk_rdec gpu avg/token` / `vk_rdec moe avg/token` lines. The + rows, pp512 / tg128: Qwen3-30B-A3B UD-IQ2_XXS 3242.0 / 123.6 + (3520.0 / 116.6: 0.92x / 1.06x; the window 153.9 ms against 142.3, the expert tiles 97.7), + Qwen3.6-35B-A3B UD-IQ2_XXS 2837.7 / 95.9 (2853.1 / 71.6: 0.99x / 1.34x), the Qwen1.5-MoE + twin 5152.9 / 142.5 (5099.8 / 173.8; its window 94.5 ms, the first measured rep after the + warmup reads 101 on every row here, a driver warm-up the bench's one warmup does not absorb). + One more pass the same day: the shared expert's K-quant planes ride beside its q8 transcode + on a load with a GPU tier armed (a CPU-only load keeps the transcode alone) and the + whole-model driver places those (the twin's token 7.06 -> 6.12 ms, tg128 162.7 = + 0.94x), the schedule's slot walks are an atomic tally and cursor (8.1 -> 0.46 ms on the 30B + window), the router tile prefetches its stage as a 64 x 32 tile (6.6 -> 4.3 ms): the 30B + 3448.9 / 124.6 (0.98x / 1.07x, the window 144.9 ms against 142.3), the 35B 2946.0 / 95.2 + (1.03x / 1.33x), the twin 5348.3 / 163.4 (1.05x / 0.94x). The remainder pass's first + lever: the routed combine rides the residual step in both chains (`ClsArComb`, sec.2.2af / + sec.2.2ag), one dispatch per layer fewer, in the two kernels' sum order so the MoE files' + bars keep their calibration; then the router tile's float4 stage (its scalar stage was + bank-conflict bound: 4.3 -> 2.4 ms per 30B window), the residual step's slot groups (eight + rows in flight, then four: the twin's one-row step 360 -> 290 us per token) and the + FFN-norm requant skipped on a layer with no shared expert (540 us per 30B window that + nothing read); then the decode GEMV family's lanes per row (`ARCHITECTURE_GPU_VULKAN_GEMM.md` + sec.2.2ah: a subgroup over one, two or four rows by the row length, the expert rows of a MoE + being the short ones - iq2s at K 768 148 -> 337 GB/s on the probe). The rows on the final + kernels, pp512 / tg128: the 30B 3482.6 / 132.0 (0.99x / 1.13x, the window 143.4 ms against + 142.3, the token 6.79 ms), the 35B 2962.8 / 107.1 (1.04x / 1.50x), the twin 5395.3 / 166.8 + (1.06x / 0.96x), the dense 4B Q4_K_M unchanged at 115-117 (its token a wash on either + form). Still open under this item: the 30B prefill's last 1% (the expert tiles ~99 ms + against the reference's ~88 - a 64-wide column for the 33-64-row buckets - the router's 2.4 + ms against 1.2 on 32 workgroups over 36 SMs, the act 2.8 and the gather 1.6; the profile + stamp after the down tiles absorbs their tail, so the residual step's own cost does not read + there), the twin's decode 4% (a token's expert dispatch is 3-4 MB, where launch and ramp cost + what the transfer does: the 30B's e_down reads 806 us over 48 layers = 240 GB/s effective + against the probe's 337 steady - fewer, larger dispatches, or the gate and up planes in one, + are the next form), the fused add+rms twin that also stores the normed row + (the router's feed, so a MoE could take the fused rail; ar1 reads 1.8-2.0 ms of the 30B + window), the CPU chain's shared expert on the same K-quant planes (it reads the q8 + transcode, so the resident-vs-CPU bar carries the two forms' rounding), an LPT order for + the device schedule's pieces (the m dispatch already leads the s one), and the two probe + arms the checklist's race rule asks for - the router tile's float4 stage against its scalar + stage and the residual step's slot groups against the plain loop, the old bodies kept as + probe twins (both ranked on before/after `DASLLAMA_GPU_PROF=1` profiles across processes, + 4.3 -> 2.4 ms per 30B window and 490 -> 440 us per twin token; ruled 2026-09-09 to ship as + stated claims). diff --git a/modules/dasLLAMA/harness/REVIEW.md b/modules/dasLLAMA/harness/REVIEW.md index c86c775210..0d2c4c9dac 100644 --- a/modules/dasLLAMA/harness/REVIEW.md +++ b/modules/dasLLAMA/harness/REVIEW.md @@ -24,9 +24,7 @@ declaring file's bodies alone - reports such a binding as never accessed, and `@ that report by leaving the binding out of the read and write binding sets the generated dispatch builds its barriers from. -**A diff that adds an arm to `vk_gemm_probe.das` joins it to one of the probe's three axes -(`../ARCHITECTURE_MEASUREMENT.md` sec.2.5): the kernel the engine serves a dense role shape -with, against its alternates; the shipped tile against a copy of itself with one lever moved -back; or the shipped tile against the reference engine's own GEMM shader, served in its place -through `DASLLAMA_VK_SPV_OVERRIDE`.** An arm off those axes has no alternate its number can be -ranked against. +**A diff that adds an arm to `vk_gemm_probe.das` ships the alternate that arm's number is ranked +against - the prior tile, the reference engine's own kernel, the other lane count, whichever the +arm exists to beat (`../ARCHITECTURE_MEASUREMENT.md` sec.2.5).** A number with no alternate +beside it ranks nothing. diff --git a/modules/dasLLAMA/harness/vk_gemm_probe.das b/modules/dasLLAMA/harness/vk_gemm_probe.das index de5a9d14a0..44f3f24dc9 100644 --- a/modules/dasLLAMA/harness/vk_gemm_probe.das +++ b/modules/dasLLAMA/harness/vk_gemm_probe.das @@ -7,9 +7,9 @@ options _dasllama_internal = true // Every arm records once, then times over repeated submits; the serving kernels' oracle gate // is tests/test_vulkan_kernels.das. // Args: gate|down|q|kv|kvm|qkv|tl pin a shape (the long-window ngfx GPU Trace vehicle); -// k4|k6 the K-quant tiles; cm2: any kq format's l/m tiles; cm2x|k6x|ref the bisect and -// reference arms; khrx the KHR kq tile's weight-stage and structural bisect; mmqx the integer tile -// prototypes. +// k4|k6 the K-quant tiles; cm2: any kq format's l/m tiles; moe: its s tile over the +// MoE expert schedule; cm2x|k6x|ref the bisect and reference arms; khrx the KHR kq tile's +// weight-stage and structural bisect; mmqx the integer tile prototypes. // Shapes default to Llama-3.2-3B Q8 geometry (dim 3072, hidden 8192, kv_dim 1024). require math @@ -19,6 +19,7 @@ require vulkan require vulkan/vulkan_boost require spirv/spirv_shader require dasllama/dasllama_vulkan_common +require dasllama/dasllama_gpu_tier // TILE_READ_SLACK: the rows the moe: arm's gathered plane carries past its last region require dasllama/dasllama_vulkan_dispatch require dasllama/dasllama_vulkan_classes require dasllama/dasllama_vulkan_prefill @@ -462,6 +463,198 @@ def private run_cm2_fmt_shape(fmt : int; name : string; d, n, cnt : int) { // delete axsh } +// ===== the expert schedule (args "moe:", "moesk:") ===== +let private MOE_NE = 128 +let private S_ROWS = int(SCHED_S_ROWS) //! the engine's s column: rows per expert region on the uniform profile +let private M_ROWS = int(SCHED_M_ROWS) //! the engine's m column +let private READ_SLACK = int(TILE_READ_SLACK) + +//! a schedule form over the buckets: the whole buckets at the s column, or the ladder's m and s halves +enum private MoeSchedForm { + whole + ladder_m + ladder_s +} + +//! the bucket rows per expert: the s column's count each, or the 30B window's profile +def private moe_bucket_counts(skew : bool) : array { + var cnt : array + cnt |> resize(MOE_NE) + if (!skew) { + for (c in cnt) { + c = S_ROWS + } + return <- cnt + } + for (e in range(MOE_NE)) { + cnt[e] = e < 9 ? 250 : (e < 17 ? 96 : (e < 21 ? 48 : (e < 69 ? 18 : 0))) + } + var placed = 0 + for (c in cnt) { + placed += c + } + cnt[0] += MOE_NE * S_ROWS - placed //! the rows left onto the biggest bucket + return <- cnt +} + +//! the device schedule's words for one form: a record per bucket piece [block base, row base, rows, +//! first wg], the per-wg map after the records, SCHED_NONE from the live workgroups to `bound` +def private moe_sched_words(cnt : array; plane_blk, wtiles, tc, bound : int; form : MoeSchedForm; var live : int&) : array { + let map_off = MOE_NE * 4 + var sched : array + sched |> resize(map_off + bound) + var ri = 0 + var wg = 0 + var row0 = 0 + for (e in range(MOE_NE)) { + let mrows = int(sched_ladder_m_rows(uint(cnt[e]))) + let rows = form == MoeSchedForm.whole ? cnt[e] : (form == MoeSchedForm.ladder_m ? mrows : cnt[e] - mrows) + let r0 = row0 + (form == MoeSchedForm.ladder_s ? mrows : 0) + row0 += cnt[e] + continue if (rows == 0) + let tiles = ((rows + tc - 1) / tc) * wtiles + sched[ri * 4] = uint(e * plane_blk) + sched[ri * 4 + 1] = uint(r0) + sched[ri * 4 + 2] = uint(rows) + sched[ri * 4 + 3] = uint(wg) + for (g in range(tiles)) { + sched[map_off + wg + g] = uint(ri) + } + wg += tiles + ri++ + } + verify(wg <= bound, "the schedule's live workgroups pass the dispatch bound") + for (g in range(wg, bound)) { + sched[map_off + g] = SCHED_NONE + } + live = wg + return <- sched +} + +[arch(at="../ARCHITECTURE_MEASUREMENT.md#one-benchmark-rig")] +def private run_moe_fmt_shape(fmt : int; name : string; d, n : int; skew : bool) { // nolint:STYLE038 — one linear measurement sweep + let unit = fmt == 0 ? 32 : 256 + let plane_blk = d * (n / unit) //! one expert's plane, in blocks + let totblk = MOE_NE * plane_blk + let bb = arena_block_bytes(fmt) + let wq_words = int(bb.wq / 4l) + let nk = MOE_NE * S_ROWS + let xrows = nk + READ_SLACK + var wqh : array + var wsuh : array + var xfh : array + wqh |> reserve(totblk * wq_words) //! the planes pass the unreserved-growth cap (128 expert planes) + wqh |> resize(totblk * wq_words) + for (i in range(totblk * wq_words)) { + wqh[i] = hash_word(uint(i) + 29u) + } + if (fmt == 0) { + wsuh |> reserve(totblk / 2 + 1) + wsuh |> resize(totblk / 2 + 1) + for (i in range(totblk / 2 + 1)) { + wsuh[i] = cool_f16_pair(uint(i) + 7u) + } + } else { + let ws_words = int(bb.ws / 4l) + wsuh |> reserve(totblk * ws_words) + wsuh |> resize(totblk * ws_words) + for (sb in range(totblk)) { + wsuh[sb * ws_words] = packHalf2x16(float2(0.0002 * float(1 + sb % 7), 0.00005 * float(1 + sb % 5))) + for (wi in range(1, ws_words)) { + wsuh[sb * ws_words + wi] = hash_word(uint(sb * ws_words + wi) + 613u) + } + } + } + xfh |> reserve(xrows * n / 2) + xfh |> resize(xrows * n / 2) + for (i in range(xrows * n / 2)) { + xfh[i] = cool_f16_pair(uint(i) * 3u + 5u) + } + let wtiles = (d + 127) / 128 + let map_off = MOE_NE * 4 + var cnt <- moe_bucket_counts(skew) + let bound_whole = (nk / S_ROWS + MOE_NE) * wtiles //! every bucket may straddle a column + let bound_ladder_m = (nk / M_ROWS + MOE_NE) * wtiles //! every bucket may own a partial m column + let bound_ladder_s = MOE_NE * wtiles //! at most one s tile per bucket + var live_whole = 0 + var live_ladder_m = 0 + var live_ladder_s = 0 + var words_whole <- moe_sched_words(cnt, plane_blk, wtiles, S_ROWS, bound_whole, MoeSchedForm.whole, live_whole) + var words_ladder_m <- moe_sched_words(cnt, plane_blk, wtiles, M_ROWS, bound_ladder_m, MoeSchedForm.ladder_m, live_ladder_m) + var words_ladder_s <- moe_sched_words(cnt, plane_blk, wtiles, S_ROWS, bound_ladder_s, MoeSchedForm.ladder_s, live_ladder_s) + let wq_bytes = int64(totblk) * bb.wq + let ws_bytes = long_length(wsuh) * 4l + let xf_bytes = int64(xrows * n) * 2l + let whole_bytes = long_length(words_whole) * 4l + let ladder_m_bytes = long_length(words_ladder_m) * 4l + let ladder_s_bytes = long_length(words_ladder_s) * 4l + let y_bytes = int64(nk * d) * 4l + let wqd = make_device_buf(wq_bytes) + let wsd = make_device_buf(ws_bytes) + let xfd = make_device_buf(xf_bytes) + let whole_d = make_device_buf(whole_bytes) + let ladder_m_d = make_device_buf(ladder_m_bytes) + let ladder_s_d = make_device_buf(ladder_s_bytes) + //! two output planes alternate across the dispatches, each under a fresh hazard rail: no barrier between + //! consecutive dispatches, the served graph's shape (one plane would serialize them on its write-after-write) + let yd_a = make_device_buf(y_bytes) + let yd_b = make_device_buf(y_bytes) + let flop = 2.0lf * double(nk) * double(d) * double(n) + let profile = skew ? "skewed " : "uniform" + unsafe { + upload_region_at(wqd, 0l, addr(wqh[0]), wq_bytes) + upload_region_at(wsd, 0l, addr(wsuh[0]), ws_bytes) + upload_region_at(xfd, 0l, addr(xfh[0]), xf_bytes) + upload_region_at(whole_d, 0l, addr(words_whole[0]), whole_bytes) + upload_region_at(ladder_m_d, 0l, addr(words_ladder_m[0]), ladder_m_bytes) + upload_region_at(ladder_s_d, 0l, addr(words_ladder_s[0]), ladder_s_bytes) + var pc = BatchArgs(n = uint(n), d = uint(d), map_off = uint(map_off)) + let begin = VkCommandBufferBeginInfo() + var raw = alloc_cmd() + vk_check(vkBeginCommandBuffer(raw, begin), null) + let gbits = fixed_array(1u, 2u, 0u, 4u, 16u) + var s_whole_a = cm2_cls_set(fmt, 32l, fixed_array(wqd, wsd, whole_d, xfd, yd_a), + fixed_array(wq_bytes, ws_bytes, whole_bytes, xf_bytes, y_bytes), gbits) + var s_whole_b = cm2_cls_set(fmt, 32l, fixed_array(wqd, wsd, whole_d, xfd, yd_b), + fixed_array(wq_bytes, ws_bytes, whole_bytes, xf_bytes, y_bytes), gbits) + for (k in range(DISPATCHES)) { + var hz : VkHaz + cm2_cls_enc(fmt, 32l, raw, hz, k % 2 == 0 ? s_whole_a : s_whole_b, pc, int64(bound_whole)) + } + vk_check(vkEndCommandBuffer(raw), null) + let us = usec_per_dispatch(raw) + let tf = flop / (us * 1000000.0lf) + print("{name} {profile} whole s-tile: d={d} K={n} wgs={bound_whole} ({live_whole} live) {us / 1000.0lf} ms/dispatch {tf} TFLOP/s timing-only\n") + var raw2 = alloc_cmd() + vk_check(vkBeginCommandBuffer(raw2, begin), null) + var s_ladder_m_a = cm2_cls_set(fmt, 128l, fixed_array(wqd, wsd, ladder_m_d, xfd, yd_a), + fixed_array(wq_bytes, ws_bytes, ladder_m_bytes, xf_bytes, y_bytes), gbits) + var s_ladder_m_b = cm2_cls_set(fmt, 128l, fixed_array(wqd, wsd, ladder_m_d, xfd, yd_b), + fixed_array(wq_bytes, ws_bytes, ladder_m_bytes, xf_bytes, y_bytes), gbits) + var s_ladder_s_a = cm2_cls_set(fmt, 32l, fixed_array(wqd, wsd, ladder_s_d, xfd, yd_a), + fixed_array(wq_bytes, ws_bytes, ladder_s_bytes, xf_bytes, y_bytes), gbits) + var s_ladder_s_b = cm2_cls_set(fmt, 32l, fixed_array(wqd, wsd, ladder_s_d, xfd, yd_b), + fixed_array(wq_bytes, ws_bytes, ladder_s_bytes, xf_bytes, y_bytes), gbits) + for (k in range(DISPATCHES)) { + var hz_m : VkHaz + var hz_s : VkHaz + cm2_cls_enc(fmt, 128l, raw2, hz_m, k % 2 == 0 ? s_ladder_m_a : s_ladder_m_b, pc, int64(bound_ladder_m)) + cm2_cls_enc(fmt, 32l, raw2, hz_s, k % 2 == 0 ? s_ladder_s_a : s_ladder_s_b, pc, int64(bound_ladder_s)) + } + vk_check(vkEndCommandBuffer(raw2), null) + let us2 = usec_per_dispatch(raw2) + let tf2 = flop / (us2 * 1000000.0lf) + print("{name} {profile} ladder m+s: d={d} K={n} wgs={bound_ladder_m}+{bound_ladder_s} ({live_ladder_m}+{live_ladder_s} live) {us2 / 1000.0lf} ms/dispatch {tf2} TFLOP/s timing-only\n") + } + delete wqh + delete wsuh + delete xfh + delete cnt + delete words_whole + delete words_ladder_m + delete words_ladder_s +} + // k4 (Q4_K) cm2 decode-in-load vs the serving kq tile - per-shape rate on the qwen role shapes def private run_k4_shape(name : string; d, n, cnt : int) { // nolint:STYLE038 — one linear measurement sweep let nsb = n / 256 @@ -2714,6 +2907,14 @@ def private run_probe { // nolint:STYLE037,STYLE038 — the flat per-arg shape only = "cm2:" var pf : KqFmt cm2_fmt = kq_fmt_of_name(slice(a, 4), pf) ? int(pf) : -1 + } elif (a |> starts_with("moe:")) { + only = "moe:" + var pf : KqFmt + cm2_fmt = kq_fmt_of_name(slice(a, 4), pf) ? int(pf) : -1 + } elif (a |> starts_with("moesk:")) { + only = "moesk:" + var pf : KqFmt + cm2_fmt = kq_fmt_of_name(slice(a, 6), pf) ? int(pf) : -1 } elif (a |> starts_with("khrprof:")) { only = "khrprof" g_prof_arm = slice(a, 8) @@ -2758,6 +2959,31 @@ def private run_probe { // nolint:STYLE037,STYLE038 — the flat per-arg shape g_gpu.decvec_on = twin_device return } + if (only == "moe:" || only == "moesk:") { + if (!g_gpu.has_coopmat2) { + print("no coopmat2 on this device\n") + return + } + if (cm2_fmt < 0 || !(cm2_fmt == int(KqFmt.q8) || kq_sb(cm2_fmt))) { + print("{only} needs q8 or a kq superblock format name ({cm2_fmt_names()})\n") + return + } + let skew = only == "moesk:" + let twin_device = g_gpu.decvec_on + let arms = twin_device ? 2 : 1 + for (round in range(2)) { + for (arm in range(arms)) { + g_gpu.decvec_on = twin_device && arm == 0 + vkd_pipes_rebuild() + verify(cm2_cls_ensure(cm2_fmt, 32l) && cm2_cls_ensure(cm2_fmt, 128l), "moe: probe pipelines must engage") + let tag = twin_device ? (arm == 0 ? "twin on " : "twin off") : "no twin " + run_moe_fmt_shape(cm2_fmt, "r{round} {tag} gate/up", 768, 2048, skew) + run_moe_fmt_shape(cm2_fmt, "r{round} {tag} down ", 2048, 768, skew) + } + } + g_gpu.decvec_on = twin_device + return + } if (only == "k6x") { if (!g_gpu.has_coopmat2) { print("no coopmat2 on this device\n") diff --git a/modules/dasLLAMA/harness/vk_gemv_probe.das b/modules/dasLLAMA/harness/vk_gemv_probe.das index 6a0b4f6b81..7392c23225 100644 --- a/modules/dasLLAMA/harness/vk_gemv_probe.das +++ b/modules/dasLLAMA/harness/vk_gemv_probe.das @@ -26,8 +26,9 @@ let private STREAM_MB = 128 //! the least a dispatch streams: four times the 3 let private YBIT = 32u //! the y buffer's hazard bit in the binding order below -//! one format at [n x d]: the shipped class over as many d-row region copies as one dispatch needs to stream STREAM_MB -def private probe_fmt(fmt : KqFmt; n, d : int) : tuple { +//! one format at [n x d] with `lanes_per_row` lanes per row (0 = the whole subgroup): the shipped class over as +//! many d-row region copies as one dispatch needs to stream STREAM_MB +def private probe_fmt(fmt : KqFmt; n, d : int; lanes_per_row : uint) : tuple { let nsb = n / 256 let nwsb = d * nsb let qsb_words = int(kq_qsb(fmt) / 4l) @@ -71,7 +72,7 @@ def private probe_fmt(fmt : KqFmt; n, d : int) : tuple let yd = make_device_buf(y_bytes) let rgd = make_device_buf(rg_bytes) var regs <- [for (i in range(copies * 2)); (i % 2 == 0) ? uint(i / 2 * nwsb) : 0u] //! region r: its own plane copy, the one x - let wgs = (int64(d) * int64(copies) + g_gpu.rows_per_wg - 1l) / g_gpu.rows_per_wg + let wgs = gemv_wgs(int64(d) * int64(copies), lanes_per_row) var raw = alloc_cmd() unsafe { upload_region_at(wqd, 0l, addr(wqh[0]), wq_bytes) @@ -86,7 +87,7 @@ def private probe_fmt(fmt : KqFmt; n, d : int) : tuple let begin = VkCommandBufferBeginInfo() vk_check(vkBeginCommandBuffer(raw, begin), null) var h : VkHaz - var pc = GemvArgs(n = uint(n), d = uint(d), nreg = uint(copies), ybase = 0u) + var pc = GemvArgs(n = uint(n), d = uint(d), nreg = uint(copies), ybase = 0u, lanes_per_row = lanes_per_row) for (_i in range(DISPATCHES)) { gemv_cls_enc(int(fmt), raw, h, s_cls, pc, wgs) vhz_dep(raw, h, YBIT, 0u, true) //! the next dispatch waits on this one's y @@ -124,13 +125,16 @@ def main : int { for (f in fmts) { verify(gemv_cls_ensure(int(f)), "vk_gemv_probe: every kq gemv class rail must engage on a live device") } - print("vk_gemv_probe: decode GEMV classes at [n {n} x d {d}], {g_gpu.rows_per_wg} rows/wg, plane copies to {STREAM_MB} MB per dispatch, {DISPATCHES} dispatches x {SUBMITS} submits after a warm-up\n") - print("| fmt | MB/dispatch | us/dispatch | GB/s | |\n") - print("| :----- | ------------: | -----------: | ------: | :---------- |\n") + print("vk_gemv_probe: decode GEMV classes at [n {n} x d {d}], {g_gpu.rows_per_wg} rows/wg, plane copies to {STREAM_MB} MB per dispatch, {DISPATCHES} dispatches x {SUBMITS} submits after a warm-up; lanes = lanes per output row (32 = one row per subgroup)\n") + print("| fmt | lanes | MB/dispatch | us/dispatch | GB/s | |\n") + print("| :----- | ----: | ------------: | -----------: | ------: | :---------- |\n") + let splits = fixed_array(0u, 16u, 8u) for (f in fmts) { - let r = probe_fmt(f, n, d) - let gbs = r.mb * 1048576.0lf / (r.us * 1000.0lf) - print("| {f} | {r.mb:13.1f} | {r.us:12.1f} | {gbs:7.1f} | timing-only |\n") + for (lanes_per_row in splits) { + let r = probe_fmt(f, n, d, lanes_per_row) + let gbs = r.mb * 1048576.0lf / (r.us * 1000.0lf) + print("| {f} | {lanes_per_row == 0u ? g_gpu.subgroup_size : int64(lanes_per_row):5d} | {r.mb:13.1f} | {r.us:12.1f} | {gbs:7.1f} | timing-only |\n") + } } return 0 } diff --git a/modules/dasLLAMA/performance/model_specs.das b/modules/dasLLAMA/performance/model_specs.das index 88eb96760b..bc7793e6c3 100644 --- a/modules/dasLLAMA/performance/model_specs.das +++ b/modules/dasLLAMA/performance/model_specs.das @@ -433,6 +433,10 @@ def model_specs() : array { // nolint:STYLE038 — flat model-set t // weights + a sigmoid-gated shared expert; QKV bias, NEOX rope); the fixture runs // BOTH MoE prefill paths off one load — bit-identical by construction parity <- ev_qwen_count([moe_reference = true])), + // the small twin of the routed-experts arch for the resident MoE chain's cells + // (tests/test_gpu_moe_shexp.das): 9.5 GB, whole on a 16 GB card with room to spare + ModelSpec(file = "Qwen1.5-MoE-A2.7B-Chat-Q4_K_M-local.gguf", + recipe = "llama-quantize --allow-requantize Qwen1.5-MoE-A2.7B-Chat.Q8_0.gguf Qwen1.5-MoE-A2.7B-Chat-Q4_K_M-local.gguf Q4_K_M"), // the qwen vision pairs (the qwen3vl arc): parity-less provenance carriers — // test_vision_chat's deepstack/qwen25o legs and the qwen tier-1 suites are their coverage ModelSpec(file = "Qwen3VL-4B-Instruct-Q8_0.gguf", display = "Qwen3-VL-4B", official = true, @@ -543,6 +547,14 @@ def model_specs() : array { // nolint:STYLE038 — flat model-set t // the MoE arm of the native-kq gate: the grouped prefill gathers per-expert token // batches off the K-form batch image (the seam the dense arms never touch) parity <- ev_qwen_count([kq_native = true, moe_reference = true])), + // the resident MoE chain's fixtures on a 16 GB card (tests/test_gpu_resident_moe.das): the 30B with no + // shared expert, and the 35B deltanet hybrid with a gated one - the whole-model fits the UD-Q4_K_M cannot make + ModelSpec(file = "Qwen3-30B-A3B-Instruct-2507-UD-IQ2_XXS.gguf", + url = "{HF}/unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF/resolve/eea7b2be5805a5f151f8847ede8e5f9a9284bf77/Qwen3-30B-A3B-Instruct-2507-UD-IQ2_XXS.gguf", + bytes = 10341814688l, sha256 = "aa0b04ac05f71aa94b542b074500b3de956608365e74f0e1d97ae0d6a6380269"), + ModelSpec(file = "Qwen3.6-35B-A3B-UD-IQ2_XXS.gguf", + url = "{HF}/unsloth/Qwen3.6-35B-A3B-GGUF/resolve/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-IQ2_XXS.gguf", + bytes = 10756586464l, sha256 = "2e8f5f705355c56311432d0a8a5d14a696dbb7e4b197d05c75ba805fc1857bef"), ModelSpec(file = "Qwen3.8-27B-Q4_K_M.gguf", display = "Qwen3.8-27B", quant_label = "q4_k_m", official = true, url = "{HF}/ggml-org/Qwen3.8-27B-GGUF/resolve/main/Qwen3.8-27B-Q4_K_M.gguf", bytes = 18973870432l, sha256 = "31629f53165ab6a7dad8c9847dcfd1fdf55829dac1e6e748f4a68581b0033d34", diff --git a/modules/dasLLAMA/performance/profile_common.das b/modules/dasLLAMA/performance/profile_common.das index 70077ed9ac..a506097a58 100644 --- a/modules/dasLLAMA/performance/profile_common.das +++ b/modules/dasLLAMA/performance/profile_common.das @@ -1934,7 +1934,7 @@ let private BENCH_ENV_VARS = fixed_array( "DASLLAMA_GPU_DENSE", "DASLLAMA_GPU_DENSE_ATTN", "DASLLAMA_GPU_DENSE_SHEXP", "DASLLAMA_GPU_DN", "DASLLAMA_GPU_DND", "DASLLAMA_GPU_HEAT", "DASLLAMA_GPU_MIN_CTX", "DASLLAMA_GPU_MOE_LAYERS", "DASLLAMA_GPU_MOE_STREAM", - "DASLLAMA_GPU_NAME", "DASLLAMA_GPU_PROF", "DASLLAMA_GPU_QKV", "DASLLAMA_GPU_SHEXP", + "DASLLAMA_GPU_NAME", "DASLLAMA_GPU_PROF", "DASLLAMA_GPU_QKV", "DASLLAMA_GPU_RESIDENT", "DASLLAMA_GPU_SHEXP", "DASLLAMA_GPU_VRAM_MB", "DASLLAMA_IMAGE", "DASLLAMA_METAL_ATTN", "DASLLAMA_METAL_ATTN_D", "DASLLAMA_METAL_ATTN_SINGLE", "DASLLAMA_METAL_BATCH_CONCURRENT", "DASLLAMA_METAL_BATCH_FUSE", "DASLLAMA_METAL_BATCH_GEMM_MIN", "DASLLAMA_METAL_BATCH_MM", "DASLLAMA_METAL_BATCH_MV", diff --git a/modules/dasLLAMA/tests/CLAUDE.md b/modules/dasLLAMA/tests/CLAUDE.md index fb9cc20ec4..d1175e4980 100644 --- a/modules/dasLLAMA/tests/CLAUDE.md +++ b/modules/dasLLAMA/tests/CLAUDE.md @@ -1,8 +1,8 @@ # modules/dasLLAMA/tests - testing discipline -The Metal suites here are wall-time-expensive (model loads dominate; a full pass holds 40GB -GGUFs), so the rules below are enforcement, not advice - an ad-hoc full-suite run turns a -one-arm fix into an afternoon. +The Metal suites here are wall-time-expensive (model loads dominate; a full pass holds the +largest GGUFs the box stocks), so the rules below are enforcement, not advice - an ad-hoc +full-suite run turns a one-arm fix into an afternoon. ## Run suites ONLY through the runner @@ -262,10 +262,15 @@ two lists together are the census. `test_vulkan_dec_tail.das` - model-free (a Vulkan device, else skips): the per-op tier's decode era against a CPU reference - the decode attention block (K-quant and q8 quads, both rope pairings, a q8 pair carrying the q/k/v projection bias, the hydrate arms), the decode FFN tail, and the whole-token decode span with its -device router + top-k against `moe_select_core`, plus the `vulkan_moe_span` override reached +device router + top-k against `moe_select_core`, the span with a shared expert in both its arms - +gated (the shared q8 triple beside the routed pair, its gate logit past the router's, one combine) +and ungated (the same at unit gate, a second span record after a reset; the reference without the +shared expert must miss the device row in both) - plus the `vulkan_moe_span` override reached through its registry. `test_vulkan_moe_cm2.das` - model-free (a cm2 device, else skips): the cm2 expert chain over a -device-side f16 gather, the streamed-group slot hand-off, and the streamed split's async head. +device-side f16 gather, the streamed-group slot hand-off, the streamed split's async head, and +the shared expert's call shape - one region over every position, the identity slot map at unit +weight. `test_vulkan_kernels.das` - model-free (a Vulkan device, else skips): the per-class CPU-oracle units of the Vulkan kernel census (`_vkd_oracles.das` runs the class methods on the CPU as the oracle; `_vkd_toy.das` is the `[vk_dispatch]` bring-up fixture). The per-format tile cells @@ -273,11 +278,26 @@ oracle; `_vkd_toy.das` is the `[vk_dispatch]` bring-up fixture). The per-format `test_vkd_cm2l_batch` / `test_vkd_cm2m_batch` / `test_vkd_cm2s_batch`, which carry no KHR arm, and q51 carries no tile cell) run four arms: the cm2 l/m/s tiles in mode 4 on an NV_coopmat2 device and the KHR 128x128 tile wherever the device has KHR coopmat at subgroup -32 - the cell skips only when the device has neither, so a KHR-only card still runs its arm; +32 - the cell skips only when the device has neither, so a KHR-only card still runs its arm; the +k4 cell dispatches two workgroups past its schedule over sentinel map words (`SCHED_NONE`), the +device-written schedules' upper-bound shape, and every arm's rows still match; `test_vkd_direct_decode` proves a `[spirv_decode]` method called from a kernel body on the plane element (the KHR arm's staging form: the index travels, the callee chains through the plane) is an ordinary call on -the device, against the same method run on the CPU. +the device, against the same method run on the CPU. `test_vkd_moe_routing` holds the resident MoE +block's routing kernels to CPU oracles: the batched router GEMM at a second layer's offset (at 21 +rows, and at 9 - fewer than the tile's half, so every invocation's second row is guarded off), +the per-row top-k against `moe_select_core` (renormalized, and scaled), the device bucket schedule +against its CPU twin - the records and maps word for word over the whole planes (the tile +ladder's two piece lists over a whole m column with an s remainder, partial m columns and +s-column buckets, the ladder spelled out in the oracle rather than called; empty experts; the +four sentinel tails), the slot-to-bucket-row map as a permutation of each bucket's rows (its +atomic cursor fixes no order within a bucket) - and the combine-folded residual step (the shared +row at its gate, ungated, and no shared expert; the in-place residual moved off its input; the +f32 normed row and the f16 twin against one oracle) - every output under a sentinel fill before +its dispatch, every bar with its own poison. `test_vkd_gemv_lane_rule` pins the decode GEMV +family's lanes-per-row rule (`gemv_lanes_per_row`) per format class and row length, the grid +formats against the k-lattice ones and q8 at the whole subgroup. `test_bench_records_schema.das` - model-free: the record store's schema (round-trip, upsert identity with `workload` in the key, annotations landing only on the rows they select, the store lister admitting `records/{box}.json` alone) and the record rig's shared seams (the @@ -308,6 +328,16 @@ with no escape compiles; an internal require does not) via spawned compiles, `load_audio_16k_mono`'s empty-on-failure contract, `decode_audio_16k_mono`'s frame cap (a synthetic `sampleRate=1` WAV bomb is refused before decode, an uncapped call still works), and `gemma4a_probe_proj_dim`'s 0-not-panic contract on `.dlim` / missing / non-GGUF inputs. +`test_mtp.das` - stocked suite, `-jit` only: the NextN loader on the Qwen3.5-0.8B-MTP Q8_0 file +(the extras load; spec off continues the plain file's fixture), the self-speculative greedy decode +token-for-token against plain decode on the counting and prose fixtures, the poisoned-verify +rollback, the 64-trunk-layer Qwen3.6-27B-MTP carrier, and the GLM-4.5-Air arm - the one +non-recurrent MTP model, so the only reach of the depth-1 step's shortcut reject (row 0's logits +and hidden stand, no re-forward): its code fixture runs plain, then again with +`set_mtp_force_reject_every(3)` rejecting every third draft, both token-for-token against plain +decode, the forced run's reject count asserted at a third of its drafts or more (the fixture's own +reject count rides the kernels' summation order and has read zero). The 27B and GLM arms are +large-tier (`DASLLAMA_PARITY_FULL=1`). `test_mtp_snapshot.das` - model-free: the speculative round's deltanet rollback sizes its two snapshot buffers on a bare session carrying a 27B-class recurrent state (151 MB, past the `max_unreserved_size` guard) and restores the state from them. @@ -330,9 +360,12 @@ deltanet decode step, and skips otherwise. `test_gpu_serving_declines.das` - model-free: the whole-model driver's decline reasons decided from a Config or a synthetic Model shell (`resident_unserved_features`, `attn_chain_unserved_features`, `resident_layer_decline`) - every unserved feature and layer -shape is named in the text a user reads, a served one yields ""; plus the KV mirror's binding cap -(`resident_binding_ctx`) on a hybrid shell whose layer 0 is recurrent, its dense twin, and a -shell with no attention layer. +shape is named in the text a user reads, a served one yields ""; the MoE names among them (the +router shapes the routed block's kernels do not serve, the expert and slot counts past their +reach) and the MoE layer helpers (`layer_is_moe`, `resident_dense_width`: a layer routes only +past the dense lead with all three expert planes, and its dense width is the shared expert's); +plus the KV mirror's binding cap (`resident_binding_ctx`) on a hybrid shell whose layer 0 is +recurrent, its dense twin, and a shell with no attention layer. `test_gpu_resident_hybrid.das` - stocked suite, `-jit` only; the whole-model resident driver on a deltanet hybrid under `DASLLAMA_GPU=1`. Each fixture is a row in `../performance/model_specs.das`: @@ -380,11 +413,43 @@ applies there. One cell is model-free: `test_kernel_census_by_name` holds that the census accessor panics on a kernel name nothing seeded, so a misspelt key cannot read as a zero count. -`test_gpu_resident_qwen2.das` - stocked suite; the whole-model resident driver on a qwen2 +`test_gpu_resident_qwen2.das` - stocked suite, `-jit` only; the whole-model resident driver on a qwen2 (Qwen2.5-0.5B-Instruct-Q8_0, `DASLLAMA_GPU=1`): the q/k/v projection bias folded into the rope stage on the device - the hybrid file's forced-feed logits-tolerance form (its K-quant 6% bar, the one-step-off control) at one window and two windows, with the arm witnesses that the model carries the bias and the driver armed on it; skips without the model or the armed tier. +`test_gpu_resident_moe.das` - stocked suite, `-jit` only; the whole-model resident driver on a MoE +(Qwen1.5-MoE-A2.7B-Chat-Q4_K_M-local, `DASLLAMA_GPU=1`): the expert stacks in the arena, the window +chain's routed block and the token command's routed block - the hybrid file's forced-feed +logits-tolerance form at the routed chain's 20% bar (the arms part on the router's near-ties from +layer 1 on; a flipped near-tie on the 35B at two windows reads 0.155-0.16 of the step's max logit +under either of two summation orders, the one-step-off controls 0.42 and above; the bar's `//!` +carries the reading) with the one-step-off control, at one window and +two windows, plus the census witnesses: the device bucket schedule and the per-row select ran once +per MoE layer per window, the token command's top-k count a whole multiple of the MoE layer count +(the command records once and resubmits); the second +fixture is the Qwen3.6-35B-A3B UD-IQ2_XXS hybrid, whose recurrent layers take the routed block +after the deltanet head, at the same two lengths; the third is the Qwen3-30B-A3B UD-IQ2_XXS, +the MoE with no shared expert (the residual step with its add partner off, the FFN-norm requant +skipped), at the same two lengths; all three are large-tier (`DASLLAMA_PARITY_FULL=1`), and the +cells skip without the file, the armed tier, or a device with no cm2 tile family (the driver +declines a MoE there by design); on a cm2 device the driver's admission of the fixture is +asserted, a decline is a red that sends the reader to the load log. Every cell pins the resident +route on for its load (`set_gpu_resident_route`) and restores the lever after. +`test_gpu_moe_shexp.das` - stocked suite, `-jit` only; the shared expert's prefill on the device +(Qwen1.5-MoE-A2.7B-Chat-Q4_K_M-local, the Q4_K_M mint of the Q8_0 carrier, `DASLLAMA_GPU=1`): the +shexp triple as one region over every position of the routed experts' chain, gated by the tier's +in-process route lever - the same prompt and fed tokens with the route on (the device arm) and off +(the CPU form), the logits within the 12% bar at the prefill and every step (the arms part on the +router's near-ties from layer 1 on, not on the shared expert's rows - the bar's `//!` carries the +reading), the one-step-off control, and the engage witness in two halves - the device prefill's +shexp layer count (`shexp_gpu_prefill_layers`) grows by the model's layers on the device arm and +not at all on the CPU arm, while `span_tokens()` grows by the fed step count on BOTH arms (the +decode span serves every step either way, the shared expert inside it only on the device arm) - +at 64 and 600 tokens; the twin is large-tier, so the cells run under +`DASLLAMA_PARITY_FULL=1`, and skip without the twin, the armed tier, or a shexp mark on every layer. +The whole-model driver is pinned off for the load (`set_gpu_resident_route`): it would take the +twin whole, and its arm is the resident MoE file's. `test_gpu_model_swap.das` - stocked suite; two models through one process on the armed tier (Qwen3-0.6B, SmolLM2-135M, `DASLLAMA_GPU=1`): a model reloaded behind the other decodes its own weights, the pin on the upload rail dropping a still-installed model's device state first; skips @@ -926,6 +991,8 @@ stale-cache red class does not exist for it. Every `[test]` file requiring a `dasllama/*` module outside this folder, each with its reason: - `utils/dasllama-server/test_openai_server*.das` - require the server by bare same-dir name (the hyphenated directory is unreachable by path require). +- `utils/dasllama-server/test_worker_dispatch.das` - requires the server (`openai_server`) by + bare same-dir name, like the server suites beside it. - `utils/dasllama-server/test_exchange_client.das` - requires `dasllama/dasllama_exchange` by registered name (nothing pins it to that directory); it stays beside the server suites because its fixed test port is coordinated with theirs (see its `TEST_PORT` note). diff --git a/modules/dasLLAMA/tests/REVIEW.md b/modules/dasLLAMA/tests/REVIEW.md index d44d07bbc9..5691816b89 100644 --- a/modules/dasLLAMA/tests/REVIEW.md +++ b/modules/dasLLAMA/tests/REVIEW.md @@ -7,8 +7,9 @@ doc: `CLAUDE.md`. Planned work: `../followup_general.md`, `../followup_vulkan.md **A kernel-unit cell - a model-less cell that dispatches one kernel class and asserts on its output - or a gate that hand-dispatches or hand-binds a kernel, wherever the diff puts it, and a diff that changes a `[metal_dispatch]` or `[vk_dispatch]` class's dispatch geometry, its -kernel-argument struct (`kargs`) or that struct's fields, apply `REVIEW_KERNEL_CELLS.md` -(beside this file) together with this list.** +kernel-argument struct (`kargs`), that struct's fields, or which branch the class takes on a +value in a buffer it binds, apply `REVIEW_KERNEL_CELLS.md` (beside this file) together with +this list.** **Every PR runs `run.das -- --suite model-free` and `run.das -- --suite stocked` on a box with the models stocked, plus every test here the change reaches - never the whole directory.** A @@ -47,16 +48,18 @@ a run of skips is not the coverage the suite owes. **A diff that registers a test file in this folder in a `CMakeLists.txt` is a defect - a `run.das` suite listing is the only registration these files get.** -**A diff that adds, removes or moves a gate - one test cell, or a file between suites - updates -the `CLAUDE.md` census entry of the file it lands in, in the same change.** A `{a,b}` shorthand -naming several files at once, or a suite roster, carries nothing to correct. +**A diff that changes what a file covers - a cell added, removed or moved, its suite, an axis or +bar an existing cell asserts - corrects that file's `CLAUDE.md` census entry, numbers included, +in the same change.** A `{a,b}` shorthand naming several files at once, or a suite roster, +carries nothing to correct. **A diff that changes the contract a gate pins - what its asserts hold fixed, an axis gained or lost - updates that gate's entry in this checklist's pinned set in the same change.** **A diff that adds, changes, or drops a gate's skip condition - the model, fixture, device or -arm it needs - updates the test file's own header in the same change, and adds or corrects the -skip clause in that file's `CLAUDE.md` entry where `CLAUDE.md` carries one.** +arm it needs - updates the test file's own header - the top comment block carrying every fact a +`t |> skip` in the file keys on (model, tier, device, arm, knob) - in the same change, and adds +or corrects the skip clause in that file's `CLAUDE.md` entry where `CLAUDE.md` carries one.** **A diff that adds, moves, or removes a `[test]` file outside `modules/dasLLAMA/` that carries a `require dasllama/...` line of its own adds, corrects, or drops its row, with the reason it @@ -70,11 +73,6 @@ block and `../CLAUDE.md` in the same change.** A data row in a table `run.das` l the surface for an agent that reads them cold; a copy the code has left behind sends that agent to a flag that no longer does what the text says. -**A new test file listed in `run.das`'s `model-free` or `stocked` suite, or in no `run.das` -suite at all, whose name does not say what it covers, gets a `CLAUDE.md` entry in the same -change** - `run.das`'s `model-free` and `stocked` lists together are the complete census, the -`CLAUDE.md` map is deliberately partial. - **A diff that adds, renames, or drops an arm name - the literal passed to `arm_on(t, name)` (`_model_tier.das`), what `--arm` matches - updates the arm census in `CLAUDE.md`'s "Arm filter mechanics" section in the same change** - an arm the census does not name is @@ -103,13 +101,11 @@ worker-local fork pools, shared queue policy. kernel regressing, adds it to the pinned set in the same change** - as a file when every cell of it pins, as a named cell otherwise. -**On every platform, a cell that neither asserts nor registers a skip is a defect.** A cell -that returns without asserting - whatever the reason - registers `t |> skip` there; `feint` is a -print, not a skip. - -**A cell whose claim needs a capability the box may lack - a device, a window server, an audio -device, a module the build omits, a stocked model - registers `t |> skip` on that fact before -it asserts: never a bare return, never a red.** +**On every platform, a cell that neither asserts nor registers a skip is a defect.** A cell that +returns without asserting - whatever the reason - registers `t |> skip` there, and one whose +claim needs a capability the box may lack (a device, a window server, an audio device, a module +the build omits, a stocked model) registers that skip on the fact before it asserts, never a +bare return and never a red; `feint` is a print, not a skip. **A cell's skip condition keys on a fact the box owns - a device capability, a run-mode knob's value, a host toolchain's presence, a compile-time module-presence check diff --git a/modules/dasLLAMA/tests/REVIEW_KERNEL_CELLS.md b/modules/dasLLAMA/tests/REVIEW_KERNEL_CELLS.md index cdcbb4be6e..3e552b37fd 100644 --- a/modules/dasLLAMA/tests/REVIEW_KERNEL_CELLS.md +++ b/modules/dasLLAMA/tests/REVIEW_KERNEL_CELLS.md @@ -21,11 +21,11 @@ the wrong buffer and passes on garbage that happens to compare. output - missing a compare against a CPU oracle that can witness the cell's property is a defect.** -**A kernel that gains a kargs field whose non-default value changes what it computes or which -elements it reads or writes - a branch selector, a row or element base, a stride - ships a -model-less kernel-unit cell that sets that field to a non-default value, in the same change.** -At the default the new field has no visible effect: a CPU oracle that ignores it and the kernel -that honors it agree. +**A kernel that gains a new branch a cell can select - a kargs field, or a sentinel value in a +bound buffer - whose non-default value changes what it computes or which elements it reads or +writes ships a model-less kernel-unit cell that sets that branch to a non-default value, in the +same change.** At the default the new branch has no visible effect: a CPU oracle that ignores it +and the kernel that honors it agree. **A kernel-unit cell fills a GPU output buffer with a sentinel before every dispatch whose output it then reads.** An unprefilled output can pass by staying stale - the previous @@ -45,10 +45,10 @@ the same gate that must RED - a poisoned input, a poisoned expectation, a discon mechanism, or a second independent lane; a gate's own reference is never its control. **A kernel-unit cell whose kernel computes at a narrower precision than its oracle at any -step - operands or accumulator - bounds that step's error by construction - f16-exact inputs, -magnitude-bounded fixtures - or states in the cell how its new bar follows from that step's -error.** A bar moved without that derivation is a loosening: the compare then measures the -narrowing until it no longer discriminates. +step - operands, accumulator, or the stored result - bounds that step's error by construction +(f16-exact inputs, magnitude-bounded fixtures) or states in the cell how its new bar follows +from that step's error.** A bar moved without that derivation is a loosening: the compare then +measures the narrowing until it no longer discriminates. **A gate for a kernel that attends inside a restricted horizon - a window, a sliding span, a block-diagonal range - writes its CPU oracle to attend strictly inside that horizon.** A leak diff --git a/modules/dasLLAMA/tests/_vkd_toy.das b/modules/dasLLAMA/tests/_vkd_toy.das index 14593d0f60..3871d5d73a 100644 --- a/modules/dasLLAMA/tests/_vkd_toy.das +++ b/modules/dasLLAMA/tests/_vkd_toy.das @@ -76,7 +76,6 @@ class ToyQ8Cm2Strip : Q8Cm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } [vk_dispatch(name = "toy_q8_cm2_twin_cls", kernel = "run", grid = "wgs", params = "wgs : int64")] @@ -84,5 +83,4 @@ class ToyQ8Cm2Twin : Q8Cm2T { typedef BT = coopmatWgB_f16_64x256 typedef ACC = coopmatWgAcc_f16_128x256 typedef ACCW = coopmatWgAcc_f32_128x256 - typedef FLO = tensorLayout2D } diff --git a/modules/dasLLAMA/tests/run.das b/modules/dasLLAMA/tests/run.das index d1b94fbb38..5d62020198 100644 --- a/modules/dasLLAMA/tests/run.das +++ b/modules/dasLLAMA/tests/run.das @@ -183,7 +183,9 @@ def suite_files(name : string) : array { // nolint:STYLE038 - a flat s "modules/dasLLAMA/tests/test_gemma4uv.das", "modules/dasLLAMA/tests/test_gemma4v.das", "modules/dasLLAMA/tests/test_gpu_model_swap.das", + "modules/dasLLAMA/tests/test_gpu_moe_shexp.das", "modules/dasLLAMA/tests/test_gpu_resident_hybrid.das", + "modules/dasLLAMA/tests/test_gpu_resident_moe.das", "modules/dasLLAMA/tests/test_gpu_resident_qwen2.das", "modules/dasLLAMA/tests/test_gpu_slot_swap.das", "modules/dasLLAMA/tests/test_kquant.das", @@ -259,7 +261,7 @@ def area_tests(area : string) : array { return <- [ "test_accel_backend.das", "test_affinity_predicates.das", "test_arch_registry.das", "test_batch_decode.das", "test_batch_grid.das", "test_chat.das", "test_deltanet.das", "test_dispatch_shaping.das", "test_facade.das", "test_flash.das", "test_forward.das", "test_fused_decode.das", "test_gguf_quant.das", "test_gguf_shards.das", - "test_gpu_model_swap.das", "test_gpu_resident_hybrid.das", "test_gpu_resident_qwen2.das", "test_gpu_serving_declines.das", "test_gpu_slot_swap.das", "test_gpu_tier.das", "test_groupn.das", + "test_gpu_model_swap.das", "test_gpu_moe_shexp.das", "test_gpu_resident_hybrid.das", "test_gpu_resident_moe.das", "test_gpu_resident_qwen2.das", "test_gpu_serving_declines.das", "test_gpu_slot_swap.das", "test_gpu_tier.das", "test_groupn.das", "test_jobque_tripwire.das", "test_kernel_backend.das", "test_kgroup_repack.das", "test_kqformat.das", "test_kquant.das", "test_kv_codec.das", "test_kv_paged.das", "test_kv_prefix.das", "test_math_activations.das", "test_math_elementwise.das", "test_matmul.das", "test_matmul_batch.das", diff --git a/modules/dasLLAMA/tests/test_gpu_moe_shexp.das b/modules/dasLLAMA/tests/test_gpu_moe_shexp.das new file mode 100644 index 0000000000..5c0ef898a0 --- /dev/null +++ b/modules/dasLLAMA/tests/test_gpu_moe_shexp.das @@ -0,0 +1,145 @@ +options gen2 +options stack = 524288 // every dasLLAMA program root takes this budget (options stack does not unify up from libs) +options persistent_heap // + explicit deletes below: the model and sessions free before exit +options _dasllama_internal = true + +require dastest/testing_boost public +require dasllama/dasllama_transformer // umbrella fires each arch [init] registration (qwen2moe) +require dasllama/dasllama // create_session + +[init] +def private allow_cpu_prefill_intent() { + allow_cpu_prefill() // the CPU-form arm prefills its shared expert on the CPU by design +} +require daslib/jobque_boost +require daslib/fio +require dasllama/dasllama_math // setup_dasllama_jobque_ (the engine-standard jobque config) +require dasllama/dasllama_gpu_resident // moe_gpu_resident_active: the route lever kept the whole-model driver off the twin +require dasllama/dasllama_gpu_tier // moe_gpu_tier_installed / gpu_want_auto / moe_gpu_shexp_on_gpu / set_moe_gpu_shexp_route / set_gpu_resident_route +require dasllama/dasllama_blocks // shexp_gpu_prefill_layers: the device arm's engage witness +require _model_tier // models_dir() + model_available() presence/size-tier gate, the logits compares + +// The shared expert's prefill on the device: a qwen2moe layer's shexp triple rides the routed +// experts' chain as one region over every position (ARCHITECTURE_GPU_VULKAN_GEMM.md sec.2.2q). The +// device chain and the CPU form are not bit-identical (the f16 gather, f32 accumulation order), +// so the gate is the forced-feed logits-tolerance form over the tier's in-process route lever: +// the same prompt and fed tokens with the route ON (the device arm) and OFF (the CPU form), the +// logits within a bar of the CPU arm after the prefill and at every step, the one-step-off +// control past it, and the engage witness - the arm's layer count grows by the model's layer +// count per prefill on the device arm and not at all on the CPU arm. The whole-model driver is +// pinned off for the load: it would take the twin whole (test_gpu_resident_moe.das is its gate). + +let private TWIN = "Qwen1.5-MoE-A2.7B-Chat-Q4_K_M-local.gguf" //! the Q4_K_M mint of the Q8_0 carrier (its recipe row in ../performance/model_specs.das) +//! of the step's max |logit|. The arms agree to 1e-8 at layer 0 and part from layer 1 on: the device chain's +//! summation order moves the residual by ulps, the next layer's Q8 requant rounds a few elements the +//! other way and the router flips a near-tie, and 24 layers of that read 0.05-0.10 on the twin under both +//! tile modes (the routed chain is bit-stable session to session; the shared expert's rows themselves sit +//! within 1e-4 of the CPU form on the sdot4 feed and 0.5% on the f16 feed - test_vulkan_moe_cm2's cells) +let private SHEXP_LOGIT_BAR_REL = 0.12 +let private FED_STEPS = 4l + +def private argmax_of(v : array) : int64 { + var best = -1e30 + var bi = 0l + for (i in range64(long_length(v))) { + if (v[i] > best) { + best = v[i] + bi = i + } + } + return bi +} + +//! prefill `prompt`, then feed `toks` one per step; the logits after the prefill and after each step +def private feed_logits(tr : Model; var s : Session; prompt : array; toks : array) : array> { + let n = long_length(prompt) + var rows : array> + rows |> reserve(long_length(toks) + 1l) + forward_prefill(tr, s, prompt, n, 0l) + rows |> emplace <| [for (i in range64(tr.config.vocab_size)); s.logits[i]] + for (st in range64(long_length(toks))) { + forward(tr, s, toks[st], n + st) + rows |> emplace <| [for (i in range64(tr.config.vocab_size)); s.logits[i]] + } + return <- rows +} + +//! one prompt of `n` tokens plus the fed steps through both arms of the route lever +def private shexp_forced_feed(t : T?; n : int64) { + if (!jit_enabled()) { + t |> skip("interpreted (dasLLAMA model tests are JIT-only)") + return + } + let path = path_join(models_dir(), TWIN) + if (!model_available(t, path)) return + if (!moe_gpu_tier_installed() || !gpu_want_auto()) { + t |> skip("the GPU tier is not armed - needs DASLLAMA_GPU=1 on a device that serves the MoE tier") + return + } + set_gpu_resident_route(false) //! the per-op rails' claims: the whole-model driver would take the twin (test_gpu_resident_moe.das holds that arm) + var tr <- load_model_(path, QuantMode.q8) + set_gpu_resident_route(true) //! the lever reads at the load alone + t |> success(!moe_gpu_resident_active(), "the route lever kept the whole-model driver off the twin") + t |> success(tr.config.n_ff_shexp > 0l && !tr.config.moe_dense_shexp, "the twin carries a gated shared expert") + let layers = tr.config.n_layers + var marked = 0l + for (l in range64(layers)) { + if (moe_gpu_shexp_on_gpu(l)) { + marked++ + } + } + if (marked != layers) { + t |> skip("the shared experts are resident on {marked} of {layers} layers - the tier's budget or the shexp mark declined (the load log says which)") + delete tr + return + } + with_job_que() { + setup_dasllama_jobque_() + var prompt <- [for (i in range64(n)); 1000l + (i * 37l) % 5000l] + var toks <- [for (i in range64(FED_STEPS)); 2000l + (i * 53l) % 4000l] + set_moe_gpu_shexp_route(true) + let served0 = shexp_gpu_prefill_layers() + let span0 = span_tokens() + var sd <- create_session(tr) + var dev <- feed_logits(tr, sd, prompt, toks) + delete sd + t |> equal(shexp_gpu_prefill_layers(), served0 + layers, "the device arm served the shared expert of every layer over the {n}-token prompt") + t |> equal(span_tokens(), span0 + FED_STEPS, "the decode span served every fed step on the device arm (the shared expert inside it)") + set_moe_gpu_shexp_route(false) + var sc <- create_session(tr) + var cpu <- feed_logits(tr, sc, prompt, toks) + delete sc + set_moe_gpu_shexp_route(true) //! the tier as found + t |> equal(shexp_gpu_prefill_layers(), served0 + layers, "the CPU arm served no shared expert on the device") + t |> equal(span_tokens(), span0 + FED_STEPS, "the CPU arm's steps took the per-layer path - the span declines a shared expert off the device") + for (st in range64(long_length(cpu))) { + let diff = logits_maxdiff(dev[st], cpu[st]) + let bar = SHEXP_LOGIT_BAR_REL * logits_maxabs(cpu[st]) + let ad = argmax_of(dev[st]) + let ac = argmax_of(cpu[st]) + let what = st == 0l ? "prefill ({n} tokens)" : "step {st - 1l} (fed {toks[st - 1l]})" + to_log(LOG_INFO, "{what}: argmax cpu {ac} ({cpu[st][ac]}), device {ad} ({dev[st][ad]}); logits maxdiff {diff} (bar {bar})\n") + t |> success(diff <= bar, "{what} logits: the device shared expert within {bar} of the CPU form (maxdiff {diff})") + if (st > 0l) { //! the control: the previous step's CPU logits are a disconnected mechanism the bar must RED + let off = logits_maxdiff(dev[st], cpu[st - 1l]) + to_log(LOG_INFO, "{what}: one-step-off control maxdiff {off} (bar {bar})\n") + t |> success(off > bar, "{what} control: the bar discriminates a one-step-off compare (maxdiff {off} vs bar {bar})") + } + } + delete dev + delete cpu + delete prompt + delete toks + } + delete tr +} + +[test] +def test_gpu_moe_shexp_prefill(t : T?) { + t |> run("Qwen1.5-MoE-A2.7B-Q4_K_M: the shared expert's prefill on the device vs the CPU form, one 64-token prompt") @(t : T?) { + shexp_forced_feed(t, 64l) + } + t |> run("Qwen1.5-MoE-A2.7B-Q4_K_M: the same over a 600-token prompt (the chain's chunking)") @(t : T?) { + shexp_forced_feed(t, 600l) + } +} diff --git a/modules/dasLLAMA/tests/test_gpu_resident_moe.das b/modules/dasLLAMA/tests/test_gpu_resident_moe.das new file mode 100644 index 0000000000..18d9561265 --- /dev/null +++ b/modules/dasLLAMA/tests/test_gpu_resident_moe.das @@ -0,0 +1,224 @@ +options gen2 +options stack = 524288 // every dasLLAMA program root takes this budget (options stack does not unify up from libs) +options persistent_heap // + explicit deletes below: the model and sessions free before exit +options _dasllama_internal = true + +require dastest/testing_boost public +require dasllama/dasllama_transformer // umbrella fires each arch [init] registration (qwen2moe) +require dasllama/dasllama // create_session, moe_gpu_drop_model +require daslib/jobque_boost +require daslib/fio +require dasllama/dasllama_math // setup_dasllama_jobque_ (the engine-standard jobque config) +require dasllama/dasllama_gpu_tier // moe_gpu_tier_installed / gpu_want_auto: is the tier armed +require dasllama/dasllama_gpu_resident // moe_gpu_resident_active: did the whole-model driver take the MoE +require ?vulkan dasllama/dasllama_vulkan_common // PF_WINDOW, the resolved coopmat mode and the kernel census, where the module exists +require _model_tier // models_dir() + model_available() presence/size-tier gate + +// The whole-model resident driver on a MoE (Qwen1.5-MoE-A2.7B-Chat-Q4_K_M-local, the Q4_K_M mint of +// the Q8_0 carrier, under DASLLAMA_GPU=1): the expert stacks in the arena beside the attention +// quads and the shared experts, the window chain's routed block (the router GEMM, the per-row +// select, the device bucket schedule, the f16 gather, the expert tiles, the gated combine) and the +// token command's routed block (the router GEMV, the top-k writing the expert GEMVs' slot regions, +// the combine). The forced-feed logits-tolerance form of test_gpu_resident_hybrid: the same fixed +// tokens through the resident driver and the all-CPU chain, the prefill logits and every step's +// within the bar, the one-step-off control past it, and the census witnesses that the routed +// block ran on the device for every MoE layer (the window's per window, the token command's once +// at its recording). Two prompt lengths: one window, two windows. Three fixtures: the twin, the +// Qwen3.6-35B UD-IQ2_XXS hybrid, whose recurrent layers take the routed block after the deltanet head, +// and the Qwen3-30B UD-IQ2_XXS, the MoE with no shared expert (the residual step's add partner off). +// All three fixtures are large-tier (DASLLAMA_PARITY_FULL=1); the file is -jit only, and skips +// without the file, the armed tier, or a device with no cm2 tile family (the routed block's only +// tile); on a cm2 device the driver's admission of every fixture is asserted, never skipped. + +[init] +def private allow_cpu_prefill_intent() { + allow_cpu_prefill() // the control arm (the model dropped off the device) prefills on the CPU by design +} + +//! a fixture file and whether it carries a shared expert +struct private Fixture { + file : string + shexp : bool +} + +let private TWIN = Fixture(file = "Qwen1.5-MoE-A2.7B-Chat-Q4_K_M-local.gguf", shexp = true) //! the Q4_K_M mint of the Q8_0 carrier (its recipe row in ../performance/model_specs.das) +let private HYBRID = Fixture(file = "Qwen3.6-35B-A3B-UD-IQ2_XXS.gguf", shexp = true) //! the deltanet hybrid MoE with a gated shared expert, the 16 GB card's whole-model fit +let private NO_SHEXP = Fixture(file = "Qwen3-30B-A3B-Instruct-2507-UD-IQ2_XXS.gguf", shexp = false) //! the MoE with no shared expert: the routed block alone at the FFN site +//! the logits bar as a fraction of the step's max |logit|: the routed chain's CPU-vs-device noise class (test_gpu_moe_shexp.das's +//! reading) - the arms agree to 1e-8 at layer 0 and part from layer 1 on, where the device chain's summation order moves the +//! residual by ulps, the next layer's Q8 requant rounds a few elements the other way and the router flips a near-tie; against +//! the all-CPU chain that reads 0.07-0.11 on the twin and the 30B at 40 and 600 tokens and 0.09-0.16 on the 35B, whose +//! two-window step 4 sits on a router near-tie that any change of the device's summation order flips (0.04 under one order, +//! 0.155-0.16 under two others); the one-step-off control reads 0.42-0.95, so the bar discriminates it by 2x and more +let private MOE_LOGIT_BAR_REL = 0.20 +let private PF_WINDOW_ROWS = 512l //! the engine's PF_WINDOW, asserted equal by the two-window cell: a prompt past it runs as sequential windows +let private FED_STEPS = 6l +let private NO_CENSUS = -1l //! the census count with no vulkan module: the routed-block witnesses stand down + +//! the served tile arm, so a red names which prefill tile it came from +def private tile_arm_tag() : string { + static_if (typeinfo builtin_module_exists(vulkan)) { + return "coopmat mode {g_gpu.coopmat_mode}" + } else { + return "no vulkan module" + } +} + +def private argmax_of(v : array) : int64 { + var best = -1e30 + var bi = 0l + for (i in range64(long_length(v))) { + if (v[i] > best) { + best = v[i] + bi = i + } + } + return bi +} + +//! the coverage census's count of a routed-block kernel's dispatches so far; NO_CENSUS with no vulkan module +def private census(kernel : string) : int64 { + static_if (typeinfo builtin_module_exists(vulkan)) { + return vk_kernel_coverage_of(kernel) + } else { + return NO_CENSUS + } +} + +//! the device carries the cm2 tile family the routed block's expert tiles need (the driver declines a MoE without it) +def private device_has_cm2() : bool { + static_if (typeinfo builtin_module_exists(vulkan)) { + return g_gpu != null && g_gpu.has_coopmat2 && g_gpu.coopmat_mode == COOPMAT_CM2 + } else { + return false + } +} + +def private feed_logits(tr : Model; var s : Session; prompt : array; toks : array) : array> { + let n = long_length(prompt) + var out : array> + out |> reserve(long_length(toks) + 1l) + forward_prefill(tr, s, prompt, n, 0l) + out |> emplace <| [for (i in range64(tr.config.vocab_size)); s.logits[i]] + for (st in range64(long_length(toks))) { + forward(tr, s, toks[st], n + st) + out |> emplace <| [for (i in range64(tr.config.vocab_size)); s.logits[i]] + } + return <- out +} + +//! prefill `n` tokens + the fed steps on the resident driver, then the same on the all-CPU chain; the +//! prefill logits and every step's within the bar, the one-step-off control past it, the routed +//! block's census growing by the MoE layer count per window and per step +def private moe_forced_feed(t : T?; n : int64; fx : Fixture) { + if (!jit_enabled()) { + t |> skip("interpreted (dasLLAMA model tests are JIT-only)") + return + } + let path = path_join(models_dir(), fx.file) + if (!model_available(t, path)) return + if (!moe_gpu_tier_installed() || !gpu_want_auto()) { + t |> skip("the GPU tier is not armed - needs DASLLAMA_GPU=1 on a device that serves the whole-model driver") + return + } + let route0 = gpu_want_resident() + set_gpu_resident_route(true) //! the whole-model driver's arm, whatever lever an earlier cell left + var tr <- load_model_(path, QuantMode.q8) + set_gpu_resident_route(route0) + t |> success(tr.config.n_expert > 0l && (tr.config.n_ff_shexp > 0l) == fx.shexp, "the fixture routes experts {fx.shexp ? "beside" : "with no"} shared expert") + if (!device_has_cm2()) { + t |> skip("the device has no cm2 tile family - the whole-model driver declines a MoE by design") + delete tr + return + } + t |> success(moe_gpu_resident_active(), "the whole-model driver took {fx.file} (a decline names its reason in the load log: the plan, a seat, a plane's format)") + if (!moe_gpu_resident_active()) { + delete tr + return + } + let layers = tr.config.n_layers + var nrec = 0l + for (l in range64(layers)) { + if (layer_is_recurrent(tr.config, l)) { + nrec++ + } + } + let windows = (n + PF_WINDOW_ROWS - 1l) / PF_WINDOW_ROWS + with_job_que() { + setup_dasllama_jobque_() + var prompt <- [for (i in range64(n)); 1000l + (i * 37l) % 5000l] + var toks <- [for (i in range64(FED_STEPS)); 2000l + (i * 53l) % 4000l] + var sg <- create_session(tr, mirror_kv_dtype()) + let served0 = moe_gpu_resident_prefills() + let sched0 = census("moe_sched_cls_spv") + let rows0 = census("topk_rows_cls_spv") + let topk0 = census("topk_cls_spv") + var gpu <- feed_logits(tr, sg, prompt, toks) + t |> success(moe_gpu_resident_active(), "the whole-model driver stayed armed through the run") + t |> equal(moe_gpu_resident_prefills(), served0 + 1l, "the resident window chain served the {n}-token prompt ({nrec} recurrent layers of {layers})") + if (sched0 != NO_CENSUS) { + t |> equal(census("moe_sched_cls_spv") - sched0, layers * windows, "the device bucket schedule ran once per MoE layer per window ({windows} windows)") + t |> equal(census("topk_rows_cls_spv") - rows0, layers * windows, "the window's per-row select ran once per MoE layer per window") + //! the token command records once and resubmits: the census counts its recording, one top-k per MoE layer + let topk = census("topk_cls_spv") - topk0 + t |> success(topk >= layers && topk % layers == 0l, "the token command recorded a top-k per MoE layer ({topk} over {layers} layers)") + } + delete sg + moe_gpu_drop_model() //! every later step is the CPU chain + var sc <- create_session(tr, mirror_kv_dtype()) + var cpu <- feed_logits(tr, sc, prompt, toks) + delete sc + for (st in range64(long_length(cpu))) { + let diff = logits_maxdiff(gpu[st], cpu[st]) + let bar = MOE_LOGIT_BAR_REL * logits_maxabs(cpu[st]) + let ag = argmax_of(gpu[st]) + let ac = argmax_of(cpu[st]) + let what = (st == 0l ? "prefill ({n} tokens)" : "step {st - 1l} (fed {toks[st - 1l]})") + " [{tile_arm_tag()}]" + to_log(LOG_INFO, "{what}: argmax cpu {ac} ({cpu[st][ac]}), resident {ag} ({gpu[st][ag]}); logits maxdiff {diff} (bar {bar})\n") + t |> success(diff <= bar, "{what} logits: resident within {bar} of the CPU chain (maxdiff {diff})") + if (st > 0l) { //! the control: the previous step's CPU logits are a disconnected mechanism the bar must RED + let off = logits_maxdiff(gpu[st], cpu[st - 1l]) + to_log(LOG_INFO, "{what}: one-step-off control maxdiff {off} (bar {bar})\n") + t |> success(off > bar, "{what} control: the bar discriminates a one-step-off compare (maxdiff {off} vs bar {bar})") + } + } + delete gpu + delete cpu + delete prompt + delete toks + } + delete tr +} + +[test] +def test_gpu_resident_moe(t : T?) { + t |> run("Qwen1.5-MoE-A2.7B-Q4_K_M: the resident MoE prefill window + the routed token command vs the CPU chain, one window") @(t : T?) { + moe_forced_feed(t, 40l, TWIN) + } + t |> run("Qwen1.5-MoE-A2.7B-Q4_K_M: the same across two windows") @(t : T?) { + static_if (typeinfo builtin_module_exists(vulkan)) { + t |> equal(PF_WINDOW, PF_WINDOW_ROWS, "the prompt is sized past the engine's own window") + } + moe_forced_feed(t, PF_WINDOW_ROWS + 88l, TWIN) + } +} + +[test] +def test_gpu_resident_moe_hybrid(t : T?) { + t |> run("Qwen3.6-35B-A3B-UD-IQ2_XXS: the deltanet hybrid MoE - the recurrent head beside the routed block, one window") @(t : T?) { + moe_forced_feed(t, 40l, HYBRID) + } + t |> run("Qwen3.6-35B-A3B-UD-IQ2_XXS: the same across two windows (the conv tail and the state carry the seam)") @(t : T?) { + moe_forced_feed(t, PF_WINDOW_ROWS + 88l, HYBRID) + } +} + +[test] +def test_gpu_resident_moe_no_shexp(t : T?) { + t |> run("Qwen3-30B-A3B-UD-IQ2_XXS: the MoE with no shared expert - the routed block alone at the FFN site, one window") @(t : T?) { + moe_forced_feed(t, 40l, NO_SHEXP) + } + t |> run("Qwen3-30B-A3B-UD-IQ2_XXS: the same across two windows") @(t : T?) { + moe_forced_feed(t, PF_WINDOW_ROWS + 88l, NO_SHEXP) + } +} diff --git a/modules/dasLLAMA/tests/test_gpu_serving_declines.das b/modules/dasLLAMA/tests/test_gpu_serving_declines.das index bc2ece2ca0..ffbcaca73f 100644 --- a/modules/dasLLAMA/tests/test_gpu_serving_declines.das +++ b/modules/dasLLAMA/tests/test_gpu_serving_declines.das @@ -131,9 +131,63 @@ def test_resident_layer_decline(t : T?) { m.wq_offs[0] = -1l t |> success(says(resident_layer_decline(m, 0l, hid0, hs, kvd), "layer 0 lacks a q/k/v plane"), "missing plane named") m.ffn_w <- [64l, 96l, 64l] - t |> success(says(resident_layer_decline(m, 1l, hid0, hs, kvd), "layer 1 FFN width 96"), "FFN width named") + t |> success(says(resident_layer_decline(m, 1l, hid0, hs, kvd), "layer 1's dense FFN width 96"), "FFN width named") delete m } + t |> run("a MoE layer's dense width is its shared expert's; a layer past the dense lead without expert planes is named") @(t : T?) { + var m <- shell(3l, 0ul) + m.config.n_expert = 8l + m.config.n_expert_used = 2l + m.config.n_ff_exp = 256l + m.config.n_ff_shexp = 512l + m.config.n_layer_dense_lead = 1l + t |> success(!layer_is_moe(m, 1l), "no expert offsets at all: no layer routes") + m.we1_offs <- [-1l, 0l, 0l] + m.we3_offs <- [-1l, 0l, 0l] + m.we2_offs <- [-1l, 0l, -1l] + t |> success(!layer_is_moe(m, 0l) && layer_is_moe(m, 1l) && !layer_is_moe(m, 2l), "a layer routes only past the dense lead with all three expert planes") + t |> equal(resident_dense_width(m, 0l), 64l, "the dense-lead layer's width is its FFN's") + t |> equal(resident_dense_width(m, 1l), 512l, "an MoE layer's width is the shared expert's") + let hs = layer_head_size(m.config, 0l) + let kvd = layer_kv_dim(m.config, 0l) + t |> success(says(resident_layer_decline(m, 2l, 512l, hs, kvd), "layer 2 lacks an expert plane"), "a layer past the dense lead without its expert planes is named") + t |> success(says(resident_layer_decline(m, 0l, 512l, hs, kvd), "layer 0's dense FFN width 64 differs"), "the dense-lead layer's width differs from the shared expert's - named") + delete m + } +} + +[test] +def test_resident_unserved_moe_features(t : T?) { + t |> run("the routed block's reach: a softmax router with plain expert stacks is served, every other router shape is named") @(t : T?) { + var c = Config() + c.n_expert = 60l + c.n_expert_used = 4l + t |> equal(resident_unserved_features(c), "", "a plain softmax MoE names nothing") + c.moe_gate = MoeGate.sigmoid + t |> equal(resident_unserved_features(c), "a non-softmax router") + c.moe_gate = MoeGate.softmax + c.moe_router_bias = true + c.moe_exp_probs = true + let s = resident_unserved_features(c) + t |> success(says(s, "a router bias") && says(s, "an expert selection bias"), "the two biases named together: {s}") + c.moe_router_bias = false + c.moe_exp_probs = false + c.n_expert = 300l + c.n_expert_used = 70l + let g = resident_unserved_features(c) + t |> success(says(g, "300 experts") && says(g, "70 routed experts"), "the kernels' reach named with the counts: {g}") + c.n_expert = 60l + c.n_expert_used = 4l + c.moe_exps_bias = true + c.ffn_act = FfnAct.swiglu_oai + c.layer_out_scale = true + let e = resident_unserved_features(c) + t |> success(says(e, "biased expert stacks") && says(e, "the clamped swiglu expert activation") && says(e, "a layer output scale"), + "the expert stack's bias, the clamped activation and the layer output scale named together: {e}") + var d = Config() + d.moe_gate = MoeGate.sigmoid + t |> equal(resident_unserved_features(d), "", "a dense model's router field is no feature") + } } [test] diff --git a/modules/dasLLAMA/tests/test_kernel_coverage.das b/modules/dasLLAMA/tests/test_kernel_coverage.das index 6f40323305..965f675e1d 100644 --- a/modules/dasLLAMA/tests/test_kernel_coverage.das +++ b/modules/dasLLAMA/tests/test_kernel_coverage.das @@ -670,6 +670,44 @@ def private vk_cov_model(t : T?; path, family : string; nbs : array) { } } +//! every prefill tile family under the coopmat modes its planes have twins in - the box's resolved mode, mm, +//! sdot4, and for the q8 and q40 planes the f16 and int8 coopmat twins too; the carriers are the qwen3 files +//! (the Q8_0 also carries the resident fused qk_norm rail; the Q4_K_M k4/k6 with the Q8_K activation quant +//! forms and qk_norm on the split rail) and the 1B requants the format arc used +[unused_argument(t)] // without vulkan the gated body is empty +def private vk_cov_mode_sweep(t : T?) { + static_if (typeinfo builtin_module_exists(vulkan)) { + let vehicles = fixed_array( + (file = "Qwen3-0.6B-Q8_0.gguf", family = "qwen3", nmodes = 5), + (file = "Llama-3.2-1B-Instruct-Q4_0-local.gguf", family = "llama", nmodes = 5), + (file = "Qwen3-4B-Instruct-2507-Q4_K_M.gguf", family = "qwen3", nmodes = 3), // k4 + k6 + (file = "Llama-3.2-1B-Instruct-IQ4_XS.gguf", family = "llama", nmodes = 3), + (file = "Llama-3.2-1B-Instruct-Q3_K_L.gguf", family = "llama", nmodes = 3), // k3 + k5 + (file = "Llama-3.2-1B-Instruct-IQ3_M.gguf", family = "llama", nmodes = 3), // iq3s + (file = "Llama-3.2-1B-Instruct-IQ3_XXS-local.gguf", family = "llama", nmodes = 3), + (file = "Llama-3.2-1B-Instruct-IQ4_NL-local.gguf", family = "llama", nmodes = 3), + (file = "Llama-3.2-1B-Instruct-Q2_K-local.gguf", family = "llama", nmodes = 3), + (file = "Llama-3.2-1B-Instruct-IQ2_XS-local.gguf", family = "llama", nmodes = 3), + (file = "Llama-3.2-1B-Instruct-IQ2_XXS-local.gguf", family = "llama", nmodes = 3), + (file = "Llama-3.2-1B-Instruct.i1-IQ3_XXS.gguf", family = "llama", nmodes = 3)) // iq2s on the attention planes + if (!vk_moe_init()) { //! the sweep reads the device's resolved mode before its first row loads (a family filter may have skipped every earlier row) + t |> skip("no Vulkan device for the coopmat mode sweep") + return + } + let none : array + let box_mode = g_gpu.coopmat_mode + let modes = fixed_array(box_mode, COOPMAT_MM, COOPMAT_SDOT4, COOPMAT_F16, COOPMAT_INT8) + for (v in vehicles) { + let vpath = path_join(models_dir(), v.file) + for (mi in range(v.nmodes)) { + g_gpu.coopmat_mode = modes[mi] + vk_cov_model(t, vpath, v.family, none) + } + g_gpu.coopmat_mode = box_mode + } + } +} + [test] def test_vulkan_kernel_coverage_serving(tst : T?) { tst |> run("Vulkan serving census: the local zoo across resident/arena/kq/deltanet/MoE") <| @(t : T?) { @@ -683,34 +721,7 @@ def test_vulkan_kernel_coverage_serving(tst : T?) { vk_cov_model(t, path_join(models_dir(), "tinyllama-1.1b-chat-v1.0.Q8_0.gguf"), "llama", [2]) // gemma3 q8: the per-op arena rails + qk_norm vk_cov_model(t, path_join(models_dir(), "gemma-3-1b-it-Q8_0.gguf"), "gemma3", none) - // every prefill tile family under the coopmat modes its planes have twins in - the - // box's resolved mode, mm, sdot4, and for the q8 and q40 planes the f16 and int8 - // coopmat twins too; the carriers are the qwen3 files (the Q8_0 also carries the - // resident fused qk_norm rail; the Q4_K_M k4/k6 with the Q8_K activation quant forms - // and qk_norm on the split rail) and the 1B requants the format arc used - let vehicles = fixed_array( - (file = "Qwen3-0.6B-Q8_0.gguf", family = "qwen3", nmodes = 5), - (file = "Llama-3.2-1B-Instruct-Q4_0-local.gguf", family = "llama", nmodes = 5), - (file = "Qwen3-4B-Instruct-2507-Q4_K_M.gguf", family = "qwen3", nmodes = 3), // k4 + k6 - (file = "Llama-3.2-1B-Instruct-IQ4_XS.gguf", family = "llama", nmodes = 3), - (file = "Llama-3.2-1B-Instruct-Q3_K_L.gguf", family = "llama", nmodes = 3), // k3 + k5 - (file = "Llama-3.2-1B-Instruct-IQ3_M.gguf", family = "llama", nmodes = 3), // iq3s - (file = "Llama-3.2-1B-Instruct-IQ3_XXS-local.gguf", family = "llama", nmodes = 3), - (file = "Llama-3.2-1B-Instruct-IQ4_NL-local.gguf", family = "llama", nmodes = 3), - (file = "Llama-3.2-1B-Instruct-Q2_K-local.gguf", family = "llama", nmodes = 3), - (file = "Llama-3.2-1B-Instruct-IQ2_XS-local.gguf", family = "llama", nmodes = 3), - (file = "Llama-3.2-1B-Instruct-IQ2_XXS-local.gguf", family = "llama", nmodes = 3), - (file = "Llama-3.2-1B-Instruct.i1-IQ3_XXS.gguf", family = "llama", nmodes = 3)) // iq2s on the attention planes - let box_mode = g_gpu.coopmat_mode - let modes = fixed_array(box_mode, COOPMAT_MM, COOPMAT_SDOT4, COOPMAT_F16, COOPMAT_INT8) - for (v in vehicles) { - let vpath = path_join(models_dir(), v.file) - for (mi in range(v.nmodes)) { - g_gpu.coopmat_mode = modes[mi] - vk_cov_model(t, vpath, v.family, none) - } - g_gpu.coopmat_mode = box_mode - } + vk_cov_mode_sweep(t) // qwen3.5 q8: the DeltaNet chains (conv/step/scan + the decode step) vk_cov_model(t, path_join(models_dir(), "Qwen3.5-0.8B-Q8_0.gguf"), "qwen35", none) // qwen3.5 9B (large tier): head-256 gated attention on the cm2 flash stamps, the @@ -719,6 +730,9 @@ def test_vulkan_kernel_coverage_serving(tst : T?) { // the MoE rows (expert FFN/qkv chains + heat cache) sit above the small tier vk_cov_model(t, path_join(models_dir(), "Qwen1.5-MoE-A2.7B-Chat.Q8_0.gguf"), "qwen2moe", none) vk_cov_model(t, path_join(models_dir(), "Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf"), "qwen3moe", none) + // the Q4_K_M twin fits a 16 GB card whole: the whole-model driver's MoE block (the router GEMM, the + // per-row select, the bucket schedule, the combine-folded residual step, the token command's routed block) + vk_cov_model(t, path_join(models_dir(), "Qwen1.5-MoE-A2.7B-Chat-Q4_K_M-local.gguf"), "qwen2moe", none) vk_kernel_coverage_report() var cov <- vk_kernel_coverage() var served = 0 diff --git a/modules/dasLLAMA/tests/test_mtp.das b/modules/dasLLAMA/tests/test_mtp.das index ea0f9f0dcd..ad4c4da00f 100644 --- a/modules/dasLLAMA/tests/test_mtp.das +++ b/modules/dasLLAMA/tests/test_mtp.das @@ -259,7 +259,10 @@ def test_mtp_invariance_27b(t : T?) { // GLM-4.5-Air Q4_K_M — the only NON-RECURRENT MTP model we have, so this is the sole arm covering // mtp_spec_eval's `recr_mask == 0` reject branch (a rejected draft consumes the verify's own row-0 // logits + h0 instead of paying a second full forward; every qwen35 MTP file is a deltanet hybrid -// and takes the restore-and-re-advance branch instead). It is also the first arch combining +// and takes the restore-and-re-advance branch instead). The code fixture's own reject count rides +// the kernels' summation order (it has read zero), so the arm reaches the branch through the +// engine's seam - every third draft rejected whatever the verify said - and holds the stream to +// plain decode there too. It is also the first arch combining // attn_qkv_bias with a NextN block — the shape where the qkv/out bias arrays were sized by the // TRUNK layer count, so blk..attn_{q,k,v}.bias never loaded and the draft read past the // end of t.bq. And glm4moe ships nextn.embed_tokens/shared_head_head as copies of the trunk's own @@ -302,10 +305,16 @@ def test_mtp_invariance_glm(t : T?) { let ref_ids <- gen_plain(tr, code, total) let spec_ids <- gen_spec(tr, code, total, nd, na) t |> success(same(ref_ids, spec_ids), "code-glm: spec decode token-for-token vs plain") - t |> success(nd > 0l, "code-glm: the draft head actually drafted ({nd} drafts)") - // a 100%-accept run would leave the reject branch untouched and the arm would prove - // nothing about it — so require that at least one draft was actually rejected - t |> success(na < nd, "code-glm: {nd - na} of {nd} drafts rejected - reject branch covered") + t |> success(nd > 0l, "code-glm: the draft head actually drafted ({nd} drafts, {nd - na} rejected on its own)") + // the reject branch on demand: every third draft rejected, the stream still plain decode's + set_mtp_force_reject_every(3l) + var nd_forced = 0l + var na_forced = 0l + let forced_ids <- gen_spec(tr, code, total, nd_forced, na_forced) + set_mtp_force_reject_every(0l) + t |> success(same(ref_ids, forced_ids), "code-glm forced: spec decode through the reject branch token-for-token vs plain") + t |> success(nd_forced >= 3l && nd_forced - na_forced >= nd_forced / 3l, + "code-glm forced: {nd_forced - na_forced} of {nd_forced} drafts rejected - reject branch covered") } delete tr } diff --git a/modules/dasLLAMA/tests/test_parity.das b/modules/dasLLAMA/tests/test_parity.das index bbc64eda69..73b4f14b78 100644 --- a/modules/dasLLAMA/tests/test_parity.das +++ b/modules/dasLLAMA/tests/test_parity.das @@ -1,4 +1,5 @@ options gen2 +options stack = 524288 // the test file is the program root under dastest; the gpt-oss forward (mx4 experts, sinks, biases) overflows the default budget options persistent_heap // + explicit deletes below: each arm frees its model/session before the next loads, so multi-GB weights never accumulate in the process options _dasllama_internal = true diff --git a/modules/dasLLAMA/tests/test_vulkan_dec_tail.das b/modules/dasLLAMA/tests/test_vulkan_dec_tail.das index beb11b63f8..acfb6e390d 100644 --- a/modules/dasLLAMA/tests/test_vulkan_dec_tail.das +++ b/modules/dasLLAMA/tests/test_vulkan_dec_tail.das @@ -12,6 +12,7 @@ require dasllama/dasllama_common // MoeGate, select_decode_override require dasllama/dasllama_blocks // registers the vulkan_moe_span decode override at init require dasllama/dasllama_moe // moe_select_core: the host select the span's device top-k mirrors require ?vulkan dasllama/dasllama_math_vulkan // the decode attention block + FFN tail under test; the test installs the hooks explicitly (no env gate) +require ?vulkan dasllama/dasllama_vulkan_decode // vk_span_reset: the span records once per model, so the second span cell forgets the first's record require daslib/f16_cvt require math require _vk_kq_fixtures @@ -222,7 +223,7 @@ def private quad_up(var q : Quad; fmt : int; base : int64; seed : int64; phase : // the expert stacks: gate/up k4 over N, down k6 over NFE; expert e's plane starts at e * NFE * N; // slot j routes expert j -def private ffn_up(t : T?; var f : Ffn) : bool { +def private ffn_up(t : T?; var f : Ffn; base_slot : int64 = 30l) : bool { var wq1 : array var ws1 : array var wq3 : array @@ -232,9 +233,9 @@ def private ffn_up(t : T?; var f : Ffn) : bool { fill_kq_stack(1, NE * NFE, N, 41l, true, wq1, ws1) fill_kq_stack(1, NE * NFE, N, 43l, true, wq3, ws3) fill_kq_stack(3, NE * N, NFE, 47l, false, wq2, ws2) - let woff1 = 30l * STRIDE - let woff3 = 31l * STRIDE - let woff2 = 32l * STRIDE + let woff1 = base_slot * STRIDE + let woff3 = (base_slot + 1l) * STRIDE + let woff2 = (base_slot + 2l) * STRIDE if (!moe_gpu_upload_stack(wq1, ws1, woff1, N, NE * NFE, 1)) { return false } @@ -252,6 +253,34 @@ def private ffn_up(t : T?; var f : Ffn) : bool { return true } +//! a shared expert's q8 triple (dequantized for the reference), its plane offsets and the two +//! layers' gate rows; width NFE like the routed experts +struct private Shexp { + w1 : array + w3 : array + w2 : array + off1 : int64 + off3 : int64 + off2 : int64 + gate : array //! 2 x N, layer-major +} + +def private shexp_up(t : T?; var sh : Shexp; base : int64) : bool { + sh.off1 = base + sh.off3 = base + STRIDE + sh.off2 = base + 2l * STRIDE + if (!plane_up(0, NFE, N, 71l, sh.off1, sh.w1)) { + return false + } + t |> success(plane_up(0, NFE, N, 73l, sh.off3, sh.w3), "dec span shexp: the shared up plane uploaded") + t |> success(plane_up(0, N, NFE, 79l, sh.off2, sh.w2), "dec span shexp: the shared down plane uploaded") + sh.gate |> resize(2l * N) + for (i in range64(2l * N)) { + sh.gate[i] = 0.03 * sin(float(i) * 0.47 + 0.9) + 0.02 * cos(float(i / N) * 1.7) + } + return true +} + // the block's reference for one position: the feed, q/k/v, rope, the mirror row, attention, wo def private block_ref(var q : Quad; x : array; pos : int64; cossin : array; var xb2 : array; var kraw : array) { var xd : array @@ -545,6 +574,83 @@ def private span_args(qa, qb : Quad; f : Ffn) : MoeSpanDec { return <- a } +//! the span's arguments with a shared expert beside the routed pair (both layers share its triple), gated or not +def private span_args_sh(qa, qb : Quad; f : Ffn; sh : Shexp; gated : bool) : MoeSpanDec { + var a <- span_args(qa, qb, f) + a.nsh = NFE + a.sh_gated = gated + a.wsh1 <- [sh.off1, sh.off1] + a.wsh3 <- [sh.off3, sh.off3] + a.wsh2 <- [sh.off2, sh.off2] + return <- a +} + +//! one span layer's reference with a shared expert: span_layer_ref's steps, the shared gate on the +//! same f32 normed row, then the shared FFN on the Q8_0 feed (its hidden row requantized Q8_0 as +//! the q8 down GEMV reads it) scaled by sigmoid(gate) - or by one, ungated - onto the residual +def private span_layer_ref_sh(var q : Quad; f : Ffn; sh : Shexp; router : array; li : int64; var x : array; pos : int64; cossin : array; var kraw : array; gated : bool) { + var xb2 : array + block_ref(q, x, pos, cossin, xb2, kraw) + for (i in range64(N)) { + x[i] += xb2[i] + } + var xb : array + xb |> resize(N) + rmsnorm(xb, x, f.norm, N, EPS) + var lg : array + lg |> resize(NE) + for (e in range64(NE)) { + var acc = 0.0 + for (i in range64(N)) { + acc += router[(li * NE + e) * N + i] * xb[i] + } + lg[e] = acc + } + var gl = 0.0 + for (i in range64(N)) { + gl += sh.gate[li * N + i] * xb[i] + } + let g = gated ? 1.0 / (1.0 + exp(-gl)) : 1.0 + var idx : array + var w : array + idx |> resize(K) + w |> resize(K) + unsafe { + moe_select_core(MoeGate.softmax, NE, K, true, 1.0, addr(lg[0]), addr(idx[0]), addr(w[0]), 0l) + } + var fd : array + var gxq1 : array + var gxs1 : array + feed_ref(x, f.norm, 1, fd, gxq1, gxs1) + var fd0 : array + var xq0 : array + var xs0 : array + feed_ref(x, f.norm, 0, fd0, xq0, xs0) + ffn_ref_sel(f, fd, idx, w, x) + var gg : array + var u : array + gemv_ref(sh.w1, NFE, N, fd0, gg) + gemv_ref(sh.w3, NFE, N, fd0, u) + var hrow : array + hrow |> resize(NFE) + for (i in range64(NFE)) { + hrow[i] = silu(gg[i]) * u[i] + } + var hq : array + var hsc : array + hq |> resize(NFE) + hsc |> resize(NFE / 32l) + quantize_q8_0_into(hrow, NFE, hq, hsc, 0l, 0l) + for (i in range64(NFE)) { + hrow[i] = float(hq[i]) * hsc[i / 32l] + } + var d : array + gemv_ref(sh.w2, N, NFE, hrow, d) + for (i in range64(N)) { + x[i] += g * d[i] + } +} + [test] def test_vulkan_dec_span_cache(t : T?) { t |> run("dec span: the cached arguments follow the carrier, not only the offsets") @(t : T?) { @@ -650,6 +756,95 @@ def test_vulkan_dec_span(t : T?) { } } +//! the span with a gated shared expert: the routed slots plus sigmoid(gate) times the shared down +//! row in one combine, the shared feed off the same normed row, one submit per position +[test] +def test_vulkan_dec_span_shexp(t : T?) { // nolint:STYLE038 — one span cell: the fixture, the hooks, two positions and the two controls, in the plain span cell's shape + static_if (typeinfo builtin_module_exists(vulkan)) { + vk_span_reset() //! the span records once per model: this cell is the second model in the process + install_moe_gpu_tier(@@vk_moe_ffn, @@vk_moe_upload_stack, @@vk_moe_cls, @@vk_moe_dense, @@vk_moe_drop_stacks, @@vk_moe_dn, @@vk_moe_attn) + set_moe_gpu_attn_dec_hooks(@@vk_moe_attn_dec, @@vk_moe_attn_dec_ok, @@vk_moe_attn_dec_claim, @@vk_moe_attn_dec_hydrate) + set_moe_gpu_span_dec_hook(@@vk_moe_span_dec) + //! bases past every other cell's (the tail pairs sit at 10-23, 40-53 and 80-93 x STRIDE, the + //! routed stacks at 30-32, the plain span's quads at 60-73): first-hit stack lookup + var f : Ffn + if (!ffn_up(t, f, 100l)) { + t |> skip("dec span shexp: no Vulkan device (or probe/budget refused the upload)") + return + } + var sh : Shexp + t |> success(shexp_up(t, sh, 103l * STRIDE), "dec span shexp: the shared gate plane uploaded") + var qa : Quad + var qb : Quad + t |> success(quad_up(qa, 1, 110l * STRIDE, 51l, 0.4, true, false), "dec span shexp: layer A's quad uploaded") + t |> success(quad_up(qb, 1, 120l * STRIDE, 63l, 0.8, true, false), "dec span shexp: layer B's quad uploaded") + var router : array + var att_norms : array + var ffn_norms : array + span_planes(qa, qb, f, router, att_norms, ffn_norms) + var no_freqs : array + var cossin : array + cossin |> resize(HS) + for (gated in fixed_array(true, false)) { + vk_span_reset() //! the record keys on the shared expert's gate: the ungated arm is a new span + let tag = gated ? "gated" : "ungated" + let gen = moe_gpu_attn_dec_claim(true) + matmul_moe_gpu_attn_dec_hydrate(qa.woq, null, null, 0l, KVD, true) + matmul_moe_gpu_attn_dec_hydrate(qb.woq, null, null, 0l, KVD, true) + t |> success(moe_gpu_attn_dec_ok(qa.woq, 0l, gen) == DAT_SERVE && moe_gpu_attn_dec_ok(qb.woq, 0l, gen) == DAT_SERVE, + "dec span shexp ({tag}): both layers serve position 0") + var a <- span_args_sh(qa, qb, f, sh, gated) + var kout : array + var vout : array + kout |> resize(2l * KVD) + vout |> resize(2l * KVD) + for (pos in range64(2)) { + build_rope_row_packed(cossin, 0l, pos, HS, 10000.0, 1.0, 1.0, no_freqs) + var x : array + fill_row(x, N, 0.5 + 0.7 * float(pos)) + var x_ref := x + var xd : array + var xq : array + var xs : array + feed_ref(x, qa.norm, 1, xd, xq, xs) + unsafe { + a.pos = pos + a.router = addr(router[0]) + a.shgate = addr(sh.gate[0]) + a.att_norm = addr(att_norms[0]) + a.ffn_norm = addr(ffn_norms[0]) + a.xq = addr(xq[0]) + a.xs = addr(xs[0]) + a.cossin = addr(cossin[0]) + a.x = addr(x[0]) + a.kout = addr(kout[0]) + a.vout = addr(vout[0]) + } + matmul_moe_gpu_span_dec(a) + var kraw_a : array + var kraw_b : array + span_layer_ref_sh(qa, f, sh, router, 0l, x_ref, pos, cossin, kraw_a, gated) + span_layer_ref_sh(qb, f, sh, router, 1l, x_ref, pos, cossin, kraw_b, gated) + check_row(t, "span shexp ({tag}) pos {pos} residual row after both layers", x, x_ref) + if (pos == 0l) { + var poison := x + poison[0] += 1.0 + 2.0 * max_abs(x_ref) + t |> success(count_bad(poison, x_ref, ABS_BAR, REL_BAR * max_abs(x_ref)) == 1, "span shexp ({tag}): the bars catch a poisoned element") + var x_no : array + fill_row(x_no, N, 0.5) + var ka : array + var kb : array + span_layer_ref(qa, f, router, 0l, x_no, 0l, cossin, ka) + span_layer_ref(qb, f, router, 1l, x_no, 0l, cossin, kb) + t |> success(count_bad(x, x_no, ABS_BAR, REL_BAR * max_abs(x_no)) > 0, "span shexp ({tag}): the reference without the shared expert misses the device row") + } + } + } + } else { + t |> skip("dasVulkan not installed; the decode span is untestable here") + } +} + [test] def test_vulkan_dec_tail(t : T?) { static_if (typeinfo builtin_module_exists(vulkan)) { diff --git a/modules/dasLLAMA/tests/test_vulkan_kernels.das b/modules/dasLLAMA/tests/test_vulkan_kernels.das index c0f77aba4c..8d493171d4 100644 --- a/modules/dasLLAMA/tests/test_vulkan_kernels.das +++ b/modules/dasLLAMA/tests/test_vulkan_kernels.das @@ -18,6 +18,10 @@ require ?vulkan dasllama/dasllama_vulkan_seams require ?vulkan dasllama/dasllama_kqformat require ?vulkan dasllama/dasllama_gemm_schema // kq_dev_ssb_i - the fixtures size their device scale planes per format require ?vulkan daslib/shader_lingua_franca // gl_GlobalInvocationID — the CPU replay drives it per lane (only vulkan-guarded arms replay) +require dasllama/dasllama_common // MoeGate: the routing cells' select oracle takes the softmax router +require dasllama/dasllama_moe // moe_select_core: the host select the window's per-row top-k mirrors + +let private TILE_SLACK_ROWS = int(TILE_READ_SLACK) //! the cm2 fixtures' rows past the last region, the engine's count: the s and m tiles load a partial column unclamped // The model-less Vulkan kernel-unit suite ([vk_dispatch] bring-up): a class kernel dispatched // through the GENERATED surface (ensure/set/enc — per-class layout, push constants, hazard rail) @@ -749,13 +753,13 @@ def test_vkd_kq_gemv_family(t0 : T?) { var host = make_host_buf(y_bytes, true, [cached = true]) var y_cls : array y_cls |> resize(rows) - let wgs = (int64(rows) + g_gpu.rows_per_wg - 1l) / g_gpu.rows_per_wg var regs : array regs |> resize(nreg * 2) for (r in range(nreg)) { regs[r * 2] = uint(r * d * nsb) regs[r * 2 + 1] = 0u } + var y_ref <- kq_gemv_oracle(fmt, wqh, wsh, xqh, xsh, regs, 512, d, nreg) unsafe { upload_region_at(wqd, 0l, addr(wqh[0]), wq_bytes) upload_region_at(wsd, 0l, addr(wsh[0]), ws_bytes) @@ -766,47 +770,52 @@ def test_vkd_kq_gemv_family(t0 : T?) { let cs = fixed_array(wq_bytes, ws_bytes, int64(nreg) * 8l, xq_bytes, xs_bytes, y_bytes) let cg = fixed_array(1u, 2u, 4u, 8u, 16u, 32u) var s_cls = gemv_cls_set(fmt, cb, cs, cg) - var raw2 = alloc_cmd() - let begin = VkCommandBufferBeginInfo() - vk_check(vkBeginCommandBuffer(raw2, begin), null) - var h2 : VkHaz - var pc = GemvArgs(n = 512u, d = uint(d), nreg = uint(nreg), ybase = 0u) - if (fmt == int(KqFmt.k4)) { - enc_kq_gemv_k4_cls(raw2, h2, s_cls, pc, wgs) - } elif (fmt == int(KqFmt.q40)) { - enc_kq_gemv_q40_cls(raw2, h2, s_cls, pc, wgs) - } elif (fmt == int(KqFmt.k5)) { - enc_kq_gemv_k5_cls(raw2, h2, s_cls, pc, wgs) - } elif (fmt == int(KqFmt.iq4xs)) { - enc_kq_gemv_iq4xs_cls(raw2, h2, s_cls, pc, wgs) - } elif (fmt == int(KqFmt.k3)) { - enc_kq_gemv_k3_cls(raw2, h2, s_cls, pc, wgs) - } elif (fmt == int(KqFmt.iq3s)) { - enc_kq_gemv_iq3s_cls(raw2, h2, s_cls, pc, wgs) - } elif (fmt == int(KqFmt.iq3xxs)) { - enc_kq_gemv_iq3xxs_cls(raw2, h2, s_cls, pc, wgs) - } elif (fmt == int(KqFmt.iq4nl)) { - enc_kq_gemv_iq4nl_cls(raw2, h2, s_cls, pc, wgs) - } elif (fmt == int(KqFmt.k2)) { - enc_kq_gemv_k2_cls(raw2, h2, s_cls, pc, wgs) - } elif (fmt == int(KqFmt.iq2s)) { - enc_kq_gemv_iq2s_cls(raw2, h2, s_cls, pc, wgs) - } elif (fmt == int(KqFmt.iq2xs)) { - enc_kq_gemv_iq2xs_cls(raw2, h2, s_cls, pc, wgs) - } elif (fmt == int(KqFmt.iq2xxs)) { - enc_kq_gemv_iq2xxs_cls(raw2, h2, s_cls, pc, wgs) - } else { - enc_kq_gemv_k6_cls(raw2, h2, s_cls, pc, wgs) + //! the three lane splits of a subgroup: one row, two rows, four rows (the 16-block row keeps every lane busy at eight) + for (lanes_per_row in fixed_array(0u, 16u, 8u)) { + let wgs = gemv_wgs(int64(rows), lanes_per_row) + nan_sentinel_upload(yd2, y_bytes) //! a lane split that stores no row leaves the sentinel, not the last split's rows + var raw2 = alloc_cmd() + let begin = VkCommandBufferBeginInfo() + vk_check(vkBeginCommandBuffer(raw2, begin), null) + var h2 : VkHaz + var pc = GemvArgs(n = 512u, d = uint(d), nreg = uint(nreg), ybase = 0u, lanes_per_row = lanes_per_row) + if (fmt == int(KqFmt.k4)) { + enc_kq_gemv_k4_cls(raw2, h2, s_cls, pc, wgs) + } elif (fmt == int(KqFmt.q40)) { + enc_kq_gemv_q40_cls(raw2, h2, s_cls, pc, wgs) + } elif (fmt == int(KqFmt.k5)) { + enc_kq_gemv_k5_cls(raw2, h2, s_cls, pc, wgs) + } elif (fmt == int(KqFmt.iq4xs)) { + enc_kq_gemv_iq4xs_cls(raw2, h2, s_cls, pc, wgs) + } elif (fmt == int(KqFmt.k3)) { + enc_kq_gemv_k3_cls(raw2, h2, s_cls, pc, wgs) + } elif (fmt == int(KqFmt.iq3s)) { + enc_kq_gemv_iq3s_cls(raw2, h2, s_cls, pc, wgs) + } elif (fmt == int(KqFmt.iq3xxs)) { + enc_kq_gemv_iq3xxs_cls(raw2, h2, s_cls, pc, wgs) + } elif (fmt == int(KqFmt.iq4nl)) { + enc_kq_gemv_iq4nl_cls(raw2, h2, s_cls, pc, wgs) + } elif (fmt == int(KqFmt.k2)) { + enc_kq_gemv_k2_cls(raw2, h2, s_cls, pc, wgs) + } elif (fmt == int(KqFmt.iq2s)) { + enc_kq_gemv_iq2s_cls(raw2, h2, s_cls, pc, wgs) + } elif (fmt == int(KqFmt.iq2xs)) { + enc_kq_gemv_iq2xs_cls(raw2, h2, s_cls, pc, wgs) + } elif (fmt == int(KqFmt.iq2xxs)) { + enc_kq_gemv_iq2xxs_cls(raw2, h2, s_cls, pc, wgs) + } else { + enc_kq_gemv_k6_cls(raw2, h2, s_cls, pc, wgs) + } + vhz_dep(raw2, h2, 32u, 0u, true) + cmd_copy_whole(raw2, yd2, host.buf, y_bytes) + vk_check(vkEndCommandBuffer(raw2), null) + submit_wait(raw2) + memcpy(addr(y_cls[0]), host.mapped, y_bytes) + let bad = mismatch_approx(y_cls, y_ref) + let lanes = lanes_per_row == 0u ? g_gpu.subgroup_size : int64(lanes_per_row) + t |> success(bad == 0, "fmt {fmt} at {lanes} lanes per row: class kernel matches the CPU oracle ({bad} of {rows} rows off)") } - vhz_dep(raw2, h2, 32u, 0u, true) - cmd_copy_whole(raw2, yd2, host.buf, y_bytes) - vk_check(vkEndCommandBuffer(raw2), null) - submit_wait(raw2) - memcpy(addr(y_cls[0]), host.mapped, y_bytes) } - var y_ref <- kq_gemv_oracle(fmt, wqh, wsh, xqh, xsh, regs, 512, d, nreg) - let bad = mismatch_approx(y_cls, y_ref) - t |> success(bad == 0, "fmt {fmt}: class kernel matches the CPU oracle ({bad} of {rows} rows off)") if (fmt == int(KqFmt.iq4xs)) { // the codebook pack has an independent float witness var y_flt <- iq4xs_gemv_float_oracle(wqh, wsh, xqh, xsh, regs, 512, d, nreg) let badf = mismatch_approx(y_ref, y_flt) @@ -2064,7 +2073,7 @@ def test_vkd_k4_cm2_batch(t0 : T?) { var xfh : array wqh |> resize(totsb * 32) wsuh |> resize(totsb * 5) - xfh |> resize((rows + 32) * n / 2) // + the s tile's 32-row read slack + xfh |> resize((rows + TILE_SLACK_ROWS) * n / 2) for (i in range(totsb * 32)) { wqh[i] = hash_word(uint(i) + 29u) // arbitrary nibble bytes } @@ -2076,12 +2085,12 @@ def test_vkd_k4_cm2_batch(t0 : T?) { wsuh[sb * 5 + 1 + wi] = hash_word(uint(sb * 4 + wi) + 613u) // sc / mn bytes } } - for (i in range((rows + 32) * n / 2)) { + for (i in range((rows + TILE_SLACK_ROWS) * n / 2)) { xfh[i] = ws_word(uint(i) * 3u + 5u) // f16 activation pairs } let wq_bytes = int64(totsb) * 128l let ws_bytes = int64(length(wsuh)) * 4l - let xf_bytes = int64((rows + 32) * n) * 2l + let xf_bytes = int64((rows + TILE_SLACK_ROWS) * n) * 2l let y_bytes = int64(rows * d) * 4l let wqd = make_device_buf(wq_bytes) let wsd = make_device_buf(ws_bytes) @@ -2102,8 +2111,11 @@ def test_vkd_k4_cm2_batch(t0 : T?) { let wgs0 = ((cnt0 + ttile - 1) / ttile) * wtiles let wgs1 = ((cnt1 + ttile - 1) / ttile) * wtiles let wgs = wgs0 + wgs1 + //! two map words past the live workgroups carry the sentinel: the dispatch runs over them as the + //! device-written schedules' upper bound does, and a tile that reads one returns before its first store + let tail = 2 var sched : array - sched |> resize(2 * 4 + wgs) + sched |> resize(2 * 4 + wgs + tail) sched[0] = 0u sched[1] = 0u sched[2] = uint(cnt0) @@ -2115,12 +2127,15 @@ def test_vkd_k4_cm2_batch(t0 : T?) { for (w in range(wgs)) { sched[8 + w] = w < wgs0 ? 0u : 1u } + for (w in range(wgs, wgs + tail)) { + sched[8 + w] = SCHED_NONE + } unsafe { upload_region_at(wqd, 0l, addr(wqh[0]), wq_bytes) upload_region_at(wsd, 0l, addr(wsuh[0]), ws_bytes) upload_region_at(xfd, 0l, addr(xfh[0]), xf_bytes) nan_sentinel_upload(yd2, y_bytes) - let sc_bytes = int64(2 * 4 + wgs) * 4l + let sc_bytes = int64(2 * 4 + wgs + tail) * 4l let scd = make_device_buf(sc_bytes) upload_region_at(scd, 0l, addr(sched[0]), sc_bytes) let bufs = fixed_array(wqd, wsd, scd, xfd, yd2) @@ -2136,13 +2151,13 @@ def test_vkd_k4_cm2_batch(t0 : T?) { var h2 : VkHaz var pc = BatchArgs(n = uint(n), d = uint(d), map_off = uint(2 * 4)) if (ml == 0) { - enc_kq_batch_k4_cm2l_cls(raw2, h2, sc, pc, int64(wgs)) + enc_kq_batch_k4_cm2l_cls(raw2, h2, sc, pc, int64(wgs + tail)) } elif (ml == 1) { - enc_kq_batch_k4_cm2m_cls(raw2, h2, sc, pc, int64(wgs)) + enc_kq_batch_k4_cm2m_cls(raw2, h2, sc, pc, int64(wgs + tail)) } elif (ml == 2) { - enc_kq_batch_k4_cm2s_cls(raw2, h2, sc, pc, int64(wgs)) + enc_kq_batch_k4_cm2s_cls(raw2, h2, sc, pc, int64(wgs + tail)) } else { - khr_cls_enc(int(KqFmt.k4), raw2, h2, sc, pc, int64(wgs)) + khr_cls_enc(int(KqFmt.k4), raw2, h2, sc, pc, int64(wgs + tail)) } vhz_dep(raw2, h2, 16u, 0u, true) cmd_copy_whole(raw2, yd2, host.buf, y_bytes) @@ -2203,7 +2218,7 @@ def test_vkd_k5_cm2_batch(t0 : T?) { var xfh : array wqh |> resize(totsb * 40) wsuh |> resize(totsb * 5) - xfh |> resize((rows + 32) * n / 2) // + the s tile's 32-row read slack + xfh |> resize((rows + TILE_SLACK_ROWS) * n / 2) for (i in range(totsb * 40)) { wqh[i] = hash_word(uint(i) + 29u) // arbitrary quant bytes } @@ -2215,12 +2230,12 @@ def test_vkd_k5_cm2_batch(t0 : T?) { wsuh[sb * 5 + 1 + wi] = hash_word(uint(sb * 4 + wi) + 613u) // sc / mn bytes } } - for (i in range((rows + 32) * n / 2)) { + for (i in range((rows + TILE_SLACK_ROWS) * n / 2)) { xfh[i] = ws_word(uint(i) * 3u + 5u) // f16 activation pairs } let wq_bytes = int64(totsb) * 160l let ws_bytes = int64(length(wsuh)) * 4l - let xf_bytes = int64((rows + 32) * n) * 2l + let xf_bytes = int64((rows + TILE_SLACK_ROWS) * n) * 2l let y_bytes = int64(rows * d) * 4l let wqd = make_device_buf(wq_bytes) let wsd = make_device_buf(ws_bytes) @@ -2342,7 +2357,7 @@ def test_vkd_q40_cm2_batch(t0 : T?) { var xfh : array wqh |> resize(totsb * 32) wsuh |> resize(totsb * 5) - xfh |> resize((rows + 32) * n / 2) // + the s tile's 32-row read slack + xfh |> resize((rows + TILE_SLACK_ROWS) * n / 2) for (i in range(totsb * 32)) { wqh[i] = hash_word(uint(i) + 29u) // arbitrary quant bytes } @@ -2353,12 +2368,12 @@ def test_vkd_q40_cm2_batch(t0 : T?) { } wsuh[sb * 5 + 4] = 0u } - for (i in range((rows + 32) * n / 2)) { + for (i in range((rows + TILE_SLACK_ROWS) * n / 2)) { xfh[i] = ws_word(uint(i) * 3u + 5u) // f16 activation pairs } let wq_bytes = int64(totsb) * 128l let ws_bytes = int64(length(wsuh)) * 4l - let xf_bytes = int64((rows + 32) * n) * 2l + let xf_bytes = int64((rows + TILE_SLACK_ROWS) * n) * 2l let y_bytes = int64(rows * d) * 4l let wqd = make_device_buf(wq_bytes) let wsd = make_device_buf(ws_bytes) @@ -2480,7 +2495,7 @@ def test_vkd_k2_cm2_batch(t0 : T?) { var xfh : array wqh |> resize(totsb * 16) wsuh |> resize(totsb * 5) - xfh |> resize((rows + 32) * n / 2) // + the s tile's 32-row read slack + xfh |> resize((rows + TILE_SLACK_ROWS) * n / 2) for (i in range(totsb * 16)) { wqh[i] = hash_word(uint(i) + 31u) // arbitrary quant bytes } @@ -2490,12 +2505,12 @@ def test_vkd_k2_cm2_batch(t0 : T?) { wsuh[sb * 5 + 1 + wi] = hash_word(uint(sb * 4 + wi) + 91u) } } - for (i in range((rows + 32) * n / 2)) { + for (i in range((rows + TILE_SLACK_ROWS) * n / 2)) { xfh[i] = ws_word(uint(i) * 3u + 7u) // f16 activation pairs } let wq_bytes = int64(totsb) * 64l let ws_bytes = int64(length(wsuh)) * 4l - let xf_bytes = int64((rows + 32) * n) * 2l + let xf_bytes = int64((rows + TILE_SLACK_ROWS) * n) * 2l let y_bytes = int64(rows * d) * 4l let wqd = make_device_buf(wq_bytes) let wsd = make_device_buf(ws_bytes) @@ -2617,7 +2632,7 @@ def test_vkd_iq4nl_cm2_batch(t0 : T?) { var xfh : array wqh |> resize(totsb * 32) wsuh |> resize(totsb * 5) - xfh |> resize((rows + 32) * n / 2) // + the s tile's 32-row read slack + xfh |> resize((rows + TILE_SLACK_ROWS) * n / 2) for (i in range(totsb * 32)) { wqh[i] = hash_word(uint(i) + 29u) // arbitrary quant bytes } @@ -2628,12 +2643,12 @@ def test_vkd_iq4nl_cm2_batch(t0 : T?) { } wsuh[sb * 5 + 4] = 0u } - for (i in range((rows + 32) * n / 2)) { + for (i in range((rows + TILE_SLACK_ROWS) * n / 2)) { xfh[i] = ws_word(uint(i) * 3u + 5u) // f16 activation pairs } let wq_bytes = int64(totsb) * 128l let ws_bytes = int64(length(wsuh)) * 4l - let xf_bytes = int64((rows + 32) * n) * 2l + let xf_bytes = int64((rows + TILE_SLACK_ROWS) * n) * 2l let y_bytes = int64(rows * d) * 4l let wqd = make_device_buf(wq_bytes) let wsd = make_device_buf(ws_bytes) @@ -2755,7 +2770,7 @@ def test_vkd_iq4xs_cm2_batch(t0 : T?) { var xfh : array wqh |> resize(totsb * 32) wsuh |> resize(totsb * 2) - xfh |> resize((rows + 32) * n / 2) // + the s tile's 32-row read slack + xfh |> resize((rows + TILE_SLACK_ROWS) * n / 2) for (i in range(totsb * 32)) { wqh[i] = hash_word(uint(i) + 29u) // arbitrary quant bytes } @@ -2763,12 +2778,12 @@ def test_vkd_iq4xs_cm2_batch(t0 : T?) { wsuh[sb * 2] = packHalf2x16(float2(0.0002 * float(1 + sb % 7), 0.0)) | (hash_word(uint(sb * 2) + 613u) << 16u) //! cool d | the row's high bits wsuh[sb * 2 + 1] = hash_word(uint(sb * 2 + 1) + 613u) //! the eight 4-bit sub-scale fields } - for (i in range((rows + 32) * n / 2)) { + for (i in range((rows + TILE_SLACK_ROWS) * n / 2)) { xfh[i] = ws_word(uint(i) * 3u + 5u) // f16 activation pairs } let wq_bytes = int64(totsb) * 128l let ws_bytes = int64(length(wsuh)) * 4l - let xf_bytes = int64((rows + 32) * n) * 2l + let xf_bytes = int64((rows + TILE_SLACK_ROWS) * n) * 2l let y_bytes = int64(rows * d) * 4l let wqd = make_device_buf(wq_bytes) let wsd = make_device_buf(ws_bytes) @@ -2890,7 +2905,7 @@ def test_vkd_k3_cm2_batch(t0 : T?) { var xfh : array wqh |> resize(totsb * 24) wsuh |> resize(totsb * 5) - xfh |> resize((rows + 32) * n / 2) // + the s tile's 32-row read slack + xfh |> resize((rows + TILE_SLACK_ROWS) * n / 2) for (i in range(totsb * 24)) { wqh[i] = hash_word(uint(i) + 29u) // arbitrary quant bytes } @@ -2900,12 +2915,12 @@ def test_vkd_k3_cm2_batch(t0 : T?) { } wsuh[sb * 5 + 4] = packHalf2x16(float2(0.0002 * float(1 + sb % 7), 0.0)) // cool d } - for (i in range((rows + 32) * n / 2)) { + for (i in range((rows + TILE_SLACK_ROWS) * n / 2)) { xfh[i] = ws_word(uint(i) * 3u + 5u) // f16 activation pairs } let wq_bytes = int64(totsb) * 96l let ws_bytes = int64(length(wsuh)) * 4l - let xf_bytes = int64((rows + 32) * n) * 2l + let xf_bytes = int64((rows + TILE_SLACK_ROWS) * n) * 2l let y_bytes = int64(rows * d) * 4l let wqd = make_device_buf(wq_bytes) let wsd = make_device_buf(ws_bytes) @@ -3027,7 +3042,7 @@ def test_vkd_iq3s_cm2_batch(t0 : T?) { var xfh : array wqh |> resize(totsb * 26) wsuh |> resize(totsb * 2) - xfh |> resize((rows + 32) * n / 2) // + the s tile's 32-row read slack + xfh |> resize((rows + TILE_SLACK_ROWS) * n / 2) for (i in range(totsb * 26)) { wqh[i] = hash_word(uint(i) + 29u) // arbitrary quant bytes } @@ -3035,12 +3050,12 @@ def test_vkd_iq3s_cm2_batch(t0 : T?) { wsuh[sb * 2] = packHalf2x16(float2(0.0002 * float(1 + sb % 7), 0.0)) | (hash_word(uint(sb * 2) + 613u) << 16u) //! cool d | the row's high bits wsuh[sb * 2 + 1] = hash_word(uint(sb * 2 + 1) + 613u) //! the eight 4-bit sub-scale fields } - for (i in range((rows + 32) * n / 2)) { + for (i in range((rows + TILE_SLACK_ROWS) * n / 2)) { xfh[i] = ws_word(uint(i) * 3u + 5u) // f16 activation pairs } let wq_bytes = int64(totsb) * 104l let ws_bytes = int64(length(wsuh)) * 4l - let xf_bytes = int64((rows + 32) * n) * 2l + let xf_bytes = int64((rows + TILE_SLACK_ROWS) * n) * 2l let y_bytes = int64(rows * d) * 4l let wqd = make_device_buf(wq_bytes) let wsd = make_device_buf(ws_bytes) @@ -3162,7 +3177,7 @@ def test_vkd_iq2s_cm2_batch(t0 : T?) { var xfh : array wqh |> resize(totsb * 18) wsuh |> resize(totsb * 5) - xfh |> resize((rows + 32) * n / 2) // + the s tile's 32-row read slack + xfh |> resize((rows + TILE_SLACK_ROWS) * n / 2) for (i in range(totsb * 18)) { wqh[i] = hash_word(uint(i) + 29u) // arbitrary quant bytes } @@ -3173,12 +3188,12 @@ def test_vkd_iq2s_cm2_batch(t0 : T?) { wsuh[sb * 5 + 3] = hash_word(uint(sb * 4 + 2) + 613u) wsuh[sb * 5 + 4] = hash_word(uint(sb * 4 + 3) + 613u) } - for (i in range((rows + 32) * n / 2)) { + for (i in range((rows + TILE_SLACK_ROWS) * n / 2)) { xfh[i] = ws_word(uint(i) * 3u + 5u) // f16 activation pairs } let wq_bytes = int64(totsb) * 72l let ws_bytes = int64(length(wsuh)) * 4l - let xf_bytes = int64((rows + 32) * n) * 2l + let xf_bytes = int64((rows + TILE_SLACK_ROWS) * n) * 2l let y_bytes = int64(rows * d) * 4l let wqd = make_device_buf(wq_bytes) let wsd = make_device_buf(ws_bytes) @@ -3300,7 +3315,7 @@ def test_vkd_iq2xs_cm2_batch(t0 : T?) { var xfh : array wqh |> resize(totsb * 16) wsuh |> resize(totsb * 5) - xfh |> resize((rows + 32) * n / 2) // + the s tile's 32-row read slack + xfh |> resize((rows + TILE_SLACK_ROWS) * n / 2) for (i in range(totsb * 16)) { wqh[i] = hash_word(uint(i) + 29u) // arbitrary quant bytes } @@ -3311,12 +3326,12 @@ def test_vkd_iq2xs_cm2_batch(t0 : T?) { wsuh[sb * 5 + 3] = hash_word(uint(sb * 4 + 2) + 613u) wsuh[sb * 5 + 4] = hash_word(uint(sb * 4 + 3) + 613u) } - for (i in range((rows + 32) * n / 2)) { + for (i in range((rows + TILE_SLACK_ROWS) * n / 2)) { xfh[i] = ws_word(uint(i) * 3u + 5u) // f16 activation pairs } let wq_bytes = int64(totsb) * 64l let ws_bytes = int64(length(wsuh)) * 4l - let xf_bytes = int64((rows + 32) * n) * 2l + let xf_bytes = int64((rows + TILE_SLACK_ROWS) * n) * 2l let y_bytes = int64(rows * d) * 4l let wqd = make_device_buf(wq_bytes) let wsd = make_device_buf(ws_bytes) @@ -3438,7 +3453,7 @@ def test_vkd_iq2xxs_cm2_batch(t0 : T?) { var xfh : array wqh |> resize(totsb * 16) wsuh |> resize(totsb * 2) - xfh |> resize((rows + 32) * n / 2) // + the s tile's 32-row read slack + xfh |> resize((rows + TILE_SLACK_ROWS) * n / 2) for (i in range(totsb * 16)) { wqh[i] = hash_word(uint(i) + 29u) // arbitrary quant bytes } @@ -3446,12 +3461,12 @@ def test_vkd_iq2xxs_cm2_batch(t0 : T?) { wsuh[sb * 2] = packHalf2x16(float2(0.0002 * float(1 + sb % 7), 0.0)) | (hash_word(uint(sb * 2) + 613u) << 16u) //! cool d8 | the row's high bits wsuh[sb * 2 + 1] = hash_word(uint(sb * 2 + 1) + 613u) //! the eight 4-bit (2ls + 1) fields } - for (i in range((rows + 32) * n / 2)) { + for (i in range((rows + TILE_SLACK_ROWS) * n / 2)) { xfh[i] = ws_word(uint(i) * 3u + 5u) // f16 activation pairs } let wq_bytes = int64(totsb) * 64l let ws_bytes = int64(length(wsuh)) * 4l - let xf_bytes = int64((rows + 32) * n) * 2l + let xf_bytes = int64((rows + TILE_SLACK_ROWS) * n) * 2l let y_bytes = int64(rows * d) * 4l let wqd = make_device_buf(wq_bytes) let wsd = make_device_buf(ws_bytes) @@ -3573,7 +3588,7 @@ def test_vkd_iq3xxs_cm2_batch(t0 : T?) { var xfh : array wqh |> resize(totsb * 24) wsuh |> resize(totsb * 2) - xfh |> resize((rows + 32) * n / 2) // + the s tile's 32-row read slack + xfh |> resize((rows + TILE_SLACK_ROWS) * n / 2) for (i in range(totsb * 24)) { wqh[i] = hash_word(uint(i) + 29u) // arbitrary quant bytes } @@ -3581,12 +3596,12 @@ def test_vkd_iq3xxs_cm2_batch(t0 : T?) { wsuh[sb * 2] = packHalf2x16(float2(0.0002 * float(1 + sb % 7), 0.0)) | (hash_word(uint(sb * 2) + 613u) << 16u) //! cool d | the row's high bits wsuh[sb * 2 + 1] = hash_word(uint(sb * 2 + 1) + 613u) //! the eight 4-bit sub-scale fields } - for (i in range((rows + 32) * n / 2)) { + for (i in range((rows + TILE_SLACK_ROWS) * n / 2)) { xfh[i] = ws_word(uint(i) * 3u + 5u) // f16 activation pairs } let wq_bytes = int64(totsb) * 96l let ws_bytes = int64(length(wsuh)) * 4l - let xf_bytes = int64((rows + 32) * n) * 2l + let xf_bytes = int64((rows + TILE_SLACK_ROWS) * n) * 2l let y_bytes = int64(rows * d) * 4l let wqd = make_device_buf(wq_bytes) let wsd = make_device_buf(ws_bytes) @@ -3708,7 +3723,7 @@ def test_vkd_k6_cm2_batch(t0 : T?) { var xfh : array wqh |> resize(totsb * 48) wsuh |> resize(totsb * 5) - xfh |> resize((rows + 32) * n / 2) // + the s tile's 32-row read slack + xfh |> resize((rows + TILE_SLACK_ROWS) * n / 2) for (i in range(totsb * 48)) { wqh[i] = hash_word(uint(i) + 41u) // arbitrary ql/qh bytes } @@ -3718,12 +3733,12 @@ def test_vkd_k6_cm2_batch(t0 : T?) { } wsuh[sb * 5 + 4] = packHalf2x16(float2(0.0002 * float(1 + sb % 7), 0.0)) // cool d: the f16 acc must stay far from 65504 } - for (i in range((rows + 32) * n / 2)) { + for (i in range((rows + TILE_SLACK_ROWS) * n / 2)) { xfh[i] = ws_word(uint(i) * 3u + 5u) } let wq_bytes = int64(totsb) * 192l let ws_bytes = int64(totsb * 5) * 4l - let xf_bytes = int64((rows + 32) * n) * 2l + let xf_bytes = int64((rows + TILE_SLACK_ROWS) * n) * 2l let y_bytes = int64(rows * d) * 4l let wqd = make_device_buf(wq_bytes) let wsd = make_device_buf(ws_bytes) @@ -3842,19 +3857,19 @@ def test_vkd_cm2m_batch(t0 : T?) { var xfh : array wqh |> resize(totblk * 8) wsh |> resize((totblk + 1) / 2) - xfh |> resize(rows * n / 2) + xfh |> resize((rows + TILE_SLACK_ROWS) * n / 2) for (i in range(totblk * 8)) { wqh[i] = hash_word(uint(i) + 29u) // arbitrary bytes — s8 quants } for (i in range((totblk + 1) / 2)) { wsh[i] = ws_word(uint(i) + 733u) // f16 scale pairs, finite halves } - for (i in range(rows * n / 2)) { + for (i in range((rows + TILE_SLACK_ROWS) * n / 2)) { xfh[i] = ws_word(uint(i) * 5u + 11u) // f16 activation pairs } let wq_bytes = int64(totblk) * 32l let ws_bytes = int64(length(wsh)) * 4l - let xf_bytes = int64(rows * n) * 2l + let xf_bytes = int64((rows + TILE_SLACK_ROWS) * n) * 2l let y_bytes = int64(rows * d) * 4l let wqd = make_device_buf(wq_bytes) let wsd = make_device_buf(ws_bytes) @@ -3946,19 +3961,19 @@ def test_vkd_cm2s_batch(t0 : T?) { var xfh : array wqh |> resize(totblk * 8) wsh |> resize((totblk + 1) / 2) - xfh |> resize((rows + 32) * n / 2) // + the s tile's 32-row read slack + xfh |> resize((rows + TILE_SLACK_ROWS) * n / 2) for (i in range(totblk * 8)) { wqh[i] = hash_word(uint(i) + 29u) // arbitrary bytes — s8 quants } for (i in range((totblk + 1) / 2)) { wsh[i] = ws_word(uint(i) + 733u) // f16 scale pairs, finite halves } - for (i in range((rows + 32) * n / 2)) { + for (i in range((rows + TILE_SLACK_ROWS) * n / 2)) { xfh[i] = ws_word(uint(i) * 5u + 11u) // f16 activation pairs } let wq_bytes = int64(totblk) * 32l let ws_bytes = int64(length(wsh)) * 4l - let xf_bytes = int64((rows + 32) * n) * 2l + let xf_bytes = int64((rows + TILE_SLACK_ROWS) * n) * 2l let y_bytes = int64(rows * d) * 4l let wqd = make_device_buf(wq_bytes) let wsd = make_device_buf(ws_bytes) @@ -7826,3 +7841,624 @@ def test_vkd_ar_f16_fused(t0 : T?) { } } } + +// ===== the resident MoE block's routing kernels (router GEMM, per-row top-k, the bucket schedule, the gated combine) ===== + +let private MR_NPOS = 37 +let private MR_NE = 20 +let private MR_K = 3 +let private MR_DIM = 2 * 64 //! two of the router tile's 64-wide K steps +let private MR_NER = MR_NE + 1 //! the shared expert's gate row rides past the experts +let private MR_SENT = 0x5A5A5A5Au //! the schedule planes' fill: a word the kernel never writes stays this on both sides + +//! one 32-bit word to a float in [-2, 2) +def private hash_unit(i : uint) : float => float(hash_word(i) % 4096u) / 1024.0 - 2.0 + +//! the device: the recorded dispatches over `raw`, then `bytes` of `dev` back into `host` +def private vkd_run_copy(var raw : VkCommandBuffer; var h : VkHaz; dev : uint64; host : HostBuf; bytes : int64; bits : uint) { + vhz_dep(raw, h, bits, 0u, true) + cmd_copy_whole(raw, dev, host.buf, bytes) + vk_check(vkEndCommandBuffer(raw), null) + submit_wait(raw) +} + +//! the schedule's bounds and map offsets for the sched cells: the s dispatch one tile per expert, the +//! m dispatch every expert's whole columns plus a partial one; the records of both lists precede the maps +struct private SchedShape { + map_off : int + map_off_mgu : int + map_off_mdn : int + bound_gu : int + bound_dn : int + bound_mgu : int + bound_mdn : int +} + +def private sched_shape(nk, ne, wt_gu, wt_dn : int) : SchedShape { + let rt_m = nk / int(SCHED_M_ROWS) + ne + var s = SchedShape(map_off = 2 * 4 * ne, bound_gu = ne * wt_gu, bound_dn = ne * wt_dn, bound_mgu = rt_m * wt_gu, bound_mdn = rt_m * wt_dn) + s.map_off_mgu = s.map_off + s.bound_gu + s.map_off_mdn = s.map_off + s.bound_dn + return s +} + +//! expert e's bucket: its first row (the slots of the experts before it) and its row count +def private bucket_start(idx : array; e : int) : int { + var run = 0 + for (r in idx) { + if (int(r) < e) { + run++ + } + } + return run +} + +def private bucket_rows(idx : array; e : int) : int { + var n = 0 + for (r in idx) { + if (int(r) == e) { + n++ + } + } + return n +} + +//! one piece's record `ri` and its map words on the three planes (the CPU twin of MoeSched.write_piece) +def private oracle_piece(var s1, s3, s2 : array; bases : uint[6]; e : int; ri : uint; row0, rows, rt, wt_gu, wt_dn, wg_gu, wg_dn, map_gu, map_dn : int) { + let rb = int(ri) * 4 + s1[rb] = bases[0] + uint(e) * bases[1] + s3[rb] = bases[2] + uint(e) * bases[3] + s2[rb] = bases[4] + uint(e) * bases[5] + s1[rb + 1] = uint(row0) + s3[rb + 1] = uint(row0) + s2[rb + 1] = uint(row0) + s1[rb + 2] = uint(rows) + s3[rb + 2] = uint(rows) + s2[rb + 2] = uint(rows) + s1[rb + 3] = uint(wg_gu) + s3[rb + 3] = uint(wg_gu) + s2[rb + 3] = uint(wg_dn) + for (g in range(rt * wt_gu)) { + s1[map_gu + wg_gu + g] = ri + s3[map_gu + wg_gu + g] = ri + } + for (g in range(rt * wt_dn)) { + s2[map_dn + wg_dn + g] = ri + } +} + +//! the CPU twin of MoeSched: the s pieces' records at [0, ne) and the m pieces' at [ne, 2 ne), each in +//! expert order (empty pieces skipped), the two dispatches' per-wg maps, the sentinel tails past the +//! real workgroup counts, and inv in slot order +def private sched_oracle(idx : array; ne, wt_gu, wt_dn : int; sh : SchedShape; bases : uint[6]; + var inv : array; var s1, s3, s2 : array) { + let nk = length(idx) + var cnt : array + cnt |> resize(ne) + for (r in range(nk)) { + cnt[int(idx[r])]++ + } + var start : array + start |> resize(ne) + var run = 0 + for (e in range(ne)) { + start[e] = run + run += cnt[e] + } + inv |> resize(nk) + var cursor := start + for (r in range(nk)) { + let e = int(idx[r]) + inv[r] = uint(cursor[e]) + cursor[e]++ + } + let words = max(sh.map_off_mgu + sh.bound_mgu, sh.map_off_mdn + sh.bound_mdn) + s1 |> resize(words) + s3 |> resize(words) + s2 |> resize(words) + for (i in range(words)) { + s1[i] = MR_SENT + s3[i] = MR_SENT + s2[i] = MR_SENT + } + var ri_s = 0u + var ri_m = uint(ne) + var wg_gu_s = 0 + var wg_dn_s = 0 + var wg_gu_m = 0 + var wg_dn_m = 0 + let s_col = int(SCHED_S_ROWS) + let m_col = int(SCHED_M_ROWS) + for (e in range(ne)) { + continue if (cnt[e] == 0) + //! the ladder spelled out (the shipped rule is the thing under test): a bucket within the s column + //! takes it whole; past it, whole m columns, and a remainder past the s column joins the last as a partial + let c = cnt[e] + let mrows = c <= s_col ? 0 : ((c % m_col) > s_col ? c : (c / m_col) * m_col) + let srows = c - mrows + if (mrows > 0) { + let rt = (mrows + m_col - 1) / m_col + oracle_piece(s1, s3, s2, bases, e, ri_m, start[e], mrows, rt, wt_gu, wt_dn, wg_gu_m, wg_dn_m, sh.map_off_mgu, sh.map_off_mdn) + wg_gu_m += rt * wt_gu + wg_dn_m += rt * wt_dn + ri_m++ + } + if (srows > 0) { + oracle_piece(s1, s3, s2, bases, e, ri_s, start[e] + mrows, srows, 1, wt_gu, wt_dn, wg_gu_s, wg_dn_s, sh.map_off, sh.map_off) + wg_gu_s += wt_gu + wg_dn_s += wt_dn + ri_s++ + } + } + for (g in range(wg_gu_s, sh.bound_gu)) { + s1[sh.map_off + g] = SCHED_NONE + s3[sh.map_off + g] = SCHED_NONE + } + for (g in range(wg_dn_s, sh.bound_dn)) { + s2[sh.map_off + g] = SCHED_NONE + } + for (g in range(wg_gu_m, sh.bound_mgu)) { + s1[sh.map_off_mgu + g] = SCHED_NONE + s3[sh.map_off_mgu + g] = SCHED_NONE + } + for (g in range(wg_dn_m, sh.bound_mdn)) { + s2[sh.map_off_mdn + g] = SCHED_NONE + } + delete cnt + delete start + delete cursor +} + +//! the router GEMM: two layers' [ner x dim] rows, the second layer's dispatched through woff; logits at the ner +//! stride; `ner` past a half-tile exercises both rows of an invocation, under it the second row's guard alone +def private moe_router_arm(t : T?; ner : int) { + static_if (typeinfo builtin_module_exists(vulkan)) { + let nw = 2 * ner * MR_DIM + var wh : array + var xh : array + wh |> resize(nw) + xh |> resize(MR_NPOS * MR_DIM) + for (i in range(nw)) { + wh[i] = hash_unit(uint(i) + 11u) * 0.25 + } + for (i in range(MR_NPOS * MR_DIM)) { + xh[i] = hash_unit(uint(i) + 7_777u) + } + let w_bytes = int64(nw) * 4l + let x_bytes = int64(MR_NPOS * MR_DIM) * 4l + let l_bytes = int64(MR_NPOS * ner) * 4l + let wd = make_device_buf(w_bytes) + let xd = make_device_buf(x_bytes) + let ld = make_device_buf(l_bytes) + var host = make_host_buf(l_bytes, true, [cached = true]) + var got : array + got |> resize(MR_NPOS * ner) + unsafe { + upload_region_at(wd, 0l, addr(wh[0]), w_bytes) + upload_region_at(xd, 0l, addr(xh[0]), x_bytes) + nan_sentinel_upload(ld, l_bytes) + var sc = set_router_gemm_cls(fixed_array(wd, xd, ld), fixed_array(w_bytes, x_bytes, l_bytes), fixed_array(0u, 1u, 2u)) + var raw = alloc_cmd() + let begin = VkCommandBufferBeginInfo() + vk_check(vkBeginCommandBuffer(raw, begin), null) + var h : VkHaz + var pc = RouterGemmArgs(npos = uint(MR_NPOS), ne = uint(ner), dim = uint(MR_DIM), woff = uint(ner * MR_DIM), ostride = uint(ner)) + enc_router_gemm_cls(raw, h, sc, pc, int64(((MR_NPOS + int(ROUTER_TP) - 1) / int(ROUTER_TP)) * ((ner + int(ROUTER_TE) - 1) / int(ROUTER_TE)))) + vkd_run_copy(raw, h, ld, host, l_bytes, 2u) + memcpy(addr(got[0]), host.mapped, l_bytes) + } + var want : array + want |> resize(MR_NPOS * ner) + for (p in range(MR_NPOS)) { + for (e in range(ner)) { + var acc = 0.0 + for (c in range(MR_DIM)) { + acc += wh[ner * MR_DIM + e * MR_DIM + c] * xh[p * MR_DIM + c] + } + want[p * ner + e] = acc + } + } + let bad = mismatch_rel(got, want, 1e-4) + t |> success(bad == 0, "router GEMM over {ner} rows matches the CPU oracle at the second layer's offset ({bad} of {MR_NPOS * ner} off)") + want[5 * ner + 3] += 0.5 + t |> success(mismatch_rel(got, want, 1e-4) == 1, "router GEMM control ({ner} rows): the bar reds one poisoned element") + delete wh + delete xh + delete got + delete want + } +} + +//! the per-row select against moe_select_core over the same rows: norm on with no scale, norm off with a scale +def private moe_topk_rows_arm(t : T?; norm : bool; wscale : float) { + static_if (typeinfo builtin_module_exists(vulkan)) { + var lh : array + lh |> resize(MR_NPOS * MR_NER) + for (i in range(MR_NPOS * MR_NER)) { + lh[i] = hash_unit(uint(i) * 3u + 101u) * 2.0 + float(i % 7) * 0.0625 //! spread past any near-tie + } + let l_bytes = int64(MR_NPOS * MR_NER) * 4l + let s_bytes = int64(MR_NPOS * MR_K) * 4l + let ld = make_device_buf(l_bytes) + let idxd = make_device_buf(s_bytes) + let wd = make_device_buf(s_bytes) + var hosti = make_host_buf(s_bytes, true, [cached = true]) + var hostw = make_host_buf(s_bytes, true, [cached = true]) + var gidx : array + var gw : array + gidx |> resize(MR_NPOS * MR_K) + gw |> resize(MR_NPOS * MR_K) + unsafe { + upload_region_at(ld, 0l, addr(lh[0]), l_bytes) + nan_sentinel_upload(idxd, s_bytes) + nan_sentinel_upload(wd, s_bytes) + var sc = set_topk_rows_cls(fixed_array(ld, idxd, wd), fixed_array(l_bytes, s_bytes, s_bytes), fixed_array(1u, 2u, 4u)) + var raw = alloc_cmd() + let begin = VkCommandBufferBeginInfo() + vk_check(vkBeginCommandBuffer(raw, begin), null) + var h : VkHaz + var pc = TopkRowsArgs(npos = uint(MR_NPOS), ne = uint(MR_NE), k = uint(MR_K), norm = norm ? 1u : 0u, lstride = uint(MR_NER), wscale = wscale) + enc_topk_rows_cls(raw, h, sc, pc, int64(MR_NPOS)) + vhz_dep(raw, h, 2u | 4u, 0u, true) + cmd_copy_whole(raw, idxd, hosti.buf, s_bytes) + cmd_copy_whole(raw, wd, hostw.buf, s_bytes) + vk_check(vkEndCommandBuffer(raw), null) + submit_wait(raw) + memcpy(addr(gidx[0]), hosti.mapped, s_bytes) + memcpy(addr(gw[0]), hostw.mapped, s_bytes) + } + var widx : array + var ww : array + widx |> resize(MR_NPOS * MR_K) + ww |> resize(MR_NPOS * MR_K) + var row : array + row |> resize(MR_NE) + for (p in range(MR_NPOS)) { + for (e in range(MR_NE)) { + row[e] = lh[p * MR_NER + e] + } + unsafe { + moe_select_core(MoeGate.softmax, int64(MR_NE), int64(MR_K), norm, wscale, addr(row[0]), addr(widx[0]), addr(ww[0]), int64(p * MR_K)) + } + } + var badi = 0 + for (i in range(MR_NPOS * MR_K)) { + if (int64(gidx[i]) != widx[i]) { + badi++ + } + } + let badw = mismatch_rel(gw, ww, 1e-4) + let arm = norm ? "renormalized" : "scaled by {wscale}" + t |> success(badi == 0, "per-row top-k picks the host select's experts ({arm}; {badi} of {MR_NPOS * MR_K} off)") + t |> success(badw == 0, "per-row top-k weights match the host select ({arm}; {badw} of {MR_NPOS * MR_K} off)") + ww[4] += 0.5 + t |> success(mismatch_rel(gw, ww, 1e-4) == 1, "per-row top-k control: the bar reds one poisoned weight") + delete lh + delete gidx + delete gw + delete widx + delete ww + delete row + } +} + +//! the bucket schedule against its CPU twin, word for word over the whole planes (three empty experts, a sentinel fill) +def private moe_sched_arm(t : T?) { + static_if (typeinfo builtin_module_exists(vulkan)) { + let nk = 4 * MR_NPOS * MR_K + let wt_gu = 3 + let wt_dn = 2 + let sh = sched_shape(nk, MR_NE, wt_gu, wt_dn) + var idx : array + idx |> resize(nk) + //! the ladder's four shapes: expert 0 a whole m column plus an s remainder, 1 and 2 one partial + //! m column each, 3 within the s column; the rest spread thin, the last three route nothing + let c0 = int(SCHED_M_ROWS) + int(SCHED_S_ROWS) - 10 + let c1 = int(SCHED_M_ROWS) - 28 + let c2 = int(SCHED_S_ROWS) + 8 + let c3 = int(SCHED_S_ROWS) - 12 + for (r in range(nk)) { + idx[r] = r < c0 ? 0u : (r < c0 + c1 ? 1u : (r < c0 + c1 + c2 ? 2u : (r < c0 + c1 + c2 + c3 ? 3u : 4u + hash_word(uint(r) + 55u) % uint(MR_NE - 7)))) + } + let bases = fixed_array(1_000u, 96u, 2_000u, 96u, 3_000u, 48u) + var winv : array + var w1 : array + var w3 : array + var w2 : array + sched_oracle(idx, MR_NE, wt_gu, wt_dn, sh, bases, winv, w1, w3, w2) + let i_bytes = int64(nk) * 4l + let s_bytes = long_length(w1) * 4l + let idxd = make_device_buf(i_bytes) + let invd = make_device_buf(i_bytes) + let s1d = make_device_buf(s_bytes) + let s3d = make_device_buf(s_bytes) + let s2d = make_device_buf(s_bytes) + var fill : array + fill |> resize(length(w1)) + for (f in fill) { + f = MR_SENT + } + var host = make_host_buf(i_bytes + 3l * s_bytes, true, [cached = true]) + var ginv : array + var g1 : array + var g3 : array + var g2 : array + ginv |> resize(nk) + g1 |> resize(length(w1)) + g3 |> resize(length(w1)) + g2 |> resize(length(w1)) + unsafe { + upload_region_at(idxd, 0l, addr(idx[0]), i_bytes) + nan_sentinel_upload(invd, i_bytes) + upload_region_at(s1d, 0l, addr(fill[0]), s_bytes) + upload_region_at(s3d, 0l, addr(fill[0]), s_bytes) + upload_region_at(s2d, 0l, addr(fill[0]), s_bytes) + var sc = set_moe_sched_cls(fixed_array(idxd, invd, s1d, s3d, s2d), fixed_array(i_bytes, i_bytes, s_bytes, s_bytes, s_bytes), fixed_array(1u, 2u, 2u, 2u, 2u)) + var raw = alloc_cmd() + let begin = VkCommandBufferBeginInfo() + vk_check(vkBeginCommandBuffer(raw, begin), null) + var h : VkHaz + var pc = SchedArgs(nk = uint(nk), ne = uint(MR_NE), wt_gu = uint(wt_gu), wt_dn = uint(wt_dn), + map_off = uint(sh.map_off), map_off_mgu = uint(sh.map_off_mgu), map_off_mdn = uint(sh.map_off_mdn), + bound_gu = uint(sh.bound_gu), bound_dn = uint(sh.bound_dn), bound_mgu = uint(sh.bound_mgu), bound_mdn = uint(sh.bound_mdn), + base1 = bases[0], stride1 = bases[1], base3 = bases[2], stride3 = bases[3], base2 = bases[4], stride2 = bases[5]) + enc_moe_sched_cls(raw, h, sc, pc, 1l) + vhz_dep(raw, h, 2u, 0u, true) + cmd_copy_range(raw, invd, 0l, host.buf, 0l, i_bytes) + cmd_copy_range(raw, s1d, 0l, host.buf, i_bytes, s_bytes) + cmd_copy_range(raw, s3d, 0l, host.buf, i_bytes + s_bytes, s_bytes) + cmd_copy_range(raw, s2d, 0l, host.buf, i_bytes + 2l * s_bytes, s_bytes) + vk_check(vkEndCommandBuffer(raw), null) + submit_wait(raw) + let hp = intptr(host.mapped) + memcpy(addr(ginv[0]), host.mapped, i_bytes) + memcpy(addr(g1[0]), reinterpret(hp + uint64(i_bytes)), s_bytes) + memcpy(addr(g3[0]), reinterpret(hp + uint64(i_bytes + s_bytes)), s_bytes) + memcpy(addr(g2[0]), reinterpret(hp + uint64(i_bytes + 2l * s_bytes)), s_bytes) + } + //! the atomic cursor fixes no order within a bucket, so the check is per bucket: a permutation of its rows + var inv_bad = 0 + var seen : array + seen |> resize(nk) + for (r in range(nk)) { + let e = int(idx[r]) + let base = uint(bucket_start(idx, e)) + let rows = uint(bucket_rows(idx, e)) + let row = ginv[r] + if (row < base || row >= base + rows || seen[int(row)]) { + inv_bad++ + } else { + seen[int(row)] = true + } + } + t |> success(inv_bad == 0, "the schedule's slot -> bucket row map is a permutation of each bucket's rows ({nk} slots, {inv_bad} off)") + delete seen + t |> success(mismatch_qbytes(g1, w1, 0) == 0, "the gate plane's s and m records, maps and sentinel tails match the CPU twin word for word ({length(w1)} words)") + t |> success(mismatch_qbytes(g3, w3, 0) == 0, "the up plane's schedule matches") + t |> success(mismatch_qbytes(g2, w2, 0) == 0, "the down plane's schedule matches (its own tile counts and tails)") + w2[sh.map_off_mdn + sh.bound_mdn - 1] = 0u + t |> success(mismatch_qbytes(g2, w2, 0) > 0, "schedule control: the m map's last sentinel word flipped reds the compare") + delete idx + delete winv + delete w1 + delete w3 + delete w2 + delete fill + delete ginv + delete g1 + delete g3 + delete g2 + } +} + +//! the residual step with the combine folded in: x += [g *] shared row + the k weighted expert rows +//! through the slot map, then the next norm - the f32 and the f16 normed-row forms against one CPU +//! oracle; `with_sh` = the layer has a shared expert (its rows ride the add partner) +def private moe_ar_comb_arm(t : T?; gated, with_sh : bool) { + static_if (typeinfo builtin_module_exists(vulkan)) { + let n = 64 + let k = 3 + let npos = 5 + let nrows = npos * k + let eps = 1.0e-6 + var sidx : array + var sw : array + var rows : array + var xs : array + var sh : array + var wn : array + var lg : array + sidx |> resize(nrows) + sw |> resize(nrows) + rows |> resize(nrows * n) + xs |> resize(npos * n) + sh |> resize(npos * n) + wn |> resize(n) + lg |> resize(npos * MR_NER) + for (i in range(nrows)) { + sidx[i] = uint((i * 7) % nrows) //! a permutation of the bucket rows + sw[i] = 0.125 + float(i % 5) * 0.25 + } + for (i in range(nrows * n)) { + rows[i] = float((i * 7) % 93) * 0.13 - 5.5 + } + for (i in range(npos * n)) { + xs[i] = float((i * 11) % 37) * 0.1 - 1.7 + sh[i] = float((i * 5) % 41) * 0.2 - 3.0 + } + for (i in range(n)) { + wn[i] = 0.5 + float(i % 7) * 0.125 + } + for (i in range(npos * MR_NER)) { + lg[i] = hash_unit(uint(i) + 9u) + } + let si_bytes = int64(nrows) * 4l + let x_bytes = int64(npos * n) * 4l + let ro_bytes = int64(nrows * n) * 4l + let wn_bytes = int64(n) * 4l + let lg_bytes = int64(npos * MR_NER) * 4l + let sid = make_device_buf(si_bytes) + let swd = make_device_buf(si_bytes) + let rod = make_device_buf(ro_bytes) + let xd = make_device_buf(x_bytes) + let shd = make_device_buf(x_bytes) + let wnd = make_device_buf(wn_bytes) + let lgd = make_device_buf(lg_bytes) + let yod = make_device_buf(x_bytes) + let yhd = make_device_buf(x_bytes / 2l) + var host = make_host_buf(3l * x_bytes, true, [cached = true]) + var got_x : array + var got_y : array + var got_h : array + got_x |> resize(npos * n) + got_y |> resize(npos * n) + got_h |> resize(npos * n / 2) + var pc = ArArgs(dim = uint(n), add_on = with_sh ? 1u : 0u, woff = 0u, eps = eps, ascale = 1.0, row0 = 0u, + slots = uint(k), lstride = uint(MR_NER), gated = gated ? 1u : 0u) + unsafe { + upload_region_at(sid, 0l, addr(sidx[0]), si_bytes) + upload_region_at(swd, 0l, addr(sw[0]), si_bytes) + upload_region_at(rod, 0l, addr(rows[0]), ro_bytes) + upload_region_at(shd, 0l, addr(sh[0]), x_bytes) + upload_region_at(wnd, 0l, addr(wn[0]), wn_bytes) + upload_region_at(lgd, 0l, addr(lg[0]), lg_bytes) + nan_sentinel_upload(yod, x_bytes) + nan_sentinel_upload(yhd, x_bytes / 2l) + //! the f32 form: x updated in place, the normed row to yo + upload_region_at(xd, 0l, addr(xs[0]), x_bytes) + var sc = set_cls_ar_comb(fixed_array(xd, shd, wnd, yod, sid, swd, rod, lgd), + fixed_array(x_bytes, x_bytes, wn_bytes, x_bytes, si_bytes, si_bytes, ro_bytes, lg_bytes), + fixed_array(1u, 2u, 0u, 4u, 8u, 8u, 16u, 32u)) + var raw = alloc_cmd() + let begin = VkCommandBufferBeginInfo() + vk_check(vkBeginCommandBuffer(raw, begin), null) + var h : VkHaz + enc_cls_ar_comb(raw, h, sc, pc, int64(npos)) + vhz_dep(raw, h, 1u | 4u, 0u, true) + cmd_copy_range(raw, xd, 0l, host.buf, 0l, x_bytes) + cmd_copy_range(raw, yod, 0l, host.buf, x_bytes, x_bytes) + vk_check(vkEndCommandBuffer(raw), null) + submit_wait(raw) + memcpy(addr(got_x[0]), host.mapped, x_bytes) + memcpy(addr(got_y[0]), reinterpret(intptr(host.mapped) + uint64(x_bytes)), x_bytes) + //! the f16 form over a fresh x + upload_region_at(xd, 0l, addr(xs[0]), x_bytes) + var sc16 = set_cls_ar_comb_f16_b(fixed_array(xd, shd, wnd, yhd, sid, swd, rod, lgd), + fixed_array(x_bytes, x_bytes, wn_bytes, x_bytes / 2l, si_bytes, si_bytes, ro_bytes, lg_bytes), + fixed_array(1u, 2u, 0u, 4u, 8u, 8u, 16u, 32u)) + var raw2 = alloc_cmd() + vk_check(vkBeginCommandBuffer(raw2, begin), null) + var h2 : VkHaz + enc_cls_ar_comb_f16_b(raw2, h2, sc16, pc, int64(npos)) + vkd_run_copy(raw2, h2, yhd, host, x_bytes / 2l, 4u) + memcpy(addr(got_h[0]), host.mapped, x_bytes / 2l) + } + var want_x : array + var want_y : array + want_x |> resize(npos * n) + want_y |> resize(npos * n) + for (p in range(npos)) { + let g = with_sh && gated ? 1.0 / (1.0 + exp(-lg[p * MR_NER + MR_NE])) : 1.0 + var ss = 0.0 + for (i in range(n)) { + var v = xs[p * n + i] + if (with_sh) { + v += g * sh[p * n + i] + } + for (j in range(k)) { + v += sw[p * k + j] * rows[int(sidx[p * k + j]) * n + i] + } + want_x[p * n + i] = v + ss += v * v + } + let inv = 1.0 / sqrt(ss / float(n) + eps) + for (i in range(n)) { + want_y[p * n + i] = wn[i] * (want_x[p * n + i] * inv) + } + } + let arm = (with_sh ? (gated ? "gated shared" : "ungated shared") : "no shared") + " expert" + t |> success(mismatch_approx(got_x, xs) > 0, "the combine-folded residual step ({arm}): the in-place residual rows moved off their input") + let bad_x = mismatch_approx(got_x, want_x) + t |> success(bad_x == 0, "the combine-folded residual step ({arm}): the residual rows match the CPU oracle ({bad_x} of {npos * n} off)") + let bad_y = mismatch_approx(got_y, want_y) + t |> success(bad_y == 0, "the combine-folded residual step ({arm}): the f32 normed rows match the CPU oracle ({bad_y} of {npos * n} off)") + var got_yf : array + got_yf |> resize(npos * n) + for (i in range(npos * n / 2)) { + let pr = unpackHalf2x16(got_h[i]) + got_yf[2 * i] = pr.x + got_yf[2 * i + 1] = pr.y + } + //! the f16 bar: a half rounds to 2^-11 of the value (4.9e-4 relative); the bar is four halves of + //! that, and the same fraction of the row's largest value as the floor the near-zero elements read + let f16_bar = 2.0e-3 + let bad_h = mismatch_bars(got_yf, want_y, f16_bar, f16_bar * max_abs(want_y)) + t |> success(bad_h == 0, "the combine-folded residual step ({arm}): the f16 normed rows within the half's rounding of the oracle ({bad_h} off)") + want_x[2 * n + 9] += 0.5 + t |> success(mismatch_approx(got_x, want_x) == 1, "combine-folded residual control: the residual bar reds one poisoned element") + want_y[3 * n + 5] += 0.5 + t |> success(mismatch_approx(got_y, want_y) == 1, "combine-folded residual control: the f32 normed-row bar reds one poisoned element") + t |> success(mismatch_bars(got_yf, want_y, f16_bar, f16_bar * max_abs(want_y)) == 1, "combine-folded residual control: the f16 normed-row bar reds one poisoned element") + delete sidx + delete sw + delete rows + delete xs + delete sh + delete wn + delete lg + delete got_x + delete got_y + delete got_h + delete got_yf + delete want_x + delete want_y + } +} + +[test] +def test_vkd_moe_routing(t0 : T?) { + t0 |> run("the resident MoE block's routing kernels == the CPU oracles: router GEMM, per-row top-k, the bucket schedule, the combine-folded residual step") <| @(t : T?) { + static_if (typeinfo builtin_module_exists(vulkan)) { + if (!(ensure_router_gemm_cls() && ensure_topk_rows_cls() && ensure_moe_sched_cls() && ensure_cls_ar_comb() && ensure_cls_ar_comb_f16_b())) { + t |> skip("no Vulkan device") + return + } + moe_router_arm(t, MR_NER) + moe_router_arm(t, 9) //! fewer rows than the tile's half: the second row of every invocation is guarded off + moe_topk_rows_arm(t, true, 0.0) + moe_topk_rows_arm(t, false, 2.5) + moe_sched_arm(t) + moe_ar_comb_arm(t, true, true) + moe_ar_comb_arm(t, false, true) + moe_ar_comb_arm(t, false, false) + } else { + t |> skip("dasVulkan not present") + } + } +} + +//! the GEMV family's lanes-per-row rule, pinned per (format, row length): the codebook and grid formats +//! take fewer lanes than the k-lattice formats at the same length, the q8 GEMV always the whole subgroup +[test] +def test_vkd_gemv_lane_rule(t0 : T?) { + t0 |> run("gemv_lanes_per_row: the measured lane split per format class and row length") <| @(t : T?) { + static_if (typeinfo builtin_module_exists(vulkan)) { + let grid = fixed_array(KqFmt.iq2s, KqFmt.iq2xs, KqFmt.iq2xxs, KqFmt.iq3xxs, KqFmt.iq4xs, KqFmt.iq4nl) + let lattice = fixed_array(KqFmt.k2, KqFmt.k3, KqFmt.k4, KqFmt.k5, KqFmt.k6, KqFmt.q40, KqFmt.iq3s) + //! row length -> (the grid formats' lanes, the k-lattice formats' lanes); 0 = the whole subgroup + let rows = fixed_array((n = 768l, g = 8u, l = 8u), (n = 1024l, g = 8u, l = 16u), (n = 1536l, g = 8u, l = 16u), + (n = 2048l, g = 16u, l = 16u), (n = 3072l, g = 16u, l = 16u), (n = 4096l, g = 16u, l = 0u), (n = 8192l, g = 16u, l = 0u)) + for (r in rows) { + for (f in grid) { + t |> equal(gemv_lanes_per_row(int(f), r.n), r.g, "{f} at K {r.n}: a grid format takes {r.g} lanes per row") + } + for (f in lattice) { + t |> equal(gemv_lanes_per_row(int(f), r.n), r.l, "{f} at K {r.n}: a k-lattice format takes {r.l} lanes per row") + } + t |> equal(gemv_lanes_per_row(int(KqFmt.q8), r.n), 0u, "q8 at K {r.n}: the whole subgroup") + } + } else { + t |> skip("dasVulkan not present") + } + } +} diff --git a/modules/dasLLAMA/tests/test_vulkan_moe_cm2.das b/modules/dasLLAMA/tests/test_vulkan_moe_cm2.das index e0791c1668..33b3987e60 100644 --- a/modules/dasLLAMA/tests/test_vulkan_moe_cm2.das +++ b/modules/dasLLAMA/tests/test_vulkan_moe_cm2.das @@ -78,7 +78,8 @@ def private ref_chain(w1, w3, w2 : array; x : array; inv : array var ws1 : array @@ -121,13 +122,14 @@ def private chain_cell(t : T?; tag : string; n, nfe, npos, k, cnt_a : int64; wof w |> resize(nk) expert_of_row |> resize(nk) for (j in range64(nk)) { - let b = (j * 7l) % nk + let b = one_region ? j : (j * 7l) % nk inv[j] = uint(b) - w[j] = 0.25 + 0.5 * float((j * 7l) % 5l) / 4.0 + w[j] = one_region ? 1.0 : 0.25 + 0.5 * float((j * 7l) % 5l) / 4.0 expert_of_row[b] = b < cnt_a ? 1 : 0 } let ege1 = nfe * n let ege2 = n * nfe + let nreg = cnt_a == nk ? 1l : 2l //! every row in region A: one region, as the shared expert calls it var offs1 <- [woff1 + ege1, 0l, cnt_a, woff1, cnt_a, nk - cnt_a] var offs3 <- [woff3 + ege1, 0l, cnt_a, woff3, cnt_a, nk - cnt_a] var offs2 <- [woff2 + ege2, 0l, cnt_a, woff2, cnt_a, nk - cnt_a] @@ -148,7 +150,7 @@ def private chain_cell(t : T?; tag : string; n, nfe, npos, k, cnt_a : int64; wof matmul_moe_gpu_ffn_combined_xf_join(gout) ref_chain(w1, w3, w2, x, inv, wg, expert_of_row, n, nfe, npos, k, refv) } else { - matmul_moe_gpu_ffn_combined_xf(gout, offs1, offs3, offs2, 2l, x, w, inv, n, nfe, nk, npos, 1, 1, 3, false) + matmul_moe_gpu_ffn_combined_xf(gout, offs1, offs3, offs2, nreg, x, w, inv, n, nfe, nk, npos, 1, 1, 3, false) ref_chain(w1, w3, w2, x, inv, w, expert_of_row, n, nfe, npos, k, refv) } let mr = max_abs(refv) @@ -208,6 +210,22 @@ def test_vulkan_moe_cm2_chain(t : T?) { } } +[test] +def test_vulkan_moe_cm2_one_region(t : T?) { + t |> run("cm2 expert chain as the shared expert calls it: one region over every position, the identity slot map at unit weight == the CPU reference") @(t : T?) { + static_if (typeinfo builtin_module_exists(vulkan)) { + install_moe_gpu_tier(@@vk_moe_ffn, @@vk_moe_upload_stack, @@vk_moe_cls, @@vk_moe_dense, @@vk_moe_drop_stacks, @@vk_moe_dn, @@vk_moe_attn) + set_moe_gpu_ffn_xf_hooks(@@vk_moe_ffn_xf, @@vk_moe_ffn_xf_ok) + //! k = 1 and every row in region A (cnt_a == nk): the one-region form; 40 rows = one s-tile column plus a partial + chain_cell(t, "one-region", 512l, 512l, 40l, 1l, 40l, 71000000l, 1000000l, false, false, true) + //! the same over a 2048-wide dim at 300 rows: the multi-row-tile region and the down GEMM's l-tile pick + chain_cell(t, "one-region-big", 2048l, 512l, 300l, 1l, 300l, 81000000l, 4000000l, false, false, true) + } else { + t |> skip("dasVulkan not installed; the cm2 expert chain is untestable here") + } + } +} + [test] def test_vulkan_moe_cm2_streamed(t : T?) { t |> run("cm2 expert chain over a streamed group (slot rebase + the timeline hand-off) == the CPU reference") @(t : T?) { diff --git a/utils/internal/review-md/test_walkers.das b/utils/internal/review-md/test_walkers.das index 16ec4b36f7..dcf709f5aa 100644 --- a/utils/internal/review-md/test_walkers.das +++ b/utils/internal/review-md/test_walkers.das @@ -104,7 +104,24 @@ let DASLLAMA_META_GREEN = "options gen2\n\nstruct Thing \{\n a : int\n b : // which the pipe-form-only count missed let DASLLAMA_META_RED = "options gen2\n\nstruct Thing \{\n a : int\n b : array\n\}\n\ndef private serialize_meta(var arch : Archive; var t : Thing) \{\n serialize(arch, t.a)\n serialize_pod_array(arch, t.b)\n\}\n" +// The Vulkan surface the gate's four Vulkan checks read: the abstract KHR stage on the cm2 template, +// the add+rms row slab, one superblock format template with its KHR class and dispatch stamp, and +// the q8 template the KHR set licenses. +let VK_CLASSES_GREEN = "options gen2\n\nclass ArBase : RmsWgBase \{\n @workgroup row : float[8192]\n\}\n\nclass template KqCm2BatchT : MoeCmBase \{\n def abstract khr_stage16(blk, e0, sbase : uint) : void\n\}\n\nclass template K4Cm2T : KqCm2BatchT \{\n\}\n\nclass template Q8Cm2T : KqCm2BatchT \{\n\}\n\n[vk_dispatch(name = \"kq_batch_k4_khr_cls\", kernel = \"run\")]\nclass K4KhrBatch : K4Cm2T \{\n\}\n" + +// the same surface with the stage given a body and a second superblock template that ships no KHR class +let VK_CLASSES_RED = "options gen2\n\nclass ArBase : RmsWgBase \{\n @workgroup row : float[8192]\n\}\n\nclass template KqCm2BatchT : MoeCmBase \{\n def khr_stage16(blk, e0, sbase : uint) : void \{\n \}\n\}\n\nclass template K4Cm2T : KqCm2BatchT \{\n\}\n\nclass template K5Cm2T : KqCm2BatchT \{\n\}\n\nclass template Q8Cm2T : KqCm2BatchT \{\n\}\n\n[vk_dispatch(name = \"kq_batch_k4_khr_cls\", kernel = \"run\")]\nclass K4KhrBatch : K4Cm2T \{\n\}\n" + +// the prefill's three KHR ladders, each with the one format's arm +let VK_PREFILL_TEXT = "options gen2\n\ndef khr_cls_ensure(fmt : int) : bool \{\n return ensure_kq_batch_k4_khr_cls()\n\}\n\ndef khr_cls_set(fmt : int) : int \{\n return set_kq_batch_k4_khr_cls()\n\}\n\ndef khr_cls_enc(fmt : int) \{\n enc_kq_batch_k4_khr_cls()\n\}\n" + +let DASLLAMA_MODULE_LIST = "\"dasllama\", \"dasllama_audio_embedder\", \"dasllama_vision_embedder\", \"dasllama_toyfam\", \"dasllama_toyvis\", \"dasllama_image\", \"dasllama_meta\", \"dasllama_vulkan_common\", \"dasllama_vulkan_classes\", \"dasllama_vulkan_prefill\", \"dasllama_blocks\"\n" + def make_dasllama_fixture(t : T?; tree, root : string) { + write_at(t, root, "modules/dasLLAMA/dasllama/dasllama_vulkan_common.das", "options gen2\n\nlet AR_MAX_DIM = 8_192l\n") + write_at(t, root, "modules/dasLLAMA/dasllama/dasllama_vulkan_classes.das", VK_CLASSES_GREEN) + write_at(t, root, "modules/dasLLAMA/dasllama/dasllama_vulkan_prefill.das", VK_PREFILL_TEXT) + write_at(t, root, "modules/dasLLAMA/dasllama/dasllama_blocks.das", "options gen2\n\ndef private attn_dec_shape_ok(c : Config) : bool \{\n return !(c.dim > 8192l)\n\}\n") write_at(t, root, "modules/dasLLAMA/dasllama/dasllama.das", "options gen2\n\ndef covered_fn() \{\n\}\n") write_at(t, root, "modules/dasLLAMA/dasllama/dasllama_audio_embedder.das", "enum AudioKind \{\n none\n toyfam\n\}\n") write_at(t, root, "modules/dasLLAMA/dasllama/dasllama_vision_embedder.das", "enum VisionKind \{\n none\n toyvis\n\}\n") @@ -112,8 +129,7 @@ def make_dasllama_fixture(t : T?; tree, root : string) { write_at(t, root, "modules/dasLLAMA/dasllama/dasllama_toyvis.das", "struct ToyvisTower \{\n\}\n") write_at(t, root, "modules/dasLLAMA/dasllama/dasllama_image.das", dasllama_image_text("64l", "cur", "1")) write_at(t, root, "modules/dasLLAMA/dasllama/dasllama_meta.das", DASLLAMA_META_GREEN) - write_at(t, root, "modules/dasLLAMA/.das_module", - "\"dasllama\", \"dasllama_audio_embedder\", \"dasllama_vision_embedder\", \"dasllama_toyfam\", \"dasllama_toyvis\", \"dasllama_image\", \"dasllama_meta\"\n") + write_at(t, root, "modules/dasLLAMA/.das_module", DASLLAMA_MODULE_LIST) write_at(t, root, "modules/dasLLAMA/REVIEW.md", "rules\n") write_at(t, root, "tutorials/dasLLAMA/01_demo.das", "covered_fn()\n// prose beside the call stays harmless\n") write_at(t, root, "doc/source/reference/tutorials/dasLLAMA_01_demo.rst", "``covered_fn`` narrated\n") @@ -180,14 +196,27 @@ def test_dasllama_gate_fixtures(t : T?) { write_at(t, fixture, "tutorials/dasLLAMA/03_vis.das", "var tw : ToyvisTower\n") write_at(t, fixture, "modules/dasLLAMA/dasllama/dasllama_unlisted.das", "options gen2\n") write_at(t, fixture, "modules/dasLLAMA/.das_module", - "\"dasllama\", \"dasllama_audio_embedder\", \"dasllama_vision_embedder\", \"dasllama_toyfam\", \"dasllama_gone\", \"dasllama_image\", \"dasllama_meta\"\n// \"dasllama_toyvis\" commented out must not count\n") + "\"dasllama\", \"dasllama_audio_embedder\", \"dasllama_vision_embedder\", \"dasllama_toyfam\", \"dasllama_gone\", \"dasllama_image\", \"dasllama_meta\", \"dasllama_vulkan_common\", \"dasllama_vulkan_classes\", \"dasllama_vulkan_prefill\", \"dasllama_blocks\"\n// \"dasllama_toyvis\" commented out must not count\n") // a moved layout constant and a changed layout helper, with the stamp left where it was write_at(t, fixture, "modules/dasLLAMA/dasllama/dasllama_image.das", dasllama_image_text("128l", "cur + 1ul", "2")) write_at(t, fixture, "modules/dasLLAMA/dasllama/dasllama_meta.das", DASLLAMA_META_RED) + // the KHR stage with a body and a superblock template with no KHR class, the slab constant off + // its slab, and a hand-built pipeline in the harness folder + write_at(t, fixture, "modules/dasLLAMA/dasllama/dasllama_vulkan_classes.das", VK_CLASSES_RED) + write_at(t, fixture, "modules/dasLLAMA/dasllama/dasllama_vulkan_common.das", "options gen2\n\nlet AR_MAX_DIM = 4_096l\n") + write_at(t, fixture, "modules/dasLLAMA/harness/hand_probe.das", "options gen2\n\nlet p = vkCreateComputePipelines(dev, null, 1u, addr(cp), null, addr(h))\n") let red = spawn_at(fixture, [path_join(fixture, "modules/dasLLAMA/REVIEW.das")]) t |> equal(1, red.rc, "planted fixture is red") for (needle in ["dasllama_image.das: the image layout closure changed", + "dasllama_vulkan_classes.das: khr_stage16 on KqCm2BatchT has a body", + "the add+rms row slab triple disagrees: AR_MAX_DIM 4096 (dasllama_vulkan_common.das), ArBase.row[8192], attn_dec_shape_ok's c.dim cap 8192", + "K5Cm2T has no `class K5KhrBatch : K5Cm2T`", + "K5Cm2T has no `[vk_dispatch(name = \"kq_batch_k5_khr_cls\"` stamp", + "khr_cls_ensure has no arm for K5Cm2T (ensure_kq_batch_k5_khr_cls(", + "khr_cls_set has no arm for K5Cm2T (set_kq_batch_k5_khr_cls(", + "khr_cls_enc has no arm for K5Cm2T (enc_kq_batch_k5_khr_cls(", + "hand_probe.das:3: hand-built Vulkan pipeline", "dasllama_meta.das:8: hand-listed Archive serializer with 2 field writes", "facade def hidden_fn appears in no tutorials/dasLLAMA/*.das", "facade def hidden_fn appears in no doc/source/reference/tutorials/dasLLAMA_*.rst",