diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_GEMM.md b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_GEMM.md index 130e81b525..d1cb8d2085 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_GEMM.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_GEMM.md @@ -147,7 +147,11 @@ hangs there too: when the device carries no `VK_NV_cooperative_matrix_decode_vec (`decvec_on` false) the served words go through `strip_decode_vector` (the capability, the extension and every load's `DecodeVectorFunc` operand removed, the scalar callback left to serve), after the override and before the shader module, so a dumped or overridden blob is -always the emitted, unstripped one. The seat is also the in-process A/B: `vkd_pipes_rebuild` +always the emitted, unstripped one. The scalar arm makes a MoE prefill window about half again as +long (the 35B-A3B's 512-row window 160 ms with the twin against 237 without on the RTX 5060 Ti), and +the tier warns at device init when the driver reports no such extension; which drivers list it, +and the reference exe's own two arms measured beside ours, are `followup_vulkan.md` item 45. The +seat is also the in-process A/B: `vkd_pipes_rebuild` marks every class slot stale, so the next ensure rebuilds it under whatever `decvec_on` says, which is how the `cm2:` probe runs both arms interleaved in one process. diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_RESIDENCY.md b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_RESIDENCY.md index beb440ec92..5c121b3af3 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_RESIDENCY.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_RESIDENCY.md @@ -52,13 +52,17 @@ tg128 reading (`-jit --for-debug-purposes -r 5 -p 512 -n 128`, `DASLLAMA_IMAGE=0 `DASLLAMA_GPU=1`, 16 threads) on the zen2 box's RTX 5060 Ti 16 GB, driver 616.56. **Where no OS answers, the auto arm's headroom is the larger of 2 GiB and 27% of the tier's -cap.** That share keeps a 16 GB card's plan near 10.7 GB and leaves an 8 GB card's plan where -the fixed term already put it; it declines every 27B file on a 16 GB card, and the ladder -behind it (the 9B hybrid at 6.8, 3.4 and 49.8 tok/s across 12.5, 12 and 11 GB plans) was taken -with every allocation tagged priority 1.0, the arm that made the planes pageable. Linux's Mesa -drivers fill `heapBudget` from the kernel's system-wide accounting and the NVIDIA driver fails -an allocation past the card out loud, so the share is a coarse stand-in there, not a measured -knee. +cap on Windows, and the 2 GiB floor alone elsewhere (`plan_headroom`).** On Windows that share +keeps a 16 GB card's plan near 10.7 GB and leaves an 8 GB card's plan where the fixed term +already put it; it declines every 27B file on a 16 GB card, and the ladder behind it (the 9B +hybrid at 6.8, 3.4 and 49.8 tok/s across 12.5, 12 and 11 GB plans) was taken with every +allocation tagged priority 1.0, the arm that made the planes pageable. The OS query is the +Windows memory manager's, so on Linux no OS ever answers; its Mesa drivers fill `heapBudget` +from the kernel's system-wide accounting and the NVIDIA driver fails an allocation past the +card out loud, so there is no demotion knee for the share to guard, and the share alone would +decline the 35B-A3B hybrid on a 16 GB card whose floor arms it at a reduced context. A decline +taken with no OS answer names the headroom it stood on and `DASLLAMA_GPU_VRAM_MB` as the pin +that reads no room. **Two guards stand whatever the plan decided.** The resident decode panics on an all-zero logits row (the first four tokens and every 256th are scanned) naming the over-commit, and at diff --git a/modules/dasLLAMA/BRINGUP.md b/modules/dasLLAMA/BRINGUP.md index 5dd4a7354c..645079e175 100644 --- a/modules/dasLLAMA/BRINGUP.md +++ b/modules/dasLLAMA/BRINGUP.md @@ -195,6 +195,13 @@ export LLAMA_BENCH_CLEAN=<...>/build-clean-cpu/bin/llama-bench export LLAMA_BENCH_STOCK=<...>/build-stock/bin/llama-bench ``` +A Vulkan reference (`-DGGML_VULKAN=ON`) is built with a glslc that knows +`GL_NV_cooperative_matrix_decode_vector` - Vulkan SDK 1.4.357 or later - and its device banner +reads `NV_coopmat2v` on a driver that lists the extension. Built with an older SDK the exe keeps +its 64-deep expert-tile k step over a scalar decode and under-reads llama.cpp by about a quarter +on a MoE model (the RTX 5060 Ti's Qwen3.6-35B-A3B UD-IQ2_XXS pp512 2080 against 2829); a banner +reading `NV_coopmat2` on such a driver is not a board-grade reference (`followup_vulkan.md` item 45). + ASR references + corpus (one tool, all legs): ```sh diff --git a/modules/dasLLAMA/ENVIRONMENT.md b/modules/dasLLAMA/ENVIRONMENT.md index f953b008c9..9b51151fda 100644 --- a/modules/dasLLAMA/ENVIRONMENT.md +++ b/modules/dasLLAMA/ENVIRONMENT.md @@ -35,7 +35,7 @@ Read by the inference engine itself, so these affect any program that loads a mo | `DASLLAMA_GPU` | flag | off | One switch for the measured-best GPU rail set; any DASLLAMA_GPU_* knob still overrides individually. | | `DASLLAMA_GPU_MOE_LAYERS` | number | -1 (auto) | How many MoE expert layers to hold resident on the GPU; -1 lets the upload walk place the split. | | `DASLLAMA_GPU_MOE_STREAM` | number | -1 (auto) | How many MoE layers to stream rather than hold resident; -1 is auto. | -| `DASLLAMA_GPU_VRAM_MB` | number | probed | Override the detected VRAM budget in MiB that sizes the resident expert stacks. | +| `DASLLAMA_GPU_VRAM_MB` | number | probed | Override the detected VRAM budget in MiB that sizes the resident expert stacks; a pinned cap takes no headroom and reads no OS room. | | `DASLLAMA_GPU_MIN_CTX` | number | built-in floor | Lower the context floor for arming the resident decode driver, for a short-context session on a small card. | | `DASLLAMA_GPU_RESIDENT` | flag | on | The whole-model resident driver for a model that fits the card, MoE included; 0 keeps the per-op rails (the A/B lever). | | `DASLLAMA_GPU_DN` | flag | follows DASLLAMA_GPU | DeltaNet rail on the GPU. | diff --git a/modules/dasLLAMA/dasllama/dasllama_env.das b/modules/dasLLAMA/dasllama/dasllama_env.das index 3acada1f87..09d5d5e445 100644 --- a/modules/dasLLAMA/dasllama/dasllama_env.das +++ b/modules/dasLLAMA/dasllama/dasllama_env.das @@ -95,7 +95,7 @@ struct public GpuTierEnv { gpu_moe_stream : Option @clarg_default_doc = "probed" - @clarg_doc = "Override the detected VRAM budget in MiB that sizes the resident expert stacks." + @clarg_doc = "Override the detected VRAM budget in MiB that sizes the resident expert stacks; a pinned cap takes no headroom and reads no OS room." gpu_vram_mb : Option @clarg_default_doc = "built-in floor" diff --git a/modules/dasLLAMA/dasllama/dasllama_gpu_resident.das b/modules/dasLLAMA/dasllama/dasllama_gpu_resident.das index 00af5d8c0a..c7d7ff7774 100644 --- a/modules/dasLLAMA/dasllama/dasllama_gpu_resident.das +++ b/modules/dasLLAMA/dasllama/dasllama_gpu_resident.das @@ -515,13 +515,18 @@ def private rdec_scratch_bytes(t : Model) : int64 { + 8l * hid + 4l * layer_head_size(c, 0l)) + c.vocab_size * 4l + RDEC_MISC_BYTES + moe } +//! The auto arm's headroom where no OS answers: the WDDM share on Windows (the demotion collapse +//! the constant's comment measures), the fixed floor elsewhere - a Linux driver refuses an +//! allocation past the card out loud instead of demoting it. +def plan_headroom(budget_bytes : int64; wddm : bool) : int64 => wddm ? max(RDEC_VRAM_HEADROOM, budget_bytes * RDEC_VRAM_HEADROOM_PCT / 100l) : RDEC_VRAM_HEADROOM + //! The room a plan may fill: the tier's cap less its headroom, capped by the OS's room when the OS //! answers - its per-process budget less what the other processes hold on the adapter and the reserve. //! A pinned cap takes no headroom and reads no room. [arch(at="../ARCHITECTURE_GPU_VULKAN_RESIDENCY.md#resident-plan")] def plan_room(budget_bytes : int64; pinned : bool; os : tuple) : tuple { let answered = !pinned && os.budget > 0l - let headroom = (pinned || answered) ? 0l : max(RDEC_VRAM_HEADROOM, budget_bytes * RDEC_VRAM_HEADROOM_PCT / 100l) + let headroom = (pinned || answered) ? 0l : plan_headroom(budget_bytes, get_platform_name() == "windows") var usable = budget_bytes - headroom var others = 0l if (answered) { @@ -612,7 +617,12 @@ def resident_plan(t : Model; seq_cap : int64; kdt, vdt : KVDtype) : ResidentPlan // a shorter context is the remedy that actually works — weights are fixed, KV is not let kv_fit = max(usable - p.weight_bytes - p.scratch_bytes, 0l) let cap_fit = (krow + vrow) > 0l ? kv_fit / (krow + vrow) : 0l - let room = p.others_bytes > 0l ? " - other processes hold {p.others_bytes >> 20l} MB of the card" : "" + var room = "" + if (p.others_bytes > 0l) { + room = " - other processes hold {p.others_bytes >> 20l} MB of the card" + } elif (p.headroom_bytes > 0l) { + room = " - the OS reported no room, so {p.headroom_bytes >> 20l} MB stands as headroom; DASLLAMA_GPU_VRAM_MB= pins the cap and reads no room" + } p.reason = (cap_fit >= 512l ? "needs {total >> 20l} MB of {usable >> 20l} MB at ctx {seq_cap} - would fit at ctx {cap_fit}{room}" : "needs {total >> 20l} MB of {usable >> 20l} MB (weights alone {p.weight_bytes >> 20l} MB){room}") diff --git a/modules/dasLLAMA/dasllama/dasllama_vulkan_common.das b/modules/dasLLAMA/dasllama/dasllama_vulkan_common.das index 8f2a2f2b9d..9841c68f35 100644 --- a/modules/dasLLAMA/dasllama/dasllama_vulkan_common.das +++ b/modules/dasLLAMA/dasllama/dasllama_vulkan_common.das @@ -1268,6 +1268,9 @@ def vk_moe_init : bool { g_init_failed = false g_gpu.full_sg_on = g_env_vulkan.vk_fullsg && compute_full_subgroups_supported(g_gpu.phys) to_log(LOG_INFO, "dasLLAMA vulkan tier: device ready (subgroup {int(sg)}, {g_gpu.rows_per_wg} rows/wg{g_gpu.full_sg_on ? ", full subgroups (DASLLAMA_VK_FULLSG)" : ""}{g_gpu.decvec_on ? ", four-wide decode (DASLLAMA_VK_DECVEC)" : ""})\n") + if (g_gpu.has_coopmat2 && g_env_vulkan.vk_decvec && !g_gpu.decvec_on) { + to_log(LOG_WARNING, "dasLLAMA vulkan tier: the driver reports no VK_NV_cooperative_matrix_decode_vector, so the cm2 tiles run the scalar decode arm - a MoE prefill window runs about half again as long as on a driver with the four-wide arm\n") + } if (dasllama_noisy()) { // storage-range is the limit that actually binds the arena; allocs/descriptors do not var lp : VkPhysicalDeviceProperties diff --git a/modules/dasLLAMA/followup_vulkan.md b/modules/dasLLAMA/followup_vulkan.md index 9ef2ba4d28..d8db765c12 100644 --- a/modules/dasLLAMA/followup_vulkan.md +++ b/modules/dasLLAMA/followup_vulkan.md @@ -969,3 +969,74 @@ module) is independent and can land any time - it is pure structure. probe twins (both ranked on before/after `DASLLAMA_GPU_PROF=1` profiles across processes, 4.3 -> 2.4 ms per 30B window and 490 -> 440 us per twin token; ruled 2026-09-09 to ship as stated claims). + +44. **The whole-model driver admits K-quant planes only off a repacked load.** Every plane the + driver places - the attention quads, the deltanet planes, the expert stacks and the shared + expert's triple - passes `kq_servable`, and that predicate (`dasllama_common.das`) is the CPU + fused chains' rule: a superblock format serves only when `t.kq_repacked` is set, which + `select_matmul_backend_for_load_` sets for a repacking backend alone. On a box that selects + the portable backend a K-quant model never reaches the driver - the Qwen3.6-35B-A3B + UD-IQ2_XXS declines on "layer 0's shared expert carries a format (2/2/3) the dense rail does + not serve" (a reviewer's reading on Linux, 2026-09-10) - while the device gather + (`moe_gpu_gather_stack_kq`) takes the layout flag as an argument and reads the disk order + too. The driver wants a device-side servability predicate (a format the tier has kernels + for, in either layout) in place of the CPU chains' rule at its four sites; done when a + portable-backend load of a K-quant model arms the driver and its resident-vs-CPU parity + file holds, which needs a box whose backend does not repack (none here: this box's backend + is x64-gen). + +45. **The resident MoE chain does not scale with the card.** Measured 2026-09-10 on a rented Linux + RTX 5080 (driver 580.173, 8 vCPU, `-t 8`) against the RTX 5060 Ti (Windows, driver 616.56), + both on the Qwen3.6-35B-A3B UD-IQ2_XXS (sha256 2e8f5f70..., the model-set row's), the bench's + `-jit --for-debug-purposes -r 3 -p 512 -n 128`: llama.cpp b10660 5352.6 / 147.7 there against + 2853.1 / 71.6 here (1.88x / 2.06x for a 2.14x bandwidth step, 36 -> 84 SMs), ours 3361.1 / 139.8 + against 2995.8 / 107.7 (1.12x / 1.30x) - 0.63x / 0.95x of llama.cpp on the 5080 where the 5060 Ti + reads 1.05x / 1.50x (b10886 5461.2 / 148.1 on the same box: the version moves 2%). The per-role + profile (`DASLLAMA_GPU_PROF=1`, both boxes) splits it three ways. + (a) The four-wide decode twin is off on Linux: `cooperative_matrix2_decode_vector_supported` + reads `VK_NV_cooperative_matrix_decode_vector`, which the 580.173 driver does not list (it lists + `VK_NV_cooperative_matrix`, `_matrix2` and `_vector`; the Windows 616.56 driver lists it), so + every cm2 tile runs the scalar arm - and so do llama.cpp's there: its shaders carry the extension + and it strips them at pipeline creation when the driver lacks it (`ggml_vk_strip_decode_vector`), + running its expert tiles' k step at 32 on that arm and 64 with the four-wide one (`mmqid_bk`; its + PR 23991 raised the step together with the four-wide B loads, neither alone consistently faster). + Same box, same source (b10660), the 5060 Ti, the 35B pp512: llama.cpp four-wide 2828.9 (the + record's 2853.1); scalar at k step 32 2385-2393 (`GGML_VK_DISABLE_COOPMAT2_DECODE_VECTOR=1` on + either build below); scalar at k step 64 2080 - the DEFAULT of a build whose glslc does not know + the extension (Vulkan SDK 1.4.350's; 1.4.357's does): its device flag reads the driver's extension + list, not the shader build, so that build keeps the 64-deep step over a scalar decode, under-reads + the reference by 27% and says `NV_coopmat2` in its device banner where the four-wide build says + `NV_coopmat2v`. Ours: twin 2995.8, scalar (k step 64, `DASLLAMA_VK_DECVEC=0`) 2049, the window + 160181 -> 236883 us. Per gate/up plane at the 30B's shape (its `GGML_VK_PERF_LOGGER=1` + `MUL_MAT_ID iq2_xxs` row against our `moe:iq2xxs` probe at uniform buckets): theirs 612 / 829 / + 1014 us (four-wide / scalar k32 / scalar k64), ours 730 / 1330-1380 (twin / scalar k64); in the + window's real schedule our e_gate plane reads 695 twin and 1229 scalar. Arm- and k-step-matched + the two engines read the same whole model (2049 against 2080), so llama.cpp's whole Linux edge on + this axis is the 32-deep k step on its scalar arm (1.15x for it), while the tiles alone stay + 1.2-1.3x behind at either matched arm (695 against 612, 1229 against 1014): its scalar decode + computes a pair's shared work once and selects the element last - the form the driver's own + two-wide commoning relies on (its PR 23541) - where ours reads its scale from the `ws` plane and + shifts per element. The expert-tile probe reads the same 0.71 ms per iq2xxs gate/up plane on the + 5080 no-twin as the 5060 Ti twin-on (0.73; twin-off 1.33); arm-matched, the tiles scale 1.80-1.90x, + qkv 1.85, conv 1.94, gather 1.82 (SMs 2.33x). Levers, in order: a 32-deep k step for the scalar + arm (a template constant; the probe twin-off at 32 against 64, and twin-on at both to check the + twin still wants 64), then the decode body's pair form. The tier warns at device init. + (b) Three window roles stay flat on the wider card whatever the arm: the deltanet scan (13037 -> + 12654 us, a serial recurrence over chunks - 8.7% of the 5080's window), the shared expert's + k5/k6 tiles (sh_gate 3518 -> 3379, sh_down 1582 -> 1452 - one dispatch per layer whose grid is a + single wave; the tile pick at 84 SMs is the first suspect) and the router (2276 -> 2371: 32 fixed + workgroups). And the routed tiles' real schedule (87 buckets, the largest 446 rows, 105 ladder + pieces per plane - one workgroup each over the expert's whole width) is one wave on either + card: splitting a piece across column groups and an LPT order over the pieces are the levers + the uniform probe (1536 workgroups) cannot show. + (c) Decode is the per-dispatch floor: the expert GEMV reads 3.3 MB per layer in 8.5 us on the + 5080 (3.4 us of transfer at 960 GB/s, ~5 fixed) and 11.3 us on the 5060 Ti (7.4 + 3.9); some 600 + dispatches per token carry it - about 3 ms of a 7.8 ms token - and every small kernel reads + SLOWER on the 5080 (sh_gate/sh_up/sh_down 0.87/0.84/0.92, ar2 0.87, topk 0.93, the deltanet step + 0.76, attention 0.88), while the GEMV probe scales exactly with bandwidth (iq2xxs 410 -> 881 GB/s + at the expert shape). The lever is fewer dispatches per layer: gate and up in one, router and + top-k in one, the shared expert's three GEMVs as extra slots of the expert dispatch, the + activation folded into the down GEMV; the host is 176 us of the token on the pod (2%). + The pod's card ran 2910-2925 MHz at 250-263 W of its 307 W limit under the probe, so no clock + cap stands behind (b) or (c). Research before any kernel work; each of (b) and (c) is a slice + of its own. diff --git a/modules/dasLLAMA/tests/test_gpu_serving_declines.das b/modules/dasLLAMA/tests/test_gpu_serving_declines.das index ffbcaca73f..4c34602622 100644 --- a/modules/dasLLAMA/tests/test_gpu_serving_declines.das +++ b/modules/dasLLAMA/tests/test_gpu_serving_declines.das @@ -219,8 +219,11 @@ def test_plan_room(t : T?) { t |> run("the plan's room: the tier cap less headroom, capped by the OS's room when it answers") @(t : T?) { let gib = 1_073_741_824l let silent = (budget = 0l, usage = 0l, adapter_dedicated = 0l, adapter_shared = 0l) + t |> equal(16l * gib * 27l / 100l, plan_headroom(16l * gib, true), "no OS answer on WDDM: 27% of a 16 GiB cap, past the 2 GiB floor, is the headroom") + t |> equal(2l * gib, plan_headroom(4l * gib, true), "no OS answer on WDDM: the 2 GiB floor wins on a small cap") + t |> equal(2l * gib, plan_headroom(16l * gib, false), "no OS answer off WDDM: the 2 GiB floor alone - the driver refuses an over-commit out loud, nothing demotes") let no_os = plan_room(16l * gib, false, silent) - t |> equal(16l * gib * 27l / 100l, no_os.headroom, "no OS answer: 27% of a 16 GiB cap, past the 2 GiB floor, is the headroom") + t |> equal(plan_headroom(16l * gib, get_platform_name() == "windows"), no_os.headroom, "no OS answer: the plan takes this platform's headroom") t |> equal(16l * gib - no_os.headroom, no_os.usable, "no OS answer: the cap less the headroom") t |> equal(0l, no_os.others, "no OS answer: nobody else is counted") let os = (budget = 14l * gib, usage = 1l * gib, adapter_dedicated = 5l * gib, adapter_shared = 0l)