Conversation
* vulkan: optimize m=1 mul_mat by swapping A/B * vulkan: Improve small M perf Allow split_k with small M. Make small vs med tile selection (for coopmat2) depend on M, not just N.
argsort had a data race in the inner loop, which VVL caught. But I don't think this was causing failures in practice. argsort_large has OOB accesses which might explain the failures in CI, but I couldn't reproduce it locally and I don't think it's a convincing explanation of the failures.
* HIP: enable mma FA for head size 256 on RDNA4, tune configs Assisted-by: Claude Assisted-by: Codex * HIP: prefer whole-tile FA grids over stream-k on AMD WMMA Assisted-by: Claude Assisted-by: Codex * revise stream_k logic * revise kernel selection logic --------- Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* server: fix speculation after an image Pass the actual position to the drafter after an image, instead of the token count. Affects every drafter, not just DFlash. * rename draft n_past to pos0 n_past is used to denote number of tokens and this parameter is meant to be a position
…ggml-org#28630) * model : fix MTP context kv cache allocation for deepseek2, glm4moe, cohere2moe architectures (ggml-org#28626) * model: add inverse architecture gating and comprehensive architecture testing for mtp layer filtering * model : slim NextN filter comment, drop test-llama-archs changes
…< 1024 (ggml-org#28692) * metal : fix idle threads in the remaining iq mul_mv kernels for ne00 < 1024 Generalize the row split from ggml-org#28086 to the six other kernels that use the same lane-to-block mapping: iq1_s, iq1_m, iq2_xxs, iq2_xs, iq2_s and iq3_s. Each of them assigns one 32-element chunk per thread, so when a row has fewer than 32 chunks the rest of the simdgroup is idle. When nb32 < 32 and nb32 divides 32, 32/nb32 threads now share each chunk and each takes a slice of the rows, reusing the FC_mul_mv_split function constant and the dispatch wrapper introduced for iq3_xxs. The plain path is untouched: wide matrices keep one thread per chunk and N_R0_<TYPE> = 4. Only the split path uses N_R0_<TYPE>_SPLIT = 8. The K-quants have the same idle-thread issue but a different lane mapping, so they are left for a separate change. * metal : offset the src0 row pointer once in the iq mul_mv kernels q2, dh, sc, qh and signs are all derived from xr, so the row slice offset only has to be applied to xr. * metal : fold iq mul_mv row split into offset0 Compute row0 and row1 before initializing the source pointers and apply the row slice directly to offset0. This keeps x and its derived pointers on the existing path while applying the split row offset once.
) * metal : rework fusion patterns into a single table All fusable op patterns for the Metal backend are now declared once in a fusion table (ggml-metal-fuse.cpp) and consumed by both the graph optimizer (ggml_metal_fuse_max, packing) and the op encoders (ggml_metal_fuse_next, compute). The two phases share the same pattern table plus ggml_can_fuse_subgraph_ext for the structural checks, and differ only in the mode used for the pattern check (STRUCTURAL at optimize time, since tensors are not allocated yet, and FULL at compute time, including Metal buffer placement). This also protects the snake activation (MUL + SIN + SQR + MUL + ADD) from being reordered during graph optimization, which was previously unprotected. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : fix absolute output indices in fusion patterns ggml_can_fuse_subgraph_ext expects the outputs array to contain absolute graph node indices (it indexes cgraph->nodes[outputs[i]]), but the fusion table query was passing a relative index (n_ops - 1). As a result the last node of every pattern was not recognized as an output and was subjected to the elidable use-count check, which failed for essentially all fusions. This silently disabled the norm/MUL fusion and caused a ~5% token-generation regression. Pass the absolute graph index of the last node instead. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : fuse gated_delta_net with cache cpy Add GGML_METAL_FUSE_GDN_CACHE to the fusion table: when the gated_delta_net kernel is followed by a cpy that scatters its recurrent state snapshots into the KV cache, the kernel writes the snapshots straight into the cache buffer and the trailing cpy is elided. The gdn output has other consumers (the attn scores view), so unlike the elision-chain patterns this is not a simple chain: a 'raw' flag on the fusion pattern skips the generic chain/shape and ggml_can_fuse_subgraph_ext checks, making the pattern-specific check callback the sole validator. Packing (ggml_metal_fuse_max) now matches on the same view-transparent node sequence that the compute phase uses, so the gdn + cache cpy group is packed along with any intermediate views and stays adjacent through the reorder. The fused cpy is a view consumer of the gdn (it writes the cache directly), so its mem-range is skipped in the encoder; the skip is restricted to CPY nodes consuming the previous fused node through a view so other fusions are unaffected. Add test_gated_delta_net_cache_fusion and register 5 cases. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : drop is_view_consumer mem-range skip The is_view_consumer skip was carried over from the upstream gated_delta_net cache-fusion draft, but it is not needed: keeping the elided cpy's mem-range in the concurrency tracker only ever adds a (conservative) memory barrier at the fusion point. It can never remove a barrier, so it cannot introduce a race. The worst case is one spurious barrier per gdn+cache-cpy fusion, which is within run-to-run noise on Qwen3.5-0.8B Q8_0. Dropping the check keeps the mem-range loop uniform for all fused groups. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : rename gated_delta_net fused state output args Rename the fused cache-write kernel argument to match the rest of the kargs: state_out_stride -> nb_out (and widen it to uint64_t), and the local buffer id bid_state_out -> bid_out. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : rename raw fusion flag to unsafe raw did not convey that the flag opts a fusion pattern out of the generic elision-chain safety net (ggml_can_fuse_subgraph_ext + chain/shape checks). rename it to 'unsafe' to make explicit that the pattern's check callback is the sole validator and must re-establish the safety guarantees itself. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : tidy fusion pattern checks and table - const-correct ggml_metal_fuse_outputs buffer - annotate unused check-callback parameters - drop a redundant size_t cast - align the ops/table initializers and add blank-line separation Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : add generic fusion stats via ad-hoc proc-address API Add a device-owned fusion context that lets a test tool count how many times each fusion pattern fires and toggle fusion. It is exposed through the ad-hoc ggml_backend_reg_get_proc_address mechanism with generic names so the testing tool is backend-agnostic: - ggml_backend_fusion_stats_init: start collecting fusion stats; when a context is created afterwards it registers the labels/counters and encodes single-threaded (n_cb == 0) so the counters are race-free - ggml_backend_fusion_stats_reset / _get_stats / _set_enabled The context lives on the metal device (not on the last backend context), so counters accumulate across contexts and reads are always consistent. The enable/disable toggle is initialized from GGML_METAL_FUSION_DISABLE and can be overridden by the test through set_enabled. Labels are synthesized from the fuse table via ggml_metal_fuse_label (e.g. "GATED_DELTA_NET+CPY"). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : add fusion count regression test with per-backend baseline test-fusion runs every dummy model generated by test-llama-archs on a single backend (single-threaded encoding, n_cb == 0) with fusion enabled and disabled, and for each mode (prefill / decode) reports the per-fusion counters and the NMSE between the fused and unfused logits, plus the NMSE against a CPU reference. A fusion pattern that silently stops matching (or fires when it should not) is caught as a regression by comparing the counters against a committed per-backend TSV baseline: - --record writes the golden baseline, --check (default) validates it - the unfused run doubles as a control: its counters must be all-zero - NMSE is skipped when it is NaN or the arch is already broken on the device (e.g. plamo2 on Metal), so the count check is the hard gate - baseline counts depend only on graph structure, not weights (verified stable across weight seeds) - the fusion stats API is resolved through the ad-hoc get_proc_address mechanism with generic names; a backend that does not export it makes the test fail with an error The committed MTL0.tsv baseline covers 110 dummy archs (298 rows). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : rename fusion api helpers to match stats_init signature Align the test with the ad-hoc fusion stats API: fusion_stats_init no longer takes an enable bool (stats are turned on by calling it), so the proc-address wrappers and typedefs are renamed to the api_* convention. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : rename backend to device in fusion test CLI The fusion test operates on a compute device (e.g. MTL0), not a backend, so rename the --backend argument to --device and the backend_name variable to device_name. Keep "backend" where it refers to the ggml backend interface (the ad-hoc proc-address mechanism). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : add --model and --help to fusion test --model FILE runs the fusion regression test over a single model file instead of enumerating a --models DIR. --models and --model are mutually exclusive. Also add a --help/-h option that prints the usage. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : use backend base name for fusion baseline output The fusion test is invoked with a specific device name (e.g. MTL0), but its output - the recorded baseline and the header it writes - should be named after the backend base name (e.g. MTL, via ggml_backend_reg_name), since the counters depend on the backend, not on the specific device index. Rename the committed baseline to MTL.tsv. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : run fusion test from ci instead of ctest The fusion test needs Metal and generates a lot of dummy models, so it does not belong in the generic ctest suite. Move it to ci/run.sh as gg_run_test_fusion, gated on GG_BUILD_METAL like gg_run_test_llama_archs_tensor_split: it generates the dummy models with test-llama-archs -o and then validates the fusion counts against the committed baseline. test-fusion.cpp is still built (llama_build) but no longer registered as a ctest. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : align fusion baseline TSV columns Pad the TSV fields to fixed widths so the columns line up regardless of the variable arch and fusion-label lengths, and trim each field on parse so the padded file is still accepted. Regenerate the committed MTL.tsv baseline in the padded format (data unchanged, verified identical modulo padding). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : widen label column and align fusion TSV header Give the label column more room (28 chars) and fix the column header widths so they match the data rows (moe/mode/label), keeping the header aligned with the values. Regenerate the MTL.tsv baseline in the new format (data unchanged). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : switch fusion baseline from TSV to CSV Use comma-separated values like the rest of the project, keeping the padded, aligned columns. Split on ',' and trim on parse. Rename the committed baseline to MTL.csv (data unchanged, verified identical modulo padding/separator). Update the ci/run.sh check path accordingly. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * cont : rebase + update MTL stats * tests : avoid graph reallocations for some archs * metal : tidy fusion debugging context and op init - simplify the shared fusion debugging context comments - shorten the ggml_metal_fusion struct comment - align the ggml_metal_fuse struct fields and comments - move the fusion parameter of ggml_metal_op_init right after dev Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : dedup fusion baseline into any mode prefill and decode always produce the same per-graph fusion count, so store a single row per label with mode = "any" and the per-graph count instead of two rows. this halves the baseline size and keeps the check stable. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * ci : move fusion model generation to a separate step the dummy models generated by test-llama-archs are reused by other tests, so generate them once in their own step instead of inside test_fusion. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : bump nmse thold * models : fix plamo2 graph * tests : remove "skip" logic from test-fusion * tests : set qwen3tts dummy vocab to codec head size the dummy qwen3tts model used a vocab of 4096 while the codec head is 3072, so the graph padded the output with -inf which made the NMSE in test-fusion produce NaN. use the exact codec head size instead so the padding is not generated at all. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : regen fusion baseline reflect the plamo2 graph fix, which changed its fusion pattern split (RMS_NORM+MUL 11->10, RMS_NORM+MUL+ADD 3->4; same total). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * ci : skip dummy model generation on OpenVINO test-llama-archs does not build on the OpenVINO platform, so do not try to generate the dummy models there. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * cont : minor * tests : enable test-llama-archs on windows * cont : disable on windows + workaround * metal : naming nits * test-fusion : add instructions to update baseline * context : fix Kimi-K3 graph reserve * fusion : update MTL * cont : fix naming * metal : rework fusion info storage Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : align fusion info API Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use opaque fusion handle in ad-hoc API Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * ci : move fusion test to dedicated workflow Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * cont : run only on ggml changes * cont : simplify * fusion : remove multi-output stuff for now * ci : fix typo
* scripts : add initial profiling script (wip) * src : add precompile headers (PCH) for models.h * common : add common.h as PCH * ggml : add PCH for ggml-impl.h * mtmd : use PCH for models.h * scripts : add script to build with Server/Tools/Tests * server : add PCH for common.h * docs: add profiling progress notes (wip) * ggml : add exclude for GCC + SVE on ARM Refs: https://github.com/ggml-org/llama.cpp/actions/runs/33393906061/job/99493756214?pr=28091 * ggml : attempt to fix use of std::hardware_destructive_inference_size Refs: https://github.com/ggml-org/llama.cpp/actions/runs/33396221677/job/99501265689?pr=28091 * squash! ggml : attempt to fix use of std::hardware_destructive_inference_size Add a version check for GCC 12 to conditionally apply the `-Winterference-size` pragma. * editorconfig : exclude profiling reports dir This directory will not be included in the merge later and this commit can be ignore at that point. Just fixing to keep CI happy. * ggml : skip PCH for gcc on non-x86 architectures * tests : add PCH for peg-parser/tests.h There are 7 peg-parser tests that can share one PCH instead of then each parsing the full tests.h. * common : add PCH for chat.h * docs : update linux build profiling full results Just updating after a number of PCH additions. These are not exact figures and will vary a bit from run to run, but they give a general idea of the performance impact of PCH. * cmake : introduce unity build for models This commit introduces a unity build for the models to improve compilation time. The improvements were roughly the following: ```console +------------------------+-----+------------+------------+------------+ | Build | TUs | Frontend | Backend | Total | +------------------------+-----+------------+------------+------------+ | Full, master | 396 | 811.0 s | 692.2 s | 1,503.2 s | | Full, with PCH | 405 | 380.0 s | 664.7 s | 1,044.7 s | | Full, with PCH + UB | 264 | 357.7 s | 635.7 s | 993.4 s | +------------------------+-----+------------+------------+------------+ TU = Translation Unit. Full = includes Server, Tools, and Tests. PCH = precompiled headers. UB = unity build for models. ``` * docs : update linux profiling table with unitiy build results * docs : update mac profiling results to include unity build [no ci] * docs: remove profiling reports * scripts : merge build profile scripts into one script I was lazy before and just copied the first script to enable Tests, Server, and Tools. This now merges them into a single script. * Revert "editorconfig : exclude profiling reports dir" [no ci] This reverts commit 2922a12. * src : rename ggml_view_2d_slice to gemma3n_view_2d_slice This is to be consistent with the rename in gemma4.cpp which was required to avoid a name clash. * cmake : add build profile script for windows [no ci] This commit adds a port of the scripts/build-profile.sh script to windows powershell. This was developed on Windows on ARM but should work on X64 as well but needs to be tested there as well.
kernel_mul_mm_id splits its NR1 = 32 token tile into two 16-row halves and skips the upper half when the expert did not fill it, on both the tensor and simdgroup paths. The tB extents are corrected to (NK, NR1H) for the [NR1][NK] row-major tile. The B tile is staged unconditionally, as on master: rows past nr1 restage a clamped duplicate of a valid row, lie in the output-row dimension so they never contribute to a valid row, and are dropped by the final store loop. test-backend-ops: re-draw the expert ids between perf iterations of test_mul_mat_id so MoE perf numbers are not warm-cache, and add token-tile boundary coverage using n_used == n_mats, which routes every token to every expert so each expert receives exactly n rows; n = 32, 33, 47, 48, 49 reach mul_mm_id and leave a last tile of 32, 1, 15, 16 and 17 rows.
…org#28759) The expected success table holds when the four requests enter the shared pool together. On a loaded runner they are admitted tens of milliseconds apart, the slot lifetimes overlap differently and the pool overflows while a short request is still resident. The decode failure aborts every slot, so a request the table marks as successful comes back with the context error instead of its generation. Such a request now passes on that error too, while any other status, a different error or a truncated generation still fails the test.
…g#16234) * Test for nrc=2 as well | i8mm kernels * Trigger only on supported HW * Remove trailing whitespace * Address review comment * test: properly prepare nrc=2 inputs with independent data per row * tests : make nrc=2 dot product inputs distinct Assisted-by: Kiro * tests : use non-trivial strides in nrc=2 dot product test * tests : fail nrc=2 dot product test on non-finite errors
* ci : run test-backend-ops as a dedicated gg test Run test-backend-ops as a separate gg test in ci/run.sh so it is executed outside ctest. With GG_BUILD_HIGH_PERF it keeps the existing CPU-only invocation (-b CPU); otherwise it runs all available backends without a backend filter. Remove the dedicated backend-ops workflow and keep test-backend-ops as a built target that is not registered with ctest to avoid duplicate runs. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * ci : run test-backend-ops earlier and enable high-perf on kleidiai Move the test-backend-ops gg test before test-llama-archs. Enable GG_BUILD_HIGH_PERF and LLAMA_ARG_THREADS on the Graviton4 KleidiAI job and use the standard self-hosted results/mnt paths. Add TODO markers for decoupling tests from libllama. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * ci : run test-backend-ops in parallel Pass -j $(nproc) to test-backend-ops in both high-perf and all-backend modes. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * ci : disable parallel tests for ROCm * cont : disable parallel tests with MoltenVK
This commit fixes an issue that I introduced when adding PCH (precompiled headers) in Commit 3bcfeb7 ("cmake : add PCH and unity build to improve build times (ggml-org#28091)". See linked issue for details. Co-authored-by: mjungnickel18 Co-authored-by: Pascal <admin@serveurperso.com> Resolves: ggml-org#28758 Refs: https://github.com/ggml-org/llama.cpp/actions/runs/34592933983/job/103262608990#step:9:1284
* server: refactor subproc handling * fix Windows build * download: keep concurrent downloads of one blob apart Every process writes the same path + .downloadInProgress, so a second download of the same blob finds that file, takes it for its own partial transfer and asks for the bytes after it, which produces a corrupt result. The in-progress file now carries the pid of the process writing it. std::rename also replaces an existing destination on POSIX but fails on Windows, so a download whose blob appeared in the meantime is dropped after every retry and an etag rewrite silently keeps the old value. std::filesystem::rename has the POSIX behaviour everywhere, and the error now carries the reason reported by the system. * Revert "download: keep concurrent downloads of one blob apart" This reverts commit 917b83f. * tests: serialize the router tests that download the same model Parallel workers share one cache, so the two tests fetch the same blob into the same in-progress file and race to rename it. They now take a file lock around the download, like the session fixture does for the preset models. * Revert "tests: serialize the router tests that download the same model" This reverts commit c368a4a. --------- Co-authored-by: Pascal <admin@serveurperso.com>
* ggml-webgpu: Update to a recent version of Dawn * No module scanning * Accept review suggestion to update comment Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com> --------- Co-authored-by: Masashi Yoshimura <yoshimura.masashi.frbs@gmail.com>
…rg#28589) * hex-row-split: add support for multi-device row spliting Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> * hex-mdev: add work splitting to fused kernels * hex-mdev: use mdev_ prefix for all multi-device state * hex-mdev: make device configuration more expressive to support device groups * hex-mdev: fix mdev session init * hex-mdev: fused nx (2x,3x) matmuls must update row counts for each w/o * hex-mdev: fix MUL_MAT work partitioning bugs introduced by mdev * hex-cont: fix crashes with new tests due to wrong striding * hex-mdev: move fences after l2flushes * hex-cont: fix work splitting for mnpu -- align chunks to cachelines * hex-mdev: fix CPY tests with multi-dev * hex-mmid: fix work partitioning with mnpu * hex-mm: fix test failures with mdev * hex-binary: fix work partitioning for mdev * hex-argsort: fix mdev partitioning * hex-mdev: fix work partitioning and general updates for all simple ops * hex-fa: fix mdev work splitting issues * hex-mdev: fixing more failing ops test * hex-mdev: update the rest of the ops * hex-mdev: refactor all mdev splitting logic to be contained within if (mdev_count > 1) {...} * hex-mdev: fix macros * hex-mdev: simplify session flush logic * hex-sync: fix recursion in session flush * hex-mdev: factor out fence buffer and allocator * hex-fence: make fence allocation more robust with reserved slots for mdev * hex-mdev: keep all mdev state in htp_mdev_group * hex-mdev: further cleanup mdev group handling at the host * hex-mdev: update group idx in the opbatch before serializing * hex-batch: remove separate op_pending and use batch_req/rsp_seq * hex-async: workaround another missing tensor_init in ggml-meta * hex-fence: cleanup and robustify fences and error handling in multi-device scenarios * hex-ar: improve ALLREDUCE error handling * hex-async: robust error handling for op_cpy_fence * hex-async: use seq0 from allreduce context to allocate fence_seq * hex-mdev: fix remaining issues with fence and barrier clearing in CPY_FENCE * hex-misc: realign macros and fix misplaces trace events * hex-misc: align macros * hex-mdev: fix unclone buffer re-entrancy * hex-glu: fix mdev partitioning logic * hex-mdev: make buffer uncloning/cleanup work with tensor-split scenarios * hex-mdev: tighten up the can_split check in act-ops * hex-mdev: factor out common bits of the partitioning logic * hex-mm: minor realignment of the macros * hex-bufs: fix incorrectly placed assert for MAX_BUFS * hex-pad: tighten up gating checks for PAD * hex-kparams: make sure all kernels properly use kparams->n_threads * hex-docs: update user and developer docs with new features and detailed guide for ops development * hex-scripts: update run script to properly parse dev groups * hex-misc: formatting * hex-sess: minor cleanup for session init * hex-ar: fix vtcm size calc in allreduce kparams * hex-scripts: fix flake8 warnings * hex-rope: update ROPE to support mdev work split * hex-ops: remove redunant checks and minor reformat * hex-dev-guide: update dev-guide to avoid redundant null checks * hex-async: improve event_wait, event_sync and fence implementations * hex-async: remove synchronous flush from event_sync * hex-async: symplify fence recovery protocol and make sync more robust * hex-async: futher simplify error recovery for fences * hex-err: return status instead of just -1 * hex-async: print all seq nums in hex * hex-async: make sure fences flush dirty ranges * hex-async: add dirty ranges merging to reduce fence flushes * hex-async: properly sync before freeing the event * hex-async: make sure fence owner session is not overriden * hex-async: more fence write order more robust * hex-async: make sure not to fuse ALLREDUCE+ADD if their dsts overlap * hex-fusion: cleanup redundant checks --------- Co-authored-by: Alexander Lu <alexlu@qti.qualcomm.com>
Walk the binding offset back until the distance to the tensor is a whole number of blocks, so block quantized views get a valid element offset in the shader.
…a8_bin` (ggml-org#28677) * opencl: add A8 Q4_K non-MoE binary kernel * opencl: fix layout compatibility * opencl: rename binary kernel selection helpers --------- Co-authored-by: Li He <lih@qti.qualcomm.com>
…g#28747) The child writes its state commands on stdout while the logger writes on stderr, and both share a single pipe. The logger emits the trailing color reset after the newline of a debug, warn or error entry, so that escape sequence has no newline of its own and the router reads it glued in front of the next command. The line prefix check then fails and the command is forwarded as a log line instead of being handled, which leaves a finished download stuck in the downloading state. Writing the command with a leading newline closes the pending line so it always starts at a line boundary.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
…org#29320) Restore get_cache_directory() as fs::path as string() can be lossy on Windows Partially reverts ggml-org#29125 Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Brings the fork from upstream b10760 to the v0.5.0 release (386 upstream commits). Every fork feature is kept; where both sides changed the same code the two changes are combined, not chosen between. Textual conflicts (13 hunks, 5 files): - ggml-cuda/mmq.cu: the fork's column-chunked MMQ (dense and MUL_MAT_ID) is kept; upstream's new mmq_args.ncols_opt is passed per chunk (dense: the chunk's column count, as upstream passes ne1; MUL_MAT_ID: upstream's RDNA3/4 expert-fill estimate applied to the chunk's token count). - ggml-cuda/ggml-cuda.cu: BF16 GEMM fallback takes upstream's per-vendor rule (ggml-org#28846, AMD switches to F32 above 32 columns) in place of the fork's unconditional one; graph_optimize takes upstream's restructured function (add_alloc_deps, top-k MoE fusion deps) with the fork's two changes applied to the MoE weighted-reduction match (ggml_cuda_moe_weighted_reduction_enabled() gate, for_alloc_deps=true). - src/llama-context.cpp: upstream's opt_ctx guard and graph-reuse guard (gf_res_prev_active) combined with the fork's layer-tap event cleanup and sequence-layout tracking. - src/llama-model.cpp: upstream's HRM_TEXT mirror guard placed ahead of the fork's tied-output and MTP split rules (the fork's own DSV4 routing is kept). - src/models/qwen4exp.cpp: upstream's [n_embd, hc] norm gammas (TENSOR_ALLOW_RESHAPE, required by its grouped_norm) with the fork's trunk/MTP load flags OR'd in; upstream's n_ff_exp() accessor. Merged without a textual conflict but broken, fixed here: - ggml-cuda/common.cuh: both sides define fast_bf16_hardware_available; upstream's is kept (identical on AMD, and the one its BF16 rule is written against), the fork's gfx906 measurement note moved onto it. - common/speculative.cpp: upstream's edits to draft_mtp::process() were applied onto the fork's relocated process_decode() body, replacing its signature; process_decode() restored exactly (identical to the fork), the entry-point process() already carried upstream's embedding-batch guard. Upstream's DFlash M-RoPE image skip (ggml-org#28587) ported from i_batch_beg/end to the fork's per-sequence i_batch_rows. - src/models/deepseek41.cpp: n_ff_exp read through v0.5.0's accessor.
GGML_HIP_RCCL defaulted OFF, so a plain HIP build silently lacked RCCL and tensor split fell back to the meta-backend butterfly: -18.6% prefill at 4 x 64K on four MI50s against the same binary with RCCL. The fork's own FEATURES.md already builds with -DGGML_HIP_RCCL=ON; this makes it the default. Opt out with -DGGML_HIP_RCCL=OFF.
nbritton
pushed a commit
to exabit-io/llama.cpp-gfx906-tuning
that referenced
this pull request
Sep 24, 2026
…) and the ADD_ADD flake re-runs Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
documentation
nbritton
pushed a commit
to exabit-io/llama.cpp-gfx906-tuning
that referenced
this pull request
Sep 24, 2026
…6-09-24) mxxm-t's fork merged with llama.cpp v0.5.0, offered upstream as mxxm-t/mx-llama.cpp#17; used for everything until mxxm-t merges it. REQUIREMENTS R3.7 + change log; plan restructured with the substrate first and its history last; CLAUDE.md, README, ROCM-SETUP, patches/README, forks list updated; 2026-09-22 docs marked history. Bin branches on exabit-io/llama.cpp moved onto it (tags gfx906/mx-merge-v0.5.0/*). binrun now refuses to measure unless the gate ran on its own base commit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This brings
masterfrom upstream b10760 to the llama.cpp v0.5.0 release (7fe450e, b11146, 386 upstream commits) with a merge, so your history is untouched. It also adds one small commit that turns onGGML_HIP_RCCLby default. Every fork feature is kept. Where both sides changed the same code, the two changes are combined rather than one being chosen.Conflicts and how they were resolved (13 hunks, 5 files)
ggml-cuda/mmq.cu: your column-chunked MMQ (dense andMUL_MAT_ID) is kept. Upstream addedmmq_args.ncols_opt, which is now passed per chunk:ne1MUL_MAT_ID: upstream's RDNA3/4 expert-fill estimate, applied to the chunk's token countggml-cuda/ggml-cuda.cu:graph_optimizeis upstream's restructured function (add_alloc_depsplus the top-k MoE fusion deps), with your two changes applied to the MoE weighted-reduction match (ggml_cuda_moe_weighted_reduction_enabled()gate,for_alloc_deps = true).src/llama-context.cpp: upstream'sopt_ctxguard and graph-reuse guard (gf_res_prev_active) are combined with your layer-tap event cleanup and sequence-layout tracking.src/llama-model.cpp: upstream'sHRM_TEXTmirror guard comes first, then your tied-output and MTP split rules. Your DSV4 routing is kept.src/models/qwen4exp.cpp:[n_embd, hc]norm gammas (TENSOR_ALLOW_RESHAPE, which its newgrouped_normrequires), with your trunk/MTP load flags OR'd inn_ff_exp()accessorMerged without a textual conflict but did not build, fixed in the merge commit
ggml-cuda/common.cuh: both sides definefast_bf16_hardware_available. Upstream's copy is kept, because it is identical on AMD and it is the one upstream's BF16 rule is written against. Your gfx906 measurement note moved onto it.common/speculative.cpp: upstream's edits todraft_mtp::process()landed on your relocatedprocess_decode()body and replaced its signature.process_decode()is restored and is now byte-identical tomaster. Theprocess()entry point already carried upstream's embedding-batch guard. Upstream's DFlash M-RoPE image skip (speculative: fix failed to decode mtmd chunk with DFlash ggml-org/llama.cpp#28587) is ported fromi_batch_beg/i_batch_endto your per-sequencei_batch_rows.src/models/deepseek41.cpp: readsn_ff_expthrough v0.5.0's accessor.ggml-hip: build with RCCL by defaultGGML_HIP_RCCLdefaulted toOFF. A plain HIP build therefore silently lacked RCCL, and tensor split fell back to the meta-backend butterfly. With RCCL, prefill was +18.6% at 4 x 64K on four MI50s, measured n=4 against the same binary.FEATURES.mdalready builds with-DGGML_HIP_RCCL=ON; this makes it the default.-DGGML_HIP_RCCL=OFFstill opts out.Testing
On 4 x MI50 (gfx906, 32 GiB, XGMI ring), ROCm 10.0, 125 W per GPU, built with
-DGGML_HIP=ON -DAMDGPU_TARGETS=gfx906 -DGGML_HIP_RCCL=ON -DGGML_CUDA_FA_QUANTS=all, and run with the fork's defaults:test-backend-ops, one process per GPU: 16273/16273 on two GPUs and 16272/16273 on the other two.ADD_ADDf16 test (sycl: fuse rms_norm+mul+add and add+add residual chains ggml-org/llama.cpp#27610), at NMSE 1.02e-7 against a 1.0e-7 limit.-sm tensor: 5.6153 +/- 0.0624, against 5.6171 +/- 0.0624 on the pre-v0.5.0 builds.-sm layer, Qwen3.8-27B greedy generation: coherent.llama-server --spec-type draft-mtp --spec-draft-n-max 2,-sm tensor: 256 tokens generated, 156 of 197 drafts accepted (79%).-sm tensor --n-cpu-moe 41 -lm mlock,LLAMA_PLE_SHARD=1: loads, and greedy generation is coherent.Not tested (no hardware or models here): CUDA/NVIDIA builds, AMD architectures other than gfx906, DeepSeek-V4/V4.1, DSpark, DFlash, gemma, MiniMax, and
-tpspipeline stages.For review
ggml_cuda_mul_mat_cublas.🤖 Generated with Claude Code