Feat/hrx port ae91949 - #115
Open
bong-water-water-bong wants to merge 10 commits into
Open
bong-water-water-bong wants to merge 10 commits into
bong-water-water-bong wants to merge 10 commits into
Conversation
Proposal for a new AMD/ROCm backend. See RFC discusion: `Insert discussion link`
… graphs)
The fused Qwen3-MoE dispatch covers the embedding inside fused kernels, but
dense (unfused) graphs fail-closed: 'unsupported HRX node 0: GET_ROWS' on the
very first decode step, because eager_capability_declared claims GET_ROWS yet
no standalone dispatch implements it.
Add a SingleOp GET_ROWS dispatch + Loom kernel:
- kernel-corpus/kernels/hrx_owned/get_rows_f32.loom: generic
output[row, col] = source[ids[row], col] with f32 and q8_0 sources
(q8_0 dequantized inline: f16 block scale x int8 quant), wave64 gfx11.
- dispatch-get-rows.{h,cpp}: matcher for GGML_OP_GET_ROWS (f32 or q8_0
source, i32 ids, contiguous), binds source/ids/output and sets the
source_format workload param. Registered in dispatch-registry.cpp and
added to ggml-hrx CMake sources + kernel-corpus manifest.
Verified: kernel parses and compiles to gfx1151 HSACO via the Loom JIT
(ggml-hrx-compile-kernel), and the dispatch engages at runtime — the
planner now covers the embedding and the fail-closed error moves past
GET_ROWS to the next unsupported op (q8_0 MUL_MAT, a separate generic-
backend item).
- loom-jit.cpp: root link uses LOOMC_LINK_MODE_LINK with config bindings materialized at link time (typed config.def), replacing the removed compile-time binding array; artifact bytes via contiguous span API; launch-config program load without nullptr args - CMakeLists.txt: IREE_HAL_DRIVER_TASK replaces LOCAL_SYNC/LOCAL_TASK - generate_kernel_corpus.py: --mode=merge replaces removed --mode=archive - kernels: func.def inline/template.decl/def/apply migration; template.apply in launch-config regions marked pure; launch template impl renamed to avoid decl==def name collision (merge dropped the def otherwise) - manifest: per-file shas + digests updated for migrated kernels Verified: fused Qwen3-30B-A3B-Instruct-2507 Q4_K_M generates on HRX0 (gfx1151) at ~81 tok/s with deterministic output and zero graph errors.
…d models to CPU - device_supports_op: detect (via the NONE weight probe) models that use quant types with no Loom kernel and delegate the whole graph to the CPU backend instead of letting the fused-pattern dispatcher fail-close. - device_buffer_type: use the host-visible buffer so delegation needs no copies. - kernel-executable-cache: accept i32 launch/workload scalars (packer previously rejected them, so the standalone GET_ROWS path could never run). - dispatch-get-rows: source_row_count is ne[1] (row count), not ne[0] (width). - manifest: source_format is a launch scalar, not a workload specialization.
…ls to CPU Kernels (Loom, in hrx_owned/): - dequant_iq3xxs_f32.loom : IQ3_XXS codebook dequantize (grid/sign tables) - mul_mat_vec_iq3xxs_f32.loom : IQ3_XXS matrix-vector product (per-row decode) Both pass the Loom verifier and are registered in the corpus manifest. Dispatch: new llm.matmul.iq3xxs_matvec_f32 route (dispatch-llm-matmul.cpp) that matches 2D f32 MUL_MAT with an IQ3_XXS weight and binds the codebook as a constant transient buffer. Runtime: because the HRX dispatcher is fused-pattern based and ggml_backend_sched splits per-op, device_supports_op now detects models containing weight types the corpus has no kernel for and delegates the whole graph to the CPU backend, so sub-4-bit UD files run correctly on HRX0. Also fixes real bugs found while bringing the IQ3_XXS path up: - kernel-executable-cache: accept i32 launch/workload scalars - dispatch-get-rows: source_row_count is ne[1], not ne[0] (width) - manifest: source_format is a launch scalar, not a workload specialization
…in-model The HRX dispatcher is fused-pattern based while ggml_backend_sched assigns nodes per-op; three coupled defects made IQ3_XXS standalone dispatches silently produce zeros or the wrong layer's weights: - ggml-hrx: device_supports_op claimed every op in eager_capability_declared() whenever the model contained no "unsupported" weight type, and otherwise delegated the whole model to the CPU through a sticky per-device flag. Replace both with an honest per-node check (can_execute_standalone_op_as_graph), so a node is claimed only when the dispatcher can actually execute it. This removes the whole-model CPU fallback and lets IQ3_XXS MUL_MAT nodes execute on HRX0. - graph: import_ggml_graph() classified a value as External only when it was never a node source, so activations produced by another backend were imported as Transients: they were never staged/uploaded, kernels read uninitialised buffer memory and wrote their result into a transient arena slot while the consumer read the host tensor. Classify by "is it produced by a node of this graph" instead, keep in-graph graph outputs external, and keep the classification consistent for a whole storage so a view (RESHAPE/VIEW) terminal value and its base share one binding. - command-program-executor: a prepared command program is reused by every node of the same shape (e.g. all layers of a model). Resident host-weight buffers were captured once at preparation time and never re-bound, so every layer after the first executed with layer 0's weights. Stage weights through the per-execution host staging path, which is re-bound to the live tensors on every execution. - dispatch-scheduler: treat GGML_OP_NONE leaf/parameter nodes as covered instead of failing the graph with "unsupported HRX node". Also fix a latent duplicate Loom symbol in the IQ3_XXS kernels: the check.benchmark result symbol shadowed the exported kernel symbol, so loom-link without --strip-check failed. Verified with Qwen3-0.6B-UD-IQ2_M (IQ3_XXS weights): test-backend-ops -b HRX0 -o MUL_MAT 30/30 (11/11 IQ3_XXS) llama-perplexity wikitext-2 20x512 HRX0 58.85 CPU 59.11 llama-perplexity ppl.txt 8x512 HRX0 12.94 CPU 13.02
Every op/shape now claimed by device_supports_op is actually executed correctly; test-backend-ops -b HRX0 passes 41/41 with 0 failures. - dispatch-add: the HRX/loom runtime submits through a fixed-size kernarg ring and fails with OUT_OF_RANGE when the grid does not fit (36 kernarg blocks per workgroup against a 262144-entry ring), and that failed submission leaves the device stream unusable for every later submission. Cap the claimed add grid at 4096 workgroups (~1M elements), well below the measured ~1.9M-element failure point, so f32 ADD over 16.7M elements is left to the CPU instead of breaking the stream for the rest of the process. -o ADD is now 11/11. - dispatch-get-rows: stop registering the standalone GET_ROWS kernel. It fails every case the claim accepts today (the Loom module fails to compile for some widths, large row counts exceed the submission ring, and the remaining shapes return wrong values), so claiming it made the backend return wrong embeddings silently. GET_ROWS nodes now fall to the CPU backend, which needs no copies because the HRX buffer type is host-visible. - get_rows_f32.loom: the launch-config signature declared a source_format scalar it never used. It is only needed by the kernel body, and the JIT feeds the launch config exactly the index workload parameters, so the extra parameter made the launch config (and the earlier "expects 4 arguments but received 3" failure) unsatisfiable.
The ggml-hrx backend files added by this port carried no license metadata. Add the HRX copyright/SPDX header to the sources and Loom kernels in the change set, and record the corpus license as a manifest key (JSON has no comment syntax). Refresh the manifest source digests for the two edited Loom kernels. Re-verified after the change: test-backend-ops -b HRX0 41/41, 0 failures; Qwen3-0.6B-UD-IQ2_M llama-perplexity -f ppl.txt -c 128 --chunks 2 HRX0 5.3308 vs CPU 5.2928.
…y run register_get_rows_dispatch() stopped registering common.get_rows_f32 because the claim accepted shapes the dispatcher could not execute. The kernel itself is sound: a direct HRX0-vs-CPU probe (f32 and q8_0 sources, random id lists) is bit-exact for every shape inside its domain. The claim was simply far wider than that domain: - get_rows_f32.loom's `index.assume` bounds are compile-time promises, not runtime guards: specialising width outside [32, 16384], source rows outside [1, 131072] or output rows outside [1, 2048] is a hard Loom compile error (confirmed: width 1/31/20000/76800, src 131073 and out 2049 all fail to compile). - The kernel indexes a flat 2D view, so batched (ne[2]/ne[3] > 1) and higher-rank gathers - which the old claim accepted - were silently wrong. - The HRX runtime charges each dispatch's kernarg footprint against a fixed 262144-entry ring (aql_block_processor: ceil_div(kernarg_length, block_size)), and that footprint scales with the *source row count*: measured on gfx1151 it is about 16 * ceil(width/256) * (source_row_count + output_row_count) blocks. Crossing the ring aborts the submission and leaves the device stream unusable, which is what turned one bad case into a cascade of failures. The claim now mirrors the kernel exactly (flat 2D only, f32/q8_0 sources with q8_0 width block-aligned, width in [32, 16384], rows within the assume bounds, and the estimated kernarg footprint under 0.75 x the ring). Shapes outside it are reported "not supported [HRX0]" and stay on the CPU, which needs no copies because the HRX buffer type is host-visible. Measured footprint boundaries: width=1024 -> source rows <= 4090, width=256 -> source rows <= 16383 (i.e. ceil(width/256) * source_rows <= 16384). Verified: test-backend-ops -b HRX0 -o GET_ROWS 2/2, 0 failures; full test-backend-ops -b HRX0 43/43, 0 failures; Qwen3-0.6B-UD-IQ2_M reference perplexity unchanged at HRX0 5.3308 vs CPU 5.2928.
Follow-up to the per-op claim change. Three defects made the honest claim path
either reject valid graphs or hang on models it used to run:
- command-program-bindings: a zero-byte external binding was reported as an error
("external value N has an empty binding"). ggml legitimately produces empty tensors
- a GET_ROWS over an empty id list during decode yields ne=[0,...] and its outputs
are [width,0] - so the whole graph was rejected and llama_decode failed with -3 on
UD-IQ2_M at c=512. A zero-length binding is a no-op, not a failure.
- dispatch-llm-matmul: the dense q4k/q6k route accepted output_size up to 262144 and
any prefill query length, so the vocab-sized LM head (151936) was claimed and
dispatched through the graph replay, whose kernarg footprint scales with
output_size x token_count and overflowed the fixed 262144-entry ring. The failed
submission left the device stream unusable, which showed up as a hang. Measured on
gfx1151 with input_size 1024: 16384x64 needs 284957 entries, 8192x512 needs 402833,
while everything up to 4096x512 fits. Cap the claim at output_size <= 4096 and
token_count <= 256; larger shapes stay on the CPU instead of breaking the stream.
- dispatch-get-rows: reject degenerate gathers with zero output or source rows.
Verified on the deployed /opt/hrx build:
test-backend-ops -b HRX0 43/43, 0 failures (IQ3_XXS MUL_MAT 11/11)
UD-IQ2_M (IQ3_XXS) c=128 HRX0 5.3308 vs CPU 5.2928; ppl.txt 20x512 12.9439;
wikitext-2 20x512 58.8460 vs CPU 59.1075
Q4_K_M (supported) c=128 HRX0 3.4908 vs CPU 3.5289; c=512 7.1220;
wikitext-2 20x512 24.6865 vs CPU 24.6607
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Additional information
Requirements