HRX: decode-split multipass path intermittently faults (HSA_STATUS_ERROR_MEMORY_FAULT) above capacity 2048

[1bit-traffic-bot]

The multi-pass KV-block reduction added by 1bit-MONSTER/llama.cpp#13 (branch feat/hrx-decode-split-multipass, commit f59f6d37a on fa1a4563) makes flash_attention.decode_split work above key_value_token_capacity 2048, but a residual out-of-bounds access intermittently faults the decode instead of declining. That violates the HRX lane's constraint that a dispatch must decline and fall back rather than fail the decode.

Symptom

gfx1151 (Strix Halo), Qwen3-Coder-30B-A3B-Instruct-Q4_K_M, -dev HRX0, llama-bench -p 0 -n 8 -d <D>:

Warning: Queue error - HSA_STATUS_ERROR_MEMORY_FAULT
E graph_compute: ... AMDGPU memory access fault at device address 0x00007f508ea94000 (reason mask 0x00000001)
E llama_decode: failed to decode, ret = -3

5-run GPU-fault counts with the partial transients aligned to 4096:

depth	capacity	blocks	faults
1900	1920	30	0/5
2000	2048	32	0/5
2100	2112	33	0/5
2500	2560	40	0/5
3000	3008	47	1/5
4800	4864	76	0/5

With the SAME code but the partial transients aligned to 256, d2100 is 5/5. GGML_HRX_DISABLE_DISPATCH=flash_attention_decode_split makes every faulting depth pass (0/5), so the fault is in the multipass path, not the fallback.

Excluded (each measured over 5 runs)

• The reducer - no-op'ing reduce_completed.multipass (inserting template.return as its first statement) still faults 5/5, so it is not the reduction math.
• The completion counter - replacing its atomic.reduce<addi> -N reset with a plain release-ordered store of 0 changes nothing; so does zero-initialising it via ConstantInitialization.
• The exact-33 assume bound - relaxing every range(..., 33, 4096) to range(..., 1, 4096) changes nothing.
• Static sizing - the dispatch's transient bytes are exactly 256 * blocks (partial_max/sum) and 16384 * blocks (partial_output), strictly linear, and the kernel's views match.
• Every readable access is in bounds - the produce's channel stores (lane_output_base = lane * 4, lane < 32, so the channel is in {0,4,...,124} and a vector<4xf16> store keeps channel + 4 <= value_head_size = 128); the scalar max/sum row stores (guarded by query_head_valid); the reduce's output store (query_head = key_value_head * query_heads_per_key_value_head + query_row < 32, output_channel < value_head_size); the K/V loads (element-guarded by bounded_key_value_token_count); pack_completed_q8's output size (q8_1_x4_byte_count(1, output_hidden_size)).
• Allocation - alignment 256 -> 4096 helps (d2100 5/5 -> 0/5); padding the transient SIZES makes it worse (d3000 5/5, d2100 4/5); 64K alignment is worse still; padding the block count to whole 64-block waves does not help.

Assessment

Every access is provably in bounds, yet the fault moves with the allocation layout and cannot be removed by adding slack. The best remaining explanation is a wrong address emitted by the JIT-specialised code for this capacity, which allocation cannot fix. Prime suspect: the multipass output pass (flash_attention_decode_split_f32_f16_wmma.loom ~911-920), the only O(blocks)-per-output-element code in the reduce.

Impact

1bit-MONSTER/llama.cpp#13 masks part of this by aligning the partial transients to 4096, but the fault is not eliminated and a decode can still fail - so this is not yet safe to land as-is.

[bong-water-water-bong]

Possibly related: #140. The ≤2048 decode-split kernel (flash_attention_decode_split_next_q8) is nondeterministic run to run even at short context. Qwen3-0.6B greedy decode gave 2 distinct outputs in 5 identical runs; with GGML_HRX_DISABLE_DISPATCH=decode_split it gave 1. Here the fault survives a no-op reducer, so it isn't in the reduction math. If the two share a cause in the decode-split family, fixing the ≤2048 case first could make this one easier to pin down.

[bong-water-water-bong]

Re-tested this while merging the pin bump (#151). llama.cpp#13's own report was 0/5 faults at depth 2100, 1/5 at depth 3000. Re-running llama-bench -dev HRX0 -p 0 -n 8 -d 2100 here faulted 4/4 times (once inside a combined -d 2100,3000 run, then 3 isolated repeats at 2100 alone) - HSA_STATUS_ERROR_MEMORY_FAULT, res=-3. Ruled out a broader GPU/driver issue: Vulkan0 ran clean immediately before and after, and HRX0 at depth 500 (below the multipass threshold) also ran clean, so this is isolated to the multipass path above 2048, same as originally diagnosed - just reproducing far more readily in this environment than in the original report. Worth knowing before prioritizing the fix: this may not be the rare edge case the 1/5-at-d3000 number suggested.

[bong-water-water-bong]

Possibly related: #140 found the exact same last-workgroup-atomic-counter-then-barrier pattern (shared by every decode-split kernel in this family, including this one) produces measurably wrong results ~19% of the time even on the plain next_q8 short-context path, with no bounds violation possible there (small, fixed block count). That points at the kernel.barrier<global> scope(workgroup) ordering(release/acquire) pair not actually guaranteeing cross-workgroup write visibility on this hardware, rather than anything specific to either kernel's indexing math - which would fit this issue's own conclusion (every access provably in bounds, fault moves with allocation layout) better than a source-level bug does. Worth checking whether strengthening that barrier's scope changes either issue's reproduction rate.

[bong-water-water-bong]

Update: built Loom's own compiler toolchain and directly inspected the barrier lowering (2026-09-26) - this narrows things down, but does not confirm the earlier barrier-visibility hypothesis; if anything it weakens it.

Built ROCm/hrx-system from source (Python 3.12 + Bazel, dev.py bazel build //loom/src/loom/tools/loom-check ...) and used loom-check's emit source-low mode to compile a minimal standalone kernel reproducing the exact same synchronization idiom used by this kernel family (kernel.barrier<global> scope(workgroup) ordering(release) on the producer side, ordering(acquire) on the consumer side, around a view.atomic.rmw<addi> {ordering = acq_rel, scope = device} completion counter), targeting amdgpu:gfx11-generic.

Actual emitted GCN/RDNA assembly:

• Acquire barrier + subsequent global load: lowers to s_barrier (intra-workgroup only) + s_waitcnt {vmcnt=0} + a global_load_b32_saddr ... {glc=1} for the load. No explicit buffer_gl0_inv/buffer_gl1_inv cache-invalidate instructions - the compiler instead uses the GLC ("globally coherent") bit on the load itself to bypass the L0 vector cache. On RDNA this is a standard, documented way to get acquire semantics for a single load and is generally considered correct, not obviously buggy.
• Release barrier + the atomic RMW itself: lowers to the data store, s_waitcnt {vmcnt=0} + s_waitcnt_vscnt {vscnt=0} (drains outstanding stores), s_barrier, then the buffer_atomic_add_u32_rtn (the counter increment), followed by s_waitcnt {vmcnt=0} and then explicit buffer_gl1_inv + buffer_gl0_inv - because that specific atomic op was declared acq_rel, not just release, so the compiler also emits real invalidate instructions there for the atomic's own acquire half.

So the compiler clearly is capable of emitting real buffer_gl0_inv/buffer_gl1_inv invalidate instructions when it decides they're needed - it just didn't for the plain acquire-barrier-then-load case, using the cheaper GLC-bypass-load idiom instead. That pairing (writeback+wait on release, GLC-bypass load on acquire) is a standard, plausible-correct AMDGPU idiom, not an obvious hole. I can't fully rule out a subtler issue (e.g. whether GLC bypass on this specific gfx1151 iGPU's cache hierarchy - L0 per-CU, L1 per shader-array, L2 device-wide - really gives full device-scope visibility, vs. only bypassing L0) without lower-level hardware documentation this box doesn't have. But the "the barrier just doesn't provide real cross-workgroup visibility" theory, as stated, is not supported by what's actually emitted.

Net effect: this rules out the simplest version of the shared hypothesis between this issue and [engine#140]. Root cause for both remains open. Repro scripts kept in case anyone wants to extend them (e.g. compiling the real multipass kernel through the full template/config pipeline and running the actual dynamic --sanitizer=race instrumentation on real hardware, which is a substantially bigger lift than this static check).

[bong-water-water-bong]

Ran the actual empirical test on real hardware (2026-09-26) - built a fresh, correctly-pinned clone of 1bit-MONSTER/llama.cpp @ 8dd75eb48 (the exact commit that shipped the multipass fix), patched the flash-attention decode-split kernel's three reduce_fused variants (direct/cooperative/multipass) to force a genuine cache invalidate before the real partial-buffer reads - a dummy view.atomic.rmw<addi> with {ordering = acq_rel, scope = device} on the completion counter, gated to a single lane, followed by a kernel.barrier<workgroup> to propagate it - based on the confirmed compiler behavior that an acq_rel atomic emits real buffer_gl1_inv/buffer_gl0_inv instructions where a plain ordering(acquire) barrier-then-load does not (see the earlier comment on this issue).

Important methodology note first: neither #123's fault nor #140's race reproduced at all on a clean, freshly-built unpatched binary under normal (low-contention) conditions - 0/54 combined attempts for #123, 0/300 for #140. Both only reproduced once I deliberately induced GPU contention with a second, concurrent llama-bench workload hammering the same HRX0 device in a loop. This matches this session's own earlier note that the divergence rate is contention-dependent, not a fixed per-request probability - worth remembering before trusting any "looks fixed" result from a quiet box.

Under matched induced contention:

• Unpatched baseline: 56/300 (18.7%) divergent logprobs - closely matching this issue's documented 58/300 (19.3%).
• Patched (forced invalidate before the reads): 208/300 (69.3%), then 170/300 (56.7%) on a repeat run - both roughly 3-3.7x worse, not better.

Conclusion: this rules out the missing-cache-invalidate hypothesis more decisively than the earlier static check did. Forcing a real, compiler-confirmed cache invalidate immediately before the reads made the race substantially more frequent, not less. The most likely explanation is that the extra synchronization changed wavefront scheduling/occupancy enough to widen whatever the actual race window is, rather than closing a coherence gap - meaning the real bug is probably a genuine scheduling/ordering hazard elsewhere in the reduction (or in how the producer side publishes its partials), not GPU cache visibility. I'm not going to guess further at the exact mechanism without stronger tooling (the dynamic --sanitizer=race instrumentation, still not attempted - would need the kernel's full template/config pipeline resolved against real device facts).

Patch discarded (this was a diagnostic experiment only, not proposed for merging). Root cause remains open for both this issue and #123.

[bong-water-water-bong]

Attempted the real dynamic race sanitizer against the actual kernel (2026-09-26) - blocked by two genuine upstream gaps, not by anything fixable from this repo.

Found that AMD's ggml-hrx layer already has plumbing for Loom's compile-time sanitizer passes via GGML_HRX_LOOM_SANITIZER (env var, loom-jit.cpp) - but it only wires up none/access/asan/value/operation/ubsan/all, never race, even though the underlying loomc C API defines LOOMC_SANITIZER_CHECK_RACE and even a ready-made LOOMC_SANITIZER_CHECKS_TSAN_LIKE preset for exactly this. Patched ggml_hrx_loom_jit_parse_sanitizer_checks to add a "race"/"tsan" case (4 lines, following the existing pattern exactly), rebuilt, and set GGML_HRX_LOOM_SANITIZER=race (plus HRX_HAL_SANITIZER=tsan for the HAL-side runtime feature) against a real llama-server serving Qwen3-0.6B on HRX0.

Two real compile-time failures surfaced immediately:

1. Missing runtime-global config: generated kernels reference a data symbol iree_tsan_config that never gets emitted - ggml-hrx's ggml_hrx_loom_jit_runtime_globals() only knows how to set LOOMC_AMDGPU_RUNTIME_GLOBAL_ASAN_CONFIG for the ACCESS check; the public C header (loomc/target/amdgpu/emit.h) doesn't define an equivalent TSAN_CONFIG global at all. This is a real gap in the public runtime-globals surface, not something fixable with a local patch.
2. The harder blocker: sanitizer-insert-race-observations cannot observe vector.fragment.load: local-memory vector operation requires lane-activity-aware race observation. Loom's race sanitizer pass cannot currently instrument WMMA fragment loads at all - and our flash-attention decode-split kernel uses exactly those (vector.fragment.load<lhs>/<rhs> for the P·V product). This affected several other WMMA-based kernels too (mul_mat_swiglu, mul_mat_q6_k_packed, mul_mat_add_f32_f32_wmma), breaking the whole forward pass when the sanitizer is enabled globally. Even with (1) fixed, the actual target kernel would still fail to compile with race instrumentation - this is an unimplemented feature in Loom's sanitizer pass itself.

Conclusion: the dynamic race sanitizer genuinely cannot be run against this kernel family with the currently pinned hrx-system version. Confirming or ruling out the actual root cause of #123/#140 further needs either an upstream fix to Loom's sanitizer pass (fragment-load support) plus the missing runtime-global plumbing, or a fundamentally different diagnostic approach (e.g. RGP/ISA-level tooling this environment doesn't have). This session tried the static compiler-output check, an empirical barrier-strengthening patch (made the race 3-3.7x worse), and now the dynamic sanitizer (blocked) - all three avenues are now closed off or exhausted with the tooling actually available. The real bug is most likely a scheduling/ordering hazard, not a cache-coherence gap, given the barrier-strengthening result, but pinning it down exactly needs the upstream sanitizer support that doesn't exist yet.

[bong-water-water-bong]

Correction to earlier comments on this issue: got the REAL production kernel to compile and disassembled it directly - the barrier lowering is NOT missing an invalidate after all.

The earlier static check (comment above) tested a minimal synthetic standalone kernel reproducing the barrier idiom in isolation, and found the acquire barrier lowered to a GLC-bypass load with no explicit invalidate instruction. That result doesn't hold for the real kernel in its full context.

Using ROCm/hrx-system's loom-compile directly with the actual production config values for this repro (Qwen3-0.6B: query_head_count=16, key_value_head_count=8, qk_head_size=128, value_head_size=128, key_value_token_capacity=2112 for a 2100-token decode - the exact #123 repro depth, landing in the multipass range), I compiled the real flash_attention_decode_split_f32_f16_wmma.loom kernel (via its actual kernel.def entry point, @ggml_flash_attention_decode_split_f32_f16_wmma_next_q8) to a real HSACO artifact, and disassembled it with llvm-objdump (from TheRock's ROCm SDK - roc-obj-extract/llvm-objdump are already present, no extra tooling needed).

The real, in-context disassembly around the completion counter and the explicit acquire barrier:

buffer_atomic_add_u32 v2, v1, s[20:23], s2 offen glc   ; completion counter increment (acq_rel)
s_waitcnt vmcnt(0)
buffer_gl1_inv
buffer_gl0_inv                                          ; <- real invalidate, atomic's own acquire half
ds_store_b32 v1, v2 offset:14592                        ; old_counter -> LDS scratch
s_barrier                                                ; kernel.barrier<workgroup> (propagate via LDS)
ds_load_b32 v1, v1 offset:14592                          ; old_counter <- LDS scratch
v_cmp_eq_i32_e64 s6, v1, 32                              ; is_last_partition? (32 = 33 blocks - 1, matches our capacity)
s_cbranch_scc0 ...                                       ; skip if not last
s_barrier                                                ; <- this is the explicit kernel.barrier<global> ordering(acquire)
s_waitcnt vmcnt(0)
buffer_gl1_inv
buffer_gl0_inv                                           ; <- a SECOND real invalidate, right here
; ... reduction over partial_max/partial_sum/partial_output begins immediately after ...

Both invalidate pairs are present and correctly placed - one for the counter atomic's own acq_rel, and a distinct second one for the explicit kernel.barrier<global> ordering(acquire) that gates entry into the reduction. The real, deployed kernel already has correct, full L0/L1 cache invalidation before reading the other workgroups' partial buffers. This directly explains the earlier empirical result (posted above): forcing an additional redundant invalidate before the reduction made the race worse rather than better, because there was nothing missing to fix - the extra synchronization was pure overhead that perturbed scheduling without closing any real gap.

This closes off the cache-coherence hypothesis for good, with ground-truth evidence rather than inference. The actual root cause of both #140's race and #123's fault is very unlikely to be a GPU memory-visibility problem. It's more likely a genuine data/algorithmic hazard - e.g. in how partial values get written by the producer side, an indexing issue that's timing-dependent, or something in the online-softmax combination itself. Confirming that precisely still needs either the (currently blocked - see above) dynamic race sanitizer, or manual disassembly analysis of the producer-side write path, which hasn't been attempted.