HRX: decode_split_next_q8 gives wrong, nondeterministic attention at short context (up to 3.66 nats between identical requests)

[bong-water-water-bong]

Greedy decode on HRX0 gives different output across identical runs (same seed, temperature 0, fresh process). The same runs on Vulkan0 are identical every time.

Repro (strixhalo, llama.cpp pin 83e1c41, HRX build): run the command below 5 times and compare the md5 of each output.

llama-completion -m Qwen3-0.6B-Q4_K_M.gguf --device HRX0 -ngl 99 -c 2048 -n 24 --temp 0 --seed 1 -no-cnv -p "The capital of France is"

run set (5 runs each, Qwen3-0.6B Q4_K_M)	distinct outputs
default	2 (4 × 460e86, 1 × d2a6c4)
GGML_HRX_DISABLE_DISPATCH=decode_split	1
GGML_HRX_DISABLE_DISPATCH=fused	1

ZAYA1-8B Q4_K_M behaves the same way: 2 of 3 runs identical. Prefill is unaffected, and perplexity at batch 512 is stable.

Where. With the decode-split dispatch disabled the output is deterministic, so it comes from flash_attention_decode_split_next_q8 (context under 2048, so the multi-pass path from #123 isn't involved). Two readings, not yet told apart:

• Benign. The split-K partials are combined in completion order, so the rounding differs run to run and greedy flips only at near-ties.
• A race. Something like HRX: decode-split multipass path intermittently faults (HSA_STATUS_ERROR_MEMORY_FAULT) above capacity 2048 #123, giving wrong values.

What would tell them apart: the logits of two differing runs at the first divergent step. A difference around 1e-3 means rounding; larger means a real race. A fixed-order reduction would make it deterministic either way.

Found while A/B-testing a ZAYA graph change on HRX: the change looked like it altered HRX decode output until repeated identical runs showed the baseline varies too.

[bong-water-water-bong]

Measured more closely. It is a real bug, not rounding.

Setup: one llama-server on HRX0 (Qwen3-0.6B Q4_K_M, pin 83e1c41). The same /completion request is sent 12 times: 24 greedy tokens, n_probs: 5, cache_prompt: false, fixed seed. Each request is compared with the last one: the largest |Δ logprob| over the top-5 at each step, or div when the greedy tokens differ. Two separate servers per setting.

setting	requests 1…12 compared with request 12
GGML_HRX_DISABLE_DISPATCH=decode_split, server 1	div div 0.142 0.0943 0.0581 0 0 0 0 0 0 0
GGML_HRX_DISABLE_DISPATCH=decode_split, server 2	div div 0.142 0.0943 0.0581 0 0 0 0 0 0 0
default, server 1	div div div 0.489 div 1.47 div div 3.66 div div 0
default, server 2	div div div div div div div div div div div 0

Two separate effects:

1. Warm-up. Without decode-split, the first ~5 requests of a process differ, then every request is bit-identical, in the same pattern on both servers. That fits kernels being specialised in the background while something else serves the first requests. It is deterministic once settled, and a first request may just be served by different kernels.
2. flash_attention_decode_split_next_q8 never settles. Its log-probabilities differ by up to 3.66 nats between identical requests. That is far outside rounding: kernel-to-kernel differences on HRX are ≤ 0.2 nats. So this kernel sometimes computes wrong attention at short context (≤ 2048 KV), on the default --device hrx decode path. It looks like a race of the same family as HRX: decode-split multipass path intermittently faults (HSA_STATUS_ERROR_MEMORY_FAULT) above capacity 2048 #123, but it is in the path that is already shipped.

Workaround until it's fixed: GGML_HRX_DISABLE_DISPATCH=decode_split. Decode then falls back to flash_attention_f32_f16_wmma, which is slower (#115) but deterministic after warm-up.

[bong-water-water-bong]

Mitigated in #148: 1bit serve --device hrx now starts llama-server with GGML_HRX_DISABLE_DISPATCH=decode_split by default. With the kernel on, Qwen3-Coder-30B-A3B faulted the GPU in 2 of 3 decode runs; with it off, 5 of 5 passed. It was also faster on Qwen3-0.6B (322 vs 169 tok/s) and ZAYA1-8B (25.5 vs 23.5 tok/s). To test a kernel fix through the engine, set ONEBIT_HRX_DECODE_SPLIT=1. Leaving this open for the kernel itself. The multi-pass work (#121, 1bit-MONSTER/llama.cpp#13) extends the same kernel family, so it should be checked against this before it's re-enabled.

[bong-water-water-bong]

Ran the diagnostic this issue proposed: captured the first-token logprob for Paris (top-1 in every run) across 300 identical requests (cache_prompt:false, seed 1, temperature 0, Qwen3-0.6B-Q4_K_M, HRX0, prompt well under 2048 tokens so this is purely the next_q8/cooperative path, #123 not involved).

Result: this is a race, not rounding. 242/300 (80.7%) give the same canonical logprob (-0.497721). The other 58/300 (19.3%) each give a different value, spread from -0.25 to -2.60 - a spread of over 2 nats, ~2000x the ~1e-3 threshold this issue proposed for benign summation-order rounding. Smaller samples (20, 60 runs) sometimes showed 0% or 5% divergence, so the rate looks contention-dependent (this is a shared box) rather than a fixed per-request probability - worth keeping in mind when anyone else tries to reproduce this with a small sample and concludes it's fixed.

The wide, unstructured spread of wrong values (not one consistent alternate answer) points toward the reducer occasionally combining a random subset of the correct partial writes with something else - stale memory, an uninitialized transient, or a partial write still in flight - rather than a simple off-by-one in which blocks get included. That would mean the producer-workgroups' kernel.barrier<global> scope(workgroup) ordering(release) before the atomic increment, and the last-workgroup's ordering(acquire) before reading, aren't actually providing a real happens-before relationship for every producer's writes on this hardware - i.e. the exact same last-workgroup-atomic-then-read pattern used by every decode-split kernel in this family, not something specific to next_q8's math.

If that's right, this isn't a next_q8-specific bug and it isn't independent of #123 either - #123's multipass path uses the identical synchronization pattern (same completion-counter/barrier structure, just far more producer workgroups), and a barrier that's too weak to guarantee cross-workgroup visibility would show up there too, plausibly as exactly the kind of wrong-address-shaped fault #123 describes (a workgroup racing ahead to read a transient before another workgroup's write to it is truly globally visible can manifest as a fault, not just a wrong value, depending on what memory it ends up touching). Someone with access to the actual barrier/atomic semantics on gfx1151 (or a way to strengthen the barrier scope and re-measure both issues' rates) is better positioned to confirm this than more black-box testing from here.

[bong-water-water-bong]

Update: built Loom's own compiler toolchain and directly inspected the barrier lowering (2026-09-26) - this narrows things down, but does not confirm the earlier barrier-visibility hypothesis; if anything it weakens it.

Built ROCm/hrx-system from source (Python 3.12 + Bazel, dev.py bazel build //loom/src/loom/tools/loom-check ...) and used loom-check's emit source-low mode to compile a minimal standalone kernel reproducing the exact same synchronization idiom used by this kernel family (kernel.barrier<global> scope(workgroup) ordering(release) on the producer side, ordering(acquire) on the consumer side, around a view.atomic.rmw<addi> {ordering = acq_rel, scope = device} completion counter), targeting amdgpu:gfx11-generic.

Actual emitted GCN/RDNA assembly:

• Acquire barrier + subsequent global load: lowers to s_barrier (intra-workgroup only) + s_waitcnt {vmcnt=0} + a global_load_b32_saddr ... {glc=1} for the load. No explicit buffer_gl0_inv/buffer_gl1_inv cache-invalidate instructions - the compiler instead uses the GLC ("globally coherent") bit on the load itself to bypass the L0 vector cache. On RDNA this is a standard, documented way to get acquire semantics for a single load and is generally considered correct, not obviously buggy.
• Release barrier + the atomic RMW itself: lowers to the data store, s_waitcnt {vmcnt=0} + s_waitcnt_vscnt {vscnt=0} (drains outstanding stores), s_barrier, then the buffer_atomic_add_u32_rtn (the counter increment), followed by s_waitcnt {vmcnt=0} and then explicit buffer_gl1_inv + buffer_gl0_inv - because that specific atomic op was declared acq_rel, not just release, so the compiler also emits real invalidate instructions there for the atomic's own acquire half.

So the compiler clearly is capable of emitting real buffer_gl0_inv/buffer_gl1_inv invalidate instructions when it decides they're needed - it just didn't for the plain acquire-barrier-then-load case, using the cheaper GLC-bypass-load idiom instead. That pairing (writeback+wait on release, GLC-bypass load on acquire) is a standard, plausible-correct AMDGPU idiom, not an obvious hole. I can't fully rule out a subtler issue (e.g. whether GLC bypass on this specific gfx1151 iGPU's cache hierarchy - L0 per-CU, L1 per shader-array, L2 device-wide - really gives full device-scope visibility, vs. only bypassing L0) without lower-level hardware documentation this box doesn't have. But the "the barrier just doesn't provide real cross-workgroup visibility" theory, as stated, is not supported by what's actually emitted.

Net effect: this rules out the simplest version of the shared hypothesis between this issue and [engine#140]. Root cause for both remains open. Repro scripts kept in case anyone wants to extend them (e.g. compiling the real multipass kernel through the full template/config pipeline and running the actual dynamic --sanitizer=race instrumentation on real hardware, which is a substantially bigger lift than this static check).

[bong-water-water-bong]

Ran the actual empirical test on real hardware (2026-09-26) - built a fresh, correctly-pinned clone of 1bit-MONSTER/llama.cpp @ 8dd75eb48 (the exact commit that shipped the multipass fix), patched the flash-attention decode-split kernel's three reduce_fused variants (direct/cooperative/multipass) to force a genuine cache invalidate before the real partial-buffer reads - a dummy view.atomic.rmw<addi> with {ordering = acq_rel, scope = device} on the completion counter, gated to a single lane, followed by a kernel.barrier<workgroup> to propagate it - based on the confirmed compiler behavior that an acq_rel atomic emits real buffer_gl1_inv/buffer_gl0_inv instructions where a plain ordering(acquire) barrier-then-load does not (see the earlier comment on this issue).

Important methodology note first: neither #123's fault nor #140's race reproduced at all on a clean, freshly-built unpatched binary under normal (low-contention) conditions - 0/54 combined attempts for #123, 0/300 for #140. Both only reproduced once I deliberately induced GPU contention with a second, concurrent llama-bench workload hammering the same HRX0 device in a loop. This matches this session's own earlier note that the divergence rate is contention-dependent, not a fixed per-request probability - worth remembering before trusting any "looks fixed" result from a quiet box.

Under matched induced contention:

• Unpatched baseline: 56/300 (18.7%) divergent logprobs - closely matching this issue's documented 58/300 (19.3%).
• Patched (forced invalidate before the reads): 208/300 (69.3%), then 170/300 (56.7%) on a repeat run - both roughly 3-3.7x worse, not better.

Conclusion: this rules out the missing-cache-invalidate hypothesis more decisively than the earlier static check did. Forcing a real, compiler-confirmed cache invalidate immediately before the reads made the race substantially more frequent, not less. The most likely explanation is that the extra synchronization changed wavefront scheduling/occupancy enough to widen whatever the actual race window is, rather than closing a coherence gap - meaning the real bug is probably a genuine scheduling/ordering hazard elsewhere in the reduction (or in how the producer side publishes its partials), not GPU cache visibility. I'm not going to guess further at the exact mechanism without stronger tooling (the dynamic --sanitizer=race instrumentation, still not attempted - would need the kernel's full template/config pipeline resolved against real device facts).

Patch discarded (this was a diagnostic experiment only, not proposed for merging). Root cause remains open for both this issue and #123.

[bong-water-water-bong]

Attempted the real dynamic race sanitizer against the actual kernel (2026-09-26) - blocked by two genuine upstream gaps, not by anything fixable from this repo.

Found that AMD's ggml-hrx layer already has plumbing for Loom's compile-time sanitizer passes via GGML_HRX_LOOM_SANITIZER (env var, loom-jit.cpp) - but it only wires up none/access/asan/value/operation/ubsan/all, never race, even though the underlying loomc C API defines LOOMC_SANITIZER_CHECK_RACE and even a ready-made LOOMC_SANITIZER_CHECKS_TSAN_LIKE preset for exactly this. Patched ggml_hrx_loom_jit_parse_sanitizer_checks to add a "race"/"tsan" case (4 lines, following the existing pattern exactly), rebuilt, and set GGML_HRX_LOOM_SANITIZER=race (plus HRX_HAL_SANITIZER=tsan for the HAL-side runtime feature) against a real llama-server serving Qwen3-0.6B on HRX0.

Two real compile-time failures surfaced immediately:

1. Missing runtime-global config: generated kernels reference a data symbol iree_tsan_config that never gets emitted - ggml-hrx's ggml_hrx_loom_jit_runtime_globals() only knows how to set LOOMC_AMDGPU_RUNTIME_GLOBAL_ASAN_CONFIG for the ACCESS check; the public C header (loomc/target/amdgpu/emit.h) doesn't define an equivalent TSAN_CONFIG global at all. This is a real gap in the public runtime-globals surface, not something fixable with a local patch.
2. The harder blocker: sanitizer-insert-race-observations cannot observe vector.fragment.load: local-memory vector operation requires lane-activity-aware race observation. Loom's race sanitizer pass cannot currently instrument WMMA fragment loads at all - and our flash-attention decode-split kernel uses exactly those (vector.fragment.load<lhs>/<rhs> for the P·V product). This affected several other WMMA-based kernels too (mul_mat_swiglu, mul_mat_q6_k_packed, mul_mat_add_f32_f32_wmma), breaking the whole forward pass when the sanitizer is enabled globally. Even with (1) fixed, the actual target kernel would still fail to compile with race instrumentation - this is an unimplemented feature in Loom's sanitizer pass itself.

Conclusion: the dynamic race sanitizer genuinely cannot be run against this kernel family with the currently pinned hrx-system version. Confirming or ruling out the actual root cause of #123/#140 further needs either an upstream fix to Loom's sanitizer pass (fragment-load support) plus the missing runtime-global plumbing, or a fundamentally different diagnostic approach (e.g. RGP/ISA-level tooling this environment doesn't have). This session tried the static compiler-output check, an empirical barrier-strengthening patch (made the race 3-3.7x worse), and now the dynamic sanitizer (blocked) - all three avenues are now closed off or exhausted with the tooling actually available. The real bug is most likely a scheduling/ordering hazard, not a cache-coherence gap, given the barrier-strengthening result, but pinning it down exactly needs the upstream sanitizer support that doesn't exist yet.

[bong-water-water-bong]

Correction to earlier comments on this issue: got the REAL production kernel to compile and disassembled it directly - the barrier lowering is NOT missing an invalidate after all.

The earlier static check (comment above) tested a minimal synthetic standalone kernel reproducing the barrier idiom in isolation, and found the acquire barrier lowered to a GLC-bypass load with no explicit invalidate instruction. That result doesn't hold for the real kernel in its full context.

Using ROCm/hrx-system's loom-compile directly with the actual production config values for this repro (Qwen3-0.6B: query_head_count=16, key_value_head_count=8, qk_head_size=128, value_head_size=128, key_value_token_capacity=2112 for a 2100-token decode - the exact #123 repro depth, landing in the multipass range), I compiled the real flash_attention_decode_split_f32_f16_wmma.loom kernel (via its actual kernel.def entry point, @ggml_flash_attention_decode_split_f32_f16_wmma_next_q8) to a real HSACO artifact, and disassembled it with llvm-objdump (from TheRock's ROCm SDK - roc-obj-extract/llvm-objdump are already present, no extra tooling needed).

The real, in-context disassembly around the completion counter and the explicit acquire barrier:

buffer_atomic_add_u32 v2, v1, s[20:23], s2 offen glc   ; completion counter increment (acq_rel)
s_waitcnt vmcnt(0)
buffer_gl1_inv
buffer_gl0_inv                                          ; <- real invalidate, atomic's own acquire half
ds_store_b32 v1, v2 offset:14592                        ; old_counter -> LDS scratch
s_barrier                                                ; kernel.barrier<workgroup> (propagate via LDS)
ds_load_b32 v1, v1 offset:14592                          ; old_counter <- LDS scratch
v_cmp_eq_i32_e64 s6, v1, 32                              ; is_last_partition? (32 = 33 blocks - 1, matches our capacity)
s_cbranch_scc0 ...                                       ; skip if not last
s_barrier                                                ; <- this is the explicit kernel.barrier<global> ordering(acquire)
s_waitcnt vmcnt(0)
buffer_gl1_inv
buffer_gl0_inv                                           ; <- a SECOND real invalidate, right here
; ... reduction over partial_max/partial_sum/partial_output begins immediately after ...

Both invalidate pairs are present and correctly placed - one for the counter atomic's own acq_rel, and a distinct second one for the explicit kernel.barrier<global> ordering(acquire) that gates entry into the reduction. The real, deployed kernel already has correct, full L0/L1 cache invalidation before reading the other workgroups' partial buffers. This directly explains the earlier empirical result (posted above): forcing an additional redundant invalidate before the reduction made the race worse rather than better, because there was nothing missing to fix - the extra synchronization was pure overhead that perturbed scheduling without closing any real gap.

This closes off the cache-coherence hypothesis for good, with ground-truth evidence rather than inference. The actual root cause of both #140's race and #123's fault is very unlikely to be a GPU memory-visibility problem. It's more likely a genuine data/algorithmic hazard - e.g. in how partial values get written by the producer side, an indexing issue that's timing-dependent, or something in the online-softmax combination itself. Confirming that precisely still needs either the (currently blocked - see above) dynamic race sanitizer, or manual disassembly analysis of the producer-side write path, which hasn't been attempted.

[bong-water-water-bong]

Tested a second, independently-motivated hypothesis: allocator-introduced buffer aliasing between dispatches. Also refuted, with the same "any perturbation makes it worse" pattern as the barrier-strengthening test.

ggml-hrx's transient buffer allocator (ggml/src/ggml-hrx/dispatch/transient-allocator.cpp, TransientAllocator::allocations_can_overlap) decides it's safe to reuse the same physical memory for two different dispatches' transient buffers using command-ordinal-based liveness intervals - a logical, program-order notion, not a guarantee about actual GPU execution-time overlap. This is structurally the same shape as an already-confirmed prior bug in this codebase ([[hrx0-fusion-overlap-bug]]: fused kernels racing on ggml-alloc-aliased buffers because there was no execution-order guarantee behind the reuse decision), so it seemed like a strong, well-motivated hypothesis for #140/#123 too.

Patched allocations_can_overlap to unconditionally return false, forcing every transient buffer (partial_max, partial_sum, partial_output, completion_counter, etc.) to get its own unique, never-reused memory allocation - eliminating any possibility of allocator-introduced false aliasing between unrelated dispatches. Rebuilt and re-ran the same induced-contention #140 repro (concurrent llama-bench workload + 300-sample logprob probe) used for the earlier barrier-strengthening test.

Result: divergence got dramatically worse, not better - 160/300 (53.3%) on the first run, 242/300 (80.7%) on a repeat, both far above the 56/300 (18.7%) unpatched-with-reuse-enabled baseline (itself matching the documented 58/300/19.3%). This refutes the buffer-aliasing hypothesis just as clearly as the earlier barrier-strengthening test refuted the cache-coherence hypothesis.

The pattern across both negative experiments is itself informative: two structurally very different interventions - adding extra synchronization/cache-invalidation, and removing all buffer reuse - both made the race substantially worse rather than better. Neither closed a real correctness gap; both just added overhead/changed memory footprint under contention. Combined with the ground-truth disassembly showing the kernel's own synchronization is textbook-correct on both the release and acquire sides, this points away from both "the kernel's own barriers are wrong" and "the allocator is aliasing buffers unsafely," and toward something more like a genuine hardware/scheduling timing sensitivity specific to this GPU under contention (gfx1151 is a very new Strix Halo APU part; this stack has already turned up other real driver/firmware-level quirks in earlier work: XRT partial-buffer-sync staleness, a from-scratch HRX0 fusion-overlap bug, etc.) - not a straightforwardly fixable software logic bug in either of these two specific places.

At this point three structurally different tooling approaches (ground-truth disassembly, empirical barrier-strengthening, empirical buffer-isolation) have converged on ruling out the two most plausible software-level hypotheses. Confirming an actual root cause from here would need either the dynamic race sanitizer (blocked upstream, see above) or hardware-level tooling (RGP, an ISA-level execution trace) this environment doesn't have.

[bong-water-water-bong]

Real GPU execution trace (2026-09-26): found continuous, heavy KFD queue eviction/restore churn correlating exactly with the contention that triggers this race - the closest thing to a concrete mechanism this investigation has produced.

Traced the server process directly under rocprofv3 (/opt/rocm-therock/bin/rocprofv3 --kernel-trace --kfd-queue-trace -- llama-server ...) during the same induced-contention #140 repro used throughout this investigation (a competing llama-bench workload on the same HRX0 device). Two notes on getting there: --attach <PID> (rocprofv3's other mode) requires the target binary to be built with rocprofiler-register attach support, which our build doesn't have - wrapping the launch directly (rocprofv3 -- <app>) works regardless. Also, the HIP/HSA API-level buffers (kernel_dispatch, hsa_api, hip_api) came back completely empty (0 records) - HRX/IREE's dispatch path doesn't go through the API layers rocprofv3 normally intercepts for those buffers, so kernel-level dispatch timing isn't directly observable this way. The KFD-level buffer, however, captured real, substantial data: 704 records over a 30-second capture, sustained at ~23 events/sec for the entire duration the competing workload was running (not a brief spike).

Decoded via rocprofiler-sdk/kfd/kfd_id.h's enums:

• 274x ROCPROFILER_KFD_EVENT_QUEUE_RESTORE_RESCHEDULED - "Queue was not restored; will be restored later." The GPU queue is repeatedly failing to restore on the first attempt and having to be rescheduled, continuously, for as long as the competing workload runs.
• 131x EVICT_SVM and 299x EVICT_USERPTR duration events - the KFD driver evicting SVM-backed and userptr-backed GPU memory (this is what the transient buffers - partial_max/partial_sum/partial_output/completion_counter - are, functionally) under memory/queue pressure from the competing workload.

Why this is a strong candidate mechanism, not just correlation: it directly explains contention-dependence (eviction pressure only exists when a competing workload creates it - exactly this issue's empirically-established trigger condition), and it sits entirely below the software layers already ruled out this investigation. The kernel's own buffer.gl0_inv/gl1_inv cache-invalidation sequence (confirmed correct via ground-truth disassembly, see above) invalidates caches - it says nothing about what happens if the underlying physical backing memory itself gets migrated/evicted and restored by the driver in the middle of the release/acquire window. If a queue eviction-restore cycle has any gap in how it re-establishes memory mappings or resyncs in-flight buffer state relative to a specific dispatch's execution, that would manifest as exactly the failure mode observed: wrong-but-plausible values, contention-dependent, present regardless of how correct the kernel's own synchronization is - and it also explains why both software-level "fixes" tried earlier (extra cache invalidation, disabling all transient buffer reuse) made things worse: both add more transient memory footprint/synchronization overhead, which would plausibly increase eviction pressure on an already-contended GPU rather than relieve it.

This is not a confirmed root cause - it's real, decisive evidence of a genuine driver/kernel-level mechanism operating exactly when and where the race occurs, one level below anything checked so far. Confirming it fully would need correlating individual eviction-event timestamps against individual divergent-response timestamps (not attempted - would need request-level tracing correlated with the KFD event stream), or a from-scratch reproduction with queue eviction suppressed (e.g. via KFD sysfs/module parameters, if any exist, to disable eviction and see whether the race disappears entirely) - a natural next experiment for anyone with the time/access to pursue it further, but not attempted here given how much ground this session has already covered.

This concludes the tooling avenues reasonably available in this environment: ground-truth disassembly (kernel sync is correct), two refuted software-level hypotheses (barrier strengthening, buffer isolation - both made things worse), a blocked dynamic sanitizer (Loom doesn't support instrumenting the WMMA ops this kernel uses), and now a real GPU-level trace pointing at driver-level queue eviction churn as the most concrete remaining lead.

[bong-water-water-bong]

Follow-up to the KFD eviction finding above: request-level correlation test, honest nuanced result - does NOT strengthen the causal claim.

Ran a second, more precise test: recorded CLOCK_MONOTONIC timestamps immediately before/after each of 150 sequential /completion requests (same clock domain as rocprofv3's KFD trace, both on strixhalo), then checked whether each request's decode window [t0, t1] overlapped a KFD eviction/restore event (RESTORE_RESCHEDULED, EVICT_SVM, EVICT_USERPTR) - both exact-window overlap and a ±20ms margin.

This run: 87/150 (58%) divergent, 42 total eviction-related events captured (a much shorter/quieter eviction window than the first trace's 704 events/30s - this capture's request burst was only ~4.2s, and eviction activity spanned ~14s, so only part of the eviction activity temporally coincides with when requests were actually firing).

Result: divergent requests did not show a higher eviction co-occurrence rate than canonical ones - if anything, slightly lower, though counts are small enough that this shouldn't be read as a confirmed refutation either:

• Exact overlap: divergent 9/87 (10.3%) vs canonical 9/63 (14.3%)
• ±20ms margin: divergent 12/87 (13.8%) vs canonical 14/63 (22.2%)

What this means: the aggregate correlation found earlier (eviction/restore churn only exists under the same contention condition that triggers the race, sustained for as long as contention exists) remains real and is still the most concrete lead. But this more precise test doesn't support the stronger, more useful claim that "an eviction event happening during a specific request's dispatch is what corrupts that request's result." Either the causal path is more indirect (e.g. a delayed effect from an earlier incomplete restore, not a same-instant one), the true corrupting window doesn't align with when the KFD event gets logged (eviction detection vs. actual corrupting window may not coincide), or eviction and the data race are both downstream symptoms of the same underlying contention pressure without one directly causing the other. With this small sample (42 eviction events against 150 requests), this test can't distinguish between those possibilities.

This is a fair, complete stopping point given available tooling: the driver-level eviction/restore churn is real and correlates with contention at the aggregate level, but pinning down precise causation would need either much larger-scale request-level sampling (to get enough eviction-event/divergent-request pairs for a meaningful statistical test) or genuinely different tooling (kernel-level tracing that timestamps the actual corrupting memory access, not just eviction detection).

[bong-water-water-bong]

The run-to-run divergence here follows the kernel migrating pages behind the process, not GPU contention: with KSM and proactive compaction off it drops from 24-38 % to 0-0.7 % on this issue's probe, and forced compaction alone brings it back (6-11 %). Full table, method and the working theory (HRX device memory on the APU is pageable/userptr-backed system memory) in #123 (comment). #170 stopped the NaN-routing fault but not this.