[ROCm/gfx1201] PFlash scorer: FP stage ~45x slower than the CUDA reference (343s for S=120k vs 7.68s on a 3090), so PFlash is net-negative at long context

[aiworkeryuan-chuxi]

Summary

On a R9700 (gfx1201, RDNA4) with the official ghcr.io/luce-org/lucebox-hub:rocm image, PFlash's scorer forward spends 98% of its time in the FP stage, and that stage runs ~45× slower than the CUDA reference for a shorter sequence:

[qwen3-0.6b-fp] forward 343.31s (S=119925, A_setup=0.01s A_alloc=0.00s A_compute=2.16s FP=337.60s
                B_warm=0.00s B_setup=0.00s B_alloc=0.00s B_copy_in=0.01s B_norm=0.15s B_compute=3.49s
                B_copy_out=0.01s)  tail-score 1.16s (layers 0-27)  total 344.47s

Compare with the CUDA log in #147 for the same stage at a longer sequence:

[qwen3-0.6b-fp] forward 7.68s (S=130197, ...)

S=119925 → 343.3 s here, S=130197 → 7.68 s there. Both are the 0.6B scorer; A_compute (2.16 s) and B_compute (3.49 s) are entirely reasonable, so it is specifically FP that is pathological.

Effect on end-to-end prefill

The compression cost is not included in the reported prefill_ms, so the engine number looks excellent while the wall clock does not:

prompt tokens	wall	engine prefill_ms	compression (wall − prefill)
40,553	55.4 s	8.8 s (4,618.7 tok/s)	46.6 s
119,926	448.8 s	100.7 s (1,190.8 tok/s)	348.1 s

Compression cost vs S: 20k → 15.5 s, 40k → 46 s, 120k → 348 s — close to O(S²), i.e. the scorer is behaving like dense attention over the whole sequence rather than a sparse/reduced path.

Resulting end-to-end gain at 120k is 1.38–1.51×, not the ~10× the engine figure implies. Same prompt, same box:

configuration	wall	NIAH retrieval
PFlash off (q4_0 KV, reference)	618.3 s	YES
PFlash always	409.2–448.8 s	YES
plain llama.cpp Vulkan (same box, same class of weights)	~235 s TTFT	YES

So on this card PFlash is a net loss against the alternative at the top of the context range, even though retrieval quality survives.

Things I checked before filing

• LUCE_FP_USE_BSA=1 LUCE_FP_ALPHA=0.85 (per PFlash NIAH benchmark on RTX 5090 #82) makes no difference on gfx1201. Startup confirms fp_use_bsa = ON and there is no fallback warning. 40k: 55.4 s vs 54.5 s without it. 120k: 448.8 s vs 445.5 s.
• Not thermals: junction 54 °C, mclk locked at 1258 MHz, 175 W, no throttling during the run.
• Not --max-ctx (Oversized --max-ctx silently destroys attention throughput (20× prefill slowdown at 32K, 8× at 64K) #10): 131072 vs a tight 81920 at 76k differs by 1.2%.
• Not the drafter GGUF: tried two (unsloth/Qwen3-0.6B-BF16.gguf and a converted variant), same behaviour. Each request logs [compress] loading drafter — worth a separate look, but the 343 s is measured inside the forward, not the load.
• Not n-max / block size: --draft-block-size 16 and 8 behave the same for the compression stage.
• Not the KV type: q8_0 and q4_0 give identical prefill at 40.5k (452.5 vs 452.2).

Hypothesis

The FP stage (flash-prefill / block-sparse scorer) has no working RDNA4 implementation in the published ROCm build, so it is silently running a fallback. Supporting evidence:

• HIP build (gfx1151) fails to link: undefined symbol flash_prefill_forward_bf16 #215: the HIP build gated DFLASH27B_HAVE_SM80_FLASHPREFILL out asymmetrically, so flash-prefill kernels were compiled out entirely on HIP.
• PFlash NIAH benchmark on RTX 5090 #82: on CUDA the scorer needs DFLASH_FP_USE_BSA=1 set explicitly; without it the path is 3.4× slower and NIAH fails at 32k. Here, setting it changes nothing — consistent with the flag having nothing to switch to.
• server/docs/HIP_PERF_PLAN.md describes RDNA effort concentrated on mul_mat_q paths.
• Related: gfx1100 (RX 7900 XTX, ROCm 6.4.1): dflash_server ~2x slower than llama.cpp, DDTree speculation never engages, SIGSEGV at long context #457 reports the rocWMMA flash-prefill kernels being unavailable on gfx1100 (DDTree silently falling back to AR decode), and the maintainer reply there confirms #456 only fixes the compile on ROCm ≥ 7.2.

What I'm asking for

1. Is the FP/flash-prefill stage expected to be usable on gfx1201/ROCm 7.2.4 today? If not, would you consider refusing or loudly warning when --prefill-compression is enabled on an arch where the fast path is missing, rather than silently spending 45× longer? Right now the only symptom is a wall-clock that contradicts the reported prefill_ms.
2. If it is expected to work, is there a build flag / env var we are missing beyond DFLASH_FP_USE_BSA / DFLASH_FP_ALPHA? I'm happy to rebuild any branch and rerun on this box.
3. Worth documenting the "compression time is not in prefill_ms" caveat next to the PFlash throughput claims — independently of the arch issue, it makes the headline number hard to compare with any other engine's end-to-end prefill.

Reproduction

Image ghcr.io/luce-org/lucebox-hub:rocm (built 2026-09-21, commit 2785c22), R9700 32 GB, ROCm 7.2.4:

docker run -d --name luceb \
  --device /dev/kfd --device /dev/dri --group-add video --group-add render \
  --security-opt seccomp=unconfined -p 8080:8080 \
  -e LUCE_FP_USE_BSA=1 -e LUCE_FP_ALPHA=0.85 \
  --entrypoint /opt/lucebox-hub/server/build/luce_server \
  ghcr.io/luce-org/lucebox-hub:rocm \
  <target.gguf> --host 0.0.0.0 --port 8080 --max-ctx 131072 \
  --draft <dflash2-q8_0.gguf> --ddtree --ddtree-budget 22 --draft-block-size 16 \
  --prefill-compression always --prefill-drafter <Qwen3-0.6B-BF16.gguf>

then POST a ~120k-token prompt to /v1/chat/completions and read the [qwen3-0.6b-fp] forward line from docker logs. Per-request figures come from the response's own prefill_ms / decode_tokens_per_sec fields, compared against wall clock measured around the HTTP call.

[Graffioh]

Hi, thanks for the report!

We are currently reworking the PFlash system, so keep an eye on this repo and our blog over the coming days and weeks, where we’ll be sharing updates

It’s a long journey, but we’ll keep improving the system to deliver the best possible user experience and speed without sacrificing quality.