qwen4exp: incremental pooled-key cache for the QSA indexer

[Rhonstin]

Overview

The QSA indexer of the qwen4exp architecture (Qwen3.8-Flash-Next) scored queries against block summary keys that were regathered and recomputed over the whole cached context every token, in every QSA layer - the dominant remaining decode cost at depth. This PR caches those summaries incrementally.

• one f32 summary row per complete position block per QSA layer, written by the graph via set_rows; per ubatch only the freshly completed blocks are pooled/normed/roped, and the score reads the cache
• row validity is a per-sequence block watermark clamped by seq_rm/seq_cp/seq_add/seq_div/clear and reset by full state loads; a PARTIAL_ONLY restore (speculative checkpoint replay) leaves the indexer cells untouched and its rollback arrives through seq_rm
• stale rows are finite and masked by the -inf bias, exactly like the previous garbage partial-block pools
• single-stream memories only; multi-stream and LLAMA_QSA_NO_POOLED_CACHE=1 fall back to the full recompute

Ported from @apepojken's qwen4exp-spec-mtp fork (commits 472b758 and the pooled-cache parts of the 843d575 review pass), with one deviation: the pooled rows are allocated one buffer per indexer-cache buffer type. The reference allocates all layers in a single buffer, which on a layer-split setup makes every layer but the first read and write its rows over the inter-GPU links; measured on an 8-GPU box that costs 2x decode throughput, and one buffer per device recovers it.

Additional information

A/B on 8 GPUs (1x RTX 3090 + 7x CMP 90HX; Qwen3.8-Flash-Next UD-Q3_K_XL; MTP n-max 2; q8_0 KV; ctx 131072), same binary toggled via the kill switch:

depth	pooled OFF	pooled ON	delta
63k tokens	22.25 t/s	24.33 t/s	+9.3%
114k tokens	25.41 t/s	27.80 t/s	+9.4%

Prefill unchanged within noise; greedy output bit-identical at both depths (64 tokens); prefix-reuse rollback (seq_rm replay) matches fresh runs.

Also examined from the same fork: 9b09d26 (graph reuse + shared QSA input) is already on master; 51c0c10, b6d995d and 6634bfd measured as Vulkan-specific on our CUDA box and are not included.

Requirements

• I have read and agree to the contributing guidelines
• AI usage disclosure: YES. The original implementation in @apepojken's fork discloses AI assistance; the port to master, the per-device allocation fix and all measurements in this PR were done with an AI agent under my direction and reviewed by me.

[ggml-gh-bot]

Hi @Rhonstin, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[Patt92]

Tried this on gfx1151 (Ryzen AI Max+ 395 / Radeon 8060S, ROCm 10.0.0) and hit a context-init failure. Reporting it because the isolation is fairly precise, with the caveat up front that I have not confirmed it reproduces on clean master — see the last paragraph.

The patch applies cleanly (only line offsets) and builds without warnings. At runtime:

llama_init_from_model: failed to initialize the context: unordered_map::at

test-llama-archs --arch qwen4exp fails to create a context on the ROCm device. Isolated it to the allocation, not the use: guarding out just the pooled-buffer block

if (mem_idx && mem_idx->get_n_stream() == 1) {   // -> if (false && ...)

makes the same build pass again (OK (9.68e-14), state round-trip OK) with the rest of the patch in place. LLAMA_QSA_NO_POOLED_CACHE=1 does not help, since it gates the graph side in qwen4exp.cpp but the buffers are still allocated.

The throw appears to come from llama_io_read_device::~llama_io_read_device in src/llama-context.cpp:

for (auto & [buft, mbuf] : mbufs_new) {
    const auto & mbuf_cur = mbufs.at(buft);

which would fit the one-buffer-per-buffer-type deviation described in the PR body: the set of buffer types seen on restore no longer matches the set recorded on save.

Caveat, and the reason I am not filing this as a confirmed bug: the tree I tested on is a fork of master that carries local Qwen3.8-Flash-Next MTP work and speculative-checkpoint machinery, so mem_storage may be exercised differently here than on master. It is entirely possible this is an interaction with that and not a defect in the PR. If it is useful I can try to reproduce on a clean master checkout.

The idea itself is very welcome here — the analogous pooled-key cache for GLM-5.3-Flash was the largest prefill win we measured on this hardware, and the per-buffer-type allocation is exactly right for a layer-split setup, which is what we run (one local GPU plus one RPC worker).

[Rhonstin]

Thanks @Patt92 — this is a real defect in the PR, and not your fork or ROCm-specific: I reproduced it on a clean checkout with the repo's own test (CPU backend): test-llama-archs --arch qwen4exp → llama_init_from_model: failed to initialize the context: unordered_map::at.

The throw is in the pooled-cache constructor, not in state I/O. It walks every layer with dsv4_compress_ratios[il] > 0 and calls mem_idx->get_k_storage(il), but the indexer cache is filtered to dense-attention layers (filter_idx = il < n_layer() && !is_recr(il)). The synthetic model built by test-llama-archs sets ratio 4 for all layers, and with full_attention_interval = 2 its even layers are recurrent — so get_k_storage(0) lands in map_layer_ids.at(0) in src/llama-kv-cache.cpp and throws std::out_of_range("unordered_map::at"). ~llama_io_read_device uses a std::map, so its at() would have reported map::at — close, but one level off, and your isolation of the allocation was exactly right.

Fixed in the branch (commit amended): both the ratio pick and the row allocation now skip recurrent layers. test-llama-archs --arch qwen4exp passes with the pooled cache active (OK (0.00e+00), state round-trip OK), and the LLAMA_QSA_NO_POOLED_CACHE=1 fallback passes too. Fair point that the kill switch only gated the graph side — the crash was in the allocation.

Thanks for testing on gfx1151 — reports from outside the NVIDIA box are exactly what this change needed.

[sodre90]

Thank you for the improvement! I am testing it and I realized vision is broken (5070 Ti + Ryzen 9950X 128G DDR5).

These are from an agent, I hope it helps:

set_input_qsa and qsa_pooled_n_dirty_max derive n_complete from two different sources, and the two disagree whenever M-RoPE ranking is active.

Filling (llama-memory-hybrid-idx.cpp, the assert site) counts cache cells:

n_complete = bid_idx[n_bid - 1] / r + 1

Under ranked mode, blocks are assigned by dense rank over live cells, so this works out to ~n_live_cells / r. Sizing (qsa_pooled_n_dirty_max) instead uses temporal position: n_complete = (q_max + 1) / ratio.

Ranked mode is switched on by the author's own code for exactly this case:

// mrope repeats one position across an image, so rank cells instead of using the position
if (dup && ubatch->is_pos_2d() && one_seq) { ... ranked = true; }

An image chunk adds ~1024 cells while advancing q_max by almost nothing. So the fill's block count outruns the sizing's by ~1024/r blocks, independent of text length.

Any single image request to a qwen4exp server with an mmproj loaded. Aborts with:

GGML_ASSERT(n_dirty <= n_dirty_max && "dirty tables sized at graph build; see qsa_pooled_n_dirty_max")

Stack: set_input_qsa <- llm_graph_input_qsa::set_input <- mtmd_helper_decode_image_chunk

The kill switch doesn't save this one either. LLAMA_QSA_NO_POOLED_CACHE=1 gates the graph side in qwen4exp.cpp, but dirty_rows is still created and set_input_qsa still asserts.

[akionux]

Heads-up from a downstream user of this branch on qwen4exp with parallel > 1: the pooled path has a second cross-sequence bug, on top of the ones already reported here.

Symptom. With two live sequences sharing one unified KV cache (llama-server serves every slot as its own sequence from position 0, and Hermes-style agents fire small auxiliary requests concurrently), the model intermittently answers as if the user's last messages were empty - it literally cannot see them, and invents a name for the empty input. Probabilistic: re-sending the same prompt succeeds (~2/4 failures with concurrency, 0/4 without).

Root cause. Pooled rows are keyed by position block only, with no sequence identity. Two sequences holding different cells at the same positions therefore write and score the same rows, and whichever wrote last wins: the loser reads the other sequence's summaries as its own tail blocks. The guard instrumentation on my tree logged cross_seq_rows=1, foreign_bids=140 on a failing turn.

Fixes in the patch below (built and run on this branch @ 141f3f5):

• rows are laid out per sequence (n_seq_max windows of pooled_rows_per_seq; a sequence's rows start at pooled_row_base()), and the score views only its own window;
• the fill stops at the last block this sequence owns - other sequences' complete groups are not ours to re-pool, holes go to the dustbin;
• the per-block bias builds an inverse block->bid map instead of treating bid-array index as block number (also filed separately as qwen4exp: fix per-block bias indexing when a unified cache holds several sequences #29166 since that part exists on master).

This sizing change also covers @sodre90's vision report: set_input_qsa counts complete blocks over the stream's whole cell array - under M-RoPE ranked mode an image chunk adds ~1024 cells while barely advancing q_max, so the position-only bound in qsa_pooled_n_dirty_max comes out short and the assert fires. Sizing from the inventory (plus slack) fixes that path too.

Cost: the pool grows to n_seq_max x rows; at --parallel 1 it is unchanged.

Verified: same concurrent shape as the repro (8.8k-token main request + small concurrent request): 2/4 wrong before, 4/4 correct after, zero guard violations.

diff --git a/src/llama-memory-hybrid-idx.cpp b/src/llama-memory-hybrid-idx.cpp
index 9087b3562..b979171e1 100644
--- a/src/llama-memory-hybrid-idx.cpp
+++ b/src/llama-memory-hybrid-idx.cpp
@@ -84,8 +84,14 @@ llama_memory_hybrid_idx::llama_memory_hybrid_idx(
 
         if (ratio > 0 && idx_dim > 0) {
             // + 1 so a partial trailing block has a slot, + 1 dustbin row for padded writes
-            pooled_rows  = kv_size/ratio + 2;
-            pooled_ratio = ratio;
+            // [TAG_QSA_OWNROW] rows are laid out per sequence: sequences sharing one unified
+            // stream hold different cells at the same positions, so one row per position block
+            // cannot serve them all - each would score the other summaries. pooled_rows stays
+            // the total (allocation + views); a sequence's rows start at pooled_row_base().
+            pooled_rows_per_seq = kv_size/ratio + 2;
+            pooled_n_seq_max    = std::max<uint32_t>(1, n_seq_max);
+            pooled_rows         = pooled_rows_per_seq * pooled_n_seq_max;
+            pooled_ratio        = ratio;
 
             // one context+buffer per device: the indexer caches of the QSA layers are spread
             // across the layer-split devices, and a row written by a device that does not own
@@ -689,7 +695,8 @@ void llama_memory_hybrid_idx::set_input_qsa(
             GGML_ASSERT(n_ns == 1 && "the pooled cache path is single-stream only");
 
             const int64_t n_dirty_max = dirty_rows->ne[0];
-            const int64_t dustbin     = (int64_t) get_pooled_rows() - 1;
+            const int64_t row_base    = pooled_row_base(seq_of_stream);   // [TAG_QSA_OWNROW]
+            const int64_t dustbin     = row_base + (int64_t) pooled_rows_per_seq - 1;
 
             int32_t * dst_d_cells = (int32_t *) dirty_cells->data;
             int32_t * dst_d_pos   = (int32_t *) dirty_pos->data;
@@ -697,9 +704,23 @@ void llama_memory_hybrid_idx::set_input_qsa(
 
             // the bids are the complete blocks, pushed in position-block order: the last
             // bid's block ends the complete range
-            const int64_t n_complete = n_bid > 0 ? (int64_t) bid_idx[n_bid - 1]/r + 1 : 0;
+            //
+            // [TAG_QSA_OWN] only a block this sequence owns may end it. In a unified cache the
+            // bid list also carries the complete groups of every other live sequence, and taking
+            // the last one let a short request adopt the whole inventory of a long one: it then
+            // pooled (and overwrote) rows belonging to that sequence. The bias masks blocks we
+            // do not own anyway, so nothing of ours is lost by stopping at our own last block.
+            int64_t n_complete = 0;
+
+            for (int32_t t = n_bid - 1; t >= 0; --t) {
+                if (one_seq || cells.seq_has((uint32_t) bid_cell[t], seq_of_stream)) {
+                    n_complete = (int64_t) bid_idx[t]/r + 1;
+                    break;
+                }
+            }
 
             auto & w = pooled_valid(seq_of_stream);
+
             w = std::min(w, n_complete);
 
             const int64_t n_dirty = n_complete - w;
@@ -710,9 +731,18 @@ void llama_memory_hybrid_idx::set_input_qsa(
             std::vector<int32_t> pb_bid(n_complete > 0 ? (size_t) n_complete : 1u, -1);
             for (int32_t t = 0; t < n_bid; ++t) {
                 const int64_t pb = bid_idx[t]/r;
-                if (pb < n_complete) {
-                    pb_bid[pb] = t;
+
+                if (pb >= n_complete) {
+                    continue;
                 }
+
+                // [TAG_QSA_OWN] a block of another sequence is not ours to pool: leave its row
+                // to whoever owns it (the bias masks it for us anyway)
+                if (!one_seq && !cells.seq_has((uint32_t) bid_cell[t], seq_of_stream)) {
+                    continue;
+                }
+
+                pb_bid[pb] = t;
             }
 
             for (int64_t i = 0; i < n_dirty_max; ++i) {
@@ -720,17 +750,40 @@ void llama_memory_hybrid_idx::set_input_qsa(
                 const int64_t b    = w + i;
                 const int32_t t    = live && b < n_complete ? pb_bid[b] : -1;
 
-                dst_d_rows[i] = live ? b : dustbin;
+                // [TAG_QSA_OWN] a hole is not ours to write: the dustbin holds it and the owner's
+                // row keeps its content, instead of being pooled from cell 0 (and certified)
+                const bool pooled = live && t >= 0;
+
+                dst_d_rows[i] = pooled ? row_base + b : dustbin;   // [TAG_QSA_OWNROW]
 
                 for (int64_t sec = 0; sec < 4; ++sec) {
-                    dst_d_pos[sec*n_dirty_max + i] = t >= 0 ? loc_blk_pos[sec*n_blocks + t] : 0;
+                    dst_d_pos[sec*n_dirty_max + i] = pooled ? loc_blk_pos[sec*n_blocks + t] : 0;
                 }
                 for (int64_t j = 0; j < r; ++j) {
-                    dst_d_cells[i*r + j] = t >= 0 ? loc_blk_cells[t*r + j] : 0;
+                    dst_d_cells[i*r + j] = pooled ? loc_blk_cells[t*r + j] : 0;
                 }
-            }
 
+            }
             w = n_complete;
+
+        }
+
+        // [TAG_QSA_BIASFIX] the block bias below is indexed by block, but the bid arrays are
+        // indexed by bid, and a unified cache with a second sequence interleaves one entry per
+        // sequence at the same position block. Build the inverse map once per ubatch, keeping the
+        // entry that owns this sequence's cells: an entry at index b then really describes block b.
+        std::vector<int32_t> blk_entry(n_blocks, -1);
+
+        for (int32_t t = 0; t < n_bid; ++t) {
+            const int64_t pb = bid_idx[t]/r;
+
+            if (pb >= n_blocks || blk_entry[pb] >= 0) {
+                continue;
+            }
+
+            if (cells.seq_has((uint32_t) bid_cell[t], seq_of_stream)) {
+                blk_entry[pb] = t;
+            }
         }
 
         for (int64_t ii = 0; ii < n_tps; ++ii) {
@@ -771,13 +824,17 @@ void llama_memory_hybrid_idx::set_input_qsa(
                 float * cur_blk_bias = dst_bias + i*n_blocks;
 
                 for (int64_t b = 0; b < n_blocks; ++b) {
-                    if (b >= n_bid || !cells.seq_has((uint32_t) bid_cell[b], seq_id)) {
+                    // [TAG_QSA_BIASFIX] the entry that describes block b (not entry b)
+                    const int32_t t = blk_entry[b];
+
+                    if (t < 0) {
+                        // no complete group of ours at this block: invisible, as before
                         cur_blk_bias[b] = -INFINITY;
                         continue;
                     }
 
                     // finite, so it can never meet a -inf and produce a nan
-                    cur_blk_bias[b] = bid_idx[b] >= tail_start ? 1e9f : 0.0f;
+                    cur_blk_bias[b] = bid_idx[t] >= tail_start ? 1e9f : 0.0f;
                 }
 
                 // the spare block holds the unpooled cells, which are the incomplete tail, so
@@ -916,6 +973,31 @@ uint32_t llama_memory_hybrid_idx_context::get_pooled_rows() const {
     return mem != nullptr ? mem->get_pooled_rows() : 0;
 }
 
+int64_t llama_memory_hybrid_idx_context::pooled_row_base(llama_seq_id seq_id) const {   // [TAG_QSA_OWNROW]
+    return mem != nullptr ? mem->pooled_row_base(seq_id) : 0;
+}
+
+int64_t llama_memory_hybrid_idx::qsa_stream_idx_max(llama_seq_id seq_id) const {
+    const llama_kv_cache * idx_cache = get_mem_idx();
+
+    if (idx_cache == nullptr) {
+        return 0;
+    }
+
+    const auto & cells = idx_cache->get_cells(seq_id);
+
+    // the fill keys blocks on the cell array: in position space the largest index is the largest
+    // position, in ranked (mrope) space it is the cell count - take the larger of the two so
+    // neither keying can outrun the tables
+    int64_t idx = (int64_t) cells.get_used() - 1;
+
+    for (llama_seq_id sq = 0; sq < LLAMA_MAX_SEQ; ++sq) {
+        idx = std::max(idx, (int64_t) cells.seq_pos_max(sq));
+    }
+
+    return std::max<int64_t>(idx, 0);
+}
+
 uint32_t llama_memory_hybrid_idx_context::qsa_pooled_n_dirty_max(const llama_ubatch & ubatch, uint32_t ratio) const {
     GGML_ASSERT(ratio > 0);
     GGML_ASSERT(mem != nullptr);
@@ -923,7 +1005,9 @@ uint32_t llama_memory_hybrid_idx_context::qsa_pooled_n_dirty_max(const llama_uba
     // the reserve pass builds worst-case graphs from a mock ubatch with no seq/pos data;
     // give it the per-ubatch bound (the refill after a state load resizes on a live ubatch)
     if (ubatch.seq_id == nullptr || ubatch.seq_id[0] == nullptr || ubatch.pos == nullptr) {
-        return (ubatch.n_tokens + ratio - 1)/ratio + 1;
+        const uint32_t res = (ubatch.n_tokens + ratio - 1)/ratio + 1;
+
+        return res;
     }
 
     // single-stream memories only (get_pooled_k gates the callers); like the block tables,
@@ -935,7 +1019,17 @@ uint32_t llama_memory_hybrid_idx_context::qsa_pooled_n_dirty_max(const llama_uba
         q_max = std::max(q_max, ubatch.pos[i]);
     }
 
-    const int64_t n_complete = (int64_t) (q_max + 1)/ratio;
+    // [TAG_QSA_SIZING] the fill counts complete blocks over the stream's whole cell array (in a
+    // unified cache that is every sequence sharing it), not over this ubatch's positions: size
+    // for that inventory plus the tokens this ubatch adds, or the tables come out short and
+    // set_input_qsa aborts. In the steady state both agree and the table stays one row.
+    const int64_t idx_max = std::max<int64_t>(mem->qsa_stream_idx_max(seq), (int64_t) q_max) +
+                            (int64_t) ubatch.n_tokens;
+
+    // +3 blocks: one for the boundary rounding, two for a slot that adds cells between the
+    // graph build and the fill. Rows past the real range are dustbin-padded, so slack costs a
+    // few pooled rows, never correctness
+    const int64_t n_complete = std::max<int64_t>((int64_t) (q_max + 1)/ratio, idx_max/ratio + 3);
     const int64_t w          = std::min(mem->pooled_valid(seq), n_complete);
 
     return (uint32_t) std::max<int64_t>(1, n_complete - w);
diff --git a/src/llama-memory-hybrid-idx.h b/src/llama-memory-hybrid-idx.h
index 6b49e2369..6a2da4150 100644
--- a/src/llama-memory-hybrid-idx.h
+++ b/src/llama-memory-hybrid-idx.h
@@ -101,11 +101,24 @@ public:
     // it and replay recomputes the range. Rows at or beyond the watermark may hold stale but
     // finite data and are masked by the -inf bias. Single-stream memories only.
     ggml_tensor * get_pooled_k(int32_t il) const;              // nullptr: no indexer / multi-stream
-    uint32_t get_pooled_rows() const { return pooled_rows; }   // rows per stream, incl. trailing dustbin row
+    uint32_t get_pooled_rows() const { return pooled_rows; }   // total rows: one window per sequence
+
+    uint32_t get_pooled_rows_per_seq() const { return pooled_rows_per_seq; }
+
+    // [TAG_QSA_OWNROW] seq_id rows live in [pooled_row_base(seq), +pooled_rows_per_seq). One
+    // window per sequence, so two sequences of a unified stream never score each other pools.
+    int64_t pooled_row_base(llama_seq_id seq_id) const {
+        return (int64_t) ((uint32_t) seq_id % pooled_n_seq_max) * (int64_t) pooled_rows_per_seq;
+    }
 
     // blocks of seq_id whose pooled rows are known valid; mutable like a cache's bookkeeping
     int64_t & pooled_valid(llama_seq_id seq_id) const;
 
+    // [TAG_QSA_SIZING] largest cell index the indexer cache of seq_id's stream holds: the fill
+    // counts complete blocks over that whole array (every sequence sharing it when the KV cache
+    // is unified), so the dirty tables have to cover it, not just one ubatch's positions.
+    int64_t qsa_stream_idx_max(llama_seq_id seq_id) const;
+
 private:
     // forget seq_id (all of it if seq_id < 0) in every cache at once, so a failed restore cannot leave the caches out of step
     // seq_id < 0 drops the whole context, as the caches themselves do on a failed restore
@@ -124,8 +137,10 @@ private:
     std::vector<ggml_backend_buffer_ptr> pooled_bufs;
     std::map<int32_t, ggml_tensor *> pooled_k;
 
-    uint32_t pooled_rows  = 0;
-    uint32_t pooled_ratio = 0;
+    uint32_t pooled_rows         = 0;   // total rows: pooled_rows_per_seq * pooled_n_seq_max
+    uint32_t pooled_rows_per_seq = 0;   // rows per sequence, incl. the trailing dustbin row
+    uint32_t pooled_n_seq_max    = 1;
+    uint32_t pooled_ratio        = 0;
 
     mutable std::unordered_map<llama_seq_id, int64_t> pooled_w;
 
@@ -195,6 +210,8 @@ public:
 
     uint32_t get_pooled_rows() const;
 
+    int64_t pooled_row_base(llama_seq_id seq_id) const;   // [TAG_QSA_OWNROW]
+
     // capacity the dirty tables need for this ubatch: completed blocks plus pending refill
     // below the watermark; stable at 1 during steady decode so graph reuse holds
     uint32_t qsa_pooled_n_dirty_max(const llama_ubatch & ubatch, uint32_t ratio) const;
diff --git a/src/models/qwen4exp.cpp b/src/models/qwen4exp.cpp
index 9fb78deb3..6fc9c9cd2 100644
--- a/src/models/qwen4exp.cpp
+++ b/src/models/qwen4exp.cpp
@@ -665,8 +665,12 @@ ggml_tensor * llama_model_qwen4exp::graph::build_qsa_top_k(
                 idx_dim, store->ne[1], store->nb[1], 0);
         ggml_build_forward_expand(gf, ggml_set_rows(ctx0, store_view, fresh, inp->dirty_rows));
 
+        // [TAG_QSA_OWNROW] score only this sequence own rows: the store holds one window per
+        // sequence, and a dense matmul over all of it would mix another sequence summaries in
+        const int64_t pooled_base = mctx_hyb->pooled_row_base(ubatch.seq_id[0][0]);
+
         pooled = ggml_view_3d(ctx0, store, idx_dim, n_blocks, 1,
-                store->nb[1], store->nb[1]*n_blocks, 0);
+                store->nb[1], store->nb[1]*n_blocks, pooled_base*store->nb[1]);
         cb(pooled, "indexer_k", il);
     } else {
         // gathers per stream: blk_cells row s indexes stream s's own cells

Happy to open a PR against this branch if that's useful.

[thregit]

Some numbers from Vulkan on AMD in case they're useful. On this machine, the re-pooling this PR removes seems to be most of the long-context decode cost, so the gain here should be much
bigger than the +9% measured on CUDA.

Setup: Ryzen AI Max+ 395 (Radeon 8060S, gfx1151, 128 GB), Ubuntu 24.04, Mesa 25.2.8 RADV, llama.cpp b11111 (4ceb171), Vulkan, a Qwen3.8-Flash-Next finetune in UD-Q4_K_XL, -fa on, K/V cache
q8_0, -c 204800, --parallel 1.

llama-server decode: 22.1 t/s at 4K context, 17.5 at 32K, 10.7 at 99K, 7.4 at 188K (45 → 57 → 94 → 135 ms per token).

I haven't run this branch. Instead I rebuilt the QSA part of one decode step as a standalone ggml graph and timed it on the same GPU. It uses the same calls in the same order as
build_qsa_top_k + build_attn_qsa in b11111: 12 QSA layers, ratio 4, top-k 2048, q8_0 caches, random data. Times are ms per token, median of 15:

┌────────┬────────────────┬─────────────────┬────────────┬───────────┐
│ cells │ whole QSA part │ indexer + top-k │ mask build │ sparse FA │
├────────┼────────────────┼─────────────────┼────────────┼───────────┤
│ 4096 │ 2.5 │ 1.9 │ 0.2 │ 0.5 │
├────────┼────────────────┼─────────────────┼────────────┼───────────┤
│ 32768 │ 10.4 │ 6.9 │ 0.2 │ 2.4 │
├────────┼────────────────┼─────────────────┼────────────┼───────────┤
│ 98304 │ 45.1 │ 40.3 │ 0.2 │ 5.8 │
├────────┼────────────────┼─────────────────┼────────────┼───────────┤
│ 188416 │ 87.6 │ 83.4 │ 0.3 │ 10.8 │
└────────┴────────────────┴─────────────────┴────────────┴───────────┘

This accounts for ~43 of the ~48 ms that decode slows down between 4K and 99K, and ~85 of ~90 ms at 188K.

GGML_VK_PERF_LOGGER at 98304 cells (12 layers, one token, 44.4 ms total):

• CONT: 88x = 20.1 ms, mostly the 48 strided pooling slices at ~0.4 ms each
• GET_ROWS: 16x = 6.0 ms
• ADD: 76x = 4.9 ms
• FLASH_ATTN_EXT: 12x = 4.7 ms
• RMS_NORM_MUL (128, 24576): 12x = 2.6 ms
• ROPE: 24x = 2.2 ms
• TOPK_QSA + TOP_K: 2.0 ms
• SCALE: 12x = 0.7 ms
• scoring MUL_MAT_VEC: 0.5 ms

So roughly 35 of the 44 ms goes to pooling, normalizing and roping blocks that are already finished. If the pooled cache removes that, a rough estimate for this machine is ~17 t/s instead
of 10.7 at 99K, and ~16 instead of 7.4 at 188K. That's just subtraction, not measured. #28213 would then mostly cover the remaining FA part.

Happy to share the benchmark source if it helps.