Skip to content

qwen4exp: fix per-block bias indexing when a unified cache holds several sequences - #29166

Draft
akionux wants to merge 1 commit into
ggml-org:masterfrom
akionux:qsa-blkbias-index-fix
Draft

akionux wants to merge 1 commit into
ggml-org:masterfrom
akionux:qsa-blkbias-index-fix

Conversation

@akionux

@akionux akionux commented Sep 20, 2026 •

Copy link
Copy Markdown

Overview

Fix the per-block (blk_bias) path of set_input_qsa: it indexes the per-sequence bid arrays by block number (bid_cell[b], bid_idx[b]), but those arrays are indexed by bid, and the block table keys a block on (sequence set, bucket). With more than one sequence in a unified cache the two no longer coincide, so a block gets the visibility of a different sequence's group - one of our blocks can be masked out, one of another sequence's can pass through.

Build an inverse block -> bid map once per ubatch, keeping the entry that owns the stream's sequence, and use it for both the seq_has test and the tail comparison. Single-sequence caches are unaffected (one_seq keeps the old correspondence).

Reproduced with two live sequences in one stream (llama-server serves each slot as its own sequence from position 0): the model stops seeing its last messages and reports the input as empty. With the change, the same concurrent shape is answered correctly in 4/4 attempts (2/4 before).

Additional information

Found while running this branch's QSA path under --parallel 4 with concurrent requests. The companion fixes for the pooled-key-cache-specific paths (per-sequence row windows, own-block fill, stream-inventory sizing) live in #28699; this PR is the part that exists on master on its own.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES - patch drafted by an AI agent (Hermes) under the author's direction on the author's account; the author is responsible for all submitted changes, per AGENTS.md/CONTRIBUTING.md.

…ral sequences

The per-block (blk_bias) path of set_input_qsa indexes the per-sequence bid arrays by
block number (bid_cell[b], bid_idx[b]), but those arrays are indexed by bid, and the
block table keys a block on (sequence set, bucket): with more than one sequence in a
unified cache the two no longer coincide, so a block gets the visibility of a different
sequence's group - one of our blocks can be masked out, one of another sequence's can
pass through. Build an inverse block -> bid map once per ubatch, keeping the entry that
owns the stream's sequence, and use it for both the seq_has test and the tail
comparison. Single-sequence caches are unaffected (one_seq keeps the old correspondence).

Reproduced with two live sequences in one stream (llama-server serves each slot as its
own sequence from position 0): the model stops seeing its last messages and reports the
input as empty. With the change, the same concurrent shape is answered correctly in 4/4
attempts (2/4 before).
@ggml-gh-bot

ggml-gh-bot Bot commented Sep 20, 2026

Copy link
Copy Markdown

Hi @akionux, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

@ggml-gh-bot ggml-gh-bot Bot added the draft PR will be changed to draft by github-actions bot label Sep 20, 2026
@github-actions
github-actions Bot marked this pull request as draft September 20, 2026 00:59
@github-actions github-actions Bot removed the draft PR will be changed to draft by github-actions bot label Sep 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant