Repository navigation
fix: opt Qwen3-Next out of no-ID chunked prefill - #2295
Merged
Merged
Conversation
Qwen3-Next resets its model-owned GatedDelta and KV state when a multi-token forward carries no sequence ID. Advertise the conservative single-pass policy so cache-less callers cannot silently discard earlier prompt pieces while preserving independent-prompt resets and stable sequence-ID continuation. Add a one-layer GatedDelta model loaded from deterministic nonzero weights. The regression forces both paths and verifies offset 3 after two independent no-ID prompts and offset 5 after two forwards with one sequence ID. The CUDA library test graph builds, the focused GPU test passes, and formatting and diff checks pass. Refs #2285
inureyes
added
status:review
`PrefillCaps` consumes the model-wide chunking capability for every token-ID prefill plan. Clarify that DirectEngine's stable sequence ID makes its state handling safe while the false capability still forces DirectEngine and cache-less callers to build one whole-prompt graph. This changes only the source comment, PR documentation, and bilingual reports. The prior focused GPU test and contract results remain applicable. Refs #2285
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Qwen3NextModel::from_weightsthat proves independent no-ID prompts reset to offset 3 while a stable sequence ID continues to offset 5.Validation
mlxcelandmlxcel-surgerylibrary test graph builds successfully, andqwen3_next_loaded_model_chunking_contract_is_safepasses 1/1 under the GPU lock.Current DirectEngine prefill keeps one stable sequence ID across its pieces, so its state handling is safe.
PrefillCapsconsumes this model-wide false capability for every token-ID prefill plan, however, so DirectEngine and cache-less callers both build one whole-prompt graph instead of bounded chunks and can use more transient memory for long prompts.Closes #2285