Repository navigation
fix: correct AFMoE no-ID prefill contract - #2301
Merged
Merged
Conversation
Disable token-ID chunked prefill because independent multi-token forwards without a sequence ID reset the model-owned sliding and full-attention KV caches. Document the whole-prompt memory tradeoff for all callers, including DirectEngine. Add a real six-layer loaded-model regression with deterministic nonzero weights that forces evaluation and checks every cache resets for a new no-ID prompt while stable-ID continuation retains all token positions. Refs #2289
Include the merged Qwen3 Next prefill contract before validating the AFMoE follow-up against the current default branch. The reviewed AFMoE source and regression remain unchanged.
inureyes
added
status:review
Merge current main through the mlxcelverse migration and revalidate mixed sliding/full attention state on MLX 243bdf09. All 29 focused CUDA tests pass, including loaded cache lifecycle and snapshots. CUDA Clippy with warnings denied, formatting, and diff checks pass. Refs #2289
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opt AFMoE out of token-ID chunked prefill because each multi-token no-ID forward starts an independent prompt and replaces the model-owned cache set. Add a loaded six-layer regression covering four rotating sliding-attention caches and two standard full-attention caches. Independent prompts reset every cache; stable sequence-ID chunks advance them continuously.
Integrate main
a2698835, including MLX243bdf09and mlxcelverse, without source conflicts. The rebuilt CUDA test-fast binary passes all 29 AFMoE tests, including loaded-model lifecycle and snapshot coverage. CUDA Clippy with warnings denied, formatting, and diff checks pass. Earlier shared integration checks passed 123 Python tests and 49 WebUI fixtures; bilingual reports preserve their provenance and record the fresh pin revalidation.Review found no remaining blocking correctness, security, or performance issue. DirectEngine's stable-ID state behavior remains intact, but the global capability makes every honoring caller use whole-prompt prefill, which may require more transient memory for long prompts. Tests use loaded synthetic weights, not an external checkpoint.
Closes #2289