docs(vision): a tier move needs the unquantized arm before it means anything - #321
Merged
Merged
Conversation
…nything A numerically wrong kernel can score higher than the correct one when its error compensates for a quantization loss on the scored sample. Measured on gemma4:31b's 9px tier: bf16, which never enters QuantizedMatmul, scores 4; nvfp4 on the fixed kernel scores 3; nvfp4 on the broken kernel scored 4. Twelve runs had split cleanly by build across both think modes, which read as a robust regression. It was not one — a deterministic bug reproduces a lucky answer perfectly, so run-to-run consistency carries no information about whether the answer was earned. Also records that gemma4:31b-mxfp8 carries a bf16 vision tower, not an mxfp8 one (its vision down_proj blob is byte-identical in size to bf16's), so it is a second bf16-encoder arm rather than an independent vision quantization — and that it still drops to 3 at think-off, meaning language-model quantization alone moves the tier. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the methodology entry behind the correction landed in #310.
The finding. A numerically wrong kernel can score higher than the correct one, because the error it introduces may compensate for a quantization loss on the scored sample. On
gemma4:31b's 9px fine-text tier, one binary (0.34.0-maxusai-8a7ba949/ payloadd9add9d1, post-MLX#3912), same window, powermode 2:31b-nvfp4[4,4,4,3,3][4,4,4,3,3]31b-mxfp8[4,4,4,3,2][4,4,4,4,3]31b-mlx-bf16[4,4,4,4,3][4,4,4,4,3]bf16 never enters
QuantizedMatmul(x/mlxrunner/model/linear.godispatches on.weight_scale; the bf16 checkpoint has zero across 1247 tensors). It scores 4, so 4 is the model's answer. nvfp4 on the fixed kernel scores 3; nvfp4 on the broken kernel scored 4. ollama#3912 did not cost a tier — it stopped hiding one.Why this was nearly missed. Twelve runs split cleanly by build across both think modes, which read as a robust regression. A deterministic bug reproduces a lucky answer perfectly, so run-to-run consistency says nothing about whether the answer was earned.
Second trap.
gemma4:31b-mxfp8carries a bf16 vision tower, not an mxfp8 one — its visionmlp.down_projblob is byte-identical in size to bf16's (9,916,560 = 1152 × 4304 × 2 + 144); only the LM is 8-bit. It is a second bf16-encoder arm, not an independent vision quantization. It still drops to 3 at think-off, so LM quantization alone moves the 9px tier — a tier is not a clean vision-encoder readout in either direction.Pairs with
quant_dims.pyfrom #316: read blob sizes before believing a tag.🤖 Generated with Claude Code