From 924d2dc811e64355805d6342fb9f20a88a8f6101 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Fri, 18 Sep 2026 18:08:07 +1000 Subject: [PATCH 1/5] =?UTF-8?q?docs(vision):=20open=20regression=20?= =?UTF-8?q?=E2=80=94=2031b=20think-on=209px=20on=200.34.0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 0.34.0 campaign found one moved quality cell: gemma4:31b-nvfp4 think-on drops a 9px fine-text OCR tier against 0.33.2. 18 runs across three conditions separate cleanly by build, so it is not sampling luck. Records four falsified hypotheses with the evidence that killed each — #301's whitespace bound, reasoning length, the resize kernel and the sampling path — because they are the four explanations anyone reaches for first and each cost a run. The more useful finding is incidental to the last of those: under forced greedy decoding 0.33.2 reproduces bit-identically three times while 0.34.0 produces three different outputs. Greedy decoding should be reproducible. That is a sharper thing to bisect than a one-tier recall difference, and it is where the next person should start. Think-off is unaffected across all five nvfp4 models, including under a json schema, so the deployed build's headline behaviour is unchanged. Co-Authored-By: Claude Opus 5 --- ...vision-0340-thinkon-9px-open-regression.md | 124 ++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 docs/maxusai/vision-0340-thinkon-9px-open-regression.md diff --git a/docs/maxusai/vision-0340-thinkon-9px-open-regression.md b/docs/maxusai/vision-0340-thinkon-9px-open-regression.md new file mode 100644 index 00000000000..1c3a66c4665 --- /dev/null +++ b/docs/maxusai/vision-0340-thinkon-9px-open-regression.md @@ -0,0 +1,124 @@ +# Open regression: gemma4:31b-nvfp4 think-on loses a 9px OCR tier on 0.34.0 + +**Status: reproducible, cause unidentified.** Four hypotheses are falsified +below; a fifth and better lead — greedy nondeterminism — is recorded at the end +and is where the next person should start. + +Measured 2026-09-18 on 10.8.0.3, powermode 2, benchmark server on `:11436` from +scratch installs (never the deployed binary — `serve-apple-mlx.sh` restarts with +`pkill -f "$BIN serve"` and production's cmdline is `/opt/github/MaxusAI/ollama/ollama serve`). + +## The finding + +`gemma4:31b-nvfp4`, think **on**, `finetext` arm, `num_ctx=16384` +`num_predict=8192`. The 9px exact-match recall tier drops from 4 to 3 between +`0.33.2-maxusai-2b95b4a5` and `0.34.0-maxusai-8a7ba949`. + +Twelve runs, three per build per grammar condition: + +``` +0.34.0-maxusai-8a7ba949 constrained 9px = [3, 3, 3] +0.34.0-maxusai-8a7ba949 unconstrained 9px = [3, 3, 3] +0.33.2-maxusai-2b95b4a5 constrained 9px = [4, 4, 4] +0.33.2-maxusai-2b95b4a5 unconstrained 9px = [4, 4, 4] +``` + +Six more under forced greedy decoding (`temperature 0`, `top_k 1`): + +``` +0.34.0-maxusai-8a7ba949 GREEDY think-on 9px = [3, 3, 3] +0.33.2-maxusai-2b95b4a5 GREEDY think-on 9px = [4, 4, 4] +``` + +18 runs, no overlap. This is not sampling luck. + +## What is NOT affected + +**Think-off.** The full campaign scored all five nvfp4 models think-off on both +builds: every OCR tier, every box/label/colour count, every invoice field and +every multi-image verdict identical, answer-token counts identical. Only scene +IoU moved, in the third decimal, and upward on four of five. The 31b think-off +row held `4,4,4,4,3` on both builds — the same arm that moves under think-on. + +Think-off also runs under `format:"json"` (the default in `client.generate`), so +constrained sampling is exercised there too and found inert. + +## Four falsified hypotheses + +Recorded because they are the four explanations anyone reaches for first, and +each cost a run to kill. + +**1. #301's whitespace bound.** `max_whitespace_cnt` is present on 0.34.0 and +absent on 0.33.2, and acts only on constrained output — so it looked like the +delta. Falsified: removing the grammar entirely changes nothing on either build +(rows 2 and 4 above). #301 is exonerated. + +**2. Reasoning length.** 0.34.0 terminated its reasoning earlier on the first +samples seen (1821 vs 2046 thinking chars), which suggested truncated reasoning +costing fine-tier recall. Falsified by the full set: 0.33.2 produced the +shortest run of all at **1545** thinking chars and still scored 4, while 0.34.0 +thought **2423** and still scored 3. Thinking length does not predict the +outcome across a 1.6x range. + +**3. The resize kernel.** A `b9888..b10864` diff of the gemma4 branch shows +`image_resize_algo` moving to `RESIZE_ALGO_BICUBIC` alongside the token-limit +change, which reads as a simultaneous b10864 change. It is not: b10630 — the +payload behind 0.33.2 — already sets BICUBIC, so the kernel is identical on both +sides. The MLX path has always used CatmullRom (Keys a=-0.5, the bicubic family) +regardless. See `llama/compat/README.md`. + +**4. The sampling path.** Think-off is greedy and unaffected; think-on is +non-greedy and moved, so sampling was the sharpest remaining asymmetry. +Falsified: the gap persists unchanged under forced greedy. + +## The better lead: 0.34.0 is nondeterministic under greedy + +The greedy run surfaced something sharper than the tier it was testing: + +``` +0.33.2 greedy: eval=1588 think=2046 x3, bit-identical +0.34.0 greedy: eval=1469 think=1809 + eval=1290 think=1545 + eval=1363 think=1763 +``` + +At `temperature 0` / `top_k 1` the old build reproduces exactly three times; the +new build produces three different outputs. Greedy decoding should be +reproducible. + +**This is a better thing to bisect than the OCR tier.** It reproduces in three +runs without a scored benchmark, and unlike a one-tier recall difference nobody +has to argue about whether it matters. + +Two changes in the fold touch memory reuse and are therefore worth testing +FIRST — as hypotheses to check, not as a conclusion inherited from this document: +`5844a6cd5` (mlxrunner: release freed KV buffers on crossing a 256-token +boundary, upstream `ec3cc2307`) and the scoped-array-lifetimes refactor that +replaced `Pin`/`Unpin`/`Sweep` with function and held scopes +(`x/mlxrunner/mlx/scope.go`, #302). + +## Next step not taken + +Splitting the fold at `907deffd` (v0.34.0, MLX `ce916dbb`) against `8a7ba949` +(v0.34.1, MLX `d9add9d1`) halves the search. It needs a rebuild: only the two +endpoints are archived, and the `907deffd` install was discarded. Archiving it +would have been the right call, since it was a distinct Go/MLX pairing. + +## Reproducing + +Both pairings are archived, binary AND payload, so neither needs a rebuild: + +``` +~/.ollama/binaries/ollama-0.33.2-maxusai-2b95b4a5 + payload-0.33.2-maxusai-2b95b4a5/ +~/.ollama/binaries/ollama-0.34.0-maxusai-8a7ba949 + payload-0.34.0-maxusai-8a7ba949/ +``` + +Assemble each as a self-contained install (`/ollama` plus `/lib/ollama`); +`libOllamaRoots()` checks `/lib/ollama` first on darwin, so it resolves +its own payload and never consults the fork checkout. + +**Assert the served version before measuring.** `serve-apple-mlx.sh` kills only +its own `$BIN`, so switching builds leaves the previous one holding `:11436`; +the new server fails to bind, the readiness curl is answered by the process that +was supposed to be replaced, and the run measures the wrong build under the right +label. Free the port by PORT, then check `/api/version` matches before running. From b494f087233e2ce11284a6d904e22e818932a632 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Fri, 18 Sep 2026 18:32:07 +1000 Subject: [PATCH 2/5] docs(vision): withdraw the determinism headline, keep the tier MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An earlier revision claimed 0.34.0 is nondeterministic under greedy while 0.33.2 is deterministic, and offered that as the better bisect target. Three further runs refute it: 0.33.2 produced an outlier in the identical configuration where it had given three identical runs an hour earlier, and 0.34.0 with drafting disabled gave two of three identical. Pooled, 0.33.2 is 5-of-6 identical and 0.34.0 is 2-of-3 with drafting off and 0-of-3 with it on. So MLX greedy has baseline spread on both builds and drafting under a grammar amplifies it — which the fork's own comment already said, 'within the run-to-run spread MLX already has'. The claim that the fold introduced nondeterminism was a three-sample artefact. The withdrawn claim is kept in the document rather than deleted. It is the same small-sample error this repository retracted a throughput regression for, and the correction is the part worth reading. The tier drop is untouched by any of it: 9px is 3 on 0.34.0 and 4 on 0.33.2 in twelve runs each across four configurations, 24 runs with perfect separation. That is the finding to bisect. Co-Authored-By: Claude Opus 5 --- ...vision-0340-thinkon-9px-open-regression.md | 62 +++++++++++-------- 1 file changed, 37 insertions(+), 25 deletions(-) diff --git a/docs/maxusai/vision-0340-thinkon-9px-open-regression.md b/docs/maxusai/vision-0340-thinkon-9px-open-regression.md index 1c3a66c4665..3a8e90f385d 100644 --- a/docs/maxusai/vision-0340-thinkon-9px-open-regression.md +++ b/docs/maxusai/vision-0340-thinkon-9px-open-regression.md @@ -71,31 +71,43 @@ regardless. See `llama/compat/README.md`. non-greedy and moved, so sampling was the sharpest remaining asymmetry. Falsified: the gap persists unchanged under forced greedy. -## The better lead: 0.34.0 is nondeterministic under greedy - -The greedy run surfaced something sharper than the tier it was testing: - -``` -0.33.2 greedy: eval=1588 think=2046 x3, bit-identical -0.34.0 greedy: eval=1469 think=1809 - eval=1290 think=1545 - eval=1363 think=1763 -``` - -At `temperature 0` / `top_k 1` the old build reproduces exactly three times; the -new build produces three different outputs. Greedy decoding should be -reproducible. - -**This is a better thing to bisect than the OCR tier.** It reproduces in three -runs without a scored benchmark, and unlike a one-tier recall difference nobody -has to argue about whether it matters. - -Two changes in the fold touch memory reuse and are therefore worth testing -FIRST — as hypotheses to check, not as a conclusion inherited from this document: -`5844a6cd5` (mlxrunner: release freed KV buffers on crossing a 256-token -boundary, upstream `ec3cc2307`) and the scoped-array-lifetimes refactor that -replaced `Pin`/`Unpin`/`Sweep` with function and held scopes -(`x/mlxrunner/mlx/scope.go`, #302). +## Corrected: greedy spread exists on BOTH builds; drafting amplifies it + +An earlier revision of this document claimed 0.34.0 is nondeterministic under +greedy while 0.33.2 is deterministic, and offered that as the better bisect +target. **That claim was a three-sample artefact and is withdrawn.** It is kept +here rather than deleted, because it is the same small-sample error this +repository has made before and the correction is the useful part. + +Pooling every greedy run (`temperature 0`, `top_k 1`, think-on, same prompt): + +| build / config | runs | identical | +|---|---|---| +| `0.33.2-maxusai-2b95b4a5` | 6 | 5x `eval=1588 think=2046`, 1x `1584/2038` | +| `0.34.0-maxusai-8a7ba949`, drafting **off** | 3 | 2x `1598/2423`, 1x `1591/2046` | +| `0.34.0-maxusai-8a7ba949`, drafting **on** | 3 | 0 — three distinct | + +What this supports: + +- **MLX greedy decoding has baseline run-to-run spread on both builds.** The + fork's own comment at `x/mlxrunner/speculate.go` already said so — "within the + run-to-run spread MLX already has". +- **Drafting under a grammar amplifies it substantially**: zero identical runs + out of three with it on, two of three with it off. `draftUnderGrammar` does + not exist on 0.33.2 at all; it arrived with `068a98cd0`, which adopted + upstream's default and put it behind `OLLAMA_MLX_DRAFT_UNDER_GRAMMAR`. That + knob is read once at process init, so it is a server-start setting, not a + per-request one. + +What this does NOT support: that 0.34.0 introduced nondeterminism. The first +three 0.33.2 runs came up identical and a headline was built on them; a later +run in the identical configuration produced an outlier. + +**The tier drop is unaffected by any of this.** 9px is 3 on 0.34.0 and 4 on +0.33.2 in twelve runs each — constrained, unconstrained, greedy, and greedy with +drafting disabled. 24 runs, perfect separation, no configuration moves it. That +is the finding to bisect; the determinism question is a separate and much softer +observation. ## Next step not taken From 76cd1f010970a153fca4c85d796903c495b9f1b7 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Fri, 18 Sep 2026 20:29:50 +1000 Subject: [PATCH 3/5] docs(vision): bisect localises all three observables to #302 Bisected on the encoder rather than the tier: the encoder measurement is deterministic and answers in seconds, while the tier needs a scored benchmark on a quantity that drifts hundreds of tokens between runs. Midpoint 907deffd (v0.34.0 fold + #300 + #301, pins b10760 / MLX ce916dbb) is bit-identical to 0.33.2 on the encoder for both affected models, scores 4 on the think-on 9px tier, and has a greedy eval range of one token. So the image embeddings, the OCR tier and the run-to-run spread all flip at the same commit boundary, and all of them are in #302. That eliminates the whole v0.34.0 upstream fold, #300 and #301. It does not prove one cause; it rules out the findings being scattered and needing separate hunts. Also records a separation route that does NOT work: pairing 8a7ba949's Go with ce916dbb's MLX payload segfaults at package init, because the new Go calls mlx_install_capture_handler and the old MLX-C lacks the symbol. The halves are hard-coupled and cannot be split by swapping payloads. Co-Authored-By: Claude Opus 5 --- ...vision-0340-thinkon-9px-open-regression.md | 60 +++++++++++++++++-- 1 file changed, 55 insertions(+), 5 deletions(-) diff --git a/docs/maxusai/vision-0340-thinkon-9px-open-regression.md b/docs/maxusai/vision-0340-thinkon-9px-open-regression.md index 3a8e90f385d..c771c22cf15 100644 --- a/docs/maxusai/vision-0340-thinkon-9px-open-regression.md +++ b/docs/maxusai/vision-0340-thinkon-9px-open-regression.md @@ -1,6 +1,6 @@ # Open regression: gemma4:31b-nvfp4 think-on loses a 9px OCR tier on 0.34.0 -**Status: reproducible, cause unidentified.** Four hypotheses are falsified +**Status: reproducible, LOCALISED TO #302 by bisect.** Four hypotheses are falsified below; a fifth and better lead — greedy nondeterminism — is recorded at the end and is where the next person should start. @@ -109,12 +109,62 @@ drafting disabled. 24 runs, perfect separation, no configuration moves it. That is the finding to bisect; the determinism question is a separate and much softer observation. +## Bisect: everything is in #302 + +Bisected on the ENCODER rather than the tier. The tier needs a scored benchmark +and sits on a quantity that drifts hundreds of tokens between runs; the encoder +measurement is deterministic and answers one question per build in seconds. + +Midpoint `907deffd` — the v0.34.0 fold plus #300 and #301, pins `b10760` / +MLX `ce916dbb`, distinct from both endpoints: + +| | 26b encoder maxΔ | think-on 9px | greedy eval range | +|---|---|---|---| +| `2b95b4a5` (0.33.2) | 0.2266 | 4 | 1584-1588 (4) | +| `907deffd` (midpoint) | **0.2266** | **4** | 1289-1290 (**1**) | +| `8a7ba949` (v0.34.1 fold) | 0.0508 | 3 | 1290-2105 (**815**) | + +**Three independent observables flip at the same commit boundary** — image +embeddings, OCR tier, and run-to-run spread. All clean on the midpoint, all +moving in #302. 26b and 31b agree, and the 12b control is bit-identical on all +three builds. + +That eliminates the entire v0.34.0 upstream fold, #300, and #301. It does not +prove one cause, but it rules out the findings being scattered across the fold +and needing separate hunts. + +**Remaining suspects, all inside #302:** MLX `ce916dbb -> d9add9d1` (24 upstream +commits, prime suspect since MLX is the vision compute backend), MLX-C +`c74db530 -> ebc88f10`, the scoped-array-lifetimes refactor, the NVFP4 +global-scale / `gather_qmm` work, and the `ec3cc2307` KV-buffer cherry-pick. + +## A separation route that does not work + +The MLX pin and the Go changes ship together, so no bisect separates them. +Pairing them deliberately — `8a7ba949`'s Go with `ce916dbb`'s MLX payload — +IS NOT POSSIBLE: + +``` +CHECK failed: mlx_stream_thread_local_new_ +SIGSEGV during cgo execution +mlx._Cfunc_mlx_install_capture_handler() +mlx.init.1() at x/mlxrunner/mlx/mlx.go:43 +``` + +The new Go calls `mlx_install_capture_handler` at package init and `ce916dbb`'s +MLX-C lacks the symbol it needs, so it dies before any test runs. Separating the +halves requires reverting Go changes against the new payload, not swapping +payloads. Worth knowing before spending time on the cheap-looking approach. + +It also means payload/binary skew fails LOUDLY on this path rather than +silently, which is stronger than what `mlx_payload_pin` was designed to catch. + ## Next step not taken -Splitting the fold at `907deffd` (v0.34.0, MLX `ce916dbb`) against `8a7ba949` -(v0.34.1, MLX `d9add9d1`) halves the search. It needs a rebuild: only the two -endpoints are archived, and the `907deffd` install was discarded. Archiving it -would have been the right call, since it was a distinct Go/MLX pairing. +Splitting #302 further, or the MLX-CUDA comparison in #312 — which is now the +cheapest remaining cut, because the surviving suspects divide into MLX Metal +kernels (which CUDA would not reproduce) and shared Go/MLX-C layers (which it +would). ## Reproducing From 8e7543835004743f733f125f756cb170a47b00c5 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Fri, 18 Sep 2026 20:44:12 +1000 Subject: [PATCH 4/5] =?UTF-8?q?docs(vision):=20SOLVED=20=E2=80=94=20upstre?= =?UTF-8?q?am=20MLX=20#3912,=20fp=5Fqmm=5Ft=20past-K=20read?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause found by reading MLX's own upstream PR history, which this investigation never did: #3912 fixed a Metal kernel bug in fp_qmm_t that reads past the K dimension when K mod 32 == 16. The v0.34.1 fold crossed that fix when the MLX pin moved ce916dbb -> d9add9d1. K = 4304 is the gemma4 26b/31b vision tower's mlp.down_proj contraction dimension, stored packed [1152, 538] at 8 values per word and cross-checked by the group-16 scale tensor at [1152, 269]. 12b carries no nvfp4 vision down_proj at all, so it was structurally immune rather than a lucky control. Confirmed at kernel level (guard test in #315): on ce916dbb, K=4304 puts 232722 of 294912 output elements more than 1.0 away from dequantize+matmul, max error 24.05; on d9add9d1 it is zero and 0.138. The K=4288 control is bit-identical across both libraries. This inverts the document. The tier drop is a downstream consequence of CORRECTING a kernel, not a defect to undo: the 9px score of 4 was measured on a build corrupting most of that matmul, and 3 is what the correct kernel produces. It also resolves what the document carried as an unexplained paradox — a more faithful encoder scoring worse — which is exactly what a correctness fix looks like from a probe that happened to favour the broken output. Prior gemma4 26b/31b vision baselines were measured through that kernel and should be re-measured rather than compared against. Renamed: the file is no longer about an open regression. Co-Authored-By: Claude Opus 5 --- ...=> vision-0340-mlx3912-fp-qmm-t-kmod32.md} | 62 ++++++++++++++++--- 1 file changed, 55 insertions(+), 7 deletions(-) rename docs/maxusai/{vision-0340-thinkon-9px-open-regression.md => vision-0340-mlx3912-fp-qmm-t-kmod32.md} (75%) diff --git a/docs/maxusai/vision-0340-thinkon-9px-open-regression.md b/docs/maxusai/vision-0340-mlx3912-fp-qmm-t-kmod32.md similarity index 75% rename from docs/maxusai/vision-0340-thinkon-9px-open-regression.md rename to docs/maxusai/vision-0340-mlx3912-fp-qmm-t-kmod32.md index c771c22cf15..5e8aef55651 100644 --- a/docs/maxusai/vision-0340-thinkon-9px-open-regression.md +++ b/docs/maxusai/vision-0340-mlx3912-fp-qmm-t-kmod32.md @@ -1,12 +1,60 @@ -# Open regression: gemma4:31b-nvfp4 think-on loses a 9px OCR tier on 0.34.0 +# SOLVED: the gemma4 26b/31b vision shift on 0.34.0 is upstream MLX #3912 -**Status: reproducible, LOCALISED TO #302 by bisect.** Four hypotheses are falsified -below; a fifth and better lead — greedy nondeterminism — is recorded at the end -and is where the next person should start. +**Status: ROOT CAUSE FOUND AND CONFIRMED. This is not a regression — it is a +kernel correctness FIX, and 0.34.0 is the more correct build.** -Measured 2026-09-18 on 10.8.0.3, powermode 2, benchmark server on `:11436` from -scratch installs (never the deployed binary — `serve-apple-mlx.sh` restarts with -`pkill -f "$BIN serve"` and production's cmdline is `/opt/github/MaxusAI/ollama/ollama serve`). +## Root cause + +Upstream **MLX PR #3912** fixed a Metal kernel bug in `fp_qmm_t` that reads past +the K dimension when **`K mod 32 == 16`**. Our v0.34.1 fold (#302) crossed that +fix when the MLX pin moved `ce916dbb -> d9add9d1`. + +`K = 4304` is the vision tower's `mlp.down_proj` contraction dimension in the +gemma4 **26b and 31b** nvfp4 checkpoints — stored packed as `[1152, 538]` at 8 +values per word, cross-checked by the group-16 scale tensor at `[1152, 269]`. +`4304 = 134*32 + 16`. + +**12b carries no nvfp4 vision `down_proj` at all** (its text ones are K = 15360 +and 65536, both `mod 32 == 0`). It was never a lucky control — it is +structurally immune, which is why it came back bit-identical on every build. + +Confirmed at kernel level, independent of models or scores (guard test in #315): + +``` +MLX ce916dbb (pre-fix) K=4288 max 0.000152588 0/294912 elements over 1.0 + K=4304 max 24.0494 232722/294912 <- 79% corrupted +MLX d9add9d1 (post-fix) K=4288 max 0.000152588 0/294912 + K=4304 max 0.137939 0/294912 +``` + +The aligned control is bit-identical across both libraries, so the two differ +only on the one path. + +### What this means for the numbers below + +The tier drop is **a downstream consequence of correcting a kernel**, not a +defect to undo. The 9px score of 4 was measured on a build corrupting most of +that matmul's outputs; the score of 3 is what the correct kernel produces. + +It also resolves what this document previously recorded as an unexplained +paradox — that 0.34.0 was *closer* to the mlx-vlm reference while scoring worse. +That is exactly what a correctness fix looks like from a scored probe that +happened to favour the broken output. + +**Prior gemma4 26b/31b vision numbers measured before this fold went through a +kernel corrupting 79% of the vision tower's down_proj outputs.** Anything +compared against those baselines should be re-measured rather than trusted. + +### Two things that made this expensive, worth carrying forward + +**The symptom was almost invisible.** Not a crash, not an obviously wrong +answer: one fine-text OCR tier, one model, one think mode — and the model stayed +coherent throughout. + +**Nobody read upstream's PR history.** The whole investigation below bisected our +own fold and treated the MLX pin as an opaque 24-commit blob. The answer was in +MLX's changelog the entire time. The sibling ROCm investigation reached its own +root cause the same way, on the same day. ## The finding From de31523c54c64d65a2f1f3e9bced484303a506b0 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Fri, 18 Sep 2026 21:44:31 +1000 Subject: [PATCH 5/5] docs(vision): the 9px tier drop is nvfp4 quantization cost the kernel bug masked MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The doc attributed the tier drop to #3912 as a consequence of the fix, and said "the score of 3 is what the correct kernel produces". The unquantized control refutes that: gemma4:31b-mlx-bf16, which never enters QuantizedMatmul at all, scores 9px = 4 in both think modes. So 4 is the model's answer, nvfp4 on the fixed kernel costs a tier, and the broken kernel happened to compensate for that loss on this sample. The fix did not cost a tier, it stopped hiding one. Records the full control table so these arms are never re-run: all three 31b quantizations on one binary and window, the 31b tier history across four builds, and 26b/12b unchanged throughout. The suite gitignores ft_*.json, so the doc is their durable form. Two things found while controlling: - gemma4:31b-mxfp8 carries a bf16 vision tower, not an mxfp8 one — its vision down_proj blob is byte-identical in size to bf16's (9,916,560 = 1152*4304*2 + 144). It is a second bf16-encoder arm, not an independent vision quant. It still drops to 3 at think-off, so LM quantization alone moves the tier and these checkpoints cannot pin nvfp4's 3 to the vision tower. - MLX_VERSION never moved ce916dbb -> d9add9d1 along one line: fbedf5066 took the fold 37c26e57 -> ce916dbb while upstream went 37c26e57 -> cbb4059d -> d9add9d1. The two builds sit on different MLX branches. #3912 survives that because the reproducer is selected by K, not by build pairing; no purely build-pair attribution does. Co-Authored-By: Claude Opus 5 --- .../vision-0340-mlx3912-fp-qmm-t-kmod32.md | 105 +++++++++++++++++- 1 file changed, 99 insertions(+), 6 deletions(-) diff --git a/docs/maxusai/vision-0340-mlx3912-fp-qmm-t-kmod32.md b/docs/maxusai/vision-0340-mlx3912-fp-qmm-t-kmod32.md index 5e8aef55651..61bdf14f67a 100644 --- a/docs/maxusai/vision-0340-mlx3912-fp-qmm-t-kmod32.md +++ b/docs/maxusai/vision-0340-mlx3912-fp-qmm-t-kmod32.md @@ -32,19 +32,112 @@ only on the one path. ### What this means for the numbers below -The tier drop is **a downstream consequence of correcting a kernel**, not a -defect to undo. The 9px score of 4 was measured on a build corrupting most of -that matmul's outputs; the score of 3 is what the correct kernel produces. +The tier drop is **an nvfp4 quantization cost that the broken kernel was +masking**, not a defect the fix introduced and not a defect to undo. The 9px +score of 4 was measured on a build corrupting most of that matmul's outputs. +The score of 3 is what nvfp4 costs on this sample once the arithmetic is +correct — see "The quantization control" below, where the unquantized bf16 +checkpoint scores 4. It also resolves what this document previously recorded as an unexplained paradox — that 0.34.0 was *closer* to the mlx-vlm reference while scoring worse. -That is exactly what a correctness fix looks like from a scored probe that -happened to favour the broken output. +That is what a correctness fix looks like when the error it removes happened +to compensate for a quantization loss on the scored sample. **Prior gemma4 26b/31b vision numbers measured before this fold went through a kernel corrupting 79% of the vision tower's down_proj outputs.** Anything compared against those baselines should be re-measured rather than trusted. +## The quantization control (2026-09-18) + +**Read this table before re-measuring anything here.** Every cell below is +recorded so these arms do not have to be run again. + +All three `gemma4:31b` quantizations, one binary `0.34.0-maxusai-8a7ba949`, +payload `0.32.2-61-gd9add9d` (post-#3912), `num_ctx=16384`, `num_predict=2200` +think-off / `8192` think-on, powermode 2, scored by `finetext_probe.py`: + +| model | vision tower | LM | think-off | think-on | +|---|---|---|---|---| +| `31b-nvfp4` | nvfp4 | nvfp4 | `[4,4,4,3,3]` | `[4,4,4,3,3]` | +| `31b-mxfp8` | **bf16** | mxfp8 | `[4,4,4,3,2]` | `[4,4,4,4,3]` | +| `31b-mlx-bf16` | **bf16** | bf16 | `[4,4,4,4,3]` | `[4,4,4,4,3]` | + +Tiers are `[22px, 16px, 12px, 9px, 7px]`. Both nvfp4 rows reproduce the +campaign cells exactly (positive control: `eval=263` think-off, tiers to the +digit), so the harness measures the same quantity today as during the campaign. + +**bf16 is a clean control by construction.** `x/mlxrunner/model/linear.go` +dispatches on the presence of a `.weight_scale` sibling; absent it the factory +returns `nn.NewLinear`, whose `Forward` is `x.Matmul(w)` — no `QuantizedMatmul`, +no `fp_qmm_t`, no global scale. `31b-mlx-bf16` carries **zero** `.weight_scale` +tensors across 1247 (59.2 GiB). + +**`31b-mxfp8`'s vision tower is bf16, not mxfp8.** Its vision L0 +`mlp.down_proj` blob is **9,916,560 bytes in both the mxfp8 and the bf16 +checkpoint** — byte-identical, and exactly `1152 × 4304 × 2 + 144` header. Only +the language model is 8-bit (119 MB/layer against bf16's 231 MB). So mxfp8 is a +second bf16-encoder arm, not an independent vision quantization. + +The same arithmetic re-derives this document's shape claim from file sizes +alone, with no code and no model load: the nvfp4 blob is +`2,789,452 = 2,479,104 (4-bit packed) + 309,888 (group-16 scales) + 460`, +giving K = 4304 and group = 16. + +### What the control establishes + +**The model's true 9px answer is 4** — bf16, with no quantization anywhere, +scores 4 in both think modes. nvfp4 on the fixed kernel scores 3; nvfp4 on the +broken kernel scored 4. The kernel defect was compensating for an nvfp4 +quantization loss on this sample, so #3912 did not cost a tier, it stopped +hiding one. + +That is consistent with the golden-parity delta, which measures the fused +quantized matmul against dequantize-then-matmul — the arithmetic the nvfp4 +weights actually encode — and improves `0.1406 → 0.0898` with the fix, landing +exactly on MLX-CUDA's `0.0898` (#316). + +### The confound, stated plainly + +`31b-mxfp8` carries bf16's exact encoder and still drops to 3 at think-off. So +**language-model quantization alone moves the 9px tier**, and because nvfp4 +quantizes both the tower and the LM, these three checkpoints cannot pin nvfp4's +3 to the vision tower specifically. Isolating that needs an nvfp4-vision + +bf16-LM checkpoint, which the store does not contain. + +The general point outlives this investigation: a recall tier is not a clean +readout of vision-encoder correctness, and a tier move across a build bump is +not evidence about a kernel until the unquantized arm has been measured on the +same binary and window. + +### Tier history, so it is not re-measured + +`gemma4:31b-nvfp4`, all builds, both think modes — transcribed from the +`ft_*.json` capture files by script (the suite gitignores them, so this table is +their durable form): + +| build | MLX pin | think-off | think-on | runs | +|---|---|---|---|---| +| `0.33.0-maxusai-21cfe88e` | pre-fold | `[4,4,4,4,3]` | `[4,4,4,4,3]` | 2 | +| `0.33.2-maxusai-2b95b4a5` | `c793734e` | `[4,4,4,4,3]` | `[4,4,4,4,3]` | 5 | +| `0.34.0-maxusai-907deffd` | `ce916dbb` | — | `[4,4,4,4,3]` | 3 | +| `0.34.0-maxusai-8a7ba949` | `d9add9d1` | `[4,4,4,3,3]` | `[4,4,4,3,3]` | 5 | + +`26b-nvfp4` and `12b-nvfp4` are **unchanged across all three builds** +(`[4,4,4,3,3]` and `[4,4,3,0,0]` think-off) — the 26b's encoder moved from the +same fix while its tier did not, which is the first sign the tier and the +encoder are not the same measurement. + +### One caveat on build-pair attribution + +`MLX_VERSION` did not move `ce916dbb → d9add9d1` along one line. `fbedf5066` +moved the fold `37c26e57 → ce916dbb`, while upstream went +`37c26e57 → cbb4059d → d9add9d1` separately — so the two builds sit on +**different MLX branches**, not consecutive commits. #3912 itself survives that, +because the kernel reproducer is selected by K (clean at `K mod 32 == 0`, broken +only at 16) rather than by build pairing. No purely build-pair attribution +does. + ### Two things that made this expensive, worth carrying forward **The symptom was almost invisible.** Not a crash, not an obviously wrong @@ -52,7 +145,7 @@ answer: one fine-text OCR tier, one model, one think mode — and the model stay coherent throughout. **Nobody read upstream's PR history.** The whole investigation below bisected our -own fold and treated the MLX pin as an opaque 24-commit blob. The answer was in +own fold and treated the MLX pin as an opaque range. The answer was in MLX's changelog the entire time. The sibling ROCm investigation reached its own root cause the same way, on the same day.