docs(vision): SOLVED — the 0.34.0 gemma4 vision shift is upstream MLX #3912

[glennneuber]

The 0.34.0 campaign found exactly one moved quality cell. This records it, the four hypotheses that turned out to be wrong, and the sharper symptom that surfaced while killing the last one.

The finding

gemma4:31b-nvfp4 think-on drops a 9px fine-text OCR tier, 4 → 3, between 0.33.2-maxusai-2b95b4a5 and 0.34.0-maxusai-8a7ba949. 18 runs, three conditions, no overlap:

0.34.0  constrained [3,3,3]   unconstrained [3,3,3]   GREEDY [3,3,3]
0.33.2  constrained [4,4,4]   unconstrained [4,4,4]   GREEDY [4,4,4]

Think-off is unaffected across all five nvfp4 models — every tier, count, field and verdict identical, including under a json schema. Only scene IoU moved, third decimal, upward on four of five.

Four falsified hypotheses

#	hypothesis	killed by
1	#301's max_whitespace_cnt	removing the grammar changes nothing on either build
2	truncated reasoning	0.33.2 scored 4 at its shortest run (1545 chars); 0.34.0 scored 3 at 2423
3	the resize kernel	b10630 already set RESIZE_ALGO_BICUBIC; identical on both sides
4	the sampling path	gap persists unchanged under temperature 0, top_k 1

Each cost a run. They're recorded because they're the four explanations anyone reaches for first — #301 in particular looked compelling, being present on one build and absent on the other and acting only on constrained output.

The better lead

Killing hypothesis 4 surfaced something sharper:

0.33.2 greedy:  eval=1588 think=2046   x3, bit-identical
0.34.0 greedy:  eval=1469 think=1809
                eval=1290 think=1545
                eval=1363 think=1763

Greedy decoding is not reproducible on 0.34.0. That is a better thing to bisect than a one-tier recall difference: it reproduces in three runs without a scored benchmark, and nobody has to argue about whether it matters.

Two fold changes touch memory reuse and are worth testing first — 5844a6cd5 (KV buffer release on a 256-token boundary, upstream ec3cc2307) and the scoped-array-lifetimes refactor. Stated as hypotheses to check, deliberately not as a conclusion to inherit.

Not done

Splitting the fold at 907deffd vs 8a7ba949 halves the search but needs a rebuild — only the two endpoints are archived. The doc says so, and says archiving 907deffd would have been right since it was a distinct Go/MLX pairing.

Reproduction notes

Both pairings are archived binary and payload, so neither needs rebuilding. The doc also records the trap that nearly produced a wrong answer here: serve-apple-mlx.sh kills only its own $BIN, so switching builds leaves the previous one holding the port and the readiness check is answered by the process that was supposed to be replaced — measuring the wrong build under the right label. Free the port by port, then assert /api/version before measuring.

🤖 Generated with Claude Code

[glennneuber]

CUDA-side context for the bisect, from the v0.34.1 fold session — evidence only, no conclusion to inherit.

• The cherry-pick 5844a6cd5 (upstream ec3cc2307) was measured for memory, not numerics. On the CUDA host, with the trace suite on qwen3.6:35b-a3b-nvfp4 (28 requests, drafting under a grammar), resident memory outside the prefix trie grew +5.3 GiB without it and +2.4 GiB with it (the deployed 0.34.0 build: +1.8; with OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0: +0.1, flat). It changes when MLX's free pool is released during a speculative generation — on crossing a 256-token boundary instead of landing on one — and nothing else. Whether an earlier pool release can alter greedy output is exactly the kind of thing your archived endpoint pair can answer in one run; I have no measurement that bears on it.
• MLX think-off cells were already not reproducible across cold loads on the CUDA host before that commit existed — recorded on 2026-09-12 against the 0.34.0 fold (cells move with prefix-cache state and flip across cold loads; ADR 0012 §4 was amended for it). That was at the suite's default sampling, so it is not greedy evidence, but it does place the non-reproducibility on the 0.34.0 side of 5844a6cd5, which argues for starting the bisect at the scoped-lifetimes refactor rather than the cherry-pick.
• GGUF is deterministic on both payloads at n = 4 (gemma4 e2b, e4b, 26b-a4b, 31b: every cell identical across four reps on b10864, and the deployed b10760 image re-run today reproduces its baseline cell for cell), so the effect you see is MLX-side, consistent with your think-off-unaffected finding.
• Your falsified hypothesis 3 has its commit: gemma4's RESIZE_ALGO_BICUBIC entered with llama.cpp #27594 between b10488 and b10630 (the 0.33.1 fold); 0.33.2 and 0.33.3 are identical on all four gemma4 GGUF models, so the kernel is out on that side too.
• One thing to check in the memory column of any repeat: the same fold session found that a grammar-constrained generation that ends by stop while speculative decoding is active retains ~400 MiB per request on 35b-a3b (14/14 on a size ladder; 0/38 for num_predict-terminated requests), so a think-on 9 px run that stops early is also a run that grows.

🤖 Generated with Claude Code

[glennneuber]

Corroboration from the other side of the estate, plus one scope note on hypothesis 3 and a list of what is already refuted so nobody re-runs it.

"Greedy decoding is not reproducible on 0.34.0" reproduces on ROCm/gfx1151

This is the part of your write-up that transfers. I hit the same thing today on a completely different backend and wrote it off as warm/cold sequencing — wrongly, I think, now that I see your numbers.

Two runs of the same 0.34.1 binary, temperature 0, think off, on qwen3.8:27b-q4_K_M, differing only in an env var that provably does not touch numerics (OLLAMA_IGPU_DIRECT_IO):

• invoice extraction 0/5 in one run, 4/5 in the other
• multi-image q1+q2+q4-bbox ❌ in one, ✅ in the other

And the tell that killed my own explanation: stock ollama/ollama:0.34.1-rocm runs direct I/O ON by default, yet tracked the dio-off fork run, not the dio-on one. So the variation is not caused by that variable. It looks like the same non-determinism you measured, on Metal/MLX and on ROCm/gfx1151 — which argues for a shared cause in the fold rather than a per-backend kernel.

Worth noting for anyone reading both threads: this means single-run cells on 0.34.x are weaker evidence than usual. (The regression below is safe on that count — 0.000 vs 0.728, reproduced across separate runs on both stock and fork.)

Scope note on hypothesis 3 (the resize kernel)

Your kill is "b10630 already set RESIZE_ALGO_BICUBIC; identical on both sides" — correct for a 0.33.2 → 0.34.0 comparison. But the BILINEAR → BICUBIC flip happened before b10630, so it sits inside a b9888 → b10864 window and outside yours. Your kill is valid for your bisect and simply does not reach that window; I am noting it only so the row is not read as closing the resize question generally.

For what it is worth, I have now killed it in my window too, by a different route: a compat patch reverting the flip for GEMMA4V/GEMMA4UV and QWEN3VL, verified present in-binary (output hash differs at identical prompt_eval_count and eval_count), does not fix the regression. Note this only refutes the algo selection; b10864 also deleted the naive resize_bilinear/resize_bicubic and routes everything through Pillow-parity resize_pillow, so the implementation change remains untested.

The ROCm-side regression, for cross-reference

Different symptom from yours (think-off, bbox rather than think-on 9px OCR) but the same era. gemma4:31b-it-q4_K_M, budget pinned, name_bbox_mean_iou: 0.728 → 0.000 at the 1120 rung only; 280 and 560 are clean on both builds. Reproduces on stock. prompt_eval_count identical between builds, so the grid is unchanged and only pixel values differ. Details in the comment on #309.

Already refuted here, so you do not have to

• fp16 accumulation in the vision tower — GGML_CUDA_CUBLAS_COMPUTE_TYPE f32 / f16 / unset are all equally broken (env var verified to reach the llama-server process)
• multi-sub-batch image decode — at num_batch 2048 the image decodes in a single decoding image batch 1/1, n_tokens_batch = 2040 and is still broken
• the resize algo selection flip, as above
• direct I/O, GPU layer placement, host drift, and our compat series (stock reproduces it)

On your two suspects

5844a6cd5 / upstream ec3cc2307 and the scoped-array-lifetimes refactor are MLX-side, but ec3cc2307 is an upstream llama.cpp commit — so if the non-determinism has one root, it may be reachable from the GGUF side too. I have not tested that.

🤖 Generated with Claude Code

[glennneuber]

Downstream follow-up: the nondeterminism has a real consequence, and it is not the one I expected.

A document-extraction pipeline running against this build on the same host audited its own dependence on byte-identical re-extraction, from its code rather than from memory. It found none:

• a document's identity for --resume is the source file's sha256, never the output
• the diff tool compares two extractions with a numeric tolerance and is read by a human
• unit tests never touch a live model
• every extracted value is judged by invariants and gates rather than trusted

So no behaviour breaks. But it did find one thing: its retry path deliberately warms the temperature, justified by the comment that "a temperature-0 retry is the same computation and stalls again."

That premise held on 0.33.x and is false on 0.34.0. The behaviour stays defensible — arguably a cold retry now has independent value it did not have before — but the stated reason silently stopped being true underneath it. Those comments have been corrected in that tree, dated and scoped to the build, with the warming kept.

Recording it here because it is the first concrete consequence outside a benchmark, and it is the shape I would expect others to take: not a crash or a wrong answer, but a design rationale that quietly stopped holding. Anywhere in this fork or its consumers that reasons "temperature 0 means the same computation" is now reasoning from a premise that needs re-checking on 0.34.0.

That also slightly raises the priority of the bisect this PR defers. A one-tier OCR difference is easy to live with; an invalidated determinism assumption is the kind of thing that is cheap to fix now and expensive to discover later.

[glennneuber]

Withdrawing the determinism headline from this PR's original body — it was a three-sample artefact. The doc has been corrected in the latest commit; this comment records what changed and why, since the original framing is in the thread above.

What I claimed

That greedy decoding is nondeterministic on 0.34.0 and reproducible on 0.33.2, and that this was a better bisect target than the OCR tier. That rested on three greedy runs of 0.33.2 coming up bit-identical.

What further runs show

Testing OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0 produced an outlier on both sides:

0.34.0 drafting OFF:  1598/2423, 1598/2423, 1591/2046
0.33.2 drafting OFF:  1584/2038, 1588/2046, 1588/2046

The knob is a no-op on 0.33.2 — draftUnderGrammar doesn't exist on that build — so that is the same configuration that gave three identical runs an hour earlier, now giving an outlier.

Pooled across every greedy run:

build / config	runs	identical
0.33.2	6	5× 1588/2046, 1× 1584/2038
0.34.0, drafting off	3	2× 1598/2423, 1× 1591/2046
0.34.0, drafting on	3	0 — three distinct

What actually holds

• MLX greedy has baseline run-to-run spread on both builds. x/mlxrunner/speculate.go already says this — "within the run-to-run spread MLX already has".
• Drafting under a grammar amplifies it substantially — 0-of-3 identical with it on, 2-of-3 with it off. draftUnderGrammar is new in this fold (068a98cd0, adopting upstream's default behind a knob), and that knob is read once at process init, so it's a server-start setting rather than a per-request one.
• 0.34.0 did not introduce nondeterminism. That claim is withdrawn.

What is untouched

The tier drop. 9px is 3 on 0.34.0 and 4 on 0.33.2 in twelve runs each — constrained, unconstrained, greedy, and greedy with drafting disabled. 24 runs, perfect separation, no configuration moves it. That remains the thing to bisect, and it is now the only open item here.

The withdrawn claim is kept in the doc rather than deleted. It is the same small-sample error that produced this repo's retracted throughput regression, made while I was actively citing that retraction as a caution — which seems worth leaving visible rather than tidying away.

[glennneuber]

Progress: the tier drop is partitioned. The vision encoder moved — the image tokens themselves differ between builds.

Prompted by two hypotheses from @glennneuber: measure whether the image tokens differ, and check for a dtype change along the pipeline. The first had a test already written.

Measurement

TestVisionGoldenParity compares EncodeMedia against the vendored mlx-vlm reference and prints the embedding statistics. Run on both builds — in-process, no server, and no rebuild of 0.33.2 needed: a worktree at 2b95b4a5 plus the archived payload dropped into its build/lib/ollama is enough, because go test compiles the Go side and dlopens the MLX library.

model	build	mean	std	norm_mean	max element Δ
31b-nvfp4	0.33.2	−0.00627	1.3209	96.749	0.1406
31b-nvfp4	0.34.0	−0.00641	1.3243	96.998	0.1094
26b-nvfp4	0.33.2	−0.00033	1.6291	86.346	0.2266
26b-nvfp4	0.34.0	−0.00059	1.6292	86.350	0.0508
12b-nvfp4	0.33.2	−0.02562	2.4605	151.310	0.0625
12b-nvfp4	0.34.0	identical in every digit

The encoder is deterministic — 0.34.0's figures reproduced to every digit across two sessions with different server lifecycles. Unlike the generation spread discussed above, these differences are real.

What it narrows to

• 31b and 26b moved; 12b did not. 12b is a built-in control.
• x/models/gemma4/vision.go is byte-identical across the fold — resize, patching and normalization are untouched. So the difference arises in the vision tower's weight loading or matmul, i.e. the quantization path, not preprocessing.
• The fold's gemma4 changes are entirely NVFP4 global-scale handling (ReadGlobalScale, GlobalScales, PrepareGatherQMMGlobalScale), consistent with that localisation.

Two things deliberately not concluded

0.34.0 is closer to the reference, not further — 26b's max element delta improves four-fold. So the encoder became more faithful while the OCR tier got worse. That pairing is unexplained and is not being explained away here.

A predicted mechanism failed. I expected the checkpoints carrying NVFP4 .global_scale tensors to be the ones that moved. They are not: 26b-nvfp4 carries zero global-scale tensors of any spelling (.global_scale, _scale_2, .input_*) and moved; 12b-nvfp4 carries 332 and did not. Global-scale presence does not predict the change. Recorded so nobody inherits it as fact.

Next

Cheapest causal test, no new code: run the think-on finetext arm on 26b (embeddings moved) and 12b (embeddings identical) across both builds. If 12b holds its tier and 26b moves, that ties embedding change to tier change.

Injection — feeding 0.33.2's embeddings to 0.34.0's decoder — is now clearly worth building if that test is ambiguous, since the decoder demonstrably receives different tokens. It needs real plumbing; there is no API path for precomputed embeddings.

[glennneuber]

Follow-up to my earlier comment: the ROCm-side regression is root-caused and fixed, and one piece of it bears on your open lead.

Cause: upstream c7d8722922a (#24233) set info.devices[id].integrated = prop.integrated on HIP builds; upstream #28211 root-causes the result as an MMQ tile-barrier race on gfx115x producing wrong output with no crash once a decode exceeds n_ubatch. Upstream reverted it in #28604 at 2026-09-08T14:19Z — our payload b10864 is 13:01Z the same day, so v0.34.1 has the defect and not the fix. Carried as compat 906; qwen3.8 scene IoU goes 0.065 → 1.000.

Relevant to your "greedy is not reproducible on 0.34.0" lead: that was the one finding of yours that transferred to ROCm, and it now has a possible shared explanation — a race that silently corrupts output past n_ubatch would produce exactly the run-to-run variation you measured at temperature 0, top_k 1. But your backend cannot carry this particular defect, since Metal/MLX does not use ggml-cuda at all.

So the useful experiment is a divergence test, and we will run our half: does 906 make greedy reproducible again on ROCm?

• 906 fixes ours and yours persists → distinct bugs, and your 5844a6cd5 / scoped-lifetimes hypotheses stand on their own rather than being a symptom of something we have now fixed.
• 906 fixes ours and yours also resolves on a payload past d4389a4dd92 → same class, different backend path, worth one shared write-up.
• ours persists too → a third thing, and neither of us should pin it on the fold's memory work yet.

Full detail and the cross-platform asks are in #313. Also recorded there: the things already refuted by experiment here, so nobody re-runs them — fp16 accumulation in the vision tower, multi-sub-batch image decode, the b10864 resize-algo flip, the gemma4 budget bump, and GGML_CUDA_NO_PINNED=1 (which aborts the runner rather than working around anything).

One correction to my earlier comment while I am here: I wrote that the resize-algo flip and the token-budget bump were "the same hunk". They are not — separate upstream PRs twelve days apart, #27594 (2026-08-23) and #28335 (2026-09-04). Neither is the cause.

🤖 Generated with Claude Code

[glennneuber]

Hypothesis 7 (integrated-GPU allocator clamp) falsified — twice. And a withdrawn claim comes back in a narrower, better-supported form.

Prompted by the ROCm result in the sibling investigation: upstream setting prop.integrated on HIP builds caused a real correctness bug there. The v0.34 fold added an integrated-GPU clamp to admit() (client.go:443) that is absent from 0.33.2 entirely and is live on Apple Silicon, lowering the MLX allocator budget to min(GPU free, system free). It looked capable of explaining both the tier and the run-to-run spread.

It fails on two independent grounds

1. The clamp never binds on this host. Pinning OLLAMA_MLX_MEMORY_LIMIT makes the runner log what it derived:

0.34.0  requested=51539607552  derived="107.0 GiB"  using="48.0 GiB"
0.33.2  requested=51539607552  derived="107.0 GiB"  using="48.0 GiB"

Identical derived on both builds. The clamp is inert here, so it cannot differ between them. (My prior reasoning used vm_stat free+inactive ≈ 55.9 GiB as a proxy for systemInfo.FreeMemory; that proxy is wrong.)

2. Equal budgets do not converge the tier. With both pinned to the same 48 GiB: 0.34.0 → 9px [3,3,3], 0.33.2 → 9px [4,4,4].

It could never have explained the encoder anyway

Raised as a question by the consuming-pipeline session before the result landed, and verified: admit() has exactly one caller, (c *Client) Load. TestVisionGoldenParity goes model.Open → mlxthread.Start → r.Load(...) on a &Runner, not a Client. The parity test never passes through the clamp or the override. So the encoder difference has a separate cause regardless — at most this hypothesis could have covered the tier and the spread.

Checking that also surfaced that vision_golden_test.go itself changed across the fold (2 lines: ClaimOSThread() removed from worker init, Sweep() → ClearCache() in the deferred teardown). EncodeMedia is invoked identically and the golden data is unchanged — but what rules the harness out properly is the 12b control: the same harness change applies to all three models, and 12b came back bit-identical while 31b and 26b moved.

Tier status

30 runs, 15 per build, five configurations, no overlap. Constrained, unconstrained, greedy, greedy-without-drafting, and greedy-with-pinned-budget. Nothing moves it.

Revisiting the withdrawn determinism claim

Earlier in this thread I claimed 0.34.0 broke greedy determinism, then withdrew it as a three-sample artefact. With nine greedy runs per build, a narrower version survives:

build	eval range	think_chars range
0.33.2	1584 – 1588	2038 – 2046
0.34.0	1290 – 2105	1545 – 3248

Both builds drift — the withdrawal was correct on that point, and "0.33.2 is deterministic" stays wrong. But 0.34.0's spread is roughly two orders of magnitude wider (4 tokens vs 815). The accurate claim is "both drift; 0.34.0 drifts far more", which the original 3-run samples could not have established in either direction.

[glennneuber]

Bisect step 1: the encoder change is in #302 (the v0.34.1 fold), not v0.34.0.

Switched the bisect target from the OCR tier to the encoder measurement. The tier needs a scored benchmark and sits on a quantity that drifts 815 tokens between runs on 0.34.0; the encoder measurement is deterministic, reproduces to every digit across sessions, runs in seconds with no server, and answers one question per build.

Midpoint: 907deffd (v0.34.0 fold + #300 + #301), pins b10760 / MLX ce916dbb — distinct from both endpoints.

	26b maxΔ	31b norm_mean	12b (control)
2b95b4a5 (0.33.2)	0.2266	96.749	0.0625
907deffd (midpoint)	0.2266	96.749	0.0625
8a7ba949 (0.34.1 fold)	0.0508	96.998	0.0625

The midpoint is bit-identical to 0.33.2 — every digit of mean, std, norm_mean and max delta, on both affected models. 26b and 31b agree, so this is not two changes straddling the midpoint. The 12b control holds across all three builds.

What this eliminates

The entire v0.34.0 upstream fold (#297), plus #300 and #301. The whitespace bound is now exonerated twice independently.

What remains

Only #302's contents:

• MLX ce916dbb → d9add9d1 — 24 upstream commits. MLX is the vision compute backend here, so this is the prime suspect.
• MLX-C c74db530 → ebc88f10
• the scoped-array-lifetimes refactor (x/mlxrunner/mlx/scope.go)
• the NVFP4 global-scale / gather_qmm work
• the ec3cc2307 KV-buffer cherry-pick
• llama.cpp b10760 → b10864 — irrelevant to the MLX path

Method notes

Two outcomes were allowed for and did not occur, flagged in advance by the consuming-pipeline session: a third value at the midpoint (meaning a change on each side, splitting the bisect in two), and 26b and 31b disagreeing about which endpoint the midpoint matches (also two changes). Recording that they were anticipated rather than ruled out after the fact.

The midpoint build is archived, binary and payload, as ollama-0.34.0-maxusai-907deffd / payload-0.34.0-maxusai-907deffd (sha256 8a0440e8072ee8c9…). The archive now holds three complete pairings, so further steps between them are copies rather than cold builds. This is the mistake recorded earlier in this PR — discarding that exact install — fixed inside the same investigation rather than filed for later.

Bearing on #312

This sharpens the MLX-CUDA ask rather than replacing it. The remaining suspects split cleanly: MLX Metal kernels (which CUDA would not reproduce) versus shared Go/MLX-C layers (which it would). One run there still answers in one step what further local bisecting answers in several.

[glennneuber]

Bisect step 2: the tier drop is in #302 as well. All three observables co-localise in one PR.

	26b encoder maxΔ	think-on 9px	greedy eval range
2b95b4a5 (0.33.2)	0.2266	4	1584 – 1588 (4)
907deffd (midpoint)	0.2266	4	1289 – 1290 (1)
8a7ba949 (v0.34.1 fold)	0.0508	3	1290 – 2105 (815)

Midpoint think-on, greedy, 3 reps: 9px = [4,4,4], eval 1290 / 1289 / 1290.

Three independent observables flip at the same commit boundary — the image embeddings, the OCR tier, and the run-to-run spread. All are clean on the midpoint; all move in #302.

This does not prove a single cause. What it does remove is the possibility that these are separate defects scattered across the fold requiring separate hunts. The whole remaining search is one PR's MLX side.

Credit: the midpoint tier run was proposed by the consuming-pipeline session, with both readings stated before the measurement existed — that the tier scoring 4 would co-localise the findings, and scoring 3 would prove them independent and exonerate the encoder for the tier. Until then the two findings were being carried as probably-related on nothing but a shared endpoint pair, which is the same unforced inference behind the retractions earlier in this thread.

A separation route that does NOT work

The MLX pin and the Go changes ship together in #302, so no bisect separates them. Pairing them deliberately — 8a7ba949's Go with ce916dbb's MLX payload — is not possible:

CHECK failed: mlx_stream_thread_local_new_
SIGSEGV during cgo execution
mlx._Cfunc_mlx_install_capture_handler()
mlx.init.1()  at x/mlxrunner/mlx/mlx.go:43

The new Go calls mlx_install_capture_handler at package init and ce916dbb's MLX-C lacks the symbol it needs, so it dies before any test runs. The two halves are hard-coupled. Separating them requires reverting Go changes against the new payload, not swapping payloads — worth knowing before someone spends an afternoon on the cheap-looking approach.

It also means payload/binary skew on this path fails loudly rather than silently, which is a stronger guarantee than mlx_payload_pin was designed to provide.

Remaining suspects (all within #302)

• MLX ce916dbb → d9add9d1 (24 upstream commits) — prime suspect; MLX is the vision compute backend
• MLX-C c74db530 → ebc88f10
• scoped-array-lifetimes refactor
• NVFP4 global-scale / gather_qmm work
• ec3cc2307 KV-buffer cherry-pick

All three pairings are archived binary+payload, so any further step between them is a copy rather than a build.

[glennneuber]

Resolved. The doc is rewritten and renamed; this PR is no longer about an open regression.

Root cause: upstream MLX ollama#3912 — a Metal kernel bug in fp_qmm_t that reads past K when K mod 32 == 16. Our v0.34.1 fold crossed the fix when the MLX pin moved ce916dbb → d9add9d1. Found by @glennneuber reading MLX's own PR history, which this investigation never did.

K=4304 is the gemma4 26b/31b vision tower's mlp.down_proj contraction dim. 12b carries no nvfp4 vision down_proj at all, so it was structurally immune rather than a lucky control.

Confirmed at kernel level, model-free (guard test in #315):

ce916dbb (pre-fix)   K=4304  max 24.0494   232722/294912 elements over 1.0
d9add9d1 (post-fix)  K=4304  max 0.137939  0/294912
                     K=4288  bit-identical on both libraries

This inverts the PR

Everything above is now reframed. The tier drop is a downstream consequence of correcting a kernel, not a defect to undo. The 9px score of 4 was measured on a build corrupting 79% of that matmul's outputs; 3 is what the correct kernel produces.

It also resolves what this thread carried for most of its length as an unexplained paradox — a more faithful encoder scoring worse. That is exactly what a correctness fix looks like from a scored probe that happened to favour the broken output. I flagged it repeatedly as not-to-be-explained-away, and it turned out to be the clue.

Consequence worth acting on: prior gemma4 26b/31b vision numbers were measured through that kernel. Anything compared against those baselines should be re-measured, not trusted.

What the seven falsified hypotheses cost, and what actually worked

The bisect was correct — it localised all three observables to #302, which is where the pin crosses ollama#3912 — but it could never have named the cause, because it treated a 24-commit upstream pin as an opaque blob. Reading the dependency's changelog would have been faster than every measurement in this thread combined.

Both root causes found on this codebase today, this one and the ROCm prop.integrated MMQ race, came from upstream issue and PR history rather than from our own tree.

[glennneuber]

Withdrawing the cross-platform corroboration I offered earlier in this thread.

I reported that ROCm shows the same non-reproducible greedy decoding you measured on 0.34.0, citing qwen3.8 flipping invoice 0/5 ↔ 4/5 and multi-image ❌ ↔ ✅ between runs at temperature 0.

That variance was observed only on unpatched 0.34.1 builds. Those were running with the HIP prop.integrated defect — an MMQ tile-barrier race on gfx115x that silently corrupts output once a decode exceeds n_ubatch (llama.cpp#28211), fixed here by compat 906 (#314).

Measured since, gemma4:31b think-on, REPEATS=3:

• ROCm 0.32.1 / b9888 → tiers 4/4/4/4/2, three times, bit-identical
• ROCm 0.34.1 + 906 → tiers 4/4/4/4/3, three times, bit-identical

No spread on either. So ROCm's non-determinism had a specific, now-removed cause, and it is not evidence that your MLX spread shares a root — particularly since Metal cannot carry that defect at all.

Net effect for you: your 5844a6cd5 / scoped-lifetimes hypotheses stand on their own rather than being corroborated by us, and the baseline spread you are designing around looks MLX-specific rather than universal. I made your lead appear better supported than it was; apologies for the noise.

Full numbers and the think-on 9px comparison are in #312.

🤖 Generated with Claude Code

[glennneuber]

The 9px tier drop is nvfp4 quantization cost, not a ollama#3912 regression — this PR's doc needs amending

Ran all three 31b quantizations on one binary (0.34.0-maxusai-8a7ba949, payload 0.32.2-61-gd9add9d, post-ollama#3912), same window as the campaign (16384/2200 think-off, 16384/8192 think-on), powermode 2.

model	vision tower	LM	think-off	think-on
31b-nvfp4	nvfp4	nvfp4	[4,4,4,3,3]	[4,4,4,3,3]
31b-mxfp8	bf16	mxfp8	[4,4,4,3,2]	[4,4,4,4,3]
31b-mlx-bf16	bf16	bf16	[4,4,4,4,3]	[4,4,4,4,3]

Both nvfp4 rows reproduce the campaign exactly (positive control, eval=263 / 1363).

bf16 is a clean control

x/mlxrunner/model/linear.go:62 dispatches on the presence of .weight_scale; absent it, nn.NewLinear → x.Matmul(w) — no QuantizedMatmul, no fp_qmm_t, no global scale. 31b-mlx-bf16 has zero .weight_scale tensors across 1247 (59.2 GiB).

31b-mxfp8's vision tower is bf16, not mxfp8

Vision L0 down_proj is 9,916,560 bytes in both the mxfp8 and the bf16 checkpoint — byte-identical (1152 × 4304 × 2 + 144 header). Only the language model is 8-bit (119 MB/layer vs bf16's 231 MB). The nvfp4 blob independently re-derives the shape from file size alone: 2,789,452 = 2,479,104 (4-bit packed) + 309,888 (group-16 scales) + 460 header, confirming K = 4304, group = 16.

What it means

The model's true 9px answer is 4 — bf16, no quantization anywhere, says 4 in both think modes.

• nvfp4 on the fixed kernel → 3
• nvfp4 on the buggy kernel (0.33.2) → 4

The bug was masking an nvfp4 quantization cost on this sample. ollama#3912 did not cause a regression; it stopped hiding one. That is consistent with the golden delta, which measures the fused quantized matmul against dequantize-then-matmul — the correct nvfp4 arithmetic — and improves 0.1406 → 0.0898 with the fix, landing exactly on CUDA's 0.0898.

Confound, stated plainly: 31b-mxfp8 carries bf16's exact encoder and still drops to 3 at think-off. So the LM quantization alone can produce a 3, and because nvfp4 quantizes both the tower and the LM, these three models cannot attribute nvfp4's 3 to the vision tower specifically. Isolating that needs an nvfp4-vision + bf16-LM checkpoint, which does not exist in the store.

Doc change needed

The doc in this PR attributes the tier drop to ollama#3912 as a consequence of the fix. That reads as "the fix cost us a tier" and would point the next reader at reverting a kernel that is wrong on 232722/294912 elements at this shape. It should instead say the fix revealed a pre-existing nvfp4 quantization cost, cite the bf16 control, and record the mxfp8 confound.

Withdrawing an earlier claim of mine

I wrote that the attribution held "by elimination of the only other candidate." The candidate set was never two. MLX_VERSION went 37c26e57 → ce916dbb on the fold (fbedf5066) while upstream went 37c26e57 → cbb4059d → d9add9d1, so the two builds sit on different MLX branches, not consecutive commits. ollama#3912 itself survives that — the reproducer is K-selective, clean at K mod 32 = 0 and broken only at 16, which no unrelated commit would produce — but no build-pair attribution does.

Practical

Production :11435 is nvfp4 and scores one tier below the model's capability at 9px. Reverting the kernel is not the lever. 31b-mxfp8 reaches 4 at think-on if that tier matters.

[glennneuber]

Correction: 31b-mxfp8 does not carry a bf16 vision tower

Withdrawing a claim I made in this thread earlier today. I wrote that
gemma4:31b-mxfp8's vision tower is bf16 and byte-identical to
31b-mlx-bf16's, and drew from it that "the language model alone flips the 9px
glyph with the encoder held constant". That inference is withdrawn.

What is true is narrower than what I claimed. The vision mlp.down_proj blob
is byte-identical between the two checkpoints — 9,916,560 bytes, digest
4b80a6f9a957ba8e…. I read that one tensor and generalised to all 356 vision
layers. Diffing the full layer set:

checkpoint	vision down_proj	vision gate/up/q/k/v/o	LM
31b-nvfp4	4-bit	4-bit	4-bit
31b-mxfp8	bf16	8-bit	8-bit
31b-mlx-bf16	bf16	bf16	bf16

162 of 31b-mxfp8's 356 vision layers are 8-bit; only down_proj was left at
bf16. So it is not a bf16-encoder arm, and none of the three checkpoints
isolates the encoder from the language model — each moves both at once.

Surfaced by the MLX-CUDA session's registry audit, which found our local
31b-nvfp4 and the library's current tag share a config digest while differing
in 194 vision layers. Confirmed locally from the manifests: our 31b-nvfp4 and
31b-mlx-bf16 do share config b72c5344… while their towers differ.

What survives, and it is stronger

The reps stand (N=10, think-off, one binary 8a7ba949 / payload d9add9d1):

31b-mlx-bf16   [4,4,4,4,4,4,4,4,4,4]   10/10
31b-mxfp8      [3,3,3,3,3,3,3,3,3,3]    0/10
31b-nvfp4      [4,4,3,3,3,3,3,3,3,3]    2/10

The cell does not order by numerical precision. 31b-mxfp8 carries a bf16
down_proj — the exact tensor ollama#3912 corrupts — plus an 8-bit tower and an 8-bit
LM, and scores 0/10, below the all-4-bit 31b-nvfp4's 2/10. A metric on which
more precision scores worse is not measuring quality and cannot be read as a
verdict on the encoder. That conclusion needed no claim about identical towers.

Also correcting a second thing from the same comment: "twelve runs split cleanly
by build" counted only one of the suite's two finetext arms
(vision_suite.py → scores_<tag>.json, finetext_probe.py →
ft_<tag>.json, independent generations). The scores_ arm holds a 0.34.0
capture scoring 4 with byte-identical provenance to an ft_ capture scoring 3 —
texts differing by one character, RNK against RMK, ground truth RNK. The
cell is bimodal: a single capture reports the minority mode about one time in
eight.

The isolating arm does not exist in our store. It is the one the MLX-CUDA
session is now pulling: the registry's 31b-nvfp4, bf16 tower with a 4-bit LM,
which against our local 4-bit/4-bit copy varies the tower alone.

[glennneuber]

Settled: on real OCR the kernel defect is undetectable

All 1000 OCRBench v1 items (echo840/OCRBench, contains-match, lmms-eval semantics via extbench.py), gemma4:31b-nvfp4 think-off, num_ctx=16384, the same local checkpoint (4-bit tower, 4-bit LM) under both archived binaries so only the build differs:

0.34.0-maxusai-8a7ba949  (MLX d9add9d1, FIXED)   835/1000 = 0.8350
0.33.2-maxusai-2b95b4a5  (MLX c793734e, BROKEN)  833/1000 = 0.8330

both right 826 | both wrong 158 | FIXED only 9 | BROKEN only 7
discordant 16   exact McNemar two-tailed p = 0.8036   churn 16/1000 = 1.6%

Statistically indistinguishable — two items in a thousand. A kernel that corrupts 232722 of 294912 elements at K = 4304, M = 256 in isolation is not detectable in end-to-end OCR accuracy.

The 200-item slice was misleading in both directions

The earlier slice gave FIXED only 4 / BROKEN only 0, reported here as directionally unanimous but underpowered. The direction was noise: across the remaining 800 items the split is 5 fixed / 7 broken.

It was also unrepresentative of the benchmark. Per-chunk accuracy on the fixed build runs 0.875, 0.845, 0.945, 0.845, 0.665 — the last 200 items are much harder. Reading the first 200 as 87.5% and calling it a match for the tech report's 88.3 reasoning-off figure was wrong. The full-benchmark number for this checkpoint is 83.5%; the gap is most plausibly nvfp4 quantization, but 31b-mlx-bf16 has not been run over the 1000, so that is plausibility rather than measurement.

What the decision rests on

Do not revert the kernel — but not for the reason the 9px tier suggested. Reverting would not cost OCR accuracy at this precision; it demonstrably would not. The argument is that the fixed kernel is arithmetically correct and matches MLX-CUDA's golden delta exactly at 0.0898, and that the 9px tier which started this was never measuring the encoder. There is no accuracy argument on either side; there is a correctness argument on one.

Full record, the discordant-item table, and both retractions in #324.