ADR 0037: keep the MLX #3912 fix and promote 0.34 on mlx-metal — campaign, OCRBench, SPEC H15–H18, results

[glennneuber]

Closes the 0.34 fold's Metal kernel question and promotes the build on the mlx-metal surface.

Decision — ADR 0037

Keep the MLX pin at d9add9d1 with ml-explore/mlx#3912. No revert, no compat patch restoring the old fp_qmm_t loop bound. Decided on correctness, not accuracy:

0.34.0-maxusai-8a7ba949 (MLX d9add9d1, fixed)    835/1000 = 0.8350
0.33.2-maxusai-2b95b4a5 (MLX c793734e, defect)   833/1000 = 0.8330
both right 826 | both wrong 158 | fixed only 9 | defect only 7
discordant 16   exact McNemar two-tailed p = 0.8036

OCRBench cannot tell the kernels apart; the golden delta can (fixed = MLX-CUDA's 0.0898 exactly). ADR 0036 already accepted the same 4→3 move on 31b's 9px cell as the cost of the faithful path on GGUF — two unrelated correctness fixes, two engines, same cell, same direction.

SPEC — vision-harness-reuse H15–H18

	rule	enforced?
H15	one capture of a quality cell is one draw — measure a rate	no
H16	both finetext arms count; the renderer shows only the suite arm	no — the renderer works against it
H17	a checkpoint is its manifest digest, not its tag or config digest	partly (quant_dims.py)
H18	cross-build accuracy needs a paired many-item test; a tier move needs the bf16 arm	no

Promotion

0.34.0-maxusai-8a7ba949 has served :11435 since 2026-09-18. README deploy line + the release matrix's mlx-metal row, generated by release_matrix.py from the build's preflight run (PASS 19 / SKIP 12) and spliced beside the CUDA row — whose run was never committed, so regenerating the whole matrix here would have erased it.

Two things found while promoting, recorded rather than fixed:

• Stamps disagree with ADR 0032. Metal builds stamp 0.34.0-maxusai-<sha>, the fold 0.34.1-dynres-0-g<sha>, same commit.
• The Metal launchd env does not set OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0, unlike the CUDA container (ADR 0033, the fold's unbounded-retention finding). Changing it is a persistent config change plus a service restart, left to the operator.

Committed results

Assembled from gitignored captures by script (ADR 0012 rule 8), one record per line:

• bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json — 1000 paired items
• bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json — N=10 × 3 quants, both-arm history
• bench-runs/vision-campaign-2026-09-18-mlx8a7ba949.json — every campaign cell, both arms
• preflight/runs/preflight-mlx-metal-0340-8a7ba949.json — force-added like the 0.33.0 fold's; the invalid --container native run is left out

Retractions (all mine, corrected in the docs they reached)

1. "Twelve runs split cleanly" counted one of two finetext arms — the cell is bimodal.
2. "31b-mxfp8 carries a bf16 vision tower" generalised from one tensor; 162 of 356 vision layers are 8-bit.
3. The 200-item OCRBench slice's 4–0 was noise (5/7 over the rest) and its 87.5% unrepresentative (83.5% over 1000).

🤖 Generated with Claude Code

[glennneuber]

Consolidated into #327, which merges this PR with the other two OCRBench streams (#323 CUDA, #324 mlx-metal, #325 ROCm/GGUF). It contains every commit from all three, so merging #327 closes them together and there is no three-way conflict to resolve later.

What moved, so nothing is lost:

• One renderer. summarize_extbench.py keeps its default table and H13 footer exactly as docs(vision): OCRBench quant ladder for gemma4:31b (q4/q8/bf16), and a paired summarizer #325 wrote them, so pasted tables stay valid; it gains --repeats, --timing and --categories from docs: OCRBench quantisation ladder for gemma4:31b, and a store/registry audit #323's ocrbench_table.py, which is deleted and its tests folded into test_summarizers.py.
• One SPEC numbering. ADR 0037: keep the MLX #3912 fix and promote 0.34 on mlx-metal — campaign, OCRBench, SPEC H15–H18, results #324's H15–H18 stand. docs: OCRBench quantisation ladder for gemma4:31b, and a store/registry audit #323's rule becomes H19 and defers its digest half to H17, whose conformance row now names store_audit.py as the enforcement it recorded as missing.
• One ADR sequence. 0037 stays the kernel decision; manifest-digest identity is 0038, the nvfp4 global-scale representation 0039.
• One entry point. docs/maxusai/ocrbench.md holds the shared method, the format, and all three hosts' numbers with a slice column, because the 1000-item run and the 200-item runs are not comparable.

All three runs agree: no quantisation difference resolves on any host, and OCRBench cannot separate the kernels where the goldens can.

🤖 Generated with Claude Code