docs: OCRBench quantisation ladder for gemma4:31b, and a store/registry audit

[glennneuber]

The maintainer asked for the Metal side's OCRBench slice to be repeated on mlx-cuda and on the llama.cpp GGUF path across the gemma4:31b quantisation ladder. This is the record, with the Metal section left empty for that session to fill, plus the two decisions the measurement forced.

Results

mlx-cuda (each arm run twice, zero items flipping in any run)

arm	accuracy	s/item
nvfp4 language model, nvfp4 tower (this store)	0.860	2.3
nvfp4 language model, bf16 tower (library today)	0.850	1.4
mxfp8 language model, bf16 tower	0.845	1.3
bf16	not run — refused admission, 75.0 GiB asked against 62.7 GiB available with the 16 GiB reserve

llama.cpp GGUF (each arm run twice, zero flips)

arm	accuracy	s/item
q4_K_M	0.855	5.0
q8_0	0.850	5.1
bf16	0.855	5.0

No pair resolves under an exact McNemar test on the shared items. The ladder is flat from 4 bits to bf16 on both engines for this slice. What does move is the clock: quantising the vision tower costs 39 % of the time per image — the two nvfp4 rows share a language model and differ only in the tower, 1.4 s against 2.3 s on identical images at the same 1115-token prefill. The bf16 tower is 3.5× the bytes, so this is kernel throughput, not bandwidth: the encode is compute-bound and MLX's quantised matmul on sm_120 runs at about half the dense rate, as the qqmm bench measured directly.

Two things the measurement forced

A published tag is not a fixed artifact. gemma4:31b-nvfp4 here and in the library differ in 194 of 1248 layers, all on the vision path, with the config blob unchanged so ollama show says nothing. 26b-nvfp4, 12b-nvfp4 and qwen3.5:0.8b-mlx moved too. Every gemma4 vision result in this repo was taken against the quantised tower, and the old artifact cannot be fetched back. ADR 0037 (proposed) makes the manifest digest the identity in any record, sends benchmark pulls to a separate store, and requires the audit on both hosts before a cross-host comparison. SPEC vision-harness-reuse H15 carries the reporting half, including that a LIMIT/OFFSET window is a stratum: rows 0–200 are four of OCRBench's ten categories, and the survey's claim that such a slice is comparable to a model card is corrected in place.

ADR 0038 (proposed) settles the nvfp4 global-scale representation that #312 turned up and that #287 runs through: store the checkpoint multiplier and convert to MLX's m × 2688 form only where MLX consumes it. The stored-MLX-form round trip misses m by one ulp for 17 of 31b's 191 vision scales, which is the entire CUDA-side encoder move across the fold, and all three wrappers that apply the scale themselves pay it. #287's prefill dequantisation inherits the fix; it stays held on its other grounds.

Tools (with tests)

• store_audit.py — diffs any store against the registry: what moved, which tensor group, the size ratio (≈3.55× reads as nvfp4 → bf16). --digests prints the digest to cite. Runs on the Metal host too.
• ocrbench_table.py — renders the ladder from extbench.py score files with repeats, exact McNemar pairs, and --categories for the question-type split.
• extbench.py now caches the row slice and retries fetches: two arms died when a DNS blip made it re-ask the datasets server for a fixed set of rows at the start of every arm.

34 tests across the three.

For the Metal session

The empty section names the four steps. The one that matters first: run store_audit.py gemma4 on that host. The CUDA 0.860 came from a 4-bit tower (digest 637cc0ff1570…, pulled 2026-08-17), so the Metal 0.875 sits on the same row only if that host also holds a 4-bit tower; if its store matches the registry, the number belongs on the bf16-tower row, where CUDA scored 0.850.

🤖 Generated with Claude Code

[glennneuber]

ROCm/gfx1151 arms for the GGUF ladder — and one correction to the "What is measured" table

I ran the same three GGUF arms on the ROCm host, one run each, before finding this PR. Overlapping work; consolidating here rather than shipping a second renderer. #325 should be folded into this PR — see the end.

The numbers

arm	host	correct / 200	accuracy	median s/item	prompt_eval
q4_K_M	gfx1151	171	0.855	7.7	1113
q8_0	gfx1151	169	0.845	8.1	1113
bf16	gfx1151	169	0.845	8.1	1113

Paired, exact McNemar: q8_0 vs bf16 0 discordant items out of 200 (p = 1.000); q4_K_M vs each of the other two is 3-vs-1 (p = 0.625). Same conclusion as yours — the ladder is flat and nothing resolves — reached independently on different silicon.

num_batch on gfx1151, and why the config table's engine row does not hold here

| engine | the deployed build 0.34.1-dynres-16-g16649e8 | GGUF arms therefore decode the image in one batch (ADR 0036) |

That inference holds on your CUDA card and not on this one. ADR 0036's own diagnostic fires on every gemma4 load here:

msg="generation batch below the image chunk, images decode in pieces" num_batch=1024 image_chunk_batch=2048

The 2048 floor is requested and denied at every quantization — 1024 for q4 at -np 1, 512 for q8 and bf16, 512 for everything at -np 2. Not for want of memory. availableMemoryForLoad takes the iGPU branch:

if systemInfo.FreeMemory > 0 && sharedGPUFree > 0 && systemInfo.FreeMemory < sharedGPUFree {
    return discreteGPUFree + systemInfo.FreeMemory, gpuFree, true
}

so the scheduler sizes the batch against 31 GiB of system RAM while logging available="95.4 GiB" of GPU. On a discrete card discreteGPUFree carries the whole figure and the branch never triggers, which is why your arms do get 2048. The premise in that branch's comment — iGPU free memory is "a static or slowly refreshed device baseline" — does not hold on a Strix Halo carve-out.

Suggest the engine row becomes per-host, or cites the logged num_batch rather than inferring it from the ADR. I have amended ADR 0036 with the measurement.

The useful part: your arms decoded the image chunk in one piece and mine decoded it split, and the ladders agree to within 0–2 items. That is a second, independent confirmation of ADR 0036's "the split costs no scored cell" — this time with the split-vs-single contrast between the two platforms rather than within one.

Config deltas before anyone treats the two ladders as one table

Same: 200 rows at offset 0, think off, /api/generate, apply_sampling=False + temperature 0 (extbench hardcodes both), commit 16649e8.

Different: num_ctx 16384 here against your 8192; ROCm/gfx1151 against CUDA/sm_120; split image chunk against one batch; one run per arm here against your two. The 1-to-2 item gaps between our q8/bf16 rows sit inside every one of those, and I would not attribute them.

Per ADR 0037 I owe you manifest digests for the three GGUF tags before these rows are cited — I will run store_audit.py here once this lands and post them.

Consolidation

Your ocrbench_table.py is strictly better than the summarize_extbench.py I wrote for #325 (repeats, categories, s.e., same exact McNemar), and SPEC H1's "a second one is the defect" applies to renderers. I will drop mine and rebase #325 onto this branch, keeping only what does not duplicate:

• extbench.py H11 provenance — client.generate() already stamps _host / _server_version on every response and extbench was discarding both, so no ext_*.json in either of our PRs can say which build produced it. My change persists them as sets, so a mid-run container restart cannot let one build vouch for another's rows. Complements your row-cache and retry changes; no overlap.
• An H13 provenance footer for ocrbench_table.py, ported from mine: a file with no host/server_version renders pre-H11 run (not recorded) instead of inheriting a sibling's. This is exactly the defect H13 was written for, reached through a different renderer.
• The ROCm section above, in your doc's format, plus the num_batch finding.

Say if you would rather I open the rebase as a separate stacked PR than push to this branch.

🤖 Generated with Claude Code

[glennneuber]

Consolidated into #327, which merges this PR with the other two OCRBench streams (#323 CUDA, #324 mlx-metal, #325 ROCm/GGUF). It contains every commit from all three, so merging #327 closes them together and there is no three-way conflict to resolve later.

What moved, so nothing is lost:

• One renderer. summarize_extbench.py keeps its default table and H13 footer exactly as docs(vision): OCRBench quant ladder for gemma4:31b (q4/q8/bf16), and a paired summarizer #325 wrote them, so pasted tables stay valid; it gains --repeats, --timing and --categories from docs: OCRBench quantisation ladder for gemma4:31b, and a store/registry audit #323's ocrbench_table.py, which is deleted and its tests folded into test_summarizers.py.
• One SPEC numbering. ADR 0037: keep the MLX #3912 fix and promote 0.34 on mlx-metal — campaign, OCRBench, SPEC H15–H18, results #324's H15–H18 stand. docs: OCRBench quantisation ladder for gemma4:31b, and a store/registry audit #323's rule becomes H19 and defers its digest half to H17, whose conformance row now names store_audit.py as the enforcement it recorded as missing.
• One ADR sequence. 0037 stays the kernel decision; manifest-digest identity is 0038, the nvfp4 global-scale representation 0039.
• One entry point. docs/maxusai/ocrbench.md holds the shared method, the format, and all three hosts' numbers with a slice column, because the 1000-item run and the 200-item runs are not comparable.

All three runs agree: no quantisation difference resolves on any host, and OCRBench cannot separate the kernels where the goldens can.

🤖 Generated with Claude Code

[glennneuber]

Pushed one mechanical commit to this branch: renumbered this branch's ADRs 0037/0038 → 0038/0039.

main took 0037 for "keep the MLX ollama#3912 kernel fix" when #324 merged, so this branch's 0037 and main's are two different decisions under one number. The merge still reports CLEAN because the filenames differ — git has no reason to object, and the collision would only surface when someone cites "ADR 0037" and gets the wrong document.

Two git mv plus the six references in README.md, ocrbench-quantisation-ladder.md and spec/vision-harness-reuse.md. The patterns are anchored to ADR 003x and the full filenames — a bare 0037 also matches the 1.003776 JPEG constant in synthetic-triggers/README.md, which is untouched. Tests still pass.

Revert the commit if you would rather renumber the other way.

[glennneuber]

Consolidating the Metal side — and one collision that git will not flag

Metal session here. Taking this PR as the consolidation point for the OCRBench test and results format rather than opening a parallel one.

1. #324 took ADR 0037 and SPEC H15 while this was open

docs/maxusai/adr/0037-keep-the-mlx-3912-kernel-fix.md and SPEC vision-harness-reuse H15–H18 are on main now. This branch adds adr/0037-a-model-is-identified-by-its-manifest-digest.md and its own H15. The filenames differ and the edits sit in different regions, so git raises no conflict — merging as it stands leaves main with two ADR 0037s and two H15s, silently. That's why this is the first item.

Fixed in the stacked PR below, as its own commit so you can drop it if you'd rather renumber differently:

• ADR 0037 (manifest digest) → ADR 0039
• SPEC H15 → H19
• ADR 0038 keeps its number — main doesn't use 0038, and main's ADR 0037 plus the mlx: opt-in dense GEMM for nvfp4 prefill (dequantise to bf16 above a row threshold) #287 park comment already cite your global-scale ADR as 0038.

Every reference moved with them (count-asserted): three links in the ladder doc plus its H15 mention, the SPEC's rule, conformance row and ADR citation, the benchmark survey, and the README deviations row.

Left for whoever merges, because it needs both trees at once: main's H17 ("a checkpoint is its manifest digest, not its tag and not its config digest") and your H19 overlap on the manifest-digest half. H19's enforcement is the stronger one — H17's conformance row says only "partly", while yours names store_audit.py --digests. Suggest H17 adopts that enforcement and H19 keeps the slice-naming half, which has no equivalent on main.

2. Which checkpoint produced the 0.875 — answered

Local manifest digest 637cc0ff1570, the nvfp4-tower artifact. That is the same digest this PR's own nvfp4 / nvfp4 tower row cites, so the Metal 0.875 and your 0.860 are the same weights on two backends, and the comparison is real.

Two caveats on that 0.875 before it goes on the ladder: it was rows 0–200 at num_ctx 16384, not the ladder's 8192, with extbench.py as it stood on main. Everything else matched (temperature 0, apply_sampling=False, num_predict 512, /api/generate). I'm re-running all Metal arms under this PR's exact protocol rather than pasting a number measured under another one.

3. store_audit.py on the Metal host — every quantised tag has moved

gemma4:12b-nvfp4     117d0d84cf2a -> ded7a2735003     6 / 737   vision 2, other 2, audio 1      2.4x
gemma4:26b-mlx-bf16  fa29b1495bbb -> 2c53b9e61626     4 / 1072  other 3                         1.39x
gemma4:26b-mxfp8     91109d36dc02 -> aee6e7cf8945   436 / 1072  language 266, vision 162        1.41x
gemma4:26b-nvfp4     c8656f50f0a6 -> f0fc7e0ae494   190 / 1072  vision 189                      3.55x  nvfp4 -> bf16
gemma4:31b-mlx-bf16  fb3f25b3bc8d -> cdca124786da     4 / 1247  other 3                         1.39x
gemma4:31b-mxfp8     1434769c1561 -> 9740f018f0d6   579 / 1247  language 411, vision 162        1.27x
gemma4:31b-nvfp4     637cc0ff1570 -> a22a363052da   194 / 1248  vision 191                      3.52x  nvfp4 -> bf16

The one that matters for the ladder: 31b-mxfp8 differs by 579 of 1247 layers — 162 in the vision tower and 411 in the language model. So a Metal mxfp8 row measured from this host's local copy is not comparable to your registry mxfp8 row; it is a different checkpoint, not a different backend. I'll label that row for what it is rather than lining it up with yours. 31b-mlx-bf16 is the opposite case: 4 layers, all tokenizer_config.json and two draft/ configs — same weights.

The three 31b checkpoints here are preserved as gemma4:31b-*-tower-* tags whose manifest digests equal the originals, so a pull can't overwrite what was measured.

4. The format proposal — one gap

Everything else I'd propose is already what this PR does: extbench.py as the one runner, ocrbench_table.py as the one renderer (its exact McNemar reproduced my hand-rolled one to the digit, 9/7, p = 0.804), manifest digest as identity, and a slice named by its rows and categories.

The gap is that the score files are gitignored and "kept on the array", so nobody else can re-render or re-check a published table. Proposal: any arm cited by a doc gets its ext_<tag>_ocrbench.json committed under vision-suite/bench-runs/ocrbench/, with a .gitignore negation, so a table regenerates from the repo with ocrbench_table.py --dir bench-runs/ocrbench. #324 committed a paired derivative of the Metal A/B on main; if you're happy with the per-arm layout I'll convert it so there's one format rather than two.

Small H11 note: your extbench now records host and server_version in the summary — the version on main when my A/B ran did not, so that data renders as "pre-H11 run (not recorded)". I've filled both in from the run script and said so in the file rather than letting them look captured.

5. Metal numbers, coming

Running now under this PR's protocol, cold server per run, two runs per arm: nvfp4 / nvfp4 tower (637cc0ff), mxfp8 / 8-bit tower (local), and bf16 — the row CUDA couldn't produce, since it was refused admission there.

Already rendered with your generator, the 1000-item kernel A/B behind #312 (same artifact 637cc0ff, same host, one archived binary per arm):

arm	engine	model	correct / scored	accuracy	±1 s.e.
0.34.0 fixed kernel	mlx-metal	gemma4:31b-nvfp4	835 / 1000	0.835	0.012
0.33.2 defect kernel	mlx-metal	gemma4:31b-nvfp4	833 / 1000	0.833	0.012

Paired: b = 9, c = 7, p = 0.804, unresolved. The MLX ollama#3912 kernel defect is undetectable end to end (ADR 0037).

--categories adds something the totals hide — where the two differ:

question type	n	fixed	defect
Regular Text Recognition	50	50	49
Irregular Text Recognition	50	41	39
Handwriting Recognition	50	35	34
Scene Text-centric VQA	200	186	185
Handwritten Mathematical Expression Recognition	100	37	39
Non-Semantic Text Recognition	50	46	47

The fixed kernel's gains sit in text recognition, where character fidelity is what's scored — consistent with the flips I inspected (Davidson/Davison, CORONADO/CORONA). The defect's two wins sit in handwritten maths, where the flips were LaTeX formatting under a contains-match scorer (OB - OD, an inserted \cdot). Not significant either way; just no longer arbitrary.

[glennneuber]

Superseded by #327, which merged as 88c34c8 and carries every commit from this branch alongside #324's and #325's.

Main now has the consolidated result: ADRs 0037 (kernel), 0038 (manifest-digest identity) and 0039 (nvfp4 global scales); one renderer in summarize_extbench.py with this branch's repeats, timing and categories folded in; store_audit.py with its tests; and docs/maxusai/ocrbench.md as the shared entry point. The later renumbering commit on this branch is redundant — #327 applied the same renumber before merging.

🤖 Generated with Claude Code