docs(vision): OCRBench quant ladder for gemma4:31b (q4/q8/bf16), and a paired summarizer

[glennneuber]

OCRBench, 200 rows, gemma4:31b on ROCm/GGUF at three quantizations — run to answer MLX-Metal's correct: 175, accuracy: 0.875 on the same slice, and to supply the unquantized control the GGUF path had never had (the 2026-09-18 learnings entry: a tier move is not evidence until bf16 is measured).

One binary 0.34.1-dynres-16649e8c (0.34.1 + compat 906 + dio knob + ADR 0036), think off, -c 16384 -np 1, offloaded 61/61 every arm, median prompt_eval_count 1113 in all three.

The aggregate column is misleading, and the paired columns say why

| model | scored | errors | empty | correct | accuracy | think | endpoint |
|---|---|---|---|---|---|---|---|
| `gemma4:31b-it-q4_K_M` | 200 | 0 | 0 | 171 | **0.855** | false | generate |
| `gemma4:31b-it-q8_0` | 200 | 0 | 0 | 169 | **0.845** | false | generate |
| `gemma4:31b-it-bf16` | 200 | 0 | 0 | 169 | **0.845** | false | generate |

| pair | both ✓ | both ✗ | A only | B only | McNemar exact p |
|---|---|---|---|---|---|
| ocr1np_q4_k_m vs ocr1np_q8_0 | 168 | 28 | 3 | 1 | 0.625 |
| ocr1np_q4_k_m vs ocr1np_bf16 | 168 | 28 | 3 | 1 | 0.625 |
| ocr1np_q8_0 vs ocr1np_bf16 | 169 | 31 | 0 | 0 | 1.000 |

q8_0 and bf16 are identical on every one of the 200 items. Zero discordant pairs. 8-bit costs nothing on this benchmark, so q8_0 is a usable stand-in for the unquantized arm. q4_K_M's apparent lead is 3 items won and 1 lost — a coin-flip disagreement.

The gap to MLX-Metal's 0.875 is inside noise at n=200 (95% interval on 0.855 is ±0.049) and cannot be sharpened without their per-item ext_*_ocrbench.json. The doc says so explicitly rather than publishing a ranking.

Note the GGUF ladder is flat where the MLX ladder was not: on Metal, nvfp4 and bf16 differed by a 9 px tier once ollama#3912 landed. Same model family, opposite result — so "quantization costs a tier" is a finding about a specific quantization and kernel, not a property of the checkpoint.

ADR 0036 is inert on gfx1151, and that is measured, not inferred

Its own diagnostic fires on every load: num_batch=1024 image_chunk_batch=2048 (q4, -np 1), 512 for q8/bf16, 512 at -np 2. Not for want of memory — the scheduler logs available="95.4 GiB" of GPU while availableMemoryForLoad returns the 31 GiB system figure, because its iGPU branch prefers live system memory whenever it is smaller than shared GPU free. That branch's premise (iGPU free memory is "a static or slowly refreshed device baseline") does not hold on a Strix Halo carve-out.

Consequence worth stating because I briefly assumed the opposite: the chunk still exceeds n_ubatch, so ADR 0036 does not incidentally mask the HIP MMQ race, and 906-revert-hip-integrated-flag.patch remains load-bearing for gemma4. ADR 0036 amended with this rather than left to read as if the optimization is active.

The 1.5–1.9× prefill gain ADR 0036 measured is unavailable on gfx1151 until that sizing is revisited. Not fixed here — it is a scheduler change with a blast radius beyond this doc.

Harness

• summarize_extbench.py — H7 generator for ext_*.json, H13 provenance footer, --paired exact McNemar. Exact and not chi-square because the uncorrected form reads p=0.317 at 3-vs-1 against the true 0.625, which is the kind of number that gets reported as "trending".
• extbench.py now persists the host/server_version that client.generate() already stamped and it was discarding (H11) — as sets, so a mid-run container restart cannot let one build vouch for another's rows. The three files here predate that and render pre-H11 run (not recorded); the build in the doc heading is attested from runner logs and labelled as inferred, per ADR 0024. Re-running purely to stamp them would cost ~5 h of GPU and ~90 GiB of pulls.
• test_summarizers.py::TestExtbenchSummary — unrecorded-file, mixed-campaign and clean-footer cases, plus the exact p-values. 139 tests pass.

Confound, stated rather than buried

q4_K_M ran at -b 1024 and the other two at -b 512 (the memory arithmetic above). So q4's 3-vs-1 edge could be quantization or batch; these data cannot separate them. q8_0 vs bf16 is clean — same batch, same everything, zero discordant items.

🤖 Generated with Claude Code

[glennneuber]

Superseded by #326, which stacks the same measurement onto #323 instead of duplicating it.

#323 landed the same OCRBench ladder from the CUDA host, with a renderer (ocrbench_table.py) that already does everything summarize_extbench.py did here plus repeats and the category split. SPEC H1's "a second one is the defect" applies to summarizers, so mine is dropped rather than merged.

What survives, in #326: the ROCm arms as a third engine section in #323's doc, the extbench.py H11 provenance fix (orthogonal to #323's row-cache work), the H13 footer ported into ocrbench_table.py, and the ADR 0036 amendment — the 2048 floor is refused at every quantisation on gfx1151, which also corrects #323's assumption that the build implies a one-batch image decode.

[glennneuber]

Consolidated into #327, which merges this PR with the other two OCRBench streams (#323 CUDA, #324 mlx-metal, #325 ROCm/GGUF). It contains every commit from all three, so merging #327 closes them together and there is no three-way conflict to resolve later.

What moved, so nothing is lost:

• One renderer. summarize_extbench.py keeps its default table and H13 footer exactly as docs(vision): OCRBench quant ladder for gemma4:31b (q4/q8/bf16), and a paired summarizer #325 wrote them, so pasted tables stay valid; it gains --repeats, --timing and --categories from docs: OCRBench quantisation ladder for gemma4:31b, and a store/registry audit #323's ocrbench_table.py, which is deleted and its tests folded into test_summarizers.py.
• One SPEC numbering. ADR 0037: keep the MLX #3912 fix and promote 0.34 on mlx-metal — campaign, OCRBench, SPEC H15–H18, results #324's H15–H18 stand. docs: OCRBench quantisation ladder for gemma4:31b, and a store/registry audit #323's rule becomes H19 and defers its digest half to H17, whose conformance row now names store_audit.py as the enforcement it recorded as missing.
• One ADR sequence. 0037 stays the kernel decision; manifest-digest identity is 0038, the nvfp4 global-scale representation 0039.
• One entry point. docs/maxusai/ocrbench.md holds the shared method, the format, and all three hosts' numbers with a slice column, because the 1000-item run and the 200-item runs are not comparable.

All three runs agree: no quantisation difference resolves on any host, and OCRBench cannot separate the kernels where the goldens can.

🤖 Generated with Claude Code