docs(vision): fold the ROCm/gfx1151 GGUF arms into the OCRBench ladder

[glennneuber]

Stacked on #323 — base is docs/ocrbench-quantisation-ladder, not main. This replaces #325, which I opened before finding #323 and which duplicated its renderer.

The AMD host ran the same three GGUF arms independently. Rather than publish a second ladder, they go into #323's doc as a third engine section, and the second renderer is dropped: ocrbench_table.py already does repeats, categories and the same exact McNemar, and SPEC H1's "a second one is the defect" applies to summarizers.

ROCm arms

| arm | engine | model | correct / scored | accuracy | ±1 s.e. | mean s/item | median | prompt_eval |
|---|---|---|---|---|---|---|---|---|
| q4_K_M | llama.cpp | `gemma4:31b-it-q4_K_M` | 171 / 200 | **0.855** | 0.025 | 7.7 | 7.7 | 1115 |
| q8_0 | llama.cpp | `gemma4:31b-it-q8_0` | 169 / 200 | **0.845** | 0.026 | 8.1 | 8.1 | 1115 |
| bf16 | llama.cpp | `gemma4:31b-it-bf16` | 169 / 200 | **0.845** | 0.026 | 8.3 | 8.1 | 1115 |

| A | B | A | B | b (A only) | c (B only) | p | resolved |
|---|---|---|---|---|---|---|---|
| q4_K_M | q8_0 | 0.855 | 0.845 | 3 | 1 | 0.625 | no |
| q4_K_M | bf16 | 0.855 | 0.845 | 3 | 1 | 0.625 | no |
| q8_0 | bf16 | 0.845 | 0.845 | 0 | 0 | 1.000 | no |

q8_0 and bf16 agree on every one of the 200 items. Same flat ladder as the CUDA arms, reached on different silicon.

The 1–2 item gaps against #323's q8_0 0.850 / bf16 0.855 sit inside three config deltas — num_ctx 16384 vs 8192, one run vs two, and split vs single image chunk — and the doc attributes them to none of them.

One correction to #323

| engine | the deployed build 0.34.1-dynres-16-g16649e8 | GGUF arms therefore decode the image in one batch (ADR 0036) |

Per-host, not a property of the build. ADR 0036's own diagnostic fires on every gemma4 load on gfx1151:

msg="generation batch below the image chunk, images decode in pieces" num_batch=1024 image_chunk_batch=2048

The 2048 floor is refused at every quantisation. availableMemoryForLoad takes its integrated-GPU branch and sizes the batch against 31 GiB of system RAM while the scheduler logs available="95.4 GiB" of GPU. On a discrete card that branch never triggers, which is why the CUDA arms get 2048.

So ADR 0036's 1.5–1.9× prefill gain is unavailable on gfx1151, and — stating it because the opposite was briefly assumed — the chunk still exceeds n_ubatch, so ADR 0036 does not mask the HIP MMQ race and compat 906 stays load-bearing for gemma4. ADR 0036 amended with the measurement.

The pair of ladders is worth more than either alone: two hosts decoded the image differently and landed within two items of each other at every rung — an independent confirmation of ADR 0036's "the split costs no scored cell", with the contrast between platforms.

Carried over from #325

• extbench.py H11. client.generate() stamps _host / _server_version on every response and extbench was discarding both, so no ext_*.json in either PR can say which build produced it. Persisted as sets, so a mid-run container restart cannot let one build vouch for another's rows. Orthogonal to docs: OCRBench quantisation ladder for gemma4:31b, and a store/registry audit #323's row-cache and retry work.
• ocrbench_table.py H13 footer. A file with no H11 fields renders pre-H11 run (not recorded) rather than inheriting a sibling's provenance — the defect H13 was written for, reached through a different renderer. Four tests including the two-build MIXED case.

14 + 8 tests pass. Still owed per ADR 0037: store_audit.py manifest digests for the three GGUF tags on this host.

🤖 Generated with Claude Code

[glennneuber]

Superseded by #327, which consolidated all three sessions' OCRBench runs and absorbed every piece of this PR — the extbench.py H11 provenance fix, summarize_extbench.py with its H13 footer, the ROCm quant-ladder doc, the ADR 0036 amendment and the learnings entry are all on main.

Worth recording: #327 kept this renderer and dropped ocrbench_table.py, the opposite of what I proposed in #323 — I had assumed the more featureful renderer should win and offered to drop mine. Good thing that was a proposal and not a push.

The one piece #327 did not absorb is #330: the CUDA ladder's engine row still reads "GGUF arms therefore decode the image in one batch", which is a per-host fact and false on gfx1151.

[glennneuber]

Heads-up before this merges: its base branch is gone and it would re-add a deleted file.

#323 is closed and #325 was closed unmerged; both were consolidated into #327, which merged as 88c34c8bb. GitHub still reports this PR CLEAN because ocrbench_table.py no longer exists in main, so re-adding it is not a textual conflict — but merging as-is would reintroduce the second renderer that #327 removed, and SPEC H1's "a second one is the defect" is exactly the reason it went.

We converged on opposite winners, which is worth stating plainly: this PR keeps ocrbench_table.py and drops summarize_extbench.py; #327 kept summarize_extbench.py — your default table and H13 provenance footer untouched, so your pasted tables stayed valid — and folded the other tool's --repeats, --timing and --categories into it. That is what is in main now, with the tests in test_summarizers.py.

Your ROCm results are already in main: the accuracy table and the per-arm timings (7.7 / 8.1 / 8.1 s) in docs/maxusai/ocrbench-gemma4-quant-ladder.md, and the three rows in the cross-host table in docs/maxusai/ocrbench.md, which carries a slice column so the 1000-item mlx-metal run is not read against the 200-item ones.

So what is left here is small. Rebasing onto main and dropping the re-added renderer would leave the genuinely new material: the ±1 s.e. column your run reports, the ADR 0036 note, and the learnings entry. summarize_extbench.py --timing emits that column, so the numbers can be re-rendered rather than hand-carried. Happy to do the rebase if you would rather not.

🤖 Generated with Claude Code

[glennneuber]

Rebased onto main as #331, which carries this PR's doc material without re-adding the deleted renderer: the ROCm section, the gfx1151 batch-floor correction, and the learnings entry in full — the 9px tier finding (bf16 scores 4, nvfp4 on the fixed kernel 3, nvfp4 on the broken kernel 4, so ollama#3912 stopped hiding a quantisation cost rather than causing one) and the mxfp8 bf16-tower trap, which this session hit independently the same day.

Two names changed in the port: summarize_extbench.py is the renderer that survived consolidation, and manifest-digest identity is ADR 0038 after #327's renumber.

Close this once #331 lands, or say the word and I will.

🤖 Generated with Claude Code

[glennneuber]

#331 is merged (f0c0c1a1f), so everything from this PR is now in main and it can close.

Where each piece landed:

• ROCm arms — your own ladder document (ocrbench-gemma4-quant-ladder.md) came in with docs: consolidate the three OCRBench runs — one test, one renderer, one format #327 and carries the tables, the batch confound and the wall times. The CUDA document points at it rather than copying it.
• The gfx1151 batch-floor correction — yours, via docs(ocrbench): the one-batch image decode is a per-host fact, not the build #330, with the availableMemoryForLoad root cause. Better than the version I had ported, so I took yours.
• The cross-host comparison — docs: fold the ROCm/gfx1151 GGUF arms in, rebased onto the consolidated tree #331, in the shared index as "Reading two hosts together": same rows and commit, different context rung, repeats, image chunk and seconds per item, and the conclusion that the two hosts decoded the image differently and landed within two items of each other at every rung.
• The learnings entry — in main: the 9px tier finding and the mxfp8 bf16-tower trap via docs(ocrbench): the one-batch image decode is a per-host fact, not the build #330, and docs: fold the ROCm/gfx1151 GGUF arms in, rebased onto the consolidated tree #331 added the resolved: no behaviour and the note that three renderers were written in a day and two dropped.

One thing worth your eye: the surviving renderer is summarize_extbench.py — yours, with the other tool's --repeats, --timing and --categories folded in and tests in test_summarizers.py.

🤖 Generated with Claude Code