docs: consolidate the three OCRBench runs — one test, one renderer, one format - #327
Merged
Merged
Conversation
…ry audit A published tag is not a fixed artifact. gemma4:31b-nvfp4 in this store and gemma4:31b-nvfp4 in the library today differ in 194 of 1248 layers, all on the vision path and each ~3.5x larger upstream: the library re-published the tag with a bf16 vision tower while our copy keeps the nvfp4 one. The config blob is unchanged, so `ollama show` surfaces nothing; 26b-nvfp4 and 12b-nvfp4 moved the same way and qwen3.5:0.8b-mlx had its tensors renamed. Every gemma4 vision measurement in this repo that names an nvfp4 tag was taken against the quantised tower, so a host that pulls today will not reproduce them. store_audit.py finds this on any host: it diffs the store's manifests against the registry and prints what moved, where, and the size ratio. --digests prints the manifest digest to cite instead of the tag. The ladder doc carries the method, the CUDA rows as they land, and an empty mlx-metal section for the Metal session, which should run the audit first so its number can be placed. ocrbench_table.py renders the tables from extbench.py score files, paired with an exact McNemar test, because a 200-item slice cannot resolve the 3-item gaps the accuracy column shows. Tests: test_store_audit.py (9), test_ocrbench_table.py (7). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…GGUF row Two 200-item arms died when a DNS blip made the /rows fetch fail: extbench re-asked the datasets server for the same fixed slice at the start of every arm, so every arm depended on that service resolving. It now caches the slice beside the images and retries with backoff; REFRESH_ROWS=1 re-fetches, and a short slice is never cached so a truncated fetch cannot poison a later arm. Also records the first llama.cpp row: q4_K_M 171/200 against mlx-cuda nvfp4's 172/200, five discordant items in both directions, p = 1.0 — one engine is not reading these images better than the other. Tests: test_extbench.py (8). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… rows 0–200 are The slice is not a sample of OCRBench. The dataset is ordered by task, so rows 0–200 are four of its ten categories at 50 items each — the text-recognition half — with no VQA, key-information extraction, digit strings or handwritten maths. Every number here, and the Metal 0.875, is an accuracy on that half and must not be called an OCRBench score or set beside a published one. Still the right slice for this comparison, since the arms answer identical items. GGUF results, both arms repeated: q4_K_M 0.855 twice and q8_0 0.850 twice, with zero items changing verdict between runs. That determinism is the control the ladder needs. Paired, q4 and q8 differ on one item out of 200 in q4's favour, so q8 buys nothing here at the same seconds per item. ocrbench_table.py gains --categories, which reads the cached row slice (no network) and splits each arm by question type: regular and artistic text are at ceiling for every arm, all the headroom is handwriting and irregular text, and no arm leads another by more than one item in any category. Tests: test_ocrbench_table.py now 10. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e per image Three MLX arms, each run twice, zero items flipping in any of them. The two nvfp4 rows share a language model and differ only in the tower: ours is nvfp4, the library's is bf16, and the bf16 one answers in 1.4 s against 2.3 s on identical images at the same 1115-token prefill. The bf16 tower is 3.5x the bytes, so this is kernel throughput and not bandwidth — MLX's quantised matmul on sm_120 runs at about half the dense bf16 rate, as the qqmm bench measured directly. No accuracy difference survives the paired test: tower precision p = 0.625, language-model precision (nvfp4 vs mxfp8 over the same tower) one item in 200. On this text-recognition slice the ladder is flat and only the clock moves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
bf16 scores 171/200 twice, the same as q4_K_M, and q8_0 sits one item below both. Paired, q4 and bf16 differ on four items, two each way; nothing resolves. Seconds per item are identical across the three, because the workload is prefill-bound with an eight-token answer, so tripling the weights costs no clock. All 61 layers of the bf16 model fit on the GPU (58 GiB weights, 1.8 GiB KV at num_ctx 8192), leaving 19 GB free on a shared card, so no arm ran partly on CPU. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ADR 0037 (proposed): a measured model is identified by its manifest digest, not its tag. gemma4:31b-nvfp4 was re-published with a bf16 vision tower — 194 layers different, config blob unchanged, ollama show silent — and the old artifact cannot be fetched back, so the store is the archive. Records cite digests, benchmark pulls go to a separate store, and a cross-host comparison runs the audit on both hosts first. ADR 0038 (proposed): nvfp4 global scales are stored as the checkpoint multiplier and converted to MLX's m x 2688 form only at the call sites that hand the scale to MLX. The stored-MLX-form round trip f32(f32(m x 2688)/2688) misses m by one ulp for 17 of 31b's 191 vision scales, which is the whole of the CUDA-side encoder move in #312; the three wrappers that apply the scale themselves — QuantizedMatmul, Dequantize, GatherQMM off Metal — all pay it. #287's prefill dequantisation runs through Dequantize, so it inherits the fix; it stays held on its other grounds. SPEC vision-harness-reuse H15: a reported model is named by its manifest digest, and a slice is named by what it contains. The survey's claim that a 200-item OCRBench slice is comparable to a model card is corrected in place: the set is ordered by task, so rows 0-200 are four of ten categories. The ladder doc now states which checkpoint produced the mlx-cuda 0.860 (this store's 4-bit-tower copy, pulled 2026-08-17, digest 637cc0ff1570), what makes the Metal 0.875 comparable to it, and the one command that settles it. The MLX bf16 arm is recorded as refused by admission — 75.0 GiB asked against 62.7 GiB available with the 16 GiB reserve — not forced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…arizer Ran OCRBench 200 rows on ROCm/GGUF at q4_K_M, q8_0 and bf16 against mlx-metal's 0.875 on the same slice, all on 0.34.1-dynres-16649e8c (906 + dio knob + ADR 0036), think off, -c 16384 -np 1. The aggregate column reads 0.855 / 0.845 / 0.845 and invites an ordering the data does not support. Paired over the shared row set: q8_0 and bf16 are identical on all 200 items (zero discordant pairs), and q4_K_M differs by 3 won / 1 lost, exact two-sided p = 0.625. So 8-bit costs nothing here and q8_0 stands in for the unquantized arm -- the control the GGUF path had never had. The cross-backend gap to 0.875 is inside noise at n=200 and cannot be sharpened without mlx-metal's per-item file. Also measured, from ADR 0036's own diagnostic firing on every load: its 2048 floor is DENIED at every quantization on gfx1151, not for want of memory but because availableMemoryForLoad sizes an integrated GPU against the 31 GiB system figure while the scheduler logs 95.4 GiB of GPU free. So the ADR is inert on the fork's own production hardware, and -- worth stating because it was briefly assumed otherwise -- the image chunk still exceeds n_ubatch, so it does not mask the HIP MMQ race and compat 906 stays load-bearing for gemma4. Harness, to keep this reproducible rather than retyped: - summarize_extbench.py: H7 generator for ext_*.json, H13 provenance footer, --paired exact McNemar. Exact and not chi-square because the uncorrected form reads p=0.317 at 3-vs-1 against the true 0.625. - extbench.py now persists the host/server_version client.generate() already stamps (H11), collected as sets so a mid-run restart cannot let one build vouch for another's rows. The three files here predate that and render as "pre-H11 run (not recorded)" rather than inheriting a sibling's provenance. - test_summarizers.py: TestExtbenchSummary pins the footer cases and the exact p-values. 139 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… ROCm ladder branch
…, one format Three sessions ran OCRBench against gemma4:31b on the same days — mlx-metal (#324, 1000 items, the kernel question), ROCm/GGUF (#325, the q4/q8/bf16 ladder with the unquantised control) and CUDA (#323, both engines plus the vision-tower axis) — and each brought its own summarizer, its own SPEC numbering and its own ADR 0037. This merges all three and removes the duplication. - One renderer. summarize_extbench.py keeps its default table and provenance footer untouched, so documents that pasted it stay valid (H7), and gains --repeats, --timing and --categories from the CUDA branch's ocrbench_table.py, which is deleted. Its tests move into test_summarizers.py beside the existing extbench ones. - One SPEC numbering. The mlx-metal campaign's H15-H18 stand; the CUDA branch's rule becomes H19 (a slice names its rows and its categories) and defers the digest half to H17, whose conformance row now names store_audit.py as the enforcement it said was missing. - One ADR sequence. 0037 stays the kernel decision; manifest-digest identity becomes 0038 and the nvfp4 global-scale representation 0039. - One entry point. docs/maxusai/ocrbench.md carries the method, the format, the two rules that bind a result, and every host's numbers in one table — with the slice column that stops the 1000-item run being read against the 200-item ones. All three runs agree on the substance: no quantisation difference resolves on any host (p 0.375 to 1.000), and OCRBench cannot tell the fixed and defective kernels apart (p = 0.804) where the vision goldens can. What moves is speed: the vision tower's precision is worth 39% of the time per image on CUDA. Tests: 144 in test_summarizers.py, 9 in test_store_audit.py, 8 in test_extbench.py. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 19, 2026
glennneuber
pushed a commit
that referenced
this pull request
Sep 19, 2026
…py of the tables Main already carries the AMD host's own ladder doc, so folding its tables into the CUDA one would rebuild the duplication #327 removed. What was genuinely unique in #326 is the cross-host comparison, and that belongs where both hosts meet: ocrbench.md now states what differed between the two runs — context rung, repeats, one-batch against split decode, seconds per item — and what the pair supports that neither supports alone: the two hosts decoded the image differently and landed within two items of each other at every rung. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three sessions ran OCRBench against gemma4:31b within a day of each other, and each arrived with its own summarizer, its own SPEC numbering and its own ADR 0037. This branch merges all three — #323 (CUDA, both engines), #324 (mlx-metal, the kernel question) and #325 (ROCm/GGUF, the quantisation ladder) — so there is one thing to merge instead of a three-way conflict.
It contains every commit from those three PRs. Merging this closes all of them; nothing is dropped and authorship is preserved.
What was duplicated, and what it is now
summarize_extbench.py(#325) andocrbench_table.py(#323)--repeats,--timingand--categoriesfrom the CUDA branch, andocrbench_table.pyis deleted with its tests folded intotest_summarizers.pystore_audit.pyas the enforcement it recorded as missingdocs/maxusai/ocrbench.mdcarries the shared method, the format, the two rules that bind a result, and every host's numbers in one tableWhat the three runs agree on
No quantisation difference resolves anywhere. Every paired test on every host returns p between 0.375 and 1.000 — 4-bit against 8-bit against bf16, on GGUF and MLX, and the fixed kernel against the defective one over the full 1000 items. Arms differ by single items in both directions, and two engines reproduced their arms item for item across repeats, so this is not noise hiding a signal.
OCRBench cannot settle a kernel question, which #324 states plainly: 1000 items cannot separate the fixed and defective
fp_qmm_t(p = 0.804) where the vision golden delta separates them exactly. Keep the goldens for numerical defects.Speed moves where quality does not. The vision tower's precision is worth 39 % of the time per image on CUDA — 1.4 s against 2.3 s for the same language model on identical images at an identical 1115-token prefill — because MLX's quantised matmul on sm_120 runs at about half the dense bf16 rate.
The cross-host table has a slice column for a reason
The mlx-metal run is 1000 items across all ten categories; the others are rows 0–200, which are four recognition categories the models find easier. 0.835 and 0.855 are not comparable numbers, and the table says so where a reader would otherwise read down it.
Production
gemma4:31b-nvfp4on:11497was promoted to the library's bf16-tower artifact on the maintainer's word (39 % faster per image, accuracy within noise). The 4-bit-tower copy every earlier measurement used is archived in the same store asgemma4:31b-nvfp4-tower4bit, digest637cc0ff1570…, per ADR 0038 — the registry no longer serves it, so that copy is the only one.Tests: 144 in
test_summarizers.py, 9 intest_store_audit.py, 8 intest_extbench.py.🤖 Generated with Claude Code