diff --git a/docs/maxusai/ocrbench-quantisation-ladder.md b/docs/maxusai/ocrbench-quantisation-ladder.md index 9b3c40d7dfc..d99db7b54dc 100644 --- a/docs/maxusai/ocrbench-quantisation-ladder.md +++ b/docs/maxusai/ocrbench-quantisation-ladder.md @@ -293,6 +293,14 @@ handwriting (16–17 misses) and irregular text (10 misses, the same count on al and no arm is more than one item better than another in any category. If a quantisation cost exists at 31b, this slice does not show it in text recognition. +## Results — llama.cpp GGUF on ROCm / gfx1151 + +The AMD host ran the same three GGUF arms on the same commit and the same 200-row slice: +q4_K_M 171/200, q8_0 and bf16 169/200 each, with q8_0 and bf16 agreeing on every one of the +200 items. Its tables, its batch confound and its wall times live in +[ocrbench-gemma4-quant-ladder.md](ocrbench-gemma4-quant-ladder.md) rather than being copied +here; what the two ladders say together is in [ocrbench.md](ocrbench.md#reading-two-hosts-together). + ## Harness note (2026-09-19) Two arms died mid-run when a DNS blip made the row fetch fail, so `extbench.py` now diff --git a/docs/maxusai/ocrbench.md b/docs/maxusai/ocrbench.md index 83007f712c6..b2b9c6a1a3a 100644 --- a/docs/maxusai/ocrbench.md +++ b/docs/maxusai/ocrbench.md @@ -79,6 +79,32 @@ comparable within a slice, never across. | 200, rows 0–200 | ROCm GGUF | `gemma4:31b-it-q8_0` | 169 / 200 | 0.845 | | 200, rows 0–200 | ROCm GGUF | `gemma4:31b-it-bf16` | 169 / 200 | 0.845 | +Seconds per item are not comparable across those rows either: CUDA runs the GGUF arms at +about 5.0 s and ROCm/gfx1151 at 7.7–8.3 s, on different silicon with a different batch. +**ADR 0036's batch floor is a per-host fact, not a property of the build**: the same commit +asks for 2048 on both, gets it on CUDA and is refused it on gfx1151, so read the logged +`num_batch` rather than assuming the floor applied. + +## Reading two hosts together + +The CUDA and ROCm GGUF ladders ran the same 200 rows at offset 0, think off, +`/api/generate`, `apply_sampling=False` with `temperature 0`, on the same commit +`16649e8`, with contains-match scoring. Three things differed, and each is large enough to +cover the one-to-two item gaps between the two hosts' `q8_0` and `bf16` rows: + +| | CUDA / sm_120 | ROCm / gfx1151 | +|---|---|---| +| `num_ctx` | 8192 | 16384 | +| runs per arm | 2 (0 items flipped) | 1 | +| image chunk | one batch | split — ADR 0036's floor is refused | +| seconds per item | 5.0 | 7.7–8.3 | + +Do not attribute those gaps to the silicon. What the pair supports is stronger than either +ladder alone: **the two hosts decoded the image differently — one chunk against several — +and still landed within two items of each other at every rung.** A split non-causal decode +did not cost this benchmark anything measurable, which is the same answer ADR 0036 reached +from the scored side. + ## What all three runs agree on **No quantisation difference resolves.** Every paired test run on every host — 4-bit diff --git a/docs/maxusai/vision-learnings-log.md b/docs/maxusai/vision-learnings-log.md index bdd3c6f045a..15848c9707b 100644 --- a/docs/maxusai/vision-learnings-log.md +++ b/docs/maxusai/vision-learnings-log.md @@ -572,11 +572,14 @@ support; the paired discordant counts settle it in one line. "quantization is costing us a tier" is a finding about a specific quantization and kernel, never a general property of the checkpoint. - **Enforced by** — `summarize_extbench.py --paired` (H7 generator) computes the - discordant counts and an **exact** McNemar. Exact, not chi-square: at 3-vs-1 - the continuity-corrected approximation gives p≈0.317 against the true 0.625, - and 0.317 is the kind of number that gets reported as "trending". + discordant counts and an **exact** McNemar, and prints `resolved: no` rather + than an ordering when a pair does not clear it. Exact, not chi-square: at + 3-vs-1 the continuity-corrected approximation gives p≈0.317 against the true + 0.625, and 0.317 is the kind of number that gets reported as "trending". `test_summarizers.py::TestExtbenchSummary` pins both the exact values and the - H13 footer behaviour. + H13 footer behaviour. Three sessions wrote a renderer for this within a day of + each other and two were dropped rather than shipped — SPEC H1's "a second one + is the defect" applies to summarizers, not just runners. - **Cost** — none this time, because the paired columns were computed before the numbers were written up. The near-miss is that the first draft of the status report quoted 171 vs 175 across backends as if the gap meant something.