From fbcf423c738f155d2d2f5c22c687d2e3fa17321c Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Sat, 19 Sep 2026 18:20:53 +1000 Subject: [PATCH 1/2] docs: fold the ROCm/gfx1151 GGUF arms in, rebased onto the consolidated tree MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ports #326's content onto main after #327 merged. #326 was stacked on the closed #323 branch and would have re-added ocrbench_table.py, the renderer #327 deleted; its doc material is the part that was genuinely new, and it is here with the two names corrected — summarize_extbench.py is the surviving renderer, and manifest-digest identity is ADR 0038 after the renumber. The ROCm rows land as a third engine section: q4_K_M 171/200, q8_0 and bf16 169/200 each, with q8_0 and bf16 agreeing on every one of the 200 items. Flat ladder, different silicon, same conclusion as CUDA. It also corrects a claim this document made. ADR 0036's batch floor is a per-host fact, not a property of the build: the same commit asks for 2048 on both hosts, gets it on CUDA and is refused it on gfx1151, so a reader must check the logged num_batch rather than assume the floor applied. The shared index says so too, next to the seconds-per-item column that is likewise not comparable across hosts. The learnings entry keeps the AMD session's finding about the 9px tier — bf16 scores 4, nvfp4 on the fixed kernel scores 3, nvfp4 on the broken kernel scored 4, so #3912 did not cost a tier, it stopped hiding one — and its independent discovery that gemma4:31b-mxfp8 carries a bf16 vision tower. Co-Authored-By: Claude Opus 5 --- docs/maxusai/ocrbench-quantisation-ladder.md | 84 +++++++++++++++++++- docs/maxusai/ocrbench.md | 6 ++ docs/maxusai/vision-learnings-log.md | 11 ++- 3 files changed, 96 insertions(+), 5 deletions(-) diff --git a/docs/maxusai/ocrbench-quantisation-ladder.md b/docs/maxusai/ocrbench-quantisation-ladder.md index 98b583490f4..59665bb59e2 100644 --- a/docs/maxusai/ocrbench-quantisation-ladder.md +++ b/docs/maxusai/ocrbench-quantisation-ladder.md @@ -120,7 +120,7 @@ LIMIT=200 OFFSET=0 THINK=false NUM_CTX=8192 NUM_PREDICT=512 SLEEP=1 TIMEOUT=1800 | `num_ctx` | 8192, every arm | one image and a short question; identical across arms so no arm truncates | | endpoint | `/api/generate` | what the published slices used | | loaded models | one at a time, unloaded between arms | GPU0 is shared; a resident model skews the next arm | -| engine | the deployed build `0.34.1-dynres-16-g16649e8` | GGUF arms therefore decode the image in one batch (ADR 0036) | +| engine | the deployed build `0.34.1-dynres-16-g16649e8` | GGUF arms on **this** host decode the image in one batch (ADR 0036); that is a per-host fact, not a property of the build — read the logged `num_batch`, because on gfx1151 the same build is refused the floor ([below](#num_batch-on-gfx1151-adr-0036-asks-for-2048-and-is-refused)) | Each arm runs in a probe container on port 11521 against GPU0 with a 16 GiB overhead reserve, never against `:11497`. @@ -293,6 +293,88 @@ handwriting (16–17 misses) and irregular text (10 misses, the same count on al and no arm is more than one item better than another in any category. If a quantisation cost exists at 31b, this slice does not show it in text recognition. +## Results — llama.cpp GGUF on ROCm / gfx1151 + +Run independently on the AMD host before this PR existed, on the same commit and the same +200-row slice, and folded in here rather than published as a second ladder. Rendered by +`summarize_extbench.py`: + +``` +| arm | engine | model | correct / scored | accuracy | ±1 s.e. | mean s/item | median | prompt_eval | +|---|---|---|---|---|---|---|---|---| +| q4_K_M | llama.cpp | `gemma4:31b-it-q4_K_M` | 171 / 200 | **0.855** | 0.025 | 7.7 | 7.7 | 1115 | +| q8_0 | llama.cpp | `gemma4:31b-it-q8_0` | 169 / 200 | **0.845** | 0.026 | 8.1 | 8.1 | 1115 | +| bf16 | llama.cpp | `gemma4:31b-it-bf16` | 169 / 200 | **0.845** | 0.026 | 8.3 | 8.1 | 1115 | + +OCRBench `echo840/OCRBench` [test] rows 0–200, think false, endpoint generate, contains-match scoring (lmms-eval semantics). + +**Paired on the same items** (first run of each arm) + +| A | B | A | B | b (A only) | c (B only) | p | resolved | +|---|---|---|---|---|---|---|---| +| q4_K_M | q8_0 | 0.855 | 0.845 | 3 | 1 | 0.625 | no | +| q4_K_M | bf16 | 0.855 | 0.845 | 3 | 1 | 0.625 | no | +| q8_0 | bf16 | 0.845 | 0.845 | 0 | 0 | 1.000 | no | + +host: pre-H11 run (not recorded) · build: pre-H11 run (not recorded) +``` + +**Same conclusion, different silicon.** The ladder is flat: `q8_0` and `bf16` agree on +every one of the 200 items — zero discordant pairs — and `q4_K_M` is 3-vs-1 against each, +which does not resolve. Nothing here orders the quantisations. + +### Before these rows are read against the CUDA ones + +Same: 200 rows at offset 0, think off, `/api/generate`, `apply_sampling=False` with +`temperature 0` (both hardcoded in `extbench.py`), commit `16649e8`, contains-match scoring. + +Different, and each difference is large enough to cover the 1–2 item gaps between the two +hosts' `q8_0` / `bf16` rows: + +| | CUDA / sm_120 | ROCm / gfx1151 | +|---|---|---| +| `num_ctx` | 8192 | 16384 | +| runs per arm | 2 (0 flips) | 1 | +| image chunk | one batch | **split** — see below | + +I would not attribute those gaps to anything. What the pair of ladders *does* support is +stronger than either alone: the two hosts decoded the image differently and landed within +two items of each other at every rung. + +### `num_batch` on gfx1151: ADR 0036 asks for 2048 and is refused + +The engine row in "What is measured" says GGUF arms decode the image in one batch. That +holds on a discrete card and **not on this one**. ADR 0036's own diagnostic fires on every +gemma4 load here: + +``` +msg="generation batch below the image chunk, images decode in pieces" num_batch=1024 image_chunk_batch=2048 +``` + +1024 for `q4_K_M` at `-np 1`, 512 for `q8_0` and `bf16`, 512 for everything at `-np 2`. Not +for want of memory — `availableMemoryForLoad` takes its integrated-GPU branch and sizes the +batch against **31 GiB of system RAM** while the scheduler logs `available="95.4 GiB"` of +GPU. On a discrete card `discreteGPUFree` carries the whole figure and the branch never +triggers, which is why the CUDA arms get 2048. The branch's premise — that iGPU free memory +is "a static or slowly refreshed device baseline" — does not hold on a Strix Halo carve-out, +where the 96 GiB is a fixed allocation and not shared with the 31 GiB the host sees. + +Two consequences beyond this table. ADR 0036's measured 1.5–1.9× prefill gain is unavailable +on gfx1151 at any quantisation, and — worth stating because the opposite was briefly assumed +— the image chunk still exceeds `n_ubatch`, so ADR 0036 does **not** incidentally mask the +HIP MMQ race and `llama/compat/906-revert-hip-integrated-flag.patch` stays load-bearing for +gemma4. ADR 0036 carries the measurement. + +### Provenance, and what is still owed + +These three files predate the H11 change in this PR and render as +`pre-H11 run (not recorded)`. The build is `0.34.1-dynres-16649e8c`, attested from the +runner logs rather than from the score files — inferred provenance, per +[ADR 0024](adr/0024-locate-faults-before-fixing-them.md). Per +[ADR 0038](adr/0038-a-model-is-identified-by-its-manifest-digest.md) the manifest digests of +the three GGUF tags on this host are still owed before these rows are cited against another +host's. + ## Harness note (2026-09-19) Two arms died mid-run when a DNS blip made the row fetch fail, so `extbench.py` now diff --git a/docs/maxusai/ocrbench.md b/docs/maxusai/ocrbench.md index 83007f712c6..8d10ec12fa5 100644 --- a/docs/maxusai/ocrbench.md +++ b/docs/maxusai/ocrbench.md @@ -79,6 +79,12 @@ comparable within a slice, never across. | 200, rows 0–200 | ROCm GGUF | `gemma4:31b-it-q8_0` | 169 / 200 | 0.845 | | 200, rows 0–200 | ROCm GGUF | `gemma4:31b-it-bf16` | 169 / 200 | 0.845 | +Seconds per item are not comparable across those rows either: CUDA runs the GGUF arms at +about 5.0 s and ROCm/gfx1151 at 7.7–8.3 s, on different silicon with a different batch. +**ADR 0036's batch floor is a per-host fact, not a property of the build**: the same commit +asks for 2048 on both, gets it on CUDA and is refused it on gfx1151, so read the logged +`num_batch` rather than assuming the floor applied. + ## What all three runs agree on **No quantisation difference resolves.** Every paired test run on every host — 4-bit diff --git a/docs/maxusai/vision-learnings-log.md b/docs/maxusai/vision-learnings-log.md index bdd3c6f045a..15848c9707b 100644 --- a/docs/maxusai/vision-learnings-log.md +++ b/docs/maxusai/vision-learnings-log.md @@ -572,11 +572,14 @@ support; the paired discordant counts settle it in one line. "quantization is costing us a tier" is a finding about a specific quantization and kernel, never a general property of the checkpoint. - **Enforced by** — `summarize_extbench.py --paired` (H7 generator) computes the - discordant counts and an **exact** McNemar. Exact, not chi-square: at 3-vs-1 - the continuity-corrected approximation gives p≈0.317 against the true 0.625, - and 0.317 is the kind of number that gets reported as "trending". + discordant counts and an **exact** McNemar, and prints `resolved: no` rather + than an ordering when a pair does not clear it. Exact, not chi-square: at + 3-vs-1 the continuity-corrected approximation gives p≈0.317 against the true + 0.625, and 0.317 is the kind of number that gets reported as "trending". `test_summarizers.py::TestExtbenchSummary` pins both the exact values and the - H13 footer behaviour. + H13 footer behaviour. Three sessions wrote a renderer for this within a day of + each other and two were dropped rather than shipped — SPEC H1's "a second one + is the defect" applies to summarizers, not just runners. - **Cost** — none this time, because the paired columns were computed before the numbers were written up. The near-miss is that the first draft of the status report quoted 171 vs 175 across backends as if the gap meant something. From 3e4e73ebd3d4d101d79c8dfcee63ec36afa8ef20 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Sat, 19 Sep 2026 18:22:40 +1000 Subject: [PATCH 2/2] docs: put the cross-host reading in the shared index, not a second copy of the tables MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Main already carries the AMD host's own ladder doc, so folding its tables into the CUDA one would rebuild the duplication #327 removed. What was genuinely unique in #326 is the cross-host comparison, and that belongs where both hosts meet: ocrbench.md now states what differed between the two runs — context rung, repeats, one-batch against split decode, seconds per item — and what the pair supports that neither supports alone: the two hosts decoded the image differently and landed within two items of each other at every rung. Co-Authored-By: Claude Opus 5 --- docs/maxusai/ocrbench-quantisation-ladder.md | 84 ++------------------ docs/maxusai/ocrbench.md | 20 +++++ 2 files changed, 25 insertions(+), 79 deletions(-) diff --git a/docs/maxusai/ocrbench-quantisation-ladder.md b/docs/maxusai/ocrbench-quantisation-ladder.md index b4e0b28f596..d99db7b54dc 100644 --- a/docs/maxusai/ocrbench-quantisation-ladder.md +++ b/docs/maxusai/ocrbench-quantisation-ladder.md @@ -295,85 +295,11 @@ cost exists at 31b, this slice does not show it in text recognition. ## Results — llama.cpp GGUF on ROCm / gfx1151 -Run independently on the AMD host before this PR existed, on the same commit and the same -200-row slice, and folded in here rather than published as a second ladder. Rendered by -`summarize_extbench.py`: - -``` -| arm | engine | model | correct / scored | accuracy | ±1 s.e. | mean s/item | median | prompt_eval | -|---|---|---|---|---|---|---|---|---| -| q4_K_M | llama.cpp | `gemma4:31b-it-q4_K_M` | 171 / 200 | **0.855** | 0.025 | 7.7 | 7.7 | 1115 | -| q8_0 | llama.cpp | `gemma4:31b-it-q8_0` | 169 / 200 | **0.845** | 0.026 | 8.1 | 8.1 | 1115 | -| bf16 | llama.cpp | `gemma4:31b-it-bf16` | 169 / 200 | **0.845** | 0.026 | 8.3 | 8.1 | 1115 | - -OCRBench `echo840/OCRBench` [test] rows 0–200, think false, endpoint generate, contains-match scoring (lmms-eval semantics). - -**Paired on the same items** (first run of each arm) - -| A | B | A | B | b (A only) | c (B only) | p | resolved | -|---|---|---|---|---|---|---|---| -| q4_K_M | q8_0 | 0.855 | 0.845 | 3 | 1 | 0.625 | no | -| q4_K_M | bf16 | 0.855 | 0.845 | 3 | 1 | 0.625 | no | -| q8_0 | bf16 | 0.845 | 0.845 | 0 | 0 | 1.000 | no | - -host: pre-H11 run (not recorded) · build: pre-H11 run (not recorded) -``` - -**Same conclusion, different silicon.** The ladder is flat: `q8_0` and `bf16` agree on -every one of the 200 items — zero discordant pairs — and `q4_K_M` is 3-vs-1 against each, -which does not resolve. Nothing here orders the quantisations. - -### Before these rows are read against the CUDA ones - -Same: 200 rows at offset 0, think off, `/api/generate`, `apply_sampling=False` with -`temperature 0` (both hardcoded in `extbench.py`), commit `16649e8`, contains-match scoring. - -Different, and each difference is large enough to cover the 1–2 item gaps between the two -hosts' `q8_0` / `bf16` rows: - -| | CUDA / sm_120 | ROCm / gfx1151 | -|---|---|---| -| `num_ctx` | 8192 | 16384 | -| runs per arm | 2 (0 flips) | 1 | -| image chunk | one batch | **split** — see below | - -I would not attribute those gaps to anything. What the pair of ladders *does* support is -stronger than either alone: the two hosts decoded the image differently and landed within -two items of each other at every rung. - -### `num_batch` on gfx1151: ADR 0036 asks for 2048 and is refused - -The engine row in "What is measured" says GGUF arms decode the image in one batch. That -holds on a discrete card and **not on this one**. ADR 0036's own diagnostic fires on every -gemma4 load here: - -``` -msg="generation batch below the image chunk, images decode in pieces" num_batch=1024 image_chunk_batch=2048 -``` - -1024 for `q4_K_M` at `-np 1`, 512 for `q8_0` and `bf16`, 512 for everything at `-np 2`. Not -for want of memory — `availableMemoryForLoad` takes its integrated-GPU branch and sizes the -batch against **31 GiB of system RAM** while the scheduler logs `available="95.4 GiB"` of -GPU. On a discrete card `discreteGPUFree` carries the whole figure and the branch never -triggers, which is why the CUDA arms get 2048. The branch's premise — that iGPU free memory -is "a static or slowly refreshed device baseline" — does not hold on a Strix Halo carve-out, -where the 96 GiB is a fixed allocation and not shared with the 31 GiB the host sees. - -Two consequences beyond this table. ADR 0036's measured 1.5–1.9× prefill gain is unavailable -on gfx1151 at any quantisation, and — worth stating because the opposite was briefly assumed -— the image chunk still exceeds `n_ubatch`, so ADR 0036 does **not** incidentally mask the -HIP MMQ race and `llama/compat/906-revert-hip-integrated-flag.patch` stays load-bearing for -gemma4. ADR 0036 carries the measurement. - -### Provenance, and what is still owed - -These three files predate the H11 change in this PR and render as -`pre-H11 run (not recorded)`. The build is `0.34.1-dynres-16649e8c`, attested from the -runner logs rather than from the score files — inferred provenance, per -[ADR 0024](adr/0024-locate-faults-before-fixing-them.md). Per -[ADR 0038](adr/0038-a-model-is-identified-by-its-manifest-digest.md) the manifest digests of -the three GGUF tags on this host are still owed before these rows are cited against another -host's. +The AMD host ran the same three GGUF arms on the same commit and the same 200-row slice: +q4_K_M 171/200, q8_0 and bf16 169/200 each, with q8_0 and bf16 agreeing on every one of the +200 items. Its tables, its batch confound and its wall times live in +[ocrbench-gemma4-quant-ladder.md](ocrbench-gemma4-quant-ladder.md) rather than being copied +here; what the two ladders say together is in [ocrbench.md](ocrbench.md#reading-two-hosts-together). ## Harness note (2026-09-19) diff --git a/docs/maxusai/ocrbench.md b/docs/maxusai/ocrbench.md index 8d10ec12fa5..b2b9c6a1a3a 100644 --- a/docs/maxusai/ocrbench.md +++ b/docs/maxusai/ocrbench.md @@ -85,6 +85,26 @@ about 5.0 s and ROCm/gfx1151 at 7.7–8.3 s, on different silicon with a differe asks for 2048 on both, gets it on CUDA and is refused it on gfx1151, so read the logged `num_batch` rather than assuming the floor applied. +## Reading two hosts together + +The CUDA and ROCm GGUF ladders ran the same 200 rows at offset 0, think off, +`/api/generate`, `apply_sampling=False` with `temperature 0`, on the same commit +`16649e8`, with contains-match scoring. Three things differed, and each is large enough to +cover the one-to-two item gaps between the two hosts' `q8_0` and `bf16` rows: + +| | CUDA / sm_120 | ROCm / gfx1151 | +|---|---|---| +| `num_ctx` | 8192 | 16384 | +| runs per arm | 2 (0 items flipped) | 1 | +| image chunk | one batch | split — ADR 0036's floor is refused | +| seconds per item | 5.0 | 7.7–8.3 | + +Do not attribute those gaps to the silicon. What the pair supports is stronger than either +ladder alone: **the two hosts decoded the image differently — one chunk against several — +and still landed within two items of each other at every rung.** A split non-causal decode +did not cost this benchmark anything measurable, which is the same answer ADR 0036 reached +from the scored side. + ## What all three runs agree on **No quantisation difference resolves.** Every paired test run on every host — 4-bit