Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/maxusai/ocrbench-quantisation-ladder.md
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,14 @@ handwriting (16–17 misses) and irregular text (10 misses, the same count on al
and no arm is more than one item better than another in any category. If a quantisation
cost exists at 31b, this slice does not show it in text recognition.

## Results β€” llama.cpp GGUF on ROCm / gfx1151

The AMD host ran the same three GGUF arms on the same commit and the same 200-row slice:
q4_K_M 171/200, q8_0 and bf16 169/200 each, with q8_0 and bf16 agreeing on every one of the
200 items. Its tables, its batch confound and its wall times live in
[ocrbench-gemma4-quant-ladder.md](ocrbench-gemma4-quant-ladder.md) rather than being copied
here; what the two ladders say together is in [ocrbench.md](ocrbench.md#reading-two-hosts-together).

## Harness note (2026-09-19)

Two arms died mid-run when a DNS blip made the row fetch fail, so `extbench.py` now
Expand Down
26 changes: 26 additions & 0 deletions docs/maxusai/ocrbench.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,32 @@ comparable within a slice, never across.
| 200, rows 0–200 | ROCm GGUF | `gemma4:31b-it-q8_0` | 169 / 200 | 0.845 |
| 200, rows 0–200 | ROCm GGUF | `gemma4:31b-it-bf16` | 169 / 200 | 0.845 |

Seconds per item are not comparable across those rows either: CUDA runs the GGUF arms at
about 5.0 s and ROCm/gfx1151 at 7.7–8.3 s, on different silicon with a different batch.
**ADR 0036's batch floor is a per-host fact, not a property of the build**: the same commit
asks for 2048 on both, gets it on CUDA and is refused it on gfx1151, so read the logged
`num_batch` rather than assuming the floor applied.

## Reading two hosts together

The CUDA and ROCm GGUF ladders ran the same 200 rows at offset 0, think off,
`/api/generate`, `apply_sampling=False` with `temperature 0`, on the same commit
`16649e8`, with contains-match scoring. Three things differed, and each is large enough to
cover the one-to-two item gaps between the two hosts' `q8_0` and `bf16` rows:

| | CUDA / sm_120 | ROCm / gfx1151 |
|---|---|---|
| `num_ctx` | 8192 | 16384 |
| runs per arm | 2 (0 items flipped) | 1 |
| image chunk | one batch | split β€” ADR 0036's floor is refused |
| seconds per item | 5.0 | 7.7–8.3 |

Do not attribute those gaps to the silicon. What the pair supports is stronger than either
ladder alone: **the two hosts decoded the image differently β€” one chunk against several β€”
and still landed within two items of each other at every rung.** A split non-causal decode
did not cost this benchmark anything measurable, which is the same answer ADR 0036 reached
from the scored side.

## What all three runs agree on

**No quantisation difference resolves.** Every paired test run on every host β€” 4-bit
Expand Down
11 changes: 7 additions & 4 deletions docs/maxusai/vision-learnings-log.md
Original file line number Diff line number Diff line change
Expand Up @@ -572,11 +572,14 @@ support; the paired discordant counts settle it in one line.
"quantization is costing us a tier" is a finding about a specific
quantization and kernel, never a general property of the checkpoint.
- **Enforced by** β€” `summarize_extbench.py --paired` (H7 generator) computes the
discordant counts and an **exact** McNemar. Exact, not chi-square: at 3-vs-1
the continuity-corrected approximation gives pβ‰ˆ0.317 against the true 0.625,
and 0.317 is the kind of number that gets reported as "trending".
discordant counts and an **exact** McNemar, and prints `resolved: no` rather
than an ordering when a pair does not clear it. Exact, not chi-square: at
3-vs-1 the continuity-corrected approximation gives pβ‰ˆ0.317 against the true
0.625, and 0.317 is the kind of number that gets reported as "trending".
`test_summarizers.py::TestExtbenchSummary` pins both the exact values and the
H13 footer behaviour.
H13 footer behaviour. Three sessions wrote a renderer for this within a day of
each other and two were dropped rather than shipped β€” SPEC H1's "a second one
is the defect" applies to summarizers, not just runners.
- **Cost** β€” none this time, because the paired columns were computed before
the numbers were written up. The near-miss is that the first draft of the
status report quoted 171 vs 175 across backends as if the gap meant something.
Loading