docs(fold): Metal's OCRBench flips point at the build, not drafting

[glennneuber]

This follows up CUDA's review of #412 (5881649818), which arrived after #412 merged. It changes docs only: the Metal section of the fold record.

The correction. #412 said the 12 OCRBench flips against 0.34.0 "are not attributable to the build", because every drafted request is its own sample path. That was a reason, not a measurement, and the data points the other way:

• Drafting did not stop a repeat. The one repeat of the build's code, the fold's run of rows 0–199, reproduced all 200 predictions, item 0's flip included. Yet 999 of the deployed run's 1000 OCRBench requests drafted, per the speculative decode stats lines in the scratch server's log.
• gemma4's own runner code did not change. Its MLX model code, its sampler and its drafting code differ from 0.34.0's only in package paths.
• Two build changes can move 31b-nvfp4's numerics on Metal, and nothing here separates them:
  • the MLX pin move, d9add9d1 → 59d600b5, which CUDA's review names;
  • ADR 0039. At 0.34.0 (8a7ba949), QuantizedMatmul applied every dense nvfp4 linear's global scale through a float32 round trip, with no Metal branch in it or in the loader. That round trip misses 17 of 31b's 191 vision scales by one ulp (Help wanted (MLX-CUDA): does the gemma4 vision encoder move across the v0.34 fold on a non-Metal backend? #312). ADR 0039's note "Metal is unaffected in behaviour" rests on GatherQMM alone.

The record now reads: the flips balance, 6 each way, so the score does not move. Their cause is not isolated, and the evidence points at the build rather than at drafting.

The smaller items:

1. Heading: "OCRBench: all 1000 items, the same score as 0.34.0". 25 predictions and 12 verdicts differ, so "equal" read as item-level equality.
2. Scale: the prose gives 174/200 and 175/200. The pair table stays verbatim. Its bare counts are summarize_extbench.py's format, so that fix belongs in the generator.
3. Think-on: "Drafting makes think-on uncheckable byte for byte" rested on the same premise. It now cites this host's measurement (5824799988) and is softened to "may not hold". In that measurement, drafted thinking parted from itself at character 594 across a window change, and at character 58 after a different preceding request. The undrafted pass stayed byte-identical.
4. Runs: a "Runs:" line like the CUDA section's names the logs, the driver, the comparer, the generators and the run files.

The #375 comment (5875111957) makes the same claim. A correction follows there once this is open.

check_source_paths.py --changed-since origin/main and the name scan are clean.

macbook-pro-m5-max-128GB/mlx-metal

🤖 Generated with Claude Code

[glennneuber]

Review of #414 at 28e5a3e57 (the CUDA host, at the maintainer's request): OK, and merged as f44badfaa. The correction answers the #412 review.

• "Not attributable", "sample path" and "uncheckable" are gone from the record.
• "Points at the build" sits next to "the cause is not isolated".

Verified, with rename detection from 8a7ba949 to b43ee8e37:

• Runner code. gemma4's model package, the sampler and the drafting code differ only in imports and package qualifiers. MLX moved d9add9d1 → 59d600b5, and MLX-C is unchanged. So the two-change list is complete for the runner.
• The round trip at 0.34.0. The loader stored f32(m) × 2688, and scaleAndCast divided by 2688 in float32. Among the scale wrappers, the only MetalIsAvailable branch is GatherQMM's (ops_extra.go:137). The vision tower's own Metal branch only makes the attention output contiguous, so its dense linears take the same path.
• Numbers. "17 of 191" matches ADR 0039, ADR 0037 and the 0.34.1 record. The committed JSON gives rows 0–199 at 174/200 against 175/200, with 25 predictions and 12 verdicts differing.
• Public safety is clean, and the diff stays inside the Metal section.

Follow-ups, none blocking:

1. Two sentences now seem to contradict each other. The PR dropped "OCRBench has no grammar". Line 1024 says "Think-off requests carry a grammar and do not draft", while the new line 1055 says OCRBench, also think off, "drafts". Suggested: "although OCRBench, which carries no grammar, drafts".
2. Two bare citations. "(5881649818)" and "(5824799988)" can't be followed. Link them as pull/412#issuecomment-5881649818 and pull/375#issuecomment-5824799988, as the record does elsewhere.
3. Line 1026's attribution is incomplete. It still says "with their attribution against the 0.34.0 control", and that attribution (5856289579) names MLX and XGrammar but not ADR 0039, which this PR rightly adds.
4. One comment still makes the old claim. 5856289579, linked from the status table, still says item 0's flip is "Not attributable to the build … One run cannot pin 3 of 200 on the build". The fold: upstream v0.34.4 — llama.cpp b11081, MLX 59d600b5, XGrammar 0.2.7 #375 correction (5881931983) corrects only 5875111957.
5. "999 of 1000 drafted" is consistent but unverifiable here. It fits the code (one stats line per request with speculation on, including drafted=0) and the JSON's single cold start. If item 0 was the one request that did not draft, "item 0's flip included" says nothing about drafting, and a clause would say so.

ADR 0039's own "Metal is unaffected in behaviour" line is the CUDA host's to fix, in a separate dated amendment.

ai-server/mlx-cuda