Repository navigation
gemma-4-E2B-it: current default cq4.zip (2026-07-09 "calibrated") collapses on multi-step reasoning — GSM8K 3/100 vs 87/100 for the previous build, at identical speed #773
Description
Summary
The current default download for google/gemma-4-E2B-it — Cactus-Compute/gemma-4-E2B-it gemma-4-e2b-it-cq4.zip, the file cactus run/cactus download resolve at --bits 4 — appears to have lost multi-step reasoning in the 2026-07-09 artifact swap. The build it replaced (now gemma-4-e2b-it-cq4-uncalibrated.zip in the same repo) does not have the problem, and the two are speed-identical, so this looks like an unintended artifact regression rather than a speed/quality trade.
Measurements
GSM8K, first 100 test questions, 0-shot CoT, greedy (temperature 0, top_k 1), max_tokens 1024, driven through cactus_complete (Metal backend, auto_handoff: false), same prompt + answer extraction for every build:
| bundle (sha256 prefix) | GSM8K |
|---|---|
cq4.zip (current default, 7a2579ad1089f626…) |
3/100 |
cq4-uncalibrated.zip (7c71f075554bd27b…) |
87/100 |
cq3.26.zip (n=25 probe) |
1/25 |
cq2.54.zip (n=25 probe) |
4/25 |
Supporting observations:
- The two cq4 builds decode at the same speed on an iPhone 17 Pro (50.2 vs 50.6 tok/s, median of 3 cold runs each) — the difference is purely output quality.
- The default build's failures are visible even on single-step questions: in a 9-item sanity prompt it answers 7×8 with "119" and "thank you in French" with "Bonjour"; the uncalibrated build gets 8/8 of its emitted items right.
- On multi-step problems the default build hedges without committing to a reading until the token budget runs out (~4.4k chars mean), the same signature across all 100 questions.
- Your docs (
docs/cactus_quants.md) report GSM8K 71.20 for CQ4 and 74.00 for the production bundle — consistent with what we measure on the uncalibrated lineage, not with the current default. - Timeline from the HF repo history: 2026-07-09 "Add locally-built calibrated CQ4 (GPTQ language, FP16 towers)" + renames made it the default; the 07-16/17 probe additions kept those graphs ("Restore original cq4 graphs; keep cloud-handoff probe").
Repro
Everything is public in https://github.com/john-rocky/apple-silicon-llm-bench:
- harness:
scripts/parity_gsm8k.py(--which cactus, drives your Python FFI; pinned question set inevaldata/gsm8k_test.jsonl) - per-question reports:
results/quality/gsm8k_cactus-*.json - session audit trail incl. device runs:
results/raw/2026-07-20-cactus-arm/SUMMARY.txt
python3 scripts/parity_gsm8k.py --which cactus \
--bundle <unzipped gemma-4-e2b-it-cq4> --cactus-repo <cactus clone> \
--n 100 --max-tokens 1024Engine commit 1ace6d78; results are deterministic under greedy decoding.
If the calibrated swap is intentional, a docs note would help users pick; if not, re-pointing the default at the uncalibrated graphs looks like the smallest fix. Happy to re-run the harness against a fixed artifact.
Activity
clumsylad21 commentedon Aug 12, 2026
Thanks for the detailed repro, @john-rocky. I spent some time trying to reproduce and narrow this down using the exact calibrated (
7a2579ad...) and uncalibrated (7c71f075...) CQ4 artifacts, along with the pinnedapple-silicon-llm-benchGSM8K harness/question set.I used greedy decoding,
max_tokens=1024, thinking disabled, and cloud handoff disabled.On ARM64 CPU, I wasn't able to reproduce the regression:
1ace6d78: calibrated 9/10, uncalibrated 8/10So at least in these CPU runs, the reported calibrated-vs-uncalibrated collapse does not appear; both builds behave similarly and produce generally correct results.
However, with Metal on engine
1ace6d78, I got a clear separation: the uncalibrated build scored 5/5, while the calibrated build scored 1/5.The failed calibrated Metal responses were also much longer (~4.3–4.9k chars) and tended to run on without reaching the correct result, which looks similar to the failure mode described in the issue.
My Metal run was on a GitHub-hosted Apple Silicon macOS runner rather than an iPhone 17 Pro, so the hardware/runtime environment isn't identical, but the backend-dependent behavior seems worth investigating further.
@ncylich @jakmro I'd be interested in digging into this and working toward a fix. Is anyone already working on it, or would it be okay for me to pick this issue up / get assigned?
clumsylad21 commentedon Aug 12, 2026
Update: I was able to narrow this down to the Metal CQ4 batched transform and opened a fix.
cactus_metal_encode_transform_batch()was binding only the firstmatrix's
input_scale_recipbuffer, while the other transform metadatawas already matrix-specific. As a result, matrices 1 and 2 in a batched
transform were incorrectly using matrix 0's reciprocal scaling.
That explains the calibrated/uncalibrated split: the uncalibrated CQ4
weights use unit input scales, so the reuse is effectively harmless,
whereas the calibrated weights have distinct per-matrix scales.
On the same current-main baseline and calibrated Gemma CQ4 Q1 repro:
pred=16,gold=18,chars=4350pred=18,gold=18,chars=1018I also added a targeted Metal regression test with deliberately distinct
reciprocal scales; it passes with the fix:
✓ PASS │ matmul_cq4_metal_distinct_recipPR: https://github.com/cactus-compute/cactus/pull/784
ncylich commentedon Aug 12, 2026
I spoke to Henry and he said it's ok for you to work on it. Thank you so much!
Lmk when the PR is ready and I'll push him to accept it.
clumsylad21 commentedon Aug 12, 2026
Thanks @ncylich , I’ve got the PR ready at — #784. I posted the root cause and validation details in my comment just above as well.
Please review it and lmk if any further improvements or changes are needed.