Independent RTX 3080 measurements + engine-matched quality comparison — Ternary Bonsai 2 27B (PTQ1_0) vs conventional 2-bit of the same base
Hi PrismML team — thanks for Bonsai 2 27B and for keeping KNOWN_ISSUES.md current; it saved this study several dead ends (the CUDA-13.3 crash cluster, reasoning_effort:"high" → 500, and the ngram no-op were all avoided because of it).
We ran an independent reproduction on a consumer Ampere card (RTX 3080 10GB, sm_86), which is absent from the README throughput table, and an engine-matched quality comparison between PTQ1_0 and a conventional 2-bit quant of the same base model (Qwen3.8-27B-UD-IQ2_XXS, unsloth). Everything below is from a self-contained repro package (runners, per-item JSONs, fixture, analysis script) that we can hand over in full.
1. Throughput rows you don't have yet: RTX 3080 (10GB)
llama-bench, fork build prism-b10735-842b188 (Linux CUDA 12.8), batch 1, depth 0, -ngl 99 -fa on; t/s are mean over llama-bench's internal repetitions (SD as printed); size = on-disk file size:
| pack |
size |
pp512 |
tg128 |
| PTQ1_0 (1.75 bpw) |
5.53 GiB |
1210.6 ± 29.8 |
52.18 ± 0.05 |
| PQ2_0 (2.13 bpw) |
6.70 GiB |
1184.5 ± 30.9 |
61.38 ± 0.26 |
| Qwen3.8-27B-UD-IQ2_XXS (2.06 bpw, same base) |
6.76 GiB |
1119.1 ± 32.8 |
44.46 ± 0.05 |
- On consumer Ampere PQ2_0 decodes ~18% faster than PTQ1_0 (61.4 vs 52.2 tg128) at pp parity. That matches your H100/A100/Blackwell ordering, not the RTX 4090/L40S/Ada rows — the sm_86 crossover point vs the Ada parts may be worth a line in the card's table.
- Both ternary packings beat the conventional IQ2_XXS at tg128 while smaller in footprint (52.2/61.4 vs 44.5).
2. Context ladder (PTQ1_0, 10GB card), every step verified with a real generation
16K q8_0 KV → 6571 MiB peak observed VRAM · 32K q8_0 = 7195 MiB (recommended serving config; ≈3,045 MiB of nominal 10-GiB capacity nominally unallocated — not reliably available to other processes) · 96K q4_0 = 8155 MiB · 128K = 8891 MiB · 160K q4_0 = 9627 MiB (highest tested working) · 192K fails at load (compute-buffer OOM). KV is cheap on the hybrid-attention backbone (~16 KV-carrying layers): 32K q8_0 costs only ~630 MiB.
3. Engine-matched quality: PTQ1_0 vs conventional 2-bit of the same base model
Framing: this compares deployable model packages (quantized checkpoint + serving recipe) under
stated settings — it is engine-matched but NOT a weights-matched quantization ablation; the two
packages may also differ in post-quant calibration.
LiveCodeBench v6, 50-problem deterministic fixture (seed 20260812; 14 easy / 14 medium / 22 hard), one sample per problem, temp 0.2, max_tokens 8192, single stream, identical prompt/evaluator for both arms (our runner is a fork of the same evaluator; per-item records included):
| arm |
pass@1 |
hard |
cap-hits (8192) |
| Qwen3.8-27B-UD-IQ2_XXS, Prism fork b10735 |
16/50 (32%) |
0/22 |
34 |
Bonsai 2 PTQ1_0, Prism fork b10735, reasoning_effort: medium |
32/50 (64%) |
6/22 |
18 |
Paired item-level analysis: 16 problems pass on PTQ1_0 and fail on IQ2_XXS, 0 the other direction (exact McNemar p = 3.1e-05, attached to the prespecified 50-item fixture and single-sample runs). The IQ2_XXS 0/22 hard tier occurred in both engine runs (13/50 on stock-llama.cpp+MTP, 16/50 on the fork; the engine switch's +3 is not significant at n=50); the ternary pack passes 6/22 at the same 8192 budget (7/22 at 16384, budgets not cross-comparable).
Two serving-relevant side findings:
reasoning_effort: medium beats the template default (xhigh) at 8192-token budgets: 32/50 vs 21/50, paired 11–0 discordants (exact McNemar p = 9.8e-04), fewer cap-hits. Consistent with your model-card guidance for moderate output limits.- Cap hits are common and budget-dependent: lifting the budget 8192→16384 (32K q8_0 KV fits easily) yields a net +3 problems and cuts cap-hits 18→14 (we report paired gains and losses below in the per-item data; a cap hit does not by itself establish the response would have passed with more tokens). Cap-hits within the both-fail items are a tie (18 vs 18) — the overall cap-hit difference lives entirely on items only PTQ1_0 passes.
4. Pack trade-off is two-sided (quality vs decode)
At identical 8192/medium settings, PQ2_0 scores 28/50 vs PTQ1_0's 32/50 — a four-problem observed quality advantage that is not statistically resolved on this fixture (2-vs-6 discordants) — while PQ2_0's decode-speed advantage (18%) is clear. Pack choice is application-dependent; we do not claim a quality-per-GB ordering from this test.
5. Notes on the runtime (all against b10735)
reasoning_effort accepted values observed: medium / xhigh; "high" → HTTP 500; "low" behaves ≈ xhigh (matches KNOWN_ISSUES).- Probing with
max_tokens ≤ ~64 returns empty content for this thinking model (entire budget goes to reasoning) — worth flagging in the README's quickstart so testers don't misread it as a broken load.
- llama-bench from the fork build handles the custom quant types cleanly (used for the rows above).
The full package (runners, fixture, per-item JSONs, analyze.py that regenerates every table and the McNemar tests, SHA256SUMS) is attached to this issue as bonsai2_3080_repro_20260925.tar.gz. Also happy to run additional configs on the 3080 if useful — it's idle and the card is at your table's disposal.
— Independent repro on ubuntu-server (RTX 3080 10GB, Ubuntu 24.04, driver 595.84), 2026-09-24/25
Independent RTX 3080 measurements + engine-matched quality comparison — Ternary Bonsai 2 27B (PTQ1_0) vs conventional 2-bit of the same base
Hi PrismML team — thanks for Bonsai 2 27B and for keeping KNOWN_ISSUES.md current; it saved this study several dead ends (the CUDA-13.3 crash cluster,
reasoning_effort:"high"→ 500, and the ngram no-op were all avoided because of it).We ran an independent reproduction on a consumer Ampere card (RTX 3080 10GB, sm_86), which is absent from the README throughput table, and an engine-matched quality comparison between PTQ1_0 and a conventional 2-bit quant of the same base model (Qwen3.8-27B-UD-IQ2_XXS, unsloth). Everything below is from a self-contained repro package (runners, per-item JSONs, fixture, analysis script) that we can hand over in full.
1. Throughput rows you don't have yet: RTX 3080 (10GB)
llama-bench, fork build prism-b10735-842b188 (Linux CUDA 12.8), batch 1, depth 0,-ngl 99 -fa on; t/s are mean over llama-bench's internal repetitions (SD as printed); size = on-disk file size:2. Context ladder (PTQ1_0, 10GB card), every step verified with a real generation
16K q8_0 KV → 6571 MiB peak observed VRAM · 32K q8_0 = 7195 MiB (recommended serving config; ≈3,045 MiB of nominal 10-GiB capacity nominally unallocated — not reliably available to other processes) · 96K q4_0 = 8155 MiB · 128K = 8891 MiB · 160K q4_0 = 9627 MiB (highest tested working) · 192K fails at load (compute-buffer OOM). KV is cheap on the hybrid-attention backbone (~16 KV-carrying layers): 32K q8_0 costs only ~630 MiB.
3. Engine-matched quality: PTQ1_0 vs conventional 2-bit of the same base model
Framing: this compares deployable model packages (quantized checkpoint + serving recipe) under
stated settings — it is engine-matched but NOT a weights-matched quantization ablation; the two
packages may also differ in post-quant calibration.
LiveCodeBench v6, 50-problem deterministic fixture (seed 20260812; 14 easy / 14 medium / 22 hard), one sample per problem, temp 0.2, max_tokens 8192, single stream, identical prompt/evaluator for both arms (our runner is a fork of the same evaluator; per-item records included):
reasoning_effort: mediumPaired item-level analysis: 16 problems pass on PTQ1_0 and fail on IQ2_XXS, 0 the other direction (exact McNemar p = 3.1e-05, attached to the prespecified 50-item fixture and single-sample runs). The IQ2_XXS 0/22 hard tier occurred in both engine runs (13/50 on stock-llama.cpp+MTP, 16/50 on the fork; the engine switch's +3 is not significant at n=50); the ternary pack passes 6/22 at the same 8192 budget (7/22 at 16384, budgets not cross-comparable).
Two serving-relevant side findings:
reasoning_effort: mediumbeats the template default (xhigh) at 8192-token budgets: 32/50 vs 21/50, paired 11–0 discordants (exact McNemar p = 9.8e-04), fewer cap-hits. Consistent with your model-card guidance for moderate output limits.4. Pack trade-off is two-sided (quality vs decode)
At identical 8192/medium settings, PQ2_0 scores 28/50 vs PTQ1_0's 32/50 — a four-problem observed quality advantage that is not statistically resolved on this fixture (2-vs-6 discordants) — while PQ2_0's decode-speed advantage (18%) is clear. Pack choice is application-dependent; we do not claim a quality-per-GB ordering from this test.
5. Notes on the runtime (all against b10735)
reasoning_effortaccepted values observed:medium/xhigh;"high"→ HTTP 500;"low"behaves ≈xhigh(matches KNOWN_ISSUES).max_tokens≤ ~64 returns empty content for this thinking model (entire budget goes to reasoning) — worth flagging in the README's quickstart so testers don't misread it as a broken load.The full package (runners, fixture, per-item JSONs,
analyze.pythat regenerates every table and the McNemar tests, SHA256SUMS) is attached to this issue asbonsai2_3080_repro_20260925.tar.gz. Also happy to run additional configs on the 3080 if useful — it's idle and the card is at your table's disposal.— Independent repro on ubuntu-server (RTX 3080 10GB, Ubuntu 24.04, driver 595.84), 2026-09-24/25