Report: Ternary kernels work on Volta (sm_70) — token-identical with sm_89, ~50 t/s on a V100

[christopherrobertbrooks-tech]

Not a bug — a datapoint for an architecture outside the tested set (75-virtual;86-real;89-real;120-real). Since the docs describe missing architectures as "untested, not merely slower," here is a tested one.

Result

Ternary-Bonsai-2-27B (PQ2_0) runs correctly on a Tesla V100 (compute 7.0) built from source against CUDA 12.9. Greedy (temp 0, 600 tokens) output is token-identical to the same build running on an RTX 4070 (compute 8.9, inside the tested set) — diff of the two outputs differs only in the loading spinner and the perf banner. Reasoning trace and final answer both coherent.

GPU	compute cap	prompt eval	generation
Tesla V100-PCIE-32GB	7.0	132.5 t/s	50.5 t/s
RTX 4070 (control)	8.9	358.2 t/s	52.9 t/s

Generation speed is effectively at parity (memory-bound; HBM2 holds up). Prefill is ~2.7x faster on Ada.

Environment

• Fork: prism branch, commit 3ae4f51 (version 0.2.0-dev, build 1)
• Model: prism-ml/Ternary-Bonsai-2-27B-gguf — Ternary-Bonsai-2-27B-PQ2_0.gguf (7.2 GB)
• GPU: Tesla V100-PCIE-32GB (Volta, sm_70), driver 580.173.02
• CUDA toolkit 12.9.86 (nvcc off PATH; passed explicitly — see below). Note this route dies with CUDA 13, which dropped Volta support.
• gcc 13.3.0, cmake 3.28.3, Ubuntu 24.04, host: i7-13700KF / 15 GB RAM

Build

cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.9/bin/nvcc \
  -DCMAKE_CUDA_ARCHITECTURES="70;89"
cmake --build build -j 4

Compiled clean — no sm_70-related warnings or errors. Load reports loaded 402 Hadamard-folded weight(s) (1 inverse-lookup) using 1 rotation(s) and 3 sign vector(s); all 65/65 layers offloaded; model buffer 6539.67 MiB.

Test

CUDA_VISIBLE_DEVICES=<n> ./build/bin/llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
  -ngl 99 --temp 0 -n 600 --single-turn \
  -p "List the first five prime numbers, then explain in one sentence why 9 is not prime."

Ran once per card; diffed the outputs. Correct answer on both (2, 3, 5, 7, 11 + a correct sentence about 9 = 3 × 3).

Happy to run additional validation (perplexity run, longer contexts, or PTQ1_0 — untested here) if useful. Filed in case it saves the next person with a Volta card the guesswork; also fine to close if sm_70 remains officially unsupported.

---

Disclosure: testing and this writeup were done with the help of an AI assistant (Claude); all figures are taken directly from the run logs on the machine described above.

[christopherrobertbrooks-tech]

Follow-up on the same machine, taking up the offer above: PTQ1_0 is now tested, and so is the vision tower. Same box, same binary, same build (3ae4f51, CUDA 12.9, -DCMAKE_CUDA_ARCHITECTURES="70;89"), one card visible at a time.

Three things: a Volta row for the throughput table, a counterexample to the "prompt processing favors PQ2_0 everywhere" claim, and confirmation that the mmproj works on sm_70.

First, a correction to my own numbers above. The 132.5 / 358.2 t/s prefill figures in the original post are not comparable to the README's table — they came from llama-cli on a ~20-token prompt, which is far too short to saturate either card. Measured properly with llama-bench, V100 prefill is 766 t/s, not 132. Please disregard the prompt-eval column in the opening post.

Throughput

llama-bench -ngl 99 -fa 1 -p 512 -n 128 -r 5, mean ± stddev:

Card	PQ2_0 TG128	PQ2_0 PP512	PTQ1_0 TG128	PTQ1_0 PP512
Tesla V100-PCIE-32GB (sm_70)	50.97 ± 0.09	766.6 ± 6.1	34.40 ± 0.04	817.3 ± 11.5
RTX 4070 (sm_89, control)	53.86 ± 0.01	1272.4 ± 12.7	54.72 ± 0.04	619.2 ± 3.8

PP512 on Volta favours PTQ1_0, not PQ2_0

The model card states that prompt processing, "being compute-bound, favors PQ2_0 everywhere." On the V100 that inverts: PTQ1_0 is ~6.6% faster at pp512 (817.3 vs 766.6), with non-overlapping error bars. Reproduced with the model order swapped and -r 8: PTQ1_0 814.9 ± 4.8 vs PQ2_0 724.2 ± 30.2.

The 4070 in the same chassis, same binary, same invocation behaves exactly as documented — PQ2_0 wins pp512 by 2.05x, and PTQ1_0 edges decode. So this is not a build or harness artifact; the two architectures genuinely disagree.

Volta is the worst case so far for PTQ1_0 decode

PTQ1_0 reaches 67.5% of PQ2_0 decode on the V100, below the A100's 74.0% (the lowest ratio currently in the table). That is consistent with the README's explanation — trit-unpacking arithmetic loses wherever batch-1 decode is not bandwidth-starved — and extends it to a generation the table does not yet cover. On this hardware PQ2_0 is the clear pick for interactive use despite the larger footprint.

Vision works on sm_70

llama-mtmd-cli with Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf alongside PQ2_0, both fully offloaded to the V100:

• Synthetic shapes and text: exact text transcription and all three shape counts correct.
• A dense five-row table (SKU codes, quantities, prices, footnotes) at --image-min-tokens 1024: every cell correct, including leading zeros (07, 04, 00), plus both values buried in the small print.

Image encode took 158 ms and 394 ms respectively. No sm_70-specific warnings from the vision path.

Minor

chat_template_kwargs: {"reasoning_effort": "none"} raises a Jinja raise_exception and surfaces as an HTTP 500 from llama-server. The template accepts xhigh/medium/low, so this is expected rather than a bug — noting it only because "none" is a common spelling elsewhere and the failure is a hard 500 rather than a fallback to the default.

Environment otherwise as described in the opening post. Happy to run perplexity or long-context numbers on either card if that would be useful.

Disclosure: testing and this writeup were done with the help of an AI assistant (Claude); all figures are taken directly from the run logs on the machine described above.