Eval bug: Bonsai 2 PQ2_0 produces corrupt logits on Windows HIP gfx1151; CPU --no-repack is coherent

[ShayGus]

Name and Version

llama-cli version: b10709 (commit 9a9394a89)
Release: prism-b10709-9a9394a
Windows prebuilt: llama-prism-b10709-9a9394a-bin-win-hip-radeon-x64.zip
ROCm userspace: 7.1
HIP runtime loaded: C:\WINDOWS\SYSTEM32\amdhip64_7.dll

A source build from the same 9a9394a commit, compiled specifically for gfx1151, reproduced the failure as well.

Operating system

Windows 11 Pro, build 10.0.26200, x64.

GGML backend

HIP/ROCm (ROCm0). CPU is a useful passing control when repacking is disabled.

Hardware

• AMD Ryzen AI MAX+ 395
• Radeon 8060S, gfx1151 / Strix Halo
• 128 GB unified system memory

Model

prism-ml/Ternary-Bonsai-2-27B-gguf

• File: Ternary-Bonsai-2-27B-PQ2_0.gguf
• Size: 7,206,168,928 bytes
• SHA-256: 3907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1
• Reported type: PQ2_0 - 2.13 bpw (group 128)

The hash matches the current Hugging Face artifact.

Problem description

The exact Prism release required by Bonsai 2 loads the model and runs quickly on HIP, but even a tiny direct llama-cli prompt produces corrupt repetitive punctuation/digits. This is independent of llama-server, chat templates, vision/mmproj, long context, and sampling.

The same binary and GGUF produce coherent tokens on CPU when --no-repack is used. HIP remains corrupt with --no-repack, including when only the 64 transformer blocks are offloaded and the output tensor remains on CPU. This localizes the remaining problem to GPU-executed model layers rather than the output head or GGUF contents.

Minimal HIP reproduction

$env:GGML_CUDA_NO_PINNED = '1'
.\llama-cli.exe `
  -m Ternary-Bonsai-2-27B-PQ2_0.gguf `
  --device ROCm0 -ngl 99 -fa on -c 32768 `
  --no-repack --temp 0 `
  -p 'What is 2 plus 2? Answer with the number only.' `
  -n 32 -st

Observed generation:

[Start thinking]
33333333333333333333333333333333

Prompt: 445.0 t/s | Generation: 178.0 t/s

Removing --no-repack also fails, with punctuation/digit gibberish instead.

CPU passing control

.\llama-cli.exe `
  -m Ternary-Bonsai-2-27B-PQ2_0.gguf `
  -ngl 0 -c 4096 --no-repack --temp 0 `
  -p 'What is 2 plus 2? Answer with the number only.' `
  -n 16 -st

Observed generation begins coherently:

[Start thinking]
We need answer user's request: "What is 2 plus 2?

Prompt: 17.3 t/s | Generation: 3.1 t/s

GPU layers with output tensor left on CPU

-ngl 64 --no-repack still produces the identical repeated-3 corruption, at 46.9 t/s generation. Therefore the failure is not limited to the vocabulary-sized output tensor.

Additional isolation already performed

All of the following still produced corrupt HIP output:

• direct llama-cli rather than llama-server;
• tiny prompt well below n_ubatch;
• --no-repack;
• --no-mmap;
• HIP_LAUNCH_BLOCKING=1;
• GGML_CUDA_NO_PINNED=1;
• output tensor left on CPU via -ngl 64;
• native Windows build from 9a9394a with:
  • -DGGML_HIP=ON
  • -DGPU_TARGETS=gfx1151
  • -DGGML_HIP_NO_VMM=ON
  • -DGGML_CUDA_FORCE_CUBLAS=ON
  • -DGGML_NATIVE=ON

The FORCE_CUBLAS build rules out the PQ2 MMQ kernel as the sole cause. GGML_CUDA_ENABLE_UNIFIED_MEMORY and experimental LLAMA_MMB_*/LLAMA_HC_* flags are not set.

The Vulkan prebuilt detects the 8060S but crashes while loading this PQ2_0 model, so it is not a usable comparison backend here.

Related evidence

• Segfault in CPU repack loading Ternary-Bonsai-2-27B-PQ2_0.gguf (output.weight) #180 reports a PQ2_0 CPU repack crash for this exact model and verifies --no-repack as a CPU workaround. That workaround makes CPU coherent here but does not repair HIP.
• Add community benchmark: Bonsai 2 27B PQ2_0 on RX 7800 XT (gfx1101) —… Bonsai-demo#199 reports the same Bonsai 2 PQ2_0 model and prism-b10709-9a9394a working on ROCm gfx1101, suggesting this failure is gfx1151-specific.
• Eval bug: HIP/ROCm on gfx1151, wrong logits (not a crash), triggered by prompts longer than n_ubatch. ggml-org/llama.cpp#28211 documents silent wrong logits on HIP gfx1151 across other architectures, although its known trigger is prompts longer than n_ubatch; this reproduction fails on a tiny prompt.
• Eval bug: GGML_CUDA_ENABLE_UNIFIED_MEMORY causes garbled output on second token and onwards halo-box/strix-llama.cpp#54 documents another gfx1151 HIP corruption path under unified memory, but unified memory is not enabled here.

Expected result

HIP output should agree semantically with the coherent CPU --no-repack control for the same model, prompt, and greedy sampling.

AI usage disclosure

OpenAI Codex assisted with running the controlled A/Bs, organizing the observed logs, searching for related reports, and drafting this issue. The user explicitly requested publication. No generated source-code patch is included.

[ShayGus]

Correction to the Vulkan statement in the issue body: the initial Vulkan load attempt was not a clean backend comparison. A llama-server.exe HIP process was still resident with all layers and a 262,144-token context on ROCm0 (PID 48360), consuming the shared GPU allocation.

After stopping that verified process and rerunning the matching prism-b10709-9a9394a Windows Vulkan prebuilt from a clean state, the same hash-verified PQ2_0 model loaded with --no-repack and generated the correct answer (4).

This is not useful PQ2_0 Vulkan GPU inference, however. Verbose placement/memory evidence showed:

Vulkan0 model memory:    55 MiB
Host model memory:     6810 MiB
Vulkan0 RS buffer:      149.62 MiB
Vulkan0 compute buffer: 136.11 MiB
Prompt: ~1.5 tokens/s
Generation: ~1.8 tokens/s

The recurrent-state and compute buffers are on Vulkan0, but almost all model weights remain on the host. So the corrected backend matrix is:

• HIP/gfx1151: full GPU execution and fast, but corrupt output.
• Vulkan/gfx1151 with --no-repack: coherent, but effectively CPU-weight execution with limited Vulkan auxiliary buffers.
• CPU with --no-repack: coherent.

The previous claim that Vulkan itself crashed loading PQ2_0 was incorrect; the failure was caused by testing while the full-context HIP server remained resident. Thanks to the user for catching the resident process.

[ShayGus]

Retested the current prism branch after the reported HIP PQ2_0 change landed. Built head ede9f73 natively on Windows for gfx1151 with GGML_HIP=ON, GPU_TARGETS=gfx1151, and GGML_HIP_NO_VMM=ON. The identical direct CLI control still generates 32 repeated 3 tokens with the hash-verified Bonsai 2 27B PQ2_0 file, despite --no-repack and a tiny prompt. Measured 369.8 prompt and 198.3 generation tokens/s. PR #200 improves the HIP PQ2_0 MMQ unpack path and validated runtime correctness on gfx1100, but it does not resolve this gfx1151 correctness failure. No newer binary release exists yet; issue remains reproducible at current branch head.

[bri-prism]

Thanks for the thorough isolation, especially the -ngl 64 and FORCE_CUBLAS runs, which rule out the output head and the PQ2_0 MMQ kernel. We don't have gfx1151 hardware to reproduce this, so the fastest way to narrow it down is the backend op tests, which compare each GPU op against the CPU. Could you run these from a build of the new release prism-b10735-842b188 or current prism and post any lines that fail?

test-backend-ops test -b ROCm0 -o MUL_MAT -p pq2_0
test-backend-ops test -b ROCm0 -o GATED_DELTA_NET
test-backend-ops test -b ROCm0 -o SSM_CONV
test-backend-ops test -b ROCm0 -o RMS_NORM
test-backend-ops test -b ROCm0 -o MUL_MAT_HADAMARD

If all of those pass, a full test-backend-ops test -b ROCm0 run would be the next step.