Name and Version
Reproduced on two llama-server builds as bundled by ollama (we run ollama in production; the repro below removes ollama entirely and drives the bare server):
bundle llama.cpp build ggml
ollama 0.32.14 7e4c0a9 (common_params_print_info: build 1 (7e4c0a9) with GNU 13.3.1) libggml.so.0.20.0
ollama 0.34.1 5d806aa libggml.so.0.23.0, libllama.so.0.4.0
Both bundles ship the same gated_delta_net_cuda<…> kernel set in rocm_v7_2/libggml-hip.so (2,576 matching symbol strings). We have not yet tested a self-built current master for gfx1151/HIP — two independent builds with different ggml versions both exhibit the defect, which argues against a since-fixed regression, but we're aware GDN correctness work has landed on master recently and are happy to run the repro against any build you point us at (or with build guidance for gfx1151 ROCm).
Environment: Linux 7.0.0-31-generic; HIP/ROCm backend rocm_v7_2 (libamdhip64.so.7.2.70201, GGML_BACKEND_PATH=…/rocm_v7_2/libggml-hip.so).
Operating systems
Linux
GGML backends
HIP
Hardware
AMD Radeon 8060S (Strix Halo, gfx1151), integrated; AMD Ryzen AI MAX+ 395; 122 GiB RAM; amdgpu.gttsize=126976 (ROCm0 reports 126,976 MiB, ~101 GiB free at load)
Models
qwen3.6:35b-a3b (official Qwen3.6-35B-A3B, Apache-2.0 — ollama run qwen3.6:35b-a3b; also on Hugging Face) — arch qwen35moe, hybrid GDN + attention MoE, Q4_K_M, ollama blob sha256-f5ee307a2982106a6eb82b62b2c00b575c9072145a759ae4660378acda8dcf2d
qwen3.8:27b (official Qwen3.8-27B, Apache-2.0 — Hugging Face Qwen/Qwen3.8-27B) — arch qwen35, dense hybrid GDN + attention, Q4_K_M, ollama blob sha256-f5f1dd8920d417aac2718b0bda3403da274301efdd6760b4f0f4b864ff2ad57d
Problem description & steps to reproduce
Summary: with the fused Gated Delta Net op enabled on HIP, a reused llama-server slot carries the previous request's recurrent (GDN) state into the next request. The leak survives both memory paths a reused slot can take — context-checkpoint restore and memory_seq_rm [0, end) with a full re-process from position 0. The visible symptom is severe: the model emits the previous request's document text, verbatim, as its answer to the current request. Deterministic at temperature 0. On longer warm-runner sequences the carried state accumulates until generation degrades into unterminated output.
This has a privacy implication worth naming: on any deployment where one server slot serves multiple users' requests in sequence, one requester's prompt content can appear in another requester's completion.
Minimal repro (one bare llama-server, one slot, three /completion requests, synthetic documents):
Both models are stock public releases; no custom weights or Modelfiles are involved — the probe sets -ngl explicitly, so it forces the fused-on condition on a default pull. Scripts at https://github.com/jgoellermaximus/gdn-hip-leak-repro (also attached as a zip): rc3/leak_probe2.py (driver + proof capture) and rc3/synth_docs.py — three deterministic synthetic documents with disjoint vocabularies ("Harbor Lantern Charter" / "Velvet Gantry Ledger Rules" / "Ember Rampart Review Procedure"), so any leak is attributable by inspection.
export LD_LIBRARY_PATH=:/rocm_v7_2 GGML_BACKEND_PATH=/rocm_v7_2/libggml-hip.so
python rc3/leak_probe2.py --blob <qwen3.6-35b-a3b.gguf> --ngl 42 --out fused_on # leaks
python rc3/leak_probe2.py --blob <qwen3.6-35b-a3b.gguf> --ngl 41 --out fused_off # does not leak
The driver starts llama-server --model --port 18080 --host 127.0.0.1 --no-webui --offline -c 24576 -np 1 --log-verbosity 4 --load-mode dio --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on -b 1024 -ub 1024 --context-shift --keep 4 -ngl N, waits for /health, then posts three /completion requests (temperature 0, seed 7, cache_prompt true, n_predict 300), ChatML with a think-off prefill, and a generic system prompt S ("extractive one-line abstract + one verbatim quote, JSON only"):
request prompt tokens server path (from the verbose log) expected observed, fused ON (-ngl 42)
R1 S + document A 1,055 fresh slot, memory_seq_rm [0, end); a context checkpoint lands at pos 30 (inside S) abstract + quote of A correct
R2 S + document B1 (same S prefix) 4,270 restored context checkpoint (pos_min = 30, pos_max = 30, n_tokens = 31, n_past = 31) → B1 processed from pos 31 abstract + quote of B1 abstract of A + a verbatim 40-char clause of A
R3 + S + document B2 (prefix broken on purpose) 3,268 checkpoint checks fail → cached n_tokens = 0, memory_seq_rm [0, end), full re-process from position 0 abstract + quote of B2 still A's abstract + a verbatim A clause
Nothing in R2/R3's prompts contains A's vocabulary. Outputs are identical across passes.
Control — same binary, same flags, -ngl 41 (layer 0 on CPU): the resolver prints fused Gated Delta Net (autoregressive) not supported, set to disabled; fused Gated Delta Net (chunked) not supported, set to disabled and the leak is gone (R3 returns the correct B2 abstract). Layer-0-on-CPU is the only available switch for the op. Placement itself is not the variable: moving 24 layers' experts to host memory with the op still enabled leaves the leak signature identical, while moving only layer 0 (793 MiB) with the op disabled is clean.
Same result on the dense sibling: qwen3.8:27b at -ngl 66 (all layers, both fused paths enabled) leaks on R2 and R3; -ngl 65 (both paths disabled) does not.
Observed vs expected:
Expected: each /completion on a slot is independent once the slot's memory is cleared (memory_seq_rm [0, end)) or restored to a checkpoint predating the previous document; the hybrid model's recurrent (GDN) state should be re-initialised from the zeroed state on every such request.
Observed: with the fused GDN op enabled on HIP, the previous request's recurrent state persists — the first ubatch of R2/R3 reads state written by R1. On longer sequences (29 unrelated documents on one warm runner, -ub 1024) the carried state accumulates until generation degrades into unterminated JSON; ubatch size moves the onset (-ub 512: ≈15 requests; -ub 1024: 2–3), not the mechanism.
Isolation: graph reuse, HIP graphs, GGML_CUDA_DISABLE_FUSION (does not reach this op on the 0.32.14 build), expert placement, --ctx-checkpoints 0, --cache-ram 0, --context-shift, f16 KV cache, and parallel-slot settings were each varied one at a time across 22 single-variable server configurations: the leak persists in all of them; only disabling the fused op clears it.
Where the state physically persists inside the fused op (a persistent workspace in ggml-cuda/gated_delta_net.cu, or the fused kernel reading the RS cell directly and bypassing the zeroed input) we could not pin from outside; the repro is intended to make that a one-sitting job for someone with the kernel in front of them.
Possibly related. Same models and operator, different symptoms (throughput/output degradation, not cross-request state persistence): #27623 (Qwen3.8-27B hybrid GDN — decode-throughput collapse past ~80K context, CUDA) and #27164 (Qwen3.8-27B garbage output traced to the CUDA DeltaNet/Gated-DeltaNet layers); #27623 also references #27442 (empty generation >16K on Metal for qwen35moe hybrids) and #27572 (draft-acceptance collapse with multi-ubatch batches) in the same family. Same kernel family on other backends, crash-class: #21383 (Qwen3.5-27B illegal memory access in the prompt-cache path with fused GDN enabled, suspected in the memory_seq_rm partial-truncation fallback for recurrent state — CUDA) and #26546 (Compute Sanitizer out-of-bounds read in gated_delta_net_cuda on Pascal).
Workaround in use and its measured cost: we re-created the affected tags with layer 0 on CPU (N−1 GPU layers) so resolve_fused_ops disables both fused GDN paths on every load. Measured on qwen3.6:35b-a3b (29-document probe, one warm runner, identical flags): prompt 0.89 → 1.10–1.12 ms/tok (+24%), decode 18.3 → 23.0–23.7 ms/tok (+26–29%). The alternative (fused op + fresh server per request) breaks even at ≈850 decode tokens/request given a 5.8 s reload.
Additional observations (companion, possibly separate issues):
On the 0.34.1 bundle, layer-0-on-CPU no longer fully clears it: a residual, deterministic 1/29 leak remains (request 14 quotes request 13's document verbatim, identical on two runs), and that build logs no fused Gated Delta Net resolver lines for these models while decoding at the fused speed — consistent with the GDN fusion having moved to a generic subgraph-fusion path that the layer-0 placement rule no longer gates. For that build we claim "defect present" only, without kernel attribution.
MTP speculative decoding (--spec-type draft-mtp, qwen3.8:27b): the draft context's resolve_fused_ops enables the fused GDN op even when the target context has it disabled; the 3-request probe does not leak on this path even fully fused, so the draft state's effect is untested.
Side effect of the unfused graph on the 0.32.14 build: with a think-off prefill the model returns EOS as its first token on a prompt-dependent subset of free-text prompts (19/24 synthetic prompts; never with the op enabled, never under a JSON grammar, 0/24 on the 0.34.1 build) — noted as a companion observation, not the same defect.
What "fixed" looks like from our side (verification we can run on request): a build that passes rc3/leak_probe2.py with the fused op ON (all layers on GPU) twice on both models, plus a 29-document warm-runner sequence with 0 cross-document leaks.
First Bad Commit
Unknown — present in both builds we can run (7e4c0a9, 5d806aa); we cannot currently bisect on this hardware (no self-built gfx1151 HIP toolchain stood up yet).
Relevant log output
fused ON (-ngl 42), qwen3.6:35b-a3b — resolver + the two memory paths:
fused Gated Delta Net (autoregressive) ... enabled
fused Gated Delta Net (chunked) ... enabled
offloaded 42/42 layers to GPU
R2:
restored context checkpoint (pos_min = 30, pos_max = 30, n_tokens = 31, n_past = 31)
R3:
checking checkpoint with [30, 30] against 3…
cached n_tokens = 0, memory_seq_rm [0, end)
control (-ngl 41):
fused Gated Delta Net (autoregressive) not supported, set to disabled
fused Gated Delta Net (chunked) not supported, set to disabled
offloaded 41/42 layers to GPU
Full verbose server logs, request/response JSON with the leak fingerprints marked, and /proc//exe + .so-map proofs for the three-request repro runs (fused-ON and the -ngl 41 / -ngl 65 controls) are in the repro repository (https://github.com/jgoellermaximus/gdn-hip-leak-repro, SHA256SUMS included) and the attached zip. The 29-document warm-runner results are summarized above (logs held privately, available to maintainers on request).
Relevant log output
fused ON (-ngl 42), qwen3.6:35b-a3b — resolver + the two memory paths:
fused Gated Delta Net (autoregressive) ... enabled
fused Gated Delta Net (chunked) ... enabled
offloaded 42/42 layers to GPU
R2:
restored context checkpoint (pos_min = 30, pos_max = 30, n_tokens = 31, n_past = 31)
R3:
checking checkpoint with [30, 30] against 3…
cached n_tokens = 0, memory_seq_rm [0, end)
control (-ngl 41):
fused Gated Delta Net (autoregressive) not supported, set to disabled
fused Gated Delta Net (chunked) not supported, set to disabled
offloaded 41/42 layers to GPU
Full verbose server logs, request/response JSON with the leak fingerprints marked, and /proc//exe + .so-map proofs for the three-request repro runs (fused-ON and the -ngl 41 / -ngl 65 controls) are in the repro repository (https://github.com/jgoellermaximus/gdn-hip-leak-repro, SHA256SUMS included) and the attached zip. The 29-document warm-runner results are summarized above (logs held privately, available to maintainers on request).
gdn-hip-leak-repro.zip
DRAFT1_llamacpp_gdn_leak_v1_2.md
Name and Version
Reproduced on two llama-server builds as bundled by ollama (we run ollama in production; the repro below removes ollama entirely and drives the bare server):
bundle llama.cpp build ggml
ollama 0.32.14 7e4c0a9 (common_params_print_info: build 1 (7e4c0a9) with GNU 13.3.1) libggml.so.0.20.0
ollama 0.34.1 5d806aa libggml.so.0.23.0, libllama.so.0.4.0
Both bundles ship the same gated_delta_net_cuda<…> kernel set in rocm_v7_2/libggml-hip.so (2,576 matching symbol strings). We have not yet tested a self-built current master for gfx1151/HIP — two independent builds with different ggml versions both exhibit the defect, which argues against a since-fixed regression, but we're aware GDN correctness work has landed on master recently and are happy to run the repro against any build you point us at (or with build guidance for gfx1151 ROCm).
Environment: Linux 7.0.0-31-generic; HIP/ROCm backend rocm_v7_2 (libamdhip64.so.7.2.70201, GGML_BACKEND_PATH=…/rocm_v7_2/libggml-hip.so).
Operating systems
Linux
GGML backends
HIP
Hardware
AMD Radeon 8060S (Strix Halo, gfx1151), integrated; AMD Ryzen AI MAX+ 395; 122 GiB RAM; amdgpu.gttsize=126976 (ROCm0 reports 126,976 MiB, ~101 GiB free at load)
Models
qwen3.6:35b-a3b (official Qwen3.6-35B-A3B, Apache-2.0 — ollama run qwen3.6:35b-a3b; also on Hugging Face) — arch qwen35moe, hybrid GDN + attention MoE, Q4_K_M, ollama blob sha256-f5ee307a2982106a6eb82b62b2c00b575c9072145a759ae4660378acda8dcf2d
qwen3.8:27b (official Qwen3.8-27B, Apache-2.0 — Hugging Face Qwen/Qwen3.8-27B) — arch qwen35, dense hybrid GDN + attention, Q4_K_M, ollama blob sha256-f5f1dd8920d417aac2718b0bda3403da274301efdd6760b4f0f4b864ff2ad57d
Problem description & steps to reproduce
Summary: with the fused Gated Delta Net op enabled on HIP, a reused llama-server slot carries the previous request's recurrent (GDN) state into the next request. The leak survives both memory paths a reused slot can take — context-checkpoint restore and memory_seq_rm [0, end) with a full re-process from position 0. The visible symptom is severe: the model emits the previous request's document text, verbatim, as its answer to the current request. Deterministic at temperature 0. On longer warm-runner sequences the carried state accumulates until generation degrades into unterminated output.
This has a privacy implication worth naming: on any deployment where one server slot serves multiple users' requests in sequence, one requester's prompt content can appear in another requester's completion.
Minimal repro (one bare llama-server, one slot, three /completion requests, synthetic documents):
Both models are stock public releases; no custom weights or Modelfiles are involved — the probe sets -ngl explicitly, so it forces the fused-on condition on a default pull. Scripts at https://github.com/jgoellermaximus/gdn-hip-leak-repro (also attached as a zip): rc3/leak_probe2.py (driver + proof capture) and rc3/synth_docs.py — three deterministic synthetic documents with disjoint vocabularies ("Harbor Lantern Charter" / "Velvet Gantry Ledger Rules" / "Ember Rampart Review Procedure"), so any leak is attributable by inspection.
export LD_LIBRARY_PATH=:/rocm_v7_2 GGML_BACKEND_PATH=/rocm_v7_2/libggml-hip.so
python rc3/leak_probe2.py --blob <qwen3.6-35b-a3b.gguf> --ngl 42 --out fused_on # leaks
python rc3/leak_probe2.py --blob <qwen3.6-35b-a3b.gguf> --ngl 41 --out fused_off # does not leak
The driver starts llama-server --model --port 18080 --host 127.0.0.1 --no-webui --offline -c 24576 -np 1 --log-verbosity 4 --load-mode dio --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on -b 1024 -ub 1024 --context-shift --keep 4 -ngl N, waits for /health, then posts three /completion requests (temperature 0, seed 7, cache_prompt true, n_predict 300), ChatML with a think-off prefill, and a generic system prompt S ("extractive one-line abstract + one verbatim quote, JSON only"):
request prompt tokens server path (from the verbose log) expected observed, fused ON (-ngl 42)
R1 S + document A 1,055 fresh slot, memory_seq_rm [0, end); a context checkpoint lands at pos 30 (inside S) abstract + quote of A correct
R2 S + document B1 (same S prefix) 4,270 restored context checkpoint (pos_min = 30, pos_max = 30, n_tokens = 31, n_past = 31) → B1 processed from pos 31 abstract + quote of B1 abstract of A + a verbatim 40-char clause of A
R3 + S + document B2 (prefix broken on purpose) 3,268 checkpoint checks fail → cached n_tokens = 0, memory_seq_rm [0, end), full re-process from position 0 abstract + quote of B2 still A's abstract + a verbatim A clause
Nothing in R2/R3's prompts contains A's vocabulary. Outputs are identical across passes.
Control — same binary, same flags, -ngl 41 (layer 0 on CPU): the resolver prints fused Gated Delta Net (autoregressive) not supported, set to disabled; fused Gated Delta Net (chunked) not supported, set to disabled and the leak is gone (R3 returns the correct B2 abstract). Layer-0-on-CPU is the only available switch for the op. Placement itself is not the variable: moving 24 layers' experts to host memory with the op still enabled leaves the leak signature identical, while moving only layer 0 (793 MiB) with the op disabled is clean.
Same result on the dense sibling: qwen3.8:27b at -ngl 66 (all layers, both fused paths enabled) leaks on R2 and R3; -ngl 65 (both paths disabled) does not.
Observed vs expected:
Expected: each /completion on a slot is independent once the slot's memory is cleared (memory_seq_rm [0, end)) or restored to a checkpoint predating the previous document; the hybrid model's recurrent (GDN) state should be re-initialised from the zeroed state on every such request.
Observed: with the fused GDN op enabled on HIP, the previous request's recurrent state persists — the first ubatch of R2/R3 reads state written by R1. On longer sequences (29 unrelated documents on one warm runner, -ub 1024) the carried state accumulates until generation degrades into unterminated JSON; ubatch size moves the onset (-ub 512: ≈15 requests; -ub 1024: 2–3), not the mechanism.
Isolation: graph reuse, HIP graphs, GGML_CUDA_DISABLE_FUSION (does not reach this op on the 0.32.14 build), expert placement, --ctx-checkpoints 0, --cache-ram 0, --context-shift, f16 KV cache, and parallel-slot settings were each varied one at a time across 22 single-variable server configurations: the leak persists in all of them; only disabling the fused op clears it.
Where the state physically persists inside the fused op (a persistent workspace in ggml-cuda/gated_delta_net.cu, or the fused kernel reading the RS cell directly and bypassing the zeroed input) we could not pin from outside; the repro is intended to make that a one-sitting job for someone with the kernel in front of them.
Possibly related. Same models and operator, different symptoms (throughput/output degradation, not cross-request state persistence): #27623 (Qwen3.8-27B hybrid GDN — decode-throughput collapse past ~80K context, CUDA) and #27164 (Qwen3.8-27B garbage output traced to the CUDA DeltaNet/Gated-DeltaNet layers); #27623 also references #27442 (empty generation >16K on Metal for qwen35moe hybrids) and #27572 (draft-acceptance collapse with multi-ubatch batches) in the same family. Same kernel family on other backends, crash-class: #21383 (Qwen3.5-27B illegal memory access in the prompt-cache path with fused GDN enabled, suspected in the memory_seq_rm partial-truncation fallback for recurrent state — CUDA) and #26546 (Compute Sanitizer out-of-bounds read in gated_delta_net_cuda on Pascal).
Workaround in use and its measured cost: we re-created the affected tags with layer 0 on CPU (N−1 GPU layers) so resolve_fused_ops disables both fused GDN paths on every load. Measured on qwen3.6:35b-a3b (29-document probe, one warm runner, identical flags): prompt 0.89 → 1.10–1.12 ms/tok (+24%), decode 18.3 → 23.0–23.7 ms/tok (+26–29%). The alternative (fused op + fresh server per request) breaks even at ≈850 decode tokens/request given a 5.8 s reload.
Additional observations (companion, possibly separate issues):
On the 0.34.1 bundle, layer-0-on-CPU no longer fully clears it: a residual, deterministic 1/29 leak remains (request 14 quotes request 13's document verbatim, identical on two runs), and that build logs no fused Gated Delta Net resolver lines for these models while decoding at the fused speed — consistent with the GDN fusion having moved to a generic subgraph-fusion path that the layer-0 placement rule no longer gates. For that build we claim "defect present" only, without kernel attribution.
MTP speculative decoding (--spec-type draft-mtp, qwen3.8:27b): the draft context's resolve_fused_ops enables the fused GDN op even when the target context has it disabled; the 3-request probe does not leak on this path even fully fused, so the draft state's effect is untested.
Side effect of the unfused graph on the 0.32.14 build: with a think-off prefill the model returns EOS as its first token on a prompt-dependent subset of free-text prompts (19/24 synthetic prompts; never with the op enabled, never under a JSON grammar, 0/24 on the 0.34.1 build) — noted as a companion observation, not the same defect.
What "fixed" looks like from our side (verification we can run on request): a build that passes rc3/leak_probe2.py with the fused op ON (all layers on GPU) twice on both models, plus a 29-document warm-runner sequence with 0 cross-document leaks.
First Bad Commit
Unknown — present in both builds we can run (7e4c0a9, 5d806aa); we cannot currently bisect on this hardware (no self-built gfx1151 HIP toolchain stood up yet).
Relevant log output
fused ON (-ngl 42), qwen3.6:35b-a3b — resolver + the two memory paths:
fused Gated Delta Net (autoregressive) ... enabled
fused Gated Delta Net (chunked) ... enabled
offloaded 42/42 layers to GPU
R2:
restored context checkpoint (pos_min = 30, pos_max = 30, n_tokens = 31, n_past = 31)
R3:
checking checkpoint with [30, 30] against 3…
cached n_tokens = 0, memory_seq_rm [0, end)
control (-ngl 41):
fused Gated Delta Net (autoregressive) not supported, set to disabled
fused Gated Delta Net (chunked) not supported, set to disabled
offloaded 41/42 layers to GPU
Full verbose server logs, request/response JSON with the leak fingerprints marked, and /proc//exe + .so-map proofs for the three-request repro runs (fused-ON and the -ngl 41 / -ngl 65 controls) are in the repro repository (https://github.com/jgoellermaximus/gdn-hip-leak-repro, SHA256SUMS included) and the attached zip. The 29-document warm-runner results are summarized above (logs held privately, available to maintainers on request).
Relevant log output
fused ON (-ngl 42), qwen3.6:35b-a3b — resolver + the two memory paths:
fused Gated Delta Net (autoregressive) ... enabled
fused Gated Delta Net (chunked) ... enabled
offloaded 42/42 layers to GPU
R2:
restored context checkpoint (pos_min = 30, pos_max = 30, n_tokens = 31, n_past = 31)
R3:
checking checkpoint with [30, 30] against 3…
cached n_tokens = 0, memory_seq_rm [0, end)
control (-ngl 41):
fused Gated Delta Net (autoregressive) not supported, set to disabled
fused Gated Delta Net (chunked) not supported, set to disabled
offloaded 41/42 layers to GPU
Full verbose server logs, request/response JSON with the leak fingerprints marked, and /proc//exe + .so-map proofs for the three-request repro runs (fused-ON and the -ngl 41 / -ngl 65 controls) are in the repro repository (https://github.com/jgoellermaximus/gdn-hip-leak-repro, SHA256SUMS included) and the attached zip. The 29-document warm-runner results are summarized above (logs held privately, available to maintainers on request).
gdn-hip-leak-repro.zip
DRAFT1_llamacpp_gdn_leak_v1_2.md