gate: production's KV cache back to f16 — it had drifted to q8_0 since 2026-08-08

[glennneuber]

This PR records a production fix on the gfx1151 host (amd-server): its KV cache is f16 again. It changes docs and the ROCm gate tooling's prod environment only, and no code. The CUDA and Metal hosts are not measured here. Their agents may want to check what their own OLLAMA_KV_CACHE_TYPE is.

What happened

ADR 0005 traced qwen3.6's think-mode runaway on grounding prompts to a q8_0 KV cache, and production was recreated with f16 by hand on 2026-08-03. That did not hold:

• The deployment's compose file (MaxusAI/ollama-deployments, docker/ollama-rocm/docker-compose.yml) had said q8_0 since its first commit.
• The 2026-08-08 cutover went back through compose, so production returned to q8_0. The 2026-08-13 and 2026-08-17 deploys used compose too.
• The 0.34.2 and 0.34.3 promotions copied the running container's arguments, and q8_0 with them.

The v0.34.4 fold's think-on protocol (#375) found it. qwen3.6 bbox_contract_real_1img did not finish at the ladder's top (131072): its thinking repeated 35 distinct lines through the second half. The llama-server command line had --cache-type-k q8_0 --cache-type-v q8_0.

What changed

• Production (done, 2026-09-26 07:23:52). ollama-rocm was recreated with OLLAMA_KV_CACHE_TYPE=f16, on the same image with every other argument unchanged. It was down for about one second. The q8_0 container is kept as ollama-rocm-0.34.3-q8kv.
• MaxusAI/ollama-deployments 31923a9. The compose file says f16, with the reason in a comment.
• This PR:
  • amd-upgrade-gate.md gets a KV cache row in the status table, and the 2026-09-26 decision with the history and the rollback;
  • ADR 0005 gets an addendum;
  • tasks/rocm-gate/gatelib.py sets PROD_ENV to f16, so the next gate run reproduces production.

Open

• The 0.34.2 rollback container still says q8_0. Starting it, as the 2026-09-25 rollback does, would bring q8_0 back. The doc says to roll back to 0.34.2 with a new container and f16 until it is recreated. Recreating it waits on the maintainer's word.
• qwen3.6 think-on under the protocol. The q8_0 pair finishes today. An f16 pair is queued after it, and its results go to fold: upstream v0.34.4 — llama.cpp b11081, MLX 59d600b5, XGrammar 0.2.7 #375.

Verification

• docker inspect ollama-rocm shows OLLAMA_KV_CACHE_TYPE=f16.
• A one-token load of a small model on production logged --cache-type-k f16 --cache-type-v f16 --flash-attn on, and the model unloaded straight away.
• check_source_paths.py --changed-since origin/main is clean.

amd-server/rocm-gfx1151

🤖 Generated with Claude Code