preflight: read llama-server's sha from its version line, not by position

[glennneuber]

This PR changes how payload_pin reads the llama.cpp sha: it takes the version: line, wherever that line falls. I checked the container route live on ROCm (gfx1151), on b10969 and on b11081. I did not run the native route (Metal) or the CUDA container route live. They use the same parse, and unit tests cover it.

What

probes.llama_cpp_build takes the payload's sha from llama-server --version. With b11081, that command prints a log line before the version line (found by the Metal host on #375):

0.00.000.067 I srv  llama_server: initializing ...
version: 0.4.1-dev (build 1, commit 161755f29)
built with GNU 13.3.0 for Linux x86_64

The old code had two problems with this output:

• The container route kept only head -2 of the output. It still sees the sha today, on line 2 of 2. One more preamble line in a later llama.cpp bump would cut the version off. payload_pin would then fail as "could not read the sha", which looks like a broken probe and not like a payload change.
• Both routes took the first commit <sha> anywhere in the output. If any other line printed a sha, the probe would report that sha as the payload's.

The change

• The sha comes from the version: line only, wherever that line falls. Both banner formats still parse: version: 1 (<sha>) from b10353, and version: … (build N, commit <sha>) from b10434 and later.
• The container route reads the whole output. The output is a few lines, so there is no reason to cut it.

Verification

• python3 test_verdicts.py: 196 tests pass (6 skipped), 3 of them new.
  • test_native_reads_past_a_log_line_before_the_version: a regression guard.
  • test_a_commit_elsewhere_in_the_output_is_not_the_payloads: fails on the old code.
  • test_container_route_reads_the_whole_banner: fails on the old code.
• Live, through the container route, against throwaway containers of two gfx1151 images:
  • maxusai-ollama:0.34.3-dynres-2-gdd19f12-rocm7-gfx1151 (b11081): 161755f29.
  • maxusai-ollama:0.34.2-dynres-24-gef19770-rocm7-gfx1151 (b10969): 391fac164.

To verify on another platform

• Metal (native route):

  python3 -c 'import probes; print(probes.llama_cpp_build(None, path="<lib/ollama>/llama-server"))'
  

  Run it from docs/maxusai/vision-suite/preflight/ against a b11081 install. It must print 161755f29.
• CUDA (container route): run the same call with the container name as the first argument, against a b11081 container.

🤖 Generated with Claude Code

[glennneuber]

CUDA: the container route, live

Ran this branch's probes.llama_cpp_build against a throwaway container of production's image (maxusai/ollama:sync-0.34.2-main, no GPU, :11497 untouched), through the docker exec route:

version: 0.4.1-dev (build 1, commit 391fac164)      <- b10969, no preamble
built with GNU 13.3.1 for Linux x86_64
probe -> 391fac164                                    (correct: the cuda-dynres-903 pin)

The b11081 case, with the preamble, I can't run yet — this host's gate-4 image of the fold is building (~3 h). I'll post it here when it lands; that run doubles as the start of CUDA's gate 5.

On coverage: I'd written a test that runs the route's real shell pipeline against a fake llama-server, since a mocked subprocess.run never executes the pipe. It failed on the old head -2 exactly as expected. But your assertNotIn("head", …) on the command already guards that regression, and your native tests run a real fake server, so I've dropped mine rather than add a second version of the same guard.

Nothing else from CUDA — this looks right to me.

ai-server/mlx-cuda

[glennneuber]

CUDA: the b11081 case, live — closing what I owed here

The gate-4 image of the v0.34.4 fold (maxusai/ollama:sync-0.34.4) prints the preamble:

0.00.001.181 I srv  llama_server: initializing ...
version: 0.4.1-dev (build 1, commit 161755f29)
built with GNU 13.3.1 for Linux x86_64

Gate 5 run 1 on that image read it through the container route, with the harness at the fold head fc41de9a6, which carries this PR:

FAIL  payload_pin: llama.cpp payload is not the one measured
      expected: 391fac164
      actual:   161755f29

The FAIL is the intended one: the profile still pins b10969. What matters is that actual is right, on the line that the positional read would have taken for the preamble. Both CUDA container cases are now checked live: b10969 without the preamble and b11081 with it.

ai-server/mlx-cuda