Skip to content

preflight: pin the MLX payload for the CUDA dynres profile - #294

Closed
glennneuber wants to merge 2 commits into
mainfrom
docs/preflight-pin-mlx-build
Closed

glennneuber wants to merge 2 commits into
mainfrom
docs/preflight-pin-mlx-build

Conversation

@glennneuber

Copy link
Copy Markdown

cuda-dynres-903 pins llama_cpp_build but carried no mlx_build, so mlx_payload_pin skipped on every CUDA preflight run: "profile records no mlx_build to assert against". These images ship mlx_cuda_v13 beside the llama.cpp payload, so the profile serves both stacks.

That is how the v0.34.0 fold's MLX bump (37c26e57 → ce916dbb) reached a preflight PASS with the MLX identity unasserted.

This PR

  • pins mlx_build to MLX_VERSION at main a523d60b, the payload these expectations were measured on;
  • widens test_every_measured_mlx_profile_pins_its_mlx_build, which only enforced profiles whose platform starts with mlx, to cover a profile that serves the MLX payload whatever its platform says.

Checked: python3 test_verdicts.py 102 pass, python3 test_summarizers.py 118 pass. With the pin removed the widened test fails, naming cuda-dynres-903 and saying mlx_payload_pin would skip.

Consequence, deliberately: an image whose MLX differs from the pin now fails this check instead of inheriting stale expectations. The v0.34.0 fold branch moves the pin to ce916dbb as part of its bump, with the preflight re-run on that payload.

🤖 Generated with Claude Code

cuda-dynres-903 pins llama_cpp_build but carried no mlx_build, so
mlx_payload_pin skipped on every CUDA run: "profile records no
mlx_build to assert against". These images ship mlx_cuda_v13 beside
llama.cpp's payload, so the profile serves both stacks, and the
v0.34.0 fold's MLX bump (37c26e57 -> ce916dbb) reached a preflight
PASS with the MLX identity unasserted.

Pin it to MLX_VERSION at main a523d60, the payload these expectations
were measured on, and widen
test_every_measured_mlx_profile_pins_its_mlx_build so a profile that
serves the MLX payload is enforced whatever its platform says. Without
the pin that test now fails naming cuda-dynres-903 (checked).

When the MLX pin moves, re-measure first and move this identity in the
same change, as the llama_cpp_build re-baselines above it do. Until
then a bumped payload fails the check loudly, which is the point.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
glennneuber added a commit that referenced this pull request Sep 12, 2026
gate-sync034h's preflight passed with mlx_payload_pin SKIPPED, because
cuda-dynres-903 recorded no mlx_build although these images ship
mlx_cuda_v13. So the fold's own MLX bump went unasserted.

Pin it to this fold's MLX_VERSION, ce916dbb, and widen the profile test
to enforce a pin on any profile that serves the MLX payload whatever its
platform says. PR #294 adds the same key on main with a523d60's
37c26e57; the merge keeps this value, since the payload here is the
bumped one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Pinning mlx_build on a CUDA profile turned the check's SKIP into a
failed run: "no MLX engine-init line in the log window". The check
reads the MLX build from the runner's engine-init line, and a CUDA
preflight loads llama.cpp models, so no MLX runner ever starts. The
payload was exactly the pinned one; nothing was wrong but the evidence.

Give it a second source. probes.mlx_build_payload reads the version
string out of the shipped libmlx.so, which is what llama_cpp_build
already does for llama-server: the payload's own identity. The
engine-init line stays first, because only a live load can catch the
binary/payload skew this check was written for, and the summary now
names which source answered.

Measured on maxusai/ollama:sync-0.34.0, where the failing run came
from.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
glennneuber added a commit that referenced this pull request Sep 12, 2026
The fold's own preflight is the run that exposed this: pinning
mlx_build on cuda-dynres-903 made gate-sync034h's preflight FAIL with
"no MLX engine-init line in the log window", because a CUDA preflight
loads llama.cpp models and no MLX runner starts.

Take the fix from #294: read the version string out of the shipped
libmlx.so when no engine-init line appears, keeping the live line as
the first source. The pin here stays ce916dbb, this fold's MLX.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@glennneuber

Copy link
Copy Markdown
Author

Pushed a follow-up, 47c8156, after running the pinned profile against a real image.

What the run showed. Pinning mlx_build on a CUDA profile turns the check's SKIP into a failed run:

FAIL  mlx_payload_pin: no MLX engine-init line in the log window
FAIL=1  PASS=20  SKIP=7   VERDICT: FAIL

The check reads the MLX build from the runner's engine-init line, and a CUDA preflight loads llama.cpp models, so no MLX runner ever starts. The payload was exactly the pinned one — nothing was wrong but the evidence. As written, this PR would have turned every CUDA preflight red, the deploy candidate's included.

The follow-up. probes.mlx_build_payload reads the version string out of the shipped libmlx.so, which is what llama_cpp_build already does for llama-server: the payload's own identity rather than the version string of whatever is checked out. The engine-init line stays the first source, because only a live load can catch the binary/payload skew this check was written for, and the summary now names which source answered.

Four tests cover it, including the failing case above and that a live line still wins over the library. With the fallback disabled, the CUDA-run test fails (checked). python3 test_verdicts.py 106 pass.

@glennneuber

Copy link
Copy Markdown
Author

Superseded by the v0.34.0 fold, #297, now merged as 19012a41.

Everything this PR proposed is on main, carried by two commits on the fold branch:

  • 41e90fa4 — mlx_build added to [profiles.cuda-dynres-903], pinned to the fold's own MLX, ce916dbb, rather than this PR's 37c26e57.
  • cbe0e9b7 — probes.mlx_build_payload() and the check_mlx_payload_pin fallback that reads the version out of the shipped libmlx.so, with the live engine-init line still preferred because only it catches binary/payload skew. Without that fallback the pin turned the CUDA preflight's SKIP into a FAIL, since a CUDA preflight never loads MLX.

Verified on the deploy candidate: PASS 21, SKIP 7, pin asserted. Closing as redundant, not as rejected — no work is lost.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant