Skip to content

docs(binaries): archive the payload, not just the binary - #307

Merged
glennneuber merged 3 commits into
mainfrom
docs/binaries-0340-and-payload-archive
Sep 18, 2026
Merged

glennneuber merged 3 commits into
mainfrom
docs/binaries-0340-and-payload-archive

Conversation

@glennneuber

Copy link
Copy Markdown

BINARIES.md has said since 2026-08-30 that "a binary is not a rollback on its own" — the MLX library is dlopen'd from build/lib/ollama and the binary does not carry it. The "Adding one" procedure never acted on that: it archived the binary alone.

So every row in the table was a one-way door in practice. The payload for each archived build existed in exactly one place — the checkout's build/lib/ollama — and was destroyed by the next build that moved a pin.

Why this stopped being theoretical

Staging 0.34.0-maxusai-8a7ba949 on :11435 meant overwriting the only copy of the 0.33.2 payload. Rolling back would have been a ~25 minute cold rebuild at 2b95b4a5 — itself possible only because Xcode's separately-downloadable Metal toolchain happened to be installed this morning. Without that, the rollback path was closed entirely.

Both pairings are now archived beside the binaries:

~/.ollama/binaries/payload-0.33.2-maxusai-2b95b4a5/
~/.ollama/binaries/payload-0.34.0-maxusai-8a7ba949/

Rolling between those two is now a copy. The rows above 0.33.2 still need a checkout and rebuild, and the section says so rather than implying otherwise.

Changes

  • "Adding one" archives both halves, and archives the outgoing pairing before overwriting it
  • "A binary is not a rollback on its own" records which rows are now copy-restorable and which still need a rebuild
  • New row for 0.34.0-maxusai-8a7ba949 — llama.cpp b10864, MLX d9add9d1, sha256 867ee5d6…
  • The 0.33.2 row loses "current", which it no longer is

On the new row's wording

It says built and staged, NOT yet serving. Preflight is PASS 19/12 against the mlx-metal-0-34-0 profile (#306), and #301's whitespace bound is confirmed behaviourally on real documents across the MLX pin move — but the :11435 kickstart has not run, and the row should not claim a deployment that hasn't happened. It needs updating to "current" once it does.

🤖 Generated with Claude Code

glennneuber and others added 3 commits September 18, 2026 12:50
BINARIES.md has said since 2026-08-30 that "a binary is not a rollback on its
own" — the MLX library is dlopen'd from build/lib/ollama and the binary does
not carry it. The "Adding one" procedure never acted on that: it archived the
binary alone. So every row in the table was a one-way door in practice. The
payload for each archived build existed in exactly one place, the checkout's
build/lib/ollama, and was destroyed by the next build that moved a pin.

That became concrete today. Staging 0.34.0-maxusai-8a7ba949 on :11435 meant
overwriting the only copy of the 0.33.2 payload; rolling back would have been
a ~25 minute cold rebuild at 2b95b4a, itself possible only because Xcode's
separately-downloadable Metal toolchain happened to get installed this
morning.

Both pairings are now archived — payload-0.33.2-maxusai-2b95b4a5/ and
payload-0.34.0-maxusai-8a7ba949/ beside the binaries — so rolling between
them is a copy. The procedure is updated to archive both halves, and to
archive the OUTGOING pairing before overwriting it.

Also adds the 0.34.0-maxusai-8a7ba949 row. It is recorded as built and
staged, NOT serving: preflight PASS 19/12 against the mlx-metal-0-34-0
profile (#306), and #301's whitespace bound confirmed behaviourally on real
documents across the MLX pin move, but the :11435 kickstart has not run. The
0.33.2 row loses "current" and says what it actually is.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The kickstart ran at 13:17. Verified three ways rather than by version
alone: :11435 reports 0.34.0-maxusai-8a7ba949 on a new pid, a live MLX load
emitted "MLX version"=0.32.2-61-gd9add9d, and the dlopen path resolved
/opt/github/MaxusAI/ollama/build/lib/ollama — so the Go and MLX halves are
the pairing that was measured, not a skew.

The 0.33.2 row becomes the rollback target and now says what is new about
it: both halves are archived, so that rollback is a copy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 0.33.2 row carries "goldens NOT recalibrated at this MLX pin" as a
caution. It turns out they did not need to be. Golden parity passes on this
build for 31b, 26b and 12b, with max sampled element deltas of 0.109, 0.051
and 0.063.

The 12b golden is the striking one: generated 2026-08-07 and never
recalibrated, it is three MLX pins behind (27fec909, c793734e, d9add9d1) and
still matches to four decimal places on mean, std and norm_mean. The 31b and
26b goldens were recalibrated once at 27fec909 (d53d33a) and have held
through both moves since.

So the vision encoder path has been numerically stable across every pin move
this fork has made, and this is the first row that can say its goldens were
checked rather than assumed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@glennneuber

Copy link
Copy Markdown
Author

Follow-up: the payload archive turned out to be load-bearing within hours of being created, which is worth recording as evidence rather than argument.

Three separate experiments on 2026-09-18 required standing 0.33.2-maxusai-2b95b4a5 back up as a working pairing to compare against the deployed build:

  1. a 3×3 finetext A/B that confirmed a think-on quality regression
  2. a 12-run constrained-vs-unconstrained A/B that falsified the first hypothesis for it
  3. a 6-run forced-greedy test that falsified the second, and surfaced a sharper symptom

Under the binary-only procedure this PR replaces, each would have required a ~25 minute cold rebuild at 2b95b4a5 — and that rebuild depends on Xcode's separately-downloadable Metal toolchain, which was itself missing on this host that morning and had to be installed before anything could build at all. So the fallback was not merely slow, it was unavailable for part of the day.

With both halves archived, restoring the old pairing is a copy. The finding in #310 exists because of that.

Deployment is confirmed and the row here reflects it: :11435 moved at 13:17, new pid, verified by version, a live MLX load reporting 0.32.2-61-gd9add9d, and the dlopen path. Rollback to 0.33.2 is now a copy plus a kickstart rather than a rebuild.

@glennneuber
glennneuber merged commit 3488461 into main Sep 18, 2026
1 check passed
@glennneuber
glennneuber deleted the docs/binaries-0340-and-payload-archive branch September 18, 2026 11:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant