TASK: merge upstream v0.34.0 into main (MLX ce916dbb; drafting under a grammar kept, with a knob) - #297
Merged
Conversation
…18219) * ci: rebuild MLX macOS test payloads the release can't supply The MLX unit test dependency payload can go stale when a PR changes the xgrammar wrapper (release dylib lacks the new symbols) or bumps MLX pins (no matching release, so MLX tests silently skipped). Compare the matched tag's build rules and wrapper sources against the checkout, rebuilding only libollama_xgrammar.dylib on drift; with no matching release, build one Metal variant at the platform default. * address comments
A structured-output request could not use a model's draft head: it decoded one token at a time, at roughly half the speculative throughput on a dense 27B MTP model. The grammar is enforced during verification instead: each draft position's logits are masked before rejection sampling, so an invalid draft is never accepted and every emitted token obeys the grammar. Drafts stay unconstrained; constraining the draft chain would stall its pipelined forwards. Speculative steps also now dispatch the drafts before the host builds the verification graph, worth 4-8% end to end at a fixed draft depth on MTP models, with or without a grammar.
The runner compiled a format as a JSON Schema, the one grammar kind its xgrammar binding exposed. A structural tag holds a schema as one node of a larger tree and also expresses what a schema cannot: free text around constrained spans, a thinking region that closes before constrained content, tool calls pinned to their schemas. The runner now compiles structural tags only; its client wraps the API's formats into one, which compiles to the same grammar as before. The JSON token and vocabulary caps go with it: neither bounds compile cost, which follows the grammar's state count. The byte and nesting caps stay.
Avoid unbounded parallel builds on small runners, and cache the payloads for repeated runs.
progress: join the render loop in stop and do the final writes after the goroutine exits, so Stop/StopAndClear cannot race an in-flight render on the shared bufio.Writer. sched: read the unload-mutable fields in runnerRef.LogValue only under a successful refMu.TryLock and omit them when contended, since slog resolves it on goroutines that may already hold refMu.
Upstream v0.34.0 is a Go- and app-side release: the native payload (MLX, MLX-C, llama.cpp) is unchanged, and none of the MLX binding's C or CMake files moved. Three files conflicted: - server/sched.go: keep the fork's runnerRef.LogValue (a leaf logMu, #289 and #291) over upstream's refMu.TryLock(), which drops name, inference, pid and num_ctx at the eleven log sites that already hold refMu. Adopt upstream's slices.Clone of the GPU list inside the lock. - x/mlxrunner/client.go: take upstream's requestGrammar (structured output compiled as an xgrammar structural tag), in line with ADR 0033. - x/mlxrunner/speculate.go: take upstream's accept(), which lets a constrained session draft. Keep the fork's comments, rewriting the park() note that said a constrained session never drafts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…est to structural tags The v0.34.0 merge took upstream's removal of pid from LogValue's shared attributes (upstream moved it into its TryLock block), so the fork's resolved function stopped logging pid. TestRunnerRefLogValueKeepsFieldsUnderRefMu caught it; put it back. client_format_test.go asserted the pre-0.34.0 wire format and called parseGrammar with a raw "json". Since v0.34.0 the client wraps every non-empty format in an xgrammar structural tag, and parseGrammar accepts only such tags. Re-assert ADR 0009 on that path: empty formats stay unconstrained, every non-empty format survives as a constraint, and a malformed or unwrapped format is an error. Rejecting a format the compiler cannot honour now happens in the native compile, checked live by the fold's GPU gate. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Adds the fold's task doc: the three conflict resolutions, what Gate 2 caught (a silent pid regression, the format test's port to structural tags, a flaky upstream test with its cause measured), and why the MLX half needs a native rebuild. Upstream changed the xgrammar shim that ships in the MLX payload, so a swapped binary cannot load the 0.33.3 library and MLX structured output answers 501. Main's live baseline refuses "yaml" with a 400; the fold's answer awaits a rebuilt payload. Corrects the first-look claim in the sync doc that 0.34.0 needed no native rebuild. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…op fix ml-explore/mlx#4452 stops the CUDA completion worker from spinning after its first batch: an idle runner held one core at 100 % with the GPU idle (measured 2026-09-10, claude-scratch/probe-idle-cpu.sh). The pin stops at that commit (10 past the old pin), not at MLX main. The next CUDA-relevant commit, ollama#4458, inserts a global_scale parameter into gather_qmm's C++ signature ahead of sorted_indices. MLX-C at our pin (c74db530) passes sorted_indices and the stream positionally, so it would bind them to the wrong parameters or fail to compile, and MLX-C has not adapted yet. The commits before ollama#4452 are Metal, CPU and Python fixes plus additive core APIs, so MLX_C_VERSION stays unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…llama#4452 Records Glenn's go-ahead for the rebuild with the MLX bump, the pin at ml-explore/mlx#4452 rather than MLX main (ollama#4458 changes gather_qmm's C++ signature, which MLX-C at our pin calls positionally), why the build misses every llama-server cache, and the MLX gates that follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…f identical on quality On the swap image, which measures the GGUF path validly: the preflight passes 20 and skips 8, the same as main, and the eight-model think-off has no OOM or error. Rendered with the ADR 0012 generators against main, every head-to-head quality row matches (56 of 56), the contract matrices are identical, and only throughput and latency move, with the host's load. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The 0.34.0 loader looks up ten symbols in libollama_xgrammar.so, and the 0.33.3 build lacks two: ollama_xgrammar_matcher_rollback and ollama_xgrammar_matcher_is_terminated. The three dynamic_matcher names the record listed are wrappers defined in dynamic.c itself. The rebuilt payload has all ten. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d; the idle core is fixed On a test image with the fold's Go binary and its freshly built MLX payload (MLX 0.32.2-37-gce916db), "json" and a schema return valid JSON and "" stays unconstrained. "yaml" is still refused with a 400, but with xgrammar's internal compile error instead of main's API-level message. The idle runner sits at 0.0-0.1 % CPU where main held a core at 100 % (ml-explore/mlx#4452). One xgrammar warning, a token offered to an already-terminated matcher, is noted for the campaign's full runner log. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…er a grammar The five-model MLX think-off on the payload-swap image finished with no OOM and no error. A mechanical diff of main's generator tables against the fold's leaves one contract flip (gemma4:12b bcadvnorm1) and small moves on qwen3.6 and gemma4:31b. Every suite request carries a grammar. Upstream 4986e92 lets a constrained request draft, and main never did, so the fold drafts in every completion. Record that mechanism, why the xgrammar terminated-matcher warning is benign, the throughput context, the knife-edge history of name_bbox, and the attribution A/B now running (fold, nodraft, main). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…hat moves outputs Gates 034e to 034i attribute the fold's MLX think-off differences from main, using four images: the fold, nodraft (the fold with main's one-line gate restored), main, and mainmlx (main with only the fold's MLX libraries). Cold, every image gives the same answers apart from knife-edge cells that already flip on main. In the full suite, nodraft and mainmlx reproduce main's answers and answer lengths on all 27 tests, while the warm-drafting fold flips bbox_contract_adv_norm1 in two of three runs. On qwen3.8 the fold's drafting also leaves MLX memory that no tracked array accounts for: +6.8 GiB by request 28, not bounded. Admission cannot see it. A decision is needed on keeping upstream's drafting under a grammar; the recommendation is to restore main's gate until the memory growth is fixed. MLX think-off cells are not bit- reproducible, whatever ADR 0012 section 4 says. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
maxusai/ollama:sync-0.34.0 (0.33.3-dynres-28-gfbedf50) carries the payload-swap image's MLX payload byte for byte and the same Go code, so the MLX gates measured there hold for it. On the real image the format checks, the idle-core check and the preflight cuda-dynres-903 pass. The deploy stays held, pending the decision on drafting under a grammar. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
gate-sync034h's preflight passed with mlx_payload_pin SKIPPED, because cuda-dynres-903 recorded no mlx_build although these images ship mlx_cuda_v13. So the fold's own MLX bump went unasserted. Pin it to this fold's MLX_VERSION, ce916dbb, and widen the profile test to enforce a pin on any profile that serves the MLX payload whatever its platform says. PR #294 adds the same key on main with a523d60's 37c26e57; the merge keeps this value, since the payload here is the bumped one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
probe-mainleak.sh sent the same 18 text requests to main-a523d60b twice on qwen3.8: without a format, where main drafts (18 of 18), and with one, where it does not (0 of 18). The untracked gap grows only in the drafting arm, to +0.72 GiB by request 17, while the other holds -0.11 to -0.14 GiB. So the leak is in the speculation path main already ships, and production meets it today on think-on and format-less requests. The fold widens its reach to structured output rather than introducing it, and restoring main's gate limits the leak without removing it. Record that in the decision, since it changes what either choice buys. Also correct the peak-memory median to the committed generator's definition (2.7, not 2.9: statistics.median averages the two middle values) and cite the promoted generators from #295 instead of the scratch scripts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The fold's own preflight is the run that exposed this: pinning mlx_build on cuda-dynres-903 made gate-sync034h's preflight FAIL with "no MLX engine-init line in the log window", because a CUDA preflight loads llama.cpp models and no MLX runner starts. Take the fix from #294: read the version string out of the shipped libmlx.so when no engine-init line appears, keeping the live line as the first source. The pin here stays ce916dbb, this fold's MLX. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Re-ran the preflight on maxusai/ollama:sync-0.34.0 with the pin and the payload fallback in place: 21 pass, 7 skip, and mlx_payload_pin reports "MLX build ce916db matches the measured payload (0.32.2-37-gce916db), from the shipped libmlx.so". The 06:40 run was 20/8 with that check skipped. Record why the second source was needed: a CUDA preflight loads llama.cpp models, so the engine-init line the check reads never appears, and pinning alone failed a correct payload. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… knob Glenn: align with upstream, keep ours as an option, post nothing to ollama/ollama. speculate.go now decides through draftingEnabled, and OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0 restores main's gate. An unrecognised value keeps upstream's default and warns: a typo that silently disabled drafting would surface later as "the fold is slow" with nothing pointing back here. ADR 0033 carries the divergence, since its own consequence section said grammar-aware speculation would have to be rebuilt on upstream's engine if the depth controller were ever fixed. The fold's decision section records what the choice buys and costs, including that the memory leak is main's too and the knob is a mitigation rather than a fix. Tests: the env parser table, and the gate across grammar, logprobs and top-logprobs requests with the knob both ways. go vet and the full x/mlxrunner suite pass in the golang:1.26.0 container. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
OLLAMA_MLX_DRAFT_UNDER_GRAMMAR sits beside OLLAMA_GPU_OVERHEAD and OLLAMA_MLX_MEMORY_LIMIT in the vision-suite README, with when to reach for it: a long-running qwen3.5-family server serving structured output, where drafting leaves memory admission cannot price. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
probe-knob.sh on maxusai/ollama:sync-0.34.0-knob (the candidate with this branch's binary layered on, payload byte-identical): with the knob unset, four format requests and two without one all draft; with it 0, the format requests draft nothing and the others still do; with an unrecognised value drafting stays on and the runner warns. All 200. So the switch gates structured output only, which is what the decision above claims. Note that the clean candidate image predates the knob: the switch needs the next full build, or this swap. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The real image had been covered by P1 and P2 — payload byte-identical, Go code equal — rather than by a campaign of its own. This is the campaign: five models, think-off, no OOM, no error, nothing left unconverged. Against main its contract matrix is identical, 50 of 50 cells, bbox_contract_adv_norm1 included: the knife-edge cell that came back ❌ on the swap image's run is ✅ here, which is what a knife-edge does. Two of 70 T1 quality cells differ from either baseline, a 0.003 IoU move and one 7 px code, and answer lengths sit inside the spread two runs of one image already show. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
PR #297 against main, opened 2026-09-13. Not merged, and the deploy stays held. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Re-reading the four existing trace logs — no new runs, no GPU — turns "memory grows" into a unit and a cadence. The gap moves by whole 144 MiB units (the 48 layers' F32 [1 48 128 128] delta state at 3 MiB each) or not at all: 15 steps of 3 units and one of 4 over the fold's 28 requests, 6 steps of exactly 1 unit over main's 17 drafting requests, and 16 zeros over main's gated arm. It is not proportional to the drafting work, so it is a per-request exit path rather than the round loop. The trie is exonerated more firmly — the tracked total, snapshot bytes, paged-out bytes and snapshot count agree request for request across the arms, and the gap keeps growing after the trie caps and the tracked total goes flat. It is not driver-side either: against the pinned MLX ce916dbb the CUDA backend makes no raw allocation outside event.cu, and its graph cache is an LRU capped at 400. That leaves a full recurrent state whose Go handle is released while MLX still holds a reference. The next probe needs a GPU, so the upstream report stays held. Also fix the gofumpt violation golangci-lint caught in the gate test: one table element per line. The macOS "[setup failed]" alongside it was a cascade — the matrix's fail-fast cancelled the Darwin payload build mid-step, and go test then ran anyway under if: always() against an unbuilt app/ui embed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The fold of upstream v0.34.0 (
d8ab4b4f, 21 commits across 88 files), cut from main at7d230fdc. The full record, with every gate and its evidence, isdocs/maxusai/tasks/upstream-sync-0.34.0.md.Merge and pins. Merge commit
ca3ff1db. MLX moves37c26e57→ce916dbb(ml-explore/mlx#4452, the idle runner that held a CPU core at 100 %), which is why this fold needed a real rebuild rather than a Go-only swap. llama.cpp and MLX-C pins are unchanged. The pin deliberately stops short of MLX main: ollama#4458 insertsglobal_scaleintogather_qmmahead ofsorted_indices, and MLX-C atc74db530passes positionally.Three conflicts.
server/sched.go— kept our leaflogMu(server: fix the data race between runnerRef.unload and LogValue #289/server: pin the in-lock runner log attrs, and correct the divergence note #291) over upstream'srefMu.TryLock(), which dropsname,inference,pidandnum_ctxat the eleven log sites already holdingrefMu. Our own test caught upstream's removal ofpidarriving as a clean auto-merge.x/mlxrunner/client.go— took upstream'srequestGrammar: formats now compile as xgrammar structural tags.x/mlxrunner/speculate.go— took upstream'saccept(), which is what lets a constrained request draft.Gates. Gate 2 green in the golang container. Gate 3 built
maxusai/ollama:sync-0.34.0. Gate 4 preflight PASS, and 21/7 once the MLX pin was made assertable. Gate 5 GGUF: every quality row identical to main across eight models. Gate 5 MLX ran twice — on the payload-swap image and, at the end, on the shipping image itself: contract matrix identical to main, 50 of 50 cells, with 2 of 70 T1 quality cells differing (a 0.003 IoU move and one 7 px OCR code). Renders inpreflight-runs/.The MLX differences, attributed. About thirty suite runs across four images — the fold, the fold with drafting gated off, main, and main carrying only the fold's MLX libraries — show that drafting under a grammar is the only change in this fold that moves MLX outputs, and what it moves are knife-edge cells that flip between runs on main too. Without it the fold reproduces main's answers test for test.
Decision (maintainer, 2026-09-12): align with upstream. Drafting under a grammar stays on by default;
OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0restores the gate this fork carried before. ADR 0033 is amended here to record the divergence, and the knob is documented beside the other serving knobs.A known issue this fold widens but does not introduce. Drafting leaves MLX memory that no tracked array accounts for, growing across requests on the qwen3.5 family, which the admission headroom cannot see. Main's own drafting does the same on think-on and format-less requests; drafting under a grammar extends the reach to structured output. The knob limits that reach without fixing it. The fix belongs upstream in the speculation path, and nothing is being filed there until we can propose one. ADR 0034 is amended in #296.
Merge order with the open PRs.
expectations.tomlconflict here, resolved by keepingce916dbb, the payload this branch ships.Not in this PR. The deploy stays held. The built candidate predates the knob commit, so shipping the knob needs the next build, or the
-knobswap image the task doc describes. The Metal half is the maintainer's.🤖 Generated with Claude Code