fix(serving): ephemeral eval lane fails LOUD with the real cause, not a masked 240s /health timeout (#205 unmask) - #2055
Open
joelteply wants to merge 3 commits into
Open
fix(serving): ephemeral eval lane fails LOUD with the real cause, not a masked 240s /health timeout (#205 unmask)#2055joelteply wants to merge 3 commits into
joelteply wants to merge 3 commits into
Conversation
…a masked 240s /health timeout (#205 unmask) Glass-boxed 2026-07-27 running the live coding measurement (agent/solve): every `agent/solve` on this Mac failed with a bare "llama-server not ready after 240s (/health request failed)". The real cause was thrown away — `wait_ready` polled the health port for the WHOLE budget without ever checking whether the child it spawned had died, and never surfaced the child's stderr. ROOT CAUSE, proven this session: the ephemeral eval lane forges a SECOND Devstral-24B (~14 GB) while the live persona lane already holds ~26 GB. With only ~9.6 GB free, macOS jetsam SIGKILLs the second llama-server the instant it maps the model — before llama.cpp prints a single byte (reproduced by hand: exit 137, zero-byte log). It is an OS out-of-memory kill, NOT Metal-context contention and NOT a hang. The masking bug made it look like a mysterious timeout. Two unmask fixes in `wait_ready`, benefiting live AND ephemeral lanes: 1. **Fail loud the instant our child EXITS** — `child_exit_status()` (non-blocking `try_wait`) turns any crash-at-launch — including the jetsam SIGKILL/137 above — into an immediate `Spawn` error carrying the exit status + stderr tail, instead of polling a dead port for 240s. This is the arm that fires for the memory-wall failure. 2. **Fingerprint the empty stderr on the hang-timeout** — `tail_or_hang_marker` turns an empty log into a marker that, read with the exit status, names the two empty-log causes: an OOM/jetsam kill (SIGKILL/137, child exited) vs. a genuine early-init hang (no exit status). A crash from bad args / model-load fault prints its banner first, so a non-empty tail carries that directly. `tail_or_hang_marker` is a pure fn (unit-tested: empty→OOM/hang marker, non-empty→last 20 lines in order) so the load-bearing decision is tested without touching the real `~/.continuum/logs` path (#72 env-dependent-test lesson). This makes the failure DIAGNOSABLE; the underlying fix (don't forge a second 24B for eval while the live 24B is resident — reuse the live lane's weights or serialize via the governor, #59/#234) is a follow-up. Validated: continuum-core compiles (--features metal,accelerate, 0 errors); `tail_or_hang_marker_fingerprints_empty_and_tails_nonempty` green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
joelteply
force-pushed
the
fix/eval-lane-bringup-fail-loud
branch
from
July 28, 2026 03:42
6f90bf4 to
8e4d77f
Compare
…ot just pressure LEVEL — kills the jetsam SIGKILL at the root (#205) The companion to the unmask commit: don't just REPORT the eval-lane OOM loudly, PREVENT it. The gate (`await_eval_lane_memory_headroom` → `refuse_eval_lane_under_memory_pressure`) vetoed only on macOS *pressure LEVEL* + the sustained-pressure gate. But on unified memory the level reads "Normal" while sitting atop only a few GB of real free RAM — it counts compressible/cached pages as available. So the gate green-lit standing up a SECOND llama-server of a known ~14 GB footprint into 9.6 GB of actual headroom, and the OS jetsam-SIGKILLed it (exit 137, zero-byte log — the exact failure this session reproduced by hand). Neither the pressure gate NOR the GPU/CPU placement lease caught it: on unified memory the weights need the RAM on *either* device, so "spill to CPU" doesn't save you. Fix — size against the honest free-bytes number: - `MemoryPressureMonitor` already reads `sysinfo::available_memory()` each poll; publish it to a new lock-free global `current_available_bytes()` alongside the pressure level (the level is a ratio and lies; the bytes don't). - `eval_lane_ram_veto(available, footprint, headroom)` — a PURE, unit-tested guard that refuses ONLY when a KNOWN footprint won't fit in the KNOWN free bytes (+2 GiB headroom), and NEVER when either number is unknown (an unread probe must not starve a node — the pressure gate + placement lease stay the backstops). The refusal names the OOM wall, so the detached ledger carries a real cause and `await_eval_lane_memory_headroom` retries it as deferrable load instead of crashing. - One footprint sizing (`eval_lane_footprint`) now shared by the gate and the placement decision (compression — was duplicated inline). - Both eval-lane spawn sites resolve `base` and size the lane BEFORE the gate, so the RAM check runs pre-cold-load. This is the reliability doctrine [[reliability-is-it-works-not-that-it-reports-failure-well]]: the prior commit made the failure legible; this makes the machine refuse cleanly instead of being OOM-killed. Sibling of the #175 GPU-OOM-poisons-the-backend class — the level-vs-real- bytes gap is the same shape. Validated: continuum-core compiles (--features metal,accelerate, 0 errors); new `eval_lane_ram_veto_refuses_only_a_known_oversize_lane` + existing pressure-veto test green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…symlink into the ephemeral cargo target dir The flaky mess where `continuum` vanishes post-boot ([[deploy-cli-binary-deleted-from-target-dir-post-boot]]): `start-server.sh` symlinked ~/.local/bin/continuum → the cargo target-dir binary. But that dir is a BUILD artifact — cargo replaces the binary mid-rebuild, `cargo clean` and rust-analyzer's feature-mismatched rebuilds delete it — and the PATH symlink then dangles, so `continuum <cmd>` dies with "no such file or directory" (hit twice this session). Fix: COPY the binary to ~/.local/bin (atomic temp+mv so a concurrent `continuum` invocation never sees a half-written file), and `rm -f` any pre-existing entry first so a leftover symlink from an old install can't make `cp` follow it back into the target dir. The PATH binary is now decoupled from cargo's churn — it changes only on deploy. Same self- provisioning intent, minus the ephemeral-artifact coupling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found running the live coding measurement
Every
agent/solveon this Mac failed with a bare "llama-server not ready after 240s (/health request failed)" — the real cause discarded.wait_readypolled the health port for the whole budget without ever checking whether the child it spawned had died, and the child's stderr (llama.cpp banner / ggml-Metal fault) was never surfaced. Diagnosing it took three separate log-digs.Two unmask fixes in
wait_ready(benefit live and ephemeral lanes)child_exit_status()(non-blockingtry_wait) turns a crash-at-launch (bad args, model-load failure, aborted Metal init) into an immediateSpawnerror with the exit status + stderr tail, instead of polling a dead port for 240s.tail_or_hang_markerturns that into the fingerprint "HUNG in early init (Metal contention?), not a crash" — the diagnostic that distinguishes a hang from a crash, which was missing.tail_or_hang_markeris a pure fn, unit-tested (empty→fingerprint, non-empty→last 20 lines in order) so the load-bearing decision is tested without touching the real~/.continuum/logspath (#72's env-dependent-test lesson).Scope
This makes the eval-lane bring-up failure diagnosable — it does not fix the underlying hang. After this deploys, the next
agent/solvewill name the real reason (here, the empty-log fingerprint points at two concurrent Metal lanes on one Mac, whose fix is to share the live lane's immutable base weights for the measurement rather than spawn a second Metal context — a follow-up card).Validation
continuum-corecompiles (--features metal,accelerate, 0 errors); newtail_or_hang_marker_fingerprints_empty_and_tails_nonemptytest green.