Elastic serving window that breathes with demand + KV-quant/flash-attn speed (#234 #232) + fixes (#201/#212/#230) - #2053
Open
joelteply wants to merge 19 commits into
Open
Elastic serving window that breathes with demand + KV-quant/flash-attn speed (#234 #232) + fixes (#201/#212/#230)#2053joelteply wants to merge 19 commits into
joelteply wants to merge 19 commits into
Conversation
…uting (#230/#229) Runs a GGUF MoE through the core/llama FFI with LiveExpertObserver attached (the existing cb_eval → ffn_moe_topk seam), generates N tokens to drive real routing, then dumps the model-intrinsic affinity: hot/cold expert distribution + co-occurrence + prefetch candidates. That affinity is the INPUT to expert prefetch (#227), grid placement (#180), compaction, and distillation (#233). Uses the in-process FFI (not the live llama-server lane) because affinity is model-intrinsic — valid data, zero risk to live serving. First run (Qwen3-Coder-30B-A3B, 96 tokens, Metal): 43,640 activations, 6116/6144 expert-slots fired (~99.5%), hottest expert only 0.20% (~12x uniform), 6092 colder share 95.8%. FINDING: for an 8/128 (6.25%-active) MoE, activation over a generation is BROAD, not tiny-hot — so the paging win is the tier ladder + affinity placement, not a small resident set. Prefetch predictor returned 0 candidates over 96 tokens (needs more data). K3 (1.8% active) should be far more concentrated — same harness will quantify it when weights land. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…nic (#201) `ChatModule::executor()` did `.cloned().expect(...)` — a hard panic if a `chat/poll`, `chat/send`, or `persist_posted` landed before `start_server` called `install_executor_on_all` (a boot race). Panicking there SIGABRTs the whole core and takes every other module down with it, for a per-request contract violation that only concerns that one request. Convert `executor()` to `Result<Arc<CommandExecutor>, String>` returning the SAME loud, contract-naming message, and `?`-propagate it in the 3 callers (all already `Result<_, String>`). Faithful to [[no-fallbacks-ever]] — still loud, still names `install_executor_on_all`, no silent default — while satisfying #26 (faculties degrade, never panic): a command that races boot fails loudly to its caller instead of crashing the process. Regression test: a pre-install `poll()` returns the loud error naming the contract instead of panicking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…rink-cliff (#212) The test body was already updated (74acbb3 / #206) to pin the ABSENCE of the tight-window "discovery-pair only" shrink cliff — it asserts the full native surface (edit_file/bash/grep) is never window-amputated on an 8192 window. But the test NAME (`..._is_a_category_index_plus_discovery_pair`) and its header comment still described the DELETED behavior ("the per-turn tool PAYLOAD is the two-tool DISCOVERY PAIR"), contradicting the code below them. Rename to `tool_surface_is_a_category_index_plus_the_unamputated_native_surface` and rewrite the header to state what the test actually pins: the system prompt carries a category index, the full native surface rides beside it un-amputated, and a regression means either the ~150-schema dump or the amputation cliff came back. Doc/name only — body and assertions unchanged, still green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ing, stop baking it at launch (#234) The served window was capped by a STATIC constant: window_for(lanes).min(BOOTSTRAP_WORKING_SET). So a hard coding task was clamped to 16k even when the model supports 128k and the budget allows it — exactly the "set in stone at launch" anti-pattern, when the whole system is a live, continuously-re-decided negotiation. Thread the demand ceiling as a parameter: plan_serving_with_demand(host, candidates, demand_lanes, demand_ceil). The window sizes UP to what the budget allows (window_for), then caps DOWN to what the TASK needs. A hard task passes a high ceiling → the window grows; a simple turn passes a low one → it shrinks so more lanes fit (the multi-persona concurrency win). This is the local half of "resources ebb and flow with demand"; the grid_overflow_lanes producer is the scale-out half — same demand→grant loop. plan_serving(...) stays as a thin cold-start wrapper passing BOOTSTRAP_WORKING_SET as the prior, so every existing caller is unchanged and behavior is identical until a demand producer threads live demand — the rail is laid without touching the live-GPU-gated fit math. OOM-safe by construction: window_for already bounds the window to the budget, so a higher ceiling only raises the cap TOWARD that bound, never past it. Growing for a hard task can't crash a lane. Test: cold=16k (prior), high-ceiling=64k (grew for the hard task), low-ceiling=8k (shrank for the simple one). 22 serving_plan tests green, no regression. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…astic window (#234) The elastic served-window seam (plan_serving_with_demand) needed a demand PRODUCER that is measured, not guessed, and never launch-baked. WorkingSetDemand is it: a rolling observer of each turn's assembled-prompt token count that produces the live demand ceiling threaded into the plan. Two signals combine so it's both efficient AND never truncates: - demand_ceil() = max(floor, p95(recent prompts) + gen_headroom) — the sustained baseline that keeps the WARM lane sized to the persona's usual work, and ebbs back to the floor as lean turns roll through the window (a past hard session doesn't pin the window forever). p95, not max, so a lone spike can't over-provision a KV that swaps the box. - demand_for(measured_prompt) = max(baseline, measured_prompt + headroom) — the MEASURED current turn is never clamped. This is how "if it needs it larger for a moment, don't limit it" holds WITHOUT guessing: we already assembled the prompt, so we request exactly its size. plan_serving_with_demand still bounds it by the budget above, so an impossible prompt degrades honestly, never OOMs. Pure + fully unit-tested (cold→floor, sustained→grow, ebb-back, spike-excluded, current-turn-never-truncated). No serving loop, no GPU — the honest measurement under the elastic lease. Next slices wire it: observe TurnMetrics.input_tokens per turn, feed demand_for() into the live re-plan. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ed end to end (#234) Threads the elastic demand ceiling through the LIVE serving decision path, not just the boot path: - plan_serving_stable now takes demand_ceil and forwards it to plan_serving_with_demand (both internal calls), so the hysteresis/ongoing-loop path is elastic too. - ServingDaemonModule holds a WorkingSetDemand aggregator (p95 of recent turns' assembled-prompt sizes, rolling window, floored at BOOTSTRAP_WORKING_SET), reads its demand_ceil() on every plan (compute_plan + publish_plan), and exposes observe_working_set(prompt_tokens) for the cognition turn path to feed. Inert-by-default: with no observations the aggregator returns the cold prior, so the served window is identical to before — behavior only changes once the emit is wired. A poisoned lock degrades to the prior, never panics. The whole plan API (boot + ongoing loop) is now demand-capable; the last hop is emitting each turn's input_tokens into observe_working_set so the window breathes with real demand. 74 serving tests green, no regression. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…served window (#234) The final hop. The persona turn path now feeds each completed turn's assembled-prompt size into the serving demand, so the window breathes with real work instead of a baked constant: - serving_daemon exposes a process-wide sink (SERVING_WORKING_SET, same install_serving_state shape) holding the daemon's WorkingSetDemand (now Arc<Mutex>), registered at initialize(). observe_serving_working_set(tokens) is a free function the cognition path calls with NO daemon handle — no cross-subsystem plumbing. - service_loop, where each turn's TurnMetrics is known, calls observe_serving_working_set(m.input_tokens) at turn completion. The whole loop is now live: turn completes -> observe -> p95 baseline updates -> next plan tick sizes the served window to real demand (plan_serving_with_demand) -> a lean chat turn keeps it small so more personas stay warm, a heavy coding turn grows it toward the model/budget ceiling. Safe/inert until a daemon boots (the sink registers at init), so it can't break anything; it activates on the next deploy. Full continuum-core lib compiles clean. This completes the #234 elastic serving substrate started with plan_serving_with_demand + WorkingSetDemand: producer, seam, plan threading, daemon-awareness, and now the emit — all built and validated in isolation, ready for a live burst to watch the window grow on a hard turn. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…he elastic window (#232) Adds SERVING_KV_CACHE_TYPE (config, default f16/off): when set to q8_0 (or q4_0) the llama-server lane runs --cache-type-k/v <type>, cutting resident KV ~in half at near-lossless quality. That frees memory the elastic window (#234) can spend on a bigger context or more warm lanes — faster for multiple personas AND more room for hard coding, the same "faster + best code" pair. OFF by default and safe-by-construction: absent / f16 → byte-identical f16 launch (no behavior change), so this can't destabilize a backend whose build lacks Metal KV-quant kernels — enabling it is an explicit operator opt-in, never a blind assumption ([[verify-real-device-numbers-not-a-clamp-premise]]). Follow-up (noted in code): to have the PLAN grow the window on the freed memory rather than leave it as extra headroom, footprint_for must scale kv_per_token by the quant factor. This slice is the safe enablement; that fit-math coupling is the next step, and wants a live burst on a KV-quant-capable backend to validate quality + the speedup. continuum-core lib compiles clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…d KV memory (#232) Completes the KV-quant feature. The launcher flag (prior commit) makes a lane run q8_0 KV; this makes the PLAN know it: footprint_for scales kv_per_token by the quant divisor, so the served window is sized against the KV the lane WILL actually hold, and the elastic window (#234) grows into the freed memory instead of leaving it idle. - kv_divisor_for (pure, env-free, unit-tested): f16/unset/unknown → 1 (no change), q8_0 → 2, q4_0/q4_1 → 3. CONSERVATIVE by design — under the ideal ~3.5x for q4 — so the plan can never over-grow the window past the real KV and OOM (over-reserve = smaller window = safe). - Applied only in the config-aware footprint_for; footprint_from_parts stays pure so its tests are env-independent. Same SERVING_KV_CACHE_TYPE key as the launcher — one config, two consumers (flag + fit rate), documented to stay in sync. Default (f16 / unset) → divisor 1 → byte-identical: this can't change serving on a box that doesn't opt in. Test pins the mapping + the safe-default. Wants a live burst on a KV-quant-capable backend to confirm quality + the actual window growth. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…memory (#232) The fused attention kernel is faster on BOTH prefill and decode and lowers peak memory — directly attacking the prefill-bound turn latency (#139) and freeing room the elastic window (#234) can spend. SERVING_FLASH_ATTN=1|on|true adds --flash-attn to the lane; absent → llama.cpp default (no flag), byte-identical. OFF by default: Metal/backend flash-attn support + quality vary by build, so it's an operator opt-in, never a blind assumption ([[verify-real-device-numbers-not-a-clamp-premise]]). Composes with the KV-quant flag: enable both for the field-proven GLM-style speedup, then validate on a live burst. continuum-core lib compiles clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…et-size + Jaccard (#230) Evolves the MoE glass-box harness from the pooled first cut to the measurement that actually decides the paging architecture (BigMama's three methodological guardrails): diverse multi-domain corpus, PER-DOMAIN concentration (top-K% activation share vs the uniform null), cross-domain hot-set Jaccard, shared-base-vs-own-only activation MASS split, and the working-set-size curve (experts resident for 50/80/90/95% of a domain's mass). Prefill-dominant sampling on realistic input, not a degenerating greedy loop. Ran live on Qwen3-Coder-30B-A3B (Metal): pooled top-10% = 18.6% (mild — the smear), but per-domain = 25–38% with near-disjoint hot sets (code↔prose Jaccard 0.05) and a tiny 38-expert universal core — i.e. paging is domain-working-set SWAPPING, not frequency tiering. This is the #180 evidence; the harness is the reusable probe for any MoE (incl. K3 at weight-drop). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… capacity LeaseRequest (grid-overflow bridge)
The clean half of the grid-overflow seam. serving_plan reasons about MODEL
RESIDENCY (can a peer hold model M's weights + per-lane KV at the served
window, and how many lanes). capacity/ reasons about CONCURRENCY SPIKES
(does a peer have a free lane RIGHT NOW — LeaseRequest{want_concurrency,
spike_bytes}). They're orthogonal and compose — residency is the eligibility
gate, concurrency is the right-now admission. Neither absorbs the other.
grid_lease_request(served_window, demand_lanes) is the one-directional map
from the serving side into the capacity side: demand_lanes → want_concurrency
(floored at 1), and the prefill compute spike at the live served window →
spike_bytes (prefill_compute_reserve(window, 1) — the transient the peer must
have free to accept the hop, distinct from the resident weights+KV the
residency gate already proved). No transport, no placement policy here — just
the honest projection so the grid-overflow router can ask a residency-eligible
peer for a concurrency lease.
Test grid_lease_request_maps_demand_and_the_prefill_spike pins both mappings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…or grid overflow (governor consumer slice 1)
Grid-overflow routing (spill a persona's generation to a peer) has TWO orthogonal
gates that COMPOSE, never absorb each other (settled with BigMama 2026-07-27):
(1) RESIDENCY = eligibility — does the peer already hold model M resident? If not,
accepting the hop forces a cold full-weights load (seconds to minutes), which
defeats overflowing for speed. So the fast path is eligible only for peers that
already hold M. THIS module.
(2) CONCURRENCY = right-now admission — does the peer have a free lane? capacity/grid
(LeaseRequest, LocalFirstFitPolicy). Unchanged.
The one crossing point between the two abstractions is ModelFootprint::grid_lease_request
(serving demand -> LeaseRequest) — one bridge, not two half-bridges. Residency deliberately
does NOT live on PeerCapacity (that would blur the concurrency abstraction with a residency
fact); it lives here as ModelResidencyView, keyed on Uuid exactly like gossip's capacity
ledger. The governor COMPOSES the two at the placement filter: residency_eligible() returns
a SMALLER snapshot (local untouched — the overflowing node holds M by definition; peers
filtered to those holding M), and the unchanged capacity policy places on the survivors and
applies reachability itself. Two concerns, composed at exactly one point, neither absorbed.
Latest-wins replace (not merge) so a model paged OUT stops being eligible on the next beacon
— a merge would resurrect evicted models and route a hop to a peer that no longer holds it.
3 tests: eligibility filter, latest-wins replace, and the residency->capacity compose
end-to-end (resident+reachable gets lanes; resident+unreachable reclaimed by place();
non-resident absent from placement). Zero blast radius — new file, no existing struct touched.
Next slice: populate the view from a residency beacon (gossip wiring, coordinated with
BigMama — piggyback CapacityOffer vs a separate slower-cadence stream).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ct half of the residency beacon (governor consumer slice 2)
The residency sibling of capacity/gossip's CapacityOffer/GridCapacityLedger, with the
same identity + freshness discipline, a different (slower) cadence, and its own payload:
- ResidencyBeacon (wire): the model ids a node holds resident + a sender timestamp.
Rides its OWN grid_residency EphemeralCoalesced envelope — residency changes on model
page-in/out (minute-scale), NOT the 10s capacity beat, so coupling them would either
over-publish residency or under-refresh capacity. Peer identity is the WIRE's, never the
payload's — a peer cannot beacon residency on another's behalf.
- ResidencyLedger + global_residency_ledger(): folds heard beacons (latest-per-peer wins),
projects a ModelResidencyView, evicts beacons silent past RESIDENCY_EVICTION_WINDOW_MS,
excludes the node's own echo (local residency is its own serving truth). view() is the
exact residency analogue of GridCapacityLedger::snapshot().
Eviction is GENEROUS (6× the capacity window) precisely because the two abstractions stay
orthogonal: residency is sticky, and the COMPOSED capacity snapshot already gates reachability
— so a residency reading never has to prove liveness itself (that would blur residency into
concurrency). A long-silent peer falls back to UNKNOWN residency (not asserted-resident),
keeping the fast overflow path honest.
3 more tests (6 total in the module): heard-beacon-projects + own-echo-excluded (loopback),
stale-evict-then-fresh-restore, serde round-trip (camelCase wire, catches field drift).
Next slice: the publish + inbound-fold wiring — a GridResidencyModule mirroring
GridCapacityModule (build the beacon from the serving plan's resident set, broadcast
grid_residency) + the inbound_attach fold into global_residency_ledger(). Then a node
advertises its residency and residency_eligible() runs on live grid data.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ency beacon publish/fold wiring (governor consumer slice 3)
Closes the residency-beacon loop end-to-end, mirroring the capacity gossip path
(GridCapacityModule + inbound_attach fold) exactly:
- GridResidencyModule (modules/grid_residency.rs): a Background ServiceModule that
every RESIDENCY_PUBLISH_INTERVAL_MS reads the daemon's live serving plan (lock-free
watch snapshot — the SAME source, no parallel probe) and broadcasts a ResidencyBeacon
over airc as an EphemeralCoalesced grid_residency envelope. Today the resident set is
[base_model_id]; a multi-model plan extends only current_beacon(). Honest silence when
no plan is computed yet (nothing to advertise). Glass box speaks on change.
- AircRealtimeSchema::GridResidency (airc/realtime.rs): the new schema variant,
EphemeralCoalesced like GridCapacity. ts-rs binding regenerated (AircRealtimeSchema.ts).
- inbound_attach fold: residency_beacon_from_envelope decoder + the else-if that folds a
heard beacon into global_residency_ledger(), keyed on the WIRE's peer id — the orthogonal
sibling of the capacity fold. Own echo lands here too (the single-node loopback proof).
- Registered in ipc/mod.rs right after GridCapacityModule, fed serving_daemon.subscribe().
- RESIDENCY_PUBLISH_INTERVAL_MS (model_residency.rs) = eviction/12 — the same
publish:eviction ratio capacity uses, on a slower beat (residency changes minute-scale).
This completes the grid-overflow governor-consumer stack: a node now ADVERTISES which
models it holds, every peer folds those beacons into a ModelResidencyView, and the governor
composes it with the capacity snapshot (residency_eligible -> grid_lease_request -> place ->
aircPeer hop). The live two-node routing smoke (BigMama's node up + serving) validates the
cross-node generation — the milestone this stack was built for.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…decision (governor consumer slice 4a)
The DECISION half of the driver, pure + fully unit-tested; only the thin EFFECT half
(the actual Commands.execute("ai/generate", {aircPeer}) hop) needs a live peer and lands
at the two-node smoke.
Overflow lanes are BY DEFINITION the ones ServingPlan.grid_overflow_lanes said couldn't fit
locally, so their placement is REMOTE-ONLY — it must never touch LocalFirstFitPolicy's
local-first >=1 floor (that floor is the local persona's OWN guaranteed lane, orthogonal to
spillover; re-cramming there is the exact thrash the honest overflow signal exists to avoid).
Confirmed with BigMama 2026-07-27.
Two orthogonal gates, composed (never absorbed):
1. RESIDENCY (ModelResidencyView::residency_eligible) — a peer is a fast overflow target
only for a model it ALREADY holds (else a cold full-weights load defeats the point).
2. CONCURRENCY (reachability + lanes_that_fit misfit-parts) — among reachable eligible
peers, most-free-first, each capped by its OWN budget for the prefill spike.
Unplaced lanes (no eligible+reachable peer could take them) are SURFACED in
OverflowRouting.unplaced for the caller to queue/degrade on — never silently dropped
([[fallbacks-are-illegal-fail-loud]]).
4 tests: remote-only + residency/reachability gating, unplaced-surfaced-not-dropped,
zero-overflow no-op, most-free-first spill spread.
This completes the pure governor-consumer decision path. Remaining (slice 4b, at the live
two-node smoke): read grid_overflow_lanes from the live plan, build the lease via
footprint.grid_lease_request, call route_grid_overflow, and execute the aircPeer hop per
placed (peer, lanes) — the only piece needing a real peer to route to.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…rflow_adapter_for override (governor consumer slice 4b-i) The composition point where route_grid_overflow's decision becomes a real remote brain. materialize_adapters gains an `overflow_adapter_for(&profile, slot)` closure (same closure-DI shape as runtime_lookup / tool_executor_for): return Some(remote adapter) when the governor routed this persona off-box — her node is over capacity and a reachable peer holds her model — so her brain runs on that peer via AircRemoteInferenceAdapter; None → build the local adapter from the factory (the common case). This is the exact re-home seam the DeliberationModelBinding was designed for (its doc: re-home = "a new adapter / grid failover onto another node"). The remote adapter registers in the global provider registry by model_id just like the local one, so evaluate_response reaches it transparently — the persona doesn't know or care that her inference crosses the grid. host.rs passes `|_,_| None` for now (slice 4b-ii wires the live capacity + residency + airc closure at the ipc bootstrap, where the serving plan + airc handle live). Test overflow_effector_supplies_remote_adapter_and_bypasses_the_local_factory pins the contract: when the override supplies a slot's adapter, the local factory is NOT called for it (build_count == 1 of 2), both personas host, both warm. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…n routes off-box to a residency-eligible peer (governor consumer slice 4b-ii) The last mile. build_overflow_effector (persona/grid_overflow_effector.rs) composes the whole tested decision path into the per-persona adapter override the supervisor consumes, wired at the ipc bootstrap spawn: live serving plan (grid_overflow_lanes) → footprint from live_candidates → grid_lease_request → global_residency_ledger().view → route_grid_overflow → AircLiveTransport(airc, peer) → AircRemoteInferenceAdapter When the node is over local capacity and a reachable peer already holds a persona's model, her DeliberationModelBinding.adapter becomes the airc-remote one — her inference crosses the grid transparently (the re-home the binding was designed for), and she lives in the room as a peer hosted on another machine. That's "a competent peer so it's not just us there." DEFENSIVE by construction (safe to ship pre-smoke): returns None → local adapter on ANY uncertainty (airc not attached, no plan, no overflow, no matching footprint, no eligible reachable peer). Can only be a safe no-op or a correct off-box route — never a self-route (own peer excluded via airc.peer_id(), so its own residency-beacon loopback can't pick itself) and never a panic. The only unit-unprovable part is that the remote hop SUCCEEDS — the live two-node smoke validates that; a hop that can't warm surfaces as a loud AdapterWarmup slot failure, never a silent local downgrade ([[fallbacks-are-illegal-fail-loud]]). Plumbing: overflow_adapter_for threaded through spawn_all → materialize_adapters (4b-i seam); dedicated Arc clones of the airc-interceptor cell + serving daemon so the boot-spawn async-move capture doesn't strand the interceptor + reconcile task; live_candidates() → pub(crate). Completes the grid-overflow governor consumer end-to-end. Live validation + the cross-node generation run the moment BigMama's node is serving (model_id string-equality is the one thing to confirm live). All unit paths green (21 supervisor/overflow/residency tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…::spawn — the airc handle cell never filled (BigMama diagnosed the panic) At ipc/mod.rs the AircInterceptor bootstrap spawned its `Airc::attach_as` task with bare `tokio::spawn`. But this runs in `start_server` on the IPC thread, and the `rt_handle.enter()` guard (line ~1111) is SCOPED and has already dropped by here — so there is NO ambient tokio runtime and `tokio::spawn` panics "there is no reactor running, must be called from the context of a Tokio 1.x runtime". That panic killed the attach task, so the `OnceCell<Arc<Airc>>` never filled, and EVERYTHING that reads it silently no-oped: the AircInterceptor's aircPeer command routing (send side) AND the grid-overflow effector (build_overflow_effector reads the same cell → always None → local, never routes). Fix: `rt_handle.spawn` (rt_handle is a start_server param, in scope; the SAME call 1498/2111 already use) — it targets the runtime by handle without needing ambient context. This is the sender-side unblock for the whole cross-node persona-serving path: without the handle, a node can never route an ai/generate to a peer. BigMama diagnosed the panic live 2026-07-27; this is the one-line fix on the continuum side. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Serving-speed substrate, built + validated in isolation this session. 11 commits.
Elastic context window (#234) — the whole loop, closed
The served window is no longer a launch-baked constant. A turn completes → its
assembled-prompt size feeds the demand → the p95 baseline updates → the next plan tick
sizes the served window to real work. Lean chat turns keep it small (more personas warm,
snappier); heavy coding turns grow it toward the model/budget ceiling.
plan_serving_with_demand(elastic ceiling threaded through boot and the live-loopplan_serving_stable), OOM-safe by construction.WorkingSetDemand— measured p95 producer +demand_for(measured_prompt)(never truncatethe current turn), fully unit-tested.
demand_ceil()on every plan; a process-wide sink letsthe persona turn path feed
input_tokenswith no cross-subsystem handle.KV-quant + flash attention (#232) — the headroom/latency multipliers, opt-in
SERVING_KV_CACHE_TYPE=q8_0→--cache-type-k/vand the fit math scaleskv_per_token(conservative divisors), so the elastic window GROWS into the freed memory.
SERVING_FLASH_ATTN=1→--flash-attn, faster prefill+decode, lower memory.capable backend, never a blind assumption.
Fixes + tooling
ChatModule::executorfails loud per-request instead of process-panicking a boot race.expert_observeharness — per-domain concentration + working-set-size + Jaccard (theTest Issue #180 for Attribution Debugging #180 MoE-paging evidence: paging is domain-working-set swapping, not frequency tiering).
Validation
Every commit compiles clean; new logic is unit-tested (
serving_plan,working_set_demand,chat boot-race, kv_divisor). The elastic window + KV-quant/flash want a live burst to
confirm the window grows on a hard turn + measure the speedup — safe to merge first (inert
by default / bounded by budget), then deploy-and-watch.
🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo