Repository navigation
Conversation
Reject oversized single-slot Metal requests before GPU prefill and before streaming HTTP headers. Estimate the loaded model's actual forward geometry, resident allocations, output KV, recurrent state and bounded SSD scratch; preserve context-error precedence and return a typed 503 without poisoning subsequent requests. Materialize lazy weights during model loading, retain cold-plan safety instead of optimistic prefix-cache credits, and document conservative over-rejection and unsupported paths. Add scheduler, provider cancellation and cross-route regression coverage. Validation: 17 admission tests and 22 Nemotron tests passed. Initial real-checkpoint calibration accepted 6,001 tokens with a 27.34 GiB estimate and 17.8 GiB measured peak; final lower-cap rejection and no-warmup validation are pending. Refs #2291
inureyes
added
status:review
Route guarded queued requests through the final single-request gate, align provider and worker scope across local and distributed paths, and cover legacy no-batch serving without changing its sequential execution model. Materialize Nemotron-H weights at shared construction, reject unsupported runtime LoRA with staged-state cleanup, and disable guarded prompt snapshot donation and background warmups outside the final admission gate. Refs #2291.
Archive attributed Nemotron-H rejection/recovery measurements for streaming, nonstreaming, no-warmup and legacy no-batch serving, plus the unaffected Qwen-family smoke and invalid override startup checks. Separate intermediate calibration from the reviewed release artifact, compare conservative estimates with process-lifetime physical peaks, and state remaining measurement limits and the distributed CI stack-frame correction explicitly. Refs #2291.
Configure the request-memory worker scope through an in-place setter after the existing builder chain. The additional by-value BatchScheduler builder frame overflowed the default model-worker stack in unoptimized Linux Pipeline Parallel CI, while the immediately preceding commit passed the same real-model smoke test. Validation: cargo test --profile test-fast --features metal,accelerate --lib request_memory -- --test-threads=1 (26 passed). Refs #2291.
Record normal and legacy Nemotron rejection/recovery and unaffected Qwen streaming checks against the exact post-setter release artifact, with its hash and unchanged admission estimates. Refs #2291.
inureyes
added
status:done
Handle both new generation events explicitly as unexpected Metal-policy events in the XLA result consumers, and add regression tests for the result-draining helper. Validation: scoped rustfmt check and git diff --check pass. The production cuda,xla-iree compile requires the GB10 CI runner's IREE runtime and was not run on this macOS host.
The Playwright harness used final control-file existence as its completion signal, while the Rust writer created that path before writing JSON. A fast reader could therefore parse an empty file and fail with Unexpected end of JSON input. Stage each private response beside its destination, sync it, and publish with persist_noclobber so visibility still means completeness and an existing response is never replaced. Add concurrent-read, private-mode, and no-clobber regression coverage. Validation: targeted test-fast private_json_is_complete_when_published_and_never_clobbered test passed; rustfmt and diff checks passed. Refs #2291.
Record bounded host and GPU snapshots around every acceptance sample so noisy WebUI performance verdicts can be separated into load, clock, power, or thermal effects without changing ordering, counts, thresholds, retries, or exclusions. Diagnostic collection uses a fixed nvidia-smi argv and records failures without deciding the verdict. Validated with the diagnostics Node tests, the WebUI verifier Python tests, Node syntax checks, and git diff checks.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add conservative request-memory admission for local Metal text Nemotron-H at effective width one, including
--no-batch. Estimate resident allocations and planned KV, recurrent, activation, and SSD memory from loaded configuration and actual prefill geometry.MLXCEL_REQUEST_MEMORY_BUDGETcan lower, not raise, finite runtime caps; invalid values fail startup and unknown/overflowing estimates fail closed.Over-budget requests receive HTTP 503 with
request_memory_budget_exceededand byte diagnostics before request GPU work or SSE headers. Preserve context-error precedence, cancellation, queue ownership, and unsupported-worker behavior. Guarded requests prefill cold without snapshot donation or background warmups. Eagerly realize weights, including fused adapters; reject Metal runtime LoRA at startup because its scratch is unmodeled.Feature-gated XLA test consumers explicitly reject unexpected Metal admission events. The browser harness publishes completed private control JSON atomically without overwriting existing responses, fixing a CI reader/writer race. Per-sample CPU load and GPU telemetry diagnose unstable performance baselines without changing acceptance thresholds, ordering, or sample counts.
Verification
Review fixed alternate-prefill/cache-work bypasses, unsupported-mode acknowledgement hangs, lazy adapter residency, and legacy streaming. An in-place scheduler setter fixed the Linux debug-stack regression; the formerly failing two-host pipeline step passed.
This is not a universal OOM guarantee. Conservative estimates may reject physically fitting requests; external GPU pressure, fragmentation, other-family/backend policies, live adapter inference, and multi-slot reservation are unverified or outside scope.
Closes #2291.