diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3b979c512..eabb0facf 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -946,6 +946,7 @@ jobs: --work-dir "$RUNNER_TEMP" \ --repo-root "$GITHUB_WORKSPACE" \ --node-bin "$(command -v node)" \ + --nvidia-smi "$(command -v nvidia-smi)" \ --virtual-display \ --activity-timeout 1800 \ --pid-file "$RUNNER_TEMP/webui-activity-gate.pid" \ diff --git a/TECHNICAL_REPORTS/2294-nemotron-h-request-memory-admission-20261014.en.md b/TECHNICAL_REPORTS/2294-nemotron-h-request-memory-admission-20261014.en.md new file mode 100644 index 000000000..dae108130 --- /dev/null +++ b/TECHNICAL_REPORTS/2294-nemotron-h-request-memory-admission-20261014.en.md @@ -0,0 +1,92 @@ +# PR #2294: Admit Nemotron-H requests against a checked memory budget + +**Date**: 2026-10-14 +**Status**: Implementation and validation complete; remaining CI jobs pending +**Risk**: High because the guard sits between HTTP streaming and GPU execution and intentionally fails closed + +## Executive summary + +PR #2294 adds conservative request-memory admission for single-slot, text-only Nemotron-H serving on Metal. Requests whose checked allocation envelope exceeds the lower of the runtime limits and `MLXCEL_REQUEST_MEMORY_BUDGET` receive a typed HTTP 503 before GPU prefill or SSE headers; context overflow still receives HTTP 400, and unsupported models and execution modes keep their prior behavior. + +## 1. Problem statement + +PR #2290 bounded Nemotron-H SSD prefill scratch, but a valid in-context request could still exceed Metal memory through resident weights and caches, output KV, recurrent state, activations, logits, attention fallback, and bounded SSD workspace. Such an allocation failure can terminate serving rather than reject one predictable request. Startup memory estimates cannot decide this at the actual call site because the relevant prompt, planned prefill geometry, requested output, retained allocations, and configured runtime ceilings are only known immediately before prefill. + +This guard is deliberately not a universal OOM guarantee. Other processes, driver allocations, fragmentation, unexpected kernel workspace, and allocations after admission can still consume memory. + +## 2. Technical review + +### 2.1 Admission boundary and error contract + +The final check runs on the scheduler thread after preceding width-one work finishes and before request GPU work. The provider waits for a one-shot admission event before an HTTP route commits streaming headers. A rejection is returned as HTTP 503 with `error.type` and `error.code` equal to `request_memory_budget_exceeded`, plus `estimated_bytes` and `budget_bytes`. Existing context validation retains precedence and returns HTTP 400. Cancellation and worker-disconnect paths complete or cancel the handshake so an abandoned or unsupported request cannot leave a waiter hanging. + +The review closed two route-level gaps. Guarded requests cannot enter batched prefill, whose sequential fallback bypassed the final single-request check, and unsupported paths acknowledge immediately instead of waiting for an admission event they never emit. Legacy `--no-batch` remains sequential but now uses the same size-one scheduler guard. + +### 2.2 Checked estimate and budget + +The pure estimator uses normalized loaded `NemotronHConfig`, the loaded embedding output dtype, actual cold `PrefillPlan` pieces, architecture-aware KV geometry, and a current MLX memory snapshot. All products and sums use checked arithmetic; unknown dimensions, unknown budgets, invalid dtypes, and overflow fail closed. The envelope charges current active allocations, cold attention KV with growth and snapshot ownership, recurrent and convolution state, per-layer activations, logits, MoE intermediates, conservative float32 attention scores, and float32 SSD buffers with the model's internal chunk size. + +The default budget is the minimum positive finite value among the device recommendation, the wired cap actually applied by mlxcel, and the allocator cap. `MLXCEL_REQUEST_MEMORY_BUDGET` accepts the existing binary size grammar but can only lower that ceiling. Empty, zero, `none`, negative, nonfinite, and overflowing explicit values stop startup. + +### 2.3 Residency and cache policy + +Metal Nemotron-H weights are evaluated at the shared construction boundary, including when warmup is disabled, so lazy graphs cannot make the first request observe a false low resident baseline. The constructor also covers fused checkpoint adapters and embedded VLM or VoiceChat users, but only eligible text serving enables admission. Runtime unfused LoRA is rejected at startup and staged adapter state is drained because arbitrary-rank float32 workspace is not modeled. + +Guarded requests prefill cold, do not restore or donate prompt snapshots before admission, and skip background prompt-cache warmups. Retained snapshots remain part of active memory. This conservative policy can reject a request that would fit with optimistic cache reuse or a fused kernel, but it avoids performing request work before the final measurement. + +### 2.4 Scope and compatibility + +The policy is restricted to local, non-speculative, text-only Metal Nemotron-H with effective width one. CUDA, ROCm, CPU, other families, multimodal input, tensor or pipeline parallel execution, disaggregated worker handoffs, speculative decoding, and multi-slot reservations retain their prior policy. The PR adds no dependency or data-format change and does not alter model numerics. + +## 3. Technical decisions + +| Decision | Chosen approach | Rejected alternative | Trade-off | +|----------|-----------------|----------------------|-----------| +| Check location | Final scheduler admission at the production prefill call site | Startup estimate or HTTP-only heuristic | Accurate request geometry and resident state, but requires an HTTP-to-worker handshake | +| Cache credit | Charge cold KV and retained active allocations | Assume a prompt snapshot hit reduces the request | Predictable safety with possible false rejection and lost cache-hit latency | +| Override | Lower-only cap over known runtime limits | Let an environment variable raise or create a budget | Operators can test stricter policy without bypassing device limits | +| Unknown values | Fail closed | Admit when geometry, budget, or arithmetic is unknown | Safer failure mode, but configuration or unsupported geometry can reject service | +| Runtime LoRA | Reject unfused adapters at construction | Guess arbitrary-rank workspace | Clear startup failure until the workspace can be modeled | + +## 4. Verification + +- Request-memory suites: 26 tests passed, covering budget boundaries, output growth, checked overflow, invalid overrides, cache accounting, context precedence, cancellation, zero-forward rejection, unsupported paths, batched-prefill bypass, legacy serving, and pre-SSE errors. +- Nemotron-H suite: 23 tests passed and 3 existing tests were ignored, including runtime-LoRA rejection and staging cleanup. +- Focused provider and native-route binaries passed 30 and 34 tests respectively; these overlap the 26 request-memory tests and are not added as unique cases. They cover all five generation dialects and the provider admission lifecycle. +- `cargo check --tests --features metal,accelerate` passed in 51.50 seconds, and `cargo clippy --lib --tests --features metal,accelerate -- -D warnings` passed in 5 minutes 5 seconds. Known C++ unused-capture warnings did not fail the Rust Clippy gate. +- `cargo build --release --features metal,accelerate` passed on the measured source in 9 minutes 42 seconds and again after the in-place-setter correction in 8 minutes 59 seconds. +- Reviewed-source real-checkpoint lifecycle passed on three normal fresh servers, one `--no-warmup` server, and one legacy `--no-batch` server. Post-setter normal and legacy lifecycle smokes repeated the rejection/recovery behavior. A Qwen 2.5 7B second-family smoke remained admitted for streaming and non-streaming requests before and after the setter correction. +- GitHub's two-host logical pipeline job reported a stack overflow from one added by-value `BatchScheduler` builder frame in an unoptimized worker. Commit `9d2a1584` replaced it with an in-place setter without changing admission behavior, and the 26 targeted tests passed. The formerly failing logical job then passed on the corrected commit; the full workflow and corrected release build still had pending jobs and were not reported as fully green. + +## 5. Benchmark evidence + +The checked-in benchmark record is `docs/benchmark_results/nemotron-h-request-memory-m1ultra-2026-10-13.md`, with machine-readable artifacts under its matching `data/` directory. Initial calibration on an M1 Ultra 128 GB admitted a 6,001-token request under 32 GiB with a 27.34 GiB estimate and a measured 17.8 GiB process peak; the harness correctly refused to present that run as rejection evidence. Intermediate lower-cap runs are retained as calibration but predated the final routing and cache review fixes. + +The final reviewed binary (`1c4673addeb123b8d3ef2758f69dc0c54811d1d061911dbcb86aa486cc6065c5`) ran three normal fresh-server repetitions. All 12 long streaming and non-streaming requests returned typed 503 before SSE, all 12 immediately following short requests and all 3 medium requests returned 200, all 3 beyond-context requests retained 400 precedence, and health remained 200. The medium estimate was 22.328 GiB in every repeat with a 17.6 GiB process physical peak; long estimates ranged from 27.337 to 27.351 GiB, short estimates from 17.813 to 17.865 GiB, and the long-rejection process peak was 16.9 GiB. One-minute host load was 6.44, 6.90, and 11.01, so this is lifecycle and safety evidence rather than a latency claim. + +One no-warmup server repeated 4 long 503 responses, 4 following short 200 responses, a medium 200, a context 400, and healthy status. One legacy `--no-batch` server repeated 4 long 503 responses, 4 following short 200 responses, and a medium 200; its unchunked largest forwards produced intentionally conservative 45.506–45.634 GiB estimates. A Qwen 2.5 7B server returned 200 for both streaming and non-streaming requests under the same budget and remained healthy, confirming the shared route handshake did not apply Nemotron-H rejection policy to the second family. + +The post-setter release artifact (`e7a5cc9997a8c0eeceba6a69769353eed35395e906b270d2c62a5f437e2a7062`) repeated one normal and one legacy lifecycle: each produced 4 typed long-request 503 responses before SSE, 4 completed short-request 200 responses, a medium 200, and health 200; the normal context check stayed 400. Estimates matched the measured artifact ranges. The Qwen control again returned 200 for streaming and non-streaming requests with health 200. These post-setter artifacts and the benchmark record were committed in `62c1cf03`. + +## 6. Change summary + +| Category | Summary | +|----------|---------| +| Safety | Adds a checked, fail-closed request envelope and lower-only runtime budget | +| HTTP | Returns typed 503 responses before SSE headers across supported generation routes | +| Scheduler | Serializes guarded admission at the final width-one prefill boundary and blocks batched-prefill bypass | +| Model loading | Materializes Metal Nemotron-H weights and rejects unmodeled runtime LoRA cleanly | +| Cache behavior | Disables pre-admission snapshot restore/donation and background warmup for guarded requests | +| Tests and docs | Adds estimator, scheduler, provider, route, and model regression tests plus operator documentation and a benchmark harness | + +## 7. Follow-up and monitoring + +- Confirm the corrected release build and remaining CI jobs before merge; do not claim a fully green workflow without evidence. +- Monitor false-rejection rates, estimate-to-observed-peak ratios, retained prompt-cache contribution, and post-admission allocation failures. +- A future policy may model additional families, runtime LoRA ranks, multi-slot reservations, distributed paths, or optimistic cache reuse, but none is promised by this PR. + +## References + +- Issue #2291: reject over-budget Nemotron-H requests before Metal prefill. +- PR #2290 and issue #2281: bound Nemotron-H SSD prefill scratch and establish the measured long-prompt baseline. +- `docs/request-memory-admission.md`: operator contract, limitations, and error shape. diff --git a/TECHNICAL_REPORTS/2294-nemotron-h-request-memory-admission-20261014.ko.md b/TECHNICAL_REPORTS/2294-nemotron-h-request-memory-admission-20261014.ko.md new file mode 100644 index 000000000..1eb1de290 --- /dev/null +++ b/TECHNICAL_REPORTS/2294-nemotron-h-request-memory-admission-20261014.ko.md @@ -0,0 +1,92 @@ +# PR #2294: 검증된 메모리 예산으로 Nemotron-H 요청 승인 + +**작성일**: 2026-10-14 +**상태**: 구현 및 검증 완료, 남은 CI job 대기 +**위험도**: HTTP 스트리밍과 GPU 실행 사이에서 동작하며 의도적으로 fail-closed하는 보호 장치이므로 높음 + +## 요약 + +PR #2294는 Metal에서 단일 슬롯으로 동작하는 text-only Nemotron-H serving에 보수적인 요청 메모리 admission을 추가한다. checked allocation envelope가 런타임 한도와 `MLXCEL_REQUEST_MEMORY_BUDGET` 중 낮은 값을 넘으면 GPU prefill이나 SSE header 전송 전에 typed HTTP 503을 반환한다. context overflow는 계속 HTTP 400이 우선하며, 지원하지 않는 모델과 실행 모드는 기존 동작을 유지한다. + +## 1. 문제 정의 + +PR #2290은 Nemotron-H SSD prefill scratch를 제한했지만 유효한 context 안의 요청도 resident weights와 cache, output KV, recurrent state, activation, logits, attention fallback, bounded SSD workspace 때문에 Metal 메모리를 초과할 수 있었다. 이런 할당 실패는 예측 가능한 단일 요청을 거절하는 대신 serving process를 종료할 수 있다. Startup memory estimate로는 실제 호출 지점의 요청을 판단할 수 없다. prompt, 실제 prefill plan, 요청 output, retained allocation, 설정된 runtime ceiling은 prefill 직전에만 함께 알 수 있기 때문이다. + +이 보호 장치는 universal OOM guarantee가 아니다. 다른 process, driver allocation, fragmentation, 예상하지 못한 kernel workspace, admission 이후의 allocation은 여전히 메모리를 소비할 수 있다. + +## 2. 기술적 검토 사항 + +### 2.1 Admission 경계와 오류 계약 + +최종 검사는 앞선 width-one 작업이 끝난 뒤 scheduler thread에서 request GPU work 전에 실행된다. Provider는 HTTP route가 streaming header를 확정하기 전에 one-shot admission event를 기다린다. 거절 시 `error.type`과 `error.code`가 `request_memory_budget_exceeded`이고 `estimated_bytes`와 `budget_bytes`를 포함하는 HTTP 503을 반환한다. 기존 context 검증은 우선순위를 유지하며 HTTP 400을 반환한다. Cancellation과 worker disconnect 경로는 handshake를 완료하거나 취소하므로, 중단된 요청이나 지원하지 않는 요청이 waiter를 영구 대기시키지 않는다. + +리뷰에서 두 route-level gap을 닫았다. Guarded request는 final single-request check를 우회하던 batched prefill의 sequential fallback에 진입할 수 없으며, unsupported path는 자신이 발생시키지 않는 admission event를 기다리지 않고 즉시 acknowledge한다. Legacy `--no-batch`는 순차 실행을 유지하면서 같은 size-one scheduler guard를 사용한다. + +### 2.2 Checked estimate와 budget + +Pure estimator는 normalized loaded `NemotronHConfig`, loaded embedding output dtype, 실제 cold `PrefillPlan` 조각, architecture-aware KV geometry, 현재 MLX memory snapshot을 사용한다. 모든 곱셈과 합산은 checked arithmetic을 사용하며 dimension이나 budget을 알 수 없거나 dtype이 잘못되거나 overflow가 발생하면 fail closed한다. Envelope는 현재 active allocation, growth와 snapshot ownership을 포함한 cold attention KV, recurrent 및 convolution state, layer별 activation, logits, MoE intermediate, 보수적인 float32 attention score, model internal chunk 크기의 float32 SSD buffer를 계산한다. + +Default budget은 device recommendation, mlxcel이 실제 적용한 wired cap, allocator cap의 양의 유한값 중 최솟값이다. `MLXCEL_REQUEST_MEMORY_BUDGET`은 기존 binary size grammar를 사용하지만 이 ceiling을 낮출 수만 있다. 명시적으로 설정한 empty, zero, `none`, negative, nonfinite, overflow 값은 startup을 중단한다. + +### 2.3 Residency와 cache 정책 + +Metal Nemotron-H weight는 warmup이 꺼져 있어도 shared construction boundary에서 평가된다. 따라서 lazy graph 때문에 첫 요청이 실제보다 작은 resident baseline을 관측하지 않는다. Constructor는 fused checkpoint adapter와 VLM 또는 VoiceChat에서 포함해 사용하는 경우에도 적용되지만, admission은 eligible text serving에서만 활성화된다. Arbitrary-rank float32 workspace가 모델링되지 않은 runtime unfused LoRA는 startup에서 거절하고 staged adapter state를 비운다. + +Guarded request는 cold prefill하며 admission 전에 prompt snapshot을 restore하거나 donate하지 않고 background prompt-cache warmup도 건너뛴다. Retained snapshot은 active memory에 계속 포함된다. 이 보수적 정책은 optimistic cache reuse나 fused kernel을 사용하면 실행 가능한 요청도 거절할 수 있지만, 최종 측정 전에 request work를 실행하지 않는다. + +### 2.4 범위와 호환성 + +정책은 effective width one인 local, non-speculative, text-only Metal Nemotron-H로 제한된다. CUDA, ROCm, CPU, 다른 family, multimodal input, tensor 또는 pipeline parallel, disaggregated worker handoff, speculative decoding, multi-slot reservation은 기존 정책을 유지한다. 새 dependency나 data format 변경은 없으며 model numeric도 바꾸지 않는다. + +## 3. 기술적 선택과 그 이유 + +| 결정 | 선택한 방식 | 선택하지 않은 방식 | 트레이드오프 | +|------|-------------|--------------------|-------------| +| 검사 위치 | Production prefill 호출 지점의 최종 scheduler admission | Startup estimate 또는 HTTP-only heuristic | 실제 request geometry와 resident state를 쓰지만 HTTP-worker handshake가 필요함 | +| Cache credit | Cold KV와 retained active allocation을 계산 | Prompt snapshot hit가 요청 크기를 줄인다고 가정 | 예측 가능한 안전성을 얻는 대신 false rejection과 cache-hit latency 손실 가능 | +| Override | 알려진 runtime limit보다 낮추기만 가능한 cap | Environment variable로 budget을 높이거나 새로 생성 | 더 엄격한 정책은 시험할 수 있지만 device limit을 우회할 수 없음 | +| Unknown value | Fail closed | Geometry, budget, arithmetic을 알 수 없을 때 admit | 안전한 실패지만 잘못된 설정이나 unsupported geometry는 service를 거절할 수 있음 | +| Runtime LoRA | Unfused adapter를 construction에서 거절 | Arbitrary-rank workspace를 추측 | Workspace 모델이 생기기 전까지 명확한 startup failure 제공 | + +## 4. 검증 + +- Request-memory suite: 26개 통과. Budget boundary, output growth, checked overflow, invalid override, cache accounting, context precedence, cancellation, zero-forward rejection, unsupported path, batched-prefill bypass, legacy serving, pre-SSE error를 포함한다. +- Nemotron-H suite: 23개 통과, 기존 3개 ignored. Runtime-LoRA rejection과 staging cleanup을 포함한다. +- Focused provider 및 native-route binary에서 각각 30개와 34개 test가 통과했다. 이 test들은 26개 request-memory test와 중복되므로 unique case로 합산하지 않는다. 5개 generation dialect와 provider admission lifecycle을 포함한다. +- `cargo check --tests --features metal,accelerate`는 51.50초에 통과했고, `cargo clippy --lib --tests --features metal,accelerate -- -D warnings`는 5분 5초에 통과했다. 알려진 C++ unused-capture warning은 Rust Clippy gate를 실패시키지 않았다. +- `cargo build --release --features metal,accelerate`는 측정 source에서 9분 42초에 통과했고 in-place-setter 수정 뒤에도 8분 59초에 통과했다. +- Reviewed source의 real-checkpoint lifecycle은 normal fresh server 3개, `--no-warmup` server 1개, legacy `--no-batch` server 1개에서 통과했다. Post-setter normal 및 legacy lifecycle smoke도 같은 rejection/recovery 동작을 반복했다. Qwen 2.5 7B second-family smoke에서는 setter 수정 전후 모두 streaming과 non-streaming request가 계속 admit되었다. +- GitHub two-host logical pipeline job은 unoptimized worker에 추가된 by-value `BatchScheduler` builder frame 하나 때문에 stack overflow를 보고했다. Commit `9d2a1584`는 admission 동작을 바꾸지 않고 이를 in-place setter로 교체했으며 targeted test 26개가 통과했다. 이전에 실패한 logical job도 수정된 commit에서 통과했다. 전체 workflow와 수정된 release build에는 아직 대기 중인 job이 있어 fully green으로 보고하지 않았다. + +## 5. 벤치마크 근거 + +추적되는 benchmark record는 `docs/benchmark_results/nemotron-h-request-memory-m1ultra-2026-10-13.md`이고 machine-readable artifact는 같은 이름의 `data/` directory에 있다. M1 Ultra 128 GB의 초기 calibration에서 32 GiB budget 아래 6,001-token 요청이 admit되었고 estimate는 27.34 GiB, 측정된 process peak는 17.8 GiB였다. Harness는 이 실행을 rejection evidence로 잘못 보고하지 않았다. 낮은 cap을 사용한 중간 실행은 calibration으로 보존하지만 final routing 및 cache review fix 전 결과이다. + +최종 reviewed binary(`1c4673addeb123b8d3ef2758f69dc0c54811d1d061911dbcb86aa486cc6065c5`)로 normal fresh-server 3회 반복을 실행했다. Long streaming 및 non-streaming request 12건은 모두 SSE 전에 typed 503을 반환했고, 바로 뒤의 short request 12건과 medium request 3건은 모두 200을 반환했다. Beyond-context request 3건은 400 우선순위를 유지했고 health는 계속 200이었다. Medium estimate는 세 번 모두 22.328 GiB이고 process physical peak는 17.6 GiB였다. Long estimate는 27.337–27.351 GiB, short estimate는 17.813–17.865 GiB였으며 long-rejection process peak는 16.9 GiB였다. 1분 host load가 6.44, 6.90, 11.01이었으므로 이는 latency 주장이 아니라 lifecycle 및 safety 근거이다. + +No-warmup server 1개에서 long 503 4건, 이어지는 short 200 4건, medium 200, context 400, 정상 health를 반복했다. Legacy `--no-batch` server 1개에서 long 503 4건, 이어지는 short 200 4건, medium 200을 반복했다. Unchunked largest forward 때문에 legacy estimate는 의도적으로 보수적인 45.506–45.634 GiB였다. Qwen 2.5 7B server는 같은 budget에서 streaming 및 non-streaming request 모두 200을 반환하고 health를 유지해 shared route handshake가 second family에 Nemotron-H rejection policy를 적용하지 않음을 확인했다. + +Post-setter release artifact(`e7a5cc9997a8c0eeceba6a69769353eed35395e906b270d2c62a5f437e2a7062`)는 normal lifecycle 1회와 legacy lifecycle 1회를 반복했다. 각각 typed long-request 503 4건을 SSE 전에 반환했고, short-request 200 4건이 모두 완료되었으며 medium 200과 health 200을 유지했다. Normal context check는 계속 400이었다. Estimate는 측정 artifact의 범위와 일치했다. Qwen control도 streaming 및 non-streaming request에서 다시 200을 반환하고 health 200을 유지했다. 이 post-setter artifact와 benchmark record는 `62c1cf03`에 commit되었다. + +## 6. 변경 요약 + +| 카테고리 | 요약 | +|----------|------| +| 안전성 | Checked fail-closed request envelope와 lower-only runtime budget 추가 | +| HTTP | 지원하는 generation route에서 SSE header 전에 typed 503 반환 | +| Scheduler | 최종 width-one prefill 경계에서 guarded admission을 직렬화하고 batched-prefill bypass 차단 | +| Model loading | Metal Nemotron-H weight를 materialize하고 모델링하지 않은 runtime LoRA를 안전하게 거절 | +| Cache 동작 | Guarded request의 pre-admission snapshot restore/donation 및 background warmup 비활성화 | +| 테스트와 문서 | Estimator, scheduler, provider, route, model regression test와 operator 문서 및 benchmark harness 추가 | + +## 7. 후속 조치와 모니터링 + +- Merge 전에 수정된 release build와 남은 CI job을 확인한다. 근거 없이 fully green workflow로 보고하지 않는다. +- False-rejection rate, estimate 대비 observed peak 비율, retained prompt-cache 기여도, post-admission allocation failure를 모니터링한다. +- 향후 정책은 다른 family, runtime LoRA rank, multi-slot reservation, distributed path, optimistic cache reuse를 모델링할 수 있지만 이 PR은 이를 보장하지 않는다. + +## 참고 자료 + +- Issue #2291: Metal prefill 전 over-budget Nemotron-H request 거절. +- PR #2290 및 issue #2281: Nemotron-H SSD prefill scratch 제한과 long-prompt 측정 baseline. +- `docs/request-memory-admission.md`: 운영자 계약, 제한 사항, error shape. diff --git a/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/calibration.json b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/calibration.json new file mode 100644 index 000000000..85af21a24 --- /dev/null +++ b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/calibration.json @@ -0,0 +1,51 @@ +{ + "stage": "initial calibration before load-time weight materialization fix", + "binary_sha256": "aaddeef2480cd829b5b41c2f6ddbcf2ebf9fa73f63f4914afa4ba4193167aebc", + "budget": "32G", + "outcome": "correctly admitted; harness aborted instead of claiming rejection", + "result": { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-cc9d645c-4ea8-4f53-a556-d228a7fdda66", + "object": "text_completion", + "created": 1791902423, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "10000000", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 6001, + "completion_tokens": 8, + "total_tokens": 6009, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 16.983042375009973, + "case": "reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 19112604467, + "current_bytes": 18575733555 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29353208076, + "budget_bytes": 34359738368, + "resident_bytes": 17778075916, + "allocator_cache_bytes": 51414376, + "admitted": true + } + } +} diff --git a/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/candidate2-no-warmup.json b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/candidate2-no-warmup.json new file mode 100644 index 000000000..b1d0f3cb7 --- /dev/null +++ b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/candidate2-no-warmup.json @@ -0,0 +1,376 @@ +[ + { + "rep": 1, + "command": [ + "target/release/mlxcel-server", + "--model", + "models/mlx/nvidia-nemotron-3-nano-30b-a3b-4bit", + "--ctx-size", + "8192", + "--jinja", + "--parallel", + "1", + "--host", + "127.0.0.1", + "--port", + "60011", + "--alias", + "repro", + "--offline", + "--no-warmup" + ], + "budget": "24G", + "load_before": [ + 11.06884765625, + 8.9833984375, + 7.34765625 + ], + "requests": [ + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994048, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994048, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 1.867893541988451, + "case": "reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18038862643, + "current_bytes": 18038862643 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994048, + "budget_bytes": 25769803776, + "resident_bytes": 17777861888, + "allocator_cache_bytes": 0, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-dac49462-6a76-40d4-a5f0-09de0d30534d", + "object": "text_completion", + "created": 1791903389, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "10000000", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 17, + "completion_tokens": 8, + "total_tokens": 25, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.5374125000089407, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584448, + "budget_bytes": 25769803776, + "resident_bytes": 17777861888, + "allocator_cache_bytes": 0, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.007565292005892843, + "case": "reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68804146, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"cmpl-14eeec72-e1c9-4e4b-afb6-a7f53ef23954\",\"object\":\"text_completion\",\"created\":1791903393,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"1\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-14eeec72-e1c9-4e4b-afb6-a7f53ef23954\",\"object\":\"text_completion\",\"created\":1791903393,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-14eeec72-e1c9-4e4b-afb6-a7f53ef23954\",\"object\":\"text_completion\",\"created\":1791903393,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-14eeec72-e1c9-4e4b-afb6-a7f53ef23954\",\"object\":\"text_completion\",\"created\":1791903393,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-14eeec72-e1c9-4e4b-afb6-a7f53ef23954\",\"object\":\"text_completion\",\"created\":1791903393,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-14eeec72-e1c9-4e4b-afb6-a7f53ef23954\",\"object\":\"text_completion\",\"created\":1791903393,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-14eeec72-e1c9-4e4b-afb6-a7f53ef23954\",\"object\":\"text_completion\",\"created\":1791903393,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-14eeec72-e1c9-4e4b-afb6-a7f53ef23954\",\"object\":\"text_completion\",\"created\":1791903393,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-14eeec72-e1c9-4e4b-afb6-a7f53ef23954\",\"object\":\"text_completion\",\"created\":1791903393,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"\",\"finish_reason\":\"length\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.456966666970402, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68804146, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=48969240844, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 48969240844, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.012999083031900227, + "case": "reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 6014, + "estimated_bytes": 48969240844, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68786986, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "chatcmpl-579b31b7-bc03-4b04-a30a-0d8118bdd313", + "object": "chat.completion", + "created": 1791903397, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "OK" + }, + "finish_reason": "stop", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 36, + "completion_tokens": 1, + "total_tokens": 37, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.42439804098103195, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68786986, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=49071280396, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 49071280396, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.013165250013116747, + "case": "reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 6014, + "estimated_bytes": 49071280396, + "budget_bytes": 25769803776, + "resident_bytes": 17879901452, + "allocator_cache_bytes": 606292, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"chatcmpl-8198f50d-2376-4d4d-a4de-84ed8e3f6b8d\",\"object\":\"chat.completion.chunk\",\"created\":1791903400,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-8198f50d-2376-4d4d-a4de-84ed8e3f6b8d\",\"object\":\"chat.completion.chunk\",\"created\":1791903401,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"content\":\"OK\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-8198f50d-2376-4d4d-a4de-84ed8e3f6b8d\",\"object\":\"chat.completion.chunk\",\"created\":1791903401,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.4262506249942817, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18253611008, + "current_bytes": 18253611008 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19284061452, + "budget_bytes": 25769803776, + "resident_bytes": 17879901452, + "allocator_cache_bytes": 606292, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-4d6513dc-560d-4aff-bede-c6e920e885bd", + "object": "text_completion", + "created": 1791903407, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "\n\n\n**Step\u2011by\u2011step", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 1501, + "completion_tokens": 8, + "total_tokens": 1509, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 4.238450374978129, + "case": "accepted_1500", + "admission": { + "prompt_tokens": 1501, + "output_tokens": 8, + "largest_forward_tokens": 1501, + "estimated_bytes": 24076936844, + "budget_bytes": 25769803776, + "resident_bytes": 17879934220, + "allocator_cache_bytes": 75550782, + "admitted": true + }, + "footprint": { + "peak_bytes": 19005230284, + "current_bytes": 18575733555 + } + } + ], + "log": "/tmp/mlxcel-2291-no-warmup-results.json.rep1.log", + "ready_s": 0.5280595830408856, + "startup_footprint": { + "peak_bytes": 2147483648, + "current_bytes": 2147483648 + }, + "context_rejection": { + "status": 400, + "content_type": "application/json", + "body": { + "error": { + "message": "request (8401 tokens) exceeds the available context size (8192 tokens), try increasing it", + "type": "exceed_context_size_error", + "code": null + } + }, + "elapsed_s": 0.009642707998864353 + }, + "final_health": { + "status": 200, + "content_type": "application/json", + "body": { + "status": "ok" + }, + "elapsed_s": 0.0006314999773167074 + }, + "load_after": [ + 10.6416015625, + 9.04833984375, + 7.4169921875 + ] + } +] \ No newline at end of file diff --git a/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/candidate2.json b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/candidate2.json new file mode 100644 index 000000000..b76694948 --- /dev/null +++ b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/candidate2.json @@ -0,0 +1,1121 @@ +[ + { + "rep": 1, + "command": [ + "target/release/mlxcel-server", + "--model", + "models/mlx/nvidia-nemotron-3-nano-30b-a3b-4bit", + "--ctx-size", + "8192", + "--jinja", + "--parallel", + "1", + "--host", + "127.0.0.1", + "--port", + "59546", + "--alias", + "repro", + "--offline" + ], + "budget": "24G", + "load_before": [ + 6.34765625, + 5.89111328125, + 6.0771484375 + ], + "requests": [ + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.008346749993506819, + "case": "reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51430760, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-cb606036-e083-4ffc-8a33-ab92f8634f48", + "object": "text_completion", + "created": 1791903237, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "10000000", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 17, + "completion_tokens": 8, + "total_tokens": 25, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.4797135000117123, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51430760, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.007507458969485015, + "case": "reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68725802, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"cmpl-867e0d6c-45d5-4398-9909-1a76ac468464\",\"object\":\"text_completion\",\"created\":1791903241,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"1\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-867e0d6c-45d5-4398-9909-1a76ac468464\",\"object\":\"text_completion\",\"created\":1791903241,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-867e0d6c-45d5-4398-9909-1a76ac468464\",\"object\":\"text_completion\",\"created\":1791903241,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-867e0d6c-45d5-4398-9909-1a76ac468464\",\"object\":\"text_completion\",\"created\":1791903241,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-867e0d6c-45d5-4398-9909-1a76ac468464\",\"object\":\"text_completion\",\"created\":1791903241,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-867e0d6c-45d5-4398-9909-1a76ac468464\",\"object\":\"text_completion\",\"created\":1791903241,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-867e0d6c-45d5-4398-9909-1a76ac468464\",\"object\":\"text_completion\",\"created\":1791903241,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-867e0d6c-45d5-4398-9909-1a76ac468464\",\"object\":\"text_completion\",\"created\":1791903241,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-867e0d6c-45d5-4398-9909-1a76ac468464\",\"object\":\"text_completion\",\"created\":1791903241,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"\",\"finish_reason\":\"length\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.5132700420217589, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68725802, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=48969240844, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 48969240844, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.013727083976846188, + "case": "reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 6014, + "estimated_bytes": 48969240844, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68873266, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "chatcmpl-cf3381c5-18ee-4d5a-b313-68a05e284670", + "object": "chat.completion", + "created": 1791903245, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "OK" + }, + "finish_reason": "stop", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 36, + "completion_tokens": 1, + "total_tokens": 37, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.47002000000793487, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68873266, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=49071214860, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 49071214860, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.013277416001074016, + "case": "reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 6014, + "estimated_bytes": 49071214860, + "budget_bytes": 25769803776, + "resident_bytes": 17879835916, + "allocator_cache_bytes": 606292, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"chatcmpl-6809a07b-3f59-4f5f-8ebd-220f2421ecba\",\"object\":\"chat.completion.chunk\",\"created\":1791903249,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-6809a07b-3f59-4f5f-8ebd-220f2421ecba\",\"object\":\"chat.completion.chunk\",\"created\":1791903249,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"content\":\"OK\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-6809a07b-3f59-4f5f-8ebd-220f2421ecba\",\"object\":\"chat.completion.chunk\",\"created\":1791903250,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.4515910420450382, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18253611008, + "current_bytes": 18253611008 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19283995916, + "budget_bytes": 25769803776, + "resident_bytes": 17879835916, + "allocator_cache_bytes": 606292, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-6984b0de-35f3-4275-ba66-b44518495a49", + "object": "text_completion", + "created": 1791903256, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "\n\n\n**Step\u2011by\u2011step", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 1501, + "completion_tokens": 8, + "total_tokens": 1509, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 4.588350916979834, + "case": "accepted_1500", + "admission": { + "prompt_tokens": 1501, + "output_tokens": 8, + "largest_forward_tokens": 1501, + "estimated_bytes": 24076936844, + "budget_bytes": 25769803776, + "resident_bytes": 17879934220, + "allocator_cache_bytes": 73815358, + "admitted": true + }, + "footprint": { + "peak_bytes": 19005230284, + "current_bytes": 18575733555 + } + } + ], + "log": "/tmp/mlxcel-2291-load-fixed-results.json.rep1.log", + "ready_s": 7.9152185000129975, + "startup_footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "context_rejection": { + "status": 400, + "content_type": "application/json", + "body": { + "error": { + "message": "request (8401 tokens) exceeds the available context size (8192 tokens), try increasing it", + "type": "exceed_context_size_error", + "code": null + } + }, + "elapsed_s": 0.009612083027604967 + }, + "final_health": { + "status": 200, + "content_type": "application/json", + "body": { + "status": "ok" + }, + "elapsed_s": 0.0007252920186147094 + }, + "load_after": [ + 7.09619140625, + 6.13623046875, + 6.15869140625 + ] + }, + { + "rep": 2, + "command": [ + "target/release/mlxcel-server", + "--model", + "models/mlx/nvidia-nemotron-3-nano-30b-a3b-4bit", + "--ctx-size", + "8192", + "--jinja", + "--parallel", + "1", + "--host", + "127.0.0.1", + "--port", + "59685", + "--alias", + "repro", + "--offline" + ], + "budget": "24G", + "load_before": [ + 7.09619140625, + 6.13623046875, + 6.15869140625 + ], + "requests": [ + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.007828917005099356, + "case": "reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51430760, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-a70c9d25-76f7-44d3-b69c-656cc4710847", + "object": "text_completion", + "created": 1791903265, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "10000000", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 17, + "completion_tokens": 8, + "total_tokens": 25, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.5078455000184476, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51430760, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.006720625038724393, + "case": "reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68714290, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"cmpl-d1d7c5f5-88bf-4ea5-8913-3fbbb7708c80\",\"object\":\"text_completion\",\"created\":1791903269,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"1\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-d1d7c5f5-88bf-4ea5-8913-3fbbb7708c80\",\"object\":\"text_completion\",\"created\":1791903269,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-d1d7c5f5-88bf-4ea5-8913-3fbbb7708c80\",\"object\":\"text_completion\",\"created\":1791903269,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-d1d7c5f5-88bf-4ea5-8913-3fbbb7708c80\",\"object\":\"text_completion\",\"created\":1791903270,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-d1d7c5f5-88bf-4ea5-8913-3fbbb7708c80\",\"object\":\"text_completion\",\"created\":1791903270,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-d1d7c5f5-88bf-4ea5-8913-3fbbb7708c80\",\"object\":\"text_completion\",\"created\":1791903270,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-d1d7c5f5-88bf-4ea5-8913-3fbbb7708c80\",\"object\":\"text_completion\",\"created\":1791903270,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-d1d7c5f5-88bf-4ea5-8913-3fbbb7708c80\",\"object\":\"text_completion\",\"created\":1791903270,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-d1d7c5f5-88bf-4ea5-8913-3fbbb7708c80\",\"object\":\"text_completion\",\"created\":1791903270,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"\",\"finish_reason\":\"length\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.5251992919947952, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68714290, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=48969240844, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 48969240844, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.013860124978236854, + "case": "reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 6014, + "estimated_bytes": 48969240844, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68725810, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "chatcmpl-88fe5edb-18f8-4499-87d2-16598ef7a537", + "object": "chat.completion", + "created": 1791903274, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "OK" + }, + "finish_reason": "stop", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 36, + "completion_tokens": 1, + "total_tokens": 37, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.509831834002398, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68725810, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=49071214860, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 49071214860, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.013520915992558002, + "case": "reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 6014, + "estimated_bytes": 49071214860, + "budget_bytes": 25769803776, + "resident_bytes": 17879835916, + "allocator_cache_bytes": 606292, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"chatcmpl-160bf966-5bd4-40f4-befd-b53bd0deff2f\",\"object\":\"chat.completion.chunk\",\"created\":1791903278,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-160bf966-5bd4-40f4-befd-b53bd0deff2f\",\"object\":\"chat.completion.chunk\",\"created\":1791903278,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"content\":\"OK\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-160bf966-5bd4-40f4-befd-b53bd0deff2f\",\"object\":\"chat.completion.chunk\",\"created\":1791903278,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.5110327499569394, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18253611008, + "current_bytes": 18253611008 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19283995916, + "budget_bytes": 25769803776, + "resident_bytes": 17879835916, + "allocator_cache_bytes": 606292, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-62f93a95-ff20-4da8-b549-a3861150835c", + "object": "text_completion", + "created": 1791903285, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "\n\n\n**Step\u2011by\u2011step", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 1501, + "completion_tokens": 8, + "total_tokens": 1509, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 4.4255018749972805, + "case": "accepted_1500", + "admission": { + "prompt_tokens": 1501, + "output_tokens": 8, + "largest_forward_tokens": 1501, + "estimated_bytes": 24076936844, + "budget_bytes": 25769803776, + "resident_bytes": 17879934220, + "allocator_cache_bytes": 75481662, + "admitted": true + }, + "footprint": { + "peak_bytes": 19005230284, + "current_bytes": 18575733555 + } + } + ], + "log": "/tmp/mlxcel-2291-load-fixed-results.json.rep2.log", + "ready_s": 3.0593283330090344, + "startup_footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "context_rejection": { + "status": 400, + "content_type": "application/json", + "body": { + "error": { + "message": "request (8401 tokens) exceeds the available context size (8192 tokens), try increasing it", + "type": "exceed_context_size_error", + "code": null + } + }, + "elapsed_s": 0.015272209013346583 + }, + "final_health": { + "status": 200, + "content_type": "application/json", + "body": { + "status": "ok" + }, + "elapsed_s": 0.0007275000098161399 + }, + "load_after": [ + 10.7333984375, + 7.03271484375, + 6.478515625 + ] + }, + { + "rep": 3, + "command": [ + "target/release/mlxcel-server", + "--model", + "models/mlx/nvidia-nemotron-3-nano-30b-a3b-4bit", + "--ctx-size", + "8192", + "--jinja", + "--parallel", + "1", + "--host", + "127.0.0.1", + "--port", + "59781", + "--alias", + "repro", + "--offline" + ], + "budget": "24G", + "load_before": [ + 10.7333984375, + 7.03271484375, + 6.478515625 + ], + "requests": [ + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.008214791014324874, + "case": "reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51426664, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-0cd06c88-9fae-450b-8c27-f9793032493d", + "object": "text_completion", + "created": 1791903295, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "10000000", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 17, + "completion_tokens": 8, + "total_tokens": 25, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.5632293749949895, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51426664, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.015676875016652048, + "case": "reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68710194, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"cmpl-38d321dd-09cf-475a-abcc-0196c7870609\",\"object\":\"text_completion\",\"created\":1791903299,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"1\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-38d321dd-09cf-475a-abcc-0196c7870609\",\"object\":\"text_completion\",\"created\":1791903299,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-38d321dd-09cf-475a-abcc-0196c7870609\",\"object\":\"text_completion\",\"created\":1791903299,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-38d321dd-09cf-475a-abcc-0196c7870609\",\"object\":\"text_completion\",\"created\":1791903299,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-38d321dd-09cf-475a-abcc-0196c7870609\",\"object\":\"text_completion\",\"created\":1791903299,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-38d321dd-09cf-475a-abcc-0196c7870609\",\"object\":\"text_completion\",\"created\":1791903299,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-38d321dd-09cf-475a-abcc-0196c7870609\",\"object\":\"text_completion\",\"created\":1791903299,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-38d321dd-09cf-475a-abcc-0196c7870609\",\"object\":\"text_completion\",\"created\":1791903300,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-38d321dd-09cf-475a-abcc-0196c7870609\",\"object\":\"text_completion\",\"created\":1791903300,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"\",\"finish_reason\":\"length\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.47889362496789545, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68710194, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=48969240844, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 48969240844, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.013379583018831909, + "case": "reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 6014, + "estimated_bytes": 48969240844, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68809778, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "chatcmpl-5e1610b4-31c6-4fbe-bba7-b56fc0c707eb", + "object": "chat.completion", + "created": 1791903304, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "OK" + }, + "finish_reason": "stop", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 36, + "completion_tokens": 1, + "total_tokens": 37, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.651069333020132, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68809778, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=49071821128, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 49071821128, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.03979179199086502, + "case": "reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 6014, + "estimated_bytes": 49071821128, + "budget_bytes": 25769803776, + "resident_bytes": 17880442184, + "allocator_cache_bytes": 24, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"chatcmpl-d362e664-c9ca-4228-a1a7-3cccd4f1d723\",\"object\":\"chat.completion.chunk\",\"created\":1791903309,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-d362e664-c9ca-4228-a1a7-3cccd4f1d723\",\"object\":\"chat.completion.chunk\",\"created\":1791903309,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"content\":\"OK\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-d362e664-c9ca-4228-a1a7-3cccd4f1d723\",\"object\":\"chat.completion.chunk\",\"created\":1791903309,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.69081629201537, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18253611008, + "current_bytes": 18253611008 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19283995916, + "budget_bytes": 25769803776, + "resident_bytes": 17879835916, + "allocator_cache_bytes": 606292, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-62117ba4-5b6f-411b-94d0-fb2d0fbe9e39", + "object": "text_completion", + "created": 1791903317, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "\n\n\n**Step\u2011by\u2011step", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 1501, + "completion_tokens": 8, + "total_tokens": 1509, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 5.1793864170322195, + "case": "accepted_1500", + "admission": { + "prompt_tokens": 1501, + "output_tokens": 8, + "largest_forward_tokens": 1501, + "estimated_bytes": 24076920460, + "budget_bytes": 25769803776, + "resident_bytes": 17879917836, + "allocator_cache_bytes": 73739582, + "admitted": true + }, + "footprint": { + "peak_bytes": 19005230284, + "current_bytes": 18575733555 + } + } + ], + "log": "/tmp/mlxcel-2291-load-fixed-results.json.rep3.log", + "ready_s": 3.329083999968134, + "startup_footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "context_rejection": { + "status": 400, + "content_type": "application/json", + "body": { + "error": { + "message": "request (8401 tokens) exceeds the available context size (8192 tokens), try increasing it", + "type": "exceed_context_size_error", + "code": null + } + }, + "elapsed_s": 0.011463707953225821 + }, + "final_health": { + "status": 200, + "content_type": "application/json", + "body": { + "status": "ok" + }, + "elapsed_s": 0.007230375020299107 + }, + "load_after": [ + 15.64404296875, + 8.61865234375, + 7.07470703125 + ] + } +] \ No newline at end of file diff --git a/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/final-no-batch.json b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/final-no-batch.json new file mode 100644 index 000000000..84614c157 --- /dev/null +++ b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/final-no-batch.json @@ -0,0 +1,365 @@ +[ + { + "rep": 1, + "binary_sha256": "1c4673addeb123b8d3ef2758f69dc0c54811d1d061911dbcb86aa486cc6065c5", + "command": [ + "target/release/mlxcel-server", + "--model", + "models/mlx/nvidia-nemotron-3-nano-30b-a3b-4bit", + "--ctx-size", + "8192", + "--jinja", + "--parallel", + "1", + "--host", + "127.0.0.1", + "--port", + "63420", + "--alias", + "repro", + "--offline", + "--no-batch" + ], + "budget": "24G", + "load_before": [ + 13.87451171875, + 11.30224609375, + 9.78857421875 + ], + "requests": [ + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=48861207180, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 48861207180, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.007982834009453654, + "case": "reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 6001, + "estimated_bytes": 48861207180, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51430760, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-2849d85e-07ea-4716-af5e-28f981cafbf3", + "object": "text_completion", + "created": 1791904775, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "10000000", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 17, + "completion_tokens": 8, + "total_tokens": 25, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.4176034589763731, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51430760, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=48861207180, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 48861207180, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.007453124970197678, + "case": "reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 6001, + "estimated_bytes": 48861207180, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68725802, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"cmpl-66e89a2e-970a-4097-ade6-269ad617ec53\",\"object\":\"text_completion\",\"created\":1791904779,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"1\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-66e89a2e-970a-4097-ade6-269ad617ec53\",\"object\":\"text_completion\",\"created\":1791904779,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-66e89a2e-970a-4097-ade6-269ad617ec53\",\"object\":\"text_completion\",\"created\":1791904779,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-66e89a2e-970a-4097-ade6-269ad617ec53\",\"object\":\"text_completion\",\"created\":1791904779,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-66e89a2e-970a-4097-ade6-269ad617ec53\",\"object\":\"text_completion\",\"created\":1791904779,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-66e89a2e-970a-4097-ade6-269ad617ec53\",\"object\":\"text_completion\",\"created\":1791904779,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-66e89a2e-970a-4097-ade6-269ad617ec53\",\"object\":\"text_completion\",\"created\":1791904779,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-66e89a2e-970a-4097-ade6-269ad617ec53\",\"object\":\"text_completion\",\"created\":1791904779,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-66e89a2e-970a-4097-ade6-269ad617ec53\",\"object\":\"text_completion\",\"created\":1791904779,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"\",\"finish_reason\":\"length\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.4752556250314228, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68725802, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=48998894860, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 48998894860, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.013013124989811331, + "case": "reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 6020, + "estimated_bytes": 48998894860, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68882730, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "chatcmpl-c2b97fd2-96b8-47b8-a8fa-cd2abd50819d", + "object": "chat.completion", + "created": 1791904783, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "OK" + }, + "finish_reason": "stop", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 36, + "completion_tokens": 1, + "total_tokens": 37, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.44092629197984934, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68882730, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=48998894860, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 48998894860, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.01316979096736759, + "case": "reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 6020, + "estimated_bytes": 48998894860, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 73814078, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"chatcmpl-c6bf1722-2c62-4f76-aaec-80e9f55a4de1\",\"object\":\"chat.completion.chunk\",\"created\":1791904787,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-c6bf1722-2c62-4f76-aaec-80e9f55a4de1\",\"object\":\"chat.completion.chunk\",\"created\":1791904787,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"content\":\"OK\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-c6bf1722-2c62-4f76-aaec-80e9f55a4de1\",\"object\":\"chat.completion.chunk\",\"created\":1791904787,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.4620456249685958, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 73814078, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-1aadfc62-4b03-4118-a604-b154d55644e6", + "object": "text_completion", + "created": 1791904793, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "\n\n\n**Step\u2011by\u2011step", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 1501, + "completion_tokens": 8, + "total_tokens": 1509, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 4.411625874985475, + "case": "accepted_1500", + "admission": { + "prompt_tokens": 1501, + "output_tokens": 8, + "largest_forward_tokens": 1501, + "estimated_bytes": 23974864524, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 73817662, + "admitted": true + }, + "footprint": { + "peak_bytes": 18897856102, + "current_bytes": 18468359372 + } + } + ], + "log": "/tmp/mlxcel-2291-reviewed-no-batch.json.rep1.log", + "ready_s": 2.857113790989388, + "startup_footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "final_health": { + "status": 200, + "content_type": "application/json", + "body": { + "status": "ok" + }, + "elapsed_s": 0.000991291948594153 + }, + "load_after": [ + 10.72607421875, + 10.79248046875, + 9.6552734375 + ] + } +] \ No newline at end of file diff --git a/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/final-no-warmup.json b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/final-no-warmup.json new file mode 100644 index 000000000..d9abbe127 --- /dev/null +++ b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/final-no-warmup.json @@ -0,0 +1,377 @@ +[ + { + "rep": 1, + "binary_sha256": "1c4673addeb123b8d3ef2758f69dc0c54811d1d061911dbcb86aa486cc6065c5", + "command": [ + "target/release/mlxcel-server", + "--model", + "models/mlx/nvidia-nemotron-3-nano-30b-a3b-4bit", + "--ctx-size", + "8192", + "--jinja", + "--parallel", + "1", + "--host", + "127.0.0.1", + "--port", + "63343", + "--alias", + "repro", + "--offline", + "--no-warmup" + ], + "budget": "24G", + "load_before": [ + 16.88916015625, + 11.56298828125, + 9.83203125 + ], + "requests": [ + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994048, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994048, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 1.8267305829795077, + "case": "reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18038862643, + "current_bytes": 18038862643 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994048, + "budget_bytes": 25769803776, + "resident_bytes": 17777861888, + "allocator_cache_bytes": 0, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-5e29c2b2-ec4a-455f-9b67-8086d32aaaeb", + "object": "text_completion", + "created": 1791904747, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "10000000", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 17, + "completion_tokens": 8, + "total_tokens": 25, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.5635405420325696, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584448, + "budget_bytes": 25769803776, + "resident_bytes": 17777861888, + "allocator_cache_bytes": 0, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.010260917013511062, + "case": "reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68804146, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"cmpl-ec7a9e12-8d9b-4abb-87ef-3cb5094f21c5\",\"object\":\"text_completion\",\"created\":1791904751,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"1\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-ec7a9e12-8d9b-4abb-87ef-3cb5094f21c5\",\"object\":\"text_completion\",\"created\":1791904751,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-ec7a9e12-8d9b-4abb-87ef-3cb5094f21c5\",\"object\":\"text_completion\",\"created\":1791904751,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-ec7a9e12-8d9b-4abb-87ef-3cb5094f21c5\",\"object\":\"text_completion\",\"created\":1791904751,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-ec7a9e12-8d9b-4abb-87ef-3cb5094f21c5\",\"object\":\"text_completion\",\"created\":1791904751,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-ec7a9e12-8d9b-4abb-87ef-3cb5094f21c5\",\"object\":\"text_completion\",\"created\":1791904751,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-ec7a9e12-8d9b-4abb-87ef-3cb5094f21c5\",\"object\":\"text_completion\",\"created\":1791904751,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-ec7a9e12-8d9b-4abb-87ef-3cb5094f21c5\",\"object\":\"text_completion\",\"created\":1791904751,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-ec7a9e12-8d9b-4abb-87ef-3cb5094f21c5\",\"object\":\"text_completion\",\"created\":1791904751,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"\",\"finish_reason\":\"length\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.5161416249466129, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68804146, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29367936268, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.013149833015631884, + "case": "reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68807730, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "chatcmpl-f9cfc403-9f94-4871-86db-b5eed68f1cf5", + "object": "chat.completion", + "created": 1791904755, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "OK" + }, + "finish_reason": "stop", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 36, + "completion_tokens": 1, + "total_tokens": 37, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.47854308295063674, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68807730, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29367936268, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.013245334033854306, + "case": "reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 75534398, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"chatcmpl-186a8148-33ef-42c2-b7ba-bdbd561dc0b8\",\"object\":\"chat.completion.chunk\",\"created\":1791904759,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-186a8148-33ef-42c2-b7ba-bdbd561dc0b8\",\"object\":\"chat.completion.chunk\",\"created\":1791904759,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"content\":\"OK\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-186a8148-33ef-42c2-b7ba-bdbd561dc0b8\",\"object\":\"chat.completion.chunk\",\"created\":1791904759,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.45454725000308827, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 75534398, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-1f369b3b-c3da-4d6d-8f58-6b548a102547", + "object": "text_completion", + "created": 1791904766, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "\n\n\n**Step\u2011by\u2011step", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 1501, + "completion_tokens": 8, + "total_tokens": 1509, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 4.206305916944984, + "case": "accepted_1500", + "admission": { + "prompt_tokens": 1501, + "output_tokens": 8, + "largest_forward_tokens": 1501, + "estimated_bytes": 23974864524, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 73879614, + "admitted": true + }, + "footprint": { + "peak_bytes": 18897856102, + "current_bytes": 18468359372 + } + } + ], + "log": "/tmp/mlxcel-2291-reviewed-no-warmup.json.rep1.log", + "ready_s": 0.5307532910373993, + "startup_footprint": { + "peak_bytes": 2469606195, + "current_bytes": 2469606195 + }, + "context_rejection": { + "status": 400, + "content_type": "application/json", + "body": { + "error": { + "message": "request (8401 tokens) exceeds the available context size (8192 tokens), try increasing it", + "type": "exceed_context_size_error", + "code": null + } + }, + "elapsed_s": 0.009281207981985062 + }, + "final_health": { + "status": 200, + "content_type": "application/json", + "body": { + "status": "ok" + }, + "elapsed_s": 0.000654041999951005 + }, + "load_after": [ + 13.87451171875, + 11.30224609375, + 9.78857421875 + ] + } +] \ No newline at end of file diff --git a/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/final-qwen.json b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/final-qwen.json new file mode 100644 index 000000000..34e352305 --- /dev/null +++ b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/final-qwen.json @@ -0,0 +1,69 @@ +{ + "command": [ + "target/release/mlxcel-server", + "--model", + "models/mlx/qwen2.5-7b-instruct-4bit", + "--offline", + "--host", + "127.0.0.1", + "--port", + "63501", + "--parallel", + "1", + "--ctx-size", + "4096", + "--alias", + "repro" + ], + "budget": "1", + "binary_sha256": "1c4673addeb123b8d3ef2758f69dc0c54811d1d061911dbcb86aa486cc6065c5", + "load_before": [ + 10.72607421875, + 10.79248046875, + 9.6552734375 + ], + "requests": [ + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-8ee73f37-de8a-4a32-8239-73969c47f829", + "object": "text_completion", + "created": 1791904798, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": " Paris. It is located in the northern", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 5, + "completion_tokens": 8, + "total_tokens": 13, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.3231982080033049 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"cmpl-352ed05c-43a5-4a67-9c21-429e323547c2\",\"object\":\"text_completion\",\"created\":1791904798,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\" Paris\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-352ed05c-43a5-4a67-9c21-429e323547c2\",\"object\":\"text_completion\",\"created\":1791904798,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\".\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-352ed05c-43a5-4a67-9c21-429e323547c2\",\"object\":\"text_completion\",\"created\":1791904798,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\" It\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-352ed05c-43a5-4a67-9c21-429e323547c2\",\"object\":\"text_completion\",\"created\":1791904798,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\" is\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-352ed05c-43a5-4a67-9c21-429e323547c2\",\"object\":\"text_completion\",\"created\":1791904798,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\" located\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-352ed05c-43a5-4a67-9c21-429e323547c2\",\"object\":\"text_completion\",\"created\":1791904798,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\" in\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-352ed05c-43a5-4a67-9c21-429e323547c2\",\"object\":\"text_completion\",\"created\":1791904798,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\" the\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-352ed05c-43a5-4a67-9c21-429e323547c2\",\"object\":\"text_completion\",\"created\":1791904798,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\" northern\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-352ed05c-43a5-4a67-9c21-429e323547c2\",\"object\":\"text_completion\",\"created\":1791904798,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"\",\"finish_reason\":\"length\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.10041912499582395 + } + ], + "health": { + "status": 200, + "content_type": "application/json", + "body": { + "status": "ok" + }, + "elapsed_s": 0.0006689999718219042 + } +} \ No newline at end of file diff --git a/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/final-startup-excerpts.txt b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/final-startup-excerpts.txt new file mode 100644 index 000000000..2d330ae4e --- /dev/null +++ b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/final-startup-excerpts.txt @@ -0,0 +1,15 @@ +results: 2026-10-13T15:17:31.976554Z INFO mlxcel::models::nemotron_h: [NemotronH] Loading weights from safetensors... +results: 2026-10-13T15:17:38.032504Z INFO mlxcel::models::nemotron_h: [NemotronH] materialized weights for request memory admission resident_bytes=17777861888 +results: 2026-10-13T15:17:38.033136Z INFO mlxcel::models::nemotron_h: [NemotronH] Model loaded successfully +results: 2026-10-13T15:18:04.165160Z INFO mlxcel::models::nemotron_h: [NemotronH] Loading weights from safetensors... +results: 2026-10-13T15:18:06.091014Z INFO mlxcel::models::nemotron_h: [NemotronH] materialized weights for request memory admission resident_bytes=17777861888 +results: 2026-10-13T15:18:06.091481Z INFO mlxcel::models::nemotron_h: [NemotronH] Model loaded successfully +results: 2026-10-13T15:18:33.051049Z INFO mlxcel::models::nemotron_h: [NemotronH] Loading weights from safetensors... +results: 2026-10-13T15:18:35.203333Z INFO mlxcel::models::nemotron_h: [NemotronH] materialized weights for request memory admission resident_bytes=17777861888 +results: 2026-10-13T15:18:35.203801Z INFO mlxcel::models::nemotron_h: [NemotronH] Model loaded successfully +no-warmup: 2026-10-13T15:19:02.062624Z INFO mlxcel::models::nemotron_h: [NemotronH] Loading weights from safetensors... +no-warmup: 2026-10-13T15:19:04.881088Z INFO mlxcel::models::nemotron_h: [NemotronH] materialized weights for request memory admission resident_bytes=17777861888 +no-warmup: 2026-10-13T15:19:04.881548Z INFO mlxcel::models::nemotron_h: [NemotronH] Model loaded successfully +no-batch: 2026-10-13T15:19:28.795392Z INFO mlxcel::models::nemotron_h: [NemotronH] Loading weights from safetensors... +no-batch: 2026-10-13T15:19:30.724244Z INFO mlxcel::models::nemotron_h: [NemotronH] materialized weights for request memory admission resident_bytes=17777861888 +no-batch: 2026-10-13T15:19:30.724708Z INFO mlxcel::models::nemotron_h: [NemotronH] Model loaded successfully diff --git a/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/final.json b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/final.json new file mode 100644 index 000000000..05a15b897 --- /dev/null +++ b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/final.json @@ -0,0 +1,1124 @@ +[ + { + "rep": 1, + "binary_sha256": "1c4673addeb123b8d3ef2758f69dc0c54811d1d061911dbcb86aa486cc6065c5", + "command": [ + "target/release/mlxcel-server", + "--model", + "models/mlx/nvidia-nemotron-3-nano-30b-a3b-4bit", + "--ctx-size", + "8192", + "--jinja", + "--parallel", + "1", + "--host", + "127.0.0.1", + "--port", + "62899", + "--alias", + "repro", + "--offline" + ], + "budget": "24G", + "load_before": [ + 6.43603515625, + 9.11181640625, + 8.89892578125 + ], + "requests": [ + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.008508292026817799, + "case": "reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51430760, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-9791fc78-00a1-4c8f-aa9a-41f217d05c06", + "object": "text_completion", + "created": 1791904663, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "10000000", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 17, + "completion_tokens": 8, + "total_tokens": 25, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.4779440410202369, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51430760, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.00712929101428017, + "case": "reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68893226, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"cmpl-06e1e193-adaa-4bc8-a0d6-bf91c65fbc19\",\"object\":\"text_completion\",\"created\":1791904667,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"1\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-06e1e193-adaa-4bc8-a0d6-bf91c65fbc19\",\"object\":\"text_completion\",\"created\":1791904667,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-06e1e193-adaa-4bc8-a0d6-bf91c65fbc19\",\"object\":\"text_completion\",\"created\":1791904667,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-06e1e193-adaa-4bc8-a0d6-bf91c65fbc19\",\"object\":\"text_completion\",\"created\":1791904667,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-06e1e193-adaa-4bc8-a0d6-bf91c65fbc19\",\"object\":\"text_completion\",\"created\":1791904667,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-06e1e193-adaa-4bc8-a0d6-bf91c65fbc19\",\"object\":\"text_completion\",\"created\":1791904667,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-06e1e193-adaa-4bc8-a0d6-bf91c65fbc19\",\"object\":\"text_completion\",\"created\":1791904667,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-06e1e193-adaa-4bc8-a0d6-bf91c65fbc19\",\"object\":\"text_completion\",\"created\":1791904667,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-06e1e193-adaa-4bc8-a0d6-bf91c65fbc19\",\"object\":\"text_completion\",\"created\":1791904667,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"\",\"finish_reason\":\"length\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.4744004160165787, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68893226, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29367936268, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.01331487501738593, + "case": "reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68725810, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "chatcmpl-25ffdbf9-4937-4a3a-a7bb-29e0146b7f58", + "object": "chat.completion", + "created": 1791904671, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "OK" + }, + "finish_reason": "stop", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 36, + "completion_tokens": 1, + "total_tokens": 37, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.431412291014567, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68725810, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29367936268, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.01298212498659268, + "case": "reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 73813310, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"chatcmpl-108fdaef-a731-483d-a9aa-492d6b3e1135\",\"object\":\"chat.completion.chunk\",\"created\":1791904674,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-108fdaef-a731-483d-a9aa-492d6b3e1135\",\"object\":\"chat.completion.chunk\",\"created\":1791904675,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"content\":\"OK\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-108fdaef-a731-483d-a9aa-492d6b3e1135\",\"object\":\"chat.completion.chunk\",\"created\":1791904675,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.44845387496752664, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 73813310, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-5adc39d7-58a6-4819-9136-fae22813cfea", + "object": "text_completion", + "created": 1791904681, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "\n\n\n**Step\u2011by\u2011step", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 1501, + "completion_tokens": 8, + "total_tokens": 1509, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 4.434573624981567, + "case": "accepted_1500", + "admission": { + "prompt_tokens": 1501, + "output_tokens": 8, + "largest_forward_tokens": 1501, + "estimated_bytes": 23974864524, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 73797698, + "admitted": true + }, + "footprint": { + "peak_bytes": 18897856102, + "current_bytes": 18468359372 + } + } + ], + "log": "/tmp/mlxcel-2291-reviewed-results.json.rep1.log", + "ready_s": 7.7716115830116905, + "startup_footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "context_rejection": { + "status": 400, + "content_type": "application/json", + "body": { + "error": { + "message": "request (8401 tokens) exceeds the available context size (8192 tokens), try increasing it", + "type": "exceed_context_size_error", + "code": null + } + }, + "elapsed_s": 0.009412874991539866 + }, + "final_health": { + "status": 200, + "content_type": "application/json", + "body": { + "status": "ok" + }, + "elapsed_s": 0.0005202500033192337 + }, + "load_after": [ + 6.90283203125, + 8.98388671875, + 8.85986328125 + ] + }, + { + "rep": 2, + "binary_sha256": "1c4673addeb123b8d3ef2758f69dc0c54811d1d061911dbcb86aa486cc6065c5", + "command": [ + "target/release/mlxcel-server", + "--model", + "models/mlx/nvidia-nemotron-3-nano-30b-a3b-4bit", + "--ctx-size", + "8192", + "--jinja", + "--parallel", + "1", + "--host", + "127.0.0.1", + "--port", + "63042", + "--alias", + "repro", + "--offline" + ], + "budget": "24G", + "load_before": [ + 6.90283203125, + 8.98388671875, + 8.85986328125 + ], + "requests": [ + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.00810525001725182, + "case": "reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51426664, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-7f8c26d3-404f-46d1-adec-87178f9d137b", + "object": "text_completion", + "created": 1791904690, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "10000000", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 17, + "completion_tokens": 8, + "total_tokens": 25, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.37033012497704476, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51426664, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.006744207988958806, + "case": "reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68869682, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"cmpl-a3b2f41c-ce86-4a24-a0ce-8fe8fb33468e\",\"object\":\"text_completion\",\"created\":1791904694,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"1\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-a3b2f41c-ce86-4a24-a0ce-8fe8fb33468e\",\"object\":\"text_completion\",\"created\":1791904694,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-a3b2f41c-ce86-4a24-a0ce-8fe8fb33468e\",\"object\":\"text_completion\",\"created\":1791904694,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-a3b2f41c-ce86-4a24-a0ce-8fe8fb33468e\",\"object\":\"text_completion\",\"created\":1791904694,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-a3b2f41c-ce86-4a24-a0ce-8fe8fb33468e\",\"object\":\"text_completion\",\"created\":1791904694,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-a3b2f41c-ce86-4a24-a0ce-8fe8fb33468e\",\"object\":\"text_completion\",\"created\":1791904694,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-a3b2f41c-ce86-4a24-a0ce-8fe8fb33468e\",\"object\":\"text_completion\",\"created\":1791904694,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-a3b2f41c-ce86-4a24-a0ce-8fe8fb33468e\",\"object\":\"text_completion\",\"created\":1791904694,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-a3b2f41c-ce86-4a24-a0ce-8fe8fb33468e\",\"object\":\"text_completion\",\"created\":1791904694,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"\",\"finish_reason\":\"length\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.4595187090453692, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68869682, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29367936268, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.020917334011755884, + "case": "reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68705834, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "chatcmpl-ff8d78b3-b4c1-4034-b8af-19fec46ea3e8", + "object": "chat.completion", + "created": 1791904698, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "OK" + }, + "finish_reason": "stop", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 36, + "completion_tokens": 1, + "total_tokens": 37, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.4696371250320226, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68705834, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29367936268, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.012423541978932917, + "case": "reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 75485246, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"chatcmpl-1dab4b6e-8bb9-438a-8689-2f52df51126b\",\"object\":\"chat.completion.chunk\",\"created\":1791904702,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-1dab4b6e-8bb9-438a-8689-2f52df51126b\",\"object\":\"chat.completion.chunk\",\"created\":1791904703,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"content\":\"OK\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-1dab4b6e-8bb9-438a-8689-2f52df51126b\",\"object\":\"chat.completion.chunk\",\"created\":1791904703,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.5936561250127852, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 75485246, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-db5d8d91-1f85-49f4-a08b-37cc7a7248b8", + "object": "text_completion", + "created": 1791904710, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "\n\n\n**Step\u2011by\u2011step", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 1501, + "completion_tokens": 8, + "total_tokens": 1509, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 4.532743457995821, + "case": "accepted_1500", + "admission": { + "prompt_tokens": 1501, + "output_tokens": 8, + "largest_forward_tokens": 1501, + "estimated_bytes": 23974864524, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 73810502, + "admitted": true + }, + "footprint": { + "peak_bytes": 18897856102, + "current_bytes": 18468359372 + } + } + ], + "log": "/tmp/mlxcel-2291-reviewed-results.json.rep2.log", + "ready_s": 2.8344444160466082, + "startup_footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "context_rejection": { + "status": 400, + "content_type": "application/json", + "body": { + "error": { + "message": "request (8401 tokens) exceeds the available context size (8192 tokens), try increasing it", + "type": "exceed_context_size_error", + "code": null + } + }, + "elapsed_s": 0.01113312499364838 + }, + "final_health": { + "status": 200, + "content_type": "application/json", + "body": { + "status": "ok" + }, + "elapsed_s": 0.0005319580086506903 + }, + "load_after": [ + 11.00830078125, + 9.67822265625, + 9.10888671875 + ] + }, + { + "rep": 3, + "binary_sha256": "1c4673addeb123b8d3ef2758f69dc0c54811d1d061911dbcb86aa486cc6065c5", + "command": [ + "target/release/mlxcel-server", + "--model", + "models/mlx/nvidia-nemotron-3-nano-30b-a3b-4bit", + "--ctx-size", + "8192", + "--jinja", + "--parallel", + "1", + "--host", + "127.0.0.1", + "--port", + "63234", + "--alias", + "repro", + "--offline" + ], + "budget": "24G", + "load_before": [ + 11.00830078125, + 9.67822265625, + 9.10888671875 + ], + "requests": [ + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.007921124983113259, + "case": "reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51426664, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-3201d890-7729-48aa-90f3-ac00527f8279", + "object": "text_completion", + "created": 1791904720, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "10000000", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 17, + "completion_tokens": 8, + "total_tokens": 25, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.45594629197148606, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51426664, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.007484582951292396, + "case": "reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68713002, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"cmpl-e71a47f3-1850-47ae-89b7-16fb4bc23d40\",\"object\":\"text_completion\",\"created\":1791904724,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"1\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-e71a47f3-1850-47ae-89b7-16fb4bc23d40\",\"object\":\"text_completion\",\"created\":1791904724,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-e71a47f3-1850-47ae-89b7-16fb4bc23d40\",\"object\":\"text_completion\",\"created\":1791904724,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-e71a47f3-1850-47ae-89b7-16fb4bc23d40\",\"object\":\"text_completion\",\"created\":1791904724,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-e71a47f3-1850-47ae-89b7-16fb4bc23d40\",\"object\":\"text_completion\",\"created\":1791904724,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-e71a47f3-1850-47ae-89b7-16fb4bc23d40\",\"object\":\"text_completion\",\"created\":1791904724,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-e71a47f3-1850-47ae-89b7-16fb4bc23d40\",\"object\":\"text_completion\",\"created\":1791904724,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-e71a47f3-1850-47ae-89b7-16fb4bc23d40\",\"object\":\"text_completion\",\"created\":1791904724,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-e71a47f3-1850-47ae-89b7-16fb4bc23d40\",\"object\":\"text_completion\",\"created\":1791904724,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"\",\"finish_reason\":\"length\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.5096182079869322, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68713002, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29367936268, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.012520707969088107, + "case": "reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68705842, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "chatcmpl-5b7c236b-864a-4dc3-b154-d2a07ccf306e", + "object": "chat.completion", + "created": 1791904728, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "OK" + }, + "finish_reason": "stop", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 36, + "completion_tokens": 1, + "total_tokens": 37, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.46989795804256573, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68705842, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29367936268, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.01306941598886624, + "case": "reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 73800510, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"chatcmpl-430c8733-0641-4cd8-9b4b-4c0def3c628f\",\"object\":\"chat.completion.chunk\",\"created\":1791904732,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-430c8733-0641-4cd8-9b4b-4c0def3c628f\",\"object\":\"chat.completion.chunk\",\"created\":1791904732,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"content\":\"OK\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-430c8733-0641-4cd8-9b4b-4c0def3c628f\",\"object\":\"chat.completion.chunk\",\"created\":1791904732,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.44576600001892075, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 73800510, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-0b7fc2b6-3c8e-459c-bf91-2cc48656af64", + "object": "text_completion", + "created": 1791904739, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "\n\n\n**Step\u2011by\u2011step", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 1501, + "completion_tokens": 8, + "total_tokens": 1509, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 4.411629874957725, + "case": "accepted_1500", + "admission": { + "prompt_tokens": 1501, + "output_tokens": 8, + "largest_forward_tokens": 1501, + "estimated_bytes": 23974864524, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 73794110, + "admitted": true + }, + "footprint": { + "peak_bytes": 18897856102, + "current_bytes": 18468359372 + } + } + ], + "log": "/tmp/mlxcel-2291-reviewed-results.json.rep3.log", + "ready_s": 3.3725628330139443, + "startup_footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "context_rejection": { + "status": 400, + "content_type": "application/json", + "body": { + "error": { + "message": "request (8401 tokens) exceeds the available context size (8192 tokens), try increasing it", + "type": "exceed_context_size_error", + "code": null + } + }, + "elapsed_s": 0.009561833983752877 + }, + "final_health": { + "status": 200, + "content_type": "application/json", + "body": { + "status": "ok" + }, + "elapsed_s": 0.0005064590368419886 + }, + "load_after": [ + 16.88916015625, + 11.56298828125, + 9.83203125 + ] + } +] \ No newline at end of file diff --git a/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/harness.py b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/harness.py new file mode 100644 index 000000000..05beb4392 --- /dev/null +++ b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/harness.py @@ -0,0 +1,160 @@ +#!/usr/bin/env python3 +"""Real checkpoint admission lifecycle, not a speed benchmark.""" +import argparse +import hashlib +import json +import os +from pathlib import Path +import re +import signal +import socket +import subprocess +import time +import urllib.error +import urllib.request + +ANSI = re.compile(r"\x1b\[[0-9;]*m") + + +def require(condition: bool, message: str) -> None: + if not condition: + raise RuntimeError(message) + + +def request(base: str, route: str, payload: dict | None = None) -> dict: + req = urllib.request.Request(base + route, data=None if payload is None else json.dumps(payload).encode(), headers={"Content-Type": "application/json"}) + started = time.monotonic() + try: + response = urllib.request.urlopen(req, timeout=900) + except urllib.error.HTTPError as exc: + response = exc + with response: + raw = response.read().decode() + try: + body = json.loads(raw) + except json.JSONDecodeError: + body = raw + return {"status": response.status, "content_type": response.headers.get("Content-Type"), "body": body, "elapsed_s": time.monotonic() - started} + + +def footprint(pid: int) -> dict: + text = subprocess.run(["vmmap", "-summary", str(pid)], capture_output=True, text=True, check=True, timeout=60).stdout + result = {} + for field, pattern in [("peak", r"Physical footprint \(peak\):\s+([0-9.]+)([KMG])"), ("current", r"Physical footprint:\s+([0-9.]+)([KMG])")]: + match = re.search(pattern, text) + require(match is not None, f"missing vmmap {field}") + result[field + "_bytes"] = int(float(match.group(1)) * {"K": 1024, "M": 1024**2, "G": 1024**3}[match.group(2)]) + return result + + +def admissions(path: Path) -> list[dict]: + result = [] + for line in ANSI.sub("", path.read_text(errors="replace")).splitlines(): + if "request memory admission" in line: + values = {k: int(v) for k, v in re.findall(r"\b([a-z_]+)=(\d+)", line)} + values["admitted"] = "admitted=true" in line + result.append(values) + return result + + +def run(args: argparse.Namespace, repetition: int) -> dict: + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)) + port = sock.getsockname()[1] + base = f"http://127.0.0.1:{port}" + log_path = Path(f"{args.output}.rep{repetition}.log") + env = {**os.environ, "RUST_LOG": "info", "NO_COLOR": "1", "MLXCEL_REQUEST_MEMORY_BUDGET": args.budget} + cmd = [args.binary, "--model", args.model, "--ctx-size", "8192", "--jinja", "--parallel", "1", "--host", "127.0.0.1", "--port", str(port), "--alias", "repro", "--offline"] + if args.no_warmup: + cmd.append("--no-warmup") + if args.no_batch: + cmd.append("--no-batch") + result = {"rep": repetition, "binary_sha256": hashlib.sha256(Path(args.binary).read_bytes()).hexdigest(), "command": cmd, "budget": args.budget, "load_before": os.getloadavg(), "requests": [], "log": str(log_path)} + started = time.monotonic() + with log_path.open("w") as log: + proc = subprocess.Popen(cmd, stdout=log, stderr=subprocess.STDOUT, env=env) + try: + deadline = time.monotonic() + 180 + while True: + require(proc.poll() is None, f"server exited: {log_path.read_text()[-3000:]}") + try: + ready = request(base, "/v1/models") + if ready["status"] == 200: + break + except (urllib.error.URLError, TimeoutError): + pass + require(time.monotonic() < deadline, "server readiness deadline") + time.sleep(0.25) + result["ready_s"] = time.monotonic() - started + result["startup_footprint"] = footprint(proc.pid) + for route in ["/v1/completions", "/v1/chat/completions"]: + for streaming in [False, True]: + for target, count in [("reject", 6000), ("accept_after_reject", 16)]: + filler = "benchmark " * count + body = {"model": "repro", "max_tokens": 8, "temperature": 0.0, "stream": streaming} + if route.endswith("/chat/completions"): + body["messages"] = [{"role": "user", "content": filler + " Reply OK."}] + body["chat_template_kwargs"] = {"enable_thinking": False} + else: + body["prompt"] = filler + before = len(admissions(log_path)) + response = request(base, route, body) + response.update({"case": target, "route": route, "stream": streaming, "footprint": footprint(proc.pid)}) + entries = admissions(log_path) + require(len(entries) > before, "request never reached real memory guard") + response["admission"] = entries[-1] + require(response["admission"]["prompt_tokens"] < 8192, "must reject within context") + if target == "reject": + require(response["status"] == 503, f"expected memory503: {response}") + require("text/event-stream" not in response["content_type"], "SSE headers committed on rejection") + error = response["body"]["error"] + require(error["type"] == error["code"] == "request_memory_budget_exceeded", "wrong typed error") + require(not response["admission"]["admitted"], "not a guard rejection") + else: + require(response["status"] == 200, f"following short failed: {response}") + require(response["admission"]["admitted"], "short not admitted") + if streaming: + require("data:" in response["body"] and "[DONE]" in response["body"], "stream incomplete") + health = request(base, "/health") + require(health["status"] == 200, "health failed after request") + response["health_status"] = health["status"] + result["requests"].append(response) + print(json.dumps({k: response[k] for k in ["case", "route", "stream", "status", "admission", "footprint"]}), flush=True) + medium = request(base, "/v1/completions", {"model": "repro", "prompt": "benchmark " * 1500, "max_tokens": 8, "temperature": 0.0}) + require(medium["status"] == 200, f"medium request failed: {medium}") + medium.update({"case": "accepted_1500", "admission": admissions(log_path)[-1], "footprint": footprint(proc.pid)}) + result["requests"].append(medium) + if not args.no_batch: + context = request(base, "/v1/completions", {"model": "repro", "prompt": "benchmark " * 8400, "max_tokens": 8, "stream": True}) + require(context["status"] == 400, f"context precedence: {context}") + result["context_rejection"] = context + result["final_health"] = request(base, "/health") + require(result["final_health"]["status"] == 200, "final health not200") + result["load_after"] = os.getloadavg() + finally: + proc.terminate() + try: + proc.wait(timeout=15) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait(timeout=5) + return result + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--binary", required=True) + parser.add_argument("--model", required=True) + parser.add_argument("--output", required=True) + parser.add_argument("--budget", default="24G") + parser.add_argument("--reps", type=int, default=3) + parser.add_argument("--no-warmup", action="store_true") + parser.add_argument("--no-batch", action="store_true") + args = parser.parse_args() + results = [] + for repetition in range(1, args.reps + 1): + results.append(run(args, repetition)) + Path(args.output).write_text(json.dumps(results, indent=2)) + +if __name__ == "__main__": + main() diff --git a/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/invalid-env.json b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/invalid-env.json new file mode 100644 index 000000000..0187042db --- /dev/null +++ b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/invalid-env.json @@ -0,0 +1,22 @@ +[ + { + "value": "0", + "exit_code": 1, + "stderr": "Error: MLXCEL_REQUEST_MEMORY_BUDGET must be a positive finite byte size, for example 12G\n" + }, + { + "value": "-1G", + "exit_code": 1, + "stderr": "Error: MLXCEL_REQUEST_MEMORY_BUDGET must be a positive finite byte size, for example 12G\n" + }, + { + "value": "NaN", + "exit_code": 1, + "stderr": "Error: MLXCEL_REQUEST_MEMORY_BUDGET must be a positive finite byte size, for example 12G\n" + }, + { + "value": "18446744073709551616", + "exit_code": 1, + "stderr": "Error: MLXCEL_REQUEST_MEMORY_BUDGET must be a positive finite byte size, for example 12G\n" + } +] \ No newline at end of file diff --git a/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/other_family.py b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/other_family.py new file mode 100644 index 000000000..ea215f6bd --- /dev/null +++ b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/other_family.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 +"""Unprotected Qwen family smoke under an intentionally tiny target-only budget.""" +import argparse +import hashlib +import json +import os +from pathlib import Path +import socket +import subprocess +import time +import urllib.error +from harness import request, require + +p = argparse.ArgumentParser() +p.add_argument('--binary', required=True) +p.add_argument('--output', required=True) +a = p.parse_args() +with socket.socket() as sock: + sock.bind(('127.0.0.1', 0)) + port = sock.getsockname()[1] +base = f'http://127.0.0.1:{port}' +cmd = [a.binary, '--model', 'models/mlx/qwen2.5-7b-instruct-4bit', '--offline', '--host', '127.0.0.1', '--port', str(port), '--parallel', '1', '--ctx-size', '4096', '--alias', 'repro'] +logpath = Path(a.output + '.log') +result = {'command': cmd, 'budget': '1', 'binary_sha256': hashlib.sha256(Path(a.binary).read_bytes()).hexdigest(), 'load_before': os.getloadavg(), 'requests': []} +with logpath.open('w') as log: + proc = subprocess.Popen(cmd, stdout=log, stderr=subprocess.STDOUT, env={**os.environ, 'MLXCEL_REQUEST_MEMORY_BUDGET': '1', 'RUST_LOG': 'info'}) + try: + deadline = time.monotonic() + 180 + while True: + require(proc.poll() is None, 'server exited') + try: + if request(base, '/v1/models')['status'] == 200: + break + except (urllib.error.URLError, TimeoutError): + pass + require(time.monotonic() < deadline, 'readiness timed out') + time.sleep(0.25) + for stream in [False, True]: + r = request(base, '/v1/completions', {'model': 'repro', 'prompt': 'The capital of France is', 'max_tokens': 8, 'temperature': 0.0, 'stream': stream}) + require(r['status'] == 200, f'Qwen incorrectly guarded: {r}') + if stream: + require('[DONE]' in r['body'], 'incomplete stream') + result['requests'].append(r) + result['health'] = request(base, '/health') + require(result['health']['status'] == 200, 'health failed') + require('request_memory_admission' not in logpath.read_text(), 'unexpected target guard') + finally: + proc.terminate() + try: + proc.wait(timeout=15) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait(timeout=5) +Path(a.output).write_text(json.dumps(result, indent=2)) +print(json.dumps({'requests': len(result['requests']), 'health': result['health']['status']})) diff --git a/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/post-setter-legacy.json b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/post-setter-legacy.json new file mode 100644 index 000000000..259bb38d2 --- /dev/null +++ b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/post-setter-legacy.json @@ -0,0 +1,365 @@ +[ + { + "rep": 1, + "binary_sha256": "e7a5cc9997a8c0eeceba6a69769353eed35395e906b270d2c62a5f437e2a7062", + "command": [ + "target/release/mlxcel-server", + "--model", + "models/mlx/nvidia-nemotron-3-nano-30b-a3b-4bit", + "--ctx-size", + "8192", + "--jinja", + "--parallel", + "1", + "--host", + "127.0.0.1", + "--port", + "50183", + "--alias", + "repro", + "--offline", + "--no-batch" + ], + "budget": "24G", + "load_before": [ + 5.51171875, + 6.05810546875, + 7.369140625 + ], + "requests": [ + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=48861207180, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 48861207180, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.007709249970503151, + "case": "reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 6001, + "estimated_bytes": 48861207180, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51430760, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-c346c620-4a07-4b0a-a964-33bb2960a6b0", + "object": "text_completion", + "created": 1791905812, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "10000000", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 17, + "completion_tokens": 8, + "total_tokens": 25, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.47218737500952557, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51430760, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=48861207180, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 48861207180, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.007228916976600885, + "case": "reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 6001, + "estimated_bytes": 48861207180, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68889642, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"cmpl-27a3565a-e41a-4bed-bd52-2a0cb0ae8295\",\"object\":\"text_completion\",\"created\":1791905816,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"1\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-27a3565a-e41a-4bed-bd52-2a0cb0ae8295\",\"object\":\"text_completion\",\"created\":1791905817,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-27a3565a-e41a-4bed-bd52-2a0cb0ae8295\",\"object\":\"text_completion\",\"created\":1791905817,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-27a3565a-e41a-4bed-bd52-2a0cb0ae8295\",\"object\":\"text_completion\",\"created\":1791905817,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-27a3565a-e41a-4bed-bd52-2a0cb0ae8295\",\"object\":\"text_completion\",\"created\":1791905817,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-27a3565a-e41a-4bed-bd52-2a0cb0ae8295\",\"object\":\"text_completion\",\"created\":1791905817,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-27a3565a-e41a-4bed-bd52-2a0cb0ae8295\",\"object\":\"text_completion\",\"created\":1791905817,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-27a3565a-e41a-4bed-bd52-2a0cb0ae8295\",\"object\":\"text_completion\",\"created\":1791905817,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-27a3565a-e41a-4bed-bd52-2a0cb0ae8295\",\"object\":\"text_completion\",\"created\":1791905817,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"\",\"finish_reason\":\"length\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.562673292006366, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68889642, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=48998894860, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 48998894860, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.012691792042460293, + "case": "reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 6020, + "estimated_bytes": 48998894860, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68709426, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "chatcmpl-4ca15015-b7dd-413d-aa0a-8f88162bd9ca", + "object": "chat.completion", + "created": 1791905821, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "OK" + }, + "finish_reason": "stop", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 36, + "completion_tokens": 1, + "total_tokens": 37, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.5394864589907229, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68709426, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=48998894860, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 48998894860, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.012445708038285375, + "case": "reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 6020, + "estimated_bytes": 48998894860, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 75468862, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"chatcmpl-5644bc3b-2c47-4605-adf8-bd657270be22\",\"object\":\"chat.completion.chunk\",\"created\":1791905824,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-5644bc3b-2c47-4605-adf8-bd657270be22\",\"object\":\"chat.completion.chunk\",\"created\":1791905825,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"content\":\"OK\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-5644bc3b-2c47-4605-adf8-bd657270be22\",\"object\":\"chat.completion.chunk\",\"created\":1791905825,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.5341448340332136, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 75468862, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-4bc80697-2aed-4125-9aaf-bbc1bcad1e5a", + "object": "text_completion", + "created": 1791905831, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "\n\n\n**Step\u2011by\u2011step", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 1501, + "completion_tokens": 8, + "total_tokens": 1509, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 4.347933334007394, + "case": "accepted_1500", + "admission": { + "prompt_tokens": 1501, + "output_tokens": 8, + "largest_forward_tokens": 1501, + "estimated_bytes": 23974864524, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 75468862, + "admitted": true + }, + "footprint": { + "peak_bytes": 18897856102, + "current_bytes": 18468359372 + } + } + ], + "log": "/tmp/mlxcel-2291-post-setter-legacy.json.rep1.log", + "ready_s": 2.827465125010349, + "startup_footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "final_health": { + "status": 200, + "content_type": "application/json", + "body": { + "status": "ok" + }, + "elapsed_s": 0.0008466249564662576 + }, + "load_after": [ + 5.1533203125, + 5.9248046875, + 7.2822265625 + ] + } +] \ No newline at end of file diff --git a/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/post-setter-normal.json b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/post-setter-normal.json new file mode 100644 index 000000000..c15199754 --- /dev/null +++ b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/post-setter-normal.json @@ -0,0 +1,376 @@ +[ + { + "rep": 1, + "binary_sha256": "e7a5cc9997a8c0eeceba6a69769353eed35395e906b270d2c62a5f437e2a7062", + "command": [ + "target/release/mlxcel-server", + "--model", + "models/mlx/nvidia-nemotron-3-nano-30b-a3b-4bit", + "--ctx-size", + "8192", + "--jinja", + "--parallel", + "1", + "--host", + "127.0.0.1", + "--port", + "50100", + "--alias", + "repro", + "--offline" + ], + "budget": "24G", + "load_before": [ + 5.89990234375, + 6.17626953125, + 7.46484375 + ], + "requests": [ + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.00812412501545623, + "case": "reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51426664, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-35dc2747-a488-4c94-9f5b-d03c71db0007", + "object": "text_completion", + "created": 1791905784, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "10000000", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 17, + "completion_tokens": 8, + "total_tokens": 25, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.5463505419902503, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 51426664, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29352994060, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.007055417052470148, + "case": "reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6001, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29352994060, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68705834, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"cmpl-fd4f20e3-65ed-42d7-964d-1201931c79ae\",\"object\":\"text_completion\",\"created\":1791905788,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"1\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-fd4f20e3-65ed-42d7-964d-1201931c79ae\",\"object\":\"text_completion\",\"created\":1791905788,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-fd4f20e3-65ed-42d7-964d-1201931c79ae\",\"object\":\"text_completion\",\"created\":1791905788,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-fd4f20e3-65ed-42d7-964d-1201931c79ae\",\"object\":\"text_completion\",\"created\":1791905788,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-fd4f20e3-65ed-42d7-964d-1201931c79ae\",\"object\":\"text_completion\",\"created\":1791905788,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-fd4f20e3-65ed-42d7-964d-1201931c79ae\",\"object\":\"text_completion\",\"created\":1791905788,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-fd4f20e3-65ed-42d7-964d-1201931c79ae\",\"object\":\"text_completion\",\"created\":1791905788,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-fd4f20e3-65ed-42d7-964d-1201931c79ae\",\"object\":\"text_completion\",\"created\":1791905788,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"0\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-fd4f20e3-65ed-42d7-964d-1201931c79ae\",\"object\":\"text_completion\",\"created\":1791905788,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"\",\"finish_reason\":\"length\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.627219290996436, + "case": "accept_after_reject", + "route": "/v1/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 17, + "output_tokens": 8, + "largest_forward_tokens": 17, + "estimated_bytes": 19126584460, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68705834, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29367936268, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.013188332959543914, + "case": "reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68869674, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "chatcmpl-6050d0c2-41bd-41f9-a997-4e38d2b551e7", + "object": "chat.completion", + "created": 1791905793, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "OK" + }, + "finish_reason": "stop", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 36, + "completion_tokens": 1, + "total_tokens": 37, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.5821980829932727, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": false, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 68869674, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 503, + "content_type": "application/json", + "body": { + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=29367936268, budget_bytes=25769803776; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded", + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776 + } + }, + "elapsed_s": 0.012780916993506253, + "case": "reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 6020, + "output_tokens": 8, + "largest_forward_tokens": 2048, + "estimated_bytes": 29367936268, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 73796926, + "admitted": false + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"chatcmpl-72d8caad-7788-450c-aa08-17ecec0a0630\",\"object\":\"chat.completion.chunk\",\"created\":1791905796,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-72d8caad-7788-450c-aa08-17ecec0a0630\",\"object\":\"chat.completion.chunk\",\"created\":1791905797,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"content\":\"OK\"},\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"chatcmpl-72d8caad-7788-450c-aa08-17ecec0a0630\",\"object\":\"chat.completion.chunk\",\"created\":1791905797,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.5606945420149714, + "case": "accept_after_reject", + "route": "/v1/chat/completions", + "stream": true, + "footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "admission": { + "prompt_tokens": 36, + "output_tokens": 8, + "largest_forward_tokens": 36, + "estimated_bytes": 19182021900, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 73796926, + "admitted": true + }, + "health_status": 200 + }, + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-557ec735-ee01-4223-8a38-1b1d8e4d4924", + "object": "text_completion", + "created": 1791905803, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": "\n\n\n**Step\u2011by\u2011step", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 1501, + "completion_tokens": 8, + "total_tokens": 1509, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 4.345065540983342, + "case": "accepted_1500", + "admission": { + "prompt_tokens": 1501, + "output_tokens": 8, + "largest_forward_tokens": 1501, + "estimated_bytes": 23974864524, + "budget_bytes": 25769803776, + "resident_bytes": 17777861900, + "allocator_cache_bytes": 73797698, + "admitted": true + }, + "footprint": { + "peak_bytes": 18897856102, + "current_bytes": 18468359372 + } + } + ], + "log": "/tmp/mlxcel-2291-post-setter-normal.json.rep1.log", + "ready_s": 5.623054875002708, + "startup_footprint": { + "peak_bytes": 18146236825, + "current_bytes": 18146236825 + }, + "context_rejection": { + "status": 400, + "content_type": "application/json", + "body": { + "error": { + "message": "request (8401 tokens) exceeds the available context size (8192 tokens), try increasing it", + "type": "exceed_context_size_error", + "code": null + } + }, + "elapsed_s": 0.00963070901343599 + }, + "final_health": { + "status": 200, + "content_type": "application/json", + "body": { + "status": "ok" + }, + "elapsed_s": 0.0004997919895686209 + }, + "load_after": [ + 5.51171875, + 6.05810546875, + 7.369140625 + ] + } +] \ No newline at end of file diff --git a/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/post-setter-qwen.json b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/post-setter-qwen.json new file mode 100644 index 000000000..4260f89fb --- /dev/null +++ b/docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13/post-setter-qwen.json @@ -0,0 +1,69 @@ +{ + "command": [ + "target/release/mlxcel-server", + "--model", + "models/mlx/qwen2.5-7b-instruct-4bit", + "--offline", + "--host", + "127.0.0.1", + "--port", + "50235", + "--parallel", + "1", + "--ctx-size", + "4096", + "--alias", + "repro" + ], + "budget": "1", + "binary_sha256": "e7a5cc9997a8c0eeceba6a69769353eed35395e906b270d2c62a5f437e2a7062", + "load_before": [ + 5.1533203125, + 5.9248046875, + 7.2822265625 + ], + "requests": [ + { + "status": 200, + "content_type": "application/json", + "body": { + "id": "cmpl-64415a6c-ff62-43ef-b84e-298fb2e3c14f", + "object": "text_completion", + "created": 1791905835, + "model": "repro", + "system_fingerprint": null, + "choices": [ + { + "index": 0, + "text": " Paris. It is located in the northern", + "finish_reason": "length", + "logprobs": null + } + ], + "usage": { + "prompt_tokens": 5, + "completion_tokens": 8, + "total_tokens": 13, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + "elapsed_s": 0.11453845800133422 + }, + { + "status": 200, + "content_type": "text/event-stream", + "body": "data: {\"id\":\"cmpl-962e0204-7d42-4cd6-93bf-76da8166a6dd\",\"object\":\"text_completion\",\"created\":1791905836,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\" Paris\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-962e0204-7d42-4cd6-93bf-76da8166a6dd\",\"object\":\"text_completion\",\"created\":1791905836,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\".\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-962e0204-7d42-4cd6-93bf-76da8166a6dd\",\"object\":\"text_completion\",\"created\":1791905836,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\" It\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-962e0204-7d42-4cd6-93bf-76da8166a6dd\",\"object\":\"text_completion\",\"created\":1791905836,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\" is\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-962e0204-7d42-4cd6-93bf-76da8166a6dd\",\"object\":\"text_completion\",\"created\":1791905836,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\" located\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-962e0204-7d42-4cd6-93bf-76da8166a6dd\",\"object\":\"text_completion\",\"created\":1791905836,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\" in\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-962e0204-7d42-4cd6-93bf-76da8166a6dd\",\"object\":\"text_completion\",\"created\":1791905836,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\" the\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-962e0204-7d42-4cd6-93bf-76da8166a6dd\",\"object\":\"text_completion\",\"created\":1791905836,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\" northern\",\"finish_reason\":null,\"logprobs\":null}]}\n\ndata: {\"id\":\"cmpl-962e0204-7d42-4cd6-93bf-76da8166a6dd\",\"object\":\"text_completion\",\"created\":1791905836,\"model\":\"repro\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"text\":\"\",\"finish_reason\":\"length\",\"logprobs\":null}]}\n\ndata: [DONE]\n\n", + "elapsed_s": 0.10268333298154175 + } + ], + "health": { + "status": 200, + "content_type": "application/json", + "body": { + "status": "ok" + }, + "elapsed_s": 0.0006745830178260803 + } +} \ No newline at end of file diff --git a/docs/benchmark_results/nemotron-h-prefill-memory-m1ultra-2026-10-13.md b/docs/benchmark_results/nemotron-h-prefill-memory-m1ultra-2026-10-13.md index cbad5164c..20e4ec978 100644 --- a/docs/benchmark_results/nemotron-h-prefill-memory-m1ultra-2026-10-13.md +++ b/docs/benchmark_results/nemotron-h-prefill-memory-m1ultra-2026-10-13.md @@ -6,6 +6,8 @@ Ship the model-internal SSD chunking for Nemotron-H. The production server path This record resolves the quadratic-prefill-memory part of issue #2281. It does **not** establish request-level rejection from a projected transient-memory budget. The current server has context-size and KV-budget admission plus an optional startup `--estimate-memory` check, but no per-request activation/transient-memory admission; a separate host/model combination can therefore still reach a fatal Metal driver OOM. The existing 8,319-token context rejection reported below is evidence only for context admission and is not treated as memory admission. +Follow-up: request-level admission is implemented separately in #2291; see [the admission policy](../request-memory-admission.md). The measurements and limitations below describe the original SSD-only change, not the subsequent guard. + ## Environment and provenance - Host: Mac Studio `Mac13,2`, Apple M1 Ultra, 128 GB unified memory, 20 logical CPUs. diff --git a/docs/benchmark_results/nemotron-h-request-memory-m1ultra-2026-10-13.md b/docs/benchmark_results/nemotron-h-request-memory-m1ultra-2026-10-13.md new file mode 100644 index 000000000..3b8dc3e29 --- /dev/null +++ b/docs/benchmark_results/nemotron-h-request-memory-m1ultra-2026-10-13.md @@ -0,0 +1,75 @@ +# Nemotron-H request memory admission — M1 Ultra, 2026-10-13/14 + +Issue: #2291. Policy: [request memory admission](../request-memory-admission.md). This is a lifecycle/safety calibration, not a throughput or latency improvement claim. + +## Setup and attribution + +- Apple M1 Ultra, 128 GiB unified memory; local `nvidia-nemotron-3-nano-30b-a3b-4bit` checkpoint, Metal + Accelerate release server. +- Context 8,192, effective parallelism one, greedy eight-token output, offline loading, 24 GiB lower-only request budget. The server-default runtime ceiling still applies. +- The [harness](data/nemotron-h-request-memory-m1ultra-2026-10-13/harness.py) requires a production admission log for every tested generation, verifies actual prompt length is within context, and refuses a rejection claim unless HTTP 503 and the typed memory error agree with that log. Streaming rejection must remain JSON, not an already-open SSE response. +- Physical footprint uses `vmmap -summary`: its peak is the **process-lifetime physical footprint**, rounded to 0.1 GiB, not a per-request MLX allocator peak. The analytical estimate is a projected conservative MLX resident/workspace requirement, so these measurements have different scopes. + +## Initial calibration and corrected loading + +An initial 32 GiB run admitted the 6,001-token completion with a 27.34 GiB estimate and about 17.8 GiB physical peak. The harness correctly stopped instead of labeling it a rejection. [Raw calibration](data/nemotron-h-request-memory-m1ultra-2026-10-13/calibration.json). This explicitly shows that a later 24 GiB policy rejection is conservative **over-rejection**, not proof the request physically cannot fit. + +Loading was hardened to materialize lazy weight graphs before admission, including `--no-warmup`. Candidate 2 (`ea183b308b19e972ae6a7e24779bc3b471aacec0e2dd218c26aaf76e4fc6dff3`) then ran three fresh servers. This binary predates the final distributed/queued-prefill/legacy/cache-warmup routing review fixes; the repeated measurements below are retained as calibration rather than misrepresented as final-source validation. + +| Candidate 2 case | Observed HTTP results | Estimate (GiB) | Process physical peak (GiB) | +|---|---|---:|---:| +| Long completion, 6,001 prompt tokens | Six 503 responses, stream and nonstream | 27.337 | 16.9 | +| Long chat, 6,020 prompt tokens | Six 503 responses, stream and nonstream | 45.606–45.702 | 16.9 | +| Short following each rejection | Twelve 200 responses; all streams completed | 17.813–17.960 | 16.9–17.0 | +| Medium completion, 1,501 prompt tokens | Three 200 responses | 22.423 | 17.7 | +| Beyond-context prompt | Three 400 responses, including stream requests | Not admitted | Not measured separately | + +The larger chat estimate used an actual 6,014-token history-boundary forward, versus a 2,048-token completion forward. Estimates follow the real plan, not merely the configured chunk size. Retained snapshots explain the later small residency increase in this intermediate candidate; final policy disables donation as well as reuse. + +Medium accepted-request estimate median was 22.423 GiB with less than 0.00002 GiB range; measured physical peak was 17.7 GiB in all three repeats at vmmap resolution. One-minute machine load at run start was 6.35, 7.10, and 10.73. HTTP model-list readiness took 7.92, 3.06, and 3.33 seconds (median 3.33, range 4.86); this is not model-load timing because worker loading may continue asynchronously. + +A separate candidate-2 `--no-warmup` run also rejected all four long cases, served all four following short cases and the medium request, and preserved context 400 precedence and health. Its first guard saw approximately 17.78 GB of active allocations rather than a lazy zero-weight baseline. [Three-repeat raw results](data/nemotron-h-request-memory-m1ultra-2026-10-13/candidate2.json), [no-warmup results](data/nemotron-h-request-memory-m1ultra-2026-10-13/candidate2-no-warmup.json). + +## Final-source validation + +Release build at source commit `07db6e30` passed; artifact SHA-256 `1c4673addeb123b8d3ef2758f69dc0c54811d1d061911dbcb86aa486cc6065c5`. This includes all admission/HTTP/security fixes. A subsequent CI-only stack-frame fix replaces a by-value scheduler builder with an in-place setter; that change does not alter these estimates or request geometry, and its final smoke is recorded below. + +| Final reviewed case | Outcome | Estimate (GiB) | Process physical peak (GiB) | +|---|---|---:|---:| +| Three normal servers: long completion/chat, both stream modes | 12 typed JSON 503 responses before SSE | 27.337–27.351 | 16.9 | +| Short request after each normal rejection | 12 successful 200 responses | 17.813–17.865 | 16.9 | +| Medium completion after each normal sequence | 3 successful 200 responses | 22.328 | 17.6 | +| One `--no-warmup` server | 4 long 503, 4 following short 200, medium 200 | Same ranges, within 12 bytes | 16.8–17.6 | +| One `--no-batch` server | 4 long 503, 4 following short 200, medium 200 | Long 45.506–45.634 | 16.9–17.6 | +| Qwen2.5-7B, target-only budget set to one byte | Stream and nonstream both 200, health 200 | Policy intentionally absent | Not measured | + +Every tested server remained healthy. All normal and no-warmup beyond-context checks returned 400 rather than memory 503. Legacy `--no-batch` context-limit handling is unchanged and was not asserted. Four invalid override startup probes (`0`, `-1G`, `NaN`, and `18446744073709551616`) failed with the explicit configuration diagnostic. + +With guarded prompt snapshot donation disabled, normal long chat and completion now both use 2,048-token largest forwards. The legacy unchunked path correctly uses the entire 6,001/6,020-token prompt and produces a larger estimate, demonstrating that the guard follows actual execution geometry rather than the global chunk setting. Medium accepted estimates were identical across all three normal repeats (median 22.328 GiB, range zero); physical peak was 17.6 GiB each at vmmap resolution. No elapsed-time speed claim is made. + +Normal run starting one-minute loads were 6.44, 6.90, and 11.01. Normal safetensor-loading-start to eager-materialization-end took 6.056, 1.926, and 2.152 seconds (median 2.152, range 4.130); these include disk I/O and sanitization, so they do not isolate added evaluation overhead. The no-warmup run measured 2.818 seconds over the same interval. There is no isolated eager-vs-lazy startup benchmark. [Startup log excerpts](data/nemotron-h-request-memory-m1ultra-2026-10-13/final-startup-excerpts.txt) record the active weight baseline of 17,777,861,888 bytes before any first request. + +Exact commands (normal run repeated three times; other modes once): + +```bash +D=docs/benchmark_results/data/nemotron-h-request-memory-m1ultra-2026-10-13 +python3 "$D/harness.py" --binary target/release/mlxcel-server --model models/mlx/nvidia-nemotron-3-nano-30b-a3b-4bit --output /tmp/final.json --budget 24G --reps 3 +python3 "$D/harness.py" --binary target/release/mlxcel-server --model models/mlx/nvidia-nemotron-3-nano-30b-a3b-4bit --output /tmp/no-warmup.json --budget 24G --reps 1 --no-warmup +python3 "$D/harness.py" --binary target/release/mlxcel-server --model models/mlx/nvidia-nemotron-3-nano-30b-a3b-4bit --output /tmp/no-batch.json --budget 24G --reps 1 --no-batch +python3 "$D/other_family.py" --binary target/release/mlxcel-server --output /tmp/qwen.json +``` + +Raw records: [normal three repeats](data/nemotron-h-request-memory-m1ultra-2026-10-13/final.json), [no warmup](data/nemotron-h-request-memory-m1ultra-2026-10-13/final-no-warmup.json), [legacy no batch](data/nemotron-h-request-memory-m1ultra-2026-10-13/final-no-batch.json), [Qwen](data/nemotron-h-request-memory-m1ultra-2026-10-13/final-qwen.json), [invalid startup values](data/nemotron-h-request-memory-m1ultra-2026-10-13/invalid-env.json). + +Local validation: 26 `request_memory` tests; 23 Nemotron-H model tests passed with three ignored; provider 30 and native-route 34 tests passed (these namespaces overlap the 26, not additive totals). Release build, `cargo check --tests --features metal,accelerate`, `cargo clippy --lib --tests --features metal,accelerate -- -D warnings`, formatting, and diff checks passed. An Ubuntu two-host logical CI test exposed a debug-worker stack overflow after adding a by-value scheduler builder; the in-place-setter correction subsequently passed the formerly failing Linux two-host logical real-model CI step. Remaining workflow status is tracked in the PR. + +## Post-setter final artifact smoke + +The final scheduler setup correction at `9d2a1584` avoids copying the large scheduler through another by-value builder frame. Its release build passed, and artifact SHA-256 `e7a5cc9997a8c0eeceba6a69769353eed35395e906b270d2c62a5f437e2a7062` was retested with the same harness: one normal server, one legacy `--no-batch` server, and the Qwen smoke. Normal and legacy each produced four within-context long-request 503 JSON responses before SSE, four successful following short requests with complete streams, and one accepted medium request; health remained 200 throughout. The normal context-overflow check remained 400. Qwen streaming and nonstreaming both returned 200 under the one-byte target-only budget. + +Estimates match the preceding reviewed artifact: normal long requests 27.337–27.351 GiB, unchunked legacy long requests 45.506–45.634 GiB, and accepted medium requests 22.328 GiB. Process-lifetime physical peaks were 16.9 GiB for the rejection/short lifecycle and 17.6 GiB after the medium request. This is a final functional smoke, not a new repeated performance measurement. The earlier three-repeat calibration and no-warmup evidence remain explicitly tied to their original artifact. + +Reproduction uses the commands above with `--reps 1` and output files named `post-setter-normal.json`, `post-setter-legacy.json`, and `post-setter-qwen.json`. Raw records: [normal](data/nemotron-h-request-memory-m1ultra-2026-10-13/post-setter-normal.json), [legacy](data/nemotron-h-request-memory-m1ultra-2026-10-13/post-setter-legacy.json), [Qwen](data/nemotron-h-request-memory-m1ultra-2026-10-13/post-setter-qwen.json). + +## Limits and unmeasured cells + +No speed claim, universal OOM guarantee, or device-wide reservation is made. External memory pressure, allocator fragmentation, unexpected kernel workspace, M4/M5, CUDA, 2-bit checkpoints, multi-slot scheduling, and distributed memory-pressure behavior are unmeasured. Runtime/unfused LoRA is explicitly rejected during Metal Nemotron-H construction because arbitrary-rank float32 scratch is not modeled; fused checkpoint weights use the common eager-construction path, but no real adapter checkpoint was measured here. Unit tests cover scope exclusion, cancellation/cleanup, checked arithmetic and final-gate routing; they do not replace the real checkpoint cases above. diff --git a/docs/environment-variables.md b/docs/environment-variables.md index 30f2d9ad7..fc0ecc7be 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -109,6 +109,7 @@ Both server entry points implement llama-server b10621's Vertex AI custom-contai | `MLXCEL_DEVICE` | `gpu`, `metal`, `cpu` | `gpu` hint | `cpu` requests CPU execution. Invalid values are ignored with a warning and treated as `gpu`; if no GPU backend is available, runtime falls back to CPU. The startup lines say which of the two put the runtime on the CPU: a `cpu` request is reported as such, next to whether a GPU backend was available, so a CPU line on a GPU host is not mistaken for a missing backend (issue #1421). | | `MLXCEL_WIRED_LIMIT` | `max`, `0`, `none`, bytes, `NK`/`NKB`, `NM`/`NMB`, `NG`/`NGB` | `max` | Apple Silicon GPU wired-memory limit. Unset/empty/`max` sets MLX's reported GPU max memory size; `0`/`none` disables the limit; numeric values set an explicit limit. | | `MLXCEL_MEMORY_LIMIT` | `0`, `none`, bytes, `NK`/`NKB`, `NM`/`NMB`, `NG`/`NGB` | unset | Soft MLX allocator memory cap. Unset/`0`/`none` lets MLX use its backend default; numeric values cap the allocator and make MLX raise an exception once allocations would push the working set past this value. Also feeds the `mlxcel inspect` / `--estimate-memory` preflight as the authoritative "available unified memory" figure when nonzero, through the same parse as the allocator cap (issue #1317), so `4G` and `4GB` produce the same preflight figure. | +| `MLXCEL_REQUEST_MEMORY_BUDGET` | positive bytes, `NK`/`NKB`, `NM`/`NMB`, `NG`/`NGB` | minimum positive runtime device/wired/allocator cap | Lower-only request admission ceiling for Metal text Nemotron-H at effective batch width one. Invalid explicit values (including zero, `none`, empty, nonfinite and overflowing values) stop server startup. Oversized requests return HTTP 503 before prefill or SSE headers; see [request memory admission](request-memory-admission.md). | | `MLXCEL_CACHE_LIMIT` | `0`, `none`, bytes, `NK`/`NKB`, `NM`/`NMB`, `NG`/`NGB` | `2GB` on ROCm builds, unset elsewhere | Bound on MLX's buffer cache (issue #627). Numeric values cap cached-buffer bytes via `set_cache_limit`; `0`/`none` leaves MLX's default cache behavior. Unset applies the backend default: none on Metal and CUDA, 2 GiB on ROCm (issue #2062), where the allocator's own default lets the exact-size cache grow to its 76.8 GiB memory limit on the gfx1151 carve-out; a value that does not parse keeps that default. Before #2062 the ROCm allocator ignored the limit, so this variable had no effect there. See [Installation](installation.md#memory-footprint) for how the default was chosen. On CUDA this is the intended way to bound cache growth now that the periodic decode-loop clear is disabled by default (see `MLXCEL_CACHE_CLEAR_INTERVAL`): it keeps the memory pool bounded without the per-step churn that defeats CUDA-graph reuse (ml-explore/mlx#2358). | | `MLXCEL_CACHE_CLEAR_INTERVAL` | `0` (disable), positive integer (token cadence) | `0` on CUDA, `256` on Metal/CPU | Cadence of the periodic `clear_memory_cache` in the decode loops and batch scheduler, in generated tokens (issue #627). `0` disables the periodic clear. The default is backend-aware: on CUDA the clear churns the memory pool and defeats CUDA-graph reuse (ml-explore/mlx#2358) so it is off by default (bound the cache with `MLXCEL_CACHE_LIMIT` instead); on Metal/CPU the cheap 256-token trim used by Python mlx-lm is kept. | | `MLX_CUDA_GRAPH_CACHE_SIZE` (MLX-native, CUDA only) | unsigned integer capacity | `2000` on CUDA builds (MLX's own default is `400`) | LRU capacity for MLX's captured CUDA-graph cache. MLX keys the cache by graph shape and its `lru_cache.h` also counts a lifetime miss counter that never resets (not on hits, not on trim), throwing a fatal `Cache thrashing` `runtime_error` once lifetime misses pass `2 * capacity` (800 at MLX's default 400). Any long-lived, shape-diverse CUDA server crosses that threshold over its lifetime, and speculative or batched decode reaches it fastest because draft/verify phases times varying batch sizes and sequence-length buckets multiply the number of distinct graph shapes; the throw is a whole-process abort, not a request-level error, so it drops every in-flight request (issue #818). mlxcel raises the default to `2000`, validated sufficient (13/13 requests across bursts on GB10). This is an LRU cap, not a preallocation, so it only costs memory as distinct graph shapes accumulate. An explicit `MLX_CUDA_GRAPH_CACHE_SIZE` always overrides. Read only by MLX's CUDA backend, so it is a harmless no-op on Metal/CPU. | @@ -140,7 +141,7 @@ Both server entry points implement llama-server b10621's Vertex AI custom-contai | `MLXCEL_DECODE_TIMEOUT` | seconds | `600` | Per-request decode watchdog: cancels a request whose generation stops producing tokens. `--decode-timeout` takes precedence. This is the control `--timeout` / `LLAMA_ARG_TIMEOUT` carried before v0.7.0 (#1432); that spelling is now the HTTP socket read/write timeout. `0` falls back to the built-in 300-second guard with a warning rather than expiring every request. | | `MLXCEL_SURGERY` | YAML file path | unset | Weight-load surgery configuration path. `--surgery` takes precedence. Active when the `surgery` feature is built, which it is by default; a `--no-default-features` build ignores it. See [Cargo feature flags](installation.md#cargo-feature-flags). | -The three size-valued variables above (`MLXCEL_WIRED_LIMIT`, +The wired, allocator, and cache size variables (`MLXCEL_WIRED_LIMIT`, `MLXCEL_MEMORY_LIMIT`, `MLXCEL_CACHE_LIMIT`) share one grammar, parsed in one place (`parse_memory_size` in `src/execution/runtime.rs`). Values are trimmed and case-insensitive; a `K`/`KB`, `M`/`MB` or `G`/`GB` suffix scales by the @@ -152,6 +153,8 @@ and `--estimate-memory` preflight reads `MLXCEL_MEMORY_LIMIT` through that same parser, so the availability figure it prints matches the cap the allocator will actually apply (issue #1317). +`MLXCEL_REQUEST_MEMORY_BUDGET` uses the same size suffixes, but an explicit value must be positive and finite; zero, `none`, empty, and overflowing values are errors, not disable switches. It only lowers the runtime request ceiling. + When `mlxcel inspect --json` is used, the same availability figure is exposed as raw bytes in `budget_bytes`; `weights_bytes`, `kv_bytes_total`, `activation_bytes`, `backend_inflight_bytes` (non-zero only on ROCm builds; part of `headroom_bytes`), `headroom_bytes`, and `total_bytes` are also raw byte counts so downstream recipe tooling can compare model and hardware fits without scraping the text banner. ## Server context sizing diff --git a/docs/installation.md b/docs/installation.md index 2d669f819..020e70b77 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -615,6 +615,7 @@ made that unnecessary; not re-measured since), gpt-oss-20b-MXFP4-Q4 about 8 tok/ | `MLXCEL_QUIET_JIT` | Suppress the one-time "compiling CUDA kernels" notice on a cold first run | unset (notice shown) | | `MLXCEL_DEVICE` | Runtime device hint (`gpu`, `metal`, or `cpu`) | `gpu` | | `MLXCEL_WIRED_LIMIT` | Apple Silicon wired-memory ceiling, e.g. `64GB`; `0`/`none` disables it | `max` | +| `MLXCEL_REQUEST_MEMORY_BUDGET` | Lower-only positive byte/size ceiling for [Nemotron-H request admission](request-memory-admission.md), e.g. `24GB`; invalid explicit values stop startup | runtime device/wired/allocator minimum | | `MLX_ROCM_*` and other ROCm backend variables | ROCm backend tuning, A/B and diagnostics, for example `MLX_ROCM_GATHER_QMV_EXPERT_BATCHED`, `MLX_ROCM_MAX_INFLIGHT_MB`, `MLX_NO_HIPBLASLT`; see [ROCm backend variables](environment-variables.md#rocm-backend-variables) | see that section | | `LLAMA_ARG_*` | Environment-backed server options accepted by clap | unset | diff --git a/docs/request-memory-admission.md b/docs/request-memory-admission.md new file mode 100644 index 000000000..9186436b7 --- /dev/null +++ b/docs/request-memory-admission.md @@ -0,0 +1,35 @@ +# Nemotron-H request memory admission + +Metal text-only Nemotron-H serving checks a conservative memory estimate before starting each request. The scheduler clamps this family to one active slot and waits for preceding work to finish before sampling active allocations. A request rejected by this guard never runs GPU forward work, and a streaming request receives HTTP 503 before SSE headers are committed. `/health` remains available; smaller requests can run afterwards. + +## Budget and errors + +The default budget is the minimum positive finite value among the device's recommended working-set size, the **actually configured** wired-memory cap, and the current MLX allocator cap. The `get_wired_limit()` MLX query reports a device recommendation, not the configured wired cap; the runtime records the latter separately. + +`MLXCEL_REQUEST_MEMORY_BUDGET=24G` can lower that budget, never raise it or supply a missing runtime budget. Size suffixes follow the existing binary-unit grammar (for example `1.5G`); explicit zero, empty, `none`, negative, nonfinite, and overflowing values fail server startup. Unknown dimensions, an unknown budget, or arithmetic overflow fail request admission closed. + +```json +{ + "error": { + "message": "request_memory_budget_exceeded: estimated_bytes=..., budget_bytes=...; shorten the prompt/output or lower retained cache usage", + "type": "request_memory_budget_exceeded", + "code": "request_memory_budget_exceeded" + } +} +``` + +The error message reports estimated and budget bytes (an unknown estimate uses `18446744073709551615`; an unknown budget uses `0`). Existing context-length checks take precedence and remain HTTP 400. Memory admission logs include prompt/output lengths, largest-forward span, resident allocations and the component breakdown. + +## What is estimated + +- Loaded, normalized model dimensions and embedding output dtype, not a checkpoint-specific size table. +- Current MLX **active** memory, including model weights and retained prompt snapshots. The shared Metal Nemotron-H constructor materializes sanitized weights (including fused adapter graphs) in place before serving, including with `--no-warmup`, so a lazy-weight graph cannot masquerade as zero residency. Reclaimable allocator free-cache is neither charged as permanent residency nor subtracted from active memory. +- Full cold attention KV for prompt plus planned output, rounded to allocation granularity, with room for growth copies and snapshots; float32 recurrent state and convolution state. With context shifting disabled, the context stop bounds output growth. With shifting enabled no unsupported trim credit is taken for Nemotron-H's model-owned caches. +- The largest padded forward in the actual `PrefillPlan`, respecting model chunking opt-out and history-boundary splits, for residuals, MLP/MoE work, logits, and a conservative float32 attention-score fallback. +- Float32 SSD linear buffers and bounded internal-chunk scratch. The SSD internal chunk is independent of the server prefill chunk. + +Guarded requests currently prefill cold without restoring or donating prompt snapshots, and skip background prompt-cache warmups: snapshot restoration could otherwise perform request work while the previous request is still active. Retained snapshots remain counted in the resident baseline. This deliberately favors safety over cache-hit latency and can reject requests that would fit with optimistic cache reuse or fused kernels. Clear or reduce retained prompt caches, or shorten prompt/output bounds, rather than expecting the override to bypass the device ceiling. + +## Limits + +This is not a universal OOM guarantee or a device-wide reservation. Other processes, driver allocations, fragmentation, and unexpected kernel workspace can consume memory after admission. CUDA/ROCm/CPU, other model families, multimodal paths, speculative decoding, distributed handoffs, and multi-slot reservations are outside this policy. Both scheduler-backed serving and legacy `--no-batch` serving use the guard. Other unsupported paths retain their existing behavior. Metal Nemotron-H runtime (unfused) LoRA fails at model construction with a clear error: its arbitrary-rank float32 activation workspace is not modeled. Use a fused adapter checkpoint instead. The shared Nemotron-H constructor also eagerly materializes weights and rejects staged runtime adapters when embedded by VLM/VoiceChat; those models do not acquire this request admission policy. The guard does not change model numerics or retry a request after an allocation failure. diff --git a/scripts/webui/verify_activity_performance.py b/scripts/webui/verify_activity_performance.py index 60f7488e0..9d680e7d3 100755 --- a/scripts/webui/verify_activity_performance.py +++ b/scripts/webui/verify_activity_performance.py @@ -622,6 +622,8 @@ def run_activity_script(h: Harness, model_id: str, inference_id: str) -> tuple[d env = {**h.env, "WEBUI_PERF_BASE": h.base_prefixed + "/", "WEBUI_PERF_KEY_FILE": str(h.key_file), "WEBUI_PERF_INFERENCE_MODEL": inference_id, "WEBUI_PERF_MODEL_ID": model_id, "WEBUI_PERF_OUTPUT": str(output), "WEBUI_PERF_MODE": h.args.perf_mode} if h.args.prompt_file: env["WEBUI_PERF_PROMPT_FILE"] = str(h.args.prompt_file.resolve()) + if h.args.nvidia_smi: + env["WEBUI_PERF_NVIDIA_SMI"] = str(h.args.nvidia_smi.resolve()) command = [h.args.node_bin, "webui/scripts/activity-performance.mjs"] with open_private_append(log_path) as log_file: proc = subprocess.Popen(command, cwd=h.args.repo_root, env=env, stdout=log_file, stderr=subprocess.STDOUT, start_new_session=True) @@ -678,6 +680,7 @@ def parse_args(argv: list[str]) -> argparse.Namespace: parser.add_argument("--work-dir", type=Path, help="Parent directory for a new private 0700 run directory; never reused directly") parser.add_argument("--repo-root", type=Path, default=Path.cwd()) parser.add_argument("--node-bin", default=shutil.which("node") or "node") + parser.add_argument("--nvidia-smi", type=Path, help="Explicit nvidia-smi executable used only for per-sample diagnostics") parser.add_argument("--prompt-file", type=Path) parser.add_argument("--port", type=int, default=0) parser.add_argument("--activity-timeout", type=float, default=1800.0) @@ -693,6 +696,8 @@ def parse_args(argv: list[str]) -> argparse.Namespace: parser.error("--model must be a checkpoint directory") if args.host_state is not None and not args.host_state.is_file(): parser.error("--host-state must name the file activity_gate_host.py precheck wrote") + if args.nvidia_smi is not None and (not args.nvidia_smi.is_file() or not os.access(args.nvidia_smi, os.X_OK)): + parser.error("--nvidia-smi must be an executable file") if args.port == 0: args.port = free_port() return args diff --git a/scripts/webui/verify_activity_performance_tests.py b/scripts/webui/verify_activity_performance_tests.py index c5a7e0deb..93ebec044 100755 --- a/scripts/webui/verify_activity_performance_tests.py +++ b/scripts/webui/verify_activity_performance_tests.py @@ -140,6 +140,9 @@ def test_parse_args_requires_build_sha_and_model_directory(self) -> None: server.chmod(0o700) model = root / "model" model.mkdir() + nvidia_smi = root / "nvidia-smi" + nvidia_smi.write_text("#!/bin/sh\n") + nvidia_smi.chmod(0o700) args = verify.parse_args([ "--server-bin", str(server), "--model", str(model), @@ -147,10 +150,12 @@ def test_parse_args_requires_build_sha_and_model_directory(self) -> None: "--source-sha", "a" * 40, "--features", "cuda,webui", "--evidence", str(root / "evidence.json"), + "--nvidia-smi", str(nvidia_smi), "--port", "53001", ]) self.assertEqual(args.features, ["cuda", "webui"]) self.assertEqual(args.port, 53001) + self.assertEqual(args.nvidia_smi, nvidia_smi) nonexec = root / "nonexec" nonexec.write_text("#!/bin/sh\n") with redirect_stderr(StringIO()): diff --git a/src/distributed/disaggregated/coordinator.rs b/src/distributed/disaggregated/coordinator.rs index d99a1f141..e2cba609a 100644 --- a/src/distributed/disaggregated/coordinator.rs +++ b/src/distributed/disaggregated/coordinator.rs @@ -903,7 +903,8 @@ fn drain_generation_events(rx: &mpsc::Receiver) -> DrainedGenerat // The prefill snapshot is a streaming-only detail of the native // completion route; a disaggregated drain has no frame to put it // on (#1441). - GenerateEvent::Prefill(_) => {} + GenerateEvent::Prefill(_) | GenerateEvent::Admitted => {} + GenerateEvent::RequestMemoryRejected(e) => error = Some(e.to_string()), GenerateEvent::Error(e) => error = Some(e), } } diff --git a/src/execution/mod.rs b/src/execution/mod.rs index 9a985dcbf..29fe53ae1 100644 --- a/src/execution/mod.rs +++ b/src/execution/mod.rs @@ -23,5 +23,6 @@ pub mod kv_arch; pub mod kv_cache_advisor; pub mod memory_estimate; pub mod quant_advisor; +pub mod request_memory; pub mod runtime; pub mod sampling; diff --git a/src/execution/request_memory.rs b/src/execution/request_memory.rs new file mode 100644 index 000000000..e50dc44bc --- /dev/null +++ b/src/execution/request_memory.rs @@ -0,0 +1,265 @@ +// Copyright 2025-2026 Lablup Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Conservative request admission for single-slot Metal Nemotron-H serving. +//! +//! This is an analytical allocation envelope, not a reservation against other +//! GPU clients. Resident *active* allocations include retained prompt snapshots; +//! allocator free-cache is reclaimable and is neither added nor subtracted. + +use crate::models::nemotron_h::NemotronHConfig; +use mlxcel_core::memory::MemorySnapshot; + +pub const REQUEST_MEMORY_BUDGET_ENV: &str = "MLXCEL_REQUEST_MEMORY_BUDGET"; + +#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)] +#[error( + "request_memory_budget_exceeded: estimated_bytes={estimated_bytes}, budget_bytes={budget_bytes}; shorten the prompt/output or lower retained cache usage" +)] +pub struct RequestMemoryRejection { + /// u64::MAX denotes an unknown/overflowing estimate (fail closed). + pub estimated_bytes: u64, + /// Zero denotes an unknown runtime budget (fail closed). + pub budget_bytes: u64, +} + +impl RequestMemoryRejection { + pub const CODE: &'static str = "request_memory_budget_exceeded"; +} + +/// The shared size grammar, but explicit zero and saturating values are invalid. +/// In particular `none`, NaN, infinity and overflowing sizes cannot disable safety. +pub fn parse_budget(value: &str) -> anyhow::Result { + super::runtime::parse_memory_size(value) + .filter(|bytes| *bytes > 0 && *bytes < u64::MAX) + .ok_or_else(|| { + anyhow::anyhow!( + "{REQUEST_MEMORY_BUDGET_ENV} must be a positive finite byte size, for example 12G" + ) + }) +} + +pub fn budget_override() -> anyhow::Result> { + match std::env::var(REQUEST_MEMORY_BUDGET_ENV) { + Ok(value) => parse_budget(&value).map(Some), + Err(std::env::VarError::NotPresent) => Ok(None), + Err(err) => Err(anyhow::anyhow!("{REQUEST_MEMORY_BUDGET_ENV}: {err}")), + } +} + +/// An override may lower a known device budget, never create or raise one. +pub fn resolve_budget( + recommended: u64, + wired: u64, + allocator: u64, + override_bytes: Option, +) -> Option { + let default = [recommended, wired, allocator] + .into_iter() + .filter(|v| *v > 0 && *v < u64::MAX) + .min()?; + Some(override_bytes.map_or(default, |value| default.min(value))) +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct RequestMemoryEstimate { + pub resident_bytes: u64, + pub kv_bytes: u64, + pub recurrent_bytes: u64, + pub activation_bytes: u64, + pub ssd_bytes: u64, + pub total_bytes: u64, +} + +fn mul(values: &[u64]) -> Option { + values.iter().try_fold(1u64, |a, b| a.checked_mul(*b)) +} +fn sum(values: &[u64]) -> Option { + values.iter().try_fold(0u64, |a, b| a.checked_add(*b)) +} + +/// Peak retained positions. With shift enabled the model-owned Nemotron cache +/// does not promise a trim, so do NOT credit the scheduler's dense-pool KV cap. +/// Without shifting the context stop bounds generation; max_tokens=0 still +/// performs one prefill/sample in the existing generation contract. +pub fn retained_tokens(prompt: u64, output: u64, stop_context: Option) -> Option { + let output = output.max(1); + match stop_context { + Some(cap) if prompt < cap => Some(prompt.checked_add(output.min(cap - prompt))?), + Some(_) => None, + None => prompt.checked_add(output), + } +} + +/// Checked envelope of the *loaded normalized config*. `forward_tokens` is the +/// largest padded piece of the actual cold PrefillPlan (not the global chunk). +/// `element_bytes` is the loaded embedding output dtype (f16/bf16=2, f32=4). +/// +/// We deliberately charge cold KV even with a warm snapshot, and triple new KV +/// for growth copies plus snapshot ownership. Layer residuals may stay live in lazy +/// graphs, so they are charged across all layers. Attention scores include a +/// float32 fallback envelope even when fused SDPA normally avoids them. SSD's +/// quadratic scratch uses its *internal* chunk, independent of the outer plan. +pub fn estimate( + config: &NemotronHConfig, + forward_tokens: u64, + retained: u64, + element_bytes: u64, + memory: MemorySnapshot, +) -> Option { + let c = config; + let pattern = c.hybrid_override_pattern.as_ref()?; + if pattern.len() != c.num_hidden_layers + || pattern.is_empty() + || forward_tokens == 0 + || retained < forward_tokens + || ![2, 4].contains(&element_bytes) + || [ + c.hidden_size, + c.vocab_size, + c.num_attention_heads, + c.num_key_value_heads, + c.mamba_num_heads, + c.mamba_head_dim, + c.ssm_state_size, + c.n_groups, + c.conv_kernel, + c.chunk_size, + ] + .contains(&0) + { + return None; + } + if pattern + .iter() + .any(|p| !matches!(p.as_str(), "*" | "M" | "-" | "E")) + { + return None; + } + let heads = c.mamba_num_heads as u64; + let dim = c.mamba_head_dim as u64; + let state = c.ssm_state_size as u64; + let hidden = c.hidden_size as u64; + let attn = pattern.iter().filter(|p| p.as_str() == "*").count() as u64; + let mamba = pattern.iter().filter(|p| p.as_str() == "M").count() as u64; + let head_dim = c.head_dim.unwrap_or(c.hidden_size / c.num_attention_heads) as u64; + if head_dim == 0 { + return None; + } + // Reuse architecture classification, but validate every product before the + // older startup estimator's saturating arithmetic can hide an overflow. + let step = mlxcel_core::hardware::KV_CACHE_ALLOC_STEP; + let rounded = retained + .checked_add(step - 1)? + .checked_div(step)? + .checked_mul(step)?; + let expected_fp16 = mul(&[attn, 2, c.num_key_value_heads as u64, head_dim, rounded, 2])?; + let arch = serde_json::json!({ + "model_type": "nemotron_h", "num_hidden_layers": c.num_hidden_layers, + "num_attention_heads": c.num_attention_heads, "num_key_value_heads": c.num_key_value_heads, + "hidden_size": c.hidden_size, "head_dim": head_dim, "hybrid_override_pattern": pattern, + }); + let classified = super::kv_arch::estimate_kv_arch_from_config(&arch, retained, false, 1)?; + if classified.total_bytes != expected_fp16 { + return None; + } + let kv_bytes = mul(&[expected_fp16 / 2, element_bytes, 3])?; + let conv_dim = sum(&[mul(&[heads, dim])?, mul(&[2, c.n_groups as u64, state])?])?; + let recurrent_bytes = mul(&[ + mamba, + 2, + sum(&[ + mul(&[heads, dim, state, 4])?, + mul(&[c.conv_kernel as u64, conv_dim, element_bytes])?, + ])?, + ])?; + let t = forward_tokens; + let inner = t.min(c.chunk_size as u64); + // Projection + convolution + float32 x/dtx + grouped B/C + dt chains + + // accumulated outputs/concatenation. B/C repeats exist only per inner chunk. + let ssd_bytes = if mamba == 0 { + 0 + } else { + sum(&[ + mul(&[ + t, + 4, + sum(&[ + mul(&[12, heads, dim])?, + mul(&[6, c.n_groups as u64, state])?, + mul(&[8, heads])?, + ])?, + ])?, + mul(&[inner, inner, heads, 4, 10])?, + mul(&[inner, heads, state, 4, 4])?, + mul(&[heads, dim, state, 4, 4])?, + ])? + }; + let experts = c.num_experts_per_tok.unwrap_or(1).max(1) as u64; + let intermediate = + (c.intermediate_size as u64).max(c.moe_intermediate_size.unwrap_or(0) as u64); + let activation_bytes = sum(&[ + mul(&[t, hidden, c.num_hidden_layers as u64, element_bytes, 4])?, + mul(&[t, c.vocab_size as u64, 4, 2])?, + mul(&[t, intermediate, experts, 4, 4])?, + mul(&[ + t, + c.moe_shared_expert_intermediate_size.unwrap_or(0) as u64, + 4, + 4, + ])?, + // Matmul temporary/dequantization workspace, counted even if the + // quantized kernel avoids it; no second copy of resident model weights. + mul(&[hidden, intermediate.max(conv_dim), 4, 3, experts])?, + if attn == 0 { + 0 + } else { + mul(&[t, retained, c.num_attention_heads as u64, 4, 3])? + }, + ])?; + let total_bytes = sum(&[ + memory.active_bytes, + kv_bytes, + recurrent_bytes, + activation_bytes, + ssd_bytes, + ])?; + Some(RequestMemoryEstimate { + resident_bytes: memory.active_bytes, + kv_bytes, + recurrent_bytes, + activation_bytes, + ssd_bytes, + total_bytes, + }) +} + +pub fn check( + estimate: Option, + budget: Option, +) -> Result { + match (estimate, budget) { + (Some(estimate), Some(budget)) if budget > 0 && estimate.total_bytes <= budget => { + Ok(estimate) + } + (estimate, budget) => Err(RequestMemoryRejection { + estimated_bytes: estimate.map_or(u64::MAX, |e| e.total_bytes), + budget_bytes: budget.unwrap_or(0), + }), + } +} + +#[cfg(test)] +#[path = "request_memory_tests.rs"] +mod tests; diff --git a/src/execution/request_memory_tests.rs b/src/execution/request_memory_tests.rs new file mode 100644 index 000000000..736c2795f --- /dev/null +++ b/src/execution/request_memory_tests.rs @@ -0,0 +1,168 @@ +// Copyright 2025-2026 Lablup Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; +use mlxcel_core::prefill_plan::{PrefillCaps, PrefillPlan}; + +fn config() -> NemotronHConfig { + let mut c: NemotronHConfig = serde_json::from_value(serde_json::json!({ + "model_type":"nemotron_h", "vocab_size":128, "hidden_size":64, + "intermediate_size":128, "num_hidden_layers":99, "num_attention_heads":4, + "num_key_value_heads":2, "mamba_num_heads":4, "mamba_head_dim":16, + "ssm_state_size":8, "conv_kernel":4, "n_groups":2, "chunk_size":32, + "layers_block_type":["mamba","attention","mlp"] + })) + .unwrap(); + c.post_init().unwrap(); + c +} + +fn estimated(t: u64, output: u64, active: u64, cache: u64) -> RequestMemoryEstimate { + estimate( + &config(), + t, + retained_tokens(t, output, None).unwrap(), + 2, + MemorySnapshot { + active_bytes: active, + cache_bytes: cache, + ..Default::default() + }, + ) + .unwrap() +} + +#[test] +fn request_memory_budget_boundaries_and_output_growth() { + let e = estimated(64, 1, 100, 0); + assert!(check(Some(e), Some(e.total_bytes)).is_ok()); + assert!(check(Some(e), Some(e.total_bytes + 1)).is_ok()); + assert!(check(Some(e), Some(e.total_bytes - 1)).is_err()); + let long = estimated(64, 1024, 100, 0); + assert!(long.kv_bytes > e.kv_bytes); + assert!(long.total_bytes > e.total_bytes); +} + +#[test] +fn request_memory_cache_accounting_counts_resident_once_not_allocator_cache() { + let cold = estimated(64, 1, 100, 0); + let free_cache = estimated(64, 1, 100, u64::MAX); + assert_eq!(cold, free_cache); + let retained_snapshot = estimated(64, 1, 500, 0); + assert_eq!(retained_snapshot.total_bytes - cold.total_bytes, 400); + assert_eq!(retained_snapshot.kv_bytes, cold.kv_bytes); +} + +#[test] +fn request_memory_budget_only_lowers_known_limits() { + assert_eq!(resolve_budget(1000, 700, 900, None), Some(700)); + assert_eq!(resolve_budget(1000, 700, 900, Some(5000)), Some(700)); + assert_eq!(resolve_budget(1000, 700, 900, Some(500)), Some(500)); + assert_eq!(resolve_budget(0, 0, 0, Some(500)), None); + assert_eq!(resolve_budget(1000, 0, 0, None), Some(1000)); + assert!(check(Some(estimated(1, 1, 0, 0)), None).is_err()); + assert!(check(None, Some(u64::MAX)).is_err()); +} + +#[test] +fn request_memory_invalid_explicit_budgets_fail_closed() { + for value in [ + "0", + "0G", + "none", + "", + "-1", + "-2G", + "NaNG", + "infG", + "1e300G", + "1.5", + "18446744073709551616", + ] { + assert!(parse_budget(value).is_err(), "{value}"); + } + assert_eq!(parse_budget("1.5G").unwrap(), 1536 * 1024 * 1024); + assert_eq!(parse_budget("1024").unwrap(), 1024); +} + +#[test] +fn request_memory_overflow_and_invalid_dimensions_fail_closed() { + assert!(retained_tokens(1, u64::MAX, None).is_none()); + assert_eq!(retained_tokens(10, u64::MAX, Some(128)), Some(128)); + assert!(estimate(&config(), u64::MAX, u64::MAX, 2, Default::default()).is_none()); + assert!( + estimate( + &config(), + 64, + 65, + 2, + MemorySnapshot { + active_bytes: u64::MAX, + ..Default::default() + } + ) + .is_none() + ); + let mut c = config(); + c.mamba_head_dim = usize::MAX; + assert!(estimate(&c, 64, 65, 2, Default::default()).is_none()); + c = config(); + c.num_attention_heads = 0; + assert!(estimate(&c, 64, 65, 2, Default::default()).is_none()); + assert!(estimate(&config(), 64, 65, 0, Default::default()).is_none()); +} + +#[test] +fn request_memory_context_stop_and_untrimmable_shift() { + assert_eq!(retained_tokens(100, 1000, Some(128)), Some(128)); + // No trim credit for Nemotron's model-owned caches when shifting is on. + assert_eq!(retained_tokens(100, 1000, None), Some(1100)); + assert_eq!(retained_tokens(100, 0, None), Some(101)); + assert_eq!(retained_tokens(128, 1, Some(128)), None); +} + +#[test] +fn request_memory_normalized_pattern_and_loaded_dtype() { + let c = config(); + assert_eq!(c.num_hidden_layers, 3); + let half = estimate(&c, 64, 65, 2, Default::default()).unwrap(); + let full = estimate(&c, 64, 65, 4, Default::default()).unwrap(); + assert_eq!(full.kv_bytes, half.kv_bytes * 2); + assert_eq!(half.ssd_bytes, full.ssd_bytes); // SSD's float32 boundary. + assert!(full.total_bytes > half.total_bytes); +} + +#[test] +fn request_memory_ssd_internal_chunk_is_not_outer_forward() { + let mut c = config(); + let small = estimate(&c, 1024, 1025, 2, Default::default()).unwrap(); + c.chunk_size = 128; + let large = estimate(&c, 1024, 1025, 2, Default::default()).unwrap(); + assert!(large.ssd_bytes > small.ssd_bytes); + assert_eq!(small.activation_bytes, large.activation_bytes); +} + +#[test] +fn request_memory_prefill_plan_honors_model_opt_out() { + let caps = PrefillCaps { + supports_chunked_prefill: false, + ..Default::default() + }; + let plan = PrefillPlan::new(8192, 256, caps); + assert!(plan.is_single_pass()); + assert_eq!(plan.pieces()[0].padded_len, 8192); + let actual = estimated(8192, 1, 0, 0); + let misleading_global_chunk = estimated(256, 1, 0, 0); + assert!(actual.total_bytes > misleading_global_chunk.total_bytes * 10); +} diff --git a/src/execution/runtime.rs b/src/execution/runtime.rs index ee4881fb3..b70edfff0 100644 --- a/src/execution/runtime.rs +++ b/src/execution/runtime.rs @@ -20,6 +20,13 @@ use std::fmt; +static APPLIED_WIRED_LIMIT: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + +/// The cap actually configured at runtime, unlike get_wired_limit (device recommendation). +pub(crate) fn applied_wired_limit() -> u64 { + APPLIED_WIRED_LIMIT.load(std::sync::atomic::Ordering::Relaxed) +} + const RUNTIME_DEVICE_ENV: &str = "MLXCEL_DEVICE"; const WIRED_LIMIT_ENV: &str = "MLXCEL_WIRED_LIMIT"; /// Issue #55: optional soft cap on the MLX allocator. When set, the @@ -256,6 +263,11 @@ pub fn initialize_runtime() -> RuntimeSetup { None }; + APPLIED_WIRED_LIMIT.store( + wired_limit_bytes.unwrap_or(0) as u64, + std::sync::atomic::Ordering::Relaxed, + ); + // Issue #55: apply optional soft allocator cap regardless of device. // The MLX no-gpu CPU allocator also honours `set_memory_limit()`, so // the preflight (#56) can use this on Linux/CI just as on Apple diff --git a/src/models/nemotron_h.rs b/src/models/nemotron_h.rs index a52a4b898..95a104eae 100644 --- a/src/models/nemotron_h.rs +++ b/src/models/nemotron_h.rs @@ -1594,6 +1594,23 @@ impl Drop for NemotronHModel { } impl NemotronHModel { + /// Loaded normalized dimensions for single-slot request admission. + pub(crate) fn request_memory_config(&self) -> (&NemotronHConfig, u64) { + let dtype = match &self.embeddings { + UnifiedEmbedding::Quantized(e) => mlxcel_core::array_dtype(&e.scales), + UnifiedEmbedding::Regular(e) => mlxcel_core::array_dtype(&e.weight), + }; + let bytes = if dtype == mlxcel_core::dtype::FLOAT16 || dtype == mlxcel_core::dtype::BFLOAT16 + { + 2 + } else if dtype == mlxcel_core::dtype::FLOAT32 { + 4 + } else { + 0 + }; + (&self.config, bytes) + } + pub fn num_layers(&self) -> usize { self.config.num_hidden_layers } @@ -2339,6 +2356,30 @@ impl NemotronHModel { mut weights: WeightMap, block_types: Vec, ) -> Result> { + // Request admission samples MLX active allocations, not on-disk weight + // sizes. Materialize the sanitized weight graph at the shared construction boundary so + // --no-warmup (or a failed warmup) cannot make the first request see a + // zero/partial resident baseline. Evaluate in place: no second weight + // copy survives loading, and failure remains a startup load error. + // This includes fused-adapter loading and the VLM/VoiceChat users of + // this constructor; they only inherit eager startup, not admission. + if cfg!(all(target_os = "macos", feature = "metal")) && mlxcel_core::default_device_is_gpu() + { + // Runtime adapters add arbitrary-rank f32 activation work not yet + // modeled by admission. Fused adapters are ordinary weights below. + if mlxcel_core::runtime_lora::has_pending() { + mlxcel_core::runtime_lora::drain_unclaimed(); + return Err("Metal Nemotron-H request memory admission does not support runtime LoRA; use a fused adapter checkpoint instead".into()); + } + for weight in weights.values() { + mlxcel_core::try_eval(weight)?; + } + tracing::info!( + resident_bytes = mlxcel_core::memory::active_memory(), + "[NemotronH] materialized weights for request memory admission" + ); + } + // Get quantization parameters let group_size = config.group_size(); let bits = config.bits(); diff --git a/src/models/nemotron_h_tests.rs b/src/models/nemotron_h_tests.rs index b9ed13c91..bd5c58525 100644 --- a/src/models/nemotron_h_tests.rs +++ b/src/models/nemotron_h_tests.rs @@ -866,3 +866,27 @@ fn nemotron_h_rejects_expert_quantization_params_that_would_abort_gather_qmm() { NemotronHModel::from_weights(config_with(0, 0), regular, vec![BlockType::MoE]) .expect("a non-quantized expert plane must not be gated on quantization params"); } + +#[test] +#[cfg(all(target_os = "macos", feature = "metal"))] +fn request_memory_runtime_lora_rejects_before_construction_and_cleans_staging() { + if !mlxcel_core::default_device_is_gpu() { + return; + } + // Empty terms suffice to represent a staged adapter without GPU work. + mlxcel_core::runtime_lora::stage(std::collections::HashMap::from([( + "backbone.layers.0.mixer.q_proj".to_string(), + Vec::new(), + )])); + let config: NemotronHConfig = serde_json::from_str(&minimal_config_json("")).unwrap(); + let result = super::NemotronHModel::from_weights( + config, + mlxcel_core::weights::WeightMap::new(), + Vec::new(), + ); + let error = result + .err() + .expect("runtime adapter must fail before weights are read"); + assert!(error.to_string().contains("does not support runtime LoRA")); + assert!(!mlxcel_core::runtime_lora::has_pending()); +} diff --git a/src/server/batch/scheduler/admission.rs b/src/server/batch/scheduler/admission.rs index 41dc67b56..859cca098 100644 --- a/src/server/batch/scheduler/admission.rs +++ b/src/server/batch/scheduler/admission.rs @@ -362,7 +362,12 @@ impl BatchScheduler { // lands in a different digest bucket. Multimodal requests with sharing // off keep the legacy cold-prefill path (their pre-injection token // stream is not self-describing). - let ctx_ref = if is_multimodal && !vlm_sharing_ok { + // A guarded request must not restore/copy a warm snapshot while a + // preceding request is running. Keep retained snapshots in the measured + // resident baseline and run this request cold after final admission. + let ctx_ref = if (is_multimodal && !vlm_sharing_ok) + || (!is_multimodal && self.request_memory_policy_enabled()) + { None } else { options.prompt_cache_ctx.as_ref() @@ -504,7 +509,8 @@ impl BatchScheduler { // empty and the donate-back short-circuits. Multimodal requests are // stored only when VLM sharing is enabled (#124 step c); otherwise // they keep opting out of the cache entirely. - if self.prompt_cache_active() + if !self.request_memory_policy_enabled() + && self.prompt_cache_active() && (!is_multimodal || vlm_sharing_ok) && let Some(mut ctx) = options.prompt_cache_ctx.clone() { diff --git a/src/server/batch/scheduler/config.rs b/src/server/batch/scheduler/config.rs index e772cccff..e7bb492ee 100644 --- a/src/server/batch/scheduler/config.rs +++ b/src/server/batch/scheduler/config.rs @@ -106,6 +106,7 @@ impl BatchScheduler { batch_metrics, batch_observability, config_eos, + request_memory_worker_path: true, prefill_chunk_policy: mlxcel_core::prefill_plan::PrefillChunkPolicy::resolve(Some( prefill_chunk_size, )), @@ -114,6 +115,8 @@ impl BatchScheduler { chunked_prefill_seq: None, #[cfg(test)] prefill_forward_lengths: Vec::new(), + #[cfg(test)] + request_memory_test_inputs: None, // A one-wide decode batch has nothing to mix a prefill into, and // that is also exactly b10621's `--no-cont-batching` gate: // upstream adds pending prompts to the batch only when diff --git a/src/server/batch/scheduler/mod.rs b/src/server/batch/scheduler/mod.rs index 2b4294b8b..6c4b0b50d 100644 --- a/src/server/batch/scheduler/mod.rs +++ b/src/server/batch/scheduler/mod.rs @@ -339,6 +339,8 @@ pub struct BatchScheduler { batch_observability: Arc, // -- Configuration -- + /// False for distributed worker paths outside local request admission. + request_memory_worker_path: bool, config_eos: Vec, /// Prompt tokens per prefill chunk with and without live decode streams. prefill_chunk_policy: mlxcel_core::prefill_plan::PrefillChunkPolicy, @@ -357,6 +359,8 @@ pub struct BatchScheduler { /// constructed. #[cfg(test)] pub(super) prefill_forward_lengths: Vec, + #[cfg(test)] + request_memory_test_inputs: Option<(mlxcel_core::memory::MemorySnapshot, Option)>, /// Issue #908 prototype gate, read once from `MLXCEL_MIXED_STEP` at /// construction so the tick loop never touches the environment. False (the @@ -966,6 +970,7 @@ mod prefill; mod prompt_cache; pub(crate) mod prompt_lookup; mod queued_adoption; +mod request_memory; mod run_loop; mod shared_budget; mod speculative_finalize; @@ -1089,3 +1094,6 @@ fn context_shift_trim_depth(live_len: i32, excess: i32, keep: i32, requested_dis }; discard.max(excess).min((n_left - 1).max(excess)) } + +#[cfg(test)] +mod request_memory_tests; diff --git a/src/server/batch/scheduler/prefill.rs b/src/server/batch/scheduler/prefill.rs index a2307ca5a..945370882 100644 --- a/src/server/batch/scheduler/prefill.rs +++ b/src/server/batch/scheduler/prefill.rs @@ -98,6 +98,15 @@ impl BatchScheduler { return; } + // Request memory admission needs the preceding single-slot work to + // finish, even if preemption was configured. Never sample a transient + // active-request baseline or admit alongside a parked decode graph. + if self.request_memory_policy_enabled() && !self.active_batch.is_empty() { + let ids = self.active_batch.sequence_ids(); + self.execute_decode_step(&ids); + return; + } + // Preemption: if batch is full and preemption is enabled, evict // a lower-priority sequence to make room. if self.active_batch.is_full() && self.enable_preemption && !self.try_evict_for_preemption() @@ -142,6 +151,29 @@ impl BatchScheduler { return; } + if seq.cancelled.load(Ordering::Acquire) { + self.prompt_cache_seq_ctx.remove(&seq.seq_id); + self.release_sequence_caches(seq.seq_id); + return; + } + if let Err(rejection) = self.check_request_memory(&seq) { + let _ = seq + .response_tx + .send(GenerateEvent::RequestMemoryRejected(rejection)); + self.prompt_cache_seq_ctx.remove(&seq.seq_id); + self.release_sequence_caches(seq.seq_id); + return; + } + // Provider-held HTTP admission waiters may now commit SSE headers. + // Non-target requests ignore this event and retain their previous path. + if matches!(self.engine.model(), LoadedModel::NemotronH(_)) + && seq.response_tx.send(GenerateEvent::Admitted).is_err() + { + self.prompt_cache_seq_ctx.remove(&seq.seq_id); + self.release_sequence_caches(seq.seq_id); + return; + } + // #122 b2: paged KV block-budget admission gate. Opt-in — a no-op // unless a budget is configured (`free_paged_block_budget()` is `None` // otherwise). When the sequence would not fit, this evicts cold diff --git a/src/server/batch/scheduler/prompt_cache.rs b/src/server/batch/scheduler/prompt_cache.rs index b6ec59d8b..9ea716478 100644 --- a/src/server/batch/scheduler/prompt_cache.rs +++ b/src/server/batch/scheduler/prompt_cache.rs @@ -892,7 +892,8 @@ impl BatchScheduler { tokens: Vec, ctx: PromptCacheRequestContext, ) { - if !self.prompt_cache_active() + if self.request_memory_policy_enabled() + || !self.prompt_cache_active() || !self.engine.model().supports_snapshot_reuse() || crate::server::prompt_cache::cache_warmup_disabled() { @@ -933,6 +934,12 @@ impl BatchScheduler { /// snapshot it already had, and its next turn still hits the #1143 boundary /// entry. A warm-up must never turn into a client-visible error. pub(super) fn run_next_prompt_cache_warmup(&mut self) { + // Guarded requests use cold plans. Neither restore nor speculative + // background forward is permitted outside their final admission gate. + if self.request_memory_policy_enabled() { + self.prompt_cache_warmups.clear(); + return; + } let Some((tokens, ctx)) = self.prompt_cache_warmups.pop_front() else { return; }; diff --git a/src/server/batch/scheduler/request_memory.rs b/src/server/batch/scheduler/request_memory.rs new file mode 100644 index 000000000..b10c59349 --- /dev/null +++ b/src/server/batch/scheduler/request_memory.rs @@ -0,0 +1,137 @@ +// Copyright 2025-2026 Lablup Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; +use crate::execution::request_memory::{self, RequestMemoryRejection}; + +impl BatchScheduler { + /// Used by: the scheduler run loop and admission regression tests. + /// A sequential fallback inside batched prefill does not pass the final + /// single-slot admission gate. Guarded workers must never enter that route, + /// even when --max-batch-prefill allows collecting multiple queued rows. + pub(super) fn execute_prefill_action(&mut self, seq_id: SequenceId) { + if !self.request_memory_policy_enabled() + && self.max_batch_prefill > 1 + && self.prefill_queue.len() >= 2 + && self.chunked_prefill_seq.is_none() + && self.batched_prefill_admits_head() + { + self.execute_batched_prefill(); + } else { + self.execute_prefill(seq_id); + } + } + + /// Keep the HTTP handshake and scheduler guard on the same local-worker scope. + /// + /// Configure this in place rather than adding another by-value builder step: + /// `BatchScheduler` contains `LoadedModel`, and another `Self -> Self` frame + /// overflows the default worker-thread stack in unoptimized Linux builds. + pub(crate) fn set_request_memory_worker_path(&mut self, supported: bool) { + self.request_memory_worker_path = supported; + } + + /// Narrow policy scope; unsupported backends, speculative and other model + /// families keep their current admission behavior. + pub(super) fn request_memory_policy_enabled(&self) -> bool { + let metal = cfg!(all(target_os = "macos", feature = "metal")) + && mlxcel_core::default_device_is_gpu(); + #[cfg(test)] + let metal = metal || self.request_memory_test_inputs.is_some(); + metal + && self.request_memory_worker_path + && self.active_batch.max_size() == 1 + && matches!(self.engine.model(), LoadedModel::NemotronH(_)) + && matches!( + self.speculative_dispatch, + crate::server::SpeculativeDispatch::Disabled + ) + } + + /// Final admission occurs on the scheduler thread, after the preceding + /// single-slot request is finalized. No optimistic prefix credit is taken: + /// snapshots remain resident, but prefix restoration is skipped at enqueue. + pub(super) fn check_request_memory( + &self, + seq: &SequenceInfo, + ) -> Result<(), RequestMemoryRejection> { + if !self.request_memory_policy_enabled() + || !seq.images.is_empty() + || !seq.audio.is_empty() + || seq.vlm_embeddings.is_some() + { + return Ok(()); + } + let LoadedModel::NemotronH(model) = self.engine.model() else { + return Ok(()); + }; + let (config, element_bytes) = model.request_memory_config(); + let plan = self.prefill_plan_for(seq); + let largest = plan + .pieces() + .iter() + .map(|p| p.padded_len as u64) + .max() + .unwrap_or(0); + let stop_context = if self.context_retention.context_shift { + None + } else { + self.max_kv_size.map(|n| n as u64) + }; + let retained = request_memory::retained_tokens( + seq.prompt_tokens.len() as u64, + seq.max_tokens as u64, + stop_context, + ); + let memory = mlxcel_core::memory::snapshot(); + #[cfg(test)] + let memory = self + .request_memory_test_inputs + .map_or(memory, |(memory, _)| memory); + let estimate = retained.and_then(|tokens| { + request_memory::estimate(config, largest, tokens.max(largest), element_bytes, memory) + }); + // Reading the recommendation does not reconstruct the configured wired + // cap: runtime explicitly records the cap it actually applied. + let budget = request_memory::budget_override() + .ok() + .and_then(|override_bytes| { + request_memory::resolve_budget( + mlxcel_core::get_wired_limit() as u64, + crate::execution::runtime::applied_wired_limit(), + memory.limit_bytes, + override_bytes, + ) + }); + #[cfg(test)] + let budget = self + .request_memory_test_inputs + .map_or(budget, |(_, budget)| budget); + let result = request_memory::check(estimate, budget); + tracing::info!( + seq_id = %seq.seq_id, + prompt_tokens = seq.prompt_tokens.len(), + output_tokens = seq.max_tokens, + largest_forward_tokens = largest, + estimated_bytes = estimate.map(|e| e.total_bytes).unwrap_or(u64::MAX), + budget_bytes = budget.unwrap_or(0), + resident_bytes = memory.active_bytes, + allocator_cache_bytes = memory.cache_bytes, + estimate = ?estimate, + admitted = result.is_ok(), + "request memory admission" + ); + result.map(|_| ()) + } +} diff --git a/src/server/batch/scheduler/request_memory_tests.rs b/src/server/batch/scheduler/request_memory_tests.rs new file mode 100644 index 000000000..7bc17e347 --- /dev/null +++ b/src/server/batch/scheduler/request_memory_tests.rs @@ -0,0 +1,193 @@ +// Copyright 2025-2026 Lablup Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use super::*; +use crate::models::nemotron_h::{BlockType, NemotronHConfig, NemotronHModel}; +use mlxcel_core::memory::MemorySnapshot; +use mlxcel_core::weights::WeightMap; + +fn scheduler() -> BatchScheduler { + let mut c: NemotronHConfig = serde_json::from_value(serde_json::json!({ + "model_type":"nemotron_h", "vocab_size":8, "hidden_size":8, + "intermediate_size":16, "num_hidden_layers":1, "num_attention_heads":2, + "num_key_value_heads":2, "mamba_num_heads":2, "mamba_head_dim":4, + "ssm_state_size":4, "conv_kernel":4, "n_groups":1, + "hybrid_override_pattern":"*" + })) + .unwrap(); + c.post_init().unwrap(); + let mut weights = WeightMap::new(); + for (name, shape) in [ + ("backbone.embeddings.weight", vec![8, 8]), + ("backbone.layers.0.norm.weight", vec![8]), + ("backbone.norm_f.weight", vec![8]), + ("lm_head.weight", vec![8, 8]), + ("backbone.layers.0.mixer.q_proj.weight", vec![8, 8]), + ("backbone.layers.0.mixer.k_proj.weight", vec![8, 8]), + ("backbone.layers.0.mixer.v_proj.weight", vec![8, 8]), + ("backbone.layers.0.mixer.o_proj.weight", vec![8, 8]), + ] { + weights.insert( + name.to_string(), + mlxcel_core::full_f32(&shape, 0.1, mlxcel_core::dtype::FLOAT32), + ); + } + let model = NemotronHModel::from_weights(c, weights, vec![BlockType::Attention]).unwrap(); + let (_, rx) = mpsc::channel(); + let mut scheduler = BatchScheduler::with_config( + LoadedModel::NemotronH(model), + MlxcelTokenizer::stub(), + vec![7], + rx, + 1, + 8, + Arc::new(BatchMetrics::new()), + Arc::new(BatchObservability::new()), + 256, + false, + PreemptionPolicy::default(), + 1, + DecodeStorageBackend::Dense, + ) + .with_max_kv_size(Some(128)); + scheduler.request_memory_test_inputs = Some((MemorySnapshot::default(), Some(1))); + install_thread_local_default_stream(scheduler.generation_stream.as_ref()); + scheduler +} + +fn enqueue(scheduler: &mut BatchScheduler, tokens: usize) -> mpsc::Receiver { + let (tx, rx) = mpsc::channel(); + scheduler.enqueue_request( + "prompt".into(), + Some(vec![1; tokens]), + super::scheduler_model_owned_cache_tests::options(), + vec![], + vec![], + vec![], + tx, + Arc::new(AtomicBool::new(false)), + true, + ); + rx +} + +#[test] +fn request_memory_rejection_has_zero_forward_and_next_request_can_run() { + let mut scheduler = scheduler(); + let rejected = enqueue(&mut scheduler, 64); + scheduler.execute_prefill(SequenceId::from_raw(0)); + assert!(matches!( + rejected.recv().unwrap(), + GenerateEvent::RequestMemoryRejected(_) + )); + assert!(scheduler.prefill_forward_lengths.is_empty()); + assert!(scheduler.prefill_queue.is_empty()); + assert!(scheduler.prompt_cache_seq_ctx.is_empty()); + assert!(scheduler.active_batch.is_empty()); + + scheduler.request_memory_test_inputs = Some((MemorySnapshot::default(), Some(1 << 30))); + let accepted = enqueue(&mut scheduler, 4); + scheduler.execute_prefill(SequenceId::from_raw(0)); + assert!(matches!(accepted.recv().unwrap(), GenerateEvent::Admitted)); + assert_eq!(scheduler.prefill_forward_lengths, vec![4]); +} + +#[test] +fn request_memory_context_precedence_and_cancelled_waiter_cleanup() { + let mut scheduler = scheduler(); + let rejected = enqueue(&mut scheduler, 128); + assert!( + matches!(rejected.recv().unwrap(), GenerateEvent::Error(message) if message.contains("context size")) + ); + assert!(scheduler.prefill_forward_lengths.is_empty()); + let (tx, rx) = mpsc::channel(); + scheduler.enqueue_request( + "prompt".into(), + Some(vec![1; 4]), + super::scheduler_model_owned_cache_tests::options(), + vec![], + vec![], + vec![], + tx, + Arc::new(AtomicBool::new(true)), + true, + ); + scheduler.execute_prefill(SequenceId::from_raw(0)); + assert!(rx.recv().is_err()); + assert!(scheduler.prefill_forward_lengths.is_empty()); + assert!(scheduler.prefill_queue.is_empty()); +} + +#[test] +fn request_memory_other_family_is_unchanged() { + let store = super::scheduler_model_owned_cache_tests::test_store(); + let scheduler = super::scheduler_model_owned_cache_tests::scheduler(store); + assert!(!scheduler.request_memory_policy_enabled()); +} + +#[test] +fn request_memory_distributed_worker_path_is_excluded() { + let mut scheduler = scheduler(); + scheduler.set_request_memory_worker_path(false); + assert!(!scheduler.request_memory_policy_enabled()); +} + +#[test] +fn request_memory_two_queued_rows_cannot_bypass_admission_via_batched_prefill() { + let mut scheduler = scheduler(); + scheduler.max_batch_prefill = 4; + let LoadedModel::NemotronH(model) = scheduler.engine.model() else { + panic!("test model"); + }; + let (config, bytes) = model.request_memory_config(); + let short_estimate = + crate::execution::request_memory::estimate(config, 4, 5, bytes, MemorySnapshot::default()) + .unwrap(); + scheduler.request_memory_test_inputs = + Some((MemorySnapshot::default(), Some(short_estimate.total_bytes))); + let long = enqueue(&mut scheduler, 64); + let short = enqueue(&mut scheduler, 4); + assert_eq!(scheduler.prefill_queue.len(), 2); + scheduler.execute_prefill_action(SequenceId::from_raw(0)); + assert!(matches!( + long.recv().unwrap(), + GenerateEvent::RequestMemoryRejected(_) + )); + assert!(scheduler.prefill_forward_lengths.is_empty()); + assert!(matches!(short.try_recv(), Err(mpsc::TryRecvError::Empty))); + scheduler.execute_prefill_action(SequenceId::from_raw(0)); + assert!(matches!(short.recv().unwrap(), GenerateEvent::Admitted)); + assert_eq!(scheduler.prefill_forward_lengths, vec![4]); + assert!(scheduler.prefill_queue.is_empty()); +} + +#[test] +fn request_memory_disables_snapshot_donation_and_background_warmup() { + let store = super::scheduler_model_owned_cache_tests::test_store(); + let mut scheduler = scheduler().with_prompt_cache(Some(store)); + let _events = enqueue(&mut scheduler, 4); + assert!(scheduler.prompt_cache_seq_ctx.is_empty()); + let ctx = super::scheduler_model_owned_cache_tests::cache_ctx(); + scheduler.handle_incoming(ModelRequest::PromptCacheWarmup { + tokens: vec![1; 64], + ctx: ctx.clone(), + }); + assert!(scheduler.prompt_cache_warmups.is_empty()); + // Even a previously queued job cannot reach restore/forward after the + // policy is enabled. Retained store entries are not evicted. + scheduler.prompt_cache_warmups.push_back((vec![1; 64], ctx)); + scheduler.run_next_prompt_cache_warmup(); + assert!(scheduler.prompt_cache_warmups.is_empty()); + assert!(scheduler.prefill_forward_lengths.is_empty()); +} diff --git a/src/server/batch/scheduler/run_loop.rs b/src/server/batch/scheduler/run_loop.rs index 207b4cbe8..0d61f5fdf 100644 --- a/src/server/batch/scheduler/run_loop.rs +++ b/src/server/batch/scheduler/run_loop.rs @@ -288,15 +288,7 @@ impl BatchScheduler { // attention mask chunked to `[chunk, L]` instead of the // unchunked `[L, L]` a single-row batched forward would // build). - if self.max_batch_prefill > 1 - && self.prefill_queue.len() >= 2 - && self.chunked_prefill_seq.is_none() - && self.batched_prefill_admits_head() - { - self.execute_batched_prefill(); - } else { - self.execute_prefill(seq_id); - } + self.execute_prefill_action(seq_id); // A classic action ran: the next tick goes back to the // in-flight speculative slice, if any (issue #734). self.speculative_slice_yielded = false; diff --git a/src/server/batch/scheduler_model_owned_cache_tests.rs b/src/server/batch/scheduler_model_owned_cache_tests.rs index 1d49d67c1..e9b389d04 100644 --- a/src/server/batch/scheduler_model_owned_cache_tests.rs +++ b/src/server/batch/scheduler_model_owned_cache_tests.rs @@ -236,6 +236,7 @@ pub(super) fn options() -> ServerGenerateOptions { grammar: None, image_soft_tokens: None, pre_rendered_prompt_tokens: None, + admission_notification: None, } } @@ -278,6 +279,10 @@ pub(super) fn run_to_completion( Ok(GenerateEvent::Token(_, _)) | Ok(GenerateEvent::TokenWithLogprobs(_, _, _)) | Ok(GenerateEvent::Prefill(_)) => {} + Ok(GenerateEvent::Admitted) => {} + Ok(GenerateEvent::RequestMemoryRejected(error)) => { + panic!("unexpected memory rejection: {error}") + } Ok(GenerateEvent::Done(result)) => return result, Ok(GenerateEvent::Error(err)) => panic!("unexpected generation error: {err}"), Err(err) => panic!("generation did not finish: {err}"), diff --git a/src/server/batch/scheduler_model_owned_lookahead_tests.rs b/src/server/batch/scheduler_model_owned_lookahead_tests.rs index 1d45de57d..e0925033c 100644 --- a/src/server/batch/scheduler_model_owned_lookahead_tests.rs +++ b/src/server/batch/scheduler_model_owned_lookahead_tests.rs @@ -92,6 +92,7 @@ fn options(max_tokens: usize) -> ServerGenerateOptions { grammar: None, image_soft_tokens: None, pre_rendered_prompt_tokens: None, + admission_notification: None, } } diff --git a/src/server/batch/scheduler_model_owned_pad_trim_tests.rs b/src/server/batch/scheduler_model_owned_pad_trim_tests.rs index 28145c02d..41fa79093 100644 --- a/src/server/batch/scheduler_model_owned_pad_trim_tests.rs +++ b/src/server/batch/scheduler_model_owned_pad_trim_tests.rs @@ -201,6 +201,7 @@ fn options(max_tokens: usize) -> ServerGenerateOptions { grammar: None, image_soft_tokens: None, pre_rendered_prompt_tokens: None, + admission_notification: None, } } @@ -267,6 +268,10 @@ fn run(prompt_len: usize, chunk: usize, align: bool, backend: DecodeStorageBacke | Ok(GenerateEvent::TokenWithLogprobs(..)) | Ok(GenerateEvent::Prefill(_)) => {} Ok(GenerateEvent::Done(result)) => break result.generated_token_ids, + Ok(GenerateEvent::Admitted) => {} + Ok(GenerateEvent::RequestMemoryRejected(error)) => { + panic!("unexpected memory rejection: {error}") + } Ok(GenerateEvent::Error(err)) => panic!("unexpected generation error: {err}"), Err(err) => panic!("generation did not finish: {err}"), } @@ -424,6 +429,10 @@ fn padded_prefill_keeps_snapshot_reuse() { loop { match rx.recv_timeout(Duration::from_secs(5)) { Ok(GenerateEvent::Done(_)) => break, + Ok(GenerateEvent::Admitted) => {} + Ok(GenerateEvent::RequestMemoryRejected(error)) => { + panic!("unexpected memory rejection: {error}") + } Ok(GenerateEvent::Error(err)) => panic!("unexpected generation error: {err}"), Ok(_) => {} Err(err) => panic!("generation did not finish: {err}"), diff --git a/src/server/batch/scheduler_muse_glimmer_support.rs b/src/server/batch/scheduler_muse_glimmer_support.rs index 441c02421..ef8455497 100644 --- a/src/server/batch/scheduler_muse_glimmer_support.rs +++ b/src/server/batch/scheduler_muse_glimmer_support.rs @@ -320,6 +320,7 @@ pub(super) fn options(max_tokens: usize) -> ServerGenerateOptions { grammar: None, image_soft_tokens: None, pre_rendered_prompt_tokens: None, + admission_notification: None, } } @@ -371,6 +372,10 @@ pub(super) fn collect_stream(rx: &mpsc::Receiver) -> StreamSummar } Ok(GenerateEvent::Prefill(_)) => {} Ok(GenerateEvent::Done(result)) => return StreamSummary { tokens, result }, + Ok(GenerateEvent::Admitted) => {} + Ok(GenerateEvent::RequestMemoryRejected(error)) => { + panic!("unexpected memory rejection: {error}") + } Ok(GenerateEvent::Error(error)) => panic!("unexpected generation error: {error}"), Err(err) => panic!("generation did not finish: {err}"), } diff --git a/src/server/batch/scheduler_muse_glimmer_tests.rs b/src/server/batch/scheduler_muse_glimmer_tests.rs index 1ced23c77..5b18e7434 100644 --- a/src/server/batch/scheduler_muse_glimmer_tests.rs +++ b/src/server/batch/scheduler_muse_glimmer_tests.rs @@ -312,6 +312,7 @@ fn options(max_tokens: usize) -> ServerGenerateOptions { grammar: None, image_soft_tokens: None, pre_rendered_prompt_tokens: None, + admission_notification: None, } } @@ -357,6 +358,10 @@ fn receive_done(rx: &mpsc::Receiver) -> crate::server::Generation Ok(GenerateEvent::Token(_, _)) | Ok(GenerateEvent::TokenWithLogprobs(_, _, _)) | Ok(GenerateEvent::Prefill(_)) => {} + Ok(GenerateEvent::Admitted) => {} + Ok(GenerateEvent::RequestMemoryRejected(error)) => { + panic!("unexpected memory rejection: {error}") + } Ok(GenerateEvent::Done(result)) => return result, Ok(GenerateEvent::Error(error)) => panic!("unexpected generation error: {error}"), Err(err) => panic!("generation did not finish: {err}"), @@ -373,6 +378,8 @@ fn receive_error(rx: &mpsc::Receiver) -> String { } Ok(GenerateEvent::Done(_)) => panic!("unexpected done event"), Ok(GenerateEvent::Prefill(_)) => panic!("unexpected prefill event"), + Ok(GenerateEvent::Admitted) => panic!("unexpected admission event"), + Ok(GenerateEvent::RequestMemoryRejected(error)) => error.to_string(), Err(err) => panic!("generation did not reject: {err}"), } } diff --git a/src/server/batch/scheduler_prompt_lookup_tests.rs b/src/server/batch/scheduler_prompt_lookup_tests.rs index 176e178dd..683de1daf 100644 --- a/src/server/batch/scheduler_prompt_lookup_tests.rs +++ b/src/server/batch/scheduler_prompt_lookup_tests.rs @@ -199,6 +199,7 @@ fn enqueue_with_sampling( grammar: None, image_soft_tokens: None, pre_rendered_prompt_tokens: None, + admission_notification: None, }; let (tx, rx) = mpsc::channel(); let cancelled = Arc::new(AtomicBool::new(false)); diff --git a/src/server/batch/scheduler_real_batch_parity_tests.rs b/src/server/batch/scheduler_real_batch_parity_tests.rs index 26ec6f580..7bbb79c7e 100644 --- a/src/server/batch/scheduler_real_batch_parity_tests.rs +++ b/src/server/batch/scheduler_real_batch_parity_tests.rs @@ -95,6 +95,7 @@ fn options() -> ServerGenerateOptions { grammar: None, image_soft_tokens: None, pre_rendered_prompt_tokens: None, + admission_notification: None, } } diff --git a/src/server/batch/serving_handoff_parity_tests.rs b/src/server/batch/serving_handoff_parity_tests.rs index 8a8dbb4f6..81b705f81 100644 --- a/src/server/batch/serving_handoff_parity_tests.rs +++ b/src/server/batch/serving_handoff_parity_tests.rs @@ -417,7 +417,10 @@ fn collect_text(rx: &mpsc::Receiver) -> String { GenerateEvent::Token(t, _) | GenerateEvent::TokenWithLogprobs(t, _, _) => { text.push_str(&t) } - GenerateEvent::Done(_) | GenerateEvent::Prefill(_) => {} + GenerateEvent::Done(_) | GenerateEvent::Prefill(_) | GenerateEvent::Admitted => {} + GenerateEvent::RequestMemoryRejected(error) => { + panic!("unexpected memory rejection: {error}") + } GenerateEvent::Error(e) => panic!("unexpected generation error event: {e}"), } } diff --git a/src/server/batch/speculative_slice_tests.rs b/src/server/batch/speculative_slice_tests.rs index 357523589..3f5a4243e 100644 --- a/src/server/batch/speculative_slice_tests.rs +++ b/src/server/batch/speculative_slice_tests.rs @@ -320,6 +320,10 @@ fn drain_events(rx: &mpsc::Receiver) -> Vec<(String, String)> { out.push(match ev { GenerateEvent::Token(t, _) => ("token".to_string(), t), GenerateEvent::TokenWithLogprobs(t, _, _) => ("token_lp".to_string(), t), + GenerateEvent::Admitted => continue, + GenerateEvent::RequestMemoryRejected(error) => { + panic!("unexpected memory rejection: {error}") + } GenerateEvent::Prefill(_) => ("prefill".to_string(), String::new()), GenerateEvent::Done(_) => ("done".to_string(), String::new()), GenerateEvent::Error(e) => panic!("unexpected error event: {e}"), diff --git a/src/server/batch/stop_sequence_tests.rs b/src/server/batch/stop_sequence_tests.rs index 6276d067f..236abdd6d 100644 --- a/src/server/batch/stop_sequence_tests.rs +++ b/src/server/batch/stop_sequence_tests.rs @@ -157,6 +157,10 @@ fn drive(text: &str, stops: &[&str], max_tokens: usize, chunk_bytes: usize) -> R GenerateEvent::Token(t, _) | GenerateEvent::TokenWithLogprobs(t, _, _) => { streamed.push_str(&t); } + GenerateEvent::Admitted => {} + GenerateEvent::RequestMemoryRejected(error) => { + panic!("unexpected memory rejection: {error}") + } GenerateEvent::Done(_) | GenerateEvent::Error(_) | GenerateEvent::Prefill(_) => {} } } diff --git a/src/server/batch/xla_worker_tests.rs b/src/server/batch/xla_worker_tests.rs index 5099d3329..3a493185e 100644 --- a/src/server/batch/xla_worker_tests.rs +++ b/src/server/batch/xla_worker_tests.rs @@ -344,10 +344,39 @@ fn receive_done(rx: &mpsc::Receiver) -> GenerationResult { | GenerateEvent::TokenWithLogprobs(_, _, _) | GenerateEvent::Prefill(_) => {} GenerateEvent::Error(error) => panic!("unexpected generation failure: {error}"), + GenerateEvent::Admitted => { + panic!("unexpected Metal memory admission on the XLA worker") + } + GenerateEvent::RequestMemoryRejected(error) => { + panic!("unexpected Metal memory rejection on the XLA worker: {error}") + } } } } +#[test] +#[should_panic(expected = "unexpected Metal memory admission on the XLA worker")] +fn receive_done_rejects_metal_memory_admission_events() { + let (sender, receiver) = mpsc::channel(); + sender.send(GenerateEvent::Admitted).unwrap(); + receive_done(&receiver); +} + +#[test] +#[should_panic(expected = "unexpected Metal memory rejection on the XLA worker")] +fn receive_done_rejects_metal_memory_rejection_events() { + let (sender, receiver) = mpsc::channel(); + sender + .send(GenerateEvent::RequestMemoryRejected( + crate::execution::request_memory::RequestMemoryRejection { + estimated_bytes: 2048, + budget_bytes: 1024, + }, + )) + .unwrap(); + receive_done(&receiver); +} + #[test] fn mixed_text_and_image_admission_keeps_public_and_effective_lengths_distinct() { let mut worker = worker(Some(image_stage())); @@ -711,6 +740,10 @@ fn stop_finish_without_a_token_event_returns_empty_content_and_no_completion_tok panic!("unexpected prefill snapshot before the result: {stats:?}") } GenerateEvent::Error(error) => panic!("unexpected generation failure: {error}"), + GenerateEvent::Admitted => panic!("unexpected Metal memory admission on the XLA worker"), + GenerateEvent::RequestMemoryRejected(error) => { + panic!("unexpected Metal memory rejection on the XLA worker: {error}") + } }; assert_eq!(result.text, ""); assert_eq!(result.completion_tokens, 0); diff --git a/src/server/config.rs b/src/server/config.rs index 2ae5927f8..2fe36d1ef 100644 --- a/src/server/config.rs +++ b/src/server/config.rs @@ -331,6 +331,8 @@ pub struct ServerGenerateOptions { /// `None` on every request that rendered through a chat template, which is /// every request that predates #1338. pub pre_rendered_prompt_tokens: Option>, + /// HTTP-only admission handshake, removed before dispatch to the scheduler. + pub admission_notification: Option, } /// Per-request context-retention overrides (b10621 `n_keep` / `n_discard`, diff --git a/src/server/engine_probe/server_engine.rs b/src/server/engine_probe/server_engine.rs index f2c4657e2..b8ee300cc 100644 --- a/src/server/engine_probe/server_engine.rs +++ b/src/server/engine_probe/server_engine.rs @@ -296,6 +296,7 @@ impl ServerEngine { grammar: None, image_soft_tokens: None, pre_rendered_prompt_tokens: Some(request.prompt_tokens.to_vec()), + admission_notification: None, }; let live = self.server.config().live_settings(); let before = self.server.observability().snapshot(); diff --git a/src/server/model_provider.rs b/src/server/model_provider.rs index 6f2b7034f..72ae5d749 100644 --- a/src/server/model_provider.rs +++ b/src/server/model_provider.rs @@ -176,8 +176,181 @@ fn uses_single_stream_queue_admission(model_path: &std::path::Path) -> bool { }) } +/// Whether the configured worker uses the local non-speculative scheduler +/// path covered by request-memory admission. Check runtime/model eligibility +/// separately: a checkpoint architecture does not identify its serving engine. +// Used by: NemotronH provider admission and model-worker scheduler policy setup. +pub(crate) fn request_memory_worker_path_supported( + has_pipeline: bool, + tp_size: usize, + serving_mode: crate::distributed::disaggregated::ServingMode, + has_speculation: bool, +) -> bool { + use crate::distributed::disaggregated::ServingMode; + !has_pipeline + && tp_size == 1 + && matches!(serving_mode, ServingMode::Hybrid | ServingMode::Router) + && !has_speculation +} + +// Used by: NemotronH scheduler-backed provider constructors; pure device input +// keeps CPU/non-Metal bypass tests independent of the process-global MLX device. +fn request_memory_provider_wait_enabled( + worker_path_supported: bool, + metal_gpu: bool, + model_type: Option, +) -> bool { + worker_path_supported + && metal_gpu + && matches!(model_type, Some(crate::models::ModelType::NemotronH)) +} + +// Used by: NemotronH legacy sequential provider. The legacy worker still uses +// BatchScheduler at width one; it forwards TP but not PP, serving roles, or +// speculation. Match its actual execution, not those ignored request settings. +fn legacy_request_memory_admission_enabled( + tp_size: usize, + metal_gpu: bool, + model_type: Option, +) -> bool { + request_memory_provider_wait_enabled( + request_memory_worker_path_supported( + false, + tp_size, + crate::distributed::disaggregated::ServingMode::Hybrid, + false, + ), + metal_gpu, + model_type, + ) +} + +// Used by: NemotronH scheduler-backed provider constructors. +fn uses_request_memory_admission( + model_path: &std::path::Path, + config: &model_worker::WorkerSchedulerConfig, +) -> bool { + request_memory_provider_wait_enabled( + request_memory_worker_path_supported( + config.pipeline_parallel_runtime.is_some(), + config.tensor_parallel.tp_size, + config.serving_mode, + !matches!( + config.speculative_dispatch, + crate::server::SpeculativeDispatch::Disabled + ), + ), + cfg!(all(target_os = "macos", feature = "metal")) && mlxcel_core::default_device_is_gpu(), + crate::models::get_model_type(model_path).ok(), + ) +} + +/// One-shot HTTP admission signal. Clones share a single sender, not multiple +/// acknowledgements. Providers remove this from generation options before enqueue. +#[derive(Debug, Clone)] +pub struct AdmissionNotification(Arc>>>>); + +impl AdmissionNotification { + fn complete(self, result: &Result) { + let result = match result { + Ok(_) => Ok(()), + Err(error) => { + if let Some(rejection) = + error.downcast_ref::() + { + Err(anyhow::Error::new(*rejection)) + } else if let Some(queue) = error.downcast_ref::() { + Err(anyhow::Error::new(QueueFullError { + max_queue_depth: queue.max_queue_depth, + })) + } else if error.downcast_ref::().is_some() { + Err(anyhow::Error::new(ChatWorkerGoneError)) + } else { + Err(anyhow::anyhow!(error.to_string())) + } + } + }; + if let Ok(mut sender) = self.0.lock() + && let Some(sender) = sender.take() + { + let _ = sender.send(result); + } + } +} + +/// Dropping an HTTP handler before admission must cancel its queued work even +/// though its blocking generation task cannot be aborted by Tokio. +pub(crate) struct AdmissionWaiter { + receiver: Option>>, + cancelled: Arc, + admitted: bool, +} + +impl AdmissionWaiter { + // Used by: NemotronH (guarded), all other serving models (immediate bypass). + pub(crate) fn arm(options: &mut ServerGenerateOptions, cancelled: Arc) -> Self { + let (sender, receiver) = tokio::sync::oneshot::channel(); + options.admission_notification = + Some(AdmissionNotification(Arc::new(Mutex::new(Some(sender))))); + Self { + receiver: Some(receiver), + cancelled, + admitted: false, + } + } + + pub(crate) async fn wait(mut self) -> Result<()> { + let result = match self.receiver.take() { + Some(receiver) => receiver + .await + .unwrap_or_else(|_| Err(anyhow::Error::new(ChatWorkerGoneError))), + None => Err(anyhow::Error::new(ChatWorkerGoneError)), + }; + self.admitted = result.is_ok(); + result + } +} + +impl Drop for AdmissionWaiter { + fn drop(&mut self) { + if !self.admitted { + self.cancelled.store(true, Ordering::Release); + } + } +} + +// Used by: NemotronH scheduler-backed text serving, independent of HTTP dialect. +fn await_request_admission( + receiver: &mpsc::Receiver, + cancelled: &AtomicBool, +) -> Result<()> { + loop { + if cancelled.load(Ordering::Acquire) { + anyhow::bail!("request cancelled before memory admission"); + } + match receiver.recv_timeout(Duration::from_millis(100)) { + Ok(GenerateEvent::Admitted) => return Ok(()), + Ok(GenerateEvent::RequestMemoryRejected(error)) => { + return Err(anyhow::Error::new(error)); + } + Ok(GenerateEvent::Error(error)) => return Err(anyhow::anyhow!(error)), + Ok(_) => { + anyhow::bail!("model worker emitted generation output before memory admission") + } + Err(mpsc::RecvTimeoutError::Timeout) => continue, + Err(mpsc::RecvTimeoutError::Disconnected) => { + return Err(anyhow::Error::new(ChatWorkerGoneError)); + } + } + } +} + /// Events from generation pub enum GenerateEvent { + /// Admission completed before request cache or GPU work. + Admitted, + /// Predictable memory rejection, preserving its typed HTTP discriminator. + RequestMemoryRejected(crate::execution::request_memory::RequestMemoryRejection), Token(String, TokenMeta), /// Token with associated log probability data (emitted when logprobs are enabled) TokenWithLogprobs(String, TokenMeta, TokenLogprobData), @@ -429,6 +602,7 @@ pub struct ModelProvider { batch_observability: Arc, max_queue_depth: usize, single_stream_queue_admission: Arc, + request_memory_admission: bool, /// b10621 `--sleep-idle-seconds` state (#1440): `true` while the serving /// worker has freed the model and is parked on the request channel. Only /// the batch worker ever sets it; every other worker leaves it `false`, @@ -583,8 +757,8 @@ impl ModelProvider { /// Same as [`Self::new_with_server_config`] but also wires the /// cross-request prompt-prefix KV cache store. - /// The legacy (`config.no_batch`) worker ignores the store because that - /// path never calls the batch scheduler that manages the cache. + /// The legacy (`config.no_batch`) worker ignores the store because its + /// size-one scheduler does not attach cross-request prompt caching. pub fn new_with_server_config_and_prompt_cache( model_path: PathBuf, adapter_path: Option, @@ -719,8 +893,8 @@ impl ModelProvider { provider.decode_hang_timeout = decode_hang_timeout; // log a warning if the operator asked for // speculative decoding but selected `--no-batch`. The legacy - // sequential worker bypasses the BatchScheduler entirely so - // the dispatch is inactive on this path. + // sequential worker does not attach speculative dispatch to its + // size-one scheduler, so dispatch is inactive on this path. if !matches!( speculative_dispatch, crate::server::SpeculativeDispatch::Disabled @@ -847,6 +1021,7 @@ impl ModelProvider { batch_observability, max_queue_depth: 1, single_stream_queue_admission: Arc::new(AtomicBool::new(false)), + request_memory_admission: false, // This path has no idle-sleep lifecycle, so the flag is truthfully // never set (#1440). sleeping: Arc::new(AtomicBool::new(false)), @@ -895,6 +1070,12 @@ impl ModelProvider { let metrics_clone = batch_metrics.clone(); let obs_clone = batch_observability.clone(); + let request_memory_admission = legacy_request_memory_admission_enabled( + tensor_parallel.tp_size, + cfg!(all(target_os = "macos", feature = "metal")) + && mlxcel_core::default_device_is_gpu(), + crate::models::get_model_type(&model_path).ok(), + ); let worker_handle = model_worker::spawn_legacy_model_worker( model_path, adapter_path, @@ -922,6 +1103,7 @@ impl ModelProvider { chat_unavailable, max_queue_depth, single_stream_queue_admission, + request_memory_admission, sleeping, prompt_cache: None, prompt_tokenizer: None, @@ -982,6 +1164,7 @@ impl ModelProvider { chat_unavailable: Arc::new(AtomicBool::new(false)), max_queue_depth: usize::MAX, single_stream_queue_admission, + request_memory_admission: false, // The OpenXLA serve worker has no idle-sleep lifecycle, so the flag // is truthfully never set on this path (#1440). sleeping: Arc::new(AtomicBool::new(false)), @@ -1317,6 +1500,7 @@ impl ModelProvider { diffusion_threshold, }; + let request_memory_admission = uses_request_memory_admission(&model_path, &sched_config); let worker_handle = model_worker::spawn_model_worker_with_batch_config( model_path, adapter_path, @@ -1344,6 +1528,7 @@ impl ModelProvider { max_queue_depth, chat_unavailable, single_stream_queue_admission, + request_memory_admission, sleeping, prompt_cache: prompt_cache_store, prompt_tokenizer: None, @@ -1438,6 +1623,7 @@ impl ModelProvider { diffusion_threshold: 0.9, }; + let request_memory_admission = uses_request_memory_admission(&model_path, &sched_config); let worker_handle = model_worker::spawn_model_worker_with_batch_config( model_path, adapter_path, @@ -1465,6 +1651,7 @@ impl ModelProvider { max_queue_depth, chat_unavailable, single_stream_queue_admission, + request_memory_admission, sleeping, prompt_cache: None, prompt_tokenizer: None, @@ -2266,6 +2453,7 @@ impl ModelProvider { ) } + // Used by: all serving models via every generation HTTP dialect and in-process callers. #[allow(clippy::too_many_arguments)] fn send_generate_request_with_cancellation_and_metadata( &self, @@ -2280,6 +2468,53 @@ impl ModelProvider { queue_reservation_mode: QueueReservationMode, ) -> Result> { let mut options = options; + let mut notification = options.admission_notification.take(); + // Unsupported paths keep their existing streaming semantics: this + // handshake must not wait for their worker's first event or reclassify + // asynchronous errors that previously followed SSE headers. + if (!self.request_memory_admission + || !images.is_empty() + || !audio.is_empty() + || !videos.is_empty()) + && let Some(notification) = notification.take() + { + notification.complete(&Ok(())); + } + let result = self.send_generate_request_with_cancellation_and_metadata_inner( + prompt, + options, + images, + audio, + videos, + media, + cancelled, + runtime, + queue_reservation_mode, + ); + if let Some(notification) = notification { + notification.complete(&result); + } + result + } + + #[allow(clippy::too_many_arguments)] + fn send_generate_request_with_cancellation_and_metadata_inner( + &self, + prompt: String, + options: ServerGenerateOptions, + images: Vec>, + audio: Vec>, + videos: Vec, + media: MediaRequestMetadata, + cancelled: Arc, + runtime: Option, + queue_reservation_mode: QueueReservationMode, + ) -> Result> { + let mut options = options; + let wait_for_admission = self.request_memory_admission + && images.is_empty() + && audio.is_empty() + && videos.is_empty(); let (response_tx, response_rx) = mpsc::channel(); media .validate_resolved_image_count() @@ -2362,13 +2597,22 @@ impl ModelProvider { media, queue_reservation, response_tx, - cancelled, + cancelled: cancelled.clone(), }) .is_err() { return Err(anyhow::Error::new(ChatWorkerGoneError)); } + let admission = if wait_for_admission { + await_request_admission(&response_rx, &cancelled) + } else { + Ok(()) + }; + if admission.is_err() { + cancelled.store(true, Ordering::Release); + } + admission?; Ok(response_rx) } } @@ -2567,6 +2811,8 @@ where GenerateEvent::Prefill(_) => Ok(None), GenerateEvent::Done(result) => Ok(Some(result)), GenerateEvent::Error(err) => Err(anyhow::anyhow!(err)), + GenerateEvent::Admitted => Ok(None), + GenerateEvent::RequestMemoryRejected(err) => Err(anyhow::Error::new(err)), })?; if !accumulated_logprobs.is_empty() { @@ -2611,6 +2857,8 @@ where } GenerateEvent::Done(result) => Ok(Some(result)), GenerateEvent::Error(err) => Err(anyhow::anyhow!(err)), + GenerateEvent::Admitted => Ok(None), + GenerateEvent::RequestMemoryRejected(err) => Err(anyhow::Error::new(err)), })?; if !accumulated_logprobs.is_empty() { @@ -2648,6 +2896,8 @@ where GenerateEvent::Prefill(_) => Ok(None), GenerateEvent::Done(result) => Ok(Some(result)), GenerateEvent::Error(err) => Err(anyhow::anyhow!(err)), + GenerateEvent::Admitted => Ok(None), + GenerateEvent::RequestMemoryRejected(err) => Err(anyhow::Error::new(err)), }) } @@ -2682,6 +2932,8 @@ where } GenerateEvent::Done(result) => Ok(Some(result)), GenerateEvent::Error(err) => Err(anyhow::anyhow!(err)), + GenerateEvent::Admitted => Ok(None), + GenerateEvent::RequestMemoryRejected(err) => Err(anyhow::Error::new(err)), }) } @@ -2721,6 +2973,8 @@ where } GenerateEvent::Done(result) => Ok(Some(result)), GenerateEvent::Error(err) => Err(anyhow::anyhow!(err)), + GenerateEvent::Admitted => Ok(None), + GenerateEvent::RequestMemoryRejected(err) => Err(anyhow::Error::new(err)), }) } @@ -2796,7 +3050,9 @@ where // must NOT open the bounded decode window: doing so would start // the hang timer while the first token is still being sampled. GenerateEvent::Prefill(_) => {} - GenerateEvent::Error(_) => {} + GenerateEvent::Error(_) + | GenerateEvent::Admitted + | GenerateEvent::RequestMemoryRejected(_) => {} } if let Some(result) = handler(event)? { diff --git a/src/server/model_provider_test_support.rs b/src/server/model_provider_test_support.rs index 4a5693caf..81c91fa56 100644 --- a/src/server/model_provider_test_support.rs +++ b/src/server/model_provider_test_support.rs @@ -31,6 +31,7 @@ impl ModelProvider { single_stream_admission, max_queue_depth, None, + None, ) } @@ -41,7 +42,28 @@ impl ModelProvider { options_tx: mpsc::Sender, speculative: SpeculativeStats, ) -> Self { - Self::recording_for_route_tests_inner(options_tx, false, usize::MAX, Some(speculative)) + Self::recording_for_route_tests_inner( + options_tx, + false, + usize::MAX, + Some(speculative), + None, + ) + } + + pub(crate) fn memory_admission_for_route_tests( + options_tx: mpsc::Sender, + ) -> Self { + Self::recording_for_route_tests_inner( + options_tx, + true, + 1, + None, + Some(crate::execution::request_memory::RequestMemoryRejection { + estimated_bytes: 2048, + budget_bytes: 1024, + }), + ) } fn recording_for_route_tests_inner( @@ -49,6 +71,7 @@ impl ModelProvider { single_stream_admission: bool, max_queue_depth: usize, speculative: Option, + memory_rejection: Option, ) -> Self { let (request_tx, request_rx) = mpsc::channel::(); let loaded = Arc::new(AtomicBool::new(true)); @@ -64,6 +87,15 @@ impl ModelProvider { .. } => { drop(queue_reservation); + if let Some(rejection) = memory_rejection { + if options.max_tokens >= 64 { + let _ = options_tx.send(options); + let _ = response_tx + .send(GenerateEvent::RequestMemoryRejected(rejection)); + continue; + } + let _ = response_tx.send(GenerateEvent::Admitted); + } // A request carrying string stop sequences is answered // as a generation that one of them ended (issue #1466), // so route tests can assert the b10621 @@ -74,9 +106,7 @@ impl ModelProvider { .stop_sequences .as_deref() .and_then(<[String]>::first) - .map(|word| { - crate::server::model_provider::StopKind::Word(word.clone()) - }) + .map(|word| crate::server::model_provider::StopKind::Word(word.clone())) .unwrap_or(crate::server::model_provider::StopKind::Eos); let _ = options_tx.send(options); let _ = response_tx.send(GenerateEvent::Done(GenerationResult { @@ -115,6 +145,7 @@ impl ModelProvider { batch_metrics, batch_observability, max_queue_depth, + request_memory_admission: memory_rejection.is_some(), single_stream_queue_admission: Arc::new(AtomicBool::new(single_stream_admission)), sleeping: Arc::new(AtomicBool::new(false)), prompt_cache: None, @@ -148,6 +179,7 @@ impl ModelProvider { batch_metrics, batch_observability: Arc::new(BatchObservability::new()), max_queue_depth: usize::MAX, + request_memory_admission: false, single_stream_queue_admission: Arc::new(AtomicBool::new(false)), sleeping: Arc::new(AtomicBool::new(false)), prompt_cache: None, @@ -239,7 +271,8 @@ impl ModelProvider { } text.push_str(&token); tokens_emitted += 1; - let _ = response_tx.send(GenerateEvent::Token(token, TokenMeta::default())); + let _ = response_tx + .send(GenerateEvent::Token(token, TokenMeta::default())); } ScriptedStreamStep::Finish => break, } @@ -279,6 +312,7 @@ impl ModelProvider { batch_metrics, batch_observability, max_queue_depth: usize::MAX, + request_memory_admission: false, single_stream_queue_admission: Arc::new(AtomicBool::new(false)), sleeping: Arc::new(AtomicBool::new(false)), prompt_cache: None, @@ -295,7 +329,6 @@ impl ModelProvider { }, ) } - } /// One step of a scripted streaming generation (#1444). @@ -317,7 +350,9 @@ pub(crate) struct ScriptedStreamHandle { impl ScriptedStreamHandle { pub(crate) fn token(&self, token: &str) { - let _ = self.step_tx.send(ScriptedStreamStep::Token(token.to_string())); + let _ = self + .step_tx + .send(ScriptedStreamStep::Token(token.to_string())); } pub(crate) fn finish(&self) { diff --git a/src/server/model_provider_tests.rs b/src/server/model_provider_tests.rs index a9633236c..95bdd43a8 100644 --- a/src/server/model_provider_tests.rs +++ b/src/server/model_provider_tests.rs @@ -13,7 +13,7 @@ // limitations under the License. use std::sync::Arc; -use std::sync::atomic::AtomicBool; +use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::mpsc; use std::time::Duration; @@ -21,10 +21,12 @@ use mlxcel_core::generate::SamplingConfig; use mlxcel_core::sampling::{LogprobsConfig, TokenLogprobData}; use super::{ - ChatWorkerGoneError, DECODE_HANG_TIMEOUT, GenerateEvent, GenerationResult, ModelProvider, - ModelRequest, PrefillStats, QueueReservationMode, RequestRuntimeDefaults, - SingleStreamQueueReservation, StopKind, TokenMeta, drain_generation_events, - drain_generation_events_with_logprobs_observing_prefill, observe_worker_exit, + AdmissionWaiter, ChatWorkerGoneError, DECODE_HANG_TIMEOUT, GenerateEvent, GenerationResult, + ModelProvider, ModelRequest, PrefillStats, QueueReservationMode, RequestRuntimeDefaults, + SingleStreamQueueReservation, StopKind, TokenMeta, await_request_admission, + drain_generation_events, drain_generation_events_with_logprobs_observing_prefill, + legacy_request_memory_admission_enabled, observe_worker_exit, + request_memory_provider_wait_enabled, request_memory_worker_path_supported, send_shutdown_signal, tokenize_prompt_for_generation, tokenize_prompt_for_generation_with_ordered_media, validated_decode_hang_timeout, }; @@ -79,6 +81,7 @@ fn sample_options() -> crate::server::ServerGenerateOptions { grammar: None, image_soft_tokens: None, pre_rendered_prompt_tokens: None, + admission_notification: None, } } @@ -237,6 +240,7 @@ fn generate_with_live_enqueues_the_captured_runtime_defaults() { batch_metrics: Arc::new(BatchMetrics::new()), batch_observability: Arc::new(BatchObservability::new()), max_queue_depth: usize::MAX, + request_memory_admission: false, single_stream_queue_admission: Arc::new(AtomicBool::new(false)), sleeping: Arc::new(AtomicBool::new(false)), prompt_cache: None, @@ -384,6 +388,7 @@ fn pre_reserved_single_stream_enqueue_does_not_double_reserve() { batch_metrics: metrics.clone(), batch_observability: Arc::new(BatchObservability::new()), max_queue_depth: 1, + request_memory_admission: false, single_stream_queue_admission: Arc::new(AtomicBool::new(true)), sleeping: Arc::new(AtomicBool::new(false)), prompt_cache: None, @@ -435,6 +440,7 @@ fn scheduler_paths_do_not_create_single_stream_reservations() { batch_metrics: Arc::new(BatchMetrics::new()), batch_observability: Arc::new(BatchObservability::new()), max_queue_depth: 0, + request_memory_admission: false, single_stream_queue_admission: Arc::new(AtomicBool::new(false)), sleeping: Arc::new(AtomicBool::new(false)), prompt_cache: None, @@ -609,6 +615,8 @@ fn drain_generation_events_impl_survives_long_prefill_before_first_token() { Ok(Some(r)) } GenerateEvent::Error(e) => Err(anyhow::anyhow!(e)), + GenerateEvent::Admitted => Ok(None), + GenerateEvent::RequestMemoryRejected(e) => Err(anyhow::Error::new(e)), }) .expect("drain_generation_events_impl must not error during long prefill"); @@ -675,6 +683,8 @@ fn drain_generation_events_impl_detects_phase2_decode_hang() { GenerateEvent::Prefill(_) => Ok(None), GenerateEvent::Done(r) => Ok(Some(r)), GenerateEvent::Error(e) => Err(anyhow::anyhow!(e)), + GenerateEvent::Admitted => Ok(None), + GenerateEvent::RequestMemoryRejected(e) => Err(anyhow::Error::new(e)), }); worker.join().expect("worker thread must not panic"); @@ -694,3 +704,235 @@ fn drain_generation_events_impl_detects_phase2_decode_hang() { "error message must describe the decode hang; got: {msg}" ); } + +#[test] +fn request_memory_admission_preserves_typed_rejection_and_context_errors() { + use crate::execution::request_memory::RequestMemoryRejection; + let (sender, receiver) = mpsc::channel(); + sender + .send(GenerateEvent::RequestMemoryRejected( + RequestMemoryRejection { + estimated_bytes: 11, + budget_bytes: 10, + }, + )) + .unwrap(); + let cancelled = AtomicBool::new(false); + let error = await_request_admission(&receiver, &cancelled).unwrap_err(); + assert_eq!( + error + .downcast_ref::() + .unwrap() + .estimated_bytes, + 11 + ); + sender + .send(GenerateEvent::Error( + "prompt exceeds the available context size".into(), + )) + .unwrap(); + let error = await_request_admission(&receiver, &cancelled).unwrap_err(); + assert!( + error + .to_string() + .contains("exceeds the available context size") + ); + sender.send(GenerateEvent::Admitted).unwrap(); + assert!(await_request_admission(&receiver, &cancelled).is_ok()); +} + +#[tokio::test] +async fn request_memory_admission_http_abandonment_cancels_pending_worker() { + let mut options = sample_options(); + let cancelled = Arc::new(AtomicBool::new(false)); + let waiter = AdmissionWaiter::arm(&mut options, cancelled.clone()); + let (sender, receiver) = mpsc::channel(); + let worker_cancelled = cancelled.clone(); + let worker = + tokio::task::spawn_blocking(move || await_request_admission(&receiver, &worker_cancelled)); + drop(waiter); + assert!(cancelled.load(Ordering::Acquire)); + let error = tokio::time::timeout(Duration::from_secs(2), worker) + .await + .unwrap() + .unwrap() + .unwrap_err(); + assert!(error.to_string().contains("cancelled")); + drop(sender); +} + +#[tokio::test] +async fn request_memory_admission_sender_exit_releases_http_waiter() { + let mut options = sample_options(); + let cancelled = Arc::new(AtomicBool::new(false)); + let waiter = AdmissionWaiter::arm(&mut options, cancelled.clone()); + drop(options); + let error = tokio::time::timeout(Duration::from_secs(2), waiter.wait()) + .await + .unwrap() + .unwrap_err(); + assert!(error.downcast_ref::().is_some()); + assert!(cancelled.load(Ordering::Acquire)); +} + +#[tokio::test] +async fn request_memory_admission_ack_does_not_cancel_generation() { + let mut options = sample_options(); + let cancelled = Arc::new(AtomicBool::new(false)); + let waiter = AdmissionWaiter::arm(&mut options, cancelled.clone()); + options + .admission_notification + .take() + .unwrap() + .complete(&Ok(())); + waiter.wait().await.unwrap(); + assert!(!cancelled.load(Ordering::Acquire)); +} + +#[test] +fn request_memory_admission_excludes_nonlocal_and_speculative_workers() { + use crate::distributed::disaggregated::ServingMode; + assert!(request_memory_worker_path_supported( + false, + 1, + ServingMode::Hybrid, + false + )); + // Router currently uses the same local worker loop as Hybrid. + assert!(request_memory_worker_path_supported( + false, + 1, + ServingMode::Router, + false + )); + for (pipeline, tp_size, mode, speculation) in [ + (true, 1, ServingMode::Hybrid, false), + (false, 2, ServingMode::Hybrid, false), + (false, 0, ServingMode::Hybrid, false), + (false, 1, ServingMode::PrefillOnly, false), + (false, 1, ServingMode::DecodeOnly, false), + (false, 1, ServingMode::Hybrid, true), + ] { + assert!(!request_memory_worker_path_supported( + pipeline, + tp_size, + mode, + speculation + )); + } +} + +#[tokio::test] +async fn request_memory_admission_bypassed_worker_needs_no_admitted_event() { + use crate::distributed::disaggregated::ServingMode; + for (pipeline, tp_size, mode, speculation, metal_gpu) in [ + (true, 1, ServingMode::Hybrid, false, true), + (false, 2, ServingMode::Hybrid, false, true), + (false, 1, ServingMode::PrefillOnly, false, true), + (false, 1, ServingMode::DecodeOnly, false, true), + (false, 1, ServingMode::Hybrid, true, true), + (false, 1, ServingMode::Hybrid, false, false), + ] { + let (options_tx, _options_rx) = mpsc::channel(); + // This worker emits Done only. Accidentally waiting for Admitted would + // turn its ordinary result into a protocol error instead of success. + let mut provider = + ModelProvider::recording_for_route_tests_with_admission(options_tx, true, 1); + provider.request_memory_admission = request_memory_provider_wait_enabled( + request_memory_worker_path_supported(pipeline, tp_size, mode, speculation), + metal_gpu, + Some(crate::models::ModelType::NemotronH), + ); + let mut options = sample_options(); + let cancelled = Arc::new(AtomicBool::new(false)); + let waiter = AdmissionWaiter::arm(&mut options, cancelled.clone()); + provider + .generate("hello".into(), options) + .expect("unsupported worker must preserve ordinary generation"); + tokio::time::timeout(Duration::from_secs(1), waiter.wait()) + .await + .expect("bypass must acknowledge immediately") + .unwrap(); + assert!(!cancelled.load(Ordering::Acquire)); + assert_eq!(provider.batch_metrics.queue_depth(), 0); + } +} + +#[test] +fn request_memory_admission_requires_metal_gpu_and_actual_target_family() { + use crate::models::ModelType; + assert!(request_memory_provider_wait_enabled( + true, + true, + Some(ModelType::NemotronH) + )); + assert!(!request_memory_provider_wait_enabled( + true, + false, + Some(ModelType::NemotronH) + )); + assert!(!request_memory_provider_wait_enabled( + false, + true, + Some(ModelType::NemotronH) + )); + assert!(!request_memory_provider_wait_enabled( + true, + true, + Some(ModelType::Llama) + )); + assert!(!request_memory_provider_wait_enabled(true, true, None)); +} + +#[test] +fn request_memory_admission_legacy_sequential_scheduler_has_matching_scope() { + use crate::models::ModelType; + assert!(legacy_request_memory_admission_enabled( + 1, + true, + Some(ModelType::NemotronH) + )); + for (tp_size, metal_gpu, model_type) in [ + (2, true, Some(ModelType::NemotronH)), + (0, true, Some(ModelType::NemotronH)), + (1, false, Some(ModelType::NemotronH)), + (1, true, Some(ModelType::Llama)), + (1, true, None), + ] { + assert!(!legacy_request_memory_admission_enabled( + tp_size, metal_gpu, model_type + )); + } +} + +#[tokio::test] +async fn request_memory_admission_legacy_provider_preserves_preheader_rejection() { + let (options_tx, _options_rx) = mpsc::channel(); + let mut provider = ModelProvider::memory_admission_for_route_tests(options_tx); + provider.request_memory_admission = + legacy_request_memory_admission_enabled(1, true, Some(crate::models::ModelType::NemotronH)); + let mut options = sample_options(); + options.max_tokens = 64; + let cancelled = Arc::new(AtomicBool::new(false)); + let waiter = AdmissionWaiter::arm(&mut options, cancelled.clone()); + let error = provider.generate("oversized".into(), options).unwrap_err(); + assert!( + error + .downcast_ref::() + .is_some() + ); + let error = tokio::time::timeout(Duration::from_secs(1), waiter.wait()) + .await + .unwrap() + .unwrap_err(); + assert!( + error + .downcast_ref::() + .is_some() + ); + assert!(cancelled.load(Ordering::Acquire)); + assert_eq!(provider.batch_metrics.queue_depth(), 0); + let mut options = sample_options(); + options.max_tokens = 1; + assert!(provider.generate("short".into(), options).is_ok()); +} diff --git a/src/server/model_worker.rs b/src/server/model_worker.rs index d2af44e34..aa80334a0 100644 --- a/src/server/model_worker.rs +++ b/src/server/model_worker.rs @@ -786,6 +786,17 @@ pub(crate) fn spawn_model_worker_with_batch_config( .file_name() .map(|name| name.to_string_lossy().into_owned()), ); + scheduler.set_request_memory_worker_path( + crate::server::model_provider::request_memory_worker_path_supported( + sched_config.pipeline_parallel_runtime.is_some(), + sched_config.tensor_parallel.tp_size, + sched_config.serving_mode, + !matches!( + sched_config.speculative_dispatch, + crate::server::SpeculativeDispatch::Disabled + ), + ), + ); // The request that woke this worker from idle sleep is prefilled by // the scheduler it just rebuilt, not answered separately (#1440): @@ -1338,6 +1349,14 @@ pub(crate) fn spawn_legacy_model_worker( )) .with_xtc_newline_token_ids(xtc_newline_token_ids) .with_reasoning_budget(reasoning_budget, thinking_ids); + scheduler.set_request_memory_worker_path( + crate::server::model_provider::request_memory_worker_path_supported( + false, + tensor_parallel.tp_size, + crate::distributed::disaggregated::ServingMode::Hybrid, + false, + ), + ); scheduler.serve(); }) }) diff --git a/src/server/request_options.rs b/src/server/request_options.rs index 297544218..7ed9cc9b6 100644 --- a/src/server/request_options.rs +++ b/src/server/request_options.rs @@ -496,6 +496,7 @@ pub(crate) fn build_server_generate_options_with_live( // carry no image parts and always leave it `None`. image_soft_tokens: None, pre_rendered_prompt_tokens: None, + admission_notification: None, // Snapshot the server-default adapter scales at admission (#1439); // the native-completion route replaces this when the request carries // its own `lora` field. diff --git a/src/server/router_webui_playwright_harness_tests.rs b/src/server/router_webui_playwright_harness_tests.rs index 518f89173..6ac6914ed 100644 --- a/src/server/router_webui_playwright_harness_tests.rs +++ b/src/server/router_webui_playwright_harness_tests.rs @@ -518,23 +518,72 @@ async fn start_harness_server( }) } -#[cfg(unix)] fn write_private_json(path: &Path, value: serde_json::Value) -> anyhow::Result<()> { use std::io::Write; - use std::os::unix::fs::OpenOptionsExt; - let mut file = std::fs::OpenOptions::new() - .write(true) - .create_new(true) - .mode(0o600) - .open(path)?; - file.write_all(serde_json::to_string_pretty(&value)?.as_bytes())?; + + let body = serde_json::to_vec_pretty(&value)?; + let parent = path + .parent() + .ok_or_else(|| anyhow::anyhow!("private JSON path has no parent: {}", path.display()))?; + // The browser treats the final name as the publication signal. Creating it + // before write_all leaves a window where it can observe and parse an empty + // or partial document. Stage a private file beside the destination, then + // publish it without replacing an earlier response. + let mut staged = tempfile::NamedTempFile::new_in(parent)?; + staged.write_all(&body)?; + staged.as_file_mut().sync_all()?; + staged + .persist_noclobber(path) + .map_err(|error| error.error)?; Ok(()) } -#[cfg(not(unix))] -fn write_private_json(path: &Path, value: serde_json::Value) -> anyhow::Result<()> { - std::fs::write(path, serde_json::to_vec_pretty(&value)?)?; - Ok(()) +#[test] +fn private_json_is_complete_when_published_and_never_clobbered() { + let temp = tempfile::tempdir().expect("tempdir"); + let path = temp.path().join("control.done.json"); + // Keep serialization long enough that publishing the final name before + // serialization/write (the original bug) gives the reader a real race. + let blob = "x".repeat(4 * 1024 * 1024); + let expected_len = blob.len(); + let writer_path = path.clone(); + let writer = std::thread::spawn(move || { + write_private_json(&writer_path, serde_json::json!({ "blob": blob })) + }); + + let deadline = std::time::Instant::now() + Duration::from_secs(10); + while !path.exists() { + assert!( + std::time::Instant::now() < deadline, + "private JSON was not published" + ); + std::thread::yield_now(); + } + let published: serde_json::Value = + serde_json::from_slice(&std::fs::read(&path).expect("read published JSON")) + .expect("the published path must always contain complete JSON"); + writer.join().expect("writer thread").expect("publish JSON"); + assert_eq!(published["blob"].as_str().map(str::len), Some(expected_len)); + + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + assert_eq!( + std::fs::metadata(&path) + .expect("published metadata") + .permissions() + .mode() + & 0o777, + 0o600 + ); + } + + let original = std::fs::read(&path).expect("original JSON"); + assert!( + write_private_json(&path, serde_json::json!({ "blob": "replacement" })).is_err(), + "publishing must retain create-new/no-clobber semantics" + ); + assert_eq!(std::fs::read(&path).expect("unchanged JSON"), original); } fn take_marker(path: &Path) -> bool { diff --git a/src/server/routes/anthropic.rs b/src/server/routes/anthropic.rs index 9d059d452..4c50c8250 100644 --- a/src/server/routes/anthropic.rs +++ b/src/server/routes/anthropic.rs @@ -103,6 +103,18 @@ fn media_rejection_response( } fn generation_error_to_response(err: anyhow::Error) -> Response { + if err + .downcast_ref::() + .is_some() + { + let response = super::generation_error_to_response(err); + return ( + response.status, + axum::Json(serde_json::json!({"type": "error", "error": response.error})), + ) + .into_response(); + } + if err.downcast_ref::().is_some() { AnthropicErrorResponse::overloaded("All slots are busy. Please try again later.") .into_response() @@ -576,6 +588,9 @@ async fn stream_messages( .map(|ids| ids.len()) .unwrap_or(0); + let admission = + crate::server::model_provider::AdmissionWaiter::arm(&mut options, cancelled.clone()); + tokio::task::spawn_blocking(move || { let message_id = format!("msg_{}", short_uuid()); @@ -829,6 +844,10 @@ async fn stream_messages( let _ = sender.send_event(&AnthropicStreamEvent::MessageStop); }); + if let Err(error) = admission.wait().await { + return generation_error_to_response(error); + } + sse_response(stream, keepalive) } diff --git a/src/server/routes/chat.rs b/src/server/routes/chat.rs index 84444768d..02069561c 100644 --- a/src/server/routes/chat.rs +++ b/src/server/routes/chat.rs @@ -1149,6 +1149,9 @@ pub(crate) async fn stream_asr_completion( let skip_chat_parsing = state.config.skip_chat_parsing; let token_events = events.clone(); + let admission = + crate::server::model_provider::AdmissionWaiter::arm(&mut options, cancelled.clone()); + tokio::task::spawn_blocking(move || { let slot = state.slots.begin( &prepared.prompt, @@ -1259,6 +1262,10 @@ pub(crate) async fn stream_asr_completion( events.done(); }); + if let Err(error) = admission.wait().await { + return generation_error_to_response(error).into_response(); + } + sse_response(stream, keepalive) } @@ -1400,6 +1407,9 @@ async fn stream_chat_completion( let echo_enabled = echo_record.is_some(); // Spawn a blocking task to handle generation + let admission = + crate::server::model_provider::AdmissionWaiter::arm(&mut options, cancelled.clone()); + tokio::task::spawn_blocking(move || { // Keep the completion controllable for exactly the lifetime of this // task; the guard unregisters the id on drop, whatever exit path @@ -1966,6 +1976,10 @@ async fn stream_chat_completion( finish_events.done(); }); + if let Err(error) = admission.wait().await { + return generation_error_to_response(error).into_response(); + } + sse_response(stream, keepalive) } diff --git a/src/server/routes/completions.rs b/src/server/routes/completions.rs index 18249135f..434c4a2ef 100644 --- a/src/server/routes/completions.rs +++ b/src/server/routes/completions.rs @@ -447,6 +447,9 @@ async fn stream_completion( let tokenizer = state.tokenizer.clone(); // Spawn a blocking task to handle generation + let admission = + crate::server::model_provider::AdmissionWaiter::arm(&mut options, cancelled.clone()); + tokio::task::spawn_blocking(move || { // Unregister the completion-control entry when this task exits, on // every path (#1444). @@ -545,6 +548,10 @@ async fn stream_completion( finish_events.done(); }); + if let Err(error) = admission.wait().await { + return generation_error_to_response(error).into_response(); + } + sse_response(stream, keepalive) } diff --git a/src/server/routes/mod.rs b/src/server/routes/mod.rs index 708777ab2..17bdd0514 100644 --- a/src/server/routes/mod.rs +++ b/src/server/routes/mod.rs @@ -171,7 +171,23 @@ pub(crate) fn chat_not_available(state: &AppState) -> Option { /// Map generation dispatch failures consistently across OpenAI-compatible /// routes. +// Used by: all serving models, all OpenAI-compatible generation routes. pub(crate) fn generation_error_to_response(err: anyhow::Error) -> ErrorResponse { + if let Some(rejection) = + err.downcast_ref::() + { + let mut response = ErrorResponse::new( + rejection.to_string(), + crate::execution::request_memory::RequestMemoryRejection::CODE, + ); + response.error.code = + Some(crate::execution::request_memory::RequestMemoryRejection::CODE.to_string()); + response.error.estimated_bytes = Some(rejection.estimated_bytes); + response.error.budget_bytes = Some(rejection.budget_bytes); + response.status = axum::http::StatusCode::SERVICE_UNAVAILABLE; + return response; + } + if err.downcast_ref::().is_some() { return ErrorResponse::service_unavailable("All slots are busy. Please try again later."); } diff --git a/src/server/routes/native_completion.rs b/src/server/routes/native_completion.rs index dea782485..f1da11a1b 100644 --- a/src/server/routes/native_completion.rs +++ b/src/server/routes/native_completion.rs @@ -876,6 +876,9 @@ async fn stream_native_completion( let want_progress = request.return_progress.unwrap_or(false); let progress_events = events.clone(); + let admission = + crate::server::model_provider::AdmissionWaiter::arm(&mut options, cancelled.clone()); + tokio::task::spawn_blocking(move || { // Slot accounting (#1440): ties this request to a numbered slot for // GET /slots and the frames' `id_slot` field. @@ -1061,6 +1064,10 @@ async fn stream_native_completion( } }); + if let Err(error) = admission.wait().await { + return generation_error_to_response(error).into_response(); + } + sse_response(stream, keepalive) } diff --git a/src/server/routes/native_route_tests.rs b/src/server/routes/native_route_tests.rs index 6dc3c53ed..8685884e5 100644 --- a/src/server/routes/native_route_tests.rs +++ b/src/server/routes/native_route_tests.rs @@ -999,3 +999,93 @@ async fn the_native_stream_carries_the_draft_keys_on_the_final_frame_only() { "the assertion must run against real partial frames, saw {timed_partials}" ); } + +#[tokio::test] +async fn request_memory_admission_rejects_before_sse_and_keeps_routes_healthy() { + let (options_tx, _options_rx) = mpsc::channel(); + let provider = Arc::new(ModelProvider::memory_admission_for_route_tests(options_tx)); + let metrics = provider.batch_metrics().clone(); + let app = create_app(state_for(provider)); + for stream in [false, true] { + for (path, mut payload) in [ + ( + "/v1/completions", + serde_json::json!({"model":"route-test-model", "prompt":"hello", "max_tokens":64}), + ), + ( + "/completion", + serde_json::json!({"prompt":"hello", "n_predict":64}), + ), + ( + "/v1/chat/completions", + serde_json::json!({"model":"route-test-model", "messages":[{"role":"user","content":"hello"}], "max_tokens":64}), + ), + ( + "/v1/responses", + serde_json::json!({"model":"route-test-model", "input":"hello", "max_output_tokens":64}), + ), + ( + "/v1/messages", + serde_json::json!({"model":"route-test-model", "messages":[{"role":"user","content":"hello"}], "max_tokens":64}), + ), + ] { + payload["stream"] = serde_json::json!(stream); + let response = tokio::time::timeout( + std::time::Duration::from_secs(5), + app.clone().oneshot(json_request(path, payload)), + ) + .await + .expect("admission must not hang") + .unwrap(); + assert_eq!( + response.status(), + StatusCode::SERVICE_UNAVAILABLE, + "{path}, stream={stream}" + ); + assert!( + !response.headers()[header::CONTENT_TYPE] + .to_str() + .unwrap() + .contains("event-stream") + ); + let bytes = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .unwrap(); + let body: serde_json::Value = serde_json::from_slice(&bytes).unwrap(); + assert_eq!( + body["error"]["type"], "request_memory_budget_exceeded", + "{path}: {body}" + ); + assert_eq!(body["error"]["code"], "request_memory_budget_exceeded"); + assert_eq!(body["error"]["estimated_bytes"], 2048); + assert_eq!(body["error"]["budget_bytes"], 1024); + if path == "/v1/messages" { + assert_eq!(body["type"], "error"); + } + let health = app + .clone() + .oneshot( + Request::builder() + .uri("/health") + .body(Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(health.status(), StatusCode::OK); + let (status, body) = post_to( + app.clone(), + "/v1/completions", + serde_json::json!({"model":"route-test-model", "prompt":"short", "max_tokens":1}), + ) + .await; + assert_eq!(status, StatusCode::OK, "short after rejection: {body}"); + } + } + assert_eq!( + metrics + .queue_depth + .load(std::sync::atomic::Ordering::Relaxed), + 0 + ); +} diff --git a/src/server/routes/responses.rs b/src/server/routes/responses.rs index fa7293bb6..8b0f54b5a 100644 --- a/src/server/routes/responses.rs +++ b/src/server/routes/responses.rs @@ -545,6 +545,9 @@ async fn stream_create_response( None }; + let admission = + crate::server::model_provider::AdmissionWaiter::arm(&mut options, cancelled.clone()); + tokio::task::spawn_blocking(move || { let mut emitter = ResponseStreamEmitter::new(); let initial_response = build_response_object(OutboundContext { @@ -1037,6 +1040,10 @@ async fn stream_create_response( } }); + if let Err(error) = admission.wait().await { + return generation_error_to_response(error).into_response(); + } + sse_response(stream, keepalive) } diff --git a/src/server/startup.rs b/src/server/startup.rs index 8572b33a6..082b5962e 100644 --- a/src/server/startup.rs +++ b/src/server/startup.rs @@ -2008,6 +2008,7 @@ pub(crate) fn warmup_model(model_provider: &ModelProvider) -> Result<()> { model_provider.generate( "Hello".to_string(), ServerGenerateOptions { + admission_notification: None, // Inert: the generation bounds are native-route fields (#1477). n_indent: 0, t_max_predict_ms: None, @@ -3152,6 +3153,8 @@ pub async fn start_server(mut startup: ServerStartupConfig) -> Result<()> { } } + // Invalid explicit request caps fail startup, rather than silently disabling admission. + crate::execution::request_memory::budget_override()?; let runtime = crate::initialize_runtime_checked()?; if let Some(invalid) = runtime.invalid_device_override.as_deref() { tracing::warn!( diff --git a/src/server/types/response.rs b/src/server/types/response.rs index bc81fb455..8551f1cda 100644 --- a/src/server/types/response.rs +++ b/src/server/types/response.rs @@ -514,6 +514,10 @@ pub struct ErrorDetail { #[serde(rename = "type")] pub error_type: String, pub code: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub estimated_bytes: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub budget_bytes: Option, } impl ErrorResponse { @@ -523,6 +527,8 @@ impl ErrorResponse { message: message.into(), error_type: error_type.into(), code: None, + estimated_bytes: None, + budget_bytes: None, }, status: axum::http::StatusCode::BAD_REQUEST, } @@ -534,6 +540,8 @@ impl ErrorResponse { message: message.into(), error_type: "server_busy".into(), code: None, + estimated_bytes: None, + budget_bytes: None, }, status: axum::http::StatusCode::SERVICE_UNAVAILABLE, } @@ -549,6 +557,8 @@ impl ErrorResponse { message: message.into(), error_type: "server_timeout".into(), code: None, + estimated_bytes: None, + budget_bytes: None, }, status: axum::http::StatusCode::GATEWAY_TIMEOUT, } @@ -564,6 +574,8 @@ impl ErrorResponse { message: message.into(), error_type: "server_error".into(), code: None, + estimated_bytes: None, + budget_bytes: None, }, status: axum::http::StatusCode::INTERNAL_SERVER_ERROR, } @@ -584,6 +596,8 @@ impl ErrorResponse { message: "The client closed the connection before the request finished.".into(), error_type: "client_closed_request".into(), code: None, + estimated_bytes: None, + budget_bytes: None, }, status: axum::http::StatusCode::from_u16(499) .unwrap_or(axum::http::StatusCode::BAD_REQUEST), @@ -605,6 +619,8 @@ impl ErrorResponse { message: message.into(), error_type: "not_supported_error".into(), code: None, + estimated_bytes: None, + budget_bytes: None, }, status: axum::http::StatusCode::NOT_IMPLEMENTED, } @@ -619,6 +635,8 @@ impl ErrorResponse { message: message.into(), error_type: "not_implemented".into(), code: None, + estimated_bytes: None, + budget_bytes: None, }, status: axum::http::StatusCode::NOT_IMPLEMENTED, } diff --git a/src/webui/assets/mlxcel-webui-manifest.json b/src/webui/assets/mlxcel-webui-manifest.json index 727e7de3b..a6bbda7f8 100644 --- a/src/webui/assets/mlxcel-webui-manifest.json +++ b/src/webui/assets/mlxcel-webui-manifest.json @@ -67,5 +67,5 @@ }, "pnpm_version": "11.18.0", "schema_version": 1, - "source_digest_sha256": "2b01882a942ae8c25371d2c53b3d58916de69812989df29a4c785ac7ff9508e2" + "source_digest_sha256": "1ebf0921089e00db6bf288cdd66fb2ba6265f9ee43a4224a07d483f1201d79c8" } diff --git a/tests/pipeline_ci_multi_stage_real_models.rs b/tests/pipeline_ci_multi_stage_real_models.rs index feaf7a246..e7dfcdd4c 100644 --- a/tests/pipeline_ci_multi_stage_real_models.rs +++ b/tests/pipeline_ci_multi_stage_real_models.rs @@ -272,6 +272,7 @@ fn assert_multi_stage_coordinator_matches_dense_baseline( grammar: None, image_soft_tokens: None, pre_rendered_prompt_tokens: None, + admission_notification: None, }; let dense = dense_provider diff --git a/tests/pipeline_server_remote_real_models.rs b/tests/pipeline_server_remote_real_models.rs index a6af4b76a..ca3a4ebbe 100644 --- a/tests/pipeline_server_remote_real_models.rs +++ b/tests/pipeline_server_remote_real_models.rs @@ -165,6 +165,7 @@ fn assert_remote_coordinator_matches_dense_baseline( grammar: None, image_soft_tokens: None, pre_rendered_prompt_tokens: None, + admission_notification: None, }; let dense = dense_provider diff --git a/tests/structured_outputs.rs b/tests/structured_outputs.rs index e69b3e7aa..d8fe4f235 100644 --- a/tests/structured_outputs.rs +++ b/tests/structured_outputs.rs @@ -402,6 +402,7 @@ fn end_to_end_constrained_chat_completion_emits_schema_conforming_json() { grammar: None, image_soft_tokens: None, pre_rendered_prompt_tokens: None, + admission_notification: None, }; let result = provider diff --git a/tests/zero_config_cluster_init.rs b/tests/zero_config_cluster_init.rs index 8656624d4..cf47a7487 100644 --- a/tests/zero_config_cluster_init.rs +++ b/tests/zero_config_cluster_init.rs @@ -263,6 +263,7 @@ fn zero_config_two_stage_cluster_matches_dense_baseline() { grammar: None, image_soft_tokens: None, pre_rendered_prompt_tokens: None, + admission_notification: None, }; let dense = dense_provider diff --git a/webui/package.json b/webui/package.json index 71d3e7b9a..d3163de62 100644 --- a/webui/package.json +++ b/webui/package.json @@ -8,7 +8,7 @@ "build": "vite build", "typecheck": "tsc --noEmit", "lint": "eslint . --max-warnings=0", - "unit": "node --test scripts/activity-stream.spec.mjs scripts/check-theme-selectors.spec.mjs && vitest run", + "unit": "node --test scripts/activity-performance-diagnostics.spec.mjs scripts/activity-stream.spec.mjs scripts/check-theme-selectors.spec.mjs && vitest run", "check:theme-selectors": "node scripts/check-theme-selectors.mjs", "browser": "playwright test --project=chromium", "browser:all": "playwright test --config playwright.engines.config.ts", diff --git a/webui/scripts/activity-performance-diagnostics.mjs b/webui/scripts/activity-performance-diagnostics.mjs new file mode 100644 index 000000000..0ac0a2f4d --- /dev/null +++ b/webui/scripts/activity-performance-diagnostics.mjs @@ -0,0 +1,81 @@ +// Copyright 2026 Lablup Inc. Licensed under the Apache License, Version 2.0. +import { execFile as execFileCallback } from 'node:child_process'; +import { readFile as readFileDefault } from 'node:fs/promises'; +import { promisify } from 'node:util'; + +const execFileDefault = promisify(execFileCallback); +const queryFields = [ + 'timestamp', + 'index', + 'pstate', + 'utilization.gpu', + 'utilization.memory', + 'temperature.gpu', + 'power.draw', + 'clocks.current.graphics', + 'clocks.current.memory', +]; +const valueNames = [ + 'timestamp', + 'index', + 'pstate', + 'gpu_utilization_percent', + 'memory_utilization_percent', + 'temperature_c', + 'power_w', + 'graphics_clock_mhz', + 'memory_clock_mhz', +]; + +function compactError(error) { + const message = error instanceof Error ? error.message : String(error); + return message.replaceAll(/\s+/g, ' ').slice(0, 300); +} + +function hostState(text) { + const fields = text.trim().split(/\s+/); + if (fields.length < 4) throw new Error('unexpected /proc/loadavg shape'); + const [running, total] = fields[3].split('/'); + return { + load_one: Number(fields[0]), + load_five: Number(fields[1]), + load_fifteen: Number(fields[2]), + running: Number(running), + total: Number(total), + }; +} + +function gpuState(stdout) { + const lines = stdout.trim().split('\n').filter(Boolean); + if (lines.length !== 1) throw new Error(`expected one GPU row, got ${lines.length}`); + const values = lines[0].split(',').map((value) => value.trim()); + if (values.length !== valueNames.length) throw new Error(`expected ${valueNames.length} GPU fields, got ${values.length}`); + return Object.fromEntries(valueNames.map((name, index) => [name, values[index]])); +} + +/** Capture bounded, best-effort runner diagnostics without affecting the acceptance verdict. */ +export async function capturePerformanceDiagnostics(nvidiaSmi, dependencies = {}) { + const execFile = dependencies.execFile ?? execFileDefault; + const readFile = dependencies.readFile ?? readFileDefault; + const now = dependencies.now ?? (() => new Date().toISOString()); + const snapshot = { observed_at: now() }; + try { + snapshot.host = { status: 'captured', ...hostState(await readFile('/proc/loadavg', 'utf8')) }; + } catch (error) { + snapshot.host = { status: 'unavailable', error: compactError(error) }; + } + if (!nvidiaSmi) { + snapshot.gpu = { status: 'not-requested' }; + return snapshot; + } + try { + const { stdout } = await execFile(nvidiaSmi, [ + `--query-gpu=${queryFields.join(',')}`, + '--format=csv,noheader,nounits', + ], { encoding: 'utf8', timeout: 10000, maxBuffer: 64 * 1024, windowsHide: true }); + snapshot.gpu = { status: 'captured', ...gpuState(stdout) }; + } catch (error) { + snapshot.gpu = { status: 'unavailable', error: compactError(error) }; + } + return snapshot; +} diff --git a/webui/scripts/activity-performance-diagnostics.spec.mjs b/webui/scripts/activity-performance-diagnostics.spec.mjs new file mode 100644 index 000000000..e5e6671d4 --- /dev/null +++ b/webui/scripts/activity-performance-diagnostics.spec.mjs @@ -0,0 +1,39 @@ +// Copyright 2026 Lablup Inc. Licensed under the Apache License, Version 2.0. +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { capturePerformanceDiagnostics } from './activity-performance-diagnostics.mjs'; + +test('captures fixed-argv bounded GPU and host diagnostics', async () => { + const calls = []; + const snapshot = await capturePerformanceDiagnostics('/usr/bin/nvidia-smi', { + now: () => '2026-10-14T00:00:00.000Z', + readFile: async (path, encoding) => { + assert.equal(path, '/proc/loadavg'); + assert.equal(encoding, 'utf8'); + return '0.10 0.20 0.30 2/100 1234\n'; + }, + execFile: async (...args) => { + calls.push(args); + return { stdout: '2026/10/14 00:00:00.000, 0, P0, 98, 12, 67, 44.5, 1200, 900\n', stderr: '' }; + }, + }); + assert.deepEqual(calls, [[ + '/usr/bin/nvidia-smi', + ['--query-gpu=timestamp,index,pstate,utilization.gpu,utilization.memory,temperature.gpu,power.draw,clocks.current.graphics,clocks.current.memory', '--format=csv,noheader,nounits'], + { encoding: 'utf8', timeout: 10000, maxBuffer: 64 * 1024, windowsHide: true }, + ]]); + assert.deepEqual(snapshot.host, { status: 'captured', load_one: 0.1, load_five: 0.2, load_fifteen: 0.3, running: 2, total: 100 }); + assert.equal(snapshot.gpu.status, 'captured'); + assert.equal(snapshot.gpu.graphics_clock_mhz, '1200'); +}); + +test('records diagnostic failures without throwing or inventing samples', async () => { + const snapshot = await capturePerformanceDiagnostics('/usr/bin/nvidia-smi', { + readFile: async () => { throw new Error('loadavg unavailable'); }, + execFile: async () => { throw new Error('unsupported query\nprivate trailing detail'.repeat(30)); }, + }); + assert.equal(snapshot.host.status, 'unavailable'); + assert.equal(snapshot.gpu.status, 'unavailable'); + assert.match(snapshot.gpu.error, /^unsupported query private trailing detail/); + assert.ok(snapshot.gpu.error.length <= 300); +}); diff --git a/webui/scripts/activity-performance.mjs b/webui/scripts/activity-performance.mjs index 379523f8b..432a1aa51 100644 --- a/webui/scripts/activity-performance.mjs +++ b/webui/scripts/activity-performance.mjs @@ -4,6 +4,7 @@ import { readFile, writeFile } from 'node:fs/promises'; import { chromium } from '@playwright/test'; import { performanceMode, assertGeometry, performanceCompletion } from './activity-performance-config.mjs'; +import { capturePerformanceDiagnostics } from './activity-performance-diagnostics.mjs'; const base = new URL(process.env.WEBUI_PERF_BASE ?? 'http://127.0.0.1:8080/'); if (!['127.0.0.1', '[::1]', 'localhost'].includes(base.hostname) || base.username || base.password || base.search || base.hash) throw new Error('Use a credential-free loopback API base (including any API prefix).'); @@ -12,6 +13,7 @@ const key = (await readFile(required('WEBUI_PERF_KEY_FILE'), 'utf8')).trim(); const model = required('WEBUI_PERF_INFERENCE_MODEL'); const modelId = required('WEBUI_PERF_MODEL_ID'); const output = required('WEBUI_PERF_OUTPUT'); +const nvidiaSmi = process.env.WEBUI_PERF_NVIDIA_SMI; const prompt = process.env.WEBUI_PERF_PROMPT_FILE ? await readFile(process.env.WEBUI_PERF_PROMPT_FILE, 'utf8') : 'Explain the difference between a process and a thread, in detail. '.repeat(100); const modeConfig = performanceMode(process.env.WEBUI_PERF_MODE); // Playwright's default Chromium arguments include --disable-backgrounding-occluded-windows and @@ -108,12 +110,14 @@ async function clients(mode) { } } async function request(mode, pair) { + const diagnosticsBefore = await capturePerformanceDiagnostics(nvidiaSmi); const response = await fetch(api('completion?autoload=false'), { method: 'POST', headers: { Authorization: `Bearer ${key}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ model, prompt, n_predict: 256, seed: 42, temperature: 0, cache_prompt: false, stream: false }), signal: AbortSignal.timeout(600000) }); if (!response.ok) throw new Error(`Native completion failed (${response.status}); no response body logged.`); const body = await response.json(); const rate = body.timings?.predicted_per_second; if (!(typeof rate === 'number' && Number.isFinite(rate) && rate > 0)) throw new Error('No valid backend predicted_per_second; do not replace with wall-clock or word-count inference.'); - const sample = { mode, pair, tokens_per_second: rate, predicted_tokens: body.timings.predicted_n, predicted_ms: body.timings.predicted_ms }; + const diagnosticsAfter = await capturePerformanceDiagnostics(nvidiaSmi); + const sample = { mode, pair, tokens_per_second: rate, predicted_tokens: body.timings.predicted_n, predicted_ms: body.timings.predicted_ms, diagnostics: { before: diagnosticsBefore, after: diagnosticsAfter } }; results.push(sample); process.stdout.write(`${JSON.stringify(sample)}\n`); await writeFile(output, JSON.stringify({ status: 'in-progress', scope: modeConfig.name, hidden_native: modeConfig.headless ? 'not-run' : 'pending', preflight, samples: results }, null, 2)); return rate;