fix: bound Nemotron-H SSD prefill memory

[inureyes]

Summary

• Bound Nemotron-H SSD prefill's quadratic surrogate-attention transient to the checkpoint-configured chunk size, carrying recurrent state and evaluating each chunk before constructing the next graph.
• Preserve the existing float32 precision boundary and add configuration validation plus focused full-step, prior-state, partial-tail, multi-batch, multi-group, f16, and bf16 coverage.
• Add load-time and production-forward attribution markers and archive the reproducible real-checkpoint memory/output harnesses and raw results.

Real-checkpoint evidence

On an M1 Ultra 128 GB host with nvidia-nemotron-3-nano-30b-a3b-4bit, three fresh-process repetitions per cell reduced median process peak memory from 20.9/31.4/48.5/72.2 GiB to 17.6/18.0/18.4/18.7 GiB at 1,519/3,019/4,519/6,019 prompt tokens. All rounded ranges were 0.0 GiB at vmmap's 0.1 GiB resolution. The candidate also completed three 8,019-token runs at 19.3 GiB. Both attribution markers were absent from every baseline run and present in every candidate run. Three substantive greedy prompts, including a 2,810-token prompt, produced byte-identical choices between baseline and candidate. No latency claim is made because the host carried background load.

Validation

• RUSTC_WRAPPER= cargo build --release --features metal,accelerate --bin mlxcel-server
• RUSTC_WRAPPER= cargo test --profile test-fast --features metal,accelerate --lib models::nemotron_h::tests:: -- --test-threads=1 (22 passed, 3 ignored)
• RUSTC_WRAPPER= cargo test --profile test-fast --features metal,accelerate --lib models::nemotron_h::tests::chunked_ssd -- --ignored --test-threads=1 (2 passed)
• cargo fmt --check
• git diff --check
• python3 scripts/ci/check_cross_repo_refs.py
• Archived benchmark artifact consistency checks and Python harness byte-compilation

Approved follow-up

The bounded SSD-prefill fix is complete. Request-level activation/transient-memory admission will be implemented in a separate follow-up, as approved by the maintainer. This PR does not provide recovery from arbitrary Metal command-buffer OOM. Issue #2281 remains open until the follow-up is linked.

Implementation and security reviews found no code defects within this scope; bilingual technical reports are included.

Refs #2281

[inureyes]

Implementation Review Summary

Intent

Bound Nemotron-H's quadratic SSD-prefill transient with chunked recurrent carry while preserving numerical behavior; request-level transient-memory admission remains outside this PR.

Findings Addressed

• Verified the production model path passes the validated checkpoint chunk size into every Nemotron-H Mamba mixer and evaluates each quadratic chunk before constructing the next graph.
• Verified cross-chunk recurrent carry, partial tails, multi-batch/multi-group behavior, float32 safety boundaries, and f16/bf16 output/state parity through focused tests and the scalar recurrence oracle.
• Verified the benchmark uses fresh processes, three repetitions per memory cell, arm-specific execution markers, process high-water memory, real-checkpoint output comparisons, and explicit unmeasured cells.

Remaining Items

• Request-level activation/transient-memory admission is not implemented (CRITICAL at issue scope, explicitly outside this PR), so arbitrary within-context Metal OOM can still terminate the process.
• GitHub currently reports fix(nemotron_h): prefill memory grows roughly quadratically with prompt length and aborts with Metal out-of-memory at ~6K tokens #2281 in this PR's closingIssuesReferences even though the body says the PR intentionally does not close it (HIGH); do not merge until the issue is split or the closing relationship is removed.

Verification

• All issue requirements implemented
• No placeholder/mock code remaining
• Integrated into project code flow
• Project conventions followed
• Existing modules reused where applicable
• No unintended structural changes
• Tests pass

No code-level correctness finding remains within the declared partial scope. Formatting, diff hygiene, benchmark artifact consistency, Python harness compilation, the 22-test Nemotron-H module suite, two serial Metal numerical tests, release server build, and current CI checks were reviewed as passing; the supplementary external reviewer could not initialize in this sandbox, so this conclusion is based on direct review and executed validation.