fp16 + VAD aborts under MPSGraph: SileroVAD::to_half() casts weights but not context/state/input

[anubissbe]

Summary

Enabling VAD on an fp16 transcriber hard-aborts the process under MPSGraph. SileroVAD::to_half() casts the VAD's weights to fp16 but leaves its context buffer, LSTM state and audio input in fp32, so the first real inference builds a graph with mismatched operand types.

This is reachable from the public API (to_half() + enable_vad()) and from example-server --fp16 --vad.

Reproduction

Build with Metal, then run the server example with both --fp16 and --vad:

example-server /tmp/pk.sock tdt-600m.safetensors tdt-600m.tokenizer.vocab \
  --model tdt-600m --gpu --fp16 --vad silero_vad_v5.safetensors

Send one request with "use_vad": true:

{"request_id":"1","audio_path":"/path/to/20s.wav","decoder":"tdt","use_vad":true}

The same request with "use_vad": false succeeds on the same warm server, which isolates the fault to the VAD path rather than to fp16 inference generally.

Actual result

The process aborts (does not throw — the whole server dies):

(mpsFileLoc): .../MPSGraphUtilities.mm:234:0: error: 'mps.add' op requires the
same element type for all operands and results
(mpsFileLoc): .../MPSGraphUtilities.mm:234:0: note: see current operation:
  %3 = "mps.add"(%arg0, %arg1) : (tensor<1x512xf32>, tensor<512xf16>) -> tensor<*xf32>
.../MPSGraphExecutable.mm:4213: failed assertion `Error: MLIR pass manager failed'

tensor<1x512xf32> is the VAD input window (context + padded audio); tensor<512xf16> is a cast weight.

Root cause

SileroVAD::to_half() (src/audio/vad.cpp:22) sets use_fp16_ and casts the model, but nothing else honours that flag:

void SileroVAD::to_half() {
    model_.to(axiom::DType::Float16);
    use_fp16_ = true;      // never read anywhere
    initialized_ = false;
}

• ensure_states() builds context_ with axiom::Tensor::zeros(...), i.e. fp32.
• process_chunk() concatenates that fp32 context with fp32 audio and feeds the result straight to model_.forward(...), which is now fp16.

Grepping use_fp16_ in src/audio/vad.cpp shows it is written but never read, so the cast is effectively unbacked.

There is also a second latent problem if the dtype plumbing is ever fixed: process_chunk() reads the result with

auto cpu_prob = prob.to_contiguous_cpu();
return cpu_prob.typed_data<float>()[0];

which would misinterpret the bits of an fp16 probability tensor. A complete fp16-VAD implementation would need the context buffer, the LSTM state, the input window and this read all made dtype-aware — plus a check that the Silero LSTM is numerically stable in fp16.

Suggested fix

Since the VAD is a ~1.2 MB preprocessor whose output is segment boundaries rather than tensors entering the ASR graph, running it in fp32 alongside an fp16 ASR model costs essentially nothing. The minimal fix is to stop casting it at the enable_vad() call sites. PR to follow.

If you would rather make fp16 VAD genuinely work, that is a larger change and I'm happy to attempt it instead — I just didn't want to guess at your intent for the LSTM numerics.

Impact

Downstream consumers that pass both flags see the sidecar die on first use. In silverstein/minutes this trips its MPSGraph crash detector, which permanently blacklists fp16 and silently falls back to fp32 — so the symptom presents as doubled GPU memory rather than as a crash, which makes it hard to trace back here.

Environment

• macOS 27.0 (arm64, Apple Silicon)
• Xcode 27.0 beta + Metal Toolchain 27A5194o
• Apple clang 21.0.0, CMake 4.4.0
• parakeet.cpp @ main (146d062), axiom submodule at pinned revision
• Model: nvidia/parakeet-tdt-0.6b-v3 converted via scripts/convert_nemo.py --model 600m-tdt