fp16 inference crashes MPSGraph: 'mps.add' f16/f32 operand type mismatch on Apple Silicon (tdt-600m)

[silverstein]

Summary

Running tdt-600m (parakeet-tdt-0.6b-v3) with fp16 enabled on Apple Silicon crashes during Metal inference. MPSGraph rejects an mps.add op because one operand is f32 and the other is f16 — the fp16 path adds an fp16 bias/weight tensor to an fp32 activation without a cast. fp32 inference works correctly on the same machine, model, and audio.

Looks like a case that slipped through the #14 (module_to_dtype fp16 casting) work.

Environment

• parakeet.cpp commit: a212eea694ab2de39d4dd1dc601806fed9ce8b19
• OS: macOS 26.6 (arm64, Apple Silicon)
• Xcode: 26.6 (Metal toolchain installed via xcodebuild -downloadComponent MetalToolchain)
• Model: parakeet-tdt-0.6b-v3 converted with scripts/convert_nemo.py --model 600m-tdt
• Build: -DCMAKE_BUILD_TYPE=Release -DPARAKEET_BUILD_SERVER_EXAMPLE=ON -DAXIOM_INSTALL=OFF -DPARAKEET_INSTALL=OFF, CMake 3.31.12
• Configure reported Metal support: YES, Target architecture: arm64; binary links Metal, MPS, MPSGraph, Accelerate

Reproduce

parakeet tdt-600m.safetensors speech.wav \
  --model tdt-600m --vocab tdt-600m.tokenizer.vocab --gpu --fp16

(also reproduces via example-server with fp16 enabled)

Error

(mpsFileLoc): .../MPSGraphUtilities.mm:234:0: error: 'mps.add' op requires the
same element type for all operands and results
note: see current operation: %3 = "mps.add"(%arg0, %arg1)
      : (tensor<1x512xf32>, tensor<512xf16>) -> tensor<*xf32>
...
MPSGraphExecutable.mm:1484: failed assertion `original module failed verification'

The offending op adds tensor<1x512xf32> (activation) and tensor<512xf16> (a 512-wide bias/weight — likely an encoder linear/bias or layernorm term) with no cast. Process aborts on the MPSGraph verification assertion.

Expected

fp16 mode either casts operands to a common element type before mps.add, or keeps that bias/weight in the same precision as the activation it's added to.

Notes

• Downstream in Minutes we detect this crash signature and auto-fall back to fp32, so it's not blocking — filing so the fp16 path (faster, lower memory) can be fixed at the source.
• The fixed 512 width may help localize it: it suggests one specific encoder tensor isn't covered by the dtype conversion rather than a systemic issue.