[ncylich]

Benchmarks

Records the Mac M4 Pro and Galaxy S26 README numbers (consolidates #725), with Parakeet-TDT now transcribing on the Apple Neural Engine on Mac.

• LLM: Gemma-4-E2B-CQ4 (CPU, no speculative decode) — 1k-prefill tps / 100-decode tps
• VLM: Gemma-4-E2B-CQ4 (NPU prefill, CPU decode), 256px — latency / decode tps
• Transcribe: Parakeet-TDT-0.6B (NPU encoder, CPU decode), 20s audio — latency

Device	Gemma4 Text	Gemma4 Vision	Parakeet	RAM
Mac M4 Pro	324 / 39	1.2s / 48	0.2s (NPU)	1385 MB
Galaxy S26	248 / 21	17s / 16	4.2s	4024 MB

On the M4 Pro the Parakeet 20s clip drops from ~1.2s (CPU encoder) to ~0.2s on the Neural Engine — ~6× faster — with the CPU path unchanged and the transcript matching CPU.

Parakeet NPU acceleration

Adds Apple Neural Engine (ANE) acceleration for the Parakeet-TDT audio encoder.

ANE-friendly encoder export. The Conformer relative-position attention used a torch.gather (plus a runtime reverse) that the ANE compiler rejects. It's replaced with a static Transformer-XL rel-shift over precomputed, reversed position keys — numerically identical to the gather — captured at a fixed sequence length as a separate npu_module. A new npu_example_inputs spec field lets the NPU export shape be set independently of the CPU graph. The torch.export _assert_tensor_metadata node, which coremltools can't translate, is registered as a no-op.

Padding mask (transcript fix). The encoder exports at a fixed window and the engine zero-pads shorter clips, but the Conformer's global attention had no padding mask, so the padded frames leaked into every real frame across all layers and drifted the transcript. The ANE encoder now derives the exact valid-frame count from the input (real mel frames are non-zero, padding is exact zero) and applies it as an additive attention key bias plus a depthwise-conv mask, matching the unpadded CPU encoder (cos 0.9997 vs 0.959 before). What previously looked like fp16 "drift" was unmasked padding — not precision or quantization.

Compute-units routing. The bundle manifest records the audio encoder's CoreML compute units (npu_audio_compute_units), threaded through to the encoder loader and honored when resolving compute units. The transpiler writes CPU_AND_NE for parakeet_tdt; the default is empty, so other encoders are unaffected. Audio longer than one NPU window falls back to the CPU encoder to preserve global attention.

Notes.

• Verified end to end through the engine STT path: the bundle auto-routes to the Neural Engine (no env var) and transcribes correctly, matching the CPU transcript.
• The rel-shift was verified equal to the original gather (max diff 0.0) across sequence lengths; the CPU graph/decoder path is unchanged.
• A MILCompilerForANE … ANECCompile() FAILED log line is emitted but is non-fatal: one sub-region falls back while the bulk runs on the ANE. Cold ANE compilation of the 24-layer Conformer is slow and variable (Apple-compiler-side, independent of the mask); it is cached as .mlmodelc after the first load.