feat: support Moondream Parakeet Ultra and Redux

[localai-org-maint-bot]

What

Adds support for moondream/parakeet-ultra and moondream/parakeet-redux, two derivatives of parakeet-tdt-0.6b-v3. Both ship as HF safetensors that only Moondream's Photon runtime reads today.

• Converter. scripts/convert_hf_parakeet_to_gguf.py reads the HF layout and maps tensor names back to the NeMo names the loader expects. Config, vocab and the mel buffers come from a v3 GGUF (--template), and every shape is checked against it. --ternary keep stores Redux's ternary weights packed. --vad keep carries the VAD head.
• Native ternary kernel for Redux. The packed weights are repacked once at load. Activations are quantized to int8 per row, and a ggml custom op pair runs the matmul. There are AVX-512 VNNI and AVX2 kernels with runtime dispatch (no global -march) and a NEON kernel. Every kernel is bit-identical to a scalar reference. The path is taken only when a tensor <name>.qweight exists, so other models build the same graph as before.
• VAD segmentation. parakeet-cli transcribe --vad cuts long audio at pauses into segments of at most 30 s, using the model's own VAD head, and offsets word and token timestamps back. parakeet-cli vad-probe prints the per-frame probabilities. The C-API gains parakeet_capi_transcribe_path_json_vad (additive, no ABI bump).
• Docs and tests. docs/ternary.md has the measurements and the evidence behind the VAD wiring. docs/conversion.md, docs/parity.md, models/MANIFEST.md, AGENTS.md and the README are updated. New tests cover the converter, the loader flags, the kernels, the VAD head, the segmenter and the end to end paths.

Results

All measured on one Ryzen 9 9950X3D. Details and raw numbers are in docs/ternary.md.

	Redux packed ternary	Ultra Q8_0 (stands in for v3 Q8_0)	Redux dequantized F16
File size	213 MB	941 MB	1.44 GB
LibriSpeech-100, per utterance, 8 threads, median RTF	75.6x	43.1x	46.1x
WER on those 100 utterances	1.96%	1.71% (Ultra)	1.92%

• Single core kernel throughput: 547 GMAC/s (VNNI) against about 87 for ggml's own Q8_0 matmul on the same shape.
• On one 180 s clip the end to end gain is only about 10 percent (single pass, load included).
• Upstream reports 113x for its own engine on a different machine. That number is theirs, and I did not reproduce or compare it.
• VAD segmentation does not change WER meaningfully on synthetic long-form clips built from LibriSpeech (differences of about 0.2 points in both directions, three clips per set).

Limits and known gaps

• The VAD wiring is inferred. Moondream publishes no reference for the head. The chosen wiring (subsampler tap, ReLU after proj and ctx, no residual) had the best mean AUC in a 108 variant search on real speech (0.93 for Ultra, 0.95 for Redux). Its pause detection is weak, and the segmenter falls back to hard cuts when it finds no pause. The evidence and caveats are in docs/ternary.md.
• Packed ternary is CPU only and offline only. A packed GGUF is refused at load on GPU backends and in streaming, with a message. Convert with --ternary dequant for those.
• Some builds fall back to the scalar kernel (about 1 GMAC/s): MSVC, Windows on ARM, and aarch64 without the dot product extension. The loader logs a warning when that happens.
• NEON was verified only under qemu-user (bit-identical to the reference). It has not run on real ARM hardware, and its speed is unmeasured.
• Known follow-up. The check that refuses a GGUF that has packed tensors but no parakeet.ternary.present flag looks at two layer 0 tensor names. A hand-crafted file with a packed tensor in a later layer gets past it. On CPU the type and size checks still prevent an out-of-bounds read. The fix is to scan every tensor name and add a negative test.
• The numbers come from this repository's standard build, which applies the in-tree ggml patches at configure time. The ggml pin itself is unchanged.

How to verify

cmake -B build -DPARAKEET_BUILD_TESTS=ON -DGGML_NATIVE=ON && cmake --build build -j
ctest --test-dir build --output-on-failure -LE model      # 25 of 25 pass

To exercise the model dependent tests, convert the two checkpoints and export the paths (steps in docs/conversion.md and docs/ternary.md):

python3 scripts/convert_hf_parakeet_to_gguf.py --hf <ultra dir> --template <v3 f16 gguf> --output ultra-f16.gguf --dtype f16
python3 scripts/convert_hf_parakeet_to_gguf.py --hf <redux dir> --template <v3 f16 gguf> --output redux-keep.gguf --dtype f16 --ternary keep
PARAKEET_TEST_GGUF_ULTRA=ultra-f16.gguf PARAKEET_TEST_GGUF_REDUX_KEEP=redux-keep.gguf ctest --test-dir build --output-on-failure

With every model variable set, 90 of 92 tests pass. The two failures, test_relpos_attention_local_chunked and test_capi_timestamps, also fail on master's own build. They expect the 110M model's baselines, and I ran them with the 0.6B v3 model.

The ced.cpp submodule was not available where this was built, so it was built with -DPARAKEET_WITH_CED=OFF and the CED tests did not run.

Assisted-by: Claude:claude-sonnet-5-5 [Claude Code]

🤖 Generated with Claude Code