fix(tts/kokoro-ane): quiet onset on long utterances — use fp32 KokoroProsody_v2 (#947) - #963
Conversation
The shipped fp16 KokoroProsody miscomputes F0/N over the first ~1-3 s of the utterance for many T_a >= 400 frames (~10 s of audio) on the Core ML CPU and ANE paths (GPU fp16 and fp32 are exact). The corrupted F0 is flat ~120 Hz and N is compressed, so the opening words come out ~12-15 dB quiet (af_heart 'Self-attention' -38 dB vs -24 dB body). Not length-monotonic: 57/99 lengths in T=20..1980 broken for en/ja (identical weights), 56/99 zh. Isolation: same phonemes into PyTorch vs Core ML; swapping torch F0/N into the Core ML chain restores the onset; LSTM alone, palettization and each sub-op exposed as an output are all clean, so this is a fused fp16 kernel issue inside the upsampling AdainResBlk1d. KokoroProsody_v2 keeps the fp16 I/O and int8 palettization, switches compute to fp32: 0/99 broken lengths, worst F0 MAE 0.3 Hz, +~3.7 ms per call at T=450. Renamed (not overwritten) so cached clients re-download. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Supertonic3 Smoke Test ✅
Runtime: 0m21s Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf. |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 1m43s • 09/25/2026, 03:08 PM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 59.8s diarization time • Test runtime: 3m 23s • 09/25/2026, 03:09 PM EST |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 145.5s processing • Test runtime: 2m 32s • 09/25/2026, 03:15 PM EST |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 12m12s • 09/25/2026, 03:21 PM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
PocketTTS Smoke Test ✅
Runtime: 0m21s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon. |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 4m 8s • 2026-09-25T19:39:15.607Z |
Adds a short convention under **HUGGINGFACE UPLOADS** in `CLAUDE.md` so every model fix can be traced from the issue to the code to the HF commit, and back. - Fixed models ship as `_v2` next to the original. - HF commit messages name the issue and link the PRs. - mobius and FluidAudio PRs link the issue, each other, and the HF commit. - A `## Changelog` row goes on the HF model card. First applied to #947 / #963 / FluidInference/mobius#108. The [kokoro-82m-coreml card](https://huggingface.co/FluidInference/kokoro-82m-coreml) now has the table, backfilled for the #836, #852, #914 and #926 uploads. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 participant
Fixes #947.
The shipped fp16
KokoroProsodymiscomputes F0/N over the first 1-3 s of the utterance on the Core ML CPU and ANE paths onceT_areaches about 400 frames (~10 s of audio). The opening words then come out ~12-15 dB quiet. Not iOS-specific: it reproduces on macOS too. All variants are affected (en and ja share weights; zh too).KokoroProsody_v2.mlmodelc(fp32 compute, same fp16 I/O). It's renamed rather than overwritten, so cached clients re-download.ModelNames.KokoroAne.requiredCoreMLModels.Verification
af_heart: onset -38.2 dB → -26.5 dB (body -24 dB); ASR WER 7.7% → 3.8%.Links
ANE/,ANE-ja/,ANE-zh/)🤖 Generated with Claude Code