Batch Parakeet v3 custom vocabulary: measurements on 500 real dictations and a real-speech exam, with audio (great recall, common words replaced, a few bugs)

[saurabhav88]

Hi, thanks for FluidAudio. We ship EnviousWispr, an on-device macOS dictation app on Parakeet TDT v3 (batch AsrManager). Users keep a personal word list (names, brands, tools), and we've spent a lot of time trying to get the recogniser to use it. We measured the current batch vocabulary boosting on real dictation and wanted to share the results, the walls we hit, and real audio, in case it helps. Happy to be told we're holding it wrong.

Setup (v0.17.4, 21493f8)

AsrManager v3 batch transcribe, then VocabularyBoostingSession(vocabulary:ctcModels:config:) → rescore(text:tokenTimings:audioSamples:) with ctc110m. For the settings sweep we called VocabularyRescorer.ctcTokenRescore directly with our own cbw / minSimilarity, on the same spotter log-probs. Terms built in code with CustomVocabularyTerm(text:aliases:). Harness: the plain transcript and the rescored transcript from the same decode, one row per clip.

Instrument check: a one-term vocabulary (zzqqxvwj) changed 0 of 138 clips.

Result 1: 500 real dictations from one user, 51-term personal list (default session settings)

278 of 500 transcripts changed. Many swaps are genuinely great; many replace a common word that was actually said:

said	default output
Send the draft to Maya tomorrow morning.	Send the draft to EMEA tomorrow morning.
Please send the invoices for this month to Sora.	…for this [listed name] to Sora.
The quick brown fox jumps over the lazy dog.	The Qwen brown fox…
…regenerated my whole Xcode project…	…my whole Codex project…
We ship analytics through post hog…	We [listed name] Qualtrics through Posthog…
…drinking lemonade with a twist of lime.	…with a Tuist [another listed name]
The application is NVS Visper.	The application is EnviousWispr ✅
I found it in Vist Whisper.	I found it in EnviousWispr ✅ (our text layer can't reach this one)

Most frequent swaps across the 500: Maya → EMEA ×114, month → [listed name] ×17, Sora → Astra ×12, twist → Tuist ×9 (e.g. "a twist up lemon in my orange juice"), Xcode → Codex ×6, quick → Qwen ×5, test → Tuist ×5.

Result 2: turning it down helps a lot on real speech

Same 500 clips, rescoring with cbw: 0 and minSimilarity: 0.7 (user's aliases included): 40 clips changed and nearly all are right. "Twist of lime" / "twist of lemon" are left alone while "coding in twist" becomes Tuist; Vist Whisper, Envy S Whisper, Envice whisper → EnviousWispr; Quen → Qwen. Two regressions, both the same shape (below).

Result 3: a real-speech exam (Common Voice 17 English test split, 983 clips, 3,238 questions)

Each question asks whether a spot in the Parakeet transcript should become a given word: 1,056 where it should (the recogniser misheard it), 2,182 lookalikes where it must not. Each clip's vocabulary is the words asked about on that clip.

cbw / minSimilarity	should-fix caught	lookalikes wrongly swapped
0 / 0.7	437 / 1,056 (41%)	202 / 2,182 (9.3%)
1 / 0.6	786 (74%)	313 (14%)
2 / 0.5	920 (87%)	459 (21%)
4.5 / 0.5	928 (88%)	501 (23%)

For dictation a wrong swap costs more than a miss (the user has to spot it), so we can't ship any row of this table on its own.

Things that look like bugs

1. The replacement eats a neighbouring word. The application is NVS Visper. → The application EnviousWispr; update my sales in Envious Labs. → update my sales Envious Labs. Also seen deleting if, for, with, heard. Looks like the replaced span, or the timing margin, takes the adjacent short word with it.
2. Results change between identical runs. Two full runs over the same 500 clips, same settings, differed on 13 of 497 clips (for example "twist of lime" swapped in one run and not the other). Parakeet's plain transcripts were identical in both runs. Isolated reruns of those 20 clips agreed with each other, and a clip run alone matched the same clip run in a batch. Both full runs happened while the machine was under heavy load, so our guess is Core ML compute-unit placement or precision varying under load. Not established.
3. detectedTerms lists every term. With minScore: nil (the session's call), every clip reports every vocabulary term as detected (37 of 37 on every trap clip, 49 to 51 of 51 on real clips, and the null term on every clip), so it can't be used as evidence of what was heard.
4. Docs: Documentation/ASR/CustomVocabulary.md (Usage Example) calls asrManager.transcribe(audioSamples, customVocabulary: vocabulary), which doesn't exist on AsrManager in v0.17.4. The working path is VocabularyBoostingSession.rescore.
5. spotterRescueEnabled has no effect above 10 terms (the largeVocabThreshold gate), so for a personal dictionary it's inert. On vs off gave byte-identical output on all our runs. Maybe worth a note in the docs where the switch is described.

What we tried ourselves

1. The CTC-spotter approach on the batch path (FluidVoice's fork, September): below no boosting on a 138-clip trap set at every weight, including weight 0.
2. Decode-time phrase boosting inside the TDT greedy loop (our fork): a reward on the next piece of a listed phrase inside the joint's top-64 shortlist, applied before the language filter; keep the boosted text only when it delivers a whole listed word; never reward past a no-token decision (without that guard, silent clips came back as hundreds of words of babble). Zero collateral on ordinary speech at a mild reward, but it recovers little (about 1 extra word in 80 real dictations), and stronger rewards break ordinary words (Okay, I mean → OpenAI). Greedy decoding can't compare twist and Tuist as whole hypotheses.
3. What works best for us so far: use the spotter as a proposer and let a separate context model decide, since "twist of lime" vs "coding in Tuist" is a meaning question, not an acoustic one. ctcTokenEvaluateCandidates is exactly the right shape for that, thank you for exposing it.

Questions

1. Precision guard: is there a supported way to require the listed word to beat the transcribed word's own CTC score by a margin at that span? cbw below 0 was our attempt; it removed almost all recall.
2. Beam search with phrase boosting for TDT v3 batch: NVIDIA reports about twice the greedy gain (arXiv 2508.07014). Is it on your roadmap?
3. Multilingual spotting: the ctc110m vocabulary has no non-ASCII pieces, so German, French, Spanish and similar terms can't be spelled. Is a multilingual spotter planned for v3?

Audio

The zip in the first comment has 13 of the clips above (16 kHz mono, from the user's dictation archive), with clips.json giving what was said, the plain transcript, the default output and the cbw 0 / sim 0.7 output. We can also share the harness, the 138-clip TTS trap set, and the exam question list.

[saurabhav88]

Audio for the examples above: 13 clips (16 kHz mono) from the user's dictation archive, plus clips/clips.json with what was meant, Parakeet's plain transcript, the default session output, and the cbw 0 / minSimilarity 0.7 output for each. Names from the user's personal list are masked as [listed name] in the JSON text.

fluidaudio-vocab-clips.zip