feat: identify speakers in the scene stream (voice-detect.cpp, C-API v9)

[localai-org-maint-bot]

Names the speakers the diarizer finds. Sortformer gives "speaker 0" and "speaker 1" in arrival order; this matches each slot's voice against enrolled speakers, so a transcript reads Ada: ... and Ben: .... It runs in the same one-pass scene stream as ASR, diarization and sound events, and it also names speakers in offline speaker-attributed ASR. Nothing changes for callers that do not pass a speaker model.

parakeet-cli enroll --model wespeaker_resnet34_f32.gguf --name Ada --input ada.wav --registry voices.bin
parakeet-cli enroll --model wespeaker_resnet34_f32.gguf --name Ben --input ben.wav --registry voices.bin
parakeet-cli scene --model asr.gguf --diar diar.gguf --speakers wespeaker_resnet34_f32.gguf --registry voices.bin --input meeting.wav

Docs: docs/speaker.md.

How it works

• voice-detect.cpp is folded in as a static submodule (third_party/voice-detect.cpp) behind PARAKEET_WITH_VOICEDETECT (default ON), the same way ced.cpp is behind PARAKEET_WITH_CED. pk::SpeakerEncoder is the only file that touches it, through voicedetect_capi.h. A voice-detect GGUF loads through parakeet_capi_load as a fourth context kind. The voice-detect side is build: allow embedding voice-detect.cpp in another ggml project localai-org/voice-detect.cpp#1 (embeddable build, VOICEDETECT_EXTERNAL_DR_WAV, voicedetect_capi_embedding_dim). This PR pins that commit.
• pk::SpeakerRegistry: enrolled speakers are the L2-normalized mean of their clip embeddings. identify takes a threshold and a runner-up margin, so a voice that is not enrolled comes out unknown instead of as the nearest name. It saves and loads as a small binary blob (version 1, not a stable interchange format yet).
• pk::SpeakerIdentifier: collects each slot's clean audio (overlap with another speaker is skipped, not resolved), embeds it once it has 2 s and again as it grows, and names the slot. A different name replaces the current one only after winning twice in a row, and an unknown match never removes a name. It takes audio from open segments as well as closed ones, so a speaker is named during a long turn, not only after their first pause.
• SceneStream runs the identifier after diarization. Words and utterances carry name and name_score, every update carries the current name of each slot in names. Words committed before a slot is identified keep their earlier label. Without a speaker part the JSON is byte for byte what it was.
• C-API v9, additive only: a speaker context kind, a registry handle (new, free, enroll, save, load, identify), parakeet_capi_scene_stream_begin_speaker, parakeet_capi_transcribe_and_diarize_named_json. Existing symbols and signatures are unchanged, and the scene options struct grows only at the end and is read only within the caller's size. Not used by LocalAI yet.

What was measured, and what was not

Everything below is one fixture, tests/fixtures/two_speakers.wav: two read-speech voices, alternating.

encoder	same voice / different voice (clip to clip cosine)	impostor against a registry with the other voice only	starting threshold
WeSpeaker ResNet34	0.869 / -0.012	-0.019	0.5
CAM++	0.894 / 0.383	0.374	0.5
ECAPA	0.934 / 0.558	0.546 to 0.566	0.7

The default accept_threshold (0.5) is a starting point and is encoder specific: ECAPA admitted an impostor near 0.57 with it. WeSpeaker ResNet34 is the encoder to start with. The genuine scores in the tests (0.92 to 0.98) come from enrolling with clips cut from the same recording, so they are optimistic; expect lower ones across sessions and check the threshold on your own audio.

Not measured yet: a third voice, noisy or overlapping speech, telephone audio, cross-session enrollment, ERes2Net. scene --sound was not run here (see below).

Tests

• Model independent: test_speaker_registry, test_speaker_identifier (fake encoder; overlap, refresh, hysteresis, unknown voice, open-segment consumption), test_write_atomic, plus extensions to test_sas_merge and test_scene_render.
• Model dependent (skip with 77 when the env vars are unset): test_speaker_encoder (also checks the folded encoder matches a standalone voice-detect build, cosine 1.000000), test_speaker_identify (real diarization plus the speaker encoder, enrollment in reverse order so arrival-order naming cannot pass, an unenrolled voice must stay unknown, ASR plus speaker utterance naming), test_capi_speaker (also under AddressSanitizer for the versioned scene options).
• Run locally with WeSpeaker, CAM++ and ECAPA: all pass, and test_diarization_accuracy still gives 100% frame agreement and the closed-loop TDT transcript on speech.wav is unchanged. I built with -DPARAKEET_WITH_CED=OFF because third_party/ced.cpp is not checked out on my machine, so test_scene_stream and the sound tests were skipped and a build with CED and voice-detect both ON has not been run outside CI. Please look at the build job for that one.
• CI: the CED-off gate now also builds with PARAKEET_WITH_VOICEDETECT=OFF and checks that enroll exits 2 with "built without speaker identification".

Notes for review

• Adding a submodule means submodules: recursive also fetches voice-detect's own third_party/ggml, which is never built (our ggml is reused). It only costs a clone.
• libparakeet.so now also exports the static dependency's voicedetect_capi_* symbols, the same pattern as ced_capi_*. Symbol visibility for the static deps would be a fine follow-up.
• voice-detect keeps its own process-global backend and device selection (VOICEDETECT_DEVICE, VOICEDETECT_THREADS), like CED_DEVICE. An embedded build does not apply voice-detect's CUDA / cuDNN patch, which does not matter on CPU.
• AGENTS.md: the ggml sentence said there are no local patches; the CMake applies the in-tree ones at configure time, so it now says that.
• Possible follow-ups: per-encoder default threshold from the GGUF architecture key, a threshold sweep on a second fixture, fsync before the registry rename.

🤖 Generated with Claude Code