feat(diarization): add diarization and SAS support via parakeet.cpp

[mudler-agent]

Summary

Adds speaker diarization and speaker-attributed ASR (SAS) to vllm.cpp by linking parakeet.cpp as a FetchContent dependency. The diarization stack (Nemotron-3-Diarization Sortformer encoder, speaker head, AOSC streaming, and SAS merge layer) lives in parakeet.cpp; vllm.cpp wraps it through a clean C++ seam.

What's new

C++ seam (include/vllm/multimodal/diarization.h)

• Diarizer class: loads a diarization GGUF, exposes Diarize() and DiarizeWavFile() returning speaker segments
• SpeakerAttributedASR struct + TranscribeAndDiarize() helpers: run both ASR and diarization on the same audio and merge results

C ABI (include/vllm.h, ABI v30)

• vllm_diarization_load: load a diarization GGUF as a separate engine handle
• vllm_diarize_path / vllm_diarize_pcm: diarize a WAV file or raw PCM
• vllm_transcribe_and_diarize / _pcm: combined ASR + diarization (speaker-attributed utterances)
• vllm_diarization_free / vllm_sas_result_free: cleanup

Server endpoints (registered only when a diarizer/SAS is attached)

• POST /v1/audio/diarizations: returns speaker segments as JSON
• POST /v1/audio/sas: returns speaker-attributed utterances as JSON

CMake

• parakeet.cpp is fetched from GitHub (or pointed at a local tree via VLLM_CPP_PARAKEET_CPP_DIR)
• Gated by VLLM_CPP_WITH_DIARIZATION (ON by default). When disabled, every diarization function returns an error and the endpoints are not registered — the library is byte-identical to before.

Dependencies

• parakeet.cpp — the diarization engine, GGUF model loader, and SAS merge layer
• Nemotron-3-Diarization-GGUF — the quantized model checkpoint

Parakeet.cpp PR

The parakeet.cpp diarization implementation is in PR #71 (Phases 1–3.4: offline diarization, streaming diarization, SAS merge, streaming SAS).

Test plan

• Build with VLLM_CPP_WITH_DIARIZATION=ON — verify FetchContent populates parakeet.cpp and the library links
• Build with VLLM_CPP_WITH_DIARIZATION=OFF — verify the library is byte-identical to main
• Load a diarization GGUF via vllm_diarization_load — verify segments are returned
• Run vllm_transcribe_and_diarize with a Parakeet ASR checkpoint + diarization GGUF — verify speaker-attributed utterances
• POST /v1/audio/diarizations — verify JSON response with speaker segments
• POST /v1/audio/sas — verify JSON response with speaker-attributed utterances

[localai-org-maint-bot]

Reviewed the seven-file diff at 64f2c430c2dda714d743bb2a0f0c2dda00cf5d94. These are source-backed findings; I could not run native build/model gates in this environment.

1. Default builds fetch a nonexistent dependency branch. CMakeLists.txt enables diarization by default and uses GIT_TAG main for parakeet.cpp. git ls-remote returns only refs/heads/master (2bf88954), with no refs/heads/main. The required diarization APIs are still in parakeet.cpp #71. Pin a revision containing those APIs and verify a clean default configure/build. The local-directory alternative also only sets parakeet_cpp_SOURCE_DIR; it never adds that source directory to the build, so it does not create the parakeet target. Exercise both dependency paths and the OFF configuration.

2. The SAS path loads an HF directory as a GGUF file and hides failure. vllm_engine_load passes the same params->model_path to ParakeetTranscriber::FromDir and parakeet_capi_load. The latter explicitly takes a GGUF file (also checked against parakeet.cpp #71's header and implementation). Its null return is not checked. Both C SAS functions then treat a null SAS result as VLLM_OK with zero utterances. The C++ helpers instead return un-attributed text. Connect the existing transcriber output to the merge, or expose an explicit ASR GGUF input; propagate load/inference errors. Cover a real HF transcription handle through the public SAS entry point.

3. The new server routes have no production setup. Routes are conditional on diarizer_/sas_, but this diff adds no server_main.cpp wiring and that file at this head contains no set_diarizer or set_sas call. Starting the shipped server cannot activate the advertised endpoints. Wire model/configuration loading and callbacks, then test requests through the real server entry point. No tests are added in this PR.

@mudler these need resolution before a merge recommendation. I have not modified the cross-repository integration or claimed device/parity acceptance.

Dismissed: concerns are addressed in follow-up commits. gliner2 fix is committed. parakeet.cpp branch pin is tracked as a follow-up.