Conversation
Boss Calling required a metered OpenAI Realtime session. This adds a second backend that runs on the user's own machine, with no key and no audio leaving the device once the models are on disk. The key finding is that this is a provider choice rather than a second voice stack. HostVoiceCallController is a state machine over the OpenAI Realtime event protocol, so a server speaking that protocol needs no controller change: huggingface/speech-to-speech (Apache-2.0) serves /v1/realtime, implements every event the controller sends or handles, carries tool calls through session.update + function_call_output, and speaks PCM16 mono 24 kHz, identical to VoiceAudioIo.FORMAT. The event check included the newer response.output_audio.* spelling rather than the legacy response.audio.*; a server on the old names connects, transcribes and is silent. What changed: - VoiceBackend + VoiceEndpointResolver: the backend decision as a pure function, with actionable failures. There is deliberately NO fallback between backends. Silently re-routing a LOCAL call through OpenAI would bill the user and put their microphone audio on the network, and neither is something a person choosing a local backend is asking for. - RealtimeTransport takes a resolved endpoint instead of a model/key pair, and omits the Authorization header entirely when there is no credential. The local server performs no authentication, so sending a live OpenAI key to an arbitrary local process would be a leak. - voice/local/: the managed runtime. Pinned install through uv (pip fallback), a child process supervised with a shutdown hook, and a readiness probe on GET /v1/pool rather than the realtime socket, which would open and abandon a session every poll. --host 127.0.0.1 is explicit because the server performs no authentication: the bind address is a security boundary. - Share surface: a LOCAL backend reports local_backend and refuses. A host who switched to local may still have an OpenAI key on disk, and without the backend check ordered ahead of the key check the share path would find it and place a metered call anyway. Remote viewers cannot reach host loopback, so unavailable is also the honest answer. - Settings: backend picker plus install/start/stop and readiness for the local runtime. Defaults to OPENAI, so an existing install is unchanged. Not yet validated at runtime against a live local server; the compatibility claims above are verified in the server's source, not in a placed call.
Follow-up to the local voice backend, addressing what the review of #422 turned up. Start race: the spawn decision and the spawn itself sat on opposite sides of the process lock, so concurrent callers (a call placed while someone presses Start in Settings, or two calls close together) could both spawn. The loser's handle was overwritten, and nothing could reap it afterwards: stop() and the shutdown hook only see that one field, and the losing monitor's `process !== started` check returns without destroying anything, leaving an unauthenticated speech server holding the port. The whole start sequence now runs under a Mutex, with the state re-checked inside it. The race test needs the interleaving pinned to be worth anything: an ungated version passed against the pre-fix code (measured spawns=1), so the injected runner parks the first caller inside spawn() until a second caller reaches the decision. Verified in both directions — it fails with the Mutex removed and again with the Starting-state guard removed, so neither piece is redundant. needsLocalSetup was computed, tested, and then dropped: the resolver set it, and the controller discarded it, so pressing Call with the local backend selected and nothing installed produced a message pointing at Settings with no way to get there. It now reaches HostCallState and draws an Open Settings button on the call bar. voiceLocalExternalUrl was untested and unvalidated. The http(s):// address a person copies out of a browser is now translated to ws(s)://, which is what the JDK transport accepts, and anything unusable is rejected against the setting by name instead of falling back to the managed runtime — the one substitution this backend must never make quietly. A non-loopback address draws a warning in Settings, since it is the one configuration that contradicts "no audio leaves the device". Also: platform detection goes through ShellCustomizationUtils rather than a second spelling of the os.name check, the Python floor is named once instead of hardcoded next to the check, and dispose()'s role is described accurately (the shutdown hook is the only reaper in production, since nothing calls dispose()).
The previous commit described the Mutex as fixing a leak: that two callers could both spawn and the loser's process would be orphaned with nothing able to reap it. That is not what the earlier code did. The liveness check and ProcessBuilder.start() both ran inside synchronized(lock), so a second caller blocked at monitor entry and could not reach its spawn decision. Verified rather than reasoned: restoring f70d8f4's LocalVoiceRuntime.kt and running the new LocalVoiceRuntimeTest unchanged passes, spawnCount 1 and no survivors. The regression test does not distinguish the two implementations, so "verified in both directions" was self-referential - removing the Mutex fails only because the spawn had already been moved out of the monitor. The Mutex stays, for the reason that is actually true: the old shape called ProcessBuilder.start() while holding a JVM monitor, pinning a Dispatchers.IO thread for the length of a process spawn. Moving the spawn out lets the start sequence suspend, and the Mutex preserves the mutual exclusion the monitor used to provide. The concurrency test guards that current structure; it is not a reproduction of a historical bug. Corrected in the startMutex and ensureRunning KDoc, the SpawnGate and race test KDoc, and AGENTS.md, which had inherited the same claim.
Owner
Author
Runtime validation found a blocker, and it breaks this PR's main claimFirst run against a real server. The install and the speech pipeline work; the LLM does not, and the reason invalidates what I wrote above. The bug
Upstream's README is explicit about what that configuration is (their "Local speech with a hosted LLM" section):
So as shipped, the "local" backend needs an OpenAI key, is metered, and sends the conversation transcript off the device. Audio stays local; nothing else does. "No key, and no audio leaves the device" is wrong as written, and the half that is true is the less important half. I read that line in the README during evaluation and did not follow it through to the default. Reading the docs is not running the thing. Why no test caught itEvery test asserts on the command we construct, never on what the server does with it. Second finding: the readiness probe is not a readiness probe
Fix in progress
The memory figures in the settings copy need revising with it. 16 GB is upstream's recommendation for the hosted-LLM configuration (~5.2 GB of speech weights). Fully local adds the LLM on top, and Qwen3-4B alone is ~8 GB. Still unprovenThe audio event naming ( Do not merge this PR yet. |
Owner
Author
Correction, and one blocker candidate retiredTwo updates to my previous comment. The main blocker stands; my second finding was wrong. Withdrawing the "readiness probe is not a readiness probe" findingI said What actually happened:
So readiness behaves correctly: the process exits, The LLM blocker is confirmed independentlyFrom the installed package's own
Blocker candidate retired: the audio event naming is correctThis was the other thing that could have made every local call silent, and it does not. Against a genuinely all-local server with no API key present, driven with the exact nested session schema Audio arrives as Worth recording: an initial probe using the older flat schema ( Measured numbers so far
Still open: the LLM fix and its tests, the corrected memory and disk figures, and the remaining protocol evidence (tool round-trip, interruption, |
Found by running the thing. The shipped serveCommand passed only --host and --port, which left --llm_backend at upstream 1.0.0's default of responses-api: a hosted OpenAI client. Against a real server the process loaded Smart Turn VAD, Silero VAD and Parakeet TDT on MPS and then died after 62.2 seconds with openai.OpenAIError: The api_key client option must be set ... Upstream's own README is explicit that this configuration carries provider API charges and that "Transcribed text, instructions, and conversation history are sent to OpenAI". So the backend advertised as needing no key and keeping data on the device did neither; only the audio stayed local. The claim in the settings copy was wrong, and no test could have caught it, because every test asserted on the argv we construct rather than on what the server does with it. serveCommand now names the pipeline: --stt parakeet-tdt, --llm_backend mlx-lm on Apple Silicon and transformers elsewhere, --model_name Qwen/Qwen3-4B-Instruct-2507, --tts qwen3, plus --device mps on macOS. Upstream's --mac-optimal-settings would cover macOS in one flag but is macOS-only and implicit, and the individual flags do not select MPS on their own, hence the explicit device. Measured, not estimated: the fully local model set is about 16 GB of downloads (HuggingFace cache 15.2 GiB plus a 1.6 GiB venv). Upstream's 16 GB memory figure describes the HOSTED-LLM configuration with ~5.2 GB of speech weights; running Qwen3-4B locally on top needs more, so the UI now asks for at least 24 GB available. No peak-RSS number is claimed because none was measured. Retired while validating: audio arrives as response.output_audio.delta / .done, matching the controller. The legacy response.audio.* spelling, which would have made every local call silent, does not appear. The older flat session schema is rejected by this server, so the nested schema the controller builds is also correct. Full :compose-ui:desktopTest green, verified in an isolated worktree. Still open, and the PR stays do-not-merge until they are closed: the shipped command has not yet been observed serving key-free end to end (the passing protocol run used a diagnostic --mac-optimal-settings server); START_TIMEOUT_SECONDS is still 180 with no measurement of cold start with a 4B model loaded; and the spawned server inherits the ambient environment, so it writes to ~/nltk_data and the default HuggingFace cache rather than anywhere under the managed home.
The spawned server inherited the ambient environment, so it wrote model and tokenizer data wherever the user's machine defaults point: the default HuggingFace cache and ~/nltk_data. That is not hypothetical - an install run died outright when ~/nltk_data was not writable, and the failure looked like a model problem rather than a path problem. LocalVoiceInstall.processEnvironment pins UV_CACHE_DIR, HF_HOME, NLTK_DATA and XDG_CACHE_HOME under home(), and both install and serve now run with it. These are the locations the pinned distribution was actually observed writing to, not a guessed list. The map is overlaid with putAll onto the inherited environment rather than replacing it, so PATH and the rest of the user's environment survive; replacing it wholesale would break the interpreter lookup this same code performs. Also measured while closing this out: the shipped serve command now reaches GET /v1/pool 200 with OPENAI_API_KEY, OPENAI_BASE_URL, OPENAI_ORG_ID and OPENAI_PROJECT removed from the environment, and a real JdkRealtimeTransport with a null apiKey completes a function-call/function_call_output round trip against it. Spawn to serving was 119.4s including a first download of Qwen3-4B, inside START_TIMEOUT_SECONDS at 180, so the constant stays as it is on evidence rather than being raised speculatively. Full :compose-ui:desktopTest green.
Groundwork for replacing VoiceDuplexGate's inference with real acoustic echo cancellation on macOS. The gate compares a microphone level against a learned attenuation of the speaker reference. That is a heuristic standing in for AEC we never had, and measurement on the local backend shows where it runs out: with a correct reference of 0.40-0.48 and true echo coupling around 0.83, the gate held its estimate at 0.10-0.24, so the bar sat at a third of the echo and the agent's own voice tripped a barge-in on frame after frame. It cannot climb out on its own either, since it only learns from frames below the bar, and a low bar means no frame qualifies. macOS ships the fix as kAudioUnitSubType_VoiceProcessingIO, the AudioUnit behind FaceTime's microphone path. It has the one thing the gate can never have: the render and capture streams in a single clock domain, sample aligned, so it subtracts the echo rather than guessing at it. This commit adds the JNA bindings and an opt-in probe that establishes the parts most likely to have sunk the idea: AudioToolbox loads, the vpio component instantiates, its input bus enables, and it accepts PCM16 mono at 24 kHz - the format VoiceAudioIo.FORMAT already uses, so nothing needs resampling between the canceller and the existing path. JNA rather than the FFM API because this module targets Java 17, where java.lang.foreign is still incubating, and JNA is already a dependency. The probe is opt-in (BOSSTERM_AEC_PROBE=1) because it needs a real audio device and CI has no mixer. It was verified to execute rather than skip by planting a failing assertion at its end and confirming the run turned red. Deliberately stops before starting the unit: that needs a realtime callback on a JNA proxy, which is the next question and worth nothing if the unit could not be created in the first place. Nothing is wired into the call path yet, so behaviour is unchanged.
Second step of moving echo cancellation off our heuristic and onto the
platform. Drives Voice Processing I/O as a full-duplex PCM16 mono device:
enable both buses, pin the format, install the microphone and speaker
callbacks, and pump audio through two rings.
Microphone bytes only become echo-cancelled by being pulled through
AudioUnitRender, so the input callback renders into a buffer list we own and
reuse rather than reading whatever the callback was handed. Both callbacks
run on CoreAudio's realtime thread, so everything they touch is allocated in
start() and only copied through afterwards, and the callback objects are held
in fields because JNA keeps no strong reference to one handed to native code.
AudioRing is the piece that took three attempts, and the concurrency test is
the reason the first two did not ship:
- Drop-oldest with eviction in the writer had BOTH threads assigning the read
cursor. The reader's own update could land after the writer's and move the
cursor backwards onto overwritten slots, returning bytes from two different
writes. Observed as a monotonic counter reading 76170 then 75915.
- Moving eviction to the reader fixed cursor ownership but not the tear: a
writer that reuses unread slots races the reader's in-progress copy however
the loss is accounted, and detecting it afterwards is too late. Observed
again at 8685 then 8430.
So the writer now refuses rather than overwrites. A consumer that falls behind
loses the same audio either way; what changes is that tearing is impossible by
construction rather than by argument, on a path whose failure mode is a click
in someone's ear.
The concurrency test earns its keep twice over. Its first version used a
repeating byte ramp and could not tell a legitimate drop that wrapped 255 to 0
from interleaved bytes, so it failed on a false positive; a monotonic 32-bit
counter has no such ambiguity. It then caught both real defects above.
Also fixes three LocalVoiceRuntimeTest fakes that used `probe = { true }`.
That was a fine stand-in for "readiness succeeds" before the runtime learned
to adopt a server already answering on the port, and afterwards it describes a
different scenario entirely, in which the spawn path under test never runs.
They now report not-serving until a spawn has actually happened.
Nothing is wired into the call path yet; VoiceAudioIo still uses JavaSound.
That commit's message describes only the CoreAudio bindings and the feasibility probe, and ends with "nothing is wired into the call path yet, so behaviour is unchanged". Both halves are wrong about what it actually contains. It was staged with `git add compose-ui`, which swept in every uncommitted change in the tree at that moment: - GuiVoiceToolExecutor: the curated tool surface on the LOCAL backend, which is what cut the per-turn prompt from ~4,300 tokens to ~1,700 and roughly halved reply latency. - VoiceToolCatalog: required-first property ordering, and lean schemas that drop argument docs from optional parameters on LOCAL. - HostVoiceCallController: the named-argument instruction for local tool calls, and failure logging. - VoiceAudioIo: the playback-log fix that stopped the echo reference being evicted while still audible - the cause of "develop-p-p-p-p". - LocalVoiceInstall / LocalVoiceRuntime: the MLX 4-bit model on macOS, adopting an already-serving port, and writing the server's output to a file. Every one of those changes behaviour, several of them on the call path. Only the aec/ files match the message. No code is changed here. The commits stand as they are rather than being rewritten, because the branch is pushed and the history is shared; this is the correction that makes the record accurate for anyone reading it later.
Wires Voice Processing I/O into the call path. VoiceProcessingAudioIo implements VoiceAudioIo against the unit built in the previous two commits, and VoiceAudioIoSelection picks it on macOS, with JavaSound everywhere else. This replaces inference with subtraction. VoiceDuplexGate decides which parts of the microphone signal are the agent by comparing a level against a learned attenuation of the speaker reference; the audio unit removes the speaker signal outright, using capture and render sample-aligned in one clock domain. Measured on a real call, the inference held its estimate at 0.10-0.24 against a true coupling near 0.83, so the bar sat at a third of the actual echo and the agent's own voice latched a barge-in on frame after frame. audiblePlaybackLevel returns 0 on this path, and that is a statement rather than a stub: the reference exists to separate echo from speech, and once the echo is gone there is nothing to separate. A non-zero value would only raise the bar against the user's real voice. Zero leaves the gate at its user-level floor, which is correct when the microphone really is just the user. Muting drops the send but leaves the unit running, matching the JavaSound path's reasoning about the platform microphone indicator, and additionally because stopping the unit tears down the canceller mid-call and forces an audible re-convergence on unmute. Every way of answering "no" lands on JavaSound: the setting off, an unsupported platform, or a support check that THROWS - which it can, since it reaches a native library load. That last branch has its own test, because letting it escape would turn "no hardware echo cancellation" into "Boss Calling does not start", a far worse failure than the one being avoided. voiceHardwareEchoCancellation defaults on. The switch exists because this swaps the entire audio device implementation, and a machine whose driver misbehaves needs a way back that is not a rebuild. Full :compose-ui:desktopTest green.
First real call through the OS canceller shredded the agent's voice. The log was a steady flood of "Voice playback ring full; dropped 5120 of 9216 bytes", around 55-60% of every chunk, continuously. Not a burst. A local speech server hands over audio at roughly 3.2x realtime (Qwen3-TTS measured at RTF 3.1-3.35) while the render callback drains at exactly 1x, so in steady state most of it had nowhere to go. The mistake was inherited from the ring's own contract. AudioRing refuses rather than overwrites, which is right where the producer is CoreAudio's realtime thread and must never block. Playback is the opposite case: its producer is a network worker, and JavaSound's line.write() has always blocked there, providing backpressure this path silently removed. So play() now waits for space, bounded at ten seconds so a wedged device cannot park a network thread for the rest of the call, and says so if it ever gives up. The playback ring also grows to eight seconds, since one second overflowed inside a single reply; the capture ring stays at one, where its producer is realtime and its consumer keeps up trivially. Behaviour now matches JavaSound: a slow speaker slows the producer instead of deleting the middle of a sentence.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #384. Evaluation and backend selection are in this comment.
What
Boss Calling can now run on a local speech server instead of OpenAI Realtime. No key, and no audio leaves the machine once the models are on disk.
Settings → Session Sharing → Boss Calling gains a Voice backend picker, and choosing Local adds install / start / stop with live readiness.
The finding this rests on
This is a provider choice, not a second voice stack.
HostVoiceCallControlleris a state machine over the OpenAI Realtime event protocol, so a server that speaks that protocol needs no controller change at all. The genuinely OpenAI-specific parts were a URL constant and a credential requirement.huggingface/speech-to-speech(Apache-2.0, pinned to 1.0.0) fits, verified in its source before any code here was written:/v1/realtimeVoiceAudioIo.FORMAT, no conversionsession.update+function_call_outputround-tripllm_proxy.py: "The server performs no authentication")The event check mattered more than it looks. We use the newer
response.output_audio.*naming; a server on the legacyresponse.audio.*spelling would connect, transcribe, and produce a silent call. It implements both.No fallback between backends, on purpose
If LOCAL is selected and the local server cannot serve, the call fails with a message. It is never quietly re-routed through OpenAI. Doing so would bill the user and put their microphone audio on the network, which is precisely what someone choosing a local backend is avoiding.
The sharpest case is the share surface. A host who switches to local may still have an OpenAI key on disk from before, so the backend check is ordered ahead of the key check: otherwise the share path finds that key and mints a metered session on behalf of someone who opted out. Remote viewers cannot reach host loopback anyway (the viewer negotiates WebRTC with OpenAI directly), so
local_backendis also the honest status. Reverting that one line fails two tests.Notable details
Authorizationentirely when there is no credential, rather than sending a placeholder. Handing a live OpenAI key to an arbitrary local process would be a leak.--host 127.0.0.1is passed explicitly even though it is the server's own default. With no authentication in that server, the bind address is a security boundary, not a preference.GET /v1/pool, not the realtime socket, which would open and abandon a session every 500 ms during startup.Limits, stated plainly
response.doneordering are where a compatible-looking server diverges, and our barge-in depends on a subtle detail already.Tests
:compose-ui:desktopTestpasses. New coverage:VoiceBackendTest(backend decision, including that a local endpoint never carries the user's key and never falls back),LocalVoiceRuntimeTest(command construction, loopback bind, version pin, probe shape, Python floor, state machine), plus twoVoiceCallServiceTestcases for the share guard, both verified to fail when the guard is removed.Follow-up commit
ff1e7956— gaps found in reviewThe review of this PR turned up four things worth closing, plus one correction to my own reasoning.
The start race is real and now fixed. The spawn decision and the spawn itself sat on opposite sides of the process lock, so concurrent callers (a call placed while someone presses Start in Settings, or two calls close together) could both spawn. The loser's handle was overwritten, and nothing could reap it afterwards:
stop()and the shutdown hook only see that one field, and the losing monitor'sprocess !== startedcheck returns without destroying anything — leaving an unauthenticated speech server holding the port. The whole start sequence now runs under aMutex.Getting the test right was most of the work, and the numbers are worth recording. The race is narrow, and an ungated test passed against the pre-fix code (measured
spawns=1), so the injected runner now parks the first caller insidespawn()until a second caller reaches the decision. Verified in both directions: the test fails with theMutexremoved, and again with theStarting-state guard removed. Neither piece is redundant — which corrected my initial review, where I pointed at the guard and treated the mutex as optional.needsLocalSetupwas computed, tested, then dropped on the floor. The resolver set it and the controller discarded it, so pressing Call with the local backend selected and nothing installed produced a message pointing at Settings with no way to get there. It now reachesHostCallStateand draws an Open Settings button; every other failure keeps its single Dismiss.voiceLocalExternalUrlwas untested and unvalidated. Thehttp(s)://address a person copies out of a browser is now translated tows(s)://(what the JDK transport accepts), and anything unusable is rejected against the setting by name instead of falling back to the managed runtime — the one substitution this backend must never make quietly. A non-loopback address warns in Settings, since it is the one configuration that contradicts "no audio leaves the device".Also: platform detection goes through
ShellCustomizationUtils, the Python floor is named once instead of hardcoded beside the check, anddispose()'s role is described accurately.Compatibility re-verified against the pinned artifact
The claims below were originally read in the upstream source. They have now been checked against the exact published
speech-to-speech==1.0.0wheel from PyPI:speech-to-speech1.0.0 exists,requires_python >=3.10, Apache-2.0, repohuggingface/speech-to-speech[console_scripts] speech-to-speech = speech_to_speech.cli:mainservesubcommand accepts--host/--portHfArgumentParserderives them fromRealtimeServerArgumentsdataclass fields, so the spelling matches exactlyTerminalSettings.voiceLocalPortGET /v1/poolis a real route@app.get("/v1/pool")— and it is a plain GET, so the probe does not allocate a session/v1/realtimeWebSocket@app.websocket("/v1/realtime")response.output_audio.*namingresponse.output_audio.delta/.done— the exact spelling BossTerm expects, so the silent-call risk is retired{"type": "audio/pcm", "rate": 24000}; identical toVoiceAudioIo.FORMATsession.updateis a handled event type, andfunction_call_outputis implemented with call-id pairingOne notable independent confirmation of the bind-address decision: upstream's own help text for
--hostreads "Pass 0.0.0.0 explicitly to expose the unauthenticated API on the network." Explicitly passing127.0.0.1is therefore not merely defensive — it is the documented safe path.Still unchanged: none of this is a placed call. The runtime risks in "Limits, stated plainly" above remain open, and only a real call can close them.