Skip to content

feat(kiosk): audio on a display: radio stations (incl. operator-ticked home-network streams) through a pinned stream relay, display tickets, transport verbs, ducking, now playing; voice turn: optional STT prompt/hotwords, repetition-loop and tool-syntax guards; kiosk 0.3.6 - #443

Merged
kh0pper merged 39 commits into
mainfrom
feat/kiosk-wm1b1
Oct 8, 2026

Conversation

@kh0pper

@kh0pper kh0pper commented Oct 5, 2026 •

Copy link
Copy Markdown
Owner

What this adds (kiosk 0.3.6)

"Play " on a paired display plays the stream through the page's own <audio> element with no model call. "Pause", "keep playing", "louder", "what's playing" and "stop" act at once with no model. A question asked over music ducks it (or, on a phone or tablet, pauses it) and brings it back. The display never receives an upstream URL or a credential.

Merge only inside a deploy window. This PR must not be merged outside an attended deploy window for the kiosk (it also moves stored volume caps once at boot; see "Volume").

  • Display tickets (server/tickets.js): one stream for one display under /display/t/<id>/stream, behind the display network gate (not a Funnel prefix). A ticket is bound to one device and one resource. At most 8 per device and 2 concurrent requests. Tickets last 12 h, and a timer aborts open requests at expiry. GET only, no-store, never logged.
  • Stream relay (server/relay.js, server/netscope.js): each upstream carries a hop policy. On every hop the name is resolved once, every resolved address is checked, and the connection goes to the address that was checked (pinned through the request's lookup). A credential is sent on the first hop only, and https→http is refused when a credential was sent. At most 3 redirects. 10 s to headers, 30 s body idle. Only audio content types pass, and only one Range is passed through.
  • Media session (server/media.js, session.js): queue, state, volume with a per-display cap. A station that drops after it has played is reloaded up to three times. Every hello carries the media state, and every turn_start gets exactly one closing frame (turn_done or turn_over). Two optional device settings, max_volume and pause_media_on_listen; no default ever writes them.
  • Sources and stations (server/sources/): a source contract (contract 1) and its first adapter, operator-entered station presets with one canonical spoken form. The repo ships no presets.
  • Play and transport verbs (play.js, executor.js, patterns.js, phrases.js, strings.js en/es): a confident station match plays at T1. "Which one?" is remembered for 30 s. The model's next is the playback verb, and "close everything" also stops audio.
  • Runtime and admin (runtime.js): station admin routes behind the dashboard session and CSRF. One invalid row refuses the whole save. Unpairing a display, or the end of the login behind a session display, ends its stream and revokes its tickets. When the instance has a station, the turn check adds a fourth sentence, which must start playback with no model.
  • Panel: a station editor with a per-row Home network tick, plus loudest volume and pause-while-listening per display.
  • Page: one <audio> element and a now-playing chip. Ducking follows the mic, the bird and the speech, with a 30 s backstop. A watchdog catches streams that never start or that stall. A blocked autoplay waits for a tap. New nowplaying window.
  • Docs en/es; kiosk 0.3.6. No funkwhale change in this PR.

Core changes (voice turn) — this PR is not kiosk-only

servers/gateway/voice/turn.js, servers/gateway/voice/turn-helpers.js, servers/gateway/ai/stt/adapters/fasterwhisper.js:

  • Optional STT hints. opts.sttPrompt (the OpenAI-style prompt, ≤ 200 characters) and opts.sttHotwords (faster-whisper's hotwords, ≤ 6 words), each a string or a function of the STT profile; the faster-whisper adapter sends both fields. Callers that pass neither are unchanged. The kiosk sends no prompt (a station-name prompt made the speech model spell letters, repeat itself and mishear short commands), and its hotwords are a switch that is OFF by default (CROW_KIOSK_STT_HOTWORDS=1), used only when both the display and the STT profile are English, from the operator's own all-capitals call signs.
  • Repetition-loop guard. A transcript that is an STT repetition loop (the same word 8 times running; 6 times when that run is most of a line of 40+ characters; a 1–3 character unit 12 times inside a non-numeric word) ends the turn as an empty transcript, before any fast path or model. Ordinary repeated speech ("no, no, no", "sí, sí", laughter, a title, long numbers) is kept.
  • Tool-syntax guard. Tool-call syntax a model writes as text (Qwen/Hermes XML tags, [TOOL_CALLS], <|python_tag|>, <function_call>, a bare or pretty-printed JSON call, a fenced json block; any case; split across deltas) is never captioned, spoken or kept in the history. Plain prose that quotes a {"name": …} object is cut too (accepted for speech).
  • Both guards apply to every caller of runVoiceTurn: the kiosk AND the glasses endpoint. The companion runs its own loop and is not covered.
  • Counts-only timings: verb (a fixed word on a no-model turn), stt_degenerate, tool_text, spoken_chars, stt_hotwords (true/false; never the words). Never transcript text.

Changed since the last push (third attended smoke and its review)

  • Pause while paused. With the music already paused, every pause-family request ("Pause.", "Paws.", "Pause it.", "Pause the music.", "Pause. Thank you.") is answered with no model, quietly, and the paused item is sent to the page again; it never resumes.
  • The sentence's own command wins. When a sentence reads as one playback command with no model ("Pause.", "Louder."), a different playback verb chosen by the model never runs; the sentence's verb runs instead. Turn metrics name the verb the model asked for (model_verbs) and the one that ran instead (model_verbs_kept), as fixed words.
  • Volume. 0–100 now spans −50…0 dB on the element (0 is mute). Each "louder"/"quieter" (or window button) moves 10 dB; "quieter" stops at a quiet but audible floor (10, −45 dB), and only "volume zero" or "mute" silences. A spoken level is that point on the scale ("volume 50" ≈ −25 dB). A new session starts at 80 (−10 dB). Stored "Loudest volume" caps move once at boot to the level that is as loud or quieter (linear 100→100, 90–60→90, 50/40→80, 30/20→70, 10→60), and are marked so the move never repeats; the move is lossy, so a rollback must restore the stored device list from a backup taken before the deploy.
  • Speech-to-text forms of short commands ("Louders.", "Quiter.", "Pons.", a trailing "thank you"/"thanks") act like the words, only as the whole utterance and only while something plays.
  • STT hotwords stay off by default (the switch remains for operators).
  • Wired tests destroy their server's open sockets on close, so a failing test ends the run instead of hanging it.

Earlier fixes (second attended smoke and two reviews)

  • A spoken reply always settles. The page's player ends a reply whose audio never finishes (only once its audio clock has stopped moving; a late Bluetooth sink is never cut; capped at twice its length + 5 s) and drops a decode that never answers. On a phone the mic is let go only once the turn is done and nothing plays (never between two sentences, and before the music comes back), so the call/media audio-mode switch no longer lands in the reply. Turn metrics add tts_stalled, tts_audio_ms, stall_ctx_state, stall_clock_moved, ctx_state.
  • Station names by voice. No STT prompt (above). A call sign one sound off ("Play BTPF" for KTPF-style names) is asked about ("Did you mean …?") with no model, never played. With radio presets, a station asked for in words the STT mangled still offers the play tool (never forces it); interjections without a vowel do not count.
  • Pause / Play. The single words "Pause." / "Play." with something loaded are answered with no model, and the whole current item is sent to the page again, so a page that shows another state, another item or none is put right. "Play." with nothing loaded answers "Nothing is paused. What would you like to hear?" (es too); while a choice is pending it answers it instead (one suggestion: plays it; several: asks again by name). Turn metrics carry the server's media state before/after (media_before, media_after) and the page's own (page_media_start, page_media_end).
  • "Already playing" re-sends the current item, so it is true where it is heard.

Known gaps

  • Some playback requests still go to the model unforced, so a claim is still possible: "Pause the music, I have a call", "The kids are too loud, turn it up" (a statement), "No, louder" (a negation). In "Play jazz and turn it up" only the first must-run tool is enforced.
  • iPhones keep the open mic (pause-while-listening still applies) until an iPhone is checked. On Android the spoken answer plays while the phone is still in call audio mode (the mic is released after it).
  • The cause of the replies that did not settle on a phone is an inference (an Android audio-mode switch stalling the page's audio clock); the metrics are there to confirm or refute it.
  • The loop guard does not catch phrase loops ("Thank you. Thank you. …") and still drops an 8× repeated word. A short reply on a sink that wakes later than its length + 2 s is cut.
  • The whisper.cpp STT adapter sends neither field.
  • The human-voice rates on a real phone are measured in the attended smoke, not here.

Home-network ("local stream") station policy

A station may point at a stream on the home network or the tailnet only when the operator ticks Home network for that row. The tick is off by default and can be set only through the admin route (dashboard session + CSRF). The model never supplies a URL.

On every hop of every station, ticked or not: an address of this host itself is refused (own_address); any address inside the on-link prefix of any interface of this host is refused (private_address), with the interface table and default-route interfaces read again on every hop (fail closed); address classes come from the shared classifier (servers/shared/ip-classify.js), and the relay additionally refuses an IPv4 embedded in a SIIT, IPv4-compatible, NAT64, 6to4 or Teredo form even when it is public.

For a ticked station the entered host:port may resolve only to an address inside a prefix of the default-route interface (RFC 1918, CGNAT, ULA, or that LAN's own global IPv6 prefix; never the network or broadcast address) or the tailnet ranges (100.64.0.0/10, fd7a:115c:a1e0::/48) when not inside another interface's prefix. Refused even with the tick: other interfaces' prefixes, an RFC 1918 address on no interface, loopback, link-local (metadata included), multicast, broadcast, reserved, embedded or translated IPv4, IPv6 site-local, this host, and any public answer for a ticked host. Pinning: the server resolves the ticked host once at save, judges every address and stores the set; every play and same-host redirect must resolve inside it (address_changed otherwise). Redirects from a ticked station may go only to the same host:port (http→https allowed) or to a public address under the public rule; no credential is ever sent on this policy.

The relay's public rule also refuses any address the shared classifier counts as one of this host's networks (isOwnNetworkIp), at netscope.js judgeAddress, on the public rule only; a guard test keeps the production predicate address-only (kiosk-relay).

How this meets the display tools on main

Tests

  • On current main (54e682d5). Full suite on this head, scratch env: 7,918 tests, 7,917 pass, 0 fail, 1 skipped.
  • build-registry --check OK, check-public-hygiene clean.

Not run here: anything on a live host (evaluation, attended smoke, deploy window). Those happen separately before merge.

@kh0pper
kh0pper force-pushed the feat/kiosk-wm1a branch 3 times, most recently from 3849872 to ec606c2 Compare October 6, 2026 02:02
@kh0pper
kh0pper changed the base branch from feat/kiosk-wm1a to main October 6, 2026 17:55
@kh0pper kh0pper closed this Oct 6, 2026
@kh0pper kh0pper reopened this Oct 6, 2026
@kh0pper kh0pper changed the title feat(kiosk): audio on a display: radio stations (incl. operator-ticked home-network streams) through a pinned stream relay, display tickets, transport verbs, ducking, now playing; kiosk 0.3.0 feat(kiosk): audio on a display: radio stations (incl. operator-ticked home-network streams) through a pinned stream relay, display tickets, transport verbs, ducking, now playing; optional STT prompt in the voice turn; kiosk 0.3.2 Oct 6, 2026
kh0pper added 17 commits October 7, 2026 12:44
…ved address checked from this host's view (its own addresses and every interface's on-link prefix are never public), the dialled address is the checked one, credentials on the first hop only, bounded redirects and timeouts, GET only, audio content types only
…ay/t, bound to a device and a resource, at most two concurrent requests, cut at expiry by a timer, never logged
… per display, a ticket per item, station reloads after a drop); media state on every hello; every turn_start gets exactly one closing frame; effect_ms in turn metrics; optional max_volume and pause_media_on_listen device settings
… station presets, with one canonical form for how a station is said; a home-network station needs an explicit per-station tick
…on plays at T1 with no model, 'Which one?' is remembered for 30 s, transport phrases fire only when they would change something and only in forms that name the music, the model's next is the playback verb, close everything also stops audio
…play turns; the ticket mount behind the display network gate; station admin routes behind the dashboard session (one invalid row refuses the save, a ticked row is checked by the server and its address set recorded, Test reads headers only); the turn check plays when there is a station
…fault; Save checks every ticked row first), loudest volume and pause-while-listening per display
…e now-playing window, ducking that follows the mic, the bird and the speech with a 30 s backstop, a watchdog for streams that never start or stall, a failure the socket could not carry is sent after reconnect
…tickets and the relay, transport, ducking, now playing); kiosk 0.3.0
…ware shared rule is marked for the public rule alone; a guard test: a ticked station still reaches the LAN's own global IPv6 /64 when the shared classifier knows this host's interfaces
… this host's networks as the shared classifier sees them (isOwnNetworkIp); the public test stays address-only, so a ticked station still reaches the LAN's own global IPv6 /64
…odel's next alone (it is the playback verb and never touches a window), so with nothing playing it says so; the panel test counts the theme select; the play test steps a recipe with words that ask for it
…audio defaults, page/server agreement, an offline chip, an automatic now-playing window; kiosk 0.3.2

- Station names by voice: the display's station names (and aliases) bias the STT through an optional, bounded prompt, and a call sign that STT spelled by its sounds finds the one station that sounds the same; two that sound alike are asked about.
- A playback word with filler ("louder louder", "a bit quieter please", "skip it") acts at once while something plays; a turn that asks the playback to change must end with a real call or the truthful "Sorry, I couldn't change the playback."
- Phones and tablets pause music while listening by default and let the mic go after each turn (Android leaves call audio mode); the reopen cost is measured per turn.
- The page never reports a pause or a block the media element did not make; a no-model turn records its verb in the metrics line.
- An offline display says so: the chip dims and does nothing, the window's buttons are off.
- The now-playing window opens by itself when playback starts on a display with a screen (one, reused, behind a card just asked for); the chip opens it.

Core change: the voice turn takes an optional STT prompt (opts.sttPrompt, bounded and cleaned) and records timings.verb on a no-model turn; the faster-whisper STT adapter sends the prompt.
…pen at a time; sound-alike stations are asked about; the now-playing window keeps out of the way; 60 s offline grace

- A playback word counts only when it acts on the music (nothing after it, a pronoun or a music word); another object (lights, heat, a timer, the TV) or the voice is never forced onto the playback tool, and crow_wm is forced only over music or when the music is named.
- "Skip this one." skips at once while something plays; a bare "Stop." with nothing playing answers "Nothing is playing." at once.
- A resume the browser refuses after pause-while-listening is reported as blocked again (chip shows play, never pause over silence).
- One microphone open at a time: taps during an open share it, a second tap is ignored and the bird shows the first was heard; an open released meanwhile is closed when it lands.
- A station that only sounds like ordinary words is asked about ("Did you mean …? Say yes or its name."); only a call-sign shape plays at once.
- The now-playing window comes to the front by itself only on an empty screen; otherwise it opens behind what is open.
- An offline page keeps its audio for 60 s (the server's grace plus two pings) and reconnects every few seconds while audio is live.
- With follow-up on, a phone keeps its mic through the follow-up and lets it go when the conversation settles; the per-turn release is Android-only until an iPhone is checked.
- No English station prompt on a Spanish display that detects its language; a station name that is itself a command is refused; the docs say a cloud STT receives the names.
…ry word is not a call sign; "Did you mean …?" lives one utterance; command-word station names are refused at the save

- "Pon la música más alta", "ponla más fuerte", "más bajito" ask the playback to change (crow_wm, never crow_play); the TV or the heating before them do not.
- A single ordinary word counts as a call sign only with four consonant sounds or more ("Play Cats." is asked about, never played).
- "Did you mean …?" is answered by the next utterance only; "No." clears it at once with "Okay."; a later yes never starts the radio.
- A command-word station name is refused at the save with the word; one already saved keeps loading, stays out of the speech hint, and the panel names it.
…hose audio never finishes (bounded by its own length plus a grace) and drops a decode that never answers; a phone lets the mic go when the turn settles instead of at the end of speech, so the audio-mode switch never lands in the reply; the turn report says whether the reply stalled and the audio context's state
…odel spell letters, loop and mishear short commands; only the call-sign-shaped words go to faster-whisper as hotwords; a call sign one sound off is asked about instead of going to the model; a transcript that is a repetition loop ends the turn as an empty one, before any fast path or model
kh0pper added 14 commits October 7, 2026 12:44
…red with no model even when nothing would change — the state is sent to the page again, so a page that disagrees is put right and 'Play.' never starts something else; every display turn records the server's media state before and after it
…d, spoken or kept; with radio presets a station asked for in words speech-to-text mangled still offers the play tool; 'already playing' re-sends the play to the page so it is true where it is heard
…n the element instead of a tenth of its linear gain, and a new session starts two steps below full
…nd-off call sign is asked about, repetition loops end the turn, bare Pause/Play re-send the state, 5 dB volume steps, the phone mic goes after the reply; kiosk 0.3.4
… plays (never on a pause between two sentences) and before the music comes back; a reply is cut only when its audio clock has stopped, never while a late speaker is still playing it; the metrics say how long a reply was and what the audio did at a stall
…ly when the display and the speech profile are both English, and taken from the operator's own all-capitals call signs
…ole current item again, so a page that shows another item, none, or the other state is put right; the turn report names the page's own media state at the tap and at the end
…repeated word in Spanish or English, laughter, a title, long numbers) and still drops the long loops; tool-call syntax in the other common formats is never heard either
…Nothing is paused. What would you like to hear?') instead of going to the model; a timer that has gone off still comes first; interjections without a vowel are not taken for call signs
…oot, to the step on the new volume curve that is as loud as it was (never louder); caps saved from now on are marked as on the new scale so the move never repeats
… loops only, Play with nothing loaded, the loudest-volume move
… suggested station plays at once, several are asked again by name
…sent (a true/false flag, never the words), the early transcript included
…d a fenced json block, are never heard either
@kh0pper kh0pper changed the title feat(kiosk): audio on a display: radio stations (incl. operator-ticked home-network streams) through a pinned stream relay, display tickets, transport verbs, ducking, now playing; optional STT prompt in the voice turn; kiosk 0.3.2 feat(kiosk): audio on a display: radio stations (incl. operator-ticked home-network streams) through a pinned stream relay, display tickets, transport verbs, ducking, now playing; voice turn: optional STT prompt/hotwords, repetition-loop and tool-syntax guards; kiosk 0.3.4 Oct 7, 2026
kh0pper added 8 commits October 7, 2026 17:22
…ered with no model in every form; a model's playback call never runs against the sentence's own command (the sentence's verb runs instead); the turn metrics name the playback verb the model asked for; more speech-to-text forms of the short commands are recognised
… a phone); a new session starts one step below full; stored loudest-volume caps move once to the step that is as loud or quieter, from the old linear scale or the previous 5 dB one
…uieter moves 10 dB; Quieter stops at a quiet but audible floor; a spoken level is that point on the scale; stored caps move once to the equally-or-less loud level, from the linear scale or the previous 10 dB one
…a short command; anywhere else it is part of the sentence
…so a failing test ends the run instead of hanging it
@kh0pper kh0pper changed the title feat(kiosk): audio on a display: radio stations (incl. operator-ticked home-network streams) through a pinned stream relay, display tickets, transport verbs, ducking, now playing; voice turn: optional STT prompt/hotwords, repetition-loop and tool-syntax guards; kiosk 0.3.4 feat(kiosk): audio on a display: radio stations (incl. operator-ticked home-network streams) through a pinned stream relay, display tickets, transport verbs, ducking, now playing; voice turn: optional STT prompt/hotwords, repetition-loop and tool-syntax guards; kiosk 0.3.6 Oct 7, 2026
@kh0pper
kh0pper merged commit f65848a into main Oct 8, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant