Skip to content

feat(health): read-only Health Skin — grain flow, system, pipeline writers - #41

Draft
guycochran wants to merge 37 commits into
masterfrom
feat/health-skin
Draft

feat(health): read-only Health Skin — grain flow, system, pipeline writers#41
guycochran wants to merge 37 commits into
masterfrom
feat/health-skin

Conversation

@guycochran

@guycochran guycochran commented Oct 9, 2026

Copy link
Copy Markdown
Owner

What

A second Skin over the same Core: a read-only facility monitor at /health, backed by GET /api/mxl/health. Brings back the grain/system visibility from the original prodbots/IBC health board — on the new commodity box — without putting any of it in the control Skin.

Core stays lean (cut/arm/take). Health is its own page + its own read-only endpoint. An engineer can watch grains; a TD can't accidentally touch the program bus from it.

Sections

  • System — CPU %, memory, load (1/5/15m), CPU count, live WebRTC viewers.
  • Grain flow — per flow: grains/s and unique-fps. frozen = grains tick at rate but unique-fps collapsed (repeat-wedge — the exact keyer symptom we hit live). Also dead / slow / ok.
  • Pipeline writers — selector / keyer / encoder: running, error, active input, format, key-on, viewers, bitrate.

Why it won't strain the $400 box

The expensive part (grain + unique-fps measurement) is already tools/grain_probe.py's job — long-lived readers writing grains.json every 3s, with hard load guards (pause >26, exit >29). The endpoint only reads that file + /proc + the three pipeline/status endpoints, and caches the aggregate ~2s so N open tabs = one scrape. No docker exec, no probe spawning, live PGM path untouched.

Tests

  • tests/js/health-info.test.js — 14 unit tests (frozen/dead/slow classification, CPU/mem/load parsing, pipeline shaping). Pure module, no network.
  • tests/test_local_ui.py — +5: air-gap (self-hosted fonts, no external hosts), read-only (no POST/PUT/DELETE on health), and separation from Core (local.html must not poll /health).
  • Full suite green: 50 node / 139 python.

Not in this PR (separate, authorized HW step)

Per the repo rules, this branch does not touch the live system. To light up the Grain-flow section on the box, tools/grain_probe.py must be running (facility bring-up / its runner). Verifying the page live is a separate HW step.

Docs

docs/HEALTH-SKIN.md.

🤖 Generated with Claude Code

guycochran and others added 3 commits October 9, 2026 01:02
@guycochran @claude
feat(health): read-only Health Skin — grain flow (+unique-fps), syste…
26f4623 
…m, pipeline writers

Second Skin over the same Core, deliberately separate from the control Skin:
- GET /api/mxl/health + GET /health (read-only; no mutation path)
- backend/health-info.js: pure aggregation — classifies flows ok/slow/frozen/dead.
  'frozen' = grains tick at rate but unique-fps collapsed (repeat-wedge detection).
- Reuses tools/grain_probe.py's grains.json snapshot (already load-guarded) + /proc
  + the 3 pipeline/status endpoints; 2s server cache so N viewers = one scrape.
  No docker exec, no probe spawning, live PGM path untouched.
- web/health.html: air-gapped (self-hosted fonts), 3s poll, Skin-styled.
- Tests: 14 node unit tests + 5 python (air-gap, read-only, separation-from-Core).
  Full suite green (50 node / 139 py).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
feat(health): read live thumbs health.json (already-running probe); g…
add96a2 
…rains.json fallback

Live HW verify on the $400 box showed tools/mxl_thumbs.py ALREADY writes health.json
beside the thumbnails (system + per-slot {age,frozen} + viewers) — same frozen-reader
detection as grain_probe, but already running. So the Health Skin reads THAT (zero new
process, no second set of MXL readers), and falls back to grain_probe's grains.json for
adopters who run it instead.

- classifyThumbsHealth(): {age,frozen} -> ok/frozen/dead; folds in load/mem/cpus/viewers.
- endpoint prefers health.json, uses the probe's host-true system numbers when Core runs
  off-box; UI renders either shape (Grains/s+Unique-fps OR Last-frame+Content-age).
- staleAfterS=25 to match the thumbs health writer's ~15s cadence.

Verified live (read-only, no box changes): /health -> 200, real payload — load 15.6,
24 cpus, mem 18%, pattern/playout/guest1/guest2 OK, guest3/layout DEAD, selector running
active_input#3 1920x1080@30, encoder 6000kbps. (It even surfaced the live 60fps-restamp
'Out of range' selector error — the monitor earning its keep.)

Tests: 55 node / 139 python, all green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran
docs(health): reflect HW finding — reads already-running thumbs healt…
724af05 
…h.json, grain_probe optional
@guycochran

guycochran commented Oct 9, 2026

Copy link
Copy Markdown
Owner Author

✅ Verified live on the $400 box (read-only, no box changes)

Ran the real endpoint against the live facility and rendered /health in a browser:

Key finding that shaped the final design: tools/mxl_thumbs.py is already running on the box and already writes health.json (system + per-slot {age, frozen} + viewers) — the same frozen-reader detection as grain_probe. So the Skin now reads that (zero new process, no second set of MXL readers), with grains.json as the adopter fallback. This directly answers the 'will it strain the $400 box' question: it adds no measurable load.

The page even surfaced a real live error (selector: read source grain … Out of range - too late — the known 60fps-restamp bug), so the monitor is already earning its keep.

Tests: 55 node / 139 python green. Still draft for review.

guycochran and others added 3 commits October 9, 2026 01:17
@guycochran @claude
feat(health): read probe snapshot over MXL_THUMBS_ORIGIN (proxy mode)…
b508360 
…, not just file

On a box where the MXL domain is only mounted inside containers, the thumbnails (and
health.json/grains.json beside them) are exposed via a small http server
(MXL_THUMBS_ORIGIN, e.g. :8086/thumbs) — same as the /api/mxl/thumbs route already
does. The health endpoint now fetches the snapshot over that origin when set, falling
back to a local file read otherwise. Keeps Core off the Docker socket + portable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
feat(health): CORS allow-origin on /api/mxl/health so the public site…
8ef61ff 
… can embed live numbers

Read-only public telemetry (GET, no secrets, no control) — mxlswitcher.com fetches it
to show the grain explainer's live data. Default '*', override via MXL_HEALTH_CORS.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
site: 'What's a grain?' layman explainer with LIVE data from the $400…
97598aa 
… box

Plain-English page (grain=one frame, flow=one source, shared memory=one copy everyone
reads) that embeds the switcher's real /api/mxl/health — live source states + 'what the
$400 box is doing right now' (load/CPU/mem). Linked first in the front-door learn cards.
Pulls from the public mxl-skin.cochran.cloud health endpoint (CORS-enabled, read-only).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran

guycochran commented Oct 9, 2026

Copy link
Copy Markdown
Owner Author

🚀 Deployed live + public explainer added

Deployed to the $400 box (cherry-picked onto the running branch, box-specific facility.json preserved, mxl-core restarted). Verified end-to-end:

  • Dashboard: https://mxl-skin.cochran.cloud/health → 200, live
  • API (CORS-enabled): /api/mxl/health → 200 from mxlswitcher.com origin
  • Needed two deploy-time fixes, both committed: read the probe snapshot over MXL_THUMBS_ORIGIN (the box exposes the domain via an http thumbs server, not a host file mount), and CORS on the read-only endpoint so the public site can embed it.

Public layman explainer: https://mxlswitcher.com/what-are-grains.html — 'What's a grain?' in plain English (grain = one frame, flow = one source, shared memory = one copy everyone reads), embedding the switcher's real live data: per-source state + 'what the $400 box is doing right now' (load/CPU/mem). Linked first in the front-door learn cards. This is the artifact for showing Jonas & co. what commodity hardware actually does under a full production.

Live sample while deploying: load ~12 / 24 cores, CPU ~58%, mem 17%, 4 live sources — a full control room with headroom.

Still draft — code's on this branch for review.

@guycochran @claude
feat(health): real viewer count from mediamtx API (MTX_API=yes), not …
5c322b1 
…the WHIP encoder

The encoder publishes WHIP and never sees WebRTC clients, so it always reports viewers=0.
The actual viewers connect to mediamtx — enable its API (:9997) and count read sessions
on the program path. Health endpoint prefers that live count; encoder card reflects it too.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran

guycochran commented Oct 9, 2026

Copy link
Copy Markdown
Owner Author

Fixed: real viewer count (was always 0)

Root cause: the WHIP encoder (mxl2webrtc) only publishes — it never sees the WebRTC clients, which connect to mediamtx directly. So encoder.viewers was structurally always 0.

Fix: enabled mediamtx's control API (MTX_API=yes, :9997) and the health endpoint now counts live read sessions on the program path via /v3/webrtcsessions/list, excluding the publisher. Verified live: 3 mediamtx sessions (1 publish + 2 read) → health reports viewers: 2. ✅

⚠️ Deploy note: enabling MTX_API required restarting mediamtx, which orphaned the encoder's WHIP publish — mxl-pgm-heal fixed it in one shot (republished + verified PGM advancing). Worth noting for the 'open it to the public' plan: a mediamtx restart needs a pgm-heal afterward.

guycochran and others added 5 commits October 9, 2026 02:04
@guycochran @claude
fix(cut): a down/bypassed keyer must never fail a cut or status
ceb012a 
The cut is the selector's active-input move; the keyer self-heal after it is best-effort.
Before, the cut path awaited mxlApi(9605) unconditionally, so a stopped keyer (unreachable
or 400) sank an otherwise-successful cut with ':9605 -> 400' — exactly what broke cutting
after the keyer went down. Now: skip the keyer entirely when MXL_KEYER_BYPASS=1, and
otherwise wrap the self-heal in try/catch so a down keyer only logs. mxlStatus() also
defaults keyer+testgen to {} so the Skin's 1s status poll survives them being down.

Robustness for the 'open it to the public' plan: switching can't break on the keyer.
+3 node tests (cut-with-keyer-down, bypass-never-pokes-keyer, status-survives-down-keyer).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran
test: allowlist mxlswitcher.com in local-server.js CORS comment (comm…
1cbf57a 
…ent-only, like quickstart.sh)
@guycochran @claude
feat(join): public phone-inject page (QR + Larix + live preview) with…
0142872 
… configurable guests/ports

The 'open it up' flow: a visitor opens /join, scans a QR with Larix, and their phone
becomes a switchable guest source. Built for a public demo.

- web/join.html: big QR, 3-step Larix instructions, app-store links, manual SRT settings,
  live program preview + a live source/viewer counter (reads /api/mxl/health). Air-gapped
  (self-hosted fonts; only the two app-store install links are external).
- ingest-info.js: MXL_GUESTS (advertise exactly the slots you run, e.g. guest1,guest2,guest3)
  + MXL_GUEST_PORTS (pin NON-contiguous real ports). Fixes the demo-breaker: the box's
  guest3 listens on :8895, but the computed scheme advertised :8892 (squatted by mediamtx)
  — a wrong QR = a dead scan. Now the QR always matches the real listener.
- QR route validates against configured slots (INGEST_SLOTS), not a hardcoded guest[12].
- /join route added. +8 node tests, +2 python tests. Full suite green (62 node / 141 py).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
feat(public): /welcome landing (Join/Watch/Monitor) + TD hand-off lin…
eda19b9 
…k to /welcome,/join

Completes the public-demo flow with a safe model:
- /welcome — public landing routing visitors to Join / Watch program / Health monitor,
  with a live 'N live sources · M watching · box at X% CPU' line. Never shows the
  token-gated control Skin.
- control Skin's 'Add your camera' modal now links /welcome + /join for TD hand-off.
Safe by construction: /join + /welcome + /program + /health are public & read-only or
contribute-only; cut/take stay token-gated at '/'. +1 python test. Suite green (62/142).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
site: add 'Try it LIVE' hero button on mxlswitcher.com -> live /welco…
6ff362b 
…me demo (put your phone on air)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran

guycochran commented Oct 9, 2026

Copy link
Copy Markdown
Owner Author

🎬 Public phone-inject + Skin flow — BUILT & LIVE

The 'open it up like IBC' flow is live on the $400 box:

Public entry points (all deployed + Playwright-verified):

  • /welcome — landing: Join / Watch / Monitor, with a live "N live sources · M watching · box at X% CPU" line
  • /join — the star: big QR, 3-step Larix instructions, app-store links, manual SRT settings, live program preview + live source counter. Scan → your phone is a switchable source.
  • /program/ — public watch (WebRTC)
  • /health — public read-only monitor

Safe-by-construction model: /welcome, /join, /program, /health are public (read-only or contribute-only). Cut/take stay token-gated at / so visitors can't touch the show.

Demo-breaker fixed: the box's guest3 listens on :8895, but the computed scheme advertised :8892 (squatted by mediamtx) → a scanned QR would've hit a dead port. Added MXL_GUESTS + MXL_GUEST_PORTS so the QR/SRT URL always matches the real listener. Verified: guest3 QR → srt://…:8895. Also installed segno on the box (QR gen was silently 501 before — never worked).

mxlswitcher.com now has a "▶ Try it LIVE — put your phone on air" hero button → the live /welcome.

Tests: 62 node / 142 python, all green. This branch (PR #41) now carries: Health Skin, viewer-count fix, keyer-tolerant cuts, configurable guests/ports, and the full public join flow.

guycochran and others added 9 commits October 9, 2026 04:40
@guycochran @claude
feat(join): multiview return grid + auto-assign a FREE slot (fix 'my …
5923faa 
…feed didn't appear')

Why the phone didn't show: /join sent everyone to guest3 (:8895), but the PTZ already
held that listener (SRT listen mode = one publisher per port) → the phone was rejected.

- /api/mxl/ingest now annotates each slot free/occupied (from the probe snapshot) and
  returns suggested_slot = first FREE slot. /join uses it, so two scanners don't collide.
- /join now shows a live MULTIVIEW return grid (source thumbnails, near-zero cost — the
  switcher already generates them): the visitor's OWN tile is ringed cyan + 'YOU' and the
  status flips to 'You're LIVE 🎥' the moment their source delivers frames. Plus the PGM
  preview below. This is the 'can we see you?' feedback — answers the design question:
  show the MULTIVIEW as the return (instant self-confirmation), PGM as 'what's on air'.
- A true encoded multiview (mv_encode.py exists) stays the TD's pop-out option — overkill
  for /join; the thumbnail grid scales to unlimited join viewers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
feat(join): multiview grid shows all guest slots dynamically (guest4-…
5053bad 
…6 + future)

The return-feed grid now orders known sources then appends any others the health
reports, and rebuilds when the source set changes — so newly-added guest slots
(guest4, guest5, …) appear automatically without a code change. Pairs with the box
now running 6 guest ingest slots for public joiners.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
feat(skin): add Health + Join links to the Core top nav (read-only, f…
eedcfa2 
…or everyone)

Health is read-only telemetry useful to anyone watching the rack — surface it from the
control Skin's header, alongside a Join link for quick QR hand-off.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
feat(reliability): guest-leg-watchdog — heal wedged SRT ingest legs s…
bfae7d0 
…o phone joins always work

The silent demo-killer (HW Oct 9): a guest ingest chain is fanout(public SRT) + video-core
+ audio-core; if EITHER internal leg listener dies, the fanout rejects the whole incoming
connection and the phone gets 'Input/output error' with no clue why. Found guest1 AND
guest2 audio legs dead, both blocking joins.

tools/guest-leg-watchdog.sh polls each guest's 3 ports every 20s; a partial chain (1-2 of
3 up) = a wedged leg → restart the chain in dependency order (audio→core→fanout) with a
per-chain cooldown. A fully-down chain = not provisioned → skipped (guest4-6 optional).
+6 tests. Full suite green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
fix(watchdog): require STRIKES consecutive degraded polls before heal…
2ca4ac0 
…ing (ignore idle-cycle blips)

An idle guest (no publisher) cycles its audio/video core — the SRT src exits 'not-linked'
each loop and the leg listener briefly unbinds between cycles. That's NORMAL, not a wedge.
The first watchdog healed on a single partial poll → it fought the idle cycle. Now a leg
must stay partial across STRIKES (default 3 → ~60s) consecutive polls before a heal, and
the strike counter resets on any healthy/fully-down poll. Cooldown raised to 90s.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
fix(watchdog): give up after GIVE_UP heals that don't stick (chronic …
763f303 
…flap = stop thrashing)

guest1's audio leg flaps 'not-linked' because the Makito sends video-only — its audio core
cycles forever and a restart can't fix it. Without a cap the watchdog would restart guest1
endlessly. Now: after GIVE_UP (4) heals that don't make it healthy, log + leave it alone;
a chain that reaches 3/3 resets its count so genuine transient wedges always get healed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
fix(watchdog): rewrite conservative — fanout-only, never touch live-v…
003a7cf 
…ideo chains, OFF by default

HARD LESSON (HW Oct 9): the first watchdog restarted whole guest chains for leg flaps. A
chain restart recreates MXL flows → the input-selector re-wires → doing that every few
seconds CHURNED the selector (router thread restarting in a loop) → PGM stuck, cuts failed,
multiview flapping, and the selector API went unresponsive (HTTP 000). The watchdog built
to help was breaking the facility.

Rewrite is minimal + safe:
- watches ONLY the public SRT listener (what a contributor connects to)
- NEVER restarts a chain whose video flow is advancing (live source = hands off)
- heal restarts ONLY the fanout, not the cores (so flows aren't recreated → no selector churn)
- acts only when public listener is down AND no live video, across STRIKES polls
- OFF by default (ENABLE=1 to arm); give-up + 300s cooldown
Tests rewritten to lock these invariants. Full suite green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
security(review): remove live hostname from docs + bind mediamtx API …
1f9b5ed 
…to localhost

Review follow-ups on #41:
- docs/index.html + docs/what-are-grains.html no longer hardcode the live server hostname
  (placeholder YOUR-SWITCHER-HOST; health page reads window.MXL_HEALTH_API override). The
  live mxlswitcher.com public/ copies keep the real URL (box-specific, like facility.json).
- quickstart: mediamtx API now binds 127.0.0.1:9997 (MTX_APIADDRESS), not *:9997 — with
  --network host the default exposed an unauthenticated control API on the LAN. The Core
  reads it via localhost. Applied to the live box too.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
docs: review-response thread — Guy's 5 decisions + Oct 9 security fix…
32129d1 
…es (hostname, API bind, employer tag)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran

guycochran commented Oct 9, 2026

Copy link
Copy Markdown
Owner Author

Security review follow-ups (Oct 9) — done on this branch:

  • Removed the live server hostname from docs/index.html + docs/what-are-grains.html (placeholder + window.MXL_HEALTH_API override). Live public/ copies keep the real URL (box-specific).
  • Bound mediamtx's API to 127.0.0.1:9997 (MTX_APIADDRESS) — with --network host the default *:9997 exposed an unauthenticated control API on the LAN. Applied to quickstart + the live box.

Full decision thread in docs/REVIEW-RESPONSE.md. Merge order confirmed: #38→#40→#37→#39→#41 (this merges last).

This was referenced Oct 9, 2026
Draft
Draft
Draft
@guycochran @claude
docs: review thread — multiview-visibility decision (open is fine) + …
8c6c7a1 
…Core-neutralization (PR #42)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran

guycochran commented Oct 9, 2026

Copy link
Copy Markdown
Owner Author

CPU budget check. The target is to stay under 60% on the 24-vCPU switcher. It's currently at about 63% (load 16–17.5). Per-container snapshot from docker stats (read-only):

  • test-generator ~3.2 cores, of which the thumbnail sampler (mxl_thumbs.py) is 1.7–2.5
  • mxl2webrtc main program x264 encode: 2.0–2.6
  • html5-keyer (CEF) ~1.5, with the key OFF
  • guest1 ~1.2 + its SRT fan-out ~1.2
  • guest4 ~0.8 + its SRT fan-out ~1.1
  • guest3 ~1.0 + fan-out ~0.15 + audio ~0.1
  • file-player ~1.15, mediamtx ~0.4, input-selector ~0.15, idle guests/other ~0.45

Questions / suggestions. Any of these may already have a reason.

  1. Fan-out CPU. The guest1 and guest4 fan-outs are copy-only SRT relays but use about 1 core each, while guest3's identical relay uses about 0.15. Is one output retrying a port with no listener in a tight loop? The leaky-queue change in contribution: 60fps restamp fix (HW-verified) + IN-005 registry/soak + IS-05 shim + ANC data-flow + v1.1.0 pin #37 might interact with that. Running ss -ulpn on the fan-out output ports would show it. Estimated saving: ~2 cores (~8 pts).
  2. Thumbnail sampler. It keeps a busy-spinning reader on every flow, including dead slots. Could it attach only to live slots, or sample on demand / at a lower rate? Estimated saving: ~1.5–2 cores (~6–8 pts).
  3. Keyer. It uses about 1.5 cores with the key off. Can it be paused, or dropped to 15 fps / 720p, when nothing is keyed? Estimated saving: ~1.5 cores (~6 pts).

Together that's about 5 cores, which would land around 40–45%. A GPU (for CEF + NVENC) is being considered later and would take another ~10–20 pts, but these software wins come first.

Please reply here with context on why any of these are the way they are before changing anything on the live box.

— Newcastle (review agent)

guycochran and others added 15 commits October 9, 2026 21:57
@guycochran @claude
feat(ingest): per-guest relay host (MXL_GUEST_HOSTS) — publish via an…
c87539f 
… SRT relay, hide the home IP

For a public phone-inject without exposing the home/box public IP: route a guest slot's SRT
to a relay host. MXL_GUEST_HOSTS='guest5=<relay-ip>' makes /join advertise (and the QR encode)
the relay address for that slot — the phone publishes to the relay, the box PULLS from it
outbound, so the box IP never appears publicly. Other slots stay on the LAN/public IP.

Both srt_url and the Larix deep-link honor the per-guest host (the QR must not leak the box IP).
+1 node test. Full suite green (63 node).

HW-deployed: guest5 -> Azure SRT relay (srt-live-transmit) proven end-to-end (phone->Azure->
box-pull->guest5 flow at 30fps); PTZ on guest3 kept on its direct LAN path untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
feat(join): prefer a relay-safe slot by default + show a 🔒 via relay …
fd52a54 
…/ LAN-only badge

The /join page now auto-picks a FREE slot whose SRT host is a public relay (non-private IP,
e.g. the Azure SRT relay) over a LAN slot — so the default public-facing join hands out the
IP-hidden path. A badge shows whether the assigned slot is '🔒 via relay' (home IP private)
or 'LAN only'. Still honors #slot=guestN to force a specific slot for testing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
docs: never-interrupt-PGM architecture for live correspondents (trade…
c1ac6f1 
…-show use case)

The requirement: correspondents join via QR + get cut live WITHOUT ever interrupting program.
Today, adding a guest re-wires the selector -> restart -> ~15s PGM freeze, and idle slots
collapse so they can't be pre-wired.

Design (assembles existing tools): per-slot flow_stabilizer.py writes a STABLE flow that never
dies -> selector pre-wired ONCE at boot to the stable flows, NEVER re-wired -> correspondent
join/drop/reconnect only touches their own volatile flow; cuts become active-input-only;
PGM never interrupted. Contribution via per-correspondent Azure SRT relay (IP hidden).
Build on a CLEAN bring-up, not patched onto hand-surgered state. Full plan + invariants inside.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
wip(never-interrupt): stabilizer-per-slot architecture validated but …
98c32fa 
…6x overloads the box (load 54)

HW findings from first build attempt:
- CORE IDEA PROVEN: stabilizer creates its stable flow at boot (even idle); selector keeps a
  wired slot on an existing-but-stale stable flow (200, holds) -> pre-wiring prevents re-wire
  on join. The architecture is sound.
- BLOCKER: a flow_stabilizer per slot is ~1 core each; 6 of them overloaded the box to load 54
  (24 cores) -> SSH stalls, selector wedges (000), stabilizer restamp freezes under starvation.
- Added stabilized_guest3..6 to facility.json, tools/run-stabilizer.sh, mxl-stabilizer@ template
  (on box, disabled). Revised approaches documented (fewer slots / lighter conform / GPU).
Box restored clean (PGM live). Decision pending before next attempt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
docs(never-interrupt): option (a) HW findings — 2 stabilizers fit (lo…
24c6137 
…ad 12 vs 54); pre-wire drops PGM so fold into boot; discover UUIDs; pgm-heal must verify-advancing before cut

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
feat(watchdog): CPU-spin guard — heal stuck-hot ingest legs that pin …
7767f37 
…a core on a silent source

A source that connects then goes silent (phone stalls behind the Azure relay, Makito
unplugged mid-stream) leaves the conform pipeline freewheeling the last frame through
videorate!videoconvert!v210 at full rate — ~1 core burned while the flow head stays
frozen. Idle slots that never had a caller sit near 0% (srtsrc blocks), so
'high CPU AND flow not advancing' uniquely identifies the stuck-hot case.

Runs for every slot regardless of listener topology (bound public listener OR outbound
pull). Restarts ONLY the fanout (same safe action as the dead-listener path) to drop the
leg back to idle; never touches a chain whose video flow is advancing. SPIN_CPU=0 disables.

HW Oct 10: caught guest5 (94% CPU, frozen flow) and healed it to 14% with PGM undisturbed.
Measured facility load 15 -> 9 after clearing this + a duplicate thumbs process.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
docs(never-interrupt): option (a) BUILT + LIVE — pgm-heal pre-wires s…
80d31c5 
…table correspondent slots; advancing-gated cut

Never-interrupt wiring is now the boot default (patched pgm-heal on the box):
correspondent slots guest5/guest6 wired to STABLE flows 57ab5e00/57ab6e00 (when the
stabilizer is up), fixed sources stay volatile. Default cut gated on a 1s head-advance
check, not file-exists (fixes the Oct-10 PGM-dark bug). HW-verified: wiring survives
source churn unchanged with PGM never dropping; boot-durable across pgm-heal re-runs.

Box copy not committed (hardcodes box-specific discovered UUIDs); repo version needs
runtime UUID discovery first (reviewer issue #2) — documented as the follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
feat(selfheal): heal_publish — auto-recover the grey screen (PGM publ…
788731b 
…ish dropped)

The recurring grey screen: the selector output keeps ADVANCING but mxl2webrtc's publish
to mediamtx silently dies downstream (ICE/renegotiation/mediamtx hiccup). The relay's
/pipeline/status still reports running:true, so heal_relay doesn't catch it — nothing did,
so PGM stayed grey until a manual pgm-heal.

heal_publish uses mediamtx itself as the signal: selector output advancing AND mediamtx
receiving 0 program bytes over a 2s window = publish dropped -> restart the encoder read of
the current program flow (keyer-out if keyed+advancing, else selector-out; same body as
pgm-heal). Requires the selector to be advancing first so it never busy-loops on a
legitimately-idle program. Runs in the existing --watch loop (15s). All flows discovered at
runtime (no hardcoded UUIDs). HW: verified no false-trigger against healthy PGM.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
feat(health): Sources panel — per-input liveness, PGM flag, never-int…
5e53d63 
…errupt tags

Adds a Sources panel to the Health Skin showing each wired selector input with a friendly
name (Makito/PTZ/Playout/Correspondent N), live/standby/idle state, which slot is PGM, and
which slots are never-interrupt stable correspondent slots. Cheap: joins data already
collected — the selector's wired UUIDs (now exposed via shapePipelines) + the thumbs probe's
per-slot liveness + a config-driven role map (MXL_HEALTH_ROLES, keyed by flow-id prefix so
it survives runtime UUID discovery). No new probes.

Also fixes a false-dead: classifyThumbsHealth reported a slot with NO reader data (age==null,
probe rebuilding) as 'dead' — now 'unknown', so a live-but-unprobed camera isn't painted red
(HW Oct 10: thumbs reader for a live Makito/PTZ can sit null while the flow is advancing).

buildSources is pure + unit-tested; stable slots with no correspondent read 'standby' (cyan),
not dead. HW-verified: panel shows Pattern/Playout/Makito/PTZ ok, 2 correspondent slots
standby, PGM flagged correctly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
fix(selfheal): heal_publish crashed under set -u (src unbound) exactl…
2f035fe 
…y when PGM dropped

The previous commit's heal_publish declared 'local src' without initializing it; under the
script's 'set -u', referencing $src on the fallback line tripped 'unbound variable' and the
whole watch loop exited (status=1) — so the one time it was needed (PGM down), it crashed
instead of healing. The healthy-path test missed it because healthy PGM returns early.

Fix: initialize 'local src="" key=""' (+ ka/kb). STRESS-TESTED on HW: killed the encoder
publish -> PGM 0 bytes -> selfheal auto-republished and PGM recovered in ~26s with no human.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
fix(never-interrupt): auto-announce no longer re-wires a healthy sele…
2f92b8d 
…ctor

The ingest fires POST /api/mxl/repair {auto:1} on every first-locked-frame, incl. reconnects.
The repair cascade STOPs+STARTs the selector from a fixed input list — which dropped the
pre-wired stable correspondent slots AND blipped PGM. This was the second re-wire path
undermining never-interrupt (the first — pgm-heal — was already converted).

Fix: for an AUTO announce, if the selector is already running cleanly with inputs wired,
SKIP the whole cascade — the reconnecting source is served by its pre-wired stable slot or
is cuttable as-is. A genuinely down selector still falls through to the full rebuild, and
manual repair (no auto) always rebuilds. Grey-screen/publish recovery is handled separately
by mxl-selfheal heal_publish, so skipping never leaves PGM grey.

HW-verified: auto-announce on a healthy selector -> wiring byte-identical, stable slots
survive, PGM uninterrupted; auto-announce on a stopped selector -> still rebuilds. 3 new
unit tests (skip-when-healthy, rebuild-when-down, manual-always-rebuilds).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
feat(selfheal): heal_thumbs — restart a wedged thumbs probe so live c…
6d8fff6 
…ameras stay visible

Symptom (recurring): a live source wired into the selector (Makito/PTZ, flow advancing per
mxl-info) shows 'unknown'/dead on the Health Skin because the thumbs probe's reader for that
slot wedged ('guestN: stalled (wedged reader or writer gone)') and won't re-attach on its own
promptly. Makes a working camera look down.

heal_thumbs reads the Core's /api/mxl/health, and if a wired non-stable source is NOT ok while
others ARE ok (facility is live -> it's the probe, not the domain), counts a strike; after
THUMBS_STRIKES (default 3) consecutive polls it restarts the thumbs unit so readers re-attach.
Guards: skips stable standby slots, needs sustained strikes (rides out the probe's own ~30s
startup), 120s cooldown (no restart storm), MXL_THUMBS_HEAL=0 disables.

HW: detected a killed/wedged probe (logged 'thumbs probe looks wedged (Makito not-ok ...)'),
sources recovered to ok. Note: the underlying guest1 reader wedge is a thumbs-probe bug (flow
advances but mxlsrc reader stalls); this keeps the display honest+self-recovering until that's
fixed upstream. All flows remained live/cuttable/publishing throughout — cameras were never
actually down, only mis-displayed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
fix(selfheal): heal_publish was fighting itself onto the DEAD keyer f…
59ddba2 
…low (grey screen)

ROOT CAUSE of the recurring 'messed up' cuts: the gst-keyer throttled the whole program chain
to ~300KB/s (vs 6Mbit) AND its flow 86efffa4 keeps a stale ADVANCING head even when the keyer
is stopped. heal_publish picked that keyer flow (saw it 'advancing'), republished the encoder
onto it every 30s -> dead flow -> 0 bytes -> 'PGM dropped' -> republish onto keyer again. An
infinite self-fight that kept PGM grey and reverted every manual fix within 30s.

Fixes:
- heal_publish ALWAYS republishes on the SELECTOR output (keyer bypassed for stability). No
  more keyer-preference — to re-enable keyed PGM, fix the keyer's throughput first.
- New _enc_restart(): stop-until-running:false -> start -> verify flow took (one retry), no
  container restart (a docker restart auto-starts the encoder on its DEFAULT keyer flow, which
  raced the re-point). This is the HW-proven reliable re-point sequence.

HW-verified: 90s watch — encoder stays on selector, PGM full-rate 720-794KB/s, selfheal silent
(no fighting); 6 clean cuts guest1<->guest3 all 200 at full rate. Also disabled mxl-gst-keyer
+ set pgm-heal MXL_KEYER_ENABLE=0 (box) so the keyer stays bypassed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
fix(selfheal): recover when the selector HTTP API dies but the contai…
092725a 
…ner stays up

HW Oct 10: the input-selector container was 'Up 7h' with the router thread running, but its
HTTP control server on :9604 had died — nothing listening, every cut/status returned 000, PGM
frozen, and selfheal's own probes (which go through that API) were blind. A /pipeline/start
can't revive a dead API; only a container restart respawns it.

heal_selector now detects an EMPTY status response (API unreachable) while the container is up,
and restarts the container (120s cooldown so it can't storm), waits for the API, then
re-points the encoder at the selector output via the proven _enc_restart path. This closes the
last 'messed up again' failure mode — previously it needed a manual docker restart + pgm-heal.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
@guycochran @claude
perf(fanout): drop 'tsparse set-timestamps=true' from the A/V SRT spl…
bd059db 
…itter

Profile (Guy, Oct 10): the two active guest fan-outs each burned ~1.1 cores for a pure TS
split. Cause: 'tsparse set-timestamps=true' re-stamps every MPEG-TS packet (PCR parse) in
software on a ~13Mbit stream — unnecessary for a splitter, since the downstream video/audio
cores run their own 'srtsrc ! tsdemux' and derive PTS there. Dropping set-timestamps keeps
tsparse's cheap packet alignment without the per-packet re-stamp.

Also brings this repo copy up to the box's HW-Oct-8 leaky-queue version (tee allow-not-linked
+ leaky=downstream per branch) that was never committed back — prevents an unbalanced-tee stall
where one slow leg SRT-errors the other.

HW: guest3-fanout 118%% -> 70%% on the live PTZ, flow stayed live, cuts clean. ~0.5 core per
active fan-out.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSKEZZz4yEc2CGq5hcTmT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@guycochran