Skip to content

render() is async on a worker thread; the player shows before the audio (1.0.0) - #44

Merged
snakajima merged 7 commits into
mainfrom
async-render
Oct 26, 2026
Merged

snakajima merged 7 commits into
mainfrom
async-render

Conversation

@snakajima

@snakajima snakajima commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

render() no longer blocks the caller, and the GUI plugin shows its card before a long render's audio is ready. Both packages go to 1.0.0, because render() now returns a Promise (breaking).

1. render() returns a Promise and renders on a worker thread (Node)

  • src/render-pool.ts / src/render-worker.ts: a small pool of worker threads (at most 2), reused between renders and unref'd while idle. The Node entry, renderToFiles and manage register it. NEUTRINO runs inside the worker, so an MCP server or GUI host keeps answering during a long piano piece or an uncached vocal.
  • Output is bit-identical: the golden hashes pass unchanged, and a test compares worker vs. in-place rendering (renderNow, the internal synchronous core).
  • Samples cross to the worker as data: loadSamples' sources can list what they hold. A caller's own SampleSource is asked for exactly the keys samplesNeeded reports.
  • Errors keep their class across the thread (JingleScriptError, SamplesNotLoadedError, Neutrino*Error, …), so instanceof still works in the CLI and manage.
  • Only RenderOptions' own fields cross the thread. A field added there is a type error until the pool passes it on, and anything else on the options object (e.g. renderToFiles' onDownload callback) stays behind.
  • When a worker cannot run, the pool falls back to rendering in place, and says so once (process warning JINGLESCRIPT_NO_RENDER_WORKER): a bundle that left render-worker.js behind, or src/ on a Node that cannot strip types. The browser entry always renders in place. Sung renders run one at a time (NEUTRINO uses every core).
  • Breaking: every render() call needs await. Tests, smoke, README, docs, eval and the playground worker are updated.

2. The card before the audio (GUI plugin)

  • manage()'s deferAudio option, used only with player: renderScore waits at most DEFER_WAIT_MS (1 s) for the audio. A short jingle comes back finished, as before. Otherwise it answers at once with a pending PlayerData (score, lanes, cues, lyrics, sheet music; no audio). awaitDeferredPlayer(id) gives the finished one, why it failed, or gone (src/deferred-renders.ts). A finished render is kept in memory until a view collects it (then 1 min more, for another mount of the card), at most 30 min, and only the newest 20. NEUTRINO and its voice are checked before rendering (checkSinger), so a missing install always reaches the LLM.

  • The plugin turns this on only when the host passes deferredAudio: true in its execute context. The view then:

    1. shows the pending card with play disabled and "Rendering the audio…";
    2. collects the audio through the host's runtime dispatch, as a { kind: "collectRenderedAudio", id } request that pluginCore.execute answers before treating the call as a tool call;
    3. saves it into the result with onUpdateResult;
    4. plays, autoplaying when asked. Freshness counts from when the audio was ready, and the voice-wait rules are unchanged.

    If the audio fails after the card appeared, the card says why and, for a fresh render, tells the LLM once. Hosts that don't set deferredAudio behave as before.

  • gui-chat-protocol/vue is now external in the plugin build, so the view injects the host's runtime by the protocol's own PLUGIN_RUNTIME_KEY.

  • CLAUDE.md: only the GUI plugin may lean on its host, and only as an opt-in. Used as a library (MulmoCast calls render()), nothing may depend on the host.

  • To enable this, MulmoTerminal needs two changes (not in this PR; see gui-plugin README): wrap the view with wrapWithPluginRuntime, and add manageJingleScript: { deferredAudio: true } to EXTRA_CONTEXT_BY_TOOL.

Testing

  • Root: 748 tests passed (100 skipped); typecheck, lint, prettier pass; scripts/smoke.sh (pack, install, render via CLI, library, browser entry, MCP) passes.
  • gui-plugin: typecheck, 204 tests, build and playground build pass.
  • Deferred path measured: a 42 s score answered in 1.2 s with a pending card; the audio arrived at 4.2 s.
  • Browser (demo page with a stand-in runtime): the pending card shows lanes and cues with play disabled; 3 s later the waveform appears, play and the MP3 download are enabled, and autoplay starts. No console errors.
  • Not done: running inside MulmoTerminal (needs the host change above).

Related: #43 (NEUTRINO render cache cleanup, follow-up from #42).

🤖 Generated with Claude Code

work in jinglescript

snakajima and others added 7 commits October 26, 2026 08:51
A long render (a piano piece, a NEUTRINO vocal) no longer blocks the caller's
event loop: render() hands the work to a small pool of worker threads
(src/render-pool.ts, src/render-worker.ts), registered by the Node entry, and
resolves with the same bit-identical result. The browser entry renders in
place. renderNow() is the synchronous core the worker runs (internal).

- Samples go to the worker as data (loadSamples' sources can list theirs).
- Errors keep their class across the thread (instanceof works for callers).
- Only RenderOptions' own fields cross; a field added there is a type error
  until the pool passes it on.
- Breaking: every render() call needs await. Tests, smoke, docs, README and
  the playground worker updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
manage()'s renderScore takes `deferAudio`: with a player, it waits at most
DEFER_WAIT_MS (1 s) for the audio, and otherwise answers at once with a
`pending` PlayerData (score, lanes, cues, lyrics, sheet music; no audio) while
the render goes on. awaitDeferredPlayer(id) gives the finished PlayerData, the
reason it failed, or "gone" (kept 30 min after it finishes, in memory).

The GUI plugin sets it when the host's context says `deferredAudio: true` (its
view has a runtime). The view shows a pending card at once with play disabled,
collects the audio through the host's dispatch (a request answered by the same
execute, never sent by the LLM), saves it with onUpdateResult, and then plays,
autoplaying when asked (fresh from when the audio was ready). A render that
fails after the card appeared is shown and, if fresh, told to the LLM once.
Hosts that do not set it get the finished card as before.

The demo page gains a pending card and a stand-in runtime; checked in a
browser. Both packages go to 1.0.0 (render() returns a Promise).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ire on time

- A pending card that unmounts before its audio arrives leaves the reply to the
  mount that shows the card now: it no longer saves, autoplays (which would
  mark the card autoplayed with no audio element) or reports a failure.
- A finished deferred render is dropped DEFERRED_KEEP_MS after it settles by an
  unref'd timer, instead of only when another render or request came along.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Where render() runs in place (Node 22.12 running src/, no type stripping, so no
worker) and ffmpeg is absent, a short render finished before even a 0 ms timer,
so the test expecting a pending player got the finished one. waitMs 0 now means
"do not wait": the answer is always pending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- tests/deferred-renders.test.ts: resolve the pending render without a
  placeholder function (sonarjs/no-extra-arguments).
- gui-plugin tests: the two that render a 42 s score get 60 s; on a CI runner
  the render and its ffmpeg encode took over the 5 s default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t fallback

- CLAUDE.md: only the GUI plugin may lean on its host (opt-in, as deferredAudio);
  as a library (MulmoCast calls render()) nothing may. manage()'s carrier
  options and awaitDeferredPlayer are allowed as API: they are not score features.
- Deferred renders: at most the newest 20 finished ones are kept, a collected
  one only a minute more; renders still running are always kept.
- The pool says once (process warning JINGLESCRIPT_NO_RENDER_WORKER) when no
  worker can run and render() works on the calling thread.
- manage() checks NEUTRINO and the voice before rendering (checkSinger), so a
  missing install reaches the LLM even when the audio would be deferred.
- Sung renders run one at a time in the pool (NEUTRINO uses every core).
- renderNow expands the score once; render() no longer expands it first.
- Tests on the deferred path no longer depend on how fast the machine is.

Not changed: samples are still copied to the worker per render (tens of MB at
most for a grand piano piece, small next to the render; a cache would have to
tell apart callers that give different data under one key).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…appeared

A deferred render that failed more than FRESH_MS after renderScore (an uncached
vocal, a render queued behind others) was never reported, so an LLM waiting for
a message (notifyWhenDone) waited forever. failedMessage now judges freshness
by when the card appeared, not when the render failed: a card replayed from
history still says nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@snakajima
snakajima merged commit f054c6e into main Oct 26, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant