Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ These are the keys as they come; every one can be changed in the help (**?**): c

| Key | Does |
| --- | --- |
| S | audio source: system audio, demo beat, file, input |
| S | audio source: system audio, demo beat, file, input, radio |
| V | background: spectrum, tunnel, spiral, pitch wheel, warp, cuts, worlds, chrome, spectrogram, scenery, rain, scope, fire, off (each described under [Backgrounds](#backgrounds)) |
| A | how strongly the ASCII background (spectrum, rain, scope) shows: soft (how it used to be), normal, bold. The code lines sit on their own bands, so they stay readable. With the worlds on, A picks their look instead: auto (film, and on each drop another look for 8 bars), night (as auto, but every world keeps a dark sky with its stars, so no cut lights the whole screen; with fewer flashes on, the worlds always have night skies), film, ASCII (the planet as text, each character in its own colour), code (the planet drawn in the song's own code the way ASCII art draws a picture: as many characters as the world is bright there, each in its colour, the world faint underneath), ink (the theme's two colours, dithered as the ID card's picture is), and the Locust Worlds page's print finishes: risograph, cyanotype, receipt, thermal camera, night vision, CRT, VHS, quadtree; and cells, the planet cut into cells of the worlds around it, turning over cell by cell into the next on each cut (after Days of Glitch's moving cells). A print look covers the planet, or the sky when the code sits over the planet. With chrome, A turns its collage off and on; with the spectrogram, it switches its colours between the rainbow and the theme's own; with the scenery, its scene: auto (a new one each song), the lake, the sea, the campfire, the snow, the city in the rain, the garden, the falls, Halloween, the sharks, the festival, the koi pond, space, the eye, and last auto again in the theme's two colours. With the handheld game on (P), A goes round the game's screens instead: Toaster's own (in the theme's colours), neon, an old LCD. Remembered |
| R | orb, over whichever background is picked (or none): after libraries.dev's Thinking Orbs, a cloud of 2,200 soft dots in 3D that holds a form for 8 bars and then flows, dot by dot, into the next, landing on the phrase's one: a sphere, a glitch ball (after animoscillator's glitch orb: the dots fade into a ball of neon noise smeared into streaks, each kick dropping fresh noise in and each bar's first kick breaking it up again), a crystal (a cube of dots on its corner), orbit paths with their dots running round them and bright particles among them, a staff the bass notes appear on as they're played, a spiral galaxy, a network its signals run through on the hats, a field of waves riding the spectrum, a sphere in slices that turn one a beat and then solve themselves (Thinking Orbs' moves), a globe's lines, a ring, an icosahedron, rings swelling with their bands of the spectrum, a burst of rays, ripples on a disc, and an outline that changes shape on the bar. The kick breathes it out, the clap spins it, the hats send a band of light round it, the bass swells the dots. Timed off the beat like the blocks |
Expand Down Expand Up @@ -94,7 +94,8 @@ These are the keys as they come; every one can be changed in the help (**?**): c
| H / F | hide header and footer / fullscreen |

The background and its strength, blocks, orb, card, glass, theme, camera effect and drop-effect
choices are remembered between launches, and so are the keys; the camera always starts off.
choices are remembered between launches, and so are the keys and the radio's address; the camera
always starts off.

## Backgrounds

Expand Down Expand Up @@ -171,9 +172,10 @@ More on all three in [docs/how-it-works.md](docs/how-it-works.md).
Everything Toaster hears and sees stays on your machine. The sound is worked out in the app, and the
camera (off until you press C) is looked at by the system's own face detector (Windows' face
detector, or the Mac's Vision framework) and by Google's MediaPipe, both running on your machine.
Nothing is uploaded. It reaches the internet for two things only: GitHub, when it starts, to see
whether there's a newer Toaster (turn that off in the help, under Updates), and strudel.cc when you
press O (your browser opens it with the code in the link).
Nothing is uploaded. It reaches the internet for three things only: GitHub, when it starts, to see
whether there's a newer Toaster (turn that off in the help, under Updates), strudel.cc when you
press O (your browser opens it with the code in the link), and the radio's stream while it plays
(from the server at the address you typed).

## Good to know

Expand All @@ -182,6 +184,8 @@ press O (your browser opens it with the code in the link).
- The song's title (and its cover, for the ID card) come from the system's media controls
(Windows' media controls, or the Mac's Now Playing), so any player that shows up there works:
Spotify, browsers and the rest.
- The radio (S) plays an internet radio stream whose address you paste in; Toaster comes with no
stations. More in [docs/how-it-works.md](docs/how-it-works.md#the-radio).
- For the finger frame, the portals and the hand signs, keep your hands a little inside the camera's picture.
- On a touch screen (an iPad, say), ☰ in the header opens every key as a button, each showing what it's set to.
- What it can and can't hear yet: the end of [docs/how-it-works.md](docs/how-it-works.md).
Expand Down
17 changes: 16 additions & 1 deletion docs/how-it-works.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,21 @@ from its first frame, so every clip is an ordinary video, nothing cut or stitche
works alongside M. While it's on, the rec button shows ✂; it costs a video encoder or two running
all the time, so it's off until you turn it on.

## The radio

The radio (in the source menu) plays an internet radio stream from the address you paste in: the
stream itself, MP3 or AAC from an Icecast or Shoutcast server, not the station's page or its .pls or
.m3u playlist (the stream's address is inside those). Toaster comes with no stations (a stream is
its station's to offer, and many don't let apps come with theirs), so you bring the address; the
last one that played is remembered. It plays through the app the way a file does, so the blocks are
timed the same way. The song titles a stream carries aren't read, so, as on an input, a new song
starts the code over when Toaster hears a new tempo or a pause. If the stream ends or drops, it
says so and the source menu comes back.

The stream is read cross-origin (CORS), the only way the analysis hears sound from another server.
The app's page is a file on your machine, which may read sound from any server; on the web page a
stream only plays if its server allows it, and an https page can't play an http stream.

## The camera

Everything the camera sees stays on this machine. While it's on, two things look at it:
Expand All @@ -74,7 +89,7 @@ a second to set up on the graphics card; after that it starts at once. If the mo
(`hand_landmarker/hand_landmarker/float16/1/`, `face_landmarker/face_landmarker/float16/1/` and
`image_segmenter/selfie_segmenter/float16/latest/`).

To look at the page in a browser (only the demo beat, files and inputs work there):
To look at the page in a browser (only the demo beat, files, inputs and some radio streams work there):
`node tools/serve.mjs`, then open http://localhost:5173.

## The blocks
Expand Down
45 changes: 37 additions & 8 deletions renderer/app.js
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,7 @@ const state = {
sceneryScene: 'auto', // and its scene (A too): auto (a new one each song), or one by name
calm: false, // fewer flashes (the help's setting; on at first if Windows' animations are off)
updates: true, // look for a newer Toaster when it starts (the help's setting)
radio: '', // the radio's address, as last played
seenVersion: '', // the version last opened, to tell what's new after an update
firstRun: false, // nothing saved yet: this is the first time Toaster has opened
hushed: false, // every part muted and the picture dimmed (a raised fist, or Q)
Expand All @@ -153,7 +154,7 @@ let col = { bg: [5, 29, 181], fg: [255, 255, 255], ink: [2, 11, 69] };

// The look you picked (background and its strength, blocks, orb, card, glass, theme, drop effect)
// is remembered between launches. The camera always starts off.
const SAVED = ['keys', 'codeHidden', 'crt', 'strength', 'blocks', 'blockLook', 'gameScreens', 'orb', 'card', 'cardV', 'cardBig', 'glass', 'theme', 'drop', 'syncTrim', 'camFx', 'calm', 'updates', 'worldLook', 'chromeCollage', 'spectroTheme', 'sceneryTheme', 'sceneryScene', 'seenVersion'];
const SAVED = ['keys', 'codeHidden', 'crt', 'strength', 'blocks', 'blockLook', 'gameScreens', 'orb', 'card', 'cardV', 'cardBig', 'glass', 'theme', 'drop', 'syncTrim', 'camFx', 'calm', 'updates', 'worldLook', 'chromeCollage', 'spectroTheme', 'sceneryTheme', 'sceneryScene', 'radio', 'seenVersion'];
function loadSettings() {
let saved = {};
try {
Expand Down Expand Up @@ -245,7 +246,7 @@ function attach(node, source, label) {
state.label = label;
tapRecording();
clips.retap();
// a file or the demo plays from its start; system audio and inputs may be halfway through a song
// a file or the demo plays from its start; system audio, inputs and the radio may be halfway through a song
freshStart(source === 'file' || source === 'demo');
$('menu').hidden = true;
document.body.classList.add('live');
Expand Down Expand Up @@ -314,17 +315,19 @@ async function useDemo() {
attach(state.demo.out, 'demo', 'demo beat');
}

async function useFile(file) {
// A file or the radio: an audio element, played to the speakers and heard by the analysis. load
// gives it what to play, once it's this pick's turn.
async function playElement(source, label, load) {
const ticket = ++picking;
await ac.resume();
if (outdated(ticket)) return;
detach();
const el = new Audio(URL.createObjectURL(file));
el.loop = true;
const el = new Audio();
load(el);
const node = ac.createMediaElementSource(el);
node.connect(ac.destination);
state.el = el;
attach(node, 'file', file.name);
attach(node, source, label);
try { await el.play(); } catch (err) {
if (state.el !== el) return; // a newer source already took over
detach();
Expand All @@ -335,7 +338,29 @@ async function useFile(file) {
$('status').textContent = restingStatus();
throw err;
}
if (outdated(ticket)) el.pause();
if (outdated(ticket)) { el.pause(); return; }
// (a stream that ends, or anything that drops, says so, as system audio does when it stops)
el.onended = el.onerror = () => { if (state.el === el) showMenu(`${label} stopped playing. Pick a source.`, 'warning'); };
}

async function useFile(file) {
await playElement('file', file.name, (el) => { el.src = URL.createObjectURL(file); el.loop = true; });
}

// The radio: the stream at an address you paste in (Toaster comes with no stations). It's read
// cross-origin, the only way the analysis hears another server's sound (see docs/how-it-works.md).
async function useRadio(address) {
let url = null;
try { url = new URL(address.trim()); } catch { /* not an address */ }
if (!['http:', 'https:'].includes(url?.protocol)) throw new Error("That isn't a stream's address: paste one that starts with http:// or https://.");
const label = url.host + url.pathname.replace(/\/$/, '');
const browser = desktopApp() ? '' : ` In a browser, a stream only plays if its server lets other sites play it, and an https page can't play an http stream: the Toaster app for Windows or Mac has neither limit (${$('b-star').href}).`;
await playElement('radio', label, (a) => { a.crossOrigin = 'anonymous'; a.src = url.href; }).catch(() => {
throw new Error(`Could not play ${label}. The station may be off the air, or this isn't the stream itself (MP3 or AAC) but the station's page or a .pls or .m3u playlist (open one of those in a text editor: the stream's address is in it).${browser}`);
});
if (state.source !== 'radio') return; // (another pick took over before it played)
state.radio = url.href; // (kept for next time once it plays)
saveSettings();
}

// What went wrong when the microphone wouldn't open, said so it can be acted on. iOS asks about the
Expand Down Expand Up @@ -498,6 +523,8 @@ function currentSong() {
if (state.source === 'system') return state.winSong;
if (state.source === 'demo') return { title: 'demo_beat', artist: 'Toaster' };
if (state.source === 'file') return { title: state.label.replace(/\.[^.]+$/, ''), artist: 'File' };
// (an input or the radio: no title, which also lets the engine drop a beat that's gone, as when
// a station goes from a song to talking: see song() in engine.js)
return null;
}

Expand Down Expand Up @@ -1527,6 +1554,7 @@ $('go-system').onclick = run(useSystem);
$('go-demo').onclick = run(useDemo);
$('go-file').onclick = () => $('file').click();
$('go-input').onclick = run(chooseInput);
$('radio').onsubmit = (e) => { e.preventDefault(); useRadio($('radio-url').value).catch(showError); }; // (its button, or Enter in the field)
$('file').onchange = () => { const f = $('file').files[0]; if (f) useFile(f).catch(showError); $('file').value = ''; };
$('b-source').onclick = () => showMenu();
// (on a touch screen, with no Esc: the menu closes with its button, or a tap beside it)
Expand Down Expand Up @@ -1740,7 +1768,7 @@ addEventListener('keydown', (e) => {
const k = e.key.toLowerCase();
if (!visMenu.hidden) { showVisMenu(false); if (k === 'escape') return; }
if (k === 'escape') { closeMenu(); return; }
if (e.target.closest('#alerts')) return; // let Enter/Space press the message box's button
if (e.target.closest('#alerts, #menu:not([hidden]) input')) return; // let Enter/Space press the message box's button, and an address be typed
if (k === '?' || k === '/') { $('help').showModal(); return; }
if (k === 'f11') { e.preventDefault(); window.nc?.toggleFullscreen(); return; }
const act = byKey.get(k);
Expand Down Expand Up @@ -1790,6 +1818,7 @@ window.nc?.version?.().then((v) => {
});

loadSettings();
$('radio-url').value = state.radio;
newsRows();
$('b-rec').dataset.title = `Record a video of the window, with the sound, into ${onMac() ? 'Movies/Toaster' : 'Videos\\Toaster'}`;
applyKeys();
Expand Down
5 changes: 3 additions & 2 deletions renderer/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html lang="en">
<head>
<meta charset="utf-8" />
<meta http-equiv="Content-Security-Policy" content="default-src 'self'; script-src 'self' 'wasm-unsafe-eval'; worker-src 'self'; style-src 'self' 'unsafe-inline'; font-src 'self'; img-src 'self' data: blob:; media-src 'self' blob: mediastream:" />
<meta http-equiv="Content-Security-Policy" content="default-src 'self'; script-src 'self' 'wasm-unsafe-eval'; worker-src 'self'; style-src 'self' 'unsafe-inline'; font-src 'self'; img-src 'self' data: blob:; media-src 'self' blob: mediastream: http: https:" />
<title>Toaster</title>
<link rel="stylesheet" href="style.css" />
</head>
Expand Down Expand Up @@ -99,6 +99,7 @@ <h1>toaster <small class="ver"></small></h1>
<button id="go-demo">✦ demo beat</button><span>built-in test track</span>
<button id="go-file">▤ audio file</button><span>or drop a file on the window</span>
<button id="go-input">◉ audio input</button><span>a mic or line-in</span>
<form id="radio" novalidate><button id="go-radio">∿ radio</button><input id="radio-url" type="url" placeholder="paste a stream's address, then Enter" aria-label="the radio stream's address" autocomplete="off" /></form>
</div>
<p class="tip">Nothing is recorded or uploaded.<span class="key-only"> Esc closes this.</span></p>
<button id="menu-close" class="menu-close">✕ close</button>
Expand Down Expand Up @@ -148,7 +149,7 @@ <h2>how it works <small class="ver"></small></h2>
<tr data-act="hide,full"><td>H / F</td><td>hide the header and footer / fullscreen</td></tr>
<tr data-act="machine"><td>I</td><td>machine readout</td></tr>
<tr class="sec"><th colspan="2">setup</th></tr>
<tr data-act="source"><td>S</td><td>audio source: system audio, demo beat, file, input</td></tr>
<tr data-act="source"><td>S</td><td>audio source: system audio, demo beat, file, input, radio</td></tr>
<tr data-act="name"><td>E</td><td>name the song yourself</td></tr>
<tr data-act="later,earlier"><td>[ / ]</td><td>blocks timing: ] if they land late, [ if early</td></tr>
</table>
Expand Down
5 changes: 3 additions & 2 deletions renderer/style.css
Original file line number Diff line number Diff line change
Expand Up @@ -398,9 +398,10 @@ html.darwin.full header { padding-left: 12px; }
.card { width: min(620px, 100%); border: 2px solid var(--fg); background: var(--bg); padding: 26px; }
.card h1 { margin: 0; font-size: 28px; }
.card .sub { margin: 6px 0 20px; color: var(--dim); }
.opts { display: grid; grid-template-columns: max-content 1fr; gap: 10px 14px; align-items: center; }
.opts { display: grid; grid-template-columns: max-content minmax(0, 1fr); gap: 10px 14px; align-items: center; }
.opts button { font-size: 14px; padding: 8px 14px; text-align: left; }
.opts span { color: var(--dim); font-size: 12px; }
#radio { display: contents; } /* (its button and field sit in the grid as the other rows do) */
.tip { font-size: 12px; line-height: 1.5; margin: 16px 0 0; color: var(--dim); }
/* a touch screen (an iPad has no Esc or S): the bars stay readable while playing, the menu has a way out, and the buttons are big enough to hit */
.menu-close { display: none; }
Expand Down Expand Up @@ -613,7 +614,7 @@ dialog .vtag { color: var(--dim); font-size: 11px; margin-left: 4px; }
dialog tr.sec th { text-align: left; padding: 12px 0 4px; color: var(--dim); font-weight: 400; letter-spacing: .12em; text-transform: uppercase; font-size: 11px; }
dialog form { text-align: right; margin-top: 14px; }
#rename-form { display: grid; gap: 10px; text-align: left; }
#rename-form input { font: inherit; padding: 6px 8px; background: transparent; color: inherit; border: 1px solid var(--dim); }
#rename-form input, #radio-url { font: inherit; padding: 6px 8px; background: transparent; color: inherit; border: 1px solid var(--dim); }
#rename-form button { justify-self: end; }
#device-list { display: grid; gap: 8px; margin: 8px 0; }
#device-list button { text-align: left; }
2 changes: 1 addition & 1 deletion tools/serve.mjs
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
// Serves the app's page as a plain web page (http://localhost:5173), for looking at the UI in a
// browser. Outside the desktop app only the demo beat, files and audio inputs work as sources.
// browser. Outside the desktop app only the demo beat, files, audio inputs and the radio work as sources.
import { createServer } from 'node:http';
import { readFile } from 'node:fs/promises';
import { extname, join, normalize, sep } from 'node:path';
Expand Down