Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/extended_checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -642,6 +642,10 @@ jobs:
run: |
set -eux
PYTHONDONTWRITEBYTECODE=1 python3 examples/dasLLAMA/wasm/test_mint_models.py
# the Pocket converter's pure predicates (which tensor lands in which form) - numpy only, no
# torch; the runner's python carries no numpy, installed the way the news step installs markdown
python3 -m pip install numpy || python3 -m pip install --user numpy || python3 -m pip install --break-system-packages numpy
PYTHONDONTWRITEBYTECODE=1 python3 modules/dasLLAMA/harness/test_convert_pocket.py

- name: "Test pr-babysit verdict core"
if: matrix.role != 'modules'
Expand Down
7 changes: 4 additions & 3 deletions .github/workflows/pages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -334,15 +334,16 @@ jobs:

# 7. dasllama.io/examples — dasLLAMA itself compiled to wasm64, one app per example
# (storyteller: stories15M writes, KittenTTS reads; storywish: you type the words,
# tinystories-instruct-27M writes). Each .das_package turns the GPU modules off, so the
# tinystories-instruct-27M writes, Pocket TTS reads; parrot: you talk, Pocket TTS clones the
# voice and reads the text you type in it). Each .das_package turns the GPU modules off, so the
# host needs no Metal/Vulkan. Non-fatal like the games; the dasllama.io stage step
# stages a card's page only when all three outputs exist. The models are prepared .dlim
# images the stage step MINTS for this very build: examples/dasLLAMA/wasm/dlim_config is
# the same wasm64 build's DlimConfiguration probe, run under node here so the stage step
# bakes against the identity the browser build actually wants (a set minted for another
# IMAGE_VERSION is declined by the program - the failure that put a black canvas on the
# storyteller page for a day).
for ex in storyteller storywish; do
for ex in storyteller storywish parrot; do
if ./bin/daslang utils/daspkg/main.das -- \
release wasm --root "examples/dasLLAMA/$ex" --out "$REPO/web/output64/examples"; then
echo "$ex wasm build OK"
Expand Down Expand Up @@ -639,7 +640,7 @@ jobs:
rm -rf "_site_dasllama/examples/$1/models"
printf '%s' "<!doctype html><html lang=\"en\"><head><meta charset=\"utf-8\"><title>$1 - building</title><style>html,body{margin:0;height:100%}body{background:#12101a;color:#d6cfbf;font-family:ui-monospace,Menlo,Consolas,monospace;display:flex;align-items:center;justify-content:center;text-align:center;padding:24px}p{max-width:30rem;line-height:1.6}</style></head><body><p>This example is being rebuilt and will be available shortly.</p></body></html>" > "_site_dasllama/examples/$1/$1.html"
}
for ex in storyteller storywish; do
for ex in storyteller storywish parrot; do
mkdir -p "_site_dasllama/examples/$ex"
if [ -f "web/output64/examples/$ex/$ex.html" ] \
&& [ -f "web/output64/examples/$ex/$ex.js" ] \
Expand Down
4 changes: 2 additions & 2 deletions REVIEW_COMMON.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,5 +135,5 @@ obligation is a rule, and it lives in the flat list above.
**Adding a rule starts with reading the whole checklist** - duplication, drift, and homeless
placement all start with a rule appended by an author who had not just read the file.

**A rule the diff adds that is longer than every rule already in the file is split, its
exception dissolved, or its extra prose moved to the architecture doc.**
**A rule the diff adds or lengthens that is longer than every other rule in the file is split,
or its extra prose moved to the architecture doc.**
22 changes: 13 additions & 9 deletions doc/source/reference/tutorials/dasLLAMA_16_text_to_speech.rst
Original file line number Diff line number Diff line change
Expand Up @@ -247,12 +247,13 @@ Cloning a voice
===============

``caps().cloning`` says whether the model takes a voice from a recording.
Pocket TTS does: a few seconds of one speaker, mono, at the model's own rate,
become a voice in the roster. ``load_audio_mono`` decodes a wav, flac, mp3 or
ogg file to that rate, and ``tts_register_voice`` adds the samples under the
name you give. From then on the name works like any bundled voice. A clip
longer than a minute is refused, and a phoneme model panics here: it has no
voice to take.
A Pocket TTS file with its codec encoder does: a few seconds of one speaker,
mono, at the model's own rate, become a voice in the roster. ``load_audio_mono``
decodes a wav, flac, mp3 or ogg file to that rate, and ``tts_register_voice``
adds the samples under the name you give. From then on the name works like any
bundled voice. A clip longer than a minute is refused; a Pocket file converted
without the encoder (a one-voice file for a page) reports ``cloning`` false and
refuses by name; a phoneme model panics here: it has no voice to take.

.. code-block:: das

Expand All @@ -262,15 +263,18 @@ voice to take.
let mine <- synthesize(m, "daslang speaks in my voice.", "me")
}

The two weight lanes
====================
The weight lanes
================

The decoder and generator matrix multiplies are served from one of two
prepared images beside the GGUF. The q8 lane holds those weights as Q8_0
quants and is what a load serves by default. The f32 lane holds the file's own
planes; it is the reference the parity tests hold the q8 lane against. A
published Pocket file already holds Q8_0 weights, so its q8 lane reads them as
they are and its f32 lane dequantizes them.
they are and its f32 lane dequantizes them. The small Pocket files hold Q4_K
planes for the backbone and the codec transformers too: an unpinned load serves
those planes as they are through the engine's K-quant kernels, a third lane,
while a pin to q8 or f32 requantizes or dequantizes them at load.

``tts_serves_q8`` answers which lane the next load takes. ``set_tts_q8`` pins
it, and ``reset_tts_q8`` returns to the default. The pin is process-wide
Expand Down
73 changes: 57 additions & 16 deletions examples/dasLLAMA/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,16 @@ checklist is `REVIEW.md` beside this file. The engine these programs drive is do
frame, KittenTTS reads each finished sentence. `main.das` is the whole program, `web_shell.html`
the page around its canvas, `.das_package` the release, `models.json` its model set.
- `storywish/` - a browser example: the typed words become a request in the TinyStoriesInstruct
corpus's layout, tinystories-instruct-27M writes the story, KittenTTS reads it. Same four files;
corpus's layout, tinystories-instruct-27M writes the story, Pocket TTS reads it in one baked
voice from a file without the codec encoder (text in, no packs, no cloning). Same four files;
`wish.das` holds the request side pure (typed line -> words -> prompt, the field-line stop) so a
test reaches it without a window.
- `parrot/` - a browser example: you press record and talk, Silero VAD ends the take when you go
quiet, Pocket TTS clones the voice from the take (a file with its codec encoder and its
roster, which speaks until a take replaces it), and the text in the box is read aloud in it on
the say button; recording again
replaces the voice. Same four files. Nothing leaves the program: the take is cloned in memory
and never written.
- `wasm/dlim_config/` - a wasm-only program: prints the running build's DlimConfiguration JSON.
`wasm/mint_models.py` - the deploy's staging step for a browser example's model set.
`wasm/run_node.js` - runs the wasm64 engine host under node.
Expand All @@ -25,8 +32,8 @@ checklist is `REVIEW.md` beside this file. The engine these programs drive is do
wasm` builds it to wasm64 for dasllama.io from the same `main.das` the desktop run uses. The
checklist's rules about the browser build bind browser examples and nothing else.
- **A witness line** is a line a browser example logs under its own name (`storyteller: ...`,
`storywish: ...`); a smoke test under `modules/dasLLAMA/tests/` matches such lines as
substrings, so their words and order are an interface.
`storywish: ...`, `parrot: ...`); a smoke test under `modules/dasLLAMA/tests/` matches such
lines as substrings, so their words and order are an interface.

## 3. Mechanisms

Expand All @@ -37,34 +44,52 @@ the desktop and `requestAnimationFrame` in the browser. The `.das_package` disab
for the wasm build, so the host needs no Metal or Vulkan, and names the shell that fetches the
models and starts the program on a click (audio needs the gesture). The desktop run and the page
therefore exercise the same code, which is why a browser-only failure is a language-runtime fact
worth a rule rather than an app bug.
worth a rule rather than an app bug. A page the browser restores from its back-forward cache
(Back to another page, then Forward) comes back with the program's workers and the audio output
frozen mid-frame and out of step, and the first sound is whatever the output ring held; the
shell reloads such a page (`pageshow` with `persisted`), so it starts from the gate again.

### 3.2 The speech thread and its stream {#speech-thread-stream}

Speech runs on its own thread so the frame loop never blocks on synthesis. The frame thread
pushes sentences into a stream as archived `Line` records and pops finished clips from a second
stream; a `SeqBox` carries the number of the story being told, so a queued sentence of a story
the user replaced is skipped instead of synthesized. The thread's own setup - the TTS model path
and the voice - rides the same sentence stream ahead of the first sentence. A string captured by
the thread's lambda would be a pointer into the frame thread's heap, which that thread reuses on
its own schedule; a browser worker starts slowly enough to read story text where the path was.
An archived message is copied out of the stream into the reader's heap, so the stream is the one
channel that is safe for a string.
pushes its requests into a stream as archived records (a sentence for the story examples; for
parrot a text to say or a take to clone, the PCM riding in the record) and pops finished clips
from a second stream; a `SeqBox` carries the number of the story (parrot: the say) being told, so
a queued sentence of one the user replaced is skipped instead of synthesized. The thread's own
setup - the TTS model path and the voice - rides the same request stream ahead of the first
request. Parrot's thread answers a say with its chunk count before the first clip, so the frame
thread can tell the last clip from a pause. A string captured by the thread's lambda would be a
pointer into the frame thread's heap, which that thread reuses on its own schedule; a browser
worker starts slowly enough to read story text where the path was. An archived message is copied
out of the stream into the reader's heap, so the stream is the one channel that is safe for a
string.

### 3.3 Input is polled {#polled-keys}

A browser example reads the keyboard with `glfwGetKey` each frame, edge-detected per key code,
never through a GLFW callback. In the browser build a callback lambda fires from a JavaScript
event outside any frame of the program, where the example's state is not live, and the program
traps. A printable GLFW key code is its upper-case ASCII, so the key range doubles as the
character range for a typed line, and repeats come from a hold timer.
character range for a typed line, and repeats come from a hold timer. The mouse is read the same
way: parrot's buttons are text, and a click is `glfwGetMouseButton` edge-detected against the
label's own box (the glyph quads rise above the pen position) in design pixels. In the browser
the surface is the document viewport, and the page's stage sits below the nav, so the picture is
letterboxed; Emscripten maps a click through the canvas element's box with one ratio per axis,
which is exact only when that box is the picture - so a shell sizes the canvas element to the
letterboxed box (`max-width`/`max-height` on the replaced element) instead of stretching it over
the stage with `object-fit`.

### 3.4 The model set is minted for the build that ships it

Each browser example's `models.json` names its source files by Hugging Face repository, file
and sha256. `wasm/mint_models.py` fetches them (cached by sha256), bakes each GGUF into a
`.dlim` image against the wasm64 build's own DlimConfiguration, copies the packs, writes
`models/manifest.json` (the file list, their sizes, the IMAGE_VERSION the images carry) and
and sha256, in three lists: `images` (a GGUF the build bakes into a `.dlim`), `packs` (a
front-end pack) and `files` (a GGUF that is its own served form - a Pocket TTS file); a fourth
list, `tree`, names a file the repository itself carries by its repo-relative path and sha256
(the voice-activity weights). `wasm/mint_models.py`
fetches the published ones (cached by sha256), bakes each image against the wasm64 build's own
DlimConfiguration, copies the packs, files and tree files as they are, writes
`models/manifest.json` (the file list, their sizes, the IMAGE_VERSION the images carry - a set
with no image carries the version the deploy expects) and
stamps that version into the page's `/* @image-version */ 0` slot. The shell reads the manifest,
refuses a set minted for another version before fetching it, and shows a program abort's last
engine lines on the page. The configuration the mint bakes against comes from the wasm build
Expand All @@ -77,6 +102,22 @@ is keyed by the build's identity and a set minted for a previous build is declin
Its `.das_package` disables the GPU tiers exactly as the browser examples' do, so the
configuration it prints is the one their programs run with.

### 3.6 Parrot's take {#the-take}

The microphone is opened at the speech model's own rate (the speech thread reports it, with
whether the file clones, before the first take), mono, and drained on the frame thread every
frame into the take; the take is resampled to 16 kHz by linear interpolation with one running
position across drains, so the stream the Silero iterator hears has no seam where the drains
met, and the iterator is the only reader of that rate. The take ends on the stop button, two
seconds after the iterator's last speech end, at the model's 60 s clip cap, or when the device
has delivered nothing for six seconds (a refused microphone opens but never delivers); the ring's
tail is drained before the device stops, since stopping frees the ring. The clip is the speech
plus a quarter second at each end, never longer than the cap, and goes to the speech thread as a
clone request, so the clone runs off the frame thread like a synthesis. A take with no speech in
it is dropped, and the status says whether the device gave nothing, silence, or too little. A
take starts by cancelling a say in flight - a clip still playing would be recorded - and the
pure side of all this (`take.das`) is what the model-free cells test.

## 4. Exception ledger

None.
24 changes: 17 additions & 7 deletions examples/dasLLAMA/REVIEW.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,26 @@
A browser example is a subfolder here with a `web_shell.html` (`ARCHITECTURE.md` sec. 2); the
rules below bind browser examples.

**Never install a GLFW callback - any `glfwSet*Callback` - in a browser example; poll each key
with `glfwGetKey` every frame and edge-detect it.** In the browser build a callback fires outside
any frame of the program and the program traps (`ARCHITECTURE.md` sec. 3.3).
**Never install a GLFW callback - any `glfwSet*Callback` - in a browser example; poll each input
every frame and edge-detect it - `glfwGetKey` for keys, `glfwGetMouseButton` for buttons.** In the
browser build a callback fires outside any frame of the program and the program traps
(`ARCHITECTURE.md` sec. 3.3).

**A browser example's `web_shell.html` gives the canvas element `max-width` / `max-height` so the
element's box is exactly the rendered image - never `object-fit` on a canvas stretched to fill the
page area around it.** A click maps through the element's box with one ratio per axis, so a
stretched box mis-maps every click (`ARCHITECTURE.md` sec. 3.3).

**A browser example's `web_shell.html` must reload a page the browser restored from its
back-forward cache - a `pageshow` handler that reloads when `persisted` is set.** Such a page
comes back with its workers and audio output frozen out of step (`ARCHITECTURE.md` sec. 3.1).

**A diff that changes or drops a witness line - a line a browser example logs under its own name -
updates every test under `modules/dasLLAMA/tests/` that matches it, in the same change.** The
smoke tests match witness lines as substrings, so the words and their order are an interface
(`ARCHITECTURE.md` sec. 2).

**A diff that adds a model file to a browser example's `models.json` names it by the repository
it is published in and its sha256, never by a local path or a branch name.** The deploy fetches
the file by that name and refuses one whose hash moved; a local path stages nothing on the
runner (`ARCHITECTURE.md` sec. 3.4).
**A diff that adds a model file to a browser example's `models.json` names its sha256 and a
location that cannot move - a Hugging Face repository, or a repo-relative path in this repository
under `tree` - never a machine-local path or a branch name.** The deploy fetches by that name and
refuses a file whose hash moved (`ARCHITECTURE.md` sec. 3.4).
19 changes: 19 additions & 0 deletions examples/dasLLAMA/parrot/.das_package
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
options gen2

require daslib/daspkg

[export]
def package() {
package_name("parrot")
package_description("Parrot: talk for a few seconds, Pocket TTS clones the voice, and the text you type is read aloud in it - dasGlfw + dasOpenGL + dasAudio over dasLLAMA")
}

[export]
def release() {
release_main("main.das") // one source for the desktop run AND wasm64 (daspkg release wasm)
release_web_shell("web_shell.html") // fetches the models into MEMFS, then a click starts the program (audio needs the gesture)
// host-only GPU tiers: absent from the wasm build, so their guarded requires resolve as absent
release_wasm_disable_module("dasvulkan")
release_wasm_disable_module("dasmetal")
release_wasm_disable_module("dasaccelerate")
}
Loading
Loading