From 697dc0d974467cfc49d588c0b8730175a2f49d76 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 4 Sep 2026 01:17:01 +0000 Subject: [PATCH 01/14] docs: wake-word detection briefing and implementation proposal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Background research and a staged proposal for adding wake-word detection to the family. Neither is a commitment to build it; both are the record of the question being asked properly. docs/wake-word-briefing.md surveys how keyword spotters actually work — the cascade, the feature front end, the model lineage from keyword/filler HMMs through Deep KWS to DS-CNN and dilated TDNN, the posterior-smoothing decision stage — and the free corpus landscape, with licences flagged where they block commercial use. docs/wake-word-plan.md proposes reusing what MuTap already has rather than importing a runtime: promote the dense/GRU arithmetic out of nn_suppressor.h into a shared tap::dsp::nn, add a mel/PCEN front end to DspTap, extend tools/ml to a second task, and ship one external. Milestones M0-M7 with pass criteria in the HANDOFF.md manner, and the evaluation harness deliberately sequenced before the model — false-accepts-per-hour is the metric that decides whether any of this works, and a harness built after the model is a harness built to flatter it. Every number in both documents is a target or a published corpus figure, never a measured result. Open decisions (repository home, wake phrase) are recorded, not assumed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01F1ZmNVvP9oKtEkAZCfdQGy --- HANDOFF.md | 1 + docs/wake-word-briefing.md | 270 ++++++++++++++++++++++++++ docs/wake-word-plan.md | 376 +++++++++++++++++++++++++++++++++++++ 3 files changed, 647 insertions(+) create mode 100644 docs/wake-word-briefing.md create mode 100644 docs/wake-word-plan.md diff --git a/HANDOFF.md b/HANDOFF.md index 2910b18..b00e259 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -756,5 +756,6 @@ Resolved since rev 1: ~~license~~ (MIT), ~~core language~~ (header-only C++20), Resolved since rev 3: ~~Max external naming~~ — settled (Rev 4): **`mutap.afc~`** (rename from the `mutap.defeed~` placeholder) and **`mutap.aec~`** for the new echo canceller, an acronym pair matching the literature. The rename executes in Stage 2 of "The next effort" above. Still open: +- **Wake-word detection — proposed, not started.** A background briefing and a staged implementation proposal landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md) and [`docs/wake-word-plan.md`](docs/wake-word-plan.md). The proposal reuses the `nn_suppressor` / `tools/ml` machinery rather than importing a runtime, and its first two milestones (a DspTap mel front end; promoting the dense/GRU kernels into `tap::dsp::nn`) pay off whether or not the spotter itself ships. Two decisions are yours before anything starts: whether MuTap's charter widens to hold it, and what the wake phrase is. - **Default engine in the external** — `@kalman` off (classic NLMS) is the shipping default purely on seniority; the measured case for flipping it is in `tests/test_fd_kalman.cpp` and book chapter 1. Decide after real-room listening. - **RIR fixtures, the measured half** — the fixture pipeline is built and three physically-modeled rooms (image-source, documented geometry) are committed baselines with regression tests. What remains yours: which MEASURED rooms join them — an academic dataset room (MYRiAD is the PEM-AFROW group's own database; openAIR is the other usual source; check each room's license allows redistribution in an MIT repo) and/or your own swept-sine measurements. Either way it is one command per room: `python3 tools/fixtures/make_rir_fixtures.py --from-wav room.wav myroom --source ""`, then a test with a freshly measured threshold. (The dataset hosts are unreachable from the remote dev container's network policy, so the WAVs have to enter via a commit.) diff --git a/docs/wake-word-briefing.md b/docs/wake-word-briefing.md new file mode 100644 index 0000000..d95b5bf --- /dev/null +++ b/docs/wake-word-briefing.md @@ -0,0 +1,270 @@ +# Wake-word detection — technical briefing + +*Background research, 4 September 2026. No commitment; see +[`wake-word-plan.md`](wake-word-plan.md) for the implementation proposal that +builds on this.* + +Formatted version: + +--- + +## 1. The constraints, not the classifier, are the problem + +A wake-word detector is a small neural classifier run continuously over a +sliding window of audio features, emitting a probability every 10 ms, with +smoothing and a threshold on top. Everything interesting about the design comes +from the constraints; the classification itself is, by modern standards, easy. + +Three numbers drive the architecture: + +- **Always-on.** The front-end stage has a power budget in the low single-digit + milliwatts and a memory budget in tens of kilobytes. +- **Streaming.** Nothing may look ahead more than a few tens of milliseconds. A + model that needs the complete utterance before deciding is useless. +- **False-accept rate.** The usual product target is well under one false wake + per day — roughly one false accept per 10–100 hours of arbitrary audio — + while still catching 95 %+ of genuine utterances across accents, distances + and noise. + +That last requirement is the whole game. You are asking for something like a +1e-7 per-frame false-positive rate, and no single small model gets there. The +architecture everyone converges on is a consequence of that arithmetic. + +## 2. The cascade + +Products stage the power-versus-accuracy conflict. Each tier wakes the next and +each is orders of magnitude more expensive. No individual stage is precise +enough; the compounded rate is. + +| Tier | Stage | Budget | Role | +|---|---|---|---| +| 1 | Acoustic VAD (analog / fixed-function) | microwatts | Rejects silence, which is most of the day | +| 2 | Keyword spotter (10k–250k params, int8) | ~1–3 mW, DSP or NPU | **"The wake-word algorithm."** Deliberately loose threshold — its only job is deciding whether to spend power | +| 3 | Verifier (+ speaker ID) | tens of mW, app processor | Re-scores the same audio with lookahead | +| 4 | Cloud ASR | network, watts | Final adjudication; can *retract* a wake already shown to the user | + +A 1–2 s pre-roll buffer is re-read by every later tier — the device needs it +anyway so the command following the wake word is not clipped. + +The tier-4 retraction path is why devices sometimes light up and then abort: the +cloud transcribed "hey, seriously?" rather than the wake word. + +## 3. Features + +Almost universally a log-mel filterbank: pre-emphasis, 20–30 ms frames at a +10 ms hop, windowed real FFT, power spectrum, 20–40 triangular mel bands, log +compression. MFCCs (a DCT on top) were standard in the HMM era; the DCT mainly +decorrelates for diagonal-covariance Gaussians and neural nets do not need it, +so log-mel is the modern default. Learned filterbanks (SincNet, LEAF) exist; +the gain is small and the cost real, so fixed mel remains the norm on-device. + +The whole front end is a few thousand MACs per 10 ms frame — which is exactly +why it can live in a fixed-point DSP loop. + +**PCEN** (per-channel energy normalization) replaces the plain log with adaptive +gain control plus compression: a one-pole smoother per mel channel, divided out, +then compressed. Markedly more robust to talker distance and channel variation +at essentially no cost; used in Google's on-device spotters. + +### Three nested time scales + +| Scale | Typical | What it sets | +|---|---|---| +| Analysis frame | 25 ms → 40 mel bands | Spectral resolution | +| Hop | 10 ms → 100 vectors/s | Decision rate | +| Model context | ~400 ms (30 frames left + 10 right) | How much of the phrase the classifier sees at once | +| Confidence window | ~1 s | How much evidence must agree before waking | + +Streaming implementations cache intermediate activations so only the newest +10 ms of work is recomputed per hop, rather than re-running the whole 400 ms +context. + +## 4. The model + +**Keyword/filler HMMs** were the original approach: one HMM for the keyword's +phone sequence, one "filler" HMM for everything else, Viterbi decoding to +compare paths. Principled likelihood ratio, fiddly decoding, weak acoustic +model. + +**Deep KWS** (Chen, Parada & Heigold, 2014) threw out the decoder. Stack a +window of frames — say 30 left and 10 right — feed a small fully-connected net, +output a posterior per keyword *word* plus a filler class. Smooth, threshold. +That is the entire system; it substantially beat the HMM approach and remains +the skeleton of everything since. + +What changed since is only the topology, for MAC-count reasons: + +- **CNNs** over the time–frequency patch (Sainath & Parada 2015). +- **Depthwise-separable CNNs (DS-CNN)** — winner of Arm's *Hello Edge* + benchmark, still a strong default for microcontroller-class targets. +- **TDNNs / dilated temporal convolutions** — long receptive field cheaply, and + they stream naturally. With activation caching you compute only the newest + frame's work each hop. This is the single most important implementation trick + for streaming efficiency, and what "streaming-aware training" in the TFLite + Micro KWS tooling refers to. +- **CRNNs and GRU/LSTM stacks** — good receptive field, but recurrent state is + awkward to quantize and the sequential dependency hurts on parallel hardware, + so convolution-with-cache has largely won on-device. +- **Small transformers / conformers** at the verifier tier. + +**End-to-end variants** skip frame-level labels: max-pooling loss over the +positive segment, CTC, or an RNN-T-style streaming decoder scoring the keyword's +label sequence. Practically important because frame alignments otherwise require +an ASR system you may not have. + +## 5. The decision stage + +Raw per-frame output is noisy; nobody thresholds it directly. The Chen et al. +formulation: smooth each class posterior over ~30 frames; form a confidence as +the geometric mean of the maximum smoothed posterior of each keyword word within +a ~100-frame window; fire when it crosses a threshold; apply a refractory period +so one utterance does not fire five times. + +The threshold is the entire operating-point knob, chosen from a **DET curve** +plotting false-rejection rate against false-accepts-per-hour over hundreds or +thousands of hours of negatives. + +> **Evaluation asymmetry.** Recall is measured *per utterance* on positives; +> false accepts are measured *per hour* on negatives. Any harness reporting a +> single accuracy figure over a balanced test set is measuring the wrong thing, +> and will make a model look finished that wakes on the radio nine times an +> evening. + +Many shipping devices adapt the threshold at runtime — tightening in noise, or +after a recent false wake. + +## 6. Training data, and why negatives are the work + +Positives are comparatively cheap: a few thousand to a few hundred thousand +recordings, multiplied by augmentation — additive noise at varied SNRs, RIR +convolution for reverberation and distance, speed/tempo perturbation, +SpecAugment-style masking, simulated codec and microphone response. Synthesized +positives from multi-speaker TTS work surprisingly well and are what make +arbitrary custom wake words feasible at all. + +Negatives are where the effort goes. Random audio is easy to reject and teaches +little. What matters is **hard negative mining** — phonetically confusable +material. For "Hey Jarvis" you need thousands of "hey Travis", "hey, Jarvis +said", "a garage is". + +Product teams choose wake words partly for this reason: three or four syllables, +unusual phonotactics, distinctive stress, preferably not a substring of common +speech. "Alexa" and "Siri" are short and pay for it; "OK Google" and "Hey +Snapdragon" are long and confusable-poor by design. + +The other perpetual source of negatives is device-side false accepts returned +from the field — simultaneously the most valuable training data available and +the reason wake-word telemetry is a privacy flashpoint. + +## 7. Custom and few-shot wake words + +**Query-by-example / embedding** systems train a speech encoder to map audio to +a fixed embedding, enrol from three repetitions, detect by cosine distance. No +retraining, arbitrary phrases, noticeably worse than a trained model. + +**Phoneme-posterior** systems run a small generic phone recognizer and match the +target phrase's phone sequence against the posteriorgram, so a wake word can be +specified as text alone. Commercial engines (Picovoice Porcupine among them) do +a variant of this, compiling a phrase into a small model offline. + +## 8. On-device numerics + +The shipping form is almost always **int8 weights and activations** with +per-channel scales, from post-training quantization with a calibration set or +from quantization-aware training. Accuracy cost typically well under a point. + +The feature front end runs in Q15 or Q31 on M4/M33/M55-class parts, with the FFT +and mel accumulation carrying the usual headroom-versus-precision negotiation, +and MVE/Helium giving the dot products a 4–8× lift. On a Cortex-M55 a DS-CNN +spotter is comfortably real-time at a few percent duty cycle; on an M4 it is +tighter but done routinely. + +Total footprint for a credible tier-2 model: ~20–100 KB of weights plus the +pre-roll ring buffer. + +## 9. Open corpora + +The field is unusually well served by open data, with one catch: the free +corpora give you *keywords*, *negatives* and *noise*, but almost nobody +publishes a real product wake word under a commercially usable licence. + +### Keyword-specific + +| Corpus | Scale | Licence | Good for | +|---|---|---|---| +| Speech Commands v2 | 105,829 one-second clips, 35 words, ~2,600 speakers, 16 kHz | CC BY 4.0 | The canonical benchmark. Contains `marvin` and `sheila` — name-shaped by design — plus a background-noise folder. Build the harness against this first. | +| Multilingual Spoken Words (MSWC) | ~23.4 M clips, 340k keywords, 50 languages | CC BY 4.0 | The big one. Force-aligned out of Common Voice. The only large keyword set unambiguously commercially usable, and the natural source of phonetically confusable hard negatives. | +| MobvoiHotwords | ~175k utterances, 2 Chinese wake phrases | OpenSLR SLR87 | Genuinely wake-word-shaped: multi-syllable, far-field, varied SNR, real negatives — which Speech Commands is not. | +| HI-MIA / HI-MIA-CW | Mandarin, far-field multi-mic | OpenSLR SLR85 | Joint wake-word + speaker verification. | +| Hey Snips | ~11k positives, ~2,200 speakers, ~86k negatives | research-use | Closest public thing to a real product set. **Availability unreliable since the Sonos acquisition — verify terms and mirrors before planning around it.** | +| Qualcomm Keyword Speech | 4 keywords, 50 speakers, ~4,000 utterances | non-commercial | Real product-style phrases, research licence only. | + +### Negatives — which set the DET curve + +| Corpus | Scale | Licence | Character | +|---|---|---|---| +| Common Voice | thousands of hours, many languages | CC0 | Cleanest licence in speech, huge speaker diversity. The default negative pool. | +| LibriSpeech / Libri-Light | ~1k h labelled / ~60k h unlabelled | CC BY 4.0 | Read speech — tonally unlike ambient audio, but cheap volume. | +| People's Speech | ~30k h | CC-BY / CC-BY-SA | One of the two largest permissive sets. | +| VoxPopuli | ~400k h unlabelled | CC0 | European Parliament recordings; enormous and unencumbered. | +| AMI Meeting Corpus | ~100 h | CC BY 4.0 | Spontaneous multi-party speech with overlap and crosstalk — much closer to what a device actually rejects all day. | + +### Noise and impulse responses + +| Corpus | Content | Licence | Role | +|---|---|---|---| +| MUSAN | ~109 h music / speech / noise | CC BY 4.0 | Standard additive-noise set in Kaldi-lineage recipes. | +| OpenSLR SLR28 | simulated + real RIRs | permissive | Standard reverberation augmentation; gives you distance and room. | +| FSD50K | labelled environmental sound | CC BY | Targeted hard negatives: television, kitchen clatter, dogs. | +| DEMAND | 16-channel real noise, 18 scenes | CC BY-SA | Realistic scene noise where a synthetic mix would be too clean. | +| WHAM! / UrbanSound8K | noise | CC BY-NC | Useful, but non-commercial. | + +## 10. The working recipe + +No open corpus contains *your* wake word, so the current standard approach — +and it works well — is to skip the recording entirely: + +1. **Synthesize positives with TTS.** Piper (MIT) plus `piper-sample-generator` + produces tens of thousands of variants of an arbitrary phrase across voices, + rates and pitches. This is exactly what openWakeWord and microWakeWord + (ESPHome) do; both ship their pipelines permissively. +2. **Augment hard.** RIR convolution from SLR28, additive noise from MUSAN and + FSD50K across 0–20 dB SNR, gain and mild speed perturbation, simulated + microphone response. +3. **Mine hard negatives from MSWC** — every keyword within small phonetic edit + distance of the target phrase. +4. **Bulk negatives from Common Voice, AMI and People's Speech**, ideally + hundreds of hours, because the false-accept target is measured per hour. +5. **Hold out real recorded positives** — even a couple of hundred utterances + from real speakers at real distances — as the *evaluation* set. + +> **The failure mode to design against.** Synthetic training with synthetic +> evaluation will lie to you cheerfully. TTS positives are too clean and too +> centred in prosody space; a model can score beautifully against them and fall +> apart on a real talker three metres from the microphone. The recorded hold-out +> set is not optional polish — it is the only thing standing between a promising +> number and a working detector. + +## 11. Licence traps + +Fine for a paper, unusable in a product: **TED-LIUM 3** is CC BY-NC-**ND**; +**VoxCeleb** is non-commercial and has had takedown friction; **GigaSpeech**'s +subsets carry mixed terms; **WHAM!** and **UrbanSound8K** are non-commercial. +**AudioSet** distributes labels, not audio — the recordings are YouTube +references and are not redistributable. + +Two further checks: both openWakeWord's and microWakeWord's *models* are trained +on synthetic data whose upstream TTS voice licences are separate from the code +licence and must be verified independently; and terms on these sets do change — +a licence confirmed at project start is not a licence confirmed at ship. + +--- + +### Confidence notes + +- Figures are as published by each project and were not re-counted here. +- Licence characterizations are a starting point for diligence, not a legal + opinion — verify each corpus at the point of use. +- Two items carry known uncertainty and are flagged in place: Hey Snips + availability, and the upstream voice licences behind synthetic-positive + pipelines. diff --git a/docs/wake-word-plan.md b/docs/wake-word-plan.md new file mode 100644 index 0000000..3327f14 --- /dev/null +++ b/docs/wake-word-plan.md @@ -0,0 +1,376 @@ +# Building `mutap.wake~` — implementation proposal + +*Proposal, rev 1 — 4 September 2026. **Nothing committed beyond this document:** +no code written, no repository changed apart from these docs. Background and +corpus survey in [`wake-word-briefing.md`](wake-word-briefing.md).* + +Formatted version: + +--- + +## 1. The proposal + +> Add a wake-word spotter to the Tap family by promoting two shared pieces down +> into DspTap, extending MuTap's existing `tools/ml` pipeline to a second task, +> and shipping one Max external. No new framework, no TFLite Micro dependency, +> no new repository until a second consumer justifies one. + +The bet behind this plan is that **MuTap has already solved the hard +infrastructure problem**, and solved it for exactly this shape of task. +`nn_suppressor.h` is a hand-written, allocation-free, `noexcept` GRU running on +ERB band energies, with its geometry carried as a value in a versioned weights +header, a Python trainer whose feature module is the declared single source of +truth, and a parity test pinning the C++ against it. A keyword spotter is the +same machine with a different head and a different loss. + +What does *not* exist yet: a mel/PCEN front end, an int8 or DS-CNN inference +path, a keyword training corpus, and — the piece that actually decides whether +this works — an evaluation harness measuring false accepts per hour rather than +accuracy. + +The plan sequences those four, deliberately putting the meter before the model. + +## 2. Where it lands + +``` +MuTap-Max mutap.afc~ mutap.aec~ [mutap.wake~] ← new, M7 + ↑ submodule pin +MuTap fd_kalman.h nn_suppressor.h [kws.h] ← new, M5 + tools/ml ───────────────────── [tools/ml/kws] ← extended, M3 + │ ↑ submodule pin + │ refactored onto + ↓ +DspTap fft.h yin.h [log_mel.h] [nn/] + ↑ new, M1 ↑ promoted, M2 +``` + +**The load-bearing move is the promotion.** M2 lifts the dense/GRU arithmetic +out of `nn_suppressor.h` into a shared `tap::dsp::nn` and refactors the +suppressor to consume it — so the spotter inherits kernels a shipping, tested +object already exercises, and the family gains one inference substrate instead +of two. This is the same promotion the family has already run three times: +`fft.h` out of MuTap and AmbiTap's duplicate copies, `sample_traits.h` from +SampleRateTap, `yin.h` out of the TapTools pitchaccum kernel. + +Two of the three new pieces (M1, M2) belong unambiguously in DspTap, which is +what makes the repository question for the third low-stakes: M1 and M2 are +correct under every option, so nothing blocks on deciding where the spotter +itself lives. + +TapTools is the wrong home and worth ruling out explicitly: its charter is +musical, object-level kernels, and a keyword spotter is neither. + +## 3. What already exists, and what it saves + +| Existing asset | What it does today | Role for the spotter | +|---|---|---| +| `nn_suppressor.h` | Dense → GRU → dense, float32, allocation-free, noexcept, geometry carried by the weights | The inference kernels, promoted in M2. The spotter is a different head on the same arithmetic. | +| `nn_geometry` / MUNN0002 | Model geometry as a value, validated at load; one inference path serves 16 kHz hop-64 and 48 kHz hop-256 | Copy the pattern verbatim for `kws_geometry` and a `MUKW0001` header. Retraining at a new geometry must not require a code change. | +| `tools/ml/features.py` | Declared single source of truth for band definitions and normalization | The precedent to copy for KWS features — and the discipline that makes the parity test meaningful. | +| `tools/ml/test_parity.py` | Pins C++ inference against the Python trainer's output | The single most valuable test in the project. A spotter that disagrees with its trainer fails silently and looks like a data problem. | +| `tools/ml/README.md` | The licensing map and measured benchmark that justified the hybrid design | The template for M3's dataset card. Same question, same rigour, different corpus. | +| Cortex-M55 QEMU rig + `scripts/icount.py` | On-target test subset in CI, instruction counting via a QEMU plugin | The embedded profile and its performance ratchet come essentially free in M6. | +| DspTap `fft.h` backend pattern | Ooura golden model; CMSIS-Helium and vDSP float32 backends re-presenting the exact contract | The mel front end's FFT, and the pattern for any future accelerated kernel. | +| DspTap `sample_traits.h` | float / Q15 / Q31 format core with documented Q-format ladders | Route to a fixed-point front end for M33-class targets, opt-in per existing convention. | + +**What this rules out:** no TFLite Micro, no CMSIS-NN dependency, no ONNX +runtime. The family's demonstrated position is hand-written inference against a +documented numeric contract, and a spotter is small enough — tens of thousands +of parameters — that this stays the cheaper option. Importing a runtime would +also break the M55 and Hexagon story MuTap already has working. + +## 4. Licensing map + +Same exercise `tools/ml/README.md` ran for the suppressor, applied to a wake +word. Comparable conclusion: every input can be MIT, CC0, CC BY or +self-generated, provided positives are synthesized rather than borrowed. + +| Component | Licence | Role | Shippable? | +|---|---|---|---| +| Speech Commands v2 | CC BY 4.0 | Benchmark; harness bring-up | yes, with attribution | +| MSWC | CC BY 4.0 | Hard negatives, phonetic near-misses | yes, with attribution | +| Common Voice | CC0 | Bulk negatives | yes | +| AMI Meeting Corpus | CC BY 4.0 | Conversational negatives | yes, with attribution | +| MUSAN + OpenSLR SLR28 | CC BY / permissive | Noise and RIR augmentation | yes | +| Piper + piper-sample-generator | MIT | Synthetic positives | code yes — **verify each voice's upstream corpus licence separately** | +| openWakeWord / microWakeWord | permissive | Pipeline design reference | yes — we reimplement, share no weights | +| Hey Snips, Qualcomm KSD | research / NC | Comparison only, if at all | **no** | +| PyTorch | BSD-3 | Trainer | never shipped — inference is dependency-free | +| Trained spotter weights | ours | The deliverable | yes, if every input above holds | + +Patent posture is comparable to the suppressor's: the windowed-classifier + +posterior-smoothing structure is published academic work from 2014 onward with +permissively licensed implementations, and the shipped inference would be a few +tens of thousands of parameters implemented from scratch. As before — not a +legal opinion; a commercial embedded product still warrants a freedom-to-operate +review. + +## 5. The numeric contracts to pin + +House rule: each header documents its geometry, conventions, normalization and +latency *as numbers*, and the tests pin them. Changing any of the following is a +breaking change for every consumer and every trained model. + +### `tap::dsp::basic_log_mel` + +- Frame length, hop, window (sqrt-Hann or Hann — stated, not implied), geometry + fixed at construction per the `prepare()`-buys-worst-case rule. +- Mel scale formula, and whether bands are Slaney- or HTK-normalized. These + differ *silently*; a model trained against one and run against the other + degrades in a way that looks like a data bug. +- Filter normalization: unit-area or unit-peak triangles. +- Log floor and any affine output normalization, as literal constants — the + suppressor's `k_log_floor` / `k_shift` / `k_scale` is the pattern. +- Latency in samples, stated. +- PCEN's smoother coefficient, gain, bias, power and epsilon, each as a number, + with the plain-log path available as the default. + +### `tap::dsp::nn` + +- Weight layout and gate ordering per layer, matching PyTorch's, exactly as + `nn_suppressor_weights` already documents it. +- Which accumulations run in double even in the float profile, and why — the + family's existing convention for numerically fragile recursions. +- For any int8 path: per-channel vs per-tensor scales, the single rounding + point, and saturation behaviour, as `sample_traits.h` already sets out for + Q15/Q31. + +### `tap::mu::kws` + +- Geometry as a value, carried by the weights and validated at load. +- Posterior smoothing window, confidence window, combination rule, refractory + period — all as numbers, all settable, all defaulted. +- The declared operating point: threshold, and the recall / false-accepts-per-hour + pair it was measured at, on a named evaluation set. +- Honest limits stated in the header, per house rule: the phrase it was trained + on, the distances and SNRs it was evaluated over, and that it is a + single-channel detector with no beamforming. + +## 6. Milestones + +Staged in the HANDOFF.md manner: each stage has a crisp pass criterion and a +committed regression fixture, so a failure is caught at the layer that caused +it. The one ordering choice worth defending is **M4 before M5** — the harness +before the model — the same call HANDOFF.md's M2 made when it declared the +closed-loop simulator a deliverable in its own right. + +### M0 — Settle the home and the phrase *(decision)* + +Two decisions, both cheap now and expensive later: which repository owns the +spotter (§9), and what the wake phrase actually is. The phrase is a technical +decision, not a branding one — three or four syllables, unusual phonotactics, +distinctive stress, not a substring of common speech. A poorly chosen phrase +costs a permanent penalty on the DET curve that no amount of training recovers. + +**Pass:** phrase chosen, with its phonetic-confusability shortlist written down; +repository decision recorded in HANDOFF.md. + +### M1 — The mel front end *(DspTap)* + +`include/tap/dsp/log_mel.h` — `basic_log_mel` with the double golden +model and float32 embedded profile, geometry fixed at construction, `noexcept` +and allocation-free processing, riding the existing `real_fft`. PCEN as a +documented option on the same object. + +Typed GoogleTest battery pinning every contract point above, plus float/double +cross-precision agreement. C ABI exposure in `tools/capi` and the `dsptap_py` +bridge, so the notebooks measure the shipping C++ rather than a Python +restatement. + +**Pass:** agreement with a reference mel implementation at a stated tolerance on +a fixed test signal, with the tolerance committed. PCEN's gain-tracking pinned +on a level-stepped input. README section added and the primitive count bumped, +per the existing checklist. + +### M2 — Promote the inference kernels *(DspTap · MuTap)* + +Lift the dense and GRU arithmetic out of `nn_suppressor.h` into `tap::dsp::nn`, +add depthwise-separable convolution and a streaming activation cache, and +refactor `nn_suppressor` to consume the promoted kernels. Pure refactor on +MuTap's side — no behaviour change intended. + +This milestone most repays being done early and most punishes being done late: +every subsequent stage builds on kernels a shipping, tested object already +exercises. + +**Pass:** `test_nn_suppressor.cpp` and `tools/ml/test_parity.py` pass unchanged, +and the M55 instruction count for the suppressor does not regress. A behaviour +change here shows up as a parity failure, which is exactly what that test is +for. + +### M3 — Corpus and dataset builder *(tools/ml)* + +A `kws_features.py` declared as source of truth alongside the existing +`features.py`; TTS positive synthesis via Piper; augmentation over MUSAN and +SLR28; hard-negative mining from MSWC by phonetic edit distance; bulk negatives +from Common Voice and AMI. One command rebuilds the dataset from a committed +manifest. + +Deliverable alongside the code: a **dataset card** in the shape of the existing +licensing map — per-corpus counts, hours, licences and attribution text. + +**Pass:** dataset rebuilds reproducibly from the manifest on a clean checkout, +and the card accounts for every hour of audio with a licence. Recorded hold-out +set collected separately and never seen by training. + +### M4 — The evaluation harness, before any model *(tools/ml · tests)* + +DET curve tooling: false-rejection rate against false-accepts-per-hour over +hundreds of hours of negatives, with the per-utterance / per-hour asymmetry +built into the meter rather than bolted on. Threshold sweep, refractory +handling, committed report format. + +Proven out against a deliberately trivial baseline — a band-energy threshold — +so the meter is known to work before it judges anything that matters. + +**Pass:** the harness produces a sane DET curve for the trivial baseline: +near-total recall at an absurd false-accept rate, collapsing to zero recall as +the threshold tightens. If that curve looks wrong, the meter is wrong, and +finding out here is the entire point of the milestone. + +### M5 — The spotter *(MuTap)* + +`kws.h` — streaming DS-CNN or dilated TDNN over the M1 features and M2 kernels, +with `kws_geometry` carried in a `MUKW0001` weights header, posterior smoothing +and confidence combination as documented constants, and a refractory period. +PyTorch trainer and exporter beside the existing ones; parity test against the +trainer. + +Architecture choice deferred to here rather than decided up front: with M4 in +place it becomes a measurement rather than an argument. + +**Pass:** parity with the trainer to float tolerance, and a stated operating +point measured on the *recorded* hold-out set — not the synthetic one — +committed as a regression baseline the way the RIR fixtures are. Target to aim +at: ≥ 95 % recall at ≤ 1 false accept per hour. Whatever is actually achieved is +what gets written down. + +### M6 — Embedded profile and the ratchet *(DspTap · MuTap)* + +Front end and inference through the bare-metal M55 QEMU rig already in CI; +instruction count per 10 ms hop measured with `scripts/icount.py` and committed +as a budget CI ratchets against. Q15 front-end profile via `sample_traits.h` if +the M33 class is a real target; int8 inference path if the budget demands it. + +**Pass:** on-target subset green under QEMU, instruction budget committed, and +any fixed-point profile agreeing with the float golden model within a stated, +tested tolerance. + +### M7 — `mutap.wake~` *(MuTap-Max)* + +One external, matching the sibling naming convention. Signal inlet; bang outlet +on detection; confidence float outlet for metering and threshold-setting by ear. +Attributes for threshold, refractory period, model path. Reference page and help +patcher demonstrating live detection with a visible confidence meter, so a user +can see the margin rather than guess at it. + +**Pass:** loads and behaves correctly in Max on both platforms, macOS binary +universal, and validated against a live microphone at conversational distance — +not only against files. + +> **Sequencing note.** M1 and M2 are worth doing regardless of whether the wake +> word ships. A mel front end is a primitive several Tap libraries would use, +> and consolidating the inference kernels removes a duplication that will +> otherwise appear the moment any second learned object is added. If the project +> stops after M2, the family is still better off — a useful property for a +> speculative effort to have. + +## 7. CI and the ratchet + +Four gates, three of which already exist and need only extending: + +- **Contract tests** — typed GoogleTest batteries in DspTap, Catch2 in MuTap. +- **Python↔C++ parity** — trainer and shipping inference agree to float + tolerance on committed fixture input. Catches the entire class of bug where a + feature definition drifts between the two sides and the model quietly + degrades. +- **On-target subset under QEMU** — the M55 rig MuTap already runs, extended to + the front end and spotter. +- **Instruction-count ratchet** — a committed budget per 10 ms hop enforced by + `scripts/icount.py`, so an innocuous-looking change that doubles the always-on + cost is caught in review rather than on hardware. + +Deliberately *not* in CI: the DET evaluation. It needs hundreds of hours of +audio and a trained model, so it belongs in the notebook verification layer — +executed, committed, re-executed when behaviour changes, exactly as +`notebooks/pitchshift.ipynb` is. The standing promise applies: every performance +claim measured, not remembered, traceable to the cell that produced it. + +## 8. Risks, honestly + +**The synthetic-positive gap.** The most likely failure is a model that scores +beautifully on TTS positives and disappoints on a real talker across a room. +Mitigation is structural rather than clever: the recorded hold-out set in M3, +and M5's pass criterion measured on it. If the gap is large, the answer is more +augmentation realism and more real recordings — both slow, neither surprising. + +**The negative corpus is a storage and time problem.** Hundreds of hours of +negatives is hundreds of gigabytes decoded, and the remote development +containers already have limited disk and a network policy that blocks some +dataset hosts — the RIR-fixture note in HANDOFF.md hit exactly this. Assume +corpus assembly happens locally and enters the repo as manifests and derived +features, not audio. + +**Charter drift.** MuTap's stated charter is adaptive filters for audio +cleaning, and its name is literally the LMS step size. A keyword spotter is +neither an adaptive filter nor audio cleaning. The counter-argument: MuTap is +already the family's speech library, already contains a learned non-adaptive +component, and already has the embedded rigs — and a second repository would +either duplicate that or pin MuTap anyway. Worth deciding on purpose rather than +by drift; see §9. + +**Scope creep toward speaker verification.** Tier 3 of the cascade usually +carries speaker ID, and it is tempting. It is a separate project with its own +corpora, enrolment UX and privacy posture. Recommend explicitly out of scope. + +**The plan's own weakest estimate.** M5's architecture and training loop is the +only milestone with genuine unknowns in it — everything else is either a known +refactor or a known harness. If a schedule is needed, treat M1–M4 as reasonably +estimable and M5 as the one to time-box with a decision point rather than +estimate. + +## 9. Open decisions + +**Which repository owns the spotter.** MuTap with a widened charter, or a new +sibling library pinning DspTap. M1 and M2 are correct under both, so this does +not block until M5. +*Recommend MuTap, charter restated as portable speech DSP for embedded targets. +Revisit only if a second consumer for keyword spotting appears in the family — +at which point the promotion pattern makes moving it cheap.* + +**The wake phrase.** Needed at M0 and genuinely load-bearing: syllable count and +phonetic distinctiveness set a ceiling on the achievable DET curve. +*Yours to choose. Technical constraints: three to four syllables, unusual +phonotactics, not a substring of common English.* + +**Model architecture.** DS-CNN, dilated TDNN, or a small GRU reusing the +promoted kernels directly. +*Defer to M5 and decide by measurement. The GRU path is cheapest to reach +because M2 delivers it; DS-CNN is the stronger default on microcontroller-class +targets. With M4 in place this is a measurement, not an argument.* + +**Fixed-point front end.** Whether M6 includes a Q15 mel profile — real work, +and only pays off on M33-class parts without usable float. +*Defer until a target is named. The M55 has float; the existing `sample_traits` +convention makes this opt-in per primitive precisely so it can wait.* + +**Whether to ship weights at all.** An alternative shape: ship `mutap.wake~` as +a runtime that loads a user-supplied model, with a documented training pipeline +and no bundled phrase. Sidesteps corpus assembly entirely and suits a Max +audience who may want their own phrase. +*Worth considering seriously as a first release, with a bundled model following +once the recorded evaluation set exists. It reorders the plan rather than +shortening it — M4 still has to happen before anyone can tell whether a trained +model is any good.* + +--- + +### Provenance + +- Proposal only — no code written, no repository changed apart from these docs. +- Milestone pass criteria state **targets, not measurements**. No number in this + document is a measured result. +- Existing-asset descriptions were read from the working checkouts of DspTap, + MuTap, TapTools and TapTools-Max on 4 September 2026. +- Corpus licences summarized from the companion briefing and require + verification at the point of use. From a8c5d9783f76ed5040e04ce935f1bdaa5b53b141 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 4 Sep 2026 02:59:55 +0000 Subject: [PATCH 02/14] docs: adversarial audit of the wake-word proposal Sixteen-agent audit of docs/wake-word-plan.md rev 1: seven adversarial reviewers (front end, codebase fit, architecture, evaluation, licensing, embedded, scope), a skeptic per lens re-checking every finding against the checkouts, a completeness critic and a cross-lens judge. Verdict: the design direction survives; the claim that MuTap has already solved the infrastructure for the learned path does not. nn_suppressor is instantiated only in double everywhere, is absent from the M55 on-target filter and the icount scenarios, and test_parity.py is a hand-run double-only random-weights check outside CI. Host sample-rate policy and the non-power-of-two 10 ms hop are undecided. The report ranks twelve root issues, records seven findings the judge discounted, and lists the amendments a rev 2 of the plan should carry. HANDOFF.md's wake-word entry points at the audit. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G --- HANDOFF.md | 2 +- docs/wake-word-audit.md | 440 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 441 insertions(+), 1 deletion(-) create mode 100644 docs/wake-word-audit.md diff --git a/HANDOFF.md b/HANDOFF.md index b00e259..c994bdd 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -756,6 +756,6 @@ Resolved since rev 1: ~~license~~ (MIT), ~~core language~~ (header-only C++20), Resolved since rev 3: ~~Max external naming~~ — settled (Rev 4): **`mutap.afc~`** (rename from the `mutap.defeed~` placeholder) and **`mutap.aec~`** for the new echo canceller, an acronym pair matching the literature. The rename executes in Stage 2 of "The next effort" above. Still open: -- **Wake-word detection — proposed, not started.** A background briefing and a staged implementation proposal landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md) and [`docs/wake-word-plan.md`](docs/wake-word-plan.md). The proposal reuses the `nn_suppressor` / `tools/ml` machinery rather than importing a runtime, and its first two milestones (a DspTap mel front end; promoting the dense/GRU kernels into `tap::dsp::nn`) pay off whether or not the spotter itself ships. Two decisions are yours before anything starts: whether MuTap's charter widens to hold it, and what the wake phrase is. +- **Wake-word detection — proposed, not started.** A background briefing and a staged implementation proposal landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md) and [`docs/wake-word-plan.md`](docs/wake-word-plan.md). The proposal reuses the `nn_suppressor` / `tools/ml` machinery rather than importing a runtime, and its first two milestones (a DspTap mel front end; promoting the dense/GRU kernels into `tap::dsp::nn`) pay off whether or not the spotter itself ships. Two decisions are yours before anything starts: whether MuTap's charter widens to hold it, and what the wake phrase is. An adversarial audit of the proposal ([`docs/wake-word-audit.md`](docs/wake-word-audit.md)) found the design direction sound but the "infrastructure already solved" claim wrong for the learned path (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI) and adds host sample-rate policy to the M0 decisions; its amendment list is the input to a rev 2 of the plan. - **Default engine in the external** — `@kalman` off (classic NLMS) is the shipping default purely on seniority; the measured case for flipping it is in `tests/test_fd_kalman.cpp` and book chapter 1. Decide after real-room listening. - **RIR fixtures, the measured half** — the fixture pipeline is built and three physically-modeled rooms (image-source, documented geometry) are committed baselines with regression tests. What remains yours: which MEASURED rooms join them — an academic dataset room (MYRiAD is the PEM-AFROW group's own database; openAIR is the other usual source; check each room's license allows redistribution in an MIT repo) and/or your own swept-sine measurements. Either way it is one command per room: `python3 tools/fixtures/make_rir_fixtures.py --from-wav room.wav myroom --source ""`, then a test with a freshly measured threshold. (The dataset hosts are unreachable from the remote dev container's network policy, so the WAVs have to enter via a commit.) diff --git a/docs/wake-word-audit.md b/docs/wake-word-audit.md new file mode 100644 index 0000000..22e24e8 --- /dev/null +++ b/docs/wake-word-audit.md @@ -0,0 +1,440 @@ +# Adversarial audit of the `mutap.wake~` proposal + +*Audit, rev 1 — 4 September 2026, of [`wake-word-plan.md`](wake-word-plan.md) +rev 1 and its companion [briefing](wake-word-briefing.md). Nothing in the plan +was changed by this audit; every amendment below is a recommendation for rev 2.* + +Formatted version: + +--- + +## 1. Verdict + +**The plan's design direction survives. Its central empirical claim does not +survive as written.** + +The direction — hand-written, allocation-free inference against a documented +numeric contract, promoted into DspTap, geometry carried as a value in a +versioned weights header, a Python feature module as the single source of +truth, no imported runtime — is sound, and nothing in the audit argues for a +framework instead. The reuse story holds for the *patterns*. + +The claim in §1 that "MuTap has already solved the hard infrastructure problem, +and solved it for exactly this shape of task" is wrong about the *oracles*. +Checked against the tree: the learned suppressor is instantiated only in +double, everywhere (its tests, the parity driver, the shipping Max external); +the instruction-count rig's "suppressor" scenario is the classical +`residual_suppressor`, not the GRU; the Cortex-M55 on-target filter does not +select the learned suppressor's tests at all; and the parity script is a +hand-run, double-only comparison against a numpy reference with random +weights, absent from CI. So the float32 embedded profile, on-target execution, +per-hop cost measurement and CI parity for a learned object are all *new work*, +not inheritance — and M2's and M6's pass criteria, which assume they exist, are +vacuous as written. + +Two further infrastructure questions were tolerable for a post-filter and are +load-bearing for a spotter, and the plan does not decide either: what happens +to a 16 kHz-trained model on a 44.1/48/96 kHz Max host, and how a 10 ms hop +(never a power of two) coexists with a geometry validator that demands one. + +Amended bet, which the plan should state instead: *MuTap has established the +patterns and the rigs; the learned path's float, on-target, cost and parity +oracles must be built before M2, and host-rate policy decided at M0, before +the spotter can inherit anything.* With those amendments the milestone +structure stands. + +## 2. How the audit was run + +Sixteen agents, three phases, every finding checked against the working +checkouts of MuTap, DspTap and MuTap-Max rather than against the plan's own +description of them. + +| Phase | Agents | What they did | +|---|---|---| +| Attack | 7 | One adversarial reviewer per lens: DSP front end and contracts; codebase fit; model architecture; evaluation and data; licensing and IP; embedded profile and performance; scope and sequencing. Each returned findings with file-and-line evidence and a stated refutation test. | +| Verify | 7 | One skeptic per lens, instructed to *refute* each finding and default to refuted when the evidence did not support the claim. Each verdict re-cites the code. | +| Critique | 2 | A completeness critic looking for what no lens examined; a cross-lens judge merging duplicates into root issues, ranking them, and naming findings the verifiers should not have let through. | + +42 findings were raised. 8 were confirmed outright, 34 were partially confirmed +(a real point, but overstated or mis-cited, with the accurate statement +recorded), 0 were refuted. The zero refutation rate is itself a signal that the +skeptics leaned lenient, so the judge's list of doubtful survivors (§5) is +carried here as a correction. The critic added five findings the lenses +structurally missed. The main session then spot-checked the top-ranked root +issue's evidence by hand; every claim in it re-checks (§3, issue 1). + +Full agent output, including every finding's evidence and the verifier's +reasoning, is in the session transcript; this document carries the merged +result. + +## 3. Root issues, ranked by how much they change what happens before work starts + +### 1. The infrastructure M2 says it inherits does not exist for the learned path — *critical* + +**What the plan says.** §3: the promotion inherits "kernels a shipping, tested +object already exercises"; the M55 rig and ratchet "come essentially free"; +`test_parity.py` "pins C++ inference against the Python trainer's output" and +is one of the three CI gates that "already exist". M2 passes when +`test_nn_suppressor.cpp` and `test_parity.py` pass unchanged and "the M55 +instruction count for the suppressor does not regress". + +**What the code says.** + +- `nn_suppressor` is instantiated as `` only: in + `tests/test_nn_suppressor.cpp`, in `tools/ml/nn_infer.cpp`, and in + `mutap.aec_tilde.cpp` (`aec_chain_nn`). There is no float typed test + and it is absent from `tests/test_float32.cpp`. The float profile the + promotion is supposed to preserve is never observed anywhere. +- `bench/icount/icount_main.cpp` includes only `fd_kalman.h` and + `postfilter.h`. The ratchet's "suppressor" scenario is the classical one. The + "does not regress" clause in M2 measures code M2 does not touch. +- `tests/bare_metal_main.cpp` applies a positive filter for the M55 leg; no + `NnSuppressor` pattern is in it. The learned path has never executed + on-target. +- `tools/ml/test_parity.py` compares C++ double output against `nn.py`'s numpy + reference using *random* weights at the legacy 16 kHz / 22-band geometry, + gain path only, run by hand behind an OFF-by-default CMake option. It never + loads trainer output, never covers the shipping 48 kHz geometry, and the + only "parity" job in `ci.yml` is the unrelated branchless check. +- The plan's own §5 asks the promoted kernels to document "which accumulations + run in double"; today the answer is none — every dot product accumulates in + `Sample`. + +**What to change.** Add a milestone between M1 and M2 that builds the oracle +the plan assumes: `nn_suppressor` typed tests with a float/double pin; +the learned suppressor in both on-target filters; an `nn_suppressor` icount +scenario seeded on M55 and Hexagon *before* the refactor; `test_parity.py` +wired into CI and run against exported trainer weights at the shipping 48 kHz +geometry. Rewrite §3's `test_parity.py` and "essentially free" rows and M2's +pass criterion to name these gates. Decide the accumulator-precision contract +before the refactor, not during it. + +### 2. Host sample rate and front-end frame geometry are undecided, and the pattern to "copy verbatim" cannot serve them — *critical* + +**What the plan says.** §3: copy the `nn_geometry` / MUNN0002 pattern +"verbatim"; §5 lists frame, hop and window without units, fmin/fmax or a +host-rate policy; §6 and §7 speak of "per 10 ms hop"; M7 must work "against a +live microphone". + +**What the code says.** + +- Every corpus in §4 is 16 kHz. Max runs at 44.1, 48 or 96 kHz. The + suppressor's precedent is retrain-per-rate: `features.py` and + `nn_suppressor.h` define bands over 0..sr/2, `make_dataset.py` spectrally + upsamples LibriSpeech 3× for the 48 kHz model, and `mutap.aec~` only prints + "detuned bands" and runs anyway when the model rate disagrees with the host. + HANDOFF already files "per-rate (44.1 kHz) models" as an open follow-up. +- No runtime resampler exists in the pinned tree. DspTap holds the FIR + substrate but its README says the converters live in SampleRateTap and + RatioTap, repositories the plan never names. The only resampler in any of + the three repos is test-support code. +- `nn_suppressor_weights::valid()` requires `hop >= 16` and a power of two. A + 10 ms hop is 160, 441 or 480 samples — none is a power of two — so the + conflict bites at *every* rate, not only 44.1 kHz. The suppressor's + `frame == 2·hop == FFT size` convention does not hold for a 25 ms / 10 ms + mel front end either, and §5 omits FFT size and zero-padding placement. +- §5 also omits streaming frame alignment (`features.py` documents that + GRU-state parity depends on exactly this), bin-0 / DC policy (the inherited + band builder forces bin 0 into band 0, the opposite of a mel with + fmin > 0), pre-emphasis, PCEN initial state and reset semantics, and the + identity of the "reference mel implementation" M1 is scored against — no + such reference exists in the family today. + +**What to change.** Make host-rate policy an M0 decision with its cost stated: +per-host-rate models on upsampled corpora (the suppressor's route, with the +band-limited-above-8 kHz risk named), or a real-time decimator as a new DspTap +primitive built on `kaiser.h` and `fir_kernels.h`. Specify in §5 that +`kws_geometry` decouples hop from FFT size (zero-padded power-of-two FFT), and +add the missing contract lines. Name the M1 reference — a committed numpy mel +in `tools/ml`, written before M1 — and make the float/double tolerance a +measured number on a signal that excites every band. + +### 3. The evaluation the plan calls its decisive instrument cannot yet produce a trustworthy number — *critical* + +**What the plan says.** M4 is "the meter before the model"; it passes when the +trivial baseline's DET curve "looks" right. M3 draws bulk negatives from +Common Voice and hard negatives from MSWC, and delivers a recorded hold-out +set "collected separately". §8 says corpus assembly happens locally and enters +the repo as "manifests and derived features, not audio". + +**What the audit found.** + +- No train/dev/eval split is specified for negatives. MSWC is force-aligned + *out of* Common Voice, so a corpus-level split leaks: false accepts per hour + would be measured on material the model trained on. +- M4's pass criterion cannot fail — any threshold sweep produces a curve that + collapses as the threshold tightens. The scoring rules that actually break + a KWS meter (hit window around a positive, one-hit-per-utterance, false + accept merging under refractory, the hours denominator) are undefined + anywhere in the plan, and a band-energy baseline cannot expose accounting + errors in them. +- The recorded hold-out set has no size, talker count, distance/SNR matrix, + owner, or consent/licence row. It is the basis for M5's only measured + number and its generalization claim is undefined. The RIR fixture precedent + the plan cites requires provenance per committed fixture. +- "Derived features" for 300 h of audio is roughly 17 GB of float32; the repo + cannot hold it and no feature store is named. As specified, the DET notebook + is re-executable on one machine only. The two existing ML notebooks already + show this failure mode. +- The FA/hour side of the operating point has no named held-out negative + corpus, and the 1 FA/hour target is looser than the briefing's product + figure without saying it is a first-release target. +- M4 as written depends on M3's corpus and M0's phrase even though §4's table + says to bring the harness up on Speech Commands first. + +**What to change.** Before M3: utterance- and speaker-disjoint splits with the +MSWC↔Common Voice overlap handled; a manifest schema (release ids, archive +checksums, decoder versions, front-end contract version); a named feature +store outside git; a hold-out specification (N talkers, condition matrix, +owner, consent row). Replace M4's pass with a planted-event oracle test with +exact expected recall and FA/hour that *must fail* on an accounting error, and +name harness↔`kws.h` decision-stage parity as a second parity surface. Let M4 +run on Speech Commands so it genuinely precedes M0 and M3. + +### 4. The licensing map inverts the risk — *major* + +**What the plan says.** §4: every input can be MIT, CC0, CC BY or +"self-generated", with TTS positives the clean case; MUSAN + SLR28 are +"CC BY / permissive"; openWakeWord / microWakeWord are "permissive". + +**What the audit found.** (Network access in this environment was limited; +these are flagged for verification at the point of use, as the plan's own +provenance section requires.) + +- Both voices that `piper-sample-generator` documents descend from the + Blizzard 2013 Lessac corpus, whose licence is research-only and names + speech-recognition products as excluded. TTS positives are the *least* + clean input, not a self-generated one. Lessac-free Piper voices exist. +- "Yes, with attribution" has no delivery mechanism: nothing in M5 or M7 + places attribution in the shipped weights header (stamped MIT) or the Max + package. The suppressor precedent credits LibriSpeech in two READMEs and + nowhere a user of the external would see. +- SLR28 is Apache 2.0, not CC BY, and its real-RIR subset carries third-party + RWCP/REVERB/AIR provenance the plan does not check. +- openWakeWord and microWakeWord code is Apache 2.0; their models are + CC BY-NC-SA and their pre-computed feature sets CC BY-NC with WHAM/CHiME6 + upstream. The row should say "design reference only; no code, models or + features imported". +- Common Voice and MSWC carry a downloader promise not to attempt speaker + re-identification. It does not restrict the plan's use, but belongs in the + dataset card. +- The patent paragraph borrows the suppressor's "comparable" prior-art + phrasing in a field where the 2014-onward publishers are the patent holders. + Neither patent the audit surfaced reads on a single-stage reimplementation; + the fix is narrower wording, not a change of design. + +**What to change.** Make voice-lineage verification an M0 pass criterion and +choose a Lessac-free voice; reword "self-generated" to "TTS-derived, +lineage-verified per voice"; relabel the SLR28 and openWakeWord rows; add a +hold-out consent/licence row; require every feature shard to be regenerated +from manifest audio; add a provenance/attribution block to the MUKW exporter +output and a notices file to the Max package as M5/M7 pass items. + +### 5. M3 pre-commits the M5 architecture the plan says it defers — *major* + +The dataset builder never decides the label form. A clip-level label is +implicit, which trains a clip classifier; but without a tracked keyword +endpoint (or frame/word alignment) the M5 "measure, don't argue" comparison +cannot include frame-wise or end-anchored losses on equal footing, and the +briefing itself (§4, end-to-end variants) says alignment is the hard part. +The `kws` contract also lacks a detection-latency constant (hops from phrase +end to the bang) and, if a GRU stays on the shortlist, a hidden-state +reset-or-carried policy for streaming training. Single-class versus per-word +head follows from the same decision. + +**What to change.** Add label form and endpoint tolerance under RIR/tempo +augmentation to M3's pass; add detection latency and streaming-state policy to +§5's `kws` contract. + +### 6. No cost ceiling exists anywhere, and the ratchet is not a budget — *major* + +`scripts/icount.py` is a ±3 % whole-binary drift gate seeded from the first +measurement. The plan calls it "a committed budget per 10 ms hop", which it is +not, and states no ceiling on MACs, instructions, weight or activation RAM, or +detection latency. M5's architecture choice therefore has one axis (DET) when +it needs two. M6's int8 option is coupled to the architecture it is supposed +to be independent of: the briefing says recurrent state is awkward to +quantize, so int8-as-requirement constrains the shortlist. + +**What to change.** State numeric ceilings up front; derive a per-hop figure +from the corpus in M6 and describe `icount.py` honestly as a drift gate plus a +separate absolute assertion; either decide float32-only for M55 now or say +that the int8 option constrains M5. + +### 7. M2 bundles consumer-less kernels and skips DspTap's checklist — *major* + +"Kernels a shipping, tested object already exercises" is true of dense and GRU +and false of the depthwise-separable convolution and streaming activation +cache M2 also adds: nothing runs them and no numpy reference covers them until +M5. M2's pass criterion also omits the DspTap-side typed battery, README +section and tidy items that M1 lists for itself, and does not say that the +weights format and dimension validation stay MuTap-owned or that the +promotion implies DspTap PR → MuTap pin bump → MuTap-Max pin bump. + +**What to change.** Restrict M2 to dense + GRU with the full DspTap checklist; +move DS-conv and the cache to M5, or give M2 a torch-versus-streaming kernel +fixture; record the ownership and pin sequence. + +### 8. The feature contract gets two owners in two repos — *major* (critic) + +M1 freezes `log_mel` — including every PCEN constant — as a DspTap contract +before any trainer exists, while M3 declares `kws_features.py` the source of +truth. Under DspTap's rule every feature iteration during M3/M5 becomes a +cross-repo breaking change through a submodule pin, and feature parameters are +exactly what a KWS pipeline iterates on. + +**What to change.** State the ownership rule: `log_mel.h` owns the +formula-level contract (mel scale, normalization, window, FFT/padding, bin-0 +policy); everything a trainer might tune (band count, fmin/fmax, log floor, +affine normalization, all PCEN parameters) is runtime geometry carried by the +MUKW weights. Retraining then never touches DspTap. + +### 9. "No CMSIS-NN" contradicts the family's filed roadmap — *major* (critic) + +§3 rules out CMSIS-NN as "the family's demonstrated position", but HANDOFF and +`tools/ml/README.md` both file "int8 + CMSIS-NN for the M55 path" as the +suppressor's next step, and the `fft.h` precedent the same table cites *is* an +optional CMSIS backend behind a fixed contract. Because M2 makes `tap::dsp::nn` +the single substrate, the plan silently decides the suppressor's roadmap too. + +**What to change.** Adopt the `fft.h` rule instead: a scalar golden kernel with +an optional, opt-in accelerated backend that must re-present the exact +contract. Design the M2 kernel API with that seam, and reconcile HANDOFF and +the README with whichever decision is taken. + +### 10. The DET will be measured on the Python model, not the shipping C++ — *major* (critic) + +§7's "standing promise" is that every performance claim is measured on the +shipping code, and §5 requires `kws.h` to declare the operating point "it was +measured at". But M5 lists no C ABI for `kws.h`, the M4 harness is Python in +`tools/ml`, and DspTap's `dsptap_py` bridge is not what MuTap's `tools/ml` +loads. The family's own v1 notebook shows the failure: the hybrid it measured +ran a numpy net over numpy features. + +**What to change.** Add a `mutap_kws_*` C ABI family and `mutap_ffi` binding to +M5; state that the final DET runs the C++ engine; make the C++-measured +operating point the one the header declares; run the M4 oracle test against +both paths. + +### 11. No compute budget or hardware is named — *major* (critic) + +The reference trainer is CPU-only by construction (no device option) and took +about 2.5 h for roughly 1 h of audio. The KWS corpus is two orders of magnitude +larger, plus tens of thousands of TTS syntheses through a PyTorch pipeline of +its own, and M5 wants several full trainings. §8 names disk and network as the +corpus problem and never mentions compute. + +**What to change.** Name where training runs, a per-run time target and a +device path in the sibling trainer; size a development negative set for +architecture selection and reserve the full corpus for the final DET; give +TTS synthesis its own runtime estimate and toolchain list. + +### 12. Lesser items — *minor* + +- **Release shape.** §9 calls "ship a runtime, not weights" a reorder. It is a + shortening: it removes the phrase as a blocking decision, the shipped + dataset card, the recorded hold-out set and the measured operating point, + while keeping M3's pipeline as the product. Say so, and reconcile with + HANDOFF's "before anything starts". +- **Operating point plumbing.** Nothing links the DET-measured threshold to + the constants in `kws.h`, the MUKW header and the Max `@threshold` default. + Put them in the MUKW payload and have the exporter emit them from the + notebook report. +- **"Ship" is undefined.** Neither repo has a tag, release, artifact upload or + signing step; users clone and symlink. State what ship means today, or add + a release job. Add the customary docs stage (book chapter, README status + rows, notices file) that every prior effort under HANDOFF carried. +- **Charter.** "Portable speech DSP for embedded targets" does not cover the + AFC/music side; the repo decision affects M3/M4 file placement, not only + M5; HANDOFF and the plan disagree on when it blocks. +- **Factual errors in asset descriptions.** MuTap's tests are GoogleTest, not + Catch2. The "`prepare()`-buys-worst-case rule" is not locatable in DspTap, + MuTap or MuTap-Max (quote DspTap's construction-time wording instead). + `sample_traits.h` is a FIR format core, not a fixed-point front-end route; + DspTap has no fixed-point FFT. `tools/ml` is suppressor-shaped and the + second task needs its own parity driver, CMake target and README re-scope. + Provenance omits MuTap-Max, which the plan clearly read. + +## 4. What survives unchanged + +Worth saying, since an audit lists only defects: + +- Promoting a mel front end and the dense/GRU kernels into DspTap is right and + pays off whether or not the spotter ships. Every lens agreed. +- No TFLite Micro, no ONNX runtime. Nothing argued for one. +- Geometry as a value in a versioned weights header, validated at load, with + the trainer's feature module as the source of truth — the pattern is + correct; only its ownership boundary (issue 8) needs stating. +- The meter before the model. The ordering is right; the meter's pass + criterion is what needs fixing. +- Recommending MuTap as the home. The audit found no reason for a new + repository. +- The honesty of the provenance section. No number in the plan is presented + as measured, and the audit found none that was. + +## 5. Findings the audit discounted + +The cross-lens judge flagged seven survivors the verifiers should have +rejected or narrowed. They are recorded here so the correction is visible: + +| Finding | Why discounted | +|---|---| +| float/double agreement "unpassable" on log features | The verifier's own measurement showed sqrt-Hann float and double log-band features agree to 0.000 decades; worst case found was ~0.27 feature units on an on-bin sine. Residual is a one-line M1 note: measure the tolerance on a signal that excites every band. | +| Network policy rationale "stale" | One day's proxy reachability from one container is an environment observation, not a plan defect. The disk half stands (issue 3). | +| Always-on duty cycle "never measured" | Misreading: the briefing's "few percent" is utilization of a continuously running spotter, which per-hop instruction counting measures directly. Only the latency omission survives (issue 5). | +| Patent paragraph | Neither cited patent reads on the plan's single-stage structure. A wording preference, kept in issue 4 as such. | +| "`prepare()` rule does not exist in the family" | Only shown absent from the three repos present; TapTools was not available. Downgraded to "not locatable here". | +| Three-repo pin sequence undecided | Documented in DspTap's CLAUDE.md and HANDOFF; residual is one clause in M2. | +| Charter wording inadequate | A judgement call, not a verifiable defect. Only the HANDOFF-versus-plan inconsistency is concrete. | + +## 6. Recommended amendments to the plan, in order + +For rev 2 of `wake-word-plan.md`: + +1. **§1** — restate the bet as patterns-and-rigs inherited, oracles to be + built (issue 1). +2. **§3** — correct the `test_parity.py`, "essentially free", `sample_traits`, + Catch2 and `prepare()` rows; replace "no CMSIS-NN" with the `fft.h` + backend rule (issues 1, 9, 12). +3. **§4** — TTS row reworded and voice lineage made an M0 gate; SLR28 and + openWakeWord rows relabelled; hold-out consent row and attribution + delivery added; patent paragraph narrowed (issue 4). +4. **§5** — add FFT size/padding, frame alignment, bin-0, pre-emphasis, PCEN + initial state, fmin/fmax in Hz, detection latency, streaming-state policy; + state the DspTap-versus-weights ownership rule (issues 2, 5, 8). +5. **M0** — add host-rate policy and voice lineage to the decisions; note + what runtime-first actually removes (issues 2, 4, 12). +6. **M1** — name the reference implementation; make the cross-precision + tolerance a measured number (issue 2). +7. **New M1.5** — build the float32, on-target, icount and CI-parity oracles + for the learned suppressor before touching it (issue 1). +8. **M2** — dense + GRU only, full DspTap checklist, ownership and pin + sequence stated (issue 7). +9. **M3** — splits, manifest schema, feature store, hold-out spec, label + form, compute budget, dataset card rows (issues 3, 5, 11). +10. **M4** — planted-event oracle pass criterion; runs on Speech Commands + first; decision-stage parity named (issue 3). +11. **M5** — C ABI and `mutap_ffi` binding; cost ceilings as a second axis; + operating point emitted by the exporter into the MUKW payload (issues 6, + 10, 12). +12. **M6** — per-hop figure derived from the corpus; ratchet described + honestly; int8 conditional on M5 (issue 6). +13. **M7** — define ship; add the docs stage and notices file (issue 12). +14. **HANDOFF entry** — reconcile on what blocks when (issue 12). + +--- + +### Provenance + +- Audit only — no code written; the plan and briefing are unchanged. +- Agent claims about the code were re-verified by a skeptic per lens and, for + the top-ranked issue, by hand in the main session against + `tests/test_nn_suppressor.cpp`, `tools/ml/nn_infer.cpp`, + `mutap.aec_tilde.cpp`, `bench/icount/icount_main.cpp`, + `tests/bare_metal_main.cpp`, `.github/workflows/ci.yml`, + `include/mutap/nn_suppressor.h` and `HANDOFF.md`. +- Licence and patent statements (issue 4) were made with limited network + access and are flagged for verification at the point of use; they are not a + legal opinion. +- TapTools and TapTools-Max were not available to the audit; claims the plan + makes about them are marked unverifiable rather than wrong. From 391265f78028a38e791691c5f4e499fbc3dc8b3e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 4 Sep 2026 03:21:09 +0000 Subject: [PATCH 03/14] docs: wake-word proposal rev 2, carrying every audit amendment Rewrites docs/wake-word-plan.md from the audit's fourteen-item list. The bet is restated as patterns and rigs inherited, oracles to be built. A new M2 builds the float, on-target, instruction-count and CI-parity verification for the learned suppressor before M3 promotes its kernels; rev 1's M2-M7 become M3-M8. M0 grows from two decisions to five (repository, host-rate policy, release shape, phrase, TTS voice lineage) plus a named compute budget, with recommendations in section 9: MuTap, a fixed internal 16 kHz with the external decimating, and runtime-first. Section 5 adds the contract points the audit found missing (FFT size and padding, frame alignment, bin-0, pre-emphasis, PCEN state, detection latency, streaming-state policy, accumulator precision) and an ownership rule: log_mel.h owns formulas, the MUKW weights carry every tunable. Section 4 corrects the TTS, SLR28 and openWakeWord rows and adds attribution delivery. M4 specifies speaker-disjoint splits across MSWC and Common Voice, a manifest schema, a feature store outside git, a hold-out specification and the label-form decision. M5's pass becomes a planted-event oracle that can fail and runs through the C ABI. Section 7 describes icount.py as a drift gate and adds absolute cost ceilings as targets. "No CMSIS-NN" is replaced by the fft.h backend rule. Factual errors in the asset table (Catch2, prepare() rule, sample_traits role) are fixed. The audit and the HANDOFF entry note that rev 2 landed and how the milestone numbers shifted. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G --- HANDOFF.md | 2 +- docs/wake-word-audit.md | 5 +- docs/wake-word-plan.md | 754 ++++++++++++++++++++++++---------------- 3 files changed, 466 insertions(+), 295 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index c994bdd..9cabe66 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -756,6 +756,6 @@ Resolved since rev 1: ~~license~~ (MIT), ~~core language~~ (header-only C++20), Resolved since rev 3: ~~Max external naming~~ — settled (Rev 4): **`mutap.afc~`** (rename from the `mutap.defeed~` placeholder) and **`mutap.aec~`** for the new echo canceller, an acronym pair matching the literature. The rename executes in Stage 2 of "The next effort" above. Still open: -- **Wake-word detection — proposed, not started.** A background briefing and a staged implementation proposal landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md) and [`docs/wake-word-plan.md`](docs/wake-word-plan.md). The proposal reuses the `nn_suppressor` / `tools/ml` machinery rather than importing a runtime, and its first two milestones (a DspTap mel front end; promoting the dense/GRU kernels into `tap::dsp::nn`) pay off whether or not the spotter itself ships. Two decisions are yours before anything starts: whether MuTap's charter widens to hold it, and what the wake phrase is. An adversarial audit of the proposal ([`docs/wake-word-audit.md`](docs/wake-word-audit.md)) found the design direction sound but the "infrastructure already solved" claim wrong for the learned path (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI) and adds host sample-rate policy to the M0 decisions; its amendment list is the input to a rev 2 of the plan. +- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. Five decisions are yours at M0 before anything starts, with recommendations in the plan's §9: repository (MuTap), host-rate policy (fixed internal 16 kHz, external decimates), release shape (runtime-first, no bundled phrase), the wake phrase if bundled, and where training runs. - **Default engine in the external** — `@kalman` off (classic NLMS) is the shipping default purely on seniority; the measured case for flipping it is in `tests/test_fd_kalman.cpp` and book chapter 1. Decide after real-room listening. - **RIR fixtures, the measured half** — the fixture pipeline is built and three physically-modeled rooms (image-source, documented geometry) are committed baselines with regression tests. What remains yours: which MEASURED rooms join them — an academic dataset room (MYRiAD is the PEM-AFROW group's own database; openAIR is the other usual source; check each room's license allows redistribution in an MIT repo) and/or your own swept-sine measurements. Either way it is one command per room: `python3 tools/fixtures/make_rir_fixtures.py --from-wav room.wav myroom --source ""`, then a test with a freshly measured threshold. (The dataset hosts are unreachable from the remote dev container's network policy, so the WAVs have to enter via a commit.) diff --git a/docs/wake-word-audit.md b/docs/wake-word-audit.md index 22e24e8..a009649 100644 --- a/docs/wake-word-audit.md +++ b/docs/wake-word-audit.md @@ -2,7 +2,10 @@ *Audit, rev 1 — 4 September 2026, of [`wake-word-plan.md`](wake-word-plan.md) rev 1 and its companion [briefing](wake-word-briefing.md). Nothing in the plan -was changed by this audit; every amendment below is a recommendation for rev 2.* +was changed by this audit; every amendment below is a recommendation for rev 2. +Rev 2 of the plan, carrying all fourteen, landed alongside this document; its +milestone numbers shifted (rev 1's M2–M7 are rev 2's M3–M8, and the "M1.5" +proposed below became M2).* Formatted version: diff --git a/docs/wake-word-plan.md b/docs/wake-word-plan.md index 3327f14..d325038 100644 --- a/docs/wake-word-plan.md +++ b/docs/wake-word-plan.md @@ -1,8 +1,11 @@ # Building `mutap.wake~` — implementation proposal -*Proposal, rev 1 — 4 September 2026. **Nothing committed beyond this document:** +*Proposal, rev 2 — 4 September 2026. **Nothing committed beyond this document:** no code written, no repository changed apart from these docs. Background and -corpus survey in [`wake-word-briefing.md`](wake-word-briefing.md).* +corpus survey in [`wake-word-briefing.md`](wake-word-briefing.md). Rev 2 +carries every amendment from the [adversarial audit](wake-word-audit.md) of +rev 1; the audit refers to rev 1's milestone numbers, and the mapping is given +in §6.* Formatted version: @@ -15,362 +18,527 @@ Formatted version: and shipping one Max external. No new framework, no TFLite Micro dependency, > no new repository until a second consumer justifies one. -The bet behind this plan is that **MuTap has already solved the hard -infrastructure problem**, and solved it for exactly this shape of task. -`nn_suppressor.h` is a hand-written, allocation-free, `noexcept` GRU running on -ERB band energies, with its geometry carried as a value in a versioned weights -header, a Python trainer whose feature module is the declared single source of -truth, and a parity test pinning the C++ against it. A keyword spotter is the -same machine with a different head and a different loss. - -What does *not* exist yet: a mel/PCEN front end, an int8 or DS-CNN inference -path, a keyword training corpus, and — the piece that actually decides whether -this works — an evaluation harness measuring false accepts per hour rather than -accuracy. - -The plan sequences those four, deliberately putting the meter before the model. +The bet behind this plan, stated as the audit corrected it: **MuTap has +established the patterns and the rigs, not the oracles.** `nn_suppressor.h` is +a hand-written, allocation-free, `noexcept` GRU running on ERB band energies, +with its geometry carried as a value in a versioned weights header and a Python +feature module declared the single source of truth. Those patterns transfer +directly. What does *not* transfer — because it was never built for the learned +path — is the verification around it: the suppressor is instantiated only in +double everywhere it runs, has never executed on the Cortex-M55 rig, has no +instruction-count scenario, and its parity script is a hand-run, double-only, +random-weights check outside CI. A keyword spotter is the same machine with a +different head and a different loss, *once those oracles exist*. Building them +is a milestone of its own (M2), and it is worth doing whether or not the +spotter ships, because it is the verification the suppressor should already +have. + +What does not exist yet at all: a mel/PCEN front end, a policy for the host +sample rate, a streaming convolutional inference path, a keyword corpus with +honest splits, and — the piece that actually decides whether this works — an +evaluation harness measuring false accepts per hour with a pass criterion that +can fail. + +The plan sequences those, deliberately putting the oracles before the refactor +and the meter before the model. ## 2. Where it lands ``` -MuTap-Max mutap.afc~ mutap.aec~ [mutap.wake~] ← new, M7 - ↑ submodule pin -MuTap fd_kalman.h nn_suppressor.h [kws.h] ← new, M5 - tools/ml ───────────────────── [tools/ml/kws] ← extended, M3 +MuTap-Max mutap.afc~ mutap.aec~ [mutap.wake~] ← new, M8 + ↑ submodule pin (transitive to DspTap) +MuTap fd_kalman.h nn_suppressor.h [kws.h] ← new, M6 + tools/ml ───────────────────── [tools/ml/kws] ← extended, M4 │ ↑ submodule pin │ refactored onto ↓ -DspTap fft.h yin.h [log_mel.h] [nn/] - ↑ new, M1 ↑ promoted, M2 +DspTap fft.h yin.h [log_mel.h] [nn/] [decimate.h] + ↑ new, M1 ↑ promoted, M3 ↑ new, M1 (if M0 picks it) ``` -**The load-bearing move is the promotion.** M2 lifts the dense/GRU arithmetic +**The load-bearing move is the promotion.** M3 lifts the dense/GRU arithmetic out of `nn_suppressor.h` into a shared `tap::dsp::nn` and refactors the -suppressor to consume it — so the spotter inherits kernels a shipping, tested -object already exercises, and the family gains one inference substrate instead -of two. This is the same promotion the family has already run three times: -`fft.h` out of MuTap and AmbiTap's duplicate copies, `sample_traits.h` from -SampleRateTap, `yin.h` out of the TapTools pitchaccum kernel. - -Two of the three new pieces (M1, M2) belong unambiguously in DspTap, which is -what makes the repository question for the third low-stakes: M1 and M2 are -correct under every option, so nothing blocks on deciding where the spotter -itself lives. - -TapTools is the wrong home and worth ruling out explicitly: its charter is -musical, object-level kernels, and a keyword spotter is neither. +suppressor to consume it — so the spotter inherits kernels a shipping object +exercises, and the family gains one inference substrate instead of two. This is +the same promotion the family has already run three times: `fft.h` out of MuTap +and AmbiTap's duplicate copies, the FIR substrate from SampleRateTap, `yin.h` +out of the TapTools pitchaccum kernel. + +The promotion implies a three-repo sequence — DspTap PR, MuTap pin bump, +MuTap-Max pin bump (MuTap-Max reaches DspTap only transitively through MuTap) +— and an ownership rule: the weights format, its loader and dimension +validation stay MuTap-owned; DspTap holds only arithmetic over caller-provided +spans. MUNN0002 images are unaffected. + +M1 and M3 belong unambiguously in DspTap, which is what keeps the repository +question for the spotter itself low-stakes. TapTools is the wrong home and +worth ruling out explicitly: its charter is musical, object-level kernels, and a +keyword spotter is neither. ## 3. What already exists, and what it saves +Read from the checkouts, with the audit's corrections applied. + | Existing asset | What it does today | Role for the spotter | |---|---|---| -| `nn_suppressor.h` | Dense → GRU → dense, float32, allocation-free, noexcept, geometry carried by the weights | The inference kernels, promoted in M2. The spotter is a different head on the same arithmetic. | -| `nn_geometry` / MUNN0002 | Model geometry as a value, validated at load; one inference path serves 16 kHz hop-64 and 48 kHz hop-256 | Copy the pattern verbatim for `kws_geometry` and a `MUKW0001` header. Retraining at a new geometry must not require a code change. | -| `tools/ml/features.py` | Declared single source of truth for band definitions and normalization | The precedent to copy for KWS features — and the discipline that makes the parity test meaningful. | -| `tools/ml/test_parity.py` | Pins C++ inference against the Python trainer's output | The single most valuable test in the project. A spotter that disagrees with its trainer fails silently and looks like a data problem. | -| `tools/ml/README.md` | The licensing map and measured benchmark that justified the hybrid design | The template for M3's dataset card. Same question, same rigour, different corpus. | -| Cortex-M55 QEMU rig + `scripts/icount.py` | On-target test subset in CI, instruction counting via a QEMU plugin | The embedded profile and its performance ratchet come essentially free in M6. | -| DspTap `fft.h` backend pattern | Ooura golden model; CMSIS-Helium and vDSP float32 backends re-presenting the exact contract | The mel front end's FFT, and the pattern for any future accelerated kernel. | -| DspTap `sample_traits.h` | float / Q15 / Q31 format core with documented Q-format ladders | Route to a fixed-point front end for M33-class targets, opt-in per existing convention. | - -**What this rules out:** no TFLite Micro, no CMSIS-NN dependency, no ONNX -runtime. The family's demonstrated position is hand-written inference against a -documented numeric contract, and a spotter is small enough — tens of thousands -of parameters — that this stays the cheaper option. Importing a runtime would -also break the M55 and Hexagon story MuTap already has working. +| `nn_suppressor.h` | Dense → GRU → dense over ERB band energies; allocation-free, noexcept; geometry carried by the weights. **Instantiated only as ``** in its tests, the parity driver and the shipping external. Every dot product accumulates in `Sample`. | The inference kernels, promoted in M3 *after* M2 gives them a float oracle. The spotter is a different head on the same arithmetic. | +| `nn_geometry` / MUNN0002 | Model geometry as a value, validated at load; 16 kHz hop-64 and 48 kHz hop-256 served by one inference path. **Validator requires a power-of-two hop.** | Copy the geometry-as-value pattern for `kws_geometry` and a `MUKW0001` header; do *not* copy the hop constraint. Retraining at a new geometry must not require a code change. | +| `tools/ml/features.py` | Declared single source of truth for band definitions and normalization; documents that streaming-state parity depends on frame alignment. | The precedent for `kws_features.py` — with the ownership boundary of §5 so the two sources of truth cannot disagree. | +| `tools/ml/test_parity.py` | C++ **double** inference against `nn.py`'s numpy reference, **random weights**, legacy 16 kHz geometry, gain path only, hand-run behind `MUTAP_BUILD_ML_TOOLS=OFF`. Not a CI job. | The shape of the test the spotter needs. M2 turns it into one: CI-run, both profiles, exported trainer weights, shipping geometry. | +| `tools/ml/README.md` | The licensing map and measured benchmark that justified the hybrid design; files "int8 + CMSIS-NN for the M55 path" as next step. | The template for M4's dataset card, and a roadmap this plan must reconcile with (§5, `tap::dsp::nn`). | +| Cortex-M55 QEMU rig + `scripts/icount.py` | On-target positive-filter test subset in CI; whole-binary instruction count per scenario with a ±3 % drift gate against `bench/baselines.json` (`fdkf`, `chain` at 16 k and 48 k, on m55 and hexagon). **No learned-path scenario; `NnSuppressor` not in the on-target filter.** | The rig the embedded profile runs on, once M2 adds the learned scenarios. The ratchet is a drift gate; the budget is a separate absolute assertion (§7). | +| DspTap `fft.h` backend pattern | Ooura golden model; CMSIS-Helium and vDSP float32 backends re-presenting the exact contract, certified by parity tests. | The mel front end's FFT, and the rule for any accelerated NN backend: optional, opt-in, parity-pinned against the scalar golden path. | +| DspTap `sample_traits.h`, `kaiser.h`, `fir_kernels.h` | The FIR substrate: float / Q15 / Q31 format core with documented Q-format ladders, Kaiser prototype design, dot kernels. No fixed-point FFT; the rate converters themselves live in SampleRateTap and RatioTap. | The material for a polyphase decimator (M1, if M0 picks that route) and the *convention* for any later Q15 front end — not a fixed-point front end in itself. | + +**What this rules out:** TFLite Micro and ONNX runtime as dependencies. The +family's demonstrated position is hand-written inference against a documented +numeric contract, and a spotter is small enough — tens of thousands of +parameters — that this stays the cheaper option. Accelerated backends (CMSIS-NN, +Helium intrinsics) are *not* ruled out: they follow the `fft.h` rule — a scalar +golden kernel, an optional backend behind the same contract, parity tests +between them — which keeps the suppressor's filed int8 + CMSIS-NN follow-up +reachable rather than foreclosed. ## 4. Licensing map Same exercise `tools/ml/README.md` ran for the suppressor, applied to a wake -word. Comparable conclusion: every input can be MIT, CC0, CC BY or -self-generated, provided positives are synthesized rather than borrowed. +word, corrected by the audit. Conclusion, narrowed: every input can be MIT, +Apache 2.0, CC0 or CC BY, **provided the TTS voices are lineage-verified** — +synthetic positives are the least clean input, not the cleanest. | Component | Licence | Role | Shippable? | |---|---|---|---| -| Speech Commands v2 | CC BY 4.0 | Benchmark; harness bring-up | yes, with attribution | -| MSWC | CC BY 4.0 | Hard negatives, phonetic near-misses | yes, with attribution | -| Common Voice | CC0 | Bulk negatives | yes | +| Speech Commands v2 | CC BY 4.0 | Benchmark; harness bring-up (M5 runs on this first) | yes, with attribution | +| MSWC | CC BY 4.0; no-reidentification term | Hard negatives, phonetic near-misses. **Force-aligned out of Common Voice** — splits must be speaker-disjoint across both. | yes, with attribution | +| Common Voice | CC0; no-reidentification term | Bulk negatives | yes | | AMI Meeting Corpus | CC BY 4.0 | Conversational negatives | yes, with attribution | -| MUSAN + OpenSLR SLR28 | CC BY / permissive | Noise and RIR augmentation | yes | -| Piper + piper-sample-generator | MIT | Synthetic positives | code yes — **verify each voice's upstream corpus licence separately** | -| openWakeWord / microWakeWord | permissive | Pipeline design reference | yes — we reimplement, share no weights | +| MUSAN | CC BY 4.0 | Additive noise | yes, with attribution | +| OpenSLR SLR28 | Apache 2.0 (simulated RIRs); real-RIR subset carries RWCP / REVERB / AIR third-party terms | Reverberation augmentation | simulated subset yes; **real subset only after its upstream terms are checked** | +| Piper + `piper-sample-generator` | MIT (code); **voice models carry their training corpus's terms** — the two voices the generator documents descend from Blizzard 2013 / Lessac, research-only, excluding speech-recognition products | Synthetic positives, TTS-derived | code yes; **each voice lineage-verified at M0, Lessac-free voices chosen** | +| openWakeWord / microWakeWord | Apache 2.0 (code); models CC BY-NC-SA 4.0; pre-computed feature sets CC BY-NC with WHAM / CHiME-6 upstream | Design reference only | design yes — **no code, models or feature sets imported**; every feature shard regenerated from manifest audio | | Hey Snips, Qualcomm KSD | research / NC | Comparison only, if at all | **no** | +| Recorded hold-out set | ours; consent and permitted use recorded per talker | The evaluation set | committed only with its consent row | | PyTorch | BSD-3 | Trainer | never shipped — inference is dependency-free | -| Trained spotter weights | ours | The deliverable | yes, if every input above holds | +| Trained spotter weights | ours, derived from CC BY inputs | The deliverable | yes, **with attribution delivered**: a provenance block in the MUKW payload and a notices file in the Max package | -Patent posture is comparable to the suppressor's: the windowed-classifier + -posterior-smoothing structure is published academic work from 2014 onward with -permissively licensed implementations, and the shipped inference would be a few -tens of thousands of parameters implemented from scratch. As before — not a -legal opinion; a commercial embedded product still warrants a freedom-to-operate -review. +Patent posture: the shipped structure is a single-stage, single-channel +windowed classifier with posterior smoothing, reimplemented from the published +literature (Chen, Parada & Heigold 2014 onward) with no imported code and hence +no Apache patent grant to rely on. The 2014-onward publishers are also the +field's patent holders, so "comparable to the suppressor's" is not claimed. Not +a legal opinion; a commercial embedded product still warrants a +freedom-to-operate review. ## 5. The numeric contracts to pin -House rule: each header documents its geometry, conventions, normalization and -latency *as numbers*, and the tests pin them. Changing any of the following is a -breaking change for every consumer and every trained model. +House rule (DspTap's wording): geometry fixed at construction, every buffer +allocated there; processing `noexcept` and allocation-free; each header +documents its packing, conventions, normalization and latency *as numbers*, and +the tests pin them. Changing a contract point is a breaking change for every +consumer and every trained model. + +**Ownership rule, so two sources of truth cannot disagree.** `log_mel.h` owns +the *formula-level* contract: mel scale, filter normalization, window, FFT size +and zero-padding placement, frame alignment, bin-0 policy, pre-emphasis. Every +parameter a trainer might tune — band count, fmin/fmax, log floor, affine +normalization, all PCEN parameters — is *runtime geometry* carried by the MUKW +weights and validated at load. `kws_features.py` is the source of truth for +the *values*; `log_mel.h` for the *formulas*. Retraining never touches DspTap. ### `tap::dsp::basic_log_mel` -- Frame length, hop, window (sqrt-Hann or Hann — stated, not implied), geometry - fixed at construction per the `prepare()`-buys-worst-case rule. -- Mel scale formula, and whether bands are Slaney- or HTK-normalized. These - differ *silently*; a model trained against one and run against the other - degrades in a way that looks like a data bug. -- Filter normalization: unit-area or unit-peak triangles. -- Log floor and any affine output normalization, as literal constants — the - suppressor's `k_log_floor` / `k_shift` / `k_scale` is the pattern. -- Latency in samples, stated. -- PCEN's smoother coefficient, gain, bias, power and epsilon, each as a number, - with the plain-log path available as the default. +- Internal sample rate, frame length, hop and FFT size **in samples at that + rate** (the reference geometry: 16 kHz, 400 / 160 / 512), with the plain + statement that frame ≠ 2·hop and the FFT is zero-padded to a power of two + at the *end* of the frame. +- Streaming alignment: frame *t* ends at the newest sample; no centring, no + prepended zero frame. Latency in samples = frame length, stated. +- Window (Hann or sqrt-Hann — stated), pre-emphasis coefficient (or 0, stated). +- Mel scale formula; Slaney or HTK normalization; unit-area or unit-peak + triangles. These differ *silently*. +- Bin-0 policy: DC excluded (fmin > 0), stated as a number. +- Log floor **relative to the sample format**, and the affine normalization — + the suppressor's `k_log_floor` / `k_shift` / `k_scale` is the pattern — as + runtime parameters with defaults. +- PCEN: smoother coefficient, gain, bias, power and epsilon as runtime + parameters; initial smoother state and `reset()` semantics as a contract + point; plain-log as the default path. +- Float/double cross-precision tolerance as a **measured** number, on a test + signal that excites every band. + +### `tap::dsp::decimate` *(only if M0 picks the fixed-internal-rate route)* + +- Integer ratios 2, 3, 6 (32 k, 48 k, 96 k → 16 k); Kaiser-designed polyphase + FIR from `kaiser.h` over `fir_kernels.h`; stopband attenuation, transition + band and group delay as numbers; latency in samples at the host rate. ### `tap::dsp::nn` - Weight layout and gate ordering per layer, matching PyTorch's, exactly as `nn_suppressor_weights` already documents it. -- Which accumulations run in double even in the float profile, and why — the - family's existing convention for numerically fragile recursions. -- For any int8 path: per-channel vs per-tensor scales, the single rounding - point, and saturation behaviour, as `sample_traits.h` already sets out for - Q15/Q31. +- Accumulator precision per kernel, stated: today every dense and GRU dot + product accumulates in `Sample`, and M3 preserves that. Any later change is + a documented contract change with a parity delta. +- Streaming convolution: the activation cache's size and the causal delay per + layer as numbers; equivalence to the non-streaming forward pass pinned by a + torch-versus-streaming fixture. +- The `fft.h` backend rule: scalar golden kernel; any accelerated backend is + opt-in and must re-present the exact contract, parity-tested. For an int8 + backend: per-channel vs per-tensor scales, the single rounding point, + saturation — in the manner `sample_traits.h` sets out for Q15/Q31. ### `tap::mu::kws` -- Geometry as a value, carried by the weights and validated at load. +- Geometry as a value, carried by the weights and validated at load; the + validator accepts any hop ≥ 1, not only powers of two. +- Label form the model was trained with (clip-level with tracked endpoint, + frame-aligned, or alignment-free) and the head shape (single-class or + per-word) — decided in M4, recorded in the header. - Posterior smoothing window, confidence window, combination rule, refractory - period — all as numbers, all settable, all defaulted. -- The declared operating point: threshold, and the recall / false-accepts-per-hour - pair it was measured at, on a named evaluation set. -- Honest limits stated in the header, per house rule: the phrase it was trained - on, the distances and SNRs it was evaluated over, and that it is a - single-channel detector with no beamforming. + period — all as numbers, all settable, all defaulted, **all carried in the + MUKW payload** alongside the threshold, so a retrain updates one place and + the Max `@threshold` default reads from the model. +- Detection latency: hops from phrase end to the bang, as a number. +- Streaming-state policy (reset per window or carried) if a recurrent layer is + present. +- The declared operating point: threshold, and the recall / false-accepts-per- + hour pair it was measured at, on a **named** evaluation set — measured on the + C++ engine through the C ABI, never on the Python model. +- Honest limits stated in the header: the phrase, the internal rate and host + rates supported, the distances and SNRs evaluated over, single-channel, no + beamforming, no pre-roll, no VAD gating. ## 6. Milestones -Staged in the HANDOFF.md manner: each stage has a crisp pass criterion and a -committed regression fixture, so a failure is caught at the layer that caused -it. The one ordering choice worth defending is **M4 before M5** — the harness -before the model — the same call HANDOFF.md's M2 made when it declared the -closed-loop simulator a deliverable in its own right. - -### M0 — Settle the home and the phrase *(decision)* - -Two decisions, both cheap now and expensive later: which repository owns the -spotter (§9), and what the wake phrase actually is. The phrase is a technical -decision, not a branding one — three or four syllables, unusual phonotactics, -distinctive stress, not a substring of common speech. A poorly chosen phrase -costs a permanent penalty on the DET curve that no amount of training recovers. - -**Pass:** phrase chosen, with its phonetic-confusability shortlist written down; -repository decision recorded in HANDOFF.md. - -### M1 — The mel front end *(DspTap)* - -`include/tap/dsp/log_mel.h` — `basic_log_mel` with the double golden -model and float32 embedded profile, geometry fixed at construction, `noexcept` -and allocation-free processing, riding the existing `real_fft`. PCEN as a -documented option on the same object. - -Typed GoogleTest battery pinning every contract point above, plus float/double -cross-precision agreement. C ABI exposure in `tools/capi` and the `dsptap_py` -bridge, so the notebooks measure the shipping C++ rather than a Python -restatement. - -**Pass:** agreement with a reference mel implementation at a stated tolerance on -a fixed test signal, with the tolerance committed. PCEN's gain-tracking pinned -on a level-stepped input. README section added and the primitive count bumped, -per the existing checklist. - -### M2 — Promote the inference kernels *(DspTap · MuTap)* - -Lift the dense and GRU arithmetic out of `nn_suppressor.h` into `tap::dsp::nn`, -add depthwise-separable convolution and a streaming activation cache, and -refactor `nn_suppressor` to consume the promoted kernels. Pure refactor on -MuTap's side — no behaviour change intended. - -This milestone most repays being done early and most punishes being done late: -every subsequent stage builds on kernels a shipping, tested object already -exercises. - -**Pass:** `test_nn_suppressor.cpp` and `tools/ml/test_parity.py` pass unchanged, -and the M55 instruction count for the suppressor does not regress. A behaviour -change here shows up as a parity failure, which is exactly what that test is -for. - -### M3 — Corpus and dataset builder *(tools/ml)* - -A `kws_features.py` declared as source of truth alongside the existing -`features.py`; TTS positive synthesis via Piper; augmentation over MUSAN and -SLR28; hard-negative mining from MSWC by phonetic edit distance; bulk negatives -from Common Voice and AMI. One command rebuilds the dataset from a committed -manifest. - -Deliverable alongside the code: a **dataset card** in the shape of the existing -licensing map — per-corpus counts, hours, licences and attribution text. - -**Pass:** dataset rebuilds reproducibly from the manifest on a clean checkout, -and the card accounts for every hour of audio with a licence. Recorded hold-out -set collected separately and never seen by training. - -### M4 — The evaluation harness, before any model *(tools/ml · tests)* - -DET curve tooling: false-rejection rate against false-accepts-per-hour over -hundreds of hours of negatives, with the per-utterance / per-hour asymmetry -built into the meter rather than bolted on. Threshold sweep, refractory -handling, committed report format. - -Proven out against a deliberately trivial baseline — a band-energy threshold — -so the meter is known to work before it judges anything that matters. - -**Pass:** the harness produces a sane DET curve for the trivial baseline: -near-total recall at an absurd false-accept rate, collapsing to zero recall as -the threshold tightens. If that curve looks wrong, the meter is wrong, and -finding out here is the entire point of the milestone. - -### M5 — The spotter *(MuTap)* - -`kws.h` — streaming DS-CNN or dilated TDNN over the M1 features and M2 kernels, -with `kws_geometry` carried in a `MUKW0001` weights header, posterior smoothing -and confidence combination as documented constants, and a refractory period. -PyTorch trainer and exporter beside the existing ones; parity test against the -trainer. - -Architecture choice deferred to here rather than decided up front: with M4 in -place it becomes a measurement rather than an argument. - -**Pass:** parity with the trainer to float tolerance, and a stated operating -point measured on the *recorded* hold-out set — not the synthetic one — -committed as a regression baseline the way the RIR fixtures are. Target to aim -at: ≥ 95 % recall at ≤ 1 false accept per hour. Whatever is actually achieved is -what gets written down. - -### M6 — Embedded profile and the ratchet *(DspTap · MuTap)* - -Front end and inference through the bare-metal M55 QEMU rig already in CI; -instruction count per 10 ms hop measured with `scripts/icount.py` and committed -as a budget CI ratchets against. Q15 front-end profile via `sample_traits.h` if -the M33 class is a real target; int8 inference path if the budget demands it. - -**Pass:** on-target subset green under QEMU, instruction budget committed, and -any fixed-point profile agreeing with the float golden model within a stated, -tested tolerance. - -### M7 — `mutap.wake~` *(MuTap-Max)* - -One external, matching the sibling naming convention. Signal inlet; bang outlet -on detection; confidence float outlet for metering and threshold-setting by ear. -Attributes for threshold, refractory period, model path. Reference page and help -patcher demonstrating live detection with a visible confidence meter, so a user -can see the margin rather than guess at it. +Staged in the HANDOFF.md manner: each stage has a pass criterion that *can +fail* and a committed regression fixture. Rev 1's M2–M7 are rev 2's M3–M8; M2 +is new. + +### M0 — Decisions *(nothing built)* + +Five decisions, each cheap now and expensive later. + +1. **Repository.** MuTap with a widened charter (§9) or a new sibling. Affects + M4/M5 file placement, so it blocks at M4, not M6. +2. **Host-rate policy.** Either (a) the spotter runs at a fixed internal + 16 kHz and `mutap.wake~` owns the conversion — integer ratios via a new + DspTap decimator in M1, 44.1 kHz via RatioTap or deferred; or (b) a model + per host rate on spectrally upsampled corpora, the suppressor's route, + accepting that bands above 8 kHz are never excited in training and that + 44.1 kHz needs a third model. Recommendation in §9. +3. **Release shape.** Runtime-first (a user-supplied model, no bundled phrase) + or bundled weights. Decides whether the phrase blocks anything (§9). +4. **The wake phrase**, if bundled: three or four syllables, unusual + phonotactics, distinctive stress, not a substring of common English. A + poorly chosen phrase costs a permanent penalty on the DET curve that no + training recovers. +5. **TTS voice lineage.** Each Piper voice's training corpus and its terms, + written down; Lessac-derived voices excluded. + +**Pass:** all five recorded in HANDOFF.md, with the phonetic-confusability +shortlist if a phrase is chosen, and the compute budget of M4 named (where +training runs, and a per-run time target). + +### M1 — The mel front end, and the decimator *(DspTap)* + +`include/tap/dsp/log_mel.h` — `basic_log_mel` per the §5 contract, with +the double golden model and float32 embedded profile, riding the existing +`real_fft` with the FFT size decoupled from the hop. PCEN as a documented option +on the same object. If M0 chose route (a), `decimate.h` beside it. + +**Before the header:** a throwaway numpy mel in `tools/ml/kws_features.py`, +written first and committed, is the reference M1 is scored against — the +family has no reference mel today, and M1's pass needs one that exists before +the C++ does. + +Typed GoogleTest battery pinning every §5 contract point; float/double +agreement measured, not assumed; C ABI exposure in `tools/capi` and the +`dsptap_py` bridge; README section and primitive count per the DspTap +checklist; `.clang-tidy` clean under the clang front end. + +**Pass:** agreement with the committed numpy reference at a committed tolerance +on a fixed multi-band test signal; PCEN gain-tracking pinned on a level-stepped +input; PCEN reset semantics pinned; streaming output identical to whole-signal +output frame for frame (the alignment contract); decimator passband ripple, +stopband attenuation and latency pinned if built. + +### M2 — Oracles for the learned path *(MuTap)* — new in rev 2 + +Build the verification the suppressor should already have, before anything +refactors it: + +- `nn_suppressor` typed tests with a float/double cross-precision pin, + and its entry in `test_float32.cpp`. +- `NnSuppressor` patterns added to both on-target positive filters (M55 and + Hexagon). +- An `nn_suppressor` scenario in `bench/icount` at both shipping geometries, + baselines seeded on m55 and hexagon. +- `test_parity.py` promoted to a CI job: both profiles, **exported trainer + weights** (`pretrained/suppressor_v2_48k.munn`) at the shipping 48 kHz + geometry as well as random weights, run under `MUTAP_BUILD_ML_TOOLS=ON` in + `ci.yml`. + +**Pass:** every item above green in CI on the *unmodified* suppressor, and the +accumulator-precision contract of §5 written down as the observed behaviour. +This milestone is worth doing even if the project stops here. + +### M3 — Promote the inference kernels *(DspTap · MuTap)* + +Lift the dense and GRU arithmetic — only those — out of `nn_suppressor.h` into +`tap::dsp::nn`, and refactor `nn_suppressor` to consume it. Pure refactor on +MuTap's side; the float profile is now observed, so "no behaviour change" is +testable in both profiles. Full DspTap checklist for the new header. +Depthwise-separable convolution and the streaming activation cache arrive in +M6 with their consumer and their torch-versus-streaming fixture, not here. + +**Pass:** the M2 battery passes unchanged in both profiles; the CI parity job +passes at float tolerance on exported weights; the `nn_suppressor` icount +scenario is within the drift gate on both targets; DspTap's typed battery for +`tap::dsp::nn` pins layout, gate order and accumulator precision; MuTap and +MuTap-Max pins bumped. + +### M4 — Corpus, splits and dataset builder *(tools/ml/kws)* + +`kws_features.py` as source of truth for the feature *values* (the M1 numpy +reference, now the real thing); Piper synthesis over lineage-verified voices; +augmentation over MUSAN and the SLR28 simulated subset; hard-negative mining +from MSWC by phonetic edit distance; bulk negatives from Common Voice and AMI. +One command rebuilds the dataset from a committed manifest. + +Decided here, because the loss depends on it: the **label form** — clip-level +with a tracked keyword endpoint, frame/word-aligned, or alignment-free — and the +endpoint tolerance under RIR and tempo augmentation. Recorded in the manifest +and the MUKW header. + +**Splits:** train / dev / eval, speaker-disjoint, with MSWC clips assigned by +their Common Voice client id so the two corpora cannot leak into each other. +The eval negative set is named and never trained on; it is the FA/hour +denominator for every number in this plan. + +**Manifest schema:** corpus release ids, archive checksums, decoder versions, +`kws_features.py` contract version, split assignment, augmentation seeds. +**Feature store:** a named location outside git (derived features for 300 h are +≈ 17 GB float32); the repo carries manifests and the builder only. + +**Hold-out specification:** N talkers (target ≥ 10), distance × SNR condition +matrix, owner, consent and permitted-use row per talker, target ≥ 200 +utterances. Committed as a fixture with provenance, as the RIR fixtures are. + +**Compute:** the trainer gains a `--device` path; a 50 h development negative +set serves architecture selection, the full corpus only the final DET. + +Deliverable alongside the code: a **dataset card** — per-corpus counts, hours, +licences, attribution text, the no-reidentification terms, and the voice +lineage table. + +**Pass:** dataset rebuilds from the manifest on a clean checkout with the feature +store mounted; every hour of audio accounted for with a licence; splits verified +speaker-disjoint by script; hold-out fixture committed with its consent rows. + +### M5 — The evaluation harness, before any model *(tools/ml · tests)* + +DET curve tooling: false-rejection rate against false-accepts-per-hour, with the +scoring semantics **defined as numbers**: hit window around each positive's +endpoint, one hit per utterance, false-accept merging under the refractory +period, the hours denominator from the eval negative set. Threshold sweep and a +committed report format. + +Brought up on **Speech Commands** (`marvin` / `sheila` as name-shaped +positives) so it precedes M0's phrase and M4's corpus, then pointed at M4's +splits. Runs the engine through the C ABI so it measures shipping code; the +Python model is for training-time validation only. + +**Pass — a test that can fail:** a planted-event oracle. Synthetic streams with +known positive endpoints and known false-accept placements, scored by the +harness against hand-computed recall and FA/hour, exact to the utterance; a +deliberately mis-accounted variant must be rejected. The trivial band-energy +baseline is run as a sanity curve, not as the pass. Harness↔`kws.h` decision +stage parity pinned on the same streams. + +### M6 — The spotter *(MuTap)* + +`kws.h` per the §5 contract, over the M1 features and M3 kernels, with the +DS-conv / activation-cache kernels landing in `tap::dsp::nn` now, alongside +their consumer and a torch-versus-streaming fixture. `kws_geometry` in a +`MUKW0001` payload that also carries the decision-stage constants, threshold and +declared operating point. PyTorch trainer and exporter beside the existing +ones; `kws_infer.cpp` parity driver and CMake target; `mutap_kws_*` C ABI +(create-from-weights, push block, posterior and confidence readout, detection +events with sample timestamps) and its `mutap_ffi` binding; `tools/ml/README.md` +re-scoped to two tasks. + +Architecture — DS-CNN, dilated TDNN, or GRU — decided here by measurement on +**two axes**: the M5 DET on the development set, and cost against the §7 +ceilings. + +**Pass:** parity with the trainer to float tolerance in both profiles, CI-run; +the streaming fixture passes; a stated operating point measured **through the +C ABI on the recorded hold-out set** and committed as a regression baseline; +attribution block present in the exporter output. Target to aim at: ≥ 95 % +recall at ≤ 1 false accept per hour on the named eval negative set — a +first-release target, looser than the briefing's product figure. Whatever is +achieved is what gets written down. + +### M7 — Embedded profile and the budget *(DspTap · MuTap)* + +Front end, decimator (if any) and spotter through the M55 and Hexagon rigs; +`kws` scenarios added to `bench/icount` at the shipping geometry, baselines +seeded on both targets; the per-hop figure derived by dividing the scenario's +count by its hop count. The §7 ceilings asserted as absolute checks beside the +drift gate. Int8 backend only if the measured cost demands it *and* the M6 +architecture admits it; Q15 front end deferred until an M33-class target is +named (a new Q-format design, since DspTap has no fixed-point FFT). + +**Pass:** on-target subsets green on both rigs; both scenarios within the +drift gate and under the absolute ceilings; any accelerated or fixed-point +backend agreeing with the scalar golden path within a stated, tested tolerance. + +### M8 — `mutap.wake~` *(MuTap-Max)* + +One external on the `mutap.aec~` pattern: signal inlet; bang outlet on +detection; confidence float outlet; attributes for threshold (defaulting from +the loaded model), refractory period and model path; host-rate handling per +M0's decision, refusing rather than warning on an unsupported rate; a no-model +state that meters and never fires, for the runtime-first shape. Reference page +and help patcher with a visible confidence meter. Package-level notices file +carrying the dataset card's attribution text. + +"Ship" means what it means for the family today: source build and a Packages +symlink. A downloadable, signed package is a separate deliverable (tagged +build, artifact upload, macOS notarization) taken on only if wanted. + +The customary docs stage: a book chapter for the spotter with numbers only from +the DET notebook and the tests; a MuTap README status row; a MuTap-Max README +roadmap row. **Pass:** loads and behaves correctly in Max on both platforms, macOS binary -universal, and validated against a live microphone at conversational distance — -not only against files. - -> **Sequencing note.** M1 and M2 are worth doing regardless of whether the wake -> word ships. A mel front end is a primitive several Tap libraries would use, -> and consolidating the inference kernels removes a duplication that will -> otherwise appear the moment any second learned object is added. If the project -> stops after M2, the family is still better off — a useful property for a -> speculative effort to have. - -## 7. CI and the ratchet - -Four gates, three of which already exist and need only extending: - -- **Contract tests** — typed GoogleTest batteries in DspTap, Catch2 in MuTap. -- **Python↔C++ parity** — trainer and shipping inference agree to float - tolerance on committed fixture input. Catches the entire class of bug where a - feature definition drifts between the two sides and the model quietly - degrades. -- **On-target subset under QEMU** — the M55 rig MuTap already runs, extended to - the front end and spotter. -- **Instruction-count ratchet** — a committed budget per 10 ms hop enforced by - `scripts/icount.py`, so an innocuous-looking change that doubles the always-on - cost is caught in review rather than on hardware. +universal; validated against a live microphone at conversational distance at +48 kHz, not only against files; the help patcher's displayed threshold equals +the model's declared operating point. + +> **Sequencing note.** M1, M2 and M3 are worth doing regardless of whether the +> wake word ships. A mel front end is a primitive several Tap libraries would +> use; the learned suppressor gains the float, on-target and CI-parity +> verification it lacks today; and consolidating the inference kernels removes a +> duplication that will otherwise appear the moment any second learned object +> is added. If the project stops after M3, the family is still better off. + +## 7. CI, the ratchet and the ceilings + +Four gates. Two exist and need extending; two are built in M2 and M5. + +- **Contract tests** — typed GoogleTest batteries in DspTap *and* MuTap + (MuTap-Max reaches Catch2 only through min-api). +- **Python↔C++ parity** — built as a CI job in M2 for the suppressor, extended + to the spotter in M6: both profiles, exported weights, shipping geometry. +- **On-target subsets** — the M55 and Hexagon rigs, with the learned path added + in M2 and the front end and spotter in M7. +- **Cost** — `scripts/icount.py` is a ±3 % *drift gate* against seeded + baselines; it catches an innocuous change that doubles the always-on cost. + It is not a budget. The budget is a separate absolute assertion on the same + counts, with these **targets set now and ratified against the first M7 + measurement** (they are not measurements): + + | Ceiling | Target | + |---|---| + | Instructions per 10 ms hop, front end + spotter, scalar float on M55 | ≤ 150 k | + | Weights | ≤ 64 KB | + | Activation and streaming state | ≤ 32 KB | + | Detection latency after phrase end | ≤ 20 hops (200 ms) | Deliberately *not* in CI: the DET evaluation. It needs hundreds of hours of audio and a trained model, so it belongs in the notebook verification layer — -executed, committed, re-executed when behaviour changes, exactly as -`notebooks/pitchshift.ipynb` is. The standing promise applies: every performance -claim measured, not remembered, traceable to the cell that produced it. +executed and committed, built by script in MuTap's convention, re-executed when +behaviour changes. It runs the C++ engine through the C ABI, and the standing +promise applies: every performance claim measured, not remembered, traceable to +the cell that produced it. ## 8. Risks, honestly **The synthetic-positive gap.** The most likely failure is a model that scores beautifully on TTS positives and disappoints on a real talker across a room. -Mitigation is structural rather than clever: the recorded hold-out set in M3, -and M5's pass criterion measured on it. If the gap is large, the answer is more -augmentation realism and more real recordings — both slow, neither surprising. - -**The negative corpus is a storage and time problem.** Hundreds of hours of -negatives is hundreds of gigabytes decoded, and the remote development -containers already have limited disk and a network policy that blocks some -dataset hosts — the RIR-fixture note in HANDOFF.md hit exactly this. Assume -corpus assembly happens locally and enters the repo as manifests and derived -features, not audio. - -**Charter drift.** MuTap's stated charter is adaptive filters for audio -cleaning, and its name is literally the LMS step size. A keyword spotter is -neither an adaptive filter nor audio cleaning. The counter-argument: MuTap is -already the family's speech library, already contains a learned non-adaptive -component, and already has the embedded rigs — and a second repository would -either duplicate that or pin MuTap anyway. Worth deciding on purpose rather than -by drift; see §9. +Mitigation is structural: the specified, consented, recorded hold-out set in +M4, and M6's pass criterion measured on it. If the gap is large, the answer is +more augmentation realism and more real recordings — both slow, neither +surprising. + +**The corpus is a disk, compute and time problem.** Hundreds of hours of +negatives is hundreds of gigabytes decoded and tens of gigabytes of features; +the reference trainer is CPU-only and took hours per hour of audio; TTS +synthesis is a PyTorch pipeline of its own. Corpus assembly and training happen +on named hardware (M0), the feature store lives outside git (M4), and the repo +carries manifests, the builder, and the executed notebook. + +**Charter drift.** MuTap's charter is adaptive filters for audio cleaning, and +its name is literally the LMS step size. A keyword spotter is neither. The +counter-argument: MuTap is already the family's speech library, already contains +a learned non-adaptive component, and already has the embedded rigs — a second +repository would either duplicate that or pin MuTap anyway. Worth deciding on +purpose rather than by drift; see §9. A widened charter also implies README and +description edits, listed in M8. **Scope creep toward speaker verification.** Tier 3 of the cascade usually -carries speaker ID, and it is tempting. It is a separate project with its own -corpora, enrolment UX and privacy posture. Recommend explicitly out of scope. +carries speaker ID, and it is tempting. Separate project, separate corpora, +enrolment UX and privacy posture. Explicitly out of scope, as are pre-roll +buffering, VAD gating and multi-keyword models — named in the header's limits. -**The plan's own weakest estimate.** M5's architecture and training loop is the -only milestone with genuine unknowns in it — everything else is either a known -refactor or a known harness. If a schedule is needed, treat M1–M4 as reasonably -estimable and M5 as the one to time-box with a decision point rather than -estimate. +**The plan's own weakest estimate.** M6's architecture and training loop is the +only milestone with genuine unknowns. M1–M5 are known refactors and known +harnesses. If a schedule is needed, treat M6 as the one to time-box with a +decision point rather than estimate. ## 9. Open decisions **Which repository owns the spotter.** MuTap with a widened charter, or a new -sibling library pinning DspTap. M1 and M2 are correct under both, so this does -not block until M5. -*Recommend MuTap, charter restated as portable speech DSP for embedded targets. -Revisit only if a second consumer for keyword spotting appears in the family — -at which point the promotion pattern makes moving it cheap.* - -**The wake phrase.** Needed at M0 and genuinely load-bearing: syllable count and -phonetic distinctiveness set a ceiling on the achievable DET curve. -*Yours to choose. Technical constraints: three to four syllables, unusual -phonotactics, not a substring of common English.* - -**Model architecture.** DS-CNN, dilated TDNN, or a small GRU reusing the -promoted kernels directly. -*Defer to M5 and decide by measurement. The GRU path is cheapest to reach -because M2 delivers it; DS-CNN is the stronger default on microcontroller-class -targets. With M4 in place this is a measurement, not an argument.* - -**Fixed-point front end.** Whether M6 includes a Q15 mel profile — real work, -and only pays off on M33-class parts without usable float. -*Defer until a target is named. The M55 has float; the existing `sample_traits` -convention makes this opt-in per primitive precisely so it can wait.* - -**Whether to ship weights at all.** An alternative shape: ship `mutap.wake~` as -a runtime that loads a user-supplied model, with a documented training pipeline -and no bundled phrase. Sidesteps corpus assembly entirely and suits a Max -audience who may want their own phrase. -*Worth considering seriously as a first release, with a bundled model following -once the recorded evaluation set exists. It reorders the plan rather than -shortening it — M4 still has to happen before anyone can tell whether a trained -model is any good.* +sibling library pinning DspTap. M1–M3 are correct under both; blocks at M4. +*Recommend MuTap, charter restated as portable speech DSP for embedded targets — +alongside, not replacing, its adaptive-filter core. Revisit only if a second +consumer for keyword spotting appears in the family.* + +**Host-rate policy.** Route (a), fixed internal 16 kHz with the external +converting, or route (b), a model per host rate. +*Recommend (a). One model, one corpus geometry, one FFT size, and the front end +never sees bands the training data cannot excite. Cost: a small polyphase +decimator in DspTap for 32 / 48 / 96 kHz, and 44.1 kHz handled by RatioTap or +declared unsupported in the first release — which HANDOFF already records as an +open slot for the suppressor.* + +**Release shape.** Runtime-first, or bundled weights. +*Recommend runtime-first for the first release. This is a shortening, not a +reorder: it removes the phrase as a blocking decision, the shipped dataset +card, the recorded hold-out set and the declared operating point from the +critical path, and adds a no-model state to the external and threshold +semantics without a measured FA/hour. M4's builder and M5's harness remain the +product a user trains with. A bundled phrase follows once a recorded +evaluation set exists.* + +**The wake phrase.** Load-bearing only under bundled weights. +*Yours to choose, when needed. Three to four syllables, unusual phonotactics, +not a substring of common English.* + +**Model architecture.** DS-CNN, dilated TDNN, or a small GRU on the promoted +kernels. +*Defer to M6 and decide by measurement on both axes. The GRU is cheapest to +reach; DS-CNN is the stronger default on microcontroller-class targets and the +easier one to quantize if int8 is ever needed.* + +**Compute.** Where training runs. +*Yours to name at M0. The plan assumes a GPU-equipped local machine or rented +hours; the remote containers are for the C++ and the harness, not the corpus.* --- ### Provenance - Proposal only — no code written, no repository changed apart from these docs. -- Milestone pass criteria state **targets, not measurements**. No number in this - document is a measured result. +- Milestone pass criteria and the §7 ceilings state **targets, not + measurements**. No number in this document is a measured result. - Existing-asset descriptions were read from the working checkouts of DspTap, - MuTap, TapTools and TapTools-Max on 4 September 2026. -- Corpus licences summarized from the companion briefing and require - verification at the point of use. + MuTap and MuTap-Max on 4 September 2026 and re-verified by the audit; TapTools + and TapTools-Max were not available to the audit and are cited only for the + promotion precedent. +- Corpus licences summarized from the companion briefing and the audit's + licensing lens, with limited network access, and require verification at the + point of use. +- Rev 2 supersedes rev 1; the audit's issue numbers refer to rev 1's sections + and milestones. From bed31641f84c47e624e77b70f33bcce901cb2629 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 4 Sep 2026 10:57:22 +0000 Subject: [PATCH 04/14] =?UTF-8?q?docs:=20wake-word=20plan=20=E2=80=94=20ho?= =?UTF-8?q?st=20rate=20is=20Max's=20job,=20not=20a=20DspTap=20decimator?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Max runs its DSP at whatever rate the audio driver offers, so a patch can run at 16 kHz outright where the interface supports it, and poly~ @down N gives a 16 kHz subpatch at 32/48/96 kHz otherwise. The spotter therefore runs at one internal rate and refuses others; the DspTap decimator is dropped from M1 and deferred until an embedded target with a fixed ADC clock needs one. 44.1 kHz has no integer path and stays unsupported in the first release. M8's help patcher and live-microphone pass now cover both the native 16 kHz and the poly~-wrapped 48 kHz cases. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G --- HANDOFF.md | 2 +- docs/wake-word-plan.md | 66 ++++++++++++++++++++++++------------------ 2 files changed, 39 insertions(+), 29 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index 9cabe66..5e23d28 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -756,6 +756,6 @@ Resolved since rev 1: ~~license~~ (MIT), ~~core language~~ (header-only C++20), Resolved since rev 3: ~~Max external naming~~ — settled (Rev 4): **`mutap.afc~`** (rename from the `mutap.defeed~` placeholder) and **`mutap.aec~`** for the new echo canceller, an acronym pair matching the literature. The rename executes in Stage 2 of "The next effort" above. Still open: -- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. Five decisions are yours at M0 before anything starts, with recommendations in the plan's §9: repository (MuTap), host-rate policy (fixed internal 16 kHz, external decimates), release shape (runtime-first, no bundled phrase), the wake phrase if bundled, and where training runs. +- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. Five decisions are yours at M0 before anything starts, with recommendations in the plan's §9: repository (MuTap), host-rate policy (fixed internal 16 kHz; the host converts — Max at 16 kHz outright or via `poly~ @down N`), release shape (runtime-first, no bundled phrase), the wake phrase if bundled, and where training runs. - **Default engine in the external** — `@kalman` off (classic NLMS) is the shipping default purely on seniority; the measured case for flipping it is in `tests/test_fd_kalman.cpp` and book chapter 1. Decide after real-room listening. - **RIR fixtures, the measured half** — the fixture pipeline is built and three physically-modeled rooms (image-source, documented geometry) are committed baselines with regression tests. What remains yours: which MEASURED rooms join them — an academic dataset room (MYRiAD is the PEM-AFROW group's own database; openAIR is the other usual source; check each room's license allows redistribution in an MIT repo) and/or your own swept-sine measurements. Either way it is one command per room: `python3 tools/fixtures/make_rir_fixtures.py --from-wav room.wav myroom --source ""`, then a test with a freshly measured threshold. (The dataset hosts are unreachable from the remote dev container's network policy, so the WAVs have to enter via a commit.) diff --git a/docs/wake-word-plan.md b/docs/wake-word-plan.md index d325038..34799fb 100644 --- a/docs/wake-word-plan.md +++ b/docs/wake-word-plan.md @@ -52,8 +52,8 @@ MuTap fd_kalman.h nn_suppressor.h [kws.h] ← new, M6 │ ↑ submodule pin │ refactored onto ↓ -DspTap fft.h yin.h [log_mel.h] [nn/] [decimate.h] - ↑ new, M1 ↑ promoted, M3 ↑ new, M1 (if M0 picks it) +DspTap fft.h yin.h [log_mel.h] [nn/] + ↑ new, M1 ↑ promoted, M3 ``` **The load-bearing move is the promotion.** M3 lifts the dense/GRU arithmetic @@ -88,7 +88,7 @@ Read from the checkouts, with the audit's corrections applied. | `tools/ml/README.md` | The licensing map and measured benchmark that justified the hybrid design; files "int8 + CMSIS-NN for the M55 path" as next step. | The template for M4's dataset card, and a roadmap this plan must reconcile with (§5, `tap::dsp::nn`). | | Cortex-M55 QEMU rig + `scripts/icount.py` | On-target positive-filter test subset in CI; whole-binary instruction count per scenario with a ±3 % drift gate against `bench/baselines.json` (`fdkf`, `chain` at 16 k and 48 k, on m55 and hexagon). **No learned-path scenario; `NnSuppressor` not in the on-target filter.** | The rig the embedded profile runs on, once M2 adds the learned scenarios. The ratchet is a drift gate; the budget is a separate absolute assertion (§7). | | DspTap `fft.h` backend pattern | Ooura golden model; CMSIS-Helium and vDSP float32 backends re-presenting the exact contract, certified by parity tests. | The mel front end's FFT, and the rule for any accelerated NN backend: optional, opt-in, parity-pinned against the scalar golden path. | -| DspTap `sample_traits.h`, `kaiser.h`, `fir_kernels.h` | The FIR substrate: float / Q15 / Q31 format core with documented Q-format ladders, Kaiser prototype design, dot kernels. No fixed-point FFT; the rate converters themselves live in SampleRateTap and RatioTap. | The material for a polyphase decimator (M1, if M0 picks that route) and the *convention* for any later Q15 front end — not a fixed-point front end in itself. | +| DspTap `sample_traits.h`, `kaiser.h`, `fir_kernels.h` | The FIR substrate: float / Q15 / Q31 format core with documented Q-format ladders, Kaiser prototype design, dot kernels. No fixed-point FFT; the rate converters themselves live in SampleRateTap and RatioTap. | The *convention* for any later Q15 front end — not a fixed-point front end in itself — and the material for a polyphase decimator should an embedded target with a fixed ADC clock ever need one (not the Max consumer; see M0). | **What this rules out:** TFLite Micro and ONNX runtime as dependencies. The family's demonstrated position is hand-written inference against a documented @@ -166,11 +166,17 @@ the *values*; `log_mel.h` for the *formulas*. Retraining never touches DspTap. - Float/double cross-precision tolerance as a **measured** number, on a test signal that excites every band. -### `tap::dsp::decimate` *(only if M0 picks the fixed-internal-rate route)* +### Host rate -- Integer ratios 2, 3, 6 (32 k, 48 k, 96 k → 16 k); Kaiser-designed polyphase - FIR from `kaiser.h` over `fir_kernels.h`; stopband attenuation, transition - band and group delay as numbers; latency in samples at the host rate. +- The spotter runs at one internal rate, 16 kHz, and `kws` and `mutap.wake~` + **refuse** any other rate rather than warning and proceeding. Rate + conversion is the host's job: Max runs its DSP at whatever the audio driver + offers, so a patch can run at 16 kHz outright where the interface supports + it, and otherwise `poly~ @down N` (N = 2, 3, 6 for 32 / 48 / 96 kHz) gives + a 16 kHz subpatch with Max's own resampling filter. The header states this, + and states that 44.1 kHz has no integer path and is unsupported until a + rational converter (RatioTap) is wired in. A DspTap decimator is deferred + until an embedded target with a fixed ADC clock needs one. ### `tap::dsp::nn` @@ -221,11 +227,13 @@ Five decisions, each cheap now and expensive later. 1. **Repository.** MuTap with a widened charter (§9) or a new sibling. Affects M4/M5 file placement, so it blocks at M4, not M6. 2. **Host-rate policy.** Either (a) the spotter runs at a fixed internal - 16 kHz and `mutap.wake~` owns the conversion — integer ratios via a new - DspTap decimator in M1, 44.1 kHz via RatioTap or deferred; or (b) a model - per host rate on spectrally upsampled corpora, the suppressor's route, - accepting that bands above 8 kHz are never excited in training and that - 44.1 kHz needs a third model. Recommendation in §9. + 16 kHz and refuses other rates, leaving conversion to the host — Max can + run its DSP at 16 kHz where the interface supports it, and `poly~ @down N` + covers 32 / 48 / 96 kHz otherwise; or (b) a model per host rate on + spectrally upsampled corpora, the suppressor's route, accepting that bands + above 8 kHz are never excited in training and that 44.1 kHz needs a third + model. Under (a) 44.1 kHz stays unsupported until RatioTap is wired in. + Recommendation in §9. 3. **Release shape.** Runtime-first (a user-supplied model, no bundled phrase) or bundled weights. Decides whether the phrase blocks anything (§9). 4. **The wake phrase**, if bundled: three or four syllables, unusual @@ -239,12 +247,12 @@ Five decisions, each cheap now and expensive later. shortlist if a phrase is chosen, and the compute budget of M4 named (where training runs, and a per-run time target). -### M1 — The mel front end, and the decimator *(DspTap)* +### M1 — The mel front end *(DspTap)* `include/tap/dsp/log_mel.h` — `basic_log_mel` per the §5 contract, with the double golden model and float32 embedded profile, riding the existing `real_fft` with the FFT size decoupled from the hop. PCEN as a documented option -on the same object. If M0 chose route (a), `decimate.h` beside it. +on the same object. **Before the header:** a throwaway numpy mel in `tools/ml/kws_features.py`, written first and committed, is the reference M1 is scored against — the @@ -259,8 +267,7 @@ checklist; `.clang-tidy` clean under the clang front end. **Pass:** agreement with the committed numpy reference at a committed tolerance on a fixed multi-band test signal; PCEN gain-tracking pinned on a level-stepped input; PCEN reset semantics pinned; streaming output identical to whole-signal -output frame for frame (the alignment contract); decimator passband ripple, -stopband attenuation and latency pinned if built. +output frame for frame (the alignment contract). ### M2 — Oracles for the learned path *(MuTap)* — new in rev 2 @@ -381,7 +388,7 @@ achieved is what gets written down. ### M7 — Embedded profile and the budget *(DspTap · MuTap)* -Front end, decimator (if any) and spotter through the M55 and Hexagon rigs; +Front end and spotter through the M55 and Hexagon rigs; `kws` scenarios added to `bench/icount` at the shipping geometry, baselines seeded on both targets; the per-hop figure derived by dividing the scenario's count by its hop count. The §7 ceilings asserted as absolute checks beside the @@ -397,10 +404,11 @@ backend agreeing with the scalar golden path within a stated, tested tolerance. One external on the `mutap.aec~` pattern: signal inlet; bang outlet on detection; confidence float outlet; attributes for threshold (defaulting from -the loaded model), refractory period and model path; host-rate handling per -M0's decision, refusing rather than warning on an unsupported rate; a no-model -state that meters and never fires, for the runtime-first shape. Reference page -and help patcher with a visible confidence meter. Package-level notices file +the loaded model), refractory period and model path; runs only at the model's +rate and refuses, rather than warns, on any other; a no-model state that meters +and never fires, for the runtime-first shape. Reference page and help patcher +with a visible confidence meter, demonstrating both a 16 kHz patch and the +`poly~ @down 3` wrapper at 48 kHz. Package-level notices file carrying the dataset card's attribution text. "Ship" means what it means for the family today: source build and a Packages @@ -412,8 +420,8 @@ the DET notebook and the tests; a MuTap README status row; a MuTap-Max README roadmap row. **Pass:** loads and behaves correctly in Max on both platforms, macOS binary -universal; validated against a live microphone at conversational distance at -48 kHz, not only against files; the help patcher's displayed threshold equals +universal; validated against a live microphone at conversational distance both natively +at 16 kHz and inside `poly~ @down 3` at 48 kHz, not only against files; the help patcher's displayed threshold equals the model's declared operating point. > **Sequencing note.** M1, M2 and M3 are worth doing regardless of whether the @@ -495,13 +503,15 @@ sibling library pinning DspTap. M1–M3 are correct under both; blocks at M4. alongside, not replacing, its adaptive-filter core. Revisit only if a second consumer for keyword spotting appears in the family.* -**Host-rate policy.** Route (a), fixed internal 16 kHz with the external +**Host-rate policy.** Route (a), fixed internal 16 kHz with the host converting, or route (b), a model per host rate. *Recommend (a). One model, one corpus geometry, one FFT size, and the front end -never sees bands the training data cannot excite. Cost: a small polyphase -decimator in DspTap for 32 / 48 / 96 kHz, and 44.1 kHz handled by RatioTap or -declared unsupported in the first release — which HANDOFF already records as an -open slot for the suppressor.* +never sees bands the training data cannot excite. Max already provides the +conversion — its DSP runs at whatever the driver offers, so 16 kHz outright on +interfaces that support it, and `poly~ @down N` for 32 / 48 / 96 kHz on those +that do not — so no DspTap decimator is needed for this consumer. 44.1 kHz has +no integer path and is declared unsupported in the first release, which HANDOFF +already records as an open slot for the suppressor.* **Release shape.** Runtime-first, or bundled weights. *Recommend runtime-first for the first release. This is a shortening, not a From 344a7a9b892c4c5e0a626197abc147d89b1f4777 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 4 Sep 2026 11:02:56 +0000 Subject: [PATCH 05/14] =?UTF-8?q?docs:=20wake-word=20plan=20=E2=80=94=20co?= =?UTF-8?q?nversion=20as=20a=20Max-layer=20@resample=20option?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Corrects the previous revision on two facts: poly~ resamples by powers of two only, so it offers no 48 -> 16 kHz path; and RatioTap is 44.1 <-> 48 kHz by charter, with SampleRateTap near-unity async, so neither covers 48 -> 16 either. The spotter stays 16 kHz-only in kws.h; conversion becomes an @resample option on mutap.wake~ in the Max layer, backed by a new integer-ratio decimator (2, 3, 6) that returns to M1 in RatioTap's design pattern, with 44.1 kHz handled by composing RatioTap's 44.1 -> 48 in front of the 3:1 stage. The decimator's home (DspTap primitive, recommended, or a RatioTap sibling) is an M0 sub-decision. RatioTap joins the existing-assets table and the diagram. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G --- HANDOFF.md | 2 +- docs/wake-word-plan.md | 99 +++++++++++++++++++++++++++--------------- 2 files changed, 64 insertions(+), 37 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index 5e23d28..1ebef87 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -756,6 +756,6 @@ Resolved since rev 1: ~~license~~ (MIT), ~~core language~~ (header-only C++20), Resolved since rev 3: ~~Max external naming~~ — settled (Rev 4): **`mutap.afc~`** (rename from the `mutap.defeed~` placeholder) and **`mutap.aec~`** for the new echo canceller, an acronym pair matching the literature. The rename executes in Stage 2 of "The next effort" above. Still open: -- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. Five decisions are yours at M0 before anything starts, with recommendations in the plan's §9: repository (MuTap), host-rate policy (fixed internal 16 kHz; the host converts — Max at 16 kHz outright or via `poly~ @down N`), release shape (runtime-first, no bundled phrase), the wake phrase if bundled, and where training runs. +- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. Five decisions are yours at M0 before anything starts, with recommendations in the plan's §9: repository (MuTap), host-rate policy (fixed internal 16 kHz in `kws.h`; conversion as an `@resample` option on the Max external — a new integer-ratio decimator, composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16), release shape (runtime-first, no bundled phrase), the wake phrase if bundled, and where training runs. - **Default engine in the external** — `@kalman` off (classic NLMS) is the shipping default purely on seniority; the measured case for flipping it is in `tests/test_fd_kalman.cpp` and book chapter 1. Decide after real-room listening. - **RIR fixtures, the measured half** — the fixture pipeline is built and three physically-modeled rooms (image-source, documented geometry) are committed baselines with regression tests. What remains yours: which MEASURED rooms join them — an academic dataset room (MYRiAD is the PEM-AFROW group's own database; openAIR is the other usual source; check each room's license allows redistribution in an MIT repo) and/or your own swept-sine measurements. Either way it is one command per room: `python3 tools/fixtures/make_rir_fixtures.py --from-wav room.wav myroom --source ""`, then a test with a freshly measured threshold. (The dataset hosts are unreachable from the remote dev container's network policy, so the WAVs have to enter via a commit.) diff --git a/docs/wake-word-plan.md b/docs/wake-word-plan.md index 34799fb..b453d27 100644 --- a/docs/wake-word-plan.md +++ b/docs/wake-word-plan.md @@ -52,8 +52,9 @@ MuTap fd_kalman.h nn_suppressor.h [kws.h] ← new, M6 │ ↑ submodule pin │ refactored onto ↓ -DspTap fft.h yin.h [log_mel.h] [nn/] - ↑ new, M1 ↑ promoted, M3 +DspTap fft.h yin.h [log_mel.h] [nn/] [decimate.h] + ↑ new, M1 ↑ promoted, M3 ↑ new, M1 (home per M0) +RatioTap 44.1 ↔ 48 only — composed by mutap.wake~ for 44.1 kHz hosts ``` **The load-bearing move is the promotion.** M3 lifts the dense/GRU arithmetic @@ -88,7 +89,8 @@ Read from the checkouts, with the audit's corrections applied. | `tools/ml/README.md` | The licensing map and measured benchmark that justified the hybrid design; files "int8 + CMSIS-NN for the M55 path" as next step. | The template for M4's dataset card, and a roadmap this plan must reconcile with (§5, `tap::dsp::nn`). | | Cortex-M55 QEMU rig + `scripts/icount.py` | On-target positive-filter test subset in CI; whole-binary instruction count per scenario with a ±3 % drift gate against `bench/baselines.json` (`fdkf`, `chain` at 16 k and 48 k, on m55 and hexagon). **No learned-path scenario; `NnSuppressor` not in the on-target filter.** | The rig the embedded profile runs on, once M2 adds the learned scenarios. The ratchet is a drift gate; the budget is a separate absolute assertion (§7). | | DspTap `fft.h` backend pattern | Ooura golden model; CMSIS-Helium and vDSP float32 backends re-presenting the exact contract, certified by parity tests. | The mel front end's FFT, and the rule for any accelerated NN backend: optional, opt-in, parity-pinned against the scalar golden path. | -| DspTap `sample_traits.h`, `kaiser.h`, `fir_kernels.h` | The FIR substrate: float / Q15 / Q31 format core with documented Q-format ladders, Kaiser prototype design, dot kernels. No fixed-point FFT; the rate converters themselves live in SampleRateTap and RatioTap. | The *convention* for any later Q15 front end — not a fixed-point front end in itself — and the material for a polyphase decimator should an embedded target with a fixed ADC clock ever need one (not the Max consumer; see M0). | +| RatioTap | Synchronous 44.1 ↔ 48 kHz, one rational pair by charter ("no other ratios"), compile-time direction type, Kaiser prototype over the DspTap substrate, instruction-count ratchet on M33/M55/Hexagon. SampleRateTap beside it is near-unity async only. | Composed by `mutap.wake~` for 44.1 kHz hosts (44.1 → 48, then 3:1 to 16 kHz), and the *design template* for the decimator: fixed ratios as types, speed-first profiles, ratchet-gated. | +| DspTap `sample_traits.h`, `kaiser.h`, `fir_kernels.h` | The FIR substrate: float / Q15 / Q31 format core with documented Q-format ladders, Kaiser prototype design, dot kernels. No fixed-point FFT; the rate converters themselves live in SampleRateTap and RatioTap. | The *convention* for any later Q15 front end — not a fixed-point front end in itself — and the substrate the integer-ratio decimator of M1 is built on, exactly as RatioTap builds on it. | **What this rules out:** TFLite Micro and ONNX runtime as dependencies. The family's demonstrated position is hand-written inference against a documented @@ -168,15 +170,27 @@ the *values*; `log_mel.h` for the *formulas*. Retraining never touches DspTap. ### Host rate -- The spotter runs at one internal rate, 16 kHz, and `kws` and `mutap.wake~` - **refuse** any other rate rather than warning and proceeding. Rate - conversion is the host's job: Max runs its DSP at whatever the audio driver - offers, so a patch can run at 16 kHz outright where the interface supports - it, and otherwise `poly~ @down N` (N = 2, 3, 6 for 32 / 48 / 96 kHz) gives - a 16 kHz subpatch with Max's own resampling filter. The header states this, - and states that 44.1 kHz has no integer path and is unsupported until a - rational converter (RatioTap) is wired in. A DspTap decimator is deferred - until an embedded target with a fixed ADC clock needs one. +- The spotter runs at one internal rate, 16 kHz. `kws.h` knows nothing about + any other rate and **refuses** one rather than warning and proceeding; rate + conversion is never at the MuTap level. +- Conversion lives in the Max layer as an option on `mutap.wake~` + (`@resample`, on by default): when the host runs at 32 / 48 / 96 kHz the + external decimates by 2 / 3 / 6 in front of the spotter; at 44.1 kHz it + composes RatioTap's 44.1 → 48 with the 3:1 stage; at 16 kHz it passes + through. Any other rate is refused. Where the interface supports 16 kHz, + running the patch there costs nothing and the option is a no-op. +- None of the family's converters covers this today — `poly~` resamples by + powers of two only, RatioTap is 44.1 ↔ 48 by charter, SampleRateTap is + near-unity async — so the integer-ratio decimator is new work (M1), built + on the same DspTap substrate RatioTap uses. + +### `tap::dsp::decimate` *(home per M0: DspTap primitive, or a RatioTap sibling)* + +- Ratios 2, 3, 6 as compile-time types, RatioTap's pattern; Kaiser-designed + polyphase FIR from `kaiser.h` over `fir_kernels.h`; stopband attenuation, + passband edge and group delay as numbers per profile; latency in samples at + the host rate; float golden model pinned against a committed scipy + reference, as RatioTap's is. ### `tap::dsp::nn` @@ -227,13 +241,14 @@ Five decisions, each cheap now and expensive later. 1. **Repository.** MuTap with a widened charter (§9) or a new sibling. Affects M4/M5 file placement, so it blocks at M4, not M6. 2. **Host-rate policy.** Either (a) the spotter runs at a fixed internal - 16 kHz and refuses other rates, leaving conversion to the host — Max can - run its DSP at 16 kHz where the interface supports it, and `poly~ @down N` - covers 32 / 48 / 96 kHz otherwise; or (b) a model per host rate on - spectrally upsampled corpora, the suppressor's route, accepting that bands - above 8 kHz are never excited in training and that 44.1 kHz needs a third - model. Under (a) 44.1 kHz stays unsupported until RatioTap is wired in. - Recommendation in §9. + 16 kHz, `kws.h` refuses other rates, and `mutap.wake~` offers conversion + as an option in the Max layer — a new integer-ratio decimator for 32 / 48 / + 96 kHz, composed with RatioTap for 44.1 kHz; or (b) a model per host rate + on spectrally upsampled corpora, the suppressor's route, accepting that + bands above 8 kHz are never excited in training and that 44.1 kHz needs a + third model. Under (a), a sub-decision: the decimator's home — a DspTap + primitive, or a sibling of RatioTap on the same substrate (RatioTap itself + refuses other ratios by charter). Recommendation in §9. 3. **Release shape.** Runtime-first (a user-supplied model, no bundled phrase) or bundled weights. Decides whether the phrase blocks anything (§9). 4. **The wake phrase**, if bundled: three or four syllables, unusual @@ -247,12 +262,14 @@ Five decisions, each cheap now and expensive later. shortlist if a phrase is chosen, and the compute budget of M4 named (where training runs, and a per-run time target). -### M1 — The mel front end *(DspTap)* +### M1 — The mel front end, and the decimator *(DspTap)* `include/tap/dsp/log_mel.h` — `basic_log_mel` per the §5 contract, with the double golden model and float32 embedded profile, riding the existing `real_fft` with the FFT size decoupled from the hop. PCEN as a documented option -on the same object. +on the same object. Under route (a), `decimate.h` beside it — ratios 2, 3 +and 6 as types, RatioTap's design path (Kaiser prototype, committed scipy +reference vectors, C ABI), in whichever home M0 chose. **Before the header:** a throwaway numpy mel in `tools/ml/kws_features.py`, written first and committed, is the reference M1 is scored against — the @@ -267,7 +284,9 @@ checklist; `.clang-tidy` clean under the clang front end. **Pass:** agreement with the committed numpy reference at a committed tolerance on a fixed multi-band test signal; PCEN gain-tracking pinned on a level-stepped input; PCEN reset semantics pinned; streaming output identical to whole-signal -output frame for frame (the alignment contract). +output frame for frame (the alignment contract); decimator passband ripple, +stopband attenuation and latency pinned per ratio against the committed +reference, if built. ### M2 — Oracles for the learned path *(MuTap)* — new in rev 2 @@ -404,11 +423,13 @@ backend agreeing with the scalar golden path within a stated, tested tolerance. One external on the `mutap.aec~` pattern: signal inlet; bang outlet on detection; confidence float outlet; attributes for threshold (defaulting from -the loaded model), refractory period and model path; runs only at the model's -rate and refuses, rather than warns, on any other; a no-model state that meters -and never fires, for the runtime-first shape. Reference page and help patcher -with a visible confidence meter, demonstrating both a 16 kHz patch and the -`poly~ @down 3` wrapper at 48 kHz. Package-level notices file +the loaded model), refractory period, model path and `@resample`; with +`@resample` on, decimates 32 / 48 / 96 kHz hosts to the model's rate and +composes RatioTap for 44.1 kHz, reporting the added latency; with it off, or +at any other rate, refuses rather than warns; a no-model state that meters and +never fires, for the runtime-first shape. Reference page and help patcher with +a visible confidence meter, demonstrating both a 16 kHz patch and a 48 kHz +patch with `@resample`. Package-level notices file carrying the dataset card's attribution text. "Ship" means what it means for the family today: source build and a Packages @@ -421,7 +442,7 @@ roadmap row. **Pass:** loads and behaves correctly in Max on both platforms, macOS binary universal; validated against a live microphone at conversational distance both natively -at 16 kHz and inside `poly~ @down 3` at 48 kHz, not only against files; the help patcher's displayed threshold equals +at 16 kHz and at 48 kHz through `@resample`, not only against files; the help patcher's displayed threshold equals the model's declared operating point. > **Sequencing note.** M1, M2 and M3 are worth doing regardless of whether the @@ -503,15 +524,21 @@ sibling library pinning DspTap. M1–M3 are correct under both; blocks at M4. alongside, not replacing, its adaptive-filter core. Revisit only if a second consumer for keyword spotting appears in the family.* -**Host-rate policy.** Route (a), fixed internal 16 kHz with the host -converting, or route (b), a model per host rate. +**Host-rate policy.** Route (a), fixed internal 16 kHz with conversion as an +option in the Max layer, or route (b), a model per host rate. *Recommend (a). One model, one corpus geometry, one FFT size, and the front end -never sees bands the training data cannot excite. Max already provides the -conversion — its DSP runs at whatever the driver offers, so 16 kHz outright on -interfaces that support it, and `poly~ @down N` for 32 / 48 / 96 kHz on those -that do not — so no DspTap decimator is needed for this consumer. 44.1 kHz has -no integer path and is declared unsupported in the first release, which HANDOFF -already records as an open slot for the suppressor.* +never sees bands the training data cannot excite. Where the interface offers +16 kHz the patch can simply run there; otherwise `@resample` on `mutap.wake~` +does the work, never `kws.h`. The conversion has to be built: `poly~` is +powers of two only, RatioTap is 44.1 ↔ 48 by charter, SampleRateTap is +near-unity async. The missing piece is small — an integer-ratio decimator for +2, 3 and 6 — and 44.1 kHz falls out by composing RatioTap's 44.1 → 48 in front +of the 3:1 stage, so no rational 441:160 design is needed. For its home, +recommend a DspTap primitive built on the substrate RatioTap already uses, +with RatioTap's design path as the template; widening RatioTap would contradict +its stated identity, and a third converter repository is more than three +ratios warrant. The decimator is also the piece an embedded target with a +fixed ADC clock would need, so it is not Max-only work.* **Release shape.** Runtime-first, or bundled weights. *Recommend runtime-first for the first release. This is a shortening, not a From 4d952e27ef17250f979348acd3a6d871581de1cc Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 4 Sep 2026 19:54:48 +0000 Subject: [PATCH 06/14] =?UTF-8?q?docs:=20wake-word=20plan=20=E2=80=94=20Pi?= =?UTF-8?q?co=202=20W=20as=20the=20M33=20target,=20training=20guide?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Names the Raspberry Pi Pico 2 W (RP2350) as the embedded M33 target. Its Cortex-M33 has a single-precision FPU, so float32 is the profile there and the plan's earlier framing of M33 parts as "without usable float" is corrected to "without double"; the nn contract now forbids double anywhere on the spotter's hot path. M2 ports RatioTap's existing M33 QEMU rig (mps2-an505 toolchain, linker script, startup, CI job, ratchet) into MuTap and seeds baselines on all three emulated targets; M7 adds an out-of-tree Pico SDK example on the board itself with the hardware cycle count recorded beside the QEMU figure. Section 7 gains a targets table and an M33 column on the ceilings. Closes the documentation gap the runtime-first release exposed: a user guide to training a phrase (book chapter plus tools/ml/kws README), drafted at M6 and finished at M8, whose commands are a script CI runs on a toy corpus. Section 6 gains a table of every documentation surface with its milestone. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G --- HANDOFF.md | 2 +- docs/wake-word-plan.md | 90 ++++++++++++++++++++++++++++++++++-------- 2 files changed, 74 insertions(+), 18 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index 1ebef87..8f755b2 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -756,6 +756,6 @@ Resolved since rev 1: ~~license~~ (MIT), ~~core language~~ (header-only C++20), Resolved since rev 3: ~~Max external naming~~ — settled (Rev 4): **`mutap.afc~`** (rename from the `mutap.defeed~` placeholder) and **`mutap.aec~`** for the new echo canceller, an acronym pair matching the literature. The rename executes in Stage 2 of "The next effort" above. Still open: -- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. Five decisions are yours at M0 before anything starts, with recommendations in the plan's §9: repository (MuTap), host-rate policy (fixed internal 16 kHz in `kws.h`; conversion as an `@resample` option on the Max external — a new integer-ratio decimator, composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16), release shape (runtime-first, no bundled phrase), the wake phrase if bundled, and where training runs. +- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. The named M33 target is the Raspberry Pi Pico 2 W (RP2350: single-precision FPU, so float32 is its profile; the M33 QEMU rig is ported from RatioTap in M2 and the board itself is exercised in M7), and the docs plan includes a user guide to training a phrase whose commands CI runs on a toy corpus. Five decisions are yours at M0 before anything starts, with recommendations in the plan's §9: repository (MuTap), host-rate policy (fixed internal 16 kHz in `kws.h`; conversion as an `@resample` option on the Max external — a new integer-ratio decimator, composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16), release shape (runtime-first, no bundled phrase), the wake phrase if bundled, and where training runs. - **Default engine in the external** — `@kalman` off (classic NLMS) is the shipping default purely on seniority; the measured case for flipping it is in `tests/test_fd_kalman.cpp` and book chapter 1. Decide after real-room listening. - **RIR fixtures, the measured half** — the fixture pipeline is built and three physically-modeled rooms (image-source, documented geometry) are committed baselines with regression tests. What remains yours: which MEASURED rooms join them — an academic dataset room (MYRiAD is the PEM-AFROW group's own database; openAIR is the other usual source; check each room's license allows redistribution in an MIT repo) and/or your own swept-sine measurements. Either way it is one command per room: `python3 tools/fixtures/make_rir_fixtures.py --from-wav room.wav myroom --source ""`, then a test with a freshly measured threshold. (The dataset hosts are unreachable from the remote dev container's network policy, so the WAVs have to enter via a commit.) diff --git a/docs/wake-word-plan.md b/docs/wake-word-plan.md index b453d27..46df0dc 100644 --- a/docs/wake-word-plan.md +++ b/docs/wake-word-plan.md @@ -198,7 +198,9 @@ the *values*; `log_mel.h` for the *formulas*. Retraining never touches DspTap. `nn_suppressor_weights` already documents it. - Accumulator precision per kernel, stated: today every dense and GRU dot product accumulates in `Sample`, and M3 preserves that. Any later change is - a documented contract change with a parity delta. + a documented contract change with a parity delta. **No double anywhere in + the hot path**, in any header on the spotter's route: the RP2350's Cortex-M33 + has a single-precision FPU only, so double is soft-float there. - Streaming convolution: the activation cache's size and the causal delay per layer as numbers; equivalence to the non-streaming forward pass pinned by a torch-versus-streaming fixture. @@ -295,10 +297,16 @@ refactors it: - `nn_suppressor` typed tests with a float/double cross-precision pin, and its entry in `test_float32.cpp`. -- `NnSuppressor` patterns added to both on-target positive filters (M55 and +- A Cortex-M33 leg ported from RatioTap (`cmake/arm-cortex-m33-mps2.cmake`, + `platform/mps2_an505`, `armv8m_startup.c`, the CI job and ratchet step), + running the float-profile on-target subset under QEMU's mps2-an505 — the + Raspberry Pi Pico 2 class of core, single-precision FPU, no FP64, no MVE — + with its own `bench/baselines.json` entries. RatioTap has run this rig since + its v0.1, so it is a port, not a design. +- `NnSuppressor` patterns added to every on-target positive filter (M33, M55, Hexagon). - An `nn_suppressor` scenario in `bench/icount` at both shipping geometries, - baselines seeded on m55 and hexagon. + baselines seeded on m33, m55 and hexagon. - `test_parity.py` promoted to a CI job: both profiles, **exported trainer weights** (`pretrained/suppressor_v2_48k.munn`) at the shipping 48 kHz geometry as well as random weights, run under `MUTAP_BUILD_ML_TOOLS=ON` in @@ -407,17 +415,30 @@ achieved is what gets written down. ### M7 — Embedded profile and the budget *(DspTap · MuTap)* -Front end and spotter through the M55 and Hexagon rigs; -`kws` scenarios added to `bench/icount` at the shipping geometry, baselines -seeded on both targets; the per-hop figure derived by dividing the scenario's -count by its hop count. The §7 ceilings asserted as absolute checks beside the -drift gate. Int8 backend only if the measured cost demands it *and* the M6 -architecture admits it; Q15 front end deferred until an M33-class target is -named (a new Q-format design, since DspTap has no fixed-point FFT). - -**Pass:** on-target subsets green on both rigs; both scenarios within the -drift gate and under the absolute ceilings; any accelerated or fixed-point -backend agreeing with the scalar golden path within a stated, tested tolerance. +Front end and spotter through the M33, M55 and Hexagon rigs; `kws` scenarios +added to `bench/icount` at the shipping geometry, baselines seeded on all +three; the per-hop figure derived by dividing the scenario's count by its hop +count. The §7 ceilings asserted as absolute checks beside the drift gate. +Float32 is the profile on every target, the RP2350 included; an int8 backend +or a Q15 front end only if the measured M33 count misses the ceiling *and* the +M6 architecture admits it (a Q15 front end is a new Q-format design, since +DspTap has no fixed-point FFT). + +**On hardware — the named M33 target is the Raspberry Pi Pico 2 W.** +`examples/pico2w/` in MuTap: a Pico SDK application, built out of tree against +the SDK (not in CI; CI is the QEMU leg), reading a MEMS microphone (PDM or I²S +through PIO) at 16 kHz, running the front end and spotter on one of the two +cores, pulsing a GPIO on detection and streaming the confidence over UART. Its +per-hop cycle count from the core's cycle counter is recorded beside the QEMU +instruction count. The board's radio is unused by the plan; a detection-over- +Wi-Fi demo is a natural follow-up, not a deliverable. + +**Pass:** on-target subsets green on all three rigs; both scenarios within the +drift gate and under the absolute ceilings on every target; the Pico 2 W +example detects at conversational distance from its own microphone, with the +hardware cycle count committed beside the QEMU figure; any accelerated or +fixed-point backend agreeing with the scalar golden path within a stated, +tested tolerance. ### M8 — `mutap.wake~` *(MuTap-Max)* @@ -445,6 +466,30 @@ universal; validated against a live microphone at conversational distance both n at 16 kHz and at 48 kHz through `@resample`, not only against files; the help patcher's displayed threshold equals the model's declared operating point. +### The documentation, across the milestones + +Every effort in this family ends under HANDOFF's honesty rule: no number in +the book that a test or notebook does not measure, no attribute described +that does not exist. The spotter's documentation is spread over the +milestones rather than left to the end, and one surface is new to the family: +a **user guide to training a phrase**, which under the runtime-first release is +the product's primary document. + +| Surface | Where | Milestone | Content | +|---|---|---|---| +| Header docstrings | `log_mel.h`, `decimate.h`, `nn/`, `kws.h` | M1, M3, M6 | Every §5 contract point as a number; the honest-limits block | +| DspTap README | `README.md` | M1, M3 | A section per primitive, count bumped, per the checklist | +| Dataset card | `tools/ml/kws/DATASET.md` | M4 | Counts, hours, licences, attribution text, no-reidentification terms, voice lineage table | +| Pipeline reference | `tools/ml/README.md`, re-scoped | M6 | Both tasks; the spotter's benchmark beside the suppressor's | +| **Training guide** | `book/src/train-your-own-phrase.md` + `tools/ml/kws/README.md` | draft M6, final M8 | Choosing a phrase (syllables, confusables); verifying a voice's lineage; synthesizing positives; which negatives and how many; running the splits; training on a named device; reading a DET curve and choosing a threshold; exporting MUKW; loading it in `mutap.wake~`; recording a small hold-out of your own voice. **Its commands are a script, and CI runs that script on a toy corpus**, so the guide cannot drift from the pipeline. | +| Executed DET notebook | `notebooks/`, script-built | M5 onward | The performance record, through the C ABI | +| Book chapter | `book/src/wake-word.md` | M8 | The spotter's design and measured numbers | +| Max reference and help | `docs/mutap.wake~.maxref.xml`, `help/mutap.wake~.maxhelp` | M8 | Attributes, both host-rate patches, a tab pointing at the training guide | +| README status rows | MuTap, MuTap-Max | M8 | Status, roadmap, charter sentence | +| Package notices | MuTap-Max `NOTICES.md` | M8 | The dataset card's attribution text, where a user of the external sees it | +| Pico 2 W example README | `examples/pico2w/README.md` | M7 | Wiring, build against the Pico SDK, the measured cycle count | +| HANDOFF | `HANDOFF.md` | M0, M8 | The five decisions; end state for the next session | + > **Sequencing note.** M1, M2 and M3 are worth doing regardless of whether the > wake word ships. A mel front end is a primitive several Tap libraries would > use; the learned suppressor gains the float, on-target and CI-parity @@ -470,11 +515,22 @@ Four gates. Two exist and need extending; two are built in M2 and M5. | Ceiling | Target | |---|---| - | Instructions per 10 ms hop, front end + spotter, scalar float on M55 | ≤ 150 k | + | Instructions per 10 ms hop, front end + spotter, scalar float, per target | ≤ 150 k on M55 and on M33 (10 % of one RP2350 core at 150 MHz) | | Weights | ≤ 64 KB | | Activation and streaming state | ≤ 32 KB | | Detection latency after phrase end | ≤ 20 hops (200 ms) | +**Targets.** The plan inherits MuTap's target set and adds the M33 leg the +family already runs elsewhere: + +| Tier | Target | Profile | Verified by | +|---|---|---|---| +| Host, the shipping consumer | macOS universal (arm64 + x86_64), Windows; Linux in MuTap's CI matrix | double golden, float32 | Full batteries, sanitizers, MuTap-Max universal-binary check | +| Embedded, emulated in CI | Cortex-M55 (QEMU mps3-an547): FP32 + FP64 + Helium | float32 | On-target subset, icount ratchet, absolute ceilings | +| Embedded, emulated in CI | Qualcomm Hexagon (qemu-hexagon, HVX auto-vectorized) | float32 | On-target full suite, icount ratchet | +| Embedded, emulated in CI *(new, M2)* | Cortex-M33 (QEMU mps2-an505): FP32 only, no FP64, no MVE — the RP2350 class | float32 | Float-profile subset, icount ratchet, absolute ceilings | +| Embedded, on hardware *(M7)* | **Raspberry Pi Pico 2 W** (RP2350: 2 × Cortex-M33 at 150 MHz, 520 KB SRAM, 4 MB flash) | float32 | `examples/pico2w/`, hardware cycle count beside the QEMU figure | + Deliberately *not* in CI: the DET evaluation. It needs hundreds of hours of audio and a trained model, so it belongs in the notebook verification layer — executed and committed, built by script in MuTap's convention, re-executed when @@ -546,8 +602,8 @@ reorder: it removes the phrase as a blocking decision, the shipped dataset card, the recorded hold-out set and the declared operating point from the critical path, and adds a no-model state to the external and threshold semantics without a measured FA/hour. M4's builder and M5's harness remain the -product a user trains with. A bundled phrase follows once a recorded -evaluation set exists.* +product a user trains with, and the training guide in §6 becomes its primary +document. A bundled phrase follows once a recorded evaluation set exists.* **The wake phrase.** Load-bearing only under bundled weights. *Yours to choose, when needed. Three to four syllables, unusual phonotactics, From 48fffb7a930a35b53523defa46b327aa7bfc1c86 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 4 Sep 2026 20:13:22 +0000 Subject: [PATCH 07/14] =?UTF-8?q?docs:=20wake-word=20plan=20=E2=80=94=20bu?= =?UTF-8?q?ild=20the=20Pico=202=20W=20example=20in=20CI?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit M7 gains a pico2w job beside the QEMU legs: the arm-none-eabi-gcc the M33/M55 legs already install, a Pico SDK pinned by tag and commit, PICO_BOARD=pico2_w, the example built, its UF2 uploaded as a workflow artifact, and the flash and RAM footprint from the linker map asserted against the section 7 ceilings. QEMU has no RP2350 model, so detection on the board stays a bench step of the pass criterion. The targets table, CI gates and documentation table say so. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G --- HANDOFF.md | 2 +- docs/wake-word-plan.md | 38 ++++++++++++++++++++++++++------------ 2 files changed, 27 insertions(+), 13 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index 8f755b2..8f7adfd 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -756,6 +756,6 @@ Resolved since rev 1: ~~license~~ (MIT), ~~core language~~ (header-only C++20), Resolved since rev 3: ~~Max external naming~~ — settled (Rev 4): **`mutap.afc~`** (rename from the `mutap.defeed~` placeholder) and **`mutap.aec~`** for the new echo canceller, an acronym pair matching the literature. The rename executes in Stage 2 of "The next effort" above. Still open: -- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. The named M33 target is the Raspberry Pi Pico 2 W (RP2350: single-precision FPU, so float32 is its profile; the M33 QEMU rig is ported from RatioTap in M2 and the board itself is exercised in M7), and the docs plan includes a user guide to training a phrase whose commands CI runs on a toy corpus. Five decisions are yours at M0 before anything starts, with recommendations in the plan's §9: repository (MuTap), host-rate policy (fixed internal 16 kHz in `kws.h`; conversion as an `@resample` option on the Max external — a new integer-ratio decimator, composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16), release shape (runtime-first, no bundled phrase), the wake phrase if bundled, and where training runs. +- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. The named M33 target is the Raspberry Pi Pico 2 W (RP2350: single-precision FPU, so float32 is its profile; the M33 QEMU rig is ported from RatioTap in M2, and M7 adds a `pico2w` CI job that builds the board example against a pinned Pico SDK, uploads the UF2 and asserts its footprint — detection itself is a bench step, since QEMU has no RP2350 model), and the docs plan includes a user guide to training a phrase whose commands CI runs on a toy corpus. Five decisions are yours at M0 before anything starts, with recommendations in the plan's §9: repository (MuTap), host-rate policy (fixed internal 16 kHz in `kws.h`; conversion as an `@resample` option on the Max external — a new integer-ratio decimator, composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16), release shape (runtime-first, no bundled phrase), the wake phrase if bundled, and where training runs. - **Default engine in the external** — `@kalman` off (classic NLMS) is the shipping default purely on seniority; the measured case for flipping it is in `tests/test_fd_kalman.cpp` and book chapter 1. Decide after real-room listening. - **RIR fixtures, the measured half** — the fixture pipeline is built and three physically-modeled rooms (image-source, documented geometry) are committed baselines with regression tests. What remains yours: which MEASURED rooms join them — an academic dataset room (MYRiAD is the PEM-AFROW group's own database; openAIR is the other usual source; check each room's license allows redistribution in an MIT repo) and/or your own swept-sine measurements. Either way it is one command per room: `python3 tools/fixtures/make_rir_fixtures.py --from-wav room.wav myroom --source ""`, then a test with a freshly measured threshold. (The dataset hosts are unreachable from the remote dev container's network policy, so the WAVs have to enter via a commit.) diff --git a/docs/wake-word-plan.md b/docs/wake-word-plan.md index 46df0dc..a1a807e 100644 --- a/docs/wake-word-plan.md +++ b/docs/wake-word-plan.md @@ -425,16 +425,27 @@ M6 architecture admits it (a Q15 front end is a new Q-format design, since DspTap has no fixed-point FFT). **On hardware — the named M33 target is the Raspberry Pi Pico 2 W.** -`examples/pico2w/` in MuTap: a Pico SDK application, built out of tree against -the SDK (not in CI; CI is the QEMU leg), reading a MEMS microphone (PDM or I²S -through PIO) at 16 kHz, running the front end and spotter on one of the two -cores, pulsing a GPIO on detection and streaming the confidence over UART. Its -per-hop cycle count from the core's cycle counter is recorded beside the QEMU -instruction count. The board's radio is unused by the plan; a detection-over- -Wi-Fi demo is a natural follow-up, not a deliverable. +`examples/pico2w/` in MuTap: a Pico SDK application reading a MEMS microphone +(PDM or I²S through PIO) at 16 kHz, running the front end and spotter on one +of the two cores, pulsing a GPIO on detection and streaming the confidence over +UART. Its per-hop cycle count from the core's cycle counter is recorded beside +the QEMU instruction count. The board's radio is unused by the plan; a +detection-over-Wi-Fi demo is a natural follow-up, not a deliverable. + +**Built in CI.** A `pico2w` job beside the QEMU legs: the `arm-none-eabi-gcc` +the M33/M55 legs already install, a Pico SDK checkout pinned by tag and +commit, `PICO_BOARD=pico2_w` and `PICO_PLATFORM=rp2350-arm-s`; it builds the +example, uploads the `.uf2` as a workflow artifact, and asserts the flash and +RAM footprint from the linker map against the §7 ceilings, so a change that +no longer fits the part fails in review. QEMU has no RP2350 board model, so +CI proves the build and the footprint; detection on the board is the bench +step of the pass criterion, and its cycle count is committed by hand. The UF2 +is the family's first CI-produced binary deliverable, which is a precedent the +M8 release question can reuse. **Pass:** on-target subsets green on all three rigs; both scenarios within the -drift gate and under the absolute ceilings on every target; the Pico 2 W +drift gate and under the absolute ceilings on every target; the `pico2w` job +green with its UF2 uploaded and its footprint under the ceilings; the Pico 2 W example detects at conversational distance from its own microphone, with the hardware cycle count committed beside the QEMU figure; any accelerated or fixed-point backend agreeing with the scalar golden path within a stated, @@ -487,7 +498,7 @@ the product's primary document. | Max reference and help | `docs/mutap.wake~.maxref.xml`, `help/mutap.wake~.maxhelp` | M8 | Attributes, both host-rate patches, a tab pointing at the training guide | | README status rows | MuTap, MuTap-Max | M8 | Status, roadmap, charter sentence | | Package notices | MuTap-Max `NOTICES.md` | M8 | The dataset card's attribution text, where a user of the external sees it | -| Pico 2 W example README | `examples/pico2w/README.md` | M7 | Wiring, build against the Pico SDK, the measured cycle count | +| Pico 2 W example README | `examples/pico2w/README.md` | M7 | Wiring, flashing the CI-built UF2, building locally against the Pico SDK, the measured cycle count | | HANDOFF | `HANDOFF.md` | M0, M8 | The five decisions; end state for the next session | > **Sequencing note.** M1, M2 and M3 are worth doing regardless of whether the @@ -505,8 +516,11 @@ Four gates. Two exist and need extending; two are built in M2 and M5. (MuTap-Max reaches Catch2 only through min-api). - **Python↔C++ parity** — built as a CI job in M2 for the suppressor, extended to the spotter in M6: both profiles, exported weights, shipping geometry. -- **On-target subsets** — the M55 and Hexagon rigs, with the learned path added - in M2 and the front end and spotter in M7. +- **On-target subsets** — the M33, M55 and Hexagon rigs, with the learned path + added in M2 and the front end and spotter in M7. +- **Pico 2 W build** — the `pico2w` job of M7: build, UF2 artifact, and a + footprint assertion from the linker map against the weight and state + ceilings below. - **Cost** — `scripts/icount.py` is a ±3 % *drift gate* against seeded baselines; it catches an innocuous change that doubles the always-on cost. It is not a budget. The budget is a separate absolute assertion on the same @@ -529,7 +543,7 @@ family already runs elsewhere: | Embedded, emulated in CI | Cortex-M55 (QEMU mps3-an547): FP32 + FP64 + Helium | float32 | On-target subset, icount ratchet, absolute ceilings | | Embedded, emulated in CI | Qualcomm Hexagon (qemu-hexagon, HVX auto-vectorized) | float32 | On-target full suite, icount ratchet | | Embedded, emulated in CI *(new, M2)* | Cortex-M33 (QEMU mps2-an505): FP32 only, no FP64, no MVE — the RP2350 class | float32 | Float-profile subset, icount ratchet, absolute ceilings | -| Embedded, on hardware *(M7)* | **Raspberry Pi Pico 2 W** (RP2350: 2 × Cortex-M33 at 150 MHz, 520 KB SRAM, 4 MB flash) | float32 | `examples/pico2w/`, hardware cycle count beside the QEMU figure | +| Embedded, on hardware *(M7)* | **Raspberry Pi Pico 2 W** (RP2350: 2 × Cortex-M33 at 150 MHz, 520 KB SRAM, 4 MB flash) | float32 | CI: `pico2w` job builds `examples/pico2w/` against a pinned Pico SDK, uploads the UF2, asserts flash and RAM footprint. Bench: detection, hardware cycle count beside the QEMU figure | Deliberately *not* in CI: the DET evaluation. It needs hundreds of hours of audio and a trained model, so it belongs in the notebook verification layer — From ce705ea79d48beb468eaa42f004e1aa91e25caa0 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 4 Sep 2026 20:22:54 +0000 Subject: [PATCH 08/14] =?UTF-8?q?docs:=20wake-word=20plan=20=E2=80=94=20fi?= =?UTF-8?q?le=20nine=20gaps=20found=20on=20review?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Music joins the evaluation negatives (MUSAN's music partition and permissively licensed Free Music Archive tracks), with the DET report splitting false accepts per hour on speech and on music, since a Max user's room has music in it. The Pico 2 W example specifies an I2S MEMS microphone, with PDM demodulation named as a budgeted follow-up rather than a free alternative, and its hardware pass becomes a bench protocol: loudspeaker playback of the hold-out set and an hour of negatives at a stated distance and level, scored by the M5 harness, numbers committed. The hold-out set is recorded through both the host interface and the Pico microphone path. The MUKW payload carries the front-end contract version and the external refuses a mismatch. The exporter also emits a C header of the weights image for flash-resident targets. Model loading in the Max external happens off the audio thread. The header gains a privacy statement and names the microphone path and the absence of a low-power tier among its limits. The training guide's toy CI run names its dependencies and time budget. The Wi-Fi follow-up notes PIO sharing with the wireless chip. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G --- HANDOFF.md | 2 +- docs/wake-word-plan.md | 76 ++++++++++++++++++++++++++++++------------ 2 files changed, 56 insertions(+), 22 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index 8f7adfd..4525796 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -756,6 +756,6 @@ Resolved since rev 1: ~~license~~ (MIT), ~~core language~~ (header-only C++20), Resolved since rev 3: ~~Max external naming~~ — settled (Rev 4): **`mutap.afc~`** (rename from the `mutap.defeed~` placeholder) and **`mutap.aec~`** for the new echo canceller, an acronym pair matching the literature. The rename executes in Stage 2 of "The next effort" above. Still open: -- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. The named M33 target is the Raspberry Pi Pico 2 W (RP2350: single-precision FPU, so float32 is its profile; the M33 QEMU rig is ported from RatioTap in M2, and M7 adds a `pico2w` CI job that builds the board example against a pinned Pico SDK, uploads the UF2 and asserts its footprint — detection itself is a bench step, since QEMU has no RP2350 model), and the docs plan includes a user guide to training a phrase whose commands CI runs on a toy corpus. Five decisions are yours at M0 before anything starts, with recommendations in the plan's §9: repository (MuTap), host-rate policy (fixed internal 16 kHz in `kws.h`; conversion as an `@resample` option on the Max external — a new integer-ratio decimator, composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16), release shape (runtime-first, no bundled phrase), the wake phrase if bundled, and where training runs. +- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. The named M33 target is the Raspberry Pi Pico 2 W (RP2350: single-precision FPU, so float32 is its profile; the M33 QEMU rig is ported from RatioTap in M2, and M7 adds a `pico2w` CI job that builds the board example against a pinned Pico SDK, uploads the UF2 and asserts its footprint — detection itself is a bench step with a loudspeaker-playback protocol and committed numbers, since QEMU has no RP2350 model), and the docs plan includes a user guide to training a phrase whose commands CI runs on a toy corpus. Five decisions are yours at M0 before anything starts, with recommendations in the plan's §9: repository (MuTap), host-rate policy (fixed internal 16 kHz in `kws.h`; conversion as an `@resample` option on the Max external — a new integer-ratio decimator, composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16), release shape (runtime-first, no bundled phrase), the wake phrase if bundled, and where training runs. - **Default engine in the external** — `@kalman` off (classic NLMS) is the shipping default purely on seniority; the measured case for flipping it is in `tests/test_fd_kalman.cpp` and book chapter 1. Decide after real-room listening. - **RIR fixtures, the measured half** — the fixture pipeline is built and three physically-modeled rooms (image-source, documented geometry) are committed baselines with regression tests. What remains yours: which MEASURED rooms join them — an academic dataset room (MYRiAD is the PEM-AFROW group's own database; openAIR is the other usual source; check each room's license allows redistribution in an MIT repo) and/or your own swept-sine measurements. Either way it is one command per room: `python3 tools/fixtures/make_rir_fixtures.py --from-wav room.wav myroom --source ""`, then a test with a freshly measured threshold. (The dataset hosts are unreachable from the remote dev container's network policy, so the WAVs have to enter via a commit.) diff --git a/docs/wake-word-plan.md b/docs/wake-word-plan.md index a1a807e..578d028 100644 --- a/docs/wake-word-plan.md +++ b/docs/wake-word-plan.md @@ -114,7 +114,8 @@ synthetic positives are the least clean input, not the cleanest. | MSWC | CC BY 4.0; no-reidentification term | Hard negatives, phonetic near-misses. **Force-aligned out of Common Voice** — splits must be speaker-disjoint across both. | yes, with attribution | | Common Voice | CC0; no-reidentification term | Bulk negatives | yes | | AMI Meeting Corpus | CC BY 4.0 | Conversational negatives | yes, with attribution | -| MUSAN | CC BY 4.0 | Additive noise | yes, with attribution | +| MUSAN | CC BY 4.0 | Additive noise; its music partition also serves as **evaluation negatives** — a Max user's room has music in it | yes, with attribution | +| Free Music Archive (CC BY / CC0 subset only) | per track | Bulk music negatives beyond MUSAN's ~42 h, filtered to permissive tracks by the manifest | yes, with attribution per track | | OpenSLR SLR28 | Apache 2.0 (simulated RIRs); real-RIR subset carries RWCP / REVERB / AIR third-party terms | Reverberation augmentation | simulated subset yes; **real subset only after its upstream terms are checked** | | Piper + `piper-sample-generator` | MIT (code); **voice models carry their training corpus's terms** — the two voices the generator documents descend from Blizzard 2013 / Lessac, research-only, excluding speech-recognition products | Synthetic positives, TTS-derived | code yes; **each voice lineage-verified at M0, Lessac-free voices chosen** | | openWakeWord / microWakeWord | Apache 2.0 (code); models CC BY-NC-SA 4.0; pre-computed feature sets CC BY-NC with WHAM / CHiME-6 upstream | Design reference only | design yes — **no code, models or feature sets imported**; every feature shard regenerated from manifest audio | @@ -220,6 +221,11 @@ the *values*; `log_mel.h` for the *formulas*. Retraining never touches DspTap. period — all as numbers, all settable, all defaulted, **all carried in the MUKW payload** alongside the threshold, so a retrain updates one place and the Max `@threshold` default reads from the model. +- The **front-end contract version** — the `log_mel` formula-contract version + the model was trained against — carried in the MUKW payload and checked at + load; a mismatch is refused with a message naming both versions. Under the + runtime-first release users train against a pipeline that will move, and + this is what keeps an old model from silently degrading on a new external. - Detection latency: hops from phrase end to the bang, as a number. - Streaming-state policy (reset per window or carried) if a recurrent layer is present. @@ -227,8 +233,12 @@ the *values*; `log_mel.h` for the *formulas*. Retraining never touches DspTap. hour pair it was measured at, on a **named** evaluation set — measured on the C++ engine through the C ABI, never on the Python model. - Honest limits stated in the header: the phrase, the internal rate and host - rates supported, the distances and SNRs evaluated over, single-channel, no - beamforming, no pre-roll, no VAD gating. + rates supported, the distances and SNRs evaluated over, the microphone path + the operating point was measured through, single-channel, no beamforming, no + pre-roll, no VAD gating, no low-power tier. +- A privacy statement, in the header and repeated in the help patcher and the + Pico README: audio is processed in place and never stored or transmitted; the + only outputs are the bang and the confidence value. ## 6. Milestones @@ -347,7 +357,10 @@ and the MUKW header. **Splits:** train / dev / eval, speaker-disjoint, with MSWC clips assigned by their Common Voice client id so the two corpora cannot leak into each other. The eval negative set is named and never trained on; it is the FA/hour -denominator for every number in this plan. +denominator for every number in this plan. It carries a **music share** — +MUSAN's music partition and permissively licensed Free Music Archive tracks, +never used in training — because a Max user's room has music playing in it +and a false-accept rate measured on speech alone says nothing about that. **Manifest schema:** corpus release ids, archive checksums, decoder versions, `kws_features.py` contract version, split assignment, augmentation seeds. @@ -356,7 +369,10 @@ denominator for every number in this plan. **Hold-out specification:** N talkers (target ≥ 10), distance × SNR condition matrix, owner, consent and permitted-use row per talker, target ≥ 200 -utterances. Committed as a fixture with provenance, as the RIR fixtures are. +utterances. Recorded through **two microphone paths** — a host audio interface +and the Pico 2 W example's own MEMS microphone — so both M6's and M7's +operating points are measured on the path they ship on. Committed as a +fixture with provenance, as the RIR fixtures are. **Compute:** the trainer gains a `--device` path; a 50 h development negative set serves architecture selection, the full corpus only the final DET. @@ -375,7 +391,8 @@ DET curve tooling: false-rejection rate against false-accepts-per-hour, with the scoring semantics **defined as numbers**: hit window around each positive's endpoint, one hit per utterance, false-accept merging under the refractory period, the hours denominator from the eval negative set. Threshold sweep and a -committed report format. +committed report format that reports false accepts per hour **separately on +speech and on music**. Brought up on **Speech Commands** (`marvin` / `sheila` as name-shaped positives) so it precedes M0's phrase and M4's corpus, then pointed at M4's @@ -394,9 +411,11 @@ stage parity pinned on the same streams. `kws.h` per the §5 contract, over the M1 features and M3 kernels, with the DS-conv / activation-cache kernels landing in `tap::dsp::nn` now, alongside their consumer and a torch-versus-streaming fixture. `kws_geometry` in a -`MUKW0001` payload that also carries the decision-stage constants, threshold and -declared operating point. PyTorch trainer and exporter beside the existing -ones; `kws_infer.cpp` parity driver and CMake target; `mutap_kws_*` C ABI +`MUKW0001` payload that also carries the decision-stage constants, threshold, +declared operating point and front-end contract version. PyTorch trainer and +exporter beside the existing ones, the exporter emitting both the `.mukw` +file and a C header of the same image for flash-resident targets, as +MuTap-Max's default-weights header already does for the suppressor; `kws_infer.cpp` parity driver and CMake target; `mutap_kws_*` C ABI (create-from-weights, push block, posterior and confidence readout, detection events with sample timestamps) and its `mutap_ffi` binding; `tools/ml/README.md` re-scoped to two tasks. @@ -409,9 +428,9 @@ ceilings. the streaming fixture passes; a stated operating point measured **through the C ABI on the recorded hold-out set** and committed as a regression baseline; attribution block present in the exporter output. Target to aim at: ≥ 95 % -recall at ≤ 1 false accept per hour on the named eval negative set — a -first-release target, looser than the briefing's product figure. Whatever is -achieved is what gets written down. +recall at ≤ 1 false accept per hour on the named eval negative set, on its +speech and its music share alike — a first-release target, looser than the +briefing's product figure. Whatever is achieved is what gets written down. ### M7 — Embedded profile and the budget *(DspTap · MuTap)* @@ -425,12 +444,24 @@ M6 architecture admits it (a Q15 front end is a new Q-format design, since DspTap has no fixed-point FFT). **On hardware — the named M33 target is the Raspberry Pi Pico 2 W.** -`examples/pico2w/` in MuTap: a Pico SDK application reading a MEMS microphone -(PDM or I²S through PIO) at 16 kHz, running the front end and spotter on one +`examples/pico2w/` in MuTap: a Pico SDK application reading an **I²S MEMS +microphone through PIO** at 16 kHz, running the front end and spotter on one of the two cores, pulsing a GPIO on detection and streaming the confidence over -UART. Its per-hop cycle count from the core's cycle counter is recorded beside -the QEMU instruction count. The board's radio is unused by the plan; a -detection-over-Wi-Fi demo is a natural follow-up, not a deliverable. +UART. I²S rather than PDM on purpose: a PDM microphone needs its own +demodulation filter from a megahertz bitstream down to 16 kHz, and on an M33 +that filter can cost more than the front end and spotter together. PDM support +is a follow-up with its own line in the §7 ceilings, not a free alternative. +The per-hop cycle count from the core's cycle counter is recorded beside the +QEMU instruction count. The board's radio is unused by the plan; a +detection-over-Wi-Fi demo is a natural follow-up, not a deliverable — and one +that must share the RP2350's three PIO blocks between the I²S microphone and +the wireless chip's PIO-driven SPI. + +**Bench protocol, so the hardware pass can fail.** The recorded hold-out set +(its Pico-microphone path) and one hour of the eval negatives, speech and +music, played through a loudspeaker at a stated distance and level, with the +board's bang line logged; hits and false accepts counted by the M5 harness's +own scoring rules; recall and FA/hour committed beside the host figures. **Built in CI.** A `pico2w` job beside the QEMU legs: the `arm-none-eabi-gcc` the M33/M55 legs already install, a Pico SDK checkout pinned by tag and @@ -446,8 +477,8 @@ M8 release question can reuse. **Pass:** on-target subsets green on all three rigs; both scenarios within the drift gate and under the absolute ceilings on every target; the `pico2w` job green with its UF2 uploaded and its footprint under the ceilings; the Pico 2 W -example detects at conversational distance from its own microphone, with the -hardware cycle count committed beside the QEMU figure; any accelerated or +example's recall and FA/hour under the bench protocol committed, with the +hardware cycle count beside the QEMU figure; any accelerated or fixed-point backend agreeing with the scalar golden path within a stated, tested tolerance. @@ -459,7 +490,10 @@ the loaded model), refractory period, model path and `@resample`; with `@resample` on, decimates 32 / 48 / 96 kHz hosts to the model's rate and composes RatioTap for 44.1 kHz, reporting the added latency; with it off, or at any other rate, refuses rather than warns; a no-model state that meters and -never fires, for the runtime-first shape. Reference page and help patcher with +never fires, for the runtime-first shape; model loading off the audio thread — +a new model may change geometry and so re-prepare the front end, which +allocates, so the object is built on the main thread and swapped in by +pointer, as `mutap.aec~` does for its weights. Reference page and help patcher with a visible confidence meter, demonstrating both a 16 kHz patch and a 48 kHz patch with `@resample`. Package-level notices file carrying the dataset card's attribution text. @@ -492,7 +526,7 @@ the product's primary document. | DspTap README | `README.md` | M1, M3 | A section per primitive, count bumped, per the checklist | | Dataset card | `tools/ml/kws/DATASET.md` | M4 | Counts, hours, licences, attribution text, no-reidentification terms, voice lineage table | | Pipeline reference | `tools/ml/README.md`, re-scoped | M6 | Both tasks; the spotter's benchmark beside the suppressor's | -| **Training guide** | `book/src/train-your-own-phrase.md` + `tools/ml/kws/README.md` | draft M6, final M8 | Choosing a phrase (syllables, confusables); verifying a voice's lineage; synthesizing positives; which negatives and how many; running the splits; training on a named device; reading a DET curve and choosing a threshold; exporting MUKW; loading it in `mutap.wake~`; recording a small hold-out of your own voice. **Its commands are a script, and CI runs that script on a toy corpus**, so the guide cannot drift from the pipeline. | +| **Training guide** | `book/src/train-your-own-phrase.md` + `tools/ml/kws/README.md` | draft M6, final M8 | Choosing a phrase (syllables, confusables); verifying a voice's lineage; synthesizing positives; which negatives and how many; running the splits; training on a named device; reading a DET curve and choosing a threshold; exporting MUKW; loading it in `mutap.wake~`; recording a small hold-out of your own voice. **Its commands are a script, and CI runs that script on a toy corpus**, so the guide cannot drift from the pipeline. The toy run installs torch and one lineage-cleared Piper voice in its job, synthesizes a handful of clips, uses a committed tiny negative set rather than downloading any corpus, and has a stated time budget. | | Executed DET notebook | `notebooks/`, script-built | M5 onward | The performance record, through the C ABI | | Book chapter | `book/src/wake-word.md` | M8 | The spotter's design and measured numbers | | Max reference and help | `docs/mutap.wake~.maxref.xml`, `help/mutap.wake~.maxhelp` | M8 | Attributes, both host-rate patches, a tab pointing at the training guide | From 45952e60017431286ec89d1d521f16c493cd23b8 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 4 Sep 2026 20:47:46 +0000 Subject: [PATCH 09/14] docs: wake-word M0 decided Records the M0 decisions in the plan and HANDOFF. Repository: MuTap, charter widened. Host rate: fixed internal 16 kHz in kws.h, conversion as an @resample option on the Max external backed by a DspTap decimate.h, composed with RatioTap for 44.1 kHz. Release shape: runtime-first, no bundled phrase. Development phrase: marvin from Speech Commands for M5 bring-up, a synthesized phrase from M4, never shipped. Compute: an Apple Silicon Mac via --device mps. TTS voice lineage verified from the Piper per-voice model cards. Most English Piper voices, libritts_r and vctk included, are fine-tuned from the Lessac voice and inherit its research-only lineage; four are trained from scratch on permissive data and are adopted: en_US-libritts-high (LibriTTS, CC BY 4.0, 904 speakers), en_US-kristin-medium and en_GB-cori-high (LibriVox, public domain), and en_US-john-medium (from Kristin). The sample generator's bundled LibriTTS-R generator is excluded until its base voice is verified. The licensing map's Piper row carries the result; section 9 is retitled as the record of arguments. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G --- HANDOFF.md | 2 +- docs/wake-word-plan.md | 42 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 39 insertions(+), 5 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index 4525796..81950a3 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -756,6 +756,6 @@ Resolved since rev 1: ~~license~~ (MIT), ~~core language~~ (header-only C++20), Resolved since rev 3: ~~Max external naming~~ — settled (Rev 4): **`mutap.afc~`** (rename from the `mutap.defeed~` placeholder) and **`mutap.aec~`** for the new echo canceller, an acronym pair matching the literature. The rename executes in Stage 2 of "The next effort" above. Still open: -- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. The named M33 target is the Raspberry Pi Pico 2 W (RP2350: single-precision FPU, so float32 is its profile; the M33 QEMU rig is ported from RatioTap in M2, and M7 adds a `pico2w` CI job that builds the board example against a pinned Pico SDK, uploads the UF2 and asserts its footprint — detection itself is a bench step with a loudspeaker-playback protocol and committed numbers, since QEMU has no RP2350 model), and the docs plan includes a user guide to training a phrase whose commands CI runs on a toy corpus. Five decisions are yours at M0 before anything starts, with recommendations in the plan's §9: repository (MuTap), host-rate policy (fixed internal 16 kHz in `kws.h`; conversion as an `@resample` option on the Max external — a new integer-ratio decimator, composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16), release shape (runtime-first, no bundled phrase), the wake phrase if bundled, and where training runs. +- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. The named M33 target is the Raspberry Pi Pico 2 W (RP2350: single-precision FPU, so float32 is its profile; the M33 QEMU rig is ported from RatioTap in M2, and M7 adds a `pico2w` CI job that builds the board example against a pinned Pico SDK, uploads the UF2 and asserts its footprint — detection itself is a bench step with a loudspeaker-playback protocol and committed numbers, since QEMU has no RP2350 model), and the docs plan includes a user guide to training a phrase whose commands CI runs on a toy corpus. **M0 is decided (4 September 2026):** repository MuTap, charter widened; host rate fixed at 16 kHz in `kws.h` with conversion as an `@resample` option on the Max external, backed by a new DspTap `decimate.h` (2/3/6) and composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16; release shape runtime-first, no bundled phrase, the training guide as the primary document; development phrase `marvin` from Speech Commands for M5 bring-up, a synthesized four-syllable phrase from M4, never shipped; TTS voices lineage-verified from the Piper model cards — `en_US-libritts-high` (from scratch, CC BY 4.0, 904 speakers), `en_US-kristin-medium`, `en_GB-cori-high` (from scratch, public domain), `en_US-john-medium` (from Kristin), with every Lessac-derived voice (most of the English set, `libritts_r` and `vctk` included) and the sample generator's bundled `.pt` generator excluded; training on an Apple Silicon Mac via `--device mps`, under an hour per run on the development set. Next: M1. - **Default engine in the external** — `@kalman` off (classic NLMS) is the shipping default purely on seniority; the measured case for flipping it is in `tests/test_fd_kalman.cpp` and book chapter 1. Decide after real-room listening. - **RIR fixtures, the measured half** — the fixture pipeline is built and three physically-modeled rooms (image-source, documented geometry) are committed baselines with regression tests. What remains yours: which MEASURED rooms join them — an academic dataset room (MYRiAD is the PEM-AFROW group's own database; openAIR is the other usual source; check each room's license allows redistribution in an MIT repo) and/or your own swept-sine measurements. Either way it is one command per room: `python3 tools/fixtures/make_rir_fixtures.py --from-wav room.wav myroom --source ""`, then a test with a freshly measured threshold. (The dataset hosts are unreachable from the remote dev container's network policy, so the WAVs have to enter via a commit.) diff --git a/docs/wake-word-plan.md b/docs/wake-word-plan.md index 578d028..cfb11ba 100644 --- a/docs/wake-word-plan.md +++ b/docs/wake-word-plan.md @@ -53,7 +53,7 @@ MuTap fd_kalman.h nn_suppressor.h [kws.h] ← new, M6 │ refactored onto ↓ DspTap fft.h yin.h [log_mel.h] [nn/] [decimate.h] - ↑ new, M1 ↑ promoted, M3 ↑ new, M1 (home per M0) + ↑ new, M1 ↑ promoted, M3 ↑ new, M1 (DspTap, per M0) RatioTap 44.1 ↔ 48 only — composed by mutap.wake~ for 44.1 kHz hosts ``` @@ -117,7 +117,7 @@ synthetic positives are the least clean input, not the cleanest. | MUSAN | CC BY 4.0 | Additive noise; its music partition also serves as **evaluation negatives** — a Max user's room has music in it | yes, with attribution | | Free Music Archive (CC BY / CC0 subset only) | per track | Bulk music negatives beyond MUSAN's ~42 h, filtered to permissive tracks by the manifest | yes, with attribution per track | | OpenSLR SLR28 | Apache 2.0 (simulated RIRs); real-RIR subset carries RWCP / REVERB / AIR third-party terms | Reverberation augmentation | simulated subset yes; **real subset only after its upstream terms are checked** | -| Piper + `piper-sample-generator` | MIT (code); **voice models carry their training corpus's terms** — the two voices the generator documents descend from Blizzard 2013 / Lessac, research-only, excluding speech-recognition products | Synthetic positives, TTS-derived | code yes; **each voice lineage-verified at M0, Lessac-free voices chosen** | +| Piper + `piper-sample-generator` | MIT (code); **voice models carry their training corpus's terms**. Verified at M0 from the per-voice model cards: most English Piper voices — `libritts_r`, `vctk`, `arctic`, `l2arctic`, `joe`, `kusal`, and the whole en_GB set except `cori` — are *fine-tuned from* the Lessac voice and inherit its research-only lineage; `hfc_*` and `semaine` are on NC datasets; `bryce`, `danny`, `kathleen`, `amy` have unverifiable base voices | Synthetic positives, TTS-derived | code yes, with the sample generator's own LibriTTS-R `.pt` generator **excluded** (base unverified); voices: **`en_US-libritts-high`** (from scratch, LibriTTS train-clean-360, CC BY 4.0, 904 speakers), **`en_US-kristin-medium`** (from scratch, LibriVox, public domain), **`en_US-john-medium`** (fine-tuned from Kristin), **`en_GB-cori-high`** (from scratch, LibriVox, public domain) | | openWakeWord / microWakeWord | Apache 2.0 (code); models CC BY-NC-SA 4.0; pre-computed feature sets CC BY-NC with WHAM / CHiME-6 upstream | Design reference only | design yes — **no code, models or feature sets imported**; every feature shard regenerated from manifest audio | | Hey Snips, Qualcomm KSD | research / NC | Comparison only, if at all | **no** | | Recorded hold-out set | ours; consent and permitted use recorded per talker | The evaluation set | committed only with its consent row | @@ -185,7 +185,7 @@ the *values*; `log_mel.h` for the *formulas*. Retraining never touches DspTap. near-unity async — so the integer-ratio decimator is new work (M1), built on the same DspTap substrate RatioTap uses. -### `tap::dsp::decimate` *(home per M0: DspTap primitive, or a RatioTap sibling)* +### `tap::dsp::decimate` *(DspTap, decided at M0)* - Ratios 2, 3, 6 as compile-time types, RatioTap's pattern; Kaiser-designed polyphase FIR from `kaiser.h` over `fir_kernels.h`; stopband attenuation, @@ -274,6 +274,37 @@ Five decisions, each cheap now and expensive later. shortlist if a phrase is chosen, and the compute budget of M4 named (where training runs, and a per-run time target). +**Decided, 4 September 2026** — recorded in HANDOFF.md; the arguments stay in +§9. + +1. **Repository: MuTap**, charter restated as portable speech DSP for embedded + targets alongside its adaptive-filter core. +2. **Host rate: fixed internal 16 kHz**; `kws.h` refuses other rates; + `@resample` on `mutap.wake~` backed by a **DspTap `decimate.h`** (ratios 2, + 3, 6 as types, RatioTap's design path), composed with RatioTap for 44.1 kHz. +3. **Release shape: runtime-first**, no bundled phrase; the training guide is + the primary document. +4. **Wake phrase: none shipped.** Development phrase for M4–M7 is a name-shaped + Speech Commands word (`marvin`, with `sheila` as the near-miss check) for + M5 bring-up, then a synthesized four-syllable phrase once Piper is in place + at M4 — chosen then, recorded in the manifest, never shipped. +5. **TTS voices, lineage-verified from the Piper model cards:** + `en_US-libritts-high` (trained from scratch on LibriTTS train-clean-360, + CC BY 4.0, 904 speakers — the primary voice, because speaker diversity is + the property that matters), `en_US-kristin-medium` and `en_GB-cori-high` + (trained from scratch on LibriVox, public domain), `en_US-john-medium` + (fine-tuned from Kristin). Excluded: every voice fine-tuned from Lessac + (`libritts_r`, `vctk`, `arctic`, `l2arctic`, `joe`, `kusal`, `alan`, + `alba`, `aru`, `northern_english_male`, `semaine`), the NC-dataset voices + (`hfc_*`, `semaine`), the voices with unverifiable base models (`bryce`, + `danny`, `kathleen`, `amy`), and the sample generator's bundled LibriTTS-R + `.pt` generator until its base voice is verified. Attribution text for + LibriTTS goes in the dataset card. + +**Compute:** an Apple Silicon Mac; the trainer gains `--device mps`; per-run +target under an hour on the 50 h development set. The remote containers are +for the C++ and the harness, never the corpus. + ### M1 — The mel front end, and the decimator *(DspTap)* `include/tap/dsp/log_mel.h` — `basic_log_mel` per the §5 contract, with @@ -620,7 +651,10 @@ only milestone with genuine unknowns. M1–M5 are known refactors and known harnesses. If a schedule is needed, treat M6 as the one to time-box with a decision point rather than estimate. -## 9. Open decisions +## 9. Decisions, with the arguments + +All of these were decided on 4 September 2026; the record is in M0 and in +HANDOFF.md. The arguments are kept here. **Which repository owns the spotter.** MuTap with a widened charter, or a new sibling library pinning DspTap. M1–M3 are correct under both; blocks at M4. From 73c6cad016b9e9b824192ae0c7a518a4edcf951d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 4 Sep 2026 22:03:15 +0000 Subject: [PATCH 10/14] docs: wake-word M1 done in DspTap; reference lives there Records M1 as built on DspTap's branch, with the measured tolerances that the plan required as committed numbers, the searched tap counts, and one deviation from the plan text: the numpy reference restatement lives in DspTap (tools/reference/make_frontend_reference.py) rather than MuTap's tools/ml, so the family has one numpy copy of the formulas and M4's feature module imports it through the submodule. Also notes that PCEN's steady state keeps a deliberate residual level dependence, which the test pins to the closed form. HANDOFF points at M2 next. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G --- HANDOFF.md | 2 +- docs/wake-word-plan.md | 24 ++++++++++++++++++++---- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index 81950a3..23dec3b 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -756,6 +756,6 @@ Resolved since rev 1: ~~license~~ (MIT), ~~core language~~ (header-only C++20), Resolved since rev 3: ~~Max external naming~~ — settled (Rev 4): **`mutap.afc~`** (rename from the `mutap.defeed~` placeholder) and **`mutap.aec~`** for the new echo canceller, an acronym pair matching the literature. The rename executes in Stage 2 of "The next effort" above. Still open: -- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. The named M33 target is the Raspberry Pi Pico 2 W (RP2350: single-precision FPU, so float32 is its profile; the M33 QEMU rig is ported from RatioTap in M2, and M7 adds a `pico2w` CI job that builds the board example against a pinned Pico SDK, uploads the UF2 and asserts its footprint — detection itself is a bench step with a loudspeaker-playback protocol and committed numbers, since QEMU has no RP2350 model), and the docs plan includes a user guide to training a phrase whose commands CI runs on a toy corpus. **M0 is decided (4 September 2026):** repository MuTap, charter widened; host rate fixed at 16 kHz in `kws.h` with conversion as an `@resample` option on the Max external, backed by a new DspTap `decimate.h` (2/3/6) and composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16; release shape runtime-first, no bundled phrase, the training guide as the primary document; development phrase `marvin` from Speech Commands for M5 bring-up, a synthesized four-syllable phrase from M4, never shipped; TTS voices lineage-verified from the Piper model cards — `en_US-libritts-high` (from scratch, CC BY 4.0, 904 speakers), `en_US-kristin-medium`, `en_GB-cori-high` (from scratch, public domain), `en_US-john-medium` (from Kristin), with every Lessac-derived voice (most of the English set, `libritts_r` and `vctk` included) and the sample generator's bundled `.pt` generator excluded; training on an Apple Silicon Mac via `--device mps`, under an hour per run on the development set. Next: M1. +- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. The named M33 target is the Raspberry Pi Pico 2 W (RP2350: single-precision FPU, so float32 is its profile; the M33 QEMU rig is ported from RatioTap in M2, and M7 adds a `pico2w` CI job that builds the board example against a pinned Pico SDK, uploads the UF2 and asserts its footprint — detection itself is a bench step with a loudspeaker-playback protocol and committed numbers, since QEMU has no RP2350 model), and the docs plan includes a user guide to training a phrase whose commands CI runs on a toy corpus. **M0 is decided (4 September 2026):** repository MuTap, charter widened; host rate fixed at 16 kHz in `kws.h` with conversion as an `@resample` option on the Max external, backed by a new DspTap `decimate.h` (2/3/6) and composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16; release shape runtime-first, no bundled phrase, the training guide as the primary document; development phrase `marvin` from Speech Commands for M5 bring-up, a synthesized four-syllable phrase from M4, never shipped; TTS voices lineage-verified from the Piper model cards — `en_US-libritts-high` (from scratch, CC BY 4.0, 904 speakers), `en_US-kristin-medium`, `en_GB-cori-high` (from scratch, public domain), `en_US-john-medium` (from Kristin), with every Lessac-derived voice (most of the English set, `libritts_r` and `vctk` included) and the sample generator's bundled `.pt` generator excluded; training on an Apple Silicon Mac via `--device mps`, under an hour per run on the development set. **M1 is done** on DspTap's `claude/mutap-wake-word-plan-2i63pe`: `log_mel.h` and `decimate.h` with their typed batteries, the numpy reference generator (`tools/reference/make_frontend_reference.py` — the family's single numpy copy of the formulas), C ABI and bridge; every tolerance in the plan's M1 record is a measured number. Next: M2, the oracles for the learned suppressor. - **Default engine in the external** — `@kalman` off (classic NLMS) is the shipping default purely on seniority; the measured case for flipping it is in `tests/test_fd_kalman.cpp` and book chapter 1. Decide after real-room listening. - **RIR fixtures, the measured half** — the fixture pipeline is built and three physically-modeled rooms (image-source, documented geometry) are committed baselines with regression tests. What remains yours: which MEASURED rooms join them — an academic dataset room (MYRiAD is the PEM-AFROW group's own database; openAIR is the other usual source; check each room's license allows redistribution in an MIT repo) and/or your own swept-sine measurements. Either way it is one command per room: `python3 tools/fixtures/make_rir_fixtures.py --from-wav room.wav myroom --source ""`, then a test with a freshly measured threshold. (The dataset hosts are unreachable from the remote dev container's network policy, so the WAVs have to enter via a commit.) diff --git a/docs/wake-word-plan.md b/docs/wake-word-plan.md index cfb11ba..ca8bf3e 100644 --- a/docs/wake-word-plan.md +++ b/docs/wake-word-plan.md @@ -314,10 +314,13 @@ on the same object. Under route (a), `decimate.h` beside it — ratios 2, 3 and 6 as types, RatioTap's design path (Kaiser prototype, committed scipy reference vectors, C ABI), in whichever home M0 chose. -**Before the header:** a throwaway numpy mel in `tools/ml/kws_features.py`, -written first and committed, is the reference M1 is scored against — the -family has no reference mel today, and M1's pass needs one that exists before -the C++ does. +**Before the header:** a numpy restatement of the contract, written first and +committed, is the reference M1 is scored against — the family has no +reference mel today, and M1's pass needs one that exists before the C++ does. +*As built:* it lives in DspTap as `tools/reference/make_frontend_reference.py` +(numpy only, generating `tests/reference/frontend_vectors.h`), not in MuTap, +so there is exactly one numpy copy of the formulas in the family; M4's +`kws_features.py` imports it through the submodule rather than restating it. Typed GoogleTest battery pinning every §5 contract point; float/double agreement measured, not assumed; C ABI exposure in `tools/capi` and the @@ -331,6 +334,19 @@ output frame for frame (the alignment contract); decimator passband ripple, stopband attenuation and latency pinned per ratio against the committed reference, if built. +**Done, 4 September 2026** (DspTap branch `claude/mutap-wake-word-plan-2i63pe`). +Measured, not estimated: C++ vs numpy on the reference signal — double +1.5e-14 (log) / 3.4e-14 (PCEN), float 6.7e-7 / 5.3e-6, pinned at 1e-13 and +1.2e-5; float vs double 6.7e-7 / 5.3e-6, pinned at 2×; decimator float vs numpy +under 3e-5 for all six ratio × profile cases, Q15 vs float 1.6e-4 at half +scale, pinned at 3.2e-4. Tap counts searched and pinned: economy 81 / 121 / 239, +transparent 259 / 389 / 773. Contract version 1. Both primitives are in the C +ABI and the `dsptap_py` bridge (`LogMel`, `Decimator`); the bridge reproduces +the numpy reference to 1.5e-14. 139 DspTap tests green, clang-format and +clang-tidy clean. One correction to §5 discovered in testing: PCEN's steady +state keeps a deliberate E^(1−α) level dependence, so "gain tracking" is +pinned to the closed form (E/(ε+E)^α + δ)^r − δ^r rather than to unity. + ### M2 — Oracles for the learned path *(MuTap)* — new in rev 2 Build the verification the suppressor should already have, before anything From 2cb1258209855664ecc7ad43e971e1c46be9f29c Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 00:19:01 +0000 Subject: [PATCH 11/14] M2: oracles for the learned suppressor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The wake-word plan's M2 — build the verification the learned path should already have, before M3 refactors its kernels into DspTap. - nn_suppressor tests typed over float and double, with a cross-precision pin (float vs double -129.4 dB measured, pinned at -120) and a learned chain gate in test_float32.cpp (-91.4 dB measured, pinned at -85). - Numeric contract written into the nn_suppressor.h docstring as observed: every dot product accumulates in Sample; nothing in the float profile touches double. - Contract defect found by the oracles and fixed on both sides: the ERB top edge round trip erb_inv(erb_rate(fs/2)) lands 2.2e-11 below fs/2 in libm at 48 kHz, so the strict band test dropped the Nyquist bin and the C++ suppressor notched bin N/2 while the numpy reference did not (48 kHz parity disagreed by 3e-2 / 8e-3). The top edge is now fs/2 exactly and the Nyquist bin belongs to the last band in nn_suppressor.h and features.py alike. - tools/ml/test_parity.py promoted to a CI job (nn-parity): both profiles (nn_infer --float), random weights at 16 k and 48 k plus the exported suppressor_v2_48k.munn; measured double <= 2.9e-8, float <= 2.9e-7, pinned at 1e-6. - Cortex-M33 leg ported from RatioTap: cmake/arm-cortex-m33-mps2.cmake, platform/mps2_an505.ld, shared armv8m_startup.c, Ooura float32 FFT (no MVE), cortex-m33-qemu CI job, m33 icount target and ratchet step. This is the Raspberry Pi Pico 2 W (RP2350) class of core. - The float nn_suppressor suite, the cross-precision pin and the chain gate added to the on-target filter on M33 and M55 (Hexagon runs everything). - bench/icount layer 4: nn_suppressor at both trained geometries with xorshift weights; m55 and m33 baselines seeded locally (the local ratchet reproduces every committed m55 baseline exactly). Hexagon's two entries are seeded from the first CI log per bench/README.md. - docs/wake-word-plan.md M2 record with the measured numbers; HANDOFF.md. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G --- .github/workflows/ci.yml | 87 +++++++++++ HANDOFF.md | 2 +- bench/README.md | 7 +- bench/baselines.json | 14 ++ bench/icount/CMakeLists.txt | 4 +- bench/icount/icount_main.cpp | 60 +++++++- cmake/arm-cortex-m33-mps2.cmake | 43 ++++++ docs/wake-word-plan.md | 51 ++++++- include/mutap/nn_suppressor.h | 19 ++- platform/mps2_an505.ld | 90 ++++++++++++ scripts/icount.py | 8 +- tests/CMakeLists.txt | 2 +- tests/bare_metal_main.cpp | 7 +- tests/test_float32.cpp | 79 ++++++++++ tests/test_nn_suppressor.cpp | 248 ++++++++++++++++++++++---------- tools/ml/README.md | 21 +++ tools/ml/features.py | 10 +- tools/ml/nn_infer.cpp | 45 +++--- tools/ml/test_parity.py | 88 ++++++++---- 19 files changed, 750 insertions(+), 135 deletions(-) create mode 100644 cmake/arm-cortex-m33-mps2.cmake create mode 100644 platform/mps2_an505.ld diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0fbdcaa..727c4ca 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -181,6 +181,37 @@ jobs: - name: Test under emulation (Ooura FFT fallback) run: ctest --test-dir build-ooura --output-on-failure + # Cortex-M33 (Raspberry Pi Pico 2 W / RP2350 class: single-precision FPU, + # no FP64, no MVE) on QEMU's MPS2+ AN505 model — the wake-word plan's named + # embedded target. Same Armv8-M startup as the M55 leg, Ooura float32 FFT + # (no Helium here), same emulation-sized float-profile selection. Ported + # from RatioTap's cortex-m33-qemu job. + cortex-m33-qemu: + name: Cortex-M33 cross (QEMU) + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 + with: + submodules: recursive + + - name: Install toolchain and QEMU + run: > + sudo apt-get update -q && + sudo apt-get install -y -q gcc-arm-none-eabi qemu-system-arm + + - name: Configure + run: > + cmake -B build + -DCMAKE_BUILD_TYPE=MinSizeRel + -DCMAKE_TOOLCHAIN_FILE=cmake/arm-cortex-m33-mps2.cmake + + - name: Build + run: cmake --build build -j 4 + + - name: Test under emulation + run: ctest --test-dir build --output-on-failure + # Cross-compile for Qualcomm Hexagon (hexagon-unknown-linux-musl, HVX # auto-vectorization on) and run the FULL test suite under qemu-hexagon # user-mode emulation: the third target of the one-core/three-targets @@ -362,6 +393,26 @@ jobs: python3 scripts/icount.py --target m55 --build-dir build-m55 --plugin /tmp/libinsncount.so + # Release (-O2) M33 workloads on the MPS2+ AN505 model: the Pico 2 W + # class core, Ooura float32 FFT (the toolchain file pins CMSIS off — + # no Helium). bench/baselines.json m33 was seeded from this exact + # toolchain/QEMU pair (gcc-arm-none-eabi 13.2.rel1, QEMU 8.2.2). + - name: Build M33 workloads + if: ${{ !cancelled() }} + run: > + cmake -B build-m33 + -DCMAKE_BUILD_TYPE=Release + -DCMAKE_TOOLCHAIN_FILE=cmake/arm-cortex-m33-mps2.cmake + -DMUTAP_BUILD_TESTS=OFF + -DMUTAP_BUILD_ICOUNT_BENCH=ON + && cmake --build build-m33 -j 4 + + - name: Ratchet M33 + if: ${{ !cancelled() }} + run: > + python3 scripts/icount.py --target m33 + --build-dir build-m33 --plugin /tmp/libinsncount.so + # qemu-system-arm's plugins cover M55; qemu-hexagon must be built with # --enable-plugins (linux-user only, ~4 min, cached thereafter). - name: Cache plugin-enabled qemu-hexagon @@ -427,6 +478,42 @@ jobs: python3 scripts/icount.py --target hexagon \ --build-dir build-hex --plugin /tmp/libinsncount.so + # Python <-> C++ parity of the learned suppressor (tools/ml/test_parity.py): + # the C++ inference against the numpy reference, in BOTH numeric profiles, + # on random weights at both trained geometries and on the shipping + # pretrained model. Previously a hand-run script; a feature definition + # drifting between features.py and nn_suppressor.h now fails here, not in + # a quietly degraded model (wake-word plan, M2). + nn-parity: + name: Suppressor Python/C++ parity + runs-on: ubuntu-latest + timeout-minutes: 20 + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 + with: + submodules: recursive + + - name: Install numpy + run: sudo apt-get update -q && sudo apt-get install -y -q python3-numpy + + - name: Build the parity driver + run: > + cmake -B build-ml -DCMAKE_BUILD_TYPE=Release + -DMUTAP_BUILD_TESTS=OFF -DMUTAP_BUILD_ML_TOOLS=ON + && cmake --build build-ml -j 4 --target nn_infer + + - name: Parity, double profile + run: | + python3 tools/ml/test_parity.py --build-dir build-ml --profile double --geometry 16k + python3 tools/ml/test_parity.py --build-dir build-ml --profile double --geometry 48k + python3 tools/ml/test_parity.py --build-dir build-ml --profile double --weights tools/ml/pretrained/suppressor_v2_48k.munn + + - name: Parity, float profile + run: | + python3 tools/ml/test_parity.py --build-dir build-ml --profile float --geometry 16k + python3 tools/ml/test_parity.py --build-dir build-ml --profile float --geometry 48k + python3 tools/ml/test_parity.py --build-dir build-ml --profile float --weights tools/ml/pretrained/suppressor_v2_48k.munn + clang-format: name: clang-format runs-on: ubuntu-latest diff --git a/HANDOFF.md b/HANDOFF.md index 23dec3b..cc24f39 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -756,6 +756,6 @@ Resolved since rev 1: ~~license~~ (MIT), ~~core language~~ (header-only C++20), Resolved since rev 3: ~~Max external naming~~ — settled (Rev 4): **`mutap.afc~`** (rename from the `mutap.defeed~` placeholder) and **`mutap.aec~`** for the new echo canceller, an acronym pair matching the literature. The rename executes in Stage 2 of "The next effort" above. Still open: -- **Wake-word detection — proposed, not started.** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. The named M33 target is the Raspberry Pi Pico 2 W (RP2350: single-precision FPU, so float32 is its profile; the M33 QEMU rig is ported from RatioTap in M2, and M7 adds a `pico2w` CI job that builds the board example against a pinned Pico SDK, uploads the UF2 and asserts its footprint — detection itself is a bench step with a loudspeaker-playback protocol and committed numbers, since QEMU has no RP2350 model), and the docs plan includes a user guide to training a phrase whose commands CI runs on a toy corpus. **M0 is decided (4 September 2026):** repository MuTap, charter widened; host rate fixed at 16 kHz in `kws.h` with conversion as an `@resample` option on the Max external, backed by a new DspTap `decimate.h` (2/3/6) and composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16; release shape runtime-first, no bundled phrase, the training guide as the primary document; development phrase `marvin` from Speech Commands for M5 bring-up, a synthesized four-syllable phrase from M4, never shipped; TTS voices lineage-verified from the Piper model cards — `en_US-libritts-high` (from scratch, CC BY 4.0, 904 speakers), `en_US-kristin-medium`, `en_GB-cori-high` (from scratch, public domain), `en_US-john-medium` (from Kristin), with every Lessac-derived voice (most of the English set, `libritts_r` and `vctk` included) and the sample generator's bundled `.pt` generator excluded; training on an Apple Silicon Mac via `--device mps`, under an hour per run on the development set. **M1 is done** on DspTap's `claude/mutap-wake-word-plan-2i63pe`: `log_mel.h` and `decimate.h` with their typed batteries, the numpy reference generator (`tools/reference/make_frontend_reference.py` — the family's single numpy copy of the formulas), C ABI and bridge; every tolerance in the plan's M1 record is a measured number. Next: M2, the oracles for the learned suppressor. +- **Wake-word detection — in progress (M0–M2 done).** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. The named M33 target is the Raspberry Pi Pico 2 W (RP2350: single-precision FPU, so float32 is its profile; the M33 QEMU rig is ported from RatioTap in M2, and M7 adds a `pico2w` CI job that builds the board example against a pinned Pico SDK, uploads the UF2 and asserts its footprint — detection itself is a bench step with a loudspeaker-playback protocol and committed numbers, since QEMU has no RP2350 model), and the docs plan includes a user guide to training a phrase whose commands CI runs on a toy corpus. **M0 is decided (4 September 2026):** repository MuTap, charter widened; host rate fixed at 16 kHz in `kws.h` with conversion as an `@resample` option on the Max external, backed by a new DspTap `decimate.h` (2/3/6) and composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16; release shape runtime-first, no bundled phrase, the training guide as the primary document; development phrase `marvin` from Speech Commands for M5 bring-up, a synthesized four-syllable phrase from M4, never shipped; TTS voices lineage-verified from the Piper model cards — `en_US-libritts-high` (from scratch, CC BY 4.0, 904 speakers), `en_US-kristin-medium`, `en_GB-cori-high` (from scratch, public domain), `en_US-john-medium` (from Kristin), with every Lessac-derived voice (most of the English set, `libritts_r` and `vctk` included) and the sample generator's bundled `.pt` generator excluded; training on an Apple Silicon Mac via `--device mps`, under an hour per run on the development set. **M1 is done** on DspTap's `claude/mutap-wake-word-plan-2i63pe`: `log_mel.h` and `decimate.h` with their typed batteries, the numpy reference generator (`tools/reference/make_frontend_reference.py` — the family's single numpy copy of the formulas), C ABI and bridge; every tolerance in the plan's M1 record is a measured number. **M2 is done** on MuTap's `claude/mutap-wake-word-plan-2i63pe`: the learned suppressor has its oracles — typed float/double tests with a −120 dB cross-precision pin, a float32 chain gate, a Python↔C++ parity CI job in both profiles on random and exported weights, a Cortex-M33 QEMU leg (mps2-an505, Ooura float32 FFT) with the float suppressor suite on-target on every leg, and `nn_suppressor` icount scenarios with m55/m33 baselines (hexagon's two are seeded from the first CI log). The oracles found and fixed a Nyquist-bin contract defect at 48 kHz (C++ notched bin N/2, numpy did not; fixed on both sides). Next: M3, promoting the dense/GRU kernels into DspTap's `tap::dsp::nn`. - **Default engine in the external** — `@kalman` off (classic NLMS) is the shipping default purely on seniority; the measured case for flipping it is in `tests/test_fd_kalman.cpp` and book chapter 1. Decide after real-room listening. - **RIR fixtures, the measured half** — the fixture pipeline is built and three physically-modeled rooms (image-source, documented geometry) are committed baselines with regression tests. What remains yours: which MEASURED rooms join them — an academic dataset room (MYRiAD is the PEM-AFROW group's own database; openAIR is the other usual source; check each room's license allows redistribution in an MIT repo) and/or your own swept-sine measurements. Either way it is one command per room: `python3 tools/fixtures/make_rir_fixtures.py --from-wav room.wav myroom --source ""`, then a test with a freshly measured threshold. (The dataset hosts are unreachable from the remote dev container's network policy, so the WAVs have to enter via a commit.) diff --git a/bench/README.md b/bench/README.md index b2e91d0..1847fff 100644 --- a/bench/README.md +++ b/bench/README.md @@ -112,7 +112,9 @@ stale, too-high baseline can never let a future regression hide in the slack — the winning commit must re-record). Scenarios mirror the wall-clock layers — `fdkf`, `suppressor`, `shadow`, -`chain` — at both certified geometries (`_48k`, `_16k`), all **float32** +`chain` — plus `nn_suppressor` (the learned post engine at its two trained +geometries, hop 256 at 48 kHz and hop 64 at 16 kHz) at both certified +geometries (`_48k`, `_16k`), all **float32** (the deployment precision; double is soft-float on the M55 and not the optimization target — the float32 parity gates in [`tests/test_float32.cpp`](../tests/test_float32.cpp) are the correctness @@ -142,6 +144,9 @@ python3 scripts/icount.py --target m55 \ --build-dir build-m55 --plugin /tmp/libinsncount.so ``` +Targets: `m55` (MPS3 AN547, CMSIS Helium FFT), `m33` (MPS2+ AN505 — the +Raspberry Pi Pico 2 W class, Ooura FFT, no MVE) and `hexagon`. + **Seeding / re-recording:** a new target starts with an empty dict, so the job reports each scenario's count and fails with `NO BASELINE`. Capture those counts by running the same command with `--update` in the target's diff --git a/bench/baselines.json b/bench/baselines.json index 2a0a582..26b05ef 100644 --- a/bench/baselines.json +++ b/bench/baselines.json @@ -9,11 +9,25 @@ "suppressor_16k": 255190544, "suppressor_48k": 251521827 }, + "m33": { + "chain_16k": 707703396, + "chain_48k": 832971429, + "fdkf_16k": 178361160, + "fdkf_48k": 306029768, + "nn_suppressor_16k": 1036433821, + "nn_suppressor_48k": 375501198, + "shadow_16k": 114513960, + "shadow_48k": 114515211, + "suppressor_16k": 435040766, + "suppressor_48k": 431364983 + }, "m55": { "chain_16k": 360481402, "chain_48k": 417303122, "fdkf_16k": 80854232, "fdkf_48k": 139974811, + "nn_suppressor_16k": 572321325, + "nn_suppressor_48k": 202455391, "shadow_16k": 51274139, "shadow_48k": 51279103, "suppressor_16k": 228296904, diff --git a/bench/icount/CMakeLists.txt b/bench/icount/CMakeLists.txt index 2336b13..8af6881 100644 --- a/bench/icount/CMakeLists.txt +++ b/bench/icount/CMakeLists.txt @@ -9,7 +9,9 @@ set(_mutap_icount_scenarios shadow_48k:2:0 shadow_16k:2:1 chain_48k:3:0 - chain_16k:3:1) + chain_16k:3:1 + nn_suppressor_48k:4:0 + nn_suppressor_16k:4:1) foreach(_sc IN LISTS _mutap_icount_scenarios) string(REPLACE ":" ";" _parts "${_sc}") diff --git a/bench/icount/icount_main.cpp b/bench/icount/icount_main.cpp index b65fe29..a965b09 100644 --- a/bench/icount/icount_main.cpp +++ b/bench/icount/icount_main.cpp @@ -17,14 +17,21 @@ // cross-run determinism. // // MUTAP_SC_LAYER: 0 = fdkf core, 1 = suppressor, 2 = shadow canceller, -// 3 = full certified chain +// 3 = full certified chain, 4 = learned nn_suppressor // MUTAP_SC_RATE: 0 = 48 kHz (2048 taps), 1 = 16 kHz (1024 taps) +// +// Layer 4 is the learned residual suppressor (nn_suppressor.h) at the two +// trained geometries — 48 kHz / hop 256 / 26 bands and 16 kHz / hop 64 / +// 22 bands, dense 64, GRU 96 — with deterministic xorshift weights (no +// file I/O on bare metal; the cost does not depend on the values). This +// is the per-hop cost the wake-word plan's ceilings are stated against. #include #include #include #include #include "mutap/fd_kalman.h" +#include "mutap/nn_suppressor.h" #include "mutap/postfilter.h" #ifndef MUTAP_SC_LAYER @@ -90,6 +97,36 @@ namespace { return tap::mu::aec_chain_preset(k_geo.block, k_geo.partitions, k_geo.fs); } + // Deterministic weights at the scenario's trained geometry (layer 4). + tap::mu::nn_suppressor_weights nn_weights() { +#if MUTAP_SC_RATE == 0 + const tap::mu::nn_geometry g{48000.0, 256, 26, 64, 96}; +#else + const tap::mu::nn_geometry g{16000.0, 64, 22, 64, 96}; +#endif + std::uint32_t s = 0x2545F491u; + auto fill = [&s](std::vector& v, std::size_t n) { + v.resize(n); + for (auto& x : v) { + s ^= s << 13; + s ^= s >> 17; + s ^= s << 5; + x = (static_cast(s) / 2147483648.0f - 1.0f) * 0.3f; + } + }; + tap::mu::nn_suppressor_weights w; + w.geometry = g; + fill(w.dense_in_w, g.dense * g.features()); + fill(w.dense_in_b, g.dense); + fill(w.gru_w_ih, 3 * g.gru * g.dense); + fill(w.gru_w_hh, 3 * g.gru * g.gru); + fill(w.gru_b_ih, 3 * g.gru); + fill(w.gru_b_hh, 3 * g.gru); + fill(w.dense_out_w, g.bands * g.gru); + fill(w.dense_out_b, g.bands); + return w; + } + // Accumulate a checksum over the timed output so nothing is dead code. double sum_out(const std::vector& e) noexcept { double s = 0.0; @@ -137,6 +174,27 @@ namespace { shadow.process_block(c.xb(i), c.yb(i), e.data()); sink += sum_out(e); } +#elif MUTAP_SC_LAYER == 4 + // The suppressor's hop is the trained one (256 at 48 kHz, 64 at + // 16 kHz); the corpus block is 256, so the 16 kHz scenario feeds + // four hops per corpus block. Same audio duration per scenario. + tap::mu::nn_suppressor::config cfg; + cfg.weights = nn_weights(); + tap::mu::nn_suppressor sup(std::move(cfg)); + const std::size_t hop = sup.block_size(); + const std::size_t hops = k_geo.block / hop; + std::vector eh(hop); + for (std::size_t i = 0; i < k_warm; ++i) { + for (std::size_t h = 0; h < hops; ++h) { + sup.process_block(c.yb(i) + h * hop, c.xb(i) + h * hop, eh.data()); + } + } + for (std::size_t i = k_warm; i < k_warm + k_timed; ++i) { + for (std::size_t h = 0; h < hops; ++h) { + sup.process_block(c.yb(i) + h * hop, c.xb(i) + h * hop, eh.data()); + sink += sum_out(eh); + } + } #else tap::mu::aec_chain chain(preset()); for (std::size_t i = 0; i < k_warm; ++i) { diff --git a/cmake/arm-cortex-m33-mps2.cmake b/cmake/arm-cortex-m33-mps2.cmake new file mode 100644 index 0000000..bd806f8 --- /dev/null +++ b/cmake/arm-cortex-m33-mps2.cmake @@ -0,0 +1,43 @@ +# Cross-compilation toolchain for Arm Cortex-M33 (bare metal, newlib + +# semihosting), executed on QEMU's MPS2+ AN505 board model. This is the +# Raspberry Pi Pico 2 W (RP2350) class of core — the wake-word plan's named +# embedded target: single-precision FPU only, no FP64, no MVE/Helium. The +# float32 profile is the profile here; anything double is soft-float and +# is excluded from the on-target selection. Ported from RatioTap's +# cmake/arm-cortex-m33-mps2.cmake (which ported it from SampleRateTap's). +# +# Usage: +# cmake -B build-m33 -DCMAKE_TOOLCHAIN_FILE=cmake/arm-cortex-m33-mps2.cmake \ +# -DCMAKE_BUILD_TYPE=MinSizeRel +# with arm-none-eabi-g++ and qemu-system-arm on PATH. +set(CMAKE_SYSTEM_NAME Generic) +set(CMAKE_SYSTEM_PROCESSOR arm) + +set(CMAKE_C_COMPILER arm-none-eabi-gcc) +set(CMAKE_CXX_COMPILER arm-none-eabi-g++) +set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY) + +set(CMAKE_C_FLAGS_INIT "-mcpu=cortex-m33 -mthumb -mfloat-abi=hard -ffunction-sections -fdata-sections") +set(CMAKE_CXX_FLAGS_INIT "${CMAKE_C_FLAGS_INIT}") + +get_filename_component(_mutap_platform "${CMAKE_CURRENT_LIST_DIR}/../platform" ABSOLUTE) +# Same startup as the M55 leg (Armv8-M, shared); the AN505 linker script +# places everything in the board's secure aliases (4 MB code, 4 MB data). +set(CMAKE_EXE_LINKER_FLAGS_INIT + "--specs=rdimon.specs -nostartfiles -Wl,--gc-sections -T${_mutap_platform}/mps2_an505.ld -x c ${_mutap_platform}/armv8m_startup.c -x none") + +set(CMAKE_CROSSCOMPILING_EMULATOR + "qemu-system-arm;-M;mps2-an505;-nographic;-semihosting;-kernel") + +set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) +set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) +set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY) + +# No Helium on the M33: DspTap defaults its CMSIS-DSP Helium FFT backend ON +# for any bare-metal Arm profile, so pin the Ooura float32 path here +# (a plain `set` of the cache entry, which DspTap's option() then respects). +set(TAP_DSP_FFT_CMSIS OFF CACHE BOOL "No MVE on the Cortex-M33: Ooura float32 FFT") + +# One-shot CTest mode (no argv on bare metal; see tests/CMakeLists.txt). +set(MUTAP_BARE_METAL ON) diff --git a/docs/wake-word-plan.md b/docs/wake-word-plan.md index ca8bf3e..e750c23 100644 --- a/docs/wake-word-plan.md +++ b/docs/wake-word-plan.md @@ -1,7 +1,8 @@ # Building `mutap.wake~` — implementation proposal -*Proposal, rev 2 — 4 September 2026. **Nothing committed beyond this document:** -no code written, no repository changed apart from these docs. Background and +*Proposal, rev 2 — 4 September 2026; **in progress**: M0 decided, M1 done +(DspTap) and M2 done (MuTap), each with a dated record of measured numbers +under its milestone in §6. Background and corpus survey in [`wake-word-briefing.md`](wake-word-briefing.md). Rev 2 carries every amendment from the [adversarial audit](wake-word-audit.md) of rev 1; the audit refers to rev 1's milestone numbers, and the mapping is given @@ -82,12 +83,12 @@ Read from the checkouts, with the audit's corrections applied. | Existing asset | What it does today | Role for the spotter | |---|---|---| -| `nn_suppressor.h` | Dense → GRU → dense over ERB band energies; allocation-free, noexcept; geometry carried by the weights. **Instantiated only as ``** in its tests, the parity driver and the shipping external. Every dot product accumulates in `Sample`. | The inference kernels, promoted in M3 *after* M2 gives them a float oracle. The spotter is a different head on the same arithmetic. | +| `nn_suppressor.h` | Dense → GRU → dense over ERB band energies; allocation-free, noexcept; geometry carried by the weights. Every dot product accumulates in `Sample`. ~~Instantiated only as ``~~ — *M2 typed its tests over float and double, pinned float vs double at −120 dB, and put the float profile on-target on M33, M55 and Hexagon.* | The inference kernels, promoted in M3 *after* M2 gives them a float oracle. The spotter is a different head on the same arithmetic. | | `nn_geometry` / MUNN0002 | Model geometry as a value, validated at load; 16 kHz hop-64 and 48 kHz hop-256 served by one inference path. **Validator requires a power-of-two hop.** | Copy the geometry-as-value pattern for `kws_geometry` and a `MUKW0001` header; do *not* copy the hop constraint. Retraining at a new geometry must not require a code change. | | `tools/ml/features.py` | Declared single source of truth for band definitions and normalization; documents that streaming-state parity depends on frame alignment. | The precedent for `kws_features.py` — with the ownership boundary of §5 so the two sources of truth cannot disagree. | -| `tools/ml/test_parity.py` | C++ **double** inference against `nn.py`'s numpy reference, **random weights**, legacy 16 kHz geometry, gain path only, hand-run behind `MUTAP_BUILD_ML_TOOLS=OFF`. Not a CI job. | The shape of the test the spotter needs. M2 turns it into one: CI-run, both profiles, exported trainer weights, shipping geometry. | +| `tools/ml/test_parity.py` | C++ inference against `nn.py`'s numpy reference, gain path only. ~~Double only, random weights, legacy 16 kHz geometry, hand-run, not a CI job~~ — *since M2 a CI job (`nn-parity`): both profiles, random weights at 16 k and 48 k plus the exported v2 weights, pinned at 1e-6.* | The shape of the test the spotter needs. M2 turns it into one: CI-run, both profiles, exported trainer weights, shipping geometry. | | `tools/ml/README.md` | The licensing map and measured benchmark that justified the hybrid design; files "int8 + CMSIS-NN for the M55 path" as next step. | The template for M4's dataset card, and a roadmap this plan must reconcile with (§5, `tap::dsp::nn`). | -| Cortex-M55 QEMU rig + `scripts/icount.py` | On-target positive-filter test subset in CI; whole-binary instruction count per scenario with a ±3 % drift gate against `bench/baselines.json` (`fdkf`, `chain` at 16 k and 48 k, on m55 and hexagon). **No learned-path scenario; `NnSuppressor` not in the on-target filter.** | The rig the embedded profile runs on, once M2 adds the learned scenarios. The ratchet is a drift gate; the budget is a separate absolute assertion (§7). | +| Cortex-M55 QEMU rig + `scripts/icount.py` | On-target positive-filter test subset in CI; whole-binary instruction count per scenario with a ±3 % drift gate against `bench/baselines.json` (`fdkf`, `chain` at 16 k and 48 k, on m55 and hexagon). ~~No learned-path scenario; `NnSuppressor` not in the on-target filter~~ — *M2 added the `nn_suppressor` scenario at both geometries, the M33 leg, and the float suppressor suite to every on-target filter.* | The rig the embedded profile runs on, once M2 adds the learned scenarios. The ratchet is a drift gate; the budget is a separate absolute assertion (§7). | | DspTap `fft.h` backend pattern | Ooura golden model; CMSIS-Helium and vDSP float32 backends re-presenting the exact contract, certified by parity tests. | The mel front end's FFT, and the rule for any accelerated NN backend: optional, opt-in, parity-pinned against the scalar golden path. | | RatioTap | Synchronous 44.1 ↔ 48 kHz, one rational pair by charter ("no other ratios"), compile-time direction type, Kaiser prototype over the DspTap substrate, instruction-count ratchet on M33/M55/Hexagon. SampleRateTap beside it is near-unity async only. | Composed by `mutap.wake~` for 44.1 kHz hosts (44.1 → 48, then 3:1 to 16 kHz), and the *design template* for the decimator: fixed ratios as types, speed-first profiles, ratchet-gated. | | DspTap `sample_traits.h`, `kaiser.h`, `fir_kernels.h` | The FIR substrate: float / Q15 / Q31 format core with documented Q-format ladders, Kaiser prototype design, dot kernels. No fixed-point FFT; the rate converters themselves live in SampleRateTap and RatioTap. | The *convention* for any later Q15 front end — not a fixed-point front end in itself — and the substrate the integer-ratio decimator of M1 is built on, exactly as RatioTap builds on it. | @@ -373,6 +374,46 @@ refactors it: accumulator-precision contract of §5 written down as the observed behaviour. This milestone is worth doing even if the project stops here. +**Done, 5 September 2026** (MuTap branch `claude/mutap-wake-word-plan-2i63pe`). +Measured, not estimated. `nn_suppressor` is now a typed suite over float and +double (unit-gain transparency pinned at −140 dB double / −120 dB float, the +comfort floor, echo-explained tracking at 1e-6 / 1e-4, the shipping 48 kHz +geometry, the chain composition); float vs double on the suppressor −129.4 dB, +pinned at −120; the learned chain in `test_float32.cpp` −91.4 dB, pinned at −85. +The parity driver runs both profiles (`nn_infer … --float`) and the CI job runs +six cases — random 16 k, random 48 k and the exported `suppressor_v2_48k.munn`, +each in double and float: double 1.6e-8 / 2.0e-8 / 2.9e-8, float 2.5e-7 / +2.0e-7 / 2.9e-7, both pinned at 1e-6. The accumulator contract is written into +the header docstring as observed: every dot product accumulates in `Sample`, +nothing in the float profile touches double. The M33 leg is ported from RatioTap +(`cmake/arm-cortex-m33-mps2.cmake`, `platform/mps2_an505.ld`, the shared +`armv8m_startup.c`, a `cortex-m33-qemu` CI job with the Ooura float32 FFT +pinned since there is no MVE) and the on-target filter on every leg carries the +float `nn_suppressor` suite, the cross-precision pin and the chain test. Layer 4 +of `bench/icount` is the suppressor at both trained geometries with xorshift +weights; baselines seeded locally on m55 and m33 (the local ratchet reproduces +every committed m55 baseline to 0.00 %, so local seeding is trustworthy) and +**not yet on hexagon** — the first CI run of the icount job reports the two +missing counts as `NO BASELINE`, and they are committed from that log per the +seeding procedure in `bench/README.md`. Per-hop cost of the *existing* GRU +suppressor, whole-binary count divided by hops processed (setup included, so an +upper bound): m55 ≈ 383 k instructions/hop at 48 kHz (hop 256) and ≈ 271 k at +16 kHz (hop 64); m33 ≈ 711 k and ≈ 491 k. Against the wake-word ceiling of +150 k/hop in §7 that is the number M6's spotter has to beat by 2–5×, which is +why the spotter is a depthwise-separable head and not this GRU. + +One contract defect found by the oracles, fixed on both sides: the ERB band +edges are built from `erb_inv(erb_rate(fs/2))`, and at 48 kHz that round trip +lands 2.2e-11 *below* fs/2 in libm (above it at 16 kHz; above at both in numpy), +so the strict `f < hi` band test dropped the Nyquist bin from the last band — +the C++ suppressor notched bin N/2 at 48 kHz while the numpy reference did +not. Before the fix the 48 kHz parity cases disagreed by 3e-2 (random) and +8e-3 (v2 model) in both profiles, and the shipping-geometry test read −27.9 dB +on unit gains. The top edge is now fs/2 exactly and the Nyquist bin belongs +to the last band, in `nn_suppressor.h` and `features.py` alike; this is the +kind of finding the audit predicted an oracle-less learned path would carry +unseen, and the reason §5's ownership rule exists. + ### M3 — Promote the inference kernels *(DspTap · MuTap)* Lift the dense and GRU arithmetic — only those — out of `nn_suppressor.h` into diff --git a/include/mutap/nn_suppressor.h b/include/mutap/nn_suppressor.h index 8660380..566008a 100644 --- a/include/mutap/nn_suppressor.h +++ b/include/mutap/nn_suppressor.h @@ -82,7 +82,16 @@ namespace tap::mu { /// /// The analysis/synthesis and the network mirror tools/ml/features.py /// and tools/ml/nn.py to float precision (tools/ml/test_parity.py - /// drives both on the same signals). The gain path is E-only: the + /// drives both on the same signals, in both profiles, in CI). + /// + /// Numeric contract: every dot product (dense_in, the GRU gates, + /// dense_out) and every band energy accumulates in Sample. Double is + /// the golden model; the float profile contains NO double arithmetic, + /// so it runs natively on parts without FP64 (the Cortex-M33 of the + /// RP2350 included) and never falls into soft-float. The float-tracks- + /// double depth is pinned by tests/test_nn_suppressor.cpp + /// (NnSuppressorCrossPrecision) and the promotion of these kernels into + /// DspTap must keep it unchanged. The gain path is E-only: the /// echo estimate Yhat feeds the FEATURES, never the signal path, so /// the worst a bad prediction can do is mis-gain a band — the /// structural safety that motivates learning gains instead of a @@ -260,6 +269,11 @@ namespace tap::mu { for (size_t i = 0; i < edges.size(); ++i) { edges[i] = erb_inv(top * static_cast(i) / static_cast(bands + 1)); } + // The top edge is fs/2 EXACTLY (features.py band_edges_hz): the ERB + // round trip lands ~1e-11 below it at 48 kHz and above it at 16 kHz, + // and used to decide whether the Nyquist bin was covered at all — + // notching it at 48 kHz in C++ while the Python reference kept it. + edges.back() = m_g.sample_rate / 2.0; m_bmat.assign(bands * bins, Sample(0)); for (size_t b = 0; b < bands; ++b) { const double lo = edges[b]; @@ -277,7 +291,8 @@ namespace tap::mu { m_bmat[b * bins + k] = static_cast(w); } } - m_bmat[0] = Sample(1); // DC belongs to the first band + m_bmat[0] = Sample(1); // DC belongs to the first band + m_bmat[(bands - 1) * bins + bins - 1] = Sample(1); // and the Nyquist bin to the last m_bnorm.assign(bins, Sample(0)); for (size_t k = 0; k < bins; ++k) { Sample s = Sample(0); diff --git a/platform/mps2_an505.ld b/platform/mps2_an505.ld new file mode 100644 index 0000000..867b9a1 --- /dev/null +++ b/platform/mps2_an505.ld @@ -0,0 +1,90 @@ +/* Linker script for the Arm MPS2+ AN505 FPGA image (Cortex-M33) as modeled + * by QEMU's mps2-an505 machine. The board boots secure with the initial + * vector table at the secure alias 0x1000_0000, so everything is placed in + * the secure aliases. QEMU's -kernel loader places the ELF directly into + * RAM (VMA == LMA, no load-time copy). + * Ported from RatioTap's platform tree (shared embedded story), which + * ported it from SampleRateTap's. + * + * Memory map (QEMU model, secure aliases): + * SSRAM1 4 MB @ 0x10000000 - vector table + code + rodata + * SSRAM2/3 4 MB @ 0x38000000 - data + bss + heap + stack + */ +MEMORY +{ + CODE (rx) : ORIGIN = 0x10000000, LENGTH = 4M + DATA (rw) : ORIGIN = 0x38000000, LENGTH = 4M +} + +__stack_top = ORIGIN(DATA) + LENGTH(DATA); + +ENTRY(Reset_Handler) + +SECTIONS +{ + .vectors : { + KEEP(*(.vectors)) + } > CODE + + .text : { + *(.text*) + *(.rodata*) + KEEP(*(.init)) + KEEP(*(.fini)) + } > CODE + + .ARM.extab : { + *(.ARM.extab* .gnu.linkonce.armextab.*) + } > CODE + + .ARM.exidx : { + __exidx_start = .; + *(.ARM.exidx* .gnu.linkonce.armexidx.*) + __exidx_end = .; + } > CODE + + .preinit_array : { + PROVIDE_HIDDEN(__preinit_array_start = .); + KEEP(*(.preinit_array*)) + PROVIDE_HIDDEN(__preinit_array_end = .); + } > CODE + + .init_array : { + PROVIDE_HIDDEN(__init_array_start = .); + KEEP(*(SORT(.init_array.*))) + KEEP(*(.init_array*)) + PROVIDE_HIDDEN(__init_array_end = .); + } > CODE + + .fini_array : { + PROVIDE_HIDDEN(__fini_array_start = .); + KEEP(*(SORT(.fini_array.*))) + KEEP(*(.fini_array*)) + PROVIDE_HIDDEN(__fini_array_end = .); + } > CODE + + .data : { + *(.data*) + } > DATA + + .bss (NOLOAD) : { + __bss_start__ = .; + *(.bss*) + *(COMMON) + __bss_end__ = .; + } > DATA + + /* Stack lives at the top of DATA; cap the heap 64 KB below it. */ + .heap (NOLOAD) : ALIGN(8) { + __heap_start__ = .; + . = ORIGIN(DATA) + LENGTH(DATA) - 64K; + __heap_end__ = .; + } > DATA + + /* MSPLIM (set in Reset_Handler): the stack may descend to the heap cap + * but no further — overflow into the heap faults instead of corrupting. */ + __stack_limit = __heap_end__; + + /* librdimon's (unused, weak) _sbrk references `end`; satisfy it. */ + PROVIDE(end = __heap_start__); +} diff --git a/scripts/icount.py b/scripts/icount.py index 68e7e55..5e5fadf 100755 --- a/scripts/icount.py +++ b/scripts/icount.py @@ -4,7 +4,7 @@ Runs every mutap_icount_* binary in a build directory under QEMU with the instruction-counting plugin, then compares against bench/baselines.json. - icount.py --target {hexagon,m55} --build-dir DIR --plugin LIB [--update] + icount.py --target {hexagon,m55,m33} --build-dir DIR --plugin LIB [--update] [--baselines bench/baselines.json] [--tolerance 0.03] The gate is two-sided: exit nonzero if any scenario regresses beyond @@ -33,6 +33,10 @@ def qemu_cmd(target: str, plugin: str, binary: str) -> list[str]: return ["qemu-system-arm", "-M", "mps3-an547", "-nographic", "-semihosting", "-d", "plugin", "-plugin", plugin, "-kernel", binary] + if target == "m33": + return ["qemu-system-arm", "-M", "mps2-an505", "-nographic", + "-semihosting", "-d", "plugin", "-plugin", plugin, + "-kernel", binary] raise SystemExit(f"unknown target {target}") @@ -55,7 +59,7 @@ def measure(target: str, plugin: str, binary: str) -> int: def main() -> int: ap = argparse.ArgumentParser() - ap.add_argument("--target", required=True, choices=["hexagon", "m55"]) + ap.add_argument("--target", required=True, choices=["hexagon", "m55", "m33"]) ap.add_argument("--build-dir", required=True) ap.add_argument("--plugin", required=True) ap.add_argument("--baselines", default="bench/baselines.json") diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index ff91258..af2d493 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -74,7 +74,7 @@ else() set(mutap_emulated_selection "") if(CMAKE_CROSSCOMPILING) set(mutap_emulated_selection TEST_FILTER - "real_fft_test/0.*:real_fft_test/1.*:RealFftCrossPrecision.*:CertifiedGeometries/fft_backend_parity.*:fdaf_test/0.*:FdafCrossPrecision.*:FdafConfigValidation.*:FdafRtContract.*:fd_kalman_test/0.*:fd_kalman_test/1.*:kalman_loop_test/0.*:FdKalmanConfigValidation.*:FdKalmanRtContract.*:Levinson.*:LpcPredictor.*:SpeechPredictor.*:WarpedLpcPredictor.*:PredictorConfigValidation.*:pem_afc_test/0.*:PemAfcConfigValidation.*:PemAfcRtContract.*:closed_loop_test/0.*:burst_test/0.*:aec_test/0.*:AdaptationControlConfigValidation.*") + "real_fft_test/0.*:real_fft_test/1.*:RealFftCrossPrecision.*:CertifiedGeometries/fft_backend_parity.*:fdaf_test/0.*:FdafCrossPrecision.*:FdafConfigValidation.*:FdafRtContract.*:fd_kalman_test/0.*:fd_kalman_test/1.*:kalman_loop_test/0.*:FdKalmanConfigValidation.*:FdKalmanRtContract.*:Levinson.*:LpcPredictor.*:SpeechPredictor.*:WarpedLpcPredictor.*:PredictorConfigValidation.*:pem_afc_test/0.*:PemAfcConfigValidation.*:PemAfcRtContract.*:closed_loop_test/0.*:burst_test/0.*:aec_test/0.*:AdaptationControlConfigValidation.*:nn_suppressor_test/0.*:NnSuppressorCrossPrecision.*:NnChainFloat32.*") endif() gtest_discover_tests(mutap_tests DISCOVERY_TIMEOUT 120 ${mutap_emulated_selection} PROPERTIES TIMEOUT 900) diff --git a/tests/bare_metal_main.cpp b/tests/bare_metal_main.cpp index 2af33a8..d4caf60 100644 --- a/tests/bare_metal_main.cpp +++ b/tests/bare_metal_main.cpp @@ -4,7 +4,9 @@ // the float32 typed suites (the embedded profile this target exists for), // the double FFT (small; exercises the soft-float path), the LP / // conditioning suite, the float closed-loop scenarios including the PEM -// canceller's tonal headline, and the float-tracks-double oracle check. +// canceller's tonal headline, the float-tracks-double oracle check, and +// the learned suppressor's float profile with its own oracle check (the +// wake-word plan's M2: the learned path had never run on target before). // // Excluded: the double-typed adaptive suites and the double closed-loop // scenarios — minutes-to-hours of soft-float virtual audio validating @@ -26,7 +28,8 @@ int main() { "Levinson.*:LpcPredictor.*:SpeechPredictor.*:WarpedLpcPredictor.*:PredictorConfigValidation.*:" "pem_afc_test/0.*:PemAfcConfigValidation.*:PemAfcRtContract.*:" "closed_loop_test/0.*:burst_test/0.*:aec_test/0.*:" - "AdaptationControlConfigValidation.*"; + "AdaptationControlConfigValidation.*:" + "nn_suppressor_test/0.*:NnSuppressorCrossPrecision.*:NnChainFloat32.*"; ::testing::InitGoogleTest(); const int rc = RUN_ALL_TESTS(); // A filter typo selects zero tests and RUN_ALL_TESTS() returns 0 — an diff --git a/tests/test_float32.cpp b/tests/test_float32.cpp index 15647e7..530c210 100644 --- a/tests/test_float32.cpp +++ b/tests/test_float32.cpp @@ -38,12 +38,14 @@ #include #include +#include #include #include #include #include "mutap/fd_kalman.h" +#include "mutap/nn_chain.h" #include "mutap/postfilter.h" #include "support/echo_scenario.h" #include "support/itu_chain.h" @@ -246,4 +248,81 @@ namespace { } } + // The learned chain (raw FD-Kalman + nn_suppressor) in the float32 + // profile, on the same synthetic echo the instruction-count workloads + // use, with a live deterministic network: finite output, the canceller + // still cancels through the learned post (output energy well below the + // microphone's), and float tracks the double golden model at a measured, + // pinned depth: -91.4 dB measured 2026-09 (the canceller's float state + // dominates; the suppressor alone tracks at -129 dB), pinned at -85 dB. The wake-word plan's M2: the learned path + // had never run in float, on target, before this test. + TEST(NnChainFloat32, TracksDoubleOnASyntheticEcho) { + constexpr size_t block = 64, partitions = 4, blocks = 300; + constexpr double fs = 16000.0; + + tap::mu::nn_suppressor_weights w; + { + std::uint32_t s = 0x2545F491u; + auto fill = [&s](std::vector& v, size_t n) { + v.resize(n); + for (auto& x : v) { + s ^= s << 13; + s ^= s >> 17; + s ^= s << 5; + x = (static_cast(s) / 2147483648.0f - 1.0f) * 0.3f; + } + }; + const auto& g = w.geometry; // default: 16 kHz / hop 64 / 22 bands + fill(w.dense_in_w, g.dense * g.features()); + fill(w.dense_in_b, g.dense); + fill(w.gru_w_ih, 3 * g.gru * g.dense); + fill(w.gru_w_hh, 3 * g.gru * g.gru); + fill(w.gru_b_ih, 3 * g.gru); + fill(w.gru_b_hh, 3 * g.gru); + fill(w.dense_out_w, g.bands * g.gru); + fill(w.dense_out_b, g.bands); + } + tap::mu::aec_chain_nn cd(tap::mu::aec_chain_nn_preset(block, partitions, fs, w)); + tap::mu::aec_chain_nn cf(tap::mu::aec_chain_nn_preset(block, partitions, fs, w)); + + // Synthetic echo: xorshift far end, sparse 3-tap path, tiny near floor. + const size_t n = blocks * block; + std::vector x(n), y(n); + std::uint32_t s = 0x9E3779B9u; + auto next = [&s]() { + s ^= s << 13; + s ^= s >> 17; + s ^= s << 5; + return (static_cast(s) / 2147483648.0 - 1.0) * 0.1; + }; + for (auto& v : x) { + v = next(); + } + for (size_t i = 0; i < n; ++i) { + const auto at = [&](size_t d) { return i >= d ? x[i - d] : 0.0; }; + y[i] = 0.25 * at(block / 2) - 0.12 * at(block) + 0.06 * at(3 * block / 2) + 0.001 * next(); + } + std::vector xf(x.begin(), x.end()), yf(y.begin(), y.end()), ef(block); + std::vector ed(block); + double err = 0.0, ref = 0.0, mic = 0.0, out = 0.0; + for (size_t b = 0; b < blocks; ++b) { + cd.process_block(&x[b * block], &y[b * block], ed.data()); + cf.process_block(&xf[b * block], &yf[b * block], ef.data()); + if (b >= blocks / 2) { + for (size_t i = 0; i < block; ++i) { + ASSERT_TRUE(std::isfinite(ef[i])); + const double d = ed[i] - static_cast(ef[i]); + err += d * d; + ref += ed[i] * ed[i]; + out += static_cast(ef[i]) * static_cast(ef[i]); + mic += y[b * block + i] * y[b * block + i]; + } + } + } + EXPECT_LT(10.0 * std::log10(out / mic), -20.0) << "the learned chain must still cancel"; + const double rel_db = 10.0 * std::log10(err / ref); + RecordProperty("float_vs_double_db", rel_db); + EXPECT_LT(rel_db, -85.0) << "float chain drifts from the double golden model"; + } + } // namespace diff --git a/tests/test_nn_suppressor.cpp b/tests/test_nn_suppressor.cpp index cbf99c6..00513ca 100644 --- a/tests/test_nn_suppressor.cpp +++ b/tests/test_nn_suppressor.cpp @@ -2,11 +2,12 @@ // Copyright 2026 MuTap contributors // // Structural tests for the learned residual suppressor's inference plumbing -// (nn_suppressor.h): the STFT analysis/synthesis path, the gain application, -// and the chain-contract surface (echo_explained, comfort noise), exercised -// with weights CONSTRUCTED to force known network outputs (numerical parity -// of the full net against the Python reference is tools/ml/test_parity.py, -// which drives real weights through both). +// (nn_suppressor.h), typed over BOTH numeric profiles — float is the +// embedded profile the Cortex-M33/M55 and Hexagon legs run, double the +// golden model — plus the float-tracks-double oracle check. Exercised with +// weights CONSTRUCTED to force known network outputs (numerical parity of +// the full net against the Python reference is tools/ml/test_parity.py, +// which drives real weights through both profiles in CI). // // - dense_out bias = +20 -> sigmoid saturates, every band gain ~= 1: the // suppressor must be a transparent one-block delay (perfect sqrt-Hann @@ -16,6 +17,13 @@ // noise off, and ~= the tracked noise floor with it on. // - echo_explained() ~= 1 when yhat ~= mic, ~= 0 when yhat = 0. // - the chain composes with nn_suppressor as its Post engine. +// - the shipping 48 kHz / hop-256 / 26-band geometry runs the same code. +// - float tracks double on a live (non-saturated) network at a measured, +// pinned tolerance: the GRU recurrence is where float error accumulates. +// +// Numeric contract pinned here (see nn_suppressor.h): every dot product +// accumulates in Sample. There is no double arithmetic in the float +// profile, which is what lets it run on parts without FP64. #include #include @@ -30,12 +38,16 @@ namespace { using tap::mu::aec_chain; + using tap::mu::nn_geometry; using tap::mu::nn_suppressor; using tap::mu::nn_suppressor_weights; constexpr size_t k_block = 64; - nn_suppressor_weights make_weights(unsigned seed, float out_bias) { + /// Deterministic weights at a geometry. bias_only zeroes dense_out_w so + /// the output is driven by out_bias alone (saturating the sigmoid); + /// otherwise the network is live and the gains vary with the input. + nn_suppressor_weights make_weights(unsigned seed, float out_bias, bool bias_only, const nn_geometry& g = {}) { std::mt19937 gen(seed); std::normal_distribution dist(0.0f, 0.3f); const auto fill = [&](std::vector& v, size_t n) { @@ -44,72 +56,86 @@ namespace { x = dist(gen); } }; - nn_suppressor_weights w; // default geometry: 16 kHz / hop 64 - fill(w.dense_in_w, 64 * 44); - fill(w.dense_in_b, 64); - fill(w.gru_w_ih, 288 * 64); - fill(w.gru_w_hh, 288 * 96); - fill(w.gru_b_ih, 288); - fill(w.gru_b_hh, 288); - fill(w.dense_out_w, 22 * 96); - w.dense_out_w.assign(22 * 96, 0.0f); // output driven by bias alone - w.dense_out_b.assign(22, out_bias); + nn_suppressor_weights w; + w.geometry = g; + fill(w.dense_in_w, g.dense * g.features()); + fill(w.dense_in_b, g.dense); + fill(w.gru_w_ih, 3 * g.gru * g.dense); + fill(w.gru_w_hh, 3 * g.gru * g.gru); + fill(w.gru_b_ih, 3 * g.gru); + fill(w.gru_b_hh, 3 * g.gru); + fill(w.dense_out_w, g.bands * g.gru); + if (bias_only) { + w.dense_out_w.assign(g.bands * g.gru, 0.0f); + } + w.dense_out_b.assign(g.bands, out_bias); return w; } - nn_suppressor::config make_config(unsigned seed, float out_bias, bool comfort) { - nn_suppressor::config cfg; - cfg.weights = make_weights(seed, out_bias); + template + typename nn_suppressor::config make_config(unsigned seed, float out_bias, bool comfort) { + typename nn_suppressor::config cfg; + cfg.weights = make_weights(seed, out_bias, true); cfg.comfort_noise = comfort; return cfg; } - std::vector noise(size_t n, unsigned seed, double rms = 1.0) { + template + std::vector noise(size_t n, unsigned seed, double rms = 1.0) { std::mt19937 gen(seed); std::normal_distribution dist(0.0, rms); - std::vector v(n); + std::vector v(n); for (auto& x : v) { - x = dist(gen); + x = static_cast(dist(gen)); } return v; } - TEST(NnSuppressor, UnitGainsAreATransparentOneBlockDelay) { - nn_suppressor sup(make_config(7, 20.0f, false)); // sigmoid(20) ~ 1 - const size_t blocks = 50; - const auto e = noise(blocks * k_block, 11); - const auto yhat = noise(blocks * k_block, 12); + template + class nn_suppressor_test : public ::testing::Test {}; + using sample_types = ::testing::Types; + TYPED_TEST_SUITE(nn_suppressor_test, sample_types); + + TYPED_TEST(nn_suppressor_test, UnitGainsAreATransparentOneBlockDelay) { + nn_suppressor sup(make_config(7, 20.0f, false)); // sigmoid(20) ~ 1 + const size_t blocks = 50; + const auto e = noise(blocks * k_block, 11); + const auto yhat = noise(blocks * k_block, 12); - std::vector out(blocks * k_block); + std::vector out(blocks * k_block); for (size_t b = 0; b < blocks; ++b) { sup.process_block(&e[b * k_block], &yhat[b * k_block], &out[b * k_block]); } - for (const double g : sup.band_gains()) { - EXPECT_NEAR(g, 1.0, 1e-8); + for (const TypeParam g : sup.band_gains()) { + EXPECT_NEAR(g, 1.0, 1e-6); } // out trails e by one block; skip the warm-up frame. double err = 0.0; double ref = 0.0; for (size_t i = 2 * k_block; i < out.size(); ++i) { - err += (out[i] - e[i - k_block]) * (out[i] - e[i - k_block]); - ref += e[i - k_block] * e[i - k_block]; + const double d = static_cast(out[i]) - static_cast(e[i - k_block]); + err += d * d; + ref += static_cast(e[i - k_block]) * static_cast(e[i - k_block]); } - EXPECT_LT(10.0 * std::log10(err / ref), -140.0) << "reconstruction should be float64-deep"; + // Reconstruction sits at the profile's own rounding floor: float64-deep + // for double, a float32 epsilon walk (~-135 dB measured) for float. + const double bound_db = std::is_same_v ? -140.0 : -120.0; + EXPECT_LT(10.0 * std::log10(err / ref), bound_db) << "reconstruction should sit at the rounding floor"; } - TEST(NnSuppressor, ZeroGainsSilenceTheOutputWithoutComfortNoise) { - nn_suppressor sup(make_config(7, -20.0f, false)); // sigmoid(-20) ~ 0 - const size_t blocks = 50; - const auto e = noise(blocks * k_block, 11); - const auto yhat = noise(blocks * k_block, 12); + TYPED_TEST(nn_suppressor_test, ZeroGainsSilenceTheOutputWithoutComfortNoise) { + nn_suppressor sup(make_config(7, -20.0f, false)); // sigmoid(-20) ~ 0 + const size_t blocks = 50; + const auto e = noise(blocks * k_block, 11); + const auto yhat = noise(blocks * k_block, 12); - std::vector out(blocks * k_block); - double energy = 0.0; + std::vector out(blocks * k_block); + double energy = 0.0; for (size_t b = 0; b < blocks; ++b) { sup.process_block(&e[b * k_block], &yhat[b * k_block], &out[b * k_block]); } for (size_t i = 0; i < out.size(); ++i) { - energy += out[i] * out[i]; + energy += static_cast(out[i]) * static_cast(out[i]); } EXPECT_LT(energy, 1e-12); } @@ -119,86 +145,164 @@ namespace { // within a few dB of the input level once the first minimum-statistics // window has completed. The floor bias intentionally overshoots the // biased-low minimum statistic, so bound both sides loosely. - TEST(NnSuppressor, ZeroGainsSettleAtTheComfortFloor) { - auto cfg = make_config(7, -20.0f, true); + TYPED_TEST(nn_suppressor_test, ZeroGainsSettleAtTheComfortFloor) { + auto cfg = make_config(7, -20.0f, true); cfg.floor_window = 32; // complete both min-statistics windows quickly - nn_suppressor sup(std::move(cfg)); + nn_suppressor sup(std::move(cfg)); const size_t blocks = 300; - const auto e = noise(blocks * k_block, 11); - const auto yhat = noise(blocks * k_block, 12); + const auto e = noise(blocks * k_block, 11); + const auto yhat = noise(blocks * k_block, 12); - std::vector out(blocks * k_block); + std::vector out(blocks * k_block); for (size_t b = 0; b < blocks; ++b) { sup.process_block(&e[b * k_block], &yhat[b * k_block], &out[b * k_block]); } double fill = 0.0; double ref = 0.0; for (size_t i = out.size() / 2; i < out.size(); ++i) { - fill += out[i] * out[i]; - ref += e[i] * e[i]; + fill += static_cast(out[i]) * static_cast(out[i]); + ref += static_cast(e[i]) * static_cast(e[i]); } const double rel_db = 10.0 * std::log10(fill / ref); EXPECT_GT(rel_db, -10.0) << "comfort fill should sit near the floor, not at silence"; EXPECT_LT(rel_db, 6.0) << "and must not exceed the input level by more than the bias"; } - TEST(NnSuppressor, EchoExplainedTracksYhatShare) { - nn_suppressor sup(make_config(7, 0.0f, false)); - const size_t blocks = 100; - const auto sig = noise(blocks * k_block, 11); - std::vector out(blocks * k_block); - std::vector zeros(blocks * k_block, 0.0); + TYPED_TEST(nn_suppressor_test, EchoExplainedTracksYhatShare) { + nn_suppressor sup(make_config(7, 0.0f, false)); + const size_t blocks = 100; + const auto sig = noise(blocks * k_block, 11); + std::vector out(blocks * k_block); + std::vector zeros(blocks * k_block, TypeParam(0)); + const double tol = std::is_same_v ? 1e-6 : 1e-4; // yhat == mic-and-then-some: E ~ 0, yhat = signal -> explained ~ 1. for (size_t b = 0; b < blocks; ++b) { sup.process_block(&zeros[b * k_block], &sig[b * k_block], &out[b * k_block]); } - EXPECT_NEAR(sup.echo_explained(), 1.0, 1e-6); + EXPECT_NEAR(sup.echo_explained(), 1.0, tol); sup.reset(); // yhat == 0: nothing explained. for (size_t b = 0; b < blocks; ++b) { sup.process_block(&sig[b * k_block], &zeros[b * k_block], &out[b * k_block]); } - EXPECT_NEAR(sup.echo_explained(), 0.0, 1e-6); + EXPECT_NEAR(sup.echo_explained(), 0.0, tol); } - TEST(NnSuppressor, RejectsWrongGeometry) { - auto cfg = make_config(7, 0.0f, false); + TYPED_TEST(nn_suppressor_test, RejectsWrongGeometry) { + auto cfg = make_config(7, 0.0f, false); cfg.block_size = 128; // != trained hop 64 - EXPECT_THROW(nn_suppressor(std::move(cfg)), std::invalid_argument); - auto bad = make_config(7, 0.0f, false); + EXPECT_THROW(nn_suppressor(std::move(cfg)), std::invalid_argument); + auto bad = make_config(7, 0.0f, false); bad.weights.gru_b_ih.resize(5); - EXPECT_THROW(nn_suppressor(std::move(bad)), std::invalid_argument); + EXPECT_THROW(nn_suppressor(std::move(bad)), std::invalid_argument); + } + + // The shipping model's geometry (tools/ml/pretrained/suppressor_v2_48k): + // 48 kHz, hop 256, 26 bands, dense 64, GRU 96. Geometry is a value the + // weights carry, so the same code must run it unchanged — pinned with + // the transparent-delay check at that hop. + TYPED_TEST(nn_suppressor_test, RunsAtTheShippingGeometry) { + const nn_geometry g{48000.0, 256, 26, 64, 96}; + typename nn_suppressor::config cfg; + cfg.weights = make_weights(9, 20.0f, true, g); + cfg.comfort_noise = false; + nn_suppressor sup(std::move(cfg)); + EXPECT_EQ(sup.block_size(), 256U); + EXPECT_EQ(sup.geometry().bands, 26U); + + const size_t hop = 256; + const size_t blocks = 40; + const auto e = noise(blocks * hop, 31); + const auto yhat = noise(blocks * hop, 32); + std::vector out(blocks * hop); + for (size_t b = 0; b < blocks; ++b) { + sup.process_block(&e[b * hop], &yhat[b * hop], &out[b * hop]); + } + double err = 0.0; + double ref = 0.0; + for (size_t i = 2 * hop; i < out.size(); ++i) { + const double d = static_cast(out[i]) - static_cast(e[i - hop]); + err += d * d; + ref += static_cast(e[i - hop]) * static_cast(e[i - hop]); + } + const double bound_db = std::is_same_v ? -140.0 : -120.0; + EXPECT_LT(10.0 * std::log10(err / ref), bound_db); } // The chain composes with the learned post engine: block sizes match up // (matched() writes the canceller's block into the post config), the // guard reads the post's echo_explained(), and processing runs. - TEST(NnSuppressor, ComposesAsTheChainPostEngine) { - using chain_t = aec_chain, nn_suppressor>; - chain_t::config cfg; + TYPED_TEST(nn_suppressor_test, ComposesAsTheChainPostEngine) { + using chain_t = aec_chain, nn_suppressor>; + typename chain_t::config cfg; cfg.canceller.block_size = k_block; cfg.canceller.partitions = 4; - cfg.postfilter = make_config(7, 20.0f, false); + cfg.postfilter = make_config(7, 20.0f, false); cfg.guard_attenuation_db = 0.0; // guard off: pass-through check below chain_t chain(cfg); - const size_t blocks = 50; - const auto x = noise(blocks * k_block, 21, 0.5); - const auto y = noise(blocks * k_block, 22, 0.5); - std::vector e(blocks * k_block); + const size_t blocks = 50; + const auto x = noise(blocks * k_block, 21, 0.5); + const auto y = noise(blocks * k_block, 22, 0.5); + std::vector e(blocks * k_block); for (size_t b = 0; b < blocks; ++b) { chain.process_block(&x[b * k_block], &y[b * k_block], &e[b * k_block]); } double energy = 0.0; - for (const double v : e) { - energy += v * v; + for (const TypeParam v : e) { + energy += static_cast(v) * static_cast(v); } EXPECT_TRUE(std::isfinite(energy)); EXPECT_GT(energy, 0.0); EXPECT_TRUE(chain.converged()) << "guard disabled reports converged"; } + // The float-tracks-double oracle on a LIVE network (dense_out_w + // populated, bias 0, so gains follow the input through the GRU): the + // float profile must reproduce the double golden model's output stream + // to a stated depth. This is the check the promotion of the kernels + // (plan M3) must keep passing unchanged. Measured 2026-09 on this + // signal: -129.4 dB; pinned at -120 dB (9 dB of margin, so a regression + // in the float path is caught rather than absorbed). + TEST(NnSuppressorCrossPrecision, FloatTracksDouble) { + nn_suppressor::config cd; + cd.weights = make_weights(5, 0.0f, false); + cd.comfort_noise = false; + nn_suppressor::config cf; + cf.weights = cd.weights; + cf.comfort_noise = false; + nn_suppressor sd(std::move(cd)); + nn_suppressor sf(std::move(cf)); + + const size_t blocks = 200; + const auto e = noise(blocks * k_block, 41, 0.3); + const auto yhat = noise(blocks * k_block, 42, 0.2); + std::vector ef(e.begin(), e.end()); + std::vector yf(yhat.begin(), yhat.end()); + std::vector od(blocks * k_block); + std::vector of(blocks * k_block); + for (size_t b = 0; b < blocks; ++b) { + sd.process_block(&e[b * k_block], &yhat[b * k_block], &od[b * k_block]); + sf.process_block(&ef[b * k_block], &yf[b * k_block], &of[b * k_block]); + } + double err = 0.0; + double ref = 0.0; + double gain_span = 0.0; + for (size_t i = k_block; i < od.size(); ++i) { + const double d = od[i] - static_cast(of[i]); + err += d * d; + ref += od[i] * od[i]; + } + for (const double g : sd.band_gains()) { + gain_span = std::max(gain_span, std::abs(g - 0.5)); + } + EXPECT_GT(gain_span, 0.05) << "the network must be live, not saturated, for this to mean anything"; + const double rel_db = 10.0 * std::log10(err / ref); + RecordProperty("float_vs_double_db", rel_db); + EXPECT_LT(rel_db, -120.0) << "float profile drifts from the golden model"; + } + } // namespace diff --git a/tools/ml/README.md b/tools/ml/README.md index b4b0f98..fa762dd 100644 --- a/tools/ml/README.md +++ b/tools/ml/README.md @@ -159,6 +159,27 @@ engine ships in `mutap.aec~` as `@postfilter 2` (embedded default weights; `@model` loads alternatives), composed in the library as `tap::mu::aec_chain_nn` (`nn_chain.h`). +## Parity, in CI + +`test_parity.py` drives the same signals through the numpy reference and +the C++ inference (`nn_infer`, built with `-DMUTAP_BUILD_ML_TOOLS=ON`) and +is a CI job (`nn-parity`): both numeric profiles, random weights at both +trained geometries, and the shipping v2 model. Measured depths (relative to +the output peak) are pinned at 1e-6 for both profiles — double 1.6e-8 to +2.9e-8, float 2.0e-7 to 2.9e-7. + +It was promoted to CI in the wake-word plan's M2 and immediately earned its +place: at 48 kHz the two sides disagreed by 3e-2 (random weights) and 8e-3 +(the v2 model) in *both* profiles. Root cause was a rounding accident in the +band definition — the ERB round trip of fs/2 lands 2e-11 below fs/2 in libm +and 4e-12 above it in numpy, and a strict `f < hi` test then decided whether +the Nyquist bin was covered by any band: the C++ notched it at 48 kHz while +the reference kept it. The contract is now explicit on both sides +(`band_edges_hz` / `nn_suppressor::build_bands`): the top edge is fs/2 +exactly, and the Nyquist bin belongs to the last band as DC belongs to the +first. The shipping v2 model had been deployed with that notch; the +difference is one bin at 24 kHz. + ## The pipeline ``` diff --git a/tools/ml/features.py b/tools/ml/features.py index aab4ded..3822127 100644 --- a/tools/ml/features.py +++ b/tools/ml/features.py @@ -85,7 +85,14 @@ def band_edges_hz(num_bands: int, fmax: float) -> np.ndarray: erb_rate = lambda f: 21.4 * np.log10(1.0 + 0.00437 * f) # noqa: E731 inv = lambda r: (10.0 ** (r / 21.4) - 1.0) / 0.00437 # noqa: E731 r = np.linspace(0.0, erb_rate(fmax), num_bands + 2) - return inv(r) + edges = inv(r) + # The top edge is fmax EXACTLY, never the ERB round trip of it: that + # round trip lands a few 1e-12 above fmax at 16 kHz and below it at + # 48 kHz (and differs between numpy and libm), which used to decide + # whether the Nyquist bin was covered at all. Coverage is a contract, + # not a rounding accident — see band_matrix(). + edges[-1] = fmax + return edges def band_matrix(geom: Geometry = GEOM16) -> np.ndarray: @@ -100,6 +107,7 @@ def band_matrix(geom: Geometry = GEOM16) -> np.ndarray: w[b, up] = (freqs[up] - lo) / max(mid - lo, 1e-9) w[b, down] = (hi - freqs[down]) / max(hi - mid, 1e-9) w[0, 0] = 1.0 # DC belongs to the first band + w[-1, -1] = 1.0 # and the Nyquist bin to the last (a triangle's weight is 0 at its edge) return w diff --git a/tools/ml/nn_infer.cpp b/tools/ml/nn_infer.cpp index ab227a3..9a86b30 100644 --- a/tools/ml/nn_infer.cpp +++ b/tools/ml/nn_infer.cpp @@ -4,12 +4,16 @@ // Offline driver for tap::mu::nn_suppressor — the C++ half of the parity // test (tools/ml/test_parity.py): reads a MUNN0001 weights file plus raw // float64 e / yhat streams, processes block-by-block, writes the cleaned -// stream. Double instantiation, so parity with the float64 numpy reference -// is tight. +// stream. Double instantiation by default (parity with the float64 numpy +// reference is tight); `--float` runs the float32 embedded profile through +// the same I/O so its depth is measured and pinned too. // -// Usage: nn_infer +// Usage: nn_infer [--float] +#include #include +#include +#include #include #include @@ -37,26 +41,33 @@ namespace { } // namespace -int main(int argc, char** argv) { - if (argc != 5) { - std::fprintf(stderr, "usage: %s \n", argv[0]); - return 1; - } - const auto e = read_f64(argv[2]); - const auto yhat = read_f64(argv[3]); - const auto n = std::min(e.size(), yhat.size()); - - tap::mu::nn_suppressor::config cfg; - cfg.weights = tap::mu::load_nn_suppressor_weights(argv[1]); +template +std::vector run(const char* weights, const std::vector& e, const std::vector& yhat) { + const auto n = std::min(e.size(), yhat.size()); + typename tap::mu::nn_suppressor::config cfg; + cfg.weights = tap::mu::load_nn_suppressor_weights(weights); // The parity reference (tools/ml/features.py + nn.py) models the gain // path alone; comfort noise would add tracked-floor fill on top. cfg.comfort_noise = false; - tap::mu::nn_suppressor sup(std::move(cfg)); + tap::mu::nn_suppressor sup(std::move(cfg)); const size_t b = sup.block_size(); - std::vector out(n, 0.0); + std::vector es(e.begin(), e.begin() + static_cast(n)); + std::vector ys(yhat.begin(), yhat.begin() + static_cast(n)); + std::vector os(n, Sample(0)); for (size_t i = 0; i + b <= n; i += b) { - sup.process_block(&e[i], &yhat[i], &out[i]); + sup.process_block(&es[i], &ys[i], &os[i]); } + return std::vector(os.begin(), os.end()); +} + +int main(int argc, char** argv) { + if (argc != 5 && !(argc == 6 && std::strcmp(argv[5], "--float") == 0)) { + std::fprintf(stderr, "usage: %s [--float]\n", argv[0]); + return 1; + } + const auto e = read_f64(argv[2]); + const auto yhat = read_f64(argv[3]); + const auto out = argc == 6 ? run(argv[1], e, yhat) : run(argv[1], e, yhat); std::FILE* f = std::fopen(argv[4], "wb"); if (f == nullptr) { diff --git a/tools/ml/test_parity.py b/tools/ml/test_parity.py index aef51ab..f2575f6 100644 --- a/tools/ml/test_parity.py +++ b/tools/ml/test_parity.py @@ -1,14 +1,18 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: MIT # Copyright 2026 MuTap contributors -"""Parity: tap::mu::nn_suppressor (C++, double) vs the numpy reference. - -Random weights, random signals; the C++ output must match the numpy -pipeline (features.py analysis + nn.py network + features.py synthesis) -to float64 depth over everything after the first frame. Run after any -change to features.py, nn.py or include/mutap/nn_suppressor.h: - - python3 tools/ml/test_parity.py [--build-dir build-ml] +"""Parity: tap::mu::nn_suppressor (C++) vs the numpy reference. + +The C++ output must match the numpy pipeline (features.py analysis + +nn.py network + features.py synthesis) — to float64 depth in the double +profile, and to a measured, pinned depth in the float32 embedded profile. +Weights are either random at a chosen geometry or a trained model +(tools/ml/pretrained/*.munn, whose geometry rides along). CI runs all +four combinations; run after any change to features.py, nn.py or +include/mutap/nn_suppressor.h: + + python3 tools/ml/test_parity.py [--build-dir build-ml] [--profile double|float] + [--weights model.munn | --geometry 16k|48k] """ from __future__ import annotations @@ -27,53 +31,79 @@ import nn # noqa: E402 -def random_weights(rng: np.random.Generator) -> dict[str, np.ndarray]: - return {name: rng.standard_normal(shape).astype(np.float32) * 0.3 - for name, shape in export_weights.ORDER} +def random_weights(rng: np.random.Generator, geometry) -> dict[str, np.ndarray]: + w = {name: rng.standard_normal(shape).astype(np.float32) * 0.3 + for name, shape in export_weights.order(geometry)} + w["geometry"] = np.asarray(geometry, dtype=" int: ap = argparse.ArgumentParser() ap.add_argument("--build-dir", default=str(pathlib.Path(__file__).resolve().parents[2] / "build-ml")) + ap.add_argument("--profile", choices=["double", "float"], default="double") + ap.add_argument("--weights", help="trained .munn to drive both sides with (else random)") + ap.add_argument("--geometry", choices=["16k", "48k"], default="16k", + help="geometry for random weights (ignored with --weights)") args = ap.parse_args() infer = pathlib.Path(args.build_dir) / "tools/ml/nn_infer" if not infer.exists(): raise SystemExit(f"{infer} missing; configure with -DMUTAP_BUILD_ML_TOOLS=ON") rng = np.random.default_rng(3) - n = 64 * 200 + if args.weights: + w = nn.load_munn(args.weights) + label = pathlib.Path(args.weights).name + else: + geometry = features.GEOM16 if args.geometry == "16k" else features.GEOM48 + w = random_weights(rng, geometry.as_array()) + label = f"random {args.geometry}" + geom = features.Geometry.from_array(w["geometry"]) + n = geom.hop * 200 e = rng.standard_normal(n) yhat = rng.standard_normal(n) * 0.5 with tempfile.TemporaryDirectory() as td: td = pathlib.Path(td) - w = random_weights(rng) np.savez(td / "w.npz", **w) export_weights.export(str(td / "w.npz"), str(td / "w.munn")) e.astype(" 1e-6: + tol = TOLERANCE[args.profile] + print(f"[{args.profile}] {label}: max abs err {err:.3e} (rel {rel:.3e}) over {m} samples, bound {tol:.1e}") + if rel > tol: print("PARITY FAIL") return 1 print("parity OK") From e87a921e3e4f61b35927ea0a0738630b02a6c040 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 00:24:08 +0000 Subject: [PATCH 12/14] icount: seed the Hexagon nn_suppressor baselines; silence per-scenario unused helpers The two learned-path scenarios had no hexagon entry, so the first ratchet run reported them as NO BASELINE (nn_suppressor_16k 465537141, nn_suppressor_48k 154160506, from the CI log of that run, per bench/README.md). The same run reproduced every m55 and m33 baseline at +0.00 %. Each icount binary is one scenario, so preset() is unreferenced in the learned-path binaries and nn_weights() in the others; both are now [[maybe_unused]] rather than warning on every cross build. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G --- bench/baselines.json | 2 ++ bench/icount/icount_main.cpp | 6 ++++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/bench/baselines.json b/bench/baselines.json index 26b05ef..0d02f66 100644 --- a/bench/baselines.json +++ b/bench/baselines.json @@ -4,6 +4,8 @@ "chain_48k": 470317154, "fdkf_16k": 92600036, "fdkf_48k": 162071335, + "nn_suppressor_16k": 465537141, + "nn_suppressor_48k": 154160506, "shadow_16k": 57863067, "shadow_48k": 57863050, "suppressor_16k": 255190544, diff --git a/bench/icount/icount_main.cpp b/bench/icount/icount_main.cpp index a965b09..e6af956 100644 --- a/bench/icount/icount_main.cpp +++ b/bench/icount/icount_main.cpp @@ -93,12 +93,14 @@ namespace { const float* yb(std::size_t i) const noexcept { return &y[(i % blocks) * block]; } }; - auto preset() { + // Each binary is one scenario, so the helper the other layers use is + // unreferenced in some of them by design. + [[maybe_unused]] auto preset() { return tap::mu::aec_chain_preset(k_geo.block, k_geo.partitions, k_geo.fs); } // Deterministic weights at the scenario's trained geometry (layer 4). - tap::mu::nn_suppressor_weights nn_weights() { + [[maybe_unused]] tap::mu::nn_suppressor_weights nn_weights() { #if MUTAP_SC_RATE == 0 const tap::mu::nn_geometry g{48000.0, 256, 26, 64, 96}; #else From a8385d6d7c7449fe9076248324590a5d35d32ba8 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 00:24:23 +0000 Subject: [PATCH 13/14] =?UTF-8?q?docs:=20M2=20record=20=E2=80=94=20hexagon?= =?UTF-8?q?=20baselines=20are=20seeded?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G --- HANDOFF.md | 2 +- docs/wake-word-plan.md | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index cc24f39..0f2420c 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -756,6 +756,6 @@ Resolved since rev 1: ~~license~~ (MIT), ~~core language~~ (header-only C++20), Resolved since rev 3: ~~Max external naming~~ — settled (Rev 4): **`mutap.afc~`** (rename from the `mutap.defeed~` placeholder) and **`mutap.aec~`** for the new echo canceller, an acronym pair matching the literature. The rename executes in Stage 2 of "The next effort" above. Still open: -- **Wake-word detection — in progress (M0–M2 done).** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. The named M33 target is the Raspberry Pi Pico 2 W (RP2350: single-precision FPU, so float32 is its profile; the M33 QEMU rig is ported from RatioTap in M2, and M7 adds a `pico2w` CI job that builds the board example against a pinned Pico SDK, uploads the UF2 and asserts its footprint — detection itself is a bench step with a loudspeaker-playback protocol and committed numbers, since QEMU has no RP2350 model), and the docs plan includes a user guide to training a phrase whose commands CI runs on a toy corpus. **M0 is decided (4 September 2026):** repository MuTap, charter widened; host rate fixed at 16 kHz in `kws.h` with conversion as an `@resample` option on the Max external, backed by a new DspTap `decimate.h` (2/3/6) and composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16; release shape runtime-first, no bundled phrase, the training guide as the primary document; development phrase `marvin` from Speech Commands for M5 bring-up, a synthesized four-syllable phrase from M4, never shipped; TTS voices lineage-verified from the Piper model cards — `en_US-libritts-high` (from scratch, CC BY 4.0, 904 speakers), `en_US-kristin-medium`, `en_GB-cori-high` (from scratch, public domain), `en_US-john-medium` (from Kristin), with every Lessac-derived voice (most of the English set, `libritts_r` and `vctk` included) and the sample generator's bundled `.pt` generator excluded; training on an Apple Silicon Mac via `--device mps`, under an hour per run on the development set. **M1 is done** on DspTap's `claude/mutap-wake-word-plan-2i63pe`: `log_mel.h` and `decimate.h` with their typed batteries, the numpy reference generator (`tools/reference/make_frontend_reference.py` — the family's single numpy copy of the formulas), C ABI and bridge; every tolerance in the plan's M1 record is a measured number. **M2 is done** on MuTap's `claude/mutap-wake-word-plan-2i63pe`: the learned suppressor has its oracles — typed float/double tests with a −120 dB cross-precision pin, a float32 chain gate, a Python↔C++ parity CI job in both profiles on random and exported weights, a Cortex-M33 QEMU leg (mps2-an505, Ooura float32 FFT) with the float suppressor suite on-target on every leg, and `nn_suppressor` icount scenarios with m55/m33 baselines (hexagon's two are seeded from the first CI log). The oracles found and fixed a Nyquist-bin contract defect at 48 kHz (C++ notched bin N/2, numpy did not; fixed on both sides). Next: M3, promoting the dense/GRU kernels into DspTap's `tap::dsp::nn`. +- **Wake-word detection — in progress (M0–M2 done).** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. The named M33 target is the Raspberry Pi Pico 2 W (RP2350: single-precision FPU, so float32 is its profile; the M33 QEMU rig is ported from RatioTap in M2, and M7 adds a `pico2w` CI job that builds the board example against a pinned Pico SDK, uploads the UF2 and asserts its footprint — detection itself is a bench step with a loudspeaker-playback protocol and committed numbers, since QEMU has no RP2350 model), and the docs plan includes a user guide to training a phrase whose commands CI runs on a toy corpus. **M0 is decided (4 September 2026):** repository MuTap, charter widened; host rate fixed at 16 kHz in `kws.h` with conversion as an `@resample` option on the Max external, backed by a new DspTap `decimate.h` (2/3/6) and composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16; release shape runtime-first, no bundled phrase, the training guide as the primary document; development phrase `marvin` from Speech Commands for M5 bring-up, a synthesized four-syllable phrase from M4, never shipped; TTS voices lineage-verified from the Piper model cards — `en_US-libritts-high` (from scratch, CC BY 4.0, 904 speakers), `en_US-kristin-medium`, `en_GB-cori-high` (from scratch, public domain), `en_US-john-medium` (from Kristin), with every Lessac-derived voice (most of the English set, `libritts_r` and `vctk` included) and the sample generator's bundled `.pt` generator excluded; training on an Apple Silicon Mac via `--device mps`, under an hour per run on the development set. **M1 is done** on DspTap's `claude/mutap-wake-word-plan-2i63pe`: `log_mel.h` and `decimate.h` with their typed batteries, the numpy reference generator (`tools/reference/make_frontend_reference.py` — the family's single numpy copy of the formulas), C ABI and bridge; every tolerance in the plan's M1 record is a measured number. **M2 is done** on MuTap's `claude/mutap-wake-word-plan-2i63pe`: the learned suppressor has its oracles — typed float/double tests with a −120 dB cross-precision pin, a float32 chain gate, a Python↔C++ parity CI job in both profiles on random and exported weights, a Cortex-M33 QEMU leg (mps2-an505, Ooura float32 FFT) with the float suppressor suite on-target on every leg, and `nn_suppressor` icount scenarios with baselines on m55, m33 and hexagon. The oracles found and fixed a Nyquist-bin contract defect at 48 kHz (C++ notched bin N/2, numpy did not; fixed on both sides). Next: M3, promoting the dense/GRU kernels into DspTap's `tap::dsp::nn`. - **Default engine in the external** — `@kalman` off (classic NLMS) is the shipping default purely on seniority; the measured case for flipping it is in `tests/test_fd_kalman.cpp` and book chapter 1. Decide after real-room listening. - **RIR fixtures, the measured half** — the fixture pipeline is built and three physically-modeled rooms (image-source, documented geometry) are committed baselines with regression tests. What remains yours: which MEASURED rooms join them — an academic dataset room (MYRiAD is the PEM-AFROW group's own database; openAIR is the other usual source; check each room's license allows redistribution in an MIT repo) and/or your own swept-sine measurements. Either way it is one command per room: `python3 tools/fixtures/make_rir_fixtures.py --from-wav room.wav myroom --source ""`, then a test with a freshly measured threshold. (The dataset hosts are unreachable from the remote dev container's network policy, so the WAVs have to enter via a commit.) diff --git a/docs/wake-word-plan.md b/docs/wake-word-plan.md index e750c23..dc634a7 100644 --- a/docs/wake-word-plan.md +++ b/docs/wake-word-plan.md @@ -392,10 +392,10 @@ pinned since there is no MVE) and the on-target filter on every leg carries the float `nn_suppressor` suite, the cross-precision pin and the chain test. Layer 4 of `bench/icount` is the suppressor at both trained geometries with xorshift weights; baselines seeded locally on m55 and m33 (the local ratchet reproduces -every committed m55 baseline to 0.00 %, so local seeding is trustworthy) and -**not yet on hexagon** — the first CI run of the icount job reports the two -missing counts as `NO BASELINE`, and they are committed from that log per the -seeding procedure in `bench/README.md`. Per-hop cost of the *existing* GRU +every committed m55 baseline to 0.00 %, so local seeding is trustworthy; CI +then reproduced both the m55 and the m33 sets at +0.00 %) and on hexagon from +the first CI run's `NO BASELINE` report, per the seeding procedure in +`bench/README.md`. Per-hop cost of the *existing* GRU suppressor, whole-binary count divided by hops processed (setup included, so an upper bound): m55 ≈ 383 k instructions/hop at 48 kHz (hop 256) and ≈ 271 k at 16 kHz (hop 64); m33 ≈ 711 k and ≈ 491 k. Against the wake-word ceiling of From 037f38313c2e1214503f0e0c991bcb364890ba91 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 00:48:59 +0000 Subject: [PATCH 14/14] M33 leg: drop the long float PEM scenarios from the on-target selection MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first full run of the Cortex-M33 selection timed out at ctest's 1800 s: kalman_loop_test/0.PemAddsStableGainOnTonal alone took 1030 s under qemu mps2-an505 (84 s on mps3-an547). The cost is not the canceller — the closed-loop and echo harnesses simulate the room in double on purpose (tests/support/closed_loop.h convolves the feedback path in double per sample, the MSG bisection measures in double) and the speech predictor's pitch search accumulates in double — which is hardware on the M55 and software on the M33. The M33 toolchain now sets MUTAP_ON_TARGET_SOFT_FP64; tests/CMakeLists.txt forwards it as a compile definition and bare_metal_main.cpp drops the four long float PEM scenarios (kalman_loop_test/0, pem_afc_test/0, burst_test/0, aec_test/0) under it. Those stay on the M55 and Hexagon legs and every host. The M55 selection is unchanged. Measured: 47 tests, 69 s, all nine learned-path tests included; the wake-word plan's M2 record carries the finding as the first concrete cost figure for double on the RP2350 class. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01JuUg1ZBxm3fyBqWgQv6H1G --- .github/workflows/ci.yml | 4 +++- cmake/arm-cortex-m33-mps2.cmake | 4 ++++ docs/wake-word-plan.md | 13 ++++++++++++- tests/CMakeLists.txt | 5 +++++ tests/bare_metal_main.cpp | 21 ++++++++++++++++++--- 5 files changed, 42 insertions(+), 5 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 727c4ca..35ceeb0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -184,7 +184,9 @@ jobs: # Cortex-M33 (Raspberry Pi Pico 2 W / RP2350 class: single-precision FPU, # no FP64, no MVE) on QEMU's MPS2+ AN505 model — the wake-word plan's named # embedded target. Same Armv8-M startup as the M55 leg, Ooura float32 FFT - # (no Helium here), same emulation-sized float-profile selection. Ported + # (no Helium here), the M55's float-profile selection minus the long float + # PEM scenarios (their test harness simulates the room in double: soft-float + # here, ~17 min for one of them — see tests/bare_metal_main.cpp). Ported # from RatioTap's cortex-m33-qemu job. cortex-m33-qemu: name: Cortex-M33 cross (QEMU) diff --git a/cmake/arm-cortex-m33-mps2.cmake b/cmake/arm-cortex-m33-mps2.cmake index bd806f8..eac3120 100644 --- a/cmake/arm-cortex-m33-mps2.cmake +++ b/cmake/arm-cortex-m33-mps2.cmake @@ -41,3 +41,7 @@ set(TAP_DSP_FFT_CMSIS OFF CACHE BOOL "No MVE on the Cortex-M33: Ooura float32 FF # One-shot CTest mode (no argv on bare metal; see tests/CMakeLists.txt). set(MUTAP_BARE_METAL ON) +# Single-precision FPU only: the on-target selection drops the long float +# PEM scenarios, whose test harness simulates the room in double (soft-float +# here; ~17 min for one of them under QEMU). See tests/bare_metal_main.cpp. +set(MUTAP_ON_TARGET_SOFT_FP64 ON) diff --git a/docs/wake-word-plan.md b/docs/wake-word-plan.md index dc634a7..0f785f1 100644 --- a/docs/wake-word-plan.md +++ b/docs/wake-word-plan.md @@ -389,7 +389,18 @@ nothing in the float profile touches double. The M33 leg is ported from RatioTap (`cmake/arm-cortex-m33-mps2.cmake`, `platform/mps2_an505.ld`, the shared `armv8m_startup.c`, a `cortex-m33-qemu` CI job with the Ooura float32 FFT pinned since there is no MVE) and the on-target filter on every leg carries the -float `nn_suppressor` suite, the cross-precision pin and the chain test. Layer 4 +float `nn_suppressor` suite, the cross-precision pin and the chain test. One +honest limit of the M33 leg, found by running it: the long float PEM +scenarios are driven by a test harness that simulates the room in double on +purpose (the closed-loop convolution and the MSG bisection), and the speech +predictor's pitch search accumulates in double — hardware on the M55, +software on the M33 — so the tonal PEM headline alone took 1030 s under qemu +mps2-an505 against 84 s on mps3-an547. The M33 selection +(`MUTAP_ON_TARGET_SOFT_FP64`) drops those four scenarios, which the M55 and +Hexagon legs and every host still run. That is also the first concrete cost +figure for "double on the RP2350" in this plan, and the reason §5's rule that +nothing on the wake-word path touches double is a budget rule, not a style +rule. Layer 4 of `bench/icount` is the suppressor at both trained geometries with xorshift weights; baselines seeded locally on m55 and m33 (the local ratchet reproduces every committed m55 baseline to 0.00 %, so local seeding is trustworthy; CI diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index af2d493..26cb911 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -56,6 +56,11 @@ if(MUTAP_BARE_METAL) # not reliably propagate exit codes through the emulator. target_sources(mutap_tests PRIVATE bare_metal_main.cpp) target_link_libraries(mutap_tests PRIVATE GTest::gtest) + if(MUTAP_ON_TARGET_SOFT_FP64) + # No FP64 hardware (Cortex-M33): bare_metal_main.cpp drops the long + # float PEM scenarios, whose test harness simulates the room in double. + target_compile_definitions(mutap_tests PRIVATE MUTAP_ON_TARGET_SOFT_FP64=1) + endif() add_test(NAME mutap_tests_emulated COMMAND mutap_tests) set_tests_properties(mutap_tests_emulated PROPERTIES PASS_REGULAR_EXPRESSION "MUTAP_TESTS_COMPLETE rc=0" diff --git a/tests/bare_metal_main.cpp b/tests/bare_metal_main.cpp index d4caf60..c81367f 100644 --- a/tests/bare_metal_main.cpp +++ b/tests/bare_metal_main.cpp @@ -12,6 +12,17 @@ // scenarios — minutes-to-hours of soft-float virtual audio validating // target-independent math already covered on every host platform — and the // bisection-heavy ASG measurements beyond the float ones kept. +// +// MUTAP_ON_TARGET_SOFT_FP64 (the Cortex-M33 leg: single-precision FPU, no +// FP64) additionally drops the long float PEM scenarios. Their cost is not +// the canceller: the closed-loop and echo harnesses simulate the room in +// double on purpose (tests/support/closed_loop.h convolves the feedback +// path in double per sample; the MSG bisection measures in double), and the +// speech predictor's pitch search accumulates in double (lpc.h). That is +// hardware on the M55 and software on the M33: the tonal PEM headline alone +// measured 1030 s under qemu mps2-an505 against 84 s on mps3-an547. Those +// scenarios stay covered on the M55 and Hexagon legs and every host; the +// M33 leg exists for the pure-float32 paths and the learned suppressor. // SPDX-License-Identifier: MIT // Copyright 2026 MuTap contributors #include @@ -24,11 +35,15 @@ int main() { "real_fft_test/0.*:real_fft_test/1.*:RealFftCrossPrecision.*:" "CertifiedGeometries/fft_backend_parity.*:" "fdaf_test/0.*:FdafCrossPrecision.*:FdafConfigValidation.*:FdafRtContract.*:" - "fd_kalman_test/0.*:fd_kalman_test/1.*:kalman_loop_test/0.*:FdKalmanConfigValidation.*:FdKalmanRtContract.*:" + "fd_kalman_test/0.*:fd_kalman_test/1.*:FdKalmanConfigValidation.*:FdKalmanRtContract.*:" "Levinson.*:LpcPredictor.*:SpeechPredictor.*:WarpedLpcPredictor.*:PredictorConfigValidation.*:" - "pem_afc_test/0.*:PemAfcConfigValidation.*:PemAfcRtContract.*:" - "closed_loop_test/0.*:burst_test/0.*:aec_test/0.*:" + "PemAfcConfigValidation.*:PemAfcRtContract.*:" + "closed_loop_test/0.*:" "AdaptationControlConfigValidation.*:" +#ifndef MUTAP_ON_TARGET_SOFT_FP64 + // The long float PEM scenarios (double harness), see the header comment. + "kalman_loop_test/0.*:pem_afc_test/0.*:burst_test/0.*:aec_test/0.*:" +#endif "nn_suppressor_test/0.*:NnSuppressorCrossPrecision.*:NnChainFloat32.*"; ::testing::InitGoogleTest(); const int rc = RUN_ALL_TESTS();