歌声（singer）の土台：計画書・データ準備・TS モジュール by snakajima · Pull Request #41 · receptron/jinglescript

snakajima · 2026-10-25T22:13:17Z

概要

jinglescript に日本語の歌声（singer）を追加するための土台。コードは MIT、モデルの重みは MIT / CC0 / CC-BY のみ。

現状

S1：lyric 付きの楽譜 → ONNX 音響モデル → DSP ボコーダー → WAV が CPU で一周（仮モデルは自作の合成データで学習した CC0。音程は正確、声はまだロボット的）
renderWithSinger() で歌声トラックが render() とマスターを通る（pan・gain・強弱も反映）
後処理チェーン（--post、既定オフ）：ライド → 低域カット＋F0 倍音を避けた共鳴ノッチ＋エア → アップワード・コンプ → M/S 幅 → リミッター
学習：training/singing/train_torch.py（CUDA / MPS / CPU 自動選択、ONNX と sha256 入り manifest まで書き出し）。MacBook Pro（MPS）がメイン、重い学習は DGX Spark
Common Voice：fetch_common_voice.py は公式 SDK（datacollective）を使用、CC0 以外は停止、--check でライセンスだけ確認。Common Voice Scripted Speech 27.0 Japanese（CC0-1.0、約15.4GB）を確認済み。ダウンロードは API キーと規約同意アカウントの不一致で未取得
onnxruntime-node は任意の peer 依存（歌わない人には入らない）。ONNX ファイルは sha256 で照合

次

S2：Common Voice で事前学習、アライナー、ITA コーパスの自前録音

…fold

socket-security · 2026-10-25T22:13:54Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/datacollective@0.7.0
	pypi/numpy@2.5.4
	pypi/tqdm@4.70.1
	pypi/torch@2.14.1
	pypi/onnxruntime@1.31.0
	pypi/onnx@1.23.2
	npm/onnxruntime-node@1.30.0
	pypi/requests@2.34.2
	pypi/librosa@1.0.0
	pypi/soundfile@0.14.0
	pypi/pyloudnorm@0.2.0

View full report

- training/singing/tiny_placeholder.py: フォルマント合成の自作コーパス(CC0)で小型MLPを学習しONNX出力 - src/singer/vocoder.ts: 帯域包絡を読む決定的な倍音+ノイズDSPボコーダー - src/singer/sing.ts: renderWithSinger()（オプトイン）、render() に prerendered オプション - onnxruntime-node を optionalDependencies に追加 - scripts/sing-demo.ts、tests/singer-sing.test.ts、計画書の S1 を更新

…README に API キー取得手順）

snakajima

Review notes. All 10 CI checks pass. Not verified: the branch was not run locally, the Python scripts were not reviewed in depth, and I did not check the vocoder's noise scaling against the analysis in tiny_placeholder.py.

The PR description is out of date. It says the singer is not connected to render() and has 8 tests, but d42c395 and 56e0c76 add RenderOptions.prerendered, renderWithSinger/singTrack, the post chain, the PyTorch training path and three test files.

Inline comments follow. The first two are bugs; the third is a packaging decision.

What checks out: the RBJ peaking filter, the Chebyshev sine recurrence and the overlap-add normalisation are correct; randomness comes only from streamRng and math goes through dmath; the plan doc keeps ONNX output out of the golden tests; no weights or audio are added, and training/ is not in package.json files.

snakajima · 2026-10-26T00:16:29Z

    });
    mixEvent(voices, event, score.tracks[event.track]?.reverb === false ? dry : wet);
  });
+  for (const [track, pcm] of options.prerendered ?? []) {


Bug: a sung track ignores its volume, dynamics and pan. Normal events go through mixEvent, which applies event.gainDb and panGains(event.pan). Prerendered PCM is added to both channels at SQRT1_2, so a singer track's pan, volume and per-note dynamics do nothing. One fix: apply the track's pan and gain here, or bake per-note gain into the PCM in singTrack.

999dc4f で修正しました。歌声トラックにも mixEvent と同じく、ノートごとに gain・pan・ベロシティをかけています（境界は5ms補間）。

snakajima · 2026-10-26T00:16:29Z

+
+/** Imports onnxruntime-node once, on first use. `load` replaces the import (tests). */
+export function loadOnnxRuntime(load: () => Promise<unknown> = () => import(ONNX_MODULE)): Promise<OnnxRuntime> {
+  runtime ??= load().then(


Bug: the loader is ignored after the first call. runtime ??= load() keeps the first result for the whole process, so a later call's load (SingOptions.loadRuntime) is ignored. In tests/singer-sing.test.ts:134, the post-chain test passes fakeRuntime({}) but gets the runtime cached by the earlier test; it passes only because of test order. Cache only the default import, or key the cache on load.

999dc4f で修正しました。キャッシュするのは既定の import だけで、渡された loader は毎回使います。テストは実行順に依存しなくなりました。

snakajima · 2026-10-26T00:16:29Z

    "vitest": "^5.0.3"
+  },
+  "optionalDependencies": {
+    "onnxruntime-node": "^1.30.0"


optionalDependencies are installed by default. npm and yarn install them unless the install fails, so every npm install jinglescript would download onnxruntime-node. That conflicts with CLAUDE.md's "no runtime dependencies beyond zod and the MCP SDK". The code already loads it by name and throws SingerRuntimeMissingError with install instructions, so an optional peer dependency (peerDependencies + peerDependenciesMeta: { optional: true }) or not listing it at all fits better.

999dc4f で任意の peerDependencies（peerDependenciesMeta optional）へ移しました。

snakajima · 2026-10-26T00:16:29Z

+  envelope: z.object({ kind: z.literal("band-log-power"), fftSize: z.number().int().positive(), maxHz: z.number().positive() }).optional(),
+  /** No `vocoder` file: the built-in harmonic + noise vocoder (src/singer/vocoder.ts) reads `envelope` frames. */
+  files: z.object({ acoustic: z.string(), vocoder: z.string().optional(), pitch: z.string().optional() }),
+  sha256: z.record(z.string(), z.string().regex(/^[0-9a-f]{64}$/)),


sha256 is required and its format is validated, but nothing hashes the ONNX files against it. Check it in readSingerManifest or before InferenceSession.create, or drop the field until the download step exists.

999dc4f で、セッション作成前に sha256 を照合するようにしました（不一致なら SingerModelCorruptError）。

snakajima · 2026-10-26T00:16:29Z

+  melBins: z.number().int().positive(),
+  phonemes: z.array(z.string()).min(1),
+  /** How the acoustic model's frames are to be read when there is no neural vocoder (built-in DSP vocoder). */
+  envelope: z.object({ kind: z.literal("band-log-power"), fftSize: z.number().int().positive(), maxHz: z.number().positive() }).optional(),


src/dsp/fft.ts only works on powers of two, but fftSize is only checked as a positive integer. A bad manifest would give wrong output instead of a clear error. Add a zod refinement (power of two, and probably >= hopSize).

999dc4f で、2のべき乗かつ hopSize 以上という zod の検査を追加しました。

snakajima · 2026-10-26T00:16:29Z

+/** 2. EQ, in place. */
+function equalize(x: Float32Array, sampleRate: number, options: Exclude<VocalChainOptions["eq"], false> = {}) {
+  const filters: Biquad[] = [highpass(options.lowCutHz ?? 90, Math.SQRT1_2, sampleRate)];
+  const resonance = findResonance(x, sampleRate);


Design note: findResonance picks the narrowest peak more than 6 dB above its neighbours in the average spectrum. On a held vowel, that will often be a harmonic of the most-sung pitch rather than a resonance, which then gets cut by up to 6 dB at Q 8. Consider excluding bins near multiples of the median F0, or requiring the peak to persist across pitches.

999dc4f で、中央値 F0 の倍音付近を除外するようにしました。

snakajima · 2026-10-26T00:16:30Z

+
+// Small kana that make a palatalized or foreign mora with the one before.
+const SMALL: Readonly<Record<string, string>> = { ゃ: "a", ゅ: "u", ょ: "o", ぁ: "a", ぃ: "i", ぅ: "u", ぇ: "e", ぉ: "o" };
+const PALATAL: Readonly<Record<string, string>> = {


Some kana combinations used in loanword lyrics are rejected or wrong: とぅ/トゥ, どぅ, つぁ, すぃ raise SingerLyricError, and でゅ comes out as d u instead of dy u.

999dc4f で対応しました。でゅは dy u になります。

- 事前レンダリングした歌声トラックにも pan・gain・ノートの強弱を mixEvent と同じ意味で適用（ノート境界は5msで補間） - loadOnnxRuntime は既定の import だけをキャッシュし、渡された loader は毎回使う - onnxruntime-node を optionalDependencies から任意の peerDependencies へ移動（開発用に devDependencies にも追加） - セッション作成前に ONNX ファイルの sha256 を照合し、不一致なら SingerModelCorruptError - manifest の fftSize は2のべき乗かつ hopSize 以上、全ファイルに sha256 必須 - findResonance は中央値 F0 の倍音付近を除外（伸ばした母音の倍音を削らない） - とぅ/トゥ・どぅ・つぁ・すぃなどに対応し、でゅを dy u に修正 - 各修正のテストを tests/singer-review.test.ts に追加

snakajima

Re-review of 999dc4f. All seven earlier comments are fixed, and the fixes have tests: per-note gain/pan/vel for prerendered tracks, the loader cache, the optional peer dependency, sha256 checking, the fftSize checks, the F0-aware notch and the loanword kana. All 10 CI checks pass.

Inline comments follow: one new bug in the harmonic filter, one silent-failure case and two minor notes. Not verified: the branch was not run locally, and the Python scripts were not read in depth beyond the new Common Voice code, which refuses non-CC0 datasets and saves under ~/.cache.

snakajima · 2026-10-26T02:32:52Z

+/** Whether a frequency is within 4% (and at least two FFT bins) of a harmonic of f0. */
+function nearHarmonic(hz: number, f0: number, binHz: number): boolean {
+  const n = Math.max(1, Math.round(hz / f0));
+  return Math.abs(hz - n * f0) <= Math.max(2 * binHz, 0.04 * n * f0);


Bug: above about 12×F0 every frequency is skipped. Every frequency is at most f0/2 from its nearest harmonic, so once 0.04·n·f0 ≥ f0/2 (n ≥ 13) everything counts as near a harmonic. Bins left open at 44.1 kHz / 4096:

F0 = 330 Hz: nothing above 4124 Hz.

F0 = 200 Hz: nothing above 2498 Hz; above 2 kHz only 2089, 2099, 2110, 2293 and 2498 Hz.

The test "still finds a real resonance between harmonics" (2530 Hz, F0 200) passes only because the 2498 Hz bin falls in a ~4 Hz gap between the n=12 and n=13 windows. Capping the tolerance fixes it, e.g. Math.max(2 * binHz, Math.min(0.04 * n * f0, f0 / 4)). A test with a resonance well above 12×F0 would cover it.

d888126 で修正しました。許容幅に f0/4 の上限を付け、12×F0 より上の共鳴（5150Hz@F0 330、4630Hz@F0 200）を検出するテストを追加しました。

snakajima · 2026-10-26T02:32:53Z

+    const index = phonemes.indexOf(span.phoneme);
+    for (let f = span.start; f < span.end && f < frames.f0.length; f++) {
+      const row = f * width;
+      if (index >= 0) out[row + index] = 1;


A phoneme missing from manifest.phonemes is silently dropped. Its frames get an all-zero one-hot and no error. The new ty/dy are not in tiny_placeholder.py's PHONEMES, so てゅ/でゅ would render as a blank phoneme. Either throw a clear error naming the phoneme, the note and the model, or fall back (ty→t, dy→d) before building features.

d888126 で修正しました。モデルにない音素は ty→t、dy→d に置き換え、それでも無ければ音素・ノート・モデル名を示す SingerPhonemeError を出します。tiny_placeholder.py の PHONEMES にも ty/dy を追加しました。

snakajima · 2026-10-26T02:32:53Z

+  const sorted = [...events].sort((a, b) => a.seconds - b.seconds);
+  const gainsOf = (event: NoteEvent | undefined): [number, number] => {
+    if (event === undefined) return [Math.SQRT1_2, Math.SQRT1_2];
+    const gain = dmath.dbToGain(event.gainDb) * (event.vel / PRERENDERED_NORMAL_VEL);


Minor: instruments scale linearly with velocity, with full velocity (1) as the reference, so a default note at 0.8 plays at 0.8×. Here a sung note at 0.8 plays at 1× and at 1.0 plays at 1.25×. That's fine if the singer is meant to be levelled at the default velocity; a comment saying so would help.

d888126 で、歌声は既定ベロシティ 0.8 で等倍になるよう意図的に揃えている旨をコメントに書きました。

snakajima · 2026-10-26T02:32:53Z

+  if (expected === undefined) throw new SingerModelMissingError(manifest.name, folder, `manifest.json has no sha256 for ${file}`);
+  let bytes: Buffer;
+  try {
+    bytes = await readFile(path);


Minor: this reads and hashes the whole model file on every singTrack. For a model of a few hundred MB that costs time on each render. Caching the verified result by path + size + mtime for the process would avoid it.

d888126 で、sha256 の照合結果をパス・サイズ・mtime ごとにプロセス内でキャッシュするようにしました。

- findResonance の倍音許容幅に f0/4 の上限を付け、12×F0 より上の共鳴も検出できるように - モデルにない音素は ty→t、dy→d に置き換え、それでも無ければ音素・ノート・モデル名を示す SingerPhonemeError - tiny_placeholder.py の PHONEMES に ty / dy を追加 - 歌声トラックのベロシティ基準（0.8）が楽器と異なる理由をコメントで説明 - ONNX の sha256 照合結果をパス・サイズ・mtime ごとにプロセス内でキャッシュ - 各修正のテストを追加

snakajima added 3 commits October 25, 2026 11:28

Singing voice groundwork: plan, data prep scripts, singer module scaf…

6ac08c9

…fold

Singing voice plan: post-processing chain for the singer output

af832a0

docs(singer): AI生成音は既存方針の範囲内である旨に修正

e8eecbd

snakajima added 4 commits October 25, 2026 13:39

歌声：PyTorch 学習経路（MPS/CUDA 自動選択・ONNX 書き出し）とボーカル後処理チェーンを追加

56e0c76

歌声：Common Voice 取得を Mozilla Data Collective 公式 SDK に移行（CC0 確認・モックテスト・…

a5855e0

…README に API キー取得手順）

歌声：Common Voice 取得に --check を追加、保存先を ~/.cache に変更、規約未同意（403）の案内

bb10b5b

snakajima commented Oct 26, 2026

View reviewed changes

snakajima marked this pull request as ready for review October 26, 2026 03:46

snakajima merged commit 5cf277e into main Oct 26, 2026
10 checks passed

Conversation

snakajima commented Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

概要

現状

次

Uh oh!

socket-security Bot commented Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

snakajima left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snakajima left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

snakajima commented Oct 25, 2026 •

edited

Loading

socket-security Bot commented Oct 25, 2026 •

edited

Loading