feat(studio): waveform strip, clip light, peak marks and video beat source (5/8)

[vanceingalls]

Summary

Seeing sound on the timeline: a waveform strip under video clips that carry sound, a latching CLIP light on the master meter at the export ceiling, peak marks where a clip redlines, and a video with sound as the beat source.

Changes

• studio-server: peak maps (peakMap) plus a /peaks route.
• studio: AudibleVideoClipContent waveform strip, ClipPeakMarks / clipPeakRuns, and a CLIP light on AudioMeterStrip (theme tokens). Each peak map is decoded once, and Duck is never offered on a carve source.
• core: the beat file accepts a video with sound as its source.

Testing

Static gates on the layer tip: bun install, bun run build, tsc --noEmit for every touched package (0 errors), oxlint and oxfmt --check on files changed vs main (clean), gen:skills-manifest --check (in sync), scripts/comment-ratchet.mjs (ok).

• Scoped vitest: cli 3957, core 3793, studio 6288, studio-server 996. All pass.

Notes

• The strip draws only at the root composition. Peaks can go stale after an in-place re-encode.
• Part of the audio-on-video stack. See the bottom PR for the overview.

🤖 Generated with Claude Code

Before

After

Timeline: strip under sound-bearing videos. At this layer the strip draws flat (1px canvas); the next layer fixes the lane height.

[miguel-heygen]

Reviewed incremental #4817 at 65b1bf6d3846389aa251407268c23e3fa65c36f7 against a244373ae644e6cf8085f7a7804cfc4a91eb08ef.

The peak accumulator preserves absolute native-channel levels and carries split frames across chunks (packages/studio-server/src/helpers/peakMap.ts:41). The CLIP light reads the program meter upstream of monitor gain, matching export intent (packages/studio/src/components/nle/AudioMeterStrip.tsx:229, existing packages/core/src/runtime/webAudioTransport.ts:147).

Blocker: packages/studio-server/src/helpers/peakMap.ts:90 — The channel probe explicitly selects a:0 (:71), but decoding leaves stream selection to FFmpeg. With a mono first stream and a stereo second stream, FFmpeg can select the second while the accumulator still treats every float as a mono frame. A real generated one-second fixture with no default dispositions returned 40 bins at binSeconds=0.05 (advertising two seconds), with peaks 0.9 from stream two instead of 0.1 from stream one. Consequently peak markers have the wrong level and time positions. Select the same stream in both commands, and add a multistream regression test.

Audited: complete peak decoder/cache/route, peak-run/marks components, audible-video strip and render dispatcher, meter latch integration and cleanup, CLI/Studio beat-source predicates, carve-source exclusion; traced waveform requests and preview program-meter placement.
Trusting: parent audio-runtime implementation and #4816 fixes; full Studio route/layout and packaged renderer; unmodified thumbnail scheduling internals. No prior reviews or comments on this PR.

Verification: 46 existing targeted tests passed at the exact head, including real generated single-track video decode tests; 1 independent real-FFmpeg multistream witness passed. Cached Vitest 3.2.4 differs from declared 4.1.11, with worktree source aliases. Initial two broader UI suites could not collect with the cached dependencies; 50 shared UI tests passed at #4818 (unchanged meter/render dispatcher source); these are supplemental results, not exact-head runs. All observed exact-head CI checks pass. The later browser fixture exercised source components only, not full Studio/backend E2E.

— Magi

Verdict: REQUEST CHANGES
Reasoning: Peak maps violate their per-bin timing/level contract on valid multistream media despite the single-stream tests and green CI.

[github-actions]

Edit accuracy: 953 passing here, 953 on the base branch

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

[miguel-heygen]

Fix-delta review at a12e2f91b6cb88300ab4569a5bc25637438ba3af, against stacked parent dbcbca85d2e088cd05cb28320015aa3109aec27d; compared with my previous review at 65b1bf6d.

The stream-selection blocker is repaired: the channel probe selects a:0, decoding explicitly maps 0:a:0 (packages/studio-server/src/helpers/peakMap.ts:69, :96), and the peaks-v2 cache invalidates previously incorrect maps (:8). The regression constructs a mono-first, louder-stereo-second fixture with no default dispositions and checks both length and 0.1 peak level.

Verification: all 6 peak-map tests pass with real FFmpeg on stack head 88255f82e; helper/test files are byte-identical to this PR head. Cached Vitest 3.2.4 and source aliases. No waveform browser, live Studio/backend, playback or export acceptance this round. No new code blocker in the two-file repair delta.

The required Studio and player captures check remains red for missing Before/After evidence. Earlier changes-requested state remains; this is not an approval.

— Magi

Verdict: COMMENT
Reasoning: The original multistream defect is fixed and regression-covered, but the required capture gate is unresolved.

[jerrai-bot-heygen]

At b9a59e5f, the first-audio-stream probe/decode repair remains, but the new peak-map decoder treats partial output followed by FFmpeg failure as success and caches an incomplete map. Inline has the concrete failure and regression request. Current-head checks pass except required Studio/player Before/After captures, which are red; no visual waiver was supplied. No local tests, build, merge or release. — Jerrai

[jerrai-bot-heygen]

A decode that emits some samples and then exits nonzero resolves here as a complete peak map. The /peaks route caches that array (routes/peaks.ts:53–64), so a truncated/corrupt audio source can permanently show an incomplete clipping map instead of the existing decode-error response. Please reject on a nonzero close code even when acc.frame > 0, and add a partial-output-then-failure regression. The first-stream probe/map fix itself is present.

[jerrai-bot-heygen]

The 24-file layer patch is preserved at the exact #4816-based head. Peak decoding selects the same first audio stream as probing and rejects a nonzero FFmpeg close before returning/caching any partial bins; peakMap.decodeFailure.test.ts runs a fake FFmpeg that emits samples then exits 1 and asserts rejection. This closes my previous code hold; the capture check passes. Miguel's prior review and remaining CI are independent merge gates. — Jerrai

[jerrai-bot-heygen]

Exact a0d266da rebase review: all 24 layer paths and patch hunks match the previously approved parent→head diff, with no concurrent parent edit on those paths. At this head FFmpeg peak decoding still rejects a nonzero exit after partial sample output before returning/caching peaks; the regression emits samples and exits 1. Code approval for this layer only; #4815 viewport CI and Miguel's older review remain independent stack gates, and the new head's own checks are still running. No merge/release action. — Jerrai

[miguel-heygen]

Re-reviewed #4817 at a0d266da27da8ed2a114ce5d51e83470502610d4 against #4816 4f68e1720; this supersedes my changes-requested review 5374292610, and confirms Jerrai's decoder repair.

Probe and decode both select the first audio stream (a:0 / -map 0:a:0), with the versioned peak cache. decodePeakMap now rejects every nonzero close before returning bins, including partial output, so the route cannot cache that result as a success. I read the helper/cache route and ran all seven peak tests, including real FFmpeg fixtures and the partial-output/exit-1 binary. Those helper and test files are byte-identical between this layer and the tested #4818 export.

Current capture and other fetched checks pass. Cached Vitest 3.2.4/source aliases; supplemental combined-tip execution, not a full isolated package build. No live Studio, waveform browser, playback or render acceptance was performed.

Verdict: APPROVE
Reasoning: the original stream mismatch and the later incomplete-decode success path are repaired and directly regression-covered. No new blocker found in the reviewed layer.

— Magi

[miguel-heygen]

Re-approved at 89fb3b8c16d43352dd00afe060d5b39cb26bb743 against stacked parent eaaf6e1b6d9e243490b0f6dc1d4e5fe366462794, following my review 5387236487.

The parent-to-head patch has identical paths and hunks to the layer I previously approved, ignoring only object indexes and hunk locations. The restack preserves the repaired matching FFmpeg audio-stream selection, cache version and nonzero-exit rejection. This is a rebase verification; I did not repeat the earlier exact-source tests or the full package suite.

Current checks have no failure, but CI remains in progress; approval does not establish merge readiness. Prior evidence limits (cached tooling, no live Studio/audio acceptance) remain. No merge, release or protection bypass performed.

Verdict: APPROVE
Reasoning: The inspected layer is the same reviewed patch on its new stacked parent, with no additional code change or new layer-specific blocker.

— Magi

[miguel-heygen]

Re-approved at 044e9696493e9b822d021a0fb38049f4a130b513 against stacked parent 9849ba697393839b2134f5a82b79a694163bcb5c, following my approval 5387641276.

I independently compared the complete old/new parent-to-head patches: same 24-path set and identical hunks, ignoring only object indexes and hunk positions. Every changed-layer source and test blob is also identical between the approved head and this head; there are no differing layer file blobs. The prior repairs, test cases and save/decode behavior, survive unchanged.

This is a rebase verification, not a new local test run. The prior focused tests and their stated cached-tooling limits remain the evidence. No live Studio/playback, full package acceptance or merge/release performed. No task worktree, server or browser was created.

Current-head checks have no failure but are still in progress. Approval does not waive any required check or establish merge readiness.

Verdict: APPROVE
Reasoning: The entire inspected layer patch and its executable sources/tests match the previously approved layer on the new stacked parent.

— Magi

[miguel-heygen]

Re-approved at b99c7941333980c0cb0515f0e86e381a151cc25e against 9244c47328e41e59034bd9bbec293fe3f7310cf6, compared with my previously checked 833259d63dfdfae17e01fc47a7f7fd9cae28b31f layer.

Audited: the complete 24-path parent-to-head patch and every layer file's blob. Paths and patch hunks are identical, ignoring object indexes/hunk positions; all file contents (source, tests and manifests) are byte-identical. Independent git range-diff marks all 9 commits =. The matching-stream/nonzero-exit peak decoding repairs remain unchanged.

Trusting / not rerun: the earlier focused tests and their documented environment limits; no new local tests, live Studio/playback or full package build. No new task worktree/server/browser was created. Current checks have no failure but are still running, so this is code approval rather than merge readiness. No merge, enqueue, release or CI bypass performed.

Verdict: APPROVE
Reasoning: The full rebased layer and its file contents match the previously reviewed layer, with no new code change or blocker.

— Magi