build(webcam): stage ONNX Runtime so the segmentation effect can actually run - #530
Merged
Merged
Conversation
…ually run The compositor has segmented the webcam since #493, and on macOS since #529 — but `ort` is linked `load-dynamic`, nothing ever put a library where `ensureOnnxRuntimeOnPath` looks, and so the feature has been dark on every platform since it landed. This provisions it. `scripts/fetch-onnxruntime.mjs`, wired into `build:mac`, `build:win` and `build:win:store`. Modelled on `fetch-ffmpeg.mjs`: pinned to an immutable release, SHA-256 verified BEFORE the archive is opened, and the extracted library checked three ways before it is allowed near `electron/native/bin/` — the archive's own LICENSE really is MIT, the binary really is a PE/Mach-O/ELF for the target, and it really carries the pinned version string. The digest alone proves only that we got the archive we asked for; it says nothing about having lifted the right file out of it, which is where a re-pin actually goes wrong. `scripts/fetch-onnxruntime.test.mjs` guards a coupling that crosses a language boundary: `crates/Cargo.toml` gives `ort` the feature `api-27`, which is the MINIMUM runtime minor version it accepts — below it `GetApi` returns null and `ort` panics rather than erroring, on the render thread, on the first frame with an effect. Nothing else would notice those two pins drifting apart; they are in different files, different languages, and touched by different tasks. CI now stages the library for the macOS compositor job and points ORT_DYLIB_PATH at it. Without that, `runtime_available()` is false and every segmentation test returns early — the suite went green having exercised no inference at all, which is exactly how the ort-panics-when-absent bug got in. Two findings, both now in the docs rather than the estimate they replace: - macOS costs 38.5 MB, not the "roughly 15 MB per platform" this was scoped as — 2.5x Windows' 15.4 MB. `strip -x` would halve it, but stripping means shipping something other than the artifact the digest vouches for. - Intel Macs cannot have it: upstream publishes no osx-x86_64 or universal asset from 1.27 on. The x64 DMG ships without it and the effect is absent there, degrading as designed. The script says so and exits 0 rather than failing that build. Linux is pinned but not wired in: its back-end has the shader half only, so the library would be 23 MB of installer for a code path that cannot run. Also adds the MediaPipe model to THIRD-PARTY-NOTICES.md. It is Apache-2.0 and ships inside app.asar, but its provenance README is stripped by electron-builder's `"!*.md"` filter — so the attribution reached no user at all.
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
EtienneLescot
added a commit
that referenced
this pull request
Aug 29, 2026
…use it #530 pinned every target but wired only `build:mac`, `build:win` and `build:win:store`, for a stated reason: the Linux compositor carried the segmentation shader and nothing that feeds it, so the library would have been 23 MB of installer for a code path that could not run. "Landing the Linux capture half makes it a one-line change," said the script. The previous commit lands it; this is the one line, plus the two places that repeated the reasoning. It also closes a gap the previous commit opened. That commit removes `supportsWebcamSegmentation`, so the camera-background control is now visible on Linux — and a control is only worth showing where a mask can actually reach the shader. Without the library staged, a packaged Linux build would offer the setting and do nothing with it, which is precisely what the gate existed to prevent. The two changes have to travel together. CI stages it for the Linux compositor job too, mirroring what #530 did for macOS and for the same reason: without a library on `ORT_DYLIB_PATH`, `runtime_available()` is false and every segmentation test returns early. The suite goes green having exercised no inference at all — which is exactly how the `ort`-panics-when-absent bug got in. It is what makes `the_whole_loop_produces_a_mask_from_compose_frame_alone` a real test on that runner rather than a skipped one. `fetch-onnxruntime.mjs` imports only node builtins, so it needs no `npm ci`, same as `fetch:ffmpeg:sdk` beside it. Verified locally rather than assumed: `node scripts/fetch-onnxruntime.mjs --target=linux-x64` downloads the pinned asset, passes the SHA-256 check before unpacking, and passes all three post-extraction checks — MIT LICENSE, ELF magic, version string — yielding a 22.6 MB `libonnxruntime.so`. Pointing `ORT_DYLIB_PATH` at THAT file (not a hand-installed one) runs the whole compositor suite green with inference actually executing. No new pin and no new digest: the `linux-x64` entry, its SHA-256 and the `api-27` cross-check in `fetch-onnxruntime.test.mjs` all already existed and are untouched.
EtienneLescot
added a commit
that referenced
this pull request
Aug 29, 2026
…use it #530 pinned every target but wired only `build:mac`, `build:win` and `build:win:store`, for a stated reason: the Linux compositor carried the segmentation shader and nothing that feeds it, so the library would have been 23 MB of installer for a code path that could not run. "Landing the Linux capture half makes it a one-line change," said the script. The previous commit lands it; this is the one line, plus the two places that repeated the reasoning. It also closes a gap the previous commit opened. That commit removes `supportsWebcamSegmentation`, so the camera-background control is now visible on Linux — and a control is only worth showing where a mask can actually reach the shader. Without the library staged, a packaged Linux build would offer the setting and do nothing with it, which is precisely what the gate existed to prevent. The two changes have to travel together. CI stages it for the Linux compositor job too, mirroring what #530 did for macOS and for the same reason: without a library on `ORT_DYLIB_PATH`, `runtime_available()` is false and every segmentation test returns early. The suite goes green having exercised no inference at all — which is exactly how the `ort`-panics-when-absent bug got in. It is what makes `the_whole_loop_produces_a_mask_from_compose_frame_alone` a real test on that runner rather than a skipped one. `fetch-onnxruntime.mjs` imports only node builtins, so it needs no `npm ci`, same as `fetch:ffmpeg:sdk` beside it. Verified locally rather than assumed: `node scripts/fetch-onnxruntime.mjs --target=linux-x64` downloads the pinned asset, passes the SHA-256 check before unpacking, and passes all three post-extraction checks — MIT LICENSE, ELF magic, version string — yielding a 22.6 MB `libonnxruntime.so`. Pointing `ORT_DYLIB_PATH` at THAT file (not a hand-installed one) runs the whole compositor suite green with inference actually executing. No new pin and no new digest: the `linux-x64` entry, its SHA-256 and the `api-27` cross-check in `fetch-onnxruntime.test.mjs` all already existed and are untouched.
EtienneLescot
added a commit
that referenced
this pull request
Aug 30, 2026
…use it #530 pinned every target but wired only `build:mac`, `build:win` and `build:win:store`, for a stated reason: the Linux compositor carried the segmentation shader and nothing that feeds it, so the library would have been 23 MB of installer for a code path that could not run. "Landing the Linux capture half makes it a one-line change," said the script. The previous commit lands it; this is the one line, plus the two places that repeated the reasoning. It also closes a gap the previous commit opened. That commit removes `supportsWebcamSegmentation`, so the camera-background control is now visible on Linux — and a control is only worth showing where a mask can actually reach the shader. Without the library staged, a packaged Linux build would offer the setting and do nothing with it, which is precisely what the gate existed to prevent. The two changes have to travel together. CI stages it for the Linux compositor job too, mirroring what #530 did for macOS and for the same reason: without a library on `ORT_DYLIB_PATH`, `runtime_available()` is false and every segmentation test returns early. The suite goes green having exercised no inference at all — which is exactly how the `ort`-panics-when-absent bug got in. It is what makes `the_whole_loop_produces_a_mask_from_compose_frame_alone` a real test on that runner rather than a skipped one. `fetch-onnxruntime.mjs` imports only node builtins, so it needs no `npm ci`, same as `fetch:ffmpeg:sdk` beside it. Verified locally rather than assumed: `node scripts/fetch-onnxruntime.mjs --target=linux-x64` downloads the pinned asset, passes the SHA-256 check before unpacking, and passes all three post-extraction checks — MIT LICENSE, ELF magic, version string — yielding a 22.6 MB `libonnxruntime.so`. Pointing `ORT_DYLIB_PATH` at THAT file (not a hand-installed one) runs the whole compositor suite green with inference actually executing. No new pin and no new digest: the `linux-x64` entry, its SHA-256 and the `api-27` cross-check in `fetch-onnxruntime.test.mjs` all already existed and are untouched.
EtienneLescot
added a commit
that referenced
this pull request
Aug 30, 2026
…use it #530 pinned every target but wired only `build:mac`, `build:win` and `build:win:store`, for a stated reason: the Linux compositor carried the segmentation shader and nothing that feeds it, so the library would have been 23 MB of installer for a code path that could not run. "Landing the Linux capture half makes it a one-line change," said the script. The previous commit lands it; this is the one line, plus the two places that repeated the reasoning. It also closes a gap the previous commit opened. That commit removes `supportsWebcamSegmentation`, so the camera-background control is now visible on Linux — and a control is only worth showing where a mask can actually reach the shader. Without the library staged, a packaged Linux build would offer the setting and do nothing with it, which is precisely what the gate existed to prevent. The two changes have to travel together. CI stages it for the Linux compositor job too, mirroring what #530 did for macOS and for the same reason: without a library on `ORT_DYLIB_PATH`, `runtime_available()` is false and every segmentation test returns early. The suite goes green having exercised no inference at all — which is exactly how the `ort`-panics-when-absent bug got in. It is what makes `the_whole_loop_produces_a_mask_from_compose_frame_alone` a real test on that runner rather than a skipped one. `fetch-onnxruntime.mjs` imports only node builtins, so it needs no `npm ci`, same as `fetch:ffmpeg:sdk` beside it. Verified locally rather than assumed: `node scripts/fetch-onnxruntime.mjs --target=linux-x64` downloads the pinned asset, passes the SHA-256 check before unpacking, and passes all three post-extraction checks — MIT LICENSE, ELF magic, version string — yielding a 22.6 MB `libonnxruntime.so`. Pointing `ORT_DYLIB_PATH` at THAT file (not a hand-installed one) runs the whole compositor suite green with inference actually executing. No new pin and no new digest: the `linux-x64` entry, its SHA-256 and the `api-27` cross-check in `fetch-onnxruntime.test.mjs` all already existed and are untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #493, follows #529. The compositor has segmented the camera since #493, and on macOS since #529 — but
ortis linkedload-dynamicand nothing has ever put a library whereensureOnnxRuntimeOnPathlooks, so the feature has been dark on every platform since it landed. This provisions it.The script
scripts/fetch-onnxruntime.mjs, wired intobuild:mac,build:winandbuild:win:store. Modelled onfetch-ffmpeg.mjs: pinned to an immutable release tag, SHA-256 verified before the archive is opened, only the plain CPU assets (thegpu_cuda*ones are 200–320 MB and carry NVIDIA redistribution terms).The extracted library is then checked three ways before it is allowed near
electron/native/bin/:LICENSEreally does beginMIT License— an asset name is not evidence;The digest alone only proves we got the archive we asked for. It says nothing about having lifted the right file out of it, and that is where a re-pin actually goes wrong: a renamed member silently vendors a 20 KB provider stub, and the failure surfaces months later as "the effect does nothing".
It never fails a build. Missing ONNX Runtime is a designed-for state — the compositor logs one line and draws the camera unsegmented.
The test earns its place
crates/Cargo.tomlgivesortthe featureapi-27. That is the minimum runtime minor version it accepts: below itGetApireturns null andortpanics rather than erroring — on the render thread, on the first frame with an effect. Above it,ortlogs a compatibility warning into a stderr no packaged app shows.Nothing would notice those two pins drifting apart. They are in different files, in different languages, edited by different tasks, and the diff of either one reads fine alone.
scripts/fetch-onnxruntime.test.mjscross-checks them in both directions:I checked that message is real by temporarily pinning 1.19.2 — the guard fires rather than passing vacuously, which is the same discipline the existing
fetch-ffmpeg.test.mjswas written with.CI now actually exercises inference
The macOS compositor job stages the library and sets
ORT_DYLIB_PATH. Without itruntime_available()is false and every segmentation test returns early — the suite went green having run no inference at all. That is precisely how theort-panics-when-absent bug got in, per this feature's own docs. ~30 MB, next to nothing beside thebrew install ffmpegalready in that job.Two findings that contradict how this was scoped
The doc estimated "roughly 15 MB per platform". Measured on the real 1.27.1 artifacts:
onnxruntime.dlllibonnxruntime.dyliblibonnxruntime.sostrip -xtakes it from 38.5 MB to 23.7 MB, but stripping means shipping something other than the artifact the digest vouches for — which is the entire point of pinning. Not done; flagged in the doc in case installer size ever forces the trade. Your call if you'd rather take the 15 MB.osx-x86_64or universal asset for any release from 1.27 on — I checked 1.27.0, 1.27.1, 1.28.0, 1.28.1 and 1.29.0. Sincebuild.ymlships an x64 DMG frommacos-15-intel, that DMG goes out without the library and the effect is simply absent there. The script says so plainly and exits 0, because failing would break that release build over a feature designed to degrade. The only alternative is building ORT from source, anfetch-ffmpeg-macos.mjs-sized script for a shrinking platform.Linux is pinned in the same table but deliberately not wired into
build:linux: its back-end carries the shader half only, so the library would be 23 MB of installer for a code path that cannot run. Landing the Linux capture half makes it a one-line change.An unrelated licence gap I hit on the way
The MediaPipe model is Apache-2.0 and ships inside
app.asar(Vite copiespublic/→dist/, anddistis infiles). Its provenance README is stripped by electron-builder's"!*.md"filter, and it had no entry inTHIRD-PARTY-NOTICES.md— so the attribution Apache-2.0 §4 asks for reached no user at all. Added, alongside the ONNX Runtime entry. Say the word if you'd rather that moved to its own PR.Verified
Same caveat as #529:
ci.ymldoes not trigger on PRs intofeat/webcam-effects, so no job runs here either. Locally, on an M1:ORT_DYLIB_PATHset to the staged file — the full-loop inference test runs for real rather than skippingBeyond the jobs: ran the script for real (fetch → verify → vendor), confirmed it is idempotent, confirmed it detects and re-fetches a corrupted file rather than trusting the filename, and confirmed the
darwin-x64branch exits 0. Also cross-fetched--target=win32-x64from this Mac, which verified the Windows DLL's PE magic and version string without a Windows machine.