feat(webcam): port segmentation to the macOS Metal back-end

[EtienneLescot]

Stacked on #493. Ports the capture/upload half of webcam segmentation to the Metal back-end, which had the shader half only.

The brief is webcam-segmentation-backend-port.md; this follows it, and updates it plus webcam-segmentation.md to match what is now true.

What was missing

ps_main already took texMask [[texture(3)]] and branched on fx.z. Nothing on macOS ever raised fx.z, because nothing captured the camera or uploaded a mask. The effect was inert and the control was hidden.

What this adds

• capture_webcam_rgb — renders the camera NV12 into a 256x144 Private target, blits to a Shared mirror, get_bytes, compacts RGBA→RGB in place. It passes the whole valid frame ([0, 0, wcw/wtw, wch/wth]), not the sub-rect drawn, so a tight user crop cannot amputate the subject at the model's input.
• set_webcam_mask — MTLPixelFormat::R8Unorm + replace_region, reallocated only when the model resolution changes.
• pump_segmentation at the top of compose_frame, before the compose command buffer exists. The worker, inbox, 30 Hz limiter, lazy start from scene.webcam_effect.model_path and the seg_failed latch are copied from Windows unchanged — that half was already platform-independent.
• fx and color on the webcam LayerCB, the mask bound at index 3, and the PiP drop shadow suppressed in cutout mode.
• supportsWebcamSegmentation now admits darwin. Linux stays gated.

Untouched, as the brief requires: the shader contract, the 25-tap blur, and segmentation.rs.

The traps, and what happened at each

• submit/sync for the capture pass. Avoided. sync() waits on last_cmd, and read_nv12_scaled needs last_cmd to stay the render_nv12 buffer — taking it would make the encode readback wait on the capture and read plans nothing had written. The capture commits and waits on its own buffer.
• effect_code must stay 0 until a mask exists. It does, and a_mode_without_a_mask_composites_exactly_like_no_effect_at_all asserts it byte for byte rather than approximately.
• Index 3 must be bound wherever the branch can be taken. In this back-end that is one draw: the mask branch is behind mode < 0.5 and fx.z > 0.5, and only the webcam layer raises fx.z. It shares an encoder with the PiP shadow, which is mode 2. No unbind afterwards — unlike Windows, that state dies with the encoder and the annotations open their own.
• synchronizeResource on a discrete GPU. Checked, not assumed: not needed. Shared is not Managed, and this is the same Private→blit→Shared shape rt_read and nv12_read_y have used in production all along, so it carries no support risk the existing readbacks do not.
• ort panics when its library is missing. runtime_available() and catch_unwind both left alone.

One thing the brief did not list, worth flagging: set_webcam_mask rewrites a texture the GPU could in principle still be sampling. It cannot here, because all three macOS frame paths drain the queue before returning (readback_direct and rgb_to_nv12 do submit + sync, read_nv12_scaled does sync), so nothing is in flight when compose_frame next pumps. That is why there is no double buffer, and it is written down on the function — if that invariant changes, this is the code that breaks.

What I actually saw on screen

poc-d3d is cfg(windows), so the --cfg C8 --scene route does not exist here. I rendered a real 1280x720 camera frame — a person against dense foliage, deliberately a hard case for hair — through the real compose_frame on this M1, with real ONNX Runtime inference, and looked at all four modes:

• none — camera whole, rounded PiP, drop shadow bottom-right. Colours match the source, so the NV12 round trip is faithful.
• cutout — subject isolated cleanly against the screen layer; the foliage is gone. The PiP frame and its shadow are gone too, which is the shadow suppression visible rather than merely asserted.
• blur — foliage smeared, subject sharp, PiP frame and shadow kept.
• custom — flat #ff2d95 behind the subject, PiP frame and shadow kept.

Honestly: there is a faint dark fringe along the hairline and shoulders, one to two pixels of the original dark background surviving the matte. It is most visible in custom mode against saturated pink. That is the 256x144 mask upscaled ~4.5x by a linear sampler, so it is a property of the shipped mask resolution and the shared shader, not of this port — Windows renders the same artefact from the same inputs. Calling it out because "the mask composites" and "the mask is correct" are different claims and the brief says so.

The harness that produced those images is committed as seg_visual_renders_the_four_modes_from_a_real_photo, opt-in behind OPENSCREEN_SEG_VISUAL + OPENSCREEN_SEG_CAM (same gate shape as tests/compose_linux.rs).

Cost

Indicative only — not a §C.2 run (no fixture, no --repeat 3, its own harness). M1, 8 GPU cores, macOS 26.5, 1280x720 preview including the synchronous readback_direct the preview already pays. A/B/A/B interleaved, 300 frames per arm, warm-up discarded; spread between repeats of the same arm ≤ 1.2 %.

arm	p50	vs off
effect off	3.04 ms	—
cutout	2.57 ms	−0.47 ms
blur	3.47 ms	+0.43 ms

Cutout is cheaper than no effect at all — it drops the PiP shadow, one fewer full-quad SDF draw, which more than pays for everything the feature adds.

What these arms do not isolate is the capture. Cutout and blur do identical capture, inference and upload work; they differ only by the shader branch and the shadow, so the 0.90 ms between them is shader work and the capture cost sits inside both arms unattributed. Separating it would need a fourth arm — effect requested, capture running, fx.z pinned at 0 — which is not a state the compositor can be asked for. So the claim is the net one, and it is a this machine claim: M1 unified memory makes the 147 KB Private→Shared blit nearly free, and on an Intel Mac with a discrete GPU it is a real bus transfer. Unmeasured there.

Tests

cargo test -p openscreen-compositor --lib --tests — 156 passed (was 146), green on a real Metal device both with and without ONNX Runtime on ORT_DYLIB_PATH. cargo build -p compositor-view-napi clean. Ten new tests, all rendering real pixels; seven post the mask by hand and need no ONNX Runtime, which is what CI has, and the full-loop one skips cleanly without it.

test	what it would catch
the_webcam_capture_comes_back_as_interleaved_rgb_at_model_resolution	a readback that is RGBA, mirrored, or reallocating every frame
the_mask_actually_cuts_the_camera_out	an unbound texture at index 3, or a mask reaching the shader as noise
the_custom_background_colour_replaces_the_masked_out_pixels	color not carried onto the webcam LayerCB
a_mode_without_a_mask_composites_exactly_like_no_effect_at_all	effect_code leaving 0 too early — the invisible-webcam trap
compose_frame_cuts_the_camera_out_once_a_mask_exists	fx not carried, by counting camera pixels rather than pinning PiP geometry
the_pip_shadow_is_suppressed_in_cutout_mode	the shadow of an invisible box — with a control render, so it cannot pass vacuously
the_whole_loop_produces_a_mask_from_compose_frame_alone	capture → inference → upload driven by compose_frame alone

The end-to-end ones go through real CVPixelBufferRef-backed AVFrames and therefore the same nv12_srvs the decoder uses; the NV12 fixture helper is #[cfg(test)] in mac_frames.rs, next to the CoreVideo FFI it reuses, so the test does not grow a second copy of it.

Note on ONNX Runtime

Still not staged for any platform, so the effect remains off at runtime until a CI job puts the library in electron/native/bin/<tag>/. That is unchanged by this PR and is tracked in webcam-segmentation.md under Not done. Removing the macOS gate reflects back-end capability, matching what Windows already does.

I ran locally against ONNX Runtime 1.27.1 osx-arm64 from the official microsoft/onnxruntime release, which matches the pinned api-27 exactly.

⚠️ CI does not run on this PR

ci.yml triggers on pull_request: branches: [main, feat/ai-edition, "release/**"]. This PR targets feat/webcam-effects, which matches none of them, so every job is skipped — including Rust test (macOS compositor), the one that would exercise this change. The only checks you see are CodeRabbit (disabled for this base) and the Discord sync.

That is the same gap the workflow's own header comment is about — it cost #167–#169 on a release branch. Flagging rather than fixing it: adding feat/webcam-effects to that list is a repo-wide change and feat/** is a short-lived branch pattern, so it is a call for whoever owns the branch strategy. #493 targets main and will run everything when it lands.

In its place I ran the CI jobs locally, on an M1 with a real Metal device:

CI job	command	result
Rust test (macOS compositor)	cd crates && cargo test -p openscreen-compositor --lib --tests	156 passed, both with and without ORT_DYLIB_PATH
Rust test (macOS compositor)	cargo build -p compositor-view-napi --release	clean
Lint	biome check on the changed file	clean
Type Check	npx tsc --noEmit	clean
Test	npm run test	181 files, 2171 passed, 2 skipped
Docs	node scripts/check-docs.mjs	OK (34 files)

Rust check (Windows compositor) I cannot run here. This PR touches no Windows file — compositor_macos.rs and mac_frames.rs are both cfg(target_os = "macos").

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 77ad2a2a-dbf0-4c67-ad2f-a71a356ce7d2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}