Repository navigation
Camera effects on the Neural Engine: Portrait blur, Studio Light, Center Stage - #16
m0d16l14n1 wants to merge 5 commits into
Conversation
…ts, wide LOAD (WIP) HWX surface names up to 64 bytes and the 'net' procedure, programs with several outputs and a BAR slot table of any size, ANE_H16_LOAD_WIDE (16 surfaces), driver minor 2; h16-sections.py handles several outputs. Large temporaries moved to the heap (stack use). KUnit 239/239. Run on a J815 (2026-11-21, same change on the gpu-gap kernel): macOS's personsemantics segmentation (person + skin outputs) ran at 15 runs/s (30 fps preview) for the camera effects in aurora-silicon/camera-denoise#16. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts, wide LOAD (WIP) HWX surface names up to 64 bytes and the 'net' procedure, programs with several outputs and a BAR slot table of any size, ANE_H16_LOAD_WIDE (16 surfaces), driver minor 2; h16-sections.py handles several outputs. Large temporaries moved to the heap (stack use). KUnit 239/239. Run on a J815 (2026-11-21, same change on the gpu-gap kernel): macOS's personsemantics segmentation (person + skin outputs) ran at 15 runs/s (30 fps preview) for the camera effects in aurora-silicon/camera-denoise#16. Assisted-by: Claude Opus 5.5 Signed-off-by: Nick <nekreshen@yandex.ru>
…ts, wide LOAD HWX surface names up to 64 bytes and the 'net' procedure, programs with several outputs and a BAR slot table of any size, ANE_H16_LOAD_WIDE (16 surfaces), driver minor 2; h16-sections.py handles several outputs. Large temporaries moved to the heap (stack use). KUnit 239/239. Run on a J815 (2026-11-21, same change on the gpu-gap kernel): macOS's personsemantics segmentation (person + skin outputs) ran at 15 runs/s (30 fps preview) for the camera effects in aurora-silicon/camera-denoise#16. Assisted-by: Claude Opus 5.5 Signed-off-by: Nick <nekreshen@yandex.ru>
|
Early note on this draft (no full review yet):
Assisted-by: LLM (Claude:Opus 5.5) |
The default filter is the strongest it can be: the filtered picture only, the full 3x3 smoothing after the temporal stage and an 8-frame running mean. On the J815 that read as soft, with smear on motion. Three [denoise] keys, live on reload: - strength 0..1: the filtered picture mixed with the camera's (1 = filtered only, the default; 0.5 = half the grain comes back); - spatial 0..1: scales the 3x3 smoothing (0 = none, 1 = default); - temporal-frames 2..8: caps the running mean (8 = default). The CPU and GPU backends take them through cdn_frame_info and stay bit-exact with each other (test_gpu); test_cpu checks that a lighter setting keeps more of the input. Out-of-range values are refused like the other keys. The plugins read the new struct cdn_frame_info fields, so CDN_BACKEND_ABI and CDN_DMABUF_ABI move to 2: a plugin and a caller built for different layouts now refuse each other instead of reading past the end of an older caller's info. The loader test's bad fixture is a plugin built for ABI 1. Signed-off-by: Nick <nekreshen@yandex.ru>
…th, tint, sharpen A last pass over each output frame, after the denoiser and also on the frames it passes through: - vignette 0..1: lens-shading compensation, gain 1 + v * r^2 (r = 1 in the corners) on luma and on the chroma's distance from neutral, for cameras whose ISP tuning leaves the corners dark; - brightness -1..1, contrast 0.5..2 (around mid grey): one luma LUT; - saturation 0..2, warmth -1..1 (Cb/Cr toward orange or blue), tint -1..1 (toward magenta or green); - sharpen 0..1: a 3x3 unsharp mask on luma. All neutral by default, and the pass is skipped then. The gain maps and the LUT are rebuilt only when the size or the settings change. On the CPU, NV12 in place; test_picture covers neutral, the vignette's centre, corners and symmetry, each control's direction and an edge for sharpen. Signed-off-by: Nick <nekreshen@yandex.ru>
A stream's camera.zoom control (or the new zoom setting) only took effect through the effects plugin, which crops and scales on the CPU. Without the plugin, the filter now does the zoom itself as its last stage: a centred crop scaled back to the stream's size. On a Mac with a scaler that takes an OUTPUT crop (the M5's MSR23, driver apple-scaler) the crop and the polyphase scaling run there, and the CPU copies the frame in and out. Without one, or after any error from it, a CPU bilinear pass does the same, so the stream keeps going. zoom-engine picks auto, cpu or scaler. aurora-camera-zoom-check times both paths on a synthetic frame and compares their outputs. Signed-off-by: Nick <nekreshen@yandex.ru>
…ter Stage A camera_effects plugin (libaurora-camera-effects-apple.so) that runs a person and skin segmentation network on the Neural Engine through the accel/ane driver. The network is the user's own: compiled for this Mac's engine by the user's macOS install and copied from it by install-apple-effects into <appledir>/segmentation/, with the four host-built sections the driver loads with it. Nothing of it ships with this package; without it the filter keeps using the companion plugin. - The driver decides where the program runs: it is loaded on each ABI 3 accel node in turn and the first one that accepts it is used. The driver validates the program and reports its surfaces (size and direction in program order); the plugin gives each surface the shape of the network's interface (one RGB input, person and skin planes, 256x192, fp16) and takes the strides from the reported sizes. It does not read the program file. - Portrait (camera.background = blur): masks refined to the frame with a guided filter, background blurred by normalized convolution. - Studio Light (camera.studio-light 0..1): the face lifted by a fill curve that spares highlights, the background dimmed; tuned here by eye. - Center Stage (camera.framing): a framing session (Kalman-filtered faces, a face/body tracker, a director, deadband / recentre / return-home, a rectangle animator) rendered as a virtual camera rotated to the viewport (sub-pixel warp). Its tuning is read at run time from the user's install (<appledir>/tuning/framing.conf, "key = value"); without it Center Stage stays off. Faces and bodies come from the ISP firmware's detector when the camera driver runs it, else from the masks; the lens is the calibration the ISP firmware returns when the driver reads it, else a pinhole. - The full-size passes run on the GPU (Vulkan) when there is a usable device, else on the CPU. - The gain reader takes V4L2_CID_ANALOGUE_GAIN when the sensor has no V4L2_CID_GAIN (the T8142 ISP). CDN_EFFECTS_DEBUG=mask|stats|faces and CDN_EFFECTS_DUMP=<dir> are debugging aids only; nothing else in the environment changes what the plugin does. Signed-off-by: Nick <nekreshen@yandex.ru>
Plain C versions of the Portrait (depth-based background blur) and Studio
Light (relighting) renderers, written to match the same passes rendered on
macOS on this Mac, as the reference a GPU port of the effects is checked
against. They are not used by the plugin yet: they need the depth network
on the Neural Engine.
- sdof: the ray pass over a 4-level colour pyramid, the guided upsampling of
the depth, the composite with its grain, the temporal filter.
- studiolight: the diffuse term from depth, the face light mask, the gain map,
its blur and upsampling, the relight at full size through a colour cube.
- focus: the focus from faces that both renderers use.
- faceexp: the face exposure statistics behind Studio Light's light estimate.
Its tuning is read at run time ("face-exposure.conf" from the user's
install); nothing of it is in the tree.
tests/test_sdof.c, test_studiolight.c and test_faceexp.c check each pass on
its own inputs against a capture of the same pass on macOS (CDN_SDOF_REF,
CDN_SL_REF, CDN_FEXP_REF). The captures are macOS's output for a picture of a
person, so they stay local and the tests skip without them. On 12 captures
(one, ten and six frames; no face, a square and a wide face, BT.709 and
BT.601 tags; one and two faces of different brightness): every face
statistic, coefficient and the light estimate exact; Studio Light's whole
frame 100% of luma within 1 LSB untagged (worst 2) and 99.96-99.97%
tagged (worst 4); Portrait's whole first frame 99.97% of luma within 1 LSB
(worst 8).
Signed-off-by: Nick <nekreshen@yandex.ru>
f317008 to
64e2479
Compare
|
@RyanTheTide thanks for the early note. All six points are addressed in
The description has a section on how the ports were written, and I'd like your call on it:
There is also an open question on the
Assisted-by: LLM (Claude:Opus 5.5) |
|
Thanks for the rework. Status of the six earlier points at 64e2479:
On your question about how the ports were written: replacing the constants is not enough. By the description and by the code,
The structure and the algorithm come from the vendor's binaries, not only the numbers, so these files can't be in this repository in any form. Please drop them, and those parts of What can stay is an implementation written from a description of the behaviour, by someone who hasn't read the binaries, and tested against macOS output as a black box with tolerances. The pass-level tests ( The previous head Other points at this head:
Assisted-by: LLM (Claude:Opus 5.5) |
…al outputs, wide LOAD What the camera's segmentation program needs from the H16 program path: - HWX surface names up to 64 bytes and procedure names other than 'main'; - programs with several outputs and a BAR slot table of any size; - ANE_H16_LOAD_WIDE, for programs of up to 16 surfaces (plain H16_LOAD refuses more than 4 with -E2BIG, since its arrays cannot report them); - driver minor 2: plain H16_LOAD now also takes a program with several outputs, where minors 0 and 1 refused one, so userspace that relies on several outputs checks for minor 2 (the uapi header says so); - the parser's large temporaries on the heap (stack use); - h16-sections.py builds the sections of a program with several outputs. Run on a J815 (2026-11-21): the user's person and skin segmentation program (two outputs) ran at 15 runs/s for the camera effects in aurora-silicon/camera-denoise#16. Signed-off-by: Nick <nekreshen@yandex.ru>
|
Closing this: I'm taking the camera effects out of this repository and continuing them on my own machine, so I won't rework the ports here. As your review asked, the earlier heads of this PR (including f317008) should be purged; I'd appreciate it if a maintainer could ask GitHub for that. Thanks for the reviews. Assisted-by: LLM (Claude:Opus 5.5) |
Draft. This is stacked on #13 (base
f78464c, which has the ABI bump). It also carries the scaler zoom commit. It needs the ANE driver from aurora-silicon/linux#2647 (programs with several outputs,H16_LOAD_WIDE).What it adds
libaurora-camera-effects-apple.so(camera_effects ABI 1) runs a person and skin segmentation network on the Neural Engine. It provides:camera.background = blur);camera.studio-light0..1);camera.framing).The network is the user's own: their macOS install compiled it for this Mac's engine, and
install-apple-effectscopies it from there. Nothing of it ships here. Without it, the filter keeps using the companion plugin.Three commits on top of #13:
8bb5d85Digital zoom on the SoC scaler, unchanged apart from the message.161ba54The plugin. The three effects, the ANE program runner, the run-time tuning reader, the installer and the host tests.5dce724Reference passes for Portrait and Studio Light in plain C (sdof,studiolight,focus,faceexp). They are the reference a GPU port of the full renderers will be checked against.Review (Ryan, 2026-11-21): what changed
framing.c/framing.h(the older controller) are gone.<appledir>/tuning/framing.conf, plainkey = valuelines covering deadbands, speeds, accelerations, durations, the default and minimum view and the camera's focal length.tuning/*.confwith the network.face-exposure.conf).segmentation/.srlv2→faceexp,ptfocus→focus.ane_prog.cno longer reads the HWX, and the installer no longer checks its magic.H16_LOAD_WIDEand reports each surface's size and direction in program order.apple_tensor_layout, host-tested).CDN_APPLE_SEG_BGRis gone. The input order is R, G, B: the masks looked right with it on the J815. B, G, R was never compared side by side.CDN_APPLE_GPUis gone: the GPU is used when there is a usable Vulkan device, else the CPU.CDN_APPLE_EFFECTS_DIRis gone: the directory is the build'sappledir.CDN_ZOOM_DEVICEis gone: the scaler is found by its driver name.test_zoomandtest_apple_pluginno longer need overrides.CDN_APPLE_FRAMING,CDN_APPLE_DETECT,CDN_APPLE_LENS) are gone too. The ISP firmware's detections and lens calibration are used when the camera driver provides them; otherwise the masks and a pinhole are used.detect/detections,detect/lens_distortion) come from ISP driver work that isn't in any pushed kernel branch yet, and the readers have not run on hardware. Until that lands, Center Stage uses the masks and the pinhole.CDN_EFFECTS_DEBUG=mask|stats|facesandCDN_EFFECTS_DUMP, is debugging only.[sensor:imx958-uw](from Tuning keys and a picture stage (vignette, colour, sharpen) #13 and config, main, data: key camera profiles on sysfs sensor name ([sensor:imx958-uw]) #14).How the ports were written (please read)
Center Stage's framing session (
cf_framing.c) and the reference passes in the third commit (sdof,studiolight,focus,faceexp) were written by reading the macOS framing code and the compiled GPU shaders of the macOS renderers. They were then matched against macOS's own output on this Mac:Parts of the plugin's
portrait.cstructure came from the same reading. The tuning no longer sits in the tree (point 1). The fixed constants are a different matter. Every numeric constant incf_framing.cand in the four reference files is the reference's, unless a comment marks it as ours. That includes:EASE_*) and its Kalman filter and tracker constants;Studio Light's colour-correction cube is vendor data and stays out: the pass takes it as input, and without it the result is still 99.93–100% within 1 LSB.
If these may not be in the tree, I can replace each constant with a value chosen here and say how it was chosen. The exact-match tests would then become tolerance tests. Tell me which you prefer.
Open question:
ce_config.studio_lightThe field is appended to
struct ce_configwithout aCE_ABIbump. That is the pattern #13 was asked to fix, so I'd rather settle it here than leave it:The options:
CE_ABIto 2. The companion plugin in aurora-silicon/omarchy-ane is then refused until it is rebuilt.CDN_DMABUF_ENTRY/CDN_QUERY_ENTRYdo.Which do you prefer?
Testing
meson test -Dwerror=truein an Arch Linux ARM container (aarch64, no GPU):apple,apple-plugin,zoom,configandpicture;gpuandapple-gpu(no Vulkan device), andapple-sdof,apple-studiolightandapple-faceexp(no local captures);gpu-dmabuffails exactly as onmainin that container (no Vulkan device).Reference checks on macOS (12 captures of one frame):
Framing replay on this head: 35 macOS traces through
cf_framing.cwith the tuning read from an exportedframing.conf: 0 state mismatches, viewports within 1e-3 rad at worst. The lens-distortion path hasn't changed since its replay on 29 traces with the real lens (28 identical, one boundary flip at frame 469 of 480). I didn't rerun that here.Hardware: not run on this head yet. The J815 row is next:
framing.confis removed.Assisted-by: LLM (Claude:Opus 5.5)