Skip to content

Camera effects on the Neural Engine: Portrait blur, Studio Light, Center Stage - #16

Closed
m0d16l14n1 wants to merge 5 commits into
aurora-silicon:mainfrom
m0d16l14n1:j815/apple-effects
Closed

m0d16l14n1 wants to merge 5 commits into
aurora-silicon:mainfrom
m0d16l14n1:j815/apple-effects

Conversation

@m0d16l14n1

@m0d16l14n1 m0d16l14n1 commented Nov 21, 2026 •

Copy link
Copy Markdown

Draft. This is stacked on #13 (base f78464c, which has the ABI bump). It also carries the scaler zoom commit. It needs the ANE driver from aurora-silicon/linux#2647 (programs with several outputs, H16_LOAD_WIDE).

What it adds

libaurora-camera-effects-apple.so (camera_effects ABI 1) runs a person and skin segmentation network on the Neural Engine. It provides:

  • Portrait background blur (camera.background = blur);
  • Studio Light (camera.studio-light 0..1);
  • Center Stage (camera.framing).

The network is the user's own: their macOS install compiled it for this Mac's engine, and install-apple-effects copies it from there. Nothing of it ships here. Without it, the filter keeps using the companion plugin.

Three commits on top of #13:

  1. 8bb5d85 Digital zoom on the SoC scaler, unchanged apart from the message.
  2. 161ba54 The plugin. The three effects, the ANE program runner, the run-time tuning reader, the installer and the host tests.
  3. 5dce724 Reference passes for Portrait and Studio Light in plain C (sdof, studiolight, focus, faceexp). They are the reference a GPU port of the full renderers will be checked against.
    • The plugin doesn't use them yet: they need the depth network on the ANE.
    • Their tests compare each pass with a capture of the same pass on macOS. Those captures are pictures of a person, so they stay local and the tests skip without them.

Review (Ryan, 2026-11-21): what changed

  1. Framing tuning copied from macOS.
    • framing.c/framing.h (the older controller) are gone.
    • Center Stage's tuning is now read at run time from the user's install: <appledir>/tuning/framing.conf, plain key = value lines covering deadbands, speeds, accelerations, durations, the default and minimum view and the camera's focal length.
    • No value of it is in the tree. Center Stage stays off without the file. The installer copies tuning/*.conf with the network.
    • The host test uses a made-up tuning.
    • Studio Light's face-exposure tuning works the same way (face-exposure.conf).
  2. Framework, class and resource names.
    • Code, docs, test names and commit messages now describe the stages by what they do: person/skin segmentation, framing session, face exposure statistics, ray pass.
    • The network's directory is segmentation/.
    • Files were renamed: srlv2 → faceexp, ptfocus → focus.
    • The reference tests read neutrally named pass files; the capture tooling outside this tree maps them.
    • "macOS" is still named as the reference the passes were matched against.
  3. No parsing of the vendor-compiled program.
    • ane_prog.c no longer reads the HWX, and the installer no longer checks its magic.
    • The driver validates the program at H16_LOAD_WIDE and reports each surface's size and direction in program order.
    • The plugin gives each surface its shape from the network's interface: one RGB input, then the person and skin planes, all 256x192 fp16. The strides come from the reported sizes, and a size that fits no dense layout is refused (apple_tensor_layout, host-tested).
    • Caveat: person and skin are told apart by program order only. That order was checked on the one program I have. The two planes are the same size, so a program that swapped them would not be caught at load time.
  4. No chip-family check. The program is loaded on each ABI 3 accel node in turn, and the first one that accepts it runs it.
  5. Environment switches.
    • CDN_APPLE_SEG_BGR is gone. The input order is R, G, B: the masks looked right with it on the J815. B, G, R was never compared side by side.
    • CDN_APPLE_GPU is gone: the GPU is used when there is a usable Vulkan device, else the CPU.
    • CDN_APPLE_EFFECTS_DIR is gone: the directory is the build's appledir.
    • CDN_ZOOM_DEVICE is gone: the scaler is found by its driver name. test_zoom and test_apple_plugin no longer need overrides.
    • The Center Stage A/B switches (CDN_APPLE_FRAMING, CDN_APPLE_DETECT, CDN_APPLE_LENS) are gone too. The ISP firmware's detections and lens calibration are used when the camera driver provides them; otherwise the masks and a pinhole are used.
    • Caveat: the two sysfs nodes those readers look for (detect/detections, detect/lens_distortion) come from ISP driver work that isn't in any pushed kernel branch yet, and the readers have not run on hardware. Until that lands, Center Stage uses the masks and the pinhole.
    • What's left, CDN_EFFECTS_DEBUG=mask|stats|faces and CDN_EFFECTS_DUMP, is debugging only.
  6. Board-keyed J815 copy: dropped. The J815 is [sensor:imx958-uw] (from Tuning keys and a picture stage (vignette, colour, sharpen) #13 and config, main, data: key camera profiles on sysfs sensor name ([sensor:imx958-uw]) #14).

How the ports were written (please read)

Center Stage's framing session (cf_framing.c) and the reference passes in the third commit (sdof, studiolight, focus, faceexp) were written by reading the macOS framing code and the compiled GPU shaders of the macOS renderers. They were then matched against macOS's own output on this Mac:

  • the framing session on traces of 35 scenarios;
  • the passes on 12 captures.

Parts of the plugin's portrait.c structure came from the same reading. The tuning no longer sits in the tree (point 1). The fixed constants are a different matter. Every numeric constant in cf_framing.c and in the four reference files is the reference's, unless a comment marks it as ours. That includes:

  • Center Stage's ease-in curves (EASE_*) and its Kalman filter and tracker constants;
  • Studio Light's per-run constants, its field of view, disparity scale and face radius;
  • the focus offsets and windows, and the face-exposure gates;
  • the ray pass's aperture and jitter tables;
  • the light-estimate and blur weights.

Studio Light's colour-correction cube is vendor data and stays out: the pass takes it as input, and without it the result is still 99.93–100% within 1 LSB.

If these may not be in the tree, I can replace each constant with a value chosen here and say how it was chosen. The exact-match tests would then become tolerance tests. Tell me which you prefer.

Open question: ce_config.studio_light

The field is appended to struct ce_config without a CE_ABI bump. That is the pattern #13 was asked to fix, so I'd rather settle it here than leave it:

  • A plugin built before the field never reads it.
  • This plugin ships with the filter.
  • But an external plugin built against the new header, run under an older filter, would read past the struct.

The options:

  1. Bump CE_ABI to 2. The companion plugin in aurora-silicon/omarchy-ane is then refused until it is rebuilt.
  2. Pass Studio Light through a companion entry symbol, the way CDN_DMABUF_ENTRY / CDN_QUERY_ENTRY do.

Which do you prefer?

Testing

meson test -Dwerror=true in an Arch Linux ARM container (aarch64, no GPU):

  • 14 OK, including apple, apple-plugin, zoom, config and picture;
  • 5 skipped: gpu and apple-gpu (no Vulkan device), and apple-sdof, apple-studiolight and apple-faceexp (no local captures);
  • gpu-dmabuf fails exactly as on main in that container (no Vulkan device).

Reference checks on macOS (12 captures of one frame):

Pass Result
Portrait (1, 10 and 6 frames) guided filter and temporal filter 100% within 2 ULP; pyramid levels 100% within 1 LSB (level 0 from the frame 99.68%); ray pass 99.83% within 1 LSB; composite 100% within 1 LSB; the whole first frame 99.97% of luma within 1 LSB (worst 8)
Studio Light (no face, a square and a wide face, BT.709 and BT.601 tags) diffuse and light mask 100% within 1 LSB; the whole frame 100% of luma within 1 LSB untagged (worst 2), 99.96–99.97% tagged (worst 4)
Face exposure (one face, and two faces of different brightness) every statistic, coefficient and the light estimate exact

Framing replay on this head: 35 macOS traces through cf_framing.c with the tuning read from an exported framing.conf: 0 state mismatches, viewports within 1e-3 rad at worst. The lens-distortion path hasn't changed since its replay on 29 traces with the real lens (28 identical, one boundary flip at frame 469 of 480). I didn't rerun that here.

Hardware: not run on this head yet. The J815 row is next:

  • install the network and the tuning;
  • run Portrait, Studio Light and Center Stage 60 s each, ON/OFF every 15 s;
  • check that no frames drop at 30 fps;
  • check per-frame time against the budget, with the GPU and CPU paths;
  • check 0 new journal errors and 0 failed units;
  • check that Center Stage stays off with one line when framing.conf is removed.

Assisted-by: LLM (Claude:Opus 5.5)

m0d16l14n1 added a commit to m0d16l14n1/linux that referenced this pull request Nov 21, 2026
…ts, wide LOAD (WIP)

HWX surface names up to 64 bytes and the 'net' procedure, programs with several
outputs and a BAR slot table of any size, ANE_H16_LOAD_WIDE (16 surfaces),
driver minor 2; h16-sections.py handles several outputs. Large temporaries
moved to the heap (stack use). KUnit 239/239.

Run on a J815 (2026-11-21, same change on the gpu-gap kernel): macOS's
personsemantics segmentation (person + skin outputs) ran at 15 runs/s (30 fps preview) for the
camera effects in aurora-silicon/camera-denoise#16.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
m0d16l14n1 added a commit to m0d16l14n1/linux that referenced this pull request Nov 21, 2026
…ts, wide LOAD (WIP)

HWX surface names up to 64 bytes and the 'net' procedure, programs with several
outputs and a BAR slot table of any size, ANE_H16_LOAD_WIDE (16 surfaces),
driver minor 2; h16-sections.py handles several outputs. Large temporaries
moved to the heap (stack use). KUnit 239/239.

Run on a J815 (2026-11-21, same change on the gpu-gap kernel): macOS's
personsemantics segmentation (person + skin outputs) ran at 15 runs/s (30 fps preview) for the
camera effects in aurora-silicon/camera-denoise#16.

Assisted-by: Claude Opus 5.5
Signed-off-by: Nick <nekreshen@yandex.ru>
m0d16l14n1 added a commit to m0d16l14n1/linux that referenced this pull request Nov 21, 2026
…ts, wide LOAD

HWX surface names up to 64 bytes and the 'net' procedure, programs with several
outputs and a BAR slot table of any size, ANE_H16_LOAD_WIDE (16 surfaces),
driver minor 2; h16-sections.py handles several outputs. Large temporaries
moved to the heap (stack use). KUnit 239/239.

Run on a J815 (2026-11-21, same change on the gpu-gap kernel): macOS's
personsemantics segmentation (person + skin outputs) ran at 15 runs/s (30 fps preview) for the
camera effects in aurora-silicon/camera-denoise#16.

Assisted-by: Claude Opus 5.5
Signed-off-by: Nick <nekreshen@yandex.ru>
@RyanTheTide

Copy link
Copy Markdown
Contributor

Early note on this draft (no full review yet):

  1. src/apple/framing.c:7-39, src/apple/framing.h:5-7 and :37, docs/effects.md:63-64 and :85-88: the framing defaults (viewport sizes, deadbands, speeds, accelerations, timings) are copied from tuning data shipped with macOS. Vendor tuning values cannot be in this tree. Either use values tuned here (as face_fov and max_fov already are) and say how they were chosen, or read them at run time from the user's own install, the way the network is, with nothing from it in the source. Drop the names of the macOS framework and file from code and docs.
  2. src/apple/portrait.c:637-645, src/apple/portrait.h:5-8, src/apple/apple_effects.c:5-18 and :50, docs/effects.md:70-74, and the PR body and commit messages: these name macOS frameworks, private classes and internal network/resource names, and cite a value range from a macOS tuning file. Describe the stages by what they do (person/skin segmentation, synthetic fill light) without those names or values.
  3. src/apple/ane_prog.c:78-120, src/apple/ane_prog.h:34-37, tests/test_apple.c:240 and :285, tools/install-apple-effects:47-49: the parser walks the internal layout of the vendor-compiled program (header magic, load commands, segment names, tensor descriptors). That layout knowledge comes from vendor binaries. Get the surface layout from the accel/ane driver's uapi (it already loads the program) instead of parsing the file here.
  4. src/apple/ane_prog.c:185-187: the device is picked by chip_family == ANE_CHIP_H17. Let the driver decide: try the load on each ANE node and keep the one that accepts the program, so any ANE that can run it is used.
  5. src/apple/apple_effects.c:183 (CDN_APPLE_SEG_BGR) and :210 (CDN_APPLE_GPU): behaviour switches through the environment. Settle the network's channel order and remove that switch; make the GPU/CPU choice a [denoise] key or automatic, as zoom-engine is. The path overrides (CDN_APPLE_EFFECTS_DIR, CDN_ZOOM_DEVICE) are better as config keys too.
  6. data/boards.conf:44-58: the J815 entry copies another board's trigger levels because it has the same sensor. Key these on the sensor the camera reports rather than adding a per-board copy.

Assisted-by: LLM (Claude:Opus 5.5)

The default filter is the strongest it can be: the filtered picture
only, the full 3x3 smoothing after the temporal stage and an 8-frame
running mean. On the J815 that read as soft, with smear on motion.
Three [denoise] keys, live on reload:

- strength 0..1: the filtered picture mixed with the camera's
  (1 = filtered only, the default; 0.5 = half the grain comes back);
- spatial 0..1: scales the 3x3 smoothing (0 = none, 1 = default);
- temporal-frames 2..8: caps the running mean (8 = default).

The CPU and GPU backends take them through cdn_frame_info and stay
bit-exact with each other (test_gpu); test_cpu checks that a lighter
setting keeps more of the input. Out-of-range values are refused like
the other keys.

The plugins read the new struct cdn_frame_info fields, so
CDN_BACKEND_ABI and CDN_DMABUF_ABI move to 2: a plugin and a caller built
for different layouts now refuse each other instead of reading past the
end of an older caller's info. The loader test's bad fixture is a plugin
built for ABI 1.

Signed-off-by: Nick <nekreshen@yandex.ru>
…th, tint, sharpen

A last pass over each output frame, after the denoiser and also on the
frames it passes through:

- vignette 0..1: lens-shading compensation, gain 1 + v * r^2 (r = 1 in
  the corners) on luma and on the chroma's distance from neutral, for
  cameras whose ISP tuning leaves the corners dark;
- brightness -1..1, contrast 0.5..2 (around mid grey): one luma LUT;
- saturation 0..2, warmth -1..1 (Cb/Cr toward orange or blue), tint
  -1..1 (toward magenta or green);
- sharpen 0..1: a 3x3 unsharp mask on luma.

All neutral by default, and the pass is skipped then. The gain maps and
the LUT are rebuilt only when the size or the settings change. On the
CPU, NV12 in place; test_picture covers neutral, the vignette's centre,
corners and symmetry, each control's direction and an edge for sharpen.

Signed-off-by: Nick <nekreshen@yandex.ru>
A stream's camera.zoom control (or the new zoom setting) only took effect
through the effects plugin, which crops and scales on the CPU. Without the
plugin, the filter now does the zoom itself as its last stage: a centred
crop scaled back to the stream's size.

On a Mac with a scaler that takes an OUTPUT crop (the M5's MSR23, driver
apple-scaler) the crop and the polyphase scaling run there, and the CPU
copies the frame in and out. Without one, or after any error from it, a
CPU bilinear pass does the same, so the stream keeps going. zoom-engine
picks auto, cpu or scaler.

aurora-camera-zoom-check times both paths on a synthetic frame and
compares their outputs.

Signed-off-by: Nick <nekreshen@yandex.ru>
…ter Stage

A camera_effects plugin (libaurora-camera-effects-apple.so) that runs a person
and skin segmentation network on the Neural Engine through the accel/ane
driver. The network is the user's own: compiled for this Mac's engine by the
user's macOS install and copied from it by install-apple-effects into
<appledir>/segmentation/, with the four host-built sections the driver loads
with it. Nothing of it ships with this package; without it the filter keeps
using the companion plugin.

- The driver decides where the program runs: it is loaded on each ABI 3
  accel node in turn and the first one that accepts it is used. The driver
  validates the program and reports its surfaces (size and direction in
  program order); the plugin gives each surface the shape of the network's
  interface (one RGB input, person and skin planes, 256x192, fp16) and takes
  the strides from the reported sizes. It does not read the program file.
- Portrait (camera.background = blur): masks refined to the frame with a
  guided filter, background blurred by normalized convolution.
- Studio Light (camera.studio-light 0..1): the face lifted by a fill curve
  that spares highlights, the background dimmed; tuned here by eye.
- Center Stage (camera.framing): a framing session (Kalman-filtered faces, a
  face/body tracker, a director, deadband / recentre / return-home, a
  rectangle animator) rendered as a virtual camera rotated to the viewport
  (sub-pixel warp). Its tuning is read at run time from the user's install
  (<appledir>/tuning/framing.conf, "key = value"); without it Center Stage
  stays off. Faces and bodies come from the ISP firmware's detector when the
  camera driver runs it, else from the masks; the lens is the calibration the
  ISP firmware returns when the driver reads it, else a pinhole.
- The full-size passes run on the GPU (Vulkan) when there is a usable
  device, else on the CPU.
- The gain reader takes V4L2_CID_ANALOGUE_GAIN when the sensor has no
  V4L2_CID_GAIN (the T8142 ISP).

CDN_EFFECTS_DEBUG=mask|stats|faces and CDN_EFFECTS_DUMP=<dir> are debugging
aids only; nothing else in the environment changes what the plugin does.

Signed-off-by: Nick <nekreshen@yandex.ru>
Plain C versions of the Portrait (depth-based background blur) and Studio
Light (relighting) renderers, written to match the same passes rendered on
macOS on this Mac, as the reference a GPU port of the effects is checked
against. They are not used by the plugin yet: they need the depth network
on the Neural Engine.

- sdof: the ray pass over a 4-level colour pyramid, the guided upsampling of
  the depth, the composite with its grain, the temporal filter.
- studiolight: the diffuse term from depth, the face light mask, the gain map,
  its blur and upsampling, the relight at full size through a colour cube.
- focus: the focus from faces that both renderers use.
- faceexp: the face exposure statistics behind Studio Light's light estimate.
  Its tuning is read at run time ("face-exposure.conf" from the user's
  install); nothing of it is in the tree.

tests/test_sdof.c, test_studiolight.c and test_faceexp.c check each pass on
its own inputs against a capture of the same pass on macOS (CDN_SDOF_REF,
CDN_SL_REF, CDN_FEXP_REF). The captures are macOS's output for a picture of a
person, so they stay local and the tests skip without them. On 12 captures
(one, ten and six frames; no face, a square and a wide face, BT.709 and
BT.601 tags; one and two faces of different brightness): every face
statistic, coefficient and the light estimate exact; Studio Light's whole
frame 100% of luma within 1 LSB untagged (worst 2) and 99.96-99.97%
tagged (worst 4); Portrait's whole first frame 99.97% of luma within 1 LSB
(worst 8).

Signed-off-by: Nick <nekreshen@yandex.ru>
@m0d16l14n1 m0d16l14n1 changed the title WIP: Apple effects on the ANE: Portrait, Studio Light, Center Stage (J815) Camera effects on the Neural Engine: Portrait blur, Studio Light, Center Stage Nov 22, 2026
@m0d16l14n1

Copy link
Copy Markdown
Author

@RyanTheTide thanks for the early note. All six points are addressed in 64e2479, which is rebased onto #13's f78464c:

  1. Framing tuning
    • framing.c/framing.h are gone.
    • Center Stage's tuning is read at run time from the user's install: <appledir>/tuning/framing.conf, key = value. No value of it is in the tree.
    • Without the file, Center Stage stays off with one log line.
    • install-apple-effects copies tuning/*.conf with the network.
    • The host test uses a made-up tuning.
  2. Names
    • Code, docs, file and test names and the commit messages now describe the stages by function.
    • The network's directory is segmentation/.
    • The reference tests read neutrally named pass files.
  3. Program layout
    • ane_prog.c no longer parses the HWX, and the installer no longer checks its magic.
    • The driver validates it and reports each surface's size and direction.
    • The plugin applies the network's known interface (RGB in; person and skin out; 256x192 fp16) and takes the strides from those sizes.
  4. Device choice: the program is loaded on each ABI 3 node, and the first one that accepts it is used.
  5. Environment
    • The channel order is fixed at R, G, B.
    • GPU vs CPU is automatic.
    • The directory is the build's appledir.
    • The scaler is found by its driver name.
    • The Center Stage A/B switches are gone: the ISP's detections and lens calibration are used when the driver provides them.
    • Only CDN_EFFECTS_DEBUG/CDN_EFFECTS_DUMP remain, for debugging.
  6. Board copy: dropped. The J815 is [sensor:imx958-uw].

The description has a section on how the ports were written, and I'd like your call on it:

  • The framing session and the Portrait/Studio Light reference passes were written by reading the macOS code and its compiled shaders. They were then matched against macOS's own output.
  • The tuning is out of the tree now, but the fixed constants of the reference are still in.
  • If those can't stay, I'll replace them with values chosen here.

There is also an open question on the ce_config.studio_light ABI.

meson test -Dwerror=true (aarch64 container, no GPU) gives 14 OK and 5 skipped. gpu-dmabuf fails exactly as on main. The J815 hardware row is still owed and will follow.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor

Thanks for the rework. Status of the six earlier points at 64e2479:

  • 3 (program layout), 4 (device choice), 5 (environment switches) and 6 (board copy) are fixed. ane_prog.c treats the program as opaque and lets H16_LOAD_WIDE validate it. Each ABI 3 node is tried. Only the debug variables remain. The J815 is [sensor:imx958-uw].
  • 1 is partly fixed. The tunables are now read from framing.conf and Center Stage stays off without it (apple_effects.c:430-432). The fixed constants are still the reference's (cf_framing.c:20-26, and the description says the same of the four reference files).
  • 2 is not fixed in the code. cf_framing.c still names the framework's methods and types: :45, :225, :245, :448, :1144, :1158, :1198, :1234, :1253, :1270, and cf_framing.h:72, :88. The reference passes name the shaders' function constants: sdof.c:367, :422, sdof.h:105, studiolight.c:200, faceexp.c:200.

On your question about how the ports were written: replacing the constants is not enough. By the description and by the code, cf_framing.c, sdof.c, studiolight.c, focus.c, faceexp.c and the parts of portrait.c taken from the same reading were translated from the macOS framework code and its compiled shaders. Some examples:

  • the operation order kept "on purpose" (cf_framing.c:2-5);
  • "Portrait's function constants" and the jitter table "as the reference uses it" (sdof.c:13-37);
  • the renderer's per-run constants "as traced" (studiolight.h:33, studiolight.c:14-24).

The structure and the algorithm come from the vendor's binaries, not only the numbers, so these files can't be in this repository in any form. Please drop them, and those parts of portrait.c, from this PR and from every commit in it (161ba54 and 64e2479 both carry them).

What can stay is an implementation written from a description of the behaviour, by someone who hasn't read the binaries, and tested against macOS output as a black box with tolerances. The pass-level tests (tests/test_sdof.c, test_studiolight.c, test_faceexp.c) compare against captures of the renderer's internal passes and its traced parameters (test_sdof.c:17). They go with the passes. docs/gpu-backend.md:27-28 should stay true of the tree.

The previous head f3170080 is still fetchable from this PR's timeline. Its src/apple/framing.c:7-20 names the vendor's tuning file and lists its values. A force-push doesn't remove it. It needs a purge request to GitHub, or this PR closed and the work reopened from a clean branch.

Other points at this head:

  • Install from the user's machine. tools/install-apple-effects:44-70 copies from any DIR that already holds model.hwx, the four sections and tuning/*.conf. Nothing in the tree builds that directory from the user's own macOS install, and macOS doesn't ship framing.conf in that form. Please add the step that produces it: an extractor that reads the user's macOS volume (or runs on it) and writes DIR, plus how the sections are built (the selftest h16-sections.py, as tools/ane/build-model.py does). Document it in docs/effects.md, with no values in the tree. Without that, the only way to get a DIR is from someone else.

  • camera.focal-length in framing.conf (used at apple_effects.c:438) is a property of the camera. It should come from the ISP calibration or the device tree, not a tuning file.

  • Untrusted conf values.

    • strtof accepts nan/inf (tuning.c:53).
    • cf_tuning_read range-checks only the views, focal length and deadband sizes (cf_framing.c:1382-1386). Speeds, accelerations, durations, padding, deadband min/max, head sizes, movement tolerance and the delays pass unchecked. For example, home.duration = 0 makes u at cf_framing.c:771 inf/NaN. NaN gets through the clamp, and warp.c:26-27 then converts a NaN to int64_t, which is undefined behaviour.
    • A line longer than 255 bytes is split, and its tail is parsed as a line of its own (tuning.c:26, :38).
    • Duplicate and unknown keys are accepted silently.

    Please add: finite checks and per-key ranges, rejection of overlong lines, duplicates and unknown keys, and a finite check on cf_step's result that falls back to cf_reset.

  • ISP lens reply (apple_effects.c:240-316). The plugin decodes the firmware's reply by byte offset. The parsing belongs in the camera driver, behind a defined interface; since that driver isn't pushed yet, this is the moment to define it. As it stands, the polynomial coefficients aren't checked for finite values (only the pixel size and radius are, :291-292).

  • Sensor geometry. The ISP preset table, the 1520x1440 binned input, LENS_CIRCLE 470 and CX/CY 760/720 (apple_effects.c:195-232, :321-330) are one sensor's values compiled in. The comment says they come from the device tree (apple,lens-circle, the board's ISP presets). Please read them from there (or from the driver's crop/bounds) and fall back to the centred crop and pinhole when they're absent.

  • Driver surfaces. These look right: the count is bounded (ane_prog.c:355), direction and size are checked before mapping (:368-391), and exactly three surfaces with dense layouts are required (apple_effects.c:409-414). Two small points:

    • an EINVAL from H16_LOAD_WIDE (a refused program) falls through to the narrow load (ane_prog.c:122-124), which hides the driver's reason;
    • the uapi structs and ioctl numbers are copied locally (ane_prog.c:16-50) for an interface that is still in review (linux#2647), so they need re-checking once it lands.
  • ce_config.studio_light. Bumping CE_ABI (your option 1) is the consistent choice after Tuning keys and a picture stage (vignette, colour, sharpen) #13.

  • Hardware. There is no row at 64e2479 yet. The J815 plan in the description covers what's needed.

Assisted-by: LLM (Claude:Opus 5.5)

m0d16l14n1 added a commit to m0d16l14n1/linux that referenced this pull request Nov 22, 2026
…al outputs, wide LOAD

What the camera's segmentation program needs from the H16 program path:

- HWX surface names up to 64 bytes and procedure names other than 'main';
- programs with several outputs and a BAR slot table of any size;
- ANE_H16_LOAD_WIDE, for programs of up to 16 surfaces (plain H16_LOAD
  refuses more than 4 with -E2BIG, since its arrays cannot report them);
- driver minor 2: plain H16_LOAD now also takes a program with several
  outputs, where minors 0 and 1 refused one, so userspace that relies on
  several outputs checks for minor 2 (the uapi header says so);
- the parser's large temporaries on the heap (stack use);
- h16-sections.py builds the sections of a program with several outputs.

Run on a J815 (2026-11-21): the user's person and skin segmentation
program (two outputs) ran at 15 runs/s for the camera effects in
aurora-silicon/camera-denoise#16.

Signed-off-by: Nick <nekreshen@yandex.ru>
@m0d16l14n1

Copy link
Copy Markdown
Author

Closing this: I'm taking the camera effects out of this repository and continuing them on my own machine, so I won't rework the ports here. As your review asked, the earlier heads of this PR (including f317008) should be purged; I'd appreciate it if a maintainer could ask GitHub for that. Thanks for the reviews.

Assisted-by: LLM (Claude:Opus 5.5)

@m0d16l14n1 m0d16l14n1 closed this Nov 22, 2026
@m0d16l14n1
m0d16l14n1 deleted the j815/apple-effects branch November 22, 2026 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants