Skip to content

Tuning keys and a picture stage (vignette, colour, sharpen) - #13

Closed
m0d16l14n1 wants to merge 2 commits into
aurora-silicon:mainfrom
m0d16l14n1:j815/tuning-picture
Closed

m0d16l14n1 wants to merge 2 commits into
aurora-silicon:mainfrom
m0d16l14n1:j815/tuning-picture

Conversation

@m0d16l14n1

@m0d16l14n1 m0d16l14n1 commented Nov 18, 2026 •

Copy link
Copy Markdown

Two commits for the MacBook Air 15" (M5, J815), each usable on any board:

  1. strength / spatial / temporal-frames. The default filter is as strong as it gets: the filtered picture only, full 3x3 smoothing and an 8-frame mean. On the J815 that looked soft, with smear on motion. These keys scale each part, live on reload:

    • the CPU and GPU backends stay bit-exact with each other (test_gpu);
    • test_cpu checks that a lighter setting keeps more of the input.
  2. Picture stage. A last pass after the denoiser, also on frames it passes through:

    • vignette: lens-shading compensation, for an ISP tuning that leaves the corners dark;
    • brightness, contrast, saturation, warmth, tint;
    • sharpen: a 3x3 unsharp mask.

    Everything is neutral by default, and the pass is skipped then. test_picture covers each control.

Changed on this head (review):

Changed on this head (DjDeveloperr's review):

  • CDN_BACKEND_ABI and CDN_DMABUF_ABI are now 2. The plugins read the three new struct cdn_frame_info fields, so an ABI 1 caller's shorter info would have been read past its end. A plugin and a caller built for different layouts now refuse each other. The loader test's bad fixture is a plugin built for ABI 1 (CDN_BACKEND_ABI - 1). The daemon and the plugins ship in one package, so nothing installed changes. Note for Camera effects on the Neural Engine: Portrait blur, Studio Light, Center Stage #16, which carries older copies of these two commits: its rebase takes the bump.
  • test_config checks the 10 new keys: accepted values, and out-of-range or non-numeric ones refused.

Tested

  • meson test on this head (Arch Linux ARM container, -Dwerror=true, no GPU): 11 OK, gpu skipped; loader, config, picture and cpu pass. gpu-dmabuf fails there the same way on main (10 OK, the same skip and the same failure): the container has no Vulkan device.
  • Earlier, on Arch Linux ARM with CDN_TEST_REQUIRE_GPU=1: 12 OK, 1 skipped.
  • On the J815, the package built from these commits (0.1.0-10) installs, reloads and streams with the GPU temporal backend and the ANE low-light backend.

Still owed (HW row): J815 with this package on the default config, then with strength=0.5 spatial=0.5 temporal-frames=4 and with vignette=0.5 sharpen=0.5, each set and reloaded. Pass: the stream runs, the GPU and CPU backends stay bit-identical on a recorded clip, per-frame time stays inside the 30 fps budget, 0 dropped frames in 5 min, and 0 new journal errors.

All defaults are unchanged, so nothing changes for existing boards unless a key is set.

Part of the J815 bring-up; board results are summarised in aurora-silicon/linux discussion #135.

Assisted-by: LLM (Claude:Opus 5.5)

@m0d16l14n1

Copy link
Copy Markdown
Author

Rebased onto current main (03cd679, after #14's sensor-keyed profiles). New head 68b1882.

Assisted-by: Claude Opus 5.5

@RyanTheTide

Copy link
Copy Markdown
Contributor

Blockers: 1.

Checked the new keys and their range checks in config.c, the CPU/GPU parity of spatial and temporal-frames (CPU ts * spatial_q8 >> 8 and age_max = frames - 1 match filter.comp's par * pc.spatial >> 8 and min(ag+1, pc.agemax); AGE_MAX 7 keeps the default bit-identical), and picture.c (gain map and LUT rebuilds, the neutral skip, the sharpen row ring). The defaults leave the output unchanged.

  1. All three commits (48a7863, b20fec5, 68b1882) have no Signed-off-by and carry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>. Each commit needs the author's Signed-off-by and no agent trailer; put Assisted-by: LLM (<Vendor>:<Model>) as the last line of the PR body instead of the generated-with line.
  2. data/boards.conf:59-73: [apple,j815] duplicates [sensor:imx958-uw] word for word, and the comment on this PR says the J815 kernel already reports imx958-uw. A board-keyed copy only adds a second place to keep in sync and would mask a sensor-keyed change; drop it.

HW row: J815 with this package on the default config, then with strength=0.5 spatial=0.5 temporal-frames=4 and vignette=0.5 sharpen=0.5 set and reloaded: stream runs, the GPU and CPU backends stay bit-identical on a recorded clip, per-frame time stays inside the frame budget at 30 fps, 0 dropped frames in 5 min, 0 new errors in the journal.

Assisted-by: LLM (Claude:Opus 5.5)

@m0d16l14n1 m0d16l14n1 changed the title J815 board data, tuning keys and a picture stage (vignette, colour, sharpen) Tuning keys and a picture stage (vignette, colour, sharpen) Nov 21, 2026
@m0d16l14n1

Copy link
Copy Markdown
Author

@RyanTheTide fixed.

  1. Both commits now carry my Signed-off-by and no agent trailer, and the body ends with Assisted-by: LLM (Claude:Opus 5.5).
  2. The [apple,j815] section (the old first commit) is dropped. [sensor:imx958-uw] covers the J815.

Rebased onto main 03cd679, and the title is updated. In a GPU-less arm64 container, meson test gives 11 OK, gpu skipped, and gpu-dmabuf failing the same way on main (no Vulkan device). The J815 row with the default and the two tuned configs is next.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor

Re-review of f6cdfdd: No blockers.

From the last review:

  1. Missing Signed-off-by and the agent co-author trailer: resolved. ef0ef81 and f6cdfdd carry only Signed-off-by: Nick, and the PR body ends with Assisted-by: LLM (Claude:Opus 5.5).
  2. The board-keyed [apple,j815] copy of [sensor:imx958-uw]: resolved. The section is gone (data/boards.conf now has [apple,j700], [sensor:imx958-uw] and [apple,j616s] only), and nothing else refers to it.

I diffed the old head 68b1882 against f6cdfdd: apart from the dropped board section, the tree is identical. The strength/spatial/temporal-frames keys and the picture stage are the code I already reviewed, so I have nothing new on them.

HW row: J815 with this package on the default config, then with strength=0.5 spatial=0.5 temporal-frames=4 and vignette=0.5 sharpen=0.5 set and reloaded. Pass: the stream runs, the GPU and CPU backends stay bit-identical on a recorded clip, per-frame time stays inside the 30 fps budget, 0 dropped frames in 5 min, 0 new journal errors, and cdn_config_print shows the J815 matched as sensor:imx958-uw.

Assisted-by: LLM (Claude:Opus 5.5)

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial review at head f6cdfdd7affdcf353a0424d3fdba4288e2257da2: Request changes (1 ABI-safety blocker + owed GPU/HW evidence on current head).

Blockers

  1. Plugin ABI break without version bump (src/backend.h:21,35-60,96, src/gpu/denoise_gpu.c:395-402,414,479): This PR extends struct cdn_frame_info from 16 bytes to 24 bytes (appending tuned, temporal_frames, and spatial_q8) and has the dlopen-loaded cdn-backend-gpu.so plugin dereference those new fields in parameters() via cdn_info_spatial_q8(info) and cdn_info_age_max(info), while keeping CDN_BACKEND_ABI == 1 and CDN_DMABUF_ABI == 1. Note src/backend.h explicitly promises that ABI v1 keeps the original ops/frame layouts intact (and tests/test_loader.c tests ABI gating across plugins). If an ABI v1 daemon/caller passing the original 16-byte struct cdn_frame_info loads the new cdn-backend-gpu.so, ops->abi == CDN_BACKEND_ABI succeeds and cdn-backend-gpu.so performs an out-of-bounds stack read past pts_ns. Fix: either bump CDN_BACKEND_ABI (and CDN_DMABUF_ABI) so mismatched binaries are cleanly rejected, or preserve the ABI v1 struct cdn_frame_info layout and deliver tuning via an optional companion symbol (matching the existing CDN_DMABUF_ENTRY / CDN_QUERY_ENTRY pattern).
  2. GPU and J815 hardware evidence still owed on f6cdfdd: On the current head f6cdfdd, test_gpu was skipped (GPU-less container) and the owed J815 hardware row (default config, strength=0.5 spatial=0.5 temporal-frames=4, and vignette=0.5 sharpen=0.5 reloaded live; CPU/GPU bit-exactness on a recorded clip; per-frame timing within 30 fps budget with the new CPU picture.c pass; 0 dropped frames in 5 min; 0 AVC / 0 failed units / 0 new kernel errors; cdn_config_print matching sensor:imx958-uw) has not been posted yet.

Checked & non-blocking notes

  • Compiled & tested locally (aarch64, -Wall -Wextra -Werror): test_picture (ok), test_cpu (PASS, tuned 9.47 dB vs default 18.30 dB), test_config (PASS), src/main.c, and SPIR-V compilation of both src/gpu/filter.comp and src/gpu/noise.comp all succeeded cleanly. CPU/NEON/GPU fixed-point arithmetic parity for spatial_q8 and age_max, and the 3-row ring buffer in src/picture.c, are sound.
  • Attribution & model gating: Clean. Both commits (ef0ef81a, f6cdfdd7) carry human Signed-off-by only, PR body ends with Assisted-by, and [apple,j815] was properly dropped in favor of [sensor:imx958-uw].
  • Non-blocking nits:
    • src/picture.c:160: vignette luma scaling hardcodes limited-range black level 16 (16 + (((v - 16) * g[x]) >> 12)), which darkens corner pixels < 16 on full-range NV12.
    • tests/test_config.c: worth adding parser/range-rejection tests for the 10 new keys in src/config.c:88-98.

Assisted-by: LLM (Google:Gemini 4 Neon)

The default filter is the strongest it can be: the filtered picture
only, the full 3x3 smoothing after the temporal stage and an 8-frame
running mean. On the J815 that read as soft, with smear on motion.
Three [denoise] keys, live on reload:

- strength 0..1: the filtered picture mixed with the camera's
  (1 = filtered only, the default; 0.5 = half the grain comes back);
- spatial 0..1: scales the 3x3 smoothing (0 = none, 1 = default);
- temporal-frames 2..8: caps the running mean (8 = default).

The CPU and GPU backends take them through cdn_frame_info and stay
bit-exact with each other (test_gpu); test_cpu checks that a lighter
setting keeps more of the input. Out-of-range values are refused like
the other keys.

The plugins read the new struct cdn_frame_info fields, so
CDN_BACKEND_ABI and CDN_DMABUF_ABI move to 2: a plugin and a caller built
for different layouts now refuse each other instead of reading past the
end of an older caller's info. The loader test's bad fixture is a plugin
built for ABI 1.

Signed-off-by: Nick <nekreshen@yandex.ru>
…th, tint, sharpen

A last pass over each output frame, after the denoiser and also on the
frames it passes through:

- vignette 0..1: lens-shading compensation, gain 1 + v * r^2 (r = 1 in
  the corners) on luma and on the chroma's distance from neutral, for
  cameras whose ISP tuning leaves the corners dark;
- brightness -1..1, contrast 0.5..2 (around mid grey): one luma LUT;
- saturation 0..2, warmth -1..1 (Cb/Cr toward orange or blue), tint
  -1..1 (toward magenta or green);
- sharpen 0..1: a 3x3 unsharp mask on luma.

All neutral by default, and the pass is skipped then. The gain maps and
the LUT are rebuilt only when the size or the settings change. On the
CPU, NV12 in place; test_picture covers neutral, the vignette's centre,
corners and symmetry, each control's direction and an edge for sharpen.

Signed-off-by: Nick <nekreshen@yandex.ru>
@m0d16l14n1

Copy link
Copy Markdown
Author

@DjDeveloperr thanks, the ABI break was real. Fixed in f78464c9:

  1. ABI: CDN_BACKEND_ABI and CDN_DMABUF_ABI are 2 (in the commit that extends struct cdn_frame_info). A v1 caller and the new cdn-backend-gpu.so now refuse each other in cdn_backend_find(), instead of reading past a 16-byte info. The loader test's BAD_ABI fixture is now a plugin built for the previous ABI (CDN_BACKEND_ABI - 1), so test_loader checks exactly that case. The backend.h comments no longer say "ABI v1 … layouts stay intact".
  2. Config tests: test_config checks the 10 new keys: accepted values, then each out-of-range or non-numeric value refused.

meson test (Arch Linux ARM container, -Dwerror=true, no GPU): 11 OK, gpu skipped, and gpu-dmabuf failing exactly as on main (no Vulkan device; main gives 10 OK with the same skip and failure).

Vignette nit: the luma path assumes limited range, which is what the ISP delivers here. I've left it as it is, and will make it range-aware if a full-range source appears.

The GPU and J815 hardware row on this head is still owed (the set in the description). #16 carries older copies of these two commits and takes the bump when it is rebased.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor

Re-review at f78464c. Nothing new from me.

Both of my earlier points are still resolved: the two commits carry only your Signed-off-by, and the board-keyed J815 section stays dropped. The new parts look right. CDN_BACKEND_ABI and CDN_DMABUF_ABI are now 2 (src/backend.h:26, :102). The loader refuses a plugin with a different ABI (src/backend.c:75) and drops a mismatched DMA-BUF companion (:87). Every in-tree backend takes the macro. The range tests in tests/test_config.c:88-127 cover the new keys. The filter and picture code is unchanged from what I reviewed.

HW row, still owed: J815 on f78464c with the default config, then with strength=0.5 spatial=0.5 temporal-frames=4, then with vignette=0.5 sharpen=0.5, each set and reloaded. Pass: the stream runs, the GPU and CPU backends stay bit-identical on a recorded clip, per-frame time stays inside the 30 fps budget, 0 dropped frames in 5 min, 0 new journal errors, and cdn_config_print shows sensor:imx958-uw.

Assisted-by: LLM (Claude:Opus 5.5)

@m0d16l14n1

Copy link
Copy Markdown
Author

Closing this: I'm taking the J815 camera work out of the upstream trees for now and continuing it on my own machine. Thanks for the reviews.

Assisted-by: LLM (Claude:Opus 5.5)

@m0d16l14n1 m0d16l14n1 closed this Nov 22, 2026
@m0d16l14n1
m0d16l14n1 deleted the j815/tuning-picture branch November 22, 2026 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants