Skip to content

media: apple: isp: H17 stream settings, lens-circle crops and MLVNR for the J813/J815 IMX958 - #2645

Closed
m0d16l14n1 wants to merge 16 commits into
aurora-silicon:integration/aurorafrom
m0d16l14n1:j815/isp-camera-fx
Closed

m0d16l14n1 wants to merge 16 commits into
aurora-silicon:integration/aurorafrom
m0d16l14n1:j815/isp-camera-fx

Conversation

@m0d16l14n1

@m0d16l14n1 m0d16l14n1 commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Draft. Stacked on #2194 (988af2c7, load the module tuning the DT names), on integration/aurora bb9f8ec. Only the last thirteen commits are new. The ANE side of MLVNR is #2647.

1. macOS's stream settings on H17 (cc44a40e, made fixed in 528ec902)

These are the commands and arguments macOS 27.2 sends the same J815 ISP firmware, taken from a hypervisor IPC trace. They started as test parameters. After review they are fixed behaviour on H17:

  • SIFR modes: SIFR skip interval 1 and highlight recovery on.
  • AE and TNR:
    • AE stability 16/8 (other firmware keeps 32/20);
    • an AE gain cap of 96.0;
    • on band-0 setfiles, TNR started with (1, 1, 0) and high strength.
  • Camera LED (T8142 only, 7069144d): CH_STATUS_LED_MBRIGHTNESS_SET is sent at macOS's start value, 529. It is clamped to the range CH_STATUS_LED_THR_GET reports (204..1023 on the J815), so the LED can never be set dark.
  • New commands, as packed structs with size asserts: CH_AE_GAIN_CAP_SET, CH_STATUS_LED_MBRIGHTNESS_SET, CH_STATUS_LED_THR_GET, CH_HR_ENABLE and CH_SIFR_SKIP_INTERVAL_SET. The opcode names are the ones in the firmware's own command table.
  • Removed:
    • the LPDP tuning, which made no measurable difference in the J815 A/B;
    • the bench-only sensor modes 7 and 9.

2. Crop inside the lens circle (669b20c2, 7c9bf6b0)

  • The J813/J815 ultra-wide lens draws an image circle smaller than the sensor. The DT crops take the full sensor width, so they show black corners.
  • Measured on one J815: the 1520x856 16:9 crop goes black at 0.855 and 0.895 of its half-diagonal at the bottom corners, and at 0.96 at the top.
  • The fix: the DT gives the circle as apple,lens-circle, per mille of the input width: 470 on the J813/J815, with the binding in 3faacebd and the dtsi in 9c17988f. Each crop is scaled about its centre until its corners are inside the circle.
    • 16:9 now crops 1242x700 of the binned frame (1.22x).
    • 4:3 crops 1140x856.
    • The portrait crops are unchanged, and each changed crop is logged.
  • Scope: only boards whose DT has the property, the J813 and J815.
    • The J813 is assumed to have the same lens; that is not measured.
    • The circle's centre may vary per unit, so the value is not shifted and leaves margin.

3. MLVNR, the ISP's ML video denoiser on the ANE (e160901d, 40b5c8ec, 2bfcf61b, deb2e98c)

The ISP firmware sends frames to the ANE over two rings; the host only sets up the link.

  • e160901d is Ryan Murray's draft from sithglan/camera/isp-ane-mlvnr (c5b5d15). It adds the ISP side and the in-kernel interface to the ANE driver, include/linux/soc/apple/ane-isp.h. The code is unchanged; only the trailer was dropped, and his Signed-off-by is kept.

  • 40b5c8ec makes it work on T8142 (H17 firmware):

    • Doorbell: the ANE firmware signals the ISP through the ISP CPU software interrupt register, ISP + 0x1668000, bitmask 0x10.
    • CISP_CMD_IPC_ENDPOINT_UNSET2 carries the whole SET2 body again.
    • Per stream: H17 gets CISP_CMD_APPLE_CH_MLVNR_MODE_CONFIG_SET (0xc11f) with macOS's auto mode, after the MBNR set-up: ctrl 2, on above 14x gain, off below 11x, 30 fps and a 30-frame ramp.
    • Loading: the ANE functions are taken with symbol_get() at the first firmware start, so apple-isp never defers on the ANE driver.
  • 2bfcf61b: what the hardware needed beyond that (J815, 2026-11-21):

    • Surface IDs: SHAREDMALLOC surfaces get IDs, and the ANE firmware maps them by ID.
    • Ring addresses: the rings are handed over as IOVA | bit 40.
    • Input shapes: the ISP firmware takes only certain (row stride, rows) pairs. isp_mlvnr_input_ok() carries that table.
      • The binned config 8 is not one of them; the firmware asserted on the input size on the J815.
      • Config 6 (3040x2880) is one.
      • MLVNR is not started outside the table, and a log line says why.
    • Gain thresholds: debugfs apple-isp/mlvnr_gain_{enter,exit} lets a test force MLVNR on.
  • deb2e98c (new, for Ryan's 2026-11-21 note): no mlvnr parameter. At probe, MLVNR is on when all of these hold:

    • the platform knows the ANE doorbell;
    • the presets name the network's sensor mode (apple,mlvnr-sensor-config = <8 6>);
    • the ANE node is enabled;
    • apple/ispane-<board>.afpp is installed. Only its head is read here; the ANE driver loads the whole file.

    Otherwise one info line says which is missing. With MLVNR on, the presets move to the full-resolution mode. If the ANE driver is not loaded at the first firmware start, or the link fails, one line says so and that mode streams without MLVNR.

  • Firmware: the network comes from the user's macOS install. It is not added here.

A trade-off to decide, please. In the J815 A/B, frame-to-frame centre luma noise was 1.29 in the binned config 8 and about 3.0 at full resolution without MLVNR. In auto mode the firmware engages MLVNR only above 14x total gain, and in the Linux runs below it stayed idle (see Testing). So with the network installed, a well-lit scene may now stream full resolution without MLVNR, and that looked noisier than binned. The options:

  1. keep this: decided at probe, as asked;
  2. stay binned until a stream has shown MLVNR working. That means switching sensor modes after probe, which the presets don't do today;
  3. run MLVNR in every light (enter/exit 1x). That is what the forced runs did, and they streamed fine.

The hardware row on this head will measure noise in each case before this leaves draft.

4. Camera light after a firmware crash (6bef90bb)

A crashed ISP firmware is forced off, so it never powers the camera module down. The sensor and its green light then stay on until the next camera use; this was seen on the J815 after an ISP assert.

  • Where the firmware can start again (hw->fw_restart), a halt after an IPC error now boots it once more, without the ANE link, and stops it the normal way.
  • This runs once.
  • On hardware, the light went off.

5. Host tests (b6544fb1)

tools/asahi/isp-preview-flow-tests.py and the isp-tests.py session suite compile driver functions against stubs, and this series broke their build. The new calls get inert stubs, and the checks themselves are unchanged.

Provenance (please read)

The same question as on #2647, and the same answer applies to both PRs. Sources:

  1. A hypervisor IPC trace of macOS 27.2 streaming on this J815. It gave the stream-start commands and arguments in section 1, the SET2/UNSET2 bodies and ring addresses, the auto-mode words, and the 984-of-987 frame count.
  2. macOS's kernel log on this J815. It gave the ISP doorbell address and the surface allocations.
  3. A reading of the compiled macOS camera and ANE drivers in the 27.2.1 kernelcache, with r2. It gave the endpoint configuration and the meaning of the SHAREDMALLOC ID answer.
  4. A reading of the ISP firmware image (26B131) for its MLVNR input check. That is the source of the isp_mlvnr_inputs table.
  5. The J815 under Linux: every one of these was then run against the firmware (Testing).

The class names, the firmware's source-file references and the IOSurface wording are gone from the code and the commit messages. Points 3 and 4 still mean this is not clean-room. Ryan, your decision on #2647 covers this PR too.

Testing

  • ISP KUnit (arm64, QEMU, LLVM) on this head: 62 tests, 61 pass.
    • The one failure, isp_test_sensor_sysfs, also fails on plain bb9f8ec.
    • isp_test_temporal_filter_band0 was fixed in 528ec902: it still expected the old (1, 0, 0) start record.
  • Host tests: isp-tests.py, isp-preview-tests.py and isp-preview-flow-tests.py pass with clang 19 and ASan/UBSan, on this head and on bb9f8ec.
  • checkpatch --strict: clean on deb2e98c. Earlier commits keep a few line-length, alignment and &-spacing notes, which I'll clean before this leaves draft.
  • DTBs: J813 and J815 build.
  • Hardware: nothing has run on this head yet. Earlier runs used the parameter form (sensor_config=6, mlvnr=1):
    • MLVNR forced on (J815, 2026-11-21, with the accel/ane: run vendor-compiled camera programs: long HWX names, several outputs, wide LOAD #2647 ANE side): two 20 s runs, each 577 frames at 28 fps. There was no ISP assert and no ANE command timeout. The ANE reported about three program runs per frame (1725 per run), and all 33 ISP surfaces were freed at stop.
    • Auto mode (14x / 11x): at 8x analogue gain indoors, MLVNR stayed idle. macOS sent the same auto-mode words and applied MLVNR on 984 of 987 frames. Either that session's total gain was above 14x, or Linux misses something that engages it. This is open.
    • The lens crop: the black corners are gone. dmesg shows crop 1520x856+0+292 -> 1242x700 (inside the lens circle).
    • The commit-1 settings: every mode streamed at 30 fps, and the firmware accepted SIFR + HR.
  • Owed on this head:
    • J815 with and without the network installed (one info line when it is absent);
    • noise in each mode (the trade-off above);
    • 0 SELinux denials, 0 failed units and 0 new kernel errors;
    • a J813 row.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor

An early note while this is WIP. Merged code should pick its behaviour itself, so before this leaves draft the six TEST: parameters (sensor_config, presets_full, hr, macos_tune, led_brightness, lpdp_macos) need to become fixed behaviour. Where a choice really belongs to the sensor or lens (the IMX958 mode, lens_circle/lens_center_y), it could be a property of the camera's DT node, like the presets already are. sensor_config and presets_full are 0444 and alias each other, so one of them can go now.

Clamping led_brightness to the firmware's own range, so the LED can't be set dark, is right for a privacy indicator. I'd keep that guarantee when it becomes a fixed value.

Assisted-by: LLM (Claude:Opus 5.5)

@m0d16l14n1 m0d16l14n1 changed the title WIP: media: apple: isp: macOS-parity knobs and lens-circle crops for the J813/J815 IMX958 WIP: media: apple: isp: macOS-parity knobs, lens-circle crops and MLVNR for the J813/J815 IMX958 Nov 21, 2026
@m0d16l14n1
m0d16l14n1 force-pushed the j815/isp-camera-fx branch 2 times, most recently from e305edc to 9b2d189 Compare November 21, 2026 10:40
@m0d16l14n1 m0d16l14n1 changed the title WIP: media: apple: isp: macOS-parity knobs, lens-circle crops and MLVNR for the J813/J815 IMX958 media: apple: isp: macOS-parity knobs, lens-circle crops and MLVNR for the J813/J815 IMX958 Nov 21, 2026
@m0d16l14n1

m0d16l14n1 commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor Author

J815 run, 2026-11-21: MLVNR streams (this PR's ISP side as a module, the ANE side from #2647 as the boot driver, sensor config 6)

Try Gain thresholds Stream IO_T2H reports taken / left FW requests ISP surfaces mapped / freed Doorbells by bit 0..7
1 (run A) 1x (forced) 20 s, 577 frames, 28 fps 1725 / 0 1759 33 / 33 0 1 0 0 0 4 1116 0
2 (run A) 14x / 11x (macOS auto) 20 s, 578 frames 0 / 0 (idle; analogue gain 8x) 66 33 / 33 0 1 0 0 0 5 0 0
1 (run B) 1x (forced) 20 s, 577 frames, 28 fps 1725 / 0 1791 33 / 33 0 1 0 0 0 5 1113 0
2 (run B) 14x / 11x (macOS auto) 20 s, 578 frames 0 / 0 34 33 / 33 0 1 0 0 0 4 0 0
  • In every try: no ANE or ISP assert, DART error status 0 on all three DARTs, and a clean link down.
  • About three reports per frame: the network has 11 procedures, and the ISP runs several of them per frame.
  • The first report is a standard event (0x0300, 0x60 bytes): UUID 0x10000001a, program 2, process 2, procedure 6, state 0, destination 1.
  • Before this change, the same set-up asserted 6.4 s in (CRPCClient.cpp, 303: out of command cache!).

Noise: the two noise runs on this boot measured the camera-denoise output in auto mode, where MLVNR was idle. Their centre luma noise was 0.32 and 0.30, at different exposures, so they say nothing about MLVNR. A raw-camera A/B (off / forced on / off / forced on) is the next run.

ISP firmware lines in the forced try: MLVNR Mode Config ctrl=2,Thres Enter:Exit=256:256,FPS Enter:Exit=30:30,RampDownFrames=30, NetInit MLVNR 0x2060310 with 11 procedures, and Load Program, programId 2 processId 2, 2.7ms. In auto mode the firmware gets 3584:2816.

Open: macOS sends the same auto-mode words (CmdMLVNRModeConfigSet: ctrl 2, TotalGainThreshold 3584/2816, 30/30 fps, ramp 30), and yet in the traced macOS session MLVNR was applied on 984 of 987 frames (BWTemporalFilterNode count). So either that session's total gain was above 14x, or Linux misses something that engages it. That is open: the 8x above is the analogue gain only (it sat at 2048, likely its cap), not the total gain the firmware compares.

@m0d16l14n1
m0d16l14n1 marked this pull request as ready for review November 21, 2026 10:57
@m0d16l14n1
m0d16l14n1 marked this pull request as draft November 21, 2026 11:02
@m0d16l14n1

Copy link
Copy Markdown
Contributor Author

Thanks. I took this out of draft too early, before acting on your note, and it is back in draft. The knobs are gone in three commits:

  • 05da2e92 (dt-bindings) adds two optional sensor-presets properties:
    • apple,lens-circle: the image-circle radius, per mille of the input width;
    • apple,mlvnr-sensor-config = <binned full>: the binned mode of the presets, and the full-resolution mode at twice its size that the MLVNR network takes.
  • b6feab18 (driver):
    • hr, macos_tune and led_brightness are fixed behaviour on H17: highlight recovery with skip interval 1 in SIFR modes, AE stability 16/8, gain cap 96.0, TNR (1, 1, 0) with high strength on band-0, and the LED at macOS's start value, 529. The LED is still clamped to the firmware's CH_STATUS_LED_THR_GET range, so it can't be set dark. Other firmware keeps its old values.
    • lpdp_macos and its 0x18-byte command are removed: the J815 A/B showed no difference.
    • sensor_config and presets_full are replaced by the DT pair. With apple_isp.mlvnr, the binned presets move to the full mode; without it they keep the DT's. mlvnr is now read at probe (0444), because the presets depend on it. The bench-only configs 7 and 9 are dropped.
    • lens_circle now comes from the DT. lens_center_y was never used and is removed.
    • The MLVNR gain thresholds moved to debugfs (apple-isp/mlvnr_gain_{enter,exit}), created only with mlvnr, for tests that force it on. The ring address form is fixed to the one macOS uses.
  • d99f2927 (dts): the J813/J815 ISP dtsi gets apple,lens-circle = <470> and apple,mlvnr-sensor-config = <8 6>.

mlvnr itself stays an opt-in parameter for now: it needs the ANE driver from #2647 and Apple's network from the user's macOS install. This builds cleanly with W=1, and the ISP host tests pass. I'll post the hardware run of this exact behaviour here before asking for review again.

Assisted-by: Claude Opus 5.5

@RyanTheTide

Copy link
Copy Markdown
Contributor

Thanks, that's the shape. The lens circle and MLVNR sensor modes are now in the sensor presets, the macOS-parity values are fixed behaviour, and the LED starts at 529, which is inside the firmware's 204–1023 range, so it can't go dark.

One left: mlvnr (0444). MLVNR already depends on things the system can see: the ANE driver bound with the ISP link, and apple/ispane-<board>.afpp present. Could it switch on when both are there (a failed request_firmware just leaves it off, with one info line), rather than a parameter?

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor

@m0d16l14n1, a tidy-up pass over the open PRs here. These of yours need something:

If any of these are no longer needed, closing them helps; otherwise a merge or rebase onto integration/aurora is usually all they need. Thanks!

@m0d16l14n1

Copy link
Copy Markdown
Contributor Author

Please wait, I will finish

m0d16l14n1 and others added 16 commits November 22, 2026 21:36
Some Macs pair a sensor with a camera module that has its own tuning
file. Let the ISP node name it, e.g. "2226_02" for the J815's IMX958.

Signed-off-by: Nick <nekreshen@yandex.ru>
The J813 and the J815 (MacBook Air 13" and 15" M5) both have an IMX958
sensor, and both have run its default 2224_01 setfile. The J815's camera
module has its own tunings, 2226_01 and 2226_02, in the H17-native form
(lens-shading records without the xx24 stride). With 2224_01 the J815's
raw picture is dark in the corners. Corner/centre brightness on one
J815, raw camera, same scene, each file loaded into a freshly started
firmware:

  2224_01  0.50
  2226_01  0.63
  2226_02  0.85

Load the tuning the ISP node's apple,module-tuning names. A board
without the property, or with a name that is not a setfile of its
sensor, keeps the sensor's default. The 2226 files are not in every
vendor firmware set yet, so a missing one falls back to 2224_01 instead
of leaving the camera untuned.

Tested on J815, with the file picked by a module parameter at the time:
all three files load and stream (no command timeouts), 2226_02 also
across several camera restarts.

KUnit: isp_test_module_tuning (match, another sensor, unknown names)
and isp_test_setfile_fallbacks (each fallback is the same sensor, no
larger, and final).

Signed-off-by: Nick <nekreshen@yandex.ru>
The J815's IMX958 module runs 2226_02: with the sensor's default
2224_01 its corners read half the centre's brightness, with 2226_02
0.85 of it.

Signed-off-by: Nick <nekreshen@yandex.ru>
Add the ISP commands macOS 27.2 sends at a J815 stream start (from a
hypervisor IPC trace) and test parameters that send them, for an A/B on
the J815: binned SIFR with highlight recovery (config 9), the AE and
gain settings, and the camera status LED. A later commit in this series
makes the useful ones fixed behaviour and drops the parameters.

Signed-off-by: Nick <nekreshen@yandex.ru>
…nned SIFR default

The J813/J815 ultra-wide lens draws an image circle smaller than the
sensor; the DT crops take the full sensor width and show black corners
(measured on a J815: the 16:9 crop goes dark at 0.855-0.96 of its
half-diagonal). lens_circle (per mille of the input width, default 470)
and lens_center_y scale a crop about its centre until its corners are
inside the circle: 1920x1080 and 1280x720 crop 1242x700 of the binned
1520x1440 frame instead of 1520x856 (1.22x).

sensor_config now defaults to 9 (binned + SIFR + highlight recovery):
config 6 measured 2.4x the centre luma noise of the binned modes.

Signed-off-by: Nick <nekreshen@yandex.ru>
Config 9 + HR measured 1.52 centre luma noise against 1.29 for config 8
on the J815 A/B; for a grain complaint keep the DT's binned mode and
leave 9 as a knob. Log each crop the lens circle changes, so a run shows
that it applied.

Signed-off-by: Nick <nekreshen@yandex.ru>
On H16-family ISPs (J616s and J700) macOS runs the camera low-light ML
noise reduction, ISP algorithm 0x143d, on the platform neural engine.
The ANE driver loads the per-board network into the ANE and both
firmwares exchange work over a pair of 16 KiB rings. The ISP driver
allocates the rings, maps them into its IOMMU, passes them to the ANE
driver and tells its firmware with CISP_CMD_IPC_ENDPOINT_SET2.

Add the ISP side and the in-kernel interface to the ANE driver
(include/linux/soc/apple/ane-isp.h):

- find the ANE from the "apple,ane" phandle and name the network after
  the board (apple/ispane-<board>.afpp);
- after the firmware starts, allocate the ring pair, ask the ANE driver
  to load the network and set up its endpoint, then send 0x300c;
- unset the endpoint (0x300d) before the firmware stops;
- per stream, enable 0x143d only when the endpoint is live. Enabling it
  without one stalls the stream, so a missing network, ANE or doorbell
  leaves the camera exactly as before.

The doorbell each side uses to signal the other is not known yet, so no
SoC sets ane_doorbell_known and nothing changes at runtime until it is.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
… auto mode

Make Ryan Murray's ISP<->ANE draft work on the M5 (T8142, H17 firmware),
opt-in behind apple_isp.mlvnr=1 (default off):

- the ANE firmware's doorbell to the ISP is the ISP CPU software
  interrupt register, ISP + 0x1668000, bitmask 0x10. macOS 27.2 logs it on
  the J815, and it is the first
  window of the ANE's address filter. The ISP's own filter already admits
  the ANE doorbell (0x421660000);
- CISP_CMD_IPC_ENDPOINT_UNSET2 carries the whole SET2 body again, as the
  macOS trace shows, not a 0xc-byte one;
- H17 does not take algorithm 0x143d: per stream it gets
  CISP_CMD_APPLE_CH_MLVNR_MODE_CONFIG_SET (0xc11f) with macOS's auto mode
  (ctrl 2, on above 14x gain, off below 11x, 30 fps, 30-frame ramp),
  after the MBNR set-up where macOS sends it;
- the ANE node comes from apple,ane or, without it, hw->ane_compatible;
- the ANE functions are taken with symbol_get() at the first firmware
  start with mlvnr=1, so apple-isp loads and probes without the ANE driver
  and never defers on it;
- ane-isp.h declares the interface unconditionally for symbol_get().

Every failure leaves MLVNR off and the camera as before. Not yet run on
hardware.

Signed-off-by: Nick <nekreshen@yandex.ru>
A crashed firmware is forced off: the stop command times out ("coproc
not in WFI"), so the firmware never sends the camera module its
power-down. The sensor stays powered, and its green status light stays
on until the next camera use. Seen on J815 (2026-11-21) after an ISP
firmware assert.

On hardware whose firmware starts again (hw->fw_restart), a halt after
an IPC error now schedules fw_recover_work. It boots the firmware once
more without the neural engine link and stops it the normal way, which
powers the module down. It runs once: a second crash is left as it is.
On J815 the light went off after the crash, and the log said "crashed
firmware: restarted and stopped it to power the camera down".

Signed-off-by: Nick <nekreshen@yandex.ru>
…hapes, gain knobs

What MLVNR needed on J815 hardware (2026-11-21, rounds 2-5):

- SHAREDMALLOC surfaces get IDs, which the ANE firmware maps them by. The rings are handed to the firmware at
  the address form macOS uses (ane_ring_fw).
- The ISP firmware's input check (26B131) takes only
  certain (row stride, rows) pairs. The binned mode (config 8, 1520 wide,
  rows of 0xc00 bytes) is not one of them, and the firmware asserted
  on the input size. The full-resolution mode macOS streams
  (config 6, 3040x2880, rows of 0x17c0) is one. isp_mlvnr_input_ok() has
  the firmware's table, and MLVNR is not started in a mode outside it;
  a log line says so instead.
- mlvnr_gain_enter / mlvnr_gain_exit (1/256 units, default 14x / 11x,
  macOS's values) set the auto-mode thresholds, so a test can force
  MLVNR on in any light.
- While the firmware is restarted after a crash, the ANE link stays
  down.

On hardware in config 6 the link works: 33 surfaces mapped, NetInit (11
procedures), LoadProgram in 0.9 ms, MLVNRProc started. The ANE then
stalled on unread reports (fixed on the ANE side), so no frame has been
denoised yet. KUnit: isp_test_mlvnr_inputs.

Signed-off-by: Nick <nekreshen@yandex.ru>
The host tests compile driver functions against fake boundaries, so they
stopped building with this series:

- isp-preview-flow-tests: isp_ch_configure_capture() now reads the
  knobs that match macOS (isp_lpdp_macos, isp_macos_tune), the remapped
  preset flag, and calls the SIFR, camera LED and MLVNR helpers. They
  get inert stubs; the order this test checks is unchanged.
- isp-tests session: the firmware stop path calls apple_isp_ane_stop()
  and the boot apple_isp_ane_start(), and reads the crash recovery work
  and flag. The fixtures have no neural engine and no fw_restart
  firmware, so recovery is never scheduled and the idle-work check
  stays as strict as before.

All ISP suites pass again with clang 19 and ASan/UBSan, and the four
session negative controls still fail on their CHECK lines.

Signed-off-by: Nick <nekreshen@yandex.ru>
…R sensor mode

Two optional properties of the sensor-presets node, for what belongs to
the camera module rather than to the driver:

- apple,lens-circle: the radius of the image circle the lens draws on
  the sensor, per mille of each preset's input width. The J813/J815
  ultra-wide lens is black outside it, so the crops are kept inside.
- apple,mlvnr-sensor-config: the binned sensor configuration of the
  presets and the full-resolution one, at twice its input size, whose
  frames the ISP's ML video noise reduction network takes.

Signed-off-by: Nick <nekreshen@yandex.ru>
Ryan Murray's review: merged code picks its behaviour itself, and what
belongs to the sensor or the lens goes in the camera's DT node. So the
TEST module parameters go:

- hr, macos_tune, led_brightness: what macOS 27.2 sends the H17 firmware
  on the J815 is now fixed behaviour on H17. That is highlight recovery
  with SIFR skip interval 1 in SIFR modes; AE stability 16/8 and an AE
  gain cap of 96.0; TNR (1, 1, 0) with high strength on band-0 setfiles;
  and the camera LED at macOS's start brightness, 529. The LED value is
  still clamped to the range the firmware reports, so the privacy
  indicator can never be set dark. Other firmware keeps its old values.
- lpdp_macos: the 0x18-byte LPDP tuning made no measurable difference
  in the J815 A/B, so it goes with its command; the driver's (1, 15)
  stays.
- sensor_config, presets_full: the full-resolution mode is only wanted
  for MLVNR, whose network takes no binned frame. With apple_isp.mlvnr,
  the presets of the binned mode in apple,mlvnr-sensor-config move to
  its full mode (input and crops doubled); without it they keep the
  DT's. mlvnr is now read at probe (0444), since the presets depend on
  it. Configs 7 and 9 were bench modes only and go.
- lens_circle, lens_center_y: the radius comes from apple,lens-circle.
  The centre shift was never used and goes.
- mlvnr_gain_enter/_exit move to debugfs (apple-isp/, with mlvnr only),
  for tests that force MLVNR on. mlvnr_ring_tag is fixed on, the form
  macOS uses.

Signed-off-by: Nick <nekreshen@yandex.ru>
…MLVNR mode

The ultra-wide IMX958 module's lens draws an image circle of radius
0.47 of the sensor width (measured on a J815; the J813 is assumed to
have the same lens), and MLVNR takes config 6 (3040x2880 with SIFR, the
mode macOS streams), not the presets' binned config 8.

Signed-off-by: Nick <nekreshen@yandex.ru>
macOS's start brightness (529) was traced on the J815 only. Other H17
firmware keeps its own default until it is measured there: a per-SoC
hw->led_brightness, set for T8142, replaces the H17-wide check.

Signed-off-by: Nick <nekreshen@yandex.ru>
Drop the mlvnr module parameter. At probe, before the presets are read,
MLVNR is on when:

- the platform's ANE doorbell is known (hw->ane_doorbell_known);
- the sensor presets name the network's sensor mode
  (apple,mlvnr-sensor-config);
- the ANE node is enabled in the DT;
- the board's network, apple/ispane-<board>.afpp, is installed (a
  partial read of its head; the ANE driver loads it whole).

Otherwise one info line says which is missing, and the camera works as
before. The presets move to the network's sensor mode only when MLVNR is
on. The ANE driver itself is still taken at the first firmware start, as
the ANE probes on its own schedule. If it is not loaded then, or the
link fails, one line says so and that mode streams without MLVNR.

The gain-threshold debugfs files stay, and they exist only with MLVNR on.

Signed-off-by: Nick <nekreshen@yandex.ru>
@m0d16l14n1 m0d16l14n1 changed the title media: apple: isp: macOS-parity knobs, lens-circle crops and MLVNR for the J813/J815 IMX958 media: apple: isp: H17 stream settings, lens-circle crops and MLVNR for the J813/J815 IMX958 Nov 22, 2026
@m0d16l14n1

Copy link
Copy Markdown
Contributor Author

@RyanTheTide I force-pushed deb2e98c, rebased onto the new #2194 (988af2c7) on bb9f8ec. It stays a draft until the hardware row on this head.

  • mlvnr parameter: gone (deb2e98c). At probe, MLVNR is on only when all of these are true: the platform knows the ANE doorbell, the presets name the network's sensor mode, the ANE node is enabled, and apple/ispane-<board>.afpp is installed. Otherwise one info line says what's missing. If the ANE driver isn't loaded at the first firmware start, that is also one line, and MLVNR stays off.
  • One consequence, which the body describes: with the network installed, the presets move to full resolution. In auto mode the firmware engages MLVNR only above 14x gain. So a well-lit scene can stream full resolution without MLVNR, and on the J815 that measured noisier than binned (about 3.0 against 1.29). The body lists three options, and I'd like your view.
  • Names: the class names, firmware source references and IOSurface wording are gone from the code and every commit message, folded into the commits that introduced them. ane-isp.h is now byte-identical to accel/ane: run vendor-compiled camera programs: long HWX names, several outputs, wide LOAD #2647's.
  • Trailers: none in any commit. Your draft (e160901d) keeps your authorship and Signed-off-by; only its Assisted-by: LLM line was dropped.
  • Provenance: the same answer as on accel/ane: run vendor-compiled camera programs: long HWX names, several outputs, wide LOAD #2647, with one extra source here. The MLVNR input table was read from the ISP firmware image. Your decision there covers this PR too.
  • Tests:
    • ISP KUnit: 62 tests, 61 pass. The one failure, isp_test_sensor_sysfs, also fails on bb9f8ec. A stale temporal_filter_band0 expectation is fixed.
    • The three tools/asahi ISP host tests pass.
    • The J813 and J815 DTBs build.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor

Thanks for the rework. The parameter is gone, MLVNR is decided at probe from what the platform and the installed files actually provide (one info line when something is missing), the names are out of the code and the messages, and the commits carry no trailers.

Provenance is the blocker, as on #2647, and here it goes further: the description says the MLVNR input table (isp_mlvnr_inputs[], the dimensions block added around line 157-182 of the diff) was read out of the ISP firmware image. Values taken from the firmware image can't be in this tree, in this PR or in its history, whatever form they take. Please drop the table. Either ask the firmware at runtime for what it accepts, or take the dimensions from a hypervisor trace of what macOS actually feeds the network, with a short write-up of the trace, rewritten from that write-up. As on #2647, the ANE/ISP command layouts and the SHAREDMALLOC answers need the same treatment. Since the earlier heads with the table are still fetchable from this PR's timeline, a fresh branch and PR would be cleaner than another force-push.

On the trade-off: option 1 shouldn't change what a well-lit stream looks like today, and from your J815 numbers it does (1.29 binned vs ~3.0 full resolution without MLVNR). I'd take whichever of 2 or 3 your noise row shows keeps well-lit output at least as clean as today's binned stream. If 3 streams fine in every light and measures cleaner, it's the simpler one.

Assisted-by: LLM (Claude:Opus 5.5)

@m0d16l14n1

Copy link
Copy Markdown
Contributor Author

@RyanTheTide thanks. Following your note, I'm closing this in favour of a fresh branch: #2843. None of its history has the MLVNR code or the firmware-derived input table. It is a draft on #2194 (988af2c7) with the parts that don't depend on MLVNR:

  • macOS's H17 stream settings as fixed behaviour (AE 16/8, gain cap 96.0, band-0 TNR (1, 1, 0) with high strength, the LED at 529 clamped to the firmware's range on T8142);
  • the lens-circle crops (apple,lens-circle);
  • the camera light after a firmware crash;
  • the host-test stubs.

Highlight recovery and the SIFR skip interval are gone there too. They only apply in a SIFR mode, and the presets stay binned, so they would be dead code until a full-resolution mode comes back.

One more provenance detail, disclosed in #2843's body: I first named its three new commands from the name table behind the firmware's command log line, extracted from the image. Each name also appears in the firmware's runtime log (Linux dmesg or macOS's log), and the code now cites only that.

MLVNR comes back as its own PR, rewritten from a hypervisor trace and its write-up, as on #2647. Your trade-off choice goes with it: option 2 or 3, picked by the noise row.

Assisted-by: LLM (Claude:Opus 5.5)

@m0d16l14n1 m0d16l14n1 closed this Nov 22, 2026
@m0d16l14n1
m0d16l14n1 deleted the j815/isp-camera-fx branch November 22, 2026 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants