Repository navigation
media: apple: isp: H17 stream settings, lens-circle crops and MLVNR for the J813/J815 IMX958 - #2645
m0d16l14n1 wants to merge 16 commits into
Conversation
|
An early note while this is WIP. Merged code should pick its behaviour itself, so before this leaves draft the six Clamping Assisted-by: LLM (Claude:Opus 5.5) |
e305edc to
9b2d189
Compare
|
J815 run, 2026-11-21: MLVNR streams (this PR's ISP side as a module, the ANE side from #2647 as the boot driver, sensor config 6)
Noise: the two ISP firmware lines in the forced try: Open: macOS sends the same auto-mode words ( |
|
Thanks. I took this out of draft too early, before acting on your note, and it is back in draft. The knobs are gone in three commits:
Assisted-by: Claude Opus 5.5 |
|
Thanks, that's the shape. The lens circle and MLVNR sensor modes are now in the sensor presets, the macOS-parity values are fixed behaviour, and the LED starts at 529, which is inside the firmware's 204–1023 range, so it can't go dark. One left: Assisted-by: LLM (Claude:Opus 5.5) |
|
@m0d16l14n1, a tidy-up pass over the open PRs here. These of yours need something:
If any of these are no longer needed, closing them helps; otherwise a merge or rebase onto integration/aurora is usually all they need. Thanks! |
|
Please wait, I will finish |
Some Macs pair a sensor with a camera module that has its own tuning file. Let the ISP node name it, e.g. "2226_02" for the J815's IMX958. Signed-off-by: Nick <nekreshen@yandex.ru>
The J813 and the J815 (MacBook Air 13" and 15" M5) both have an IMX958 sensor, and both have run its default 2224_01 setfile. The J815's camera module has its own tunings, 2226_01 and 2226_02, in the H17-native form (lens-shading records without the xx24 stride). With 2224_01 the J815's raw picture is dark in the corners. Corner/centre brightness on one J815, raw camera, same scene, each file loaded into a freshly started firmware: 2224_01 0.50 2226_01 0.63 2226_02 0.85 Load the tuning the ISP node's apple,module-tuning names. A board without the property, or with a name that is not a setfile of its sensor, keeps the sensor's default. The 2226 files are not in every vendor firmware set yet, so a missing one falls back to 2224_01 instead of leaving the camera untuned. Tested on J815, with the file picked by a module parameter at the time: all three files load and stream (no command timeouts), 2226_02 also across several camera restarts. KUnit: isp_test_module_tuning (match, another sensor, unknown names) and isp_test_setfile_fallbacks (each fallback is the same sensor, no larger, and final). Signed-off-by: Nick <nekreshen@yandex.ru>
The J815's IMX958 module runs 2226_02: with the sensor's default 2224_01 its corners read half the centre's brightness, with 2226_02 0.85 of it. Signed-off-by: Nick <nekreshen@yandex.ru>
Add the ISP commands macOS 27.2 sends at a J815 stream start (from a hypervisor IPC trace) and test parameters that send them, for an A/B on the J815: binned SIFR with highlight recovery (config 9), the AE and gain settings, and the camera status LED. A later commit in this series makes the useful ones fixed behaviour and drops the parameters. Signed-off-by: Nick <nekreshen@yandex.ru>
…nned SIFR default The J813/J815 ultra-wide lens draws an image circle smaller than the sensor; the DT crops take the full sensor width and show black corners (measured on a J815: the 16:9 crop goes dark at 0.855-0.96 of its half-diagonal). lens_circle (per mille of the input width, default 470) and lens_center_y scale a crop about its centre until its corners are inside the circle: 1920x1080 and 1280x720 crop 1242x700 of the binned 1520x1440 frame instead of 1520x856 (1.22x). sensor_config now defaults to 9 (binned + SIFR + highlight recovery): config 6 measured 2.4x the centre luma noise of the binned modes. Signed-off-by: Nick <nekreshen@yandex.ru>
Config 9 + HR measured 1.52 centre luma noise against 1.29 for config 8 on the J815 A/B; for a grain complaint keep the DT's binned mode and leave 9 as a knob. Log each crop the lens circle changes, so a run shows that it applied. Signed-off-by: Nick <nekreshen@yandex.ru>
On H16-family ISPs (J616s and J700) macOS runs the camera low-light ML noise reduction, ISP algorithm 0x143d, on the platform neural engine. The ANE driver loads the per-board network into the ANE and both firmwares exchange work over a pair of 16 KiB rings. The ISP driver allocates the rings, maps them into its IOMMU, passes them to the ANE driver and tells its firmware with CISP_CMD_IPC_ENDPOINT_SET2. Add the ISP side and the in-kernel interface to the ANE driver (include/linux/soc/apple/ane-isp.h): - find the ANE from the "apple,ane" phandle and name the network after the board (apple/ispane-<board>.afpp); - after the firmware starts, allocate the ring pair, ask the ANE driver to load the network and set up its endpoint, then send 0x300c; - unset the endpoint (0x300d) before the firmware stops; - per stream, enable 0x143d only when the endpoint is live. Enabling it without one stalls the stream, so a missing network, ANE or doorbell leaves the camera exactly as before. The doorbell each side uses to signal the other is not known yet, so no SoC sets ane_doorbell_known and nothing changes at runtime until it is. Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
… auto mode Make Ryan Murray's ISP<->ANE draft work on the M5 (T8142, H17 firmware), opt-in behind apple_isp.mlvnr=1 (default off): - the ANE firmware's doorbell to the ISP is the ISP CPU software interrupt register, ISP + 0x1668000, bitmask 0x10. macOS 27.2 logs it on the J815, and it is the first window of the ANE's address filter. The ISP's own filter already admits the ANE doorbell (0x421660000); - CISP_CMD_IPC_ENDPOINT_UNSET2 carries the whole SET2 body again, as the macOS trace shows, not a 0xc-byte one; - H17 does not take algorithm 0x143d: per stream it gets CISP_CMD_APPLE_CH_MLVNR_MODE_CONFIG_SET (0xc11f) with macOS's auto mode (ctrl 2, on above 14x gain, off below 11x, 30 fps, 30-frame ramp), after the MBNR set-up where macOS sends it; - the ANE node comes from apple,ane or, without it, hw->ane_compatible; - the ANE functions are taken with symbol_get() at the first firmware start with mlvnr=1, so apple-isp loads and probes without the ANE driver and never defers on it; - ane-isp.h declares the interface unconditionally for symbol_get(). Every failure leaves MLVNR off and the camera as before. Not yet run on hardware. Signed-off-by: Nick <nekreshen@yandex.ru>
A crashed firmware is forced off: the stop command times out ("coproc
not in WFI"), so the firmware never sends the camera module its
power-down. The sensor stays powered, and its green status light stays
on until the next camera use. Seen on J815 (2026-11-21) after an ISP
firmware assert.
On hardware whose firmware starts again (hw->fw_restart), a halt after
an IPC error now schedules fw_recover_work. It boots the firmware once
more without the neural engine link and stops it the normal way, which
powers the module down. It runs once: a second crash is left as it is.
On J815 the light went off after the crash, and the log said "crashed
firmware: restarted and stopped it to power the camera down".
Signed-off-by: Nick <nekreshen@yandex.ru>
…hapes, gain knobs What MLVNR needed on J815 hardware (2026-11-21, rounds 2-5): - SHAREDMALLOC surfaces get IDs, which the ANE firmware maps them by. The rings are handed to the firmware at the address form macOS uses (ane_ring_fw). - The ISP firmware's input check (26B131) takes only certain (row stride, rows) pairs. The binned mode (config 8, 1520 wide, rows of 0xc00 bytes) is not one of them, and the firmware asserted on the input size. The full-resolution mode macOS streams (config 6, 3040x2880, rows of 0x17c0) is one. isp_mlvnr_input_ok() has the firmware's table, and MLVNR is not started in a mode outside it; a log line says so instead. - mlvnr_gain_enter / mlvnr_gain_exit (1/256 units, default 14x / 11x, macOS's values) set the auto-mode thresholds, so a test can force MLVNR on in any light. - While the firmware is restarted after a crash, the ANE link stays down. On hardware in config 6 the link works: 33 surfaces mapped, NetInit (11 procedures), LoadProgram in 0.9 ms, MLVNRProc started. The ANE then stalled on unread reports (fixed on the ANE side), so no frame has been denoised yet. KUnit: isp_test_mlvnr_inputs. Signed-off-by: Nick <nekreshen@yandex.ru>
The host tests compile driver functions against fake boundaries, so they stopped building with this series: - isp-preview-flow-tests: isp_ch_configure_capture() now reads the knobs that match macOS (isp_lpdp_macos, isp_macos_tune), the remapped preset flag, and calls the SIFR, camera LED and MLVNR helpers. They get inert stubs; the order this test checks is unchanged. - isp-tests session: the firmware stop path calls apple_isp_ane_stop() and the boot apple_isp_ane_start(), and reads the crash recovery work and flag. The fixtures have no neural engine and no fw_restart firmware, so recovery is never scheduled and the idle-work check stays as strict as before. All ISP suites pass again with clang 19 and ASan/UBSan, and the four session negative controls still fail on their CHECK lines. Signed-off-by: Nick <nekreshen@yandex.ru>
…R sensor mode Two optional properties of the sensor-presets node, for what belongs to the camera module rather than to the driver: - apple,lens-circle: the radius of the image circle the lens draws on the sensor, per mille of each preset's input width. The J813/J815 ultra-wide lens is black outside it, so the crops are kept inside. - apple,mlvnr-sensor-config: the binned sensor configuration of the presets and the full-resolution one, at twice its input size, whose frames the ISP's ML video noise reduction network takes. Signed-off-by: Nick <nekreshen@yandex.ru>
Ryan Murray's review: merged code picks its behaviour itself, and what belongs to the sensor or the lens goes in the camera's DT node. So the TEST module parameters go: - hr, macos_tune, led_brightness: what macOS 27.2 sends the H17 firmware on the J815 is now fixed behaviour on H17. That is highlight recovery with SIFR skip interval 1 in SIFR modes; AE stability 16/8 and an AE gain cap of 96.0; TNR (1, 1, 0) with high strength on band-0 setfiles; and the camera LED at macOS's start brightness, 529. The LED value is still clamped to the range the firmware reports, so the privacy indicator can never be set dark. Other firmware keeps its old values. - lpdp_macos: the 0x18-byte LPDP tuning made no measurable difference in the J815 A/B, so it goes with its command; the driver's (1, 15) stays. - sensor_config, presets_full: the full-resolution mode is only wanted for MLVNR, whose network takes no binned frame. With apple_isp.mlvnr, the presets of the binned mode in apple,mlvnr-sensor-config move to its full mode (input and crops doubled); without it they keep the DT's. mlvnr is now read at probe (0444), since the presets depend on it. Configs 7 and 9 were bench modes only and go. - lens_circle, lens_center_y: the radius comes from apple,lens-circle. The centre shift was never used and goes. - mlvnr_gain_enter/_exit move to debugfs (apple-isp/, with mlvnr only), for tests that force MLVNR on. mlvnr_ring_tag is fixed on, the form macOS uses. Signed-off-by: Nick <nekreshen@yandex.ru>
…MLVNR mode The ultra-wide IMX958 module's lens draws an image circle of radius 0.47 of the sensor width (measured on a J815; the J813 is assumed to have the same lens), and MLVNR takes config 6 (3040x2880 with SIFR, the mode macOS streams), not the presets' binned config 8. Signed-off-by: Nick <nekreshen@yandex.ru>
macOS's start brightness (529) was traced on the J815 only. Other H17 firmware keeps its own default until it is measured there: a per-SoC hw->led_brightness, set for T8142, replaces the H17-wide check. Signed-off-by: Nick <nekreshen@yandex.ru>
Drop the mlvnr module parameter. At probe, before the presets are read, MLVNR is on when: - the platform's ANE doorbell is known (hw->ane_doorbell_known); - the sensor presets name the network's sensor mode (apple,mlvnr-sensor-config); - the ANE node is enabled in the DT; - the board's network, apple/ispane-<board>.afpp, is installed (a partial read of its head; the ANE driver loads it whole). Otherwise one info line says which is missing, and the camera works as before. The presets move to the network's sensor mode only when MLVNR is on. The ANE driver itself is still taken at the first firmware start, as the ANE probes on its own schedule. If it is not loaded then, or the link fails, one line says so and that mode streams without MLVNR. The gain-threshold debugfs files stay, and they exist only with MLVNR on. Signed-off-by: Nick <nekreshen@yandex.ru>
78b274b to
deb2e98
Compare
|
@RyanTheTide I force-pushed
Assisted-by: LLM (Claude:Opus 5.5) |
|
Thanks for the rework. The parameter is gone, MLVNR is decided at probe from what the platform and the installed files actually provide (one info line when something is missing), the names are out of the code and the messages, and the commits carry no trailers. Provenance is the blocker, as on #2647, and here it goes further: the description says the MLVNR input table ( On the trade-off: option 1 shouldn't change what a well-lit stream looks like today, and from your J815 numbers it does (1.29 binned vs ~3.0 full resolution without MLVNR). I'd take whichever of 2 or 3 your noise row shows keeps well-lit output at least as clean as today's binned stream. If 3 streams fine in every light and measures cleaner, it's the simpler one. Assisted-by: LLM (Claude:Opus 5.5) |
|
@RyanTheTide thanks. Following your note, I'm closing this in favour of a fresh branch: #2843. None of its history has the MLVNR code or the firmware-derived input table. It is a draft on #2194 (
Highlight recovery and the SIFR skip interval are gone there too. They only apply in a SIFR mode, and the presets stay binned, so they would be dead code until a full-resolution mode comes back. One more provenance detail, disclosed in #2843's body: I first named its three new commands from the name table behind the firmware's command log line, extracted from the image. Each name also appears in the firmware's runtime log (Linux dmesg or macOS's log), and the code now cites only that. MLVNR comes back as its own PR, rewritten from a hypervisor trace and its write-up, as on #2647. Your trade-off choice goes with it: option 2 or 3, picked by the noise row. Assisted-by: LLM (Claude:Opus 5.5) |
Draft. Stacked on #2194 (
988af2c7, load the module tuning the DT names), on integration/aurora bb9f8ec. Only the last thirteen commits are new. The ANE side of MLVNR is #2647.1. macOS's stream settings on H17 (
cc44a40e, made fixed in528ec902)These are the commands and arguments macOS 27.2 sends the same J815 ISP firmware, taken from a hypervisor IPC trace. They started as test parameters. After review they are fixed behaviour on H17:
7069144d):CH_STATUS_LED_MBRIGHTNESS_SETis sent at macOS's start value, 529. It is clamped to the rangeCH_STATUS_LED_THR_GETreports (204..1023 on the J815), so the LED can never be set dark.CH_AE_GAIN_CAP_SET,CH_STATUS_LED_MBRIGHTNESS_SET,CH_STATUS_LED_THR_GET,CH_HR_ENABLEandCH_SIFR_SKIP_INTERVAL_SET. The opcode names are the ones in the firmware's own command table.2. Crop inside the lens circle (
669b20c2,7c9bf6b0)apple,lens-circle, per mille of the input width: 470 on the J813/J815, with the binding in3faacebdand the dtsi in9c17988f. Each crop is scaled about its centre until its corners are inside the circle.3. MLVNR, the ISP's ML video denoiser on the ANE (
e160901d,40b5c8ec,2bfcf61b,deb2e98c)The ISP firmware sends frames to the ANE over two rings; the host only sets up the link.
e160901dis Ryan Murray's draft fromsithglan/camera/isp-ane-mlvnr(c5b5d15). It adds the ISP side and the in-kernel interface to the ANE driver,include/linux/soc/apple/ane-isp.h. The code is unchanged; only the trailer was dropped, and hisSigned-off-byis kept.40b5c8ecmakes it work on T8142 (H17 firmware):CISP_CMD_IPC_ENDPOINT_UNSET2carries the whole SET2 body again.CISP_CMD_APPLE_CH_MLVNR_MODE_CONFIG_SET(0xc11f) with macOS's auto mode, after the MBNR set-up: ctrl 2, on above 14x gain, off below 11x, 30 fps and a 30-frame ramp.symbol_get()at the first firmware start, so apple-isp never defers on the ANE driver.2bfcf61b: what the hardware needed beyond that (J815, 2026-11-21):isp_mlvnr_input_ok()carries that table.apple-isp/mlvnr_gain_{enter,exit}lets a test force MLVNR on.deb2e98c(new, for Ryan's 2026-11-21 note): nomlvnrparameter. At probe, MLVNR is on when all of these hold:apple,mlvnr-sensor-config = <8 6>);apple/ispane-<board>.afppis installed. Only its head is read here; the ANE driver loads the whole file.Otherwise one info line says which is missing. With MLVNR on, the presets move to the full-resolution mode. If the ANE driver is not loaded at the first firmware start, or the link fails, one line says so and that mode streams without MLVNR.
Firmware: the network comes from the user's macOS install. It is not added here.
A trade-off to decide, please. In the J815 A/B, frame-to-frame centre luma noise was 1.29 in the binned config 8 and about 3.0 at full resolution without MLVNR. In auto mode the firmware engages MLVNR only above 14x total gain, and in the Linux runs below it stayed idle (see Testing). So with the network installed, a well-lit scene may now stream full resolution without MLVNR, and that looked noisier than binned. The options:
The hardware row on this head will measure noise in each case before this leaves draft.
4. Camera light after a firmware crash (
6bef90bb)A crashed ISP firmware is forced off, so it never powers the camera module down. The sensor and its green light then stay on until the next camera use; this was seen on the J815 after an ISP assert.
hw->fw_restart), a halt after an IPC error now boots it once more, without the ANE link, and stops it the normal way.5. Host tests (
b6544fb1)tools/asahi/isp-preview-flow-tests.pyand theisp-tests.pysession suite compile driver functions against stubs, and this series broke their build. The new calls get inert stubs, and the checks themselves are unchanged.Provenance (please read)
The same question as on #2647, and the same answer applies to both PRs. Sources:
isp_mlvnr_inputstable.The class names, the firmware's source-file references and the IOSurface wording are gone from the code and the commit messages. Points 3 and 4 still mean this is not clean-room. Ryan, your decision on #2647 covers this PR too.
Testing
isp_test_sensor_sysfs, also fails on plain bb9f8ec.isp_test_temporal_filter_band0was fixed in528ec902: it still expected the old (1, 0, 0) start record.isp-tests.py,isp-preview-tests.pyandisp-preview-flow-tests.pypass with clang 19 and ASan/UBSan, on this head and on bb9f8ec.--strict: clean ondeb2e98c. Earlier commits keep a few line-length, alignment and&-spacing notes, which I'll clean before this leaves draft.sensor_config=6,mlvnr=1):crop 1520x856+0+292 -> 1242x700 (inside the lens circle).Assisted-by: LLM (Claude:Opus 5.5)