Repository navigation
media: apple: ave: hardware H.264/HEVC encode on the M3 (T8122, J615) with the macOS 27.2 firmware - #2644
Conversation
…er states The M3-generation encoder nodes (T8122, T6030, T6031 ave0/ave1, T6032 ave0-ave3, added disabled in 982f330 "arm64: dts: apple: describe disabled M3 peripheral resources") list the five register windows in ADT order (codec, coproc, mailbox, pmgr, pio) without names, and the power domains as sys, pipe4, pipe5, me0, me1, dma, again unnamed. The AVE driver and binding find the windows by name in the order codec, mailbox, coproc, and attach the domains by name as dma, pipe4, pipe5, me0, me1, as on T8112. Describe them as the T8112 node does: reorder the windows and add reg-names, list the five encoder power states with power-domain-names, and leave the VENC system state to the encoder DARTs, which is where the ADT puts it (ps_venc_dma has it as its parent). The T6031 ave1 and T6032 ave1/ave3 nodes, which already lacked the system state, now list the same five states as their ave0 siblings. Addresses, interrupts and power states are unchanged and match the ADT of every M3 board (J613, J615, J433, J434, J504, J514s, J516s, J514c, J516c, J514m, J516m, J575d both dies) and our own IODeviceTree capture of a J615 under macOS 27.2. The nodes stay disabled. Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
T8122, T6030 and T6031 were listed as encoders the driver does not bind, with five unnamed register windows, and T6032 had a branch of its own with the same unnamed form. The driver now binds all four, by register and power-domain name. Move T6032 into the "<soc>-ave", "apple,ave" list and drop its branch, and require reg-names and power-domain-names (and at least five power domains) on all four M3 SoCs, as on the other bound encoders. T6020, T6021, T6022 and T8103 keep the unnamed form. Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The M3-generation encoders (T8122, T6030, T6031 ave0/ave1, T6032 ave0-ave3) have the T8112 encoder's register windows, interrupt order, power states and DART layout in the ADT, but iBoot preloads a firmware build of their own: on a J615 under macOS 27.2 the ADT segment-ranges give TEXT 0x134000 and DATA 0x128000 bytes, which matches none of the builds the driver knows (T8112: TEXT 0xd0000). So none of the existing match data can be reused: its sizes, TEXT CRC, boot sequence, device numbers and command layout all belong to other builds. Match the four M3 compatibles to one probe-only variant that pins no build: it takes any segment sizes from the bootloader's handoff, still checks the reservation, the DART mapping and both loader CRCs, and logs the sizes and CRCs so the build can be qualified. ave_fw_check_boot() refuses to start the coprocessor because the variant is not boot-qualified and has no registers, so no stream can start, whatever layout a TEXT CRC might select. The probe-only message now also names the segment sizes. A KUnit case checks that the M3 variant stays probe-only even with a TEXT CRC that selects a known layout. Builds on the probe-only path of the T6041 and T8140 bring-up and the AVE driver by the Aurora Silicon contributors. Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The macOS 27.2 build of the encoder firmware takes the 26A428 Start commands unchanged, but its H.264 and HEVC Process commands are 0x10 bytes longer (0x6e08 and 0x78c8). Every field the driver writes stays at its 26A428 offset; the added bytes are left zero. Share the 26A428 field description between the two layouts and add ave_layout_26b123 with the longer Process commands. No firmware selects it yet. A KUnit case checks that the two layouts differ only in the Process sizes. Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
…irmware The M3 (T8122) encoder only bound probe-only. With the macOS 27.2 (26B123) stub, as on the MacBook Air 15" (J615), the bootloader preloads TEXT 0x134000 and DATA 0x128000, TEXT CRC 918d70d3, the same on every boot. That build belongs to the 27.2 firmware of the M4 generation: it takes the boot configuration handshake as device 17 with handshake type 17, and speaks the 26B123 command layout. The coprocessor around it is still the M2's: the same register family and start, the bootloader's code base left as it is, the mailbox idle word cleared before start, and DATA restored from the boot snapshot before every start. The LFS reference buffer uses scale 2. Give T8122 its own variant. The other M3-generation chips (T603x) stay probe-only until their builds are seen. Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The bootloader hands the M3 encoder's preloaded firmware over on the MacBook Air 15" (J615). Enable the encoder and its two DARTs; the DARTs get their VENC power state from the SoC description. Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
DjDeveloperr
left a comment
There was a problem hiding this comment.
Reviewed d994b2b. Approving the code. The merge waits for CI and two J615 items.
Shared driver: the T8140 and T8112 paths are unchanged, so no Neo row is needed for this PR.
ave-drv.conly adds code: theave_t8122andave_m3variants and fourof_matchentries. No existing variant struct changes.- The command layout is chosen by the firmware's TEXT CRC (
ave_layout_for_crc(): T81220x918d70d3and T81320x99fc929a→ave_layout_26b123). The variant comes from the block's compatible, and its build is pinned by segment sizes and CRC. There's no chip gate. - The 26A428 refactor into
AVE_LAYOUT_26A428_FIELDSis exact. I compared the oldave_layout_26a428body with the macro field by field: 253 of 253 identical.ave_layout_26b123differs only in the two Process sizes (+0x10). ave-fw.cadds the CRC to the known list and one probe-only branch,!boot_qualified && !fw_text_size. Every existing variant pinsfw_text_size, so that branch only reachesave_m3. The probe-only log line now also prints the sizes.- The M3 start uses fields that already exist (
mbox_idle,restore_data_first,vcbar_mask/vcbar_flags), with the values T8112 already uses formbox_idle. The coprocessor code itself isn't touched.
DT and binding.
- The t8122 and t603x
regorder now matchesreg-names(codec, mailbox, coproc, pmgr, pio). The driver maps by name, so this is what lets T8122 bind. - Dropping
ps_venc_sysfrom the encoder is fine:ps_venc_dma's parent isps_venc_sys(t8122-pmgr.dtsi:1083), and both DARTs keep it. - The binding rework keeps the T8140/T6041/T6050/T8142
maxItems: 3limit, and the Neo's nodes have 3 named windows. dt-checks validates every Apple DTB at base and head, so a green run settles it. It's still queued.
ABI test. ave_26b123_layout covers the new layout, but only through the T8132 CRC.
- Please also assert
ave_layout_for_crc(AVE_FW_TEXT_CRC_T8122_26B123) == &ave_layout_26b123, plus theave_t8122pinned build (0x134000/0x128000,text_crc,boot_qualified). This PR is what makes that mapping live. - Nit: the new test sits between the "M4 Max loader sets one TEXT word" comment and
ave_t6041_text_boot_word(), which that comment documents.
Before merge:
- CI green on the exact head. All 8 checks, including dt-checks, kunit and rust-build, are queued on d994b2b.
- Name the head and Stage 2 the J615 rows ran on. The body says "this exact head" for the second cold boot, but gives no SHA. The Stage 2 it names is aura main + #354/#334, which aura#420 now supersedes. Please give the kernel head (d994b2b or otherwise) and the aura head for the 8 rows, the byte-identical repeats and the PM cycling. Also point the dependency at aura#420.
- One J615 boot of this head on aura main without #420. This DT enables both AVE DARTs and the encoder unconditionally, so a J615 on today's aura gets it before the T8122 handoff lands. Pass: the encoder declines to bind with a clear log line, there are no DART faults or SErrors, the boot reaches the desktop, and s2idle works. Alternatively, merge this only after aura#420.
Once this merges, #2362 closes as superseded. Its three commits are carried here unchanged: git patch-id matches for c93c19c/e30360cc, d5fd9b7/9540263a and f14e055/eb9bfdad.
Assisted-by: Claude Opus 5.5
The encoder and its two DARTs were enabled for the J615 only. Nothing in
the encoder is specific to that board: every T8122 Mac has the same
block, and whether it can run is decided at boot from what the hardware
and firmware describe, as on T6000 and T8112, which enable the encoder in
the SoC tree. The bootloader hands the preloaded firmware over only when
the ADT encoder is pre-loaded with a valid segment list, and disables the
node otherwise; without a handoff the driver finds no firmware reservation
and fails probe ("no preloaded firmware") before it powers anything.
Enable the encoder and its DARTs in t8122-peripherals.dtsi and drop the
J615 override, so the J613, J504, J433 and J434 get it as well.
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
DjDeveloperr
left a comment
There was a problem hiding this comment.
Re-checked 0d15803. The only new commit moves the encoder and DART enables from t8122-j615.dts into t8122-peripherals.dtsi. That's the right shape: no board override, and the loader handoff plus the driver's "no preloaded firmware" probe failure decide at boot. T8140, T8112 and the driver are untouched, so my earlier review stands.
What changes is the reach. Every T8122 Mac (J613, J504, J433, J434, J615) now boots with both AVE DARTs and the encoder okay, including on today's aura, which doesn't hand over the T8122 encoder until aura#420 lands. So item 3 matters more now. Before merge:
- CI green on 0d15803. dt-checks now covers the SoC-level enable on all five T8122 DTBs.
- The heads behind the 8 encode rows. Name the kernel head and the aura head (aura#420's, not #354/#334) in the body.
- One J615 boot of this head on aura main without #420. The encoder must fail probe with "no preloaded firmware", the DARTs must probe without faults or SErrors, the desktop must come up, and s2idle must work. A J613 too if one is available, but the J615 is enough as the same SoC and firmware. Alternatively, merge this only after aura#420.
- Your "still to run" row: a J615 boot with the SoC-level enable plus aura#420's head, with the 8 encode rows.
Approving the code. #2362 closes as superseded once this merges.
Assisted-by: Claude Opus 5.5
|
J615 row on aura#420's current head (999ab16c merged with aura main), driver = this PR's head (the later 0d15803 changes only the DT):
Assisted-by: LLM (Claude:Opus 5.5) |
On a counted command layout the driver refuses every session that is not
30 fps with B-frames, apart from H.264 Baseline and intra-only. That
left ffmpeg's h264_v4l2m2m and hevc_v4l2m2m unusable on the M3 with the
macOS 27.2 firmware: FFmpeg's V4L2 encoder wrapper always asks for no
B-frames, so every Main/High and HEVC session failed with -EOPNOTSUPP
("26B123 layout: only 30 fps with 3 B-frames is qualified").
The 27.2 (26B123) build encodes those sessions. On a J615 (T8122, TEXT
CRC 918d70d3), with the refusal skipped, ffmpeg encoded H.264 Main and
High and HEVC Main without B-frames, plus H.264 Baseline, at 24, 30 and
60 fps, and gst v4l2h264enc/v4l2h265enc did the same without B-frames
and at 60 fps. All 11 streams decode in full with no errors. Against the
encoder's own NV12 input, PSNR is 40.4 to 46.0 dB at 8 Mbit/s: Main
44.9, HEVC 45.9 and Baseline 43.6 at 30 fps; 46.0 at 24 fps; 40.4 and
41.0 at 60 fps. libx264 gives 47.3 dB at the same rate. There were no
coprocessor faults, DART faults or firmware timeouts.
Describe that on the layout: any_gop marks a counted layout whose
build runs any GOP structure and rate, and the refusal now applies only
to counted layouts without it. 26B123 sets it; 26A428 keeps the
refusal until its own rows. The T8132 build (99fc929a) speaks the same
26B123 layout and gets the same behaviour; its encode rows on a J713
will cover it.
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
|
1d21390 lifts the B-frame/30 fps refusal for the 27.2 (26B123) layout. Evidence, J615, lab module = this driver with that refusal skipped (boot 2e1d5f80):
libx264 at the same 8 Mbit/s on the same source: 47.3 dB. There were 0 coprocessor faults, 0 DART faults and 0 firmware timeouts; Enforcing, 0 failed units. KUnit apple-ave 46/46. Assisted-by: LLM (Claude:Opus 5.5) |
DjDeveloperr
left a comment
There was a problem hiding this comment.
Re-checked 1d21390. The change is minimal and in the right place. The firmware build's layout now says whether it runs any GOP structure and rate (any_gop), the refusal in ave_sess_params() applies only to counted layouts without it, and KUnit pins both values plus the T8122 CRC → 26B123 mapping. 26A428 is untouched. The commit message gives the reason for keying this per build, and the rows back it: 11 sessions decode in full and land 40.4 to 46.0 dB, against libx264's 47.3 at the same rate.
Before merge:
- CI green on 1d21390. It's running now.
- One row on the exact head build. The rows above ran a lab module with the refusal skipped. That's equivalent for this commit, but please run one ffmpeg
h264_v4l2m2mMain session without B-frames at 30 fps and onehevc_v4l2m2msession at 60 fps on a 1d21390 build. Frames, decode errors and 0 faults are enough. - Item 3 from my last review still applies. aura#420 is still open, so either one J615 boot of this head on aura main without #420 (the encoder fails probe with "no preloaded firmware", the DARTs probe cleanly, desktop up, s2idle works), or merge after aura#420.
T8132: the T8132 26B123 build gets any_gop too, but its encoder only starts with #2643. #2643's rows on the M4 should include the same sessions without B-frames and at 60 fps once it's rebased on this.
Approving the code.
Assisted-by: LLM (Claude:Opus 5.5)
|
J615 (15" MacBook Air M3, T8122, 27.2 stub) rows on this PR's head 1d21390. The DT is the same as 0d15803; the new commit only touches the driver. Heads. The kernel is the J615's default 7.3.0-aurora-fleet77, whose
So on today's aura, the SoC-level enable fails closed as intended. Assisted-by: LLM (Claude:Opus 5.5) |
|
Merged at 1d21390 (4ea3629). On the J615 with this head's DT and driver:
Post-merge, on a build of 1d21390 or later: one ffmpeg #2362's three commits are carried here unchanged, so #2362 can close as superseded when you're ready. Assisted-by: LLM (Claude:Opus 5.5) |
|
Exact-head row for 1d21390 (J615, driver built from this head as-is, no lab parameter): the same 11 sessions all pass. That covers ffmpeg A separate row on a fleet77-based kernel carrying this head's DT and driver: without the aura#420 handoff the encoder fails probe cleanly ( Assisted-by: LLM (Claude:Opus 5.5) |
The AVE binding was restructured on integration (#2644). Carry this branch's one change to it into the new layout: apple,t6050-ave takes the apple,ave fallback list and leaves the three-register group. Signed-off-by: Diljit Singh <hey@djdev.me>
Problem
The video encoder on the MacBook Air 15" M3 (J615, T8122) only bound probe-only, so there was no H.264/HEVC hardware encode.
Change
t8122-peripherals.dtsi, so every T8122 Mac gets them (J613, J615, J504, J433, J434), as T6000 and T8112 do in their SoC trees. The DARTs keep theirps_venc_syspower domain, which they need: an enabled AVE DART without it stops the J615 before userspace. Whether the encoder runs is decided at boot from the hardware and firmware: the bootloader hands the firmware over only when the ADT encoder is pre-loaded with a valid segment list, and disables the node otherwise. Without a handoff the driver fails probe ("no preloaded firmware") before it powers anything.apple,t8122-ave) and the command layout by the preloaded firmware's TEXT CRC, because the firmware describes neither its handshake nor its command layout. The driver has no SoC, board or OS-version checks.any_gopon the layout). Until now the driver refused them, so ffmpeg'sh264_v4l2m2m/hevc_v4l2m2m, which always ask for 0 B-frames, failed with -EOPNOTSUPP. 26A428 keeps the refusal until its own rows.Needs the encoder handoff in aura: aurora-silicon/aura#420 (aurora-silicon/aura#334 + aurora-silicon/aura#354 rebased onto main). Without that handoff the node has no firmware regions and the driver declines to bind.
Testing
J615, kernel 7.3.0 d1 lab build with this driver as a module, Stage 2 = aura main + #354/#334. Sources were gst videotestsrc smpte, moving; the same script and the same sources (identical md5) were run on a J616s (M4 Pro) with the production driver.
Every row: rc=0, and the full frame count decodes with ffmpeg with 0 errors.
Quality matches the M4 Pro to within 0.03 dB. Throughput is 0.68x of the M4 Pro on every row.
Repeat runs give byte-identical bitstreams (md5).
The firmware restarts after each runtime-PM power cycle, and the encoder suspends with all five VENC power states off.
0 SError/DART faults, 0 failed units, SELinux enforcing.
KUnit apple-ave: 31/31.
dtbs_check(CHECK_DTBS=y) on all five T8122 DTBs (j433, j434, j504, j613, j615): no warnings for the encoder or its DARTs, which areokayin each. The other warnings in that output are for nodes this PR doesn't touch (aliases; isp, aop, gpu without schemas) and are on integration/aurora too.Each commit builds.
This exact head, as a module on a second cold boot, passes all 8 rows, and every bitstream is byte-identical to the first boot's.
Sessions the driver used to refuse, run on the J615 with that refusal skipped (lab module): ffmpeg H.264 Main/High/Baseline and HEVC without B-frames, at 24/30/60 fps, and gst without B-frames and at 60 fps. All 11 streams decode in full with 0 errors. PSNR against the encoder's NV12 input at 8 Mbit/s is 40.4–46.0 dB (libx264: 47.3), with 0 faults.
This exact head on the J615 runs the same 11 sessions with byte-identical output. A fleet77-based kernel with this DT and driver, but without the aura#420 handoff, fails probe cleanly ("no preloaded firmware") and the desktop comes up.
Still to run
Reviewer detail
coprocessor lost its code base (vcbar 0x10000000001). The bootloader sets the valid bit, so the variant usesvcbar_mask = GENMASK_ULL(41, 11)andvcbar_flags = AVE_COPROC_VCBAR_VALID.restart: CTRL 0x0 BOOT_ARG 0x0->BOOT_ARG after start 0x10400 (wrote 0x10000)->firmware running, 992/993 ring slots. Both DARTs are audited before and after the start: 1 live stream each, 0 unsafe.Assisted-by: LLM (Claude:Opus 5.5)