Repository navigation
Conversation
RyanTheTide: accel/ane: h17: recover a hung engine and retry a failed boot With aurora-silicon#1863 (idle park) kept: the idle park leaves the engine alone while a hung-request recovery is queued or running. aurora-silicon#1303 is not taken (aurora-silicon#1768 carries its retry).
6eda4a2 to
74eccdd
Compare
|
idle_park_ms in ane_fw_cold.c adds a private module parameter for the idle park delay, and its 5000 ms default has no measurement behind it. This is what runtime PM autosuspend does: pm_runtime_set_autosuspend_delay() with a default based on the measured cold start, and the standard power/autosuspend_delay_ms attribute for anyone who needs a different value. Please move the park onto runtime PM and drop the parameter. Until then this should stay a draft. |
74eccdd to
7402a9c
Compare
|
Moved onto runtime PM: the bound device uses autosuspend, the runtime suspend callback parks the engine, and the resume callback boots it for the next request. The default is 2 s, from the measured wake cost: a request on a parked engine took ~16–20 ms end to end on the J700 (cold boot + replay), and the H16 driver logs a cold boot from the preload at 9–11 ms. At 2 s the wake costs under 1% of any pause long enough to trigger it, and the engine cycles its power at most once every 2 s. Rebased onto |
The H17 engine stays off until its first request and is parked again after each system sleep, but once a request has brought it up it stays powered for the rest of the boot: ANE_SYS, the engine and its DARTs keep drawing power until the next sleep. Hand the engine's ANE_SYS reference to runtime PM once it serves programs, and park it through runtime autosuspend: - The bound device uses autosuspend with a 2 s delay. A request that finds the engine parked waits for a cold boot from the preload and the replay of its programs, ~18 ms on a J700; 2 s keeps that under 1% of a longer pause and the engine's power cycles at most one per 2 s. The standard power/autosuspend_delay_ms changes it, and a negative value keeps the engine up until the next system sleep. - The runtime suspend callback takes the way down a system sleep takes: close the session and wait for requests in flight, stop the firmware, power the engine off and leave it parked, with the session closed, as after a wake. A ring with something still pending refuses with -EBUSY and autosuspend tries again later. - A request for a parked engine resumes the device; the resume callback boots it from the preload and replays every open program, the path a wake already uses. If runtime PM never suspended the device (runtime PM forbidden) while a system sleep parked the engine, the request boots it itself. - An execute holds the device and its end restarts the delay. - A system sleep holds the device across its own way down and then suspends it at once, so ANE_SYS still goes off before the sleep. - A quarantined engine keeps its power: the quarantine pins the device. This applies to every engine whose firmware parks the macOS way (macos_sleep), T8140 and T8142 today. The callbacks take b->lock only when they have work to do (a powered engine to park, a parked engine at the program stage to boot), and the paths that call into runtime PM under b->lock do so only with the engine already powered off or before it serves programs, so neither waits for a callback that needs the lock. Suggested-by: Ryan Murray <ryan@aurorasilicon.org> Signed-off-by: Bartosz Nitka <niteria@gmail.com>
7402a9c to
d271ab4
Compare
|
Re-reviewed at d271ab4. The earlier point is fixed: Checked the runtime PM paths:
No blocker. The PR is no longer a draft, but the description still lists the T8142 (M5) row and the H16/M1–M2 regression runs under "Still to run". Those need to land before merge: on H16 at least one request after a 2 s idle (park + wake in the log, result unchanged) and one deep sleep with the engine parked. Assisted-by: LLM (Claude:Opus 5.5) |
|
Tested
Updated the PR description with the J813/T8142 rows. The H16 and M1/M2 regression runs remain under "Still to run". Assisted-by: LLM (Gemini) |
|
Ran the H16G regression check on J713 (M4 MacBook Air 13", T8132,
We don't have M1/M2 hardware locally to run that regression check. By inspection, the new Assisted-by: LLM (Gemini) |
Problem
After its first request, the H17 Neural Engine stays powered until the next system sleep. On a J700,
ane_sys, the engine and its three DARTs were still on 10 minutes after the last request. Between boot and the first request, and after each wake, the engine is already parked: firmware stopped, power off.Change
Once the engine serves programs, its
ANE_SYSreference goes to runtime PM, and runtime autosuspend parks it when idle:power/autosuspend_delay_mschanges it, and a negative value keeps the engine up until the next system sleep. There is no module or command-line parameter.-EBUSYwith a fresh last-busy, so autosuspend tries again later.ANE_SYSstill goes off before the sleep. A quarantined engine keeps its power.The 2 s default comes from the measured cost of a wake: a request that found the engine parked took ~18 ms end to end on a J700, against ~0.13 ms warm, and the H16 driver logs a cold boot from the preload at 9–11 ms. At 2 s the wake costs under 1% of any pause long enough to trigger it, and the engine cycles its power at most once every 2 s.
It applies to every engine whose firmware parks the macOS way (the backend's existing
macos_sleepproperty): T8140 (J700) and T8142 (M5) today. The ANE platform driver gains runtime PM callbacks. They return 0 for every device that isn't the bound H17 engine, which is what the PM core does when a driver has none, so H16 and the M1/M2 devices behave as before.Thanks to Ryan Murray for suggesting runtime PM for this.
Locking
The callbacks take
b->lockonly when they have work to do: a powered engine to park, or a parked engine at the program stage to boot. The paths that call into runtime PM while holdingb->lockdo so only with the engine already powered off (power-down, quarantine) or before it serves programs (the first bring-up). Neither can wait for a callback that needs the lock. The session wake callspm_runtime_resume_and_get()without any lock held. The execute hooks run under the session lock and only take a no-resume reference or drop one withpm_runtime_put_autosuspend().If runtime suspend is forbidden (
power/control=on) or the delay is negative, a system sleep can park the engine while the device stays active. The wake then finds the engine still parked afterpm_runtime_resume_and_get()and boots it itself.Testing
drivers/accel/ane/,W=1, J700 config with the ANE built inboot.kunitconfig(KASAN, UBSAN)ane-h16-hwx, which crashes the same way onintegration/aurora(15 pass, 2 crashed)pm.kunitconfigtools/asahi/run-host-tests.pyintegration/aurora: the one failure,pmp-floor-tests --mutant class-twice, fails on bothcheckpatch.pl --strictb8c5bb03+ this commit + J700 encoder-0 DT / ALS sleep fix) — park & wake18.2 ms(1024/1024 exact, 0 ulp, p50127.1 us); parks ~2.4 s after last request (ane_sys off-0, engine + 3 DARTssuspended,DARTs off before ANE_SYS); wakes in15.8 ms(1024/1024 exact, 0 ulp, p50126.2 us)ane_sys off-0) each time, replayed on wake (16.8 / 19.5 / 20.0 mswake+replay+exec),1024/1024 exact, 0 ulpon every runpower/autosuspend_delay_ms-1and back-1wakes and keepsane_sys onpast 5 s idle; restoring2000parks immediately; next request wakes in15.5 ms(1024/1024 exact)rtcwake -m mem)suspend_stats/fail=0): parked, ~40 ms after use while active (stops firmware and dropsane_sysbefore sleep), and withautosuspend_delay_ms=-1(fallback unpark path on next request); all1024/1024 exactpost-wake31/525)2171.2 mWparked vs2182.6 mWidle-up (+11.4 mW, paired[+15.5, +6.7, +5.6, +18.0] mW); SMCPPMC649.9vs661.8 mW(+11.9 mW, 4/4 pairs positive)ab9352c6+#2325+#2747+d271ab48, ANE as moduleane_t6021.ko) — park & wake420000000.ane+ both DARTs (421800000.iommu,421840000.iommu)suspended,ane_sys off-0. First bring-up16.5 ms(1024/1024 exact, 0 ulp, p50125.2 us); parks ~2.3 s after UNLOAD (firmware stopped, status 0x6e,powered down,idle: parked until the next request,DARTs off before ANE_SYS,ane_sys off-0); wakes in11.0 ms(1024/1024 exact, 0 ulp, p50146.5 us)10.94 ms), 3 × 3 s idle pauses; parked (ane_sys off-0, both DARTssuspended) each time, replayed on wake (18.43 / 22.30 / 21.60 mswake+replay+exec),1024/1024 exact, 0 ulpevery cycle; close → re-parked cleanlypower/autosuspend_delay_ms-1and back-1wakes (ane_sys on), warm load0.6 ms, stays active past 4.5 s idle; restoring2000parks immediately (ane_sys off-0), wake run18.9 ms(1024/1024 exact), re-parkedapple_sleep,t8142_wake_s=20)suspend_stats/fail=0): parked (ane_sys off-0), ~31 ms after UNLOAD while active (firmware stopped, status 0x6e,powered down,DARTs off before ANE_SYS,10 coprocessors up and stopped,resume ok: parked until the next request), and withautosuspend_delay_ms=-1(fallback unpark path in11.2 ms); all1024/1024 exactpost-wake, 0 ANE/DART/PMGR errorsdc771820+d271ab48, ANE as moduleane_t6021.ko) — H16 regression (idle + deep sleep)ane_t6021confirmed carryingane_rtclient_runtime_*/ane_fw_platform_runtime_*.500000000.aneprobed unchanged,power/runtime_status=unsupportedthroughout.add128(16384/16384 exact, 0 ulp) passed before (0.6 msload, p50137.8 us) and after 3.5 s idle (0.4 msload, p50115.4 us). 1/1 deep sleep (rtcwake -d rtc1 -m mem -s 15,suspend_stats/fail=0):H16 ANE: resume ok: cold, firmware ready 11 ms, post-wakeadd128passed immediately (0.4 msload, p50113.0 us) and again after 2.5 s idle (0.6 msload, p50206.9 us). 0 failed units, 0 ANE/DART/PMGR errors or warningsStill to run
M1/M2 regression run: we do not have M1/M2 hardware locally. The new runtime PM callbacks in
ane_t6021_rtclient_main.creturn0without touching hardware for every device that isn't the bound H17 engine, so M1/M2 behavior is unchanged by inspection; happy for someone with an M1/M2 to run a check if desired.Clean-room: no Apple code, strings, firmware, RE output or tuning values
No chip/board/firmware-version gate where a hardware property can decide
Works on the default kernel command line; no hacks, delays or lab-only paths
Safety limits come from the machine, not from constants in code
Hardware safety rules followed for every test boot
Third-party commits unchanged; reporters and testers credited
Commits: human Signed-off-by only; no agent trailers of any kind
Tested on the affected hardware; no regression (J700, J813/M5, and J713/H16G done; no M1/M2 hardware locally)
Assisted-by: LLM (Claude:Opus 5.5, Gemini)