Repository navigation
accel/ane: h16, h17: give program memory and DVA back after close - #2240
Conversation
|
Thanks, this is a clear fix for both limits. The arena rewind helps every backend. The give-back is opt-in through The code looks good to us. Two things before it leaves draft:
And the Signed-off-by, as you noted. |
|
Thank you for the review and for offering both runs.
niteria will add the Signed-off-by before this leaves draft. |
|
Thanks, the two H16 commits match the H17 shape well. The range goes into We will run the H17 rows on our A18 Pro (Neo) at this head on plain integration, and your H16 table on our M4 Pro, including the two-client case with lockdep on. Results will follow here. |
The transport copies every command into a 256 KiB pool arena. Each EXECUTE sequence gives its space back once the firmware reported the request complete, but LOAD, CREATE, TERMINATE, UNLOAD and the gating commands outside a request stay until the next firmware boot: 1164 bytes per program loaded and closed. About 210 programs into one firmware boot, h16_pool_room() refuses every LOAD with -ENOSPC. On H17 an idle park or a system sleep boots the firmware again and starts a new arena; a client that keeps the engine busy, or a platform that does not park, runs into it. Take the arena position at a LOAD that finds no program alive in this firmware boot. When the last program is torn down cleanly, with nothing in flight and the session not quarantined, the firmware is done with every command published since, so release the arena to that position. A program still loaded keeps its commands and every later one; a new firmware boot drops the position. Assisted-by: LLM Signed-off-by: Bartosz Nitka <niteria@gmail.com>
Program memory never goes back before reset. A closed program's surfaces become spares, which serve only a later mapping of the same size and permission; the table holds 64, and anything beyond it, or a LOAD that failed before or at the firmware, keeps its memory mapped and unused. A client that loads programs of many shapes (a sweep of held-out conv layers, camera frames at several sizes) thus fills the backend's DVA window: on the J700, 337 programs into a boot, every H16_LOAD failed with -ENOSPC until reboot, and the memory behind the window stayed allocated too. Add an optional ops->unmap that unmaps a surface, frees its memory and its DVA range, and use it for memory the firmware no longer uses: - Keep the spares oldest first, and free the oldest beyond 32 entries or 128 MiB after every give-back and LOAD. A spare table that is full frees the surface instead of keeping it. - When a mapping finds no room (-ENOSPC or -ENOMEM), free every spare and try once more. - A LOAD that fails before the firmware saw the program gives its memory back and frees the program. One the firmware refused, at LOAD or CREATE, joins the programs held for the verified power cycle of the recovery, which gives them back. The DART invalidation of an unmap must reach the engine, so nothing is freed while the session is asleep or parked; the spares keep it until the next LOAD, which runs with the engine up. Every caller holds the session lock. Without ops->unmap nothing changes. Assisted-by: LLM Signed-off-by: Bartosz Nitka <niteria@gmail.com>
The H17 backend hands out program and SHAREDMALLOC mappings by bumping a pointer through a 512 MiB DVA window, and never gives anything back: once the window is used up, every program mapping fails with -ENOSPC until reboot. On the J700 that happened after 337 program loads of a held-out layer sweep and some camera-denoise trials, and from then on every H16_LOAD failed, even of a 6 KB program that had run before. Allocate window ranges from a drm_mm instead, and implement the session's ops->unmap for program surfaces: find the mapping, unmap it from the DARTs (iommu_unmap() invalidates their TLBs before it returns), then free its pages and its range. A short unmap keeps the mapping, its memory and its range, as a failed mapping does. SHAREDMALLOC, pool, IPC, heap and preload mappings are never unmapped. A failed iommu_map() no longer leaves a partial mapping at the DVA the next allocation is given, since its range stays allocated with it. Assisted-by: LLM Signed-off-by: Bartosz Nitka <niteria@gmail.com>
ane_h16_alloc_prot() puts a mapping on h->mappings before it allocates anything, so a 16K block, vmap() or sg table that fails leaves the mapping and the blocks it already got on the list. Before RUN the rollback frees them; from RUN on nothing does, and every program mapping that fails with -ENOMEM (a fragmented buddy allocator, a camera-sized surface) keeps its partial memory until reboot. Free the memory of a mapping that fails before iommu_map_sgtable() right away: nothing of it is visible to the engine. A mapping that iommu_map_sgtable() refused is still recorded and kept until reset, as before. The rollback frees mappings through the same helper. Assisted-by: LLM Signed-off-by: Bartosz Nitka <niteria@gmail.com>
The H16 backend hands out program mappings by bumping a pointer through a 1 GiB DVA window and never gives anything back: once the window is used up, every program mapping fails with -ENOSPC until reboot. A client that loads programs of many shapes gets there, as the H17 backend did on the J700 after 337 loads of a held-out layer sweep. Allocate program ranges from a drm_mm instead, and implement the session's ops->unmap: find the program mapping, unmap it from the DARTs (iommu_unmap() invalidates their TLBs before it returns), then free its pages and its range. A short unmap keeps the mapping, its memory and its range, as a failed mapping does. SHAREDMALLOC, preload, heap tail and identity MMIO mappings have no range in the window and are never unmapped. The session calls ops->unmap only with the engine up, never while the session is asleep for system sleep or a recovery power cycle, so the invalidation reaches a powered engine. A new lock guards the mappings list and the window, since SHAREDMALLOC can be mapped while a program is unmapped. Assisted-by: LLM Signed-off-by: Bartosz Nitka <niteria@gmail.com>
24073e1 to
049ae04
Compare
|
H17 rows on our A18 Pro (MacBook Neo, J700, T8140) at this head (049ae04), on plain integration as promised: integration/aurora 0abc2f3 with this PR merged. The test kernel also carried #1065, #1881 and #2143, which touch other drivers. It did not carry #1768 or #1863, and had no local allocation change. Firmware 25G83. Everything below ran in one firmware boot, with Four of the five rows pass. The two-client row fails.
No WARN, oops, quarantine, DART fault or short unmap was logged in any row. Two-client finding. The arena rewind only happens when the last program is torn down ("At a LOAD that finds no program alive, the session records the arena mark"). Any long-lived program, such as a daemon or a model kept resident, therefore pins the mark. Every other client's load/close cycle then leaks its LOAD/CREATE/TERMINATE/UNLOAD commands (0x48c bytes each), exactly as before this PR. After about 190 cycles the arena is full, and it stops the resident client's EXECUTE batches too. So one client's churn can starve every other client until all programs close. The rewind fixes the single-client case. The multi-client case needs the commands of an unloaded program freed per program, rather than a single mark. For example:
Either way, an EXECUTE on a loaded program should not fail because another client loaded programs. A KUnit case for this would be one program held while another does N > 210 load/close cycles. The M4 Pro (H16) table, including this two-client case with lockdep on, follows when our slot on that machine comes up. Raw row linesThe rows use a small ABI 3 runner: open, H16_LOAD, H16_EXEC, close per cycle. Outputs are compared exactly against the expected file where there is one, otherwise against the program's first run. Assisted-by: LLM (Claude:Opus 5.5) |
The command arena went back to its start only once the last program of the firmware boot was torn down. A program that stays loaded, such as a model kept resident by a daemon, therefore pinned it: every other client's load/close cycle still left its LOAD, CREATE, TERMINATE and UNLOAD (1164 bytes) behind. After about 190 cycles every LOAD failed with -ENOSPC, and so did every EXECUTE of the resident program, until all programs were closed. Keep a program's LOAD and CREATE, which the firmware may read while the program lives, in a record of their own: 128 records of 1088 bytes, taken from the arena at the first LOAD of each firmware boot. A record is free again once the firmware acknowledged the program's UNLOAD, after which it keeps no pointer into the program's commands (the previous rewind relied on the same). A new firmware boot frees every record; a refused LOAD or a quarantined session keeps its record until then. Everything else goes to the end of the arena and now comes back as soon as the firmware is done with it: an EXECUTE batch when it completes, as before, and a gate, TERMINATE or UNLOAD outside a batch once acknowledged. EXECUTE batches thus never compete with loaded programs for space, and a LOAD with every record taken is a clean -ENOSPC before anything is mapped or sent. The transport gains a placed range for this: ane_fw_transport_reserve() takes it once per firmware boot, ane_fw_transport_publish_at() copies a command to a slot inside it, and ane_fw_transport_release() never gives back below it. Fixes: 0fb022b ("accel/ane: h16: give back the command arena of unloaded programs") Assisted-by: LLM Signed-off-by: Bartosz Nitka <niteria@gmail.com>
|
Thank you for running these, and for the two-client row in particular. You're right: the rewind only helps when every program goes away. One resident program pinned the mark, and every other client's load/close cycle then left its LOAD, CREATE, TERMINATE and UNLOAD in the arena, as before the PR. Your numbers match a KUnit replay of your row on 049ae04, and the same row on our M4 Air: the cycling client's LOAD and the resident program's EXECUTE both get What changed. One new commit on top, f1623ba "accel/ane: h16: give each program's commands a record of their own" (
Our earlier Air two-client row was flawed, so it didn't catch this. It ran Tests
M4 Air (J713, T8132, H16 firmware 25G83). Our Air kernel, which carries this PR's first five commits (19f7319), with and without this commit (the same patch, cherry-picked). Each row ran in one firmware boot. The window between the runner's kmsg marks was checked in the kernel journal (the ring buffer overflows during a row): no firmware boot or recovery, no
We haven't rerun the H17 rows on our Neo with this commit yet. Could you rerun the two-client row on your Neo at the new head? And when your M4 Pro slot comes up, it would help to include the same case in the H16 table, with lockdep on. The other four H17 rows exercise code this commit only touches at LOAD and close, but they're cheap to repeat if you're running anyway. |
|
H16 rows on our M4 Pro (J616s, T6040) at this head (049ae04). Kernel: integration/aurora 0abc2f3 with this PR merged (also #1065, #1881 and #2143, which touch other drivers), built with The new H16 code works for single clients: give-back, DVA reuse and memory return. The two-client row fails the same way as on H17. Lockdep found one report, and it is not from this PR.
Lockdep (not from this PR). On the first boot the s2idle row produced one report,
That is an ABBA between bring-up and a suspend that starts at the same moment. The code is identical on the base and on our current production kernel, so it predates this PR. Registering the notifier once at probe, or outside So for H16 the summary is the same as for H17: one client at a time is fixed, but a resident program still lets another client's load/close cycles fill the command arena and starve everyone. The per-program command release suggested in the H17 comment covers both backends, since the arena code is shared. Raw row linesAssisted-by: LLM (Claude:Opus 5.5) |
|
The two-client row now passes on both chips at f1623ba. Thanks for the record-per-program change. Kernels: integration/aurora 2c35f50 with this PR (f1623ba) merged, plus #2505 (the lockdep fix below), with
No row logged a WARN from the ANE, an oops, The lockdep report we mentioned in the M4 Pro results ( Still not run on our side: the H16 Raw row linesThe runner is the same ABI 3 tool as before: open, H16_LOAD, H16_EXEC, close per cycle; the held program's output is compared with its first run, the add with the expected file. Assisted-by: LLM (Claude:Opus 5.5) |
DjDeveloperr
left a comment
There was a problem hiding this comment.
Thanks @niteria, this is a really nice piece of work, and thanks for turning the two-client failure around so quickly with the per-program records.
I read the series at f1623ba. The lifetime rules hold up:
- Surfaces only reach
ops->unmapfrom the spare table. - Spares only come from programs whose TERMINATE/UNLOAD was acknowledged, that the firmware never saw, that were lost with an old firmware, or that were held until a verified power cycle.
- Mmapped surfaces wait for their last export.
- Both backends free pages and the range only after a full
iommu_unmap().
The records allocator, the epoch guard in h16_record_put() and ane_fw_transport_release() refusing to go below the placed range all look right to me. window_lock stays a leaf in both backends. Ryan's rows on the Neo and the M4 Pro at this head (two and three clients, sleep, lockdep together with #2505) cover what I was worried about. It merges cleanly with #2505 (now on integration as 1473d15), so no rebase is needed.
One small thing I'd like, as a fixup here or a follow-up:
-
h16_can_unmap()(ane_h16_program.c:743) treats "not asleep and not parked" as "engine up". On H17 a quarantined engine can be powered off while the session is open:- The QUARANTINED branch of
ane_fw_sleep_prepare()(ane_fw_cold.c:1068) powers down withoutane_h16_session_sleep(true). - A failed wake (
ane_fw_cold.c:1033-1039) reopens the session afterane_fw_quarantine().
If an export is dropped in that state,
h16_export_work()can trim spares and reachiommu_unmap()and the DART invalidation.ane_fw_power_down_staged()says that must not happen until ANE_SYS is back. It's narrow (it needs a lost program with a live mmap), but&& !s->quarantinedcloses it, and it also matches "a quarantined session keeps every mapping". Once the quarantine ends, the next LOAD trims as it does after a park. AnEXPECT_FALSE(f->s.quarantined)infw_unmap()would pin it. - The QUARANTINED branch of
Follow-ups, not blocking:
- When the spare table is full and unmap isn't allowed,
h16_give_spare()(ane_h16_program.c:786) still drops the surface for good.ane_h16_session_recovered()gives back every held program while the session is asleep, so a recovery with more than about 32 held surfaces leaks their memory and DVA until reboot. Keeping the overflow on a list for the next trim, or giving held programs back at the reopen, would close the last leak. ane_h16_alloc_prot()keeps the pages and the range wheniommu_map_sgtable()fails (ane_h16_backend.c:1100), even though the call already unmapped what it mapped. Under real memory pressure each failed retry pinssizebytes until reboot. Freeing them there looks safe to me.- The first bullet of the PR description (and the message of the first commit) still describes the single rewind mark that the last commit replaced. Our merges keep the in-series
Fixes:valid, so squashing is up to you, but please refresh the description so it matches the code. - A merge note: #2505 added a notifier-failure path in
ane_h16_backend_probe()that doeskfree(h)without thedrm_mm_takedown()you added to the other two. It's harmless (the window is empty there); I'm only mentioning it for symmetry.
Two rows are still open, and both are fine to run after merge: the H16 -ENOMEM flush-and-retry on hardware (on the session side, unmap_pressure covers the same branch) and a hang-recovery row at this head.
Approving. Thanks again!
|
Build check of head 811f6d3 (the merge with
Full build log, W=1 log and all module hashes: https://gist.github.com/RyanTheTide/53fde0a01ba687cd1204b44dc08daab2
This head's tree ( |
0a7c49b
into
aurora-silicon:integration/aurora
On H17 (
ane_fw_cold.c), program memory is never given back. Everyane_fw_program_map()andane_fw_alloc()takes DVA from one 512 MiB window through a bump pointer (b->next_dva) that nothing ever lowers, and every mapping stays onb->mappingswith its pages until reboot. The session layer (ane_h16_program.c) reuses a closed program's surfaces only for a later mapping of the same size and permission, and drops them once its 64-entry spare table is full. A workload with varied shapes takes fresh DVA for almost every program. On a J700, a sweep of random conv shapes made everyH16_LOADreturn-ENOSPCafter about 340 loads, even for a 6 KB program that had passed earlier. The memory behind the window stayed allocated too. The H16 backend (ane_h16_backend.c, M4 class) has the same pattern with a 1 GiB window.A second, smaller limit: LOAD, CREATE, TERMINATE, UNLOAD and gate commands are never released from the 256 KiB command arena (only EXECUTE batches are). That is 0x48c bytes per program, so about 210 load/close cycles fit in one firmware boot. After that
h16_pool_room()returns-ENOSPCuntil something re-inits the transport: a system sleep, a recovery, or an idle park (#1863).Six commits, one change each:
session_restart) drops the mark.ops->unmap.-ENOSPC/-ENOMEMfrees all spares and retries once.s->held, so a recovery releases it.ops->unmap, behaviour is unchanged. Both backends implement it in the commits below.drm_mm(DRM_MM_INSERT_LOW, 16K alignment) under a newwindow_lock, which also guardsb->mappings.ane_fw_program_unmap()callsiommu_unmap()(the DART TLB invalidation happens inside it), then frees the pages and the range.ane_h16_alloc_prot()put a mapping onh->mappingsbefore it allocated anything. When a 16K block, thevmap()or the sg table failed, the blocks already allocated stayed on the list, and after RUN nothing freed them: every program mapping that failed with-ENOMEMleaked its partial memory until reboot. The memory is now freed at once, and the mapping is recorded only just beforeiommu_map_sgtable(). A mapping thatiommu_map_sgtable()refused is still kept until reset, as before.h->program_next) becomes adrm_mmof offsets fromh->program_start(DRM_MM_INSERT_LOW, 16K alignment). A newwindow_lockguards it andh->mappings, since SHAREDMALLOC can be mapped while a program surface is unmapped.ane_h16_program_unmap()(ops->unmap) unlinks the program mapping, callsiommu_unmap()(the TLB invalidation of the three DARTs happens inside it), and only then frees the pages and the range. On a short unmap it keeps everything.ops->unmapwith the engine off.Fixes:the arena commit). A program still loaded pinned the rewind mark, so with one resident program every other client's load/close leaked 0x48c bytes again. A program's LOAD and CREATE now go into one of 128 fixed records (1088 bytes each, 136 KiB) taken from the arena at the first LOAD of a firmware boot, and the record is freed once the UNLOAD is acknowledged. A gate outside a batch, and TERMINATE + UNLOAD at a clean close, are released from the end of the arena once acknowledged, so the end of the arena is back at the records whenever nothing runs. A new firmware boot frees all records.The arena rewind assumes the firmware keeps no pointer into LOAD/CREATE command buffers once every program is UNLOADed. The EXECUTE arena already relies on the same assumption after completion. The device test below checks it.
Testing
KUnit (arm64 QEMU, KASAN/UBSAN): 265/265 on this branch. The new cases in
ane_h16_program_test.care:pool_rewindunmap_window-ENOSPC, and 2000 cycles pass withops->unmap.unmap_pressure-ENOSPCfrees the spares and retries.unmap_bytesunmap_parkedunmap_failed_mapunmap_refused_loadpool_residentpool_records_reuse,pool_records_full,pool_records_no_room,pool_placed_bounds,pool_records_restart-ENOSPC; no room at the first LOAD; placement bounds; a cold boot re-takes records and a lost program can't free a replayed program's record.ane_fw_cold.candane_h16_backend.care not built under KUnit; the backends'ops->unmapimplements the contract these cases test with a fake backend. A W=1 build ofdrivers/accel/ane/(the Neo config builds the H16 backend intoane_t6021) gives no warnings in the touched files, at every commit. The fulltools/asahi/run-host-tests.pygives the same result on this branch as on its base.H17
On a J700 (MacBook Neo, T8140), Aura 1.0.0b9, with ANE built in. These rows ran on a combined kernel, not on this exact head: it also carried #1768 (hang recovery), #1863 (idle park, with the close-while-parked fix) and a local non-contiguous allocation change. The third commit was resolved against that change: pages are freed with
dma_unmap_sgtable+vunmapand mapped withiommu_map_sgtable(). On plain integration/aurora it applies as in this PR. Booted withoutcma=.idle_park_ms=0-ENOSPC, no short unmap. The 6 KB program that used to fail still passes at the end, with byte-identical output.Kernel memory outside every
/proc/meminfobucket stayed flat across the run (about 104 MB before, 56–83 MB after 48 large loads). Nothing logged a WARN, an oops or a quarantine apart from the deliberate hang. (Two of my own compiler's conv tilings also hung the engine during the sweep. That was a bug in the tilings, not in this code, and the driver recovered both.)H16
Run on an M4 MacBook Air (J713, T8132; firmware 26.6.2 / 25G83), with the Air's kernel (07e8b02, before #2135) plus these commits cherry-picked, Aura Stage 2 with aura#260 for the firmware hand-off. Programs with surfaces that are not a 16 KiB multiple were run with 16 KiB-sized sections: on H16 the driver currently passes the exact section size in EXECUTE, which the firmware rejects for those surfaces (a separate bug, fixed by fbalazsy's 90cf680 from #1916).
H16 stage 5 program window 10001f00000..10041efffff, the same as without the PR, no WARN-ENOSPCon EXECUTE after about 200 cycles (the command arena; H16 never parks, so nothing re-inits it)-ENOSPC, 23 distinct DVAs,MemAvailableback within 8 MBstress-ng --vmhog-ENOMEMreached userspace, no OOM, memory fully back-ENOSPCat cycle 198Notes
.hungto the session ops next to the new.unmap, and accel/ane: T6050: pin the firmware code, and admit the 27.2 build #1957/accel/ane: T8142: admit the macOS 27.2 (26B123) firmware build #1899 touchane_fw_cold.caround preload. These are textual overlaps only; arm64: apple: t6041: support J616c (M4 Max) on macOS 27.2 (26B123) firmware #1954 and Hibernation pm_test dry runs on Apple silicon #1814 touchane_h16_backend.cand merge cleanly with this branch. I can rebase onto whichever lands first.--strictsays the newwindow_lockhas no comment of its own. The comment just abovedva_windowcovers it: the lock guards the DVA window and the mappings list.The commits have
Assisted-by: LLMand niteria'sSigned-off-by.🤖 Generated with Claude Code