Repository navigation
Conversation
|
Claude Opus 5.5 on @satchlj's behalf: hardware run.
Build: phellos (MacBook Air M2, J413, macOS 27.2 / 26B123 stub), default rebuilt 2026-11-20 = linux |
|
Thanks for the split. This step drops all the module parameters and the timed re-kick, and selects the path by G14G plus One point is left. We avoid sleep/poll loops where the firmware gives a signal:
Could completion and ring service run from the firmware's endpoint messages (as the existing G14 path does), with the idle poll gone, and |
|
Claude Opus 5.5 on @satchlj's behalf. @RyanTheTide: done in
One finding the old poll had hidden. With completion now sub-millisecond, the fixed round-robin over the 48 pool slots stopped at the first reuse: the firmware kept the retired command's tracking slot (+0x94) linked indefinitely, and a diagnostic run showed it linking slot 0 again after releasing it. Pump now takes the next slot that is neither in flight nor held. If all 48 were held, it kicks the retirement scan once and the next notification, submission or scheduler timeout retries. I don't know why the firmware holds slots this way; the PR body says so.
Trials d1–d7 were the failures that led here: the never-satisfied command-service condition (d1), then the held-slot stall at command 48 (d2–d7, including the diagnostic run). Each one stopped the firmware cleanly and returned unattended. Untested: everything in the body's Untested list, and in particular all 48 slots held at once, concurrent clients, and runs much longer than 303 dispatches. |
|
Thanks for 9454ef4, it covers what we asked:
The waits that remain are bounded, poll words the firmware writes without sending a message, and say so in a comment. That's fine. The held tracking-slot finding is worth keeping in the description, as you did. We'll run rows on our M2 (J413, 27.2) on this head: compute workloads, rmmod and reload, and one s2idle cycle. |
|
Claude Opus 5.5 on @satchlj's behalf. New commit f890a9c on this branch fixes a pool-slot stall in the notification rework (9454ef4). #2545 has it merged as 6c5d3db. Symptom. On phellos (J413, firmware 27.2) the 300 × 4096 compute check stalled at the first pool wrap on most boots. Every slot was still linked (+0x94), so the 49th command was never published: Bisect. The asahi tree of the up2 trial that passed (d8) is byte-identical to this branch, so the base kernel was not the cause.
Cause. Dumps of the 48 tracking slots and the queue's timestamp queue at the wrap (diagnostic module, 7 runs):
The firmware writes Fix. Tested on hardware. phellos (J413, macOS 27.2 stub 26B123), up3 default kernel 7.3.0-1-up3-ARCH, this branch's module built against it, loaded with no parameters, trial loader Host tests. Untested. Workloads other than this serial Vulkan compute check (many commands in flight at once, several clients). Render is still out of scope. The boot job's own tracking slot is never retired; it stays linked at the tail of the list, which these runs show is harmless. 🤖 Generated with Claude Code |
|
Thanks for f890a9c and the clear bisect. Advancing a slot's cursor only once the firmware has recorded We will run our M2 (J413, 27.2) rows on f890a9c rather than on 9454ef4: compute workloads, rmmod and reload, and one s2idle cycle. We'll post them here. |
|
Claude Opus 5.5 on @satchlj's behalf. Open finding at f890a9c (not fixed): s2idle with the GPU bound, then a hang around unload. On phellos (J413, 27.2, up4 kernel with this series, GPU published by aura#402 at 10373aae, aurora autoloaded), one boot, 2026-11-20:
So the 27.2 runtime has no suspend handling. The gfx domain can't reach its target while the firmware is up, and an unload after that may not be safe. Until this is handled, the series should keep the GPU unpublished by default. Next steps:
Untested: s2idle with the GPU idle but bound, without any later unload. 🤖 Generated with Claude Code |
|
Thanks for flagging the s2idle finding at f890a9c rather than leaving it for a reviewer to hit. Agreed that the GPU should stay unpublished by default until suspend is handled. For the fix, we would prefer a suspend callback that stops the firmware through its own shutdown handshake and starts it again on resume, over refusing suspend, so a bound GPU does not block sleep. If the stop path turns out to be what hangs after the s2idle, that would be good to know first. Our M2 rows on f890a9c will cover compute, rmmod and reload now; we will keep the s2idle row for the head that adds suspend handling. |
|
Claude Opus 5.5 on @satchlj's behalf. Suspend handling is in as 0aac788 ("let the G14 firmware-27.2 GPU through suspend-to-idle"), merged into #2545 as a8d3cd3. That merge only moves the callbacks' cfg gate to What hung, first. It was not the stop path. The driver had no callback for suspend-to-idle (its Your preferred design, tried first. Stop through the firmware's handshake (idle policy allow-off, then wait until it reports not running) and start it again on resume (keep-powered).
I did not go further into that. A cold restart (ASC halt, then start again) is not possible with this runtime, because its stop keeps the firmware's memory for the rest of the boot. What the commit does instead.
Rows (phellos, up4 default kernel df62a8c + our PRs, this commit as the module, aura#402 Stage 2 with the property). Each boot ran 3 × 1024 + 300 × 4096 Vulkan compute dispatches with every word checked, then one rtc1 s2idle, then the same again. No unload anywhere: rmmod after s2idle was never run.
For s3–s5, the test's watchdog pinger kept the test's SSH session open, so the reboot step never ran; WD1 reset the machine instead. A control boot without s2idle (s6) did the same, and s7 with the fix rebooted cleanly. Every boot logged "GPU held for sleep: submitted=303 completed=303 pending=0" and "released after sleep: running=Ok(1)". Untested: sleep with commands queued in |
|
Thanks for trying the handshake route first and for the clear account of why it can't work here. Knowing that the hang was genpd gating the domain under a running firmware, not the stop path, is very useful, and given the restart crash, holding publications, draining, and keeping the domain through We'll run our s2idle row on 0aac788 with the #2614 property: compute before and after, a few cycles in one boot, and the power drawn during the sleep, since the GPU now stays powered through it. |
949b48b to
259dd1a
Compare
0aac788 to
81446c7
Compare
259dd1a to
a111d34
Compare
81446c7 to
f5dfd26
Compare
DjDeveloperr
left a comment
There was a problem hiding this comment.
Pre-PR checklist review at head f5dfd2617395 (stacked on #2543).
Blockers
- §3 Safety — hardcoded SRAM voltage fallback and duplicated thermal/alarm literals (
drivers/gpu/drm/asahi/gpu/v27_2.rs).- In
RootV27_2::build()(aroundsram_mv), ifadt-perf-states-sramis absent under/chosen/aurora-gpu-resources/sgx, the code silently falls back toNone => (*mv).max(850). Per §3, safety/voltage limits must come from the machine (ADT/DT) and never fall back to a guessed constant; if the loader does not supplyadt-perf-states-sram,sram_table()must return an error (EINVAL/ENODEV) and refuse to start the GPU rather than inventing an 850 mV floor. - In
RootV27_2::controller(), thermal/alarm thresholds (11000,900,125) and sensor correction coefficients (5.3) are written as bare literals instead of being read fromcfg.da(unk_8cc,unk_87c,unk_e24) andcfg.unk_coef_a(whichhw/t8112.rsalready holds) or DT/ADT.
- In
- Concurrency bug —
Runtime::submit()vsRuntime::stop()race (drivers/gpu/drm/asahi/g14_v27_2.rs).
AsahiPlatformDriver::unbindcallsbackend.stop()before_reg(the DRM registration) is torn down, so DRM scheduler kthreads can still invokeJobImpl::run()->Runtime::submit().submit()checksif !self.healthy()outsideself.state.lock(), whereasstop()setshealthy = false, detachesnotify, drainsself.queue, and only then acquiresself.state.lock()toguard.take(). Asubmit()that passes the unlockedhealthy()check beforestop()flips it can acquireself.state.lock()afterqueue.drain()and beforeguard.take(), push topending, and callpump()— publishing a new command and ringingEP_DOORBELLwhilestop()is shutting down / halting the ASC.- Fix: re-check
self.healthy()after acquiringself.state.lock()insubmit()(and guardpump()onself.healthy()), returningErr(EIO)/Err(ENODEV)without mutatingpendingor publishing.
- Fix: re-check
- Partial multi-command publication & OOM-after-publish bugs (
drivers/gpu/drm/asahi/g14_v27_2.rs,pump()&service_locked()).- Drop of
VmBindafter firmware publication onENOMEM:pump()publishes the command to the queue ring and sendsEP_DOORBELLbefore callingstate.inflight.push(...). Ifpushfails withENOMEM,bind(mmu::VmBind) is dropped immediately while the firmware is executing the command on that ASID slot. Reserve capacity instate.inflightbefore publishing to the firmware sopushcannot fail post-publication. - Leaked
VmJobGuard/ premature quarantine on partial multi-command packet failure: if aPackethas multiple compute commands, command 0 is pushed ontostate.inflightwithlast = falseandpacket.mark_published()is set. If command 1 then fails inpump(),pending.remove(0)drops the packet,packet.release_guard()is skipped (becauseindex != 0), andpacket.finish(Err(e))immediately quarantinesvm_joband signals the error fence while command 0 is still executing. When command 0 subsequently retires inservice_locked(),if done.last { done.packet.retired(); }is false, sorelease_guard()is never invoked on retirement. Track the last published command (or remaining published count) per packet so retirement of the last published command cleans up the guard properly rather than leaking/quarantining it across a clean retirement.
- Drop of
- §1 Redundant chip-ID gate & firmware-version gate rationale (
drivers/gpu/drm/asahi/driver.rs,drivers/gpu/drm/asahi/g14_v27_2.rs).g14_v27_2::start()has a redundantmatch cfg.chip_id { 0x8112 => ... }check; attach the 27.2 register table / writable mask toHwConfig(or pass it cleanly fromcfg) rather than matching a literal chip ID in code.driver.rsgates on exactapple,firmware-version == [27, 2, 0]. Per §1, if an exact firmware-version gate is unavoidable (e.g., 27.3+ ABI may diverge), document why in the commit message and ensure it uses the narrowest firmware/layout property.
Non-blocking Notes
Runtime::sleep_prepare()currently wakes every 50 ms viaidle.wait_timeoutbecauseservice()signalsidleonly when bothinflight.is_empty() && pending.is_empty(). Signallingidlewheneverstate.inflight.is_empty() && (state.pending.is_empty() || self.asleep.load(Ordering::Acquire))removes the 50 ms polling wait during suspend drain.STATE_CAP = 100ingpu/v27_2.rscaps performance at P1 as a bring-up limit; make sure a follow-up tracks lifting it once higher operating points are verified.- Consider rebasing onto #2543 rather than carrying merge commit
66c50471once #2543 is updated.
Pre-PR Checklist (§1–§6)
- §1 No model gating: Blocked (see Blocker #4).
- §2 Production-ready, no hacks: Blocked by concurrency/error-handling bugs (#2, #3); no module parameters remain (good).
- §3 Safety: Blocked (see Blocker #1).
- §4 Clean-room & attribution: Pass. All 4 non-merge commits carry human author/committer and human
Signed-off-by:only, with no AI trailers; PR body discloses agent assistance. - §5 Evidence: Partial. CI checks on this base (
source,rust-build,kunit, DWC3/ATC matrix) are green (host-testsdoes not trigger against a non-integration base branch). Still needed on the final head: J413 regression counts (0 SELinux denials, 0 failed units, 0 new kernel errors),rmmodafter s2idle, and a G13/G14 13.5-stub no-regression boot row covering the touched shared files (driver.rs,gpu.rs,mmu.rs,fw/initdata.rs,event.rs,object.rs,regs.rs). - §6 Naming: Pass.
Assisted-by: LLM (Gemini 4 Neon)
GPU firmware 27.2 on G14 (the M2's G14G) takes a new root: four opaque records (status, control, power, an extra block) after the UAT level descriptors, and a runtime directory of 26.x-style channels in place of the 13.5 RuntimePointers. Add that root as G14 V27_2 and a compute-only runtime on it. - fw/initdata.rs: the raw InitData and HwDataB layouts gain V27_2 (a new AGX_V27_2 version group instantiates only these two), with layout checks against 13.5. - gpu/v27_2.rs: builds the root on the records of a 13.5 G14 manager (GpuManager::build_graph with GraphProfile::G14V27_2, which maps no registers and keeps the private range below the 27.2 timestamp aperture): hardware data from the 13.5 values, the chip's 27.2 register table (hw/t8112.rs), the power and thermal controller, the idle-policy records, and one queue with a kernel-owned boot compute job. - g14_v27_2.rs: start sequence and runtime. The driver's kernel window is mirrored into the loader's shared second-level table and named in TTBAT context 0 (mmu::FirmwareWindow); the ASC is started, the GPU held powered and the boot job retired; clients then submit through the 26.x front end (m3_submit), each compute command re-encoded in the 27.2 layout from a 48-slot pool and published in order. Completion follows the queue's retired position. The firmware's event, statistics, log and trace rings are consumed as they fill. Render commands get EINVAL. - driver.rs: a G14G whose DT apple,firmware-version is 27.2.0 takes this path, before anything starts the coprocessor. There are no module parameters. Inputs come from the DT (operating points, power and thermal properties, apple,coredump-copy-buffer, the SRAM voltages the loader publishes, the UAT regions), the GPU's ID registers and the T8112 chip configuration. Unbind stops the firmware (allow-off, RTKit power-off if it does not stop, ASC halt) and keeps its memory for the rest of the boot. Signed-off-by: Satya Benson <satya@benson.earth>
Review asked for no sleep or poll loops where the firmware gives a signal. The runtime polled from delayed work every 1 ms while busy and every 20 ms when idle, and dropped every endpoint message. The firmware sends endpoint 0x20 a message after it publishes event, statistics or log records. Each message now queues one run of the service, which consumes the rings and retires completed commands; there is no idle poll and no timer. The DRM scheduler's job timeout runs the service once more, so a command the firmware never completes still hits the command timeout. A firmware crash also runs it. The service runs on its own ordered workqueue. Stop waits for published work on a condition variable the service signals, detaches the message route so nothing queues more work, and drains the workqueue, replacing the fsleep loops and the fixed 20 ms sleep. Start waits for the firmware's ready status (+0x14) instead of a fixed 1 s, and fails if the command handlers are not installed in time instead of ignoring the timeout. Installation is acknowledged by the firmware clearing the request word (+0x4c24); the old wait also required the powered-stage word (+0x4c00) to read 2, which it never did after the handlers were installed, so it always ran out its 500 ms. A failed start names the step. These and the idle-policy and running-flag waits stay bounded checks of a shared word: the firmware writes it and sends no message, as with the existing G14 device-control channel. The completion stamp is written before the firmware's post-processing advances the queue's retired position, which sends no message. When the stamp is there and the position is not, the service waits for the position for at most 2 ms. Faster retirement showed that the firmware keeps a retired command's tracking slot (+0x94) linked for a while, and can link one again after releasing it; with a fixed round-robin the pool stopped at the first reuse. Pump now takes the next slot that is neither in flight nor held. If all are held it kicks the firmware's retirement scan once and the next notification, submission or scheduler timeout retries; there is no message for a slot's release. The stop line reports the notifications received, how often the position wait ran and how many held slots were passed over. Signed-off-by: Satya Benson <satya@benson.earth>
…are records it On phellos (J413, firmware 27.2) the 300-dispatch compute check stalled at the first pool wrap on most boots: every slot was still linked in the queue's tracking list (+0x94), so no slot could be reused and the 49th command timed out (submitted=48 completed=48 skipped-slots=480). Dumps of the tracking slots at the wrap show why. The firmware records the value a slot's retirement cursor must reach (+0xc, 1 for every use) in its post-processing, which can run after the queue's retired position advances. The service retires a command as soon as the position moves, so it usually read +0xc as 0 and wrote 0 to the cursor. The firmware then never unlinked the slot. In the boots that passed, the read happened to land after the record for one slot; only that slot was ever unlinked, and the pool reused it for the rest of the run (skipped-slots near 47 per command). The polled runtime before the notification rework retired 1 to 20 ms later and usually missed the window, which is why it mostly passed. Retirement now advances the cursor only once +0xc is recorded. A slot retired before that is kept in a pending set, and its cursor is advanced when the service next runs: on the firmware's next notification, submission or scheduler timeout. That is no fixed wait. The stop line counts the deferred cursors. Signed-off-by: Satya Benson <satya@benson.earth>
With the G14 firmware-27.2 runtime bound, a suspend-to-idle on phellos
(J413) left the GPU unusable. At the wake the GFX power domain did not
come back ("PS gfx: Failed to reach power state 0xf"), and the next GPU
access, a compute dispatch or an unbind, hung a CPU. The driver had no
callback for that sleep. Linux gated the domain under the running
firmware, as on J713 before apple,sleep-retained.
The domain is now kept through the sleep by the device tree
(apple,sleep-retained on the J413 firmware-27 GFX domain). Before
suspend-to-idle, the runtime holds new publications and lets published
commands retire, so the sleep starts with nothing in flight; after the
wake it publishes what was held. The firmware keeps running with its
keep-powered policy. The ASC, the root and every mapping stay as they
are.
Its own power-off handshake (idle policy allow-off before the sleep,
keep-powered after it) was tried and not kept: the firmware crashed in
the wake that followed it.
The sleep is refused (EBUSY) if published work does not retire within
the stop drain time. A sleep that powers the SoC down is still refused:
the firmware would be lost, and this runtime has no cold restart.
Signed-off-by: Satya Benson <satya@benson.earth>
f5dfd26 to
4c4f313
Compare
a111d34 to
119cce5
Compare
|
Re-checked at 4c4f313. This is a rebase onto the updated Assisted-by: LLM (Claude:Opus 5.5) |
|
Test rows from a second G14G machine: Mac mini M2 (J473, T8112) on the macOS 27.2 stub, GPU published by the Stage 2 from aurora-silicon/aura#402. Kernel: This stack, unchanged (state cap 100):
Two dispatches that hang the firmware at the first state, both deterministic:
After the timeout the runtime is marked failed and every later client gets ENODEV until reboot. A lab build that prints the MMU fault status on a timeout reads Performance states (lab builds, only the globals +0x98 cap changed):
So the cap tracks the state index in hundredths, states 1–5 run here, and the hang starts at state 6 (1236 MHz / 875 mV in the DT). A fixed 500 would be another magic number, so we are not proposing it as a change; it narrows where the 27.2 state-change inputs go wrong. Not tested here: render (refused, as expected), suspend, more than one client at a time. Row details
Assisted-by: LLM (Claude:Opus 5.5) |
Claude Opus 5.5 on @satchlj's behalf.
Second of three PRs that replace #2473: the compute backend as the G14 firmware-27.2 path. It builds on #2543. #2545 removes its build dependency on
CONFIG_DRM_ASAHI_M3_T8122.Selection. A G14G GPU (
apple,agx-t8112, the existing T8112 configuration) whose DTapple,firmware-versionis 27.2.0 takes this path inprobe, before anything starts the coprocessor. There is no board name and no extra compatible. Other firmware versions keep the existing code.What changes
fw/initdata.rs,rust/macros/versions.rs: rawInitDataandHwDataBgain aV27_2layout. A newAGX_V27_2version group instantiates only those two types for G14/V27_2. Compile-time checks pin the 27.2 offsets against 13.5.GpuManagerand the other versioned types are unchanged.gpu/v27_2.rs: the 27.2 root. It is built on the records of a 13.5 G14 manager (GpuManager::build_graphwithGraphProfile::G14V27_2: no register mappings, and the private range stays below the 27.2 timestamp aperture). It holds the hardware data from the 13.5 values, the chip's 27.2 register table (hw/t8112.rs), the power and thermal controller, the two idle-policy records, and one queue with a kernel-owned boot compute job. The firmware never sees the 13.5 root.g14_v27_2.rs: start sequence, runtime and DRM backend. It mirrors the kernel window into the loader's shared second-level table and names it in TTBAT context 0 (mmu::FirmwareWindow). It then starts the ASC, sends the root, holds the GPU powered and retires the boot job. Clients submit through the 26.x front end (m3_submit). Each compute command is encoded in the 27.2 layout from a 48-slot pool and published in order on the root's queue. The runtime runs from the firmware's endpoint-0x20 messages, which it sends after publishing event, statistics or log records: each one queues a run of the service, which consumes the event, statistics, log and trace rings (the firmware waits when one is full) and retires completed commands. Completion follows the queue's retired position, which the firmware advances in order. There is no poll or timer; the DRM scheduler's job timeout runs the service as the backstop for a hung command. Each command goes into the next of 48 pool slots that is neither in flight nor still held by the firmware (tracking +0x94). The scheduler has one run queue per priority. Render commands getEINVAL. Unbind lets published work retire, detaches the message route, drains the service's workqueue, then stops the firmware (allow-off, RTKit power-off if it is still running, ASC halt) and keeps its memory for the rest of the boot.CrashLog::from_firmware,GpuArray::shared_index,EventManager::try_get_polled, the Fender operating-point read,rtkit_dump.c(crash log to devcoredump), and the G14 skip of the 26.x CDM launch check, which decodes the G15 launch format.Against #2473
apple,coredump-copy-buffer, the UAT regions, the loader's per-state SRAM voltages under/chosen/aurora-gpu-resources/sgx), from the GPU's ID registers (core mask and count), and from the T8112 configuration (sensor masks, leakage table, preemption size, the 27.2 register table).t8112_dashboard_real=1overrodet8112_dashboard_ram=1, so slot 9 was always the real push-telemetry page mapped read-write, and the hand-saved snapshot was never read. The drm/asahi: T8112 G14G GPU on firmware 27.2: compute DRM device #2473 trial logs show slot 9 at 0x23b3d0000, RW. This PR deletes the RAM substitute and its seed files; nothing replaces them.mmu.rs.mmu.rskeeps onlyFirmwareWindow: map the loader regions, mirror, install, announce.Tested, this head (
f890a9c9, slot-cursor fix) on phellos (J413, firmware 27.2), the up3 default kernel (7.3.0-1-up3-ARCH), this branch's module built against it (through #2545), loaded with no parameters under the same watchdog arrangement, trials h1–h8: 8/8 PASS. Each ran 3 × 1024 and 300 × 4096 words, every word checked, 303 of 303 dispatches; "GPU stopped: allow-off=Ok(()) running=Ok(0) forced-off=None submitted=303 completed=303 notifications=307 late=0 skipped-slots=0 deferred-cursors=45…48"; cleanrmmod; unattended return. Cause and bisect are in the PR comment of 2026-11-20.Tested, previous head (
9454ef44, review rework) on phellos, J413 with firmware 27.2, the up2 default kernel (integration/aurora458f759 + open PRs; this module built from the #2545 head over it), loaded with no parameters under the same watchdog arrangement, trial d8: PASS (later, on the up3 kernel, this head passed only 2 of 7 boots: the pool-slot cursor race fixed by f890a9c), 3 × 1024 and 300 × 4096 words, every word checked, 303 of 303 dispatches; "GPU stopped: allow-off=Ok(()) running=Ok(0) forced-off=None submitted=303 completed=303 notifications=307 late=0 skipped-slots=11945"; cleanrmmod; unattended return to the default. The 300 dispatches took 77 ms (6.3 s with the old 20 ms idle poll). Start takes about 5 ms instead of 1.5 s.Tested, earlier heads on phellos, a J413 (M2 MacBook Air) with firmware 27.2. The packaged default kernel (
integration/aurora61295c2 + open PRs) loaded this PR's module with no parameters, under a watchdog with unattended return:DRM_ASAHI_G15andDRM_ASAHI_M3_T8122): PASS, 303 of 303 dispatches, 0 bad words; normal stop; cleanrmmod; return to the default;Each trial ran Vulkan compute through Mesa's Borealis driver: 3 × 1024 and 300 × 4096 words,
data[i] = data[i]*mul + add + i, every word checked on the CPU after each dispatch. That is 303 dispatches, past the 256-entry submission ring. Then a normal stop (allow-off acknowledged, firmware reported off, no forced power-off) and a cleanrmmod.The boot loader on those trials was a trial Stage 2 that publishes the GPU on 27.2. Its DTB names the GPU
apple,agx-t8112-identify,apple,agx-t8112; the first compatible is the loader's own trigger. The Aura in phellos's default (main + aura#381) leaves the GPU disabled on 27.2, so this path needs an Aura change before it runs on a default install. That change is not part of this series.Waits that remain. Each is a bounded check of a word the firmware writes and sends no message for: the loader's handoff answer, the ready status (+0x14), the command-handler installation (+0x4c24 cleared), the idle-policy acknowledgement and the running flag at stop (the existing G14 path waits for its device-control channel the same way), and, after a notification, at most 2 ms for the retired position when the completion stamp is already there (never needed in the trials:
late=0). The stop line reports the notifications received, those waits, and how many held slots were passed over.Untested
EBUSY), suspend, and GPU fault recovery.-identifycompatible.Assisted-by: LLM (Anthropic:Claude Opus 5.5)