Skip to content

drm/asahi: g18: async submit follow-ups (comment, rate limit, device cap, teardown) - #2850

Open
sithglan wants to merge 2 commits into
aurora-silicon:integration/aurorafrom
sithglan:thomas/g18-async-followup
Open

sithglan wants to merge 2 commits into
aurora-silicon:integration/aurorafrom
sithglan:thomas/g18-async-followup

Conversation

@sithglan

@sithglan sithglan commented Nov 22, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #2679: the four points from the review there (comment).

Changes

  1. Comment (g18/sync_input.c). dma_fence_chain_walk() unlinks the earlier points of a timeline once they have signalled. A job's inputs have all signalled when it runs, so of a timeline only the head point's own fence is checked. The comment now says that instead of "each fence is checked".
  2. Rate limit. A failed asynchronous job is reported at most once a second, with a separate line counting the failures not reported in between. This follows the fault_log_ratelimit pattern in gpu.rs. The output fence and the queue's next submit still carry every error.
  3. Device-wide cap. The device holds at most 256 accepted jobs waiting to run, over all queues and files (DEVICE_DEPTH). The per-queue cap stays 32. JobQueue::wait_room() waits interruptibly before the submit publishes anything, next to the existing per-queue wait, so a signal restarts the ioctl. The cap is soft: submits that wait together can each still add one job.
  4. Teardown. Jobs run on the device's own unbound workqueue (aurora_g18_jobs) instead of system_unbound. When G18DrmGpu is dropped, the queue is taken out of the job state and destroyed, which waits for the worker to return. So no work item of the module can still be running when G18DrmGpu goes away. A submit after that gets -ENODEV. The drop never runs on the worker itself: a job holds the runtime, the VM and the queue's error flag, but not G18DrmGpu, and a queue waits for its last job before it is released.
    • On the J873 today, the drop only happens on the probe error path after G18DrmGpu::new(). Once the firmware session starts, the driver pins its own module until reboot (retain.c), and the G18 platform driver suppresses bind attributes. So neither rmmod nor unbind can reach this path on hardware; see the row below. The drain covers that error path and any later removal support.

Testing

  • arm64 build with LLVM=-22: drivers/gpu/drm/asahi compiles with no new warnings. The 20 dead-code warnings are all in other files and are there at the base too.

  • checkpatch.pl --strict: clean on both commits.

  • CI host-tests: pass. No host test covers g18/drm.rs.

  • M6 (J873g) row. The M6 desktop GPU module (the failfast state) was run A/B, with and without the runtime commit of this PR. It applied unchanged; the comment commit has nothing to apply to there, because that tree predates asahi_g18_fence_error(). Mesa is the system build with mesa#601, so GL queues are async (glFlush returns in 0.0 ms). The test tool is g18burst, a raw-ioctl test.

    Test Without (A) With (B)
    burst: 320 sync-only jobs over 16 async queues behind an 80 ms draw, 3 runs all submits returned at once (0.1-0.2 ms in all) exactly submit usb: typec: tipd: Reconcile initial SPMI events #256 waited for the draw (79.1, 80.3, 80.6 ms); every other submit returned at once
    burst: 600 jobs over 64 queues all 600 accepted at once submit usb: typec: tipd: Reconcile initial SPMI events #256 waited 80.7 ms, the rest at once
    flood: 40 jobs failing after their submit returned, then 1 more after 1.2 s 41 G18 job failed lines 1 line, then G18: 39 more failed jobs not reported with the 41st
    next submit returns each late failure 41 of 41 41 of 41
    • On B, all outputs signalled, and no submit failed.
    • synctest (indirect dispatch, XFB, GS, render to compute, shared context): 5 x 50/50, GL error 0.
    • Unload: with sway stopped and no client left, rmmod aurora is refused, because the module is pinned for the firmware session. /sys/bus/platform/drivers/aurora-g18/unbind does not exist. That is expected on this driver, so the drain is not exercised on hardware.
    • The M6 is back on its default module afterwards.

Assisted-by: LLM (Claude:Opus 5.5)

asahi_g18_fence_error() walks a dma_fence_chain with
dma_fence_chain_walk(), which unlinks the earlier points that have
signalled. A job's inputs have all signalled when it runs, so of a
timeline only the head point's own fence is checked, and an error on an
earlier point is not seen. The comment said every fence was checked.

Signed-off-by: Thomas Glanzmann <thomas@glanzmann.de>
Three follow-ups to the asynchronous submit path (aurora-silicon#2679):

- A failed job is reported at most once a second, with the number of
  failures not reported in between. A client that submits failing jobs
  in a loop could otherwise flood the log; the output fence and the
  queue's next submit still carry every error.
- The device holds at most 256 accepted jobs waiting to run, over all
  queues and files. Each queue was bounded at 32, but a file can open
  any number of queues. A submit waits for room before it publishes
  anything, so a signal restarts the ioctl as the existing waits do.
  The cap is soft: submits that wait together can each add one job.
- Jobs run on the device's own unbound workqueue instead of
  system_unbound. When G18DrmGpu is dropped, the queue is taken out of
  the job state and destroyed, which waits for the worker to return, so
  no job work item outlives the device. A submit after that is refused
  with -ENODEV. On the J873 this happens only on the probe error path
  today: once the firmware session starts, the module is pinned until
  reboot and the platform driver has no bind attributes.

Signed-off-by: Thomas Glanzmann <thomas@glanzmann.de>
@RyanTheTide

Copy link
Copy Markdown
Contributor

Review at 6252dbc.

Thanks, this addresses the #2679 points cleanly. The teardown ordering looks right to me: the worker holds only Arc<JobQueue>, shut_down() drops the queue outside the state lock, and push() refuses with -ENODEV once workers is gone. A few things:

  1. g18/drm.rs:487: the job worker signals dma_fences, and memory reclaim can wait on those. Since this is now a dedicated queue, could it be OwnedQueue::new_unbound_reclaim() (WQ_MEM_RECLAIM)? g17 does the same for its submit queue (g17/job.rs:584, new_ordered_reclaim). system_unbound had the same gap, so this is not a regression, but now it costs one word to fix.
  2. g18/drm.rs:543-549: wait_room() drops the lock before push(). So every submitter woken at 255 gets through, and the bound is 256 plus the number of threads submitting at once, which userspace controls. A reserved count, taken in wait_room() under the lock and given back in push() and its error paths, would make it a hard cap for very little code. If you would rather keep it soft, please say in the comment why that is acceptable.
  3. g18/drm.rs:355: please say in the comment what DEVICE_DEPTH bounds (kernel memory and mappings pinned by accepted jobs?) and why 256 (8 x QUEUE_DEPTH?). Also, the cap is device-wide, so one file can fill all 256 slots and then block another client's submit ioctl (a compositor, say) in wait_room(). Its job would run after those 256 anyway, but before this change the ioctl itself did not block. A per-file share, or at least a note that this is intended, would help.
  4. Nit, g18/drm.rs:553: fault_log_ratelimit() in gpu.rs never throttles when VerboseFaults is set. It would be good to do the same here, so the two reports behave alike.
  5. Testing: the M6 A/B ran on a different tree, where the first commit did not apply. Could you rerun g18burst (both burst cases, plus the flood) and synctest on J873 at this head, 6252dbc, ideally with CONFIG_PROVE_LOCKING, as there is a new state-lock user on the worker? A short sway/GL session would also help.

The comment change in sync_input.c matches what dma_fence_chain_walk() does. Other GPU generations are untouched: both files are under g18/, and the backend binds only apple,agx-t8152.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants