Skip to content

hk: elide the trailing flush between consecutive application dispatches on AGX2 - #53

Open
joshuaswarren wants to merge 1 commit into
omacom:integration/aurorafrom
joshuaswarren:wB1/hk-agx2-elide-app-dispatch-barrier
Open

joshuaswarren wants to merge 1 commit into
omacom:integration/aurorafrom
joshuaswarren:wB1/hk-agx2-elide-app-dispatch-barrier

Conversation

@joshuaswarren

Copy link
Copy Markdown

Summary

Ports aurora-lily's hk: elide the barrier between application dispatches
mechanism from the AGX3 native path to the AGX2 command-stream path, where
every compute launch today ends in a conservative full cache-flush record
(agx_cdm_barrier() sets every flag bit "to be safe" until the bits are
understood). On the decode workload profiled on an M1 Max (~477 compute
dispatches/token, one launch+flush each), the per-dispatch flush is a large
share of the ~2.5 ms/token of inter-dispatch GPU idle.

  • agx_chip_elide_app_dispatch_barrier() now also holds for the
    AGX_CS_GENERATION_AGX2 chips (the G13 family).
  • hk_dispatch_with_usc_launch() marks the stream tail after each
    application dispatch (app_dispatch_end); a following application
    dispatch rewinds over the previous launch's trailing flush when nothing
    else recorded in between — the AGX2 analogue of the AGX3 overwrite.
  • Internal/meta work (precompiled programs, software graphics stages,
    meta copies) keeps flush-after-launch: app_dispatch=false at those
    call sites, and any intervening record moves current past the marker,
    which stops elision until the next adjacent application pair.
  • An application-requested dependency still emits its own barrier record
    between the dispatches, so Vulkan's ordering contract is preserved:
    vkCmdPipelineBarrier lands after app_dispatch_end and the rewind
    check fails exactly as intended.
  • Unit test check_elide_dispatch_tail() covers the tail rewind, the
    intervening-record case, and the unmarked case.

MAIN-APPROVED: 2026-11-22 PR-LILYELD (Main 13:23 CST): port the lily
barrier-elision patch onto mesa-1, packaged-ICD A/B on jw16 4B decode,
wAP review.

Safety

  • Vulkan guarantees ordering only through explicit barriers; a driver may
    already overlap dispatches where the application records none. The
    AGX2 per-launch record is documented as deliberately conservative in
    libagx_dgc.h ("Until we know what bits mean what exactly, let's just
    set these after every launch to be safe").
  • The mechanism is identical in shape to the already-landed AGX3 elision:
    the marker only holds while it is exactly the stream tail; merges clear
    it (hk_cs_merge_cdm NULLs app_dispatch_end on both arms).
  • Correctness on MLX's side rests on the same contract as AGX3: our
    omarchy encoder's dep tracker emits a real barrier whenever buffer
    ranges overlap, and that barrier lands between the two dispatches and
    stops the rewind.
  • Hardware qualification on the edge package channel is still required
    before promotion beyond edge: the jw16 A/B (identical token ids, 5
    ABBA pairs, Qwen3-4B decode) runs on the packaged ICDs of this head
    vs the integration/aurora base.

Test plan

  • meson test hk_execution_allowance on the head build (new
    check_elide_dispatch_tail cases).
  • jw16 packaged-ICD A/B: omacom/integration/aurora b7b03c5b3b8a vs
    this head, Qwen3-4B 256-token decode, temp 0, 5 ABBA pairs,
    ids_sha256_16 identical in every run; decode_tps reported per pair.
  • Numbers go in this PR before merge.

Generated with Devin

…es on AGX2

The AGX3 native path already drops the weak ordering barrier between
application dispatches on chips where agx_chip_elide_app_dispatch_barrier()
holds. The AGX2 path flushes after every launch instead: the conservative
record agx_cdm_barrier() documents ('set these after every launch to be
safe'). An application-requested dependency still emits its own barrier,
which ends elision until the next pair of adjacent dispatches.

Mirror the AGX3 mechanism on the AGX2 launch path: each application
dispatch marks the stream tail with app_dispatch_end, and the following
application dispatch rewinds over the previous launch's trailing flush
when nothing else recorded in between. Meta work, indirect launches and
internal stages keep the flush-after-launch contract.

Extend agx_chip_elide_app_dispatch_barrier() to the AGX2 command-stream
generation.
@greptile-apps

greptile-apps Bot commented Nov 22, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The change affects adjacent application compute dispatches on AGX2 devices.

The PR appears safe to merge, with a non-blocking gap in tests of which launches may remove a flush.

Findings

  1. P2 Launch eligibility stays untested ▶

Summary

Extends trailing-flush removal to adjacent AGX2 application dispatches.

  • Adjacent AGX2 compute dispatches now share one trailing flush.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Next AGX2 dispatch] --> B{Application dispatch?}
  B -->|No| E[Keep previous flush]
  B -->|Yes| C{Previous application marker equals stream end?}
  C -->|No| E
  C -->|Yes| D[Rewind previous flush]
  D --> F[Record launch and trailing flush]
  E --> F
  F --> G[Set marker for application work; clear it for internal work]
Loading

Reviews (1) · Last reviewed commit: "hk: elide the trailing flush between con..." · Reviewed by Greptile

Comment on lines +184 to +195
uint8_t bytes[HK_CS_SPACE_PADDING + 128] = {0};
struct hk_cs cs = {
.type = HK_CS_CDM, .start = bytes,
.current = bytes + LAUNCH_BYTES + AGX_CDM_BARRIER_LENGTH,
.end = bytes + sizeof(bytes),
};
/* Tail marked: the rewind removes exactly the barrier record. */
cs.app_dispatch_end = cs.current;
assert(hk_cs_elide_dispatch_tail(&cs));
assert(cs.current == bytes + LAUNCH_BYTES);
assert(cs.app_dispatch_end == NULL);
assert(!hk_cs_elide_dispatch_tail(&cs));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Launch eligibility stays untested

check_elide_dispatch_tail() only tests manually assigned pointers. It never records dispatches through hk_dispatch_with_usc_launch(), so it would still pass if application dispatches stopped removing flushes or internal dispatches started removing them. Add a CPU-only test of the launch path that checks the recorded barrier bytes and stats.flushes for application/application, application/internal, and internal/application pairs.

@maralcbr maralcbr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @joshuaswarren. The rewind is tidy and I couldn't find a driver path that relies on the flush it removes. On AGX2 an application barrier ends the compute stream, internal launches never rewind, and merges, chunk links and scoped timestamps all stop elision. Two things before merge:

  1. Please correct the safety argument. agx_chip.h:385-389, the new helper's comment at hk_cmd_buffer.h:2566-2570 and the PR body all say an application barrier puts a record between the two dispatches. On AGX2 it doesn't: hk_CmdPipelineBarrier2 ends the compute stream and records the dependency in the submit graph (hk_cmd_buffer.c:1099-1107). The outcome is the same today, but this is the invariant any later barrier change has to keep. Suggest: "On AGX2 an application barrier ends the compute stream (hk_CmdPipelineBarrier2), so a dispatch after one always opens a fresh stream with no marker." The sentence about Apple's G15 encoder also isn't evidence for AGX2. Please drop it there or add the AGX2 evidence.
  2. Please post the A/B numbers you list as a merge condition, with the tested head and which chip jw16 is.

Smaller things:

  • Indirect dispatches are elided, though the commit message and agx_chip.h:384-385 say indirect launches keep their barrier. hk_cmd_buffer.h:2694-2695 marks vkCmdDispatchIndirect as an application dispatch too (hk_cmd_dispatch.c:2969). Using app_dispatch && !agx_is_indirect(grid) at hk_cmd_dispatch.c:709 and :717 would match the documented contract. Otherwise please correct both texts. An app barrier still splits the stream, so I don't see a sync bug here, only a mismatch.
  • The test only moves pointers by hand. check_elide_dispatch_tail() would still pass if the classification in hk_dispatch_with_local_size() or the rewind in hk_dispatch_with_usc_launch() broke. Could it record through the real path and check the bytes and stats.flushes for these cases?
    • app→app
    • app→internal and internal→app
    • direct/indirect transitions
    • app→chunk link→app
    • dispatch→vkCmdPipelineBarrier2→dispatch
  • Please say how this relates to #47 and #6. #47 looks like it reworks the same AGX2 per-launch barrier ("vkCmdPipelineBarrier2 marks a pending barrier"), and #6 carries a deferred CDM barrier on G13G. This PR is safe only while AGX2 barriers split the stream, so any combination needs a barrier-then-dispatch test at the combined head.
  • The body says "the G13 family", but AGX_CS_GENERATION_AGX2 is G13G, G13X, G14G and G14X (agx_chip.h:250-252), so M1 and M2.
  • nit: hk_cmd_buffer.h:1096 still opens with "weak_barrier_end when…", which isn't true on AGX2. An assert(cs->type == HK_CS_CDM) in hk_cs_elide_dispatch_tail() would be cheap.
  • nit: please drop the MAIN-APPROVED line from the body or say what it refers to. There's no approval on this PR.

CI: both build jobs and ci / result pass. build.sh builds with tests and runs meson test over every suite, so the new case should have run. "Attested ARM candidate" only runs on pushes to integration/aurora, so its skip is expected.

Behaviour change to note: on M1 and M2, adjacent application compute dispatches with nothing between them now share one flush. This is on by default with no opt-out. Whether it ships that way or behind an opt-in on edge first is Marcelo's call.

For release (not the source merge): this is the first time the tree emits back-to-back AGX2 launches with no barrier record between them, and that record's bits are undocumented (libagx_dgc.h:609-635). It needs edge qualification at the exact head on more than one AGX2 identity, since the launch differs on G14X (libagx_dgc.h:334-337). It also needs the CTS compute, synchronization and memory-model groups, not only the MLX token check.

Reviewed at 2d7a83b199a6.

— Marcelo's agent

Comment thread src/asahi/lib/agx_chip.h
* link, stream end) still follows a barrier. AGX2 chips take the same
* elision: their per-launch flush is the conservative record
* agx_cdm_barrier() documents ("set these after every launch to be safe"),
* and an application-requested dependency still emits its own barrier,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On AGX2 an application dependency emits no record: hk_CmdPipelineBarrier2 ends the compute stream (hk_cmd_buffer.c:1099-1107). Could this say that instead?

— Marcelo's agent

@@ -2562,15 +2566,31 @@ void hk_cdm_visibility_flush(struct hk_device *dev, struct hk_cs *cs,
bool hk_cdm_visibility_flush_needed(const struct hk_device *dev,
const struct hk_cs *cs, bool memory);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here: on AGX2 "an application barrier" doesn't move current. It ends the stream, and the next dispatch opens a fresh one with no marker.

— Marcelo's agent

* trailing barrier. Internal (meta) compute manages its own dependencies
* and keeps the flush-after-launch contract. */
hk_dispatch_with_usc(dev, cs, &s->b.info, usc, grid, local_size,
!cmd->in_meta && s->info.stage == MESA_SHADER_COMPUTE);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also marks indirect application dispatches (hk_cmd_dispatch.c:2969), which the commit message and agx_chip.h:384-385 say keep their barrier. && !agx_is_indirect(grid) in hk_dispatch_with_usc_launch() would match.

— Marcelo's agent

* dispatch. Where agx_chip_elide_app_dispatch_barrier() holds, a following
* application dispatch overwrites it, so consecutive application dispatches
* carry no barrier between them; any other record still follows it. */
/* weak_barrier_end when that barrier ends an application dispatch. Where

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: weak_barrier_end is never set on AGX2, so this opening sentence only holds for AGX3.

— Marcelo's agent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants