Skip to content

soc: apple: vote GPU activity to the PMP on T6001 — packaged default - #515

Open
joshuaswarren wants to merge 19 commits into
omacom:integration/aurorafrom
joshuaswarren:w72/t600x-agx-vote
Open

joshuaswarren wants to merge 19 commits into
omacom:integration/aurorafrom
joshuaswarren:w72/t600x-agx-vote

Conversation

@joshuaswarren

Copy link
Copy Markdown

Summary

Re-opens the T6001 AGX PMP vote as the packaged-default change. Closed #334 carried the same head (b36d0192); its content was consolidated into aurora-silicon/linux as #1290 (squash 9bec84ba, merged 2026-11-09). omacom/linux-aurora — the tree the linux-aurora package pins — still lacks it, so every packaged-kernel user on an M1 Max runs GPU workloads with memory held at the PMP idle floor.

The patchset: the asahi driver reports GPU activity to the PMP (apple_pmp_agx_activity_begin/end), and pmp_report requests the AGX power bit (soc-device record 9, PS-REQ bit 9) while work is in flight, releasing it 250 ms after the last operation. The pmp_report.agx_vote sysfs param remains as a manual override. T6001 only.

Measured A/B (fresh, this PR's evidence)

jw16 (M1 Max, T6001), the P23 scorecard protocol — one load, 5 fresh-cache rounds, ctx 512, 128 greedy tokens, ABBA arms through the serial GPU queue, identical generated-token digests in every arm:

model vote off vote on delta
Qwen3-4B (4-bit) 54.03 tok/s 67.31 tok/s +24.6%
Qwen3.8-2B (4-bit) 97.08 tok/s 119.72 tok/s +23.3%

Receipt: omarchy-mplus-private receipts/2026-11-22-w7W-jw16-agx-vote-ab.md (commit 58963b408), run dir jw16:~/scratch/w7W-4b-profile/vote-ab-20261122T180803Z. Prior independent evidence: Nov-3 matrix +24.6–33.8% decode, Nov-10 timed A/B 56.55→71.85 tok/s at ctx512; the microbenchmark basis in #334 (241→368 GB/s read roof).

Test

In-tree coverage rides in the head: drivers/pmdomain/apple/pmp-report-test.c (KUnit, .kunitconfig.apple-pmp) covering the vote policy state machine — including the five cases the #334 review exercised plus the regression cases added for the begin-during-clear race and the retry residual. The extracted-C policy reproducer (tools/asahi/t6030-pmp-graph-tests.c) is also in-tree.

Battery behaviour

The vote is asserted only while GPU operations are in flight, plus a 250 ms idle hold; it clears on its own after that — no daemon, no userspace. On battery, a decode burst holds the higher memory state for the burst plus ≤250 ms, and the burst itself finishes ~25% sooner, so energy per token is roughly a wash while interactive latency improves. Between bursts the request is absent. System sleep: report entries already withdraw requests in suspend_noirq; the vote follows the same path, so nothing is held across s2idle. Unbind stops the vote and refuses new begins (the stopped flag added for review finding 3). Setting pmp_report.agx_vote=0 restores the old behaviour at run time; =1 forces the request.

State of review

This head was re-reviewed in #334 (scottjones, 2026-11-11): the original blockers (boot-path NULL workqueue, begin-during-clear, unbind race, sleep question, in-tree coverage) are all addressed at b36d0192. It is ~2 weeks behind current integration/aurora; happy to rebase onto the current head on request. Cross-family review of this head requested separately (wAP).

Generated with Devin

…n t600x

On the M1 Max (t6001) the PMP keeps memory at its idle floor for a busy
GPU until the AGX power request, bit 9 of the PMP report's PS-REQ word, is
set. Nothing sets it on Linux, so GPU reads top out at 241 GB/s. The same
chip reads 361 GB/s on macOS.

Report GPU activity to the PMP. The GPU driver calls
apple_pmp_agx_activity_begin() when a submission starts and
apple_pmp_agx_activity_end() when it finishes. pmp-report sets bit 9 when
the first operation starts and clears it 100 ms after the last one ends.
It also clears it at once when the GPU driver unbinds or the report goes
away, so the request is never left set past the GPU's power-domain
release. The register sequence is the one apple_pmp_report_set_state()
already uses for report entries: read 0xf80, write 0x107c0, poll 0x1000,
status 0x10, 50 ms acknowledge. A set that is not acknowledged is cleared
again and not retried for a second.

The vote is off by default. pmp_report.agx_vote=1 turns it on, and
pmp_report.agx_vote_idle_ms (default 100) sets the hold. With it off
nothing is written. It applies to apple,t6001 only; t6000 is untested.

The decisions (set, clear, hold, retry, wrap of the 32-bit millisecond
clock) are plain functions in pmp-agx-vote.h with no kernel dependency.

Setting the bit by hand with a test module on a 16-inch M1 Max moved a GPU
read microbenchmark from 241 to 368 GB/s. This patch has not yet been run
on hardware.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…one power request bit

The PS-REQ word of the PMP report is shared by every client of the PMP:
the report entries (display, ISP, video decode, the AFNC IOAs), the T6001
AGX vote (bit 9) and, out of tree, the ANE clock level of omarchy-ane
(bit 10). The report entries and the AGX vote read-modify-write the word
under rep->lock. A module outside this driver could only map the window
and read-modify-write it without that lock, so a concurrent write of
another bit could be lost.

apple_pmp_report_set_request(report_np, id, enable, no_ack) is the one
writer a client outside this file may use. It calls the same
apple_pmp_report_set_state() the entries use, so it takes the same lock
and does the same acknowledge poll (at most 50 ms). The caller owns the
bit it asks for.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…ment clearing

apple_pmp_report_set_request() lets a client outside pmp-report update one
bit of the PS-REQ word. On the T6001 bit 9 is the AGX request, which the
GPU vote owns: a second writer could clear it while the GPU works, or set
it and leave it set. Refuse it from the export with -EBUSY on a t600x
report. The vote itself writes through apple_pmp_report_set_state() and is
not affected.

Document what a caller must do after -ETIMEDOUT: the request was written
but not acknowledged and stays in the word, so the caller clears it with
@enable false and @no_ack true before giving up.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…st operation

A 9B model at 8192 tokens of context had one gap of up to 111.5 ms between
two decoded tokens in every process, once per run, with the GPU idle for a
good part of it. The 100 ms default hold dropped the request in that gap,
which costs one slow token while the PMP raises memory again. The longest
gap in any other configuration measured was 64 ms.

Raise the default to 250 ms, about 2.2 times the longest gap seen. The cost
is a few hundred milliseconds of the higher memory state after the last
job. pmp_report.agx_vote_idle_ms still overrides it at run time.

The default is one constant, AGX_VOTE_IDLE_MS_DEFAULT, used for both the
parameter and the initial state. The userspace test has a case that rides
through a 112 ms gap with the default and still drops the request after a
long idle.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
… it across unbind and sleep

Review of the AGX vote found four problems.

1. agx_vote=1 on the kernel command line runs the parameter setter from
   parse_args(), before workqueue_init_early() has created the workqueues,
   and the setter queued work on a NULL workqueue. kick() now returns while
   system_dfl_wq does not exist; begin() and the work function read the
   parameter themselves, so the first GPU operation votes. It also uses
   system_dfl_wq instead of the deprecated system_wq.

2. An operation that began while the clear was being written saw voted
   still true and did not kick; after the clear nothing scheduled a set, so
   the GPU ran at the idle floor until the next begin. agx_vote_next()
   now reports a set that is owed (1 ms) when users > 0, the request is not
   voted and no set has failed.

3. release() did not stop a later begin() from setting the bit again, and
   the report's removal had the same shape. A stopped flag, set under the
   vote lock at release and at report removal, is checked by begin(),
   decide() and next(); it is cleared by a new
   apple_pmp_agx_activity_resume() called at GPU probe and at report probe.

4. The request is withdrawn on PM_SUSPEND_PREPARE and PM_HIBERNATION_PREPARE
   and decided again on PM_POST_SUSPEND and PM_POST_HIBERNATION: holding the
   bit through s2idle may cost power (a held display request cost about
   0.25 W on J514S), and the PMP may drop the word across sleep, which would
   leave voted true with nothing on the wire.

Each case has a test in tools/ptd-vote of the research repository that
carries the policy header (cases 14 to 17 of the vote simulator, and a source
check for the boot-time kick); they were written to fail first.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…d report probe now uses

The report probe tells the AGX vote that a T6001 report came back
(apple_pmp_agx_activity_resume) and compares rep->offsets with the T6001
table. The harness extracts the probe verbatim and compiles it standalone, so
production.h failed with an undeclared apple_pmp_offsets_t600x and an
implicit declaration of apple_pmp_agx_activity_resume.

An empty T6001 table that never matches and an empty resume function are
enough: the harness models the T6030 graph. All 21 graph cases pass, and the
supplier harness passes (ASan needs ASLR off on this host: setarch -R).

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…ontrols

The vote policy is a header of static inline transition functions, so the
suite compiles pmp-agx-vote-tests.c against a copy of it and drives the same
begin, decide, next and applied calls the kernel glue makes, with a fake
clock, a fake PMP bit and a fake timer. 18 cases: the parameter off, the set
and idle drop, decode-like bursts, a drop cancelled by new work, the failed
set backoff, unbalanced ends, the parameter switched off mid-flight, the zero
idle tick, millisecond clock wraps, release and refuse, resume with and
without work in flight, sleep and resume, and the two asserts from review
(the begin inside a clear write must be re-set, and begin must refuse while
stopped).

--mutation and --mutations remove one guard each (the owed-set branch, the
!failures in it, and the stopped check in begin and decide) and require the
suite to fail. The two mutants that survived before the races fix (begin not
refusing while stopped, and the retry deadline indistinguishable from the 1 ms
owed-set kick) are the stopped-in-begin and owed-set-branch controls: at
5b54de3 (the pre-fix header) both survive because the guarded lines do not
exist; at this head each is caught at the case that pins it.

Registered in tools/aurora-ci/host-tests.json (base, --mutations, four
single-mutant entries, the .c as a helper).

Proof: plain run passes (18 cases, AGX-VOTE-SELFCHECK-OK); --mutations exits 0
with all four caught; --ref 5b54de3 --mutation stopped-in-begin and
--mutation owed-set-branch print SURVIVED there.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…he mutation modes

Review of 1142df9 by w7X, two non-blocking items folded in before push.

Case 19 pins the signed 32-bit compare in agx_vote_reached: with idle_at
0xFFFFFFF0, idle_ms 100 and voted, decide at 0xFFFFFFF8 must be NONE and next
must return the 92 ms to the wrapped deadline, and a retry_at that has wrapped
must not read as reached either. The wrap-reached mutant (now >= t) survives
the old cases because they jump straight to the wrapped deadline; it dies at
line 306. Registered as a fifth mutation entry.

The mutation modes now assert the unmutated suite passed first, so a base
failure cannot make every mutant read as caught. The negative control (--ref)
is exempt: the pre-fix header cannot compile this suite, which is the point.

The suite runs under setarch -R when available; ASan startup crashes under
ASLR on a loaded host otherwise.

Proof: plain run passes; --mutations exits 0 with all five mutants caught;
wrap-reached CAUGHT at line 306; at 5b54de3 the owed-set-branch and
stopped-in-begin mutants still print SURVIVED.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…ff kick storm and the revived retry

Three cases for chriskearney's findings on AsahiLinux#334, written first and red at
this commit:

- Case 20: an end lands between decide (SET) and applied. The end sees
  voted false, arms nothing, and applied then sets the vote with no user and
  an already-reached idle deadline. next() must return 1 there, not 0: 0 is
  the suite's "nothing pending", so the caller would never run the reconcile
  that clears the vote.
- Case 21: an idle gap beyond half the 32-bit millisecond range makes the
  wrapped arithmetic read an expired retry deadline as future, which would
  keep the vote off for days. A retry deadline more than one backoff window
  in the future cannot be real and counts as reached.
- Case 22: during backoff, begin must not ask for a reconciliation decide
  would refuse; the retry timer already drives the set.

Three mutants pin them (set-write-ends, retry-revival, begin-backoff-kick),
and stopped-in-begin now matches the begin line the fix will produce. Red
here: case 20 fails at `agx_vote_next(&v, 5001) == 1`.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…try deadline and quiet the backoff kicks

Three findings from chriskearney's review of AsahiLinux#334; the suite cases landed
first (825e9ac, red) and this is the smallest change that turns them green.

- end() during the SET write (between decide and applied) sees voted false
  and arms nothing; applied then leaves the vote set with no user and an
  already-reached idle deadline, and next() answered 0, which the caller
  reads as nothing pending. An already-reached deadline now returns 1, so
  the caller runs the reconcile that clears the vote. It settles one step
  later: decide CLEAR, applied, next 0.
- A retry deadline more than one backoff window in the future cannot be
  real (an idle gap beyond half the 32-bit millisecond range reads an
  expired deadline as future through the wrapped compare), so the new
  agx_vote_retry_reached() treats it as expired instead of keeping the vote
  off for days.
- begin() no longer asks for a reconciliation while a failed set is in
  backoff: decide would return NONE, and the retry timer already drives the
  set. One kick per operation during the window was pure waste.

The mutants in tools/asahi/pmp-agx-vote-tests.py pin each: set-write-ends
dies at the race assert, retry-revival at the bound assert,
begin-backoff-kick at the kick-count assert, and the five earlier mutants
still die. Plain suite: 22 cases, selfcheck ok.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…ot strand or spam the vote

w7X's review of the first cut (a7330a5) found a blocking regression and two
correct behaviours to keep. The gate on begin had to know the time, so
agx_vote_begin() now takes now (the caller reads it under its lock; the glue
passes apple_pmp_agx_now_ms() beside its existing end and next reads):

- begin returns false during a failed-set backoff before the deadline (decide
  would return NONE; the retry timer drives the set), and true again once the
  deadline has passed. The first cut's unconditional !v->failures stranded the
  vote after one failed set whose retry timer fired idle: failures is only
  cleared by a successful SET, so every later begin refused forever. Case 23
  is that sequence (failed set, end before the retry, timer fires idle, a
  later begin must kick and the set must land); the begin-ignores-retry
  mutant is exactly the first cut's expression and dies at it (line 363).
- The other direction is pinned too: begin-backoff-kick (kick during the
  window regardless of the deadline) dies at the kick-storm case (line 357),
  so the gate cannot silently decay into "always kick" again.

Case 23 exists because the simulator cases kept the operation alive through
the backoff; this one ends it first. The begin signature change touches the
suite's helper, four header-only call sites, and the one glue call.

Proof: plain suite 23 cases selfcheck ok; --mutations catches all nine
mutants (owed-set-branch, owed-failures, stopped-in-begin, stopped-in-decide,
wrap-reached, set-write-ends, begin-backoff-kick, begin-ignores-retry,
retry-revival); check-kick-guard still passes on pmp-report.c; the tree
runner passes the registered vote entries.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
w7X's re-review nit: next() takes the first match, so the second entry could
never be selected. Nine distinct mutants; --mutations still passes.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…ommit)

chriskearney's second blocker at efff25c: the last operation ends during a
SET ack that then times out. applied records the failure with no user and no
timer (next returns 0 on that path), and a begin before the retry deadline
must arm the remaining delay instead of reporting no kick, or the retry never
runs. The case also walks the release/resume cycle: an idle release does not
write a clear, an idle resume does not kick, and a begin past the deadline
still kicks.

Red here: the begin result is 0 where the remaining 998 ms delay is required
(FAIL line 397).

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…e path could not

chriskearney's second blocker at efff25c: when the last operation ends during
a SET ack that then times out, applied records the failure with users already
0, so next() returns 0 on that path and no timer is armed. A begin before the
retry deadline reported "no kick" (the clock gate from 2ec389c), so nothing
ever ran the retry: the vote stayed off until some unrelated reconcile.

agx_vote_begin() now returns the same delay contract as agx_vote_end(): 1 run
the reconcile now, N arm it N ms out, 0 nothing pending. A begin inside a
failed-set backoff arms the remaining delay; past the deadline it returns 1 as
before; a voted overlap, stopped or disabled still return 0. The glue passes
the delay to the kicker instead of a boolean, and an idle release/resume cycle
does not clear the retry state, so the next begin still arms it.

Case 24 (fb8cba3, red at line 397) walks the whole sequence including the
release/resume variant. Controls re-pointed at the new return: begin-no-retry-
arm (returns 0 where the delay belongs) and begin-backoff-kick (returns 1,
the kick storm) both die; begin-ignores-retry is gone because the delay branch
subsumes it.

Proof: plain suite 24 cases selfcheck ok; --mutations catches all nine
mutants; the tree runner passes the registered entries.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
The residual w7X flagged on the delay contract: the end lands inside a failed
SET write and no new operation starts. The write path arms nothing (users is
already 0 when applied runs), so failures lingers with no timer, and a begin
before the deadline reports "no kick". Case 25 walks it: the retry must be
armed from the failure state itself, and when the retry fires with nobody
waiting the backoff must be cleared once instead of re-arming forever. The
header-alone asserts pin next(): 500 at 500 ms before the deadline, 1 at it.

Red here: FAIL line 424, s.armed is false after the race.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…the operation

The residual from the delay-contract review: when the last operation ends
inside a failed SET write, applied records the failure with users already 0,
and next() returned 0 on that path, so no timer existed and a begin before the
deadline reported "no kick" (2ec389c's clock gate) - the retry never ran.

next() now arms the retry from the failure state itself (users 0, not voted,
failures set): the remaining delay, or 1 when the deadline is already reached.
That alone would re-arm forever once the retry fired idle, so the work
function also clears a spent backoff: a reconcile that finds nobody waiting
and an expired deadline (decide returned NONE) sets failures to 0, and the
state settles. A later operation then starts a fresh attempt, exactly like
the first.

Case 25 (4574c14, red at s.armed) walks the sequence through the
release/resume cycle; the retry-dropped-when-idle mutant removes the new
branch and dies there. The sim's begin helper now arms the returned delay
(delay 1 is "due now" within a tick), case 23's spent-backoff assert and
case 24's next() assert updated to the new contract. Plain suite 25 cases,
ten mutants, all caught; the tree runner passes the registered entries.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
Treat retry deadlines beyond the backoff window as expired using unsigned distance. Cover the exact half-range boundary and keep the mutation control aligned.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
chriskearney's re-review at c3487a5 reproduced a stranded retry: with the
timer already armed by the write path, a begin before the deadline returned
the remaining delay, and a caller that queue_works the delay pushes the
deadline out. mod_delayed_work only pulls a deadline earlier, so the pushed
timer would fire after retry_at and reconcile a decide that returns NONE.

begin() now returns the two-value contract only: 1 when the deadline has
passed, 0 otherwise. The retry arming belongs to next() alone, which already
arms from the failure state (85a68bf) and is monotone under the same
queueing rule. A begin during the window cannot strand the vote and cannot
delay it.

Case 26 walks the repro end to end (end during a failed SET ack, begin
before the deadline, vote lands inside the new operation) and pins begin at
the deadline edge and 1 ms before it. Case 22 gains the monotone-deadline
assert. The begin-no-retry-arm control now reads the real return, and the
new begin-drops-users mutant (drop the users++ behind the stopped check)
dies at the base case. The tree runner registers the mutant entry.

Proof: plain suite 26 cases selfcheck ok; --mutations catches all ten
mutants; host-tests.py runs the vote entries green (base, --mutations and
the new mutant entry). The whole tools group was not run here: it exceeds
the 20 minute budget on this host and is CI's job.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
The device trees job on bfb4186 died in "Install dtschema and yamllint": apt-get update stalled on the azure mirror for about 45 minutes and the job was cancelled at its timeout, before any dts check ran. That is a runner stall, not a code failure, and rerunning the failed job needs admin rights on omacom, so this empty commit retriggers the run. No code change.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@greptile-apps

greptile-apps Bot commented Nov 22, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The PR changes GPU power requests on T6001 and adds a shared PMP request API.

Enable the promised packaged-default vote before merging; the host-test fixes are non-blocking.

Findings

  1. P1 Packaged vote stays disabled ▶
  2. P2 Fake timer hides postponed work ▶
  3. P2 Restricted hosts cannot start tests ▶

Summary

Adds GPU activity hooks and a T6001 PMP vote policy with a 250 ms idle hold.

  • T6001 GPU work now holds an AGX power request.
  • Kernel callers can now set individual PMP request bits.

Previously addressed blockers: joshuaswarren states that boot-path NULL workqueue access, begin-during-clear, unbind, sleep, and coverage findings were addressed in the supplied head; no duplicate findings are reported.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[GPU operation starts] --> B{Vote enabled and allowed?}
  B -->|Yes| C[Queue vote work]
  B -->|No| D[Count operation only]
  C --> E[Set T6001 PMP request]
  E --> F[Last operation ends]
  F --> G[Wait 250 ms]
  G --> H[Clear request if still idle]
  I[Suspend or unbind] --> J[Stop new votes and clear request]
Loading

Reviews (1) · Last reviewed commit: "ci: retrigger the device trees job" · Reviewed by Greptile


/* T6001 AGX request (see the engine below set_state): AGX is soc-device record 9 on the T6001. */
#define APPLE_PMP_T6001_AGX_ID 9
static bool agx_vote;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Packaged vote stays disabled

agx_vote defaults to false, so an unchanged installation does not get the promised automatic vote. Without a manual pmp_report.agx_vote=1, agx_vote_begin() schedules nothing. No in-tree boot or package setting enables it. Enable the policy by default while keeping the T6001 hardware check and manual opt-out.

Comment on lines +25 to +29
uint32_t at = s->now + delay_ms;
if (!s->armed || (int32_t)(at - s->timer_at) < 0) {
s->armed = true;
s->timer_at = at;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Fake timer hides postponed work

arm() only moves a timer earlier, but the kernel's mod_delayed_work() can move it later too. A worker can compute a delayed retry, a parameter write can queue immediate work, and the worker's later kick can replace that immediate request with the old delay. The tests cannot catch this ordering problem. Make the queue behavior match the intended deadline rule and test both kick orders.

Knowledge Base Used: Aurora CI and validation tooling

Comment on lines +69 to +70
prefix = ['setarch', '-R'] if shutil.which('setarch') else [] # ASan startup can crash under ASLR on a loaded host
ran = subprocess.run(prefix + [str(out / 'vote')], capture_output=True, text=True)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Restricted hosts cannot start tests

If a host has setarch installed but blocks the personality call required by setarch -R, this launcher exits before the test binary runs. The base suite and every mutation entry then fail despite working test code. Check whether disabling address randomization is allowed and fall back to running the binary directly when it is not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant