Skip to content

phy: apple: atc: T8132: keep a CIO-mode port across suspend-to-idle - #2705

Open
RyanTheTide wants to merge 3 commits into
airs-wifibt2/t8132-s2idle-device-rolefrom
j713-s2r/atc-s2idle-cio-on-2700
Open

RyanTheTide wants to merge 3 commits into
airs-wifibt2/t8132-s2idle-device-rolefrom
j713-s2r/atc-s2idle-cio-on-2700

Conversation

@RyanTheTide

Copy link
Copy Markdown
Contributor

Problem

With a USB4 cable in (a dock, or a host-to-host link to another Mac), a T8132 port runs in CIO mode under its USB4 router, and DWC3 does not run on it. #2700 admits suspend-to-idle while DWC3 runs, in either role, so on such a port the M4 MacBook Air still refused every suspend-to-idle: atcphy_suspend() -EBUSY.

Change

Also admit suspend-to-idle for a port in CIO mode. The router's power states stay on across suspend-to-idle, because the ACIO's links to them are syscore devices, as the glue keeps the controller's on for DWC3. So the link stays as it is. A deeper sleep still needs the native CIO sleep's handoff. The T8132 selftest gets the CIO case and a negative control for it.

Stacked on #2700.

Testing

  • tools/testing/selftests/drivers/apple/atc-t8132-usb3.py: 13/13. The new CIO case admits s2idle and refuses deep; the no-CIO mutation fails as it should.
  • tools/asahi/atc-usb3-tests.py --mutations: 38/38 cases, all mutations killed.
  • W=1 clean.
  • J713 on the macOS 27.2 stub, with a USB4 host-to-host link to a J616s up (port in CIO mode), an external monitor on the other port, and a lab kernel = J713 B9 + phy: apple: atc: admit suspend-to-idle with DWC3 running in device role #2700's change + this one: s2idle 3/3 with RTC wake (suspend_stats 3 / 0, Wi-Fi back each time). Without this change, the same setup refused with phy-apple-atc 402a90000.phy: atcphy_suspend returns -16.

Assisted-by: LLM (Claude:Opus 5.5)

With a USB4 cable in (to a dock, or a host-to-host link to another
Mac), the port runs in CIO mode under its USB4 router and DWC3 is not
running on it, so suspend-to-idle was still refused there. The router's
power states stay on across suspend-to-idle (the ACIO's links to them are
syscore devices), as the glue keeps the controller's on for DWC3: the
link stays as it is. Admit that case too; a deeper sleep still needs the
native CIO sleep's handoff.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Blockers: 1. The comment at atc.c:7856-7863 says the port survives suspend-to-idle because the router's power states stay on.
The ACIO keeps atcN_cio, atcN_cio_pcie, atcN_cio_usb and atcN_pcie on (t8132-usb4.dtsi:469-476, thunderbolt/apple.c:3448).
The PHY's domain is atcN_usb (t8132.dtsi:1176), a sibling under atcN_common. With DWC3 in reset, dwc3-apple is no longer syscore,
so nothing visible here keeps atcN_usb on through genpd's system-sleep power-off. The driver's own comment at atc.c:4417
describes exactly that. If it drops, atcphy_resume() runs atcphy_restore_after_pd_off(), which sets the mode to OFF and
reconfigures CIO through the atcphy_power_off() branch of atcphy_configure_t8132(), a PHY power cycle under the live router.
Nothing logs it (the resume warning only checks host_active). s2idle 3/3 with suspend_stats clean doesn't distinguish the two.
Could you either hold the PHY's own domain while it is in CIO mode (syscore or the awake path, as dwc3-apple does for its role),
or add a row showing atc0_usb stays on and the USB4 link (thunderbolt device, traffic) survives the sleep? Please also extend the
resume warning to CIO mode. Smaller point: the check is on the PHY mode only, not on a running router, so a T8132 board with CIO
tunables and no ACIO (J604) now also admits s2idle with a USB4 partner. Rows worth adding: a dock with a DP tunnel across s2idle,
and one J604/J773g USB4 partner s2idle as a regression row.

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

J713 row, run on the J713 default kernel (7.3.0-bj713t6+ = the integration tip plus open J713 PR heads, including this one at 766641b; default cmdline; 27.2). Test: s2idle three times, 15 s each, with the USB4 host-to-host link to the J616s up and carrying traffic (5 Hz ping over thunderbolt-net).

  • s2idle 3/3. The CIO link was up after every cycle: thunderbolt0 up and peer 0-1 present.
  • Ping got 148 of 150 replies in each cycle; the two lost replies fall at the sleep transitions.
  • atc0_usb, atc0_cio, atc0_pcie, atc0_common and atc0_cio_pcie read on before and after each cycle. There were zero Timeout reading AT lines.

This is the row alternative to holding the domain in code. It covers a host-to-host link only; a dock with a DP tunnel is not covered.

Assisted-by: LLM (Claude:Opus 5.5)

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial review at 766641b3586c against pre-pr-checklist.md.

Blockers

  1. §3 (Correctness — ps_atcN_usb is not kept on across s2idle in CIO mode, and atcphy_restore_after_pd_off() silently tears down/re-initializes the live CIO PHY on resume):

    • drivers/phy/apple/atc.c:7860-7862 assumes a CIO-mode PHY stays powered because the USB4 router's power states stay on. In DT, usb4_0_acio only owns ps_atc0_cio, ps_atc0_cio_pcie, ps_atc0_cio_usb, and ps_atc0_pcie (arch/arm64/boot/dts/apple/t8132-usb4.dtsi:468-471, marked syscore in drivers/thunderbolt/apple.c:3448). Those keep parent ps_atc0_common on, not ps_atc0_usb.
    • atcphy0 sits in ps_atc0_usb (t8132.dtsi:1176), a sibling under ps_atc0_common (t8132-pmgr.dtsi:1237-1244; t8132-j713-pmgr.dtsi:594-596 does not mark it always-on or sleep-retained). With DWC3 in reset (dwc3_running == false), dwc3-apple drops syscore, so nothing in ps_atc0_usb prevents genpd_finish_suspend() (drivers/pmdomain/core.c:1605-1608) from powering off ps_atc0_usb and firing GENPD_NOTIFY_OFF → atcphy->pd_was_off = true (atc.c:6845-6846).
    • On resume (atc.c:7879-7903), the pd_was_off warning at line 7887 only checks atcphy->host_active (false in CIO mode), so nothing warns. Then atcphy_restore_after_pd_off() (atc.c:4439-4508, enabled by atcphy_hw_t8132.restore_after_pd_off = true since cio_sleep.prepared is false) runs: it calls atcphy_forget_common_state() (wiping tracked DP/tunnel clock state), sets atcphy->mode = APPLE_ATCPHY_MODE_OFF at line 4467 while atcphy->t8132_common_owned is still true, and then calls atcphy_configure(atcphy, APPLE_ATCPHY_MODE_USB4) → atcphy_configure_t8132(). Because atcphy->mode was already overwritten to OFF, line 4033–4040 skips atcphy_t8132_phy_stop() and executes atcphy_power_off(atcphy) followed by a full CIO re-init underneath the live USB4 router.
    • Checking that atc0_usb reads on before and after s2idle (issuecomment-7074095251) does not prove it stayed on during s2idle, because genpd_finish_resume() powers it back on before userspace reads it.
    • Required fix: Explicitly hold ps_atcN_usb across s2idle while the PHY is kept in CIO mode (releasing it when leaving CIO or for deep sleep), and extend the pd_was_off resume warning at atc.c:7887 to CIO mode (and device-role dwc3_running).
  2. §3 (Cross-board regression — mode-only check admits CIO s2idle without an active ACIO router):

    • atcphy_suspend() checks only atcphy_mode_is_cio(atcphy->mode). Boards such as t8132-j604 (M4 Mac mini) include t8132-j604-atcphy-tunables.dtsi (apple,tunable-cio-*, so atcphy->cio_tunables is true) without any usb4_*_acio node in t8132-j604.dts. Without a running ACIO router holding ps_atcN_common, even ps_atcN_common is not kept on across s2idle. Gate admission on the PHY/router actually keeping the required power domain(s) on (or holding ps_atcN_usb, which itself keeps parent ps_atcN_common on).
  3. CI / Stack:

    • Base branch is unmerged PR #2700 (airs-wifibt2/t8132-s2idle-device-role), and 0 CI check runs have executed on 766641b3586c3dd22c17066d9e161b5615b7dc90. Rebase/retarget onto integration/aurora once #2700 lands so CI runs cleanly on the live head.

What checks out & remaining §5 rows

  • §4 & §6: Clean (human author/committer + human Signed-off-by: only, Assisted-by: in PR body).
  • §5 (for re-review after fixes): Include genpd_summary off-count/off-time delta (or pd_was_off verification) proving ps_atcN_usb had 0 off transitions during s2idle, plus explicit regression counts (0 SELinux AVCs, 0 failed units, 0 new kernel errors). A dock with a DP tunnel across s2idle is also worth covering.

Assisted-by: LLM (Gemini 4 Neon)

…c-s2idle-cio-on-2700

Bring in the resume and shutdown checks of a running DWC3 in either role,
which the CIO-mode suspend-to-idle change builds on.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
…in CIO mode

A T8132 port in CIO mode was admitted to suspend-to-idle on the
assumption that its USB4 router keeps the PHY powered. It does not: the
ACIO holds the CIO and PCIe power states, and through them ATCn_COMMON,
but the PHY sits in ATCn_USB, a sibling of those. With DWC3 held in
reset nothing else holds ATCn_USB, so genpd may switch it off in the
noirq phase. atcphy_resume() then runs atcphy_restore_after_pd_off(),
which power-cycles and sets up CIO again under the live router, and the
resume warning only looked at USB3, so nothing reported it. A board with
CIO tunables and no ACIO at all, such as the J604, was admitted the same
way.

Hold the domain from the PHY itself. At probe, a T8132 PHY with CIO
tunables attaches a second consumer to its own domain: a syscore
virtual device, as the ACIO uses for its own states, which the system
sleep leaves alone and which stays runtime suspended, so runtime PM of
the domain is unchanged. In its noirq suspend the PHY gives up that
consumer's share with dev_pm_genpd_suspend(), and takes it back on
resume, unless this suspend-to-idle keeps a CIO-mode port: then the
domain, and its parent ATCn_COMMON, have no off transition. Leaving CIO
mode, a deeper sleep (refused, or taken over by the router's CIO sleep
handoff) and hibernation release it as before.

Admit CIO mode only with the hold in place, and extend the resume
warning to a port in CIO mode, next to DWC3 running in either role.

The T8132 host test models genpd's system-sleep count of the domain and
checks: no off transition and no warning for a held CIO port, the
warning when it was lost anyway, refusal without a hold and in a deeper
sleep, and the release once the port left CIO mode or the router took
the sleep. Negative controls reject admission without a hold, a hold
never kept, a hold never released and a resume warning blind to CIO.
The KUnit suspend case covers the CIO admission and now sets
dwc3_running for its host case, which admission checks since DWC3 in
device role was admitted.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Updated to 0ed3520. The branch first takes the current #2700 head (da78109) with a normal merge, 1866ac2. That merge is needed because both PRs change the resume warning and the T8132 host test. The fix is a single commit on top.

1. ps_atcN_usb is not held across s2idle in CIO mode. The PHY now holds the domain itself.

  • At probe, a T8132 PHY with CIO tunables attaches a second consumer to its own domain: atcphy_probe_pd_hold(), drivers/phy/apple/atc.c:6891. This is a syscore genpd virtual device, the same pattern the ACIO uses for its own states. It stays runtime suspended, so runtime PM of the domain does not change.
  • atcphy_suspend() (atc.c:7926-7929) sets pd_hold_sleep when it admits a CIO-mode port to suspend-to-idle.
  • In atcphy_sleep_suspend_noirq() (atc.c:8021), the PHY gives up the hold's share of the domain with dev_pm_genpd_suspend() and reclaims it in resume_noirq. It skips both while pd_hold_sleep is set. For a kept CIO port genpd's suspended_count therefore never reaches device_count, so ps_atcN_usb has no off transition and parent ps_atcN_common stays on with it.
  • The hold is released as before in four cases: when the port has left CIO mode, in deep sleep (refused, or handed to the router's CIO sleep through cio_sleep.prepared), for an idle port, and in hibernation.
  • The pd_was_off resume warning now also covers a port in CIO mode, next to DWC3 running in either role from phy: apple: atc: admit suspend-to-idle with DWC3 running in device role #2700: atc.c:7953-7958, "power domain went off under an active USB4/USB3 link".

2. Admission without a router or hold (J604). Admission now keys on atcphy_cio_can_hold() (atc.c:1484), which requires CIO mode plus the hold device, not the mode alone. A board without a domain to hold refuses CIO s2idle with -EBUSY. On a board with CIO tunables and no ACIO, such as the J604, the PHY holds ps_atcN_usb and through it ps_atcN_common, so admission no longer depends on a router being present.

3. Stack / CI. Still stacked on #2700. Retarget to integration/aurora once #2700 lands.

Also fixed: the KUnit atcphy_test_t8132_system_suspend host case only set host_active, and it already failed at the #2700 head because admission checks dwc3_running. It now sets both, and the test covers the CIO cases.

Testing (no hardware in this round):

  • drivers/phy/apple/atc.o builds with W=1 and adds no warnings. The kernel-doc warning count is unchanged at 60, all pre-existing. checkpatch --strict: 0 errors, 0 warnings, 0 checks.
  • tools/testing/selftests/drivers/apple/atc-t8132-usb3.py: 13/13. It now models genpd's system-sleep count of the domain and checks:
    • a held CIO port: 0 off transitions, no restore, no warning;
    • a CIO port that loses the domain anyway: 1 warning;
    • refusal in deep sleep and without a hold;
    • after leaving CIO, and for a router-handled sleep: exactly 1 off transition.
      New negative controls, all killed:
    • admission on CIO mode alone;
    • a hold that is never kept;
    • a hold that is always released;
    • a hold that is never released;
    • a resume warning blind to CIO.
  • tools/asahi/run-host-tests.py --only atc-: 4/4. That covers atc-cio-sleep-tests (41 cases, 11 negative controls), atc-usb3-tests (38 cases) and atc-usb3-tests --mutations (all killed). --only dwc3: 1/1.
  • KUnit, arm64 under qemu, phy-apple-atc*: 38/38 (phy-apple-atc-mux 33, phy-apple-atc-t8132-pm 5). As a negative control, the same run with admission on CIO mode alone fails at the new no-hold expectation.

Still to run on hardware:

  • J713, kernel = this head plus usb: dwc3: apple: take a device role off the bus for suspend-to-idle #2728, default cmdline, USB4 host-to-host link to a J616s carrying traffic (ping over thunderbolt-net), s2idle x3. For each cycle, record before and after the cycle:

    • the atc0_usb usage and time columns from /sys/kernel/debug/pm_genpd/atc0_usb/idle_states;
    • atc0_usb in /sys/kernel/debug/pm_genpd/pm_genpd_summary;
    • atc0_common the same way.

    The pass condition:

    • the usage delta is 0 for every cycle;
    • genpd:0:402a90000.phy is listed under the domain's devices;
    • there is no "power domain went off under an active USB4 link" line;
    • the link and peer are back after every cycle.

    Also give the counts: 0 SELinux AVCs, 0 failed units, 0 new kernel errors.

  • Nice to have: a USB4/TB dock with a DP tunnel across s2idle on the J713.

  • Not in our lab: a J604/J773g USB4-partner s2idle regression row.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Review at 0ed3520: no blockers.

Both code findings from the review at 766641b are fixed. The PHY now holds ATCn_USB itself: a syscore genpd virtual consumer that is given up in noirq unless this suspend-to-idle keeps a CIO port. CIO admission requires that hold, so a CIO-tunables board with no hold is refused. The resume warning now covers CIO mode and DWC3 running in either role. I found no get/put imbalance and no lifetime bug. Two non-blocking points are below. The J713 genpd off-count row asked for in the earlier review has not been posted for this head, and the PR still needs a retarget once #2700 lands.

Non-blocking:

  1. drivers/phy/apple/atc.c:8000 and the commit message say the hold is released "as before" in hibernation, and that genpd "counts differently" there. In this tree, genpd_freeze_noirq() and genpd_poweroff_noirq() both go through genpd_finish_suspend(), the same as suspend. .freeze_noirq and .poweroff_noirq are wired to the plain atcphy_suspend_noirq(), so the hold is never counted there. On every T8132 PHY with CIO tunables (J713, J715, J604), ATCn_USB and ATCn_COMMON now stay on through freeze and poweroff, where before they could go off. The counts stay balanced, so nothing leaks and thaw/restore are fine. Either point freeze/poweroff/thaw/restore at the sleep wrappers (pd_hold_sleep is false there, so the hold is released), or fix the comment and the commit message.
  2. atc.c:7929: pd_hold_sleep is latched in atcphy_suspend(). t8132_suspended does not block a change to OFF or USB2 (atcphy_configure_t8132() only refuses USB3/DP/CIO), so a port that leaves CIO between atcphy_suspend() and noirq keeps the domain on for that one sleep. That costs power only, and the next resume clears it. atcphy_pd_hold_release() could re-check atcphy_cio_can_hold().
Full review

Earlier findings:

  • 1 (ATCn_USB not held in CIO s2idle; silent restore under the live router). Fixed. atcphy_probe_pd_hold() (atc.c:6891) attaches genpd:0:<phy> to the PHY's own domain with genpd_dev_pm_attach_by_id(dev, 0) and marks it syscore. The PM core never prepares or suspends it, so it stays in device_count and out of suspended_count. genpd_sync_power_off() therefore needs it released before it can turn the domain off. atcphy_sleep_suspend_noirq() releases it with dev_pm_genpd_suspend() only after atcphy_suspend_noirq() succeeds, and not when pd_hold_sleep is set. atcphy_sleep_resume_noirq() reclaims it only if it was released (pd_hold_released). The hold runs runtime suspended, so runtime genpd_power_off() ignores it and idle runtime PM is unchanged. The resume warning (atc.c:7954) now fires for CIO mode, or for a common epoch with dwc3_running, on pd_was_off.
  • 2 (mode-only CIO admission, J604). Fixed. Admission uses atcphy_cio_can_hold() = CIO && pd_hold. CIO mode always sets t8132_common_owned (atcphy_configure_t8132()), so the refusal path is reached. With no hold the result is -EBUSY. On J604 the hold attaches, and through it ATCn_COMMON is held too.
  • 3 (stack/CI). Still open, as the PR says. The base is airs-wifibt2/t8132-s2idle-device-role = da78109 = phy: apple: atc: admit suspend-to-idle with DWC3 running in device role #2700's current head. 1866ac2 is a real merge (parents 766641b, da78109), and git log rv2700..rv2705 = 766641b, 1866ac2, 0ed3520 only. Nothing stale. No CI runs on this head.

Refcount walk for the hold:

  • s2idle CIO kept: no release, no reclaim.
  • s2idle with DWC3 or an idle port, and deep sleep: release, then reclaim.
  • Router CIO-sleep handoff (cio_sleep.prepared): pd_hold_sleep is false, so release, then reclaim. Its resume early-return does not matter because reclaim is in resume_noirq.
  • Suspend refused in atcphy_suspend(), or aborted before noirq: neither runs.
  • Own suspend_noirq fails: no release, and the PM core skips resume_noirq.
  • A later device's noirq fails, or genpd's stop_dev fails after the callback (genpd calls resume_noirq): reclaim balances.
  • Repeated cycles: pd_hold_released is cleared on every reclaim, and pd_hold_sleep is cleared at every suspend entry and resume.
  • Hibernation: never counted (see the non-blocking point).
  • Probe: -EPROBE_DEFER is propagated. Other errors or NULL give no hold (info line), and CIO s2idle is then refused. The devm detach is registered right after the attach, so later probe failures and remove detach it. No PM callback can run after unbind.

Locking:

  • The Apple pmgr domains are GENPD_FLAG_IRQ_SAFE (pmgr-pwrstate.c:902), so genpd_switch_state() takes the genpd spinlock. Async noirq of other ATCn_USB consumers cannot race the counts.
  • The release runs under atcphy->lock before genpd counts the PHY itself, so it cannot be the transition that powers off. The PD notifier only does WRITE_ONCE anyway, so there is no recursion on atcphy->lock.
  • The reclaim runs after genpd already powered the domain on in genpd_finish_resume(). Same pattern as the ACIO's own dev_pm_genpd_suspend/resume in apple-cio-sleep.c.

Other checks:

  • Scope: only apple,t8132-atcphy of-data (gen T8132), plus t8132_ss (reg-names) and cio_tunables (DT properties). No board or firmware if-gates. Other generations are untouched. T6040/T6050 are not gen T8132.
  • No new cmdline or module params. No new delays.
  • Trailers: every commit in integration/aurora..0ed3520 (including the merge) carries only Signed-off-by: Ryan Murray. The PR body ends with the Assisted-by line.
  • git merge-tree onto integration/aurora 95a13cf: clean.
  • Reproduced here:
    • run-host-tests --only atc- 4/4, including --mutations.
    • atc-t8132-usb3.py 13/13.
    • W=1 clang build of drivers/phy/apple/atc.o on the fleet81 config: no warnings.
  • KUnit 38/38 is from the PR's log; I did not re-run it.

Still to run (from the earlier review's section 5): J713 at this head, USB4 link with traffic, s2idle x3, with the atc0_usb idle_states usage delta = 0 per cycle and genpd:0:...phy listed in its devices. Also 0 AVC, 0 failed units, 0 new kernel errors. A DP-tunnel dock row would be good too. Then retarget onto integration/aurora after #2700.

Assisted-by: LLM (Claude:Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants