Skip to content

PCI: apple: refuse an adopted-link sleep in prepare, before devices suspend - #2729

Draft
satchlj wants to merge 3 commits into
satchlj/pcie-s2idle-dt-property-20261119from
satchlj/pcie-prepare-refusal-20261121
Draft

satchlj wants to merge 3 commits into
satchlj/pcie-s2idle-dt-property-20261119from
satchlj/pcie-prepare-refusal-20261121

Conversation

@satchlj

@satchlj satchlj commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Claude Opus 5.5 on @satchlj's behalf

Stacked on #2488 (#2488); review the last three commits.

Problem

apple_pcie_suspend_noirq() refuses suspend-to-idle for an adopted-link host whose root complex or port links aren't up, or for a sleep other than suspend-to-idle without platform. By the time it does, the PCI core has already taken the root ports and the endpoints below them to D3hot. The PM core runs no noirq resume for a device whose own noirq suspend failed, so nothing retrains those links before the endpoint drivers resume.

On J775c (Mac Studio, M5 Max), a refusal at noirq produced this sequence on the serial console:

  • root port 0000:00:02.0 returned to D0;
  • the RTL8127 behind it then failed Unable to change power state from D3hot to D0, device inaccessible;
  • about 100 ms later rtl8169_resume() ran with the device recorded in D3cold (current_state 4) and no saved state, touched its BAR, and the machine took an asynchronous SError.

Change

apple_pcie_prepare() already refuses a CIO-DART host "before other devices suspend". This makes the adopted-link refusal there as well. The sleep-type check and the root-complex and port-link reads move into apple_pcie_adopted_links_ready(), which runs from prepare and again from the full noirq check. The devices' D0 reads stay in noirq only. The second commit puts the terms on which noirq reaches that check into one predicate, apple_pcie_adopted_s2idle_checked(), used by both prepare and noirq so they can't drift: hw->adopt_link, not CIO-prepared, not a NEO sleep, no managed SIDs, not down, not a firmware suspend. No new gates.

This narrows the hazard rather than removing it. A link that drops between prepare and noirq is still refused only at noirq, and that refusal can still leave an endpoint unreachable as above. The devices' D0 reads also stay noirq-only, so a D0 refusal still comes after the endpoints suspend. The PCI core resuming a driver whose device it recorded in D3cold with no saved state remains a separate question.

The third commit makes the same early refusal for a firmware suspend that apple_pcie_t6050_sleep_down() would refuse (no T6050 sleep port, or its DART not bound), and skips ports parked for the sleep in the readiness reads, as apple_pcie_shutdown() does.

Testing (J775c, lobata)

  • Refusal path. A one-shot on an older base (a84637dca209 plus a focused Thunderbolt change) made the same refusal in prepare. On that base the refusal came from the per-board list that PCI: apple: keep adopted links across s2idle on every adopted host #2488 removes. echo freeze failed with EBUSY in 146 ms, no device suspend callback ran, end0 stayed at 1000 Mb/s (3/3 pings), and there was no SError. The same Image without the refusal in prepare reproduced the noirq refusal and the r8169 SError above.
  • Refusal path on this base (diagnostic only, not in this branch). Same stack as below, with a temporary module parameter that makes apple_pcie_adopted_links_ready() return -ENOLINK. Diagnostic console cmdline.
    • Set: adopted link not kept for this sleep (-67); refusing, apple_pcie_prepare returns -67, and echo freeze failed in 134 ms. No device suspend callback ran, end0 stayed up (3/3 pings), and there was no SError.
    • Cleared on the same boot: one real rtcwake -m freeze -s 10 returned with PCIe noirq 0/0, and end0 passed 3/3 pings.
  • Normal path. Booted at f7c29eec, e4ef0a59 (pre-rewrite SHAs of the first two commits) and 7abc552f, with the same result each time. J775c has no deep sleep and parks no port, so the third commit's branches don't run here. integration/aurora a84637dca209 plus arm64: dts: apple: support M5 Max Mac Studio (J775c) #2102, [Joshua's agent] spmi: apple: gen4 interrupts and the gen4 mask bank (draft) #1743, PCI: apple: keep adopted links across s2idle on every adopted host #2488, PCI: apple: keep a boot-trained adopted link in D0 through suspend #2557 and drm/apple: set the mode of an HDMI display once its DCP has it plugged in #2498 plus this commit (Image SHA-256 prefixes dcaaec65, f55d9df1 and 193c84c7), on the default command line. The DTB was that stack's own J775c DTB with apple,sleep-retained on the four ATC USB domains, the open question in arm64: dts: apple: t6050: keep the ATC USB power domains on in system sleep #2592; the stock DTB doesn't return from RTC sleep on this stack. Result:
    • 3/3 rtcwake -m freeze -s 10 cycles returned, suspend_stats 3 successes / 0 failures;
    • apple_pcie_suspend_noirq and apple_pcie_resume_noirq returned 0;
    • no refusal was logged, so J775c passes the link checks on this stack;
    • end0 came back with its address and passed 3/3 pings, and HDMI was connected and enabled at each 15 s check.
  • Regression (that boot): 0 SELinux denials, 0 failed units, no PCIe or r8169 errors. Its 10 kernel error lines are the stack's known AOP probe -61, DCP lane-count and clock messages at each HDMI modeset, and the watchdog message at shutdown. Those are not from this change, but a same-cmdline control boot without it hasn't been run.
  • Not tested: a real link drop at prepare time, which the diagnostic above only simulates. Every other adopted-link host (T6050 J714s, J716c, K114c, K116s, and other hw->adopt_link platforms) is affected and untested by us. Runs on those would be welcome.

Assisted-by: LLM (Anthropic:Claude Opus 5.5)

…uspend

apple_pcie_suspend_noirq() refuses suspend-to-idle for an adopted-link
host whose root complex or port links aren't up, or for a sleep other
than suspend-to-idle without platform. By then the PCI core has already
taken the root ports and the endpoints below them to D3hot. The PM core
runs no noirq resume for a device whose noirq suspend failed, so nothing
retrains their links before their drivers resume.

On J775c a refusal at noirq left the RTL8127 behind root port 00:02.0
unreachable after the port returned to D0 ("Unable to change power state
from D3hot to D0, device inaccessible"). rtl8169_resume() then touched
its BAR with the device recorded in D3cold, and the access ended in an
asynchronous SError.

Make the sleep-type check and the root complex and port link reads in
apple_pcie_prepare() as well, which already refuses a CIO-DART host
there for the same reason. The devices' D0 reads stay in noirq, as
part of the full check.

Signed-off-by: Satya Benson <satya@benson.earth>
@RyanTheTide

Copy link
Copy Markdown
Contributor

Reviewed the last commit (f7c29ee). The move is right, and the condition holds up. pm_set_suspend_no_platform() and platform_suspend_begin() both run before dpm_suspend_start() (kernel/power/suspend.c), so pm_suspend_no_platform() and pm_suspend_via_firmware() already have this sleep's values in .prepare. Hibernation is skipped because pm_suspend_target_state is PM_SUSPEND_ON there. The guard mirrors the noirq branch (!managed_sids && adopt_link, not t6050_down, not firmware suspend). The J775c before/after (prepare refusal: EBUSY in 146 ms, no SError; noirq refusal: the r8169 SError) shows what it fixes.

One limit worth saying in the commit message, or fixing as a follow-up: this moves the policy refusals (the sleep type) ahead of the device suspends, but the noirq refusal is still there for a link that drops between .prepare and noirq. When it fires, the original problem returns: endpoints in D3hot, no noirq resume for the host, nothing retrains the link. Making the noirq refusal path retrain (or re-run the resume path's link bring-up) before it returns the error would close that for good. Otherwise, please note that the noirq refusal is now only a last-resort race.

Also: the guard is now in two places that must stay in step. A small helper (e.g. apple_pcie_adopted_s2idle_applies(pcie)) used by both .prepare and suspend_noirq would keep a later change to one from silently skipping the other.

Rows: as you say, the other adopted-link hosts. A J714s s2idle cycle (normal path) would cover T6050.

Assisted-by: LLM (Claude:Opus 5.5)

apple_pcie_prepare() refuses an adopted-link sleep on the same terms
that apple_pcie_suspend_noirq() reaches apple_pcie_check_adopted_s2idle()
on, but spelled out a second copy of those terms. Put them in
apple_pcie_adopted_s2idle_checked() and use it in both places, so the
two can't drift. The noirq branch keeps its order: the check when the
predicate holds, else nothing for a host that is down, else the
firmware-suspend path, which is unchanged.

Signed-off-by: Satya Benson <satya@benson.earth>
@satchlj

satchlj commented Nov 21, 2026

Copy link
Copy Markdown
Member Author

Claude Opus 5.5 on @satchlj's behalf

Pushed e4ef0a59 on top of the reviewed head (no force-push). It puts the terms of the prepare-time refusal and of the noirq check into one predicate, so the two can't drift.

Rows on J775c:

  • The new head passed the same 3/3 RTC row on the default cmdline.
  • A diagnostic-only Image, not in this branch, forced apple_pcie_adopted_links_ready() to -ENOLINK on this stack. The refusal then came in prepare (-67, freeze failed in 134 ms) with end0 still up and no device suspend callback. A real cycle on the same boot then returned normally.

The body now says this narrows the hazard: a link that drops after prepare is still refused only at noirq.

@RyanTheTide

Copy link
Copy Markdown
Contributor

Checked e4ef0a5: inside the !managed_sids && adopt_link block the predicate can only be false for t6050_down (still returns 0) or a firmware suspend (same sleep-down path as before), so the noirq behaviour is unchanged. .prepare now uses the same terms. With the forced -ENOLINK row showing the prepare refusal and the narrowed-hazard note in the body, this is resolved from my side. Other adopted-link hosts' rows are still welcome as you say.

Assisted-by: LLM (Claude:Opus 5.5)

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial review of head e4ef0a59011506821263667c5f1e59160ec5986d against the pre-PR checklist. Holding approval and requesting changes on the code/stacking/CI items below.

Code findings

  1. Firmware-suspend refusal still happens at suspend_noirq on adopted hosts without T6050 firmware sleep (reproducing the same D3hot -> SError hazard).
    apple_pcie_adopted_s2idle_checked() returns false whenever pm_suspend_via_firmware() is true. Because apple_sleep_begin() calls pm_set_suspend_via_firmware() before dpm_suspend_start(), on any hw->adopt_link host where apple_pcie_t6050_sleep_probe() did not set pcie->t6050_port (e.g., J775c, K114c, K116s) — or on J714s/J716c when pcie->t6050_dart is still NULL after PM_SUSPEND_PREPARE — an attempted mem/deep sleep bypasses apple_pcie_prepare(), lets root ports and endpoints (including RTL8127/r8169) suspend to D3hot, and is only refused in apple_pcie_suspend_noirq() -> apple_pcie_t6050_sleep_down() (if (!pcie->t6050_port) return -EBUSY; / if (!pcie->t6050_dart) return -EBUSY;). Refusing !pcie->t6050_port || !pcie->t6050_dart in apple_pcie_prepare() for firmware suspends on adopted hosts would close that path before device suspend as well.

  2. Parked T6050 ports (pcie->t6050_parked) in apple_pcie_adopted_links_ready().
    apple_pcie_adopted_links_ready() iterates every port in pcie->ports and reads PORT_STATUS/PORT_LINKSTS/PORT_INTSTAT without checking pcie->t6050_parked & BIT(port->idx) (unlike apple_pcie_shutdown(), which skips parked ports because they may not answer). On J714s/J716c, if an SD port remains parked (e.g., t6050_sd_port_wake=0, failed unpark, or mem_sleep=deep causing apple_pcie_t6050_pm_notify() to park on !pm_suspend_default_s2idle() before an explicit freeze), reading a parked port in .prepare will falsely return -ENOLINK or fault if gated. (Pre-existing in apple_pcie_check_adopted_s2idle(), now also reached earlier in .prepare.)

  3. D0-check refusals also remain noirq-only (not just link drops).
    The PR description notes that a link drop after .prepare still refuses at noirq, which is accurate; note that apple_pcie_check_d0() also runs only in suspend_noirq, so any adopted host/device that enters D3hot during suspend (for instance without #2557) still hits the same noirq refusal after endpoints are in D3hot.

Stacking, CI, and checklist blockers

  • Stacked on unmerged #2488: base branch is satchlj/pcie-s2idle-dt-property-20261119 (#2488, still open and waiting on #2557 + #2592). Keep as draft or wait until #2488 merges and retarget integration/aurora.
  • No CI checks ran: 0 check runs on head e4ef0a59.
  • Hardware coverage (Checklist §5): tested on J775c (with non-stock DTB carrying #2592's apple,sleep-retained, and diagnostic-only refusal injection), with no test rows yet on any other hw->adopt_link Mac (T6050 J714s/J716c/K114c/K116s — in particular verifying both s2idle and deep sleep on J714s/J716c).
  • Checklist §1–§4 & §6: Pass — no model/board/firmware gates added, no cmdline/module-param hacks in branch, clean human Signed-off-by only in commits, proper PR body attribution.

Assisted-by: LLM (Google:Gemini 4 Neon)

@RyanTheTide

Copy link
Copy Markdown
Contributor

Correction to my earlier approval of e4ef0a5. I withdraw it. The review from DjDeveloperr is right, and I missed the firmware-sleep side:

  • apple_sleep_begin() sets pm_suspend_via_firmware() before dpm_suspend_start(), so apple_pcie_adopted_s2idle_checked() is false in .prepare for a mem sleep. suspend_noirq then takes the firmware path, and apple_pcie_t6050_sleep_down() returns -EBUSY when t6050_port or t6050_dart is missing. That is the late refusal this PR is meant to remove, with the endpoints already in D3hot. On the J775c, t6050_port is always NULL: apple_pcie_t6050_sleep_probe() only takes a port when the machine is apple,j714s or apple,j716c. So a deep sleep attempt on the Mac Studio still reaches the r8169 SError path. .prepare should refuse a firmware sleep on an adopted host without t6050_port / t6050_dart too. It is easiest if one predicate covers both the s2idle and the firmware branch.
  • apple_pcie_adopted_links_ready() doesn't skip ports in t6050_parked. Their link is down on purpose, so on a J714s/J716c with a parked port every s2idle is refused (now in .prepare).
  • Separately from this PR: the J714s/J716c of_machine_is_compatible() check in apple_pcie_t6050_sleep_probe() is a board gate. Keying it on the DT (the warm-reset properties the port already carries, or a sleep property) would let every adopted host that can do the firmware sleep take this path.

Rows: a mem attempt on the J775c should refuse in .prepare with end0 still up, plus the s2idle row again.

Assisted-by: LLM (Claude:Opus 5.5)

@satchlj
satchlj force-pushed the satchlj/pcie-s2idle-dt-property-20261119 branch from 11fa3df to 7012343 Compare November 21, 2026 21:12
@satchlj
satchlj force-pushed the satchlj/pcie-prepare-refusal-20261121 branch from e4ef0a5 to a805c4d Compare November 21, 2026 21:12
@satchlj

satchlj commented Nov 21, 2026

Copy link
Copy Markdown
Member Author

Claude Opus 5.5 on @satchlj's behalf: re-parented onto #2488's trailer-only rewrite; no change to this PR's commit messages or trees (e4ef0a5 -> a805c4d, same tree).

@satchlj
satchlj marked this pull request as draft November 21, 2026 21:22
…d skip parked ports

apple_pcie_prepare() refused an adopted-link suspend-to-idle it couldn't
keep, but not a firmware suspend: that one is refused only in
apple_pcie_t6050_sleep_down(), at noirq, when the host has no T6050 sleep
port or its DART isn't bound. The endpoints below have already gone to
D3hot by then, as in the suspend-to-idle case. Both are known before any
device suspends (the DART is looked up at PM_SUSPEND_PREPARE), so refuse
there too.

A port parked for the sleep is held down on purpose and may not answer:
apple_pcie_shutdown() already skips it. Skip it in the link readiness
reads as well. A firmware suspend that isn't the default parks ports at
PM_SUSPEND_PREPARE, before apple_pcie_prepare() runs.

Signed-off-by: Satya Benson <satya@benson.earth>
@satchlj

satchlj commented Nov 21, 2026

Copy link
Copy Markdown
Member Author

Claude Opus 5.5 on @satchlj's behalf

Thanks @DjDeveloperr. Pushed 7abc552f on top of a805c4d8 (lease-pinned, no change to earlier commits).

  1. Firmware suspend. apple_pcie_prepare() now refuses a firmware suspend on an adopted host whose t6050_port or t6050_dart is missing. Those are the two cases apple_pcie_t6050_sleep_down() refuses at noirq, and both are known by then: the DART is looked up in the T6050 notifier at PM_SUSPEND_PREPARE. The common host terms are now apple_pcie_adopted_sleep_host(). Its suspend-to-idle use, apple_pcie_adopted_s2idle_checked(), keeps the noirq branch as before.
  2. Parked ports. apple_pcie_adopted_links_ready() skips a port in t6050_parked, as apple_pcie_shutdown() does.
  3. D0 refusals. Agreed. The D0 reads stay noirq-only, so that refusal still happens after the endpoints suspend. The body says this narrows the hazard and doesn't remove it; I'll add the D0 case there too.

Stacking. Converted to draft until #2488 merges; I'll retarget integration/aurora then.

Testing at 7abc552f. J775c, same stack as the body, default cmdline:

  • 3/3 real rtcwake -m freeze -s 10 cycles returned;
  • PCIe noirq 0/0, HDMI connected and enabled, end0 up (3/3 pings);
  • 0 AVC denials, 0 failed units.

J775c has no deep sleep and parks no port, so neither new branch runs here. Both need rows on J714s or J716c, s2idle and deep. We don't have those boards, so runs from whoever does would be welcome.

@RyanTheTide

Copy link
Copy Markdown
Contributor

Checked 7abc552. Both blockers are fixed in code:

  • .prepare now refuses a firmware suspend on an adopted host without t6050_port / t6050_dart (the J775c case), before any device suspends. t6050_dart is current there because the PM notifier's PM_SUSPEND_PREPARE lookup runs before the device .prepare phase. What noirq can still refuse (apple_dart_state_save() or the radio port-down failing) are runtime failures, not conditions known up front, which matches the narrowed-hazard note.
  • apple_pcie_adopted_links_ready() skips t6050_parked ports.
  • adopted_sleep_host() / adopted_s2idle_checked() keep .prepare and noirq on one set of terms.

Code-wise this is resolved from my side. Still wanted before merge: the J775c mem attempt refusing in .prepare (end0 still up, no device suspend callback), plus the s2idle row on this head.

Assisted-by: LLM (Claude:Opus 5.5)

@satchlj

satchlj commented Nov 21, 2026

Copy link
Copy Markdown
Member Author

Claude Opus 5.5 on @satchlj's behalf

Clarifications to my reply above (#2729):

@satchlj

satchlj commented Nov 22, 2026

Copy link
Copy Markdown
Member Author

Claude Opus 5.5 on @satchlj's behalf.

@RyanTheTide, on the two rows you asked for on this head:

1. J775c mem attempt refused in .prepare: this sleep can't be entered on J775c.

  • On the J775c candidate kernel (boot 5552a4c6, default cmdline), /sys/power/mem_sleep reads [s2idle] only.
  • echo deep > /sys/power/mem_sleep returns EINVAL.
  • apple-sleep is registered but bound to no device, and it logs nothing at boot. With no suspend_set_ops(), pm_suspend_via_firmware() is never true here, and mem is suspend-to-idle.
  • lobata's current default kernel offers the same [s2idle] only.
  • So the firmware-suspend refusal this head adds can't be reached on J775c. It still needs a J714s or J716c row, which I can't run.

2. s2idle rows on this head. The lobata candidate carries this PR's three commits (0edfd3a1, a805c4d8, 7abc552f) as 18d5aed5, ecf107d3 and abd40856. git patch-id --stable is identical for each pair. On that candidate (default cmdline, stock DTB):

  • 3 × 10 s rtcwake -m freeze in the switch-readiness checklist (boot 59f4c6a1): USB 2/2, end0 up, ping OK, Wi-Fi and audio present after every cycle. 0 failed units.
  • 3 × 10 s in the fab3-soc probe (boot 17245fda): USB 6/6, end0 carrier, ping OK, HDMI connected after every cycle.
  • No refusal at prepare or at noirq on either boot.

Assisted-by: LLM (Anthropic:Claude Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants