Skip to content

PCI: apple: keep a boot-trained adopted link in D0 through suspend - #2557

Open
satchlj wants to merge 2 commits into
integration/aurorafrom
satchlj/pcie-apple-adopted-d0-20261120
Open

satchlj wants to merge 2 commits into
integration/aurorafrom
satchlj/pcie-apple-adopted-d0-20261120

Conversation

@satchlj

@satchlj satchlj commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Claude Opus 5.5 on @satchlj's behalf. s2idle works on J775c with this PR plus #2592 (cand-10: pm_test=platform passes, and 3/3 real RTC-woken freezes; rows on #2488 and below). This PR passes the host's D0 check. #2592 fixes the noirq resume hang that followed. Earlier rows are kept below for the record.

PCI: apple: mark a root port the boot loader trained on an adopted link, and every function below it, PCI_DEV_FLAGS_NO_D3 when it is enabled. The PCI core's suspend paths then leave them in D0, which the host's s2idle check (adopted link requires accessible D0 devices for s2idle) requires.

Selection: the port's own state, not a board or a command-line option. Only an adopted link the boot loader trained is marked. An adopted port that Linux trains itself (pwren-gpios) is not.

Cost: those functions also lose runtime D3. That is what lobata's default kernel has today, since its root ports have no driver.

Base: integration/aurora f0c51766e9c5 (rebased 2026-11-20 14:4xZ; #2088, #2101 and #2447 since c92ed8517b79 touch none of this PR's files).

Why

On J775c with #2488's head, s2idle was refused on every attempt. A diagnostic print in the D0 check showed the device it refused (cand-5, d2f8af83bf98; see #2488):

pcieport 0000:00:01.0: adopted link requires accessible D0 devices for s2idle (DIAG: vendor 0x106b, PMCSR 0x000b, current_state D3, below a boot-trained port)

It was the same in 6 of 6 cycles, including 3 with runtime PM forbidden on every PCI device. So it is the PCI core's system-suspend path that puts the boot-trained root port 00:01.0 in D3hot before the host's noirq check.

Testing

J775c (Mac Studio, M5 Max, T6050), on lobata.

Kernel: cand-7, 324280cadc47 = integration/aurora d91db387c720 + 85, several merges behind this base (none touch pcie-apple.c). It includes #2488's head and this commit, cherry-picked with a context-only conflict next to the Clock-PM helper. No diagnostic print.

Wake path, checked before the cycle:

  • rtc0 (macsmc-rtc) has power/wakeup = enabled.
  • Its alarm IRQ 151 is routed through apple_spmi, with count 0.
  • The live DT's RTC node has wakeup-source.

Result, 1 cycle: rtcwake -d rtc0 -m freeze -s 20 began at 12:23:01Z. The console showed Filesystems sync: 0.003 seconds and then nothing more. The console suspends with the system, so its silence after that line tells nothing.

There was no network and no resume after 7 minutes, so lobata was reset over DebugUSB. The journal ends before the freeze.

So it isn't known whether the D0 check passed and the wake failed, or suspend hung earlier. The earlier refusal (-16, returning at once) did not appear on the console, but the console can't show it either way.

cand-8 run (7b5df9391f5f = c92ed8517b79 + 84, with #2488's head and this PR, no_console_suspend), pm_test with pm_debug_messages and pm_print_times:

Level Result
freezer, devices resumed
platform apple_pcie_suspend_noirq returned 0 after 49 usecs: the D0 check passes. Then noirq resume stalls. The last callbacks were apple_pcie_resume_noirq, pci_pm_resume_noirq on 00:02.0, on the Wi-Fi function 01:00.0 and the Bluetooth function 01:00.2, then PHY genpd_resume_noirq. About 80 s later came an RCU stall, with CPU 7 not answering the backtrace NMI.

The pm_test levels resume on their own, so this hang doesn't involve a wake source. It is neither "suspend hung" nor "wake failed". The 12:23Z cand-7 freeze most likely hit the same resume hang.

The plan below was written before the cand-8 run and is kept for the record:

  1. Bisect with /sys/power/pm_test at freezer, devices, platform and core. Each level resumes on its own, so devices alone shows whether the D0 check now passes, with no dependence on a wake source.
  2. Check the wake arming: /sys/kernel/irq/151/wakeup once rtcwake has armed the alarm, the SPMI node's interrupt-controller / #interrupt-cells in the live DT, and whether the SPMI irqchip implements irq_set_wake (or sets IRQCHIP_SKIP_SET_WAKE). Hypothesis: if the wake is never armed at the parent interrupt controller, the RTC alarm can't wake the system, which would look exactly like the 12:23Z non-resume.
  3. Only then, a real freeze with no_console_suspend and pm_debug_messages, a second wake source I can trigger, and /sys/power/pm_wakeup_irq logged after each cycle.

I'll post it here before taking this out of draft.

Untested: other adopted hosts (J775D), and the runtime-PM cost in idle power.

Assisted-by: LLM (Codex:GPT-6.2)

@satchlj
satchlj force-pushed the satchlj/pcie-apple-adopted-d0-20261120 branch 2 times, most recently from 9dc8f01 to 6ff3e34 Compare November 20, 2026 14:03
@satchlj

satchlj commented Nov 20, 2026

Copy link
Copy Markdown
Member Author

Claude Opus 5.5 on @satchlj's behalf: J775c row on cand-8. With this PR the D0 check passes, and resume then hangs in the noirq phase. This run doesn't depend on any wake source.

Kernel: cand-8, 7b5df9391f5f = integration/aurora c92ed8517b79 (this PR's base) + 84. It includes #2488's head, this PR, #2565 and #2556, and has no diagnostic commits. Command line … no_console_suspend. Booted 2026-11-20 14:10Z.

Method: /sys/power/pm_test at each level with echo freeze > /sys/power/state, and pm_debug_messages plus pm_print_times on. Each level resumes by itself.

  • devices stops before the host's noirq check.
  • platform stops after it, just before the s2idle loop.
Level Result
freezer suspend entry, 5 s wait, suspend exit
devices passed: resumed and the script continued
platform apple_pcie_suspend_noirq returned 0 after 49 usecs: the check that returned -16 on every earlier cycle now passes. Then the 5 s wait, then noirq resume, which never completed.

The hang. The last resume callbacks on the console, in order:

  • apple_pcie_resume_noirq (1cb0000000.pcie);
  • pci_pm_resume_noirq on pcieport 0000:00:02.0;
  • pci_pm_resume_noirq on apple_centauri_t6050 0000:01:00.0 (Wi-Fi) and btcentauri_t6050 0000:01:00.2 (Bluetooth);
  • genpd_resume_noirq for three PHY domains.

About 80 s later the RCU watchdog reported rcu_sched detected stalls … these CPUS still haven't responded to the NMI: 7 and rcu_sched kthread starved. No backtrace for CPU 7 came out. Recovery took the documented order: the DebugUSB reset didn't return the machine, and a 180 s power cycle on the plug returned the known-good kernel.

What this shows:

  • This PR fixes the refusal: the boot-trained root port is no longer in D3hot at the host's noirq check.
  • System suspend now goes past the point where it used to stop, and the noirq resume path hangs. A pm_test run hangs the same way, so cand-7's 12:23Z freeze was most likely this resume hang, not a failed RTC wake.

Not known:

  • Which callback hangs, or what CPU 7 was doing. Hypothesis: a noirq resume touches a block whose power domain the noirq suspend turned off, with interrupts masked.
  • Whether this PR's NO_D3 contributes, or whether the hang was always behind the refusal. Before this PR, the refusal undid the noirq phase before any other device's noirq resume ran.

Wake arming on the same boot, for the real freeze later:

  • The live SPMI node has interrupt-controller and #interrupt-cells = 2.
  • rtc0 has wakeup enabled, and IRQ 151 (macsmc-rtc) is routed through apple_spmi.
  • The SPMI irqchip skips irq_set_wake by design, because its chained parent interrupt stays live.
  • end0 supports Wake-on-LAN (g) as a second wake source.

Next run:

  1. pm_test=platform with initcall_debug and an NMI backtrace on the stall (rcupdate.rcu_cpu_stall_ftrace_dump, sysctl kernel.panic_on_rcu_stall=0, nmi_backtrace), to name the stuck callback.
  2. The same with the Wi-Fi and Bluetooth functions unbound, to split the PCIe functions from the PHYs.

This PR stays in draft.

🤖 Generated with Claude Code

@satchlj
satchlj force-pushed the satchlj/pcie-apple-adopted-d0-20261120 branch from 6ff3e34 to 8d09565 Compare November 20, 2026 14:42
@satchlj

satchlj commented Nov 20, 2026

Copy link
Copy Markdown
Member Author

Claude Opus 5.5 on @satchlj's behalf: cand-9 row. The resume hang doesn't follow the Wi-Fi/BT functions. Probes point at the USB/Thunderbolt power domains (hypothesis).

Kernel: cand-9, df1005dceaa0 = integration/aurora f0c51766e9c5 (this PR's base) + 84. It is cand-8's commits replayed, with no diagnostic commits. Command line adds no_console_suspend initcall_debug rcupdate.rcu_cpu_stall_ftrace_dump=1 ftrace_dump_on_oops. Booted 2026-11-20 14:47Z.

Method:

  • kprobe and kretprobe events (in_* / out_*) on pci_pm_resume_noirq, genpd_resume_noirq, apple_pcie_resume_noirq and pci_pm_resume_early, each recording the device name. They go into the ring buffer that the RCU stall dump prints.
  • 01:00.0, 01:00.1 and 01:00.2 (the Apple Wi-Fi/BT functions below 00:00.0) unbound first.
  • Then pm_test=platform.

Result:

  • apple_pcie_suspend_noirq returned 0 after 38 usecs: the D0 check passes again. Then the 5 s wait.

  • Noirq resume stops at 14:49:00Z. The RCU stall followed, with rcu_sched kthread timer wakeup didn't happen for 30152 jiffies, and the ftrace buffer was dumped.

  • In the dump, every PCI pci_pm_resume_noirq has a matching return: 00:00.0–00:03.0, 02:00.0 (SD) and 03:00.0 (10GbE). So does apple_pcie_resume_noirq.

  • The genpd_resume_noirq entries with no return are:

    • 38a280000.usb;
    • two ACIO power domains (genpd:1:381ac0000.acio, and a second genpd:1:…acio cut short by interleaving);
    • the ATC PHY domains genpd:2:2a3000000.phy and genpd:3:2a3000000.phy.

    Resume runs asynchronously, so several of these were in flight at once. The console interleaves CPUs, so this list is read from a partly garbled dump.

What it shows:

  • The hang is not in the Wi-Fi/BT functions, which were unbound. It is not in any PCI function's noirq resume either: all of those returned.
  • Hypothesis: the power-domain resume of the USB-C/Thunderbolt blocks (dwc3, ACIO, ATC PHY) doesn't complete after a noirq phase that runs to its end. Before this PR, the host's D0 refusal aborted the noirq phase, so those domains were never powered off and on again in s2idle. The "timer wakeup didn't happen" line fits a CPU stuck with interrupts masked in that path.
  • This PR's selection (NO_D3 on a boot-trained adopted link) only moves s2idle past the point where it used to stop. The hang is in other drivers, and the fix doesn't belong in this PR.

Next:

  1. The same run with the Type-C/Thunderbolt consumers of those domains unbound, or with those genpds kept on through suspend, to confirm.
  2. Then name the domain, from apple_pmgr_ps and genpd debug.

This PR stays in draft. s2idle is still not shown to work.

🤖 Generated with Claude Code

@satchlj

satchlj commented Nov 20, 2026

Copy link
Copy Markdown
Member Author

Claude Opus 5.5 on @satchlj's behalf: s2idle now works on J775c with this PR plus #2592, so this is out of draft.

Kernel: cand-10, 6ebd4f505fd5 = integration/aurora df62a8c4d38d + 84 (#2488's head, this PR, and #2592's DT change). Booted 2026-11-20 17:45Z.

What each change does:

Selection is unchanged: the port's adopted-link state.

🤖 Generated with Claude Code

@satchlj
satchlj marked this pull request as ready for review November 20, 2026 17:54
@RyanTheTide

Copy link
Copy Markdown
Contributor

The selection by the port's own state looks right, and the cand-5 log shows exactly the device this keeps in D0.

  • apple_pcie_on_boot_trained_link() (pcie-apple.c:3663-3674) walks the ports the same way as the existing apple_pcie_below_kernel_trained_port() (pcie-apple.c:6740-6753) and returns its negation. One helper used in both places would keep the two definitions of "boot-trained" from drifting.
  • adopt_link is set for all of T6050, so this also marks the N1 functions on the J716c, J714s, K114c and K116s, and they lose runtime D3 there too. The comment above apple_pcie_check_d0() says the J716c radio drivers already stay out of D3, so the cost may be nothing, but the description only covers the J775c. An s2idle cycle and an idle power reading on a J714s or J716c with this change would cover the laptops.

Assisted-by: LLM (Claude:Opus 5.5)

@satchlj

satchlj commented Nov 21, 2026

Copy link
Copy Markdown
Member Author

Codex GPT-6.2 Sol on @satchlj's behalf.

Both the NO_D3 setup and the s2idle D0 check now use one helper to identify the root port and its training source. The D0 check remains conservative when it cannot identify the port. drivers/pci/controller/pcie-apple.o compiled for ARM64 with 16 KiB pages. This revision is compile-checked only and untested on hardware. We don't have a J714s or J716c available for the laptop s2idle and idle-power row.

@RyanTheTide

Copy link
Copy Markdown
Contributor

Re-review of 14c0fd3: Blockers: 1.

Earlier points:

  • Two copies of the "boot-trained" walk: resolved. drivers/pci/controller/pcie-apple.c:3663-3675 is now the only walk, used by the NO_D3 marking (pcie-apple.c:3702-3703) and the s2idle D0 check (pcie-apple.c:6762-6764). The behaviour is unchanged: bootloader == !port->pwren gives the old !pwren and !!pwren results, and an unmatched port still returns false, so the D0 check still runs on it. The new adopt_link test does not change the D0 check, because that check only runs on adopted hosts.
  • NO_D3 on the other T6050 machines (J716c, J714s, K114c, K116s): not resolved. The code still marks their N1 functions, and no s2idle or idle-power row from a laptop exists yet. This is not a code blocker, but the description should say these machines are untested, and a row should come before they are counted as covered.
  1. Commit 8d09565 ("PCI: apple: keep a boot-trained adopted link in D0 through suspend") has no Signed-off-by and ends with a Co-Authored-By: Claude Opus 5.5 trailer. It needs the author's Signed-off-by and that trailer dropped (or fold 14c0fd3 into it with one sign-off).

HW row: J775c at 14c0fd3 (with #2592): pm_test=platform and 3 RTC-woken rtcwake -m freeze cycles resume with rc 0; plus one J714s or J716c s2idle cycle and an idle power reading with and without this change; 0 AVC, 0 failed units, 0 new kernel errors.

Assisted-by: LLM (Claude:Opus 5.5)

@satchlj

satchlj commented Nov 21, 2026

Copy link
Copy Markdown
Member Author

Claude Opus 5.5 on @satchlj's behalf: trailer-only rewrite, removed Co-Authored-By from 1 commit(s); same trees, Signed-off-by kept (14c0fd3 -> f90f377, --force-with-lease).

@DjDeveloperr

Copy link
Copy Markdown
Contributor

J714s (Apple M5 Pro / T6050) combined-stack s2idle + deep hardware row: PASS

  • Exact tested stack: base cc145705c592 + #2007 (36ef4f8b60cf, incl. #1743) + #2592 (a142752404ed) + #2557 (f90f377c8c5e) + #2664 (6fb5e78f9b46) + #2739 (075bd14d3dec, T6050 PCIe-C s2idle_hold_domain), rebuilt t6050-j714s.dtb (1fbcc4fe2c55, MTE enabled; no_console_suspend ignore_loglevel appended for serial capture; Dell Thunderbolt dock attached with active PCIe-C tunnel + USB3/DP).
  • s2idle (rtcwake -d rtc1 -m freeze -s 15): 3/3 PASS (asleep 14.5 s / 14.0 s / 14.6 s, rtc_delta=20 s each, macsmc-rtc IRQ 118 0 -> 1 -> 2 -> 3, pciec_ebusy=0, refusal=0, disp=3/3, usb=20, wifi=1, bt=1, nvme=1, aer=0, link_err=0, gpu_lost=0).
  • deep regression (rtcwake -d rtc0 -m mem -s 15): 1/1 PASS (asleep=13.1 s, rtc_delta=23 s, refusal=0, disp=3/3, usb=20, wifi=1, bt=1, nvme=1, aer=0, gpu_lost=0).
  • Regression line: suspend_stats success=0 -> 4, fail=0; GPU LIVE (vulkaninfo + compute dispatch) pre/post; 0 kernel oops/WARN, 0 DART faults, 0 SELinux AVCs, 0 new failed systemd units (only pre-existing offline tailscaled.service).
  • Scope note: Combined-stack J714s verification (not isolated from the other four s2idle PRs listed above).

Assisted-by: LLM (Gemini 4 Neon)

@satchlj
satchlj force-pushed the satchlj/pcie-apple-adopted-d0-20261120 branch from f90f377 to 6747193 Compare November 21, 2026 21:53
@satchlj

satchlj commented Nov 21, 2026

Copy link
Copy Markdown
Member Author

Claude Opus 5.5 on @satchlj's behalf: added the missing Signed-off-by: Satya Benson to 1 commit(s); same trees, nothing else changed (f90f377 -> 6747193, --force-with-lease).

The host's s2idle check refuses an adopted link unless every function on
it is accessible in D0. On J775c the PCI core's system-suspend path puts
the boot-trained root port 00:01.0 in D3hot first (PMCSR 0x000b at the
host's noirq check, with runtime PM forbidden on every PCI device), so
s2idle is refused on every attempt.

Mark a root port the boot loader trained on an adopted link, and every
function below it, PCI_DEV_FLAGS_NO_D3 when it is enabled. The selection
is the port's adopted-link state: an adopted port that Linux trains
itself (pwren-gpios) is not marked. Those functions also lose runtime
D3.

Signed-off-by: Satya Benson <satya@benson.earth>
Use one root-port walk and training test when setting NO_D3 and
checking D0 for s2idle. Still check D0 when the root port is unknown.

Signed-off-by: Satya Benson <satya@benson.earth>
@satchlj
satchlj force-pushed the satchlj/pcie-apple-adopted-d0-20261120 branch from 6747193 to 513067f Compare November 22, 2026 10:43
@RyanTheTide

Copy link
Copy Markdown
Contributor

Re-checked at 513067f. f90f377 and 6747193 have the same tree as 14c0fd3, and the move to 513067f is a rebase onto d1b022d that only changes hunk context, so the code is what I reviewed. Both commits are now signed off, which closes the attribution point.

On the other T6050 machines: the J714s combined-stack row above (s2idle 3/3, deep 1/1, at an identical tree) covers suspend on one laptop. Two things are left:

  • The idle-power cost of NO_D3 on a laptop has still not been measured. That needs one idle reading on a J714s or J716c with and without this change.
  • The "Untested" line in the description should list the J716c, K114c and K116s next to the J775D, and cite the J714s row.

Assisted-by: LLM (Claude:Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants