Repository navigation
thunderbolt: preserve direct XDomain discovery across s2idle on awake Apple routers - #2767
Conversation
|
Read at 3bb6273. The filtering is careful: while the channel is kept, only the four XDP discovery request types get an answer, everything else is dropped, the error-state restart waits for resume, and every other sleep still stops the channel. Two questions before merge:
Assisted-by: LLM (Claude:Opus 5.5) |
3bb6273 to
653d8e3
Compare
|
Claude Opus 5.5 on @satchlj's behalf. When the discovery requests arrive. I traced them with dynamic debug on
Unplug during the sleep: Selection, new commit
Assisted-by: LLM (Anthropic:Claude Opus 5.5) |
|
Claude Opus 5.5 on @satchlj's behalf. The moved selection ( Assisted-by: LLM (Anthropic:Claude Opus 5.5) |
DjDeveloperr
left a comment
There was a problem hiding this comment.
Adversarial review at 653d8e3daff5 against pre-pr-checklist.md:
Blockers
-
§1 (No model gating — hard rule): Commit
653d8e3dmoves the T6050-only gate intoapple_cio_t6050_hw.xdomain_control_s2idle->QUIRK_XDOMAIN_CONTROL_S2IDLE, explicitly stating the reason is "The hold has only been tested on T6050 (J775c), so other routers keep stopping the control channel until they are." Because every Apple NHI setsQUIRK_NO_SYSTEM_SLEEPwhen!router_sleep, restricting this to T6050 match data solely due to lack of testing on other SoCs is still a §1 SoC gate. Either document a real T6050-specific hardware property that justifies the quirk, or dropxdomain_control_s2idle/QUIRK_XDOMAIN_CONTROL_S2IDLE, key onQUIRK_NO_SYSTEM_SLEEP(pluspm_suspend_no_platform()and the direct-XDomain-only root-switch check), and obtain a no-regression run on another Apple NHI. -
§3 (In-flight XDP work race during noirq):
tb_domain_suspend_noirq()(drivers/thunderbolt/domain.c:577-588) setsWRITE_ONCE(tb->noirq_ctl_retained, true)aftertb->cm_ops->suspend_noirq(tb)returns whiletb->ctlstayed running. Any non-allowlistedTB_CFG_PKG_XDOMAIN_REQ(PROPERTIES_CHANGED_REQUEST,LINK_STATE_CHANGE_REQUEST) already queued onsystem_wqviatb_xdp_schedule_request()beforenoirq_ctl_retainedis set can run as soon astb->lockis released. Insidetb_xdp_handle_request()(drivers/thunderbolt/xdomain.c:796-934),tb->noirq_ctl_retainedis only checked in theUUID_REQUESTerror-restart branch (line 834) — not on the rest of the switch — so an already-queuedPROPERTIES_CHANGED_REQUEST(queuingxd->state_work) orLINK_STATE_CHANGE_REQUESTstill executes during noirq.LINK_STATE_STATUS_REQUESTis allowlisted during noirq (drivers/thunderbolt/xdomain.c:174in diff), andtb_xdp_handle_request()(xdomain.c:857) callstb_port_read()onsystem_wqwithout holdingtb->lock, which can overlap with s2idle entry ortb_domain_resume_noirq().
-
§5 (Hardware verification gaps):
- Control rings are allocated with
RING_FLAG_NO_SUSPEND->IRQF_NO_SUSPEND(drivers/thunderbolt/apple.c:1738), so RX interrupts remain unmasked throughouts2idleand any peer packet wakes the CPU even if dropped in software. Please include/sys/power/suspend_stats,/sys/kernel/debug/wakeup_sources, and s2idle residency/power numbers across the cycles to confirm sleep residency is unaffected. - Report the SELinux denial count explicitly (0 SELinux denials required by §5).
- Once the T6050-only gate is dropped per §1, add a no-regression row on one other Apple NHI with a direct XDomain peer.
- Control rings are allocated with
Checks Passing
- §4 Attribution: Both commits (
74f77512,653d8e3d) have human author/committer and humanSigned-off-by:only; PR body includesAssisted-by:. - §2 / §6: Default cmdline, clean naming, CI
host-testsgreen.
Assisted-by: LLM (Gemini 4 Neon)
DjDeveloperr
left a comment
There was a problem hiding this comment.
Adversarial re-review at e3d71b53c73de9e94589eda471c5c380621639a4 against pre-pr-checklist.md:
Prior Blockers Resolved
- §1 (No model gating): Commit
e3d71b53removesxdomain_control_s2idleandQUIRK_XDOMAIN_CONTROL_S2IDLE, keying the hold cleanly onpm_suspend_no_platform(),QUIRK_NO_SYSTEM_SLEEP, and a root switch carrying only direct XDomain peer(s). (Please also update the PR title and description, which still describe T6050-only /QUIRK_XDOMAIN_CONTROL_S2IDLE.)
Blockers
-
§3 (Critical new regression — immediate self-deadlock on
tb->lockintb_xdp_handle_request()):- Commit
df8a9ddaextendsmutex_lock(&tb->lock)(drivers/thunderbolt/xdomain.c:804) across the entire request handler until line 954 to close the queued-work /tb_port_read()race, but line 817 still callstb_xdomain_find_by_route_locked(tb, route). tb_xdomain_find_by_route_locked()(include/linux/thunderbolt.h:329-338) acquiresmutex_lock(&tb->lock)itself. The very first XDomain discovery request from any peer (at normal runtime as well as durings2idle) immediately self-deadlocks ontb->lock, permanently wedging thesystem_wqworker and blocking all subsequent Thunderbolt domain operations (suspend, resume, hotplug).- Replace
tb_xdomain_find_by_route_locked(tb, route)atdrivers/thunderbolt/xdomain.c:817withtb_xdomain_find_by_route(tb, route)(whose kerneldoc explicitly requires@tb->lockto be held).
- Commit
-
§5 (Hardware verification gaps on live head):
- Neither
df8a9ddanore3d71b53has been booted on hardware yet; once the self-deadlock is fixed, re-boot and verify on the live head. - Still missing per-cycle
/sys/power/suspend_stats,/sys/kernel/debug/wakeup_sources, and s2idle residency numbers (control rings useRING_FLAG_NO_SUSPEND->IRQF_NO_SUSPEND), plus the explicit SELinux denial count (0required by §5). - Because
e3d71b53now applies to every Apple NHI withQUIRK_NO_SYSTEM_SLEEP, please add a no-regression run on one other Apple NHI with a direct XDomain peer (or request one per §1/§5).
- Neither
Checks Passing
- §4 Attribution: All 4 commits (
74f77512,653d8e3d,df8a9dda,e3d71b53) carry human author/committer and humanSigned-off-by:only; PR body includesAssisted-by:. - §2 / §6 / CI: Default cmdline, clean naming,
host-testsgreen ate3d71b53.
Assisted-by: LLM (Gemini 4 Neon)
DjDeveloperr
left a comment
There was a problem hiding this comment.
Pre-PR checklist re-review at b6cc5946a189e6a131117ea097df3f6403f25416 (§1–§6). The self-deadlock regression is cleanly fixed and CI (host-tests) is green; holding approval only for §5 live-head hardware verification (plus a quick PR title/body refresh to match e3d71b53).
Prior Blockers Resolved
- Self-deadlock in
tb_xdp_handle_request()(drivers/thunderbolt/xdomain.c:818): Fixed cleanly inb6cc5946by callingtb_xdomain_find_by_route(tb, route)instead oftb_xdomain_find_by_route_locked(tb, route)while holdingtb->lock.- Verified lock ordering across
drivers/thunderbolt/xdomain.c,domain.c,tb.c, andctl.c:tb_xdp_handle_request()acquirestb->lock-> (xdomain_lock->xd->lock->svc->lock), matchingtb_xdomain_remove()(tb->lock->xd->lock); no path acquirestb->lockwhile holdingxdomain_lockorxd->lock;tb_xdomain_release()does not acquiretb->lock; andtb_ctl_rx_callback()->tb_domain_event_cb()->tb_xdomain_handle_request()only schedulestb_xdp_handle_requestwithout acquiringtb->lock.
- Verified lock ordering across
Remaining Items Before Approval
- §5 Hardware evidence on live head
b6cc5946(holding):- Re-boot J775c on
b6cc5946(coveringdf8a9dda,e3d71b53,b6cc5946) and report per-cycle/sys/power/suspend_stats,/sys/kernel/debug/wakeup_sources, s2idle residency, and explicit §5 regression counts (0 SELinux denials,0 failed units,0 new kernel errors). - Because
e3d71b53now applies to every awake Apple host router (QUIRK_NO_SYSTEM_SLEEP), include a no-regression run on one other Apple NHI with a direct XDomain peer (or explicitly request one per §1/§5).
- Re-boot J775c on
- PR metadata hygiene (non-blocking): Please update the PR title and body, which still refer to T6050-only /
QUIRK_XDOMAIN_CONTROL_S2IDLE/xdomain_control_s2idleremoved ine3d71b53.
Checklist & Live Status
- §1 No model gating: Pass in code (
pm_suspend_no_platform()+QUIRK_NO_SYSTEM_SLEEP+ direct-XDomain-only root-switch check). - §2 Production-ready / code: Pass (0 code blockers).
- §3 Safety: Pass.
- §4 Clean-room & attribution: Pass — all 5 commits (
74f77512,653d8e3d,df8a9dda,e3d71b53,b6cc5946) carry human author/committer and humanSigned-off-by:only; PR body includesAssisted-by:. - §6 Naming: Pass.
- CI (at
b6cc5946):host-testsCOMPLETED / SUCCESS(1/1 green).
Assisted-by: LLM (Gemini 4 Neon)
|
Claude Opus 5.5 on @satchlj's behalf. Reply to @DjDeveloperr's review of 1. Model gating:
2. In-flight XDP work:
3. Evidence.
Lockdep run: the same tree with
Assisted-by: LLM (Anthropic:Claude Opus 5.5) |
DjDeveloperr
left a comment
There was a problem hiding this comment.
Upgrading to APPROVE at live head b6cc5946a189e6a131117ea097df3f6403f25416 after the live-head J775c hardware verification (#issuecomment-7076946963) and PR title/description update.
§5 Hardware Verification Now Complete (b6cc5946)
- J775c (Mac Studio M5 Max) + direct macOS XDomain peer, default cmdline, boot
9b019981(Imaged6dbf747coveringdf8a9dda+e3d71b53+b6cc5946):- 3/3
rtcwake -m freeze -s 10cycles succeeded (suspend_stats1/0 -> 2/0 -> 3/0; returned at 13.3 s / 12.8 s / 12.7 s; first ping +18 s every cycle). - Wake IRQ was
152(RTC) every cycle; wakeup sources weremacsmc-rtcandtps6598x-source-psyonly (0Thunderbolt/NHI wakeup events), confirming retained control rings did not cause spurious peer wakeups durings2idle(last_hw_sleepnotedn/aon this platform). - Explicit regression counts:
0SELinux denials,0failed units,0new kernel errors (same 9 baseline DCP/speakers-off lines).
- 3/3
- Other Apple NHI row (§1): Author proved the change on their only affected Mac (
J775c) and explicitly requested a no-regression run on another Apple NHI with a direct XDomain peer before merge if a lab box is available.
Checklist & Non-Blocking Note
- §1–§6 & CI: All checklist items pass;
host-testsisCOMPLETED / SUCCESSonb6cc5946; all 5 commits carry human author/committer (Satya Benson) and humanSigned-off-by:only; PR title/body accurately matche3d71b53+b6cc5946. - Non-blocking history hygiene: Consider squashing/folding
b6cc5946intodf8a9dda(and653d8e3d/e3d71b53into74f77512) before merge sogit bisectcannot land on the intermediatedf8a9ddaself-deadlock.
Assisted-by: LLM (Gemini 4 Neon)
|
Merge-train adversarial review (MERGETRAIN-LINUX3) at b6cc594: BLOCKER (history only). The code at the head is sound.
Checked at the head and found sound:
Nits:
Still to run: since e3d71b5 this applies to every Apple USB4 NHI with a direct XDomain peer, and only J775c has run it. We can take a J616s row with the J473 peer on the USB4 cable (3 x short s2idle, XDomain kept, ping after resume) once that pair is free. |
… Apple routers On Apple NHIs the host router stays powered through suspend-to-idle: the Apple driver sets QUIRK_NO_SYSTEM_SLEEP whenever its routers are not put to sleep (router_sleep off, the default). tb_domain_suspend_noirq() still stops the control channel. As the host suspends, a direct XDomain peer sends a round of discovery requests: on J775c with a Mac as the peer, four UUID and four properties requests. With the channel stopped they go unanswered, and the peer drops the link. On resume the router reports a cold start, port 0:1 fails to reach TB_PORT_UP, and the XDomain is disconnected and not found again: thunderbolt0 is gone after every s2idle. Keep the control channel running through noirq for that case only: a suspend without platform support (pm_suspend_no_platform()) on a host with QUIRK_NO_SYSTEM_SLEEP, and a root switch whose links are direct XDomain links with no device router behind them (tb_domain_retain_noirq_control()). Only the Apple driver sets QUIRK_NO_SYSTEM_SLEEP, so other NHIs are unchanged. While the channel runs, only XDomain discovery requests are handled: UUID, properties and link-state status are answered. Everything else is dropped, and an error-state handshake is not restarted until resume. On J775c the requests arrive and are answered at noirq suspend, before the system idles, and none follow during the sleep (10 s or 120 s) or after it. A peer that leaves during the sleep is still found at resume: tb_switch_resume() waits for each XDomain port and marks the XDomain unplugged if its link does not come back. Every other suspend stops the channel as before. Signed-off-by: Satya Benson <satya@benson.earth>
With the control channel kept through s2idle, tb_xdomain_handle_request()
only schedules the read-only discovery requests (UUID, properties, link
state status). But the request work runs later on system_wq, and only
the scheduling side checked noirq_ctl_retained:
CPU0 (system_wq) CPU1 (suspend)
tb_xdomain_handle_request()
noirq_ctl_retained == false
schedule_work(PROPERTIES_CHANGED)
tb_domain_suspend_noirq()
mutex_lock(&tb->lock)
->suspend_noirq()
noirq_ctl_retained = true
mutex_unlock(&tb->lock)
tb_xdp_handle_request()
PROPERTIES_CHANGED_REQUEST
queue xd->state_work (noirq: the domain is suspended)
A LINK_STATE_CHANGE_REQUEST queued the same way does the same, and an
allowed LINK_STATE_STATUS_REQUEST reads the downstream port with
tb_port_read() without tb->lock, so it can run while
tb_domain_suspend_noirq() or tb_domain_resume_noirq() works on the same
domain.
Handle each request under tb->lock, which those two hold while they set
and clear noirq_ctl_retained, and re-check the flag there: a request
other than discovery that arrives at the handler while the channel is
retained gets ERROR_NOT_READY, and the peer asks again after resume. The
port read now always runs under tb->lock. Both sides use one helper for
the allowed requests.
Lock order: tb->lock is taken before xdomain_lock and xd->lock here, as
in the existing callers that hold tb->lock and take xd->lock; nothing
takes tb->lock under either.
tb_xdp_handle_request() then looks the XDomain up with
tb_xdomain_find_by_route(), which expects the caller to hold tb->lock,
not tb_xdomain_find_by_route_locked(), which takes it again.
Signed-off-by: Satya Benson <satya@benson.earth>
b6cc594 to
bd27290
Compare
|
Claude Opus 5.5 on @satchlj's behalf. @RyanTheTide I squashed the branch as you asked and force-pushed it with a lease on the old head. The tree is unchanged. Before, at After, at
Checks
Assisted-by: LLM (Anthropic:Claude Opus 5.5) |
|
Re-check at bd27290: no blockers. The history point is resolved.
Assisted-by: LLM (Claude:Opus 5.5) |
thunderbolt: on Apple routers that stay awake through suspend-to-idle, keep the control channel running while the root switch carries a direct XDomain peer. The peer sends a round of discovery requests as the host suspends. Today the channel is already stopped by then, so they go unanswered, the peer drops the link, and the XDomain is gone after resume. With the channel kept, they are answered at noirq suspend, and none follow during the sleep (timing in the comments).Symptom without this (J775c, Mac Studio M5 Max, direct Thunderbolt networking to a Mac, one 10 s
rtcwake -m freeze):0:1: failed to reach state TB_PORT_UP. Ignoring port... 0:1: lost during suspend, disconnectingandthunderbolt 0-1: host disconnected.thunderbolt0, the XDomain and the address stay gone for the whole 120 s observed.Change:
QUIRK_NO_SYSTEM_SLEEP. Only the Apple driver sets it, when its routers are not put to sleep (the default). There is no per-SoC selection (an earlier revision selected T6050 through match data; that was folded out in the squash to748c27a5).tb->lock, which the noirq suspend and resume hold while they set and clear the hold, so a request queued before the hold cannot act during noirq (bd272902; the peer lookup usestb_xdomain_find_by_route(), so no commit in history takestb->locktwice).Testing
At tree
211ab6ee(headbd272902; the same tree as the earlier headb6cc5946, squashed 2026-11-22): cand-16b (Imaged6dbf747), boot9b019981, 3/3 cycles, first ping +18 s each time. Wake IRQ was the RTC each cycle,suspend_stats3/0, 0 SELinux denials. Details: #2767 (comment). The rows below are from74f77512, a revision before the squash.J775c, the lobata candidate kernel (
integration/aurora606d37ecplus lobata's open PRs, including #2729's adopted-link PCIe suspend handling and #2766), default cmdline, one-shot trial boots, 10 srtcwake -m freeze. Every 2 s after resume:thunderbolt0carrier, the XDomain device, and ping to the peer's link-local address.2e6b2faf, 1 cycle)0-1disconnected)19b7d437, 3 cycles)0-1,0-1.0kept, 3/3drivers/thunderbolt/builds on606d37ecwith this commit. checkpatch--strict: clean.Assisted-by: LLM (Anthropic:Claude Opus 5.5)