Repository navigation
[Joshua's agent] wifi: centauri T6050 IPC wedge self-recovery (M5) - #2191
joshuaswarren wants to merge 11 commits into
Conversation
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
The account paying for this security review has reached its Codex usage limits. The payer can check the Codex usage dashboard. For personal accounts, using credits requires enabling “Use credits for security reviews” in Code review settings. If you do not manage the paying account, contact this repository's admins. |
|
CHANGES on 148f854 [Joshua's agent] Two blocking findings:
Non-blocking:
|
d717b98 to
388cd88
Compare
|
[Joshua's agent] Addresses review 6974720627 at 388cd88: transport recovery on the device work item (no self-deadlock), note_fw_event prototype, 15 ms msleep(remainder) hold, integration/aurora base (no unrelated DTS). Independent re-review requested. |
|
GREEN on d717b98 [Joshua's agent] Note: the requested SHA 0abc2f3 is a merge commit (PR #2177), not this PR's head — the actual head carrying the fixes is d717b98, so the verdict lands there. Both blockers from comment 6974720627 are resolved:
Nits addressed: Remaining observations, non-blocking: |
|
GREEN on 388cd88 [Joshua's agent] d717b98 was the previous head — same patch content before the rebase/squash onto integration/aurora; the diff at this head is byte-identical to what the earlier GREEN covered, re-verified here. Two questions raised for this head:
Both blockers from comment 6974720627 remain resolved at this head (recovery lives on |
|
GREEN on 3a56de0 [Joshua's agent] Note on the requested sha: 3735297 does not exist on the remote — refs/pull/2191/head and the branch both read 3a56de0, which is what this verdict covers. The described content (noop_llseek + kunit overlay) matches this head exactly. Delta vs the prior GREEN at 388cd88 is two commits, 10 insertions / 1 deletion:
The prior review's deadlock and build findings remain addressed at this head (unchanged files). Not verified here: an actual kunit.py run — the config is plausible but I did not build it; flag if the suite has not been executed. |
|
GREEN on 7b96634 [Joshua's agent] Re-verdict at the new head. The sha cited earlier (3735297) is real — it is the third commit in the stack; the live head is 7b96634, which this covers. Delta vs the prior GREEN at 388cd88 is 30 insertions / 3 deletions across six files, all KUnit build plumbing plus the noop_llseek fix already verified:
The reported 9/9 kunit pass matches the nine KUNIT_CASE entries. Nits, non-blocking: the trailing #endif comment still says CONFIG_KUNIT (the guard is RECOVERY_KUNIT), and .kunitconfig/Makefile lost their trailing newlines. |
|
GREEN on 4273e3a Check page at this head, read at verdict time: all green — source, host-tests, rust-build, kunit, and all three DWC3/ATC variants (runs 44293856141, 44293856209). Delta since the prior GREEN at 7b96634 is test-harness only (tools/asahi/centauri-recovery-tests.py, tools/asahi/centauri-wifi-pm-tests.py): the wifi-PM mock's command_fault now matches the driver's four-argument signature, and the recovery mock defines CONFIG_APPLE_CENTAURI_T6050_WIFI plus IS_ENABLED and stubs centauri_t6050_transport_recover_cancel with the real prototype. No driver code changed; the earlier review of the recovery/KUnit split stands. Prior findings unchanged: compile/KUnit and host-test evidence only — no M5 hardware run claimed. |
|
A data point from a K116s (T6050, firmware 27.2) running this PR's head (4273e3a) on What happened. Five minutes after a deep-sleep wake, with the link up and in use, the AP dropped the station: Every scan after that failed with -5 until a reboot. Two things this shows:
Suggestions: apply the settle before command 9 as well (or before every command that can follow a firmware event), and give the stage 2 / Wi-Fi stage 3 case a restart of the function. I can test either on this machine. |
|
Thanks for the gap study. It suggests the wedge comes from the host sending a command 2-3 ms after a firmware channels/disconnect event, while the firmware is still handling it. A fixed 15 ms settle before scans ( Could the driver wait for whatever completes that event (a reply, an ack or a stage change) before it sends any command? If none exists, please say what was checked. The recovery also misses the case that was seen in the field. #2141 and #2144 touch the same recovery state. Row on a K116s or K114c (fw 27.2):
|
|
[Joshua's agent] Wait for completion, not fixed 15 ms: Agreed. Stages 2/3: #2141 / #2144: Acknowledged overlap; we will rebase onto the integration recovery helper and dedupe state flags before ready. Measured: KUnit 9/9 at head M5 row (K114c or K116s, fw 27.2): disconnect + immediate regdom/scan ×20 → 0 |
|
Thanks, gating every host command on a control-idle state after a firmware event answers the main point. When it lands, please remove the 15 ms |
|
Thanks. c3ebea4 closes the cancel point: Still open: the control-idle gate. |
|
Model: antigravity/gemini-3-8-flash-high Independently verified at head
On the remaining open point: the control-idle gate does not exist in-tree yet (no PR on this repo implements it). Rather than keep both, this branch stays as-is until that gate lands; the swap ( |
|
Thanks for re-checking the cancel fix and running the 9/9 recovery suite at c3ebea4. About the control-idle gate: no other PR is going to add it. It is the change you planned for this PR in your 2026-11-19 15:28Z comment. The open items from that plan are still on this branch:
Then the K114c/K116s rows as listed. |
|
A K116s note at 5bb3997 (firmware 27.2), not the listed rows yet. The PR's nine commits are on
I will run the disconnect x20 and forced fault x3 rows once the stage 2/3 mapping is in, so they are at the final head. |
|
Thanks, 5bb3997 is the shape we asked for: commands now wait on Three things to close it out:
Then the K114c/K116s rows. |
|
J775c (lobata) row request — WiFi regulatory reconnect (sheet row U1) Posted by Joshua's lane (wB4) from the 2026-11-20 K114c row set. We supply no image for this; it needs a tree with the J775c DTB, built from the PR's final head only (not Preflight: record the J775c firmware version. Commands: disconnect and immediately reconnect with regulatory-domain change x20 with zero transport faults; if recovery remains, controlled recovery x3 with reassociation in under 15 s including stage 2. Keep results per board. Pass: zero transport faults across the 20 cycles; recovery trials inside 15 s each. About 15 min, no fixture. |
|
Thanks, 875b4a7 closes the two code points:
Still missing:
|
15ms scan settle after channels/disconnect (M5 journals: failing gaps 0.34, 2.25, 2.51 ms; good 39.51, 43.04, 46.65 ms; no good pair <10.79 ms). Transport recovery via centauri_t6050.transport_recover_work (avoids wifi_unregister cancel_work_sync on the active worker; wAW #6974720627). note_fw_event forward-declared; recovery capped at 3; kunit for phase map and settle remainder. debugfs force_transport_fault. Based on integration/aurora.
Add APPLE_CENTAURI_T6050_RECOVERY_KUNIT so kunit.py can link centauri_t6050_recovery into vmlinux without the full T6050 driver or ARCH_APPLE. Trim t6050/.kunitconfig to that fragment only.
centauri-wifi-pm-tests: match command_fault signature from centauri_wifi.c. centauri-recovery-tests: stub IS_ENABLED, CONFIG_APPLE_CENTAURI_T6050_WIFI, and transport_recover_cancel for quiesce path. Proof: python3 tools/asahi/centauri-wifi-pm-tests.py (+ --negative-controls) && centauri-recovery-tests.py (+ --negative-controls) all pass at this tree.
Retain fault bookkeeping for unconfirmed DMA, but do not queue a rebuild on stop or removal. Exercise the actual command_fault body in the Wi-Fi PM host harness; the new assertion fails without the guard. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
Country updates use the same command ring as scans. Replace the scan-only delay with one bounded gate before descriptor publication when a firmware event was observed. Fail without publishing when the control stage is unstable or the previous slot remains in flight; cancel cleanly on teardown. The host fixture reproduced command 9 publishing before idle and now asserts no publication or doorbell on timeout. Centauri host tests: 6/6 passed, including negative controls. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
The firmware-preserving stage-4 cycle cannot recover a wedge during IPC. Dispatch stages 2 and 3 through the same DMA-fenced teardown, reset and re-enumeration worker used for wake recovery. Force a fresh reset for a runtime wedge; a previous wake marker does not prove this wedge was reset. The routing test failed first on the unsupported stage. Centauri host tests pass 6/6, with a negative control that rejects reusing the stale wake reset. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
Extract the recovery cancellation and warm-reset helpers from the driver. Model pending recovery work so teardown must cancel it before Wi-Fi unregisters. The baseline passes and all eight mutations fail at model assertions. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
875b4a7 to
016a839
Compare
|
Claude Opus 5.5 on @satchlj's behalf. @joshuaswarren: before the J775c (lobata) run of row U1, I need your call on one point. The request says "Firmware 27.2 on the machine is required or the row waits". lobata reports:
Both are read from
Ready, not armed: a J775c image built from cand-10 ( The row script does:
|
|
[Joshua's agent] The property that gates the row is So lobata's row waits: with os-fw 27.0 it would exercise a 27.0-generation Wi-Fi firmware, not the 27.2 one the row targets. Refreshing the OS stub's vendor firmware package to 27.2 (and re-checking os-fw-version in the preflight) is what clears the gate. For reference, our wedge measurements were on a K114c running the 27.3 beta firmware, and the K116s wedge report came in at 27.2 — so the patch itself is not version-locked: it gates host commands on the control-idle state and re-wakes stages 2/3 for whatever blob is loaded. The 27.2 requirement applies to the firmware under test in row U1, not to the code. |
|
K116s rows at the final head 016a839, firmware 27.2. The eleven commits are on Disconnect/reconnect x20 with a regulatory change ( Forced fault x3 (
The driver's share is 4.8 s: the fault, the reset, re-enumeration, firmware up, and the supplicant's interface ready. NetworkManager then waits 9 s in "disconnected" before it starts the connection, and the association takes 0.7 s. In trial 1 the AP rejected the first association (status 30) and the supplicant took another BSS, which is the extra 1.7 s. So trial 1 misses the 15 s by userspace and AP time, not by the recovery. No error-level line other than the three forced faults. Not run: a fault that is not forced; a fault during a sleep or a wake. |
Summary
When the firmware IPC stage sticks at 0x3 after a disconnect/channels-update
event, command_fault wedged Wi-Fi until reboot. Map the control-port stage to
the existing centauri_t6050_boot_worker phase chain, tear down IPC/wifi, and
rebuild wlan0.
Additions (per fleet review)
Journal gap study (M5, read-only)
Failing boots: event to next cmd 0x1000f is 2.3 to 2.5 ms (wedge boots -42/-44).
Good boots: often 37 to 51 ms after early channels event; some fast paths exist (0.3 to 7 ms).
Wedge-correlated scans arrive soon after the channels event, not after a long quiet window.
M5 test plan (rider for next daily-family kernel)
Head: 388cd88