Repository navigation
thermal: apple-pmp: keep the power budget over a dark sensor sample - #2699
Conversation
When every PMP die sensor is dark past the gated hold, apple_pmp_sample() caches -ENODATA and the thermal guard holds the CPU floor for that sample. The next power-budget sample then turned the cached error into -EAGAIN, and apple_pmp_power_fail() put the P, E and GPU budgets at full restriction. That budget comes back only after three good samples and then one step per second, larger weight first, so after a single dark sample on a burst out of idle the P cluster spent about 20 s well below its clocks (1260 -> 3096 MHz over more than 10 s on a 1T load). The rails were read; nothing about the power telemetry failed. Skip that budget sample instead of failing it while the sensors are only dark and the thermal sample is fresh. The thermal guard still holds the floor for the dark sample, and a dark span longer than the policy's 2 s gap, a stale thermal sample, a disabled zone or any other error still fail the budget closed as before. Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
|
Blockers: 1. What I checked. The change adds no temperature or power constant: the dark branch reuses the existing 500 ms freshness test (drivers/thermal/apple-pmp-thermal.c:207), and nothing substitutes a value for the missing reading. On the CPU side a dark sample still fails safe.
Nit: the comment at :204 and the body say a dark span longer than the 2 s gap "still fails closed". While the span lasts, the gap check never runs, because Not changed here, but relevant to the safety reading: the budget this protects is the existing provisional policy (pmp-power-policy.h:4-14, 50: fixed sustainable watts, burst energy and temperature target). It does not come from the machine, so this PR neither improves nor worsens that. Body: drop the internal lane name in the first paragraph and the lab-store path in the details. Describe the run itself (machine, load, numbers). HW row: J713, kernel built from this head (not an insmod'd module), default command line. Run the 8 x (all-core 12 s, idle 3 s) loop at 5 Hz Assisted-by: LLM (Claude:Opus 5.5) |
The previous change skips the budget sample while every sensor is dark, but it still republished the GPU's last budget level with the old sample time, so the GPU kept that level (possibly 0) until the publication went stale 3 s later. The CPU clusters do not depend on the budget there, because the thermal guard holds them at the floor for the dark sample. Publish nothing for the GPU in the dark branch instead: a zero publication selects full protection, and the next real budget sample republishes its level with a fresh time, so the GPU comes back at once and no release ladder runs. Also say what closes a long dark span: the 3 s APPLE_PMP_POWER_STALE_NS, then the gap path at the first sample after it. Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
|
Thanks, all addressed in 96ea511 (pushed on top, no force push):
Assisted-by: LLM (Claude:Opus 5.5) |
|
Merge-train adversarial review (MERGETRAIN-LINUX2) at 96ea511: NO BLOCKERS. Code read only, nothing built here. Before the merge it still needs Dj's verdict at this head; the missing rows are listed below. Full review#2699 thermal: apple-pmp: keep the power budget over a dark sensor sample @96ea511eVERDICT: NO BLOCKERS What I checked in drivers/thermal/apple-pmp-thermal.c:
Nits:
Missing rows:
|
DjDeveloperr
left a comment
There was a problem hiding this comment.
Adversarial review at live head 96ea511e2cd79093942468a67120b0782bdc5d67 against the pre-PR checklist: 0 code/policy/attribution/CI blockers, and §5 hardware verification is complete on J713 (the sole T8132 machine in the lab).
Checked & clean
- §1–§2 No model gating / no hacks: No chip/board/firmware gates, no new module/cmdline parameters, no magic sleeps or retries; reuses the existing 500 ms freshness check.
- §3 Safety & deep code correctness (
drivers/thermal/apple-pmp-thermal.c):- The dark-skip predicate (
!ret && pmp_thermal.enabled && pmp_thermal.cached_error == -ENODATA && now - pmp_thermal.cooling.last_ns <= 500000000ULL) is appropriately narrow: SMC rail-read errors and the all-zero firmware-reset check (!mw[0] && !mw[1]→ret = -ENODATAat line 192) bypassdarkbecause!retis false, andcached_error == -ENODATAoriginates solely fromapple_pmp_sample_all()when every die sensor is dark with no other error. - CPU clusters remain fail-closed via
apple_pmp_cooling_sample()(level = 8,hard = true) →apple_pmp_guard(8). - GPU remains fail-closed via
apple_pmp_gpu_publish(pmp_thermal.sensors, epoch, 8, 0)(zero publication selects level 8 inapple_pmp_gpu_lease_level()), and newly attached GPU epochs still initializep->level[2] = 8beforeif (dark), then recover immediately on the next real sample without the 23 s release ladder. - Prolonged darkness still fails closed:
p->last_nsdoes not advance whiledarkis true, soapple_pmp_guard()and hwmon expire afterAPPLE_PMP_POWER_STALE_NS(3 s) and the first real sample after a >2 s span takesapple_pmp_power_sample()'s gap path (apple_pmp_power_fail()). - Interaction with #2730: Complementary (
#2699handles emitted-ENODATAsamples such as hot-to-idle transitions above 90 C where the gated hold does not apply);#2730touches adjacent lines inapple_pmp_power_work()and can rebase cleanly once#2533lands.
- The dark-skip predicate (
- §4 Attribution: Both commits (
1a925b273afd,96ea511e2cd7) carry human author/committer (Ryan Murray <ryan@aurorasilicon.org>) and humanSigned-off-by:only (zero AI trailers); PR body includesAssisted-by:. - §5 Hardware & CI evidence:
host-testsCI is green on96ea511e(includingpmp-power-tests). J713 hardware run (macOS 27.2, default cmdline, both commits compiled in) exercises the exact dark-skip path including GPUpower=8(sensor=Some(ENODATA)) →power=042 ms after the next real sample, with A/B telemetry-fault counts and regression summary (0 AVC,0 failed units,0new driver errors). - §6 Naming: Clean.
Non-blocking nits
- Bisectability: Commit
1a925b27alone briefly leaves the GPU budget un-invalidated for up to 3 s over a dark sample until96ea511efixes it; squashing on merge keeps every commit fail-closed. 500000000ULLis now repeated on lines 210, 213, and 486 — worth naming in a future cleanup.
Assisted-by: LLM (Gemini 4 Neon)
Problem
On the M4 MacBook Air (macOS 27.2 firmware), a load that starts right after the PMP die sensors go dark can run well below its clocks for about 20 s. The log shows
sensor error -61; CPU minimum caps retainedfollowed bypower telemetry error -11, retaining minimum CPU/GPU ceilings; on a single-thread job the P cluster climbed from 1260 to 3096 MHz over more than 10 s.The -61 is every die sensor being dark for one sample (their clusters powered down, for example right after a hot load stops). The thermal guard handles that sample by itself and is back in about 1.6 s. But if the 500 ms power-budget worker runs inside the dark window, it turns the cached thermal error into -EAGAIN, and
apple_pmp_power_fail()sets the P, E and GPU budgets to full restriction. The budget then waits for three good samples and gives back one step per second, GPU first, so the P cluster stays restricted for about 23 s although the rails were read and nothing in the power telemetry failed.Change
A dark span longer than
APPLE_PMP_POWER_STALE_NS(3 s) still fails closed: the guard and the GPU treat the budget as stale, and the first sample after the span takes the policy's gap path. Every other error is unchanged. No new constants, parameters or board checks.Testing
MacBook Air 13" M4 (J713), macOS 27.2 firmware, AC, default command line.
Kernel built with both commits (the machine's production kernel plus this PR; the PR's own base lacks the 27.2 PMP record layout, so on this machine its sensors never read at all). Load: 8 x (all ten CPUs busy 12 s, then idle 3 s),
power_statusat 5 Hz, then the same with vkmark running on the GPU throughout:In the CPU run one budget sample fell inside a dark window: that sample was skipped (
power_statuskept its previous sample), the GPU driver loggedpower=8withsensor=Some(ENODATA), and 42 ms after the next real budget sample it was back atpower=0, with no release ladder. Under GPU load the dark windows lasted 5-40 ms and no budget sample landed in one. 0 AVC, 0 failed units, no firmware crash; 7 kernel errors, the same six this machine always logs (display, power-state, SEP) plus one SEP nonce timeout, none from this driver.Earlier A/B with change 1 alone as a hot-swapped module (same loop, before the GPU change):
Both stock faults followed the path above: levels 8 on err=-11, then one step per ~1.04 s, and the P cluster was free again ~23 s later. W=1 build and checkpatch --strict are clean on integration/aurora.
Still to run
Reviewer detail
apple_pmp_sample()caches-ENODATAonce the gated hold no longer applies (last reading >= 90 C, or dark longer than the hold).apple_pmp_power_work()mapped anycached_errorto-EAGAIN->apple_pmp_power_sample(..., error)->apple_pmp_power_fail(): levels {8,8,8},good = 0. Recovery:goodreaches 3 after three >= 400 ms samples, then one level per >= 1 s, choosing the largest weight-normalised level (GPU weight 4), so P/E come back last.apple_pmp_power_sample();power_errorstays 0, rails are published,p->last_nsdoes not advance, andapple_pmp_gpu_publish(..., 8, 0)invalidates the GPU publication for that sample (aschange_modedoes). If darkness lasts beyondAPPLE_PMP_POWER_STALE_NS,apple_pmp_guard()treats the budget as level 8 and the GPU publication is already invalid; the next real sample seesdt > 2 sand fails closed through the existing gap check.sensor=Some(ENODATA) ... backstop=8 limit=S1); commit 2 makes the budget side fail closed as well instead of relying on that.Assisted-by: LLM (Claude:Opus 5.5)