Skip to content

thermal: apple-pmp: keep the power budget over a dark sensor sample - #2699

Merged
RyanTheTide merged 2 commits into
integration/aurorafrom
j713-aop/pmp-dark-budget
Nov 22, 2026
Merged

RyanTheTide merged 2 commits into
integration/aurorafrom
j713-aop/pmp-dark-budget

Conversation

@RyanTheTide

@RyanTheTide RyanTheTide commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem

On the M4 MacBook Air (macOS 27.2 firmware), a load that starts right after the PMP die sensors go dark can run well below its clocks for about 20 s. The log shows sensor error -61; CPU minimum caps retained followed by power telemetry error -11, retaining minimum CPU/GPU ceilings; on a single-thread job the P cluster climbed from 1260 to 3096 MHz over more than 10 s.

The -61 is every die sensor being dark for one sample (their clusters powered down, for example right after a hot load stops). The thermal guard handles that sample by itself and is back in about 1.6 s. But if the 500 ms power-budget worker runs inside the dark window, it turns the cached thermal error into -EAGAIN, and apple_pmp_power_fail() sets the P, E and GPU budgets to full restriction. The budget then waits for three good samples and gives back one step per second, GPU first, so the P cluster stays restricted for about 23 s although the rails were read and nothing in the power telemetry failed.

Change

  1. While every sensor is only dark (-ENODATA), the zone is enabled and the thermal sample is fresh, the budget worker skips that budget sample instead of failing it. The thermal guard still holds both CPU clusters at the floor for the dark sample.
  2. For the same dark sample the GPU gets full protection: its budget publication is dropped (a zero publication selects level 8), and the next real budget sample republishes its level at once, so no release ladder runs.

A dark span longer than APPLE_PMP_POWER_STALE_NS (3 s) still fails closed: the guard and the GPU treat the budget as stale, and the first sample after the span takes the policy's gap path. Every other error is unchanged. No new constants, parameters or board checks.

Testing

MacBook Air 13" M4 (J713), macOS 27.2 firmware, AC, default command line.

Kernel built with both commits (the machine's production kernel plus this PR; the PR's own base lacks the 27.2 PMP record layout, so on this machine its sensors never read at all). Load: 8 x (all ten CPUs busy 12 s, then idle 3 s), power_status at 5 Hz, then the same with vkmark running on the GPU throughout:

run dark-sensor errors power telemetry errors budget fault P / GPU budget restricted
CPU load 67 0 0 s 0 s / 0 s
CPU + GPU load 16 0 0 s 0 s / 0 s

In the CPU run one budget sample fell inside a dark window: that sample was skipped (power_status kept its previous sample), the GPU driver logged power=8 with sensor=Some(ENODATA), and 42 ms after the next real budget sample it was back at power=0, with no release ladder. Under GPU load the dark windows lasted 5-40 ms and no budget sample landed in one. 0 AVC, 0 failed units, no firmware crash; 7 kernel errors, the same six this machine always logs (display, power-state, SEP) plus one SEP nonce timeout, none from this driver.

Earlier A/B with change 1 alone as a hot-swapped module (same loop, before the GPU change):

module dark-sensor errors power telemetry errors P budget restricted GPU budget restricted
stock 49 2 46.0 s 32.4 s
change 1 45 0 0 s 0 s

Both stock faults followed the path above: levels 8 on err=-11, then one step per ~1.04 s, and the P cluster was free again ~23 s later. W=1 build and checkpatch --strict are clean on integration/aurora.

Still to run

  • The driver binds only on T8132, and the J713 is our only T8132 in the lab.
Reviewer detail
  • Old path: apple_pmp_sample() caches -ENODATA once the gated hold no longer applies (last reading >= 90 C, or dark longer than the hold). apple_pmp_power_work() mapped any cached_error to -EAGAIN -> apple_pmp_power_sample(..., error) -> apple_pmp_power_fail(): levels {8,8,8}, good = 0. Recovery: good reaches 3 after three >= 400 ms samples, then one level per >= 1 s, choosing the largest weight-normalised level (GPU weight 4), so P/E come back last.
  • New path: a dark sample with a fresh cooling sample (<= 500 ms) and an enabled zone skips apple_pmp_power_sample(); power_error stays 0, rails are published, p->last_ns does not advance, and apple_pmp_gpu_publish(..., 8, 0) invalidates the GPU publication for that sample (as change_mode does). If darkness lasts beyond APPLE_PMP_POWER_STALE_NS, apple_pmp_guard() treats the budget as level 8 and the GPU publication is already invalid; the next real sample sees dt > 2 s and fails closed through the existing gap check.
  • The bug is timing-dependent: the dark window after a hot load lasted ~50-700 ms in these runs against the 500 ms worker period, so a single hot -> idle transition faults only sometimes; count over many cycles.
  • On this kernel the G16 driver also applies the shared cooling backstop during a dark sample (logged sensor=Some(ENODATA) ... backstop=8 limit=S1); commit 2 makes the budget side fail closed as well instead of relying on that.

Assisted-by: LLM (Claude:Opus 5.5)

When every PMP die sensor is dark past the gated hold, apple_pmp_sample()
caches -ENODATA and the thermal guard holds the CPU floor for that sample.
The next power-budget sample then turned the cached error into -EAGAIN,
and apple_pmp_power_fail() put the P, E and GPU budgets at full
restriction. That budget comes back only after three good samples and
then one step per second, larger weight first, so after a single dark
sample on a burst out of idle the P cluster spent about 20 s well below
its clocks (1260 -> 3096 MHz over more than 10 s on a 1T load).

The rails were read; nothing about the power telemetry failed. Skip that
budget sample instead of failing it while the sensors are only dark and
the thermal sample is fresh. The thermal guard still holds the floor for
the dark sample, and a dark span longer than the policy's 2 s gap, a
stale thermal sample, a disabled zone or any other error still fail the
budget closed as before.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Blockers: 1.

What I checked. The change adds no temperature or power constant: the dark branch reuses the existing 500 ms freshness test (drivers/thermal/apple-pmp-thermal.c:207), and nothing substitutes a value for the missing reading. On the CPU side a dark sample still fails safe. apple_pmp_sample() passes the -ENODATA to apple_pmp_cooling_sample(), which sets level 8 and hard (include/linux/soc/apple/pmp-cooling.h:94-104), and apple_pmp_guard() applies max(thermal, budget), so both clusters sit at the floor for the dark sample whatever the budget says. Because apple_pmp_power_sample() is not called (apple-pmp-thermal.c:222), the budget levels cannot be released during darkness. They stay where they were. If darkness persists, p->last_ns stops advancing, and after APPLE_PMP_POWER_STALE_NS the guard (:355), the GPU lease (pmp-power-policy.h:195-197) and hwmon (:250) all fall back to full restriction or -EAGAIN. The first sample after the dark span then takes the gap path (pmp-power-policy.h:90-92).

  1. drivers/thermal/apple-pmp-thermal.c:222-225: the GPU loses its fail-closed response for the dark window. Before this change, a dark sample put level[2] at 8 through apple_pmp_power_fail(). apple_pmp_guard() covers only the CPU clusters, and with get_temp() returning the error the governor does not act either. So that budget step was the only restriction the GPU got when the sensors went dark. Now the GPU keeps its previous budget level (possibly 0), re-published with the old p->last_ns, until the publication goes stale 3 s after the last real budget sample. That includes the case where the last valid reading was at or above APPLE_PMP_GATED_HOLD_MC, which pmp-cooling.h:24-27 itself treats as a failed sensor. The fix keeps the CPU result and costs the GPU no slow release ladder: in the dark branch, invalidate the GPU publication instead of re-publishing the old level, apple_pmp_gpu_publish(pmp_thermal.sensors, epoch, 8, 0), as change_mode already does at :535. A zero publication selects full protection (pmp-power-policy.h:230-231), and the next real budget sample republishes p->level[2] with a fresh timestamp, so the GPU returns to its budget level at once.

Nit: the comment at :204 and the body say a dark span longer than the 2 s gap "still fails closed". While the span lasts, the gap check never runs, because apple_pmp_power_sample() is skipped. What closes it is the 3 s APPLE_PMP_POWER_STALE_NS, then the gap at the first sample after. Please say that.

Not changed here, but relevant to the safety reading: the budget this protects is the existing provisional policy (pmp-power-policy.h:4-14, 50: fixed sustainable watts, burst energy and temperature target). It does not come from the machine, so this PR neither improves nor worsens that.

Body: drop the internal lane name in the first paragraph and the lab-store path in the details. Describe the run itself (machine, load, numbers).

HW row: J713, kernel built from this head (not an insmod'd module), default command line. Run the 8 x (all-core 12 s, idle 3 s) loop at 5 Hz power_status, and add a variant with a GPU load running across the dark samples, showing gpu_level held at 8 for the dark samples only and no release during them. Then 0 AVC, 0 failed units, 0 new kernel errors.

Assisted-by: LLM (Claude:Opus 5.5)

The previous change skips the budget sample while every sensor is dark,
but it still republished the GPU's last budget level with the old
sample time, so the GPU kept that level (possibly 0) until the
publication went stale 3 s later. The CPU clusters do not depend on the
budget there, because the thermal guard holds them at the floor for the
dark sample.

Publish nothing for the GPU in the dark branch instead: a zero
publication selects full protection, and the next real budget sample
republishes its level with a fresh time, so the GPU comes back at once
and no release ladder runs. Also say what closes a long dark span: the
3 s APPLE_PMP_POWER_STALE_NS, then the gap path at the first sample
after it.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Thanks, all addressed in 96ea511 (pushed on top, no force push):

  1. Blocker: for a dark sample the GPU publication is now dropped (apple_pmp_gpu_publish(..., 8, 0)), so the GPU takes full protection for that sample, and the next real budget sample republishes level[2] with a fresh time. On hardware: one budget sample landed in a dark window in the CPU run, and the GPU driver logged power=8 ... sensor=Some(ENODATA), then power=0 42 ms after the next real sample, with no ladder. (For the record, on this kernel the G16 driver also applies the shared cooling backstop for a dark sample, backstop=8 limit=S1, but the budget side now fails closed by itself.)
  2. Nit: the comment now says the 3 s APPLE_PMP_POWER_STALE_NS closes a long dark span, then the gap path at the first sample after it.
  3. Body: lane name and lab paths removed; the runs are described directly.
  4. HW row: kernel built with both commits, default command line, the 8 x (12 s all-core, 3 s idle) loop with power_status at 5 Hz, and the same with vkmark on the GPU. 0 power telemetry errors, 0 s of budget fault or restriction, 0 AVC, 0 failed units, and no kernel errors from this driver. Table in the body. One limit: the kernel is the J713's production kernel plus these two commits rather than the bare PR head, because integration/aurora alone lacks the 27.2 PMP record layout and its sensors never read on this machine. Under GPU load the dark windows were 5-40 ms, so no budget sample fell inside one there; the CPU run shows the GPU path.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Merge-train adversarial review (MERGETRAIN-LINUX2) at 96ea511: NO BLOCKERS.

Code read only, nothing built here. Before the merge it still needs Dj's verdict at this head; the missing rows are listed below.

Full review

#2699 thermal: apple-pmp: keep the power budget over a dark sensor sample @96ea511e

VERDICT: NO BLOCKERS

What I checked in drivers/thermal/apple-pmp-thermal.c:

  • The dark test at :209-211 needs all three of: enabled, cached_error == -ENODATA and a fresh
    cooling sample.
  • cached_error == -ENODATA comes only from apple_pmp_sample_all() with no readable sensor and no
    other error (:303-327, :462). guard_ret is a freq_qos errno and never -ENODATA. The probe seeds
    -EAGAIN (:802), so a first-sample-dark boot still fails closed.
  • On that same sample the thermal side really does hold the CPUs: apple_pmp_cooling_sample() with
    an error sets level 8 / hard / reason=sensor (include/linux/soc/apple/pmp-cooling.h:87-107),
    apple_pmp_guard(8) runs (:352-381), and release needs 10 good reads spaced ≥20 ms.
  • cooling.last_ns advances on error samples too, so the freshness test (≤500 ms) holds while the
    poller runs (5/100 ms cadence, :42-44, :529-538).
  • A long dark span still fails closed. p->last_ns does not advance, so apple_pmp_guard() forces
    power_level 8 after APPLE_PMP_POWER_STALE_NS (:365-367), and the next real sample takes the gap
    path.
  • Integrating energy across skipped samples uses the next sample's power over the whole dt. The
    clusters are powered down during the dark span, so this over-counts, which is the conservative
    direction.
  • No new parameters, constants or board gates. The 500 ms literal was there before.

Nits:

  1. Bisectability. Commit 1a925b2 alone republishes the GPU's last level (possibly 0) with the old
    p->last_ns for every dark sample (:222-224 at that commit). The GPU budget then stays released
    for up to 3 s of darkness, which commit 2 fixes. Squash the two, or swap their order, so that no
    tree has the GPU released over a dark span.
  2. GPU performance. apple_pmp_gpu_publish(..., 8, 0) (:231) sets the GPU to full protection for
    every dark sample. The PR shows the dark windows also happen under GPU load (5-40 ms), so a
    GPU-bound load with idle CPUs takes a level-8 dip of up to one budget period (about 500 ms) each
    time a budget tick lands in a window. That is far better than base (a 23 s ladder), but it is not
    free. The comment at :226-229, "Nothing else restricts the GPU's budget while the sensors are
    dark", also contradicts the PR's own note that the G16 driver applies the cooling backstop on a
    dark sample.
  3. 500000000ULL now appears twice in the same condition (:210, :213). Give it a name.

Missing rows:

  • A GPU-bound load with mostly idle CPUs (a game or glmark2 with the CPU at rest) on J713: frame-time
    dips correlated with power=8 sensor=ENODATA (nit 2).
  • A hot-dark case: dark with valid_mc ≥ the gated-hold limit (90 C). That is the only case where
    the gated hold does not cover the sample and the budget-skip decision rests entirely on the
    cooling guard. The rows shown do not say whether any dark sample started from ≥90 C.
  • The driver binds only on T8132 and the J713 is the only T8132 in the lab, so this cannot be tested
    on another Mac. Other T8132 boards are unproven. The G16 GPU consumer is shared with any future
    binder.

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial review at live head 96ea511e2cd79093942468a67120b0782bdc5d67 against the pre-PR checklist: 0 code/policy/attribution/CI blockers, and §5 hardware verification is complete on J713 (the sole T8132 machine in the lab).

Checked & clean

  • §1–§2 No model gating / no hacks: No chip/board/firmware gates, no new module/cmdline parameters, no magic sleeps or retries; reuses the existing 500 ms freshness check.
  • §3 Safety & deep code correctness (drivers/thermal/apple-pmp-thermal.c):
    • The dark-skip predicate (!ret && pmp_thermal.enabled && pmp_thermal.cached_error == -ENODATA && now - pmp_thermal.cooling.last_ns <= 500000000ULL) is appropriately narrow: SMC rail-read errors and the all-zero firmware-reset check (!mw[0] && !mw[1] → ret = -ENODATA at line 192) bypass dark because !ret is false, and cached_error == -ENODATA originates solely from apple_pmp_sample_all() when every die sensor is dark with no other error.
    • CPU clusters remain fail-closed via apple_pmp_cooling_sample() (level = 8, hard = true) → apple_pmp_guard(8).
    • GPU remains fail-closed via apple_pmp_gpu_publish(pmp_thermal.sensors, epoch, 8, 0) (zero publication selects level 8 in apple_pmp_gpu_lease_level()), and newly attached GPU epochs still initialize p->level[2] = 8 before if (dark), then recover immediately on the next real sample without the 23 s release ladder.
    • Prolonged darkness still fails closed: p->last_ns does not advance while dark is true, so apple_pmp_guard() and hwmon expire after APPLE_PMP_POWER_STALE_NS (3 s) and the first real sample after a >2 s span takes apple_pmp_power_sample()'s gap path (apple_pmp_power_fail()).
    • Interaction with #2730: Complementary (#2699 handles emitted -ENODATA samples such as hot-to-idle transitions above 90 C where the gated hold does not apply); #2730 touches adjacent lines in apple_pmp_power_work() and can rebase cleanly once #2533 lands.
  • §4 Attribution: Both commits (1a925b273afd, 96ea511e2cd7) carry human author/committer (Ryan Murray <ryan@aurorasilicon.org>) and human Signed-off-by: only (zero AI trailers); PR body includes Assisted-by:.
  • §5 Hardware & CI evidence: host-tests CI is green on 96ea511e (including pmp-power-tests). J713 hardware run (macOS 27.2, default cmdline, both commits compiled in) exercises the exact dark-skip path including GPU power=8 (sensor=Some(ENODATA)) → power=0 42 ms after the next real sample, with A/B telemetry-fault counts and regression summary (0 AVC, 0 failed units, 0 new driver errors).
  • §6 Naming: Clean.

Non-blocking nits

  • Bisectability: Commit 1a925b27 alone briefly leaves the GPU budget un-invalidated for up to 3 s over a dark sample until 96ea511e fixes it; squashing on merge keeps every commit fail-closed.
  • 500000000ULL is now repeated on lines 210, 213, and 486 — worth naming in a future cleanup.

Assisted-by: LLM (Gemini 4 Neon)

@RyanTheTide
RyanTheTide merged commit bb2ccbe into integration/aurora Nov 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants