Skip to content

J615: expose die temperatures and preserve T8122 GPU thermal fallback - #410

Open
ericjuta wants to merge 3 commits into
omacom:integration/aurorafrom
ericjuta:fix/j615-thermal-safe-cap
Open

ericjuta wants to merge 3 commits into
omacom:integration/aurorafrom
ericjuta:fix/j615-thermal-safe-cap

Conversation

@ericjuta

@ericjuta ericjuta commented Nov 7, 2026

Copy link
Copy Markdown

Summary

Separate the opt-in T8122 GPU performance ceiling from its boot/sensor-failure cap, and expose the complete J615 SoC die sensor set.

Three commits:

  1. hwmon: add J615's six GPU-family and six CPU die keys; require all twelve to be discovered before registering the zone.
  2. drm/asahi: preserve the state-2 boot/failure cap even when asahi.t8122_pstate_cap=5 is requested. Off/hold cannot boost above the safe cap; lower requested ceilings still win. Add regression coverage for missing, failed, stale, stuck and implausible readings, pacing, hot/critical temperatures and recovery.
  3. docs/tools: explain the guarded ceiling, qualification procedure and older-kernel warning; update one-shot help.

The default ceiling remains 2. Non-T8122 cap selection is unchanged. This also conservatively bounds T8122 boards without usable sensor feedback.

Validation

  • rustc --edition=2021 --test tools/asahi/m3-runtime-tests.rs -o /tmp/j615-thermal-runtime-tests and execution: 24 passed on the rebased branch.
  • bash -n tools/aurora-sep/air-gpu/air-gpu-oneshot.sh: passed.
  • Existing running J615 custom kernel 7.2.11-dcpexp-thermal+, configured with asahi.t8122_pstate_cap=5 asahi.m3_thermal=on, logs runtime cap 5 -> 4 -> 3 -> 2 at 80.3–80.5 C, then recovery 2 -> 3 -> 4 -> 5 at 66.5–67.7 C. This is observed runtime behavior, not proof that the freshly rebased branch is the running binary.
  • The newly rebased branch has not had a full kernel build/boot test. Sensor-failure behavior is covered by host tests, not injected hardware faults. No claim of calibrated temperatures or hardware safety certification.

Scope

Based on current integration/aurora (8eddf5a). Excludes existing display/brightness PRs #354, #356 and #357, local boot deployment scripts, and USB power workarounds. The intermittent USB-C -28/revalidation failure is unresolved and is not claimed fixed here.

@ericjuta

ericjuta commented Nov 8, 2026

Copy link
Copy Markdown
Author

Additional J615 hardware observations (2026-11-08): the currently running combined kernel is 7.2.11-dcpexp-usb-recovery-20261108, with asahi.t8122_pstate_cap=5 and asahi.m3_thermal=on. All nine locally modified thermal-related source/docs files are byte-for-byte identical between the earlier thermal working tree and this USB-recovery build tree. The installed Image matches the build Image by SHA-256; installed asahi/macsmc-hwmon module build IDs match the build artifacts. This is provenance for the combined local build, not a claim that the rebased PR branch was built or booted unchanged.

The current boot registered all 12 listed J615 die keys. Runtime logs show cap 2 -> 3 -> 4 -> 5 with cool readings, and 5 -> 4 -> 3 -> 2 with readings of 80.5–83.8 C; the firmware summary reports peak state 5. The user reports stable unplug/replug and lid-open/closed use on this combined kernel. No sensor-failure injection or sustained thermal qualification was performed in this audit.

Please keep the sensor/fallback correctness work separate from raising default performance limits. Review coordination with overlapping #135 is still needed, particularly explicit thermal-Off semantics. The all-keys requirement concerns the selected sensor list; it does not establish calibrated coverage of the hottest physical location, nor does it prevent an explicit partial sensor-list override. These observations do not establish hardware safety or qualify #317.

Signed-off-by: Eric Juta <ericjohnjuta@gmail.com>
Signed-off-by: Eric Juta <ericjohnjuta@gmail.com>
Signed-off-by: Eric Juta <ericjohnjuta@gmail.com>
@ericjuta

Copy link
Copy Markdown
Author

Rebased onto integration/aurora 39d5f88ab4e5 (the branch was 32 commits behind) and force-pushed; new head 6fc58762cd8f. No conflicts. All three commits now carry a Signed-off-by, which they lacked.

One real fix came out of this: the branch edits a help line in tools/aurora-sep/air-gpu/air-gpu-oneshot.sh but did not update that script's SHA-256 pin in install-aurora-sep.sh, so 11 cases of tools/aurora-sep/tests/test_air_gpu.py failed ("does not match its published checksum"). The old head had this too; full CI never ran on it. The pin is now updated in the docs commit and test_air_gpu.py passes.

Tested at this head on an arm64 Arch host (clang 23.1.5), not on the CI image:

  • host-tests.py --group aurora-sep --group tools: 163 suites, 152 passed, 6 known failures, 5 failed. None of the 5 comes from this PR:
    • tools/asahi/g15-pb-growth-tests.py and g15-stats-drain-tests.py need rustup, which this host does not have.
    • tools/testing/selftests/drivers/gpu/apple-dcp/run.sh (6 cases) and tools/aurora-sep/recipes/mesa-m3/tests/test-session-env.sh (sh and bash) fail identically on plain 39d5f88ab4e5 here, so I take them as host differences. CI is the judge for those.
  • tools/asahi/m3-runtime-tests.rs and m3-batching-tests.rs pass.
  • check-commits.py: all signed off; one checkpatch warning on the docs commit.
  • compile-check.py builds the changed C object (macsmc-hwmon); it does not compile the Rust sources.

Not tested: this exact head has not been booted. The J615 hardware results above are from a combined local kernel that carries these changes with others.

@ericjuta
ericjuta force-pushed the fix/j615-thermal-safe-cap branch from da758c5 to 6fc5876 Compare November 16, 2026 10:41
@ericjuta

Copy link
Copy Markdown
Author

J615 finding next to this PR: the board has no CPU thermal zones

This PR's governor limits the GPU only. Separately, the J615 device tree (t8122-j615) has no thermal-zones node, so nothing slows the CPU on temperature. The die sensors this PR exposes made that visible.

Without zones (combined local kernel, asahi.m3_thermal=on, cap 5):

  • 8 busy CPU threads: 105.5 C after 150 s, still rising.
  • -j8 kernel build: 110-114.6 C held for about 22 minutes.
  • The GPU cap dropped 5 -> 2 as designed; CPU cooling states stayed 0.

With the J613 block from 30db282 ("t8122-j613: Add CPU thermal zones") carried to the J615 unchanged (six zones, 58/95 C passive, 110 C critical, 6 W sustainable-power, E cluster capped at state 3), applied as a local device tree overlay:

  • 8 threads, 170 s: peak 87.7 C; P cluster 3.1-3.6 GHz (from 4.06).
  • -j8 kernel build, 20 min: two-minute means 89-92.6 C, peak 97.2 C, 6 of 545 samples at or above 95 C, no upward trend after minute 6; P cluster mean 2.93 GHz; both clusters unthrottled within 25 s.
  • 8 W was also tried: peak 96.3 C on the 8-thread test with no P-cluster frequency gain, so I went back to 6 W.

Limits: one machine, local overlay on the packaged DTB, combined local kernels (7.2.11-dcpexp-combined-20261111 and 7.2.11-dcpext-pr356-combined-20261116), not this PR's head. CPU + GPU load together and the 110 C critical power-off are not tested.

This does not change anything in this PR. It is a reason not to read the GPU governor as full thermal protection on the J615. I can send the zones as a separate t8122-j615 DTS patch if that is wanted.

@freber

freber commented Nov 17, 2026

Copy link
Copy Markdown

Claude Code wrote this up from my tests.

J615 (15-inch M3 Air) results for this PR's code. My kernel has #410's changed lines exactly as in 6fc58762cd8f. They sit on bed544b here and on 39d5f88ab4e5 in the PR, so this is not a boot of the exact head. The kernel also has #491, the J615 display and memory floor patches and a T8122 series that lets the GPU use states 6 to 8 under the m3_thermal limit. It's my daily kernel since 2026-11-17 (three boots so far).

  • macsmc_soc_die lists all 12 keys (Tg0D Tg0P Tg0X Tg0b Tg0j Tg0v Te05 Te0P Tp05 Tp0L Tp0b Tp0r). On each boot the first die reading came 8.2 to 9.0 s after the driver's "no reading yet" line at 2.7 s. The runtime cap stayed at 2 until then. There were no fallback lines after that.
  • Which key runs hottest depends on the load, so the governor needs all 12. In 600 s of GPU-only load at state 2 it was a P-core key (Tp0b 86.4 C, Tg0P 80.5 C). With a higher limit the GPU ran at states 2 to 5 and the GPU keys led (Tg0P 93.8 C with the limit at 90 C, 95.9 C at 92 C). Under a combined CPU and GPU load it was the E-cluster key Te0P.
  • The critical step goes to state 2 as it should. With the default 80 C limit, a combined load started all 8 cores at once and the die jumped past the 88 C critical point between two readings. The cap went from 8 to 2 in one step (logged as critical at 95.9 C). It stayed at 2 for the whole 300 s while arm64: dts: apple: t8122-j615: Add CPU thermal zones #491's zones held the die at 95 C (max 95.8, mean 94.9).
  • With the default 80 C limit, 600 s of GPU-only load stepped down to state 2 within 5 s and stayed there. The hottest key ended at 86.4 C.

The session summaries found no asahi errors, faults or timeouts in the kernel log. The throughput and tuning numbers belong to the state 6 to 8 series (#317). I'll post them with the PR for it.

@freber

freber commented Nov 18, 2026

Copy link
Copy Markdown

Claude Code wrote this up from my tests.

I ported this PR and #491 to 7.3 in aurora-silicon/linux#2203, together with #500, since 7.3 ships from aurora-silicon. Your four commits keep you as author and keep your Signed-off-by. My Signed-off-by follows yours.

On 7.3 the die keys already come from apple,soc-die-keys in t8122.dtsi, so the port drops the J615 key table. In the port the hwmon commit keeps the rule that the J615 registers macsmc_soc_die only with all 12 keys present. If you'd rather open the 7.3 version of #410 and #491 yourself, I can drop them from #2203 and stack on yours.

@ericjuta

Copy link
Copy Markdown
Author

@freber keep them in aurora-silicon/linux#2203, thanks. Authorship and sign-off as you have them is fine, and I'd rather not split the 7.3 review across two PRs. Dropping the J615 key table in favour of apple,soc-die-keys makes sense. Ping me if anything needs changing on my side.

freber added a commit to freber/linux that referenced this pull request Nov 22, 2026
… range

Two gaps in the host tests that the rest of this series leans on.

The nine unit tests in m3/t8122_knobs.rs have never run. No file in
tools/asahi includes the module. It does not compile on its own either,
because it takes its default InitData version from m3_firmware, which
needs the kernel crate. pstate_cap_accepts_one_to_five_only, the test
that pins the range of asahi.t8122_pstate_cap, is one of them. Include
the module in tools/asahi/m3-runtime-tests.rs, the suite the kernel CI
already runs, with a stand-in for that one constant. The stand-in's
value is checked by defaults_are_the_plan_baseline, so it cannot drift
from m3_firmware without a failing test.

Policy in m3_thermal_policy.rs moves the runtime cap between the safe
cap and the ceiling. The cap_tests from aurora-silicon#410 drive it with a ceiling of
5 over a safe cap of 2. Nothing covers ceilings 6 to 8, or the check
that finds a firmware ignoring the cap. Add tests for the rules as
they are, for ceilings 6 to 8 above a safe cap of 5 and for ceilings 1
to 5:

- the cap starts at the safe cap and rises one state per 2 s on cool
  readings, never above the ceiling;
- between cool and hot it holds;
- hot lowers one state per fresh reading, at most every 500 ms, never
  below the safe cap;
- critical, no reading, a failed, stale, implausible (including the
  -2.849 C an unpowered SMC GPU key reads) or stuck reading returns to
  the safe cap at once, with the previous cap tolerated for 1 s;
- after a fallback the cap climbs again at the same pace;
- hold never leaves the safe cap;
- a ceiling equal to the safe cap never moves;
- a firmware that stays above a lowered cap is found after 20 checks
  spanning at least 3 s.

Three mutations of Policy (critical keeps the cap, the 3 s span dropped,
the raise not bounded by the ceiling) each fail one to three of the new
tests.

No driver code changes.

Link: omacom/linux-aurora#317
Link: omacom/linux-aurora#410
Signed-off-by: Fredrik Silfver <1684291+freber@users.noreply.github.com>
freber added a commit to freber/linux that referenced this pull request Nov 22, 2026
The thermal limit moves the runtime cap only inside
Config::check_pstate. That runs after a job only in two conditions,
which a ceiling above the safe cap makes matter.

First, the job thread runs the check only when asahi.m3_retire_mmio has
bit 2 (the performance-state register read). The parameter is
root-writable at runtime, so writing 0 to 3 stops both the thermal limit
and the check that the firmware stays within the runtime cap. Once the
ceiling is above the safe cap, the runtime cap is the only thing that
keeps the firmware below the ceiling. A frozen governor would leave it
at whatever state it last allowed. Add Policy::above_safe()
(host-testable), Governor::enforces() (the limit is on or holding and
the ceiling is above the safe cap) and Config::pstate_check_required().
Run the after-job check when bit 2 is set or the check is required. Bit
2 is the T8122 default, so this forces only what runs by default. The
first retirement of a module load at which the check runs although bit 2
is clear logs one line, so a write that leaves bit 2 out is not silently
overridden.

Second, the check runs only at a retirement with no later batch in
flight. The T8122 default pipeline depth is 2. A render batch of the
same VM is published behind the running one while its passes fit the
4000 us budget. So one client that keeps submitting short passes, with
no other client rendering (a still or blanked screen, offscreen work),
can keep the check from running at all, at 90 C too. While the ceiling
is above the safe cap, pipeline_slot() now refuses a slot behind batches
in flight once the last check is 500 ms or more old
(Policy::overlap_allowed, CHECK_EVERY_NS), so the pipeline drains and
the existing checked retirement runs. The sensor is read every 500 ms
while busy, so more frequent checks would see no newer reading. The
longest time between checks is then 500 ms plus the run time of the
batches in flight. Every firmware-visible action stays where it runs
today, after the last batch has retired and the pipes are idle. The
other way, moving the cap on a pipelined retirement, would be a Globals
store with a later batch running on the GPU, which has never been done
on this firmware.

With aurora-silicon#410 the T8122 safe cap is 2, so both already apply to boots that
set asahi.t8122_pstate_cap to 3 to 5 with the thermal limit on or
holding. Those boots now drain the pipeline once per 500 ms under
sustained pipelined load. With the ceiling at the safe cap nothing
changes, since overlap_allowed() returns true before reading the clock.

New tests in m3_thermal_policy.rs simulate a client that keeps the
pipeline full (depth 1 to 4, batches of 0.1 to 50 ms). Without the bound
it is checked once, at boot, so 90 C leaves the cap at 8. With it the
longest gap between checks is within 500 ms plus the batches in flight,
the cap falls from 8 to the safe cap within that time of a critical
reading, a firmware that ignores the cap in hold mode is found. With the
ceiling at the safe cap, the publication order is identical to today's.
Three mutations of the bound each fail two to four of them.

Link: omacom/linux-aurora#317
Link: omacom/linux-aurora#410
Signed-off-by: Fredrik Silfver <1684291+freber@users.noreply.github.com>
freber added a commit to freber/linux that referenced this pull request Nov 22, 2026
… limit

asahi.t8122_pstate_cap refuses anything above 5 (PSTATE_CAP_MAX in
t8122_knobs.rs). The stated reason is that J613 and J615 had no SoC die
temperature zone, so there was no temperature feedback above the
driver's no-feedback limit. Both boards have one now (macsmc_soc_die).
The rest of the path already exists. The firmware always gets the whole
eight-state table. With the thermal limit on, PstatePolicy::new already
makes the cap the ceiling, so only the knob range keeps the governor
below 6.

Accept 6 to 8, as the thermal limit's ceiling only. With aurora-silicon#410 below
this, the runtime cap of every T8122 boot starts at state 2 and falls
back to it, whatever the ceiling.

- asahi.m3_thermal=on (the default). The ceiling is the cap. The runtime
  cap starts at 2 and rises one state per 2 s only on usable, cool
  readings of the SoC die zone. No zone, or a failed, stale,
  implausible, stuck or critical reading, puts it back to 2 at once.
  These are the governor's existing rules.
- asahi.m3_thermal=hold. 6 to 8 are lowered to 5, with one boot line.
  The firmware gets 5 and the runtime cap stays at 2.
- asahi.m3_thermal=off. 6 to 8 are lowered to 5, with one boot line.
  aurora-silicon#410's fixed cap without the limit then gives the firmware 2. These
  values were refused before in every mode (ENODEV, no GPU for that
  boot). A cap left on the command line from a ranging boot while the
  thermal limit is switched off should not cost the GPU.

Values of 0, 9 and up are refused as before.

Whether the zone is usable can only be decided at runtime. On a J615 the
GPU firmware starts about 1.6 s into boot from the initramfs. The
macsmc_soc_die zone appears at about 7.6 s, when macsmc_hwmon registers
it. A probe-time zone check would refuse every boot, so the zone
condition is the governor's existing rule that only a usable reading
raises the runtime cap. The line the governor logs for a usable reading
now ends in "the runtime cap may rise above 2 up to N (MHz)" while the
ceiling is above the safe cap, so the boot log shows when the higher
states became possible. It is one line per change of the reading's
source, as before.

pstate_cap_use() in t8122_knobs.rs states the rules above in plain code
that the host tests drive. PstatePolicy::new gets the lowered ceiling
from admitted_cap(). Contents::select refuses the start if what
PstatePolicy::new computed is ever more than pstate_cap_use() allows, so
the two cannot drift apart silently. A host test checks that they are
equal for every cap and mode. Const assertions tie the no-feedback limit
(5) and the fallback (2) to the two constants of m3_thermal_policy.rs
that PstatePolicy::new uses. The boot lines name the case (lowered,
ranging or held) and the knob's value, which asahi.m3_max_pstate can
still lower later. When that parameter is given they say so and point to
the "firmware highest state" line, which has the result.
tools/aurora-sep/M3-GPU.md gets a paragraph on caps 6 to 8.

Tested on a J615 (15-inch M3 MacBook Air) with this series and aurora-silicon#410 on
bed544b (the 7.2 line), asahi.t8122_pstate_cap=8 and the thermal limit
at its defaults. On a cool die the runtime cap rose from 2 to 8 within
10 s of boot. Under GPU load the die passed 80 C within about 2 s at
state 8 and the cap fell back to 2 one state per 500 ms. A 600 s GPU
load and a 300 s combined CPU and GPU load ran without asahi errors,
faults or timeouts.

Link: omacom/linux-aurora#317
Link: omacom/linux-aurora#410
Signed-off-by: Fredrik Silfver <1684291+freber@users.noreply.github.com>
freber added a commit to freber/linux that referenced this pull request Nov 22, 2026
The M3 thermal limit lowers the runtime cap at asahi.m3_thermal_hot
(default 80 C, at most 85 C) and raises it again only 10 C below that.
On a J615 under a sustained GPU load the die then settles between the
two (83 to 84 C with hot at 85). The cap stays at the safe state 2 for
the whole load, since nothing ever reaches the cool point again.

Let both be set further for boards and users that want it. The defaults
do not change.

- asahi.m3_thermal_hot goes up to 92 C. Critical stays 8 C above it, so
  at most 100 C, below the 110 C critical trip of the J613 and J615 CPU
  thermal zones.
- asahi.m3_thermal_cool_gap (default 10 C, 3 to 10 C) sets how far below
  hot the die must be before the cap rises again.
- Both are writable at runtime in /sys/module/asahi/parameters. The
  governor reads them at each update, applies the clamped values from
  the next reading with the usual pacing and logs the new thresholds.

The bounds and the clamping move into m3_thermal_policy so the host
tests cover them. New tests cover the clamping, the default gap, a
narrower gap, a higher hot and thresholds changed at runtime.

Tested on a J615 with asahi.t8122_pstate_cap=8, stepping the two values
at runtime. A 600 s GPU load (gpubench w10, depth 2) ran at 80.0 fps
with the defaults, 85.8 fps with hot 88 and gap 3, 88.1 fps with hot 90
and gap 3 and 95.2 fps with hot 92 and gap 3. A 300 s combined CPU and
GPU load stayed at the safe state 2 at every setting. The CPU thermal
zones kept the hottest die sensor at 92 to 94 C on average, with peaks
of 93.9 to 97.6 C.

Link: omacom/linux-aurora#317
Link: omacom/linux-aurora#410
Signed-off-by: Fredrik Silfver <1684291+freber@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants