Skip to content

arm64: dts: apple: t6050: Describe the CPU caches - #2654

Merged
DjDeveloperr merged 1 commit into
integration/aurorafrom
djdev/t6050-l2-cache-nodes
Nov 21, 2026
Merged

DjDeveloperr merged 1 commit into
integration/aurorafrom
djdev/t6050-l2-cache-nodes

Conversation

@DjDeveloperr

@DjDeveloperr DjDeveloperr commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

The T6050 (M5 Pro/Max/Ultra) CPU nodes have no next-level-cache and no cache sizes, and the DT has no L2 nodes. Without a cache description, arm64 cacheinfo falls back to CLIDR (use_arch_info):

  • sysfs reports every level above L1 as shared by all 18 CPUs, with no size;
  • no CPU has a valid last-level cache, so llc_sibling stays the CPU itself and cpu_coregroup_mask() falls back to the package;
  • as a result, the scheduler's MC domain spans all three clusters.

This describes the caches the way the ADT and macOS do:

Level M tier (apple,sawtooth, 2 clusters x 6) P tier (apple,everest, 1 cluster x 6) Source
L1I 128 KiB 192 KiB macOS hw.perflevel1/0.l1icachesize
L1D 64 KiB 128 KiB macOS hw.perflevel1/0.l1dcachesize
L2 (unified, one per cluster) 8 MiB 16 MiB ADT /cpus/cpuN l2-cache-size (J714c/s, J716c/s, J775c/d, J873s); macOS cpusperl2 = 6

The K114/K116 boards use the same CPU list. The system level cache sits behind the memory controllers and is shared with every agent. It is not part of the CPUs' cache hierarchy, so it is not described here.

The L2 nodes use compatible = "cache", cache-level = <2>, cache-unified and cache-size, at the root as in t604x. There is one node per cluster, 6 on the Ultra.

Build check: all 11 T6050 board DTBs compile with no new dtc warnings. With the L2 phandles held fixed, every decompiled DTB differs from the base only by these additions: 3 L2 nodes (6 on the Ultra) plus next-level-cache, i-cache-size and d-cache-size on every CPU.

Tested on J714s (M5 Pro, 18 CPUs)

Kernel 7.3.0 (integration/aurora with the J714s sleep series), the same kernel, initrd and boot loader on both boots. The only difference is the DTB: the shipped J714s DTB, then the same DTB with exactly these cache nodes and properties added.

Cacheinfo (sysfs cpuN/cache/index*, lscpu -C):

Before After
M core L1D / L1I no size 64K / 128K
P core L1D / L1I no size 128K / 192K
M cluster L2 no size, shared 0-17 8M, shared 0-5 and 6-11
P cluster L2 no size, shared 0-17 (CLIDR also gives the P cores a level 3, shared 12-17) 16M, shared 12-17

These are the values macOS reports on the same machine (hw.perflevelN.*, hw.cacheconfig 18 1 6 0).

The P cores' CLIDR reports three levels. The latency curve shows a private ~1 MiB middle level (2.3-2.8 ns per load, against 7.7-8.8 ns in the 16 MiB cluster cache). macOS reports the cluster cache as the L2 and does not expose that level. This DT follows macOS and gives no size for it.

Regression run (both boots):

  • Failed units: only the one that already fails on this rootfs.
  • Kernel errors: the same set.
  • SELinux denials: 0.

Scheduler-sensitive runs, unpinned, before -> after:

Run Before After
Pipe ping-pong, 1 pair (µs/roundtrip, median of 3) 10.2 11.3
Pipe ping-pong, 9 pairs (roundtrips/s) 751k 694k
perf bench sched messaging (s) 0.56 0.56
perf bench sched messaging -p (s) 0.30 0.38
Thread/socket messaging, 400 threads (s) 0.16 0.17
STREAM triad, 12 threads (GB/s) 220 234

The pipe-based wakeup runs are 8-27% slower afterwards: perf bench sched messaging -p, 0.29-0.32 s before against 0.34-0.39 s after, does not overlap. This is the cost of the correct topology. The scheduler's idle-CPU search for a wakeup now stays inside the waker's 6-core cluster instead of spanning all 18 CPUs, and a cross-cluster wakeup goes through an IPI. Toggling sched/llc_balancing/enabled (CONFIG_SCHED_CACHE, active once there is more than one LLC) on the after boot shows no consistent difference, so it is not the cause. The pinned latency and bandwidth curves are unchanged. Each side is one boot with n=3.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor

This overlaps with #2574, which describes the T6050 caches in the same three files. The L2 values agree: 8 MiB per M-tier cluster and 16 MiB for the P cluster, unified and per cluster. The two differ in two places:

Only one of the two should land.

Assisted-by: LLM (Claude:Opus 5.5)

@satchlj

satchlj commented Nov 21, 2026

Copy link
Copy Markdown
Member

Gemini 4 Neon on @satchlj's behalf:

One critical correction on the comparison between #2654 and #2574 before either merges:

#2574's cache-level = <3> node is NOT the system-level cache (SLC). On T6050, CLIDR_EL1 itself reports 3 cache levels (4 leaves: L1D, L1I, private L2, shared L3) on P-cores (apple,everest, cpu12..17) and 2 cache levels (3 leaves: L1D, L1I, shared L2) on M-cores (apple,sawtooth, cpu0..11). On J775c (M5 Max), pointer-chase latency curves match this (0.7 ns up to 128 KiB, 2.8 ns up to 512 KiB private L2, 5–11 ns up to 8–16 MiB cluster-shared L3 on a P-core, vs 1.0 ns up to 64 KiB and 4.7 ns up to 3–8 MiB on an M-core).

Because arch/arm64/kernel/cacheinfo.c:init_cache_level() takes max(CLIDR_EL1, fw_level), P-cores always have cache_leaves(cpu) == 4 (3 levels). If the DT only walks 2 levels (cpu12 -> l2_cache_pc -> NULL), as #2654 does:

  1. drivers/base/cacheinfo.c:cache_setup_of_node(12) breaks after index == 3 != cache_leaves(12) == 4 and returns -ENOENT.
  2. cache_setup_properties(12) hits if (ret && use_arch_cache_info()) { use_arch_info = true; ret = 0; }.
  3. use_arch_info is a single file-static global in drivers/base/cacheinfo.c. Once flipped on CPU 12, cache_leaves_are_shared() falls back to (level != 1) for all CPUs in the system, discarding the DT cache sharing topology for both M-clusters and the P-cluster and collapsing every non-L1 cache back to system-wide shared.

Live /sys/devices/system/cpu/cpu{0,6,12}/cache/ on J775c (M5 Max) with #2574 booted confirms all 3 levels populate cleanly with per-cluster sharing:

  • cpu0 (M0): index0 L1D (shared=0), index1 L1I (shared=0), index2 L2 Unified 8192K (shared=0-5)
  • cpu6 (M1): index0 L1D (shared=6), index1 L1I (shared=6), index2 L2 Unified 8192K (shared=6-11)
  • cpu12 (P): index0 L1D (shared=12), index1 L1I (shared=12), index2 L2 Unified private (shared=12), index3 L3 Unified 16384K (shared=12-17)

We are folding the verified L1 sizes from #2654 (128 KiB I / 64 KiB D on M-tier, 192 KiB I / 128 KiB D on P-tier) into #2574 and verifying them in J775c sysfs so #2574 covers both L1 sizes and the 3-level P-core hierarchy without tripping use_arch_info.

@RyanTheTide

Copy link
Copy Markdown
Contributor

Thanks, my comparison was wrong there: #2574's level-3 node is the P cluster's shared L3 that CLIDR reports, not the system level cache. The use_arch_info point settles it. It's a single global in drivers/base/cacheinfo.c, so a two-level DT walk on the P cores (where cache_leaves() is 4) would drop the DT sharing for every CPU. The J775c sysfs listing shows #2574's hierarchy populating per cluster. Folding #2654's L1 sizes into #2574 and keeping that one sounds right.

Assisted-by: LLM (Claude:Opus 5.5)

The T6050 CPU nodes have no next-level-cache and no cache sizes, and
there are no L2 nodes. Without a cache description in the DT, arm64
cacheinfo falls back to CLIDR (use_arch_info): sysfs reports every
level above L1 as shared by all 18 CPUs with no size, no CPU has a
valid last-level cache, so llc_sibling stays the CPU itself and
cpu_coregroup_mask() falls back to the package. The scheduler's MC
domain then spans all three clusters.

Describe the caches as the ADT and macOS do. Each cluster has its own
unified L2, shared by its six cores: the ADT's /cpus/cpuN l2-cache-size
is 8 MiB on the M-tier (apple,sawtooth) cores and 16 MiB on the P-tier
(apple,everest) cores in the J714c/s, J716c/s, J775c/d and J873s ADTs
(the K114/K116 boards use the same CPU list), and macOS reports
hw.perflevelN.cpusperl2 = 6 on both tiers. The L1 sizes are macOS's
hw.perflevelN.l1icachesize and l1dcachesize: 128 KiB I / 64 KiB D on
the M tier, 192 KiB I / 128 KiB D on the P tier. The system level cache
sits behind the memory controllers and is shared with every agent; it
is not part of the CPUs' cache hierarchy and is not described here.

On every T6050 board the compiled DTB changes only by these additions
(3 L2 nodes, 6 on the Ultra, and next-level-cache plus the L1 sizes on
each CPU).

Signed-off-by: Diljit Singh <hey@djdev.me>
@DjDeveloperr
DjDeveloperr force-pushed the djdev/t6050-l2-cache-nodes branch from 7d3a1da to 1bd5f40 Compare November 21, 2026 05:01
@DjDeveloperr
DjDeveloperr marked this pull request as ready for review November 21, 2026 12:24
@DjDeveloperr
DjDeveloperr merged commit a84637d into integration/aurora Nov 21, 2026
2 checks passed
funkzy added a commit to funkzy/linux that referenced this pull request Nov 21, 2026
…eduler gets the M5 Ultra's M and P core capacities (754 and 1024, not 1024 on all thirty)

The M5 Ultra's CPU nodes have no cpu-map, so Linux sees thirty CPUs in
one package with no clusters. parse_dt_topology() reads
capacity-dmips-mhz only while it walks cpu-map, so the M and P core
capacities that t6050-base.dtsi gives every T6050 CPU (dj's aurora-silicon#1399) never
reach the scheduler: on the J775D (Mac Studio) every CPU reports
cpu_capacity 1024, and cluster_cpus_list reads thirty clusters of one.

Describe the six clusters the way the one-die description does, commit
68c10dd ("arm64: dts: apple: t6050: Describe the CPU clusters"):
MPIDR Aff1 0, 1 and 2 are die 0's M0, M1 and P clusters and 0x10, 0x11
and 0x12 are die 1's, five cores each, as the ADT's cluster-id and
die-id give them. The clusters are listed flat, cluster0-2 on die 0 and
cluster3-5 on die 1, as the M1 and M2 Ultra descriptions (t6002.dtsi,
t6022.dtsi) list theirs; the topology parser does not support clusters
of clusters. The boot loader prunes the entries of cores it did not
start. The capacities in t6050-base.dtsi are kept as they are.

The caches are described already: dj's commit 1bd5f40 ("arm64:
dts: apple: t6050: Describe the CPU caches") gives each cluster its L2
and points each CPU's next-level-cache at it, and each cluster here is
the set of CPUs that share one of those L2s.

Only t6050-j775d.dts includes t6050-ultra.dtsi, so the J775D is the only
board whose device tree changes: the ten other T6050 board device trees
(J714c, J714s, J716c, J716s, J775c, J873s, K114c, K114s, K116c, K116s)
are byte-identical with and without the series. The one-die description
(t6050-max.dtsi) has its cpu-map already; this commit leaves it
unchanged.

Testing, on J775D, integration/aurora's head of 2026-11-21 (a84637d,
the merge of aurora-silicon#2654), one boot with and one without the series, both
with a local display fix that is not part of it: cpu_capacity reads 754
on the twenty M cores and 1024 on the ten P cores (1024 on all thirty
without), and cluster_cpus_list reads six lists of five (thirty lists of
one without). Each CPU's L2 reads the same either way: 8 MiB or 16 MiB,
shared by its own cluster's five.

Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants