Repository navigation
Conversation
…f too A pipe or futex ping-pong between two threads costs 13-17 us per round trip on an M5 Max (T6050) against 2.4-2.7 us on macOS on the same machine, and 1.1 us when both threads are pinned to one CPU. Traced with sched_waking/ipi/cpu_idle/sched_switch events, every wake-up of the partner's idle CPU takes about 3 us from the IPI to the core leaving WFI (the hardware powers idle cores down in WFI) and about 5 us until the wakee runs; and with each CPU busy only ~10% of the time, schedutil keeps the clusters at 1.3-2.3 GHz instead of 4.4-4.6 GHz. Even with the performance governor a cross-CPU round trip stays at 5-7 us. When the wakee and the waker both run only briefly between sleeps and the waker is alone on its CPU, wake the wakee there. Such a pair takes turns, so the wakee runs as soon as the waker sleeps, without an IPI or a core wake-up, and the pair keeps one CPU busy at a high clock. "Briefly" is a decaying average of the run time between sleeps below sysctl_sched_migration_cost / 8 (62.5 us by default) for both tasks, and the waker's current run below the same limit, so a task that keeps running does not hold its wakee up for long. Requiring the waker to be alone stacks at most one wakee on a CPU, and wake_wide() still spreads tasks that wake many others. The run time between sleeps is accounted in dequeue_task_fair() when a task goes to sleep. The WAKE_SHORT_LOCAL scheduler feature turns it off at run time. Signed-off-by: Satya Benson <satya@benson.earth>
Fork balancing starts a new task in the idlest group of the widest domain with SD_BALANCE_FORK. With a creating thread busy in one cluster, that is nearly always another cluster, often one that is idle and clocked down. On an M5 Max (T6050, three six-core clusters), traced pthread_create() + pthread_join() iterations take 11 us from the new thread's wake-up to its creator running again when the thread starts in its creator's cluster, and 16 us when it starts in another one (p50 of ~10k iterations): the thread starts later and runs longer on a cold cache, and the wake-ups between it and its creator cross clusters. The whole create+join loop costs 25-32 us against 12 us on macOS. Start a new thread, a task sharing its creator's mm, on an idle CPU in the creating CPU's LLC domain when there is one. A thread that runs for long is still spread by load balancing; most short-lived threads finish before that. New processes keep the usual fork and exec balancing. The FORK_LOCAL scheduler feature turns it off at run time. Signed-off-by: Satya Benson <satya@benson.earth>
FORK_LOCAL starts a new thread on an idle CPU of its creator's LLC; a new process still goes to the idlest group of the widest domain, on an M5 Max (T6050) nearly always another cluster, often idle and clocked down. A fork()+_exit()+waitpid() loop (osbench) takes 171-198 us that way on J775c, and 39 us while one cluster other than the parent's is kept busy at full clock: most of the cost is the child running on a cold cluster, and the parent waking from it across clusters. Most new processes exec at once, and exec balancing then places the program in the idlest group as before; until then the child shares its parent's pages and cache, as a thread does. Start new processes like new threads. FORK_LOCAL turns both off at run time. (A separate feature would be the 33rd, past the width of sysctl_sched_features.) Signed-off-by: Satya Benson <satya@benson.earth>
Each scheduler feature is a bit of sysctl_sched_features, an unsigned int, while the code that sets, clears and prints the bits already shifts 1UL. With every feature this tree has, the 32 bits are used up: a 33rd would be lost from the mask (and its debugfs state shown wrong) without a warning. Widen the mask to unsigned long and check at build time that the features fit. Signed-off-by: Satya Benson <satya@benson.earth>
…gest CPUs are full On an M5 Max (T6050) the six Everest cores (capacity 1024) form one cluster and the twelve Sawtooth cores (730) two more. STREAM triad with 6 threads gets 179 GB/s on Linux against 379 GB/s on macOS on the same machine, and 315 against 382 GB/s with 12: memory bandwidth is limited per cluster (6 threads: about 180-210 GB/s on the Everest cluster, 138 on a Sawtooth cluster), and the scheduler keeps all six threads on the Everest cluster. Bound two per cluster they get 359 GB/s, and 12 threads bound four per cluster 353 GB/s. The asymmetric-capacity rules keep them there by design: a group of bigger CPUs running one task each is never the busiest group for a smaller CPU, a smaller CPU never pulls the only task of a bigger one, tasks on smaller CPUs are misfits that bigger idle CPUs pull back, and wake-ups pick the first idle CPU the task fits. That is right for compute-bound threads: 6-thread 7-Zip does 73.4k MIPS on the Everest cluster and 65.5k spread two per cluster (6-thread SHA-256 gains 2%). Add ASYM_SPREAD, off by default. While at least as many CPUs are saturated as there are CPUs of the highest capacity, as decided when the root domain is balanced, balance as if capacities were equal: no misfit tasks, no protection of fully busy bigger groups or of lone tasks on bigger CPUs, nohz idle balancing kicked as on symmetric systems, and wake-ups looking for an idle CPU in the target's LLC. Below that, tasks fill the biggest CPUs first as before, so one to five threads still run on the Everest cores. Whether it should be on depends on the workload mix: the scheduler cannot tell a bandwidth-bound thread from a compute-bound one. Signed-off-by: Satya Benson <satya@benson.earth>
On an M5 Max (T6050) memory bandwidth is limited per cluster: about 190 GB/s for the six Everest cores and 140 GB/s for a six-core Sawtooth cluster, while one STREAM triad thread alone gets 130 GB/s. Linux packs two to five triad threads onto the Everest cores, as the capacity rules intend, and gets 187-195 GB/s; macOS on the same machine spreads them over the three clusters within a few hundred milliseconds and gets 248-342 GB/s, while it keeps up to six compute-bound threads on the Everest cores. ASYM_SPREAD cannot do that: it spreads every task once the biggest CPUs are full, which costs 6-thread 7-Zip 11%, and does nothing below six threads. Tell the two kinds of task apart from the CPU PMU. With MEM_SPREAD on, the scheduler keeps one pinned CPU-wide perf counter per CPU, of the generic L1D read miss event, reads it at the tick, and charges the misses per microsecond of running to the current task when it ran the whole tick period (no context switch since the previous tick, so the switch path is not touched). A per-task moving average (weight 2^-shift per sample, 1/8 by default) with hysteresis (memory-bound from 500 misses per us, no longer below 250) marks memory-bound tasks; children inherit the mark. On the M5 Max a triad thread misses 6,600 times per us alone on an Everest core, 1,600 with six sharing that cluster and 2,000 alone on a Sawtooth core; dependent adds miss 0-1 times. Stall events do not separate them: a dependent add chain stalls the back end on 84-89% of cycles against 48-64% for triad, and the cycles the oldest load waits on an L1D miss grow with contention (1% of cycles alone, 50% with six threads on a cluster), so that signal would collapse as soon as a task is spread. Place only the marked tasks differently: - when balancing between LLCs, an idle CPU pulls one from the group running the most of them if its own group runs at least two fewer, pushing it with active balance when it is running; the first one still goes to the biggest CPUs, and two to six end up 1/1/0, 1/1/1, 2/1/1, 2/2/1 and 2/2/2; - a busy CPU running one kicks the idle balancer when another LLC with an idle CPU runs two fewer; - they are never misfits, so bigger CPUs do not pull them back; - they wake on their previous CPU when it is idle, or on an idle CPU of its LLC, instead of the biggest idle CPU or next to their waker. Other tasks are placed as before. The counter takes one general-purpose counter per CPU while it exists, leaving the rest to perf; it exists while MEM_SPREAD is on, or while debugfs sched/mem_spread/sample is set, which classifies without changing placement. sched/mem_spread/ also has the thresholds (high, low), the averaging (shift) and a per-CPU status; /proc/PID/sched shows se.mem_rate and se.mem_bound. Without a PMU that counts L1D read misses nothing is classified. Off by default. Signed-off-by: Satya Benson <satya@benson.earth>
On the M5 Max (T6050) MEM_SPREAD brings STREAM triad with 2-7 OpenMP threads from 178-289 GB/s to 247-372 GB/s and the pthread triad with 2-6 threads to 249-383 GB/s, against macOS's 248-350, with the placements it was designed for (1/1/0 to 2/2/2 over the three clusters). Compute stays on the Everest cores: 7-Zip scores the same with 6 threads and with 18, and keeping the counters on every CPU costs no idle power and no wake-up latency. Turn it on by default; it can still be turned off at run time in debugfs sched/features. The counters were only set up when the feature was switched on in debugfs. Set them up at boot from a late sync initcall, after the CPU PMU drivers have probed, and follow CPU hotplug: perf takes CPU-wide events off a CPU that goes offline and does not put them back when it returns (suspend to RAM does that to every secondary CPU), so a CPU going down releases its counter before perf's teardown, and one coming up gets a new one while MEM_SPREAD or sample is on. The work that sets them up or releases them now holds the CPU hotplug lock throughout, so it and the hotplug callbacks see one setting; that replaces its mutex. The first setup logs how many CPUs count. Signed-off-by: Satya Benson <satya@benson.earth>
|
Thanks for the STREAM work. The comparison with macOS is very useful. A few things before this can become a default:
|
The previous commit turned MEM_SPREAD on wherever the CPU PMU provides the generic L1D read-miss event. Its thresholds (500 and 250 misses per microsecond, shift 3) were set from M5 Max (T6050) cores alone, so the default would be chosen by where it was tried, not by a hardware property. Leave it off until rows from other cores back a default or the thresholds scale with a hardware property. It can be turned on at run time in debugfs sched/features. The boot-time counter setup and the CPU hotplug handling stay; they take effect when MEM_SPREAD or sched/mem_spread/sample is turned on. Signed-off-by: Satya Benson <satya@benson.earth>
|
Claude Opus 5.5 on @satchlj's behalf: thanks, you're right on all three.
I've updated the description with all of this. 🤖 Generated with Claude Code |
ASYM_SPREAD balanced load as if CPU capacities were equal once the biggest CPUs were saturated. It was kept off because it costs compute-bound work (6-thread 7-Zip lost 11% on an M5 Max), and MEM_SPREAD now spreads only the tasks that are memory-bound. Remove it rather than keep a switch nobody should turn on: the feature bit, update_asym_spread() and the root domain's asym_spread, and its checks in misfit tracking, wake-up placement, load balancing and the nohz kick. Those paths are as they were before it, except that misfit tracking still treats a memory-bound task as fitting when MEM_SPREAD is on. Signed-off-by: Satya Benson <satya@benson.earth>
|
Claude Opus 5.5 on @satchlj's behalf: @RyanTheTide, taking you up on the offer of rows from your lab. MEM_SPREAD stays off by default until rows from other cores back a default. Lobata won't carry it on locally, so the J775c switch waits on these rows. What J775c shows
KernelThis PR's head (stacked on #2481 → #2480) plus #2517. #2517 maps the generic L1D read-miss event. Its first commit covers M4 and T6050, which includes your M4 Pro and A18 Pro ( Recipe (same boot, MEM_SPREAD off and on)
Then
The 500/250 thresholds were set on M5 Max cores, so the rows will also show whether they need to scale. 🤖 Generated with Claude Code |
|
Claude Opus 5.5 on @satchlj's behalf: ASYM_SPREAD is dropped in
Those paths are back as they were, except that misfit tracking still treats a memory-bound task as fitting when MEM_SPREAD is on. Tested: built and booted on J775c in lobata's cand-4. That is
Untested:
The lab rows requested above would cover the first and the last. 🤖 Generated with Claude Code |
|
Thanks for leaving MEM_SPREAD off by default and dropping ASYM_SPREAD, and for the detailed recipe. We will run it on our M1 Pro, M2, M4 Pro and A18 Pro on a kernel with this head and #2517 (including 9491b93), checking first that |
c0d8efc to
23e3003
Compare
e6103ff to
6c3280c
Compare
Claude Opus 5.5 on @satchlj's behalf.
Update after review (2026-11-20):
Stacked on #2481 (fork/exec placement), which is stacked on #2480. Review the last five commits:
sched: Make sysctl_sched_features an unsigned long. The new features need bits past 31.sched/fair: Optionally spread load across CPU capacities once the biggest CPUs are full(ASYM_SPREAD, off). It is kept off because it costs compute-bound work; MEM_SPREAD replaces it.sched/fair: Spread memory-bound tasks across LLCs (MEM_SPREAD). It classifies tasks from a pinned per-CPU L1D-read-miss counter and spreads only the memory-bound ones across LLCs.sched/fair: Turn MEM_SPREAD on by default. It sets the counters up at boot and follows CPU hotplug.sched/fair: Leave MEM_SPREAD off by default. It keeps commit 4's boot setup and hotplug handling.The commit messages carry the design and the thresholds. Without a PMU that counts L1D read misses, nothing is classified.
It depends on the LLC topology, like #2481: on T6050 it needs the CPU cache DT, which is a separate DT PR to follow.
Evidence (J775c, Mac Studio M5 Max, earlier private lobata kernels with the cache DT and a PMU cache map)
Testing
integration/aurora26294e82).6ae9c3caplus 63 commits), which does not yet have the cache DT.26294e82, and not measured on the tip.Assisted-by: LLM (Anthropic:Claude Opus 5.5)