Skip to content

sched/fair: spread memory-bound tasks across LLCs (MEM_SPREAD, stacked on #2481) - #2482

Open
satchlj wants to merge 9 commits into
satchlj/sched-fork-exec-llc-20261119from
satchlj/sched-mem-spread-20261119
Open

satchlj wants to merge 9 commits into
satchlj/sched-fork-exec-llc-20261119from
satchlj/sched-mem-spread-20261119

Conversation

@satchlj

@satchlj satchlj commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Claude Opus 5.5 on @satchlj's behalf.

Update after review (2026-11-20):

Stacked on #2481 (fork/exec placement), which is stacked on #2480. Review the last five commits:

  1. sched: Make sysctl_sched_features an unsigned long. The new features need bits past 31.
  2. sched/fair: Optionally spread load across CPU capacities once the biggest CPUs are full (ASYM_SPREAD, off). It is kept off because it costs compute-bound work; MEM_SPREAD replaces it.
  3. sched/fair: Spread memory-bound tasks across LLCs (MEM_SPREAD). It classifies tasks from a pinned per-CPU L1D-read-miss counter and spreads only the memory-bound ones across LLCs.
  4. sched/fair: Turn MEM_SPREAD on by default. It sets the counters up at boot and follows CPU hotplug.
  5. sched/fair: Leave MEM_SPREAD off by default. It keeps commit 4's boot setup and hotplug handling.

The commit messages carry the design and the thresholds. Without a PMU that counts L1D read misses, nothing is classified.

It depends on the LLC topology, like #2481: on T6050 it needs the CPU cache DT, which is a separate DT PR to follow.

Evidence (J775c, Mac Studio M5 Max, earlier private lobata kernels with the cache DT and a PMU cache map)

  • MEM_SPREAD, toggled in the same boot (2026-11-06): STREAM OpenMP triad with 2–7 threads went from 200 / 276 / 250 / 189 / 178 / 289 to 247 / 343 / 340 / 317 / 350 / 372 GB/s. 7-Zip was unchanged: about 79.1k at 6 threads, and 188.6k vs 187.5k at 18.
  • With it on by default (2026-11-06): triad with 2–7 threads gave 252 / 354 / 357 / 359 / 384 / 385 GB/s. macOS on the same machine (2026-11-05) gave 248 / 319 / 342 / – / 350.
  • ASYM_SPREAD (2026-11-05): 6-thread STREAM went from 266 to 352 GB/s, but 6-thread 7-Zip lost 11% (73.4k to 65.5k MIPS), which is why it is off.
  • Caveats:
    • The CPU-hotplug path has not been exercised on hardware.
    • One 16-thread sample dropped from 362 to 319 GB/s; a 3-round repeat showed it was noise.
    • The feature holds one general-purpose PMU counter per CPU while it is on.

Testing

Assisted-by: LLM (Anthropic:Claude Opus 5.5)

…f too

A pipe or futex ping-pong between two threads costs 13-17 us per round
trip on an M5 Max (T6050) against 2.4-2.7 us on macOS on the same machine,
and 1.1 us when both threads are pinned to one CPU. Traced with
sched_waking/ipi/cpu_idle/sched_switch events, every wake-up of the
partner's idle CPU takes about 3 us from the IPI to the core leaving WFI
(the hardware powers idle cores down in WFI) and about 5 us until the
wakee runs; and with each CPU busy only ~10% of the time, schedutil keeps
the clusters at 1.3-2.3 GHz instead of 4.4-4.6 GHz. Even with the
performance governor a cross-CPU round trip stays at 5-7 us.

When the wakee and the waker both run only briefly between sleeps and the
waker is alone on its CPU, wake the wakee there. Such a pair takes turns,
so the wakee runs as soon as the waker sleeps, without an IPI or a core
wake-up, and the pair keeps one CPU busy at a high clock.

"Briefly" is a decaying average of the run time between sleeps below
sysctl_sched_migration_cost / 8 (62.5 us by default) for both tasks, and
the waker's current run below the same limit, so a task that keeps
running does not hold its wakee up for long. Requiring the waker to be
alone stacks at most one wakee on a CPU, and wake_wide() still spreads
tasks that wake many others. The run time between sleeps is accounted in
dequeue_task_fair() when a task goes to sleep.

The WAKE_SHORT_LOCAL scheduler feature turns it off at run time.

Signed-off-by: Satya Benson <satya@benson.earth>
Fork balancing starts a new task in the idlest group of the widest domain
with SD_BALANCE_FORK. With a creating thread busy in one cluster, that is
nearly always another cluster, often one that is idle and clocked down.

On an M5 Max (T6050, three six-core clusters), traced pthread_create() +
pthread_join() iterations take 11 us from the new thread's wake-up to its
creator running again when the thread starts in its creator's cluster,
and 16 us when it starts in another one (p50 of ~10k iterations): the
thread starts later and runs longer on a cold cache, and the wake-ups
between it and its creator cross clusters. The whole create+join loop
costs 25-32 us against 12 us on macOS.

Start a new thread, a task sharing its creator's mm, on an idle CPU in the
creating CPU's LLC domain when there is one. A thread that runs for long
is still spread by load balancing; most short-lived threads finish before
that. New processes keep the usual fork and exec balancing.

The FORK_LOCAL scheduler feature turns it off at run time.

Signed-off-by: Satya Benson <satya@benson.earth>
FORK_LOCAL starts a new thread on an idle CPU of its creator's LLC; a new
process still goes to the idlest group of the widest domain, on an M5 Max
(T6050) nearly always another cluster, often idle and clocked down. A
fork()+_exit()+waitpid() loop (osbench) takes 171-198 us that way on
J775c, and 39 us while one cluster other than the parent's is kept busy at
full clock: most of the cost is the child running on a cold cluster, and
the parent waking from it across clusters.

Most new processes exec at once, and exec balancing then places the program
in the idlest group as before; until then the child shares its parent's
pages and cache, as a thread does. Start new processes like new threads.
FORK_LOCAL turns both off at run time. (A separate feature would be the
33rd, past the width of sysctl_sched_features.)

Signed-off-by: Satya Benson <satya@benson.earth>
Each scheduler feature is a bit of sysctl_sched_features, an unsigned
int, while the code that sets, clears and prints the bits already shifts
1UL. With every feature this tree has, the 32 bits are used up: a 33rd
would be lost from the mask (and its debugfs state shown wrong) without
a warning. Widen the mask to unsigned long and check at build time that
the features fit.

Signed-off-by: Satya Benson <satya@benson.earth>
…gest CPUs are full

On an M5 Max (T6050) the six Everest cores (capacity 1024) form one
cluster and the twelve Sawtooth cores (730) two more. STREAM triad with 6
threads gets 179 GB/s on Linux against 379 GB/s on macOS on the same
machine, and 315 against 382 GB/s with 12: memory bandwidth is limited
per cluster (6 threads: about 180-210 GB/s on the Everest cluster, 138
on a Sawtooth cluster), and the scheduler keeps all six threads on the
Everest cluster. Bound two per cluster they get 359 GB/s, and 12 threads
bound four per cluster 353 GB/s.

The asymmetric-capacity rules keep them there by design: a group of
bigger CPUs running one task each is never the busiest group for a
smaller CPU, a smaller CPU never pulls the only task of a bigger one,
tasks on smaller CPUs are misfits that bigger idle CPUs pull back, and
wake-ups pick the first idle CPU the task fits. That is right for
compute-bound threads: 6-thread 7-Zip does 73.4k MIPS on the Everest
cluster and 65.5k spread two per cluster (6-thread SHA-256 gains 2%).

Add ASYM_SPREAD, off by default. While at least as many CPUs are
saturated as there are CPUs of the highest capacity, as decided when the
root domain is balanced, balance as if capacities were equal: no misfit
tasks, no protection of fully busy bigger groups or of lone tasks on
bigger CPUs, nohz idle balancing kicked as on symmetric systems, and
wake-ups looking for an idle CPU in the target's LLC. Below that, tasks
fill the biggest CPUs first as before, so one to five threads still run
on the Everest cores.

Whether it should be on depends on the workload mix: the scheduler cannot
tell a bandwidth-bound thread from a compute-bound one.

Signed-off-by: Satya Benson <satya@benson.earth>
On an M5 Max (T6050) memory bandwidth is limited per cluster: about
190 GB/s for the six Everest cores and 140 GB/s for a six-core Sawtooth
cluster, while one STREAM triad thread alone gets 130 GB/s. Linux packs
two to five triad threads onto the Everest cores, as the capacity rules
intend, and gets 187-195 GB/s; macOS on the same machine spreads them
over the three clusters within a few hundred milliseconds and gets
248-342 GB/s, while it keeps up to six compute-bound threads on the
Everest cores. ASYM_SPREAD cannot do that: it spreads every task once
the biggest CPUs are full, which costs 6-thread 7-Zip 11%, and does
nothing below six threads.

Tell the two kinds of task apart from the CPU PMU. With MEM_SPREAD on,
the scheduler keeps one pinned CPU-wide perf counter per CPU, of the
generic L1D read miss event, reads it at the tick, and charges the misses
per microsecond of running to the current task when it ran the whole
tick period (no context switch since the previous tick, so the switch
path is not touched). A per-task moving average (weight 2^-shift per
sample, 1/8 by default) with hysteresis (memory-bound from 500 misses
per us, no longer below 250) marks memory-bound tasks; children inherit
the mark. On the M5 Max a triad thread misses 6,600 times per us alone
on an Everest core, 1,600 with six sharing that cluster and 2,000 alone
on a Sawtooth core; dependent adds miss 0-1 times. Stall events do not
separate them: a dependent add chain stalls the back end on 84-89% of
cycles against 48-64% for triad, and the cycles the oldest load waits on
an L1D miss grow with contention (1% of cycles alone, 50% with six
threads on a cluster), so that signal would collapse as soon as a task
is spread.

Place only the marked tasks differently:
- when balancing between LLCs, an idle CPU pulls one from the group
  running the most of them if its own group runs at least two fewer,
  pushing it with active balance when it is running; the first one
  still goes to the biggest CPUs, and two to six end up 1/1/0, 1/1/1,
  2/1/1, 2/2/1 and 2/2/2;
- a busy CPU running one kicks the idle balancer when another LLC with
  an idle CPU runs two fewer;
- they are never misfits, so bigger CPUs do not pull them back;
- they wake on their previous CPU when it is idle, or on an idle CPU of
  its LLC, instead of the biggest idle CPU or next to their waker.
Other tasks are placed as before.

The counter takes one general-purpose counter per CPU while it exists,
leaving the rest to perf; it exists while MEM_SPREAD is on, or while
debugfs sched/mem_spread/sample is set, which classifies without
changing placement. sched/mem_spread/ also has the thresholds (high,
low), the averaging (shift) and a per-CPU status; /proc/PID/sched shows
se.mem_rate and se.mem_bound. Without a PMU that counts L1D read misses
nothing is classified. Off by default.

Signed-off-by: Satya Benson <satya@benson.earth>
On the M5 Max (T6050) MEM_SPREAD brings STREAM triad with 2-7 OpenMP
threads from 178-289 GB/s to 247-372 GB/s and the pthread triad with 2-6
threads to 249-383 GB/s, against macOS's 248-350, with the placements
it was designed for (1/1/0 to 2/2/2 over the three clusters). Compute
stays on the Everest cores: 7-Zip scores the same with 6 threads and
with 18, and keeping the counters on every CPU costs no idle power and
no wake-up latency. Turn it on by default; it can still be turned off
at run time in debugfs sched/features.

The counters were only set up when the feature was switched on in
debugfs. Set them up at boot from a late sync initcall, after the CPU
PMU drivers have probed, and follow CPU hotplug: perf takes CPU-wide
events off a CPU that goes offline and does not put them back when it
returns (suspend to RAM does that to every secondary CPU), so a CPU
going down releases its counter before perf's teardown, and one coming
up gets a new one while MEM_SPREAD or sample is on. The work that sets
them up or releases them now holds the CPU hotplug lock throughout, so
it and the hotplug callbacks see one setting; that replaces its mutex.
The first setup logs how many CPUs count.

Signed-off-by: Satya Benson <satya@benson.earth>
@RyanTheTide

Copy link
Copy Markdown
Contributor

Thanks for the STREAM work. The comparison with macOS is very useful.

A few things before this can become a default:

  • kernel/sched/fair.c:6445-6449 asks for the generic PERF_TYPE_HW_CACHE L1D read-miss event. In this tree the Apple CPU PMU drivers pass no cache map (drivers/perf/apple_m1_cpu_pmu.c:791,802,809,815), so the event is refused and MEM_SPREAD classifies nothing on any Apple SoC, T6050 included. The numbers must have come from a kernel with a cache map. That map belongs in the PMU driver's per-core match data (the l1d-load-misses events are already there for M4 and T6050).
  • kernel/sched/fair.c:6373-6375 sets the thresholds (500/250 misses per µs, shift 3) from M5 Max cores, but features.h:125 turns MEM_SPREAD on for every machine with such a counter. Please back the default with rows on other cores, or scale the thresholds by a hardware property. We can run STREAM/7-Zip rows on our M1 Pro, M2, M4 Pro and A18 Pro.
  • features.h:116 ASYM_SPREAD stays off because it costs compute. If MEM_SPREAD replaces it, could it be dropped instead of kept as a switch?

The previous commit turned MEM_SPREAD on wherever the CPU PMU provides
the generic L1D read-miss event. Its thresholds (500 and 250 misses per
microsecond, shift 3) were set from M5 Max (T6050) cores alone, so the
default would be chosen by where it was tried, not by a hardware
property. Leave it off until rows from other cores back a default or
the thresholds scale with a hardware property. It can be turned on at
run time in debugfs sched/features.

The boot-time counter setup and the CPU hotplug handling stay; they
take effect when MEM_SPREAD or sched/mem_spread/sample is turned on.

Signed-off-by: Satya Benson <satya@benson.earth>
@satchlj

satchlj commented Nov 20, 2026

Copy link
Copy Markdown
Member Author

Claude Opus 5.5 on @satchlj's behalf: thanks, you're right on all three.

  • Which kernel: the STREAM and 7-Zip rows came from lobata's private kernels (2026-11-05/06). Their PMU driver had its own cache map, which is not in this tree and won't be ported. In this tree the event is refused, as you found, so the numbers say nothing about this PR as posted. Step 0 found no open PR with an Apple PMU cache map, so I wrote one first: perf: apple_m1: map the generic L1D read-miss event on M1, M2, M4 and T6050 #2517. It maps L1D | OP_READ | RESULT_MISS to the counter the M4 and T6050 drivers already name l1d-load-misses, in each core's own table. The private map counted the same T6050 event number (0x5a3). I'll re-measure the rows on this tree with perf: apple_m1: map the generic L1D read-miss event on M1, M2, M4 and T6050 #2517 and post them here. This PR now depends on it.
  • Default: c784abcba7 leaves MEM_SPREAD off by default. The thresholds come from M5 Max cores only, so I won't propose a default until rows from other cores support it or the thresholds scale with a hardware property. Your M1 Pro, M2, M4 Pro and A18 Pro STREAM/7-Zip rows would be the basis for that. M1 and M2 would also need their L1D read-miss event mapped in perf: apple_m1: map the generic L1D read-miss event on M1, M2, M4 and T6050 #2517 first.
  • ASYM_SPREAD: yes, I'll drop it rather than keep a switch. It shares the placement path at fair.c asym_spread() || mem_spread_task() and the load-balance checks, so I'll remove it as its own commit once it builds. A plain revert conflicts with the MEM_SPREAD commit.

I've updated the description with all of this.

🤖 Generated with Claude Code

ASYM_SPREAD balanced load as if CPU capacities were equal once the
biggest CPUs were saturated. It was kept off because it costs
compute-bound work (6-thread 7-Zip lost 11% on an M5 Max), and
MEM_SPREAD now spreads only the tasks that are memory-bound. Remove it
rather than keep a switch nobody should turn on: the feature bit,
update_asym_spread() and the root domain's asym_spread, and its checks
in misfit tracking, wake-up placement, load balancing and the nohz kick.
Those paths are as they were before it, except that misfit tracking
still treats a memory-bound task as fitting when MEM_SPREAD is on.

Signed-off-by: Satya Benson <satya@benson.earth>
@satchlj

satchlj commented Nov 20, 2026

Copy link
Copy Markdown
Member Author

Claude Opus 5.5 on @satchlj's behalf: @RyanTheTide, taking you up on the offer of rows from your lab. MEM_SPREAD stays off by default until rows from other cores back a default. Lobata won't carry it on locally, so the J775c switch waits on these rows.

What J775c shows

  • Cross-kernel, not an A/B: STREAM triad at 6 threads is 245.6 GB/s on lobata's cand-3, against 358.7 GB/s on lobata's default kernel. cand-3 is integration/aurora 61295c26 + 80 commits (this PR at c784abcba, MEM_SPREAD off). The default runs MEM_SPREAD on through a private PMU cache map. 12 threads: 308.3 vs 354.6. 18: 329.6 vs 344.3. 1T: 131.3 vs 124.7.
  • Same-boot toggle, private kernel (2026-11-06): triad with 2–7 threads went from 200 / 276 / 250 / 189 / 178 / 289 to 247 / 343 / 340 / 317 / 350 / 372 GB/s. 7-Zip was unchanged at 6 and 18 threads.

Kernel

This PR's head (stacked on #2481 → #2480) plus #2517. #2517 maps the generic L1D read-miss event. Its first commit covers M4 and T6050, which includes your M4 Pro and A18 Pro (t6040/t8140 use the M4 tables). I've just added 9491b9317 for M1 and M2 (L1D_CACHE_MISS_LD 0xa3). Without it, MEM_SPREAD classifies nothing on your M1 Pro and M2, and those rows would show no change by construction. It isn't built yet, so please confirm perf stat -e L1-dcache-load-misses counts there.

Recipe (same boot, MEM_SPREAD off and on)

  1. Toggle, as root: echo MEM_SPREAD > /sys/kernel/debug/sched/features (on) and echo NO_MEM_SPREAD > /sys/kernel/debug/sched/features (off). Turning it on should log sched: MEM_SPREAD: counting L1D load misses on N CPUs once. If that line is missing, the machine has no usable counter and the row is "no classification".
  2. STREAM 5.10 (upstream stream.c, rev 6703f75), built as:
    clang -O3 -fopenmp -DSTREAM_ARRAY_SIZE=100000000 -DNTIMES=20 stream.c -o stream_omp.
    Run OMP_NUM_THREADS=<n> ./stream_omp with no OMP_PROC_BIND, and report the best Triad rate.
    Thread counts: n = 1, n = the number of performance cores, and n = all cores.
  3. CPU-bound control: 7-Zip 26.05 (official arm64 7zz). Run 7zz b 3 -mmt1, then 7zz b 3 -mmt<performance cores>, then 7zz b 3 (all threads), and report the Tot: rating.
  4. Order: off, on, off, on, off, on (three rounds of each), in one boot, with the machine otherwise idle. Report each round, not only the mean.
  5. With each row: the machine, kernel head, uname -rv, and per-cluster core counts (/sys/devices/system/cpu/cpufreq/policy*/related_cpus).

Then

  • If no core regresses (STREAM or 7-Zip) with MEM_SPREAD on, I'll propose default-on in this PR, with your rows as the evidence.
  • If one does, I'll look for the hardware property that separates those cases and select on that. Candidates: per-cluster memory bandwidth relative to one core, or the number of LLCs per capacity class. I won't use a SoC list.

The 500/250 thresholds were set on M5 Max cores, so the rows will also show whether they need to scale.

🤖 Generated with Claude Code

@satchlj

satchlj commented Nov 20, 2026

Copy link
Copy Markdown
Member Author

Claude Opus 5.5 on @satchlj's behalf: ASYM_SPREAD is dropped in e6103ff0e, as you asked, rather than kept as a switch. The commit removes:

  • the feature bit, update_asym_spread() and rd->asym_spread;
  • their checks in misfit tracking, wake-up placement, load balancing and the nohz kick.

Those paths are back as they were, except that misfit tracking still treats a memory-bound task as fitting when MEM_SPREAD is on.

Tested: built and booted on J775c in lobata's cand-4. That is 7400d465c320 = integration/aurora e0ee46a1 + 80 commits, with this PR's head plus this commit, booted 2026-11-20 05:57Z. MEM_SPREAD was off, the default. Its CPU rows match the previous candidate, which still had ASYM_SPREAD (off):

Row without this commit with it
7-Zip 1T on cpu0 9662 9668
7-Zip 1T unpinned 12716 12731
7-Zip 18T 185607 185182

Untested:

  • MEM_SPREAD switched on with this commit;
  • CPU hotplug;
  • any SoC other than T6050.

The lab rows requested above would cover the first and the last.

🤖 Generated with Claude Code

@RyanTheTide

Copy link
Copy Markdown
Contributor

Thanks for leaving MEM_SPREAD off by default and dropping ASYM_SPREAD, and for the detailed recipe. We will run it on our M1 Pro, M2, M4 Pro and A18 Pro on a kernel with this head and #2517 (including 9491b93), checking first that perf stat -e L1-dcache-load-misses counts and that the MEM_SPREAD: counting line appears. Rows will follow here in the format you listed, each round reported separately.

@satchlj

satchlj commented Nov 21, 2026

Copy link
Copy Markdown
Member Author

Claude Opus 5.5 on @satchlj's behalf: trailer-only rewrite, removed Co-Authored-By from 9 commit(s); same trees, Signed-off-by kept (e6103ff -> 6c3280c, --force-with-lease).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants