Skip to content

sched/fair: wake brief tasks on the waking CPU when the waker is brief too - #2480

Open
satchlj wants to merge 1 commit into
integration/aurorafrom
satchlj/sched-brief-wake-20261119
Open

satchlj wants to merge 1 commit into
integration/aurorafrom
satchlj/sched-brief-wake-20261119

Conversation

@satchlj

@satchlj satchlj commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

WAKE_SHORT_LOCAL keeps a brief wakee on the waking CPU when the waker is also brief and alone. The wake path adds the unaccounted time since exec_start to the waker's recorded runtime since its last sleep using the refreshed runqueue task clock.

EAS gets first choice. The T6050 CPU DT on this base has no OPP power data or dynamic-power coefficient to register its cpufreq energy model. The prior lobata evidence has no runtime EAS readback for the measured boot.

This is the base of the scheduler stack. #2481 adds fork placement, and #2482 adds MEM_SPREAD.

Evidence

On earlier J775c kernels, pinned S↔S pipe ping-pong took 12.8–13.1 µs and condvar ping-pong took 13.5 µs. A kernel that combined this heuristic with fork placement, the busy boost (#2479), and cpufreq fast switch measured 1.10 µs for pipe and 1.24 µs for condvar. The isolated effect of this change remains unmeasured. The heuristic applies to task-context wakes.

Testing

  • ARM64 kernel/sched/fair.o built with W=1.
  • The revised head is untested on hardware. Compare the feature on and off on J775c and a non-T6050 Mac, record EAS state, and measure unpinned ping-pong, hackbench, and schbench p99.

Assisted-by: LLM (OpenAI:GPT-6.2 Sol)

@RyanTheTide

Copy link
Copy Markdown
Contributor

Blockers: 2.

  1. The commit carries Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>. Drop that trailer and keep the Signed-off-by:; the disclosure goes in the PR body's last line as Assisted-by: LLM (Claude:Opus 5.5). Right now the body opens with "Claude Opus 5.5 on @satchlj's behalf" and ends with the "Generated with Claude Code" line.
  2. kernel/sched/fair.c:9967: waker->se.sum_exec_runtime is only brought up to date by update_curr() (tick, schedule, dequeue). Nothing on the ttwu path updates it before this read, so "the waker's current run below the limit" can be stale by up to a tick (4 ms at HZ=250), far above the 62.5 us limit. Measure the current run from waker->se.exec_start against the rq task clock instead, or drop that condition from the commit message.
  3. kernel/sched/fair.c:10007: on a system with EAS active (asymmetric capacities, an energy model and schedutil), find_energy_efficient_cpu() returns a CPU first whenever the root domain is not overutilized. That is exactly the case targeted here, two ping-pong threads each ~10% busy. So wake_short_local() at line 10016 never runs there. Either show that EAS is off on the target (/proc/sys/kernel/sched_energy_aware, or no rd->pd), or decide where the check belongs relative to EAS and say so in the commit message. As it stands the commit message describes a path the measured kernel may not take. (Not a blocker if EAS is shown to be off.)

The burst accounting in dequeue_task_fair() is correct for delayed dequeue: the sleep is counted on the DEQUEUE_SLEEP call that sets sched_delayed, the later DEQUEUE_DELAYED call is excluded, and __dequeue_task() has already run update_curr().

HW row: J775c, this commit alone on integration/aurora 26294e8, same boot, WAKE_SHORT_LOCAL on vs off: pipe/futex ping-pong round trip unpinned, plus hackbench -p -g 8 and schbench p99 wakeup latency (no regression with the feature on), sched_energy_aware value stated. Repeat the on/off pair on one non-T6050 Mac (M1/M2), and include the regression summary (0 AVC denials, 0 failed units, 0 new kernel errors).

Assisted-by: LLM (Claude:Opus 5.5)

…f too

A pipe or futex ping-pong between two threads costs 13-17 us per round
trip on an M5 Max (T6050) against 2.4-2.7 us on macOS on the same machine,
and 1.1 us when both threads are pinned to one CPU. Traced with
sched_waking/ipi/cpu_idle/sched_switch events, every wake-up of the
partner's idle CPU takes about 3 us from the IPI to the core leaving WFI
(the hardware powers idle cores down in WFI) and about 5 us until the
wakee runs; and with each CPU busy only ~10% of the time, schedutil keeps
the clusters at 1.3-2.3 GHz instead of 4.4-4.6 GHz. Even with the
performance governor a cross-CPU round trip stays at 5-7 us.

When the wakee and the waker both run only briefly between sleeps and the
waker is alone on its CPU, wake the wakee there. Such a pair takes turns,
so the wakee runs as soon as the waker sleeps, without an IPI or a core
wake-up, and the pair keeps one CPU busy at a high clock.

"Briefly" is a decaying average of the run time between sleeps below
sysctl_sched_migration_cost / 8 (62.5 us by default) for both tasks, and
the waker's runtime since its last sleep below the same limit. Include
the unaccounted portion from exec_start to the refreshed rq task clock
under the local rq lock, so a scheduler tick is not needed to see it.
Requiring the waker to be alone stacks at most one wakee on a CPU.
wake_wide() still spreads tasks that wake many others. The run time
between sleeps is accounted in dequeue_task_fair() when a task sleeps.

Give energy-aware scheduling first choice. This keeps its placement
when it finds a CPU and leaves the short-wake heuristic for the normal
fair wake path. The T6050 tables on this base have neither OPP power
nor a CPU dynamic-power coefficient, so the DT-based cpufreq energy
model is unavailable.

The WAKE_SHORT_LOCAL scheduler feature turns it off at run time.

Signed-off-by: Satya Benson <satya@benson.earth>
@satchlj
satchlj force-pushed the satchlj/sched-brief-wake-20261119 branch from dd4abcc to c86e358 Compare November 21, 2026 12:33
@satchlj

satchlj commented Nov 21, 2026

Copy link
Copy Markdown
Member Author

Codex GPT-6.2 Sol on @satchlj's behalf

Removed the co-author trailer and updated the PR footer. The waker check now adds the time from se.exec_start to the refreshed rq task clock to the runtime already accounted since its last sleep. It samples under the local rq lock and falls back to normal placement if the trylock fails.

EAS keeps first choice when it returns a CPU. The T6050 DT on this base supplies neither OPP power data nor a dynamic-power coefficient for the cpufreq energy model. I found no saved runtime EAS readback for the measured lobata boot.

ARM64 kernel/sched/fair.o built with W=1. The revised head is untested on hardware. The isolated J775c and non-T6050 comparisons, EAS readback, and regression checks are pending.

@satchlj

satchlj commented Nov 21, 2026

Copy link
Copy Markdown
Member Author

Codex GPT-6.2 Sol on @satchlj's behalf

J775c same-boot A/B on exact c86e358c4a4c, parent 26294e82e9c87a0162a7973c5e23c599ceaea22f. That parent was 370 commits behind integration/aurora a84637dca209 when checked on Nov 21. All changed files merge cleanly in a text-only check against that tip. Runtime on the merged tip is untested. The one-shot used the J775c carrier DTB, PCI SDHCI disabled in the trial config, and arm64.nomte console=tty0 console=ttyDC0.

Three independent invocations per state, ordered on/off/off/on/on/off. The unpinned pipe and pthread condvar/futex probes each ran five internal passes of 50,000 round trips. The table gives median and range across invocations.

Workload WAKE_SHORT_LOCAL on WAKE_SHORT_LOCAL off
Pipe round trip, µs 1.746 (1.728–1.780) 15.025 (14.481–15.138)
Condvar/futex round trip, µs 1.965 (1.946–1.968) 23.840 (21.533–25.259)
hackbench -p -g 8, s 0.013 (0.013–0.015) 0.015 (0.015–0.017)
schbench wake-up p99, µs 2,572 (2,492–2,628) 2,620 (2,524–2,628)

schbench used one message thread, 18 workers, 15 s runtime and 3 s warmup. Initial die temperatures were 47.3/82.2/83.9 °C for on and 81.0/82.9/84.0 °C for off. The schbench ranges overlap. sched_energy_aware=1 read back, but the Energy Model debugfs listing and archived registration logs had no perf-domain entries; active root-domain EAS was not directly read. The production head contains SCHED_FEAT(WAKE_SHORT_LOCAL, true). The A/B used the existing scheduler debugfs feature toggle and restored the original on state.

0 AVC denials, one failed unit (ac-power-mode.service), and three trial boot error lines absent from Aura (Gen4 SPMI IRQ and AOP). Two further DCP errors were also present on Aura. Warm handback verified default Aura, both ESPs offline and direct Thunderbolt peer 2/2. Non-T6050 hardware remains untested here.

@RyanTheTide

Copy link
Copy Markdown
Contributor

Re-review of c86e358: No blockers.

Points from the last review:

  1. Co-author trailer: resolved. c86e358 carries only the human Signed-off-by, and the body ends with the Assisted-by: line.
  2. Stale waker runtime: resolved. kernel/sched/fair.c:9979-9986 now takes the local rq lock (trylock, falling back to normal placement), refreshes the clock and adds rq_clock_task() - exec_start to the runtime accounted since the last sleep, so the check no longer waits for a tick.
  3. EAS ordering: resolved. fair.c:10029-10036 keeps EAS first and the commit message says so. The J775c row shows sched_energy_aware=1 with no perf domains registered, so EAS was not active in the measured boot.

What changed since then is the locked runtime sample in wake_short_local(). That is sound: select_task_rq_fair() runs with p->pi_lock held and IRQs off, pi_lock -> rq lock is the normal order, the trylock cannot self-deadlock, and the lock is dropped before ttwu_queue() takes any rq lock. The J775c A/B (pipe 15.0 -> 1.7 us, condvar 23.8 -> 2.0 us, hackbench and schbench p99 not worse) covers the T6050 side.

HW row: this head merged onto the current integration/aurora tip on the default command line (the posted row was 370 commits behind, with arm64.nomte and PCI SDHCI off), on J775c and on one non-T6050 Mac (M1 or M2), WAKE_SHORT_LOCAL on/off in the same boot: unpinned pipe/futex round trip, hackbench -p -g 8 -l 10000 (enough loops that the 1 ms output resolution is not the limit) and schbench p99. Pass criteria: no regression with the feature on, 0 AVC, 0 failed units, 0 new kernel errors. The ac-power-mode.service failure and the SPMI/AOP error lines from the trial boot should be gone on the tip, or shown to come from that base.

Assisted-by: LLM (Claude:Opus 5.5)

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-PR checklist review (§1–§6) on head c86e358c4a4c725ea3eb822317f142f0ad40ffef.

Blockers

  1. Unsigned underflow in wake_short_local() (kernel/sched/fair.c) — burst += rq_clock_task(rq) - waker->se.exec_start; performs an unclamped unsigned u64 subtraction before adding to burst. Both update_se() and preempt_sync() in kernel/sched/fair.c explicitly guard against rq_clock_task(rq) < se->exec_start with a signed check (s64 delta = rq_clock_task(rq) - waker->se.exec_start; if (delta > 0) burst += delta;). Without clamping, a negative delta wraps and can falsely push burst below limit (when burst > limit) or overflow.
  2. Missing core-scheduling cookie check in wake_short_local() (kernel/sched/fair.c) — WAKE_SHORT_LOCAL is enabled by default (SCHED_FEAT(WAKE_SHORT_LOCAL, true)) for all architectures and returns cpu directly, bypassing select_idle_sibling() / __select_idle_cpu(). On SMT systems with CONFIG_SCHED_CORE active, returning cpu without checking sched_cpu_cookie_match(rq, p) can stack an incompatible-cookie wakee onto waker's CPU and cause unnecessary forced-idle on SMT siblings.
  3. Burst-accounting gaps across throttled cgroups and scheduling-class transitions (kernel/sched/fair.c):
    • In dequeue_task_fair(), the early return for if (task_is_throttled(p)) runs before update_burst_avg(p). A task sleeping while its CFS bandwidth group is throttled skips recording prev_sleep_sum_runtime, corrupting its next burst sample.
    • When a task switches into fair_sched_class (e.g., SCHED_FIFO/SCHED_RR/SCHED_DEADLINE -> SCHED_NORMAL in switched_to_fair()), prev_sleep_sum_runtime is never resynced to sum_exec_runtime (which RT/DL also advance via update_curr_common() -> update_se()), so its first fair sleep attributes all prior RT/DL runtime as a single burst.
  4. §2 (Default command line) & §5 (Cross-Mac hardware evidence & clean regression run):
    • The J775c A/B row (c86e358c4a4c) was collected 370 commits behind integration/aurora, with non-default boot parameters/config (arm64.nomte console=tty0 console=ttyDC0, PCI SDHCI disabled), and reported 1 failed unit (ac-power-mode.service) and 3 new kernel errors (Gen4 SPMI IRQ + AOP). Under §2 and §5, re-prove on current integration/aurora tip on the default command line with 0 AVC denials, 0 failed units, and 0 new kernel errors.
    • Zero non-T6050 Mac testing: Across arch/arm64/boot/dts/apple/, M1/M2/M3/M4/A18 SoCs (t8103, t600x, t8112, t602x, t8122, t8132, t6040/t6041, t8140) all define opp-microwatt and run with EAS active (unlike t6050-base.dtsi, which currently has none). That means on every other Apple Silicon Mac, find_energy_efficient_cpu() preempts wake_short_local() whenever !is_rd_overutilized(rd) (so the feature won't fire for light ping-pong there, and will stop firing on T6050 once T6050 gains OPP power data), while it does fire whenever the root domain is overutilized — a regime that is completely untested on any asymmetric E/P Mac. Please run same-boot WAKE_SHORT_LOCAL on/off A/B (including hackbench -p -g 8 -l 10000, temperature-matched schbench p99, and unpinned ping-pong) on J775c and at least one M1/M2/M4 Mac with EAS state recorded.

Non-blocking Notes

  • __sched_fork() initializes burst_avg = 0, which is below limit (62.5 µs), and update_avg() uses a 1/8 EWMA step — meaning a newly created worker can run bursts up to ~500 µs before burst_avg exceeds limit and stops stacking onto a brief dispatcher. Worth checking under fan-out / thread-pool workloads.

Checklist Summary

  • §1 No model gating: Pass.
  • §2 Production-ready / default cmdline: Fail (J775c proof used arm64.nomte + disabled PCI SDHCI on a 370-commit-stale base).
  • §3 Safety: Pass.
  • §4 Clean-room & attribution: Pass (commit has human author/committer + human Signed-off-by: only; PR body discloses Assisted-by:).
  • §5 Evidence: Fail (1 failed unit + 3 kernel errors on stale base; zero non-T6050 M1/M2/M4 A/B; low-resolution hackbench and skewed initial die temps on schbench).
  • §6 Naming: Pass.
  • CI: host-tests passed on c86e358c4a4c.

Assisted-by: LLM (Gemini 4 Neon)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants