Repository navigation
Add ResourceUsage to report subprocess CPU time and memory consumption - #226
Open
jakepetroules wants to merge 4 commits into
Open
jakepetroules wants to merge 4 commits into
jakepetroules wants to merge 4 commits into
Conversation
jakepetroules
force-pushed
the
eng/PR-resource-usage
branch
2 times, most recently
from
March 17, 2026 17:39
a8f9745 to
66ae4c3
Compare
compnerd
reviewed
Mar 18, 2026
jakepetroules
commented
Mar 22, 2026
jakepetroules
force-pushed
the
eng/PR-resource-usage
branch
2 times, most recently
from
April 13, 2026 19:02
a357701 to
ed3aa46
Compare
jakepetroules
force-pushed
the
eng/PR-resource-usage
branch
from
May 13, 2026 17:05
ed3aa46 to
9630762
Compare
jakepetroules
force-pushed
the
eng/PR-resource-usage
branch
from
May 21, 2026 07:19
9630762 to
d173b7d
Compare
jakepetroules
force-pushed
the
eng/PR-resource-usage
branch
from
June 3, 2026 23:15
d173b7d to
006121c
Compare
jakepetroules
force-pushed
the
eng/PR-resource-usage
branch
from
June 11, 2026 22:17
006121c to
578e47a
Compare
jakepetroules
force-pushed
the
eng/PR-resource-usage
branch
4 times, most recently
from
August 9, 2026 02:49
abd8399 to
0211f39
Compare
jakepetroules
force-pushed
the
eng/PR-resource-usage
branch
from
October 2, 2026 00:52
0211f39 to
a7cdef9
Compare
jakepetroules
force-pushed
the
eng/PR-resource-usage
branch
from
October 14, 2026 05:49
05defd1 to
5b525fc
Compare
Introduce a public ResourceUsage struct exposing userTime and systemTime (as Duration) and maxRSS (peak resident set size, in bytes) for every terminated subprocess, surfaced as ExecutionResult.resourceUsage. Collecting this is something only Subprocess can do: the data has to be read after the child has become a zombie but before its pid is reaped, and Subprocess owns the reap. Doing it for an arbitrary pid from outside is notoriously hard to do portably without also reaping the pid. User time, system time, and maxRSS are the most commonly used getrusage metrics; on Unix the raw rusage struct is also exposed as ResourceUsage.rusage for everything else, and on Windows callers can use DuplicateHandle to get a process handle that outlives Subprocess's ownership of the process and query further metrics from it once it is known to have terminated. Unix: the child is now reaped with wait4 on every platform, which is the only interface every supported libc exposes that reports rusage alongside the wait status. It is reached through a _subprocess_wait4 C shim because musl declares wait4 only under _GNU_SOURCE/_BSD_SOURCE, so it isn't reliably visible through every Swift overlay. Wrapping the libc entry point, rather than issuing the Linux 5-argument waitid syscall, also keeps libc's struct rusage layout in play: the kernel's __kernel_old_timeval fields disagree with libc's wherever time_t is 64-bit on a 32-bit architecture. TerminationStatus is now decoded from the wait status rather than siginfo_t, and the unused non-blocking reap() is removed. maxRSS is normalized from KiB to bytes on Linux, Android, FreeBSD, and OpenBSD, saturating rather than trapping if a peak above 2 GiB is reported on a 32-bit platform. Windows: GetProcessTimes provides CPU time (converted from FILETIME 100ns units) and K32GetProcessMemoryInfo provides PeakWorkingSetSize, both read while the process handle is still open. If either query fails, its fields are left zero rather than failing the reap, since the exit status is the caller's primary result. ResourceUsage is Hashable, but equality and hashing consider only userTime, systemTime, and maxRSS: rusage is a C struct that conforms to neither protocol, and retroactively conforming a type this package doesn't own would collide with anyone else doing the same. Result.swift deliberately doesn't import _SubprocessCShims. On some platforms (notably glibc) that would attribute rusage to the shims module rather than the libc overlay, and ResourceUsage.rusage could then not be public without publicly re-exporting an underscored module. The conversions that need C struct members in scope therefore live in Subprocess+Unix.swift and Subprocess+Windows.swift, next to the reaping code that calls them.
Switching the reap to wait4 truncated exit codes to 8 bits. A wait-family wait status holds only the low 8 bits of the value a child passes to exit, but Darwin and FreeBSD preserve the full int and report it through waitid's siginfo, so a child exiting 0x12345 was reported as .exited(0x45) where main reports .exited(0x12345). Reported by Jonathan Grynspan on the pitch thread. Reap in two steps instead. A blocking waitid(WEXITED | WNOWAIT) reads the termination status from siginfo, exactly as main did, without consuming the zombie; wait4 then reaps it and supplies the rusage. Subprocess is the only thing that reaps this child, so both calls observe the same zombie. This restores TerminationStatus.init(_ siginfo:) from main, and drops the wait-status decoding that only wait4 needed. On Linux, where the kernel keeps only 8 bits regardless, this costs one extra non-blocking system call per reap. FreeBSD's wait6 could return both in one call, but that is an optimization that can come later. The regression test runs Perl, because a shell truncates its own exit status to 8 bits; Perl keeps 16, which is enough to tell the two apart.
Respond to the review on the pitch thread: - Drop the raw `rusage` member (Franz Busch). Exposing a C struct put a platform-conditional member in the public API, and it was the only reason ResourceUsage needed hand-written equality and Result.swift needed to avoid importing _SubprocessCShims. The rusage fields that mean the same thing on every Unix platform become typed properties instead: minor and major page faults, and voluntary and involuntary context switches. Block I/O is left out because Linux derives ru_inblock/ru_oublock from bytes in 512-byte units while the BSDs count operations. - Rename maxRSS to peakMemoryUsage (Franz Busch). It's still bytes: peak resident set size on Unix, peak working set size on Windows. - Make every property optional, and nil rather than zero when a platform doesn't report a measurement or querying it fails (Franz Busch). As BlueSparrow pointed out, zero can be a real measurement. On Windows that covers a failed GetProcessTimes or K32GetProcessMemoryInfo, and the page fault and context switch properties, which have no direct equivalent there. - Make the properties `var` and add a public memberwise initializer whose parameters default to nil (Franz Busch), so callers can construct values, for example in tests, and fields can be added later without breaking them. - Drop Equatable and Hashable from ResourceUsage (Jordan Rose, Steve Canon). No two runs measure alike, so equality over a whole measurement has no use; individual Duration and Int fields remain comparable. ExecutionResult synthesizes Equatable and Hashable, which requires every stored property to conform. Rather than give that up, which would be a breaking change, it stores the measurement in an internal _ExcludedFromEquality wrapper whose instances all compare equal, and exposes it through a computed `resourceUsage`. ExecutionResult's equality is therefore exactly what it was before this change: resource usage describes how an execution ran, not what it produced (Tony Allevato). ExecutionOutcome does the same for its internal copy. Reading resourceUsage through a computed property consumes the whole noncopyable ExecutionOutcome, so API.swift now reads it before `value`.
FreeBSD samples resident memory on clock ticks rather than tracking a high-water mark on every fault. dd copying its 32 MiB block once exits before enough ticks land, and CI measured 15.7 MB against the 16 MiB the test requires. Copy the block 64 times instead, which keeps the buffer resident for on the order of 100 ms.
jakepetroules
force-pushed
the
eng/PR-resource-usage
branch
from
October 21, 2026 08:27
47836f7 to
178141d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
ExecutionResult.resourceUsage: the CPU time, peak memory, page faults, and context switches of every terminated subprocess, for everyrunoverload.Pitch: Subprocess: Reporting the CPU time and peak memory of terminated child processes
Why Subprocess has to do this
The kernel hands back a child's resource accounting through the same call that reaps it, so the data exists only between the child's exit and its reap. Subprocess owns that window: monitoring and reaping happen inside
run(), and the PID is gone by the time it returns.getrusage(RUSAGE_CHILDREN)is a running total over every child the process has waited for, so it can't be attributed to one child when several run concurrently. Reaping the child yourself takes it away from Subprocess, which then can't report its termination status.API
nilwhen the platform doesn't report that measurement or querying it failed. Zero is a real measurement, never a placeholder.ResourceUsageis deliberately notEquatableorHashable: no two runs measure alike. IndividualDurationandIntproperties remain comparable.ExecutionResultsynthesizesEquatableandHashable, which requires every stored property to conform. It stores the measurement in an internal wrapper whose instances all compare equal and exposes it through a computed property, soExecutionResult's equality is unchanged by this PR andresourceUsagetakes no part in it. Removing those conformances instead would be a breaking change.userTime,systemTimepeakMemoryUsagenilImplementation
Unix. The reap takes two calls, because no interface on every supported platform reports both pieces completely:
waitid(WEXITED | WNOWAIT)reads the termination status fromsiginfo_t, exactly asmaindoes, without consuming the zombie. This preserves the fullintthe child passed toexitwhere the kernel keeps it (Darwin, FreeBSD).wait4then reaps the zombie and supplies therusage. It's the only interface every supported libc exposes that reportsrusagefor a specific child. Its wait status is discarded, since it holds only the low 8 bits of the exit code.Subprocess is the only thing that reaps this child, so both calls observe the same zombie.
wait4is reached through a_subprocess_wait4C shim, because musl declares it only under_GNU_SOURCE/_BSD_SOURCE. Wrapping the libc entry point, rather than issuing Linux's 5-argumentwaitidsyscall, also keeps libc'sstruct rusagelayout in play: the kernel's__kernel_old_timevalfields disagree with libc's wherevertime_tis 64-bit on a 32-bit architecture.ru_maxrssis normalized from KiB to bytes on Linux, Android, FreeBSD, and OpenBSD, saturating rather than trapping if a peak above 2 GiB is reported on a 32-bit platform.Windows.
GetProcessTimesandK32GetProcessMemoryInfoare read just beforeGetExitCodeProcess, while the process handle is still open. If either fails, its properties arenilrather than failing the reap, since the exit status is the caller's primary result.Commits
wait4alone.wait4truncated exit codes to 8 bits, so a child exiting0x12345was reported as.exited(0x45)on Darwin and FreeBSD. Adds a regression test.rusagemember in favor of typed properties, renamesmaxRSStopeakMemoryUsage, makes properties optionalvars with a public initializer, and removesEquatable/HashablefromResourceUsage.Commit 1 on its own contains the exit-status regression, so this should be squash-merged or have 1 and 2 squashed before merging.
Open issue: peak memory on Linux
Since #365, Subprocess spawns with
posix_spawnon Linux, which glibc implements withclone(CLONE_VM | CLONE_VFORK); musl does the same. Atexec, the kernel folds the high-water RSS of the address space being replaced into the child'smaxrss(exec_mm_put_oldinfs/exec.c). Because that address space is the parent's, every child'sru_maxrssis at least the parent's peak RSS. CI shows it: a 1 KiB and a 32 MiBddboth report the test runner's ~80 MB. CPU times, page faults, and context switches are unaffected.testResourceUsagePeakMemoryUsageScalesWithAllocationcurrently fails on Linux because of this, and should keep failing until it's resolved. Options are being discussed on the pitch thread.Testing
testExitCodeWiderThanEightBits(Darwin, FreeBSD): runsperl -e 'exit 0x1234'and expects.exited(0x1234); fails without commit 2. A shell can't be used because it truncates its own exit status to 8 bits.rusageconversion, including KiB scaling, the initializer defaults, andExecutionResultequality ignoring resource usage.