Use all available CPUs in CI instead of a fixed -j3 - #938
Merged
Conversation
GitHub's hosted Linux runners currently have four CPUs, and larger or self-hosted runners have more, so -j3 leaves capacity unused. Use $(nproc) instead of a fixed number. This covers the jobs which run on Linux hosts or in Linux containers, where nproc is available. The other platforms are handled separately.
macOS has no nproc, so query the CPU count with sysctl instead.
The BSDs have no nproc. Query the CPU count once with sysctl at the top of each VM script and reuse it, rather than repeating the command at every call site.
Solaris has neither nproc nor sysctl -n hw.ncpu, but getconf reports the number of online processors. The compilation steps are left serial, as they were before.
These steps run under PowerShell, so take the CPU count from NUMBER_OF_PROCESSORS rather than from a shell command.
A stray semicolon before && makes the line unparseable. The whole remote script is a single argument to "bash -c", which is parsed before anything is executed, so this stops the entire job rather than just that line: neither the autoconf nor the CMake build ever starts. The job only runs on workflow_dispatch, which is presumably why this has gone unnoticed.
Member
|
Thank you! These changes are a nice maintenance improvement. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every CI job hard-codes
-j3formakeandctest. GitHub's hosted Linux runners currently have four CPUs, and larger or self-hosted runners have more, so this leaves capacity unused.There is no portable way to ask for the CPU count, so each platform gets the spelling it supports:
$(nproc)$(sysctl -n hw.ncpu)NPROC=sysctl -n hw.ncpu`` once per VM scriptNPROC=getconf NPROCESSORS_ONLN``-j $env:NUMBER_OF_PROCESSORSOn the BSDs and Solaris the count is computed once at the top of the VM script and reused, rather than repeating the command at each call site.
One commit per platform, so the unusual ones can be reviewed or dropped individually.
No functional change: the same jobs run the same builds and tests.
Two things deliberately left alone:
ctest -j3. I don't know whethernprocexists in the zopen environment, and that line has a pre-existing syntax error (--output-on-failure; &&) which I've included a patch to fixctestchanges.