Repository navigation
Conversation
… thread counts are 1 under MPI Under MPI with more than one rank, import proteus (and proteus.Comm, which knows the real rank count) sets OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, MKL_NUM_THREADS and VECLIB_MAXIMUM_THREADS to 1 unless the user set them. A serial run, or a single rank, is untouched and may use threads. OPENBLAS_NUM_THREADS alone is not enough: conda-forge's OpenBLAS is built on OpenMP and ignores it (13.8 cores busy in one process with only that set; one with OMP_NUM_THREADS=1). The variables are read at library load, so when numpy was imported before proteus, Comm limits the loaded libraries through threadpoolctl if it is installed (added to the dev environments) and logs a warning if not. Tests cover the rules on a fake environment, import proteus under each launcher variable, and real mpiexec ranks measuring BLAS CPU time per wall time with proteus first and with numpy first. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… the ranks' environment clean_environment dropped every OMPI_*, PMIX_* and PMI_* variable, including CI's OMPI_MCA_* settings that keep Open MPI off UCX; two-rank launches then failed to create a UCP worker and hung until the timeout on linux-64 and linux-aarch64. Only the launcher's size announcements are removed now. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…r until it has enough points It used the module's shared generator and 40 draws. Where that generator stood depended on which tests the xdist worker had run first, and for n=3 the peak is narrow enough that one linux-aarch64 run got 4 points above the cutoff and failed 'assert checked >= 5'. A generator of its own, and draws until 8 points are above the cutoff, make it independent of order. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Under MPI with more than one rank,
import proteussetsOMP_NUM_THREADS,OPENBLAS_NUM_THREADS,MKL_NUM_THREADSandVECLIB_MAXIMUM_THREADSto 1 unless the user already set them. A serial run, or one rank, is untouched, so the no-MPI mode keeps whatever threads the libraries choose (as decided: threads stay supported there).proteus/_threads.pyholds the rules;proteus/__init__.pyapplies them from the launcher's announced size (Open MPI, MPICH/Hydra, Slurm, PMIx), andproteus/Comm.pyapplies them again from the real communicator size.Why
C30 baseline, section 6.2 of
ON_NODE_PERFORMANCE_BASELINE.md: the legacy kernels are limited by instruction issue on one core, and OpenBLAS starts a thread per core in every rank (16 per rank on the M4 Max), which oversubscribes a node run one rank per core.Two things the measurement showed
OPENBLAS_NUM_THREADS=1alone does nothing on conda-forge's OpenBLAS (built on OpenMP): 13.8 cores busy in one process with only it set, 1.0 withOMP_NUM_THREADS=1. All four variables are set. The baseline doc's wording named only OpenBLAS; corrected there.Commthen limits the loaded libraries throughthreadpoolctlif installed (added to the three dev environments) and logs a warning if not.Tests (
test/test_thread_defaults.py, 14)Rules on a fake environment;
import proteusunder each launcher variable, one rank, and a user value; and realmpiexec -n 2ranks measuring BLAS CPU time over wall time (proteus first, numpy first, one rank, user value kept). Negative control: with the runtime limit disabled the numpy-first test fails at 8.2.Not in this PR
The generated Kokkos kernels will read
OMP_NUM_THREADStoo, so rank-per-core stays the default there and hybrid runs set it explicitly.🤖 Generated with Claude Code