Conversation
ONNX Runtime sizes a session's intra-op thread pool at one thread per
physical core unless told otherwise, and keeps every thread busy while
the session runs. One model stream can then occupy most of a machine,
which leaves nothing for a second model or for the rest of the program.
ort exposes SessionBuilder::with_intra_threads; this passes it through
as an optional fourth argument to Ortex.load/4:
Ortex.load(path, [:cpu], 3, intra_threads: 4)
Unset, the session is built exactly as before, so ONNX Runtime's own
default and its ORT_INTRA_OP_NUM_THREADS environment fallback still
apply. A value that is not a positive integer, or an unknown option,
raises ArgumentError before the NIF is called.
Matching the whole validated keyword list meant a second option would break every clause. Read the key out of the validated options instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an
intra_threadsoption toOrtex.load/4, passed toort'sSessionBuilder::with_intra_threads.Why
ONNX Runtime gives a session one intra-op thread per physical core, so one
model stream can take most of a machine and starve a second model. Ortex has
no way to change that.
ORT_INTRA_OP_NUM_THREADSworks only from the OSenvironment at process start and applies to every session in the VM.
One stream, RF-DETR at 576x576, CPU provider, ONNX Runtime 1.30.0, 15-core
Apple M5 Pro:
Past 5 threads latency still falls, but each inference costs more CPU.
Relation to #50
#50 passes session config entries through. On ONNX Runtime 1.28+ the key
session.intra_op_num_threadssizes the pool. On the 1.19.2 runtime Ortexbundles, that key does not exist and is silently ignored. This PR calls
with_intra_threadsdirectly, so it works on every runtime Ortex links;checked by counting OS threads (
intra_threads: 1adds 0 workers, 6 adds 5).Both PRs take the fourth argument of
load/4. If #50 lands first I can rebasethis as one more key in the same keyword list.
Details
nil, the default, leaves the builder untouched; existing callers areunchanged.
nil, elseArgumentError.Keyword.validate!/2refuses an unknown option.
Cargo.lockunchanged.and a misspelt key.