Repository navigation
Conversation
Set enable_mem_pattern=False and enable_cpu_mem_arena=False on SessionOptions for both YoloX and Detectron2 ONNX sessions. These flags control pre-allocation strategies that trade memory for speed on repeated inference. With both disabled, peak memory drops ~36% (553→351 MB) on the YoloX model with negligible latency impact.
|
The review correctly identified that sessions are cached via However, the idle memory savings are also real and substantial: the arena holds ~209 MB of pre-allocated memory that persists after inference completes. With it disabled, the system allocator returns that memory. Peak RSS during inference is also modestly lower (−50 MB). The tradeoff is: ~15% higher inference latency vs. ~209 MB lower idle RSS per model session. |
128436c to
040a460
Compare
Replace unconditional disable of enable_mem_pattern and enable_cpu_mem_arena with opt-in via ONNX_DISABLE_MEMORY_ARENA=1. Default behavior is unchanged (arena enabled, ~15% faster inference). Setting the env var disables both options, saving ~209 MB idle RSS per session at a ~15% latency cost. Env var is read once at module init time.
040a460 to
098b9ef
Compare
098b9ef to
1916f8b
Compare
1916f8b to
8b406d5
Compare
|
Superseded by #532, which rebases this onto current |
Disable
enable_mem_patternandenable_cpu_mem_arenaonSessionOptionsfor both YoloX and Detectron2 ONNX sessions.By default ONNX Runtime pre-allocates a memory arena and traces allocation patterns from the first
Run()to reuse memory on subsequent calls. This trades higher idle memory for faster repeated inference. For our use-case (one model loaded per worker, infrequent re-inference on the same session), the arena and pattern buffer are mostly wasted — they keep ~200 MB of pre-allocated native memory alive between requests.With both disabled, idle session memory drops significantly with negligible latency impact on inference.
Benchmark
Measured with memray (
memray run+memray stats --json), 3 inference iterations per configuration, on Apple M3 Max / Python 3.12. Uses the actualyolox_l0.05.onnxmodel with 1700x2200 input (letter page at 200 DPI).Reproduce
bench_onnx_mem_pattern.py