Repository navigation
[Bug]: RunInference ModelManager NNLS model approximates different data points #40636
Description
What happened?
It appears that the NNLS approximation is done over data points that sometimes represent the deltas in the memory usage, and sometimes the total currently used memory. This can reduce the quality of the approximation / prediction of future memory usage.
All content below , including Suggested Fix, is LLM-generated:
ResourceEstimator receives inconsistent observations (delta vs. total memory)
ModelManager._spawn_new_modelcallsadd_observation({tag: 1}, peak_during_load - baseline_snap), which records a delta.ModelManager.release_modelcallsadd_observation(snapshot_of_all_models, peak_during_job), which records total GPU memory.ResourceEstimator._solveadds a+1bias column to every row. So delta rows saycost + bias ≈ delta, while total rows sayΣ count·cost + bias ≈ total. The NNLS system contradicts itself.- When only one model is loaded, both kinds of reading land in the same
history[(("tag", 1),)]bucket. - The spawn delta is also noisy in the concurrent path.
peakis a 30s sliding window that isn't reset before the load, so it can include earlier spikes.
Impact: per-model memory estimates are skewed, so admission control and eviction decisions can be wrong (overcommit → OOM, or needless waiting/eviction).
Suggested fix
Feed the estimator only total-memory observations with a consistent model snapshot, or drop the spawn-time observation.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Infrastructure
- Component: Spark Runner
- Component: Flink Runner
- Component: Prism Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
Activity
tvalentyn commentedon Oct 16, 2026
cc: @AMOOOMA
schizophrenicmaniac commentedon Oct 16, 2026
.take-issue
Hi, I'd like to work on this. _spawn_new_model passes a load delta to add_observation while release_model passes total peak memory, so I'd drop the spawn-time delta and only feed the estimator total-memory readings with the full model snapshot, plus a test in model_manager_test.py.
alisha-1000 commentedon Oct 22, 2026
Hi @tvalentyn,
I reviewed the issue and the proposed fix in #40646. The inconsistency between delta-based and total-memory observations is an important edge case for the resource estimator.
I'd be interested in helping validate the fix, particularly around concurrent model loading, memory estimation accuracy, and admission-control behavior. Are there any additional edge cases or regression tests that would benefit from investigation alongside the existing PR?
Thank you.
tvalentyn commentedon Oct 22, 2026
@alisha-1000 are you using multiple identities?
alisha-1000 commentedon Oct 22, 2026
Hi @tvalentyn,
I'm contributing through my GitHub account,
@alisha-1000. Could you clarify what prompted the question about multiple identities? I’d be happy to clarify any confusion.Thank you.
tvalentyn commentedon Oct 22, 2026
sorry, I misread the above comment as: "I reviewed the issue and proposed the fix in...", but the fix was from someone else, hence the question.
tvalentyn commentedon Oct 22, 2026
you can check the conversation in that PR if you are interested.