Residency unification, disk GC, worker-side VRAM juggling (#369, #370, #371)#49
Merged
Merged
Conversation
fe5a952 to
4592bae
Compare
…/promote juggling (#369, #370, #371) - #369: per-slot allocator-delta measurement registers each ref with its own measured vram_bytes AND the pipeline object; tenant-loaded refs split the residual. Loads serialize under a load lock. demote()/promote() only perform transitions they can execute (movable obj + RAM headroom); record teardown books every held ref back to disk. Free-VRAM probes sum across all CUDA devices. ModelStore.bind_loop fixes dropped thread-side residency events. - #370: pre-download disk headroom gate + LRU disk GC over DISK-tier refs (keep + in-use pins + grace, keep-pressure escape hatch emits EVICTED), persisted ref-index, boot-time rescan, fail-fast insufficient_disk. - #371: make_room before load (idle LRU pipelines demote to warm RAM instead of the new load degrading down the offload ladder); hub UNLOAD demotes instead of destroying; next RunJob/LOAD promotes RAM->VRAM.
4592bae to
ac4c9e3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes tracker issues #369, #370, #371 (agents/completed.md).
#369 — residency registry owns the executor's pipelines
_injection_kwargs: each worker-built pipeline is registered under its ref with its own measuredvram_bytesAND the object (demote/promote actually move memory). Tenant-loaded refs (path-annotated setup slots) split the residual delta — multi-model endpoints no longer reportvram_bytes=0.Executor._load_lock— concurrent setups can no longer cross-contaminate allocator deltas orplace_pipeline's free-VRAM read.demote()/promote()only perform transitions they can actually execute (movable object + host-RAM headroom); otherwise the executor vacates the owning record and books every held ref back to disk — registry state and instance state cannot diverge.ModelStore.bind_loop).#370 — keep-set disk retention (disk GC exists now)
ModelStore.ensure_local(snapshot sizes are known up front): GC first, thenModelEvent{FAILED, insufficient_disk}+ typedInsufficientDiskError(fails the job RETRYABLE without self-disabling the function).keep+ in-use pins + a 300s grace window; keep-pressure escape hatch evicts LRU keep refs too, still emitting EVICTED.models/disk_gc.py: CAS snapshot dir + hardlink-count orphan-blob sweep; HFmodels--*repo dir.ref-index.json+ boot-time rescan wired intolifecycle.startup()so Hello.models matches disk truth across restarts.#371 — worker-side VRAM juggling
ensure_setupruns make_room before loading (estimate: per-refvram_hint, else declaredresources.vram_gb); idle LRU pipelines demote to the warm RAM tier instead of the new load degrading down the offload ladder. Non-movable victims vacate via record teardown. Offload ladder remains the fallback.UNLOADdemotes instead of destroying (instance stays ready); the next RunJob/LOAD promotes RAM→VRAM in seconds.Tests
uv run --extra dev pytest -q: 193 passed, 1 skipped (was 182+1s on master).New:
tests/test_executor_residency.py(6: per-ref measured bytes, residual attribution, concurrent-setup non-contamination, UNLOAD→RAM→promote round trip, tenant-loaded teardown, alternating demote/promote swap with IN_VRAM⇄IN_RAM events),tests/test_disk_gc.py(5: LRU non-keep eviction with real file deletion, keep escape hatch, fail-fast insufficient_disk, in-use pin, restart rescan).