Skip to content

Residency unification, disk GC, worker-side VRAM juggling (#369, #370, #371)#49

Merged
PaulFidika merged 2 commits into
masterfrom
issue/369-370-371-residency-diskgc-vram
Jul 4, 2026
Merged

Residency unification, disk GC, worker-side VRAM juggling (#369, #370, #371)#49
PaulFidika merged 2 commits into
masterfrom
issue/369-370-371-residency-diskgc-vram

Conversation

@PaulFidika

Copy link
Copy Markdown
Contributor

Closes tracker issues #369, #370, #371 (agents/completed.md).

#369 — residency registry owns the executor's pipelines

  • Per-slot allocator-delta measurement in _injection_kwargs: each worker-built pipeline is registered under its ref with its own measured vram_bytes AND the object (demote/promote actually move memory). Tenant-loaded refs (path-annotated setup slots) split the residual delta — multi-model endpoints no longer report vram_bytes=0.
  • Model loads serialize under Executor._load_lock — concurrent setups can no longer cross-contaminate allocator deltas or place_pipeline's free-VRAM read.
  • demote()/promote() only perform transitions they can actually execute (movable object + host-RAM headroom); otherwise the executor vacates the owning record and books every held ref back to disk — registry state and instance state cannot diverge.
  • Free-VRAM probes (residency + StateDelta) sum across ALL CUDA devices.
  • Bug found en route: residency events raised from worker threads were silently dropped when no loop had been captured (ModelStore.bind_loop).

#370 — keep-set disk retention (disk GC exists now)

  • Pre-download headroom gate in ModelStore.ensure_local (snapshot sizes are known up front): GC first, then ModelEvent{FAILED, insufficient_disk} + typed InsufficientDiskError (fails the job RETRYABLE without self-disabling the function).
  • LRU disk GC over Residency DISK-tier refs honoring keep + in-use pins + a 300s grace window; keep-pressure escape hatch evicts LRU keep refs too, still emitting EVICTED.
  • Deletion units in models/disk_gc.py: CAS snapshot dir + hardlink-count orphan-blob sweep; HF models--* repo dir.
  • Persisted ref-index.json + boot-time rescan wired into lifecycle.startup() so Hello.models matches disk truth across restarts.

#371 — worker-side VRAM juggling

  • ensure_setup runs make_room before loading (estimate: per-ref vram_hint, else declared resources.vram_gb); idle LRU pipelines demote to the warm RAM tier instead of the new load degrading down the offload ladder. Non-movable victims vacate via record teardown. Offload ladder remains the fallback.
  • Hub UNLOAD demotes instead of destroying (instance stays ready); the next RunJob/LOAD promotes RAM→VRAM in seconds.

Tests

uv run --extra dev pytest -q: 193 passed, 1 skipped (was 182+1s on master).
New: tests/test_executor_residency.py (6: per-ref measured bytes, residual attribution, concurrent-setup non-contamination, UNLOAD→RAM→promote round trip, tenant-loaded teardown, alternating demote/promote swap with IN_VRAM⇄IN_RAM events), tests/test_disk_gc.py (5: LRU non-keep eviction with real file deletion, keep escape hatch, fail-fast insufficient_disk, in-use pin, restart rescan).

@PaulFidika PaulFidika force-pushed the issue/369-370-371-residency-diskgc-vram branch from fe5a952 to 4592bae Compare July 4, 2026 08:43
…/promote juggling (#369, #370, #371)

- #369: per-slot allocator-delta measurement registers each ref with its own
  measured vram_bytes AND the pipeline object; tenant-loaded refs split the
  residual. Loads serialize under a load lock. demote()/promote() only perform
  transitions they can execute (movable obj + RAM headroom); record teardown
  books every held ref back to disk. Free-VRAM probes sum across all CUDA
  devices. ModelStore.bind_loop fixes dropped thread-side residency events.
- #370: pre-download disk headroom gate + LRU disk GC over DISK-tier refs
  (keep + in-use pins + grace, keep-pressure escape hatch emits EVICTED),
  persisted ref-index, boot-time rescan, fail-fast insufficient_disk.
- #371: make_room before load (idle LRU pipelines demote to warm RAM instead
  of the new load degrading down the offload ladder); hub UNLOAD demotes
  instead of destroying; next RunJob/LOAD promotes RAM->VRAM.
@PaulFidika PaulFidika force-pushed the issue/369-370-371-residency-diskgc-vram branch from 4592bae to ac4c9e3 Compare July 4, 2026 08:44
@PaulFidika PaulFidika merged commit 88a8daf into master Jul 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant