feat(nvsnap): download each model once per cluster into a shared volume for Helm functions - #2106
Open
balajinvda wants to merge 11 commits into
Open
balajinvda wants to merge 11 commits into
balajinvda wants to merge 11 commits into
Conversation
First step of docs/proposals/helm-shared-model-volume.md. For any GPU pod at admission, internal/modelid answers which artifact the pod will download, as a URI shared by every chart, version and namespace that names it, and where the bytes land and what backs that path today. Identity sources, as the field uses them: engine args (--model, --model-path, --model=, positional `vllm serve <x>`, --revision), engine env (HF_MODEL_ID, MODEL_ID, MODEL_PATH, with $VAR expansion from the container's own env), download init containers (NGC CLI with NGC_MODEL_NAME, huggingface-cli download with --local-dir, aws s3 sync, KServe storage-initializer args), NIM images with NIM_MODEL_PROFILE, and for LeaderWorkerSet workers that name no model the leader template of their group. URIs: hf://org/repo[@rev], ngc://org/team/model:ver, s3://bucket/key, nim://image@profile, path:///abs for pre-filled paths. The landing volume is the mount at or above the download destination in the main container: emptyDir or rootfs are substitutable, a customer's PVC, hostPath or other volume means sharing is already solved and the pod is left alone. KServe pvc:// is likewise not a downloader. Tests are built from real specs: the NVCF Helm function observed on prd11 (NGC init download, positional MODEL_PATH), the Dynamo operator sample, SGLang, NIM, KServe, huggingface-cli and S3 inits, the upstream LWS vLLM example, and non-downloaders (Dynamo frontend, Ray worker, etcd). Mutation-checked: PVC marked substitutable, revision split dropped and env expansion dropped each turn tests red. Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com>
Step 2 of docs/proposals/helm-shared-model-volume.md. For a GPU pod whose model identity resolves (internal/modelid), the webhook replaces the volume the download lands in with the per-identity model volume and turns the download into a write-once step; no pod is gated. internal/modelvolume names the claim per identity (nvsnap-model-<key>), creates the writer claim (ReadWriteMany on a distributed filesystem, ReadWriteOnce on NVMesh), looks the identity up cluster-wide and records completion as a label. The webhook elects the writer with the existing Lease elector keyed by identity; the writer's landing emptyDir becomes the claim, its engine mounts the model read-only, and its download init is wrapped to touch <volume>/.nvsnap-complete on success. Readers on a distributed filesystem mount the same claim and their init waits for the marker, downloading themselves after the deadline (decided: always fall back). Readers on NVMesh keep their emptyDir, are labelled pending and wait for the agent to bind the completed volume in (step 3). Charts whose engine downloads itself get an injected huggingface-cli init that lands the model in the volume, with the engine started offline and the registry credentials forwarded. Compile caches are redirected into <volume>/.nvsnap/cache/<engine key> on a distributed filesystem and into the local cachedir on NVMesh; the model env of the cachedir template is left alone because the model lives in the landing volume now. Tests use the prd11 function shape and a stock vLLM Deployment: writer on Block, pending reader on Block, reader on RWX, injected init with forwarded token and offline engine, completed volume skipping the election, and the customer-PVC / no-GPU / no-model pods left alone. Mutation-checked: read-only mount dropped, reader fallback dropped, Block reader substituting the volume, and writer marker dropped each turn tests red. Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com>
…Mesh
Step 3 of docs/proposals/helm-shared-model-volume.md. The agent gains a
model volume controller. For writers, on any node: when the download init
the webhook named on the pod exits 0, the writer claim is labelled
complete and, on block storage, SharedVolumePromoter.MintReadOnly exposes
it as a read-only claim (primary PV retained, secondary static PV with the
namespace-rewritten NVMesh handle, pre-bound claim). The writer claim
stays because the writer still runs on it and later writers must find it
complete. For pending readers on this node: once the identity is complete
the read-only claim is minted in the reader's namespace, attached to the
node through a mount-holder pod, bind-mounted read-only onto the reader's
hostPath landing under the Bidirectional overlays root
(<overlay-root>/models/<key>), and the pod is un-pended. The writer touched
the marker inside the volume, so the reader's wait init sees it as soon as
the bind lands. Every agent watches writers (idempotent marking and
minting); only the reader's own agent binds.
The webhook gives Block-mode readers a hostPath landing with
HostToContainer propagation on the engine and init mounts, and names the
writer's download init for the agent. Storage profiles gain
`modelVolume: {mode, storageClass, size}`; block is the default for
shared-volume strategies, rwx must be declared. Agent flags
--model-volume and --model-volume-wait-deadline, Helm
agent.modelVolume.{enabled,waitDeadline}, and pods patch in the agent
ClusterRole.
Tests with fake clients: writer completion marks and mints the read-only
PV and claim with the handle rewritten and the primary retained; a
pending reader on its node gets the claim in its own namespace, one
attach, one bind at the hostPath, and is un-pended, a second reader of
the same identity reuses the bind, readers on other nodes are ignored.
Mutation-checked: node filter dropped, completion on any exit code,
binding before completion, and un-pend dropped each turn tests red.
Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com>
…b in huggingface_hub 1.x Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com>
Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com>
The first cluster run of the model volume on NVMesh exposed a constraint the design had missed: a volume attached read-write by a running pod cannot be attached read-only anywhere else (AttachVolume failed on the read-only PV while the writer pod held the primary). The L2 promote never met it because it deletes the writer claim before readers attach. So the download step cannot live inside a pod that goes on to serve. The download is now a Job per identity per cluster, nvsnap-model-dl-<key>, created idempotently by the webhook on first sight: the chart's own download init (image, command, env, pull secrets, tolerations) wrapped to touch the marker, or `hf download` on the engine image when the engine fetches the model itself. Its exit releases the volume. Every workload pod is a reader; there is no writer pod and no Lease election on this path, because Job create is atomic. The agent completes the identity when the Job succeeds and mints the read-only claim on block storage; readers are bound as before. The design doc records the constraint and the reason. Tests: Job spec derived from the NGC init and from a stock vLLM pod (hf download, forwarded token, claim mounted at the init's path), idempotent create, no Job once the volume is complete, controller completes on Job success only. Mutation-checked: Job never created, Job created when complete, and completion on a running Job each turn tests red. Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com>
…detaches A Succeeded pod keeps its volumes attached. On NVMesh that let the read-only attach succeed only on the Job's own node (dev1 2026-09-26); every other node failed until the pod was deleted. The Job now carries ttlSecondsAfterFinished=30; completion state lives on the claim label. MarkComplete tolerates the update race between agents by re-reading. Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com>
…a dedicated host root Two findings from the second dev1 run of the model volume on NVMesh. The retained volume must have no read-write attachment before any node can attach it read-only: with the Job's claim still bound, the read-only attach succeeded only on the Job's own node and failed everywhere else. On block storage MarkComplete now labels the retained PV with identity and completion, sets Retain, and deletes the download claim, the same release the L2 promote performs; Lookup reads completion from the PV; readers' read-only claims are minted from that PV, and only after no VolumeAttachment references it (Detached). RWX mode keeps the shared claim and labels it as before. The bind root moves out of the overlays root: the overlay sweeper removes entries it does not own and deleted the model binds under it. Completed model volumes are bound under a dedicated Bidirectional hostPath (agent.hostPaths.nvsnapModels, default /var/lib/containerd/nvsnap-models), mounted at the same path in the agent and on the host. Tests: block completion releases the claim and labels the PV, RWX labels the claim, the reader is not served while a VolumeAttachment exists and is served once it is gone (mutation-checked: dropping the detach gate turns the test red), the read-only PV carries the reader namespace's handle. Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com>
…on dev1 Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com>
…d when stale The controller kept "bound" in memory. After the volume for an identity was replaced (and after a manual unmount during cleanup on dev1) it un-pended readers without a bind and they waited on an empty hostPath. The mount table is now the truth: a reader is served only when the device mounted at the bind target belongs to the identity's primary PV (NVMesh device csi-id against the volume handle); a missing mount is bound again, a mount of a replaced volume is unbound and redone, and a bind that leaves nothing mounted does not un-pend the pod. Tests: rebinding after an unmount, replacing a stale bind, device/handle matching. Mutation-checked: dropping the stale unbind and the missing mount check each turn tests red. Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com>
Co-Authored-By: Balaji Ganesan <bganesan@nvidia.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: NVIDIA/nvcf/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Customers deploy Helm charts of N identical GPU workers, and every worker
downloads and compiles on its own. The gate-and-promote election in #2104
cannot serve multi-node instances (LWS, StatefulSet groups, Dynamo podgangs):
holding any member deadlocks the group. The invariant we want is simpler
and storage-independent: the model is downloaded once per cluster and every
pod attaches it; compile caches are shared the same way. Design and the
decisions taken with the product owner:
docs/proposals/helm-shared-model-volume.md.What changed
internal/modelid: identity and landing volume for any chart. Identity asa URI (
hf://,ngc://,s3://,nim://) from engine args (includingpositional
vllm serve <x>and$VARexpansion), engine env, downloadinit containers (NGC CLI,
hf download, S3, KServe storage-initializer),NIM images, and LeaderWorkerSet workers via the leader template. Landing
volume kind decides substitutability; a customer's PVC, hostPath or image
is left alone. Tests from the prd11 NVCF function, the Dynamo sample,
KServe, NIM and the upstream LWS example.
internal/modelvolume+ webhook: one download Job per identity percluster (
nvsnap-model-dl-<key>, idempotent create, so no election), builtfrom the chart's own download init wrapped to touch a marker, or from
hf downloadon the engine image with credentials forwarded. Everyworkload pod is a reader: its download init becomes a wait for the marker
(deadline, then self-download), the engine is started offline when it used
to download itself. RWX mode mounts the shared claim; Block mode (NVMesh)
gives readers a hostPath landing with HostToContainer propagation.
ModelVolumeController: on Job success labels the retained PVcomplete and releases the claim (NVMesh refuses read-only attaches while
a read-write attachment exists, including a Succeeded pod's); for pending
readers on its node, once the volume is detached, mints the read-only PV
and claim in the reader's namespace, attaches through a mount-holder,
bind-mounts under a dedicated Bidirectional hostPath and un-pends. Binds
are verified against the mount table and the volume handle, and redone
when missing or stale.
modelVolume: {mode, storageClass, size}; block bydefault on shared-volume strategies, rwx must be declared. Helm
agent.modelVolume.{enabled,waitDeadline},agent.hostPaths.nvsnapModels;agent RBAC gains PVC update/patch and pods patch.
Customer Release Notes
Helm chart functions download each model once per cluster; every other
worker, later version, scale-up and redeploy attaches the downloaded volume.
No chart change. Requires NVMesh or a distributed filesystem.
Plan Summary
Helm: new values (off by default), one extra Bidirectional hostPath on the
agent DaemonSet, ClusterRole verbs. No new controllers or CRDs.
Usage
Testing
dev1, 2026-09-26, NVMesh, stock
vllm-workerschart (Qwen2.5-32B TP=4,two replicas on two nodes, no nvsnap markers):
Unit: identity forms and classifier; Job spec derivation and idempotence;
webhook matrix (Block and RWX readers, injected download, complete state,
customer volumes left alone); controller completion, release, detach gate,
per-namespace read-only minting, rebind on missing or stale mounts, no
un-pend without a mount. All mutation-checked as described in the commits.
Notes
Stacked on #2104 (base
nvsnap/helm-election). Open items in the designdoc: compile caches on block storage (cache-only capture),
cacheModeandthe Lustre flock check, retention including unbinding when the last reader
leaves a node and holder cleanup, and the RWX path needs a cluster with a
distributed filesystem for e2e (dev1 has none).
References
Relates to #2099
Related Pull Requests
#2104, #2101
Dependencies
None