feat(nvsnap): elect one downloader per cache hash at admission

[balajinvda]

Why

Customers predominantly deploy Helm charts, and a chart creates N identical
model workers. Today each one downloads and initialises on its own: the pod
template cannot name a capture hash, restore-from: auto never made a pod a
capture source, and the watcher's hash never matched admission's. Measured
on dev1 with a TinyLlama TP=2 Deployment, both replicas were admitted "no
capture for hash; admitting pod unchanged" and the scale-up pod cold-started.

The goal, in two sentences: if the hash exists, the pods that would download
the model get the cache volume instead. If it does not, exactly one of them
downloads and the rest get the volume once it exists.

What changed

The webhook decides per pod, with no template change.

• Classifier: a pod is a downloader when it requests a GPU and its main
  container names a model (--model/--model-path in one arg or as
  Dynamo's separate list items, --model=, HF_MODEL_ID/MODEL_ID/
  NIM_MODEL_NAME, or a NIM image). Frontends, routers, etcd, nats are
  left alone. Covers stock vLLM/SGLang and vLLM/SGLang under Dynamo.
• Hash once: composed from the incoming spec, stamped on the pod
  (annotation full, label short). The watcher and orchestrator capture
  under the stamped hash instead of recomposing from the live pod.
• Promoted capture exists: cachedir restore decoration.
• Otherwise elect: Lease nvsnap-capture-<hash> created with an id the
  webhook mints (pods have no UID at CREATE admission). The one admission
  that succeeds is the leader and gets the capture decoration plus the
  capture label. Every other pod is a follower: restore decoration against
  the deterministic rox-<hash> claim, a schedulingGate (no node, no GPU
  held), a gated label.
• nvsnap-server releases the gates when the agent posts pvc-state=ready
  and deletes the Lease. On failed, a leader pod that is gone or
  terminated, or a Lease past its deadline, the reconciler evicts gated
  followers that have a controller so they are recreated and re-elected.
• Followers need ReadOnlyMany storage; on per-pod-clone storage they start
  cold. Any error fails open.
• Off by default: agent.election.enabled, agent.election.deadline
  (60m). Server RBAC gains leases get/list/watch.

Design: docs/proposals/helm-chart-cache-election.md.

Claims across namespaces

NVCF runs every chart in its own namespace and a PVC is namespaced. The
promoter gains EnsureClaim(hash, ns): shared-volume mints one more
secondary static PV per consumer namespace with the CSI handle rewritten for
it, pre-bound to a rox-<hash> claim there (NVCA's pattern, zero copy);
snapshot-clone with ReadOnlyMany re-exposes the promote's snapshot handle in
the target namespace through a pre-provisioned VolumeSnapshotContent +
VolumeSnapshot and clones it; per-pod clone reports unsupported and the pod
stays cold. Mount mints on a miss; the promote mints for every namespace
that already holds stamped pods before publishing ready. Delete reaps by
label. Agent RBAC gains volumesnapshotcontents.

Customer Release Notes

Helm chart functions with several identical GPU model workers download and
initialise the model once; the other workers start from that worker's
cache, and a redeploy or scale-up of the same chart starts every worker
from the cache. Requires a ReadOnlyMany storage class for the cache.

Plan Summary

Helm: new agent.election values (off by default), --election flags on
the agent DaemonSet, server ClusterRole leases verbs. No new objects.

Usage

helm upgrade nvsnap deploy/helm/nvsnap --set agent.election.enabled=true
kubectl get lease -n nvsnap-system -l nvsnap.io/kind=capture-election
kubectl get pods -l nvsnap.io/gated=true -A

Testing

dev1, 2026-09-25, stock TinyLlama TP=2 Deployment with no nvsnap markers,
agent v0.2.75-election2, server v0.0.32-election2, NVMesh:

install replicas=2      leader + gated follower decided in the same second
leader Ready            70s
followers released      +124s (server: "capture promoted; followers released" released=1)
follower Ready          53s after release, other node, 0 downloads, serves " Paris."
scale to 3              third pod role=restore, no election, Ready 53s
uninstall + reinstall   both role=restore, Ready 50s
cold, same node         94s

Cross-namespace, capture only in nvsnap-system, same stock chart in
nvsnap-xns with two replicas admitted in the same second:

webhook   "L2 claim minted in restore namespace" then both "restoring"
claim     nvsnap-xns/rox-<hash> Bound to a per-namespace PV, handle rewritten for nvsnap-xns
pods      both role=restore, Ready 60s, 0 downloads, serve " Paris."

Dynamo sample (frontend + prefill + decode, Qwen3-0.6B): both workers hash
the same, prefill leader, decode follower, frontend ignored. The gang did
not schedule under kai-scheduler (see Notes).

Unit: model-identity forms and the classifier; Lease election (one leader
of four, uid-less pods, error surfacing); webhook matrix (frontend ignored,
leader, gated follower with wait/seed/prewarm inits, cold fallbacks,
restore without election, unbound capture left alone, explicit
restore-from bypass, metadata maps bootstrapped once); server release,
eviction of owned pods only, leader liveness; orchestrator and watcher
honouring the stamped hash. Mutation-checked: dropping the gate, the
leader label, the single bootstrap, the restore-before-elect branch and
the stamped-hash override each turned tests red after compiling.

Notes

Stacked on #2101 (base is nvsnap/helm-chart-support); merge that first.

Disaggregated workers: Dynamo prefill and decode workers differ by a role
flag and the etcd/NATS/DYN_* wiring. The hash leaves those out
(stripRoleFlags), so both roles share one leader and one rox; compile
caches recompile into the per-pod writable shadow where they differ. Flags
that change the download (--model, --revision, --tokenizer) stay in.

Gang scheduling: Grove/KAI place a DynamoGraphDeployment as one podgang, so
a follower that is unschedulable until rox-<hash> exists holds the leader
too and nothing downloads. Grove also rewrites schedulingGates. Options
and a recommendation are on #2099; this PR keeps stock charts working and
does not change behaviour for gang-scheduled workers beyond the shared hash.

Restore-namespace NetworkPolicy: the chart's policy allows egress only to
nvsnap-server and selects every pod, so a namespace without other egress
allows loses DNS. NVCF function namespaces carry NVCA's allows; noted in the
design doc for a follow-up.

References

Relates to #2099

Related Pull Requests

#2101

Dependencies

None

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: NVIDIA/nvcf/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ee8dfbd1-5577-4c57-b703-7859e5622a0a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}