nvsnap: support Helm-chart functions

[balajinvda]

Goal

Helm-chart functions get no cache reuse. Customers predominantly deploy Helm
charts, so this is the main deployment mode, not an edge case. Every worker in a
release downloads the model and recompiles its kernels independently. The goal is
that one worker warms, and every other worker reads what it produced.

nvsnap is the only component that can do this. NVCA's model cache covers NGC
registry models only: not HuggingFace, not custom model endpoints, and not
compile artifacts at all. nvsnap never asks what to cache, it snapshots what the
workload wrote, which is why it covers all three. Nothing here proposes changing
NVCA.

What is already proven

Measured on dev1, 2026-09-24, two identical vLLM TP=2 pods on different nodes,
each warmed to Ready and then sent an identical inference so first-request JIT
fired.

Cache content is shareable:

model/hub            15 files  2100.43 MB  byte-identical
cache/.triton        60 files     5.26 MB  keys identical, 2 PTX differ by 2-3 bytes
cache/torchinductor  30 files     1.61 MB  keys identical, 12 leaf names differ
96 of 108 paths identical, 37 of 37 compile-cache keys identical

The model, which is 99.9 percent of the bytes, is byte-identical. Compilation
output is not deterministic, but the cache keys are, and a hit depends on the key
rather than on the bytes. Byte-identical trees were never the requirement.

The capture is complete. All seven redirected caches land inside the captured
directory, including FlashInfer, which arrives under the HOME redirect at
/opt/nvsnap/cache/.cache/flashinfer and needs no variable of its own. The
/root/.cache/flashinfer and /root/.cache/pip directories are image-baked, not
runtime output. /tmp/root/cutlass_python_cache is empty.

What is missing, in dependency order

1. The capture trigger is unobservable, and did not fire

A pod that satisfies every documented precondition was not captured: label
nvsnap.io/capture=true, PodReady True, 2 GPUs, correct node, correct namespace,
watcher confirmed running on that node with --rootfs-capture. No capture
manifest appeared and no rootfsonly log line was emitted in 15 minutes, across
pod creation with the label and a later label remove and re-add.

The reason it cannot be diagnosed is in internal/rootfsonly/watcher.go. Every
early return in handlePodEvent is silent, and the one Info line is gated on
gpus < 2:

if !w.isLabeledForCapture(pod) { return }
if !IsPodReady(pod)            { return }
if gpus < 2 { log.Info("...single-GPU pod...") }
if _, already := w.captured.LoadOrStore(pod.UID, struct{}{}); already { return }
go w.runCapture(ctx, pod.DeepCopy())

For the multi-GPU case this path exists to serve, a silently skipped capture is
indistinguishable from one that was never triggered. runCapture logs on commit
and on failure; neither appeared.

The leading hypothesis is the captured sync.Map, which is keyed by pod UID and
cleared only by EnsureRetryable, so a UID marked by an attempt that then bailed
is a permanent no-op. That is a hypothesis, not a finding, and it cannot be
confirmed without instrumenting the agent, which is the point.

This blocks everything below: election and identity are irrelevant if the trigger
does not fire or cannot be diagnosed when it does not.

2. Chart pods are never decorated

cacheDirCapturePatches is opt-in on nvsnap.io/capture=true, and the comment
names the case:

// Un-labeled pods (system/infra, helm-chart miniservice,
// anything not meant for capture) are left untouched.

Observed directly: without the label the pod keeps HF_HOME from the image, the
model lands in /root/.cache, and /opt/nvsnap is not mounted. Nothing is
captured and nothing is shareable.

3. Nothing elects a single downloader

Capture is triggered by labelling one pod, which is an explicit step for
container functions. A Helm release has no such step, so either every worker
cold-starts, or every worker waits and none warms. The wait gate already exists:
nvsnap-l2-wait is prepended at init index 0 and blocks until the capture is
promoted. What is missing is a release-scoped decision, made before consumers
start, that inspects the rendered chart, identifies which workload downloads, and
elects one warmer.

4. The sharing key is not stable across workers

composeEngineCompatFlags hashes Command and Args wholesale and positionally, so
workers differing only in node rank or leader address produce different hashes
and miss each other's capture. Either normalise the rank-bearing arguments or let
the identity be declared.

Note that Compose() is spec-derived, not content-derived, despite
checkpointstore's package comment calling the store content-addressed. Two
captures of the same spec share a hash whatever their bytes.

5. Storage selection is not agnostic

Partly addressed by #2100, which stops promoting on capability an unqualified
driver never proved. The rest is to stop holding a second opinion about the same
StorageClasses: NVCA publishes qualified access modes in the
nvcf-storage-capabilities ConfigMap, key storage-provider-capabilities.yaml,
installed by the NVCA chart with an embedded fallback. nvsnap can read it at
runtime, keeping its own behavioural fields (snapshotClass,
volumeHandleTransform, mountOptions) which the capability catalog does not
express. That is a read of an existing ConfigMap, not a change to NVCA, and the
two Go modules stay independent.

Still unmeasured

Whether a fresh pod handed a warm cache reuses it or recompiles. The key match
above is strong indirect evidence; the direct test is to mount one worker's cache
into a new pod and confirm no new compile keys appear. Blocked on item 1.

Whether ranks that legitimately differ produce divergent compile caches. Every
workload in the repository is single-pod, so this needs a multi-pod chart built
first. Symmetric tensor parallelism should match; pipeline or expert parallelism
may not.

Sequence

1. Make the capture trigger observable, and find out why it did not fire.
2. Prove reuse: warm, capture, restore into a fresh pod, confirm no recompile.
3. Decorate and elect for chart pods.
4. Stabilise the sharing key across ranks.
5. Catalog-derived storage selection.

References

Relates to #1009
Relates to #2100

[balajinvda]

Step 1 and step 2 done, single namespace

Reuse is now proven directly, not only inferred from matching cache keys.
Measured on dev1, 2026-09-24, agent v0.2.72-shim1, all in nvsnap-system.

capture             vllm-tp2-a and vllm-tp2-b, different nodes, one hash a387a272d77d
                    108 files, 2107 MiB, second capture deduplicated onto the first
restore             fresh pod with nvsnap.io/restore-from: <hash>
time to Ready       59s          (cold baseline on the same node: 74s)
inference           "The capital of France is" -> " Paris."   HTTP 200
model downloaded    no           0 download lines in the log
kernels recompiled  no           37 of 37 captured compile-cache keys present, 0 new, 0 missing

The saving is small here because TinyLlama is small; what gets skipped is the
download and the compile, both of which scale with the model.

Two defects had to be fixed to get there, both on nvsnap/helm-chart-support:

• 189b87a: the watcher's dedup set marked a pod UID as scheduled and treated
  it as done. Three of five exits from runCapture left the mark in place with no
  capture, so an abort poisoned the pod for the life of the agent, and because
  the only log line was gated on gpus < 2 a multi-GPU pod produced no output
  on any branch. Both pods captured within seconds of the fixed agent starting.
• 67d15d8: the restore shim exec'd the CAPTURED pod's entry argv. For the
  bash-wrapper convention the pid resolver had recorded ["sleep","30"], so
  the restored pod seeded 2.21GB perfectly, ran sleep 30, exited, and never
  served. The shim now execs the restoring pod's own command.

Correction to the whole-rootfs removal scoping discussed alongside this issue:
nvsnap-rootfs-restore is NOT reachable only on the rootfs branch. It is the
prewarm and exec shim for cachedir restores too, and it ran in this test. That
removal needs re-scoping before it goes anywhere.

Next per the agreed order: cross-namespace. NVCF runs charts in their own
namespace, so the reader PVC has to reach a namespace other than the one the
capture was written in. NVCA's catalog already states the constraint: RWX
reaches other namespaces natively, ROX does not except on NVMesh via CSI
volume-handle rewrite, and nvsnap's SharedVolumePromoter already carries a
VolumeHandleTransform: "nvmesh" for exactly that. Whether the two are
equivalent is the next thing to establish.

[balajinvda]

Correction and final shape of step 2

The shim is gone from the cachedir restore, not patched. The earlier comment
above said nvsnap-rootfs-restore runs on cachedir restores and so blocks the
whole-rootfs removal; that was true of the code as it stood and is no longer
true. A cachedir restore is now exactly the design: mount the rox read-only,
seed the writable cache shadow, run the pod's own command untouched.

Re-measured on dev1, agent v0.2.73-noshim1, single namespace:

time to Ready       59s   (cold 74s; with the shim's page-cache prewarm: also 59s)
inference           " Paris."   HTTP 200
model downloaded    no
kernels recompiled  no    37 of 37 keys, 0 new, 0 missing
shim / tools mount  none

The prewarm the shim existed for bought no measurable time. The 15s saving is
small because TinyLlama is small; download and compile are what get skipped.

Consequence: nvsnap-rootfs-restore is reachable only from the whole-rootfs
branch again, so that path's removal scoping (about 1640 lines, no cachedir
impact) holds. It is cleanly separable and best as its own PR.

Also observed: deleting both source pods left the capture intact. The manifest
ConfigMap persists and the rox PVC rox-<hash> stays Bound to
nvsnap-ro-pv-<hash>. That bound read-only volume is the object cross-namespace
sharing has to reach, which is the next step.

[balajinvda]

Scale-up test with a real chart: cache reuse does not engage

Throwaway Helm chart, TinyLlama TP=2 vLLM Deployment, pod template carrying
nvsnap.io/capture: "true" and nvsnap.io/restore-from: "auto" (the only
template a chart can have: identical for every replica). dev1, 2026-09-25,
agent v0.2.74.

helm install replicas=1   webhook: "no capture for hash; admitting pod unchanged (cold start)" hash=c8bbf555
                          pod: no cache env, no /opt/nvsnap, nothing captured
kubectl scale replicas=2  webhook: same line, same hash; second pod cold too (94 s)

Compare the raw-Pod flow, which works: capture pod without restore-from
gets the cache decoration and is captured; restore pod with an explicit hash
restores (59 s vs 74 s TinyLlama, 247 s vs 632 s Llama-70B).

Three gaps, in the order they bite:

1. A pod with restore-from that misses is admitted unchanged
   (internal/webhook/mutate.go, the cold-start return after the L2 miss).
   cacheDirCapturePatches runs only on the no-annotation branch. So the
   first replica of a chart is never a capture source, and nothing ever
   exists for the second replica to restore from.
2. Even with (1) fixed, auto cannot match: the watcher composes the hash
   from the live pod (resolved image digest from status.containerStatuses,
   plus the env the webhook injected), admission composes from the incoming
   spec (image tag, original env). TestMutate_AutoUsesComposerHash
   composes both sides from the same admission object and cannot see this.
3. Deployment pods have no name at admission (generateName), so the
   webhook's log lines and anything keyed on pod.Name cannot identify
   them; the admission lines above carry no pod= field.

Plan: elect one downloader at admission, gate the rest from scheduling

Per hash, decided by the webhook when each pod is created:

• Hash once, stamp it. Compose from the incoming spec and write
  nvsnap.io/hash on the pod. The watcher reads the annotation instead of
  recomposing. Removes (2) by construction.
• Elect atomically. Try to create Lease nvsnap-capture-<hash> with the
  pod UID as holder. create is atomic, so of N concurrent admissions
  exactly one wins. Winner gets the capture decoration (fixes (1) for the
  leader). Same primitive the L2 promote already uses one step later.
• Gate followers. Every loser gets the restore decoration for that hash plus
  a schedulingGates entry (nvsnap.io/wait-for-cache). A gated pod holds
  no node and no GPU. nvsnap-server already receives the promote state
  (POST /checkpoints/by-hash/{hash}/pvc-state); on ready it lists pods
  with that hash annotation and removes the gate. The rox is Bound at that
  point, the follower schedules and starts warm. nvsnap-l2-wait stays as a
  second check and should exit at once.
• Already ready: plain restore (scale-up after warm, reinstall).
• Leader fails before commit: the Lease has a TTL the agent renews during
  capture and the watcher's committed defer releases; the server ungates
  followers with nvsnap.io/cold-start=true and they run cold. Correctness
  never depends on the election.

Trade-offs to settle: gated pods depend on nvsnap-server being up, so the
gate carries an admission-time deadline the agents also honour; and which
pods participate, since NVCF chart pods carry no nvsnap labels (item 2
above), so the selector is GPU-requesting pods in function namespaces with
the label as override.

Cluster requirement: schedulingGates GA since Kubernetes 1.30 (dev1 is
1.35).