fix(self-managed): run bundled images and hooks as non-root

[sbaum1994]

TL;DR

Run self-managed bootstrap, migration, API Keys, NATS, and helper workloads as non-root. The remaining OCI metadata work for NVCF-distributed images is tracked in a public subissue.

Additional Details

The stack needs non-root defaults at both layers: built image metadata and the pod specs rendered by Helm charts. Cassandra migrations were addressed in #1712. This PR covers the repository-owned worker-init and API Keys images, OpenBao runtime images, and the charts that launch these and mirrored helper images.

What changed

• Declare numeric runtime users in worker-init, API Keys, OpenBao server, and OpenBao migrations images. The API Keys Bazel image contract verifies Config.User=1000:1000.
• Set non-root contexts for generated function/task bootstrap containers, NATS server and reloader, OpenBao migration and setup hooks, API Keys, account bootstrap, upgrade receipt, and addon migration Jobs.
• Forward API Keys resource and probe overrides through Helmfile so constrained local environments can be validated without changing chart defaults.
• Add render and image contract checks for the new security contexts.
• Stabilize local Helmfile BDD fixtures with API Keys startup overrides and a compute-reachable invocation callback URL.

For the Reviewer

Please review the generated worker pod fsGroup behavior, the NATS volume ownership context, and OpenBao's preserved UID 100/GID 1000. This draft groups related changes so it can be split by owner after review.

For QA

Single-cluster k3d Helmfile install passed with 24 deployed releases. The changed local API Keys and OpenBao charts then upgraded successfully. NATS server and reloader reported UID/GID 1000; the OpenBao migration Job reported UID 100/GID 1000 and completed. The isolated single cluster was destroyed before multi-cluster testing.

Multi-cluster control-plane and compute-plane Helmfile installs both passed. The control plane has 23 deployed releases; the NVCA operator is ready on the compute plane and its NVCFBackend reports healthy. The LLM addon migration Job completed as UID 100/GID 1000. The published API Keys chart needed a local QA override of 1 CPU and a longer startup/liveness window to finish under the k3d CPU budget.

Full live Helmfile BDDs passed: single cluster (10 scenarios, 89 steps) and multi cluster (10 scenarios, 93 steps). Both exercised published sample task and function charts with and without resource requests and limits. The multi-cluster run used one control-plane agent and two compute agents on fresh k3d clusters with an isolated kubeconfig. Cross-cluster image, Helm chart, gRPC, and LLM function invocations passed.

The published BDD chart inputs were inference-test 0.1.2 and task-helmchart-test 0.1.1 with CPU and memory requests and limits, plus inference-test 0.1.0-pr1887 and task-helmchart-test 0.2 without them. The function charts use fastapi-echo-sample; the task charts use task-simple-sample. All four chart packages and their container images were reachable, so no new test chart was needed.

Customer Release Notes

Self-managed bootstrap and migration workloads run as non-root by default.

Plan Summary

No new Kubernetes resources. Existing Jobs and pods gain explicit non-root security contexts and, where needed, a matching filesystem group.

Usage

Custom worker init images must support UID/GID 1000 and writable shared volumes. The requirement is documented in docs/compute-plane/cluster-management/self-managed.md.

Testing

Passed: worker-init OCI contents and package checks; translator and targeted NVCA checks; API Keys Bazel OCI image contract with Java 25; NVCA transport TLS Bazel test; NATS, OpenBao, API Keys, addon migration, account bootstrap, and upgrade receipt chart tests; make -C deploy/stacks/self-managed test; make -C deploy/stacks/nvcf-compute-plane test-local; ./tools/ci/check-docs; single-cluster Helmfile install and local chart upgrades; multi-cluster control-plane and compute-plane Helmfile installs and healthy NVCA backend; full live single-cluster Helmfile BDD (10 scenarios, 89 steps); full live multi-cluster Helmfile BDD (10 scenarios, 93 steps); focused BDD wiring tests; git diff --check.

The full NVCA MiniService package test needs KUBEBUILDER_ASSETS, which is absent here. The first NVCA CI run found a missing Bazel dependency, fixed in the current commit and verified by the focused Bazel test. The live BDDs used existing published stack and sample artifacts; they do not validate rebuilt OCI metadata from this branch. QA is needed for rebuilt published images before merge.

Notes

The current mirrored NATS server, NATS reloader, and Alpine Kubernetes helper images still have unset OCI User metadata. Their runtime pods are non-root in this PR; repackaging and publication are tracked in #2085. ddcs-dist-kv, gxcache-service, gxcache-webhook, gxcache-init, and streaming-proxy are not owned by NVCF and are out of scope for this PR and #2071. Scanner closure for NVCF-distributed images requires checking the rebuilt image digests. No dependency or NOTICE changes.

References

#1712

Related Pull Requests

#1610, #1712

Dependencies

None. License review and NOTICE update are not required.

Issues

Relates to #2071
Relates to #2072
Relates to #2073
Relates to #2074
Relates to #2084
Relates to #2085

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-fix-nonroot-self-managed-hooks.docs.buildwithfern.com/nvcf