Make visible devices configurable

[sstrk]

As adjusting this env var is necessary for the current suggested workaround for issues 1 and 2.
This saves overhead for operators who previously had to adjust the env var via a kustomization or helm post-renderer.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[myeolenv]

Thanks for the contribution. We will review this PR.

[cdesiniotis]

@sstrk my preference would be to not add this option to the top-level helm values, as those are typically reserved for common config shared by both k8s-device-plugin and GFD (which can both be deployed by the chart). I would be more open to introducing a common field for passing arbitrary envvars to the device-plugin main container. For example:

devicePlugin:
  containers:
    init: {}
    main:
      env:
        - name: NVIDIA_VISIBLE_DEVICES
          value: "all"
    sidecar: {}

@tariq1890 @rahulait let me know if you have any strong opinions.