Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions deploy/helm/openshell/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,23 @@ namespace. The gateway and workspace releases can then be upgraded and removed
independently. Use Kubernetes `operator` workspace mode when one gateway serves
multiple pre-provisioned workspace namespaces.

## Certificate hook metadata

Use `pkiInitJob.podLabels` and `pkiInitJob.podAnnotations` to add metadata to the
Pod templates of the `<fullname>-certgen` and `<fullname>-certgen-backend-ca`
Jobs. These settings apply to both hooks when enabled and do not affect gateway
or sandbox Pods. Chart-owned Pod labels take precedence when custom keys
conflict.

For example, disable Istio sidecar injection for the hook Pods in an injected
namespace:

```yaml
pkiInitJob:
podAnnotations:
sidecar.istio.io/inject: "false"
```
## Cluster-scoped vs namespaced objects
Most objects in this chart are namespaced and land in the release namespace.
Expand Down Expand Up @@ -450,6 +467,8 @@ discovery endpoint or its TLS CA.
| openshiftRoute.host | string | `""` | Hostname for the Route. Must match a SAN on the gateway's server cert. |
| pkiInitJob.enabled | bool | `true` | Run a pre-install/pre-upgrade Job that creates gateway and client mTLS Secrets. When certManager.enabled=true, cert-manager owns TLS and this same hook runs in JWT-only mode even if pkiInitJob.enabled remains true. |
| pkiInitJob.failOnTimeout | bool | `true` | Fail the helm install/upgrade if cert-manager does not issue the certificate within the polling timeout. When true (default), the install fails immediately if the timeout is reached, providing clear feedback that BackendTLSPolicy is non-functional. When false, the hook succeeds with a warning and you can run `helm upgrade` after cert-manager issues the certificate to create the backend CA ConfigMap. If you set this to false and see "TLS error: Secret is not supplied by SDS" when connecting to the gateway, check if the TLS secret exists and run `helm upgrade` to create the ConfigMap. |
| pkiInitJob.podAnnotations | object | `{}` | Additional annotations on both certificate hook Pods. |
| pkiInitJob.podLabels | object | `{}` | Additional labels on both certificate hook Pods. Chart-owned labels take precedence. |
| pkiInitJob.securityContext.allowPrivilegeEscalation | bool | `false` | Whether the certgen hook container can gain additional privileges. |
| pkiInitJob.securityContext.capabilities.drop | list | `["ALL"]` | Linux capabilities dropped from the certgen hook container. |
| pkiInitJob.securityContext.runAsNonRoot | bool | `true` | Require the certgen hook container to run as a non-root user. |
Expand Down
17 changes: 17 additions & 0 deletions deploy/helm/openshell/README.md.gotmpl
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,23 @@ namespace. The gateway and workspace releases can then be upgraded and removed
independently. Use Kubernetes `operator` workspace mode when one gateway serves
multiple pre-provisioned workspace namespaces.

## Certificate hook metadata

Use `pkiInitJob.podLabels` and `pkiInitJob.podAnnotations` to add metadata to the
Pod templates of the `<fullname>-certgen` and `<fullname>-certgen-backend-ca`
Jobs. These settings apply to both hooks when enabled and do not affect gateway
or sandbox Pods. Chart-owned Pod labels take precedence when custom keys
conflict.

For example, disable Istio sidecar injection for the hook Pods in an injected
namespace:

```yaml
pkiInitJob:
podAnnotations:
sidecar.istio.io/inject: "false"
```

## Cluster-scoped vs namespaced objects

Most objects in this chart are namespaced and land in the release namespace.
Expand Down
13 changes: 11 additions & 2 deletions deploy/helm/openshell/templates/certgen.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
{{- if or .Values.pkiInitJob.enabled .Values.certManager.enabled }}
{{- $hookName := printf "%s-certgen" (include "openshell.fullname" .) }}
{{- $ns := .Release.Namespace }}
{{- $podLabels := merge (include "openshell.certgenPodLabels" . | fromYaml) (.Values.pkiInitJob.podLabels | default dict) }}
apiVersion: v1
kind: ServiceAccount
metadata:
Expand Down Expand Up @@ -75,7 +76,11 @@ spec:
template:
metadata:
labels:
{{- include "openshell.certgenPodLabels" . | nindent 8 }}
{{- toYaml $podLabels | nindent 8 }}
{{- with .Values.pkiInitJob.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
restartPolicy: OnFailure
serviceAccountName: {{ $hookName }}
Expand Down Expand Up @@ -143,7 +148,11 @@ spec:
template:
metadata:
labels:
{{- include "openshell.certgenPodLabels" . | nindent 8 }}
{{- toYaml $podLabels | nindent 8 }}
{{- with .Values.pkiInitJob.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
restartPolicy: OnFailure
serviceAccountName: {{ $hookName }}
Expand Down
74 changes: 74 additions & 0 deletions deploy/helm/openshell/tests/certgen_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -192,3 +192,77 @@ tests:
app.kubernetes.io/instance: openshell
app.kubernetes.io/component: certgen
documentIndex: 4

- it: adds configured metadata to both certificate hook Pod templates
template: templates/certgen.yaml
set:
certManager.enabled: true
grpcRoute.backendTLSPolicy.enabled: true
pkiInitJob.podLabels:
example.com/workload: certificate-bootstrap
pkiInitJob.podAnnotations:
sidecar.istio.io/inject: "false"
asserts:
- equal:
path: spec.template.metadata.labels["example.com/workload"]
value: certificate-bootstrap
documentIndex: 3
- equal:
path: spec.template.metadata.annotations["sidecar.istio.io/inject"]
value: "false"
documentIndex: 3
- notExists:
path: metadata.labels["example.com/workload"]
documentIndex: 3
- notExists:
path: metadata.annotations["sidecar.istio.io/inject"]
documentIndex: 3
- equal:
path: spec.template.metadata.labels["example.com/workload"]
value: certificate-bootstrap
documentIndex: 4
- equal:
path: spec.template.metadata.annotations["sidecar.istio.io/inject"]
value: "false"
documentIndex: 4
- notExists:
path: metadata.labels["example.com/workload"]
documentIndex: 4
- notExists:
path: metadata.annotations["sidecar.istio.io/inject"]
documentIndex: 4

- it: preserves chart-owned Pod labels when custom keys conflict
template: templates/certgen.yaml
set:
certManager.enabled: true
grpcRoute.backendTLSPolicy.enabled: true
pkiInitJob.podLabels:
app.kubernetes.io/name: openshell
app.kubernetes.io/instance: custom-release
app.kubernetes.io/component: gateway
asserts:
- equal:
path: spec.template.metadata.labels["app.kubernetes.io/name"]
value: openshell-certgen
documentIndex: 3
- equal:
path: spec.template.metadata.labels["app.kubernetes.io/instance"]
value: openshell
documentIndex: 3
- equal:
path: spec.template.metadata.labels["app.kubernetes.io/component"]
value: certgen
documentIndex: 3
- equal:
path: spec.template.metadata.labels["app.kubernetes.io/name"]
value: openshell-certgen
documentIndex: 4
- equal:
path: spec.template.metadata.labels["app.kubernetes.io/instance"]
value: openshell
documentIndex: 4
- equal:
path: spec.template.metadata.labels["app.kubernetes.io/component"]
value: certgen
documentIndex: 4
4 changes: 4 additions & 0 deletions deploy/helm/openshell/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -661,6 +661,10 @@ pkiInitJob:
# Secrets. When certManager.enabled=true, cert-manager owns TLS and this same
# hook runs in JWT-only mode even if pkiInitJob.enabled remains true.
enabled: true
# -- Additional labels on both certificate hook Pods. Chart-owned labels take precedence.
podLabels: {}
# -- Additional annotations on both certificate hook Pods.
podAnnotations: {}
# -- Extra DNS SANs to append to the server certificate.
serverDnsNames: []
# -- Extra IP SANs to append to the server certificate.
Expand Down
12 changes: 12 additions & 0 deletions docs/kubernetes/setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,18 @@ helm upgrade --install openshell \

The chart automatically generates PKI secrets on first install using pre-install Helm hooks. No manual secret creation is required.

To add labels or annotations to both certificate hook Pod templates, use
`pkiInitJob.podLabels` and `pkiInitJob.podAnnotations`. Chart-owned Pod labels
take precedence when custom keys conflict.
For example, disable sidecar injection for hook Pods in an Istio-injected
namespace:

```yaml
pkiInitJob:
podAnnotations:
sidecar.istio.io/inject: "false"
```
The default Kubernetes driver puts all workspaces in one sandbox namespace. To
have the gateway create a namespace for each workspace, follow
[Workspace on Kubernetes](/kubernetes/workspaces).
Expand Down
1 change: 1 addition & 0 deletions skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -1279,6 +1279,7 @@ credential failures.
| `helm upgrade` fails with an `autoscaling.*` message | HPA values invalid: missing `resources.requests` (or `resources.limits`), `maxReplicas` above 1 without `server.externalDbSecret` (or on a StatefulSet without `workload.allowMultiReplicaStatefulSet`), no metric target, or min/max out of order. "`minReplicas` and `maxReplicas` are not set" means `--reuse-values` kept a release without the chart's autoscaling defaults | Fix the values named in the error; upgrade with `--reset-then-reuse-values` instead of `--reuse-values` |
| HPA shows `<unknown>` targets | No metrics-server for CPU/memory, or the metrics adapter does not serve the custom metric | `kubectl -n openshell describe hpa openshell`, `kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1` |
| One replica holds most sessions after a rollout | Expected: sessions stay where they reconnected | `openshell_server_supervisor_sessions` per pod; it fades as sandboxes are recreated |
| Certificate hook Job stays active after `certgen` exits successfully | An injected sidecar is still running in the hook Pod | Inspect all container statuses. Configure `pkiInitJob.podLabels` or `pkiInitJob.podAnnotations` as required by the injector, then retry the Helm install or upgrade to recreate the hook Pods |
| Gateway or `<release>-certgen` hook pod rejected with `violates PodSecurity "restricted:latest"` | The rendered container lacks `runAsNonRoot` or `seccompProfile`: an older chart version does not set them, or an override in `securityContext` / `pkiInitJob.securityContext` unset them | `kubectl -n openshell get events --field-selector reason=FailedCreate`; upgrade the chart, then remove overrides that unset those fields |
| OpenShift gateway pod fails to start with an SCC/`runAsUser` error (e.g. `unable to validate against any security context constraint`) | Chart's default `podSecurityContext`/`securityContext` hardcodes `runAsUser`/`fsGroup`, which the restricted-v2 SCC rejects; it must instead inject the namespace-assigned UID/GID range | `oc -n openshell describe pod <pod>`; deploy with `podSecurityContext: null` and clear `securityContext.runAsUser` (see `deploy/helm/openshell/ci/values-openshift-scc.yaml`) |
| OpenShift `<release>-supervisor` Deployment never creates pods after setting `supervisor_topology = "gateway"` (SCC `runAsUser`/`fsGroup` not in the namespace range) | `supervisor.podSecurityContext` defaults `runAsUser`, `runAsGroup`, and `fsGroup` to 65534, outside the namespace's assigned UID/GID ranges | `kubectl -n openshell get events --field-selector reason=FailedCreate`; compare with the `openshift.io/sa.scc.*` annotations from `oc describe ns openshell`; set `supervisor.podSecurityContext.runAsUser`, `.runAsGroup`, and `.fsGroup` to `null`, as `deploy/helm/openshell/ci/values-openshift-scc.yaml` does, and keep `runAsNonRoot` and `seccompProfile` |
Expand Down
Loading