fix(chart): the kates NetworkPolicy lets the API reach the Kubernetes API, Trogdor, and Kafka in networkPolicy.kafka.namespace

.github/workflows/ci.yml

@@ -216,6 +216,26 @@ jobs:
           fi
           echo "OK: backup without the bundled database is refused"
 
+      # networkPolicy.kafka.namespace went unread for releases: the policy
+      # always allowed `kafka`, kept Trogdor's 8889 to `monitoring` and had no
+      # Kubernetes API rule, and nothing rendered it to notice.
+      - name: Assert kates NetworkPolicy egress follows its values
+        run: |
+          np=$(helm template kates charts/kates/ \
+                 --show-only templates/networkpolicy.yaml \
+                 --set networkPolicy.kafka.namespace=kates-stack)
+          fail() { echo "::error::kates NetworkPolicy: $1"; exit 1; }
+          grep -q 'kubernetes.io/metadata.name: "kates-stack"' <<<"${np}" \
+            || fail "no egress to networkPolicy.kafka.namespace"
+          if grep -qE 'kubernetes.io/metadata.name: "?kafka"?$' <<<"${np}"; then
+            fail "egress still names the kafka namespace"
+          fi
+          grep -q 'port: 6443' <<<"${np}" || fail "no Kubernetes API egress"
+          trogdor=$(awk '/kubernetes.io\/metadata.name:/ {ns=$2} /port: 8889/ {print ns}' <<<"${np}")
+          [ "${trogdor}" = '"kates-stack"' ] \
+            || fail "8889 must be open once, to the Kafka namespace (got: ${trogdor:-none})"
+          echo "OK: the egress follows networkPolicy.kafka.namespace"
+
       # The Strimzi version is declared in five places. If values.yaml's
       # strimziVersion drifts from the Chart.yaml dependency version, the CRD
       # hook applies CRDs for a different operator than the one installed —

README.md

@@ -213,7 +213,7 @@ When you install charts with Helm directly:
 | [`kafka-ui`](charts/kafka-ui/) | 0.3.0 | v1.5.0 | A Helm chart for deploying Kafka UI (Kafbat) with Strimzi SCRAM-SHA-512 authentication |
 | [`kates-chaos`](charts/kates-chaos/) | 2.2.1 | 3.28.0 | Kates Chaos Engineering — wraps the LitmusChaos execution plane (operator, exporter, CRDs) with Kafka-specific RBAC, experiment/engine templating, and monitoring |
 | [`kates-platform`](charts/kates-platform/) | 0.7.0 | 1.0.0 | Umbrella chart for the full Kates platform — Kafka, Kates, and supporting infrastructure |
-| [`kates`](charts/kates/) | 0.11.0 | 1.25.1 | Kates — Kafka Advanced Testing & Engineering Suite |
+| [`kates`](charts/kates/) | 0.11.1 | 1.25.1 | Kates — Kafka Advanced Testing & Engineering Suite |
 | [`legacy-kafka`](charts/legacy-kafka/) | 0.2.0 | 3.9.1 | A deliberately old Kafka (2.x on ZooKeeper, or 3.x on KRaft) to migrate FROM — the replication source that the pinned Strimzi release cannot deploy |
 | [`mirror-maker2`](charts/mirror-maker2/) | 0.11.4 | 4.3.1 | Strimzi KafkaMirrorMaker2 — cross-cluster replication, DR, and cross-version migration (2.x/3.x → 4.x) |
 | [`monitoring`](charts/monitoring/) | 1.6.0 | 82.4.3 | Kates Monitoring — wraps kube-prometheus-stack with Kates-specific dashboards and configuration |

charts/kates/Chart.yaml

@@ -2,11 +2,11 @@ apiVersion: v2
 name: kates
 description: Kates — Kafka Advanced Testing & Engineering Suite
 type: application
-# Minor: backup.enabled without the bundled PostgreSQL now fails rendering,
-# where it rendered no backup and said nothing, and values-prod.yaml drops its
-# backup block. A failed pg_dump fails the backup job before the prune and
-# leaves no archive. appVersion is unchanged.
-version: 0.11.0
+# Patch: the NetworkPolicy's Kafka egress follows networkPolicy.kafka.namespace
+# instead of always `kafka`, opens Trogdor's 8889 there instead of in
+# `monitoring`, and a new rule opens TCP 443/6443 for the Kubernetes API.
+# appVersion is unchanged.
+version: 0.11.1
 # The non-blocking health endpoints this chart's probes now point at come from
 # a BUILD-TIME property (quarkus.messaging.health.enabled=false), so they only
 # exist in the image built from this branch. Keep appVersion and
@@ -31,7 +31,7 @@ annotations:
   artifacthub.io/license: Apache-2.0
   artifacthub.io/changes: |
     - kind: fixed
-      description: "ACTION REQUIRED for values that set backup.enabled with externalDatabase — those backups never ran. The backup CronJob dumps the bundled PostgreSQL only and rendered only with postgresql.enabled, so values-prod.yaml's backup block (and the README's production example) rendered no CronJob and no PVC, while NOTES printed a backup schedule. backup.enabled without postgresql.enabled now fails rendering; back the external database up with its own tooling and set backup.enabled=false. values-prod.yaml no longer sets it. A pg_dump that failed also left a valid, empty gzip archive, the job passed, and the prune then deleted the real backups. The script runs with set -eo pipefail and dumps to a .partial file it renames only once the dump succeeds, so a failed dump fails the job before the prune and leaves no archive behind"
+      description: "On a CNI that enforces NetworkPolicy, the chart's NetworkPolicy cut the backend off from the Kubernetes API, so every chaos fault, pod listing, KafkaTopic CR and orphan recovery failed; its Kafka egress always named namespace kafka, whatever networkPolicy.kafka.namespace said; and it allowed Trogdor's 8889 only to namespace monitoring, where no coordinator runs. The Kafka egress now follows networkPolicy.kafka.namespace (default kafka), 8889 moves from monitoring to that namespace, and a new ports-only rule opens TCP 443 and 6443 to any address for the Kubernetes API. Upgrading: a release that set networkPolicy.kafka.namespace to anything but the namespace Kafka runs in loses Kafka, as the value now takes effect; a Trogdor coordinator outside that namespace needs a networkPolicy.egressRules entry"
     - kind: changed
       description: "appVersion 1.25.1 and the image tag move to 1.25.1, the release of the deployment hardening in #368: engine timeout and reaper tunables wired through values and the configmap, stranded runs recovered more robustly, and network policy updates. Templates are otherwise unchanged."
     - kind: fixed

charts/kates/README.md

@@ -57,7 +57,7 @@ All configuration is in [values.yaml](values.yaml). Key sections:
 | `ingress.certManager.issuerName` | `""` | cert-manager issuer name |
 | `ingress.certManager.issuerKind` | `ClusterIssuer` | Issuer kind |
 | `networkPolicy.enabled` | `true` | Enable NetworkPolicy |
-| `networkPolicy.kafka.namespace` | `kafka` | Kafka egress target namespace |
+| `networkPolicy.kafka.namespace` | `kafka` | Namespace the brokers (`networkPolicy.kafka.port`) and the Trogdor coordinator (8889) run in; `kates deploy` sets it |
 | `networkPolicy.kafka.port` | `9092` | Kafka egress target port |
 | `networkPolicy.prometheus.namespace` | `monitoring` | Prometheus egress target namespace: the one `prometheus.url` points into |
 | `networkPolicy.prometheus.port` | `9090` | Prometheus egress target port |
@@ -70,6 +70,16 @@ effect and moves the egress there while `prometheus.url` points into
 `monitoring`, so the backend loses Prometheus. Remove that line, or set it to
 the namespace your Prometheus runs in.
 
+Upgrading from chart 0.11.0 or earlier: those charts ignored
+`networkPolicy.kafka.namespace` and always allowed egress to `kafka`, allowed
+Trogdor's 8889 only to `monitoring`, and had no egress to the Kubernetes API,
+so on a CNI that enforces policy every chaos fault and pod listing failed. The
+Kafka egress now follows the value, with 8889 next to the brokers, and TCP
+443/6443 is open to any address. A release that set the value to anything but
+where Kafka runs loses Kafka after the upgrade. A Trogdor coordinator outside
+that namespace (`kates/docs/deployment.md` runs one in `kates`) needs a
+`networkPolicy.egressRules` entry.
+
 ### Security
 
 | Parameter | Default | Description |

charts/kates/templates/networkpolicy.yaml

@@ -30,6 +30,13 @@ spec:
           protocol: UDP
         - port: 53
           protocol: TCP
+    # The Kubernetes API, for chaos faults, pod listing, KafkaTopic CRs and the
+    # orphan reconciler; 6443 is the API server's own port after the Service DNAT.
+    - ports:
+        - port: 443
+          protocol: TCP
+        - port: 6443
+          protocol: TCP
     {{- if .Values.postgresql.enabled }}
     - to:
         - podSelector:
@@ -60,27 +67,25 @@ spec:
         - port: {{ .Values.networkPolicy.cdcDatabase.port | default 5432 }}
           protocol: TCP
     {{- end }}
+    # The brokers, and the Trogdor coordinator on 8889 where the default
+    # trogdor.coordinatorUrl puts it.
+    # shortcut: a coordinator in another namespace needs a networkPolicy.egressRules entry.
     - to:
         - namespaceSelector:
             matchLabels:
-              kubernetes.io/metadata.name: kafka
+              kubernetes.io/metadata.name: {{ .Values.networkPolicy.kafka.namespace | default "kafka" | quote }}
       ports:
         - port: {{ .Values.networkPolicy.kafka.port | default 9092 }}
           protocol: TCP
+        - port: 8889
+          protocol: TCP
     - to:
         - namespaceSelector:
             matchLabels:
               kubernetes.io/metadata.name: {{ .Values.networkPolicy.prometheus.namespace | default "monitoring" }}
       ports:
         - port: {{ .Values.networkPolicy.prometheus.port | default 9090 }}
           protocol: TCP
-    - to:
-        - namespaceSelector:
-            matchLabels:
-              kubernetes.io/metadata.name: monitoring
-      ports:
-        - port: 8889
-          protocol: TCP
     {{- with .Values.networkPolicy.egressRules }}
     {{- toYaml . | nindent 4 }}
     {{- end }}

charts/kates/values.yaml

@@ -447,7 +447,7 @@ networkPolicy:
       ports:
         - port: 8080
           protocol: TCP
-  # -- Kafka egress configuration
+  # -- Kafka egress: the brokers on port and Trogdor on 8889 (kates deploy sets namespace)
   kafka:
     namespace: kafka
     port: 9092

cli/cmd/deploy_components.go

@@ -1681,6 +1681,7 @@ data:
 		katesArgs = append(katesArgs,
 			"--set", "kafka.bootstrapServers="+katesBootstrap,
 			"--set", "kafka.topicNamespace="+kafkaNS,
+			"--set", "networkPolicy.kafka.namespace="+kafkaNS,
 			"--set", fmt.Sprintf("monitoring.enabled=%t", deployWithMonitoring))
 		katesArgs = append(katesArgs, dc.scrapeArgs("charts/kates")...)
 		katesArgs = append(katesArgs, dc.prometheusArgs()...)

cli/cmd/deploy_test.go

@@ -201,6 +201,9 @@ func TestDeployCommand_SingleTopology(t *testing.T) {
 			if !strings.Contains(cmd, "-n kates-test") {
 				t.Errorf("Expected Kates to be deployed in kates-test namespace, got: %s", cmd)
 			}
+			if !strings.Contains(cmd, "--set networkPolicy.kafka.namespace=kates-test") {
+				t.Errorf("Expected the Kates NetworkPolicy to allow Kafka in kates-test, got: %s", cmd)
+			}
 		}
 	}
 
@@ -276,6 +279,9 @@ func TestDeployCommand_IsolatedTopology(t *testing.T) {
 			if !strings.Contains(cmd, "-n app-sys") {
 				t.Errorf("Expected Kates to be deployed in app-sys namespace, got: %s", cmd)
 			}
+			if !strings.Contains(cmd, "--set networkPolicy.kafka.namespace=kafka-sys") {
+				t.Errorf("Expected the Kates NetworkPolicy to allow Kafka in kafka-sys, got: %s", cmd)
+			}
 		}
 		if strings.Contains(cmd, "helm upgrade --install chaos") {
 			foundChaos = true

docs/book/appendix-d-versions.md

@@ -22,7 +22,7 @@ Every version the platform pins, in one place. This table is generated from the
 | `kafka-ui` chart | 0.3.0 (app v1.5.0) | `charts/kafka-ui/Chart.yaml` |
 | `kates-chaos` chart | 2.2.1 (app 3.28.0) | `charts/kates-chaos/Chart.yaml` |
 | `kates-platform` chart | 0.7.0 (app 1.0.0) | `charts/kates-platform/Chart.yaml` |
-| `kates` chart | 0.11.0 (app 1.25.1) | `charts/kates/Chart.yaml` |
+| `kates` chart | 0.11.1 (app 1.25.1) | `charts/kates/Chart.yaml` |
 | `legacy-kafka` chart | 0.2.0 (app 3.9.1) | `charts/legacy-kafka/Chart.yaml` |
 | `mirror-maker2` chart | 0.11.4 (app 4.3.1) | `charts/mirror-maker2/Chart.yaml` |
 | `monitoring` chart | 1.6.0 (app 82.4.3) | `charts/monitoring/Chart.yaml` |

docs/stretched-kafka/04-test-and-chaos-scenarios.md

@@ -486,7 +486,7 @@ service:
   type: NodePort
   nodePort: "30083"
 networkPolicy:
-  enabled: false        # its egress rules leave out the Kubernetes API, which the chaos backend calls
+  enabled: false        # untested in the lab with it on; see below
 extraEnv:
   - name: KATES_CHAOS_PROVIDER                # simultaneous kills
     value: kubernetes
@@ -512,7 +512,7 @@ kates ctx set site-a --url http://localhost:30083 --api-key "$KEY_A"
 kates ctx set site-b --url http://localhost:30084 --api-key "$KEY_B"
 ```
 
-`POD_KILL` needs no extra RBAC; leave `rbac.directChaos` off. The chart's NetworkPolicy allows egress only to DNS, PostgreSQL, Kafka and the `monitoring` namespace, so on a CNI that enforces policies it cuts Kates off from the API server its chaos backend and pod watcher use: disable it in the lab, or add a `networkPolicy.egressRules` entry for the API servers on the target. The chart's default `resources` request 2 GiB; lower them on a laptop. Reach each Kates through a NodePort or load balancer in its own site, not `kubectl port-forward`, which needs the API server that T1 and T9 take down. Read the API keys before any test for the same reason.
+`POD_KILL` needs no extra RBAC; leave `rbac.directChaos` off. The lab values above leave the chart's NetworkPolicy off. Since chart 0.11.1 its egress allows TCP 443 and 6443 to any address, which covers the API server its chaos backend and pod watcher use, and Kafka in `networkPolicy.kafka.namespace` (`kafka` here); turning it on in the lab is untested. The chart's default `resources` request 2 GiB; lower them on a laptop. Reach each Kates through a NodePort or load balancer in its own site, not `kubectl port-forward`, which needs the API server that T1 and T9 take down. Read the API keys before any test for the same reason.
 
 ### Sustained Workloads