From bba4ad11b3dee4296898315120bfa0f00b11d97d Mon Sep 17 00:00:00 2001 From: Said Boudjelda Date: Sat, 17 Oct 2026 19:23:35 +0200 Subject: [PATCH] fix(chart): the kates NetworkPolicy lets the API reach the Kubernetes API, Trogdor, and Kafka in networkPolicy.kafka.namespace On a CNI that enforces NetworkPolicy, the default kates policy cut the API off from three things it needs. It had no egress to the Kubernetes API, so every fabric8 call (chaos faults, disruption plans, pod listing, KafkaTopic CRs, the orphan reconciler) failed after its retry backoff. Its Kafka rule named the namespace `kafka` literally and never read networkPolicy.kafka.namespace, so `kates deploy --topology single -n kates-stack`, which runs Kafka in kates-stack, left Kates with no Kafka at all while the pod still went Ready on its cached Kafka check. And Trogdor's 8889 was allowed only to `monitoring`, where nothing listens on it, while the default coordinator URL is trogdor-coordinator.kafka.svc:8889. The policy now opens TCP 443 and 6443 to any address (the ports-only shape kafka-common uses; 6443 is the API server's port after the Service DNAT), takes the Kafka namespace from networkPolicy.kafka.namespace, and opens 8889 next to the brokers in that namespace. The `monitoring` 8889 rule is gone. `kates deploy` sets networkPolicy.kafka.namespace to the namespace it installs Kafka in, and the two topology tests assert it. The helm-lint job renders the policy with the value set and checks the egress follows it. The kates chart moves to 0.11.1, with README.md and the book's version matrix to match. The chart README's upgrade note and a new artifacthub.io/changes entry say what changes for a release that set the value elsewhere, and that a coordinator outside the Kafka namespace still needs an egressRules entry. The stretched-kafka lab no longer says the policy blocks the API server. --- .github/workflows/ci.yml | 20 ++++++++++++++++++ README.md | 2 +- charts/kates/Chart.yaml | 12 ++++++----- charts/kates/README.md | 12 ++++++++++- charts/kates/templates/networkpolicy.yaml | 21 ++++++++++++------- charts/kates/values.yaml | 2 +- cli/cmd/deploy_components.go | 1 + cli/cmd/deploy_test.go | 6 ++++++ docs/book/appendix-d-versions.md | 2 +- .../04-test-and-chaos-scenarios.md | 4 ++-- 10 files changed, 63 insertions(+), 19 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index dacc670e..e956ebc4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -191,6 +191,26 @@ jobs: - name: Template validation run: helm template kates charts/kates/ > /dev/null + # networkPolicy.kafka.namespace went unread for releases: the policy + # always allowed `kafka`, kept Trogdor's 8889 to `monitoring` and had no + # Kubernetes API rule, and nothing rendered it to notice. + - name: Assert kates NetworkPolicy egress follows its values + run: | + np=$(helm template kates charts/kates/ \ + --show-only templates/networkpolicy.yaml \ + --set networkPolicy.kafka.namespace=kates-stack) + fail() { echo "::error::kates NetworkPolicy: $1"; exit 1; } + grep -q 'kubernetes.io/metadata.name: "kates-stack"' <<<"${np}" \ + || fail "no egress to networkPolicy.kafka.namespace" + if grep -qE 'kubernetes.io/metadata.name: "?kafka"?$' <<<"${np}"; then + fail "egress still names the kafka namespace" + fi + grep -q 'port: 6443' <<<"${np}" || fail "no Kubernetes API egress" + trogdor=$(awk '/kubernetes.io\/metadata.name:/ {ns=$2} /port: 8889/ {print ns}' <<<"${np}") + [ "${trogdor}" = '"kates-stack"' ] \ + || fail "8889 must be open once, to the Kafka namespace (got: ${trogdor:-none})" + echo "OK: the egress follows networkPolicy.kafka.namespace" + # The Strimzi version is declared in five places. If values.yaml's # strimziVersion drifts from the Chart.yaml dependency version, the CRD # hook applies CRDs for a different operator than the one installed — diff --git a/README.md b/README.md index 75cd1c36..a7629dcd 100644 --- a/README.md +++ b/README.md @@ -213,7 +213,7 @@ When you install charts with Helm directly: | [`kafka-ui`](charts/kafka-ui/) | 0.3.0 | v1.5.0 | A Helm chart for deploying Kafka UI (Kafbat) with Strimzi SCRAM-SHA-512 authentication | | [`kates-chaos`](charts/kates-chaos/) | 2.2.1 | 3.28.0 | Kates Chaos Engineering — wraps the LitmusChaos execution plane (operator, exporter, CRDs) with Kafka-specific RBAC, experiment/engine templating, and monitoring | | [`kates-platform`](charts/kates-platform/) | 0.7.0 | 1.0.0 | Umbrella chart for the full Kates platform — Kafka, Kates, and supporting infrastructure | -| [`kates`](charts/kates/) | 0.10.11 | 1.25.1 | Kates — Kafka Advanced Testing & Engineering Suite | +| [`kates`](charts/kates/) | 0.11.1 | 1.25.1 | Kates — Kafka Advanced Testing & Engineering Suite | | [`legacy-kafka`](charts/legacy-kafka/) | 0.2.0 | 3.9.1 | A deliberately old Kafka (2.x on ZooKeeper, or 3.x on KRaft) to migrate FROM — the replication source that the pinned Strimzi release cannot deploy | | [`mirror-maker2`](charts/mirror-maker2/) | 0.11.4 | 4.3.1 | Strimzi KafkaMirrorMaker2 — cross-cluster replication, DR, and cross-version migration (2.x/3.x → 4.x) | | [`monitoring`](charts/monitoring/) | 1.6.0 | 82.4.3 | Kates Monitoring — wraps kube-prometheus-stack with Kates-specific dashboards and configuration | diff --git a/charts/kates/Chart.yaml b/charts/kates/Chart.yaml index caf1ec4d..c939b235 100644 --- a/charts/kates/Chart.yaml +++ b/charts/kates/Chart.yaml @@ -2,11 +2,11 @@ apiVersion: v2 name: kates description: Kates — Kafka Advanced Testing & Engineering Suite type: application -# Patch: the test run cleanup CronJob deletes runs at last, through the -# API's DELETE /api/tests with the apiKey Secret's key, and cleanup.statuses -# picks which finished runs it deletes. appVersion is unchanged; the job -# takes an image newer than it. -version: 0.10.11 +# Patch: the NetworkPolicy's Kafka egress follows networkPolicy.kafka.namespace +# instead of always `kafka`, opens Trogdor's 8889 there instead of in +# `monitoring`, and a new rule opens TCP 443/6443 for the Kubernetes API. +# appVersion is unchanged. +version: 0.11.1 # The non-blocking health endpoints this chart's probes now point at come from # a BUILD-TIME property (quarkus.messaging.health.enabled=false), so they only # exist in the image built from this branch. Keep appVersion and @@ -30,6 +30,8 @@ annotations: artifacthub.io/category: integration-delivery artifacthub.io/license: Apache-2.0 artifacthub.io/changes: | + - kind: fixed + description: "On a CNI that enforces NetworkPolicy, the chart's NetworkPolicy cut the backend off from the Kubernetes API, so every chaos fault, pod listing, KafkaTopic CR and orphan recovery failed; its Kafka egress always named namespace kafka, whatever networkPolicy.kafka.namespace said; and it allowed Trogdor's 8889 only to namespace monitoring, where no coordinator runs. The Kafka egress now follows networkPolicy.kafka.namespace (default kafka), 8889 moves from monitoring to that namespace, and a new ports-only rule opens TCP 443 and 6443 to any address for the Kubernetes API. Upgrading: a release that set networkPolicy.kafka.namespace to anything but the namespace Kafka runs in loses Kafka, as the value now takes effect; a Trogdor coordinator outside that namespace needs a networkPolicy.egressRules entry" - kind: changed description: "appVersion 1.25.1 and the image tag move to 1.25.1, the release of the deployment hardening in #368: engine timeout and reaper tunables wired through values and the configmap, stranded runs recovered more robustly, and network policy updates. Templates are otherwise unchanged." - kind: fixed diff --git a/charts/kates/README.md b/charts/kates/README.md index e7907b62..9aff3d7d 100644 --- a/charts/kates/README.md +++ b/charts/kates/README.md @@ -57,7 +57,7 @@ All configuration is in [values.yaml](values.yaml). Key sections: | `ingress.certManager.issuerName` | `""` | cert-manager issuer name | | `ingress.certManager.issuerKind` | `ClusterIssuer` | Issuer kind | | `networkPolicy.enabled` | `true` | Enable NetworkPolicy | -| `networkPolicy.kafka.namespace` | `kafka` | Kafka egress target namespace | +| `networkPolicy.kafka.namespace` | `kafka` | Namespace the brokers (`networkPolicy.kafka.port`) and the Trogdor coordinator (8889) run in; `kates deploy` sets it | | `networkPolicy.kafka.port` | `9092` | Kafka egress target port | | `networkPolicy.prometheus.namespace` | `monitoring` | Prometheus egress target namespace: the one `prometheus.url` points into | | `networkPolicy.prometheus.port` | `9090` | Prometheus egress target port | @@ -70,6 +70,16 @@ effect and moves the egress there while `prometheus.url` points into `monitoring`, so the backend loses Prometheus. Remove that line, or set it to the namespace your Prometheus runs in. +Upgrading from chart 0.11.0 or earlier: those charts ignored +`networkPolicy.kafka.namespace` and always allowed egress to `kafka`, allowed +Trogdor's 8889 only to `monitoring`, and had no egress to the Kubernetes API, +so on a CNI that enforces policy every chaos fault and pod listing failed. The +Kafka egress now follows the value, with 8889 next to the brokers, and TCP +443/6443 is open to any address. A release that set the value to anything but +where Kafka runs loses Kafka after the upgrade. A Trogdor coordinator outside +that namespace (`kates/docs/deployment.md` runs one in `kates`) needs a +`networkPolicy.egressRules` entry. + ### Security | Parameter | Default | Description | diff --git a/charts/kates/templates/networkpolicy.yaml b/charts/kates/templates/networkpolicy.yaml index 885b317c..096273fe 100644 --- a/charts/kates/templates/networkpolicy.yaml +++ b/charts/kates/templates/networkpolicy.yaml @@ -30,6 +30,13 @@ spec: protocol: UDP - port: 53 protocol: TCP + # The Kubernetes API, for chaos faults, pod listing, KafkaTopic CRs and the + # orphan reconciler; 6443 is the API server's own port after the Service DNAT. + - ports: + - port: 443 + protocol: TCP + - port: 6443 + protocol: TCP {{- if .Values.postgresql.enabled }} - to: - podSelector: @@ -60,13 +67,18 @@ spec: - port: {{ .Values.networkPolicy.cdcDatabase.port | default 5432 }} protocol: TCP {{- end }} + # The brokers, and the Trogdor coordinator on 8889 where the default + # trogdor.coordinatorUrl puts it. + # shortcut: a coordinator in another namespace needs a networkPolicy.egressRules entry. - to: - namespaceSelector: matchLabels: - kubernetes.io/metadata.name: kafka + kubernetes.io/metadata.name: {{ .Values.networkPolicy.kafka.namespace | default "kafka" | quote }} ports: - port: {{ .Values.networkPolicy.kafka.port | default 9092 }} protocol: TCP + - port: 8889 + protocol: TCP - to: - namespaceSelector: matchLabels: @@ -74,13 +86,6 @@ spec: ports: - port: {{ .Values.networkPolicy.prometheus.port | default 9090 }} protocol: TCP - - to: - - namespaceSelector: - matchLabels: - kubernetes.io/metadata.name: monitoring - ports: - - port: 8889 - protocol: TCP {{- with .Values.networkPolicy.egressRules }} {{- toYaml . | nindent 4 }} {{- end }} diff --git a/charts/kates/values.yaml b/charts/kates/values.yaml index 1554ec55..d33e7302 100644 --- a/charts/kates/values.yaml +++ b/charts/kates/values.yaml @@ -447,7 +447,7 @@ networkPolicy: ports: - port: 8080 protocol: TCP - # -- Kafka egress configuration + # -- Kafka egress: the brokers on port and Trogdor on 8889 (kates deploy sets namespace) kafka: namespace: kafka port: 9092 diff --git a/cli/cmd/deploy_components.go b/cli/cmd/deploy_components.go index 23bd6937..2460f3b9 100644 --- a/cli/cmd/deploy_components.go +++ b/cli/cmd/deploy_components.go @@ -1681,6 +1681,7 @@ data: katesArgs = append(katesArgs, "--set", "kafka.bootstrapServers="+katesBootstrap, "--set", "kafka.topicNamespace="+kafkaNS, + "--set", "networkPolicy.kafka.namespace="+kafkaNS, "--set", fmt.Sprintf("monitoring.enabled=%t", deployWithMonitoring)) katesArgs = append(katesArgs, dc.scrapeArgs("charts/kates")...) katesArgs = append(katesArgs, dc.prometheusArgs()...) diff --git a/cli/cmd/deploy_test.go b/cli/cmd/deploy_test.go index 32e03037..3436ff43 100644 --- a/cli/cmd/deploy_test.go +++ b/cli/cmd/deploy_test.go @@ -201,6 +201,9 @@ func TestDeployCommand_SingleTopology(t *testing.T) { if !strings.Contains(cmd, "-n kates-test") { t.Errorf("Expected Kates to be deployed in kates-test namespace, got: %s", cmd) } + if !strings.Contains(cmd, "--set networkPolicy.kafka.namespace=kates-test") { + t.Errorf("Expected the Kates NetworkPolicy to allow Kafka in kates-test, got: %s", cmd) + } } } @@ -276,6 +279,9 @@ func TestDeployCommand_IsolatedTopology(t *testing.T) { if !strings.Contains(cmd, "-n app-sys") { t.Errorf("Expected Kates to be deployed in app-sys namespace, got: %s", cmd) } + if !strings.Contains(cmd, "--set networkPolicy.kafka.namespace=kafka-sys") { + t.Errorf("Expected the Kates NetworkPolicy to allow Kafka in kafka-sys, got: %s", cmd) + } } if strings.Contains(cmd, "helm upgrade --install chaos") { foundChaos = true diff --git a/docs/book/appendix-d-versions.md b/docs/book/appendix-d-versions.md index 37f3f255..5012b7cd 100644 --- a/docs/book/appendix-d-versions.md +++ b/docs/book/appendix-d-versions.md @@ -22,7 +22,7 @@ Every version the platform pins, in one place. This table is generated from the | `kafka-ui` chart | 0.3.0 (app v1.5.0) | `charts/kafka-ui/Chart.yaml` | | `kates-chaos` chart | 2.2.1 (app 3.28.0) | `charts/kates-chaos/Chart.yaml` | | `kates-platform` chart | 0.7.0 (app 1.0.0) | `charts/kates-platform/Chart.yaml` | -| `kates` chart | 0.10.11 (app 1.25.1) | `charts/kates/Chart.yaml` | +| `kates` chart | 0.11.1 (app 1.25.1) | `charts/kates/Chart.yaml` | | `legacy-kafka` chart | 0.2.0 (app 3.9.1) | `charts/legacy-kafka/Chart.yaml` | | `mirror-maker2` chart | 0.11.4 (app 4.3.1) | `charts/mirror-maker2/Chart.yaml` | | `monitoring` chart | 1.6.0 (app 82.4.3) | `charts/monitoring/Chart.yaml` | diff --git a/docs/stretched-kafka/04-test-and-chaos-scenarios.md b/docs/stretched-kafka/04-test-and-chaos-scenarios.md index 3529a602..612045e5 100644 --- a/docs/stretched-kafka/04-test-and-chaos-scenarios.md +++ b/docs/stretched-kafka/04-test-and-chaos-scenarios.md @@ -486,7 +486,7 @@ service: type: NodePort nodePort: "30083" networkPolicy: - enabled: false # its egress rules leave out the Kubernetes API, which the chaos backend calls + enabled: false # untested in the lab with it on; see below extraEnv: - name: KATES_CHAOS_PROVIDER # simultaneous kills value: kubernetes @@ -512,7 +512,7 @@ kates ctx set site-a --url http://localhost:30083 --api-key "$KEY_A" kates ctx set site-b --url http://localhost:30084 --api-key "$KEY_B" ``` -`POD_KILL` needs no extra RBAC; leave `rbac.directChaos` off. The chart's NetworkPolicy allows egress only to DNS, PostgreSQL, Kafka and the `monitoring` namespace, so on a CNI that enforces policies it cuts Kates off from the API server its chaos backend and pod watcher use: disable it in the lab, or add a `networkPolicy.egressRules` entry for the API servers on the target. The chart's default `resources` request 2 GiB; lower them on a laptop. Reach each Kates through a NodePort or load balancer in its own site, not `kubectl port-forward`, which needs the API server that T1 and T9 take down. Read the API keys before any test for the same reason. +`POD_KILL` needs no extra RBAC; leave `rbac.directChaos` off. The lab values above leave the chart's NetworkPolicy off. Since chart 0.11.1 its egress allows TCP 443 and 6443 to any address, which covers the API server its chaos backend and pod watcher use, and Kafka in `networkPolicy.kafka.namespace` (`kafka` here); turning it on in the lab is untested. The chart's default `resources` request 2 GiB; lower them on a laptop. Reach each Kates through a NodePort or load balancer in its own site, not `kubectl port-forward`, which needs the API server that T1 and T9 take down. Read the API keys before any test for the same reason. ### Sustained Workloads