fix(chart): the kates NetworkPolicy lets the API reach the Kubernetes API, Trogdor, and Kafka in networkPolicy.kafka.namespace

[bmscomp]

Plan PR 14 of the audit fix plan (finding 10). Lands after #372 and #375; see Landing.

Why

On a CNI that enforces NetworkPolicy (Calico, Cilium or GKE Dataplane V2, EKS network policy), the kates chart's default policy cuts the Kates API off from three things it needs. Checked in charts/kates/templates/networkpolicy.yaml on main:

• No Kubernetes API egress. The egress list allows DNS, PostgreSQL, Kafka, Prometheus and 8889 to monitoring, and nothing on 443 or 6443. Every fabric8 call fails after its retry backoff: every chaos fault, disruption plan, pod listing and KafkaTopic CR, and the orphan reconciler. kates-chaos's own policy has this rule; the kates chart never did.
• Kafka is always namespace kafka. The Kafka rule names kubernetes.io/metadata.name: kafka literally and never reads networkPolicy.kafka.namespace, although NOTES.txt prints that value as the Kafka egress target. With kates deploy --topology single -n kates-stack, Kafka runs in kates-stack, and Kates cannot reach it at all. The pod still goes Ready, because readiness reads a cached Kafka check, so the deploy looks healthy.
• Trogdor's 8889 is open only to monitoring. The default trogdor.coordinatorUrl is http://trogdor-coordinator.kafka.svc:8889 (values.yaml:388), and nothing in the repo listens on 8889 in monitoring. fix(kates): resolve strict network policy blocks and static init crash #35 (8eab57e) added that rule as Trogdor egress to any namespace, and feat(networkpolicy): harden all NetworkPolicies for default-deny clusters #42 (cf00a9f) narrowed it to monitoring by mistake. This part matters only for runs on the trogdor backend; engine.defaultBackend is native.

docs/stretched-kafka/04-test-and-chaos-scenarios.md documented the API-server gap and told the lab to disable the policy. That does not cover the chart default or kates deploy on a generic cluster, and after this PR the text is wrong.

Changes

charts/kates/templates/networkpolicy.yaml

• A ports-only rule after DNS opens TCP 443 and 6443 to any address, for the Kubernetes API. 6443 is the API server's own port after the Service DNAT. It is the same shape as kafka-common.netpol.apiServerEgress, written inline because the kates chart does not depend on kafka-common.
• The Kafka rule's namespace is {{ .Values.networkPolicy.kafka.namespace | default "kafka" | quote }}, and it opens 8889 next to the broker port, for the Trogdor coordinator where the default URL puts it. A shortcut: comment says a coordinator in another namespace needs a networkPolicy.egressRules entry.
• The monitoring 8889 rule is deleted.

charts/kates/values.yaml: the # -- Kafka egress configuration comment is reworded in place to say the rule covers the brokers on port and Trogdor on 8889, and that kates deploy sets the namespace. No key and no line is added, so values.schema.json is unchanged and no caveat ref into values.yaml moves.

cli/cmd/deploy_components.go: the kates helm call sets networkPolicy.kafka.namespace=<kafkaNS> next to kafka.topicNamespace. kafkaNS is the namespace kates deploy installs Kafka in, so --topology single gets the shared namespace and the isolated topology gets the Kafka one.

Chart version and docs

• charts/kates/Chart.yaml: 0.11.1, with the patch comment rewritten and a kind: fixed artifacthub.io/changes entry that says what changes on upgrade. appVersion stays 1.25.1.
• README.md chart table and docs/book/appendix-d-versions.md: 0.11.1.
• charts/kates/README.md: the networkPolicy.kafka.namespace row says the namespace is where the brokers and the Trogdor coordinator run, and that kates deploy sets it. A new "Upgrading from chart 0.11.0 or earlier" note says the value now takes effect, so a release that set it to anything but where Kafka runs loses Kafka, and that a coordinator outside that namespace (kates/docs/deployment.md runs one in kates) needs an egressRules entry.
• docs/stretched-kafka/04-test-and-chaos-scenarios.md: the lab values comment and the paragraph below no longer say the policy blocks the API server. The lab still runs with the policy off, and the text says turning it on there is untested.

.github/workflows/ci.yml: a helm-lint step, after Template validation, renders the policy with networkPolicy.kafka.namespace=kates-stack and checks the namespace is "kates-stack", no egress names kafka, 6443 is there, and 8889 is open once, to the Kafka namespace. Nothing rendered and checked the value before, which is how it went unread.

Tests

Added:

• TestDeployCommand_SingleTopology: the kates helm call carries --set networkPolicy.kafka.namespace=kates-test, the namespace Kafka and Kates share.
• TestDeployCommand_IsolatedTopology: the kates helm call carries --set networkPolicy.kafka.namespace=kafka-sys while Kates installs in app-sys.
• The ci.yml render assertion above.

The backend is unchanged: this PR touches nothing under kates/. The full backend suite (./mvnw verify) ran on main and passed, and gives the same result here.

Run locally, passing:

• git diff --name-only origin/main...HEAD: 1 commit on a128f85. The paths trigger ci.yml (helm-lint, kyverno, yaml-validation, go-tests, dependency-scan, build-and-test), lint.yml (actionlint, yamllint, golangci-lint), docs.yml (chart table, version matrix, book style, contrast, mermaid, lychee), book.yml and the ci-kafka-charts dashboards job.
• cd cli && go test ./...: every package ok, cli/cmd in 33 s. This covers the two new assertions and TestMCPCaveatRefAnchors.
• go vet ./...: no findings. golangci-lint run --new-from-rev=origin/main ./...: 0 issues. govulncheck ./...: no vulnerabilities.
• scripts/check-cli-style.sh and scripts/check-cli-compat.sh: OK.
• helm lint charts/kates/: 1 chart linted, 0 failed.
• helm template kates charts/kates/ with the defaults and with values-dev, -generic, -kind, -local, -native-local, -native and -staging: each renders. values-prod and values-corporate render once externalDatabase.existingSecret is set; values-corporate needs it too, like prod.
• The new ci.yml step, run as a bash script: OK, with the kates-stack namespace, no literal kafka, 6443 present, and 8889 only in kates-stack.
• scripts/check-versions.sh: every pin agrees. scripts/check-chart-test-paths.sh: both chart-test /api paths resolve.
• helm template with kyvernoPolicy.enabled piped to kyverno apply --resource kates/k8s/deployment.yaml: 0 fail, 0 error.
• yamllint -c .yamllint.yaml .github config charts (the lint.yml scope): no errors. actionlint: clean, shellcheck over the new run block included.
• scripts/gen-chart-table.sh --check (README table at 0.11.1) and scripts/gen-version-matrix.sh --check (appendix-d matches the pins).
• scripts/check-book-style.sh: OK, ratchet within baseline. check-book-contrast.py: all 170 colour pairs meet WCAG AA. check-mermaid.sh: 156 diagrams, 0 invalid.
• A scratch-copy HTML render of the book (prepare-book-render.sh, quarto render --to html): 38 pages, and check-book-fragments.py finds all 7,458 fragment links landing on an id.
• scripts/check-dashboards.py: 12 dashboards pass.

Skipped locally:

• gen-dashboards.py --check: PyYAML is not installed. The PR touches no dashboard or chart dashboard copy.
• lychee --offline --include-fragments: lychee is not installed. The docs diff adds or changes no link, only version cells and prose.
• ci.yml yaml-validation (PyYAML over config/**): config/ is unchanged and PyYAML is missing.
• shellcheck and make check-help: no shell script or Makefile changed.

CI only:

• kates ./mvnw verify, surefire and the *IT failsafe tests (Build & Test, triggered by cli/** and ci.yml)
• Dependency CVE scan (Trivy over kates/)
• kubeconform over every rendered chart (not installed locally)
• Lint and template of all charts with fetched dependencies (kafka-cluster, kates-chaos, monitoring, mirror-maker2, connect-cluster, kates-platform), plus strimzi-operator lint and overlays
• ci-kafka-charts dashboards job: gen-dashboards.py --check, promtool PromQL checks, Grafana render and install bundle checks
• docs.yml lychee markdown link check
• book.yml PDF render
• Native image build: native.yml is not triggered, as no kates/** path changes
• Kind and live tests: integration.yml. metrics-live is not triggered by charts/kates
• Docker image builds (ci-docker.yml)
• CodeQL and security.yml scans

Decisions

• Chart 0.11.1. fix(chart): a failed pg_dump fails the kates backup job, and backup.enabled without the bundled database fails rendering #372 (plan PR 3) takes 0.11.0 and is meant to land first, so this one takes the next patch. The chart README's upgrade note reads "from chart 0.11.0 or earlier" to match.
• 443/6443 open to any address, the ports-only rule kafka-common uses, rather than a new networkPolicy.apiServer.ipBlock value. A new value would add values.yaml and schema lines and move the caveat ref at mcp_tools_scenario.go, which anchors values.yaml:605-673.
• 8889 in the Kafka-namespace rule, where the default trogdor.coordinatorUrl puts the coordinator, and the monitoring 8889 rule deleted. Deriving the namespace from the URL, or a new networkPolicy.trogdor.namespace key, was left out.
• The ci.yml render assertion is added. It touches .github, which brings in actionlint.
• Stretched-kafka doc: text only. The lab values keep networkPolicy.enabled: false; turning it on there is untested.
• NOTES.txt unchanged. Its line naming networkPolicy.kafka.namespace as the Kafka egress target is true once the template reads the value.
• The namespace renders quoted ("kafka" by default), because of | quote. The CI assertion's check for a leftover kafka matches both the quoted and the bare form.
• kates-platform umbrella gap, documented here and not fixed. Checked by rendering: the umbrella installs Kafka and Kates in the release namespace but never sets kates.networkPolicy.kafka.namespace, so installed in kates-stack, its kates policy still sends Kafka egress to "kafka". It was broken before this PR too (the literal kafka). Fixing it means setting that value at install time, because Helm values cannot read .Release.Namespace.

Deliberately not done

• No live test on a cluster whose CNI enforces policy: Docker is off here. values-kind.yaml turns the policy off, so the Kind smoke test in CI does not exercise it either; only the static render assertion guards the template.
• kubeconform was not run locally (not installed).
• OTel trace export to jaeger-collector.monitoring.svc:4317 is still outside the egress list. A follow-up.
• The 443/6443 rule opens HTTPS egress to every address, which also covers webhook targets and the OAuth token endpoint. Narrowing it needs the apiServer.ipBlock value left out above.
• A Trogdor coordinator outside the Kafka namespace, including one in Kates' own namespace as kates/docs/deployment.md lays it out, is still blocked without an egressRules entry. The chart README says so.
• A stale ref that predates this PR is left as is: docs/mirror-maker2-cli-migration-plan.md cites .github/workflows/ci.yml:359 for go test -p 1, which was already at :418 on main and is at :438 after this PR.

Landing

Lands after #372 (plan PR 3) and #375 (plan PR 6), which edit files this PR also edits:

• fix(chart): a failed pg_dump fails the kates backup job, and backup.enabled without the bundled database fails rendering #372: .github/workflows/ci.yml, README.md, charts/kates/Chart.yaml, charts/kates/README.md, charts/kates/values.yaml, docs/book/appendix-d-versions.md. The ci.yml step (both add one after Template validation), the Chart.yaml version, comment and changes entries, README.md:216 and appendix-d:25 conflict mechanically: keep both ci.yml steps and both changes entries, and take 0.11.1. The charts/kates/README.md and values.yaml hunks are in different sections.
• fix(chaos): compound runs and resilience faults go through the safety guard and the disruption lease #375: docs/stretched-kafka/04-test-and-chaos-scenarios.md, in a different hunk (:594 against :489 and :515).

Rebase recipe: rebase onto main once both have merged, resolve as above, then re-point any caveat ref the merged changes moved from a diff line map (this PR moves none of its own: the values.yaml edit keeps its line count), and rerun cd cli && go test ./cmd -run TestMCPCaveat along with gen-chart-table.sh --check, gen-version-matrix.sh --check and the ci.yml assertion.