fix(chaos): orphan recovery undoes every fault left from before the Kates API started, and NetworkPolicies in any namespace

[bmscomp]

Why

Startup orphan recovery left every fault that a normal restart leaves behind. Checked in the code (line numbers are main's):

• The cutoff was "now minus 15 minutes". reconcileNetworkPolicies and reconcileScaleDowns (DisruptionOrphanReconciler.java:107, :148) took System.currentTimeMillis() - minAgeSec * 1000. kates.chaos.orphan-recovery.min-age-sec was 900 both in the @ConfigProperty default and in application.properties:198.
• The reason for that age no longer holds. The class javadoc said Kates may run more than one replica, so a recent fault could belong to a live plan on a peer. values.schema.json pins replicaCount to 1 and strategy.type to Recreate, so no peer can exist, and a restart always lands inside the window.
• Case. A plan's SCALE_DOWN step lowers brokers-sigma. The Kates API is OOM-killed by its in-JVM benchmark, or a helm upgrade arrives. KatesExecutor.shutdown() doesn't interrupt the plan thread, so no rollback runs. The new pod is up about 90 s later. Recovery sees a 2-minute-old scaled-down-at snapshot and skips it, and since it runs only once, at startup, it never looks again. The pool stays a broker short until some later restart comes more than 15 minutes after the fault, and the report says only INTERRUPTED. A kubernetes-provider partition behaves the same way: kates-netpol-<pod> keeps the broker cut off.
• Other namespaces, never. Policies were listed and deleted in kates.chaos.kafka.namespace only (:113, :124). The consumer-isolation playbook targets namespace kates, so its policies were never recovered at any age, although KubernetesChaosProvider.cleanup() deletes managed-by=kates policies inAnyNamespace().
• Book 07 and book 11 documented the 15-minute gap.

The finding's fix (cutoff at the process start, default 0, list in any namespace) needed four corrections, all applied here:

• application.properties:198 sets 900 explicitly, so it has to change too, not only the @ConfigProperty default.
• With inAnyNamespace(), each delete has to use the policy's own namespace, not kafkaNamespace.
• A cutoff of exactly startedAt has a gap. creationTimestamp has whole seconds, so a policy this process makes in its first second can be stamped before startedAt and deleted while live. The cutoff is truncated to whole seconds. The scaled-down-at annotation is in milliseconds, so truncating doesn't change anything for scale-downs.
• Scale-down snapshots outside the Kafka namespace are still not restored, so book 07 has to say so rather than claim full coverage.

Changes

kates/.../disruption/DisruptionOrphanReconciler.java

• New cutoff(startedAt): the process start, which markInterruptedReports already uses, truncated to whole seconds, minus min-age-sec. reconcile passes startedAt to reconcileNetworkPolicies and reconcileScaleDowns, and both use it in place of their currentTimeMillis() cutoffs.
• The min-age-sec default is "0".
• reconcileNetworkPolicies lists managed-by=kates policies inAnyNamespace() and deletes each one from its own namespace. The log line names namespace/name. The method is package-private now, for the test.
• The list call keeps its try and its log message, and returns 0 when it fails. Each delete has its own try, which logs namespace/name and moves on to the next policy.
• Scale-downs are still restored in kates.chaos.kafka.namespace only.
• The class javadoc's "Why age matters" is now "Why the process start is the cutoff", with the one-replica reason. The comment in onStart says recovery cleans faults made before this process started.

kates/src/main/resources/application.properties

• Line 198: kates.chaos.orphan-recovery.min-age-sec=0, edited in place so that no CLI ref into the file moves.

kates mcp refs

• Refs into DisruptionOrphanReconciler.java are re-pointed through a diff line map: mcp_tools_chaos.go:285 goes from 53-54,62-69,85-92,152-203 to 54-55,63-70,86-93,171-222, and mcp_tools_runs.go:2391 and the comment at :1931 go from 152-203 to 171-222.
• No caveat text changes. The refs into application.properties are untouched.

Docs

• Book 07, the orphan recovery paragraph: policies in every namespace, every fault made before the Kates API started however recent (the chart runs one pod), and min-age-sec, 0 by default, moving the cutoff earlier.
• Book 07, the Safety Guardrails gaps: policies in every namespace, scale-downs in the Kafka namespace only, and how to put such a StatefulSet back by hand from its kates.io/original-replicas annotation.
• Book 11, the Orphan recovery note.
• docs/stretched-kafka/03-kubernetes-and-strimzi.md:581 drops "only once they are 15 min old" and keeps the rest of the sentence. The scope's docs list didn't name this file, but the clause is false after this change.
• cli/examples/resilience-scale-down.yaml: orphan recovery brings the broker back the next time the Kates API starts.

Tests

• DisruptionOrphanReconcilerTest (fabric8 crud mock server, VertxPerMockClient kept), 2 new tests, 3 in all:
  • restoresANodePoolAbandonedMidPlanAndLeavesARecentScaleDownAlone, updated: minAgeSec = 0 and the cutoff from startedAt. A pool scaled down 2 minutes before the start (the 90-second restart) is now restored. One scaled down 1 s after the start stays at 0.
  • deletesKatesNetworkPoliciesFromBeforeTheStartInEveryNamespace: Kates-managed policies in kafka and kates, and one unlabelled policy. With startedAt 60 s ago nothing is deleted. With startedAt 60 s ahead, both Kates policies are deleted from their own namespaces, and the unlabelled one stays. The fabric8 7.2 crud handler overwrites creationTimestamp on create, so the test moves the start instead (the split the scope allowed).
  • aPolicyStampedInTheSecondTheProcessStartedIsKept: startedAt is the mock's stamp plus 700 ms, and the policy is kept. This guards the truncation.
• Run locally, all passing unless marked skipped:
  • git diff --name-only origin/main...HEAD: one commit on origin/main, touching 9 paths: kates/** (DisruptionOrphanReconciler.java and its test, application.properties), cli/** (mcp_tools_chaos.go, mcp_tools_runs.go, examples/resilience-scale-down.yaml), docs/book/** (07, 11) and docs/stretched-kafka/03. CI jobs triggered: ci.yml code (Build & Test, CLI, govulncheck), lint.yml go/java/yaml, docs.yml, book.yml, native.yml.
  • cd kates && ./mvnw -o test -Dquarkus.grpc.codegen.skip=true -Dquarkus.http.test-port=0: BUILD SUCCESS, 1,468 tests, 0 failures, 0 errors, 0 skipped. DisruptionOrphanReconcilerTest ran all 3 tests, and ResilienceExamplesTest read the edited cli/examples file.
  • spotless:check on JDK 25 (25.0.2-graalce): BUILD SUCCESS.
  • cd cli && go test ./...: all packages ok, with cli/cmd run fresh rather than from cache. That covers TestMCPCaveatRefAnchors, which checks the moved DisruptionOrphanReconciler.java refs.
  • cd cli && go vet ./...: clean.
  • cd cli && golangci-lint run --new-from-rev=origin/main ./...: 0 issues.
  • cd cli && govulncheck ./...: no vulnerabilities found.
  • bash scripts/check-cli-style.sh and bash scripts/check-cli-compat.sh: OK.
  • scripts/check-book-style.sh: OK, and the structure ratchet holds for all 38 pages.
  • scripts/gen-version-matrix.sh --check and scripts/gen-chart-table.sh --check: in sync (the chart table check also covers the docs/stretched-kafka change).
  • python3 scripts/check-book-contrast.py: all 170 colour pairs meet WCAG AA.
  • scripts/check-mermaid.sh --quiet: 156 diagrams, 0 invalid.
  • A scratch-copy HTML render of the book (prepare-book-render.sh, then quarto render docs/book --to html): 39 chapters prepared, 38 pages rendered, 0 warnings in the render log. The worktree wasn't touched.
  • python3 scripts/check-book-fragments.py on that render: all 7,458 fragment links in 38 pages land on an id.
  • yamllint with .yamllint.yaml on .github, config and charts (CI's yaml job scope), and on cli/examples/resilience-scale-down.yaml: 0 errors. The only warnings were already there (for example charts/kates/values-corporate.yaml:142 empty-lines), and the edited example lints clean.
  • lychee --offline --include-fragments (docs.yml's link check): skipped, since lychee isn't installed locally. The docs diff adds or removes no Markdown link, so CI's run is the only check.
  • actionlint, shellcheck, make check-help, helm lint/helm template and check-versions.sh: not triggered, since this PR changes no .github/**, scripts/**, Makefile, charts/** or versions.env.
• Runs only in CI:
  • the Failsafe *IT tests (ci.yml Build & Test runs ./mvnw verify, and Docker stays off locally);
  • the native image build and smoke test (native.yml, triggered by kates/**);
  • the book PDF render (book.yml pdf job);
  • the Markdown link check with lychee (docs.yml);
  • the Kind/integration tests and the Docker image builds;
  • JUnit/JaCoCo report publishing;
  • the github-advanced-security AI scan, which is red on every PR for a GitHub-side reason.

Decisions

• min-age-sec defaults to 0, with the cutoff truncated to whole seconds. A margin would move the cutoff earlier, so it would skip orphans the old process made in its last seconds before it died, as in an in-place OOM restart where a native boot takes about 1 s. It protects nothing the truncation doesn't already protect.
• The explicit =0 line stays in application.properties:198, edited in place. Deleting it and relying on the @ConfigProperty default would shift four CLI refs at line 274 and below.
• Each policy delete gets its own try. The whole loop used to share one try, so the first delete that failed (a 403 in one namespace, say) stopped recovery in every other namespace, and that matters more now that the loop covers all of them. This goes beyond the finding. It is a few more lines than the three-line move the scope described, because the list call keeps its own try rather than nesting one try inside another.
• Scale-down snapshots outside the Kafka namespace are still not restored. ScaleDownSnapshots.restore works per namespace, while SCALE_DOWN can target spec.targetNamespace(). Listing StatefulSets cluster-wide is a bigger RBAC and cost question. Book 07 documents the gap, and it is left as a follow-up.

Deliberately not done

• Scale-downs in other namespaces (see Decisions). This is a follow-up, and no chip is filed yet.
• RBAC is unchanged. Deleting in any namespace needs cluster-scope delete on networkpolicies. The chart grants it only under rbac.directChaos, in a ClusterRole (charts/kates/templates/rbac.yaml:54-67). kates/k8s/rbac.yaml grants only get and list, so deletes there still fail with 403 and are logged, as before.
• Clock skew. The API server stamps creationTimestamp. If the Kates node's clock is ahead by Δ, a partition this process creates within Δ of startup can be deleted early. If it is behind, an orphan created within Δ before startup is left. NTP keeps Δ to milliseconds.
• A second Kates pod is not guarded against. With min-age 0, a user who bypasses the schema and runs two pods would have one pod's startup remove the other's live chaos. The chart and kates/k8s/deployment.yaml (1 replica, Recreate) rule that out. min-age-sec is still there for such a deployment.
• An interrupted plan's report still says only INTERRUPTED. Nothing records whether its faults were undone. This PR doesn't change that.

Landing

This PR shares no file with the earlier plan PRs (#370, #371, #372, #373), so it can land at any time.