Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cli/cmd/mcp_tools_chaos.go
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ var mcpCaveatsChaos = []mcpCaveat{
"unknown or was unavailable when the backend started, the backend falls back to noop, which injects " +
"nothing and marks every step Skipped, and says so only in its log.",
Refs: []string{
mcpAnchoredRef(mcpJava+"chaos/CompoundChaosOrchestrator.java:150-169",
mcpAnchoredRef(mcpJava+"chaos/CompoundChaosOrchestrator.java:155-174",
"List<String> availableProviders()", `" (available)" : " (unavailable)"`, "return names;"),
mcpAnchoredRef(mcpJava+"chaos/ChaosCoordinator.java:34-75",
`name = "kates.chaos.provider"`, "matches no chaos provider"),
Expand Down
4 changes: 2 additions & 2 deletions cli/cmd/mcp_tools_runs.go
Original file line number Diff line number Diff line change
Expand Up @@ -2392,9 +2392,9 @@ var mcpCaveatsRuns = []mcpCaveat{
mcpJava + "disruption/DisruptionTemplateResource.java:63-65",
mcpJava + "disruption/DisruptionScheduler.java:95-107",
mcpJava + "disruption/DisruptionReportEntity.java:39-47",
mcpAnchoredRef(mcpJava+"disruption/DisruptionAnalysisResource.java:110-141",
mcpAnchoredRef(mcpJava+"disruption/DisruptionAnalysisResource.java:118-192",
`@Path("/compound")`, `"results", outcome.results()`),
mcpAnchoredRef(mcpJava+"resilience/ResilienceResource.java:97-127",
mcpAnchoredRef(mcpJava+"resilience/ResilienceResource.java:107-137",
"StreamingOutput executeWithKeepAlive(", "objectMapper.writeValue(os, payload);",
`"Failed to execute resilience test"`),
},
Expand Down
2 changes: 1 addition & 1 deletion cli/cmd/mcp_tools_scenario.go
Original file line number Diff line number Diff line change
Expand Up @@ -1286,7 +1286,7 @@ var mcpCaveatsScenario = []mcpCaveat{
"switch (task.getWorkloadType())", "case INTEGRITY -> runIntegrity(task, state);",
"void runIntegrity(BenchmarkTask task, WorkerState state)", "state.chaosStartNanos.get()"),
mcpJava + "engine/TrogdorBackend.java:162-163",
mcpAnchoredRef(mcpJava+"resilience/ResilienceOrchestrator.java:144-145",
mcpAnchoredRef(mcpJava+"resilience/ResilienceOrchestrator.java:158-159",
"chaosCoordinator.triggerFault(", "testOrchestrator.markChaosStart(runId, injectedAt)"),
},
},
Expand Down
2 changes: 1 addition & 1 deletion docs/book/01-introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ Every run produces a JSON report, which you can export as CSV, as JUnit XML for

### 5. Safe by Default

Kates refuses a disruption plan that would hit every broker, or more brokers than the plan's `maxAffectedBrokers` allows. Before each fault, it checks that every Kafka pod is Ready. By default, when a step fails, [rollback](appendix-a-glossary.md#gl-disruption-rollback) gives back a broker that a scale-down removed and deletes the NetworkPolicies Kates created for a network partition. This safety guard counts brokers, not replicas. It doesn't read the ISR or the KRaft quorum, and a [resilience run](appendix-a-glossary.md#gl-resilience-run) doesn't go through it. [Chaos Engineering in Practice](07-chaos-practice.md#safety-guardrails) says exactly what it checks.
Kates refuses a disruption plan that would hit every broker, or more brokers than the plan's `maxAffectedBrokers` allows. Before each fault, it checks that every Kafka pod is Ready. By default, when a step fails, [rollback](appendix-a-glossary.md#gl-disruption-rollback) gives back a broker that a scale-down removed and deletes the NetworkPolicies Kates created for a network partition. This safety guard counts brokers, not replicas. It doesn't read the ISR or the KRaft quorum, and a [resilience run](appendix-a-glossary.md#gl-resilience-run) goes through only part of it: the broker count and the one-at-a-time rule before its test starts, with no Ready check before the fault and no rollback (see [Limits of the Guard](07-chaos-practice.md#limits-of-the-guard)). [Chaos Engineering in Practice](07-chaos-practice.md#safety-guardrails) says exactly what it checks.

## Feature Overview

Expand Down
4 changes: 2 additions & 2 deletions docs/book/02-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ Kates's [disruption types](appendix-a-glossary.md#gl-disruption-type) run from k

### Safety Guardrails

Before a disruption plan touches the cluster, Kates counts the brokers its steps would hit, and refuses the plan if that's every broker, or more than the plan's `maxAffectedBrokers`. Before each fault it checks that every Kafka pod is Running and Ready. When a step fails with `autoRollback` on, [rollback](appendix-a-glossary.md#gl-disruption-rollback) gives a scaled-down node pool its broker back, or deletes the [NetworkPolicies](appendix-a-glossary.md#gl-networkpolicy) Kates created for a network partition. The safety guard doesn't read the [ISR](appendix-a-glossary.md#gl-isr) or the [KRaft](appendix-a-glossary.md#gl-kraft) quorum, and a resilience run doesn't go through it: [Chaos Engineering in Practice](07-chaos-practice.md#safety-guardrails) says exactly what it checks.
Before a disruption plan touches the cluster, Kates counts the brokers its steps would hit, and refuses the plan if that's every broker, or more than the plan's `maxAffectedBrokers`. Before each fault it checks that every Kafka pod is Running and Ready. When a step fails with `autoRollback` on, [rollback](appendix-a-glossary.md#gl-disruption-rollback) gives a scaled-down node pool its broker back, or deletes the [NetworkPolicies](appendix-a-glossary.md#gl-networkpolicy) Kates created for a network partition. The safety guard doesn't read the [ISR](appendix-a-glossary.md#gl-isr) or the [KRaft](appendix-a-glossary.md#gl-kraft) quorum, and a resilience run goes through only part of it: the broker count and the one-at-a-time rule before its test starts, with no Ready check before the fault and no rollback (see [Limits of the Guard](07-chaos-practice.md#limits-of-the-guard)). [Chaos Engineering in Practice](07-chaos-practice.md#safety-guardrails) says exactly what it checks.

## CLI Architecture

Expand Down Expand Up @@ -519,7 +519,7 @@ kates report show <id>
- Kates has two halves: the Go CLI on your machine and the Kates API, one Quarkus service in the cluster that serves REST and gRPC.
- The `TestOrchestrator` owns a run's lifecycle: resolve defaults, create the topic, launch workers, poll status and collect heatmap rows; reports are built on request.
- Latency measurement rests on HdrHistogram (1µs–60s, microsecond precision), compressed into 25 heatmap buckets for export.
- The safety guard refuses a disruption plan that would hit too many brokers, and rollback undoes some faults; a resilience run skips both.
- The safety guard refuses a disruption plan or resilience run that would hit too many brokers, and rollback undoes some faults; a resilience run skips rollback.
- Results live in PostgreSQL, not Kafka; heatmaps, security grades and built reports live only in the Kates API's memory.

[The Cluster Under Test](03-cluster.md) comes next: it builds the Kubernetes and Kafka environment that the Kates API runs in and tests.
Expand Down
2 changes: 1 addition & 1 deletion docs/book/05-test-types.md
Original file line number Diff line number Diff line change
Expand Up @@ -485,7 +485,7 @@ scenarios:

### Integrity + Chaos

The real power of INTEGRITY tests emerges when combined with chaos engineering. An INTEGRITY run injects no [fault](appendix-a-glossary.md#gl-fault) by itself: `kates resilience run` pairs a test request with one fault, injected while the test runs. That pairing is a [resilience run](appendix-a-glossary.md#gl-resilience-run), one of the two ways Kates runs a [chaos experiment](appendix-a-glossary.md#gl-chaos-experiment), and it doesn't go through the [safety guard](appendix-a-glossary.md#gl-safety-guard) that checks a disruption plan (see [Chaos Engineering in Practice](07-chaos-practice.md)):
The real power of INTEGRITY tests emerges when combined with chaos engineering. An INTEGRITY run injects no [fault](appendix-a-glossary.md#gl-fault) by itself: `kates resilience run` pairs a test request with one fault, injected while the test runs. That pairing is a [resilience run](appendix-a-glossary.md#gl-resilience-run), one of the two ways Kates runs a [chaos experiment](appendix-a-glossary.md#gl-chaos-experiment), and it goes through only part of the [safety guard](appendix-a-glossary.md#gl-safety-guard) that checks a disruption plan: the broker count and the one-at-a-time rule before its test starts, with no Ready check before the fault and no rollback (see [Limits of the Guard](07-chaos-practice.md#limits-of-the-guard)):

```yaml
# resilience-integrity.yaml
Expand Down
16 changes: 8 additions & 8 deletions docs/book/07-chaos-practice.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,8 @@ The table sets a plan and a resilience run side by side. Read the guard, rollbac
| Faults | One per step | One |
| Workload | None | The Kates test in `testRequest` |
| What it measures | Pod recovery, Prometheus snapshots before and after each fault, and ISR and lag when asked | The test's throughput, latency and error rate just before the fault, and again over the whole run after the recovery wait |
| Safety guard | Checks the plan before the first fault, and the Kafka pods before each fault | None, apart from the [fault parameter limits](#fault-parameter-limits) |
| Another plan running | Refused | Not checked |
| Safety guard | Checks the plan before the first fault, and the Kafka pods before each fault | Counts the brokers the fault hits, and holds it to the [fault parameter limits](#fault-parameter-limits), before the test starts; doesn't check the Kafka pods before the fault |
| Another disruption running | Refused | Refused with `409` before the test starts; a run that finds one when `steadyStateSec` is up ends `ERROR` with no fault |
| Rollback | After a step fails its recovery check or errors, with `autoRollback` | None |
| Grade | An [SLA grade](appendix-a-glossary.md#gl-sla-grade) when the plan has an `sla` block | None |
| Report | Saved: `kates disruption status <id>` and `kates chaos list` | Printed once; the [test run](appendix-a-glossary.md#gl-test-run) is saved like any other |
Expand All @@ -85,8 +85,8 @@ The table sets a plan and a resilience run side by side. Read the guard, rollbac

Both paths get the fault itself from the same chaos provider, which [Choosing a Chaos Provider](#choosing-a-chaos-provider) describes, so the same fault spec (a plan step's `faultSpec`, a resilience file's `chaosSpec`) does the same thing either way. The one difference is aiming: only a plan step looks up a partition's [leader](appendix-a-glossary.md#gl-partition-leader) for you, when the step names a `targetTopic`.

::: {.callout-caution title="A Resilience Run Has No Safety Guard"}
Nothing checks a resilience run's `chaosSpec` against `maxAffectedBrokers` or waits for a running plan to finish, and nothing rolls its fault back. Aim it with a selector that matches only the pods you mean to hit, such as the brokers.
::: {.callout-caution title="A Resilience Run Has No Rollback"}
Kates refuses a resilience run whose `chaosSpec` would leave no broker untouched, and one started while another disruption holds the cluster. A resilience run has no `maxAffectedBrokers`, though, nothing checks that the Kafka pods are Ready before its fault, and nothing rolls the fault back. Aim it with a selector that matches only the pods you mean to hit, such as the brokers.
:::

## Disruption Types
Expand Down Expand Up @@ -578,7 +578,7 @@ Kates runs the plan anyway, with a warning in `validationWarnings`, in three cas
- A step is a `SCALE_DOWN`, whose broker stays removed until rollback puts it back.
- The `sla` block sets a threshold a plan can't evaluate (see [SLA Grading](#sla-grading)).

Only one plan runs against the cluster at a time. While one runs, Kates refuses a second, with `409 Conflict` over the API, because both would rely on the original replica counts that rollback restores from.
Only one disruption runs against the cluster at a time, whether a plan, a compound run or a resilience run's fault. While one runs, Kates refuses another, with `409 Conflict` over the API, because two would take down brokers together, and two plans would both rely on the original replica counts that rollback restores from. A resilience run is refused before its test starts, and checked again just before its fault: one that finds the cluster taken then ends `ERROR` with no fault injected.

### What Kates Checks Before Each Fault

Expand All @@ -588,7 +588,7 @@ If one isn't, the step fails with `Cluster is not in a stable baseline state bef

### Fault Parameter Limits

Every number in a fault spec has a range, and Kates refuses a fault with one outside it before anything is injected. A plan is checked before its first fault, whether you post it or it comes from a playbook, a template or a schedule, and one with a fault outside its range ends `REJECTED`. A resilience run's `chaosSpec` is checked before its test starts, and every fault of a compound run (`POST /api/disruptions/compound`) before any goes in; both get a `400` naming each parameter. The ranges are generous on purpose: they stop a typo or a runaway value, not a fault you mean to run. The table gives each range with the defaults, and the Kates API setting that raises its ceiling:
Every number in a fault spec has a range, and Kates refuses a fault with one outside it before anything is injected. A plan is checked before its first fault, whether you post it or it comes from a playbook, a template or a schedule, and one with a fault outside its range ends `REJECTED`. A resilience run's `chaosSpec` is checked before its test starts, and every fault of a compound run (`POST /api/disruptions/compound`) before any goes in; both get a `400` naming each parameter. Both are then counted by the safety guard, each fault as a plan step, and get a `400` with the guard's reasons when it refuses them. The ranges are generous on purpose: they stop a typo or a runaway value, not a fault you mean to run. The table gives each range with the defaults, and the Kates API setting that raises its ceiling:

| Parameter | Range with the defaults | Setting for the ceiling |
|-----------|-------------------------|-------------------------|
Expand Down Expand Up @@ -656,7 +656,7 @@ The guard counts brokers, reads pod readiness and, for a drain, where pods run,
- A step whose selector looks in another namespace, such as the consumers `consumer-isolation` isolates in `kates`, counts no broker, whatever it hits, unless it's a `NODE_DRAIN`.
- Steps that pick one pod at random from the same selector count as one broker between them, although each can hit a different broker.
- A `NODE_DRAIN` step counts the brokers on its node as the pods run when you submit the plan. A pod that moves to another node before the step runs takes the drain with it.
- A resilience run (`kates resilience run`) doesn't go through the guard at all: no broker count, no one-plan rule, no check before the fault and no rollback. Only the [fault parameter limits](#fault-parameter-limits) apply to it.
- A resilience run (`kates resilience run`) gets the broker count before its test starts, with no `maxAffectedBrokers`, and the one-at-a-time rule then and again just before its fault. It gets no check of the Kafka pods before the fault and no rollback.
- `make gameday` doesn't go through it either: it deletes a broker pod with `kubectl`.
- The one-plan rule lives in the Kates API process, so it holds only while one Kates API pod runs. The `kates` chart refuses a second replica and rolls with `Recreate`, so it never runs two.
- Orphan recovery covers the Kafka namespace only. A NetworkPolicy that a Kates API left in another namespace when it died stays until you run `kubectl delete networkpolicy -n <namespace> -l managed-by=kates`.
Expand Down Expand Up @@ -817,7 +817,7 @@ The Kates API emits these events under the plan's name, not under the disruption

## Resilience Testing: Performance + Chaos Combined

The `kates resilience run` command combines a performance test with chaos injection, providing a **before/after impact analysis**. It is the second of the [Two Ways to Run a Fault](#two-ways-to-run-a-fault), and it doesn't go through the safety guard: see [Limits of the Guard](#limits-of-the-guard). The diagram shows its phases:
The `kates resilience run` command combines a performance test with chaos injection, providing a **before/after impact analysis**. It is the second of the [Two Ways to Run a Fault](#two-ways-to-run-a-fault), and the safety guard checks less of it than of a plan: see [Limits of the Guard](#limits-of-the-guard). The diagram shows its phases:

```mermaid
%%| label: fig-practice-resilience-run
Expand Down
2 changes: 1 addition & 1 deletion docs/book/10-cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -1067,7 +1067,7 @@ kates chaos show <id>

### Resilience

Combined performance and chaos testing: a resilience run starts one Kates test, injects one fault while it runs, and prints the change in throughput, latency and error rate from before the fault. Unlike a disruption plan, it has no safety guard, rollback or grade, as [Chaos Engineering in Practice](07-chaos-practice.md) explains.
Combined performance and chaos testing: a resilience run starts one Kates test, injects one fault while it runs, and prints the change in throughput, latency and error rate from before the fault. Unlike a disruption plan, it has no rollback or grade, and the safety guard checks less of it, as [Chaos Engineering in Practice](07-chaos-practice.md) explains.

```bash
kates resilience run -f resilience-test.yaml
Expand Down
Loading
Loading