fix(tune): no command or endpoint ranks tuning steps that no run measured: the tuning report is gone

[bmscomp]

Plan PR 15 of the audit fix plan (finding 13). Breaking: GET /api/tests/{id}/report/tuning now answers 404 and kates tune report no longer exists, so scripts that call either one fail. Lands after the earlier plan PRs; see Landing.

Why

kates tune report and the endpoint behind it ranked configurations that no run measured. Checked in the code on main:

• A tuning run measures one configuration. kates tune run <type> sends no spec. TestTypeDefaults gives every TUNE_* type the LOAD replication factor, partitions and min ISR with acks "all", and TestOrchestrator (case TUNE_REPLICATION, …, :1725-1726) builds one produce task, <id>-tune-0, from that spec.
• The report invented the rest. TuningTestRunner.buildReport took 3 to 6 labelled configurations from the stepsFor tables ("RF=1, acks=1", "RF=2, acks=all", "RF=3, min.isr=2, acks=all" for TUNE_REPLICATION) and gave every step the same summary of that one run.
• The best step was always step 0. The best index moved only when summary.avgThroughputRecPerSec() > bestThroughput, starting from -1. All steps carry the same value, so only step 0 passes, and with no summary the index stays 0 anyway.
• So the recommendation was always the first, least durable option. buildRecommendation read "RF=1, acks=1 balances durability with acceptable throughput" for TUNE_REPLICATION and "acks=0 gives the best latency for the required safety level" for TUNE_ACKS. TUNE_COMPRESSION recommended none, TUNE_BATCHING 16KB / 0ms and TUNE_PARTITIONS 1 partition / 1 producer. On the tie, findWorstStep also picked step 0, so the CLI table showed a BEST row and no WORST row.
• Only agents were warned. The MCP caveat tuning-one-measurement recorded this. CLI and REST users got the recommendation with no warning.
• kates tune types repeated the claim. It reported stepsFor(type).size() steps (3 to 6) and descriptions such as "Sweeps batch.size and linger.ms to find optimal throughput". The README, the book, kates --help, kates test types and kates explain also called these types sweeps.

The ranking is deleted rather than faked. kates report show <id> already shows the one real measurement.

Changes

Backend (kates/)

• TuningTestRunner: buildReport, buildRecommendation and the stepsFor tables are deleted, with the repository and reportGenerator fields and the imports only they used. availableTuningTests reports steps: 1, and each description says what a run measures ("Measures throughput and latency at the spec's acks setting").
• report/TuningReport.java is deleted, and so are its two entries (TuningReport, TuningReport.TuningStep) in NativePayloadReflectionConfig.
• ReportResource: the injected tuningRunner field and GET {id}/report/tuning are gone. GET tuning/types stays; it calls the static availableTuningTests. Every line after the endpoint moves up 22.
• ReportingApiIT.tuningCatalogIsServedIndependentlyOfAnyRun: the 404 check for /report/tuning, its comment and the blank line above them are removed. The check would still pass, because a missing route also answers 404, but its comment would be false. The catalog assertion stays.

CLI (cli/)

• cmd/tune.go: tune report is gone, with tuneReportResult, tuneStepRow, buildTuneReport, renderTuneReport, metricValue, metricStr, findWorstStep and the tuneWorst style. tune run's hint reads View results: kates report show <id>. The tune and tune run Long texts say a run measures one configuration and compares none, and tune run points to kates lab for comparing settings.
• client/client.go and client/types.go: ReportTuning, TuningReport and TuningStep are deleted. TuningTypes and TuningTypeInfo stay.
• cmd/helpers.go describeType: the five TUNE_* descriptions, used by kates test types and kates explain, say "Measures throughput and latency at one …" instead of naming a sweep.
• cmd/root.go: the tune line in kates --help reads "Tuning tests, each measuring one configuration".
• cmd/data/doc_entries.yaml: the tune report entry is deleted. tune run and tune types are reworded. In tune run's see_also, report show replaces tune report; tune types just loses it.

kates mcp

• mcp_caveats.go: the tuning-one-measurement caveat keeps its ID. Its text is now "A TUNE_* run executes one produce task with the spec's single configuration: it measures one setting and compares none." Its ref to TuningTestRunner.java:101-139, which no longer exists, is dropped. The anchored TestOrchestrator.java:1725-1726 ref and SpecFactory.java:38-39 are unchanged.
• The comment refs into ReportResource.java move up 22, each checked against the new file: mcp_tools_runs.go 318-327 → 296-305 (computeDeltas), 293-305 → 271-283 (the compare loop), 307-313 → 285-291 (the deltas), 141-142 → 119-120 (Test run not found), and mcp_tools_runs_test.go 152-153 → 130-131.

Docs

• docs/book/10-cli-reference.md: the tune report section is deleted. The family table lists tune run, types and asks how one setting performs, not which performs best. The Tuning Commands intro says a run measures one setting, points to kates report show <id>, and points to kates lab, whose auto-sweep runs one test per value. The type table says each type measures the default.
• docs/book/05-test-types.md and appendix-a-glossary.md: "parameter-sweep" and "(parameter sweeps)" are dropped.
• README.md: "plus configuration sweeps (kates tune run)" is dropped.
• skills/kates-cli/SKILL.md: tune report leaves the JSON command list. The TUNE_* rule says a run measures one setting and compares none, and to read it with report show.

Tests

Added:

• TestTune_JSON's run subtest runs kates tune run batching without -o json and checks the output names kates report show run-2.

Removed with the code they tested:

• client_test.go TestReportTuning and the ReportTuning case in escape_test.go.
• TestTune_JSON's report subtest and its fixture, and runsBackend's tuning map with its handler case.

Kept: TestMCPGetRunTuningRun and TestMCPAssessRunScenarioAndTuning, since the caveat stays. The latter's check that no /report/tuning request is made stays valid.

Run locally, passing:

• git diff --name-only origin/main...HEAD: one commit, on a128f85. It changes 22 paths across README.md, cli/** (client, cmd, data/doc_entries.yaml), docs/book/** (05, 10, appendix-a), kates/** (main plus ReportingApiIT) and skills/kates-cli/SKILL.md. Sets that apply: Java, CLI, Book, Docs and yamllint (a .yaml file changed). Sets that do not: Charts, Makefile, workflows and shell. Main has since gained fix(make): the undeploy targets remove only their own Helm release, never the namespace's volumes or workloads #370, which touches none of these paths; the branch merges into main cleanly.
• cd kates && ./mvnw -o test -Dquarkus.grpc.codegen.skip=true -Dquarkus.http.test-port=0: BUILD SUCCESS in 3:19, 1466 tests, 0 failures, 0 errors, 0 skipped. Test-compile also compiled ReportingApiIT.
• ./mvnw -o spotless:check on JDK 25 (25.0.2-graalce): exit 0; the only output was sun.misc.Unsafe deprecation warnings.
• cd cli && go test ./...: every package ok, including cmd with the caveat-ref anchor tests. go test -p 1 ./... -race -count=1, the CI CLI Tests flags: exit 0.
• go vet ./...: clean. golangci-lint run --new-from-rev=origin/main ./...: 0 issues. govulncheck ./...: no vulnerabilities.
• scripts/check-cli-style.sh and scripts/check-cli-compat.sh: OK.
• scripts/check-book-style.sh: exit 0. The ratchet prints below-baseline counts with an optional --update hint; they appear on unchanged pages too, and nothing fails.
• scripts/gen-version-matrix.sh --check: in sync. scripts/gen-chart-table.sh --check: README chart table in sync.
• python3 scripts/check-book-contrast.py: all 170 colour pairs meet WCAG AA. scripts/check-mermaid.sh: 156 diagrams, 0 invalid.
• A scratch-copy HTML render of the book (prepare-book-render.sh, quarto render docs/book --to html): 39 chapters prepared, 38 pages rendered, 0 warnings. check-book-fragments.py on it: all 7,453 fragment links land on an id.
• yamllint -c .yamllint.yaml .github config charts plus cli/cmd/data/doc_entries.yaml: clean.

Skipped locally:

• lychee --offline --include-fragments: not installed. I checked by hand the 4 link targets the diff touches (#tuning-commands, 05-test-types.md#test-type-overview, 10-cli-reference.md, 10b-lab.md); all exist, and the rendered fragment check passed.
• helm lint / template and check-versions.sh: no charts/** or helm-filter path changed.
• make check-help, actionlint and shellcheck: no Makefile, .github/** or scripts/** change.

CI only:

• ci.yml Build & Test: ./mvnw verify, which runs the *IT tests (including the edited ReportingApiIT) under Docker/Testcontainers
• ci.yml Dependency CVE scan: Trivy over the Maven dependencies
• native.yml: Native image build and smoke test (kates/** changed). Removing a registered DTO is what this job checks.
• integration.yml: Kind Cluster Smoke Tests (cli/** and kates/** changed)
• ci-docker.yml: Docker image builds
• codeql.yml: CodeQL analysis
• docs.yml link-check: lychee offline (not installed locally)
• book.yml: PDF render
• github-advanced-security AI scan (always red, GitHub-side)

Decisions

• Remove the tuning report (option b) rather than report one honest step. It is mostly deletion, and kates report show <id> already shows the one real measurement. The cost is the breaking removal above.
• stepsFor is dropped too. kates tune types reports 1 step, with reworded descriptions. Keeping the tables would have left tune types advertising the same invented sweep.
• The caveat is kept and rewritten, not retired. A TUNE_* run still measures one setting, and an agent should still be told not to rank settings from it.
• bestStepIndex and recommendation go with the types. Nothing returns them now.
• @ApplicationScoped stays on TuningTestRunner. Nothing injects it any more, but it is harmless.
• The hint is kates report show <id>. kates report <id> does not exist: reportCmd has no RunE.
• Wording. tune run's Long text and the book say a run measures "the backend's defaults for the type", which is what tune run gets, since it sends no spec. The caveat keeps the decided wording, "the spec's single configuration", which also covers a TUNE_* run created with a spec of its own.
• skills/kates-cli/SKILL.md is edited now. eval/mcp/run.py records the skill's skill_sha256 in a run's manifest (check_manifest), so a pending MCP eval run resumed after this merges will refuse until it is restarted or its manifest is updated.

Deliberately not done

• eval/mcp/test_metrics.py:179 still has "kates tune report x": False. It passes, because classify works on strings, and the decisions do not cover it.
• TestTune_JSON's types subtest fixture still serves Steps: 4 and "Sweep batch.size" from the fake backend. It only checks that the CLI relays the answer.
• plans/mcp-server.md still cites TuningTestRunner.java:101-139 and describes the tuning report. It is an old plan doc whose refs were already stale (TestOrchestrator.java:973-974), so it is left as is.
• No deprecation period for the endpoint or the command: both go in this PR.

Landing

Lands after #370 (plan PR 1), #371 (plan PR 2), #372 (plan PR 3), #373 (plan PR 4), #374 (plan PR 5), #375 (plan PR 6), #376 (plan PR 7), #377 (plan PR 8), #378 (plan PR 9), #379 (plan PR 10), #380 (plan PR 11), #381 (plan PR 12), #382 (plan PR 13) and #384 (plan PR 14). The files they share with this PR:

• fix(chart): a failed pg_dump fails the kates backup job, and backup.enabled without the bundled database fails rendering #372 and fix(chart): the kates NetworkPolicy lets the API reach the Kubernetes API, Trogdor, and Kafka in networkPolicy.kafka.namespace #384: README.md.
• fix(migrate): up and run refuse a lab whose mirrored topics already exist on the target, and a run refused at its prerequisites removes nothing #373, fix(engine): run retention keeps the run a baseline names, so regression checks keep working #378 and fix(cli): kates ctx use, delete, export, import and current exit 1 when they fail #380: docs/book/10-cli-reference.md (fix(engine): run retention keeps the run a baseline names, so regression checks keep working #378 also cli/cmd/mcp_caveats.go).
• fix(chaos): orphan recovery undoes every fault left from before the Kates API started, and NetworkPolicies in any namespace #374: cli/cmd/mcp_tools_runs.go.
• fix(chaos): compound runs and resilience faults go through the safety guard and the disruption lease #375: cli/cmd/mcp_tools_runs.go, docs/book/05-test-types.md, docs/book/10-cli-reference.md, docs/book/appendix-a-glossary.md.
• fix(engine): each INTEGRITY run reads in a group of its own, and a consumer with no committed offset starts at its task's first record, not the topic's oldest #376: cli/cmd/mcp_caveats.go, cli/cmd/mcp_tools_runs.go, docs/book/05-test-types.md.
• fix(engine): a status poll writes only over the status it read, so a cancelled run stays cancelled #377: cli/cmd/mcp_caveats.go, cli/cmd/mcp_tools_runs.go, cli/cmd/mcp_tools_runs_test.go.
• fix(security): the audit and pentest read each listener's protocol, so Kafka's default GSSAPI and named listeners no longer pass #379: cli/cmd/mcp_caveats.go, skills/kates-cli/SKILL.md.
• fix(cli): report exports read each phase's name, metrics and SLA verdict #382: cli/client/client_test.go, cli/client/types.go.
• fix(make): the undeploy targets remove only their own Helm release, never the namespace's volumes or workloads #370 (merged), fix(native): the native image registers the API keys file's YAML model for reflection, so it starts with a named keys file #371 and fix(cli): kates test apply refuses a scenario spec number key that holds something other than a number #381 share no file with this PR.

A trial merge of this branch with each open PR's head is clean, and none of them adds a ref into ReportResource.java or TuningTestRunner.java or mentions tune report.

Rebase recipe: rebase onto main once they have merged. Then re-point any caveat or comment ref the merged changes moved, from a diff line map. The five ReportResource.java refs here are this PR's own move. The tuning caveat's TestOrchestrator.java:1725-1726 ref is anchored, so a merged edit that moves it shows up in the anchor test. Then rerun cd cli && go test ./cmd -run TestMCPCaveat.