Repository navigation
fix(cli): report exports read each phase's name, metrics and SLA verdict - #382
Merged
Merged
Conversation
The backend writes a report phase as {phaseName, metrics, slaVerdict}, but
the CLI read name, status, recordsSent, throughputRecordsPerSec,
p99LatencyMs and errorRate, none of which exist. Every phase of a LOAD
report, produce and consume alike, came out of `report export --format md`
and `--format html` as "main ✓ 0 0.0 rec/s", a phase whose SLA failed
included. The client test encoded the Go struct, so it agreed with itself.
ReportPhase now has the backend's shape. Both renderers take the numbers
from the phase's metrics and mark the column, renamed from Status to SLA,
✓ or ✖ from its slaVerdict (— when there is none). The phase name is the
user's scenario text, so the HTML export escapes it and the Markdown one
escapes |. report show no longer sums phases to fill in a summary of 0:
they are built from the same rows as the summary, so the sum is 0 too.
`report show -o json` now prints phaseName, metrics and slaVerdict in
place of the old keys, which were always empty. TestReport serves the
backend's JSON as written.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The CLI never decoded a report's phases. Checked in the code:
ReportGeneratorbuilds each phase as aPhaseReportwithphaseName,metrics(aReportSummary) andslaVerdict(ReportGenerator.java:239-249,PhaseReport.java:18-22).docs/book/11-api-reference.mddocuments the same shape.client.ReportPhasereadname,status,recordsSent,throughputRecordsPerSec,p99LatencyMsanderrorRate(cli/client/types.go:289-296). None of these keys exist, so every field stayed empty.kates report export --format mdwrote each one as| main | ✓ | 0 | 0.0 rec/s | 0.00 | 0.0000% |: the empty name becamemain, and the empty status got the pass mark.--format htmldid the same with abadge-pass. A phase whose SLA failed was marked as passed too. These exports go into CI artifacts and PRs.ReportPhaseby hand in the Go shape, andTestReportserved the Go struct encoded as JSON, so the client agreed with itself.The finding missed two things:
phaseNamedecodes, it holds the phase names from the user's scenario file. fix(engine): refuse scenario values the database cannot store #329 limits only their length and NUL bytes.report_html.gowrote the name into the HTML unescaped, which would be an HTML injection in a file people share. In Markdown, a|in the name would split the row.report showfallback. When the summary is missing or its throughput is 0,report showsummed the phases to fill it in. Phases are built from the same rows as the summary, so their sum is 0 too. The branch never did anything, and it could only read the broken fields.Changes
cli/client/types.goReportPhasehas the backend's shape:Name(phaseName),Metrics *ReportSummaryandSlaVerdict *SlaVerdict, with a comment namingPhaseReport.cli/cmd/report_markdown.go,cli/cmd/report_html.gometrics. A phase with no metrics falls back to a zeroReportSummary.slaVerdict, and — when there is none. In HTML the cell is a✓ PASSEDor✖ FAILEDbadge, the same text as the overall SLA Verdict card.mainstand-in. HTML escapes it withhtml.EscapeString, and Markdown escapes|the way the violations table already does.cli/cmd/report.goreport showno longer sums phases. Only the phase branch of the fallback is gone. TheGetTestfallback, which sums the run's results, is unchanged apart from its indent.Visible changes
kates report show -o jsonre-encodesclient.Report, so each phase now printsphaseName,metricsandslaVerdict. The old keys (name,status,recordsSent,throughputRecordsPerSec,p99LatencyMs,errorRate) are gone. They were always empty or 0. Book 10 points users at this command for the full report as JSON.GET /api/tests/{id}, through theGetTestfallback. It reads the same rows, so the outcome is the same.Tests
TestReport(cli/client/client_test.go) serves a raw JSON literal in the backend's shape: a summary and two phases, each withphaseName,metricsandslaVerdict, the second one failed. It asserts two phases decode, the produce phase's throughput and P99, and the consume phase's name and failed verdict. The old struct decodes this JSON to two phases with no name and no numbers.TestRenderMarkdownReport_WithPhasesasserts:| produce | ✓ | 600 | 10000.0 rec/s | 15.50 | 0.0000% |;a|brenders asa\|b, and a phase with no verdict shows —;| main |row appears.TestRenderHTMLReport_WithPhasesasserts a passing phase getsbadge-pass, a failing one getsbadge-fail, and a phase named<script>renders as<script>with no raw<script>in the output.git diff --name-only origin/main...HEAD: 7 files, allcli/**Go (client/types.go,client_test.go,cmd/report*.goand their tests). They triggerci.yml(code filter),lint.yml(go), CodeQL (go),integration.ymlandci-docker.yml.cd cli && go test -p 1 -race -count=1 ./..., CI's flags: all packages ok (cmdin 147.8 s). TheTestMCPCaveat*tests ran inside thecmdpackage and pass.cd cli && go vet ./...: clean.cd cli && golangci-lint run --new-from-rev=origin/main ./...: 0 issues, with golangci-lint 2.14.0, which matches theGOLANGCI_LINT_VERSIONpin.cd cli && govulncheck ./...: no vulnerabilities found, with govulncheck v1.7.0, which matches the pin.scripts/check-cli-style.sh, withGOMEMLIMIT=2GiB GOGC=50as in CI.scripts/check-cli-compat.sh: all terminal-compatibility checks ok.helm lint charts/kates/,helm template kates charts/kates/andscripts/check-versions.sh. Theci.ymlHelm Lint job runs on the code filter, so it runs here too. The charts are unchanged; its Strimzi operator lint steps were not run.kates/file is touched. The full backend suite ran on main and passed, so it gives the same result here.integration.ymlKind Cluster Smoke Tests (cli/**trigger);ci-docker.ymlDocker image build validation (cli/**trigger);codeql.ymlGo CodeQL analysis;ci.ymlBuild & Test, the Java suite with the*ITtests (kates/unchanged);native.ymlnative image;github-advanced-securityAI scan, which is always red (GitHub-side).Decisions
ReportGeneratorjudges phases on their SLA alone, and the status belongs to the run.slaVerdict. The current backend always sets one, so this only matters for odd input, and it doesn't claim a pass nothing judged.report.go's fallback is deleted. TheGetTestbranch is just as dead against the current backend, which always sends a summary built from the same rows, but it still matters for an older Kates API that sent none.cli/cmd/data/doc_entries.yaml:817-818is left for a follow-up.Deliberately not done
cli/cmd/data/doc_entries.yaml:817-818still saysreport showdisplays per-phase breakdowns.report showrenders the summary and the SLA verdict, not the phases. This is left for a follow-up, as decided.ReportPhasehas nophaseTypeortimeSeries.ReportGeneratornever setsphaseType, and nothing in the CLI renderstimeSeries.Landing
Shares no file with the earlier plan PRs, so it can land at any time.