Skip to content

test(chaos): ProbeExecutorTest's exec stub keeps the stream open until the client has read the exit status - #383

Merged
bmscomp merged 1 commit into
mainfrom
fix/probe-exec-status-race
Oct 17, 2026
Merged

bmscomp merged 1 commit into
mainfrom
fix/probe-exec-status-race

Conversation

@bmscomp

@bmscomp bmscomp commented Oct 17, 2026

Copy link
Copy Markdown
Owner

ProbeExecutorTest.cmdProbeComparesTheOutputOfACommandThatExitedZero failed CI on #374 (run 39061200842, attempt 1) with the command in pod krafter-brokers-0 ended without an exit status.

Cause: the mock server, not ProbeExecutor

fabric8 mockwebserver 7.2.0 WebSocketSession.onOpen calls send() once per message. Each call adds the message to pendingMessages, then schedules it on a 1-thread executor. After each send, the executor closes the socket if pendingMessages is empty. When the executor sent the output before onOpen had added the status, it closed the socket first. The status went out after the close, and exitCode() completed with null. The waitFor(50) from #368 did not help: a ready-made StatusStreamMessage keeps a delay of 0.

Production cannot hit this. fabric8's VertxWebSocket reports the close from Vert.x's endHandler, which runs in frame order, so a status the API server sends before its close frame is always read first.

Fix

execWithStatus also expects a client message that never comes (.expect("never sent").andEmit("unused").always()). While the mock is waiting for a client message, it never closes the socket itself. The client closes it once it has read the status. waitFor(50) becomes immediately(), since it never added a delay.

exec(...) is unchanged. It sends one message and no status, for the test of a stream that ends without one, and a single message cannot race.

Checks (local)

  • On main's stub, with @RepeatedTest(500) on the two exit-status tests: 1 failure in 1,000 runs, with the CI error.
  • With the fix, @RepeatedTest(2000): 0 failures in 4,000 runs, and no slowdown (about 117 ms per run against 108 ms).
  • ProbeExecutorTest, 20 runs in a row: all 48 tests passed every time.
  • The whole kates suite: 1,466 tests passed. spotless:check on JDK 25 passed, and so did go test ./cmd -run TestMCPCaveat. No caveat ref cites this file.

…l the client has read the exit status

cmdProbeComparesTheOutputOfACommandThatExitedZero failed PR #374's CI
with "ended without an exit status". The cause is in fabric8's
mockwebserver. Its WebSocketSession queues the open messages one at a
time, and its executor closes the socket as soon as no message is
pending. When it sent the output before the status was queued, it closed
the socket first, and exitCode() completed with null. waitFor(50) did not
help, because a ready-made StatusStreamMessage keeps a delay of 0. On
main, the two exit-status tests failed 1 run in 1,000.

execWithStatus now also expects a client message that never comes. The
mock keeps a socket open while it waits for one, so the client closes
the socket itself once it has read the status: 0 failures in 4,000 runs.

ProbeExecutor is unchanged. fabric8's Vert.x socket reports the close in
frame order, so a status the API server sends before its close frame is
always read first.
@bmscomp
bmscomp merged commit 3e94c67 into main Oct 17, 2026
56 checks passed
@bmscomp
bmscomp deleted the fix/probe-exec-status-race branch October 17, 2026 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant