Skip to content

[CI] Wait for the flush in testBatchSizeFromConfig instead of sleeping 4 s - #42

Merged
PetrHeinz merged 1 commit into
mainfrom
claude/fix-flaky-batch-config-size-test
Sep 29, 2026
Merged

PetrHeinz merged 1 commit into
mainfrom
claude/fix-flaky-batch-config-size-test

Conversation

@PetrHeinz

@PetrHeinz PetrHeinz commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

LogtailAppenderBatchConfigSizeTest.testBatchSizeFromConfig logs 200 lines so that the batch size from logback-batch-test.xml makes the appender send the batch on another thread, to the real ingesting endpoint. It then slept a fixed 4 s and required exactly one API call and a 202 response by then.

Why it flakes

It failed six attempts in the last 54 Java Build runs (since 2026-07-17), each time passing on a later attempt thanks to the nick-fields/retry wrapper:

When Jobs Failure What the appender logged
2026-09-29 build (11), build (17) testBatchSizeFromConfig:66->isOk:76, bare AssertionError nothing, neither an error nor a retry: the request was simply still in flight at 4 s
2026-09-08, push to main all four jobs of run 34244332450 three times testBatchSizeFromConfig:64 expected:<1> but was:<2>, once isOk:76 Connection reset / Remote host terminated the handshake, then Retrying to send 200 logs to Better Stack (1 / 5)

So it fails both when a request takes longer than the sleep and when the appender has to retry a failed request, which is the appender doing its job. Apart from 60 s build timeouts, the only other test that failed an attempt and passed on a retry in those runs is testConnectTimeout (3 jobs), which #36 reworks.

Change

  • Replace Thread.sleep(4000) with LogtailAppenderDecorator.awaitFlushCompletion(). It takes and releases the appender's flushLock, so it returns once the flush started by the 200th line is done, retries included.
  • Assert exactly one call accepted by the endpoint (acceptedCalls, new in the decorator) instead of apiCalls == 1. The test still checks that the batch size from the XML sends the batch in one request, but no longer fails when a first attempt was reset and retried. The assertions that nothing is sent before the 200th line keep using apiCalls.
  • In the usual case the test no longer waits the full 4 s.

LogtailAppenderShutdownTest and LogtailAppenderIntegrationTest don't have this pattern, so they're unchanged: the 2 s sleep in the shutdown test is the window in which nothing may be sent (5 lines, batch size 10, 3 s interval), and the integration test already calls awaitFlushCompletion() before isOk().

Verification

The test needs a real source token, which I don't have locally, so the run against the real endpoint is this PR's CI: run 36562884171 is green on Java 8, 11, 17 and 20. There testBatchSizeFromConfig took 3.4 s (11), 3.1 s (17), 2.2 s (20) and 2.8 s / 1.8 s (8), so the flush alone usually takes 2-3.5 s and the old 4 s sleep left as little as 0.6 s of headroom. Build (8) needed a second attempt because of testConnectTimeout:150, the flake #36 reworks; testBatchSizeFromConfig passed in both of its attempts.

Locally I pointed BETTER_STACK_INGESTING_HOST at an HTTPS stand-in:

Stand-in answers origin/main this branch
202 after 5 s fails at isOk:76, bare AssertionError, 4.1 s passes, 5.2 s, one POST with 200 lines
503, then 202 to the appender's retry fails at :64 expected:<1> but was:<2>, 4.1 s passes, 0.6 s
202 right away not run 30 of 30 runs pass, 0.75 s in total

The whole suite against the stand-in passes except testConnectTimeout and testReadTimeout, which expect 1 ms timeouts that a localhost connection doesn't produce.

Notes

  • awaitFlushCompletion() still gives the flush thread 10 ms to take the lock before waiting on it. testBatchDefaultBatchSize relies on the same timing (it asserts apiCalls == 1 10 ms after the 1000th line) and didn't fail once in those 54 runs, so I left the helper as it is.
  • If the endpoint fails all six attempts the appender makes, the batch is dropped and isOk() still fails, with the error printed.
  • Only test files are touched: LogtailAppenderBatchConfigSizeTest and LogtailAppenderDecorator, neither of which the open PRs T-1365 Keep the appender running while the JVM shuts down #34-Explain the ingesting host placeholder in the examples #41 change.

🤖 Generated with Claude Code

…g 4 s

The 200th line starts the flush on another thread and the request goes to
the real ingesting endpoint. The test slept a fixed 4 s and then required
exactly one API call and a 202 response, so it failed whenever the request
took longer (a bare AssertionError from isOk()) or had failed and been
retried by the appender by then (expected:<1> but was:<2>) - six times
since 2026-09-08, hidden by the retry wrapper in the Java Build workflow.

Wait for the flush with LogtailAppenderDecorator.awaitFlushCompletion()
instead, and count the calls the endpoint accepted rather than all calls,
so the test still checks that the batch went out in exactly one request
while the appender's retries do their job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PetrHeinz
PetrHeinz marked this pull request as ready for review September 29, 2026 12:11
@PetrHeinz
PetrHeinz merged commit 7b6dcb3 into main Sep 29, 2026
6 checks passed
@PetrHeinz
PetrHeinz deleted the claude/fix-flaky-batch-config-size-test branch September 29, 2026 12:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant