bench: fail fast on lost produce; kafka: survive host stalls#204
Merged
Conversation
The throughput bench produced fire-and-forget: no delivery callback and an unchecked flush(30), so events librdkafka silently expired after message.timeout.ms still counted as "produced" and the gate math judged a phantom load. Now every produce carries an on_delivery hook, the paced loop aborts on the first surfaced failure, and the final flush must drain clean. The dev stand's single-node KRaft broker heartbeats its own controller over TCP; a host stall past the default 9s session makes it fence itself and drop every partition leader. Session timeout raised to 45s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DORA Metrics
|
…ess loss) The paced100 series documented runs up to 1h; the two 4h attempts failed for stand and harness reasons worth keeping: r1 proves the apply path dedups a 292k replay storm exactly under checkpoint-storage failure, r3 proves the path processes 100% of delivered events across broker fencing blips and motivates the bench delivery guard. Multi-hour gate stays open until a clean pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A 4-hour soak run exposed two stacked stand/harness defects (neither in the serving path):
flush(30): when the broker briefly lost partition leadership, librdkafka silently expired queued messages for the rest of the run while the pace loop kept counting them as produced — the gate then judged a phantom load. Everyproduce()now carries anon_deliveryhook, the loop aborts (exit 3) on the first surfaced delivery failure, and the finalflush(120)must drain clean. Transient blips that librdkafka rides out withinmessage.timeout.msdo not abort.KAFKA_BROKER_SESSION_TIMEOUT_MSraised to 45 s in the dev compose.Also guards misnamed root-level audit/plan notes in .gitignore.
Verified: ruff check + format clean, py_compile, compose YAML valid; the guarded bench is currently driving a live 4 h re-run.
🤖 Generated with Claude Code