Conversation
osiris_writer:handle_batch/2 evaluates the tracking snapshot before
writing the batch's own chunk, but passed it a tracking state that
already contained that batch's updates, and osiris_tracking:snapshot/3
additionally cleared the pending deltas. When a batch rolled the segment
over, the resulting CHNK_TRK_SNAPSHOT therefore recorded tracking for
entries that were only written in the chunk following it, and that chunk
was written with an empty trailer:
chunk_id=4 CHNK_TRK_SNAPSHOT sequences=[{<<"app">>,{4,40}}]
chunk_id=5 CHNK_USER trailer_size=0 <- holds the msg for 40
A node dying in between those two writes - a plain SIGKILL suffices, and
recover_tracking/1 only scans the current segment - then recovers a
writer sequence for a message that is not in the log.
For RabbitMQ this breaks stream publisher deduplication in the worst
possible direction. Clients that resume from query_publisher_sequence,
which is how exactly-once read-process-write loops are built on streams,
skip the messages the sequence claims were written. They cannot repair
the gap either: writing those messages again is detected as a duplicate,
so they are dropped while still being confirmed. The messages are
unrecoverable and nothing reports an error. Replication does not help,
as the snapshot chunk is replicated and committed independently of the
chunk that follows it.
Build the snapshot from the tracking as it exists in the log: every
tracking id with a pending update is reverted to the value last written
for it, and left out entirely when it has never been written. Keep the
pending deltas so the batch's own chunk carries them in its trailer
again, restoring the property that a message and its writer sequence are
always in the same chunk, i.e. persisted atomically. Skip the snapshot
chunk when nothing has been written yet, since recover_tracking/1 treats
a snapshot as the complete tracking state at that point in the log.
Deduplication and query behaviour are unchanged: query/3 and overview/1
keep reporting pending values. The chunk format is unchanged, so logs
stay readable by and replicable to and from older versions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
osiris_writer:handle_batch/2 evaluates the tracking snapshot before writing the batch's own chunk, but passed it a tracking state that already contained that batch's updates, and osiris_tracking:snapshot/3 additionally cleared the pending deltas. When a batch rolled the segment over, the resulting CHNK_TRK_SNAPSHOT therefore recorded tracking for entries that were only written in the chunk following it, and that chunk was written with an empty trailer:
chunk_id=4 CHNK_TRK_SNAPSHOT sequences=[{<<"app">>,{4,40}}]
chunk_id=5 CHNK_USER trailer_size=0 <- holds the msg for 40
A node dying in between those two writes - a plain SIGKILL suffices, and recover_tracking/1 only scans the current segment - then recovers a writer sequence for a message that is not in the log.
For RabbitMQ this breaks stream publisher deduplication in the worst possible direction. Clients that resume from query_publisher_sequence, which is how exactly-once read-process-write loops are built on streams, skip the messages the sequence claims were written. They cannot repair the gap either: writing those messages again is detected as a duplicate, so they are dropped while still being confirmed. The messages are unrecoverable and nothing reports an error. Replication does not help, as the snapshot chunk is replicated and committed independently of the chunk that follows it.
Build the snapshot from the tracking as it exists in the log: every tracking id with a pending update is reverted to the value last written for it, and left out entirely when it has never been written. Keep the pending deltas so the batch's own chunk carries them in its trailer again, restoring the property that a message and its writer sequence are always in the same chunk, i.e. persisted atomically. Skip the snapshot chunk when nothing has been written yet, since recover_tracking/1 treats a snapshot as the complete tracking state at that point in the log.
Deduplication and query behaviour are unchanged: query/3 and overview/1 keep reporting pending values. The chunk format is unchanged, so logs stay readable by and replicable to and from older versions.