Skip to content

Don't write tracking snapshots ahead of the log - #240

Draft
ansd wants to merge 1 commit into
mainfrom
rollover
Draft

ansd wants to merge 1 commit into
mainfrom
rollover

Conversation

@ansd

@ansd ansd commented Aug 17, 2026

Copy link
Copy Markdown
Member

osiris_writer:handle_batch/2 evaluates the tracking snapshot before writing the batch's own chunk, but passed it a tracking state that already contained that batch's updates, and osiris_tracking:snapshot/3 additionally cleared the pending deltas. When a batch rolled the segment over, the resulting CHNK_TRK_SNAPSHOT therefore recorded tracking for entries that were only written in the chunk following it, and that chunk was written with an empty trailer:

chunk_id=4 CHNK_TRK_SNAPSHOT sequences=[{<<"app">>,{4,40}}]
chunk_id=5 CHNK_USER trailer_size=0 <- holds the msg for 40

A node dying in between those two writes - a plain SIGKILL suffices, and recover_tracking/1 only scans the current segment - then recovers a writer sequence for a message that is not in the log.

For RabbitMQ this breaks stream publisher deduplication in the worst possible direction. Clients that resume from query_publisher_sequence, which is how exactly-once read-process-write loops are built on streams, skip the messages the sequence claims were written. They cannot repair the gap either: writing those messages again is detected as a duplicate, so they are dropped while still being confirmed. The messages are unrecoverable and nothing reports an error. Replication does not help, as the snapshot chunk is replicated and committed independently of the chunk that follows it.

Build the snapshot from the tracking as it exists in the log: every tracking id with a pending update is reverted to the value last written for it, and left out entirely when it has never been written. Keep the pending deltas so the batch's own chunk carries them in its trailer again, restoring the property that a message and its writer sequence are always in the same chunk, i.e. persisted atomically. Skip the snapshot chunk when nothing has been written yet, since recover_tracking/1 treats a snapshot as the complete tracking state at that point in the log.

Deduplication and query behaviour are unchanged: query/3 and overview/1 keep reporting pending values. The chunk format is unchanged, so logs stay readable by and replicable to and from older versions.

osiris_writer:handle_batch/2 evaluates the tracking snapshot before
writing the batch's own chunk, but passed it a tracking state that
already contained that batch's updates, and osiris_tracking:snapshot/3
additionally cleared the pending deltas. When a batch rolled the segment
over, the resulting CHNK_TRK_SNAPSHOT therefore recorded tracking for
entries that were only written in the chunk following it, and that chunk
was written with an empty trailer:

  chunk_id=4 CHNK_TRK_SNAPSHOT sequences=[{<<"app">>,{4,40}}]
  chunk_id=5 CHNK_USER         trailer_size=0  <- holds the msg for 40

A node dying in between those two writes - a plain SIGKILL suffices, and
recover_tracking/1 only scans the current segment - then recovers a
writer sequence for a message that is not in the log.

For RabbitMQ this breaks stream publisher deduplication in the worst
possible direction. Clients that resume from query_publisher_sequence,
which is how exactly-once read-process-write loops are built on streams,
skip the messages the sequence claims were written. They cannot repair
the gap either: writing those messages again is detected as a duplicate,
so they are dropped while still being confirmed. The messages are
unrecoverable and nothing reports an error. Replication does not help,
as the snapshot chunk is replicated and committed independently of the
chunk that follows it.

Build the snapshot from the tracking as it exists in the log: every
tracking id with a pending update is reverted to the value last written
for it, and left out entirely when it has never been written. Keep the
pending deltas so the batch's own chunk carries them in its trailer
again, restoring the property that a message and its writer sequence are
always in the same chunk, i.e. persisted atomically. Skip the snapshot
chunk when nothing has been written yet, since recover_tracking/1 treats
a snapshot as the complete tracking state at that point in the log.

Deduplication and query behaviour are unchanged: query/3 and overview/1
keep reporting pending values. The chunk format is unchanged, so logs
stay readable by and replicable to and from older versions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant