Skip to content

Fix multi-row binlog checkpointing - #60

Merged
coding-chimp merged 1 commit into
masterfrom
bb/checkpoint-safety
Sep 30, 2026
Merged

coding-chimp merged 1 commit into
masterfrom
bb/checkpoint-safety

Conversation

@coding-chimp

@coding-chimp coding-chimp commented Sep 28, 2026 •

Copy link
Copy Markdown

Summary

Fix the existing risk of checkpointing a partially applied binlog transaction. Per-row coordinates are progress, not proof that every change sharing those coordinates has reached the ghost table.

  • Send ordered transaction-completion markers through the existing reader, dispatcher, and apply queues.
  • Advance a separate applied-transaction watermark only after preceding destination SQL commits; persist that completed boundary rather than sampled reader or heartbeat progress.
  • Preserve bounded queues and DML batching across markers. Failed SQL batches cannot acknowledge collected markers.
  • Stop synchronous dispatch after listener errors and reject marker acknowledgement after abort. Make reader operations, checkpoint waits, and checkpoint SQL observe migration cancellation.
  • Preserve MySQL and MariaDB GTID coordinate ownership through cloning.

Testing

  • Full GOTOOLCHAIN=auto go test ./... suite passed.
  • Short suite and targeted race tests passed.
  • golangci-lint v2.11.4: 0 issues.

Route ordered completion markers through the existing DML queues and persist a separate applied-transaction watermark only after preceding SQL commits. Preserve batching and reject acknowledgements after failed delivery or migration cancellation.

Make reader operations and checkpoint SQL cancellation-aware. Cover multi-row events, MySQL/MariaDB GTID cloning, partial-apply replay, failed batches, and cancellation during both checkpoint polling and an in-flight INSERT. Keep go-mysql at v1.15.0; compressed payload support is a separate change.
@coding-chimp
coding-chimp changed the base branch from master to sync-upstream-2026-11-22 September 28, 2026 15:54
@coding-chimp
coding-chimp changed the base branch from sync-upstream-2026-11-22 to master September 28, 2026 15:54
@coding-chimp
coding-chimp added this pull request to stack #62 September 28, 2026 15:54
@coding-chimp coding-chimp self-assigned this Sep 28, 2026
Coordinates: coords.Clone(),
TransactionComplete: true,
}
if err := base.SendWithContext(gmr.migrationContext.GetContext(), entriesChannel, marker); err != nil {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're increasing the throughput of entriesChannel. I suspect this shouldn't realistically affect anything, but just pointing out there's an extra BinlogEntry for every transaction.

@coding-chimp
coding-chimp merged commit e3122d4 into master Sep 30, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants