Repository navigation
chore: rebase the Bytebase patches onto upstream v1.1.11, and fix an instant-DDL deadlock - #10
Open
d-bytebase wants to merge 76 commits into
Open
d-bytebase wants to merge 76 commits into
d-bytebase wants to merge 76 commits into
Conversation
…ithub#1661) * Fix Warning 1300 for varbinary columns with bytes invalid as utf8mb4 When gh-ost replays a binlog DML event, the go-mysql library returns varbinary column values as a Go `string` (not `[]byte`). In convertArg, the existing code only converted string → []byte when the column had a non-empty Charset (e.g. utf8mb4 for varchar). varbinary columns have no character set, so Charset is always "", and the string fell through unconverted. The Go MySQL driver sends `string` args as MYSQL_TYPE_VAR_STRING with the connection's utf8mb4 charset metadata attached, causing MySQL to validate the bytes. If a varbinary value (e.g. a binary UUID) contains byte sequences that are invalid utf8mb4, MySQL emits Warning 1300. With gh-ost's panic-on-warnings enabled, this aborts the migration. Fix: add an else-if branch that detects binary storage types by MySQLType (binary, varbinary, *blob) and returns []byte, so the driver sends MYSQL_TYPE_BLOB (binary data) with no charset validation. MySQLType is used rather than Charset == "" alone because test Column objects built via NewColumnList leave MySQLType unset, which would have changed the return type for all no-charset columns in existing tests. In production, inspect.go always populates MySQLType from information_schema.data_type. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Address PR review feedback on varbinary Warning 1300 fix - Use MySQLType "varbinary(16)" in tests instead of bare "varbinary", matching the real value produced by information_schema COLUMN_TYPE (which includes length). Guards against future refactors that might switch from substring matching to exact matching. - Correct test comment: MySQLType is populated from COLUMN_TYPE, not data_type. - Broaden types.go comment to say "the connection's charset/collation metadata (often utf8mb4)" since gh-ost's connection charset is configurable via --charset. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * appease linter --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: meiji163 <meiji163@github.com>
…binlog sentinel (github#1637) * Prevent permanent worker deadlock when cutover times out waiting for binlog sentinel Buffer allEventsUpToLockProcessed to MaxRetries() so the applier's send always completes immediately even after waitForEventsUpToLock has timed out and exited. --------- Co-authored-by: meiji163 <meiji163@github.com>
…ithub#1666) PR github#1637 buffered allEventsUpToLockProcessed to MaxRetries() to prevent a goroutine deadlock when waitForEventsUpToLock times out during cutover. However, when --default-retries is set to a very large value (e.g. 9999999999999), Go tries to allocate a channel with trillions of buffer slots, causing an immediate OOM crash before the migration even starts. Replace the MaxRetries()-sized buffer with a buffer of 1 and overwrite-oldest (latest-wins) send semantics. When the buffer is full (receiver timed out on a previous attempt), the stale message is drained before sending the current sentinel. This guarantees: - The current sentinel is always delivered (no message loss) - The executeWriteFuncs worker is never blocked (no deadlock) - Memory usage is constant regardless of MaxRetries() (no OOM) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ithub#1671) * Prevent throttle() from blocking forever when context is cancelled When abort() is called (e.g. due to heartbeat failures after a MySQL failover), the context is cancelled and initiateThrottlerChecks exits via ctx.Done() without calling SetThrottled(false). The throttle() loop only checks IsThrottled(), so it would spin indefinitely, preventing the migration from ever returning. Replace time.Sleep with a select on ctx.Done() so throttle() unblocks immediately on context cancellation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Address copilot feedback --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…b#1675) * Add MigrationContext.Hooks for in-process hook implementations gh-ost's only hook extension point is on-disk scripts globbed from --hooks-path. Library callers that embed Migrator must either ship scripts and their dependencies alongside their binary or maintain a parallel Go layer that bridges script side effects back into the host application. Introduce a Hooks interface in go/base with one method per lifecycle event, and an optional MigrationContext.Hooks field. NewMigrator reads the field once at construction and falls back to the existing HooksExecutor when unset, so CLI behavior is unchanged. A CompositeHooks helper in go/logic lets callers run the on-disk script executor and their own Go implementation side-by-side. HooksExecutor's previously package-private method names are renamed (onStartup -> OnStartup, etc.) so external types can satisfy the interface. The struct and constructor were already exported but the methods weren't, so no usable external API is displaced. * Skip nil entries in CompositeHooks and self-contain doc example Address PR review feedback: - CompositeHooks.OnX methods skip nil members instead of panicking, allowing callers to conditionally append optional hooks. - doc/hooks.md embedded-usage snippet now defines ctx and version so it is self-contained.
…p across 1 directory (github#1606) * Bump golang.org/x/crypto in the go_modules group across 1 directory Bumps the go_modules group with 1 update in the / directory: [golang.org/x/crypto](https://github.com/golang/crypto). Updates `golang.org/x/crypto` from 0.37.0 to 0.45.0 - [Commits](golang/crypto@v0.37.0...v0.45.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.45.0 dependency-type: indirect dependency-group: go_modules ... Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: meiji163 <meiji163@github.com>
…locks (github#1677) Co-authored-by: meiji163 <meiji163@github.com>
…github#1684) * Fix resume data loss: route heartbeat coords through applyEventsQueue onChangelogHeartbeatEvent was mutating applier.CurrentCoordinates directly from the streamer goroutine, before any DML that preceded the heartbeat was applied to the ghost table. The checkpoint loop reads CurrentCoordinates as "applied through this GTID" and could persist a checkpoint whose LastTrxCoords was ahead of what was actually applied. If gh-ost crashed before applyEventsQueue drained, --resume read that checkpoint and called StartSyncGTID with the persisted set; MySQL treated the un-applied GTIDs as already-seen and never re-streamed them. The ghost table silently lost those DMLs and cut-over produced a stale table. Fix: enqueue a tableWriteFunc onto applyEventsQueue that performs the coords bump. The apply goroutine executes it in order, after the DMLs the streamer enqueued before the heartbeat, restoring the invariant. Adds TestMigratorHeartbeatDoesNotAdvancePastUnappliedDML, which fails at the previous HEAD and passes after the fix; also asserts queue ordering to guard against future changes that wrap the heartbeat enqueue in a goroutine. Co-authored-by: Bastian Bartmann <bastian.bartmann@shopify.com> * Replace direct channel write with SendWithContext Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Bastian Bartmann <bastian.bartmann@shopify.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Add Datadog/statsd with simple client emitting startup * Add go runtime metrics to statsd reporting --------- Co-authored-by: meiji163 <meiji163@github.com>
Refactor newTestMigrationContext to set ServeSocketFile via os.TempDir() and remove runtime.Caller-based path derivation.
* Disable CGO for release builds * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…uery latency, sleep (github#1701) * metrics: introduce unified Emitter interface and consolidate go_runtime helpers Introduce a single Emitter interface (Gauge, Count, Histogram) on metrics.Client so all metric helpers share one testable contract. The interface replaces the narrow MemStatsGaugeEmitter type used only by the Go runtime reporter. - Add Emitter interface and Histogram method on Client. - Move go_runtime.go and go_runtime_test.go into emit.go and emit_test.go so the metrics package has a single place for helper functions and tests. - Switch MigrationContext.Metrics from *metrics.Client to metrics.Emitter so consumers can be tested with spies. * Refactor status reporting around migrationProgressSnapshot. Sample row-copy, DML, backlog, and lag once per tick into a snapshot passed to printStatus, with reportStatus as the single entry point for status output. Co-authored-by: Cursor <cursoragent@cursor.com> * metrics: emit row-copy progress gauges Add gh_ost.row_copy.rows_copied, gh_ost.row_copy.rows_estimate, and gh_ost.dml.events_applied gauges on each reportStatus tick, sampled from migrationProgressSnapshot. * metrics: emit binlog backlog gauges Add gh_ost.binlog.backlog_size, gh_ost.binlog.backlog_capacity, and gh_ost.binlog.backlog_utilization gauges on each reportStatus tick from the applyEventsQueue depth captured in migrationProgressSnapshot. * metrics: emit replication and heartbeat lag gauges Add gh_ost.lag.replication_seconds and gh_ost.lag.heartbeat_seconds gauges on each reportStatus tick, tagged with throttled:true|false. These are point-in-time readings (not distributions), so gauges are used rather than histograms — DogStatsD histogram aggregation exposes count/max series that do not match the log line lag values in Prometheus/Grafana. * metrics: emit throttle active and throttled-interval metrics Record throttle active state at a debounced cadence (gh_ost.throttle.active) and emit duration plus event metrics when a throttled interval completes (gh_ost.throttle.duration_seconds histogram and gh_ost.throttle.events_total count), each tagged with the throttling reason. * metrics: emit cut-over phase, attempt, and total duration metrics Add cut-over metric helpers and instrument cut-over attempts, phase durations, and terminal duration. Metrics emitted: - gh_ost.cut_over.attempts_total tagged with outcome - gh_ost.cut_over.phase_duration_milliseconds tagged with phase and outcome - gh_ost.cut_over.total_duration_milliseconds tagged with outcome Phase coverage includes the magic lock, original table lock, magic rename, and unlock paths. Durations are reported in milliseconds to preserve sub-second granularity. The atomic rename phase duration is recorded after the rename completes so the histogram reflects the full operation. * metrics: emit query latency histogram Emit gh_ost.query.duration_milliseconds for representative source-side and target-side queries (row count and binlog apply), tagged with side, kind, and outcome (ok|error). Helper validates inputs and is nil-safe. * metrics: emit per-stage sleep histograms Add sleep metric helpers and instrument the main migration sleep/wait paths. Metrics emitted: - gh_ost.sleep.duration_milliseconds tagged with stage - gh_ost.sleep.total_milliseconds tagged with stage Stages covered: - cut_over_postpone - chunk_throttle - retry_backoff - replica_wait Use millisecond units so sub-second waits, such as replica polling and nice-ratio throttling, are not truncated to zero. Skip sub-millisecond chunk-throttle samples to avoid emitting zero-valued sleeps that would distort the histogram. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Jan Grodowski <jan.grodowski@shopify.com>
* Document missing command-line flags. Add sections for command-line flags that were registered in the CLI but missing from command-line-flags.md, including logging, throttling, hooks, StatsD, trigger, revert, and basic connection flags. Update the documented cut-over default from the stale "safe" value to the current "atomic" default.
Rework the replica-test harness to run the localtests suite against MySQL 5.7, 8.0, 8.4 and Percona Server 8.0 from one parameterized setup. - Move docker-compose into script/ and split per-version server config into docker/<flavor>-<major.minor>/ dirs (common.cnf, create_user.sql, start_replication.sql), selected from TEST_MYSQL_IMAGE. Optional flavor.env per dir for knobs (Percona reads /etc/my.cnf.d). - Configure replication from the harness once both servers are up rather than at container init, so that replica connects on the first try. - Keep mysql_native_password for 5.7/8.0/percona, caching_sha2_password for 8.4. - Speed-ups: native mysql client with docker-exec fallback, detect server version once, shorter replication polls, configurable copy-throttle window (THROTTLE_SECONDS, default 2s to still exercise binlog DML replay). - toxiproxy split into its own compose file.
* feat: support MariaDB 11.4 and 11.8 gh-ost mostly works against MariaDB unchanged; this makes the few MariaDB-specific spots correct and runs the replica test suite against MariaDB 11.4 and 11.8 (LTS) alongside the MySQL/Percona flavors. Core: - ReplicaTermFor keeps the legacy slave/master replication terminology on MariaDB. MariaDB reports versions >= 10 (greater than the 8.4 cutoff) but never adopted the new replica/source terms, and its SHOW REPLICA STATUS still emits legacy column names, so the MySQL 8.4 term map doesn't apply. - Accept BINLOG MONITOR as well as REPLICATION CLIENT in the grant check (MariaDB 10.5+ renamed the privilege). Test harness (localtests): - test.sh: pick slave/Master terminology for MariaDB, wait for replica catch-up via MASTER_GTID_WAIT/gtid_binlog_pos, and skip gtid_mode-gated tests where @@gtid_mode is unsupported instead of aborting the run. - Resolve the per-flavor config dir by specificity (docker/<flavor>-<major.minor>, then <flavor>-<major>, then bare <flavor>) so versions that share one config live in a single dir. All MariaDB versions use one docker/mariadb dir (log-bin + ROW + log-slave-updates, native-auth users, MariaDB-GTID replication, MYSQL_CLIENT=mariadb since 11.x dropped the mysql-named client); a version (or major line) needing different config overrides it by adding its own dir. The compose healthcheck honours MYSQL_CLIENT. - Make the zero-date failure tests deterministic across flavors with an explicit sql_mode (NO_ZERO_IN_DATE,NO_ZERO_DATE,...); they relied on MySQL's default sql_mode, which MariaDB doesn't share. - Drop the incidental NOT NULL from the generated-column tests (MariaDB forbids NOT NULL on generated columns) so they run on MariaDB; keep generated-columns-unique skipped (it puts a generated column in the PRIMARY KEY, which MariaDB forbids, Error 1903). - Add mariadb:11.4.12 and mariadb:11.8.8 to the CI matrix. * feat: support MariaDB 10.5, 10.6 and 10.11 Add the MariaDB 10.x LTS lines (10.5, 10.6, 10.11) to the replica test suite. MariaDB versions below 11.1 need one core fix: gh-ost passes transaction_isolation as a DSN param, which the driver applies as `SET transaction_isolation=<v>` on each connection. That system variable does not exist on MariaDB < 11.1 (which has only tx_isolation), so the connection fails with Error 1193 "Unknown system variable 'transaction_isolation'". tx_isolation was in turn removed in MySQL 8.0, so no single variable name is portable across the supported servers. - mysql.OpenDB strips the transaction_isolation param and instead runs the SQL-standard `SET SESSION TRANSACTION ISOLATION LEVEL <level>` on each new connection via a connector wrapper, accepted by every supported MySQL and MariaDB version. GetDBUri still emits the param unchanged, so DSN construction and its tests are unaffected. - Add mariadb:10.5.29, mariadb:10.6.27 and mariadb:10.11.18 to the CI matrix.
Upgrade go-mysql to v1.15.0
* build: standardize release asset naming (github#1728) Rework build.sh so released assets have stable, predictable names that are easy to automate against: - drop the generated timestamp from tarball names - drop the "-binary" infix from tarball names - name macOS tarballs with "darwin" (matches GOOS) instead of "osx" - drop the version from .rpm/.deb filenames; the release download URL already carries it. Package metadata still records version + epoch - build arm64 .rpm and .deb in addition to amd64, using each ecosystem's arch convention (x86_64/aarch64 for rpm, amd64/arm64 for deb) - stop publishing the bare "gh-ost" binary; it was just the last build (darwin/arm64) and was ambiguous. Tarballs and packages cover every OS/arch - emit a SHA256SUMS manifest so downloads can be verified with `sha256sum -c SHA256SUMS` Resulting assets: gh-ost-{linux,darwin}-{amd64,arm64}.tar.gz gh-ost.{x86_64,aarch64}.rpm gh-ost.{amd64,arm64}.deb SHA256SUMS Also refactor build() to take GOOS/GOARCH directly, share common fpm flags via an array, and update the README build.sh description. * build: simplify packaging Dockerfile and dock script (github#1728) Slim down the release packaging tooling: - Dockerfile.packaging: collapse the per-package RUN layers into one, drop packages already provided by the golang:bookworm base (git, tar, curl, gcc, g++, bash) and the unused rsync, and drop the legacy GOPATH src layout (the build is module-mode). Only ruby/ruby-dev/build-essential (for fpm) and rpm (for rpmbuild) are installed now. - Use a multi-stage build with a `scratch` artifacts stage so `docker build --output type=local` exports the assets straight to the host. This drops the intermediate container, bind mount, -it TTY and find|xargs cp dance from `script/dock pkg`. - build.sh: stage the fpm package tree in the system temp dir instead of under $buildpath, so it never lands among the exported release artifacts.
gh-ost hard-coded the go-mysql binlog syncer to MySQLFlavor and parsed every GTID set as MySQL, so --gtid could not be used against MariaDB. go-mysql already speaks the MariaDB GTID dialect; this wires it up. - Detect server flavor from the version string (IsMariaDB / FlavorFor) and set it on the BinlogSyncerConfig. - Make GTIDBinlogCoordinates hold the flavor-agnostic gomysql.GTIDSet interface; parse via ParseGTIDSet(flavor, ...). - Handle MariadbGTIDEvent alongside GTIDEvent in the streamer, and read the committed GTID set from XIDEvent.GSet without a MySQL-only cast. - Read MariaDB GTID positions from @@global.gtid_binlog_pos, Gtid_IO_Pos and @@global.gtid_slave_pos (MariaDB has no Executed_Gtid_Set column nor gtid_mode / enforce_gtid_consistency). - Skip the gtid_mode / enforce_gtid_consistency validation on MariaDB, where GTIDs are always recorded when binary logging is enabled. localtests: - Detect the server version once and reuse it; run gtid_mode=ON tests on MariaDB (normalize its current_gtid_mode to ON) and trim the GTID diagnostics that don't exist on MariaDB. - Enable gtid_strict_mode on the MariaDB test servers. Because --test-on-replica makes gh-ost write locally on the replica, give the replica its own GTID domain so its writes never collide with the primary's domain-0 stream under strict mode. - Add the gtid-resume case: interrupt a --gtid migration mid-copy via the interactive 'panic' command, then --resume, exercising the GTID checkpoint round-trip (WriteCheckpoint persists the GTID set, ReadLastCheckpoint parses it back via NewGTIDBinlogCoordinates(flavor)). Passes across the MySQL, MariaDB and Percona CI matrix. Verified: full localtests suite passes on MySQL 5.7/8.0/8.4 and on the MariaDB matrix (10.5, 10.6, 10.11, 11.4, 11.8) with gtid_strict_mode on.
…upstream-analyze-cutover Add opt-in --analyze-ghost-table-before-cutover
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 7. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v4...v7) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
…actions/upload-artifact-5 Bump actions/upload-artifact from 4 to 7
Signed-off-by: vladdoster <vladislav.doster@icloud.com>
docs: correct spelling
The binlog reader updates its current coordinates while streaming, while go-mysql's BinlogSyncer retains and mutates the GTID set passed to StartSyncGTID. Clone the caller-provided coordinates for the reader and clone the GTID set again for the syncer. This prevents both components from mutating the caller's reconnect coordinates and eliminates concurrent access to the same MysqlGTIDSet.
The retry-hook test redirected os.Stdout and os.Stderr to capture status and hook output. Migrate has background status and logging goroutines, so reassigning those process-global streams races with their output. Inject a status writer into Migrator, retaining stdout as the default. The test supplies thread-safe buffers for the outputs, preserving the separate stdout and stderr assertions without mutating global state.
Fix data races and enable race detector in CI
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Add release workflow
d-bytebase
added a commit
to bytebase/bytebase
that referenced
this pull request
Nov 2, 2026
The require line above is cosmetic on its own: the replace below shadows it, go.sum holds no upstream gh-ost entries, and that version is never compiled. What makes this bump real is repointing the replace. The fork was pinned at a commit based on upstream v1.1.8, so Bytebase shipped none of upstream's MariaDB support, the migration-state data-race fixes, or the binlog decode optimization that skips rows for unrelated tables. gh-ost dispatch is directive-based with no engine gate and SQL Review recommends gh-ost for MariaDB, so MariaDB and OceanBase gh-ost runs were reachable on that build. Repoints at bytebase/gh-ost2#10, which rebases the nine Bytebase patches onto upstream v1.1.11 and fixes a deadlock that the rebase exposed: with --attempt-instant-ddl, which Bytebase enables for every migration, Migrate() never returned, because the changelog listener parked on an unbuffered send while holding the mutex the binlog parser needs and finalCleanup waited on the parser through BinlogSyncer.Close. TestGhostSchemaUpdate reproduces it. No source change is needed here: the API surface Bytebase uses is unchanged. go mod tidy raises go-mysql to v1.15.0 and pulls in DataDog/datadog-go, which gh-ost's new statsd code needs but never constructs here. The require line for pingcap/tidb/pkg/parser moves too, but a replace pins that module to the Bytebase fork, so the compiled parser is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GhostDatabaseName puts the ghost, old and changelog tables in a schema of their own instead of beside the original table, and the temporary-table prefix becomes `~` rather than `_`. Only the checkpoint table stays with the original table, since it records that table's progress. Grant validation now requires the privileges on every database the migration writes to and names the ones that are missing, and --include-triggers bails out when the ghost database differs, because a trigger cannot follow its table into another schema.
go-sql-driver registers TLS configs in a process-global map keyed by name. Keying on the server name alone makes concurrent migrations against the same host overwrite each other's config, so the key now carries gh-ost's own uuid. An empty key keeps the previous "ghost-<host>" form.
ServeSocketFile is empty when gh-ost runs in-process rather than as a CLI, so BindUnixSocket is never called and unixListener stays nil. The accept loop panicked on it; the TCP loop already had this guard.
Teardown closed only the database handle, leaking the binlog reader's connection for the lifetime of the process that embeds gh-ost. It now closes the reader too, and both it and Close tolerate the nil reader and nil handle left behind when InitDBConnections fails early.
The goroutine logged "Done streaming" after handing the error to panicAbort, which reads as a clean finish in the log of a migration that is in fact aborting.
Every step after LockOriginalTable runs with a write lock held on the original table. A failure in any of them returned without unlocking, so the table stayed unwritable for the life of the process that embeds gh-ost.
The streamer reconnects the binlog reader on every read failure, so bad credentials or a host the server refuses produced an unbounded reconnect loop that reads as a brute-force attempt and gets the host firewalled. Consecutive authentication failures are now counted on the migration context, which outlives the reader the streamer recreates, and the failure that reaches MaxAuthFailures is wrapped in ErrMaxAuthFailures so the streamer stops instead of reconnecting. MaxAuthFailures of zero keeps the old unbounded behavior.
A caller that reaches the server through a tunnel can now register a dial context with go-sql-driver and name it in ConnectionConfig.Network, which replaces tcp in the DSN. ConnectionConfig.Dialer carries the same tunnel to go-mysql's binlog connections, which do not go through go-sql-driver. Both are empty or nil by default, which keeps a plain tcp dial.
d-bytebase
force-pushed
the
bytebase/rebase-v1.1.11
branch
from
November 2, 2026 12:32
b5db74f to
8a2e2d7
Compare
d-bytebase
added a commit
to bytebase/bytebase
that referenced
this pull request
Nov 2, 2026
The require line above is cosmetic on its own: the replace below shadows it, go.sum holds no upstream gh-ost entries, and that version is never compiled. What makes this bump real is repointing the replace. The fork was pinned at a commit based on upstream v1.1.8, so Bytebase shipped none of upstream's MariaDB support, the migration-state data-race fixes, or the binlog decode optimization that skips rows for unrelated tables. gh-ost dispatch is directive-based with no engine gate and SQL Review recommends gh-ost for MariaDB, so MariaDB and OceanBase gh-ost runs were reachable on that build. Repoints at bytebase/gh-ost2#10, which rebases the nine Bytebase patches onto upstream v1.1.11 and fixes a deadlock that the rebase exposed: with --attempt-instant-ddl, which Bytebase enables for every migration, Migrate() never returned, because the changelog listener parked on an unbuffered send while holding the mutex the binlog parser needs and finalCleanup waited on the parser through BinlogSyncer.Close. TestGhostSchemaUpdate reproduces it. No source change is needed here: the API surface Bytebase uses is unchanged. go mod tidy raises go-mysql to v1.15.0 and pulls in DataDog/datadog-go, which gh-ost's new statsd code needs but never constructs here. The require line for pingcap/tidb/pkg/parser moves too, but a replace pins that module to the Bytebase fork, so the compiled parser is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A migration that --attempt-instant-ddl completes never returned. Three
goroutines formed a cycle:
- the changelog listener parks on the unbuffered GhostTableMigrated
send, which the instant-DDL path returns without ever reading;
- notifyListeners held listenersMutex across that synchronous call, so
shouldDecodeRowsEvent — called on the binlog parser's goroutine since
upstream github#1699 — stalled on the same mutex;
- BinlogSyncer.Close, which finalCleanup calls, waits for that parser
goroutine.
Migrate() therefore blocked until the migration context was cancelled.
notifyListeners now collects the matching listeners under the mutex and
invokes them after releasing it, and ghostTableMigrated gets a one-slot
buffer, since the changelog carries exactly one such state per migration.
This is an upstream defect, still present on gh-ost master. It surfaces
here because Bytebase enables --attempt-instant-ddl for every migration,
so every ALTER that MySQL can apply instantly hung.
d-bytebase
force-pushed
the
bytebase/rebase-v1.1.11
branch
from
November 2, 2026 12:57
8a2e2d7 to
6ff100b
Compare
d-bytebase
added a commit
to bytebase/bytebase
that referenced
this pull request
Nov 2, 2026
The require line above is cosmetic on its own: the replace below shadows it, go.sum holds no upstream gh-ost entries, and that version is never compiled. What makes this bump real is repointing the replace. The fork was pinned at a commit based on upstream v1.1.8, so Bytebase shipped none of upstream's MariaDB support, the migration-state data-race fixes, or the binlog decode optimization that skips rows for unrelated tables. gh-ost dispatch is directive-based with no engine gate and SQL Review recommends gh-ost for MariaDB, so MariaDB and OceanBase gh-ost runs were reachable on that build. Repoints at bytebase/gh-ost2#10, which rebases the nine Bytebase patches onto upstream v1.1.11 and fixes a deadlock that the rebase exposed: with --attempt-instant-ddl, which Bytebase enables for every migration, Migrate() never returned, because the changelog listener parked on an unbuffered send while holding the mutex the binlog parser needs and finalCleanup waited on the parser through BinlogSyncer.Close. TestGhostSchemaUpdate reproduces it. No source change is needed here: the API surface Bytebase uses is unchanged. go mod tidy raises go-mysql to v1.15.0 and pulls in DataDog/datadog-go, which gh-ost's new statsd code needs but never constructs here. The require line for pingcap/tidb/pkg/parser moves too, but a replace pins that module to the Bytebase fork, so the compiled parser is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
d-bytebase
added a commit
to bytebase/bytebase
that referenced
this pull request
Nov 2, 2026
* fix(deps): update module github.com/github/gh-ost to v1.1.11 * chore(deps): point the gh-ost fork at upstream v1.1.11 The require line above is cosmetic on its own: the replace below shadows it, go.sum holds no upstream gh-ost entries, and that version is never compiled. What makes this bump real is repointing the replace. The fork was pinned at a commit based on upstream v1.1.8, so Bytebase shipped none of upstream's MariaDB support, the migration-state data-race fixes, or the binlog decode optimization that skips rows for unrelated tables. gh-ost dispatch is directive-based with no engine gate and SQL Review recommends gh-ost for MariaDB, so MariaDB and OceanBase gh-ost runs were reachable on that build. Repoints at bytebase/gh-ost2#10, which rebases the nine Bytebase patches onto upstream v1.1.11 and fixes a deadlock that the rebase exposed: with --attempt-instant-ddl, which Bytebase enables for every migration, Migrate() never returned, because the changelog listener parked on an unbuffered send while holding the mutex the binlog parser needs and finalCleanup waited on the parser through BinlogSyncer.Close. TestGhostSchemaUpdate reproduces it. No source change is needed here: the API surface Bytebase uses is unchanged. go mod tidy raises go-mysql to v1.15.0 and pulls in DataDog/datadog-go, which gh-ost's new statsd code needs but never constructs here. The require line for pingcap/tidb/pkg/parser moves too, but a replace pins that module to the Bytebase fork, so the compiled parser is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Danny Xu <d@bytebase.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebases the Bytebase-only patches onto upstream tag
v1.1.11. The fork'smainsat on
v1.1.8(2026-05-12) and so shipped none of upstream's MariaDB support,the migration-state data-race fixes, or the binlog decode optimisation that skips
rows for unrelated tables.
Base was
v1.1.8+ 27 commits. 7 of those were upstream-authored and are alreadyin
v1.1.11, so they drop out; the branch isv1.1.11+ 9 commits, one perlogical patch, plus one fix for an upstream deadlock the rebase exposed (see
below — read that section first). A ninth, the transaction-isolation patch, is
dropped because upstream now handles that case itself; see Dropped patches.
Preserved patches
Each is verified present in the rebased tree, and the per-patch notes say where
upstream's new structure changed how it is applied.
~instead of_as the temp-table prefixfeat: ghost tables in a separate database, with a ~ name prefixAnalyzeGhostTableand tobuildRangeInsertQueryTwoColumn(the composite-PK path from github#1686), both of which needed the original/ghost database split. Upstream's MariaDBBINLOG MONITORgrant alias is kept whilevalidateGrantsmoves to a per-database check.chore: use the migration uuid as the TLS config keyunixListenerfixfix: do not accept on a nil unixListenerfix: close the binlog reader when tearing down the streamerpanicAbortchore: return after reporting a streaming failure to panicAbortfix: release the original table lock when two-step cut-over failsmetrics.RecordCutOverPhasecalls.chore: circuit breaker for repeated authentication failuresresetAuthFailureCountrather than ahandleAuthError(nil, …)call on the hot event path.isAuthenticationError's error-code constants are renamed to Go style (errAccessDenied, …).feat: support a custom MySQL dial networkEvery receiver was renamed upstream (
this→mctx/apl/isp/mgtr/es/thlr/srv/gmr/con), so all eight patches were reapplied by hand against upstream's current code rather than replayed textually. Error strings now follow upstream's lowercase convention.One fix beyond the rebase
The rebase itself is nine commits. A tenth,
fix: do not deadlock when instant DDL completes the migration, is not a Bytebase patch being carried forward — itfixes an upstream defect that the rebase exposed, and without it this branch is
unusable from Bytebase.
Before the fix,
TestGhostSchemaUpdatein bytebase/bytebase failed on thisbranch 3 times out of 3 (~140s each, the rollout wait timing out) while
origin/mainpassed it 3 out of 3 (~23s each). A focused reproduction, nowcommitted as
TestMigrator/TestMigrateInstantDDLReturns, pins it down to a three-goroutinecycle after
--attempt-instant-ddlcompletes a migration:GhostTableMigratedsend, whichthe instant-DDL path in
Migratereturns without ever reading;notifyListenersheldlistenersMutexacross that synchronous call, soshouldDecodeRowsEvent— called on the binlog parser's goroutine sinceupstream Don’t parse binlog events of unrelated tables github/gh-ost#1699, first released in v1.1.11 — stalled on the same mutex;
BinlogSyncer.Close, whichfinalCleanupcalls, waits for that parsergoroutine (
b.wg.Wait()).Migrate()then blocks until the migration context is cancelled. The fixcollects the matching listeners under the mutex and invokes them after releasing
it, and gives
ghostTableMigrateda one-slot buffer.It also sets
CutOverCompleteFlagonce the instant DDL succeeds, beforefinalCleanup, as the normal cut-over and revert paths already do. Without itcanStopStreamingstays false, soStreamEventstreats the close thatfinalCleanupperforms as an unexpected error and reconnects after its sleep —leaking a replication connection and a goroutine past the point where
Migrate()has already returned nil. That one is visible in a Bytebase task-run log as
StreamEvents encountered unexpected error: Sync was closedafterDone migrating, once per instant-DDL migration.The same commit guards
dmlEventsinTestStreamEventsandTestStreamEventsAutomaticallyReconnectswith a mutex. The goroutineEventsStreamer.StreamEventsspawns to notify listeners outlivesStreamEvents,so
group.Wait()never ordered the listener's appends against theLenassertionthat follows — a latent race in upstream's tests, independent of anything here,
which
-racecaught once during a full-suite run on this branch. Running thattest alone on plain
v1.1.11three times did not reproduce it, so I cannot saywhether the production change makes it more likely; the guard is correct either
way.
This is still present on gh-ost
masterand is worth sending upstream. It stayedunnoticed there because
--attempt-instant-ddldefaults to off upstream, whileBytebase sets it for every migration — so on this branch every ALTER that MySQL
can apply instantly hung.
Dropped patches
chore: rename test_utils.go to test_utils_test.go014cdc80 refactor(logic): scope testcontainers helpers to the test build.chore: drop deprecated linters from golangci config, plus thechore: update/Revert "chore: update"pair that touched.golangci.yml4d37b8a9 Update golangci-lint to v2.11, which already movesgofmttoformattersand disablesnoctx/prealloc. The three fork commits also cancelled out:git diff 0270a289..main -- .golangci.ymlwas already empty.StreamEventsloop rewrite tofor !canStopStreaming(), thePos:literal reformat, and lowercasingunknown DML type— all carried inside the circuit-breaker commitsTestAuthFailureCounterResetandTestAuthFailureRecoveryScenarioAuthFailureCountthemselves and asserted their own arithmetic, so they exercised no production code.TestHandleAuthErrorTripsAtMaxAuthFailuresandTestHandleAuthErrorCountsOnlyAuthFailurescover the same ground throughhandleAuthError.chore: no need to set tx isolation level)mysql.OpenDBstrips thetransaction_isolationDSN parameter and applies the portableSET SESSION TRANSACTION ISOLATION LEVELstatement on each new connection instead, precisely because the variable does not exist on MariaDB < 11.1 andtx_isolationis gone from MySQL 8.0+ — the incompatibility this patch was working around. Every connection in gh-ost goes throughOpenDBorGetDB, so the parameter never reaches a server. Proven on live servers: with the raw DSN parameter, MariaDB 10.11 answersError 1193: Unknown system variable 'transaction_isolation', while throughOpenDBthe level is applied correctly on both 10.11 and 11.4. This is a behavior change for Bytebase: the patch had been disabling the isolation level outright, so gh-ost ran at the server default; it now gets the REPEATABLE-READ (or READ-COMMITTED for RocksDB) that upstream intends.git cherry -v v1.1.11 mainlists none of them as absent, sov1.1.11already contains all 7 by patch-id.There is no "testcontainers port API fix" to carry. Fork PR #8's branch was named
fix-testcontainers-port-api, but its two commits are thetest_utilsrename andthe golangci trim above — the fork contains no port-API source change, and its
testcontainers pin (
v0.37.0) was never bumped.v1.1.11is onv0.43.0.Also fixed, because the
~prefix reaches themlocaltests/gtid-resume/test.shis new upstream and hard-codes_gh_ost_test_gho.TestRevert'sCHECKSUM TABLE %s, %spassed unquoted names, which is a syntaxerror once the reverted table is
~testing_rev_del. The old fork had this fix;it is reapplied at upstream's new call site.
getTestRevertedTableNamestill returned a_-prefixed name.What was run
go build ./...,gofmt -s -l go/clean.golangci-lint run --config=.golangci.ymlat v2.11.4, the version.github/workflows/golangci-lint.ymlpins: 0 issues.go test -count=1 -race -covermode=atomic ./go/...— all 6 packages pass, nodata races.
go/logicholds the testcontainers suites (TestApplier,TestMigrator,TestEventsStreamer) at ~64% coverage.TestEventsStreameralone,-race, 3 runs on this branch and 3 on plainv1.1.11: clean on both.TestMigrator/TestMigrateInstantDDLReturns, the new regression lock: hung past60s and failed before the deadlock fix, passes in 0.26s after it.
localtestson MariaDB 11.4.13 and MariaDB 10.11.19 primary/replicapairs with MariaDB GTID, via the flavor harness upstream added in Multi-flavor docker replica-test harness github/gh-ost#1722: all
90 pass on each. 10.11 is below 11.1, so it is the version that proves the
transaction_isolationtranslation above.localtestson a MySQL 8.0.46 primary/replica pair with GTID: all 90 pass,including the two the fork touches (
panic-on-warnings-batch-middleandpanic-on-warnings-update-pk-with-duplicate-on-new-unique-index). The wholesuite was run rather than just those two, because the
~prefix reaches theshared harness (
localtests/test.sh) and so every test in it.What to review
GitHub shows 77 commits and 1505 files, because the merge base is the old fork's
base and so the diff includes all of upstream v1.1.9–v1.1.11 plus its
vendor/.The Bytebase part is the last ten commits:
That is 23 files and 864 added lines — the whole of what this branch adds
on top of the upstream tag.
How to land this
GitHub reports the PR as conflicting, which is inherent to a fork rebase: this
branch is not a descendant of
mainand rewrites the same files, so there is nomerge commit to make. The branch is meant to replace
main's history — land itby fast-forwarding
maintobytebase/rebase-v1.1.11(or by force-updatingmain), not by merging.This repository's
pull_requestworkflows have not run since July 2026 (only thescheduled CodeQL job does), so expect no CI on this PR. The run below is local.
MariaDB
Upstream's MariaDB work (github#1724, github#1731) arrives with this rebase, and it does
work: the whole
localtestssuite passes against MariaDB 11.4 and 10.11 withMariaDB GTID, and a Bytebase rollout completes a gh-ost ALTER on MariaDB 11.4
end to end (dry run with a full row copy, then the real migration).
Two gh-ost flags are needed on MariaDB, neither caused by this branch:
--skip-metadata-lock-check. MariaDB's performance_schema has nowait/lock/metadata/sql/mdlinstrument, soStateMetadataLockInstrumentcannever be satisfied there.
--assume-master-host. Without it,initiateInspectorwalks the topologyvia
getMasterConnectionConfig, which runsSHOW SLAVE STATUS. MariaDB 10.5split the old
REPLICATION CLIENTintoBINLOG MONITORandSLAVE MONITOR,and
GRANT REPLICATION CLIENTnow yields only the former, so the query returnsError 1227 … you need … the SLAVE MONITOR privilege(s). Verified: with--assume-master-hostset and noSLAVE MONITORgranted, the migrationcompletes.
Two things here are arguably upstream bugs worth fixing separately:
validateGrantsacceptsBINLOG MONITORfor its REPLICATION CLIENTrequirement and logs that privileges are sufficient, then a later
SHOW SLAVE STATUSfails on exactly the privilege it did not check. On MariaDBit should also require
SLAVE MONITOR/REPLICA MONITOR, or stop claiming theprivileges are sufficient.
restartReplicationdiscards the error fromGetMasterKeyFromSlaveStatus(
masterKey, _ :=), so the same 1227 is logged at ERROR twice per migration ona server where it is expected and harmless. It reads like a failure in the log
of a migration that succeeds.
🤖 Generated with Claude Code