Skip to content

chore: rebase the Bytebase patches onto upstream v1.1.11, and fix an instant-DDL deadlock - #10

Open
d-bytebase wants to merge 76 commits into
mainfrom
bytebase/rebase-v1.1.11
Open

d-bytebase wants to merge 76 commits into
mainfrom
bytebase/rebase-v1.1.11

Conversation

@d-bytebase

@d-bytebase d-bytebase commented Nov 2, 2026 •

Copy link
Copy Markdown

Rebases the Bytebase-only patches onto upstream tag v1.1.11. The fork's main
sat on v1.1.8 (2026-05-12) and so shipped none of upstream's MariaDB support,
the migration-state data-race fixes, or the binlog decode optimisation that skips
rows for unrelated tables.

Base was v1.1.8 + 27 commits. 7 of those were upstream-authored and are already
in v1.1.11, so they drop out; the branch is v1.1.11 + 9 commits, one per
logical patch, plus one fix for an upstream deadlock the rebase exposed (see
below — read that section first). A ninth, the transaction-isolation patch, is
dropped because upstream now handles that case itself; see Dropped patches.

Preserved patches

Each is verified present in the rebased tree, and the per-patch notes say where
upstream's new structure changed how it is applied.

patch commit note
ghost database name, and ~ instead of _ as the temp-table prefix feat: ghost tables in a separate database, with a ~ name prefix Extended to upstream's new AnalyzeGhostTable and to buildRangeInsertQueryTwoColumn (the composite-PK path from github#1686), both of which needed the original/ghost database split. Upstream's MariaDB BINLOG MONITOR grant alias is kept while validateGrants moves to a per-database check.
uuid as the TLS config key (fork #4) chore: use the migration uuid as the TLS config key
nil unixListener fix fix: do not accept on a nil unixListener
close the binlog reader on teardown fix: close the binlog reader when tearing down the streamer
teardown-and-return on panicAbort chore: return after reporting a streaming failure to panicAbort
unlock the original table when two-step cut-over fails fix: release the original table lock when two-step cut-over fails Applied around upstream's new metrics.RecordCutOverPhase calls.
authentication-failure circuit breaker chore: circuit breaker for repeated authentication failures The success-path reset is now its own resetAuthFailureCount rather than a handleAuthError(nil, …) call on the hot event path. isAuthenticationError's error-code constants are renamed to Go style (errAccessDenied, …).
custom MySQL dial network (fork #9) feat: support a custom MySQL dial network This is what routes gh-ost through the SSH tunnel for bytebase/bytebase#20304.

Every receiver was renamed upstream (this → mctx/apl/isp/mgtr/es/thlr/srv/gmr/con), so all eight patches were reapplied by hand against upstream's current code rather than replayed textually. Error strings now follow upstream's lowercase convention.

One fix beyond the rebase

The rebase itself is nine commits. A tenth, fix: do not deadlock when instant DDL completes the migration, is not a Bytebase patch being carried forward — it
fixes an upstream defect that the rebase exposed, and without it this branch is
unusable from Bytebase.

Before the fix, TestGhostSchemaUpdate in bytebase/bytebase failed on this
branch 3 times out of 3 (~140s each, the rollout wait timing out) while
origin/main passed it 3 out of 3 (~23s each). A focused reproduction, now
committed as
TestMigrator/TestMigrateInstantDDLReturns, pins it down to a three-goroutine
cycle after --attempt-instant-ddl completes a migration:

  • the changelog listener parks on the unbuffered GhostTableMigrated send, which
    the instant-DDL path in Migrate returns without ever reading;
  • notifyListeners held listenersMutex across that synchronous call, so
    shouldDecodeRowsEvent — called on the binlog parser's goroutine since
    upstream Don’t parse binlog events of unrelated tables github/gh-ost#1699, first released in v1.1.11 — stalled on the same mutex;
  • BinlogSyncer.Close, which finalCleanup calls, waits for that parser
    goroutine (b.wg.Wait()).

Migrate() then blocks until the migration context is cancelled. The fix
collects the matching listeners under the mutex and invokes them after releasing
it, and gives ghostTableMigrated a one-slot buffer.

It also sets CutOverCompleteFlag once the instant DDL succeeds, before
finalCleanup, as the normal cut-over and revert paths already do. Without it
canStopStreaming stays false, so StreamEvents treats the close that
finalCleanup performs as an unexpected error and reconnects after its sleep —
leaking a replication connection and a goroutine past the point where Migrate()
has already returned nil. That one is visible in a Bytebase task-run log as
StreamEvents encountered unexpected error: Sync was closed after
Done migrating, once per instant-DDL migration.

The same commit guards dmlEvents in TestStreamEvents and
TestStreamEventsAutomaticallyReconnects with a mutex. The goroutine
EventsStreamer.StreamEvents spawns to notify listeners outlives StreamEvents,
so group.Wait() never ordered the listener's appends against the Len assertion
that follows — a latent race in upstream's tests, independent of anything here,
which -race caught once during a full-suite run on this branch. Running that
test alone on plain v1.1.11 three times did not reproduce it, so I cannot say
whether the production change makes it more likely; the guard is correct either
way.

This is still present on gh-ost master and is worth sending upstream. It stayed
unnoticed there because --attempt-instant-ddl defaults to off upstream, while
Bytebase sets it for every migration — so on this branch every ALTER that MySQL
can apply instantly hung.

Dropped patches

dropped reason
chore: rename test_utils.go to test_utils_test.go Upstream does it in 014cdc80 refactor(logic): scope testcontainers helpers to the test build.
chore: drop deprecated linters from golangci config, plus the chore: update / Revert "chore: update" pair that touched .golangci.yml Upstream migrated to the golangci-lint v2 schema in 4d37b8a9 Update golangci-lint to v2.11, which already moves gofmt to formatters and disables noctx/prealloc. The three fork commits also cancelled out: git diff 0270a289..main -- .golangci.yml was already empty.
the StreamEvents loop rewrite to for !canStopStreaming(), the Pos: literal reformat, and lowercasing unknown DML type — all carried inside the circuit-breaker commits Upstream now has all three.
TestAuthFailureCounterReset and TestAuthFailureRecoveryScenario Both incremented AuthFailureCount themselves and asserted their own arithmetic, so they exercised no production code. TestHandleAuthErrorTripsAtMaxAuthFailures and TestHandleAuthErrorCountsOnlyAuthFailures cover the same ground through handleAuthError.
the transaction-isolation patch (chore: no need to set tx isolation level) Upstream's new mysql.OpenDB strips the transaction_isolation DSN parameter and applies the portable SET SESSION TRANSACTION ISOLATION LEVEL statement on each new connection instead, precisely because the variable does not exist on MariaDB < 11.1 and tx_isolation is gone from MySQL 8.0+ — the incompatibility this patch was working around. Every connection in gh-ost goes through OpenDB or GetDB, so the parameter never reaches a server. Proven on live servers: with the raw DSN parameter, MariaDB 10.11 answers Error 1193: Unknown system variable 'transaction_isolation', while through OpenDB the level is applied correctly on both 10.11 and 11.4. This is a behavior change for Bytebase: the patch had been disabling the isolation level outright, so gh-ost ran at the server default; it now gets the REPEATABLE-READ (or READ-COMMITTED for RocksDB) that upstream intends.
the 7 upstream-authored commits in the old range (by Gabriel Gilder, yosefbs, meiji163, Jan Grodowski) Verified rather than assumed: git cherry -v v1.1.11 main lists none of them as absent, so v1.1.11 already contains all 7 by patch-id.

There is no "testcontainers port API fix" to carry. Fork PR #8's branch was named
fix-testcontainers-port-api, but its two commits are the test_utils rename and
the golangci trim above — the fork contains no port-API source change, and its
testcontainers pin (v0.37.0) was never bumped. v1.1.11 is on v0.43.0.

Also fixed, because the ~ prefix reaches them

  • localtests/gtid-resume/test.sh is new upstream and hard-codes _gh_ost_test_gho.
  • TestRevert's CHECKSUM TABLE %s, %s passed unquoted names, which is a syntax
    error once the reverted table is ~testing_rev_del. The old fork had this fix;
    it is reapplied at upstream's new call site.
  • getTestRevertedTableName still returned a _-prefixed name.

What was run

  • go build ./..., gofmt -s -l go/ clean.
  • golangci-lint run --config=.golangci.yml at v2.11.4, the version
    .github/workflows/golangci-lint.yml pins: 0 issues.
  • go test -count=1 -race -covermode=atomic ./go/... — all 6 packages pass, no
    data races. go/logic holds the testcontainers suites (TestApplier,
    TestMigrator, TestEventsStreamer) at ~64% coverage.
  • TestEventsStreamer alone, -race, 3 runs on this branch and 3 on plain
    v1.1.11: clean on both.
  • TestMigrator/TestMigrateInstantDDLReturns, the new regression lock: hung past
    60s and failed before the deadlock fix, passes in 0.26s after it.
  • localtests on MariaDB 11.4.13 and MariaDB 10.11.19 primary/replica
    pairs with MariaDB GTID, via the flavor harness upstream added in Multi-flavor docker replica-test harness github/gh-ost#1722: all
    90 pass on each
    . 10.11 is below 11.1, so it is the version that proves the
    transaction_isolation translation above.
  • localtests on a MySQL 8.0.46 primary/replica pair with GTID: all 90 pass,
    including the two the fork touches (panic-on-warnings-batch-middle and
    panic-on-warnings-update-pk-with-duplicate-on-new-unique-index). The whole
    suite was run rather than just those two, because the ~ prefix reaches the
    shared harness (localtests/test.sh) and so every test in it.

What to review

GitHub shows 77 commits and 1505 files, because the merge base is the old fork's
base and so the diff includes all of upstream v1.1.9–v1.1.11 plus its vendor/.
The Bytebase part is the last ten commits:

git fetch origin bytebase/rebase-v1.1.11 && git diff v1.1.11...origin/bytebase/rebase-v1.1.11

That is 23 files and 864 added lines — the whole of what this branch adds
on top of the upstream tag.

How to land this

GitHub reports the PR as conflicting, which is inherent to a fork rebase: this
branch is not a descendant of main and rewrites the same files, so there is no
merge commit to make. The branch is meant to replace main's history — land it
by fast-forwarding main to bytebase/rebase-v1.1.11 (or by force-updating
main), not by merging.

This repository's pull_request workflows have not run since July 2026 (only the
scheduled CodeQL job does), so expect no CI on this PR. The run below is local.

MariaDB

Upstream's MariaDB work (github#1724, github#1731) arrives with this rebase, and it does
work: the whole localtests suite passes against MariaDB 11.4 and 10.11 with
MariaDB GTID, and a Bytebase rollout completes a gh-ost ALTER on MariaDB 11.4
end to end (dry run with a full row copy, then the real migration).

Two gh-ost flags are needed on MariaDB, neither caused by this branch:

  • --skip-metadata-lock-check. MariaDB's performance_schema has no
    wait/lock/metadata/sql/mdl instrument, so StateMetadataLockInstrument can
    never be satisfied there.
  • --assume-master-host. Without it, initiateInspector walks the topology
    via getMasterConnectionConfig, which runs SHOW SLAVE STATUS. MariaDB 10.5
    split the old REPLICATION CLIENT into BINLOG MONITOR and SLAVE MONITOR,
    and GRANT REPLICATION CLIENT now yields only the former, so the query returns
    Error 1227 … you need … the SLAVE MONITOR privilege(s). Verified: with
    --assume-master-host set and no SLAVE MONITOR granted, the migration
    completes.

Two things here are arguably upstream bugs worth fixing separately:

  • validateGrants accepts BINLOG MONITOR for its REPLICATION CLIENT
    requirement and logs that privileges are sufficient, then a later
    SHOW SLAVE STATUS fails on exactly the privilege it did not check. On MariaDB
    it should also require SLAVE MONITOR/REPLICA MONITOR, or stop claiming the
    privileges are sufficient.
  • restartReplication discards the error from GetMasterKeyFromSlaveStatus
    (masterKey, _ :=), so the same 1227 is logged at ERROR twice per migration on
    a server where it is expected and harmless. It reads like a failure in the log
    of a migration that succeeds.

🤖 Generated with Claude Code

ggilder and others added 30 commits April 22, 2026 16:52
…ithub#1661)

* Fix Warning 1300 for varbinary columns with bytes invalid as utf8mb4

When gh-ost replays a binlog DML event, the go-mysql library returns
varbinary column values as a Go `string` (not `[]byte`). In convertArg,
the existing code only converted string → []byte when the column had a
non-empty Charset (e.g. utf8mb4 for varchar). varbinary columns have no
character set, so Charset is always "", and the string fell through
unconverted.

The Go MySQL driver sends `string` args as MYSQL_TYPE_VAR_STRING with
the connection's utf8mb4 charset metadata attached, causing MySQL to
validate the bytes. If a varbinary value (e.g. a binary UUID) contains
byte sequences that are invalid utf8mb4, MySQL emits Warning 1300. With
gh-ost's panic-on-warnings enabled, this aborts the migration.

Fix: add an else-if branch that detects binary storage types by
MySQLType (binary, varbinary, *blob) and returns []byte, so the driver
sends MYSQL_TYPE_BLOB (binary data) with no charset validation.

MySQLType is used rather than Charset == "" alone because test Column
objects built via NewColumnList leave MySQLType unset, which would have
changed the return type for all no-charset columns in existing tests.
In production, inspect.go always populates MySQLType from
information_schema.data_type.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Address PR review feedback on varbinary Warning 1300 fix

- Use MySQLType "varbinary(16)" in tests instead of bare "varbinary",
  matching the real value produced by information_schema COLUMN_TYPE
  (which includes length). Guards against future refactors that might
  switch from substring matching to exact matching.
- Correct test comment: MySQLType is populated from COLUMN_TYPE, not
  data_type.
- Broaden types.go comment to say "the connection's charset/collation
  metadata (often utf8mb4)" since gh-ost's connection charset is
  configurable via --charset.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* appease linter

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: meiji163 <meiji163@github.com>
…binlog sentinel (github#1637)

* Prevent permanent worker deadlock when cutover times out waiting for binlog sentinel

Buffer allEventsUpToLockProcessed to MaxRetries() so the applier's send always
completes immediately even after waitForEventsUpToLock has timed out and exited.

---------

Co-authored-by: meiji163 <meiji163@github.com>
…ithub#1666)

PR github#1637 buffered allEventsUpToLockProcessed to MaxRetries() to prevent
a goroutine deadlock when waitForEventsUpToLock times out during cutover.
However, when --default-retries is set to a very large value (e.g.
9999999999999), Go tries to allocate a channel with trillions of buffer
slots, causing an immediate OOM crash before the migration even starts.

Replace the MaxRetries()-sized buffer with a buffer of 1 and
overwrite-oldest (latest-wins) send semantics. When the buffer is full
(receiver timed out on a previous attempt), the stale message is drained
before sending the current sentinel. This guarantees:

- The current sentinel is always delivered (no message loss)
- The executeWriteFuncs worker is never blocked (no deadlock)
- Memory usage is constant regardless of MaxRetries() (no OOM)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ithub#1671)

* Prevent throttle() from blocking forever when context is cancelled

When abort() is called (e.g. due to heartbeat failures after a MySQL
failover), the context is cancelled and initiateThrottlerChecks exits
via ctx.Done() without calling SetThrottled(false). The throttle() loop
only checks IsThrottled(), so it would spin indefinitely, preventing the
migration from ever returning.

Replace time.Sleep with a select on ctx.Done() so throttle() unblocks
immediately on context cancellation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Address copilot feedback

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…b#1675)

* Add MigrationContext.Hooks for in-process hook implementations

gh-ost's only hook extension point is on-disk scripts globbed from
--hooks-path. Library callers that embed Migrator must either ship
scripts and their dependencies alongside their binary or maintain a
parallel Go layer that bridges script side effects back into the host
application.

Introduce a Hooks interface in go/base with one method per lifecycle
event, and an optional MigrationContext.Hooks field. NewMigrator reads
the field once at construction and falls back to the existing
HooksExecutor when unset, so CLI behavior is unchanged. A CompositeHooks
helper in go/logic lets callers run the on-disk script executor and
their own Go implementation side-by-side.

HooksExecutor's previously package-private method names are renamed
(onStartup -> OnStartup, etc.) so external types can satisfy the
interface. The struct and constructor were already exported but the
methods weren't, so no usable external API is displaced.

* Skip nil entries in CompositeHooks and self-contain doc example

Address PR review feedback:

- CompositeHooks.OnX methods skip nil members instead of panicking,
  allowing callers to conditionally append optional hooks.
- doc/hooks.md embedded-usage snippet now defines ctx and version so it
  is self-contained.
…p across 1 directory (github#1606)

* Bump golang.org/x/crypto in the go_modules group across 1 directory

Bumps the go_modules group with 1 update in the / directory: [golang.org/x/crypto](https://github.com/golang/crypto).


Updates `golang.org/x/crypto` from 0.37.0 to 0.45.0
- [Commits](golang/crypto@v0.37.0...v0.45.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.45.0
  dependency-type: indirect
  dependency-group: go_modules
...

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: meiji163 <meiji163@github.com>
…locks (github#1677)

Co-authored-by: meiji163 <meiji163@github.com>
…github#1684)

* Fix resume data loss: route heartbeat coords through applyEventsQueue

onChangelogHeartbeatEvent was mutating applier.CurrentCoordinates directly
from the streamer goroutine, before any DML that preceded the heartbeat was
applied to the ghost table. The checkpoint loop reads CurrentCoordinates as
"applied through this GTID" and could persist a checkpoint whose
LastTrxCoords was ahead of what was actually applied.

If gh-ost crashed before applyEventsQueue drained, --resume read that
checkpoint and called StartSyncGTID with the persisted set; MySQL treated
the un-applied GTIDs as already-seen and never re-streamed them. The ghost
table silently lost those DMLs and cut-over produced a stale table.

Fix: enqueue a tableWriteFunc onto applyEventsQueue that performs the
coords bump. The apply goroutine executes it in order, after the DMLs the
streamer enqueued before the heartbeat, restoring the invariant.

Adds TestMigratorHeartbeatDoesNotAdvancePastUnappliedDML, which fails at
the previous HEAD and passes after the fix; also asserts queue ordering to
guard against future changes that wrap the heartbeat enqueue in a goroutine.

Co-authored-by: Bastian Bartmann <bastian.bartmann@shopify.com>

* Replace direct channel write with SendWithContext

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Bastian Bartmann <bastian.bartmann@shopify.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Add Datadog/statsd with simple client emitting startup

* Add go runtime metrics to statsd reporting

---------

Co-authored-by: meiji163 <meiji163@github.com>
Refactor newTestMigrationContext to set ServeSocketFile via os.TempDir() and remove runtime.Caller-based path derivation.
* Disable CGO for release builds

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…uery latency, sleep (github#1701)

* metrics: introduce unified Emitter interface and consolidate go_runtime helpers

Introduce a single Emitter interface (Gauge, Count, Histogram) on metrics.Client
so all metric helpers share one testable contract. The interface replaces the
narrow MemStatsGaugeEmitter type used only by the Go runtime reporter.

- Add Emitter interface and Histogram method on Client.
- Move go_runtime.go and go_runtime_test.go into emit.go and emit_test.go so the
  metrics package has a single place for helper functions and tests.
- Switch MigrationContext.Metrics from *metrics.Client to metrics.Emitter so
  consumers can be tested with spies.

* Refactor status reporting around migrationProgressSnapshot.

Sample row-copy, DML, backlog, and lag once per tick into a snapshot
passed to printStatus, with reportStatus as the single entry point for
status output.

Co-authored-by: Cursor <cursoragent@cursor.com>

* metrics: emit row-copy progress gauges

Add gh_ost.row_copy.rows_copied, gh_ost.row_copy.rows_estimate, and
gh_ost.dml.events_applied gauges on each reportStatus tick, sampled from
migrationProgressSnapshot.

* metrics: emit binlog backlog gauges

Add gh_ost.binlog.backlog_size, gh_ost.binlog.backlog_capacity, and
gh_ost.binlog.backlog_utilization gauges on each reportStatus tick from
the applyEventsQueue depth captured in migrationProgressSnapshot.

* metrics: emit replication and heartbeat lag gauges

Add gh_ost.lag.replication_seconds and gh_ost.lag.heartbeat_seconds gauges
on each reportStatus tick, tagged with throttled:true|false. These are
point-in-time readings (not distributions), so gauges are used rather than
histograms — DogStatsD histogram aggregation exposes count/max series that
do not match the log line lag values in Prometheus/Grafana.

* metrics: emit throttle active and throttled-interval metrics

Record throttle active state at a debounced cadence (gh_ost.throttle.active)
and emit duration plus event metrics when a throttled interval completes
(gh_ost.throttle.duration_seconds histogram and gh_ost.throttle.events_total
count), each tagged with the throttling reason.

* metrics: emit cut-over phase, attempt, and total duration metrics

Add cut-over metric helpers and instrument cut-over attempts, phase
durations, and terminal duration.

Metrics emitted:
- gh_ost.cut_over.attempts_total tagged with outcome
- gh_ost.cut_over.phase_duration_milliseconds tagged with phase and outcome
- gh_ost.cut_over.total_duration_milliseconds tagged with outcome

Phase coverage includes the magic lock, original table lock, magic rename,
and unlock paths. Durations are reported in milliseconds to preserve
sub-second granularity. The atomic rename phase duration is recorded after
the rename completes so the histogram reflects the full operation.

* metrics: emit query latency histogram

Emit gh_ost.query.duration_milliseconds for representative source-side and
target-side queries (row count and binlog apply), tagged with side, kind,
and outcome (ok|error). Helper validates inputs and is nil-safe.

* metrics: emit per-stage sleep histograms

Add sleep metric helpers and instrument the main migration sleep/wait paths.

Metrics emitted:
- gh_ost.sleep.duration_milliseconds tagged with stage
- gh_ost.sleep.total_milliseconds tagged with stage

Stages covered:
- cut_over_postpone
- chunk_throttle
- retry_backoff
- replica_wait

Use millisecond units so sub-second waits, such as replica polling and
nice-ratio throttling, are not truncated to zero. Skip sub-millisecond
chunk-throttle samples to avoid emitting zero-valued sleeps that would
distort the histogram.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jan Grodowski <jan.grodowski@shopify.com>
* Document missing command-line flags.

Add sections for command-line flags that were registered in the CLI but missing from command-line-flags.md, including logging, throttling, hooks, StatsD, trigger, revert, and basic connection flags.

Update the documented cut-over default from the stale "safe" value to the current "atomic" default.
Rework the replica-test harness to run the localtests suite against
MySQL 5.7, 8.0, 8.4 and Percona Server 8.0 from one parameterized setup.

- Move docker-compose into script/ and split per-version server config
  into docker/<flavor>-<major.minor>/ dirs (common.cnf, create_user.sql,
  start_replication.sql), selected from TEST_MYSQL_IMAGE. Optional
  flavor.env per dir for knobs (Percona reads /etc/my.cnf.d).
- Configure replication from the harness once both servers are up rather
  than at container init, so that replica connects on the first try.
- Keep mysql_native_password for 5.7/8.0/percona, caching_sha2_password for 8.4.
- Speed-ups: native mysql client with docker-exec fallback, detect server
  version once, shorter replication polls, configurable copy-throttle
  window (THROTTLE_SECONDS, default 2s to still exercise binlog DML replay).
- toxiproxy split into its own compose file.
* feat: support MariaDB 11.4 and 11.8

gh-ost mostly works against MariaDB unchanged; this makes the few
MariaDB-specific spots correct and runs the replica test suite against
MariaDB 11.4 and 11.8 (LTS) alongside the MySQL/Percona flavors.

Core:
- ReplicaTermFor keeps the legacy slave/master replication terminology on
  MariaDB. MariaDB reports versions >= 10 (greater than the 8.4 cutoff) but
  never adopted the new replica/source terms, and its SHOW REPLICA STATUS
  still emits legacy column names, so the MySQL 8.4 term map doesn't apply.
- Accept BINLOG MONITOR as well as REPLICATION CLIENT in the grant check
  (MariaDB 10.5+ renamed the privilege).

Test harness (localtests):
- test.sh: pick slave/Master terminology for MariaDB, wait for replica
  catch-up via MASTER_GTID_WAIT/gtid_binlog_pos, and skip gtid_mode-gated
  tests where @@gtid_mode is unsupported instead of aborting the run.
- Resolve the per-flavor config dir by specificity (docker/<flavor>-<major.minor>,
  then <flavor>-<major>, then bare <flavor>) so versions that share one config
  live in a single dir. All MariaDB versions use one docker/mariadb dir (log-bin
  + ROW + log-slave-updates, native-auth users, MariaDB-GTID replication,
  MYSQL_CLIENT=mariadb since 11.x dropped the mysql-named client); a version (or
  major line) needing different config overrides it by adding its own dir. The
  compose healthcheck honours MYSQL_CLIENT.
- Make the zero-date failure tests deterministic across flavors with an
  explicit sql_mode (NO_ZERO_IN_DATE,NO_ZERO_DATE,...); they relied on MySQL's
  default sql_mode, which MariaDB doesn't share.
- Drop the incidental NOT NULL from the generated-column tests (MariaDB
  forbids NOT NULL on generated columns) so they run on MariaDB; keep
  generated-columns-unique skipped (it puts a generated column in the PRIMARY
  KEY, which MariaDB forbids, Error 1903).
- Add mariadb:11.4.12 and mariadb:11.8.8 to the CI matrix.

* feat: support MariaDB 10.5, 10.6 and 10.11

Add the MariaDB 10.x LTS lines (10.5, 10.6, 10.11) to the replica test suite.

MariaDB versions below 11.1 need one core fix: gh-ost passes transaction_isolation as
a DSN param, which the driver applies as `SET transaction_isolation=<v>` on each
connection. That system variable does not exist on MariaDB < 11.1 (which has
only tx_isolation), so the connection fails with Error 1193 "Unknown system
variable 'transaction_isolation'". tx_isolation was in turn removed in MySQL
8.0, so no single variable name is portable across the supported servers.

- mysql.OpenDB strips the transaction_isolation param and instead runs the
  SQL-standard `SET SESSION TRANSACTION ISOLATION LEVEL <level>` on each new
  connection via a connector wrapper, accepted by every supported MySQL and
  MariaDB version. GetDBUri still emits the param unchanged, so DSN
  construction and its tests are unaffected.
- Add mariadb:10.5.29, mariadb:10.6.27 and mariadb:10.11.18 to the CI matrix.
* build: standardize release asset naming (github#1728)

Rework build.sh so released assets have stable, predictable names that
are easy to automate against:

- drop the generated timestamp from tarball names
- drop the "-binary" infix from tarball names
- name macOS tarballs with "darwin" (matches GOOS) instead of "osx"
- drop the version from .rpm/.deb filenames; the release download URL
  already carries it. Package metadata still records version + epoch
- build arm64 .rpm and .deb in addition to amd64, using each ecosystem's
  arch convention (x86_64/aarch64 for rpm, amd64/arm64 for deb)
- stop publishing the bare "gh-ost" binary; it was just the last build
  (darwin/arm64) and was ambiguous. Tarballs and packages cover every
  OS/arch
- emit a SHA256SUMS manifest so downloads can be verified with
  `sha256sum -c SHA256SUMS`

Resulting assets:
  gh-ost-{linux,darwin}-{amd64,arm64}.tar.gz
  gh-ost.{x86_64,aarch64}.rpm
  gh-ost.{amd64,arm64}.deb
  SHA256SUMS

Also refactor build() to take GOOS/GOARCH directly, share common fpm
flags via an array, and update the README build.sh description.

* build: simplify packaging Dockerfile and dock script (github#1728)

Slim down the release packaging tooling:

- Dockerfile.packaging: collapse the per-package RUN layers into one, drop
  packages already provided by the golang:bookworm base (git, tar, curl,
  gcc, g++, bash) and the unused rsync, and drop the legacy GOPATH src
  layout (the build is module-mode). Only ruby/ruby-dev/build-essential
  (for fpm) and rpm (for rpmbuild) are installed now.
- Use a multi-stage build with a `scratch` artifacts stage so
  `docker build --output type=local` exports the assets straight to the
  host. This drops the intermediate container, bind mount, -it TTY and
  find|xargs cp dance from `script/dock pkg`.
- build.sh: stage the fpm package tree in the system temp dir instead of
  under $buildpath, so it never lands among the exported release artifacts.
gh-ost hard-coded the go-mysql binlog syncer to MySQLFlavor and parsed
every GTID set as MySQL, so --gtid could not be used against MariaDB.
go-mysql already speaks the MariaDB GTID dialect; this wires it up.

- Detect server flavor from the version string (IsMariaDB / FlavorFor)
  and set it on the BinlogSyncerConfig.
- Make GTIDBinlogCoordinates hold the flavor-agnostic gomysql.GTIDSet
  interface; parse via ParseGTIDSet(flavor, ...).
- Handle MariadbGTIDEvent alongside GTIDEvent in the streamer, and read
  the committed GTID set from XIDEvent.GSet without a MySQL-only cast.
- Read MariaDB GTID positions from @@global.gtid_binlog_pos,
  Gtid_IO_Pos and @@global.gtid_slave_pos (MariaDB has no
  Executed_Gtid_Set column nor gtid_mode / enforce_gtid_consistency).
- Skip the gtid_mode / enforce_gtid_consistency validation on MariaDB,
  where GTIDs are always recorded when binary logging is enabled.

localtests:
- Detect the server version once and reuse it; run gtid_mode=ON tests on
  MariaDB (normalize its current_gtid_mode to ON) and trim the GTID
  diagnostics that don't exist on MariaDB.
- Enable gtid_strict_mode on the MariaDB test servers. Because
  --test-on-replica makes gh-ost write locally on the replica, give the
  replica its own GTID domain so its writes never collide with the
  primary's domain-0 stream under strict mode.
- Add the gtid-resume case: interrupt a --gtid migration mid-copy via the
  interactive 'panic' command, then --resume, exercising the GTID
  checkpoint round-trip (WriteCheckpoint persists the GTID set,
  ReadLastCheckpoint parses it back via NewGTIDBinlogCoordinates(flavor)).
  Passes across the MySQL, MariaDB and Percona CI matrix.

Verified: full localtests suite passes on MySQL 5.7/8.0/8.4 and on the
MariaDB matrix (10.5, 10.6, 10.11, 11.4, 11.8) with gtid_strict_mode on.
ericyan and others added 12 commits August 7, 2026 11:15
…upstream-analyze-cutover

Add opt-in --analyze-ghost-table-before-cutover
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 7.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v4...v7)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
…actions/upload-artifact-5

Bump actions/upload-artifact from 4 to 7
Signed-off-by: vladdoster <vladislav.doster@icloud.com>
The binlog reader updates its current coordinates while streaming, while go-mysql's BinlogSyncer retains and mutates the GTID set passed to StartSyncGTID.

Clone the caller-provided coordinates for the reader and clone the GTID set again for the syncer. This prevents both components from mutating the caller's reconnect coordinates and eliminates concurrent access to the same MysqlGTIDSet.
The retry-hook test redirected os.Stdout and os.Stderr to capture status
and hook output. Migrate has background status and logging goroutines,
so reassigning those process-global streams races with their output.

Inject a status writer into Migrator, retaining stdout as the default.
The test supplies thread-safe buffers for the outputs, preserving the
separate stdout and stderr assertions without mutating global state.
Fix data races and enable race detector in CI
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@d-bytebase d-bytebase changed the title chore: rebase the Bytebase patches onto upstream v1.1.11 chore: rebase the Bytebase patches onto upstream v1.1.11, and fix an instant-DDL deadlock Nov 2, 2026
d-bytebase added a commit to bytebase/bytebase that referenced this pull request Nov 2, 2026
The require line above is cosmetic on its own: the replace below shadows it,
go.sum holds no upstream gh-ost entries, and that version is never compiled.
What makes this bump real is repointing the replace.

The fork was pinned at a commit based on upstream v1.1.8, so Bytebase shipped
none of upstream's MariaDB support, the migration-state data-race fixes, or the
binlog decode optimization that skips rows for unrelated tables. gh-ost dispatch
is directive-based with no engine gate and SQL Review recommends gh-ost for
MariaDB, so MariaDB and OceanBase gh-ost runs were reachable on that build.

Repoints at bytebase/gh-ost2#10, which rebases the nine Bytebase patches onto
upstream v1.1.11 and fixes a deadlock that the rebase exposed: with
--attempt-instant-ddl, which Bytebase enables for every migration, Migrate()
never returned, because the changelog listener parked on an unbuffered send
while holding the mutex the binlog parser needs and finalCleanup waited on the
parser through BinlogSyncer.Close. TestGhostSchemaUpdate reproduces it.

No source change is needed here: the API surface Bytebase uses is unchanged.

go mod tidy raises go-mysql to v1.15.0 and pulls in DataDog/datadog-go, which
gh-ost's new statsd code needs but never constructs here. The require line for
pingcap/tidb/pkg/parser moves too, but a replace pins that module to the
Bytebase fork, so the compiled parser is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GhostDatabaseName puts the ghost, old and changelog tables in a schema of
their own instead of beside the original table, and the temporary-table
prefix becomes `~` rather than `_`. Only the checkpoint table stays with
the original table, since it records that table's progress.

Grant validation now requires the privileges on every database the
migration writes to and names the ones that are missing, and
--include-triggers bails out when the ghost database differs, because a
trigger cannot follow its table into another schema.
go-sql-driver registers TLS configs in a process-global map keyed by name.
Keying on the server name alone makes concurrent migrations against the
same host overwrite each other's config, so the key now carries gh-ost's
own uuid. An empty key keeps the previous "ghost-<host>" form.
ServeSocketFile is empty when gh-ost runs in-process rather than as a
CLI, so BindUnixSocket is never called and unixListener stays nil. The
accept loop panicked on it; the TCP loop already had this guard.
Teardown closed only the database handle, leaking the binlog reader's
connection for the lifetime of the process that embeds gh-ost. It now
closes the reader too, and both it and Close tolerate the nil reader and
nil handle left behind when InitDBConnections fails early.
The goroutine logged "Done streaming" after handing the error to
panicAbort, which reads as a clean finish in the log of a migration that
is in fact aborting.
Every step after LockOriginalTable runs with a write lock held on the
original table. A failure in any of them returned without unlocking, so
the table stayed unwritable for the life of the process that embeds
gh-ost.
The streamer reconnects the binlog reader on every read failure, so bad
credentials or a host the server refuses produced an unbounded reconnect
loop that reads as a brute-force attempt and gets the host firewalled.

Consecutive authentication failures are now counted on the migration
context, which outlives the reader the streamer recreates, and the
failure that reaches MaxAuthFailures is wrapped in ErrMaxAuthFailures so
the streamer stops instead of reconnecting. MaxAuthFailures of zero keeps
the old unbounded behavior.
A caller that reaches the server through a tunnel can now register a dial
context with go-sql-driver and name it in ConnectionConfig.Network, which
replaces tcp in the DSN. ConnectionConfig.Dialer carries the same tunnel
to go-mysql's binlog connections, which do not go through go-sql-driver.
Both are empty or nil by default, which keeps a plain tcp dial.
@d-bytebase
d-bytebase force-pushed the bytebase/rebase-v1.1.11 branch from b5db74f to 8a2e2d7 Compare November 2, 2026 12:32
d-bytebase added a commit to bytebase/bytebase that referenced this pull request Nov 2, 2026
The require line above is cosmetic on its own: the replace below shadows it,
go.sum holds no upstream gh-ost entries, and that version is never compiled.
What makes this bump real is repointing the replace.

The fork was pinned at a commit based on upstream v1.1.8, so Bytebase shipped
none of upstream's MariaDB support, the migration-state data-race fixes, or the
binlog decode optimization that skips rows for unrelated tables. gh-ost dispatch
is directive-based with no engine gate and SQL Review recommends gh-ost for
MariaDB, so MariaDB and OceanBase gh-ost runs were reachable on that build.

Repoints at bytebase/gh-ost2#10, which rebases the nine Bytebase patches onto
upstream v1.1.11 and fixes a deadlock that the rebase exposed: with
--attempt-instant-ddl, which Bytebase enables for every migration, Migrate()
never returned, because the changelog listener parked on an unbuffered send
while holding the mutex the binlog parser needs and finalCleanup waited on the
parser through BinlogSyncer.Close. TestGhostSchemaUpdate reproduces it.

No source change is needed here: the API surface Bytebase uses is unchanged.

go mod tidy raises go-mysql to v1.15.0 and pulls in DataDog/datadog-go, which
gh-ost's new statsd code needs but never constructs here. The require line for
pingcap/tidb/pkg/parser moves too, but a replace pins that module to the
Bytebase fork, so the compiled parser is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A migration that --attempt-instant-ddl completes never returned. Three
goroutines formed a cycle:

  - the changelog listener parks on the unbuffered GhostTableMigrated
    send, which the instant-DDL path returns without ever reading;
  - notifyListeners held listenersMutex across that synchronous call, so
    shouldDecodeRowsEvent — called on the binlog parser's goroutine since
    upstream github#1699 — stalled on the same mutex;
  - BinlogSyncer.Close, which finalCleanup calls, waits for that parser
    goroutine.

Migrate() therefore blocked until the migration context was cancelled.
notifyListeners now collects the matching listeners under the mutex and
invokes them after releasing it, and ghostTableMigrated gets a one-slot
buffer, since the changelog carries exactly one such state per migration.

This is an upstream defect, still present on gh-ost master. It surfaces
here because Bytebase enables --attempt-instant-ddl for every migration,
so every ALTER that MySQL can apply instantly hung.
@d-bytebase
d-bytebase force-pushed the bytebase/rebase-v1.1.11 branch from 8a2e2d7 to 6ff100b Compare November 2, 2026 12:57
d-bytebase added a commit to bytebase/bytebase that referenced this pull request Nov 2, 2026
The require line above is cosmetic on its own: the replace below shadows it,
go.sum holds no upstream gh-ost entries, and that version is never compiled.
What makes this bump real is repointing the replace.

The fork was pinned at a commit based on upstream v1.1.8, so Bytebase shipped
none of upstream's MariaDB support, the migration-state data-race fixes, or the
binlog decode optimization that skips rows for unrelated tables. gh-ost dispatch
is directive-based with no engine gate and SQL Review recommends gh-ost for
MariaDB, so MariaDB and OceanBase gh-ost runs were reachable on that build.

Repoints at bytebase/gh-ost2#10, which rebases the nine Bytebase patches onto
upstream v1.1.11 and fixes a deadlock that the rebase exposed: with
--attempt-instant-ddl, which Bytebase enables for every migration, Migrate()
never returned, because the changelog listener parked on an unbuffered send
while holding the mutex the binlog parser needs and finalCleanup waited on the
parser through BinlogSyncer.Close. TestGhostSchemaUpdate reproduces it.

No source change is needed here: the API surface Bytebase uses is unchanged.

go mod tidy raises go-mysql to v1.15.0 and pulls in DataDog/datadog-go, which
gh-ost's new statsd code needs but never constructs here. The require line for
pingcap/tidb/pkg/parser moves too, but a replace pins that module to the
Bytebase fork, so the compiled parser is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
d-bytebase added a commit to bytebase/bytebase that referenced this pull request Nov 2, 2026
* fix(deps): update module github.com/github/gh-ost to v1.1.11

* chore(deps): point the gh-ost fork at upstream v1.1.11

The require line above is cosmetic on its own: the replace below shadows it,
go.sum holds no upstream gh-ost entries, and that version is never compiled.
What makes this bump real is repointing the replace.

The fork was pinned at a commit based on upstream v1.1.8, so Bytebase shipped
none of upstream's MariaDB support, the migration-state data-race fixes, or the
binlog decode optimization that skips rows for unrelated tables. gh-ost dispatch
is directive-based with no engine gate and SQL Review recommends gh-ost for
MariaDB, so MariaDB and OceanBase gh-ost runs were reachable on that build.

Repoints at bytebase/gh-ost2#10, which rebases the nine Bytebase patches onto
upstream v1.1.11 and fixes a deadlock that the rebase exposed: with
--attempt-instant-ddl, which Bytebase enables for every migration, Migrate()
never returned, because the changelog listener parked on an unbuffered send
while holding the mutex the binlog parser needs and finalCleanup waited on the
parser through BinlogSyncer.Close. TestGhostSchemaUpdate reproduces it.

No source change is needed here: the API surface Bytebase uses is unchanged.

go mod tidy raises go-mysql to v1.15.0 and pulls in DataDog/datadog-go, which
gh-ost's new statsd code needs but never constructs here. The require line for
pingcap/tidb/pkg/parser moves too, but a replace pins that module to the
Bytebase fork, so the compiled parser is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Danny Xu <d@bytebase.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.