Repository navigation
chore(deps): point the gh-ost fork at upstream v1.1.11 - #21860
Closed
d-bytebase wants to merge 1 commit into
Closed
d-bytebase wants to merge 1 commit into
d-bytebase wants to merge 1 commit into
Conversation
The replace pinned github.com/bytebase/gh-ost2 at a commit based on upstream v1.1.8, so Bytebase shipped none of upstream's MariaDB support, the migration-state data-race fixes, or the binlog decode optimization that skips rows for unrelated tables. gh-ost dispatch is directive-based with no engine gate and SQL Review recommends gh-ost for MariaDB, so MariaDB and OceanBase gh-ost runs were reachable on that build. Repoints at bytebase/gh-ost2#10, which rebases the nine Bytebase patches onto upstream v1.1.11 and fixes a deadlock that the rebase exposed: with --attempt-instant-ddl, which Bytebase enables for every migration, Migrate() never returned, because the changelog listener parked on an unbuffered send while holding the mutex the binlog parser needs and finalCleanup waited on the parser through BinlogSyncer.Close. TestGhostSchemaUpdate reproduces it. No source change is needed here: the API surface Bytebase uses is unchanged. The require line stays as it is, since the replace shadows it. go mod tidy raises go-mysql to v1.15.0 and pulls in DataDog/datadog-go, which gh-ost's new statsd code needs but never constructs here. The require line for pingcap/tidb/pkg/parser moves too, but a replace pins that module to the Bytebase fork, so the compiled parser is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repoints the
github.com/github/gh-ostreplace at the Bytebase fork rebased ontoupstream
v1.1.11(bytebase/gh-ost2#10). The old pin,v1.1.7-0.20260512035759-ca2794d6ebea, sat on upstreamv1.1.8and so shippednone of upstream's MariaDB support, the migration-state data-race fixes, or the
binlog decode optimisation that skips rows for unrelated tables.
Why it matters beyond tidiness: gh-ost dispatch is directive-based with no engine
gate —
backend/runner/taskrun/database_migrate_executor.go:125routes torunGhostMigrationwheneverghost.IsGhostEnabled(sheet.Statement)holds, on anyinstance — and SQL Review registers an online-migration advisor for MariaDB at
backend/plugin/advisor/mariadb/rule_online_migration.go:25that recommendsgh-ost. MariaDB and OceanBase gh-ost runs were therefore reachable on a build
that predated upstream's MariaDB work.
The
require github.com/github/gh-ost v1.1.10line is left alone: the replaceshadows it,
go.sumholds no upstream gh-ost entries, and that version is nevercompiled. Renovate's #21794 is not folded in.
What the fork branch carries
Nine Bytebase patches, one per commit, each reapplied by hand against upstream's
current code (upstream renamed every receiver). The preserved and dropped lists,
with reasons, are in bytebase/gh-ost2#10.
It also carries one fix that is not a carried-forward patch. Without it this
bump hangs every gh-ost migration that MySQL can apply instantly, which is all
of them that qualify, since
defaultConfig.attemptInstantDDListrue. Therebase first failed
TestGhostSchemaUpdatehere whilemainpassed it 3/3 in~23s; a focused reproduction in gh-ost pinned it to a three-goroutine cycle after
--attempt-instant-ddlshort-circuits the migration:GhostTableMigratedsend, whichthat path returns without reading;
notifyListenersheldlistenersMutexacross the synchronous call, soshouldDecodeRowsEvent— on the binlog parser's goroutine since upstreamfix: using bash syntax instead of sh #1699, first released in v1.1.11 — stalled on the same mutex;
BinlogSyncer.Close, called fromfinalCleanup, waits for that parsergoroutine.
So
migrator.Migrate()never returned and the task run sat until its context wascancelled. The defect is upstream's, is still on gh-ost
master, and is invisiblethere because
--attempt-instant-ddldefaults to off. bytebase/gh-ost2#10 fixesit and adds
TestMigrator/TestMigrateInstantDDLReturnsas the regression lock; itshould go upstream too.
No source fallout
No Go file in this repo needed a change. The API surface Bytebase uses is
unchanged:
ghostbase.{MigrationContext,NewMigrationContext,CutOverAtomic,ParseLoadMap,SendWithContext},ghostsql.NewParserFromAlterStatement,logic.NewMigrator, and theGhostDatabaseName/Network/Dialer/MaxAuthFailuresfields thatbackend/component/ghost/config.gosets all still exist with the samesignatures. The temp-table prefix stays
~and the ghost schema stayscommon.BackupDatabaseNameOfEngine(MYSQL), sobackend/runner/plancheck/ghost_sync_executor.goandbackend/runner/taskrun/database_migrate_executor.gostill drop the righttables.
Behavior Bytebase now inherits
unconditional
GET_LOCK('gh-ost::<db>.<table>', 0)ininitiateApplier, withno flag to disable it. Two gh-ost runs against the same table on one server
used to race; the second now errors with "another gh-ost process is already
migrating ...". A plan check re-run while a rollout task is mid-migration is the
realistic way to hit this, and failing is the right outcome. The lock
connection is opened from the same
ConnectionConfig, so it goes through theSSH tunnel like every other connection.
MariaDB GTID migrations (upstream chore: add metric for OpenAPI #1724, refactor: clickhouse cluster support step 1 #1731).
reader/syncer ownership.
range query for two-column keys).
it brings
DataDog/datadog-go/v5in as a new module, butMigrationContext.Metricsis only assigned in gh-ost's ownmain.go;Bytebase builds the context itself, so it stays nil, and all 11
emitter-taking functions in
go/metrics/emit.goreturn early on nil. Nosocket is opened and no packet is sent.
--analyze-ghost-table-before-cutoveris opt-in and defaults to false.Bytebase never sets it.
gh-ost task-run logs shifts slightly. Nothing in this repo matches on them.
Dependency changes
go mod tidybrought these along. The list is the diff ofgo list -m allagainst
origin/main, so it is what actually compiles, not whatgo.modsays:go-mysql-org/go-mysqlDataDog/datadog-go/v5BurntSushi/tomlpingcap/errorspingcap/failpointpingcap/logsiddontang/go-logtestcontainers-go/modules/mysqltestcontainers-goitself did not move.Masterminds/semveralso leaves the// indirectblock, but the module is stillin the graph at the same version (
go list -m allhas it before and after) — onlythe bookkeeping changed, because nothing in the build list needs it declared any
more.
go.mod'srequireline forpingcap/tidb/pkg/parseris bumped from20241125141335to20260504140133by MVS (go-mysql v1.15.0 requires it), butthat is cosmetic:
github.com/pingcap/tidb/pkg/parser => github.com/bytebase/tidb/pkg/parser v0.0.0-20251104040057-d29df9dd1b3bshadows it.
go list -mconfirms the compiled parser is byte-identical toorigin/main's. The same holds forpingcap/tidb. So the SQL parser Bytebaseuses does not change in this PR.
Verification
gofmt: no Go file changed.golangci-lint run -j 8at v2.15.0, the version.github/workflows/ci.ymlpins: 0 issues.
go build -ldflags "-w -s" -p=16 -o ./bytebase-build/bytebase ./backend/bin/server/main.go: clean.go test -v -count=1 ./backend/component/ghost ./backend/runner/plancheck ./backend/runner/taskrun: pass.go test -v -count=1 ./backend/plugin/parser/tidb ./backend/plugin/schema/tidb ./backend/plugin/advisor/tidb:pass. Run on top of the required set because the
pingcap/errors,failpointand
logbumps are reachable from the TiDB parser.go test -v -count=1 ./backend/tests/ -run '^TestGhostSchemaUpdate$' -timeout 10m:pass. This is the end-to-end gh-ost rollout against a real MySQL instance, and
it is the test that caught the instant-DDL deadlock. With the fix it passes 2/2
(34.9s, 47.2s under heavy load on this box). Without it, the same pin failed 3/3
at ~140s each on the rollout wait, while
origin/mainpassed 3/3 at ~23s.go test -race ./go/...(all 6 packages, 0 dataraces) and all 90
localtestsagainst a MySQL 8.0.46 primary/replica pair.Noted, not changed
Both runners deregister the driver TLS config as
gomysql.DeregisterTLSConfig(migrationContext.Uuid), but gh-ost registers itunder
GetDBTLSConfigKey(uuid, serverName), i.e."<uuid>-<host>". Thederegister therefore removes nothing and the driver's global TLS config map grows
by an entry per TLS gh-ost migration for the life of the process. This predates
this PR — the fork has keyed on
tlsKey-serverNamesince fork PR #4 — so it isleft alone here, together with the comment above each call that still says the
key is the uuid.
Not verified
MariaDB was not exercised. Upstream's MariaDB support arrives with this bump,
but nothing here ran gh-ost against a MariaDB server, so this PR is not evidence
that the MariaDB gh-ost path works end to end. That needs its own run against a
MariaDB instance before anyone relies on it.
Pre-PR checklist
Sections 3 (composite-PK), 4 (pagination), 5 (transaction lock ordering) and 6
(image compatibility) are skipped by their own skip conditions: the diff is
go.modandgo.sumonly, and touches no query, no migration and no versionmetadata. Section 2 found no breaking change — no API, schema, proto, config or
UI surface moves, and the gh-ost flags Bytebase exposes are unchanged.
🤖 Generated with Claude Code