Skip to content

chore(deps): point the gh-ost fork at upstream v1.1.11 - #21860

Closed
d-bytebase wants to merge 1 commit into
mainfrom
d/ghost-v1.1.11
Closed

d-bytebase wants to merge 1 commit into
mainfrom
d/ghost-v1.1.11

Conversation

@d-bytebase

Copy link
Copy Markdown
Collaborator

Repoints the github.com/github/gh-ost replace at the Bytebase fork rebased onto
upstream v1.1.11 (bytebase/gh-ost2#10). The old pin,
v1.1.7-0.20260512035759-ca2794d6ebea, sat on upstream v1.1.8 and so shipped
none of upstream's MariaDB support, the migration-state data-race fixes, or the
binlog decode optimisation that skips rows for unrelated tables.

Why it matters beyond tidiness: gh-ost dispatch is directive-based with no engine
gate — backend/runner/taskrun/database_migrate_executor.go:125 routes to
runGhostMigration whenever ghost.IsGhostEnabled(sheet.Statement) holds, on any
instance — and SQL Review registers an online-migration advisor for MariaDB at
backend/plugin/advisor/mariadb/rule_online_migration.go:25 that recommends
gh-ost. MariaDB and OceanBase gh-ost runs were therefore reachable on a build
that predated upstream's MariaDB work.

The require github.com/github/gh-ost v1.1.10 line is left alone: the replace
shadows it, go.sum holds no upstream gh-ost entries, and that version is never
compiled. Renovate's #21794 is not folded in.

What the fork branch carries

Nine Bytebase patches, one per commit, each reapplied by hand against upstream's
current code (upstream renamed every receiver). The preserved and dropped lists,
with reasons, are in bytebase/gh-ost2#10.

It also carries one fix that is not a carried-forward patch. Without it this
bump hangs every gh-ost migration that MySQL can apply instantly
, which is all
of them that qualify, since defaultConfig.attemptInstantDDL is true. The
rebase first failed TestGhostSchemaUpdate here while main passed it 3/3 in
~23s; a focused reproduction in gh-ost pinned it to a three-goroutine cycle after
--attempt-instant-ddl short-circuits the migration:

  • the changelog listener parks on the unbuffered GhostTableMigrated send, which
    that path returns without reading;
  • notifyListeners held listenersMutex across the synchronous call, so
    shouldDecodeRowsEvent — on the binlog parser's goroutine since upstream
    fix: using bash syntax instead of sh #1699, first released in v1.1.11 — stalled on the same mutex;
  • BinlogSyncer.Close, called from finalCleanup, waits for that parser
    goroutine.

So migrator.Migrate() never returned and the task run sat until its context was
cancelled. The defect is upstream's, is still on gh-ost master, and is invisible
there because --attempt-instant-ddl defaults to off. bytebase/gh-ost2#10 fixes
it and adds TestMigrator/TestMigrateInstantDDLReturns as the regression lock; it
should go upstream too.

No source fallout

No Go file in this repo needed a change. The API surface Bytebase uses is
unchanged: ghostbase.{MigrationContext,NewMigrationContext,CutOverAtomic,ParseLoadMap,SendWithContext},
ghostsql.NewParserFromAlterStatement, logic.NewMigrator, and the
GhostDatabaseName / Network / Dialer / MaxAuthFailures fields that
backend/component/ghost/config.go sets all still exist with the same
signatures. The temp-table prefix stays ~ and the ghost schema stays
common.BackupDatabaseNameOfEngine(MYSQL), so
backend/runner/plancheck/ghost_sync_executor.go and
backend/runner/taskrun/database_migrate_executor.go still drop the right
tables.

Behavior Bytebase now inherits

  • Concurrent migrations of one table now fail fast. Upstream feat: implement schema check API with OpenAPI specification #1693 added an
    unconditional GET_LOCK('gh-ost::<db>.<table>', 0) in initiateApplier, with
    no flag to disable it. Two gh-ost runs against the same table on one server
    used to race; the second now errors with "another gh-ost process is already
    migrating ...". A plan check re-run while a rollout task is mid-migration is the
    realistic way to hit this, and failing is the right outcome. The lock
    connection is opened from the same ConnectionConfig, so it goes through the
    SSH tunnel like every other connection.
  • MariaDB server-flavor detection, MariaDB replication and privileges, and
    MariaDB GTID migrations
    (upstream chore: add metric for OpenAPI #1724, refactor: clickhouse cluster support step 1 #1731).
  • Data-race fixes in migration state and GTID coordinate cloning for
    reader/syncer ownership.
  • Binlog events of unrelated tables are no longer decoded (upstream fix: using bash syntax instead of sh #1699).
  • Composite-primary-key row copy is faster (upstream chore: dispose autocomplete provider when unmount #1686, a UNION-based
    range query for two-column keys).
  • statsd/Datadog instrumentation is linked but inert. Upstream added it, and
    it brings DataDog/datadog-go/v5 in as a new module, but
    MigrationContext.Metrics is only assigned in gh-ost's own main.go;
    Bytebase builds the context itself, so it stays nil, and all 11
    emitter-taking functions in go/metrics/emit.go return early on nil. No
    socket is opened and no packet is sent.
  • --analyze-ghost-table-before-cutover is opt-in and defaults to false.
    Bytebase never sets it.
  • gh-ost's own error strings are lowercase upstream now, so the wording in
    gh-ost task-run logs shifts slightly. Nothing in this repo matches on them.

Dependency changes

go mod tidy brought these along. The list is the diff of go list -m all
against origin/main, so it is what actually compiles, not what go.mod says:

module change reach
go-mysql-org/go-mysql v1.11.0 → v1.15.0 gh-ost's binlog reader only. Nothing in this repo imports it.
DataDog/datadog-go/v5 added v5.8.3 gh-ost's statsd client, never constructed (above).
BurntSushi/toml v1.5.0 → v1.6.0 pulled by go-mysql.
pingcap/errors 20250523 → 20260310 pulled by go-mysql; also reached by the TiDB parser fork.
pingcap/failpoint 20240528 → 20260406 same.
pingcap/log 20250917 → 20260227 same.
siddontang/go-log dropped go-mysql v1.15 stopped using it.
testcontainers-go/modules/mysql v0.41.0 → v0.43.0 test-only; testcontainers-go itself did not move.

Masterminds/semver also leaves the // indirect block, but the module is still
in the graph at the same version (go list -m all has it before and after) — only
the bookkeeping changed, because nothing in the build list needs it declared any
more.

go.mod's require line for pingcap/tidb/pkg/parser is bumped from
20241125141335 to 20260504140133 by MVS (go-mysql v1.15.0 requires it), but
that is cosmetic: github.com/pingcap/tidb/pkg/parser => github.com/bytebase/tidb/pkg/parser v0.0.0-20251104040057-d29df9dd1b3b
shadows it. go list -m confirms the compiled parser is byte-identical to
origin/main's. The same holds for pingcap/tidb. So the SQL parser Bytebase
uses does not change in this PR.

Verification

  • gofmt: no Go file changed.
  • golangci-lint run -j 8 at v2.15.0, the version .github/workflows/ci.yml
    pins: 0 issues.
  • go build -ldflags "-w -s" -p=16 -o ./bytebase-build/bytebase ./backend/bin/server/main.go: clean.
  • go test -v -count=1 ./backend/component/ghost ./backend/runner/plancheck ./backend/runner/taskrun: pass.
  • go test -v -count=1 ./backend/plugin/parser/tidb ./backend/plugin/schema/tidb ./backend/plugin/advisor/tidb:
    pass. Run on top of the required set because the pingcap/errors, failpoint
    and log bumps are reachable from the TiDB parser.
  • go test -v -count=1 ./backend/tests/ -run '^TestGhostSchemaUpdate$' -timeout 10m:
    pass. This is the end-to-end gh-ost rollout against a real MySQL instance, and
    it is the test that caught the instant-DDL deadlock. With the fix it passes 2/2
    (34.9s, 47.2s under heavy load on this box). Without it, the same pin failed 3/3
    at ~140s each on the rollout wait, while origin/main passed 3/3 at ~23s.
  • On the fork branch itself: go test -race ./go/... (all 6 packages, 0 data
    races) and all 90 localtests against a MySQL 8.0.46 primary/replica pair.

Noted, not changed

Both runners deregister the driver TLS config as
gomysql.DeregisterTLSConfig(migrationContext.Uuid), but gh-ost registers it
under GetDBTLSConfigKey(uuid, serverName), i.e. "<uuid>-<host>". The
deregister therefore removes nothing and the driver's global TLS config map grows
by an entry per TLS gh-ost migration for the life of the process. This predates
this PR — the fork has keyed on tlsKey-serverName since fork PR #4 — so it is
left alone here, together with the comment above each call that still says the
key is the uuid.

Not verified

MariaDB was not exercised. Upstream's MariaDB support arrives with this bump,
but nothing here ran gh-ost against a MariaDB server, so this PR is not evidence
that the MariaDB gh-ost path works end to end. That needs its own run against a
MariaDB instance before anyone relies on it.

Pre-PR checklist

Sections 3 (composite-PK), 4 (pagination), 5 (transaction lock ordering) and 6
(image compatibility) are skipped by their own skip conditions: the diff is
go.mod and go.sum only, and touches no query, no migration and no version
metadata. Section 2 found no breaking change — no API, schema, proto, config or
UI surface moves, and the gh-ost flags Bytebase exposes are unchanged.

🤖 Generated with Claude Code

The replace pinned github.com/bytebase/gh-ost2 at a commit based on upstream
v1.1.8, so Bytebase shipped none of upstream's MariaDB support, the
migration-state data-race fixes, or the binlog decode optimization that skips
rows for unrelated tables. gh-ost dispatch is directive-based with no engine
gate and SQL Review recommends gh-ost for MariaDB, so MariaDB and OceanBase
gh-ost runs were reachable on that build.

Repoints at bytebase/gh-ost2#10, which rebases the nine Bytebase patches onto
upstream v1.1.11 and fixes a deadlock that the rebase exposed: with
--attempt-instant-ddl, which Bytebase enables for every migration, Migrate()
never returned, because the changelog listener parked on an unbuffered send
while holding the mutex the binlog parser needs and finalCleanup waited on the
parser through BinlogSyncer.Close. TestGhostSchemaUpdate reproduces it.

No source change is needed here: the API surface Bytebase uses is unchanged. The
require line stays as it is, since the replace shadows it.

go mod tidy raises go-mysql to v1.15.0 and pulls in DataDog/datadog-go, which
gh-ost's new statsd code needs but never constructs here. The require line for
pingcap/tidb/pkg/parser moves too, but a replace pins that module to the
Bytebase fork, so the compiled parser is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@d-bytebase
d-bytebase requested a review from a team as a code owner November 2, 2026 12:09
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-02T12:10:51.508660Z c115f78 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@d-bytebase

Copy link
Copy Markdown
Collaborator Author

Folded into #21794 instead, so the require bump Renovate already opened and the replace repoint land together. Same two commits' worth of change; #21794 carries Renovate's commit plus the replace repoint, rebased onto current main.

@d-bytebase d-bytebase closed this Nov 2, 2026
@d-bytebase
d-bytebase deleted the d/ghost-v1.1.11 branch November 2, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant