feat: rebuild shared MySQL platform

[viasnake]

Problem

Why

• The single-node shared MySQL service turns OS and MySQL maintenance into downtime.
• Read-heavy work needs a replica path instead of competing with writes on one server.
• The platform should provide a practical place to learn MySQL replication and Router operations.
• Enterprise-scale automatic HA is unnecessary for the current environment.
• A simple asynchronous InnoDB ReplicaSet is the right starting point.
• Prometheus-based monitoring is retired in favor of one Zabbix stack.

Changes

Architecture

Component	Placement	Responsibility
MySQL Server 8.4	mysql-shared01, mysql-shared02	Database engine, GTID, row-based binary logs
InnoDB ReplicaSet	both MySQL nodes	Asynchronous primary/secondary topology
MySQL Router 8.4	web01, workbench01, control01	Local RW, RO, and split endpoints
XtraBackup 8.4	both MySQL nodes	Role-aware physical backup and restore validation
Zabbix 7.0 LTS	monitor01 and Agent 2 on all hosts	Metrics, health checks, and alerting
NSD	both authoritative DNS servers	Stable node records and operator-facing role aliases

Application databases and users are declared as tenants. Application repositories remain responsible for schema migrations.

MySQL ReplicaSet

• Adds mysql-shared02 at 10.10.10.222 with a unique server_id.
• Pins MySQL Server and Shell to the 8.4 LTS series.
• Enables GTID, row-based binary logging, log_replica_updates, relay recovery, TLS-required remote traffic, and seven-day binlog retention.
• Uses MySQL Shell AdminAPI to create, add, rejoin, inspect, switch, and force-fail over the ReplicaSet.
• Treats mysql-shared01 only as the initial bootstrap seed. Runtime status owns the PRIMARY and SECONDARY roles.
• Keeps topology changes out of normal site.yml convergence.

MySQL Router

Router runs beside each client rather than on a central host. Bootstrap registers each Router in ReplicaSet metadata, stores the generated credential in its keyring, and reuses the generated TLS certificate and private key. Router binds only to loopback and UNIX sockets; Router-to-MySQL and metadata traffic requires TLS.

Read/write routing

• 127.0.0.1:6446 and /run/mysqlrouter/mysql-rw.sock route to the current primary.
• 127.0.0.1:6447 and /run/mysqlrouter/mysql-ro.sock prefer the current secondary and use Router's primary fallback when needed.
• 127.0.0.1:6450 and /run/mysqlrouter/mysql-rw-split.sock use statement-aware read/write splitting with connection sharing.
• Split routing enables wait_for_my_writes=1 with a one-second timeout. Workloads requiring cross-connection read-after-write guarantees must use the RW endpoint.

Backup and restore

• Installs XtraBackup 8.4 and identical tooling on both MySQL nodes.
• Scheduled jobs inspect the local runtime role and run a normal full backup only on the healthy secondary.
• A primary backup requires an exact target and an explicit override.
• Requires an absolute mounted NFS, NFS4, CIFS, or SSHFS repository. There is no local fallback.
• Prepares each backup, copies it through an atomic incoming path, and archives closed binlogs by node identity and server UUID with checksums and GTID metadata.
• Adds an isolated restore test that copies a prepared backup into a guarded scratch datadir, starts a network-disabled temporary mysqld, runs SELECT 1, verifies expected databases, proves process ownership before shutdown, and cleans up.
• Exposes machine-readable backup and restore status for Zabbix.

Zabbix migration

• Rebuilds monitor01 as Zabbix Server 7.0 LTS, frontend, and local PostgreSQL. Monitoring does not depend on the shared MySQL service.
• Installs Zabbix Agent 2 through foundation provisioning on every managed host.
• Links the packaged Linux and MySQL Agent 2 templates.
• Adds custom MySQL role, replication, GTID, row-binlog, DNS consistency, backup, restore, and Router synthetic checks.
• Uses a local UNIX-socket MySQL monitoring session and source-restricted monitoring accounts.
• Rotates the packaged Zabbix API bootstrap password to the required operator secret during local API reconciliation.

DNS model

Stable identities remain mysql-shared01.srv.alflag.internal and mysql-shared02.srv.alflag.internal. Topology operations derive the operator aliases from fresh ReplicaSet status:

• mysql-shared-primary.srv.alflag.internal
• mysql-shared-replica.srv.alflag.internal

The aliases are not application routing endpoints. The updater validates and atomically writes the runtime fragment on both authoritative servers, handles SOA serial wrap using RFC 1982 ordering, requires matching serials and record hashes, reloads NSD, and records the successfully loaded state. Forced failover omits the replica alias while no healthy secondary exists.

Impact

Breaking changes

• mysql-shared01 may be rebuilt.
• mysql-shared02 is newly introduced.
• The old MySQL service alias is removed.
• The old zone-prefixed MySQL FQDN is removed.
• Prometheus is removed.
• Grafana is removed.
• Alertmanager is removed.
• blackbox_exporter is removed.
• node_exporter is removed.
• Monitoring is replaced by Zabbix.
• No compatibility layer is provided.

Validation

• mise run check passes: 51 unit tests, seven playbook syntax checks, and production-profile YAML/Ansible lint with zero failures and zero warnings.
• ansible-inventory --graph resolves two MySQL nodes, the three Router clients, and monitor01 in the intended groups.
• node --check passes for the MySQL Shell program; all 22 repository Python files compile without generated bytecode.
• git diff --check passes.
• Repository searches find zero old monitoring component references, obsolete MySQL names, fixed runtime role declarations, or deleted-role includes.
• All 230 tracked files were checked: no tracked symlinks or world-writable files are present.
• A read-only NetBox query reports zero assignments for 10.10.10.222 and no device or VM named as a backup, NAS, or storage candidate.
• Read-only host checks reach control01, workbench01, monitor01, both authoritative DNS servers, and mysql-shared01. The existing mysql-shared01 reports MySQL 8.4.10.
• A negative failover check proves the playbook stops without an exact target and force:<target> confirmation.

Operational procedures

Current procedures and secret contracts are documented in docs/mysql-platform.md. The explicit topology and data-safety operations are:

.venv/bin/ansible-playbook playbooks/operations/mysql-switchover.yml \
  -e mysql_target_primary=<current-secondary>

.venv/bin/ansible-playbook playbooks/operations/mysql-failover.yml \
  -e mysql_target_primary=<current-secondary> \
  -e mysql_failover_confirmation=force:<current-secondary>

.venv/bin/ansible-playbook playbooks/operations/mysql-backup.yml
.venv/bin/ansible-playbook playbooks/operations/mysql-restore-test.yml

Planned maintenance switches primary away from the node being serviced, applies targeted OS/MySQL maintenance, rejoins and catches up the node, then optionally switches primary back.

Limitations

• No environment apply was attempted. mysql-shared02 is unreachable with No route to host, and web01 SSH times out.
• The ignored secret file is absent, and NetBox/repository inspection found no existing off-host backup destination. The roles intentionally fail preflight until the required secrets and mysql_backup_repository mount are supplied.
• Runtime ReplicaSet creation, Router bootstrap on all three clients, actual switchover/failover, secondary backup, isolated restore, Zabbix reconciliation, and post-operation DNS checks remain unverified in the environment. They are blocked, not passed.
• InnoDB ReplicaSet uses asynchronous replication. Unexpected primary loss requires manual, explicitly confirmed failover; automatic election and RPO=0 are not provided.