feat: backup and restore (vault + mysql) by Deezzir · Pull Request #876 · canonical/snap-openstack

Deezzir · 2026-07-06T05:28:28Z

Overview

Adds sunbeam backup, sunbeam list-backups, and sunbeam restore commands for stateful workloads in the OpenStack model.
Introduces a component-driven backup/restore framework (BackupComponent) so command flow is generic while component specifics (target resolution, parse/list/restore plans) stay in one registry.
Supports MySQL and Vault, including backup inventory listing, restore execution, and manifest output.

Automation

In addition to the new commands, I am working on an experimental feature to automate the deployment and configuration of the s3-integrator charm for each MySQL and Vault component detected in the cluster. The branch is currently based on the current changes.

See: #890

The feature, at a minimum, deploys the s3-integrator for each target application, but the operator can choose to interactively set up the S3 location (single for all integrators).

Prerequisites

The current changes are blocked by the sunbeam-charms feature introduced in the PR, adding the pause/resume actions for the control plane services, like keystone, cinder, etc. The feature allows stopping the control-plane service's communication with the corresponding databases, so the restore can complete without concurrent writes.

QA

The new commands were validated end-to-end on a new Sunbeam Cluster (MAAS) on the personal SE-Cloud.

Demo

Backup

List Backups

Restore

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

Deezzir · 2026-07-20T06:39:21Z

+            params[self.component.backup_id_param] = latest
+
+        try:
+            tenacity.Retrying(


I found a race condition during a MySQL component restore. After the corresponding control-plane service is paused, there is a brief window during which MySQL may have dangling client connections, so the restore action will fail to close them. Having a retry here solves the issue, but I would like to come up with something more robust and reliable. Open to suggestions

issue: I find this aggressive retry policy a bit concerning.

Was this tested with applications unpaused?

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

gboutry

I think this PR needs another pass with a more holistic view of the backup and restore workflows.

There is a lot of indirection, which makes the operational sequence and its safety properties difficult to reason about. In particular, BackupComponent does not appear to simplify the implementation: component-specific behavior still leaks into generic steps, especially around action parameters, point-in-time restore, pause/resume coordination, and failure recovery.

The force handling illustrates the problem. It is propagated through several layers without a clear definition of what it is supposed to bypass. In some paths it has no effect, while in others it becomes an action parameter even when the target charm does not support that parameter. This makes --force both unreliable and capable of breaking otherwise valid operations.

Before adding more fixes, I think we should simplify the design around explicit MySQL and Vault workflows, define the guarantees for backup, restore, rollback, and force behavior, and then test those workflows against the actual charm action contracts and supported database topologies. The current mock-heavy tests pass while still encoding assumptions that are invalid with Juju.

gboutry · 2026-07-20T14:23:40Z

+            params[self.component.backup_id_param] = latest
+
+        try:
+            tenacity.Retrying(


issue: I find this aggressive retry policy a bit concerning.

Was this tested with applications unpaused?

gboutry · 2026-07-20T14:26:52Z

+    return sorted(successful, key=lambda b: b.backup_id)[-1].backup_id
+
+
+def _api_app_for_mysql(app_name: str) -> str:


The default MySQL topology is single MySQL app, and if we follow this Pull request, then, to which application is mysql tied to?

gboutry · 2026-07-20T14:35:22Z

+
+    return [
+        _PauseAppStep(jhelper, api_app, force=force, timeout=timeout, model=model),
+        _ScaleAppStep(


I don't see in the official docs where it's advised to scale mysql down to 1: https://canonical.com/data/mysql/docs/8.0/how-to/back-up-and-restore/restore-a-backup/

Anyway, won't that leave the PVCs on the host, and just provoke more juju hooks churn?

What are we trying to achieve here?

In the docs, it explicitly says to scale the replicas to 1 before the restore

Uh, you're right. And there's no conflict when it's scaled back up to 3 with the PVCs?

No, I guess it just reattaches them back

Yeah, but then we have "old" pvc on 2 units, and restored PVC 1 unit. I really would like some input on DPE, just to ensure it's a case they have considered.

I think the behaviour is the following: K8s preserves the old PVCs for the scaled-down units. During the restore, the PRIMARY cluster member rebuilds the database on its storage. After the scale-back, the SECONDARY rebuilds its copies on the old PVCs, replacing the stale data on the existing storage using the MySQL Clone Plugin from the fresh snapshot.

gboutry · 2026-07-20T14:36:34Z

+        super().__init__("Restore app", f"Restoring {target.app}")
+        self.jhelper = jhelper
+        self.target = target
+        self.restore_to_time = restore_to_time


I feel like the restore_to_time is always None, why include it then?

Any reason we're not looking for PITR recovery mechanism?

gboutry · 2026-07-20T14:38:39Z

+    return app_name
+
+
+def _current_scale(jhelper: JujuHelper, app: str, model: str) -> int:


If we can't figure out the scale of the application, this means juju is in a bad state, and we should not proceed with this operation. We should fail the operation.

gboutry · 2026-07-20T14:39:55Z

+                component=target.component,
+                backup=backup,
+            )
+        except Exception as e:


This exception catch is a bit too broad, can we narrow it down?

gboutry · 2026-07-20T14:41:28Z

+    return backups
+
+
+def _parse_backup(action_result: dict) -> BackupOutcome | None:


This allows for None, and we don't gate on that result later, is that expected?

I'd surmise either we've successfully created a backup, or it's a failure?

gboutry · 2026-07-20T14:43:06Z

+            if component is None:
+                continue
+            for app in apps:
+                target = component.resolve_backup_target(


Isn't it concerning that a discovered app for backup is gone once we try to resolve the target?

gboutry · 2026-07-20T14:44:49Z

+        self.scale = scale
+        self.timeout = timeout
+        self.model = model
+        self.force = force


Is this actually used?

gboutry · 2026-07-20T14:47:13Z

+    def run(self, context: StepContext) -> Result:
+        """Dispatch the backup action, recording the outcome on ``self.result``."""
+        target = self.target
+        params: dict[str, str | bool] = {"force": True} if self.force else {}


MySQL supports force: https://github.com/canonical/mysql-operators/blob/84489e49fbb12a27f12496ecc7beae7370219e4c/kubernetes/actions.yaml#L49
Vault k8s does not: https://github.com/canonical/vault-k8s-operator/blob/e8e103a558beeeb6007fa664d3418fc521ba24c4/k8s/charmcraft.yaml#L204

What I found is that Juju will ignore provided parameters if they are not defined in the action, so it is safe to pass

Is that documented behaviour? I'd rather not depend on an implementation detail that's not properly documented by the juju team if possible.

gboutry · 2026-07-21T07:33:10Z

+# Public component registry
+# ---------------------------------------------------------------------------
+BACKUP_COMPONENTS: list[BackupComponent] = [
+    BackupComponent(


I think I'd rather have subclasses VaultBackupComponent and so on, rather than generic object with helper functions. This would make the whole ordeal much better to follow.

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

Deezzir · 2026-07-22T05:37:58Z

@gboutry I've refactored the logic:

Explicit VaultBackupComponent and MySQLBackupComponent classes that encapsulate the per-component backup/restore logic.
Cleaned up stale step inputs (parameters)
Removed tenacity retry in the Restore step; to fight the race condition, the MySQL restore plan now scales all mysql-router apps to clean up the stale client connections and proceed to restore.
Cleaned up generic exceptions.
Added logic to handle multi/single MySQL cluster topology

gboutry self-requested a review July 9, 2026 11:45

hmlanigan requested a review from freyes July 10, 2026 19:06

Deezzir force-pushed the feat/backup-restore branch from 50b43d1 to a5e8558 Compare July 15, 2026 04:38

Deezzir added 5 commits July 15, 2026 00:40

feat: backup command (vault + mysql)

6308c59

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

tests(backup): add unit tests

41cc8f5

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

feat: restore command (vault + mysql)

361e9a6

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

tests(restore): add unit tests

fc7864f

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

refactor: single module for backup/restore

8c4cbe5

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

Deezzir force-pushed the feat/backup-restore branch 9 times, most recently from c03c37b to 253ec99 Compare July 17, 2026 01:04

fix: use OPENSTACK_MODEL

68a665b

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

Deezzir force-pushed the feat/backup-restore branch 4 times, most recently from 6d10439 to 63907a8 Compare July 17, 2026 06:39

refactor: backup/restore component agnostic wrappers

552f912

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

Deezzir force-pushed the feat/backup-restore branch 5 times, most recently from 691aaa8 to 141d379 Compare July 17, 2026 08:44

refactor: centralize unresolved target handling in list inventory flow

e33a28b

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

Deezzir force-pushed the feat/backup-restore branch from 141d379 to e33a28b Compare July 18, 2026 11:07

fix: restore safety

7338fbe

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

Deezzir force-pushed the feat/backup-restore branch from 0aab721 to 7338fbe Compare July 20, 2026 05:09

Deezzir mentioned this pull request Jul 20, 2026

feat: disaster-recovery experimental feature #890

Open

Deezzir commented Jul 20, 2026

View reviewed changes

fix: restore retry to allow resources to settle

434862a

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

Deezzir force-pushed the feat/backup-restore branch from c1983b3 to 434862a Compare July 20, 2026 07:05

Deezzir marked this pull request as ready for review July 20, 2026 07:06

gboutry requested changes Jul 20, 2026

View reviewed changes

gboutry reviewed Jul 21, 2026

View reviewed changes

refactor: explicit BackupComponent registry class

80c3324

Signed-off-by: Deezzir <yurii.kondrakov@canonical.com>

Deezzir requested a review from gboutry July 22, 2026 05:38

		return sorted(successful, key=lambda b: b.backup_id)[-1].backup_id


		def _api_app_for_mysql(app_name: str) -> str:

		return app_name


		def _current_scale(jhelper: JujuHelper, app: str, model: str) -> int:

		return backups


		def _parse_backup(action_result: dict) -> BackupOutcome \| None:

Uh oh!

Conversation

Deezzir commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Automation

Prerequisites

QA

Demo

Backup

List Backups

Restore

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gboutry left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gboutry Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Deezzir Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Deezzir commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Deezzir commented Jul 6, 2026 •

edited

Loading

gboutry Jul 21, 2026 •

edited

Loading

Deezzir Jul 22, 2026 •

edited

Loading