Skip to content

Implement backup deletion - #1645

Draft
mxsrc wants to merge 13 commits into
mainfrom
backup-deletion
Draft

mxsrc wants to merge 13 commits into
mainfrom
backup-deletion

Conversation

@mxsrc

@mxsrc mxsrc commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Implements backup deletion in the control plane (previously unimplemented:
delete_backup/delete_backups removed FDB records unconditionally,
regardless of whether the S3 data they describe ever went away).

  • Chain-wide guard (DBController.claim_backup_chain_op): one FDB
    transaction reads every backup of a volume's chain and refuses a
    delete/merge if any sibling is already MERGING/DELETING, or if the
    chain is being read by an active restore (new JobSchedule.restore_chain_s3_id
    index, derived from each restore task's own chain_ids — no manual
    counter to leak).
  • Manifest state (BackupManifestState.ACTIVE/DELETING on
    BackupManifest): marked before any S3 object is touched, mirroring
    GroupBackupManifest's existing state machine. import_backups/
    import_from_bucket refuse/skip a non-ACTIVE manifest.
  • delete_backup/delete_backups rewritten: claim → mark manifest →
    queue FN_BACKUP_DELETE, all-or-nothing revert if marking a manifest
    fails. A backup with a completed successor is routed to trigger_merge
    instead (deleting it and merging it into that successor are the same
    request). Resuming an already-DELETING backup re-queues rather than
    re-claims.
  • trigger_merge now goes through the same guard (previously wrote
    MERGING directly) and returns (ok, reason) instead of silently no-op'ing.
  • FN_BACKUP_DELETE task (_run_delete + finalize_resource revert
    branch) in tasks_runner_backup.py, single-cycle since
    bdev_lvol_s3_delete has no paired stat RPC to poll — it isn't implemented
    on the data plane yet, so real object deletion stays a no-op RPC call
    until that lands; the metadata lifecycle is complete and will start
    working the moment it does.
  • API/CLI: DELETE /clusters/{id}/backups/{backup_id} +
    sbctl backup delete-backup; both bulk-delete endpoints now return 202
    (deletion is async) instead of 204.

Not included

  • sbctl cluster check-backups (FDB-vs-manifest consistency checker) — not
    started.
  • Conditional-write (ETag) hardening on manifest writes — flagged as a cheap
    follow-up, not done here.

Test plan

  • tox run-parallel -e lint,types — clean
  • tox run -e unit — 6965 passed, 29 skipped
  • tox run -e integration — run in CI

@github-actions github-actions Bot added the target/main Targets the development branch label Nov 13, 2026
@mxsrc
mxsrc marked this pull request as ready for review November 13, 2026 16:58
@mxsrc
mxsrc marked this pull request as draft November 16, 2026 10:45
@mxsrc mxsrc added this to the 26.4 Beta 7 milestone Nov 17, 2026
mxsrc added 13 commits November 20, 2026 17:19
Backup deletion needs mutual exclusion between a delete, a merge and a
restore on the same volume's backup chain. Add
DBController.claim_backup_chain_op: one FDB transaction that reads
every backup of a volume's chain and refuses if any sibling is already
MERGING/DELETING or is named by an active restore task's chain, else
flips the target(s) to the new status -- the status field itself is
then the exclusion every later caller checks.

Restore contention is tracked via a new JobSchedule index
(restore_chain_s3_id) rather than a manual counter: it derives directly
from each active FN_BACKUP_RESTORE task's own chain_ids, so entries
appear and disappear with the task's own writes, nothing to leak or
decrement by hand. Restore task creation now goes through
DBController.create_backup_restore_task_if_chain_clear, which joins
the same sibling check into the transaction that creates the task,
closing the gap between validating a chain restorable and the task
existing.

add_backup_restore_task gains a leading source_lvol_id parameter;
callers and the two tests asserting on its positional call signature
are updated.

Not yet wired into delete_backup/trigger_merge -- the guard exists but
nothing calls it yet.
…rewrite

Fixes a real bug found while wiring up claim_backup_chain_op: siblings were
keyed by the composite get_id() but every caller holds a bare uuid, so the
target-exemption and lookup checks never matched. Siblings are now keyed by
uuid, and a target already in its own claimed state no longer blocks
re-claiming itself (resuming an interrupted delete), while a genuinely
different busy sibling still does.

trigger_merge now goes through claim_backup_chain_op instead of writing
MERGING directly, and returns (ok, reason) instead of silently doing
nothing, so delete_backup can surface a real refusal to its own caller.

manifest.py: BackupManifestState (ACTIVE/DELETING) and a state field on
BackupManifest, mirroring GroupBackupManifest. mark_deleting/mark_active are
idempotent read-modify-writes; controller.py wraps them as
mark_manifest_deleting/revert_manifest_deleting. import_backups refuses a
non-ACTIVE manifest for an explicit target; import_from_bucket skips one
instead, logging it, the same split import_group_backup/
import_group_backup_from_bucket already use for group manifests.

delete_backup/delete_backups rewritten around the guard: claim the chain,
mark the manifest(s) DELETING, queue FN_BACKUP_DELETE -- reverting every
claim just taken if marking a manifest fails, so a partial failure never
leaves FDB and the bucket disagreeing. A backup with a completed successor
is routed to trigger_merge instead (deleting it and merging it into that
successor are the same request). Resuming an already-DELETING backup
re-queues rather than re-claims. delete_backups additionally refuses while
any backup of the volume is still being created or merging, and removes
FAILED backups (no manifest, nothing to claim) directly.

tasks_runner_backup.py: FN_BACKUP_DELETE handler (_run_delete) and its
finalize_resource revert branch, following the existing FN_BACKUP_MERGE
shape. Single-cycle: bdev_lvol_s3_delete has no paired stat RPC to poll,
unlike backup/restore/merge.

New unit tests for the manifest state functions (fake in-memory bucket, no
DB). Integration test file for delete_backup/delete_backups/trigger_merge
rewritten for the new async, claim-based behavior, including the new
restore-vs-delete contention path -- NOT verified against real FDB in this
sandbox; the testcontainers-provisioned FDB and a manually-configured
single-process fallback both failed to come up usably here (environment
limitation, not a code issue surfaced by either attempt). Lint, mypy, and
the full unit suite (1778 tests) are green.
DELETE /clusters/{cluster_id}/backups/{backup_id} (route name
clusters:backups:delete-one) calls backup_controller.delete_backup, mapping
PreconditionError to 409 -- the same convention delete_group_backup already
uses. sbctl backup delete-backup <backup_id> is the CLI equivalent
(backup__delete_backup in clibase.py, mirroring group_backup__delete).

Both per-volume bulk-delete endpoints (the current one under
storage-pools/.../volumes/{id}/backups, and the deprecated
/clusters/{id}/backups/{volume_id}) now return 202 instead of 204: deletion
is asynchronous since delete_backups only claims and queues a task, it does
not finish before returning. Their error status moves from 400 to 409 for
the same reason delete_group_backup uses 409 over HTTPException's usual 400
-- these are precondition conflicts, not malformed requests.

Running the full unit suite after this surfaced two real gaps the new task
type exposed: TaskDTO.function_name's Literal type didn't list
"s3_backup_delete" (every /tasks/ list request with one in it 500'd), and
test_backup_family_is_owned's exact-set assertion over
tasks_runner_backup.SPEC.function_names didn't expect it either. Both fixed.

Full unit suite (6965 tests) and lint/types green.
The backup analog of check_indices: controller.check_backups walks every
COMPLETED/DELETING backup in a cluster (or every cluster) and its manifest
in both directions. A record whose manifest is missing or disagrees on
state/s3_id/prev_backup_id is reported, and --repair rewrites it from the
record (today's source of truth). A manifest naming no known record is
reported but never auto-repaired -- it may be an unimported
disaster-recovery backup, which looks identical to drift from here.

MERGED/MERGING backups are explicitly excluded: the merge path doesn't mark
manifest state at all, so a leftover manifest there is a known gap in that
path, not something this check can resolve.

manifest._read_if_present is now public (read_if_present) since
check_backups needs the same "read or None" shape mark_deleting/mark_active
already used internally.

sbctl cluster check-backups [--cluster-id] [--repair] wires it into the CLI,
mirroring check-indices. New integration test file (real FDB, fake
in-memory bucket) covering the clean case, missing/disagreeing/orphaned
manifests, the MERGED exclusion, and cluster scoping -- reviewed but, like
the rest of this branch's integration coverage, not run against real FDB in
this sandbox.
CI (Integration tests py3.11/py3.14t) is where the sandbox's broken FDB
testcontainer never let this get verified. It found six real problems:

Real regressions in the implementation:
- delete_backup had no path for a FAILED backup -- it fell through to
  "not completed, refuse", when delete_backups (the bulk path) already
  handled this correctly (nothing was ever uploaded, so there is no
  manifest or S3 data to claim; remove it directly).
- group_backup._retire_members treated calling delete_backup on a
  successor-less member as immediately retiring it, which was true when
  delete_backup was synchronous and is not now: a member only claimed
  (DELETING) is not yet gone, and advance_group_backup_delete must keep
  returning PENDING until it actually is. Fixed by checking the member's
  real post-call state rather than assuming it.
- group_backup._members_touched (abandon_group_backup_delete's "was any
  member touched" check) didn't count DELETING as touched, so a timed-out
  delete whose only progress was claiming a member for deletion would have
  been reported as untouched and the group backup restored to COMPLETED
  over a member whose S3 objects might still be getting destroyed.

Test bugs in this branch's own new test file (test_check_backups.py):
- its s3_id and the Backup fixture's s3_id were derived from different
  strings (the manifest's from the uuid, the record's from a short name),
  so they never matched -- every manifest looked like an orphan.
- the Backup fixture had no snapshot_id, which build_manifest (the
  --repair path) requires; repairing crashed instead of rewriting.
Both are now also handled defensively in check_backups itself:
_rewrite_manifest_from_record no longer propagates a build/write failure
out of the whole check -- it reports that one backup as unrepaired and
keeps going.

Legacy stubbed-DB tests (test_legacy.py's own documented pattern, migrated
per tests/AGENTS.md -- "convert it to real FDB, don't add a matching patch
for the new module"): TestEvaluatePolicy, TestTriggerMerge and
TestDeleteBackups.test_success mocked away write_to_db or db_controller
entirely, which worked while trigger_merge/delete_backups only mutated
in-memory objects and called a no-op write. Now that both read the chain's
real siblings through DBController.claim_backup_chain_op, the backups they
exercise have to actually be in FDB. Converted to write real records and
assert against what the database holds afterward, following the pattern
already used in this branch's own test_delete_backup.py.

Also updated two test_group_backup_delete.py assertions that depended on
delete_backup being synchronous (a leaf member is now DELETING after one
retirement pass, not already gone -- its own task removes it, simulated
here the same way the existing test already simulated a merge finishing).

Lint, mypy, and the full unit suite (6965 tests) are green locally;
pushing for CI to re-verify the integration tier, which is what actually
caught all of this.
Its backup used the short placeholder "b-1", which every other test in
this file gets away with since nothing used to parse it -- but
delete_backups now reaches manifest.mark_deleting, which does
UUID(backup.uuid) (via manifest_key / build_manifest's backup_id). "b-1"
is not shaped like one. CI's last run (py3.11 and py3.14t integration,
both green otherwise) was the only thing still failing.
bdev_lvol_s3_delete does not exist on the data plane and never will --
confirmed live, it always answers Method not found. A tip or whole-chain
delete needs no data-plane involvement at all: the control plane can
remove a backup's S3 objects itself, the same way it already removes a
manifest. Where the data plane genuinely has to be involved -- a backup
something still chains against -- that already goes through
bdev_lvol_s3_merge instead, which stays unchanged.

Adds backup_manifest.delete_data_objects (list+batch-delete under the
backup's s3_id prefix) and backup_controller.delete_backup_data, and
switches tasks_runner_backup._run_delete to call it instead of the RPC.
Removes the now-dead RPCClient.bdev_lvol_s3_delete method and the test
asserting it exists, and fixes a db_controller.py comment that referenced
the same nonexistent RPC.
…bjects

Found live, against a real MinIO backend: delete_objects (multi-object
delete) requires a legacy Content-MD5 header that botocore 1.36+ no
longer computes by default in favor of its newer flexible checksums --
MinIO (RELEASE.2024-01-16) doesn't understand those and answers
MissingContentMD5, and setting request_checksum_calculation back to
when_required doesn't restore the header for this operation either.
Single-object delete has no such requirement, so delete_data_objects now
calls delete_object once per key instead of batching.
delete_backup, group backup member retirement, and orphan retirement each
picked an arbitrary successor via next()/dict construction when a backup
had multiple dependents, silently dropping any sibling fork instead of
failing loudly. Use single_or_none so a real fork raises instead of being
masked.
The own-targets exemption from the MERGING/DELETING block only checked
that a backup's id was in target_ids, not that its current status already
matched new_status. A backup claimed DELETING by one operation could
therefore be silently re-claimed as MERGING by a different, concurrent
one (e.g. trigger_merge racing delete_backup's earlier, non-transactional
status read), overwriting the in-progress claim.

Now a target only exempts itself when it is already in new_status; a
target held in the other status still blocks, like any other sibling.
single_or_none was applied only to the COMPLETED-filtered subset of
dependents, so a fork with one COMPLETED sibling and one PENDING/
IN_PROGRESS sibling went undetected: the filter hid the non-completed one
from the uniqueness check, and the completed one was merged into
immediately. Check uniqueness over every dependent first, then the
single survivor's status, so any fork is caught regardless of which
sibling's status it involves.
Covers the three fixes that lacked a dedicated test:
- delete_backup raising on a predecessor with multiple dependents of any
  status (controller.py), not just multiple COMPLETED ones.
- _retire_members raising on a forked successor lookup (group_backup.py).
- retire_orphaned_member_backups raising on a forked by_prev index instead
  of silently keeping whichever backup the construction saw last.

claim_backup_chain_op's fix already had a regression test added earlier
(test_a_backup_claimed_for_delete_refuses_a_concurrent_merge_claim).
delete_backup, _retire_members and retire_orphaned_member_backups each
hand-rolled their own prev_backup_id lookup to find a backup's successor,
with inconsistent fork detection (single_or_none's ValueError in two
places, a manual dict in the third). Add SuccessorIndex to chain.py --
the forward-lookup counterpart to BackupChain's backward walk -- and
migrate all three sites to it, so a chain fork is now always reported as
PreconditionError from one place.

Also points simplyblock_core/AGENTS.md at chain.py as the canonical home
for backup-chain invariants, since the original backup-deletion
implementation missed BackupChain entirely due to the module's docstring
being framed only around restore.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

target/main Targets the development branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant