Repository navigation
Conversation
mxsrc
marked this pull request as ready for review
November 13, 2026 16:58
mxsrc
marked this pull request as draft
November 16, 2026 10:45
Backup deletion needs mutual exclusion between a delete, a merge and a restore on the same volume's backup chain. Add DBController.claim_backup_chain_op: one FDB transaction that reads every backup of a volume's chain and refuses if any sibling is already MERGING/DELETING or is named by an active restore task's chain, else flips the target(s) to the new status -- the status field itself is then the exclusion every later caller checks. Restore contention is tracked via a new JobSchedule index (restore_chain_s3_id) rather than a manual counter: it derives directly from each active FN_BACKUP_RESTORE task's own chain_ids, so entries appear and disappear with the task's own writes, nothing to leak or decrement by hand. Restore task creation now goes through DBController.create_backup_restore_task_if_chain_clear, which joins the same sibling check into the transaction that creates the task, closing the gap between validating a chain restorable and the task existing. add_backup_restore_task gains a leading source_lvol_id parameter; callers and the two tests asserting on its positional call signature are updated. Not yet wired into delete_backup/trigger_merge -- the guard exists but nothing calls it yet.
…rewrite Fixes a real bug found while wiring up claim_backup_chain_op: siblings were keyed by the composite get_id() but every caller holds a bare uuid, so the target-exemption and lookup checks never matched. Siblings are now keyed by uuid, and a target already in its own claimed state no longer blocks re-claiming itself (resuming an interrupted delete), while a genuinely different busy sibling still does. trigger_merge now goes through claim_backup_chain_op instead of writing MERGING directly, and returns (ok, reason) instead of silently doing nothing, so delete_backup can surface a real refusal to its own caller. manifest.py: BackupManifestState (ACTIVE/DELETING) and a state field on BackupManifest, mirroring GroupBackupManifest. mark_deleting/mark_active are idempotent read-modify-writes; controller.py wraps them as mark_manifest_deleting/revert_manifest_deleting. import_backups refuses a non-ACTIVE manifest for an explicit target; import_from_bucket skips one instead, logging it, the same split import_group_backup/ import_group_backup_from_bucket already use for group manifests. delete_backup/delete_backups rewritten around the guard: claim the chain, mark the manifest(s) DELETING, queue FN_BACKUP_DELETE -- reverting every claim just taken if marking a manifest fails, so a partial failure never leaves FDB and the bucket disagreeing. A backup with a completed successor is routed to trigger_merge instead (deleting it and merging it into that successor are the same request). Resuming an already-DELETING backup re-queues rather than re-claims. delete_backups additionally refuses while any backup of the volume is still being created or merging, and removes FAILED backups (no manifest, nothing to claim) directly. tasks_runner_backup.py: FN_BACKUP_DELETE handler (_run_delete) and its finalize_resource revert branch, following the existing FN_BACKUP_MERGE shape. Single-cycle: bdev_lvol_s3_delete has no paired stat RPC to poll, unlike backup/restore/merge. New unit tests for the manifest state functions (fake in-memory bucket, no DB). Integration test file for delete_backup/delete_backups/trigger_merge rewritten for the new async, claim-based behavior, including the new restore-vs-delete contention path -- NOT verified against real FDB in this sandbox; the testcontainers-provisioned FDB and a manually-configured single-process fallback both failed to come up usably here (environment limitation, not a code issue surfaced by either attempt). Lint, mypy, and the full unit suite (1778 tests) are green.
DELETE /clusters/{cluster_id}/backups/{backup_id} (route name
clusters:backups:delete-one) calls backup_controller.delete_backup, mapping
PreconditionError to 409 -- the same convention delete_group_backup already
uses. sbctl backup delete-backup <backup_id> is the CLI equivalent
(backup__delete_backup in clibase.py, mirroring group_backup__delete).
Both per-volume bulk-delete endpoints (the current one under
storage-pools/.../volumes/{id}/backups, and the deprecated
/clusters/{id}/backups/{volume_id}) now return 202 instead of 204: deletion
is asynchronous since delete_backups only claims and queues a task, it does
not finish before returning. Their error status moves from 400 to 409 for
the same reason delete_group_backup uses 409 over HTTPException's usual 400
-- these are precondition conflicts, not malformed requests.
Running the full unit suite after this surfaced two real gaps the new task
type exposed: TaskDTO.function_name's Literal type didn't list
"s3_backup_delete" (every /tasks/ list request with one in it 500'd), and
test_backup_family_is_owned's exact-set assertion over
tasks_runner_backup.SPEC.function_names didn't expect it either. Both fixed.
Full unit suite (6965 tests) and lint/types green.
The backup analog of check_indices: controller.check_backups walks every COMPLETED/DELETING backup in a cluster (or every cluster) and its manifest in both directions. A record whose manifest is missing or disagrees on state/s3_id/prev_backup_id is reported, and --repair rewrites it from the record (today's source of truth). A manifest naming no known record is reported but never auto-repaired -- it may be an unimported disaster-recovery backup, which looks identical to drift from here. MERGED/MERGING backups are explicitly excluded: the merge path doesn't mark manifest state at all, so a leftover manifest there is a known gap in that path, not something this check can resolve. manifest._read_if_present is now public (read_if_present) since check_backups needs the same "read or None" shape mark_deleting/mark_active already used internally. sbctl cluster check-backups [--cluster-id] [--repair] wires it into the CLI, mirroring check-indices. New integration test file (real FDB, fake in-memory bucket) covering the clean case, missing/disagreeing/orphaned manifests, the MERGED exclusion, and cluster scoping -- reviewed but, like the rest of this branch's integration coverage, not run against real FDB in this sandbox.
CI (Integration tests py3.11/py3.14t) is where the sandbox's broken FDB testcontainer never let this get verified. It found six real problems: Real regressions in the implementation: - delete_backup had no path for a FAILED backup -- it fell through to "not completed, refuse", when delete_backups (the bulk path) already handled this correctly (nothing was ever uploaded, so there is no manifest or S3 data to claim; remove it directly). - group_backup._retire_members treated calling delete_backup on a successor-less member as immediately retiring it, which was true when delete_backup was synchronous and is not now: a member only claimed (DELETING) is not yet gone, and advance_group_backup_delete must keep returning PENDING until it actually is. Fixed by checking the member's real post-call state rather than assuming it. - group_backup._members_touched (abandon_group_backup_delete's "was any member touched" check) didn't count DELETING as touched, so a timed-out delete whose only progress was claiming a member for deletion would have been reported as untouched and the group backup restored to COMPLETED over a member whose S3 objects might still be getting destroyed. Test bugs in this branch's own new test file (test_check_backups.py): - its s3_id and the Backup fixture's s3_id were derived from different strings (the manifest's from the uuid, the record's from a short name), so they never matched -- every manifest looked like an orphan. - the Backup fixture had no snapshot_id, which build_manifest (the --repair path) requires; repairing crashed instead of rewriting. Both are now also handled defensively in check_backups itself: _rewrite_manifest_from_record no longer propagates a build/write failure out of the whole check -- it reports that one backup as unrepaired and keeps going. Legacy stubbed-DB tests (test_legacy.py's own documented pattern, migrated per tests/AGENTS.md -- "convert it to real FDB, don't add a matching patch for the new module"): TestEvaluatePolicy, TestTriggerMerge and TestDeleteBackups.test_success mocked away write_to_db or db_controller entirely, which worked while trigger_merge/delete_backups only mutated in-memory objects and called a no-op write. Now that both read the chain's real siblings through DBController.claim_backup_chain_op, the backups they exercise have to actually be in FDB. Converted to write real records and assert against what the database holds afterward, following the pattern already used in this branch's own test_delete_backup.py. Also updated two test_group_backup_delete.py assertions that depended on delete_backup being synchronous (a leaf member is now DELETING after one retirement pass, not already gone -- its own task removes it, simulated here the same way the existing test already simulated a merge finishing). Lint, mypy, and the full unit suite (6965 tests) are green locally; pushing for CI to re-verify the integration tier, which is what actually caught all of this.
Its backup used the short placeholder "b-1", which every other test in this file gets away with since nothing used to parse it -- but delete_backups now reaches manifest.mark_deleting, which does UUID(backup.uuid) (via manifest_key / build_manifest's backup_id). "b-1" is not shaped like one. CI's last run (py3.11 and py3.14t integration, both green otherwise) was the only thing still failing.
bdev_lvol_s3_delete does not exist on the data plane and never will -- confirmed live, it always answers Method not found. A tip or whole-chain delete needs no data-plane involvement at all: the control plane can remove a backup's S3 objects itself, the same way it already removes a manifest. Where the data plane genuinely has to be involved -- a backup something still chains against -- that already goes through bdev_lvol_s3_merge instead, which stays unchanged. Adds backup_manifest.delete_data_objects (list+batch-delete under the backup's s3_id prefix) and backup_controller.delete_backup_data, and switches tasks_runner_backup._run_delete to call it instead of the RPC. Removes the now-dead RPCClient.bdev_lvol_s3_delete method and the test asserting it exists, and fixes a db_controller.py comment that referenced the same nonexistent RPC.
…bjects Found live, against a real MinIO backend: delete_objects (multi-object delete) requires a legacy Content-MD5 header that botocore 1.36+ no longer computes by default in favor of its newer flexible checksums -- MinIO (RELEASE.2024-01-16) doesn't understand those and answers MissingContentMD5, and setting request_checksum_calculation back to when_required doesn't restore the header for this operation either. Single-object delete has no such requirement, so delete_data_objects now calls delete_object once per key instead of batching.
delete_backup, group backup member retirement, and orphan retirement each picked an arbitrary successor via next()/dict construction when a backup had multiple dependents, silently dropping any sibling fork instead of failing loudly. Use single_or_none so a real fork raises instead of being masked.
The own-targets exemption from the MERGING/DELETING block only checked that a backup's id was in target_ids, not that its current status already matched new_status. A backup claimed DELETING by one operation could therefore be silently re-claimed as MERGING by a different, concurrent one (e.g. trigger_merge racing delete_backup's earlier, non-transactional status read), overwriting the in-progress claim. Now a target only exempts itself when it is already in new_status; a target held in the other status still blocks, like any other sibling.
single_or_none was applied only to the COMPLETED-filtered subset of dependents, so a fork with one COMPLETED sibling and one PENDING/ IN_PROGRESS sibling went undetected: the filter hid the non-completed one from the uniqueness check, and the completed one was merged into immediately. Check uniqueness over every dependent first, then the single survivor's status, so any fork is caught regardless of which sibling's status it involves.
Covers the three fixes that lacked a dedicated test: - delete_backup raising on a predecessor with multiple dependents of any status (controller.py), not just multiple COMPLETED ones. - _retire_members raising on a forked successor lookup (group_backup.py). - retire_orphaned_member_backups raising on a forked by_prev index instead of silently keeping whichever backup the construction saw last. claim_backup_chain_op's fix already had a regression test added earlier (test_a_backup_claimed_for_delete_refuses_a_concurrent_merge_claim).
delete_backup, _retire_members and retire_orphaned_member_backups each hand-rolled their own prev_backup_id lookup to find a backup's successor, with inconsistent fork detection (single_or_none's ValueError in two places, a manual dict in the third). Add SuccessorIndex to chain.py -- the forward-lookup counterpart to BackupChain's backward walk -- and migrate all three sites to it, so a chain fork is now always reported as PreconditionError from one place. Also points simplyblock_core/AGENTS.md at chain.py as the canonical home for backup-chain invariants, since the original backup-deletion implementation missed BackupChain entirely due to the module's docstring being framed only around restore.
mxsrc
force-pushed
the
backup-deletion
branch
from
November 20, 2026 16:20
f04c016 to
665bd31
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements backup deletion in the control plane (previously unimplemented:
delete_backup/delete_backupsremoved FDB records unconditionally,regardless of whether the S3 data they describe ever went away).
DBController.claim_backup_chain_op): one FDBtransaction reads every backup of a volume's chain and refuses a
delete/merge if any sibling is already
MERGING/DELETING, or if thechain is being read by an active restore (new
JobSchedule.restore_chain_s3_idindex, derived from each restore task's own
chain_ids— no manualcounter to leak).
BackupManifestState.ACTIVE/DELETINGonBackupManifest): marked before any S3 object is touched, mirroringGroupBackupManifest's existing state machine.import_backups/import_from_bucketrefuse/skip a non-ACTIVEmanifest.delete_backup/delete_backupsrewritten: claim → mark manifest →queue
FN_BACKUP_DELETE, all-or-nothing revert if marking a manifestfails. A backup with a completed successor is routed to
trigger_mergeinstead (deleting it and merging it into that successor are the same
request). Resuming an already-
DELETINGbackup re-queues rather thanre-claims.
trigger_mergenow goes through the same guard (previously wroteMERGINGdirectly) and returns(ok, reason)instead of silently no-op'ing.FN_BACKUP_DELETEtask (_run_delete+finalize_resourcerevertbranch) in
tasks_runner_backup.py, single-cycle sincebdev_lvol_s3_deletehas no paired stat RPC to poll — it isn't implementedon the data plane yet, so real object deletion stays a no-op RPC call
until that lands; the metadata lifecycle is complete and will start
working the moment it does.
DELETE /clusters/{id}/backups/{backup_id}+sbctl backup delete-backup; both bulk-delete endpoints now return 202(deletion is async) instead of 204.
Not included
sbctl cluster check-backups(FDB-vs-manifest consistency checker) — notstarted.
follow-up, not done here.
Test plan
tox run-parallel -e lint,types— cleantox run -e unit— 6965 passed, 29 skippedtox run -e integration— run in CI