Repository navigation
fix: run overlapping scheduled backups of the same volume in turn - #4879
neurekadev wants to merge 7 commits into
Conversation
…ping Scheduled system backups and system-managed volume backups share one admission and both default to 03:00. The one that started second was skipped, and the Activity Center reported the skip as a success. Scheduled runs now wait for the admission, up to six hours, and fail visibly if it never frees. Manual runs still return a conflict right away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ention Scheduled runs now take the admission before reading their policy, so a policy edited or disabled during the wait is respected. Retention and runs resumed after a restart wait for the admission too. Before this, retention failed and marked a successful backup as failed whenever the other scheduled job grabbed the admission first, and a resumed backup could lose the admission to a redelivered waiting job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…starts A manual system-managed volume backup started while another backup runs returns not-found or validation errors again instead of the conflict. Scheduled runs already hold the admission before calling prepare. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Manual system and volume backup workflows run one task at a time, and a new manual start parks the admission for its queued task. A resumed manual run that waited for the admission would block on that task until the six-hour timeout, so manual resumes fail fast again. Scheduled policy workflows never park the admission, so their resumes still wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A scheduled backup of a volume that another backup holds was skipped and reported as succeeded, and a scheduled system-managed run only counted that volume as skipped. Scheduled backups now wait for the volume's admission. Manual backups still fail fast, because a manual start parks the admission for its queued task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…olume Scheduled per-volume and system-managed backups now take the volume admission before resolving their plan, so a policy edited, disabled, or deleted during the wait is respected. A per-volume policy disabled or deleted meanwhile ends its run as canceled, and a volume-level policy enabled meanwhile takes the volume out of the system-managed run. Manual starts keep their validation, authorization, then conflict order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🔍 Code ReviewPR #4879 ·
1 · Scheduled volume backup uses policy settings read before the wait
🤖 Claude Opus 5.5 (High) · T3 Code (Claude Code) · Round 1 of 3 |
|
@neurekadev If you want to stack PR's use GitHub stacked pr's its makes it easier to see/review, I think there's a skill out there somewhere too for agents. |
|
Thanks @kmendell! I did look at GitHub's stacked PRs, but I don't think I can use them from a fork right now. All the branches in a stack have to be in the same repository, and cross-fork stacks aren't supported (About stacked pull requests). I don't have push access here, so my branches live on That's why I've been doing it by hand. Each PR targets Would you prefer I do it differently? I see two options:
Happy to go with whatever's easiest for you. |
getarcaneapp#4876 merged as 40e65a7. Resolved the conflict by keeping upstream's removal of the admissionWait comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in upstream main through getarcaneapp#4879. Resolved the en.json conflict by keeping upstream's new keys, then this branch's keys appended after them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| if trigger == types.VolumeBackupTriggerScheduled { | ||
| var err error | ||
| if lease, err = s.deps.Engine.AwaitRun(ctx, backup.VolumeAdmissionScope, volumeName); err != nil { | ||
| return nil, backupPlan{}, nil, err | ||
| } | ||
| } | ||
| plan, err := resolve() |
There was a problem hiding this comment.
S3 checks use stale settings.
runScheduledBackup checks remote storage before calling CreateBackup, but this new wait can load a different S3 destination afterward. For example, a local-only policy can gain S3 while waiting. The new plan uses S3 without ever calling CheckScheduledRemote for it.
If that destination has retained backups but its repository is missing, the run bypasses the normal disable-and-warn path. CreateSnapshot uses --init, so it can recreate the repository and report success while older backups remain unavailable. Move the scheduled remote check and its disable/fallback decision under the volume admission, using the policy that will actually run.
Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/internal/volume/children/backup/service.go
Line: 369-375
Comment:
**S3 checks use stale settings.** `runScheduledBackup` checks remote storage before calling `CreateBackup`, but this new wait can load a different S3 destination afterward. For example, a local-only policy can gain S3 while waiting. The new plan uses S3 without ever calling `CheckScheduledRemote` for it.
If that destination has retained backups but its repository is missing, the run bypasses the normal disable-and-warn path. `CreateSnapshot` uses `--init`, so it can recreate the repository and report success while older backups remain unavailable. Move the scheduled remote check and its disable/fallback decision under the volume admission, using the policy that will actually run.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Important
PR 2 of 3. Merge in order: #4876 (merged) → #4879 → #4888.
Checklist
mainbranchm.*()) (no new strings)What This PR Implements
Fixes: #4878
#4876 makes overlapping scheduled system backups wait for each other. This does the same for overlapping scheduled backups of the same volume:
RunStatusSkipped, which shows as success.skippedand move on.Both now wait for the volume's admission. Manual volume backups still fail fast.
Changes Made
volume/children/backup:prepareBackuptakes a resolve callback.Engine.AwaitRun(from fix: run overlapping scheduled system backups in turn instead of skipping #4876), then resolve the plan under it, so a policy edited during the wait is respected. This covers per-volume policies (CreateBackup) and system-managed runs (CreateSystemManagedBackup).StartBackupauthorizes first) and keep the single try, so their validation, authorization, then conflict order is unchanged.ErrVolumeBackupPolicyOverridden, andexecuteSystemVolumeBackupscounts the volume as skipped, the same as its pre-check.volume/children/backup/children/policies: the scheduled skip branch and its now-unusedAlreadyRunningdependency are removed.TestVolumeBackupPolicy_ScheduledRunCreatesActivityused the skip as its way to end the run before a real backup. It now disables the policy after registration, which ends the run before a backup the same way, and still checks that the activity and its metadata are recorded.Why waiting is deadlock-free here:
volume-backupworkflow. Scheduled waits run involume-backup-policyorsystem-volume-backup-policy.entityjobsnever pre-acquires the volume admission for policy runs.Testing Done
just format go,just lint backend(0 issues),go build -tags exclude_frontend ./....go test ./internal/volume/... ./internal/system/... ./internal/backup/...passes.AI Tool Used (if applicable)
Claude Code (Claude Opus 5.5) investigated this, wrote the code and drafted this description, at my direction.
Additional Context
Local review round 1 is in the comments below. It moved policy resolution after the wait, matching #4876.
🤖 Generated with Claude Code
Disclaimer Greptiles Reviews use AI, make sure to check over its work.
To better help train Greptile on our codebase, if the comment is useful and valid Like the comment, if its not helpful or invalid Dislike
To have Greptile Re-Review the changes, mention
greptileai.The PR should not merge until the S3 check uses the policy that will actually run.
Fix with agent prompt
Summary
Scheduled backups of the same volume now wait their turn, while manual backups still fail fast when the volume is busy.
No new findings. Tests were not run during this review. Access to both related repositories was denied.
Reviews (4) · Last reviewed commit: "Merge upstream main into fix/4878-wait-o..." · Reviewed by Greptile