Skip to content

fix: run overlapping scheduled backups of the same volume in turn - #4879

Open
neurekadev wants to merge 7 commits into
getarcaneapp:mainfrom
neurekadev:fix/4878-wait-overlapping-volume-backups
Open

neurekadev wants to merge 7 commits into
getarcaneapp:mainfrom
neurekadev:fix/4878-wait-overlapping-volume-backups

Conversation

@neurekadev

@neurekadev neurekadev commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Important

PR 2 of 3. Merge in order: #4876 (merged) → #4879 → #4888.

Checklist

  • This PR is not opened from my fork’s main branch
  • All new user-facing strings are translated via Paraglide (m.*()) (no new strings)

What This PR Implements

Fixes: #4878

#4876 makes overlapping scheduled system backups wait for each other. This does the same for overlapping scheduled backups of the same volume:

  • A scheduled per-volume policy whose volume is already being backed up used to return RunStatusSkipped, which shows as success.
  • A scheduled system-managed run used to count that volume as skipped and move on.

Both now wait for the volume's admission. Manual volume backups still fail fast.

Changes Made

  • volume/children/backup: prepareBackup takes a resolve callback.
    • Scheduled triggers wait for the volume admission with Engine.AwaitRun (from fix: run overlapping scheduled system backups in turn instead of skipping #4876), then resolve the plan under it, so a policy edited during the wait is respected. This covers per-volume policies (CreateBackup) and system-managed runs (CreateSystemManagedBackup).
    • Other triggers resolve first (and StartBackup authorizes first) and keep the single try, so their validation, authorization, then conflict order is unchanged.
    • A scheduled per-volume run whose policy was disabled or deleted during the wait ends as canceled ("Backup policy disabled or deleted").
    • A scheduled system-managed run re-checks for a volume-level policy after admission. If one was enabled meanwhile, it returns ErrVolumeBackupPolicyOverridden, and executeSystemVolumeBackups counts the volume as skipped, the same as its pre-check.
  • volume/children/backup/children/policies: the scheduled skip branch and its now-unused AlreadyRunning dependency are removed.
  • TestVolumeBackupPolicy_ScheduledRunCreatesActivity used the skip as its way to end the run before a real backup. It now disables the policy after registration, which ends the run before a backup the same way, and still checks that the activity and its metadata are recorded.

Why waiting is deadlock-free here:

  • Manual starts park the volume admission only for tasks in the manual volume-backup workflow. Scheduled waits run in volume-backup-policy or system-volume-backup-policy.
  • Nothing that holds a volume admission waits for the system admission.
  • entityjobs never pre-acquires the volume admission for policy runs.
  • Manual runs keep failing fast for the same reason as in fix: run overlapping scheduled system backups in turn instead of skipping #4876.

Testing Done

  • just format go, just lint backend (0 issues), go build -tags exclude_frontend ./....
  • go test ./internal/volume/... ./internal/system/... ./internal/backup/... passes.
  • Not yet done: a live overlap in a dev environment.

AI Tool Used (if applicable)

Claude Code (Claude Opus 5.5) investigated this, wrote the code and drafted this description, at my direction.

Additional Context

Local review round 1 is in the comments below. It moved policy resolution after the wait, matching #4876.

🤖 Generated with Claude Code

Disclaimer Greptiles Reviews use AI, make sure to check over its work.

To better help train Greptile on our codebase, if the comment is useful and valid Like the comment, if its not helpful or invalid Dislike

To have Greptile Re-Review the changes, mention greptileai.

RetriggerView in GreptileConfidence Score: 4/5

The PR should not merge until the S3 check uses the policy that will actually run.

Fix All in CodexFindings

  1. P1 S3 checks use stale settings. ▶
Fix with agent prompt
### Issue 1
backend/internal/volume/children/backup/service.go:369-375
`runScheduledBackup` checks remote storage before calling `CreateBackup`, but this new wait can load a different S3 destination afterward. For example, a local-only policy can gain S3 while waiting. The new plan uses S3 without ever calling `CheckScheduledRemote` for it.

If that destination has retained backups but its repository is missing, the run bypasses the normal disable-and-warn path. `CreateSnapshot` uses `--init`, so it can recreate the repository and report success while older backups remain unavailable. Move the scheduled remote check and its disable/fallback decision under the volume admission, using the policy that will actually run.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Scheduled backups of the same volume now wait their turn, while manual backups still fail fast when the volume is busy.

  • Scheduled volume backups wait their turn and use the current policy.
  • System-managed volume backups wait and yield to per-volume policies.

No new findings. Tests were not run during this review. Access to both related repositories was denied.

Reviews (4) · Last reviewed commit: "Merge upstream main into fix/4878-wait-o..." · Reviewed by Greptile

neurekadev and others added 5 commits November 18, 2026 11:07
…ping

Scheduled system backups and system-managed volume backups share one
admission and both default to 03:00. The one that started second was
skipped, and the Activity Center reported the skip as a success.

Scheduled runs now wait for the admission, up to six hours, and fail
visibly if it never frees. Manual runs still return a conflict right away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ention

Scheduled runs now take the admission before reading their policy, so a
policy edited or disabled during the wait is respected. Retention and runs
resumed after a restart wait for the admission too. Before this, retention
failed and marked a successful backup as failed whenever the other
scheduled job grabbed the admission first, and a resumed backup could lose
the admission to a redelivered waiting job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…starts

A manual system-managed volume backup started while another backup runs
returns not-found or validation errors again instead of the conflict.
Scheduled runs already hold the admission before calling prepare.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Manual system and volume backup workflows run one task at a time, and a
new manual start parks the admission for its queued task. A resumed manual
run that waited for the admission would block on that task until the
six-hour timeout, so manual resumes fail fast again. Scheduled policy
workflows never park the admission, so their resumes still wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A scheduled backup of a volume that another backup holds was skipped and
reported as succeeded, and a scheduled system-managed run only counted
that volume as skipped. Scheduled backups now wait for the volume's
admission. Manual backups still fail fast, because a manual start parks
the admission for its queued task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…olume

Scheduled per-volume and system-managed backups now take the volume
admission before resolving their plan, so a policy edited, disabled, or
deleted during the wait is respected. A per-volume policy disabled or
deleted meanwhile ends its run as canceled, and a volume-level policy
enabled meanwhile takes the volume out of the system-managed run. Manual
starts keep their validation, authorization, then conflict order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@neurekadev

Copy link
Copy Markdown
Contributor Author

🔍 Code Review

PR #4879 · 630ae1453..7eb5083c6 · 3 files · ✅ No blocking issues remain

Critical: 0 High: 0 Medium: 1 Low: 0 Informational: 0

# Severity Finding Status
1 Medium Scheduled volume backup uses policy settings read before the wait
backend/internal/volume/children/backup/service.go:359 · correctness
✅ Fixed 5877c7fe7
1 · Scheduled volume backup uses policy settings read before the wait
  • Scenario: A manual backup holds volume V. The scheduled policy P for V loads P (StopContainers=true) and waits. Meanwhile the user disables or deletes P, turns off stop-containers, or retargets S3. After the wait, P's run backs up with the old settings anyway. In the system-managed case, a volume-level policy enabled during the wait doesn't stop the centralized backup of V.
  • Evidence: CreateBackup and CreateSystemManagedBackup called resolveBackupPlan before prepareBackup's AwaitRun, and executeBackup used that plan without reloading it. executeSystemVolumeBackups checked HasEnabledBackupPolicy before the per-volume wait. Reviewers split Medium/Low; the user chose to fix it.
  • Fix: prepareBackup takes a resolve callback. Scheduled runs wait first and then resolve under the admission, and a policy disabled or deleted after the wait ends the run as canceled. A scheduled system-managed run re-checks for a volume-level policy after admission and is counted as skipped, like the pre-check. Other triggers still resolve (and StartBackup still authorizes) before the single try. (5877c7fe7)
  • Fix check: Pass: the plan is read after AwaitRun, disabled or deleted policies end Canceled, PolicyOverridden is wired in production and skipped like the pre-check, manual ordering is unchanged, and the lease is released on every error path (Release is nil-safe).

🤖 Claude Opus 5.5 (High) · T3 Code (Claude Code) · Round 1 of 3

@kmendell

Copy link
Copy Markdown
Member

@neurekadev If you want to stack PR's use GitHub stacked pr's its makes it easier to see/review, I think there's a skill out there somewhere too for agents.

@neurekadev

Copy link
Copy Markdown
Contributor Author

Thanks @kmendell! I did look at GitHub's stacked PRs, but I don't think I can use them from a fork right now. All the branches in a stack have to be in the same repository, and cross-fork stacks aren't supported (About stacked pull requests). I don't have push access here, so my branches live on neurekadev/arcane, and an upstream PR can't use a fork branch as its base anyway (github/gh-stack#510). Fork support is planned (github/gh-stack#46), but there's no date yet.

That's why I've been doing it by hand. Each PR targets main, has a merge-order note at the top, and links an "only this PR's changes" compare so you can see just that layer's diff. I know the full diff on the upper PR still isn't great to review.

Would you prefer I do it differently? I see two options:

Happy to go with whatever's easiest for you.

getarcaneapp#4876 merged as 40e65a7. Resolved the conflict by keeping upstream's
removal of the admissionWait comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
neurekadev added a commit to neurekadev/arcane that referenced this pull request Nov 18, 2026
Brings in upstream main through getarcaneapp#4879. Resolved the en.json conflict by
keeping upstream's new keys, then this branch's keys appended after them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment on lines +369 to +375
if trigger == types.VolumeBackupTriggerScheduled {
var err error
if lease, err = s.deps.Engine.AwaitRun(ctx, backup.VolumeAdmissionScope, volumeName); err != nil {
return nil, backupPlan{}, nil, err
}
}
plan, err := resolve()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 S3 checks use stale settings. runScheduledBackup checks remote storage before calling CreateBackup, but this new wait can load a different S3 destination afterward. For example, a local-only policy can gain S3 while waiting. The new plan uses S3 without ever calling CheckScheduledRemote for it.

If that destination has retained backups but its repository is missing, the run bypasses the normal disable-and-warn path. CreateSnapshot uses --init, so it can recreate the repository and report success while older backups remain unavailable. Move the scheduled remote check and its disable/fallback decision under the volume admission, using the policy that will actually run.

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/internal/volume/children/backup/service.go
Line: 369-375

Comment:
**S3 checks use stale settings.** `runScheduledBackup` checks remote storage before calling `CreateBackup`, but this new wait can load a different S3 destination afterward. For example, a local-only policy can gain S3 while waiting. The new plan uses S3 without ever calling `CheckScheduledRemote` for it.

If that destination has retained backups but its repository is missing, the run bypasses the normal disable-and-warn path. `CreateSnapshot` uses `--init`, so it can recreate the repository and report success while older backups remain unavailable. Move the scheduled remote check and its disable/fallback decision under the volume admission, using the policy that will actually run.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code Fix in Cursor

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐞 Bug: Overlapping scheduled backups of the same volume are skipped instead of waiting

2 participants