Skip to content

fix: run overlapping scheduled system backups in turn instead of skipping - #4876

Merged
kmendell merged 5 commits into
getarcaneapp:mainfrom
neurekadev:fix/4874-wait-overlapping-scheduled-backups
Nov 18, 2026
Merged

kmendell merged 5 commits into
getarcaneapp:mainfrom
neurekadev:fix/4874-wait-overlapping-scheduled-backups

Conversation

@neurekadev

@neurekadev neurekadev commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Important

PR 1 of 2. Merge in order: #4876 → #4879.

Checklist

  • This PR is not opened from my fork’s main branch
  • All new user-facing strings are translated via Paraglide (m.*()) (no new strings)

What This PR Implements

Fixes: #4874

The scheduled system backup and the system-managed volume backups share the system backup admission and both default to 0 3 * * *. Whichever started second only tried the admission, returned RunStatusSkipped, and was never retried. jobActivity shows Skipped as a success, so a backup that never ran looked fine.

Scheduled runs now wait for the admission, so overlapping schedules run one after the other. If the admission doesn't free within six hours, the run fails with a timeout error, which shows as a failed activity instead of a skip. Manual runs still return the 409 conflict immediately, which the user sees.

Changes Made

  • internal/backup: new Engine.AwaitRun. It tries the admission, logs when it has to wait, then blocks on Admission.Acquire (the same wait GitOps syncs use) up to admissionWait (6h), and returns ErrTimeout when that runs out.
  • system/children/backup:
    • runScheduledBackup takes the admission first, then reads and checks the policy under it, so a policy edited or disabled during the wait is respected. CreateBackup (its only caller) is folded in.
    • The lease is released before the remote-environment backups and retention.
    • applyRetention waits for the admission instead of a single try. Otherwise it failed, and marked the successful backup as failed, whenever the other scheduled job grabbed the admission first.
    • reconcileBackup takes its acquire step from the caller. A scheduled resume after a restart waits, and a manual resume still fails fast.
  • system/children/backup/children/volumes:
    • runScheduledSystemVolumeBackup takes the admission first and passes it to prepareSystemVolumeBackups. Start keeps the immediate alreadyRunning conflict, and validation and not-found errors still come first.
    • backUpSelection still takes a parked manual claim. Without one, a resumed scheduled run waits and a manual run fails fast.
    • The scheduled skip branches are removed.

Why waiting is deadlock-free: the two policy workflows have separate Concurrency: 1 slots, a waiting step holds no lease, and nothing that runs under the admission tries to take it again.

Testing Done

  • just format go, just lint backend (0 issues), go build -tags exclude_frontend ./....
  • go test ./internal/backup/... ./internal/system/... ./internal/scheduler/runs/... passes.
  • Added TestAwaitRunWaitsForConflictingRun. A held admission blocks the waiter, which is admitted once the holder releases, and a canceled wait returns the context error. The test passes with -count=3.
  • Local review rounds 1 and 2 are in the comments below.
  • Not yet done: a live run with both schedules at the same minute in a dev environment.

AI Tool Used (if applicable)

Claude Code (Claude Opus 5.5) investigated this, wrote the code and test, and drafted this description, at my direction.

Additional Context

  • Manual resumes keep failing fast on purpose. The manual workflows run one task at a time and a new manual start parks the admission for its queued task, so a waiting resume could block until the timeout.
  • Per-volume backup policies (volume/children/backup/children/policies) still skip when a backup of the same volume is running. That's tracked separately.

🤖 Generated with Claude Code

Disclaimer Greptiles Reviews use AI, make sure to check over its work.

To better help train Greptile on our codebase, if the comment is useful and valid Like the comment, if its not helpful or invalid Dislike

To have Greptile Re-Review the changes, mention greptileai.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge based on this follow-up review.

Summary

Scheduled system and system-managed volume backups now wait their turn instead of skipping when another backup is running. Manual requests still fail fast when busy.

  • Overlapping scheduled backups now take turns instead of skipping.

Reviews (7) · Last reviewed commit: "Apply suggestion from @kmendell" · Reviewed by Greptile

…ping

Scheduled system backups and system-managed volume backups share one
admission and both default to 03:00. The one that started second was
skipped, and the Activity Center reported the skip as a success.

Scheduled runs now wait for the admission, up to six hours, and fail
visibly if it never frees. Manual runs still return a conflict right away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@neurekadev
neurekadev marked this pull request as ready for review November 18, 2026 19:16
@neurekadev
neurekadev requested a review from a team November 18, 2026 19:16
…ention

Scheduled runs now take the admission before reading their policy, so a
policy edited or disabled during the wait is respected. Retention and runs
resumed after a restart wait for the admission too. Before this, retention
failed and marked a successful backup as failed whenever the other
scheduled job grabbed the admission first, and a resumed backup could lose
the admission to a redelivered waiting job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@neurekadev

Copy link
Copy Markdown
Contributor Author

🔍 Code Review

PR #4876 · 8612ea241..d9b7c4118 · 4 files · ✅ No blocking issues remain

Critical: 0 High: 1 Medium: 1 Low: 1 Informational: 0

# Severity Finding Status
1 High Retention fails after a scheduled backup when the other job waits
backend/internal/system/children/backup/service.go:1628 · correctness
✅ Fixed 5b9dc2468
2 Medium Resume after restart can lose the admission to a waiting run
backend/internal/system/children/backup/service.go:1839 · reliability
✅ Fixed 5b9dc2468
3 Low Scheduled runs act on the policy read before a long wait
backend/internal/system/children/backup/children/volumes/service.go:524 · correctness
✅ Fixed 5b9dc2468
1 · Retention fails after a scheduled backup when the other job waits
  • Scenario: Both schedules fire at 03:00 and the system backup (retention 7) wins. When it releases the admission, the waiting volume job takes it within 250ms. applyRetention's single try then returns ErrSystemBackupAlreadyRunning, so runScheduledBackup reports the successful backup as failed and nothing is pruned.
  • Evidence: applyRetention used s.acquireRun (TryAcquire). CreateBackup released the lease before backUpRemoteEnvironments and applyRetention. Admission.Acquire polls every 250ms. Found while preparing the fixes below; the fix checker confirmed the scenario and the fix.
  • Fix: applyRetention waits for the admission with Engine.AwaitRun. All its callers are scheduled or resume paths that have already released their lease. (5b9dc2468)
  • Fix check: Pass: all four call sites run after the caller's lease is released, and deleteRuns takes no admission, so retention now waits for the volume job instead of failing.
2 · Resume after restart can lose the admission to a waiting run
  • Scenario: The volume backup holds the admission and the scheduled system backup waits. Arcane restarts, and both jobs are redelivered. The waiter's fresh run wins TryAcquire before the holder's resume, so reconcileBackup or backUpSelection gives up and the interrupted backup fails.
  • Evidence: flow/task.go classify maps context cancellation to ErrJobRejected, so the job is redelivered. Admission.TryAcquire checks only in-memory state after a restart. reconcileBackup returned outcome on !admitted, and backUpSelection returned alreadyRunning, which resumeSelection turns into NeedsAttention. Reviewers split Medium/Low; the user chose to fix it.
  • Fix: reconcileBackup waits with AwaitRun. backUpSelection still takes a parked manual claim through AcquireRun, and waits with AwaitRun when none is parked. (5b9dc2468)
  • Fix check: Pass: both resume paths now wait whichever job wins after the restart, and the manual Hold handoff is unchanged. Residual: a resumed manual run can wait up to 6h if a new manual run of the same workflow parks the lease within the 250ms poll gap. Narrow and bounded.
Low and informational (1)
  • 3 · Scheduled runs act on the policy read before a long wait (backend/internal/system/children/backup/children/volumes/service.go:524)
    • Scenario: An admin disables or edits a policy (selection, S3, retention) while its scheduled run waits behind the other backup. The run still executes once with the settings read before the wait.
    • Evidence: runScheduledSystemVolumeBackup and prepareSystemVolumeBackups resolved the policy before acquire. runScheduledBackup checked Enabled and the remote, and read RetentionCount, before CreateBackup waited.
    • Fix: Both scheduled handlers take the admission first and read and check the policy under it. CreateBackup (only caller) is folded into runScheduledBackup, and the lease is released before remote-environment backups and retention. (5b9dc2468)
    • Fix check: Pass: every return path releases the lease (Release is idempotent), nothing under the lease takes the admission again, and manual Start and StartBackup still fail fast.

🤖 Claude Opus 5.5 (High) · T3 Code (Claude Code) · Round 1 of 3

…starts

A manual system-managed volume backup started while another backup runs
returns not-found or validation errors again instead of the conflict.
Scheduled runs already hold the admission before calling prepare.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@neurekadev

Copy link
Copy Markdown
Contributor Author

🔍 Code Review

PR #4876 · d9b7c4118..5b9dc2468 · 4 files · ✅ No blocking issues remain

Critical: 0 High: 0 Medium: 0 Low: 1 Informational: 0

# Severity Finding Status
1 Low Manual volume Start reports "already running" in place of other errors
backend/internal/system/children/backup/children/volumes/service.go:353 · correctness
✅ Fixed fe5491b3c
Low and informational (1)
  • 1 · Manual volume Start reports "already running" in place of other errors (backend/internal/system/children/backup/children/volumes/service.go:353)
    • Scenario: A scheduled backup holds the admission and a user calls Start with a just-deleted PolicyID or an invalid custom config. They get 409 already-running instead of 404/400, and see the real error only after retrying.
    • Evidence: 5b9dc24 moved acquire(ctx) above policy resolution in prepareSystemVolumeBackups. Start's acquire returns s.alreadyRunning when TryAcquireRun isn't admitted. Base and d9b7c41 resolved the policy first.
    • Fix: prepareSystemVolumeBackups resolves the policy before acquiring again. The scheduled caller already holds its lease before calling prepare, so its policy read stays under the admission. (fe5491b3c)
    • Fix check: Pass: not-found and validation errors return before any acquire, every error path after acquire releases the lease, and the scheduled path still reads the policy under its up-front AwaitRun lease.

🤖 Claude Opus 5.5 (High) · T3 Code (Claude Code) · Round 2 of 3

Manual system and volume backup workflows run one task at a time, and a
new manual start parks the admission for its queued task. A resumed manual
run that waited for the admission would block on that task until the
six-hour timeout, so manual resumes fail fast again. Scheduled policy
workflows never park the admission, so their resumes still wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread backend/internal/backup/service.go Resolved
@kmendell
kmendell merged commit 40e65a7 into getarcaneapp:main Nov 18, 2026
16 of 18 checks passed
neurekadev added a commit to neurekadev/arcane that referenced this pull request Nov 18, 2026
getarcaneapp#4876 merged as 40e65a7. Resolved the conflict by keeping upstream's
removal of the admissionWait comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐞 Bug: Overlapping scheduled backups are skipped silently and reported as succeeded

2 participants