Repository navigation
fix(roles/duplicity): throttle duba to the free temp space, fix excludes of divided sources - #751
Merged
Merged
Conversation
On start, every duplicity backup aborts with exit code 35 if its temp dir has less than (concurrency + 2) volumes plus 30% of one free, which is 860 MB for duba's --concurrency 2 --volsize 200 (660 MB with 3.0.0, 460 MB before). duba started up to 7 backups into the same /tmp, so the ones started later found less space and failed, seen on hosts with a 1 GiB /tmp. duba now runs only as many backups at the same time as fit into the free temp space, and logs at INFO when it throttles itself. The process limit in run_cmd_in_background() is exact now; it let one process more run than asked. The restore space check reads TMPDIR from the config env as well. Verified against the duplicity source (check_resources(), rel.0.8.23 to rel.3.2.3.dev2) and with a stub that mimics the check on ubi9: with 1000 MB free, the old duba fails 3 of 4 backups with 35, the new one runs all 4.
…ckup With divide: true, every first-level entry is a backup with its own root, and duba passed all excludes of the source to each of them. duplicity aborts a backup with exit code 14 (file_prefix_error) if an absolute exclude cannot match its root or a parent, so /data/Archiv broke the backup of /data/lucymwangama and of every other entry. Since dfe44ab fails the run on any failed source, such a config fails every night. duba now passes an absolute exclude only to the backups whose root it can match, comparing the leading path components with fnmatch (** matches anything). Other patterns go to duplicity unchanged. Verified with duplicity 3.0.7 on Fedora 43: the old duba fails all three divided backups with 14, the new one backs them up with the excludes applied.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes for
duba(role:duplicity):dubastarted up to 7 backups into the same/tmp, so on hosts with a 1 GiB/tmpthe later ones failed.dubanow runs only as many backups at the same time as fit, and logs at INFO when it throttles itself.divide: true, every first-level entry is its own backup, and an absolute exclude outside its root makes duplicity abort with exit code 14. One exclude like/data/Archivbroke the backups of all other entries of/data, every night since dfe44ab fails the run on any failed source.Verified against the duplicity source, with duplicity 3.0.7 in a Fedora 43 container, and with
molecule test --scenario-name duplicityon rocky8 and rocky10 (verify gained a throttled backup on a 900 MB tmpfs and an exclude of a sibling entry in the divided source).