Skip to content

fix(roles/duplicity): throttle duba to the free temp space, fix excludes of divided sources - #751

Merged
markuslf merged 2 commits into
mainfrom
fix/duba-temp-space-and-divided-excludes
Nov 16, 2026
Merged

markuslf merged 2 commits into
mainfrom
fix/duba-temp-space-and-divided-excludes

Conversation

@markuslf

Copy link
Copy Markdown
Member

Two fixes for duba (role:duplicity):

  • Throttle backups to the free temp space. Every duplicity backup aborts on start with exit code 35 if its temp dir has less than 860 MB free (660 MB with duplicity 3.0.0, 460 MB before). duba started up to 7 backups into the same /tmp, so on hosts with a 1 GiB /tmp the later ones failed. duba now runs only as many backups at the same time as fit, and logs at INFO when it throttles itself.
  • Pass only the excludes that fit each divided backup. With divide: true, every first-level entry is its own backup, and an absolute exclude outside its root makes duplicity abort with exit code 14. One exclude like /data/Archiv broke the backups of all other entries of /data, every night since dfe44ab fails the run on any failed source.

Verified against the duplicity source, with duplicity 3.0.7 in a Fedora 43 container, and with molecule test --scenario-name duplicity on rocky8 and rocky10 (verify gained a throttled backup on a 900 MB tmpfs and an exclude of a sibling entry in the divided source).

On start, every duplicity backup aborts with exit code 35 if its temp dir
has less than (concurrency + 2) volumes plus 30% of one free, which is
860 MB for duba's --concurrency 2 --volsize 200 (660 MB with 3.0.0, 460 MB
before). duba started up to 7 backups into the same /tmp, so the ones
started later found less space and failed, seen on hosts with a 1 GiB /tmp.

duba now runs only as many backups at the same time as fit into the free
temp space, and logs at INFO when it throttles itself. The process limit
in run_cmd_in_background() is exact now; it let one process more run than
asked. The restore space check reads TMPDIR from the config env as well.

Verified against the duplicity source (check_resources(), rel.0.8.23 to
rel.3.2.3.dev2) and with a stub that mimics the check on ubi9: with 1000 MB
free, the old duba fails 3 of 4 backups with 35, the new one runs all 4.
…ckup

With divide: true, every first-level entry is a backup with its own root,
and duba passed all excludes of the source to each of them. duplicity
aborts a backup with exit code 14 (file_prefix_error) if an absolute
exclude cannot match its root or a parent, so /data/Archiv broke the
backup of /data/lucymwangama and of every other entry. Since dfe44ab
fails the run on any failed source, such a config fails every night.

duba now passes an absolute exclude only to the backups whose root it can
match, comparing the leading path components with fnmatch (** matches
anything). Other patterns go to duplicity unchanged.

Verified with duplicity 3.0.7 on Fedora 43: the old duba fails all three
divided backups with 14, the new one backs them up with the excludes
applied.
@markuslf
markuslf merged commit c0098b9 into main Nov 16, 2026
11 checks passed
@markuslf
markuslf deleted the fix/duba-temp-space-and-divided-excludes branch November 16, 2026 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant