Skip to content

fix: stage Rustic CA bundle in a volume instead of copying into the container - #4875

Open
neurekadev wants to merge 1 commit into
getarcaneapp:mainfrom
neurekadev:fix/4870-rustic-ca-trust-volume
Open

neurekadev wants to merge 1 commit into
getarcaneapp:mainfrom
neurekadev:fix/4870-rustic-ca-trust-volume

Conversation

@neurekadev

@neurekadev neurekadev commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Checklist

  • This PR is not opened from my fork’s main branch
  • All new user-facing strings are translated via Paraglide (m.*()) (no new strings)

What This PR Implements

Fixes: #4870 (regression of #4616)

Since #4797, system backups fail on Docker Engine before 29.9.0 with:

failed to stage CA certificates for Rustic container: copy archive to container: Error response from daemon: openat data/arcane.db: read-only file system

CopyToContainer on a stopped container makes the daemon replay every mount and call createIfNotExists on each mount target, which opens the target with O_CREATE|O_WRONLY. The system backup container mounts the VACUUM INTO database copies as files under the read-only /data mount, so that open fails with EROFS. moby fixed this in 29.9.0 (moby/moby#53716, backported in moby/moby#53724), but older engines still fail.

This brings back #4621's approach: a stager container that mounts only a fresh volume receives the bundle, and that volume is mounted read-only at /etc/ssl/certs. #4780 (ZFS) failed because the stager was removed right after the copy ("dataset is busy"). Now the stager is removed only after Rustic exits, with retries, and a failed removal is logged instead of failing the backup.

Changes Made

  • pkg/rustic/trust.go: stageTrustBundle creates the volume and stager, copies the bundle, and returns a cleanup function that removes the stager (retrying) and then the volume.
  • pkg/rustic/run.go: mounts the staged volume read-only and defers cleanup so it runs after the Rustic container is removed. Removes the CopyToContainer into the Rustic container.

Testing Done

  • just format go, just lint backend (0 issues), go build -tags exclude_frontend ./..., go test ./internal/backup/....
  • Reproduced in docker:29.6.0-dind with the same mount layout (read-only volume at /data, read-only volume-subpath file at /data/arcane.db). docker cp into that container fails with the exact error above. docker cp into a stager that mounts only the trust volume succeeds, a container with the layout plus the read-only trust volume reads both the bundle and the file, and removing the stager afterwards succeeds. The same docker cp succeeds on 29.10.0.
  • Not yet done: an end-to-end system backup with this build in a running Arcane dev environment, and ZFS testing (I don't have a ZFS host). Human verification is still pending, so this stays a draft.

AI Tool Used (if applicable)

Claude Code (Claude Opus 5.5) investigated this, wrote the code and drafted this description, at my direction.

Additional Context

Docker Engine 29.9.0 and later avoid the error without this change. That's a workaround for affected users until this lands.

🤖 Generated with Claude Code

Disclaimer Greptiles Reviews use AI, make sure to check over its work.

To better help train Greptile on our codebase, if the comment is useful and valid Like the comment, if its not helpful or invalid Dislike

To have Greptile Re-Review the changes, mention greptileai.

RetriggerView in GreptileConfidence Score: 5/5

No code issues were found that block merging; the author’s planned human testing remains pending.

Summary

This PR stages Rustic’s CA certificates in a temporary Docker volume instead of copying them into the backup container.

  • Rustic now reads host certificates from a staged read-only volume.

No actionable issues found. neurekadev explicitly notes that end-to-end and ZFS human testing remain pending and keeps the PR in draft for that reason. No tests were run during this review.

Reviews (1) · Last reviewed commit: "fix: stage Rustic CA bundle in a volume ..." · Reviewed by Greptile

…ontainer

Copying the bundle into the stopped Rustic container makes the daemon
create every mount target first. System backups mount the database copies
as files under the read-only /data mount, so that fails with "openat
data/arcane.db: read-only file system".

Stage the bundle in a volume through a stager that mounts only that volume,
and remove the stager after Rustic exits, retrying, so ZFS no longer fails
with a busy dataset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@neurekadev

Copy link
Copy Markdown
Contributor Author

🔍 Code Review

PR #4875 · 8612ea241..ed8ad8c09 · 2 files · ✅ No findings

Critical: 0 High: 0 Medium: 0 Low: 0 Informational: 0

🤖 Claude Opus 5.5 (High) · T3 Code (Claude Code) · Round 1 of 3

@kmendell

Copy link
Copy Markdown
Member

Does this spin up two containers just to copy the CA Certs?

Im not sure im a fan of that as a "workaround" per se.

@neurekadev

Copy link
Copy Markdown
Contributor Author

Yeah, it does: a short-lived stager container (created, never started) plus a temp volume per Rustic run, just to get the bundle in. Fair point that it's heavy for one file.

The real bug is Docker's: docker cp into a stopped container fails when it has file mounts under a read-only mount, and Engine 29.9.0 fixed it (moby/moby#53716). So two leaner options:

  1. Keep the current single-container copy and require Docker 29.9+ for system backups, with a clear error on older engines.
  2. Write the bundle into Arcane's data dir and mount just that file into the Rustic container (the same volume-subpath mount system backups already use for the DB copies). One container, works on older Docker too.

I'd lean towards 2, but happy to do 1 if you'd rather not carry a workaround for an old Docker bug. Which do you prefer?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

System backup fails again when staging Rustic CA certificates (regression of #4616)

2 participants