Repository navigation
feat: export and import projects with their volumes - #4888
neurekadev wants to merge 13 commits into
Conversation
…ping Scheduled system backups and system-managed volume backups share one admission and both default to 03:00. The one that started second was skipped, and the Activity Center reported the skip as a success. Scheduled runs now wait for the admission, up to six hours, and fail visibly if it never frees. Manual runs still return a conflict right away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ention Scheduled runs now take the admission before reading their policy, so a policy edited or disabled during the wait is respected. Retention and runs resumed after a restart wait for the admission too. Before this, retention failed and marked a successful backup as failed whenever the other scheduled job grabbed the admission first, and a resumed backup could lose the admission to a redelivered waiting job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…starts A manual system-managed volume backup started while another backup runs returns not-found or validation errors again instead of the conflict. Scheduled runs already hold the admission before calling prepare. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Manual system and volume backup workflows run one task at a time, and a new manual start parks the admission for its queued task. A resumed manual run that waited for the admission would block on that task until the six-hour timeout, so manual resumes fail fast again. Scheduled policy workflows never park the admission, so their resumes still wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A scheduled backup of a volume that another backup holds was skipped and reported as succeeded, and a scheduled system-managed run only counted that volume as skipped. Scheduled backups now wait for the volume's admission. Manual backups still fail fast, because a manual start parks the admission for its queued task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…olume Scheduled per-volume and system-managed backups now take the volume admission before resolving their plan, so a policy edited, disabled, or deleted during the wait is respected. A per-volume policy disabled or deleted meanwhile ends its run as canceled, and a volume-level policy enabled meanwhile takes the volume out of the system-managed run. Manual starts keep their validation, authorization, then conflict order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Project export can now include the project's named volumes in one portable bundle: a gzip tar with an arcane-project.json manifest, the existing project archive, and one existing volume archive per Compose key. The project can be stopped during export for a consistent copy. Import detects a bundle, refuses before changing anything if a target volume already exists, recreates the project and its volumes by Compose key, rolls back on a failed volume restore, and can start the project. Plain exports and plain project archives behave as before. The volume classification shared with project transfer moves to pkg/projects, so transfer, export, and the planned project backups use one rule. The UI gets Export on the project page and Import on the projects list; the CLI gets `projects export` and `projects import`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The export preview marks named volumes that don't exist yet as missing, so default exports of undeployed projects no longer fail. An explicitly requested missing volume is rejected before the project is stopped. Exports are written straight into their upload session instead of a temp file that was then copied, so they need one copy of disk space. Browsers download them through a new streaming upload-session download route instead of holding the whole bundle in memory; the session is removed once it is fully sent. Proxied upload chunk reads and downloads now check the upload kind's permission, so remote exports work for non-admin users. The CLI refuses a plain archive from a server that ignores the volume request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An empty named volume exports as an archive with only its root directory. The volume restore rejects empty uploads, so importing such a bundle failed and rolled the whole project back. Import now recognizes an empty volume archive and creates the Compose-labelled volume directly, tracked for rollback like restored volumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
getarcaneapp#4876 merged as 40e65a7. Resolved the conflict by keeping upstream's removal of the admissionWait comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in upstream main through getarcaneapp#4879. Resolved the en.json conflict by keeping upstream's new keys, then this branch's keys appended after them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🔍 Code ReviewPR #4888 · 1 · Export fails for undeployed projects with default volume selection
2 · Browser holds the whole bundle in memory before saving
3 · Server writes the whole bundle twice on disk during export
4 · Import of a bundle with an empty volume fails and rolls back
Low and informational (2)
🤖 Claude Opus 5.5 (High) · T3 Code (Claude Code) · Round 1 of 3 |
Import was a menu action, which folded Update Projects into an Actions menu with it and broke the E2E tests that click Update Projects. Import is now a secondary button, so the projects header keeps its existing buttons. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| // ClassifyProjectVolumes sorts a project's Compose volumes by key and includes only named volumes by default; | ||
| // external volumes and driver_opts volumes keep their data outside Docker's volume storage. | ||
| func ClassifyProjectVolumes(project types.Project) []types.ProjectVolume { | ||
| classified := make([]types.ProjectVolume, 0, len(project.Volumes)) | ||
| for _, key := range slices.Sorted(maps.Keys(project.Volumes)) { | ||
| config := project.Volumes[key] | ||
| kind := types.ProjectVolumeNamed | ||
| switch { | ||
| case bool(config.External): | ||
| kind = types.ProjectVolumeExternal | ||
| case len(config.DriverOpts) > 0: | ||
| kind = types.ProjectVolumeDriverOpts | ||
| } | ||
| classified = append(classified, types.ProjectVolume{ | ||
| Labels: config.Labels, | ||
| DriverOpts: config.DriverOpts, | ||
| Key: key, | ||
| Name: config.Name, | ||
| Kind: kind, | ||
| Driver: config.Driver, | ||
| Included: kind == types.ProjectVolumeNamed, | ||
| }) | ||
| } | ||
| return classified | ||
| } |
There was a problem hiding this comment.
external volumes can still live in dockers volumes, this is how i do it, cause i hate the way compose names volumes.
…d import Import checks the volume names the imported project actually uses before restoring anything, and only rolls back volumes it created, so a volume that existed before can no longer be deleted. Empty volumes keep the archive root's owner and mode. Export with stop only stops the project's running containers, holds the project lock while volumes are copied, and starts exactly those containers again instead of redeploying the whole project. The CLI downloads into a temporary file and only replaces the output once the download completes, so a failed re-export keeps the previous bundle; pipes such as /dev/stdout are still written directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| running := slices.DeleteFunc(found.Items, func(c container.Summary) bool { return c.State != container.StateRunning }) | ||
| switch { | ||
| case len(running) == 0: | ||
| return stopped, nil |
There was a problem hiding this comment.
Restarting containers escape export stops.
stopRunningContainers drops every container whose state is not running, including containers waiting for Docker to restart them. Such a container can start writing again while the volumes are copied. Both passes skip it, so an export with stopProject can still return a bundle marked consistent: true.
Stop containers with a pending restart too, or refuse the export before copying their data.
Knowledge Base Used: Containers, environments, and workloads
Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/internal/project/service.go
Line: 1980-1983
Comment:
**Restarting containers escape export stops.** `stopRunningContainers` drops every container whose state is not `running`, including containers waiting for Docker to restart them. Such a container can start writing again while the volumes are copied. Both passes skip it, so an export with `stopProject` can still return a bundle marked `consistent: true`.
Stop containers with a pending restart too, or refuse the export before copying their data.
**Knowledge Base Used:** [Containers, environments, and workloads](https://app.greptile.com/-/switch-tenant?tenant=getarcaneapp&next=%2Fcustom-context%2Fknowledge-base%2Fgetarcaneapp%2Farcane%2F-%2Fdocs%2Fworkload-management.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| var errs []error | ||
| for _, id := range containerIDs { | ||
| if _, startErr := dockerClient.ContainerStart(ctx, id, client.ContainerStartOptions{}); startErr != nil { | ||
| errs = append(errs, fmt.Errorf("start container %s: %w", id, startErr)) | ||
| } | ||
| } | ||
| return errors.Join(errs...) |
There was a problem hiding this comment.
Export ignores service startup order.
startContainers starts containers in the same order returned by Docker, without checking Compose dependencies or waiting for them to become healthy. For a project whose app uses depends_on with service_healthy, the app can start before its database and exit. ContainerStart can still succeed, so the export reports success while leaving the app down.
Restore only the previously running containers, but honor their dependency order and readiness checks.
Knowledge Base Used: Containers, environments, and workloads
Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/internal/project/service.go
Line: 2006-2012
Comment:
**Export ignores service startup order.** `startContainers` starts containers in the same order returned by Docker, without checking Compose dependencies or waiting for them to become healthy. For a project whose app uses `depends_on` with `service_healthy`, the app can start before its database and exit. `ContainerStart` can still succeed, so the export reports success while leaving the app down.
Restore only the previously running containers, but honor their dependency order and readiness checks.
**Knowledge Base Used:** [Containers, environments, and workloads](https://app.greptile.com/-/switch-tenant?tenant=getarcaneapp&next=%2Fcustom-context%2Fknowledge-base%2Fgetarcaneapp%2Farcane%2F-%2Fdocs%2Fworkload-management.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Important
PR 3 of 3. Merge in order: #4876 (merged) → #4879 → this PR. It is stacked on #4879, so the diff looks large: it includes #4879's changes until that merges. Only this PR's changes.
Importing a bundle with non-empty volumes also needs #4887 (merged) and getarcaneapp/tools#22 (approved) for #4885. Without the tools change, the existing volume restore this PR reuses fails with
chown: not found.Checklist
mainbranchm.*())What This PR Implements
Addresses discussion #4881. Feature requests live in Discussions here, so there's no issue. This is also the first step of #4880, which uses the same manifest and volume selection for project backups.
Project Export can now include the project's volume data in one portable bundle, and Import recreates the project and its volumes from it on any Arcane, as a project-level copy that doesn't need two connected environments. Plain export and plain project archives behave exactly as before, and project copy/move between environments is unchanged.
Changes Made
Bundle (
pkg/projects/archive.go): a gzip tar named<project>.arcane.tar.gzholdingarcane-project.json(manifest, first entry),project.tar.gz(the existing project archive), andvolumes/<compose key>.tar.gz(the existing volume archive). Reading it checks entry names and order, archive paths, keys and version, and caps the manifest at 1 MiB. Anything that isn't a bundle is imported as a plain project archive, as today.Volume selection (
pkg/projects/volume_migration.go):ClassifyProjectVolumesis now the single rule, shared with project copy/move. Named volumes are included by default; external anddriver_optsvolumes are listed but not exported. Volumes that don't exist yet are marked missing.Export (
internal/project):GET /agents/{id}/projects/{projectId}/exportpreviews the volumes and whether the project is running.POST …/exporttakes an optional{volumes, stopProject}. Including volumes needsvolumes:backup;stopProjectstops the project for a consistent copy (needsprojects:down/projects:deploy) and always starts it again.Import:
startdeploys afterwards (needsprojects:deploy).volumes:createandvolumes:upload.Downloads:
GET /agents/{id}/uploads/{kind}/{uploadId}/downloadstreams a complete session, so browsers don't hold the bundle in memory. The session is deleted once fully sent.UI:
CLI:
arcane projects export <project> [file] [--volume <key>]... [--all-volumes] [--stop]andarcane projects import <file> [--name] [--start]. Against an older server that ignores the volume request, export refuses instead of saving a plain archive.Testing Done
just format all,just lint all(0 issues; svelte-check 0 errors),go build -tags exclude_frontend ./....WriteSessionstreaming and download cleanupAI Tool Used (if applicable)
Claude Code (Claude Opus 5.5) planned and implemented this, ran the local review and the evidence, and drafted this description, at my direction.
Additional Context
🤖 Generated with Claude Code
📸 Evidence
Before
main@8612ea241→ Afterfeat/4881-project-export-import-volumes@c8fa87e17· fresh Arcane with a seeded test project evidence-shop (nginx + busybox; volumes site, cache, external shared) on a sandbox Docker engine; test account only, no private data. Screens and recordings from c8fa87e; API excerpts from 402e0ea (import with #4887 applied and the tools#22 image), and that code hasn't changed since🖥️ Project actions menu — desktop
8612ea241c8fa87e17🖥️ Projects list header — desktop 1280×800
8612ea241c8fa87e17🎬 Export dialog — recording, 0:07
🎬 Export dialog on a phone — recording, 0:09
🎬 Import dialog — recording, 0:08
🔌 API response —
POST /api/agents/0/projects/{id}/export {"volumes":["site","cache"],"stopProject":true}, trimmed to changed fields🔌 API response —
POST /api/agents/0/projects/import {"uploadId":"{id}","name":"evidence-shop-copy","start":true}, trimmed to changed fields📸 Captured by Claude Opus 5.5 (High) · T3 Code (Claude Code) · Playwright · Chromium (mcr.microsoft.com/playwright, --docker); Arcane images built with docker/Dockerfile · 2026-11-18 23:16 UTC
Disclaimer Greptiles Reviews use AI, make sure to check over its work.
To better help train Greptile on our codebase, if the comment is useful and valid Like the comment, if its not helpful or invalid Dislike
To have Greptile Re-Review the changes, mention
greptileai.Fix the two export lifecycle issues before merging.
Fix with agent prompt
Summary
This PR adds portable project bundles with selected volume data, browser downloads, and CLI import/export commands.
Review checks were code-based; no runtime tests were run.
Reviews (2) · Last reviewed commit: "fix: keep existing volumes and stopped s..." · Reviewed by Greptile