Skip to content

feat: export and import projects with their volumes - #4888

Open
neurekadev wants to merge 13 commits into
getarcaneapp:mainfrom
neurekadev:feat/4881-project-export-import-volumes
Open

neurekadev wants to merge 13 commits into
getarcaneapp:mainfrom
neurekadev:feat/4881-project-export-import-volumes

Conversation

@neurekadev

@neurekadev neurekadev commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Important

PR 3 of 3. Merge in order: #4876 (merged) → #4879 → this PR. It is stacked on #4879, so the diff looks large: it includes #4879's changes until that merges. Only this PR's changes.

Importing a bundle with non-empty volumes also needs #4887 (merged) and getarcaneapp/tools#22 (approved) for #4885. Without the tools change, the existing volume restore this PR reuses fails with chown: not found.

Checklist

  • This PR is not opened from my fork’s main branch
  • All new user-facing strings are translated via Paraglide (m.*())

What This PR Implements

Addresses discussion #4881. Feature requests live in Discussions here, so there's no issue. This is also the first step of #4880, which uses the same manifest and volume selection for project backups.

Project Export can now include the project's volume data in one portable bundle, and Import recreates the project and its volumes from it on any Arcane, as a project-level copy that doesn't need two connected environments. Plain export and plain project archives behave exactly as before, and project copy/move between environments is unchanged.

Changes Made

Bundle (pkg/projects/archive.go): a gzip tar named <project>.arcane.tar.gz holding arcane-project.json (manifest, first entry), project.tar.gz (the existing project archive), and volumes/<compose key>.tar.gz (the existing volume archive). Reading it checks entry names and order, archive paths, keys and version, and caps the manifest at 1 MiB. Anything that isn't a bundle is imported as a plain project archive, as today.

Volume selection (pkg/projects/volume_migration.go): ClassifyProjectVolumes is now the single rule, shared with project copy/move. Named volumes are included by default; external and driver_opts volumes are listed but not exported. Volumes that don't exist yet are marked missing.

Export (internal/project):

  • GET /agents/{id}/projects/{projectId}/export previews the volumes and whether the project is running.
  • POST …/export takes an optional {volumes, stopProject}. Including volumes needs volumes:backup; stopProject stops the project for a consistent copy (needs projects:down/projects:deploy) and always starts it again.
  • The bundle is written straight into its upload session, so only one copy is on disk.

Import:

  • Before changing anything, refuses if a target volume already exists.
  • Creates the project, then recreates each volume by Compose key with Compose labels. Empty volumes are created directly; the rest go through the existing restore.
  • If anything fails, removes the project and only the volumes it created.
  • start deploys afterwards (needs projects:deploy).
  • Volumes need volumes:create and volumes:upload.

Downloads:

  • New GET /agents/{id}/uploads/{kind}/{uploadId}/download streams a complete session, so browsers don't hold the bundle in memory. The session is deleted once fully sent.
  • Proxied upload chunk reads and downloads now check the upload kind's permission, so remote exports work for non-admin users.

UI:

  • Export in the project page's Manage menu, with a dialog for picking volumes and stopping the project.
  • Import in the projects list's new Actions menu, with name and "start after import".

CLI: arcane projects export <project> [file] [--volume <key>]... [--all-volumes] [--stop] and arcane projects import <file> [--name] [--start]. Against an older server that ignores the volume request, export refuses instead of saving a plain archive.

Testing Done

  • just format all, just lint all (0 issues; svelte-check 0 errors), go build -tags exclude_frontend ./....
  • Full backend and CLI test suites pass.
  • New tests:
    • volume classification
    • bundle round trip, legacy detection, and nine invalid-bundle cases
    • empty-volume archive detection
    • import refusing existing volumes before creating anything
    • import creating empty volumes without a restore
    • export skipping volumes that don't exist yet
    • WriteSession streaming and download cleanup
    • proxy permissions for upload reads
  • Ran end to end against images built from this branch with a seeded test project (see Evidence):
    • Export produced the bundle and stopped and restarted the project.
    • Import recreated both volumes with their data and the right root owner, and started the project.
    • A second import was refused.
  • That run found and fixed one bug: a bundle with an empty volume failed to import. It also found 🐞 Bug: Project copy/move with volumes fails: chown not found in the tools helper image #4885.

AI Tool Used (if applicable)

Claude Code (Claude Opus 5.5) planned and implemented this, ran the local review and the evidence, and drafted this description, at my direction.

Additional Context

  • Exports run inside the request. A very large export on a remote agent can outrun the 30-minute proxy timeout. The agent still finishes and the unclaimed session is purged after 24h. Making export a background job is left for later.

🤖 Generated with Claude Code

📸 Evidence

Surfaces: 7 Captured: 7 Open: 0 Recording: 3

Before main @ 8612ea241 → After feat/4881-project-export-import-volumes @ c8fa87e17 · fresh Arcane with a seeded test project evidence-shop (nginx + busybox; volumes site, cache, external shared) on a sandbox Docker engine; test account only, no private data. Screens and recordings from c8fa87e; API excerpts from 402e0ea (import with #4887 applied and the tools#22 image), and that code hasn't changed since

Surface Viewport Capture Look for
🖥️ Project actions menu Desktop 1280×800 Before / After New Export entry in the Manage group
🖥️ Projects list header Desktop 1280×800 Before / After New Import button; Update Projects stays as it was
🎬 Export dialog Desktop 1280×800 Recording 0:07 Named volumes preselected, external volume disabled with its reason, stop-project toggle
🎬 Export dialog on a phone Phone 390×844 Recording 0:09 Dialog fits as a bottom sheet with no horizontal overflow
🎬 Import dialog Desktop 1280×800 Recording 0:08 Archive drop zone, project name, Start after import
🔌 Export API — Response excerpt Bundle with manifest, project archive and one archive per selected volume; project stopped and restarted; session removed after download
🔌 Import API (with #4887 and getarcaneapp/tools#22) — Response excerpt Project and both volumes recreated with data and the archive root's owner/mode; a second import is refused before changing anything

🖥️ Project actions menu — desktop

Before · 8612ea241 After · c8fa87e17
Before: Project actions menu After: Project actions menu
🖥️ Projects list header — desktop 1280×800
Before · 8612ea241 After · c8fa87e17
Before: Projects list header After: Projects list header
🎬 Export dialog — recording, 0:07 After: Export dialog

▶️ Full-quality MP4 (148 KB)

🎬 Export dialog on a phone — recording, 0:09 After: Export dialog on a phone

▶️ Full-quality MP4 (148 KB)

🎬 Import dialog — recording, 0:08 After: Import dialog

▶️ Full-quality MP4 (116 KB)

🔌 API response — POST /api/agents/0/projects/{id}/export {"volumes":["site","cache"],"stopProject":true}, trimmed to changed fields
+ filename: evidence-shop.arcane.tar.gz
+ entries:  arcane-project.json, project.tar.gz, volumes/cache.tar.gz, volumes/site.tar.gz
+ manifest: formatVersion 1, consistent: true, shared (external) listed with included: false
+ GET /api/agents/0/uploads/project-archive/{id} after the download → 404
🔌 API response — POST /api/agents/0/projects/import {"uploadId":"{id}","name":"evidence-shop-copy","start":true}, trimmed to changed fields
+ success: true, name: evidence-shop-copy
+ volumes: evidence-shop-copy_cache (empty, created directly), evidence-shop-copy_site
+ evidence-shop-copy_site root 999:999 700, index.html restored
+ evidence-shop-copy-web-1 Up, evidence-shop-copy-worker-1 Up
+ same bundle again → 409 these volumes already exist: evidence-shop-copy_cache, evidence-shop-copy_site
- without #4887/tools#22: failed to restore volume root ownership: chown: not found (project and volumes rolled back)

📸 Captured by Claude Opus 5.5 (High) · T3 Code (Claude Code) · Playwright · Chromium (mcr.microsoft.com/playwright, --docker); Arcane images built with docker/Dockerfile · 2026-11-18 23:16 UTC

Disclaimer Greptiles Reviews use AI, make sure to check over its work.

To better help train Greptile on our codebase, if the comment is useful and valid Like the comment, if its not helpful or invalid Dislike

To have Greptile Re-Review the changes, mention greptileai.

RetriggerView in GreptileConfidence Score: 3/5

Fix the two export lifecycle issues before merging.

Fix All in CodexFindings

  1. P1 Restarting containers escape export stops ▶
  2. P1 Export ignores service startup order ▶
Fix with agent prompt
### Issue 1
backend/internal/project/service.go:1980-1983
**Restarting containers escape export stops.** `stopRunningContainers` drops every container whose state is not `running`, including containers waiting for Docker to restart them. Such a container can start writing again while the volumes are copied. Both passes skip it, so an export with `stopProject` can still return a bundle marked `consistent: true`.

Stop containers with a pending restart too, or refuse the export before copying their data.

### Issue 2
backend/internal/project/service.go:2006-2012
**Export ignores service startup order.** `startContainers` starts containers in the same order returned by Docker, without checking Compose dependencies or waiting for them to become healthy. For a project whose app uses `depends_on` with `service_healthy`, the app can start before its database and exit. `ContainerStart` can still succeed, so the export reports success while leaving the app down.

Restore only the previously running containers, but honor their dependency order and readiness checks.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

This PR adds portable project bundles with selected volume data, browser downloads, and CLI import/export commands.

  • Project exports can include selected named-volume data in a portable bundle.
  • Project imports can recreate the bundle’s selected volumes with the project.
  • Complete upload sessions can stream to a download without loading the whole file into a browser.
  • Project copy and export use the same rules to classify volumes.
  • Scheduled volume backups wait their turn and use the current policy.

Review checks were code-based; no runtime tests were run.

Reviews (2) · Last reviewed commit: "fix: keep existing volumes and stopped s..." · Reviewed by Greptile

neurekadev and others added 11 commits November 18, 2026 11:07
…ping

Scheduled system backups and system-managed volume backups share one
admission and both default to 03:00. The one that started second was
skipped, and the Activity Center reported the skip as a success.

Scheduled runs now wait for the admission, up to six hours, and fail
visibly if it never frees. Manual runs still return a conflict right away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ention

Scheduled runs now take the admission before reading their policy, so a
policy edited or disabled during the wait is respected. Retention and runs
resumed after a restart wait for the admission too. Before this, retention
failed and marked a successful backup as failed whenever the other
scheduled job grabbed the admission first, and a resumed backup could lose
the admission to a redelivered waiting job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…starts

A manual system-managed volume backup started while another backup runs
returns not-found or validation errors again instead of the conflict.
Scheduled runs already hold the admission before calling prepare.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Manual system and volume backup workflows run one task at a time, and a
new manual start parks the admission for its queued task. A resumed manual
run that waited for the admission would block on that task until the
six-hour timeout, so manual resumes fail fast again. Scheduled policy
workflows never park the admission, so their resumes still wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A scheduled backup of a volume that another backup holds was skipped and
reported as succeeded, and a scheduled system-managed run only counted
that volume as skipped. Scheduled backups now wait for the volume's
admission. Manual backups still fail fast, because a manual start parks
the admission for its queued task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…olume

Scheduled per-volume and system-managed backups now take the volume
admission before resolving their plan, so a policy edited, disabled, or
deleted during the wait is respected. A per-volume policy disabled or
deleted meanwhile ends its run as canceled, and a volume-level policy
enabled meanwhile takes the volume out of the system-managed run. Manual
starts keep their validation, authorization, then conflict order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Project export can now include the project's named volumes in one
portable bundle: a gzip tar with an arcane-project.json manifest, the
existing project archive, and one existing volume archive per Compose
key. The project can be stopped during export for a consistent copy.
Import detects a bundle, refuses before changing anything if a target
volume already exists, recreates the project and its volumes by Compose
key, rolls back on a failed volume restore, and can start the project.
Plain exports and plain project archives behave as before.

The volume classification shared with project transfer moves to
pkg/projects, so transfer, export, and the planned project backups use
one rule. The UI gets Export on the project page and Import on the
projects list; the CLI gets `projects export` and `projects import`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The export preview marks named volumes that don't exist yet as missing,
so default exports of undeployed projects no longer fail. An explicitly
requested missing volume is rejected before the project is stopped.

Exports are written straight into their upload session instead of a
temp file that was then copied, so they need one copy of disk space.
Browsers download them through a new streaming upload-session download
route instead of holding the whole bundle in memory; the session is
removed once it is fully sent. Proxied upload chunk reads and downloads
now check the upload kind's permission, so remote exports work for
non-admin users. The CLI refuses a plain archive from a server that
ignores the volume request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An empty named volume exports as an archive with only its root
directory. The volume restore rejects empty uploads, so importing such a
bundle failed and rolled the whole project back. Import now recognizes
an empty volume archive and creates the Compose-labelled volume
directly, tracked for rollback like restored volumes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
getarcaneapp#4876 merged as 40e65a7. Resolved the conflict by keeping upstream's
removal of the admissionWait comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in upstream main through getarcaneapp#4879. Resolved the en.json conflict by
keeping upstream's new keys, then this branch's keys appended after them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@neurekadev

Copy link
Copy Markdown
Contributor Author

🔍 Code Review

PR #4888 · 5877c7fe7..93d2e81b3 · 23 files · ✅ No blocking issues remain

Critical: 0 High: 0 Medium: 4 Low: 2 Informational: 0

# Severity Finding Status
1 Medium Export fails for undeployed projects with default volume selection
backend/internal/project/service.go:1814 · correctness
✅ Fixed 607660297
2 Medium Browser holds the whole bundle in memory before saving
frontend/src/lib/services/upload-service.ts:97 · reliability
✅ Fixed 607660297
3 Medium Server writes the whole bundle twice on disk during export
backend/internal/project/service.go:1800 · performance
✅ Fixed 607660297
4 Medium Import of a bundle with an empty volume fails and rolls back
backend/internal/project/service.go:1900 · correctness
✅ Fixed 402e0ea9b
5 Low CLI export silently drops volumes on an older server
cli/pkg/projects/actions.go:388 · correctness
✅ Fixed 607660297
6 Low Long exports can outrun the 30-minute proxy and CLI timeouts
cli/pkg/projects/actions.go:370 · reliability
📝 Reported
1 · Export fails for undeployed projects with default volume selection
  • Scenario: A project that was never deployed (or whose volumes were removed) is exported with the dialog defaults or --all-volumes; ExportVolumeArchive returns NotFound and no archive is produced.
  • Evidence: ExportOptions returned ClassifyProjectVolumes from the compose model with named volumes included; writeBundle aborted on the first ExportVolume error.
  • Fix: The preview checks each named volume exists and marks missing ones not included (missing); the dialog disables them, --all-volumes skips them, and an explicitly requested missing volume is rejected before stopping the project or writing anything. (607660297)
  • Fix check: Pass: existence uses the compose-resolved names; nothing is stopped or written for missing volumes (tested).
2 · Browser holds the whole bundle in memory before saving
  • Scenario: Exporting a project with a 20 GB volume collects every chunk as a Blob before saving; Safari/iOS run out of memory and the export is wasted.
  • Evidence: downloadSession pushed each chunk Blob into an array and built one Blob at the end; no streaming route existed for upload sessions.
  • Fix: New streaming GET /agents/{id}/uploads/{kind}/{uploadId}/download (same permission as chunk reads, agent allowlist entry), removing the session once fully sent; the dialog downloads through downloadFromUrl. (607660297)
  • Fix check: Pass: deletion only after a complete read; traversal-safe IDs; HEAD/aborts keep the session.
3 · Server writes the whole bundle twice on disk during export
  • Scenario: 40 GB of compressed volume data with 60 GB free: the temp bundle plus its byte copy into the upload root fills the disk.
  • Evidence: ingestExport wrote a temp file in the projects directory and IngestSession copied it with WriteFrom.
  • Fix: UploadService.WriteSession streams straight into the session's data file; plain and bundle export use it. (607660297)
  • Fix check: Pass: metadata matches CreateSession plus all chunks, so transfer's chunk relay still works (tested).
4 · Import of a bundle with an empty volume fails and rolls back
  • Scenario: Found in the evidence run: exporting evidence-shop with an empty cache volume and importing the bundle failed with "uploaded archive appears empty or invalid", removing the imported project.
  • Evidence: An empty volume exports as an archive with only ./; the volume restore refuses empty uploads on purpose (restore/service.go ~562).
  • Fix: Import detects an empty volume archive and creates the Compose-labelled volume directly, tracked for rollback. (402e0ea9b)
  • Fix check: Rerun against a build of 402e0ea: both volumes recreated (tests added for detection and for skipping the restore).
Low and informational (2)
  • 5 · CLI export silently drops volumes on an older server (cli/pkg/projects/actions.go:388)
    • Scenario: arcane projects export web --volume db against a 3.1.0 server returns a plain archive (Huma ignores the body) and the CLI reports the volume as included.
    • Evidence: The CLI skipped the preview and never checked the returned filename.
    • Fix: The CLI requires the .arcane.tar.gz bundle name when volumes were requested, deleting the session and failing otherwise. (607660297)
    • Fix check: Pass: project names are sanitized, so a plain archive can't end in that suffix.
  • 6 · Long exports can outrun the 30-minute proxy and CLI timeouts (cli/pkg/projects/actions.go:370)
    • Scenario: A multi-GB export on a remote agent passes 30 minutes; the client errors while the agent finishes and leaves a session until the 24h purge.
    • Evidence: Export runs inside the POST; ProxyResponseHeaderTimeout and the CLI timeout are 30 minutes; the activity context outlives the request.
    • 📝 Reported: Making export a background job is a larger change; noted in the PR description.

🤖 Claude Opus 5.5 (High) · T3 Code (Claude Code) · Round 1 of 3

Import was a menu action, which folded Update Projects into an Actions
menu with it and broke the E2E tests that click Update Projects. Import
is now a secondary button, so the projects header keeps its existing
buttons.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
neurekadev added a commit to neurekadev/arcane that referenced this pull request Nov 18, 2026
@neurekadev
neurekadev marked this pull request as ready for review November 18, 2026 23:29
@neurekadev
neurekadev requested a review from a team November 18, 2026 23:29
Comment thread backend/internal/project/service.go Resolved
Comment thread backend/internal/project/service.go Resolved
Comment thread backend/internal/project/service.go Resolved
Comment thread backend/internal/project/service.go Resolved
Comment thread cli/internal/cmdutil/upload.go Resolved
Comment on lines +131 to +155
// ClassifyProjectVolumes sorts a project's Compose volumes by key and includes only named volumes by default;
// external volumes and driver_opts volumes keep their data outside Docker's volume storage.
func ClassifyProjectVolumes(project types.Project) []types.ProjectVolume {
classified := make([]types.ProjectVolume, 0, len(project.Volumes))
for _, key := range slices.Sorted(maps.Keys(project.Volumes)) {
config := project.Volumes[key]
kind := types.ProjectVolumeNamed
switch {
case bool(config.External):
kind = types.ProjectVolumeExternal
case len(config.DriverOpts) > 0:
kind = types.ProjectVolumeDriverOpts
}
classified = append(classified, types.ProjectVolume{
Labels: config.Labels,
DriverOpts: config.DriverOpts,
Key: key,
Name: config.Name,
Kind: kind,
Driver: config.Driver,
Included: kind == types.ProjectVolumeNamed,
})
}
return classified
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

external volumes can still live in dockers volumes, this is how i do it, cause i hate the way compose names volumes.

…d import

Import checks the volume names the imported project actually uses before
restoring anything, and only rolls back volumes it created, so a volume
that existed before can no longer be deleted. Empty volumes keep the
archive root's owner and mode.

Export with stop only stops the project's running containers, holds the
project lock while volumes are copied, and starts exactly those
containers again instead of redeploying the whole project.

The CLI downloads into a temporary file and only replaces the output once
the download completes, so a failed re-export keeps the previous bundle;
pipes such as /dev/stdout are still written directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment on lines +1980 to +1983
running := slices.DeleteFunc(found.Items, func(c container.Summary) bool { return c.State != container.StateRunning })
switch {
case len(running) == 0:
return stopped, nil

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Restarting containers escape export stops. stopRunningContainers drops every container whose state is not running, including containers waiting for Docker to restart them. Such a container can start writing again while the volumes are copied. Both passes skip it, so an export with stopProject can still return a bundle marked consistent: true.

Stop containers with a pending restart too, or refuse the export before copying their data.

Knowledge Base Used: Containers, environments, and workloads

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/internal/project/service.go
Line: 1980-1983

Comment:
**Restarting containers escape export stops.** `stopRunningContainers` drops every container whose state is not `running`, including containers waiting for Docker to restart them. Such a container can start writing again while the volumes are copied. Both passes skip it, so an export with `stopProject` can still return a bundle marked `consistent: true`.

Stop containers with a pending restart too, or refuse the export before copying their data.

**Knowledge Base Used:** [Containers, environments, and workloads](https://app.greptile.com/-/switch-tenant?tenant=getarcaneapp&next=%2Fcustom-context%2Fknowledge-base%2Fgetarcaneapp%2Farcane%2F-%2Fdocs%2Fworkload-management.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code Fix in Cursor

Comment on lines +2006 to +2012
var errs []error
for _, id := range containerIDs {
if _, startErr := dockerClient.ContainerStart(ctx, id, client.ContainerStartOptions{}); startErr != nil {
errs = append(errs, fmt.Errorf("start container %s: %w", id, startErr))
}
}
return errors.Join(errs...)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Export ignores service startup order. startContainers starts containers in the same order returned by Docker, without checking Compose dependencies or waiting for them to become healthy. For a project whose app uses depends_on with service_healthy, the app can start before its database and exit. ContainerStart can still succeed, so the export reports success while leaving the app down.

Restore only the previously running containers, but honor their dependency order and readiness checks.

Knowledge Base Used: Containers, environments, and workloads

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/internal/project/service.go
Line: 2006-2012

Comment:
**Export ignores service startup order.** `startContainers` starts containers in the same order returned by Docker, without checking Compose dependencies or waiting for them to become healthy. For a project whose app uses `depends_on` with `service_healthy`, the app can start before its database and exit. `ContainerStart` can still succeed, so the export reports success while leaving the app down.

Restore only the previously running containers, but honor their dependency order and readiness checks.

**Knowledge Base Used:** [Containers, environments, and workloads](https://app.greptile.com/-/switch-tenant?tenant=getarcaneapp&next=%2Fcustom-context%2Fknowledge-base%2Fgetarcaneapp%2Farcane%2F-%2Fdocs%2Fworkload-management.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code Fix in Cursor

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants