Repository navigation
feat(cli): Incus workspace deployments (M4) - #5430
Open
yannickmonney wants to merge 6 commits into
Open
yannickmonney wants to merge 6 commits into
yannickmonney wants to merge 6 commits into
Conversation
A snapshot on Incus is a set of native volume snapshots `tale-<id>`,
taken under one freeze of the volumes' running users. The freeze,
snapshot and thaw calls of a set go out in parallel per phase, and a
user the freeze took from healthy is awaited until it is again. A set
holds data volumes only; `backups`, a restore's holders and the
CLI-owned volumes are refused (D-28, D-28a, R3-16, R-07).
Manifest v2 adds per-volume `native {snapshot, createdAt, volume,
usedBytes, quotaBytes}` and the freeze window, with the archive fields
optional only beside `native`; v1 manifests still read. On Incus a
manifest is written as `manifest.<gen>.json`, read back, and the older
generations deleted; readers take the newest that parses (R3-29).
Docker keeps writing `manifest.json` v1.
`tale backup --archive` and `tale backup export <id> [--to file]`
write today's tar.gz of each volume from a copy of its snapshot through
the helper, owners kept as numbers. `tale backup import <file>` streams
the archives back (Docker: a helper cat; Incus: the volume file API in
pieces, HTTP range reads out), checking each sha256 on arrival.
`tale restore` on Incus never rolls back in place (D-29, R3-08, R3-22):
pool space and the project's pool quota are checked for every copy
before anything changes, `restore-pending` is journaled with the
devices and the users, then per volume: stop, detach under the ETag,
rename to `<v>.pre-restore-<ts>` (the manifests follow the history to
the holder, R-06), copy the snapshot (or extract the archive) into a
fresh `v` with the snapshot's own size, attach, start. `--resume`
finishes a crash at any step and `--abort` undoes it. A volume whose
native snapshot is gone (a reset took it) restores from its archive.
Rotation deletes expired native snapshots, archives, sets a crash left
without any manifest file and holders no kept manifest names; `tale backup prune` lists
holders and deletes only unreferenced ones; verify checks native
existence and the freeze window beside the archives' sha256.
`reset --all` and `uninstall --purge` on Incus archive the newest set
into `backups` first, or refuse without `--discard-snapshots`, with
runtime-specific confirmation text, and the reset records `resetAt`
(R-05). StackRuntime gains `snapshotKind`, `snapshot`,
`restoreSnapshot`, `readVolumeFile` and `writeVolumeFile`; Docker's
argv is unchanged and it refuses the native verbs. `@tale/incus` gains
a range read and the volume usage read, and its fake both.
A workspace that declares the Incus runtime now deploys on it (design 7.2, 7.3). The selection reads tale.json's runtime.incus and the workspace's .tale/incus credentials; the stack is the project id and its state lives in the workspace. tale deploy converges scope by scope: images and volumes (a missing data volume of a stack that ran is refused, R3-22), the resolvers and the switch (stop-gated, R-02), the stateful tier behind the downtime gate with the spawner drained, then the flip. The idle colour comes up in its slots and takes traffic once the switch admits it; every replica is healthy; admit removes a stale marker; the old colour is withdrawn by its markers before its drain, stopped and deleted, and the switch is never reloaded. The flip is journaled by phase, so an interrupted one resumes. --services rolls the live colour one replica at a time; --plan-only and --plan plan every scope; the node is checked against the stack contract first. tale rollback runs the same flip toward the previous release; an interrupted deploy flip still starting turns toward it, and one past its admission is refused until tale deploy finishes it. tale stop and tale start stop and start the whole environment in the three stop groups on both runtimes (Docker: docker stop -t <grace>). tale status shows each replica's admission and a service stopped after repeated crashes with the console lines Incus kept; migrate and auth reset-owner address the serving colour. Each instance carries its healthcheck, so a runtime that compiled nothing judges it, and the spawner gets the zone tale incus bootstrap made.
tale runtime migrate --to incus moves a workspace's Docker deployment to the Incus runtime tale.json declares in runtime.next (design 14). Each step is journaled in .tale/migration.json, so an interrupted run resumes after the last one it finished: tale incus check with its forwarding probes (a FORWARD drop is refused with the DOCKER-USER rules first), the CLI deploying the version Docker runs, then the drains, every Docker container stopped with docker stop (the proxy too), a Docker archive, export, the Incus images and volumes made, import (sha256 each way), restore into fresh volumes, the same version deployed with the same .env once no running container publishes 80 or 443, the public smoke checks, and runtime.engine flipped to incus. The Docker stack stays stopped. --to docker deletes the Incus instances (their volumes stay), puts tale.json and Docker's colour state back and starts the Docker stack. --finish removes the stopped Docker stack after a confirmation and keeps its backups volume. tale deploy, start and rollback refuse while a migration is under way. runtime.next is part of the runtime declaration; tale incus bootstrap, check and contract read it on a Docker workspace, and bootstrap writes its endpoint there instead of flipping the engine. A loopback site is smoke-checked at the Incus edge's own address, since NAT proxy devices take no loopback traffic.
The CLI Incus E2E gains the workspace suites of design 15.5 and its r2 and r3 additions. They share a workspace of their own on the lab node (tools/cli/e2e/incus/workspace.ts), deployed by the compiled CLI built twice from the checkout, as releases N and N+1 (a workspace deploys the release its CLI was built as, and a rollback needs two a patch apart); images.sh tags the stack's images by version for it (TAG). - flip: twenty flips under HTTP, server-sent event streams, a 1 Hz /api/health poller and a sandbox session's calls, all but the session through main's edge in front of the switch: no 5xx, edge 502/503/504, maintenance page or UPSTREAM_UNAVAILABLE; by serving identity, the retired colour serves nothing new after its withdrawal window; a stream ends only at its colour's retirement and reconnects at once. - streams: a stream opened on the new colour survives the rest of the flip and an in-place roll of the other role; a roll of its own role ends one replica's streams at a time. - infra-skew: N+1's CLI defers N's switch and resolvers (same instance, image and upstream files); --stop replaces them, files unchanged. - services-rollback, resume (the CLI killed while a colour comes up, right after its admission, with the old colour withdrawn, and in a roll), withdraw-restart (root's markers, a restart outside the CLI). - backup, reset-keeps-backups, restore-history (the CLI killed after each restore step: deploy and start refuse and make no volume, --resume and --abort converge). - reboot (incus admin shutdown) and migrate (a Docker deployment on the same host, DOCKER-USER rules, a dockerd restart) run only on a lab that is the run's own: the CI runner sets TALE_E2E_HOST_RESTART and TALE_E2E_RESTART_DOCKERD. Main's switched roles serve no WebSocket (the 0.5 backend replaced the Convex socket with HTTP and the /events hint stream), so the streams are server-sent events. Defects the lab found, fixed here: - An in-place roll left the switch without a reachable replica for about 20 s: a replica made again keeps its slot's address with a new MAC, and the switch's neighbour cache kept the old one. The applier now flushes that entry in the switch and waits until the switch's own admission probe passes before the next replica is withdrawn (workspace and managed hooks). - tale backup, restore and reset on a workspace refused every one-shot job: the runtime knew the helper image only from a deploy. It now takes the image the stack's edge runs. - tale restore refused to list after reset --all on Incus: the backups volume alone now names the namespace. - A snapshot right after a restore failed on a volume whose usage Incus answers without `used` (omitted when zero, as for a fresh ZFS copy); it now reads as 0. The workflow runs the eleven suites after the managed ones and before secrets, with a longer job timeout.
The install, operate and configuration pages still said a workspace on Incus is refused and that backups do not work there. They now describe the runtime as M4 built it, in English, German and French. - install/incus: the workspace path (tale init --runtime incus, bootstrap into .tale/, tale deploy with --plan-only and --plan), tale logs --consume, tale stop and tale start, the Docker FORWARD remedies (DOCKER-USER first; ip-forward-no-drop only after a reboot), and a full guide to moving a workspace between Docker and Incus (runtime.next, tale runtime migrate, --to docker, --finish, another host, back to Docker by export and import). The limits and the verified scope follow the suites' results. - operate/backups-and-restore: native snapshot sets and --archive, retention, restore aside under a pending restore (--resume, --abort, about twice the space, checked first), backup prune, export and import across hosts and runtimes, and what reset --all and uninstall --purge keep on each runtime (--discard-snapshots). - operate/upgrades: the Incus flip (admission probe, no reload, /api/health unchanged, streams reconnect to the new colour), the larger stop-gated tier, --plan-only, tale stop before a shared host's shutdown. - configuration/tls-and-domains: the proxy-device and routed ingress for ports 80 and 443; environment-reference: TALE_INCUS_DIR's readers. Quickstart and the install index point to the runtime. - install/cli-install's command reference: backup --archive, backup export/import/prune, restore --resume/--abort and --discard-snapshots on reset and uninstall; the prerequisites name an Incus workspace. - tools/cli/README.md and the compose.yml header cover the workspace path, stop/start, backup export/import/prune, restore --resume/--abort and runtime migrate.
CI runs every CLI test file in one Bun process, in path order, and mock.module is process-global. color-lifecycle.test.ts replaced get-current-color with a spy its afterEach reset to answer undefined, and never put it back; incus-deploy.test.ts runs right after it, so every Incus deploy built a flip record with retiring: undefined and the journal schema refused it (16 failures). A run that ordered the files otherwise passed. Move rollback.test.ts's restoring mockModule into a shared fixture, createModuleMocks, and use it in both suites. A test runs the two suites in CI's order in one process and fails without the fix.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone M4 of the Tale-on-Incus program, on the stable line (main, 0.5.x). A workspace that declares the Incus runtime now deploys, flips colours, backs up, restores and migrates on Incus. The branch sits on main at 137ef55 (#5429) and has one commit per lane.
Commits
feat(cli): snapshot and restore stack volumes on Incusbackup --archive/export/import/prune.restoreon Incus copies to a side volume and never rolls back in place (--resume,--abort). Rotation.reset --allanduninstall --purgearchive the newest set or refuse. Proof: unit tests (backup, actions, runtime, commands,@tale/incus), with mutations caught by the adversarial review, and a real script run on lab Incus that reported ALL CHECKS PASSED (freeze, archive, restore aside, a crash after rename then--resume, export and import with owners kept, rotation).feat(cli): deploy and flip colours on Incustale deployon an Incus workspace converges scope by scope: images and volumes, resolvers and switch, the stop-gated stateful tier, then the journaled blue/green flip through switch admission with no reload. Also--servicesin-place rolls,--plan-only/--plan,rollback,stop/starton both runtimes, and admission instatus. Proof: unit tests, and a library-driven deploy on lab Incus.feat(cli): migrate a deployment between Docker and Incustale runtime migrate --to incus(every step journaled and resumable),--to dockerand--finish, plusruntime.nextin bootstrap, check and contract. Proof: a real Docker to Incus migration on lab-vm-3. The org and the file came back on Incus, then on Docker after--to docker, then on Incus again after a second migrate.--finishkept the backups volume. Two defects the lab found are fixed in this commit: bootstrap flipped the engine, and the smoke check could not reach loopback.test(e2e): prove workspace deploys and data safety on Incusdocs: run workspace deployments on Incustools/cli/README.mdand thecompose.ymlheader.Gate (on this branch, after the rebase onto 137ef55)
Run on ops-lab3-ci with Node 22.21.1:
format:check,lint:conflicts,lint:links,lint:manual,knip:checkandlint:sast.@tale/cli generateleaves the tree clean.turbo run lint typecheck test test:ui --continue: 55 of 57 tasks pass.@tale/incuspasses 131 tests with 0 failures.cached site plans preserve Compose overrides, environment interpolation and image identity. The lab has no compose plugin. The cli suite is 4720 pass, 1 fail.backend/testing/harness.test.ts"keep polling ... fakes the timers": a timing test that fails under load. Run alone it passes 14 of 14.The drain-status mismatch TI-18 found (Docker's drain read
inFlight, the spawner answerssessions) is fixed on main by #5429, which this branch is now rebased onto.Unproven items each lane reported
TI-16
deployment/apply.ts,INCUS_APPLY_SENTENCES.snapshot). Lifting it needs the IncusRuntime insideapplyDeployment.tale backupandtale restorewere proven through the library and the real script, not through the compiled CLI on a workspace. TI-23's backup suites cover that path.TI-17
TI-18
TI-23
incus admin shutdownwould stop every project on the shared VM) and migrate (no docker compose plugin, and host port 443 is held by another lab). The CI dispatch run (gh workflow run cli-incus-e2e.yml --ref feat/cli-incus-workspace) is still needed.docker ps -aon ops-lab-ci still shows the foreign containerlab-objects./sandbox/tunnelto the spawner and/ws_api, and neither goes through the switch.Caddyfile.switchhas none.sandbox.e2e.tsasserts that a session cannot reach the proxy.incus execargv. It is a short-lived signature, not the token, andsandbox.e2e.tsalready relays this way.tale backupafterreset --allnow gets past the "no data volumes" check, because the backups volume counts as the namespace. It then refuses increateNativeSnapshotwith "No data volumes found in the Incus project".TI-24
--finishsays only "a Docker workspace that has the same .env and version". It does not say how to get that workspace (a newinit, or settingruntime.engineback), because the code defines no safe in-place way.