Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 107 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ the first or has to be started by hand with different settings.
| One stack per machine, fixed host ports | One stack per worktree, a port set per worktree |
| Writes into the checkout (`pods/`, mounted sources) | Writes nothing into the worktree; sources are bind-mounted by your compose file |
| Pods, targets and overlays | Compose profiles and extra compose files |
| Mints a Vault token per service | Checks that the variables in `require_env` are set and passes them through |
| Mints a Vault token per service | Mints one Vault token shared by every stack, and checks `require_env` |
| Build and deploy path for production | Local development and testing only |

cage remains the deploy tool. aviary replaces it only for running a repository
Expand Down Expand Up @@ -166,16 +166,59 @@ path, so a worktree at the same path gets the same ports on every machine.
aviary skips ports another stack holds or something on the machine is
listening on. A worktree keeps its ports until `aviary rm` or `aviary gc`.

### Required environment
### Required environment, Vault tokens, and host files

```yaml
vault:
addr: https://vault.example.com
ttl: 8h

host_env:
GOOGLE_CREDENTIALS_HOST_PATH:
path: ~/.config/gcloud/application_default_credentials.json
required: true
hint: ask the user to run `gcloud auth application-default login`

require_env: [VAULT_TOKEN]
```

Commands that create containers (`up`, `test`, `init`, `run`, `build`, and
`aviary compose up` or `run`) refuse to start while any of these is unset or
empty, and name every missing one. The host environment passes through to
compose, so compose files can interpolate them.
`aviary compose up` or `run`) resolve credentials in this order before calling
compose:

1. **`vault`**: a `VAULT_TOKEN` already in the environment wins. When
`~/.vault-token` also exists, aviary first checks the variable with
`lookup-self` and falls back to `~/.vault-token` if Vault rejects it, since
agent shells often inherit stale exports. Otherwise aviary mints a
non-renewable child token from `~/.vault-token`, caches it under
`$XDG_STATE_HOME/aviary/vault-token.json` (`0600`), and reuses it across
worktrees and shells until half of its lease has passed. aviary trusts the
cached token's recorded expiry without asking Vault, so commands work
offline while it lasts; after `vault token revoke`, delete the cache file.
The child token has every policy your login has, and Vault caps its lease
at your login's remaining time, so aviary warns when Vault grants less
than `ttl`. `VAULT_ADDR` is set to `vault.addr`, overriding the shell's.
`VAULT_TOKEN` is passed to compose subprocesses and never printed by
`aviary env`.
2. **`host_env`**: each unset variable is filled with the expanded `path` when
that file exists on the host. When `required: true` and the file is missing,
aviary stops before touching Docker and prints `hint`.
3. **`require_env`**: refuses to start while any listed variable is still unset
or empty, naming every missing one.

### Prepare steps

```yaml
prepare:
- command: [node, scripts/copy-api-artifacts.mts, --all]
```

Commands that run code from the worktree (`up`, `init`, `run`, and any suite
without `prepare: false`) run `prepare` on the host in the worktree root first,
so gitignored build/codegen inputs exist in fresh worktrees without an extra
step. The steps run at most once per command and never in two aviary commands
in one worktree at once. Init runs them even for a suite with
`prepare: false`, because init steps may need their output.

### Definitions outside the worktree

Expand Down Expand Up @@ -231,19 +274,68 @@ suites:
needs: [postgres]
command: [npm, run, test:migrations]
init: false
worker-host:
host: true
workdir: src/worker
init: false
needs: [postgres]
env:
TEST_DATABASE_URL: postgresql://postgres@127.0.0.1:${AVIARY_PORT_POSTGRES}/postgres
command: [uv, run, pytest]
```

`aviary test api`:
`aviary test api` prints one line first, naming the suite and whether init will run or was skipped. The first init in a worktree can take several minutes. Then it:

1. runs the init steps, each as `docker compose run --rm <service> <command>`,
1. runs `prepare` on the host in the worktree root (skipped with `prepare: false`);
2. runs the init steps, each as `docker compose run --rm <service> <command>`,
if init has not run for this stack yet (skipped with `init: false`);
2. starts `postgres` and `redis` and waits until they are healthy;
3. runs `npm test` in a fresh `api` container.
3. starts `postgres` and `redis` and waits until they are healthy;
4. runs `npm test` in a fresh `api` container (or directly on the host in
`workdir` when `host: true`).

A suite's `env` is interpolated against the stack's `AVIARY_*` variables and
resolved credentials, then the host environment; `$$` is a literal `$`. For
container suites the values reach compose through its environment, not its
command line, so they never show up in `ps`.

Arguments after the suite name, with or without a `--`, are appended to its
command, so `aviary test api -- --grep users` and `aviary test api --help`
reach the test runner. aviary exits with the suite's status. `aviary test`
alone lists the suites; `aviary test -h` prints the command's help.
reach the test runner. When a suite sets `workdir` (either a `host: true` suite
or a container suite whose working directory mounts that package), path
arguments pointing inside `workdir` from the worktree root or caller directory
are rewritten relative to `workdir` (`aviary test api -- src/api/test/foo.ts`
passes `test/foo.ts`), preserving `:line` and `::test` selectors. aviary exits
with the suite's status. `aviary test` alone lists the suites; `aviary test -h`
prints the command's help.

### Volume snapshots

When seeding a database takes minutes, `snapshot` caches the seeded volumes
under `$XDG_CACHE_HOME/aviary/snapshots` (`~/.cache/aviary/snapshots` by
default) so later worktrees restore them in seconds:

```yaml
after_restore:
- service: migrate
command: [rake, db:migrate]
snapshot:
volumes: [postgres]
key_files: [db/seeds.rb, db/reference_data.sql]
stamp_files: [db/structure.sql]
```

The snapshot key hashes the volume names, the images mounting them, the `init`
steps, and every file or directory in `key_files`. The stamp hashes
`stamp_files`. On `init` (unless `aviary init --force` is passed):

- **Exact `(key, stamp)` hit**: stops the stack's containers, restores the
volumes, and skips both `init` and `after_restore`.
- **Same `key`, different `stamp`** (when `after_restore` is non-empty):
restores the newest cached snapshot for `key`, runs `after_restore`, stops
the stack to flush database state, and saves the migrated snapshot under the
new `stamp`.
- **Cache miss** (or `aviary init --force`): runs `init`, stops the stack to
flush database state, and saves the snapshot under `(key, stamp)`.

Init runs once per stack, so a new worktree runs it from scratch. Step times
go to stderr. A failing step stops the sequence, names the step, and leaves
Expand Down Expand Up @@ -289,6 +381,9 @@ Run from anywhere inside a worktree. `aviary <command> --help` describes each.
- `aviary down` stops it and keeps its volumes; `--volumes` removes them too.
- `aviary test <suite> [args...]` runs a suite; with no suite it lists them.
- `aviary init [--force]` runs the init steps by hand.
- `aviary pull [service...]` pulls the stack's images across every profile
(using `gcloud` Application Default Credentials automatically when present so
GCR pulls never hang on an interactive reauth prompt).
- `aviary run <service> [command...]` runs a one-off command in a fresh
container and exits with its status.
- `aviary exec <service> <command...>` runs a command in a running container.
Expand Down Expand Up @@ -460,7 +555,7 @@ and those of similar compose wrappers, map like this:
| `exec`, `shell` | `aviary exec <service> <command...>` |
| `test` | `aviary test <suite>`, with the suites named in `aviary.yml` |
| `logs`, `build` | `aviary logs`, `aviary build` |
| `pull` | `aviary compose pull` |
| `pull` | `aviary pull [service...]` |
| init, seed | `aviary init`; `aviary down --volumes` first to start from empty volumes |
| source mount, unmount | none: the compose files bind-mount the worktree's sources, so each stack always runs its own code |
| targets, environments | more files under `compose.files`, or [profiles](#profiles) |
Expand Down
117 changes: 117 additions & 0 deletions cmd/aviary/integration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,123 @@ func TestMinimalExampleEndToEnd(t *testing.T) {
}
}

func TestSnapshotSaveRestoreAndAfterRestoreEndToEnd(t *testing.T) {
if out, err := exec.Command("docker", "compose", "version").CombinedOutput(); err != nil {
t.Fatalf("the integration test needs docker compose: %v\n%s", err, out)
}

tmp := t.TempDir()
bin := filepath.Join(tmp, "aviary")
if out, err := exec.Command("go", "build", "-o", bin, ".").CombinedOutput(); err != nil {
t.Fatalf("go build: %v\n%s", err, out)
}
env := append(withoutAviaryVars(os.Environ()),
"XDG_STATE_HOME="+filepath.Join(tmp, "state"),
"XDG_CACHE_HOME="+filepath.Join(tmp, "cache"),
)

const aviaryYml = `compose:
files: [compose.yaml]
ports: [postgres, redis]
init:
- service: client
command: [psql, -f, /src/seed.sql]
after_restore:
- service: client
command: [psql, -f, /src/migrate.sql]
snapshot:
volumes: [postgres]
key_files: [seed.sql]
stamp_files: [schema.sql]
suites:
db:
service: client
needs: [postgres]
command: [psql, -tAc, "SELECT count(*) FROM seeded"]
`

makeWorktree := func(name, schema, migrate string) string {
dir := filepath.Join(tmp, name)
if err := os.CopyFS(dir, os.DirFS(filepath.Join("..", "..", "examples", "minimal"))); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "aviary.yml"), []byte(aviaryYml), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "seed.sql"), []byte("CREATE TABLE seeded (id int); INSERT INTO seeded VALUES (1), (2);\n"), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "schema.sql"), []byte(schema), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "migrate.sql"), []byte(migrate), 0o644); err != nil {
t.Fatal(err)
}
return dir
}

runIn := func(dir string, args ...string) (stdout, stderr string) {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()
cmd := exec.CommandContext(ctx, bin, args...)
cmd.Dir = dir
cmd.Env = env
var outBuf, errBuf bytes.Buffer
cmd.Stdout = &outBuf
cmd.Stderr = &errBuf
if err := cmd.Run(); err != nil {
t.Fatalf("aviary %s in %s: %v\nstdout:\n%s\nstderr:\n%s", strings.Join(args, " "), dir, err, &outBuf, &errBuf)
}
return outBuf.String(), errBuf.String()
}

wt1 := makeWorktree("snap1", "-- v1\n", "SELECT 1;\n")
wt2 := makeWorktree("snap2", "-- v1\n", "SELECT 1;\n")
wt3 := makeWorktree("snap3", "-- v2\n", "INSERT INTO seeded VALUES (3);\n")

for _, dir := range []string{wt1, wt2, wt3} {
out, _ := runIn(dir, "env", "--json")
var e map[string]string
if err := json.Unmarshal([]byte(out), &e); err != nil {
t.Fatal(err)
}
proj := e["AVIARY_PROJECT"]
t.Cleanup(func() { _ = compose.DownProject(proj) })
}

// First worktree seeds from scratch and saves the snapshot.
out1, err1 := runIn(wt1, "test", "db")
if got := lastLine(out1); got != "2" {
t.Fatalf("wt1 test db = %q, want 2", got)
}
if !strings.Contains(err1, "saved snapshot") {
t.Errorf("wt1 stderr missing saved snapshot:\n%s", err1)
}

// Second worktree on the same schema restores without running init steps.
out2, err2 := runIn(wt2, "test", "db")
if got := lastLine(out2); got != "2" {
t.Fatalf("wt2 test db = %q, want 2", got)
}
if !strings.Contains(err2, "restoring snapshot") || strings.Contains(err2, "init: client psql") {
t.Errorf("wt2 stderr should restore snapshot without running init:\n%s", err2)
}

// Third worktree on a newer schema restores and runs after_restore.
out3, err3 := runIn(wt3, "test", "db")
if got := lastLine(out3); got != "3" {
t.Fatalf("wt3 test db = %q, want 3", got)
}
if !strings.Contains(err3, "after_restore") {
t.Errorf("wt3 stderr should run after_restore:\n%s", err3)
}

for _, dir := range []string{wt1, wt2, wt3} {
runIn(dir, "rm")
}
}

// withoutAviaryVars drops variables that would point the binary at a
// different definition or registry than the test's.
func withoutAviaryVars(environ []string) []string {
Expand Down
Loading
Loading