From 290290114f78d31e8fe751793291f11f6ba31bf2 Mon Sep 17 00:00:00 2001 From: Amp Date: Fri, 31 Jul 2026 14:49:34 +0000 Subject: [PATCH 01/12] docs: refine spec-driven agent workflow Amp-Thread-ID: https://ampcode.com/threads/T-019fb8a3-1679-72d5-902f-95123721c3bb Co-authored-by: Arjun Komath --- AGENT.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/AGENT.md b/AGENT.md index a7030e83..8c2c940c 100644 --- a/AGENT.md +++ b/AGENT.md @@ -43,6 +43,12 @@ For any requested code or configuration change, work through these phases in order. Do not collapse requirements, specification, and implementation planning into a single step. +- Use subagents where helpful for bounded research, investigation, independent + analysis, and synthesizing findings or answers. +- As each stage is completed, compact the working context into its agreed + deliverable before progressing. Preserve material decisions, constraints, + assumptions, unresolved questions, and risks. + ### 1. Research and refine requirements Understand the problem before designing a solution. From a876cd3b24bc84682654486f3666483a05efe618 Mon Sep 17 00:00:00 2001 From: Arjun Komath Date: Sat, 1 Aug 2026 08:34:04 +1000 Subject: [PATCH 02/12] docs: document safe registry garbage collection Amp-Thread-ID: https://ampcode.com/threads/T-019fb886-b4f9-71a0-94ab-1062b9d8cc5a Co-authored-by: Amp --- docs/infrastructure/registry.mdx | 90 ++++++++++++++++++++++++++++-- registry/README.md | 96 +++++++++++++++++++++++++++++--- 2 files changed, 174 insertions(+), 12 deletions(-) diff --git a/docs/infrastructure/registry.mdx b/docs/infrastructure/registry.mdx index 5c11bd56..8d3f6fdc 100644 --- a/docs/infrastructure/registry.mdx +++ b/docs/infrastructure/registry.mdx @@ -31,16 +31,96 @@ Agents receive registry credentials automatically during [registration](/agents/ Images are stored on the local filesystem in a persistent Docker volume (`registry-data`). Delete operations are enabled for garbage collection. -### Garbage Collection +### Garbage collection -To reclaim disk space from deleted images, run garbage collection manually: +Garbage collection removes blobs that are no longer referenced by a manifest. It does not choose which tagged images to retain. Delete unwanted tags first, then run garbage collection to reclaim their storage. + +Techulus Cloud does not install or schedule registry garbage collection automatically. + +#### Preview garbage collection + +You can run a dry run while the registry is serving traffic: + +```bash +REGISTRY=techulus-cloud-registry-1 + +docker exec "$REGISTRY" \ + /bin/registry garbage-collect \ + --dry-run \ + --delete-untagged \ + /etc/docker/registry/config.yml +``` + +`--delete-untagged` removes untagged manifests so that their unreferenced blobs can also be collected. + +#### Run garbage collection + +Garbage collection must not race with image pushes. The following maintenance script stops the registry while collecting, so the registry is briefly unavailable. It prevents overlapping runs and restarts the registry if garbage collection fails. + +Install it as `/usr/local/sbin/techulus-registry-gc` on the control-plane server: + +```bash +cat >/usr/local/sbin/techulus-registry-gc <<'EOF' +#!/bin/sh +set -eu + +exec 9>/run/lock/techulus-registry-gc.lock +/usr/bin/flock -n 9 || exit 0 + +REGISTRY=${REGISTRY_CONTAINER:-techulus-cloud-registry-1} +IMAGE=$(/usr/bin/docker inspect --format '{{.Config.Image}}' "$REGISTRY") + +restart_registry() { + /usr/bin/docker start "$REGISTRY" >/dev/null 2>&1 || true +} + +trap restart_registry EXIT +trap 'exit 1' HUP INT TERM + +/usr/bin/docker stop "$REGISTRY" +/usr/bin/docker run --rm \ + --volumes-from "$REGISTRY" \ + --entrypoint /bin/registry \ + "$IMAGE" \ + garbage-collect \ + --delete-untagged \ + /etc/docker/registry/config.yml + +restart_registry +trap - EXIT HUP INT TERM +EOF + +chmod 0755 /usr/local/sbin/techulus-registry-gc +``` + +Run it manually with: + +```bash +/usr/local/sbin/techulus-registry-gc +``` + +#### Schedule weekly garbage collection + +To run garbage collection every Sunday at 03:00, create `/etc/cron.d/techulus-registry-gc` manually: + +```cron +SHELL=/bin/sh +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + +0 3 * * 0 root /usr/local/sbin/techulus-registry-gc 2>&1 | /usr/bin/logger -t techulus-registry-gc +``` + +View scheduled-run output with: ```bash -docker exec registry /bin/registry garbage-collect /etc/docker/registry/config.yml +journalctl -t techulus-registry-gc ``` -For automatic cleanup, add a daily cron job: +#### Verify garbage collection + +After garbage collection, wait for the registry to report `healthy` and check its storage usage: ```bash -0 2 * * * docker exec registry /bin/registry garbage-collect /etc/docker/registry/config.yml +docker inspect --format '{{.State.Health.Status}}' techulus-cloud-registry-1 +docker run --rm --volumes-from techulus-cloud-registry-1 alpine du -sh /var/lib/registry ``` diff --git a/registry/README.md b/registry/README.md index 90e67500..9b45af81 100644 --- a/registry/README.md +++ b/registry/README.md @@ -27,19 +27,101 @@ Should only be accessible via WireGuard mesh - not exposed publicly. ## Garbage Collection -Clean up unreferenced image layers to reclaim storage space. +Garbage collection removes blobs that are no longer referenced by a manifest. +It does not choose which tagged images to retain: delete unwanted tags first, +then run garbage collection to reclaim their storage. This repository does not +install or schedule garbage collection automatically. + +### Dry run + +A dry run can run while the registry is serving traffic: -**Dry-run** (see what would be deleted): ```bash -docker exec registry /bin/registry garbage-collect --dry-run /etc/docker/registry/config.yml +REGISTRY=techulus-cloud-registry-1 + +docker exec "$REGISTRY" \ + /bin/registry garbage-collect \ + --dry-run \ + --delete-untagged \ + /etc/docker/registry/config.yml ``` -**Run GC**: +`--delete-untagged` removes untagged manifests so that their unreferenced blobs +can also be collected. + +### Run garbage collection + +Garbage collection must not race with image pushes. The following maintenance +script stops the registry while collecting, so the registry is briefly +unavailable. It also prevents overlapping runs and restarts the registry if +garbage collection fails. + +Install it as `/usr/local/sbin/techulus-registry-gc`: + ```bash -docker exec registry /bin/registry garbage-collect /etc/docker/registry/config.yml +cat >/usr/local/sbin/techulus-registry-gc <<'EOF' +#!/bin/sh +set -eu + +exec 9>/run/lock/techulus-registry-gc.lock +/usr/bin/flock -n 9 || exit 0 + +REGISTRY=${REGISTRY_CONTAINER:-techulus-cloud-registry-1} +IMAGE=$(/usr/bin/docker inspect --format '{{.Config.Image}}' "$REGISTRY") + +restart_registry() { + /usr/bin/docker start "$REGISTRY" >/dev/null 2>&1 || true +} + +trap restart_registry EXIT +trap 'exit 1' HUP INT TERM + +/usr/bin/docker stop "$REGISTRY" +/usr/bin/docker run --rm \ + --volumes-from "$REGISTRY" \ + --entrypoint /bin/registry \ + "$IMAGE" \ + garbage-collect \ + --delete-untagged \ + /etc/docker/registry/config.yml + +restart_registry +trap - EXIT HUP INT TERM +EOF + +chmod 0755 /usr/local/sbin/techulus-registry-gc ``` -**Scheduled GC** (daily at 2 AM via cron): +Run it manually with: + +```bash +/usr/local/sbin/techulus-registry-gc +``` + +### Schedule weekly garbage collection + +To run garbage collection every Sunday at 03:00, create +`/etc/cron.d/techulus-registry-gc` manually: + +```cron +SHELL=/bin/sh +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + +0 3 * * 0 root /usr/local/sbin/techulus-registry-gc 2>&1 | /usr/bin/logger -t techulus-registry-gc +``` + +View scheduled-run output with: + +```bash +journalctl -t techulus-registry-gc +``` + +### Verify + +After garbage collection, wait for the registry to report `healthy` and check +its storage usage: + ```bash -0 2 * * * docker exec registry /bin/registry garbage-collect /etc/docker/registry/config.yml >> /var/log/registry-gc.log 2>&1 +docker inspect --format '{{.State.Health.Status}}' techulus-cloud-registry-1 +docker run --rm --volumes-from techulus-cloud-registry-1 alpine du -sh /var/lib/registry ``` From 66ebe230f14a6f0f1f812660727e58c9ecf31586 Mon Sep 17 00:00:00 2001 From: Amp Date: Fri, 31 Jul 2026 22:34:12 +0000 Subject: [PATCH 03/12] fix command search input font size Amp-Thread-ID: https://ampcode.com/threads/T-019fba2a-4a0d-758e-b433-061969977625 Co-authored-by: Arjun Komath --- web/components/ui/command.tsx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/components/ui/command.tsx b/web/components/ui/command.tsx index 7848260a..968efb14 100644 --- a/web/components/ui/command.tsx +++ b/web/components/ui/command.tsx @@ -74,7 +74,7 @@ function CommandInput({ Date: Sat, 1 Aug 2026 00:07:21 +0000 Subject: [PATCH 04/12] Clean up registry images Amp-Thread-ID: https://ampcode.com/threads/T-019fba5b-5b73-7227-9f59-22ae4c279baa Co-authored-by: Arjun Komath --- agent/internal/agent/handlers.go | 10 +- agent/internal/build/build.go | 76 +++-- agent/internal/build/build_test.go | 39 +++ agent/internal/http/client.go | 5 +- agent/internal/http/client_test.go | 26 ++ docs/infrastructure/registry.mdx | 13 +- registry/Dockerfile | 4 +- registry/README.md | 30 +- registry/config.yml | 2 +- registry/entrypoint.sh | 4 +- web/actions/projects.ts | 192 ++++++++--- web/app/api/inngest/route.ts | 2 + .../api/v1/agent/builds/[id]/status/route.ts | 91 +++-- web/db/schema.ts | 1 + web/lib/inngest/functions/build-workflow.ts | 28 +- web/lib/inngest/functions/crons.ts | 15 + web/lib/inngest/functions/index.ts | 1 + .../functions/service-deletion-workflow.ts | 89 ++++- web/lib/registry-retention.ts | 316 ++++++++++++++++++ web/lib/service-revisions.ts | 17 +- web/tests/build-status-route.test.ts | 99 +++++- web/tests/build-workflow.test.ts | 27 +- web/tests/inngest-route.test.ts | 1 + web/tests/registry-retention.test.ts | 295 ++++++++++++++++ web/tests/service-revision-build.test.ts | 20 +- 25 files changed, 1237 insertions(+), 166 deletions(-) create mode 100644 web/lib/registry-retention.ts create mode 100644 web/tests/registry-retention.test.ts diff --git a/agent/internal/agent/handlers.go b/agent/internal/agent/handlers.go index 5dd68cb0..8d1c528d 100644 --- a/agent/internal/agent/handlers.go +++ b/agent/internal/agent/handlers.go @@ -148,7 +148,7 @@ func (a *Agent) ProcessBuild(item agenthttp.WorkQueueItem) error { } log.Printf("[build] starting build %s for commit %s (timeout: %d minutes)", Truncate(payload.BuildID, 8), Truncate(buildDetails.Build.CommitSha, 8), timeoutMinutes) - if err := a.Client.UpdateBuildStatus(payload.BuildID, "cloning", "", ""); err != nil { + if err := a.Client.UpdateBuildStatus(payload.BuildID, "cloning", "", "", ""); err != nil { log.Printf("[build] failed to update status to cloning: %v", err) } @@ -185,24 +185,24 @@ func (a *Agent) ProcessBuild(item agenthttp.WorkQueueItem) error { } onStatusChange := func(status string) { - if err := a.Client.UpdateBuildStatus(payload.BuildID, status, "", buildConfig.ResolvedCommitSha); err != nil { + if err := a.Client.UpdateBuildStatus(payload.BuildID, status, "", buildConfig.ResolvedCommitSha, ""); err != nil { log.Printf("[build] failed to update status to %s: %v", status, err) } } ctx, cancel := context.WithTimeout(context.Background(), time.Duration(timeoutMinutes)*time.Minute) defer cancel() - err = a.Builder.Build(ctx, buildConfig, checkCancelled, onStatusChange) + artifact, err := a.Builder.Build(ctx, buildConfig, checkCancelled, onStatusChange) if err != nil { log.Printf("[build] build %s failed: %v", Truncate(payload.BuildID, 8), err) - if updateErr := a.Client.UpdateBuildStatus(payload.BuildID, "failed", err.Error(), buildConfig.ResolvedCommitSha); updateErr != nil { + if updateErr := a.Client.UpdateBuildStatus(payload.BuildID, "failed", err.Error(), buildConfig.ResolvedCommitSha, ""); updateErr != nil { log.Printf("[build] failed to update status to failed: %v", updateErr) } return err } log.Printf("[build] build %s completed successfully", Truncate(payload.BuildID, 8)) - if err := a.Client.UpdateBuildStatus(payload.BuildID, "completed", "", buildConfig.ResolvedCommitSha); err != nil { + if err := a.Client.UpdateBuildStatus(payload.BuildID, "completed", "", buildConfig.ResolvedCommitSha, artifact); err != nil { log.Printf("[build] failed to update status to completed: %v", err) } diff --git a/agent/internal/build/build.go b/agent/internal/build/build.go index e54ed907..6a99aaa8 100644 --- a/agent/internal/build/build.go +++ b/agent/internal/build/build.go @@ -5,6 +5,7 @@ import ( "context" "crypto/sha256" "encoding/hex" + "encoding/json" "errors" "fmt" "log" @@ -59,6 +60,7 @@ type dockerfileConfig struct { var managedTempArtifactPattern = regexp.MustCompile(`^(backup|restore)-[0-9a-fA-F-]{36}\.tar\.gz$|^restore-extract-[0-9a-fA-F-]{36}$`) var windowsAbsoluteRootPattern = regexp.MustCompile(`^[A-Za-z]:[\\/]`) +var imageDigestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) func NewBuilder(dataDir string, logSender LogSender) *Builder { return &Builder{ @@ -67,11 +69,11 @@ func NewBuilder(dataDir string, logSender LogSender) *Builder { } } -func (b *Builder) Build(ctx context.Context, config *Config, checkCancelled func() bool, onStatusChange func(status string)) error { +func (b *Builder) Build(ctx context.Context, config *Config, checkCancelled func() bool, onStatusChange func(status string)) (string, error) { buildDir := filepath.Join(b.dataDir, "builds", config.BuildID) if err := os.MkdirAll(buildDir, 0755); err != nil { - return fmt.Errorf("failed to create build directory: %w", err) + return "", fmt.Errorf("failed to create build directory: %w", err) } defer func() { @@ -80,31 +82,27 @@ func (b *Builder) Build(ctx context.Context, config *Config, checkCancelled func }() if checkCancelled() { - return fmt.Errorf("build cancelled") + return "", fmt.Errorf("build cancelled") } if err := b.clone(ctx, config, buildDir); err != nil { - return fmt.Errorf("clone failed: %w", err) - } - - if config.CommitSha == "HEAD" && config.ImageRepository != "" && config.ResolvedCommitSha != "" { - config.ImageURI = fmt.Sprintf("%s:%s", config.ImageRepository, config.ResolvedCommitSha) - b.sendLog(config, fmt.Sprintf("Resolved image tag %s", config.ImageURI)) + return "", fmt.Errorf("clone failed: %w", err) } if checkCancelled() { - return fmt.Errorf("build cancelled") + return "", fmt.Errorf("build cancelled") } if onStatusChange != nil { onStatusChange("building") } - if err := b.buildAndPush(ctx, config, buildDir); err != nil { - return fmt.Errorf("build failed: %w", err) + artifact, err := b.buildAndPush(ctx, config, buildDir) + if err != nil { + return "", fmt.Errorf("build failed: %w", err) } - return nil + return artifact, nil } func truncateStr(s string, maxLen int) string { @@ -224,10 +222,10 @@ func (b *Builder) resolveCommitSha(ctx context.Context, config *Config, buildDir return resolvedCommitSha, nil } -func (b *Builder) buildAndPush(ctx context.Context, config *Config, buildDir string) error { +func (b *Builder) buildAndPush(ctx context.Context, config *Config, buildDir string) (string, error) { contextDir, err := resolveBuildContext(buildDir, config.RootDir) if err != nil { - return err + return "", err } if config.RootDir != "" { b.sendLog(config, fmt.Sprintf("Using root directory: %s", config.RootDir)) @@ -235,8 +233,12 @@ func (b *Builder) buildAndPush(ctx context.Context, config *Config, buildDir str dockerfile, err := resolveDockerfile(contextDir, config.Secrets) if err != nil { - return err + return "", err + } + if config.ImageRepository == "" { + return "", fmt.Errorf("image repository is required") } + metadataPath := filepath.Join(buildDir, "build-metadata.json") buildkitAddr := os.Getenv("BUILDKIT_HOST") if buildkitAddr == "" { @@ -257,10 +259,7 @@ func (b *Builder) buildAndPush(ctx context.Context, config *Config, buildDir str if len(config.TargetPlatforms) > 0 { platform = config.TargetPlatforms[0] } - arch := strings.Split(platform, "/")[1] - - archImageUri := config.ImageURI + "-" + arch - archOutputFlag := fmt.Sprintf("type=image,name=%s,push=true,registry.insecure=true", archImageUri) + outputFlag := fmt.Sprintf("type=image,name=%s,push=true,push-by-digest=true,registry.insecure=true", config.ImageRepository) if dockerfile.found { log.Printf("[build:%s] building with Dockerfile via buildctl for %s", truncateStr(config.BuildID, 8), platform) @@ -269,7 +268,7 @@ func (b *Builder) buildAndPush(ctx context.Context, config *Config, buildDir str } else { b.sendLog(config, "Using existing Dockerfile") } - b.sendLog(config, fmt.Sprintf("Building and pushing %s", archImageUri)) + b.sendLog(config, fmt.Sprintf("Building and pushing %s", config.ImageRepository)) args := []string{ "--addr", buildkitAddr, @@ -279,7 +278,8 @@ func (b *Builder) buildAndPush(ctx context.Context, config *Config, buildDir str "--local", fmt.Sprintf("dockerfile=%s", dockerfile.directory), "--opt", fmt.Sprintf("filename=%s", dockerfile.filename), "--opt", fmt.Sprintf("platform=%s", platform), - "--output", archOutputFlag, + "--output", outputFlag, + "--metadata-file", metadataPath, } args = append(args, secretArgs...) @@ -290,7 +290,7 @@ func (b *Builder) buildAndPush(ctx context.Context, config *Config, buildDir str if err != nil { log.Printf("[build:%s] buildctl failed with output: %s", truncateStr(config.BuildID, 8), output) b.sendLog(config, fmt.Sprintf("Build error: %s", output)) - return fmt.Errorf("buildctl build failed:\n%s", tailLines(output, 20)) + return "", fmt.Errorf("buildctl build failed:\n%s", tailLines(output, 20)) } } else { log.Printf("[build:%s] building with Railpack via buildctl for %s", truncateStr(config.BuildID, 8), platform) @@ -308,7 +308,7 @@ func (b *Builder) buildAndPush(ctx context.Context, config *Config, buildDir str if err != nil { log.Printf("[build:%s] railpack prepare failed with output: %s", truncateStr(config.BuildID, 8), output) b.sendLog(config, fmt.Sprintf("Railpack prepare error: %s", output)) - return fmt.Errorf("railpack prepare failed:\n%s", tailLines(output, 20)) + return "", fmt.Errorf("railpack prepare failed:\n%s", tailLines(output, 20)) } b.sendLog(config, fmt.Sprintf("Building for %s...", platform)) @@ -322,7 +322,8 @@ func (b *Builder) buildAndPush(ctx context.Context, config *Config, buildDir str "--local", "dockerfile=.", "--opt", "filename=railpack-plan.json", "--opt", fmt.Sprintf("platform=%s", platform), - "--output", archOutputFlag, + "--output", outputFlag, + "--metadata-file", metadataPath, } secretsHash := computeSecretsHash(config.Secrets) @@ -338,12 +339,33 @@ func (b *Builder) buildAndPush(ctx context.Context, config *Config, buildDir str if err != nil { log.Printf("[build:%s] buildctl failed for %s: %s", truncateStr(config.BuildID, 8), platform, output) b.sendLog(config, fmt.Sprintf("Build error (%s): %s", platform, output)) - return fmt.Errorf("buildctl build failed for %s:\n%s", platform, tailLines(output, 20)) + return "", fmt.Errorf("buildctl build failed for %s:\n%s", platform, tailLines(output, 20)) } } + digest, err := readImageDigest(metadataPath) + if err != nil { + return "", err + } b.sendLog(config, "Build completed") - return nil + return fmt.Sprintf("%s@%s", config.ImageRepository, digest), nil +} + +func readImageDigest(metadataPath string) (string, error) { + data, err := os.ReadFile(metadataPath) + if err != nil { + return "", fmt.Errorf("failed to read build metadata: %w", err) + } + var metadata struct { + Digest string `json:"containerimage.digest"` + } + if err := json.Unmarshal(data, &metadata); err != nil { + return "", fmt.Errorf("failed to parse build metadata: %w", err) + } + if !imageDigestPattern.MatchString(metadata.Digest) { + return "", fmt.Errorf("build metadata contains invalid containerimage.digest") + } + return metadata.Digest, nil } func resolveBuildContext(buildDir, rootDir string) (string, error) { diff --git a/agent/internal/build/build_test.go b/agent/internal/build/build_test.go index 380676d1..8f5b9bb6 100644 --- a/agent/internal/build/build_test.go +++ b/agent/internal/build/build_test.go @@ -258,6 +258,45 @@ func TestResolveDockerfileFallsBackToRailpack(t *testing.T) { } } +func TestReadImageDigest(t *testing.T) { + digest := "sha256:" + strings.Repeat("a", 64) + tests := []struct { + name string + content string + want string + }{ + {name: "valid", content: `{"containerimage.digest":"` + digest + `"}`, want: digest}, + {name: "missing digest", content: `{}`}, + {name: "wrong algorithm", content: `{"containerimage.digest":"sha512:` + strings.Repeat("a", 64) + `"}`}, + {name: "wrong length", content: `{"containerimage.digest":"sha256:abc"}`}, + {name: "non hex", content: `{"containerimage.digest":"sha256:` + strings.Repeat("g", 64) + `"}`}, + {name: "malformed json", content: `{`}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + path := filepath.Join(t.TempDir(), "metadata.json") + if err := os.WriteFile(path, []byte(tt.content), 0600); err != nil { + t.Fatal(err) + } + got, err := readImageDigest(path) + if tt.want == "" { + if err == nil { + t.Fatalf("readImageDigest() = %q, want error", got) + } + return + } + if err != nil || got != tt.want { + t.Fatalf("readImageDigest() = %q, %v, want %q, nil", got, err, tt.want) + } + }) + } + + if _, err := readImageDigest(filepath.Join(t.TempDir(), "missing.json")); err == nil { + t.Fatal("readImageDigest() missing file: want error") + } +} + func runGit(t *testing.T, args ...string) string { t.Helper() output, err := exec.Command("git", args...).CombinedOutput() diff --git a/agent/internal/http/client.go b/agent/internal/http/client.go index ee2d3c7b..7e96243e 100644 --- a/agent/internal/http/client.go +++ b/agent/internal/http/client.go @@ -391,7 +391,7 @@ func (c *Client) ClaimBuild(buildID string) (*BuildDetails, error) { return &result, nil } -func (c *Client) UpdateBuildStatus(buildID, status, errorMsg, resolvedCommitSha string) error { +func (c *Client) UpdateBuildStatus(buildID, status, errorMsg, resolvedCommitSha, imageURI string) error { payload := map[string]string{ "status": status, } @@ -401,6 +401,9 @@ func (c *Client) UpdateBuildStatus(buildID, status, errorMsg, resolvedCommitSha if resolvedCommitSha != "" { payload["resolvedCommitSha"] = resolvedCommitSha } + if imageURI != "" { + payload["imageUri"] = imageURI + } body, err := json.Marshal(payload) if err != nil { diff --git a/agent/internal/http/client_test.go b/agent/internal/http/client_test.go index 3ad5c145..41c89322 100644 --- a/agent/internal/http/client_test.go +++ b/agent/internal/http/client_test.go @@ -1,9 +1,11 @@ package http import ( + "encoding/json" "io" stdhttp "net/http" "net/http/httptest" + "strings" "testing" "techulus/cloud-agent/internal/crypto" @@ -38,3 +40,27 @@ func TestSignedJSONRequests(t *testing.T) { t.Fatalf("unexpected error response: %v", err) } } + +func TestUpdateBuildStatusImageURI(t *testing.T) { + keyPair, err := crypto.GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + digestURI := "registry.example/repository@sha256:" + strings.Repeat("a", 64) + server := httptest.NewServer(stdhttp.HandlerFunc(func(w stdhttp.ResponseWriter, r *stdhttp.Request) { + var payload map[string]string + if err := json.NewDecoder(r.Body).Decode(&payload); err != nil { + t.Error(err) + } + if payload["status"] != "completed" || payload["imageUri"] != digestURI { + t.Errorf("unexpected payload: %#v", payload) + } + w.WriteHeader(stdhttp.StatusOK) + })) + defer server.Close() + + client := NewClient(server.URL, "server-1", keyPair, "") + if err := client.UpdateBuildStatus("build-1", "completed", "", "commit-sha", digestURI); err != nil { + t.Fatal(err) + } +} diff --git a/docs/infrastructure/registry.mdx b/docs/infrastructure/registry.mdx index 8d3f6fdc..b3fb465d 100644 --- a/docs/infrastructure/registry.mdx +++ b/docs/infrastructure/registry.mdx @@ -35,7 +35,7 @@ Images are stored on the local filesystem in a persistent Docker volume (`regist Garbage collection removes blobs that are no longer referenced by a manifest. It does not choose which tagged images to retain. Delete unwanted tags first, then run garbage collection to reclaim their storage. -Techulus Cloud does not install or schedule registry garbage collection automatically. +Registry 3 is required because its `--delete-untagged` behavior safely preserves manifests referenced by retained multi-platform indexes. Failed or interrupted builds can leave digest-only artifacts until the next weekly collection (between 0 and 7 days). Techulus Cloud does not install or schedule garbage collection automatically. #### Preview garbage collection @@ -48,14 +48,14 @@ docker exec "$REGISTRY" \ /bin/registry garbage-collect \ --dry-run \ --delete-untagged \ - /etc/docker/registry/config.yml + /etc/distribution/config.yml ``` `--delete-untagged` removes untagged manifests so that their unreferenced blobs can also be collected. #### Run garbage collection -Garbage collection must not race with image pushes. The following maintenance script stops the registry while collecting, so the registry is briefly unavailable. It prevents overlapping runs and restarts the registry if garbage collection fails. +Garbage collection must not race with image pushes. The following maintenance script stops the registry while collecting, so the registry is briefly unavailable. It prevents overlapping runs and restarts the registry if garbage collection fails. Builds overlapping this window may fail, and transient platform manifests not yet referenced by an index can be collected; retry the build after maintenance. Snapshot registry storage before the first Registry 3 collection. Install it as `/usr/local/sbin/techulus-registry-gc` on the control-plane server: @@ -84,7 +84,7 @@ trap 'exit 1' HUP INT TERM "$IMAGE" \ garbage-collect \ --delete-untagged \ - /etc/docker/registry/config.yml + /etc/distribution/config.yml restart_registry trap - EXIT HUP INT TERM @@ -118,9 +118,12 @@ journalctl -t techulus-registry-gc #### Verify garbage collection -After garbage collection, wait for the registry to report `healthy` and check its storage usage: +After garbage collection, wait for the registry to report `healthy`, pull a known retained multi-platform tag on every supported platform, inspect its index, and check storage usage: ```bash docker inspect --format '{{.State.Health.Status}}' techulus-cloud-registry-1 +docker pull --platform linux/amd64 REGISTRY/IMAGE:TAG +docker pull --platform linux/arm64 REGISTRY/IMAGE:TAG +docker buildx imagetools inspect REGISTRY/IMAGE:TAG docker run --rm --volumes-from techulus-cloud-registry-1 alpine du -sh /var/lib/registry ``` diff --git a/registry/Dockerfile b/registry/Dockerfile index 31bcf66a..b9b2174a 100644 --- a/registry/Dockerfile +++ b/registry/Dockerfile @@ -1,8 +1,8 @@ -FROM registry:2 +FROM registry:3.1.1 RUN apk add --no-cache apache2-utils curl -COPY config.yml /etc/docker/registry/config.yml +COPY config.yml /etc/distribution/config.yml COPY entrypoint.sh /entrypoint.sh RUN chmod +x /entrypoint.sh diff --git a/registry/README.md b/registry/README.md index 9b45af81..6e012cf1 100644 --- a/registry/README.md +++ b/registry/README.md @@ -29,8 +29,14 @@ Should only be accessible via WireGuard mesh - not exposed publicly. Garbage collection removes blobs that are no longer referenced by a manifest. It does not choose which tagged images to retain: delete unwanted tags first, -then run garbage collection to reclaim their storage. This repository does not -install or schedule garbage collection automatically. +then run garbage collection to reclaim their storage. Registry 3 is required +because its `--delete-untagged` behavior safely preserves manifests referenced +by retained multi-platform indexes. + +Failed or interrupted builds can leave digest-only artifacts. These remain +until the next weekly garbage-collection run (between 0 and 7 days). This +repository does not install or schedule garbage collection automatically; the +offline weekly cron remains a manual host setup step. ### Dry run @@ -43,7 +49,7 @@ docker exec "$REGISTRY" \ /bin/registry garbage-collect \ --dry-run \ --delete-untagged \ - /etc/docker/registry/config.yml + /etc/distribution/config.yml ``` `--delete-untagged` removes untagged manifests so that their unreferenced blobs @@ -54,7 +60,12 @@ can also be collected. Garbage collection must not race with image pushes. The following maintenance script stops the registry while collecting, so the registry is briefly unavailable. It also prevents overlapping runs and restarts the registry if -garbage collection fails. +garbage collection fails. Builds that overlap this offline window may fail. +Transient platform manifests that have been pushed but are not yet referenced +by a multi-platform index can be collected; retry the build after maintenance. + +Snapshot the registry storage volume before the first Registry 3 garbage +collection run. Install it as `/usr/local/sbin/techulus-registry-gc`: @@ -83,7 +94,7 @@ trap 'exit 1' HUP INT TERM "$IMAGE" \ garbage-collect \ --delete-untagged \ - /etc/docker/registry/config.yml + /etc/distribution/config.yml restart_registry trap - EXIT HUP INT TERM @@ -118,10 +129,15 @@ journalctl -t techulus-registry-gc ### Verify -After garbage collection, wait for the registry to report `healthy` and check -its storage usage: +After garbage collection, wait for the registry to report `healthy`. Pull a +known retained multi-platform tag on every supported platform and verify its +index still references the expected platform manifest digests. Finally, check +storage usage: ```bash docker inspect --format '{{.State.Health.Status}}' techulus-cloud-registry-1 +docker pull --platform linux/amd64 REGISTRY/IMAGE:TAG +docker pull --platform linux/arm64 REGISTRY/IMAGE:TAG +docker buildx imagetools inspect REGISTRY/IMAGE:TAG docker run --rm --volumes-from techulus-cloud-registry-1 alpine du -sh /var/lib/registry ``` diff --git a/registry/config.yml b/registry/config.yml index b3e8f468..882989ef 100644 --- a/registry/config.yml +++ b/registry/config.yml @@ -20,7 +20,7 @@ http: auth: htpasswd: realm: basic-realm - path: /etc/docker/registry/htpasswd + path: /etc/distribution/htpasswd health: storagedriver: diff --git a/registry/entrypoint.sh b/registry/entrypoint.sh index 60a1c2e8..3dbb198b 100644 --- a/registry/entrypoint.sh +++ b/registry/entrypoint.sh @@ -2,7 +2,7 @@ set -e if [ -n "$REGISTRY_USERNAME" ] && [ -n "$REGISTRY_PASSWORD" ]; then - htpasswd -Bbn "$REGISTRY_USERNAME" "$REGISTRY_PASSWORD" > /etc/docker/registry/htpasswd + htpasswd -Bbn "$REGISTRY_USERNAME" "$REGISTRY_PASSWORD" > /etc/distribution/htpasswd fi -exec registry serve /etc/docker/registry/config.yml +exec /bin/registry serve /etc/distribution/config.yml diff --git a/web/actions/projects.ts b/web/actions/projects.ts index cdb8d75c..e20c8e95 100644 --- a/web/actions/projects.ts +++ b/web/actions/projects.ts @@ -2,7 +2,16 @@ import { randomUUID } from "node:crypto"; import cronstrue from "cronstrue"; -import { and, desc, eq, inArray, isNotNull, isNull, sql } from "drizzle-orm"; +import { + and, + desc, + eq, + inArray, + isNotNull, + isNull, + or, + sql, +} from "drizzle-orm"; import { revalidatePath } from "next/cache"; import { ZodError, z } from "zod"; import { db } from "@/db"; @@ -40,6 +49,10 @@ import { validateDockerImageInternal } from "@/lib/docker-image"; import { inngest } from "@/lib/inngest/client"; import { inngestEvents } from "@/lib/inngest/events"; import { allocatePort } from "@/lib/port-allocation"; +import { + cleanupRegistryArtifactsForService, + prepareRegistryArtifactCleanup, +} from "@/lib/registry-retention"; import { containerPathSchema, githubRepoUrlSchema, @@ -129,24 +142,27 @@ export async function deleteProject( } } - await deleteBackupsForServices(projectServices.map((service) => service.id)); - await db.delete(projects).where(eq(projects.id, id)); - return { success: true }; -} - -async function deleteBackupsForServices(serviceIds: string[]) { - if (serviceIds.length === 0) { - return; - } - - const backups = await db - .select({ id: volumeBackups.id }) - .from(volumeBackups) - .where(inArray(volumeBackups.serviceId, serviceIds)); - - for (const backup of backups) { - await deleteBackup(backup.id, { revalidate: false }); + for (const service of projectServices) { + await hardDeleteService(service.id); } + await db.transaction(async (tx) => { + const locked = await tx.execute( + sql`select id from projects where id = ${id} for update`, + ); + if (locked.rows.length === 0) throw new Error("Project not found"); + const remainingServices = await tx + .select({ id: services.id }) + .from(services) + .where(eq(services.projectId, id)) + .limit(1); + if (remainingServices.length > 0) { + throw new Error( + "Project services changed during deletion; retry deletion", + ); + } + await tx.delete(projects).where(eq(projects.id, id)); + }); + return { success: true }; } export async function updateProjectName(projectId: string, name: string) { @@ -240,8 +256,26 @@ export async function deleteEnvironment(environmentId: string) { .from(services) .where(eq(services.environmentId, environmentId)); - await deleteBackupsForServices(envServices.map((service) => service.id)); - await db.delete(environments).where(eq(environments.id, environmentId)); + for (const service of envServices) { + await hardDeleteService(service.id); + } + await db.transaction(async (tx) => { + const locked = await tx.execute( + sql`select id from environments where id = ${environmentId} for update`, + ); + if (locked.rows.length === 0) throw new Error("Environment not found"); + const remainingServices = await tx + .select({ id: services.id }) + .from(services) + .where(eq(services.environmentId, environmentId)) + .limit(1); + if (remainingServices.length > 0) { + throw new Error( + "Environment services changed during deletion; retry deletion", + ); + } + await tx.delete(environments).where(eq(environments.id, environmentId)); + }); return { success: true }; } @@ -368,14 +402,51 @@ export async function createService(input: CreateServiceInput) { } async function hardDeleteService(serviceId: string) { - const service = await db - .select() - .from(services) - .where(eq(services.id, serviceId)) - .then((r) => r[0]); + const service = await db.transaction(async (tx) => { + await tx.execute(sql`select pg_advisory_xact_lock(hashtext(${serviceId}))`); + const freshService = await tx + .select() + .from(services) + .where( + and( + eq(services.id, serviceId), + or( + isNull(services.deletionStatus), + eq(services.deletionStatus, "failed"), + ), + ), + ) + .then((rows) => rows[0]); + if (!freshService) return undefined; + const now = new Date(); + const claimed = await tx + .update(services) + .set({ + deletedAt: now, + purgeAfter: now, + deletionStatus: "deleting", + deletionError: null, + }) + .where(eq(services.id, serviceId)) + .returning() + .then((rows) => rows[0]); + if (!claimed) return undefined; + return { + service: claimed, + registryCleanupReady: await prepareRegistryArtifactCleanup(tx, serviceId), + }; + }); if (!service) { - throw new Error("Service not found"); + throw new Error( + "Service not found or another service operation is in progress", + ); + } + if (!service.registryCleanupReady) { + throw new Error( + "Service deletion deferred while registry manifest work is processing", + ); } + const claimedService = service.service; const allDeployments = await db .select() @@ -397,14 +468,14 @@ async function hardDeleteService(serviceId: string) { await db.delete(deployments).where(eq(deployments.serviceId, serviceId)); - if (service.stateful && service.lockedServerId) { + if (claimedService.stateful && claimedService.lockedServerId) { const volumes = await db .select() .from(serviceVolumes) .where(eq(serviceVolumes.serviceId, serviceId)); if (volumes.length > 0) { - await enqueueWork(service.lockedServerId, "cleanup_volumes", { + await enqueueWork(claimedService.lockedServerId, "cleanup_volumes", { serviceId, }); } @@ -419,6 +490,7 @@ async function hardDeleteService(serviceId: string) { await deleteBackup(backup.id, { revalidate: false }); } + await cleanupRegistryArtifactsForService(serviceId); await db.delete(secrets).where(eq(secrets.serviceId, serviceId)); await db.delete(services).where(eq(services.id, serviceId)); @@ -507,10 +579,29 @@ export async function deleteService( } } - await db - .update(services) - .set({ deletionStatus: "backing_up", deletionError: null }) - .where(eq(services.id, serviceId)); + const claimed = await db.transaction(async (tx) => { + await tx.execute(sql`select pg_advisory_xact_lock(hashtext(${serviceId}))`); + return tx + .update(services) + .set({ deletionStatus: "backing_up", deletionError: null }) + .where( + and( + eq(services.id, serviceId), + isNull(services.deletedAt), + or( + isNull(services.deletionStatus), + eq(services.deletionStatus, "failed"), + ), + ), + ) + .returning({ id: services.id }) + .then((rows) => rows[0]); + }); + if (!claimed) { + throw new Error( + "Deletion cannot start while another service operation is in progress", + ); + } try { await inngest.send( @@ -618,14 +709,35 @@ export async function restoreDeletedService(serviceId: string) { } } - await db - .update(services) - .set({ - deletionStatus: "restoring", - deletionError: null, - lockedServerId: targetServerId ?? service.lockedServerId, - }) - .where(eq(services.id, serviceId)); + await db.transaction(async (tx) => { + await tx.execute(sql`select pg_advisory_xact_lock(hashtext(${serviceId}))`); + const current = await tx + .select({ id: services.id }) + .from(services) + .where( + and( + eq(services.id, serviceId), + isNotNull(services.deletedAt), + or( + isNull(services.deletionStatus), + eq(services.deletionStatus, "failed"), + ), + ), + ) + .limit(1) + .then((rows) => rows[0]); + if (!current) { + throw new Error("A deletion or restore operation is already in progress"); + } + await tx + .update(services) + .set({ + deletionStatus: "restoring", + deletionError: null, + lockedServerId: targetServerId ?? service.lockedServerId, + }) + .where(eq(services.id, serviceId)); + }); try { await inngest.send( diff --git a/web/app/api/inngest/route.ts b/web/app/api/inngest/route.ts index fbc85d0c..5e209019 100644 --- a/web/app/api/inngest/route.ts +++ b/web/app/api/inngest/route.ts @@ -13,6 +13,7 @@ import { oldBackupsCleanup, onDeploymentFailed, onRestoreFailed, + registryArtifactRetention, restoreTriggerWorkflow, restoreWorkflow, rolloutWorkflow, @@ -38,6 +39,7 @@ export const { GET, POST, PUT } = serve({ staleItemsCleanup, controlPlaneUpdateCheck, agentUpgradeTimeoutCheck, + registryArtifactRetention, migrationWorkflow, backupWorkflow, restoreWorkflow, diff --git a/web/app/api/v1/agent/builds/[id]/status/route.ts b/web/app/api/v1/agent/builds/[id]/status/route.ts index e5af4d82..ac7e80f8 100644 --- a/web/app/api/v1/agent/builds/[id]/status/route.ts +++ b/web/app/api/v1/agent/builds/[id]/status/route.ts @@ -1,4 +1,4 @@ -import { and, eq, inArray } from "drizzle-orm"; +import { and, eq, inArray, isNull, sql } from "drizzle-orm"; import { type NextRequest, NextResponse } from "next/server"; import { db } from "@/db"; import { @@ -21,6 +21,7 @@ type StatusUpdate = { status: "cloning" | "building" | "pushing" | "completed" | "failed"; error?: string; resolvedCommitSha?: string; + imageUri?: string; }; const validStatuses = new Set([ @@ -45,17 +46,16 @@ const transitionSources: Record = { failed: ["pending", "claimed", "cloning", "building", "pushing"], }; -function platformImageForTarget(finalImage: string, targetPlatform: string) { - const [operatingSystem, architecture, ...extra] = targetPlatform.split("/"); - if ( - operatingSystem !== "linux" || - !architecture || - extra.length > 0 || - !["amd64", "arm64"].includes(architecture) - ) { - throw new Error(`Invalid build target platform: ${targetPlatform}`); - } - return `${finalImage}-${architecture}`; +function imageRepository(image: string) { + const withoutDigest = image.split("@", 1)[0]; + const lastSlash = withoutDigest.lastIndexOf("/"); + const tag = withoutDigest.lastIndexOf(":"); + return tag > lastSlash ? withoutDigest.slice(0, tag) : withoutDigest; +} + +function digestImageRepository(image: string) { + const match = /^(.*)@sha256:[0-9a-f]{64}$/.exec(image); + return match?.[1] ?? null; } async function sendBuildCompletedEvent(data: { @@ -169,20 +169,24 @@ export async function POST( let platformImageUri: string | null = null; if (update.status === "completed") { - try { - platformImageUri = platformImageForTarget( - specification.image, - build.targetPlatform, + if (!update.imageUri || !digestImageRepository(update.imageUri)) { + return NextResponse.json( + { error: "Completed build requires a valid image digest" }, + { status: 400 }, ); - } catch (error) { + } + if ( + digestImageRepository(update.imageUri) !== + imageRepository(specification.image) + ) { return NextResponse.json( { - error: - error instanceof Error ? error.message : "Invalid build target", + error: "Completed build artifact does not match its service revision", }, - { status: 500 }, + { status: 409 }, ); } + platformImageUri = update.imageUri; } const updateData: Record = { status: update.status }; @@ -351,26 +355,41 @@ export async function POST( groupBuilds.every((candidate) => { if (candidate.status !== "completed") return false; return ( - candidate.imageUri === - platformImageForTarget(specification.image, candidate.targetPlatform) + candidate.imageUri && + digestImageRepository(candidate.imageUri) === + imageRepository(specification.image) ); }); if (allCompleted) { - const images = groupBuilds.map((candidate) => - platformImageForTarget(specification.image, candidate.targetPlatform), - ); - await enqueueWork( - auth.serverId, - "create_manifest", - { - images, - finalImageUri: specification.image, - serviceId: build.serviceId, - serviceRevisionId: build.serviceRevisionId, - buildGroupId: build.buildGroupId, - }, - { id: `manifest-work-${build.buildGroupId}` }, + const images = groupBuilds.map( + (candidate) => candidate.imageUri as string, ); + await db.transaction(async (tx) => { + await tx.execute( + sql`select pg_advisory_xact_lock(hashtext(${build.serviceId}))`, + ); + const activeService = await tx + .select({ id: services.id }) + .from(services) + .where( + and(eq(services.id, build.serviceId), isNull(services.deletedAt)), + ) + .limit(1) + .then((rows) => rows[0]); + if (!activeService) return; + await enqueueWork( + auth.serverId, + "create_manifest", + { + images, + finalImageUri: specification.image, + serviceId: build.serviceId, + serviceRevisionId: build.serviceRevisionId, + buildGroupId: build.buildGroupId, + }, + { id: `manifest-work-${build.buildGroupId}`, tx }, + ); + }); } await sendBuildCompletedEvent({ diff --git a/web/db/schema.ts b/web/db/schema.ts index ca7122ea..e90100d8 100644 --- a/web/db/schema.ts +++ b/web/db/schema.ts @@ -608,6 +608,7 @@ export const serviceRevisions = pgTable( .$type() .notNull(), actor: jsonb("actor").$type(), + artifactDeletedAt: timestamp("artifact_deleted_at", { withTimezone: true }), createdAt: timestamp("created_at", { withTimezone: true }) .defaultNow() .notNull(), diff --git a/web/lib/inngest/functions/build-workflow.ts b/web/lib/inngest/functions/build-workflow.ts index 7b34d478..723b9762 100644 --- a/web/lib/inngest/functions/build-workflow.ts +++ b/web/lib/inngest/functions/build-workflow.ts @@ -29,7 +29,7 @@ const nonTerminalBuildStatuses: BuildStatus[] = [ "pushing", ]; -function platformImageForTarget(finalImage: string, targetPlatform: string) { +function validateTargetPlatform(targetPlatform: string) { const [operatingSystem, architecture, ...extra] = targetPlatform.split("/"); if ( operatingSystem !== "linux" || @@ -39,7 +39,21 @@ function platformImageForTarget(finalImage: string, targetPlatform: string) { ) { throw new Error(`Invalid build target platform: ${targetPlatform}`); } - return `${finalImage}-${architecture}`; +} + +function imageRepository(image: string) { + const withoutDigest = image.split("@", 1)[0]; + const lastSlash = withoutDigest.lastIndexOf("/"); + const tag = withoutDigest.lastIndexOf(":"); + return tag > lastSlash ? withoutDigest.slice(0, tag) : withoutDigest; +} + +function isDigestForRepository(image: string, repository: string) { + return ( + image.startsWith(`${repository}@sha256:`) && + image.length === repository.length + "@sha256:".length + 64 && + /^[0-9a-f]{64}$/.test(image.slice(-64)) + ); } async function getGroupBuilds( @@ -130,18 +144,16 @@ function validateCompletedGroup( manifest: Extract, ) { if (groupBuilds.length === 0) throw new Error("Build group is missing"); + const repository = imageRepository(manifest.finalImageUri); const expectedImages = groupBuilds.map((build) => { if (build.status !== "completed") { throw new Error("Build group is not complete"); } - const expectedImage = platformImageForTarget( - manifest.finalImageUri, - build.targetPlatform, - ); - if (build.imageUri !== expectedImage) { + validateTargetPlatform(build.targetPlatform); + if (!build.imageUri || !isDigestForRepository(build.imageUri, repository)) { throw new Error("Platform build artifact does not match its revision"); } - return expectedImage; + return build.imageUri; }); const expected = [...expectedImages].sort(); const actual = [...manifest.images].sort(); diff --git a/web/lib/inngest/functions/crons.ts b/web/lib/inngest/functions/crons.ts index da5ef12c..c137e4b3 100644 --- a/web/lib/inngest/functions/crons.ts +++ b/web/lib/inngest/functions/crons.ts @@ -5,6 +5,7 @@ import { } from "@/lib/acme-manager"; import { cleanupOldBackups, runScheduledBackups } from "@/lib/backup-scheduler"; import { checkAndPersistControlPlaneUpdate } from "@/lib/control-plane-updates"; +import { cleanupRegistryArtifactsDaily } from "@/lib/registry-retention"; import { checkAndRecoverStaleServers, checkAndRunScheduledDeployments, @@ -156,3 +157,17 @@ export const agentUpgradeTimeoutCheck = inngest.createFunction( }); }, ); + +export const registryArtifactRetention = inngest.createFunction( + { + id: "cron-registry-artifact-retention", + triggers: [cron("0 5 * * *")], + singleton: { mode: "skip" }, + }, + async ({ step }) => { + await step.run("cleanup-registry-artifacts", async () => { + console.log("[cron] cleaning up registry artifacts"); + await cleanupRegistryArtifactsDaily(); + }); + }, +); diff --git a/web/lib/inngest/functions/index.ts b/web/lib/inngest/functions/index.ts index ef862f53..9fbff375 100644 --- a/web/lib/inngest/functions/index.ts +++ b/web/lib/inngest/functions/index.ts @@ -7,6 +7,7 @@ export { challengeCleanup, controlPlaneUpdateCheck, oldBackupsCleanup, + registryArtifactRetention, scheduledBackupsCheck, scheduledDeploymentsCheck, staleItemsCleanup, diff --git a/web/lib/inngest/functions/service-deletion-workflow.ts b/web/lib/inngest/functions/service-deletion-workflow.ts index a7fd41be..847359db 100644 --- a/web/lib/inngest/functions/service-deletion-workflow.ts +++ b/web/lib/inngest/functions/service-deletion-workflow.ts @@ -31,6 +31,10 @@ import { markDeploymentRemoved, observedReadyPhases, } from "@/lib/deployment-status"; +import { + cleanupRegistryArtifactsForService, + prepareRegistryArtifactCleanup, +} from "@/lib/registry-retention"; import { parseServiceRevisionSpec } from "@/lib/service-revision-changes"; import { enqueueWork } from "@/lib/work-queue"; import { inngest } from "../client"; @@ -46,7 +50,16 @@ async function markServiceDeletionFailed(serviceId: string, error: unknown) { deletionError: error instanceof Error ? error.message : "Service operation failed", }) - .where(eq(services.id, serviceId)); + .where( + and( + eq(services.id, serviceId), + isNull(services.deletedAt), + or( + eq(services.deletionStatus, "backing_up"), + eq(services.deletionStatus, "deleting"), + ), + ), + ); } export const serviceDeletionWorkflow = inngest.createFunction( @@ -288,7 +301,7 @@ export const serviceDeletionWorkflow = inngest.createFunction( .limit(1) .then((rows) => rows[0]); if (!current) throw new Error("Service not found"); - await tx + const deleted = await tx .update(services) .set({ deletedAt, @@ -298,7 +311,17 @@ export const serviceDeletionWorkflow = inngest.createFunction( deletionStatus: null, deletionError: null, }) - .where(eq(services.id, serviceId)); + .where( + and( + eq(services.id, serviceId), + isNull(services.deletedAt), + eq(services.deletionStatus, "deleting"), + ), + ) + .returning({ id: services.id }); + if (deleted.length === 0) { + throw new Error("Service deletion ownership was lost"); + } }); }); @@ -645,23 +668,65 @@ export const expiredDeletedServicesPurge = inngest.createFunction( or( isNull(services.deletionStatus), eq(services.deletionStatus, "failed"), + eq(services.deletionStatus, "deleting"), ), lte(services.purgeAfter, new Date()), ), ); for (const service of expiredServices) { - const backups = await db - .select({ id: volumeBackups.id }) - .from(volumeBackups) - .where(eq(volumeBackups.serviceId, service.id)); + try { + const claimed = await db.transaction(async (tx) => { + await tx.execute( + sql`select pg_advisory_xact_lock(hashtext(${service.id}))`, + ); + const claimed = await tx + .update(services) + .set({ deletionStatus: "deleting", deletionError: null }) + .where( + and( + eq(services.id, service.id), + isNotNull(services.deletedAt), + isNotNull(services.purgeAfter), + lte(services.purgeAfter, new Date()), + or( + isNull(services.deletionStatus), + eq(services.deletionStatus, "failed"), + eq(services.deletionStatus, "deleting"), + ), + ), + ) + .returning({ id: services.id }) + .then((rows) => rows[0]); + if (!claimed) return undefined; + return { + ...claimed, + registryCleanupReady: await prepareRegistryArtifactCleanup( + tx, + service.id, + ), + }; + }); + if (!claimed) continue; + if (!claimed.registryCleanupReady) continue; + await cleanupRegistryArtifactsForService(service.id); + const backups = await db + .select({ id: volumeBackups.id }) + .from(volumeBackups) + .where(eq(volumeBackups.serviceId, service.id)); + + for (const backup of backups) { + await deleteBackupInternal(backup.id); + } - for (const backup of backups) { - await deleteBackupInternal(backup.id); + await db.delete(secrets).where(eq(secrets.serviceId, service.id)); + await db.delete(services).where(eq(services.id, service.id)); + } catch (error) { + console.error( + `[service-purge] failed to purge service ${service.id}`, + error, + ); } - - await db.delete(secrets).where(eq(secrets.serviceId, service.id)); - await db.delete(services).where(eq(services.id, service.id)); } }); }, diff --git a/web/lib/registry-retention.ts b/web/lib/registry-retention.ts new file mode 100644 index 00000000..0d5e7a90 --- /dev/null +++ b/web/lib/registry-retention.ts @@ -0,0 +1,316 @@ +import { and, eq, inArray, isNull, sql } from "drizzle-orm"; +import { db } from "@/db"; +import { builds, serviceRevisions, workQueue } from "@/db/schema"; +import { parseServiceRevisionSpec } from "@/lib/service-revision-changes"; + +type RegistryCleanupTransaction = Parameters< + Parameters[0] +>[0]; + +const MANIFEST_ACCEPT = [ + "application/vnd.oci.image.index.v1+json", + "application/vnd.oci.image.manifest.v1+json", + "application/vnd.docker.distribution.manifest.list.v2+json", + "application/vnd.docker.distribution.manifest.v2+json", +].join(", "); +const DAILY_BATCH_SIZE = 100; + +type RevisionArtifact = { + id: string; + serviceId: string; + specification: unknown; + artifactDeletedAt: Date | null; +}; + +type ArtifactCandidate = RevisionArtifact & { image: string }; + +export async function prepareRegistryArtifactCleanup( + tx: RegistryCleanupTransaction, + serviceId: string, +): Promise { + await tx + .update(workQueue) + .set({ status: "failed" }) + .where( + and( + eq(workQueue.type, "create_manifest"), + eq(workQueue.status, "pending"), + sql`${workQueue.payload}::jsonb ->> 'serviceId' = ${serviceId}`, + ), + ); + const processing = await tx + .select({ id: workQueue.id }) + .from(workQueue) + .where( + and( + eq(workQueue.type, "create_manifest"), + eq(workQueue.status, "processing"), + sql`${workQueue.payload}::jsonb ->> 'serviceId' = ${serviceId}`, + ), + ) + .limit(1); + return processing.length === 0; +} + +function registryConfig() { + const rawUrl = process.env.REGISTRY_URL; + const rawHost = process.env.REGISTRY_HOST; + const username = process.env.REGISTRY_USERNAME; + const password = process.env.REGISTRY_PASSWORD; + if (!rawUrl || !rawHost || !username || !password) { + throw new Error("Registry retention requires registry configuration"); + } + const url = new URL( + /^[a-z][a-z\d+.-]*:\/\//i.test(rawUrl) ? rawUrl : `http://${rawUrl}`, + ); + const hostUrl = new URL( + /^[a-z][a-z\d+.-]*:\/\//i.test(rawHost) ? rawHost : `https://${rawHost}`, + ); + if (hostUrl.pathname !== "/" || hostUrl.search || hostUrl.hash) { + throw new Error("REGISTRY_HOST must contain only a registry authority"); + } + return { + url, + host: hostUrl.host.toLowerCase(), + authorization: `Basic ${Buffer.from(`${username}:${password}`).toString("base64")}`, + }; +} + +function managedReference(image: string, registryHost: string) { + const withoutScheme = image.replace(/^[a-z][a-z\d+.-]*:\/\//i, ""); + const slash = withoutScheme.indexOf("/"); + if ( + slash <= 0 || + withoutScheme.slice(0, slash).toLowerCase() !== registryHost + ) { + throw new Error("Malformed or unmanaged registry image reference"); + } + const pathAndReference = withoutScheme.slice(slash + 1); + const digestAt = pathAndReference.lastIndexOf("@"); + const colon = pathAndReference.lastIndexOf(":"); + const separator = digestAt >= 0 ? digestAt : colon; + if ( + separator <= 0 || + (colon < pathAndReference.lastIndexOf("/") && digestAt < 0) + ) { + throw new Error("Malformed or unmanaged registry image reference"); + } + const repository = pathAndReference.slice(0, separator); + const reference = pathAndReference.slice(separator + 1); + if ( + !repository || + repository.split("/").some((part) => !part) || + !reference + ) { + throw new Error("Malformed or unmanaged registry image reference"); + } + return { repository, reference, isDigest: digestAt >= 0 }; +} + +function manifestUrl(registryUrl: URL, repository: string, reference: string) { + const base = registryUrl.toString().replace(/\/$/, ""); + const encodedRepository = repository + .split("/") + .map(encodeURIComponent) + .join("/"); + return `${base}/v2/${encodedRepository}/manifests/${encodeURIComponent(reference)}`; +} + +async function deleteTag( + registryUrl: URL, + authorization: string, + repository: string, + tag: string, +) { + const response = await fetch(manifestUrl(registryUrl, repository, tag), { + method: "DELETE", + headers: { Accept: MANIFEST_ACCEPT, Authorization: authorization }, + }); + if (response.status !== 404 && !response.ok) { + throw new Error(`Registry manifest DELETE failed (${response.status})`); + } +} + +async function deleteArtifactReferences( + revision: RevisionArtifact, + completedBuilds: Array<{ imageUri: string | null }>, +) { + if (revision.artifactDeletedAt) return false; + const specification = parseServiceRevisionSpec(revision.specification); + if (specification.source.type !== "github") return false; + const config = registryConfig(); + const finalReference = managedReference(specification.image, config.host); + if (finalReference.isDigest) { + throw new Error("Managed final image reference must use a tag"); + } + const references = [finalReference]; + for (const build of completedBuilds) { + if (build.imageUri && !build.imageUri.includes("@")) { + const reference = managedReference(build.imageUri, config.host); + if (!reference.isDigest) references.push(reference); + } + } + const unique = references.filter( + (item, index) => + references.findIndex( + (other) => + other.repository === item.repository && + other.reference === item.reference, + ) === index, + ); + for (const item of unique) { + await deleteTag( + config.url, + config.authorization, + item.repository, + item.reference, + ); + } + return true; +} + +export async function cleanupRevisionArtifact(revision: RevisionArtifact) { + const specification = parseServiceRevisionSpec(revision.specification); + if (revision.artifactDeletedAt || specification.source.type !== "github") { + return false; + } + const revisions = await db + .select({ id: serviceRevisions.id }) + .from(serviceRevisions) + .where( + and( + eq(serviceRevisions.serviceId, revision.serviceId), + sql`${serviceRevisions.specification} ->> 'image' = ${specification.image}`, + ), + ); + const revisionIds = revisions.map(({ id }) => id); + const completedBuilds = revisionIds.length + ? await db + .select({ imageUri: builds.imageUri }) + .from(builds) + .where( + and( + inArray(builds.serviceRevisionId, revisionIds), + eq(builds.serviceId, revision.serviceId), + eq(builds.status, "completed"), + ), + ) + : []; + await deleteArtifactReferences(revision, completedBuilds); + await db + .update(serviceRevisions) + .set({ artifactDeletedAt: new Date() }) + .where( + and( + eq(serviceRevisions.serviceId, revision.serviceId), + sql`${serviceRevisions.specification} ->> 'image' = ${specification.image}`, + isNull(serviceRevisions.artifactDeletedAt), + ), + ); + return true; +} + +export async function cleanupRegistryArtifactsDaily() { + const result = await db.execute(sql` + with completed_ranked as ( + select r.service_id, sr.specification ->> 'image' as image, + row_number() over (partition by r.service_id order by r.completed_at desc nulls last, r.created_at desc, r.id desc) as rank + from rollouts r join service_revisions sr on sr.id = r.service_revision_id + where r.status = 'completed' + ) + select min(sr.id) as id, sr.service_id as "serviceId", + min(sr.specification::text)::jsonb as specification, + null::timestamptz as "artifactDeletedAt", sr.specification ->> 'image' as image + from service_revisions sr + join services s on s.id = sr.service_id and s.deleted_at is null + where sr.artifact_deleted_at is null + and sr.specification -> 'source' ->> 'type' = 'github' + and exists (select 1 from rollouts r join service_revisions x on x.id = r.service_revision_id + where x.service_id = sr.service_id and x.specification ->> 'image' = sr.specification ->> 'image' + and r.status in ('failed','rolled_back','completed')) + and not exists (select 1 from rollouts r join service_revisions x on x.id = r.service_revision_id + where x.service_id = sr.service_id and x.specification ->> 'image' = sr.specification ->> 'image' + and r.status in ('queued','in_progress')) + and not exists (select 1 from deployments d join service_revisions x on x.id = d.service_revision_id + where x.service_id = sr.service_id and x.specification ->> 'image' = sr.specification ->> 'image' + and d.runtime_desired_state <> 'removed') + and not exists (select 1 from completed_ranked cr where cr.service_id = sr.service_id + and cr.image = sr.specification ->> 'image' and cr.rank <= 10) + group by sr.service_id, sr.specification ->> 'image' + order by min(sr.id) limit ${DAILY_BATCH_SIZE} + `); + for (const candidate of result.rows) { + try { + await db.transaction(async (tx) => { + await tx.execute( + sql`select pg_advisory_xact_lock(hashtext(${candidate.serviceId}))`, + ); + const eligible = await tx.execute< + RevisionArtifact & { imageUri: string | null } + >(sql` + with completed_ranked as ( + select sr.specification ->> 'image' as image, + row_number() over (order by r.completed_at desc nulls last, r.created_at desc, r.id desc) as rank + from rollouts r join service_revisions sr on sr.id = r.service_revision_id + where r.service_id = ${candidate.serviceId} and r.status = 'completed' + ), artifact_revisions as ( + select sr.* from service_revisions sr join services s on s.id = sr.service_id + where sr.service_id = ${candidate.serviceId} + and sr.specification ->> 'image' = ${candidate.image} + and s.deleted_at is null + ), eligibility as ( + select exists (select 1 from rollouts r join artifact_revisions x on x.id = r.service_revision_id where r.status in ('failed','rolled_back','completed')) + and not exists (select 1 from rollouts r join artifact_revisions x on x.id = r.service_revision_id where r.status in ('queued','in_progress')) + and not exists (select 1 from deployments d join artifact_revisions x on x.id = d.service_revision_id where d.runtime_desired_state <> 'removed') + and not exists (select 1 from completed_ranked where image = ${candidate.image} and rank <= 10) as eligible + ), representative as ( + select e.* from artifact_revisions e cross join eligibility + where eligibility.eligible and e.artifact_deleted_at is null + and e.specification -> 'source' ->> 'type' = 'github' + order by e.id limit 1 + ) + select r.id, r.service_id as "serviceId", r.specification, + r.artifact_deleted_at as "artifactDeletedAt", b.image_uri as "imageUri" + from representative r + left join artifact_revisions e on true + left join builds b on b.service_revision_id = e.id + and b.service_id = r.service_id and b.status = 'completed' + `); + const revision = eligible.rows[0]; + if (!revision) return; + await deleteArtifactReferences(revision, eligible.rows); + await tx.execute(sql`update service_revisions set artifact_deleted_at = now() + where service_id = ${candidate.serviceId} and specification ->> 'image' = ${candidate.image} + and artifact_deleted_at is null`); + }); + } catch (error) { + console.error( + `[registry-retention] failed to clean artifact ${candidate.id}`, + error, + ); + } + } + return result.rows.length; +} + +export async function cleanupRegistryArtifactsForService(serviceId: string) { + const revisions = await db + .select() + .from(serviceRevisions) + .where( + and( + eq(serviceRevisions.serviceId, serviceId), + isNull(serviceRevisions.artifactDeletedAt), + ), + ); + const artifacts = new Map(); + for (const revision of revisions) { + const specification = parseServiceRevisionSpec(revision.specification); + if (specification.source.type !== "github") continue; + const image = specification.image; + if (!artifacts.has(image)) artifacts.set(image, revision); + } + for (const revision of artifacts.values()) { + await cleanupRevisionArtifact(revision); + } +} diff --git a/web/lib/service-revisions.ts b/web/lib/service-revisions.ts index 4ba35cfd..bf670261 100644 --- a/web/lib/service-revisions.ts +++ b/web/lib/service-revisions.ts @@ -321,7 +321,10 @@ export async function createRolloutWithServiceRevision( let overrides: ServiceRevisionSpecOverrides | undefined; if (runtimeBaseRevisionId) { const baseRevision = await tx - .select({ specification: serviceRevisions.specification }) + .select({ + specification: serviceRevisions.specification, + artifactDeletedAt: serviceRevisions.artifactDeletedAt, + }) .from(serviceRevisions) .where( and( @@ -339,6 +342,9 @@ export async function createRolloutWithServiceRevision( if (baseSpecification.source.type !== "github") { throw new Error("GitHub runtime base revision is not a GitHub build"); } + if (baseRevision.artifactDeletedAt) { + throw new Error("Service revision artifact is no longer available"); + } overrides = { image: baseSpecification.image, source: baseSpecification.source, @@ -369,6 +375,12 @@ export async function cloneActiveRevisionAndQueueSystemRollout( ) { return db.transaction(async (tx) => { await tx.execute(sql`select pg_advisory_xact_lock(hashtext(${serviceId}))`); + const activeService = await tx + .select({ id: services.id }) + .from(services) + .where(and(eq(services.id, serviceId), isNull(services.deletedAt))) + .then((rows) => rows[0]); + if (!activeService) throw new Error("Service not found"); const pending = await tx .select({ id: rollouts.id }) .from(rollouts) @@ -451,6 +463,9 @@ export async function createRolloutForServiceRevision( } const specification = parseServiceRevisionSpec(revision.specification); + if (specification.source.type === "github" && revision.artifactDeletedAt) { + throw new Error("Service revision artifact is no longer available"); + } if (specification.image !== artifactImageUri) { throw new Error("Built artifact does not match the service revision"); } diff --git a/web/tests/build-status-route.test.ts b/web/tests/build-status-route.test.ts index b7c1e45f..81bc5a33 100644 --- a/web/tests/build-status-route.test.ts +++ b/web/tests/build-status-route.test.ts @@ -10,6 +10,7 @@ const mocks = vi.hoisted(() => { from: vi.fn(() => query), innerJoin: vi.fn(() => query), where: vi.fn(() => query), + limit: vi.fn(() => query), // biome-ignore lint/suspicious/noThenProperty: Drizzle query builders are awaitable. then: ( resolve: (value: unknown[]) => unknown, @@ -41,6 +42,8 @@ const mocks = vi.hoisted(() => { db: { select: vi.fn(() => selectQuery(selectResults.shift() ?? [])), update: vi.fn(() => updateQuery(updateResults.shift() ?? [])), + execute: vi.fn().mockResolvedValue({ rows: [] }), + transaction: vi.fn(async (callback) => callback(mocks.db)), }, verifyAgentRequest: vi.fn(), enqueueWork: vi.fn(), @@ -74,6 +77,9 @@ import { POST } from "@/app/api/v1/agent/builds/[id]/status/route"; const commitSha = "0123456789abcdef0123456789abcdef01234567"; const finalImage = "registry.test/project-1/service-1:revision-revision-1"; +const repository = "registry.test/project-1/service-1"; +const amd64Image = `${repository}@sha256:${"a".repeat(64)}`; +const arm64Image = `${repository}@sha256:${"b".repeat(64)}`; const specification = { schemaVersion: 2, @@ -121,11 +127,16 @@ function build(status: string, overrides: Record = {}) { }; } -function post(status: string) { +function post( + status: string, + imageUri: string | null | undefined = status === "completed" + ? amd64Image + : undefined, +) { return POST( new Request("http://localhost/api/v1/agent/builds/build-amd64/status", { method: "POST", - body: JSON.stringify({ status, resolvedCommitSha: commitSha }), + body: JSON.stringify({ status, resolvedCommitSha: commitSha, imageUri }), }) as NextRequest, { params: Promise.resolve({ id: "build-amd64" }) }, ); @@ -137,6 +148,8 @@ describe("agent build status transitions", () => { mocks.selectResults.length = 0; mocks.updateResults.length = 0; mocks.updateSets.length = 0; + mocks.db.transaction.mockClear(); + mocks.db.execute.mockClear(); mocks.verifyAgentRequest.mockResolvedValue({ success: true, serverId: "server-1", @@ -186,7 +199,7 @@ describe("agent build status transitions", () => { it("stores completion and the platform artifact atomically", async () => { const completedBuild = build("completed", { - imageUri: `${finalImage}-amd64`, + imageUri: amd64Image, }); const githubSpecification = { ...specification, @@ -209,9 +222,10 @@ describe("agent build status transitions", () => { build("completed", { id: "build-arm64", targetPlatform: "linux/arm64", - imageUri: `${finalImage}-arm64`, + imageUri: arm64Image, }), ], + [{ id: "service-1" }], ); mocks.updateResults.push([completedBuild]); @@ -221,20 +235,20 @@ describe("agent build status transitions", () => { expect(mocks.updateSets).toHaveLength(1); expect(mocks.updateSets[0]).toMatchObject({ status: "completed", - imageUri: `${finalImage}-amd64`, + imageUri: amd64Image, completedAt: expect.any(Date), }); expect(mocks.enqueueWork).toHaveBeenCalledWith( "server-1", "create_manifest", { - images: [`${finalImage}-amd64`, `${finalImage}-arm64`], + images: [amd64Image, arm64Image], finalImageUri: finalImage, serviceId: "service-1", serviceRevisionId: "revision-1", buildGroupId: "group-1", }, - { id: "manifest-work-group-1" }, + { id: "manifest-work-group-1", tx: mocks.db }, ); expect(mocks.createBuildCompleted).toHaveBeenCalledWith( expect.objectContaining({ status: "success" }), @@ -254,6 +268,27 @@ describe("agent build status transitions", () => { ); }); + it("does not enqueue manifest work after the service is deleted", async () => { + const completedBuild = build("completed", { imageUri: amd64Image }); + mocks.selectResults.push( + [build("pushing")], + [{ specification }], + [completedBuild], + [], + ); + mocks.updateResults.push([completedBuild]); + + const response = await post("completed"); + + expect(response.status).toBe(200); + expect(mocks.db.transaction).toHaveBeenCalledOnce(); + expect(mocks.enqueueWork).not.toHaveBeenCalled(); + expect(mocks.createBuildCompleted).toHaveBeenCalledWith( + expect.objectContaining({ status: "success" }), + { id: "build-completed-build-amd64" }, + ); + }); + it("does not overwrite a concurrent cancellation", async () => { mocks.selectResults.push( [build("pushing")], @@ -272,7 +307,7 @@ describe("agent build status transitions", () => { it("rejects reversal of a completed build to failed", async () => { const completedBuild = build("completed", { - imageUri: `${finalImage}-amd64`, + imageUri: amd64Image, }); mocks.selectResults.push( [completedBuild], @@ -287,4 +322,52 @@ describe("agent build status transitions", () => { expect(mocks.enqueueWork).not.toHaveBeenCalled(); expect(mocks.send).not.toHaveBeenCalled(); }); + + it.each([ + ["missing", null], + ["malformed", `${repository}@sha256:ABC`], + ])("rejects a %s completion digest before persistence", async (_case, imageUri) => { + mocks.selectResults.push([build("pushing")], [{ specification }]); + + const response = await post("completed", imageUri); + + expect(response.status).toBe(400); + expect(mocks.db.update).not.toHaveBeenCalled(); + }); + + it("rejects a completion digest for another repository", async () => { + mocks.selectResults.push([build("pushing")], [{ specification }]); + + const response = await post( + "completed", + `registry.test/other/service@sha256:${"c".repeat(64)}`, + ); + + expect(response.status).toBe(409); + expect(mocks.db.update).not.toHaveBeenCalled(); + }); + + it("accepts only an identical digest when replaying completion", async () => { + const completedBuild = build("completed", { imageUri: amd64Image }); + mocks.selectResults.push( + [completedBuild], + [{ specification }], + [completedBuild], + [completedBuild], + ); + mocks.updateResults.push([]); + + expect((await post("completed", amd64Image)).status).toBe(200); + + mocks.selectResults.push( + [completedBuild], + [{ specification }], + [completedBuild], + ); + mocks.updateResults.push([]); + expect( + (await post("completed", `${repository}@sha256:${"d".repeat(64)}`)) + .status, + ).toBe(409); + }); }); diff --git a/web/tests/build-workflow.test.ts b/web/tests/build-workflow.test.ts index d129e7fd..ee047bcc 100644 --- a/web/tests/build-workflow.test.ts +++ b/web/tests/build-workflow.test.ts @@ -43,6 +43,10 @@ vi.mock("@/lib/inngest/events", () => ({ import { buildWorkflow } from "@/lib/inngest/functions/build-workflow"; +function digest(image: string, character = "a") { + return `${image.slice(0, image.lastIndexOf(":"))}@sha256:${character.repeat(64)}`; +} + function invoke(serviceRevisionId: string, buildGroupId: string) { const step = { run: vi.fn(async (_name: string, operation: () => unknown) => operation()), @@ -73,20 +77,23 @@ function completedGroup( buildGroupId: string, image: string, ) { + const repository = image.slice(0, image.lastIndexOf(":")); + const amd64Image = `${repository}@sha256:${"a".repeat(64)}`; + const arm64Image = `${repository}@sha256:${"b".repeat(64)}`; const group = [ { id: `${buildGroupId}-amd64`, status: "completed", serviceRevisionId, targetPlatform: "linux/amd64", - imageUri: `${image}-amd64`, + imageUri: amd64Image, }, { id: `${buildGroupId}-arm64`, status: "completed", serviceRevisionId, targetPlatform: "linux/arm64", - imageUri: `${image}-arm64`, + imageUri: arm64Image, }, ]; mocks.queryResults.push( @@ -99,7 +106,7 @@ function completedGroup( serviceRevisionId, buildGroupId, finalImageUri: image, - images: [`${image}-amd64`, `${image}-arm64`], + images: [amd64Image, arm64Image], }), }, ], @@ -168,7 +175,7 @@ describe("revision-first build completion", () => { const completed = { ...pending, status: "completed", - imageUri: `${image}-amd64`, + imageUri: digest(image), }; mocks.queryResults.push( [pending], @@ -181,7 +188,7 @@ describe("revision-first build completion", () => { serviceRevisionId: "revision-missed-build-event", buildGroupId: "group-missed-build-event", finalImageUri: image, - images: [`${image}-amd64`], + images: [digest(image)], }), }, ], @@ -210,7 +217,7 @@ describe("revision-first build completion", () => { id: "missed-manifest-amd64", status: "completed", targetPlatform: "linux/amd64", - imageUri: `${image}-amd64`, + imageUri: digest(image), }, ]; mocks.queryResults.push( @@ -224,7 +231,7 @@ describe("revision-first build completion", () => { serviceRevisionId: "revision-missed-manifest-event", buildGroupId: "group-missed-manifest-event", finalImageUri: image, - images: [`${image}-amd64`], + images: [digest(image)], }), }, ], @@ -253,13 +260,13 @@ describe("revision-first build completion", () => { id: "incomplete-amd64", status: "completed", targetPlatform: "linux/amd64", - imageUri: `${image}-amd64`, + imageUri: digest(image), }, { id: "incomplete-arm64", status: "completed", targetPlatform: "linux/arm64", - imageUri: `${image}-arm64`, + imageUri: digest(image, "b"), }, ]; mocks.queryResults.push( @@ -272,7 +279,7 @@ describe("revision-first build completion", () => { serviceRevisionId: "revision-incomplete-manifest", buildGroupId: "group-incomplete-manifest", finalImageUri: image, - images: [`${image}-amd64`], + images: [digest(image)], }), }, ], diff --git a/web/tests/inngest-route.test.ts b/web/tests/inngest-route.test.ts index e5fb14ef..eb7ca73a 100644 --- a/web/tests/inngest-route.test.ts +++ b/web/tests/inngest-route.test.ts @@ -20,6 +20,7 @@ const mocks = vi.hoisted(() => { onRestoreFailed: { id: "on-restore-failed" }, restoreTriggerWorkflow: { id: "restore-trigger-workflow" }, restoreWorkflow: { id: "restore-workflow" }, + registryArtifactRetention: { id: "registry-artifact-retention" }, rolloutWorkflow: { id: "rollout-workflow" }, scheduledBackupsCheck: { id: "scheduled-backups-check" }, scheduledDeploymentsCheck: { id: "scheduled-deployments-check" }, diff --git a/web/tests/registry-retention.test.ts b/web/tests/registry-retention.test.ts new file mode 100644 index 00000000..5b2a7376 --- /dev/null +++ b/web/tests/registry-retention.test.ts @@ -0,0 +1,295 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; + +const mocks = vi.hoisted(() => { + let buildRows: Array<{ imageUri: string | null }> = []; + let serviceRows: unknown[] = []; + let selectRows: unknown[] = []; + const selectQuery = { + from: vi.fn(), + where: vi.fn(), + limit: vi.fn(), + // biome-ignore lint/suspicious/noThenProperty: Drizzle query builders are awaitable. + then: (resolve: (rows: unknown[]) => unknown) => + Promise.resolve(selectRows).then(resolve), + }; + selectQuery.from.mockReturnValue(selectQuery); + selectQuery.where.mockReturnValue(selectQuery); + selectQuery.limit.mockReturnValue(selectQuery); + const updateQuery = { set: vi.fn(), where: vi.fn() }; + updateQuery.set.mockReturnValue(updateQuery); + updateQuery.where.mockResolvedValue(undefined); + return { + transaction: vi.fn(), + execute: vi.fn(), + db: { + select: vi.fn((selection?: Record) => { + selectRows = selection + ? "imageUri" in selection + ? buildRows + : [{ id: "revision-1" }] + : serviceRows; + return selectQuery; + }), + update: vi.fn(() => updateQuery), + execute: vi.fn((...args: unknown[]) => mocks.execute(...args)), + transaction: vi.fn((...args: unknown[]) => mocks.transaction(...args)), + }, + updateQuery, + setBuildRows: (rows: Array<{ imageUri: string | null }>) => { + buildRows = rows; + }, + setServiceRows: (rows: unknown[]) => { + serviceRows = rows; + }, + }; +}); + +vi.mock("@/db", () => ({ db: mocks.db })); +vi.mock("@/lib/service-revision-changes", () => ({ + parseServiceRevisionSpec: (value: unknown) => value, +})); + +import { + cleanupRegistryArtifactsDaily, + cleanupRegistryArtifactsForService, + cleanupRevisionArtifact, + prepareRegistryArtifactCleanup, +} from "@/lib/registry-retention"; + +const digest = (character: string) => `sha256:${character.repeat(64)}`; +const githubRevision = (image: string) => ({ + id: "revision-1", + serviceId: "service-1", + artifactDeletedAt: null, + specification: { image, source: { type: "github" } }, +}); +const response = (status: number) => new Response(null, { status }); + +describe("registry retention", () => { + beforeEach(() => { + vi.restoreAllMocks(); + mocks.setBuildRows([]); + mocks.setServiceRows([]); + mocks.db.select.mockClear(); + mocks.db.update.mockClear(); + mocks.execute.mockReset(); + mocks.transaction.mockReset(); + mocks.updateQuery.set.mockClear(); + mocks.updateQuery.where.mockClear(); + process.env.REGISTRY_URL = "http://registry:5000"; + process.env.REGISTRY_HOST = "https://Registry.Example.com:5443"; + process.env.REGISTRY_USERNAME = "retention"; + process.env.REGISTRY_PASSWORD = "secret"; + }); + + it("routes a public registry reference through the internal API", async () => { + const fetchMock = vi + .spyOn(globalThis, "fetch") + .mockResolvedValue(response(202)); + + await cleanupRevisionArtifact( + githubRevision("registry.example.com:5443/team/a b:final"), + ); + + expect(fetchMock).toHaveBeenNthCalledWith( + 1, + "http://registry:5000/v2/team/a%20b/manifests/final", + expect.objectContaining({ + method: "DELETE", + headers: expect.objectContaining({ + Accept: expect.stringContaining( + "application/vnd.oci.image.index.v1+json", + ), + Authorization: `Basic ${Buffer.from("retention:secret").toString("base64")}`, + }), + }), + ); + }); + + it.each([ + { processingRows: [], ready: true }, + { processingRows: [{ id: "manifest-1" }], ready: false }, + ])("fails pending manifest work and reports processing readiness as $ready", async ({ + processingRows, + ready, + }) => { + const update = { + set: vi.fn(), + where: vi.fn().mockResolvedValue(undefined), + }; + update.set.mockReturnValue(update); + const select = { + from: vi.fn(), + where: vi.fn(), + limit: vi.fn().mockResolvedValue(processingRows), + }; + select.from.mockReturnValue(select); + select.where.mockReturnValue(select); + const tx = { + update: vi.fn(() => update), + select: vi.fn(() => select), + } as never; + + await expect(prepareRegistryArtifactCleanup(tx, "service-1")).resolves.toBe( + ready, + ); + expect(update.set).toHaveBeenCalledWith({ status: "failed" }); + expect(update.where).toHaveBeenCalledOnce(); + expect(select.limit).toHaveBeenCalledWith(1); + }); + + it("does not delete an old revision artifact shared with a protected newer revision", async () => { + mocks.execute.mockResolvedValue({ + rows: [ + { + id: "revision-old", + serviceId: "service-1", + image: "registry.example.com:5443/team/app:shared", + specification: { + image: "registry.example.com:5443/team/app:shared", + source: { type: "github" }, + }, + artifactDeletedAt: null, + }, + ], + }); + mocks.transaction.mockImplementation(async (callback) => + callback({ + execute: vi + .fn() + .mockResolvedValueOnce({ rows: [] }) + // Full locked recheck returns no rows because the newer + // same-image revision is in the protected completed set. + .mockResolvedValueOnce({ rows: [] }), + }), + ); + const fetchMock = vi.spyOn(globalThis, "fetch"); + + await cleanupRegistryArtifactsDaily(); + + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it("deletes only the modern final tag and leaves digest build children for GC", async () => { + mocks.setBuildRows([ + { imageUri: `registry.example.com:5443/team/app@${digest("b")}` }, + ]); + const fetchMock = vi + .spyOn(globalThis, "fetch") + .mockResolvedValue(response(202)); + + await cleanupRevisionArtifact( + githubRevision("registry.example.com:5443/team/app:final"), + ); + + expect(fetchMock).toHaveBeenCalledTimes(1); + expect(fetchMock.mock.calls[0][1]?.method).toBe("DELETE"); + expect(String(fetchMock.mock.calls[0][0])).toMatch(/\/manifests\/final$/); + expect(fetchMock.mock.calls.flat().join(" ")).not.toContain(digest("b")); + }); + + it("deletes the final tag then unique legacy architecture tags", async () => { + mocks.setBuildRows([ + { imageUri: "registry.example.com:5443/team/app:final" }, + { imageUri: "registry.example.com:5443/team/app:amd64" }, + { imageUri: "registry.example.com:5443/team/app:arm64" }, + { imageUri: "registry.example.com:5443/team/app:amd64" }, + ]); + const fetchMock = vi + .spyOn(globalThis, "fetch") + .mockResolvedValue(response(202)); + + await cleanupRevisionArtifact( + githubRevision("registry.example.com:5443/team/app:final"), + ); + + expect(fetchMock.mock.calls.map((call) => call[1]?.method)).toEqual([ + "DELETE", + "DELETE", + "DELETE", + ]); + expect(fetchMock.mock.calls.map((call) => String(call[0]))).toEqual([ + "http://registry:5000/v2/team/app/manifests/final", + "http://registry:5000/v2/team/app/manifests/amd64", + "http://registry:5000/v2/team/app/manifests/arm64", + ]); + }); + + it("treats a missing tag as deleted and marks the revision", async () => { + vi.spyOn(globalThis, "fetch").mockResolvedValue(response(404)); + + await expect( + cleanupRevisionArtifact( + githubRevision("registry.example.com:5443/team/app:gone"), + ), + ).resolves.toBe(true); + expect(mocks.db.update).toHaveBeenCalledOnce(); + }); + + it.each([ + 405, 500, + ])("does not mark when registry tag deletion returns %s", async (status) => { + vi.spyOn(globalThis, "fetch").mockResolvedValue(response(status)); + await expect( + cleanupRevisionArtifact( + githubRevision("registry.example.com:5443/team/app:final"), + ), + ).rejects.toThrow("DELETE failed"); + expect(mocks.db.update).not.toHaveBeenCalled(); + }); + + it("rejects a digest-addressed final image without network access", async () => { + const fetchMock = vi.spyOn(globalThis, "fetch"); + await expect( + cleanupRevisionArtifact( + githubRevision(`registry.example.com:5443/team/app@${digest("a")}`), + ), + ).rejects.toThrow("must use a tag"); + expect(fetchMock).not.toHaveBeenCalled(); + expect(mocks.db.update).not.toHaveBeenCalled(); + }); + + it("rejects external images without network access", async () => { + const fetchMock = vi.spyOn(globalThis, "fetch"); + await expect( + cleanupRevisionArtifact(githubRevision("evil.example/team/app:final")), + ).rejects.toThrow("unmanaged"); + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it("ignores non-GitHub external image revisions", async () => { + const fetchMock = vi.spyOn(globalThis, "fetch"); + await expect( + cleanupRevisionArtifact({ + ...githubRevision("evil.example/team/app:final"), + specification: { + image: "evil.example/team/app:final", + source: { type: "image", image: "evil.example/team/app:final" }, + }, + }), + ).resolves.toBe(false); + expect(fetchMock).not.toHaveBeenCalled(); + expect(mocks.db.update).not.toHaveBeenCalled(); + }); + + it("uses a GitHub representative when an external revision shares its image", async () => { + const image = "registry.example.com:5443/team/app:shared"; + mocks.setServiceRows([ + { + ...githubRevision(image), + id: "external-revision", + specification: { image, source: { type: "image", image } }, + }, + { ...githubRevision(image), id: "github-revision" }, + ]); + const fetchMock = vi + .spyOn(globalThis, "fetch") + .mockResolvedValue(response(202)); + + await cleanupRegistryArtifactsForService("service-1"); + + expect(fetchMock).toHaveBeenCalledOnce(); + expect(String(fetchMock.mock.calls[0][0])).toMatch(/\/manifests\/shared$/); + expect(mocks.db.update).toHaveBeenCalledOnce(); + }); +}); diff --git a/web/tests/service-revision-build.test.ts b/web/tests/service-revision-build.test.ts index b038673a..e35bd21e 100644 --- a/web/tests/service-revision-build.test.ts +++ b/web/tests/service-revision-build.test.ts @@ -171,7 +171,7 @@ describe("GitHub build service revisions", () => { it("combines current runtime config with a GitHub base artifact", async () => { const base = sourceSpecification(); mocks.selectResults.push( - [{ specification: base }], + [{ specification: base, artifactDeletedAt: null }], [], [ { @@ -220,4 +220,22 @@ describe("GitHub build service revisions", () => { }); expect(inserted.specification.image).not.toContain(":latest"); }); + + it("rejects a redeployment whose managed artifact was deleted", async () => { + mocks.selectResults.push([ + { + specification: sourceSpecification(), + artifactDeletedAt: new Date("2026-07-31T00:00:00.000Z"), + }, + ]); + + await expect( + createRolloutWithServiceRevision( + "service-1", + { type: "system" }, + "revision-expired", + ), + ).rejects.toThrow("Service revision artifact is no longer available"); + expect(mocks.tx.insert).not.toHaveBeenCalled(); + }); }); From cd4f7fca2e8ef58660b84d255baffa28547bee48 Mon Sep 17 00:00:00 2001 From: Amp Date: Sat, 1 Aug 2026 00:16:29 +0000 Subject: [PATCH 05/12] Document registry retention configuration Amp-Thread-ID: https://ampcode.com/threads/T-019fba5b-5b73-7227-9f59-22ae4c279baa Co-authored-by: Arjun Komath --- web/.env.example | 7 +++++-- web/lib/registry-retention.ts | 2 ++ 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/web/.env.example b/web/.env.example index cca92dac..439028b7 100644 --- a/web/.env.example +++ b/web/.env.example @@ -38,8 +38,11 @@ VM_USERNAME=username VM_PASSWORD=password VM_RETENTION=30d -# Docker Registry for builds (optional) -REGISTRY_HOST=registry.example.com +# Docker Registry for builds and artifact retention (optional) +REGISTRY_URL=http://localhost:5002 +REGISTRY_HOST=localhost:5002 +REGISTRY_USERNAME=your-registry-username +REGISTRY_PASSWORD=your-registry-password # Inngest (local dev via ../compose.dev.yml) INNGEST_BASE_URL=http://localhost:8288 diff --git a/web/lib/registry-retention.ts b/web/lib/registry-retention.ts index 0d5e7a90..82c68e4d 100644 --- a/web/lib/registry-retention.ts +++ b/web/lib/registry-retention.ts @@ -122,6 +122,8 @@ async function deleteTag( repository: string, tag: string, ) { + // Registry 3.1.1 supports exact-tag deletion. Resolving and deleting the + // digest would also remove retained aliases that point to the same manifest. const response = await fetch(manifestUrl(registryUrl, repository, tag), { method: "DELETE", headers: { Accept: MANIFEST_ACCEPT, Authorization: authorization }, From 81e597e59608652b3b4296a44ba52031a7d65422 Mon Sep 17 00:00:00 2001 From: Amp Date: Sat, 1 Aug 2026 00:15:38 +0000 Subject: [PATCH 06/12] feat: add global registry authentication Amp-Thread-ID: https://ampcode.com/threads/T-019fac42-0f87-77cd-a0fb-a9e99ccb8d65 Co-authored-by: Arjun Komath --- agent/cmd/agent/main.go | 56 +-- agent/internal/agent/agent.go | 22 +- agent/internal/agent/drift.go | 18 +- agent/internal/agent/handlers.go | 49 +- agent/internal/agent/registries_test.go | 45 ++ agent/internal/agent/run.go | 25 +- agent/internal/agent/workqueue.go | 2 + agent/internal/api/client.go | 16 +- agent/internal/build/build.go | 11 +- agent/internal/container/runtime.go | 86 +--- agent/internal/container/runtime_test.go | 21 +- agent/internal/container/types.go | 3 +- agent/internal/crypto/keys.go | 11 +- agent/internal/crypto/keys_test.go | 29 ++ agent/internal/http/client.go | 32 +- agent/internal/http/client_registry_test.go | 57 +++ agent/internal/http/client_test.go | 2 +- agent/internal/reconcile/reconcile.go | 25 +- agent/internal/registryauth/manager.go | 433 ++++++++++++++++++ agent/internal/registryauth/manager_test.go | 175 +++++++ docs/infrastructure/registry.mdx | 16 +- web/actions/compose.ts | 2 +- web/actions/projects.ts | 23 +- web/actions/registry-credentials.ts | 125 +++++ .../(dashboard)/dashboard/settings/page.tsx | 20 +- web/app/api/v1/agent/register/route.ts | 4 - web/app/api/v1/agent/registries/route.ts | 28 ++ web/components/settings/global-settings.tsx | 15 + web/components/settings/registry-settings.tsx | 214 +++++++++ web/db/schema.ts | 25 + web/db/types.ts | 2 + web/lib/agent-auth.ts | 3 +- web/lib/agent/expected-state.ts | 9 +- web/lib/crypto.ts | 23 + web/lib/docker-image.ts | 256 +---------- web/lib/registry-credentials.ts | 194 ++++++++ web/lib/registry-reference.ts | 126 +++++ web/lib/trigger-build.ts | 6 +- web/lib/work-queue.ts | 33 +- web/tests/agent-auth-signature.test.ts | 58 +++ web/tests/agent-registries-route.test.ts | 82 ++++ web/tests/docker-image.test.ts | 158 +------ web/tests/registry-credentials.test.ts | 64 +++ web/tests/registry-reference.test.ts | 47 ++ web/tests/registry-work-queue.test.ts | 92 ++++ 45 files changed, 2139 insertions(+), 604 deletions(-) create mode 100644 agent/internal/agent/registries_test.go create mode 100644 agent/internal/crypto/keys_test.go create mode 100644 agent/internal/http/client_registry_test.go create mode 100644 agent/internal/registryauth/manager.go create mode 100644 agent/internal/registryauth/manager_test.go create mode 100644 web/actions/registry-credentials.ts create mode 100644 web/app/api/v1/agent/registries/route.ts create mode 100644 web/components/settings/registry-settings.tsx create mode 100644 web/lib/registry-credentials.ts create mode 100644 web/lib/registry-reference.ts create mode 100644 web/tests/agent-auth-signature.test.ts create mode 100644 web/tests/agent-registries-route.test.ts create mode 100644 web/tests/registry-credentials.test.ts create mode 100644 web/tests/registry-reference.test.ts create mode 100644 web/tests/registry-work-queue.test.ts diff --git a/agent/cmd/agent/main.go b/agent/cmd/agent/main.go index 4605ad7c..7f710f97 100644 --- a/agent/cmd/agent/main.go +++ b/agent/cmd/agent/main.go @@ -25,6 +25,7 @@ import ( "techulus/cloud-agent/internal/network" "techulus/cloud-agent/internal/paths" "techulus/cloud-agent/internal/reconcile" + "techulus/cloud-agent/internal/registryauth" "techulus/cloud-agent/internal/routeowners" "techulus/cloud-agent/internal/traefik" "techulus/cloud-agent/internal/wireguard" @@ -126,6 +127,9 @@ func main() { if err != nil { log.Fatalf("Failed to load config: %v", err) } + if err := configuration.Save(config); err != nil { + log.Fatalf("Failed to rewrite config: %v", err) + } log.Printf("Loaded config: serverID=%s, subnetId=%d, wireguardIP=%s", config.ServerID, config.SubnetID, config.WireGuardIP) @@ -194,29 +198,14 @@ func main() { respMetricsEndpoint = *resp.MetricsEndpoint } - var registryURL, registryUsername, registryPassword string - if resp.RegistryURL != nil { - registryURL = *resp.RegistryURL - } - if resp.RegistryUsername != nil { - registryUsername = *resp.RegistryUsername - } - if resp.RegistryPassword != nil { - registryPassword = *resp.RegistryPassword - } - config = &agent.Config{ - ServerID: resp.ServerID, - SubnetID: resp.SubnetID, - WireGuardIP: resp.WireGuardIP, - EncryptionKey: resp.EncryptionKey, - IsProxy: isProxy, - LoggingEndpoint: respLoggingEndpoint, - MetricsEndpoint: respMetricsEndpoint, - RegistryURL: registryURL, - RegistryUsername: registryUsername, - RegistryPassword: registryPassword, - RegistryInsecure: resp.RegistryInsecure, + ServerID: resp.ServerID, + SubnetID: resp.SubnetID, + WireGuardIP: resp.WireGuardIP, + EncryptionKey: resp.EncryptionKey, + IsProxy: isProxy, + LoggingEndpoint: respLoggingEndpoint, + MetricsEndpoint: respMetricsEndpoint, } if logsEndpointFlag != "" { @@ -275,15 +264,20 @@ func main() { } } - if config.RegistryURL != "" && config.RegistryUsername != "" { - log.Printf("[registry] attempting login to %s", config.RegistryURL) - if err := container.Login(config.RegistryURL, config.RegistryUsername, config.RegistryPassword, config.RegistryInsecure); err != nil { - log.Printf("[registry] warning: failed to login to registry: %v", err) - } - } - - reconciler := reconcile.NewReconciler(config.EncryptionKey, dataDir, config.RegistryInsecure) client := agenthttp.NewClient(controlPlaneURL, config.ServerID, signingKeyPair, dataDir) + registryManager, err := registryauth.NewManager(dataDir, config.EncryptionKey, client) + if err != nil { + log.Fatalf("Failed to initialize registry authentication: %v", err) + } + if err := registryManager.MarkDirty("startup"); err != nil { + log.Fatalf("Failed to require initial registry synchronization: %v", err) + } + initialCtx, initialCancel := context.WithTimeout(context.Background(), 30*time.Second) + if err := registryManager.Sync(initialCtx); err != nil { + log.Printf("[registry] initial sync failed: %v", err) + } + initialCancel() + reconciler := reconcile.NewReconciler(config.EncryptionKey, dataDir, registryManager) var logCollector *logs.Collector var traefikLogCollector *logs.TraefikCollector @@ -339,7 +333,7 @@ func main() { privateIP := network.PrivateIP() log.Printf("Agent %s started. Public IP: %s, Private IP: %s. Tick interval: %v", agent.Version, publicIP, privateIP, agent.TickInterval) - agentInstance := agent.NewAgent(client, reconciler, config, publicIP, privateIP, dataDir, logCollector, traefikLogCollector, metricsSender, routeOwners, builder, config.IsProxy, disableDNS) + agentInstance := agent.NewAgent(client, reconciler, config, publicIP, privateIP, dataDir, logCollector, traefikLogCollector, metricsSender, routeOwners, builder, registryManager, config.IsProxy, disableDNS) agentInstance.Run(ctx) if agentLogFlusherDone != nil { diff --git a/agent/internal/agent/agent.go b/agent/internal/agent/agent.go index eeef2be7..da96c6f8 100644 --- a/agent/internal/agent/agent.go +++ b/agent/internal/agent/agent.go @@ -11,6 +11,7 @@ import ( agenthttp "techulus/cloud-agent/internal/http" "techulus/cloud-agent/internal/logs" "techulus/cloud-agent/internal/reconcile" + "techulus/cloud-agent/internal/registryauth" "techulus/cloud-agent/internal/routeowners" ) @@ -28,17 +29,13 @@ const ( ) type Config struct { - ServerID string `json:"serverId"` - SubnetID int `json:"subnetId"` - WireGuardIP string `json:"wireguardIp"` - EncryptionKey string `json:"encryptionKey"` - IsProxy bool `json:"isProxy"` - LoggingEndpoint string `json:"loggingEndpoint,omitempty"` - MetricsEndpoint string `json:"metricsEndpoint,omitempty"` - RegistryURL string `json:"registryUrl,omitempty"` - RegistryUsername string `json:"registryUsername,omitempty"` - RegistryPassword string `json:"registryPassword,omitempty"` - RegistryInsecure bool `json:"registryInsecure"` + ServerID string `json:"serverId"` + SubnetID int `json:"subnetId"` + WireGuardIP string `json:"wireguardIp"` + EncryptionKey string `json:"encryptionKey"` + IsProxy bool `json:"isProxy"` + LoggingEndpoint string `json:"loggingEndpoint,omitempty"` + MetricsEndpoint string `json:"metricsEndpoint,omitempty"` } type ActualState struct { @@ -89,6 +86,7 @@ type Agent struct { MetricsSender MetricsSender RouteOwners *routeowners.Registry Builder *build.Builder + RegistryAuth *registryauth.Manager isBuilding bool buildMutex sync.Mutex currentBuildID string @@ -107,6 +105,7 @@ func NewAgent( metricsSender MetricsSender, routeOwners *routeowners.Registry, builder *build.Builder, + registryAuth *registryauth.Manager, isProxy bool, disableDNS bool, ) *Agent { @@ -126,6 +125,7 @@ func NewAgent( MetricsSender: metricsSender, RouteOwners: routeOwners, Builder: builder, + RegistryAuth: registryAuth, IsProxy: isProxy, DisableDNS: disableDNS, deploymentDeployLocks: map[string]*sync.Mutex{}, diff --git a/agent/internal/agent/drift.go b/agent/internal/agent/drift.go index 75c02a47..15e98655 100644 --- a/agent/internal/agent/drift.go +++ b/agent/internal/agent/drift.go @@ -4,12 +4,12 @@ import ( "context" "fmt" "log" - "strings" "time" "techulus/cloud-agent/internal/container" "techulus/cloud-agent/internal/dns" agenthttp "techulus/cloud-agent/internal/http" + "techulus/cloud-agent/internal/registryauth" "techulus/cloud-agent/internal/retry" "techulus/cloud-agent/internal/traefik" "techulus/cloud-agent/internal/wireguard" @@ -403,21 +403,7 @@ func (a *Agent) planReconcile(expected *agenthttp.ExpectedState, actual *ActualS } func normalizeImage(image string) string { - digest := "" - if digestIndex := strings.Index(image, "@"); digestIndex != -1 { - digest = image[digestIndex:] - image = image[:digestIndex] - } - - image = strings.TrimPrefix(image, "docker.io/library/") - image = strings.TrimPrefix(image, "docker.io/") - - lastSlash := strings.LastIndex(image, "/") - lastColon := strings.LastIndex(image, ":") - if digest == "" && lastColon <= lastSlash { - image = image + ":latest" - } - return image + digest + return registryauth.NormalizeImage(image) } func desiredContainerState(container agenthttp.ExpectedContainer) string { diff --git a/agent/internal/agent/handlers.go b/agent/internal/agent/handlers.go index 8d1c528d..0b8aeb29 100644 --- a/agent/internal/agent/handlers.go +++ b/agent/internal/agent/handlers.go @@ -16,6 +16,7 @@ import ( "techulus/cloud-agent/internal/crypto" agenthttp "techulus/cloud-agent/internal/http" "techulus/cloud-agent/internal/paths" + "techulus/cloud-agent/internal/registryauth" ) func (a *Agent) ProcessRestart(item agenthttp.WorkQueueItem) error { @@ -137,6 +138,11 @@ func (a *Agent) ProcessBuild(item agenthttp.WorkQueueItem) error { a.buildMutex.Unlock() }() + snapshot, releaseRegistryAuth, err := a.RegistryAuth.Acquire() + if err != nil { + return err + } + defer releaseRegistryAuth() buildDetails, err := a.Client.ClaimBuild(payload.BuildID) if err != nil { return fmt.Errorf("failed to claim build: %w", err) @@ -182,6 +188,8 @@ func (a *Agent) ProcessBuild(item agenthttp.WorkQueueItem) error { RootDir: buildDetails.RootDir, Secrets: decryptedSecrets, TargetPlatforms: buildDetails.TargetPlatforms, + DockerConfigDir: snapshot.DockerConfigDir, + TargetTLSVerify: snapshot.TLSVerify(buildDetails.ImageURI), } onStatusChange := func(status string) { @@ -231,13 +239,37 @@ func (a *Agent) ProcessCreateManifest(item agenthttp.WorkQueueItem) error { } log.Printf("[create_manifest] creating manifest for %s with %d images", payload.FinalImageUri, len(payload.Images)) + snapshot, releaseRegistryAuth, err := a.RegistryAuth.Acquire() + if err != nil { + return err + } + defer releaseRegistryAuth() + tls := snapshot.TLSVerify(payload.FinalImageUri) + host, err := registryauth.ImageHost(payload.FinalImageUri) + if err != nil { + return fmt.Errorf("invalid manifest target reference") + } + for _, img := range payload.Images { + imageHost, hostErr := registryauth.ImageHost(img) + if hostErr != nil || imageHost != host { + return fmt.Errorf("manifest references must use one registry host") + } + if snapshot.TLSVerify(img) != tls { + return fmt.Errorf("manifest registry TLS policies differ") + } + } - craneArgs := []string{"index", "append", "--insecure", "-t", payload.FinalImageUri} + craneArgs := []string{"index", "append"} + if !tls { + craneArgs = append(craneArgs, "--insecure") + } + craneArgs = append(craneArgs, "-t", payload.FinalImageUri) for _, img := range payload.Images { craneArgs = append(craneArgs, "-m", img) } cmd := exec.Command(paths.CranePath, craneArgs...) + cmd.Env = append(os.Environ(), "DOCKER_CONFIG="+snapshot.DockerConfigDir, "HOME="+snapshot.EmptyHome) output, err := cmd.CombinedOutput() if err != nil { log.Printf("[create_manifest] crane failed: %s", string(output)) @@ -247,3 +279,18 @@ func (a *Agent) ProcessCreateManifest(item agenthttp.WorkQueueItem) error { log.Printf("[create_manifest] manifest created successfully for %s", payload.FinalImageUri) return nil } + +func (a *Agent) ProcessSyncRegistries(item agenthttp.WorkQueueItem) error { + var payload struct { + Version string `json:"version"` + } + if err := json.Unmarshal([]byte(item.Payload), &payload); err != nil || payload.Version == "" { + return fmt.Errorf("invalid sync_registries payload") + } + if err := a.RegistryAuth.MarkDirty(payload.Version); err != nil { + return fmt.Errorf("mark registry sync required: %w", err) + } + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + return a.RegistryAuth.Sync(ctx) +} diff --git a/agent/internal/agent/registries_test.go b/agent/internal/agent/registries_test.go new file mode 100644 index 00000000..d92fe05d --- /dev/null +++ b/agent/internal/agent/registries_test.go @@ -0,0 +1,45 @@ +package agent + +import ( + "context" + "errors" + "testing" + + agenthttp "techulus/cloud-agent/internal/http" + "techulus/cloud-agent/internal/registryauth" +) + +type failingRegistryBundleSource struct{} + +func (failingRegistryBundleSource) GetRegistryBundle(context.Context) (*registryauth.Bundle, error) { + return nil, errors.New("control plane unavailable") +} + +func TestProcessSyncRegistriesMarksAuthenticationDirtyBeforeFetch(t *testing.T) { + manager, err := registryauth.NewManager(t.TempDir(), "00", failingRegistryBundleSource{}) + if err != nil { + t.Fatal(err) + } + a := &Agent{RegistryAuth: manager} + err = a.ProcessSyncRegistries(agenthttp.WorkQueueItem{ + Type: "sync_registries", + Payload: `{"version":"v2"}`, + }) + if err == nil { + t.Fatal("sync unexpectedly succeeded") + } + if _, _, err := manager.Acquire(); err == nil { + t.Fatal("registry authentication remained usable after failed required sync") + } +} + +func TestProcessSyncRegistriesRejectsInvalidPayload(t *testing.T) { + manager, err := registryauth.NewManager(t.TempDir(), "00", failingRegistryBundleSource{}) + if err != nil { + t.Fatal(err) + } + a := &Agent{RegistryAuth: manager} + if err := a.ProcessSyncRegistries(agenthttp.WorkQueueItem{Payload: `{}`}); err == nil { + t.Fatal("invalid payload was accepted") + } +} diff --git a/agent/internal/agent/run.go b/agent/internal/agent/run.go index 1f9baa09..607685fe 100644 --- a/agent/internal/agent/run.go +++ b/agent/internal/agent/run.go @@ -9,7 +9,6 @@ import ( "net/http" "time" - "techulus/cloud-agent/internal/container" agenthttp "techulus/cloud-agent/internal/http" "techulus/cloud-agent/internal/metrics" "techulus/cloud-agent/internal/serverless" @@ -32,14 +31,6 @@ func (a *Agent) Run(ctx context.Context) { log.Printf("[cache] expected state unavailable for initial Traefik attribution: %v", err) } } - if a.Config.RegistryURL != "" && a.Config.RegistryUsername != "" && a.Config.RegistryPassword != "" { - if err := container.Login(a.Config.RegistryURL, a.Config.RegistryUsername, a.Config.RegistryPassword, a.Config.RegistryInsecure); err != nil { - log.Printf("[registry] login failed: %v", err) - } else { - log.Printf("[registry] logged in to %s", a.Config.RegistryURL) - } - } - ticker := time.NewTicker(TickInterval) defer ticker.Stop() @@ -78,6 +69,7 @@ func (a *Agent) Run(ctx context.Context) { go a.StatusReportLoop(ctx) go a.WorkQueueWakeLoop(ctx) + go a.RegistrySyncLoop(ctx) a.Tick() @@ -110,6 +102,21 @@ func (a *Agent) Run(ctx context.Context) { } } +func (a *Agent) RegistrySyncLoop(ctx context.Context) { + t := time.NewTicker(5 * time.Minute) + defer t.Stop() + for { + select { + case <-ctx.Done(): + return + case <-t.C: + if err := a.RegistryAuth.Sync(ctx); err != nil { + log.Printf("[registry] fallback sync failed: %v", err) + } + } + } +} + func (a *Agent) TraefikMetricsLoop(ctx context.Context) { ticker := time.NewTicker(traefikMetricsInterval) defer ticker.Stop() diff --git a/agent/internal/agent/workqueue.go b/agent/internal/agent/workqueue.go index 4c806853..5791a21f 100644 --- a/agent/internal/agent/workqueue.go +++ b/agent/internal/agent/workqueue.go @@ -146,6 +146,8 @@ func (a *Agent) ProcessWorkItem(item agenthttp.WorkQueueItem) error { return a.ProcessRestoreVolume(item) case "create_manifest": return a.ProcessCreateManifest(item) + case "sync_registries": + return a.ProcessSyncRegistries(item) case "upgrade_agent": return a.ProcessAgentUpgrade(item) default: diff --git a/agent/internal/api/client.go b/agent/internal/api/client.go index 187e7b13..b4ad14af 100644 --- a/agent/internal/api/client.go +++ b/agent/internal/api/client.go @@ -33,16 +33,12 @@ type RegisterRequest struct { } type RegisterResponse struct { - ServerID string `json:"serverId"` - SubnetID int `json:"subnetId"` - WireGuardIP string `json:"wireguardIp"` - EncryptionKey string `json:"encryptionKey"` - LoggingEndpoint *string `json:"loggingEndpoint"` - MetricsEndpoint *string `json:"metricsEndpoint"` - RegistryURL *string `json:"registryUrl"` - RegistryUsername *string `json:"registryUsername"` - RegistryPassword *string `json:"registryPassword"` - RegistryInsecure bool `json:"registryInsecure"` + ServerID string `json:"serverId"` + SubnetID int `json:"subnetId"` + WireGuardIP string `json:"wireguardIp"` + EncryptionKey string `json:"encryptionKey"` + LoggingEndpoint *string `json:"loggingEndpoint"` + MetricsEndpoint *string `json:"metricsEndpoint"` } func (c *Client) Register(token, wireguardPublicKey, signingPublicKey, publicIP, privateIP string, isProxy bool) (*RegisterResponse, error) { diff --git a/agent/internal/build/build.go b/agent/internal/build/build.go index 6a99aaa8..42d34970 100644 --- a/agent/internal/build/build.go +++ b/agent/internal/build/build.go @@ -35,6 +35,8 @@ type Config struct { RootDir string Secrets map[string]string TargetPlatforms []string + DockerConfigDir string + TargetTLSVerify bool } type LogSender interface { @@ -259,7 +261,10 @@ func (b *Builder) buildAndPush(ctx context.Context, config *Config, buildDir str if len(config.TargetPlatforms) > 0 { platform = config.TargetPlatforms[0] } - outputFlag := fmt.Sprintf("type=image,name=%s,push=true,push-by-digest=true,registry.insecure=true", config.ImageRepository) + outputFlag := fmt.Sprintf("type=image,name=%s,push=true,push-by-digest=true", config.ImageRepository) + if !config.TargetTLSVerify { + outputFlag += ",registry.insecure=true" + } if dockerfile.found { log.Printf("[build:%s] building with Dockerfile via buildctl for %s", truncateStr(config.BuildID, 8), platform) @@ -285,7 +290,7 @@ func (b *Builder) buildAndPush(ctx context.Context, config *Config, buildDir str cmd := exec.CommandContext(ctx, paths.BuildctlPath, args...) cmd.Dir = contextDir - cmd.Env = append(os.Environ(), secretEnv...) + cmd.Env = append(os.Environ(), append(secretEnv, "DOCKER_CONFIG="+config.DockerConfigDir)...) output, err := b.runCommandStreaming(cmd, config) if err != nil { log.Printf("[build:%s] buildctl failed with output: %s", truncateStr(config.BuildID, 8), output) @@ -334,7 +339,7 @@ func (b *Builder) buildAndPush(ctx context.Context, config *Config, buildDir str cmd = exec.CommandContext(ctx, paths.BuildctlPath, args...) cmd.Dir = contextDir - cmd.Env = append(os.Environ(), secretEnv...) + cmd.Env = append(os.Environ(), append(secretEnv, "DOCKER_CONFIG="+config.DockerConfigDir)...) output, err = b.runCommandStreaming(cmd, config) if err != nil { log.Printf("[build:%s] buildctl failed for %s: %s", truncateStr(config.BuildID, 8), platform, output) diff --git a/agent/internal/container/runtime.go b/agent/internal/container/runtime.go index 412fd36d..3671b0d7 100644 --- a/agent/internal/container/runtime.go +++ b/agent/internal/container/runtime.go @@ -2,14 +2,12 @@ package container import ( "context" - "encoding/base64" "encoding/json" "errors" "fmt" "log" "os" "os/exec" - "path/filepath" "strings" "time" @@ -133,11 +131,7 @@ func Deploy(config *DeployConfig) (*DeployResult, error) { } func buildPodmanPullArgs(config *DeployConfig) []string { - args := []string{"pull"} - if config.RegistryInsecure { - args = append(args, "--tls-verify=false") - } - return append(args, config.Image) + return []string{"pull", "--authfile", config.AuthFile, fmt.Sprintf("--tls-verify=%t", config.TLSVerify), config.Image} } func buildPodmanRunArgs(config *DeployConfig, image string) []string { @@ -375,84 +369,6 @@ func CheckPrerequisites() error { return nil } -func Login(registryURL, username, password string, insecure bool) error { - if registryURL == "" || username == "" { - return nil - } - - log.Printf("[podman:login] logging in to registry %s", registryURL) - - args := []string{"login"} - if insecure { - args = append(args, "--tls-verify=false") - } - args = append(args, "-u", username, "-p", password, registryURL) - - cmd := exec.Command("podman", args...) - output, err := cmd.CombinedOutput() - if err != nil { - return fmt.Errorf("failed to login to registry: %s: %w", string(output), err) - } - - log.Printf("[podman:login] successfully logged in to registry %s", registryURL) - - if err := writeDockerConfig(registryURL, username, password); err != nil { - log.Printf("[registry] failed to write docker config: %v", err) - } - - registryHost := strings.TrimPrefix(registryURL, "https://") - registryHost = strings.TrimPrefix(registryHost, "http://") - registryHost = strings.TrimSuffix(registryHost, "/") - - craneArgs := []string{"auth", "login", "-u", username, "-p", password, registryHost} - craneCmd := exec.Command("/usr/local/bin/crane", craneArgs...) - if out, err := craneCmd.CombinedOutput(); err != nil { - log.Printf("[crane:login] failed: %s: %v", string(out), err) - } else { - log.Printf("[crane:login] successfully logged in to %s", registryHost) - } - - return nil -} - -func writeDockerConfig(registryURL, username, password string) error { - registryHost := strings.TrimPrefix(registryURL, "https://") - registryHost = strings.TrimPrefix(registryHost, "http://") - registryHost = strings.TrimSuffix(registryHost, "/") - - homeDir, err := os.UserHomeDir() - if err != nil { - return err - } - - dockerDir := filepath.Join(homeDir, ".docker") - if err := os.MkdirAll(dockerDir, 0700); err != nil { - return err - } - - auth := base64.StdEncoding.EncodeToString([]byte(username + ":" + password)) - config := map[string]interface{}{ - "auths": map[string]interface{}{ - registryHost: map[string]string{ - "auth": auth, - }, - }, - } - - configBytes, err := json.MarshalIndent(config, "", " ") - if err != nil { - return err - } - - configPath := filepath.Join(dockerDir, "config.json") - if err := os.WriteFile(configPath, configBytes, 0600); err != nil { - return err - } - - log.Printf("[registry] wrote docker config to %s", configPath) - return nil -} - func ImagePrune() error { cmd := exec.Command("podman", "image", "prune", "-a", "-f", "--filter", "until=168h") if output, err := cmd.CombinedOutput(); err != nil { diff --git a/agent/internal/container/runtime_test.go b/agent/internal/container/runtime_test.go index 629febf4..f9b09853 100644 --- a/agent/internal/container/runtime_test.go +++ b/agent/internal/container/runtime_test.go @@ -7,26 +7,27 @@ import ( func TestBuildPodmanPullArgs(t *testing.T) { tests := []struct { - name string - insecure bool - want []string + name string + tlsVerify bool + want []string }{ { - name: "does not disable TLS by default", - want: []string{"pull", "registry.example.com/app:latest"}, + name: "does not disable TLS by default", + tlsVerify: true, + want: []string{"pull", "--authfile", "/managed/config.json", "--tls-verify=true", "registry.example.com/app:latest"}, }, { - name: "disables TLS verification when configured", - insecure: true, - want: []string{"pull", "--tls-verify=false", "registry.example.com/app:latest"}, + name: "disables TLS verification when configured", + want: []string{"pull", "--authfile", "/managed/config.json", "--tls-verify=false", "registry.example.com/app:latest"}, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { got := buildPodmanPullArgs(&DeployConfig{ - Image: "registry.example.com/app:latest", - RegistryInsecure: tt.insecure, + Image: "registry.example.com/app:latest", + AuthFile: "/managed/config.json", + TLSVerify: tt.tlsVerify, }) if !slices.Equal(got, tt.want) { t.Fatalf("buildPodmanPullArgs() = %q, want %q", got, tt.want) diff --git a/agent/internal/container/types.go b/agent/internal/container/types.go index ea1730f7..ae8e2f0f 100644 --- a/agent/internal/container/types.go +++ b/agent/internal/container/types.go @@ -28,7 +28,8 @@ type BuildLogFunc func(stream string, message string) type DeployConfig struct { Name string Image string - RegistryInsecure bool + AuthFile string + TLSVerify bool ServiceID string ServiceName string DeploymentID string diff --git a/agent/internal/crypto/keys.go b/agent/internal/crypto/keys.go index e01eea46..485746e0 100644 --- a/agent/internal/crypto/keys.go +++ b/agent/internal/crypto/keys.go @@ -88,6 +88,15 @@ func KeyPairExists(dir string) bool { } func DecryptSecret(encryptedBase64 string, keyHex string) (string, error) { + return decryptAESGCM(encryptedBase64, keyHex, nil) +} + +func DecryptRegistryCredential(encryptedBase64, keyHex, id, canonicalHost string) (string, error) { + aad := []byte("registry-credential:v1\x00" + id + "\x00" + canonicalHost) + return decryptAESGCM(encryptedBase64, keyHex, aad) +} + +func decryptAESGCM(encryptedBase64 string, keyHex string, aad []byte) (string, error) { key, err := hex.DecodeString(keyHex) if err != nil { return "", fmt.Errorf("invalid encryption key: %w", err) @@ -120,7 +129,7 @@ func DecryptSecret(encryptedBase64 string, keyHex string) (string, error) { } ciphertextWithTag := append(ciphertext, authTag...) - plaintext, err := gcm.Open(nil, iv, ciphertextWithTag, nil) + plaintext, err := gcm.Open(nil, iv, ciphertextWithTag, aad) if err != nil { return "", fmt.Errorf("decryption failed: %w", err) } diff --git a/agent/internal/crypto/keys_test.go b/agent/internal/crypto/keys_test.go new file mode 100644 index 00000000..2392f8e0 --- /dev/null +++ b/agent/internal/crypto/keys_test.go @@ -0,0 +1,29 @@ +package crypto + +import ( + "crypto/aes" + "crypto/cipher" + "encoding/base64" + "encoding/hex" + "testing" +) + +func TestDecryptRegistryCredentialAAD(t *testing.T) { + key := make([]byte, 32) + block, _ := aes.NewCipher(key) + gcm, _ := cipher.NewGCM(block) + iv := make([]byte, gcm.NonceSize()) + aad := []byte("registry-credential:v1\x00id-1\x00docker.io") + sealed := gcm.Seal(nil, iv, []byte("password"), aad) + framed := append(append([]byte{}, iv...), sealed[len(sealed)-gcm.Overhead():]...) + framed = append(framed, sealed[:len(sealed)-gcm.Overhead()]...) + encoded := base64.StdEncoding.EncodeToString(framed) + + got, err := DecryptRegistryCredential(encoded, hex.EncodeToString(key), "id-1", "docker.io") + if err != nil || got != "password" { + t.Fatalf("decrypt = %q, %v", got, err) + } + if _, err := DecryptRegistryCredential(encoded, hex.EncodeToString(key), "id-2", "docker.io"); err == nil { + t.Fatal("decrypt succeeded with wrong AAD") + } +} diff --git a/agent/internal/http/client.go b/agent/internal/http/client.go index 7e96243e..03c126ab 100644 --- a/agent/internal/http/client.go +++ b/agent/internal/http/client.go @@ -16,6 +16,7 @@ import ( "techulus/cloud-agent/internal/crypto" "techulus/cloud-agent/internal/health" + "techulus/cloud-agent/internal/registryauth" ) type Client struct { @@ -26,6 +27,35 @@ type Client struct { dataDir string } +func (c *Client) GetRegistryBundle(ctx context.Context) (*registryauth.Bundle, error) { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.baseURL+"/api/v1/agent/registries", nil) + if err != nil { + return nil, fmt.Errorf("create registry request: %w", err) + } + c.signRequest(req, "") + resp, err := c.client.Do(req) + if err != nil { + return nil, fmt.Errorf("fetch registry bundle: %w", err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("fetch registry bundle returned status %d", resp.StatusCode) + } + limited := io.LimitReader(resp.Body, 2*1024*1024+1) + data, err := io.ReadAll(limited) + if err != nil { + return nil, fmt.Errorf("read registry response: %w", err) + } + if len(data) > 2*1024*1024 { + return nil, errors.New("registry response too large") + } + var bundle registryauth.Bundle + if err = json.Unmarshal(data, &bundle); err != nil { + return nil, errors.New("invalid registry response") + } + return &bundle, nil +} + func NewClient(baseURL, serverID string, keyPair *crypto.KeyPair, dataDir string) *Client { return &Client{ baseURL: baseURL, @@ -40,7 +70,7 @@ func NewClient(baseURL, serverID string, keyPair *crypto.KeyPair, dataDir string func (c *Client) signRequest(req *http.Request, body string) { timestamp := strconv.FormatInt(time.Now().UnixMilli(), 10) - message := timestamp + ":" + body + message := "agent-request:v2\x00" + timestamp + "\x00" + req.Method + "\x00" + req.URL.RequestURI() + "\x00" + body signature := c.keyPair.Sign([]byte(message)) req.Header.Set("x-server-id", c.serverID) diff --git a/agent/internal/http/client_registry_test.go b/agent/internal/http/client_registry_test.go new file mode 100644 index 00000000..8db2028a --- /dev/null +++ b/agent/internal/http/client_registry_test.go @@ -0,0 +1,57 @@ +package http + +import ( + "context" + stdhttp "net/http" + "net/http/httptest" + "testing" + + "techulus/cloud-agent/internal/crypto" +) + +func TestGetRegistryBundle(t *testing.T) { + keyPair, err := crypto.GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + server := httptest.NewServer(stdhttp.HandlerFunc(func(w stdhttp.ResponseWriter, r *stdhttp.Request) { + if r.Method != stdhttp.MethodGet || r.URL.Path != "/api/v1/agent/registries" { + t.Errorf("unexpected request %s %s", r.Method, r.URL.Path) + } + for _, header := range []string{"x-server-id", "x-timestamp", "x-signature"} { + if r.Header.Get(header) == "" { + t.Errorf("missing %s header", header) + } + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"version":"v1","registries":[]}`)) + })) + defer server.Close() + + client := NewClient(server.URL, "server-1", keyPair, t.TempDir()) + bundle, err := client.GetRegistryBundle(context.Background()) + if err != nil { + t.Fatal(err) + } + if bundle.Version != "v1" || len(bundle.Registries) != 0 { + t.Fatalf("unexpected bundle: %+v", bundle) + } +} + +func TestGetRegistryBundleDoesNotIncludeErrorBody(t *testing.T) { + keyPair, err := crypto.GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + server := httptest.NewServer(stdhttp.HandlerFunc(func(w stdhttp.ResponseWriter, _ *stdhttp.Request) { + w.WriteHeader(stdhttp.StatusServiceUnavailable) + _, _ = w.Write([]byte(`{"encryptedPassword":"must-not-be-returned"}`)) + })) + defer server.Close() + + client := NewClient(server.URL, "server-1", keyPair, t.TempDir()) + _, err = client.GetRegistryBundle(context.Background()) + if err == nil || err.Error() != "fetch registry bundle returned status 503" { + t.Fatalf("unexpected error: %v", err) + } +} diff --git a/agent/internal/http/client_test.go b/agent/internal/http/client_test.go index 41c89322..d402012b 100644 --- a/agent/internal/http/client_test.go +++ b/agent/internal/http/client_test.go @@ -19,7 +19,7 @@ func TestSignedJSONRequests(t *testing.T) { server := httptest.NewServer(stdhttp.HandlerFunc(func(w stdhttp.ResponseWriter, r *stdhttp.Request) { body, _ := io.ReadAll(r.Body) if r.URL.Path == "/api/v1/agent/status" { - message := r.Header.Get("x-timestamp") + ":" + string(body) + message := "agent-request:v2\x00" + r.Header.Get("x-timestamp") + "\x00" + r.Method + "\x00" + r.URL.RequestURI() + "\x00" + string(body) if r.Method != stdhttp.MethodPost || r.Header.Get("Content-Type") != "application/json" || r.Header.Get("x-server-id") != "server-1" || r.Header.Get("x-signature") != keyPair.Sign([]byte(message)) { t.Error("request method, headers, or signature not preserved") } diff --git a/agent/internal/reconcile/reconcile.go b/agent/internal/reconcile/reconcile.go index 1817ff5f..6e5d8cff 100644 --- a/agent/internal/reconcile/reconcile.go +++ b/agent/internal/reconcile/reconcile.go @@ -8,23 +8,29 @@ import ( "techulus/cloud-agent/internal/container" "techulus/cloud-agent/internal/crypto" agenthttp "techulus/cloud-agent/internal/http" + "techulus/cloud-agent/internal/registryauth" ) type Reconciler struct { - encryptionKey string - dataDir string - registryInsecure bool + encryptionKey string + dataDir string + registryAuth *registryauth.Manager } -func NewReconciler(encryptionKey, dataDir string, registryInsecure bool) *Reconciler { +func NewReconciler(encryptionKey, dataDir string, registryAuth *registryauth.Manager) *Reconciler { return &Reconciler{ - encryptionKey: encryptionKey, - dataDir: dataDir, - registryInsecure: registryInsecure, + encryptionKey: encryptionKey, + dataDir: dataDir, + registryAuth: registryAuth, } } func (r *Reconciler) Deploy(exp agenthttp.ExpectedContainer) error { + snapshot, releaseRegistryAuth, err := r.registryAuth.Acquire() + if err != nil { + return err + } + defer releaseRegistryAuth() portMappings := make([]container.PortMapping, len(exp.Ports)) for i, p := range exp.Ports { portMappings[i] = container.PortMapping{ @@ -65,10 +71,11 @@ func (r *Reconciler) Deploy(exp agenthttp.ExpectedContainer) error { } } - _, err := container.Deploy(&container.DeployConfig{ + _, err = container.Deploy(&container.DeployConfig{ Name: exp.Name, Image: exp.Image, - RegistryInsecure: r.registryInsecure, + AuthFile: snapshot.AuthFile, + TLSVerify: snapshot.TLSVerify(exp.Image), ServiceID: exp.ServiceID, ServiceName: exp.ServiceName, DeploymentID: exp.DeploymentID, diff --git a/agent/internal/registryauth/manager.go b/agent/internal/registryauth/manager.go new file mode 100644 index 00000000..c005c426 --- /dev/null +++ b/agent/internal/registryauth/manager.go @@ -0,0 +1,433 @@ +package registryauth + +import ( + "context" + "crypto/rand" + "encoding/base64" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "log" + "net" + "os" + "path/filepath" + "regexp" + "strconv" + "strings" + "sync" + + "techulus/cloud-agent/internal/crypto" +) + +var dnsLabelPattern = regexp.MustCompile(`^[a-z0-9](?:[a-z0-9-]*[a-z0-9])?$`) + +type Registry struct { + ID string `json:"id"` + Host string `json:"host"` + AuthKey string `json:"authKey"` + Username string `json:"username"` + EncryptedPassword string `json:"encryptedPassword"` + TLSVerify bool `json:"tlsVerify"` + System bool `json:"system"` +} +type Bundle struct { + Version string `json:"version"` + Registries []Registry `json:"registries"` +} +type BundleSource interface { + GetRegistryBundle(context.Context) (*Bundle, error) +} + +type generationState struct { + Version string `json:"version"` + TLS map[string]bool `json:"tls"` +} + +type Snapshot struct { + GenerationDir, AuthFile, DockerConfigDir, EmptyHome string + tls map[string]bool + Ready bool +} + +func (s Snapshot) TLSVerify(ref string) bool { + h, err := ImageHost(ref) + if err != nil { + return true + } + v, ok := s.tls[h] + return !ok || v +} + +type Manager struct { + mu sync.Mutex + root, key string + source BundleSource + snapshot Snapshot + version string + generations map[string]*generationLease + syncRequired bool +} + +type generationLease struct { + readers int + retired bool +} + +func NewManager(dataDir, encryptionKey string, source BundleSource) (*Manager, error) { + m := &Manager{root: filepath.Join(dataDir, "registry"), key: encryptionKey, source: source, generations: map[string]*generationLease{}} + for _, d := range []string{m.root, filepath.Join(m.root, "generations"), filepath.Join(m.root, "empty-home")} { + if err := os.MkdirAll(d, 0700); err != nil { + return nil, err + } + } + if _, err := os.Stat(filepath.Join(m.root, "sync-required")); err == nil { + m.syncRequired = true + } else if !os.IsNotExist(err) { + return nil, fmt.Errorf("inspect registry synchronization marker: %w", err) + } + current, _ := filepath.EvalSymlinks(filepath.Join(m.root, "current")) + generationsDir := filepath.Join(m.root, "generations") + if current != "" && strings.HasPrefix(current, generationsDir+string(os.PathSeparator)) { + var state generationState + stateData, stateErr := os.ReadFile(filepath.Join(current, "state.json")) + configInfo, configErr := os.Stat(filepath.Join(current, "config.json")) + if stateErr == nil && configErr == nil && !configInfo.IsDir() && json.Unmarshal(stateData, &state) == nil && state.Version != "" { + m.version = state.Version + m.snapshot = Snapshot{ + GenerationDir: current, + AuthFile: filepath.Join(current, "config.json"), + DockerConfigDir: current, + EmptyHome: filepath.Join(m.root, "empty-home"), + tls: state.TLS, + Ready: true, + } + m.generations[current] = &generationLease{} + } + } + entries, err := os.ReadDir(generationsDir) + if err != nil { + return nil, fmt.Errorf("read registry generations: %w", err) + } + for _, e := range entries { + p := filepath.Join(generationsDir, e.Name()) + if e.IsDir() && p != current { + if err := os.RemoveAll(p); err != nil { + return nil, fmt.Errorf("remove stale registry generation: %w", err) + } + } + } + return m, nil +} +func (m *Manager) MarkDirty(version string) error { + m.mu.Lock() + defer m.mu.Unlock() + return m.markDirtyLocked(version) +} +func (m *Manager) markDirtyLocked(version string) error { + if version == "" { + return errors.New("registry sync version is required") + } + m.syncRequired = true + if err := writeSyncedFile(filepath.Join(m.root, "sync-required"), []byte(version), 0600); err != nil { + return err + } + return syncDir(m.root) +} +func (m *Manager) Acquire() (Snapshot, func(), error) { + m.mu.Lock() + defer m.mu.Unlock() + s := m.snapshot + s.Ready = s.Ready && !m.syncRequired + if !s.Ready { + return s, nil, errors.New("registry authentication unavailable or synchronization required") + } + lease := m.generations[s.GenerationDir] + if lease == nil || lease.retired { + return s, nil, errors.New("registry authentication generation unavailable") + } + lease.readers++ + var once sync.Once + release := func() { + once.Do(func() { + m.mu.Lock() + defer m.mu.Unlock() + if current := m.generations[s.GenerationDir]; current != nil && current.readers > 0 { + current.readers-- + } + if err := m.cleanupRetiredLocked(); err != nil { + log.Printf("[registry] failed to remove retired authentication generation: %v", err) + } + }) + } + return s, release, nil +} +func (m *Manager) Sync(ctx context.Context) error { + m.mu.Lock() + defer m.mu.Unlock() + return m.sync(ctx) +} +func (m *Manager) sync(ctx context.Context) error { + b, err := m.source.GetRegistryBundle(ctx) + if err != nil { + return fmt.Errorf("fetch registry bundle: %w", err) + } + if b == nil || b.Version == "" { + return errors.New("invalid registry bundle") + } + if b.Version == m.version && m.snapshot.GenerationDir != "" && m.snapshot.Ready { + if err := m.cleanupRetiredLocked(); err != nil { + return err + } + if err := removeIfPresent(filepath.Join(m.root, "sync-required")); err != nil { + return err + } + if err := syncDir(m.root); err != nil { + return err + } + m.syncRequired = false + m.snapshot.Ready = true + return nil + } + if err := m.markDirtyLocked(b.Version); err != nil { + return fmt.Errorf("persist required registry synchronization: %w", err) + } + auths := map[string]map[string]string{} + tls := map[string]bool{} + for _, r := range b.Registries { + h, err := CanonicalHost(r.Host) + if err != nil || h != r.Host || r.ID == "" || r.AuthKey == "" || r.Username == "" || r.EncryptedPassword == "" { + return errors.New("invalid registry bundle entry") + } + if _, ok := tls[h]; ok { + return errors.New("duplicate registry host") + } + if r.AuthKey != RegistryAuthKey(h) { + return errors.New("invalid registry authentication key") + } + if _, ok := auths[r.AuthKey]; ok { + return errors.New("duplicate registry authentication key") + } + p, err := crypto.DecryptRegistryCredential(r.EncryptedPassword, m.key, r.ID, h) + if err != nil { + return errors.New("invalid encrypted registry credential") + } + auths[r.AuthKey] = map[string]string{"auth": base64.StdEncoding.EncodeToString([]byte(r.Username + ":" + p))} + tls[h] = r.TLSVerify + } + generation, err := generationID() + if err != nil { + return fmt.Errorf("generate registry state id: %w", err) + } + dir := filepath.Join(m.root, "generations", generation) + if err := os.Mkdir(dir, 0700); err != nil { + return err + } + linked := false + defer func() { + if !linked { + _ = os.RemoveAll(dir) + } + }() + data, err := json.Marshal(struct { + Auths map[string]map[string]string `json:"auths"` + }{auths}) + if err != nil { + return err + } + if err = writeSyncedFile(filepath.Join(dir, "config.json"), data, 0600); err != nil { + return err + } + stateData, err := json.Marshal(generationState{Version: b.Version, TLS: tls}) + if err != nil { + return err + } + if err = writeSyncedFile(filepath.Join(dir, "state.json"), stateData, 0600); err != nil { + return err + } + if err = syncDir(dir); err != nil { + return err + } + tmpID, err := generationID() + if err != nil { + return fmt.Errorf("generate registry symlink id: %w", err) + } + tmp := filepath.Join(m.root, "current.tmp-"+tmpID) + if err = os.Symlink(filepath.Join("generations", filepath.Base(dir)), tmp); err != nil { + return err + } + defer os.Remove(tmp) + if err = os.Rename(tmp, filepath.Join(m.root, "current")); err != nil { + return err + } + linked = true + previousGeneration := m.snapshot.GenerationDir + m.version = b.Version + m.snapshot = Snapshot{GenerationDir: dir, AuthFile: filepath.Join(dir, "config.json"), DockerConfigDir: dir, EmptyHome: filepath.Join(m.root, "empty-home"), tls: tls, Ready: true} + m.generations[dir] = &generationLease{} + if previousGeneration != "" && previousGeneration != dir { + if previous := m.generations[previousGeneration]; previous != nil { + previous.retired = true + } + } + if err = syncDir(m.root); err != nil { + return err + } + if err = m.cleanupRetiredLocked(); err != nil { + return err + } + if err = removeIfPresent(filepath.Join(m.root, "sync-required")); err != nil { + return err + } + if err = syncDir(m.root); err != nil { + return err + } + m.syncRequired = false + return nil +} + +func (m *Manager) cleanupRetiredLocked() error { + for dir, generation := range m.generations { + if !generation.retired || generation.readers != 0 { + continue + } + if err := os.RemoveAll(dir); err != nil { + return fmt.Errorf("remove retired registry generation: %w", err) + } + delete(m.generations, dir) + } + return nil +} + +func CanonicalHost(raw string) (string, error) { + if raw == "" || raw != strings.TrimSpace(raw) || strings.Contains(raw, "://") || strings.ContainsAny(raw, "/?#@") { + return "", errors.New("invalid registry host") + } + s := strings.ToLower(raw) + if s == "index.docker.io" || s == "registry-1.docker.io" { + s = "docker.io" + } + var hostname string + if strings.HasPrefix(s, "[") { + end := strings.IndexByte(s, ']') + if end < 0 || net.ParseIP(s[1:end]) == nil || (len(s) > end+1 && (s[end+1] != ':' || !validPort(s[end+2:]))) { + return "", errors.New("invalid registry host") + } + hostname = s[1:end] + } else if host, port, ok := strings.Cut(s, ":"); ok { + if host == "" || !validPort(port) || strings.Contains(host, ":") { + return "", errors.New("invalid registry host") + } + hostname = host + } else { + hostname = s + } + if strings.ContainsAny(s, " \\") { + return "", errors.New("invalid registry host") + } + if net.ParseIP(hostname) == nil { + if len(hostname) > 253 { + return "", errors.New("invalid registry host") + } + for _, label := range strings.Split(hostname, ".") { + if len(label) > 63 || !dnsLabelPattern.MatchString(label) { + return "", errors.New("invalid registry host") + } + } + } + return s, nil +} +func validPort(s string) bool { + p, err := strconv.Atoi(s) + return err == nil && p > 0 && p <= 65535 && strconv.Itoa(p) == s +} +func RegistryAuthKey(host string) string { + if host == "docker.io" { + return "https://index.docker.io/v1/" + } + return host +} +func ImageHost(ref string) (string, error) { + s := strings.TrimSpace(ref) + if s == "" { + return "", errors.New("empty image") + } + first := strings.SplitN(s, "/", 2)[0] + if !strings.Contains(s, "/") || (!strings.Contains(first, ".") && !strings.Contains(first, ":") && first != "localhost" && !strings.HasPrefix(first, "[")) { + return "docker.io", nil + } + return CanonicalHost(first) +} +func NormalizeImage(ref string) string { + h, e := ImageHost(ref) + if e != nil { + return ref + } + s := ref + first := strings.SplitN(s, "/", 2) + if h == "docker.io" { + if len(first) == 1 { + s = "library/" + s + } else if first[0] == "docker.io" || first[0] == "index.docker.io" || first[0] == "registry-1.docker.io" { + s = first[1] + } + if !strings.Contains(s, "/") { + s = "library/" + s + } + s = "docker.io/" + s + } + at := strings.Index(s, "@") + name := s + if at >= 0 { + name = s[:at] + } + slash := strings.LastIndex(name, "/") + if at < 0 && strings.LastIndex(name, ":") <= slash { + s += ":latest" + } + return s +} + +func generationID() (string, error) { + var b [16]byte + if _, err := rand.Read(b[:]); err != nil { + return "", err + } + return hex.EncodeToString(b[:]), nil +} + +func writeSyncedFile(path string, data []byte, mode os.FileMode) error { + f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, mode) + if err != nil { + return err + } + if _, err = f.Write(data); err == nil { + err = f.Sync() + } + closeErr := f.Close() + if err != nil { + return err + } + return closeErr +} + +func syncDir(path string) error { + dir, err := os.Open(path) + if err != nil { + return err + } + err = dir.Sync() + closeErr := dir.Close() + if err != nil { + return err + } + return closeErr +} + +func removeIfPresent(path string) error { + err := os.Remove(path) + if os.IsNotExist(err) { + return nil + } + return err +} diff --git a/agent/internal/registryauth/manager_test.go b/agent/internal/registryauth/manager_test.go new file mode 100644 index 00000000..ad7c10bd --- /dev/null +++ b/agent/internal/registryauth/manager_test.go @@ -0,0 +1,175 @@ +package registryauth + +import ( + "context" + "crypto/aes" + "crypto/cipher" + "encoding/base64" + "encoding/hex" + "encoding/json" + "errors" + "os" + "path/filepath" + "testing" +) + +type fakeSource struct { + bundle *Bundle + err error +} + +func (f *fakeSource) GetRegistryBundle(context.Context) (*Bundle, error) { return f.bundle, f.err } + +func encrypt(t *testing.T, key []byte, id, host, password string) string { + t.Helper() + block, _ := aes.NewCipher(key) + gcm, _ := cipher.NewGCM(block) + iv := make([]byte, gcm.NonceSize()) + sealed := gcm.Seal(nil, iv, []byte(password), []byte("registry-credential:v1\x00"+id+"\x00"+host)) + framed := append(append([]byte{}, iv...), sealed[len(sealed)-gcm.Overhead():]...) + framed = append(framed, sealed[:len(sealed)-gcm.Overhead()]...) + return base64.StdEncoding.EncodeToString(framed) +} + +func TestReferenceVectors(t *testing.T) { + tests := map[string]string{"alpine": "docker.io", "library/alpine": "docker.io", "index.docker.io/library/alpine": "docker.io", "registry-1.docker.io/x/y": "docker.io", "localhost:5000/x": "localhost:5000", "[::1]:5000/x": "[::1]:5000", "ghcr.io/x/y": "ghcr.io"} + for ref, want := range tests { + got, err := ImageHost(ref) + if err != nil || got != want { + t.Errorf("ImageHost(%q)=%q,%v want %q", ref, got, err, want) + } + } +} + +func TestManagerInstallDirtyAndPermissions(t *testing.T) { + key := make([]byte, 32) + source := &fakeSource{} + dir := t.TempDir() + m, err := NewManager(dir, hex.EncodeToString(key), source) + if err != nil { + t.Fatal(err) + } + if err := m.MarkDirty("v1"); err != nil { + t.Fatal(err) + } + if _, _, err := m.Acquire(); err == nil { + t.Fatal("dirty manager was ready") + } + source.bundle = &Bundle{Version: "v1", Registries: []Registry{{ID: "id", Host: "docker.io", AuthKey: "https://index.docker.io/v1/", Username: "user", EncryptedPassword: encrypt(t, key, "id", "docker.io", "pass"), TLSVerify: false}}} + if err := m.Sync(context.Background()); err != nil { + t.Fatal(err) + } + snapshot, release, err := m.Acquire() + if err != nil { + t.Fatal(err) + } + release() + var cfg struct { + Auths map[string]struct { + Auth string `json:"auth"` + } `json:"auths"` + } + data, _ := os.ReadFile(snapshot.AuthFile) + if err := json.Unmarshal(data, &cfg); err != nil { + t.Fatal(err) + } + if got := cfg.Auths["https://index.docker.io/v1/"].Auth; got != base64.StdEncoding.EncodeToString([]byte("user:pass")) { + t.Fatal("wrong managed auth") + } + for _, p := range []string{snapshot.GenerationDir, snapshot.AuthFile, filepath.Join(snapshot.GenerationDir, "state.json")} { + info, err := os.Stat(p) + if err != nil { + t.Fatal(err) + } + want := os.FileMode(0700) + if !info.IsDir() { + want = 0600 + } + if info.Mode().Perm() != want { + t.Errorf("%s mode %o want %o", p, info.Mode().Perm(), want) + } + } + + restarted, err := NewManager(dir, hex.EncodeToString(key), &fakeSource{err: errors.New("offline")}) + if err != nil { + t.Fatal(err) + } + restartedSnapshot, releaseRestarted, err := restarted.Acquire() + if err != nil { + t.Fatalf("cached generation unavailable after restart: %v", err) + } + defer releaseRestarted() + if restartedSnapshot.TLSVerify("docker.io/library/alpine") { + t.Fatal("cached TLS policy was not restored") + } +} + +func TestManagerRetainsLeasedGenerationUntilRelease(t *testing.T) { + key := make([]byte, 32) + source := &fakeSource{bundle: &Bundle{ + Version: "v1", + Registries: []Registry{{ + ID: "id", + Host: "registry.example.com", + AuthKey: "registry.example.com", + Username: "user", + EncryptedPassword: encrypt(t, key, "id", "registry.example.com", "first"), + TLSVerify: true, + }}, + }} + m, err := NewManager(t.TempDir(), hex.EncodeToString(key), source) + if err != nil { + t.Fatal(err) + } + if err := m.Sync(context.Background()); err != nil { + t.Fatal(err) + } + old, release, err := m.Acquire() + if err != nil { + t.Fatal(err) + } + source.bundle = &Bundle{ + Version: "v2", + Registries: []Registry{{ + ID: "id", + Host: "registry.example.com", + AuthKey: "registry.example.com", + Username: "user", + EncryptedPassword: encrypt(t, key, "id", "registry.example.com", "second"), + TLSVerify: true, + }}, + } + if err := m.Sync(context.Background()); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(old.GenerationDir); err != nil { + t.Fatalf("leased generation was removed early: %v", err) + } + release() + if _, err := os.Stat(old.GenerationDir); !os.IsNotExist(err) { + t.Fatalf("retired generation was not removed after release: %v", err) + } +} + +func TestManagerFailsClosedWhenDirtyMarkerCannotBePersisted(t *testing.T) { + key := make([]byte, 32) + source := &fakeSource{bundle: &Bundle{Version: "v1"}} + dataDir := t.TempDir() + m, err := NewManager(dataDir, hex.EncodeToString(key), source) + if err != nil { + t.Fatal(err) + } + if err := m.Sync(context.Background()); err != nil { + t.Fatal(err) + } + marker := filepath.Join(dataDir, "registry", "sync-required") + if err := os.Mkdir(marker, 0700); err != nil { + t.Fatal(err) + } + if err := m.MarkDirty("v2"); err == nil { + t.Fatal("dirty marker persistence unexpectedly succeeded") + } + if _, _, err := m.Acquire(); err == nil { + t.Fatal("authentication remained available after dirty marker failure") + } +} diff --git a/docs/infrastructure/registry.mdx b/docs/infrastructure/registry.mdx index b3fb465d..ed2320ad 100644 --- a/docs/infrastructure/registry.mdx +++ b/docs/infrastructure/registry.mdx @@ -21,11 +21,25 @@ The registry runs as a Docker container alongside the control plane, available a | Variable | Description | | --- | --- | +| `REGISTRY_HOST` / `REGISTRY_URL` | Registry endpoint aliases (an optional `http://` or `https://` scheme is accepted; paths are not) | | `REGISTRY_USERNAME` | Basic auth username | | `REGISTRY_PASSWORD` | Basic auth password | +| `REGISTRY_INSECURE` | Set to `true` to disable TLS certificate verification on agents | | `REGISTRY_HTTP_SECRET` | Internal HTTP signing secret | -Agents receive registry credentials automatically during [registration](/agents/setup#registration). +The built-in credentials are included in the same encrypted registry bundle as custom credentials. Registered agents fetch that complete bundle over a signed, non-cacheable endpoint; registration itself does not return registry passwords. + +## Global custom registries + +Administrators can add private registry credentials under **Settings → Registries**. A registry host (including an explicit port, when required) is globally unique and cannot include a scheme or path. Docker Hub aliases are treated as the same host, and hosts used by the built-in registry configuration are reserved. + +Registry changes are transactionally queued for every registered agent, including offline agents. Rotating a password replaces the encrypted credential while preserving the host; deleting an entry removes it from the next complete bundle. Use pull-only robot or service-account tokens wherever possible. + +Passwords are write-only in the control plane UI. The control plane validates image reference syntax but neither decrypts custom registry passwords nor contacts registries to test them. An agent's actual image pull is the authoritative credential and image availability check. + +TLS certificate verification is enabled by default. Disable it only for a deliberately trusted registry using a private or self-signed certificate: doing so allows credential interception if the network is compromised. + +The TLS setting applies to agent-managed Podman pulls, build exports, and manifest operations. Dockerfile `FROM` pulls are resolved by the separate BuildKit daemon; using an insecure registry there also requires configuring that host in BuildKit's daemon configuration. Credentials for TLS-verified registries are forwarded to BuildKit automatically. ## Storage diff --git a/web/actions/compose.ts b/web/actions/compose.ts index 9778f40e..ed7afa64 100644 --- a/web/actions/compose.ts +++ b/web/actions/compose.ts @@ -99,7 +99,7 @@ export async function importCompose( if (!validation.valid) { errors.push({ service: validation.service, - message: `Invalid image '${validation.image}': ${validation.error || "Image not found"}`, + message: `Invalid image syntax '${validation.image}': ${validation.error || "Invalid image reference"}`, }); } } diff --git a/web/actions/projects.ts b/web/actions/projects.ts index e20c8e95..b034026b 100644 --- a/web/actions/projects.ts +++ b/web/actions/projects.ts @@ -49,6 +49,7 @@ import { validateDockerImageInternal } from "@/lib/docker-image"; import { inngest } from "@/lib/inngest/client"; import { inngestEvents } from "@/lib/inngest/events"; import { allocatePort } from "@/lib/port-allocation"; +import { resolveRegistryImageHost } from "@/lib/registry-reference"; import { cleanupRegistryArtifactsForService, prepareRegistryArtifactCleanup, @@ -303,6 +304,12 @@ const SERVICE_CARD_WIDTH = 320; export async function createService(input: CreateServiceInput) { await requireDeveloperRole(); const { projectId, environmentId, name, image, github } = input; + if (!github) { + const validation = await validateDockerImageInternal(image); + if (!validation.valid) { + throw new Error(validation.error ?? "Invalid image reference"); + } + } const resourceLimits = input.resourceLimits ?? { cpuCores: null, memoryMb: null, @@ -334,10 +341,7 @@ export async function createService(input: CreateServiceInput) { let githubRootDir: string | null = null; if (github) { - const registryHost = process.env.REGISTRY_HOST; - if (!registryHost) { - throw new Error("REGISTRY_HOST environment variable is required"); - } + const registryHost = resolveRegistryImageHost(); finalImage = `${registryHost}/${projectId}/${id}:latest`; sourceType = "github"; githubRepoUrl = github.repoUrl; @@ -877,10 +881,7 @@ export async function updateServiceGithubRepo( }; if (normalizedUrl) { - const registryHost = process.env.REGISTRY_HOST; - if (!registryHost) { - throw new Error("REGISTRY_HOST environment variable is required"); - } + const registryHost = resolveRegistryImageHost(); updateData.image = `${registryHost}/${service.projectId}/${serviceId}:latest`; } @@ -1222,6 +1223,12 @@ export async function updateServiceConfig( if (!service) { throw new Error("Service not found"); } + if (config.source) { + const validation = await validateDockerImageInternal(config.source.image); + if (!validation.valid) { + throw new Error(validation.error ?? "Invalid image reference"); + } + } if (config.source || config.healthCheck !== undefined) { await db.transaction(async (tx) => { diff --git a/web/actions/registry-credentials.ts b/web/actions/registry-credentials.ts new file mode 100644 index 00000000..01590d37 --- /dev/null +++ b/web/actions/registry-credentials.ts @@ -0,0 +1,125 @@ +"use server"; + +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { revalidatePath } from "next/cache"; +import { db } from "@/db"; +import { registryCredentials } from "@/db/schema"; +import { requireAdminRole } from "@/lib/auth"; +import { encryptRegistryPassword } from "@/lib/crypto"; +import { + calculateRegistryBundleVersion, + getReservedSystemRegistryHosts, +} from "@/lib/registry-credentials"; +import { canonicalizeRegistryHost } from "@/lib/registry-reference"; +import { enqueueRegistrySyncForAllRegisteredServers } from "@/lib/work-queue"; + +export type RegistryCredentialInput = { + host: string; + username: string; + password: string; + tlsVerify?: boolean; +}; + +async function requireAdmin() { + if (!(await requireAdminRole())) throw new Error("Unauthorized"); +} + +function validateUsername(username: string) { + const hasControlCharacter = [...username].some((character) => { + const codePoint = character.codePointAt(0) ?? 0; + return codePoint <= 0x1f || codePoint === 0x7f; + }); + if ( + !username || + username !== username.trim() || + username.length > 255 || + username.includes(":") || + hasControlCharacter + ) + throw new Error("Registry username is required"); +} + +async function mutateAndFanout( + mutation: ( + tx: Parameters[0]>[0], + ) => Promise, +) { + await db.transaction(async (tx) => { + await mutation(tx); + const rows = await tx.select().from(registryCredentials); + await enqueueRegistrySyncForAllRegisteredServers( + await calculateRegistryBundleVersion(rows), + tx, + ); + }); + revalidatePath("/dashboard/settings"); + return { success: true }; +} + +export async function createRegistryCredential(input: RegistryCredentialInput) { + await requireAdmin(); + const host = canonicalizeRegistryHost(input.host); + validateUsername(input.username); + if (!input.password) throw new Error("Registry password is required"); + if (getReservedSystemRegistryHosts().has(host)) + throw new Error("This host is reserved by the built-in registry"); + const id = randomUUID(); + const encryptedPassword = await encryptRegistryPassword( + input.password, + id, + host, + ); + return mutateAndFanout(async (tx) => { + await tx.insert(registryCredentials).values({ + id, + host, + username: input.username, + encryptedPassword, + tlsVerify: input.tlsVerify ?? true, + }); + }); +} + +export async function updateRegistryCredential( + id: string, + input: { username: string; password?: string; tlsVerify: boolean }, +) { + await requireAdmin(); + validateUsername(input.username); + return mutateAndFanout(async (tx) => { + const existing = await tx + .select() + .from(registryCredentials) + .where(eq(registryCredentials.id, id)) + .then((rows) => rows[0]); + if (!existing) throw new Error("Registry credential not found"); + await tx + .update(registryCredentials) + .set({ + username: input.username, + tlsVerify: input.tlsVerify, + ...(input.password + ? { + encryptedPassword: await encryptRegistryPassword( + input.password, + id, + existing.host, + ), + } + : {}), + }) + .where(eq(registryCredentials.id, id)); + }); +} + +export async function deleteRegistryCredential(id: string) { + await requireAdmin(); + return mutateAndFanout(async (tx) => { + const deleted = await tx + .delete(registryCredentials) + .where(eq(registryCredentials.id, id)) + .returning({ id: registryCredentials.id }); + if (!deleted.length) throw new Error("Registry credential not found"); + }); +} diff --git a/web/app/(dashboard)/dashboard/settings/page.tsx b/web/app/(dashboard)/dashboard/settings/page.tsx index 892927d1..83f33650 100644 --- a/web/app/(dashboard)/dashboard/settings/page.tsx +++ b/web/app/(dashboard)/dashboard/settings/page.tsx @@ -2,7 +2,23 @@ import { listMembers } from "@/actions/members"; import { SetBreadcrumbs } from "@/components/core/breadcrumb-data"; import { GlobalSettings } from "@/components/settings/global-settings"; import { getGlobalSettings, listServers } from "@/db/queries"; +import { requireAdminRole } from "@/lib/auth"; import { AdminNotConfiguredError } from "@/lib/members"; +import { listRegistryMetadata } from "@/lib/registry-credentials"; + +async function getRegistryData() { + try { + if (!(await requireAdminRole())) return null; + return await listRegistryMetadata(); + } catch (error) { + if ( + error instanceof Error && + (error.message === "Unauthorized" || error.message === "Forbidden") + ) + return null; + throw error; + } +} async function getMembersData() { try { @@ -32,10 +48,11 @@ async function getMembersData() { } export default async function SettingsPage() { - const [servers, settings, membersData] = await Promise.all([ + const [servers, settings, membersData, registries] = await Promise.all([ listServers(), getGlobalSettings(), getMembersData(), + getRegistryData(), ]); return ( @@ -58,6 +75,7 @@ export default async function SettingsPage() { servers={servers} membersData={membersData} initialSettings={settings} + registries={registries} appVersion={ process.env.TECHULUS_CLOUD_VERSION ?? process.env.NEXT_PUBLIC_APP_VERSION ?? diff --git a/web/app/api/v1/agent/register/route.ts b/web/app/api/v1/agent/register/route.ts index a06bc793..4a095b87 100644 --- a/web/app/api/v1/agent/register/route.ts +++ b/web/app/api/v1/agent/register/route.ts @@ -112,10 +112,6 @@ export async function POST(request: NextRequest) { encryptionKey, loggingEndpoint: process.env.VICTORIA_LOGS_URL ?? null, metricsEndpoint: process.env.VICTORIA_METRICS_URL ?? null, - registryUrl: process.env.REGISTRY_URL ?? null, - registryUsername: process.env.REGISTRY_USERNAME ?? null, - registryPassword: process.env.REGISTRY_PASSWORD ?? null, - registryInsecure: process.env.REGISTRY_INSECURE === "true", }); } catch (error) { console.error("Agent registration error:", error); diff --git a/web/app/api/v1/agent/registries/route.ts b/web/app/api/v1/agent/registries/route.ts new file mode 100644 index 00000000..dab33ffd --- /dev/null +++ b/web/app/api/v1/agent/registries/route.ts @@ -0,0 +1,28 @@ +import { type NextRequest, NextResponse } from "next/server"; +import { verifyAgentRequest } from "@/lib/agent-auth"; +import { getRegistryBundle } from "@/lib/registry-credentials"; + +const PRIVATE_HEADERS = { + "Cache-Control": "private, no-store", + Pragma: "no-cache", +}; + +export async function GET(request: NextRequest) { + const auth = await verifyAgentRequest(request); + if (!auth.success) + return NextResponse.json( + { error: auth.error }, + { status: auth.status, headers: PRIVATE_HEADERS }, + ); + try { + return NextResponse.json(await getRegistryBundle(), { + headers: PRIVATE_HEADERS, + }); + } catch (error) { + console.error("Registry bundle error:", error); + return NextResponse.json( + { error: "Registry credentials unavailable" }, + { status: 503, headers: PRIVATE_HEADERS }, + ); + } +} diff --git a/web/components/settings/global-settings.tsx b/web/components/settings/global-settings.tsx index e131fd23..f60e6972 100644 --- a/web/components/settings/global-settings.tsx +++ b/web/components/settings/global-settings.tsx @@ -30,6 +30,7 @@ import { } from "@/components/settings/edge-domain-settings"; import { EmailSettings } from "@/components/settings/email-settings"; import { MemberSettings } from "@/components/settings/member-settings"; +import { RegistrySettings } from "@/components/settings/registry-settings"; import { TwoFactorSettings } from "@/components/settings/two-factor-settings"; import { Badge } from "@/components/ui/badge"; import { Button } from "@/components/ui/button"; @@ -61,6 +62,7 @@ import type { ControlPlaneUpdateState, ControlPlaneUpgradeState, } from "@/lib/control-plane-updates"; +import type { RegistryMetadata } from "@/lib/registry-credentials"; import type { EmailAlertsConfig } from "@/lib/settings-keys"; type Props = { @@ -93,6 +95,7 @@ type Props = { controlPlaneUpgradeState: ControlPlaneUpgradeState | null; }; appVersion: string | null; + registries: RegistryMetadata[] | null; }; const CONTROL_PLANE_UPGRADE_DOCS_URL = @@ -103,6 +106,7 @@ export function GlobalSettings({ membersData, initialSettings, appVersion, + registries, }: Props) { const router = useRouter(); const [tab, setTab] = useQueryState("tab", { @@ -274,6 +278,11 @@ export function GlobalSettings({ Members )} + {registries && ( + + Registries + + )} Update @@ -460,6 +469,12 @@ export function GlobalSettings({ )} + {registries && ( + + + + )} +
diff --git a/web/components/settings/registry-settings.tsx b/web/components/settings/registry-settings.tsx new file mode 100644 index 00000000..d4aec126 --- /dev/null +++ b/web/components/settings/registry-settings.tsx @@ -0,0 +1,214 @@ +"use client"; + +import { useState } from "react"; +import { toast } from "sonner"; +import { + createRegistryCredential, + deleteRegistryCredential, + updateRegistryCredential, +} from "@/actions/registry-credentials"; +import { Button } from "@/components/ui/button"; +import { Input } from "@/components/ui/input"; +import { Label } from "@/components/ui/label"; +import type { RegistryMetadata } from "@/lib/registry-credentials"; + +export function RegistrySettings({ + registries, +}: { + registries: RegistryMetadata[]; +}) { + const [host, setHost] = useState(""); + const [username, setUsername] = useState(""); + const [password, setPassword] = useState(""); + const [tlsVerify, setTlsVerify] = useState(true); + const [busy, setBusy] = useState(false); + + async function create() { + setBusy(true); + try { + await createRegistryCredential({ host, username, password, tlsVerify }); + toast.success("Registry added"); + window.location.reload(); + } catch (error) { + toast.error( + error instanceof Error ? error.message : "Failed to add registry", + ); + setBusy(false); + } + } + + return ( +
+
+
+

Add registry credentials

+

+ Credentials are distributed globally to every registered agent. The + password is write-only. +

+
+
+
+ + setHost(event.target.value)} + placeholder="registry.example.com:5000" + /> +
+
+ + setUsername(event.target.value)} + /> +
+
+ + setPassword(event.target.value)} + /> +
+
+ + {!tlsVerify && ( +

+ Warning: disabling TLS verification exposes registry credentials to + interception. +

+ )} + +
+
+ {registries.length === 0 && ( +

+ No registry credentials configured. +

+ )} + {registries.map((registry) => ( + + ))} +
+
+ ); +} + +function RegistryRow({ registry }: { registry: RegistryMetadata }) { + const [username, setUsername] = useState(registry.username); + const [password, setPassword] = useState(""); + const [tlsVerify, setTlsVerify] = useState(registry.tlsVerify); + const [busy, setBusy] = useState(false); + if (registry.system) + return ( +
+
+
+

{registry.host}

+

+ Built-in registry · {registry.username} · TLS verification{" "} + {registry.tlsVerify ? "enabled" : "disabled"} +

+
+ Read-only +
+
+ ); + async function save() { + setBusy(true); + try { + await updateRegistryCredential(registry.id, { + username, + password: password || undefined, + tlsVerify, + }); + toast.success("Registry updated"); + window.location.reload(); + } catch (error) { + toast.error( + error instanceof Error ? error.message : "Failed to update registry", + ); + setBusy(false); + } + } + async function remove() { + if ( + !window.confirm( + `Delete credentials for ${registry.host}? Existing image pulls may fail.`, + ) + ) + return; + setBusy(true); + try { + await deleteRegistryCredential(registry.id); + toast.success("Registry deleted"); + window.location.reload(); + } catch (error) { + toast.error( + error instanceof Error ? error.message : "Failed to delete registry", + ); + setBusy(false); + } + } + return ( +
+

{registry.host}

+
+ setUsername(event.target.value)} + /> + setPassword(event.target.value)} + /> +
+ + {!tlsVerify && ( +

+ TLS verification is disabled. +

+ )} +
+ + +
+
+ ); +} diff --git a/web/db/schema.ts b/web/db/schema.ts index e90100d8..95e4e01c 100644 --- a/web/db/schema.ts +++ b/web/db/schema.ts @@ -355,6 +355,25 @@ export const servers = pgTable("servers", { .notNull(), }); +export const registryCredentials = pgTable( + "registry_credentials", + { + id: text("id").primaryKey(), + host: text("host").notNull(), + username: text("username").notNull(), + encryptedPassword: text("encrypted_password").notNull(), + tlsVerify: boolean("tls_verify").default(true).notNull(), + createdAt: timestamp("created_at", { withTimezone: true }) + .defaultNow() + .notNull(), + updatedAt: timestamp("updated_at", { withTimezone: true }) + .defaultNow() + .$onUpdate(() => new Date()) + .notNull(), + }, + (table) => [uniqueIndex("registry_credentials_host_idx").on(table.host)], +); + export const projects = pgTable("projects", { id: text("id").primaryKey(), name: text("name").notNull(), @@ -781,6 +800,7 @@ export const workQueue = pgTable( "restore_volume", "create_manifest", "upgrade_agent", + "sync_registries", ], }).notNull(), payload: text("payload").notNull(), @@ -806,6 +826,11 @@ export const workQueue = pgTable( .where( sql`${table.type} = 'upgrade_agent' AND ${table.status} IN ('pending', 'processing')`, ), + uniqueIndex("work_queue_one_pending_registry_sync_idx") + .on(table.serverId) + .where( + sql`${table.type} = 'sync_registries' AND ${table.status} = 'pending'`, + ), ], ); diff --git a/web/db/types.ts b/web/db/types.ts index 4860a6f8..1cc45813 100644 --- a/web/db/types.ts +++ b/web/db/types.ts @@ -6,6 +6,7 @@ import type { environments, memberInvitations, projects, + registryCredentials, rollouts, secrets, servers, @@ -30,6 +31,7 @@ export type DeploymentPort = typeof deploymentPorts.$inferSelect; export type Rollout = typeof rollouts.$inferSelect; export type Build = typeof builds.$inferSelect; export type WorkQueue = typeof workQueue.$inferSelect; +export type RegistryCredential = typeof registryCredentials.$inferSelect; export type User = typeof user.$inferSelect; export type MemberInvitation = typeof memberInvitations.$inferSelect; export type MemberRole = User["role"]; diff --git a/web/lib/agent-auth.ts b/web/lib/agent-auth.ts index 37d756c9..86ddecf5 100644 --- a/web/lib/agent-auth.ts +++ b/web/lib/agent-auth.ts @@ -54,7 +54,8 @@ export async function verifyAgentRequest( }; } - const messageToVerify = `${timestamp}:${body ?? ""}`; + const url = new URL(request.url); + const messageToVerify = `agent-request:v2\0${timestamp}\0${request.method}\0${url.pathname}${url.search}\0${body ?? ""}`; const isValid = verifyEd25519Signature( server.signingPublicKey, messageToVerify, diff --git a/web/lib/agent/expected-state.ts b/web/lib/agent/expected-state.ts index fc05f4d5..8916efe7 100644 --- a/web/lib/agent/expected-state.ts +++ b/web/lib/agent/expected-state.ts @@ -15,6 +15,7 @@ import { observedReadyPhases, runtimeExpectedStates, } from "@/lib/deployment-status"; +import { normalizeImageReference } from "@/lib/registry-reference"; import { selectRoutingSyncRolloutIds } from "@/lib/routing-sync"; import { parseServiceRevisionSpec } from "@/lib/service-revision-changes"; import { @@ -830,13 +831,7 @@ function compareServerlessUpstreams( } function normalizeImage(image: string) { - if (!image.includes("/")) { - return `docker.io/library/${image}`; - } - if (!image.includes(".") && image.split("/").length === 2) { - return `docker.io/${image}`; - } - return image; + return normalizeImageReference(image); } export function buildRuntimeRoutePorts( diff --git a/web/lib/crypto.ts b/web/lib/crypto.ts index eb55eee0..4d067f88 100644 --- a/web/lib/crypto.ts +++ b/web/lib/crypto.ts @@ -11,6 +11,29 @@ const ALGORITHM = "aes-256-gcm"; const IV_LENGTH = 12; const AUTH_TAG_LENGTH = 16; +export function registryCredentialAad( + id: string, + canonicalHost: string, +): Buffer { + return Buffer.from(`registry-credential:v1\0${id}\0${canonicalHost}`, "utf8"); +} + +export async function encryptRegistryPassword( + plaintext: string, + id: string, + canonicalHost: string, +): Promise { + const key = await resolveEncryptionKey(); + const iv = randomBytes(IV_LENGTH); + const cipher = createCipheriv(ALGORITHM, key, iv); + cipher.setAAD(registryCredentialAad(id, canonicalHost)); + const encrypted = Buffer.concat([ + cipher.update(plaintext, "utf8"), + cipher.final(), + ]); + return Buffer.concat([iv, cipher.getAuthTag(), encrypted]).toString("base64"); +} + export async function encryptSecret(plaintext: string): Promise { const key = await resolveEncryptionKey(); const iv = randomBytes(IV_LENGTH); diff --git a/web/lib/docker-image.ts b/web/lib/docker-image.ts index 0fc90305..e664bdda 100644 --- a/web/lib/docker-image.ts +++ b/web/lib/docker-image.ts @@ -1,265 +1,33 @@ +import { parseImageReference } from "@/lib/registry-reference"; + export function imageUsesMutableReference(image: string): boolean { if (image.includes("@")) return false; - const lastSlash = image.lastIndexOf("/"); const lastColon = image.lastIndexOf(":"); - if (lastColon <= lastSlash) return true; - - return image.slice(lastColon + 1) === "latest"; + return lastColon <= lastSlash || image.slice(lastColon + 1) === "latest"; } export function imageIsUnqualified(image: string): boolean { - const imageWithoutDigest = image.split("@")[0]; - return !imageWithoutDigest.includes("/"); + return !image.split("@")[0]?.includes("/"); } export function imageNeedsProductionPinning(image: string): boolean { return image !== "" && imageUsesMutableReference(image); } -const DOCKER_AUTH_URL = "https://auth.docker.io/token"; -const DOCKER_MANIFEST_BASE = "https://registry-1.docker.io/v2"; -const DOCKER_TAGS_BASE = "https://hub.docker.com/v2/repositories"; -const GHCR_TOKEN_URL = "https://ghcr.io/token"; -const GHCR_MANIFEST_BASE = "https://ghcr.io/v2"; -const MANIFEST_ACCEPT = - "application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json"; - -function isValidImageReferencePart(reference: string): boolean { - const tagPattern = /^[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}$/; - const digestPattern = /^[A-Za-z0-9_+.-]+:[0-9a-fA-F]{32,256}$/; - - return ( - reference === "latest" || - tagPattern.test(reference) || - digestPattern.test(reference) - ); -} - -function isValidImageNamePart(part: string): boolean { - const segmentPattern = /^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$/; - return part.split("/").every((segment) => segmentPattern.test(segment)); -} - -function isValidRegistry(registry: string): boolean { - if (!/^(?:\[[0-9A-Fa-f:.]+\]|[A-Za-z0-9.-]+)(?::[0-9]+)?$/.test(registry)) { - return false; - } - try { - const url = new URL(`https://${registry}`); - return ( - url.protocol === "https:" && - url.username === "" && - url.password === "" && - url.pathname === "/" && - url.search === "" && - url.hash === "" && - url.hostname !== "" - ); - } catch { - return false; - } -} - -function encodePathSegments(value: string): string { - return value - .split("/") - .map((segment) => encodeURIComponent(segment)) - .join("/"); -} - -function getBearerToken(value: unknown): string | null { - if (value === null || typeof value !== "object" || Array.isArray(value)) { - return null; - } - const record = value as Record; - for (const candidate of [record.token, record.access_token]) { - if ( - typeof candidate === "string" && - candidate.trim() === candidate && - candidate.length > 0 && - !candidate.includes("\r") && - !candidate.includes("\n") - ) { - return candidate; - } - } - return null; -} - -async function readBearerToken(response: Response): Promise { - try { - const value: unknown = await response.json(); - return getBearerToken(value); - } catch { - return null; - } -} - -function parseImageReference(image: string): { - registry: string; - repositoryPath: string; - tag: string | null; - digest: string | null; -} { - let registry = "docker.io"; - let tag: string | null = null; - let digest: string | null = null; - let imagePath = image; - - const digestIndex = imagePath.indexOf("@"); - if (digestIndex !== -1) { - digest = imagePath.substring(digestIndex + 1); - imagePath = imagePath.substring(0, digestIndex); - } - const tagIndex = imagePath.lastIndexOf(":"); - if (tagIndex > imagePath.lastIndexOf("/")) { - tag = imagePath.substring(tagIndex + 1); - imagePath = imagePath.substring(0, tagIndex); - } else if (!digest) { - tag = "latest"; - } - - const parts = imagePath.split("/"); - const first = parts[0] ?? ""; - const hasExplicitRegistry = - parts.length > 1 && - (first.toLowerCase() === "localhost" || - first.includes(".") || - first.includes(":") || - first.toLowerCase() !== first); - const nameParts = hasExplicitRegistry ? parts.slice(1) : parts; - if (hasExplicitRegistry) { - registry = - first.toLowerCase() === "index.docker.io" - ? "docker.io" - : first.toLowerCase(); - } - let repositoryPath = nameParts.join("/"); - if (registry === "docker.io" && nameParts.length === 1) { - repositoryPath = `library/${repositoryPath}`; - } - - return { registry, repositoryPath, tag, digest }; -} - export async function validateDockerImageInternal( image: string, ): Promise<{ valid: boolean; error?: string }> { try { - const { registry, repositoryPath, tag, digest } = - parseImageReference(image); - const reference = digest || tag || "latest"; - - if ( - !isValidImageReferencePart(reference) || - (tag !== null && !isValidImageReferencePart(tag)) - ) { - return { valid: false, error: "Invalid image tag or digest" }; - } - if (!isValidRegistry(registry) || !isValidImageNamePart(repositoryPath)) { - return { valid: false, error: "Invalid image name" }; - } - const encodedRepositoryPath = encodePathSegments(repositoryPath); - const encodedReference = encodeURIComponent(reference); - - if (registry === "docker.io") { - if (digest) { - const tokenUrl = new URL(DOCKER_AUTH_URL); - tokenUrl.search = new URLSearchParams({ - service: "registry.docker.io", - scope: `repository:${repositoryPath}:pull`, - }).toString(); - const tokenResponse = await fetch(tokenUrl, { redirect: "error" }); - if (!tokenResponse.ok) { - return { - valid: false, - error: "Failed to authenticate with Docker Hub", - }; - } - const token = await readBearerToken(tokenResponse); - if (!token) { - return { - valid: false, - error: "Failed to authenticate with Docker Hub", - }; - } - const manifestUrl = `${DOCKER_MANIFEST_BASE}/${encodedRepositoryPath}/manifests/${encodedReference}`; - const manifestResponse = await fetch(manifestUrl, { - redirect: "error", - headers: { - Authorization: `Bearer ${token}`, - Accept: MANIFEST_ACCEPT, - }, - }); - if (manifestResponse.status === 404) { - return { - valid: false, - error: "Image digest not found on Docker Hub", - }; - } - if (!manifestResponse.ok) { - return { valid: false, error: "Failed to validate image" }; - } - return { valid: true }; - } - - const url = `${DOCKER_TAGS_BASE}/${encodedRepositoryPath}/tags/${encodedReference}`; - const response = await fetch(url, { - method: "GET", - redirect: "error", - }); - if (response.status === 404) { - return { valid: false, error: "Image or tag not found on Docker Hub" }; - } - if (!response.ok) { - return { valid: false, error: "Failed to validate image" }; - } - return { valid: true }; - } - - if (registry === "ghcr.io") { - const tokenUrl = new URL(GHCR_TOKEN_URL); - tokenUrl.search = new URLSearchParams({ - scope: `repository:${repositoryPath}:pull`, - }).toString(); - const tokenResponse = await fetch(tokenUrl, { redirect: "error" }); - if (!tokenResponse.ok) { - return { - valid: false, - error: "Image not found on GitHub Container Registry", - }; - } - const token = await readBearerToken(tokenResponse); - if (!token) { - return { - valid: false, - error: "Image not found on GitHub Container Registry", - }; - } - const manifestUrl = `${GHCR_MANIFEST_BASE}/${encodedRepositoryPath}/manifests/${encodedReference}`; - const manifestResponse = await fetch(manifestUrl, { - redirect: "error", - headers: { - Authorization: `Bearer ${token}`, - Accept: MANIFEST_ACCEPT, - }, - }); - if (manifestResponse.status === 404) { - return { - valid: false, - error: `Image ${digest ? "digest" : "tag"} not found on GitHub Container Registry`, - }; - } - if (!manifestResponse.ok) { - return { valid: false, error: "Failed to validate image" }; - } - return { valid: true }; - } - + parseImageReference(image); return { valid: true }; } catch (error) { - console.error("Image validation error:", error); - return { valid: false, error: "Failed to validate image" }; + return { + valid: false, + error: + error instanceof Error + ? error.message + : "Invalid image reference syntax", + }; } } diff --git a/web/lib/registry-credentials.ts b/web/lib/registry-credentials.ts new file mode 100644 index 00000000..549269b5 --- /dev/null +++ b/web/lib/registry-credentials.ts @@ -0,0 +1,194 @@ +import { createHmac } from "node:crypto"; +import { asc } from "drizzle-orm"; +import { db } from "@/db"; +import { registryCredentials } from "@/db/schema"; +import type { RegistryCredential } from "@/db/types"; +import { encryptRegistryPassword } from "@/lib/crypto"; +import { resolveEncryptionKey } from "@/lib/kms"; +import { + parseRegistryEndpoint, + registryAuthKey, +} from "@/lib/registry-reference"; + +export type RegistryMetadata = { + id: string; + host: string; + username: string; + tlsVerify: boolean; + system: boolean; + updatedAt: string | null; +}; +export type AgentRegistry = { + id: string; + host: string; + authKey: string; + username: string; + encryptedPassword: string; + tlsVerify: boolean; + system: boolean; +}; +export type AgentRegistryBundle = { + version: string; + registries: AgentRegistry[]; +}; + +type SystemCredential = { + id: string; + host: string; + username: string; + password: string; + tlsVerify: boolean; +}; + +type RegistryEnvironment = Record; + +export function resolveSystemRegistryCredentials( + env: RegistryEnvironment = process.env, +): SystemCredential[] { + const endpointValues = [env.REGISTRY_HOST, env.REGISTRY_URL].filter( + (value): value is string => Boolean(value), + ); + const username = env.REGISTRY_USERNAME; + const password = env.REGISTRY_PASSWORD; + const insecure = env.REGISTRY_INSECURE; + const configured = + endpointValues.length > 0 || + username !== undefined || + password !== undefined || + insecure !== undefined; + if (!configured) return []; + if ( + endpointValues.length === 0 || + !username || + !password || + (insecure !== undefined && insecure !== "true" && insecure !== "false") + ) { + throw new Error("Built-in registry configuration is incomplete"); + } + const hosts = [...new Set(endpointValues.map(parseRegistryEndpoint))].sort(); + return hosts.map((host) => ({ + id: `system:${host}`, + host, + username, + password, + tlsVerify: insecure !== "true", + })); +} + +async function readCustomCredentials(): Promise { + return db + .select() + .from(registryCredentials) + .orderBy(asc(registryCredentials.host)); +} + +function assertNoSystemCollisions( + custom: RegistryCredential[], + system: SystemCredential[], +) { + const reserved = new Set(system.map((credential) => credential.host)); + const collision = custom.find((credential) => reserved.has(credential.host)); + if (collision) + throw new Error( + "A custom registry collides with the built-in registry configuration", + ); +} + +export async function listRegistryMetadata(): Promise { + const [custom, system] = await Promise.all([ + readCustomCredentials(), + Promise.resolve(resolveSystemRegistryCredentials()), + ]); + assertNoSystemCollisions(custom, system); + return [ + ...system.map((entry) => ({ + id: entry.id, + host: entry.host, + username: entry.username, + tlsVerify: entry.tlsVerify, + system: true, + updatedAt: null, + })), + ...custom.map((entry) => ({ + id: entry.id, + host: entry.host, + username: entry.username, + tlsVerify: entry.tlsVerify, + system: false, + updatedAt: entry.updatedAt.toISOString(), + })), + ].sort((a, b) => a.host.localeCompare(b.host)); +} + +export async function getRegistryBundle(): Promise { + const custom = await readCustomCredentials(); + const system = resolveSystemRegistryCredentials(); + assertNoSystemCollisions(custom, system); + const version = await calculateRegistryBundleVersion(custom, system); + const systemEntries = await Promise.all( + system.map(async (entry) => ({ + id: entry.id, + host: entry.host, + authKey: registryAuthKey(entry.host), + username: entry.username, + encryptedPassword: await encryptRegistryPassword( + entry.password, + entry.id, + entry.host, + ), + tlsVerify: entry.tlsVerify, + system: true, + })), + ); + const registries = [ + ...custom.map((entry) => ({ + id: entry.id, + host: entry.host, + authKey: registryAuthKey(entry.host), + username: entry.username, + encryptedPassword: entry.encryptedPassword, + tlsVerify: entry.tlsVerify, + system: false, + })), + ...systemEntries, + ].sort((a, b) => a.host.localeCompare(b.host)); + return { version, registries }; +} + +export async function calculateRegistryBundleVersion( + custom: Pick< + RegistryCredential, + "id" | "host" | "username" | "encryptedPassword" | "tlsVerify" + >[], + system: SystemCredential[] = resolveSystemRegistryCredentials(), +): Promise { + const versionInput = [ + ...custom.map((entry) => [ + entry.id, + entry.host, + entry.username, + entry.encryptedPassword, + entry.tlsVerify, + ]), + ...system.map((entry) => [ + entry.id, + entry.host, + entry.username, + entry.password, + entry.tlsVerify, + ]), + ].sort((a, b) => String(a[1]).localeCompare(String(b[1]))); + const key = await resolveEncryptionKey(); + return createHmac("sha256", key) + .update("registry-bundle-version:v1\0") + .update(JSON.stringify(versionInput)) + .digest("hex"); +} + +export async function getRegistryBundleVersion(): Promise { + return (await getRegistryBundle()).version; +} + +export function getReservedSystemRegistryHosts(): Set { + return new Set(resolveSystemRegistryCredentials().map((entry) => entry.host)); +} diff --git a/web/lib/registry-reference.ts b/web/lib/registry-reference.ts new file mode 100644 index 00000000..6b2e1861 --- /dev/null +++ b/web/lib/registry-reference.ts @@ -0,0 +1,126 @@ +export const DOCKER_HUB_HOST = "docker.io"; +export const DOCKER_HUB_AUTH_KEY = "https://index.docker.io/v1/"; + +const HOST_PATTERN = + /^(?:\[[0-9a-fA-F:.]+\]|[a-zA-Z0-9](?:[a-zA-Z0-9.-]*[a-zA-Z0-9])?)(?::(?:[1-9]\d{0,4}))?$/; +const NAME_SEGMENT = /^[a-z0-9]+(?:[._-][a-z0-9]+)*$/; +const TAG = /^[\w][\w.-]{0,127}$/; +const DIGEST = /^[A-Za-z][A-Za-z0-9_+.-]*:[0-9a-fA-F]{32,256}$/; + +export function canonicalizeRegistryHost(host: string): string { + const value = host.trim(); + if ( + !value || + host !== value || + value.includes("://") || + !HOST_PATTERN.test(value) + ) { + throw new Error( + "Registry host must be a hostname with an optional port and no scheme or path", + ); + } + const parsed = new URL(`https://${value}`); + if (Number(parsed.port) > 65535) throw new Error("Invalid registry port"); + const hostname = parsed.hostname.toLowerCase(); + if (!hostname.startsWith("[") && !/^\d+(?:\.\d+){3}$/.test(hostname)) { + if ( + hostname.length > 253 || + hostname + .split(".") + .some( + (label) => + label.length > 63 || + !/^[a-z0-9](?:[a-z0-9-]*[a-z0-9])?$/.test(label), + ) + ) { + throw new Error("Invalid registry hostname"); + } + } + const canonical = `${hostname}${parsed.port ? `:${parsed.port}` : ""}`; + return canonical === "index.docker.io" || canonical === "registry-1.docker.io" + ? DOCKER_HUB_HOST + : canonical; +} + +export function parseRegistryEndpoint(endpoint: string): string { + const value = endpoint.trim(); + if (!value) throw new Error("Registry endpoint is required"); + const url = new URL(value.includes("://") ? value : `https://${value}`); + if ( + !/^(https?):$/.test(url.protocol) || + url.username || + url.password || + url.pathname !== "/" || + url.search || + url.hash + ) { + throw new Error( + "Registry endpoint must not contain credentials, a path, query, or fragment", + ); + } + return canonicalizeRegistryHost(url.host); +} + +export function resolveRegistryImageHost( + env: Record = process.env, +): string { + const host = env.REGISTRY_HOST; + if (!host) throw new Error("REGISTRY_HOST environment variable is required"); + return parseRegistryEndpoint(host); +} + +export function registryAuthKey(host: string): string { + return host === DOCKER_HUB_HOST ? DOCKER_HUB_AUTH_KEY : host; +} + +export type ParsedImageReference = { + host: string; + repository: string; + tag: string | null; + digest: string | null; + normalized: string; +}; + +export function parseImageReference(input: string): ParsedImageReference { + if (!input || input !== input.trim() || /[\\?#\s]/.test(input)) + throw new Error("Invalid image reference syntax"); + const at = input.indexOf("@"); + if (at !== input.lastIndexOf("@")) throw new Error("Invalid image digest"); + let path = at < 0 ? input : input.slice(0, at); + const digest = at < 0 ? null : input.slice(at + 1); + if (digest !== null && !DIGEST.test(digest)) + throw new Error("Invalid image digest"); + const lastSlash = path.lastIndexOf("/"); + const colon = path.lastIndexOf(":"); + const tag = colon > lastSlash ? path.slice(colon + 1) : null; + if (tag !== null) { + if (!TAG.test(tag)) throw new Error("Invalid image tag"); + path = path.slice(0, colon); + } + const parts = path.split("/"); + const first = parts[0] ?? ""; + const explicit = + parts.length > 1 && + (first.includes(".") || + first.includes(":") || + first === "localhost" || + first.startsWith("[")); + const host = explicit ? canonicalizeRegistryHost(first) : DOCKER_HUB_HOST; + const names = explicit ? parts.slice(1) : parts; + if (!names.length || names.some((part) => !NAME_SEGMENT.test(part))) + throw new Error("Invalid image name"); + if (host === DOCKER_HUB_HOST && names.length === 1) names.unshift("library"); + const repository = names.join("/"); + const suffix = digest ? `@${digest}` : tag ? `:${tag}` : ""; + return { + host, + repository, + tag, + digest, + normalized: `${host}/${repository}${suffix}`, + }; +} + +export function normalizeImageReference(image: string): string { + return parseImageReference(image).normalized; +} diff --git a/web/lib/trigger-build.ts b/web/lib/trigger-build.ts index 3350db15..06942fbb 100644 --- a/web/lib/trigger-build.ts +++ b/web/lib/trigger-build.ts @@ -9,6 +9,7 @@ import { canonicalGitHubRepository, resolvePersistedSourceFromRows, } from "@/lib/public-api"; +import { resolveRegistryImageHost } from "@/lib/registry-reference"; import type { ServiceRevisionActor } from "@/lib/service-revision-actor"; import { parseServiceRevisionSpec } from "@/lib/service-revision-changes"; import { @@ -106,10 +107,7 @@ async function queueResolvedBuild( throw new Error("GitHub source changed before the build was queued"); } - const registryHost = process.env.REGISTRY_HOST?.replace(/\/+$/, ""); - if (!registryHost) { - throw new Error("REGISTRY_HOST environment variable is required"); - } + const registryHost = resolveRegistryImageHost(); const serviceRevisionId = input.idempotencyKey ? deterministicRevisionId(input.idempotencyKey) : randomUUID(); diff --git a/web/lib/work-queue.ts b/web/lib/work-queue.ts index 5048d65f..3d762f31 100644 --- a/web/lib/work-queue.ts +++ b/web/lib/work-queue.ts @@ -1,5 +1,5 @@ import { randomUUID } from "node:crypto"; -import { and, eq, inArray, sql } from "drizzle-orm"; +import { and, eq, inArray, isNotNull, sql } from "drizzle-orm"; import { db } from "@/db"; import { deployments, servers, volumeBackups, workQueue } from "@/db/schema"; import type { WorkQueue } from "@/db/types"; @@ -70,6 +70,7 @@ export type WorkPayloadByType = { buildGroupId: string; }; upgrade_agent: { targetVersion: string; expectedSha256: string }; + sync_registries: { version: string }; }; export type WorkItemResult = { @@ -141,6 +142,32 @@ export async function enqueueReconcileForAllOnlineServers( } } +export async function enqueueRegistrySyncForAllRegisteredServers( + version: string, + tx: WorkQueueTransaction, +) { + const registeredServers = await tx + .select({ id: servers.id }) + .from(servers) + .where(isNotNull(servers.signingPublicKey)); + for (const server of registeredServers) { + await tx + .insert(workQueue) + .values({ + id: randomUUID(), + serverId: server.id, + type: "sync_registries", + payload: JSON.stringify({ version }), + }) + .onConflictDoUpdate({ + target: workQueue.serverId, + targetWhere: sql`${workQueue.type} = 'sync_registries' AND ${workQueue.status} = 'pending'`, + set: { payload: JSON.stringify({ version }), createdAt: new Date() }, + }); + await notifyWorkAvailable(server.id, tx); + } +} + export async function completeWorkItemResults( serverId: string, results: WorkItemResult[], @@ -245,7 +272,9 @@ export async function claimNextWorkItem( SELECT id FROM work_queue WHERE ${claimable} - ORDER BY created_at ASC + ORDER BY + CASE WHEN type = 'sync_registries' THEN 0 ELSE 1 END, + created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1 ) diff --git a/web/tests/agent-auth-signature.test.ts b/web/tests/agent-auth-signature.test.ts new file mode 100644 index 00000000..6e1df98f --- /dev/null +++ b/web/tests/agent-auth-signature.test.ts @@ -0,0 +1,58 @@ +import { NextRequest } from "next/server"; +import { beforeEach, describe, expect, it, vi } from "vitest"; + +const mocks = vi.hoisted(() => ({ + verifyEd25519Signature: vi.fn(), +})); + +vi.mock("@/db", () => ({ + db: { + select: vi.fn(() => ({ + from: vi.fn(() => ({ + where: vi.fn().mockResolvedValue([ + { + id: "server-1", + name: "Worker 1", + signingPublicKey: "public-key", + }, + ]), + })), + })), + }, +})); +vi.mock("@/lib/crypto", () => ({ + verifyEd25519Signature: mocks.verifyEd25519Signature, +})); + +import { verifyAgentRequest } from "@/lib/agent-auth"; + +describe("agent request signatures", () => { + beforeEach(() => { + vi.clearAllMocks(); + mocks.verifyEd25519Signature.mockReturnValue(true); + }); + + it("binds the signature to the method, path, query, and body", async () => { + const timestamp = String(Date.now()); + const request = new NextRequest( + "https://control.example/api/v1/agent/status?mode=full", + { + method: "POST", + headers: { + "x-server-id": "server-1", + "x-timestamp": timestamp, + "x-signature": "signature", + }, + }, + ); + + await expect( + verifyAgentRequest(request, '{"ready":true}'), + ).resolves.toMatchObject({ success: true, serverId: "server-1" }); + expect(mocks.verifyEd25519Signature).toHaveBeenCalledWith( + "public-key", + `agent-request:v2\0${timestamp}\0POST\0/api/v1/agent/status?mode=full\0{"ready":true}`, + "signature", + ); + }); +}); diff --git a/web/tests/agent-registries-route.test.ts b/web/tests/agent-registries-route.test.ts new file mode 100644 index 00000000..bee94be2 --- /dev/null +++ b/web/tests/agent-registries-route.test.ts @@ -0,0 +1,82 @@ +import type { NextRequest } from "next/server"; +import { beforeEach, describe, expect, it, vi } from "vitest"; + +const mocks = vi.hoisted(() => ({ + verifyAgentRequest: vi.fn(), + getRegistryBundle: vi.fn(), +})); + +vi.mock("@/lib/agent-auth", () => ({ + verifyAgentRequest: mocks.verifyAgentRequest, +})); +vi.mock("@/lib/registry-credentials", () => ({ + getRegistryBundle: mocks.getRegistryBundle, +})); + +import { GET } from "@/app/api/v1/agent/registries/route"; + +function request() { + return new Request("http://localhost/api/v1/agent/registries") as NextRequest; +} + +describe("agent registry bundle endpoint", () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + it("requires a signed agent request and never reads credentials on failure", async () => { + mocks.verifyAgentRequest.mockResolvedValue({ + success: false, + status: 401, + error: "Invalid signature", + }); + + const response = await GET(request()); + + expect(response.status).toBe(401); + expect(response.headers.get("cache-control")).toBe("private, no-store"); + expect(response.headers.get("pragma")).toBe("no-cache"); + expect(mocks.getRegistryBundle).not.toHaveBeenCalled(); + }); + + it("returns the complete encrypted desired state without caching", async () => { + mocks.verifyAgentRequest.mockResolvedValue({ + success: true, + serverId: "server-1", + }); + mocks.getRegistryBundle.mockResolvedValue({ + version: "opaque-version", + registries: [ + { + id: "credential-1", + host: "registry.example.com", + authKey: "registry.example.com", + username: "robot", + encryptedPassword: "ciphertext", + tlsVerify: true, + system: false, + }, + ], + }); + + const response = await GET(request()); + + expect(response.status).toBe(200); + expect(response.headers.get("cache-control")).toBe("private, no-store"); + expect(response.headers.get("pragma")).toBe("no-cache"); + expect(await response.json()).toEqual({ + version: "opaque-version", + registries: [ + { + id: "credential-1", + host: "registry.example.com", + authKey: "registry.example.com", + username: "robot", + encryptedPassword: "ciphertext", + tlsVerify: true, + system: false, + }, + ], + }); + }); +}); diff --git a/web/tests/docker-image.test.ts b/web/tests/docker-image.test.ts index 3d7be37d..dcf1297f 100644 --- a/web/tests/docker-image.test.ts +++ b/web/tests/docker-image.test.ts @@ -1,158 +1,32 @@ -import { afterEach, describe, expect, it, vi } from "vitest"; +import { describe, expect, it, vi } from "vitest"; import { validateDockerImageInternal } from "@/lib/docker-image"; -function jsonResponse(body: unknown, status = 200) { - return new Response(JSON.stringify(body), { - status, - headers: { "content-type": "application/json" }, - }); -} - -describe("Docker image validation requests", () => { - afterEach(() => { - vi.unstubAllGlobals(); - }); - - it("uses fixed Docker Hub origins and encoded digest paths", async () => { - const fetchMock = vi - .fn() - .mockResolvedValueOnce(jsonResponse({ token: "docker-token" })) - .mockResolvedValueOnce(new Response(null, { status: 200 })); - vi.stubGlobal("fetch", fetchMock); - const digest = `sha256:${"a".repeat(64)}`; - - await expect( - validateDockerImageInternal(`alpine@${digest}`), - ).resolves.toEqual({ valid: true }); - - expect(String(fetchMock.mock.calls[0]?.[0])).toBe( - "https://auth.docker.io/token?service=registry.docker.io&scope=repository%3Alibrary%2Falpine%3Apull", - ); - expect(fetchMock.mock.calls[0]?.[1]).toEqual({ redirect: "error" }); - expect(fetchMock).toHaveBeenNthCalledWith( - 2, - `https://registry-1.docker.io/v2/library/alpine/manifests/sha256%3A${"a".repeat(64)}`, - { - redirect: "error", - headers: { - Authorization: "Bearer docker-token", - Accept: expect.any(String), - }, - }, - ); - }); - - it("validates nested Docker Hub repositories instead of treating them as registries", async () => { - const fetchMock = vi - .fn() - .mockResolvedValue(new Response(null, { status: 200 })); - vi.stubGlobal("fetch", fetchMock); - - await expect( - validateDockerImageInternal("owner/team/api:release-1"), - ).resolves.toEqual({ valid: true }); - - expect(fetchMock).toHaveBeenCalledWith( - "https://hub.docker.com/v2/repositories/owner/team/api/tags/release-1", - { method: "GET", redirect: "error" }, - ); - }); - - it("uses fixed GHCR origins and accepts access_token responses", async () => { - const fetchMock = vi - .fn() - .mockResolvedValueOnce(jsonResponse({ access_token: "ghcr-token" })) - .mockResolvedValueOnce(new Response(null, { status: 200 })); - vi.stubGlobal("fetch", fetchMock); - - await expect( - validateDockerImageInternal("ghcr.io/acme/api:release"), - ).resolves.toEqual({ valid: true }); - - expect(String(fetchMock.mock.calls[0]?.[0])).toBe( - "https://ghcr.io/token?scope=repository%3Aacme%2Fapi%3Apull", - ); - expect(fetchMock.mock.calls[0]?.[1]).toEqual({ redirect: "error" }); - expect(fetchMock).toHaveBeenNthCalledWith( - 2, - "https://ghcr.io/v2/acme/api/manifests/release", - { - redirect: "error", - headers: { - Authorization: "Bearer ghcr-token", - Accept: expect.any(String), - }, - }, - ); - }); - +describe("Docker image syntax validation", () => { it.each([ - {}, - { token: "" }, - { token: " " }, - { token: 123 }, - { access_token: null }, - null, - [], - ])("does not request a manifest for malformed bearer token %#", async (tokenBody) => { - const fetchMock = vi - .fn() - .mockResolvedValue(jsonResponse(tokenBody)); - vi.stubGlobal("fetch", fetchMock); - - await expect( - validateDockerImageInternal("ghcr.io/acme/api:release"), - ).resolves.toEqual({ - valid: false, - error: "Image not found on GitHub Container Registry", - }); - expect(fetchMock).toHaveBeenCalledOnce(); - }); - - it("handles a non-JSON token response as an authentication failure", async () => { - const fetchMock = vi - .fn() - .mockResolvedValue(new Response("not json", { status: 200 })); + "alpine", + "owner/api:release-1", + "ghcr.io/acme/api:release", + `localhost:5000/api@sha256:${"a".repeat(64)}`, + ])("accepts %s without network access", async (image) => { + const fetchMock = vi.fn(); vi.stubGlobal("fetch", fetchMock); - - await expect( - validateDockerImageInternal(`alpine@sha256:${"a".repeat(64)}`), - ).resolves.toEqual({ - valid: false, - error: "Failed to authenticate with Docker Hub", - }); - expect(fetchMock).toHaveBeenCalledOnce(); - }); - - it.each([ - "docker.io.evil.example/owner/api:tag", - "docker.io:443/owner/api:tag", - "ghcr.io.evil.example/owner/api:tag", - "localhost:5000/owner/api:tag", - "REGISTRY/owner/api:tag", - ])("does not contact unsupported registry %s", async (image) => { - const fetchMock = vi.fn(); - vi.stubGlobal("fetch", fetchMock); - await expect(validateDockerImageInternal(image)).resolves.toEqual({ valid: true, }); expect(fetchMock).not.toHaveBeenCalled(); + vi.unstubAllGlobals(); }); it.each([ - "docker.io/acme//api:tag", - "docker.io/acme\\evil/api:tag", - "docker.io/acme/api?query:tag", - "docker.io/acme/%2f:tag", - "ghcr.io/acme/api#fragment:tag", - ])("rejects structural URL characters in %s", async (image) => { - const fetchMock = vi.fn(); - vi.stubGlobal("fetch", fetchMock); - + "", + "https://ghcr.io/acme/api:tag", + "ghcr.io/acme//api:tag", + "ghcr.io/acme/api?x:tag", + "UPPER/repo:tag", + "alpine@sha256:short", + ])("rejects %s", async (image) => { await expect(validateDockerImageInternal(image)).resolves.toMatchObject({ valid: false, }); - expect(fetchMock).not.toHaveBeenCalled(); }); }); diff --git a/web/tests/registry-credentials.test.ts b/web/tests/registry-credentials.test.ts new file mode 100644 index 00000000..223ab8b6 --- /dev/null +++ b/web/tests/registry-credentials.test.ts @@ -0,0 +1,64 @@ +import { afterEach, describe, expect, it } from "vitest"; +import { resetEncryptionKeyCacheForTests } from "@/lib/kms"; +import { + calculateRegistryBundleVersion, + resolveSystemRegistryCredentials, +} from "@/lib/registry-credentials"; + +describe("registry credential bundles", () => { + afterEach(() => { + delete process.env.ENCRYPTION_KEY; + resetEncryptionKeyCacheForTests(); + }); + + it("deduplicates built-in endpoint aliases and canonicalizes Docker Hub", () => { + const credentials = resolveSystemRegistryCredentials({ + REGISTRY_HOST: "docker.io", + REGISTRY_URL: "https://index.docker.io", + REGISTRY_USERNAME: "robot", + REGISTRY_PASSWORD: "token", + REGISTRY_INSECURE: "false", + }); + expect(credentials).toHaveLength(1); + expect(credentials[0]).toMatchObject({ + host: "docker.io", + tlsVerify: true, + }); + }); + + it("rejects partial built-in configuration", () => { + expect(() => + resolveSystemRegistryCredentials({ + REGISTRY_URL: "registry.example.com", + }), + ).toThrow("incomplete"); + }); + + it("produces a deterministic opaque version independent of row ordering", async () => { + process.env.ENCRYPTION_KEY = "ab".repeat(32); + resetEncryptionKeyCacheForTests(); + const rows = [ + { + id: "2", + host: "z.example", + username: "z", + encryptedPassword: "cipher-z", + tlsVerify: true, + }, + { + id: "1", + host: "a.example", + username: "a", + encryptedPassword: "cipher-a", + tlsVerify: false, + }, + ]; + const first = await calculateRegistryBundleVersion(rows, []); + const second = await calculateRegistryBundleVersion( + [...rows].reverse(), + [], + ); + expect(first).toMatch(/^[0-9a-f]{64}$/); + expect(second).toBe(first); + }); +}); diff --git a/web/tests/registry-reference.test.ts b/web/tests/registry-reference.test.ts new file mode 100644 index 00000000..98bfca47 --- /dev/null +++ b/web/tests/registry-reference.test.ts @@ -0,0 +1,47 @@ +import { describe, expect, it } from "vitest"; +import { + canonicalizeRegistryHost, + normalizeImageReference, + parseRegistryEndpoint, + registryAuthKey, + resolveRegistryImageHost, +} from "@/lib/registry-reference"; + +describe("registry references", () => { + it.each([ + ["Example.COM:5000", "example.com:5000"], + ["index.docker.io", "docker.io"], + ["registry-1.docker.io", "docker.io"], + ["docker.io", "docker.io"], + ])("canonicalizes %s", (input, expected) => + expect(canonicalizeRegistryHost(input)).toBe(expected)); + it.each([ + "https://example.com/path", + "user@example.com", + "example.com?x=1", + "http://example.com/#x", + "a..b", + "a.-b.example", + "a.b-.example", + `${"a".repeat(64)}.example`, + ])("rejects endpoint decorations in %s", (input) => + expect(() => parseRegistryEndpoint(input)).toThrow()); + it("accepts an optional built-in endpoint scheme", () => + expect(parseRegistryEndpoint("https://REGISTRY.example:5443")).toBe( + "registry.example:5443", + )); + it("canonicalizes the configured public image host", () => + expect( + resolveRegistryImageHost({ + REGISTRY_HOST: "https://REGISTRY.example:5443", + }), + ).toBe("registry.example:5443")); + it("uses Docker's special auth key", () => + expect(registryAuthKey("docker.io")).toBe("https://index.docker.io/v1/")); + it.each([ + ["alpine", "docker.io/library/alpine"], + ["index.docker.io/acme/api:v1", "docker.io/acme/api:v1"], + ["ghcr.io/acme/api", "ghcr.io/acme/api"], + ])("normalizes image %s", (input, expected) => + expect(normalizeImageReference(input)).toBe(expected)); +}); diff --git a/web/tests/registry-work-queue.test.ts b/web/tests/registry-work-queue.test.ts new file mode 100644 index 00000000..0eb731e6 --- /dev/null +++ b/web/tests/registry-work-queue.test.ts @@ -0,0 +1,92 @@ +import { getTableConfig, PgDialect } from "drizzle-orm/pg-core"; +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { workQueue } from "@/db/schema"; + +const mocks = vi.hoisted(() => ({ + notifyWorkAvailable: vi.fn(), +})); + +vi.mock("@/lib/work-queue-notifications", () => ({ + notifyWorkAvailable: mocks.notifyWorkAvailable, +})); + +import { enqueueRegistrySyncForAllRegisteredServers } from "@/lib/work-queue"; + +type RegistrySyncTransaction = Parameters< + typeof enqueueRegistrySyncForAllRegisteredServers +>[1]; + +function awaitable(value: T) { + return { + // biome-ignore lint/suspicious/noThenProperty: Drizzle query builders are awaitable. + then: ( + resolve?: ((value: T) => TResult1 | PromiseLike) | null, + reject?: ((reason: unknown) => TResult2 | PromiseLike) | null, + ) => Promise.resolve(value).then(resolve, reject), + }; +} + +describe("registry synchronization fan-out", () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + it("targets every registered server and coalesces only pending sync work", async () => { + const inserted: Array> = []; + const conflicts: Array> = []; + const selectQuery = { + from: vi.fn(() => selectQuery), + where: vi.fn(() => awaitable([{ id: "online" }, { id: "offline" }])), + }; + const tx = { + select: vi.fn(() => selectQuery), + insert: vi.fn(() => ({ + values: vi.fn((values: Record) => { + inserted.push(values); + return { + onConflictDoUpdate: vi.fn((config: Record) => { + conflicts.push(config); + return Promise.resolve(); + }), + }; + }), + })), + } as unknown as RegistrySyncTransaction; + + await enqueueRegistrySyncForAllRegisteredServers("version-2", tx); + + expect(inserted).toHaveLength(2); + expect(inserted).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + serverId: "online", + type: "sync_registries", + payload: JSON.stringify({ version: "version-2" }), + }), + expect.objectContaining({ + serverId: "offline", + type: "sync_registries", + payload: JSON.stringify({ version: "version-2" }), + }), + ]), + ); + expect(conflicts).toHaveLength(2); + expect(mocks.notifyWorkAvailable.mock.calls).toEqual([ + ["online", tx], + ["offline", tx], + ]); + + const pendingSyncIndex = getTableConfig(workQueue).indexes.find( + (index) => + index.config.name === "work_queue_one_pending_registry_sync_idx", + ); + if (!pendingSyncIndex?.config.where) + throw new Error("pending registry sync index is missing"); + expect(pendingSyncIndex?.config.unique).toBe(true); + expect( + new PgDialect().sqlToQuery(pendingSyncIndex.config.where).sql, + ).toContain( + `"work_queue"."type" = 'sync_registries' AND "work_queue"."status" = 'pending'`, + ); + }); +}); From af3ade0685bda908e7338d27d8956a40404167c3 Mon Sep 17 00:00:00 2001 From: Amp Date: Sat, 1 Aug 2026 05:21:43 +0000 Subject: [PATCH 07/12] refactor: make registry credentials immutable Amp-Thread-ID: https://ampcode.com/threads/T-019fac42-0f87-77cd-a0fb-a9e99ccb8d65 Co-authored-by: Arjun Komath --- docs/infrastructure/registry.mdx | 2 +- web/actions/registry-credentials.ts | 32 --------- web/components/settings/registry-settings.tsx | 72 +++++-------------- 3 files changed, 17 insertions(+), 89 deletions(-) diff --git a/docs/infrastructure/registry.mdx b/docs/infrastructure/registry.mdx index ed2320ad..71861d4b 100644 --- a/docs/infrastructure/registry.mdx +++ b/docs/infrastructure/registry.mdx @@ -33,7 +33,7 @@ The built-in credentials are included in the same encrypted registry bundle as c Administrators can add private registry credentials under **Settings → Registries**. A registry host (including an explicit port, when required) is globally unique and cannot include a scheme or path. Docker Hub aliases are treated as the same host, and hosts used by the built-in registry configuration are reserved. -Registry changes are transactionally queued for every registered agent, including offline agents. Rotating a password replaces the encrypted credential while preserving the host; deleting an entry removes it from the next complete bundle. Use pull-only robot or service-account tokens wherever possible. +Registry changes are transactionally queued for every registered agent, including offline agents. Registry entries cannot be edited after creation. To change a username, password, or TLS setting, delete the registry and add it again. Deleting an entry removes it from the next complete bundle. Use pull-only robot or service-account tokens wherever possible. Passwords are write-only in the control plane UI. The control plane validates image reference syntax but neither decrypts custom registry passwords nor contacts registries to test them. An agent's actual image pull is the authoritative credential and image availability check. diff --git a/web/actions/registry-credentials.ts b/web/actions/registry-credentials.ts index 01590d37..e8d3a0f1 100644 --- a/web/actions/registry-credentials.ts +++ b/web/actions/registry-credentials.ts @@ -81,38 +81,6 @@ export async function createRegistryCredential(input: RegistryCredentialInput) { }); } -export async function updateRegistryCredential( - id: string, - input: { username: string; password?: string; tlsVerify: boolean }, -) { - await requireAdmin(); - validateUsername(input.username); - return mutateAndFanout(async (tx) => { - const existing = await tx - .select() - .from(registryCredentials) - .where(eq(registryCredentials.id, id)) - .then((rows) => rows[0]); - if (!existing) throw new Error("Registry credential not found"); - await tx - .update(registryCredentials) - .set({ - username: input.username, - tlsVerify: input.tlsVerify, - ...(input.password - ? { - encryptedPassword: await encryptRegistryPassword( - input.password, - id, - existing.host, - ), - } - : {}), - }) - .where(eq(registryCredentials.id, id)); - }); -} - export async function deleteRegistryCredential(id: string) { await requireAdmin(); return mutateAndFanout(async (tx) => { diff --git a/web/components/settings/registry-settings.tsx b/web/components/settings/registry-settings.tsx index d4aec126..e11c2fa5 100644 --- a/web/components/settings/registry-settings.tsx +++ b/web/components/settings/registry-settings.tsx @@ -5,7 +5,6 @@ import { toast } from "sonner"; import { createRegistryCredential, deleteRegistryCredential, - updateRegistryCredential, } from "@/actions/registry-credentials"; import { Button } from "@/components/ui/button"; import { Input } from "@/components/ui/input"; @@ -44,7 +43,8 @@ export function RegistrySettings({

Add registry credentials

Credentials are distributed globally to every registered agent. The - password is write-only. + password is write-only. To change credentials later, delete the + registry and add it again.

@@ -111,9 +111,6 @@ export function RegistrySettings({ } function RegistryRow({ registry }: { registry: RegistryMetadata }) { - const [username, setUsername] = useState(registry.username); - const [password, setPassword] = useState(""); - const [tlsVerify, setTlsVerify] = useState(registry.tlsVerify); const [busy, setBusy] = useState(false); if (registry.system) return ( @@ -130,23 +127,6 @@ function RegistryRow({ registry }: { registry: RegistryMetadata }) {
); - async function save() { - setBusy(true); - try { - await updateRegistryCredential(registry.id, { - username, - password: password || undefined, - tlsVerify, - }); - toast.success("Registry updated"); - window.location.reload(); - } catch (error) { - toast.error( - error instanceof Error ? error.message : "Failed to update registry", - ); - setBusy(false); - } - } async function remove() { if ( !window.confirm( @@ -167,46 +147,26 @@ function RegistryRow({ registry }: { registry: RegistryMetadata }) { } } return ( -
-

{registry.host}

-
- setUsername(event.target.value)} - /> - setPassword(event.target.value)} - /> -
- - {!tlsVerify && ( -

- TLS verification is disabled. -

- )} -
- +
+
+
+

{registry.host}

+

+ {registry.username} · TLS verification{" "} + {registry.tlsVerify ? "enabled" : "disabled"} +

+

+ Delete and add this registry again to change its credentials or TLS + setting. +

+
From b383d7b30038a6f7e2bb8decd33fdc0f89933ee1 Mon Sep 17 00:00:00 2001 From: Amp Date: Sat, 1 Aug 2026 05:31:40 +0000 Subject: [PATCH 08/12] chore: simplify registry settings copy Amp-Thread-ID: https://ampcode.com/threads/T-019fac42-0f87-77cd-a0fb-a9e99ccb8d65 Co-authored-by: Arjun Komath --- web/components/settings/registry-settings.tsx | 4 ---- 1 file changed, 4 deletions(-) diff --git a/web/components/settings/registry-settings.tsx b/web/components/settings/registry-settings.tsx index e11c2fa5..0e933206 100644 --- a/web/components/settings/registry-settings.tsx +++ b/web/components/settings/registry-settings.tsx @@ -155,10 +155,6 @@ function RegistryRow({ registry }: { registry: RegistryMetadata }) { {registry.username} · TLS verification{" "} {registry.tlsVerify ? "enabled" : "disabled"}

-

- Delete and add this registry again to change its credentials or TLS - setting. -