Skip to content

feat: add external secret sources for variables - #4861

Open
kmendell wants to merge 9 commits into
mainfrom
feat/extensible-external-secrets
Open

kmendell wants to merge 9 commits into
mainfrom
feat/extensible-external-secrets

Conversation

@kmendell

@kmendell kmendell commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Checklist

  • This PR is not opened from my fork's main branch
  • All new user-facing strings are translated via Paraglide (m.*())

What This PR Implements

Arcane can now pull variables from an external secrets manager, store them as variables, and sync them to environments and agents. Each source is described by a YAML definition, so a new provider needs a definition file and no new Go code.

Shipped presets: Infisical, Doppler, Vault KV v2 (AppRole and token auth), 1Password Connect, generic HTTP JSON, and generic container. Custom definitions can be pasted in and validated before saving.

Addresses discussion #4036 (external secrets managers). This is a discussion, not an issue, so there is no issue number to close.

Changes Made

  • Backend domain: backend/internal/variable/children/source adds create, read, update, delete, refresh, test, definition validation, and preset listing under /api/variables/sources.
  • Fetch engine: backend/pkg/secretsource parses definitions, fetches over HTTP or from a container, extracts keys, renders templates, and loads presets from backend/resources/secretsources/*.yaml.
  • Scope: a source applies to all environments or to a selected set, stored through environment.ReplaceScopeRows.
  • Webhooks: a source can be refreshed by webhook, using target type secretsource.
  • Permissions: six secret-sources:* permissions (list, read, create, update, delete, refresh) in the authz catalog.
  • Migration 00117: creates secret_sources and secret_source_environments. It also grants the new permissions to roles and API keys that already hold every prior permission (full admins). Partial roles and keys get nothing. The down migration removes the tables, the grants, and the webhooks targeting sources.
  • Outbound fetch: backend/pkg/httpx/safe_remote.go is reworked for remote source fetches.
  • Shared regex: projects.EnvKeyPattern replaces a duplicate regex in the lifecycle service.
  • CLI: arcane variables sources commands in cli/pkg/variables/sources.go.
  • Frontend: a Secrets page with a table, create and edit sheet, refresh, and delete. The variables table and form are updated, and the environment scope select is new. Services, query keys, and types are added. All strings go through Paraglide.

Testing Done

  • TestMigration117_AddsSecretSourcesAndGrantsFullAdmins covers the up migration, full-admin grants, partial role and key exclusion, name uniqueness, and the down migration.
  • TestMigration102_CleansUpRBACPermissions now migrates to 117.
  • New service tests in backend/internal/variable/children/source/service_test.go.
  • backend/internal/variable/service_test.go is updated for the new variable flow.

Not verified yet:

  • The Secrets page has not been opened in a browser.
  • No live fetch has been run against real Infisical, Doppler, Vault, or 1Password instances.

AI Tool Used (if applicable)

Additional Context

  • Review migration 00117 first. Existing full-admin roles and API keys gain the six secret-sources:* permissions automatically.
  • Credentials are stored encoded and are never returned by the list or get endpoints.
  • Route, API paths, and permission IDs follow the secret-sources naming in the code.

Disclaimer Greptiles Reviews use AI, make sure to check over its work.

To better help train Greptile on our codebase, if the comment is useful and valid Like the comment, if its not helpful or invalid Dislike

To have Greptile Re-Review the changes, mention greptileai.

RetriggerView in GreptileConfidence Score: 2/5 Tier: plus

Not ready to merge: concurrent syncs can still send secrets to the wrong agent, and editing container inputs can erase saved tokens.

Fix All in CodexFindings

  1. P1 Security Secrets reach wrong agents. ▶
  2. P1 Input edits erase saved tokens. ▶
  3. P2 Fetches leave unused volumes. ▶
  4. P2 Old checks replace current fields. ▶
  5. P2 Flag errors go unexplained. ▶
  6. P2 File credentials lose spaces. ▶
  7. P2 Imports break the Go guide. ▶
Fix with agent prompt
### Issue 1
backend/internal/variable/children/source/service.go:402-404
An in-flight sync can pass the scope check for agent A before an edit moves the source to agent B and changes its secret path. After the edit refreshes the cache, this lookup can return B's new secrets to the sync for A. The cache-miss path also reloads the source without rechecking its scope. Read the scope and values from the same locked source settings before returning them.

**How this was verified:** `EffectiveLayers` checks an earlier row's scope, `cachedEntries` reads values keyed only by source ID, and `SyncEnvironment` sends those values to the already-selected agent.

### Issue 2
frontend/src/lib/components/sheets/secret-source-form-sheet.svelte:111-113
Changing the command on a saved `generic-container` source makes `mergeConfig` drop its stored credentials. The sheet still shows the token as saved and omits untouched credentials from `buildInput`. Because this token is optional, saving succeeds and silently erases it. Require re-entry or an explicit clear before saving an edit that drops a stored credential.

### Issue 3
backend/pkg/secretsource/container.go:undefined-122
If the source image declares `VOLUME`, Docker creates anonymous volumes for each run. This cleanup only sets `Force`, so those volumes survive container removal after tests and refreshes. Set `RemoveVolumes` for these temporary containers.

### Issue 4
frontend/src/lib/components/sheets/secret-source-form-sheet.svelte:128-135
An older YAML check can finish after a newer one and overwrite `customDefinition`. Then `resolveDefinition` trusts the old result, while `buildInput` sends the current YAML with fields from the old definition. Users can see outdated fields or have a valid save rejected. Ignore responses that no longer match the current text and check.

### Issue 5
cli/pkg/variables/sources.go:undefined-283
This new call discards the error from `MarkFlagRequired` without explanation. The Go guide requires handling every error or adding a comment explaining why it can be ignored. Handle it, or document that `bindSourceFlags` already registered the flag.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 6
cli/pkg/variables/sources.go:123-128
`readCredentials` trims each whole line before splitting at `=`, so a password ending in a space loses that character. Custom definitions can use credentials as passwords, so the source gets saved or tested with the wrong password. Trim only the key and a separate copy used to detect blank or comment lines; preserve the value.

### Issue 7
backend/pkg/secretsource/container.go:undefined-17
`container.go` renames `github.com/containerd/errdefs` to `cerrdefs`, but no other import conflicts with `errdefs`. The repository guide requires default import names and allows aliases only to fix a real conflict. Remove this alias and use `errdefs.IsNotFound`.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds external secret sources that feed variables across selected environments, with provider presets, custom definitions, webhooks, CLI commands, and a shared Variables & Secrets page.

  • Arcane fetches external secrets and syncs their keys as variables.
  • The variables page now includes a tab for managing secret sources.
  • A webhook can refresh a secret source and start syncing its variables.
  • The CLI can list, manage, test, and refresh secret sources.
  • Source data and permissions are added to Arcane's access controls.
  • Outbound HTTP checks now support secret-source fetches with a private-network policy.

Verification: compared all 237 catalog permissions with the migration lists and checked permission paths. Browser, live-provider, and related-repository checks were not completed; access to the related repositories was denied.

Reviews (8) · Last reviewed commit: "refactor: move secret source permissions..." · Reviewed by Greptile

@kmendell
kmendell requested a review from a team November 18, 2026 00:35
Comment thread cli/pkg/variables/sources.go Resolved
Comment thread cli/pkg/variables/sources.go Resolved
Comment thread backend/pkg/secretsource/http.go Resolved
- Add secret sources that sync key/value pairs from Doppler, Infisical, Vault, 1Password Connect, and generic HTTP or container providers into variables
- Scope each source to all environments or selected ones
- Keep credentials out of API responses
- Add migration 00117 granting secret-sources permissions to full admins and their API keys
- Add secret source API routes, `/secrets` page, and CLI `variables sources` commands
- Require secret sources create permission to test an unsaved source
- Require update permission only when a saved test changes fetch settings
- Update saved sources by ID and return not found when no row matches
- Add a switch to clear a saved optional credential in the source form
- Reset the refresh button's loading state before showing the result
- Print JSON from `secrets sources test` when JSON output is enabled
- Check the error from w.Write in serveSecrets instead of discarding it
- Panic on failure so lint no longer flags the ignored error
- Move external secret sources into a Secrets tab on the Variables page
- Remove the standalone /secrets route, its sidebar entry, and its auth prefix
- Gate the Variables page on variables or secret-sources permissions and show each tab only when its permission is granted
- Rename the Variables nav item and page to "Variables & Secrets"
- Wrap the variables page parallel loads in tryCatch
- Route load failures through throwPageLoadError so the page error boundary renders them
@kmendell
kmendell force-pushed the feat/extensible-external-secrets branch from 05d7b71 to 0cfc6d4 Compare November 18, 2026 20:51
…nal-secrets

# Conflicts:
#	backend/internal/database/database_test.go
#	frontend/messages/en.json
Comment on lines +402 to +404
func (s *Service) cachedEntries(ctx context.Context, record SecretSource) ([]types.EnvVariable, error) {
if entries, ok, _ := s.entries.Get(record.ID); ok {
return withInject(entries, record.InjectIntoContainers), nil

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 security Secrets reach wrong agents. An in-flight sync can pass the scope check for agent A before an edit moves the source to agent B and changes its secret path. After the edit refreshes the cache, this lookup can return B's new secrets to the sync for A. The cache-miss path also reloads the source without rechecking its scope. Read the scope and values from the same locked source settings before returning them.

How this was verified: EffectiveLayers checks an earlier row's scope, cachedEntries reads values keyed only by source ID, and SyncEnvironment sends those values to the already-selected agent.

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/internal/variable/children/source/service.go
Line: 402-404

Comment:
**Secrets reach wrong agents.** An in-flight sync can pass the scope check for agent A before an edit moves the source to agent B and changes its secret path. After the edit refreshes the cache, this lookup can return B's new secrets to the sync for A. The cache-miss path also reloads the source without rechecking its scope. Read the scope and values from the same locked source settings before returning them.

**How this was verified:** `EffectiveLayers` checks an earlier row's scope, `cachedEntries` reads values keyed only by source ID, and `SyncEnvironment` sends those values to the already-selected agent.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

Comment on lines +111 to +113
function credentialIsSet(field: SecretSourceField): boolean {
return !!sourceToEdit && sourceToEdit.credentialKeys.includes(field.key);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Input edits erase saved tokens. Changing the command on a saved generic-container source makes mergeConfig drop its stored credentials. The sheet still shows the token as saved and omits untouched credentials from buildInput. Because this token is optional, saving succeeds and silently erases it. Require re-entry or an explicit clear before saving an edit that drops a stored credential.

Knowledge Base Used: Web application experience

Prompt To Fix With AI
This is a comment left during a code review.
Path: frontend/src/lib/components/sheets/secret-source-form-sheet.svelte
Line: 111-113

Comment:
**Input edits erase saved tokens.** Changing the command on a saved `generic-container` source makes `mergeConfig` drop its stored credentials. The sheet still shows the token as saved and omits untouched credentials from `buildInput`. Because this token is optional, saving succeeds and silently erases it. Require re-entry or an explicit clear before saving an edit that drops a stored credential.

**Knowledge Base Used:** [Web application experience](https://app.greptile.com/-/switch-tenant?tenant=getarcaneapp&next=%2Fcustom-context%2Fknowledge-base%2Fgetarcaneapp%2Farcane%2F-%2Fdocs%2Fweb-application.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

defer func() {
removeCtx, removeCancel := context.WithTimeout(context.WithoutCancel(ctx), timeouts.DefaultDockerAPI)
defer removeCancel()
if _, removeErr := dockerClient.ContainerRemove(removeCtx, created.ID, client.ContainerRemoveOptions{Force: true}); removeErr != nil && !cerrdefs.IsNotFound(removeErr) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Fetches leave unused volumes. If the source image declares VOLUME, Docker creates anonymous volumes for each run. This cleanup only sets Force, so those volumes survive container removal after tests and refreshes. Set RemoveVolumes for these temporary containers.

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/pkg/secretsource/container.go
Line: 122

Comment:
**Fetches leave unused volumes.** If the source image declares `VOLUME`, Docker creates anonymous volumes for each run. This cleanup only sets `Force`, so those volumes survive container removal after tests and refreshes. Set `RemoveVolumes` for these temporary containers.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

Comment on lines +128 to +135
const result = await tryCatch(secretSourceService.validateDefinition(customYaml));
validating = false;
if (result.error) {
customDefinition = null;
definitionError = extractApiErrorMessage(result.error);
return null;
}
customDefinition = result.data;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Old checks replace current fields. An older YAML check can finish after a newer one and overwrite customDefinition. Then resolveDefinition trusts the old result, while buildInput sends the current YAML with fields from the old definition. Users can see outdated fields or have a valid save rejected. Ignore responses that no longer match the current text and check.

Knowledge Base Used: Web application experience

Prompt To Fix With AI
This is a comment left during a code review.
Path: frontend/src/lib/components/sheets/secret-source-form-sheet.svelte
Line: 128-135

Comment:
**Old checks replace current fields.** An older YAML check can finish after a newer one and overwrite `customDefinition`. Then `resolveDefinition` trusts the old result, while `buildInput` sends the current YAML with fields from the old definition. Users can see outdated fields or have a valid save rejected. Ignore responses that no longer match the current text and check.

**Knowledge Base Used:** [Web application experience](https://app.greptile.com/-/switch-tenant?tenant=getarcaneapp&next=%2Fcustom-context%2Fknowledge-base%2Fgetarcaneapp%2Farcane%2F-%2Fdocs%2Fweb-application.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

},
}
bindSourceFlags(command, &opts)
_ = command.MarkFlagRequired("name")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Flag errors go unexplained. This new call discards the error from MarkFlagRequired without explanation. The Go guide requires handling every error or adding a comment explaining why it can be ignored. Handle it, or document that bindSourceFlags already registered the flag.

Rule Used: --- name: go-struct-literals description: Write Go 1.27+ in a declarative, composite-literal style. Every value (structs, slices, maps, configs, errors, servers, test tables, constructor results) is built with one keyed composite literal instead of f... (source)

Prompt To Fix With AI
This is a comment left during a code review.
Path: cli/pkg/variables/sources.go
Line: 283

Comment:
**Flag errors go unexplained.** This new call discards the error from `MarkFlagRequired` without explanation. The Go guide requires handling every error or adding a comment explaining why it can be ignored. Handle it, or document that `bindSourceFlags` already registered the flag.

**Rule Used:** --- name: go-struct-literals description: Write Go 1.27+ in a declarative, composite-literal style. Every value (structs, slices, maps, configs, errors, servers, test tables, constructor results) is built with one keyed composite literal instead of f... ([source](https://app.greptile.com/-/switch-tenant?tenant=getarcaneapp&next=%2Fcustom-context%3Fmemory%3D214b40a8-9695-4738-986d-5949b5d65ff1))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Codex Fix in Claude Code

…nal-secrets

# Conflicts:
#	frontend/src/lib/components/sheets/webhook-form-sheet.svelte
#	frontend/src/routes/(app)/projects/variables/+page.svelte
Comment on lines +123 to +128
for number, line := range strings.Split(string(content), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
key, value, ok := strings.Cut(line, "=")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 File credentials lose spaces. readCredentials trims each whole line before splitting at =, so a password ending in a space loses that character. Custom definitions can use credentials as passwords, so the source gets saved or tested with the wrong password. Trim only the key and a separate copy used to detect blank or comment lines; preserve the value.

Prompt To Fix With AI
This is a comment left during a code review.
Path: cli/pkg/variables/sources.go
Line: 123-128

Comment:
**File credentials lose spaces.** `readCredentials` trims each whole line before splitting at `=`, so a password ending in a space loses that character. Custom definitions can use credentials as passwords, so the source gets saved or tested with the wrong password. Trim only the key and a separate copy used to detect blank or comment lines; preserve the value.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

"strings"
"time"

cerrdefs "github.com/containerd/errdefs"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Imports break the Go guide. container.go renames github.com/containerd/errdefs to cerrdefs, but no other import conflicts with errdefs. The repository guide requires default import names and allows aliases only to fix a real conflict. Remove this alias and use errdefs.IsNotFound.

Context Used: AGENTS.md (source)

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/pkg/secretsource/container.go
Line: 17

Comment:
**Imports break the Go guide.** `container.go` renames `github.com/containerd/errdefs` to `cerrdefs`, but no other import conflicts with `errdefs`. The repository guide requires default import names and allows aliases only to fix a real conflict. Remove this alias and use `errdefs.IsNotFound`.

**Context Used:** AGENTS.md ([source](https://github.com/getarcaneapp/arcane/blob/main/AGENTS.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Codex Fix in Claude Code

- Rename secret-sources:* permissions to variables:sources:* in the authz catalog, access policy, routes, webhook refresh mapping, and frontend permission checks
- Update migration 00118 and its tests to the new permission strings (feature is unreleased)
- Remove the separate "External Secrets" permission resource
- List its actions under Variables as sources-list through sources-refresh
- Relabel them "... external secrets" and update the Variables description

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants