diff --git a/apps/docs/content/docs/platform/self-hosting/environment-variables.mdx b/apps/docs/content/docs/platform/self-hosting/environment-variables.mdx index e5c2f872541..d5263cdb977 100644 --- a/apps/docs/content/docs/platform/self-hosting/environment-variables.mdx +++ b/apps/docs/content/docs/platform/self-hosting/environment-variables.mdx @@ -16,14 +16,14 @@ Production Compose derives `DATABASE_URL` and `BETTER_AUTH_URL` on its service d | `DATABASE_URL` | PostgreSQL connection string | | `BETTER_AUTH_SECRET` | Auth secret, at least 32 characters: `openssl rand -hex 32`. **Must be identical on the app and realtime services** | | `BETTER_AUTH_URL` | Your app URL — must be the real public origin, not `localhost` | -| `ENCRYPTION_KEY` | Encryption key, exactly 64 hex characters (32 bytes): `openssl rand -hex 32` | +| `ENCRYPTION_KEY` | Encryption key, exactly 64 hex characters (32 bytes): `openssl rand -hex 32`. A key in any other format leaves [account token sealing](/platform/self-hosting/security#account-token-sealing) off | | `INTERNAL_API_SECRET` | Internal API secret, at least 32 characters: `openssl rand -hex 32` | | `NEXT_PUBLIC_APP_URL` | Public app URL | `openssl rand -hex 32` prints 64 hex characters. Encryption keys require that format. The app skips its env schema validation; individual consumers can reject invalid values when first used. Realtime validates `BETTER_AUTH_SECRET` and `INTERNAL_API_SECRET` at boot and rejects values shorter than 32 characters. - Retain `ENCRYPTION_KEY` and `API_ENCRYPTION_KEY` across upgrades and restores. There is no supported in-place key rotation procedure; losing a key makes values encrypted with it unreadable — workspace and personal environment variables, stored provider API keys, MCP OAuth credentials, and deployment/chat secrets in the first case, user-generated Sim API keys in the second. Back them up separately from the database. + Retain `ENCRYPTION_KEY` and `API_ENCRYPTION_KEY` across upgrades and restores. There is no supported in-place key rotation procedure; losing a key makes values encrypted with it unreadable — workspace and personal environment variables, stored provider API keys, MCP OAuth credentials, connected-account OAuth tokens, and deployment/chat secrets in the first case, user-generated Sim API keys in the second. Back them up separately from the database. ## Feature prerequisites and coordination [#strongly-recommended] @@ -39,6 +39,7 @@ Production Compose derives `DATABASE_URL` and `BETTER_AUTH_URL` on its service d | `AUTH_TRUSTED_PROXIES` | Comma-separated reverse-proxy IPs/CIDRs so the client IP cannot be forged through `X-Forwarded-For` | | `INTERNAL_API_BASE_URL` | Internal URL for server-side self-calls, e.g. `http://sim-app.simstudio.svc.cluster.local:3000`. Defaults to the public base URL. The Trigger.dev worker ignores it and uses the public app URL. | | `SIM_MCP_URL` | Public URL of the [Sim MCP server](/mcp/overview) when you serve it on its own host, e.g. `https://mcp.example.com/mcp`. Point that host at the app; Sim serves only the MCP endpoint and its OAuth metadata there, and stops serving `/api/mcp` on the app host so clients use one URL. Optional — defaults to `/api/mcp` | +| `ENCRYPT_ACCOUNT_TOKENS` | Set to `false` to stop sealing connected-account OAuth tokens on write and during `migrate`. Sealing is on by default with a 64-hex `ENCRYPTION_KEY`; see [Account token sealing](/platform/self-hosting/security#account-token-sealing) | | `DATABASE_REPLICA_URL` | Read-replica connection string for log listing, audit logs, and dashboard aggregations. Falls back to the primary when unset | ## AI providers diff --git a/apps/docs/content/docs/platform/self-hosting/security.mdx b/apps/docs/content/docs/platform/self-hosting/security.mdx index e6d9a2cea50..1cfd611ff44 100644 --- a/apps/docs/content/docs/platform/self-hosting/security.mdx +++ b/apps/docs/content/docs/platform/self-hosting/security.mdx @@ -41,7 +41,7 @@ Five secrets drive the security of a deployment. Generate each with `openssl ran | Secret | Protects | Rotatable | |---|---|---| | `BETTER_AUTH_SECRET` | Session tokens | Yes — invalidates all sessions | -| `ENCRYPTION_KEY` | Workspace env vars, stored provider keys, MCP OAuth credentials, deployment/chat secrets | **No** — see below | +| `ENCRYPTION_KEY` | Workspace env vars, stored provider keys, MCP OAuth credentials, connected-account OAuth tokens, deployment/chat secrets | **No** — see below | | `API_ENCRYPTION_KEY` | Encrypted stored API-key material when configured | Rotation needs re-encryption; existing hash-based authentication continues, but old encrypted material cannot be decrypted with a new key | | `INTERNAL_API_SECRET` | Service-to-service calls | Yes — roll app and realtime together | | `CRON_SECRET` | Background job endpoints | Yes — roll app and cron together | @@ -52,6 +52,19 @@ Five secrets drive the security of a deployment. Generate each with `openssl ran `BETTER_AUTH_SECRET` must be **identical** on the app and realtime services — they share sessions through the database, and a mismatch means realtime rejects every authenticated socket. +### Account token sealing + +Sim seals the OAuth access and refresh tokens of connected accounts at rest with `ENCRYPTION_KEY`. Each token is encrypted and bound to its column, so a value copied from the database is unusable without the key. + +- **On by default** whenever `ENCRYPTION_KEY` is exactly 64 hex characters (`openssl rand -hex 32`). Each process logs `Account token sealing: enabled` with a key id at startup, never the key itself. +- **Without a usable key**, tokens are stored in plain text and the app keeps working. Each process logs a single warning at startup: `Account token sealing: disabled (ENCRYPTION_KEY is not a 64-character hex key)`. Other features that use `ENCRYPTION_KEY` keep their current behavior. +- **Tokens stored earlier** are sealed by the migrations container during `migrate`, in small batches. The step needs `ENCRYPTION_KEY` on the migrations container: the Helm chart and the Compose files pass the app's value. Until the key is usable, the step does nothing and stays pending, so it runs on a later `migrate` once sealing is on. It also waits for a later `migrate` when it can't finish because rows are busy, rather than failing the upgrade. +- **Opt out** with `ENCRYPT_ACCOUNT_TOKENS=false` on the app, background workers, and migrations container. New writes are then stored in plain text, and tokens that are already sealed stay readable. + + + Once tokens are sealed, do not roll back to a release older than v0.9.33. Those releases cannot read sealed tokens, so every connected account would have to be reconnected. Opting out does not unseal stored tokens. + + ### Storing them Choose how to supply secrets: diff --git a/apps/docs/content/docs/platform/self-hosting/upgrades.mdx b/apps/docs/content/docs/platform/self-hosting/upgrades.mdx index ebfc1e225e2..e0fb6e303d3 100644 --- a/apps/docs/content/docs/platform/self-hosting/upgrades.mdx +++ b/apps/docs/content/docs/platform/self-hosting/upgrades.mdx @@ -227,6 +227,8 @@ PROFILE=gpu # or cpu docker compose -f docker-compose.ollama.yml --profile "$PROFILE" up -d --build ``` +Never roll back below v0.9.33 once account tokens are sealed: those releases cannot read them, and every connected account would have to be reconnected. See [Account token sealing](/platform/self-hosting/security#account-token-sealing). + **Rollback after an incompatible schema change** requires restoring the pre-upgrade database backup. This loses writes made since the snapshot. Stop all writers before restoring: 1. Stop traffic and all processes that can write to the database, including the app, realtime service, scheduled jobs, and any background workers. @@ -270,6 +272,10 @@ Migrations are Drizzle SQL files plus registered script (data) migrations, appli Migrations are **forward-only**. Take the backup described in the upgrade procedure before applying them; there are no down-migrations. + + Upgrading from a release older than v0.9.33 with a rolling deploy leaves old pods running next to new ones for a while. Old pods cannot read sealed account tokens, so integrations can fail on them until the rollout finishes. The `migrate` backfill waits an hour after such an upgrade before it seals stored tokens, but tokens the new pods write are sealed straight away. To avoid that window, upgrade to v0.9.33 first, stop the old pods before starting the new ones, or set `ENCRYPT_ACCOUNT_TOKENS=false` for the upgrade and remove it once every pod runs the new release. + + Migrations apply in sequence, so skipping several releases at once generally works mechanically. It is riskier — those combinations get less testing, and you take several releases' worth of behavior changes in one step. Upgrade in smaller increments where you can. ### How concurrent replicas are handled @@ -289,6 +295,7 @@ These control the migrator and are set on the `migrations` container, not the ap | Variable | Purpose | |---|---| | `MIGRATION_DATABASE_URL` | A **direct, non-pooled** database connection URL (DSN). Falls back to `DATABASE_URL`. Setting it also arms the backend-pid check that aborts the run if the session — and with it the advisory lock — is silently recycled mid-migration; with only `DATABASE_URL` set, that check is skipped | +| `ENCRYPTION_KEY`, `ENCRYPT_ACCOUNT_TOKENS` | Let the migrator seal account tokens stored before [account token sealing](/platform/self-hosting/security#account-token-sealing) was on. The Helm init container reads both from the app Secret, and the Compose `migrations` services pass the `.env` values. Without a usable key the step stays pending and does nothing | Set `MIGRATION_DATABASE_URL` if anything sits between Sim and Postgres. Session advisory locks and session-level `SET`s do not survive PgBouncer transaction pooling — the lock silently does not hold, and consecutive statements can land on different backends. Point it at the database directly, bypassing the pooler. @@ -299,7 +306,7 @@ Supply the migration URL separately from the app URL: | Deployment | How to pass `MIGRATION_DATABASE_URL` | |---|---| -| Helm | The init container reads only the database Secret. Include the key in a Secret supplied through `postgresql.auth.existingSecret` or `externalDatabase.existingSecret`; `app.env` does not reach it. The chart-managed database Secret carries only the password, and there is no dedicated values key for the migration URL | +| Helm | The init container reads the database Secret, plus only `ENCRYPTION_KEY` and `ENCRYPT_ACCOUNT_TOKENS` from the app Secret. Include the key in a Secret supplied through `postgresql.auth.existingSecret` or `externalDatabase.existingSecret`; `app.env` does not reach it. The chart-managed database Secret carries only the password, and there is no dedicated values key for the migration URL | | Compose | Add the variable to the `migrations` service's explicit `environment:` list. Setting it in `.env` alone does not pass it to that service | | Separate migration run | Run the migration image yourself with a direct DSN before upgrading | diff --git a/apps/sim/.env.example b/apps/sim/.env.example index d75989de4b5..35cd8c85c38 100644 --- a/apps/sim/.env.example +++ b/apps/sim/.env.example @@ -55,7 +55,7 @@ NEXT_PUBLIC_APP_URL=http://localhost:3000 # DAYTONA_SHELL_SNAPSHOT_ID= # Mothership shell snapshot ref, required when Mothership runs code on Daytona # Security (Required) -ENCRYPTION_KEY=your_encryption_key # Use `openssl rand -hex 32` to generate, used to encrypt environment variables +ENCRYPTION_KEY=your_encryption_key # Use `openssl rand -hex 32` to generate, used to encrypt environment variables and connected-account OAuth tokens (ENCRYPT_ACCOUNT_TOKENS=false opts the tokens out) INTERNAL_API_SECRET=your_internal_api_secret # Use `openssl rand -hex 32` to generate, used to encrypt internal api routes API_ENCRYPTION_KEY=your_api_encryption_key # Use `openssl rand -hex 32` to generate, used to encrypt api keys CRON_SECRET=your_cron_secret # Use `openssl rand -hex 32` to generate. Authenticates the scheduler against the background job endpoints (scheduled workflows, polling triggers, connector syncs) diff --git a/apps/sim/background/seal-account-tokens.test.ts b/apps/sim/background/seal-account-tokens.test.ts index bc48e14cce7..5785ba51dbe 100644 --- a/apps/sim/background/seal-account-tokens.test.ts +++ b/apps/sim/background/seal-account-tokens.test.ts @@ -28,7 +28,6 @@ function emptyAccountTable() { describe('runSealAccountTokens', () => { beforeEach(() => { vi.stubEnv('ENCRYPTION_KEY', ENCRYPTION_KEY) - vi.stubEnv('ENCRYPT_ACCOUNT_TOKENS', 'true') databaseMockFns.mockResolveDbUrl.mockReturnValue('postgresql://fixture/sim') mockPostgres.mockReset() mockPostgres.mockImplementation(emptyAccountTable) @@ -48,12 +47,18 @@ describe('runSealAccountTokens', () => { expect(mockPostgres).not.toHaveBeenCalled() }) - it('refuses to seal while this worker writes plaintext tokens, before connecting', async () => { + it('seals by default once the worker holds a usable key', async () => { + await expect( + runSealAccountTokens({ mode: 'seal', dryRun: false, expectedKeyId: WORKER_KEY_ID }) + ).resolves.toEqual({ keyId: WORKER_KEY_ID, scanned: 0, pending: 0, updated: 0, foreignKey: 0 }) + }) + + it('refuses to seal while this worker opts out of sealing, before connecting', async () => { vi.stubEnv('ENCRYPT_ACCOUNT_TOKENS', 'false') await expect( runSealAccountTokens({ mode: 'seal', dryRun: false, expectedKeyId: WORKER_KEY_ID }) - ).rejects.toThrow('ENCRYPT_ACCOUNT_TOKENS is not enabled') + ).rejects.toThrow('Refusing to seal: ENCRYPT_ACCOUNT_TOKENS is false') expect(mockPostgres).not.toHaveBeenCalled() }) diff --git a/apps/sim/background/seal-account-tokens.ts b/apps/sim/background/seal-account-tokens.ts index 0f0ae8a8dd3..ef4f8ba4d83 100644 --- a/apps/sim/background/seal-account-tokens.ts +++ b/apps/sim/background/seal-account-tokens.ts @@ -1,6 +1,6 @@ import { resolveDbUrl } from '@sim/db' import { sealAccountTokens } from '@sim/db/script-migrations/0037_seal_account_tokens' -import { isAccountTokenWriteSealingEnabled, readEncryptionKey } from '@sim/db/sealed-text' +import { accountTokenSealingDisabledReason, readEncryptionKey } from '@sim/db/sealed-text' import { createLogger } from '@sim/logger' import { sealingKeyId } from '@sim/security/encryption' import { task } from '@trigger.dev/sdk' @@ -31,7 +31,8 @@ function assertIntegerInRange(name: string, value: number | undefined, min: numb * Runs the account-token sealing backfill (`0037_seal_account_tokens`) from a worker, which already * holds the database and the fleet's `ENCRYPTION_KEY`. Refuses before connecting when the payload * names a key id other than this worker's, so a pass can never seal rows under a key the fleet - * cannot open, and refuses `seal` while this worker's writers still store plaintext. + * cannot open, and refuses `seal` while this worker's writers still store plaintext (sealing opted + * out with `ENCRYPT_ACCOUNT_TOKENS=false`). * * Opens its own single connection rather than reusing the `@sim/db` pool: the pass binds JS arrays * through the postgres.js tag, which needs the array type map the shared pool skips @@ -50,9 +51,10 @@ export async function runSealAccountTokens(payload: SealAccountTokensPayload) { if (expectedKeyId !== keyId) { throw new Error(`ENCRYPTION_KEY has key id ${keyId}, not the expected ${expectedKeyId}`) } - if (mode === 'seal' && !isAccountTokenWriteSealingEnabled()) { + const disabledReason = accountTokenSealingDisabledReason() + if (mode === 'seal' && disabledReason) { throw new Error( - 'Refusing to seal: ENCRYPT_ACCOUNT_TOKENS is not enabled on this worker, so writers still store plaintext' + `Refusing to seal: ${disabledReason} on this worker, so writers still store plaintext` ) } diff --git a/apps/sim/lib/core/config/env.ts b/apps/sim/lib/core/config/env.ts index dbb91782927..7f4d0abeb25 100644 --- a/apps/sim/lib/core/config/env.ts +++ b/apps/sim/lib/core/config/env.ts @@ -144,7 +144,7 @@ export const env = createEnv({ TURNSTILE_SECRET_KEY: z.string().min(1).optional(), // Cloudflare Turnstile secret key for captcha verification ENCRYPTION_KEY: z.string().min(32), // Key for encrypting sensitive data API_ENCRYPTION_KEY: z.string().min(32).optional(), // Dedicated key for encrypting API keys (optional for OSS) - ENCRYPT_ACCOUNT_TOKENS: z.boolean().optional(), // Seal OAuth access/refresh tokens on write (read by @sim/db); takes effect only with a 64-hex ENCRYPTION_KEY. Enable only once every app, worker, and background-task deployment runs a release that reads sealed tokens + ENCRYPT_ACCOUNT_TOKENS: z.boolean().optional(), // Opt-out: OAuth access/refresh tokens are sealed on write (read by @sim/db) and backfilled by migrate whenever ENCRYPTION_KEY is 64 hex characters; false or 0 writes plaintext. Sealed tokens need v0.9.33+ on every process that reads them INTERNAL_API_SECRET: z.string().min(32), // Secret for internal API authentication INTERNAL_JWT_SECRET: z.string().min(32).optional(), // Dedicated signing key for internal JWTs (falls back to INTERNAL_API_SECRET); separating limits blast radius if one leaks diff --git a/apps/sim/lib/oauth/credential-revocation.integration.ts b/apps/sim/lib/oauth/credential-revocation.integration.ts index 4f88564c700..cc38d3281d0 100644 --- a/apps/sim/lib/oauth/credential-revocation.integration.ts +++ b/apps/sim/lib/oauth/credential-revocation.integration.ts @@ -89,8 +89,10 @@ async function rawTokens() { return row } +/** Sealing is the default with the fixture's hex key; `false` is the operator opt-out. */ function sealWrites(enabled: boolean) { - process.env.ENCRYPT_ACCOUNT_TOKENS = enabled ? 'true' : 'false' + if (enabled) Reflect.deleteProperty(process.env, 'ENCRYPT_ACCOUNT_TOKENS') + else process.env.ENCRYPT_ACCOUNT_TOKENS = 'false' } async function resolveToken(rowId = accountId) { @@ -490,7 +492,7 @@ describe('OAuth refresh chains with sealed account tokens', () => { }) }) - it('keeps writing readable tokens when sealing is requested without a usable key', async () => { + it('keeps writing readable tokens when ENCRYPTION_KEY cannot seal them', async () => { sealWrites(true) process.env.ENCRYPTION_KEY = 'not-hex'.repeat(6) providerAnswer = REFRESHED @@ -499,6 +501,17 @@ describe('OAuth refresh chains with sealed account tokens', () => { expect(await rawTokens()).toMatchObject({ access_token: 'fresh-access' }) }) + it('writes plaintext tokens when the operator opts out of sealing', async () => { + sealWrites(false) + providerAnswer = REFRESHED + + await expect(resolveToken()).resolves.toEqual({ accessToken: 'fresh-access', refreshed: true }) + expect(await rawTokens()).toMatchObject({ + access_token: 'fresh-access', + refresh_token: 'issued-refresh', + }) + }) + it('rotates a chain stored before sealing was enabled', async () => { sealWrites(true) providerAnswer = REFRESHED diff --git a/docker-compose.local.yml b/docker-compose.local.yml index 48daa577a06..eceba713eb5 100644 --- a/docker-compose.local.yml +++ b/docker-compose.local.yml @@ -91,6 +91,10 @@ services: working_dir: /app/packages/db environment: - DATABASE_URL=postgresql://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:-postgres}@db:5432/${POSTGRES_DB:-simstudio} + # Seals account tokens stored before sealing was on; a no-op until ENCRYPTION_KEY is + # 64 hex characters. ENCRYPT_ACCOUNT_TOKENS=false opts out here and on the app. + - ENCRYPTION_KEY=${ENCRYPTION_KEY:-} + - ENCRYPT_ACCOUNT_TOKENS=${ENCRYPT_ACCOUNT_TOKENS:-} depends_on: db: condition: service_healthy diff --git a/docker-compose.ollama.yml b/docker-compose.ollama.yml index 4311732add4..501a16032ee 100644 --- a/docker-compose.ollama.yml +++ b/docker-compose.ollama.yml @@ -81,6 +81,10 @@ services: dockerfile: docker/db.Dockerfile environment: - DATABASE_URL=postgresql://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:-postgres}@db:5432/${POSTGRES_DB:-simstudio} + # Seals account tokens stored before sealing was on; a no-op until ENCRYPTION_KEY is + # 64 hex characters. ENCRYPT_ACCOUNT_TOKENS=false opts out here and on the app. + - ENCRYPTION_KEY=${ENCRYPTION_KEY:-} + - ENCRYPT_ACCOUNT_TOKENS=${ENCRYPT_ACCOUNT_TOKENS:-} depends_on: db: condition: service_healthy diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index 0c85846041b..226a9a49829 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -110,6 +110,10 @@ services: working_dir: /app/packages/db environment: - 'DATABASE_URL=postgresql://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:?is required. For a new install set it in .env to the output of: openssl rand -hex 24. An install created before it was required uses the password postgres - set POSTGRES_PASSWORD=postgres to keep its data.}@db:5432/${POSTGRES_DB:-simstudio}' + # Seals account tokens stored before sealing was on; a no-op until ENCRYPTION_KEY is + # 64 hex characters. ENCRYPT_ACCOUNT_TOKENS=false opts out here and on the app. + - ENCRYPTION_KEY=${ENCRYPTION_KEY:-} + - ENCRYPT_ACCOUNT_TOKENS=${ENCRYPT_ACCOUNT_TOKENS:-} depends_on: db: condition: service_healthy diff --git a/helm/sim/Chart.yaml b/helm/sim/Chart.yaml index 482676a7a34..c98331a9f4c 100644 --- a/helm/sim/Chart.yaml +++ b/helm/sim/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: sim description: A Helm chart for Sim - the open-source AI workspace where teams build, deploy, and manage AI agents type: application -version: 1.11.9 +version: 1.11.10 appVersion: "v0.8.26" kubeVersion: ">=1.25.0-0" home: https://sim.ai diff --git a/helm/sim/templates/_helpers.tpl b/helm/sim/templates/_helpers.tpl index 06e0d6c9c07..fa54c75df87 100644 --- a/helm/sim/templates/_helpers.tpl +++ b/helm/sim/templates/_helpers.tpl @@ -683,3 +683,50 @@ Validate Copilot configuration {{- end -}} {{- end -}} {{- end }} + +{{/* +Inline env entries the app container applies over its Secret, in precedence order (a later entry +for the same name wins): envDefaults the operator did not override, existingSecret-mode app.env +values, then extraEnvVars. Shared with the migrations init container so both resolve a key alike. +*/}} +{{- define "sim.appEnvOverrides" -}} +{{- /* + Skip envDefaults keys that the user has explicitly overridden in app.env + with a non-empty value. K8s `env` takes precedence over `envFrom`, so an + inline default would otherwise mask the user's Secret-bound value. + Also skip envDefaults entirely in existingSecret mode: the pre-existing + Secret is the source of truth, and inlining localhost URL defaults would + silently shadow keys (NEXT_PUBLIC_APP_URL, BETTER_AUTH_URL, etc.) the + user stored in their Secret but left empty in app.env. +*/}} +{{- $appEnv := .Values.app.env | default dict }} +{{- $useExistingSecret := and .Values.app.secrets.existingSecret.enabled (not .Values.externalSecrets.enabled) }} +{{- if not $useExistingSecret }} +{{- range $key, $value := .Values.app.envDefaults | default dict }} +{{- $override := index $appEnv $key }} +{{- if and (ne (toString $value) "") (ne (toString $value) "") (or (not $override) (eq (toString $override) "") (eq (toString $override) "")) }} +- name: {{ $key }} + value: {{ $value | quote }} +{{- end }} +{{- end }} +{{- end }} +{{- /* + In existingSecret mode the chart-managed Secret is not rendered, so + non-empty values in app.env would otherwise go nowhere — inline them + here so the user's overrides actually reach the pod. Skipped in ESO + mode (ESO is the source of truth; inlining would mask the synced value) + and in inline mode (values flow through the chart-managed Secret). +*/}} +{{- if and .Values.app.secrets.existingSecret.enabled (not .Values.externalSecrets.enabled) }} +{{- $chartComputed := list "DATABASE_URL" "SOCKET_SERVER_URL" "OLLAMA_URL" "PII_URL" }} +{{- range $key, $value := $appEnv }} +{{- if and (ne (toString $value) "") (ne (toString $value) "") (not (has $key $chartComputed)) }} +- name: {{ $key }} + value: {{ $value | quote }} +{{- end }} +{{- end }} +{{- end }} +{{- with .Values.extraEnvVars }} +{{ toYaml . }} +{{- end }} +{{- end }} diff --git a/helm/sim/templates/deployment-app.yaml b/helm/sim/templates/deployment-app.yaml index 2cc1cd6af13..a6437716b0e 100644 --- a/helm/sim/templates/deployment-app.yaml +++ b/helm/sim/templates/deployment-app.yaml @@ -57,6 +57,32 @@ spec: cd /app/packages/db export DATABASE_URL="{{ include "sim.databaseUrl" . }}" bun run db:migrate + {{- /* + The account token backfill script migration seals with the app's key and honors its + ENCRYPT_ACCOUNT_TOKENS opt-out. Each resolves as it does on the app container: the last + matching entry of the app's inline overrides (envDefaults, existingSecret inlining, + extraEnvVars), else the app Secret. Without a usable key the step stays pending. + */}} + env: + {{- $appOverrides := include "sim.appEnvOverrides" . | fromYamlArray }} + {{- range $key := list "ENCRYPTION_KEY" "ENCRYPT_ACCOUNT_TOKENS" }} + {{- $override := dict }} + {{- range $appOverrides }} + {{- if eq (toString .name) $key }} + {{- $override = . }} + {{- end }} + {{- end }} + {{- if $override }} + - {{ toYaml $override | nindent 14 | trim }} + {{- else }} + - name: {{ $key }} + valueFrom: + secretKeyRef: + name: {{ include "sim.appSecretName" $ }} + key: {{ $key }} + optional: {{ eq $key "ENCRYPT_ACCOUNT_TOKENS" }} + {{- end }} + {{- end }} envFrom: {{- if .Values.postgresql.enabled }} - secretRef: @@ -91,42 +117,6 @@ spec: value: {{ include "sim.ollamaUrl" . | quote }} - name: PII_URL value: {{ include "sim.piiUrl" . | quote }} - {{- /* - Skip envDefaults keys that the user has explicitly overridden in app.env - with a non-empty value. K8s `env` takes precedence over `envFrom`, so an - inline default would otherwise mask the user's Secret-bound value. - Also skip envDefaults entirely in existingSecret mode: the pre-existing - Secret is the source of truth, and inlining localhost URL defaults would - silently shadow keys (NEXT_PUBLIC_APP_URL, BETTER_AUTH_URL, etc.) the - user stored in their Secret but left empty in app.env. - */}} - {{- $appEnv := .Values.app.env | default dict }} - {{- $useExistingSecret := and .Values.app.secrets.existingSecret.enabled (not .Values.externalSecrets.enabled) }} - {{- if not $useExistingSecret }} - {{- range $key, $value := .Values.app.envDefaults | default dict }} - {{- $override := index $appEnv $key }} - {{- if and (ne (toString $value) "") (ne (toString $value) "") (or (not $override) (eq (toString $override) "") (eq (toString $override) "")) }} - - name: {{ $key }} - value: {{ $value | quote }} - {{- end }} - {{- end }} - {{- end }} - {{- /* - In existingSecret mode the chart-managed Secret is not rendered, so - non-empty values in app.env would otherwise go nowhere — inline them - here so the user's overrides actually reach the pod. Skipped in ESO - mode (ESO is the source of truth; inlining would mask the synced value) - and in inline mode (values flow through the chart-managed Secret). - */}} - {{- if and .Values.app.secrets.existingSecret.enabled (not .Values.externalSecrets.enabled) }} - {{- $chartComputed := list "DATABASE_URL" "SOCKET_SERVER_URL" "OLLAMA_URL" "PII_URL" }} - {{- range $key, $value := $appEnv }} - {{- if and (ne (toString $value) "") (ne (toString $value) "") (not (has $key $chartComputed)) }} - - name: {{ $key }} - value: {{ $value | quote }} - {{- end }} - {{- end }} - {{- end }} {{- if .Values.telemetry.enabled }} {{- $nodeEnv := default (default "production" (index (.Values.app.envDefaults | default dict) "NODE_ENV")) (index (.Values.app.env | default dict) "NODE_ENV") }} # OpenTelemetry configuration @@ -139,9 +129,8 @@ spec: - name: OTEL_RESOURCE_ATTRIBUTES value: "service.name=sim-app,service.version={{ .Chart.AppVersion }},deployment.environment={{ $nodeEnv }}" {{- end }} - {{- with .Values.extraEnvVars }} - {{- toYaml . | nindent 12 }} - {{- end }} + {{- /* Operator overrides last, so they win over the defaults above and over envFrom. */}} + {{- include "sim.appEnvOverrides" . | trim | nindent 12 }} envFrom: # Bundled Redis URL. Listed FIRST so any operator-supplied REDIS_URL # in a later source overrides it — see templates/configmap-redis.yaml. diff --git a/helm/sim/tests/secret-modes_test.yaml b/helm/sim/tests/secret-modes_test.yaml index ef7361c0444..8c296f5b87b 100644 --- a/helm/sim/tests/secret-modes_test.yaml +++ b/helm/sim/tests/secret-modes_test.yaml @@ -139,3 +139,105 @@ tests: - equal: path: stringData.NEXT_PUBLIC_APP_URL value: "https://app.example.com" + + - it: the migrations init container reads the sealing key and opt-out from the app Secret + template: deployment-app.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + asserts: + - contains: + path: spec.template.spec.initContainers[0].env + content: + name: ENCRYPTION_KEY + valueFrom: + secretKeyRef: { name: t-sim-app-secrets, key: ENCRYPTION_KEY, optional: false } + - contains: + path: spec.template.spec.initContainers[0].env + content: + name: ENCRYPT_ACCOUNT_TOKENS + valueFrom: + secretKeyRef: { name: t-sim-app-secrets, key: ENCRYPT_ACCOUNT_TOKENS, optional: true } + + - it: existingSecret mode hands an inline sealing opt-out to the migrations init container + template: deployment-app.yaml + set: + app.secrets.existingSecret.enabled: true + app.secrets.existingSecret.name: my-existing-secret + app.env.ENCRYPT_ACCOUNT_TOKENS: "false" + postgresql.auth.password: xxxxxxxx + asserts: + - contains: + path: spec.template.spec.initContainers[0].env + content: { name: ENCRYPT_ACCOUNT_TOKENS, value: "false" } + - contains: + path: spec.template.spec.initContainers[0].env + content: + name: ENCRYPTION_KEY + valueFrom: + secretKeyRef: { name: my-existing-secret, key: ENCRYPTION_KEY, optional: false } + + - it: an extraEnvVars sealing opt-out reaches the migrations init container + template: deployment-app.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + extraEnvVars: + - name: ENCRYPT_ACCOUNT_TOKENS + value: "false" + asserts: + - contains: + path: spec.template.spec.initContainers[0].env + content: { name: ENCRYPT_ACCOUNT_TOKENS, value: "false" } + - notContains: + path: spec.template.spec.initContainers[0].env + content: + name: ENCRYPT_ACCOUNT_TOKENS + valueFrom: + secretKeyRef: { name: t-sim-app-secrets, key: ENCRYPT_ACCOUNT_TOKENS, optional: true } + + - it: an envDefaults sealing opt-out reaches the migrations init container + template: deployment-app.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + app.envDefaults.ENCRYPT_ACCOUNT_TOKENS: "false" + postgresql.auth.password: xxxxxxxx + asserts: + - contains: + path: spec.template.spec.initContainers[0].env + content: { name: ENCRYPT_ACCOUNT_TOKENS, value: "false" } + + - it: an extraEnvVars ENCRYPTION_KEY override reaches the migrations init container + template: deployment-app.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + extraEnvVars: + - name: ENCRYPTION_KEY + valueFrom: + secretKeyRef: { name: vault-synced, key: encryption-key } + asserts: + - contains: + path: spec.template.spec.initContainers[0].env + content: + name: ENCRYPTION_KEY + valueFrom: + secretKeyRef: { name: vault-synced, key: encryption-key } + - contains: + path: spec.template.spec.containers[0].env + content: + name: ENCRYPTION_KEY + valueFrom: + secretKeyRef: { name: vault-synced, key: encryption-key } diff --git a/helm/sim/values.yaml b/helm/sim/values.yaml index 7817b60cb1b..678b52905c6 100644 --- a/helm/sim/values.yaml +++ b/helm/sim/values.yaml @@ -108,6 +108,11 @@ app: # Generate secure 32-character secrets using: openssl rand -hex 32 BETTER_AUTH_SECRET: "" # REQUIRED - set via --set flag or external secret manager ENCRYPTION_KEY: "" # REQUIRED - set via --set flag or external secret manager + # A 64-hex ENCRYPTION_KEY also seals stored OAuth account tokens, and the migrations init + # container seals the ones stored before. To opt out, set ENCRYPT_ACCOUNT_TOKENS to "false" + # or "0" wherever the app reads its env: here (inline or existingSecret mode), as a key in the + # pre-created Secret, mapped in externalSecrets.remoteRefs.app (External Secrets mode), or in + # app.envDefaults / extraEnvVars. The init container resolves it the same way as the app. INTERNAL_API_SECRET: "" # REQUIRED - set via --set flag or external secret manager, used for internal service-to-service authentication # Optional: Scheduled Jobs Authentication @@ -2004,6 +2009,9 @@ externalSecrets: CRON_SECRET: "" # Path to API_ENCRYPTION_KEY in external store (optional) API_ENCRYPTION_KEY: "" + # Path to ENCRYPT_ACCOUNT_TOKENS in external store (optional; "false" or "0" opts out of + # account token sealing) + ENCRYPT_ACCOUNT_TOKENS: "" # Path to MSHIP_SYSPROMPT_OVERRIDE in external store (optional) MSHIP_SYSPROMPT_OVERRIDE: "" # Path to REDIS_URL in external store (optional) diff --git a/packages/db/script-migrations/0037_seal_account_tokens.integration.ts b/packages/db/script-migrations/0037_seal_account_tokens.integration.ts index 0e3bb85de79..1a20acdff36 100644 --- a/packages/db/script-migrations/0037_seal_account_tokens.integration.ts +++ b/packages/db/script-migrations/0037_seal_account_tokens.integration.ts @@ -1,10 +1,16 @@ -import { sealAccountTokens } from '@sim/db/script-migrations/0037_seal_account_tokens' +import { + sealAccountTokens, + sealAccountTokensMigration, +} from '@sim/db/script-migrations/0037_seal_account_tokens' +import { runScriptMigrations } from '@sim/db/script-migrations/index' import { openText } from '@sim/db/sealed-text' import { readTestDatabaseUrl } from '@sim/db/testing/test-infrastructure' import { seal, sealingKeyId } from '@sim/security/encryption' +import { createDeferred } from '@sim/testing/helpers/deferred' +import { sleep } from '@sim/utils/helpers' import { generateId } from '@sim/utils/id' import postgres, { type Sql } from 'postgres' -import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest' +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest' const databaseUrl = readTestDatabaseUrl() const KEY_HEX = 'a'.repeat(64) @@ -196,4 +202,175 @@ describe('sealing stored account tokens in PostgreSQL', () => { refresh_token: 'own-refresh', }) }) + + describe('during migrate', () => { + const MIGRATION = sealAccountTokensMigration.name + + beforeEach(async () => { + await sql`DROP TABLE IF EXISTS script_migrations` + await sql`CREATE TABLE script_migrations ( + name text PRIMARY KEY, applied_at timestamptz NOT NULL DEFAULT now())` + await markSealedReaderRelease('2 days') + }) + + afterEach(async () => { + vi.unstubAllEnvs() + await admin.unsafe(`DROP FUNCTION IF EXISTS "${schemaName}".on_account_update() CASCADE`) + }) + + /** When the database first ran the migrate step of a release that reads sealed tokens. */ + async function markSealedReaderRelease(ago: string) { + await sql`INSERT INTO script_migrations (name, applied_at) + VALUES ('0036_archive_legacy_scheduled_jobs', now() - ${ago}::interval) + ON CONFLICT (name) DO UPDATE SET applied_at = EXCLUDED.applied_at` + } + + async function migrate() { + await runScriptMigrations(sql, [sealAccountTokensMigration]) + } + + async function isRecorded() { + const rows = await sql`SELECT 1 FROM script_migrations WHERE name = ${MIGRATION}` + return rows.length > 0 + } + + it('seals a table of plaintext and sealed tokens, then records the backfill', async () => { + const key = Buffer.from(KEY_HEX, 'hex') + const sealedAccess = seal('sealed-access', key, ACCESS_CONTEXT) + const sealedRefresh = seal('sealed-refresh', key, REFRESH_CONTEXT) + const presealed = await insertAccount(sealedAccess, sealedRefresh) + await insertAccount('plain-access', 'plain-refresh') + await insertAccount(null, 'refresh-only') + + await migrate() + + const rows = await storedRows() + expect(rows.find((row) => row.id === presealed)).toMatchObject({ + access_token: sealedAccess, + refresh_token: sealedRefresh, + }) + expect( + rows.map((row) => ({ + access: row.access_token && openText(row.access_token, ACCESS_CONTEXT), + refresh: row.refresh_token && openText(row.refresh_token, REFRESH_CONTEXT), + })) + ).toEqual( + expect.arrayContaining([ + { access: 'sealed-access', refresh: 'sealed-refresh' }, + { access: 'plain-access', refresh: 'plain-refresh' }, + { access: null, refresh: 'refresh-only' }, + ]) + ) + for (const row of rows) { + if (row.access_token !== null) expect(row.access_token).toMatch(/^enc:v1:/) + expect(row.refresh_token).toMatch(/^enc:v1:/) + } + expect(await isRecorded()).toBe(true) + }) + + it.each([ + { name: 'sealing is opted out with false', env: { ENCRYPT_ACCOUNT_TOKENS: 'false' } }, + { name: 'sealing is opted out with 0', env: { ENCRYPT_ACCOUNT_TOKENS: '0' } }, + { + name: 'ENCRYPTION_KEY is not hex', + env: { ENCRYPTION_KEY: 'z'.repeat(64) }, + }, + ])( + 'stays pending without touching a token while $name, and runs once sealing is on', + async ({ env }) => { + await insertAccount('plain-access', 'plain-refresh') + const before = await storedRows() + for (const [name, value] of Object.entries(env)) vi.stubEnv(name, value) + + await migrate() + await migrate() + + expect(await storedRows()).toEqual(before) + expect(await isRecorded()).toBe(false) + + vi.unstubAllEnvs() + await migrate() + + expect((await storedRows())[0].access_token).toMatch(/^enc:v1:/) + expect(await isRecorded()).toBe(true) + } + ) + + it('waits out the rollout after an upgrade from a release that cannot read sealed tokens', async () => { + await insertAccount('plain-access', 'plain-refresh') + const before = await storedRows() + await markSealedReaderRelease('1 minute') + + await migrate() + + expect(await storedRows()).toEqual(before) + expect(await isRecorded()).toBe(false) + + await markSealedReaderRelease('2 hours') + await migrate() + + expect((await storedRows())[0].access_token).toMatch(/^enc:v1:/) + expect(await isRecorded()).toBe(true) + }) + + it('defers instead of failing the deploy when a batch times out on a locked row', async () => { + const locked = await insertAccount('locked-access', 'locked-refresh') + const held = createDeferred() + const release = createDeferred() + const holding = sql.begin(async (tx) => { + await tx`SELECT id FROM account WHERE id = ${locked} FOR UPDATE` + held.resolve() + await release.promise + }) + await held.promise + + try { + await migrate() + expect(await isRecorded()).toBe(false) + } finally { + release.resolve() + await holding + } + expect((await storedRows())[0].access_token).toBe('locked-access') + }, 30_000) + + /** Makes every account UPDATE in this schema run `body` first, as the batch's own statement. */ + async function onAccountUpdate(body: string) { + await admin.unsafe(`CREATE FUNCTION "${schemaName}".on_account_update() RETURNS trigger + LANGUAGE plpgsql AS $$ BEGIN ${body}; RETURN NEW; END $$`) + await admin.unsafe(`CREATE TRIGGER on_account_update BEFORE UPDATE ON "${schemaName}".account + FOR EACH ROW EXECUTE FUNCTION "${schemaName}".on_account_update()`) + } + + it('defers when a batch hits its statement timeout', async () => { + await insertAccount('slow-access', 'slow-refresh') + await onAccountUpdate( + `RAISE EXCEPTION 'canceling statement due to statement timeout' USING ERRCODE = '57014'` + ) + + await migrate() + + expect(await isRecorded()).toBe(false) + expect((await storedRows())[0].access_token).toBe('slow-access') + }) + + it('fails the run when an operator cancels a batch', async () => { + await insertAccount('cancelled-access', 'cancelled-refresh') + await onAccountUpdate('PERFORM pg_sleep(20)') + + const run = migrate() + for (let attempt = 0; attempt < 100; attempt++) { + const [cancelled] = await admin<{ cancelled: boolean }[]>` + SELECT pg_cancel_backend(pid) AS cancelled FROM pg_stat_activity + WHERE datname = current_database() AND wait_event = 'PgSleep' + AND query LIKE '%UPDATE account%'` + if (cancelled?.cancelled) break + await sleep(50) + } + + await expect(run).rejects.toThrow('canceling statement due to user request') + expect(await isRecorded()).toBe(false) + expect((await storedRows())[0].access_token).toBe('cancelled-access') + }, 30_000) + }) }) diff --git a/packages/db/script-migrations/0037_seal_account_tokens.ts b/packages/db/script-migrations/0037_seal_account_tokens.ts index 6a4354bf8c7..0c5f5a4d207 100644 --- a/packages/db/script-migrations/0037_seal_account_tokens.ts +++ b/packages/db/script-migrations/0037_seal_account_tokens.ts @@ -1,7 +1,15 @@ import { account } from '@sim/db/schema' -import { openText, readEncryptionKey, sealedTextContext, sealText } from '@sim/db/sealed-text' +import { type ScriptMigration, ScriptMigrationDeferred } from '@sim/db/script-migrations/types' +import { + accountTokenSealingDisabledReason, + openText, + readEncryptionKey, + sealedTextContext, + sealText, +} from '@sim/db/sealed-text' import { createLogger } from '@sim/logger' import { sealedKeyId, sealingKeyId } from '@sim/security/encryption' +import { getPostgresCancellationReason } from '@sim/utils/errors' import { sleep } from '@sim/utils/helpers' import type { Sql } from 'postgres' @@ -18,7 +26,7 @@ interface AccountTokenSealingOptions { /** Counts the rows that would change without writing any. */ dryRun?: boolean batchSize?: number - /** Pause between batches, which keeps the backfill off the write path's latency. */ + /** Pause after each batch that rewrote rows, which keeps the backfill off the write path's latency. */ pauseMs?: number } @@ -129,8 +137,64 @@ export async function sealAccountTokens( } logger.info('Account token batch processed', { mode, dryRun, afterId, ...result }) - if (pauseMs > 0) await sleep(pauseMs) + if (pauseMs > 0 && changes.length > 0 && !dryRun) await sleep(pauseMs) } return result } + +/** Recorded by the migrate step of v0.9.33, the first release whose processes read sealed tokens. */ +const SEALED_READER_RELEASE_MIGRATION = '0036_archive_legacy_scheduled_jobs' + +/** + * How long after an upgrade from a release older than v0.9.33 the backfill waits, so a rolling + * deploy has replaced every process that cannot read sealed tokens before all of them are sealed. + */ +const SEALED_READER_ROLLOUT_GRACE = '1 hour' + +/** + * The lock and statement timeouts a batch sets on itself; either leaves the rest for the next run. + * An operator's cancel (`user_cancel`) is not one of them, so it fails the run visibly. + */ +const BATCH_TIMEOUTS = new Set(['lock_timeout', 'statement_timeout']) + +/** + * Seals the account tokens stored before this deployment sealed its writes, during `migrate`. + * Defers, leaving itself unrecorded, while writes would still store plaintext (sealing opted out, + * or no usable `ENCRYPTION_KEY` on the migrations container), so it runs once an operator enables + * sealing. Also defers within the rollout grace after an upgrade from a release that cannot read + * sealed tokens, and when a batch times out on a lock: an unfinished pass resumes on the next run, + * which is safer than failing the deploy over a backfill that reads never need. + */ +export const sealAccountTokensMigration: ScriptMigration = { + name: '0037_seal_account_tokens', + async up(sql) { + const disabledReason = accountTokenSealingDisabledReason() + if (disabledReason) { + throw new ScriptMigrationDeferred(`account token writes are unsealed: ${disabledReason}`) + } + const [marker] = await sql<{ recent: boolean }[]>` + SELECT applied_at > now() - ${SEALED_READER_ROLLOUT_GRACE}::interval AS recent + FROM script_migrations WHERE name = ${SEALED_READER_RELEASE_MIGRATION} + ` + if (marker?.recent) { + throw new ScriptMigrationDeferred( + `v0.9.33, the first release that reads sealed tokens, migrated this database less than ${SEALED_READER_ROLLOUT_GRACE} ago` + ) + } + try { + const result = await sealAccountTokens(sql, { + mode: 'seal', + expectedKeyId: sealingKeyId(readEncryptionKey()), + }) + if (result.foreignKey > 0) { + logger.warn('Account tokens sealed under another key were left as they are', result) + } + logger.info('Account token backfill finished', result) + } catch (error) { + const reason = getPostgresCancellationReason(error) + if (!reason || !BATCH_TIMEOUTS.has(reason)) throw error + throw new ScriptMigrationDeferred(`an account token batch hit its ${reason}`) + } + }, +} diff --git a/packages/db/script-migrations/index.ts b/packages/db/script-migrations/index.ts index d36b11985ac..24e766f610a 100644 --- a/packages/db/script-migrations/index.ts +++ b/packages/db/script-migrations/index.ts @@ -15,6 +15,8 @@ import { validateFileOwnershipMigration } from '@sim/db/script-migrations/0033_v import { backfillPublicShareEntitiesMigration } from '@sim/db/script-migrations/0034_backfill_public_share_entities' import { validateFileWorkspaceBindingMigration } from '@sim/db/script-migrations/0035_validate_file_workspace_binding' import { archiveLegacyScheduledJobsMigration } from '@sim/db/script-migrations/0036_archive_legacy_scheduled_jobs' +import { sealAccountTokensMigration } from '@sim/db/script-migrations/0037_seal_account_tokens' +import { createLogger } from '@sim/logger' import type { Sql } from 'postgres' import { backfillTableOrderKeys } from './0001_backfill_table_order_keys' import { backfillPausedBillingAttribution } from './0002_backfill_paused_billing_attribution' @@ -25,10 +27,13 @@ import { repairUnknownTableRowProvenanceSecondPass } from './0006_repair_unknown import { repairUnknownWorkspaceFileProvenance } from './0007_repair_unknown_workspace_file_provenance' import { backfillCredentialGroupResourcePolicies } from './0010_backfill_credential_group_resource_policies' import { remapLegacyKnowledgeConnectorCredentialsMigration } from './0011_remap_legacy_knowledge_connector_credentials' -import type { ScriptMigration } from './types' +import { type ScriptMigration, ScriptMigrationDeferred } from './types' export type { ScriptMigration } from './types' +/** The migrations container's progress log, which operators read on every upgrade. */ +const logger = createLogger('ScriptMigrations', { enabled: true, logLevel: 'INFO' }) + /** * Ordered, append-only registry of script migrations. An entry may be deleted * once a later migration supersedes it (accepting that deployments which @@ -72,6 +77,8 @@ export const scriptMigrations: readonly ScriptMigration[] = [ validateFileWorkspaceBindingMigration, /** Scheduled jobs run again from the schedule tick; none written for the retired runner may start. */ archiveLegacyScheduledJobsMigration, + /** Defers until this deployment seals account token writes; see its `up`. */ + sealAccountTokensMigration, ] /** @@ -83,6 +90,8 @@ export const scriptMigrations: readonly ScriptMigration[] = [ * * Fails fast: a missing required env var or a throwing `up` aborts the run * before the name is recorded, so the migration retries on the next upgrade. + * An `up` that throws `ScriptMigrationDeferred` is left unrecorded without + * failing the run. * * `migrations` defaults to the registry; callers may supply a scoped list. */ @@ -121,7 +130,7 @@ export async function runScriptMigrations( const pending = migrations.filter((migration) => !applied.has(migration.name)) if (pending.length === 0) { - console.log('No pending script migrations.') + logger.info('No pending script migrations.') return } @@ -133,14 +142,25 @@ export async function runScriptMigrations( ) } } - console.log(`Applying script migration ${migration.name}...`) + logger.info(`Applying script migration ${migration.name}...`) const startedAt = Date.now() - await migration.up(sql) + try { + await migration.up(sql) + } catch (error) { + if (!(error instanceof ScriptMigrationDeferred)) throw error + logger.info( + `Script migration ${migration.name} deferred; it runs again on the next migrate.`, + { + reason: error.message, + } + ) + continue + } await sql.begin(async (tx) => { for (const name of [migration.name, ...(migration.supersedes ?? [])]) { await tx`INSERT INTO script_migrations (name) VALUES (${name}) ON CONFLICT (name) DO NOTHING` } }) - console.log(`Script migration ${migration.name} applied in ${Date.now() - startedAt}ms.`) + logger.info(`Script migration ${migration.name} applied in ${Date.now() - startedAt}ms.`) } } diff --git a/packages/db/script-migrations/types.ts b/packages/db/script-migrations/types.ts index 9228e7ebd97..5b32bbbe8ca 100644 --- a/packages/db/script-migrations/types.ts +++ b/packages/db/script-migrations/types.ts @@ -20,6 +20,11 @@ import type { Sql } from 'postgres' * script from the registry in the same PR instead. * - **Owns its transactions.** The runner deliberately does not wrap `up` in * one: backfills commit per batch and cannot be rolled back wholesale. + * - **May defer.** A migration that needs a configuration the deployment may + * not have yet throws {@link ScriptMigrationDeferred}: nothing is recorded, + * later migrations still run, and the runner tries it again on every run + * until it applies. No later migration may depend on the work of one that + * can defer. */ export interface ScriptMigration { /** Unique stable identifier recorded in `script_migrations`; never rename after release. */ @@ -35,3 +40,11 @@ export interface ScriptMigration { */ up(sql: Sql): Promise } + +/** Thrown by `up` to leave a migration unrecorded, without failing the run, until a later run. */ +export class ScriptMigrationDeferred extends Error { + constructor(reason: string) { + super(reason) + this.name = 'ScriptMigrationDeferred' + } +} diff --git a/packages/db/sealed-text.test.ts b/packages/db/sealed-text.test.ts index 0c99a2d0bb2..1297fc723c3 100644 --- a/packages/db/sealed-text.test.ts +++ b/packages/db/sealed-text.test.ts @@ -19,3 +19,29 @@ describe('sealedTextEquals', () => { expect(params).toEqual(['refresh-token', stored]) }) }) + +describe('account token sealing without a usable ENCRYPTION_KEY', () => { + beforeEach(() => { + vi.resetModules() + vi.stubEnv('ENCRYPTION_KEY', 'z'.repeat(64)) + vi.stubEnv('ENCRYPT_ACCOUNT_TOKENS', undefined) + }) + + it('writes plaintext and warns once at startup with the reason, without failing', async () => { + const output = vi.spyOn(console, 'log').mockImplementation(() => undefined) + const { account: freshAccount } = await import('@sim/db/schema') + const { reportAccountTokenSealing } = await import('@sim/db/sealed-text') + + reportAccountTokenSealing() + expect(freshAccount.accessToken.mapToDriverValue('access-token')).toBe('access-token') + reportAccountTokenSealing() + + const lines = output.mock.calls.map((call) => call.join(' ')) + const sealingLines = lines.filter((line) => line.includes('Account token sealing')) + expect(sealingLines).toHaveLength(1) + expect(sealingLines[0]).toContain('WARN') + expect(sealingLines[0]).toContain( + 'Account token sealing: disabled (ENCRYPTION_KEY is not a 64-character hex key)' + ) + }) +}) diff --git a/packages/db/sealed-text.ts b/packages/db/sealed-text.ts index bf9a11e3904..be66af57a78 100644 --- a/packages/db/sealed-text.ts +++ b/packages/db/sealed-text.ts @@ -4,10 +4,12 @@ import { toRecord } from '@sim/utils/object' import { type SQL, sql } from 'drizzle-orm' import { type AnyPgColumn, customType } from 'drizzle-orm/pg-core' -const logger = createLogger('SealedText') -/** Operators read the key id from this report, so it logs at any configured level. */ +/** Operators read the key id and the sealing state from this report, so it logs at any configured level. */ const reportLogger = createLogger('AccountTokenSealing', { enabled: true, logLevel: 'INFO' }) +const OPTED_OUT_REASON = 'ENCRYPT_ACCOUNT_TOKENS is false' +const UNUSABLE_KEY_REASON = 'ENCRYPTION_KEY is not a 64-character hex key' + let reportedUnusableKey = false let reportedSealing = false @@ -23,41 +25,51 @@ export function readEncryptionKey(): Buffer { return key } -function isWriteSealingRequested(): boolean { - const flag = process.env.ENCRYPT_ACCOUNT_TOKENS?.toLowerCase() - return flag === 'true' || flag === '1' +function isWriteSealingOptedOut(): boolean { + const flag = process.env.ENCRYPT_ACCOUNT_TOKENS?.trim().toLowerCase() + return flag === 'false' || flag === '0' +} + +/** + * Why this process writes account tokens unsealed, or null when it seals them. Sealing is on + * whenever `ENCRYPTION_KEY` is a usable 64-character hex key, unless `ENCRYPT_ACCOUNT_TOKENS` opts + * out with `false` or `0`. + */ +export function accountTokenSealingDisabledReason(): string | null { + if (isWriteSealingOptedOut()) return OPTED_OUT_REASON + return readEncryptionKeyOrNull() ? null : UNUSABLE_KEY_REASON +} + +function reportUnusableKey(): void { + if (reportedUnusableKey) return + reportedUnusableKey = true + reportLogger.warn(`Account token sealing: disabled (${UNUSABLE_KEY_REASON})`, { keyId: null }) } /** * The key writes are sealed with, or null to write plaintext. Reads open sealed values regardless, - * so every process can read both forms before any process writes the sealed one. Sealing requested - * without a usable key writes plaintext, which every process can still read, and reports the - * misconfiguration once instead of failing every token write. + * so every process can read both forms. A key that cannot seal writes plaintext, which every + * process can still read, and reports the misconfiguration once instead of failing every token + * write. */ function writeSealingKey(): Buffer | null { - if (!isWriteSealingRequested()) return null + if (isWriteSealingOptedOut()) return null const key = readEncryptionKeyOrNull() - if (!key && !reportedUnusableKey) { - reportedUnusableKey = true - logger.error( - 'ENCRYPT_ACCOUNT_TOKENS is set but ENCRYPTION_KEY is not a 64-character hex key; account tokens are written unsealed' - ) - } + if (!key) reportUnusableKey() return key } -/** Whether this process seals account tokens it writes: the flag is on and the key is usable. */ -export function isAccountTokenWriteSealingEnabled(): boolean { - return writeSealingKey() !== null -} - -/** Logs once per process whether writes seal account tokens, and the key id (never the key). */ +/** Logs once per process whether writes seal account tokens, why not, and the key id (never the key). */ export function reportAccountTokenSealing(): void { if (reportedSealing) return reportedSealing = true - const writeKey = writeSealingKey() - const key = writeKey ?? readEncryptionKeyOrNull() - reportLogger.info(`Account token sealing: ${writeKey ? 'enabled' : 'disabled'}`, { + const reason = accountTokenSealingDisabledReason() + if (reason === UNUSABLE_KEY_REASON) { + reportUnusableKey() + return + } + const key = readEncryptionKeyOrNull() + reportLogger.info(`Account token sealing: ${reason ? `disabled (${reason})` : 'enabled'}`, { keyId: key ? sealingKeyId(key) : null, }) }