Skip to content

Commit cf11071

Browse files
committed
feat(db): seal account tokens by default and backfill them during migrate
1 parent 94de080 commit cf11071

21 files changed

Lines changed: 423 additions & 47 deletions

‎apps/docs/content/docs/platform/self-hosting/environment-variables.mdx‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,14 +16,14 @@ Production Compose derives `DATABASE_URL` and `BETTER_AUTH_URL` on its service d
1616
| `DATABASE_URL` | PostgreSQL connection string |
1717
| `BETTER_AUTH_SECRET` | Auth secret, at least 32 characters: `openssl rand -hex 32`. **Must be identical on the app and realtime services** |
1818
| `BETTER_AUTH_URL` | Your app URL — must be the real public origin, not `localhost` |
19-
| `ENCRYPTION_KEY` | Encryption key, exactly 64 hex characters (32 bytes): `openssl rand -hex 32` |
19+
| `ENCRYPTION_KEY` | Encryption key, exactly 64 hex characters (32 bytes): `openssl rand -hex 32`. A key in any other format leaves [account token sealing](/platform/self-hosting/security#account-token-sealing) off |
2020
| `INTERNAL_API_SECRET` | Internal API secret, at least 32 characters: `openssl rand -hex 32` |
2121
| `NEXT_PUBLIC_APP_URL` | Public app URL |
2222

2323
`openssl rand -hex 32` prints 64 hex characters. Encryption keys require that format. The app skips its env schema validation; individual consumers can reject invalid values when first used. Realtime validates `BETTER_AUTH_SECRET` and `INTERNAL_API_SECRET` at boot and rejects values shorter than 32 characters.
2424

2525
<Callout type="error">
26-
Retain `ENCRYPTION_KEY` and `API_ENCRYPTION_KEY` across upgrades and restores. There is no supported in-place key rotation procedure; losing a key makes values encrypted with it unreadable — workspace and personal environment variables, stored provider API keys, MCP OAuth credentials, and deployment/chat secrets in the first case, user-generated Sim API keys in the second. Back them up separately from the database.
26+
Retain `ENCRYPTION_KEY` and `API_ENCRYPTION_KEY` across upgrades and restores. There is no supported in-place key rotation procedure; losing a key makes values encrypted with it unreadable — workspace and personal environment variables, stored provider API keys, MCP OAuth credentials, connected-account OAuth tokens, and deployment/chat secrets in the first case, user-generated Sim API keys in the second. Back them up separately from the database.
2727
</Callout>
2828

2929
## Feature prerequisites and coordination [#strongly-recommended]
@@ -39,6 +39,7 @@ Production Compose derives `DATABASE_URL` and `BETTER_AUTH_URL` on its service d
3939
| `AUTH_TRUSTED_PROXIES` | Comma-separated reverse-proxy IPs/CIDRs so the client IP cannot be forged through `X-Forwarded-For` |
4040
| `INTERNAL_API_BASE_URL` | Internal URL for server-side self-calls, e.g. `http://sim-app.simstudio.svc.cluster.local:3000`. Defaults to the public base URL. The Trigger.dev worker ignores it and uses the public app URL. |
4141
| `SIM_MCP_URL` | Public URL of the [Sim MCP server](/mcp/overview) when you serve it on its own host, e.g. `https://mcp.example.com/mcp`. Point that host at the app; Sim serves only the MCP endpoint and its OAuth metadata there, and stops serving `/api/mcp` on the app host so clients use one URL. Optional — defaults to `<NEXT_PUBLIC_APP_URL>/api/mcp` |
42+
| `ENCRYPT_ACCOUNT_TOKENS` | Set to `false` to stop sealing connected-account OAuth tokens on write and during `migrate`. Sealing is on by default with a 64-hex `ENCRYPTION_KEY`; see [Account token sealing](/platform/self-hosting/security#account-token-sealing) |
4243
| `DATABASE_REPLICA_URL` | Read-replica connection string for log listing, audit logs, and dashboard aggregations. Falls back to the primary when unset |
4344

4445
## AI providers

‎apps/docs/content/docs/platform/self-hosting/security.mdx‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ Five secrets drive the security of a deployment. Generate each with `openssl ran
4141
| Secret | Protects | Rotatable |
4242
|---|---|---|
4343
| `BETTER_AUTH_SECRET` | Session tokens | Yes — invalidates all sessions |
44-
| `ENCRYPTION_KEY` | Workspace env vars, stored provider keys, MCP OAuth credentials, deployment/chat secrets | **No** — see below |
44+
| `ENCRYPTION_KEY` | Workspace env vars, stored provider keys, MCP OAuth credentials, connected-account OAuth tokens, deployment/chat secrets | **No** — see below |
4545
| `API_ENCRYPTION_KEY` | Encrypted stored API-key material when configured | Rotation needs re-encryption; existing hash-based authentication continues, but old encrypted material cannot be decrypted with a new key |
4646
| `INTERNAL_API_SECRET` | Service-to-service calls | Yes — roll app and realtime together |
4747
| `CRON_SECRET` | Background job endpoints | Yes — roll app and cron together |
@@ -52,6 +52,19 @@ Five secrets drive the security of a deployment. Generate each with `openssl ran
5252

5353
`BETTER_AUTH_SECRET` must be **identical** on the app and realtime services — they share sessions through the database, and a mismatch means realtime rejects every authenticated socket.
5454

55+
### Account token sealing
56+
57+
Sim seals the OAuth access and refresh tokens of connected accounts at rest with `ENCRYPTION_KEY`. Each token is encrypted and bound to its column, so a value copied from the database is unusable without the key.
58+
59+
- **On by default** whenever `ENCRYPTION_KEY` is exactly 64 hex characters (`openssl rand -hex 32`). Each process logs `Account token sealing: enabled` with a key id at startup, never the key itself.
60+
- **Without a usable key**, tokens are stored in plain text and the app keeps working. Each process logs a single warning at startup: `Account token sealing: disabled (ENCRYPTION_KEY is not a 64-character hex key)`. Other features that use `ENCRYPTION_KEY` keep their current behavior.
61+
- **Tokens stored earlier** are sealed by the migrations container during `migrate`, in small batches. The step needs `ENCRYPTION_KEY` on the migrations container: the Helm chart and the Compose files pass the app's value. Until the key is usable, the step does nothing and stays pending, so it runs on a later `migrate` once sealing is on. It also waits for a later `migrate` when it can't finish because rows are busy, rather than failing the upgrade.
62+
- **Opt out** with `ENCRYPT_ACCOUNT_TOKENS=false` on the app, background workers, and migrations container. New writes are then stored in plain text, and tokens that are already sealed stay readable.
63+
64+
<Callout type="error">
65+
Once tokens are sealed, do not roll back to a release older than v0.9.33. Those releases cannot read sealed tokens, so every connected account would have to be reconnected. Opting out does not unseal stored tokens.
66+
</Callout>
67+
5568
### Storing them
5669

5770
Choose how to supply secrets:

‎apps/docs/content/docs/platform/self-hosting/upgrades.mdx‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -227,6 +227,8 @@ PROFILE=gpu # or cpu
227227
docker compose -f docker-compose.ollama.yml --profile "$PROFILE" up -d --build
228228
```
229229

230+
Never roll back below v0.9.33 once account tokens are sealed: those releases cannot read them, and every connected account would have to be reconnected. See [Account token sealing](/platform/self-hosting/security#account-token-sealing).
231+
230232
**Rollback after an incompatible schema change** requires restoring the pre-upgrade database backup. This loses writes made since the snapshot. Stop all writers before restoring:
231233

232234
1. Stop traffic and all processes that can write to the database, including the app, realtime service, scheduled jobs, and any background workers.
@@ -270,6 +272,10 @@ Migrations are Drizzle SQL files plus registered script (data) migrations, appli
270272

271273
Migrations are **forward-only**. Take the backup described in the upgrade procedure before applying them; there are no down-migrations.
272274

275+
<Callout type="warn">
276+
Upgrading from a release older than v0.9.33 with a rolling deploy leaves old pods running next to new ones for a while. Old pods cannot read sealed account tokens, so integrations can fail on them until the rollout finishes. The `migrate` backfill waits an hour after such an upgrade before it seals stored tokens, but tokens the new pods write are sealed straight away. To avoid that window, upgrade to v0.9.33 first, stop the old pods before starting the new ones, or set `ENCRYPT_ACCOUNT_TOKENS=false` for the upgrade and remove it once every pod runs the new release.
277+
</Callout>
278+
273279
Migrations apply in sequence, so skipping several releases at once generally works mechanically. It is riskier — those combinations get less testing, and you take several releases' worth of behavior changes in one step. Upgrade in smaller increments where you can.
274280

275281
### How concurrent replicas are handled
@@ -289,6 +295,7 @@ These control the migrator and are set on the `migrations` container, not the ap
289295
| Variable | Purpose |
290296
|---|---|
291297
| `MIGRATION_DATABASE_URL` | A **direct, non-pooled** database connection URL (DSN). Falls back to `DATABASE_URL`. Setting it also arms the backend-pid check that aborts the run if the session — and with it the advisory lock — is silently recycled mid-migration; with only `DATABASE_URL` set, that check is skipped |
298+
| `ENCRYPTION_KEY`, `ENCRYPT_ACCOUNT_TOKENS` | Let the migrator seal account tokens stored before [account token sealing](/platform/self-hosting/security#account-token-sealing) was on. The Helm init container reads both from the app Secret, and the Compose `migrations` services pass the `.env` values. Without a usable key the step stays pending and does nothing |
292299

293300
<Callout type="warn">
294301
Set `MIGRATION_DATABASE_URL` if anything sits between Sim and Postgres. Session advisory locks and session-level `SET`s do not survive PgBouncer transaction pooling — the lock silently does not hold, and consecutive statements can land on different backends. Point it at the database directly, bypassing the pooler.
@@ -299,7 +306,7 @@ Supply the migration URL separately from the app URL:
299306

300307
| Deployment | How to pass `MIGRATION_DATABASE_URL` |
301308
|---|---|
302-
| Helm | The init container reads only the database Secret. Include the key in a Secret supplied through `postgresql.auth.existingSecret` or `externalDatabase.existingSecret`; `app.env` does not reach it. The chart-managed database Secret carries only the password, and there is no dedicated values key for the migration URL |
309+
| Helm | The init container reads the database Secret, plus only `ENCRYPTION_KEY` and `ENCRYPT_ACCOUNT_TOKENS` from the app Secret. Include the key in a Secret supplied through `postgresql.auth.existingSecret` or `externalDatabase.existingSecret`; `app.env` does not reach it. The chart-managed database Secret carries only the password, and there is no dedicated values key for the migration URL |
303310
| Compose | Add the variable to the `migrations` service's explicit `environment:` list. Setting it in `.env` alone does not pass it to that service |
304311
| Separate migration run | Run the migration image yourself with a direct DSN before upgrading |
305312

‎apps/sim/.env.example‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ NEXT_PUBLIC_APP_URL=http://localhost:3000
5555
# DAYTONA_SHELL_SNAPSHOT_ID= # Mothership shell snapshot ref, required when Mothership runs code on Daytona
5656

5757
# Security (Required)
58-
ENCRYPTION_KEY=your_encryption_key # Use `openssl rand -hex 32` to generate, used to encrypt environment variables
58+
ENCRYPTION_KEY=your_encryption_key # Use `openssl rand -hex 32` to generate, used to encrypt environment variables and connected-account OAuth tokens (ENCRYPT_ACCOUNT_TOKENS=false opts the tokens out)
5959
INTERNAL_API_SECRET=your_internal_api_secret # Use `openssl rand -hex 32` to generate, used to encrypt internal api routes
6060
API_ENCRYPTION_KEY=your_api_encryption_key # Use `openssl rand -hex 32` to generate, used to encrypt api keys
6161
CRON_SECRET=your_cron_secret # Use `openssl rand -hex 32` to generate. Authenticates the scheduler against the background job endpoints (scheduled workflows, polling triggers, connector syncs)

‎apps/sim/background/seal-account-tokens.test.ts‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,6 @@ function emptyAccountTable() {
2828
describe('runSealAccountTokens', () => {
2929
beforeEach(() => {
3030
vi.stubEnv('ENCRYPTION_KEY', ENCRYPTION_KEY)
31-
vi.stubEnv('ENCRYPT_ACCOUNT_TOKENS', 'true')
3231
databaseMockFns.mockResolveDbUrl.mockReturnValue('postgresql://fixture/sim')
3332
mockPostgres.mockReset()
3433
mockPostgres.mockImplementation(emptyAccountTable)
@@ -48,12 +47,18 @@ describe('runSealAccountTokens', () => {
4847
expect(mockPostgres).not.toHaveBeenCalled()
4948
})
5049

51-
it('refuses to seal while this worker writes plaintext tokens, before connecting', async () => {
50+
it('seals by default once the worker holds a usable key', async () => {
51+
await expect(
52+
runSealAccountTokens({ mode: 'seal', dryRun: false, expectedKeyId: WORKER_KEY_ID })
53+
).resolves.toEqual({ keyId: WORKER_KEY_ID, scanned: 0, pending: 0, updated: 0, foreignKey: 0 })
54+
})
55+
56+
it('refuses to seal while this worker opts out of sealing, before connecting', async () => {
5257
vi.stubEnv('ENCRYPT_ACCOUNT_TOKENS', 'false')
5358

5459
await expect(
5560
runSealAccountTokens({ mode: 'seal', dryRun: false, expectedKeyId: WORKER_KEY_ID })
56-
).rejects.toThrow('ENCRYPT_ACCOUNT_TOKENS is not enabled')
61+
).rejects.toThrow('Refusing to seal: ENCRYPT_ACCOUNT_TOKENS is false')
5762
expect(mockPostgres).not.toHaveBeenCalled()
5863
})
5964

‎apps/sim/background/seal-account-tokens.ts‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
import { resolveDbUrl } from '@sim/db'
22
import { sealAccountTokens } from '@sim/db/script-migrations/0037_seal_account_tokens'
3-
import { isAccountTokenWriteSealingEnabled, readEncryptionKey } from '@sim/db/sealed-text'
3+
import { accountTokenSealingDisabledReason, readEncryptionKey } from '@sim/db/sealed-text'
44
import { createLogger } from '@sim/logger'
55
import { sealingKeyId } from '@sim/security/encryption'
66
import { task } from '@trigger.dev/sdk'
@@ -31,7 +31,8 @@ function assertIntegerInRange(name: string, value: number | undefined, min: numb
3131
* Runs the account-token sealing backfill (`0037_seal_account_tokens`) from a worker, which already
3232
* holds the database and the fleet's `ENCRYPTION_KEY`. Refuses before connecting when the payload
3333
* names a key id other than this worker's, so a pass can never seal rows under a key the fleet
34-
* cannot open, and refuses `seal` while this worker's writers still store plaintext.
34+
* cannot open, and refuses `seal` while this worker's writers still store plaintext (sealing opted
35+
* out with `ENCRYPT_ACCOUNT_TOKENS=false`).
3536
*
3637
* Opens its own single connection rather than reusing the `@sim/db` pool: the pass binds JS arrays
3738
* through the postgres.js tag, which needs the array type map the shared pool skips
@@ -50,9 +51,10 @@ export async function runSealAccountTokens(payload: SealAccountTokensPayload) {
5051
if (expectedKeyId !== keyId) {
5152
throw new Error(`ENCRYPTION_KEY has key id ${keyId}, not the expected ${expectedKeyId}`)
5253
}
53-
if (mode === 'seal' && !isAccountTokenWriteSealingEnabled()) {
54+
const disabledReason = accountTokenSealingDisabledReason()
55+
if (mode === 'seal' && disabledReason) {
5456
throw new Error(
55-
'Refusing to seal: ENCRYPT_ACCOUNT_TOKENS is not enabled on this worker, so writers still store plaintext'
57+
`Refusing to seal: ${disabledReason} on this worker, so writers still store plaintext`
5658
)
5759
}
5860

‎apps/sim/lib/core/config/env.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -144,7 +144,7 @@ export const env = createEnv({
144144
TURNSTILE_SECRET_KEY: z.string().min(1).optional(), // Cloudflare Turnstile secret key for captcha verification
145145
ENCRYPTION_KEY: z.string().min(32), // Key for encrypting sensitive data
146146
API_ENCRYPTION_KEY: z.string().min(32).optional(), // Dedicated key for encrypting API keys (optional for OSS)
147-
ENCRYPT_ACCOUNT_TOKENS: z.boolean().optional(), // Seal OAuth access/refresh tokens on write (read by @sim/db); takes effect only with a 64-hex ENCRYPTION_KEY. Enable only once every app, worker, and background-task deployment runs a release that reads sealed tokens
147+
ENCRYPT_ACCOUNT_TOKENS: z.boolean().optional(), // Opt-out: OAuth access/refresh tokens are sealed on write (read by @sim/db) and backfilled by migrate whenever ENCRYPTION_KEY is 64 hex characters; false or 0 writes plaintext. Sealed tokens need v0.9.33+ on every process that reads them
148148
INTERNAL_API_SECRET: z.string().min(32), // Secret for internal API authentication
149149
INTERNAL_JWT_SECRET: z.string().min(32).optional(), // Dedicated signing key for internal JWTs (falls back to INTERNAL_API_SECRET); separating limits blast radius if one leaks
150150

‎apps/sim/lib/oauth/credential-revocation.integration.ts‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -89,8 +89,10 @@ async function rawTokens() {
8989
return row
9090
}
9191

92+
/** Sealing is the default with the fixture's hex key; `false` is the operator opt-out. */
9293
function sealWrites(enabled: boolean) {
93-
process.env.ENCRYPT_ACCOUNT_TOKENS = enabled ? 'true' : 'false'
94+
if (enabled) Reflect.deleteProperty(process.env, 'ENCRYPT_ACCOUNT_TOKENS')
95+
else process.env.ENCRYPT_ACCOUNT_TOKENS = 'false'
9496
}
9597

9698
async function resolveToken(rowId = accountId) {
@@ -490,7 +492,7 @@ describe('OAuth refresh chains with sealed account tokens', () => {
490492
})
491493
})
492494

493-
it('keeps writing readable tokens when sealing is requested without a usable key', async () => {
495+
it('keeps writing readable tokens when ENCRYPTION_KEY cannot seal them', async () => {
494496
sealWrites(true)
495497
process.env.ENCRYPTION_KEY = 'not-hex'.repeat(6)
496498
providerAnswer = REFRESHED
@@ -499,6 +501,17 @@ describe('OAuth refresh chains with sealed account tokens', () => {
499501
expect(await rawTokens()).toMatchObject({ access_token: 'fresh-access' })
500502
})
501503

504+
it('writes plaintext tokens when the operator opts out of sealing', async () => {
505+
sealWrites(false)
506+
providerAnswer = REFRESHED
507+
508+
await expect(resolveToken()).resolves.toEqual({ accessToken: 'fresh-access', refreshed: true })
509+
expect(await rawTokens()).toMatchObject({
510+
access_token: 'fresh-access',
511+
refresh_token: 'issued-refresh',
512+
})
513+
})
514+
502515
it('rotates a chain stored before sealing was enabled', async () => {
503516
sealWrites(true)
504517
providerAnswer = REFRESHED

‎docker-compose.local.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,10 @@ services:
9191
working_dir: /app/packages/db
9292
environment:
9393
- DATABASE_URL=postgresql://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:-postgres}@db:5432/${POSTGRES_DB:-simstudio}
94+
# Seals account tokens stored before sealing was on; a no-op until ENCRYPTION_KEY is
95+
# 64 hex characters. ENCRYPT_ACCOUNT_TOKENS=false opts out here and on the app.
96+
- ENCRYPTION_KEY=${ENCRYPTION_KEY:-}
97+
- ENCRYPT_ACCOUNT_TOKENS=${ENCRYPT_ACCOUNT_TOKENS:-}
9498
depends_on:
9599
db:
96100
condition: service_healthy

‎docker-compose.ollama.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,10 @@ services:
8181
dockerfile: docker/db.Dockerfile
8282
environment:
8383
- DATABASE_URL=postgresql://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:-postgres}@db:5432/${POSTGRES_DB:-simstudio}
84+
# Seals account tokens stored before sealing was on; a no-op until ENCRYPTION_KEY is
85+
# 64 hex characters. ENCRYPT_ACCOUNT_TOKENS=false opts out here and on the app.
86+
- ENCRYPTION_KEY=${ENCRYPTION_KEY:-}
87+
- ENCRYPT_ACCOUNT_TOKENS=${ENCRYPT_ACCOUNT_TOKENS:-}
8488
depends_on:
8589
db:
8690
condition: service_healthy

0 commit comments

Comments
 (0)