diff --git a/.changeset/migrate-firebase.md b/.changeset/migrate-firebase.md new file mode 100644 index 000000000..bef7d6bf1 --- /dev/null +++ b/.changeset/migrate-firebase.md @@ -0,0 +1,5 @@ +--- +"clerk": minor +--- + +Add `clerk migrate export firebase`, and let `clerk migrate import` read Firebase exports with their password hashes. The project's hash parameters come from the export, the `--firebase-*` flags, or `CLERK_FIREBASE_SIGNER_KEY`, `CLERK_FIREBASE_SALT_SEPARATOR`, `CLERK_FIREBASE_ROUNDS` and `CLERK_FIREBASE_MEM_COST`. `clerk migrate` is still experimental and needs `CLERK_EXPERIMENTAL=migrate`. diff --git a/packages/cli-core/src/commands/migrate/README.md b/packages/cli-core/src/commands/migrate/README.md index a8ffca9a0..bb0b165cf 100644 --- a/packages/cli-core/src/commands/migrate/README.md +++ b/packages/cli-core/src/commands/migrate/README.md @@ -215,6 +215,7 @@ Gets users **out** of a source platform, so there is something to feed clerk migrate export # pick a platform clerk migrate export clerk --output users.json clerk migrate export supabase --db-url "postgres://postgres:...@db.xxx.supabase.co:5432/postgres" +clerk migrate export firebase --service-account ./service-account.json ``` The platform is an optional positional. Omitted, you get a picker built from @@ -222,11 +223,12 @@ the registry; given, it runs directly. Each platform resolves its own flags — what a Clerk export needs (a secret key) has nothing in common with what a database export needs. -**A credential the far end rejects is asked for again.** A connection string is -long, pasted by hand, masked as it is typed, and wrong in ways nothing local can -check: a typo'd host, a revoked password, the right server but the wrong -database. Only the connection can say, and by then the operator has answered -every other question the command asked. So that step — and only that step, +**A credential the far end rejects is asked for again.** Connection strings and +Firebase service account keys are long, pasted by hand, masked as they are +typed, and wrong in ways nothing local can check: a typo'd host, a revoked key, +an expired token, the right server but the wrong database. Only the connection +or the token exchange can say, and by then the operator has answered every +other question the command asked. So that step — and only that step, never a fetch already under way or a file already written — runs inside a retry: the failure is explained, the prompt comes back, and the rest of the export continues against whichever credential worked. Agent mode and a non-TTY @@ -237,6 +239,7 @@ told not to. | ---------- | -------------------------------- | ------------------- | | `clerk` | Clerk Backend API | `--source clerk` | | `supabase` | Supabase Postgres (`auth.users`) | `--source supabase` | +| `firebase` | Firebase Identity Toolkit | `--source firebase` | Every export is a [run](#the-run-store), and the file lands in the run folder as `export.json`. `--output` writes it somewhere else instead, @@ -257,20 +260,23 @@ The file is an envelope around the users: ``` `source` is what lets `clerk migrate import ` run with no -`--source`. +`--source`. A Firebase export adds `firebase`, the project's hash +parameters, when it could read them: the import then needs no `--firebase-*` +flags. When it couldn't, pass them to the import, or enter them at its prompt. `--json` prints the result on stdout instead — `{ target, run, output, users, skipped, coverage, next }` — and never prompts, so a missing credential exits 2 naming the flag to pass. `next`, and the commands printed after a human run, keep `--runs-dir`: the import looks the run ID up there. -| Flag | Platforms | Description | -| --------------------- | ---------- | --------------------------------------------------------- | -| `-o, --output ` | all | Write the export here instead of the run folder | -| `-y, --yes` | all | Do not prompt: fail on a bad credential | -| `--json` | all | Print the result as JSON; never prompts | -| `--runs-dir ` | all | Where runs are kept (see [the run store](#the-run-store)) | -| `--db-url ` | `supabase` | Postgres connection string | +| Flag | Platforms | Description | +| -------------------------- | ---------- | --------------------------------------------------------- | +| `-o, --output ` | all | Write the export here instead of the run folder | +| `-y, --yes` | all | Do not prompt: fail on a bad credential | +| `--json` | all | Print the result as JSON; never prompts | +| `--runs-dir ` | all | Where runs are kept (see [the run store](#the-run-store)) | +| `--db-url ` | `supabase` | Postgres connection string | +| `--service-account ` | `firebase` | Path to a service account key JSON file | `export clerk` also takes the targeting flags — it reads from a Clerk instance, so it resolves a key the same way `clerk migrate import` does, with one extra @@ -384,6 +390,94 @@ user to reset their password; this one carries the bcrypt digests across. It also keeps `raw_app_meta_data`, which is what the import's [checks](#checks) read for each user's providers. +#### `firebase` + +```sh +clerk migrate export firebase --service-account ./service-account.json +``` + +Create a key at **Project settings → Service accounts → Generate new private +key**. The account needs to read users and the project's password hash +parameters (`signIn.hashConfig`). Either of these works, both verified live: + +- **Firebase Authentication Admin** (`roles/firebaseauth.admin`). The simplest + option, but it can also modify users and auth settings. +- **Read-only:** Firebase Authentication Viewer (`roles/firebaseauth.viewer`) + plus a custom role that adds `firebaseauth.configs.getHashConfig`: + + ```sh + gcloud iam roles create clerkMigrateHashExport --project=PROJECT_ID \ + --title="Clerk migrate hash export" \ + --permissions=firebaseauth.configs.get,firebaseauth.configs.getHashConfig,firebaseauth.users.get + gcloud projects add-iam-policy-binding PROJECT_ID \ + --member=serviceAccount:SA_EMAIL --role=roles/firebaseauth.viewer + gcloud projects add-iam-policy-binding PROJECT_ID \ + --member=serviceAccount:SA_EMAIL --role=projects/PROJECT_ID/roles/clerkMigrateHashExport + ``` + +Without `firebaseauth.configs.getHashConfig`, users still export, but the +export can't read the hash parameters, and Firebase may redact the password +hashes themselves: the same permission covers both. It says which permission is +missing. A redacted hash can't be brought back by anything the import is given, +so to keep passwords, grant the permission and export again. When the hashes +came through and only the parameters are missing, the import needs +`--firebase-signer-key`, `--firebase-salt-separator`, `--firebase-rounds` and +`--firebase-mem-cost` (from **Authentication → Users → ⋮ → Password hash +parameters** in the Firebase console). Each flag can come +from its environment variable instead, `CLERK_FIREBASE_SIGNER_KEY`, +`CLERK_FIREBASE_SALT_SEPARATOR`, `CLERK_FIREBASE_ROUNDS` and +`CLERK_FIREBASE_MEM_COST`, which keeps the signer key out of `ps`. Set it from a +secret manager or a file (`export CLERK_FIREBASE_SIGNER_KEY="$(cat signer.key)"`) rather +than typing the key, or it lands in shell history anyway. A flag wins over the +export file, but a variable only fills in for an export file without the +parameters, so one left set from another project can't replace them, and a +partial flag set is refused rather than completed from them. At a terminal, +without either, the import +asks, masking the signer key. The export file keeps the parameters, signer key +included, owner-only in its run folder. + +Without `--service-account` you are prompted for it, the way `export supabase` +prompts for its connection string. The answer can be a path to the downloaded +file _or_ the key's JSON pasted whole, so a key kept in a password manager or a +CI secret never has to be written to disk. The prompt is masked, since the key +carries a private key. Agent mode cannot prompt, so it names the flag instead. + +Either way the key is validated before anything reaches the network, so +downloading the web app config by mistake fails in a second with the right +console page named rather than after an auth round-trip. Key material never +appears in output. + +Firebase's scrypt is a modified variant, so a digest is worthless without the +project's four hash parameters. The export **reads them from the project** and +saves them in the export file's envelope, so the import needs nothing more: + +``` +Password hash parameters +Read from the project and saved in the export file, so the import needs nothing more. +``` + +Reading the config needs a broader role than listing users, so if it is denied +the export still succeeds and points at **Authentication → Users → (⋮) → +Password hash parameters** instead. An export with no password hashes says so +and asks for nothing. + +A user whose hash is present but whose salt is not (or the reverse) has both +dropped: half a credential produces a user nobody can sign in as. + +`FIREBASE_AUTH_EMULATOR_HOST` is honoured, so this works against the local +Firebase emulator as well as production. The target line names the emulator +when it is set. + +**No `firebase-admin`.** The spike the plan called for was run and _passed_ — a +compiled binary can import the SDK and complete `listUsers`, so the known +Firestore-under-compile bug does not reach the Auth Admin surface. It was still +not adopted: the SDK is 74 MB across 158 packages, including Firestore and +Cloud Storage, which would roughly double the ~62 MB binary every user +downloads, to serve one subcommand. What it does here is an RS256 JWT traded +for an access token, then two Identity Toolkit endpoints, `accounts:batchGet` +(paged, 1000 users a request) and the project config, and Bun's Web Crypto +signs RS256 with no dependency at all. + ### `clerk migrate import` Reads an exported user file, maps it onto Clerk's user schema, checks every @@ -399,24 +493,30 @@ clerk migrate import users.json --source clerk --new-run --yes clerk migrate import users.json --source clerk --json --yes clerk migrate import users.json --source clerk --require-password --yes clerk migrate import users.json --source clerk --skip-legal-checks --yes +clerk migrate import users.json --source firebase --firebase-signer-key SIGNER_KEY \ + --firebase-salt-separator SALT_SEPARATOR --firebase-rounds 8 --firebase-mem-cost 14 --yes clerk migrate import users.json --source clerk --runs-dir ./runs --yes clerk migrate import users.json --source clerk --app app_123 --instance prod --yes clerk migrate import users.json --source clerk --secret-key sk_test_... -y clerk migrate import # a human is asked ``` -| Flag | Description | -| ----------------------- | ------------------------------------------------------------------- | -| `[file\|export-run-id]` | The export file, or the ID of the export run that wrote it | -| `--source ` | Where the file came from: one of the [sources](#sources) | -| `--dry-run` | Run the [checks](#checks) against the instance, and write nothing | -| `--allow-partial` | Import the users that pass, and record the rest as skipped | -| `--new-run` | Start a new run instead of [continuing](#re-running) an earlier one | -| `--require-password` | Import only users that carry a password digest | -| `--skip-legal-checks` | Import users with no legal acceptance into an instance requiring it | -| `-y, --yes` | Import without prompting | -| `--json` | Output as JSON. Never prompts, so importing needs `--yes` | -| `--runs-dir ` | Where runs are kept (see [the run store](#the-run-store)) | +| Flag | Description | +| --------------------------------------- | ------------------------------------------------------------------------------------------------- | +| `[file\|export-run-id]` | The export file, or the ID of the export run that wrote it | +| `--source ` | Where the file came from: one of the [sources](#sources) | +| `--dry-run` | Run the [checks](#checks) against the instance, and write nothing | +| `--allow-partial` | Import the users that pass, and record the rest as skipped | +| `--new-run` | Start a new run instead of [continuing](#re-running) an earlier one | +| `--require-password` | Import only users that carry a password digest | +| `--skip-legal-checks` | Import users with no legal acceptance into an instance requiring it | +| `--firebase-signer-key ` | Firebase base64 signer key (overrides the export file), or `CLERK_FIREBASE_SIGNER_KEY` (does not) | +| `--firebase-salt-separator ` | Firebase base64 salt separator, or `CLERK_FIREBASE_SALT_SEPARATOR` | +| `--firebase-rounds ` | Firebase scrypt rounds, or `CLERK_FIREBASE_ROUNDS` | +| `--firebase-mem-cost ` | Firebase scrypt memory cost, or `CLERK_FIREBASE_MEM_COST` | +| `-y, --yes` | Import without prompting | +| `--json` | Output as JSON. Never prompts, so importing needs `--yes` | +| `--runs-dir ` | Where runs are kept (see [the run store](#the-run-store)) | Plus the targeting flags from the table above: `--secret-key`, `--app` and `--instance`. @@ -426,7 +526,7 @@ has changed since (an `--output` path another export or an edit overwrote), the import exits 2 and imports nothing. The import records it as `fromExport`. A file `clerk migrate export` wrote carries its source, so it needs no `--source`, and a `--source` that contradicts it exits 2. Any other -file needs `--source`: a JSON array, a CSV, or NDJSON, one user per line (what +file needs `--source`: a JSON array, Firebase's own `{ "users": [...] }`, a CSV, or NDJSON, one user per line (what Auth0's bulk export job writes). NDJSON is read always for `.ndjson` and `.jsonl`, and for a `.json` file that doesn't parse whole. A leading BOM is ignored in JSON and CSV. A file that isn't valid JSON is named in the error. @@ -441,7 +541,9 @@ what to pass. `Import N users?`, and declining writes nothing. `--yes` skips the question. Without either — an agent, a non-TTY run, `--json` — the run prints the checks and exits 2 with the exact command to run. Printed commands shell-quote their -paths, keep `--json`, and put `` in place of a secret key. +paths, keep `--json`, and put `` in place of a secret key and +`SIGNER_KEY`, `SALT_SEPARATOR`, `ROUNDS` and `MEM_COST` in place of the +Firebase parameters. **Every run prints its target first**, then which [case](#re-running) applies, then the checks. @@ -829,6 +931,7 @@ brings across. Adding a platform is one file in `sources/` plus one line in | Key | Reads | Passwords | MFA | Metadata | | ---------- | ---------------------------- | --------- | ------- | -------- | | `clerk` | Clerk Dashboard export | partial | partial | partial | +| `firebase` | `firebase auth:export` | yes | no | no | | `supabase` | Supabase `auth.users` export | yes | no | partial | An unknown `--source` exits 2 with the list of valid keys. @@ -853,7 +956,7 @@ which style it uses. An identifier the source never confirmed is routed to field, because Clerk creates primary identifiers **verified** — sending an unconfirmed address there would silently promote it. -- **Boolean**: `true`/`false`. A CSV export stringifies these, so `"false"` is +- **Boolean** (`firebase`): `true`/`false`. A CSV export stringifies these, so `"false"` is read as false, not as a non-empty string. `TRUE`, `FALSE`, `t` and `f` read too, as a spreadsheet or psql writes them. - **Timestamp** (`supabase`): a nullable confirmation time. Any real value @@ -861,6 +964,36 @@ unconfirmed address there would silently promote it. A Clerk export keeps an unverified primary email or phone unverified. +### Firebase hash parameters + +Firebase uses a modified scrypt, so Clerk needs the project's four parameters +alongside each digest. Find them in the Firebase console under +**Authentication → Users → (⋮) → Password hash parameters**, and put them in +place of `SIGNER_KEY`, `SALT_SEPARATOR` and the two numbers: + +```sh +clerk migrate import users.json --source firebase -y \ + --firebase-signer-key SIGNER_KEY --firebase-salt-separator SALT_SEPARATOR \ + --firebase-rounds 8 --firebase-mem-cost 14 +``` + +A project with no salt separator passes `--firebase-salt-separator ""` (or +`CLERK_FIREBASE_SALT_SEPARATOR=""`). + +All four are **required as a set** — supplying some but not all is a usage error +naming what is missing. Rejecting a partial set prevents importing a digest that +cannot verify: users would import successfully and then be unable to sign in. + +The flags are read first, then the export file's envelope, which carries the +parameters when `clerk migrate export firebase` could read them from the +project, then the `CLERK_FIREBASE_*` variables, and last a prompt at a terminal. +The flags win, for an export file with no parameters or the wrong ones. They +must be the parameters that produced the exported hashes. +The envelope sits in the gitignored run folder, since the signer key is a +Firebase secret. + +An export with no password hashes needs no parameters at all. + ## Schema fields What a source maps _onto_. Every user is validated against this schema @@ -958,6 +1091,19 @@ The checks also read the instance's Frontend API `GET /v1/environment` attributes and enabled social providers. Nothing in `clerk migrate` writes instance settings: the checks print the `clerk config patch` to run instead. +`export firebase` talks to Google rather than to Clerk: + +| Method | Path | Used by | +| ------ | ---------------------------------------------- | ---------------------------------------------------- | +| `POST` | `https://oauth2.googleapis.com/token` | `export firebase` — RS256 assertion → access token | +| `GET` | `…/v1/projects/{project_id}/accounts:batchGet` | `export firebase` — pages users, 1000 at a time | +| `GET` | `…/admin/v2/projects/{project_id}/config` | `export firebase` — reads the scrypt hash parameters | + +The two Identity Toolkit paths are on `identitytoolkit.googleapis.com`, or on +`FIREBASE_AUTH_EMULATOR_HOST` when that is set. The emulator needs no token, so +the export skips that request, and it may not carry the hash parameters: pass +them to the import. + ## Notes - `userId` in the source file becomes the Clerk user's `external_id`. That is diff --git a/packages/cli-core/src/commands/migrate/export/firebase.test.ts b/packages/cli-core/src/commands/migrate/export/firebase.test.ts new file mode 100644 index 000000000..34e5fa804 --- /dev/null +++ b/packages/cli-core/src/commands/migrate/export/firebase.test.ts @@ -0,0 +1,589 @@ +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from "bun:test"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { CliError } from "../../../lib/errors.ts"; +import type { UserLine } from "../lib/run-store.ts"; +import { useCaptureLog } from "../../../test/lib/stubs.ts"; +import { setAssumeYes } from "../lib/assume-yes.ts"; +import { + buildFirebaseExport, + exportFirebase, + fetchAccessToken, + fetchAllFirebaseUsers, + fetchHashConfig, + formatHashConfigGuidance, + loadServiceAccount, + mapFirebaseUserToExport, + readServiceAccount, + signServiceAccountJwt, + type ServiceAccount, +} from "./firebase.ts"; + +const captured = useCaptureLog(); + +let workDir: string; +let originalCwd: string; +let originalFetch: typeof globalThis.fetch; +let requests: { url: string; body: unknown }[]; +let account: ServiceAccount; + +let originalMode: string | undefined; + +beforeAll(async () => { + originalMode = process.env.CLERK_MODE; + originalCwd = process.cwd(); + originalFetch = globalThis.fetch; + workDir = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), "clerk-migrate-fb-"))); + process.chdir(workDir); + + // A real RSA key, so the signing path is genuinely exercised. + const pair = await crypto.subtle.generateKey( + { + name: "RSASSA-PKCS1-v1_5", + modulusLength: 2048, + publicExponent: new Uint8Array([1, 0, 1]), + hash: "SHA-256", + }, + true, + ["sign", "verify"], + ); + const pkcs8 = await crypto.subtle.exportKey("pkcs8", pair.privateKey); + const body = btoa(String.fromCharCode(...new Uint8Array(pkcs8))).replace(/(.{64})/g, "$1\n"); + + account = { + project_id: "demo-fb", + client_email: "exp@demo-fb.iam.gserviceaccount.com", + private_key: `-----BEGIN PRIVATE KEY-----\n${body}\n-----END PRIVATE KEY-----\n`, + }; + fs.writeFileSync( + path.join(workDir, "sa.json"), + JSON.stringify({ type: "service_account", ...account }), + ); +}); + +afterAll(() => { + if (originalMode === undefined) delete process.env.CLERK_MODE; + else process.env.CLERK_MODE = originalMode; + globalThis.fetch = originalFetch; + process.chdir(originalCwd); + fs.rmSync(workDir, { recursive: true, force: true }); +}); + +beforeEach(() => { + // Tests that need a prompt set human mode themselves; without this a + // leaked "human" from an earlier test stops a later one on the destination prompt. + process.env.CLERK_MODE = "agent"; + requests = []; + delete process.env.FIREBASE_AUTH_EMULATOR_HOST; + fs.rmSync(path.join(workDir, ".clerk"), { recursive: true, force: true }); + fs.rmSync(path.join(workDir, "exports"), { recursive: true, force: true }); +}); + +afterEach(() => { + globalThis.fetch = originalFetch; + delete process.env.FIREBASE_AUTH_EMULATOR_HOST; +}); + +/** Answers the token exchange, then one page per entry in `pages`. */ +function stubFirebase(pages: Record[][], hashConfig?: unknown) { + let page = 0; + globalThis.fetch = (async (input: string | URL | Request, init?: RequestInit) => { + const url = input.toString(); + requests.push({ url, body: init?.body ?? null }); + + if (url.includes("oauth2.googleapis.com/token")) { + return Response.json({ access_token: "tok" }); + } + if (url.includes("/config")) { + return hashConfig === undefined + ? new Response("forbidden", { status: 403 }) + : Response.json(hashConfig); + } + const current = pages[page++] ?? []; + const hasMore = page < pages.length; + return Response.json({ users: current, ...(hasMore ? { nextPageToken: `p${page}` } : {}) }); + }) as unknown as typeof fetch; +} + +const fbUser = (i: number, overrides: Record = {}) => ({ + localId: `fb${i}`, + email: `u${i}@fb.dev`, + emailVerified: true, + displayName: `User ${i}`, + passwordHash: `SGFzaA${i}`, + salt: `U2FsdA${i}`, + createdAt: "1704067200000", + ...overrides, +}); + +describe("loadServiceAccount", () => { + // The prompt takes either, so a key pasted out of a password manager never + // has to be written to disk first. + test("accepts the key JSON pasted whole", () => { + expect(loadServiceAccount(` ${JSON.stringify(account)} `).project_id).toBe("demo-fb"); + }); + + test("accepts a path to the key file", () => { + expect(loadServiceAccount("./sa.json").project_id).toBe("demo-fb"); + }); + + test("rejects a paste that is not valid JSON", () => { + expect(() => loadServiceAccount('{"project_id":')).toThrow(/pasted key is not valid JSON/); + }); + + test("rejects a paste missing a required field", () => { + expect(() => loadServiceAccount('{"type":"service_account"}')).toThrow( + /The pasted key is not a usable service account key: "project_id" is missing/, + ); + }); +}); + +describe("readServiceAccount", () => { + test("reads a valid key file", () => { + expect(readServiceAccount("./sa.json").project_id).toBe("demo-fb"); + }); + + test("reports a path that is not there", () => { + expect(() => readServiceAccount("./nope.json")).toThrow(/No service account file at/); + }); + + test("reports a file that is not JSON", () => { + fs.writeFileSync(path.join(workDir, "bad.json"), "not json"); + expect(() => readServiceAccount("./bad.json")).toThrow(/is not valid JSON/); + }); + + // Downloading the web app config instead of a service account key is the + // usual mistake, and the two files look similar at a glance. + test("points at the right console page for a web app config", () => { + fs.writeFileSync(path.join(workDir, "web.json"), JSON.stringify({ apiKey: "x" })); + expect(() => readServiceAccount("./web.json")).toThrow(/"project_id" is missing/); + }); + + test("names a wrong type explicitly", () => { + fs.writeFileSync(path.join(workDir, "wrong.json"), JSON.stringify({ type: "authorized_user" })); + expect(() => readServiceAccount("./wrong.json")).toThrow( + /"type" is "authorized_user".*Generate new private key/s, + ); + }); + + test.each([["project_id"], ["client_email"], ["private_key"]])( + "reports a missing %s", + (field) => { + const partial: Record = { type: "service_account", ...account }; + delete partial[field]; + fs.writeFileSync(path.join(workDir, `no-${field}.json`), JSON.stringify(partial)); + expect(() => readServiceAccount(`./no-${field}.json`)).toThrow( + new RegExp(`"${field}" is missing`), + ); + }, + ); + + // Pasting a key through a form that eats newlines is common, and the failure + // would otherwise surface as an opaque crypto error. + test("catches a private key whose newlines were mangled", () => { + fs.writeFileSync( + path.join(workDir, "mangled.json"), + JSON.stringify({ type: "service_account", ...account, private_key: "mangled" }), + ); + expect(() => readServiceAccount("./mangled.json")).toThrow(/newlines survived copying/); + }); + + test("raises CliError so the global handler formats it", () => { + expect(() => readServiceAccount("./nope.json")).toThrow(CliError); + }); +}); + +describe("signServiceAccountJwt", () => { + test("produces a three-segment RS256 JWT", async () => { + const jwt = await signServiceAccountJwt(account); + expect(jwt.split(".")).toHaveLength(3); + }); + + test("claims the right issuer, audience and scopes", async () => { + const jwt = await signServiceAccountJwt(account, 1_700_000_000); + const claims = JSON.parse( + atob((jwt.split(".")[1] as string).replace(/-/g, "+").replace(/_/g, "/")), + ); + + expect(claims).toMatchObject({ + iss: "exp@demo-fb.iam.gserviceaccount.com", + aud: "https://oauth2.googleapis.com/token", + iat: 1_700_000_000, + exp: 1_700_003_600, + }); + expect(claims.scope).toContain("cloud-platform"); + }); + + test("declares RS256 in the header", async () => { + const jwt = await signServiceAccountJwt(account); + const header = JSON.parse( + atob((jwt.split(".")[0] as string).replace(/-/g, "+").replace(/_/g, "/")), + ); + expect(header).toEqual({ alg: "RS256", typ: "JWT" }); + }); + + test("rejects a private key that is not valid base64", async () => { + await expect( + signServiceAccountJwt({ + ...account, + private_key: "-----BEGIN PRIVATE KEY-----\n!!!\n-----END PRIVATE KEY-----", + }), + ).rejects.toThrow(CliError); + }); +}); + +describe("fetchAccessToken", () => { + test("exchanges the assertion for a token", async () => { + stubFirebase([[]]); + + expect(await fetchAccessToken(account)).toBe("tok"); + expect(String(requests[0]?.body)).toContain("grant-type%3Ajwt-bearer"); + }); + + test("explains a rejection rather than surfacing a raw status", async () => { + globalThis.fetch = (async () => + new Response(JSON.stringify({ error_description: "Invalid JWT Signature" }), { + status: 400, + })) as unknown as typeof fetch; + + await expect(fetchAccessToken(account)).rejects.toThrow( + /Google rejected the service account \(400\): Invalid JWT Signature/, + ); + }); + + test("names the role the service account usually lacks", async () => { + globalThis.fetch = (async () => new Response("{}", { status: 403 })) as unknown as typeof fetch; + await expect(fetchAccessToken(account)).rejects.toThrow(/revoked or deleted/); + }); + + // The emulator has no token endpoint; `firebase-admin` uses the same bearer. + test("skips the exchange entirely against the emulator", async () => { + process.env.FIREBASE_AUTH_EMULATOR_HOST = "127.0.0.1:9099"; + globalThis.fetch = (async () => { + throw new Error("should not have been called"); + }) as unknown as typeof fetch; + + expect(await fetchAccessToken(account)).toBe("owner"); + }); +}); + +describe("fetchAllFirebaseUsers", () => { + test("follows nextPageToken until it stops coming", async () => { + stubFirebase([ + Array.from({ length: 1000 }, (_, i) => fbUser(i)), + Array.from({ length: 7 }, (_, i) => fbUser(1000 + i)), + ]); + + const all = await fetchAllFirebaseUsers({ account, token: "tok" }); + + expect(all).toHaveLength(1007); + expect(requests[1]?.url).toContain("nextPageToken=p1"); + }); + + test("asks for the endpoint's maximum page size", async () => { + stubFirebase([[]]); + await fetchAllFirebaseUsers({ account, token: "tok" }); + expect(requests[0]?.url).toContain("maxResults=1000"); + }); + + test("targets the project named in the key", async () => { + stubFirebase([[]]); + await fetchAllFirebaseUsers({ account, token: "tok" }); + expect(requests[0]?.url).toContain("/projects/demo-fb/accounts:batchGet"); + }); + + test("routes through the emulator when one is configured", async () => { + process.env.FIREBASE_AUTH_EMULATOR_HOST = "127.0.0.1:9099"; + stubFirebase([[]]); + + await fetchAllFirebaseUsers({ account, token: "owner" }); + + expect(requests[0]?.url).toStartWith("http://127.0.0.1:9099/"); + }); + + test("raises a clear error on a failed page", async () => { + globalThis.fetch = (async () => + new Response("nope", { status: 500 })) as unknown as typeof fetch; + + await expect(fetchAllFirebaseUsers({ account, token: "tok" })).rejects.toThrow( + /Firebase returned 500 listing users/, + ); + }); +}); + +describe("mapFirebaseUserToExport", () => { + test("keeps the fields the firebase transformer maps from", () => { + expect(mapFirebaseUserToExport(fbUser(0))).toEqual({ + localId: "fb0", + email: "u0@fb.dev", + displayName: "User 0", + createdAt: "1704067200000", + emailVerified: true, + passwordHash: "SGFzaA0", + salt: "U2FsdA0", + }); + }); + + test("drops project internals the import has no use for", () => { + const mapped = mapFirebaseUserToExport( + fbUser(0, { + providerUserInfo: [{ providerId: "password" }], + lastLoginAt: "1704153600000", + customAttributes: '{"role":"x"}', + validSince: "1704067200", + }), + ); + for (const noise of ["providerUserInfo", "lastLoginAt", "customAttributes", "validSince"]) { + expect(noise in mapped).toBe(false); + } + }); + + // A digest without its salt cannot be verified, so exporting one alone would + // produce a user nobody can sign in as. + test.each([ + ["hash without salt", { passwordHash: "H", salt: undefined }], + ["salt without hash", { passwordHash: undefined, salt: "S" }], + ])("drops a %s", (_label, overrides) => { + const mapped = mapFirebaseUserToExport(fbUser(0, overrides)); + expect("passwordHash" in mapped).toBe(false); + expect("salt" in mapped).toBe(false); + }); + + test("keeps emailVerified when it is false", () => { + expect(mapFirebaseUserToExport(fbUser(0, { emailVerified: false })).emailVerified).toBe(false); + }); + + test("copes with a phone-only user", () => { + const mapped = mapFirebaseUserToExport({ localId: "fb9", phoneNumber: "+15555550100" }); + expect(mapped).toEqual({ localId: "fb9", phoneNumber: "+15555550100" }); + }); +}); + +test("mapFirebaseUserToExport keeps disabled only when it is true", () => { + expect(mapFirebaseUserToExport(fbUser(0, { disabled: true })).disabled).toBe(true); + expect("disabled" in mapFirebaseUserToExport(fbUser(0, { disabled: false }))).toBe(false); +}); + +describe("buildFirebaseExport", () => { + test("counts coverage and records each user", () => { + const lines: UserLine[] = []; + const { users, coverage } = buildFirebaseExport( + [fbUser(0), { localId: "fb1", phoneNumber: "+1555" }], + (line) => lines.push(line), + ); + + expect(users).toHaveLength(2); + const byLabel = Object.fromEntries(coverage.map((c) => [c.label, c.count])); + expect(byLabel["have a password hash"]).toBe(1); + expect(byLabel["have a phone number"]).toBe(1); + expect(lines).toHaveLength(2); + }); + + // Firebase returns an empty passwordHash for users it did not hash itself. + test("counts password users whose hash Firebase did not return", () => { + const { users, unreadablePasswords } = buildFirebaseExport([ + fbUser(0, { providerUserInfo: [{ providerId: "password" }] }), + fbUser(1, { passwordHash: "", providerUserInfo: [{ providerId: "password" }] }), + fbUser(2, { passwordHash: "", providerUserInfo: [{ providerId: "google.com" }] }), + ]); + expect(unreadablePasswords).toBe(1); + expect("passwordHash" in users[1]!).toBe(false); + }); + + // Firebase sends base64 "REDACTED" when the caller may not read hashes: a + // digest that could never verify, which must not be exported as one. + test("treats a redacted hash as none, and counts it", () => { + const { users, redactedPasswords } = buildFirebaseExport([ + fbUser(0, { passwordHash: "UkVEQUNURUQ=", providerUserInfo: [{ providerId: "password" }] }), + ]); + expect(redactedPasswords).toBe(1); + expect("passwordHash" in users[0]!).toBe(false); + expect("salt" in users[0]!).toBe(false); + }); +}); + +describe("fetchHashConfig", () => { + test("reads the project's scrypt parameters", async () => { + stubFirebase([[]], { + signIn: { + hashConfig: { signerKey: "KEY=", saltSeparator: "Bw==", rounds: 8, memoryCost: 14 }, + }, + }); + + expect(await fetchHashConfig(account, "tok")).toEqual({ + signerKey: "KEY=", + saltSeparator: "Bw==", + rounds: 8, + memoryCost: 14, + }); + }); + + // Firebase leaves the field out when the project has no separator. + test("reads a project with no salt separator as an empty one", async () => { + stubFirebase([[]], { + signIn: { hashConfig: { signerKey: "KEY=", rounds: 8, memoryCost: 14 } }, + }); + + expect(await fetchHashConfig(account, "tok")).toEqual({ + signerKey: "KEY=", + saltSeparator: "", + rounds: 8, + memoryCost: 14, + }); + }); + + // Reading the config needs a broader role than listing users, so a project + // where it is denied must still export. + test("returns null rather than failing when the call is not permitted", async () => { + stubFirebase([[]]); + expect(await fetchHashConfig(account, "tok")).toBeNull(); + // Names the permission, so the operator can grant it rather than guess. + expect(captured.err).toContain("firebaseauth.configs.getHashConfig"); + }); + + // Every digest carries these, so a set Clerk would refuse is not written. + test("leaves out parameters Clerk would refuse, and says so", async () => { + stubFirebase([[]], { + signIn: { + hashConfig: { signerKey: "KEY=", saltSeparator: "Bw==", rounds: 17, memoryCost: 14 }, + }, + }); + expect(await fetchHashConfig(account, "tok")).toBeNull(); + expect(captured.err).toContain( + "won't work in Clerk (rounds must be a whole number from 1 to 16", + ); + }); + + test("returns null when the response carries no hash config", async () => { + stubFirebase([[]], { signIn: {} }); + expect(await fetchHashConfig(account, "tok")).toBeNull(); + }); +}); + +describe("formatHashConfigGuidance", () => { + const config = { signerKey: "KEY==", saltSeparator: "Bw==", rounds: 8, memoryCost: 14 }; + + // They are in the envelope now, so the import needs no flags for them. + test("says the parameters travel in the export file when the project gave them", () => { + const text = formatHashConfigGuidance(config, 3).join("\n"); + expect(text).toContain("saved in the export file"); + expect(text).not.toContain("--firebase-signer-key"); + }); + + test("says where to find them when the project would not say", () => { + const text = formatHashConfigGuidance(null, 3).join("\n"); + expect(text).toContain("Password hash parameters"); + expect(text).toContain("Authentication → Users"); + expect(text).toContain("--firebase-signer-key"); + }); + + // Nothing to configure, so nothing to tell them to configure. + test("says nothing is needed when the export has no hashes", () => { + expect(formatHashConfigGuidance(null, 0).join("\n")).toContain("no hash parameters are needed"); + }); +}); + +/** The envelope the one export run in this project wrote. */ +function onlyExportFile(): string { + const dir = path.join(workDir, ".clerk", "migrate"); + const entries = fs.readdirSync(dir); + expect(entries).toHaveLength(1); + return path.join(dir, entries[0] as string, "export.json"); +} + +/** The users inside that envelope. */ +function exportedUsers(): Record[] { + return ( + JSON.parse(fs.readFileSync(onlyExportFile(), "utf-8")) as { users: Record[] } + ).users; +} + +describe("exportFirebase", () => { + // `-y` is "do not prompt", at a terminal too. + test("does not ask for a service account key under -y", async () => { + process.env.CLERK_MODE = "human"; + setAssumeYes(true); + try { + await expect(exportFirebase({})).rejects.toThrow(/cannot prompt here/); + } finally { + setAssumeYes(false); + } + }); + + test("exports end to end and reports coverage", async () => { + stubFirebase([[fbUser(0), fbUser(1)]], { + signIn: { + hashConfig: { signerKey: "KEY=", saltSeparator: "Bw==", rounds: 8, memoryCost: 14 }, + }, + }); + + await exportFirebase({ serviceAccount: "./sa.json" }); + expect(JSON.parse(fs.readFileSync(onlyExportFile(), "utf-8"))).toMatchObject({ + source: "firebase", + firebase: { + base64_signer_key: "KEY=", + base64_salt_separator: "Bw==", + rounds: 8, + mem_cost: 14, + }, + }); + const written = exportedUsers(); + expect(written).toHaveLength(2); + expect(captured.err).toContain("Field coverage"); + expect(captured.err).toContain("demo-fb project"); + }); + + // Firebase's own variable, so it's honoured, but an emulator's users are + // not the project's: the target line says which. + test("names the emulator in the target line when one is set", async () => { + stubFirebase([[fbUser(0)]], { signIn: {} }); + process.env.FIREBASE_AUTH_EMULATOR_HOST = "localhost:9099"; + try { + await exportFirebase({ serviceAccount: "./sa.json" }); + } finally { + delete process.env.FIREBASE_AUTH_EMULATOR_HOST; + } + expect(captured.err).toContain("Source: firebase (emulator at localhost:9099)"); + }); + + test("names the command that consumes the file", async () => { + stubFirebase([[fbUser(0)]], { signIn: {} }); + // The suggestion now rides the gutter's Next steps block, which only + // renders in human mode. + process.env.CLERK_MODE = "human"; + await exportFirebase({ serviceAccount: "./sa.json", output: "exports/mine.json" }); + expect(captured.err).toMatch(/clerk migrate import \d{8}-\d{6}-[0-9a-f]{4}/); + }); + + test("--output controls the destination", async () => { + stubFirebase([[fbUser(0)]], { signIn: {} }); + await exportFirebase({ serviceAccount: "./sa.json", output: "fb.json" }); + expect(fs.existsSync(path.join(workDir, "fb.json"))).toBe(true); + }); + + // Human runs get a prompt instead; an agent has nobody to ask, so it is told + // which flag to pass. + test("agent mode names the flag rather than prompting", async () => { + await expect(exportFirebase({})).rejects.toThrow(/needs a service account key file/); + }); + + test("validates the key file before making any request", async () => { + stubFirebase([[fbUser(0)]]); + await expect(exportFirebase({ serviceAccount: "./nope.json" })).rejects.toThrow(CliError); + expect(requests).toHaveLength(0); + }); + + test("never puts key material in the output", async () => { + stubFirebase([[fbUser(0)]], { signIn: {} }); + await exportFirebase({ serviceAccount: "./sa.json" }); + expect(captured.err).not.toContain("BEGIN PRIVATE KEY"); + expect(captured.err).not.toContain(account.private_key.slice(40, 80)); + }); + + test("skips the hash-parameter section when nothing has a password", async () => { + stubFirebase([[{ localId: "fb9", phoneNumber: "+1555" }]], { signIn: {} }); + await exportFirebase({ serviceAccount: "./sa.json" }); + expect(captured.err).toContain("no hash parameters are needed"); + }); +}); diff --git a/packages/cli-core/src/commands/migrate/export/firebase.ts b/packages/cli-core/src/commands/migrate/export/firebase.ts new file mode 100644 index 000000000..0630fe887 --- /dev/null +++ b/packages/cli-core/src/commands/migrate/export/firebase.ts @@ -0,0 +1,643 @@ +/** + * `clerk migrate export firebase` — pull users out of Firebase Authentication. + * + * Ported from the standalone migration-tool's `src/export/firebase.ts`, but + * **without `firebase-admin`**. + * + * The spike the ticket asked for was run first, and it passed: a + * `bun build --compile` binary imports `firebase-admin`, initializes it, and + * completes `listUsers` against Identity Toolkit. The known Firestore-under- + * compile bug does not reach the Auth Admin surface. + * + * The SDK was still not adopted, on the second measurement: it is **74 MB + * across 158 packages**, including `@google-cloud/firestore` and + * `@google-cloud/storage`, neither of which this command touches. The compiled + * `clerk` binary is ~65 MB today, so that roughly doubles the artifact every + * user downloads — to serve one subcommand. + * + * What the SDK actually does here is two REST calls and an RS256 JWT, and Bun's + * Web Crypto signs RS256 with no dependency at all (verified compiled). So this + * adds **zero** packages, and its HTTP goes through `loggedFetch`, so a + * `--verbose` run shows the requests — which an SDK doing its own fetch would + * not. + */ + +import fs from "node:fs"; +import path from "node:path"; +import { CliError, ERROR_CODE, throwUsageError } from "../../../lib/errors.ts"; +import { bold, dim } from "../../../lib/color.ts"; +import { loggedFetch } from "../../../lib/fetch.ts"; +import { log } from "../../../lib/log.ts"; +import { password as passwordPrompt } from "../../../lib/prompts.ts"; +import { isHuman } from "../../../mode.ts"; +import { isAssumeYes } from "../lib/assume-yes.ts"; +import { firebaseHashConfigProblem } from "../lib/firebase-hash.ts"; +import { withGutter, withSpinner, type SpinnerControls } from "../../../lib/spinner.ts"; +import type { UserLine } from "../lib/run-store.ts"; +import { printTarget } from "../lib/target.ts"; +import type { FirebaseHashConfig } from "../types.ts"; +import { isCredentialStatus, throwApiFailure, withInputRetry } from "../lib/input-retry.ts"; +import { finishExport, startExportRun } from "./shared.ts"; + +/** Identity Toolkit's maximum for `accounts:batchGet`. */ +const PAGE_SIZE = 1000; + +const TOKEN_URL = "https://oauth2.googleapis.com/token"; +const SCOPES = [ + "https://www.googleapis.com/auth/cloud-platform", + "https://www.googleapis.com/auth/firebase", +].join(" "); + +const DOCS_URL = "https://clerk.com/docs/guides/development/migrating/firebase"; + +export type ExportFirebaseOptions = { + serviceAccount?: string; + output?: string; + /** Where runs are kept; overrides `CLERK_MIGRATE_DIR`. */ + runsDir?: string; + /** Print the result as JSON on stdout; never prompts. */ + json?: boolean; +}; + +export type ServiceAccount = { + project_id: string; + client_email: string; + private_key: string; +}; + +/** + * Validates already-parsed JSON as a service-account key. + * + * Every failure names the field, because the usual causes are downloading the + * wrong JSON from the console (a web app config rather than a service account) + * or pasting a key with its newlines mangled. + * + * @param label how to refer to the source in an error — a file name, or + * "the pasted key" when it came from the prompt. + */ +function validateServiceAccount(parsed: unknown, label: string): ServiceAccount { + const account = parsed as Partial & { type?: string }; + const invalid = (problem: string): never => { + throwUsageError(`${label} is not a usable service account key: ${problem}`, DOCS_URL); + }; + + if (account.type && account.type !== "service_account") { + invalid( + `its "type" is "${account.type}", not "service_account". Download a private key from ` + + "Project settings → Service accounts → Generate new private key.", + ); + } + for (const field of ["project_id", "client_email", "private_key"] as const) { + if (typeof account[field] !== "string" || account[field].length === 0) { + invalid(`"${field}" is missing`); + } + } + if (!account.private_key?.includes("PRIVATE KEY")) { + invalid('"private_key" does not look like a PEM key — check its newlines survived copying'); + } + + return account as ServiceAccount; +} + +/** Reads and validates a service-account key file. */ +export function readServiceAccount(file: string): ServiceAccount { + const resolved = path.resolve(process.cwd(), file); + + if (!fs.existsSync(resolved)) { + throw new CliError(`No service account file at ${resolved}.`, { + code: ERROR_CODE.FILE_NOT_FOUND, + docsUrl: DOCS_URL, + }); + } + + let parsed: unknown; + try { + parsed = JSON.parse(fs.readFileSync(resolved, "utf-8")); + } catch (error) { + throw new CliError(`${file} is not valid JSON: ${(error as Error).message}`, { + code: ERROR_CODE.INVALID_JSON, + docsUrl: DOCS_URL, + }); + } + + return validateServiceAccount(parsed, file); +} + +/** + * Accepts what the prompt accepts: a path to the downloaded key file, or the + * key's JSON pasted in whole. Console downloads land as a file, but a key + * copied out of a password manager or CI secret never touches disk. + */ +export function loadServiceAccount(source: string): ServiceAccount { + const trimmed = source.trim(); + if (!trimmed.startsWith("{")) return readServiceAccount(trimmed); + + let parsed: unknown; + try { + parsed = JSON.parse(trimmed); + } catch (error) { + throw new CliError(`The pasted key is not valid JSON: ${(error as Error).message}`, { + code: ERROR_CODE.INVALID_JSON, + docsUrl: DOCS_URL, + }); + } + + return validateServiceAccount(parsed, "The pasted key"); +} + +/** + * Resolves the key: the flag, then a prompt — the shape `export supabase` uses + * for its connection string. Prompted as a password: the JSON carries a private + * key, and a path typed blind is short enough to survive being masked. + */ +async function resolveServiceAccount(options: ExportFirebaseOptions): Promise { + if (options.serviceAccount) return loadServiceAccount(options.serviceAccount); + + // `-y` is "do not prompt", as for the other exports. + if (options.json || isAssumeYes() || !isHuman()) { + throwUsageError( + "`clerk migrate export firebase` needs a service account key file and cannot prompt here. " + + "Pass --service-account .", + DOCS_URL, + undefined, + [ + { + command: "clerk migrate export firebase --service-account ./service-account.json", + description: "Export using a downloaded service account key", + }, + ], + ); + } + + log.info( + dim("Firebase console → Project settings → Service accounts → Generate new private key."), + ); + + return promptServiceAccount(); +} + +/** + * Asks for the key, masked. + * + * Masked because the JSON carries a private key, and a path typed blind is + * short enough to survive it. What the file cannot tell us — whether Google + * still accepts the key — is left to the token exchange, which is why this is + * separate from {@link resolveServiceAccount}: a revoked key has to be asked + * for again after that call fails, not before it is made. + */ +async function promptServiceAccount(): Promise { + const answer = await passwordPrompt({ + message: "Path to the service account key file, or paste the key JSON", + validate: (value) => { + try { + loadServiceAccount(value ?? ""); + return undefined; + } catch (error) { + return error instanceof CliError ? error.message : String(error); + } + }, + }); + + return loadServiceAccount(answer); +} + +function base64Url(input: string | Uint8Array): string { + const binary = + typeof input === "string" ? input : String.fromCharCode(...(input as unknown as number[])); + return btoa(binary).replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, ""); +} + +/** Imports the PEM private key for RS256 signing. */ +async function importPrivateKey(pem: string): Promise { + const body = pem.replace(/-----[^-]+-----/g, "").replace(/\s+/g, ""); + let der: Uint8Array; + try { + der = Uint8Array.from(atob(body), (character) => character.charCodeAt(0)); + } catch { + throwUsageError("The service account's private_key is not valid base64.", DOCS_URL); + } + + try { + return await crypto.subtle.importKey( + "pkcs8", + der, + { name: "RSASSA-PKCS1-v1_5", hash: "SHA-256" }, + false, + ["sign"], + ); + } catch (error) { + throwUsageError( + `The service account's private_key could not be read: ${(error as Error).message}`, + DOCS_URL, + ); + } +} + +/** + * Signs the assertion Google exchanges for an access token. + * + * @param now - Seconds since the epoch; injectable so tests are not clock-bound. + */ +export async function signServiceAccountJwt( + account: ServiceAccount, + now: number = Math.floor(Date.now() / 1000), +): Promise { + const key = await importPrivateKey(account.private_key); + const claims = { + iss: account.client_email, + scope: SCOPES, + aud: TOKEN_URL, + iat: now, + exp: now + 3600, + }; + const body = `${base64Url(JSON.stringify({ alg: "RS256", typ: "JWT" }))}.${base64Url(JSON.stringify(claims))}`; + + const signature = await crypto.subtle.sign( + "RSASSA-PKCS1-v1_5", + key, + new TextEncoder().encode(body), + ); + + return `${body}.${base64Url(new Uint8Array(signature))}`; +} + +/** + * Exchanges the signed assertion for an Identity Toolkit access token. + * + * Against the emulator there is nothing to exchange with — Google's token + * endpoint is not part of it — so the run uses the `owner` bearer the emulator + * accepts, matching what `firebase-admin` does. + */ +export async function fetchAccessToken(account: ServiceAccount): Promise { + if (process.env.FIREBASE_AUTH_EMULATOR_HOST) return "owner"; + + const assertion = await signServiceAccountJwt(account); + + const response = await loggedFetch(new URL(TOKEN_URL), { + tag: "firebase", + method: "POST", + headers: { "Content-Type": "application/x-www-form-urlencoded" }, + body: new URLSearchParams({ + grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer", + assertion, + }).toString(), + }); + + const body = (await response.json().catch(() => ({}))) as { + access_token?: string; + error_description?: string; + error?: string; + }; + + const detail = body.error_description ?? body.error ?? "no access token returned"; + if (!response.ok && !isCredentialStatus(response.status)) { + throwApiFailure( + response.status, + `Google did not issue a token (${response.status}): ${detail}. Try again shortly.`, + DOCS_URL, + ); + } + if (!body.access_token) { + throwUsageError( + `Google rejected the service account (${response.status}): ${detail}\n` + + "Check the key has not been revoked or deleted, in the Google Cloud console under IAM → Service accounts.", + DOCS_URL, + ); + } + + return body.access_token; +} + +/** + * Base URL for Identity Toolkit. + * + * Honours `FIREBASE_AUTH_EMULATOR_HOST`, the variable Firebase's own tooling + * uses, so this works against the local emulator as well as production. + */ +function identityToolkitBase(): string { + const emulator = process.env.FIREBASE_AUTH_EMULATOR_HOST; + return emulator + ? `http://${emulator}/identitytoolkit.googleapis.com` + : "https://identitytoolkit.googleapis.com"; +} + +export type FirebaseUser = Record & { localId?: string }; + +/** Pages through every user in the project. */ +export async function fetchAllFirebaseUsers(options: { + account: ServiceAccount; + token: string; + spinner?: SpinnerControls; +}): Promise { + const all: FirebaseUser[] = []; + let pageToken: string | undefined; + + do { + const url = new URL( + `${identityToolkitBase()}/v1/projects/${options.account.project_id}/accounts:batchGet`, + ); + url.searchParams.set("maxResults", String(PAGE_SIZE)); + if (pageToken) url.searchParams.set("nextPageToken", pageToken); + + const response = await loggedFetch(url, { + tag: "firebase", + method: "GET", + headers: { Authorization: `Bearer ${options.token}`, Accept: "application/json" }, + }); + + if (!response.ok) { + throwApiFailure( + response.status, + `Firebase returned ${response.status} listing users: ${await response.text()}`, + DOCS_URL, + ); + } + + const body = (await response.json()) as { users?: FirebaseUser[]; nextPageToken?: string }; + all.push(...(body.users ?? [])); + options.spinner?.update(`Fetching users from Firebase: ${all.length} so far...`); + pageToken = body.nextPageToken; + } while (pageToken); + + return all; +} + +export type HashConfig = { + signerKey: string; + saltSeparator: string; + rounds: number; + memoryCost: number; +}; + +/** + * Reads the project's scrypt parameters. + * + * These are the whole reason a Firebase migration keeps its passwords: without + * them Clerk cannot verify a single digest. Fetching them here saves the user + * hunting through the console — and if the call is not permitted, the run says + * exactly where to look instead. + * + * @returns `null` when the config could not be read. + */ +export async function fetchHashConfig( + account: ServiceAccount, + token: string, +): Promise { + try { + const url = new URL(`${identityToolkitBase()}/admin/v2/projects/${account.project_id}/config`); + const response = await loggedFetch(url, { + tag: "firebase", + method: "GET", + headers: { Authorization: `Bearer ${token}`, Accept: "application/json" }, + }); + if (!response.ok) { + if (response.status === 403) { + log.warn( + "The service account cannot read the project's password hash parameters: it needs the " + + "`firebaseauth.configs.getHashConfig` permission. Grant it and export again, or pass the " + + "parameters to the import with --firebase-*.", + ); + } else { + log.debug(`firebase: ${response.status} reading the project config`); + } + return null; + } + + const body = (await response.json()) as { + signIn?: { hashConfig?: Partial & { algorithm?: string } }; + }; + const config = body.signIn?.hashConfig; + if (!config?.signerKey) return null; + + const hashConfig = { + signerKey: config.signerKey, + // Firebase omits an empty separator. + saltSeparator: config.saltSeparator ?? "", + rounds: Number(config.rounds ?? 8), + memoryCost: Number(config.memoryCost ?? 14), + }; + // Every digest the import builds carries these, so a set Clerk would + // refuse is left out rather than written: the import then asks for them. + const problem = firebaseHashConfigProblem({ + base64_signer_key: hashConfig.signerKey, + base64_salt_separator: hashConfig.saltSeparator, + rounds: hashConfig.rounds, + mem_cost: hashConfig.memoryCost, + }); + if (problem) { + log.warn( + `The project's password hash parameters won't work in Clerk (${problem}), so the export carries none. ` + + "Pass the right ones to the import with --firebase-*.", + ); + return null; + } + return hashConfig; + } catch (error) { + log.debug(`firebase: could not read the project config: ${String(error)}`); + return null; + } +} + +/** + * Keeps the fields the `firebase` transformer maps from. + * + * A Firebase user also carries provider records, custom claims and sign-in + * timestamps that would bloat the export and mean nothing to the import. + */ +export function mapFirebaseUserToExport(user: FirebaseUser): Record { + const exported: Record = {}; + + for (const field of ["localId", "email", "displayName", "phoneNumber", "createdAt"] as const) { + if (user[field]) exported[field] = user[field]; + } + // Meaningful when false, so copied on presence rather than truthiness. + if (user.emailVerified !== undefined) exported.emailVerified = user.emailVerified; + // Only when true: every active user would otherwise carry a `false`. + if (user.disabled === true) exported.disabled = true; + + // Both halves or neither: a digest without its salt cannot be verified. A + // redacted hash is no hash at all. + if (user.passwordHash && user.passwordHash !== REDACTED_HASH && user.salt) { + exported.passwordHash = user.passwordHash; + exported.salt = user.salt; + } + + return exported; +} + +/** + * What Firebase sends as `passwordHash` when the caller may not read hashes: + * base64 for "REDACTED". `firebase-admin` treats it as no hash. + */ +export const REDACTED_HASH = "UkVEQUNURUQ="; + +function hasPasswordProvider(user: FirebaseUser): boolean { + const providers = user.providerUserInfo; + return ( + Array.isArray(providers) && + providers.some((provider) => (provider as { providerId?: unknown })?.providerId === "password") + ); +} + +export function buildFirebaseExport( + users: FirebaseUser[], + record: (line: UserLine) => void = () => {}, +) { + const exported: Record[] = []; + const counts = { email: 0, verified: 0, password: 0, name: 0, phone: 0 }; + // Password users Firebase returned no hash for: it only returns digests it + // made itself (scrypt), and an empty string for users uploaded with bcrypt, + // HMAC or any other hasher. + let unreadablePasswords = 0; + // Hashes Firebase redacted because the caller may not read them. + let redactedPasswords = 0; + + for (const user of users) { + const userId = String(user.localId ?? ""); + try { + const mapped = mapFirebaseUserToExport(user); + exported.push(mapped); + + if (mapped.email) counts.email++; + if (mapped.emailVerified) counts.verified++; + if (mapped.passwordHash) counts.password++; + else if (user.passwordHash === REDACTED_HASH) redactedPasswords++; + else if (hasPasswordProvider(user)) unreadablePasswords++; + if (mapped.displayName) counts.name++; + if (mapped.phoneNumber) counts.phone++; + + record({ sourceId: userId, status: "exported" }); + } catch (error) { + record({ sourceId: userId, status: "skipped", error: (error as Error).message }); + } + } + + return { + users: exported, + unreadablePasswords, + redactedPasswords, + coverage: [ + { label: "have an email address", count: counts.email }, + { label: "have a verified email", count: counts.verified }, + { label: "have a password hash", count: counts.password }, + { label: "have a display name", count: counts.name }, + { label: "have a phone number", count: counts.phone }, + ], + }; +} + +/** + * What the reader needs to know about the hash parameters. + * + * When they were read, they are already in the export file, so the import + * needs nothing extra. When they were not, the import has to be given them. + */ +export function formatHashConfigGuidance( + config: HashConfig | null, + passwordCount: number, +): string[] { + if (passwordCount === 0) { + return [dim("No password hashes in this export, so no hash parameters are needed.")]; + } + + if (!config) { + return [ + bold("Password hash parameters"), + "This export carries password hashes, which Clerk can only verify with the project's", + "scrypt parameters. Find them in the Firebase console under", + "Authentication → Users → (⋮) → Password hash parameters, then pass them to the import:", + dim( + " --firebase-signer-key --firebase-salt-separator --firebase-rounds --firebase-mem-cost", + ), + ]; + } + + return [ + bold("Password hash parameters"), + dim("Read from the project and saved in the export file, so the import needs nothing more."), + ]; +} + +/** The hash parameters in the shape the import reads them. */ +export function toFirebaseHashConfig(config: HashConfig): FirebaseHashConfig { + return { + base64_signer_key: config.signerKey, + base64_salt_separator: config.saltSeparator, + rounds: config.rounds, + mem_cost: config.memoryCost, + }; +} + +export async function exportFirebase(options: ExportFirebaseOptions): Promise { + // Read and validate before anything reaches the network, so a wrong file + // fails in a second rather than after an auth round-trip. + const resolved = await resolveServiceAccount(options); + + await withGutter("Exporting users from Firebase", async () => { + // The emulator variable is Firebase's own, so it is honoured, but named: + // an export from it is not the production project's users. + const emulator = process.env.FIREBASE_AUTH_EMULATOR_HOST; + if (!options.json) { + printTarget({ platform: emulator ? `firebase (emulator at ${emulator})` : "firebase" }); + } + // Only Google can say whether a well-formed key is still a valid one, so a + // revoked or deleted key fails here and is asked for again. + const { value: token, input: account } = await withInputRetry( + resolved, + promptServiceAccount, + async (candidate) => { + log.info(`Exporting from the ${candidate.project_id} project.`); + return withSpinner("Authenticating with Google...", async () => + fetchAccessToken(candidate), + ); + }, + options, + ); + + const users = await withSpinner("Fetching users from Firebase...", async (spinner) => + fetchAllFirebaseUsers({ account, token, spinner }), + ); + + const run = await startExportRun(options, { platform: "firebase" }); + const { + users: exported, + coverage, + unreadablePasswords, + redactedPasswords, + } = buildFirebaseExport(users, run.append); + + // Read before the file is written, so the envelope carries them and the + // import needs no --firebase-* flags. + const passwordCount = coverage.find((entry) => entry.label.includes("password"))?.count ?? 0; + const hashConfig = passwordCount > 0 ? await fetchHashConfig(account, token) : null; + + finishExport({ + run, + options, + users: exported, + coverage, + ...(hashConfig ? { firebase: toFirebaseHashConfig(hashConfig) } : {}), + }); + + if (!options.json) { + log.blank(); + for (const line of formatHashConfigGuidance(hashConfig, passwordCount)) log.info(line); + } + + if (redactedPasswords > 0) { + log.warn( + `${redactedPasswords} user${redactedPasswords === 1 ? "'s" : "s'"} password hash came back redacted: ` + + "the service account can't read hashes. Grant it `firebaseauth.configs.getHashConfig` and export again, " + + "or those users are imported without a password and need to reset it.", + ); + } + + if (unreadablePasswords > 0) { + log.warn( + `${unreadablePasswords} user${unreadablePasswords === 1 ? " has" : "s have"} a password Firebase did not return. ` + + "Firebase only returns hashes it made itself; users imported into Firebase with bcrypt, HMAC or another hasher come back without one. " + + "They will be imported without a password and need to reset it.", + ); + } + }); +} diff --git a/packages/cli-core/src/commands/migrate/export/index.ts b/packages/cli-core/src/commands/migrate/export/index.ts index 944f5799d..ba4e163a0 100644 --- a/packages/cli-core/src/commands/migrate/export/index.ts +++ b/packages/cli-core/src/commands/migrate/export/index.ts @@ -3,6 +3,7 @@ import { throwUsageError } from "../../../lib/errors.ts"; import { select } from "../../../lib/listage.ts"; import { isAgent, isHuman } from "../../../mode.ts"; import { exportClerk } from "./clerk.ts"; +import { exportFirebase } from "./firebase.ts"; import { exportSupabase } from "./supabase.ts"; import type { DbExportOptions } from "./db-options.ts"; import { RUNS_DIR_DESCRIPTION, RUNS_DIR_FLAG } from "../lib/run-store.ts"; @@ -49,6 +50,7 @@ const handlers = { picker: exportPicker, clerk: exportClerk, supabase: exportSupabase, + firebase: exportFirebase, }; /** The platforms that read a database, which share `--db-url`. */ @@ -120,6 +122,24 @@ export function registerMigrateExport(migrateCommand: Command<[], Record[0]), ); + exportCommand + .command("firebase") + .description("Export users from a Firebase project") + .option("--service-account ", "Path to a service account key JSON file") + .option("-o, --output ", "Write the export here instead of the run folder") + .option("-y, --yes", "Do not prompt: fail on a rejected credential") + .option(RUNS_DIR_FLAG, RUNS_DIR_DESCRIPTION) + .option("--json", "Print the result as JSON; never prompts") + .setExamples([ + { + command: "clerk migrate export firebase --service-account ./service-account.json", + description: "Export using a downloaded service account key", + }, + ]) + .action(async (_opts, cmd) => + handlers.firebase(cmd.optsWithGlobals() as Parameters[0]), + ); + // Each takes exactly one connection string, so they are registered from a // table rather than near-identical blocks. for (const platform of DB_PLATFORMS) { diff --git a/packages/cli-core/src/commands/migrate/export/registry.test.ts b/packages/cli-core/src/commands/migrate/export/registry.test.ts index 3131db58e..4a92d606b 100644 --- a/packages/cli-core/src/commands/migrate/export/registry.test.ts +++ b/packages/cli-core/src/commands/migrate/export/registry.test.ts @@ -4,7 +4,7 @@ import { exportPlatformKeys, exportPlatforms, getExportPlatform } from "./regist describe("export registry", () => { test("registers every source platform", () => { - expect(exportPlatformKeys()).toEqual(["clerk", "supabase"]); + expect(exportPlatformKeys()).toEqual(["clerk", "supabase", "firebase"]); }); test.each([...exportPlatforms])("$key carries a label and description", (entry) => { diff --git a/packages/cli-core/src/commands/migrate/export/registry.ts b/packages/cli-core/src/commands/migrate/export/registry.ts index 57b4b6cd9..0ecb9108e 100644 --- a/packages/cli-core/src/commands/migrate/export/registry.ts +++ b/packages/cli-core/src/commands/migrate/export/registry.ts @@ -11,6 +11,7 @@ */ import { exportClerk } from "./clerk.ts"; +import { exportFirebase } from "./firebase.ts"; import { exportSupabase } from "./supabase.ts"; export type ExportRegistryEntry = { @@ -37,6 +38,13 @@ export const exportPlatforms: ExportRegistryEntry[] = [ sourceKey: "supabase", run: async (options) => exportSupabase(options), }, + { + key: "firebase", + label: "Firebase", + description: "A Firebase project, via Identity Toolkit", + sourceKey: "firebase", + run: async (options) => exportFirebase(options), + }, ]; export function exportPlatformKeys(): string[] { diff --git a/packages/cli-core/src/commands/migrate/index.test.ts b/packages/cli-core/src/commands/migrate/index.test.ts index 962624b7f..b80201419 100644 --- a/packages/cli-core/src/commands/migrate/index.test.ts +++ b/packages/cli-core/src/commands/migrate/index.test.ts @@ -54,6 +54,10 @@ describe("registerMigrate", () => { "--new-run", "--require-password", "--skip-legal-checks", + "--firebase-signer-key", + "--firebase-salt-separator", + "--firebase-rounds", + "--firebase-mem-cost", "--yes", "--json", "--secret-key", @@ -64,9 +68,12 @@ describe("registerMigrate", () => { }); test("registers an export subcommand per registered platform", () => { - expect(findCommand(["migrate", "export"])?.commands.map((cmd) => cmd.name())).toEqual( - exportPlatformKeys(), - ); + // Help order is not registry order, so compare as sets. + expect( + findCommand(["migrate", "export"]) + ?.commands.map((cmd) => cmd.name()) + .sort(), + ).toEqual(exportPlatformKeys().sort()); }); // Bare `migrate export` runs the picker rather than defaulting to a @@ -97,6 +104,12 @@ describe("registerMigrate", () => { ).toEqual(["--db-url", "--output", "--yes", "--runs-dir", "--json"]); }); + test("migrate export firebase takes exactly its flags", () => { + expect( + findCommand(["migrate", "export", "firebase"])?.options.map((option) => option.long), + ).toEqual(["--service-account", "--output", "--yes", "--runs-dir", "--json"]); + }); + test.each(exportPlatformKeys())( "migrate export %s names the run folder as the default output", (platform) => { diff --git a/packages/cli-core/src/commands/migrate/index.ts b/packages/cli-core/src/commands/migrate/index.ts index 98ffcb17d..de2b44b55 100644 --- a/packages/cli-core/src/commands/migrate/index.ts +++ b/packages/cli-core/src/commands/migrate/index.ts @@ -1,5 +1,6 @@ import type { Program } from "../../cli-program.ts"; import { isExperimentEnabled, requireExperiment } from "../../lib/experimental.ts"; +import { parseIntegerOption } from "../../lib/option-parsers.ts"; import { setMode } from "../../mode.ts"; import { setAssumeYes } from "./lib/assume-yes.ts"; import { registerMigrateExport } from "./export/index.ts"; @@ -91,6 +92,24 @@ export function registerMigrate(program: Program, env: NodeJS.ProcessEnv = proce "--skip-legal-checks", "Import users with no legal acceptance on record into an instance that requires it", ) + .option( + "--firebase-signer-key ", + "Firebase base64 signer key (overrides the export file), or CLERK_FIREBASE_SIGNER_KEY (does not)", + ) + .option( + "--firebase-salt-separator ", + "Firebase base64 salt separator, or CLERK_FIREBASE_SALT_SEPARATOR", + ) + .option( + "--firebase-rounds ", + "Firebase scrypt rounds, or CLERK_FIREBASE_ROUNDS", + (value: string) => parseIntegerOption(value, "--firebase-rounds", { min: 1 }), + ) + .option( + "--firebase-mem-cost ", + "Firebase scrypt memory cost, or CLERK_FIREBASE_MEM_COST", + (value: string) => parseIntegerOption(value, "--firebase-mem-cost", { min: 1 }), + ) .option("-y, --yes", "Import without prompting") .option("--json", "Output as JSON; never prompts, so pair it with --yes to import") .option("--secret-key ", "Backend API secret key to use") diff --git a/packages/cli-core/src/commands/migrate/lib/checks.test.ts b/packages/cli-core/src/commands/migrate/lib/checks.test.ts index 70539dc6e..9c8d697e4 100644 --- a/packages/cli-core/src/commands/migrate/lib/checks.test.ts +++ b/packages/cli-core/src/commands/migrate/lib/checks.test.ts @@ -1177,6 +1177,8 @@ describe("hashShapeProblem", () => { test.each([ [BCRYPT, "bcrypt"], ["hash$salt$signer$sep$8$14", "scrypt_firebase"], + // A project with no salt separator. + ["hash$salt$signer$$8$14", "scrypt_firebase"], ["$argon2id$v=19$m=65536,t=3,p=4$c2FsdA$aGFzaA", "argon2id"], [`scrypt:16384:16:1$${"a".repeat(32)}$${"b".repeat(128)}`, "scrypt_werkzeug"], // Not checked: a bad digest there fails only at sign-in. @@ -1191,6 +1193,10 @@ describe("hashShapeProblem", () => { ["$2a$03$N9qo8uLOickgx2ZMRZoMyeIjZAgcfl7p92ldGxad68LJZdL17lhWy", "bcrypt"], ["hash$salt$signer$sep$eight$14", "scrypt_firebase"], ["hash$salt", "scrypt_firebase"], + ["hash$$signer$sep$8$14", "scrypt_firebase"], + // Clerk decodes every segment as base64. + ["not base64!$salt$signer$sep$8$14", "scrypt_firebase"], + ["hash$sa=lt$signer$sep$8$14", "scrypt_firebase"], ["argon2id$...", "argon2id"], ["scrypt:16384:16:1$salt$not-hex!", "scrypt_werkzeug"], ])("rejects %s as %s", (password, hasher) => { diff --git a/packages/cli-core/src/commands/migrate/lib/checks.ts b/packages/cli-core/src/commands/migrate/lib/checks.ts index e80ad51a4..9f3c1c551 100644 --- a/packages/cli-core/src/commands/migrate/lib/checks.ts +++ b/packages/cli-core/src/commands/migrate/lib/checks.ts @@ -25,6 +25,7 @@ import { isEnabled, isRequired, type AttributeName } from "../../users/interacti import { splitIdentifiers } from "../import-users.ts"; import type { User } from "../types.ts"; import { analyzeFields, hasValue } from "./analysis.ts"; +import { decodesLikeClerk } from "./firebase-hash.ts"; import { clerkOffersProvider, enabledSocialProviders, @@ -145,7 +146,10 @@ export function hashShapeProblem(password: string, hasher: string): string | und const parts = password.split("$"); const numeric = (value: string | undefined) => /^\d+$/.test(value ?? ""); return parts.length === 6 && - parts.slice(0, 4).every(Boolean) && + // The salt separator (parts[3]) may be empty. + parts.slice(0, 3).every(Boolean) && + // Clerk decodes all four as base64 (`Validate` in pkg/hash/scrypt.go). + parts.slice(0, 4).every(decodesLikeClerk) && numeric(parts[4]) && numeric(parts[5]) ? undefined diff --git a/packages/cli-core/src/commands/migrate/lib/firebase-hash.test.ts b/packages/cli-core/src/commands/migrate/lib/firebase-hash.test.ts new file mode 100644 index 000000000..10e585414 --- /dev/null +++ b/packages/cli-core/src/commands/migrate/lib/firebase-hash.test.ts @@ -0,0 +1,172 @@ +import { describe, expect, test } from "bun:test"; +import { + fingerprintFirebaseHashConfig, + firebaseHashConfigProblem, + resolveFirebaseHashConfig, +} from "./firebase-hash.ts"; + +const ALL_FLAGS = { + firebaseSignerKey: "SIGNER", + firebaseSaltSeparator: "Bw==", + firebaseRounds: 8, + firebaseMemCost: 14, +}; + +describe("gating on the source", () => { + test.each([["clerk"], ["supabase"], ["auth0"], ["authjs"], ["betterauth"]])( + "ignores even explicit flags for the %s source", + (transformer) => { + expect(resolveFirebaseHashConfig(ALL_FLAGS, transformer)).toBeUndefined(); + }, + ); + + test("resolves nothing before the platform is known", () => { + expect(resolveFirebaseHashConfig(ALL_FLAGS, undefined)).toBeUndefined(); + }); +}); + +describe("on a firebase run", () => { + test("builds the config from flags", () => { + expect(resolveFirebaseHashConfig(ALL_FLAGS, "firebase")).toEqual({ + base64_signer_key: "SIGNER", + base64_salt_separator: "Bw==", + rounds: 8, + mem_cost: 14, + }); + }); + + // A digest built from a partial set is well-formed but verifies against + // nothing, so every migrated user would silently fail to sign in. + test.each([ + ["firebaseSignerKey", "--firebase-signer-key"], + ["firebaseSaltSeparator", "--firebase-salt-separator"], + ["firebaseRounds", "--firebase-rounds"], + ["firebaseMemCost", "--firebase-mem-cost"], + ] as const)("rejects a flag set missing %s, naming it", (omit, flag) => { + const partial = { ...ALL_FLAGS }; + delete (partial as Record)[omit]; + + expect(() => resolveFirebaseHashConfig(partial, "firebase")).toThrow(new RegExp(flag)); + }); + + test("names every missing flag at once", () => { + expect(() => resolveFirebaseHashConfig({ firebaseSignerKey: "SIGNER" }, "firebase")).toThrow( + /--firebase-salt-separator.*--firebase-rounds.*--firebase-mem-cost/, + ); + }); + + test("returns nothing when no flag supplies a config", () => { + expect(resolveFirebaseHashConfig({}, "firebase", {})).toBeUndefined(); + }); +}); + +// The variables keep the signer key out of argv, shell history and `ps`. +describe("from CLERK_FIREBASE_* variables", () => { + const ENV = { + CLERK_FIREBASE_SIGNER_KEY: "SIGNER", + CLERK_FIREBASE_SALT_SEPARATOR: "Bw==", + CLERK_FIREBASE_ROUNDS: "8", + CLERK_FIREBASE_MEM_COST: "14", + }; + + test("builds the config from all four", () => { + expect(resolveFirebaseHashConfig({}, "firebase", ENV)).toEqual({ + base64_signer_key: "SIGNER", + base64_salt_separator: "Bw==", + rounds: 8, + mem_cost: 14, + }); + }); + + test("a flag wins over its variable", () => { + expect( + resolveFirebaseHashConfig({ firebaseSignerKey: "ROTATED" }, "firebase", ENV) + ?.base64_signer_key, + ).toBe("ROTATED"); + }); + + test("a partial set names each missing flag and variable", () => { + expect(() => + resolveFirebaseHashConfig({}, "firebase", { CLERK_FIREBASE_SIGNER_KEY: "SIGNER" }), + ).toThrow(/--firebase-rounds \(or CLERK_FIREBASE_ROUNDS\)/); + }); + + test("refuses a cost that is not a whole number", () => { + expect(() => + resolveFirebaseHashConfig({}, "firebase", { ...ENV, CLERK_FIREBASE_ROUNDS: "eight" }), + ).toThrow(/CLERK_FIREBASE_ROUNDS must be a positive whole number/); + }); +}); + +// Clerk's bounds: clerk_go pkg/hash/scrypt.go. +describe("firebaseHashConfigProblem", () => { + const good = { + base64_signer_key: "SIGNER", + base64_salt_separator: "Bw==", + rounds: 8, + mem_cost: 14, + }; + + test("passes Firebase's usual parameters", () => { + expect(firebaseHashConfigProblem(good)).toBeUndefined(); + }); + + // Clerk pads a short final group itself, and reads the URL-safe alphabet. + test.each(["Bw", "Bw==", "a-b_", "YWJjZA"])("passes %s, which Clerk decodes", (separator) => { + expect( + firebaseHashConfigProblem({ ...good, base64_salt_separator: separator }), + ).toBeUndefined(); + }); + + // Clerk decodes it to no bytes; some projects have no separator. + test("passes an empty salt separator", () => { + expect(firebaseHashConfigProblem({ ...good, base64_salt_separator: "" })).toBeUndefined(); + }); + + test.each([ + // Clerk pads to a multiple of four, then decodes strictly. + ["a one-character key", { base64_signer_key: "A" }, /signer key is not base64/], + ["padding after a full group", { base64_signer_key: "AAAA=" }, /signer key is not base64/], + ["two pads after a full group", { base64_signer_key: "AAAA==" }, /signer key is not base64/], + ["an empty key", { base64_signer_key: "" }, /signer key is not base64/], + [ + "a signer key that is not base64", + { base64_signer_key: "not base64!" }, + /signer key is not base64/, + ], + [ + "a separator with the digest's $", + { base64_salt_separator: "Bw$" }, + /salt separator is not base64/, + ], + ["rounds of 0", { rounds: 0 }, /rounds must be a whole number from 1 to 16, not 0/], + ["rounds of 17", { rounds: 17 }, /rounds must be a whole number from 1 to 16, not 17/], + ["a fractional memory cost", { mem_cost: 14.5 }, /memory cost must be a whole number/], + ])("names %s", (_label, change, expected) => { + expect(firebaseHashConfigProblem({ ...good, ...change })).toMatch(expected); + }); +}); + +describe("fingerprintFirebaseHashConfig", () => { + const config = { + base64_signer_key: "SIGNER", + base64_salt_separator: "Bw==", + rounds: 8, + mem_cost: 14, + }; + + test("is the same for the same parameters, and holds no signer key", () => { + const fingerprint = fingerprintFirebaseHashConfig(config); + expect(fingerprintFirebaseHashConfig({ ...config })).toBe(fingerprint); + expect(fingerprint).toMatch(/^[0-9a-f]{64}$/); + expect(fingerprint).not.toContain("SIGNER"); + }); + + test("changes with any parameter", () => { + const fingerprint = fingerprintFirebaseHashConfig(config); + expect(fingerprintFirebaseHashConfig({ ...config, base64_signer_key: "OTHER" })).not.toBe( + fingerprint, + ); + expect(fingerprintFirebaseHashConfig({ ...config, rounds: 9 })).not.toBe(fingerprint); + }); +}); diff --git a/packages/cli-core/src/commands/migrate/lib/firebase-hash.ts b/packages/cli-core/src/commands/migrate/lib/firebase-hash.ts new file mode 100644 index 000000000..1772de072 --- /dev/null +++ b/packages/cli-core/src/commands/migrate/lib/firebase-hash.ts @@ -0,0 +1,149 @@ +/** + * Firebase's four scrypt parameters, read from the `--firebase-*` flags or the + * matching `CLERK_FIREBASE_*` environment variables. + * + * **Only read when the source is `firebase`.** `migrate import` is one + * command serving every platform, and nothing but the Firebase source + * reads the config off {@link TransformContext}. + */ + +import { scryptSync } from "node:crypto"; +import { throwUsageError } from "../../../lib/errors.ts"; +import type { FirebaseHashConfig } from "../types.ts"; + +/** The `--firebase-*` flags, keyed by their option name, with their environment variables. */ +export const FIREBASE_FLAGS = [ + ["firebaseSignerKey", "--firebase-signer-key", "CLERK_FIREBASE_SIGNER_KEY"], + ["firebaseSaltSeparator", "--firebase-salt-separator", "CLERK_FIREBASE_SALT_SEPARATOR"], + ["firebaseRounds", "--firebase-rounds", "CLERK_FIREBASE_ROUNDS"], + ["firebaseMemCost", "--firebase-mem-cost", "CLERK_FIREBASE_MEM_COST"], +] as const; + +export type FirebaseHashFlags = { + firebaseSignerKey?: string; + firebaseSaltSeparator?: string; + firebaseRounds?: number; + firebaseMemCost?: number; +}; + +/** + * Resolves the four parameters from the flags, each falling back to its + * environment variable. The variables keep the signer key out of argv, and so + * out of shell history and `ps`. + * + * The four are required as a set: a digest built from a partial set is + * well-formed but verifies against nothing, so every migrated user would fail + * to sign in with no error at import time. + * + * @param source - The platform being migrated. Anything but `firebase` + * returns immediately. + * @returns The config, or `undefined` when none was supplied — which is fine + * for an export that carries no password hashes. + */ +export function resolveFirebaseHashConfig( + options: FirebaseHashFlags, + source: string | undefined, + env: Record = process.env, +): FirebaseHashConfig | undefined { + if (source !== "firebase") return undefined; + + const flags: FirebaseHashFlags = { ...options }; + for (const [key, flag, envVar] of FIREBASE_FLAGS) { + const raw = env[envVar]; + if (flags[key] !== undefined || raw === undefined) continue; + if (key === "firebaseRounds" || key === "firebaseMemCost") { + const value = Number(raw.trim()); + if (!Number.isInteger(value) || value < 1) { + throwUsageError(`${envVar} must be a positive whole number, like ${flag}. Got "${raw}".`); + } + flags[key] = value; + } else { + flags[key] = raw; + } + } + + const provided = FIREBASE_FLAGS.filter(([key]) => flags[key] !== undefined); + if (provided.length === 0) return undefined; + + if (provided.length < FIREBASE_FLAGS.length) { + const missing = FIREBASE_FLAGS.filter(([key]) => flags[key] === undefined).map( + ([, flag, envVar]) => `${flag} (or ${envVar})`, + ); + throwUsageError( + `The Firebase hash parameters must be supplied together. Missing: ${missing.join(", ")}.\n` + + "Find all four in the Firebase console under Authentication → Users → (⋮) → Password hash parameters.", + "https://clerk.com/docs/guides/development/migrating/firebase", + ); + } + + return { + base64_signer_key: flags.firebaseSignerKey as string, + base64_salt_separator: flags.firebaseSaltSeparator as string, + rounds: flags.firebaseRounds as number, + mem_cost: flags.firebaseMemCost as number, + }; +} + +/** Clerk's bounds on Firebase scrypt costs (clerk_go `pkg/hash/scrypt.go`). */ +export const MAX_SCRYPT_COST = 16; + +/** One whole base64 string: full groups of four, padded only at the end. */ +const STRICT_BASE64 = /^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/; + +/** + * True when Clerk can decode it: it maps the URL-safe alphabet onto the + * standard one and pads to a multiple of four (`normalizeBase64` in clerk_go + * `pkg/hash/scrypt.go`), then decodes strictly. So `Bw` passes and `A` or + * `AAAA=` do not. + */ +export function decodesLikeClerk(value: string): boolean { + let normalized = value.replace(/-/g, "+").replace(/_/g, "/"); + const rem = normalized.length % 4; + if (rem !== 0) normalized += "=".repeat(4 - rem); + return STRICT_BASE64.test(normalized); +} + +/** + * What is wrong with a set of Firebase hash parameters, or `undefined`. + * + * Each one goes into every `scrypt_firebase` digest, so a bad one costs every + * password in the import: Clerk refuses a cost outside 1..16 and a key that is + * not base64, and a `$` would break the digest's segments. + */ +export function firebaseHashConfigProblem(config: FirebaseHashConfig): string | undefined { + // An empty salt separator is a valid one: Clerk decodes it to no bytes. + const keys = [ + ["signer key", config.base64_signer_key, false], + ["salt separator", config.base64_salt_separator, true], + ] as const; + for (const [label, value, mayBeEmpty] of keys) { + if (typeof value !== "string" || (!value && !mayBeEmpty) || !decodesLikeClerk(value)) { + return `the ${label} is not base64`; + } + } + const costs = [ + ["rounds", config.rounds], + ["memory cost", config.mem_cost], + ] as const; + for (const [label, value] of costs) { + if (!Number.isInteger(value) || value < 1 || value > MAX_SCRYPT_COST) { + return `${label} must be a whole number from 1 to ${MAX_SCRYPT_COST}, not ${String(value)}`; + } + } + return undefined; +} + +/** + * A fingerprint of the hash parameters, for a run record: two runs built + * their digests alike exactly when these match. Derived with scrypt, so the + * signer key is not written to disk, nor a fast hash of it. + */ +export function fingerprintFirebaseHashConfig(config: FirebaseHashConfig): string { + const { base64_signer_key, base64_salt_separator, rounds, mem_cost } = config; + // A fixed salt: the fingerprint has to come out the same on every run. + return scryptSync( + JSON.stringify([base64_signer_key, base64_salt_separator, rounds, mem_cost]), + "clerk-migrate-firebase-hash-config", + 32, + ).toString("hex"); +} diff --git a/packages/cli-core/src/commands/migrate/lib/run-store.ts b/packages/cli-core/src/commands/migrate/lib/run-store.ts index 69780dfd1..1ca85c585 100644 --- a/packages/cli-core/src/commands/migrate/lib/run-store.ts +++ b/packages/cli-core/src/commands/migrate/lib/run-store.ts @@ -89,6 +89,12 @@ export type RunRecord = { * continue for those. */ flags?: ImportFlags; + /** + * A Firebase import's hash parameters, as SHA-256 of the four (never the + * parameters themselves): every digest it builds carries them, so a continue + * with others is refused. + */ + firebaseHash?: string; /** The import run an undo reverses. */ undoes?: string; /** The undo run that reversed this one. */ diff --git a/packages/cli-core/src/commands/migrate/lib/transform.ts b/packages/cli-core/src/commands/migrate/lib/transform.ts index 385a4891a..c638899a1 100644 --- a/packages/cli-core/src/commands/migrate/lib/transform.ts +++ b/packages/cli-core/src/commands/migrate/lib/transform.ts @@ -457,7 +457,7 @@ async function readUsersFromFile( } // A pre-transform's rows win over the file, CSV or not. - if (preExtracted) return preExtracted; + if (preExtracted) return assertUserRows(preExtracted, file); if (type === "text/csv") return readCsv(filePath, csvHeaders); return assertUserRows(readJsonFile(filePath), file); diff --git a/packages/cli-core/src/commands/migrate/run-interactive.test.ts b/packages/cli-core/src/commands/migrate/run-interactive.test.ts index 786c02cb7..34b985a6a 100644 --- a/packages/cli-core/src/commands/migrate/run-interactive.test.ts +++ b/packages/cli-core/src/commands/migrate/run-interactive.test.ts @@ -26,6 +26,7 @@ import { listageStubs, useCaptureLog } from "../../test/lib/stubs.ts"; const mockSelect = mock(async () => "clerk" as unknown); const mockText = mock(async () => "export.json" as unknown); +const mockPassword = mock(async () => "" as unknown); let confirmAnswer = true; /** User settings the instance serves, or none (the settings read fails). */ let instanceSettings: Record | undefined; @@ -47,7 +48,7 @@ mock.module("../../lib/prompts.ts", () => ({ }, multiselect: async () => [], text: (...args: unknown[]) => mockText(...(args as [])), - password: async () => "", + password: (...args: unknown[]) => mockPassword(...(args as [])), editor: async () => "{}", })); @@ -104,6 +105,7 @@ beforeEach(() => { confirmMessages = []; mockSelect.mockReset(); mockText.mockReset(); + mockPassword.mockClear(); mockSelect.mockResolvedValue("clerk"); mockText.mockResolvedValue("export.json"); fs.rmSync(path.join(workDir, ".clerk"), { recursive: true, force: true }); @@ -276,6 +278,64 @@ describe("consent", () => { }); }); +describe("firebase hash parameters", () => { + // `-y` is "import without prompting", at a terminal too. + test("-y never asks for them, and names the flags instead", async () => { + fs.writeFileSync( + path.join(workDir, "export.json"), + JSON.stringify({ + users: [{ localId: "fb1", email: "a@x.dev", passwordHash: "SGFzaA==", salt: "U2FsdA==" }], + }), + ); + + await expect(run({ ...importOptions, source: "firebase", yes: true })).rejects.toThrow( + /--firebase-signer-key --firebase-salt-separator --firebase-rounds --firebase-mem-cost/, + ); + expect(mockPassword).not.toHaveBeenCalled(); + expect(mockText).not.toHaveBeenCalled(); + expect(created()).toHaveLength(0); + }); + + // The suggested retry runs with --yes, which never asks again. + test("parameters entered at the prompt go into the retry the import suggests", async () => { + fs.writeFileSync( + path.join(workDir, "export.json"), + JSON.stringify({ + users: [ + { localId: "fb1", email: "a@x.dev", passwordHash: "SGFzaA==", salt: "U2FsdA==" }, + { localId: "fb2", email: "b@x.dev", passwordHash: "SGFzaA==", salt: "U2FsdA==" }, + ], + }), + ); + mockPassword.mockResolvedValueOnce("c2lnbmVy"); + mockText.mockImplementation((async (prompt: { message: string }) => + prompt.message.includes("salt separator") + ? "c2VwYXJhdG9y" + : prompt.message === "rounds" + ? "8" + : "14") as unknown as typeof mockText); + + // fb2's create fails, so the run ends partial and suggests the retry. + const clerk = globalThis.fetch; + globalThis.fetch = (async (input: string | URL | Request, init?: RequestInit) => { + const body = typeof init?.body === "string" ? init.body : ""; + return init?.method === "POST" && body.includes('"fb2"') + ? Response.json( + { errors: [{ message: "taken", code: "form_identifier_exists" }] }, + { status: 422 }, + ) + : clerk(input, init); + }) as typeof fetch; + await run({ ...importOptions, source: "firebase" }); + + const err = Bun.stripANSI(captured.err); + expect(err).toContain( + "--firebase-signer-key SIGNER_KEY --firebase-salt-separator SALT_SEPARATOR --firebase-rounds ROUNDS --firebase-mem-cost MEM_COST", + ); + expect(err).not.toContain("c2lnbmVy"); + }); +}); + describe("legal consent", () => { beforeEach(() => { instanceSettings = { diff --git a/packages/cli-core/src/commands/migrate/run.test.ts b/packages/cli-core/src/commands/migrate/run.test.ts index 17af33d43..73ecb0cc7 100644 --- a/packages/cli-core/src/commands/migrate/run.test.ts +++ b/packages/cli-core/src/commands/migrate/run.test.ts @@ -308,7 +308,7 @@ describe("run", () => { describe("export envelopes", () => { /** An export run whose envelope holds `users`, as `clerk migrate export` writes it. */ - function exportRun(source: string, rows: unknown[]) { + function exportRun(source: string, rows: unknown[], extra: Record = {}) { const run = startRun(runsDir(), { kind: "export", target: { platform: source }, source }); const file = path.join(run.dir, "export.json"); fs.writeFileSync( @@ -319,6 +319,7 @@ describe("run", () => { exportedAt: "2026-09-01T00:00:00.000Z", runId: run.record.id, users: rows, + ...extra, }), ); run.update({ file: { path: file, sha256: sha256File(file) } }); @@ -610,6 +611,146 @@ describe("run", () => { ); expect(requests).toHaveLength(0); }); + + test("reads Firebase's hash parameters from the envelope", async () => { + const firebase = { + base64_signer_key: "SIGNER", + base64_salt_separator: "Bw==", + rounds: 8, + mem_cost: 14, + }; + const { record } = exportRun( + "firebase", + [{ localId: "f1", email: "f@x.dev", passwordHash: "HASH", salt: "SALT" }], + { firebase }, + ); + + await run({ ...noSource, input: record.id }); + + const created = requests.find((r) => r.url.endsWith("/v1/users")); + expect(created?.body).toMatchObject({ + password_hasher: "scrypt_firebase", + password_digest: "HASH$SALT$SIGNER$Bw==$8$14", + }); + }); + + // Each goes into every digest: a bad one would cost every password. + test("refuses Firebase hash parameters Clerk would reject, from the envelope or the flags", async () => { + const { record } = exportRun( + "firebase", + [{ localId: "f1", email: "f@x.dev", passwordHash: "HASH", salt: "SALT" }], + { + firebase: { + base64_signer_key: "SIGNER", + base64_salt_separator: "Bw==", + rounds: 0, + mem_cost: 14, + }, + }, + ); + + await expect(run({ ...noSource, input: record.id })).rejects.toThrow( + /from the export file won't work: rounds must be a whole number from 1 to 16/, + ); + await expect( + run({ + ...noSource, + input: record.id, + firebaseSignerKey: "SIGNER", + firebaseSaltSeparator: "Bw==", + firebaseRounds: 8, + firebaseMemCost: 17, + }), + ).rejects.toThrow(/from the --firebase-\* flags won't work: memory cost/); + expect(requests.filter((r) => r.url.endsWith("/v1/users"))).toHaveLength(0); + }); + + test("lets the --firebase-* flags override the envelope", async () => { + const { record } = exportRun( + "firebase", + [{ localId: "f1", email: "f@x.dev", passwordHash: "HASH", salt: "SALT" }], + { + firebase: { + base64_signer_key: "OLD", + base64_salt_separator: "Bw==", + rounds: 8, + mem_cost: 14, + }, + }, + ); + + await run({ + ...noSource, + input: record.id, + firebaseSignerKey: "NEW", + firebaseSaltSeparator: "Bw==", + firebaseRounds: 8, + firebaseMemCost: 14, + }); + + const created = requests.find((r) => r.url.endsWith("/v1/users")); + expect((created!.body as { password_digest: string }).password_digest).toContain("$NEW$"); + }); + + // Variables left over from another project would hash every password wrongly. + test("keeps the envelope's parameters over the CLERK_FIREBASE_* variables", async () => { + const { record } = exportRun( + "firebase", + [{ localId: "f1", email: "f@x.dev", passwordHash: "HASH", salt: "SALT" }], + { + firebase: { + base64_signer_key: "FILE", + base64_salt_separator: "Bw==", + rounds: 8, + mem_cost: 14, + }, + }, + ); + const env = { + CLERK_FIREBASE_SIGNER_KEY: "STALE", + CLERK_FIREBASE_SALT_SEPARATOR: "Bw==", + CLERK_FIREBASE_ROUNDS: "8", + CLERK_FIREBASE_MEM_COST: "14", + }; + Object.assign(process.env, env); + try { + await run({ ...noSource, input: record.id }); + } finally { + for (const key of Object.keys(env)) delete process.env[key]; + } + + const created = requests.find((r) => r.url.endsWith("/v1/users")); + expect((created!.body as { password_digest: string }).password_digest).toContain("$FILE$"); + }); + + test("does not complete a partial flag set from the variables when the envelope has parameters", async () => { + const { record } = exportRun( + "firebase", + [{ localId: "f1", email: "f@x.dev", passwordHash: "HASH", salt: "SALT" }], + { + firebase: { + base64_signer_key: "FILE", + base64_salt_separator: "Bw==", + rounds: 8, + mem_cost: 14, + }, + }, + ); + const env = { + CLERK_FIREBASE_SALT_SEPARATOR: "Bw==", + CLERK_FIREBASE_ROUNDS: "8", + CLERK_FIREBASE_MEM_COST: "14", + }; + Object.assign(process.env, env); + try { + await expect( + run({ ...noSource, input: record.id, firebaseSignerKey: "NEW" }), + ).rejects.toThrow(/must be supplied together/); + } finally { + for (const key of Object.keys(env)) delete process.env[key]; + } + expect(requests.filter((r) => r.url.endsWith("/v1/users"))).toHaveLength(0); + }); }); test("gitignores the project's .clerk folder before writing a run", async () => { @@ -1893,12 +2034,118 @@ describe("run", () => { expect(error.message).toContain('Unknown source "nope"'); }); + test("firebase imports its wrapped export and builds the scrypt digest", async () => { + fs.writeFileSync( + path.join(workDir, "export.json"), + JSON.stringify({ + users: [ + { + localId: "fb1", + email: "a@x.dev", + emailVerified: true, + passwordHash: "SGFzaA==", + salt: "U2FsdA==", + }, + ], + }), + ); + + await run({ + ...baseOptions, + source: "firebase", + firebaseSignerKey: "SIGNER", + firebaseSaltSeparator: "Bw==", + firebaseRounds: 8, + firebaseMemCost: 14, + }); + + const body = requests.find((r) => r.url.endsWith("/v1/users"))?.body as Record< + string, + unknown + >; + expect(body).toMatchObject({ + external_id: "fb1", + password_digest: "SGFzaA==$U2FsdA==$SIGNER$Bw==$8$14", + password_hasher: "scrypt_firebase", + }); + }); + + // The users a run created carry its parameters; the rest must match them. + test("a firebase run continues only with the hash parameters it started with", async () => { + fs.writeFileSync( + path.join(workDir, "export.json"), + JSON.stringify({ + users: ["fb1", "fb2"].map((localId) => ({ + localId, + email: `${localId}@x.dev`, + emailVerified: true, + passwordHash: "SGFzaA==", + salt: "U2FsdA==", + })), + }), + ); + const flags = { + ...baseOptions, + source: "firebase", + firebaseSignerKey: "SIGNER", + firebaseSaltSeparator: "Bw==", + firebaseRounds: 8, + firebaseMemCost: 14, + }; + stubClerk({ failing: new Set(["fb2"]) }); + await run(flags); + expect(readRun(runsDir(), listRuns(runsDir())[0]!.id)?.firebaseHash).toMatch( + /^[0-9a-f]{64}$/, + ); + + requests = []; + process.exitCode = 0; + stubClerk(); + await expect(run({ ...flags, firebaseRounds: 9 })).rejects.toThrow( + /different Firebase hash parameters/, + ); + expect(created()).toEqual([]); + + await run(flags); + expect(created()).toEqual(["fb2"]); + }); + + test("the printed command keeps the --firebase-* flags, as placeholders", async () => { + fs.writeFileSync( + path.join(workDir, "export.json"), + JSON.stringify({ users: [{ localId: "fb1", email: "a@x.dev", emailVerified: true }] }), + ); + + const error = (await run({ + ...baseOptions, + yes: false, + source: "firebase", + firebaseSignerKey: "SIGNER", + firebaseSaltSeparator: "Bw==", + firebaseRounds: 8, + firebaseMemCost: 14, + }).catch((caught: unknown) => caught)) as CliError; + + expect(error.examples?.[0]?.command).toContain( + "--firebase-signer-key SIGNER_KEY --firebase-salt-separator SALT_SEPARATOR --firebase-rounds ROUNDS --firebase-mem-cost MEM_COST", + ); + }); + + test("a partial firebase flag set fails before anything is read", async () => { + await expect( + run({ ...baseOptions, source: "firebase", firebaseSignerKey: "SIGNER" }), + ).rejects.toThrow(/--firebase-salt-separator/); + expect(requests).toHaveLength(0); + }); + test("an unknown source fails listing the valid keys", async () => { const error = (await run({ ...baseOptions, source: "nope" }).catch( (caught: unknown) => caught, )) as CliError; expect(error.exitCode).toBe(EXIT_CODE.USAGE); - expect(error.message).toContain('Unknown source "nope". Valid sources: clerk, supabase.'); + expect(error.message).toContain( + 'Unknown source "nope". Valid sources: clerk, firebase, supabase.', + ); expect(requests).toHaveLength(0); }); }); diff --git a/packages/cli-core/src/commands/migrate/run.ts b/packages/cli-core/src/commands/migrate/run.ts index 6297d958e..acfc6ca23 100644 --- a/packages/cli-core/src/commands/migrate/run.ts +++ b/packages/cli-core/src/commands/migrate/run.ts @@ -49,6 +49,13 @@ import { importUsers } from "./import-users.ts"; import { ALREADY_IMPORTED, checkImport, type ImportChecks } from "./lib/checks.ts"; import { fetchInstanceSettings, fetchUserCount } from "./lib/clerk-config.ts"; import { readEnvelope, type ExportEnvelope } from "./lib/export-file.ts"; +import { + FIREBASE_FLAGS, + firebaseHashConfigProblem, + fingerprintFirebaseHashConfig, + resolveFirebaseHashConfig, + type FirebaseHashFlags, +} from "./lib/firebase-hash.ts"; import { DEV_USER_LIMIT, resolveLimits, type InstanceType } from "./lib/instance.ts"; import { continueRun, @@ -96,7 +103,7 @@ import { } from "./lib/transform.ts"; import { resolveSource, sourceKeys } from "./sources/registry.ts"; import type { ImportSummary, User } from "./types.ts"; -import { promptForFile, promptForSource } from "./wizard.ts"; +import { promptForFile, promptForFirebaseHashConfig, promptForSource } from "./wizard.ts"; import { login } from "../auth/login.ts"; import { link } from "../link/index.ts"; @@ -126,7 +133,7 @@ export type MigrateRunOptions = { instance?: string; /** Where runs are kept; overrides `CLERK_MIGRATE_DIR`. */ runsDir?: string; -}; +} & FirebaseHashFlags; /** * Validates what a run needs before anything is read or sent. @@ -651,6 +658,12 @@ function commandFor( if (!fromExport && options.source) parts.push("--source", quoteArg(options.source)); if (options.newRun) parts.push("--new-run"); parts.push(...importFlags(options)); + // Names, not `<…>`: pasted as is, a shell reads `` as a redirect. + if (options.firebaseSignerKey) parts.push("--firebase-signer-key", "SIGNER_KEY"); + if (options.firebaseSaltSeparator !== undefined) + parts.push("--firebase-salt-separator", "SALT_SEPARATOR"); + if (options.firebaseRounds) parts.push("--firebase-rounds", "ROUNDS"); + if (options.firebaseMemCost) parts.push("--firebase-mem-cost", "MEM_COST"); parts.push(...carry); if (options.json) parts.push("--json"); return [...parts, ...extra].join(" "); @@ -784,6 +797,47 @@ export async function run(rawOptions: MigrateRunOptions): Promise { // The flags every suggested command needs to reach this runs folder and instance. const carry = carryFlags(options); + // A flag is typed for this run, so it wins over the export file. A variable + // is ambient, and may be left over from another project, so it only fills in + // for an export file without the parameters: with one, a partial flag set is + // refused rather than completed from the variables. + const fromEnvelope = source === "firebase" ? envelope?.firebase : undefined; + const fromFlagsAtAll = FIREBASE_FLAGS.some(([key]) => options[key] !== undefined); + const fromFlags = + fromFlagsAtAll || !fromEnvelope + ? resolveFirebaseHashConfig(options, source, fromEnvelope ? {} : process.env) + : undefined; + let firebaseHashConfig = fromFlags ?? fromEnvelope; + let configFrom = fromFlags + ? fromFlagsAtAll + ? "the --firebase-* flags" + : "the CLERK_FIREBASE_* variables" + : "the export file"; + // `-y` is "import without prompting": it gets the usage error naming the flags. + if (source === "firebase" && !firebaseHashConfig && !options.yes && canPrompt(options)) { + firebaseHashConfig = await promptForFirebaseHashConfig(); + configFrom = "the parameters entered"; + // A suggested command runs with --yes, which never asks: it names the flags. + if (firebaseHashConfig) { + options = { + ...options, + firebaseSignerKey: firebaseHashConfig.base64_signer_key, + firebaseSaltSeparator: firebaseHashConfig.base64_salt_separator, + firebaseRounds: firebaseHashConfig.rounds, + firebaseMemCost: firebaseHashConfig.mem_cost, + }; + } + } + // Wherever they came from, each goes into every password digest. + const problem = firebaseHashConfig && firebaseHashConfigProblem(firebaseHashConfig); + if (problem) { + throwUsageError( + `The Firebase hash parameters from ${configFrom} won't work: ${problem}. Nothing was imported.\n` + + "Find all four in the Firebase console under Authentication → Users → (⋮) → Password hash parameters.", + "https://clerk.com/docs/guides/development/migrating/firebase", + ); + } + await withGutter( "Migrating users to Clerk", async ({ setNextSteps }) => { @@ -813,6 +867,23 @@ export async function run(rawOptions: MigrateRunOptions): Promise { ? { kind: "new" } : findResume(runsDir, match, carry); + // The users this run created carry digests built from its parameters, so + // the rest must be built from the same ones. A run from before this was + // recorded has none to compare, and continues as it did. + const firebaseHash = firebaseHashConfig + ? fingerprintFirebaseHashConfig(firebaseHashConfig) + : undefined; + if ( + resume.kind === "continue" && + resume.record.firebaseHash && + resume.record.firebaseHash !== firebaseHash + ) { + throwUsageError( + `Run ${resume.record.id} imported this file with different Firebase hash parameters, and the users it created carry digests built from them. Nothing was imported.\n` + + "Pass the same --firebase-* flags to continue it, or --new-run to start a new run.", + ); + } + if (resume.kind === "complete") { // Imported by path and named now by its export run ID: the record // links the export, so the cleanup names the folder holding the data. @@ -882,7 +953,7 @@ export async function run(rawOptions: MigrateRunOptions): Promise { } const loaded = await withSpinner(`Loading users from ${file}...`, async () => - loadUsersFromFile(file, source), + loadUsersFromFile(file, source, { context: { firebaseHashConfig } }), ); let users = loaded.users.filter((user) => !done.has(user.userId)); const failures = loaded.failures.filter((failure) => !done.has(failure.userId)); @@ -1149,6 +1220,7 @@ export async function run(rawOptions: MigrateRunOptions): Promise { source, file: { path: filePath, sha256 }, ...(input.fromExport ? { fromExport: input.fromExport } : {}), + ...(firebaseHash ? { firebaseHash } : {}), flags, }); // Printed now, not on the way out: on a Ctrl-C the signal handler exits diff --git a/packages/cli-core/src/commands/migrate/sources/firebase.ts b/packages/cli-core/src/commands/migrate/sources/firebase.ts new file mode 100644 index 000000000..7bacec418 --- /dev/null +++ b/packages/cli-core/src/commands/migrate/sources/firebase.ts @@ -0,0 +1,125 @@ +import { CliError, ERROR_CODE, throwUsageError } from "../../../lib/errors.ts"; +import { readJsonFile } from "../lib/export-file.ts"; +import type { PreTransformResult, SourceEntry } from "../types.ts"; +import { isVerified, routeByVerification, splitName, toIsoDate } from "./shared.ts"; + +/** + * Column order of `firebase auth:export --format=csv`, which writes no header + * row. Without these the CSV parser would treat the first user as the header. + */ +const FIREBASE_CSV_HEADERS = + "localId,email,emailVerified,passwordHash,passwordSalt,displayName,photoUrl," + + "googleId,googleEmail,googleDisplayName,googlePhotoUrl," + + "facebookId,facebookEmail,facebookDisplayName,facebookPhotoUrl," + + "twitterId,twitterEmail,twitterDisplayName,twitterPhotoUrl," + + "githubId,githubEmail,githubDisplayName,githubPhotoUrl," + + "createdAt,lastSignedInAt,phoneNumber,disabled,customAttributes,providerUserInfo"; + +/** + * Firebase → Clerk transformer. + * + * Handles both shapes `firebase auth:export` produces: a headerless CSV, and + * JSON wrapped in `{ users: [...] }`. + * + * Firebase's scrypt is a modified variant, so Clerk needs the project's four + * hash parameters alongside each digest. They arrive on the run's + * {@link TransformContext} from the `--firebase-*` flags, the export file, the + * matching `CLERK_FIREBASE_*` environment variables, or a prompt. The + * export file keeps them, signer key included, owner-only in its run folder. + * + * See https://clerk.com/docs/guides/development/migrating/firebase + */ +const firebaseSource = { + key: "firebase", + label: "Firebase", + description: + "Works with `firebase auth:export` (CSV or JSON). Requires the project's four password hash parameters to migrate passwords.", + + preTransform: (filePath: string, fileType: string): PreTransformResult => { + // The CSV has no header row; name the columns rather than writing a copy + // with one, which would leave the hashes and salts in a temp file. + if (fileType === "text/csv") return { filePath, csvHeaders: FIREBASE_CSV_HEADERS.split(",") }; + + if (fileType === "application/json") { + const parsed = readJsonFile(filePath); + if (Array.isArray(parsed)) return { filePath, data: parsed as Record[] }; + + const users = (parsed as { users?: unknown })?.users; + if (Array.isArray(users)) return { filePath, data: users as Record[] }; + + throw new CliError( + "Invalid Firebase JSON export: expected `{ users: [...] }` or an array of users.", + { code: ERROR_CODE.INVALID_JSON }, + ); + } + + return { filePath }; + }, + + carries: { + passwords: { + level: "yes", + note: "scrypt hashes come across with the project's hash parameters, read by the export or passed as `--firebase-*`.", + }, + mfa: { level: "no", note: "Firebase exports no MFA enrolments. Users enrol again in Clerk." }, + metadata: { level: "no", note: "Custom claims are not exported." }, + }, + transformer: { + localId: "userId", + email: "email", + emailVerified: "emailVerified", + passwordHash: "passwordHash", + passwordSalt: "salt", + phoneNumber: "phone", + displayName: "name", + disabled: "banned", + }, + + postTransform: (user, context) => { + const passwordHash = user.passwordHash; + const salt = user.salt; + + if (passwordHash && salt) { + const config = context.firebaseHashConfig; + if (!config) { + throwUsageError( + "This export contains Firebase password hashes, which need the project's hash parameters to import.\n" + + "Find them in the Firebase console under Authentication → Users → (⋮) → Password hash parameters, then pass:\n" + + " --firebase-signer-key --firebase-salt-separator --firebase-rounds --firebase-mem-cost\n" + + "or set CLERK_FIREBASE_SIGNER_KEY, CLERK_FIREBASE_SALT_SEPARATOR, CLERK_FIREBASE_ROUNDS and CLERK_FIREBASE_MEM_COST.", + "https://clerk.com/docs/guides/development/migrating/firebase", + ); + } + + // Clerk's scrypt_firebase hasher expects every parameter inline: + // hash$salt$signerKey$saltSeparator$rounds$memCost + user.password = [ + passwordHash, + salt, + config.base64_signer_key, + config.base64_salt_separator, + config.rounds, + config.mem_cost, + ].join("$"); + + delete user.passwordHash; + delete user.salt; + } + + routeByVerification(user, "email", "emailVerified", "boolean"); + // Firebase exports timestamps as Unix milliseconds, often as strings. + user.createdAt = toIsoDate(user.createdAt, true); + splitName(user); + + // A disabled Firebase account is Clerk's banned. Runs before + // normalizeUserData, so accept CSV's "true" as well. + if (isVerified(user.banned, "boolean")) user.banned = true; + else delete user.banned; + }, + + defaults: { + passwordHasher: "scrypt_firebase" as const, + }, +} satisfies SourceEntry; + +export default firebaseSource; diff --git a/packages/cli-core/src/commands/migrate/sources/registry.ts b/packages/cli-core/src/commands/migrate/sources/registry.ts index abf898004..f6f14f758 100644 --- a/packages/cli-core/src/commands/migrate/sources/registry.ts +++ b/packages/cli-core/src/commands/migrate/sources/registry.ts @@ -10,9 +10,10 @@ import { throwUsageError } from "../../../lib/errors.ts"; import type { SourceEntry } from "../types.ts"; import clerkSource from "./clerk.ts"; +import firebaseSource from "./firebase.ts"; import supabaseSource from "./supabase.ts"; -export const sources: SourceEntry[] = [clerkSource, supabaseSource]; +export const sources: SourceEntry[] = [clerkSource, firebaseSource, supabaseSource]; export const ACCOUNT_LINKING_URL = "https://clerk.com/docs/guides/configure/auth-strategies/social-connections/account-linking"; diff --git a/packages/cli-core/src/commands/migrate/sources/sources.test.ts b/packages/cli-core/src/commands/migrate/sources/sources.test.ts index 5a5881207..b0ac09a54 100644 --- a/packages/cli-core/src/commands/migrate/sources/sources.test.ts +++ b/packages/cli-core/src/commands/migrate/sources/sources.test.ts @@ -1,11 +1,20 @@ -import { afterAll, beforeAll, describe, expect, test } from "bun:test"; +import { afterAll, beforeAll, describe, expect, spyOn, test } from "bun:test"; import fs from "node:fs"; import os from "node:os"; import path from "node:path"; +import { CliError } from "../../../lib/errors.ts"; import { loadUsersFromFile, transformUsers } from "../lib/transform.ts"; +import type { FirebaseHashConfig } from "../types.ts"; import { getSource, sourceKeys, sources } from "./registry.ts"; import { isVerified } from "./shared.ts"; +const FIREBASE_HASH: FirebaseHashConfig = { + base64_signer_key: "SIGNERKEY==", + base64_salt_separator: "Bw==", + rounds: 8, + mem_cost: 14, +}; + let workDir: string; let originalCwd: string; @@ -37,7 +46,7 @@ const one = (key: string, record: Record, context = {}) => describe("registry", () => { test("registers the built-in platforms", () => { - expect(sourceKeys()).toEqual(["clerk", "supabase"]); + expect(sourceKeys()).toEqual(["clerk", "firebase", "supabase"]); }); test.each([...sources])("$key maps a source field to userId", (source) => { @@ -104,6 +113,91 @@ describe("isVerified", () => { }); }); +describe("firebase", () => { + const base = { localId: "fb1", email: "a@x.dev", emailVerified: true }; + + test.each([ + [true, true], + ["true", true], + [false, undefined], + [undefined, undefined], + ])("disabled=%p carries as banned=%p", (disabled, expected) => { + expect(one("firebase", { ...base, disabled })?.banned).toBe(expected as boolean | undefined); + }); + const withHash = { ...base, passwordHash: "SGFzaA==", salt: "U2FsdA==" }; + + test("builds the scrypt digest Clerk expects, parameters inline", async () => { + const { users } = await load("firebase", { users: [withHash] }, "json", { + firebaseHashConfig: FIREBASE_HASH, + }); + expect(users[0]?.password).toBe("SGFzaA==$U2FsdA==$SIGNERKEY==$Bw==$8$14"); + expect(users[0]?.passwordHasher).toBe("scrypt_firebase"); + }); + + test("refuses to import hashes without the project's hash parameters", async () => { + await expect(load("firebase", { users: [withHash] })).rejects.toThrow( + /Firebase password hashes/, + ); + }); + + test("imports a passwordless export with no hash parameters at all", async () => { + const { users } = await load("firebase", { users: [base] }); + expect(users).toHaveLength(1); + expect(users[0]?.password).toBeUndefined(); + }); + + test("unwraps the { users: [...] } export shape", async () => { + const { users } = await load("firebase", { users: [base, { ...base, localId: "fb2" }] }); + expect(users.map((u) => u.userId)).toEqual(["fb1", "fb2"]); + }); + + test("accepts a bare array too", async () => { + const { users } = await load("firebase", [base]); + expect(users).toHaveLength(1); + }); + + test("rejects a JSON export that is neither", async () => { + await expect(load("firebase", { records: [] })).rejects.toThrow(CliError); + }); + + test("rejects a wrapped row that is not a user object", async () => { + await expect(load("firebase", { users: [base, null] })).rejects.toThrow( + /row 2 is not a user object/, + ); + }); + + // Named, not prepended: a copy with a header row would leave the hashes and + // salts in a temp file nobody deletes. + test("names the columns of a headerless CSV export, writing no copy", async () => { + const csv = "fb9,a@x.dev,true,,,Ada Lovelace,,,,,,,,,,,,,,,,,,1704067200000,,,,,\n"; + const mkdtemp = spyOn(fs, "mkdtempSync"); + try { + const { users } = await load("firebase", csv, "csv"); + expect(users[0]).toMatchObject({ userId: "fb9", email: "a@x.dev", firstName: "Ada" }); + expect(mkdtemp).not.toHaveBeenCalled(); + } finally { + mkdtemp.mockRestore(); + } + }); + + test.each([ + ["1704067200000", "2024-01-01T00:00:00.000Z"], + [1704067200000, "2024-01-01T00:00:00.000Z"], + ])("converts the Unix-millisecond createdAt %p", (createdAt, expected) => { + expect(one("firebase", { ...base, createdAt })?.createdAt).toBe(expected); + }); + + test.each([ + [true, true], + ["true", true], + [false, false], + ["false", false], + ])("emailVerified=%p keeps the address primary: %p", (emailVerified, verified) => { + const user = one("firebase", { ...base, emailVerified }); + expect(user?.email !== undefined).toBe(verified); + }); +}); + describe("supabase", () => { // What `export supabase` hands over: first_name only when the metadata has // one, the rest left in the metadata for this source to split. @@ -437,7 +531,10 @@ describe("fields the CLI's own export adds", () => { }); describe("invalid records", () => { - const INVALID: [string, Record][] = [["supabase", { id: "a5" }]]; + const INVALID: [string, Record][] = [ + ["firebase", { localId: "a4" }], + ["supabase", { id: "a5" }], + ]; test.each(INVALID)( "%s reports a user with no identifier instead of crashing", @@ -456,6 +553,7 @@ describe("invalid records", () => { }); /** The per-platform source field that becomes a Clerk identifier. */ -function identifierFor(_key: string, email = "ok@x.dev"): Record { +function identifierFor(key: string, email = "ok@x.dev"): Record { + if (key === "firebase") return { email, emailVerified: true }; return { email, email_confirmed_at: "2024-01-01 00:00:00+00" }; } diff --git a/packages/cli-core/src/commands/migrate/wizard.test.ts b/packages/cli-core/src/commands/migrate/wizard.test.ts index 387d42d79..c29372f99 100644 --- a/packages/cli-core/src/commands/migrate/wizard.test.ts +++ b/packages/cli-core/src/commands/migrate/wizard.test.ts @@ -13,6 +13,7 @@ type SelectPrompt = Prompt & { choices: { name: string; value: string }[] }; // This file is the only consumer of the mocked prompt modules. const mockSelect = mock(async (_config: SelectPrompt) => undefined as unknown); const mockText = mock(async (_config: Prompt) => "" as unknown); +const mockPassword = mock(async (_config: Prompt) => "" as unknown); mock.module("../../lib/listage.ts", () => ({ ...listageStubs, @@ -22,11 +23,11 @@ mock.module("../../lib/listage.ts", () => ({ mock.module("../../lib/prompts.ts", () => ({ confirm: async () => true, text: (config: Prompt) => mockText(config), - password: async () => "", + password: (config: Prompt) => mockPassword(config), editor: async () => "{}", })); -const { promptForFile, promptForSource } = await import("./wizard.ts"); +const { promptForFile, promptForFirebaseHashConfig, promptForSource } = await import("./wizard.ts"); let workDir: string; let originalCwd: string; @@ -48,6 +49,7 @@ afterAll(() => { beforeEach(() => { mockSelect.mockReset(); mockText.mockReset(); + mockPassword.mockReset(); }); /** The config object passed to the Nth `text`/`select` prompt. */ @@ -59,7 +61,11 @@ describe("promptForSource", () => { mockSelect.mockResolvedValue("supabase"); expect(await promptForSource()).toBe("supabase"); - expect(selectCall(0)?.choices.map((choice) => choice.value)).toEqual(["clerk", "supabase"]); + expect(selectCall(0)?.choices.map((choice) => choice.value)).toEqual([ + "clerk", + "firebase", + "supabase", + ]); }); test("labels each choice with the source's display name", async () => { @@ -96,3 +102,40 @@ describe("promptForFile", () => { expect(check(value)).toMatch(message); }); }); + +describe("promptForFirebaseHashConfig", () => { + test("collects all four parameters as a set, the signer key masked", async () => { + mockPassword.mockResolvedValueOnce("SIGNER"); + mockText.mockResolvedValueOnce("Bw==").mockResolvedValueOnce("8").mockResolvedValueOnce("14"); + + expect(await promptForFirebaseHashConfig()).toEqual({ + base64_signer_key: "SIGNER", + base64_salt_separator: "Bw==", + rounds: 8, + mem_cost: 14, + }); + }); + + // An export with no password hashes needs none of them. + test("stops when the signer key is left blank", async () => { + mockPassword.mockResolvedValueOnce(""); + + expect(await promptForFirebaseHashConfig()).toBeUndefined(); + expect(mockText).toHaveBeenCalledTimes(0); + }); + + // Clerk refuses a cost above 16: the prompt says so, not the run afterwards. + test.each([ + ["17", "from 1 to 16"], + ["0", "from 1 to 16"], + ["16", undefined], + ])("a cost of %s is checked at its prompt", async (value, message) => { + mockPassword.mockResolvedValueOnce("SIGNER"); + mockText.mockResolvedValueOnce("Bw==").mockResolvedValueOnce("8").mockResolvedValueOnce("14"); + await promptForFirebaseHashConfig(); + const rounds = mockText.mock.calls.find(([config]) => config.message === "rounds")?.[0]; + const result = rounds?.validate?.(value); + if (message) expect(result).toContain(message); + else expect(result).toBeUndefined(); + }); +}); diff --git a/packages/cli-core/src/commands/migrate/wizard.ts b/packages/cli-core/src/commands/migrate/wizard.ts index bb0fbd70a..3d7249590 100644 --- a/packages/cli-core/src/commands/migrate/wizard.ts +++ b/packages/cli-core/src/commands/migrate/wizard.ts @@ -1,17 +1,21 @@ /** * The prompts behind an interactive `clerk migrate import`. * - * Each one fills in exactly one thing the command was not given: the file, or - * the source. + * Each one fills in exactly one thing the command was not given: the file, + * the source when the file does not name its own, and Firebase's hash + * parameters when neither the flags nor the export carry them. * * Nothing here runs for an agent, a non-TTY run or `--json`: `run` raises a * usage error naming what to pass instead. */ import { select } from "../../lib/listage.ts"; -import { text } from "../../lib/prompts.ts"; +import { log } from "../../lib/log.ts"; +import { password, text } from "../../lib/prompts.ts"; +import { MAX_SCRYPT_COST } from "./lib/firebase-hash.ts"; import { fileExists, getFileType } from "./lib/transform.ts"; import { sources } from "./sources/registry.ts"; +import type { FirebaseHashConfig } from "./types.ts"; /** Trims a description down to a single readable hint line. */ function hint(description: string): string { @@ -45,3 +49,52 @@ export async function promptForFile(): Promise { }); return answer.trim(); } + +async function askNumber(label: string): Promise { + const answer = await text({ + message: label, + validate: (value) => { + // Clerk refuses a cost above 16: caught here, not after all four answers. + const parsed = Number(value?.trim()); + return Number.isInteger(parsed) && parsed >= 1 && parsed <= MAX_SCRYPT_COST + ? undefined + : `Enter a whole number from 1 to ${MAX_SCRYPT_COST}`; + }, + }); + return Number(answer.trim()); +} + +/** + * Collects Firebase's four hash parameters. + * + * Asked as a set because a partial set produces a digest that verifies against + * nothing. Pressing enter at the first leaves the config unset, which is + * correct for an export with no password hashes. + */ +export async function promptForFirebaseHashConfig(): Promise { + log.info( + "Firebase password hashes need the project's hash parameters. Find them in the Firebase console under Authentication → Users → (⋮) → Password hash parameters.", + ); + log.info("Pass the four --firebase-* flags to skip these prompts on the next run."); + + // Masked: with the exported hashes and salts, the signer key is what lets + // anyone verify, or crack, the passwords offline. + const signerKey = ( + await password({ + message: "base64 signer key (leave blank if this export has no passwords)", + }) + ).trim(); + if (!signerKey) return undefined; + + // Blank is a valid separator: some projects have none. + const saltSeparator = ( + await text({ message: "base64 salt separator (leave blank if the project has none)" }) + ).trim(); + + return { + base64_signer_key: signerKey, + base64_salt_separator: saltSeparator, + rounds: await askNumber("rounds"), + mem_cost: await askNumber("mem cost"), + }; +} diff --git a/packages/cli-core/src/test/integration/completion.test.ts b/packages/cli-core/src/test/integration/completion.test.ts index 3c148aee4..886f339b2 100644 --- a/packages/cli-core/src/test/integration/completion.test.ts +++ b/packages/cli-core/src/test/integration/completion.test.ts @@ -230,7 +230,7 @@ describe("generateCompletions", () => { "--source", "", ]).completions.map((c) => c.name); - expect(names).toEqual(["clerk", "supabase"]); + expect(names).toEqual(["clerk", "firebase", "supabase"]); } finally { if (saved === undefined) delete process.env.CLERK_EXPERIMENTAL; else process.env.CLERK_EXPERIMENTAL = saved; diff --git a/packages/cli-core/src/test/preload.ts b/packages/cli-core/src/test/preload.ts index bc5b5a2dd..4621ac3db 100644 --- a/packages/cli-core/src/test/preload.ts +++ b/packages/cli-core/src/test/preload.ts @@ -13,3 +13,14 @@ import { MCP_CLIENT_ENV_VARS } from "./lib/mcp-client-env.ts"; for (const key of MCP_CLIENT_ENV_VARS) delete process.env[key]; + +// `clerk migrate import --source firebase` falls back to these when a flag is +// missing, so a developer's own would leak into tests that pass only some. +for (const key of [ + "CLERK_FIREBASE_SIGNER_KEY", + "CLERK_FIREBASE_SALT_SEPARATOR", + "CLERK_FIREBASE_ROUNDS", + "CLERK_FIREBASE_MEM_COST", +]) { + delete process.env[key]; +}