Add manual production database refresh for development

.github/workflows/copy-production-db-to-dev.yml

@@ -0,0 +1,54 @@
+name: Copy production database to dev
+
+on:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  refresh-database:
+    name: Replace dev data with a production snapshot
+    if: vars.XEVENTS_DATABASE_REFRESH_ENABLED == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    # This environment is restricted to main. It has its own private credentials.
+    environment:
+      name: development-refresh
+    concurrency:
+      group: xevents-development
+      cancel-in-progress: false
+    steps:
+      - name: Require the trusted workflow branch
+        env:
+          SELECTED_REF: ${{ github.ref }}
+        run: |
+          if [ "$SELECTED_REF" != 'refs/heads/main' ]; then
+            echo 'Run this database-only workflow from main.' >&2
+            exit 1
+          fi
+      - name: Check out the database-copy tools
+        uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
+        with:
+          ref: ${{ github.sha }}
+          persist-credentials: false
+      - name: Read the current development migrations
+        uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
+        with:
+          ref: dev
+          path: .wrangler/development-code
+          persist-credentials: false
+      - name: Set up Node.js
+        uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
+        with:
+          node-version: 22
+          cache: npm
+      - name: Install database tools
+        run: npm ci
+      - name: Copy production data into development only
+        env:
+          CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
+          CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
+          WRANGLER_CONFIG_JSON: ${{ secrets.WRANGLER_CONFIG_JSON }}
+          PRODUCTION_CONFIG_JSON: ${{ secrets.PRODUCTION_CONFIG_JSON }}
+        run: node scripts/copy-production-database.mjs

AGENTS.md

@@ -65,9 +65,14 @@ existing invitations, and users' private event data when making changes.
 - Work on `dev` and verify the affected flows there before promoting to `main`.
   `main` deploys production; `dev` deploys the separate development Worker and D1.
   Keep all bot credentials, app URLs and database bindings isolated by environment.
-- The production-to-dev workflow merges code without force-pushing. Optional D1
-  copying is schema-only into an empty development database. Never copy real
-  guest records, media references, payment state or delivery queues into tests.
+- Keep code synchronization and database refresh separate. The user explicitly
+  authorizes the manual production-to-dev database snapshot workflow to copy real
+  business records into the private development D1. Replace development data,
+  preserve its settings/whitelist, clear sessions and delivery/coordination state,
+  and require the development snapshot access/payment/notification safeguards.
+  Never merge code, deploy Workers, write production data or publish database
+  exports as part of this database-only workflow. Automated tests still use
+  fictional local fixtures, never actual production snapshots.
 - Run deployment automation from one canonical repository; mirrors run checks
   only. Store private configuration in GitHub environment secrets and ignored files.
 

docs/development.md

@@ -37,7 +37,7 @@ Create GitHub environments named `production` and `development`. Configure these
 | `WRANGLER_CONFIG_JSON` | The complete private Wrangler configuration for that environment, as valid JSON. Use different Worker names, D1 IDs, bot usernames and app URLs. |
 | `PRODUCTION_CONFIG_JSON` | Development only: the private production configuration used to reject production resource collisions. It is a workflow reference, never the development Worker's runtime configuration. |
 
-`PRODUCTION_CONFIG_JSON` must match the actual production target. Also create the `development-refresh` environment with the same four secrets as development; restrict this environment to `main`. It is used only for the optional schema-copy job. Production needs only the first three secrets. Keep private configuration in GitHub secrets, never repository variables, checked-in files or workflow text.
+`PRODUCTION_CONFIG_JSON` must match the actual production target. Also create the `development-refresh` environment with the same four secrets as development; restrict this environment to `main`. It is used for manual database refreshes. Production needs only the first three secrets. Keep private configuration in GitHub secrets, never repository variables, checked-in files or workflow text.
 
 The deployment helper, `scripts/deploy-environment.mjs`, applies migrations to the selected environment and deploys with `--keep-vars`. The private configuration is materialized for that run and must not become a build artifact or committed file. Keep existing production bindings and secrets intact.
 
@@ -59,7 +59,8 @@ Keep the real development configuration in an ignored local file, such as `wrang
   "preview_urls": false,
   "vars": {
     "BOT_USERNAME": "YOUR_DEVELOPMENT_BOT_USERNAME",
-    "APP_URL": "https://YOUR_DEVELOPMENT_WORKER.YOUR_SUBDOMAIN.workers.dev/app"
+    "APP_URL": "https://YOUR_DEVELOPMENT_WORKER.YOUR_SUBDOMAIN.workers.dev/app",
+    "TEST_WHITELIST_ENABLED": "false"
   },
   "assets": {
     "directory": "./public",
@@ -90,22 +91,60 @@ Store these secrets on the **development Cloudflare Worker**, using private inte
 - `TELEGRAM_BOT_TOKEN`: the development bot's token.
 - `TELEGRAM_WEBHOOK_SECRET`: a new random secret for the development webhook.
 - `SUPER_ADMIN_ID`: your administrator Telegram identity, if needed.
+- `WHITELIST_USER_IDS`: optional initial allowed Telegram IDs for development, separated by commas or whitespace. Leave it blank to permit everyone.
 
 These bot secrets do not belong in GitHub source or `WRANGLER_CONFIG_JSON`, and the deployment workflow does not need copies of them in GitHub. Set `vars.BOT_USERNAME` to the development bot and `vars.APP_URL` to the development Mini App URL. The [self-hosting guide](self-hosting.md#3-set-worker-secrets) explains interactive secret setup and webhook registration.
 
 Do not connect production's bot to the development Worker. Telegram uses a single webhook per bot, so doing that would redirect production updates. A separate bot is also necessary for authentication and media isolation: Telegram [`file_id` values are specific to each bot](https://core.telegram.org/bots/api#sending-files).
 
 Stars payments use Telegram's real Bot API endpoints in the current application. A separate regular bot does not create a payment sandbox. Keep payment/refund verification mocked unless deliberately performing a controlled real transaction; never reuse production charge IDs or payment records in development.
 
-## Sync production to dev
+## Restrict development access
+
+Set `vars.TEST_WHITELIST_ENABLED` to `"true"` in the private development configuration to enable the test whitelist. The deployment helper sets `APP_ENV` to `development` and defaults this flag to `"false"` when omitted. Production always ignores the whitelist, and the deployment helper writes `"false"` for production even if its input enables the flag. For local testing, set `APP_ENV=development` and `TEST_WHITELIST_ENABLED=true` in the ignored `.dev.vars` file.
+
+An empty whitelist permits everyone. Once it contains IDs, the development bot and authenticated Mini App APIs allow only those users and the superadmin configured by the `SUPER_ADMIN_ID` Worker secret. The superadmin always retains access so they can manage the list.
+
+Send these commands to the development bot as the configured superadmin:
+
+| Command | Effect |
+| --- | --- |
+| `/whitelist` | Show the current access setting and allowed IDs. |
+| `/whitelist add ID...` | Add one or more numeric Telegram user IDs. |
+| `/whitelist remove ID...` | Remove one or more IDs. |
+| `/whitelist clear` | Empty the list and permit everyone. |
+
+Keep real IDs in private Worker secrets or bot commands. `WHITELIST_USER_IDS` supplies the initial list and is never serialized into generated deployment configuration. The first command that changes the list saves an override in that environment's D1 database; subsequent requests and commands use that override instead of the secret seed. `/whitelist clear` saves an empty override, so it keeps access open even if the seed secret still contains IDs. Development and production use separate D1 databases, so these settings do not cross environments.
+
+Removing the final tester also opens access to everyone; every change confirms whether access is restricted or open. Invalid configuration restricts access until the superadmin resets it. Payment and refund confirmations already issued by Telegram are still recorded after a tester is removed. Existing queued notifications are preserved.
+
+This restriction applies to the Cloudflare Worker, including its webhook and Mini App APIs. Use `wrangler dev` for local access-control testing; the standalone `npm start` polling process does not enforce this hosted feature. The whitelist command is advertised only in the configured superadmin's private chat.
+
+## Copy production data to dev
+
+Open **Actions → Copy production database to dev → Run workflow**, select `main`, and run it whenever you need the latest production records in development. This replaces the existing development database contents. It does not merge either branch, deploy a Worker, change bot credentials, or write to production.
+
+Enable this action only in the canonical repository by setting the repository variable `XEVENTS_DATABASE_REFRESH_ENABLED=true`. Leave it unset or `false` in mirrors. This switch is independent of `XEVENTS_DEPLOY_ENABLED`. The `development-refresh` environment needs all four secrets listed above, including its own `CLOUDFLARE_API_TOKEN` with D1 export and import access. A missing token must be configured before the action can run.
+
+The action exports production, prepares the snapshot privately, and imports one replacement SQL file into the verified development D1. It copies events, users, profiles, guest responses, invitation links, ticket/check-in records, payment audits and media references. Development settings, including the test whitelist, are retained; production settings are not copied. Sessions, queued deliveries, processed-update markers and coordination locks are cleared. Already-applied migrations are preserved, and pending migrations from the current `dev` checkout are applied offline before import. An incompatible migration set stops the refresh before writing development data.
+
+The development Worker must already include the database-refresh safeguards; the action checks its health capability before exporting. A copied database is restricted to the superadmin and explicitly allowed testers, even if the normal whitelist switch is disabled. An empty whitelist means **admin only** for a production snapshot; `/whitelist clear` cannot expose it to everyone. Add testers with `/whitelist add ID...` as usual. Copied reminders are paused, outbound messages are limited to these permitted accounts, and Stars invoices, checkout and refunds are disabled while the snapshot marker is present. Stored financial audit records remain intact.
+
+Telegram media references are copied, but their `file_id` values belong to the production bot. Some existing images or files will need to be uploaded again through the development bot. Never use the production bot token to work around this isolation.
+
+Exports and prepared SQL files are temporary, private files deleted after the run. They are never printed or uploaded as artifacts. D1 export can briefly block production database requests; import can briefly block development requests. A failed D1 file import rolls back the replacement. Both development deployment and database refresh use the same concurrency group. See Cloudflare's [D1 import and export guide](https://developers.cloudflare.com/d1/best-practices/import-export-data/).
+
+Automated tests continue to use fictional local fixtures and mocked Telegram calls. Production snapshots are private operational data, never test fixtures or public artifacts.
+
+## Sync production code to dev
 
 Run **Sync production code to development** manually from GitHub Actions on `main`. It merges production code into `dev` without resetting the branch or force-pushing. It preserves development work and stops if there is a merge conflict. Resolve that conflict in a normal checkout and push the reviewed result. After the merge, it explicitly starts the development deployment workflow because a push using GitHub's workflow token does not start another push workflow. That development run skips automated tests.
 
 The `copy_database_schema` input defaults to `false`. With that default, the workflow syncs code only and leaves the development database unchanged.
 
 Select `copy_database_schema` only when the development database is empty and you want to initialize it from production's schema. `scripts/refresh-development.mjs` verifies the separate targets and refuses a nonempty development database, preserving existing test data. It exports **schema only**, imports it into development, and applies reviewed `main` migrations; the subsequent development deployment applies any additional `dev` migrations. Schema export can briefly block production database requests, so use it deliberately. Cloudflare documents the supported [D1 schema export and import](https://developers.cloudflare.com/d1/best-practices/import-export-data/).
 
-Only schema and applied migration filenames are transferred, so already-applied schema changes are not replayed. This workflow never copies production users, events, invitations, sessions, payment or refund records, queued notifications, reminders, configuration rows or media references. It does not promote development data to production. Create fictional test events in the development bot after initialization. A production database backup is private operational data and must never be uploaded as a public workflow artifact or committed as a test fixture.
+Only schema and applied migration filenames are transferred by that optional initialization step, so already-applied schema changes are not replayed. For actual records, use the separate **Copy production database to dev** action above. Neither workflow promotes development data to production.
 
 ## Before promoting a change