Repository navigation
feat(db): native Postgres foundation (database package, schema, tests) — #251 - #252
Conversation
src/database gives connect() (one transaction per block, dict rows, UTC, lock and statement timeouts), transaction() with isolation, read-only and an advisory lock, advisory_lock() and serializable() with retry. Nothing uses it yet; the SQLite connect() stays until each area moves over (#251). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…able Numbered SQL files run in order under an advisory lock, each in its own transaction, at start-up when DATABASE_URL is set and from python -m src.database.migrate. 0001_baseline.sql covers the 61 tables of schema.sql plus _upgrade(): identity ids, timestamptz, boolean, jsonb, COLLATE "C" on every text column, and admin_audit refusing UPDATE, DELETE and TRUNCATE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tests that take the postgres fixture run against TEST_DATABASE_URL in a database the session creates and migrates once; every table is emptied before each test. Build gets a pgvector/pgvector:pg17 service and runs for the postgres branch too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…ately A returned connection kept session settings, temp tables, LISTENs, session advisory locks and autocommit for the next borrower; the pool now resets each one (DISCARD ALL's parts, minus DEALLOCATE ALL, which psycopg 3.3 stops noticing after a connection's first reset). Connections pin search_path to public. advisory_lock() waits up to 20 s, not the 5 s row lock bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runner stops when an applied version has no file or a pending file is numbered below the newest applied one. The patreon_sync_state row moves from the baseline to seeds.sql, inserted at start-up with ON CONFLICT DO NOTHING, so the copy from SQLite starts from empty tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… tables The parity test also compares column defaults, CHECK expressions, foreign key delete actions and partial-index predicates with schema.sql. The per-test reset truncates only tables with rows or a used identity, on one kept connection (about 6 ms instead of 50 ms). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Reviewed by Muse (muse-wonder-full) over two rounds; round 2 returned VERDICT: OK (46 database tests, and a 158/158 column-type check against a freshly migrated database). The dead import nit is fixed. Merging into the |
The foundation for moving the site data to native Postgres (#251): a
src/databasepackage with a pooled psycopg 3 connection, a migration runner with a typed baseline for every province.db table, pytest fixtures against a real Postgres, and a Postgres service in CI. Nothing reads or writes site data through it yet; the SQLitesrc.skins.db.connect()is untouched, so the unconverted areas and their tests keep running. Area PRs intopostgresmove each area over.What it adds
backend/src/database/(psycopg[binary]==3.3.6,psycopg-pool==3.3.3)connect()borrows a connection from a process-wide pool opened fromDATABASE_URL(sizesDATABASE_POOL_MIN_SIZE/DATABASE_POOL_MAX_SIZE, default 1 and 10; 10 s checkout timeout; each borrow checks the connection). The block is one transaction: it commits when it ends normally and rolls back on an exception. Rows are dicts (dict_row). Every connection starts withsearch_path=public(the database is shared with Precedent and patch notes),TimeZone=UTC,lock_timeout=5s,statement_timeout=30sandidle_in_transaction_session_timeout=60s. A returned connection is reset before the next borrower gets it: the pool rolls back, then_resetrunsDISCARD ALL's parts (settings back to their start-up values; cursors, temp tables, LISTENs and session advisory locks gone) and putsautocommit, isolation, read-only and the row factory back to their defaults. It leaves outDEALLOCATE ALL: psycopg 3.3 notices aDISCARD ALLonly the first time a connection runs it, and after the second reset it named prepared statements the server had dropped. psycopg's own prepared statements are safe to keep.transaction(isolation=…, read_only=…, lock=…)isconnect()withSET TRANSACTIONapplied first and, optionally, an advisory lock taken before anything else.advisory_lock(conn, name, wait=True)takespg_advisory_xact_lock(hashtextextended(name, 0)); commit or rollback releases it.wait=FalsereturnsFalseinstead of waiting. The wait is bounded by its ownADVISORY_LOCK_TIMEOUTof 20 s (set withset_config(…, true)for the acquisition only, then the transaction's previouslock_timeoutcomes back). That is longer than the 5 s row-lock bound, because an advisory holder may run a whole Patreon recompute or LuckPerms snapshot, and shorter than the 30 sstatement_timeout, so a waiter fails withLockNotAvailablerather than a cancelled statement. Callers need no retry.serializable(work, attempts=5)runswork(conn)in a SERIALIZABLE transaction and retries it on serialization failure or deadlock, with a short random backoff.UniqueViolation,ForeignKeyViolation,SerializationFailure,LockNotAvailable, …) andJsonbare re-exported.Migrations.
src/database/migrations/NNNN_name.sql, applied in order bysrc/database/migrate.py, each file in its own transaction with its row inschema_migrations(version, name, SHA-256, applied time). A session advisory lock lets one runner work at a time, so two backends starting together never apply a file twice. The run stops, changing nothing, when an applied file has changed, an applied version has no file, or a pending file is numbered below the newest applied one. Rows the site needs live insrc/database/seeds.sql(idempotentON CONFLICT DO NOTHINGinserts; today onlypatreon_sync_stateid 1) and are applied after the migrations withseed=True. The backend migrates and seeds at start-up whenDATABASE_URLis set (previews have none, so they skip it). The copy from SQLite migrates without seeding, so every table starts empty and SQLite's ownpatreon_sync_staterow is copied in. By hand:python -m src.database.migrate [--seed] [--status] [--database-url URL].0001_baseline.sqlcreates all 61 tables inschema.sqlat their final shape, which includes whatskins/db.py's_upgrade()adds (the role, placeholder, acting-user, nickname, link method, minted-via, tier sets, held-until and roster death columns;rpc_player_meta.donator_tierdropped) and its two indexes (idx_users_role,idx_user_sessions_acting). The tables that only old database files hold are left out:armourshop_player_metaanddrink_player_meta(dropped from the schema in #85) andplayer_keys(removed in99359c83). No code reads them; the copy tool should list any table it skips.INTEGER PRIMARY KEY AUTOINCREMENTisbigint GENERATED BY DEFAULT AS IDENTITY, so the copy tool can insert the existing ids (and mustsetvaleach sequence afterwards). Singletons stayinteger PRIMARY KEY CHECK (id = 1), other integers arebigint,REALisdouble precision.timestamptz, 0/1 flagsboolean, JSONjsonb(table below). Three kinds of column staytext: day strings (day,*_day), which are compared as strings and used in file paths; ledgerfounded_at, which is hashed intofaction_keyand must stay byte-exact; andcharacter_roster.created_at, which holds RPCharacters' epoch-seconds string.COLLATE "C". The sites' existingprovincesystemdatabases were created with the image'sen_US.utf8default, which sorts_,a,B,é,Z; "C" sorts by code point (B,Z,_,a,é), as SQLite's BINARY did, and itslower()folds ASCII only, as SQLite's does. Putting it on the columns makes the schema independent of how the database was created. Future migrations must do the same;test_schema.pyfails on any text column without it.active = 1predicates onpatreon_linksbecomeWHERE active.microsoft_link_states UNIQUE (user_id)(it replaces the plain index), a partial unique index onpatreon_alerts (kind) WHERE acked_at IS NULL, and two backstops for invariants SQLite's lock already kept:will_requests (realm_id, character_id) WHERE status = 'pending'andlp_changes (target_type, target) WHERE status IN ('pending', 'sent').patreon_alertsis the one that live data might break, becausealert()is also called outside a lock (patreon/client.py:172); the copy tool should report duplicate open alerts.idx_codes_hash,idx_sessions_token,idx_discord_links_discord,idx_discord_link_codes_hash,idx_war_declare_codes_hash,idx_character_roster_player,idx_character_roster_player_realm).admin_auditis append-only: one plpgsql function behind aBEFORE UPDATE OR DELETE … FOR EACH ROWtrigger and aBEFORE TRUNCATE … FOR EACH STATEMENTtrigger. An upsert onto an existing id fires the UPDATE trigger, so SQLite's replace trigger has no counterpart.Tests.
src/database/testing.pyprovides the fixtures, registered inbackend/conftest.py. The session creates its own database on the server inTEST_DATABASE_URL(defaulten_US.utf8locale, like the sites), applies the migrations and seeds once and points the pool at it (and reopens it if a test replaced it). A test opts in by taking thepostgresfixture, or a whole module withpytestmark = pytest.mark.usefixtures("postgres"). Before each such test one query finds the tables with rows or a used identity sequence (a rolled-back insert still advances it), and one TRUNCATE empties them with identities restarted (undersession_replication_role = replica, which skips the audit TRUNCATE trigger). The seed rows are copied back. It runs on one kept connection, taking about 6 ms instead of about 50 ms for truncating every table.scratch_databasegives an empty database for runner tests. WithoutTEST_DATABASE_URLthe tests skip, except underCI, where they fail. SQLite tests are unchanged.New tests:
test_connection.py:search_pathlock_timeout, gives up after its own bound, and putslock_timeoutbackserializableautocommit, read-only, row factory, prepared statementsautocommitcan no longer commit a block that raisedtest_migrate.py:test_schema.py:schema.sqlaftermigrate(): tables, column order, allowed type per SQLite affinity, nullability, primary keys, unique sets, foreign keys with their ON DELETE action, column defaults, CHECK expressions and partial-index predicates (both sides normalised:INvs= ANY (ARRAY[…]), casts,flag = 1vsflag)admin_auditrefusing UPDATE, DELETE, TRUNCATE and upserttests/test_server_lifespan.pycovers the start-up migration with and withoutDATABASE_URL.CI.
build.ymlgets apgvector/pgvector:pg17service (tmpfs data) and setsTEST_DATABASE_URLfor the backend step. Build also runs for pushes and PRs onpostgres; Deploy still follows onlymain.Type changes callers must handle
Each changed column returns a different Python value. Writes are more forgiving than reads: psycopg sends
strparameters untyped, so an ISO string still lands in atimestamptzcolumn and ajson.dumps()string in ajsonbcolumn, and an ISO string compared with atimestamptzcolumn inWHEREis cast. Integers are not cast toboolean.timestamptzdatetimein UTC (tzinfo=timezone.utc)datetime(preferred), or an ISO stringnow()jsonbdict/list/scalarJsonb(value); a baredictcannot be adaptedcol ->> 'key'replacesjson_extractbooleanboolbool;0/1ints are refusedWHERE col/NOT col;= 1,= 0andSUM(col)failbigint,integer,double precisionint,floatas beforetextcolumn compared with anintparameter failstimestamptz(nowdatetime)jsonb(nowdict/list)boolean(nowbool)codescreated_at,expires_at,redeemed_atrevokedsubmissionscreated_at,reviewed_at,applied_attiers,tier_aliases,helmet_3d_tiers,name_colours,name_styles,tier_scrolls,tier_setsadd_name,staffsessionsexpires_at,created_atdiscord_linkslinked_at,left_guild_at,grace_untildiscord_link_codescreated_at,expires_at,used_atwar_declare_codescreated_at,expires_at,redeemed_atrevokedskin_notificationscreated_at,delivered_atpayloadplugin_noticescreated_at,delivered_atpayloadplayer_warningscreated_atmoderation_notificationscreated_at,delivered_atpayloadarmourshop_catalogupdated_atpayloadlore_item_customisationsready_at,applied_at,updated_atname_colours,name_styles,lore_jsoncharacter_wardrobe_slotsupdated_atapply_pendingcharacter_create_wardrobeupdated_atdrink_submissionscreated_at,reviewed_at,applied_atrecipe_jsonnew_texturedrink_texturescreated_atdrink_catalogupdated_atpayloadrpc_player_metaupdated_atskin_kinds_json,permission_flags_jsonallow_drink_texture,allow_drink_message,allow_armor_3d_helmetdrink_notificationscreated_at,delivered_atpayloadcosmetic_mint_resetsreset_atmap_chronicle_snapshotsmanifestmap_chronicle_snapshots_archivemanifestmap_ledger_dayscaptured_atglobalscompletemap_ledger_factionsfirst_seen_at,last_seen_at,deleted_atmap_ledger_faction_dayswealth_breakdown,prestige_breakdown,subjects,warspatreon_membersdeclined_since,updated_atdata_jsonpatreon_linkslinked_at,person_changed_atactivepatreon_tokensexpires_atpatreon_leasesexpires_atpatreon_sync_statelast_sync_atlast_sync_ok,brake_heldpatreon_appliedtiers_jsonever_grantedpatreon_desiredgrace_untilpatreon_changesgrace_until,created_at,dispatched_at,acked_at,cancelled_atremove_jsondm_suppressedpatreon_alertscreated_at,acked_atpatreon_oauth_statesexpires_at,used_atpatreon_pending_linksexpires_at,used_atpatreon_webhook_membersrecorded_atcreation_catalogupdated_atpayloadcharacter_createscreated_at,applied_at,held_untilpayloadcharacter_rosterupdated_at,died_atkit_statuses_json,sheet_jsoncharacter_estatessource_updated_at,received_atpayloadwill_requestscreated_at,acked_atcharacter_player_metaupdated_atkit_cooldowns_jsoneighteen,real_age_setuserscreated_at,updated_at,last_login_atplaceholderuser_sessionsguild_checked_at,created_at,expires_atguild_memberdiscord_oauth_statesexpires_atpreview_sign_in_ticketsguild_checked_at,expires_atguild_membermicrosoft_link_statesexpires_at,used_atadmin_auditcreated_atdetail_jsonlp_statereceived_at,checked_at,polled_atlp_group_nodescontextsvaluelp_tracksgroups_jsonlp_user_nodescontextsvaluelp_commandsreceived_atpayloadlp_changescreated_at,sent_at,finished_atops_jsondiagnostic_reportscreated_atreport_jsonPlaces the type sweep found that break or change behaviour silently, for the area PRs:
strptime(value, "%Y-%m-%dT%H:%M:%SZ"),_parse_iso,parse()orfromisoformaton a read value raises or misbehaves on adatetime; so doesstr(value)(it gives2026-10-16 10:00:00+00:00). Examples:skins/codes.py:220/865(cooldown),skins/discord_link.py:80/495,wars/declare_codes.py:131,auth/users.py:105/202/235,luckperms/mirror.py:40,luckperms/changes.py:56,patreon/resolver.py:47.datetimeas2026-10-16T10:00:00+00:00, where the stored strings were mostly…Z. Pin the format where a client parses it strictly: TFMCWeb (last_mint_at), the preview sign-in hand-off (preview_sign_in.py:118-143expects the Z form), and tests that assert exact strings.json.loadsinsideexcept (…, TypeError)would quietly return{}/[]on an already-parsed value: the four notification payloads,drink_catalog.payload,recipe_json, chroniclemanifest(scripts/chronicle/store.py:339-343reads the TypeError as "referencing"), ledgerglobals/breakdowns/subjects/wars, rosterkit_statuses_json/sheet_json, estatespayload. Reads without the fallback fail loudly instead (patreon/service.py,lp_*,creation_catalog,armourshop_catalog,lore_json,report_json).patreon/service.py:158/162compare storedgrace_untilwith a fresh ISO string (ondatetimethey would never match, bumping the generation and re-sending DMs every sync);service.py:224buildslink_eventasid + ":" + linked_at, which must keep the same string (iso(linked_at)reproducesisoformat());characters/estates.py:230compares a pushed string with the storedsource_updated_at.= 1/= 0onplaceholder,active,revoked,complete,ever_granted,staff(for exampleauth/users.py:41/135,auth/admin.py:46/142,patreon/service.py:81…595,scripts/ledger/store.py:510,characters/lore_items.py:769/771);int(value)on aboolstill works.skins/submissions.py:1302comparesreviewed_atwith the plugin's rawsincequery string; a malformed value becomes a Postgres cast error, so validate it first.character_player_meta.kit_cooldowns_jsonis written back as the read value (characters/roster.py:661); wrap it inJsonb().Design note for the area PRs
Connections and transactions. Use
with database.connect() as conn:once per unit of work and passconnto helpers. Placeholders are%s(or%(name)s); a literal%in SQL with parameters is written%%. Rows are dicts:row[0]no longer works, and when a join returns two columns with the same namedict_rowkeeps the last one, wheresqlite3.Rowreturned the first. Alias every duplicate column name (SELECT s.id AS session_id, u.id AS user_id …), and avoidSELECT *across joins.INSERT … RETURNING idreplaceslastrowid. Do not callconn.commit()inside the block unless the work really is two transactions; never keep a connection outside its block. An error aborts the whole transaction, so code that catches a database error and carries on must wrap the failing statement inwith conn.transaction():(a savepoint), asskins/submissions.py:967will need for slug conflicts. Async routes should call database code throughrun_in_threadpool.Locking. Postgres runs READ COMMITTED, so a read followed by a write no longer sees the world frozen as SQLite's
BEGIN IMMEDIATEdid. Pick, in order of preference:UPDATE … SET used_at = %s WHERE key = %s AND used_at IS NULL AND expires_at > %s RETURNING …, orDELETE … RETURNING. No row back means someone else won; re-read only to choose the error.INSERT … ON CONFLICT … DO NOTHING RETURNING …(orDO UPDATE) where the race is two inserts of one key.SELECT … FOR UPDATE(orFOR SHAREwhen you only need the row to stay as read) on the rows the check reads, when several statements follow. Lock in a consistent order (ORDER BY id).database.transaction(lock="area:what:key")oradvisory_lock(conn, …)as the first statement, when the check is on absence or an aggregate with no row to lock. Every path that changes what the check reads must take the same key. The wait is bounded at 20 s by the helper, with no retry needed.database.serializable(work): for the rare transaction too tangled for the above;workmay run several times, so it must not call out.Shared recipes used below:
session_user_in(conn, token, lock=True)(auth/users.py:186). The opt-inlock=TrueaddsFOR SHARE OF s, u, so a demotion or revocation waits for the action, or the action sees it. Ordinary auth reads keeplock=Falseand take no lock._end_where(auth/view_as.py:164) selects the view sessionsFOR UPDATE OF s, so two stops end and audit a view once.drop_view_leftovers(auth/users.py:28), andview_as.start, opens withdatabase.transaction(lock="auth:placeholders"). The lock is taken as the transaction's first statement, before any row lock, never insidedrop_view_leftovers. ItsDELETE … WHERE placeholder AND id NOT IN (…)could otherwise delete a placeholder whose view is being committed.patreonaround anything that runs_compute/_plan(patreon/service.py:214), which reads and rewrites every Patreon table.lp_state(ON CONFLICT (id) DO NOTHING), thenSELECT … FROM lp_state WHERE id = 1 FOR UPDATEas the transaction's first lock.LuckPerms lock order:
lp_statefirst, then users and sessions (X1), thenlp_changesrows, in every LuckPerms transaction.The 42
BEGIN IMMEDIATEpaths:auth/users.py:98consume_stateDELETE … WHERE state_hash = %s RETURNING return_to, expires_atauth/users.py:126sign_inusers.discord_user_idwithRETURNING id; X3auth/admin_cli.py:28_set_rootORDER BY id FOR UPDATEand count, or the target row for a grant; X3auth/admin.py:56_staff_writeusersrowsORDER BY id FOR UPDATE, then X1; X3auth/view_as.py:35startON CONFLICT (discord_user_id) DO NOTHING, re-select); X2auth/view_as.py:134stop,:198endauth/view_as.py:215end_lapsedauth/preview_sign_in.py:97redeemDELETE … RETURNINGjoined tousersapi/admin_routes.py:267_server_writeauth/microsoft_link.py:38startUNIQUE (user_id):ON CONFLICT (user_id) DO UPDATEauth/microsoft_link.py:61consumeUPDATE … SET used_at WHERE … AND used_at IS NULL AND expires_at > %s RETURNINGskins/discord_link.py:321complete_link_bind(ON CONFLICT DO NOTHING RETURNING, re-select for the error)skins/discord_link.py:353link_verified_profileFOR SHAREon the state and session rows inmicrosoft_link.still_valid; C in_bindskins/codes.py:716start_linked_profile_sessionFOR SHAREinview_as.view_still_open; nothing withoutstill_allowedskins/codes.py:930start_site_sessioncosmetic_mint:<lower(uuid)>(across realms). The TFMCWeb mint insert (skins/codes.py:286, inissue_code) takes the same lock, so both mint paths and the cooldown read are serialised.skins/codes.py:503redeem_code,:750redeem_drink_code(noBEGIN IMMEDIATEtoday)SELECT … FROM codes WHERE id = %s FOR UPDATEand re-validate (not revoked, not expired, no submission yet) inside the submission-insert transaction:create_submission(skins/submissions.py:874-882) for skins,create_drink_submission(skins/drinks.py:797-800) for drinks. The submission insert is what consumes the code; the redeem step only opens a session, so it can stay a plain read.characters/wills.py:134create_will_requestFOR UPDATE; the new partial unique index backs itcharacters/wills.py:212ack_will_resultsUPDATE … WHERE id = %s AND status = 'pending' RETURNINGluckperms/mirror.py:125replace_snapshotluckperms/changes.py:325queue_changelp_stateand take itFOR SHAREfirst, sobuild_*_opsandcheck_ops_allowedread one consistent mirror whilereplace_snapshotwaits; then X1, then C on the new partial unique index (or D onlp_change:<type>:<target>)luckperms/changes.py:496fetch_for_bridgerecord_poll's upsert first (row-lockslp_state);_current_actor(changes.py:444) reads the actor rowFOR SHARE; then B per row (… WHERE status = 'pending')luckperms/changes.py:553record_resultslp_state FOR UPDATEbefore anylp_changesrow, matchingfetch_for_bridge's order; todayapply_statelocks it after the updates, which can deadlock); then B:… WHERE id = %s AND status IN ('sent', 'unknown') RETURNINGpatreon/client.py:164refresh_tokenstoken_refreshlease rowFOR UPDATE; the token UPDATE is already Bpatreon/service.py:30acquire_leaseON CONFLICT (name) DO UPDATE … WHERE expires_at <= %s RETURNING ownerpatreon/service.py:141create_or_update_linkpatreon/service.py:259recompute_link,:267unlink,:495release_brake,:566staff_link,:583import_linkspatreon/service.py:309_sync_failureUPDATE patreon_sync_statelocks the row; C on the newpatreon_alertsindex inalert()patreon/service.py:333_run_fetchsynclease rowpatreon/service.py:430list_changes_plan's cancelpatreon/service.py:454ack_changespatreon/linking.py:57start_link,:159_finish_callbackpatreon/linking.py:88consume_stateRETURNING)patreon/linking.py:174_pendingRETURNINGwhenconsume; today line 180 ignores the row count, so two confirms would both linkpatreon/linking.py:214_confirm_linkskins/db.py:101_upgradedbstudio/studio.py:246_siteRead-then-write races without
BEGIN IMMEDIATEthat the same review found:luckperms/changes.py:460-469settle_timeouts(B: add the status to theWHERE),skins/codes.py:592-635redeem_profile_code(B), the skin and drink submission inserts (in the table above),skins/codes.py:125-153ensure_lore_upload_code(C oncode_hash), andpatreon/service.py:50alert(C). None of the 42 paths catches a database error and then keeps using its connection.Tests. Convert an area's tests to take
postgres(module-levelusefixtures), drop theirDB_PATHpatching andmigrate()calls, and seed through the area's own functions or plain SQL. Tests that only check SQLite behaviour (iterdump,PRAGMA, legacy upgrades) go with the area's SQLite code.Checks
src/databaseandtests/test_server_lifespan.py: 46 passed againstpgvector/pgvector:pg17locally.TEST_DATABASE_URLset: 2,432 passed, 4 skipped, 175 subtests passed (13 min 8 s), the new database tests included.Not deployed anywhere:
postgresis not deployed until the copy tool and host plumbing land (plan on #251).🤖 Generated with Claude Code