feat(db): chronicle and ledger on native Postgres — #251

[ryanbarlow97]

The map chronicle, its wipe audit and the economy ledger move to native Postgres (#251, plan item 7): every read and write in the area goes through src.database with %s placeholders, RETURNING, ON CONFLICT, the new column types and COPY for a day's rows. The ledger read routes take one snapshot per request, a promotion is backed by an advisory lock, and every ledger ETag is unchanged for the same data. No unconverted area calls into this one with a SQLite connection, so there are no PG-MIGRATION marks.

Files converted

• backend/src/scripts/ledger/store.py: every read and write; open_connection() is now a pooled database.connect() block, and read_view() is new.
• backend/src/scripts/ledger/ingest.py: index_snapshot is one with block that takes ledger:index:<map> first.
• backend/src/scripts/ledger/wipe.py, reindex.py: Postgres connections. Their CLIs validate the map first and run src.database.migrate (no seed) for writes; --dry-run applies no migrations and database failures become usage errors. remap_wealth_keys.py never touched the database and drops its migrate().
• backend/src/scripts/chronicle/store.py, audit.py, wipe.py (CLI migrates like the ledger's), restore.py. capture.py and verify.py only call store and are unchanged.
• backend/src/scripts/maps/archive_copy.py: the chronicle rows are cloned with one INSERT … SELECT … RETURNING day in the same transaction as the delete, in place of a read and an upsert per day.
• backend/src/scripts/tools/generate_spoof_ledger.py: existing_ledger_days answers [] or the days, and raises LedgerIndexUnreadable when DATABASE_URL is unset or the database cannot be reached. The old "no DB_PATH file means no history" answer is gone, because on Postgres there is no file to look for.
• backend/src/api/ledger_routes.py: /ledger/index, /ledger/series and /ledger/faction/{key} each read inside one store.read_view().
• backend/src/api/maps_routes.py: /maps/accessible is a plain def, so the archived chapters' chronicle reads run in the threadpool rather than on the event loop.
• backend/src/api/chronicle_routes.py, chronicle_staff_routes.py, data_routes.py: only call the converted modules. The chronicle index now reads its day rows and wipe stamp in one read-only repeatable-read snapshot; the staff routes already used run_in_threadpool.

Dialect fixes

Where	SQLite	Postgres
scripts/ledger/store.py:399-416 touch_registry	ON CONFLICT … SET first_seen_day = MIN(first_seen_day, …) with unqualified columns and scalar MIN/MAX	table alias AS f, LEAST(f.first_seen_day, excluded.first_seen_day), GREATEST(f.last_seen_day, …), every right-hand column qualified
scripts/ledger/store.py:424-451 mark_deleted	CREATE TEMP TABLE + INSERT OR IGNORE of the present keys, NOT IN (SELECT …)	faction_key <> ALL(%s::text[]), one array parameter for up to 2,000 keys
scripts/ledger/store.py:612 read_faction_days	IN (?, ?, …)	= ANY(%s::text[])
scripts/ledger/store.py:504 latest_complete_day	complete = 1	complete
scripts/ledger/store.py:293-300 day rows	executemany INSERT	COPY … FROM STDIN (a third of the time for 2,000 rows)
scripts/chronicle/wipe.py:63-73 _archive_rows	INSERT OR REPLACE … SELECT	INSERT … SELECT … ON CONFLICT (map_id, day, archived_at) DO UPDATE
scripts/chronicle/restore.py:139-146 _reinsert_rows	read every archive row, INSERT OR IGNORE one at a time	one INSERT … SELECT … WHERE day <> ALL(%s) ON CONFLICT (map_id, day) DO NOTHING
scripts/chronicle/audit.py:91 record_wipe	cursor.lastrowid	RETURNING id
scripts/chronicle/audit.py last_wiped_at	caught sqlite3.OperationalError for a missing table	removed; the migrations create the table
scripts/maps/archive_copy.py:109-121	read + upsert per day	INSERT … SELECT … RETURNING day

json_extract, SQL strftime and rowid do not occur in this area. The only strftime calls are Python's, on aware datetimes, for day strings. json_extract was in api/database_routes.py, which #253 converts.

Type adaptations

• map_ledger_days.captured_at, map_ledger_factions.first_seen_at/last_seen_at/deleted_at (timestamptz): writes still pass the snapshot's …Z string. Reads go through store._iso() (store.py:123), which turns them back into isoformat() with Z, the form schema.normalize_snapshot writes: …T12:00:00Z, …T08:30:15.123456Z, …T23:59:59.500000Z. The API output and the series ETag inputs are byte-identical to SQLite's. Nanoseconds (only in old daily files, since normalize_snapshot already truncates to microseconds) round to microseconds once on write. Times are never compared: the registry's first/last-seen logic compares the *_day text columns.
• map_ledger_days.globals (jsonb): jsonb reorders keys, and the series ETag hashes the block's repr. Promotion always indexes the snapshot read back from its packed file, which ingest._pack writes with sorted keys, so SQLite held them sorted. store._global_block() sorts them again, including nested keys.
• Faction JSON columns (wealth_breakdown, prestige_breakdown, subjects, wars) and chronicle manifest: these arrive parsed. The json.loads … except (TypeError, ValueError) fallbacks that would have returned {}/[] on every row are gone. A manifest that is not an object still degrades to {} (chronicle/store.py:127). days_referencing treats such a manifest as a blocker, as the old code meant to do for an unparsable one; before, a list manifest raised AttributeError there. Writes wrap values in Jsonb().
• complete (boolean): written as bool, read as bool (it was already coerced).
• founded_at stays text, so faction_key = sha1(id \0 founded_at) is unchanged. A test stores a nine-digit founded_at and reads it back byte-exact.
• Chronicle captured_at, wipe stamps: unchanged bigint epoch seconds.

ETags. test_series_etags_match_the_sqlite_index pins the four ledger ETags (series core, series full, index, faction) for a fixed dataset with microsecond and nanosecond times. The values were taken from the SQLite build on postgres and match exactly, so clients keep their 304s across the cutover. Chronicle and ledger bodies that are hashed whole are encoded with sort_keys=True, so jsonb's key order cannot reach them.

Locking and races

#252 lists no BEGIN IMMEDIATE path or read-then-write race in this area. Promotion, reindex, wipe and restore are already serialised per map by the cross-process file locks (scripts/util/maplock.py), which are unchanged. Postgres's READ COMMITTED needed two additions:

• Read routes (store.read_view(), store.py:160): /ledger/series builds its ETag from the day rows and registry and then reads the faction rows. Under READ COMMITTED each statement sees its own snapshot, so a promotion committing in between could tag a body with an ETag from a different state, or return a faction row for a day the axis lacks. The routes now read inside one REPEATABLE READ, READ ONLY transaction, which keeps the "one consistent view" the code already promised.
• Index writes (ingest.py:236): index_snapshot takes ledger:index:<map> (D) as its first statement, so two index writes for one map cannot interleave their 2,000-row upserts even without the file lock.

Tests in scripts/ledger/test_postgres.py, each racing two connections:

• test_a_reader_never_sees_half_a_promotion: the writer is paused inside its transaction after all five writes but before commit, and the reader sees the old day.
• test_a_read_view_keeps_one_snapshot_across_a_promotion: a promotion commits mid-request, and the view still sees one day.
• test_index_writes_for_one_map_take_turns: the writer waits for a held lock.
• test_index_writes_for_other_maps_do_not_wait.

I checked that the second and third fail when the isolation level or the advisory lock is removed.

Timings

backend/benchmarks/ledger_promotion.py builds --days of 2,000 factions and 200 guilds per day, then times promotion and the routes through the real code. Results for 365 days (730,000 faction-day rows), Postgres 17 in Docker on the same 4-core host, against the same script on SQLite (the postgres branch, province.db on local disk):

	Postgres	SQLite
promote_day, 2,000 factions (raw read, daily file write, index)	316 ms median, 393 ms max	248 ms, 254 ms
index_snapshot only (n=360)	147 ms median, 260 ms max	239 ms, 448 ms
re-promote an indexed day (replace every row)	318 ms	371 ms
/ledger/series default (top 12, core, 365 days)	426 ms median, 814 ms max	418 ms, 459 ms
/ledger/series 40 factions, fields=full	684 ms	722 ms
/ledger/index (2,000-faction registry)	42 ms	28 ms

With 30 days the series routes take about 100 ms on Postgres. Promotion runs as a background task after the upload's 200, so it costs no request latency. Before COPY, the Postgres index write took 270 ms; the faction rows were most of that. The default series' time grows with history on both engines, mostly in count_factions_in_range's COUNT(DISTINCT faction_key) over the range. That was already the case on SQLite.

PG-MIGRATION marks

None. Nothing in this area takes a caller's connection from an unconverted area, and nothing here reads tables that still live in SQLite. Map access (Skins sessions) is a separate call that keeps its own SQLite connection.

Tests

Converted to the postgres fixture: scripts/ledger/* (through conftest.env), scripts/chronicle/test_audit, test_capture, test_restore, test_verify, test_wipe, scripts/maps/test_archive_copy, scripts/tools/test_generate_spoof_ledger, api/test_ledger_routes. api/test_chronicle_routes, test_chronicle_staff_routes, test_chronicle_archive_routes and test_data_routes_ledger take postgres and keep their SQLite isolation, because the full app's start-up and the staff session checks still touch province.db.

• The connection-count regressions (test_series_opens_one_connection_per_request, the chronicle index ones) now count pool borrows. The chronicle index borrows once for a repeatable-read snapshot containing both the day rows and the newest wipe stamp. It still does not grow with the number of days.
• The SQLite variable-limit test now checks that 1,201 present keys go in as one parameter.
• "Unparsable manifest" tests became "manifest that is not an object", since jsonb cannot hold invalid JSON.
• New tests: scripts/ledger/test_postgres.py (10: time format, nanosecond rounding, parsed JSON columns, byte-exact founded_at, pinned ETags, four races, read-only view) and test_a_later_manifest_that_is_not_an_object_blocks_the_overwrite.
• scripts/chronicle/test_schema_indexes.py still checks schema.sql on SQLite and stays until area 10 removes it.

Full backend suite against pgvector/pgvector:pg17 with TEST_DATABASE_URL set:

• postgres (b5e161e): 2,432 passed, 4 skipped, 175 subtests passed (18 min 0 s)
• this branch: 2,443 passed, 4 skipped, 175 subtests passed (16 min 12 s). The 11 more are the 10 in test_postgres.py, the chronicle manifest test, and one net spoof-ledger test (the "no database file" pass removed, "reads the index" and "raises without a database" added).

Review fixes and current validation

The first review found a NUL character could permanently wedge a saved day. Normalisation now recursively strips NUL from keys and values before files are saved; indexing applies the same scrub to older daily files. Colliding scrubbed guild ids keep the last row. Further fixes add a faction-key ordering tiebreak, recursive key sorting, real lock-wait assertions and CLI error checks. The whole-number float boundary is pinned: JSONB preserves float values through 1e15; values written in exponent notation at 1e16 can return as integers.

• After those fixes, on the original foundation base: 2,457 passed, 4 skipped, 175 subtests (/tmp/ps-pg-ledger-full2.log).
• Rebased onto accounts (postgres at 7d2a1ee7): 440 affected tests passed, including chronicle/ledger, archive copy, spoof data, and API routes. The final full suite is checked by CI for the rebased head.

Not deployed anywhere.

🤖 Generated with Claude Code

Sol round 2 follow-up

The independent Sol review confirmed the NUL, identity/deletion-safety, COPY rollback, snapshot and ETag fixes. One CLI gap remained: dry-run wipes on an unmigrated database raised UndefinedTable, a ProgrammingError rather than an OperationalError. The ledger and chronicle wipe handlers now catch DatabaseError. Two fresh-schema tests reproduced the old failure, then passed with the fix; all 50 targeted Postgres and wipe tests passed. The tests prove dry runs apply no migrations and leave the database schema empty. Final independent Sol review: VERDICT: OK at ce6ae3e9, with all 50 targeted tests independently passing. The new Build is the remaining integration gate.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 96d35d4f-7fde-4abc-9dfe-905942243deb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}