Skip to content

Add pg_dump schema extraction and the catalog drift fingerprint - #4

Merged
roachitect-aman merged 1 commit into
mainfrom
task-4-schema-extraction
Aug 25, 2026
Merged

Add pg_dump schema extraction and the catalog drift fingerprint#4
roachitect-aman merged 1 commit into
mainfrom
task-4-schema-extraction

Conversation

@roachitect-aman

Copy link
Copy Markdown
Contributor

Task 4 of the MVP plan. First real consumer of the run_command / run_pg_dump
helpers from task 3.

collect_schema(config, server_version_num) returns the schema DDL as a string
together with a catalog fingerprint taken before the dump, so the after-collection
recheck in task 10 covers drift during the dump itself.

The client check runs first

An older pg_dump does not refuse a newer server outright in every case — it can
emit a dump that is quietly wrong, and discovering that midway through a migration is
expensive. probe_pg_dump_major() opens no connection, and a test asserts nothing
else executes when the check fails.

UnsupportedClientVersion is deliberately separate from UnsupportedServerVersion:
the remedy differs, since the operator upgrades the local client, not the server.

Dump arguments

--schema-only --no-owner --no-privileges, plus -n / -N for the schema filters.
Roles and grants are the customer's access control, not schema shape a migration
needs. A test joins the whole argv and asserts none of the URL, host, user, password,
or database appears anywhere in it.

Fingerprint

SHA-256 over sorted (nspname, relname, relkind) from pg_class joined to
pg_namespace. Three details worth review:

Scope filtering happens in Python, not in SQL. Building the WHERE clause from
--schema-include would mean assembling SQL inline, which the house rule forbids
precisely so --check-safety can see every statement the tool can issue.
SQL_SCHEMA_FINGERPRINT stays a fixed module constant and the rows are filtered after
they come back. That has a second benefit: a schema the operator excluded cannot abort
the run by changing underneath it. Tests cover both directions — out-of-scope churn is
ignored, in-scope churn is still detected.

Fields and rows are separated by ASCII unit and record separators (\x1f, \x1e).
Neither can occur in a PostgreSQL identifier, so (public, ab) and (publica, b)
cannot collide into the same digest. That is a test, not a comment claiming it.

User schemas are selected with left(nspname, 3) <> 'pg_' rather than a LIKE
pattern. One comparison covers pg_catalog, pg_toast, pg_temp_N and
pg_toast_temp_N, and it needs no backslash escaping inside a Python string. There is
no relkind filter — TOAST relations are already excluded by the schema test, and a
narrower filter would only create blind spots.

Deviations from the plan

collect_schema takes the server version and runs three child processes, not two.
The checklist said "one dump and one fingerprint query". It also checks the client
itself rather than trusting the orchestrator to remember, which adds a
pg_dump --version probe. Failing before we produce anything is worth the extra
process.

--schema-include validation moved to config time, in build_postgres_config, so
the operator finds out before we connect. PostgreSQL reserves the pg_ prefix, so
pg_myschema is rejected too — it cannot be a real user schema. --schema-exclude is
not validated: excluding something already out of scope is harmless.

Verification

  • 125 tests pass (33 new).
  • python3 dbprofiler.py --check-safety exits 0; the new SQL_SCHEMA_FINGERPRINT
    clears the relation allowlist, and a test asserts audit_sql() returns nothing
    for it.
  • Adversarially probed: swapping pg_class for pg_statistic in that exact query
    shape is caught by the audit with its reason.
  • ruff check clean. Source re-verified as 3.9-parseable.

collect_schema(config, server_version_num) returns the schema DDL as a
string together with a catalog fingerprint taken before the dump, so the
after-collection recheck in task 10 covers drift during the dump itself.

The client check runs first. An older pg_dump does not refuse a newer
server outright in every case; it can emit a dump that is quietly wrong,
and discovering that during a migration is expensive. probe_pg_dump_major
opens no connection, and a test asserts nothing else runs when the check
fails.

build_pg_dump_args passes --schema-only --no-owner --no-privileges plus the
-n and -N schema filters. Roles and grants are the customer's access
control, not schema shape a migration needs. A test joins the whole argv
and asserts none of the URL, host, user, password, or database appears in
it.

Scope filtering for the fingerprint happens in Python rather than in SQL.
Building the WHERE clause from --schema-include would mean assembling SQL
inline, which the house rule forbids so that --check-safety can see every
statement. Filtering the returned rows has a second benefit: a schema the
operator excluded cannot abort the run by changing underneath it.

The canonical form separates fields and rows with ASCII unit and record
separators. Neither can occur in a PostgreSQL identifier, so no pair of
relation names can forge a field boundary and collide into one digest.
There is a test for that rather than a comment claiming it.

User schemas are selected with left(nspname, 3) <> 'pg_'. One comparison
covers pg_catalog, pg_toast, pg_temp_N and pg_toast_temp_N, and it needs no
backslash escaping inside a Python string. No relkind filter: a narrower
one would only create blind spots, and TOAST relations are already excluded
by the schema test.

--schema-include validation moved to config time, so the operator finds out
before we connect. PostgreSQL reserves the pg_ prefix, so pg_myschema
cannot be a real user schema either. --schema-exclude is not validated;
excluding something already out of scope is harmless.

UnsupportedClientVersion is separate from UnsupportedServerVersion. The
remedy differs: upgrade the local client, not the server.

125 tests pass.

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
@roachitect-aman
roachitect-aman merged commit 95c4cd4 into main Aug 25, 2026
7 checks passed
@roachitect-aman
roachitect-aman deleted the task-4-schema-extraction branch August 25, 2026 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant