Task 10: orchestration and the postgres subcommand - #10
Merged
Conversation
run_postgres() wires the collectors together: probe the server version, check the client, fingerprint the catalog, dump the schema, read the catalog, read the statistics views, re-fingerprint, normalize, and publish. Fixes a bug the orchestration tests found. build_postgres_config resolved the whole output path, and Path.resolve() follows a symlinked final component, so by the time the publication-time symlink check ran there was no symlink left to see -- the tool would have written straight through it. The config now resolves the parent directory and keeps the final component as given. The tokenization key and the destination are both validated before a connection is opened. A missing DBPROFILER_TOKEN_KEY discovered after collection costs the operator a full pass over a production catalog, and so does an unwritable destination. An empty pg_dump is fatal: pg_dump exits zero when the role can see no objects, which would otherwise publish a bundle with an empty schema.sql and no other complaint. REQUIRED_BUNDLE_PATHS covers the same class of failure for the profile and the three catalog CSVs, which may not degrade to a warning. Source.collected_schemas is derived from what the catalog returned rather than from --schema-include, so a bundle that covered less than the operator expected says so on its face. Progress goes to stderr and stdout carries the bundle path alone, so the run composes in a shell substitution. KeyboardInterrupt exits 130; write_bundle already removes its own temporary file, so the handler only has something to say. The test fake dispatches on the SQL constant a call carries rather than on call position, so the order assertion is one comparison against the documented sequence and an override names the step it replaces. Secrecy is asserted for five values across stdout, stderr and the bundle bytes, on the success path and on two failure paths where a server error deliberately echoes the URL and the password back. As in task 9, the guarantees were mutation-tested: dropping the after-collection fingerprint comparison, neutering the comparison, publishing an incomplete bundle, and deferring the key load until after collection are each caught. 354 tests pass; --check-safety and ruff are clean; the source parses under Python 3.9. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
run_postgres()wires the collectors together andpostgres --outputnow produces a bundle end to end: probe the server version → check the client → fingerprint the catalog → dump the schema → read the catalog → read the statistics views → re-fingerprint → normalize → publish.A bug the tests found
build_postgres_configresolved the whole output path, andPath.resolve()follows a symlinked final component. By the time task 9's publication-time symlink check ran there was no symlink left to see, so the tool would have written straight through it. The config now resolves the parent directory and keeps the final component as given. Two regression tests: the symlink survives into the config, and a..in the directory part is still normalized away.Fail early, fail completely
The tokenization key and the destination are both validated before a connection is opened — a missing
DBPROFILER_TOKEN_KEYdiscovered after collection costs the operator a full pass over a production catalog, and so does an unwritable destination. Two tests assert those failures happen with zero child processes spawned.An empty
pg_dumpis fatal: it exits zero when the role can see no objects, which would otherwise publish a bundle with an emptyschema.sqland no other complaint.REQUIRED_BUNDLE_PATHScovers the same class of failure forprofile.jsonand the three catalog CSVs — those may not degrade to a warning, and a bundle missing one of them looks complete and is not.SchemaDriftis its own error class, so concurrent DDL is distinguishable from a command failure by a caller and not only by message text.KeyboardInterruptexits 130;write_bundlealready removes its own temporary file, so the handler has something to say and nothing to clean up.Every failure path asserts the same three things: the exit code, that nothing was left in the destination directory, and that an existing bundle survives untouched.
Reporting
Progress goes to stderr; stdout is the bundle path and nothing else, so the run composes in a shell substitution.
Source.collected_schemasis derived from what the catalog actually returned rather than from--schema-include, so a bundle that covered less than the operator expected says so on its face.Testing
The fake child process dispatches on the SQL constant a call carries, not on call position. A positional
side_effectlist needs renumbering every time a step moves, and asserting "17 calls happened" proves nothing about their order; keying on the query makes the order assertion one comparison against the documented sequence, and a list-valued override is consumed one entry per call, which is how the two fingerprint reads are given different answers to simulate concurrent DDL.Secrecy is asserted for five values — URL, password, user, host, tokenization key — against stdout, stderr and the bundle bytes, on the success path and on two failure paths where a server error deliberately echoes the URL and the password back. Two further tests assert the credentials reach the child through
env=and never throughargv, and that neitherDBPROFILER_*variable is passed down.As in task 9, the guarantees were mutation-tested: dropping the after-collection fingerprint comparison, neutering the comparison itself, publishing an incomplete bundle, and deferring the key load until after collection are each caught by the suite.
Verification
python3 dbprofiler.py --check-safety— OK, 14 SQL constantsruff check— clean