Normalize the contract and estimate foreign-key fan-out - #8
Merged
Conversation
Turns collected catalog rows into the profile contract. Two rules govern the section: PostgreSQL-native names stay out of the contract and live in provenance instead, and nothing is invented -- where PostgreSQL has no estimate the contract carries None, because a plausible number is worse than a gap and a gap gets investigated. Token domains follow foreign-key chains to their root. Resolving only one hop would have A.x tokenize under B.y while B.y tokenized under C.z, making the A-to-B join invisible: the exact failure the domains exist to prevent. Cycles terminate, since self-referential keys are legal. Fan-out excludes nulls from the numerator; a null foreign key references no parent. A composite key is estimated only from extended statistics covering exactly its column set, never from the product of its columns' distinct counts -- foreign-key columns are almost never independent, and multiplying would understate fan-out by orders of magnitude. p99 comes from the most-common-value frequencies, which is what makes the hot-parent shape survive tokenization. The fixture's orders.user_id has a mean of 10 and a p99 of 100; sizing a migration on the mean alone would be wrong by an order of magnitude. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task 8 of the MVP plan.
Turns collected catalog rows into the profile contract. Two rules govern the section:
provenanceinstead. A consumer readingprofile.jsonshould not have to know thatreltuplesis an estimate or thatn_distinctis negative when it means a fraction. The native encodings are recorded rather than discarded — a number without its derivation is not auditable. A test assertsreltuples,n_distinct,null_frac,attnumandrelkindappear in no contract field name.Noneand the fan-out saysinsufficient_statistics.n_distinct = 0means "no estimate", not zero distinct values — resolving it as a count would put a zero in every fan-out denominator.Token domains follow foreign-key chains to their root. A child column borrows its parent's domain so both sides tokenize alike, but with
A.x → B.yandB.y → C.z, resolving one hop would haveA.xtokenize underB.ywhileB.ytokenized underC.z— making the A-to-B join invisible, the exact failure domains exist to prevent. Cycles terminate; self-referential keys are legal.Fan-out excludes nulls from the numerator — a null foreign key references no parent, and counting those rows inflates every estimate on a nullable key. A composite key uses the most-null column's fraction, since PostgreSQL has no joint null fraction, so the estimate errs high; that direction is deliberate, because an under-reported hot parent is otherwise discovered during the migration.
A composite key is never estimated from the product of its columns.
user_idhas 500 distinct values andplaced_at2500, but orders are not placed at independently random times — the real pair count is 4200, not 1,250,000. Only extended statistics covering exactly the column set count, matched as a set rather than a sequence.p99comes from the most-common-value frequencies, which is what makes the hot-parent shape survive tokenization. The fixture'sorders.user_idhas a mean of 10 and a p99 of 100: sizing a migration on the mean alone would be wrong by an order of magnitude, which is why the field exists. Where the MCV list is shorter than the percentile's rank, the last MCV's count is returned as an upper bound.Verification: 253 tests pass,
--check-safetyclean,ruff checkclean, 3.9 grammar verified. A planted email from the statistics fixture is proven absent from the normalized records.