Skip to content

Normalize the contract and estimate foreign-key fan-out - #8

Merged
roachitect-aman merged 1 commit into
mainfrom
task-8-normalization
Aug 25, 2026
Merged

Normalize the contract and estimate foreign-key fan-out#8
roachitect-aman merged 1 commit into
mainfrom
task-8-normalization

Conversation

@roachitect-aman

Copy link
Copy Markdown
Contributor

Task 8 of the MVP plan.

Turns collected catalog rows into the profile contract. Two rules govern the section:

  • PostgreSQL-native names stay out of the contract and live in provenance instead. A consumer reading profile.json should not have to know that reltuples is an estimate or that n_distinct is negative when it means a fraction. The native encodings are recorded rather than discarded — a number without its derivation is not auditable. A test asserts reltuples, n_distinct, null_frac, attnum and relkind appear in no contract field name.
  • Nothing is invented. Where PostgreSQL has no estimate the contract carries None and the fan-out says insufficient_statistics. n_distinct = 0 means "no estimate", not zero distinct values — resolving it as a count would put a zero in every fan-out denominator.

Token domains follow foreign-key chains to their root. A child column borrows its parent's domain so both sides tokenize alike, but with A.x → B.y and B.y → C.z, resolving one hop would have A.x tokenize under B.y while B.y tokenized under C.z — making the A-to-B join invisible, the exact failure domains exist to prevent. Cycles terminate; self-referential keys are legal.

Fan-out excludes nulls from the numerator — a null foreign key references no parent, and counting those rows inflates every estimate on a nullable key. A composite key uses the most-null column's fraction, since PostgreSQL has no joint null fraction, so the estimate errs high; that direction is deliberate, because an under-reported hot parent is otherwise discovered during the migration.

A composite key is never estimated from the product of its columns. user_id has 500 distinct values and placed_at 2500, but orders are not placed at independently random times — the real pair count is 4200, not 1,250,000. Only extended statistics covering exactly the column set count, matched as a set rather than a sequence.

p99 comes from the most-common-value frequencies, which is what makes the hot-parent shape survive tokenization. The fixture's orders.user_id has a mean of 10 and a p99 of 100: sizing a migration on the mean alone would be wrong by an order of magnitude, which is why the field exists. Where the MCV list is shorter than the percentile's rank, the last MCV's count is returned as an upper bound.

Verification: 253 tests pass, --check-safety clean, ruff check clean, 3.9 grammar verified. A planted email from the statistics fixture is proven absent from the normalized records.

Turns collected catalog rows into the profile contract. Two rules govern the
section: PostgreSQL-native names stay out of the contract and live in
provenance instead, and nothing is invented -- where PostgreSQL has no
estimate the contract carries None, because a plausible number is worse than
a gap and a gap gets investigated.

Token domains follow foreign-key chains to their root. Resolving only one hop
would have A.x tokenize under B.y while B.y tokenized under C.z, making the
A-to-B join invisible: the exact failure the domains exist to prevent. Cycles
terminate, since self-referential keys are legal.

Fan-out excludes nulls from the numerator; a null foreign key references no
parent. A composite key is estimated only from extended statistics covering
exactly its column set, never from the product of its columns' distinct
counts -- foreign-key columns are almost never independent, and multiplying
would understate fan-out by orders of magnitude.

p99 comes from the most-common-value frequencies, which is what makes the
hot-parent shape survive tokenization. The fixture's orders.user_id has a
mean of 10 and a p99 of 100; sizing a migration on the mean alone would be
wrong by an order of magnitude.

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
@roachitect-aman
roachitect-aman merged commit 726c654 into main Aug 25, 2026
7 checks passed
@roachitect-aman
roachitect-aman deleted the task-8-normalization branch August 25, 2026 01:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant