Tokenize statistic values with keyed HMAC - #7
Merged
Conversation
Catalog statistics embed literal customer values: pg_stats most-common values and histogram bounds are actual rows. This replaces each one with an HMAC-SHA-256 token. The property that has to survive is equality within a domain. A foreign key is only visible in the profile if the same value on the child and the parent tokenizes identically, and skew is only visible if a hot value stays a single distinct token. Everything here exists to preserve that and nothing more -- no ordering, no length, no prefix. The type name steers canonicalization but is never hashed, because PostgreSQL permits a foreign key across int4 and int8 and folding the type into the material would break the equality the tokens exist to preserve. Canonicalization is opt-in by type: numerics through Decimal.normalize so 42, 42.0 and 4.2e1 agree, uuid through uuid.UUID so case and hyphenation agree. Text is excluded -- "0001" and "1" are different strings, and collapsing them would merge two most-common values into one token. A uuid value's token is reshaped as a uuid. The profile is meant to be replayed into a CockroachDB schema for sizing, and a hex string in a uuid column would force the migration team to retype it, at which point the shape under test is no longer the shape being migrated. The version bits are left as digest bits rather than faked as v4. The key lives in a Tokenizer whose repr redacts it, is read only from the environment with no default, and is rejected below 16 characters. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task 7 of the MVP plan.
Catalog statistics embed literal customer values —
pg_statsmost-common values and histogram bounds are actual rows. This replaces each with an HMAC-SHA-256 token.The property that has to survive is equality within a domain. A foreign key is only visible in the profile if the same value on the child and the parent tokenizes identically, and skew is only visible if a hot value stays a single distinct token. Everything here exists to preserve that and nothing more — no ordering, no length, no prefix.
The type name steers canonicalization but is never hashed. PostgreSQL permits a foreign key across
int4andint8; folding the type into the material would break exactly the equality the tokens exist to preserve. Canonicalization is opt-in by type: numerics throughDecimal.normalize()so42,42.0and4.2e1agree,uuidthroughuuid.UUIDso case and hyphenation agree. Text is deliberately excluded —"0001"and"1"are different strings, and collapsing them would merge two most-common values into one token and corrupt the frequency it carries.A uuid value's token is reshaped as a uuid. The profile is meant to be replayed into a CockroachDB schema for sizing; a 64-character hex string in a uuid column would force the migration team to retype it, at which point the shape under test is no longer the shape being migrated. Version and variant bits are left as digest bits rather than faked as v4 — any 128-bit value is a valid uuid to both PostgreSQL and CockroachDB, and a token advertising itself as random would be a lie about where it came from.
Key handling. Read only from
DBPROFILER_TOKEN_KEY, never fromargv, with no default — a default would tokenize every deployment identically. Held by aTokenizerwhosereprredacts it, rather than a module global that would land in the first traceback. Rejected below 16 characters; neither rejection message echoes the value. An explicitly passed environment is never topped up fromos.environ.token_domainjoins schema/table/column with NUL rather than dots, because PostgreSQL permits a dot inside a quoted identifier and a dotted join would let one column's values impersonate another's.Verification: 215 tests pass,
--check-safetyclean,ruff checkclean, 3.9 grammar verified.