Skip to content

Tokenize statistic values with keyed HMAC - #7

Merged
roachitect-aman merged 1 commit into
mainfrom
task-7-tokenization
Aug 25, 2026
Merged

Tokenize statistic values with keyed HMAC#7
roachitect-aman merged 1 commit into
mainfrom
task-7-tokenization

Conversation

@roachitect-aman

Copy link
Copy Markdown
Contributor

Task 7 of the MVP plan.

Catalog statistics embed literal customer values — pg_stats most-common values and histogram bounds are actual rows. This replaces each with an HMAC-SHA-256 token.

The property that has to survive is equality within a domain. A foreign key is only visible in the profile if the same value on the child and the parent tokenizes identically, and skew is only visible if a hot value stays a single distinct token. Everything here exists to preserve that and nothing more — no ordering, no length, no prefix.

The type name steers canonicalization but is never hashed. PostgreSQL permits a foreign key across int4 and int8; folding the type into the material would break exactly the equality the tokens exist to preserve. Canonicalization is opt-in by type: numerics through Decimal.normalize() so 42, 42.0 and 4.2e1 agree, uuid through uuid.UUID so case and hyphenation agree. Text is deliberately excluded — "0001" and "1" are different strings, and collapsing them would merge two most-common values into one token and corrupt the frequency it carries.

A uuid value's token is reshaped as a uuid. The profile is meant to be replayed into a CockroachDB schema for sizing; a 64-character hex string in a uuid column would force the migration team to retype it, at which point the shape under test is no longer the shape being migrated. Version and variant bits are left as digest bits rather than faked as v4 — any 128-bit value is a valid uuid to both PostgreSQL and CockroachDB, and a token advertising itself as random would be a lie about where it came from.

Key handling. Read only from DBPROFILER_TOKEN_KEY, never from argv, with no default — a default would tokenize every deployment identically. Held by a Tokenizer whose repr redacts it, rather than a module global that would land in the first traceback. Rejected below 16 characters; neither rejection message echoes the value. An explicitly passed environment is never topped up from os.environ.

token_domain joins schema/table/column with NUL rather than dots, because PostgreSQL permits a dot inside a quoted identifier and a dotted join would let one column's values impersonate another's.

Verification: 215 tests pass, --check-safety clean, ruff check clean, 3.9 grammar verified.

Catalog statistics embed literal customer values: pg_stats most-common
values and histogram bounds are actual rows. This replaces each one with an
HMAC-SHA-256 token.

The property that has to survive is equality within a domain. A foreign key
is only visible in the profile if the same value on the child and the parent
tokenizes identically, and skew is only visible if a hot value stays a
single distinct token. Everything here exists to preserve that and nothing
more -- no ordering, no length, no prefix.

The type name steers canonicalization but is never hashed, because
PostgreSQL permits a foreign key across int4 and int8 and folding the type
into the material would break the equality the tokens exist to preserve.
Canonicalization is opt-in by type: numerics through Decimal.normalize so
42, 42.0 and 4.2e1 agree, uuid through uuid.UUID so case and hyphenation
agree. Text is excluded -- "0001" and "1" are different strings, and
collapsing them would merge two most-common values into one token.

A uuid value's token is reshaped as a uuid. The profile is meant to be
replayed into a CockroachDB schema for sizing, and a hex string in a uuid
column would force the migration team to retype it, at which point the shape
under test is no longer the shape being migrated. The version bits are left
as digest bits rather than faked as v4.

The key lives in a Tokenizer whose repr redacts it, is read only from the
environment with no default, and is rejected below 16 characters.

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
@roachitect-aman
roachitect-aman merged commit fe3a097 into main Aug 25, 2026
7 checks passed
@roachitect-aman
roachitect-aman deleted the task-7-tokenization branch August 25, 2026 01:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant