Repository navigation
Merged
Conversation
…ted file repeats no persona: a placeholder is held by at least ten times the field's records per value; each generated part after the first pairs every surname with another persona's given name Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012o8JNBKSY1aREPbMQDG8g2
… one rerun before an increment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012o8JNBKSY1aREPbMQDG8g2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
A real phone value was read as a placeholder on large files. A value of a compared field held by 64+ customers, whose pairs outnumber all the others', was erased as a placeholder. On a file of a million records, every value of a 4-digit phone is held by about a hundred records, so the field was erased wholesale, and phone agreement stopped counting.
Run._placeholdersnow also asks that a placeholder be held by at leastSPREAD= 10 times the field's records per value (filled ÷ distinct).PLACEHOLDERSgoes from 2 to 3: a workspace run before this PR asks for one rerun before its next increment, so its stored records and a batch are told placeholders the same way.Generated files past 50,000 entities repeated every persona.
er.auto._partsgenerates in parts of 50,000 entities. Each part drew from the same 60,000 personas, so at a million records each full name, in its prefecture, belonged to about 15 unrelated entities. The generator's own rule forbids exactly these clones, which no matcher can tell apart. This is what made keeping the phones look like it added false pairs: same name and same postal area, a different phone.er.auto.part_persons: the first part draws from the personas as they are, so files of one part (train, val, the tests) are byte-identical.Results
Medium and hard files of 1M records, regenerated without clones (seed 0,
config/final.toml). Pairs listed by rules or keys; "false" excludes pairs of one household.Tests:
test_a_short_phone_held_by_many_records_each_is_kept_and_a_dummy_far_past_them_is_still_read_as_missingtest_the_parts_of_a_large_generated_file_draw_no_persona_twice: two parts of 3,000 share 118 full names on main and 2 here.Full suite: 303 passed on 3.14 (
-n auto).Checklist (CONTRIBUTING.md)
shortcut: …), and every store that grows says what bounds it (rules 3 and 4)🤖 Generated with Claude Code