Skip to content

A 4-digit phone on a large file is no placeholder; a large generated file repeats no persona - #146

Merged
mvsbm merged 2 commits into
mainfrom
score-fix
Nov 15, 2026
Merged

mvsbm merged 2 commits into
mainfrom
score-fix

Conversation

@mvsbm

@mvsbm mvsbm commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

What and why

A real phone value was read as a placeholder on large files. A value of a compared field held by 64+ customers, whose pairs outnumber all the others', was erased as a placeholder. On a file of a million records, every value of a 4-digit phone is held by about a hundred records, so the field was erased wholesale, and phone agreement stopped counting.

  • Run._placeholders now also asks that a placeholder be held by at least SPREAD = 10 times the field's records per value (filled ÷ distinct).
    • A dummy far past every real value (9999 on 3,000 records, beside 100 values on 200 each) is still read as missing.
    • The real values are kept.
    • Customer IDs and small files are unchanged: their records per value is near 1, so the count stays at the budget's oversized-block size.
  • The note names the count used.
  • PLACEHOLDERS goes from 2 to 3: a workspace run before this PR asks for one rerun before its next increment, so its stored records and a batch are told placeholders the same way.

Generated files past 50,000 entities repeated every persona. er.auto._parts generates in parts of 50,000 entities. Each part drew from the same 60,000 personas, so at a million records each full name, in its prefecture, belonged to about 15 unrelated entities. The generator's own rule forbids exactly these clones, which no matcher can tell apart. This is what made keeping the phones look like it added false pairs: same name and same postal area, a different phone.

  • er.auto.part_persons: the first part draws from the personas as they are, so files of one part (train, val, the tests) are byte-identical.
  • Each later part shuffles given names, with their sex and age, among the surnames, by the part's seed.
  • The rehearsal note, the commands' descriptions and data.md now say what still repeats past the reference: company names (31,596).

Results

Medium and hard files of 1M records, regenerated without clones (seed 0, config/final.toml). Pairs listed by rules or keys; "false" excludes pairs of one household.

file true 4+ recall at 4+ true 6+ false 4+ false 6+
medium 1M main 63,971 28.9% 63,838 35 35
medium 1M this PR 98,796 44.6% 95,011 332 209
hard 1M main 39,139 17.6% 31,802 50 16
hard 1M this PR 41,341 18.6% 36,472 59 32
  • The 209 false pairs at 6+ on medium:
    • 119 have no name on either side (same address, same phone). A pair with no name compared always goes to people and never merges on its points.
    • Most of the rest have the same name and the same 4-digit phone in another area.
  • "Same name + same postal code, phone differs": 7,081 true pairs to 13 false.
  • Unchanged: train, val and a medium 200k file (no crowded phone there). Same pairs; scores equal to 5e-15.
  • Wall clock: medium 1M 31.7 s → 34.6 s; hard 1M 31.7 s → 33.1 s. More phone pairs are listed and scored.

Tests:

  • test_a_short_phone_held_by_many_records_each_is_kept_and_a_dummy_far_past_them_is_still_read_as_missing
  • test_the_parts_of_a_large_generated_file_draw_no_persona_twice: two parts of 3,000 share 118 full names on main and 2 here.

Full suite: 303 passed on 3.14 (-n auto).

Checklist (CONTRIBUTING.md)

  • Reviewed before merging: a review pass on this pull request's changes (rule 10)
  • Unhappy paths tested at the real size: two at once, an input twice, the real data size, a lost server (rule 9)
  • Every new limit names itself (shortcut: …), and every store that grows says what bounds it (rules 3 and 4)
  • Docs and diagrams describe the code as it is

🤖 Generated with Claude Code

Marvin Ma and others added 2 commits November 16, 2026 08:25
…ted file repeats no persona: a placeholder is held by at least ten times the field's records per value; each generated part after the first pairs every surname with another persona's given name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012o8JNBKSY1aREPbMQDG8g2
… one rerun before an increment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012o8JNBKSY1aREPbMQDG8g2
@mvsbm
mvsbm merged commit 9ceba69 into main Nov 15, 2026
6 of 7 checks passed
@mvsbm
mvsbm deleted the score-fix branch November 15, 2026 23:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant