Skip to content

About

Config-driven entity matching & resolution for Japanese customer records (Splink, Fellegi-Sunter, Optuna)

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Entity resolution for Japanese customer records

Finds which customer records belong to the same person or company, and gives each of them one ID. It reads names in katakana, addresses in any format, the last four phone digits, customer IDs and birth dates. It needs no labels to run.

flowchart LR
  A["your records"] --> S["1 · Standardize<br/>one way to write each fact"]
  S --> R["2 · Rules<br/>settle the certain cases"]
  R --> C["3 · Candidates<br/>list the pairs worth comparing"]
  C --> F["4 · Score<br/>weigh the evidence, in bits"]
  F --> G["5 · Cluster<br/>linked records become one"]
  G --> O["one ID per person<br/>or company"]
Loading
  1. Standardize. The same fact is written many ways: full-width digits, kana variants, addresses with or without the prefecture or the building. Each is rewritten in one form.
  2. Rules. Pairs that agree on strong evidence, such as the same customer ID and the same name, are matched at once. Rules are cheap and easy to audit.
  3. Candidates. Comparing every pair is impossible on a large file: a million records make 500 billion pairs. Shared keys, such as the same address and phone, or a name one letter apart in the same postal area, list about 5 pairs per record instead.
  4. Score. Each field adds evidence for or against, in bits. Each bit doubles the odds that the two records are one entity. A pair is accepted at 6 bits or more, if a name was compared; without one, a person decides.
  5. Cluster. Records linked by accepted pairs become one entity. If A matches B and B matches C, all three are one.

One pair, scored field by field: chapter 5.

How well it works

These numbers come from synthetic records built from real Japanese names, companies and addresses, where the truth is known:

  • It finds almost half the duplicates: 48.2% of the true pairs on test (95% interval 46.2–50.2%); the rules alone find 9.6%. Test is a split never used for any choice.
  • The data is hard on purpose. About 1% of true pairs are direct matches (the same customer ID and name, or the same name, address code and phone), and 11.3% share nothing usable, so no method finds them all (data).
  • It merged no two unrelated customers: 0 false merges on test, so the rate is at most 0.26%. Two people of one household (a spouse, a parent or a sibling at one address) can score as high as a duplicate; their pairs are counted apart, 45 on test. On val, people see 27 of the 32 such pairs it accepted labelled "maybe one household" (usage).
  • It tells persons from companies better than the file's flag: on generated files with one name column for both, and company names held out of its name model's training, it calls a kind where it is sure and takes the flag's elsewhere, flagging where they disagree. It called 180 to 1,641 persons companies, against 308 to 2,398 by the flag alone; recall is 42.6–46.1%, against 39.0–45.5% by the flag, with no more false merges (chapter 7).
  • A million records take 53 seconds, with peak memory 6.64 GB under an 8 GB limit, and about 15 seconds before that for the reader to read the file as it comes. Every step's work grows about linearly with the file.

Real data may differ, and only a run on it can tell (limits).

Use it

./setup_venv.sh
.venv/bin/nayose create workspace customers                                        # workspaces/customers: one matching job
.venv/bin/nayose load records customers.csv --workspace workspaces/customers       # CSV, TSV or Parquet: copied in, read as it comes, then described
# no file yet: nayose generate records --records 10000 --difficulty medium --workspace workspaces/customers
# repairs it would have to guess (a row whose fields don't line up, a garbled value) wait for you: nayose accept repairs …
# a field it can't find: the check prints the line, nayose set columns id=CUSTOMER_NO --workspace workspaces/customers
.venv/bin/nayose resolve records --workspace workspaces/customers
.venv/bin/nayose open ui                                                           # or all of it in a browser, on this machine only

After an update (git pull), start nayose open ui again: it reads its commands once, when it starts.

On Windows, in PowerShell: powershell -ExecutionPolicy Bypass -File setup_venv.ps1, then .venv\Scripts\nayose in place of .venv/bin/nayose.

The workspace holds your file (inputs/, copied in, and as it was read: each value in its own encoding, control bytes and placeholders fixed, what would be a guess proposed for you to accept), its mapping (columns.toml), gold.parquet (each record and its entity's ID, and each one's kind: 1 person, 2 company, 0 not sure), review.parquet (pairs for a person to decide or audit), kind_review.parquet (records for a person to decide person or company), qa.md (checks that need no labels) and report.txt (the run on one page with no record's value in it, to send when asking for help). New records join the run later (nayose add records). A new workspace joins nothing before people check. People decide the pairs (nayose list scored, show records for their full rows, decide pair, apply decisions), with a details file's rows about each record in its full row if one is loaded (nayose load details: any number of rows per record ID, read from the file itself, never by matching) and answer person or company (list kinds, answer kind, apply answers); they can let pairs merge by conditions they set (nayose set merging: points, rules, fields), or by a trust line (nayose lower trust) that comes down one score range at a time as each range's random sample proves clean. Deciding pairs and merging by conditions run nothing again; lowering the trust line and applying answers match again; nayose serve inbox takes batches, decisions and answers from an inbox, one file at a time. nayose (名寄せ, gathering one customer's records) reads as a sentence, verb then noun: every step is in usage, every command in commands.

Read next

To Read
run it on your records usage, and before a run on your file
look up a command commands
change the code: the rules, and what checks each contributing
see every step on a small file, with charts the demo
see what each optimization saves the optimizations notebook
see how it is built architecture
understand each method, and why it was chosen the guide, seven chapters
look up a term the glossary
check the numbers results · performance · choose report · kind report · benchmark
know where the data comes from data
learn from what went wrong lessons

Limits

  • Measured on synthetic data only. Test comes from the same generator as the data the choices were made on. Other seeds, emptier fields, or a prior off by a factor of two keep recall between 41.7% and 49.3%; twice the typos lowers it to 33.4%. None of these made a false merge (benchmark). Real data has not been measured.
  • False pairs appear on files fitted on themselves. On the generated files of the cost benchmark, the run joined 2 false pairs at 12,532 records, through 1 false link, and 4 at 74,959, through 3. Each is a model link between 8.2 and 10.1 bits (the threshold is 8), between near-namesakes with a kana typo who share an address or a phone. The model expects them: the sum of 1 − match probability over its links, which every run reports without labels, is 0.84 at 12,532 records, 4.38 at 74,959, and 30.11 of 38,786 links at a million.
  • A company that looks like its representative. A company record under its representative's name, at the representative's address and phone and flagged person, looks like the person's own record in every column. Nothing tells them apart, so it can merge into the person (chapter 7).
  • Close names rely on other keys. A Jaro–Winkler level guarantees no shared key, so two close but unequal names (or company names) are compared only if another key lists them. By the model's own bound, up to 13.2% of matches at a million records rest on such patterns; on labelled train, every pair that could reach the threshold shares a key.
  • Machine-read kana. Given-name readings come from SudachiPy, and some differ from the intended reading.

About

Config-driven entity matching & resolution for Japanese customer records (Splink, Fellegi-Sunter, Optuna)

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages