Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ Writes each table as Parquet under a temporary name, with the reports (9), and `

### 9 · Report, for people · `O(review pairs)` + `O(N)` in SQL

`Run.report` builds, and step 8 writes with the run's files: `qa.md`, the label-free checks; `report.txt`, the run on one page with no record's value in it, which can leave the machine that holds the records; `review.parquet`, with the evidence for each pair; and, on a full run that tells persons from companies, `kind_review.parquet`, the records for people to answer person or company ([usage](usage.md#6--review-and-decide)). The review sheet's *decide* rows are every pair of clusters whose best link scored in the threshold rule's safety margin: from the `[threshold]` grid point `margin_steps` below the threshold (4 bits) up to it, and every pair above it with no name compared. Its *audit* rows are about 50 of the pairs the run accepted (`matches`), up to 4 bits above it. With a trust line (`report.trust_sheet`), the *decide* rows are a random sample of the next range down instead, `[trust] checks` less those decided, and the *audit* rows `watch` spot-checks above the line.
`Run.report` builds, and step 8 writes with the run's files: `qa.md`, the label-free checks; `report.txt`, the run on one page with no record's value in it, which can leave the machine that holds the records, headed by what limits what the run finds (`report.diagnosis`: [usage](usage.md#7--when-something-is-wrong)); `review.parquet`, with the evidence for each pair; and, on a full run that tells persons from companies, `kind_review.parquet`, the records for people to answer person or company ([usage](usage.md#6--review-and-decide)). The review sheet's *decide* rows are every pair of clusters whose best link scored in the threshold rule's safety margin: from the `[threshold]` grid point `margin_steps` below the threshold (4 bits) up to it, and every pair above it with no name compared. Its *audit* rows are about 50 of the pairs the run accepted (`matches`), up to 4 bits above it. With a trust line (`report.trust_sheet`), the *decide* rows are a random sample of the next range down instead, `[trust] checks` less those decided, and the *audit* rows `watch` spot-checks above the line.

- **Why:** the margin holds pairs that were clean on cross-validated train, held back in case of noise: on the synthetic val split, 55 pairs from 4 to 6 bits, none of them false; one grid step lower, at 2 bits, val's first false pairs appear. So people decide them. Without labels, the audit rows and the largest clusters are how a person catches over-merging.
- **Code:** `Run.report` → `report.qa` (`report.fill_rates`, `match.review_floor`, `report.review_sheet`, `report.carry_over`, `match.pattern_coverage`), `report.report_text`, `kind.kind_review_sheet` (its rows picked in DuckDB, so only they reach Python), `io.markdown`
Expand Down
2 changes: 1 addition & 1 deletion docs/diagrams/full-run.puml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ package "every step after load reads records" as after_load {
rectangle "<b>7 · Cluster</b>: merge the rules' and the model's pairs at the trust line (none set: every rule pair, and the model's at its threshold with a name compared), or, with the workspace's merge conditions, the pairs that meet them; people's matches, less their no_match; join them into entities, from single records\n<i>O(C + E·d + N)</i>\n<size:10><color:#65717E>Run._load_decisions · folder.latest_decisions · folder.parts · Run.cluster · Run._accept · Run.scored · match.named_sql · Run._links · results.merge_where</color></size>\n<size:10><color:#65717E>cluster.merge · Run._gold · run.customer_kinds · Run._expected_false · Run._alone · Run._line · Run._sized · match.review_floor · results.own_merging</color></size>" as cluster
rectangle "<b>7b · Probe</b>: the matches no key lists: pairs drawn within groups matches share (a phone, a postal area), scored; those at the line the likely links don't join, counted and scaled\n<i>O(N + P·K + L)</i>\n<size:10><color:#65717E>Run.probe · probe.missed · Run._sized</color></size>" as probe
rectangle "<b>8 · Write</b>: save the answer, the pairs at 0 points or more with each comparison's level (scored/), the state for increments and the reports (9, built first); the old run goes only once all is written\n<i>O(N + C)</i>\n<size:10><color:#65717E>Run.write · config.generator_dict · Run._links · Run.scored · match.named_sql · folder.next_part · folder.stage · folder.finish_writes · folder.history · folder.brief · Run._log · folder.log · io.code_version · io.replace_text · io.patiently</color></size>\n<size:10><color:#65717E>the round its method is in: folder.round_ran · folder.rounds · folder.method · config.toml_of · Run._alone · Run._sized</color></size>" as write
rectangle "<b>9 · Report</b>: the checks and the sheets for people, what they typed in and haven't given back kept, saved with the run (8)\n<i>O(review pairs) + O(N) in SQL</i>\n<size:10><color:#65717E>Run.report · report.qa · report.trust_sheet · review_state._trust_links · review_state.applied · report.fill_rates · match.review_floor · match.named_sql · report.review_sheet · report.households · report.carry_over</color></size>\n<size:10><color:#65717E>match.pattern_coverage · match.levels · io.markdown · kind.kind_review_sheet · report.carry_kinds</color></size>\n<size:10><color:#65717E>report.report_text · match.scaled_prior · source.filled · match.parameters · config.sources_text · io.code_version · io.read_parquet</color></size>" as report
rectangle "<b>9 · Report</b>: the checks and the sheets for people, what they typed in and haven't given back kept, saved with the run (8)\n<i>O(review pairs) + O(N) in SQL</i>\n<size:10><color:#65717E>Run.report · report.qa · report.trust_sheet · review_state._trust_links · review_state.applied · report.fill_rates · match.review_floor · match.named_sql · report.review_sheet · report.households · report.carry_over</color></size>\n<size:10><color:#65717E>match.pattern_coverage · match.levels · io.markdown · kind.kind_review_sheet · report.carry_kinds</color></size>\n<size:10><color:#65717E>report.report_text · match.scaled_prior · source.filled · match.parameters · config.sources_text · io.code_version · io.read_parquet</color></size>\n<size:10><color:#65717E>report.diagnosis · report.feature_path · match._ranked · match.search_floor</color></size>" as report
}
rectangle "<b>4b · Keys</b>: with [keys], the keys learned on this file: the pairs the model accepts, found by sampling every way two records agree, listed by the fewest pairs. Without, the method's generators, but one whose agreement weighs nothing and whose sampled pairs at the review floor other keys or rules all list\n<i>O(A·N + f·N·b + K·T + K'·N)</i>\n<size:10><color:#65717E>Run.keys · keys.learn · keys.idle · match.review_floor · source.known</color></size>" as keys
rectangle "<b>7c · Gap</b>: what the keys miss against comparing every pair: the keys step's sample of records (only those that could reach the floor), each paired with every record sharing a value or a near one, scored; the pairs at the search floor, the review floor, the line and 20 bits no key or rule listed, counted and scaled\n<i>O(f·N·b)</i>\n<size:10><color:#65717E>Run.gap · keys.gap</color></size>" as gap
Expand Down
2 changes: 1 addition & 1 deletion docs/diagrams/full-run.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
18 changes: 17 additions & 1 deletion docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ None of these runs anything: they read the workspace's files.
| `proposed/` | the model's pair below the trust line | both IDs, `match_weight`, `match_probability`, `sources`; empty with no trust line | the pairs waiting for people, or for the line to come down |
| `scored/` | pair at 0 points or more, and every rule's pair | both IDs, `match_weight`, `match_probability`, `sources` (none: a rule alone found it), `rule`, `named`, `keep`, and `gamma_<comparison>`, the level each comparison reached | list, filter and review any pair without a run; merge by your own conditions ([section 6](#merge-by-your-own-conditions)). A workspace whose run wrote none gets it once, on first use, from the pairs that run stored, scored again |
| `qa.md` | | the summary (with `missed`: the matches no pair key lists, sampled within `[probe]` groups, as estimated by the probe); each field's fill rate (flagged under 5%), the share that looks right and the share of its most common value; what was turned off; each step's Big-O, time and memory; pairs by weight; cluster sizes; the 10 largest clusters; match patterns no key guarantees; the review queue; records and customers by kind; clusters holding both a sure person and a sure company; the oversized blocks and what was decided for each | check the run without labels |
| `report.txt` | | the run on one page of plain text, with no record's value in it: what ran (the code, the tracked files changed in this copy, the pipeline, the machine's limits), the summary, the fields, the columns no field reads (filled, distinct, the field each looks most like), scores and links by weight with the false links expected, the commonest patterns of levels below and from the review floor, cluster sizes, each step's time and memory (the fit's own sample and the u measured on the file apart; the gate's samples), where each step's time went in the string comparisons (per function: the wall seconds while it ran, the threads that ran it at once, the values it compared, per value), the pairs each EM session learned from (with fit em `choose`, each session judged: its pairs, the share of its matches agreeing on the given name, kept or dropped and why) and the model's m, u and bits per level, the costliest blocks (a full run), the match patterns no key pairs | send it to whoever helps with the run ([section 7](#7--when-something-is-wrong)) |
| `report.txt` | | the run on one page of plain text, with no record's value in it: what ran (the code, the tracked files changed in this copy, the pipeline, the machine's limits), DIAGNOSIS (what limits what the run finds, below), the summary, the fields, the columns no field reads (filled, distinct, the field each looks most like), scores and links by weight with the false links expected, the commonest patterns of levels below and from the review floor, cluster sizes, each step's time and memory (the fit's own sample and the u measured on the file apart; the gate's samples), where each step's time went in the string comparisons (per function: the wall seconds while it ran, the threads that ran it at once, the values it compared, per value), the pairs each EM session learned from (with fit em `choose`, each session judged: its pairs, the share of its matches agreeing on the given name, kept or dropped and why) and the model's m, u and bits per level, the costliest blocks (a full run), the match patterns no key pairs | send it to whoever helps with the run ([section 7](#7--when-something-is-wrong)) |
| `review.parquet` | pair for people | `why` (`decide` or `audit`); an empty `decision`; both records' compared fields; per comparison, the level reached and its bits; `accepted`, whether the run joined the pair; `household`, whether it may be two people of one household ([section 6](#6--review-and-decide)) | decide the next range below the trust line and audit above it, or, with no line, decide the pairs just below the threshold and those above it with no name compared, and audit those just above ([section 6](#6--review-and-decide)) |
| `kind_review.parquet` | record for people | `why` (`decide` or `measure`), `reason` in words, `engine_says`, `name_says`, `flag` and `birth_date` if the file has them, the name; `engine` and `engine_p` (the engine's kind and its probability of a company), `name_p`, `drawn`; an empty `kind` | answer 1 person, 2 company or 0 can't tell ([section 6](#6--review-and-decide)) |
| `decisions/` | decided pair | both IDs and `decision`; `why` (`decide`, `audit`, or `split` from `split entity`) when there is one; `round`, the round it was decided in; one part per file people gave back | what people decided; a full run applies it again |
Expand Down Expand Up @@ -422,6 +422,22 @@ Wrong audit pairs are also the evidence for changing the matching. In `qa.md`, a

## 7 · When something is wrong

`report.txt` starts with DIAGNOSIS: what limits what the run finds, at most eight lines, the most pairs at stake first,
each with a number and the command that fixes it where there is one (`--workspace …`: the run's workspace). It names:

- a compared field filled in under 30% of the records (a person's name and a company's aside: each kind fills one), the
pairs below the review floor its comparison can't compare, and a column no field reads whose values look like the
field, with the `nayose set feature` that adds it as a fallback;
- EM sessions of the run's own fit whose matches mostly share no given name: they learned households (`nayose choose em`);
- pairs below the review floor that agree exactly on a name and another field (the birth date, the phone, the
address, …), by pattern, and the true pairs the model itself expects among all the pairs below the review floor;
- levels that learned nothing (m = u, 0 bits) yet hold pairs (`nayose try sample`);
- pairs at the review floor no key lists (the gap's estimate and its interval), and oversized blocks skipped whose best
sampled pair came within 2 bits of the search floor (`nayose describe method`);
- EM sessions that learned from under 200 pairs (more records: `nayose choose em … --records`, or `nayose try sample`).

What is fine is left out.

What the run handles by itself, and reports in its summary:

| Found | Done |
Expand Down
Loading
Loading