From f437c799500696653700b234f2b133120bcb508f Mon Sep 17 00:00:00 2001 From: Marvin Ma Date: Mon, 16 Nov 2026 03:03:44 +0900 Subject: [PATCH 1/2] report.txt starts with DIAGNOSIS: what limits what the run finds, the most pairs at stake first, at most eight lines, each with a number and the command that fixes it; no record's value in it - er.report.diagnosis, in order of pairs at stake: EM sessions of the run's own fit that learned households (nayose choose em); compared fields under 30% filled, one comparison a line, with the pairs below the review floor it can't compare and an unmapped column that looks like the field (nayose set feature with it as a fallback); pairs below the review floor agreeing exactly on a name and another field, by pattern, with the true pairs the model expects below the floor; levels that learned nothing yet hold pairs (nayose try sample); the gap at the review floor and skipped oversized blocks that came within 2 bits of the search floor (nayose describe method); EM sessions of under 200 pairs, last. Items with nothing found are left out. - er.match.train_fs: every EM session of the method's own fit (in turn, or kept apart) is judged as fit em "choose" judges it, in model.json "er" em_given (session, pairs, given-name share); er.match.judged split from chosen. - Docs: usage section 7 lists the items; architecture names report.diagnosis; full-run.puml's report box names its functions. - Tests: the method's own sessions, in turn and kept apart, judged on a household frame (phone under half, birth over); on the population run, each item constructed in turn (a sparse field with a look-alike column, a household session, three pairs moved below the floor, a level that learned nothing, the gap and a skipped block, a thin session), each line and its fix said, the most pairs first, at most eight lines; the report-holds-no-value test passes. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_012o8JNBKSY1aREPbMQDG8g2 --- docs/architecture.md | 2 +- docs/diagrams/full-run.puml | 2 +- docs/diagrams/full-run.svg | 2 +- docs/usage.md | 18 ++++++- er/match.py | 61 ++++++++++++--------- er/report.py | 104 ++++++++++++++++++++++++++++++++++-- tests/test_cli.py | 34 ++++++++++++ tests/test_scoring.py | 8 +++ 8 files changed, 198 insertions(+), 33 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index d0acbe2..6a868e5 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -208,7 +208,7 @@ Writes each table as Parquet under a temporary name, with the reports (9), and ` ### 9 · Report, for people · `O(review pairs)` + `O(N)` in SQL -`Run.report` builds, and step 8 writes with the run's files: `qa.md`, the label-free checks; `report.txt`, the run on one page with no record's value in it, which can leave the machine that holds the records; `review.parquet`, with the evidence for each pair; and, on a full run that tells persons from companies, `kind_review.parquet`, the records for people to answer person or company ([usage](usage.md#6--review-and-decide)). The review sheet's *decide* rows are every pair of clusters whose best link scored in the threshold rule's safety margin: from the `[threshold]` grid point `margin_steps` below the threshold (4 bits) up to it, and every pair above it with no name compared. Its *audit* rows are about 50 of the pairs the run accepted (`matches`), up to 4 bits above it. With a trust line (`report.trust_sheet`), the *decide* rows are a random sample of the next range down instead, `[trust] checks` less those decided, and the *audit* rows `watch` spot-checks above the line. +`Run.report` builds, and step 8 writes with the run's files: `qa.md`, the label-free checks; `report.txt`, the run on one page with no record's value in it, which can leave the machine that holds the records, headed by what limits what the run finds (`report.diagnosis`: [usage](usage.md#7--when-something-is-wrong)); `review.parquet`, with the evidence for each pair; and, on a full run that tells persons from companies, `kind_review.parquet`, the records for people to answer person or company ([usage](usage.md#6--review-and-decide)). The review sheet's *decide* rows are every pair of clusters whose best link scored in the threshold rule's safety margin: from the `[threshold]` grid point `margin_steps` below the threshold (4 bits) up to it, and every pair above it with no name compared. Its *audit* rows are about 50 of the pairs the run accepted (`matches`), up to 4 bits above it. With a trust line (`report.trust_sheet`), the *decide* rows are a random sample of the next range down instead, `[trust] checks` less those decided, and the *audit* rows `watch` spot-checks above the line. - **Why:** the margin holds pairs that were clean on cross-validated train, held back in case of noise: on the synthetic val split, 55 pairs from 4 to 6 bits, none of them false; one grid step lower, at 2 bits, val's first false pairs appear. So people decide them. Without labels, the audit rows and the largest clusters are how a person catches over-merging. - **Code:** `Run.report` → `report.qa` (`report.fill_rates`, `match.review_floor`, `report.review_sheet`, `report.carry_over`, `match.pattern_coverage`), `report.report_text`, `kind.kind_review_sheet` (its rows picked in DuckDB, so only they reach Python), `io.markdown` diff --git a/docs/diagrams/full-run.puml b/docs/diagrams/full-run.puml index 241ffa7..661b10f 100644 --- a/docs/diagrams/full-run.puml +++ b/docs/diagrams/full-run.puml @@ -33,7 +33,7 @@ package "every step after load reads records" as after_load { rectangle "7 · Cluster: merge the rules' and the model's pairs at the trust line (none set: every rule pair, and the model's at its threshold with a name compared), or, with the workspace's merge conditions, the pairs that meet them; people's matches, less their no_match; join them into entities, from single records\nO(C + E·d + N)\nRun._load_decisions · folder.latest_decisions · folder.parts · Run.cluster · Run._accept · Run.scored · match.named_sql · Run._links · results.merge_where\ncluster.merge · Run._gold · run.customer_kinds · Run._expected_false · Run._alone · Run._line · Run._sized · match.review_floor · results.own_merging" as cluster rectangle "7b · Probe: the matches no key lists: pairs drawn within groups matches share (a phone, a postal area), scored; those at the line the likely links don't join, counted and scaled\nO(N + P·K + L)\nRun.probe · probe.missed · Run._sized" as probe rectangle "8 · Write: save the answer, the pairs at 0 points or more with each comparison's level (scored/), the state for increments and the reports (9, built first); the old run goes only once all is written\nO(N + C)\nRun.write · config.generator_dict · Run._links · Run.scored · match.named_sql · folder.next_part · folder.stage · folder.finish_writes · folder.history · folder.brief · Run._log · folder.log · io.code_version · io.replace_text · io.patiently\nthe round its method is in: folder.round_ran · folder.rounds · folder.method · config.toml_of · Run._alone · Run._sized" as write - rectangle "9 · Report: the checks and the sheets for people, what they typed in and haven't given back kept, saved with the run (8)\nO(review pairs) + O(N) in SQL\nRun.report · report.qa · report.trust_sheet · review_state._trust_links · review_state.applied · report.fill_rates · match.review_floor · match.named_sql · report.review_sheet · report.households · report.carry_over\nmatch.pattern_coverage · match.levels · io.markdown · kind.kind_review_sheet · report.carry_kinds\nreport.report_text · match.scaled_prior · source.filled · match.parameters · config.sources_text · io.code_version · io.read_parquet" as report + rectangle "9 · Report: the checks and the sheets for people, what they typed in and haven't given back kept, saved with the run (8)\nO(review pairs) + O(N) in SQL\nRun.report · report.qa · report.trust_sheet · review_state._trust_links · review_state.applied · report.fill_rates · match.review_floor · match.named_sql · report.review_sheet · report.households · report.carry_over\nmatch.pattern_coverage · match.levels · io.markdown · kind.kind_review_sheet · report.carry_kinds\nreport.report_text · match.scaled_prior · source.filled · match.parameters · config.sources_text · io.code_version · io.read_parquet\nreport.diagnosis · report.feature_path · match._ranked · match.search_floor" as report } rectangle "4b · Keys: with [keys], the keys learned on this file: the pairs the model accepts, found by sampling every way two records agree, listed by the fewest pairs. Without, the method's generators, but one whose agreement weighs nothing and whose sampled pairs at the review floor other keys or rules all list\nO(A·N + f·N·b + K·T + K'·N)\nRun.keys · keys.learn · keys.idle · match.review_floor · source.known" as keys rectangle "7c · Gap: what the keys miss against comparing every pair: the keys step's sample of records (only those that could reach the floor), each paired with every record sharing a value or a near one, scored; the pairs at the search floor, the review floor, the line and 20 bits no key or rule listed, counted and scaled\nO(f·N·b)\nRun.gap · keys.gap" as gap diff --git a/docs/diagrams/full-run.svg b/docs/diagrams/full-run.svg index 76079c2..8a8f6ab 100644 --- a/docs/diagrams/full-run.svg +++ b/docs/diagrams/full-run.svg @@ -1 +1 @@ -LLD 1 · A full run: Run.full, its reports written with it (nayose resolve records … --workspace …)inputsevery step after load reads recordsthe workspace (--workspace)the input: the files given, read as they come (inputs/)config/final.toml · config/settings.tomlconfig/kind_names.npz: the name modelapply answers · kind_answers/part-n.parquetdecisions/part-n.parquet2 · Budget: count every key's pairs before any join; flag blocks too big to join blindO(N) per keyRun.budget · candidates.estimate · Run.capacity · Run.spill_room · run.size_bytes3 · Fit: learn the model from the file itself, without labels; measure u on this fileO(n + S·K + P_em·K) + O(N + S·K)Run.fit · match.train_fs · match.scored_umatch.u_basis · match.fit_model · match.fit_sample_sql4 · Rules: join the certain cases, the first rule wins; score each pair with the model; cluster them (they steer the candidates)O(N + ΣP_r·L²) + O(R·K) + O(E·d + N)Run.rules · Run._rules · Run._untrusted · match.rule_edges · match.score_fs · cluster.merge · source.filled · Run._alone · Run._sized5 · Candidates: list the pairs worth scoring, keeping only those that can reach 0; check big blocks on a sampleO(N·L + ΣP_g)Run.candidates · run.listed · candidates.candidates · Run._reaching · match.reaching_sql · match.bound_reads · Run._gate · Run._scored · match.score_fs · match.bound_tests · Run._gated · Run._joinedmatch.search_floor · Run.capacity · Run.spill_room · run.size_bytes · Run._sized · Run._alone · match.rough_reads6 · Score: give every candidate pair that can reach 0 its evidence in bits, the rest left below 0 by a bound, a million at a time, saying how far it is; count the patterns they fall inO(C·K·L²)Run.score · Run._scored · match.score_fs · match.bound_tests · Run._patterns · Run._bands · results.banded · match.named_sql · match.review_floor · Run._line · results.own_merging7 · Cluster: merge the rules' and the model's pairs at the trust line (none set: every rule pair, and the model's at its threshold with a name compared), or, with the workspace's merge conditions, the pairs that meet them; people's matches, less their no_match; join them into entities, from single recordsO(C + E·d + N)Run._load_decisions · folder.latest_decisions · folder.parts · Run.cluster · Run._accept · Run.scored · match.named_sql · Run._links · results.merge_wherecluster.merge · Run._gold · run.customer_kinds · Run._expected_false · Run._alone · Run._line · Run._sized · match.review_floor · results.own_merging7b · Probe: the matches no key lists: pairs drawn within groups matches share (a phone, a postal area), scored; those at the line the likely links don't join, counted and scaledO(N + P·K + L)Run.probe · probe.missed · Run._sized8 · Write: save the answer, the pairs at 0 points or more with each comparison's level (scored/), the state for increments and the reports (9, built first); the old run goes only once all is writtenO(N + C)Run.write · config.generator_dict · Run._links · Run.scored · match.named_sql · folder.next_part · folder.stage · folder.finish_writes · folder.history · folder.brief · Run._log · folder.log · io.code_version · io.replace_text · io.patientlythe round its method is in: folder.round_ran · folder.rounds · folder.method · config.toml_of · Run._alone · Run._sized9 · Report: the checks and the sheets for people, what they typed in and haven't given back kept, saved with the run (8)O(review pairs) + O(N) in SQLRun.report · report.qa · report.trust_sheet · review_state._trust_links · review_state.applied · report.fill_rates · match.review_floor · match.named_sql · report.review_sheet · report.households · report.carry_overmatch.pattern_coverage · match.levels · io.markdown · kind.kind_review_sheet · report.carry_kindsreport.report_text · match.scaled_prior · source.filled · match.parameters · config.sources_text · io.code_version · io.read_parquetgold.parquet · steps.jsonmatches/part-n.parquet · rule_kept/part-n.parquetproposed/part-n.parquet · set_aside/part-n.parquetscored/part-n.parquet: every pair at 0 points or more, each comparison's levelstate/records/part-n.parquet · state/rule_clusters.parquetstate/model.json · state/scoring.json · state/placeholders.json · state/config.txtkind_answers/part-n.parquet: the sheet given, as givenqa.md · report.txt · review.parquet · kind_review.parquet1 · Load: read the file, check it, standardize each record (chunks in parallel processes), and tell persons from companiesO(N·L)Run.load · run.ahead · Run._answers · folder.answer_parts · Run._given · source.filledsource.open_source · source.suggest · report.unmapped_columns · source.fit · settings.present_fields · settings.fit_to_fileRun._set_aside · Run._set_aside_in_order · kind.classifier · source.answers · source.known · kind.merge_answers · standardize.prepare · source.file_facts · Run._placeholders · Run._lacksRun._prepared_key · kind.century · folder.prepared · Run._kept · folder.forget_prepared · folder.keep_prepared · io.alive · io.read_parquet · match.compared_exactly4b · Keys: with [keys], the keys learned on this file: the pairs the model accepts, found by sampling every way two records agree, listed by the fewest pairs. Without, the method's generators, but one whose agreement weighs nothing and whose sampled pairs at the review floor other keys or rules all listO(A·N + f·N·b + K·T + K'·N)Run.keys · keys.learn · keys.idle · match.review_floor · source.known7c · Gap: what the keys miss against comparing every pair: the keys step's sample of records (only those that could reach the floor), each paired with every record sharing a value or a near one, scored; the pairs at the search floor, the review floor, the line and 20 bits no key or rule listed, counted and scaledO(f·N·b)Run.gap · keys.gaprecordsset_asiderule_clustersrule_keptcandsscoresrule_scoresmatchesproposedkeptgolder_decisionsbuilt first, saved togetherbox: a step, what it does, its cost, and the code it runscylinder: a working table · note: a filesolid arrow: makes or uses · dashed arrow: runs next \ No newline at end of file +LLD 1 · A full run: Run.full, its reports written with it (nayose resolve records … --workspace …)inputsevery step after load reads recordsthe workspace (--workspace)the input: the files given, read as they come (inputs/)config/final.toml · config/settings.tomlconfig/kind_names.npz: the name modelapply answers · kind_answers/part-n.parquetdecisions/part-n.parquet2 · Budget: count every key's pairs before any join; flag blocks too big to join blindO(N) per keyRun.budget · candidates.estimate · Run.capacity · Run.spill_room · run.size_bytes3 · Fit: learn the model from the file itself, without labels; measure u on this fileO(n + S·K + P_em·K) + O(N + S·K)Run.fit · match.train_fs · match.scored_umatch.u_basis · match.fit_model · match.fit_sample_sql4 · Rules: join the certain cases, the first rule wins; score each pair with the model; cluster them (they steer the candidates)O(N + ΣP_r·L²) + O(R·K) + O(E·d + N)Run.rules · Run._rules · Run._untrusted · match.rule_edges · match.score_fs · cluster.merge · source.filled · Run._alone · Run._sized5 · Candidates: list the pairs worth scoring, keeping only those that can reach 0; check big blocks on a sampleO(N·L + ΣP_g)Run.candidates · run.listed · candidates.candidates · Run._reaching · match.reaching_sql · match.bound_reads · Run._gate · Run._scored · match.score_fs · match.bound_tests · Run._gated · Run._joinedmatch.search_floor · Run.capacity · Run.spill_room · run.size_bytes · Run._sized · Run._alone · match.rough_reads6 · Score: give every candidate pair that can reach 0 its evidence in bits, the rest left below 0 by a bound, a million at a time, saying how far it is; count the patterns they fall inO(C·K·L²)Run.score · Run._scored · match.score_fs · match.bound_tests · Run._patterns · Run._bands · results.banded · match.named_sql · match.review_floor · Run._line · results.own_merging7 · Cluster: merge the rules' and the model's pairs at the trust line (none set: every rule pair, and the model's at its threshold with a name compared), or, with the workspace's merge conditions, the pairs that meet them; people's matches, less their no_match; join them into entities, from single recordsO(C + E·d + N)Run._load_decisions · folder.latest_decisions · folder.parts · Run.cluster · Run._accept · Run.scored · match.named_sql · Run._links · results.merge_wherecluster.merge · Run._gold · run.customer_kinds · Run._expected_false · Run._alone · Run._line · Run._sized · match.review_floor · results.own_merging7b · Probe: the matches no key lists: pairs drawn within groups matches share (a phone, a postal area), scored; those at the line the likely links don't join, counted and scaledO(N + P·K + L)Run.probe · probe.missed · Run._sized8 · Write: save the answer, the pairs at 0 points or more with each comparison's level (scored/), the state for increments and the reports (9, built first); the old run goes only once all is writtenO(N + C)Run.write · config.generator_dict · Run._links · Run.scored · match.named_sql · folder.next_part · folder.stage · folder.finish_writes · folder.history · folder.brief · Run._log · folder.log · io.code_version · io.replace_text · io.patientlythe round its method is in: folder.round_ran · folder.rounds · folder.method · config.toml_of · Run._alone · Run._sized9 · Report: the checks and the sheets for people, what they typed in and haven't given back kept, saved with the run (8)O(review pairs) + O(N) in SQLRun.report · report.qa · report.trust_sheet · review_state._trust_links · review_state.applied · report.fill_rates · match.review_floor · match.named_sql · report.review_sheet · report.households · report.carry_overmatch.pattern_coverage · match.levels · io.markdown · kind.kind_review_sheet · report.carry_kindsreport.report_text · match.scaled_prior · source.filled · match.parameters · config.sources_text · io.code_version · io.read_parquetreport.diagnosis · report.feature_path · match._ranked · match.search_floorgold.parquet · steps.jsonmatches/part-n.parquet · rule_kept/part-n.parquetproposed/part-n.parquet · set_aside/part-n.parquetscored/part-n.parquet: every pair at 0 points or more, each comparison's levelstate/records/part-n.parquet · state/rule_clusters.parquetstate/model.json · state/scoring.json · state/placeholders.json · state/config.txtkind_answers/part-n.parquet: the sheet given, as givenqa.md · report.txt · review.parquet · kind_review.parquet1 · Load: read the file, check it, standardize each record (chunks in parallel processes), and tell persons from companiesO(N·L)Run.load · run.ahead · Run._answers · folder.answer_parts · Run._given · source.filledsource.open_source · source.suggest · report.unmapped_columns · source.fit · settings.present_fields · settings.fit_to_fileRun._set_aside · Run._set_aside_in_order · kind.classifier · source.answers · source.known · kind.merge_answers · standardize.prepare · source.file_facts · Run._placeholders · Run._lacksRun._prepared_key · kind.century · folder.prepared · Run._kept · folder.forget_prepared · folder.keep_prepared · io.alive · io.read_parquet · match.compared_exactly4b · Keys: with [keys], the keys learned on this file: the pairs the model accepts, found by sampling every way two records agree, listed by the fewest pairs. Without, the method's generators, but one whose agreement weighs nothing and whose sampled pairs at the review floor other keys or rules all listO(A·N + f·N·b + K·T + K'·N)Run.keys · keys.learn · keys.idle · match.review_floor · source.known7c · Gap: what the keys miss against comparing every pair: the keys step's sample of records (only those that could reach the floor), each paired with every record sharing a value or a near one, scored; the pairs at the search floor, the review floor, the line and 20 bits no key or rule listed, counted and scaledO(f·N·b)Run.gap · keys.gaprecordsset_asiderule_clustersrule_keptcandsscoresrule_scoresmatchesproposedkeptgolder_decisionsbuilt first, saved togetherbox: a step, what it does, its cost, and the code it runscylinder: a working table · note: a filesolid arrow: makes or uses · dashed arrow: runs next \ No newline at end of file diff --git a/docs/usage.md b/docs/usage.md index 60818f8..d820a4a 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -240,7 +240,7 @@ None of these runs anything: they read the workspace's files. | `proposed/` | the model's pair below the trust line | both IDs, `match_weight`, `match_probability`, `sources`; empty with no trust line | the pairs waiting for people, or for the line to come down | | `scored/` | pair at 0 points or more, and every rule's pair | both IDs, `match_weight`, `match_probability`, `sources` (none: a rule alone found it), `rule`, `named`, `keep`, and `gamma_`, the level each comparison reached | list, filter and review any pair without a run; merge by your own conditions ([section 6](#merge-by-your-own-conditions)). A workspace whose run wrote none gets it once, on first use, from the pairs that run stored, scored again | | `qa.md` | | the summary (with `missed`: the matches no pair key lists, sampled within `[probe]` groups, as estimated by the probe); each field's fill rate (flagged under 5%), the share that looks right and the share of its most common value; what was turned off; each step's Big-O, time and memory; pairs by weight; cluster sizes; the 10 largest clusters; match patterns no key guarantees; the review queue; records and customers by kind; clusters holding both a sure person and a sure company; the oversized blocks and what was decided for each | check the run without labels | -| `report.txt` | | the run on one page of plain text, with no record's value in it: what ran (the code, the tracked files changed in this copy, the pipeline, the machine's limits), the summary, the fields, the columns no field reads (filled, distinct, the field each looks most like), scores and links by weight with the false links expected, the commonest patterns of levels below and from the review floor, cluster sizes, each step's time and memory (the fit's own sample and the u measured on the file apart; the gate's samples), where each step's time went in the string comparisons (per function: the wall seconds while it ran, the threads that ran it at once, the values it compared, per value), the pairs each EM session learned from (with fit em `choose`, each session judged: its pairs, the share of its matches agreeing on the given name, kept or dropped and why) and the model's m, u and bits per level, the costliest blocks (a full run), the match patterns no key pairs | send it to whoever helps with the run ([section 7](#7--when-something-is-wrong)) | +| `report.txt` | | the run on one page of plain text, with no record's value in it: what ran (the code, the tracked files changed in this copy, the pipeline, the machine's limits), DIAGNOSIS (what limits what the run finds, below), the summary, the fields, the columns no field reads (filled, distinct, the field each looks most like), scores and links by weight with the false links expected, the commonest patterns of levels below and from the review floor, cluster sizes, each step's time and memory (the fit's own sample and the u measured on the file apart; the gate's samples), where each step's time went in the string comparisons (per function: the wall seconds while it ran, the threads that ran it at once, the values it compared, per value), the pairs each EM session learned from (with fit em `choose`, each session judged: its pairs, the share of its matches agreeing on the given name, kept or dropped and why) and the model's m, u and bits per level, the costliest blocks (a full run), the match patterns no key pairs | send it to whoever helps with the run ([section 7](#7--when-something-is-wrong)) | | `review.parquet` | pair for people | `why` (`decide` or `audit`); an empty `decision`; both records' compared fields; per comparison, the level reached and its bits; `accepted`, whether the run joined the pair; `household`, whether it may be two people of one household ([section 6](#6--review-and-decide)) | decide the next range below the trust line and audit above it, or, with no line, decide the pairs just below the threshold and those above it with no name compared, and audit those just above ([section 6](#6--review-and-decide)) | | `kind_review.parquet` | record for people | `why` (`decide` or `measure`), `reason` in words, `engine_says`, `name_says`, `flag` and `birth_date` if the file has them, the name; `engine` and `engine_p` (the engine's kind and its probability of a company), `name_p`, `drawn`; an empty `kind` | answer 1 person, 2 company or 0 can't tell ([section 6](#6--review-and-decide)) | | `decisions/` | decided pair | both IDs and `decision`; `why` (`decide`, `audit`, or `split` from `split entity`) when there is one; `round`, the round it was decided in; one part per file people gave back | what people decided; a full run applies it again | @@ -422,6 +422,22 @@ Wrong audit pairs are also the evidence for changing the matching. In `qa.md`, a ## 7 · When something is wrong +`report.txt` starts with DIAGNOSIS: what limits what the run finds, at most eight lines, the most pairs at stake first, +each with a number and the command that fixes it where there is one (`--workspace …`: the run's workspace). It names: + +- a compared field filled in under 30% of the records (a person's name and a company's aside: each kind fills one), the + pairs below the review floor its comparison can't compare, and a column no field reads whose values look like the + field, with the `nayose set feature` that adds it as a fallback; +- EM sessions of the run's own fit whose matches mostly share no given name: they learned households (`nayose choose em`); +- pairs below the review floor that agree exactly on a name and another field (the birth date, the phone, the + address, …), by pattern, and the true pairs the model itself expects among all the pairs below the review floor; +- levels that learned nothing (m = u, 0 bits) yet hold pairs (`nayose try sample`); +- pairs at the review floor no key lists (the gap's estimate and its interval), and oversized blocks skipped whose best + sampled pair came within 2 bits of the search floor (`nayose describe method`); +- EM sessions that learned from under 200 pairs (more records: `nayose choose em … --records`, or `nayose try sample`). + +What is fine is left out. + What the run handles by itself, and reports in its summary: | Found | Done | diff --git a/er/match.py b/er/match.py index 84fc95f..f47c07b 100644 --- a/er/match.py +++ b/er/match.py @@ -457,30 +457,27 @@ def em_sql(model: FSModel, rule: tuple[Key, ...]) -> str: ENOUGH = 10 # a comparison takes its m from the kept sessions when they hold this many pairs agreeing at its first level; else from every session -def chosen(model: FSModel, t: pd.DataFrame, learned: dict[tuple[Key, ...], dict[tuple[str, str], float]], - pairs: dict[tuple[Key, ...], int]) -> tuple[dict[tuple[str, str], list[float]], list[dict], list[str]]: - """Fit em "choose": each session (learned: its m per comparison and level) judged by the matches it learned. A session - is kept when at least GIVEN_SHARE of them agree on the given name (the name comparison's levels of exact or near-exact - agreement, its given-name level among them where that is filled in GIVEN_FILLED of the named records): pairs that - share a phone or an address teach EM households, whose members share a family name only. A session keyed on a name - field is kept by its key (it learned no name m). Each level's m is then the median of the kept sessions that learned - it; a comparison but the name's that those barely saw (fewer than ENOUGH of their pairs agree at its first level, - exactly) takes every session's. Returns those m lists ({} when none is kept), each session's row, and the comparisons that took every - session's. O(sessions · comparisons · n).""" +def judged(model: FSModel, t: pd.DataFrame, learned: dict[tuple[Key, ...], dict[tuple[str, str], float]], + pairs: dict[tuple[Key, ...], int]) -> tuple[list[dict], Comparison | None]: + """Each EM session (learned: its m per comparison and level) judged by the matches it learned: kept when at least + GIVEN_SHARE of them agree on the given name (the name comparison's levels of exact or near-exact agreement, its + given-name level among them where that is filled in GIVEN_FILLED of the named records): pairs that share a phone or + an address teach EM households, whose members share a family name only. A session keyed on a name field is kept by + its key (it learned no name m). Each session's row (session, field as set fit takes it, pairs, given_share, kept, + why), and the name comparison judged by. O(sessions · levels).""" reads = lambda c: {col for lv in c.levels for col, _ in lv.any_of} anchor = next((c for c in model.comparisons if GIVEN in reads(c)), None) or next((c for c in model.comparisons if PERSON in reads(c)), None) - rows = [dict(session=" + ".join(k.column for k in r), field="+".join(":".join((k.column, *k.normalize)) for k in r), pairs=n, given_share=None) for r, n in pairs.items()] # field: as set fit --em takes it + rows = [dict(session=" + ".join(k.column for k in r), field="+".join(":".join((k.column, *k.normalize)) for k in r), pairs=n, given_share=None) for r, n in pairs.items()] if anchor is None: - return {}, [dict(x, kept=False, why="no person's name compared") for x in rows], [] + return [dict(x, kept=False, why="no person's name compared") for x in rows], None names = reads(anchor) | set(anchor.null_if_missing) given_col = next((comp.column(col) for lv in anchor.levels for col, comp in lv.any_of if col == GIVEN), None) named = t[[c for c in anchor.null_if_missing if c in t]].notna().any(axis=1).sum() given = (t[given_col].notna().sum() if given_col in t else 0) / max(named, 1) >= GIVEN_FILLED agree = {lv.label for lv in anchor.levels if all(comp.function == "exact" or (comp.function in SIMILARITIES and (comp.threshold or 0) >= 0.9) for _, comp in lv.any_of) and (given or GIVEN not in {col for col, _ in lv.any_of})} - kept = [] for r, row in zip(pairs, rows): - m = {lv: v for (c, lv), v in learned[r].items() if c == anchor.name} + m = {lv: v for (c, lv), v in learned.get(r, {}).items() if c == anchor.name} if {k.column for k in r} & names: row.update(kept=True, why="keyed on a name") elif not m: @@ -488,8 +485,17 @@ def chosen(model: FSModel, t: pd.DataFrame, learned: dict[tuple[Key, ...], dict[ else: row["given_share"] = share = sum(v for lv, v in m.items() if lv in agree) / sum(m.values()) row.update(kept=share >= GIVEN_SHARE, why="matches agree on the given name" if share >= GIVEN_SHARE else "matches share no given name: households") - if row["kept"]: - kept.append(r) + return rows, anchor + + +def chosen(model: FSModel, t: pd.DataFrame, learned: dict[tuple[Key, ...], dict[tuple[str, str], float]], + pairs: dict[tuple[Key, ...], int]) -> tuple[dict[tuple[str, str], list[float]], list[dict], list[str]]: + """Fit em "choose": the sessions judged; each level's m the median of the kept sessions that learned it; a + comparison but the name's that those barely saw (fewer than ENOUGH of their pairs agree at its first level, exactly) + takes every session's. Returns those m lists ({} when none is kept), each session's row, and the comparisons that + took every session's. O(sessions · comparisons · n).""" + rows, anchor = judged(model, t, learned, pairs) + kept = [r for r, row in zip(pairs, rows) if row["kept"]] if not kept: return {}, rows, [] median, wide = {}, [] @@ -537,19 +543,19 @@ def train_fs(model: FSModel, df: pd.DataFrame, db: DuckDBAPI, uid: str, labels: prior = [r for r in model.prior_rules if rule_pairs(t, r)] em_pairs = {r: n for r in model.em_rules if (n := rule_pairs(t, r))} if model.m_method == "em" else {} # each EM session's pairs in the sample em = list(em_pairs) - judged = {r: n for r in model.em_choose if (n := rule_pairs(t, r))} if model.m_method == "em" else {} # fit em "choose": these judged first + choose = {r: n for r in model.em_choose if (n := rule_pairs(t, r))} if model.m_method == "em" else {} # fit em "choose": these judged first if not prior: raise InputError(f"no two records of the {len(t):,} the fit samples agree on any near-certain rule " f"({'; '.join(map(fields, model.prior_rules))}), so the chance that two records match can't be estimated", fix="map the fields those rules read, or give a model fitted before with --model") - if model.m_method == "em" and not em and not judged: + if model.m_method == "em" and not em and not choose: raise InputError(f"no two records of the {len(t):,} the fit samples share {' or '.join(map(fields, model.em_rules))}, " "the pairs it learns from", fix="map those fields to filled columns, or give a model fitted before with --model") settings = fs_settings(model, uid, u) # unestimated levels: handled here tf_levels = any(c.tf and any(lv.tf for lv in c.levels) for c in model.comparisons) # none: EM iterates on each agreement pattern's count, not on every pair median: dict[tuple[str, str], list[float]] = {} choice, wide = [], [] - with narrowed(t, db, read_by(settings, *(em_sql(model, r) for r in (*prior, *em, *judged))) | {uid, "__label__"}) as records: + with narrowed(t, db, read_by(settings, *(em_sql(model, r) for r in (*prior, *em, *choose))) | {uid, "__label__"}) as records: linker = Linker(records, settings, log_level=logging.ERROR, validate_settings=False) # its checks only log warnings, which ERROR hides: sqlglot parses for nothing linker.training.estimate_probability_two_random_records_match([keys_sql(r) for r in prior], recall=model.prior_recall) @@ -564,16 +570,20 @@ def apart(rules: Iterable[tuple[Key, ...]]) -> dict[tuple[Key, ...], dict[tuple[ for lv in c["comparison_levels"] if c["output_column_name"] not in fixed and "m_probability" in lv and not lv.get("is_null_level")} one.table_management.invalidate_cache() return out - if judged: - median, choice, wide = chosen(model, t, apart(judged), judged) + if choose: + median, choice, wide = chosen(model, t, apart(choose), choose) if median: - em_pairs = {r: judged[r] for r, row in zip(judged, choice) if row["kept"]} + em_pairs = {r: choose[r] for r, row in zip(choose, choice) if row["kept"]} elif not em: raise InputError(f"no session of the {len(t):,} records the fit samples learned matches that agree on the given name, and no two " f"share {' or '.join(map(fields, model.em_rules))}, the method's own", fix="map those fields to filled columns, or give a model fitted before with --model") + learned = {} # the method's sessions, each judged as fit em "choose" judges them (er "em_given"): what DIAGNOSIS says of them for rule in em if not model.em_median and not median else (): - linker.training.estimate_parameters_using_expectation_maximisation(em_sql(model, rule), estimate_without_term_frequencies=not tf_levels) - for ms in (apart(em) if model.em_median and not median else {}).values(): + session = linker.training.estimate_parameters_using_expectation_maximisation(em_sql(model, rule), estimate_without_term_frequencies=not tf_levels) + learned[rule] = {(c.output_column_name, lv.label_for_charts): lv._m_probability for c in session.core_model_settings.comparisons + for lv in c.comparison_levels if not lv.is_null_level and isinstance(lv._m_probability, float)} + learned |= apart(em) if model.em_median and not median else {} + for ms in learned.values() if model.em_median else (): for key, v in ms.items(): median.setdefault(key, []).append(v) if model.m_method == "labels": @@ -600,7 +610,8 @@ def apart(rules: Iterable[tuple[Key, ...]]) -> dict[tuple[Key, ...], dict[tuple[ skipped = [fields(r) for r in (*model.prior_rules, *(model.em_rules if model.m_method == "em" and not kept else ())) if r not in (*prior, *em)] return {**model_json, "er": dict(fit_rows=len(t), tf=tf, filled=dict(u=filled_u, m=filled_m), skipped=skipped, adjusted=adjusted, em_pairs={fields(r): n for r, n in em_pairs.items()}, - **({"em_choice": choice, "em_wide": wide} if choice else {}))} + **({"em_choice": choice, "em_wide": wide} if choice else {}), + **({"em_given": [{k: x[k] for k in ("session", "pairs", "given_share")} for x in judged(model, t, learned, em_pairs)[0]]} if learned else {}))} def compared_exactly(model: FSModel) -> list[str]: # the fields a comparison's first level tests for equality alone: an agreement on them is that level's diff --git a/er/report.py b/er/report.py index 637c8d3..b547c03 100644 --- a/er/report.py +++ b/er/report.py @@ -17,13 +17,14 @@ import numpy as np import pandas as pd -from er.config import COPIES, DeletionKeys, InputError, Key, KeyBlocker, Pipeline, SortedNeighbourhood, TfidfNeighbours, bits, lit, q, sources_text +from er.config import COPIES, GIVEN, DeletionKeys, InputError, Key, KeyBlocker, Pipeline, SortedNeighbourhood, TfidfNeighbours, bits, lit, q, sources_text from er.folder import PENDING, history, latest_decisions, method, rounds, runs from er.io import code_version, read_parquet, table_after from er.kind import COMPANY, PERSON, SAYS, UNSURE, sheet_parts -from er.match import ELSE, levels, named_sql, parameters, pattern_coverage, review_floor, scaled_prior +from er.match import ELSE, GIVEN_SHARE, _ranked, levels, named_sql, parameters, pattern_coverage, review_floor, scaled_prior, search_floor from er.source import filled, fit from er.review_state import _trust_links, applied, read +from er.settings import DEFAULT_FEATURES if TYPE_CHECKING: from er.run import Run @@ -189,11 +190,105 @@ def qa(run: "Run", fields: list[str] | None = None, review: int = 50, previous: review_sheet=sheet) +SPARSE = 0.3 # a compared field filled in fewer records than this: DIAGNOSIS names it +THIN = 200 # a kept EM session that learned from fewer pairs: DIAGNOSIS asks for more records +NEAR = 2.0 # bits: a skipped oversized block whose best sampled pair came this close to the search floor +SHOWN = 8 # DIAGNOSIS's lines at most +WS = "--workspace …" + + +def feature_path(field: str) -> str | None: # where set feature puts a field's columns + if field in ("person_name", "given_name"): + return "Name/Full name" + ("/Given name" if field == "given_name" else "") + return next((f"{g}/{f['name']}" for g, feats in DEFAULT_FEATURES for f in feats if f.get("field") == field), None) + + +def diagnosis(run: "Run", tables: dict[str, pd.DataFrame], lo: float) -> list[str]: + """What limits what the run finds, the most pairs at stake first, at most SHOWN lines, each what, a number and the + fix (a command where there is one); what is fine is left out: EM sessions of the run's own fit that learned + households, compared fields under SPARSE filled (with a column no field reads that looks like one), pairs below the + review floor that agree exactly on a name and another field, levels that learned nothing yet hold pairs, what the + search leaves (the gap at the review floor, oversized blocks skipped that came within NEAR bits), and, last, EM + sessions of under THIN pairs. No record's value: + field, column and level names, counts and shares. O(C) over the scores.""" + con, model, m, fs = run.con, run.scoring, run.scoring["er"], run.pipeline.fs + gammas = {c for c, *_ in con.execute("DESCRIBE scores").fetchall() if c.startswith("gamma_")} + below = f"match_weight >= 0 AND match_weight < {lo}" + count = lambda where: con.execute(f"SELECT count(*) FROM scores WHERE {where}").fetchall()[0][0] + n_below, expected = con.execute(f"SELECT count(*), coalesce(sum(match_probability), 0) FROM scores WHERE {below}").fetchall()[0] + gamma = {c["output_column_name"]: {lv["label_for_charts"]: g for g, lv in _ranked(c)} for c in model["comparisons"]} + out: list[tuple[float, str]] = [] + + if households := [x for x in m.get("em_given", []) if x["given_share"] is not None and x["given_share"] < GIVEN_SHARE]: # EM taught by households: first of equals + out.append((n_below, "EM learned households from " + ", ".join(f"{x['session']} ({x['given_share']:.0%} of its matches agree on the given name)" for x in households) + + f": it scores the {n_below:,} pairs below review → nayose choose em {WS}")) + + fill = tables["fill"].set_index("field")["filled"] # compared fields mostly empty (one comparison a line), and a column that looks like one + either = {f for group in run.pipeline.fields.one_of for f in group} # a person's name or a company's: each kind fills one + cols = getattr(run, "unmapped", None) + for c in (c for c in fs.model.comparisons if f"gamma_{c.name}" in gammas): + sparse = [f for f in dict.fromkeys([*(col for lv in c.levels for col, _ in lv.any_of), *c.null_if_missing]) if f in fill and fill[f] < SPARSE and f not in either] + if not sparse: + continue + if not (stake := count(f"{below} AND {q('gamma_' + c.name)} = -1")): + continue + line = f"{', '.join(f'{f} {fill[f]:.0%}' for f in sparse)} filled: {stake:,} pairs below review compare no {c.name}" + for f in sparse: + like = cols[(cols["looks_like"] == f) & (cols["share"] >= 0.5)].sort_values("filled", ascending=False) if cols is not None and len(cols) else [] + if len(like) and (path := feature_path(f)): + col, now = like.iloc[0]["column"], ",".join("+".join(alt) for alt in run.present.get(f, ())) + line += f"; column {col} looks like {f} ({like.iloc[0]['filled']:.0%} filled) → nayose set feature \"{path}={now + ',' if now else ''}{col}\" {WS}" + break + out.append((stake, line)) + + agree, named = [], ["given name"] # near misses: a name and another field agree exactly, below the review floor (a phone and an address alone: a household's) + for c in fs.model.comparisons: + named += [c.name] if {col for lv in c.levels for col, _ in lv.any_of} & either else [] + if f"gamma_{c.name}" in gammas: + exact = [gamma[c.name][lv.label] for lv in c.levels if all(x.function == "exact" for _, x in lv.any_of) and GIVEN not in {col for col, _ in lv.any_of}] + given = [gamma[c.name][lv.label] for lv in c.levels if GIVEN in {col for col, _ in lv.any_of}] + agree += [f"CASE WHEN {q('gamma_' + c.name)} IN ({', '.join(map(str, gs))}) THEN '{what}' END" for what, gs in ((c.name, exact), ("given name", given)) if gs] + if len(agree) > 1 and n_below: + got = con.execute(f"""SELECT p, count(*) AS n FROM (SELECT concat_ws(' + ', {', '.join(agree)}) AS p FROM scores WHERE {below}) + WHERE p LIKE '% + %' AND list_has_any(string_split(p, ' + '), {named!r}) GROUP BY 1 ORDER BY n DESC, p""").fetchall() + if total := sum(n for _, n in got): + out.append((total, f"{total:,} pairs below review agree exactly on a name and another field ({', '.join(f'{p} {n:,}' for p, n in got[:3])}); " + f"the model expects {expected:,.0f} true pairs among the {n_below:,} below review → nayose list patterns {WS}")) + + held, empty = [], 0 # levels that learned nothing (m = u: 0 bits) + for c, lv in (tuple(x) for x in m.get("filled", {}).get("m", [])): + if lv != ELSE and f"gamma_{c}" in gammas and lv in gamma.get(c, {}): + if n := count(f"{q('gamma_' + c)} = {gamma[c][lv]}"): + held.append((n, f"{c} {lv} {n:,}")) + else: + empty += 1 + if held: + out.append((sum(n for n, _ in held), "levels that learned nothing hold pairs, at 0 bits: " + ", ".join(x for _, x in sorted(held, reverse=True)) + + (f" ({empty} more hold none)" if empty else "") + f" → nayose try sample {WS}")) + + g = next((x for x in run.steps if x["step"] == "gap"), None) # what the search leaves + if g and (at := next((x for x in g["thresholds"] if x["bits"] >= lo), None)) and at["missed"]: + out.append((at["missed"], f"no key lists about {at['missed']:,} pairs at {at['bits']:g}+ ({at['missed_lo']:,}–{at['missed_hi']:,}) → nayose describe method {WS}")) + gated, floor = getattr(run, "gated", pd.DataFrame()), search_floor(run.pipeline) + if len(gated) and "best" in gated: + near = gated[gated["decision"].str.startswith("skipped") & (gated["best"] >= floor - NEAR)] + if len(near): + out.append((int(near["pairs"].sum()), f"{len(near):,} oversized blocks skipped ({int(near['pairs'].sum()):,} pairs) held a sampled pair within {NEAR:g} bits " + f"of {floor:g} → nayose describe method {WS}")) + + kept = [(x["session"], x["pairs"]) for x in m["em_choice"] if x["kept"]] if m.get("em_choice") else list(m.get("em_pairs", {}).items()) # thin sessions + if thin := [(x, n) for x, n in kept if n < THIN]: + fix = f"nayose choose em {WS} --records {2 * m['fit_rows']}" if m.get("em_choice") else f"nayose try sample {WS}" + out.append((0, f"EM learned from few pairs: {', '.join(f'{x} {n:,}' for x, n in thin)} → {fix}")) + lines = [x for _, x in sorted(out, key=lambda x: -x[0])[:SHOWN]] or ["nothing found"] + return ["DIAGNOSIS (what limits what the run finds, the most pairs at stake first)", *(f" {x}" for x in lines), ""] + + def report_text(run: "Run", tables: dict[str, pd.DataFrame], pipeline: Path) -> str: """The run on a page of plain text that holds no record's value (no ID, name, phone or address: the notes that quote one are run.examples, in qa.md only), so it can leave the machine that holds the records. `tables` is qa()'s. - It shows what ran (the code, the tracked files changed in this copy, the pipeline, the machine's limits), the - summary, each field's fill and shape, the columns no field reads, scores and accepted links by weight with the false links the + It shows what ran (the code, the tracked files changed in this copy, the pipeline, the machine's limits), what limits + what the run finds (diagnosis), the summary, each field's fill and shape, the columns no field reads, scores and accepted links by weight with the false links the model expects, each score band's commonest patterns, cluster sizes, each step's time and memory, its time in each string comparison, the model (the prior, the pairs each EM session learned from, and m, u and bits per level), the costliest blocks, and the patterns no key pairs. O(C + E) over scores and matches.""" con, fs = run.con, run.pipeline.fs @@ -219,6 +314,7 @@ def pct(x: object) -> str: f"{threads} threads · memory limit {memory} · spill room {run.spill_room() / 1e9:,.0f} GB · " f"took {sum(x['seconds'] for x in run.steps):,.1f} s · peak {max(x['peak_memory_gb'] for x in run.steps):.2f} GB", ""] + out += diagnosis(run, tables, lo) summary = tables["summary"][tables["summary"]["what"] != "example"] out += ["SUMMARY", *(f" {what:<12} {value}" for what, value in summary.itertuples(index=False)), ""] diff --git a/tests/test_cli.py b/tests/test_cli.py index 04e26a8..d9d740f 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -833,6 +833,40 @@ def test_a_customer_id_held_by_many_records_is_a_placeholder_every_step_and_the_ assert new["customer_id"].isna().all() and ("birth_date" not in new or new["birth_date"].isna().all()) +def test_the_diagnosis_names_what_limits_the_run_the_most_pairs_first_each_with_its_fix(tmp_path, population): + from er.match import _ranked, review_floor, search_floor + from er.report import SHOWN, diagnosis, qa + from er.run import Run + s, p, g = population + g.df.to_parquet(tmp_path / "in.parquet", index=False) + run = Run(p, s).full(tmp_path / "in.parquet", tmp_path / "run") + tables, lo, m = qa(run, [f for f in s.columns if f not in ("id", "entity")]), review_floor(run.pipeline), run.scoring["er"] + said = lambda: diagnosis(run, tables, lo)[1:-1] + top = {c["output_column_name"]: max(g for g, _ in _ranked(c)) for c in run.scoring["comparisons"]} + assert said() and len(said()) <= SHOWN + tables["fill"].loc[tables["fill"]["field"] == "birth_date", "filled"] = 0.1 # a compared field mostly empty, and a column like it + run.unmapped = pd.DataFrame([dict(column="BORN", filled=0.9, distinct=0.5, looks_like="birth_date", share=0.8)]) + run.con.execute(f"UPDATE scores SET gamma_birth = -1, match_weight = {lo - 1} WHERE rowid IN (SELECT rowid FROM scores LIMIT 5)") + assert any("birth_date 10% filled" in x and 'nayose set feature "Birth date/Birth date=' in x and ",BORN\" --workspace" in x for x in said()) + m["em_given"] = [dict(session="phone", pairs=100, given_share=0.1)] # EM taught by households + assert any(x.startswith(" EM learned households from phone (10%") and "nayose choose em" in x for x in said()) + near = lambda: next((int(x.split()[0].replace(",", "")) for x in said() if "agree exactly on a name and another field" in x and "the model expects" in x), 0) + was = near() # near misses: three pairs from the review floor moved below it, agreeing on the name and the phone + run.con.execute(f"UPDATE scores SET gamma_name = {top['name']}, gamma_phone = {top['phone']}, match_weight = {lo - 1} WHERE rowid IN (SELECT rowid FROM scores WHERE match_weight >= {lo} LIMIT 3)") + assert near() == was + 3 + m["filled"]["m"].append(["phone", "exact"]) # a level that learned nothing, holding pairs + assert any("levels that learned nothing hold pairs" in x and "phone exact" in x for x in said()) + gap = next(x for x in run.steps if x["step"] == "gap") + gap["thresholds"] = [dict(bits=lo, pairs=500, missed=50, missed_lo=40, missed_hi=60)] + run.gated = pd.DataFrame([dict(generator="phone", value="x", records=900, pairs=404_550, sampled=2000, best=search_floor(run.pipeline) - 1, hits=0, decision="skipped: over capacity")]) + assert any("no key lists about 50 pairs" in x and "(40–60)" in x for x in said()) and any("1 oversized blocks skipped (404,550 pairs)" in x for x in said()) + m["em_pairs"] = {"phone": 40} # a thin session + assert any("EM learned from few pairs: phone 40" in x for x in said()) and len(said()) <= SHOWN + stakes = said() + assert stakes[0].startswith(" 1 oversized blocks skipped") # the most pairs first + assert stakes[-1].startswith(" EM learned from few pairs") # advice, no pairs: last + + def test_the_run_report_holds_no_record_value(tmp_path, population): import re from er.report import qa, report_text diff --git a/tests/test_scoring.py b/tests/test_scoring.py index 7ba7538..1ca4273 100644 --- a/tests/test_scoring.py +++ b/tests/test_scoring.py @@ -565,3 +565,11 @@ def test_em_choose_with_no_session_kept_learns_as_the_method_does(con_db): method = train_fs(household_model(em_choose=()), records, db, UID) assert [x["kept"] for x in got["er"]["em_choice"]] == [False] and got["er"]["em_pairs"] == method["er"]["em_pairs"] assert [c["comparison_levels"] for c in got["comparisons"]] == [c["comparison_levels"] for c in method["comparisons"]] + + +def test_the_method_s_own_em_sessions_are_judged_by_the_given_name_too(con_db): + _, db = con_db + records, two = households(), ((Key("phone"),), (Key("birth_date"),)) + for model in (replace(household_model(em_choose=()), em_rules=two), replace(household_model(em_choose=()), em_rules=two, em_median=True)): # in turn, and kept apart + got = {x["session"]: x["given_share"] for x in train_fs(model, records, db, UID)["er"]["em_given"]} + assert got["phone"] < 0.5 <= got["birth_date"], got # what DIAGNOSIS says of a run's own EM From c9c6be2ed82146c5b8d4c93f1a54edc8e2a4cf53 Mon Sep 17 00:00:00 2001 From: Marvin Ma Date: Mon, 16 Nov 2026 03:14:20 +0900 Subject: [PATCH 2/2] A test's loop variable no longer shadows the model it imports (ruff F402) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_012o8JNBKSY1aREPbMQDG8g2 --- tests/test_scoring.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_scoring.py b/tests/test_scoring.py index 1ca4273..7d2b144 100644 --- a/tests/test_scoring.py +++ b/tests/test_scoring.py @@ -570,6 +570,6 @@ def test_em_choose_with_no_session_kept_learns_as_the_method_does(con_db): def test_the_method_s_own_em_sessions_are_judged_by_the_given_name_too(con_db): _, db = con_db records, two = households(), ((Key("phone"),), (Key("birth_date"),)) - for model in (replace(household_model(em_choose=()), em_rules=two), replace(household_model(em_choose=()), em_rules=two, em_median=True)): # in turn, and kept apart - got = {x["session"]: x["given_share"] for x in train_fs(model, records, db, UID)["er"]["em_given"]} + for fitted in (replace(household_model(em_choose=()), em_rules=two), replace(household_model(em_choose=()), em_rules=two, em_median=True)): # in turn, and kept apart + got = {x["session"]: x["given_share"] for x in train_fs(fitted, records, db, UID)["er"]["em_given"]} assert got["phone"] < 0.5 <= got["birth_date"], got # what DIAGNOSIS says of a run's own EM