diff --git a/docs/architecture.md b/docs/architecture.md
index 6a868e5..60f52b5 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -211,7 +211,7 @@ Writes each table as Parquet under a temporary name, with the reports (9), and `
`Run.report` builds, and step 8 writes with the run's files: `qa.md`, the label-free checks; `report.txt`, the run on one page with no record's value in it, which can leave the machine that holds the records, headed by what limits what the run finds (`report.diagnosis`: [usage](usage.md#7--when-something-is-wrong)); `review.parquet`, with the evidence for each pair; and, on a full run that tells persons from companies, `kind_review.parquet`, the records for people to answer person or company ([usage](usage.md#6--review-and-decide)). The review sheet's *decide* rows are every pair of clusters whose best link scored in the threshold rule's safety margin: from the `[threshold]` grid point `margin_steps` below the threshold (4 bits) up to it, and every pair above it with no name compared. Its *audit* rows are about 50 of the pairs the run accepted (`matches`), up to 4 bits above it. With a trust line (`report.trust_sheet`), the *decide* rows are a random sample of the next range down instead, `[trust] checks` less those decided, and the *audit* rows `watch` spot-checks above the line.
- **Why:** the margin holds pairs that were clean on cross-validated train, held back in case of noise: on the synthetic val split, 55 pairs from 4 to 6 bits, none of them false; one grid step lower, at 2 bits, val's first false pairs appear. So people decide them. Without labels, the audit rows and the largest clusters are how a person catches over-merging.
-- **Code:** `Run.report` → `report.qa` (`report.fill_rates`, `match.review_floor`, `report.review_sheet`, `report.carry_over`, `match.pattern_coverage`), `report.report_text`, `kind.kind_review_sheet` (its rows picked in DuckDB, so only they reach Python), `io.markdown`
+- **Code:** `Run.report` → `report.qa` (`report.fill_rates`, `match.review_floor`, `report.review_sheet`, `report.carry_over`, `match.pattern_coverage`), `report.diagnosis` (kept in the step's facts in `steps.json`, for `describe workspace`), `report.report_text` (`report.diagnosis_lines`), `kind.kind_review_sheet` (its rows picked in DuckDB, so only they reach Python), `io.markdown`
- **From a written folder:** `report.describe_workspace` (what the folder holds), `report.describe_entity` (one record's entity and what joined it), and `report.describe_matching`: the pipeline as a graph, each stage with its method, parameters and fields, and the run's numbers (pairs per rule and how the model scores them, per generator its pairs, links and those only it found, each level's m, u, bits and pairs, the weight bands with what happened in each, the review sheet's patterns, entities and where to look first) with real examples; `explain.describe_key` inspects one pair key (records per value, one group). `nayose describe workspace`, `describe entity`, `describe matching` and `describe key` print them, and `nayose open ui` draws them.
- **Increment:** the same for the pairs the batch made, plus the previous sheet's rows still open: a run keeps no scores, so a row dropped from the sheet could never come back. With a trust line, the range people check, its checks and its false pairs are counted over the workspace's links too, less the open rows kept, so an increment asks for no check a full run wouldn't (`report.trust_sheet`). No `kind_review.parquet`: a record an increment holds back is on the next full run's sheet.
diff --git a/docs/commands.md b/docs/commands.md
index 7180659..6afb311 100644
--- a/docs/commands.md
+++ b/docs/commands.md
@@ -878,6 +878,7 @@ Records and entities, the largest entities, each run so far, the pairs and kinds
| `details` | the details file people see above each record (load details), or null |
| `stamp` | what people and runs change, by its files' sizes and times (sheets, decisions, answers, method, mapping, entities): a page coming back reads again when it moves |
| `ready` | whether a run can start on the file loaded last: run (true or false), why not, and the fix |
+| `diagnosis` | what limits what the last run finds, as report.txt's DIAGNOSIS says it, the most pairs first: what, pairs (at stake), fix (a command, or empty); null before a run that kept it |
| `next` | the commands that usually follow |
```bash
diff --git a/docs/diagrams/full-run.puml b/docs/diagrams/full-run.puml
index 661b10f..beb95c7 100644
--- a/docs/diagrams/full-run.puml
+++ b/docs/diagrams/full-run.puml
@@ -33,7 +33,7 @@ package "every step after load reads records" as after_load {
rectangle "7 · Cluster: merge the rules' and the model's pairs at the trust line (none set: every rule pair, and the model's at its threshold with a name compared), or, with the workspace's merge conditions, the pairs that meet them; people's matches, less their no_match; join them into entities, from single records\nO(C + E·d + N)\nRun._load_decisions · folder.latest_decisions · folder.parts · Run.cluster · Run._accept · Run.scored · match.named_sql · Run._links · results.merge_where\ncluster.merge · Run._gold · run.customer_kinds · Run._expected_false · Run._alone · Run._line · Run._sized · match.review_floor · results.own_merging" as cluster
rectangle "7b · Probe: the matches no key lists: pairs drawn within groups matches share (a phone, a postal area), scored; those at the line the likely links don't join, counted and scaled\nO(N + P·K + L)\nRun.probe · probe.missed · Run._sized" as probe
rectangle "8 · Write: save the answer, the pairs at 0 points or more with each comparison's level (scored/), the state for increments and the reports (9, built first); the old run goes only once all is written\nO(N + C)\nRun.write · config.generator_dict · Run._links · Run.scored · match.named_sql · folder.next_part · folder.stage · folder.finish_writes · folder.history · folder.brief · Run._log · folder.log · io.code_version · io.replace_text · io.patiently\nthe round its method is in: folder.round_ran · folder.rounds · folder.method · config.toml_of · Run._alone · Run._sized" as write
- rectangle "9 · Report: the checks and the sheets for people, what they typed in and haven't given back kept, saved with the run (8)\nO(review pairs) + O(N) in SQL\nRun.report · report.qa · report.trust_sheet · review_state._trust_links · review_state.applied · report.fill_rates · match.review_floor · match.named_sql · report.review_sheet · report.households · report.carry_over\nmatch.pattern_coverage · match.levels · io.markdown · kind.kind_review_sheet · report.carry_kinds\nreport.report_text · match.scaled_prior · source.filled · match.parameters · config.sources_text · io.code_version · io.read_parquet\nreport.diagnosis · report.feature_path · match._ranked · match.search_floor" as report
+ rectangle "9 · Report: the checks and the sheets for people, what they typed in and haven't given back kept, saved with the run (8)\nO(review pairs) + O(N) in SQL\nRun.report · report.qa · report.trust_sheet · review_state._trust_links · review_state.applied · report.fill_rates · match.review_floor · match.named_sql · report.review_sheet · report.households · report.carry_over\nmatch.pattern_coverage · match.levels · io.markdown · kind.kind_review_sheet · report.carry_kinds\nreport.report_text · match.scaled_prior · source.filled · match.parameters · config.sources_text · io.code_version · io.read_parquet\nreport.diagnosis · report.diagnosis_lines · report.feature_path · match._ranked · match.search_floor" as report
}
rectangle "4b · Keys: with [keys], the keys learned on this file: the pairs the model accepts, found by sampling every way two records agree, listed by the fewest pairs. Without, the method's generators, but one whose agreement weighs nothing and whose sampled pairs at the review floor other keys or rules all list\nO(A·N + f·N·b + K·T + K'·N)\nRun.keys · keys.learn · keys.idle · match.review_floor · source.known" as keys
rectangle "7c · Gap: what the keys miss against comparing every pair: the keys step's sample of records (only those that could reach the floor), each paired with every record sharing a value or a near one, scored; the pairs at the search floor, the review floor, the line and 20 bits no key or rule listed, counted and scaled\nO(f·N·b)\nRun.gap · keys.gap" as gap
diff --git a/docs/diagrams/full-run.svg b/docs/diagrams/full-run.svg
index 8a8f6ab..954c5c2 100644
--- a/docs/diagrams/full-run.svg
+++ b/docs/diagrams/full-run.svg
@@ -1 +1 @@
-
\ No newline at end of file
+
\ No newline at end of file
diff --git a/docs/usage.md b/docs/usage.md
index d820a4a..b1c3616 100644
--- a/docs/usage.md
+++ b/docs/usage.md
@@ -436,7 +436,10 @@ each with a number and the command that fixes it where there is one (`--workspac
sampled pair came within 2 bits of the search floor (`nayose describe method`);
- EM sessions that learned from under 200 pairs (more records: `nayose choose em … --records`, or `nayose try sample`).
-What is fine is left out.
+What is fine is left out. The run keeps the same items in `steps.json` (what, the pairs at stake, the fix);
+`nayose describe workspace` gives them as `diagnosis`, and *Find matches* lists them under *What limits what the run
+finds?*, a fix the *Method* page runs as its button there, any other as its command line to copy. A run written before
+the items were kept shows no card.
What the run handles by itself, and reports in its summary:
@@ -533,10 +536,10 @@ open ui` prints a new link: the old one says to open it.
| *Results*, after a run | *What did the run do?* The records and the entities they became, and whether anything merges without you, with *Review N pairs*. Then a tile per question, each with one number, a chart and a way to its data: *How did the scores come out?* (every pair compared, the keys' and the rules' alike, the full run's and each increment's, by points; a run logged before each counted its own says which records added since it leaves out, until matching runs again), *What is waiting for you?* (the pairs from your review points, by range, and about how long they take to review if each takes 20 seconds), *What has joined?* (entities by size, and who joined them), *Does anything merge without you?*, *Could it miss a real match?* (what the probe found among the pairs sharing a phone or postal code that no key listed; a match sharing neither isn't checked), *Were any groups too big to compare?* (values on 64 or more records, by the key that made them; it opens their list in *How it works*), *Person or company?* (the records flagged: the engine sure against the file's flag). In a workspace auto mode made, *what should I send?* Its report, with *Copy the report*. Every run, folded | `describe workspace`, `describe results`, `try everything` |
| 1 Choose a workspace | *What will this do for me?* An example: three records written differently become one entity, one ID, and *Create a workspace*, or *Auto mode*: every step on one file, each listed as it ends. *Can I pick up where I left off?* The latest workspaces, each with where it stands, and *Open* | `list workspaces`, `create workspace`, `try everything` |
| 2 Prepare your data | Two cards side by side, built alike: a question, its answer, a search, one list, then *Load another file*. *Customer information*: a records file dropped anywhere on the card or chosen (*Drop or choose a file*), sent into the workspace and loaded from there, or its path on this computer, with *Load*; or synthetic records by size and difficulty, folded, with *Generate* ([Synthetic records](#synthetic-records)); once loaded, the file, its records and columns, what waits for a repair or what is still needed (with *Suggest the columns*), and what may hurt the matching: fields mostly empty, and fields no column is mapped to with the ways of finding pairs they turn off. Then the repairs: each kind the reader proposed, its values as given and repaired, with *Accept these N*; what it fixed by itself; the rows it can't read. Then one list of the columns, group by group, a group folded unless something in it needs a look: each feature with how full it is and what its values say is off, with the button that sets it right, and under it its columns as rows in the order they are read (the first; where it is empty, the next), each with how full it is and an example. A row is dragged by ⠿ above or below another, onto one to be joined to it, or down among *Other columns* (the file's columns no feature reads) to be taken off, a folded group opening as it is held over; Alt+↑↓ moves it by keys. A column clicked lists the file's columns to put another in its place, and + adds one read where the others are empty; *Suggest* shows what the file's contents place there, kept with *Keep*; one search over every group. *Policy information*, in view as the left side scrolls: the details file ([Details beside each record](#details-beside-each-record)), dropped, chosen or its path, with *Load*, and its ID column, folded; once loaded, its rows and the share of records that have some, then one list of its columns, each with how full it is and an example: those shown in reviews first, in their order, each dragged by ⠿ (or moved with Alt+↑↓), the first naming each policy; then the others, ticked to add. A file dragged over the page shows the card that takes it; a file's upload says how far it is in the panel, as a long command does. As a file is read, its columns show at once among the features, then each one's share filled in its first 200,000 rows, read only until the whole of it is read and its suggestions and hints come. The line between the two cards is dragged to make one wider, and the next visit keeps it. A change shows at once and is saved behind it, one after another | `load records`, `generate records`, `describe records`, `accept repairs`, `set feature`, `set names`, `set preparation`, `add feature`, `remove feature`, `load details`, `set details`, `describe workspace` |
-| 3 Find matches | Before a run, *how long will it take?* The records and about how long, and *Find matches*. During it, *is it working?* What it is doing now, the seconds so far and about how long is left, and each stage filling in as its step ends. Before a run, the estimate scales this workspace's last full run to the records, else this machine's reading of the file. During it, the time left is only what this machine measured on this file: a step that counts its work (records loaded, rules applied, keys joined, pairs scored) by its own pace so far, any other by its time in this workspace's last full run, scaled to the records, and the steps after it by that run. What nothing measured yet covers (a first run's later steps) makes it *at least* the time it can count, and with nothing to count it shows only the time so far. The bar is the work done, not the time spent: a step running past its last time holds it and says it is taking longer than last time, rather than stretching the estimate. Left and opened again while it runs, step 3 shows the run as it stands. After it, the records and the entities they became, and *Results*. Then *how did it find matches?* The stages in order, each opening its own question answered: *What did it read?* · *Which pairs did the rules match, and why?* (how the scoring rates the rules' pairs, and where it disagrees) · *Which pairs did it compare, and could a real match have been missed?* (all the possible pairs, how long comparing them all would take, the pairs compared; then a sample of the pairs that share a phone or a postal area, scored: the matches the method missed, with their estimate and the pairs found; each key opens `describe key`) · *How did it score each pair?* (one pair of the run as a waterfall, *Another pair*, and what counts most; every comparison's staircase folded) · *Where did it draw the line, and how many could be wrong?* (the line and its odds, the pairs matched and the wrong ones the model expects, pairs by points; every pattern folded) · *How did matched pairs become entities?* (with what to look at first) | `resolve records`, `add records`, `match records`, `rerun records`, `describe workspace`, `list patterns`, `explain pair`, `compare records`, `describe key` |
+| 3 Find matches | Before a run, *how long will it take?* The records and about how long, and *Find matches*. During it, *is it working?* What it is doing now, the seconds so far and about how long is left, and each stage filling in as its step ends. Before a run, the estimate scales this workspace's last full run to the records, else this machine's reading of the file. During it, the time left is only what this machine measured on this file: a step that counts its work (records loaded, rules applied, keys joined, pairs scored) by its own pace so far, any other by its time in this workspace's last full run, scaled to the records, and the steps after it by that run. What nothing measured yet covers (a first run's later steps) makes it *at least* the time it can count, and with nothing to count it shows only the time so far. The bar is the work done, not the time spent: a step running past its last time holds it and says it is taking longer than last time, rather than stretching the estimate. Left and opened again while it runs, step 3 shows the run as it stands. After it, the records and the entities they became, and *Results*; then *what limits what the run finds?* ([section 7](#7--when-something-is-wrong)). Then *how did it find matches?* The stages in order, each opening its own question answered: *What did it read?* · *Which pairs did the rules match, and why?* (how the scoring rates the rules' pairs, and where it disagrees) · *Which pairs did it compare, and could a real match have been missed?* (all the possible pairs, how long comparing them all would take, the pairs compared; then a sample of the pairs that share a phone or a postal area, scored: the matches the method missed, with their estimate and the pairs found; each key opens `describe key`) · *How did it score each pair?* (one pair of the run as a waterfall, *Another pair*, and what counts most; every comparison's staircase folded) · *Where did it draw the line, and how many could be wrong?* (the line and its odds, the pairs matched and the wrong ones the model expects, pairs by points; every pattern folded) · *How did matched pairs become entities?* (with what to look at first) | `resolve records`, `add records`, `match records`, `rerun records`, `describe workspace`, `list patterns`, `explain pair`, `compare records`, `describe key` |
| *Review* | Tab *Same customer?*: the pairs waiting for you, best first, from *Review from* points (this browser keeps it), 100 a page, or a record ID's. The pair first and wide: its points and what found it; *Same*, *Different*, *Not sure* (M, N, S; ↑↓ move) and what each does; then both full rows, with every other candidate of either record beside them ([Full rows](#full-rows)), and how it scored, folded, with *Why?* beside each step ([Why this match?](#why-this-match)). Beside it, how many wait and how many are decided, *Apply decisions*, then the list, a decided pair marked; the line between the two dragged (or moved with ←→) widens either, on every page with one, and this browser keeps it. *Auto-merge* opens the conditions under which pairs merge without you ([section 6](#merge-by-your-own-conditions)): points, rules, fields that agree, all or any of them, what they would merge counted as they change, then *Apply*; *Turn off* parts what they joined. Tab *Person or company?*: two parts, *Flagged* (the engine sure against the file's flag, surest first) and *Teach the engine* (optional: the draw at random). *Is this record a person or a company?* The name, large; what the engine thinks, and why the record is asked; the answers (1 person, 2 company, 0 can't tell), saved as given; *Apply answers* matches again with them ([Persons and companies](#persons-and-companies)) | `list scored`, `show records`, `show details`, `explain pair`, `decide pair`, `apply decisions`, `describe results`, `describe merging`, `set merging`, `list kinds`, `answer kind`, `apply answers` |
| *Explore* | Tab *Pairs*: *Which pairs scored highest?* The score chart (what it leaves out said, with *Run matching again*), then every pair at 0 points or more, as many as it says, best first, 100 a page, filtered by points, how it was found, its status or a record ID; a pair opens its full rows and how it scored. Tab *Entities*: *Which records became one customer?* Entities by size, then those of two records or more, largest first, or a record ID's; one opens the links that joined its records (a rule, the model, you, or your conditions) and their full rows. Tab *Records*: *Which record?* A record ID's full row as the file gave it, its entity, and the pairs it is in. Tab *How it works*: the stages, as on *Find matches* | `list scored`, `list entities`, `show records`, `show details`, `describe entity`, `explain pair`, `describe results`, `describe matching`, `describe key`, `list patterns`, `compare records` |
-| *Method*, under Setup | *Is the matching good enough yet?* From the pairs people checked: not known yet, so far yes (with *Review pairs*), or how many were wrong, with *Look at them*: the pairs people said differ, in *Explore*. *What could be better?* Each change worth trying in words, with what it would do, and *Try*; a trial tried, with *Adopt*; where to draw the line, folded; the rounds, rules, pair keys and comparison steps folded ([section 6](#improve-the-method)) | `describe method`, `try method`, `adopt method`, `reset method`, `describe key` |
+| *Method*, under Setup | *Is the matching good enough yet?* From the pairs people checked: not known yet, so far yes (with *Review pairs*), or how many were wrong, with *Look at them*: the pairs people said differ, in *Explore*. *What could be better?* Each change worth trying in words, with what it would do, and *Try*; a trial tried, with *Adopt*; where to draw the line, folded. *How many records should the model learn from?* The fit as set, and *Try sample sizes*. *Which fields should EM learn from?* As tuned, chosen by each run, or the fields named, and *✦ Choose EM fields*: each session's pairs, its share agreeing on the given name, kept or dropped and why, then *Find matches*. The rounds, rules, pair keys and comparison steps folded ([section 6](#improve-the-method)) | `describe method`, `try method`, `adopt method`, `reset method`, `try sample`, `set fit`, `choose em`, `describe key` |
### Full rows
diff --git a/er/cli.py b/er/cli.py
index 536852d..61d7758 100644
--- a/er/cli.py
+++ b/er/cli.py
@@ -820,7 +820,7 @@ def describe_workspace_(a: dict, emit: "Emit") -> dict:
told = out / "auto" / "report.md" # the last auto mode's report, to send
auto = dict(report=str(told), markdown=told.read_text(encoding="utf-8")) if told.is_file() else None
return {**d, "inputs": files, "columns": "own" if columns_file(out) else "project", "auto": auto, "index": index.status(out, s), "rerun_owed": owed,
- "details": details.described(out, s.data.unique_id), "next": then}
+ "details": details.described(out, s.data.unique_id), "diagnosis": folder.diagnosis(out), "next": then}
def describe_entity_(a: dict, emit: "Emit") -> dict:
@@ -1460,9 +1460,11 @@ def open_ui(a: dict, emit: "Emit") -> dict:
"details": "the details file people see above each record (load details), or null",
"stamp": "what people and runs change, by its files' sizes and times (sheets, decisions, answers, method, mapping, entities): a page coming back reads again when it moves",
"ready": "whether a run can start on the file loaded last: run (true or false), why not, and the fix",
+ "diagnosis": "what limits what the last run finds, as report.txt's DIAGNOSIS says it, the most pairs first: what, pairs (at stake), fix (a command, or empty); null before a run that kept it",
"next": "the commands that usually follow"},
"nayose describe workspace workspaces/customers", describe_workspace_, quick=True,
- show={"largest": ("entity", "records"), "runs": ("at", "what", "seconds"), "inputs": ("name", "files", "bytes", "read"), "auto": (), "index": ()}),
+ show={"largest": ("entity", "records"), "runs": ("at", "what", "seconds"), "inputs": ("name", "files", "bytes", "read"), "auto": (), "index": (),
+ "diagnosis": ("what", "pairs", "fix")}),
Command("describe", "entity", "One record's entity, and what joined it",
"The entity of the record ID: its size, each of its records (the first 200, the ID among them) with its fields and "
"kind, and every link between them: a rule, the model with its weight in bits, people, or the workspace's merge "
diff --git a/er/folder.py b/er/folder.py
index 9312ac5..2780509 100644
--- a/er/folder.py
+++ b/er/folder.py
@@ -168,6 +168,12 @@ def history(out: Path) -> list[dict]: # every run steps.json holds, in brief, t
return [*old.get("earlier", []), *map(brief, old.get("runs", []))]
+def diagnosis(out: Path) -> list[dict] | None: # what limits what the run finds, as report.txt says it: the last report's items; None before a run that kept them
+ old = json.loads((out / STEPS).read_text(encoding="utf-8")) if (out / STEPS).exists() else {}
+ step = next((s for x in reversed(old.get("runs", [])) for s in x["steps"] if s["step"] == "report"), {}) # decisions write no report
+ return step.get("diagnosis")
+
+
def log(out: Path, entry: dict, append: bool, earlier: list[dict] | None = None) -> str:
"""steps.json with this entry: one per run, increment or set of decisions (a full run starts it again, with `earlier`,
the runs before it in brief: history, read before its write replaced the old log). The text, for stage to write with
diff --git a/er/report.py b/er/report.py
index b547c03..ad06163 100644
--- a/er/report.py
+++ b/er/report.py
@@ -203,9 +203,10 @@ def feature_path(field: str) -> str | None: # where set feature puts a field's
return next((f"{g}/{f['name']}" for g, feats in DEFAULT_FEATURES for f in feats if f.get("field") == field), None)
-def diagnosis(run: "Run", tables: dict[str, pd.DataFrame], lo: float) -> list[str]:
- """What limits what the run finds, the most pairs at stake first, at most SHOWN lines, each what, a number and the
- fix (a command where there is one); what is fine is left out: EM sessions of the run's own fit that learned
+def diagnosis(run: "Run", tables: dict[str, pd.DataFrame], lo: float) -> list[dict]:
+ """What limits what the run finds, the most pairs at stake first, at most SHOWN items, each what (with its number),
+ the pairs at stake and the fix (a command where there is one): report.txt prints them (diagnosis_lines), steps.json
+ keeps them for describe workspace. What is fine is left out: EM sessions of the run's own fit that learned
households, compared fields under SPARSE filled (with a column no field reads that looks like one), pairs below the
review floor that agree exactly on a name and another field, levels that learned nothing yet hold pairs, what the
search leaves (the gap at the review floor, oversized blocks skipped that came within NEAR bits), and, last, EM
@@ -280,15 +281,19 @@ def diagnosis(run: "Run", tables: dict[str, pd.DataFrame], lo: float) -> list[st
if thin := [(x, n) for x, n in kept if n < THIN]:
fix = f"nayose choose em {WS} --records {2 * m['fit_rows']}" if m.get("em_choice") else f"nayose try sample {WS}"
out.append((0, f"EM learned from few pairs: {', '.join(f'{x} {n:,}' for x, n in thin)} → {fix}"))
- lines = [x for _, x in sorted(out, key=lambda x: -x[0])[:SHOWN]] or ["nothing found"]
+ return [dict(what=what, pairs=int(n), fix=fix) for n, x in sorted(out, key=lambda x: -x[0])[:SHOWN] for what, _, fix in [x.partition(" → ")]]
+
+
+def diagnosis_lines(items: list[dict]) -> list[str]: # report.txt's DIAGNOSIS, from diagnosis's items
+ lines = [x["what"] + (f" → {x['fix']}" if x["fix"] else "") for x in items] or ["nothing found"]
return ["DIAGNOSIS (what limits what the run finds, the most pairs at stake first)", *(f" {x}" for x in lines), ""]
-def report_text(run: "Run", tables: dict[str, pd.DataFrame], pipeline: Path) -> str:
+def report_text(run: "Run", tables: dict[str, pd.DataFrame], pipeline: Path, items: list[dict] | None = None) -> str:
"""The run on a page of plain text that holds no record's value (no ID, name, phone or address: the notes that
quote one are run.examples, in qa.md only), so it can leave the machine that holds the records. `tables` is qa()'s.
It shows what ran (the code, the tracked files changed in this copy, the pipeline, the machine's limits), what limits
- what the run finds (diagnosis), the summary, each field's fill and shape, the columns no field reads, scores and accepted links by weight with the false links the
+ what the run finds (diagnosis's `items`, made here when not given), the summary, each field's fill and shape, the columns no field reads, scores and accepted links by weight with the false links the
model expects, each score band's commonest patterns, cluster sizes, each step's time and memory, its time in each string comparison, the model (the prior, the pairs each EM session learned from, and m, u and bits per level), the costliest
blocks, and the patterns no key pairs. O(C + E) over scores and matches."""
con, fs = run.con, run.pipeline.fs
@@ -314,7 +319,7 @@ def pct(x: object) -> str:
f"{threads} threads · memory limit {memory} · spill room {run.spill_room() / 1e9:,.0f} GB · "
f"took {sum(x['seconds'] for x in run.steps):,.1f} s · peak {max(x['peak_memory_gb'] for x in run.steps):.2f} GB", ""]
- out += diagnosis(run, tables, lo)
+ out += diagnosis_lines(diagnosis(run, tables, lo) if items is None else items)
summary = tables["summary"][tables["summary"]["what"] != "example"]
out += ["SUMMARY", *(f" {what:<12} {value}" for what, value in summary.itertuples(index=False)), ""]
diff --git a/er/run.py b/er/run.py
index 6fd2512..8bb31b1 100644
--- a/er/run.py
+++ b/er/run.py
@@ -77,7 +77,7 @@
from er.kind import KIND_COLUMNS, NAME_MODEL, century, classifier as kind_classifier, kind_review_sheet, merge_answers
from er.match import ELSE, bound_tests, rough_reads, compared_exactly, fit_model, fit_sample_sql, named_sql, bound_reads, reaching_sql, refreshed_u, review_floor, rule_edges, search_floor, score_fs, scored_u, train_fs, u_basis
from er.probe import missed
-from er.report import carry_kinds, carry_over, no_run, qa, report_text, review_sheet, unmapped_columns
+from er.report import carry_kinds, carry_over, diagnosis, no_run, qa, report_text, review_sheet, unmapped_columns
from er.explain import scored_again
from er.review_state import trust_state
from er.results import BANDS, own_merging, banded, check_merging, merge_where
@@ -883,7 +883,8 @@ def report(self, path: str | Path, out: Path, pipeline: Path) -> tuple[list[tupl
returned as texts and frames for write() to stage with them: a run never lands without its reports, which a
retry could not make again, since nothing is new by then. `path` is the input, `pipeline` the pipeline file, both
named in the reports. Keeps qa's tables, the summary first, in self.tables, and its weight bands (counts) in the
- step's facts (steps.json), for describe matching and describe method."""
+ step's facts (steps.json), for describe matching and describe method, with what limits what the run finds
+ (report.diagnosis), for describe workspace."""
t = time.perf_counter()
tables = qa(self, previous=out / "review.parquet" if self.added is not None else None)
sheet = tables.pop("review_sheet")
@@ -898,9 +899,10 @@ def report(self, path: str | Path, out: Path, pipeline: Path) -> tuple[list[tupl
frames.append((carry_kinds(self, kind_review_sheet(self.con, "records", self.uid, self.pipeline.kind, self.settings.flag_company, answered),
out / "kind_review.parquet"), out / "kind_review.parquet"))
self.tables = tables
- texts = [("\n".join(text), out / "qa.md"), (report_text(self, tables, pipeline), out / "report.txt")]
+ items = diagnosis(self, tables, review_floor(self.pipeline)) # report.txt prints them, describe workspace reads them here
+ texts = [("\n".join(text), out / "qa.md"), (report_text(self, tables, pipeline, items), out / "report.txt")]
bands = json.loads(tables["weight_bands"].round(4).to_json(orient="values")) # as qa.md shows them: describe matching and method read them here
- self._done("report", "O(review pairs) + O(N)", f"O({len(frames[0][0]):,} review pairs + {self.n:,})", t, weight_bands=bands)
+ self._done("report", "O(review pairs) + O(N)", f"O({len(frames[0][0]):,} review pairs + {self.n:,})", t, weight_bands=bands, diagnosis=items)
return texts, frames
# ---------- increments ----------
diff --git a/er/ui.html b/er/ui.html
index 8651b68..9efc78e 100644
--- a/er/ui.html
+++ b/er/ui.html
@@ -84,6 +84,7 @@
.qa > .q { flex: 1 1 420px; min-width: 0; }
.qa h2.question { font-size: 22px; font-weight: 400; margin: 0 0 4px; }
.qa .answer { font-size: 15px; margin: 0; color: var(--ink); } .qa .answer b { font-weight: 600; } .qa .answer .muted { color: var(--muted); }
+.limits { margin-top: 12px; } .limit { padding: 10px 0; border-top: 1px solid var(--line); } .limit .equiv { margin: 8px 0 0; }
.qa .big, .qa h2.question.big { font-size: 28px; line-height: 1.25; font-variant-numeric: tabular-nums; }
.checks { display: flex; flex-direction: column; gap: 6px; font-size: 14px; }
.checks > div { display: flex; gap: 10px; align-items: baseline; } .checks > div::before { width: 14px; flex: none; text-align: center; font-weight: 700; }
@@ -652,7 +653,7 @@
{"command": "add records", "args": {"input": "@input", "workspace": "@workspace", "compute": "@compute"}, "form": ["input"], "reads": ["summary"]},
{"command": "match records", "args": {"input": "@input", "workspace": "@workspace"}, "form": ["input"], "reads": ["counts", "records"]},
{"command": "rerun records", "args": {"workspace": "@workspace", "compute": "@compute"}, "form": [], "reads": ["summary"]},
- {"command": "describe workspace", "args": {"workspace": "@workspace"}, "reads": ["records", "entities", "pairs", "trust", "inputs"]},
+ {"command": "describe workspace", "args": {"workspace": "@workspace"}, "reads": ["records", "entities", "pairs", "trust", "inputs", "diagnosis"]},
{"command": "list patterns", "args": {"workspace": "@workspace"}, "reads": ["pairs", "fields", "patterns"]},
{"command": "compare records", "args": {"first": "@first", "others": "@others", "workspace": "@workspace"}, "reads": ["first", "records", "candidates"]},
{"command": "explain pair", "args": {"first": "@first", "second": "@second", "workspace": "@workspace"}, "reads": ["weight", "prior", "pairs_all", "pairs_same", "common", "same_customer", "household", "records", "comparisons", "tests", "found", "group_a", "group_b", "across"]},
@@ -683,6 +684,7 @@
{"command": "try method", "args": {"workspace": "@workspace", "line": "@line", "drop": "@drop", "add": "@add", "compute": "@compute"}, "reads": ["trial"]},
{"command": "try sample", "args": {"workspace": "@workspace", "compute": "@compute"}, "reads": ["steps", "settled", "floor", "report"]},
{"command": "set fit", "args": {"workspace": "@workspace", "records": "@records", "em": "@em"}, "reads": ["method"]},
+ {"command": "choose em", "args": {"workspace": "@workspace", "records": "@records", "compute": "@compute"}, "reads": ["records", "sessions", "kept", "wide", "method"]},
{"command": "adopt method", "args": {"workspace": "@workspace", "trial": "@trial", "compute": "@compute"}, "reads": ["summary"]},
{"command": "reset method", "args": {"workspace": "@workspace", "compute": "@compute"}, "reads": ["summary"]},
{"command": "describe key", "args": {"name": "@name", "workspace": "@workspace"}, "reads": ["name", "pairs", "matched", "only", "examples", "sizes", "largest", "group", "around"]},
@@ -1808,7 +1810,7 @@
Search
${item("lookup", "search", "Look up")} (COMMANDS[c] || {}).steps || []; // a run's steps, in the order they end, as the CLI declares them
const BUSY = {"resolve records": "Finding matches", "rerun records": "Matching again", "add records": "Adding the new records", "match records": "Previewing the new records",
- "load records": "Loading your file", "upload file": "Uploading your file", "generate records": "Generating records", "accept repairs": "Accepting the repairs", "try everything": "Auto mode", "try method": "Trying a method", "try sample": "Trying sample sizes", "set fit": "Setting the fit",
+ "load records": "Loading your file", "upload file": "Uploading your file", "generate records": "Generating records", "accept repairs": "Accepting the repairs", "try everything": "Auto mode", "try method": "Trying a method", "try sample": "Trying sample sizes", "set fit": "Setting the fit", "choose em": "Choosing EM's fields",
"set merging": "Rebuilding the entities", "apply answers": "Matching again with your answers", "apply decisions": "Applying your decisions",
"adopt method": "Matching with the method tried", "reset method": "Matching with the tuned method", "lower trust": "Lowering the trust line",
"raise trust": "Raising the trust line", "split entity": "Splitting the entity"};
@@ -2913,7 +2915,7 @@
Same customer? ${p.said ? statusChip(p.status) : ""}
}
$("flowq").textContent = "How did it find matches?";
$("out").innerHTML = `
${esc(nameOf(file))} · ${plural(file.rows, "record")}, not matched yet
` : ""}`; // a file loaded since: added, or matched alone
$("out").querySelector("[data-first]")?.addEventListener("click", () => start("resolve records", "Replace the run with them",
@@ -3025,7 +3027,7 @@
Same customer? ${p.said ? statusChip(p.status) : ""}
return w;
},
- async improve(el) { // the loop: what people's decisions say about the method; a change tried, adopted, and the new round reviewed
+ async improve(el, arg) { // the loop: what people's decisions say about the method; a change tried, adopted, and the new round reviewed; arg: a button to press once drawn ({press, records})
const foot = `
Same customer? ${p.said ? statusChip(p.status) : ""}
}
};
$("wrong")?.addEventListener("click", () => go("results", {tab: "pairs", status: "different"})); // the pairs people said differ, in Explore
+ if (arg?.press && $(arg.press)) { if (arg.records) $(arg.press).dataset.records = arg.records; $(arg.press).scrollIntoView({block: "center"}); $(arg.press).click(); } // a fix the run's page asked for
},
pairs: (el, arg) => pairsPage(el, arg),
kinds: (el, note) => review(el, "kinds", note),
};
+// What limits what the run finds (describe workspace's diagnosis, report.txt's DIAGNOSIS): the most pairs first, each
+// with its fix; one the Method page runs is a button there, another its command line to copy. None kept: no card.
+const FIXES = {"choose em": "choose-em", "try sample": "sizes"};
+function diagCard(items, ws) {
+ if (!items) return "";
+ if (!items.length) return `
${ask("What limits what the run finds?", "Nothing found.")}
${ask("What limits what the run finds?", `${plural(items.length, "limit")}, the most pairs first`)}
${items.map(row).join("")}
`;
+}
+
// How large a sample the model needs (try sample): pairs at the floor as each sample grows, both ways EM learns.
-const FIT_EM = {rules: "EM as set", keys: "EM per key"};
+const FIT_EM = {rules: "EM as set", keys: "EM per key", choose: "EM chosen by the given name"};
+const fitEm = (em) => FIT_EM[em] || `EM per ${[].concat(em).join(", ")}`; // a list of fields, or a word set fit takes
+const fitNow = (fit) => fit ? `${fmt(fit.records)} records, ${fitEm(fit.em)}` : "As tuned.";
+const emNow = (fit) => !fit ? "As tuned." : fit.em === "choose" ? "Each run chooses them by the given name." : Array.isArray(fit.em) ? `${fit.em.map(words).join(", ")}.` : `${fitEm(fit.em)}.`;
+function emView(box, r) { // choose em: each session kept or dropped and why, and what the next run does
+ if (r.event === "error") { box.innerHTML = banner(r); return; }
+ box.innerHTML = `
${icon("check")}
${r.kept.length ? `Kept now: ${r.sessions.filter((x) => x.kept).map((x) => words(x.session)).join(", ")}` : "None kept now: EM as set"}.
snack('Copied', 3000), () => {});
if (e.target.closest("[data-pick]")) pick();
if (e.target.closest("[data-auto]")) autoMode();
});
diff --git a/tests/test_cli.py b/tests/test_cli.py
index d9d740f..7019d3d 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -835,13 +835,13 @@ def test_a_customer_id_held_by_many_records_is_a_placeholder_every_step_and_the_
def test_the_diagnosis_names_what_limits_the_run_the_most_pairs_first_each_with_its_fix(tmp_path, population):
from er.match import _ranked, review_floor, search_floor
- from er.report import SHOWN, diagnosis, qa
+ from er.report import SHOWN, diagnosis, diagnosis_lines, qa
from er.run import Run
s, p, g = population
g.df.to_parquet(tmp_path / "in.parquet", index=False)
run = Run(p, s).full(tmp_path / "in.parquet", tmp_path / "run")
tables, lo, m = qa(run, [f for f in s.columns if f not in ("id", "entity")]), review_floor(run.pipeline), run.scoring["er"]
- said = lambda: diagnosis(run, tables, lo)[1:-1]
+ said = lambda: diagnosis_lines(diagnosis(run, tables, lo))[1:-1]
top = {c["output_column_name"]: max(g for g, _ in _ranked(c)) for c in run.scoring["comparisons"]}
assert said() and len(said()) <= SHOWN
tables["fill"].loc[tables["fill"]["field"] == "birth_date", "filled"] = 0.1 # a compared field mostly empty, and a column like it
@@ -929,6 +929,27 @@ def test_the_run_report_counts_each_score_bands_patterns(reported):
assert shown == stored and len(lines) > 3
+def test_describe_workspace_gives_the_diagnosis_report_txt_prints(reported, capsys):
+ from er import cli
+ from er.report import diagnosis_lines
+ _, text, out = reported
+ assert cli.main(["describe", "workspace", str(out), "--format", "json"]) == 0
+ items = json.loads(capsys.readouterr().out.splitlines()[-1])["diagnosis"]
+ assert isinstance(items, list) and all(set(x) == {"what", "pairs", "fix"} for x in items)
+ assert [x["pairs"] for x in items] == sorted((x["pairs"] for x in items), reverse=True) # the most pairs first
+ assert section(text, "DIAGNOSIS")[1:] == diagnosis_lines(items)[1:-1] # one source: report.txt prints the items the page reads
+ was = (out / "steps.json").read_text(encoding="utf-8")
+ steps = json.loads(was)
+ for x in steps["runs"]: # a run an older version wrote: no items, and none said
+ x["steps"] = [{k: v for k, v in s.items() if k != "diagnosis"} for s in x["steps"]]
+ (out / "steps.json").write_text(json.dumps(steps), encoding="utf-8")
+ try:
+ assert cli.main(["describe", "workspace", str(out), "--format", "json"]) == 0
+ assert json.loads(capsys.readouterr().out.splitlines()[-1])["diagnosis"] is None
+ finally: # the module's run as it was, for the tests after
+ (out / "steps.json").write_text(was, encoding="utf-8")
+
+
def test_the_run_report_says_what_each_em_session_learned_from(reported):
run, text, _ = reported
em = run.model["er"]["em_pairs"]
diff --git a/tests/test_ui_page.py b/tests/test_ui_page.py
index 07c50c7..2f7bb9f 100644
--- a/tests/test_ui_page.py
+++ b/tests/test_ui_page.py
@@ -1512,3 +1512,82 @@ def test_a_list_shows_the_answer_to_the_last_thing_asked_whichever_answer_comes_
page.wait_for_timeout(1500)
page.screenshot(path=str(tmp_path / f"{screen}.png"), full_page=True)
assert page.evaluate(mine), screen # the record's pairs stay: the answer asked before them, come late, is dropped
+
+
+def test_the_run_s_page_says_what_limits_it_and_the_method_page_chooses_em_s_fields_once_per_click(ui_page, tmp_path, population):
+ from er import cli, folder
+ s, p, g = population
+ g.df.to_parquet(tmp_path / "all.parquet", index=False)
+ ws, home = tmp_path / "run", tmp_path / "workspaces"
+ assert cli.main(["resolve", "records", str(tmp_path / "all.parquet"), "--workspace", str(ws), "--format", "json"]) == 0
+ items = folder.diagnosis(ws)
+ page, server, httpd = ui_page(str(ws), str(home), viewport={"width": 1280, "height": 900})
+ shot = lambda name: page.screenshot(path=str(tmp_path / f"{name}.png"), full_page=True)
+ page.wait_for_selector(".tiles")
+ page.click(".navitem[data-screen=run]")
+ page.wait_for_selector("#out h2.question.big")
+ said = page.inner_text("#out")
+ assert "What limits what the run finds?" in said and (all(x["what"] in said for x in items) if items else "Nothing found." in said) # report.txt's items
+ assert "--workspace …" not in said
+ shot("diagnosis-light")
+ page.emulate_media(color_scheme="dark")
+ shot("diagnosis-dark")
+ page.emulate_media(color_scheme="light")
+
+ page.click(".navitem[data-screen=improve]")
+ page.wait_for_selector("#choose-em")
+ assert page.inner_text("#em-now") == "As tuned."
+ shot("choose-em-before")
+ sent = []
+ page.on("request", lambda r: r.url.endswith("/api/run") and json.loads(r.post_data)["command"] == "choose em" and sent.append(r))
+ page.dblclick("#choose-em") # pressed twice: one fit
+ assert page.is_disabled("#choose-em")
+ page.wait_for_selector("#em-out .banner", timeout=120_000)
+ assert len(sent) == 1 and folder.method(ws)["fit"]["em"] == "choose" and not page.is_disabled("#choose-em")
+ assert page.inner_text("#em-now") == "Each run chooses them by the given name." and "EM chosen by the given name" in page.inner_text("#out")
+ rows = page.query_selector_all("#em-table tbody tr")
+ assert rows and all(r.query_selector(".chip").inner_text() in ("kept", "dropped") for r in rows)
+ shot("choose-em-after")
+ page.emulate_media(color_scheme="dark")
+ shot("choose-em-after-dark")
+ page.emulate_media(color_scheme="light")
+
+ page.click("#em-out .banner a[data-go=run]") # the next run chooses them itself
+ page.click("[data-act='rerun records']")
+ page.click("#dlg button[value=run]")
+ page.wait_for_selector("#out .steps-card", timeout=10_000)
+ page.wait_for_selector("#out .steps-card", state="detached", timeout=120_000)
+ page.wait_for_selector("#out h2.question.big")
+ after = folder.diagnosis(ws)
+ assert after is not None and ("Nothing found." in page.inner_text("#out") if not after else all(x["what"] in page.inner_text("#out") for x in after))
+ shot("diagnosis-after-rerun")
+ page.evaluate("""() => { const box = document.createElement("div"); box.id = "probe"; document.querySelector("#out").append(box); // a fix the Method page runs: its button
+ box.innerHTML = diagCard([{what: "EM learned from few pairs: phone 40", pairs: 0, fix: "nayose choose em --workspace … --records 4000"},
+ {what: "no key lists about 50 pairs", pairs: 50, fix: "nayose describe method --workspace …"}], state.workspace); }""")
+ assert page.query_selector("#probe [data-press=choose-em][data-records='4000']") and f"describe method --workspace {ws}" in page.inner_text("#probe code")
+ shot("diagnosis-buttons")
+ page.click("#probe [data-press=choose-em]") # to the Method page, its button pressed with the records asked
+ page.wait_for_selector("#em-out .banner", timeout=120_000)
+ assert folder.method(ws)["fit"] == {"records": 4000, "em": "choose"}
+
+ steps = json.loads((ws / "steps.json").read_text(encoding="utf-8")) # a run an older version wrote: no card, no error
+ for x in steps["runs"]:
+ x["steps"] = [{k: v for k, v in st.items() if k != "diagnosis"} for st in x["steps"]]
+ (ws / "steps.json").write_text(json.dumps(steps), encoding="utf-8")
+ page.click(".navitem[data-screen=run]")
+ page.wait_for_selector("#out h2.question.big")
+ assert "What limits" not in page.inner_text("#out") and not page.query_selector("#out .banner.bad")
+ assert cli.main(["create", "workspace", "empty", "--home", str(home), "--format", "json"]) == 0 # a workspace without a run: no card
+ page.evaluate(f"setWorkspace({json.dumps(str(home / 'empty'))})")
+ page.click(".navitem[data-screen=run]")
+ page.wait_for_selector("#out .card")
+ assert "What limits" not in page.inner_text("#out")
+ page.evaluate(f"setWorkspace({json.dumps(str(ws))})")
+ page.click(".navitem[data-screen=improve]")
+ page.wait_for_selector("#choose-em")
+ httpd.shutdown() # the server lost: said above the page, the button back
+ httpd.server_close()
+ page.click("#choose-em")
+ page.wait_for_selector("#lost", timeout=30_000)
+ page.wait_for_selector("#choose-em:not([disabled])", timeout=30_000)
+ shot("choose-em-lost")