Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ Writes each table as Parquet under a temporary name, with the reports (9), and `
`Run.report` builds, and step 8 writes with the run's files: `qa.md`, the label-free checks; `report.txt`, the run on one page with no record's value in it, which can leave the machine that holds the records, headed by what limits what the run finds (`report.diagnosis`: [usage](usage.md#7--when-something-is-wrong)); `review.parquet`, with the evidence for each pair; and, on a full run that tells persons from companies, `kind_review.parquet`, the records for people to answer person or company ([usage](usage.md#6--review-and-decide)). The review sheet's *decide* rows are every pair of clusters whose best link scored in the threshold rule's safety margin: from the `[threshold]` grid point `margin_steps` below the threshold (4 bits) up to it, and every pair above it with no name compared. Its *audit* rows are about 50 of the pairs the run accepted (`matches`), up to 4 bits above it. With a trust line (`report.trust_sheet`), the *decide* rows are a random sample of the next range down instead, `[trust] checks` less those decided, and the *audit* rows `watch` spot-checks above the line.

- **Why:** the margin holds pairs that were clean on cross-validated train, held back in case of noise: on the synthetic val split, 55 pairs from 4 to 6 bits, none of them false; one grid step lower, at 2 bits, val's first false pairs appear. So people decide them. Without labels, the audit rows and the largest clusters are how a person catches over-merging.
- **Code:** `Run.report` → `report.qa` (`report.fill_rates`, `match.review_floor`, `report.review_sheet`, `report.carry_over`, `match.pattern_coverage`), `report.report_text`, `kind.kind_review_sheet` (its rows picked in DuckDB, so only they reach Python), `io.markdown`
- **Code:** `Run.report` → `report.qa` (`report.fill_rates`, `match.review_floor`, `report.review_sheet`, `report.carry_over`, `match.pattern_coverage`), `report.diagnosis` (kept in the step's facts in `steps.json`, for `describe workspace`), `report.report_text` (`report.diagnosis_lines`), `kind.kind_review_sheet` (its rows picked in DuckDB, so only they reach Python), `io.markdown`
- **From a written folder:** `report.describe_workspace` (what the folder holds), `report.describe_entity` (one record's entity and what joined it), and `report.describe_matching`: the pipeline as a graph, each stage with its method, parameters and fields, and the run's numbers (pairs per rule and how the model scores them, per generator its pairs, links and those only it found, each level's m, u, bits and pairs, the weight bands with what happened in each, the review sheet's patterns, entities and where to look first) with real examples; `explain.describe_key` inspects one pair key (records per value, one group). `nayose describe workspace`, `describe entity`, `describe matching` and `describe key` print them, and `nayose open ui` draws them.
- **Increment:** the same for the pairs the batch made, plus the previous sheet's rows still open: a run keeps no scores, so a row dropped from the sheet could never come back. With a trust line, the range people check, its checks and its false pairs are counted over the workspace's links too, less the open rows kept, so an increment asks for no check a full run wouldn't (`report.trust_sheet`). No `kind_review.parquet`: a record an increment holds back is on the next full run's sheet.

Expand Down
1 change: 1 addition & 0 deletions docs/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -878,6 +878,7 @@ Records and entities, the largest entities, each run so far, the pairs and kinds
| `details` | the details file people see above each record (load details), or null |
| `stamp` | what people and runs change, by its files' sizes and times (sheets, decisions, answers, method, mapping, entities): a page coming back reads again when it moves |
| `ready` | whether a run can start on the file loaded last: run (true or false), why not, and the fix |
| `diagnosis` | what limits what the last run finds, as report.txt's DIAGNOSIS says it, the most pairs first: what, pairs (at stake), fix (a command, or empty); null before a run that kept it |
| `next` | the commands that usually follow |

```bash
Expand Down
2 changes: 1 addition & 1 deletion docs/diagrams/full-run.puml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ package "every step after load reads records" as after_load {
rectangle "<b>7 · Cluster</b>: merge the rules' and the model's pairs at the trust line (none set: every rule pair, and the model's at its threshold with a name compared), or, with the workspace's merge conditions, the pairs that meet them; people's matches, less their no_match; join them into entities, from single records\n<i>O(C + E·d + N)</i>\n<size:10><color:#65717E>Run._load_decisions · folder.latest_decisions · folder.parts · Run.cluster · Run._accept · Run.scored · match.named_sql · Run._links · results.merge_where</color></size>\n<size:10><color:#65717E>cluster.merge · Run._gold · run.customer_kinds · Run._expected_false · Run._alone · Run._line · Run._sized · match.review_floor · results.own_merging</color></size>" as cluster
rectangle "<b>7b · Probe</b>: the matches no key lists: pairs drawn within groups matches share (a phone, a postal area), scored; those at the line the likely links don't join, counted and scaled\n<i>O(N + P·K + L)</i>\n<size:10><color:#65717E>Run.probe · probe.missed · Run._sized</color></size>" as probe
rectangle "<b>8 · Write</b>: save the answer, the pairs at 0 points or more with each comparison's level (scored/), the state for increments and the reports (9, built first); the old run goes only once all is written\n<i>O(N + C)</i>\n<size:10><color:#65717E>Run.write · config.generator_dict · Run._links · Run.scored · match.named_sql · folder.next_part · folder.stage · folder.finish_writes · folder.history · folder.brief · Run._log · folder.log · io.code_version · io.replace_text · io.patiently</color></size>\n<size:10><color:#65717E>the round its method is in: folder.round_ran · folder.rounds · folder.method · config.toml_of · Run._alone · Run._sized</color></size>" as write
rectangle "<b>9 · Report</b>: the checks and the sheets for people, what they typed in and haven't given back kept, saved with the run (8)\n<i>O(review pairs) + O(N) in SQL</i>\n<size:10><color:#65717E>Run.report · report.qa · report.trust_sheet · review_state._trust_links · review_state.applied · report.fill_rates · match.review_floor · match.named_sql · report.review_sheet · report.households · report.carry_over</color></size>\n<size:10><color:#65717E>match.pattern_coverage · match.levels · io.markdown · kind.kind_review_sheet · report.carry_kinds</color></size>\n<size:10><color:#65717E>report.report_text · match.scaled_prior · source.filled · match.parameters · config.sources_text · io.code_version · io.read_parquet</color></size>\n<size:10><color:#65717E>report.diagnosis · report.feature_path · match._ranked · match.search_floor</color></size>" as report
rectangle "<b>9 · Report</b>: the checks and the sheets for people, what they typed in and haven't given back kept, saved with the run (8)\n<i>O(review pairs) + O(N) in SQL</i>\n<size:10><color:#65717E>Run.report · report.qa · report.trust_sheet · review_state._trust_links · review_state.applied · report.fill_rates · match.review_floor · match.named_sql · report.review_sheet · report.households · report.carry_over</color></size>\n<size:10><color:#65717E>match.pattern_coverage · match.levels · io.markdown · kind.kind_review_sheet · report.carry_kinds</color></size>\n<size:10><color:#65717E>report.report_text · match.scaled_prior · source.filled · match.parameters · config.sources_text · io.code_version · io.read_parquet</color></size>\n<size:10><color:#65717E>report.diagnosis · report.diagnosis_lines · report.feature_path · match._ranked · match.search_floor</color></size>" as report
}
rectangle "<b>4b · Keys</b>: with [keys], the keys learned on this file: the pairs the model accepts, found by sampling every way two records agree, listed by the fewest pairs. Without, the method's generators, but one whose agreement weighs nothing and whose sampled pairs at the review floor other keys or rules all list\n<i>O(A·N + f·N·b + K·T + K'·N)</i>\n<size:10><color:#65717E>Run.keys · keys.learn · keys.idle · match.review_floor · source.known</color></size>" as keys
rectangle "<b>7c · Gap</b>: what the keys miss against comparing every pair: the keys step's sample of records (only those that could reach the floor), each paired with every record sharing a value or a near one, scored; the pairs at the search floor, the review floor, the line and 20 bits no key or rule listed, counted and scaled\n<i>O(f·N·b)</i>\n<size:10><color:#65717E>Run.gap · keys.gap</color></size>" as gap
Expand Down
2 changes: 1 addition & 1 deletion docs/diagrams/full-run.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading