Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,8 @@ Scores every candidate, with the u step 3 measured on this file: for each field,

- **Why:** bits add up, so every score can be explained field by field.
- **Code:** `Run.score` → `match.score_fs`, `scaled_prior` (λ at this file's size)
- **Increment:** only the pairs touching the batch that can reach 0 are scored, as above, a slice at a time: `O(C_Δ·K·L²)`.
- **The floor:** a pair is scored and stored only if it can reach the workspace's floor: 0 points, or `method.toml` `keep` (`set floor`). A lower floor keeps pairs people can review that 0 drops, such as a moved customer with a new ID, for more pairs scored and stored. The oversized blocks' gate keeps its own floor (step 5).
- **Increment:** only the pairs touching the batch that can reach the floor are scored, as above, a slice at a time: `O(C_Δ·K·L²)`.

### 7 · Cluster: from pairs to entities · `O(C + E·d + N)`

Expand Down
26 changes: 26 additions & 0 deletions docs/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ Written by `scripts/document_commands.py` from the declarations in `er/cli.py`;
| [`try method`](#nayose-try-method) | Try a change of the method beside the method as it is |
| [`try sample`](#nayose-try-sample) | Find how large a sample the model needs to learn from |
| [`set fit`](#nayose-set-fit) | Set the sample the model learns from, and how EM learns |
| [`set floor`](#nayose-set-floor) | Set the points from which the run keeps pairs |
| [`choose em`](#nayose-choose-em) | Choose the fields EM learns from, by the given name |
| [`try everything`](#nayose-try-everything) | Try everything on a file, and write a report to send that holds no record's value |
| [`adopt method`](#nayose-adopt-method) | Adopt a method tried, and find matches with it |
Expand Down Expand Up @@ -1217,6 +1218,31 @@ Keeps them in the workspace (method.toml fit): the records the next run's fit sa
nayose set fit --workspace workspaces/customers --records 64000 --em keys
```

## nayose set floor

Set the points from which the run keeps pairs.

```
nayose set floor --workspace WORKSPACE --points POINTS [--pipeline PIPELINE] [--format text|json]
```

Keeps it in the workspace (method.toml keep): a pair whose best case can't reach it is never compared in full or stored, so people can't review it. 0 unless set; lower keeps more pairs and costs more. The next run uses it. O(1).

| Parameter | |
|---|---|
| `--workspace WORKSPACE` | the workspace: the folder that keeps one matching job (required) |
| `--points POINTS` | the points to keep pairs from, for example -10 (required) |
| `--pipeline PIPELINE` | the pipeline file (default: config/final.toml) |

| Result | |
|---|---|
| `method` | the workspace's method.toml |
| `next` | the commands that usually follow |

```bash
nayose set floor --workspace workspaces/customers --points -10
```

## nayose choose em

Choose the fields EM learns from, by the given name.
Expand Down
2 changes: 1 addition & 1 deletion docs/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ reads the file before these steps, about 15 s a million records from CSV, 7 s fr
| 9 | report: qa.md, report.txt and the review sheet, built before the write and landing with it | `Run.report` | O(review pairs) + O(N) | N | 0.78 s |
| | **total** | | | | **53.0 s** |

**Pairs that can't reach 0 points.** Nothing reads a candidate's score below 0: `scored/` keeps 0 points and up, and the line, the trust steps and review all start there. So a pair is scored only when its bound reaches 0: the prior, each comparison whose tests are equality, Hamming or one edit at its own level, and each other at the best level the pair may land in (`bound_tests`): DuckDB settles each level's test where it can in C++, a name or a company name in katakana by Jaro–Winkler on its one-byte copy, and only a test it can't settle (text outside katakana and ASCII, an edit distance past one) counts as passing. 0 where its fields are missing. Each test is computed once: the outcomes index a table of the bounds they allow, since DuckDB computes an expression as often as it is written. A pair a rule found is scored whatever its bound. A full run's candidates step already leaves the rest out as its keys list them (below); an increment's are counted (`unscored`, one row "below 0" in report.txt's SCORES and the score chart) and never compared by name, company or address: on a file whose address and phone are mostly empty, that is nearly every candidate. Every score written is the same to the last bit. The gate on oversized blocks and the probe do the same at their own floors, the review floor and the line. K_c is the comparisons settled first, C₀ the candidates whose bound reaches 0.
**Pairs that can't reach 0 points.** Nothing reads a candidate's score below the floor: `scored/` keeps 0 points and up (or `method.toml` `keep`), and the line, the trust steps and review all start there. So a pair is scored only when its bound reaches the floor: the prior, each comparison whose tests are equality, Hamming or one edit at its own level, and each other at the best level the pair may land in (`bound_tests`): DuckDB settles each level's test where it can in C++, a name or a company name in katakana by Jaro–Winkler on its one-byte copy, and only a test it can't settle (text outside katakana and ASCII, an edit distance past one) counts as passing. 0 where its fields are missing. Each test is computed once: the outcomes index a table of the bounds they allow, since DuckDB computes an expression as often as it is written. A pair a rule found is scored whatever its bound. A full run's candidates step already leaves the rest out as its keys list them (below); an increment's are counted (`unscored`, one row "below 0" in report.txt's SCORES and the score chart) and never compared by name, company or address: on a file whose address and phone are mostly empty, that is nearly every candidate. Every score written is the same to the last bit. The gate on oversized blocks and the probe do the same at their own floors, the review floor and the line. K_c is the comparisons settled first, C₀ the candidates whose bound reaches 0.

At a million records, scoring the candidates (16.05 s) and the fit (9.40 s) take nearly half the time. The fit costs
nearly the same at any size: it learns on n records, and its u sample of the file is S pairs for each of name, company
Expand Down
2 changes: 1 addition & 1 deletion docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,7 @@ None of these runs anything: they read the workspace's files.
| `review.parquet` | pair for people | `why` (`decide` or `audit`); an empty `decision`; both records' compared fields; per comparison, the level reached and its bits; `accepted`, whether the run joined the pair; `household`, whether it may be two people of one household ([section 6](#6--review-and-decide)) | decide the next range below the trust line and audit above it, or, with no line, decide the pairs just below the threshold and those above it with no name compared, and audit those just above ([section 6](#6--review-and-decide)) |
| `kind_review.parquet` | record for people | `why` (`decide` or `measure`), `reason` in words, `engine_says`, `name_says`, `flag` and `birth_date` if the file has them, the name; `engine` and `engine_p` (the engine's kind and its probability of a company), `name_p`, `drawn`; an empty `kind` | answer 1 person, 2 company or 0 can't tell ([section 6](#6--review-and-decide)) |
| `decisions/` | decided pair | both IDs and `decision`; `why` (`decide`, `audit`, or `split` from `split entity`) when there is one; `round`, the round it was decided in; one part per file people gave back | what people decided; a full run applies it again |
| `method.toml` | | the workspace's changes to the tuned method: `trust` (the trust line), `merge` (the conditions under which pairs merge without people), `line`, `drop_rules`, `drop_keys`, `add_keys` | every command on the workspace runs the method it says ([section 6](#the-trust-line)) |
| `method.toml` | | the workspace's changes to the tuned method: `trust` (the trust line), `merge` (the conditions under which pairs merge without people), `line`, `drop_rules`, `drop_keys`, `add_keys`, `fit`, `keep` (the points from which pairs are stored, 0 unless `set floor` sets it) | every command on the workspace runs the method it says ([section 6](#the-trust-line)) |
| `rounds.json` | round | per method the workspace ran: its `config` hash, `method`, when, the links it made and the false links the model expects | see each round's results side by side |
| `trials/` | trial | each trial's result (`<n>.json`, the last 5); the clusters of the method it was tried against (`base.parquet`, `base.json`) | `adopt method --trial n` |
| `index/` | key of a record | every key of every record and every searched value, sorted (`<id>.duckdb`); the records as compared (`<id>.records.duckdb`); what changed since (`<id>-<n>.duckdb`); the person/company classifier's fitted part (`<id>.kind.pkl`); `manifest.json`, which names them and what they rest on | `match record`, `search records`, the *Look up* page |
Expand Down
17 changes: 16 additions & 1 deletion er/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -683,6 +683,15 @@ def set_fit(a: dict, emit: "Emit") -> dict:
return write_fit(a, fit_records(a), em)


def set_floor(a: dict, emit: "Emit") -> dict: # method.toml's keep, checked, then written
ws, points = Path(a["workspace"]), number(a["points"], "points", fix="a number of points, for example -10 or 0")
with folder.run_lock(ws): # read, checked and written as one
method = {**folder.method(ws), "keep": points}
setup(a, method) # refused before anything is written
folder.write_method(ws, method)
return {"method": method, "next": [dict(command=nayose("rerun", "records", "--workspace", ws), why="find matches with it")]}


def choose_em(a: dict, emit: "Emit") -> dict:
s, p, _ = setup(a)
records = fit_records(a) if a.get("records") else p.fit_rows
Expand Down Expand Up @@ -1631,6 +1640,12 @@ def open_ui(a: dict, emit: "Emit") -> dict:
Param("em", "rules: as the method has it; keys: one session per equality key; choose: as choose em, at each fit; or fields, comma-separated: one session per field", required=True, metavar="rules|keys|choose|FIELDS"), PIPELINE),
{"method": "the workspace's method.toml", "next": "the commands that usually follow"},
"nayose set fit --workspace workspaces/customers --records 64000 --em keys", set_fit),
Command("set", "floor", "Set the points from which the run keeps pairs",
"Keeps it in the workspace (method.toml keep): a pair whose best case can't reach it is never compared in full or "
"stored, so people can't review it. 0 unless set; lower keeps more pairs and costs more. The next run uses it. O(1).",
(WORKSPACE, Param("points", "the points to keep pairs from, for example -10", required=True, metavar="POINTS"), PIPELINE),
{"method": "the workspace's method.toml", "next": "the commands that usually follow"},
"nayose set floor --workspace workspaces/customers --points -10", set_floor),
Command("choose", "em", "Choose the fields EM learns from, by the given name",
"Fits the model on the run's fit sample (--records of them, default the method's), against the u the run measured, "
"in one EM session per field a comparison's first level tests for equality (an identifying field left out). "
Expand Down Expand Up @@ -1878,7 +1893,7 @@ def open_ui(a: dict, emit: "Emit") -> dict:
"apply answers", "describe merging", "set merging", "list entities", "describe entity", "split entity")),
("Look up", ("build index", "match record", "search records")),
("Look inside", ("describe workspace", "describe matching", "describe key", "list patterns", "explain pair", "compare records")),
("Method", ("describe method", "try method", "try sample", "set fit", "choose em", "adopt method", "reset method", "describe review", "lower trust", "raise trust", "draw pairs", "list pairs")),
("Method", ("describe method", "try method", "try sample", "set fit", "set floor", "choose em", "adopt method", "reset method", "describe review", "lower trust", "raise trust", "draw pairs", "list pairs")),
)


Expand Down
10 changes: 7 additions & 3 deletions er/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -435,6 +435,7 @@ class ThresholdRule:
max_fp_rate_hi: float
margin_steps: int
search_floor: float = 0.0 # bits: the oversized-block gate and the gap aim here (er.match.search_floor); the idle-key check, the review floor
keep_from: float = field(default=0.0, repr=False) # bits: a pair the best case says can't reach it is never stored (method.toml keep); in the run's hash only when set (run_config)


@dataclass(frozen=True, kw_only=True)
Expand Down Expand Up @@ -726,7 +727,7 @@ def rekeyed(b: Blocker, old: str, new: str) -> Blocker: # the generator, readin
fs=replace(p.fs, model=model, blockers=p.fs.blockers + tuple(c for b in p.fs.blockers for c in copies.get(b.name, ()))))


METHOD_KEYS = ("line", "drop_rules", "drop_keys", "trust", "add_keys", "merge", "fit")
METHOD_KEYS = ("line", "drop_rules", "drop_keys", "trust", "add_keys", "merge", "fit", "keep")


GIVEN, PERSON = "given_name", "person_name" # agreement on the given name tells one person's records from a household's; the full name, where it is sparse (er.match.chosen)
Expand Down Expand Up @@ -754,7 +755,7 @@ def adjust(p: "Pipeline", method: Mapping[str, object]) -> "Pipeline":
`fit`, the sample the model learns from: `records` (fit_rows) and `em`, "rules" (em_rules), "keys" (one session
per equality key, each level's m their median), as try sample shows them, "choose" (one session per compared field,
kept by the given name: em_candidates), or a list of fields (`field` or `field:normalizer`), one session blocked on
each, in place of em_rules.
each, in place of em_rules; `keep`, the points from which the run stores pairs (keep_from, default 0).
Nothing changed: the same pipeline, so the same config hash. O(rules + generators)."""
if not method:
return p
Expand All @@ -764,6 +765,9 @@ def adjust(p: "Pipeline", method: Mapping[str, object]) -> "Pipeline":
line = method.get("line", p.fs.min_match_weight)
if isinstance(line, bool) or not isinstance(line, int | float) or not all(isinstance(method.get(k, []), list) for k in METHOD_KEYS[1:3]):
raise InputError("method.toml: line is a weight in bits, drop_rules and drop_keys lists of names", fix)
keep = method.get("keep", p.threshold.keep_from)
if isinstance(keep, bool) or not isinstance(keep, int | float) or not math.isfinite(keep):
raise InputError("method.toml: keep is the points from which pairs are stored, for example -10", fix)
trust = method.get("trust")
if trust is not None and trust != "none" and (isinstance(trust, bool) or not isinstance(trust, int | float) or float(trust) not in p.trust.steps):
raise InputError(f'method.toml: trust is "none" or one of the steps {list(p.trust.steps)}', fix)
Expand Down Expand Up @@ -795,7 +799,7 @@ def adjust(p: "Pipeline", method: Mapping[str, object]) -> "Pipeline":
model = replace(model, em_rules=sessions)
return replace(p, rules=replace(p.rules, stages=stages), fs=replace(p.fs, blockers=blockers, min_match_weight=float(line), model=model),
trust=p.trust if trust is None else replace(p.trust, line=math.inf if trust == "none" else float(trust)),
fit_rows=fit.get("records", p.fit_rows))
fit_rows=fit.get("records", p.fit_rows), threshold=replace(p.threshold, keep_from=float(keep)))


def load_pipeline(path: Path, std: Standardization, chains: Mapping[str, tuple[Normalizer, ...]],
Expand Down
Loading
Loading