feat(structured): tables declared on the ontology, one node for a row and its prose mention - #328
Merged
Merged
Conversation
Every data edge is written with `type == "RELATES"` and its semantic type in a `rel_type` property. Four places decided whether two edges were the same edge using only `(start, end)` or `(start, "RELATES", end)`, so they read as "one edge per entity pair" and silently discarded every fact after the first. - deduplicator: the survivor is now bound by its own MATCH before the MERGE. Written inline, `s` is unbound and MERGE creates the entire pattern, forking the survivor into a second type-label-less node that takes the remapped edges while the real survivor is left with none. Reproducible from three CREATEs. - deduplicator: the RELATES MERGE is keyed on rel_type. Without it a survivor's WORKS_AT and a duplicate's FOUNDED collapse onto one edge, destroying a fact and leaving its source_chunk_ids attached to the other. - graph_store: same key on the write path, batch and per-item fallback both. - resolution: the dedup key includes properties["rel_type"]. exact_match had an inline copy of remap_relationships that had drifted, so both carried the same bug; it now calls the shared helper. coalesce(rel_type, '') keeps the merge key non-null: FalkorDB rejects a MERGE keyed on a null property, which would abort the whole remap for an untyped edge. Tests: 6 new, each verified to fail without the fix. Two guard against overshoot, since a finer key must still dedupe identical facts.
…e deleting it The remap migrates edges only, so DETACH DELETE took the duplicate's properties with it. That silently lost whatever only the duplicate knew: the `description` entity vector search embeds, and every typed value a structured source supplied. Observed end to end: a graph built from CSVs, then one document ingested. The merge picked the prose node as survivor and every structured property was gone afterwards, with no error. age, title, hq_country, employee_count all deleted. Properties are copied with keep_existing semantics, so a value already on the survivor always wins and a merge can never overwrite what the survivor knew. `id` and `embedding` are never carried. Both merge paths call it, exact and fuzzy.
…euse it Moves _build_lexical_graph and _write_mentions out of IngestionPipeline into a base that depends on nothing but self.graph_store. Verbatim move, no behaviour change; IngestionPipeline now inherits them. A structured pipeline cannot subclass IngestionPipeline: its __init__ requires a chunker and an LLM extractor, and a structured source has neither, so subclassing means passing None and relying on which methods happen to be called. Sharing a base instead keeps both paths writing byte-identical Document, Chunk and MENTIONED_IN shapes, which is what lets one retrieval path serve both. The full suite passes unchanged, which is the point of this commit. Also fixes the mock in test_deduplicate_entities_merges_duplicates, which enumerated exactly six query_raw results. The property carry added in the previous commit makes a seventh call, so the mock ran dry and the delete failed silently. The list is now explicit about which call each entry answers.
Text-to-Cypher rows reached the answering LLM as bare values with the
column names discarded. For a row of entity names that is survivable; for
an aggregate it is not. `RETURN avg(p.age) AS average_age` arrived as the
single token "39.5", which the final model cannot attribute to anything,
so it answered "the context does not provide enough information" while the
query it had just run returned the correct number.
Two changes, both in the path between execution and the prompt:
- Each value is labelled with its returned column name, so the aggregate
reads `average_age: 39.5`. Nodes and maps are left alone: they already
read as described things, and prefixing them would only add noise.
A driver that omits the header degrades to unlabelled rows rather than
failing the retrieval, since a header is never essential.
- The section names the question the query was generated from. A labelled
scalar is still unscoped on its own ("39.5 for whom?"), and saying which
question the rows answer is what makes it usable.
Measured on the same question and graph: "not enough information" before,
"39.5" after. Only reachable with `enable_cypher=True`, which is off by
default, so no existing retrieval path changes.
The survivor of a merge was chosen by longest description alone. When one of the duplicates has an id derived from a declared key, that rule picks the wrong node, and the damage only appears on the next ingest. A node written from a mapping is keyed on a column the source declared, so every later ingest of that table recomputes the same id. A node extracted from prose is keyed on whatever surface form the model produced, and prose usually wins on description length. So the prose node survives, the keyed id stops existing, and re-ingesting a corrected export recreates it as a second node: measured as two `E-1` people, one titled "Engineer" and one "engineer", each holding half the facts and neither complete. The rank is now, in order: derived from a declared key, then real over placeholder, then longest description. `is_stub` is the marker because only a mapped source writes it, so `None` means the node came from extraction. The same rule replaces array order in the semantic phase, where the survivor was simply whichever id appeared first. For a corpus with no structured sources every candidate ranks (0, 1, len), which is the previous ordering exactly, so prose-only graphs are unaffected.
Ingest a CSV through the extraction pipeline and it loses the two things a
table has that a document does not: a stable key per row and a known type
per column. Every cell arrives as a described entity, so `age` is the
string "34" if it survives at all, and nothing can be averaged, filtered
numerically, or joined on a key. Measured on a real run before this: the
`age` property did not exist on any node, and all eight entities carried
the identical key set an extracted entity always has.
This adds the other route. The caller declares what the columns mean and
the graph is written deterministically, with no model in the loop:
ORGS = Table("Organization", key="org_id", name="org_name",
employee_count=Column("employee_count", "INTEGER"))
await rag.ingest("orgs.csv", mapping=ORGS)
A record is a chunk. That is what makes the two halves the same shape: a
row is retrievable and traceable to its source exactly as a paragraph is,
while the typed projection lives on the entity where aggregation reads it.
Records are deliberately not chained with NEXT_CHUNK, because rows have no
reading order and cypher_generation tells the model that edge means "the
next sequential Chunk".
How a row and a sentence become one node: a source holding both a key and
a name publishes the id an extractor would independently compute for the
same thing, in `alias_ids`. `finalize()` then folds them with the ordinary
resolver and carries the typed columns onto the survivor. The bridge is
exact string equality on computed ids, never similarity, so it cannot merge
two different people. Variant names (`A. Smith`) still do not bridge; that
is a floor, not an oversight.
Three things the mapping layer refuses, each because the failure is silent
and remote from its cause:
- `name` cannot be declared as a property. The extraction path merges
ontology-declared attributes over the system properties it just built, so
a declared `name` invites the extractor to answer it with a null for
every prose mention. Symptom: CSVs ingest fine, then documents produce
nameless nodes that resolve into nothing. Found by measuring it.
- A record stream is a factory, not an iterator. The write path walks the
records twice; a one-shot iterator is empty the second time and ingests
zero rows without raising.
- A mapping is validated against the source's real header before anything
is written, so a mapping that does not fit leaves the graph untouched.
A foreign key is written as a reference: ON CREATE only, carrying its key
so it is joinable, flagged `is_stub` until the source that owns the entity
arrives. Order between sources does not matter.
Ontology registration is additive. The store refuses to modify an existing
label, which is right for extraction but wrong here, since `employees.csv`
may declare `Organization` by key alone and `orgs.csv` later add
`employee_count` to it. New labels go through `register()`; existing ones
are extended through the evolution primitives, which still raise on a type
contradiction.
The declared ontology is what makes the columns queryable: text-to-Cypher
can see that `age` is an INTEGER instead of guessing everything is a
described entity. On a mixed graph, all three classes of question are now
answerable through one `completion()` call, including the one that needs
both halves ("who works at the company that reported the revenue miss, and
how old are they") which no single source can answer.
Includes `examples/11_structured_ingestion.py`, a docs page, 71 unit tests
and 15 integration tests against a live FalkorDB.
Two problems with one cause: the record loader invented its own convention for a Document id. It defaulted to the bare file name while the text path derives `os.path.normpath(source)`. So `ingest(path, mapping=M)` filed the source under "orgs.csv" and `update(path, mapping=M)` would look for "/tmp/x/orgs.csv" and not find it — and its own docstring claimed the name was "the stable handle update() and delete_document() use", which was exactly backwards. It also collided: two same-named exports in different directories became one Document, silently interleaving their rows. Also adds `RecordMapping.fingerprint`, a digest of the declaration itself, for the update path to fold into its content hash. Identical rows under a changed mapping produce a different graph, so without it, adding a column to a declaration would look like unchanged data and be skipped. And makes the types say what construction already guarantees: `alias` is non-optional once set (`handle`), and `properties` holds Columns after normalisation (`typed_properties`). Both were mypy errors under the repo's strict config, and the second is now read once per source instead of once per record — for a large table that is millions of redundant dict builds.
A row deleted from a source stayed in the graph forever. Re-ingesting an export rewrote the rows that changed and added the rows that appeared, but a departed row had nothing left to rewrite it, so it persisted with an orphaned record chunk still attached to its Document. Measured on a live graph: re-ingested a two-row CSV with one row removed, and the removed organization was still there with the document still reporting two chunks. `update()` was no help — it rejected `mapping` outright with a TypeError, so the SDK's whole incremental-update feature did not apply to tables. `update(..., mapping=...)` now drives the existing crash-safe state machine. Only the content-producing phase differs: the records are written under the pending Document and everything after that — the cleanup-state write, the commit marker, the cutover, the scoped orphan sweep — is the same code the prose path runs. Phases 3b through 6 moved into `_finish_update` so there is one copy rather than two; the ordering in there *is* the crash-safety contract, and a second copy would drift without anything failing until a crash landed in the gap. Deletion then falls out rather than being implemented: a departed row has no new chunk, so the cutover deletes its old one and Phase 6 removes the entity — unless another source still mentions it, because the sweep is scoped to this document's candidates. Writing under the pending id is load-bearing: record chunk ids derive from the effective document id, so pointing them at the canonical id would MERGE them onto the very chunks the cutover is about to delete. Ingesting a source that is already present now routes through that same path. A table is a snapshot, not an addition, so "ingest this export again" means "this is its current state" — which is also the call people reach for first, and leaving it as an in-place write both kept departed rows and doubled a source's chunks (measured: two chunks for one row, because the post-cutover ids are pending-derived). Also fixes the sync wrappers, which is why this class of bug survived: `ingest_sync` had no `mapping` parameter at all despite a docstring saying to keep it in step, and neither `ingest` overload mentioned it, so a type checker rejected the exact call the docs teach. There was already a tripwire test for wrapper drift covering update / delete_document / apply_changes — `ingest` was simply not in its list. It is now.
A mapping declares its columns into the ontology so generated Cypher can see their types. That also puts them in front of the extractor, which then answers them from prose. Measured: a job title arrived as "engineer" from a memo and overwrote the "Engineer" the HR export spelled, because the merge keeps the survivor's value and prose had written it first. This is the same shape as the earlier `name` bug, where declaring an attribute the SDK owns let the extractor null it out. That one was fixed by not declaring `name`. These columns have to stay declared — that is the whole point of declaring them — so instead they now have an owner. Extraction may not write a property a mapping declared. Only the value is discarded: the entity, its name, its description and every unclaimed property are exactly as extracted, and what was dropped is logged, because silently discarding a value looks identical to never extracting one. Ownership is persisted as `Property.structured` in the ontology graph, so it survives a reload rather than holding only in whichever process ran the ingest, and it is sticky: a later declaration that omits the flag cannot hand the column back to the extractor. This implements the recommended answer to an open question (does the table or the document win?) rather than a settled one. It is deliberately one guard in one place, `_drop_structured_properties`, so reversing it or softening it to fill-only-when-absent is a local change. Found while testing it: rebuilding GraphData by listing fields dropped `mentions`, which silently cost every extracted entity its MENTIONED_IN edges — a merged node that no longer remembered the document it came from, visible only as missing provenance two steps later. Uses `model_copy` now, so a field added in future cannot go the same way.
Declaring that a column is an INTEGER exists so that a question can be answered by querying instead of by finding a passage that states the answer. Two things stood between that and anyone using it. The option was not on the client. The documented way to get an aggregate was to hand-build a MultiPathRetrieval out of `rag._graph_store` and `rag._vector_store` — our own example and docs page instructed people to reach into internals, which is both a bad advertisement and a promise we never made. `GraphRAG(enable_cypher=True)` now does it. Still off by default: it costs an extra LLM call per question and wants an ontology with real property types, which is exactly what a structured source provides. And the strategy was holding an ontology from before the data existed. It was constructed in `__init__` and refreshed only on the ontology's first load, so every later change was invisible to it — including every structured ingest, which is precisely when typed columns get declared. Measured: after ingesting a CSV the facade knew `Organization.employee_count` while the strategy's copy of Organization had no properties at all. Enabling Cypher alone would have generated queries blind to the columns the mapping existed to declare. The working ontology is now a property whose setter republishes to the strategy, so all seven assignment sites stay in step rather than the one that happened to remember. Strategies adopt it through a new `set_ontology` hook on the base class, a no-op by default so a strategy that ignores the ontology need not implement it. Adds `GraphRAG.query()` while removing the last reason to reach past the facade: there was no public way to read your own graph, which is why the examples verified their own work through private attributes. It runs the Cypher it is given, with parameters, and returns rows. The example and docs page now use both, and neither touches an underscore.
A review pass over the whole path, probing what the tests had not: hostile names, non-unique keys, wrong-path updates, and size. Four real problems, two of them silent data loss. **An update could re-interpret a table as prose.** `update(path)` without the mapping loaded the CSV as text, replaced its record chunks with one text chunk, and deleted every entity with them. Measured: two organizations before the call, none after, nothing raised. The same call arrives from `apply_changes(modified=[...])`, so a scheduled sync would have emptied a table on its first run. A Document already recorded how it was written; it is now read back and the mismatch refused in both directions, with apply_changes surfacing it per entry. **Rows sharing a key disappeared.** A chunk's id came from the row's key, so two rows keyed K1 produced one chunk holding the first row's cells while the ingest reported two records. Each row now keeps its own chunk, and both passes over the source take their chunk ids from one shared walk so they cannot drift. A key that is not unique is logged with the column and the repeated values, because it usually means the wrong column was declared. Ids for unique keys are byte-identical to before, so nothing moves in graphs that already exist. **Writing n nodes cost O(n^2).** Nothing indexed `id`, and a MERGE can only use an index on the label in its own pattern — indexing `__Entity__.id` does not help, because that label arrives in a later SET. Measured on a structured ingest: 4k rows 15.2s, and 50k projected to hours. With a range index per written label, created before the write rather than after: 4k in 1.9s, 50k in 38.5s, memory flat because records stream. Prose gains too, where the cost was hidden by documents contributing few nodes each. Labels come from mappings and ontologies, so they are indexed on first write rather than declared up front. **A declaration that could not be written failed deep in the driver.** A property named with a backtick surfaced as `DatabaseError: Invalid input at end of input`, from a query the caller never wrote. Property names and relationship types must now be identifiers, since generated Cypher writes them bare; the escape for an awkward column is to name the property and point it at the column. Labels stay permissive — `Legal Entity` and `Ünïcode` are fine — but are refused when the sanitiser would silently rewrite them into something the graph holds under a different name. Injection itself held: a hostile label was sanitised, a hostile key value was slugged into an id, and the graph stayed intact in both cases. Two casting fixes while there: `FLOAT` rejects `nan` and `inf`, which are valid literals that turn an `avg()` over the whole column into NaN with nothing to point at; `LIST` is parsed as a CSV row so a quoted element may contain a comma instead of silently becoming two. Docs carry the measured throughput table, the naming rules, the repeated-key behaviour and the update guard. Three graph-store tests asserted a raw query count and now assert the write they care about, so they are not tripwires for an unrelated index.
Found by running an outside team's demo of this feature. It reported `entities_deduplicated: 0` and its own instrumentation said no question ever saw context from both halves, while the graph looked fully populated. The cause is that ingest order silently decided whether a table and a document could ever join. Resolution matches on name *and* label, which is what stops "Apple" the company merging with "Apple" the fruit. But an extractor can only label an entity with a label the ontology already has. Read a document before any mapping is declared and "Carbon Farming" is filed under a guessed `Concept`; the CSV then declares it a `MitigationPractice`; the labels differ, so resolution correctly refuses to merge them. Two nodes, same name, no error, and the feature's whole premise quietly absent. Measured on their files with only the order changed: prose first merged 0, tables first merged 5. The fix is not to document the ordering. A mapping *declares* that a name is a `MitigationPractice`; the extractor only *guessed* `Concept`, from a list that did not yet contain the real label. Those are the same thing described by two sources, one of which knows. So exactly one cross-label merge is now allowed: when a name exists under one label a mapping declared and one or more labels nothing declared, the declared label survives and absorbs the rest, keeping the document's description and mentions. This is the rule that already governs declared columns, applied to the label. The guard that matters is untouched. Two *declared* labels sharing a name are a real modelling conflict, not a guess to correct, and are left alone. A name under only undeclared labels is the Apple case and is left alone too. Both are now reported as `FinalizeResult.unmerged_name_collisions` rather than showing up as a bare `deduplicated=0` with no explanation. On their demo, unchanged and in its original order: 0 merges before, 4 after, and no silent collisions left. Also adds `GraphRAG(mappings=[...])` and `declare_mapping()`, which register a mapping's labels and column types without writing data. That was my first attempt at a fix and it is the wrong shape for one — it needs the caller to know — but it remains worth having: the extractor then has the real labels while it reads, and gets more of them right first time. Same corpus, 9 merges instead of 4. Quality, not correctness.
Declaring a table took two shapes. `Table(...)` for one record, one entity;
`RecordMapping(nodes=[NodeMapping(...)], edges=[EdgeMapping(...)])` once a
record produced more than one. The second was strictly more capable and the
first was sugar over it, which sounds harmless until a table turns out to
have a foreign key in it. Then the sugar has to be thrown away and
re-expressed in a different shape: the worst kind of seam, because the
moment you learn something new about your data you pay for having started
simple. It also made the surface twice as large to explain, which it
demonstrably was.
`Table` is now the only form and it grows by argument:
ORGS = Table("Organization", key="org_id", name="org_name",
employee_count=Column("employee_count", "INTEGER"))
EMPLOYEES = Table("Person", key="employee_id", name="full_name",
age=Column("age", "INTEGER"),
links=[Link("WORKS_AT", to="Organization", by="org_id")])
`Link` says the thing worth saying about a table: `org_id` is not text, it is
an edge. It carries the relationship, the label it points at and the column
holding the key, plus optionally the target's denormalised name and
properties for the edge itself. The target is still written ON CREATE only,
so a pointer can never overwrite the source that owns the entity.
`Table` is also a class now rather than a function wearing a capital letter,
so the linter suppression goes with it. `RecordMapping` stays as its base and
the internal representation the pipeline consumes; `NodeMapping`,
`EdgeMapping` and `RecordMapping` are no longer exported, because nothing a
caller writes needs them. Public signatures say `Table`.
Two links to the same label get distinct handles automatically, which is the
case that used to require hand-written aliases. A link on the record's own
key is refused, since it would join a record to itself.
Doing this now because the branch is unreleased with one known consumer, and
their three mappings already use `Table`, so nothing of theirs changes.
Review pass over what changed since the last audit: the Table/Link refactor and the declared-label adoption. One real defect. Three links to the same label failed. Handles are derived, and the disambiguator was the relationship type, which collides the moment two links share one — `PARTY_TO` twice to `Organization` produced two nodes wanting the handle `Organization__PARTY_TO`. It surfaced as "duplicate node aliases", naming a concept the caller never wrote and cannot set. The disambiguator is now the column, which is the thing that actually differs between two links. Two links genuinely naming the same target by the same column are refused with that said plainly. The adoption pass deletes nodes, so it got its own battery: edges moving in both directions with their direction intact, provenance moving so the survivor still remembers its document, property carry-over with the declared value winning, two declared labels left alone, two guessed labels left alone, a guessed node with no declared twin left alone, and finalize twice being a no-op. All correct; tests added for the ones a unit test can reach and an integration test for edge movement, which needs edges written directly because the scripted extraction fixture emits `"relationships": []` and a prose-driven version of it would pass while proving nothing. Re-ran the earlier adversarial battery on the unified API: hostile labels, property names and relationship types are all refused at declaration; a hostile key value is still slugged into an id with the graph intact; rows sharing a key still keep a chunk each; updating a table as prose is still refused; a deleted row still leaves. Scale is unchanged by the refactor: 50,000 rows in 38.6s against 38.5s before, memory flat. One thing left as-is and worth naming: when two *guessed* labels share a name and both carry a description, which description lands on the survivor depends on fetch order, since neither is authoritative. Same non-determinism the existing same-label dedup already has, and both values are equally valid.
A mapping written by hand chooses a label by hand, and a label chosen by hand can invent `Person` on a graph whose entities are all `Employee`. Both then exist, one real person is held as two nodes, and the facts split between them — resolution matches on name *and* label and will not join across the two, correctly, because nothing says they mean the same thing. `propose_mapping(source)` cannot do that, because it must choose a label the ontology already has. One model call, at authoring time, and it decides as little as possible. Everything a reading of the data settles is measured: the key column is the one unique and complete across the sample; a column's type is whatever every value actually parses as, widening on a single failure because a declared type is enforced at ingest; and a foreign key is found by computing the entity id its value would have and looking it up, which proves a reference instead of guessing from a column name. Labels are offered with how many entities each already holds. That leaves the model the part no measurement answers: which existing concept the rows describe, and what to call the relationships. A label outside the ontology is rejected and fed back through the same `extract_with_retry` contract ontology discovery already uses, so this is the repo's existing pattern applied one layer down rather than a new one. The result is a reviewable `MappingProposal` carrying the evidence for every choice, `as_code()` for committing, and `requested_new_label` when nothing fitted. Nothing is written and nothing is applied. Committing the code is the intended path: a proposal regenerated per run puts a model back in the ingest path, and a declared mapping exists so that none runs there. Writing the mapping yourself is unchanged and needs no model. Two silent routes into a bad graph are closed alongside it. A `.csv` with no mapping took the text path: the whole file became one chunk with its commas intact, an extractor pulled out whatever it noticed, and no column kept its type. Measured on a two-row export — one entity written, `age` absent entirely, nothing raised. It now raises and names both ways out, because the graph looks populated afterwards and a log line would not save anyone. Passing a loader explicitly stays the escape, and it is a real case: a table of support tickets or survey answers is prose that happens to live in columns. `TextLoader` is exported so that escape is reachable, which it was not when the message first recommended it. And a hand-written mapping introducing a label beside one already in use now logs which labels hold entities. Only a warning: declaring a new type is ordinary on a fresh graph, and refusing would make that tiresome to no purpose. What the author was missing is the list, not permission. Two defects found while building it. The first attempt chose `Person` over an `Employee` holding data, because `Person` is a built-in default and so legally in the ontology — constraining to the ontology is not enough while the pool holds both, which is what the entity counts fixed. The second was found by mypy: a column named `links`, `key` or `name` collides with `Table`'s own parameters and raised `TypeError: got multiple values for keyword argument`. Such a column is left unmapped and reported. Known limitation, documented rather than worked around: the choice is per file, not per column. A mapping never extracts from text, so a source like `ticket_id, customer, body` either keeps its typed identity and loses what `body` mentions, or gets the entities from `body` and loses the identity.
Reading that a table and a document end up on one node is not the same as watching it happen. This walks through it against a live graph and prints a query at every step to paste into the FalkorDB browser, so the claim is checkable rather than asserted. Generates its own data: two PDFs and three CSVs about a fictional energy company. The PDFs are written by a small PDF writer included in the notebook, because no PDF library is installed by default and a demo that needs one installed first is a demo people do not run. Nothing binary is committed and a re-run starts from an empty graph. The names are the substance of the example. `Northwind Energy`, `Maya Ellison` and `Kestrel Grid` appear identically in the prose and in the tables, because the join is exact string equality on the name — a table whose display names are phrases the documents never use joins to nothing, and that is the one design decision worth being explicit about in a demo. Covers the parts that are easy to claim and hard to show: a proposal fitted to the ontology that already exists, a hand-written mapping with a foreign key, two links to the same label from one row, the merge at `finalize()`, all three classes of question, a re-sync that removes a row deleted from the export, and both refusals. Verified by extracting the code cells and running them end to end: exit 0, six entities reachable from both a PDF and a CSV, `Northwind Energy` from five sources, and all four questions answered correctly, including the two that need both halves. Two things the run itself taught, now in the notebook. The helper rendered nodes as `<Node object at 0x...>`, which is useless in a notebook even though returning whole nodes is what draws the picture in the browser; it renders them as `Label(name)` now. And the model's half of a proposal genuinely varies between runs — one run made `country` a property, another linked it to `Location` — so the notebook says so and points at `as_code()` as the way to freeze one answer, which is also what keeps a load free of a model.
Validating the notebook step by step turned up a flaw in the notebook itself: it ended with two `Northwind Energy` nodes, which is precisely the split the whole walkthrough is about. Not an SDK fault. Step 9 demonstrates the explicit-loader escape by reading a ticket export as prose, that ticket mentions Northwind Energy, and `finalize()` had already run back in step 6. So a fresh entity was created for a company the graph already held, and nothing merged it. Which makes it worth showing rather than hiding, because it is the easiest call in the SDK to forget: `finalize()` belongs after the *last* ingest, and it is cheap to repeat. The notebook now says so and runs it, going from two Northwinds to one, with the ticket ending up on the same node as the two PDFs and the three CSVs. Verified by a 33-check validation of a run from an empty graph: files generated and their text extractable, join names verbatim in the PDF, the proposal reusing a label already in use and introducing no new type, employee_count arriving as an int rather than a string, WORKS_AT written, both sides of a contract reaching two different organizations, entities spanning a PDF and a CSV, the arithmetic in the table-only answer, the departed row deleted and the promoted row keeping its identity, an unchanged re-load being a no-op, both refusals, and no duplicate names left in the graph. 33/33, exit 0, about a minute end to end.
The notebook's questions answered correctly, and I had been treating that as proof the halves were joined. Checking what the generated Cypher actually does says otherwise: asked who works at the company that reported the shortfall, it produced `WHERE o.name CONTAINS 'shortfall'` and returned zero rows. The answer came from retrieval putting the prose chunk and the record chunks in the model's context separately, which works just as well on a graph where nothing merged at all. Which is the same weakness worth criticising in anyone's demo of this: an answer can look hybrid while the graph is two disconnected islands, and the answers alone cannot tell you which you have. So the notebook now ends with the version that can only work joined. Every clause of its WHERE comes from a different source — the document from a PDF, `org_id` from a column only the CSV supplied, `WORKS_AT` from an edge only the CSV created — and it walks from a sentence about a shortfall to two ages in an HR export. On a disconnected graph it returns nothing, which is what makes it worth running. Section 7 now says up front what it does and does not prove, and points here. Also states the test to apply to any demo of this: ask whether the question could still be answered if the halves were never joined. If it could, the demo is showing retrieval. Verified: 33/33 checks from an empty graph, exit 0, and both closing queries return what they claim — Maya Ellison 34 and Tomas Reyes 47 through the traversal, and both companies reached through prose answering with the CSV's headcount and revenue.
The notebook opened with 98 lines of hand-rolled PDF writer and inline CSV string literals before the reader saw a single SDK call. The point of the example is how tables and documents join, and none of that plumbing was about it. The seven files now live in examples/data/hybrid_walkthrough/ as real files on disk, with a README naming the shared spellings that make the join work. Step 0 is a directory listing. A reader can open the CSVs, edit them, or point the notebook at their own. The changed export is a committed employees_v2.csv rather than a cell that rewrote a tracked file, and both loads pass the same document_id, so the re-sync is explicit instead of a side effect of the filename. Also: *.pdf was globally ignored, so the two fixtures needed a narrow exception; and the WORKS_AT step claimed four edges where the graph holds six before finalize() — the board review asserts two of them in prose. Verified by executing the notebook itself with nbclient from an empty graph, three times: 0 error cells, entities_deleted=1, no_op=True on the unchanged reload, and the section 10 traversal still returns Maya Ellison and Tomas Reyes.
The narrative kept getting interrupted by code that was not about the SDK.
Cell 7 was 36 lines of `_render` / `look` / `counts` — printing nodes as
Label(name), wrapping Cypher, truncating row lists. Real work, but the same
kind of noise as the PDF writer that came out in the previous commit.
It now lives in examples/walkthrough.py, and the notebook imports three
names. Every code cell left in the notebook calls the SDK.
Also:
- dropped `await rag.__aenter__()`. It is `return self` — a no-op that
read like required setup.
- the browser URL and graph name are printed once at connect instead of
on every one of the ten `look()` calls.
- `glob("*.[cp][sd][vf]")` was a puzzle; it is a suffix check now.
- edges printed as <falkordb.edge.Edge object at 0x...>; they render as
-[WORKS_AT]-> so the rel_type is visible, which is the point of them.
- `textwrap` was imported at connect and used once, fifteen cells later.
- merged an orphan one-line markdown cell into the one above it.
19 code cells / 353 lines -> 17 / 234, with nothing removed from what the
notebook demonstrates. Executed with nbclient from an empty graph after
each change: 0 error cells, entities_deleted=1, no_op=True, and section 10
still walks from the board review to Maya Ellison and Tomas Reyes.
Each of these produced a graph that looked correct and reported success. Found by reviewing the feature against real exports rather than the fixtures it was built with, and every one is now a regression test. finalize() deleted whole tables. Step 1's legacy NULL-name cleanup was unscoped, so a fact export declared without a name column — a reading keyed on reading_id with a sensor code and a value — lost every row during the call the docs tell you to make. Measured two Reading nodes before and none after, reported only as a legacy-stub count. Now scoped to nodes no mapping wrote. key="id" cost every row its provenance. The key column was written verbatim, so `id` overwrote the node's graph id and MENTIONED_IN edges were never created: no provenance and no way to join to prose, on a load that reported success. Reserved and awkward headers now route through safe_property_name, so key="id" stores under col_id and the graph keeps its own. The same rule fixes a header containing a space, which used to reach the driver inside a parameter map and surface as a raw DatabaseError from a query the caller never wrote. Every number in a European export was out by a factor of ten. INTEGER and FLOAT stripped all commas, so 880,5 became 8805.0. They now decide the cases that are decidable — both separators present means the rightmost is the decimal point; a lone comma not followed by three digits is a decimal comma — and refuse the genuinely ambiguous 1,234 for FLOAT while reading it as grouping for INTEGER, where no fractional part is on offer. A blank key cell dropped its row and said nothing. records: 3 for a four-row file is true about what was written and false about what the file said. rows_skipped and rows_in_source now appear on the result when something was skipped, with a warning naming the column and the lines. One bad cell left a broken graph. The chunk pass committed the Document, a chunk per row and the content hash before the mapping pass ran, so a cast failure left a document of orphan chunks — and because the document record then existed, the retry compared hashes instead of writing. Both passes now complete before anything is written, which is what makes the documented "leaves the graph untouched" true. Two people called John Smith became one. Dedup groups by display name, so it merged them and deleted one, reporting the loss as a successful dedup. A declared key is an assertion of identity and now outranks a shared name. Also adds the report that was missing underneath all of it: near-miss names that probably denote one thing and did not merge are surfaced in FinalizeResult.probable_duplicates. Rules, not embeddings — measured, "same" scored 0.601-0.980 and "different" 0.706-0.930 over realistic pairs, so no threshold exists; the rules get 10/10 recall with no false positives. Reported and never merged: a merge in the graph does not hold, because the next read of that document recreates the node it deleted. 43 regression tests. 1387 passed, 1 pre-existing failure unrelated to this change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ows and prose Move the table declaration onto the ontology — Ontology(tables=[TableMapping(...)]) — so a mapping is registered before anything is extracted and the graph is independent of the order sources arrive in. A row and a mention now derive the same entity id from name and label, so a CSV row and a PDF mention of the same person land on one node at finalize(). Every property a table writes is signed with its source, so two tables disagreeing keep both values and prose can never overwrite a declared column. Also: an undeclared table is read as-is rather than refused, with a proposed mapping reported; re-syncing a source is the same ingest() call; the data-safety guards on the structured path; and the finalize() report (property_conflicts, probable_duplicates, proposed_mappings, unresolved_references, stale_signed_properties, mapping_changed, ...). Replaces the walkthrough script and notebook 12 with examples/13_documents_and_tables.ipynb. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…by default A table's "Priya Raman" and a note's "Ms. Raman" arrive in different ingest() calls, so the resolver that decides such pairs within one document never met them, and the graph-wide deduplicator only merged spellings. finalize() now runs a resolver over every entity at once — by default an LLMVerifiedResolution over the instance's llm and embedder, asking from cosine 0.6 because names differ more across sources than within one document. finalize(resolver=...) substitutes your own; finalize(resolve=False) reports instead of asking. The resolver decides identity; the merge keeps the table's node and every value it signed, and never folds two rows of one table into each other. A NO is remembered as a DISTINCT_FROM edge and listed in rejected_duplicates, so the same pair is not asked twice; name-rule near-misses are handed to the resolver whatever they scored; two rows of one table are declared distinct up front. The hand-off is four optional ctx.metadata keys on ResolutionStrategy. Also in this change: - a property a table owns is not offered to the extractor - mapping lookup by basename, so a source moved on disk keeps its declaration; the table's Document id is its basename too - propose_mapping() through the SDK's LLM with structured output; drop_table() - MockEmbedder uses a stable, zero-centred digest: hash() salted per process and all-positive components made unrelated names score as duplicates on some seeds Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
When the resolver judged `Austria` and `Republik Österreich` one country the survivor was chosen by description length, so the hub the tables pointed at (five rows, four mentions) was renamed after the one-mention citation and `c.name CONTAINS 'Austria'` came back empty. Degree (RELATES either way + MENTIONED_IN) now ranks ahead of description; a row or placeholder still outranks any prose node. Found running three papers and three tables from FalkorDB-POCs/GraphRAG-SDK-Docs-Example; PDFs-first went 24/25, the same order now 25/25. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Found by the wave-2 eval (12 tables, 6 PDFs, one JSON, dirty files). - References to one (label, key) within a batch derived different ids when one column named the target and another did not: two Paper placeholders for one arXiv id. Collapse them before the graph lookup — the row wins, then the named reference — and remap edges. Also lands a self-reference (manager_id) on its own file's rows. - Strip surrounding whitespace from keys before storing entity_key; the id derivation already ignored it, so `inst-10 ` derived the INST-10 id but could never be found by key. - LLMVerifiedResolution leaves apart any pair whose names differ only in digits (P-011 / P-021, GPT-3 / GPT-4): they embed above the hard threshold and the model is inconsistent on them. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The ontology fragment a mapping contributes describes a label it did not
create only by what the table did ("Declared by a structured source, keyed
on exp_id", "Referenced by key from a structured source"), and the store's
coalesce took that text over the description the user declared. Measured
on twelve tables: every declared description was gone after the first
load, and propose_mapping() asked to place an undeclared grants table chose
Experiment three times out of three, because Experiment now read "keyed on
exp_id" instead of "a field experiment measuring methane flux". The
extractor and text-to-Cypher read the same descriptions.
_register_structured_ontology keeps an existing label's description; a
label nothing has described still takes the table's note. The proposal
prompt now lists each label's description next to its properties. With
both, the same table is placed under a new Funding label three of three.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…es, JSON examples/14_research_group_knowledge_base: a reproducible corpus that fits the design about 80%. Twelve CSVs written by different people (keys that disagree, a bare junction table, an undeclared file, 'austria'/'Austria ', a ';'-joined FK cell), six arXiv PDFs (two connected only by prose, one that must stay an island), two notes and a JSON scheduler dump. download_pdfs.py six pinned arXiv ids, 15 MB, gitignored ontology.py 14 entity types, 11 TableMappings, each trait commented ingest.py prose -> tables -> JSON -> finalize(resolve=True) verify.py 32 expectations recomputed from the CSVs, checked in Cypher ask.py 18 aggregate / hybrid / lookup questions with sources used dirty_files.py six single-fault loads on <graph>_dirty README records what to expect (tables 6 s for 1,785 rows, PDFs 12-20 min, finalize 4 min; 6/6 aggregates exact) and the limitations the corpus exposes. CHANGELOG, docs and the examples table point at it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A foreign key now keys a node a document created first: upsert_reference_nodes adds its signed key ON MATCH, entity_key and is_stub via coalesce, and never the name. Node-level source_chunk_ids is a union on every write and on a resolver merge, and a deleted chunk's id is stripped by the post-cutover cleanup. drop_table() removes the signed key column from graph and ontology on every label the table wrote and releases entity_key/is_stub unless another table still keys the node. The description overwrite by a second document is left as is: with no per-chunk attribution an append could not be undone by update(), so it is the ownership question, not an operator bug. Unit tests pin the Cypher shapes; integration tests pin the behaviour against FalkorDB. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
Merged Conflict notes, since two of them were real overlaps:
CI: |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
VectorStore’s new _id_indices_ensured cache can incorrectly short-circuit id-range index creation after broader index cache resets, leading to missing pre-MERGE indexes in subsequent runs.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 57/58 changed files
- Comments generated: 1
- Review effort level: Lite
Contributor
|
[High] Failed structured writes are certified as complete, so retrying the same table skips the missing data (newly found).
Reproduction: Against isolated FalkorDB, ingest a one-row employee CSV with a Fix: Record the structured Document's hash only after all required writes complete. Propagate reported shortfalls through the structured result and update/cutover metadata so an incomplete write cannot receive a completion hash, and keep an ordinary retry eligible to repair it. Add failure-and-retry coverage for entity, provenance, and relation writes; the prose pipeline's complete-write gate does not currently protect this structured path. |
Naseem77
approved these changes
Sep 9, 2026
Blockers - rename_entity() follows through to the stored TableMapping, its links and link columns; drop_entity() refuses a label a table maps rows to or links to, naming the table and drop_table(). Before, a rename left the stored ontology in a shape the validator refuses on every later load. - Ontology initialisation and registration are serialised (asyncio.Lock per event loop); the store collapses a duplicated :TableMapping node, reads such a graph once per source meanwhile, keys MappedLink by (type, to, by) and MappedLinkColumn by link target, and skips a malformed stored mapping with a warning instead of raising from load(). - A second proposed mapping no longer retires a prior proposal with the same signature; it falls through to the signature-collision check. - CI integration job runs the whole suite under RUN_INTEGRATION=1. Should-fix - A label a Link points at has one key column: refused at registration (whichever declaration completes the shape) and at proposal time, where the model is told why. Two tables keying one label differently with no link to it remain the supported multi-source shape. - find_near_misses is blocked on comparable tokens rather than pairwise; same result set, 5k entities under one label in under a second. - A merge carries the longer description. - Link order is sorted before handle derivation, so an unchanged export no longer re-syncs by the order the graph returned its links. Tests: 20 regression tests (18 verified failing on the pre-fix source); test_rename_entity_relabels_data_nodes reworked to assert data, since FalkorDB keeps a label in db.labels() after its last node is relabelled. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
RetrievalStrategy.set_ontology() currently has no no-op body, which is a Python syntax error that will break imports.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 58/60 changed files
- Comments generated: 1
- Review effort level: Lite
…ites The structured pipeline wrote content_hash with the Document, before the rows, references and edges. A Person write that raised, or a RELATES batch the store logged and skipped, left a Document whose hash certified the table complete; the retry of the identical file routed through update(), matched the hash and returned no_op=True against the missing data (Naseem77 on #328). The hash is now the last write, through the LexicalGraphWriter's shared _mark_content_hash, and is withheld when the lexical or data edge write came up short — the same gate IngestionPipeline applies. The shortfall is reported as StructuredIngestionResult.incomplete_writes (and in as_dict, so the re-sync cutover in _finish_update already honours it), and an ordinary retry repairs the table. _build_lexical_graph loses its content_hash parameter: nothing writes the hash up front any more. Tests: entity write raising, PART_OF and RELATES coming up short, and a short re-sync — each followed by a retry of the identical file that must not be a no-op. All four fail on the previous source. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
@Naseem77 — two commits since your last pass, plus a merge of
|
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
It introduces a large new ingestion mode and modifies core storage/resolution/retrieval behaviors across many modules, warranting careful human validation of edge cases and backward compatibility.
Review details
- Files reviewed: 58/60 changed files
- Comments generated: 0 new
- Review effort level: Lite
The integration job now runs the whole suite, which surfaced the same runner problem main fixed in dda742e for test_facade: the default GLiNER extractor cannot load its deberta tokenizer on the CI runner, the chunk counts as an incomplete write, content_hash is withheld and update() is no longer a no-op. The test is about the hash short-circuit, not NER — give it an empty step-1 extractor and keep the scripted step-2 LLM. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The PR introduces a large new ingestion mode and changes multiple core behaviors (ontology model, ingestion, resolution, retrieval, indexing), so it warrants a final human review despite no specific defects found in this pass.
Review details
- Files reviewed: 59/61 changed files
- Comments generated: 0 new
- Review effort level: Lite
… pattern nodes Two review threads on #328 that were real: - delete_all() reset _indices_ensured but not _id_indices_ensured, so after a graph drop the per-label id range indexes were never recreated in the same process and the next ingest paid the unindexed MERGE cost. - add_relation_pattern_node() copied type/description but not `structured` to the new pattern's Property node, and load() kept the first row it saw, so a declared relation property could read back unstructured depending on row order. Copy the flag and OR it across nodes on load. Also the two code-quality nits: an unused first `renamed` assignment in a test, and the implicit string concatenation inside a list literal in the proposal prompt. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
It introduces a broad new ingestion mode and substantial cross-cutting changes across ingestion, storage, resolution, retrieval, docs, and CI that warrant careful human validation.
Review details
- Files reviewed: 60/62 changed files
- Comments generated: 0 new
- Review effort level: Lite
Reconciles the finalize-time deduplicator fixes from main (#323) with the structured-ingestion merge rules: - One `_survivor_rank`: declared key, real over placeholder, long form over acronym, degree, description, name length, id. - One merge path, `_absorb`: remap (MATCH-first, rel_type-keyed), read both nodes, join descriptions with " | ", record aliases, union source_chunk_ids, carry every property the survivor lacks, delete the duplicate in the same statement, detect a vanished survivor. - Grouping stays on `canonical_key`, now folding accents, abbreviation dots and a leading English article too; acronym groups are unioned on top of it. - `properties_to_carry` joins descriptions instead of keeping the longer, so the dedup and the table-row reconcile in GraphStore agree. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
One critical and six moderate findings remain unresolved, including transactional identity reconciliation, custom-loader dispatch, and table-owned edge cleanup.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (10)
CHANGELOG.md:17
- The release note says table identity comes from the declared key, but this feature's central shared-node behavior derives a named row's node id from
nameand uses the key asentity_keyfor links and re-sync. This misleading summary conflicts with the immediately following explanation and the public API contract.
docs/api-reference.mdx:796 - This promises that passing any
RecordLoaderStrategyenables another format, but the current dispatch only routes four filename suffixes to structured ingestion; a.parquet/database-backed source with this argument is sent through prose (andupdate()rejects it). The implementation and this public API reference need to agree on explicit custom-loader dispatch.
docs/api-reference.mdx:743 - This says
keyderives the node id, but the implementation derives a named row's id fromname(viacompute_entity_id) so it can share the node with prose;keyis stored asentity_keyand used to find rows, with id derivation only when no usable name exists. As written, consumers may expect a corrected display name to keep the old id and misunderstand the prose-join behavior.
docs/api-reference.mdx:701 - This API overview says identity comes from the key, but named table rows use the name-derived node id so they share ids with prose; the key is the
entity_keyused for links and re-sync. Please align this summary with the detailedTableMapping.keyandnamedescriptions below.
docs/structured-ingestion.mdx:553 - A row's node id does not generally come from its declared key: when
nameis present it is name-derived, and the key is the lookup/ownership value used during re-sync. This sentence documents the wrong update behavior and obscures why a rename can move the node while preserving its keyed row identity.
docs/structured-ingestion.mdx:35 - This overview repeats the incorrect identity rule: a named row's node id is derived from
nameto match prose, while the declared key is retained asentity_keyfor links and re-sync (and is only the id fallback without a name). The current wording conflicts with the guide's own mapping section and can mislead users about why prose and rows share nodes.
docs/structured-ingestion.mdx:884 - This note documents that a custom loader is still restricted to the four delimited suffixes, which contradicts the feature contract that other formats (for example Parquet) work through
RecordLoaderStrategy. It also conflicts with the custom-source example/API text above; once dispatch is corrected, this limitation must be removed or rewritten.
graphrag_sdk/src/graphrag_sdk/core/tables.py:301 datetimeis a subclass ofdate, so a Python datetime supplied by a custom record loader takes this branch and is stored as2026-01-01T12:00:00rather than the date-only value produced by the string branch below. A column declared asDATEtherefore has different semantics depending on loader representation and can sort/compare incorrectly. Strip the time component before serializing datetime inputs.
graphrag_sdk/src/graphrag_sdk/ingestion/resolution_strategies/base.py:329- When two relationships become the same
(start, type, rel_type, end)after entity resolution, this branch keeps only the first relationship object. Itssource_chunk_idsare therefore the only provenance retained; deleting that chunk later can remove the edge even though the discarded relationship's chunk still supports it. Merge the duplicate relationship's provenance (and preserve the applicable properties) instead of silently dropping it.
graphrag_sdk/src/graphrag_sdk/ingestion/structured_pipeline.py:459 - This pipeline docstring says identity comes from the declared key, contradicting
reconcile_keyed_identity()below and the shared-node contract: named rows use the name-derived id, while the key is stored and used for row/link resolution. Please correct the description so maintainers do not preserve the wrong identity invariant.
- Files reviewed: 60/63 changed files
- Comments generated: 2
- Review effort level: Lite
Comment on lines
+600
to
+603
| for (label, signed_key), rows in keyed.items(): | ||
| result.identity_moved.update( | ||
| await self.graph_store.reconcile_keyed_identity(label, signed_key, rows) | ||
| ) |
Comment on lines
+875
to
+879
| for prop, column in edge_columns[(edge.type, edge.source, edge.target)].items(): | ||
| value = column.cast(record.get(column.name)) | ||
| if value is not None: | ||
| # Signed, exactly as node properties are. Two tables | ||
| # declaring one edge property used to overwrite each |
galshubeli
added a commit
that referenced
this pull request
Sep 12, 2026
main moved after #323 and #328 landed. Only CHANGELOG.md conflicted: #328 added a large ### Fixed block to [Unreleased] at the same position as this branch's ### Removed entry. Resolved keep-both, leaving [Unreleased] as Added / Changed / Fixed / Removed. Verified on the merged tree: 1807 passed, 227 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes FalkorDB/research#82 · design in #290 · supersedes #74
What
GraphRAG-SDK reads prose (text, PDF, Markdown). This adds tables — CSV/TSV/PSV out of the box, any other format through a
RecordLoaderStrategy— as a first-class source, without an LLM call per row, and so that a row and the passage that names the same thing land on one node.Docs:
docs/structured-ingestion.mdx(guide),docs/api-reference.mdx§ Structured Ingestion. Example:examples/11_structured_ingestion.py. CHANGELOG Unreleased has the full list.API surface
graphrag_sdk.core.tables:TableMapping,Column,Link,MappingError. Mappings live onOntology.tablesand are looked up by the source's basename —ingest()/update()take nomapping=.ingest(path)routes.csv/.tsv/.psv/.tabto the record path (loader=TextLoader()opts out); returnsStructuredIngestionResult(records,rows_skipped,entities_created/updated/deleted,references_ambiguous, …).record_loader=accepts anyRecordLoaderStrategy(Parquet, a database cursor, in-memory rows);strict_mapping=refuses unmapped columns.ingest(path, document_id="employees.csv")re-syncs a table from a renamed export.update(path)re-syncs a table to its new export (rows added, changed, removed);delete_document(table)and newdrop_table()take the table's columns back off every node while a document's own facts stay.TableMappingis not refused: when an LLM is configured the model proposes one from the header, a sample and the existing labels (deterministicnatural_mappingotherwise). The proposal is storedderived=True, reported byfinalize().proposed_mappingsuntil a declaration replaces it, and every key/type it picks is checked against the data.finalize()also reportsprobable_duplicatesit declined to merge and why, andproperty_conflictswith how many entities two tables actually disagree on.Design (method A, "shared node")
compute_entity_id(name, label)— the same function prose extraction uses — so the two meet at write time, no merge pass. The key derives the id only when the row has no name column or the names are ambiguous.employees__age), and the table's key is kept asentity_key;is_stubmarks placeholders raised by aLinkto a row not yet seen. Prose never writes signed props; a table never writes unsigned ones exceptname/description. That is what lets re-sync, redeclaration anddrop_tableremove exactly what the table said and nothing else.Chunk{kind:"record"}on the table'sDocument, withMENTIONED_IN, soupdate()/delete_document()orphan cleanup, chunk retrieval and Cypher work unchanged. Edges areRELATES{rel_type}— no new edge convention for retrieval to learn.Method B (a separate
Recordnode per row linked to the entity) was implemented and compared on the same corpus; A won on connectivity and retrieval hops. The comparison lives on branchfeat/structured-ingestion-separate-nodesfor reference.Behaviour changes to know about
finalize()runsLLMVerifiedResolutionby default (resolve=Falserestores the old exact-only pass)..csvis no longer read as one prose chunk.ingest()of a table returnsStructuredIngestionResult, not anIngestionResultsubclass.id(andentity_key) are created per label before writes — prose ingest gets the O(n) write path too.Fixed along the way (prose path too)
Merge dropped the duplicate's properties; edge identity ignored
rel_type(one edge per pair); unboundMERGEin the remap could fork the survivor;source_chunk_idswas last-writer; digit-only name differences (GPT-3/GPT-4) were merged; reopening a graph logged an ERROR per label for already-existing indexes; text-to-Cypher rows lost their column names; retrieval held a stale ontology.Tests
PYTHONPATH=src pytest tests/— 1498 passed, 210 skipped.RUN_INTEGRATION=1): the structured, graph-store, resolver, identity, loader and finalize-report suites — 580 passed.test_ontology_evolve_integration::test_rename_entity_relabels_data_nodesfails onmainas well.ruff check src && ruff format --check src;node scripts/check_mdx.mjs;mypy src/graphrag_sdk— 137 errors vs 141 onmain.Review guide
Start with
core/tables.py(the declaration), theningestion/structured_pipeline.py(rows → GraphData), thenapi/main.py—_ingest_structured,_run_post_cutover_cleanup,_retract_table,_retract_what_the_declaration_dropped. Storage:graph_store.pyresolve_by_entity_key/reconcile_keyed_identity/release_entity_keys/rows_outside_document;deduplicator.py_survivor_rankand the keyed guards in_deduplicate_fuzzy.Summary by CodeRabbit
New Features
Bug Fixes
Documentation