Add inline text ingestion support - #2384
Conversation
Greptile SummaryThis PR adds a
|
| Filename | Overview |
|---|---|
| nemo_retriever/src/nemo_retriever/common/inline_text.py | New module introducing inline text primitives: normalize/validate, deterministic source IDs, blank-corpus guard, and inline URI detection. |
| nemo_retriever/src/nemo_retriever/common/modality/txt/split.py | Refactored to extract text_to_chunks_df from txt_file_to_chunks_df and txt_bytes_to_chunks_df, adding empty_text_chunks_df() and inline-source-path preservation in txt_bytes_to_chunks_df. |
| nemo_retriever/src/nemo_retriever/ingestor/graph_ingestor.py | Adds .texts(), removes .extract_txt() (breaking API change flagged in previous review), changes _extraction_mode default from pdf to None, and adds inline-text routing for inprocess/batch/branch paths. |
| nemo_retriever/src/nemo_retriever/ingestor/branch_extraction.py | Adds inline_rows field to ExtractionBranchExecutor; _partition_branch_inputs and _inline_rows_by_path helpers separate file, buffer, and inline inputs per branch for both inprocess and Ray batch paths. |
| nemo_retriever/src/nemo_retriever/service/service_ingestor.py | Adds .texts() to ServiceIngestor; _collect_inputs packs inline texts as InMemoryUpload items without writing temp files; blank-corpus early exit and _has_mixed_inline_sources auto-routing for mixed payloads. |
| nemo_retriever/src/nemo_retriever/service/client.py | Adds InMemoryUpload NamedTuple and UploadInput union type; extracts _upload_source helper; broadens ingest_documents and aingest_documents_stream to accept list[UploadInput] instead of list[Path]. |
| nemo_retriever/src/nemo_retriever/operators/extract/txt/ray_data.py | TxtSplitCPUActor now dispatches on text column (inline path) vs bytes column (file/buffer path); exception handler now logs with exc_info=True via logger.warning before continuing. |
| nemo_retriever/src/nemo_retriever/service/utils/file_type.py | Adds early-return in infer_extraction_mode_from_filename for inline:// URIs so the server correctly routes in-memory text uploads to the text extractor. |
Reviews (19): Last reviewed commit: "Remove redundant shutdown lease assertio..." | Re-trigger Greptile
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
7888b78 to
57bc3e9
Compare
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
|
|
||
| def _execute_batch(self) -> Any: | ||
| _ray, cluster_resources = self.ensure_batch_runtime() | ||
| ray, cluster_resources = self.ensure_batch_runtime() |
There was a problem hiding this comment.
why did you make this public? I think it should stay private no?
| upload_failures: list[tuple[str, str]] = [] | ||
|
|
||
| async def _upload_one_file(fpath: Path) -> None: | ||
| async def _upload_one_file(source: UploadInput) -> None: |
There was a problem hiding this comment.
is there anyway to refactor this internal logic and that from the aingest_document_stream internal into a single helper, that maybe calls a another function that handles the differences?
| if result_schema == "compact" | ||
| else ["text", "content", "path", "page_number", "metadata"] | ||
| ) | ||
| return pd.DataFrame(columns=columns).astype({"page_number": "int64"}) |
There was a problem hiding this comment.
what is the difference between this and regular empty_text_chunks_df? Seems confusing to have both. Is this only because of the columns available, and the dtype conversion?
| self._last_job_id = None | ||
| result = ServiceIngestResult() | ||
| if return_results: | ||
| result.dataframe = _empty_inline_text_dataframe(result_schema) |
There was a problem hiding this comment.
Like here, why can't we use just empty dataframe.
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
…xt-ingestion # Conflicts: # nemo_retriever/src/nemo_retriever/common/modality/txt/split.py
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
Summary
.texts()for ingesting raw Python strings without temporary filesinline://...text sources.extract(split_config=...)interface; no separate.extract_txt()or Graph.split()APICloses #2345.
Validation