Repository navigation
fix: pass the OCR_AGENT env var to partition() - #596
Open
theluckystrike wants to merge 1 commit into
Open
theluckystrike wants to merge 1 commit into
theluckystrike wants to merge 1 commit into
Conversation
partition() defaults ocr_agent to Tesseract and does not read OCR_AGENT, so the API always ran Tesseract even when OCR_AGENT named PaddleOCR. Resolve the agent the way the library does (legacy names included) and pass it to partition(). Parallel-mode workers resolve it from their own environment, so it is not sent as a form field. Fixes Unstructured-IO#532
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #532.
pipeline_api builds partition_kwargs without ocr_agent. In unstructured, partition_pdf and partition_image take ocr_agent with a default of OCR_AGENT_TESSERACT and do not read the OCR_AGENT variable themselves, so the API always ran Tesseract. An image that installs PaddleOCR and sets OCR_AGENT, as in the issue, still spawns tesseract processes.
pipeline_api now resolves the agent with OCRAgent._get_ocr_agent_cls_qname() from unstructured and adds it to partition_kwargs (and to the debug log of partition inputs). That is the same lookup the library uses for OCR_AGENT, so the legacy names tesseract and paddle are still mapped to the full class names instead of failing the module whitelist check. With OCR_AGENT unset nothing changes, the value is the Tesseract default. In parallel mode the field is dropped from the form sent to workers, because it is not a form parameter and each worker resolves OCR_AGENT from its own environment. Table OCR keeps the library default for table_ocr_agent, since OCR_AGENT does not cover it in the library either. The CHANGELOG has a 0.1.17 entry and version is bumped to match.
Tests. test_general/api/test_ocr_agent.py is new. It posts an image with strategy hi_res and a mocked partition, and checks the ocr_agent passed for OCR_AGENT unset, set to the PaddleOCR class name, and set to the legacy name paddle. A fourth test checks that ocr_agent is not in the form sent to a parallel worker. test_parallel_mode_passes_params now sets OCR_AGENT and expects it in the partition call. The five tests fail on main and pass with this change.
Commands run locally on Python 3.13 were pytest test_general, ruff format and ruff check on prepline_general, test_general and scripts/smoketest.py, and mypy prepline_general with the Makefile flags. pytest gave 137 passed, 53 failed, 1 skipped, 24 xfailed and 4 xpassed, and main with the new test file present gave 133 passed and 57 failed. The two runs fail the same 53 tests apart from the 4 new ones. All 53 are environment failures, because I installed unstructured without the all-docs extras to avoid pulling torch, so pdfminer and the pdf, docx and pptx partitioners are missing. For the same reason unstructured_inference was replaced by a local one-class import shim during the run. Ruff is clean. mypy reports one error at the response.body.strip() line in the CSV branch, and main reports the same error, so it is not from this change. I did not build the Docker image, did not run PaddleOCR (no Paddle wheels or model weights were downloaded), and did not run scripts/version-sync.sh because it needs GNU sed. The CHANGELOG heading and version both say 0.1.17.