Repository navigation
Conversation
cragwolfe
left a comment
There was a problem hiding this comment.
Reviewed the PDF partitioning changes at e79d99a. I did not establish an actionable Critical, High, or Medium defect in the reviewed scope.
One qualification to the bit-identical claim: with float32 coordinates under NumPy 2, the new intersection accumulation can use a different precision from the previous scalar loop. The available source does not establish whether a supported production path passes such coordinates or whether a resulting threshold difference has a user-visible effect. A dtype and threshold-boundary equivalence test would help establish that contract.
This is a scoped source review; it does not cover metrics or staging behavior, uv.lock, or CHANGELOG.md, and it is not a statement about CI or merge readiness.
(authored by codex)
…redundant slicing
Seed the cumulative sums with the input dtype so float32 coordinates accumulate in float32 like the scalar loop did, and pin bit-for-bit equality against that loop for float32 and float64.
e79d99a to
ac92ac9
Compare
Summary
aggregate_embedded_text_by_block()runs once per layout region that needs text, frommerge_out_layout_with_ocr_layout()(OCR) andmerge_inferred_with_extracted_layout()(hi_res). Per call it:if sum(mask):— builtinsumiterates a numpy bool array element by element; nowmask.any();source_regionswith the same mask three times; now once;_aggregated_iouwith a Python loop over the selected boxes; now vectorized with numpy.Numerical equivalence
_aggregated_iouaccumulates withnp.cumsum, which adds left to right like the old loop (np.sum's pairwise summation could differ in the last bit), and seeds the sums with the coordinates' own dtype, so float32 input still accumulates in float32 as the loop did. Results are bit-identical, with the same return dtype, for both float32 and float64.test_aggregated_iou_matches_scalar_loop_bit_for_bitpins this: it compares against the old scalar loop over 500 random cases per dtype, including zero-size and inverted boxes, and requires exact equality. (In production,TextRegionsconverts coordinates to float64, so the float32 case isn't currently reachable; the test keeps the claim unconditional.)Performance
Walltime,
pytest-codspeedrun locally (Apple Silicon),merge_out_layout_with_ocr_layoutwith 60 regions and 1,500 OCR words on one page:Validation
test_unstructured/partition/pdf_image/test_pdfminer_processing.py,test_ocr.py,test_pdf.pyandtest_merge_elements.pypass.