Repository navigation
Conversation
cragwolfe
left a comment
There was a problem hiding this comment.
The source review found no actionable critical, high, or medium defect in the authored change. With valid configuration and successful NLP calls, the reordered checks preserve the classification result.
The short circuit also changes observable failure and diagnostic behavior: a malformed non-alpha threshold can now raise before the capitalization check, while a malformed capitalization threshold or an NLP initialization failure can be bypassed for text rejected by the non-alpha check. A maintainer should decide whether to accept those compatibility effects. A focused test that asserts low-alpha text skips the NLP path would protect the optimization.
The current head differs from main in version metadata because the branch is behind; this is not part of the author diff. The exact-head check-version and partition runtime comparison checks have failed, so integration readiness remains separate from this source review.
(authored by codex)
053cf73 to
3f2c850
Compare
Summary
is_possible_narrative_text()calledexceeds_cap_ratio()beforeunder_non_alpha_ratio().exceeds_cap_ratio()callssentence_count()andword_tokenize(), which run the spaCy pipeline;under_non_alpha_ratio()is a single character scan. Both checks only ever returnFalse, so running the cheap one first gives the same result and skips spaCy for text like----------,$ 1,234.56 $ 7,890.12or*** === +++.is_possible_title()already orders its checks this way.A new test,
test_is_possible_narrative_text_rejects_low_alpha_text_without_nlp, makesword_tokenize,pos_tagandsent_tokenizeraise and asserts low-alpha text is still rejected, so the optimization can't silently regress. It fails on the previous check order.Behavior notes for maintainers
With valid configuration the classification result is unchanged. Three observable side effects change, all only for text that fails the non-alpha check:
sentence_count().UNSTRUCTURED_NARRATIVE_TEXT_NON_ALPHA_THRESHOLDis now read first, so a malformed value raisesValueErroreven for text the cap-ratio check would have rejected first (e.g."RISK FACTORS"). Conversely, a malformedUNSTRUCTURED_NARRATIVE_TEXT_CAP_THRESHOLDis no longer read for low-alpha text. Onmaineither malformed value already raises for most text.Performance
Walltime,
pytest-codspeedrun locally (Apple Silicon), 120 strings:is_possible_narrative_textThis is a worst case: the benchmark clears the spaCy caches every round, so documents with repeated strings gain less. The gain also depends on how much of a document is symbol- or number-heavy; 5 of the 12 sample strings were. Prose-only input gains little.
Validation
test_unstructured/partition/test_text_type.pypasses (119 tests).