Repository navigation
Conversation
… signs as letters, split sentences on danda)
chrikrah
left a comment
There was a problem hiding this comment.
@shlokbam blocking: unicodedata.category(char)[0] == "M" also counts U+FE0F (variation selector, Mn) and U+20E3 (keycap, Me), so an emoji-only line now passes under_non_alpha_ratio. is_possible_title never reaches _is_word, so these lines come out as Title and open a new chunk in chunk_by_title.
# trees from `git archive`: head 6544f77 and base 6544f77~1, CPython 3.12.14, clone .venv
# probe.py prints unstructured.__file__, then for each line: is_possible_title(s), element types of partition_text(text=s)
$ PYTHONPATH=/tmp/p34e/base .venv/bin/python probe.py
/tmp/p34e/base/unstructured/__init__.py
'⭐️⭐️⭐️⭐️⭐️' False ['Text']
'1️⃣ 2️⃣ 3️⃣' False ['Text']
'✅️ ✅️ ✅️' False ['Text']
'⚠️' False ['Text']
$ PYTHONPATH=/tmp/p34e/head .venv/bin/python probe.py
/tmp/p34e/head/unstructured/__init__.py
'⭐️⭐️⭐️⭐️⭐️' True ['Title']
'1️⃣ 2️⃣ 3️⃣' True ['Title']
'✅️ ✅️ ✅️' True ['Title']
'⚠️' True ['Title']
# head 6544f77, your tests pass and fail when the source hunks are reverted
$ .venv/bin/python -m pytest test_unstructured/partition/test_text_type.py test_unstructured/nlp/test_tokenize.py -q -p no:cacheprovider
151 passed in 25.04s
# same tree, unstructured/ reverted with `git apply -R` of the PR's source hunks
$ .venv/bin/python -m pytest test_unstructured/partition/test_text_type.py test_unstructured/nlp/test_tokenize.py -q -p no:cacheprovider
10 failed, 141 passed in 21.70s
# not run: ruff, test_unstructured/ outside these files
A fix that keeps the Indic cases: count a mark only when the previous non-space character is a letter or an attached mark. Matras, viramas and decomposed accents still count. A test that partition_text(text="⭐️⭐️⭐️⭐️⭐️") stays Text would pin it. Can you add that guard in this PR?
… letter Counting every Unicode mark (M*) as a letter also counted the variation selector U+FE0F and the keycap mark U+20E3, so emoji-only lines like "⭐️⭐️⭐️⭐️⭐️" or "1️⃣ 2️⃣ 3️⃣" passed under_non_alpha_ratio and were classified as Title. Add one helper, _letter_flags, that counts a mark only when the previous non-space character is a letter or a counted mark, and use it in both _is_word and under_non_alpha_ratio. Indic vowel signs and viramas still count; emoji marks do not.
|
@chrikrah thanks for catching this, and for the probe. Added the guard in ffae7b8.
The existing Hindi and Bengali cases still pass. |
cragwolfe
left a comment
There was a problem hiding this comment.
Medium — Keep the non-alpha check's memory use bounded
_letter_flags() builds a list entry for every non-space character, and under_non_alpha_ratio() builds the whole list before counting it. The previous implementation used two counters. Large paragraphs or extracted text blocks can reach this check without an element-length cap, so this adds a potentially large allocation and can exhaust a partition worker. The spaCy input limit does not bound the original string passed to the ratio check. Could the ratio use streaming counts and the word check use a small amount of state while preserving the new combining-mark rules? A focused long-input regression would help protect this behavior.
Compatibility decision: This also changes the public sentence tokenizer for danda-separated text, including its use by translation, and applies the combining-mark classification rule across Unicode scripts. The intended classification changes can alter element types and by-title chunk boundaries. Please confirm that this broader behavior is acceptable as part of the fix.
The earlier emoji regression is addressed by the new mark-handling rule and checked-in cases. This is a static review; exact-head CI requires approval and is not a successful test result.
(authored by codex)
_letter_flags built a list with an entry for every non-space character, and under_non_alpha_ratio built the whole list before counting it. Large extracted text blocks can reach this check, so the allocation grew with the input. Replace it with _iter_letter_flags, a generator that keeps only whether the previous non-space character counted. under_non_alpha_ratio uses two running counters again and _is_word stops at the first character that does not count. Results are unchanged.
|
@cragwolfe thanks for the review. Memory: fixed in 72cbaf3. Version / changelog: CONTRIBUTING says ordinary PRs don't edit the frozen Scope:
I think that change is a fix too, but if you'd rather keep this PR narrow, I can limit the mark rule to the Indic blocks. |
Problem
Text in Hindi, Bengali and other Indic scripts is misclassified in both directions:
Paragraphs become
Title.partition_texton six two-sentence Hindi paragraphs returns 3Titles, and on three Bengali paragraphs returns 2Titles.chunk_by_titlethen starts a new chunk at each false title:Title, 4 chunksTitle, 1 chunkTitle, 3 chunksTitle, 1 chunkTitle, 1 chunkTitle, 1 chunk (unchanged)(
chunk_by_title(elements, max_characters=2000, combine_text_under_n_chars=0))Real headings can't be detected as
Title.is_possible_title("हिंदी")("Hindi") returnsFalse, becauseunder_non_alpha_ratio("हिंदी")says the word is mostly non-letters.Root causes
str.isalpha(). In Devanagari and Bengali, vowel signs (matras), viramas and anusvaras are Unicode combining marks (categoryM*), so"हिंदी".isalpha()isFalse: only 2 of its 5 code points are letters.under_non_alpha_ratiocounts the marks as non-letters, so short words like "हिंदी" fall under the 0.5 threshold.exceeds_cap_ratiokeeps only tokens wheretk.isalpha()is true. In a Hindi sentence where every word carries a vowel sign, nothing is left, and the "no tokens" branch returnsTrue("too many capitals"). The paragraph then fails the narrative check.।,॥) isn't a reliable sentence end. The English spaCy model sometimes splits at।and sometimes doesn't. When it doesn't, a two-sentence paragraph counts as one sentence. That sends it into the single-sentence branch ofexceeds_cap_ratioabove, and into the English verb check.Fix
text_type.py: a character counts as a letter if it is alphabetic or a combining mark.under_non_alpha_ratio: counts marks as letters, written inline for speed (char.isalpha() or unicodedata.category(char)[0] == "M").exceeds_cap_ratio: a token counts as a word if it has at least one letter and every character is a letter or a mark (_is_word), instead oftk.isalpha().nlp/tokenize.py: after spaCy's sentence segmentation, each sentence is split again on(?<=[।॥])\s+. This is inside the existinglru_cached_tokenize_for_cache, so caching is unchanged andsent_tokenize/sentence_countpick it up.Unchanged: thresholds, the English verb check,
is_possible_narrative_text/is_possible_titlelogic, and behavior for text without combining marks or dandas. All existing tests in the touched files pass.Tests
New tests in
test_text_type.py,nlp/test_tokenize.pyandpartition/test_text.py:under_non_alpha_ratio("हिंदी")isFalse;is_possible_title("हिंदी")isTrueexceeds_cap_ratioisFalsefor "हिंदी सबसे अधिक बोली जाने वाली भाषा है।" (every word has a vowel sign)sentence_countof "हिंदी सबसे अधिक बोली जाने वाली भाषा है। गंगा पवित्र नदी है।" is 2 (was 1);sent_tokenizesplits on।and॥is_possible_narrative_textisTruefor each Hindi/Bengali paragraph;partition_textreturns onlyNarrativeTextfor both documentsunder_non_alpha_ratioon full paragraphs,is_possible_title("परिचय"), and the existing English title/narrative cases12 of the new tests fail on
mainand pass with this change; the rest guard behavior that already worked.test_text_type.py,nlp/test_tokenize.pyandpartition/test_text.py: 246 passed.ruff checkandruff format --checkpass on the changed files.To reproduce:
Performance
10,000 calls each, best of 5, Apple M2, Python 3.13:
mainunder_non_alpha_ratioexceeds_cap_ratiounder_non_alpha_ratioexceeds_cap_ratioEnglish is within noise. Hindi
under_non_alpha_ratiodoes about 1 µs more work because it now looks up the category of each mark. Theexceeds_cap_rationumbers are the cached path; classifying a new element is dominated by the spaCy run (milliseconds), which this PR doesn't change.Known limitation (possible follow-up)
A single-sentence Hindi/Bengali paragraph can still become a
Title, e.g. "दिल्ली में सरकार की बैठकें होती हैं।", "भारत एक विशाल देश है।" or "রবীন্দ্রনাথ ঠাকুর নোবেল পুরস্কার পেয়েছিলেন।". Both character checks now pass, but with fewer than 2 sentencesis_possible_narrative_textruns the English-only verb check (contains_verb), whose result for Hindi depends on spaCy happening to tag some word as an English verb.partition_textdetectshin(it's inelement.metadata.languages) but doesn't passlanguagestois_possible_narrative_text;is_possible_narrative_text(t, languages=["hin"])returnsTrue. I'm happy to send that as a separate PR.Overlap with #4679
#4679 reorders the
under_non_alpha_ratioandexceeds_cap_ratiocalls insideis_possible_narrative_text. This PR only changes the bodies of those two functions (plus new helpers aboveis_possible_narrative_text), so the two shouldn't conflict, and the reorder doesn't change any result here since both checks can only make the function returnFalseearly.