Repository navigation
fix(affixes): resolve destructive affix removal and protect root words (fixes #26) - #39
Open
AbdelrahmanBayoumi wants to merge 1 commit into
Open
AbdelrahmanBayoumi wants to merge 1 commit into
AbdelrahmanBayoumi wants to merge 1 commit into
Conversation
fixes #26) - enforce minStemLength (default: 3) to protect root consonants from being mangled - add support for compound prefixes (بال, كال, ولل, فال, فلل) and conjunctions (و) - add comprehensive suffix handling without stripping ambiguous dual endings - export AffixOptions interface for fine-grained prefix/suffix/length control - check both raw and stripped string in tashfeerBannedWords to fix false negatives (e.g. فلسطين, بالقدس) - update tests and docs to verify non-destructive linguistic behavior Closes #26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of Changes
Fixes #26 - Resolves destructive affix removal in
removeArabicAffixesand prevents word corruption.1. Core Fixes in
removeArabicAffixes:minStemLength >= 3): Ensures root letters are never stripped (e.g. "بيت", "تفاح", "أمل", "نجم", "يوم", "ولد" now remain intact instead of being mutilated into "يت", "فاح", "مل", etc.).بالـ,كالـ,وللـ,فالـ,فللـ(e.g.بالكتاب->كتاب,وللقدس->قدس).الـ,للـ(e.g.الكتاب->كتاب,للمعلمين->معلم).و(e.g.وكتاب->كتاب).ون,ين,ات,هم,نا,ة,ه,ي,ية) while protecting singular words with root "ان" (such as "إنسان" and "قرآن").AffixOptionsinterface allowing callers to configureminStemLength(default 3),prefixes(default true), andsuffixes(default true).2. Fixes in
tashfeerBannedWords:Verification
npm test: 84/84 tests passed across 5 test suites (100% statement, branch, function, and line coverage onsrc/scripts/affixes.ts).npm run lint: 0 errors, 0 warnings.npm run build:cjs,npm run build:esm,npm run bundle: All builds and bundles generated cleanly.Closes #26