Conversation
Signed-off-by: ealbasiri <ealbasiri@gradcenter.cuny.edu>
for more information, see https://pre-commit.ci
Signed-off-by: anand-nv <105917641+anand-nv@users.noreply.github.com>
for more information, see https://pre-commit.ci
Signed-off-by: anand-nv <105917641+anand-nv@users.noreply.github.com>
Signed-off-by: anand-nv <105917641+anand-nv@users.noreply.github.com>
for more information, see https://pre-commit.ci
Signed-off-by: anand-nv <105917641+anand-nv@users.noreply.github.com>
…d fraction homograph Cardinal (TN/ITN): - Expand to millions with place-value 3-digit period decomposition - Orthography: مئة (drop silent alif), مئتين, separated compound hundreds (ثلاث مئة), مئتي construct-state before counted nouns - Use genitive tens (عشرين) and اثنين by default - Fix counted-noun (تمييز) agreement: trailing 3-10 -> plural (110000 -> مئة وعشرة آلاف), otherwise singular; applied to آلاف and ملايين Fraction (TN/ITN): - Resolve homograph collision between dual fractions and cardinals by using construct-state dual denominators without final nun (سبعي=2/7 vs سبعين=70); both directions now round-trip losslessly - Verbalizer accepts اثنين numerator Tests: - Fix pre-existing NameError (RUN_AUDIO_BASED_TESTS) in ar/test_measure.py - Regenerate cardinal/fraction/decimal/money/measure test data (TN+ITN) to the new canonical spelling; decimal/money/measure grammars unchanged - Full ar suite: 306 passed
TN (text_normalization/ar):
- Add time tagger + verbalizer for HH:MM, HH:MM:SS, and H <suffix> inputs
- Formal/regular style: hour as feminine ordinal (24h; midnight 0 -> الثانية عشرة),
minutes/seconds as <number> دقيقة/دقائق
- Counted-noun agreement:
* teens via 13_19 data (11 -> إحدى عشرة, 15 -> خمس عشرة)
* 3-10 plural (خمس دقائق), 1 -> دقيقة واحدة, 2 -> دقيقتان
* 21-59 ones digit feminized by polarity (خمس/ثمان/إحدى/اثنتان ...)
* minutes/seconds number is nominative (عشرون/ثلاثون/أربعون/خمسون)
- Constrain suffix verbalization to suffix.tsv values (required for a clean inverse)
- Fix suffix.tsv tab separators
- Wire TimeFst into tokenize_and_classify.py and verbalize.py
- Add data: time/{suffix,time_zone}.tsv, ordinal/ordinals.tsv, number/{3_10mas,13_19}.tsv
ITN (inverse_text_normalization/ar):
- Add time tagger (inverts the TN time verbalizer) + verbalizer (-> HH:MM[:SS])
- Wire into tokenize_and_classify.py and verbalize.py
Tests:
- test_time.py: 30 TN (test_norm) + 19 ITN (test_denorm) hand-written cases
- Full ar suite: 355 passed
…. tests 399 passed.
…months) Signed-off-by: Enas Albasiri <ealbasiri@gradcenter.cuny.edu>
mgrafu
reviewed
Jul 9, 2026
| @@ -0,0 +1,113 @@ | |||
| # Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
Collaborator
There was a problem hiding this comment.
Copyright (c) 2026 for any new file
mgrafu
reviewed
Jul 9, 2026
| @@ -1 +1 @@ | |||
| # Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
Collaborator
There was a problem hiding this comment.
unchanged for any existing files
mgrafu
reviewed
Jul 9, 2026
| @@ -1,4 +1,4 @@ | |||
| # Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. | |||
| # Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. | |||
Collaborator
There was a problem hiding this comment.
let's add these 3 new test files to sparrowhawk test file too
mgrafu
reviewed
Jul 9, 2026
| ) | ||
| self.final_graph_wo_negative = optional_graph_negative + self.final_graph_wo_sign | ||
|
|
||
| final_graph = self.add_tokens(self.final_graph_wo_negative) |
Collaborator
There was a problem hiding this comment.
do we need to remove coverage for negatives?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Adds the Arabic (
ar) grammar contributions: expanded cardinals, time, and date — each in both text normalization (TN) and inverse text normalization (ITN).Scope (3 semiotic areas)
Cardinal (TN + ITN)
Time (TN + ITN)
Date (TN + ITN)
-,/,\,.(dotted requires full d.m.y so decimals stay decimals) and spelled-out months.dd/mm[/yyyy](or month name + year), Hijri suffix preserved.Testing
pytest --cpufullarsuite: 399 passed, 0 failed.Before your PR is "Ready for review"
Pre checks:
git commit -s)pytest --cpu— 399 passed. (Sparrowhawk: not run.)pytesttest cases (cardinal/time/date + regenerated dependent data). (Sparrowhawk cases: not added.)__init__.pyfor new data folders (months, ordinal, time).en/graph_utils.pycopy header — N/A.aralready documented).tools/text_processing_deployment/pynini_export.py—aralready registered.PR Type: