hi_itn_electronic#437
Conversation
Signed-off-by: Mayuri S <mayuris@nvidia.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
Signed-off-by: mayuris-00 <mayuris@nvidia.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
mayuris-00
force-pushed
the
hi-itn-electronic-clean
branch
from
3e004de to
01b2fc4
Compare
| sharda शारदा | ||
| universities यूनिवर्सिटीज़ | ||
| mcdonald मैक्डॉनल्ड | ||
| southmountaincc साउथ माउन्टेन सी सी |
There was a problem hiding this comment.
let's trim this list to only be the most common cases
Signed-off-by: Mayuri S <mayuris@nvidia.com>
Signed-off-by: Mayuri S <mayuris@nvidia.com>
mayuris-00
force-pushed
the
hi-itn-electronic-clean
branch
from
812333f to
99a6eef
Compare
| @@ -0,0 +1,167 @@ | |||
| Users यूज़र्स | |||
There was a problem hiding this comment.
let's add everything lowercased and accept several kinds of inputs if needed
https://github.com/NVIDIA/NeMo-text-processing/blob/main/nemo_text_processing/inverse_text_normalization/en/utils.py#L50
| graph_hundred_component_at_least_one_none_zero_digit | ||
| ) | ||
|
|
||
| # Transducer for eleven hundred -> 1100 or twenty one hundred eleven -> 2111 |
There was a problem hiding this comment.
let's not modify any scripts unnecessarily
| common_map_lower = (common_map @ to_lower).optimize() | ||
|
|
||
| latin_run = pynini.closure( | ||
| pynini.union(*[pynini.accep(chr(c)) for c in range(ord('A'), ord('Z') + 1)]) |
There was a problem hiding this comment.
is this equivalent to NEMO_ALPHA?
| ) | ||
| latin_run_lower = (latin_run @ to_lower).optimize() | ||
|
|
||
| drive_letter = letter_map_upper |
There was a problem hiding this comment.
this leave space for the entire alphabet instead of the previously restricted set. is that on purpose?
There was a problem hiding this comment.
Yes, on purpose Windows drive letters can be any letter A–Z (not just C/D), so letter_map_upper is intentional. It only matches within the full X:\… path structure, so it won't cause false positives on lone letters.
|
|
||
| url_slash = delete_space + sym["forwardslash"] | ||
|
|
||
| lit_slash_seg = pynini.cross(" / ", "/") |
There was a problem hiding this comment.
again, let's not hardcode the symbol, and let's use delete_space
| + pynini.closure( | ||
| pynutil.add_weight( | ||
| (delete_space + path_atom) | ||
| | win_hyphen |
There was a problem hiding this comment.
it looks like a lot of these follow the same pattern. can we create one rule for all symbols instead of individual rules?
| """ | ||
| Finite state transducer for classifying serial strings, whose segments are | ||
| joined by a hyphen (a literal "-" or the spoken word "हाइफ़न"). | ||
| e.g. कोविड-उन्नीस -> tokens { serial { name: "कोविड-19" } } |
There was a problem hiding this comment.
'serial' is not a semiotic class in the semiotic classes proto
| def __init__(self, cardinal: GraphFst): | ||
| super().__init__(name="serial", kind="classify") | ||
|
|
||
| not_quote = pynini.closure(pynini.difference(NEMO_SIGMA, pynini.accep('"')), 1) |
There was a problem hiding this comment.
| super().__init__(name="serial", kind="classify") | ||
|
|
||
| not_quote = pynini.closure(pynini.difference(NEMO_SIGMA, pynini.accep('"')), 1) | ||
| strip_cardinal_tags = pynutil.delete('cardinal { integer: "') + not_quote + pynutil.delete('" }') |
There was a problem hiding this comment.
let's define and use the necessary graphs before adding cardinal tags so we don't need to strip them
| number_words = pynini.arcmap(pynini.project(number, "input"), map_type="rmweight").optimize() | ||
|
|
||
| devanagari_letter = pynini.union( | ||
| *[chr(c) for c in range(0x0900, 0x0966)], |
There was a problem hiding this comment.
can we define this in graph_utils so we can use it as needed in other classes (like electronic)
| segment = word | number | ||
|
|
||
| word_hyphen = ( | ||
| delete_space + (pynutil.delete("हाइफ़न") | pynutil.delete("हाइफन")) + delete_space + pynutil.insert("-") |
There was a problem hiding this comment.
let's use the symbols as defined in electronics instead of hardcoding here
| letter_map_upper = (letter_map_lower @ TO_UPPER).optimize() | ||
|
|
||
| sym = load_symbols(get_abs_path("data/electronic/symbols.tsv")) | ||
| lit_open_paren = delete_space + pynutil.delete("(") + pynutil.insert("(") + delete_zero_or_one_space |
| | pynutil.add_weight(digit_words, 0.10) | ||
| | pynutil.add_weight(letter_map_upper, 0.84) | ||
| ) | ||
| alnum_run = alnum_token + delete_space + alnum_token + pynini.closure(delete_space + alnum_token, 0) |
There was a problem hiding this comment.
alnum_run = alnum_token + pynini.closure(delete_space + alnum_token, 1)
| @@ -0,0 +1,37 @@ | |||
| # Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. | |||
There was a problem hiding this comment.
The TN Hindi electronic files use the same Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. header, so I kept it consistent with those. Happy to switch these to 2024 to match the rest of the hi ITN module if you'd prefer just let me know which you'd like as the standard.
| class ElectronicFst(GraphFst): | ||
| """ | ||
| ITN verbalizer for electronic expressions. | ||
| All fields pass content through unchanged. |
There was a problem hiding this comment.
let's follow the same descriptions as for other languages / classes
|
|
||
| class SerialFst(GraphFst): | ||
| """ | ||
| Verbalizer for serial expressions. |
There was a problem hiding this comment.
let's follow the same descriptions as for other languages / classes, and make sure we're only adding tagger / verbalizer for classes that exist in the proto
| ) | ||
| graph = delete_space + pynini.closure(graph + delete_extra_space) + graph + delete_space | ||
| self.fst = graph | ||
| self.fst = (graph @ PostProcessor().fst).optimize() |
There was a problem hiding this comment.
let's follow other languages in their use of the postprocessor. it should be optional and passed as a flag
|
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
2 participants
What does this PR do ?
Add a one line overview of what this PR aims to accomplish.
Before your PR is "Ready for review"
Pre checks:
git commit -sto sign.pytestor (if your machine does not have GPU)pytest --cpufrom the root folder (given you marked your test cases accordingly@pytest.mark.run_only_on('CPU')).bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...pytestand Sparrowhawk here.__init__.pyfor every folder and subfolder, includingdatafolder which has .TSV files?Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.to all newly added Python files?Copyright 2015 and onwards Google, Inc.. See an example here.try import: ... except: ...) if not already done.PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.