Japanese TN fix and improvement#444
Conversation
Signed-off-by: Mai Anh <palasek182@gmail.com>
for more information, see https://pre-commit.ci
| @@ -0,0 +1,52 @@ | |||
| A エー | |||
There was a problem hiding this comment.
instead of mapping upper and lower case separately, let's add only one and capitalize the inputs in the graph itself
| @@ -0,0 +1 @@ | |||
| # Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. | |||
There was a problem hiding this comment.
let's add full copyright in all files
| @@ -0,0 +1,3 @@ | |||
| 万 万 | |||
There was a problem hiding this comment.
we shouldn't need to have the chars map to themselves, should be enough just to have the list of chars
| @@ -0,0 +1,2 @@ | |||
| Henry | |||
There was a problem hiding this comment.
let's instead only use common terms like century or chapter, but not cover just a couple of king names
| @@ -0,0 +1 @@ | |||
| World War II World War II | |||
| @@ -14,26 +14,20 @@ | |||
|
|
|||
|
|
|||
There was a problem hiding this comment.
i think right now normalize.py isn't wired to use this. can you please include it so we have it for both pynini and sh?
|
please update jenkinsfile |
| input=$PROJECT_DIR/ja/data_text_normalization/test_cases_whitelist.txt | ||
| runtest $input | ||
| } | ||
|
|
| Finite state transducer for classifying Japanese electronic expressions. | ||
|
|
||
| Examples: | ||
| abc@abc.com -> name: "abc アット abc ドット com" |
There was a problem hiding this comment.
this should be the electronic class instead of 'name'
| Finite state transducer for classifying Japanese address-like expressions. | ||
|
|
||
| Examples: | ||
| 東京都千代田区丸の内1-1-1 -> name: "東京都千代田区丸の内一の一の一" |
There was a problem hiding this comment.
let's make sure all tagger outputs are fully formatted properly
| @@ -1,5 +1,4 @@ | |||
| ゼロ 0 | |||
| 零 0 | |||
There was a problem hiding this comment.
why did we change this only for ITN? and do we need to update test cases for this?
| dr. ドクター | ||
| Mr. ミスター | ||
| mr. ミスター | ||
| Mrs. ミセス |
There was a problem hiding this comment.
let's not repeat terms in different tsv files, please keep in either this one or the original one
|
|
||
| digit = pynini.string_file(get_abs_path("data/numbers/digit.tsv")) | ||
| zero = pynini.string_file(get_abs_path("data/numbers/zero.tsv")) | ||
| zero_maru = pynini.cross("0", "〇") |
There was a problem hiding this comment.
let's add this as a tsv instead of hardcoding here
| @@ -0,0 +1 @@ | |||
| 0 零 | |||
There was a problem hiding this comment.
why is this different from zero.tsv and both from the ITN zero?
| graph_integer = ( | ||
| pynutil.insert('integer_part: \"') | ||
| + ( | ||
| pynini.cross("0", "零") |
There was a problem hiding this comment.
wouldn't this be zero_decimal?
| def __init__(self, deterministic: bool = True): | ||
| super().__init__(name="punctuation", kind="classify", deterministic=deterministic) | ||
| s = "!#$%&'()*+,-./:;<=>?@^_`{|}。，；：《》“”·~【】！？、‘’.<>-——_、。.「」『』‘`／・；’”“”‷･〔〕々〃ゝゞヽ〲〱〳〴〵ヾ〆，~" | ||
| s = "!#$%&'()*+,-./:;<=>?@^_`{|}~" |
There was a problem hiding this comment.
are we sure we don't need this punctuation?
| Examples: | ||
| B2A23C -> name: "ビー 二 エー 二三 シー" | ||
| MIG-25/235212-asdg -> name: "エムアイジー ハイフン 二五 スラッシュ 二三五二一二 ハイフン エーエスディージー" | ||
| Room 301 -> name: "Room 三〇一" |
There was a problem hiding this comment.
why would this be serial and not cardinal?
| covid_style = special_word + pynutil.delete("-") + insert_space + (NEMO_DIGIT**2 @ cardinal.just_cardinals) | ||
|
|
||
| model_number = pynini.accep("型番") + insert_space + delimited | ||
| room_number = pynini.accep("Room") + pynutil.delete(" ") + insert_space + digit_group |
There was a problem hiding this comment.
we definitely don't want something this restricted
| division = pynini.string_file(get_abs_path("data/time/division.tsv")) | ||
|
|
||
| division_component = pynutil.insert("suffix: \"") + division + pynutil.insert("\"") | ||
| hour_number = pynutil.add_weight(pynini.cross("0", "零"), -0.1) | graph_cardinal |
There was a problem hiding this comment.
again looks like zero_decimal?
|
|
||
| @parameterized.expand( | ||
| [ | ||
| ("119", "百十九"), |
There was a problem hiding this comment.
this should definitely not be here
2 participants
What does this PR do ?
Add a one line overview of what this PR aims to accomplish.
Before your PR is "Ready for review"
Pre checks:
git commit -sto sign.pytestor (if your machine does not have GPU)pytest --cpufrom the root folder (given you marked your test cases accordingly@pytest.mark.run_only_on('CPU')).bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...pytestand Sparrowhawk here.__init__.pyfor every folder and subfolder, includingdatafolder which has .TSV files?Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.to all newly added Python files?Copyright 2015 and onwards Google, Inc.. See an example here.try import: ... except: ...) if not already done.PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.