Skip to content

Japanese TN fix and improvement #444

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
folivoramanh wants to merge 2 commits into
base: staging/ja_tn_v1
Choose a base branch
from
Open

Japanese TN fix and improvement #444

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ffa2255
Japanese TN fix and improvement
folivoramanh Jul 2, 2026
86b45aa
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion nemo_text_processing/inverse_text_normalization/ja/data/numbers/zero.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
ゼロ 0
零 0

@mgrafu mgrafu Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why did we change this only for ITN? and do we need to update test cases for this?

れい 0
○ 0
〇 0
Expand Down
2 changes: 1 addition & 1 deletion nemo_text_processing/inverse_text_normalization/ja/data/time_hours.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@
二十二 22
二十三 23
二十四 24
零 0
ゼロ 0
4 changes: 2 additions & 2 deletions nemo_text_processing/inverse_text_normalization/ja/taggers/fraction.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ def __init__(self, cardinal: GraphFst, decimal: GraphFst):
e.g.,
四分の三 -> fraction { denominator: "4" numerator: "3" }
一と四分の三 -> fraction { integer: "1" denominator: "4" numerator: "3" }
一荷四分の三 -> fraction { integer: "1" denominator: "4" numerator: "3" }
一と四分の三 -> fraction { integer: "1" denominator: "4" numerator: "3" }
ルート三分の一 -> fraction { denominator: "√3" numerator: "1" }
一点六五分の五十 -> fraction { denominator: "1.65" numerator: "50" }
二ルート六分の三 -> -> fraction { denominator: "2√6 " numerator: "3" }
Expand All @@ -40,7 +40,7 @@ def __init__(self, cardinal: GraphFst, decimal: GraphFst):
pynutil.delete("分の") | pynutil.delete(" 分 の　") | pynutil.delete("分 の　") | pynutil.delete("分 の")
)

integer_word = pynutil.delete("と") | pynutil.delete("荷")
integer_word = pynutil.delete("と")
root_word = pynini.accep("√") | pynini.cross("ルート", "√")

graph_sign = (
Expand Down
13 changes: 13 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/electronic/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
2 changes: 2 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/electronic/card_cues.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
カード末尾
カード番号
6 changes: 6 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/electronic/file_extensions.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
.jpg ドット jpg
.JPG ドット JPG
.png ドット png
.PNG ドット PNG
.pdf ドット pdf
.PDF ドット PDF
1 change: 1 addition & 0 deletions nemo_text_processing/text_normalization/ja/data/electronic/special_digit_runs.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
01 ゼロイチ
5 changes: 5 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/electronic/symbol.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
@ アット
. ドット
- ハイフン
/ スラッシュ
: コロン
13 changes: 13 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/latin/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
52 changes: 52 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/latin/letters.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
A エー

@mgrafu mgrafu Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of mapping upper and lower case separately, let's add only one and capitalize the inputs in the graph itself

B ビー
C シー
D ディー
E イー
F エフ
G ジー
H エイチ
I アイ
J ジェー
K ケー
L エル
M エム
N エヌ
O オー
P ピー
Q キュー
R アール
S エス
T ティー
U ユー
V ブイ
W ダブリュー
X エックス
Y ワイ
Z ゼット
a エー
b ビー
c シー
d ディー
e イー
f エフ
g ジー
h エイチ
i アイ
j ジェー
k ケー
l エル
m エム
n エヌ
o オー
p ピー
q キュー
r アール
s エス
t ティー
u ユー
v ブイ
w ダブリュー
x エックス
y ワイ
z ゼット
13 changes: 13 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/measure/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
33 changes: 33 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/measure/per_unit.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
h 時
ｈ 時
hour 時
hours 時
s 秒
ｓ 秒
sec 秒
second 秒
seconds 秒
min 分
ｍｉｎ 分
minute 分
minutes 分
kg キロ
ｋｇ キロ
g グラム
ｇ グラム
m メートル
ｍ メートル
km キロ
ｋｍ キロ
cm センチメートル
ｃｍ センチメートル
mm ミリメートル
ｍｍ ミリメートル
L リットル
Ｌ リットル
l リットル
ｌ リットル
ml ミリリットル
mL ミリリットル
ｍｌ ミリリットル
ｍＬ ミリリットル
4 changes: 4 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/measure/rate_numerator.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
km キロ
ｋｍ キロ
m メートル
ｍ メートル
79 changes: 79 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/measure/unit.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
kg キロ
ｋｇ キロ
キロ キロ
キログラム キログラム
g グラム
ｇ グラム
グラム グラム
mg ミリグラム
ｍｇ ミリグラム
ミリグラム ミリグラム
μg マイクログラム
µg マイクログラム
マイクログラム マイクログラム
km キロ
ｋｍ キロ
キロメートル キロ
m メートル
ｍ メートル
メートル メートル
cm センチ
ｃｍ センチ
センチ センチ
センチメートル センチメートル
mm ミリメートル
ｍｍ ミリメートル
ミリメートル ミリメートル
m2 平方メートル
m² 平方メートル
ｍ2 平方メートル
ｍ² 平方メートル
平方メートル 平方メートル
cm2 平方センチメートル
cm² 平方センチメートル
平方センチメートル 平方センチメートル
L リットル
l リットル
Ｌ リットル
ｌ リットル
リットル リットル
mL ミリリットル
ml ミリリットル
ｍＬ ミリリットル
ｍｌ ミリリットル
ミリリットル ミリリットル
% パーセント
％ パーセント
パーセント パーセント
° 度
º 度
度 度
°C 度
℃ 度
ºC 度
°F 度エフ
℉ 度エフ
W ワット
w ワット
ワット ワット
kW キロワット
kw キロワット
キロワット キロワット
V ボルト
v ボルト
ボルト ボルト
Hz ヘルツ
hz ヘルツ
ヘルツ ヘルツ
kHz キロヘルツ
khz キロヘルツ
MHz メガヘルツ
mhz メガヘルツ
GHz ギガヘルツ
ghz ギガヘルツ
GB ギガバイト
gb ギガバイト
ギガバイト ギガバイト
MB メガバイト
mb メガバイト
メガバイト メガバイト
1 change: 1 addition & 0 deletions nemo_text_processing/text_normalization/ja/data/money/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.

@mgrafu mgrafu Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's add full copyright in all files

27 changes: 27 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/money/currency_major.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
¥ 円
￥ 円
JPY 円
jpy 円
円 円
$ ドル
US$ ドル
USD ドル
usd ドル
ドル ドル
米ドル ドル
€ ユーロ
EUR ユーロ
eur ユーロ
ユーロ ユーロ
£ ポンド
GBP ポンド
gbp ポンド
ポンド ポンド
₩ ウォン
KRW ウォン
krw ウォン
ウォン ウォン
CNY 元
cny 元
元 元
人民元 人民元
2 changes: 2 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/money/currency_minor.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
銭 銭
セント セント
15 changes: 15 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/money/currency_minor_by_major.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
¥ 銭
￥ 銭
JPY 銭
jpy 銭
円 銭
$ セント
US$ セント
USD セント
usd セント
ドル セント
米ドル セント
€ セント
EUR セント
eur セント
ユーロ セント
19 changes: 19 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/money/currency_prefix.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
¥ 円
￥ 円
JPY 円
jpy 円
$ ドル
US$ ドル
USD ドル
usd ドル
€ ユーロ
EUR ユーロ
eur ユーロ
£ ポンド
GBP ポンド
gbp ポンド
₩ ウォン
KRW ウォン
krw ウォン
CNY 元
cny 元
3 changes: 3 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/money/quantity.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
万 万

@mgrafu mgrafu Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we shouldn't need to have the chars map to themselves, should be enough just to have the list of chars

億 億
兆 兆
2 changes: 1 addition & 1 deletion nemo_text_processing/text_normalization/ja/data/numbers/zero.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0 零
0 ゼロ
1 change: 1 addition & 0 deletions nemo_text_processing/text_normalization/ja/data/numbers/zero_decimal.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0 零

@mgrafu mgrafu Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this different from zero.tsv and both from the ITN zero?

13 changes: 13 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/roman/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
4 changes: 4 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/roman/key_cardinal.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Chapter
chapter
Part
Super Bowl
2 changes: 2 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/roman/key_ordinal.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Henry

@mgrafu mgrafu Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's instead only use common terms like century or chapter, but not cover just a couple of king names

Louis
1 change: 1 addition & 0 deletions nemo_text_processing/text_normalization/ja/data/roman/preserve.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
World War II World War II

@mgrafu mgrafu Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we preserve this?

13 changes: 13 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/roman/roman_numerals.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
I 1
V 5
X 10
L 50
C 100
D 500
M 1000
IV 4
IX 9
XL 40
XC 90
CD 400
CM 900
13 changes: 13 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/serial/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
3 changes: 3 additions & 0 deletions nemo_text_processing/text_normalization/ja/data/serial/words.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
covid コビッド
COVID コビッド
Covid コビッド
Loading