Fix/el teen gender forms and sparrowhawk server#453
Closed
georgepar wants to merge 4 commits into
Closed
Conversation
Add full TN and ITN grammar coverage for Greek (el), wired into Normalizer / InverseNormalizer and the Sparrowhawk export tooling. TN taggers/verbalizers: cardinal, ordinal (full gender/case agreement), decimal, fraction, date, time, measure, money, telephone, electronic, roman, range, serial, whitelist, punctuation, word. Numbers render in neuter citation form with feminine thousands, up to 10^12. ITN taggers/verbalizers: cardinal, ordinal, decimal, fraction, date, time, measure, money, telephone, electronic, whitelist, punctuation, word. Deployment: - Register el in tools/text_processing_deployment/pynini_export.py and add the Greek Sparrowhawk test scripts. - Commit the exported TN/ITN classify+verbalize .far grammars; gitignore the regenerable grammar cache .far files. - Add tools/text_processing_deployment/GREEK_TN_ITN_GUIDE.md with usage and Linux Sparrowhawk deployment instructions. Notes: - Greek acronyms are handled only via the whitelist; there is no generic all-caps letter-split (it mis-spells all-caps words like ΙΝΤΡΑΚΟΜ and Greek acronym pronunciation is irregular and lexicon-dependent). - Measure ITN uses SI symbols as canonical output (e.g. μέτρα -> m). Signed-off-by: Giorgos Paraskevopoulos <geopar@central.ntua.gr>
Greek cardinals 3 and 4 inflect for gender, and digit.tsv already lists both forms (τέσσερα/τέσσερις). The teens that embed them — 13 and 14 — were only present in their neuter form in teen.tsv, so feminine inputs like "δεκατέσσερις Ιουλίου" (the natural way to say "14th of July") had no path and failed, while neuter "δεκατέσσερα" worked. - teen.tsv: add δεκατρείς (13), δεκατέσσερις and variant δεκατέσσερεις (14) - inverse_normalize.py: register 'el' in the --language argparse choices (omitted when Greek was added, so the CLI rejected el) - add ITN cardinal/date regression cases - regenerate el_itn_grammars_cased classify FAR to match Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HC8SksXmtP6LS8u84H3WqW Signed-off-by: Giorgos Paraskevopoulos <g.paraskevopoulos@athenarc.gr>
FastAPI service that serves both Text Normalization and Inverse Text Normalization from one container via Google Sparrowhawk, running the compiled .far grammars exported by nemo_text_processing. - Two persistent normalizer_main processes (TN + ITN), each behind a worker pool, wrapped by /tn, /itn, /normalize, /health endpoints - normalizer.py launches the C++ binary under `stdbuf -oL` and drains stderr on a side thread, so the block-buffered pipe can't deadlock the request/response loop (matches the test_sparrowhawk_*.sh contract) - Dockerfile builds FROM the existing sparrowhawk:latest base and bakes in the exported grammars; parameterized per language via build args or bind mounts - entrypoint.sh wires each direction's .far + ascii_proto config into a cwd-relative serving dir - build_and_run.sh one-shot export/build/run helper - tests/: fake normalizer_main + full-stack local test (no image needed) - README with API, deploy, config, and the buffering rationale Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HC8SksXmtP6LS8u84H3WqW Signed-off-by: Giorgos Paraskevopoulos <g.paraskevopoulos@athenarc.gr>
Satisfy the repo pre-commit hooks (black --line-length=119 --skip-string-normalization, isort --line-width=119, and requirements-txt-fixer) over the Greek TN/ITN and Sparrowhawk server files so the pre-commit.ci check passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JxHtWQkMhX77mE3EeQsVHi Signed-off-by: Giorgos Paraskevopoulos <g.paraskevopoulos@athenarc.gr>
georgepar
force-pushed
the
fix/el-teen-gender-forms-and-sparrowhawk-server
branch
from
b5f1819 to
b1ab13b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Add a one line overview of what this PR aims to accomplish.
Before your PR is "Ready for review"
Pre checks:
git commit -sto sign.pytestor (if your machine does not have GPU)pytest --cpufrom the root folder (given you marked your test cases accordingly@pytest.mark.run_only_on('CPU')).bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...pytestand Sparrowhawk here.__init__.pyfor every folder and subfolder, includingdatafolder which has .TSV files?Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.to all newly added Python files?Copyright 2015 and onwards Google, Inc.. See an example here.try import: ... except: ...) if not already done.PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.