Repository navigation
Localized voice files, fonts and SCPUI files via language subfolders - #8062
Goober5000 wants to merge 6 commits into
Conversation
|
@caasi @NovachenFS2 @Achille44 this should be relevant to your interests |
|
Thanks, this covers most of #8054. The part it leaves out is a voice language separate from the text language (for example, English text with a Chinese voice). Would you be OK with me adding that as a small follow-up PR on top of this one? |
I'd be OK with that. I am a little surprised that anyone would want to do that, since it sounds like a recipe for being very confused. But it can be an option. |
8c63aeb to
f0acaaa
Compare
I'd guess it's because voice acting is not that strong here in Taiwan. China used to be like Taiwan, but voice acting is a real industry there now, with many good young voice actors. In my experience, games like the MGS or The Division series have no Chinese voice option, so I pick the English or Japanese voices and read Chinese text, to enjoy the work of professional voice actors. I'm excited to see this PR merged. I will reorganize my mod to follow your folder structure. |
I actually support this, it is very common in many countries, to have localized Text to english Voice acting. Also in FS2 the official German localization only covers Voice Files for Command Briefing and In-Mission. |
dustinspace217
left a comment
There was a problem hiding this comment.
Reviewed at f0acaaa. The fix and the voice lookup both work in the game for the layout the description shows. Two layouts behave differently from what I expected, and one font path is left out. None of it blocks; they're yours to weigh.
Tested (Debug builds of master 5af50bd and this head, retail FS2 data plus a small test mod, Linux, OpenGL, sound on). The mod's originals are 1-second tones and its gr copies are 3-second tones, and I asked the engine through Lua whether a bare name exists and how long the voice stream it opens is. German was set with -language German; the mod carries a first font with an unrecognized checksum, because auto-detection otherwise wins over that flag.
voice/briefing/x1.oggplusvoice/briefing/gr/x1.ogg: on master an English player gets the German copy (the hole). With this PR English gets the original and German gets the copy.voice/briefing/gr/x3.oggwith no original: master finds it for everyone. With this PR English doesn't find it and German plays it.tables/gr/zzlang-test.tbm: found by bare name on master, not with this PR, as the mod author note says.- The four
CFileLanguageSubdirTesttests pass, with the rest of the cfile suites.
Two layouts that surprised me (details inline):
- An original in a subfolder,
voice/briefing/act1/x2.oggwithact1/gr/x2.oggbeside it: German still gets the original, on master and with this PR. - A language folder that isn't the innermost one,
voice/special/gr/act1/x4.ogg: still found by bare name in English with this PR.
From reading, not run:
- SCPUI loads its fonts by listing
*.ttfand*.otfinload_fonts()(rocket_ui.cpp:137) and opens them throughRocketFileInterfacewith a plaincfopen. A font that exists only indata/fonts/tw/is no longer listed, so SCPUI never loads it, and a same-named copy there is never swapped in. So thedata/fonts/tw/<name>.ttfnote holds for fonts.tbl fonts but not for SCPUI. cf_get_file_list_preallocated()has no sub-path filter, so pilot and squad images in a language folder are listed but can't be loaded by bare name.- Lua
message.VoiceFile.Filenamenow returnsgr/x.wavfor a localized message voice.
Not tested: fonts, mission messages and training voices in a mission, French and Polish, a mod-defined language, files inside a .vp, Windows.
| // except that localized files are only found by naming their language subfolder | ||
| if (search.empty()) { | ||
| return true; | ||
| return index.empty() || !is_language_sub_path(index); |
There was a problem hiding this comment.
Only the innermost folder is checked, and the localized lookup asks for exactly gr/<name>. Two results from my test mod:
voice/briefing/act1/x2.oggwithact1/gr/x2.oggnext to it: the copy is hidden from the bare lookup here, andcf_localize_filename()looks forgr/x2.ogg, which doesn't match the sub pathact1/gr/. German plays the original and nothing is logged. So "next to the originals" works at the top of a folder likevoice/briefing, but not for originals a mod keeps in subfolders.voice/special/gr/act1/x4.ogg: the innermost folder isact1, so this still matches a barex4.oggfor every player.
If both are intended, a line in the mod author notes would cover it.
A lookup that doesn't name a subfolder matches a file in any subfolder, first match in root order. So a mod that ships localized files in a language subfolder (data/fonts/gr/font01.vf, data/voice/briefing/tw/...) replaced the originals for every player, whatever language they chose. lcl_add_dir_to_path_with_filename() and its font fallback depend on the bare lookup finding only the original. The subfolder support from scp-fs2open#4771 noted that file localization conflicts with it and would need changes to the subfolder code; this is that change. Bare lookups and listings now skip files whose innermost folder is named after the extension of any language in Lcl_languages. Naming the subfolder (gr/font01.vf) or a */ glob still finds them. Ordinary subfolders and override priority are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
message_play_training_voice() opened the file by its exact name to decide whether to stream it or load it. The loaders it chooses between accept any audio extension, but the size check didn't, so a mission that named x.wav with only x.ogg on disk played no training voice. Look the file up with the audio extension list instead, and take the size from the lookup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With a language that has an extension, voice files now load from the language subfolder when a copy exists there, like fonts: German players get data/voice/briefing/gr/x.ogg in place of data/voice/briefing/x.ogg. Without a localized copy, the original plays. The localized copy wins wherever it is found. Overrides go by file name, so a mod that replaces x.ogg doesn't hide the base game's gr/x.ogg; a mod that wants to replace the localized line ships its own gr/x.ogg. cf_localize_filename() returns the name to load. It is used by voice streams in audiostream_open() (briefings, command briefings, debriefings, the fiction viewer, red alert, loop briefs, long training voices and Lua voice streams), by message_load_wave(), by the short training voice, and by the debriefing persona and FS1 Command checks, so a file that exists only in the language subfolder counts. The text language selects the voice language. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only .vf fonts were localized: loadFontOld() tried gr/<name> by hand before the bare name, while loadNVGFont() opened .ttf and .otf files as named. Both loaders now open their files with cfopen()'s localize flag, so a translation can ship a font with the glyphs it needs (data/fonts/tw/<name>.ttf) without changing the fonts other players get. The localize parameter of cfopen() was left unused when cfile gained subfolder support (scp-fs2open#4771). It now opens the copy given by cf_localize_filename() when reading, if that copy satisfies the location flags, and otherwise the file as named. Every existing caller passes false. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
f0acaaa to
4e9a1dc
Compare
|
@dustinspace217 Thanks for the thorough testing.
|
dustinspace217
left a comment
There was a problem hiding this comment.
Re-checked at 4e9a1dc. The fixes and the doc notes look right to me, but this head doesn't build with gcc or clang. The Linux and Mac CI jobs all fail on it; the Windows ones pass.
rocket_ui.cpp:155 calls SCP_vector_contains_lcase() with a string, and that template in vmallocator.h:65 calls lcase_equal(), which is only declared further down the header (line 118). MSVC accepts that; gcc and clang don't. A declaration above the template fixes it. I tested with extern bool lcase_equal(const std::string& _Left, const std::string& _Right); placed just before it.
With that one line added (Debug, Linux, OpenGL, retail data plus the same test mod as last time), English and German give the same results as in my first review, and the 46 cfile and language unit tests pass, including the new checks in language_subdir_only_localized. I didn't test the SCPUI changes in game.
4e9a1dc to
bf0cc8b
Compare
|
@dustinspace217 fixed compilation |
dustinspace217
left a comment
There was a problem hiding this comment.
Re-checked at bf0cc8b. The compile fix works: gcc builds this head without warnings, and the 46 cfile and language unit tests pass.
One thing left for CI: clang-tidy flags list_fonts in rocket_ui.cpp as a static definition inside an anonymous namespace. That's the only failing job; the other Linux jobs were cancelled after it.
|
|
||
| void load_fonts() | ||
| // the fonts with this extension, including those that only exist in the current language's subfolder | ||
| static SCP_vector<SCP_string> list_fonts(const char* ext) |
There was a problem hiding this comment.
clang-tidy (readability-static-definition-in-anonymous-namespace): the anonymous namespace already makes this internal.
| static SCP_vector<SCP_string> list_fonts(const char* ext) | |
| SCP_vector<SCP_string> list_fonts(const char* ext) |
RocketFileInterface::Open() now opens files with cfopen()'s localize flag, so a translation can ship its own copy of an RML document, a stylesheet or a font in the language subfolder. SCPUI paths only choose the path type and cfile finds the file by name, so this follows the same rule as other localized files. Images are loaded through bmpman, so they are not affected. load_fonts() also lists the fonts in the current language's subfolder, because the bare listing skips them, and a font that exists only there would otherwise not be loaded. libRocket picks fonts by the family name stored in the file, so a localized font has to keep the original's family name to replace it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bf0cc8b to
e60ade5
Compare
The engine has long had a way to load a localized copy of a file from a subfolder named after the language's extension, e.g.
data/fonts/gr/font01.vffor German. Fonts were its only user, and since cfile gained subfolder support (#4771) the scheme has had a hole. This PR fixes the hole, then extends the scheme to voice files, TrueType fonts and SCPUI files.A translation or voice pack can now ship its files next to the originals, under the same names:
With no localized copy, the original plays. English, retail hybrid and untranslated have no extension, so for them nothing changes and no extra lookups happen.
1. Keep bare-name lookups out of language subfolders
Since #4771, a lookup that doesn't name a subfolder matches a file in any subfolder, and the first match in root order wins. So a higher-priority mod that shipped
data/fonts/gr/font01.vfordata/voice/briefing/gr/x.oggreplaced the original for every player, whatever their language. #4771 noted that localization would need changes to the subfolder code; this is that change.Lookups and listings that don't name a subfolder now skip files whose innermost folder is the extension of any language in
Lcl_languages. Naming the subfolder (gr/font01.vf) or using a*/glob still finds them. Ordinary subfolders and override priority are unchanged.2. Find training voices with any audio extension (retail bug)
message_play_training_voice()opened the file by its exact name to decide whether to stream it or load it. Both loaders accept any audio extension, but the size check didn't, so a mission namingx.wavwith onlyx.oggon disk played no training voice. The size now comes from an extension-agnostic lookup. This is independent of localization, and it's committed separately.3. Load voice files from the language subfolder
New
cf_localize_filename()returns the name to load: the localized copy if it exists, otherwise the name as given. It's used by:audiostream_open()forASF_VOICEstreams: briefings, command briefings, debriefings, the fiction viewer, red alert, loop briefs, long training voices and LuaAUDIOSTREAM_VOICEstreams;message_load_wave(), for mission and built-in messages;Design decisions:
x.oggreplaces onlyx.ogg; it doesn't hide the base game'sgr/x.ogg. A mod that wants to replace the localized line ships its owngr/x.ogg. This matches how fonts already behaved, and how text works: a mod's new lines aren't translated until it supplies translations.MAX_FILENAME_LEN. If the localized name would be 32 characters or longer, the original is used and a line is logged.4. Bring back
cfopen()'slocalizeflag; TrueType fonts_cfopen()has carried an unusedbool localizeparameter since #4771. When reading, it now opens the localized copy if there is one and it satisfies the caller's location flags, and otherwise the file as named. Write modes ignore it. Every existing caller passesfalse; they only fill that position to reachlocation_flags, and they all open user or generated files (pilot and campaign files, control presets, the shader cache).Both font loaders now use
cfopen(name, "rb", CF_TYPE_ANY, true)..vffonts behave as before, but the hand-written fallback is gone..ttfand.otffonts are localized for the first time, so a translation can ship a font with the glyphs it needs (data/fonts/tw/<name>.ttf) without changing anyone else's fonts.5. SCPUI files
RocketFileInterface::Open()now opens every file withlocalize, so a translation can ship its own RML documents, stylesheets and fonts in the language subfolder. SCPUI paths only choose the path type, and cfile finds the file by name, so the same rule applies as for other localized files. Images are loaded through bmpman and aren't affected.load_fonts()also lists the current language's font subfolder, so a font that exists only there is still loaded. libRocket picks fonts by the family name inside the file, so a localized font must keep the original's family name to replace it.Notes for mod authors
+Extension:instrings.tblor a*-lcl.tbm. The built-in German, French and Polish usegr,frandpl.gr,fr,plor one of its own language extensions can no longer reach them by bare name.data/voice/briefing/gr/x.ogg. If the original is in an ordinary subfolder (data/voice/briefing/act1/x.ogg), the localized copy still goes indata/voice/briefing/gr/, notact1/gr/. Only the innermost folder counts as a language folder, so files ingr/act1/are ordinary files that every player gets.Filenameof the message's voice file includes the subfolder (gr\x.wav).🤖 Generated with Claude Code