Repository navigation
word_segmenter: rename string_tokenizer and segment words on every platform - #548
Merged
Merged
Conversation
…atform `string_tokenizer` is taken on pub.dev and only covered macOS/iOS, returning token strings without positions. Rename it to `word_segmenter` and replace the API with `wordBoundaries(text)`, which returns contiguous UTF-16 ranges and null when no segmenter is available. Backends, all through the operating system so CJK text is split with its dictionaries (Flutter's bundled ICU omits them): - macOS/iOS: CFStringTokenizer - Windows: icu.dll (icuuc.dll on 1703-1809) - Linux: libicuuc.so.N with versioned symbols - Android: NDK libicu.so (API 31+) - web: Intl.Segmenter The ffigen bindings are replaced by a few hand-written lookups, which also fixes a leaked UTF-8 buffer per call. The example now targets all platforms and runs an integration test on Windows, macOS and Linux in CI, plus the unit tests in Chrome. Migration: `tokenize(text, options: [TokenizerUnit.wordBoundary])` becomes `wordBoundaries(text)!.map((r) => r.textInside(text))`. The other CFStringTokenizer units are gone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
string_tokenizeris taken on pub.dev and only ran on macOS/iOS. This renames it toword_segmenterand makes it segment words on every platform with the operating system's dictionaries, which Flutter's bundled ICU leaves out (TextPainter.getWordBoundarysplits Chinese per character).mixin_markdown_widgetwill use it for CJK double-click selection in a follow-up PR.nullmeans no segmenter (e.g. Android < 31), so callers fall back.ubrk_open_74); the loader probeslibicuuc.so.120down to.50, with and without the suffix.toNativeUtf8()buffer per call.word_segmenteris a separate release step.Migration for flutter-app (pinned to
66a8ed5, unaffected until it bumps):Evidence
wordBoundaries('Hello, 世界和平')→[Hello, ",", " ", 世界, 和平](macOS and Chrome).flutter test, integration test-d macosflutter test --platform chrome(JS and--wasm)icu.dll72.1)remote-testunit + integrationremote-testunitremote-testintegrationTests assert contiguous coverage,
Hello, world!→Hello | , | " " | world | !,我们今天讨论一下contains我们 / 今天 / 讨论,👍🏽kept whole, empty text →[].Before: the web build failed: Chrome threw
TypeError: Cannot convert undefined or null to objectinnew Intl.Segmenter(null, …).After: locales passed as an empty array; Chrome passes 5/5.
Not verified: Android below API 31 returning
null(no image locally), Windows 10 1703–1809icuuc.dll.Merge Danger
Door: two-way
The package was never published; the pub.dev name is only claimed at release time.
Blast Radius: none
No published consumer. flutter-app depends on a pinned git commit of
packages/string_tokenizerand keeps building; moving it toword_segmenteris a code change there (see migration). Touching rootpubspec.yamlmakes this PR run every integration target once.