[siracozmen01]

Problem

An editor asks about one position at a time: Backspace removes the grapheme cluster that ends at the caret, the arrow keys move to the cluster or word boundary before or after it, and a click selects the word under the pointer. The iterator only steps forward from the start of the text, so maul-ui finds the boundary before an offset by stepping a fresh iterator from the paragraph's start (ClusterAround in its text_delete.c, text_mask.c, text_move.c and text_rules.c). Each key press then costs time linear in the paragraph before the caret. ICU answers the same question with preceding() and following().

Change

muniSegmentIterator_FindBreaksAround(iterator, offset, &before, &after) gives the last boundary at or before a byte offset and the first after it, as the iterator reports them, for every kind and setting: grapheme clusters, words and sentences, line breaks at every strictness and writing system, class overrides and a word segmenter included. It needs an iterator given its whole text at init, reads only that text and the settings, and leaves the iterator as it was wherever its own iteration stands. The struct does not change: the call is additive, a minor release.

The call backs up from the offset to a boundary the rules make whatever came before, and reads forward from there. Each rule file gets a window function: given the one or two code points before a position, it says whether they fix every part of the rule set's state that a later decision reads. A part may stay open only if it can make a segmenter started at the window's start join at the position where the whole text breaks, never the reverse, and the code point at the position sets it again. The windows were derived from the rule code, and each one's comment says why it holds:

• Grapheme clusters: one code point, unless it is InCB=Extend, which carries a GB9c linker on. An Extend may leave an emoji sequence open and a regional indicator the count, but those only decide after a ZWJ (InCB=Extend since Unicode 15.1, which a test pins) and between two regional indicators, where the window's segmenter joins. A line iterator that breaks anywhere uses the same window.
• Words: two code points, neither skipped by WB4, the second no regional indicator. A hold can be open only at the second, and whether it is depends on the two alone.
• Sentences: a paragraph separator; or a terminator, or a terminator and a space, with no Extend or Format before the terminator.
• Lines: a hard break, or two code points of AL, HL, ID, EB, EM or NU of which the second may be a space.
• With a word segmenter set, no position between two SA code points.

Whether there is a boundary at a window's end, the engine decides: a segmenter started at the window's start reads a piece cut after the code point at the position, so the check costs a few code points and the whole call stays linear in the stretch it reads. Code point starts are found as reading from the start finds them, ill-formed UTF-8 included.

• include/maul-unicode/segment.h: the call and its documentation.
• src/segment.c: the search and the run; src/segmenter.h, src/grapheme.c, src/word.c, src/sentence.c, src/line_rules.h, src/line.c, src/line_tailored.c, src/line_anywhere.c, src/complex.c: the windows.
• docs/user/segmentation.md: "Around the caret", with a snippet for Backspace and the arrow keys, run by test/test_guide.c. docs/internals/segmentation.md: how the windows and the check work.

Verification

test/test_segment.c checks the call at every byte offset against stepping from the start, at every kind and setting (13 setups: grapheme, anywhere, word, word with overrides, word with a breaker, sentence, sentence with overrides, line, line with a breaker, strict, loose, normal and loose for Chinese and Japanese), half the offsets on an iterator already stepped to its end:

• every case of GraphemeBreakTest.txt (853), WordBreakTest.txt (1944), SentenceBreakTest.txt (512) and LineBreakTest.txt (19346) under the setups of its kind;
• the cases of each file joined into one text of up to 8 KiB, where the call backs up across cases;
• 616 random texts of the code points the four files use, SA letters and marks and ill-formed UTF-8, 16 of them about a kilobyte, under all 13 setups;
• hand-written edge texts for what a wrong window would miss: a GB9c conjunct whose linker a nukta carries on, Thai runs with a breaker, five regional indicators, the abbreviations of SB7 and SB8 with a mark before the full stop, an aksara sequence around U+25CC, a prefix before an opening bracket.

Each text sits alone in its allocation, so AddressSanitizer sees any read past it. The refusals are tested too: a NULL argument or an offset past the end (muni_errorInvalid), text in pieces (muni_errorState), a call from inside the word segmenter (muni_errorBusy), an iterator no init call set up.

Mutants of the windows, each run against that test: dropping any one condition fails it (an InCB=Extend code point as a grapheme window, an Extend or regional indicator in a word window, a word or line window inside an SA run, an Extend or Format before a sentence terminator, a combining mark in a line window), as do a line window with no condition and leaving out the boundary check. Nine mutants, nine killed. The first push carried four more conditions (Extend, ZWJ and regional indicators in the grapheme window, U+25CC in the line window) that no mutant could fail; the boundary check or the window's own claim already covered them, so the second commit removes them. The grapheme window's claim rests on U+200D being InCB=Extend, which test_segment.c now asserts.

test/fuzz_segment.c checks the call at every offset of an input up to 32 bytes and about 16 of a longer one, against the boundaries the iterator reports for the input's kind, its class overrides, its line settings and the wild breaker. A 300 second run from the committed seeds (125,300 runs) found nothing.

Time per offset on one paragraph of English prose (the UDHR preamble, one long sentence, repeated), mean of 64 offsets spread over it, best of 5, gcc-14 -O2 Release, against stepping a fresh iterator from the start past the offset as maul-ui does now:

Kind	1 KiB: step, around	16 KiB: step, around	256 KiB: step, around
Grapheme clusters	10.0 µs, 92 ns	151 µs, 88 ns	2.36 ms, 84 ns
Words	7.4 µs, 299 ns	114 µs, 299 ns	1.80 ms, 303 ns
Lines	8.2 µs, 491 ns	127 µs, 453 ns	1.97 ms, 415 ns
Sentences	8.0 µs, 12.9 µs	74.5 µs, 26.5 µs	1.03 ms, 24.4 µs

Grapheme clusters, words and lines cost the same at any length. Sentences back up to the last terminator, which in this text comes once every 1.9 KB, so within the first kilobyte the call reads back to the start and costs about 1.6 times stepping; in text with shorter sentences it costs a few hundred bytes.

On the branch rebased on main: tools/check.sh changed passes (the rules with the nodiscard check, format, clang-tidy in Release and Debug, linux-clang-20-asan's 13 suites), and linux-gcc-14 builds with -Werror and passes 13/13.

Closes #240

[siracozmen01]

Review at f06a8fa: ready. Read, not run.

1. The windows are sound for their rules. A segmenter started at a window's start reaches the window's end in the state the whole text gives, for everything a later decision reads. Where it does not, BreaksAt only refuses a restart, so the error goes the safe way.

• Grapheme, and line "anywhere". The window keeps codePoint < 0x80 || InCB != Extend.
  • Every field of muniGraphemeAdvance is set by the code point alone except three:
    • the linker, which InCB=Extend carries on: excluded;
    • the emoji state, which Extend and ZWJ carry on;
    • the regional-indicator parity.
  • An open emoji state matters only after a ZWJ (GB11). U+200D is InCB=Extend, which the new assertion pins, so a window never ends at a ZWJ.
  • After an Extend with the state open, the next code point either joins by GB9 (an Extend or ZWJ), so BreaksAt refuses it, or resets the emoji state in both runs.
  • A fresh odd count only ever joins two regional indicators, so BreaksAt refuses where the whole text would break.
  • A Linker passes the window. That is right for today's engine, where a Linker sets linker alone. As you say, it must be excluded once #279 makes the linker need a consonant before it.
• Words.
  • Two code points neither skipped by WB4, the second no regional indicator, are the actual, previous and before classes.
  • The holding rules (WB6, WB7b, WB12) hold across one middle code point only. So a hold opened before the window has been resolved by its second code point, and one opening at its end depends on those two and on lookahead, which BreaksAt cannot get from a piece ending after next. It then returns false.
  • Overrides go through the segmenter's own Lookup.
  • An SA run with a breaker is refused only between two SA code points. A run cut at the window's start is answered only for boundaries inside the window, which the final scan drops.
• Sentences.
  • A ParaSep resets everything later decisions read.
  • A terminator starts its ending whatever came before. With first not Ignored, the class SB7 reads before an ATerm is first's own.
  • Terminator + Sp keeps the ending in the window.
• Lines.
  • After a hard break, only LB19a, 20a, 21a and 28a read the class before the previous one, and only after QU, HY or a virama, never after BK.
  • Two Plain (AL, HL, ID, EB, EM, NU), the second possibly SP, fix the previous and before classes, the last class other than SP (so no OP-SP*, CL/CP-SP* or B2-SP* hold open), the LB25 number state and the RI parity.
  • Dropping U+25CC is sound. Every LB28a hold that involves it opens at or after the window's start, where the local segmenter sees it.
  • The tailoring's allowances read the previous class and code point and the next, all inside the window.

2. The survivors. The grapheme window is now the minimal sound one, and the line window lost the U+25CC condition. The two added-option mutants (a space in the sentence window, OP in the line window) are documented as left out. No condition in the shipped code is dead.

3. The iterator is const, and the offset is checked.

• The call loads the iterator into a local copy and builds fresh local segmenters on its text. It writes only *beforeOut and *afterOut.
• An offset past the end is refused before any byte is read. offset == length returns at once.
• Align, CodePointAt and BreaksAt read only below length: start <= offset < length, and the end of BreaksAt's piece is position + nextSize <= length.
• Overrides exist only for words and sentences, and InitLocal copies both counts, the line settings and the breaker.

Docs. "Around the caret" and the internals section describe the windows as they now are. The body has Problem, Change and Verification, and feat: fits an additive 1.x call.

[siracozmen01]

Correction to my review at f06a8fa: drop my note that a Linker must leave the grapheme window "once #279 ..."

#279 was not a bug. Unicode 18.0's GB9c joins a linker to the next consonant with no consonant before it. The repo's GraphemeBreakTest.txt (18.0.0) has ÷ 094D × 0915 ÷. I had read it against the 15.1 rule.

So the engine's state, a Linker setting linker on its own, is right. A window ending at a Linker is sound as it stands, and the verdict is unchanged: ready.