feat: refuse an iterator that no init call set up

.github/workflows/ci.yml

@@ -248,7 +248,7 @@ jobs:
         #
         # A dispatch's fuzz-seconds must leave time for the minimize and
         # save, or the timeout kills the job and the corpus is lost: the
-        # ten targets run in three waves of four, and the night's 50
+        # eleven targets run in three waves of four, and the night's 50
         # minutes less about 10 for the build, the merge and the save leave
         # 40, so at most 800 seconds a wave. libFuzzer reads a negative
         # time as no limit.

CMakeLists.txt

@@ -224,7 +224,7 @@ if(MAUL_UNICODE_FUZZ)
     target_compile_options(maul-unicode PRIVATE -fsanitize=fuzzer-no-link,address,undefined
                            -fno-sanitize-recover=all)
     target_link_options(maul-unicode PUBLIC -fsanitize=address,undefined -fno-sanitize-recover=all)
-    set(MUNI_FUZZ_AREAS encoding segment bidi bidi_long script)
+    set(MUNI_FUZZ_AREAS encoding segment bidi bidi_long script iterators)
     if(MAUL_UNICODE_NORMALIZATION)
         list(APPEND MUNI_FUZZ_AREAS normalize)
     endif()
@@ -249,6 +249,9 @@ if(MAUL_UNICODE_FUZZ)
         target_link_options(fuzz_${area} PRIVATE -fsanitize=fuzzer,address,undefined
                             -fno-sanitize-recover=all)
     endforeach()
+    # fuzz_iterators sets the fields of the iterators' state, which only
+    # src/ declares.
+    target_include_directories(fuzz_iterators PRIVATE src)
     # ICU is an oracle for the tests alone, never linked into the library.
     if(TARGET fuzz_icu)
         find_package(PkgConfig REQUIRED)

docs/internals/overview.md

@@ -77,7 +77,7 @@ Outside the promise:
 - `MUNI_MAX_SCRIPT_EXTENSIONS`, `MUNI_MAX_RESOLVED_SCRIPTS` and `MUNI_MAX_DECOMPOSITION` hold for one Unicode version and may rise in a minor release; the calls always report the count they need.
 - The size of the iterator structs across minor releases.
 - Everything outside `include/`: `src/`, the generated tables and their sizes, speed, the tools, tests, samples and benchmarks, and the HarfBuzz versions the adapter builds with.
-- What a call does when its caller breaks a header's contract, such as a workspace that overlaps its text or an iterator that was never initialized.
+- What a call does when its caller breaks a header's contract, such as a workspace that overlaps its text, or an iterator whose bytes were changed by hand while keeping the cookie its init call wrote. An iterator no init call set up is refused with `muni_errorInvalid`, which is promised.
 
 The library reads and writes no asset or state format, so it promises none, and its headers name no other Maul library that would have to reach 1.0.0 first.
 

docs/internals/segmentation.md

@@ -8,7 +8,9 @@ UAX #29 defines three segmentations of the same shape: walk the text and decide,
 
 The public struct is opaque storage. [`src/segment.c`](../../src/segment.c) copies the internal `muniSegmenter` in and out of it with `memcpy`, which keeps every access within the C aliasing rules, and dispatches on the kind once per boundary, not per code point.
 
-Its size follows the state: a `static_assert` in [`src/segmenter.h`](../../src/segmenter.h) keeps `muniSegmenter` within it, so a change that grows the state fails to build until the public iterator grows with it. On 64-bit targets `muniSegmenter` fills all 128 bytes, with a line iterator's strictness and writing system packed in the byte after the hold's kind and the class overrides' pointer in the last 8. The next switch or piece of state therefore grows the public iterator, which changes the ABI, so its pull request's title carries `!`.
+The caller holds that storage, so every call checks what it copied out before it reads it, as the family's rule on state a caller holds asks ([maul-conventions.md](maul-conventions.md)): the cookie an init call writes in the cursor, every count against the storage it counts (the cursor's pending bytes and position, the override count), every class, state and hold kind against the values its rule set writes, and every flag's byte as a 0 or 1 before the copy is read as a `bool`. It also checks what the assertions downstream rely on: a waiting cursor holds less than a code point, and a hold lies before the cursor. An iterator no init call set up is refused with `muni_errorInvalid`, so no bit pattern makes a call read outside the state; the pointers it keeps (the text, the overrides, the caller's breaker) cannot be checked, and one forged there is outside the contract. The script iterator ([`src/script_runs.c`](../../src/script_runs.c)) checks its state the same way. The checks run once per call, not per code point. They are inlined into each call (`MUNI_ALWAYS_INLINE`), since the compiler left them outlined at 1 to 2 ns more per call, and they read the flags' bytes from the caller's iterator rather than from the copy just made of it, so that no load waits on the copy's stores. On the bench, where every call reports one boundary, they cost a grapheme break about 2 ns (11%) and a line break about 5 ns (20%), all of it instructions executed: they add no branch or cache miss. `test/fuzz_iterators.c` sets every byte outside those pointers, and the lengths and counts that go with them, from its input and calls every function again.
+
+Its size follows the state: a `static_assert` in [`src/segmenter.h`](../../src/segmenter.h) keeps `muniSegmenter` within it, so a change that grows the state fails to build until the public iterator grows with it, and on 64-bit targets a second one keeps the iterator from growing a whole `uint64_t` more than the state needs. On 64-bit targets `muniSegmenter` fills all 128 bytes, with the cookie in the two bytes that were the cursor's padding, a line iterator's strictness and writing system packed in the byte after the hold's kind and the class overrides' pointer in the last 8. The next switch or piece of state therefore grows the public iterator, which changes the ABI; from 1.0.0 that is allowed in a minor release, so its pull request is a `feat` with no `!`.
 
 ## The engine and its holds
 

docs/operations/checks.md

@@ -45,7 +45,7 @@ A test that fails without a fault in the change under test is quarantined, as [m
 
 Every fuzz target (`test/fuzz_<area>.c`) checks properties that must hold for any input rather than known answers: no crash, no broken assertion, piecewise iteration giving the whole-text answer, round trips, idempotence where the standard promises it, agreement between parts (validation and conversion, normalization and its quick check, NFKC_Casefold and the normal forms), and output through a short buffer matching output through a large one. `fuzz_icu` compares with ICU, a test-only oracle, where both Unicode versions agree; its header lists what it leaves out and why. The targets take at most 256 bytes, but `fuzz_bidi_long` repeats its input into a text of 2 KiB to 128 KiB, so work that grows faster than the text runs past libFuzzer's 30-second timeout and is a finding.
 
-Each target's corpus carries over from run to run in one Actions cache entry, `fuzz-corpus-<run id>`, holding a directory per target. Every run restores the latest entry, a pull request's included, and adds the committed seeds, which are the floor. After fuzzing, runs on `main` and at night minimize each target's corpus through `tools/fuzz_target.sh --merge` into a fresh directory, swap it in only when the merge finished, and save a new entry. A clean merge prints one line, the target and the size of its merged corpus; an input that crashes the target during the merge fails the step and prints the same report as a fuzzing finding, the target, the kind of finding and the sanitizers' report, never the input. A run whose fuzzing or merge found something saves nothing. A pull request's run reads the corpus too, so a change that breaks a harness's promise fails on its own pull request and not later on `main`. A pull request, or a dispatch on any other branch, never minimizes or saves it, so only `main` writes the corpus. A target whose minimized corpus passes 25 MB fails the step, and the corpora are not saved, so one target cannot crowd the others out of the cache. The `prune-caches` job then deletes the entry the new one replaced. The organization's other libraries run one fuzz job per target, each with its own cache. This repository has ten targets in one job, four at a time, because ten more containers per run would cost more runner time than the targets' corpora need, and the corpora are still kept and minimized separately.
+Each target's corpus carries over from run to run in one Actions cache entry, `fuzz-corpus-<run id>`, holding a directory per target. Every run restores the latest entry, a pull request's included, and adds the committed seeds, which are the floor. After fuzzing, runs on `main` and at night minimize each target's corpus through `tools/fuzz_target.sh --merge` into a fresh directory, swap it in only when the merge finished, and save a new entry. A clean merge prints one line, the target and the size of its merged corpus; an input that crashes the target during the merge fails the step and prints the same report as a fuzzing finding, the target, the kind of finding and the sanitizers' report, never the input. A run whose fuzzing or merge found something saves nothing. A pull request's run reads the corpus too, so a change that breaks a harness's promise fails on its own pull request and not later on `main`. A pull request, or a dispatch on any other branch, never minimizes or saves it, so only `main` writes the corpus. A target whose minimized corpus passes 25 MB fails the step, and the corpora are not saved, so one target cannot crowd the others out of the cache. The `prune-caches` job then deletes the entry the new one replaced. The organization's other libraries run one fuzz job per target, each with its own cache. This repository has eleven targets in one job, four at a time, because eleven more containers per run would cost more runner time than the targets' corpora need, and the corpora are still kept and minimized separately.
 
 A crash a fuzzer finds becomes a test in the suite of its module (see [development.md](development.md#reproducing-a-fuzz-finding)).
 

docs/user/segmentation.md

@@ -1,6 +1,6 @@
 # Characters, words, sentences and lines
 
-`segment.h` finds the boundaries of grapheme clusters, words and sentences (UAX #29) and the line break opportunities (UAX #14). All four use one iterator type, `muniSegmentIterator`, a plain struct the caller keeps on the stack that needs no cleanup.
+`segment.h` finds the boundaries of grapheme clusters, words and sentences (UAX #29) and the line break opportunities (UAX #14). All four use one iterator type, `muniSegmentIterator`, a plain struct the caller keeps on the stack that needs no cleanup. A copy of an initialized iterator is one too, and every call refuses an iterator no init call set up, such as a zeroed one, with `muni_errorInvalid`.
 
 ## Grapheme clusters, words and sentences
 

docs/user/shaping.md

@@ -18,7 +18,7 @@ if (muniInitScriptIterator(&scripts, text, length, false) == muni_success)
 }
 ```
 
-Like the segment iterators, it takes text in pieces through `muniScriptIterator_Feed` after `muni_needMoreText`. `muniFindScriptRuns` writes the runs of a whole text into an array.
+Like the segment iterators, it takes text in pieces through `muniScriptIterator_Feed` after `muni_needMoreText`, and its calls refuse an iterator no init call set up with `muni_errorInvalid`. `muniFindScriptRuns` writes the runs of a whole text into an array.
 
 A text layout pipeline itemizes by paragraph ([bidi](bidi.md)), then by script and bidi run, then by font, and shapes each item.
 

include/maul-unicode/script.h

@@ -28,7 +28,10 @@ extern "C"
 #endif
 
     // An iterator over script runs. Its contents are private; it is plain
-    // data the caller keeps anywhere and needs no cleanup.
+    // data the caller keeps anywhere and needs no cleanup. A copy of an
+    // iterator an init call set up is one too; every call refuses one no
+    // init call set up, such as a zeroed one or one copied from other
+    // bytes, as an invalid argument.
     typedef struct muniScriptIterator
     {
         uint64_t opaque[26];
@@ -65,8 +68,9 @@ extern "C"
     /// @param moreFollows  true when still more pieces will be fed.
     /// @return `muni_success`; `muni_errorState` for an iterator that did
     ///         not ask for more text, such as one initialized without more
-    ///         text to follow; `muni_errorInvalid` for a NULL argument.
-    ///         `muni_errorInvalid` is reported before `muni_errorState`.
+    ///         text to follow; `muni_errorInvalid` for a NULL argument or
+    ///         an iterator no init call set up. `muni_errorInvalid` is
+    ///         reported before `muni_errorState`.
     /// @par Thread safety
     /// Safe from any thread; an iterator is used by one thread at a time.
     MUNI_NODISCARD MUNI_API muniResult muniScriptIterator_Feed(muniScriptIterator* iterator,
@@ -83,7 +87,8 @@ extern "C"
     ///                  returns `muni_success`.
     /// @return `muni_success` with a run; `muni_done` after the last one;
     ///         `muni_needMoreText` when the run's end or script depends on
-    ///         text not yet fed; `muni_errorInvalid` for a NULL argument.
+    ///         text not yet fed; `muni_errorInvalid` for a NULL argument or
+    ///         an iterator no init call set up.
     /// @par Thread safety
     /// Safe from any thread; an iterator is used by one thread at a time.
     MUNI_NODISCARD MUNI_API muniResult muniScriptIterator_NextRun(muniScriptIterator* iterator,

include/maul-unicode/segment.h

@@ -36,7 +36,9 @@ extern "C"
     // An iterator over grapheme cluster, word or sentence boundaries or
     // line break opportunities. Its contents are private; it is plain data
     // the caller keeps anywhere, usually on the stack, and needs no
-    // cleanup.
+    // cleanup. A copy of an iterator an init call set up is one too; every
+    // call refuses one no init call set up, such as a zeroed one or one
+    // copied from other bytes, as an invalid argument.
     typedef struct muniSegmentIterator
     {
         uint64_t opaque[16];
@@ -192,7 +194,8 @@ extern "C"
     /// @return `muni_success`; `muni_errorState` for an iterator already
     ///         asked for a break; `muni_errorBusy` when called from inside
     ///         the iterator's word segmenter; `muni_errorInvalid` for a NULL
-    ///         iterator or another kind of iterator. `muni_errorInvalid` is
+    ///         iterator, one no init call set up or another kind of
+    ///         iterator. `muni_errorInvalid` is
     ///         reported before `muni_errorBusy`, and that before
     ///         `muni_errorState`.
     /// @par Thread safety
@@ -236,7 +239,8 @@ extern "C"
     ///                   assigns.
     /// @return `muni_success`; `muni_errorState` for an iterator already
     ///         asked for a boundary; `muni_errorInvalid` for a NULL
-    ///         iterator, NULL overrides with a nonzero count, another kind
+    ///         iterator or one no init call set up, NULL overrides with a
+    ///         nonzero count, another kind
     ///         of iterator, a count over 0x10F800, a code point past
     ///         U+10FFFF or a surrogate, code points out of order or
     ///         repeated, or a value that is not a class of the iterator's
@@ -257,7 +261,8 @@ extern "C"
     /// @return `muni_success`; `muni_errorState` for an iterator already
     ///         asked for a break; `muni_errorBusy` when called from inside
     ///         the iterator's word segmenter; `muni_errorInvalid` for a NULL
-    ///         iterator, another kind of iterator or an unknown strictness.
+    ///         iterator, one no init call set up, another kind of iterator
+    ///         or an unknown strictness.
     ///         `muni_errorInvalid` is reported before `muni_errorBusy`, and
     ///         that before `muni_errorState`.
     /// @par Thread safety
@@ -270,7 +275,8 @@ extern "C"
     /// @param iterator  The iterator.
     /// @return The strictness muniSegmentIterator_SetLineBreakStrictness
     ///         set, muni_lineBreakNormal when none was set, and for a NULL
-    ///         iterator or another kind of iterator.
+    ///         iterator, one no init call set up or another kind of
+    ///         iterator.
     /// @par Thread safety
     /// Safe from any thread; an iterator is used by one thread at a time.
     MUNI_API muniLineBreakStrictness
@@ -284,7 +290,8 @@ extern "C"
     /// @return `muni_success`; `muni_errorState` for an iterator already
     ///         asked for a break; `muni_errorBusy` when called from inside
     ///         the iterator's word segmenter; `muni_errorInvalid` for a NULL
-    ///         iterator, another kind of iterator or an unknown writing
+    ///         iterator, one no init call set up, another kind of iterator
+    ///         or an unknown writing
     ///         system. `muni_errorInvalid` is reported before
     ///         `muni_errorBusy`, and that before `muni_errorState`.
     /// @par Thread safety
@@ -296,8 +303,8 @@ extern "C"
     ///
     /// @param iterator  The iterator.
     /// @return The writing system muniSegmentIterator_SetWritingSystem set,
-    ///         muni_writingOther when none was set, and for a NULL iterator
-    ///         or another kind of iterator.
+    ///         muni_writingOther when none was set, and for a NULL iterator,
+    ///         one no init call set up or another kind of iterator.
     /// @par Thread safety
     /// Safe from any thread; an iterator is used by one thread at a time.
     MUNI_API muniWritingSystem
@@ -314,9 +321,9 @@ extern "C"
     ///         not ask for more text, such as one initialized without more
     ///         text to follow or one fed before it asked; `muni_errorBusy`
     ///         when called from inside the iterator's word segmenter;
-    ///         `muni_errorInvalid` for a NULL argument. `muni_errorInvalid`
-    ///         is reported before `muni_errorBusy`, and that before
-    ///         `muni_errorState`.
+    ///         `muni_errorInvalid` for a NULL argument or an iterator no
+    ///         init call set up. `muni_errorInvalid` is reported before
+    ///         `muni_errorBusy`, and that before `muni_errorState`.
     /// @par Thread safety
     /// Safe from any thread; an iterator is used by one thread at a time.
     MUNI_NODISCARD MUNI_API muniResult muniSegmentIterator_Feed(muniSegmentIterator* iterator,
@@ -333,7 +340,8 @@ extern "C"
     ///         one; `muni_needMoreText` when the next boundary depends on
     ///         text not yet fed; `muni_errorBusy` when called from inside
     ///         the iterator's word segmenter, and then the iterator is left
-    ///         as it was; `muni_errorInvalid` for a NULL argument.
+    ///         as it was; `muni_errorInvalid` for a NULL argument or an
+    ///         iterator no init call set up.
     /// @par Thread safety
     /// Safe from any thread; an iterator is used by one thread at a time.
     MUNI_NODISCARD MUNI_API muniResult muniSegmentIterator_NextBreak(muniSegmentIterator* iterator,
@@ -390,7 +398,8 @@ extern "C"
     ///                  `muni_success`.
     /// @return `muni_success`; `muni_errorBusy` when called from inside the
     ///         iterator's word segmenter; `muni_errorInvalid` for a NULL
-    ///         argument or an iterator of another kind.
+    ///         argument, an iterator no init call set up or one of another
+    ///         kind.
     /// @par Thread safety
     /// Safe from any thread; an iterator is used by one thread at a time.
     MUNI_NODISCARD MUNI_API muniResult