Repository navigation
docs(maul): say how state a caller holds is checked - #119
Conversation
maul-locale#334 asked how far a library checks state a caller holds: an opaque stamped formatter, a value struct with documented fields, and caller memory kept as checked. The conventions now give one rule for each: opaque state is checked by stamp and bounds only; a public value struct names what it came from instead of carrying a raw pointer and size; kept caller memory is read only within the extents recorded.
siracozmen01
left a comment
There was a problem hiding this comment.
Review at b90128f: changes needed (2). Both are wording; the three-part rule itself is sound.
1. The value-struct clause forbids outputs it should not. As written, "A value struct with documented fields, which a caller may copy and assign, carries no raw pointer and size into the library's data" covers every value struct, including ones the library only writes out and never reads back. maul-locale#334's own survey lists such outputs as fine: "Outputs pointing into validated data, which the library never reads back: mlocIndexBucket.label, mlocMessagePartInfo.name and mlocMessageParameter's name and value". The danger the clause guards against exists only when a call takes the struct back and reads through its fields, as with mlocTimeZone.zones. Each session audits against this rule, so the literal text would make every such getter's view into library data a bug to file. Please scope it, for example: "A value struct with documented fields that a call takes back from the caller carries no raw pointer and size into the library's data: ..." Optionally add: "a struct the library only writes out may point into its data, valid for as long as its documentation says."
2. "Stamp" already means something else in this file. The ABI rule (line 86) uses "stamp" for PP_ABI_VERSION in a root def: "each one carries the stamp", and "A library without a root def has no stamp". The new rule checks opaque state "by its stamp", so a library with no root def, or opaque state that is not a def, reads as having nothing to check by. Please name what is checked, in the file's existing terms: its cookie, and its ABI version where the state records one (a state format records its writer's PP_ABI_VERSION, line 142). Or define "stamp" here as the opaque state's own check word.
No contradiction with the root and id rules. The id rule covers objects a root owns; the new rule covers state the caller holds outside them. "Names the object or data it came from" is satisfied by a typed id for a root-owned object, which is still refused when stale, or by an opaque, checked object such as mlocData. The rule could say so ("by a typed id or by an opaque object checked as above") to close the loop, but this is optional. The opaque clause is also consistent with itself: bounds make every bit pattern memory-safe within the extents the state records, and a forged one gets no semantic checks, which matches "a forged one is outside the contract".
Structs that would break it (not blocking). Known: maul-locale's mlocTimeZone (zones, zonesSize), and through it mlocMessageOperand.zone and mlocMessageOptions.timeZone, per #334. I could not search the other libraries' headers from this session, so the rest is not verified. The test for each session's audit: a public struct with documented fields that a call takes back and reads through, holding a pointer into memory the library allocated or validated. Defs that point at the caller's own memory (shader bytes, mesh positions, text spans) are not affected.
Read, not run.
siracozmen01
left a comment
There was a problem hiding this comment.
Re-review at eaf35b1: ready. Both points are resolved in the rule.
- The value-struct clause now applies to "a value struct with documented fields that a call takes back from the caller", and the new sentence "A struct the library only fills in for the caller, such as a label it points at, may point into its data" keeps the output views maul-locale#334 lists as fine (
mlocIndexBucket.label,mlocMessagePartInfo.name,mlocMessageParameter). - Opaque state is now checked "by its cookie, and its ABI version where it records one", which uses the file's own terms and no longer collides with the root def's stamp (line 86) or reads as nothing to check in a library without a root def.
One leftover, not blocking: the body's Opaque state bullet still says "checked by its stamp". The body becomes the squash commit message, so you may want it to say "its cookie, and its ABI version where it records one" too. It does not affect the file.
Read, not run.
## Problem `.github` v0.0.48 adds the family's `tools/tidy_filter.sh`, which prints clang-tidy's header filter for a build tree, and v0.0.50 and v0.0.51 add a conventions rule on how state a caller holds is checked (Rawframe-Project/.github#119 and Rawframe-Project/.github#120). This repository lacks it, carries an older `docs/internals/maul-conventions.md`, pins the reusable workflows at v0.0.42 and v0.0.45, and builds the filter itself in two places. ## Change - The family files are synced to v0.0.51 with `family_sync.py --write`: `tools/tidy_filter.sh` is added, and `docs/internals/maul-conventions.md` gains the rule that every clang-tidy call takes its header filter from it and the rule on state a caller holds. `AGENTS.md` lists `tidy_filter.sh` among the shared files. - Every reusable workflow and the docs-check action are pinned at `972ea41c6abf2652d3b067e57bf0713f1661cf0e # v0.0.51`. Between the old pins and v0.0.51 only `pr-title.yml` changed among the workflows, and its old pin, v0.0.45, already had that change. - ci.yml's clang-tidy step and `tools/check.sh`'s `tidy()` set `filter=$(tools/tidy_filter.sh <tree>)` on a line of its own, so a tree without the cache entry stops the step instead of tidying with an empty filter, and pass `--header-filter="$filter"`. `tools/check.sh`'s own `header_filter()` is gone. A changed header still tidies every source. ## Verification - `family_sync.py --check` from v0.0.51 reports 0 differences. - In a fresh clone of this branch, with only a header changed (`return value == value;` planted in `src/bits.h`), `tools/check.sh changed` fails on `misc-redundant-expression` in `src/bits.h`, and the ci.yml step's commands report it on the Release and Debug presets. `tools/tidy_filter.sh` on a tree whose cache lacks the entry exits 1. - `tools/check.sh changed` passes on the clean branch, and `shellcheck -S warning` is clean on `tools/check.sh` and `tools/tidy_filter.sh`. Closes #261
## Problem .github v0.0.50 and v0.0.51 add one rule to the family conventions, on state a caller holds (Rawframe-Project/.github#119 and #120). This library still carried v0.0.48's `maul-conventions.md`, so the nightly family check reported it as differing. ## Change - The family files are synced to v0.0.51 (`family_sync.py --write`). Only `docs/internals/maul-conventions.md` changes, by the rule's one paragraph. - Every reusable workflow pin moves to v0.0.51 (972ea41). No script or workflow changed since v0.0.48. The library follows the rule. The audit is #297: the only structs a call takes back that point into the library's data are clip tracks and timings, which the values layer reads as caller-built input. #299 documents them that way. ## Change of results None. ## Verification - `family_sync.py --check` at v0.0.51 reports 0 differences. - `tools/check.sh changed` passes; no test is reached by the changes. - The organization's docs check passes. Closes #300
## Problem maul2d carries the v0.0.47 family files, pins its workflows to v0.0.47, and builds clang-tidy's anchored header filter inline in `tools/check.sh` and `ci.yml`. `.github` v0.0.51 (`972ea41`) has a family `tools/tidy_filter.sh` that does the same, a third `NOLINTNEXTLINE` in `src/cpu.h`, and two conventions rules (#310). ## Change - `family_sync.py --write` at v0.0.51: `tools/tidy_filter.sh` (new, executable), `src/cpu.h` (its comment and a `NOLINTNEXTLINE(readability-redundant-declaration)` on the third declaration, Rawframe-Project/.github#118) and `docs/internals/maul-conventions.md` (the tidy row names `tools/tidy_filter.sh`, plus Rawframe-Project/.github#119 and #120). - `tools/check.sh` and both clang-tidy steps of `ci.yml`: the filter comes from `filter=$(tools/tidy_filter.sh <build tree>)` on its own line, so a failure there stops the step instead of passing an empty filter. The inline `CMAKE_HOME_DIRECTORY` read and escape are gone. - `.github/workflows/ci.yml`, `pr-gate.yml` and `pr-title.yml`: the eight `Rawframe-Project/.github` references are pinned to `972ea41c6abf2652d3b067e57bf0713f1661cf0e # v0.0.51`. Between v0.0.47 and v0.0.51 no file under `.github/workflows/` or `.github/actions/` of `.github` changed, so no shared workflow behaves differently. `AGENTS.md`'s list of shared family files does not gain `tools/tidy_filter.sh` here; no check reads that list. ## Verification - `family_sync.py --check` reports 0 differences, and the synced `tools/tidy_filter.sh` equals the one at v0.0.51. - A fresh clone of this branch, from its root, with `return a == a;` planted in a function in `src/core.h` and nothing else changed: `tools/check.sh changed` runs clang-tidy on all 68 sources and fails on `misc-redundant-expression` at `src/core.h`. Without the plant, it passes, and the `linux-clang-20-asan` preset runs 25 of 25 tests. - `tools/check.sh` parses with `bash -n`, every workflow file parses as YAML, and `tools/check_source.py` passes. Closes #310
## Problem The family conventions now say that opaque state only the library writes is checked by its cookie and by every index, slot and length in it being in bounds, so that no bit pattern makes a call read outside the state (Rawframe-Project/.github#119). `muniScriptIterator` and `muniSegmentIterator` carried no cookie, and their calls trusted the counts and flags in caller memory. A script iterator of 0x41 bytes made `muniScriptIterator_NextRun` copy 65 bytes into a 4-byte window on the stack, and 0x02 or 0x04 bytes made both iterators load a `bool` that was neither 0 nor 1. ## Change - Each init call writes a 16-bit cookie with its top bit set into the iterator's cursor, in two bytes that were its padding, and every call that takes an iterator back checks it, then checks every count against the storage it counts (the cursor's pending bytes and position, the bracket count, the override count), every class, state, hold kind and script index against the values the rules write, and every flag's byte as a 0 or 1 before the copy is read as a `bool`. It also checks the two cross-field facts the assertions downstream rely on: a waiting cursor holds less than a code point, and a hold lies before the cursor. The cookie only turns away an iterator no init call set up; the checks after it keep any other bit pattern from reaching outside the state too. They run once per call. - An iterator that fails them is refused with `muni_errorInvalid`, the result each header already gives for a bad argument, and left as it was; `muniSegmentIterator_GetLineBreakStrictness` and `muniSegmentIterator_GetWritingSystem` give their defaults for it, as for NULL. Each header now names it, and says that a copy of an initialized iterator is one too. The pointers an iterator keeps (the text, the overrides, the caller's breaker) cannot be checked: an iterator forged there with the cookie kept stays outside the contract, as `docs/internals/overview.md` now says. - Neither public iterator changes size: the cookie fills the cursor's padding on 32-bit and 64-bit targets alike, and a new `static_assert` beside `muniSegmenter`, like the script state's, keeps the iterator from growing a whole `uint64_t` more than the state needs. A first version put a 32-bit cookie in front of each state, which grew both iterators by 8 bytes; the segmenter's copies in and out of caller memory then passed 128 bytes and became `memcpy` calls, which cost a grapheme break about 8 ns. - Since a call that was undefined behavior now refuses with a result the headers document, this is a `feat`, so the next release is 1.1.0. The ABI is unchanged. - The segment check is inlined into each call and checks only the fields of the iterator's rule set; line's anywhere setting checks only the grapheme state it keeps. The flags' bytes are read from the caller's iterator rather than from the copy just made of it, so that no load waits on the copy's stores. The largest value each rule set writes in a state is named in `src/segmenter.h` and tied to the rule set's own enumeration by a `static_assert`. - The script state moves to `src/script_state.h` so that the new fuzz target can reach its fields. - `docs/internals/segmentation.md` describes the checks and where the cookie lives, and no longer asks for `!` when the iterator grows, which 1.x forbids. Who sees it: a caller that passed an iterator no init call set up, which was undefined behavior, now gets `muni_errorInvalid`. Every result for an initialized iterator is the same; no result of the results version changes. ## Verification - `test_segment` and `test_script` fill an iterator with 0x00 (zeroed, as a static one is), 0x41, 0x02 and 0x04 and pass it to every call that takes one back: each refuses it and changes nothing in it. On main the 0x41 script fill is a stack buffer overflow under AddressSanitizer and the 0x02 and 0x04 fills stop UBSan on a `bool` load. - `fuzz_iterators`, a new fuzz target, sets up each kind of iterator over a fixed text, answers a few calls, then sets bytes of its state from the input, everywhere but the pointers and the lengths and counts that go with them, and makes every call again: none may read outside the state or the text or trip an assertion. 300 seconds of it ran 7,227,223 inputs with no finding. - `tools/check.sh changed` and `tools/check.sh pr` pass: the generated tables, the HarfBuzz and AddressSanitizer presets with every suite, clang-tidy on every source in Release and Debug, and ten seconds of each fuzz target, the new one included. - Speed, Release builds with clang 20, best of 15 passes over a text of 1 Mi code points, the two builds run eight times each in turn and the best of each kept. Every call here reports one run or boundary, the most calls a text can need, so the difference is the check's whole cost per call: | Call, each reporting one run or boundary | main | this change | difference | |---|---|---|---| | `muniScriptIterator_NextRun`, alternating Latin and Greek | 31.1 ns | 32.4 ns | +1.3 ns (4%) | | `muniSegmentIterator_NextBreak`, grapheme clusters of ASCII | 17.0 ns | 18.9 ns | +1.9 ns (11%) | | `muniSegmentIterator_NextBreak`, lines of "a " repeated | 25.0 ns | 29.9 ns | +4.9 ns (20%) | The medians of the eight runs differ by the same: +0.6, +2.0 and +5.4 ns. `perf stat` over the whole bench counts the same branch misses (302 thousand) and L1 misses (1.5 million) for both builds and 29% more instructions, so the cost is the instructions of the check, which a line call runs more of: the cursor's checks and the line rules' classes, flags, number state and hold kind. The first version of this change, which called a validator per rule set out of line, cost +2.7 ns (16%) and +6.9 ns (28%) on the same two calls. On ordinary prose the cost is small: line `NextBreak` over the English preamble of the Universal Declaration of Human Rights, repeated to 1 MiB (168,979 breaks, one per 6.2 bytes), takes 107.2 ns per break on main and 108.2 ns with this change at best, +1.0 ns (1%), and 109.6 against 112.3 ns at the median, +2.6 ns (2%), over the same eight alternating runs. Closes #263
## Problem The family files from `.github` v0.0.51 (972ea41) had not reached maul-vfx, which is on v0.0.48 since #285. v0.0.49 and v0.0.50 are skipped because v0.0.51 holds their changes. The only family file that changes is `docs/internals/maul-conventions.md`, which gains the rule on how state a caller holds is checked (Rawframe-Project/.github#119 and Rawframe-Project/.github#120). ## Change - Ran `family_sync.py --write maul-vfx` from a clone of `.github` checked out at the v0.0.51 tag, not the live checkout. It writes `docs/internals/maul-conventions.md` only. - Pinned the reusable workflows and actions to 972ea41 (v0.0.51) instead of 7eeaaa4 (v0.0.48), in `ci.yml`, `pr-gate.yml` and `pr-title.yml`. The workflows and actions are the same at both tags. No code changes. The library already follows the new rule; the audit was done against the public headers: | The rule says | maul-vfx | | --- | --- | | A value struct a call takes back carries no raw pointer and size into the library's data | `mvfxEffectRefusal` (filled by the create, read and write calls, taken by `mvfxGetEffectFieldPath`) holds no pointer: inline tags and items with a `tagCount` checked against below 0 and past `MVFX_MAX_FIELD_DEPTH`. The ids are an index and a generation, checked on every call. | | A struct the library only fills in may point into its data | The views (`mvfxEmitterView`, `mvfxMeshView`, `mvfxGridView`, `mvfxLayoutView` and the rest), `mvfxEmitterPlan` and `mvfxParticleArrays` point into the library's data and are never taken back from the caller. | | A struct also documented as caller-built input may point where its builder chose, read as untrusted | `mvfxEffectDef`, which `mvfxReadEffect` fills into the caller's arena, is checked field by field by every call that takes it. The other defs, `mvfxSpawnRequest` and the output buffers point into caller memory. | | Caller memory a call keeps "as checked" is read only within the extents it recorded | `mvfxInstance_SetDepthImage` borrows the pixels. It checks the image once and keeps its own prepared copy of the extents, so later steps read within them. | ## Verification - `family_sync.py --check maul-vfx` from the same clone: 0 differences. - `git grep` finds no `.github` pin other than 972ea41 `# v0.0.51`. - `tools/check.sh changed` passes. Only documentation and workflows changed, so there is no build. - The organization's documentation check (`scripts/docs_check.py --strict`) from the v0.0.51 clone: ok.
Problem
maul-locale#334 (and its comments on #376 and the bound messages) asks how far a library checks state that a caller holds. The conventions say how a stale id is refused, but nothing for three other kinds of state:
The issue has waited for that ruling since 2026-10-31.
Change
maul/files/docs/internals/maul-conventions.md: one rule beside the root-object rule, covering each kind.Verification