Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion maul/files/docs/internals/maul-conventions.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ The libraries are one product family: a program may link several of them, and a
- Tests use the family's harness header and register with CTest through `maul_add_test`. A suite that reaches internal headers is marked `WHITEBOX` and is skipped in shared builds, where internal symbols are hidden. A quarantined suite is marked `QUARANTINE`, and a test registered another way takes the label through `maul_quarantine`.
- CI runs the tests under AddressSanitizer, UndefinedBehaviorSanitizer and ThreadSanitizer, with warnings as errors.
- Every parser of external bytes treats its input as hostile and has a fuzz target.
- A fuzz run publishes no input. CI runs each target through `tools/fuzz_target.sh`, which keeps a finding, its seed and the target's log on the runner and prints only the finding's kind and sanitizer report. No workflow uploads a finding, a fuzz log or a corpus as an artifact; a corpus is kept in the Actions cache. Every run restores it, the library's own pull_request runs included, so a change that breaks a harness's promise fails on its own pull request and not later on main; only a clean run on main saves it, and a run that found something does not, since its corpus holds inputs near the finding. A corpus is merged or minimized through `tools/fuzz_target.sh --merge`, which prints no log and fails when the merge met a finding, since libFuzzer's merge reruns every input and exits 0 on a crash; that corpus is not saved either. AddressSanitizer's register dump is off, since registers may hold the input. The cache is not secret: a fork's pull request can run its own workflow and restore it, so it holds nothing the committed seeds would not lead to. A harness's own failure message names the broken promise, never the input's bytes.
- A fuzz run publishes no input. CI runs each target through `tools/fuzz_target.sh`, which keeps a finding, its seed and the target's log on the runner and prints only the finding's kind and sanitizer report. No workflow uploads a finding, a fuzz log or a corpus as an artifact; a corpus is kept in the Actions cache. Every run restores it, the library's own pull_request runs included, so a change that breaks a harness's promise fails on its own pull request and not later on main; only a clean run on main saves it, and a run that found something does not, since its corpus holds inputs near the finding. A corpus is merged or minimized through `tools/fuzz_target.sh --merge`, which prints no log and fails when the merge met a finding, since libFuzzer's merge reruns every input and exits 0 on a crash; that corpus is not saved either. AddressSanitizer's register dump is off, since registers may hold the input. The cache is not secret: a fork's pull request can run its own workflow and restore it, so it holds nothing the committed seeds would not lead to. A harness's own failure message names the broken promise and the values that broke it, never the input's bytes, as one `FILE:LINE: expectation failed: ...` line, the form the report keeps.
- A failing test is never rerun to pass. When a test fails without a fault in the change under test, it is quarantined: an issue records the failure and its run, the test (or the one check that failed, split into a test of its own) takes the `quarantine` label, the required runs exclude that label, and a nightly run executes every quarantined test through `tools/run_quarantine.sh`, which runs by build directory: the job passes the preset's options it needs, and each test gets 300 seconds unless it says otherwise. A quarantined test's failure is a known flake: it is reported as a warning and in the job summary, and it does not fail the nightly, which would teach everyone to ignore red nights. A skip or a test that did not run still fails it, as do a sanitizer report, a crash on a bad address and any failure outside the label: a flake is timing or a daemon, never a memory error. A test leaves quarantine when its issue's fix lands and the nightly has passed it 14 nights in a row.
- Every C snippet of `docs/user/` is built and run by a test as written, so documentation cannot drift from the API. A fragment no test can build everywhere is marked `<!-- guide: not run: REASON -->` right above its fence.

Expand Down
5 changes: 4 additions & 1 deletion maul/files/tools/fuzz_target.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@
# machine that ran it. Every run's log is public in CI, and libFuzzer
# prints a finding's input there as bytes, text and Base64, so on a
# finding this reports only the target, the kind of each finding (crash,
# leak, timeout, oom), the sanitizers' runtime error lines and the
# leak, timeout, oom), the sanitizers' runtime error lines, the
# harness's own "FILE:LINE: expectation failed: ..." lines (the promise
# it broke and the values that broke it, never the input's bytes) and the
# report from its "==PID== ERROR:" line to its "SUMMARY:" line, all
# taken from the log above libFuzzer's first input dump.
#
Expand Down Expand Up @@ -105,6 +107,7 @@ head=$(sed -E '/^(MS: [0-9]|0x[0-9a-fA-F]{1,2},|artifact_prefix=|Base64: )/,$d'
find "$crashes" -maxdepth 1 -type f -newer "$stamp" -printf '%f\n' |
sed 's/-.*//' | sort | uniq -c
grep -E -m 5 '^[^ ]+:[0-9]+:[0-9]+: runtime error: ' <<<"$head" || true
grep -E -m 5 '^[^ ]+:[0-9]+: expectation failed: ' <<<"$head" || true
# head may close the pipe early on a long report; that is not an error.
sed -n '/^==[0-9]*== *ERROR: /,/^SUMMARY: /p' <<<"$head" | head -n 80 || true
} >"$report"
Expand Down
Loading