From 8d738178291894eabf4df3d6a4d6a25ac94f9894 Mon Sep 17 00:00:00 2001 From: Sirac Ozmen Date: Thu, 5 Nov 2026 14:50:54 +0300 Subject: [PATCH] fix(maul): keep a harness's expectation lines in the fuzz report --- maul/files/docs/internals/maul-conventions.md | 2 +- maul/files/tools/fuzz_target.sh | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/maul/files/docs/internals/maul-conventions.md b/maul/files/docs/internals/maul-conventions.md index 2a5c6c6..0fdb4e1 100644 --- a/maul/files/docs/internals/maul-conventions.md +++ b/maul/files/docs/internals/maul-conventions.md @@ -119,7 +119,7 @@ The libraries are one product family: a program may link several of them, and a - Tests use the family's harness header and register with CTest through `maul_add_test`. A suite that reaches internal headers is marked `WHITEBOX` and is skipped in shared builds, where internal symbols are hidden. A quarantined suite is marked `QUARANTINE`, and a test registered another way takes the label through `maul_quarantine`. - CI runs the tests under AddressSanitizer, UndefinedBehaviorSanitizer and ThreadSanitizer, with warnings as errors. - Every parser of external bytes treats its input as hostile and has a fuzz target. -- A fuzz run publishes no input. CI runs each target through `tools/fuzz_target.sh`, which keeps a finding, its seed and the target's log on the runner and prints only the finding's kind and sanitizer report. No workflow uploads a finding, a fuzz log or a corpus as an artifact; a corpus is kept in the Actions cache. Every run restores it, the library's own pull_request runs included, so a change that breaks a harness's promise fails on its own pull request and not later on main; only a clean run on main saves it, and a run that found something does not, since its corpus holds inputs near the finding. A corpus is merged or minimized through `tools/fuzz_target.sh --merge`, which prints no log and fails when the merge met a finding, since libFuzzer's merge reruns every input and exits 0 on a crash; that corpus is not saved either. AddressSanitizer's register dump is off, since registers may hold the input. The cache is not secret: a fork's pull request can run its own workflow and restore it, so it holds nothing the committed seeds would not lead to. A harness's own failure message names the broken promise, never the input's bytes. +- A fuzz run publishes no input. CI runs each target through `tools/fuzz_target.sh`, which keeps a finding, its seed and the target's log on the runner and prints only the finding's kind and sanitizer report. No workflow uploads a finding, a fuzz log or a corpus as an artifact; a corpus is kept in the Actions cache. Every run restores it, the library's own pull_request runs included, so a change that breaks a harness's promise fails on its own pull request and not later on main; only a clean run on main saves it, and a run that found something does not, since its corpus holds inputs near the finding. A corpus is merged or minimized through `tools/fuzz_target.sh --merge`, which prints no log and fails when the merge met a finding, since libFuzzer's merge reruns every input and exits 0 on a crash; that corpus is not saved either. AddressSanitizer's register dump is off, since registers may hold the input. The cache is not secret: a fork's pull request can run its own workflow and restore it, so it holds nothing the committed seeds would not lead to. A harness's own failure message names the broken promise and the values that broke it, never the input's bytes, as one `FILE:LINE: expectation failed: ...` line, the form the report keeps. - A failing test is never rerun to pass. When a test fails without a fault in the change under test, it is quarantined: an issue records the failure and its run, the test (or the one check that failed, split into a test of its own) takes the `quarantine` label, the required runs exclude that label, and a nightly run executes every quarantined test through `tools/run_quarantine.sh`, which runs by build directory: the job passes the preset's options it needs, and each test gets 300 seconds unless it says otherwise. A quarantined test's failure is a known flake: it is reported as a warning and in the job summary, and it does not fail the nightly, which would teach everyone to ignore red nights. A skip or a test that did not run still fails it, as do a sanitizer report, a crash on a bad address and any failure outside the label: a flake is timing or a daemon, never a memory error. A test leaves quarantine when its issue's fix lands and the nightly has passed it 14 nights in a row. - Every C snippet of `docs/user/` is built and run by a test as written, so documentation cannot drift from the API. A fragment no test can build everywhere is marked `` right above its fence. diff --git a/maul/files/tools/fuzz_target.sh b/maul/files/tools/fuzz_target.sh index 08f00fa..4f19338 100755 --- a/maul/files/tools/fuzz_target.sh +++ b/maul/files/tools/fuzz_target.sh @@ -14,7 +14,9 @@ # machine that ran it. Every run's log is public in CI, and libFuzzer # prints a finding's input there as bytes, text and Base64, so on a # finding this reports only the target, the kind of each finding (crash, -# leak, timeout, oom), the sanitizers' runtime error lines and the +# leak, timeout, oom), the sanitizers' runtime error lines, the +# harness's own "FILE:LINE: expectation failed: ..." lines (the promise +# it broke and the values that broke it, never the input's bytes) and the # report from its "==PID== ERROR:" line to its "SUMMARY:" line, all # taken from the log above libFuzzer's first input dump. # @@ -105,6 +107,7 @@ head=$(sed -E '/^(MS: [0-9]|0x[0-9a-fA-F]{1,2},|artifact_prefix=|Base64: )/,$d' find "$crashes" -maxdepth 1 -type f -newer "$stamp" -printf '%f\n' | sed 's/-.*//' | sort | uniq -c grep -E -m 5 '^[^ ]+:[0-9]+:[0-9]+: runtime error: ' <<<"$head" || true + grep -E -m 5 '^[^ ]+:[0-9]+: expectation failed: ' <<<"$head" || true # head may close the pipe early on a long report; that is not an error. sed -n '/^==[0-9]*== *ERROR: /,/^SUMMARY: /p' <<<"$head" | head -n 80 || true } >"$report"