Repository navigation
ci(fuzz): tolerate only Go's bare -fuzztime timeout, not real findings - #706
Conversation
The fuzz jobs treated any failing run as a coordinator boundary timeout unless some line looked like an indented `_test.go:N:` reference. A panic reports through `testing.go:N:` and its stack frames end in `+0x..`, not a colon, so real crashes (and build errors) passed CI. Reproduced locally: a fuzz target that writes to a nil map fails with "Failing input written to testdata/fuzz/..." and the old check exits 0. Port the stricter check from datadog-agent's tasks/fuzz.py (DataDog/datadog-agent#58502): a failing run passes only when it printed "context deadline exceeded" and Go wrote no failing input. Both fuzz jobs now share scripts/fuzz-run.sh instead of two copies of the inline function. Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
This comment will be updated automatically if new data arrives. 🔗 Commit SHA: 5075c3e | Docs | View more details | Give us feedback! |
julesmcrt
left a comment
There was a problem hiding this comment.
Reviewed 5075c3eb8d4acc341e9f4270549e96ef15734a9c.
This is a real improvement to the fuzz gate's signal. The old inline fuzz_run tolerated any non-zero go test exit whose log lacked a *_test.go:NN: line — which meant a panic whose stack frames were all in production code, a worker hang, a coordinator internal failure, or an overall test-timeout panic were all silently relabeled "coordinator boundary timeout" and the job went green. The new script narrows tolerance to exactly the two-property signature of Go's bare -fuzztime cancellation: a non-zero exit whose log contains "context deadline exceeded" and does not contain "Failing input written to". Both properties check out against Go's source: every real crash/hang finding is funneled through the crashError path and always prints "Failing input written to" (even a hang whose termination suffix is itself a deadline error), while the bare boundary cancellation is a plain context error with no corpus write. I fed thirteen synthetic terminal-class logs through the new script with a mocked go binary: eight classes the old logic masked (production-frame crashes, hangs, internal failures, test-timeout panics, deadline-plus-failing-input mixes, near-empty kills) now fail the job with go's original exit code preserved, and only the genuine bare-timeout class is tolerated.
One residual boundary gap worth tightening before merge: grep -q 'context deadline exceeded' matches the string anywhere in the log. A real assertion failure on a seed-corpus entry prints foo_test.go:42: context deadline exceeded and — because seed inputs are already on disk — Go prints no "Failing input written to" for it, so both guards are satisfied and that finding would exit 0. The old inline logic actually failed this class, so it's a narrow regression. None of the current fuzz targets can produce that text in a failure message (the t.Fatal(err) paths only fire on file-write errors, the ip targets word timeouts as "timed out (possible hang)", and the others return early on t.Context().Err()), and deterministic seed failures are usually caught by the preceding seed-run step — so it isn't reachable today — but it's exactly the shape of bug this gate exists to catch. One-line fix, verified against the synthetic logs: anchor the pattern to the bare-message line shape, which matches the coordinator cancellation (message alone on its line) and not an assertion failure (file:line prefix on the same line):
grep -qE '^[[:space:]]*context deadline exceeded[[:space:]]*$' "$tmpfile"
Workflow side looks right: the script is invoked as the loop body in both fuzz jobs (no other callers), there's no || true or continue-on-error anywhere near the fuzz command, go's exit code is preserved on the fail path, tee still streams the full log to the Actions log on the tolerated path, and the script is plain portable bash invoked explicitly. CI-only change, merge-tree clean, no production impact. Happy to approve as-is if you'd rather follow up the anchor separately, but it's a one-liner and I'd take it here.
What does this PR do?
Makes the fuzz jobs' timeout check strict. A failing
go test -fuzzrun now counts as a pass only when Go printedcontext deadline exceededand wrote no failing input (Failing input written to). Both thefuzzandfuzz-differentialjobs now call one sharedscripts/fuzz-run.shinstead of keeping two copies of the inlinefuzz_runfunction.Motivation
Ports the check from datadog-agent's
tasks/fuzz.py(DataDog/datadog-agent#58502, commitc37564820f4).The current check passes any failing run unless some line matches an indented
_test.go:N:. Real crashes don't print that line:testing.go:NNNN: panic: ..., and its stack frames end in_test.go:N +0x.., with no trailing colon../x_test.go:3:1: ...), so the leading-whitespace pattern misses it.So CI can go green on a real fuzz finding.
Testing
Reproduced locally with a throwaway fuzz target that writes to a nil map once the fuzzer finds the input:
scripts/fuzz-run.shFailing input written to testdata/fuzz/...)context deadline exceeded, no failing input (stubbedgo)Failing input written to(stubbed)The script is committed without the executable bit (the commit-signing path can't keep it), so the workflow runs it with
bash.Checklist
🤖 Generated with Claude Code