Skip to content

ci(fuzz): tolerate only Go's bare -fuzztime timeout, not real findings - #706

Merged
matt-dz merged 1 commit into
mainfrom
matthew.deguzman/fuzz-strict-timeout-check
Oct 29, 2026
Merged

matt-dz merged 1 commit into
mainfrom
matthew.deguzman/fuzz-strict-timeout-check

Conversation

@matt-dz

@matt-dz matt-dz commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Makes the fuzz jobs' timeout check strict. A failing go test -fuzz run now counts as a pass only when Go printed context deadline exceeded and wrote no failing input (Failing input written to). Both the fuzz and fuzz-differential jobs now call one shared scripts/fuzz-run.sh instead of keeping two copies of the inline fuzz_run function.

Motivation

Ports the check from datadog-agent's tasks/fuzz.py (DataDog/datadog-agent#58502, commit c37564820f4).

The current check passes any failing run unless some line matches an indented _test.go:N:. Real crashes don't print that line:

  • A panic is reported as testing.go:NNNN: panic: ..., and its stack frames end in _test.go:N +0x.., with no trailing colon.
  • A compile error starts at column 0 (./x_test.go:3:1: ...), so the leading-whitespace pattern misses it.

So CI can go green on a real fuzz finding.

Testing

Reproduced locally with a throwaway fuzz target that writes to a nil map once the fuzzer finds the input:

Case Old inline check scripts/fuzz-run.sh
Real panic (Failing input written to testdata/fuzz/...) exit 0 (tolerated as "boundary timeout") exit 1
Bare context deadline exceeded, no failing input (stubbed go) exit 0 exit 0
Timeout text plus Failing input written to (stubbed) n/a exit 1
Compile error (stubbed) exit 0 exit 1
Clean run exit 0 exit 0

The script is committed without the executable bit (the commit-signing path can't keep it), so the workflow runs it with bash.

Checklist

  • Tests added/updated
  • Documentation updated (if applicable)

🤖 Generated with Claude Code

The fuzz jobs treated any failing run as a coordinator boundary timeout
unless some line looked like an indented `_test.go:N:` reference. A
panic reports through `testing.go:N:` and its stack frames end in
`+0x..`, not a colon, so real crashes (and build errors) passed CI.
Reproduced locally: a fuzz target that writes to a nil map fails with
"Failing input written to testdata/fuzz/..." and the old check exits 0.

Port the stricter check from datadog-agent's tasks/fuzz.py
(DataDog/datadog-agent#58502): a failing run passes only when it printed
"context deadline exceeded" and Go wrote no failing input. Both fuzz
jobs now share scripts/fuzz-run.sh instead of two copies of the inline
function.

Environment: Datadog workspace

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@matt-dz
matt-dz marked this pull request as ready for review October 28, 2026 19:19
@matt-dz
matt-dz requested review from a team as code owners October 28, 2026 19:19
@matt-dz
matt-dz requested a review from san-jos October 28, 2026 19:19
@datadog-official

datadog-official Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Pipelines

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5075c3e | Docs | View more details | Give us feedback!

@julesmcrt julesmcrt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 5075c3eb8d4acc341e9f4270549e96ef15734a9c.
This is a real improvement to the fuzz gate's signal. The old inline fuzz_run tolerated any non-zero go test exit whose log lacked a *_test.go:NN: line — which meant a panic whose stack frames were all in production code, a worker hang, a coordinator internal failure, or an overall test-timeout panic were all silently relabeled "coordinator boundary timeout" and the job went green. The new script narrows tolerance to exactly the two-property signature of Go's bare -fuzztime cancellation: a non-zero exit whose log contains "context deadline exceeded" and does not contain "Failing input written to". Both properties check out against Go's source: every real crash/hang finding is funneled through the crashError path and always prints "Failing input written to" (even a hang whose termination suffix is itself a deadline error), while the bare boundary cancellation is a plain context error with no corpus write. I fed thirteen synthetic terminal-class logs through the new script with a mocked go binary: eight classes the old logic masked (production-frame crashes, hangs, internal failures, test-timeout panics, deadline-plus-failing-input mixes, near-empty kills) now fail the job with go's original exit code preserved, and only the genuine bare-timeout class is tolerated.
One residual boundary gap worth tightening before merge: grep -q 'context deadline exceeded' matches the string anywhere in the log. A real assertion failure on a seed-corpus entry prints foo_test.go:42: context deadline exceeded and — because seed inputs are already on disk — Go prints no "Failing input written to" for it, so both guards are satisfied and that finding would exit 0. The old inline logic actually failed this class, so it's a narrow regression. None of the current fuzz targets can produce that text in a failure message (the t.Fatal(err) paths only fire on file-write errors, the ip targets word timeouts as "timed out (possible hang)", and the others return early on t.Context().Err()), and deterministic seed failures are usually caught by the preceding seed-run step — so it isn't reachable today — but it's exactly the shape of bug this gate exists to catch. One-line fix, verified against the synthetic logs: anchor the pattern to the bare-message line shape, which matches the coordinator cancellation (message alone on its line) and not an assertion failure (file:line prefix on the same line):

grep -qE '^[[:space:]]*context deadline exceeded[[:space:]]*$' "$tmpfile"

Workflow side looks right: the script is invoked as the loop body in both fuzz jobs (no other callers), there's no || true or continue-on-error anywhere near the fuzz command, go's exit code is preserved on the fail path, tee still streams the full log to the Actions log on the tolerated path, and the script is plain portable bash invoked explicitly. CI-only change, merge-tree clean, no production impact. Happy to approve as-is if you'd rather follow up the anchor separately, but it's a one-liner and I'd take it here.

@matt-dz
matt-dz added this pull request to the merge queue Oct 29, 2026
Merged via the queue into main with commit a0fd4fb Oct 29, 2026
89 of 92 checks passed
@matt-dz
matt-dz deleted the matthew.deguzman/fuzz-strict-timeout-check branch October 29, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants