Repository navigation
cherry: procmon memory fix (#503, #507) - #508
Merged
Merged
Conversation
Audit events were kept in ProcessMonitor.Events forever. send() on a connected socket is audited as sendto without an address, so those events never became ready and were never released, and the agent grew by about 1.4 GiB per minute on a localhost-streaming workload (harden-runner#709). Under the same load the netlink socket overflowed (ENOBUFS), receive returned and audit monitoring stopped for the rest of the job. Use go-libaudit's Reassembler, which Auditbeat uses, with Auditbeat's defaults: at most 50 events in flight, a 2 second timeout and maintenance every 500 ms. Every event is delivered once, complete or evicted, so nothing is kept after delivery. The first qualifying PATH is kept, as before. Like Auditbeat, receive stops only when the client is closed: other errors, including ENOBUFS, are counted and receiving continues, and records with an invalid header are dropped. Sequence gaps and records whose fields cannot be parsed are counted and logged on powers of two. Bump go-libaudit from v2.3.2 to v2.6.2: v2.3.2 panics in parseSockaddr on a SOCKADDR record shorter than expected (elastic/go-libaudit#152), which stops the agent. This replaces the event map and idle sweep from #501, reverted in the previous commit. (cherry picked from commit 018a9e6)
go-libaudit v2.6.2 pins github.com/elastic/go-licenser in rule/tools.go behind the tools build tag. It is not built into the agent, but go mod vendor needs its checksum. (cherry picked from commit 3ac31cc)
go-libaudit's reassembler delivers events in sequence order. Records that are not part of an audited syscall, such as BPF program loads and unloads or ANOM_PROMISCUOUS, can arrive alone without a record that completes an event, so each one held every later event until it timed out (up to 2 s, or until 50 events were in flight). Handlers that look up the parent process in /proc can miss it once it has exited. Pass a record to the reassembler only if it is a SYSCALL record or belongs to an event whose SYSCALL record was passed. Every record the agent uses follows the SYSCALL record. Records that arrive after their event was delivered are dropped instead of starting a new event. Pushing, maintenance and the final flush take the same lock, so an event cannot be given up between a record being admitted and being pushed. (cherry picked from commit efd33da)
h0x0er
approved these changes
Nov 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of the procmon memory fix from
inttomain: #503 and #507.Commits (picked with
-x):ad2fe96fix(procmon): group audit records with go-libaudit's reassembler (int018a9e6, fix(procmon): group audit records with go-libaudit's reassembler #503)068ce96build: add go-libaudit v2.6.2 tool dependency to go.mod and go.sum (int3ac31cc, fix(procmon): group audit records with go-libaudit's reassembler #503)b14af87fix(procmon): only reassemble records of audited syscall events (intefd33da, fix(procmon): only reassemble records of audited syscall events #507)#503 also reverted #501 on
int. #501 was never onmain, so that revert is not included. Before #501, these files were identical onmainandint.After this PR,
procmon.go,procmon_linux.go,procmon_test.go,procmon_linux_test.go,go.modandgo.sumare identical toint(cce1d50).fileMonitorAuditRuleis defined once.Testing
-race(Go 1.27.3 container,Test_addAuditRulesskipped as it needs iptables): 48 pass.mainhas 29; the only test removed isTestProcessMonitor_PrepareEvent, replaced byTestEvent_addRecordin fix(procmon): group audit records with go-libaudit's reassembler #503.TestRunfails the same way onmainin the container.int, where it was tested end to end: see fix(procmon): group audit records with go-libaudit's reassembler #503 and fix(procmon): only reassemble records of audited syscall events #507 (memory flat under a heavy workload, detections unchanged, int canary workflows passing).